AI summary by TwinMind

Jay McClelland: Neural Networks and the Emergence of Cognition | Lex Fridman Podcast #222

Lex Fridman

2h 31m 57s

Introduction and Background

  • Jay McClelland: cognitive scientist at Stanford, seminal figure in AI/neural networks history
  • Co-wrote Parallel Distributed Processing book with David Rumelhart
  • Rumelhart co-authored backpropagation paper with Jeff Hinton
  • Their work foundational to neural network-based machine learning revolution of past 15 years

Beauty of Neural Networks

  • Most beautiful aspect: linking biology with mysteries of thought
  • Late 60s/early 70s: cognitive psychology emerging as field, book "Cognitive Psychology" (1967) claimed nervous system study only peripheral interest
  • McClelland disagreed - felt mind emerges from physical being ("dust to dust, ashes to ashes")
  • Concept of "awakening from the Cartesian dream"

Descartes and Mind-Body Problem

  • Descartes' hydraulic statue story at Versailles gardens - stepping on stones moved statues via water pressure
  • Developed mechanistic theory of animal behavior through hydraulic "fibers"
  • Believed humans had animal body plus divine element for thought
  • Physical world included body/action but not thought according to Descartes
  • Study of physiology focused on sensory/motor systems, cognition tied to abstract algorithms

Evolution and Intelligence

  • Darwin's struggle with complexity arguments (eye development)
  • Had nightmares about complexity, delayed publication due to fear of own ideas
  • 20th century: evolution accepted for producing range of animals
  • Continuity of species (apes, chimpanzees, bonobos) challenged Descartes' view
  • Huxley had to demonstrate hippocampus exists in chimpanzee brains (late 19th century)

Language and Human Intelligence

  • Chomsky argued genetic fluke 100,000 years ago created language capability
  • Language provided special separation from other animals
  • McClelland: language comes with evolution of sociality, mutual engagement, rich world-organizing mechanisms
  • Language as tool for collective intelligence
  • Evolution: punctuated equilibrium - long stasis periods followed by sudden changes
  • Similar pattern in child development (Piaget's stage theory)

Early Career and Neural Network Discovery

  • 1974: Assistant professorship at UCSD
  • Rumelhart had written "Explorations in Cognition" with Don Norman
  • Created spirit of playful exploration of ideas
  • James Anderson conference talk: used linear algebra to create neural network models of perception/categorization/memory
  • 1977: Walking across campus revelation - "if I think about the mind in terms of a neural network it will help me answer questions about the mind"

Parallel Distributed Processing Concepts

  • "Parallel": each neuron as independent computational unit
  • Billions of little computational units working simultaneously vs single central processor
  • Revolutionary compared to sequential computer science approaches
  • Convolutional neural networks: layers of thousands/tens of thousands of neurons
  • Each neuron contributes tiny bit to whole computation

David Rumelhart Background

  • From small town western South Dakota
  • Mother: librarian, Father: newspaper editor
  • Four competitive brothers encouraged by father
  • Standout undergraduate, went to Stanford for mathematical psychology PhD
  • Mathematical psychology: using mathematics/equations to model mental processes
  • One of first assistant professors at UCSD psychology department (1967)

Interactive Model of Reading

  • Rumelhart's breakthrough paper on reading comprehension
  • Every level of analysis depends on all other levels
  • Pixels → letters → words → ideas (hierarchical but interactive)
  • Top-down and bottom-up processing simultaneously
  • Experts at each level updating hypotheses at other levels
  • Completely interactive bi-directional parallel distributed process

Connectionism Definition

  • Knowledge exists in connections between units, not separate dictionary
  • Unit for word "time" defined by connections to constituent letters
  • Knowledge distributed across connection weights
  • Counter-intuitive - knowledge not explicitly stored anywhere

Semantic Dementia and Rumelhart's Decline

  • Progressive neurological condition affecting meaning comprehension
  • Starts with inability to differentiate similar categories (dog breeds)
  • Progresses to calling all medium animals "dogs", small ones "cats", large ones "horses"
  • Loss of conceptual grounding from all input modalities
  • Pyramids and Palm Trees test: can't relate concepts (pyramids go with palm trees vs pine trees)
  • Rumelhart suffered from this condition - tragic irony given his work on distributed representation
  • Unlike Hawking (motor system affected), Rumelhart's semantics/self-semantics affected
  • Could still bowl, navigate spatially, point to menu items but couldn't name them

PDP Research Group Formation (1982)

  • Hinton returned to UCSD on sabbatical
  • Formed PDP research group with McClelland and Rumelhart
  • Francis Crick joined from Salk Institute
  • Paul Smolensky and others also involved

Backpropagation Development

  • Hinton's key insight: stop looking to biology for learning rules
  • Instead: define problem, figure out how to adjust connection weights to solve it
  • Objective function approach - measure error/loss, minimize through weight adjustment
  • Gradient descent already known for single layer (Widrow-Hoff delta rule)
  • Rumelhart generalized to multiple layers - "generalized delta rule"
  • Later called "backpropagation" - error signals propagated backward through connections
  • Hinton initially skeptical, preferred Boltzmann machine approach
  • Backprop worked better than Boltzmann machine learning

Jeff Hinton's Contributions

  • Two 1981 papers: essentially transformers concept, early semantic cognition work
  • 1977 unpublished paper: neural networks doing recursive computation via fast connection weight changes
  • Visual/intuitive thinking style - draws pictures instead of equations in lab meetings
  • Uses geometric metaphors (hands showing ravine shape for gradient descent)
  • Descendant of logician Boole, long line of English academics
  • Wanted to contribute to reasoning itself, not just human reasoning
  • Boltzmann machine: connecting Boolean tradition with continuous probabilistic constraint satisfaction

Computational Intelligence

  • McClelland uses term "computational intelligence" to characterize field
  • Human-oriented computational intelligence researcher
  • Interested in human solution but also going beyond human nervous system limitations
  • Deep learning scaling human intelligence mechanisms through computational power

Mathematical Cognition

  • Mathematics as tools for exploring idealized worlds often relevant to real world
  • Objects with idealized properties, precise relationships
  • Natural numbers as cultural construction, not innate endowment
  • Tristan Needham quote: studying math as symbol manipulation like studying music without hearing notes
  • Mathematics: development of systems for thinking about properties/relations among idealized objects
  • Mathematical notation just way of expressing propositions, not deep representation

Mathematical Reasoning and Intuition

  • Poincaré quote: "by logic we prove, by intuition we discover"
  • Combination of intuitive connectionist knowledge with formal reasoning
  • Flash of insight from constellation of simultaneous constraints
  • Archimedes, Newton, Einstein, Hinton examples of sudden insights
  • Deep learning systems showing similar creative synthesis (text generation, novel story creation)
  • Alpha Zero chess: novel brilliant moves combining intuition with search

Formal vs Natural Thinking

  • 3000+ years of formal system development (Thales, Euclid's Elements)
  • Academia created modes of thought that become deeply ingrained
  • Chomsky example: born into academic family, deeply enculturated in formal reasoning
  • Lila Gleitman's discovery: linguists' intuitions about language differ from ordinary people
  • Formally trained academics have systematicity-organized engagement with experience
  • Different from natural human mind without formal immersion

Expert Blind Spot

  • Things self-evident to experts that they can't explain to non-experts
  • Kronecker: "God made natural numbers, all else is work of man" - wrong, natural numbers are cultural construction
  • Mathematical ability often attributed to biological differences rather than dinner table conversations
  • Beginner's mind vs expert limitations

Introspection and Self-Knowledge

  • Nisbet and Wilson paper: limits on awareness of factors causing our choices
  • Explanations of behavior often post-hoc rationalizations, not actual causal processes
  • Our insight into beliefs/attitudes/feelings subject to culturally transmitted understanding
  • Ability to introspect itself product of culture and belief practice

Life Advice

  • Find what you're intrinsically motivated to engage with
  • Celebrate that discovery when it happens
  • McClelland's wandering undergraduate years until 1968 Columbia revolution
  • Psychology offered chance to run own experiments (vs predetermined physics/chemistry labs)
  • Intrinsic motivation leads to immersion, which creates expertise opportunity
  • "Mozart effect": synergy of innate resonance with intensive experience
  • Different siblings found different passions despite shared environment

Mortality and Legacy

  • More afraid of degeneration than death itself
  • Fear of losing ability to engage at cutting edge of ideas
  • Motivated by limited time to tackle hard problems (mathematical cognition)
  • Legacy: contributing to science through collaborative opportunities
  • Joy of science: partially formed thoughts crystallizing through discourse into scientific progress
  • Interactive activation model and backpropagation as examples
  • Don't let others label you - continue growing beyond assigned categories

Meaning of Life

  • Humans make meaning for themselves
  • Inspired by meaning others made in past (religious thinkers of first millennium BC)
  • Emergent result of natural process without guidance
  • Meaning is what we make of it, not what we find
  • Very personal and individual - comes through particular combination of fabric, experience, context
  • Local context creates peculiar emergent processes that tell stories about themselves

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome