AI summary by TwinMind
Vladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence | Lex Fridman Podcast #71
Lex Fridman
1h 44m 55s
Introduction and Background
- Conversation with Vladimir Vapnik (APNIC), co-inventor of support vector machines
- Previously worked at Institute of Control Sciences Moscow, AT&T, NEC Labs, Facebook AI Research
- Currently professor at Columbia University, cited over 200,000 times
- Discussion follows his MIT lecture on "Complete Statistical Theory of Learning"
Engineering vs. Science of Intelligence
- Engineering intelligence: imitation of human activity, creating devices that behave like humans
- Science of intelligence: understanding what intelligence actually is
- Relates to Vladimir Propp's work: found 31 predicates/units explaining human behavior in Russian tales
- Same predicates apply to movies, TV serials, theater - universally applicable
Predicates and Plato's Theory of Forms
- Predicates as functions from input space (e.g., pixel space for images)
- Belief in Plato's world of ideas - vault of pure ideas exists
- Intelligence combines ideas with reality to create invariances
- Example predicates for 2D images:
- Symmetry (various types: vertical, horizontal, diagonal, anti-symmetry)
- Structure (formal definition of spatial relationships)
- Predicates are universal, not task-specific
Symmetry as Core Predicate
- Four types of symmetry discussed
- Degree of symmetry important, not just binary existence
- Example: Letter 'S' has vertical anti-symmetry
- Symmetry helps create invariants for classification tasks
Convergence Theory
- Strong convergence: functions converge pointwise with small squared difference
- Weak convergence: convergence in inner product for any function in Hilbert space
- Weak convergence more powerful - creates admissible sets of functions
- Key insight: need uniform convergence for learning, not just convergence
Admissible Sets of Functions
- Goal: reduce huge set of functions to smaller admissible set
- Predicates help select functions that maintain specific properties
- Good predicates significantly reduce VC dimension
- Fewer training examples needed with good predicates
Deep Learning Critique
- Deep learning uses single predicate (convolution) for translational invariance
- Neural networks create piecewise linear functions - subset of all possible functions
- Better to explicitly specify admissible functions using predicates
- Current approach uses heuristics rather than principled predicate selection
Handwritten Digit Recognition Challenge
- Proposes achieving state-of-the-art MNIST performance using 100x fewer examples (600 instead of 60,000)
- Requires discovering good predicates for visual understanding
- Success would demonstrate intelligence principles
- Predicates should be universal for images, not digit-specific
Discovering Good Predicates
- Cannot be done automatically - infinite predicates exist in Hilbert space
- Requires human insight like Vladimir Propp's literary analysis
- Process similar to physics: find contradictions, formulate new predicates
- Good predicates significantly reduce admissible function sets
Philosophy and Mathematics Connection
- Philosophy provides understanding of life at abstract level
- Plato's forms connect to mathematical convergence theory
- Implementation follows philosophical understanding
- Took 50 years to realize importance of weak convergence
Music and Art as Predicate Sources
- Music critics use predicates to describe compositions
- Bach vs. Chopin require different vocabularies/predicates
- Art criticism might inform visual predicates discovery
- Collaborated with Russian poetry expert to describe digits poetically
Future Directions
- Focus on simple problems first (digit recognition)
- Avoid complex problems like language until fundamentals solved
- Collect descriptions from music/art critics to find common predicates
- Language understanding not feasible in 21st century
Personal Philosophy
- Modest about own contributions
- Appreciates brilliance in literature, art, life understanding
- Believes in focusing on solvable problems
- Views mortality pragmatically - hopes to continue working
