AI summary by TwinMind
Jitendra Malik: Computer Vision | Lex Fridman Podcast #110
Lex Fridman
1h 41m 36s
Introduction and Background
- Conversation with Jitendra Malik, Berkeley professor and seminal computer vision researcher
- Cited over 180,000 times, mentored many world-class researchers
- Discussion of sponsors: BetterHelp and ExpressVPN
Why Computer Vision is Underestimated
- 1966 Seymour Papert MIT Summer Vision Project - proposed solving computer vision in one summer
- Vision appears easy because most visual processing is subconscious/unconscious
- Large part of cerebral cortex devoted to visual processing in humans and primates
- "Fallacy of the successful first step" - 50% solution takes 1 minute, 90% takes 1 day, 99% takes 5 years, 99.99% may not happen in lifetime
- Vision requires understanding at all levels - sensation, perception, cognition are artificial divides
Autonomous Driving Challenges
- Malik is pessimist on fully autonomous driving in near future
- 0.01% of cases require sophisticated cognitive reasoning
- Tesla autopilot approach: 8 cameras, single neural network (HydraNet) with multiple heads
- Highway driving more solvable but still has edge cases
- Vision fundamental problem includes predicting behavior of other agents
- Example: Tesla system failing with skateboarder due to lack of understanding typical skateboarder behavior patterns
Learning Paradigms - Human vs Machine
- Current systems: tabula rasa supervised learning with lots of data
- Human driving: 16-year-old already visual genius from 0-16 years of development
- By age 2: understand 3D world, object perspectives, occlusion, dynamics, intuitive physics
- Driver education teaches control, not visual understanding
- Need evolution of learning techniques beyond current supervised learning
- Child learning includes: exploration, manipulation, self-supervised multi-modal signals
Computational Power and Efficiency
- Hans Moravec calculations from 1990s - may have comparable computing power to brain today
- Different style: GPUs far more power hungry than biological computation
- Neurons slow vs transistors but high connectivity vs lower connectivity in silicon
General Problem of Computer Vision
- Fundamental purpose: perception to guide action (evolved from simple creatures 500 million years ago)
- Perception-action loops originally for finding food, avoiding predators
- Modern humans have "hyper-evolved" visual system for broader purposes
- Scientists impose structure: object detection, 3D reconstruction modules
- All processes run simultaneously, originally to guide action
Static vs Dynamic Vision
- Much progress on static images due to historical compute/storage limitations
- Video recognition "feels like 10 years behind object recognition"
- Performance on challenging video datasets ~30% (similar to 2009 object detection)
- Video computing still challenging even for large companies like Google/Facebook
Three R's of Computer Vision
- Recognition: Labeling (cat, dog, specific breeds, abstract categories)
- Reconstruction: Inverse graphics - building internal 3D model from 2D images
- Reorganization: Creating entities/structure rather than uniform pixel grid
- All three interact in triangle relationship, sharing common neural network representations
Bottom-up vs Top-down Processing
- Current systems: purely feed-forward (bottom-up) at test time
- Biology: shallower networks (~7 layers retina to cortex) with feedback connections
- Current training: too much top-down supervision (hand-labeled boundaries)
- Humans use bottom-up cues like motion (common fate principle)
- Need both approaches: less supervision in training, more feedback at test time
Learning from Children - Six Lessons
- From Smith & Gasser paper: be multimodal, incremental, physical, explore, social, use language
- Multimodal example: Child holding ball - tactile and visual signals self-calibrate
- Audio-visual learning: glass breaking provides synchronized signals across modalities
- Physical embodiment: Real robotics and simulation environments (Habitat project)
- Active learning enables breaking correlation vs causation barrier through controlled experiments
Segmentation and Scene Understanding
- Segmentation: identifying pixel groups as objects without necessarily naming them
- Enables weak supervision - child can map words to pre-segmented objects
- Medical applications: precise delineation of brain bleeds in CT scans
- Not semantics-free: involves perception and cognition, revised with experience
Vision Benchmarks and Tests
- Current benchmarks test "adult" capabilities rather than child-like learning
- Need data collection of child's visual environment (privacy challenges)
- Good vision test: aid to the blind - ultimate performance in real world
- Hilbert problems of computer vision: long-form video understanding with goals, intentionality, prediction
3D Understanding Challenges
- Two disconnected approaches: multi-view geometry vs single-view supervised learning
- Need learning-as-you-move approach: build 3D representations from visual experiences
- Learn implicit "CAD models" from seeing objects from different viewpoints
- Then predict 3D from single novel views
Explainability and Black Boxes
- Humans themselves are black boxes to each other
- Explainability matters in high-stakes settings (medical diagnosis)
- Important for understanding error bounds, data bias, reference class problems
- Willing to sacrifice interpretability for high performance in many applications
Language vs Vision Fundamentality
- Vision more fundamental than language (contra Chomsky)
- Phylogeny: Vision 500M years ago, bipedalism/tool use ~5M years ago, language after that
- Ontogeny: Child develops spatial intelligence before language
- Language builds on substrate of space, time, causality from perception/action
- All human languages have constructs depending on spatial/temporal concepts
Intelligence Testing
- Against single test like Turing test - prefer multiple capabilities
- Rather have 10 different tasks: manipulation, navigation, visual understanding, reading comprehension
- Favorite language test: reading novel and answering arbitrary questions
- Vision test: comprehensive aid to blind person
Future of AI and Existential Risk
- Two answers: in principle yes, pragmatically not in next 20 years
- Known unknowns vs unknown unknowns (Donald Rumsfeld reference)
- Vision/robotics: known unknowns; NLP/high-level cognition: unknown unknowns
- Positive surprise possible (like deep learning success despite non-convex optimization)
- AI risks exist today, not future AGI: biases, self-driving car deaths, recommendation algorithms
- Continuous responsibility as systems deployed, not sudden future problem
Mentoring and Research Philosophy
- Lucky to work in field during productive phase (34 years at Berkeley)
- Key mentoring contribution: sense of picking right problems
- Peter Medawar: "research is art of the soluble" - problems with "soft underbelly"
- Intellectual breadth connecting psychology, neuroscience, applied math
- Students bring narrow depth, mentor provides shallow breadth
