AI summary by TwinMind

Jitendra Malik: Computer Vision | Lex Fridman Podcast #110

Lex Fridman

1h 41m 36s

Introduction and Background

  • Conversation with Jitendra Malik, Berkeley professor and seminal computer vision researcher
  • Cited over 180,000 times, mentored many world-class researchers
  • Discussion of sponsors: BetterHelp and ExpressVPN

Why Computer Vision is Underestimated

  • 1966 Seymour Papert MIT Summer Vision Project - proposed solving computer vision in one summer
  • Vision appears easy because most visual processing is subconscious/unconscious
  • Large part of cerebral cortex devoted to visual processing in humans and primates
  • "Fallacy of the successful first step" - 50% solution takes 1 minute, 90% takes 1 day, 99% takes 5 years, 99.99% may not happen in lifetime
  • Vision requires understanding at all levels - sensation, perception, cognition are artificial divides

Autonomous Driving Challenges

  • Malik is pessimist on fully autonomous driving in near future
  • 0.01% of cases require sophisticated cognitive reasoning
  • Tesla autopilot approach: 8 cameras, single neural network (HydraNet) with multiple heads
  • Highway driving more solvable but still has edge cases
  • Vision fundamental problem includes predicting behavior of other agents
  • Example: Tesla system failing with skateboarder due to lack of understanding typical skateboarder behavior patterns

Learning Paradigms - Human vs Machine

  • Current systems: tabula rasa supervised learning with lots of data
  • Human driving: 16-year-old already visual genius from 0-16 years of development
  • By age 2: understand 3D world, object perspectives, occlusion, dynamics, intuitive physics
  • Driver education teaches control, not visual understanding
  • Need evolution of learning techniques beyond current supervised learning
  • Child learning includes: exploration, manipulation, self-supervised multi-modal signals

Computational Power and Efficiency

  • Hans Moravec calculations from 1990s - may have comparable computing power to brain today
  • Different style: GPUs far more power hungry than biological computation
  • Neurons slow vs transistors but high connectivity vs lower connectivity in silicon

General Problem of Computer Vision

  • Fundamental purpose: perception to guide action (evolved from simple creatures 500 million years ago)
  • Perception-action loops originally for finding food, avoiding predators
  • Modern humans have "hyper-evolved" visual system for broader purposes
  • Scientists impose structure: object detection, 3D reconstruction modules
  • All processes run simultaneously, originally to guide action

Static vs Dynamic Vision

  • Much progress on static images due to historical compute/storage limitations
  • Video recognition "feels like 10 years behind object recognition"
  • Performance on challenging video datasets ~30% (similar to 2009 object detection)
  • Video computing still challenging even for large companies like Google/Facebook

Three R's of Computer Vision

  • Recognition: Labeling (cat, dog, specific breeds, abstract categories)
  • Reconstruction: Inverse graphics - building internal 3D model from 2D images
  • Reorganization: Creating entities/structure rather than uniform pixel grid
  • All three interact in triangle relationship, sharing common neural network representations

Bottom-up vs Top-down Processing

  • Current systems: purely feed-forward (bottom-up) at test time
  • Biology: shallower networks (~7 layers retina to cortex) with feedback connections
  • Current training: too much top-down supervision (hand-labeled boundaries)
  • Humans use bottom-up cues like motion (common fate principle)
  • Need both approaches: less supervision in training, more feedback at test time

Learning from Children - Six Lessons

  • From Smith & Gasser paper: be multimodal, incremental, physical, explore, social, use language
  • Multimodal example: Child holding ball - tactile and visual signals self-calibrate
  • Audio-visual learning: glass breaking provides synchronized signals across modalities
  • Physical embodiment: Real robotics and simulation environments (Habitat project)
  • Active learning enables breaking correlation vs causation barrier through controlled experiments

Segmentation and Scene Understanding

  • Segmentation: identifying pixel groups as objects without necessarily naming them
  • Enables weak supervision - child can map words to pre-segmented objects
  • Medical applications: precise delineation of brain bleeds in CT scans
  • Not semantics-free: involves perception and cognition, revised with experience

Vision Benchmarks and Tests

  • Current benchmarks test "adult" capabilities rather than child-like learning
  • Need data collection of child's visual environment (privacy challenges)
  • Good vision test: aid to the blind - ultimate performance in real world
  • Hilbert problems of computer vision: long-form video understanding with goals, intentionality, prediction

3D Understanding Challenges

  • Two disconnected approaches: multi-view geometry vs single-view supervised learning
  • Need learning-as-you-move approach: build 3D representations from visual experiences
  • Learn implicit "CAD models" from seeing objects from different viewpoints
  • Then predict 3D from single novel views

Explainability and Black Boxes

  • Humans themselves are black boxes to each other
  • Explainability matters in high-stakes settings (medical diagnosis)
  • Important for understanding error bounds, data bias, reference class problems
  • Willing to sacrifice interpretability for high performance in many applications

Language vs Vision Fundamentality

  • Vision more fundamental than language (contra Chomsky)
  • Phylogeny: Vision 500M years ago, bipedalism/tool use ~5M years ago, language after that
  • Ontogeny: Child develops spatial intelligence before language
  • Language builds on substrate of space, time, causality from perception/action
  • All human languages have constructs depending on spatial/temporal concepts

Intelligence Testing

  • Against single test like Turing test - prefer multiple capabilities
  • Rather have 10 different tasks: manipulation, navigation, visual understanding, reading comprehension
  • Favorite language test: reading novel and answering arbitrary questions
  • Vision test: comprehensive aid to blind person

Future of AI and Existential Risk

  • Two answers: in principle yes, pragmatically not in next 20 years
  • Known unknowns vs unknown unknowns (Donald Rumsfeld reference)
  • Vision/robotics: known unknowns; NLP/high-level cognition: unknown unknowns
  • Positive surprise possible (like deep learning success despite non-convex optimization)
  • AI risks exist today, not future AGI: biases, self-driving car deaths, recommendation algorithms
  • Continuous responsibility as systems deployed, not sudden future problem

Mentoring and Research Philosophy

  • Lucky to work in field during productive phase (34 years at Berkeley)
  • Key mentoring contribution: sense of picking right problems
  • Peter Medawar: "research is art of the soluble" - problems with "soft underbelly"
  • Intellectual breadth connecting psychology, neuroscience, applied math
  • Students bring narrow depth, mentor provides shallow breadth

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome