AI summary by TwinMind

Anca Dragan: Human-Robot Interaction and Reward Engineering | Lex Fridman Podcast #81

Lex Fridman

1h 38m 33s

Introduction and Background

  • Anca Dragan, UC Berkeley professor working on human robot interaction algorithms
  • Focus on robot behavior that accounts for interaction and coordination with humans
  • Also consults at Waymo autonomous vehicle company
  • Personal journey: Started with math competitions in Romania, moved to programming, then AI and robotics
  • Applied to Carnegie Mellon Robotics Institute - only acceptance after Berkeley and Stanford rejections

Early Robotics Experiences and Inspiration

  • Transformative moment: Riding in Google self-driving car (Firefly) at RSS 2014 in Berkeley
  • Another magical robot experience: Boston Dynamics Spot Mini - anthropomorphism effect
  • Favorite fictional robot: Wall-E due to expressiveness through motion and cameras
  • Personal connection: Husband proposed using actuated Wall-E robot with 7 degrees of freedom
  • Early programming: Started in 4th grade with QBasic, drawing circles
  • Math Olympiad background in Romania from 5th grade

Expressivity and Anthropomorphism in Robots

  • Challenge of creating expressive robots like Wall-E
  • Easy in narrow settings with handcrafted animations (like Anki's Cosmo)
  • Hard part: Doing it autonomously and generally across different tasks
  • Robots can communicate internal states through movement style
  • Example: Kids being rude to Alexa because it doesn't react - need for robots to respond appropriately
  • Expanding state space to include human internal state and perceptions

Human Complexity in Robot Interaction

  • Two main changes when moving from isolated robot tasks to human interaction:
    1. Multiple agents acting in same space (coordination problem)
    2. Robot must optimize for human preferences, not programmer specifications
  • Understanding humans requires both prediction (anticipating actions) and inference (understanding preferences/intent)
  • Human behavior assumed to be optimal with respect to their goals

Inverse Reinforcement Learning and Human Models

  • Inverse reinforcement learning: Infer reward function from human behavior demonstrations
  • Based on utility maximization from economics (1940s)
  • Boltzmann rationality: Accounts for human noise and approximation (late 1950s)
  • Success with simple models until hitting complex tasks like lunar lander control
  • People appear much more noisy and irrational in difficult tasks

Beyond Simple Rationality Models

  • Behavioral economics (1970s): People are messy, emotional, irrational with domain-specific heuristics
  • Alternative approach: Humans may be rational but under different assumptions about world
  • Example: Lunar lander - people may operate under simpler intuitive physics models
  • Can infer human's intuitive physics model and correct their actions accordingly
  • Bounded rationality: People work under computational constraints

Active Information Gathering

  • Robot doesn't need to passively observe - can act to gather information
  • Example: Autonomous car testing driving style of nearby human driver
  • Car can nudge toward lane change to see if human slows down (defensive) or maintains speed (aggressive)
  • Information gathering actions help disambiguate human intentions and preferences

Game Theoretic Aspects

  • Human-robot interaction as game theory problem
  • Both agents have overlapping but not identical objectives
  • Example: Both want to avoid accidents, but have different destinations and timing preferences
  • Under-actuated system perspective: Robot influences but doesn't control human degrees of freedom

Driving and Human Complexity

  • Removing all humans from driving scenario makes problem much simpler
  • Perception challenges exist but significant progress made
  • Main difficulty is human element in driving
  • Current systems (like Waymo) successfully operating in constrained domains
  • Question: How quickly can we expand to more complex environments like downtown San Francisco

Learning vs Rule-Based Systems

  • Learning is inevitable but pure end-to-end learning has limitations
  • Planning, search, and optimization remain crucial for decision-making
  • Need combination: Learning for world models, planning for behavior generation
  • Simulation useful but requires good human models (combination of learned and constructed)

Semi-Autonomous Systems

  • Level 2 autonomy systems (Tesla Autopilot, Cadillac Super Cruise) with human supervision
  • Concern: Assumption that supervising human performs as well as actively driving human
  • Technical issue: Robot takes human to "off-policy" states they wouldn't normally encounter
  • Human factors issue: Passive supervision vs active engagement creates different cognitive states
  • Potential: If designed properly, human-robot team could outperform either alone

Reward Function Design Challenges

  • Even without human interaction, specifying good reward functions is extremely difficult
  • Need to anticipate all possible situations and ensure specified rewards incentivize desired behavior
  • Current approach: Sample representative situations and tune until optimal behavior matches desired behavior
  • Moved complexity from behavior specification to reward specification

Collaborative Reward Design

  • Human-robot collaboration on reward function design itself
  • Designer's specified reward as evidence of preferences, not absolute truth
  • Robot maintains uncertainty about true reward function
  • Multiple information sources: demonstrations, corrections, environment state, physical interactions

Leaked Information and Physical Interaction

  • Humans "leak" reward information through various channels
  • Physical corrections (pushing robot away) provide reward signal
  • E-stop button presses indicate very negative reward for intended action
  • Environment state itself contains information about human preferences (shoes arranged in line)

Universal Principles vs Customization

  • Three Laws of Robotics too simplistic - need mathematical precision
  • Should be constantly adjusting rather than fixed universal laws
  • Interpret designer specifications as evidence in context, not universal truths
  • Robot maintains uncertainty and continuously learns from multiple signals

Personal Philosophy and Influences

  • Key book: Russell & Norvig "AI: A Modern Approach" in 12th grade
  • Captivated by autonomous goal achievement through messy situations
  • Approach: Write math about human behavior, combine with data, give to robots for autonomous planning
  • Preference for existing due to life's finite nature
  • Meaning of life: Impact local communities, be there for other humans
  • Small acts of kindness: High school physics teacher Nicole McConnell provided free tutoring and SAT prep

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome