AI summary by TwinMind
Anca Dragan: Human-Robot Interaction and Reward Engineering | Lex Fridman Podcast #81
Lex Fridman
1h 38m 33s
Introduction and Background
- Anca Dragan, UC Berkeley professor working on human robot interaction algorithms
- Focus on robot behavior that accounts for interaction and coordination with humans
- Also consults at Waymo autonomous vehicle company
- Personal journey: Started with math competitions in Romania, moved to programming, then AI and robotics
- Applied to Carnegie Mellon Robotics Institute - only acceptance after Berkeley and Stanford rejections
Early Robotics Experiences and Inspiration
- Transformative moment: Riding in Google self-driving car (Firefly) at RSS 2014 in Berkeley
- Another magical robot experience: Boston Dynamics Spot Mini - anthropomorphism effect
- Favorite fictional robot: Wall-E due to expressiveness through motion and cameras
- Personal connection: Husband proposed using actuated Wall-E robot with 7 degrees of freedom
- Early programming: Started in 4th grade with QBasic, drawing circles
- Math Olympiad background in Romania from 5th grade
Expressivity and Anthropomorphism in Robots
- Challenge of creating expressive robots like Wall-E
- Easy in narrow settings with handcrafted animations (like Anki's Cosmo)
- Hard part: Doing it autonomously and generally across different tasks
- Robots can communicate internal states through movement style
- Example: Kids being rude to Alexa because it doesn't react - need for robots to respond appropriately
- Expanding state space to include human internal state and perceptions
Human Complexity in Robot Interaction
- Two main changes when moving from isolated robot tasks to human interaction:
- Multiple agents acting in same space (coordination problem)
- Robot must optimize for human preferences, not programmer specifications
- Understanding humans requires both prediction (anticipating actions) and inference (understanding preferences/intent)
- Human behavior assumed to be optimal with respect to their goals
Inverse Reinforcement Learning and Human Models
- Inverse reinforcement learning: Infer reward function from human behavior demonstrations
- Based on utility maximization from economics (1940s)
- Boltzmann rationality: Accounts for human noise and approximation (late 1950s)
- Success with simple models until hitting complex tasks like lunar lander control
- People appear much more noisy and irrational in difficult tasks
Beyond Simple Rationality Models
- Behavioral economics (1970s): People are messy, emotional, irrational with domain-specific heuristics
- Alternative approach: Humans may be rational but under different assumptions about world
- Example: Lunar lander - people may operate under simpler intuitive physics models
- Can infer human's intuitive physics model and correct their actions accordingly
- Bounded rationality: People work under computational constraints
Active Information Gathering
- Robot doesn't need to passively observe - can act to gather information
- Example: Autonomous car testing driving style of nearby human driver
- Car can nudge toward lane change to see if human slows down (defensive) or maintains speed (aggressive)
- Information gathering actions help disambiguate human intentions and preferences
Game Theoretic Aspects
- Human-robot interaction as game theory problem
- Both agents have overlapping but not identical objectives
- Example: Both want to avoid accidents, but have different destinations and timing preferences
- Under-actuated system perspective: Robot influences but doesn't control human degrees of freedom
Driving and Human Complexity
- Removing all humans from driving scenario makes problem much simpler
- Perception challenges exist but significant progress made
- Main difficulty is human element in driving
- Current systems (like Waymo) successfully operating in constrained domains
- Question: How quickly can we expand to more complex environments like downtown San Francisco
Learning vs Rule-Based Systems
- Learning is inevitable but pure end-to-end learning has limitations
- Planning, search, and optimization remain crucial for decision-making
- Need combination: Learning for world models, planning for behavior generation
- Simulation useful but requires good human models (combination of learned and constructed)
Semi-Autonomous Systems
- Level 2 autonomy systems (Tesla Autopilot, Cadillac Super Cruise) with human supervision
- Concern: Assumption that supervising human performs as well as actively driving human
- Technical issue: Robot takes human to "off-policy" states they wouldn't normally encounter
- Human factors issue: Passive supervision vs active engagement creates different cognitive states
- Potential: If designed properly, human-robot team could outperform either alone
Reward Function Design Challenges
- Even without human interaction, specifying good reward functions is extremely difficult
- Need to anticipate all possible situations and ensure specified rewards incentivize desired behavior
- Current approach: Sample representative situations and tune until optimal behavior matches desired behavior
- Moved complexity from behavior specification to reward specification
Collaborative Reward Design
- Human-robot collaboration on reward function design itself
- Designer's specified reward as evidence of preferences, not absolute truth
- Robot maintains uncertainty about true reward function
- Multiple information sources: demonstrations, corrections, environment state, physical interactions
Leaked Information and Physical Interaction
- Humans "leak" reward information through various channels
- Physical corrections (pushing robot away) provide reward signal
- E-stop button presses indicate very negative reward for intended action
- Environment state itself contains information about human preferences (shoes arranged in line)
Universal Principles vs Customization
- Three Laws of Robotics too simplistic - need mathematical precision
- Should be constantly adjusting rather than fixed universal laws
- Interpret designer specifications as evidence in context, not universal truths
- Robot maintains uncertainty and continuously learns from multiple signals
Personal Philosophy and Influences
- Key book: Russell & Norvig "AI: A Modern Approach" in 12th grade
- Captivated by autonomous goal achievement through messy situations
- Approach: Write math about human behavior, combine with data, give to robots for autonomous planning
- Preference for existing due to life's finite nature
- Meaning of life: Impact local communities, be there for other humans
- Small acts of kindness: High school physics teacher Nicole McConnell provided free tutoring and SAT prep
