AI summary by TwinMind

Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics | Lex Fridman Podcast #241

Lex Fridman

3h 14m 14s

Boris Sofman Background & Introduction

  • Senior Director of Engineering and Head of Trucking at Waymo (autonomous vehicle company, formerly Google self-driving car project)
  • Previously co-founder and CEO of Anki, robotics company that created Cosmo robot
  • PhD from Carnegie Mellon University Robotics Institute
  • Discusses favorite fictional robots: Wall-E and R2D2 for emotional communication without language; Terminator series for spectrum of AI capabilities

Anki Origins & Vision

  • Started Anki while PhD students at CMU Robotics Institute (2011-2013)
  • Focused on consumer applications of robotics and AI
  • Motivated by cost reduction in components due to smartphone industry
  • iPhone (2007) enabled new interface possibilities for physical world experiences
  • Strategy: push complexity from physical hardware into software/AI
  • Entertainment chosen as starting domain due to low innovation, forgiving environment for mistakes

Cosmo Robot Design Philosophy

  • Core Constraint: Only 4 degrees of freedom - 2 treads for movement, head up/down, lift up/down, plus sound and low-resolution screen
  • Character-driven approach over game-driven approach
  • Intentionally non-humanoid form to avoid uncanny valley and cost constraints
  • Inspired by Pixar's Wall-E - "bringing a Pixar character to life in the real world"
  • Cost of goods under $50, retail under $200
  • Emphasis on emotional connection through minimal design elements

Character Development & Animation

  • Partnered with Carlos Baena (9 years at Pixar, worked on Wall-E)
  • Created parameterized emotional response library across dimensions: happy, sad, angry, surprised, confused, scared
  • Behavioral engine mapped real-world context to probability distributions of appropriate responses
  • Eye contact experiments: doubling eye contact frequency increased play time engagement by 40%
  • Games served to create context for emotional responses, not primary entertainment
  • Mistakes became endearing when robot showed appropriate emotional reaction

Technical Implementation

  • Maya animation tools with plugin for real-time physical robot testing
  • Fusion of three key elements: hardware, AI, and character
  • Early fusion of sensor data (camera, microphones) rather than late fusion
  • Intentionally avoided voice to maintain universal appeal across demographics and ages
  • Randomness and variability built in to maintain illusion of life
  • Statistics tracked on emotional response usage to expand underutilized areas

Anki Business Challenges & Closure (2019)

  • Sold millions of units, reached close to $100M annual revenue
  • 85% of volume concentrated in Q4 (holiday season) creating brutal cash flow challenges
  • Expensive to market and explain new product category
  • Hardware companies facing valuation challenges (1x revenue multiples)
  • Final financing round fell through, company wound down
  • Team stayed together, many joined Waymo
  • Received extensive fan mail from customers, especially families with autistic children

Waymo Overview & Structure

  • Waymo Driver: core autonomous driving system across different vehicle platforms
  • Waymo One: consumer transportation/ride sharing
  • Waymo Via: goods transportation/trucking
  • 5th generation hardware and software system
  • Leverage between passenger vehicle and trucking applications

Autonomous Trucking Vision

  • Starting with L4 autonomy for Class 8 trucks (53-foot trailers)
  • Transfer hub model: hub-to-hub autonomous operation, manual last-mile initially
  • Dallas facility opening Q1 as first fully ground-up designed driverless hub
  • Routes: Dallas-Houston, Dallas-Phoenix, Dallas-Austin
  • Long-term: expand to surface streets, different vehicle types, broader network

Trucking Industry Context

  • 80,000+ truck driver shortage, expected to grow to hundreds of thousands
  • Some fleets have 95% attrition rate
  • Average truck driver age approaching 50
  • Long-haul routes most dangerous and personally challenging for drivers
  • 1.3 million miles between serious accidents, 28 million miles between serious injuries
  • Opportunity to optimize human utilization and improve working conditions

Technical Challenges - Trucking vs Cars

  • Sensor suite: ~2x more sensors than cars, main sensor pods positioned where mirrors would be (not center-mounted due to trailer occlusion)
  • Extended range requirements: need to see much further due to 80,000 pound braking profile
  • Dynamic constraints: jackknifing, rollover risks, load-dependent handling characteristics
  • Testing: facility with training wheels to push trucks past safety limits to define boundaries
  • Unique blind spots: trailer creates different occlusion patterns than passenger vehicles

Machine Learning & Data Strategy

  • Massive ML problems across perception, behavior prediction, planning
  • Heavy reuse of car-side data for trucking models
  • 85% of behavior prediction data comes from passenger vehicles
  • Early fusion approach for sensor integration (lidar, camera, radar)
  • Simulation: 1000 miles simulated for every mile driven physically
  • Data pipeline and labeling workflows shared between car and truck programs

Sensor Technology & Philosophy

  • Lidar: Waymo develops own lidar in-house, provides consistency and precision
  • Cameras: High resolution, long range capability (past 1km), but localization challenges at distance
  • Radar: Penetrates weather, provides speed information, longer range than lidar
  • Fusion approach: Not thinking of problems as sensor-specific but as fusion problems
  • Redundancy: Multiple sensor pods provide built-in failure tolerance

Safety & Validation Framework

  • L4 definition: fully autonomous in well-defined operating domain
  • Evaluation is the hardest problem in autonomous driving
  • Human accident rates: serious freeway accident every 1.3M miles for trucks
  • Portfolio of approaches: structured testing, statistical arguments, simulation
  • Borrowing validation methods from aerospace industry
  • Challenge of continuous software updates while maintaining safety bar

Tesla vs Waymo Comparison

  • Waymo advantages: L4-focused from beginning, custom sensor suite, deployed driverless service, 5th generation hardware learnings
  • Tesla advantages: Massive data collection through deployed fleet, business model funds R&D
  • Key difference: Waymo uses lidar+camera+radar, Tesla vision-only
  • Both approaches face significant challenges, timeline uncertain
  • Perception long-tail problems particularly challenging for vision-only approach

Robotics Industry Challenges

  • Most robotics companies fail due to technology-solution mismatch
  • Consumer products especially difficult: must hit function AND price point
  • Mass market appeal requires 2M+ units, not just enthusiast adoption
  • Product-market fit critical - easy to get outside viable price range
  • Entertainment chosen for Anki because low existing quality allowed leapfrogging competition

Humanoid Robotics Perspective

  • Currently more research than practical application
  • Wheels more efficient than legs for most tasks
  • Humanoid form often pursuit of form factor rather than application
  • Tesla Bot factory application questionable - would optimize for specific tasks rather than human form
  • Human-robot interaction potentially compelling use case for humanoid form
  • Boston Dynamics impressive technically but application unclear

Future of Robotics & Society

  • Autonomous driving will feel natural in hindsight, like internet/mobile phones
  • Will enable distributed living/working, reduce urban parking needs
  • Supply chain and logistics revolution potential
  • 95% of accidents preventable with automation strengths
  • Job displacement concerns, but trucking has existing shortage
  • Long-haul routes (most dangerous/undesirable) automated first
  • Short-haul/urban routes (more pleasant) remain human longer

Career & Life Advice

  • Find overlap between passion and growing market opportunity
  • Machine learning as versatile foundation skill set
  • Balance obsessive work dedication with relationships and family
  • Robotics attracts collaborative, passionate people due to historical non-lucrative nature
  • Startup advice: market timing often dominates team quality and execution
  • Co-founders essential for emotional support through brutal ups/downs
  • Consumer robotics requires exceptional clarity on value proposition

Home Robotics & Amazon Astro

  • Skeptical of Amazon Astro success at $1500 price point
  • Mobility advantage not utilized effectively
  • Character development likely underinvested
  • Trust and personality challenges for mobile home robots
  • Roomba succeeded by anchoring on familiar vacuum cleaner concept
  • Need more thoughtful stepping stones in robotics applications

PhD & Academic Advice

  • Robotics requires breadth across multiple disciplines
  • Collaborative nature due to system complexity
  • CMU experience included large team projects (UPI off-road autonomous vehicle)
  • Field moving from academic to commercial focus
  • Organizational challenges as significant as technical challenges
  • No established modularity/APIs like operating systems
  • Multi-dimensional problems strain human mental capacity for system design

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome