AI summary by TwinMind

Jim Fan on Nvidia’s Embodied AI Lab and Jensen Huang’s Prediction that All Robots will be Autonomous

Sequoia Capital

49m 14s

Jim Fan's Background and Journey to Embodied AI

  • First intern at OpenAI (summer 2016)
    • Mentored by Andrej Karpathy and Ilya Sutskever
    • Worked on "World of Bits" project - AI agent reading computer screens, controlling keyboard/mouse
    • Part of larger OpenAI Universe initiative
    • Used reinforcement learning (no Transformers existed in 2016)
    • Challenge: worked on specific tasks but didn't generalize to arbitrary language instructions
  • Stanford PhD (2016-2021) with Fei-Fei Li
    • Witnessed transition from static computer vision to embodied computer vision
    • Agents learning perception and taking actions in interactive environments (virtual simulation or physical world)
  • Joined Nvidia after PhD graduation, continuing embodied AI work

Nvidia's Embodied AI Initiative - GEAR Team

  • GEAR = Generalist Embodied Agent Research
  • Mission: "generate actions" - build embodied AI agents taking actions in different worlds
    • Virtual world actions = gaming AI and simulation
    • Physical world actions = robotics
  • Project Groot unveiled at GTC March 2024
    • Nvidia's moonshot effort building foundation models for humanoid robotics
    • Goal: AI brain for humanoid robots and beyond

Nvidia's Competitive Advantages in Robotics

  • Compute resources for scaling foundation models
    • Belief in scaling laws for embodied AI and robotics (yet to be studied)
  • Simulation expertise
    • Nvidia's graphics company heritage - years of physics simulation, rendering, real-time GPU acceleration
    • Simulation heavily used in robotics approach

Three-Bucket Data Strategy for Robot Foundation Models

  • Internet-scale data: Text and videos online
    • Most diverse, encode common sense prior
    • Human-centered content (selfies, instructional videos)
    • Learn human-object interactions and object behaviors
    • Weakness: no action/motor control signals available
  • Simulation data: Nvidia simulation tools generating synthetic data
    • Provides actions and consequences of actions
    • Infinite data scaling with compute
    • 10,000x real-time acceleration possible
    • Weakness: sim-to-real gap, physics/visuals differences, limited diversity
  • Real robot data: Teleoperated robot data collection
    • No sim-to-real gap
    • Weakness: expensive, limited by 24 hours/day, requires human operators
  • Strategy: Combine complementary strengths, remove individual weaknesses

Vision for Robotics Breakthrough Timeline

  • Research breakthrough in robot foundation models: 2-3 years ("GPT-3 moment for robotics")
  • Mass market adoption: longer timeline due to affordability, mass production, safety, privacy, regulations

Defining the "GPT-3 Moment" for Robotics

  • System 1 vs System 2 thinking (from "Thinking Fast and Slow")
    • System 1: Low-level motor control, unconscious, fast (e.g., grasping cup)
    • System 2: Slow, deliberate reasoning and planning
  • GPT-3 moment will be System 1 breakthrough
  • Example: Understanding verb "open" across different contexts
    • Opening door vs window vs bottle vs phone - different motions, same concept
    • Need model generalizing low-level motor control for abstract verbs
  • System 2 already strong with existing LLMs (reasoning, planning, coding)
  • Integration challenge: System 1 operates at 1000Hz, System 2 at 1Hz
  • Open question: Monolithic model vs cascaded approach with communication between systems

Scaling Law Hypothesis for Embodied AI

  • Haven't pushed Transformer limits yet for robotics
  • Transformers fundamentally take tokens in, output tokens
  • Quality of tokens determines model quality
  • Complex robotics data strategy (internet + simulation + real robot data)
  • Once high-quality action data tokenized and scaled, expect emerging properties
  • Optimistic about Transformer-based scaling approach

Humanoid Robot Focus Rationale

  • World design: Everything built for human form factor
    • Restaurants, factories, hospitals, equipment, tools designed for human hands/body
    • Sufficiently good humanoid hardware can support any reasonable human task
  • Data availability: Abundant human-centered online content
    • Videos of humans doing daily tasks, having fun
    • Humanoid form factor closest to human - easier transfer learning
    • Few videos of robot arms/grippers online vs many videos of five-finger hands
  • Generality first: Train for full generality, then specialize to robot arms/specific forms

Current Robotics Landscape and "Why Now"

  • Hardware improvements: Surge of new robot hardware since late 2023
    • Tesla Optimus, Boston Dynamics, many startups
    • Better dexterous hands, whole-body reliability
  • Pricing revolution:
    • 2001: NASA humanoid robot >$1.5M
    • Recent: $30,000 full-fledged humanoids (comparable to car price)
    • Raw materials only 4% of car cost - potential for exponential price decrease
  • Foundation model advances:
    • System 2 reasoning/planning addressed by frontier models (GPTs, Claude, Llama)
    • Multimodal models improving computer vision/perception
    • Coding abilities leveraged for robot solutions (Eureka project)

Eureka and Dr. Eureka Research Projects

  • Eureka: LLM writes reward function code in Isaac Sim API
    • Automates tedious manual reward function engineering
    • Demonstrated with five-finger robot hand pen spinning (superhuman performance)
    • General technique for complex task specification
  • Dr. Eureka: Extension using domain randomization
    • LLM writes code specifying simulation randomization parameters
    • Trains across 10,000 different simulation configurations
    • Zero-shot transfer to real world without fine-tuning
    • Demo: Robot dog walking on yoga ball (better than real dogs)

Virtual Worlds and Gaming AI Research

  • MineDojo: Platform for general-purpose Minecraft agents
    • Collected internet data: gameplay videos, Wiki pages, Reddit discussions
    • Multimodal datasets for training Minecraft-playing models
  • Voyager: GPT-4 powered Minecraft agent using code as action
    • Converts 3D Minecraft world to text representation
    • Agent writes code using action APIs
    • Self-reflection loop for error correction
    • Skill library saves successful programs for reuse
    • Automated curriculum - agent proposes next appropriate difficulty tasks
    • Emerging property: discovers novel items/tools autonomously

Foundation Agent Vision

  • Single model working across virtual and physical agents
  • Three generalization axes:
    • Skills: Different capabilities
    • Embodiments: Different body forms/platforms
    • Worlds: Different realities (virtual/physical)
  • Physical world as "10,001st virtual simulation" in domain randomization approach

Gaming AI Future Vision

  • Intelligent NPCs:
    • Non-player characters with memory, personality, dynamic behavior
    • Interactive conversations changing narrative/story
    • Infinite replay value with unique experiences
  • Generated game worlds:
    • Text-to-3D asset generation
    • Text-to-video models
    • Language agents for storylines
    • Real-time world generation during gameplay
  • Current models (Llama-level) not quite sufficient - need more diverse, engaging behaviors
  • Inference cost challenge for deployment (cloud hosting vs local device)

Alternative Architectures and Future Directions

  • Haven't pushed Transformer limits for robotics yet
  • Data remains primary bottleneck (action data not downloadable from internet)
  • Interested in alternatives: Mamba, test-time training
    • More efficient inference mechanisms
    • Need scaling to frontier model sizes for proper comparison
  • Video generation as world simulator
    • Learning physics and rendering from data alone
    • Data-driven simulation for embodied AI training

Long-term AI Vision (10+ years)

  • Coding agents: Human-level software engineers accelerating development
  • Humanoid robots: Reliability and agility matching/exceeding humans
  • Economic impact: Robots in households (laundry, cooking, elderly care), restaurants, hospitals, factories
  • Jensen's quote: "Everything that moves will eventually be autonomous"
  • Vision: As many intelligent robots as iPhones within 10+ years

Research Philosophy and Advice

  • Importance of identifying right problems vs solving problems
  • Keep up with accessible research literature
  • Try open-source tools and simulators
  • Get hands dirty with practical experimentation
  • Long-term vision thinking - work on future needs today

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome