AI summary by TwinMind

Decart’s Dean Leitersdorf on AI-Generated Video Games and Worlds

Sequoia Capital

46m 35s

Oasis Launch and Technology Overview

  • Launched Oasis few weeks ago - first real-time interactive video model
  • Key breakthrough: real-time video model responding to user actions
    • Move around world, break/place blocks
    • Game without traditional game engine
  • Vision extends beyond Oasis 1 to future versions (Oasis 3)
    • "Magical mirror" concept - voice commands create visual changes
    • Example: request sword, change size/color, transform environment to Game of Thrones
    • Connects imagination to visual output on screens

Company Background and Leadership

  • Dean Leitersdorf: youngest PhD from Technion at 23 (brother beat record at 21)
  • Grew up between Israel and United States
  • Company focus: delivering delightful AI experiences
  • Fully vertically integrated from CUDA kernels to model training to user experiences
  • Sequoia Capital partners Sean McGuire and Sonia Huang conducting interview

Fundamental Technology Philosophy

  • Comparison to ChatGPT's impact: overcomes communication barriers
    • ChatGPT bridges human unstructured language with computer structured language
    • Oasis bridges visual communication barrier between humans and computers
  • Platform approach: solve fundamental limitation, not specific problems
    • Enables building multiple applications on top
    • From next-gen Snapchat/TikTok to fighter pilot simulators

Technical Architecture and Real-Time Implementation

  • Model similar to Sora but prompt is user actions instead of text
    • Input: keyboard actions + past frames
    • Output: next frame generation
  • Data preprocessing: label gameplay recordings with corresponding actions
    • Small model trained for action labeling
    • Team recorded gameplay data for training
  • Pure pixel representation vs world models
    • End-to-end approach: keystroke to frame
    • Alternative: text-to-3D with existing game engines
    • Pixel approach more flexible for live editing

Real-Time Inference Breakthrough

  • Industry belief: real-time video required Blackwell chips, H100s insufficient
  • Decart achieved real-time on H100s through dual optimization:
    • Model architecture changes for real-time capability
    • Systems-level optimization: custom CUDA kernels, rewrote PyTorch components
  • Training convergence: 20 hours vs industry standard 2 weeks
  • Vertical integration essential for both model and systems work

Training Infrastructure Challenges

  • 200-page internal documentation of training issues
  • GPU cluster reliability problems across industry
    • Clusters crash every few hours
    • Random failures from dust on cables between nodes
    • Only Google and OpenAI have stable training infrastructure
  • Example failure: synthetic data generation caused network bandwidth issues
    • Led to Python data loader lock file conflicts
    • Cascading system failures from seemingly unrelated operations
  • Built distributed file system when clusters lacked storage
  • Utilized clusters at 200% capacity (GPUs, CPUs, InfiniBand networking)

Vertical Integration Strategy

  • Comparison to Google's early advantage
    • Google's distributed systems expertise with commodity hardware
    • 10x cost advantage over competitors using expensive Sun Microsystems servers
    • Fault tolerance through software vs expensive hardware
  • Current AI training similar to 1990s CPU reliability issues
  • Advantages of vertical integration:
    • Cost reduction/higher margins
    • Speed to market (1-2 years ahead of competitors)
    • Essential for real-time performance delivery

Game Engine Disruption Potential

  • Traditional game engines: one person creates world, others interact
  • AI-powered approach enables natural language modifications
    • "Turn this into Elsa themed" - instant transformation
    • "Add flying elephant" - interactive elements appear
  • Modding revolution: much faster iteration, interactive modifications
  • Programming through words instead of code
  • Example: Pokemon mod for Minecraft already exists

Future Technical Architecture

  • Current: prompt-to-pixels (keystrokes to pixels)
  • Future convergence: two-model approach
    • Model 1: Transformer managing game state (unrelated to pixels)
    • Model 2: Renders state to pixels
    • Combines advantages of world models and diffusion models
  • Plans to build both models

Market Positioning and Competition

  • Three major AI company opportunities identified:
    • Nvidia competitor (next-gen AI chip)
    • Next AWS (new cloud for changing workloads)
    • New experiences (chosen path)
  • Character AI case study: TikTok as new moat
    • Weak network effect through social media presence
    • User awareness concentrated on single platform
    • Brand/distribution advantages in social media age

Long-term Vision and Moats

  • Short-term: technical moat through vertical integration
  • Long-term: hardware moats most durable
    • Google's optical interconnects example (Jupiter Rising project)
    • 30% data center power consumption reduction
    • Hardware deployment timescales create lasting advantages
  • "Generated Experiences" (GX) company
    • Term coined by James from Sequoia
    • "UX is dead, Long Live GX"
    • Platform for human-computer interaction matching natural preferences

Personal Background

  • First programming at 13: RuneScape bots
  • Favorite scientist: Leonardo da Vinci
    • Combination of scientist, engineer, and fundraiser
    • Tesla and Descartes as close seconds
  • Company name inspiration: Descartes' "I think therefore I am" relates to AI
  • Favorite AI apps: ChatGPT and Character AI
  • Prediction: AI-generated creative platforms before fully AI-generated content

Timeline and Expectations

  • Company timeline: minimum 5 years to major impact
  • Potential for faster development than previous tech titans
  • Focus on fundamental limitation solving vs problem solving
  • No celebration until major victory achieved

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome