AI summary by TwinMind
Decart’s Dean Leitersdorf on AI-Generated Video Games and Worlds
Sequoia Capital
46m 35s
Oasis Launch and Technology Overview
- Launched Oasis few weeks ago - first real-time interactive video model
- Key breakthrough: real-time video model responding to user actions
- Move around world, break/place blocks
- Game without traditional game engine
- Vision extends beyond Oasis 1 to future versions (Oasis 3)
- "Magical mirror" concept - voice commands create visual changes
- Example: request sword, change size/color, transform environment to Game of Thrones
- Connects imagination to visual output on screens
Company Background and Leadership
- Dean Leitersdorf: youngest PhD from Technion at 23 (brother beat record at 21)
- Grew up between Israel and United States
- Company focus: delivering delightful AI experiences
- Fully vertically integrated from CUDA kernels to model training to user experiences
- Sequoia Capital partners Sean McGuire and Sonia Huang conducting interview
Fundamental Technology Philosophy
- Comparison to ChatGPT's impact: overcomes communication barriers
- ChatGPT bridges human unstructured language with computer structured language
- Oasis bridges visual communication barrier between humans and computers
- Platform approach: solve fundamental limitation, not specific problems
- Enables building multiple applications on top
- From next-gen Snapchat/TikTok to fighter pilot simulators
Technical Architecture and Real-Time Implementation
- Model similar to Sora but prompt is user actions instead of text
- Input: keyboard actions + past frames
- Output: next frame generation
- Data preprocessing: label gameplay recordings with corresponding actions
- Small model trained for action labeling
- Team recorded gameplay data for training
- Pure pixel representation vs world models
- End-to-end approach: keystroke to frame
- Alternative: text-to-3D with existing game engines
- Pixel approach more flexible for live editing
Real-Time Inference Breakthrough
- Industry belief: real-time video required Blackwell chips, H100s insufficient
- Decart achieved real-time on H100s through dual optimization:
- Model architecture changes for real-time capability
- Systems-level optimization: custom CUDA kernels, rewrote PyTorch components
- Training convergence: 20 hours vs industry standard 2 weeks
- Vertical integration essential for both model and systems work
Training Infrastructure Challenges
- 200-page internal documentation of training issues
- GPU cluster reliability problems across industry
- Clusters crash every few hours
- Random failures from dust on cables between nodes
- Only Google and OpenAI have stable training infrastructure
- Example failure: synthetic data generation caused network bandwidth issues
- Led to Python data loader lock file conflicts
- Cascading system failures from seemingly unrelated operations
- Built distributed file system when clusters lacked storage
- Utilized clusters at 200% capacity (GPUs, CPUs, InfiniBand networking)
Vertical Integration Strategy
- Comparison to Google's early advantage
- Google's distributed systems expertise with commodity hardware
- 10x cost advantage over competitors using expensive Sun Microsystems servers
- Fault tolerance through software vs expensive hardware
- Current AI training similar to 1990s CPU reliability issues
- Advantages of vertical integration:
- Cost reduction/higher margins
- Speed to market (1-2 years ahead of competitors)
- Essential for real-time performance delivery
Game Engine Disruption Potential
- Traditional game engines: one person creates world, others interact
- AI-powered approach enables natural language modifications
- "Turn this into Elsa themed" - instant transformation
- "Add flying elephant" - interactive elements appear
- Modding revolution: much faster iteration, interactive modifications
- Programming through words instead of code
- Example: Pokemon mod for Minecraft already exists
Future Technical Architecture
- Current: prompt-to-pixels (keystrokes to pixels)
- Future convergence: two-model approach
- Model 1: Transformer managing game state (unrelated to pixels)
- Model 2: Renders state to pixels
- Combines advantages of world models and diffusion models
- Plans to build both models
Market Positioning and Competition
- Three major AI company opportunities identified:
- Nvidia competitor (next-gen AI chip)
- Next AWS (new cloud for changing workloads)
- New experiences (chosen path)
- Character AI case study: TikTok as new moat
- Weak network effect through social media presence
- User awareness concentrated on single platform
- Brand/distribution advantages in social media age
Long-term Vision and Moats
- Short-term: technical moat through vertical integration
- Long-term: hardware moats most durable
- Google's optical interconnects example (Jupiter Rising project)
- 30% data center power consumption reduction
- Hardware deployment timescales create lasting advantages
- "Generated Experiences" (GX) company
- Term coined by James from Sequoia
- "UX is dead, Long Live GX"
- Platform for human-computer interaction matching natural preferences
Personal Background
- First programming at 13: RuneScape bots
- Favorite scientist: Leonardo da Vinci
- Combination of scientist, engineer, and fundraiser
- Tesla and Descartes as close seconds
- Company name inspiration: Descartes' "I think therefore I am" relates to AI
- Favorite AI apps: ChatGPT and Character AI
- Prediction: AI-generated creative platforms before fully AI-generated content
Timeline and Expectations
- Company timeline: minimum 5 years to major impact
- Potential for faster development than previous tech titans
- Focus on fundamental limitation solving vs problem solving
- No celebration until major victory achieved
