AI summary by TwinMind
How To Build Generative AI Models Like OpenAI's Sora
Y Combinator
34m 5s
Sora Video Generation Analysis
- OpenAI's Sora represents major breakthrough in text-to-video generation
- Demonstrated with custom prompts: 2050 humanoid robot walking golden retriever, drone footage of Golden Gate Bridge
- Key improvements over previous models:
- Accurate text spelling in videos (major advance from image models like Stable Diffusion, DALL-E)
- Realistic physics simulation (robot walking mechanics, dog movement)
- Long-term visual consistency across 1-minute videos
- Architectural coherence and world continuity
- Technical limitations identified:
- Floating objects appearing in scenes
- Geographic inaccuracies (Golden Gate Bridge terrain)
- Cars driving on wrong side of road
- Imperfect fluid simulation for water/waves
- Structural discontinuities in bridge columns
Sora Technical Architecture
- Combination of transformer model (typically used for text) + diffusion model (used in DALL-E, Midjourney)
- Uses "SpaceTime patches" - 3x3 pixel matrices with spatial and temporal components
- Variable patch sizes in XYZ dimensions
- Built on prior work: Google's "Image is Worth 16x16" (2020) visual transformer paper
- Incorporates World Model concepts from 2018 robotics papers
- Estimated 10x more computationally expensive than GPT-4
- GPT-4: ~1 trillion parameters, Sora estimated: ~10 trillion parameters
- Requires ~10x more GPUs than GPT-4 (which used 20-30K GPUs)
YC Companies Building Foundation Models with $500K
Infinity AI (Winter 24 batch)
- Creates deepfake videos of specific people (e.g., Elon Musk replica)
- Trained model on just ~1 hour of YouTube video data from Light Cone podcast
- Generated realistic AI replica for podcast demonstration
- Built by current YC batch companies during 3-month program
SyncLabs
- Real-time lip syncing API
- Trained models on single A100 GPU
- Demonstrated accurate Hindi lip syncing on English speaker
- Data optimization: compressed video to lower resolution (241p vs 1080p = quadratic reduction in data)
- Leveraged YC's Azure GPU cluster deal for 100x faster iteration
Sonado (Winter 24 batch)
- Text-to-song model with artist voice replication
- Only 2-3 models worldwide with this capability, potentially best performance
- Founded by 21-year-old college graduates
- Built entire system in 2 months during YC batch
- Self-taught through online AI papers, no PhD ML background required
MetalWare
- Hardware design co-pilot
- Founded by former SpaceX hardware engineers, no AI background initially
- Used high-quality textbook data instead of massive datasets
- Chose smaller GPT-2.5 model (1 billion parameters vs GPT-4's trillion)
- Proved smaller model + better data can outperform larger models
GuaB
- Building explainable foundation models
- Addresses "black box" problem in deep learning
- Trained model during YC batch to explain model outputs
- Enables understanding of feature weights and decision processes
Find
- Software co-pilot generating answers better than Stack Overflow
- Founded by college graduates without extensive AI background
- Used synthetic data generation for programming competitions
- Infinite high-quality synthetic data generation approach
Synthetic Data Revolution
- Previously controversial concept ("mosquito drinking its own blood" meme)
- Now proven effective for model training
- Enables infinite high-quality dataset generation
- Works due to LLM reasoning capabilities creating positive feedback loop
- Similar to self-driving car simulation training (10:1 simulation to real data ratio)
- Sora likely uses Unreal Engine/Unity generated footage for physics training
YC Infrastructure Advantages
- Azure GPU cluster partnership providing $500K+ credits
- 24-hour access to dedicated GPU resources
- No resource contention unlike public cloud providers
- Companies didn't need to touch YC's $500K investment for model training
- Eliminated typical GPU access barriers (prepayment, 2025 delivery times)
Physics Simulation Applications Beyond Entertainment
Atmo Weather Prediction
- Foundation model approach 1 million times more efficient than traditional physics calculations
- Seed-stage startup outperforming NOAA's $1B+ weather model
- Demonstrates AI replacing expensive physics-based simulations
Biotech Applications
Theseus Bio
- Generative AI for protein and drug molecule design
- Founded by AI expert (published in Nature)
- Custom GPU kernels for faster foundation model training
- Specialized in gene therapies and new drug discovery
Pyramidal Neurotech
- Foundation model for human brain/EEG signal prediction
- EEG data treated like video (electrical impulses over time)
- SpaceTime chunking approach similar to Sora
- Applications: stroke prediction, potential brain reading
- Reduced runtime complexity quadratically through data chunking
- Single model iteration: 800 hours GPU compute
Engineering Applications
DraftAI
- CAD design AI models
- Replaces expensive legacy kernels (FORTRAN-based)
- Accelerates physics calculations (force, shear, tolerances)
- Shortcuts traditional computational geometry processes
Strategic Insights for Founders
When to Build vs Buy Models
- Build own model for specific/niche applications (like MetalWare hardware)
- Use existing models (GPT-4) for general language tasks
- Domain expertise can overcome compute/data limitations
Success Factors Analysis
- Expertise: Often overrated - smart founders can self-teach through papers
- Compute: YC partnership solves access problems
- Data: High-quality, domain-specific data beats massive generic datasets
- Synthetic data generation emerging as key competitive advantage
Competitive Positioning
- Specialized foundation models can compete with OpenAI in verticals
- Example: Playground AI (founder Suhail Doshi) competing with Midjourney
- Hard pivot success story: Suhail locked himself away for month reading AI papers
- Field so new that 6-9 months intensive study can reach cutting edge
Robotics Renaissance
- Historical context: OpenAI originally focused on robotics (Rubik's cube robots)
- Transformer architecture was initially side project
- Sora's physics simulation could enable AI robotics breakthrough
- K-Scale Labs (current YC batch): consumer humanoid robots
- Founder Ben built Tesla's Optimus robotics foundation model
Market Opportunity Assessment
- Many valuable AI applications remain undiscovered (EEG example)
- SpaceTime concept applicable to diverse domains beyond video
- Traditional industries ripe for AI disruption (weather, CAD, biotech)
- Founders shouldn't fear competing with well-funded AI companies
- Specialized models can outperform general-purpose solutions in verticals
