AI summary by TwinMind

How To Build Generative AI Models Like OpenAI's Sora

Y Combinator

34m 5s

Sora Video Generation Analysis

  • OpenAI's Sora represents major breakthrough in text-to-video generation
  • Demonstrated with custom prompts: 2050 humanoid robot walking golden retriever, drone footage of Golden Gate Bridge
  • Key improvements over previous models:
    • Accurate text spelling in videos (major advance from image models like Stable Diffusion, DALL-E)
    • Realistic physics simulation (robot walking mechanics, dog movement)
    • Long-term visual consistency across 1-minute videos
    • Architectural coherence and world continuity
  • Technical limitations identified:
    • Floating objects appearing in scenes
    • Geographic inaccuracies (Golden Gate Bridge terrain)
    • Cars driving on wrong side of road
    • Imperfect fluid simulation for water/waves
    • Structural discontinuities in bridge columns

Sora Technical Architecture

  • Combination of transformer model (typically used for text) + diffusion model (used in DALL-E, Midjourney)
  • Uses "SpaceTime patches" - 3x3 pixel matrices with spatial and temporal components
  • Variable patch sizes in XYZ dimensions
  • Built on prior work: Google's "Image is Worth 16x16" (2020) visual transformer paper
  • Incorporates World Model concepts from 2018 robotics papers
  • Estimated 10x more computationally expensive than GPT-4
  • GPT-4: ~1 trillion parameters, Sora estimated: ~10 trillion parameters
  • Requires ~10x more GPUs than GPT-4 (which used 20-30K GPUs)

YC Companies Building Foundation Models with $500K

Infinity AI (Winter 24 batch)

  • Creates deepfake videos of specific people (e.g., Elon Musk replica)
  • Trained model on just ~1 hour of YouTube video data from Light Cone podcast
  • Generated realistic AI replica for podcast demonstration
  • Built by current YC batch companies during 3-month program

SyncLabs

  • Real-time lip syncing API
  • Trained models on single A100 GPU
  • Demonstrated accurate Hindi lip syncing on English speaker
  • Data optimization: compressed video to lower resolution (241p vs 1080p = quadratic reduction in data)
  • Leveraged YC's Azure GPU cluster deal for 100x faster iteration

Sonado (Winter 24 batch)

  • Text-to-song model with artist voice replication
  • Only 2-3 models worldwide with this capability, potentially best performance
  • Founded by 21-year-old college graduates
  • Built entire system in 2 months during YC batch
  • Self-taught through online AI papers, no PhD ML background required

MetalWare

  • Hardware design co-pilot
  • Founded by former SpaceX hardware engineers, no AI background initially
  • Used high-quality textbook data instead of massive datasets
  • Chose smaller GPT-2.5 model (1 billion parameters vs GPT-4's trillion)
  • Proved smaller model + better data can outperform larger models

GuaB

  • Building explainable foundation models
  • Addresses "black box" problem in deep learning
  • Trained model during YC batch to explain model outputs
  • Enables understanding of feature weights and decision processes

Find

  • Software co-pilot generating answers better than Stack Overflow
  • Founded by college graduates without extensive AI background
  • Used synthetic data generation for programming competitions
  • Infinite high-quality synthetic data generation approach

Synthetic Data Revolution

  • Previously controversial concept ("mosquito drinking its own blood" meme)
  • Now proven effective for model training
  • Enables infinite high-quality dataset generation
  • Works due to LLM reasoning capabilities creating positive feedback loop
  • Similar to self-driving car simulation training (10:1 simulation to real data ratio)
  • Sora likely uses Unreal Engine/Unity generated footage for physics training

YC Infrastructure Advantages

  • Azure GPU cluster partnership providing $500K+ credits
  • 24-hour access to dedicated GPU resources
  • No resource contention unlike public cloud providers
  • Companies didn't need to touch YC's $500K investment for model training
  • Eliminated typical GPU access barriers (prepayment, 2025 delivery times)

Physics Simulation Applications Beyond Entertainment

Atmo Weather Prediction

  • Foundation model approach 1 million times more efficient than traditional physics calculations
  • Seed-stage startup outperforming NOAA's $1B+ weather model
  • Demonstrates AI replacing expensive physics-based simulations

Biotech Applications

Theseus Bio

  • Generative AI for protein and drug molecule design
  • Founded by AI expert (published in Nature)
  • Custom GPU kernels for faster foundation model training
  • Specialized in gene therapies and new drug discovery

Pyramidal Neurotech

  • Foundation model for human brain/EEG signal prediction
  • EEG data treated like video (electrical impulses over time)
  • SpaceTime chunking approach similar to Sora
  • Applications: stroke prediction, potential brain reading
  • Reduced runtime complexity quadratically through data chunking
  • Single model iteration: 800 hours GPU compute

Engineering Applications

DraftAI

  • CAD design AI models
  • Replaces expensive legacy kernels (FORTRAN-based)
  • Accelerates physics calculations (force, shear, tolerances)
  • Shortcuts traditional computational geometry processes

Strategic Insights for Founders

When to Build vs Buy Models

  • Build own model for specific/niche applications (like MetalWare hardware)
  • Use existing models (GPT-4) for general language tasks
  • Domain expertise can overcome compute/data limitations

Success Factors Analysis

  • Expertise: Often overrated - smart founders can self-teach through papers
  • Compute: YC partnership solves access problems
  • Data: High-quality, domain-specific data beats massive generic datasets
  • Synthetic data generation emerging as key competitive advantage

Competitive Positioning

  • Specialized foundation models can compete with OpenAI in verticals
  • Example: Playground AI (founder Suhail Doshi) competing with Midjourney
  • Hard pivot success story: Suhail locked himself away for month reading AI papers
  • Field so new that 6-9 months intensive study can reach cutting edge

Robotics Renaissance

  • Historical context: OpenAI originally focused on robotics (Rubik's cube robots)
  • Transformer architecture was initially side project
  • Sora's physics simulation could enable AI robotics breakthrough
  • K-Scale Labs (current YC batch): consumer humanoid robots
  • Founder Ben built Tesla's Optimus robotics foundation model

Market Opportunity Assessment

  • Many valuable AI applications remain undiscovered (EEG example)
  • SpaceTime concept applicable to diverse domains beyond video
  • Traditional industries ripe for AI disruption (weather, CAD, biotech)
  • Founders shouldn't fear competing with well-funded AI companies
  • Specialized models can outperform general-purpose solutions in verticals

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome