AI summary by TwinMind

How to Scale AI Application Inference 100x ft. Fireworks’ Lin Qiao

Sequoia Capital

10m 18s

Introduction and Company Overview

  • Lynn Chow presenting on behalf of Fireworks AI
  • Fireworks positioned as leading, reliable, performant inference provider globally
  • Focus on reliability, scale, performance for AI inference

Future State of Inference Through Developer Lens

  • Perspective from app developers and innovative product developers
  • Development journey: build/ideation → scale after product-market fit
  • Core concept: alignment process towards audience throughout entire development cycle

Multi-Phase Alignment Process

  • User Behavior Optimization: Drive specific user behaviors through product design
  • Product Design Alignment: Traditional area with established tools (product analytics, logging)
  • Model Integration Challenge: New weak area as models become intrinsic to product stack
    • Most app developers use off-the-shelf models
    • Limited knowledge on infusing product knowledge into models
    • Primary steering method: prompt engineering
  • Data Flywheel: Key differentiator for successful applications and products

Application-Specific Alignment Gap

  • Core Problem: Align data distribution in application workload with training data distribution
  • Research Assumptions: Researchers make assumptions about model usage that don't fully connect with real-life applications
  • Guaranteed Gap: Almost always exists between assumptions and actual use cases
  • Inference System Design: Not one-size-fits-all, requires production data for optimization

Three-Dimensional Scaling Law

  • Traditional Focus: Scaling law in training
  • Future Focus: Scaling law across three dimensions in inference:
    • Quality
    • Speed
    • User concurrency (cost)
  • Customer Demands: "OpenAI quality, light speed, high concurrency" (fraud detection-level performance)
  • Solution Approach: Multi-dimensional optimization problem specific to each application

Co-Optimization Strategy

  • Key Innovation: Combine post-training and inference for co-optimization
  • Goal: Drive down inference costs by 10x to 100x from current state
  • Market Impact: "Iceberg situation" - when costs drop dramatically, applications achieving product-market fit can scale to sustainable businesses

Technical Complexity Challenges

  • Multi-Token Prediction: Predict 10 tokens at once instead of one token at a time
  • Numeric Alignment: Align positions towards application data distribution
  • Hardware Selection: Each hardware type has unique advantages for different data distributions
  • Model Sharding: Optimize towards application distribution
  • Cross-Host Distributed Inference: Optimize for application distribution
  • Kernel Optimization: Select most optimized kernel for application distribution
  • Quality Tuning: Various tuning mechanisms for application-specific quality
  • Combinatorial Explosion: Over 100,000 combinations to choose from

Fireworks Platform Architecture

  • Virtual Cloud Infrastructure: Manages complexity of GPU sourcing with guaranteed high quality/reliability
  • Multi-Hardware Support: Different vendors, SKUs with varying capabilities
    • Some hardware: higher FLOPS
    • Some hardware: higher memory bandwidth
    • Optimized for different application performance requirements
  • Open Model Libraries: Easy selection and steering towards application needs
  • Multiple Tuning Mechanisms: Speed tuning, quality tuning, various approaches
  • Production Data Integration: Reinforcement tuning using production data for application-specific needs
  • Developer Tools: Comprehensive suite for accessibility

Platform Launch and Success Metrics

  • Launch Timeline: Developer-facing self-serve platform launched 1.5 years ago
  • Food Chain Company Case Study: Scaled from 1 shop AI feature to 1,000 shops in 3 months
  • Software Development Company Case Study: Scaled from 100,000 developers to 25 million developers in 3 months
  • Optimization Goal: Achieve optimal sweet spot across quality, cost, and speed during scaling

Platform Accessibility

  • Easy onboarding and experimentation process
  • Self-serve platform for developer access

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome