AI summary by TwinMind
How to Scale AI Application Inference 100x ft. Fireworks’ Lin Qiao
Sequoia Capital
10m 18s
Introduction and Company Overview
- Lynn Chow presenting on behalf of Fireworks AI
- Fireworks positioned as leading, reliable, performant inference provider globally
- Focus on reliability, scale, performance for AI inference
Future State of Inference Through Developer Lens
- Perspective from app developers and innovative product developers
- Development journey: build/ideation → scale after product-market fit
- Core concept: alignment process towards audience throughout entire development cycle
Multi-Phase Alignment Process
- User Behavior Optimization: Drive specific user behaviors through product design
- Product Design Alignment: Traditional area with established tools (product analytics, logging)
- Model Integration Challenge: New weak area as models become intrinsic to product stack
- Most app developers use off-the-shelf models
- Limited knowledge on infusing product knowledge into models
- Primary steering method: prompt engineering
- Data Flywheel: Key differentiator for successful applications and products
Application-Specific Alignment Gap
- Core Problem: Align data distribution in application workload with training data distribution
- Research Assumptions: Researchers make assumptions about model usage that don't fully connect with real-life applications
- Guaranteed Gap: Almost always exists between assumptions and actual use cases
- Inference System Design: Not one-size-fits-all, requires production data for optimization
Three-Dimensional Scaling Law
- Traditional Focus: Scaling law in training
- Future Focus: Scaling law across three dimensions in inference:
- Quality
- Speed
- User concurrency (cost)
- Customer Demands: "OpenAI quality, light speed, high concurrency" (fraud detection-level performance)
- Solution Approach: Multi-dimensional optimization problem specific to each application
Co-Optimization Strategy
- Key Innovation: Combine post-training and inference for co-optimization
- Goal: Drive down inference costs by 10x to 100x from current state
- Market Impact: "Iceberg situation" - when costs drop dramatically, applications achieving product-market fit can scale to sustainable businesses
Technical Complexity Challenges
- Multi-Token Prediction: Predict 10 tokens at once instead of one token at a time
- Numeric Alignment: Align positions towards application data distribution
- Hardware Selection: Each hardware type has unique advantages for different data distributions
- Model Sharding: Optimize towards application distribution
- Cross-Host Distributed Inference: Optimize for application distribution
- Kernel Optimization: Select most optimized kernel for application distribution
- Quality Tuning: Various tuning mechanisms for application-specific quality
- Combinatorial Explosion: Over 100,000 combinations to choose from
Fireworks Platform Architecture
- Virtual Cloud Infrastructure: Manages complexity of GPU sourcing with guaranteed high quality/reliability
- Multi-Hardware Support: Different vendors, SKUs with varying capabilities
- Some hardware: higher FLOPS
- Some hardware: higher memory bandwidth
- Optimized for different application performance requirements
- Open Model Libraries: Easy selection and steering towards application needs
- Multiple Tuning Mechanisms: Speed tuning, quality tuning, various approaches
- Production Data Integration: Reinforcement tuning using production data for application-specific needs
- Developer Tools: Comprehensive suite for accessibility
Platform Launch and Success Metrics
- Launch Timeline: Developer-facing self-serve platform launched 1.5 years ago
- Food Chain Company Case Study: Scaled from 1 shop AI feature to 1,000 shops in 3 months
- Software Development Company Case Study: Scaled from 100,000 developers to 25 million developers in 3 months
- Optimization Goal: Achieve optimal sweet spot across quality, cost, and speed during scaling
Platform Accessibility
- Easy onboarding and experimentation process
- Self-serve platform for developer access
