AI summary by TwinMind
Microsoft CTO Kevin Scott on How Far Scaling Laws Will Extend | Training Data
Sequoia Capital
1h 0m 28s
Kevin Scott's Background and Career Journey
- Kevin Scott, 52, CTO of Microsoft for 7.5 years
- Grew up in rural Virginia, first in family to attend university
- Majored in computer science, minored in English Literature
- Nearly pursued PhD in English Literature but chose pragmatic path due to financial constraints
- Started PhD in computer science focusing on compiler optimization and programming languages
- Left academia after realizing limited real-world impact of academic research
- Joined Google in 2003 after cold-emailing resume
- Interviewed by many compiler experts at Google despite not working on compilers there
- Career progression: Google → AdMob startup → back to Google → LinkedIn (helped go public) → Microsoft (via LinkedIn acquisition)
- Attributes success to "right place at right time" during major technology waves (PC, internet, mobile)
AI Team Building and Talent Strategy
- For complex AI platform work (distributed systems, training/inference, algorithms): PhD very helpful
- Need extensive prior knowledge and rigorous training
- PhD demonstrates ability to complete complex projects
- Challenge: transitioning from individual PhD work to collaborative team environments
- For AI applications (education, healthcare, developer tools): PhD helpful but not necessary
- Most AI work involves building complete platform portfolios beyond just core technology
Microsoft's AI Strategy and Platform Approach
- Microsoft positions itself as AI platform company
- Strategy: build comprehensive technology platform enabling others to build useful applications
- Focus areas:
- Frontier models to small language models
- Highly optimized inference infrastructure
- Hyperscale training and inference with economies of scale
- Making platform more accessible (cheaper, more powerful)
- Developer tools, safety infrastructure, testing
- Listen intently to developers and AI builders to fill gaps
- Successful examples: GitHub Copilot showing strong market traction
- Partnership approach with OpenAI central to strategy
Self-Assessment: Strengths and Weaknesses
- Strengths:
- Made powerful AI dramatically more accessible with OpenAI partnership
- Seeing customers building AI applications who otherwise wouldn't
- Clear point of view on AI platform architecture
- Good partnering approach
- Weaknesses:
- Late to basic AI investments
- Had fragmented AI spending across multiple projects instead of focused scale approach
- Didn't prioritize scale early enough (scale identified as key success factor)
The Scale Revelation and OpenAI Partnership
- Kevin became CTO in 2017, identified AI progress as biggest gap
- Mid-2017: developed "religion" that AI would be major focus
- 2018: BERT paper from Google crystallized belief in need for acceleration
- Restructured Microsoft AI efforts for focus
- 2019: First OpenAI deal
- Key insight: AI breakthroughs consistently come from scale (data + compute)
- At certain scale points, becomes economically impossible to bet on multiple exponentially scaling approaches
- OpenAI partnership based on shared platform belief and track record of execution through scale points
- GPT-3 and GPT-4 validated forecasting approach
Computing Infrastructure and Hardware Evolution
- Training costs significant but inference expected to dwarf training costs
- Hardware improvements exceed Moore's Law for AI:
- A100: 3-3.5x better price/performance than V100
- H100: close to similar improvement
- Next generation looks very promising
- Key hardware advances:
- Process technology improvements
- Architectural innovations
- Narrower word sizes (less than 64-bit arithmetic)
- Massive parallelism extraction
- Advanced networking for multi-GPU/multi-node systems
- Power scaling challenges since 2012 (Dennard scaling ended)
- Density issues with power dissipation
- Different architectures needed for training vs inference
- Inference environments easier to experiment with and upgrade
Supply/Demand Balance and Market Dynamics
- Currently more demand than supply for AI compute
- Powerful incentives driving software stack optimization
- Demand likely to increase if platform capabilities continue expanding
- Shape of demand evolving
- Questions about number of frontier models actually needed
- Analogy: building frontier model to deliver application like building smartphone hardware/OS for mobile app
- Expects lots of inference growth indicating successful product-market fit
- Caution about speculative infrastructure investment patterns
Training Data Quality and Economics
- Quality of training data at least as important as volume
- Economic framework emerging for data partnerships
- Large language models not databases - poor for fact retrieval
- Two types of data needs:
- Training data (for model development)
- Application data (for reasoning/retrieval)
- Business models developing around both types
- Training data attribution challenging due to diffuse contribution (like reading Moby Dick)
- Retrieval data will have clearer business models (subscription, rev share, licensing, new advertising formats)
- Opportunity for entrepreneurs to develop new ad units for AI agents
Reasoning Capabilities and Benchmarks
- Benchmarks (GPQA, MMLU) being rapidly saturated by new model generations
- Expensive to run granular experiments on training data contribution
- "Textbooks are all you need" paper example of evaluating token contribution quality
- Strong incentives to optimize synthetic data generation
- Value function construction remains challenging for broader domains beyond games
Current AI Capabilities and Deployment Philosophy
- Microsoft chose "co-pilot" naming deliberately to emphasize augmentation over replacement
- Focus on assistive rather than substitutive technology
- Easier to deploy when narrowed to specific domains
- GitHub Copilot major success story
- General co-pilots harder to achieve high precision autonomous action
- Partnership with Devon as example of optimized human-AI collaboration
- Emphasis on being "very good before introduction" to avoid user rejection
The 80/20 Problem and Future Scaling
- Common pattern: can automate 80% of task quickly, last 20% very difficult
- Microsoft not at diminishing marginal returns on scale-up
- Next frontier model sample coming (timing undisclosed)
- Each generation improves brittleness, cost, and enables more complex capabilities
- Architectural advice: build flexibility to "snap to" new frontier models
- Avoid getting trapped by over-investing in current limitations
Long-term Optimism and Societal Impact
- Technology historically turns zero-sum problems into non-zero-sum through abundance creation
- Healthcare example: Kevin's mother's thyroid condition misdiagnosed multiple times
- GPT-4 could have immediately suggested correct TSH test
- Rural healthcare system overburdened, lacking resources
- Current AI could alleviate suffering if properly deployed
- Opportunities in:
- Education advancement
- Healthcare system improvements
- Scientific research tools
- Carbon capture catalyst discovery
- Safer transportation
- Post-carbon economy transition
- Concern about getting distracted by "noise" rather than focusing on deployment benefits
- Emphasis on considering cost of NOT deploying beneficial technology
Person Most Admired in AI
- Ray Solomonoff from 1950s Dartmouth Workshop
- Advocated probabilistic methods for AI from beginning
- Contrarian view against prevailing rule-based/symbolic reasoning approaches
- Proved correct despite being judged wrong for decades
- Admired for persistence with beliefs throughout career
