AI summary by TwinMind

Microsoft CTO Kevin Scott on How Far Scaling Laws Will Extend | Training Data

Sequoia Capital

1h 0m 28s

Kevin Scott's Background and Career Journey

  • Kevin Scott, 52, CTO of Microsoft for 7.5 years
  • Grew up in rural Virginia, first in family to attend university
  • Majored in computer science, minored in English Literature
  • Nearly pursued PhD in English Literature but chose pragmatic path due to financial constraints
  • Started PhD in computer science focusing on compiler optimization and programming languages
  • Left academia after realizing limited real-world impact of academic research
  • Joined Google in 2003 after cold-emailing resume
    • Interviewed by many compiler experts at Google despite not working on compilers there
  • Career progression: Google → AdMob startup → back to Google → LinkedIn (helped go public) → Microsoft (via LinkedIn acquisition)
  • Attributes success to "right place at right time" during major technology waves (PC, internet, mobile)

AI Team Building and Talent Strategy

  • For complex AI platform work (distributed systems, training/inference, algorithms): PhD very helpful
    • Need extensive prior knowledge and rigorous training
    • PhD demonstrates ability to complete complex projects
    • Challenge: transitioning from individual PhD work to collaborative team environments
  • For AI applications (education, healthcare, developer tools): PhD helpful but not necessary
  • Most AI work involves building complete platform portfolios beyond just core technology

Microsoft's AI Strategy and Platform Approach

  • Microsoft positions itself as AI platform company
  • Strategy: build comprehensive technology platform enabling others to build useful applications
  • Focus areas:
    • Frontier models to small language models
    • Highly optimized inference infrastructure
    • Hyperscale training and inference with economies of scale
    • Making platform more accessible (cheaper, more powerful)
    • Developer tools, safety infrastructure, testing
  • Listen intently to developers and AI builders to fill gaps
  • Successful examples: GitHub Copilot showing strong market traction
  • Partnership approach with OpenAI central to strategy

Self-Assessment: Strengths and Weaknesses

  • Strengths:
    • Made powerful AI dramatically more accessible with OpenAI partnership
    • Seeing customers building AI applications who otherwise wouldn't
    • Clear point of view on AI platform architecture
    • Good partnering approach
  • Weaknesses:
    • Late to basic AI investments
    • Had fragmented AI spending across multiple projects instead of focused scale approach
    • Didn't prioritize scale early enough (scale identified as key success factor)

The Scale Revelation and OpenAI Partnership

  • Kevin became CTO in 2017, identified AI progress as biggest gap
  • Mid-2017: developed "religion" that AI would be major focus
  • 2018: BERT paper from Google crystallized belief in need for acceleration
  • Restructured Microsoft AI efforts for focus
  • 2019: First OpenAI deal
  • Key insight: AI breakthroughs consistently come from scale (data + compute)
  • At certain scale points, becomes economically impossible to bet on multiple exponentially scaling approaches
  • OpenAI partnership based on shared platform belief and track record of execution through scale points
  • GPT-3 and GPT-4 validated forecasting approach

Computing Infrastructure and Hardware Evolution

  • Training costs significant but inference expected to dwarf training costs
  • Hardware improvements exceed Moore's Law for AI:
    • A100: 3-3.5x better price/performance than V100
    • H100: close to similar improvement
    • Next generation looks very promising
  • Key hardware advances:
    • Process technology improvements
    • Architectural innovations
    • Narrower word sizes (less than 64-bit arithmetic)
    • Massive parallelism extraction
    • Advanced networking for multi-GPU/multi-node systems
  • Power scaling challenges since 2012 (Dennard scaling ended)
  • Density issues with power dissipation
  • Different architectures needed for training vs inference
  • Inference environments easier to experiment with and upgrade

Supply/Demand Balance and Market Dynamics

  • Currently more demand than supply for AI compute
  • Powerful incentives driving software stack optimization
  • Demand likely to increase if platform capabilities continue expanding
  • Shape of demand evolving
  • Questions about number of frontier models actually needed
  • Analogy: building frontier model to deliver application like building smartphone hardware/OS for mobile app
  • Expects lots of inference growth indicating successful product-market fit
  • Caution about speculative infrastructure investment patterns

Training Data Quality and Economics

  • Quality of training data at least as important as volume
  • Economic framework emerging for data partnerships
  • Large language models not databases - poor for fact retrieval
  • Two types of data needs:
    • Training data (for model development)
    • Application data (for reasoning/retrieval)
  • Business models developing around both types
  • Training data attribution challenging due to diffuse contribution (like reading Moby Dick)
  • Retrieval data will have clearer business models (subscription, rev share, licensing, new advertising formats)
  • Opportunity for entrepreneurs to develop new ad units for AI agents

Reasoning Capabilities and Benchmarks

  • Benchmarks (GPQA, MMLU) being rapidly saturated by new model generations
  • Expensive to run granular experiments on training data contribution
  • "Textbooks are all you need" paper example of evaluating token contribution quality
  • Strong incentives to optimize synthetic data generation
  • Value function construction remains challenging for broader domains beyond games

Current AI Capabilities and Deployment Philosophy

  • Microsoft chose "co-pilot" naming deliberately to emphasize augmentation over replacement
  • Focus on assistive rather than substitutive technology
  • Easier to deploy when narrowed to specific domains
  • GitHub Copilot major success story
  • General co-pilots harder to achieve high precision autonomous action
  • Partnership with Devon as example of optimized human-AI collaboration
  • Emphasis on being "very good before introduction" to avoid user rejection

The 80/20 Problem and Future Scaling

  • Common pattern: can automate 80% of task quickly, last 20% very difficult
  • Microsoft not at diminishing marginal returns on scale-up
  • Next frontier model sample coming (timing undisclosed)
  • Each generation improves brittleness, cost, and enables more complex capabilities
  • Architectural advice: build flexibility to "snap to" new frontier models
  • Avoid getting trapped by over-investing in current limitations

Long-term Optimism and Societal Impact

  • Technology historically turns zero-sum problems into non-zero-sum through abundance creation
  • Healthcare example: Kevin's mother's thyroid condition misdiagnosed multiple times
    • GPT-4 could have immediately suggested correct TSH test
    • Rural healthcare system overburdened, lacking resources
    • Current AI could alleviate suffering if properly deployed
  • Opportunities in:
    • Education advancement
    • Healthcare system improvements
    • Scientific research tools
    • Carbon capture catalyst discovery
    • Safer transportation
    • Post-carbon economy transition
  • Concern about getting distracted by "noise" rather than focusing on deployment benefits
  • Emphasis on considering cost of NOT deploying beneficial technology

Person Most Admired in AI

  • Ray Solomonoff from 1950s Dartmouth Workshop
  • Advocated probabilistic methods for AI from beginning
  • Contrarian view against prevailing rule-based/symbolic reasoning approaches
  • Proved correct despite being judged wrong for decades
  • Admired for persistence with beliefs throughout career

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome