AI summary by TwinMind

Moore's Law is Not Dead (Jim Keller) | AI Podcast Clips

Lex Fridman

26m 2s

Moore's Law Definition and Evolution

  • Gordon Moore's original statement: double transistor count every 2 years
  • Operational model: increase computer performance 2x every 2-3 years
  • Current cadence: shrink factor of 0.6 every 2 years (not 0.5)
    • Shrink factor refers to making transistors smaller by 0.6x
    • Results in 1/0.6 more transistors for constant chip area
  • Performance delivery methods have changed substantially over time

Moore's Law Death Predictions and Reality

  • Speaker aware of Moore's Law for 30 years, designing computers for 40 years
  • Consistent predictions of death in "10-15 years" throughout career
    • Pattern: death predicted in 10-15 years, then 5 years, then back to 10 years
  • Joined Intel when "everybody said Moore's law is dead"
  • Observation: humans like apocalyptic statements (running out of food, air, room)
  • 50+ years of people predicting Moore's Law death

Innovation Cascade Behind Moore's Law

  • Moore's Law appears as single trend but contains thousands of innovations
  • Each innovation has own diminishing return curve
  • Result: cascade of diminishing return curves creating exponential curve overall
  • Experts in single innovation see plateau, predict end
  • Meanwhile, other groups work on different innovations

Physical Limits and Future Potential

  • Modern transistor: ~1000x1000x1000 atoms
  • Quantum effects occur at 2-10 atoms
  • Theoretical minimum: transistor as small as 10x10x10 atoms
  • Potential: million times smaller than current
  • Current transistor width: ~120 atoms, can be made thinner
  • Gate wrapped around inner space with complex geometry
  • Techniques exist to place atoms in single atomic layer
  • Manufacturing challenge: placing atoms individually takes time
    • Example: 10 minutes per atom × 10²³ atoms = impractical timeline

Innovation Stack Breadth

  • Equipment, optics, chemistry, physics, material science, metallurgy
  • Material interaction stability over temperature
  • Repeatability requirements
  • Thousands of technologies involved in shrinking alone
  • Roadmap request: path to 100x shrink
    • Team achieved 50x shrink in 2 weeks
    • Continued work for additional improvement

Design Strategy for Transistor Abundance

  • Two design stances for computer architects:
    • Expect shrinking: design for more transistors
    • Prepare for complexity management of increased transistor count
  • Analogy: building with bricks that halve in size every 2 years
    • Same building methods would exponentially increase construction time
    • Solution: design equipment and methods for smaller bricks
    • Take advantage of benefits (strength, efficiency, less material)
  • Roadmap enables collateral design to leverage and cope with changes

Human and Team Constraints

  • People don't get significantly smarter over time
  • Teams can't grow much beyond certain limits
    • Effective teams: ~10 people
    • Teams of 100: can know each other
    • Beyond 100: require organizational boundaries
  • Hard constraints require divide-and-conquer approaches

Abstraction Layers and Design Methodology

  • Power of abstraction layers essential for scaling
  • Evolution: transistors → logic cells → functional units → computers
  • Team specialization by abstraction level
  • Critical decisions about when to shift gears between levels
  • Use faster computers to build faster computers
    • Some algorithms run 2x faster on new computers
    • Many algorithms are N² complexity
    • Computer with 2x transistors may take 4x longer to run
    • Requires software refactoring

Mathematical Hierarchy in Computing

  • Evolution from simple equations (A = B + C × D)
  • Vector units enabled matrix equations
  • Modern AI: convolutional neural networks
  • Hierarchy: simple equation → linear equations → matrix equations → deeper computation
  • Large datasets creating topology problems
  • Data organized in immense shapes requiring specialized computation

AI Computational Evolution

  • Historical progression:
    • Rule sets applied to simple computational situations
    • Chess: deep search through move/result databases
    • Modern: large image datasets training weight sets for convolution
  • Completely different phenomena at each stage
  • Mathematical graph progression with increasing computational requirements

Search vs. Projection Debate

  • Neural network inner layers perform projections, not search
  • Example: shadow projections creating attribute recognition
  • Layers describe specific features (pointy ears, round eyes, fuzziness)
  • Training involves projection extraction, not search
  • Deep networks have layers with unknown representations that are essential
  • Distinction between optimization (search) and inference (projection)
  • Difference between given search space vs. found search space

Fundamental Operations Consistency

  • Basic operations remain: add, subtract, load, compare, branch
  • Building blocks: atoms → transistors → logic gates → computers → functional units
  • Mathematics building blocks: adds, subtracts, multiplies
  • Mathematical space essentially infinite despite consistent computational building blocks
  • Algorithm requirements vary:
    • Sparse data, 32-bit data
    • 8-bit convolution operations
    • TensorFlow data types imply optimization sets

Quantum and Analog Computing Possibilities

  • Quantum researchers pursuing radical changes to basic operations
  • Analog computing research inspired by brain's analog-like processing
  • Current: million-fold increase in computation capability
  • Relationship between computational intensity and mathematical abstraction ability unclear

Computational Magnitude and Quality Changes

  • Raja Koduri observation: every 10x computation change fundamentally changes what computation does
  • "Difference in quantity is difference in kind"
  • Examples: ant vs. anthill, neuron vs. brain
  • Quantity changes create quality changes
  • Pattern observed multiple times in mathematics
  • Expected to continue happening

Hardware-Software Dialogue

  • Awareness of software stacks running on computational loads
  • Considerations for petabyte memory with sparse access patterns
  • AI programmer requirements inform hardware design
  • Interaction between hardware capabilities and software needs
  • Fundamental operations remain adders, subtractors, multipliers at chip level

Software Optimization Philosophy

  • Problem of early optimization in software development
  • Optimizing first components rarely addresses performance limiters
  • Once complete system works, 2x performance gains possible through right optimizations
  • Creative tension: optimize existing vs. write new software stack
  • Hardware improvement continues during software development (Moore's Law benefit)
  • AI research should expect hardware improvements and make different choices accordingly

Semiconductor Innovation Examples

  • Nanowires announced by fabricators
  • Evolution: FET with gate → little wire with better control
  • Obvious shrinking steps from nanowire technology
  • Metallurgy around wire stacks has clear shrinking abilities
  • Factor of 100 improvement expected from shrinking innovations
  • Timeline: 10-15 years for significant advancement

Bell's Law and Computational Evolution

  • Historical progression: mainframes → minis → workstations → PCs → mobile
  • Moore's Law drove faster, smaller computers
  • Every 10x generates new computation type:
    • Scalar → vector → matrix → topological computation
  • Industry trends: mainframes → mini-computers → PCs → internet → mobile → 5G
  • 5G wireless: 1 millisecond latency enabling smart world
  • Smart world: everything knows and recognizes users
  • Transformations unpredictable in nature

Personal Perspective on Technology Impact

  • Feels lucky to work in technology and get paid for it
  • Millions of people working on technology globally
  • Actions and outcomes highly unpredictable
  • Individual contribution part of larger vector of parallel developments
  • If not present, others would continue the work
  • Acknowledges tools are changing the world
  • Uncertain about ultimate purpose or direction

Philosophical Observations

  • Physics and philosophy have highest GRE scores
  • Both disciplines attempt to answer "why is there anything?"
  • Philosophers: theological approach
  • Physicists: material approach
  • 100 billion galaxies with 100 billion stars each
  • Approaching 10 billion people on Earth
  • Difficulty determining ultimate purpose

Mathematics and Computation Relationship

  • Physical world inherently generates mathematics
  • Example: sphere surface area grows by r²
  • Mathematicians and physicists collaborate extensively
  • Computation historically pedestrian (binary algebra)
  • Recent advancement: sophisticated mathematical computations beyond human understanding
  • Machine learning finds patterns in large datasets without intuitive functions
  • Results may not be completely mathematically describable
  • Computation achieving interesting results compared to simple "A = B + C"

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome