AI summary by TwinMind
Moore's Law is Not Dead (Jim Keller) | AI Podcast Clips
Lex Fridman
26m 2s
Moore's Law Definition and Evolution
- Gordon Moore's original statement: double transistor count every 2 years
- Operational model: increase computer performance 2x every 2-3 years
- Current cadence: shrink factor of 0.6 every 2 years (not 0.5)
- Shrink factor refers to making transistors smaller by 0.6x
- Results in 1/0.6 more transistors for constant chip area
- Performance delivery methods have changed substantially over time
Moore's Law Death Predictions and Reality
- Speaker aware of Moore's Law for 30 years, designing computers for 40 years
- Consistent predictions of death in "10-15 years" throughout career
- Pattern: death predicted in 10-15 years, then 5 years, then back to 10 years
- Joined Intel when "everybody said Moore's law is dead"
- Observation: humans like apocalyptic statements (running out of food, air, room)
- 50+ years of people predicting Moore's Law death
Innovation Cascade Behind Moore's Law
- Moore's Law appears as single trend but contains thousands of innovations
- Each innovation has own diminishing return curve
- Result: cascade of diminishing return curves creating exponential curve overall
- Experts in single innovation see plateau, predict end
- Meanwhile, other groups work on different innovations
Physical Limits and Future Potential
- Modern transistor: ~1000x1000x1000 atoms
- Quantum effects occur at 2-10 atoms
- Theoretical minimum: transistor as small as 10x10x10 atoms
- Potential: million times smaller than current
- Current transistor width: ~120 atoms, can be made thinner
- Gate wrapped around inner space with complex geometry
- Techniques exist to place atoms in single atomic layer
- Manufacturing challenge: placing atoms individually takes time
- Example: 10 minutes per atom × 10²³ atoms = impractical timeline
Innovation Stack Breadth
- Equipment, optics, chemistry, physics, material science, metallurgy
- Material interaction stability over temperature
- Repeatability requirements
- Thousands of technologies involved in shrinking alone
- Roadmap request: path to 100x shrink
- Team achieved 50x shrink in 2 weeks
- Continued work for additional improvement
Design Strategy for Transistor Abundance
- Two design stances for computer architects:
- Expect shrinking: design for more transistors
- Prepare for complexity management of increased transistor count
- Analogy: building with bricks that halve in size every 2 years
- Same building methods would exponentially increase construction time
- Solution: design equipment and methods for smaller bricks
- Take advantage of benefits (strength, efficiency, less material)
- Roadmap enables collateral design to leverage and cope with changes
Human and Team Constraints
- People don't get significantly smarter over time
- Teams can't grow much beyond certain limits
- Effective teams: ~10 people
- Teams of 100: can know each other
- Beyond 100: require organizational boundaries
- Hard constraints require divide-and-conquer approaches
Abstraction Layers and Design Methodology
- Power of abstraction layers essential for scaling
- Evolution: transistors → logic cells → functional units → computers
- Team specialization by abstraction level
- Critical decisions about when to shift gears between levels
- Use faster computers to build faster computers
- Some algorithms run 2x faster on new computers
- Many algorithms are N² complexity
- Computer with 2x transistors may take 4x longer to run
- Requires software refactoring
Mathematical Hierarchy in Computing
- Evolution from simple equations (A = B + C × D)
- Vector units enabled matrix equations
- Modern AI: convolutional neural networks
- Hierarchy: simple equation → linear equations → matrix equations → deeper computation
- Large datasets creating topology problems
- Data organized in immense shapes requiring specialized computation
AI Computational Evolution
- Historical progression:
- Rule sets applied to simple computational situations
- Chess: deep search through move/result databases
- Modern: large image datasets training weight sets for convolution
- Completely different phenomena at each stage
- Mathematical graph progression with increasing computational requirements
Search vs. Projection Debate
- Neural network inner layers perform projections, not search
- Example: shadow projections creating attribute recognition
- Layers describe specific features (pointy ears, round eyes, fuzziness)
- Training involves projection extraction, not search
- Deep networks have layers with unknown representations that are essential
- Distinction between optimization (search) and inference (projection)
- Difference between given search space vs. found search space
Fundamental Operations Consistency
- Basic operations remain: add, subtract, load, compare, branch
- Building blocks: atoms → transistors → logic gates → computers → functional units
- Mathematics building blocks: adds, subtracts, multiplies
- Mathematical space essentially infinite despite consistent computational building blocks
- Algorithm requirements vary:
- Sparse data, 32-bit data
- 8-bit convolution operations
- TensorFlow data types imply optimization sets
Quantum and Analog Computing Possibilities
- Quantum researchers pursuing radical changes to basic operations
- Analog computing research inspired by brain's analog-like processing
- Current: million-fold increase in computation capability
- Relationship between computational intensity and mathematical abstraction ability unclear
Computational Magnitude and Quality Changes
- Raja Koduri observation: every 10x computation change fundamentally changes what computation does
- "Difference in quantity is difference in kind"
- Examples: ant vs. anthill, neuron vs. brain
- Quantity changes create quality changes
- Pattern observed multiple times in mathematics
- Expected to continue happening
Hardware-Software Dialogue
- Awareness of software stacks running on computational loads
- Considerations for petabyte memory with sparse access patterns
- AI programmer requirements inform hardware design
- Interaction between hardware capabilities and software needs
- Fundamental operations remain adders, subtractors, multipliers at chip level
Software Optimization Philosophy
- Problem of early optimization in software development
- Optimizing first components rarely addresses performance limiters
- Once complete system works, 2x performance gains possible through right optimizations
- Creative tension: optimize existing vs. write new software stack
- Hardware improvement continues during software development (Moore's Law benefit)
- AI research should expect hardware improvements and make different choices accordingly
Semiconductor Innovation Examples
- Nanowires announced by fabricators
- Evolution: FET with gate → little wire with better control
- Obvious shrinking steps from nanowire technology
- Metallurgy around wire stacks has clear shrinking abilities
- Factor of 100 improvement expected from shrinking innovations
- Timeline: 10-15 years for significant advancement
Bell's Law and Computational Evolution
- Historical progression: mainframes → minis → workstations → PCs → mobile
- Moore's Law drove faster, smaller computers
- Every 10x generates new computation type:
- Scalar → vector → matrix → topological computation
- Industry trends: mainframes → mini-computers → PCs → internet → mobile → 5G
- 5G wireless: 1 millisecond latency enabling smart world
- Smart world: everything knows and recognizes users
- Transformations unpredictable in nature
Personal Perspective on Technology Impact
- Feels lucky to work in technology and get paid for it
- Millions of people working on technology globally
- Actions and outcomes highly unpredictable
- Individual contribution part of larger vector of parallel developments
- If not present, others would continue the work
- Acknowledges tools are changing the world
- Uncertain about ultimate purpose or direction
Philosophical Observations
- Physics and philosophy have highest GRE scores
- Both disciplines attempt to answer "why is there anything?"
- Philosophers: theological approach
- Physicists: material approach
- 100 billion galaxies with 100 billion stars each
- Approaching 10 billion people on Earth
- Difficulty determining ultimate purpose
Mathematics and Computation Relationship
- Physical world inherently generates mathematics
- Example: sphere surface area grows by r²
- Mathematicians and physicists collaborate extensively
- Computation historically pedestrian (binary algebra)
- Recent advancement: sophisticated mathematical computations beyond human understanding
- Machine learning finds patterns in large datasets without intuitive functions
- Results may not be completely mathematically describable
- Computation achieving interesting results compared to simple "A = B + C"
