AI summary by TwinMind
AI isn't gonna keep improving
Theo - t3․gg
22m 10s
Moore's Law Background and CPU Performance Plateau
- Moore's Law: transistor count doubles every 2 years, cost halves
- Historical reality: computers 2x faster every 2-4 years through 2000s
- Current reality: Apple M1 (2020) vs 2024 machines show minimal performance difference
- Physics limitations: silicon manufacturing hitting atomic scale barriers
- Manufacturing complexity: only TSMC can produce most efficient chips
- Intel, Apple, Nvidia all depend on TSMC manufacturing
- CPU performance gains dropped from 1.5x/year to 1.1x/year
- Personal example: 2020 10700k processor still competitive with latest hardware
GPU Architecture Advantages and Limitations
- GPU cores significantly simpler than CPU cores
- Performance scaling through adding more cores/chips
- Nvidia claims continued GPU compute growth
- Reality: scaling requires massive physical infrastructure (room-sized GPUs)
- Alternative architectures emerging: IBM analog AI chips (similar to Bitcoin ASIC evolution)
- Specialized chips potentially faster than general-purpose CPUs/GPUs
LLM Performance Plateau Evidence
- Mistral Large 2 improvements: code generation, mathematics, reasoning, multilingual, function calling
- Yann LeCun (Meta AI head) advice: students shouldn't focus on LLMs for next-gen AI
- Performance gap analysis:
- ChatGPT 2→3→3.5→4: massive improvements
- GPT-4→4 Turbo→4o: significantly smaller gains
- Claude 1→2→3: major wins, but slowing
- Claude 3.5 Sonnet: solid improvement but diminishing returns
- Each iteration requires more money, time, compute, energy for smaller gains
- Models becoming commoditized like Windows laptops
The Bitter Lesson Analysis
- Rich Sutton's 2019 blog post: general AI methods with massive compute most effective
- Historical pattern: human knowledge baking into AI eventually outperformed by brute force compute
- Examples: custom chess/Go engines vs DeepBlue/AlphaZero
- Counter-evidence to pure compute scaling:
- Moore's Law fading
- Transformer architectures carefully handcrafted by humans
- Algorithm optimization often exceeds Moore's Law gains
Fast Inverse Square Root Case Study
- Doom lighting/rendering breakthrough through math hack, not hardware
- 32-bit floating point treated as integer, subtracted from magic constant
- Enabled 3D gaming revolution through clever algorithm, not faster processors
- Demonstrates software innovation trumping hardware limitations
- Parallel to current AI needs: smarter methods vs more compute
AI Hype Cycle and Environmental Concerns
- Gartner hype cycle: excitement → inflated expectations → trough of disillusionment → slope of reality
- AI popularity now exceeds Moore's Law capabilities
- Environmental footprint concerns from massive compute requirements
- Research increasingly restricted to large corporations with compute resources
- Current focus areas: synthetic data, edge AI, model optimization, knowledge graphs
ARC AGI Benchmark Challenge
- $1M competition for beating ARC AGI benchmark
- Measures general intelligence (skill acquisition) vs current benchmarks (skill demonstration)
- Pattern recognition puzzles easy for humans, difficult for AI
- LLMs trained on vast data but can't adapt to simple unseen problems
- Progress toward AGI stalled on general intelligence measures
- Advisors include Nat Friedman (ex-GitHub CEO), Mike Knoop
Future AI Architecture Predictions
- Hybrid approaches combining handwritten code + AI
- CPU/GPU task division model applied to code/AI collaboration
- Video generation showing promise for specialized applications
- Multiple specialized models vs single general-purpose LLM
- Efficiency, speed, novel applications as competitive advantages
- Need for entirely different AI architectures beyond LLMs
