AI summary by TwinMind
Running Neural Networks on Meshes of Light
Asianometry
13m 42s
Energy Efficiency Problem in AI Computing
- 2016 AlphaGo defeated Lee Sedol using 48 TPUs consuming 40 watts each
- Human brain with 100 billion neurons runs on ~20 watts
- Neural networks growing bigger, energy consumption increasing
- Silicon photonics emerging as promising solution for "meshes of light" neural networks
Current AI Accelerator Technology
- GPU enabled training large neural networks in reasonable time
- Google founded multi-billion dollar AI accelerator industry with TPU
- AI accelerators conceptually simple: optimized for matrix multiplication
- Neural networks represented as mathematical matrices
- Over 90% of inference involves matrix multiplication operations
- TPU contains 2D array of tens of thousands of MAC (multiply-accumulator) units
- Each MAC unit multiplies two numbers at high precision, adds to accumulation sum
- TPU outperforms GPU like specialized arm wrestler vs general athlete
Energy Consumption Analysis
- 80% of TPU's 40-watt power budget spent on connections and data transfer
- Electric connections use electrons, which interact with particles generating heat losses
- AI accelerators highly parallel - MAC units run side by side using more energy
- Design improvements: bringing model weights physically closer to MAC circuits
- AI accelerators now 20x more power efficient per MAC than GPUs
- AlphaGo 2016: 48 TPUs across network
- Alpha Zero 2017: 4 TPUs in single machine with better performance
Silicon Photonics Fundamentals
- Replaces electrical connections with light-based ones
- Uses same principles as fiber optic cables
- Light signals transmitted around chip with minimal losses
- Two key benefits over traditional silicon:
- Very high bandwidth: 100 terahertz and above
- Very low power consumption potential: only energy for sending/receiving light
- Theoretical 80% power savings from eliminated data connection losses
- Practical achievable gain: potential 10x improvement over status quo
Hybrid Implementation Approach
- Simplest implementation fuses silicon photonics with traditional semiconductors
- Traditional semiconductors store inputs, weights, intermediate results between layers
- Data crosses to photonic circuit through DACs (digital-to-analog converters)
- Light processing occurs in photonic circuit
- Output returns through ADCs (analog-to-digital converters)
Lightmatter Case Study
- MIT spinoff raised over $100 million from VCs and enterprises
- Nanophotonic processor replaces TPU's 2D MAC array
- Uses mesh of Mach-Zehnder Interferometers (MZI) as building blocks
- MZI splits and recombines light when voltage applied
- Recombined light changes map to multiplication results
- Weight and input matrices converted to light arrangements
- Matrix multiplications executed by sending light through MZI mesh
- Calculations complete in ~100 picoseconds due to light speed
- Photons detected, collected, mapped to computation results
- Small networks achieve 3 orders of magnitude better power efficiency
Current Limitations and Challenges
Accuracy Issues
- 2017 Lightmatter paper: 76.7% accuracy recognizing vowel sounds
- Simulations suggested 91% should be achievable
- Large accuracy gap due to analog-to-digital conversion errors
- Photons cause measurement errors during encoding/decoding
- Potential improvements: better light contrast for detection
- Analog signals inherently less accurate than digital signals
- Silicon photonics likely limited to inference, not training
- Training still requires NVIDIA GPUs for digital precision
Size and Scale Constraints
- 2017 system: only 2 layers, 56 Mach-Zehnder interferometers
- 1980s Bell Labs optical transistors failed due to bulky optics
- MZIs still bulky: ~10,000 square micrometers each
- NVIDIA A100 has hundreds of cores, trillions of floating point operations/second
- Question: Can photonic meshes scale while retaining advantages?
Scaling Solutions and Progress
- Lightmatter 2020 Mars chip: 64x64 MZI matrix on 90nm process
- Global Foundries and Intel investing heavily in silicon photonics platforms
- Commercialization possible through fabrication improvements
- Theoretical 1D scaling approach:
- Single row of interferometers performs like 2D mesh
- Replaces one spatial dimension with time dimension
- Pathway to larger networks without advanced fabrication
Market Potential and Competition
- Economic rationale: energy usage direct operating cost for data centers
- 10x energy reduction would attract Google, Amazon adoption
- Multiple startups entering space beyond Lightmatter
- Technology shows potential but needs performance delivery
- Competition heating up indicates market viability
Research Developments
- Cornell group training neural networks without high digital accuracy
- Using traditional digital methods for unusual network architectures
- Potential pathway to overcome accuracy obstacles
