AI summary by TwinMind

Running Neural Networks on Meshes of Light

Asianometry

13m 42s

Energy Efficiency Problem in AI Computing

  • 2016 AlphaGo defeated Lee Sedol using 48 TPUs consuming 40 watts each
  • Human brain with 100 billion neurons runs on ~20 watts
  • Neural networks growing bigger, energy consumption increasing
  • Silicon photonics emerging as promising solution for "meshes of light" neural networks

Current AI Accelerator Technology

  • GPU enabled training large neural networks in reasonable time
  • Google founded multi-billion dollar AI accelerator industry with TPU
  • AI accelerators conceptually simple: optimized for matrix multiplication
  • Neural networks represented as mathematical matrices
  • Over 90% of inference involves matrix multiplication operations
  • TPU contains 2D array of tens of thousands of MAC (multiply-accumulator) units
  • Each MAC unit multiplies two numbers at high precision, adds to accumulation sum
  • TPU outperforms GPU like specialized arm wrestler vs general athlete

Energy Consumption Analysis

  • 80% of TPU's 40-watt power budget spent on connections and data transfer
  • Electric connections use electrons, which interact with particles generating heat losses
  • AI accelerators highly parallel - MAC units run side by side using more energy
  • Design improvements: bringing model weights physically closer to MAC circuits
  • AI accelerators now 20x more power efficient per MAC than GPUs
  • AlphaGo 2016: 48 TPUs across network
  • Alpha Zero 2017: 4 TPUs in single machine with better performance

Silicon Photonics Fundamentals

  • Replaces electrical connections with light-based ones
  • Uses same principles as fiber optic cables
  • Light signals transmitted around chip with minimal losses
  • Two key benefits over traditional silicon:
    • Very high bandwidth: 100 terahertz and above
    • Very low power consumption potential: only energy for sending/receiving light
  • Theoretical 80% power savings from eliminated data connection losses
  • Practical achievable gain: potential 10x improvement over status quo

Hybrid Implementation Approach

  • Simplest implementation fuses silicon photonics with traditional semiconductors
  • Traditional semiconductors store inputs, weights, intermediate results between layers
  • Data crosses to photonic circuit through DACs (digital-to-analog converters)
  • Light processing occurs in photonic circuit
  • Output returns through ADCs (analog-to-digital converters)

Lightmatter Case Study

  • MIT spinoff raised over $100 million from VCs and enterprises
  • Nanophotonic processor replaces TPU's 2D MAC array
  • Uses mesh of Mach-Zehnder Interferometers (MZI) as building blocks
  • MZI splits and recombines light when voltage applied
  • Recombined light changes map to multiplication results
  • Weight and input matrices converted to light arrangements
  • Matrix multiplications executed by sending light through MZI mesh
  • Calculations complete in ~100 picoseconds due to light speed
  • Photons detected, collected, mapped to computation results
  • Small networks achieve 3 orders of magnitude better power efficiency

Current Limitations and Challenges

Accuracy Issues

  • 2017 Lightmatter paper: 76.7% accuracy recognizing vowel sounds
  • Simulations suggested 91% should be achievable
  • Large accuracy gap due to analog-to-digital conversion errors
  • Photons cause measurement errors during encoding/decoding
  • Potential improvements: better light contrast for detection
  • Analog signals inherently less accurate than digital signals
  • Silicon photonics likely limited to inference, not training
  • Training still requires NVIDIA GPUs for digital precision

Size and Scale Constraints

  • 2017 system: only 2 layers, 56 Mach-Zehnder interferometers
  • 1980s Bell Labs optical transistors failed due to bulky optics
  • MZIs still bulky: ~10,000 square micrometers each
  • NVIDIA A100 has hundreds of cores, trillions of floating point operations/second
  • Question: Can photonic meshes scale while retaining advantages?

Scaling Solutions and Progress

  • Lightmatter 2020 Mars chip: 64x64 MZI matrix on 90nm process
  • Global Foundries and Intel investing heavily in silicon photonics platforms
  • Commercialization possible through fabrication improvements
  • Theoretical 1D scaling approach:
    • Single row of interferometers performs like 2D mesh
    • Replaces one spatial dimension with time dimension
    • Pathway to larger networks without advanced fabrication

Market Potential and Competition

  • Economic rationale: energy usage direct operating cost for data centers
  • 10x energy reduction would attract Google, Amazon adoption
  • Multiple startups entering space beyond Lightmatter
  • Technology shows potential but needs performance delivery
  • Competition heating up indicates market viability

Research Developments

  • Cornell group training neural networks without high digital accuracy
  • Using traditional digital methods for unusual network architectures
  • Potential pathway to overcome accuracy obstacles

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome