AI summary by TwinMind
Why Brain-like Computers Are Hard
Asianometry
17m 43s
Von Neumann Bottleneck and Brain Motivation
- Von Neumann architecture stores programs/data in same memory bank
- Both travel same road to/from CPU, creating bottleneck
- System limited by memory bandwidth, not CPU/GPU computational power
- Brain operates at 12-20 watts (20% body's metabolic rate)
- Desktop computers: ~175 watts
- Nvidia H100 AI accelerators: 300-700 watts
- Brain clock frequency: ~10 Hz (varies by activity/mental state)
- Bee brain example: <1 million neurons, <1 watt power
- Can fly, navigate, socialize, calculate
- Comparable to 18 billion transistor system-on-chip
Brain Architecture and Parallelism
- Human brain: 86 billion neurons operating without central clock
- Neurons fire spikes based on neighbor neuron inputs
- Brain = exaflop-level compute device (comparable to advanced supercomputers)
- Nvidia H100: ~60 teraflops (depending on variant)
- CPU synchronization costs time/energy for clock distribution
- Brain uses chaos as feature, not bug
- Majority of neural spikes result in "noise" - no action
- Extensive connections ensure enough spikes reach correct neurons
- Modern 3.2 GHz Intel CPU: noise signal once every 24 hours
- Brain achieves efficiency through fault tolerance and redundancy
Neuron Structure and Diversity
- Basic neuron components: dendrites, soma, axon
- Typical cerebrum neuron: 10,000 inputs
- Typical cerebellum neuron: up to 250,000 inputs
- Signal transmission through synapses (electrochemical connections)
- Synaptic plasticity enables long-term learning/memory changes
- 86 billion neurons exhibit incredible diversity
- Signal speeds: faster than Taiwan high-speed rail to slower than sea turtle
- Motor neuron length: up to 1 meter (multiple meters in giraffes)
- Some neurons: only 0.1mm long
- 1981 Nobel Prize (Hubel & Wiesel): neurons have preferred stimuli/orientations
Brain vs Computer Fundamental Differences
- Brain propagates random spike waves through diverse neurons
- Many spikes become noise, but relevant ones carry computation
- Neurons/synapses merge computation and memory functions
- No Von Neumann bottleneck - memory and compute not separated
- Artificial neural networks (ANNs) run on Von Neumann hardware
- Perceptrons: simplified mathematical neuron models
- Spiking neural networks differentiate from traditional ANNs/deep networks
Silicon Neuromorphic Hardware Examples
IBM TrueNorth (2014)
- First widely distributed neuromorphic chip
- 28nm CMOS process, 5.4 billion transistors
- 4,096 neurosynaptic cores
- Each core: 256 neurons, 256 axons, 256x256 crossbar SRAM synapses
- 256 million programmable synapses total
- Power consumption: ~70mW
- Runs convolutional neural networks
- No global clock, asynchronous operation at 10 Hz
- Random number generators for noise/randomness simulation
Intel Loihi and Loihi 2
- Intel Labs neuromorphic AI hardware
- Low power neuromorphic processing
European Human Brain Project (2013-2023)
- BrainScales project: first release 2011, second version 2020
- Mixed-signal ASIC using analog circuits
- Mimics spiking neurons but less flexible than digital counterparts
Tianjic Project (2019)
- Hybrid chip from Tsinghua University, China
- Runs various neuron network types including neuromorphic hardware
Silicon Neuromorphic Limitations
- Trade-offs between energy efficiency and accuracy
- Large circuit size due to distributed memory in each core
- Von Neumann systems benefit from single dense central memory
- Digital devices ill-suited for analog brain behavior
- Flexibility limitations when incorporating analog elements
- Scalability challenges for larger models
Memristor Technology
Historical Development
- 1971: Leon Chua (UC Berkeley) proposed fourth circuit element
- 2008: HP Labs created physical memristor implementation
- Metal-dielectric-metal sandwich structure
- Platinum electrodes, titanium oxide dielectric
- Two terminals/connection points
Memristor Characteristics
- Electrical resistance depends on voltage history
- Non-volatile memory (retains state when power off)
- Can switch between insulating/conducting states
- Behavior similar to biological synapses
- Suitable for in-memory computing applications
Memristor Challenges
- Manufacturing uniformity and scalability issues
- Endurance: handling many resistance switch cycles
- Data retention longevity questions
- Nonlinear resistance changes complicate programming
- May require new software paradigms and neural network programming approaches
Competition and Future Outlook
- Nvidia/GPU advancement following Huang's Law
- AI chips double performance every 2 years
- GPU AI performance improved 1,000x in past 10 years
- Hybrid systems potential: combining neuromorphic and Von Neumann chiplets
- "Best of both worlds" approach for different system advantages
