AI summary by TwinMind

Why Brain-like Computers Are Hard

Asianometry

17m 43s

Von Neumann Bottleneck and Brain Motivation

  • Von Neumann architecture stores programs/data in same memory bank
  • Both travel same road to/from CPU, creating bottleneck
  • System limited by memory bandwidth, not CPU/GPU computational power
  • Brain operates at 12-20 watts (20% body's metabolic rate)
  • Desktop computers: ~175 watts
  • Nvidia H100 AI accelerators: 300-700 watts
  • Brain clock frequency: ~10 Hz (varies by activity/mental state)
  • Bee brain example: <1 million neurons, <1 watt power
    • Can fly, navigate, socialize, calculate
    • Comparable to 18 billion transistor system-on-chip

Brain Architecture and Parallelism

  • Human brain: 86 billion neurons operating without central clock
  • Neurons fire spikes based on neighbor neuron inputs
  • Brain = exaflop-level compute device (comparable to advanced supercomputers)
  • Nvidia H100: ~60 teraflops (depending on variant)
  • CPU synchronization costs time/energy for clock distribution
  • Brain uses chaos as feature, not bug
  • Majority of neural spikes result in "noise" - no action
  • Extensive connections ensure enough spikes reach correct neurons
  • Modern 3.2 GHz Intel CPU: noise signal once every 24 hours
  • Brain achieves efficiency through fault tolerance and redundancy

Neuron Structure and Diversity

  • Basic neuron components: dendrites, soma, axon
  • Typical cerebrum neuron: 10,000 inputs
  • Typical cerebellum neuron: up to 250,000 inputs
  • Signal transmission through synapses (electrochemical connections)
  • Synaptic plasticity enables long-term learning/memory changes
  • 86 billion neurons exhibit incredible diversity
  • Signal speeds: faster than Taiwan high-speed rail to slower than sea turtle
  • Motor neuron length: up to 1 meter (multiple meters in giraffes)
  • Some neurons: only 0.1mm long
  • 1981 Nobel Prize (Hubel & Wiesel): neurons have preferred stimuli/orientations

Brain vs Computer Fundamental Differences

  • Brain propagates random spike waves through diverse neurons
  • Many spikes become noise, but relevant ones carry computation
  • Neurons/synapses merge computation and memory functions
  • No Von Neumann bottleneck - memory and compute not separated
  • Artificial neural networks (ANNs) run on Von Neumann hardware
  • Perceptrons: simplified mathematical neuron models
  • Spiking neural networks differentiate from traditional ANNs/deep networks

Silicon Neuromorphic Hardware Examples

IBM TrueNorth (2014)

  • First widely distributed neuromorphic chip
  • 28nm CMOS process, 5.4 billion transistors
  • 4,096 neurosynaptic cores
  • Each core: 256 neurons, 256 axons, 256x256 crossbar SRAM synapses
  • 256 million programmable synapses total
  • Power consumption: ~70mW
  • Runs convolutional neural networks
  • No global clock, asynchronous operation at 10 Hz
  • Random number generators for noise/randomness simulation

Intel Loihi and Loihi 2

  • Intel Labs neuromorphic AI hardware
  • Low power neuromorphic processing

European Human Brain Project (2013-2023)

  • BrainScales project: first release 2011, second version 2020
  • Mixed-signal ASIC using analog circuits
  • Mimics spiking neurons but less flexible than digital counterparts

Tianjic Project (2019)

  • Hybrid chip from Tsinghua University, China
  • Runs various neuron network types including neuromorphic hardware

Silicon Neuromorphic Limitations

  • Trade-offs between energy efficiency and accuracy
  • Large circuit size due to distributed memory in each core
  • Von Neumann systems benefit from single dense central memory
  • Digital devices ill-suited for analog brain behavior
  • Flexibility limitations when incorporating analog elements
  • Scalability challenges for larger models

Memristor Technology

Historical Development

  • 1971: Leon Chua (UC Berkeley) proposed fourth circuit element
  • 2008: HP Labs created physical memristor implementation
  • Metal-dielectric-metal sandwich structure
  • Platinum electrodes, titanium oxide dielectric
  • Two terminals/connection points

Memristor Characteristics

  • Electrical resistance depends on voltage history
  • Non-volatile memory (retains state when power off)
  • Can switch between insulating/conducting states
  • Behavior similar to biological synapses
  • Suitable for in-memory computing applications

Memristor Challenges

  • Manufacturing uniformity and scalability issues
  • Endurance: handling many resistance switch cycles
  • Data retention longevity questions
  • Nonlinear resistance changes complicate programming
  • May require new software paradigms and neural network programming approaches

Competition and Future Outlook

  • Nvidia/GPU advancement following Huang's Law
  • AI chips double performance every 2 years
  • GPU AI performance improved 1,000x in past 10 years
  • Hybrid systems potential: combining neuromorphic and Von Neumann chiplets
  • "Best of both worlds" approach for different system advantages

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome