AI summary by TwinMind

How do Graphics Cards Work? Exploring GPU Architecture

Branch Education

28m 30s

GPU vs CPU Architecture Comparison

  • Graphics cards perform 36 trillion calculations/second for realistic games like Cyberpunk 2077
  • Equivalent to 4,400 Earths of people doing 1 calculation/second each
  • GPU has 10,000+ cores vs CPU's 24 cores
  • GPU analogy: massive cargo ship (high capacity, slower rate)
  • CPU analogy: jumbo jet (lower capacity, faster rate)
  • Trade-off: massive parallel calculations at slower rate vs few calculations at much faster rate
  • CPUs more flexible: run operating systems, interface with devices, networks
  • GPUs limited to simple arithmetic instructions, can't run OS

GA102 GPU Physical Architecture

  • Built from 28.3 billion transistors
  • Hierarchical organization:
    • 7 Graphics Processing Clusters (GPCs)
    • 12 Streaming Multiprocessors (SMs) per GPC
    • 4 warps per SM + 1 ray tracing core
    • 32 Cuda/shading cores per warp + 1 tensor core
  • Total cores across GPU:
    • 10,752 Cuda cores
    • 336 tensor cores
    • 84 ray tracing cores
  • Three core types:
    • Cuda cores: simple binary calculators (addition, multiply buttons)
    • Tensor cores: matrix multiplication/addition calculators
    • Ray tracing cores: largest but fewest, execute ray tracing algorithms

GPU Chip Binning Process

  • Same GA102 chip used in 3080, 3090, 3080 TI, 3090 TI
  • Manufacturing defects create damaged areas
  • Engineers isolate/deactivate defective regions
  • Chips categorized by number of working cores:
    • 3090 TI: 10,752 cores (flawless)
    • 3090: 10,496 cores
    • 3080 TI: 10,240 cores
    • 3080: 8,704 cores (16 damaged SMs)
  • Different cards also vary by clock speed and memory specs

Individual Cuda Core Design

  • Layout of ~410,000 transistors
  • 50,000 transistors perform A * B + C operation (Fused Multiply Add/FMA)
  • Most common graphics operation
  • Half cores execute FMA using 32-bit floating point
  • Other half use 32-bit integers or floating point
  • Other sections handle negative numbers, bit shifting, bit masking
  • Instruction queuing and result accumulation
  • Simple calculator with limited functions
  • One multiply + one add per clock cycle
  • 3090: 10,496 cores × 1.7 GHz = 35.6 trillion calculations/second

Additional GPU Components

  • 12 Graphics memory controllers around chip edge
  • NV link controllers and PCIe interface
  • 6 MB level 2 SRAM memory cache
  • Giga thread engine manages processing clusters/multiprocessors
  • Special function units (4 per SM) handle division, square root, trigonometric functions

Graphics Card Hardware Components

  • Display ports on one side
  • 12V power connector on other side
  • PCIe pins for motherboard connection
  • Voltage regulator module: converts 12V to 1.1V, supplies hundreds of watts
  • Heat sink with 4 heat pipes carries heat from GPU/memory to radiator fins
  • Fans remove heat from radiator

Graphics Memory (GDDR6X)

  • 24 GB GDDR6X SDRAM manufactured by Micron
  • Loading screens transfer 3D models from SSD to graphics memory
  • GPU has small 6 MB shared L2 cache
  • Continuous transfer between graphics memory and GPU needed
  • 24 chips transfer combined 384 bits at once (bus width)
  • Total bandwidth: ~1.15 TB/second
  • CPU RAM: 64-bit bus width, ~64 GB/second max bandwidth
  • GDDR6X uses PAM 4 encoding: 2 bits via 4 voltage levels
  • GDDR7 uses PAM 3 encoding: 3 voltage levels (0, 1, -1)
    • Three encoding schemes combine binary to ternary digits
    • 276 binary bits sent using 176 ternary digits
  • PAM 3 benefits: reduced encoder complexity, better signal-to-noise ratio, improved power efficiency

High Bandwidth Memory (HBM)

  • Micron's HBM surrounds AI chips
  • Built from stacked DRAM chips
  • Through Silicon Vias (TSVs) connect stack into single chip
  • Forms cube of AI memory
  • HBM 3E: single cube has 24-36 GB
  • Up to 192 GB high-speed memory around AI chip
  • Micron's HBM3 uses 30% less power than competitors
  • AI accelerator systems cost $25,000-$40,000, back-ordered for years

SIMD/SIMT Computational Architecture

  • Embarrassingly parallel operations: little effort to divide into parallel tasks
  • Video game rendering and Bitcoin mining fit this category
  • SIMD: Single Instruction Multiple Data
  • Same instructions repeated across thousands to millions of numbers
  • Example: 3D cowboy hat with 28,000 triangles, 14,000 vertices
  • Model space coordinates converted to world space coordinates
  • Single instruction adds hat origin position to each vertex coordinate
  • Applied to 5,629 objects with 8.3 million vertices total
  • Results in 25 million addition calculations
  • Key: no dependency between calculations, all can run in parallel

Thread Architecture Mapping

  • Each instruction completed by thread
  • Thread matched to single Cuda core
  • 32 threads bundled into warps
  • Same instruction sequence issued to all threads in warp
  • Warps grouped into thread blocks (handled by streaming multiprocessor)
  • Thread blocks grouped into grids (computed across overall GPU)
  • Giga thread engine manages/schedules computations
  • Maps thread blocks to available streaming multiprocessors

SIMD vs SIMT Evolution

  • SIMD (pre-2016): all 32 threads in warp follow same instructions in lockstep
  • SIMT (post-2016): Single Instruction Multiple Threads
  • SIMT difference: individual threads don't need lockstep, can progress at different rates
  • Each thread gets own program counter
  • Threads within SM use shared 128 KB L1 cache
  • Data output by one thread can be used by separate thread
  • More flexibility with warp divergence via data-dependent conditional branching
  • Easier thread reconvergence to reach barrier synchronization
  • "Warp" term from weaving, not warp drives - from 1804 Jacquard Loom using programmable punch cards

Bitcoin Mining Application

  • SHA-256 hashing algorithm creates blockchain blocks
  • Input: transactions, timestamp, additional data, random number (nonce)
  • Output: random 256-bit value
  • Algorithm like lottery ticket generator
  • Winner: first number with first 80 bits all zeros
  • Reward: 3 Bitcoin per winning block
  • GPUs ran thousands of SHA-256 iterations with different nonce values
  • This graphics card: ~95 million SHA-256 hashes/second
  • Modern ASIC miners: 250 trillion hashes/second (equivalent to 2,600 graphics cards)
  • GPUs now obsolete for Bitcoin mining compared to ASICs

Tensor Cores for AI/Neural Networks

  • Take 3 matrices: multiply first two, add third, output result
  • Matrix multiplication and addition calculations done concurrently
  • Neural networks require trillions to quadrillions of matrix operations
  • Typically uses much larger matrices than example shown
  • Designed for generative AI and neural network computations

Ray Tracing Cores

  • Covered in separate video already released
  • Execute ray tracing algorithms for realistic lighting/reflections

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome