AI summary by TwinMind
How do Graphics Cards Work? Exploring GPU Architecture
Branch Education
28m 30s
GPU vs CPU Architecture Comparison
- Graphics cards perform 36 trillion calculations/second for realistic games like Cyberpunk 2077
- Equivalent to 4,400 Earths of people doing 1 calculation/second each
- GPU has 10,000+ cores vs CPU's 24 cores
- GPU analogy: massive cargo ship (high capacity, slower rate)
- CPU analogy: jumbo jet (lower capacity, faster rate)
- Trade-off: massive parallel calculations at slower rate vs few calculations at much faster rate
- CPUs more flexible: run operating systems, interface with devices, networks
- GPUs limited to simple arithmetic instructions, can't run OS
GA102 GPU Physical Architecture
- Built from 28.3 billion transistors
- Hierarchical organization:
- 7 Graphics Processing Clusters (GPCs)
- 12 Streaming Multiprocessors (SMs) per GPC
- 4 warps per SM + 1 ray tracing core
- 32 Cuda/shading cores per warp + 1 tensor core
- Total cores across GPU:
- 10,752 Cuda cores
- 336 tensor cores
- 84 ray tracing cores
- Three core types:
- Cuda cores: simple binary calculators (addition, multiply buttons)
- Tensor cores: matrix multiplication/addition calculators
- Ray tracing cores: largest but fewest, execute ray tracing algorithms
GPU Chip Binning Process
- Same GA102 chip used in 3080, 3090, 3080 TI, 3090 TI
- Manufacturing defects create damaged areas
- Engineers isolate/deactivate defective regions
- Chips categorized by number of working cores:
- 3090 TI: 10,752 cores (flawless)
- 3090: 10,496 cores
- 3080 TI: 10,240 cores
- 3080: 8,704 cores (16 damaged SMs)
- Different cards also vary by clock speed and memory specs
Individual Cuda Core Design
- Layout of ~410,000 transistors
- 50,000 transistors perform A * B + C operation (Fused Multiply Add/FMA)
- Most common graphics operation
- Half cores execute FMA using 32-bit floating point
- Other half use 32-bit integers or floating point
- Other sections handle negative numbers, bit shifting, bit masking
- Instruction queuing and result accumulation
- Simple calculator with limited functions
- One multiply + one add per clock cycle
- 3090: 10,496 cores × 1.7 GHz = 35.6 trillion calculations/second
Additional GPU Components
- 12 Graphics memory controllers around chip edge
- NV link controllers and PCIe interface
- 6 MB level 2 SRAM memory cache
- Giga thread engine manages processing clusters/multiprocessors
- Special function units (4 per SM) handle division, square root, trigonometric functions
Graphics Card Hardware Components
- Display ports on one side
- 12V power connector on other side
- PCIe pins for motherboard connection
- Voltage regulator module: converts 12V to 1.1V, supplies hundreds of watts
- Heat sink with 4 heat pipes carries heat from GPU/memory to radiator fins
- Fans remove heat from radiator
Graphics Memory (GDDR6X)
- 24 GB GDDR6X SDRAM manufactured by Micron
- Loading screens transfer 3D models from SSD to graphics memory
- GPU has small 6 MB shared L2 cache
- Continuous transfer between graphics memory and GPU needed
- 24 chips transfer combined 384 bits at once (bus width)
- Total bandwidth: ~1.15 TB/second
- CPU RAM: 64-bit bus width, ~64 GB/second max bandwidth
- GDDR6X uses PAM 4 encoding: 2 bits via 4 voltage levels
- GDDR7 uses PAM 3 encoding: 3 voltage levels (0, 1, -1)
- Three encoding schemes combine binary to ternary digits
- 276 binary bits sent using 176 ternary digits
- PAM 3 benefits: reduced encoder complexity, better signal-to-noise ratio, improved power efficiency
High Bandwidth Memory (HBM)
- Micron's HBM surrounds AI chips
- Built from stacked DRAM chips
- Through Silicon Vias (TSVs) connect stack into single chip
- Forms cube of AI memory
- HBM 3E: single cube has 24-36 GB
- Up to 192 GB high-speed memory around AI chip
- Micron's HBM3 uses 30% less power than competitors
- AI accelerator systems cost $25,000-$40,000, back-ordered for years
SIMD/SIMT Computational Architecture
- Embarrassingly parallel operations: little effort to divide into parallel tasks
- Video game rendering and Bitcoin mining fit this category
- SIMD: Single Instruction Multiple Data
- Same instructions repeated across thousands to millions of numbers
- Example: 3D cowboy hat with 28,000 triangles, 14,000 vertices
- Model space coordinates converted to world space coordinates
- Single instruction adds hat origin position to each vertex coordinate
- Applied to 5,629 objects with 8.3 million vertices total
- Results in 25 million addition calculations
- Key: no dependency between calculations, all can run in parallel
Thread Architecture Mapping
- Each instruction completed by thread
- Thread matched to single Cuda core
- 32 threads bundled into warps
- Same instruction sequence issued to all threads in warp
- Warps grouped into thread blocks (handled by streaming multiprocessor)
- Thread blocks grouped into grids (computed across overall GPU)
- Giga thread engine manages/schedules computations
- Maps thread blocks to available streaming multiprocessors
SIMD vs SIMT Evolution
- SIMD (pre-2016): all 32 threads in warp follow same instructions in lockstep
- SIMT (post-2016): Single Instruction Multiple Threads
- SIMT difference: individual threads don't need lockstep, can progress at different rates
- Each thread gets own program counter
- Threads within SM use shared 128 KB L1 cache
- Data output by one thread can be used by separate thread
- More flexibility with warp divergence via data-dependent conditional branching
- Easier thread reconvergence to reach barrier synchronization
- "Warp" term from weaving, not warp drives - from 1804 Jacquard Loom using programmable punch cards
Bitcoin Mining Application
- SHA-256 hashing algorithm creates blockchain blocks
- Input: transactions, timestamp, additional data, random number (nonce)
- Output: random 256-bit value
- Algorithm like lottery ticket generator
- Winner: first number with first 80 bits all zeros
- Reward: 3 Bitcoin per winning block
- GPUs ran thousands of SHA-256 iterations with different nonce values
- This graphics card: ~95 million SHA-256 hashes/second
- Modern ASIC miners: 250 trillion hashes/second (equivalent to 2,600 graphics cards)
- GPUs now obsolete for Bitcoin mining compared to ASICs
Tensor Cores for AI/Neural Networks
- Take 3 matrices: multiply first two, add third, output result
- Matrix multiplication and addition calculations done concurrently
- Neural networks require trillions to quadrillions of matrix operations
- Typically uses much larger matrices than example shown
- Designed for generative AI and neural network computations
Ray Tracing Cores
- Covered in separate video already released
- Execute ray tracing algorithms for realistic lighting/reflections
