AI summary by TwinMind
The Special Memory Powering the AI Revolution
Asianometry
13m 6s
Introduction to HBM and AI Accelerators
- Referenced AI capacity constraints post on Semi Analysis by Dylan
- Focus on NVIDIA H100 AI super accelerator
- Costs ~$20,000
- Not for gaming
- Uses HBM3 memory from SK Hynix
- AMD Mi 300 accelerator also uses HBM
- Video explores HBM technology powering deep learning revolution
HBM Standard Definition and Architecture
- HBM = High Bandwidth Memory, not special DRAM cell type
- Standard administered by Joint Electron Device Engineering Council (JEDEC)
- Core concept: stacking DRAM dies with multiple independent memory channels
- Goal: provide very high data rate transfers for AI applications amenable to parallelism
- Optional logic base die for signal redistribution, test logic, external communications
- JEDEC defines system requirements but allows vendor differentiation
- Deployment options:
- Direct stacking on CPU/GPU die
- Silicon interposer connecting multiple HBM stacks to GPU (AMD/NVIDIA approach)
Memory Standards Comparison
- JEDEC also administers DDR, LPDDR, GDDR standards
- DDR: general purpose PC memory modules
- GDDR: Graphics Double Data Rate for graphics cards
- Current generation: GDDR6
- Used in gaming cards
- Need for HBM driven by GDDR limitations for AI processing
GDDR Limitations for AI
- Point-to-point connections: each memory channel connects to single module
- Contrast with HBM: channels run through all modules in stack
- Scaling challenges:
- Hard to scale total memory capacity
- Requires shrinking DRAM cells further
- Manufacturing difficulties with new capacitor structures (pillar capacitors)
- HBM vertical stacking enables easier capacity scaling
- Physical size benefits: less horizontal floor space for more memory
- More capacity = less reliance on secondary memory systems
HBM3 Capacity Expansion
- April 2023: SK Hynix unveiled HBM3 with 12 layers, 24GB memory
- Expansion from previous 8-layer HBM3 configurations
Memory Bandwidth Analysis
- Bandwidth = rate of data read/write from memory
- Components: clock frequency, data transfers per clock cycle, memory bus width
- GDDR approach: narrow memory bus, high clock speed ("thin but fast stream")
- Great for sequential calculations, streaming data, physics, rendering
- Downsides: engineering challenges, high power consumption, heat generation
- HBM approach: "wide and slow"
- 1,024 bits per HBM3 stack vs DDR5's 64 bits
- Lower clock speed, wider memory bus ("slow moving but wide river")
- Result: more data transfer with less energy per bit
- 2013 AMD data: 40% decrease in memory power consumption, 65% system performance boost
AMD's HBM Development History
- AMD started HBM development 2007-2008 during die stacking research
- 2015 interview: Brian Black (AMD senior fellow, "Godfather of chiplet")
- Team architected HBM after seeing GDDR5 limits, especially power consumption
- 2011: AMD announced partnership with SK Hynix to replace GDDR standard
Technical Implementation: TSVs and Microbumps
- Through Silicon Vias (TSVs) and microbumps enable HBM functionality
- Thousands of TSVs per HBM package
- TSVs: small holes drilled in memory dies for data transfer to interposer/CPU/GPU
- Advantages over wire bonding:
- Shorter and thinner
- Less parasitic capacitance
- Less real estate usage
- Higher speed, lower power consumption
- Manufacturing challenges:
- Deep reactive ion etching process (anisotropic etch with etchant/inhibitor cycles)
- Substantial packaging and test challenges
- Testing 10,000+ TSVs per HBM die
- Repair difficulties for failed TSVs
Ecosystem Development Challenges
- First commercial HBM product: 2015 AMD Radeon R9 Fury GPU
- Biggest challenge: ecosystem development, not single technical issue
- Long list of "newness" problems requiring solutions
- Supply chain complexity:
- SK Hynix: HBM memory
- UMC (Taiwanese foundry): silicon interposer
- ASE (Taiwan OSAT): packaging and testing
- Underrated aspect: supply chain and ecosystem construction for commercialization
HBM Standard Evolution
- HBM2: 2018
- HBM3: 2020
- NVIDIA H100 equipped with HBM3 or HBM2e (modified HBM2)
Market Analysis and Growth
- Pre-ChatGPT: HBM ~1.5% of dynamic RAM market
- Mordor Intelligence forecast: $2B (2023) to $6.3B (2028)
- Current suppliers and market share (mid-2023 TrendForce data):
- SK Hynix: 53% (leading supplier, pioneered with AMD)
- Samsung: 38% (fierce competitor, disputes market share figures)
- Micron: 9% (only American supplier, significant laggard)
Manufacturing Expansion Plans
- SK Hynix and Samsung each investing 1 trillion won for capacity expansion
- SK Hynix facilities:
- Current: Icheon city fabs
- New: Cheongju city fab
- Samsung facilities:
- Core HBM line: Chonan city (former LCD factory)
- Goal: double HBM3 capacity by end of 2024
- Micron: July announcement of HBM3 samples with 12-die stacking
Financial Impact on Memory Companies
- HBM as bright spot in depressed memory market
- Q2 2023 financial results:
- SK Hynix: $2.2B loss
- Samsung: 95% decline in operating profit ($7.5B to $500M)
- Volatile memory business conditions
Future Applications Beyond AI
- AMD's original vision: not GPU-only technology
- Lower power consumption benefits for constrained environments
- Potential applications:
- Edge computing
- Automotive (abundant optical data, but price/qualification challenges)
- Current focus: AI market driving demand
- HBM transformed from sleepy high-end market to hottest technology
- Supply chain effects still propagating
