AI summary by TwinMind

The Special Memory Powering the AI Revolution

Asianometry

13m 6s

Introduction to HBM and AI Accelerators

  • Referenced AI capacity constraints post on Semi Analysis by Dylan
  • Focus on NVIDIA H100 AI super accelerator
    • Costs ~$20,000
    • Not for gaming
    • Uses HBM3 memory from SK Hynix
  • AMD Mi 300 accelerator also uses HBM
  • Video explores HBM technology powering deep learning revolution

HBM Standard Definition and Architecture

  • HBM = High Bandwidth Memory, not special DRAM cell type
  • Standard administered by Joint Electron Device Engineering Council (JEDEC)
  • Core concept: stacking DRAM dies with multiple independent memory channels
  • Goal: provide very high data rate transfers for AI applications amenable to parallelism
  • Optional logic base die for signal redistribution, test logic, external communications
  • JEDEC defines system requirements but allows vendor differentiation
  • Deployment options:
    • Direct stacking on CPU/GPU die
    • Silicon interposer connecting multiple HBM stacks to GPU (AMD/NVIDIA approach)

Memory Standards Comparison

  • JEDEC also administers DDR, LPDDR, GDDR standards
  • DDR: general purpose PC memory modules
  • GDDR: Graphics Double Data Rate for graphics cards
    • Current generation: GDDR6
    • Used in gaming cards
  • Need for HBM driven by GDDR limitations for AI processing

GDDR Limitations for AI

  • Point-to-point connections: each memory channel connects to single module
  • Contrast with HBM: channels run through all modules in stack
  • Scaling challenges:
    • Hard to scale total memory capacity
    • Requires shrinking DRAM cells further
    • Manufacturing difficulties with new capacitor structures (pillar capacitors)
  • HBM vertical stacking enables easier capacity scaling
  • Physical size benefits: less horizontal floor space for more memory
  • More capacity = less reliance on secondary memory systems

HBM3 Capacity Expansion

  • April 2023: SK Hynix unveiled HBM3 with 12 layers, 24GB memory
  • Expansion from previous 8-layer HBM3 configurations

Memory Bandwidth Analysis

  • Bandwidth = rate of data read/write from memory
  • Components: clock frequency, data transfers per clock cycle, memory bus width
  • GDDR approach: narrow memory bus, high clock speed ("thin but fast stream")
    • Great for sequential calculations, streaming data, physics, rendering
    • Downsides: engineering challenges, high power consumption, heat generation
  • HBM approach: "wide and slow"
    • 1,024 bits per HBM3 stack vs DDR5's 64 bits
    • Lower clock speed, wider memory bus ("slow moving but wide river")
    • Result: more data transfer with less energy per bit
  • 2013 AMD data: 40% decrease in memory power consumption, 65% system performance boost

AMD's HBM Development History

  • AMD started HBM development 2007-2008 during die stacking research
  • 2015 interview: Brian Black (AMD senior fellow, "Godfather of chiplet")
  • Team architected HBM after seeing GDDR5 limits, especially power consumption
  • 2011: AMD announced partnership with SK Hynix to replace GDDR standard

Technical Implementation: TSVs and Microbumps

  • Through Silicon Vias (TSVs) and microbumps enable HBM functionality
  • Thousands of TSVs per HBM package
  • TSVs: small holes drilled in memory dies for data transfer to interposer/CPU/GPU
  • Advantages over wire bonding:
    • Shorter and thinner
    • Less parasitic capacitance
    • Less real estate usage
    • Higher speed, lower power consumption
  • Manufacturing challenges:
    • Deep reactive ion etching process (anisotropic etch with etchant/inhibitor cycles)
    • Substantial packaging and test challenges
    • Testing 10,000+ TSVs per HBM die
    • Repair difficulties for failed TSVs

Ecosystem Development Challenges

  • First commercial HBM product: 2015 AMD Radeon R9 Fury GPU
  • Biggest challenge: ecosystem development, not single technical issue
  • Long list of "newness" problems requiring solutions
  • Supply chain complexity:
    • SK Hynix: HBM memory
    • UMC (Taiwanese foundry): silicon interposer
    • ASE (Taiwan OSAT): packaging and testing
  • Underrated aspect: supply chain and ecosystem construction for commercialization

HBM Standard Evolution

  • HBM2: 2018
  • HBM3: 2020
  • NVIDIA H100 equipped with HBM3 or HBM2e (modified HBM2)

Market Analysis and Growth

  • Pre-ChatGPT: HBM ~1.5% of dynamic RAM market
  • Mordor Intelligence forecast: $2B (2023) to $6.3B (2028)
  • Current suppliers and market share (mid-2023 TrendForce data):
    • SK Hynix: 53% (leading supplier, pioneered with AMD)
    • Samsung: 38% (fierce competitor, disputes market share figures)
    • Micron: 9% (only American supplier, significant laggard)

Manufacturing Expansion Plans

  • SK Hynix and Samsung each investing 1 trillion won for capacity expansion
  • SK Hynix facilities:
    • Current: Icheon city fabs
    • New: Cheongju city fab
  • Samsung facilities:
    • Core HBM line: Chonan city (former LCD factory)
    • Goal: double HBM3 capacity by end of 2024
  • Micron: July announcement of HBM3 samples with 12-die stacking

Financial Impact on Memory Companies

  • HBM as bright spot in depressed memory market
  • Q2 2023 financial results:
    • SK Hynix: $2.2B loss
    • Samsung: 95% decline in operating profit ($7.5B to $500M)
  • Volatile memory business conditions

Future Applications Beyond AI

  • AMD's original vision: not GPU-only technology
  • Lower power consumption benefits for constrained environments
  • Potential applications:
    • Edge computing
    • Automotive (abundant optical data, but price/qualification challenges)
  • Current focus: AI market driving demand
  • HBM transformed from sleepy high-end market to hottest technology
  • Supply chain effects still propagating

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome