AI summary by TwinMind

Building makemore Part 3: Activations & Gradients, BatchNorm

Andrej Karpathy

1h 55m 57s

Introduction and Setup

  • Continuing make more implementation from previous lecture
  • Previously implemented multilayer perceptron following Bengio 2003 for character-level language modeling
  • Goal: understand activations and gradients in neural networks before moving to RNNs/LSTMs/GRUs
  • Starting code: cleaned up version with 11,000 parameters, 200,000 training steps, batch size 32
  • Achieved train/val loss ~2.16

Initialization Problems - Softmax Confidence Issue

  • Initial loss at iteration 0: 27 (way too high)
  • Expected initial loss: -log(1/27) = 3.29 for uniform distribution over 27 characters
  • Problem: network confidently wrong at initialization due to extreme logit values
  • Solution: Scale down final layer weights (W2) by 0.1 to get logits closer to zero
  • Result: Initial loss improved from 27 to ~3.3, final validation loss improved from 2.17 to 2.13

Hidden Layer Saturation Problem

  • Examined hidden state activations (H) - many values at -1 or +1 (saturated tanh)
  • Pre-activations (H_preact) had very broad distribution (-5 to 15)
  • Issue: Saturated neurons kill gradients in backprop
    • tanh derivative: 1 - t²
    • When t ≈ ±1, gradient ≈ 0
  • Dead neurons: When entire column saturated, neuron never learns
  • Solution: Scale down W1 by 0.2 to get better pre-activation distribution
  • Result: Validation loss improved from 2.13 to 2.10

Principled Weight Initialization

  • Kaiming initialization: Scale weights by gain/√(fan_in)
  • For tanh nonlinearity: gain = 5/3
  • Fan_in = embed_dim × block_size = 30
  • Standard deviation = (5/3)/√30 ≈ 0.3
  • Modern innovations make precise initialization less critical:
    • Residual connections
    • Normalization layers (batch norm, layer norm, group norm)
    • Better optimizers (Adam vs SGD)

Batch Normalization Introduction

  • Core idea: If you want unit Gaussian activations, just normalize them
  • Formula: (x - μ)/σ where μ, σ calculated over batch
  • Scale and shift: Add learnable γ (gain) and β (bias) parameters
  • Running statistics: Maintain exponential moving average of μ and σ for inference
  • Benefits: Stabilizes training of deep networks, allows higher learning rates

Batch Normalization Implementation Details

  • Training mode: Use batch statistics, update running statistics
  • Inference mode: Use fixed running statistics
  • Momentum: Controls running average update rate (default 0.1, use 0.001 for small batches)
  • Epsilon: Small constant (1e-5) to prevent division by zero
  • Bias removal: Remove bias from layers before batch norm (it gets subtracted anyway)

Batch Normalization Issues and Tradeoffs

  • Coupling problem: Examples in batch affect each other's forward pass
  • Regularization effect: Jittering from batch coupling acts as data augmentation
  • Inference complexity: Need separate calibration step or running statistics
  • Bugs: Coupling creates many edge cases and debugging difficulties
  • Alternatives: Layer normalization, group normalization, instance normalization

Diagnostic Tools for Neural Network Training

  • Activation histograms: Monitor saturation levels, aim for ~5% saturation
  • Gradient histograms: Check for vanishing/exploding gradients
  • Weight statistics: Monitor parameter distributions and updates
  • Update-to-data ratio: Key metric for learning rate tuning
    • Target: ~1e-3 (log scale -3)
    • Too high: Learning rate too large
    • Too low: Learning rate too small

PyTorch-style Implementation

  • Created modular Linear, BatchNorm1d, and Tanh classes
  • Linear layer: Implements W×X + b with Kaiming initialization
  • BatchNorm1d: Full implementation with training/eval modes, running stats
  • Parameters vs buffers: Parameters trained via backprop, buffers updated via exponential moving average
  • API compatibility: Matches PyTorch nn.Module interface

Deep Network Analysis

  • 6-layer MLP: Linear → BatchNorm → Tanh repeated
  • Gain importance: Without proper gain (5/3 for tanh), activations shrink to zero
  • Gradient flow: Proper initialization ensures gradients don't vanish/explode
  • Last layer problem: Output layer often has different gradient scale
  • Batch norm benefits: More robust to initialization, but still need to tune learning rate

Key Insights and Recommendations

  • Initialization matters: Especially for deep networks without modern techniques
  • Monitor diagnostics: Always plot activation/gradient histograms and update ratios
  • Batch norm tradeoffs: Stabilizes training but introduces coupling and complexity
  • Update ratio heuristic: Target 1e-3 for stable training
  • Modern techniques: Residual connections, better optimizers, normalization layers make training more robust
  • Historical context: Pre-2015 deep network training was extremely finicky and required precise calibration

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome