AI summary by TwinMind
Building makemore Part 3: Activations & Gradients, BatchNorm
Andrej Karpathy
1h 55m 57s
Introduction and Setup
- Continuing make more implementation from previous lecture
- Previously implemented multilayer perceptron following Bengio 2003 for character-level language modeling
- Goal: understand activations and gradients in neural networks before moving to RNNs/LSTMs/GRUs
- Starting code: cleaned up version with 11,000 parameters, 200,000 training steps, batch size 32
- Achieved train/val loss ~2.16
Initialization Problems - Softmax Confidence Issue
- Initial loss at iteration 0: 27 (way too high)
- Expected initial loss: -log(1/27) = 3.29 for uniform distribution over 27 characters
- Problem: network confidently wrong at initialization due to extreme logit values
- Solution: Scale down final layer weights (W2) by 0.1 to get logits closer to zero
- Result: Initial loss improved from 27 to ~3.3, final validation loss improved from 2.17 to 2.13
Hidden Layer Saturation Problem
- Examined hidden state activations (H) - many values at -1 or +1 (saturated tanh)
- Pre-activations (H_preact) had very broad distribution (-5 to 15)
- Issue: Saturated neurons kill gradients in backprop
- tanh derivative: 1 - t²
- When t ≈ ±1, gradient ≈ 0
- Dead neurons: When entire column saturated, neuron never learns
- Solution: Scale down W1 by 0.2 to get better pre-activation distribution
- Result: Validation loss improved from 2.13 to 2.10
Principled Weight Initialization
- Kaiming initialization: Scale weights by gain/√(fan_in)
- For tanh nonlinearity: gain = 5/3
- Fan_in = embed_dim × block_size = 30
- Standard deviation = (5/3)/√30 ≈ 0.3
- Modern innovations make precise initialization less critical:
- Residual connections
- Normalization layers (batch norm, layer norm, group norm)
- Better optimizers (Adam vs SGD)
Batch Normalization Introduction
- Core idea: If you want unit Gaussian activations, just normalize them
- Formula: (x - μ)/σ where μ, σ calculated over batch
- Scale and shift: Add learnable γ (gain) and β (bias) parameters
- Running statistics: Maintain exponential moving average of μ and σ for inference
- Benefits: Stabilizes training of deep networks, allows higher learning rates
Batch Normalization Implementation Details
- Training mode: Use batch statistics, update running statistics
- Inference mode: Use fixed running statistics
- Momentum: Controls running average update rate (default 0.1, use 0.001 for small batches)
- Epsilon: Small constant (1e-5) to prevent division by zero
- Bias removal: Remove bias from layers before batch norm (it gets subtracted anyway)
Batch Normalization Issues and Tradeoffs
- Coupling problem: Examples in batch affect each other's forward pass
- Regularization effect: Jittering from batch coupling acts as data augmentation
- Inference complexity: Need separate calibration step or running statistics
- Bugs: Coupling creates many edge cases and debugging difficulties
- Alternatives: Layer normalization, group normalization, instance normalization
Diagnostic Tools for Neural Network Training
- Activation histograms: Monitor saturation levels, aim for ~5% saturation
- Gradient histograms: Check for vanishing/exploding gradients
- Weight statistics: Monitor parameter distributions and updates
- Update-to-data ratio: Key metric for learning rate tuning
- Target: ~1e-3 (log scale -3)
- Too high: Learning rate too large
- Too low: Learning rate too small
PyTorch-style Implementation
- Created modular Linear, BatchNorm1d, and Tanh classes
- Linear layer: Implements W×X + b with Kaiming initialization
- BatchNorm1d: Full implementation with training/eval modes, running stats
- Parameters vs buffers: Parameters trained via backprop, buffers updated via exponential moving average
- API compatibility: Matches PyTorch nn.Module interface
Deep Network Analysis
- 6-layer MLP: Linear → BatchNorm → Tanh repeated
- Gain importance: Without proper gain (5/3 for tanh), activations shrink to zero
- Gradient flow: Proper initialization ensures gradients don't vanish/explode
- Last layer problem: Output layer often has different gradient scale
- Batch norm benefits: More robust to initialization, but still need to tune learning rate
Key Insights and Recommendations
- Initialization matters: Especially for deep networks without modern techniques
- Monitor diagnostics: Always plot activation/gradient histograms and update ratios
- Batch norm tradeoffs: Stabilizes training but introduces coupling and complexity
- Update ratio heuristic: Target 1e-3 for stable training
- Modern techniques: Residual connections, better optimizers, normalization layers make training more robust
- Historical context: Pre-2015 deep network training was extremely finicky and required precise calibration
