AI summary by TwinMind
Building makemore Part 4: Becoming a Backprop Ninja
Andrej Karpathy
1h 55m 24s
Introduction and Motivation
- Continuing implementation of makemore neural network from previous lectures
- Focus: removing PyTorch autograd (loss.backward()) and implementing manual backpropagation
- Backpropagation described as "leaky abstraction" - not magical, requires understanding internals
- Historical context: 10 years ago manual backprop was standard practice in deep learning
- Examples from 2006 Hinton paper, 2010 MATLAB implementations, 2014 fragmented embeddings work
- Manual implementation provides better debugging capabilities and intuitive understanding
Exercise 1: Full Manual Backpropagation
- Starting with two-layer MLP with batch normalization from previous lecture
- Forward pass expanded into manageable chunks with intermediate tensors
- Each tensor gets corresponding gradient tensor (prepended with 'd')
- Initialization changed to small random numbers instead of zeros to avoid masking gradient errors
- Bias kept in first layer despite batch normalization for gradient checking purposes
Gradient Calculations (Bottom-Up)
- dlogprobs: Derivative of loss w.r.t. log probabilities
- Shape: 32×27 (same as logprobs)
- Negative 1/n at correct indices, zero elsewhere
- Loss = -mean(logprobs[range(n), yb])
- dprobs: Chain rule through log operation
- Local derivative: 1/probs (derivative of log)
- Result: dlogprobs / probs
- dcounts_sum_inv: Backprop through power operation (-0.5)
- Power rule: -0.5 × counts_sum^(-1.5)
- dcounts_sum: Backprop through sum operation
- Broadcasting: sum becomes replication in backward pass
- Need to maintain dimensions with keepdim=True
- dcounts: Two branches - direct and through counts_sum
- Plus equals for accumulating gradients from multiple uses
- dnorm_logits: Element-wise exp backward
- Local derivative: exp(norm_logits) = counts
- dlogits and dlogit_maxes: Backprop through subtraction with broadcasting
- dlogits gets direct copy, dlogit_maxes gets negative sum
- dlogits (second branch): Through max operation using indices
- One-hot encoding to scatter gradients to max positions
- F.one_hot with logits.max(1).indices
Linear Layer Backpropagation
- Matrix multiplication backward: A×B + C format
- dh: dlogits @ W2.T (shape matching: 32×64)
- dW2: h.T @ dlogits (shape matching: 64×27)
- db2: dlogits.sum(0) (eliminate batch dimension)
- Shape-based intuition: only one way to make dimensions work
Activation and Batch Normalization
- dh_preact: Through tanh activation
- Local derivative: 1 - h²
- Batch norm parameters:
- dbn_gain: (bn_raw × dh_preact).sum(0, keepdim=True)
- dbn_bias: dh_preact.sum(0, keepdim=True)
- dbn_raw: bn_gain × dh_preact
- Batch norm internals: Multiple intermediate steps
- dbn_var_inv: Power rule for (-0.5) exponent
- dbn_var: Broadcasting and summation for variance
- dbn_diff2: Element-wise square backward (2×bn_diff)
- dbn_diff: Two branches - from bn_raw and bn_diff2
- dhprebn: Final accumulation from mean and diff branches
Embedding Layer
- First linear layer: Same pattern as second layer
- Concatenation: View operation backward - reshape gradients
- Embedding lookup: For loops to route gradients back to correct rows
- Multiple occurrences of same character require gradient accumulation
Exercise 2: Efficient Cross-Entropy Backward
- Mathematical derivation of cross-entropy loss gradient
- Single analytical expression instead of multiple atomic operations
- Result: softmax probabilities with -1 at correct positions
- Implementation: F.softmax(logits, dim=1) then subtract 1 at correct indices
- Significant speedup over atomic backpropagation
- Gradient intuition: "forces" pulling down incorrect probabilities, pulling up correct ones
Exercise 3: Efficient Batch Normalization Backward
- Analytical derivation of batch norm backward pass
- Mathematical setup: μ, σ², x̂, y with proper variable dependencies
- Bessel's correction discussion: using 1/(n-1) instead of 1/n for unbiased variance
- PyTorch train/test mismatch issue in batch normalization
- Four-step derivation process:
- DL/dx̂ = DL/dy × γ
- DL/dσ² with summation over all examples
- DL/dμ with cancellation of second term
- DL/dx with three contribution paths
- Final compact implementation in single line of code
- Handles 64 neurons in parallel with proper broadcasting
Exercise 4: Complete Manual Implementation
- Full neural network training without loss.backward()
- Gradient checking against PyTorch (approximate equality)
- Training loop with manual parameter updates
- torch.no_grad() context for efficiency
- Final results comparable to PyTorch autograd
- Complete backward pass in ~20 lines of code
Historical Context and Practical Notes
- MATLAB era (2010): Manual backprop was standard
- Python transition and autograd adoption
- Gradient checking importance for correctness
- Shape-based intuition for deriving gradients
- Duality: sum in forward → broadcast in backward, broadcast in forward → sum in backward
