AI summary by TwinMind

Building makemore Part 4: Becoming a Backprop Ninja

Andrej Karpathy

1h 55m 24s

Introduction and Motivation

  • Continuing implementation of makemore neural network from previous lectures
  • Focus: removing PyTorch autograd (loss.backward()) and implementing manual backpropagation
  • Backpropagation described as "leaky abstraction" - not magical, requires understanding internals
  • Historical context: 10 years ago manual backprop was standard practice in deep learning
  • Examples from 2006 Hinton paper, 2010 MATLAB implementations, 2014 fragmented embeddings work
  • Manual implementation provides better debugging capabilities and intuitive understanding

Exercise 1: Full Manual Backpropagation

  • Starting with two-layer MLP with batch normalization from previous lecture
  • Forward pass expanded into manageable chunks with intermediate tensors
  • Each tensor gets corresponding gradient tensor (prepended with 'd')
  • Initialization changed to small random numbers instead of zeros to avoid masking gradient errors
  • Bias kept in first layer despite batch normalization for gradient checking purposes

Gradient Calculations (Bottom-Up)

  • dlogprobs: Derivative of loss w.r.t. log probabilities
    • Shape: 32×27 (same as logprobs)
    • Negative 1/n at correct indices, zero elsewhere
    • Loss = -mean(logprobs[range(n), yb])
  • dprobs: Chain rule through log operation
    • Local derivative: 1/probs (derivative of log)
    • Result: dlogprobs / probs
  • dcounts_sum_inv: Backprop through power operation (-0.5)
    • Power rule: -0.5 × counts_sum^(-1.5)
  • dcounts_sum: Backprop through sum operation
    • Broadcasting: sum becomes replication in backward pass
    • Need to maintain dimensions with keepdim=True
  • dcounts: Two branches - direct and through counts_sum
    • Plus equals for accumulating gradients from multiple uses
  • dnorm_logits: Element-wise exp backward
    • Local derivative: exp(norm_logits) = counts
  • dlogits and dlogit_maxes: Backprop through subtraction with broadcasting
    • dlogits gets direct copy, dlogit_maxes gets negative sum
  • dlogits (second branch): Through max operation using indices
    • One-hot encoding to scatter gradients to max positions
    • F.one_hot with logits.max(1).indices

Linear Layer Backpropagation

  • Matrix multiplication backward: A×B + C format
  • dh: dlogits @ W2.T (shape matching: 32×64)
  • dW2: h.T @ dlogits (shape matching: 64×27)
  • db2: dlogits.sum(0) (eliminate batch dimension)
  • Shape-based intuition: only one way to make dimensions work

Activation and Batch Normalization

  • dh_preact: Through tanh activation
    • Local derivative: 1 - h²
  • Batch norm parameters:
    • dbn_gain: (bn_raw × dh_preact).sum(0, keepdim=True)
    • dbn_bias: dh_preact.sum(0, keepdim=True)
    • dbn_raw: bn_gain × dh_preact
  • Batch norm internals: Multiple intermediate steps
    • dbn_var_inv: Power rule for (-0.5) exponent
    • dbn_var: Broadcasting and summation for variance
    • dbn_diff2: Element-wise square backward (2×bn_diff)
    • dbn_diff: Two branches - from bn_raw and bn_diff2
    • dhprebn: Final accumulation from mean and diff branches

Embedding Layer

  • First linear layer: Same pattern as second layer
  • Concatenation: View operation backward - reshape gradients
  • Embedding lookup: For loops to route gradients back to correct rows
    • Multiple occurrences of same character require gradient accumulation

Exercise 2: Efficient Cross-Entropy Backward

  • Mathematical derivation of cross-entropy loss gradient
  • Single analytical expression instead of multiple atomic operations
  • Result: softmax probabilities with -1 at correct positions
  • Implementation: F.softmax(logits, dim=1) then subtract 1 at correct indices
  • Significant speedup over atomic backpropagation
  • Gradient intuition: "forces" pulling down incorrect probabilities, pulling up correct ones

Exercise 3: Efficient Batch Normalization Backward

  • Analytical derivation of batch norm backward pass
  • Mathematical setup: μ, σ², x̂, y with proper variable dependencies
  • Bessel's correction discussion: using 1/(n-1) instead of 1/n for unbiased variance
  • PyTorch train/test mismatch issue in batch normalization
  • Four-step derivation process:
    1. DL/dx̂ = DL/dy × γ
    2. DL/dσ² with summation over all examples
    3. DL/dμ with cancellation of second term
    4. DL/dx with three contribution paths
  • Final compact implementation in single line of code
  • Handles 64 neurons in parallel with proper broadcasting

Exercise 4: Complete Manual Implementation

  • Full neural network training without loss.backward()
  • Gradient checking against PyTorch (approximate equality)
  • Training loop with manual parameter updates
  • torch.no_grad() context for efficiency
  • Final results comparable to PyTorch autograd
  • Complete backward pass in ~20 lines of code

Historical Context and Practical Notes

  • MATLAB era (2010): Manual backprop was standard
  • Python transition and autograd adoption
  • Gradient checking importance for correctness
  • Shape-based intuition for deriving gradients
  • Duality: sum in forward → broadcast in backward, broadcast in forward → sum in backward

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome