AI summary by TwinMind
The spelled-out intro to neural networks and backpropagation: building micrograd
Andrej Karpathy
2h 25m 52s
Introduction to Micrograd
- Andre introduces micrograd: autograd engine released on GitHub 2 years ago
- Autograd = automatic gradient, implements backpropagation
- Backpropagation calculates gradient of loss function w.r.t. neural network weights
- Enables iterative weight tuning to minimize loss and improve network accuracy
- Core of modern deep learning libraries like PyTorch, JAX
- Micrograd is scalar-valued (vs tensor-based production libraries)
- Only ~150 lines of code total: engine.py (100 lines) + nn.py (50 lines)
Mathematical Foundation and Derivatives
- Started with scalar function f(x) = 3x² - 4x + 5
- Derivative definition: limit as h→0 of [f(x+h) - f(x)]/h
- Numerical approximation using small h (0.001)
- Examples: derivative at x=3 is 14, at x=-3 is -22, at x=2/3 is 0
- Extended to multi-variable case: d = a*b + c
- Chain rule application for composite functions
Building the Value Class
- Core data structure wrapping scalar values
- Attributes: data, grad (gradient), _prev (children), _op (operation)
- Implemented add and mul operators
- Added convenience for mixed operations (value + scalar)
- Implemented rmul for commutative operations (scalar * value)
- Graph visualization using Graphviz
Manual Backpropagation Example
- Expression: L = (a*b + c) * f where a=2, b=-3, c=10, f=-2
- Forward pass: L = -8
- Backward pass starting with L.grad = 1
- Plus node: distributes gradient equally (local derivative = 1)
- Times node: local derivatives are the other operand values
- Chain rule: multiply local derivative by upstream gradient
- Final gradients: a.grad=6, b.grad=-4, c.grad=-2, f.grad=4
Implementing Automatic Backpropagation
- Added _backward function to each Value node
- Plus operation: self.grad += 1.0 * out.grad, other.grad += 1.0 * out.grad
- Multiply operation: self.grad += other.data * out.grad, other.grad += self.data * out.grad
- Topological sort to ensure correct backward pass ordering
- Gradient accumulation bug fix: use += instead of = to handle multiple uses
Neural Network Components
- Neuron: implements w·x + b followed by activation function
- Implemented tanh activation: (e^(2x) - 1)/(e^(2x) + 1)
- Added exp, pow, and division operations
- Layer: collection of independent neurons
- MLP: multi-layer perceptron with sequential layers
Training Process
- Dataset: 4 examples with binary classification targets
- Loss function: mean squared error Σ(y_pred - y_target)²
- Gradient descent: update weights using -learning_rate * gradient
- Learning rate tuning: too high causes instability, too low is slow
- Zero grad bug: must reset gradients to zero before each backward pass
- Training loop: forward pass → backward pass → parameter update
Advanced Operations
- Implemented tanh as composite of exp operations
- Power function with derivative: n * x^(n-1)
- Subtraction as addition of negation
- Mathematical equivalence verification between atomic and composite operations
PyTorch Comparison
- Showed equivalent implementation using PyTorch tensors
- Same API structure: data, grad attributes, backward() method
- Confirmed identical results for forward and backward passes
- Production libraries use tensors for efficiency and parallelization
Micrograd Codebase Overview
- engine.py: Value class with all operations and backpropagation
- nn.py: Neural network modules (Neuron, Layer, MLP)
- Matches PyTorch API design patterns
- Test suite comparing against PyTorch
- Demo notebook with 2D binary classification example
