AI summary by TwinMind

The spelled-out intro to neural networks and backpropagation: building micrograd

Andrej Karpathy

2h 25m 52s

Introduction to Micrograd

  • Andre introduces micrograd: autograd engine released on GitHub 2 years ago
  • Autograd = automatic gradient, implements backpropagation
  • Backpropagation calculates gradient of loss function w.r.t. neural network weights
  • Enables iterative weight tuning to minimize loss and improve network accuracy
  • Core of modern deep learning libraries like PyTorch, JAX
  • Micrograd is scalar-valued (vs tensor-based production libraries)
  • Only ~150 lines of code total: engine.py (100 lines) + nn.py (50 lines)

Mathematical Foundation and Derivatives

  • Started with scalar function f(x) = 3x² - 4x + 5
  • Derivative definition: limit as h→0 of [f(x+h) - f(x)]/h
  • Numerical approximation using small h (0.001)
  • Examples: derivative at x=3 is 14, at x=-3 is -22, at x=2/3 is 0
  • Extended to multi-variable case: d = a*b + c
  • Chain rule application for composite functions

Building the Value Class

  • Core data structure wrapping scalar values
  • Attributes: data, grad (gradient), _prev (children), _op (operation)
  • Implemented add and mul operators
  • Added convenience for mixed operations (value + scalar)
  • Implemented rmul for commutative operations (scalar * value)
  • Graph visualization using Graphviz

Manual Backpropagation Example

  • Expression: L = (a*b + c) * f where a=2, b=-3, c=10, f=-2
  • Forward pass: L = -8
  • Backward pass starting with L.grad = 1
  • Plus node: distributes gradient equally (local derivative = 1)
  • Times node: local derivatives are the other operand values
  • Chain rule: multiply local derivative by upstream gradient
  • Final gradients: a.grad=6, b.grad=-4, c.grad=-2, f.grad=4

Implementing Automatic Backpropagation

  • Added _backward function to each Value node
  • Plus operation: self.grad += 1.0 * out.grad, other.grad += 1.0 * out.grad
  • Multiply operation: self.grad += other.data * out.grad, other.grad += self.data * out.grad
  • Topological sort to ensure correct backward pass ordering
  • Gradient accumulation bug fix: use += instead of = to handle multiple uses

Neural Network Components

  • Neuron: implements w·x + b followed by activation function
  • Implemented tanh activation: (e^(2x) - 1)/(e^(2x) + 1)
  • Added exp, pow, and division operations
  • Layer: collection of independent neurons
  • MLP: multi-layer perceptron with sequential layers

Training Process

  • Dataset: 4 examples with binary classification targets
  • Loss function: mean squared error Σ(y_pred - y_target)²
  • Gradient descent: update weights using -learning_rate * gradient
  • Learning rate tuning: too high causes instability, too low is slow
  • Zero grad bug: must reset gradients to zero before each backward pass
  • Training loop: forward pass → backward pass → parameter update

Advanced Operations

  • Implemented tanh as composite of exp operations
  • Power function with derivative: n * x^(n-1)
  • Subtraction as addition of negation
  • Mathematical equivalence verification between atomic and composite operations

PyTorch Comparison

  • Showed equivalent implementation using PyTorch tensors
  • Same API structure: data, grad attributes, backward() method
  • Confirmed identical results for forward and backward passes
  • Production libraries use tensors for efficiency and parallelization

Micrograd Codebase Overview

  • engine.py: Value class with all operations and backpropagation
  • nn.py: Neural network modules (Neuron, Layer, MLP)
  • Matches PyTorch API design patterns
  • Test suite comparing against PyTorch
  • Demo notebook with 2D binary classification example

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome