AI summary by TwinMind

Building makemore Part 2: MLP

Andrej Karpathy

1h 15m 39s

Introduction and Problem Statement

  • Continuing makemore implementation from bigram language model
  • Previous lecture: implemented bigram model using counts and single linear layer neural network
  • Problem with bigram approach: only uses single character context, predictions not very good
  • Scaling issue: table size grows exponentially with context length
    • 1 character context: 27 possibilities
    • 2 characters: 27×27 = 729 possibilities
    • 3 characters: 20,000 possibilities
    • Results in too many rows, too few counts per possibility

Multi-Layer Perceptron Approach

  • Following Bengio et al. 2003 paper (19 pages, very influential)
  • Paper uses word-level language model (17,000 vocabulary)
  • We adapt to character-level with 27 characters
  • Key innovation: word/character embeddings
    • Each word gets 30-dimensional feature vector
    • Initially random, tuned via backpropagation
    • Similar words cluster together in embedding space
    • Enables generalization through embedding similarity

Neural Network Architecture

  • Input: 3 previous characters (block_size = 3)
  • Embedding lookup table C: 17,000 × 30 (paper) / 27 × 2 (our implementation)
  • Hidden layer: 100 neurons (hyperparameter)
  • Output layer: 17,000 neurons (paper) / 27 neurons (our case)
  • Softmax for probability distribution
  • Training via maximum log likelihood

Dataset Creation

  • Block size = 3 (context length)
  • Process all words to create examples
  • Each example: 3 input characters → 1 target character
  • Padding with dots for start of sequences
  • From 5 words: 32 examples initially
  • Full dataset: 228,000 examples

Implementation Details

Embedding Layer

  • PyTorch indexing: C[x] retrieves embeddings for all integers in x
  • Alternative: one-hot encoding @ C (equivalent but less efficient)
  • Embedding table C: 27 × 2 dimensions initially

Hidden Layer

  • Input concatenation: 3 × 2 = 6 dimensional input
  • Methods: torch.cat vs tensor.view()
  • tensor.view() much more efficient (no memory copying)
  • W1: 6 × 100 weights, b1: 100 biases
  • Activation: tanh

Output Layer

  • W2: 100 × 27 weights, b2: 27 biases
  • Logits → softmax → probabilities
  • Loss: F.cross_entropy (more efficient than manual implementation)

Training Process

Learning Rate Finding

  • Exponential search from 0.001 to 1.0
  • Plot learning rate vs loss to find optimal range
  • Sweet spot around 0.1 (10^-1)

Mini-batch Training

  • Batch size: 32 examples
  • Much faster than full dataset forward/backward
  • Approximate gradients but more frequent updates

Optimization Results

  • Initial loss: ~17
  • After training: ~2.3 (better than bigram 2.45)
  • 3,400 parameters initially

Model Scaling and Improvements

Embedding Visualization

  • 2D embeddings allow visualization
  • Vowels (a,e,i,o,u) cluster together
  • Special characters (q, dot) positioned separately
  • Shows learned structure

Scaling Up

  • Hidden layer: 100 → 300 neurons
  • Embedding dimensions: 2 → 10
  • Total parameters: ~11,000
  • Final performance: ~2.17 validation loss

Data Splitting and Evaluation

  • Training split: 80% (25,000 words)
  • Validation split: 10% (3,000 words)
  • Test split: 10% (3,204 words)
  • Training optimizes parameters
  • Validation tunes hyperparameters
  • Test evaluates final performance (use sparingly)

Sampling from Model

  • Start with context of dots
  • Forward pass → logits → softmax → probabilities
  • Sample next character using torch.multinomial
  • Shift context window, repeat until end token
  • Generated names much more realistic than bigram

Key Technical Concepts

PyTorch Internals

  • Tensor storage: always 1D in memory
  • tensor.view(): manipulates metadata, no memory copy
  • Extremely efficient reshaping operation

Cross-Entropy Implementation

  • F.cross_entropy vs manual implementation
  • Numerical stability: subtracts max logit to prevent overflow
  • Fused kernels for efficiency
  • Simpler backward pass derivatives

Optimization Techniques

  • Learning rate decay: 0.1 → 0.01
  • Batch size considerations: 32 → larger for stability
  • Tracking training vs validation loss
  • Early stopping when overfitting begins

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome