AI summary by TwinMind
The spelled-out intro to language modeling: building makemore
Andrej Karpathy
1h 57m 45s
Introduction and Dataset Overview
- Building "makemore" - character-level language model that generates name-like sequences
- Dataset: names.txt with 32,000 names from government website
- Model generates new unique names by learning character-level patterns
- Examples: "dontel", "irot", "zhendi" - sound name-like but are unique
Bigram Language Model Fundamentals
- Character-level approach: treats each name as sequence of individual characters
- Bigram model: predicts next character based only on previous character
- Each word contains multiple training examples (e.g., "isabella" has 8 bigrams)
- Special tokens: start token (.) and end token to mark word boundaries
Data Processing and Counting
- Extract bigrams from dataset using sliding window approach
- Use zip(w, w[1:]) to get consecutive character pairs
- Add special start/end tokens to capture word boundaries
- Count bigram frequencies using dictionary: b[bigram] = b.get(bigram, 0) + 1
- Most common bigrams: "n." (ending with n), "an" (a followed by n)
2D Array Representation
- Convert dictionary counts to 28x28 PyTorch tensor (27 chars + 1 special token)
- Rows: first character of bigram, Columns: second character
- Create character-to-integer mapping: s2i = {char: i for i, char in enumerate(chars)}
- Visualize count matrix with matplotlib showing bigram frequencies
Probability Distributions and Sampling
- Convert counts to probabilities by normalizing rows: p = n.float() / n.sum(1, keepdim=True)
- Use torch.multinomial() for sampling from probability distributions
- Implement sampling loop: start with dot token, sample next character, continue until end token
- Generated names quality limited by bigram model's simplicity
Broadcasting and Tensor Operations
- Broadcasting rules: dimensions must be equal, one, or non-existent
- Critical importance of keepdim=True in sum operations
- Without keepdim: creates row vector, broadcasts incorrectly (normalizes columns)
- With keepdim: creates column vector, broadcasts correctly (normalizes rows)
- Demonstrates subtle bugs possible with broadcasting
Model Evaluation and Loss Function
- Evaluate model quality using negative log-likelihood loss
- Likelihood = product of all probabilities assigned to actual next characters
- Log-likelihood = sum of log probabilities (more convenient)
- Negative log-likelihood = -log-likelihood (loss function to minimize)
- Training loss: ~2.45 for bigram model on full dataset
- Model smoothing: add fake counts (e.g., +1) to prevent zero probabilities
Neural Network Implementation
- Convert bigram model to neural network framework
- Input: one-hot encoded characters (27-dimensional vectors)
- Architecture: single linear layer (27x27 weight matrix W)
- Forward pass: x_enc @ W produces logits
- Softmax: exponentiate logits and normalize to get probabilities
Training Data Preparation
- Create training examples: inputs (x) and targets (y) as integer indices
- Convert to tensors: torch.tensor() vs torch.Tensor() differences
- One-hot encoding: F.one_hot(x, num_classes=27).float()
- Example: "emma" produces 5 training examples (dot→e, e→m, m→m, m→a, a→dot)
Gradient-Based Optimization
- Initialize weights W randomly: torch.randn(27, 27, requires_grad=True)
- Forward pass: logits = x_enc @ W, probs = softmax(logits)
- Loss calculation: -torch.log(probs[range(len(y)), y]).mean()
- Backward pass: loss.backward() computes gradients
- Update: W.data -= learning_rate * W.grad
Training Loop Implementation
- Set gradients to None (more efficient than zero)
- Multiple iterations of forward/backward/update
- Learning rate experimentation: start with 1, can go up to 50+
- Convergence to ~2.45 loss (same as counting approach)
- 100+ iterations typically needed for full dataset
Model Equivalence
- Neural network approach achieves identical results to counting method
- W matrix becomes log-counts after optimization
- One-hot encoding + matrix multiplication = table lookup
- Gradient-based framework more flexible for complex models
Regularization and Smoothing
- L2 regularization: add 0.01 * W.square().mean() to loss
- Pushes weights toward zero (equivalent to count smoothing)
- Regularization strength controls uniformity vs. data fitting
- Higher regularization = more uniform distributions
Sampling from Neural Network
- Same sampling procedure as before
- Convert index to one-hot, multiply by W, apply softmax
- Sample from resulting probability distribution
- Identical samples to counting approach (same underlying model)
