AI summary by TwinMind

The spelled-out intro to language modeling: building makemore

Andrej Karpathy

1h 57m 45s

Introduction and Dataset Overview

  • Building "makemore" - character-level language model that generates name-like sequences
  • Dataset: names.txt with 32,000 names from government website
  • Model generates new unique names by learning character-level patterns
  • Examples: "dontel", "irot", "zhendi" - sound name-like but are unique

Bigram Language Model Fundamentals

  • Character-level approach: treats each name as sequence of individual characters
  • Bigram model: predicts next character based only on previous character
  • Each word contains multiple training examples (e.g., "isabella" has 8 bigrams)
  • Special tokens: start token (.) and end token to mark word boundaries

Data Processing and Counting

  • Extract bigrams from dataset using sliding window approach
  • Use zip(w, w[1:]) to get consecutive character pairs
  • Add special start/end tokens to capture word boundaries
  • Count bigram frequencies using dictionary: b[bigram] = b.get(bigram, 0) + 1
  • Most common bigrams: "n." (ending with n), "an" (a followed by n)

2D Array Representation

  • Convert dictionary counts to 28x28 PyTorch tensor (27 chars + 1 special token)
  • Rows: first character of bigram, Columns: second character
  • Create character-to-integer mapping: s2i = {char: i for i, char in enumerate(chars)}
  • Visualize count matrix with matplotlib showing bigram frequencies

Probability Distributions and Sampling

  • Convert counts to probabilities by normalizing rows: p = n.float() / n.sum(1, keepdim=True)
  • Use torch.multinomial() for sampling from probability distributions
  • Implement sampling loop: start with dot token, sample next character, continue until end token
  • Generated names quality limited by bigram model's simplicity

Broadcasting and Tensor Operations

  • Broadcasting rules: dimensions must be equal, one, or non-existent
  • Critical importance of keepdim=True in sum operations
  • Without keepdim: creates row vector, broadcasts incorrectly (normalizes columns)
  • With keepdim: creates column vector, broadcasts correctly (normalizes rows)
  • Demonstrates subtle bugs possible with broadcasting

Model Evaluation and Loss Function

  • Evaluate model quality using negative log-likelihood loss
  • Likelihood = product of all probabilities assigned to actual next characters
  • Log-likelihood = sum of log probabilities (more convenient)
  • Negative log-likelihood = -log-likelihood (loss function to minimize)
  • Training loss: ~2.45 for bigram model on full dataset
  • Model smoothing: add fake counts (e.g., +1) to prevent zero probabilities

Neural Network Implementation

  • Convert bigram model to neural network framework
  • Input: one-hot encoded characters (27-dimensional vectors)
  • Architecture: single linear layer (27x27 weight matrix W)
  • Forward pass: x_enc @ W produces logits
  • Softmax: exponentiate logits and normalize to get probabilities

Training Data Preparation

  • Create training examples: inputs (x) and targets (y) as integer indices
  • Convert to tensors: torch.tensor() vs torch.Tensor() differences
  • One-hot encoding: F.one_hot(x, num_classes=27).float()
  • Example: "emma" produces 5 training examples (dot→e, e→m, m→m, m→a, a→dot)

Gradient-Based Optimization

  • Initialize weights W randomly: torch.randn(27, 27, requires_grad=True)
  • Forward pass: logits = x_enc @ W, probs = softmax(logits)
  • Loss calculation: -torch.log(probs[range(len(y)), y]).mean()
  • Backward pass: loss.backward() computes gradients
  • Update: W.data -= learning_rate * W.grad

Training Loop Implementation

  • Set gradients to None (more efficient than zero)
  • Multiple iterations of forward/backward/update
  • Learning rate experimentation: start with 1, can go up to 50+
  • Convergence to ~2.45 loss (same as counting approach)
  • 100+ iterations typically needed for full dataset

Model Equivalence

  • Neural network approach achieves identical results to counting method
  • W matrix becomes log-counts after optimization
  • One-hot encoding + matrix multiplication = table lookup
  • Gradient-based framework more flexible for complex models

Regularization and Smoothing

  • L2 regularization: add 0.01 * W.square().mean() to loss
  • Pushes weights toward zero (equivalent to count smoothing)
  • Regularization strength controls uniformity vs. data fitting
  • Higher regularization = more uniform distributions

Sampling from Neural Network

  • Same sampling procedure as before
  • Convert index to one-hot, multiply by W, apply softmax
  • Sample from resulting probability distribution
  • Identical samples to counting approach (same underlying model)

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome