Introduction and Problem Statement

• Continuing makemore implementation from bigram language model
• Previous lecture: implemented bigram model using counts and single linear layer neural network
• Problem with bigram approach: only uses single character context, predictions not very good
• Scaling issue: table size grows exponentially with context length
  • 1 character context: 27 possibilities
  • 2 characters: 27×27 = 729 possibilities
  • 3 characters: 20,000 possibilities
  • Results in too many rows, too few counts per possibility

Multi-Layer Perceptron Approach

• Following Bengio et al. 2003 paper (19 pages, very influential)
• Paper uses word-level language model (17,000 vocabulary)
• We adapt to character-level with 27 characters
• Key innovation: word/character embeddings
  • Each word gets 30-dimensional feature vector
  • Initially random, tuned via backpropagation
  • Similar words cluster together in embedding space
  • Enables generalization through embedding similarity

Neural Network Architecture

• Input: 3 previous characters (block_size = 3)
• Embedding lookup table C: 17,000 × 30 (paper) / 27 × 2 (our implementation)
• Hidden layer: 100 neurons (hyperparameter)
• Output layer: 17,000 neurons (paper) / 27 neurons (our case)
• Softmax for probability distribution
• Training via maximum log likelihood

Dataset Creation

• Block size = 3 (context length)
• Process all words to create examples
• Each example: 3 input characters → 1 target character
• Padding with dots for start of sequences
• From 5 words: 32 examples initially
• Full dataset: 228,000 examples

Implementation Details

Embedding Layer

• PyTorch indexing: C[x] retrieves embeddings for all integers in x
• Alternative: one-hot encoding @ C (equivalent but less efficient)
• Embedding table C: 27 × 2 dimensions initially

Hidden Layer

• Input concatenation: 3 × 2 = 6 dimensional input
• Methods: torch.cat vs tensor.view()
• tensor.view() much more efficient (no memory copying)
• W1: 6 × 100 weights, b1: 100 biases
• Activation: tanh

Output Layer

• W2: 100 × 27 weights, b2: 27 biases
• Logits → softmax → probabilities
• Loss: F.cross_entropy (more efficient than manual implementation)

Training Process

Learning Rate Finding

• Exponential search from 0.001 to 1.0
• Plot learning rate vs loss to find optimal range
• Sweet spot around 0.1 (10^-1)

Mini-batch Training

• Batch size: 32 examples
• Much faster than full dataset forward/backward
• Approximate gradients but more frequent updates

Optimization Results

• Initial loss: ~17
• After training: ~2.3 (better than bigram 2.45)
• 3,400 parameters initially

Model Scaling and Improvements

Embedding Visualization

• 2D embeddings allow visualization
• Vowels (a,e,i,o,u) cluster together
• Special characters (q, dot) positioned separately
• Shows learned structure

Scaling Up

• Hidden layer: 100 → 300 neurons
• Embedding dimensions: 2 → 10
• Total parameters: ~11,000
• Final performance: ~2.17 validation loss

Data Splitting and Evaluation

• Training split: 80% (25,000 words)
• Validation split: 10% (3,000 words)
• Test split: 10% (3,204 words)
• Training optimizes parameters
• Validation tunes hyperparameters
• Test evaluates final performance (use sparingly)

Sampling from Model

• Start with context of dots
• Forward pass → logits → softmax → probabilities
• Sample next character using torch.multinomial
• Shift context window, repeat until end token
• Generated names much more realistic than bigram

Key Technical Concepts

PyTorch Internals

• Tensor storage: always 1D in memory
• tensor.view(): manipulates metadata, no memory copy
• Extremely efficient reshaping operation

Cross-Entropy Implementation

• F.cross_entropy vs manual implementation
• Numerical stability: subtracts max logit to prevent overflow
• Fused kernels for efficiency
• Simpler backward pass derivatives

Optimization Techniques

• Learning rate decay: 0.1 → 0.01
• Batch size considerations: 32 → larger for stability
• Tracking training vs validation loss
• Early stopping when overfitting begins