AI summary by TwinMind
Building makemore Part 5: Building a WaveNet
Andrej Karpathy
56m 21s
Introduction and Setup
- Continuing implementation of "makemore" character-level language model from Kyoto hotel room
- Previous architecture: multi-layer perceptron with 3 previous characters predicting 4th character
- Goal: complexify architecture to take more characters as input and avoid crushing information too quickly
- Target: implement hierarchical structure similar to WaveNet (2016 paper)
- WaveNet: autoregressive model predicting audio sequences using tree-like hierarchical structure
- Progressive fusion of information rather than single hidden layer approach
Code Refactoring and Cleanup
- Started from part 3 codebase (part 4 was manual backprop exercise)
- Fixed jagged loss visualization by averaging consecutive values
- Used tensor.view() to reshape loss list into 200x1000 rows
- Applied mean() along rows for smoother plotting
- Simplified forward pass by creating embedding and flatten layers
- Implemented Embedding class with vocab_size and n_embed parameters
- Implemented FlattenConsecutive class for tensor reshaping
- Moved all layers into unified structure
- Introduced Sequential container class
- Mimics PyTorch's nn.Sequential functionality
- Passes input through layers sequentially
- Simplified model definition and forward pass
Data Scaling and Architecture Changes
- Increased block_size from 3 to 8 characters for more context
- Dataset now has 8 characters predicting 9th character
- Baseline performance with simple scaling: validation loss improved from 2.10 to 2.02
- Model parameters increased to ~22K due to larger input layer
Hierarchical Architecture Implementation
- Problem: current architecture crushes all characters into single layer too quickly
- Solution: implement tree-like progressive fusion similar to WaveNet
- Process pairs of characters first (bigrams)
- Then process pairs of bigrams (4-grams)
- Continue hierarchically to build deeper network
- Key insight: linear layers in PyTorch support higher-dimensional inputs
- Matrix multiplication works on last dimension
- Earlier dimensions treated as batch dimensions
- Enables parallel processing of consecutive character groups
FlattenConsecutive Layer Enhancement
- Modified to group consecutive elements rather than full flattening
- Takes parameter n for number of consecutive elements to group
- Reshapes 4x8x10 tensor to 4x4x20 (grouping pairs of 10-dim embeddings)
- Handles spurious dimensions with squeeze() operation
- Enables hierarchical processing of character sequences
BatchNorm Bug Fix
- Issue: BatchNorm1D assumed 2D input, but now receiving 3D tensors
- Problem: maintaining separate statistics for each position rather than treating positions as batch dimension
- Solution: modify dim parameter in mean/variance calculations
- 2D input: reduce over dim=0 (batch dimension)
- 3D input: reduce over dim=(0,1) (both batch and sequence dimensions)
- Fixed running_mean shape from 1x4x68 to 1x1x68 (proper channel-wise statistics)
Performance Results and Analysis
- Hierarchical model with bug fix: validation loss 2.022 vs 2.029 (slight improvement)
- Scaled up model (24 embeddings, more hidden units): 76K parameters, validation loss 1.993
- Performance progression logged:
- Block size 3: 2.10
- Block size 8: 2.02
- Hierarchical: 2.022
- Scaled hierarchical: 1.993
Relationship to Convolutional Neural Networks
- Current implementation processes single examples independently
- Convolutions enable efficient sliding of model over input sequences
- Example: processing all 8 positions of "DeAndre" simultaneously
- Convolutions provide:
- Efficient for-loop execution in CUDA kernels
- Variable reuse across overlapping windows
- Same hierarchical structure but with efficiency gains
- WaveNet uses dilated causal convolutions as implementation detail for speed
Development Process Insights
- Heavy reliance on PyTorch documentation (noted as incomplete/unclear)
- Significant time spent on tensor shape gymnastics
- Prototyping workflow: Jupyter notebook → VS Code → experiment execution
- Shape debugging critical for multi-dimensional tensor operations
- Typical deep learning development involves extensive experimentation harness (not shown here)
Future Directions
- Implement dilated causal convolutional layers for efficiency
- Add residual and skip connections from WaveNet paper
- Build proper experimental harness for systematic hyperparameter tuning
- Cover RNNs, LSTMs, GRUs, and Transformers
- Potential improvements: embedding dimensions, channel allocation, initialization, optimization tuning
