AI summary by TwinMind

Building makemore Part 5: Building a WaveNet

Andrej Karpathy

56m 21s

Introduction and Setup

  • Continuing implementation of "makemore" character-level language model from Kyoto hotel room
  • Previous architecture: multi-layer perceptron with 3 previous characters predicting 4th character
  • Goal: complexify architecture to take more characters as input and avoid crushing information too quickly
  • Target: implement hierarchical structure similar to WaveNet (2016 paper)
    • WaveNet: autoregressive model predicting audio sequences using tree-like hierarchical structure
    • Progressive fusion of information rather than single hidden layer approach

Code Refactoring and Cleanup

  • Started from part 3 codebase (part 4 was manual backprop exercise)
  • Fixed jagged loss visualization by averaging consecutive values
    • Used tensor.view() to reshape loss list into 200x1000 rows
    • Applied mean() along rows for smoother plotting
  • Simplified forward pass by creating embedding and flatten layers
    • Implemented Embedding class with vocab_size and n_embed parameters
    • Implemented FlattenConsecutive class for tensor reshaping
    • Moved all layers into unified structure
  • Introduced Sequential container class
    • Mimics PyTorch's nn.Sequential functionality
    • Passes input through layers sequentially
    • Simplified model definition and forward pass

Data Scaling and Architecture Changes

  • Increased block_size from 3 to 8 characters for more context
  • Dataset now has 8 characters predicting 9th character
  • Baseline performance with simple scaling: validation loss improved from 2.10 to 2.02
  • Model parameters increased to ~22K due to larger input layer

Hierarchical Architecture Implementation

  • Problem: current architecture crushes all characters into single layer too quickly
  • Solution: implement tree-like progressive fusion similar to WaveNet
    • Process pairs of characters first (bigrams)
    • Then process pairs of bigrams (4-grams)
    • Continue hierarchically to build deeper network
  • Key insight: linear layers in PyTorch support higher-dimensional inputs
    • Matrix multiplication works on last dimension
    • Earlier dimensions treated as batch dimensions
    • Enables parallel processing of consecutive character groups

FlattenConsecutive Layer Enhancement

  • Modified to group consecutive elements rather than full flattening
  • Takes parameter n for number of consecutive elements to group
  • Reshapes 4x8x10 tensor to 4x4x20 (grouping pairs of 10-dim embeddings)
  • Handles spurious dimensions with squeeze() operation
  • Enables hierarchical processing of character sequences

BatchNorm Bug Fix

  • Issue: BatchNorm1D assumed 2D input, but now receiving 3D tensors
  • Problem: maintaining separate statistics for each position rather than treating positions as batch dimension
  • Solution: modify dim parameter in mean/variance calculations
    • 2D input: reduce over dim=0 (batch dimension)
    • 3D input: reduce over dim=(0,1) (both batch and sequence dimensions)
  • Fixed running_mean shape from 1x4x68 to 1x1x68 (proper channel-wise statistics)

Performance Results and Analysis

  • Hierarchical model with bug fix: validation loss 2.022 vs 2.029 (slight improvement)
  • Scaled up model (24 embeddings, more hidden units): 76K parameters, validation loss 1.993
  • Performance progression logged:
    • Block size 3: 2.10
    • Block size 8: 2.02
    • Hierarchical: 2.022
    • Scaled hierarchical: 1.993

Relationship to Convolutional Neural Networks

  • Current implementation processes single examples independently
  • Convolutions enable efficient sliding of model over input sequences
  • Example: processing all 8 positions of "DeAndre" simultaneously
  • Convolutions provide:
    • Efficient for-loop execution in CUDA kernels
    • Variable reuse across overlapping windows
    • Same hierarchical structure but with efficiency gains
  • WaveNet uses dilated causal convolutions as implementation detail for speed

Development Process Insights

  • Heavy reliance on PyTorch documentation (noted as incomplete/unclear)
  • Significant time spent on tensor shape gymnastics
  • Prototyping workflow: Jupyter notebook → VS Code → experiment execution
  • Shape debugging critical for multi-dimensional tensor operations
  • Typical deep learning development involves extensive experimentation harness (not shown here)

Future Directions

  • Implement dilated causal convolutional layers for efficiency
  • Add residual and skip connections from WaveNet paper
  • Build proper experimental harness for systematic hyperparameter tuning
  • Cover RNNs, LSTMs, GRUs, and Transformers
  • Potential improvements: embedding dimensions, channel allocation, initialization, optimization tuning

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome