AI summary by TwinMind
Let's build GPT: from scratch, in code, spelled out.
Andrej Karpathy
1h 56m 20s
Introduction to ChatGPT and Language Models
- ChatGPT has revolutionized AI community - allows text-based tasks through natural language interaction
- System generates text sequentially (left to right) in probabilistic manner
- Multiple prompts can produce different outputs due to probabilistic nature
- Language model predicts sequences of words/characters/tokens based on learned patterns
- Goal: build ChatGPT-like system from scratch using Transformer architecture
Transformer Architecture Foundation
- Based on "Attention is All You Need" (2017) landmark paper
- GPT = Generatively Pre-trained Transformer
- Transformer architecture became foundation for most AI applications in following 5 years
- Original paper focused on machine translation but architecture proved universally applicable
- Will build character-level language model for educational purposes
Dataset and Tokenization Setup
- Using "Tiny Shakespeare" dataset (1MB, ~1 million characters)
- Character-level tokenization: 65 unique characters in vocabulary
- Alternative tokenization methods:
- Google's SentencePiece (subword level)
- OpenAI's tiktoken (byte-pair encoding, ~50k tokens)
- Word-level tokenization
- Character-level chosen for simplicity despite longer sequences
- Train/validation split: 90%/10%
Data Loading and Batch Processing
- Block size = 8 (context length for predictions)
- Batch size = 4 (independent sequences processed in parallel)
- Each chunk contains multiple training examples (8 examples in 9-character chunk)
- Training on contexts from length 1 to block_size for robustness during inference
- Batched processing for GPU efficiency
Baseline Bigram Language Model
- Simplest possible model: predicts next character based only on current character
- Uses token embedding table (vocab_size × vocab_size)
- No communication between tokens initially
- Cross-entropy loss for training
- Initial loss ~4.87, expected ~4.17 for random uniform distribution
- Added positional embeddings (though not useful for bigram model)
Mathematical Trick for Attention
- Weighted aggregation of past tokens using matrix multiplication
- Lower triangular matrix structure prevents future information leakage
- Three equivalent implementations:
- Explicit for-loop averaging
- Matrix multiplication with normalized weights
- Softmax with masked negative infinity
- Softmax version allows data-dependent affinities between tokens
Self-Attention Implementation
- Each token emits query (what I'm looking for) and key (what I contain)
- Affinities calculated via dot product between queries and keys
- Added value vector (what I communicate if found interesting)
- Scaled attention: divide by sqrt(head_size) to control variance
- Prevents softmax from becoming too sharp at initialization
- Single head implementation with head_size = 32
Multi-Head Attention
- Multiple attention heads running in parallel
- 4 heads × 8 dimensions each = 32 total dimensions
- Each head captures different types of relationships
- Similar to group convolution concept
- Concatenate outputs from all heads
- Validation loss improved from 2.4 to 2.28
Feed-Forward Networks
- Added computation after communication phase
- Simple linear → ReLU → linear structure
- Applied per-token independently (no communication)
- Allows tokens to "think" on aggregated information
- Inner layer 4× larger than embedding dimension (following paper)
- Validation loss improved to 2.24
Residual Connections and Deep Networks
- Skip connections crucial for training deep networks
- Residual pathway: x = x + self_attention(x)
- Gradients flow unimpeded through addition operations
- Blocks initialized to contribute minimally, come online during training
- Added projection layers back to residual pathway
- Enables much deeper architectures
Layer Normalization
- Applied before transformations (pre-norm formulation)
- Normalizes features to unit Gaussian per token
- Different from batch norm (normalizes across batch dimension)
- Layer norm normalizes across feature dimension
- Helps optimization of deep networks
- Added at multiple points in architecture
Scaling Up the Model
- Increased parameters significantly:
- Batch size: 64
- Block size: 256 (context length)
- Embedding dimension: 384
- Number of heads: 6
- Number of layers: 6
- Dropout: 0.2
- Validation loss improved dramatically to 1.48
- Training time: ~15 minutes on A100 GPU
- Generated more coherent Shakespeare-like text
Decoder vs Encoder-Decoder Architecture
- Implemented decoder-only Transformer (like GPT)
- Uses triangular mask for autoregressive generation
- Original paper used encoder-decoder for translation:
- Encoder: processes input sequence (e.g., French)
- Decoder: generates output sequence (e.g., English)
- Cross-attention connects encoder outputs to decoder
- Decoder-only sufficient for language modeling tasks
NanoGPT Codebase Overview
- Production-ready implementation in ~300 lines
- train.py: training loop with checkpointing, distributed training
- model.py: Transformer architecture implementation
- Batched multi-head attention for efficiency
- GELU activation (for OpenAI checkpoint compatibility)
- Can load pre-trained GPT-2 weights
ChatGPT Training Process
- Two-stage process:
- Pre-training: Large-scale language modeling on internet data
- Fine-tuning: Alignment to be helpful assistant
- Pre-training scale comparison:
- Our model: 10M parameters, 300k tokens
- GPT-3: 175B parameters, 300B tokens
- Fine-tuning involves:
- Supervised fine-tuning on Q&A format
- Reward model training from human preferences
- RLHF (Reinforcement Learning from Human Feedback)
- Alignment stage transforms document completer into question answerer
