AI summary by TwinMind

Let's build GPT: from scratch, in code, spelled out.

Andrej Karpathy

1h 56m 20s

Introduction to ChatGPT and Language Models

  • ChatGPT has revolutionized AI community - allows text-based tasks through natural language interaction
  • System generates text sequentially (left to right) in probabilistic manner
  • Multiple prompts can produce different outputs due to probabilistic nature
  • Language model predicts sequences of words/characters/tokens based on learned patterns
  • Goal: build ChatGPT-like system from scratch using Transformer architecture

Transformer Architecture Foundation

  • Based on "Attention is All You Need" (2017) landmark paper
  • GPT = Generatively Pre-trained Transformer
  • Transformer architecture became foundation for most AI applications in following 5 years
  • Original paper focused on machine translation but architecture proved universally applicable
  • Will build character-level language model for educational purposes

Dataset and Tokenization Setup

  • Using "Tiny Shakespeare" dataset (1MB, ~1 million characters)
  • Character-level tokenization: 65 unique characters in vocabulary
  • Alternative tokenization methods:
    • Google's SentencePiece (subword level)
    • OpenAI's tiktoken (byte-pair encoding, ~50k tokens)
    • Word-level tokenization
  • Character-level chosen for simplicity despite longer sequences
  • Train/validation split: 90%/10%

Data Loading and Batch Processing

  • Block size = 8 (context length for predictions)
  • Batch size = 4 (independent sequences processed in parallel)
  • Each chunk contains multiple training examples (8 examples in 9-character chunk)
  • Training on contexts from length 1 to block_size for robustness during inference
  • Batched processing for GPU efficiency

Baseline Bigram Language Model

  • Simplest possible model: predicts next character based only on current character
  • Uses token embedding table (vocab_size × vocab_size)
  • No communication between tokens initially
  • Cross-entropy loss for training
  • Initial loss ~4.87, expected ~4.17 for random uniform distribution
  • Added positional embeddings (though not useful for bigram model)

Mathematical Trick for Attention

  • Weighted aggregation of past tokens using matrix multiplication
  • Lower triangular matrix structure prevents future information leakage
  • Three equivalent implementations:
    1. Explicit for-loop averaging
    2. Matrix multiplication with normalized weights
    3. Softmax with masked negative infinity
  • Softmax version allows data-dependent affinities between tokens

Self-Attention Implementation

  • Each token emits query (what I'm looking for) and key (what I contain)
  • Affinities calculated via dot product between queries and keys
  • Added value vector (what I communicate if found interesting)
  • Scaled attention: divide by sqrt(head_size) to control variance
  • Prevents softmax from becoming too sharp at initialization
  • Single head implementation with head_size = 32

Multi-Head Attention

  • Multiple attention heads running in parallel
  • 4 heads × 8 dimensions each = 32 total dimensions
  • Each head captures different types of relationships
  • Similar to group convolution concept
  • Concatenate outputs from all heads
  • Validation loss improved from 2.4 to 2.28

Feed-Forward Networks

  • Added computation after communication phase
  • Simple linear → ReLU → linear structure
  • Applied per-token independently (no communication)
  • Allows tokens to "think" on aggregated information
  • Inner layer 4× larger than embedding dimension (following paper)
  • Validation loss improved to 2.24

Residual Connections and Deep Networks

  • Skip connections crucial for training deep networks
  • Residual pathway: x = x + self_attention(x)
  • Gradients flow unimpeded through addition operations
  • Blocks initialized to contribute minimally, come online during training
  • Added projection layers back to residual pathway
  • Enables much deeper architectures

Layer Normalization

  • Applied before transformations (pre-norm formulation)
  • Normalizes features to unit Gaussian per token
  • Different from batch norm (normalizes across batch dimension)
  • Layer norm normalizes across feature dimension
  • Helps optimization of deep networks
  • Added at multiple points in architecture

Scaling Up the Model

  • Increased parameters significantly:
    • Batch size: 64
    • Block size: 256 (context length)
    • Embedding dimension: 384
    • Number of heads: 6
    • Number of layers: 6
    • Dropout: 0.2
  • Validation loss improved dramatically to 1.48
  • Training time: ~15 minutes on A100 GPU
  • Generated more coherent Shakespeare-like text

Decoder vs Encoder-Decoder Architecture

  • Implemented decoder-only Transformer (like GPT)
  • Uses triangular mask for autoregressive generation
  • Original paper used encoder-decoder for translation:
    • Encoder: processes input sequence (e.g., French)
    • Decoder: generates output sequence (e.g., English)
    • Cross-attention connects encoder outputs to decoder
  • Decoder-only sufficient for language modeling tasks

NanoGPT Codebase Overview

  • Production-ready implementation in ~300 lines
  • train.py: training loop with checkpointing, distributed training
  • model.py: Transformer architecture implementation
  • Batched multi-head attention for efficiency
  • GELU activation (for OpenAI checkpoint compatibility)
  • Can load pre-trained GPT-2 weights

ChatGPT Training Process

  • Two-stage process:
    1. Pre-training: Large-scale language modeling on internet data
    2. Fine-tuning: Alignment to be helpful assistant
  • Pre-training scale comparison:
    • Our model: 10M parameters, 300k tokens
    • GPT-3: 175B parameters, 300B tokens
  • Fine-tuning involves:
    • Supervised fine-tuning on Q&A format
    • Reward model training from human preferences
    • RLHF (Reinforcement Learning from Human Feedback)
  • Alignment stage transforms document completer into question answerer

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome