AI summary by TwinMind

Coding a ChatGPT Like Transformer From Scratch in PyTorch

StatQuest with Josh Starmer

31m 11s

Introduction and Setup

  • Tutorial by Josh Starmer on StatQuest
  • Building decoder-only Transformer (foundation for ChatGPT) from scratch in PyTorch
  • Code available for free via pinned comment
  • Charity mention: give internet.org for underprivileged students

Library Imports

  • torch: tensors and helper functions
  • torch.nn: module, linear, embedding classes
  • torch.nn.functional: softmax function for attention
  • optim: neural network fitting with backpropagation
  • TensorDataset, DataLoader from torch.utils.data: large-scale training
  • lightning as L: easier coding, auto optimization, cloud scaling

Training Dataset Creation

  • Simple example with two prompts:
    • "what is stat quest" → "awesome"
    • "stat quest is what" → "awesome"
  • Vocabulary tokens: what, is, stat, quest, awesome, EOS
  • Token-to-ID mapping: dictionary for PyTorch embedding function
  • ID-to-token mapping: reverse dictionary for output interpretation

Input/Output Formatting

  • Example: "what is stat quest" + "awesome"
  • Input sequence: what → is → stat → quest → EOS → awesome
  • Output sequence: is → stat → quest → EOS → awesome → EOS
  • Each token generates next token in sequence
  • Training data created using TensorDataset and DataLoader

Word Embedding

  • Using nn.embedding for token-to-vector conversion
  • Handles conversion automatically

Positional Encoding

  • Uses alternating sine/cosine functions
  • Equations:
    • sin(pos/10000^(2i/d_model))
    • cos(pos/10000^(2i/d_model))
  • pos: token position, i: embedding dimension index, d_model: embedding size
  • Example with 2 tokens, 4 embeddings (d_model=4)
  • Pre-computed matrix for efficiency

Positional Encoding Implementation

  • Class PositionalEncoding inherits from nn.Module
  • Parameters: d_model=2, max_len=6 (for simple example)
  • Creates PE matrix filled with zeros initially
  • position: column matrix of token positions
  • embedding_index: row matrix of embedding indices (step=2)
  • div_term: divisor for scaling
  • PE matrix filled with sin/cos values
  • register_buffer: ensures GPU compatibility

Masked Self-Attention

  • Query, Key, Value calculation using matrix multiplication
  • Weight matrices for Q, K, V transformations
  • Attention class inherits from nn.Module
  • Parameters: d_model (embedding dimension)
  • WQ, WK, WV: linear layers for query/key/value calculation (bias=False)
  • Flexibility for encoder-decoder attention

Attention Forward Method

  • Calculates Q, K, V from input encodings
  • Similarity calculation: Q × K^T
  • Scaling by sqrt(key_dimension)
  • Mask application using mask_fill (-1e9 for masked positions)
  • Softmax for attention percentages
  • Final attention: attention_percentages × V

Decoder-Only Transformer Class

  • DecoderOnlyTransformer inherits from lightning.Module
  • Parameters: num_tokens, d_model, max_len
  • Components:
    • we: word embedding (nn.Embedding)
    • pe: positional encoding
    • attention: attention mechanism
    • fc: fully connected layer (nn.Linear)
    • loss_fn: CrossEntropyLoss

Forward Method Implementation

  • Token embedding → positional encoding
  • Mask creation using torch.ones and torch.tril
  • Lower triangle mask (ones below diagonal, zeros above)
  • Mask conversion: zeros→True, ones→False
  • Attention calculation with mask
  • Residual connections
  • Fully connected layer output

Training Configuration

  • configure_optimizers method using Adam optimizer
  • Learning rate: 0.1 (faster for simple model, typically 0.001)
  • training_step method: splits batch into inputs/labels
  • Forward pass → loss calculation → return loss

Model Testing and Training

  • Pre-training test: "what is stat quest EOS" → "EOS" (incorrect)
  • Lightning trainer: 30 epochs
  • Post-training results:
    • "what is stat quest EOS" → "awesome EOS" ✓
    • "stat quest is what EOS" → "awesome EOS" ✓
  • Model works as expected

Self-Promotion

  • StatQuest PDF study guides
  • Book: "StatQuest Illustrated Guide to Machine Learning"
  • Available at statquest.org
  • Support options: subscribe, Patreon, channel membership, merchandise

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome