AI summary by TwinMind
The matrix math behind transformer neural networks, one step at a time!!!
StatQuest with Josh Starmer
23m 42s
Introduction & Setup
- Josh Starmer presents complete matrix math behind encoder-decoder Transformer neural networks
- Example: translating English "let's go" to Spanish "vamos"
- Assumes prior knowledge of Transformer basics and essential matrix algebra
- Focus on encoder-decoder architecture (covers all Transformer types)
Word Embeddings & Input Processing
- Input tokens: SOS (start of sequence), "lets", "go"
- Word embedding network: each input has 2 connections to activation functions
- Matrix notation: input matrix (one-hot encoded) × weight matrix
- First row: SOS token weights, second row: "lets" weights, third row: "go" weights
- Matrix multiplication produces word embeddings for all input tokens
Positional Encoding
- Add positional encoding to word embeddings
- Y-axis coordinates from sine and cosine curves added to each token position
- Values precomputed and stored in lookup table
- Element-by-element addition: word embeddings + positional encodings
- Results in encoded values for input tokens
Self-Attention Calculation (Encoder)
- Calculate queries (Q), keys (K), values (V) using encoded values
- Query calculation example: SOS token
- First encoded value (1.16) × 2.22 + second encoded value (0.23) × 0.17
- Matrix form: encoded values × query weights matrix
- Query weights matrix labeled with transpose symbol (PyTorch requirement)
- Same process for key and value matrices
Attention Score Computation
- Multiply Q × K^T (transpose of K)
- Dot product calculation: multiply corresponding numbers, sum results
- Example: SOS query × SOS key = -4.49 (unscaled similarity)
- Dot products measure unscaled similarity (related to cosine similarity)
- Results: unscaled dot product similarities between all query-key combinations
Attention Scaling & Softmax
- Scale by √(d_k) where d_k = dimension of key matrix (2 in this example)
- Scaling factor: √2 = 1.414
- Apply softmax to each row of scaled similarities
- Softmax ensures each row sums to 1 (percentage relationships)
- Example: SOS token 1% similar to itself, 99% to "lets", 0% to "go"
Final Attention Calculation
- Multiply percentage matrix × values matrix (V)
- Weighted combination of values based on attention percentages
- Example: 1% × SOS value + 99% × "lets" value + 0% × "go" value = 1.54
- Results in self-attention scores for each input token
- Add residual connections (original word + position encodings)
- Final matrix = encoder output
Decoder Setup & Training
- Training goal: SOS + "lets" + "go" → "vamos" + EOS
- Decoder vocabulary: SOS, "vamos", "y" (Spanish "and"), EOS
- Teacher forcing during training: use known output values to initialize decoders
- Input tokens: SOS, "vamos"
- Word embedding matrix: 5 columns (vocabulary size) × 2 rows (tokens)
Decoder Self-Attention with Masking
- Calculate Q, K, V matrices using decoder-specific weights
- Compute unscaled dot product similarities
- Key detail: SOS query × "vamos" key = 0.12 unscaled, 0.08 scaled
- Apply masking to prevent "cheating" (looking at future tokens)
- Mask matrix: zeros for allowed values, negative infinity for blocked values
- After softmax: SOS has 100% self-similarity, 0% to future tokens
- "Vamos" can use both SOS and itself for attention calculation
Encoder-Decoder Attention
- Q matrix: created from decoder values
- K and V matrices: created from encoder output values
- Same attention calculation process as standard self-attention
- Results in encoder-decoder attention scores
- Add residual connections
Fully Connected Layer & Output
- Fully connected layer: 2 inputs (token values) → 5 outputs (vocabulary size)
- Matrix multiplication for both tokens simultaneously
- Add bias terms to each column
- Apply softmax function to final outputs
- First decoder output: "vamos" = 1, others = 0
- Second decoder output: EOS has highest value (translation complete)
Results
- Successful translation: "let's go" → "vamos" + EOS
- Matrix operations enable parallel computation during training
- Sequential processing during inference (no future token access)
