AI summary by TwinMind

The matrix math behind transformer neural networks, one step at a time!!!

StatQuest with Josh Starmer

23m 42s

Introduction & Setup

  • Josh Starmer presents complete matrix math behind encoder-decoder Transformer neural networks
  • Example: translating English "let's go" to Spanish "vamos"
  • Assumes prior knowledge of Transformer basics and essential matrix algebra
  • Focus on encoder-decoder architecture (covers all Transformer types)

Word Embeddings & Input Processing

  • Input tokens: SOS (start of sequence), "lets", "go"
  • Word embedding network: each input has 2 connections to activation functions
  • Matrix notation: input matrix (one-hot encoded) × weight matrix
  • First row: SOS token weights, second row: "lets" weights, third row: "go" weights
  • Matrix multiplication produces word embeddings for all input tokens

Positional Encoding

  • Add positional encoding to word embeddings
  • Y-axis coordinates from sine and cosine curves added to each token position
  • Values precomputed and stored in lookup table
  • Element-by-element addition: word embeddings + positional encodings
  • Results in encoded values for input tokens

Self-Attention Calculation (Encoder)

  • Calculate queries (Q), keys (K), values (V) using encoded values
  • Query calculation example: SOS token
    • First encoded value (1.16) × 2.22 + second encoded value (0.23) × 0.17
    • Matrix form: encoded values × query weights matrix
  • Query weights matrix labeled with transpose symbol (PyTorch requirement)
  • Same process for key and value matrices

Attention Score Computation

  • Multiply Q × K^T (transpose of K)
  • Dot product calculation: multiply corresponding numbers, sum results
  • Example: SOS query × SOS key = -4.49 (unscaled similarity)
  • Dot products measure unscaled similarity (related to cosine similarity)
  • Results: unscaled dot product similarities between all query-key combinations

Attention Scaling & Softmax

  • Scale by √(d_k) where d_k = dimension of key matrix (2 in this example)
  • Scaling factor: √2 = 1.414
  • Apply softmax to each row of scaled similarities
  • Softmax ensures each row sums to 1 (percentage relationships)
  • Example: SOS token 1% similar to itself, 99% to "lets", 0% to "go"

Final Attention Calculation

  • Multiply percentage matrix × values matrix (V)
  • Weighted combination of values based on attention percentages
  • Example: 1% × SOS value + 99% × "lets" value + 0% × "go" value = 1.54
  • Results in self-attention scores for each input token
  • Add residual connections (original word + position encodings)
  • Final matrix = encoder output

Decoder Setup & Training

  • Training goal: SOS + "lets" + "go" → "vamos" + EOS
  • Decoder vocabulary: SOS, "vamos", "y" (Spanish "and"), EOS
  • Teacher forcing during training: use known output values to initialize decoders
  • Input tokens: SOS, "vamos"
  • Word embedding matrix: 5 columns (vocabulary size) × 2 rows (tokens)

Decoder Self-Attention with Masking

  • Calculate Q, K, V matrices using decoder-specific weights
  • Compute unscaled dot product similarities
  • Key detail: SOS query × "vamos" key = 0.12 unscaled, 0.08 scaled
  • Apply masking to prevent "cheating" (looking at future tokens)
  • Mask matrix: zeros for allowed values, negative infinity for blocked values
  • After softmax: SOS has 100% self-similarity, 0% to future tokens
  • "Vamos" can use both SOS and itself for attention calculation

Encoder-Decoder Attention

  • Q matrix: created from decoder values
  • K and V matrices: created from encoder output values
  • Same attention calculation process as standard self-attention
  • Results in encoder-decoder attention scores
  • Add residual connections

Fully Connected Layer & Output

  • Fully connected layer: 2 inputs (token values) → 5 outputs (vocabulary size)
  • Matrix multiplication for both tokens simultaneously
  • Add bias terms to each column
  • Apply softmax function to final outputs
  • First decoder output: "vamos" = 1, others = 0
  • Second decoder output: EOS has highest value (translation complete)

Results

  • Successful translation: "let's go" → "vamos" + EOS
  • Matrix operations enable parallel computation during training
  • Sequential processing during inference (no future token access)

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome