AI summary by TwinMind
Decoder-Only Transformers, ChatGPTs specific Transformer, Clearly Explained!!!
StatQuest with Josh Starmer
36m 44s
Introduction and Overview
- Josh Starmer introduces decoder-only Transformers used in ChatGPT
- Example: StatQuest types "what is stat Quest" → ChatGPT responds "stat Quest is awesome"
- Decoder-only Transformers are specific type used for chat applications
- Will explain each concept step-by-step, including for those unfamiliar with basic Transformers
Word Embedding - Converting Words to Numbers
- Neural networks require numerical inputs, not words
- Word embedding uses simple neural network with one input per vocabulary word/symbol
- Simple vocabulary includes: "what", "is", "stat", "Quest", "awesome", "EOS" (end of sequence)
- Each input called a "token" (mix of words, word fragments, symbols)
- Inputs connected to activation functions via weighted connections
- Example: "what" → input value 1, others 0
- Multiplies weights: -2.3 × 1 = -2.38, 0.10 × 1 = 0.10
- Other words send zeros since input values are 0
- Activation functions are identity functions (output = input)
- Word "what" represented by values: -2.38, 0.10
Weight Optimization and Training
- Weights determined using back propagation
- Similar to fitting line to data: starts with random values, iteratively optimizes
- Neural network weights start random, optimized through training on known documents
- Training = process of optimizing weights through back propagation
- Reuses same word embedding network for all words in prompt
- Allows handling different prompt lengths by copying network as needed
Positional Encoding - Tracking Word Order
- Word order crucial: "Squatch eats pizza" vs "Pizza eats Squatch" have different meanings
- Positional encoding uses alternating sine and cosine functions
- Each position gets specific values from squiggles
- First word "what": position values 0, 1 from y-axis coordinates
- Second word "is": different x-axis position gives different y-coordinates
- Repetitive patterns but wider squiggles for larger positions create unique sequences
- Final result: word embeddings + positional encoding for each word
Masked Self-Attention - Word Relationships
- Tracks relationships among words in sentences
- Example: "the pizza came out of the oven and it tasted good"
- Word "it" could refer to "pizza" or "oven"
- Masked self-attention helps associate "it" with "pizza"
- Works by calculating similarity between each word and all preceding words
- Cannot look ahead to future words (hence "masked")
- Also called "auto-regressive method"
Masked Self-Attention Details
- Uses three types of numbers: queries, keys, values
- For word "is":
- Create query numbers: -2.4, 2.6
- Create key numbers for "is" and "what"
- Calculate dot product similarities
- Query "is" with key "is": 5.9
- Query "is" with key "what": -25.7
- Similarity scores run through softmax function
- Converts to percentages (0-1, sum to 1)
- High similarity "is" to itself: 100% weight
- Low similarity "is" to "what": 0% weight
- Create value numbers, scale by softmax percentages
- Add scaled values together = masked self-attention output
Reused Weight Sets
- Same query weights reused for all words
- Same key weights reused for all words
- Same value weights reused for all words
- Enables handling different prompt lengths
- Original GPT used 12 stacked masked self-attention cells
- Each cell captures different word relationships
Residual Connections
- Add position-encoded values to self-attention values
- Called "residual connections" or "bypasses"
- Makes training complex networks easier
- Allows attention layer to focus on relationships without preserving word/position info
Output Generation - Fully Connected Layer
- Need to generate word following each input word
- Takes masked self-attention values through fully connected layer
- Example: values -5.28, 0.2 for "what" → 5 outputs (vocabulary size)
- Outputs run through final softmax function
- Original GPT reused word embedding weights in reverse
- Alternative: separate fully connected layer
Training vs Inference Differences
- During training: compare generated words to known correct words
- Mistakes used to modify weights/biases via back propagation
- During inference: generated words become actual output
- Model generates until EOS token produced
Complete Process Review
- Word embedding converts words to numbers
- Positional encoding tracks word order
- Masked self-attention determines word relationships
- Residual connections aid training
- Fully connected layer + softmax generates next word
- All components reused for different prompt lengths
- Encoding can happen in parallel (fast with multiple cores)
Output Generation Process
- Uses exact same steps as prompt encoding
- Starts with EOS token to initialize generation
- Alternative: some use SOS (start of sequence) token
- EOS token gets position encoding for 4th position
- Masked self-attention includes entire prompt history
- Generates "awesome" as first output word
- Continues until EOS token generated, then stops
Decoder-Only vs Regular Transformers
Decoder-Only Transformer:
- Single unit for encoding input AND generating output
- Uses masked self-attention exclusively
- Masked self-attention applied to input AND output
- Uses masked self-attention during training AND inference
- Includes input + output in attention calculations
Regular Transformer:
- Separate encoder unit for input, decoder unit for output
- Encoder uses full self-attention (sees all input words)
- Uses encoder-decoder attention (queries from decoder, keys/values from encoder)
- During training: uses masked self-attention only on output tokens
- During inference: uses self-attention + encoder-decoder attention
Three Key Differences Summary
- Architecture: Normal has separate encoder/decoder units; decoder-only has single unit
- Attention types: Normal uses self-attention + encoder-decoder attention; decoder-only uses only masked self-attention
- Training: Normal uses masked self-attention only on output during training; decoder-only uses masked self-attention on everything always
