AI summary by TwinMind
Long Short-Term Memory (LSTM), Clearly Explained
StatQuest with Josh Starmer
20m 45s
Introduction and Prerequisites
- StatQuest episode on Long Short-Term Memory (LSTM) networks
- Host: Josh Starmer
- Prerequisites: familiarity with recurrent neural networks and vanishing/exploding gradient problem
- LSTM as stepping stone to understanding Transformers
Vanishing/Exploding Gradient Problem Review
- Basic vanilla RNNs suffer from gradient issues with sequential data
- Weight > 1 (e.g., 2): causes exploding gradients
- 50 sequential data points → 2^50 = huge number → gradient explodes
- Weight < 1 (e.g., 0.5): causes vanishing gradients
- 50 sequential data points → 0.5^50 ≈ 0 → gradient vanishes
- Makes basic RNNs hard to train
LSTM Core Concept
- Uses two separate paths for predictions:
- One path for long-term memories
- One path for short-term memories
- Avoids exploding/vanishing gradient problem
- More complex unit compared to vanilla RNN
Activation Functions in LSTM
- Sigmoid function: converts any input to value between 0 and 1
- Example: input 10 → output 0.999995
- Example: input -5 → output 0.01
- TanH function: converts any input to value between -1 and 1
- Example: input 2 → output 0.96
- Example: input -5 → output -1
LSTM Unit Architecture
- Cell State (green line): represents long-term memory
- Flows across top of unit
- No direct weights/biases modifying it
- Allows long-term memories to flow without gradient issues
- Hidden State (pink line): represents short-term memories
- Directly connected to modifiable weights
LSTM Gates and Mathematical Operations
Stage 1: Forget Gate
- Determines what percentage of long-term memory is remembered
- Mathematical example:
- Previous long-term memory: 2
- Previous short-term memory: 1
- Input: 1
- Calculation: (1 × 2.7) + (1 × 1.63) + 1.62 = 5.95
- Sigmoid(5.95) = 0.997
- New long-term memory: 2 × 0.997 = 1.99
- Sigmoid output determines retention percentage (0-100%)
Stage 2: Input Gate
- Right block: creates potential long-term memory using TanH
- Calculation: (1 × weight) + (1 × weight) + bias = 2.03
- TanH(2.03) = 0.97 (potential memory)
- Left block: determines percentage of potential memory to add
- Sigmoid activation determines retention percentage
- Example result: 1.0 (100% retention)
- Final: 1.99 + 0.97 = 2.96 (new long-term memory)
Stage 3: Output Gate
- Updates short-term memory
- Uses new long-term memory as input to TanH: TanH(2.96) = 0.99
- Sigmoid determines percentage to pass on: 0.99
- New short-term memory: 0.99 × 0.99 = 0.98
- This becomes the unit's output
Real Data Example: Stock Price Prediction
- Company A data: Day 1=0, Day 2=0.5, Day 3=1, Day 4=0.5, Day 5=0
- Company B data: Day 1=1, Day 2=0.5, Day 3=1, Day 4=0.5, Day 5=1
- Only differences on Day 1 and Day 5
- LSTM must remember Day 1 to predict Day 5
Company A Processing
- Initialize memories to 0
- Day 1 input (0) → long-term: -0.20, short-term: -0.13
- Sequential processing through Days 2-4 with same weights/biases
- Final output: 0.0 (correctly predicts Company A Day 5 value)
Company B Processing
- Initialize memories to 0
- Day 1 input (1) → long-term: 0.5, short-term: 0.28
- Sequential processing through Days 2-4 with same weights/biases
- Final output: 1.0 (correctly predicts Company B Day 5 value)
Key Advantages
- Separate paths prevent gradient explosion/vanishing
- Can handle longer sequences than vanilla RNNs
- Same weights/biases handle variable sequence lengths
- Successfully demonstrates long-term memory retention
