AI summary by TwinMind
Long Short-Term Memory with PyTorch + Lightning
StatQuest with Josh Starmer
33m 23s
Introduction and Prerequisites
- Tutorial by Josh Starmer on implementing LSTM with PyTorch Lightning
- Special thanks to co-worker Adrian Vouchley for help creating tutorial
- Assumes familiarity with:
- Long short-term memory concepts
- Previous StatQuest on coding neural networks with PyTorch Lightning
- All code downloadable for free (details in pinned comment)
Problem Setup and Data
- Stock price data from two companies (Company A and Company B)
- Y-axis: stock value, X-axis: day recorded
- Key differences between companies:
- Day 1: Company A = 0, Company B = 1
- Day 5: Company A = 0, Company B = 1
- Days 2, 3, 4: both companies have identical values
- Goal: LSTM remembers day 1 to predict day 5 correctly
- Sequential data from days 1-4 fed through unrolled LSTM to predict day 5
Library Imports and Setup
torch- create tensors for data, weights, biasestorch.nn- make weight/bias tensors part of neural networktorch.nn.functional- activation functionsAdamoptimizer - faster convergence than SGD, less stochasticlightning as L- simplify training codeTensorDatasetandDataLoaderfromtorch.utils.data
Custom LSTM Implementation (LSTM_by_hand class)
Class Structure
- Inherits from
lightning.LightningModule - Methods implemented:
__init__()- initialize weights and biaseslstm_unit()- LSTM mathematicsforward()- forward pass through unrolled LSTMconfigure_optimizers()- setup Adam optimizertraining_step()- calculate loss and log progress
Weight Initialization
- Uses normal distribution (mean=0.0, std=1.0) for random weight initialization
- Creates tensors for
meanandstdparameters - All weights initialized with
torch.normal()using distribution - Biases initialized to zero
- All parameters set with
requires_grad=Truefor optimization
LSTM Unit Mathematics
Stage 1: Long-term Memory Retention
- Calculate
long_remember_percentusing sigmoid activation - Formula: sigmoid(short_term_memory * weight + input * weight + bias)
Stage 2: Potential Long-term Memory
- Calculate
potential_remember_percentusing sigmoid - Calculate
potential_long_memoryusing tanh activation - Update long-term memory: scaled existing + percentage of potential
Stage 3: Short-term Memory Creation
- Calculate percentage for short-term memory output
- Scale tanh of updated long-term memory
- Return updated long and short-term memories
Forward Method Implementation
- Input: array of first 4 days stock values
- Initialize long_memory and short_memory to zero
- Unroll LSTM through 4 time steps:
- Day 1: pass day1, long_memory, short_memory to lstm_unit
- Day 2: pass day2, updated memories to lstm_unit
- Day 3: pass day3, updated memories to lstm_unit
- Day 4: pass day4, updated memories to lstm_unit
- Return final short-term memory as prediction
Training Configuration
- Adam optimizer configured in
configure_optimizers() training_step()method:- Calculate loss (sum of squared residuals)
- Log loss with label "train_loss"
- Log predictions separately for Company A ("out_0") and Company B ("out_1")
- Return loss for backpropagation
Training Process and Results
Initial Predictions (Random Weights)
- Company A prediction: -0.04 (target: 0) - relatively close
- Company B prediction: -0.04 (target: 1) - horrible
Training Data Setup
inputstensor: 2 arrays of days 1-4 values for both companieslabelstensor: [0, 1] for Company A and B respectively- Wrapped in
TensorDatasetandDataLoader
Training Iterations
First Training (2000 epochs)
- Lightning trainer with max_epochs=2000
- Results after training:
- Company A: 0.43 (worse than initial)
- Company B: improved but not awesome
TensorBoard Analysis
- Loss graph: decreasing throughout training, not flattened
- Company A predictions: got worse initially, then improving
- Company B predictions: approaching target value 1
- All graphs suggest more training needed
Additional Training Sessions
- Added 1000 more epochs (total 3000):
- Company A: closer to 0
- Company B: closer to 1
- Added 2000 more epochs (total 5000):
- Company A: very close to 0
- Company B: very close to 1
- All TensorBoard curves flattened - training complete
Lightning Training Features
- Automatic checkpoint saving
- Resume training from checkpoints using
ckpt_pathparameter - Easy to add epochs without starting over
Built-in PyTorch LSTM Implementation
LightningLSTM Class
- Much simpler implementation using
nn.LSTM() - Parameters:
input_size=1- single feature (stock value)hidden_size=1- single output value
- Note:
hidden_sizecan be larger to connect to other networks
Forward Method Simplification
- Transpose input from row to column using
view(len(input), 1) - Pass to LSTM:
lstm_out = self.lstm(input) - Extract final prediction:
lstm_out[-1](last element) - Return prediction
Training Configuration
- Learning rate increased to 0.1 for faster convergence
- Only 300 epochs needed (vs 5000 for handmade)
- Log interval set to 2 steps for better graph visualization
Results Comparison
- After 300 epochs:
- Company A: super close to 0
- Company B: close to 1
- TensorBoard shows stabilized results (flattened curves)
- Demonstrates efficiency of optimized implementation
TensorBoard Usage
- Launch from terminal:
tensorboard --logdir=lightning_logs - Copy URL to browser for visualization
- Graphs show:
- Training loss over time
- Predictions for each company
- Progress indicators for training completion
