AI summary by TwinMind

Long Short-Term Memory with PyTorch + Lightning

StatQuest with Josh Starmer

33m 23s

Introduction and Prerequisites

  • Tutorial by Josh Starmer on implementing LSTM with PyTorch Lightning
  • Special thanks to co-worker Adrian Vouchley for help creating tutorial
  • Assumes familiarity with:
    • Long short-term memory concepts
    • Previous StatQuest on coding neural networks with PyTorch Lightning
  • All code downloadable for free (details in pinned comment)

Problem Setup and Data

  • Stock price data from two companies (Company A and Company B)
  • Y-axis: stock value, X-axis: day recorded
  • Key differences between companies:
    • Day 1: Company A = 0, Company B = 1
    • Day 5: Company A = 0, Company B = 1
    • Days 2, 3, 4: both companies have identical values
  • Goal: LSTM remembers day 1 to predict day 5 correctly
  • Sequential data from days 1-4 fed through unrolled LSTM to predict day 5

Library Imports and Setup

  • torch - create tensors for data, weights, biases
  • torch.nn - make weight/bias tensors part of neural network
  • torch.nn.functional - activation functions
  • Adam optimizer - faster convergence than SGD, less stochastic
  • lightning as L - simplify training code
  • TensorDataset and DataLoader from torch.utils.data

Custom LSTM Implementation (LSTM_by_hand class)

Class Structure

  • Inherits from lightning.LightningModule
  • Methods implemented:
    • __init__() - initialize weights and biases
    • lstm_unit() - LSTM mathematics
    • forward() - forward pass through unrolled LSTM
    • configure_optimizers() - setup Adam optimizer
    • training_step() - calculate loss and log progress

Weight Initialization

  • Uses normal distribution (mean=0.0, std=1.0) for random weight initialization
  • Creates tensors for mean and std parameters
  • All weights initialized with torch.normal() using distribution
  • Biases initialized to zero
  • All parameters set with requires_grad=True for optimization

LSTM Unit Mathematics

Stage 1: Long-term Memory Retention

  • Calculate long_remember_percent using sigmoid activation
  • Formula: sigmoid(short_term_memory * weight + input * weight + bias)

Stage 2: Potential Long-term Memory

  • Calculate potential_remember_percent using sigmoid
  • Calculate potential_long_memory using tanh activation
  • Update long-term memory: scaled existing + percentage of potential

Stage 3: Short-term Memory Creation

  • Calculate percentage for short-term memory output
  • Scale tanh of updated long-term memory
  • Return updated long and short-term memories

Forward Method Implementation

  • Input: array of first 4 days stock values
  • Initialize long_memory and short_memory to zero
  • Unroll LSTM through 4 time steps:
    • Day 1: pass day1, long_memory, short_memory to lstm_unit
    • Day 2: pass day2, updated memories to lstm_unit
    • Day 3: pass day3, updated memories to lstm_unit
    • Day 4: pass day4, updated memories to lstm_unit
  • Return final short-term memory as prediction

Training Configuration

  • Adam optimizer configured in configure_optimizers()
  • training_step() method:
    • Calculate loss (sum of squared residuals)
    • Log loss with label "train_loss"
    • Log predictions separately for Company A ("out_0") and Company B ("out_1")
    • Return loss for backpropagation

Training Process and Results

Initial Predictions (Random Weights)

  • Company A prediction: -0.04 (target: 0) - relatively close
  • Company B prediction: -0.04 (target: 1) - horrible

Training Data Setup

  • inputs tensor: 2 arrays of days 1-4 values for both companies
  • labels tensor: [0, 1] for Company A and B respectively
  • Wrapped in TensorDataset and DataLoader

Training Iterations

First Training (2000 epochs)

  • Lightning trainer with max_epochs=2000
  • Results after training:
    • Company A: 0.43 (worse than initial)
    • Company B: improved but not awesome

TensorBoard Analysis

  • Loss graph: decreasing throughout training, not flattened
  • Company A predictions: got worse initially, then improving
  • Company B predictions: approaching target value 1
  • All graphs suggest more training needed

Additional Training Sessions

  • Added 1000 more epochs (total 3000):
    • Company A: closer to 0
    • Company B: closer to 1
  • Added 2000 more epochs (total 5000):
    • Company A: very close to 0
    • Company B: very close to 1
    • All TensorBoard curves flattened - training complete

Lightning Training Features

  • Automatic checkpoint saving
  • Resume training from checkpoints using ckpt_path parameter
  • Easy to add epochs without starting over

Built-in PyTorch LSTM Implementation

LightningLSTM Class

  • Much simpler implementation using nn.LSTM()
  • Parameters:
    • input_size=1 - single feature (stock value)
    • hidden_size=1 - single output value
  • Note: hidden_size can be larger to connect to other networks

Forward Method Simplification

  • Transpose input from row to column using view(len(input), 1)
  • Pass to LSTM: lstm_out = self.lstm(input)
  • Extract final prediction: lstm_out[-1] (last element)
  • Return prediction

Training Configuration

  • Learning rate increased to 0.1 for faster convergence
  • Only 300 epochs needed (vs 5000 for handmade)
  • Log interval set to 2 steps for better graph visualization

Results Comparison

  • After 300 epochs:
    • Company A: super close to 0
    • Company B: close to 1
  • TensorBoard shows stabilized results (flattened curves)
  • Demonstrates efficiency of optimized implementation

TensorBoard Usage

  • Launch from terminal: tensorboard --logdir=lightning_logs
  • Copy URL to browser for visualization
  • Graphs show:
    • Training loss over time
    • Predictions for each company
    • Progress indicators for training completion

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome