AI summary by TwinMind

Long Short-Term Memory (LSTM), Clearly Explained

StatQuest with Josh Starmer

20m 45s

Introduction and Prerequisites

  • StatQuest episode on Long Short-Term Memory (LSTM) networks
  • Host: Josh Starmer
  • Prerequisites: familiarity with recurrent neural networks and vanishing/exploding gradient problem
  • LSTM as stepping stone to understanding Transformers

Vanishing/Exploding Gradient Problem Review

  • Basic vanilla RNNs suffer from gradient issues with sequential data
  • Weight > 1 (e.g., 2): causes exploding gradients
    • 50 sequential data points → 2^50 = huge number → gradient explodes
  • Weight < 1 (e.g., 0.5): causes vanishing gradients
    • 50 sequential data points → 0.5^50 ≈ 0 → gradient vanishes
  • Makes basic RNNs hard to train

LSTM Core Concept

  • Uses two separate paths for predictions:
    • One path for long-term memories
    • One path for short-term memories
  • Avoids exploding/vanishing gradient problem
  • More complex unit compared to vanilla RNN

Activation Functions in LSTM

  • Sigmoid function: converts any input to value between 0 and 1
    • Example: input 10 → output 0.999995
    • Example: input -5 → output 0.01
  • TanH function: converts any input to value between -1 and 1
    • Example: input 2 → output 0.96
    • Example: input -5 → output -1

LSTM Unit Architecture

  • Cell State (green line): represents long-term memory
    • Flows across top of unit
    • No direct weights/biases modifying it
    • Allows long-term memories to flow without gradient issues
  • Hidden State (pink line): represents short-term memories
    • Directly connected to modifiable weights

LSTM Gates and Mathematical Operations

Stage 1: Forget Gate

  • Determines what percentage of long-term memory is remembered
  • Mathematical example:
    • Previous long-term memory: 2
    • Previous short-term memory: 1
    • Input: 1
    • Calculation: (1 × 2.7) + (1 × 1.63) + 1.62 = 5.95
    • Sigmoid(5.95) = 0.997
    • New long-term memory: 2 × 0.997 = 1.99
  • Sigmoid output determines retention percentage (0-100%)

Stage 2: Input Gate

  • Right block: creates potential long-term memory using TanH
    • Calculation: (1 × weight) + (1 × weight) + bias = 2.03
    • TanH(2.03) = 0.97 (potential memory)
  • Left block: determines percentage of potential memory to add
    • Sigmoid activation determines retention percentage
    • Example result: 1.0 (100% retention)
  • Final: 1.99 + 0.97 = 2.96 (new long-term memory)

Stage 3: Output Gate

  • Updates short-term memory
  • Uses new long-term memory as input to TanH: TanH(2.96) = 0.99
  • Sigmoid determines percentage to pass on: 0.99
  • New short-term memory: 0.99 × 0.99 = 0.98
  • This becomes the unit's output

Real Data Example: Stock Price Prediction

  • Company A data: Day 1=0, Day 2=0.5, Day 3=1, Day 4=0.5, Day 5=0
  • Company B data: Day 1=1, Day 2=0.5, Day 3=1, Day 4=0.5, Day 5=1
  • Only differences on Day 1 and Day 5
  • LSTM must remember Day 1 to predict Day 5

Company A Processing

  • Initialize memories to 0
  • Day 1 input (0) → long-term: -0.20, short-term: -0.13
  • Sequential processing through Days 2-4 with same weights/biases
  • Final output: 0.0 (correctly predicts Company A Day 5 value)

Company B Processing

  • Initialize memories to 0
  • Day 1 input (1) → long-term: 0.5, short-term: 0.28
  • Sequential processing through Days 2-4 with same weights/biases
  • Final output: 1.0 (correctly predicts Company B Day 5 value)

Key Advantages

  • Separate paths prevent gradient explosion/vanishing
  • Can handle longer sequences than vanilla RNNs
  • Same weights/biases handle variable sequence lengths
  • Successfully demonstrates long-term memory retention

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome