AI summary by TwinMind

Word Embedding in PyTorch + Lightning

StatQuest with Josh Starmer

32m 2s

Introduction and Setup

  • Tutorial on building word embedding networks using PyTorch + Lightning
  • Presenter: Josh Starmer from StatQuest
  • Three approaches covered: from scratch, using nn.Linear, using nn.Embedding
  • Code available for free download (link in pinned comment)

Required Imports and Libraries

  • torch: tensors, raw data storage, helper functions
  • torch.nn: weights creation, helper functions
  • optim: Adam optimizer for backpropagation training
  • uniform: weight initialization
  • TensorDataset, DataLoader: large-scale data handling, batching, shuffling
  • lightning as L: code simplification, automatic optimization, cloud scaling
  • pandas, matplotlib, seaborn: visualization

Training Data Setup

  • Two simple sentences: "Troll 2 is great" and "Jim Kata is great"
  • Unique tokens: Troll 2, is, great, Jim Kata
  • One-hot encoding implementation:
    • Troll 2: [1, 0, 0, 0]
    • is: [0, 1, 0, 0]
    • great: [0, 0, 1, 0]
    • Jim Kata: [0, 0, 0, 1]

Prediction Task Definition

  • Goal: predict next token given current token
  • Training pairs:
    • Troll 2 → is
    • is → great
    • great → Jim Kata (artificial for demo)
    • Jim Kata → is
  • Convert to tensors using torch.tensor
  • Create TensorDataset and DataLoader for batch processing

From Scratch Implementation - Class Structure

  • Class: WordEmbeddingFromScratch inherits from lightning.LightningModule
  • Methods required:
    • init: weight initialization, loss function setup
    • forward: forward pass through network
    • configure_optimizers: Adam optimizer setup
    • training_step: loss calculation using cross-entropy

Weight Initialization

  • Uniform distribution range: -0.5 to 0.5 (matches nn.Linear default)
  • Variables: min_value = -0.5, max_value = 0.5
  • Create parameters for all input-to-hidden weights
  • Create parameters for all hidden-to-output weights
  • Example initial values: Troll 2 weights = [-0.38, 0.42]
  • Loss function: nn.CrossEntropyLoss()

Forward Method Implementation

  • Input: one-hot encoded tensor (remove extra brackets)
  • Calculate inputs_to_top_hidden: multiply inputs by top weights, sum
  • Calculate inputs_to_bottom_hidden: multiply inputs by bottom weights, sum
  • Calculate outputs: multiply hidden values by output weights
  • Return: torch.stack of output values (preserves gradients)
  • Note: Cross-entropy loss handles softmax internally

Optimizer and Training Configuration

  • Adam optimizer with learning rate = 0.1
  • Training method: training_step splits batch into input/labels
  • Forward pass through network
  • Loss calculation using cross-entropy
  • Return loss for backpropagation

Model Creation and Initial Visualization

  • Create model_from_scratch instance
  • Print randomly initialized weights using named_parameters()
  • Organize weights into pandas DataFrame:
    • W1: weights to top activation function
    • W2: weights to bottom activation function
  • Create scatter plot: W1 on x-axis, W2 on y-axis
  • Initial observation: Troll 2 and Jim Kata embeddings very different

Training Process

  • Lightning trainer with max_epochs=100
  • Call trainer.fit(model_from_scratch, data_loader)
  • Training duration: few seconds for simple network
  • Post-training visualization shows Troll 2 and Jim Kata embeddings now similar
  • Verification: input Troll 2 predicts "is" correctly after softmax

Simplified Implementation with nn.Linear

  • Class: WordEmbeddingWithLinear
  • Two nn.Linear layers replace manual weight creation:
    • input_to_hidden: nn.Linear(4, 2, bias=False)
    • hidden_to_output: nn.Linear(2, 4, bias=False)
  • Forward method simplified: just pass through linear layers
  • Significantly reduced code compared to from-scratch implementation

Linear Implementation Training

  • Same training process: 100 epochs with Adam optimizer
  • Weight access: model.input_to_hidden.weight.detach().numpy()
  • Use indices [0] and [1] for top/bottom activation function weights
  • Results identical to from-scratch implementation
  • Post-training: Troll 2 and Jim Kata embeddings cluster together

Pre-trained Embeddings with nn.Embedding

  • Load existing embedding weights from trained model
  • Weights arranged in rows, need transposition for nn.Embedding
  • Create nn.Embedding object: nn.Embedding.from_pretrained(weights.T)
  • Access embeddings by index: word_embeddings(torch.tensor([0]))
  • Create token-to-index dictionary for easier access
  • Ready for integration with larger networks (e.g., Transformers)

Visualization and Verification

  • Scatter plots show embedding evolution during training
  • Initial: tokens scattered randomly
  • Post-training: semantically similar tokens (Troll 2, Jim Kata) cluster together
  • Quantitative verification: softmax outputs match expected predictions
  • DataFrame organization makes weight inspection easier

Key Technical Details

  • Cross-entropy loss includes softmax calculation
  • torch.stack preserves gradients for backpropagation
  • Uniform weight initialization matches nn.Linear defaults
  • Lightning module provides automatic optimization and scaling
  • DataLoader enables batch processing and data shuffling

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome