AI summary by TwinMind
Word Embedding in PyTorch + Lightning
StatQuest with Josh Starmer
32m 2s
Introduction and Setup
- Tutorial on building word embedding networks using PyTorch + Lightning
- Presenter: Josh Starmer from StatQuest
- Three approaches covered: from scratch, using nn.Linear, using nn.Embedding
- Code available for free download (link in pinned comment)
Required Imports and Libraries
- torch: tensors, raw data storage, helper functions
- torch.nn: weights creation, helper functions
- optim: Adam optimizer for backpropagation training
- uniform: weight initialization
- TensorDataset, DataLoader: large-scale data handling, batching, shuffling
- lightning as L: code simplification, automatic optimization, cloud scaling
- pandas, matplotlib, seaborn: visualization
Training Data Setup
- Two simple sentences: "Troll 2 is great" and "Jim Kata is great"
- Unique tokens: Troll 2, is, great, Jim Kata
- One-hot encoding implementation:
- Troll 2: [1, 0, 0, 0]
- is: [0, 1, 0, 0]
- great: [0, 0, 1, 0]
- Jim Kata: [0, 0, 0, 1]
Prediction Task Definition
- Goal: predict next token given current token
- Training pairs:
- Troll 2 → is
- is → great
- great → Jim Kata (artificial for demo)
- Jim Kata → is
- Convert to tensors using torch.tensor
- Create TensorDataset and DataLoader for batch processing
From Scratch Implementation - Class Structure
- Class: WordEmbeddingFromScratch inherits from lightning.LightningModule
- Methods required:
- init: weight initialization, loss function setup
- forward: forward pass through network
- configure_optimizers: Adam optimizer setup
- training_step: loss calculation using cross-entropy
Weight Initialization
- Uniform distribution range: -0.5 to 0.5 (matches nn.Linear default)
- Variables: min_value = -0.5, max_value = 0.5
- Create parameters for all input-to-hidden weights
- Create parameters for all hidden-to-output weights
- Example initial values: Troll 2 weights = [-0.38, 0.42]
- Loss function: nn.CrossEntropyLoss()
Forward Method Implementation
- Input: one-hot encoded tensor (remove extra brackets)
- Calculate inputs_to_top_hidden: multiply inputs by top weights, sum
- Calculate inputs_to_bottom_hidden: multiply inputs by bottom weights, sum
- Calculate outputs: multiply hidden values by output weights
- Return: torch.stack of output values (preserves gradients)
- Note: Cross-entropy loss handles softmax internally
Optimizer and Training Configuration
- Adam optimizer with learning rate = 0.1
- Training method: training_step splits batch into input/labels
- Forward pass through network
- Loss calculation using cross-entropy
- Return loss for backpropagation
Model Creation and Initial Visualization
- Create model_from_scratch instance
- Print randomly initialized weights using named_parameters()
- Organize weights into pandas DataFrame:
- W1: weights to top activation function
- W2: weights to bottom activation function
- Create scatter plot: W1 on x-axis, W2 on y-axis
- Initial observation: Troll 2 and Jim Kata embeddings very different
Training Process
- Lightning trainer with max_epochs=100
- Call trainer.fit(model_from_scratch, data_loader)
- Training duration: few seconds for simple network
- Post-training visualization shows Troll 2 and Jim Kata embeddings now similar
- Verification: input Troll 2 predicts "is" correctly after softmax
Simplified Implementation with nn.Linear
- Class: WordEmbeddingWithLinear
- Two nn.Linear layers replace manual weight creation:
- input_to_hidden: nn.Linear(4, 2, bias=False)
- hidden_to_output: nn.Linear(2, 4, bias=False)
- Forward method simplified: just pass through linear layers
- Significantly reduced code compared to from-scratch implementation
Linear Implementation Training
- Same training process: 100 epochs with Adam optimizer
- Weight access: model.input_to_hidden.weight.detach().numpy()
- Use indices [0] and [1] for top/bottom activation function weights
- Results identical to from-scratch implementation
- Post-training: Troll 2 and Jim Kata embeddings cluster together
Pre-trained Embeddings with nn.Embedding
- Load existing embedding weights from trained model
- Weights arranged in rows, need transposition for nn.Embedding
- Create nn.Embedding object: nn.Embedding.from_pretrained(weights.T)
- Access embeddings by index: word_embeddings(torch.tensor([0]))
- Create token-to-index dictionary for easier access
- Ready for integration with larger networks (e.g., Transformers)
Visualization and Verification
- Scatter plots show embedding evolution during training
- Initial: tokens scattered randomly
- Post-training: semantically similar tokens (Troll 2, Jim Kata) cluster together
- Quantitative verification: softmax outputs match expected predictions
- DataFrame organization makes weight inspection easier
Key Technical Details
- Cross-entropy loss includes softmax calculation
- torch.stack preserves gradients for backpropagation
- Uniform weight initialization matches nn.Linear defaults
- Lightning module provides automatic optimization and scaling
- DataLoader enables batch processing and data shuffling
