AI summary by TwinMind
Word Embedding and Word2Vec, Clearly Explained!!!
StatQuest with Josh Starmer
16m 11s
Introduction and Problem Setup
- Josh Starmer introduces word embeddings and word2vec for converting words to numbers
- Machine learning algorithms (including neural networks) don't work well with words directly
- Need method to convert words into numerical representations for ML processing
Random Number Assignment Limitations
- Simple approach: assign each word random number (e.g., "Troll 2" = 4.2, "awesome!" = -32.1)
- Problem: similar words ("great!", "awesome!") get very different numbers
- Neural network requires more complexity/training since learning one word doesn't help with similar words
- Need similar words to have similar numbers for efficient learning
Word Context Considerations
- Same words used in different contexts (positive vs sarcastic "great")
- Multiple numbers per word needed to capture different contextual uses
- Example: "StatQuest is great!" (positive) vs "My cell phone is broken, great" (sarcastic)
Neural Network Architecture for Word Embeddings
- Training data: "Troll 2 is great!" and "Gymkata is great!"
- 4 unique words = 4 inputs in neural network
- Each input connects to activation functions (using identity function)
- Number of activation functions = number of embeddings per word (example uses 2)
- Weights on connections become the word embeddings
- Initial weights are random, optimized through backpropagation
Training Process
- Goal: predict next word in phrase using current word
- Input representation: one-hot encoding (1 for target word, 0s elsewhere)
- Training examples:
- "Troll 2" → predict "is"
- "is" → predict "great!"
- "Gymkata" → predict "is"
- Outputs run through softmax function for classification
- Uses cross entropy loss function for backpropagation
Weight Visualization and Results
- Before training: weights plotted on 2D graph (x-axis: top activation weights, y-axis: bottom activation weights)
- Initial positions: "Troll 2" (0.38, 0.42), words scattered randomly
- After training: "Troll 2" and "Gymkata" positioned closer together (similar context)
- Final predictions: 100% accuracy for all training examples
Word2Vec Methods
- Continuous Bag-of-Words (CBOW): uses surrounding words to predict middle word
- Example: "Troll 2" + "great!" → predict "is"
- Skip-gram: uses middle word to predict surrounding words
- Example: "is" → predict "Troll 2", "great!", "Gymkata"
Real-World Implementation Scale
- Production systems use 100+ activation functions (not just 2)
- Training data: entire Wikipedia (not just 2 sentences)
- Vocabulary: ~3 million words/phrases (not just 4)
- Total weights: 3M words × 100 embeddings × 2 (input/output) = 600 million weights
- Training can be very slow due to massive scale
Negative Sampling Optimization
- Word2vec uses negative sampling to speed up training
- Randomly selects subset of words we don't want to predict
- Example: predicting "aardvark" → "a"
- Ignores ~300M weights from non-target input words (multiply by zero)
- Randomly selects negative examples (e.g., "abandon")
- Typically selects 2-20 negative words per training step
- Result: optimizes only ~300 weights per step instead of 600 million
- Enables efficient training on large vocabularies
