AI summary by TwinMind

Word Embedding and Word2Vec, Clearly Explained!!!

StatQuest with Josh Starmer

16m 11s

Introduction and Problem Setup

  • Josh Starmer introduces word embeddings and word2vec for converting words to numbers
  • Machine learning algorithms (including neural networks) don't work well with words directly
  • Need method to convert words into numerical representations for ML processing

Random Number Assignment Limitations

  • Simple approach: assign each word random number (e.g., "Troll 2" = 4.2, "awesome!" = -32.1)
  • Problem: similar words ("great!", "awesome!") get very different numbers
  • Neural network requires more complexity/training since learning one word doesn't help with similar words
  • Need similar words to have similar numbers for efficient learning

Word Context Considerations

  • Same words used in different contexts (positive vs sarcastic "great")
  • Multiple numbers per word needed to capture different contextual uses
  • Example: "StatQuest is great!" (positive) vs "My cell phone is broken, great" (sarcastic)

Neural Network Architecture for Word Embeddings

  • Training data: "Troll 2 is great!" and "Gymkata is great!"
  • 4 unique words = 4 inputs in neural network
  • Each input connects to activation functions (using identity function)
  • Number of activation functions = number of embeddings per word (example uses 2)
  • Weights on connections become the word embeddings
  • Initial weights are random, optimized through backpropagation

Training Process

  • Goal: predict next word in phrase using current word
  • Input representation: one-hot encoding (1 for target word, 0s elsewhere)
  • Training examples:
    • "Troll 2" → predict "is"
    • "is" → predict "great!"
    • "Gymkata" → predict "is"
  • Outputs run through softmax function for classification
  • Uses cross entropy loss function for backpropagation

Weight Visualization and Results

  • Before training: weights plotted on 2D graph (x-axis: top activation weights, y-axis: bottom activation weights)
  • Initial positions: "Troll 2" (0.38, 0.42), words scattered randomly
  • After training: "Troll 2" and "Gymkata" positioned closer together (similar context)
  • Final predictions: 100% accuracy for all training examples

Word2Vec Methods

  • Continuous Bag-of-Words (CBOW): uses surrounding words to predict middle word
    • Example: "Troll 2" + "great!" → predict "is"
  • Skip-gram: uses middle word to predict surrounding words
    • Example: "is" → predict "Troll 2", "great!", "Gymkata"

Real-World Implementation Scale

  • Production systems use 100+ activation functions (not just 2)
  • Training data: entire Wikipedia (not just 2 sentences)
  • Vocabulary: ~3 million words/phrases (not just 4)
  • Total weights: 3M words × 100 embeddings × 2 (input/output) = 600 million weights
  • Training can be very slow due to massive scale

Negative Sampling Optimization

  • Word2vec uses negative sampling to speed up training
  • Randomly selects subset of words we don't want to predict
  • Example: predicting "aardvark" → "a"
    • Ignores ~300M weights from non-target input words (multiply by zero)
    • Randomly selects negative examples (e.g., "abandon")
    • Typically selects 2-20 negative words per training step
  • Result: optimizes only ~300 weights per step instead of 600 million
  • Enables efficient training on large vocabularies

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome