AI summary by TwinMind

One-Hot, Label, Target and K-Fold Target Encoding, Clearly Explained!!!

StatQuest with Josh Starmer

15m 23s

Introduction and Problem Setup

  • Josh Starmer from StatQuest explaining encoding methods for discrete variables
  • Example dataset: favorite color (blue, red, green) and height to predict "loves Troll 2" movie
  • Discrete features problematic for neural networks and many ML algorithms
  • Need to convert discrete data to numerical values

One Hot Encoding

  • Used when discrete variable has 3+ options
  • Create new column for each option (blue, red, green)
  • Set value to 1 if original column had that option, 0 otherwise
  • Binary target (loves Troll 2: yes/no) converted to 1/0
  • Works fine with few options but problematic with many
    • Example: 41,683 postal codes would create 41,683 new columns
    • Makes data difficult to work with

Label Encoding

  • Alternative when too many discrete options exist
  • Simply assign arbitrary numbers to each option
    • Blue = 0, Red = 1, Green = 2
  • Problem: arbitrary numbers may be interpreted as meaningful order by algorithms
    • Decision trees forced to group colors based on random number assignments

Basic Target Encoding

  • Calculate mean value of target for each discrete option
  • Blue: 1/3 people love Troll 2 → replace with 0.33
  • Red: 0/1 people love Troll 2 → replace with 0.00
  • Green: 2/3 people love Troll 2 → replace with 0.67
  • Uses target variable to determine encoding values

Weighted Mean Target Encoding (Bayesian Mean Encoding)

  • Addresses confidence issues when limited data per category
  • Red only has 1 person vs 3 people each for blue/green
  • Formula combines category mean with overall target mean
  • Parameters:
    • Category mean for specific option
    • n = number of people in category
    • Overall target mean = 3/7 (overall people loving Troll 2)
    • m = user-defined weight parameter (set to 2 in example)
  • Results with m=2:
    • Blue: 0.37 (similar to before due to sufficient data)
    • Red: 0.29 (closer to overall mean due to limited data)
    • Green: 0.57 (similar to before due to sufficient data)
  • Less data → encoding closer to overall mean (best guess)
  • More data → encoding based more on actual category data

Data Leakage Problem

  • Target encoding uses target variable to modify input features
  • Creates data leakage leading to overfitting
  • Models work great on training data, poor on testing data

K-Fold Target Encoding Solution

  • Reduces data leakage by splitting data into k equal subsets
  • "Fold" = equal-sized subsets, "k" = number of subsets
  • Example with k=2 (two-fold):
    • Subset A and Subset B
    • To encode blue in Subset A: ignore target values in A, use only Subset B values
    • Blue in A: uses B's data → 0.22
    • Blue in B: uses A's data → 0.50
  • Different subsets get different values for same category (acceptable for continuous variables)
  • Process for all colors and subsets, then merge back together

Leave-One-Out Target Encoding

  • Special case where k = number of rows (k=7 in example)
  • Each row uses all other target values except its own
  • Mixed success reported online
  • Alternative: k=5 also shows success in practice

Key Takeaways

  • One hot: good for few categories, creates too many columns for many categories
  • Label: simple but arbitrary ordering can mislead algorithms
  • Target: uses target mean but causes data leakage
  • Weighted target: handles low-data categories better
  • K-fold target: reduces leakage, k=5 or leave-one-out commonly successful

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome