AI summary by TwinMind
One-Hot, Label, Target and K-Fold Target Encoding, Clearly Explained!!!
StatQuest with Josh Starmer
15m 23s
Introduction and Problem Setup
- Josh Starmer from StatQuest explaining encoding methods for discrete variables
- Example dataset: favorite color (blue, red, green) and height to predict "loves Troll 2" movie
- Discrete features problematic for neural networks and many ML algorithms
- Need to convert discrete data to numerical values
One Hot Encoding
- Used when discrete variable has 3+ options
- Create new column for each option (blue, red, green)
- Set value to 1 if original column had that option, 0 otherwise
- Binary target (loves Troll 2: yes/no) converted to 1/0
- Works fine with few options but problematic with many
- Example: 41,683 postal codes would create 41,683 new columns
- Makes data difficult to work with
Label Encoding
- Alternative when too many discrete options exist
- Simply assign arbitrary numbers to each option
- Blue = 0, Red = 1, Green = 2
- Problem: arbitrary numbers may be interpreted as meaningful order by algorithms
- Decision trees forced to group colors based on random number assignments
Basic Target Encoding
- Calculate mean value of target for each discrete option
- Blue: 1/3 people love Troll 2 → replace with 0.33
- Red: 0/1 people love Troll 2 → replace with 0.00
- Green: 2/3 people love Troll 2 → replace with 0.67
- Uses target variable to determine encoding values
Weighted Mean Target Encoding (Bayesian Mean Encoding)
- Addresses confidence issues when limited data per category
- Red only has 1 person vs 3 people each for blue/green
- Formula combines category mean with overall target mean
- Parameters:
- Category mean for specific option
- n = number of people in category
- Overall target mean = 3/7 (overall people loving Troll 2)
- m = user-defined weight parameter (set to 2 in example)
- Results with m=2:
- Blue: 0.37 (similar to before due to sufficient data)
- Red: 0.29 (closer to overall mean due to limited data)
- Green: 0.57 (similar to before due to sufficient data)
- Less data → encoding closer to overall mean (best guess)
- More data → encoding based more on actual category data
Data Leakage Problem
- Target encoding uses target variable to modify input features
- Creates data leakage leading to overfitting
- Models work great on training data, poor on testing data
K-Fold Target Encoding Solution
- Reduces data leakage by splitting data into k equal subsets
- "Fold" = equal-sized subsets, "k" = number of subsets
- Example with k=2 (two-fold):
- Subset A and Subset B
- To encode blue in Subset A: ignore target values in A, use only Subset B values
- Blue in A: uses B's data → 0.22
- Blue in B: uses A's data → 0.50
- Different subsets get different values for same category (acceptable for continuous variables)
- Process for all colors and subsets, then merge back together
Leave-One-Out Target Encoding
- Special case where k = number of rows (k=7 in example)
- Each row uses all other target values except its own
- Mixed success reported online
- Alternative: k=5 also shows success in practice
Key Takeaways
- One hot: good for few categories, creates too many columns for many categories
- Label: simple but arbitrary ordering can mislead algorithms
- Target: uses target mean but causes data leakage
- Weighted target: handles low-data categories better
- K-fold target: reduces leakage, k=5 or leave-one-out commonly successful
