AI summary by TwinMind
Live Stream - More details about Target Encoding/AMA/Silly Songs
StatQuest with Josh Starmer
51m 17s
Introduction and Setup
- Josh Starmer hosting StatQuest live stream
- Woke up at 1am, been awake 11+ hours, apologetic about being sleepy
- Apologized for not practicing Tabla as promised in previous stream
- Mentioned having Tambora instrument, considering friend collaboration for drone background
- Encouraged questions in chat, will address at end of session
Basic Encoding Strategies Review
- Dataset: favorite color (blue/red/green) + height to predict "loves Troll 2"
- One-hot encoding works for few discrete values but problematic with many options
- Example: 41,683 postal codes in US would create 41,683 new columns
- Target encoding as alternative: replace each option with weighted mean of target
Standard Target Encoding Process
- Weighted mean formula: (n × option_mean + m × overall_mean) / (n + m)
- Blue example: (3 × 1/3 + 2 × 3/7) / (3 + 2) = 0.37
- Red example: (2 × 0/2 + 2 × 3/7) / (2 + 2) = 0.29
- Green example: (2 × 2/2 + 2 × 3/7) / (2 + 2) = 0.57
- M=2 hyperparameter: need ≥3 rows before option mean becomes more important than overall mean
Leakage Problem
- Using target values to modify feature values creates leakage
- Results in overfitted models: great on training data, poor on testing data
- Relationship established between target and encoded features
K-Fold Target Encoding Solution
- Split data into K equal-sized subsets
- Two-fold example: subset A (4 rows), subset B (3 rows)
- Encode subset A using only target values from subset B
- Encode subset B using only target values from subset A
- Blue in subset A: 0.22, Blue in subset B: 0.5
- Different values for same category across subsets is acceptable
- Leave-one-out encoding: K=7 (number of total rows)
- Alternative: K=5 commonly used with large datasets
CatBoost Ordered Target Encoding
- Treats samples as sequential input
- Only uses data from preceding rows for encoding
- Different equation: (count_in_class + prior) / (total_count + 1)
- Prior typically 0.05 (from documentation example)
- First occurrences get prior value (0.05)
- Subsequent occurrences use accumulated statistics from previous rows
- Blue sequence: 0.05 → 0.525 → 0.35
- Green sequence: 0.05 → 0.525 → 0.67
CatBoost Justification Critique
- Authors claim leave-one-out causes leakage with single option
- Josh argues this justification doesn't make sense
- When only 1-2 options exist, everyone uses one-hot encoding
- CatBoost itself uses one-hot encoding for few categories
- Many practitioners successfully use leave-one-out encoding
- Despite weak justification, CatBoost works well in practice
Machine Learning Philosophy
- Unlike pure mathematics, ML focuses on results over perfect logic
- Sometimes requires "getting messy" to achieve good performance
- Pragmatic approach: if it works well, it's valuable
Q&A Session Highlights
- Inference on new data: use all training data to encode (no leakage concern)
- Regression targets: same process, just calculate means of continuous values
- Book availability: Amazon for US/Europe, working on India/Australia/Brazil printing
- Jobs: StatQuest, Lightning AI educator, book writing
- Leakage definition: when prediction target influences feature encoding
- Feature engineering: normalization, ratios, combining features
- Ordinal data: often treated as numeric without encoding needed
