AI summary by TwinMind
CatBoost Part 1: Ordered Target Encoding
StatQuest with Josh Starmer
8m 32s
Introduction & Context
- StatQuest episode by Josh Starmer on CatBoost part one: ordered target encoding
- Assumes familiarity with target encoding, gradient boost, XGBoost
- CatBoost = categorical boosting, fundamentally similar to gradient boost/XGBoost
- Focus on how CatBoost handles categorical variables differently
Target Encoding Background
- Previous example: using favorite color + height to predict if someone loves Troll 2 movie
- Applied one-hot label and target encoding to favorite color
- Basic target encoding problem: each row's target value used to modify same row's feature value
- Creates "leakage" - models work great on training data, poor on testing data
- Solution discussed: k-fold target encoding splits data into K groups to reduce leakage
CatBoost Authors' Leakage Concern
- Original CatBoost manuscript identified specific leakage scenario
- Example: if everyone loved color blue (single category)
- Leave-one-out target encoding results:
- All rows with favorite color = 0.33 correspond to people who love Troll 2
- All rows with favorite color = 0.5 correspond to people who don't love Troll 2
- Any tree splitting on favorite color < 0.42 classifies training data perfectly
- Author's opinion: scenario seems silly/unrealistic
- Standard practice: convert single/binary categories to 0s and 1s rather than target encoding
- Would just convert blue to zero instead of target encoding
CatBoost Ordered Target Encoding Method
- CatBoost creators developed new categorical encoding method
- "Ordered Target Encoding" - treats each row as fed sequentially into algorithm
Sequential Processing Approach
- First row with blue: treated as if it's all data received so far
- Ignores all other rows when target encoding first occurrence of blue
- Each subsequent row uses only data from preceding rows
CatBoost Equation Differences
- Uses defined prior/guess instead of overall mean (example: 0.05)
- Simplifies denominator: adds 1 to number of rows rather than using weight
- Formula: (option_count + prior) / (n + 1)
Step-by-Step Example Calculation
- Row 1 (blue): option_count = 0, n = 0 → (0 + 0.05)/(0 + 1) = 0.05
- Row 2 (red): option_count = 0, n = 0 → 0.05
- Row 3 (green): option_count = 0, n = 0 → 0.05
- Row 4 (blue, 2nd occurrence):
- Previous blue person loved Troll 2
- option_count = 1, n = 1 → (1 + 0.05)/(1 + 1) = 0.525
- Row 5 (green, 2nd occurrence): option_count = 1, n = 1 → 0.525
- Row 6 (green, 3rd occurrence):
- Two previous green lovers both liked Troll 2
- option_count = 2, n = 2 → (2 + 0.05)/(2 + 1) = 0.683
- Row 7 (blue, 3rd occurrence):
- Only 1 of 2 previous blue lovers liked Troll 2
- option_count = 1, n = 2 → (1 + 0.05)/(2 + 1) = 0.35
Model Implementation
- Order of data affects encoding results
- For new data classification: entire dataset used for target encoding
- Method works well despite questionable theoretical justification
- Machine learning lesson: results matter more than perfect theoretical justification
Promotional Content
- StatQuest PDF study guides available
- Book: "StatQuest Illustrated Guide to Machine Learning" at statquest.org
- Support options: Patreon, channel membership, merchandise, donations
