AI summary by TwinMind

CatBoost Part 1: Ordered Target Encoding

StatQuest with Josh Starmer

8m 32s

Introduction & Context

  • StatQuest episode by Josh Starmer on CatBoost part one: ordered target encoding
  • Assumes familiarity with target encoding, gradient boost, XGBoost
  • CatBoost = categorical boosting, fundamentally similar to gradient boost/XGBoost
  • Focus on how CatBoost handles categorical variables differently

Target Encoding Background

  • Previous example: using favorite color + height to predict if someone loves Troll 2 movie
  • Applied one-hot label and target encoding to favorite color
  • Basic target encoding problem: each row's target value used to modify same row's feature value
  • Creates "leakage" - models work great on training data, poor on testing data
  • Solution discussed: k-fold target encoding splits data into K groups to reduce leakage

CatBoost Authors' Leakage Concern

  • Original CatBoost manuscript identified specific leakage scenario
  • Example: if everyone loved color blue (single category)
  • Leave-one-out target encoding results:
    • All rows with favorite color = 0.33 correspond to people who love Troll 2
    • All rows with favorite color = 0.5 correspond to people who don't love Troll 2
  • Any tree splitting on favorite color < 0.42 classifies training data perfectly
  • Author's opinion: scenario seems silly/unrealistic
    • Standard practice: convert single/binary categories to 0s and 1s rather than target encoding
    • Would just convert blue to zero instead of target encoding

CatBoost Ordered Target Encoding Method

  • CatBoost creators developed new categorical encoding method
  • "Ordered Target Encoding" - treats each row as fed sequentially into algorithm

Sequential Processing Approach

  • First row with blue: treated as if it's all data received so far
  • Ignores all other rows when target encoding first occurrence of blue
  • Each subsequent row uses only data from preceding rows

CatBoost Equation Differences

  • Uses defined prior/guess instead of overall mean (example: 0.05)
  • Simplifies denominator: adds 1 to number of rows rather than using weight
  • Formula: (option_count + prior) / (n + 1)

Step-by-Step Example Calculation

  • Row 1 (blue): option_count = 0, n = 0 → (0 + 0.05)/(0 + 1) = 0.05
  • Row 2 (red): option_count = 0, n = 0 → 0.05
  • Row 3 (green): option_count = 0, n = 0 → 0.05
  • Row 4 (blue, 2nd occurrence):
    • Previous blue person loved Troll 2
    • option_count = 1, n = 1 → (1 + 0.05)/(1 + 1) = 0.525
  • Row 5 (green, 2nd occurrence): option_count = 1, n = 1 → 0.525
  • Row 6 (green, 3rd occurrence):
    • Two previous green lovers both liked Troll 2
    • option_count = 2, n = 2 → (2 + 0.05)/(2 + 1) = 0.683
  • Row 7 (blue, 3rd occurrence):
    • Only 1 of 2 previous blue lovers liked Troll 2
    • option_count = 1, n = 2 → (1 + 0.05)/(2 + 1) = 0.35

Model Implementation

  • Order of data affects encoding results
  • For new data classification: entire dataset used for target encoding
  • Method works well despite questionable theoretical justification
  • Machine learning lesson: results matter more than perfect theoretical justification

Promotional Content

  • StatQuest PDF study guides available
  • Book: "StatQuest Illustrated Guide to Machine Learning" at statquest.org
  • Support options: Patreon, channel membership, merchandise, donations

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome