AI summary by TwinMind

Gradient Boost Part 3 (of 4): Classification

StatQuest with Josh Starmer

17m 2s

Introduction and Prerequisites

  • Part 3 of gradient boost series focusing on classification
  • Requires knowledge of gradient boost part 1 (regression) and logistic regression
  • Presenter: Josh Dahmer from StatQuest

Training Data Setup

  • Dataset: 6 people's popcorn preferences
  • Features: age, favorite color, loved Troll 2 movie (yes/no)
  • Target: binary classification (loves Troll 2 or not)
  • 4 people love Troll 2, 2 people do not

Initial Prediction (Leaf)

  • Classification starts with log odds (logistic regression equivalent of average)
  • Log odds = log(4/2) = 0.7 (rounded)
    • Exact value: 0.6931
  • Convert to probability using logistic function: P = e^(log odds) / (1 + e^(log odds))
  • Initial probability = 0.7 (rounded)
    • Exact value: 0.666
  • Classification threshold: 0.5 (can be adjusted)
  • Initial prediction: everyone classified as loving Troll 2

First Tree Construction

Residual Calculation

  • Pseudo residuals = observed - predicted probabilities
  • Red dots (don't love): probability = 0, residual = 0 - 0.7 = -0.7
  • Blue dots (love): probability = 1, residual = 1 - 0.7 = 0.3
  • Build tree using age, favorite color, likes popcorn to predict residuals

Tree Structure and Leaf Limits

  • Tree limited to 3 leaves (for example simplicity)
  • Practice: typically 8-32 leaves maximum
  • Three groups of data points distributed across leaves

Output Value Calculations

  • Formula for leaf output values: Σ(residuals) / Σ(previous_prob × (1 - previous_prob))
  • Leaf 1 (single residual): -3.3
  • Leaf 2 (two residuals): -1.0
  • Leaf 3 (multiple residuals): 1.4
  • Formula derivation saved for part 4 (technical details)

First Prediction Update

Learning Rate Application

  • Learning rate: 0.8 (large for illustration; 0.1 more common in practice)
  • New log odds = previous prediction + (learning rate × tree output)
  • Person 1: 0.7 + (0.8 × 1.4) = 1.8 → probability = 0.9
  • Person 2: 0.7 + (0.8 × -1.0) = -0.1 → probability = 0.5
  • Some predictions improve, others worsen (reason for multiple trees)

Second Tree Construction

New Residual Calculation

  • Each person now has different predicted probability
  • Calculate residuals using new individual predictions
  • Plot observed vs predicted probabilities for each person

Second Tree Output Values

  • Leaf 1 (person 2): output = 2.0
  • Leaf 2 (person 3): output = -2.0
  • Leaf 3 (multiple people): output = 0.6

Final Model Structure

  • Initial leaf: single prediction for all
  • Tree 1: based on initial residuals, scaled by learning rate
  • Tree 2: based on updated residuals, scaled by learning rate
  • Process continues until maximum trees reached or convergence

Classification Example

New Person Prediction

  • Start with initial leaf: 0.7
  • Run through Tree 1: add scaled output
  • Run through Tree 2: add scaled output
  • Final log odds: 2.3
  • Convert to probability: 0.9
  • Classification: loves Troll 2 (0.9 > 0.5 threshold)

Key Differences from Regression

  • Uses log odds instead of direct averages
  • Complex transformation formula for leaf outputs
  • Probability conversion required for final classification
  • Cannot simply add residuals due to log odds vs probability mismatch

Practical Considerations

  • Typical trees: 8-32 leaves (not 3 as in example)
  • Small example due to limited training data
  • Part 4 will cover mathematical derivations

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome