Introduction and Prerequisites

• Part 3 of gradient boost series focusing on classification
• Requires knowledge of gradient boost part 1 (regression) and logistic regression
• Presenter: Josh Dahmer from StatQuest

Training Data Setup

• Dataset: 6 people's popcorn preferences
• Features: age, favorite color, loved Troll 2 movie (yes/no)
• Target: binary classification (loves Troll 2 or not)
• 4 people love Troll 2, 2 people do not

Initial Prediction (Leaf)

• Classification starts with log odds (logistic regression equivalent of average)
• Log odds = log(4/2) = 0.7 (rounded)
  • Exact value: 0.6931
• Convert to probability using logistic function: P = e^(log odds) / (1 + e^(log odds))
• Initial probability = 0.7 (rounded)
  • Exact value: 0.666
• Classification threshold: 0.5 (can be adjusted)
• Initial prediction: everyone classified as loving Troll 2

First Tree Construction

Residual Calculation

• Pseudo residuals = observed - predicted probabilities
• Red dots (don't love): probability = 0, residual = 0 - 0.7 = -0.7
• Blue dots (love): probability = 1, residual = 1 - 0.7 = 0.3
• Build tree using age, favorite color, likes popcorn to predict residuals

Tree Structure and Leaf Limits

• Tree limited to 3 leaves (for example simplicity)
• Practice: typically 8-32 leaves maximum
• Three groups of data points distributed across leaves

Output Value Calculations

• Formula for leaf output values: Σ(residuals) / Σ(previous_prob × (1 - previous_prob))
• Leaf 1 (single residual): -3.3
• Leaf 2 (two residuals): -1.0
• Leaf 3 (multiple residuals): 1.4
• Formula derivation saved for part 4 (technical details)

First Prediction Update

Learning Rate Application

• Learning rate: 0.8 (large for illustration; 0.1 more common in practice)
• New log odds = previous prediction + (learning rate × tree output)
• Person 1: 0.7 + (0.8 × 1.4) = 1.8 → probability = 0.9
• Person 2: 0.7 + (0.8 × -1.0) = -0.1 → probability = 0.5
• Some predictions improve, others worsen (reason for multiple trees)

Second Tree Construction

New Residual Calculation

• Each person now has different predicted probability
• Calculate residuals using new individual predictions
• Plot observed vs predicted probabilities for each person

Second Tree Output Values

• Leaf 1 (person 2): output = 2.0
• Leaf 2 (person 3): output = -2.0
• Leaf 3 (multiple people): output = 0.6

Final Model Structure

• Initial leaf: single prediction for all
• Tree 1: based on initial residuals, scaled by learning rate
• Tree 2: based on updated residuals, scaled by learning rate
• Process continues until maximum trees reached or convergence

Classification Example

New Person Prediction

• Start with initial leaf: 0.7
• Run through Tree 1: add scaled output
• Run through Tree 2: add scaled output
• Final log odds: 2.3
• Convert to probability: 0.9
• Classification: loves Troll 2 (0.9 > 0.5 threshold)

Key Differences from Regression

• Uses log odds instead of direct averages
• Complex transformation formula for leaf outputs
• Probability conversion required for final classification
• Cannot simply add residuals due to log odds vs probability mismatch

Practical Considerations

• Typical trees: 8-32 leaves (not 3 as in example)
• Small example due to limited training data
• Part 4 will cover mathematical derivations