AI summary by TwinMind

XGBoost Part 2 (of 4): Classification

StatQuest with Josh Starmer

25m 17s

Introduction and Overview

  • XGBoost classification tutorial (Part 2 of series)
  • Assumes familiarity with XGBoost regression and gradient boosting classification
  • Uses simple training data: 4 drug dosages (green dots = effective, red dots = not effective)
  • XGBoost is complex algorithm with many simple, understandable parts

Initial Prediction Setup

  • First step: make initial prediction (default = 0.5 probability regardless of regression/classification)
  • Default prediction: 50% chance drug is effective for any dosage
  • Graph setup: y-axis represents probability drug is effective
  • Effective dosages (green) moved to probability = 1, ineffective (red) at probability = 0
  • Residuals calculated as differences between observed and predicted values

Similarity Score Formula for Classification

  • New formula for classification similarity scores
  • Numerator: sum of residuals squared (same as regression)
  • Denominator: lambda + sum for each observation of (previous probability × (1 - previous probability))
  • Different from regression denominator but closely related (explained in Part 3)

Building the First Tree

  • Tree starts as single leaf with all residuals
  • Root similarity score = 0 (residuals cancel out in numerator)
  • Split threshold: dosage < 15 (average between last two observations)
  • Left leaf: 3 residuals (dosage < 15), Right leaf: 1 residual (dosage ≥ 15)
  • Lambda = 0 for simplicity (reduces similarity scores when > 0)

Similarity Score Calculations

  • Left leaf similarity: 0.33 (two residuals cancel, one remains)
  • Right leaf similarity: 1.0
  • Gain calculation: 0.33 + 1.0 - 0 = 1.33
  • Dosage < 15 threshold provides highest gain

Further Tree Splitting

  • Split left branch further: dosage < 5 vs dosage < 10
  • Dosage < 5 threshold: gain = 2.66
  • Dosage < 10 threshold: gain = 0.66
  • Choose dosage < 5 (higher gain)
  • Tree limited to 2 levels

Cover and Minimum Leaf Requirements

  • Cover = denominator of similarity score - lambda
  • For classification: cover = sum of (previous probability × (1 - previous probability))
  • For regression: cover = number of residuals in leaf
  • Default minimum cover = 1
  • Example calculations show leaves would be rejected with default minimum
  • Set minimum cover = 0 (min_child_weight = 0) to allow tree building

Tree Pruning

  • Prune by comparing gain to gamma parameter
  • If (gain - gamma) < 0, prune the branch
  • Example: gamma = 2 → no pruning, gamma = 3 → prune branches
  • Lambda > 0 reduces similarity scores → lower gains → easier pruning
  • Lambda reduces tree sensitivity to individual observations

Output Values for Leaves

  • Formula: sum of residuals / (sum of previous probability × (1 - previous probability) + lambda)
  • Similar to non-extreme gradient boost formula (except lambda)
  • Example calculations with lambda = 0: output values = -2, 2, -2
  • Lambda > 0 shrinks output values toward zero (regularization effect)

Making New Predictions

  • Convert initial probability (0.5) to log odds = 0
  • New prediction = initial log odds + (learning rate × tree output)
  • Default learning rate (eta) = 0.3
  • Convert log odds back to probability using logistic function
  • Example: dosage 2 → log odds = -0.6 → probability = 0.35

Prediction Examples

  • Dosage 2: 0.5 → 0.35 (residual reduced)
  • Dosage 8: 0.5 → 0.65 (residual reduced)
  • All new predictions have smaller residuals (progress toward correct values)

Building Additional Trees

  • Build second tree using new residuals
  • Previous probabilities now different for each observation
  • Similarity score calculations more complex (mixed previous probabilities)
  • Continue building trees until residuals very small or max trees reached

Summary of Key Concepts

  • Similarity scores and gain determine data splitting
  • Pruning using gamma parameter (gain - gamma comparison)
  • Lambda regularization: shrinks similarity scores and output values
  • Cover metric determines minimum residuals per leaf
  • Classification requires log odds conversions and logistic function
  • Process continues iteratively building trees on residuals

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome