AI summary by TwinMind

XGBoost Part 1 (of 4): Regression

StatQuest with Josh Starmer

25m 46s

Introduction and Prerequisites

  • XGBoost = Extreme Gradient Boost
  • Presenter: Josh Stormer from StatQuest
  • Assumes familiarity with:
    • Gradient boost regression concepts
    • Regularization main ideas
  • XGBoost = big machine learning algorithm with many simple parts
  • Series structure: Part 1 (regression), Part 2 (classification), Part 3 (mathematical details)

Training Data Setup

  • Simple dataset used for demonstration (XGBoost designed for large/complex datasets)
  • X-axis: drug dosages
  • Y-axis: drug effectiveness measurements
  • Two observations: large positive values (drug helpful)
  • Two observations: large negative values (drug harmful)

Initial Prediction Process

  • First step: make initial prediction
  • Default initial prediction = 0.5 (regardless of regression/classification)
  • Corresponds to horizontal line on graph
  • Calculate residuals = observed - predicted values
  • Shows quality of initial prediction

XGBoost Tree Construction vs Standard Gradient Boost

  • Standard gradient boost: uses regular off-the-shelf regression trees
  • XGBoost: uses unique regression trees ("XGBoost trees")
  • Focus on most common way to build XGBoost trees for regression

Similarity Score Calculation

  • Each tree starts as single leaf with all residuals
  • Calculate quality/similarity score for residuals
  • Formula: Similarity = (sum of residuals)² / (number of residuals + λ)
  • λ (lambda) = regularization parameter
  • Initially set λ = 0 for examples
  • Example calculation: residuals 7.5, -7.5, 6.5, -10.5
    • Sum = -4 (cancellation occurs)
    • Similarity = (-4)² / 4 = 4

Tree Splitting Process

  • Test different thresholds to find best split
  • Threshold 1: dosage < 15
    • Left leaf: 1 residual (-10.5), similarity = 110.25
    • Right leaf: 3 residuals, similarity = 14.08
  • Threshold 2: dosage < 22.5, gain = 4
  • Threshold 3: dosage < 30, gain = 56.33
  • Choose threshold with largest gain

Gain Calculation

  • Formula: Gain = Left similarity + Right similarity - Root similarity
  • Threshold dosage < 15: Gain = 110.25 + 14.08 - 4 = 120.33
  • Best threshold selected based on highest gain
  • Continue splitting process for remaining nodes

Tree Pruning with Gamma

  • Pruning based on gain values
  • Gamma (γ) = tree complexity parameter (user-defined number)
  • Pruning rule: If (Gain - γ) < 0, remove branch
  • Example with γ = 130: keep branches (positive difference)
  • Example with γ = 150: remove branches (negative difference)
  • Prevents overfitting through aggressive pruning

Regularization Parameter Lambda Effects

  • When λ = 1 (vs λ = 0):
    • Root similarity: 3.2 (vs 4.0) - 20% decrease
    • Left leaf similarity: 55.12 (vs 110.25) - 50% decrease
    • Right leaf similarity: 10.56 (vs 14.08) - 25% decrease
  • Pattern: Decrease inversely proportional to number of residuals
  • Fewer residuals = larger decrease in similarity score
  • Results in smaller gain values, easier pruning
  • Can create negative gains even with γ = 0

Output Value Calculation

  • Formula: Output = (sum of residuals) / (number of residuals + λ)
  • Different from similarity score (no squaring of sum)
  • When λ = 0: output = average of residuals
  • When λ > 0: reduces individual observation impact
  • Example calculations:
    • Left leaf: -10.5/1 = -10.5 (λ=0) vs -5.25 (λ=1)
    • Middle leaf: 7 (average of residuals)
    • Right leaf: -7.5

Making New Predictions

  • Formula: New prediction = Initial prediction + η × Tree output
  • η (eta) = learning rate, default = 0.3
  • Example: dosage=10, new prediction = 0.5 + 0.3×(-10.5) = -2.65
  • New residuals smaller than original (progress in right direction)
  • Repeat process: build tree → make predictions → calculate residuals

Iterative Process

  • Continue building trees on new residuals
  • Stop when:
    • Residuals become very small
    • Maximum number of trees reached
  • Each iteration takes small step toward better predictions

Key Parameters Summary

  • λ (lambda): regularization parameter
    • Controls similarity scores and output values
    • Higher λ = more pruning, smaller outputs
    • Reduces prediction sensitivity to individual observations
  • γ (gamma): tree complexity parameter
    • Controls pruning threshold
    • Higher γ = more aggressive pruning
  • η (eta): learning rate, default = 0.3
    • Controls step size in predictions
  • Tree depth: default maximum = 6 levels

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome