AI summary by TwinMind

Gradient Boost Part 2 (of 4): Regression Details

StatQuest with Josh Starmer

26m 45s

Introduction and Prerequisites

  • Josh Starmer from StatQuest tutorial on gradient boost algorithmic details for regression
  • Assumes viewers watched "gradient boost part one regression main ideas"
  • Recommends understanding gradient descent (optional but helpful)
  • Will walk through original gradient boost algorithm step-by-step

Training Dataset Setup

  • Super simple dataset with 3 people
  • Features: height measurements, favorite color, gender
  • Target: weight
  • n = 3 samples total
  • Data notation: x_i (feature rows), y_i (weights), i goes from 1 to n

Loss Function Selection

  • Uses differentiable loss function: 1/2 * (observed - predicted)²
  • Same as linear regression loss function (without 1/2 multiplier)
  • 1/2 factor chosen for mathematical convenience
  • When differentiated: derivative = -(observed - predicted) = negative residual
  • Makes math easier since gradient boost uses derivatives frequently
  • Residuals = differences between observed and predicted weights

Step 1: Model Initialization

  • Initialize model with constant value F₀(x)
  • Constant determined by arg min over γ of sum of loss functions
  • Minimize sum of squared residuals/2 across all samples
  • Solution: γ = average of observed weights
  • F₀(x) = 73.3 (average weight)
  • Initial prediction: leaf predicting all samples weigh 73.3

Step 2: Tree Building Loop (M iterations)

  • Generic: build M trees (typically M = 100 in practice)
  • Little m = individual tree index (1 to M)
  • Starting with m = 1 (first tree)

Step 2A: Calculate Pseudo-Residuals

  • r_{i,m} = negative gradient of loss function
  • r_{i,m} = observed - predicted = y_i - F_{m-1}(x_i)
  • For m=1: use F₀(x) = 73.3 as previous prediction
  • Calculate residuals for all samples:
    • r_{1,1} = 88 - 73.3 = 14.7
    • r_{2,1} = 76 - 73.3 = 2.7
    • r_{3,1} = 66 - 73.3 = -7.3
  • Called "pseudo-residuals" because with other loss functions wouldn't be exact residuals

Step 2B: Fit Regression Tree to Residuals

  • Build regression tree using height, color, gender to predict residuals (not weights)
  • Creates tree/stump with decision rule
  • Example: if height ≤ 1.55 → left leaf, else → right leaf
  • Sample x₃ → left leaf (residual -7.3)
  • Samples x₁, x₂ → right leaf (residuals 14.7, 2.7)
  • Create terminal regions R_{j,m} (leaves labeled R₁,₁ and R₂,₁)

Step 2C: Calculate Output Values for Each Leaf

  • For each leaf, compute γ_{j,m} that minimizes sum of loss functions
  • Only include samples that end up in that specific leaf
  • Left leaf R₁,₁ (only x₃):
    • Minimize: 1/2 * (66 - (73.3 + γ))²
    • Solution: γ₁,₁ = -7.3
  • Right leaf R₂,₁ (x₁ and x₂):
    • Minimize sum over both samples
    • Solution: γ₂,₁ = 8.7 (average of residuals in leaf)
  • Output values always equal average of residuals in each leaf

Step 2D: Make New Predictions

  • F_m(x) = F_{m-1}(x) + ν * Σ γ{j,m} * I(x ∈ R{j,m})
  • ν = learning rate (set to 0.1)
  • Small learning rate reduces each tree's effect, improves long-term accuracy
  • New predictions F₁(x):
    • x₁: 73.3 + 0.1 * 8.7 = 74.2
    • x₂: 73.3 + 0.1 * 8.7 = 74.2
    • x₃: 73.3 + 0.1 * (-7.3) = 71.6
  • All predictions improved (closer to observed weights)

Iteration Process

  • Repeat Step 2 for m = 2, 3, ..., M
  • Each iteration:
    • Calculate new residuals using latest predictions
    • Build new tree to predict residuals
    • Calculate leaf output values
    • Update predictions
  • Assuming M = 2 for demonstration (practice uses M ≥ 100)

Step 3: Final Output

  • Final model: F_M(x) = F₂(x) after M iterations
  • For new data prediction:
    • Start with initial leaf value (73.3)
    • Add contributions from all trees scaled by learning rate
    • Example prediction: 73.3 + 0.1(-7.3) + 0.1(-15.6) = 70 kg

Important Notes

  • Tutorial uses stumps due to small dataset (only 3 samples)
  • Gradient boost typically uses larger trees in practice
  • Next tutorial will cover gradient boost for classification

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome