AI summary by TwinMind

Regression Trees, Clearly Explained!!!

StatQuest with Josh Starmer

22m 33s

Introduction and Prerequisites

  • Regression trees tutorial by Josh Starman from StatQuest
  • Assumes familiarity with:
    • Bias-variance tradeoff in machine learning
    • Basic decision tree concepts
    • Basic regression concepts

Drug Dosage Example - Linear vs Non-Linear Data

  • Hypothetical drug study for common cold cure
  • Clinical trial measuring effectiveness vs dosage
  • Linear case: Higher dose = more effective, straight line fits well
    • 27mg dose → 62% effectiveness prediction
  • Non-linear case: Complex relationship
    • Low dosages: not effective
    • Moderate dosages: work really well
    • Higher dosages: ~50% effectiveness
    • High dosages: not effective at all
    • Straight line prediction: 20mg dose → 45% (incorrect, should be 100%)

Regression Trees vs Classification Trees

  • Regression trees: leaves contain numeric values
  • Classification trees: leaves contain true/false or discrete categories
  • Better suited for non-linear relationships than straight lines

Single Predictor Regression Tree Structure

  • Root question: dosage < 14.5?
  • Left branch (dosage < 14.5):
    • 6 observations in training data
    • Average effectiveness: 4.2%
    • Prediction: 4.2%
  • Right branch splits further:
    • If dosage ≥ 29: 4 observations, average 2.5% → prediction 2.5%
    • If 23.5 ≤ dosage < 29: 5 observations, average 52.8% → prediction 52.8%
    • If 14.5 ≤ dosage < 23.5: 4 observations, average 100% → prediction 100%

Multi-Predictor Capability

  • Trees easily handle 3+ predictors (dosage, age, sex)
  • Example prediction for patient:
    • Age > 50? No → right branch
    • Dosage ≥ 29? No → right branch
    • Female? Yes → left branch
    • Prediction: 100% (actual: 98%)
  • Graphs become impossible with 3+ dimensions, trees remain manageable

Building Regression Trees from Scratch

Threshold Selection Process

  • Start with smallest dosages, calculate average (3)
  • Build simple tree: dosage < 3
    • Left leaf: 0% (1 observation)
    • Right leaf: 38.8% (all other observations)
  • Calculate sum of squared residuals for each threshold
  • Test all possible thresholds between consecutive dosage values
  • Optimal threshold: dosage < 14.5 (smallest sum of squared residuals)

Sum of Squared Residuals Calculation

  • For each observation: (observed - predicted)²
  • Sum all squared residuals
  • Example with dosage < 3 threshold: 27,468.5
  • Dosage < 14.5 had lowest sum of squared residuals

Tree Building Algorithm

Left Side Processing (dosage < 14.5)

  • 6 observations total
  • Could theoretically split further
  • Overfitting problem: perfect predictions on training data
    • Individual predictions: 20% → 20%, 5% → 5%, four 0% → 0%
    • Zero bias but potentially large variance

Preventing Overfitting

  • Minimum observations rule: typically 20, example uses 7
  • Since left side has only 6 observations < 7, make it a leaf
  • Output: average of 6 observations = 4.2%

Right Side Processing (dosage ≥ 14.5)

  • 13 observations > 7 minimum → can split
  • Find optimal threshold using sum of squared residuals
  • Split at dosage = 29
  • Right sub-branch (dosage ≥ 29): 4 observations < 7 → leaf with 2.5%
  • Left sub-branch (14.5 ≤ dosage < 29): 9 observations → can split again
  • Final split at dosage = 23.5
  • Final leaves:
    • 14.5 ≤ dosage < 23.5: 100%
    • 23.5 ≤ dosage < 29: 52.8%

Multi-Predictor Tree Building

Candidate Selection Process

  • Step 1: Find best dosage threshold → candidate for root
  • Step 2: Find best age threshold → candidate for root
  • Step 3: Find best sex threshold (only one possible) → candidate for root
  • Step 4: Compare sum of squared residuals for all candidates
  • Winner: age > 50 (lowest sum of squared residuals) becomes root

Tree Growth Process

  • Continue same process for each branch
  • Compare optimal thresholds across all predictors
  • Stop when node has < minimum observations (7 in example, typically 20)
  • Each leaf outputs average value for observations in that cluster

Key Algorithm Summary

  • Objective: Minimize sum of squared residuals at each split
  • Process: Try all possible thresholds for each predictor
  • Selection: Choose threshold/predictor combination with lowest sum of squared residuals
  • Stopping criteria: Node has fewer than minimum observations
  • Output: Average of observations in each leaf

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome