AI summary by TwinMind

How to Prune Regression Trees, Clearly Explained!!!

StatQuest with Josh Starmer

16m 15s

Introduction and Prerequisites

  • Host: Josh Starmer, StatQuest
  • Topic: Cost complexity pruning (aka weakest link pruning) for regression trees
  • Prerequisites: familiarity with regression trees and cross-validation
  • Purpose: prevent overfitting to improve performance on testing data

Overfitting Problem Example

  • Dataset: drug dosage (x-axis) vs effectiveness (y-axis)
  • Pattern: low/high dosages ineffective, medium dosages very effective, moderately high dosages moderately effective
  • Full regression tree fits training data well (each leaf = average effectiveness from observation clusters)
  • Testing data analysis reveals overfitting:
    • Some predictions close to actual values (small residuals)
    • Four observations with 100% effectiveness appear as outliers
    • Large residuals indicate overfitting to training data

Pruning Solution Concept

  • Remove leaves and replace splits with leaf averaging larger observation groups
  • Example: observations between 14.5-29 go to single leaf on far right
  • Trade-off: worse fit to training data but better performance on testing data
  • Progressive pruning possible: can remove multiple leaf pairs sequentially

Cost Complexity Pruning Methodology

Step 1: Calculate Sum of Squared Residuals (SSR)

  • Original full-sized tree SSR breakdown:
    • Dosages < 14.5: SSR = 320.8
    • Dosages ≥ 29: SSR = 75
    • Dosages ≥ 23 and < 29: SSR = 148.8
    • Dosages ≥ 14.5 and < 23.5: SSR = 0
    • Total SSR = 543.8
  • Pruned trees SSR calculations:
    • 3-leaf subtree: SSR = 5,494.8
    • 2-leaf subtree: SSR = 19,243.7
    • 1-leaf subtree: SSR = 28,897.2

Step 2: Tree Scoring Formula

  • Tree Score = SSR + α × T
  • Where: α = tuning parameter, T = number of leaves
  • Tree complexity penalty = α × T (compensates for leaf count differences)

Step 3: Example with α = 10,000

  • Full tree (4 leaves): Score = 543.8 + 10,000 × 4 = 40,543.8
  • 3-leaf subtree: Score = 5,494.8 + 10,000 × 3 = 35,494.8 (lowest score)
  • 2-leaf subtree: Score = 19,243.7 + 10,000 × 2 = 39,243.7
  • 1-leaf subtree: Score = 28,897.2 + 10,000 × 1 = 38,897.2

Step 4: Alpha Impact Analysis

  • With α = 22,000: 1-leaf subtree has lowest score
  • Alpha value determines optimal tree selection

Complete Pruning Process

Phase 1: Build Full Tree and Find Alpha Values

  • Use all data to build full-sized regression tree
  • α = 0: full tree has lowest score (no complexity penalty)
  • Incrementally increase α to find pruning thresholds:
    • α = 10,000: prune to 3-leaf subtree
    • α = 15,000: prune to 2-leaf subtree
    • α = 22,000: prune to 1-leaf subtree
  • Creates sequence from full tree to single leaf

Phase 2: Cross-Validation Process

  • Split full dataset into training/testing sets
  • Using training data only, build tree sequence with found α values
  • Calculate SSR for each tree using testing data
  • Example results: tree with α = 10,000 had smallest testing SSR
  • Repeat with new training/testing splits
  • Continue until 10-fold cross-validation complete

Phase 3: Final Selection

  • Identify α value with lowest average testing SSR across all folds
  • Example: α = 10,000 produced optimal results on average
  • Return to original full-data trees, select tree corresponding to optimal α
  • This becomes final pruned regression tree

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome