AI summary by TwinMind

StatQuest: Random Forests Part 2: Missing data and clustering

StatQuest with Josh Starmer

11m 53s

Introduction and Dataset Overview

  • Josh Stommer presenting StatQuest on Random Forests Part 2
  • Focus: missing data handling and sample clustering (presenter's favorite aspect)
  • Dataset: 4 patients with some missing data for patient #4
  • Two types of missing data scenarios covered

Missing Data Types in Random Forests

  • Type 1: Missing data in original dataset used to create random forest
  • Type 2: Missing data in new sample requiring categorization
  • Starting with Type 1 scenario

Initial Guess Method for Missing Values

  • Patient #4 missing: blocked arteries status and weight
  • Patient #4 confirmed no heart disease
  • Categorical data (blocked arteries): Use most common value from samples with same outcome
    • Among no heart disease patients: "no" appears 2/2 times
    • Initial guess: "no" for blocked arteries
  • Numeric data (weight): Use median value from same outcome group
    • Median weight for no heart disease patients: 167.5
    • Initial guess: 167.5 for weight

Proximity Matrix and Similarity Determination

  • Step 1: Build random forest
  • Step 2: Run all data through all trees
  • Similarity defined: samples ending in same leaf node are similar
  • Proximity matrix: rows and columns for each sample
    • Increment by 1 when samples end in same leaf node
    • Track across all trees in forest

Proximity Matrix Calculation Process

  • Tree 1: Samples 3 and 4 end in same leaf → add 1 to matrix
  • Tree 2: Samples 2, 3, and 4 end in same leaf → increment relevant pairs
  • Tree 3: Only samples 3 and 4 end in same leaf → increment again
  • Continue for all trees (example uses 10 trees)
  • Final step: Divide each proximity value by total number of trees

Weighted Frequency Calculation for Categorical Data

  • Calculate weighted frequency for "yes" and "no" using proximity values as weights
  • For "yes":
    • Frequency: 1/3 of samples
    • Weight: proximity for "yes" samples ÷ sum of all proximities = 0.1
    • Weighted frequency: 0.03
  • For "no":
    • Frequency: 2/3 of samples
    • Weight: proximity for "no" samples ÷ sum of all proximities = 0.9
    • Weighted frequency: 0.6
  • Result: "no" selected due to higher weighted frequency

Weighted Average Calculation for Numeric Data

  • Use proximities to calculate weighted average for weight
  • Sample 1 weighted contribution: proximity/sum × weight
  • Sample 2 (180 lbs) and Sample 3 (210 lbs) weighted contributions calculated
  • Final weighted average weight: 198.5

Iterative Refinement Process

  • Repeat entire process 6-7 times until convergence
  • Each iteration: rebuild random forest → recalculate proximities → recalculate missing values
  • Stop when missing values no longer change

Distance Matrix and Visualization Applications

  • Proximity matrix before division by tree count
  • Perfect similarity (samples in same leaf all trees) = proximity of 1.0
  • Distance = 1 - proximity value
  • Applications:
    • Heat maps for data visualization
    • MDS (Multidimensional Scaling) plots
    • Works with any data type (ranks, multiple-choice, numeric) if trees can be built

Type 2 Missing Data: New Sample Classification

  • Scenario: Pre-built random forest, new patient needs classification
  • New patient missing blocked arteries status
  • Method:
    • Create two copies: one with heart disease, one without
    • Use iterative method to guess missing values for both copies
    • Run both samples through forest trees
    • Count correct classifications for each option
  • Example results:
    • Heart disease option: correctly labeled in 3/3 trees
    • No heart disease option: correctly labeled in 1/3 trees
    • Select option with more correct classifications

Conclusion and Resources

  • Summary of missing data handling and sample classification
  • References to additional StatQuest content for heat maps and MDS plots
  • Support options: subscriptions, Patreon, merchandise, donations

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome