AI summary by TwinMind

Decision and Classification Trees, Clearly Explained!!!

StatQuest with Josh Starmer

18m 8s

Introduction to Decision Trees

  • Decision trees make statements and decisions based on true/false outcomes
  • Two main types:
    • Classification trees: classify things into categories
    • Regression trees: predict numeric values (e.g., using diet to predict mouse size)
  • Video focuses on classification trees
  • Trees can combine numeric and yes/no data types
  • Numeric thresholds can vary for same data type
  • Final classifications can be repeated

Tree Navigation and Terminology

  • Navigation: start at top, work down until reaching endpoint
  • Arrow conventions: true = left, false = right (sometimes unlabeled)
  • Tree structure terminology:
    • Root node: very top of tree
    • Internal nodes/branches: have arrows pointing to and from them
    • Leaf nodes/leaves: have arrows pointing to them but none pointing away

Building Classification Trees from Data

  • Example dataset: loves popcorn, loves soda, age, loves Cool as Ice (1991 movie)
  • Goal: predict whether someone loves Cool as Ice

Step 1: Select Root Node Feature

  • Test each feature (loves popcorn, loves soda, age) to determine best predictor
  • Run data through simple trees for each feature
  • Track outcomes in leaves (yes/no counts)

Measuring Impurity

  • Pure leaf: contains only one category
  • Impure leaf: contains mixture of categories
  • Gini impurity calculation for leaves:
    • Formula: 1 - (probability of yes)² - (probability of no)²
    • Example: left leaf = 1 - (1/4)² - (3/4)² = 0.375
    • Right leaf = 1 - (2/3)² - (1/3)² = 0.444
  • Total Gini impurity = weighted average of leaf impurities
    • Loves popcorn: 0.405
    • Loves soda: 0.214

Handling Numeric Data (Age)

  • Sort rows by age (lowest to highest)
  • Calculate average age for all adjacent people
  • Test each average as potential threshold
  • Example thresholds: 9.5, 12.5, 15, 44
  • Age thresholds 15 and 44 tied for lowest impurity: 0.343
  • Choose threshold 15

Final Feature Selection

  • Compare Gini impurities:
    • Loves popcorn: 0.405
    • Loves soda: 0.214 (lowest)
    • Age: 0.343
  • Select "loves soda" as root node feature

Tree Construction Process

  • Root: loves soda splits data into left (loves soda) and right (doesn't love soda) nodes
  • Left node (4 people): 3 love Cool as Ice, 1 doesn't (impure)
  • Further split left node:
    • Test loves popcorn: Gini impurity = 0.25
    • Test age < 12.5: Gini impurity = 0 (perfect split)
  • Choose age < 12.5 for split (lower impurity)
  • Right node (3 people): becomes leaf (no further splitting needed)

Leaf Output Assignment

  • Output = majority category in each leaf
  • Three leaves with "does not love Cool as Ice" output
  • One leaf with "loves Cool as Ice" output

Making Predictions

  • Example: new person loves soda, age 15
  • Path: loves soda → left → age < 12.5 is false → right leaf
  • Prediction: loves Cool as Ice

Overfitting and Solutions

  • Problem: leaves with very few people (low confidence in predictions)
  • Two main solutions:
    1. Pruning (separate StatQuest video available)
    2. Growth limits (e.g., minimum 3 people per leaf)
  • Impure leaves still need output values (use majority category)
  • Cross-validation used to determine optimal parameters

Alternative Impurity Measures

  • Gini impurity (focus of video)
  • Entropy and information gain
  • All methods numerically similar
  • Gini impurity chosen for being straightforward and popular

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome