AI summary by TwinMind

Classification Trees in Python from Start to Finish

StatQuest with Josh Starmer

1h 6m 23s

Introduction and Setup

  • StatQuest webinar on decision trees from start to finish in Python
  • Using scikit-learn and cost complexity pruning for classification tree
  • Dataset: UCI heart disease data (continuous and categorical variables)
  • Goal: predict patient heart disease status
  • Materials: Jupyter notebook + Python code to be emailed to participants
  • Decision trees useful for interpretable ML, especially in medical field where decision rationale must be traceable

Module Imports and Environment Setup

  • Python 3 required with specific module versions
  • Key imports: pandas (data manipulation), numpy (calculations), matplotlib (visualization), scikit-learn (classification trees, confusion matrices, cross validation)
  • Jupyter notebook execution: click cell + play button or Ctrl+Enter

Data Import and Initial Exploration

  • Dataset: UCI heart disease repository via pandas.read_csv()
  • 303 total rows of patient data
  • Features: age, sex, chest pain, resting blood pressure, cholesterol, fasting blood sugar, resting ECG, max heart rate, exercise angina, oldpeak, slope, ca, thal
  • Target variable: hd (heart disease levels 0-4)
  • Added descriptive column names from UCI website documentation

Missing Data Identification

  • Used df.dtypes to check data types
  • ca and thal columns showed 'object' type (should be numeric) - indicating mixed data
  • Investigated unique values: both contained question marks (?) representing missing data
  • ca: blood vessels lit up during diagnostic (values: 0,1,2,3,?)
  • thal: thallium heart scan results (values: 3,6,7,?)

Missing Data Handling

  • Scikit-learn decision trees don't support missing values
  • Options: delete rows or impute values
  • Counted missing data: 6 rows out of 303 (2%)
  • Decision: remove missing rows (still 297 rows sufficient for training)
  • Created df_no_missing dataframe excluding rows with question marks
  • Verified removal: ca and thal no longer contain question marks

Data Formatting for Decision Trees

  • Split data into X (features) and y (target)
  • X: all columns except 'hd'
  • y: 'hd' column only
  • Reason for splitting after missing data removal: ensures row correspondence between X and y

One-Hot Encoding Theory

  • Scikit-learn decision trees support continuous data natively but not categorical
  • Problem with treating categorical as continuous: assumes numerical ordering (e.g., chest pain types 1,2,3,4 aren't inherently ordered)
  • Solution: one-hot encoding converts categorical columns to multiple binary (0/1) columns
  • Each category becomes separate column with 1 for that category, 0 for others

One-Hot Encoding Implementation

  • Method: pandas.get_dummies() (chosen for teaching clarity over sklearn.ColumnTransformer)
  • Applied to categorical columns with >2 categories: cp, restecg, slope, thal
  • Result: chest pain split into cp_1.0, cp_2.0, cp_3.0, cp_4.0 columns
  • Binary categorical columns (sex, fbs, exang) left unchanged (already 0/1)
  • Final encoded dataset: X_encoded

Target Variable Preprocessing

  • Original y contained 5 levels (0=no disease, 1-4=varying degrees)
  • Simplified to binary classification: 0=no disease, 1=any disease
  • Converted all values >0 to 1 using boolean indexing
  • Final y: only 0s and 1s

Preliminary Classification Tree

  • Split data: train_test_split() with random_state=42 (70/30 default split)
  • Built initial DecisionTreeClassifier() with default parameters
  • Fitted on training data
  • Visualized: massive, complex tree (likely overfitted)
  • Evaluation on test set: 71% accuracy (no disease), 79% accuracy (has disease)
  • Baseline performance established for comparison

Cost Complexity Pruning Theory

  • Decision trees notorious for overfitting training data
  • Many parameters available (max_depth, min_samples_split, etc.)
  • Cost complexity pruning simplifies optimization process
  • Controls pruning via alpha parameter: higher alpha = more pruning
  • Goal: find alpha value that improves test set accuracy

Alpha Parameter Exploration

  • Extracted available alpha values using ccp_alphas
  • Built trees for each alpha value
  • Plotted training vs testing accuracy as function of alpha
  • Observation: full tree (alpha=0) best on training, poor on testing
  • As alpha increases (more pruning): testing accuracy improves
  • Initial estimate: optimal alpha ≈ 0.016

Cross-Validation for Alpha Selection

  • Single train/test split may give misleading results
  • Demonstrated alpha sensitivity to data splitting
  • Used cross-validation to find robust optimal alpha
  • Cross-validation showed different optimal value: ~0.014
  • Refined search between 0.014-0.015
  • Final optimal alpha: 0.01447 (converted from series to float)

Final Optimized Tree

  • Built DecisionTreeClassifier with ccp_alpha=0.01447
  • Trained on full training dataset
  • Evaluation improvements:
    • No heart disease: 74% → 81% accuracy
    • Has heart disease: 79% → 85% accuracy
  • Significantly smaller, more interpretable tree structure

Tree Interpretation

  • Each node shows: splitting criterion, Gini impurity, sample count, class distribution, predicted class
  • Root node: ca ≤ 0.5 split (222 total samples: 118 no disease, 104 disease)
  • Node colors: orange (majority no disease), blue (majority disease)
  • Color intensity reflects Gini impurity (darker = lower impurity = better separation)
  • Key features identified: ca (blood vessels), thal (thallium scan), oldpeak
  • Leaf nodes: final classifications without further splits

Conclusion and Key Learnings

  • Complete pipeline: data import → missing data handling → formatting → one-hot encoding → preliminary tree → optimization → final evaluation
  • Pruning significantly improved performance and interpretability
  • Overfitting analogy: memorizing exam answers vs understanding concepts
  • Final tree much smaller yet more accurate than initial complex tree

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome