AI summary by TwinMind
Classification Trees in Python from Start to Finish
StatQuest with Josh Starmer
1h 6m 23s
Introduction and Setup
- StatQuest webinar on decision trees from start to finish in Python
- Using scikit-learn and cost complexity pruning for classification tree
- Dataset: UCI heart disease data (continuous and categorical variables)
- Goal: predict patient heart disease status
- Materials: Jupyter notebook + Python code to be emailed to participants
- Decision trees useful for interpretable ML, especially in medical field where decision rationale must be traceable
Module Imports and Environment Setup
- Python 3 required with specific module versions
- Key imports: pandas (data manipulation), numpy (calculations), matplotlib (visualization), scikit-learn (classification trees, confusion matrices, cross validation)
- Jupyter notebook execution: click cell + play button or Ctrl+Enter
Data Import and Initial Exploration
- Dataset: UCI heart disease repository via pandas.read_csv()
- 303 total rows of patient data
- Features: age, sex, chest pain, resting blood pressure, cholesterol, fasting blood sugar, resting ECG, max heart rate, exercise angina, oldpeak, slope, ca, thal
- Target variable: hd (heart disease levels 0-4)
- Added descriptive column names from UCI website documentation
Missing Data Identification
- Used df.dtypes to check data types
- ca and thal columns showed 'object' type (should be numeric) - indicating mixed data
- Investigated unique values: both contained question marks (?) representing missing data
- ca: blood vessels lit up during diagnostic (values: 0,1,2,3,?)
- thal: thallium heart scan results (values: 3,6,7,?)
Missing Data Handling
- Scikit-learn decision trees don't support missing values
- Options: delete rows or impute values
- Counted missing data: 6 rows out of 303 (2%)
- Decision: remove missing rows (still 297 rows sufficient for training)
- Created df_no_missing dataframe excluding rows with question marks
- Verified removal: ca and thal no longer contain question marks
Data Formatting for Decision Trees
- Split data into X (features) and y (target)
- X: all columns except 'hd'
- y: 'hd' column only
- Reason for splitting after missing data removal: ensures row correspondence between X and y
One-Hot Encoding Theory
- Scikit-learn decision trees support continuous data natively but not categorical
- Problem with treating categorical as continuous: assumes numerical ordering (e.g., chest pain types 1,2,3,4 aren't inherently ordered)
- Solution: one-hot encoding converts categorical columns to multiple binary (0/1) columns
- Each category becomes separate column with 1 for that category, 0 for others
One-Hot Encoding Implementation
- Method: pandas.get_dummies() (chosen for teaching clarity over sklearn.ColumnTransformer)
- Applied to categorical columns with >2 categories: cp, restecg, slope, thal
- Result: chest pain split into cp_1.0, cp_2.0, cp_3.0, cp_4.0 columns
- Binary categorical columns (sex, fbs, exang) left unchanged (already 0/1)
- Final encoded dataset: X_encoded
Target Variable Preprocessing
- Original y contained 5 levels (0=no disease, 1-4=varying degrees)
- Simplified to binary classification: 0=no disease, 1=any disease
- Converted all values >0 to 1 using boolean indexing
- Final y: only 0s and 1s
Preliminary Classification Tree
- Split data: train_test_split() with random_state=42 (70/30 default split)
- Built initial DecisionTreeClassifier() with default parameters
- Fitted on training data
- Visualized: massive, complex tree (likely overfitted)
- Evaluation on test set: 71% accuracy (no disease), 79% accuracy (has disease)
- Baseline performance established for comparison
Cost Complexity Pruning Theory
- Decision trees notorious for overfitting training data
- Many parameters available (max_depth, min_samples_split, etc.)
- Cost complexity pruning simplifies optimization process
- Controls pruning via alpha parameter: higher alpha = more pruning
- Goal: find alpha value that improves test set accuracy
Alpha Parameter Exploration
- Extracted available alpha values using ccp_alphas
- Built trees for each alpha value
- Plotted training vs testing accuracy as function of alpha
- Observation: full tree (alpha=0) best on training, poor on testing
- As alpha increases (more pruning): testing accuracy improves
- Initial estimate: optimal alpha ≈ 0.016
Cross-Validation for Alpha Selection
- Single train/test split may give misleading results
- Demonstrated alpha sensitivity to data splitting
- Used cross-validation to find robust optimal alpha
- Cross-validation showed different optimal value: ~0.014
- Refined search between 0.014-0.015
- Final optimal alpha: 0.01447 (converted from series to float)
Final Optimized Tree
- Built DecisionTreeClassifier with ccp_alpha=0.01447
- Trained on full training dataset
- Evaluation improvements:
- No heart disease: 74% → 81% accuracy
- Has heart disease: 79% → 85% accuracy
- Significantly smaller, more interpretable tree structure
Tree Interpretation
- Each node shows: splitting criterion, Gini impurity, sample count, class distribution, predicted class
- Root node: ca ≤ 0.5 split (222 total samples: 118 no disease, 104 disease)
- Node colors: orange (majority no disease), blue (majority disease)
- Color intensity reflects Gini impurity (darker = lower impurity = better separation)
- Key features identified: ca (blood vessels), thal (thallium scan), oldpeak
- Leaf nodes: final classifications without further splits
Conclusion and Key Learnings
- Complete pipeline: data import → missing data handling → formatting → one-hot encoding → preliminary tree → optimization → final evaluation
- Pruning significantly improved performance and interpretability
- Overfitting analogy: memorizing exam answers vs understanding concepts
- Final tree much smaller yet more accurate than initial complex tree
