AI summary by TwinMind
ROC and AUC, Clearly Explained!
StatQuest with Josh Starmer
16m 17s
Introduction and Prerequisites
- StatQuest tutorial on ROC (Receiver Operating Characteristic) curves and AUC (Area Under Curve)
- Builds on confusion matrix, sensitivity, and specificity concepts
- Example uses logistic regression but applies to other classification methods
- Presenter: Josh Starmer
Dataset and Logistic Regression Setup
- Dataset: mouse weight vs obesity classification
- Y-axis: two categories (obese/not obese)
- Blue dots: obese mice, Red dots: not obese mice
- X-axis: weight measurements
- Logistic regression curve fitted to convert to probability of obesity
- Heavy mice: high probability of obesity
- Light mice: low probability of obesity
Classification Thresholds
- Need threshold to convert probabilities to classifications
- Standard threshold: 0.5
- Probability > 0.5 → classified as obese
- Probability ≤ 0.5 → classified as not obese
Evaluation with Test Data
- Test set: 4 known non-obese mice + 4 known obese mice
- Threshold 0.5 results:
- 3 correctly classified as obese
- 1 false positive (predicted obese, actually not obese)
- 3 correctly classified as not obese
- 1 false negative (predicted not obese, actually obese)
Alternative Thresholds
- Threshold 0.1 (lower):
- Correctly classifies all obese mice
- Increases false positives
- Reduces false negatives
- Use case: disease detection (e.g., Ebola) where missing positive cases is critical
- Threshold 0.9 (higher):
- Same number of correctly classified obese samples as 0.5
- No false positives
- One additional correctly classified non-obese sample
- Same false negatives as threshold 0.5
ROC Curve Construction
- Problem: too many possible thresholds create confusing number of confusion matrices
- Solution: ROC graphs summarize all threshold information
- Y-axis: True Positive Rate (TPR) = sensitivity = TP/(TP+FN)
- Proportion of obese samples correctly classified
- X-axis: False Positive Rate (FPR) = 1-specificity = FP/(FP+TN)
- Proportion of non-obese samples incorrectly classified as obese
Step-by-Step ROC Construction
- Start with threshold classifying all samples as obese:
- TPR = 4/4 = 1.0
- FPR = 4/4 = 1.0
- Plot point at (1,1)
- Progressively increase threshold, calculate TPR/FPR for each:
- Point at (0.75, 1)
- Point at (0.5, 1)
- Point at (0.25, 1)
- Point at (0, 0.75)
- Point at (0, 0.5)
- Point at (0, 0.25)
- End with point at (0,0) - all classified as not obese
- Green diagonal line: TPR = FPR
- Points left of diagonal: better performance
- Connected points form ROC curve
Area Under Curve (AUC)
- AUC = 0.9 for example curve
- Enables easy comparison between different models
- Higher AUC = better performance
- Red curve (higher AUC) better than blue curve (lower AUC)
- Use case: compare logistic regression vs random forest
Alternative Metrics
- Precision as alternative to False Positive Rate
- Precision = TP/(TP+FP)
- Proportion of positive results correctly classified
- More useful when large imbalance in dataset
- Precision unaffected by number of true negatives
- Common in rare disease studies with many more healthy than diseased samples
Summary
- ROC curves identify optimal thresholds for classification decisions
- AUC helps compare different categorization methods
- Tools for evaluating classification model performance across all possible thresholds
