Summary by TwinMind · 15m 12s

Summary by TwinMind · 15m 12s

ROC and AUC in R

ROC and AUC in R

StatQuest with Josh Starmer

StatQuest with Josh Starmer

Introduction and Setup

  • Tutorial on drawing ROC graphs and calculating AUC in R using pROC library
  • Builds on previous ROC and AUC StatQuest content
  • Required packages: pROC, randomForest
  • Installation: install.packages("pROC"), install.packages("randomForest")
  • Set random seed to 420 for reproducible results

Dataset Creation

  • Generate 100 sample dataset (vs 8 in previous example)
  • num.samples = 100
  • Weight variable: rnorm(100, mean=172, sd=29) - average man weight data
  • Sort weights from low to high
  • Obesity classification using rank function:
    • Rank weights 1-100 (lightest to heaviest)
    • Scale ranks by dividing by 100 (0.01 to 1.0)
    • Compare scaled ranks to random numbers 0-1
    • If random < scaled rank → obese (1), else not obese (0)
  • Results: lighter samples mostly 0s, heavier samples mostly 1s

Logistic Regression Model

  • Use GLM function to fit logistic regression curve
  • Store results in glm.fit variable
  • glm.fit$fitted.values contains estimated probabilities each sample is obese
  • Plot data with fitted curve showing predicted obesity probability

Basic ROC Curve Generation

  • Use ROC function from pROC library
  • Input: known classifications (obese/not obese) + estimated probabilities
  • Set plot=TRUE to draw graph, not just calculate numbers
  • Output includes:
    • Sample counts (not obese=0, obese=1)
    • Area Under Curve (AUC) value
  • Default: specificity on x-axis (1 to 0), sensitivity on y-axis

ROC Graph Customization

  • Remove ugly padding: par(pty="s") for square plot type
  • Alternative x-axis: legacy.axes=TRUE for 1-specificity instead of specificity
  • Percentage format: percent=TRUE for percentage vs 0-1 values
  • Custom axis labels: xlab="False Positive Percentage", ylab="True Positive Percentage"
  • Line customization: col=RGB values from ColorBrewer, lwd for thickness

Threshold Analysis

  • Save ROC calculations in variable to access thresholds
  • Create data frame with:
    • True Positive Percentages (sensitivities × 100)
    • False Positive Percentages ((1-specificities) × 100)
    • Thresholds
  • head() shows threshold=-Inf → TPP=100%, FPP=100% (upper right corner)
  • tail() shows threshold=+Inf → TPP=0%, FPP=0% (bottom left corner)
  • Filter thresholds for specific TPP ranges (e.g., 60-80%)

Partial AUC Calculation

  • Focus on specific ROC curve regions (low false positive rates)
  • Parameters:
    • print.auc=TRUE
    • print.auc.x=45 (x-axis position for text)
    • partial.auc=c(100,90) (specificity range 100%-90%)
    • auc.polygon=TRUE to draw shaded area
    • auc.polygon.col with RGB + transparency values

Overlapping ROC Curves Comparison

  • Create random forest classifier with same dataset
  • Draw original logistic regression ROC curve
  • Add second ROC curve using plot.roc function:
    • Random forest input: number of trees voting correctly
    • Different color (green vs blue RGB values)
    • add=TRUE to overlay on existing graph
    • print.auc.y=40 to position AUC text below first curve
  • Add legend in bottom right corner
  • Reset graphical parameters: par(pty="m") after completion

Technical Notes

  • Specificity range values based on specificity, not 1-specificity
  • 100% specificity = 0% on 1-specificity axis
  • 90% specificity = 10% on 1-specificity axis
  • ColorBrewer website for RGB color values
  • Semi-transparent colors: add two digits to RGB values