AI summary by TwinMind

ROC and AUC in R

StatQuest with Josh Starmer

15m 12s

Introduction and Setup

  • Tutorial on drawing ROC graphs and calculating AUC in R using pROC library
  • Builds on previous ROC and AUC StatQuest content
  • Required packages: pROC, randomForest
  • Installation: install.packages("pROC"), install.packages("randomForest")
  • Set random seed to 420 for reproducible results

Dataset Creation

  • Generate 100 sample dataset (vs 8 in previous example)
  • num.samples = 100
  • Weight variable: rnorm(100, mean=172, sd=29) - average man weight data
  • Sort weights from low to high
  • Obesity classification using rank function:
    • Rank weights 1-100 (lightest to heaviest)
    • Scale ranks by dividing by 100 (0.01 to 1.0)
    • Compare scaled ranks to random numbers 0-1
    • If random < scaled rank → obese (1), else not obese (0)
  • Results: lighter samples mostly 0s, heavier samples mostly 1s

Logistic Regression Model

  • Use GLM function to fit logistic regression curve
  • Store results in glm.fit variable
  • glm.fit$fitted.values contains estimated probabilities each sample is obese
  • Plot data with fitted curve showing predicted obesity probability

Basic ROC Curve Generation

  • Use ROC function from pROC library
  • Input: known classifications (obese/not obese) + estimated probabilities
  • Set plot=TRUE to draw graph, not just calculate numbers
  • Output includes:
    • Sample counts (not obese=0, obese=1)
    • Area Under Curve (AUC) value
  • Default: specificity on x-axis (1 to 0), sensitivity on y-axis

ROC Graph Customization

  • Remove ugly padding: par(pty="s") for square plot type
  • Alternative x-axis: legacy.axes=TRUE for 1-specificity instead of specificity
  • Percentage format: percent=TRUE for percentage vs 0-1 values
  • Custom axis labels: xlab="False Positive Percentage", ylab="True Positive Percentage"
  • Line customization: col=RGB values from ColorBrewer, lwd for thickness

Threshold Analysis

  • Save ROC calculations in variable to access thresholds
  • Create data frame with:
    • True Positive Percentages (sensitivities × 100)
    • False Positive Percentages ((1-specificities) × 100)
    • Thresholds
  • head() shows threshold=-Inf → TPP=100%, FPP=100% (upper right corner)
  • tail() shows threshold=+Inf → TPP=0%, FPP=0% (bottom left corner)
  • Filter thresholds for specific TPP ranges (e.g., 60-80%)

Partial AUC Calculation

  • Focus on specific ROC curve regions (low false positive rates)
  • Parameters:
    • print.auc=TRUE
    • print.auc.x=45 (x-axis position for text)
    • partial.auc=c(100,90) (specificity range 100%-90%)
    • auc.polygon=TRUE to draw shaded area
    • auc.polygon.col with RGB + transparency values

Overlapping ROC Curves Comparison

  • Create random forest classifier with same dataset
  • Draw original logistic regression ROC curve
  • Add second ROC curve using plot.roc function:
    • Random forest input: number of trees voting correctly
    • Different color (green vs blue RGB values)
    • add=TRUE to overlay on existing graph
    • print.auc.y=40 to position AUC text below first curve
  • Add legend in bottom right corner
  • Reset graphical parameters: par(pty="m") after completion

Technical Notes

  • Specificity range values based on specificity, not 1-specificity
  • 100% specificity = 0% on 1-specificity axis
  • 90% specificity = 10% on 1-specificity axis
  • ColorBrewer website for RGB color values
  • Semi-transparent colors: add two digits to RGB values

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome