AI summary by TwinMind

Why Dividing By N Underestimates the Variance

StatQuest with Josh Starmer

17m 14s

Introduction and Context

  • Josh Starmer presents StatQuest on why dividing by n underestimates variance
  • Assumes prior knowledge of population parameter estimation
  • Builds on previous StatQuest covering mean, variance, and standard deviation estimation
  • References gene X measurement in liver cells example from previous episode

Population vs Sample Statistics Review

  • Population mean μ = sum of measurements / number of measurements = average measurement
  • Population variance = average of squared distances between data and population mean
  • Population standard deviation = square root of variance
  • Estimated population mean x̄ = sum of measurements / number of measurements
  • Estimated population variance = sum of squared differences / (n-1)
  • Estimated standard deviation = square root of estimated variance
  • Key point: dividing by (n-1) compensates for using sample mean instead of population mean

First Example Demonstration

  • Used 5 measurements to demonstrate variance calculation around different points
  • Calculated variance around zero: 391
  • Calculated variance around point 5: 240
  • Showed variance around sample mean vs population mean
  • Key finding: variance around sample mean < variance around population mean when dividing by n
  • Conclusion: using sample mean with division by n underestimates population variance

Second Example Confirmation

  • Collected 5 new measurements from same population
  • Calculated variance around zero: 616
  • Repeated process with multiple points
  • Confirmed same pattern: minimum variance occurs at sample mean
  • Again showed sample mean division by n underestimates population variance

Mathematical Proof Setup

  • Replaced x̄ with unknown value V in variance formula
  • Created general formula for variance around any point V
  • Used calculus to find minimum variance point

Derivative Calculation

  • Applied chain rule to find derivative of variance formula with respect to V
  • Derivative = -2/n × Σ(Xi - V)
  • Set derivative = 0 to find minimum variance point
  • Solved for V where slope = 0

Proof with Specific Data

  • Plugged in n=5 and actual measurements
  • Multiplied both sides by 5/(-2) to solve for V
  • Result: V = sample mean x̄ = 17.6
  • Confirmed minimum variance occurs when V = sample mean

Proof with General 5 Measurements

  • Replaced specific data with unknown values X₁, X₂, X₃, X₄, X₅
  • Applied same derivative process
  • Result: V = average of 5 measurements = sample mean x̄
  • Conclusion: regardless of specific measurements, minimum variance at sample mean

Proof for Any Sample Size

  • Extended proof to n measurements (X₁ through Xₙ)
  • Derivative = -2/n × Σ(Xi - V)
  • Set equal to zero and solved for V
  • Result: V = sample mean x̄ for any sample size
  • Universal conclusion: sample mean always minimizes variance

Why Squaring vs Absolute Value

  • Explained choice of squared differences over absolute value
  • Squared differences create smooth curve suitable for calculus
  • Absolute value creates sharp angle at minimum, making derivative undefined
  • Calculus approach much easier with squared terms

Final Summary and Implications

  • Division by n underestimates population variance
  • Sample mean differences consistently smaller than population mean differences
  • Population mean differences result in larger average (what we're estimating)
  • Therefore: use (n-1) for population variance estimation
  • Teaser: future StatQuest on expected values will explain why (n-1) specifically, not (n-0.5) or (n-2)

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome