AI summary by TwinMind

Lowess and Loess, Clearly Explained!!!

StatQuest with Josh Starmer

10m 10s

Introduction and Background

  • StatQuest educational video from UNC Chapel Hill genetics department
  • Topic: LOESS smoothing (curve fitting to data)
  • Pronunciation uncertainty for "LOESS" - requests viewer comments
  • Previous episode covered linear fitting with least squares
  • Today: extending to curve fitting

Core LOESS Concepts

  • Two main ideas:
    • Use sliding window to divide data into smaller blobs
    • At each data point use weighted least squares to fit line
  • Building on least squares knowledge

Step-by-Step LOESS Process

Window Size 5 Example

  • Start with simple dataset using window size 5
  • First point becomes "focal point" of window
  • 5 points in first window ranked by distance from focal point:
    • Focal point: most weight
    • 1 unit away: second most weight
    • 2 units away: third most weight
    • Furthest point: least weight

Weighted Least Squares Implementation

  • Closer points to focal point = higher weight/influence on line fit
  • Comparison: unweighted vs weighted least squares
    • Weighted version: distant points have less influence, don't pull slope as much
  • Use fitted line to define first point in curve (preliminary draft)

Moving Through Data Points

  • Second point becomes new focal point
  • Window contains 4 closest points
  • May use same window as before if same points are closest
  • Distances and weights change based on new focal point
  • Continue process for each subsequent point

Distance Calculations

  • Only X-axis distances matter for window selection
  • Y-axis distances not factored into point selection at this stage
  • Points equidistant on X-axis get equal initial weights

Outlier Handling and Refinement

Initial Results

  • All preliminary points affected by outliers
  • Outliers pull curve away from true pattern

Additional Weighting System

  • Create second set of weights based on Y-axis distance between old and new points
  • Points close together along Y-axis = high weights
  • Points far apart along Y-axis = low weights
  • Outliers get really low weights

Iterative Refinement

  • Combine two weight sets:
    • Original weights (X-axis distance)
    • New weights (Y-axis distance from fitted points)
  • Repeat entire process with combined weights
  • Results in smoother curve
  • Process can be repeated multiple times until satisfactory smoothness

Advanced Considerations

Line vs Parabola Fitting

  • Choice between fitting lines or parabolas in each window
  • Demonstration with complex dataset:
    • Lines: basic curve fitting
    • Parabolas: slightly better data fit
  • Recommendation: examine original data to decide approach

R Implementation Details

  • lowess() function: fits lines only
  • loess() function: can fit lines or parabolas
    • Default setting: parabola fitting
    • Same algorithm, different defaults
  • loess() allows confidence interval drawing

Window Size Selection

  • Adjustable window size affects fit
  • Examples shown:
    • 1/5 of total points per window
    • 1/3 of total points per window
    • 75% of points (loess default)
  • Usually specify proportion rather than exact point count

Weight Functions

Function Selection

  • Current weight formulas chosen empirically (work well in practice)
  • No physical/biological justification for specific formulas
  • Two weight function graphs shown:
    • X-axis distance weights
    • Y-axis distance weights (old vs new points)
  • Overlapping standard functions show subtle differences
  • Alternative weight functions possible and acceptable

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome