AI summary by TwinMind
Lowess and Loess, Clearly Explained!!!
StatQuest with Josh Starmer
10m 10s
Introduction and Background
- StatQuest educational video from UNC Chapel Hill genetics department
- Topic: LOESS smoothing (curve fitting to data)
- Pronunciation uncertainty for "LOESS" - requests viewer comments
- Previous episode covered linear fitting with least squares
- Today: extending to curve fitting
Core LOESS Concepts
- Two main ideas:
- Use sliding window to divide data into smaller blobs
- At each data point use weighted least squares to fit line
- Building on least squares knowledge
Step-by-Step LOESS Process
Window Size 5 Example
- Start with simple dataset using window size 5
- First point becomes "focal point" of window
- 5 points in first window ranked by distance from focal point:
- Focal point: most weight
- 1 unit away: second most weight
- 2 units away: third most weight
- Furthest point: least weight
Weighted Least Squares Implementation
- Closer points to focal point = higher weight/influence on line fit
- Comparison: unweighted vs weighted least squares
- Weighted version: distant points have less influence, don't pull slope as much
- Use fitted line to define first point in curve (preliminary draft)
Moving Through Data Points
- Second point becomes new focal point
- Window contains 4 closest points
- May use same window as before if same points are closest
- Distances and weights change based on new focal point
- Continue process for each subsequent point
Distance Calculations
- Only X-axis distances matter for window selection
- Y-axis distances not factored into point selection at this stage
- Points equidistant on X-axis get equal initial weights
Outlier Handling and Refinement
Initial Results
- All preliminary points affected by outliers
- Outliers pull curve away from true pattern
Additional Weighting System
- Create second set of weights based on Y-axis distance between old and new points
- Points close together along Y-axis = high weights
- Points far apart along Y-axis = low weights
- Outliers get really low weights
Iterative Refinement
- Combine two weight sets:
- Original weights (X-axis distance)
- New weights (Y-axis distance from fitted points)
- Repeat entire process with combined weights
- Results in smoother curve
- Process can be repeated multiple times until satisfactory smoothness
Advanced Considerations
Line vs Parabola Fitting
- Choice between fitting lines or parabolas in each window
- Demonstration with complex dataset:
- Lines: basic curve fitting
- Parabolas: slightly better data fit
- Recommendation: examine original data to decide approach
R Implementation Details
lowess()function: fits lines onlyloess()function: can fit lines or parabolas- Default setting: parabola fitting
- Same algorithm, different defaults
loess()allows confidence interval drawing
Window Size Selection
- Adjustable window size affects fit
- Examples shown:
- 1/5 of total points per window
- 1/3 of total points per window
- 75% of points (loess default)
- Usually specify proportion rather than exact point count
Weight Functions
Function Selection
- Current weight formulas chosen empirically (work well in practice)
- No physical/biological justification for specific formulas
- Two weight function graphs shown:
- X-axis distance weights
- Y-axis distance weights (old vs new points)
- Overlapping standard functions show subtle differences
- Alternative weight functions possible and acceptable
