AI summary by TwinMind
UMAP Dimension Reduction, Main Ideas!!!
StatQuest with Josh Starmer
18m 51s
Introduction to UMAP
- UMAP (Uniform Manifold Approximation and Projection) takes high-dimensional data that can't be graphed and helps visualize it
- Takes data with 3+ features and outputs low-dimensional graph for easy visualization
- Popular because relatively fast with large datasets and similar samples cluster together
- Useful for identifying similarities and outliers
Problem with High-Dimensional Data Visualization
- 2D graphs work for weight/height data
- Adding 3rd dimension (age) creates awkward 3D visualization
- 4+ dimensions impossible to visualize on 2D screens
- PCA alternative but only works when first two principal components account for most variation
- Complex datasets require alternative to PCA
UMAP vs Simple Projection
- Simple projection onto x-axis or y-axis creates mish-mash of points, loses cluster structure
- UMAP calculates similarity scores to identify clustered points
- Preserves clustering in low-dimensional output
High-Dimensional Similarity Score Calculation
- Calculate distances between all pairs of high-dimensional points
- For each point, create graph showing distances to all other points
- Draw curve over data to calculate similarity scores
- Curve shape depends on number of high-dimensional neighbors parameter
- Common default: 15 neighbors
- Example uses 3 neighbors (includes point itself, so 2 actual neighbors)
- Mathematical approach: log base 2 of number of neighbors
- 3 neighbors → log₂(3) = 1.6
- Curve shaped so nearest neighbors' y-axis coordinates sum to target (1.6)
- Points far away get similarity scores near zero
Asymmetric Similarity Scores
- Similarity score for C relative to B (0.6) differs from B relative to C (1.0)
- Different curves for each point ensure every point similar to at least one other
- UMAP makes scores symmetrical using averaging method
- Process repeated for all points in dataset
Low-Dimensional Graph Optimization
- Initialize low-dimensional graph (often not ideal initially)
- Iterative improvement process:
- Randomly select pair of points to move closer (proportional to high-dimensional scores)
- Higher probability for higher-scoring pairs
- Flip coin to decide which point moves
- Select point from different cluster to move away from (equal probability)
- Low-dimensional similarity scores use fixed bell-shaped curve from t-distribution
- T-distribution similar to Gaussian but shorter with fatter tails
- All low-dimensional curves same size
Movement Strategy
- Maximize low-dimensional score for points that should be close
- Minimize low-dimensional score for points that should be far
- Small incremental movements for better optimization with large datasets
- Continue until clusters match high-dimensional structure
UMAP vs t-SNE Comparison
- Main ideas essentially the same with subtle differences
- Two major differences:
- Initialization: t-SNE uses random initialization (different each run), UMAP uses spectral embedding (consistent initialization)
- Point movement: t-SNE moves every point each iteration, UMAP can move subset of points (better scaling for large datasets)
Number of Neighbors Parameter Impact
- Low number of neighbors: small independent clusters (details, not big picture)
- High number of neighbors: big picture view with less detail
- Worth experimenting with different values for specific datasets
- Key user-adjustable parameter affecting results
Practical Considerations
- Lightning and Grid sponsorship for model building and cloud training
- Mathematical details available in follow-up quest
- Parameter tuning important for optimal results
