AI summary by TwinMind
How TikTok dances trained an AI to see
Vox
8m 8s
AI Training Data Analogy
- Machine learning uses flash card system: question on front, answer on back
- Ground truth = answers on flash cards that computers can rely on being correct
- Different data sets for different tasks:
- Language: sentences for completion
- Image generation: pictures with definitions
- More varied and better ground truth = more AI model can learn and test
TikTok Dance Dataset for 3D Vision
- Researcher Yazamin collected 600 custom selected TikTok videos
- Challenge: train AI to make 3D images from 2D single viewpoint
- Started with Render People dataset:
- Contains video of people + real 3D scans
- Shows exact spatial positions
- Insufficient variety/backgrounds for training
- TikTok videos chosen for diversity, not similarity:
- Show people clearly in motion
- Different dances, indoor/outdoor settings
- Various backgrounds, jackets, shapes
- Processing method:
- Used program to remove backgrounds
- Another program estimated 3D movement in space
- Model predicts depth, checks work like flipping flash card
- Successfully created 3D mesh from custom dance video
Mannequin Challenge Dataset
- Google researchers used 2,000 mannequin challenge videos
- Advantage: people staying still = multiple camera angles simultaneously
- Real world messiness built in vs controlled studio setup
- Used as ground truth for depth perception training
- Model learns to predict scene appearance from different camera positions
- Variety in settings, people types, locations
Research Applications
- Scene geometry learning for photo completion:
- Paint out parts of images
- Use scene knowledge to fill back in
- Mannequin challenge verified accuracy
- Virtual correspondence matching:
- Match 3D scene points from different angles
- Tested on movie scenes and TV shows (Friends)
- Researcher watched Friends 10+ times, used screenshots
- Carnegie Mellon comparison dataset:
- People in camera-filled sphere
- Perfect 3D scans but narrow/controlled environment
- Mannequin challenge provided real-world testing
Broader Context
- Papers built on Facebook 3D dataset (manual face dots)
- 10,000 real estate videos helped computers learn camera positions
- Machines learning but humans still teachers
- AI only as good as training flashcards humans create
