AI summary by TwinMind

How TikTok dances trained an AI to see

Vox

8m 8s

AI Training Data Analogy

  • Machine learning uses flash card system: question on front, answer on back
  • Ground truth = answers on flash cards that computers can rely on being correct
  • Different data sets for different tasks:
    • Language: sentences for completion
    • Image generation: pictures with definitions
  • More varied and better ground truth = more AI model can learn and test

TikTok Dance Dataset for 3D Vision

  • Researcher Yazamin collected 600 custom selected TikTok videos
  • Challenge: train AI to make 3D images from 2D single viewpoint
  • Started with Render People dataset:
    • Contains video of people + real 3D scans
    • Shows exact spatial positions
    • Insufficient variety/backgrounds for training
  • TikTok videos chosen for diversity, not similarity:
    • Show people clearly in motion
    • Different dances, indoor/outdoor settings
    • Various backgrounds, jackets, shapes
  • Processing method:
    • Used program to remove backgrounds
    • Another program estimated 3D movement in space
    • Model predicts depth, checks work like flipping flash card
  • Successfully created 3D mesh from custom dance video

Mannequin Challenge Dataset

  • Google researchers used 2,000 mannequin challenge videos
  • Advantage: people staying still = multiple camera angles simultaneously
  • Real world messiness built in vs controlled studio setup
  • Used as ground truth for depth perception training
  • Model learns to predict scene appearance from different camera positions
  • Variety in settings, people types, locations

Research Applications

  • Scene geometry learning for photo completion:
    • Paint out parts of images
    • Use scene knowledge to fill back in
    • Mannequin challenge verified accuracy
  • Virtual correspondence matching:
    • Match 3D scene points from different angles
    • Tested on movie scenes and TV shows (Friends)
    • Researcher watched Friends 10+ times, used screenshots
  • Carnegie Mellon comparison dataset:
    • People in camera-filled sphere
    • Perfect 3D scans but narrow/controlled environment
    • Mannequin challenge provided real-world testing

Broader Context

  • Papers built on Facebook 3D dataset (manual face dots)
  • 10,000 real estate videos helped computers learn camera positions
  • Machines learning but humans still teachers
  • AI only as good as training flashcards humans create

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome