AI summary by TwinMind
Mark Zuckerberg: First Interview in the Metaverse | Lex Fridman Podcast #398
Lex Fridman
1h 4m 38s
Codec Avatars Technology Overview
- Photorealistic 3D avatars with spatial audio allowing remote presence feeling
- Captures facial expressions, nuances, micro-movements for emotional communication
- Both participants scanned at Meta facilities for avatar creation
- Computer model built from multiple expressions, collapsed into codec
- Headset tracks face/expressions, sends encoded version over network
- More bandwidth efficient than full 3D video transmission
- Captures imperfections (freckles, wrinkles, asymmetry) for realism
- Eyes crucial for communication - most human interaction non-verbal
Current Limitations & Future Accessibility
- Currently requires lengthy scanning process (hours) at specialized facilities
- Goal: 2-3 minute smartphone scan for same quality
- Working on streamlined phone-based scanning process
- Production scalability main remaining challenge
- Meta headset (existing product) can drive these avatars
- Timeline: rolling out progressively over next few years
Quest 3 Mixed Reality Capabilities
- First mainstream mixed reality device at $500 (vs Quest Pro at $1500)
- Better mixed reality than Quest Pro despite lower cost
- Multiple times better resolution for mixed reality cameras/sensors
- Qualcomm next-gen chipset (custom version)
- 40% thinner, better display, better graphics
- Combines VR experiences with physical world visibility
- Hand tracking without controllers
- Pass-through technology integrates digital objects into physical space
- Launching fall 2023
Mixed Reality Applications & Vision
- Safer VR experiences (see physical surroundings while gaming)
- Remote meetings with some people physical, others as holograms
- Digital objects/screens as holograms in physical space
- Games integrated with real furniture/environment
- Future: normal glasses showing holograms alongside physical objects
- By end of decade: as many holograms as physical objects in rooms
- "Real world" = combination of physical + digital worlds
AI Integration Plans
- AI avatars can join meetings as realistic team members
- Photorealistic avatars provide acceptance bridge to new technology
- People comfortable blending photorealistic and cartoon elements
- Future uncertainty: long-term preference for photorealistic vs expressive avatars
AI Personas & Assistant Development
- Multiple AI types: assistant (Meta AI), business-focused, entertainment
- Snoop Dogg as dungeon master for text-based adventures
- Cultural figures: Jane Austen, Marcus Aurelius personalities
- Creator AI assistants planned for next year
- AI Studio platform for user-generated AI creation
- Business AI for customer service, reservations, purchases
- Challenge: ensuring AI represents creators accurately
Llama 2 & Future AI Models
- Llama 2 exceeded download/usage expectations
- Open sourcing decision validated - benefits outweigh risks
- Priority: integrating Llama 2 into consumer products
- Future foundation models in development
- Will require similar red-teaming process for safety
- Hope to open source next version when ready
Philosophical Implications
- Identity questions: avatar vs physical self representation
- Potential for idealized self-presentation in avatars
- Death/memorialization in digital spaces
- Harm definitions different in digital vs physical worlds
- New forms of anonymity-enabled toxic behavior
- Breaking physics constraints (teleportation, instant presence)
- Consciousness-based reality vs physical space reality
Technical Challenges
- Full body scanning vs face-only optimization
- Compute requirements for real-time rendering
- Temporal accuracy (avatar may not match current appearance)
- Camera positioning vs eye placement for natural perspective
- Real-time AI processing for perspective mapping
- Balancing fidelity with performance requirements
