AI summary by TwinMind
Google I/O Afterparty: The Future of Human-AI Collaboration, From Veo to Mariner
Sequoia Capital
53m 51s
Google's AI Momentum and Public Opinion Shift
- Public opinion on Google's AI position flipped dramatically after Google I/O
- Models are primary driver of perception change
- Google now leads many AI leaderboards with state-of-the-art capabilities
- Internal feeling: end of chapter one, start of chapter 2 in AI development
- Three years of consistent improvement in Gen AI finally visible externally
Whisk and Flow: Creative Video Generation (Thomas Iljik)
Development History
- Started 2.5-3 years ago with control net paper enabling model channeling
- LoRAs allowed showing model range of possibilities for iteration
- Trained internal teams on ComfyUI-type workflows
- Created half-episode animation "Not So Super Villain" on YouTube
- Key convictions: creation must be iterative, media comes with blueprint, show-and-tell approach
Product Strategy
- Whisk: consumer-focused visual language tool for remixing
- Flow: high-end filmmaking tool - "generative AI camera" for AI filmmakers
- Target users: AI filmmakers (not traditional filmmakers) - people without $100K budgets
- Flow designed for world-building then shooting inside created worlds
Technical Progress
- VO3 beat VO2 in rankings, major improvement in adherence
- Physics getting better, six-finger problem resolved
- Audio co-generation with video launched - major breakthrough
- Challenges remain: multiple character choreography, cross-scene consistency, output refinement
Future Vision
- Video generation, simulation, games converging into same technology
- World-building approach: define stage, assets, appearance, then shoot
- Show-and-tell interface preferred over pure text prompts
- New formats between static images/videos and full games
- Sharing experiences rather than just content
Mariner: Computer Use Agent (Jaclyn Consulman)
Technical Architecture
- Action-tuned model on recent Gemini version
- Leverages multimodal capabilities for planning and reasoning
- Takes screenshots, understands UI, continues trajectory toward user goals
- Moved from Chrome extension to VM-based background execution
Development Timeline
- Project started last year after Google AI Studio/Gemini API graduated from Labs
- December launch as Chrome extension with trusted testers
- Feedback: "cool but I want my browser back"
- Recent launch: web app with virtual machines for background tasks
User Experience Evolution
- Companion extension provides context from open browser tabs
- Can handle 10 concurrent tasks vs single task limitation
- Users can watch agent in full-screen mode or run in background
- Task summaries provided instead of full conversation logs
- Pause/takeover functionality for user oversight
Current Capabilities and Limitations
- Research prototype with experiment label
- Major improvements from December to current version
- Speed optimization ongoing priority
- Model quality improvements needed
- Application-level improvements for seamless context provision
- Planning and reasoning enhancements required
Future Development
- Three categories: agent intelligence, environment expansion, ecosystem integration
- Agent: smarter models, tool use, memory, context
- Environment: local desktop background operation, cross-device capability
- Ecosystem: agent-to-agent interaction, business model evolution
Notebook LM: Personalized AI Content (Simon Takamina)
Viral Success and Evolution
- Audio overviews went viral unexpectedly
- "Audience of one" content hypothesis - personal rather than mass content
- First months focused on infrastructure scaling (TPU capacity)
- Join mode launched end of last year for podcast interaction
Technical Infrastructure Upgrade
- Moved from research-grade model to native Gemini audio
- International audio overviews launched
- Mobile applications launched Monday
- Collaboration across Labs and GDM teams
Core User Patterns
- Longer-running projects in hobbies and work
- Knowledge workers and students as primary users
- Projects as units of knowledge accumulation
- Context criticality for AI system usefulness
Three Notebook Superpowers
- Information accumulation over time (database technologies)
- Bundled intelligence (Gemini 1.5 Pro, thinking models)
- Adaptive content transformation (audio, visual, interactive formats)
Content Format Innovation
- Beyond two-host podcast format
- LinkedIn feedback analysis use case
- Different show types in development
- Comic book, mind map, short movie adaptations
- 150-page PhD dissertation → comic book example
Mobile-First Innovation
- Desktop-mobile companion experience rather than carbon copy
- Native voice recording for conversations
- Leveraging mobile form factor, sensors, always-on presence
- Novel mobile experience development priority
Cross-Labs Collaboration and Future Predictions
Favorite Unreleased Projects
- Virtual try-on for shopping (Thomas, Simon)
- Stitch for front-end design generation (Jaclyn)
2025 Predictions
- Video as breakout application
- Remixable content generation and sharing
- Show-and-tell interface paradigm adoption
Lessons Learned
- Timing consistently challenging - often too early
- Show-and-tell over pure text prompting validated
- Sticking with LLM technology when business viability questioned
- Inference cost curves enabling new possibilities
- Two-year forward planning approach in Labs
