AI summary by TwinMind
Sierra co-founder Clay Bavor on Making Customer-Facing AI Agents Delightful
Sequoia Capital
1h 12m 31s
Clay Bavor Background and Google Experience
- Bay Area native, 18 years at Google until March 2022
- Started in Search, then ads, ran product/design teams for Workspace (Gmail, Google Docs, Google Drive)
- Last 10 years focused on forward-looking bets: AR/VR, Google Lens, AI applications
- Early exposure to large language models at Google (Mina/Lambda, PaLM 540B parameter model)
- Key moment: PaLM explained 2008 financial crisis using movie references ("like Inception except instead of dreams within dreams it was debt within debt")
- Avocado chairs from DALL-E paper inspired office decor at Sierra
Sierra Founding and Vision
- Co-founded with Brett Taylor (longtime friend from Google APM program) in December 2022
- Monthly poker group connection maintained friendship
- Vision: Enable every company to create branded customer-facing AI agents
- Parallel to historical technology shifts: 90s websites, 2000s social networks, 2015 mobile apps
- Now: conversational AI agents that can understand, generate language, reason, and make decisions
Current Sierra Performance and Customers
- 15 months old company
- Customers: Weight Watchers, Sonos, SiriusXM, Olay
- Resolving 70%+ of incoming customer inquiries
- Customer satisfaction: mid-4s out of 5 stars (often higher than human agents)
- Use cases: points advice, subscription management, radio troubleshooting, product recommendations
Agent OS Technical Architecture
Foundation Layer
- Data governance, PII detection/masking/encryption built-in
- Customer data stays customer data, protected before logging
Agent SDK
- Declarative programming language purpose-built for agents
- Express high-level goals and guardrails
- Define behavior scaffolding for complex tasks
- Example: Healthcare-adjacent customer can discuss products without dispensing medical advice
Runtime System
- Abstracts complexity from developers (define "what" not "how")
- May involve 5-10 separate LLM calls to different models
- Document retrieval for accurate answers
- Equivalent to OS handling IO and resource utilization
Supervisory Architecture and Quality Control
- "Solution to many problems with AI is more AI"
- LLMs better at detecting errors in output than avoiding them initially
- Multiple supervisory agents act as "Jiminy Cricket" oversight
- Check for: factual accuracy, medical/financial advice, prompt injection attacks
- Different personas elicit more discerning behavior from supervisors
TAU Benchmark Research
- Tool Agent User Benchmark - first research paper from Sierra
- Existing benchmarks inadequate for real-world agent performance
- Tests conversational interactions with user simulator agent
- Results: Best LLMs alone achieve only 61% success rate (returns), 35% (airline reservations)
- Pass@K metric: 0.61^8 = ~2.5% reliability over multiple interactions
- Demonstrates need for sophisticated agent architectures beyond raw LLMs
Engineering vs Research Balance
- Primary approach: engineering challenge using composed foundation models
- Fine-tuned open-source models with proprietary datasets
- Supplemented with retrieval systems (RAG) for grounding
- External reasoning scaffolding (planning, task generation, supervisors)
- Can swap in better/specialized models as "IQ upgrades"
- Continuous progress in agent architecture and tooling
Customer Deployment Process
New Software Paradigm
- Programming with English language vs traditional code
- Non-deterministic models vs predictable input/output
- Structured input to messy human language
- Model upgrades change behavior fundamentally
Agent Development Lifecycle
- Declarative programming language for agent definition
- Conversation simulator for testing (hundreds/thousands of steps)
- New approaches to unit/integration testing
- Release management and quality assurance
Deep Customer Partnership
- Solution includes technology + teaching + auditing/improvement
- Deployment team acts as product managers/engineers
- Each agent built as new product for customer
- Work with customer experience + marketing teams
- Define voice, values, vibe (emoji usage varies by brand)
- Extract business processes from institutional knowledge (not documented manuals)
- Handle "policy behind the policy" scenarios
Experience Manager Platform
- Command center for customer experience teams
- Rich analytics on trending/new issues
- Proactive issue detection (shipping depot outages, app crashes 8-10 hours early)
- Biased sampling to surface problem conversations
- Coaching feedback system for continuous improvement
- Test creation for every improvement to prevent regression
Pricing and Business Model
- Resolution-based/outcome-based pricing
- Only charge when customer problem fully solved
- Aligned incentives: Sierra wants high resolution rates, customers want to send more cases
- Fraction of cost compared to 20-minute phone calls (~$12-13 average cost per call)
- Simple ROI calculation for customers
Current Capabilities and Trust Level
- Simple: FAQ answers
- Complex: Product recommendations based on fit/sizing understanding
- Advanced: Technical troubleshooting (Sonos system setup/connectivity)
- High-value: Churn prevention, subscription downgrades, upsell/cross-sell
- Revenue preservation through consistent, patient interactions
- Understanding customer usage patterns for personalized offers
Future Vision (Next 5 Years)
- Comparison to Robert Heinlein's "The Moon is a Harsh Mistress" - talking computers, photorealistic video generation already achieved
- Foundation model improvements will provide "IQ upgrades"
- Multimodal capabilities (80% of customer service is voice-based)
- Companies operating at their best in every interaction
- Individual creative force multiplication
- Fast path from idea to creation (75-minute game development with 8-year-old using AI tools)
- Dramatic acceleration of idea-to-manifestation timeline
