AI summary by TwinMind

Sierra co-founder Clay Bavor on Making Customer-Facing AI Agents Delightful

Sequoia Capital

1h 12m 31s

Clay Bavor Background and Google Experience

  • Bay Area native, 18 years at Google until March 2022
  • Started in Search, then ads, ran product/design teams for Workspace (Gmail, Google Docs, Google Drive)
  • Last 10 years focused on forward-looking bets: AR/VR, Google Lens, AI applications
  • Early exposure to large language models at Google (Mina/Lambda, PaLM 540B parameter model)
  • Key moment: PaLM explained 2008 financial crisis using movie references ("like Inception except instead of dreams within dreams it was debt within debt")
  • Avocado chairs from DALL-E paper inspired office decor at Sierra

Sierra Founding and Vision

  • Co-founded with Brett Taylor (longtime friend from Google APM program) in December 2022
  • Monthly poker group connection maintained friendship
  • Vision: Enable every company to create branded customer-facing AI agents
  • Parallel to historical technology shifts: 90s websites, 2000s social networks, 2015 mobile apps
  • Now: conversational AI agents that can understand, generate language, reason, and make decisions

Current Sierra Performance and Customers

  • 15 months old company
  • Customers: Weight Watchers, Sonos, SiriusXM, Olay
  • Resolving 70%+ of incoming customer inquiries
  • Customer satisfaction: mid-4s out of 5 stars (often higher than human agents)
  • Use cases: points advice, subscription management, radio troubleshooting, product recommendations

Agent OS Technical Architecture

Foundation Layer

  • Data governance, PII detection/masking/encryption built-in
  • Customer data stays customer data, protected before logging

Agent SDK

  • Declarative programming language purpose-built for agents
  • Express high-level goals and guardrails
  • Define behavior scaffolding for complex tasks
  • Example: Healthcare-adjacent customer can discuss products without dispensing medical advice

Runtime System

  • Abstracts complexity from developers (define "what" not "how")
  • May involve 5-10 separate LLM calls to different models
  • Document retrieval for accurate answers
  • Equivalent to OS handling IO and resource utilization

Supervisory Architecture and Quality Control

  • "Solution to many problems with AI is more AI"
  • LLMs better at detecting errors in output than avoiding them initially
  • Multiple supervisory agents act as "Jiminy Cricket" oversight
  • Check for: factual accuracy, medical/financial advice, prompt injection attacks
  • Different personas elicit more discerning behavior from supervisors

TAU Benchmark Research

  • Tool Agent User Benchmark - first research paper from Sierra
  • Existing benchmarks inadequate for real-world agent performance
  • Tests conversational interactions with user simulator agent
  • Results: Best LLMs alone achieve only 61% success rate (returns), 35% (airline reservations)
  • Pass@K metric: 0.61^8 = ~2.5% reliability over multiple interactions
  • Demonstrates need for sophisticated agent architectures beyond raw LLMs

Engineering vs Research Balance

  • Primary approach: engineering challenge using composed foundation models
  • Fine-tuned open-source models with proprietary datasets
  • Supplemented with retrieval systems (RAG) for grounding
  • External reasoning scaffolding (planning, task generation, supervisors)
  • Can swap in better/specialized models as "IQ upgrades"
  • Continuous progress in agent architecture and tooling

Customer Deployment Process

New Software Paradigm

  • Programming with English language vs traditional code
  • Non-deterministic models vs predictable input/output
  • Structured input to messy human language
  • Model upgrades change behavior fundamentally

Agent Development Lifecycle

  • Declarative programming language for agent definition
  • Conversation simulator for testing (hundreds/thousands of steps)
  • New approaches to unit/integration testing
  • Release management and quality assurance

Deep Customer Partnership

  • Solution includes technology + teaching + auditing/improvement
  • Deployment team acts as product managers/engineers
  • Each agent built as new product for customer
  • Work with customer experience + marketing teams
  • Define voice, values, vibe (emoji usage varies by brand)
  • Extract business processes from institutional knowledge (not documented manuals)
  • Handle "policy behind the policy" scenarios

Experience Manager Platform

  • Command center for customer experience teams
  • Rich analytics on trending/new issues
  • Proactive issue detection (shipping depot outages, app crashes 8-10 hours early)
  • Biased sampling to surface problem conversations
  • Coaching feedback system for continuous improvement
  • Test creation for every improvement to prevent regression

Pricing and Business Model

  • Resolution-based/outcome-based pricing
  • Only charge when customer problem fully solved
  • Aligned incentives: Sierra wants high resolution rates, customers want to send more cases
  • Fraction of cost compared to 20-minute phone calls (~$12-13 average cost per call)
  • Simple ROI calculation for customers

Current Capabilities and Trust Level

  • Simple: FAQ answers
  • Complex: Product recommendations based on fit/sizing understanding
  • Advanced: Technical troubleshooting (Sonos system setup/connectivity)
  • High-value: Churn prevention, subscription downgrades, upsell/cross-sell
  • Revenue preservation through consistent, patient interactions
  • Understanding customer usage patterns for personalized offers

Future Vision (Next 5 Years)

  • Comparison to Robert Heinlein's "The Moon is a Harsh Mistress" - talking computers, photorealistic video generation already achieved
  • Foundation model improvements will provide "IQ upgrades"
  • Multimodal capabilities (80% of customer service is voice-based)
  • Companies operating at their best in every interaction
  • Individual creative force multiplication
  • Fast path from idea to creation (75-minute game development with 8-year-old using AI tools)
  • Dramatic acceleration of idea-to-manifestation timeline

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome