AI summary by TwinMind

Why Vertical LLM Agents Are The New $1 Billion SaaS Opportunities

Y Combinator

37m 6s

Company Background & Pre-LLM Era (2012-2023)

  • CaseText founded 11-12 years ago by Jake Heller (lawyer + CS background)
  • Mission: bring best technology to legal space, solve inefficient legal research
  • Initial problem: lawyers spent days manually reading documents/case law vs. simple Google searches for basic info
  • First 10 years: $0 to $100M valuation through incremental improvements

Early Pivot Attempts

  • Original concept: user-generated content (UGC) site for lawyer annotations of case law
  • Heroes: Stack Overflow, Wikipedia, GitHub
  • Total failure: lawyers bill by hour, no time for free contributions vs. Wikipedia editors with excess time
  • Shifted to natural language processing/machine learning automation

Pre-GPT-4 Technology Stack

  • Used recommendation algorithms (similar to Pandora/Spotify)
  • Citation network analysis: cases referencing other cases
  • Features: check lawyer work, suggest missed relevant cases
  • Reality: only incremental improvements on legal workflow
  • Client resistance: "$5M/year lawyers" didn't want change, billed by hour so efficiency wasn't incentive

The GPT-4 Breakthrough Moment

Early Access & Company Transformation

  • Close relationship with OpenAI research labs through decade of AI investment
  • Got early GPT-4 access months before public release (under NDA)
  • 48-hour decision window: saw GPT-4 on Friday, decided over weekend
  • Monday executive offsite: complete company pivot announcement

Dramatic Company Restructuring

  • 120-person company (corrected from initial "20 people" statement)
  • 100% of workforce shifted from existing projects to GPT-4 product in 48 hours
  • "Whole company didn't sleep for months before GPT-4" public launch
  • Built Co-Counsel: AI legal assistant concept

Product Development Philosophy

The "Skills" Framework

  • Co-Counsel designed as AI legal assistant - "new member of firm"
  • Tasks: read millions of documents, detect fraud, summarize, legal research memos
  • "Skills" approach: break complex tasks into step-by-step processes
  • Model: how would best attorney in world approach this problem

Technical Implementation Deep Dive

  • Research workflow breakdown:
    • English query → specialized search queries (SQL-like syntax)
    • Execute dozen+ searches against legal databases
    • 100+ results per search
    • Read every single result (cases, statutes, regulations)
    • Take notes, summarize, create outline
    • Compile final research memo with citations
  • Each step = individual prompt in chain (dozen to two dozen prompts total)

Test-Driven Development for AI

Rigorous Testing Framework

  • Started with few dozen tests → few hundred → few thousand
  • Gold standard answers for each prompt in chain
  • Example: English query breakdown into search queries had clear "good" examples
  • Target: 1,199/1,200 correct answers (99.9%+ accuracy)
  • Test-first approach: write tests before prompts

Accuracy Requirements

  • Legal domain demands 100% accuracy vs. 70% consumer tolerance
  • One bad experience = lost customer for year+
  • Conservative lawyers need perfect first impression
  • Voice in head from 10 years of lawyer feedback on smallest mistakes

Model Evolution & Capabilities

GPT Progression Analysis

  • GPT-3/3.5: "plausible sounding English, sounds like lawyer but making stuff up wildly"
  • Bar exam performance: GPT-3.5 = 10th percentile, GPT-4 = 90th percentile
  • Extensive prompt work required for earlier models to cite accurately
  • GPT-4: fundamental change, not incremental improvement

OpenAI O1 Breakthrough Testing

  • Test: 40-page legal brief with subtle alterations (adding "not" changes meaning entirely)
  • Previous LLMs: "nothing wrong, perfectly right"
  • O1: identifies precise changes like "and" to "neither nor"
  • Sits for full minute thinking before answering
  • Passes tests requiring "precise detail thinking"

Business Model & Market Dynamics

Revenue & Valuation Journey

  • Pre-Co-Counsel: $15-20M ARR, 70-80% YoY growth
  • 2 months post-Co-Counsel launch: acquisition conversations began
  • 6 months total: $650M exit to Thomson Reuters
  • Classic product-market fit indicators: servers crashing, can't hire fast enough, major media coverage (CNN, MSNBC)

Customer Transformation

  • Pre-ChatGPT: "I make $5M/year, don't want anything to change"
  • Post-ChatGPT public release: same lawyers calling "we need to get ahead of this"
  • Market perception shift: incremental → fundamental change
  • First-time AI experience for many lawyers (pre-ChatGPT era)

Technical Architecture & IP

Beyond "GPT Wrapper" Criticism

  • Proprietary datasets: law itself + automated annotations
  • Customer database integrations (legal document management systems)
  • OCR optimization for legal documents
  • Edge cases: handwritten notes, tilted scans, 4-pages-per-page printing
  • Reading order logic (1-2-3-4 vs. across pattern)
  • Dozens of components before reaching LLM

Comparison to Traditional SaaS

  • SaaS historically = "SQL wrapper" with business logic
  • AI applications = LLM + business logic + domain integrations
  • Gap between "almost works" (70%, $20/month) and "actually works" (100%, $500-1000/month)
  • Value creation in last mile optimization

Founder Mode & Change Management

Internal Resistance Management

  • 10-year veteran employees seen multiple direction changes
  • "Here Jake goes again with crazy new technology"
  • Founder credibility limited after previous pivots
  • Solution: lead by example - Jake built first version personally despite 120-person team

Early Customer Validation

  • NDA initially limited to Jake + co-founder only
  • Extended NDA to handful of customers for months of pre-launch testing
  • Customer reactions on Zoom calls convinced skeptical employees
  • "Existential crises live on Zoom calls"
  • Senior attorneys: "well I have to retire soon, don't want to deal with this"

Strategic Insights & Future Implications

Prompting Innovation with O1

  • Investigating teaching O1 "what to think about during thinking process"
  • Inject domain expertise from "best lawyers in country"
  • New prompting technique: not just examples of good answers, but how to think
  • Potential to overcome limitations of contractor intelligence in training data

Industry Transformation Predictions

  • Millions spent on document review → strategic thinking focus
  • Jobs become more interesting rather than disappearing
  • Unlock for companies currently paying millions in salaries
  • Path exists despite "hallucination/accuracy" tropes
  • Thousands of potential billion-dollar AI companies across verticals

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome