AI summary by TwinMind

Dan Kokotov: Speech Recognition with AI and Humans | Lex Fridman Podcast #151

Lex Fridman

1h 28m 46s

Introduction and Background

  • Conversation with Dan Kokotov, VP of Engineering at Rev.ai
  • Rev.ai considered best speech-to-text AI engine by many metrics
  • Rev is company doing captioning/transcription by humans and AI
  • Host has been using Rev services for couple years, planning to add captions/transcripts to podcast episodes
  • Sponsors mentioned: Athletic Greens, Blinkist, Business Wars podcast, Cash App

Science Fiction Discussion

  • Greatest sci-fi novel: Dune by Frank Herbert
  • Second greatest: Children of Dune, third: God Emperor of Dune
  • God Emperor of Dune philosophical concept: Leto becomes tyrant/god emperor merged with sandworm
    • Oppresses people to break through civilization stagnation
    • Forces humanity to explode outward and colonize new worlds
    • Philosophy: need pressure/suffering to make progress, avoid getting too comfortable

Rev Company Origins and Model

  • Founded to improve on Upwork freelancer marketplace model
  • CEO Jason was early Upwork employee, familiar with pain points
  • Original Upwork problems:
    • Customers had to browse/choose freelancers without expertise
    • Potential for regret with bad choices
    • Freelancers spent time on profile optimization vs actual work
  • Rev solution: Remove barriers, pick standardizable verticals
  • Started with translation, added audio transcription
  • Hide complexity from users - simple file upload, get results back
  • Originally 48 hours turnaround, made progressively shorter
  • Rush processing now available

Service Categories and Pricing

  • Focus on work-from-home, computer-based, standardizable tasks
  • Translation: Birth certificates, legal documents (fairly well-defined)
  • Audio transcription: Per audio minute pricing
  • Current pricing: $1.25 per minute (was $1.00), automated version $0.25/minute
  • Rev.com for human/end-user services, Rev.ai for ASR API services

Freelancer Demographics and Experience

  • Two-sided marketplace: customers and "Revvers" (freelancers)
  • Challenge: Balancing supply/demand of work and workers
  • Demographics: Mostly US-based (English transcription requirement)
  • Wide variety: Work-from-home moms, students, people with social anxiety, career break professionals
  • Freelancers often knowledgeable about technical concepts, do research during work
  • Many enjoy learning while transcribing (documentaries, lectures)

Transcription Process and Tools

  • Freelancers log into workspace, see available audio list
  • Tools to filter by preferences: length, subject, region
  • Specialized editor built for transcription efficiency
  • Start with ASR (speech recognition) draft, then correct
  • Quality depends on audio: good audio (like podcasts) vs poor audio (back of auditorium)
  • Speed: Real-time transcription very difficult, maybe 2-3x real-time for good audio

Automatic Speech Recognition (ASR) Technology

  • ASR = Automatic Speech Recognition, audio-to-text conversion
  • Rev believes they have world's best ASR for conversational speech
  • Different from voice assistants (Siri) - limited vocabulary vs unstructured conversation
  • Current accuracy: ~14% word error rate on test suite
  • Word error rate calculation: insertions + deletions + substitutions
  • Human accuracy estimated at 2-3% word error rate
  • Compete against Google, Amazon, Microsoft
  • Beat competitors in internal testing

Data Advantage and Machine Learning

  • Biggest factor in ASR success: high-quality labeled data
  • Rev's business model = being paid to annotate data ("magical flywheel")
  • Have final transcripts, original audio, AND edit sequences from Revvers
  • Early stages of utilizing all available data signals
  • Edit sequence data potentially valuable for training improvement models

Rev.ai Platform and Vision

  • Rev.ai brings ASR technology to broader market
  • Evolution: Human-only → Temi (consumer ASR) → Human transcriptionists using ASR → Rev.ai API
  • Vision: Enable new applications with highly accurate speech-to-text
  • Goal: Make all audio/meetings as accessible as written notes
  • Potential for indexed, archived, searchable meeting content
  • Impact on remote work and collaboration

Podcast Industry and Transcription

  • Current podcast discovery limited to title/description keyword search
  • Transcription could enable content-based search within episodes
  • Examples: Finding product recommendations, specific quotes
  • Reporter use cases for pulling quotes and references
  • Host planning to implement transcripts on website for better accessibility

Technical Challenges and Competition

  • Competing with major tech companies (Google, YouTube auto-captions)
  • Working toward 3% error rate goal (human-level performance)
  • RSS feed integration proposals for standard transcript inclusion
  • Criticism of exclusive platform deals vs open podcast ecosystem

Management Transition and Leadership Philosophy

  • Dan's transition from programmer to VP of Engineering
  • Loss of direct creation dopamine, replaced by team success satisfaction
  • Management as "programming humans" - understanding different motivations
  • People respond differently: some need criticism/push, others need constant encouragement
  • "Manage by exception" philosophy - tailor approach to each individual
  • Reference: "First Break All the Rules" management book

Book Recommendations

  • Dune series (6 main books by Frank Herbert, skip son's additions)
  • Brave New World by Aldous Huxley - prescient about genetic/social sorting
  • Brazil (movie) - dystopian vision of authoritarian incompetence vs efficient evil dictatorship
  • Creative Selection by Ken Cocienda - Apple engineer's perspective on working with Steve Jobs
  • Discussion of various dystopian works and their relevance to current technology/society

Philosophy and Meaning of Life

  • Meaning found in contributing to humanity as evolving organism
  • Creation as core drive - from simple HTML buttons to complex systems
  • Parallel between human creativity and religious concept of being created "in God's image"
  • Living, creating, raising unique children as fundamental purposes
  • Programming as god-like act of bringing non-existent things to life

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome