AI summary by TwinMind

I changed databases again (please learn from my mistakes)

Theo - t3․gg

1h 11m 0s

Migration Background & Previous Setup

  • 4th database migration for T3 Chat - previous migrations covered in other videos
  • Original setup: Local-first with Dexie (IndexDB wrapper) + Redis for persistence
    • All data stored locally in IndexDB using Dexie
    • Data serialized with SuperJSON, gzipped, stored in Upstash blob storage
    • Simple key-value store: entire database serialized and uploaded
    • Later broke into separate keys (message:user:id format) for scalability
  • Moved to PlanetScale with Drizzle after Redis throughput issues
    • Schema stored all data in single JSON blob field (SuperJSON string)
    • No traditional columns for content, status, title, model
    • 4x data bloat due to SuperJSON overhead

Core Problems with Previous Architecture

  • IndexDB issues:
    • Only one connection per browser instance (multi-tab problems)
    • Race conditions in Safari
    • API fundamentally broken, painful to work with
  • Split brain problem: definitions in multiple places
    • Local Dexie schema vs database schema
    • Complex client-side migrations (v3, v4, v5, v6)
    • Client as source of truth caused reliability issues
  • Sync engine complexity:
    • Delete persistence never worked correctly (5-6 different approaches)
    • Complex data streaming between server/client/DB
    • 619-line create message file, nearly 1000-line Dexie file
  • Team blocked by sync engine complexity
    • All schema changes required touching sync engine
    • Features like preferences, pinned models couldn't sync

Migration Goals & Requirements

Essential Requirements:

  • Exit IndexDB completely
  • Single source of truth (eliminate split brain)
  • Good optimistic updates
  • Strong signed-out experience support
  • Unblock team from sync engine dependency

Additional Benefits Sought:

  • Resumable streams (handle disconnections during AI generation)
  • Client no longer source of truth
  • Simplified architecture

Solution Evaluation Process

  • Zero (Replicache): Ruled out due to:
    • PostgreSQL requirement
    • Self-hosted infrastructure needs
    • Multiple schema definitions required (SQL + sync layer + client + permissions)
    • Complex permission system setup
    • Still split brain problem
  • Various state management solutions: Still required multiple sources of truth
  • Convex: Chosen as application database solution
    • TypeScript-based queries and mutations
    • Built-in sync engine via WebSocket
    • Single schema definition
    • Good signed-out experience via sessions

Convex Implementation Details

Schema & Architecture:

  • Single schema file defines all tables (threads, messages, users, etc.)
  • User-defined IDs combined with Convex internal IDs
  • Thread IDs remain user-defined for optimistic updates
  • Messages reference thread via user-defined thread ID

Key Improvements:

  • Title updates: Previously 300+ lines of SSE parsing/streaming code
    • Now: Single convexClient.mutation call updates DB directly
    • Client automatically receives update via WebSocket
  • Message updates: Direct DB writes instead of complex streaming protocols
  • Permissions: Built into TypeScript mutation handlers
  • Query invalidation: Automatic when touched data changes

Migration Process:

  • Feature flag system for gradual rollout
  • Cookie migration (HTTP-only false for WebSocket access)
  • Chunked data migration from PlanetScale
    • 500 threads per chunk, 100 messages per chunk
  • User-driven migration (not bulk migration)
  • Migration status tracking in Convex

Major Debugging Crisis

Initial Deployment Disaster:

  • Shipped 11:39 PM Friday, immediately got infinite loop reports
  • Users stuck in "migration complete" loop
  • Panicked, reverted deployment
  • 4-day debugging session described as "murder mystery"

Root Cause Investigation:

  • Initial assumptions: OAuth/cookie issues, Convex cache problems
  • Brave browser bugs discovered:
    • HTTP-only cookie handling with max-age but no expires
    • WebSocket reconnection issues (6-year-old bug)
    • Led to patches being made in Brave browser
  • Actual root cause: OpenAuth library breaking change
    • Early T3 Chat users (first 2 weeks) had incorrect JWT subject format
    • Subject changed from "google:123456" to "user:xyz" format
    • Library update fixed new users but old users remained broken
    • Had to use properties.id instead of subject for user identification

Resolution:

  • Fixed user ID extraction from JWT
  • Added extensive logging and debugging
  • Feature flag system for safer rollouts
  • 286 users successfully migrated via beta system

Technical Challenges & Solutions

Streaming Performance:

  • Convex sends full query results on updates (not diffs)
  • Token-by-token updates would send entire message history each time
  • Solution: Chunked updates every 500ms + optimistic SSE layer
  • Future: Resumable streams with Vercel's new package + Redis storage

Migration Complexity:

  • 5200 lines added, 3300 removed
  • 140+ commits across 6 feature branches
  • Effectively application rewrite, not just DB migration
  • All client-side message/thread logic rewritten

Performance Trade-offs:

  • Some operations slower (forking now requires server round-trip)
  • Overall experience more reliable
  • Cheaper than previous setup
  • Better team velocity

Current State & Future Plans

Achievements:

  • ✅ Off IndexDB completely
  • ✅ Single source of truth
  • ✅ Team unblocked
  • ✅ Resumable streams foundation
  • ✅ Better signed-out experience
  • ✅ Simplified codebase (deleted 1000-line Dexie file)

Remaining Work:

  • Implement optimistic updates for forking
  • Add resumable stream ID field to messages
  • Integrate Vercel's resumable stream package
  • Migrate to proper auth provider (tired of rolling own auth)

Lessons Learned

  • Don't roll your own auth ("would gladly pay $8k/month to avoid this")
  • Application databases vs traditional databases serve different purposes
  • Local-first development has significant hidden costs
  • Friday deployments actually good (lowest traffic)
  • Convex significantly simplified architecture despite initial skepticism
  • Database migrations at this scale are application rewrites

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome