AI summary by TwinMind
I changed databases again (please learn from my mistakes)
Theo - t3․gg
1h 11m 0s
Migration Background & Previous Setup
- 4th database migration for T3 Chat - previous migrations covered in other videos
- Original setup: Local-first with Dexie (IndexDB wrapper) + Redis for persistence
- All data stored locally in IndexDB using Dexie
- Data serialized with SuperJSON, gzipped, stored in Upstash blob storage
- Simple key-value store: entire database serialized and uploaded
- Later broke into separate keys (message:user:id format) for scalability
- Moved to PlanetScale with Drizzle after Redis throughput issues
- Schema stored all data in single JSON blob field (SuperJSON string)
- No traditional columns for content, status, title, model
- 4x data bloat due to SuperJSON overhead
Core Problems with Previous Architecture
- IndexDB issues:
- Only one connection per browser instance (multi-tab problems)
- Race conditions in Safari
- API fundamentally broken, painful to work with
- Split brain problem: definitions in multiple places
- Local Dexie schema vs database schema
- Complex client-side migrations (v3, v4, v5, v6)
- Client as source of truth caused reliability issues
- Sync engine complexity:
- Delete persistence never worked correctly (5-6 different approaches)
- Complex data streaming between server/client/DB
- 619-line create message file, nearly 1000-line Dexie file
- Team blocked by sync engine complexity
- All schema changes required touching sync engine
- Features like preferences, pinned models couldn't sync
Migration Goals & Requirements
Essential Requirements:
- Exit IndexDB completely
- Single source of truth (eliminate split brain)
- Good optimistic updates
- Strong signed-out experience support
- Unblock team from sync engine dependency
Additional Benefits Sought:
- Resumable streams (handle disconnections during AI generation)
- Client no longer source of truth
- Simplified architecture
Solution Evaluation Process
- Zero (Replicache): Ruled out due to:
- PostgreSQL requirement
- Self-hosted infrastructure needs
- Multiple schema definitions required (SQL + sync layer + client + permissions)
- Complex permission system setup
- Still split brain problem
- Various state management solutions: Still required multiple sources of truth
- Convex: Chosen as application database solution
- TypeScript-based queries and mutations
- Built-in sync engine via WebSocket
- Single schema definition
- Good signed-out experience via sessions
Convex Implementation Details
Schema & Architecture:
- Single schema file defines all tables (threads, messages, users, etc.)
- User-defined IDs combined with Convex internal IDs
- Thread IDs remain user-defined for optimistic updates
- Messages reference thread via user-defined thread ID
Key Improvements:
- Title updates: Previously 300+ lines of SSE parsing/streaming code
- Now: Single
convexClient.mutationcall updates DB directly - Client automatically receives update via WebSocket
- Now: Single
- Message updates: Direct DB writes instead of complex streaming protocols
- Permissions: Built into TypeScript mutation handlers
- Query invalidation: Automatic when touched data changes
Migration Process:
- Feature flag system for gradual rollout
- Cookie migration (HTTP-only false for WebSocket access)
- Chunked data migration from PlanetScale
- 500 threads per chunk, 100 messages per chunk
- User-driven migration (not bulk migration)
- Migration status tracking in Convex
Major Debugging Crisis
Initial Deployment Disaster:
- Shipped 11:39 PM Friday, immediately got infinite loop reports
- Users stuck in "migration complete" loop
- Panicked, reverted deployment
- 4-day debugging session described as "murder mystery"
Root Cause Investigation:
- Initial assumptions: OAuth/cookie issues, Convex cache problems
- Brave browser bugs discovered:
- HTTP-only cookie handling with max-age but no expires
- WebSocket reconnection issues (6-year-old bug)
- Led to patches being made in Brave browser
- Actual root cause: OpenAuth library breaking change
- Early T3 Chat users (first 2 weeks) had incorrect JWT subject format
- Subject changed from "google:123456" to "user:xyz" format
- Library update fixed new users but old users remained broken
- Had to use
properties.idinstead ofsubjectfor user identification
Resolution:
- Fixed user ID extraction from JWT
- Added extensive logging and debugging
- Feature flag system for safer rollouts
- 286 users successfully migrated via beta system
Technical Challenges & Solutions
Streaming Performance:
- Convex sends full query results on updates (not diffs)
- Token-by-token updates would send entire message history each time
- Solution: Chunked updates every 500ms + optimistic SSE layer
- Future: Resumable streams with Vercel's new package + Redis storage
Migration Complexity:
- 5200 lines added, 3300 removed
- 140+ commits across 6 feature branches
- Effectively application rewrite, not just DB migration
- All client-side message/thread logic rewritten
Performance Trade-offs:
- Some operations slower (forking now requires server round-trip)
- Overall experience more reliable
- Cheaper than previous setup
- Better team velocity
Current State & Future Plans
Achievements:
- ✅ Off IndexDB completely
- ✅ Single source of truth
- ✅ Team unblocked
- ✅ Resumable streams foundation
- ✅ Better signed-out experience
- ✅ Simplified codebase (deleted 1000-line Dexie file)
Remaining Work:
- Implement optimistic updates for forking
- Add resumable stream ID field to messages
- Integrate Vercel's resumable stream package
- Migrate to proper auth provider (tired of rolling own auth)
Lessons Learned
- Don't roll your own auth ("would gladly pay $8k/month to avoid this")
- Application databases vs traditional databases serve different purposes
- Local-first development has significant hidden costs
- Friday deployments actually good (lowest traffic)
- Convex significantly simplified architecture despite initial skepticism
- Database migrations at this scale are application rewrites
