AI summary by TwinMind
I Only Test In Production
Theo - t3․gg
18m 40s
Video Sponsorship Disclosure
- PostHog sponsors video but had no creative input on content
- Theo discovered blog post independently, fits current sponsor agreement
- Full editorial independence maintained
Testing in Production Definition
- Testing new code with real infrastructure and production data vs local/staging with synthetic data
- Only way to know if code actually works as expected in real conditions
- Mock data and staging environments test theoretical behavior, not real-world performance
- Concept of "fail small" - build safety nets not guard rails for easier recovery
Types of Production Testing Techniques
- Real user monitoring: tracking app query performance, site performance, error rates, logs
- Load/spike/soak testing: checking code under high volume/stress conditions
- Shadowing/mirroring/dark launching: using clones of production database for testing
- Twitch example: cloned production DB every 2-3 weeks, obfuscated user data
- Integration testing: checking services/features/infrastructure work together post-deployment
- Alerts: notifying relevant people when issues occur
- Usage tracking: analytics, session replays, A/B testing
- Feedback and surveys: criminally underrated for understanding user behavior
When NOT to Test in Production
- Risk factors: test failures can impact real users
- Depends on three factors:
- Size of business
- Potential negative impact of changes
- Speed to identify and resolve issues
- Theo disagrees with size mattering - argues good pipelines for identification/resolution make size irrelevant
- Examples given: UI changes with feature flags (safe) vs automated financial trading algorithms (risky)
PostHog's Open Source Context
- Fully open source on GitHub with MIT license
- Self-hostable with Docker images and scripts
- Hard to hide new features since code is public
- Reduces risk of feature leaks since everything visible in GitHub
PostHog's Three Main Reasons for Testing in Production
1. Production is the Real World
- Development environments can never completely match production
- Diminishing returns trying to replicate production locally
- Quote: "In theory, theory and practice are the same. In practice they are not"
- Handles massive data amounts requiring big machines - expensive to replicate locally
- Production reveals bugs missed by local testing
- Gets feedback from teammates and beta testers as releases widen
2. Dogfooding and Collaboration
- PostHog are their own best customers
- Features developed are most useful to them internally
- Example: used early access management feature to manage its own beta
- Enables team collaboration in production vs managing in-progress branches
- Feature flags allow adding teammates for feedback, then transition to phased rollouts
- Theo's Twitch Studio example: rolled out to staff first, got immediate feedback when employees had to opt out
3. No Staging Environment Maintenance
- Skipped staging environment, dropped associated maintenance
- Previously had demo environment but shut it down due to:
- High maintenance overhead
- Frequent breaks and bugs different from production
- Effort better used elsewhere
- Shifted efforts to improving onboarding and faster project setup
- Theo's example: complex S3-to-Lambda callback system requiring CloudFlare proxy layer for staging
- Staging environments suffer from confirmation bias - work/break in expected ways, miss unexpected issues
Implementation Process
Local Testing
- Run frontend unit tests, visual regression tests, backend tests, end-to-end tests
- Tests run locally and on new pull requests
- Ensure merged code doesn't cause bugs/regressions/degradations
- Note: Uses Husky pre-commit hooks (Theo strongly disagrees with this approach)
Deployment vs Release Separation
- Deployment: code gets to production but not activated for users
- Release: controlling which users get access via feature flags
- Feature flags start rolled out only to developer responsible for change
Feature Flag Options Mentioned
- PostHog's own feature flags
- LaunchDarkly (industry standard but heavy)
- GrowthBook (fully open source, Y Combinator batch, JSON-based)
- Custom implementation (Theo's gallery app example with user permissions field)
Query Performance Example
- Team makes query performance improvements using production data/machines
- Load testing with real queries via production app or Grafana
- Monitor error tracking for exceptions
- Use spike/soak/shadowing/mirroring/integration tests
Release Process
- After deployment tests pass, expand feature flag rollout
- Use usage analytics, session replays, monitoring, feedback, surveys
- Include error tracking and bug reports
- Uncover issues and get feedback fast
Additional Considerations
- Big PRs problem: Feature flags enable merging code earlier in smaller chunks
- Each change reviewable independently without massive conflict chains
- Code can be merged and hidden under feature flags until ready
- Graphite mentioned as tool fighting against giant pull requests
