AI summary by TwinMind

I Only Test In Production

Theo - t3․gg

18m 40s

Video Sponsorship Disclosure

  • PostHog sponsors video but had no creative input on content
  • Theo discovered blog post independently, fits current sponsor agreement
  • Full editorial independence maintained

Testing in Production Definition

  • Testing new code with real infrastructure and production data vs local/staging with synthetic data
  • Only way to know if code actually works as expected in real conditions
  • Mock data and staging environments test theoretical behavior, not real-world performance
  • Concept of "fail small" - build safety nets not guard rails for easier recovery

Types of Production Testing Techniques

  • Real user monitoring: tracking app query performance, site performance, error rates, logs
  • Load/spike/soak testing: checking code under high volume/stress conditions
  • Shadowing/mirroring/dark launching: using clones of production database for testing
    • Twitch example: cloned production DB every 2-3 weeks, obfuscated user data
  • Integration testing: checking services/features/infrastructure work together post-deployment
  • Alerts: notifying relevant people when issues occur
  • Usage tracking: analytics, session replays, A/B testing
  • Feedback and surveys: criminally underrated for understanding user behavior

When NOT to Test in Production

  • Risk factors: test failures can impact real users
  • Depends on three factors:
    • Size of business
    • Potential negative impact of changes
    • Speed to identify and resolve issues
  • Theo disagrees with size mattering - argues good pipelines for identification/resolution make size irrelevant
  • Examples given: UI changes with feature flags (safe) vs automated financial trading algorithms (risky)

PostHog's Open Source Context

  • Fully open source on GitHub with MIT license
  • Self-hostable with Docker images and scripts
  • Hard to hide new features since code is public
  • Reduces risk of feature leaks since everything visible in GitHub

PostHog's Three Main Reasons for Testing in Production

1. Production is the Real World

  • Development environments can never completely match production
  • Diminishing returns trying to replicate production locally
  • Quote: "In theory, theory and practice are the same. In practice they are not"
  • Handles massive data amounts requiring big machines - expensive to replicate locally
  • Production reveals bugs missed by local testing
  • Gets feedback from teammates and beta testers as releases widen

2. Dogfooding and Collaboration

  • PostHog are their own best customers
  • Features developed are most useful to them internally
  • Example: used early access management feature to manage its own beta
  • Enables team collaboration in production vs managing in-progress branches
  • Feature flags allow adding teammates for feedback, then transition to phased rollouts
  • Theo's Twitch Studio example: rolled out to staff first, got immediate feedback when employees had to opt out

3. No Staging Environment Maintenance

  • Skipped staging environment, dropped associated maintenance
  • Previously had demo environment but shut it down due to:
    • High maintenance overhead
    • Frequent breaks and bugs different from production
    • Effort better used elsewhere
  • Shifted efforts to improving onboarding and faster project setup
  • Theo's example: complex S3-to-Lambda callback system requiring CloudFlare proxy layer for staging
  • Staging environments suffer from confirmation bias - work/break in expected ways, miss unexpected issues

Implementation Process

Local Testing

  • Run frontend unit tests, visual regression tests, backend tests, end-to-end tests
  • Tests run locally and on new pull requests
  • Ensure merged code doesn't cause bugs/regressions/degradations
  • Note: Uses Husky pre-commit hooks (Theo strongly disagrees with this approach)

Deployment vs Release Separation

  • Deployment: code gets to production but not activated for users
  • Release: controlling which users get access via feature flags
  • Feature flags start rolled out only to developer responsible for change

Feature Flag Options Mentioned

  • PostHog's own feature flags
  • LaunchDarkly (industry standard but heavy)
  • GrowthBook (fully open source, Y Combinator batch, JSON-based)
  • Custom implementation (Theo's gallery app example with user permissions field)

Query Performance Example

  • Team makes query performance improvements using production data/machines
  • Load testing with real queries via production app or Grafana
  • Monitor error tracking for exceptions
  • Use spike/soak/shadowing/mirroring/integration tests

Release Process

  • After deployment tests pass, expand feature flag rollout
  • Use usage analytics, session replays, monitoring, feedback, surveys
  • Include error tracking and bug reports
  • Uncover issues and get feedback fast

Additional Considerations

  • Big PRs problem: Feature flags enable merging code earlier in smaller chunks
  • Each change reviewable independently without massive conflict chains
  • Code can be merged and hidden under feature flags until ready
  • Graphite mentioned as tool fighting against giant pull requests

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome