AI summary by TwinMind
The Consequences of Your Code
Tom Scott
6m 15s
Personal STI Testing Experience
- Received chlamydia notification from ex via creative poem format
- Visited London STI clinic next morning for testing
- Observed diverse cross-section of London life in clinic queue
- Submitted urine sample, received business card with phone number and passcode
- System design: automated results for negative tests, human transfer for positive results
System Failure and User Experience
- Called next day, entered passcode, received test results message
- Experienced dramatic 10-second pause creating artificial tension
- System unexpectedly transferred to human staff member despite negative results
- Staff confirmed all clear/negative results but couldn't explain system transfer
- Receptionist dismissed glitch as normal occurrence ("just does that sometimes")
Technical Analysis of System Failure
- Potential causes: database lookup failure, timeout handling, file annotation confusion
- Poor error handling: system defaulted to human transfer without explanation
- No transparency in failure messaging or reasoning
- System used for serious medical conditions beyond treatable STIs
Broader Implications for Public Systems
- Distinction between hobby projects (bodging acceptable) vs public systems (higher standards required)
- Trade-off considerations: time/money investment vs edge case handling
- Design assumption: lookups fail 1 in million times, not worth extra messaging
- Reality: frequent enough failures that staff consider normal
- System inertia: too expensive to fix once deployed, users must tolerate bugs
Big Tech Parallels and Consequences
- Facebook benefits: unprecedented communication, emotional labor, maintaining friendships
- Facebook harms: enabling stalkers/abusers, data misuse, election meddling
- YouTube benefits: democratized publishing, creator income, human connection
- YouTube harms: radicalization, conspiracy theories, child trauma
- Incompetence vs malice: betting on lack of foresight rather than intentional harm
Design Philosophy and Recommendations
- No crystal ball for predicting consequences (dating app example: 1,000 marriages vs 3 murders)
- STI system benefits: reduced staff workload, out-of-hours convenience, discretion
- Simple fix: additional voice message "sorry I can't find your result, just a moment"
- Minimize potential harm in system design
- Attack your own systems: explore failure states, deliberate testing
- Consider malicious use cases and bad actor scenarios
