AI summary by TwinMind
Why DxOMark Smartphone Camera Scores are Wrong
Mrwhosetheboss
6m 42s
DxOMark Testing Methodology Overview
- Independent benchmark assessing image quality on phones/cameras
- Companies frequently brag about DxOMark scores
- Actually thorough testing: indoor/outdoor, day/night conditions
- Tests zoom, bokeh, flash quality comprehensively
- Detailed testing not the core issue
Problem 1: Rapidly Evolving Smartphone Features
- Pixel 3 example: scored 101 points
- DxOMark factored: wide-angle front camera, better bokeh, refined zoom
- Excluded features that don't affect noise/texture/exposure:
- Top Shot: fixes people's faces after photo taken
- Night Sight mode: transforms low-light photos from average to best available
- Inconsistent evaluation: praised Galaxy Note 9 low-light but ignored Pixel 3 Night Sight
- Makes Pixel 3 review unrepresentative
Problem 2: Manufacturer Score Withholding
- Huawei P20 Pro currently leads leaderboard
- Mate 20 Pro (improved version) absent from rankings
- Reason: Mate 20 Pro scores too highly
- Huawei strategy: withhold better scores until competitors overtake P20 Pro
- Creates inherently biased leaderboard
- Shows only phones where manufacturers happy with results
Problem 3: Subjective Camera Evaluation Complexity
- Easy to compare detail between two photos
- Difficult to assign numbers to complex trade-offs:
- Huawei night mode: better shots but 4-second stability requirement
- OnePlus: 1-second requirement
- Mate 20 Pro light painting mode: incredible but niche feature
- Single number approach problematic for subjective assessments
- Most people only see final number, not detailed analysis
Problem 4: Missing Audio Quality Assessment
- Video example: P20 Pro scores 109 vs iPhone 8 scores 92
- iPhone 8 actually superior video and audio quality
- Microphone quality not considered in scoring
- Audio represents ~30% of video experience
- Major exclusion from evaluation criteria
Problem 5: Conflict of Interest - Consultation Business
- DxOMark's main business: consulting companies on camera improvements
- Advice focuses on "how to do well on our tests"
- Companies can purchase DxO Analyzer package:
- Massive, expensive hardware/software/training bundle
- Simulates DxOMark test conditions
- Equivalent to "revising for test with answers"
- Creates unfair advantage for purchasing companies
- Non-purchasing companies disadvantaged
Problem 6: Industry Distortion Effects
- Companies compete for highest benchmark scores
- Could hinder smartphone camera progress
- Improvements weighted toward DxOMark priorities
- Diverts focus from new/exciting features
- Optimization for tests rather than user experience
Problem 7: Single Number Inadequacy
- One overall score insufficient for complex assessment
- Detailed website analysis actually thorough and well-researched
- Single number enables misleading marketing claims
- Gives little indication of real-world user suitability
Proposed Solution: Subcategory Scoring Only
- Issue only specific category scores
- Companies could highlight: "best exposure handling" or "highest zoom rating"
- Removes subjectivity of overall encompassing score
- More meaningful numbers for end consumers
- Eliminates weird comparative results:
- 2018 LG G7 = 2015 Galaxy S6 Edge performance
- Xiaomi Mi Mix 2S > iPhone 8 Plus (despite inferior video/audio)
