AI summary by TwinMind

Why DxOMark Smartphone Camera Scores are Wrong

Mrwhosetheboss

6m 42s

DxOMark Testing Methodology Overview

  • Independent benchmark assessing image quality on phones/cameras
  • Companies frequently brag about DxOMark scores
  • Actually thorough testing: indoor/outdoor, day/night conditions
  • Tests zoom, bokeh, flash quality comprehensively
  • Detailed testing not the core issue

Problem 1: Rapidly Evolving Smartphone Features

  • Pixel 3 example: scored 101 points
  • DxOMark factored: wide-angle front camera, better bokeh, refined zoom
  • Excluded features that don't affect noise/texture/exposure:
    • Top Shot: fixes people's faces after photo taken
    • Night Sight mode: transforms low-light photos from average to best available
  • Inconsistent evaluation: praised Galaxy Note 9 low-light but ignored Pixel 3 Night Sight
  • Makes Pixel 3 review unrepresentative

Problem 2: Manufacturer Score Withholding

  • Huawei P20 Pro currently leads leaderboard
  • Mate 20 Pro (improved version) absent from rankings
  • Reason: Mate 20 Pro scores too highly
  • Huawei strategy: withhold better scores until competitors overtake P20 Pro
  • Creates inherently biased leaderboard
  • Shows only phones where manufacturers happy with results

Problem 3: Subjective Camera Evaluation Complexity

  • Easy to compare detail between two photos
  • Difficult to assign numbers to complex trade-offs:
    • Huawei night mode: better shots but 4-second stability requirement
    • OnePlus: 1-second requirement
    • Mate 20 Pro light painting mode: incredible but niche feature
  • Single number approach problematic for subjective assessments
  • Most people only see final number, not detailed analysis

Problem 4: Missing Audio Quality Assessment

  • Video example: P20 Pro scores 109 vs iPhone 8 scores 92
  • iPhone 8 actually superior video and audio quality
  • Microphone quality not considered in scoring
  • Audio represents ~30% of video experience
  • Major exclusion from evaluation criteria

Problem 5: Conflict of Interest - Consultation Business

  • DxOMark's main business: consulting companies on camera improvements
  • Advice focuses on "how to do well on our tests"
  • Companies can purchase DxO Analyzer package:
    • Massive, expensive hardware/software/training bundle
    • Simulates DxOMark test conditions
    • Equivalent to "revising for test with answers"
  • Creates unfair advantage for purchasing companies
  • Non-purchasing companies disadvantaged

Problem 6: Industry Distortion Effects

  • Companies compete for highest benchmark scores
  • Could hinder smartphone camera progress
  • Improvements weighted toward DxOMark priorities
  • Diverts focus from new/exciting features
  • Optimization for tests rather than user experience

Problem 7: Single Number Inadequacy

  • One overall score insufficient for complex assessment
  • Detailed website analysis actually thorough and well-researched
  • Single number enables misleading marketing claims
  • Gives little indication of real-world user suitability

Proposed Solution: Subcategory Scoring Only

  • Issue only specific category scores
  • Companies could highlight: "best exposure handling" or "highest zoom rating"
  • Removes subjectivity of overall encompassing score
  • More meaningful numbers for end consumers
  • Eliminates weird comparative results:
    • 2018 LG G7 = 2015 Galaxy S6 Edge performance
    • Xiaomi Mi Mix 2S > iPhone 8 Plus (despite inferior video/audio)

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome