AI summary by TwinMind

We need to talk about this benchmark

Theo - t3․gg

42m 25s

Initial Benchmark Results and Controversy

  • Viral JavaScript framework benchmark showed Next.js performing poorly
    • React: 537 ops/second (5,300 total pages rendered)
    • Next.js: only 41 ops/second
    • 14x performance gap between frameworks
  • Ryan Carniato (Solid.js creator) and presenter agree benchmark doesn't mean much
  • Solid.js performing worse than React in these benchmarks despite known superior performance
    • React 8x slower than Marco
    • Solid 10x slower than Marco
    • Red flag for benchmark validity since Solid typically faster than React

Benchmark Methodology Issues

  • Original benchmark by Adam Rackis, surfaced from Dax's testing
  • Tests server-side rendering speed: request → page generation → response
  • Vanilla React example doesn't output valid HTML
  • Missing critical context: rendering time is tiny fraction of real-world request time
  • Database queries, authentication, network requests dominate actual response times

Real-World Performance Context

  • Rendering performance spectrum analogy:
    • React rendering: small time slice
    • Next.js rendering: 14x longer but still tiny
    • Actual production workload (DB queries, auth, etc.): massive time slice
  • Even 50x rendering improvement negligible in production context
  • 99.95% of apps won't notice these differences

Ryan Carniato's Perspective on Benchmarking

  • Benchmarking stream least popular content despite high Twitter engagement
  • Three topics people claim to care about but don't actually watch videos on:
    • Vendor lock-in
    • Unexpected bills
    • Benchmarks
  • Used as weapons to attack frameworks rather than genuine decision-making tools
  • Quote: "Benchmarking is easy, unintentionally cheating benchmarks is easier, creating useful benchmarks is hard, having those benchmarks be fair is harder"

Framework Optimization Tricks

  • Many frameworks optimize by opting out of features for benchmarks
  • Happy path optimizations: basically not using framework, just rendering HTML
  • Million.js early benchmark example: overhead skewed results
  • Solid.js initial implementation missed required details, inflated performance
  • Frameworks can unintentionally cheat by making assumptions

Hydration vs Non-Hydration Performance

  • Benchmark shows "times 1" vs "times 2" performance difference
  • Times 1: faster, no hydration data sent
  • Times 2: slower, includes data required for client-side hydration
  • Non-hydrated apps just spit out HTML, no client interactivity
  • Hydrated apps double payload but enable dynamic client behavior

Benchmark Modifications and Testing

  • Added 100ms data fetching delay (realistic for DB/auth operations)
  • Performance gap reduced from 14x to 1.5x
  • Added concurrency testing (5 parallel requests)
  • With concurrency: gap reduced to 1.28x slower
  • Real-world conditions make benchmark differences negligible

React vs Next.js Output Comparison

  • React example output:
    • Not valid HTML (just stream of TR/TD elements)
    • No JavaScript for client interaction
    • No head section or document structure
    • Uses simple reactdom.server.renderToPipeableStream
  • Next.js output:
    • Valid HTML document
    • Includes JavaScript for hydration
    • Data serialization for client-side updates
    • Enables navigation, pre-fetching, dynamic updates
    • Proper escaping and security measures

Framework Improvements from Benchmark Pressure

  • View team found 2x performance improvement
  • Next.js team made optimizations:
    • Avoid extra render to readable stream when content empty
    • Remove unnecessary await stream.allReady
    • Increase progressive chunk size
    • Skip server HTML insertion when not needed
  • Improvements are micro-optimizations for edge cases
  • If 2x wins weren't found until now, shows how unimportant these metrics are

Next.js Engineering Priorities

  • Jimmy (Next.js engineering manager) response:
    • Raw throughput not main concern
    • Focus on stability and developer experience over micro-benchmarks
    • Routing, rendering, and response streaming pipeline complexity
    • Web streams vs Node streams tradeoff for edge compatibility
  • Parallel routing hot reload bugs more important than benchmark scores
  • App router stability prioritized over raw performance numbers

Technical Architecture Decisions

  • Next.js uses web streams for node/edge compatibility
    • Can run on Cloudflare Workers, edge runtime
    • Could optimize for Node.js specifically but lose edge support
    • Following web standards vs proprietary optimizations
  • Server components add overhead but enable powerful features
  • Framework complexity enables developer and user experience benefits

Benchmark Tool Evolution

  • Benchmark.js archived after 6+ years due to viral attention
  • Tiny bench became new standard overnight
  • Muhammad (Tiny bench author) added concurrency support
  • Concurrency testing showed much closer performance numbers
  • Marco performance degraded significantly with parallel requests

Final Conclusions

  • Benchmarks useful for framework authors to catch regressions
  • Not useful for developers choosing frameworks
  • Real bottlenecks: database queries, authentication, network requests
  • Framework choice should be based on features, DX, ecosystem
  • Using benchmarks to shame technical decisions is counterproductive
  • Production performance dominated by factors other than rendering speed

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome