AI summary by TwinMind
We need to talk about this benchmark
Theo - t3․gg
42m 25s
Initial Benchmark Results and Controversy
- Viral JavaScript framework benchmark showed Next.js performing poorly
- React: 537 ops/second (5,300 total pages rendered)
- Next.js: only 41 ops/second
- 14x performance gap between frameworks
- Ryan Carniato (Solid.js creator) and presenter agree benchmark doesn't mean much
- Solid.js performing worse than React in these benchmarks despite known superior performance
- React 8x slower than Marco
- Solid 10x slower than Marco
- Red flag for benchmark validity since Solid typically faster than React
Benchmark Methodology Issues
- Original benchmark by Adam Rackis, surfaced from Dax's testing
- Tests server-side rendering speed: request → page generation → response
- Vanilla React example doesn't output valid HTML
- Missing critical context: rendering time is tiny fraction of real-world request time
- Database queries, authentication, network requests dominate actual response times
Real-World Performance Context
- Rendering performance spectrum analogy:
- React rendering: small time slice
- Next.js rendering: 14x longer but still tiny
- Actual production workload (DB queries, auth, etc.): massive time slice
- Even 50x rendering improvement negligible in production context
- 99.95% of apps won't notice these differences
Ryan Carniato's Perspective on Benchmarking
- Benchmarking stream least popular content despite high Twitter engagement
- Three topics people claim to care about but don't actually watch videos on:
- Vendor lock-in
- Unexpected bills
- Benchmarks
- Used as weapons to attack frameworks rather than genuine decision-making tools
- Quote: "Benchmarking is easy, unintentionally cheating benchmarks is easier, creating useful benchmarks is hard, having those benchmarks be fair is harder"
Framework Optimization Tricks
- Many frameworks optimize by opting out of features for benchmarks
- Happy path optimizations: basically not using framework, just rendering HTML
- Million.js early benchmark example: overhead skewed results
- Solid.js initial implementation missed required details, inflated performance
- Frameworks can unintentionally cheat by making assumptions
Hydration vs Non-Hydration Performance
- Benchmark shows "times 1" vs "times 2" performance difference
- Times 1: faster, no hydration data sent
- Times 2: slower, includes data required for client-side hydration
- Non-hydrated apps just spit out HTML, no client interactivity
- Hydrated apps double payload but enable dynamic client behavior
Benchmark Modifications and Testing
- Added 100ms data fetching delay (realistic for DB/auth operations)
- Performance gap reduced from 14x to 1.5x
- Added concurrency testing (5 parallel requests)
- With concurrency: gap reduced to 1.28x slower
- Real-world conditions make benchmark differences negligible
React vs Next.js Output Comparison
- React example output:
- Not valid HTML (just stream of TR/TD elements)
- No JavaScript for client interaction
- No head section or document structure
- Uses simple reactdom.server.renderToPipeableStream
- Next.js output:
- Valid HTML document
- Includes JavaScript for hydration
- Data serialization for client-side updates
- Enables navigation, pre-fetching, dynamic updates
- Proper escaping and security measures
Framework Improvements from Benchmark Pressure
- View team found 2x performance improvement
- Next.js team made optimizations:
- Avoid extra render to readable stream when content empty
- Remove unnecessary await stream.allReady
- Increase progressive chunk size
- Skip server HTML insertion when not needed
- Improvements are micro-optimizations for edge cases
- If 2x wins weren't found until now, shows how unimportant these metrics are
Next.js Engineering Priorities
- Jimmy (Next.js engineering manager) response:
- Raw throughput not main concern
- Focus on stability and developer experience over micro-benchmarks
- Routing, rendering, and response streaming pipeline complexity
- Web streams vs Node streams tradeoff for edge compatibility
- Parallel routing hot reload bugs more important than benchmark scores
- App router stability prioritized over raw performance numbers
Technical Architecture Decisions
- Next.js uses web streams for node/edge compatibility
- Can run on Cloudflare Workers, edge runtime
- Could optimize for Node.js specifically but lose edge support
- Following web standards vs proprietary optimizations
- Server components add overhead but enable powerful features
- Framework complexity enables developer and user experience benefits
Benchmark Tool Evolution
- Benchmark.js archived after 6+ years due to viral attention
- Tiny bench became new standard overnight
- Muhammad (Tiny bench author) added concurrency support
- Concurrency testing showed much closer performance numbers
- Marco performance degraded significantly with parallel requests
Final Conclusions
- Benchmarks useful for framework authors to catch regressions
- Not useful for developers choosing frameworks
- Real bottlenecks: database queries, authentication, network requests
- Framework choice should be based on features, DX, ecosystem
- Using benchmarks to shame technical decisions is counterproductive
- Production performance dominated by factors other than rendering speed
