AI summary by TwinMind
Fixing serverless node.js (by adding servers?)
Theo - t3․gg
52m 7s
Introduction & Sponsorship
- Vercel shipped in-function concurrency feature after ending sponsorship relationship
- Episode not sponsored by Vercel
- Unkey sponsorship demo: API rate limiting setup
- Added rate limiter with 2 requests per 30 seconds per IP
- Open source solution, can self-host or pay for service
Expert Opinions on Vercel's Innovation
- AJ (AWS Hero, DataDog staff engineer): remarkable accomplishment, hopes AWS builds similar features
- Dax: Previously critical of Vercel but acknowledges this is "pretty cool"
- Feature enables serious workloads on Vercel platform
- Industry consensus: significant innovation in serverless space
Serverless History & Node.js Fundamentals
- Lambda revolutionized batch workloads with 1:1 function-to-invocation mapping
- Traditional serverless: each request gets own machine/server
- Problem: wasted CPU cycles during I/O wait times
- Ryan Dahl's original Node.js vision (2009): event-driven, non-blocking I/O
- Millions of clock cycles wasted waiting for database/network responses
- Callback-based async model allowed concurrent request handling
- "This is how I/O should be done for fast, high-concurrency servers"
Vercel's Technical Implementation
- 20-50% reduction in compute usage and costs without latency impact
- Built for interactive workloads: server rendering, APIs, AI applications
- Native Node support with V8 bytecode acceleration and instance pre-warming
Architecture Components
- Edge routing layer
- Function load balancer
- Function invocation service (manages up to 100k instances per function)
- Rust-powered runtime rewrite for performance and stability
- Bidirectional TCP socket streaming back to invocation service
Database Connection Problem Solved
- Traditional serverless: each request creates new database connection
- Connection limits (e.g., Heroku free tier: 10 connections)
- Cold start delays from connection establishment
- New model: connection reuse across multiple requests in same instance
- Reduces need for heavy connection pooling systems
Efficiency Gains & Real-World Results
- Beta customers seeing 20-50% compute usage reduction
- Verse example: 50% cost reduction during traffic surge (10M+ visitors)
- Another customer: 35% savings on 3000+ compute hours
- Author's limited results due to low traffic patterns
- Savings scale with request volume and I/O wait times
Concurrency Model Comparisons
Traditional VPS
- Concurrency handled entirely in Node.js
- Single server, multiple requests, shared database connection
Lambda (Traditional)
- Concurrency through separate server instances
- Each request gets own machine and database connection
- No shared resources between requests
Cloudflare Workers
- V8 isolate-based concurrency
- Multiple customers on same server
- No billing during I/O wait times
- Limited Node.js API access (no filesystem)
- Shared V8 instance across multiple applications
Vercel's New Model
- Combines Lambda isolation with Node.js concurrency
- Multiple requests per Lambda instance
- Maintains serverless benefits (auto-scaling, zero maintenance)
Technical Trade-offs & Limitations
- Beta phase: limited concurrent invocations per instance
- Potential latency increase for CPU-bound workloads
- Memory leak risks with shared instances
- File system conflicts if using fixed file names
- Best for I/O-heavy applications, less beneficial for compute-heavy tasks
React/Next.js Connection
- App router increases server-side work and async calls
- Server components reduce client-side API requests
- More granular routes mirror data requirements
- Sophisticated caching mitigates some overhead
- Strong alignment between React, Next.js, and Vercel infrastructure
Availability & Future
- Available for Pro/Enterprise customers
- Simple checkbox toggle in settings
- Maintains all serverless characteristics
- Automatic scaling, maintenance-free operation
- Preview deployments remain cost-effective
