AI summary by TwinMind
A cool new way to think about servers
Theo - t3․gg
37m 8s
Evolution of Compute Models
- Traditional dedicated servers: always-on, handles traffic in parallel, requires manual scaling
- Fixed cost model problematic for variable workloads
- Peak traffic requires expensive servers running 24/7
- Preview deploys need dedicated boxes per branch
- Serverless functions: stateless, spin up on demand, die when not used, auto-scale
- Each instance serves one user at a time
- Concurrency handled by spinning up new VMs vs. within server
- Billing: pay for each VM's usage time per user
- Cost significantly higher than dedicated servers for same workload
AI's Impact on Serverless Economics
- Traditional web requests optimized down to 300ms (upload thing P95)
- AI chat streaming can take 800+ seconds per request
- Most AI request time spent idling, waiting for next token from provider
- Node.js excellent for this pattern due to event loop handling I/O
- Serverless billing becomes prohibitively expensive for long-running, low-compute AI requests
Vercel's Streaming Solution Architecture
- Lambda traditionally didn't support streaming
- AWS Gateway killed Lambda after first chunk sent
- Vercel built custom Edge layer replacing AWS Gateway
- Uses Lambda API invocation mode instead of request-response
- Forms persistent connection (websocket-like) between Lambda and Edge
- Enables streaming responses, wait-until functionality for background tasks
- Edge can route to existing Lambda instances vs. spinning new ones
Function Concurrency Evolution
- Released earlier this year as intermediate step
- Allows multiple requests per Lambda if CPU utilization not too high
- Customer testing showed 20-50% reduction in compute usage
- Still billing for full duration including idle time
- Limited hack on traditional Lambda invocations
Fluid Compute Technical Implementation
- Only bills for actual CPU time used, not idle time
- Built custom Rust runtime layer before function code
- Supports Node.js and Python currently
- Default max duration increased to 800 seconds (from much lower limit)
- 99.8% of requests served hot (no cold starts)
- Can handle multiple concurrent requests efficiently
Performance and Cost Benefits
- 20% compute savings achieved immediately
- Some customers seeing 50-85% cost reductions
- Similar model to Cloudflare Workers (CPU time billing)
- Maintains Node.js compatibility vs. Cloudflare's V8 isolate limitations
- Better than self-hosting on AWS due to complexity of implementation
Industry Context and Comparison
- Google Cloud Run had similar concept from day zero
- Cloudflare Workers use V8 isolates for similar efficiency
- Bill only for CPU time, unlimited duration
- Average 8ms CPU time for upload thing requests
- Lose Node.js compatibility and file system access
- Vercel's solution combines best of both approaches
Implementation Considerations
- Not enabled by default due to concurrency changes
- Only 3 out of thousands of beta customers had issues
- Requires code to be safe for multiple concurrent requests
- No global variables that could conflict between requests
