AI summary by TwinMind

Fixing serverless node.js (by adding servers?)

Theo - t3․gg

52m 7s

Introduction & Sponsorship

  • Vercel shipped in-function concurrency feature after ending sponsorship relationship
  • Episode not sponsored by Vercel
  • Unkey sponsorship demo: API rate limiting setup
    • Added rate limiter with 2 requests per 30 seconds per IP
    • Open source solution, can self-host or pay for service

Expert Opinions on Vercel's Innovation

  • AJ (AWS Hero, DataDog staff engineer): remarkable accomplishment, hopes AWS builds similar features
  • Dax: Previously critical of Vercel but acknowledges this is "pretty cool"
  • Feature enables serious workloads on Vercel platform
  • Industry consensus: significant innovation in serverless space

Serverless History & Node.js Fundamentals

  • Lambda revolutionized batch workloads with 1:1 function-to-invocation mapping
  • Traditional serverless: each request gets own machine/server
  • Problem: wasted CPU cycles during I/O wait times
  • Ryan Dahl's original Node.js vision (2009): event-driven, non-blocking I/O
    • Millions of clock cycles wasted waiting for database/network responses
    • Callback-based async model allowed concurrent request handling
    • "This is how I/O should be done for fast, high-concurrency servers"

Vercel's Technical Implementation

  • 20-50% reduction in compute usage and costs without latency impact
  • Built for interactive workloads: server rendering, APIs, AI applications
  • Native Node support with V8 bytecode acceleration and instance pre-warming

Architecture Components

  • Edge routing layer
  • Function load balancer
  • Function invocation service (manages up to 100k instances per function)
  • Rust-powered runtime rewrite for performance and stability
  • Bidirectional TCP socket streaming back to invocation service

Database Connection Problem Solved

  • Traditional serverless: each request creates new database connection
  • Connection limits (e.g., Heroku free tier: 10 connections)
  • Cold start delays from connection establishment
  • New model: connection reuse across multiple requests in same instance
  • Reduces need for heavy connection pooling systems

Efficiency Gains & Real-World Results

  • Beta customers seeing 20-50% compute usage reduction
  • Verse example: 50% cost reduction during traffic surge (10M+ visitors)
  • Another customer: 35% savings on 3000+ compute hours
  • Author's limited results due to low traffic patterns
  • Savings scale with request volume and I/O wait times

Concurrency Model Comparisons

Traditional VPS

  • Concurrency handled entirely in Node.js
  • Single server, multiple requests, shared database connection

Lambda (Traditional)

  • Concurrency through separate server instances
  • Each request gets own machine and database connection
  • No shared resources between requests

Cloudflare Workers

  • V8 isolate-based concurrency
  • Multiple customers on same server
  • No billing during I/O wait times
  • Limited Node.js API access (no filesystem)
  • Shared V8 instance across multiple applications

Vercel's New Model

  • Combines Lambda isolation with Node.js concurrency
  • Multiple requests per Lambda instance
  • Maintains serverless benefits (auto-scaling, zero maintenance)

Technical Trade-offs & Limitations

  • Beta phase: limited concurrent invocations per instance
  • Potential latency increase for CPU-bound workloads
  • Memory leak risks with shared instances
  • File system conflicts if using fixed file names
  • Best for I/O-heavy applications, less beneficial for compute-heavy tasks

React/Next.js Connection

  • App router increases server-side work and async calls
  • Server components reduce client-side API requests
  • More granular routes mirror data requirements
  • Sophisticated caching mitigates some overhead
  • Strong alignment between React, Next.js, and Vercel infrastructure

Availability & Future

  • Available for Pro/Enterprise customers
  • Simple checkbox toggle in settings
  • Maintains all serverless characteristics
  • Automatic scaling, maintenance-free operation
  • Preview deployments remain cost-effective

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome