AI summary by TwinMind

A cool new way to think about servers

Theo - t3․gg

37m 8s

Evolution of Compute Models

  • Traditional dedicated servers: always-on, handles traffic in parallel, requires manual scaling
    • Fixed cost model problematic for variable workloads
    • Peak traffic requires expensive servers running 24/7
    • Preview deploys need dedicated boxes per branch
  • Serverless functions: stateless, spin up on demand, die when not used, auto-scale
    • Each instance serves one user at a time
    • Concurrency handled by spinning up new VMs vs. within server
    • Billing: pay for each VM's usage time per user
    • Cost significantly higher than dedicated servers for same workload

AI's Impact on Serverless Economics

  • Traditional web requests optimized down to 300ms (upload thing P95)
  • AI chat streaming can take 800+ seconds per request
  • Most AI request time spent idling, waiting for next token from provider
  • Node.js excellent for this pattern due to event loop handling I/O
  • Serverless billing becomes prohibitively expensive for long-running, low-compute AI requests

Vercel's Streaming Solution Architecture

  • Lambda traditionally didn't support streaming
  • AWS Gateway killed Lambda after first chunk sent
  • Vercel built custom Edge layer replacing AWS Gateway
  • Uses Lambda API invocation mode instead of request-response
  • Forms persistent connection (websocket-like) between Lambda and Edge
  • Enables streaming responses, wait-until functionality for background tasks
  • Edge can route to existing Lambda instances vs. spinning new ones

Function Concurrency Evolution

  • Released earlier this year as intermediate step
  • Allows multiple requests per Lambda if CPU utilization not too high
  • Customer testing showed 20-50% reduction in compute usage
  • Still billing for full duration including idle time
  • Limited hack on traditional Lambda invocations

Fluid Compute Technical Implementation

  • Only bills for actual CPU time used, not idle time
  • Built custom Rust runtime layer before function code
  • Supports Node.js and Python currently
  • Default max duration increased to 800 seconds (from much lower limit)
  • 99.8% of requests served hot (no cold starts)
  • Can handle multiple concurrent requests efficiently

Performance and Cost Benefits

  • 20% compute savings achieved immediately
  • Some customers seeing 50-85% cost reductions
  • Similar model to Cloudflare Workers (CPU time billing)
  • Maintains Node.js compatibility vs. Cloudflare's V8 isolate limitations
  • Better than self-hosting on AWS due to complexity of implementation

Industry Context and Comparison

  • Google Cloud Run had similar concept from day zero
  • Cloudflare Workers use V8 isolates for similar efficiency
    • Bill only for CPU time, unlimited duration
    • Average 8ms CPU time for upload thing requests
    • Lose Node.js compatibility and file system access
  • Vercel's solution combines best of both approaches

Implementation Considerations

  • Not enabled by default due to concurrency changes
  • Only 3 out of thousands of beta customers had issues
  • Requires code to be safe for multiple concurrent requests
  • No global variables that could conflict between requests

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome