Summary by TwinMind · 25m 41s

Summary by TwinMind · 25m 41s

Was Porting To Rust A Mistake?

Was Porting To Rust A Mistake?

Theo - t3․gg

Theo - t3․gg

Migration Overview

  • Turbo repo completed migration from Go to Rust after 15 months
  • Ported 70,000 lines of Go code to 151,000 lines of Rust code
  • JavaScript/TypeScript build system focused on performance and stability
  • Used innovative "rust-go-rust sandwich" architecture with Zig compiler assistance

Technical Architecture & Challenges

  • Go Sandwich Strategy: Rust binary calls Go binary which contains statically linked Rust libraries
  • Cross-platform Compilation: Required building for 6 different platforms
  • Windows ARM Issues: Go's MW conflicted with Rust's MSVC, forced to use x86-64 emulation
  • Zig CC Compiler: Used for cross-compilation support, bundled libraries for all target platforms
  • Incremental Porting: Sandwich allowed gradual migration while maintaining functionality

Key Components Migrated

  • File Hashing: Mechanism for creating content fingerprints for cache comparisons
  • Lock File Analysis: Dependency checking in repositories
  • Cache Signature Verification: Ensuring cache integrity with private keys
  • Package Graph: Graph data structure of packages and dependencies
  • Prune Command: Removes everything except single package and dependencies

Performance Impact & Caching

  • 2 million hours saved in developer compute time (comparable to author's 2.6M YouTube watch hours)
  • Cache system uses tar files stored locally and in Vercel remote cache
  • Hashing includes source code, environment variables, dependencies
  • Global hash for repo-wide changes, task-specific hashes for individual components
  • Environment variables compiled into code affect cache validity

Hash Stability & Testing

  • Used Cap'n Proto serialization format for cross-language compatibility
  • CI runs tasks on both Go and Rust code paths, comparing hashes
  • Any hash divergence causes CI failure
  • Found bugs in null value handling, package change detection, filter flag processing
  • Ensured perfect matching between Go and Rust behavior

Release Process

  • Used feature flags for local development without shipping to users
  • Built "run outline" - all-Rust version with stubbed functionality
  • Dog-fooded internally at Vercel for 72 hours before v1.11 release
  • Integration tests matched Go version "log line for log line"

Lessons Learned

  • Should have invested in single serialization format from start
  • Needed faster shipping of porting strategies
  • Code quality improvement before porting would have accelerated process
  • Should have fully specified core behaviors (globbing, file watching, hashing) earlier
  • Many undocumented behaviors discovered during port

Performance Concerns & Criticisms

  • Repo Size: 839MB before installation, massive codebase
  • Feature Stagnation: Minimal new features shipped during 15-month migration
  • Rust Compilation: Slower compile times compared to Go alternatives
  • Development Speed: Contrast with esbuild's 4-month Go development cycle
  • Industry Trend: Concern about over-adoption of Rust for JavaScript tooling

Future Plans

  • Integration with Vercel ecosystem
  • Conformance and code owners features
  • Improved stability and CI performance
  • Deeper turbo pack integration