AI summary by TwinMind
Was Porting To Rust A Mistake?
Theo - t3․gg
25m 41s
Migration Overview
- Turbo repo completed migration from Go to Rust after 15 months
- Ported 70,000 lines of Go code to 151,000 lines of Rust code
- JavaScript/TypeScript build system focused on performance and stability
- Used innovative "rust-go-rust sandwich" architecture with Zig compiler assistance
Technical Architecture & Challenges
- Go Sandwich Strategy: Rust binary calls Go binary which contains statically linked Rust libraries
- Cross-platform Compilation: Required building for 6 different platforms
- Windows ARM Issues: Go's MW conflicted with Rust's MSVC, forced to use x86-64 emulation
- Zig CC Compiler: Used for cross-compilation support, bundled libraries for all target platforms
- Incremental Porting: Sandwich allowed gradual migration while maintaining functionality
Key Components Migrated
- File Hashing: Mechanism for creating content fingerprints for cache comparisons
- Lock File Analysis: Dependency checking in repositories
- Cache Signature Verification: Ensuring cache integrity with private keys
- Package Graph: Graph data structure of packages and dependencies
- Prune Command: Removes everything except single package and dependencies
Performance Impact & Caching
- 2 million hours saved in developer compute time (comparable to author's 2.6M YouTube watch hours)
- Cache system uses tar files stored locally and in Vercel remote cache
- Hashing includes source code, environment variables, dependencies
- Global hash for repo-wide changes, task-specific hashes for individual components
- Environment variables compiled into code affect cache validity
Hash Stability & Testing
- Used Cap'n Proto serialization format for cross-language compatibility
- CI runs tasks on both Go and Rust code paths, comparing hashes
- Any hash divergence causes CI failure
- Found bugs in null value handling, package change detection, filter flag processing
- Ensured perfect matching between Go and Rust behavior
Release Process
- Used feature flags for local development without shipping to users
- Built "run outline" - all-Rust version with stubbed functionality
- Dog-fooded internally at Vercel for 72 hours before v1.11 release
- Integration tests matched Go version "log line for log line"
Lessons Learned
- Should have invested in single serialization format from start
- Needed faster shipping of porting strategies
- Code quality improvement before porting would have accelerated process
- Should have fully specified core behaviors (globbing, file watching, hashing) earlier
- Many undocumented behaviors discovered during port
Performance Concerns & Criticisms
- Repo Size: 839MB before installation, massive codebase
- Feature Stagnation: Minimal new features shipped during 15-month migration
- Rust Compilation: Slower compile times compared to Go alternatives
- Development Speed: Contrast with esbuild's 4-month Go development cycle
- Industry Trend: Concern about over-adoption of Rust for JavaScript tooling
Future Plans
- Integration with Vercel ecosystem
- Conformance and code owners features
- Improved stability and CI performance
- Deeper turbo pack integration
