AI summary by TwinMind

So I've Been Trying Other Languages...

Theo - t3․gg

30m 59s

Project Background

  • 1 billion row challenge recorded in February, delayed 3 months
  • Testing file reading performance across different languages
  • File: measurements.txt, 13.8 GB, 1 billion rows
  • Baseline: wc -l takes 11.93 seconds to count lines

Go Implementation - Basic Version

  • Simple single-threaded solution using bufio scanner
  • Function readFile with deferred file close pattern
  • Scanner.Scan() reads line by line by default
  • Build time: 0.08 seconds (impressive vs TypeScript)
  • Performance: 15.05 seconds (without reading text), 16.91 seconds (with scanner.Text())
  • Go lacks built-in watch feature, requires external tools like air
  • Language focuses on code quality over developer experience features

Go Implementation - Optimized Version

  • Uses bytes.Count() with newline separator
  • Reads in 32KB chunks (32 * 1024 buffer)
  • Counts newline bytes in each chunk
  • Performance: 3.2 seconds (significantly faster than wc -l at 11 seconds)
  • Proves task isn't purely I/O bound

JavaScript/Node.js Implementation

  • Uses readline.createInterface() with file stream
  • crlfDelay: Infinity option for cross-platform line breaks
  • Async/await syntax for line iteration
  • Performance: 223 seconds (full order of magnitude slower than Go)
  • Attempted parallel processing but got wrong answers or errors

Gleam Implementation - Initial Version

  • Uses file_stream package for Erlang file streams
  • Functional programming approach with recursion instead of loops
  • Pattern matching with case statements (ok/error handling)
  • Recursive function calls (1 billion times due to tail call optimization)
  • Performance: 1,571 seconds (~26 minutes, 12x slower than JavaScript)
  • Language lacks explicit return statements (last expression returned)

Gleam Performance Issues Analysis

  • Filed issue with library maintainer about slow performance
  • Library author (Louie) responsive but initially missed performance severity
  • Root cause: file I/O bouncing context via Erlang processes (lots of copying)
  • Raw mode disabled, using utf8 line reading instead of binary streams
  • Testing showed 15+ minutes for billion line count

Gleam Implementation - Optimized Version

  • Community member (Yakomo) provided Erlang-based solution
  • Required dropping to native Erlang code for performance
  • Performance: ~115 seconds (faster than JS, slower than Go)
  • Highlights need to write Erlang for optimal performance

Bun Attempt

  • Bun's file API lacks reasonable line-by-line reading
  • Attempted readline implementation ran for 1 hour using all CPU
  • Currently slowest solution tested
  • Jared proposed implementing bun.lines() in Zig layer

Language Comparison Results

  • Go (optimized): 3.2 seconds
  • wc -l baseline: 11.93 seconds
  • Go (basic): ~16 seconds
  • Rust (mentioned): ~53 seconds
  • Gleam (optimized): ~115 seconds
  • JavaScript/Node: 223 seconds
  • Gleam (basic): 1,571 seconds
  • Bun: Failed/extremely slow

Key Insights

  • Go's simple solution performs excellently with minimal optimization
  • Ecosystem and standard library quality matters more than raw language speed
  • Go's bufio package provides excellent performance out of the box
  • New languages face standard library maturity challenges
  • Team responsiveness and community support crucial for adoption

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome