Project Background

• 1 billion row challenge recorded in February, delayed 3 months
• Testing file reading performance across different languages
• File: measurements.txt, 13.8 GB, 1 billion rows
• Baseline: wc -l takes 11.93 seconds to count lines

Go Implementation - Basic Version

• Simple single-threaded solution using bufio scanner
• Function readFile with deferred file close pattern
• Scanner.Scan() reads line by line by default
• Build time: 0.08 seconds (impressive vs TypeScript)
• Performance: 15.05 seconds (without reading text), 16.91 seconds (with scanner.Text())
• Go lacks built-in watch feature, requires external tools like air
• Language focuses on code quality over developer experience features

Go Implementation - Optimized Version

• Uses bytes.Count() with newline separator
• Reads in 32KB chunks (32 * 1024 buffer)
• Counts newline bytes in each chunk
• Performance: 3.2 seconds (significantly faster than wc -l at 11 seconds)
• Proves task isn't purely I/O bound

JavaScript/Node.js Implementation

• Uses readline.createInterface() with file stream
• crlfDelay: Infinity option for cross-platform line breaks
• Async/await syntax for line iteration
• Performance: 223 seconds (full order of magnitude slower than Go)
• Attempted parallel processing but got wrong answers or errors

Gleam Implementation - Initial Version

• Uses file_stream package for Erlang file streams
• Functional programming approach with recursion instead of loops
• Pattern matching with case statements (ok/error handling)
• Recursive function calls (1 billion times due to tail call optimization)
• Performance: 1,571 seconds (~26 minutes, 12x slower than JavaScript)
• Language lacks explicit return statements (last expression returned)

Gleam Performance Issues Analysis

• Filed issue with library maintainer about slow performance
• Library author (Louie) responsive but initially missed performance severity
• Root cause: file I/O bouncing context via Erlang processes (lots of copying)
• Raw mode disabled, using utf8 line reading instead of binary streams
• Testing showed 15+ minutes for billion line count

Gleam Implementation - Optimized Version

• Community member (Yakomo) provided Erlang-based solution
• Required dropping to native Erlang code for performance
• Performance: ~115 seconds (faster than JS, slower than Go)
• Highlights need to write Erlang for optimal performance

Bun Attempt

• Bun's file API lacks reasonable line-by-line reading
• Attempted readline implementation ran for 1 hour using all CPU
• Currently slowest solution tested
• Jared proposed implementing bun.lines() in Zig layer

Language Comparison Results

• Go (optimized): 3.2 seconds
• wc -l baseline: 11.93 seconds
• Go (basic): ~16 seconds
• Rust (mentioned): ~53 seconds
• Gleam (optimized): ~115 seconds
• JavaScript/Node: 223 seconds
• Gleam (basic): 1,571 seconds
• Bun: Failed/extremely slow

Key Insights

• Go's simple solution performs excellently with minimal optimization
• Ecosystem and standard library quality matters more than raw language speed
• Go's bufio package provides excellent performance out of the box
• New languages face standard library maturity challenges
• Team responsiveness and community support crucial for adoption