AI summary by TwinMind
The Secret Language Scaling WhatsApp and Discord
Theo - t3․gg
28m 31s
Introduction to Erlang/Elixir for Scaling
- Big companies use Rust/Go but secret weapon: Erlang from 1980s
- Built for phone companies, now powers real-time apps
- Elixir: modern syntax on top of Erlang VM (BEAM)
- Author excited about Elixir Conference August 27-30 Orlando
Origins and History
- Erlang developed late 1980s by Ericsson (Swedish telecom)
- Goal: massive concurrency + high availability for telecom systems
- Pre-internet era - designed for phone companies
- Lightweight processes, message passing, fault tolerance
- BEAM (Bogdan/Bjorn Erlang Abstract Machine) - virtual machine concept
- Revolutionary at time - abstract machine vs direct hardware compilation
Elixir Creation Story
- Created 2011 by Jose Valim (Ruby developer)
- Built OAuth implementation for Ruby
- Frustrated with Ruby's single-threaded scaling limitations
- Load balancing invented largely to handle Ruby's scaling issues
- Loved Erlang capabilities but hated syntax
- Combined Erlang power with Ruby-like beautiful syntax
- Added functional programming features
Five Core Features Making BEAM Powerful
1. Fault Tolerance
- "Let it crash" philosophy
- Individual process failures don't crash entire system
- Supervisors automatically restart failed processes
- Processes isolated - failures contained
- Hot code swapping possible - change code live without downtime
2. Scalability
- Handles inter-node communication seamlessly
- Simple node connection:
Node.connect("node_two@ip") - Run code on different machines:
Node.spawn(function) - GenServer for distributed computing
- Chunks arrays across multiple nodes automatically
- No complex distributed systems like Kafka/RabbitMQ needed
3. Distribution
- Treats local/remote processes identically
- Easy to add nodes to cluster
- No complex load balancers or sharding strategies
- Beam handles distribution automatically
- Same source code assumed available on all nodes
4. Responsiveness
- Preemptive scheduler with execution windows
- Max 500 function calls per process before switching
- Similar to JavaScript event loop but multi-node
- IO operations delegated to separate threads
- No process blocking from long-running operations
5. Live Code Updates
- Hot code swapping without restarting application
- Critical for real-time apps requiring continuous operation
- Update WebSocket servers without disconnecting users
- Fix bugs/add features without service interruption
Performance Philosophy Discussion
- Performance != just raw computation speed
- System performance more important than individual function speed
- Rust faster for single functions, Elixir better for distributed systems
- Elixir makes scaling to multiple servers trivial
- Parallelization easier than micro-optimizations
- When need more performance: spin up another server
Case Study 1: WhatsApp
- Founded 2009 by Brian Acton & John Koum
- Chose Erlang for fault tolerance, scalability, distribution
- Based on ejabberd (open source XMPP server in Erlang)
- Exponential growth to 1+ billion daily active users by 2018
- At acquisition: 32 engineers total, ~6 on infrastructure
- $19 billion acquisition with 6 engineers serving 450M users
- Handled millions messages/day with minimal downtime
- Anton Lavrik (WhatsApp engineer): concurrency Erlang's best attribute
Case Study 2: Discord
- Founded 2015 by Jason Citron & Stanislav Vishnevsky
- Originally for gamers, chose Python + Elixir
- Core real-time chat still powered by Elixir
- Popular servers (Fortnite, Minecraft) handle hundreds of thousands concurrent users
- 2019 milestone: 11+ million concurrent users
- Used Rust for member list via NIFs (Native Implemented Functions)
- Combined Rust performance with Elixir scalability/productivity
Case Study 3: Food Focus App
- Author's personal project: live cooking video + chat
- Smartwatch integration with synchronized recipe timers
- Tech requirements: real-time features, ready-to-use tools, good DX
- Chose Elixir/Phoenix backend (Flutter frontend - author regrets)
- Key tools used:
- Phoenix Channels (WebSocket-based real-time API)
- ETS (Erlang Term Storage) - built-in key-value store
- Ecto (database operations)
- Oban (background jobs)
- Electric SQL (local-first queries)
- Delivered on time and budget
- Less time configuring third-party libraries, more time building features
Other Companies Using Erlang/Elixir
- Spotify, Pinterest, PepsiCo, Financial Times, Heroku
- Fly.io (built entirely on Elixir)
- Twitch (used for marathons)
- All achieve high concurrency, fault tolerance, scalability
Comparison with Other Languages
- Gleam: same benefits when using Erlang backend
- JavaScript: similar event loop concepts but single-node
- Go: easier parallelization than Rust
- Rust: great for single-core performance, struggles with async/concurrency
- Java: similar VM concept but not used for distribution
Technical Details
- Lightweight processes: thousands/millions concurrent without performance degradation
- Message passing without shared memory
- Node discovery and orchestration built-in
- Pipe operator for functional programming
- Pattern matching (one of earliest languages with it, after ML)
- RPC calls across nodes trivial
