How Discord Stores Trillions of Messages

By early 2022 Discord's message store held trillions of messages across 177 Cassandra nodes. Capacity was fine and the data was correct at every scale. What had degraded was predictability: the on-call team got paged constantly, tail latency wandered, and routine maintenance had become too expensive to run.

Discord fixed the pain on two fronts at once. They swapped the storage engine for ScyllaDB, and they built a Rust service tier in front of it to shield the database from its own worst access pattern. Most retellings focus on the engine swap, but the service tier they put in front of it is the part worth copying.

The first Discord was built in under two months in early 2015, and everything lived in a single MongoDB replica set, deliberately, with a planned exit. Around November 2015 they hit 100 million stored messages, and the data plus its index no longer fit in RAM. Latencies turned unpredictable, and it was time to move.

In 2017 they migrated to Cassandra, chosen for good reasons: linear scaling with no manual re-sharding, automatic failover, proven at scale at Netflix and Apple, and open source. The promise that quietly mattered most was low maintenance, and it was the first one to break at 177 nodes.

Cassandra was the right call for years, meeting a genuine and well-reasoned list of requirements. The 2023 story is what happens when a system grows an order of magnitude past the point where those assumptions held.

Every ID in Discord is a Snowflake, chronologically sortable. Messages are partitioned by the channel they were sent in, along with a bucket, a static ~10-day time window. All messages for a given channel and bucket sit together, replicated across three nodes at replication factor 3.

The bucket exists for a reason. Cassandra warns about partitions over 100MB, since large partitions cause pressure during compaction and can't be distributed. Time-bucketing keeps each partition small. To read recent messages, the service generates a bucket range from now backward and queries partitions in sequence until it has enough.

In Cassandra, reads cost more than writes. A write appends to a commit log and an in-memory memtable; a read must consult the memtable and potentially many on-disk SSTables. Pile enough concurrent reads on one partition and you get a hot partition, and because queries use quorum consistency, every node serving that partition slows down. One busy channel could raise latency across the whole cluster.

Two more taxes stacked on top. Nodes fell behind on compaction, forcing a manual "gossip dance": pull a node from rotation, let it compact, bring it back to catch up via hinted handoff, repeat. And the JVM garbage collector produced pauses ranging from latency spikes to stop-the-world stalls long enough that an operator had to reboot and babysit the node.

Discord was suspicious that a shiny new database alone would fix everything, because hot partitions are an access-pattern problem, not an engine bug. So before touching the engine, they built data services in Rust: thin intermediary services between the API monolith and the database, roughly one gRPC endpoint per query, with no business logic.

The core trick is request coalescing. If many users request the same row at once, the first spins up a worker task and the rest subscribe to it; the worker queries the DB once and hands the result to everyone. Paired with consistent-hash routing on channel ID, every request for a channel lands on the same instance, so coalescing actually catches the duplicates.

An @everyone announcement on a huge server, the classic hot-partition maker, gets absorbed into one database read. Discord is honest that this did not cure hot partitions on Cassandra; it made them rarer and bought time to prepare ScyllaDB. Rust fit because its fearless concurrency and the Tokio ecosystem made safe concurrent code easy to write.

ScyllaDB is Cassandra-compatible but written in C++. The appeal was concrete: a shard-per-core architecture for stronger workload isolation, faster repairs, and, above all, no JVM to tune. As Discord puts it plainly, ScyllaDB is not void of issues, it is void of a garbage collector.

One blocker remained. In testing, reverse queries (scanning against a table's sort order) were too slow for Discord's read path. The ScyllaDB team prioritized and shipped performant reverse queries, an upstream feature driven directly by Discord's use case, which cleared the last obstacle.

The requirement: move trillions of messages with no downtime, and do it fast, because Cassandra was still catching fire. The first plan dual-wrote new data to both databases and backfilled history with ScyllaDB's Spark migrator. After tuning, the estimate came back at three months, plus a pile of dual-world complexity. Rejected.

Instead they extended their existing Rust data-service library into a migrator, in an afternoon. It reads token ranges from Cassandra, checkpoints progress locally in SQLite, and firehoses rows into ScyllaDB. The new estimate: nine days, fast enough to abandon the time-based cutover and migrate everything at once.

The migrator hit speeds up to 3.2 million messages per second, then jammed at 99.9999%. The final token ranges kept timing out because they held gigantic runs of tombstones Cassandra had never compacted away. Compacting that range finished the migration seconds later. The tombstones from 2017, one last time.

They cut over in May 2022, validating by mirroring a small fraction of reads to both databases and comparing results. The node count dropped from 177 Cassandra nodes to 72 ScyllaDB nodes, each holding 9 TB versus Cassandra's ~4 TB average, roughly half the machines and half the disk.

The tail latencies collapsed and, more importantly, stopped wandering. During the 2022 World Cup Final, one of the busiest traffic moments Discord had seen, the latency graph stayed flat. The system finally had headroom to spare.