In this video, the narrator discusses how Discord faced a significant challenge in storing trillions of messages as their user base grew exponentially. Initially using Cassandra for their database, they encountered scaling issues. They ultimately migrated to ScyllaDB, a high-performance NoSQL database written in Rust, which eliminated the issues caused by Java's garbage collector. The video also mentions the use of Rust and gRPC for data services, which improved performance and safety. They implemented coalescing to optimize message reading for multiple users simultaneously. This migration process was complex and a significant undertaking for Discord.
Here are the key facts extracted from the text:
1. Discord stores trillions of messages.
2. In 2017, Discord faced scaling issues with their MongoDB database.
3. Discord switched to a Cassandra database in 2017.
4. By 2021, Discord had over 140 million users.
5. Discord's Cassandra database had grown to 177 nodes with trillions of messages.
6. The Discord engineering team had trouble maintaining the Cassandra database.
7. Discord decided to migrate to a new database called ScyllaDB.
8. ScyllaDB is a high-performance, NoSQL database that is Cassandra-compatible.
9. ScyllaDB is written in C++ and offers many features that set it apart from other databases.
10. Discord chose Rust as the programming language for their data services.
11. Rust is a popular programming language known for its performance, security, and reliability.
12. Discord used Rust and the Tokyo ecosystem to manage asynchronous requests.
13. Discord's data services were designed to optimize database queries and reduce latency.
14. Discord's migration to ScyllaDB took nine days to complete.
15. The migration involved rewriting the migration tool in Rust to make it 10 times faster.
16. After the migration, Discord's message latency improved from 40-125 milliseconds to 15 milliseconds.
17. Inserting a message took 5 milliseconds to 70 milliseconds and is now down to a consistent 5 milliseconds.
18. Discord's new system handled the communication during the World Cup finals without any issues.
19. The migration was a success, and Discord's engineers were able to monitor the system's performance in real-time.
20. The article is based on a blog post by Bo Ingram, a senior software engineer at Discord.