Scaling Databases with Sharding: Techniques and Challenges

In this episode, we explore two key techniques for scaling databases when they run out of memory: vertical scaling and horizontal scaling. Our focus is on sharding, a powerful form of horizontal scaling that distributes the database across multiple machines, improving performance and capacity. We dive into how sharding works by using a partition function, typically a hash function, to determine which machine holds a particular piece of data. We also discuss the pros and cons of sharding, including the potential for hotspots—where one machine becomes overloaded—and the complexity of remapping data when adding new machines. Lastly, we address the challenge of keeping the database available for read and write operations during remapping. Tune in to learn how to effectively scale your database and navigate the complexities of sharding!