Divide and Conquer: Mastering Sharding (Designing Data-Intensive Applications Chapter 7)
In the previous chapters of our journey through Designing Data-Intensive Applications (Second Edition) , we explored the internals of storage engines and the complexities of replication. But what happens when your dataset grows so massive or your write throughput soars so high that a single machine, even with replicas, simply cannot keep up? This is where Chapter 7, "Sharding," comes in. Authors Martin Kleppmann and Chris Riccomini shift the focus from copying the same data (replication) to splitting the data into smaller subsets, known as shards or partitions . Sharding by Any Other Name While the authors use the term "sharding," they note that the industry is full of synonyms. Depending on the tool you use, you might hear shards called partitions (Kafka), regions (HBase), tablets (Bigtable), or vnodes (Riak). Regardless of the name, the goal is the same: to break a large database into smaller, self-contained pieces where each record belongs to exactly one...
