The Copycat Problem: Navigating Replication (Designing Data-Intensive Applications Chapter 6)

In our journey through Designing Data-Intensive Applications, we have covered everything from architectural trade-offs to the internal mechanics of storage engines. But so far, we have largely assumed our data lives on a single machine. In Chapter 6, "Replication," authors Martin Kleppmann and Chris Riccomini tackle the challenge of keeping copies of the same data on multiple machines connected via a network.

Replication isn't just about having a backup; it's a critical tool for achieving high availability (keeping the system running if a node fails), reduced latency (placing data closer to users), and increased read throughput (scaling out the number of machines that can serve queries). However, if the data changes over time, replication becomes remarkably tricky.

The Three Families of Replication

Almost all distributed databases fall into one of three architectural families:

1. Single-Leader Replication

This is the most common approach, used by PostgreSQL, MySQL, and Kafka. One replica is designated as the leader (or primary), and all writes must be sent to it. The leader then sends a stream of these changes to followers as part of a replication log. 

• The Trade-off: You must choose between synchronous replication (the leader waits for followers to confirm the write, ensuring durability but risking availability if a follower is slow) and asynchronous replication (the leader doesn't wait, which is faster but risks data loss if the leader fails before replicating).

2. Multi-Leader Replication

Commonly used for systems distributed across multiple regions or for local-first software that needs to work offline, this model allows more than one node to accept writes. Each leader simultaneously acts as a follower to the other leaders.

• The Challenge: The biggest problem here is conflicting writes. If two users edit the same record on different leaders at the same time, the system needs a way to resolve the clash, such as Last Write Wins (LWW) or more sophisticated Conflict-free Replicated Datatypes (CRDTs).

3. Leaderless Replication

Popularized by Amazon's original Dynamo system, this approach allows clients to send writes directly to several replicas. 

• The Quorum Rule: To ensure consistency, leaderless systems rely on quorums. If you have n replicas, every write must be confirmed by w nodes, and every read must query r nodes. As long as w + r > n, you expect to get an up-to-date value because the read and write sets will overlap.

The Reality of Replication Lag

While replication offers great benefits, it introduces the headache of eventual consistency. If a user writes to a leader but then reads from a lagging follower, they might see a "disappearing" update. To combat this, the authors highlight three essential consistency models:

• Read-after-write consistency: Ensuring users always see updates they submitted themselves.
• Monotonic reads: Guaranteeing that once a user has seen a certain piece of data, they won't later see it "move backward" in time.
• Consistent prefix reads: Ensuring that if a sequence of writes happens in a certain order, anyone reading them will see them in that same order (critical for preserving causality).

Conclusion

Chapter 6 serves as a powerful reminder that in distributed systems, there is no such thing as a free lunch. Every benefit of replication, whether it's fault tolerance or performance, comes with a cost in complexity. By understanding the "biblical" trade-offs between these three replication models, you can better design systems that remain robust even when the network is unreliable and machines inevitably fail.

Reference 

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems

Book by Martin Kleppmann and Chris Riccomini

This blog post is a summary of my personal notes and understanding from reading "Designing Data-Intensive Applications" by Martin Kleppmann. All credit for the original ideas belongs to the author.