The Art of Staying Flexible: Encoding and Evolution (Designing Data-Intensive Applications Chapter 5)

Image source; Google Gemini

As Heraclitus of Ephesus famously noted, "Everything changes and nothing stands still". This is especially true for modern software, where new features are constantly launched and user requirements shift daily. In Chapter 5 of *Designing Data-Intensive Applications* (Second Edition), "Encoding and Evolution," Martin Kleppmann and Chris Riccomini explore how to build systems that can adapt to this inevitable change without breaking.

The core challenge is that in large systems, code changes cannot happen instantaneously. To avoid downtime, we often use rolling upgrades, where new code is deployed to a few nodes at a time. This means that for a period, different nodes are running different versions of the code, and they must still be able to understand each other’s data. 

To handle this, we need to ensure two types of compatibility:

• Backward Compatibility: New code can read data that was written by old code.
• Forward Compatibility: Old code can read data that was written by new code.

The Evolution of Data Formats

The chapter begins by looking at how we turn data structures into bytes on the wire or on disk, a process known as encoding (or serialization).

While many programming languages have built-in encoding (like Java’s `Serializable`), these are often problematic because they lock you into a specific language, have security vulnerabilities, or fail to provide versioning. This has led to the rise of standardized, language-independent formats.

1. JSON, XML, and CSV

Textual formats like JSON and XML are incredibly popular due to their human-readability. However, they have well-known quirks: they are often ambiguous about numbers (like distinguishing integers from floats), and they don't natively support binary strings without base64 encoding. 

2. Binary Encodings: Protobuf, Thrift, and Avro

To save space and improve performance, many systems use binary formats that require a schema.

• Protocol Buffers (Protobuf) and Thrift: These use field tags (numbers) instead of field names in the encoded data. Because each field has a unique tag, you can add or remove fields in the schema while maintaining compatibility.
• Avro: The most compact of all, Avro does not use tag numbers. Instead, it uses a writer’s schema and a reader’s schema and resolves the differences between them by field name during decoding. This makes Avro particularly well-suited for dynamically generated schemas, such as database dumps.

The Three Modes of Dataflow

Compatibility isn't just a technical detail; it's a relationship between the process that encodes data and the one that decodes it. The authors identify three primary ways data flows through a system:

I. Dataflow Through Databases

When you write to a database, you are effectively "sending a message to your future self". Because data outlives code, a database might contain records written five milliseconds ago alongside records written five years ago. This requires the system to handle both forward and backward compatibility over long periods.

II. Dataflow Through Services (REST and RPC)

In service-oriented or microservice architectures, clients and servers communicate over a network. While REST remains the dominant style for public APIs, internal communication often uses RPC (Remote Procedure Call) frameworks like gRPC. A key design goal here is that clients and servers can be deployed independently, provided the API evolution is handled with care.

This section also introduces durable execution frameworks like Temporal, which provide "exactly-once" semantics for complex workflows by logging every state change and RPC call to durable storage.

III. Event-Driven Architectures

In asynchronous systems, processes communicate by sending events through a message broker (like Kafka or RabbitMQ). This decouples the sender from the recipient, as the sender doesn't wait for a response. Like databases, these systems must be careful to preserve unknown fields when re-publishing messages to ensure data isn't lost when different code versions coexist.

Conclusion

Chapter 5 serves as a bridge between single-node storage and distributed systems. It reminds us that evolvability, the ease with which we can adapt to change, is a first-class requirement. By choosing the right encoding formats and being mindful of compatibility, we can ensure our applications continue to run smoothly even as the world around them changes.

Reference 

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
Book by Martin Kleppmann and Chris Riccomini

This blog post is a summary of my personal notes and understanding from reading "Designing Data-Intensive Applications" by Martin Kleppmann. All credit for the original ideas belongs to the author.