System designCore12 min

Schema evolution & encoding

Data outlives the code that wrote it. Old phones keep sending last year's format, a rolling deploy runs two versions at once, and a log or warehouse keeps records for years. How you encode records, and which changes you allow to their schema, decides whether old and new code can keep reading each other's data.

1 · The problem

Old and new code read each other's data, for longer than you think

A language-learning app sends a LessonCompleted event each time someone finishes a lesson: who, which lesson, the score, how long it took, and when. The backend uses the events to update streaks, a warehouse keeps them for years, and the team changes the format every few months. Each change meets old versions in three places:

Where	Old and new together	For how long
Rolling deploy	Some servers run the new code, some the old	Minutes to hours
Mobile apps	Phones on last spring's version keep sending its format	Months; some never update
Stored data	Events written by every version since launch	As long as you keep them

So every change has to work in two directions:

Backward compatibility: new code can read data written by old code. The new streak service must read events from old phones and from the warehouse's first year.
Forward compatibility: old code can read data written by new code. An old server still running mid-deploy, or an old phone receiving a response, must cope with fields it has never heard of.

Backward compatibility is the easier one, because the new code's authors know the old format. Forward compatibility needs a promise made in advance: old code must skip what it doesn't understand instead of failing. Whether it can depends heavily on how the data is encoded.

2 · Encoding

JSON spells everything out; binary formats rely on a schema

Here is one event as compact JSON, 91 bytes:

LessonCompleted as JSON

json

{"user_id":48213377,"lesson_id":3071,"score":92,"duration_s":274,"completed_at":1759390000}

Every record carries its field names, and every number is written out as decimal text. Binary formats move the names into a schema, a definition both sides share, and write only the values. The same event in the two most common schema-based formats:

JSON: 91 bytes
Protocol Buffers: 19 bytes
Avro: 15 bytes

Measured by encoding the event with each format's library: 91, 19 and 15 bytes. Protocol Buffers writes a one-byte field number before each value; Avro writes the values alone, in schema order.

Size is only part of it. JSON has no schema, so nothing checks that score is a number, and it is vague about numbers themselves: implementations agree exactly only on integers up to 2⁵³ − 1. JavaScript numbers are doubles, so a 64-bit ID such as 9007199254740993 is parsed as 9007199254740992. APIs that send large IDs to browsers send them as strings (Unique ID generation).

JSON's strength is that anyone can read it, which is why public APIs use it (APIs: REST, gRPC & GraphQL). Its evolution rules are informal: readers should ignore fields they don't know, and nothing enforces that. The binary formats make the rules explicit, in two different ways.

3 · Protocol Buffers

Protocol Buffers identifies fields by number, so numbers are forever

In Protocol Buffers, every field has a number, and the number, not the name, is what goes on the wire. Version 5.2 of the app adds how many hearts were left:

lesson.proto

protobuf

syntax = "proto3";

message LessonCompleted {
  optional int64 user_id      = 1;
  optional int32 lesson_id    = 2;
  optional int32 score        = 3;
  optional int32 duration_s   = 4;
  optional int64 completed_at = 5;
  optional int32 hearts_left  = 6;  // added in app 5.2
}

Each encoded field starts with a tag combining the field number and a wire type, so a parser can always tell how long a field is, even one it doesn't know. An old server reading a 5.2 event meets field 6, can't place it, and keeps it as an unknown field. If it modifies the event and writes it back, field 6 goes out with it. Converting the message to JSON, or copying it field by field into a new message, loses it.

Phone, app 5.2

Old server

Event store

fields 1 to 5, field 6 = 3

field 6 unknown: kept

corrects score: 95

fields 1 to 6

The old server never learns what field 6 means, yet passes it on intact, so hearts_left survives a trip through code written before it existed.

Change	Safe?	Why
Add a field with a new number	Yes	Old code skips it; new code sees the default in old data
Remove a field	Yes, if its number is reserved	Old data still has it; new code skips it
Reuse a removed field's number	Never	Old data decodes into the wrong field
Rename a field	On the wire, yes	Names aren't encoded, but JSON forms use them
Change a field's type	Rarely	int32 to int64 parses, but big values get truncated

Reuse is the dangerous one. Suppose duration_s (field 4) is removed, and a year later someone adds string device = 4. Every stored event and every old phone still writes a number as field 4, and new code tries to read it as a string. Depending on the data, that is a parse error or a silently wrong value. Writing reserved 4; in the schema makes the compiler refuse any future field 4.

Two more rules come from the same idea:

No required fields. A field that must always be present can never be removed, since old readers would reject every message without it. Proto3 dropped them entirely; mark fields optional and document what is required.
Know what "missing" reads as. An absent field reads as its default, zero for a number. With optional, code can ask whether it was set; without it, a score of 0 and a missing score look the same.

4 · Avro

Avro writes no field ids at all, so readers need the writer's schema

Avro is smaller still because it writes nothing but values, in the order the schema lists them. Bytes alone can't even be split into fields. To read them, you need the exact schema they were written with, the writer's schema. The reader also has its own, the reader's schema, and Avro resolves the two field by field, matching by name:

Writer's schema	Reader's schema	Result
Has hearts_left	Doesn't	The value is skipped
Doesn't have hearts_left	Has it, default 5	The reader sees 5
Doesn't have hearts_left	Has it, no default	Error: the record can't be read
int score	long score	Promoted; int widens to long, float or double

So in Avro, the default value is what makes a new field compatible. Add hearts_left without a default and the new streak service can't read a single old event; the library reports there is no default for the field. Renames can use aliases, alternate names the reader's schema lists for a field, but alias support is optional in Avro, so check that your library honors them.

Since every reader needs the writer's schema, it has to travel with the data. Avro's container files store it once in the file header. For one message at a time, as in a Kafka topic, that would be too heavy, so a schema registry stores each schema once and gives it an ID, and every message carries only the ID:

Events API

Schema registry

Kafka topic

Streak service

compatible: ID 7

0 | ID 7 | Avro bytes

same message

schema 7?

v3 (cached after this)

map to its schema

Confluent's format: one version byte, a 4-byte schema ID, then the data. The registry also refuses a new version that breaks the compatibility rule set for the topic.

5 · Compatibility

Choose which direction must hold, and let the registry enforce it

A registry checks each new schema version against older ones before accepting it. The rule is chosen per subject, usually per topic, and it also decides who must upgrade first:

Mode	Guarantees	Upgrade first
BACKWARD (Confluent's default)	New readers read data from the previous version	Consumers
FORWARD	Readers of the previous version read new data	Producers
FULL	Both	Either, independently
…_TRANSITIVE	The same, against every earlier version, not just the last	As above

BACKWARD is the default because it lets consumers rewind to the start of a topic and read everything. The catch is in the word "previous". Plain BACKWARD checks a new version only against the latest one. Each step can pass while the chain breaks:

v1
The original LessonCompleted, with no hearts_left.
v2 adds hearts_left with default 5
Compatible with v1: a v2 reader fills 5 into old events.
v3 removes the default
Compatible with v2 alone under Avro's rules, since every v2 event has the field. The registry accepts it.
The warehouse reloads year one
A v3 reader meets v1 events without hearts_left and no default to fill in. The load fails.

For anything stored for years, such as this warehouse, use BACKWARD_TRANSITIVE, which checks every new version against all earlier ones. It suits Protocol Buffers in general. When a change truly can't be compatible, don't force it into the same topic: publish a new event type or topic and move consumers over (Event-driven architecture).

6 · The hard changes

A rename or a type change is an add, a wait and a remove

The team decides score, an integer from 0 to 100, should become accuracy, a fraction from 0.0 to 1.0. In place, that is a rename and a type change at once, and both readers and stored data would break. Done as a sequence of compatible steps, nothing breaks:

Add the new field
Add optional float accuracy = 7. Old readers skip it.
Write both
App 5.4 and the backend fill score and accuracy. Readers prefer accuracy when present and fall back to score ÷ 100.
Wait for the old writers
Until phones still sending only score are rare enough to accept, often by raising the minimum app version. Stored events keep score forever, so the fallback stays in readers that read history.
Stop writing the old field
Remove score from the schema and reserve number 3 and the name, so neither is reused.

Every step is compatible with the one before, so any step can be rolled back. The waiting is the expensive part, and it is set by the slowest writer: a server fleet upgrades in an hour, phones in months. Databases face the same problem with column renames, and use the same pattern (Zero-downtime migrations).

A few changes have no compatible path at all. Changing what a field means while keeping its name and type, say completed_at switching from UTC to local time, passes every check a registry can make, because the schema didn't change. Schema rules protect the shape of data, not its meaning; a change of meaning needs a new field.

7 · In practice

Pick the encoding by who reads it, and the rule by how long it lives

Format	Fields identified by	Schema needed to read	Fits
JSON	Names, in every record	No	Public APIs, browsers, debugging
Protocol Buffers	Numbers, in every record	No to parse; yes to know the meaning	Service-to-service calls, gRPC
Avro	Position in the writer's schema	Yes, always	Event streams and files with a registry or schema header

For the language app, the checklist:

Assume two versions are always live, for as long as the slowest writer or the oldest stored record.
Only add optional fields, with defaults in Avro; never make a new field required.
Never reuse a field number or name; reserve them when a field is removed.
Make renames and type changes an add, a wait and a remove.
Register schemas and enforce a mode: BACKWARD_TRANSITIVE for anything stored long-term, FULL when producers and consumers deploy independently.
Send big integers to browsers as strings, and treat a change of meaning as a new field.

Check yourself

5 questions

1. The new streak service must process events from phones still running last year's app. Which property does the service's schema need?

2. A year after removing duration_s (field 4), a developer adds string device = 4. What happens to events still arriving from old phones?

3. An Avro schema gains hearts_left with no default. What happens when the new service reads old events?

4. Under the registry's default BACKWARD mode, versions v1, v2 and v3 were each accepted. Why can a v3 reader still fail on v1 data?

5. An old server parses a Protocol Buffers event containing field 6, which it doesn't know, changes the score and writes the event back in binary. Is field 6 still there?

Takeaways

Remember this

Old and new code read each other's data during deploys, across app versions and in storage, so changes must be backward and forward compatible.
JSON carries names in every record and is loose about numbers; binary formats rely on a schema and are several times smaller.
Protocol Buffers identifies fields by number: add with new numbers, reserve removed ones, and never reuse them.
Avro needs the writer's schema to read anything, resolves it against the reader's by name, and needs defaults on added fields.
Let a registry enforce compatibility, transitively for long-lived data, and turn renames and type changes into add, wait and remove.