Serialization and schema evolution

Serialization turns an object in memory into bytes that can cross a network or sit on a disk, and deserialization rebuilds an object from those bytes, often in another language, on another machine, months later. Schema evolution is the craft of changing that data’s shape (adding a field, retiring one, widening a type) while every reader and writer still running the old shape keeps working. The format decides how big the bytes are; the evolution rules decide whether next Tuesday’s deploy breaks anyone.

Picture it first: three kinds of courier slip

Over the years a courier company has accepted three kinds of booking slip.

• A handwritten note: “Name: Asha, Pincode: 560001, Weight: 2 kg”. Anyone can read it, and a clerk who has never seen a new “Fragile: yes” line just skips it. But every note repeats every label.
• A numbered form: boxes 1, 4 and 7 filled in, no labels. Each clerk keeps a legend saying box 4 is the pincode. Slips are shorter, and an old clerk can skip a new box 9 because every box has edges. One rule is sacred: reprint the form with box 4 as a phone number and every old slip in the archive is misread.
• An unlabelled strip: “Asha | 560001 | 2”. Shortest of all, and meaningless unless you know which template the sender used, so every strip carries a template number and the counter keeps a binder of every template ever printed.

Map it back: the note is JSON (names in every message), the numbered form is Protocol Buffers (field numbers in every message), and the strip is Avro with a schema registry. A company reprinting its forms while old slips sit in the archive and old clerks still staff the counters is every system that changes its data model: schema evolution.

What serialization does, and why you can’t just send memory

An object in memory is not a tidy run of bytes. A Java Order is a header plus references, with its status string elsewhere on the heap; a Python dict is a hash table of pointers. Copying those bytes onto the wire fails for three reasons:

• Pointers mean nothing elsewhere. An address on your machine points at nothing on mine. A serializer follows references and writes the values inline, which is why it is also called flattening.
• Layouts differ. Byte order, integer widths, struct padding and string encodings vary by CPU, compiler and language.
• The reader is different code. A Go service, a Kotlin app and a Python job must agree on one byte format that belongs to none of their object models.

So the sender encodes (serializes, marshals) the object into a defined format, and the receiver decodes (deserializes, unmarshals) it into its own objects. Everything that crosses a process boundary goes through this step: RPC and HTTP bodies, webhook payloads, Kafka messages, cache values, database blobs. The API style that carries the bytes is a separate choice (REST vs GraphQL vs gRPC); this page is about the bytes.

Why language-native serializers are a trap

Java’s Serializable, Python’s pickle and Ruby’s Marshal serialize almost any object in one call: handy inside one program, a liability between services.

• Lock-in. A pickle can only be read by Python, with the same classes importable.
• Accidental versioning. The bytes mirror your class definitions, so a refactor silently changes the format.
• Code execution. These formats name classes to construct and can run code while decoding. The Python documentation is blunt: pickle “is not secure”, and crafted data can “execute arbitrary code during unpickling”.

Text and binary formats

Cross-language formats fall into three families, plus one niche.

Text formats (JSON, XML, CSV) are human-readable and supported everywhere. JSON and XML carry field names in every record. You pay in size and parse time, and JSON has no integer, date or raw-bytes type, so binary data travels as base64, turning every 3 bytes into 4 characters (33% bigger).

Binary, schemaless formats (MessagePack, and CBOR, the IETF standard in RFC 8949) keep JSON’s data model but write type headers and binary integers instead of text. They parse faster, yet still repeat every key name, so the size saving is modest.

Binary, schema-based formats (Protocol Buffers from Google, Apache Avro from the Hadoop world, Apache Thrift from Facebook) move field names into a schema. Protobuf and Thrift replace each name with a small field number; Avro drops even that and relies on field order. Messages are small, decoding is fast and you get generated classes; the price is bytes that are opaque without the schema.

Zero-copy formats such as FlatBuffers and Cap’n Proto let the reader access fields in place inside the received buffer, with no decode step, at the cost of alignment padding and a clumsier API.

JSON numbers and the 2^53 trap

RFC 8259 lets each JSON implementation choose its own number range. It only promises that integers between -(2^53)+1 and 2^53-1 are interoperable, because so much software, JavaScript included, stores every number as an IEEE 754 double.

A double has 1 sign bit, 11 exponent bits and 52 fraction bits plus an implied leading 1, so it holds integers exactly up to 53 bits. The largest integer you can trust, with every integer below it, is 2^53 − 1 = 9,007,199,254,740,991, JavaScript’s Number.MAX_SAFE_INTEGER. Above that the gaps begin: JSON.parse("9007199254740993") returns 9007199254740992. A 64-bit snowflake-style order ID such as 1234567890123456789 comes back as 1234567890123456768, a different order, and nothing throws. Signed 64-bit integers reach 2^63 − 1, 2^10 = 1,024 times further, and timestamp-prefixed IDs live up there.

Money has the mirror problem: doubles can’t represent most decimal fractions, so 0.1 + 0.2 is 0.30000000000000004. The fixes are boring and essential:

• Send 64-bit IDs as strings: "id": "1234567890123456789".
• Send money as integer minor units plus a currency ("amount_paise": 49900, "currency": "INR" for ₹499.00) or as a decimal string, never as a float.
• Send timestamps as RFC 3339 strings with an explicit offset (2026-10-27T16:23:25+05:30) or as UTC epoch integers whose unit is in the name (created_at_ms).

One order, byte by byte

Take a Swiggy-style order with three fields: id 1500, status “PAID” and amount_paise 49900 (₹499). Here it is in four formats.

JSON, minified, is {"id":1500,"status":"PAID","amount_paise":49900}: 48 bytes. Only 13 of them are the values themselves (1500, PAID, 49900). The key names with their quotes take 4 + 8 + 14 = 26 bytes, the quotes around PAID take 2, and braces, colons and commas take 7: 26 + 2 + 7 = 35 bytes of overhead.

CBOR or MessagePack writes the same map in 35 bytes: a 1-byte header per string instead of quotes, and each integer as a type byte plus 2 binary bytes. The key names are still all there.

Protobuf needs a schema first:

syntax = "proto3";
message Order {
  int64  id           = 1;
  string status       = 2;
  int64  amount_paise = 3;
}

Each field becomes a record: a tag, then the value. The tag is (field_number << 3) | wire_type written as a varint, where wire type 0 means “a varint follows” and 2 means “a length, then that many bytes”. A varint stores 7 bits of the number per byte, lowest group first, and sets the top bit of every byte except the last to say “more follows”.

• id: tag (1 << 3) | 0 = 8 = 08. Then 1500 = 11 × 128 + 92: the low group is 92, sent with the top bit set as 92 + 128 = 220 = DC, then 11 = 0B. 3 bytes.
• status: tag (2 << 3) | 2 = 18 = 12, length 04, then 50 41 49 44 (“PAID” in ASCII). 6 bytes.
• amount_paise: tag (3 << 3) | 0 = 24 = 18. Then 49900 = 389 × 128 + 108 and 389 = 3 × 128 + 5, so the groups are 108, 5 and 3: EC 85 03. 4 bytes.

That is 3 + 6 + 4 = 13 bytes.

Avro has a schema too, written in JSON:

{"type": "record", "name": "Order", "fields": [
  {"name": "id",           "type": "long"},
  {"name": "status",       "type": "string"},
  {"name": "amount_paise", "type": "long"}
]}

Avro writes the values in schema order with no tags at all. Integers are zig-zag encoded before the varint, mapping 0, −1, 1, −2 … to 0, 1, 2, 3 …, so a non-negative n becomes 2n and small negative numbers stay small.

• id: 1500 becomes 3000 = 23 × 128 + 56, so B8 17 (56 + 128 = 184, then 23). 2 bytes.
• status: the length 4 becomes 8 = 08, then the four letters. 5 bytes.
• amount_paise: 49900 becomes 99800 = 779 × 128 + 88, and 779 = 6 × 128 + 11, so D8 8B 06. 3 bytes.

That is 2 + 5 + 3 = 10 bytes.

What the numbers mean for a real system

Avro wins on size because it sends nothing but values, which is also why it cannot be read without the writer’s schema. Naming that schema costs bytes: with Confluent’s schema registry each Kafka record starts with a magic byte (0) and a 4-byte schema ID, so the 10-byte Avro order becomes 15 bytes; Protobuf gets the same 5 bytes plus a message index, usually one byte, so 13 becomes 19. On a 2 KB event that framing is noise; on a 10-byte body it adds 50%.

Compression narrows the gap further. In a quick experiment with 1,000 random orders shaped like this one, a JSON array came to 55,028 bytes and Avro to 14,220, a 3.9× gap. After gzip they were 10,146 and 7,769 bytes, only 1.3× apart, because gzip squeezes out the repeated key names. The catch: compression costs CPU on both ends and does nothing for single small messages; gzipping the 48-byte order alone produced 68 bytes. HTTP negotiates compression for you (see HTTP and HTTPS); Kafka compresses whole batches.

So schema-based binary pays off when volume or CPU genuinely matters: hot internal RPCs, Kafka topics with billions of events a day, phones on patchy 3G during a Flipkart sale. Elsewhere, JSON’s readability in logs and curl is often worth more than the bytes.

Data outlives code

If every reader and writer upgraded at the same instant, evolution would be trivial. They never do; as the Protobuf best-practices guide warns, even when you try, one side may be rolled back. Old and new shapes always overlap:

• Rolling and canary deploys. For minutes or hours, v1 and v2 instances serve side by side, and a rollback puts v1 back after v2 has already written data (see deployment strategies).
• Data at rest. A Kafka topic keeps records for days, or indefinitely with long retention or compaction. Database rows and archived files live for years. A consumer that replays from offset 0 reads every version ever written.
• Clients you don’t control. Mobile users update weeks late, and a partner’s integration may never be redeployed.

The definitions, from the reader’s point of view:

• Backward compatible: new code can read data written by old code. The reader looks back in time.
• Forward compatible: old code can read data written by new code, ignoring what it doesn’t understand. Data from the future reaches an old reader.
• Full: both at once.

Registry modes and deploy order

A schema registry turns those definitions into an automatic check on every new schema version. Confluent’s modes are the de facto vocabulary:

BACKWARD is the default because it lets consumers rewind to the start of a topic. The non-transitive modes only compare with the last version, so a topic that keeps data for a year wants BACKWARD_TRANSITIVE or FULL_TRANSITIVE; Confluent recommends BACKWARD_TRANSITIVE for Protobuf.

Protobuf: the field number is the identity

The bytes carry numbers and wire types, never names, so a field is its number. Numbers run from 1 to 536,870,911 (2^29 − 1, because 3 of a 32-bit tag’s bits hold the wire type), and 1 to 15 fit in a one-byte tag, so give them to your most common fields. Version 3 of the order might look like this:

message Order {
  reserved 4;                   // was: string coupon_code
  reserved "coupon_code";
  int64  id           = 1;
  string status       = 2;
  int64  amount_paise = 3;
  optional int64 tip_paise = 5; // added in v3
}

The rules follow from the encoding:

• Never change or reuse a number. Changing one is a delete plus an add. Reusing one is worse: if old records hold discount_paise as field 4 and someone reuses 4 for loyalty_points, a ₹50 discount silently becomes 5,000 points; the docs warn of corruption and leaked personal data. reserved makes the compiler refuse.
• Adding and removing fields is wire-safe. Old binaries skip unknown numbers, because the wire type tells them how many bytes to jump.
• Unknown fields survive. An old binary that parses and re-serializes a newer message keeps the fields it doesn’t know, unless you convert it to JSON or copy it field by field.
• Presence needs optional. A plain proto3 scalar can’t tell “set to zero” from “not sent”. For a new field where zero means something, such as a ₹0 tip versus no tip data, declare it optional.
• Enums start with …_UNSPECIFIED = 0. Adding a value, say UPI_LITE to a payment-method enum, is wire-safe, but an old reader sees a number it has no name for: C++ and Go keep the raw integer, Java maps it to UNRECOGNIZED. An exhaustive switch without a default branch is the real bug.
• Type changes are lossy at best. int32, uint32, int64, uint64 and bool can be swapped on the wire, but values that don’t fit are truncated; sint32 pairs only with sint64; string and bytes swap if the bytes are valid UTF-8. Widen int32 to int64 and keep writing small values until every reader has the new schema.
• There is no required. Proto3 removed it because so many teams found it harmful: a required field can never be dropped without breaking every reader that still checks it.

One trap: Protobuf’s JSON form is stricter than its binary form. Old code parsing ProtoJSON fails on an added or renamed field, so use binary between services.

Avro: the writer’s schema meets the reader’s

Avro bytes carry no tags, so a reader needs the exact schema the writer used. It also brings its own reader’s schema, the shape its code expects, and the library resolves one against the other, field by field, by name:

• A field the writer has and the reader lacks is skipped.
• A field the reader has and the writer lacks gets the reader’s default; with no default, decoding fails.
• Field order may differ, and types may be promoted: int to long, float or double; long to float or double; float to double; string to bytes and back.
• A reader’s field can list aliases, old names to match against the writer.
• An enum symbol the reader doesn’t know becomes the reader’s enum default, or an error.

{"name": "tip_paise", "type": ["null", "long"], "default": null}

That one line is how you add an optional field: a union with null first and a default of null. The rules then write themselves. Add a field only with a default, or new readers can’t read old data. Remove a field only if it had a default, or old readers can’t read new data. Rename by adding the old name as an alias in readers, upgrading readers, then changing writers.

Where does the writer’s schema come from? Container files store it once in the header, ahead of millions of records. Avro’s single-object encoding prefixes each record with 2 marker bytes and an 8-byte schema fingerprint. On Kafka, a schema registry hands out a small ID instead.

The registry is a lookup service, not a proxy: records never pass through it, and clients cache each ID’s schema, so a short registry outage mostly blocks new schemas rather than records whose schemas are already cached.

JSON: tolerant readers and explicit contracts

JSON carries names, not numbers, so the name is the identity. Reordering keys is free, and a rename is a removal plus an addition. Evolution works by convention:

• Writers only add. New fields are optional; nothing is removed while a reader still uses it.
• Readers are tolerant. They ignore unknown fields, treat missing fields as defaults, and survive enum strings they have never seen.

Your library’s defaults decide whether the second rule holds. Java’s popular Jackson 2.x mapper enables FAIL_ON_UNKNOWN_PROPERTIES by default, so one new field in a payload makes every Java consumer that never switched it off throw. Jackson 3.0 flipped the default to off. Check yours before you assume additions are safe.

JSON Schema (or an OpenAPI spec) adds a written contract and validation. A closed schema with "additionalProperties": false turns every new field into a validation failure for old readers. And because JSON has no types beyond string, number, boolean, array and object, the contract must spell out formats: 64-bit IDs as strings, money in paise, timestamps in RFC 3339. Duplicate keys are another trap: RFC 8259 calls their behaviour unpredictable, and parsers disagree on which value wins. API-level versioning (URL versions, dated headers, deprecation) is covered in What is an API?.

Which changes are safe

Going deeper: what breaks in production

Renames: expand, migrate, contract

You can’t rename across independently deployed services in one step, so you do it in several, each compatible with the one before. To turn amount into amount_paise:

1. Expand: writers send both fields.
2. Migrate: readers switch to amount_paise, falling back to amount for old records.
3. Backfill stored data, or wait for retention to age out old records.
4. Contract: writers stop sending amount; once no old reader or record remains, delete the fallback and, in Protobuf, reserve the old number and name.

Database columns follow the same dance (see online schema and data migrations). If the meaning changes, rupees to paise for example, add a new field rather than renaming: an old reader that silently reads a 100× bigger number is worse than one that crashes.

The unknown-field round trip

No error is raised anywhere, which is what makes this bug dangerous. Defences: keep unknown fields when you decode (Protobuf binary does; many JSON mappers can collect them into a map), update only the fields a service owns instead of writing whole documents back, and upgrade every writer of a shared record before any of them starts writing the new field.

Events kept for years

In event sourcing and long-retention Kafka topics, every version ever written must stay readable, so register schemas under a transitive mode. When the shape drifts too far, write upcasters: small functions that lift a v1 event to v2 and v2 to v3 as it is read, so business logic only sees the latest shape. For a truly breaking change, start a new topic such as orders.v2, publish to both, move consumers over, then retire the old one. Change data capture feeds deserve extra care, because a column rename in the source database becomes a breaking change for every downstream consumer.

Make the registry a CI gate

Compatibility belongs in the pull request, not in production. Check every changed schema against the registered versions, or against the main branch for .proto files, and fail the build on a breaking change. Register schemas from CI rather than from producers at startup, so nothing unreviewed reaches a topic. The same goes for payloads you might forget are serialized: opaque pagination cursors, cache values and resumable job state all deserve a version byte.

Cheat sheet

How to say it in the interview

“At the public edge I’d use JSON, with 64-bit IDs as strings and money in paise, because every client can parse and debug it. Between services I’d use Protobuf over gRPC, and for Kafka events Avro or Protobuf with a schema registry, which cuts payloads three to five times. What matters more is evolution: v1 and v2 always coexist during deploys, and events outlive code. So changes are additive with defaults, Protobuf field numbers are never reused, the registry runs in full-transitive mode for long-lived topics, and its compatibility check runs in CI. Renames go through expand, migrate, contract.”

How do you rename a field without downtime? Expand and contract: write both names, move readers to the new one with a fallback, backfill or let old data expire, then stop writing the old name and reserve it.

Producer or consumer first? Backward compatibility means consumers first; forward means producers first. Since a rollback reverses the order, aim for full compatibility.

Why does Protobuf use field numbers? Numbers are smaller than names and let an old parser skip fields it doesn’t know, which is what makes adding fields safe. The cost: a number can never be reused.

JSON or Protobuf for a mobile app’s API? JSON by default, for simplicity and debuggability, with gzip on. Protobuf if payload size or battery truly matter. Either way, old app versions live for months, so treat every response as a public contract.

How do you evolve events already stored in Kafka for a year? Use a transitive registry mode so new readers handle every old version, upcast old events on read, and move a truly breaking change to a new topic with dual publishing.

Where to go next

• What is an API?: contracts, breaking changes and versioning at the HTTP level.
• REST vs GraphQL vs gRPC: the API styles that carry these bytes.
• API design for system design interviews: payload design in practice.
• Kafka and event-driven architecture: where schemas live longest.
• Event sourcing and CQRS: events stored forever, and upcasting.
• Deployment strategies: why versions overlap.
• Online schema and data migrations: expand and contract for databases.

References and further reading

1. Protocol Buffers: Encodingprotobuf.devThe wire format: base-128 varints with a continuation bit, tags as (field_number << 3) | wire_type, the wire types, ZigZag and length-delimited strings.
2. Protocol Buffers: Language Guide (proto3)protobuf.devField numbers (1 to 536,870,911; one byte for 1–15), reserved, defaults and presence, enum zero values, wire-safe and wire-compatible changes, unknown-field retention.
3. Proto Best Practices (dos and don’ts)protobuf.devNever reuse a tag number, reserve deleted ones, don’t change types or defaults, and why required fields were removed from proto3.
4. Apache Avro 1.12.0 Specificationavro.apache.orgBinary encoding with zig-zag varints and no names or tags, defaults, aliases, unions, single-object encoding and writer/reader schema resolution.
5. Confluent Schema Registry: Schema evolution and compatibilitydocs.confluent.ioDefinitions of BACKWARD, FORWARD, FULL, their transitive forms and NONE, why BACKWARD is the default, and which clients to upgrade first.
6. Confluent: Formats, serializers and deserializers (wire format)docs.confluent.ioThe record prefix: a magic byte of 0 and a 4-byte big-endian schema ID, plus Protobuf’s message-index bytes.
7. RFC 8259 — The JavaScript Object Notation (JSON) Data Interchange Formatrfc-editor.orgThe JSON grammar; integers in [-(2^53)+1, 2^53-1] are interoperable; duplicate names behave unpredictably.
8. MDN: Number.MAX_SAFE_INTEGERdeveloper.mozilla.orgWhy JavaScript numbers are exact only up to 2^53 − 1 = 9,007,199,254,740,991.
9. Python documentation: pickle — Python object serializationdocs.python.orgThe official warning that unpickling untrusted data can execute arbitrary code.
10. Jackson databind: DeserializationFeature (source)github.comFAIL_ON_UNKNOWN_PROPERTIES: enabled by default in Jackson 2.x, disabled by default as of 3.0.