Relational sharding
0-day streakReview

Your progress

Level 1

0 XP100 XP to level 2
0day streak
0best streak
0topics started
0cards reviewed
    Review cards

    Progress is stored only in this browser. No account is required.

    Sharding a relational database: the 60-second story

    Sharding a relational database

    VACUUM is stalling, and wraparound could stop every write.

    Notion’s PostgreSQL monolith, before it sharded. Splitting it is harder than splitting a NoSQL store, because years of code assume one machine.

    Split the database, and six SQL promises break

    Figma called it an order of magnitude harder than its earlier scaling. Its first sharded table took about nine months.

    Put workspace_id first in every key

    Pick the entity most requests are scoped to. A workspace’s joins, foreign keys and transactions then never leave its shard.

    Store workspace_id in comment, though page has it?

    Bad: No: join through page

    The router can’t place the row, nor check its foreign key locally.

    Good: Yes, denormalise on purpose

    Key it (workspace_id, comment_id); reference (workspace_id, page_id).

    Who knows the shard map?

    Your own code (Instagram, Notion), a MySQL proxy (Vitess’s VTGate), or Citus’s coordinator inside PostgreSQL. The shards look the same.

    Every query path changes. How do you test that before moving data?

    The trick

    Shard logically first: Figma routed traffic through a view per hash range on one host, by percentage. Rollback was a config change.

    5 min

    of downtime, hiding in the catch-up

    Once writes stopped, Notion’s script still had to drain its backlog. A week more tuning, to under 30 seconds, could have made it invisible.

    Cutover day is the start

    1. Expand-and-contract schema changes, tracked per shard
    2. Feed a warehouse by CDC for reporting
    3. Backups, restores and alerts are per shard
    4. Every pooler connects to every shard: budget it

    Remember

    Root every table in one entity’s ID, then pick who routes: your code, Vitess or Citus. Migrate by sync, backfill, dark reads and per-shard cutover, with a reverse stream.