Adding one column takes 1 ms. Why is checkout down?
A report holds a read lock for 90 s. The ALTER waits for exclusive access, and every new query queues behind the waiting ALTER.
The ALTER isn’t slow; its waiting is the outage. So what’s the free fix?
The fix
A short lock timeout, retried with jitter. The worst case becomes a 2-second stall, not an 85-second outage.
Rename phone to mobile?
Bad: One rename statement
Instant, but servers still on the old code break the moment it commits.
Good: Expand, migrate, contract
Add mobile beside phone, move readers and writers across, drop phone last.
Mid-deploy, old and new code share one database
For minutes, versions N and N+1 run side by side, and a rollback brings N back. So every schema state must work with both.
40,000
small transactions instead of one UPDATE
200 million rows in 5,000-row batches, throttled on replica lag. One UPDATE would lock every row it touched until commit, and a failure at 90% rolls back everything.
Moving orders to a new database? Same shape.
Capture changes, backfill, verify with shadow reads, then flip reads and writes behind flags. The old store stays in sync: rollback is a flag flip.
Where migrations still bite
- A long transaction stalls DDL and concurrent index builds
- Rewrites and backfills leave replicas lagging
- At cut-over, writes land on the old side
- Nobody contracts; old columns and flags live forever
The interview line
Never one big ALTER on a hot table. Lock timeout and retry; expand, migrate, contract so each schema state serves two app versions; drop last.