Migrating a service without downtime (and the one outage)
The migration moved a service from a single instance with a local database to two instances sharing a managed one. Nobody wanted a maintenance window, so we did the dull version: expand, migrate, contract.
The plan that worked
- Add the new column and code that writes to both, while reading the old one.
- Backfill in batches, throttled, with progress logged.
- Flip reads to the new column behind a flag.
- Delete the old column a week later, once nothing referenced it.
Six days of boring work, no downtime, and the deletion commit was tiny because everything depended on it already being unused.
The outage
Step three, the flag flip, caused twelve seconds of errors. The cause was not the migration. It was a cache warm-up: after the flip, every request missed cache, the database saw a burst it had never seen, and connection limits did what connection limits do.
The fix was a rate-limited rollout of the flag, ten percent at a time, which cost us twenty extra minutes and would have cost us nothing else.
The migration is not the risk. The new access pattern is the risk.