When is it safe to roll back during an expand-contract migration?

Rollback is safe while the old schema, old write path, and old-readable data are still intact. That usually includes expand, dual-write, and backfill. It stops being a normal rollback once production contains only-new-readable data, or once the old storage path has been removed.

Why not deploy the code and migration together?

Because production rarely switches from old to new in one clean instant. A previous pod, worker, queue consumer, or scheduled job may still run after the migration has applied. If the migration removes or changes something that old code needs, those remaining processes can fail on real traffic.

What is dual-write, and why is it risky?

Dual-write means each request writes both the old and new representations during the compatibility window. It lets either application version read correct data, but it also creates consistency risk: every write path must update both places, and retries, partial failures, and missed background writers need careful handling.

Is a backfill part of the deploy or a separate operation?

It is usually treated as an operational job rather than a single blocking deploy step. The backfill should run in bounded batches, be restartable, and be observable. Large one-shot updates can hold locks, overload replicas, enlarge write-ahead logs, and interfere with normal production writes.

Migrations Against Live Traffic

Migrations against live traffic are schema or data changes performed while production requests continue and multiple application versions may be running. They use an expand-contract sequence so old and new code can read and write safely until the compatibility window closes and any irreversible change is made.

The problem is that deploys are not instantaneous. During a rolling release, old pods, workers, queue consumers, Lambdas, cron jobs, and in-flight requests may still execute while new code is already writing data. If the database changes too early, old code may read a missing column, parse a value in the wrong shape, or depend on data that has just been deleted.

The usual mechanism is expand, dual-write, backfill, read-switch, then contract. First add new nullable columns, tables, indexes, or formats without removing the old ones. Then deploy code that writes both old and new representations. Historical rows are copied in bounded batches. Reads move to the new path only after it is populated. Finally, after every old runtime has drained, the obsolete path is removed.

The trade-off is temporary complexity. For a while, the system has two representations of the same fact, extra write logic, backfill jobs, monitoring, and edge cases around partial progress. Backfills can create lock pressure, replication lag, write amplification, or large transaction logs if run carelessly. The honest rollback answer is: it depends on whether old code can still understand every committed production state.

The exact rollback boundary is the first irreversible contract step, or the first committed request that creates data only the new code can understand. Before that, a normal deploy rollback should work because the old schema and old data path still exist. After a dropped column, destructive rewrite, or removal of old-format messages, rollback becomes a forward repair or restore problem.

Engineers meet this in database migrations, rolling Kubernetes deployments, background workers, message consumers, API versioning, and webhook payload changes. Common tooling helps but does not remove the compatibility obligation: online schema migration tools, concurrent index creation, and batched jobs reduce blocking, while the application still has to tolerate every live version until the oldest one is gone.

Go deeper

DeploymentMigrations Against Live TrafficRun an expand-contract migration and name the exact window in which rollback stops being possible.Open the lesson

Common questions

When is it safe to roll back during an expand-contract migration?: Rollback is safe while the old schema, old write path, and old-readable data are still intact. That usually includes expand, dual-write, and backfill. It stops being a normal rollback once production contains only-new-readable data, or once the old storage path has been removed.
Why not deploy the code and migration together?: Because production rarely switches from old to new in one clean instant. A previous pod, worker, queue consumer, or scheduled job may still run after the migration has applied. If the migration removes or changes something that old code needs, those remaining processes can fail on real traffic.
What is dual-write, and why is it risky?: Dual-write means each request writes both the old and new representations during the compatibility window. It lets either application version read correct data, but it also creates consistency risk: every write path must update both places, and retries, partial failures, and missed background writers need careful handling.
Is a backfill part of the deploy or a separate operation?: It is usually treated as an operational job rather than a single blocking deploy step. The backfill should run in bounded batches, be restartable, and be observable. Large one-shot updates can hold locks, overload replicas, enlarge write-ahead logs, and interfere with normal production writes.

Every term