Is the deployment problem just about avoiding downtime?

No. Availability is only part of it. The sharper question is whether every accepted request, message, transaction, and connection is handled correctly across the change. A service can remain reachable while still losing background jobs, breaking old sessions, reading incompatible rows, or producing writes that make rollback unsafe.

Why is draining old work necessary?

Because stopping new traffic is not the same as finishing existing work. A process may still own active requests, open connections, uncommitted transactions, or queue messages it has reserved. Draining gives those units a chance to complete, acknowledge, or hand off through a defined retry path before the old process disappears.

What is commonly misunderstood about rollback?

Rollback is often treated as undo, but it is another live deployment in the opposite direction. It only works if S0 can still run against the current configuration, schema, and data. If S1 has performed a forward-only migration or written a new format, reverting code may create a worse outage.

Problem, Stated

The stated deployment problem is the engineering problem of changing a live system from S0 to S1 without losing, corrupting, or misrouting work the system has already accepted. The system state includes code, configuration, schema, processes, data, traffic routing, and the compatibility rules that must hold while old and new versions overlap.

Deployment is hard because a running service is not just files being replaced. It is handling requests, jobs, transactions, streams, sessions, and queued messages at the moment the change begins. Some units of work finish quickly; others run long enough to outlive the rollout step that started them. If the old process is killed, the schema changes underneath it, or traffic moves too abruptly, accepted work can be dropped or completed incorrectly.

A safe deployment treats the change as a controlled state transition. Start capacity for the new version, prove it is ready, stop giving new work to the old version, then let the old version drain the work it already owns before termination. During the overlap, both versions must understand shared data, message formats, API responses, and database rows. If work can be retried, the retry path must be explicit, durable, and safe to repeat.

The trade-off is that deployment safety usually requires temporary complexity. You may need backwards-compatible schemas, feature flags, dual readers, idempotency keys, durable queues, longer drain periods, and staged traffic movement. Rollback is not magic either: it is another transition, S1 to S0, and can fail if the newer version has written data the older version cannot parse or dependably ignore.

Engineers meet this problem in rolling updates, readiness probes, load balancer draining, queue consumers, database migrations, and rollback plans. In Kubernetes, old and new ReplicaSets can run together while endpoints change gradually. In serverless and proxy-based systems, aliases or routing weights can shift traffic in stages. In relational databases, expand-and-contract migrations let old and new application versions coexist while data is backfilled and callers are moved.

Go deeper

DeploymentThe Problem, StatedState the deployment problem precisely: moving a live system between states without dropping work already in flight.Open the lesson

Common questions

Is the deployment problem just about avoiding downtime?: No. Availability is only part of it. The sharper question is whether every accepted request, message, transaction, and connection is handled correctly across the change. A service can remain reachable while still losing background jobs, breaking old sessions, reading incompatible rows, or producing writes that make rollback unsafe.
Why is draining old work necessary?: Because stopping new traffic is not the same as finishing existing work. A process may still own active requests, open connections, uncommitted transactions, or queue messages it has reserved. Draining gives those units a chance to complete, acknowledge, or hand off through a defined retry path before the old process disappears.
What is commonly misunderstood about rollback?: Rollback is often treated as undo, but it is another live deployment in the opposite direction. It only works if S0 can still run against the current configuration, schema, and data. If S1 has performed a forward-only migration or written a new format, reverting code may create a worse outage.

Every term