Skip to content
Klay

Have an undo you have actually run

Make every deploy reversible, and rehearse the rollback before you need it.

EngineersSystem designersvercelawskubernetesdockergithub

The question during an incident is never 'what caused this' — it is 'how do we stop it', and the fastest answer is almost always to put back the thing that worked. A rollback path that has never been exercised is a hypothesis, and incidents are a bad time to test one. This is also what makes shipping often SAFE rather than reckless: small, reversible changes fail small.

What goes wrong: An incident extended by an hour because the rollback needed a step nobody had performed before.

You are violating it when

  • Nobody on the team has rolled back this service.
  • Rolling back requires a migration to be undone.
  • Your deploy documentation has no rollback section.

The usual objection: That rollback is a button your platform provides. The button reverts your code; it does not revert the schema change, the cache format or the message your consumers already ate.

A real undo is an operational path, not an optimistic design note. For each production change, the team needs to know what exact command, pipeline action, flag flip, data restore, or compatibility mode returns the system to the last known-good behavior, and that path needs to have been exercised before the emergency. The rehearsal is what turns rollback from a hope into a usable control.

This works because incidents reward the fastest safe reduction in blast radius. Small deployments, staged rollouts, health checks, and rollback triggers make it possible to stop harm before the whole fleet or customer base is affected. Rehearsal also exposes the parts a deployment platform cannot infer: database migrations, cached representations, background jobs, queues, external consumers, configuration drift, and data already written in a new shape.

The misconception is that rollback is a button your platform provides. That button may restore a prior artifact, deployment template, or routing target, but it cannot automatically un-send messages, un-migrate data, or prove that old code can still read new state. Treating platform rollback as complete rollback leaves the hardest recovery steps to be discovered during the outage.

When an agent writes the code, the undo path has to be part of the requested work, not an afterthought. Agent-produced changes often span application code, schema, config, generated files, and tests; the review needs to verify that the reversal covers the same surface area and has actually been run in an environment close enough to production to catch integration failures.

Install it

npx klay practices add reversible-deploys
  • .klay/practices/reversible-deploys.mdcreate
    # Reversible deploys

The question during an incident is never "what caused this". It is "how do we
stop it", and the fastest answer is almost always to put back the thing that
worked.

A rollback path nobody has exercised is a hypothesis. An incident is a bad time
to test one.

## What "reversible" actually requires

Your platform's rollback button reverts **your code**. It does not revert:

- the **schema change** that shipped with it,
  • docs/runbook.mdcreate
    # Runbook

The document someone reads at 3am who did not write this service. Keep it
short, keep it current, and put real commands in it.

## Rollback

**The command:**

```bash
# REPLACE ME with the actual command for this service.
# e.g. vercel rollback <deployment-url>
#      kubectl rollout undo deployment/<name>
#      gcloud run services update-traffic <svc> --to-revisions=<prev>=100

The previews are the first lines of each file; the command writes them in full. Existing files are never overwritten.

How you know it stuck

npx klay practices audit reports this check for this practice:

  • rollback-documented

Where this comes from

  1. REL08-BP01 Use runbooks for standard activities such as deployment - Reliability PillarAWS Well-Architected · Official docsFirst-party reliability guidance directly requires tested, non-disruptive reversals and names untested reversal as an anti-pattern.
  2. Google SRE: Role of Release Engineer and Best PracticesGoogle SRE · Official docsConnects frequent release practice to smaller, cheaper, safer reversions.
  3. DeploymentsKubernetes Docs · Official docsShows the boundary of infrastructure rollback: workload template history is not full application state.
  4. Performing an Instant Rollback on a DeploymentVercel Docs · Official docsProvides a concrete rollback feature while documenting state caveats that remain outside the button.
  5. Code Orange: Fail Small — our resilience plan following recent incidentsCloudflare Blog · Engineering blogAdds an incident-driven rollout model with metrics, staged progression, and automatic rollback triggers.

Questions

We deploy from CI on every merge. Is that not risky?
It is the opposite, provided each deploy is small and reversible. Risk concentrates in big infrequent releases, not in frequent tiny ones.