Health Checks and Draining
Health checks and draining are the release mechanisms that tell infrastructure whether an instance may receive traffic, whether it should be restarted, and how to stop it safely. Readiness removes an instance from routing, liveness proves the process is still recoverable, and draining lets in-flight work finish before termination.
The problem appears during deploys, autoscaling, crashes, and node maintenance. A process can be alive but not ready for new work, or shutting down while clients still reuse existing connections. If the platform sends traffic until the process disappears, users see failed requests and workers lose leased jobs. If probes treat a shared dependency outage as process death, every replica can be restarted even though restarting them cannot repair the dependency.
In Kubernetes, readiness controls whether a Pod is listed as a Service endpoint. When readiness fails, the container keeps running but new traffic should stop arriving after propagation through the load balancer. Liveness is different: repeated liveness failures tell the kubelet to kill and restart the container. A safe shutdown usually handles SIGTERM by failing readiness first, waiting for routing to drain, finishing active requests or jobs, then exiting cleanly.
The trade-off is that health checks must be honest without being too ambitious. Readiness can include local capacity, such as whether the server socket is bound and whether queues are under control. Liveness should usually be shallow, such as whether the event loop can still respond. A common mistake is putting database or cache checks in liveness; during a shared outage that can create a restart cascade across otherwise healthy application processes.
Engineers meet this in Kubernetes readinessProbe and livenessProbe settings, service mesh draining, load balancer deregistration, worker shutdown code, and server connection handling. HTTP services may need to stop accepting new keep-alive reuse, while HTTP/2 systems may send GOAWAY to prevent new streams. Background workers need the same idea expressed differently: stop leasing new jobs, complete the current lease, or requeue it before the grace period expires.
Common questions
- Should readiness and liveness use the same endpoint?
- Usually no. They answer different questions. Readiness asks whether this instance should receive new work right now. Liveness asks whether the process is so stuck that restarting it is useful. Sharing one deep endpoint means a temporary Redis or Postgres failure can both remove traffic and trigger restarts, which is often the wrong response.
- What should happen when an instance receives SIGTERM?
- The application should first make readiness fail, before expensive cleanup begins. Then it should allow time for routers and load balancers to stop selecting it, finish active handlers or jobs, and exit successfully. The shutdown grace period must be longer than the expected drain time, or the platform may kill the process mid-request.
- What belongs in a readiness check?
- It depends on what receiving work requires. Good readiness checks test local ability to accept work: the server is listening, required local initialisation has completed, and internal queues are not overloaded. They may include critical dependencies if the instance truly cannot serve without them, but avoid turning every remote hiccup into total fleet withdrawal.
- What belongs in a liveness check?
- Liveness should be a small proof that the process can still make progress and that a restart would help. Examples include a responsive event loop, an unwedged worker loop, or a simple in-process endpoint. It should not usually query databases, caches, or third-party APIs, because their failure is not fixed by restarting every application container.