08.04 · Concept
Health Checks and Draining
Set readiness and liveness so a shutting-down instance stops taking work, and avoid the deep-check restart cascade.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Deployment lesson by lesson; 29 of 56 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Readiness removes an instance from routing; liveness restarts it. During shutdown, flip readiness before cleanup, drain existing work, then exit cleanly. Keep liveness shallow so shared dependency outages do not trigger pointless restarts across the fleet, while readiness reflects whether the instance can safely accept new work.
What this lesson answers
- how should readiness and liveness differ
- how do I drain traffic during shutdown
- why deep liveness checks cause restart cascades
Notes
Health Checks and Draining — Health checks and draining exist so load balancers stop sending new work to an instance before it exits; without them, deployments produce dropped requests, killed in-flight jobs, or restart cascades from probes that mistake dependency failures for process failure.
Key Concepts: - Kubernetes uses `readinessProbe` to decide Service endpoints: when readiness returns non-2xx/3xx, the Pod is removed from endpoint routing but the container is not restarted.
Common questions
- What should a readiness check actually test?
- Readiness should answer whether this instance can take new work right now. That usually means the server is listening, local queues are within acceptable bounds, and the process is not shutting down. It should fail before shutdown cleanup begins, so routers stop sending fresh requests while existing work finishes.
- Why should liveness checks avoid database and cache calls?
- Liveness is used to decide whether to kill and restart the process. If it depends on Postgres, Redis, or another shared service, an external outage can make every instance look dead. Restarting application processes will not fix that dependency, and can turn a recoverable outage into a fleet-wide restart cascade.
- How does draining differ for web requests and workers?
- For web services, draining means stop receiving new requests, discourage reuse of existing connections, and let active handlers complete before exit. For workers, it means stop leasing new jobs, then finish or safely return leased jobs so another worker can take them later without losing work.
Short definition: what is Health Checks and Draining?