06.09 · Walkthrough
Autoscaling and Cold Starts
Choose between scale-to-zero and a warm pool by pricing the cold start honestly: loading tens of gigabytes of weights is the cold start, and no amount of container tuning removes it.
LLM cold starts are dominated by moving model weights into GPU memory, not by container startup. Autoscaling choices should price the byte path from storage to host to GPU, then compare that delay against user-visible latency targets and the cost of keeping warm replicas resident.
What this lesson answers
- why are LLM cold starts so slow
- scale to zero versus warm pool LLM serving
- how to price cold starts for inference autoscaling
Notes
A cold start for an LLM serving replica is the interval from scheduling a new worker to accepting its first token request, and for fleet sizing the irreducible term is weight materialization, not container boot: , with and usually .
Common questions
- Why does container tuning not solve LLM cold starts?
- Container startup is usually not the limiting step. The expensive part is materialising the checkpoint: reading large model weights from storage, transferring them through host memory, copying them to GPU memory, and initialising any serving engine state. A faster image pull helps only if the weights are already close to, or inside, the GPUs.
- When is scale-to-zero reasonable for an inference service?
- Scale-to-zero is reasonable when requests are rare, the product can tolerate the first request waiting for weights to load, and the saved idle GPU cost exceeds the expected penalty from cold misses. It is a poor fit for interactive traffic with tight time-to-first-token targets, because the first request after zero replicas fails that target by design.
- What actually counts as warm capacity for an LLM?
- The meaningful warm state is weights already resident in GPU memory and ready to serve. Having the container image cached is only image warm. Having checkpoint files on local storage is better than fetching from remote storage, but still not equivalent to normal serving latency. Warm pools must be sized around cold-miss risk and burst concurrency.
Short definition: what is Autoscaling and Cold Starts?