Autoscaling and Cold Starts
Autoscaling and cold starts are the capacity policy and startup delay involved in adding inference replicas under changing traffic. For LLM serving, the cold start is usually dominated by making model weights available in GPU memory, not by launching a container, so scale-to-zero only works when that first-request delay is acceptable or correctly priced.
The problem is that inference traffic is often bursty, while GPUs are expensive to leave idle. Autoscaling tries to add replicas only when needed, and scale-to-zero removes them completely during quiet periods. That saves money until the next request arrives for a model with no hot worker. For large models, the delay is not mostly orchestration overhead. The service must read a very large checkpoint, move shards to the right devices, initialise the runtime, and only then accept token work.
A cold start is the path from scheduling a worker to serving its first token request. In concrete terms, the system pulls or reuses an image, starts the process, finds the checkpoint or engine files, reads weights from object storage, network disk, or local NVMe, copies them through host memory to GPUs, and prepares kernels or cached plans. The slowest part is often the byte path for the weights. Peer GPU loading, local caches, and pre-staged shards help because they shorten that path.
The trade-off is between paying for warm capacity and paying users with latency when capacity is absent. A warm pool keeps weights already resident in HBM, which gives normal time to first token but burns GPU cost while idle. Scale-to-zero avoids that idle spend, but the first request after idleness pays the full materialisation cost. Commonly misunderstood: container tuning can improve the edges, but it cannot remove the need to put the model’s bytes on the GPUs.
Engineers meet this in Kubernetes autoscalers, model-serving platforms, and LLM runtimes such as vLLM, SGLang, TensorRT-LLM, Dynamo, and llm-d. The important operational distinction is image warm, weights local, and weights in HBM. Only the last state is genuinely ready for low-latency inference. Autoscaling signals also matter: average QPS or GPU utilisation can hide burst concurrency, long prefills, KV pressure, and cold-miss probability.
Common questions
- Is a cold start just the time to start a container?
- No. For LLM inference, container startup is usually a smaller component. The expensive step is materialising the model weights: reading checkpoint or engine files, placing shards, copying bytes to GPUs, and initialising runtime state. If the weights are not already in HBM, the replica is not truly warm, even if the pod reports ready soon after launch.
- When is scale-to-zero a good idea?
- It depends on the arrival pattern, the user-visible latency budget, and the cost of idle GPUs. Scale-to-zero can be sensible for rare, isolated requests where a long first-token delay is acceptable. It is a poor fit for interactive traffic with tight SLOs, because the first request after zero replicas violates the latency target by design.
- Do faster inference kernels remove cold starts?
- No. FlashAttention, PagedAttention, speculative decoding, batching, and KV-cache improvements help once the model is hot and serving requests. They do not eliminate the checkpoint I/O and host-to-GPU transfer needed before the model can run. Some engine formats or auxiliary draft models can even add bytes that must be loaded during startup.
- How should a warm pool be sized?
- Size it from cold-miss probability, burst concurrency, model placement, and prefill behaviour, not just average utilisation. A single warm replica may still leave a burst queued behind long prefills. Many model variants make the problem worse because each variant may need its own warm shard. The honest question is which models need weights already in HBM.