08.08 · Walkthrough
Serving Two Models
Co-locate a small model beside a large one, whether a verifier, a draft model or a router, and divide the device between them: weights, KV budget and scheduling priority. Show what happens to the big model's throughput when the small one is not given its own priority class.
Co-locating a small auxiliary model with a large LLM is a memory and scheduling problem, not just a loading problem. The large model needs protected weight, KV-cache and execution budget, while the smaller verifier, draft model or router needs explicit limits so it cannot steal bandwidth and collapse decode throughput.
What this lesson answers
- how to serve two models on one GPU
- why small verifier model slows large LLM
- how to budget KV cache for co-located models
Notes
Co-locating two LLMs on one accelerator means reserving disjoint memory budgets for two weight sets and two KV-cache pools, then sharing the same attention/MLP execution fabric under a scheduler with explicit priority or rate limits. For model , the hard memory constraint is summed over co-resident models, where is sharded weight memory, is KV memory, is transient activation/workspace, and is allocator fragmentation.
References
Common questions
- Why can a small model hurt a much larger model's throughput?
- During decode, both models compete for the same memory bandwidth and scheduler slots. A small model may have fewer weights, but repeated tiny batches can still interrupt the large model's continuous batch. The result is lower batch residency, more gaps between decode iterations and fewer large-model tokens produced per unit of device time.
- What must be reserved when two LLMs share one accelerator?
- Each model needs its own weight allocation, KV-cache pool and workspace allowance, with extra room for allocator fragmentation. The KV budget is often the part that breaks the plan, because it scales with live sequences, context length, layers, KV heads and element size. Fitting weights alone is not enough.
- How should a verifier, draft model or router be scheduled?
- It should run in a separate priority or service class with bounded KV blocks, batch size and token rate. The large model normally owns the latency-critical path, while auxiliary work runs in slack or under explicit caps. A single FIFO queue is unsafe because it treats bandwidth-heavy auxiliary steps as neutral.
Short definition: what is Serving Two Models?