Is serving two models the same as speculative decoding?

No. Speculative decoding is one reason to serve two models: a smaller draft proposes tokens and the large model verifies them. Serving two models is the broader deployment problem of co-locating any auxiliary model with a main one, including verifiers and routers, and controlling their memory and scheduling interference.

Why can a small model hurt the large model so much?

During decode, a model may stream a large fraction of its weights and touch KV state for each generated token. If the small model runs many tiny batches in the same queue, it burns bandwidth and scheduler turns that could have kept the large model’s continuous batch full. Parameter count alone hides that interference.

What should be reserved separately for each model?

Reserve weight memory, KV-cache blocks, workspace, and enough slack for allocator fragmentation. Then reserve service capacity too: maximum running batch, request admission, priority, or token-rate limits. Without both memory and scheduling isolation, the auxiliary model can either exhaust cache blocks or delay the primary model’s decode loop.

When does co-locating a draft, verifier, or router stop helping?

It depends on whether the auxiliary work reduces more large-model work than it adds. A draft model helps only when accepted tokens save enough main-model decode steps to cover draft and verification cost. Routers and verifiers turn negative when they sit on the critical path or inspect too much synchronously.

Serving Two Models

Serving two models is co-locating a primary model and an auxiliary model on the same accelerator, while explicitly dividing memory, KV cache, and scheduler priority between them. The auxiliary may be a verifier, draft model, or router, but it still competes for bandwidth and execution slots.

The need appears when a system wants the benefit of a small model without paying for a separate device. Speculative decoding may use a draft model, safety or quality checks may use a verifier, and request dispatch may use a router. The mistake is treating the smaller model as nearly free because it has fewer parameters. In decode-heavy serving, the scarce resource is often HBM bandwidth and scheduler residency, not just nominal model size.

Concretely, each resident model needs its own weights, KV-cache allocation, temporary workspace, and allocator slack. The scheduler then decides which model gets decode or prefill work on the shared execution fabric. A small model request can still stream much of its weight set for a tiny batch. If those jobs enter the same FIFO path as the large model, they interrupt continuous batching and consume bandwidth while producing no large-model tokens.

The trade-off is that co-location can save hardware only if the auxiliary work is bounded and amortised. You may have to quantise weights, reduce KV budgets, cap batch size, or move one model to another slice or worker. Commonly misunderstood: fitting both weight files is not enough. KV growth, fragmentation, launch overheads, and priority inversion can make a configuration that fits in memory still fail latency or throughput objectives.

Engineers meet this in vLLM, SGLang, TensorRT-LLM, Dynamo, llm-d, or similar serving stacks when adding speculative decoding, verification, or routing beside a main LLM. The practical pattern is separate model instances or engines, reserved KV blocks, admission control, and distinct priority queues or token-rate limits. If the auxiliary model has no priority class, traces show falling large-model batch residency, scheduler gaps, and throughput loss proportional to stolen bandwidth.

Go deeper

Inference EngineeringServing Two ModelsCo-locate a small model beside a large one, whether a verifier, a draft model or a router, and divide the device between them: weights, KV budget and scheduling priority. Show what happens to the big model's throughput when the small one is not given its own priority class.Open the lesson

Common questions

Is serving two models the same as speculative decoding?: No. Speculative decoding is one reason to serve two models: a smaller draft proposes tokens and the large model verifies them. Serving two models is the broader deployment problem of co-locating any auxiliary model with a main one, including verifiers and routers, and controlling their memory and scheduling interference.
Why can a small model hurt the large model so much?: During decode, a model may stream a large fraction of its weights and touch KV state for each generated token. If the small model runs many tiny batches in the same queue, it burns bandwidth and scheduler turns that could have kept the large model’s continuous batch full. Parameter count alone hides that interference.
What should be reserved separately for each model?: Reserve weight memory, KV-cache blocks, workspace, and enough slack for allocator fragmentation. Then reserve service capacity too: maximum running batch, request admission, priority, or token-rate limits. Without both memory and scheduling isolation, the auxiliary model can either exhaust cache blocks or delay the primary model’s decode loop.
When does co-locating a draft, verifier, or router stop helping?: It depends on whether the auxiliary work reduces more large-model work than it adds. A draft model helps only when accepted tokens save enough main-model decode steps to cover draft and verification cost. Routers and verifiers turn negative when they sit on the critical path or inspect too much synchronously.

Every term