Is LoRA multi-tenancy just cheaper fine-tuning?

No. Cheap adaptation is only part of the story. The serving benefit is that many fine-tuned behaviours can pass through the same base-model replicas. The runtime swaps or selects small adapter deltas per request, so the fleet avoids duplicating the base weights and can keep larger, healthier batches across tenants.

Does the server merge the LoRA adapter into the model weights?

Sometimes, but it often does not. A common path computes the normal base layer output and then adds the adapter contribution from the low-rank matrices. Popular adapters may be pre-merged or cached if memory and shapes make that worthwhile. The honest answer depends on adapter popularity, rank, batching strategy, and memory pressure.

When does one deployment per tenant make more sense?

A dedicated deployment can win when a tenant is large enough to keep replicas busy, needs strong isolation, uses high-rank or broad adapters, or has latency requirements hurt by adapter loading and grouping. Multi-tenancy works best when the base model is shared, adapters are relatively small, and traffic has enough locality for caching and batching.

What is commonly misunderstood about batching with LoRA?

People often assume all tenant requests can be batched exactly like ordinary requests once the base model is shared. In practice, each token row may reference different adapter matrices. The runtime must account for that difference, usually by grouping, segmented operations, or specialised kernels, otherwise small adapter computations and cache misses can dominate the benefit.

Multi-Tenancy and LoRA

Multi-tenancy with LoRA is a serving pattern where many tenant-specific fine-tunes share one loaded base model, while each request selects a small low-rank adapter that modifies selected layers. The base weights stay resident; the runtime routes, caches, batches, and applies adapters per request instead of running a separate model deployment for every tenant.

The problem is that full fine-tunes duplicate the most expensive object in the system: the base model weights. If each tenant gets a private deployment, memory use, GPU reservation, warm-up time, and batch fragmentation all grow with the tenant count. Many tenants may have modest customisation needs, but still force the fleet to behave as if each one owns a complete model. LoRA multi-tenancy exists to share the heavy part while preserving tenant-specific behaviour.

Mechanically, LoRA keeps the original weight matrix frozen and represents a tenant’s change as two small low-rank matrices whose product is added to the base layer’s output. At inference time, a request carries an adapter identifier. The runtime looks up that adapter, keeps hot adapters in GPU memory when possible, may spill colder ones to CPU, and computes the base projection plus the adapter’s skinny matrix multiplications, often without ever building a merged full weight matrix.

The trade-off is that adapters are not free. They add extra kernel work, memory traffic, cache policy, versioning, and scheduling complexity. Batching also becomes harder because different rows in the same decode step may need different adapters. Systems group by adapter, use segmented batched operations, or cache merged forms for popular adapters. If adapter variety is high, ranks are large, or cold adapters are constantly streamed, the shared design can lose to dedicated deployments.

Engineers meet this in inference servers and fleet schedulers rather than in training code alone. Frameworks expose adapter-aware routing, adapter caches, inflight or continuous batching, and placement policies across GPUs and nodes. Operationally, an API request may name a tenant or adapter version, while the serving layer decides where the base replica lives, whether the adapter is already hot, whether to prefetch it, and which batch it can safely join.

Go deeper

Inference EngineeringMulti-Tenancy and LoRAServe many fine-tunes from one base model by swapping low-rank adapters per request, and state what that buys over one deployment per tenant.Open the lesson

Common questions

Is LoRA multi-tenancy just cheaper fine-tuning?: No. Cheap adaptation is only part of the story. The serving benefit is that many fine-tuned behaviours can pass through the same base-model replicas. The runtime swaps or selects small adapter deltas per request, so the fleet avoids duplicating the base weights and can keep larger, healthier batches across tenants.
Does the server merge the LoRA adapter into the model weights?: Sometimes, but it often does not. A common path computes the normal base layer output and then adds the adapter contribution from the low-rank matrices. Popular adapters may be pre-merged or cached if memory and shapes make that worthwhile. The honest answer depends on adapter popularity, rank, batching strategy, and memory pressure.
When does one deployment per tenant make more sense?: A dedicated deployment can win when a tenant is large enough to keep replicas busy, needs strong isolation, uses high-rank or broad adapters, or has latency requirements hurt by adapter loading and grouping. Multi-tenancy works best when the base model is shared, adapters are relatively small, and traffic has enough locality for caching and batching.
What is commonly misunderstood about batching with LoRA?: People often assume all tenant requests can be batched exactly like ordinary requests once the base model is shared. In practice, each token row may reference different adapter matrices. The runtime must account for that difference, usually by grouping, segmented operations, or specialised kernels, otherwise small adapter computations and cache misses can dominate the benefit.

Every term