01.09 · Walkthrough
First Batch
Understand batching as a throughput lever: how multiple sequences share a forward pass, why batch size is a latency/throughput tradeoff, and how a scheduler picks requests to batch together.
Batching groups active generation requests into shared model passes so the GPU produces next-token logits for multiple sequences at once. It raises throughput by amortising kernel overhead and cache work, but can add waiting time. The scheduler decides which requests fit together under token, memory and latency limits.
What this lesson answers
- how does batching improve LLM inference throughput
- why does batch size affect latency in generation
- how do inference schedulers choose batches
Notes
First batch is the set of active generation requests that a scheduler groups into one model forward pass so the GPU computes logits for many sequences at once: for batch size , sequence lengths , hidden size , the input tensor is shaped roughly for prefill or for decode, and the model returns logits for next-token sampling.
References
Common questions
- What is a batch in LLM generation?
- A batch is a group of active requests sent through the model together. During prefill, the batch contains prompt tokens from multiple sequences. During decode, it usually contains one next-token input per active sequence, plus each sequence’s KV cache. The model returns separate logits for each request.
- Why can batching make latency worse while improving throughput?
- Batching improves throughput because one forward pass can serve several requests, making better use of GPU work. Latency can worsen because a request may wait in a queue until compatible requests arrive or until the scheduler admits it. The server emits more tokens overall, but an individual request may start later.
- What does the inference scheduler optimise when forming batches?
- The scheduler tries to pack useful work into each step without exceeding token budgets, KV cache memory, or latency limits. It may prefer FIFO ordering for fairness, best-fit packing for higher utilisation, or continuous batching so finished sequences leave and new ones enter between decode iterations.