Launch Overhead
Launch overhead is the fixed CPU-side cost of asking the GPU to run a kernel before that kernel does useful work. It includes driver and runtime scheduling work, and can dominate latency when an inference step is made of many tiny kernels whose actual device work is brief.
The problem appears when the GPU is capable of finishing each small operation quickly, but the host must still submit every operation separately. In small-batch decode, an inference step may run normalisation, RoPE, attention fragments, cache updates, residual adds and sampling as separate kernels. If each kernel does little work, the lower bound is no longer memory bandwidth or arithmetic throughput. It is the repeated cost of launching the next piece of work.
Mechanically, a launch is a command placed by the CPU runtime and driver onto a CUDA stream so the device can later execute a kernel with particular arguments. You can measure the fixed part with an empty or near-empty kernel, after warmup, timing many launches without accidentally synchronising inside the loop. The useful test is batch doubling: run the same step at batch B and then at 2B. If latency barely changes, fixed launch and setup costs are dominating.
The trade-off is that removing launch overhead usually requires more structure. Kernel fusion reduces the number of launches but may produce larger, less reusable kernels and more complicated code. CUDA Graphs record a stable launch graph and replay it cheaply, but they like fixed shapes, stable tensor addresses and predictable control flow. Padding, bucketing, graph capture management or dynamic sampling paths can cost more than the saved launches. The honest answer is workload-dependent.
Engineers meet launch overhead when profiling GPU inference and seeing many short kernels separated by CPU submission gaps. It is common in eager PyTorch paths, custom decode loops, and serving stacks at small batch sizes. CUDA Graph replay in TensorRT-LLM, vLLM or SGLang is often used to avoid per-token host orchestration. If doubling batch roughly doubles runtime, look instead at memory traffic, tensor utilisation or KV-cache locality.
Common questions
- Is launch overhead the same as GPU execution time?
- No. GPU execution time is the time the device spends running the kernel body. Launch overhead is the host-side submission and scheduling cost paid before useful device work can begin. Profilers may show both close together, but optimising one does not automatically optimise the other.
- How do I tell if my kernel is launch-bound?
- Use a representative benchmark and double the batch while keeping the model and sequence conditions comparable. If runtime barely rises, the extra work was cheap and the fixed launch path is probably dominant. If runtime scales with the added batch, the device is doing substantial memory or compute work.
- Do CUDA Graphs always fix launch overhead?
- They fix a specific source of overhead when the same launch pattern can be captured and replayed. They do not help much when the step is already device-bound. They can also hurt if requests constantly change shape, batch composition, LoRA choice or sampling path, forcing padding, bucketing or recapture.