02.11 · Walkthrough
Launch Overhead
Measure kernel launch overhead. Apply the batch doubling test: double the work and if runtime barely rises, the kernel is overhead bound. Recognise when CUDA graphs eliminate the problem.
Kernel launch overhead is a fixed CPU-side scheduling cost that can dominate small GPU decode steps. Measure it with empty launches, then double the batch: if latency barely changes, launches are the bottleneck. CUDA Graphs help when the kernel sequence is stable enough to replay instead of repeatedly scheduled.
What this lesson answers
- how do I measure CUDA kernel launch overhead
- when is GPU inference launch overhead bound
- when do CUDA Graphs improve decode latency
Notes
Kernel launch overhead is the fixed host-to-device scheduling cost paid before a GPU kernel can begin useful work, distinct from the kernel’s device execution time. For one decode step with kernels, the lower-bound latency model is , and the launch-bound regime is the one where .
Common questions
- What is kernel launch overhead in CUDA inference?
- It is the fixed cost of asking the GPU to run a kernel before useful device work starts. In small-batch inference, many short kernels can spend more time being scheduled from the host than doing arithmetic or memory movement, so latency stops reflecting the GPU’s raw bandwidth or compute capability.
- How does the batch doubling test reveal launch overhead?
- Run the same decode workload at one batch size, then run it again with double the batch while keeping the model and sequence conditions fixed. If latency barely increases, the work added by the extra batch was small compared with fixed launch and setup costs. If latency scales strongly, the GPU is doing real device work.
- When do CUDA Graphs fix launch overhead?
- CUDA Graphs help when the hot path has repeatable shapes, stable tensor addresses and a consistent kernel sequence. The runtime records the launch pattern once and replays it cheaply. They do not help much when execution is already bandwidth or compute bound, or when dynamic batching forces padding, copying or frequent recapture.
Short definition: what is Launch Overhead?