What should I look for first in a forward-pass trace?

Start with the GPU lanes. If they are mostly filled with long kernels, inspect which kernels dominate and compare them with the expected compute or memory floor. If they contain blank gaps, look back to the CPU lanes for delayed launches, blocking synchronisation, host-side sampling, shape logic, logging, or copies that prevent the GPU from receiving work.

Is low GPU utilisation always a kernel problem?

No. This is commonly misunderstood. Low utilisation may mean kernels are inefficient, but it may also mean the GPU is waiting for the CPU, a data transfer, an allocator event, a collective operation, or a remote worker. The trace matters because it shows whether the device is busy running poor kernels or simply not being fed.

Why profile prefill and decode separately?

They stress different parts of the system. Prefill usually has larger matrix operations and attention over the prompt, so the trace often contains longer GPU kernels. Decode often runs smaller per-token work, where launch overhead, KV-cache access, sampling, and scheduler gaps become more visible. Mixing them can hide the actual bottleneck.

Can the profiler itself change the result?

Yes. Profiling is not free, and the overhead can be large enough to mislead when the measured region is short. Extra stack, shape, and memory recording can perturb CPU dispatch and allocation behaviour. Use warmups, fixed shapes, a narrow profiled interval, and avoid inserting synchronisation inside the path you are trying to measure.

Profiling a Forward Pass

Profiling a forward pass is recording the CPU work, CUDA calls, GPU kernels, and memory movement that occur while a model computes outputs from inputs. The result is a timeline trace that shows where time is spent, where the GPU waits, which kernels run, and whether data copies or synchronisation are limiting inference.

A forward pass can be slow for reasons that are invisible from wall-clock timing alone. The model may be doing real GPU work, or the GPU may be idle while Python prepares launches, the runtime waits on a synchronisation point, or tensors move between host and device. Without a trace, these very different failures all look like “inference is slow”. Profiling separates useful kernel time from launch gaps, blocking copies, allocator effects, and scheduler overhead.

In practice, you warm up the model, profile only the interval that computes the chosen prefill or decode step, and export the trace to a timeline viewer. CPU lanes show Python, framework dispatch, and CUDA API calls. GPU lanes show kernels such as attention, matmul, normalisation, dequantisation, and cache updates. The horizontal length of a GPU bar is kernel duration; blank space on a stream means no work is running; copy lanes expose host-device or device-device transfers.

The main cost is measurement distortion. Profilers add overhead, especially when collecting stack traces, tensor shapes, or memory history, and that matters most for short decode steps where launch overhead is already significant. CUDA is asynchronous, so synchronising in the wrong place can either hide unfinished GPU work or create an artificial stall. Captured execution paths and fused kernels can also make attribution harder, because many logical operations may appear as one launch.

Engineers meet forward-pass profiling when optimising PyTorch inference, investigating low GPU utilisation, validating fused attention or graph capture, and debugging serving systems such as paged KV-cache runtimes. It is most useful when compared with a hardware floor: if kernel time is close to the memory or compute limit, the implementation may be healthy; if the trace shows long blank regions, the bottleneck is likely scheduling, synchronisation, transfer, or distributed coordination.

Go deeper

Inference EngineeringProfiling a Forward PassUse the PyTorch profiler to capture a trace of a forward pass. Read the trace: identify CPU-GPU synchronisation gaps, kernel durations, and memory transfers.Open the lesson

Common questions

What should I look for first in a forward-pass trace?: Start with the GPU lanes. If they are mostly filled with long kernels, inspect which kernels dominate and compare them with the expected compute or memory floor. If they contain blank gaps, look back to the CPU lanes for delayed launches, blocking synchronisation, host-side sampling, shape logic, logging, or copies that prevent the GPU from receiving work.
Is low GPU utilisation always a kernel problem?: No. This is commonly misunderstood. Low utilisation may mean kernels are inefficient, but it may also mean the GPU is waiting for the CPU, a data transfer, an allocator event, a collective operation, or a remote worker. The trace matters because it shows whether the device is busy running poor kernels or simply not being fed.
Why profile prefill and decode separately?: They stress different parts of the system. Prefill usually has larger matrix operations and attention over the prompt, so the trace often contains longer GPU kernels. Decode often runs smaller per-token work, where launch overhead, KV-cache access, sampling, and scheduler gaps become more visible. Mixing them can hide the actual bottleneck.
Can the profiler itself change the result?: Yes. Profiling is not free, and the overhead can be large enough to mislead when the measured region is short. Extra stack, shape, and memory recording can perturb CPU dispatch and allocation behaviour. Use warmups, fixed shapes, a narrow profiled interval, and avoid inserting synchronisation inside the path you are trying to measure.

Every term