02.10 · Walkthrough
Profiling a Forward Pass
Use the PyTorch profiler to capture a trace of a forward pass. Read the trace: identify CPU-GPU synchronisation gaps, kernel durations, and memory transfers.
PyTorch profiling of a forward pass is a trace-reading exercise: capture only the warmed-up compute interval, then separate useful GPU work from host delays, synchronisation barriers, kernel launch overhead and memory copies. The goal is to tell whether latency comes from the hardware floor, PyTorch scheduling, data movement or accidental blocking.
What this lesson answers
- how to profile a PyTorch forward pass
- how to read CUDA gaps in profiler
- why is GPU idle during inference
Notes
Profiling a forward pass means recording a time-ordered trace of host work, CUDA runtime calls, GPU kernels, and memory copies for exactly the interval that computes , with timings interpreted against the hardware floor , where is FLOPs, peak FLOP/s, bytes moved from the limiting memory level, bandwidth, and launch/synchronization latency.
Common questions
- What should be included when profiling a forward pass?
- Capture a warmed-up prefill or decode step with fixed shapes, including CPU activity, CUDA activity, memory information and useful named ranges. Exclude startup effects such as compilation, allocator growth and lazy library initialisation. Synchronise after the measured region so the trace contains completed GPU work without inserting a barrier into the path being diagnosed.
- How do I spot CPU-GPU synchronisation in a trace?
- Look for an idle GPU stream while the CPU is doing Python, ATen dispatch, sampling, shape logic or blocking copies. Calls such as stream synchronisation, device-to-host transfers, `.item()` and `.cpu()` often create visible gaps. The key sign is not CPU busyness, but missing GPU work between launches.
- When is the PyTorch profiler misleading for inference?
- It can distort very short decode paths, especially with stack capture, shape recording and memory tracing enabled. Captured execution can also collapse many logical operations into one launch, hiding attribution. In distributed inference, a blank local GPU lane may reflect network, collective or remote cache waiting rather than Python overhead.
Short definition: what is Profiling a Forward Pass?