02.09 · Walkthrough
Reading the Device
Interpret nvidia-smi output. Understand what GPU-Util actually measures --- and what it does not. Diagnose why a GPU reporting 100% utilisation can still be memory bound.
`GPU-Util` in `nvidia-smi` means the device had a kernel active during the sampler window, not that tensor cores, SMs, or memory bandwidth are efficiently used. LLM decode can show full utilisation while stalled on HBM traffic from weights and KV cache, so real diagnosis needs profiler counters and workload context.
What this lesson answers
- what does nvidia-smi GPU-Util actually mean
- why is full GPU utilisation still memory bound
- how to read nvidia-smi for LLM inference
Notes
`nvidia-smi` is a low-rate NVML sampler, not a profiler: the headline `GPU-Util` is reported by NVML as the fraction of the last sampling period during which at least one kernel was executing on the device. A precise mental model is , where “active” means the GPU has work resident, not that SM pipelines are saturated. The companion `Memory-Usage` is allocated framebuffer, not bandwidth, and `Memory-Util` is similarly a coarse activity flag for memory reads/writes, not .
Common questions
- Does GPU-Util show how busy the tensor cores are?
- No. It reports whether kernels were resident on the GPU during the sampling interval. A kernel can keep the device continuously active while doing little tensor-core work. To know whether tensor cores or SM pipelines are saturated, use profiler counters such as SM activity, tensor pipe use, achieved FLOP rate, and kernel timelines.
- Why can LLM decoding show full GPU utilisation but low throughput?
- Decode often streams model weights and KV cache through HBM for each generated token. That can keep kernels running without providing enough arithmetic work to saturate compute units. The GPU is not idle, but the limiting resource is memory movement, so `GPU-Util` stays high while tokens per second remain constrained.
- Is Memory-Usage in nvidia-smi the same as memory bandwidth?
- No. `Memory-Usage` is allocated framebuffer capacity, not traffic rate. It tells you how much GPU memory is reserved or occupied, but not how fast data is being read or written. For bandwidth questions, inspect profiler metrics such as DRAM throughput, achieved HBM bandwidth, and memory-related kernel counters.