Published November 16, 2026
What NVML can tell you about a single kernel
My Kokkos energy tooling stopped at the scale of the whole GPU. On my own RTX 3080 Ti I measured how often NVML’s power and energy readings change, how exact its energy counter is, and whether CUPTI kernel timestamps can split board power between kernels. They can, as long as the kernels do not share the GPU.
The Kokkos energy post ended on two open questions. NVML is too coarse to charge energy to a single kernel, and I had no clean answer for kernels that run at the same time on two streams. On an AMD MI210, a cumulative energy counter answered part of the first question. NVIDIA GPUs have one too, nvmlDeviceGetTotalEnergyConsumption, and I had never measured it. So I did, on the GPU I have at home: an RTX 3080 Ti, driver 610.88, under Windows 11. Every probe is a short Python script, and together they take about five minutes to run.
How often the numbers change
NVML gives three power values: nvmlDeviceGetPowerUsage, which most profilers call, and two fields of the field-value API, the instantaneous power and the average power. I read all three and the energy counter in a loop for 20 s, about once every 1.3 ms under load, and recorded each time a value changed.
| value | under load | at idle |
|---|---|---|
| power usage, instant, average | 500 ms | 500 ms or 1 s |
| energy counter | 100 ms | 100 ms |
On this card the power values move twice a second. That is five times slower than the 100 ms that Yang, Adamek and Armour report for data center GPUs. A consumer board under Windows is not an H100, and this probe says nothing about one. The energy counter, on the other hand, moves every 100 ms, under load and at idle.
The three power values are not three measurements either. nvmlDeviceGetPowerUsage returns exactly the average field, read for read. When a load starts, the instantaneous field reads 349 W at its first update. The value most profilers read goes through 207 W, then 316 W, and only reaches 351 W a second later. It lags the same way when the load stops.
nvmlDeviceGetPowerUsage returns is an average and takes several updates to catch up, on the way up and on the way down.The counter, read too fast
A counter should make the read rate irrelevant: the energy between two reads is the difference of two values, and nothing in between is missed. To check, I ran a constant matmul load, read the counter at a fixed rate for a fixed time, and divided the energy it reported by that time, 8 s per rate, in increasing order, in a single run. The power value read 351 W throughout.
| read rate | power from the counter | against 351 W |
|---|---|---|
| 1 Hz | 338 W | −4% |
| 2 Hz | 335 W | −5% |
| 5 Hz | 328 W | −6% |
| 10 Hz | 301 W | −14% |
| 14 Hz | 301 W | −14% |
| 19 Hz | 270 W | −23% |
| 46 Hz | 158 W | −55% |
| 102 Hz | 115 W | −67% |
| 235 Hz | 29 W | −92% |
| 766 Hz | 12 W | −97% |
Read once or twice a second, the counter agrees with the power value within 5%. Read faster, it loses energy, and read in a tight loop it loses almost all of it. I do not know why, and I have not found it documented. The rule I take from it is a practical one: on this card, read the counter rarely, over windows of a second or more, and never at kernel boundaries.
Splitting board power between kernels
NVML only ever sees the whole board. CUPTI, NVIDIA’s profiling interface, sees the kernels: when each one starts and ends. The question was whether the two can be joined. The model is the simplest one possible: board power is an idle term plus one term per kind of kernel, weighted by the fraction of the time CUPTI shows that kind running. Fit it on blocks where the kinds run one at a time, then use it to split power between kinds.
The first probe runs 2 s blocks of a single kind (an fp32 matmul, an fp16 matmul, and a memory-bound pass over a 512 MB vector) at a 25% or 50% duty cycle, in shuffled order. Over ten runs, the fit explains 89 to 97% of the variance of the power samples (R² 0.89 to 0.97), once the power trace is shifted 0.25 to 0.5 s behind the kernels: even the instantaneous field, which these probes sample every 10 ms, trails the activity. Against the energy counter read at the borders of each block, the model is within 32 W for every kind and duty cycle (the mean of its three blocks) in every run. The coefficients move with the size of the kernels and the duty cycle, so they say nothing like "fp16 costs less than fp32"; they only hold for the workload they were fitted on.
The second probe is the one the Kokkos post could not answer: two streams, an fp32 matmul on one and an fp16 matmul on the other. The model is fitted on blocks where each stream runs alone, then asked to predict blocks where both run.
| kernels | overlap | measured | predicted |
|---|---|---|---|
| one 8192×8192 matmul per stream at a time | none | 265 W | 265 W |
| bursts of 40 small 2048×2048 matmuls | about a tenth of the busy time | 216 W | 228 W |
When the big kernels fill the GPU in turn, the model is right on average, within 1 W over the five runs, although a single run can be off by up to 13 W either way. When small kernels from the two streams overlap, it over-predicts by about 13 W: two kernels sharing the GPU draw less than the model adds up from each of them alone. I would not lean too hard on that number, though. In the same runs the model already over-predicts the single-stream blocks by 3 to 5 W, and from one run to the next the measured power of the same blocks moves by about 10 W, so 13 W is a hint in the right direction, not a measurement of the effect.
What this changes for the Kokkos tooling
The NVML connector I wrote at ORNL (#301) samples nvmlDeviceGetPowerUsage on a side thread and integrates it over each region. On this card, that value only changes twice a second and lags the GPU by about a second, so a region shorter than a few seconds gets the power of whatever ran before it. Reading the energy counter at region borders would fix the lag but not the resolution, and read too often the counter under-counts. What holds up is what these probes did: read the counter on a slow timer, and split each window between kernels with CUPTI timestamps and a fitted model, accepting that the split is only as good as the model when kernels overlap.
None of this transfers automatically to an H100 or an A100 under Linux, where NVML is known to behave differently. The point of keeping the probes small is that they can be rerun there before trusting any number. The next step is to do exactly that on the node where the poster runs were made.
The scripts, the raw outputs and the script that redraws the figures are on GitHub: nvml-energy-probes. The next post uses these rules to measure a real workload.