Published October 15, 2026
Reading GPU energy instead of guessing it, on an AMD MI210
NVML only reports a power value every 100 ms, so my Kokkos energy connector could not see short kernels. AMD GPUs expose a cumulative energy counter instead. On an MI210 it moves about every millisecond, but one read costs a third of a millisecond, which decides how a profiler has to use it. Measured with it, ArborX fdbscan-densebox takes 38% less time and 42% less energy than fdbscan.
The previous post ended on a limit. NVIDIA’s NVML gives a power value every 100 ms that only covers the last 25 ms of the interval, so a Kokkos Tools connector built on it can charge energy to long regions, never to a kernel that lasts a few milliseconds. In October 2026 I got time on illyad, a node of the University of Oregon’s Frank cluster: two AMD Instinct MI210 and an H100, on the node where the poster’s runs were made. AMD GPUs do not make you integrate power at all. Through AMD SMI they expose a counter of the energy the GPU has used since the driver loaded, and the question was what that counter is actually good for.
What the counter gives you
amdsmi_get_energy_count returns three numbers: an accumulator, its resolution, and a timestamp. Multiply the first two and you have joules. On the MI210 the resolution is 15.3 µJ exactly, the counter is readable without root, and over every probe and the 140,033 samples of the study below it never went backwards. It does restart from zero when the driver reloads: it read 33.9 MJ one week and 4.4 MJ after the node rebooted.
The interesting number is how often it moves. Read in a tight loop, it changes value about every 1.2 ms, and the firmware timestamp it returns moves by the same amount. Under load, those steps look like this:
With NVML, energy is an estimate: you integrate a curve you only see a quarter of the time. Here, the energy between any two reads is the difference of two counter values. Nothing between them is missed, whatever the GPU did.
A read costs a third of a millisecond
The obvious way to use such a counter is the one that fails with NVML: read it when a region begins, read it when it ends, subtract. With a counter that would even be correct. It is too slow.
| calls | median | p99 |
|---|---|---|
| back to back | 0.26 to 0.29 ms | 0.51 to 0.60 ms |
| 10 ms apart | 0.47 to 0.56 ms | 0.88 to 0.95 ms |
Half a millisecond per read is longer than many kernels. Reading the counter at each region boundary would cost more than some of the regions it measures. So the counter has to be read the same way as NVML: by a side thread, on a fixed period. At 10 ms that thread spends about 5% of one core inside AMD SMI.
Why a call 10 ms after the last one costs twice as much as one right after it shows up in the driver. Reading the same counter straight from gpu_metrics in sysfs takes 42 µs, and the value there changes every 1.000 ms, almost to the microsecond. My reading, which I have not checked in the driver source: amdgpu keeps a copy of the firmware’s metrics table for one millisecond, and the first read after it expires waits for a fresh one. A sampling thread that reads every 10 ms always pays that wait. The 1.2 ms step seen through AMD SMI is that millisecond plus the time until the next ~0.3 ms call lands. Two processes reading at once are not serialized; their p99 grows by about half.
What does the 10 ms period cost in accuracy? Less than it seems. The energy between two samples is exact; only its split at a region boundary is interpolated. At 200 W, that is at most about 2 J per boundary, under 0.1% of a region lasting several seconds, and a lot for a kernel of a few milliseconds. The hardware moves every millisecond; the profiler sees every ten.
The DBSCAN study, again, on the MI210
I repeated the poster’s case study with this counter: the same 497,838,606-point HACC input, the same parameters, 64 runs of each ArborX implementation, alternating run by run, through an experimental connector that reads the counter every 10 ms and writes the trace format of energy-dashboard-for-kokkos.
| implementation | time | energy | mean power |
|---|---|---|---|
fdbscan | 6.948 s | 1465 J | 211 W |
fdbscan-densebox | 4.309 s | 846 J | 197 W |
The dense implementation takes 38.0% less time and 42.2% less energy, and the bootstrap intervals over the runs are a tenth of a point wide. The runs are also remarkably regular: the quartiles of fdbscan span 11 ms and 3 J, and no run is more than 30% slower than its median. Nothing like the slow first batch of the H100 series showed up.
The shape of the result is the same as on the H100: the faster implementation is also the more frugal one, and the energy gap is wider than the time gap because the mean power drops too, by 6.8% here. A time profile understates the gain again.
What this does not show
It is tempting to put this table next to the poster’s and read off a comparison between AMD and NVIDIA. It would not hold. The H100 numbers come from power sampled through NVML, an ArborX from July 2025 and the Kokkos of the time; these come from the energy counter, the ArborX of October 2026 and Kokkos 5.0.2. The gap between the two implementations is wider here, 38% against 19% in time, and I cannot tell how much of that is the GPU and how much is fifteen months of ArborX development.
Two more limits. I did not compare the counter with an external power meter, so the absolute joules are worth what AMD’s telemetry is worth; the comparisons between implementations do not depend on it. And the machine changed under me: during the study the MI210 idled at about 41 W, and after a reboot a week later the same GPU idled at 58 W with its clocks at their maximum. I do not know why. The study numbers belong to the first state.
Where it goes
The connector reads the counter through a single class that is the only code touching AMD SMI, built to slot into the provider layout of the energy tooling proposed for Kokkos Tools. It has its own sampling thread for now; the upstream version should use the daemon under review in Kokkos Tools (#350), so it waits for that merge. The next measurement is the one that would make the comparison fair: the same ArborX build on the H100 of the same node, read through NVML’s own cumulative energy counter rather than its power value.
The connector, the probes, the raw data and the scripts that recompute every number in this post are on GitHub: mi210-energy-counter. This research used resources on the Frank cluster at the University of Oregon.