Measuring where the energy goes on the GPU
Energy-measurement tooling for Kokkos, the C++ performance-portability library behind many US Department of Energy codes. I wrote a power-sampling profiler for Kokkos Tools, with NVML and Variorum back ends, whose sampling daemon is merged upstream, and energy-dashboard-for-kokkos, the open-source analysis tool I rewrote in September 2026.
The problem
Kokkos runs one C++ source on NVIDIA, AMD, and Intel GPUs, and the same kernel draws different power on each. Kokkos Tools could only estimate a kernel’s energy from two power readings, at its start and end, which NVML’s 100 ms refresh makes unreliable; nothing integrated a continuous power trace per region, on DOE machines where power is now a first-class constraint.
How it is built
The tools attach at run time through Kokkos Tools, so an application is measured without a rebuild or a patch. The connectors I submitted upstream (#301, #302) read power through NVML or Variorum; an AMD path through ROCm SMI, not public yet, is the version I ran on Frontier. energy-dashboard-for-kokkos, rewritten in Rust in September 2026, has a documented trace format, tests on a real GPU trace, and CI-built releases archived on Zenodo with a DOI. The traces and script behind the DBSCAN figures are public, and CI recomputes them.
Where it stands
The rest of the profiler is three open pull requests I wrote: the core and timing export (#299), the NVML connector (#301, with changes requested), and the Variorum connector (#302, a draft of about 2,700 lines that includes unit tests). Its sampling daemon (#300) was merged into kokkos-tools in March 2026, after my ORNL mentor, Jakob Bludau, reworked it through review once my stay had ended. Nine pull requests to kokkos-tools and LAMMPS in all, some later split into smaller ones; three merged: the daemon and two build fixes. One poster, with Daniel Arndt, Jakob Bludau and Damien Lebrun-Grandié, presented at SMC 2025 and at an ORNL internal session, and cited in the S4PST 2024–2025 project report.
Stack C++ · Kokkos · CUDA · NVML · ROCm SMI · Variorum · Slurm · Rust

