heapline

Heap usage of a Linux program, rank by rank, without recompiling it.

heapline is a small shared library loaded with LD_PRELOAD. It counts allocations and releases, tracks the bytes in use and their peak, and writes one record per process. heapline-report merges the records of a job and tells how much memory each rank used and when it peaked: what you need to size the --mem request of a Slurm job and to spot imbalanced ranks.

It is not a debugger and not a full heap profiler. It keeps lock-free counters on the allocation path and allocates nothing there. Its cost is paid per allocation. In a process that holds its data, as a simulation does, four threads that allocate continuously add about 20% to the run time, and so does a single thread. A small program whose threads do nothing but allocate adds the most, about 170%. See Performance history. When you need every allocation with its call stack, use heaptrack or Valgrind’s massif instead (see Limitations and alternatives).

• Getting started
• Tutorial: measure the memory of a program
• Tutorial: how much memory does my MPI job need?
• Measuring an MPI job
• heapline-report
• Output format
• What “peak memory” means
• Case study: miniAMR, LULESH and CloverLeaf
• Limitations and alternatives
• How heapline is tested
• Performance history
• Changelog