Tutorial: measure the memory of a program

This tutorial measures the memory that a program allocates, and when it needs it. It takes about ten minutes on any Linux machine, needs no MPI and no change to the program, which here is a Python one-liner. For a parallel job, follow it first, then the MPI tutorial.

1. Get heapline

Build it from source:

git clone https://github.com/ethan-puyaubreau/heapline.git
cd heapline
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build

This builds build/libheapline.so and build/heapline-report. A prebuilt archive also exists, see Getting started; its lib/libheapline.so and bin/heapline-report replace the two files above in the commands below.

2. Run a program under heapline

Preload the library; the program needs no change and no rebuild:

LD_PRELOAD=$PWD/build/libheapline.so python3 -c "x = [bytes(10**6) for i in range(300)]"

The last line on stderr is the summary:

heapline: 4154 allocs, 4125 frees, peak 288.2 MiB, live at exit 400.2 KiB

Python allocated 300 blocks of one million bytes, 286.1 MiB. heapline counts the usable size of each block, which the allocator rounds up, so they weigh 287.1 MiB, and Python’s own allocations add about 1 MiB. peak is the highest amount of heap in use at any time, and live at exit what was still allocated when the program ended. The 29 allocations that were never released are what Python keeps until it exits.

3. Keep a record and read it

Set HEAPLINE_OUTPUT to also write a record of the process, then read it with heapline-report:

LD_PRELOAD=$PWD/build/libheapline.so HEAPLINE_OUTPUT=run.csv python3 -c "x = [bytes(10**6) for i in range(300)]"
cat run.csv
build/heapline-report run.csv

The record is a CSV file of three lines, described in Output format:

# heapline 1
rank,host,pid,job,elapsed_s,allocs,frees,peak_heap_bytes,peak_heap_time_s,live_heap_at_exit_bytes,vmhwm_bytes,vmrss_at_exit_bytes
-1,node01,17438,,0.015,4154,4125,302236152,0.007,409800,11534336,10485760

The host name is shown here as node01. Your pid will differ, and so will the times and VmHWM, slightly. The report of this one process reads:

heapline-report: 1 rank on 1 node, longest process 0.0 s

Peak heap
  max        288.2 MiB, rank -1 at 0.0 s
  median     288.2 MiB
  min        288.2 MiB
  imbalance  1.00 (max / mean)

Per node, sum of VmHWM (an upper bound to compare with --mem)
  node01  1 rank   11.0 MiB

Ranks by peak heap (1 of 1)
  rank  host     peak heap  at (s)       VmHWM  heap / VmHWM
    -1  node01   288.2 MiB     0.0    11.0 MiB  26.20

Slurm: sacct MaxRSS is a per-task maximum sampled during the job. Compare it
with the largest VmHWM above, 11.0 MiB; a short peak can fall between samples.

The rank is -1 because the program did not run under MPI. The report is made for jobs of many processes, so with one it repeats the same figures. The lines about --mem and Slurm matter for cluster jobs; see the MPI tutorial.

4. Understand the numbers

Two figures look contradictory: the heap peak is 288.2 MiB, yet VmHWM, the highest amount of memory that the process had in RAM, is only 11.0 MiB, and the ratio of the two, 26.20, is far above 1. Python asks the allocator for large zeroed blocks, and the kernel provides them without using RAM until a page is written. Python writes almost none of them, only the header page of each block, about 1 MiB in total. The program allocated 288 MiB and used almost none of it. Make it write every byte:

LD_PRELOAD=$PWD/build/libheapline.so HEAPLINE_OUTPUT=touched.csv python3 -c "x = [b'x' * 10**6 for i in range(300)]"
build/heapline-report touched.csv

The heap peak is the same, 288.2 MiB, and the ranks table becomes:

Ranks by peak heap (1 of 1)
  rank  host     peak heap  at (s)       VmHWM  heap / VmHWM
    -1  node01   288.2 MiB     0.2   297.1 MiB  0.97

Now VmHWM exceeds the heap peak by 8.9 MiB. That gap is memory that heapline does not count as heap: the code of Python, its libraries, stacks. It is constant, so it matters on a small heap and hardly at all on a large one.

The two figures answer two questions. The heap peak says how much the program asked for. VmHWM says how much RAM it used, which is what a job scheduler limits, so it is the one to compare with a memory request. What “peak memory” means lists every measure.

5. When does the peak happen?

The record holds the time of the heap peak, peak_heap_time_s. Compare a program that allocates first and then idles for two seconds with one that idles first:

LD_PRELOAD=$PWD/build/libheapline.so HEAPLINE_OUTPUT=early.csv python3 -c "import time; x = bytes(300 * 10**6); del x; time.sleep(2)"
LD_PRELOAD=$PWD/build/libheapline.so HEAPLINE_OUTPUT=late.csv python3 -c "import time; time.sleep(2); x = bytes(300 * 10**6)"
build/heapline-report early.csv | grep '^  max'
build/heapline-report late.csv | grep '^  max'

  max        287.2 MiB, rank -1 at 0.0 s
  max        287.2 MiB, rank -1 at 2.0 s

Both peak at 287.2 MiB, one at the start and one at the end. For a job that shares a node with others, or that must not exceed a limit at one moment, the time of the peak tells where to look in the program.

6. A pitfall: wrapper processes

LD_PRELOAD reaches every process that the preloaded command starts. If you run the program through a shell, the shell is measured too, and it usually exits last and replaces the record of the program:

LD_PRELOAD=$PWD/build/libheapline.so HEAPLINE_OUTPUT=wrapped.csv bash -c "python3 -c \"x = [bytes(10**6) for i in range(300)]\"; echo done"

heapline: 4154 allocs, 4125 frees, peak 288.2 MiB, live at exit 400.2 KiB
done
heapline: 897 allocs, 278 frees, peak 84.5 KiB, live at exit 83.9 KiB

wrapped.csv holds the second line, the shell’s, not the 288 MiB of Python. Preload the program itself, not a wrapper around it; Wrapper processes gives the ways to do so.

7. Next steps