# Measuring an MPI job

## With srun

Install heapline once, from the source tree where you built it in the
[MPI tutorial](tutorial-mpi.md):

```
cmake --install build --prefix $HOME/opt/heapline
```

The library goes to `$HOME/opt/heapline/lib`, or `lib64` on some
distributions, such as RHEL, Rocky or SUSE, and `heapline-report` to
`$HOME/opt/heapline/bin`.

Prepare the output directory first: heapline does not create it.

```
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8

out=$SCRATCH/heapline/$SLURM_JOB_ID
mkdir -p "$out"
# libheapline.so is in lib (or lib64): check with ls $HOME/opt/heapline
srun env LD_PRELOAD=$HOME/opt/heapline/lib/libheapline.so \
         HEAPLINE_OUTPUT=$out/heapline.%h.%r.csv ./app
$HOME/opt/heapline/bin/heapline-report "$out"
```

Pass `LD_PRELOAD` through `env` on the `srun` line rather than exporting it
in the script: exported, it would also preload heapline into `srun` and
every other command of the script.

`./app` must be the program itself. If it is a script that starts the real
program, read [Wrapper processes](#wrapper-processes) first: the script is
preloaded too, and its record can replace the program's.

The rank comes from `SLURM_PROCID`, set by `srun` for each task, unless the MPI
library sets one of its own variables (see the next section); a PMI variable
set by `srun` holds the same value.

## With mpirun

```
mkdir -p records
mpirun -n 16 env LD_PRELOAD=$PWD/build/libheapline.so HEAPLINE_OUTPUT=$PWD/records/heapline.%h.%r.csv ./app
```

With Open MPI on a machine with fewer than 16 cores, add `--oversubscribe`
after `mpirun` (MPICH has no such option). With an installed copy, use the
path of the installed library instead of `$PWD/build/libheapline.so`. With the
RPM or the Debian package installed on every node, `libheapline.so` without a
path is enough (see [Packages](getting-started.md#packages)).

Keep `env` directly in front of the program. A script or `/usr/bin/time`
between them would be preloaded too, and its record can replace the program's:
see [Wrapper processes](#wrapper-processes).

Open MPI sets `OMPI_COMM_WORLD_RANK`, MPICH and Intel MPI `PMI_RANK`,
MVAPICH `MV2_COMM_WORLD_RANK`, and PMIx-based launchers `PMIX_RANK`. heapline
reads `OMPI_COMM_WORLD_RANK`, `PMIX_RANK`, `PMI_RANK`, `MV2_COMM_WORLD_RANK`
and then `SLURM_PROCID`, in this order, and takes the first one that is set.
`SLURM_PROCID` comes last because the ranks started by `mpirun` inside a Slurm
allocation inherit the `SLURM_PROCID` of the process that started them: 0 on
the first node, and the index of the node's daemon on the other nodes. Either
way it is not their rank.

## Wrapper processes

`LD_PRELOAD` reaches every process started by the preloaded command. A shell
script run without `exec`, `/usr/bin/time` or any other process that stays the
parent of the program is preloaded too. It has the same rank and host as the
program and usually exits last, so with `HEAPLINE_OUTPUT=heapline.%h.%r.csv`
its record overwrites the program's, and nothing warns you.

There are two ways to avoid it:

- Start the program with `exec` in wrapper scripts, and put `LD_PRELOAD` on the
  program itself rather than on the wrapper. The last line of the script
  becomes `exec env LD_PRELOAD=... HEAPLINE_OUTPUT=... ./app`.
- Or add `%p` to the file name, as in `heapline.%h.%r.%p.csv`. Every process
  then keeps its own record. `heapline-report` shows both, and its warning
  about duplicate ranks names the two files, the wrapper's among them. Remove
  the wrapper's file before reading the figures: its small record lowers the
  minimum and shifts the median and the imbalance.

## Sizing --mem

`heapline-report` prints, per node, the sum of `VmHWM` of its ranks. Compare
the largest one with the `--mem` request, which applies per node, and keep a
margin for the page cache and the system.

`--mem-per-cpu` is also enforced per node: the limit is the request times the
CPUs allocated on the node. Comparing it with the same per-node sum is
therefore the like-for-like check. Comparing the largest `VmHWM` of a rank with
the request times the CPUs of a task is a stricter one.