Open MPI

There are three different Open MPI choices in the list of MPI implementations to choose from in Arm® Forge when debugging or profiling for Open MPI.

Open MPI – the job is launched with a custom launch agent that, in turn, launches the Arm daemons.
Open MPI (Compatibility) – mpirun launches the Arm daemons directly. This startup method does not take advantage of the Arm scalable tree.
Open MPI for Cray XT/XE/XK/XC – for Open MPI running on Cray XT/XE/XK/XC systems. This method is fully capable of using the Arm scalable tree infrastructure.

To launch with aprun (instead of mpirun), enter one of these commands:
```
ddt --mpi="OpenMPI (Cray XT/XE/XK)" --mpiexec aprun [arguments]
```
```
map --mpi="OpenMPI (Cray XT/XE/XK)" --mpiexec aprun [arguments]
```

Known issues

Message queue debugging does not work with the UCX or Yalla PML, due to UCX and Yalla not storing the required information.
The version of Open MPI packaged with Ubuntu has the Open MPI debug libraries stripped. This prevents the Message Queues feature of Arm® DDT from working.
On Infiniband systems, Open MPI and CUDA can conflict in a manner that results in failure to start processes, or a failure for processes to be debugged. To enable CUDA interoperability with Infiniband, set the CUDA environment variable to 1.

These versions of Open MPI do not work with Arm® Forge because of bugs in the Open MPI debug interface:

Open MPI 3.0.0 when compiled with the Arm® Compiler for Linux on Arm®v8 (AArch64) systems.
Open MPI 3.0.x when compiled with some versions of the GNU compiler on Arm®v8 (AArch64) systems.
Open MPI <= 3.x.4 and <= 4.0.1 when compiled with some versions of IBM XLC/XLF, NVIDIA HPC, or PGI compilers.
Open MPI <= 3.1.6, <= 4.0.5 and <= 4.1.0 when compiled with -O2 or -O3 optimization flags, and PGI 19.x, 20.1 and NVIDIA HPC compilers.
Open MPI 3.1.0 and 3.1.1.
Open MPI 3.x with any version of PMIx earlier than 2.0.
Open MPI 4.0.1 with PMIx 3.1.2.
Open MPI up to version 3.1.4 and 3.0.4 on Ubuntu 20.04 x86_64 systems.

To resolve any of these issues, select Open MPI (Compatibility) for the MPI Implementation.

Open MPI 3.x on IBM Power with the GNU compiler

To use Open MPI versions 3.0.0 to 3.0.4 (inclusive) and Open MPI versions 3.1.0 to 3.1.3 (inclusive) with the GNU compiler on IBM Power systems, you must configure the Open MPI build with CFLAGS=-fasynchronous-unwind-tables. This fixes a startup bug where Arm® Forge is unable to step out of MPI_Init into your main function.

The startup bug occurs because of missing debug information and optimization in the Open MPI library. If you already configure with -g, you do not need to add this extra flag.

An example configure command is:

./configure --prefix=/software/openmpi-3.1.2 CFLAGS=-fasynchronous-unwind-tables

An alternative workaround if you do not have the option to recompile your MPI, is to select Open MPI (Compatibility) for the MPI Implementation.

This issue is fixed in later versions.