Use custom MPI scripts (MAP)

On some systems, a custom mpirun replacement is used to start jobs, such as mpiexec. MAP typically uses whatever the default for your MPI implementation is, so for Open MPI it would look for mpirun and not mpiexec, for SLURM it would use srun, and so on. Here we explain how to configure MAP to use a custom mpirun command for job startup.

MAP supports two ways that you can use to start jobs using a custom script.

In the first way, you pass all the arguments on the command-line, like this:

mpiexec -n 4 /home/<user>/program/chains.exe /tmp/mydata

There are several key variables in this command that MAP can complete for you:

  • The number of processes (4 in the above example).

  • The name of your program (/home/<user>/program/chains.exe).

  • One or more arguments passed to your program (/tmp/mydata).

Everything else, like the name of the command and the format of its arguments, remains constant.

To use a command like this in MAP, you adapt the queue submission system described in Start a job in a queue (MAP). For this mpiexec example, the settings are :

Using custom MPI scripts

As you can see, most of the settings are left blank.

There are some differences between the Submit command in Linaro MAP and what you would type at the command-line:

  • The number of processes is replaced with NUM_PROCS_TAG.

  • The name of the program is replaced by the full path to forge-backend, used by both DDT and MAP.

  • The program arguments are replaced by PROGRAM_ARGUMENTS_TAG.

Note

It is not necessary to specify the program name here. MAP takes care of that during its own startup process. The important thing is to make sure your MPI implementation starts forge-backend instead of your program, but with the same options.

In the second way, you start a job using a custom mpirun replacement with a settings file:

mpiexec -config /home/<user>/myapp.nodespec

Where myfile.nodespec might contain something similar to the following:

comp00 comp01 comp02 comp03 : /home/<user>/program/chains.exe /tmp/mydata

If you specify a template file, Linaro MAP can automatically generate simple configuration files like this every time you run your program. For the above example, the template file myfile.template would contain the following:

comp00 comp01 comp02 comp03 : DDTPATH_TAG/libexec/forge-backend DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_ARGUMENTS_TAG

This follows the same replacement rules described above and in detail in Integration with queuing systems.

The settings in the Options window for this example might be:

Using substitute MPI commands

Note Submit command and Submission template file in particular. Linaro MAP creates a new file and appends it to the submit command before executing it. In this case what would actually be executed might be mpiexec -config /tmp/linaro-temp-0112 or similar. Therefore, any argument like -config must be last on the line, because Linaro MAP will add a file name to the end of the line. Other arguments, if there are any, can come first.

Linaro recommend that you read the section on queue submission, because there are many features described there that might be useful to you if your system uses a non-standard start-up command.

If you do use a non-standard command, contact Forge Support.