Use custom MPI scripts

On some systems a custom mpirun replacement is used to start jobs, such as mpiexec. DDT normally uses whatever the default for your MPI implementation is, so for Open MPI it would look for mpirun and not mpiexec. Here we explain how to configure DDT to use a custom mpirun command for job start up.

There are typically two ways you might want to start jobs using a custom script, and DDT supports them both.

In the first way, you pass all the arguments on the command-line, like this:

mpiexec -n 4 /home/mark/program/chains.exe /tmp/mydata

There are several key variables in this command that DDT can fill in for you:

The number of processes (4 in the above example).
The name of your program (/home/mark/program/chains.exe).
One or more arguments passed to your program (/tmp/mydata).

Everything else, like the name of the command and the format of its arguments remains constant. To use a command like this in DDT, you adapt the queue submission system described in Job scheduling with jsrun. For this mpiexec example, the settings are:

As you can see, most of the settings are left blank. There are some differences between the Submit command in DDT and what you would type at the command-line:

The number of processes is replaced with NUM_PROCS_TAG.
The name of the program is replaced by the full path to forge-backend.
The program arguments are replaced by PROGRAM_ARGUMENTS_TAG.

Note

It is not necessary to specify the program name here. DDT takes care of that during its own startup process. The important thing is to make sure your MPI implementation starts forge-backend instead of your program, but with the same options.

In the second way, you start a job using a custom mpirun replacement with a settings file:

mpiexec -config /home/mark/myapp.nodespec

Where myfile.nodespec might contain something similar to the following:

comp00 comp01 comp02 comp03 : /home/mark/program/chains.exe /tmp/mydata

DDT can automatically generate simple configuration files like this every time you run your program, you need to specify a template file. For the above example, the template file myfile.ddt would contain:

comp00 comp01 comp02 comp03 : DDTPATH_TAG/libexec/forge-backend DDT_DEBUGGER_ARGUMENTS_TAG PROGRAM_ARGUMENTS_TAG

This follows the same replacement rules described above and in detail in Integration with queuing systems.

The settings in the Options window for this example might be:

Note Submit command and Submission template file in particular. Arm® DDT creates a new file and appends it to the submit command before executing it. In this case, mpiexec -config /tmp/ddt-temp-0112 or similar is executed. Therefore, any argument like -config must be last on the line, because Arm® DDT adds a file name to the end of the line. If there are any other arguments, they can come first.

If your system uses a non-standard startup command, Arm recommends that you read Job scheduling with jsrun, which describes many features that might be useful to you.

If you do use a non-standard command, contact Arm support.