Run-time job progress reporting

In offline mode, you can compile a snapshot of a job, including its stacks and variables, and update the session log with that information. This includes writing the HTML log file, which otherwise is only written when the session has completed.

Snapshots can be triggered periodically via a command-line option, or at any point in the session by sending a signal to the front-end.

Periodic snapshots

Snapshots can be triggered periodically throughout a debugging session with the command-line option --snapshot-interval=MINUTES. For example, to log a snapshot every three minutes:

ddt --offline -o log.html --snapshot-interval=3 \\
mpiexec -n 8 ./myprog

Signal-triggered snapshots

Snapshots can also be triggered by sending a SIGUSR1 signal to the front-end process (called forge.bin in process lists), regardless of whether or not the --snapshot-interval command-line option was specified. For example, after running the following:

ddt --offline -o log.html mpiexec -n 8 ./myprog

A snapshot can be triggered by running (in another terminal):

# Find PID of DDT front-end:
pgrep forge.bin
> 18032
> 18039

# Use pstree to identify the parent if there are multiple PIDs:
pstree -p

# Trigger the snapshot:
kill -SIGUSR1 18032