The I/O Trap (Moving to Parallel MPI-IO)

To cure the Asymmetric Funnel, we need to stop routing all our data through Rank 0. Instead, we will use Parallel MPI-IO.

If this application is run on a cluster with a shared parallel file system (like Lustre or GPFS, as was the case when the current example was prepared), MPI-IO allows every rank to independently calculate its absolute byte-offset and write its chunk directly to the disk simultaneously.

Let’s look at the implementation in fire_io.cpp. We replace MPI_Gatherv with MPI_File_write_at_all:

// Calculate offsets
MPI_Offset header_offset = 2*sizeof(int);
MPI_Offset global_offset = (MPI_Offset)findex*NX_GLOBAL*NY_GLOBAL*sizeof(double);
MPI_Offset local_offset = (MPI_Offset)y_global_start*NX_GLOBAL*sizeof(double);
MPI_Offset total_byte_offset = header_offset + global_offset + local_offset;
// Write in parallel
MPI_File_write_at_all(fh, total_byte_offset, fire.get_temp() + NX_GLOBAL, NX_GLOBAL*Ny_loc, MPI_DOUBLE, MPI_STATUS_IGNORE);

The resulting Linaro MAP Volume-wise matrix is completely cured:

../../_images/map_mpi_comm_matrix_fire_io_volume.png

Fig. 29 Linaro MAP Volume-wise communication matrix for the MPI-IO version of the wildfire application run using an 800x800 grid for 2000 steps on 16 processes.

The solid column of network traffic to Rank 0 disappears entirely because the data goes straight to the file system.

If we look at the profile summary, we see that the total wall time remained the same (if even slightly went up), as (at least at this scale, on a shared-memory system) parallel IO alleviates the data funnel problem, but does not take less time than the combination of MPI_Gatherv and serial IO (note however that now Linaro MAP factored out MPI-IO time from the general MPI into a dedicated I/O field, while their sum remained near constant).

Linaro MAP profiling summary
============================
Profiling time:      17 seconds (between MPI init and finalize)
Peak process memory: 77455360 B (~73.9 MiB)

Compute:              17.9%     (2.9s) |=|
MPI:                  66.0%    (10.7s) |======|
I/O:                  16.1%     (2.6s) |=|
  (based on time on the main thread)

Note that on a shared-memory system without parallel filesystem, MPI-IO may carry somewhat higher overhead of and bring the total wall time slightly up. However, this is not a problem, or a flaw in our first patch. The goal of it is not to make application run faster, but to remove the potentially massive data funnel. If we scale this proxy app to thousands of cores on an HPC cluster to run a huge, high-resolution grid, MPI_Gatherv will attempt to pull gigabytes if not terabytes of data into Rank 0’s memory all at once. Rank 0 will then crash with an Out-Of-Memory (OOM) error, destroying the simulation. We accept a minor overhead on a single node to guarantee architectural survival and leverage true parallel bandwidth across multiple nodes.

Remaining Imbalance

If we now check the new Time-wise communication matrix, we don’t observe any qualitative changes:

../../_images/map_mpi_comm_matrix_fire_io_time.png

Fig. 30 Linaro MAP Time-wise communication matrix for the MPI-IO version of the wildfire application run using an 800x800 grid for 2000 steps on 16 processes.

There is still a significant time difference between the waiting time spent in collective MPI calls in the central-upper and lower (as well a couple of upper-most) ranks, due to the same reasons as described previously - heavy load-imbalance. This is addressed in the next section.