Introduction and Problem Statement
The Scenario
Our proxy application simulates a wildfire spreading across a 2D grid. The top half of our domain is covered in dense, old-growth Forest, while the bottom half is covered in light, fast-burning Brush. Fire starts in the very center of the grid, at the boundary of the Forest and Brush. Below you can see three frames from the beginning, middle, and end of the 2000-step simulation on a 800 x 800 grid.
Fig. 22 1st frame (step 10) of the baseline wildfire application run using an 800x800 grid on 16 processes - fire is ignited in the center of the grid (at the border of the Forest and the Brush) at a very high temperature.
Fig. 23 50th frame (step 500) of the baseline wildfire application run using an 800x800 grid on 16 processes - fire is propagating from the center of the grid asymmetrically - Forest (upper half) burns slower but resulting in higher temperatures.
Fig. 24 198th frame (step 2000) of the baseline wildfire application run using an 800x800 grid on 16 processes - fire has propagated much further from the center of the grid, with heterogeneity between the two substrates maintained. One can see that the center of the Brush has burned through and is progressively cooling down, while center of the Forest is still burning, albeit at a lower temperature.
The Computational Challenge
In nature, different fuels burn differently, which is leveraged in our simulation to create a significant computational disparity across the two domains:
The Forest (Heavy Compute): Burning dense and tall wood requires calculating stiff Arrhenius chemical kinetics. For every burning forest cell, the CPU must execute a computationally expensive
std::exp()function inside a many-iteration (exact number depends on the step size, with 500 iterations performed for the initial time step of 1.0) sub-loop to maintain mathematical stability.The Brush (Light Compute): Dry and much less dense brush and grass burn fast and therefore fire propagation in this domain can be simulated with linear approximation. It requires only a handful of basic floating-point multiplications per cell.
The Architecture
The application is written in C++ using MPI. It utilizes a classic 5-point stencil with double-buffered Jacobi heat diffusion. The 2D grid is divided among the MPI processes using a standard 1D Block Row Decomposition - meaning each MPI rank gets a contiguous horizontal slice of the terrain and performs a standard halo exchange with its top and bottom neighbors.
The Goal
At first glance, dividing a simulation grid evenly across the available MPI ranks seems reasonable - every processor gets (mostly) the same number of rows. However, as we will see, treating all rows equally in a heterogeneous and dynamic physics environment can create performance bottlenecks.
We will use MAP’s MPI Comm Matrix to capture these bottlenecks, analyze the generated MPI communication matrices, and step-by-step engineer our way out of the traps.