Guiding Principles for Running MPI

Owned by Kester Scandrett
Last updated: 29 Mar, 2021 by Duy Pham
2 min read

Four main factors will determine how you configure a model to optimize the efficiency of a simulation.  To run a model most efficiently requires balancing these factors:

  1. Number of grid cells

  2. Number of constituents to be simulated

  3. Number of physical cores available for processing

  4. The amount of data exchanged between sub-domains in each time step

1. Number of grid cells

A general rule of thumb is that the number of cells assigned to a given core should be in the order of some thousands. A minimum of approximately 1,000 cells is needed to make an MPI computation worthwhile.  An individual core would be expected to be assigned from 500 to 10,000 cells.  20,000 cells should be the maximum number of cells on any one core. For example, if we have a hydrodynamic model with 60,000 cells and 6 cores, then the model could initially be divided into six subdomains, each with 10,000 cells.  The user should then adjust the run configuration by increasing or decreasing subdomains.

2. Number of constituents

Besides hydrodynamics, each transport process added to the simulation such as salinity, temperature, dye, sediment, toxic, or water quality parameters, requires more equations to be solved. Although the processing of the constituents in each sub-domain can be parallelized using OMP, if more constituents need to be considered, then it is recommended that more domains be used.

3. Number of physical cores

The goal of the modeler should be to ensure that an equal computational load is placed on each processor.  This means that if some models have different numbers of layers in different parts of the grid, thought should be given to how to divide the domain.  For example, a Sigma-Zed model could fewer layers in some parts of the domain.  The domain should then be divided in such a way that more horizontal cells are in that one sub-domain so that the total number of vertical and horizontal cells are roughly equal for each sub-domain.

4. The amount of data exchanged between sub-domains in each time step

The information at the interfaces of the subdomain is exchanged between the subdomains at each time step. If more constituents are simulated, especially for water quality problems, the amount of data exchanged between the subdomain could be very significant. This should be minimized by optimization of the number of cell connections between the subdomains at the interfaces.