serial meshes to be partitioned and saved in parallel format
using one processor. This capability allows MFEM to work around
the current bottleneck which requires every MPI rank to load
the big serial mesh before it can be partitioned.
This new capability is added to the `mesh-explorer` miniapp with
the new menu option `D` and is based on two new classes:
* `MeshPartitioner`, constructed from a serial mesh and any
partitioning array. Once constructed, it can extract any part
of the mesh consisting of the elements with ids `elem_id` such
that `partitioning[elem_id] == part_id`. The extracted mesh
part is given in the form of a `MeshPart` object.
* `MeshPart`, which is currently construced by a `MeshPartitioner`.
In the future, it can be created by other methods to facilitate
other capabilities such as parallel mesh re-partitioing. Once
constructed, the `MeshPart` can be saved to a file using the
text-based parallel MFEM format. Support for other formats can
be added as well. Another capability that can be added is the
MPI communication of `MeshPart` objects between different ranks
which can be used, for example, for implementing a `ParMesh`
constructor that needs the serial mesh only on one processor.
Current limitations:
* Non-conforming and NURBS meshes are not supported.
* Meshes with nodes (e.g. high-order or periodic meshes) are
not supported.
Small extension: if `Mesh::SetCurvature` is called with `order = 0`
then the method will remove the nodal `GridFunction` and use the
vertices array instead. This "curvature removal" capability can be
used from the `mesh-explorer` miniapp with the `c` menu option,
by specifying 0 when prompted to enter order.
Temporary change: `ex1p.cpp` is modified to disregard the (serial)
mesh specified with the option `-m`/`--mesh`. Instead, it loads
the parallel mesh `../miniapps/meshing/mesh-explorer.mesh.<rank>`
produced by the `D` menu option of the `mesh-explorer` miniapp.
performs MPI initialization using MPI_Init_thread; added method
Mpi::Init_auto that selects what MPI init mode to use based on
the configured external packages used by MFEM -- currently it
calls Mpi::Init_thread(MPI_THREAD_MULTIPLE) when using STRUMPACK
built with SLATE or PT-Scotch, otherwise it calls Mpi::Init.
In the examples that can use STRUMPACK, use Mpi::Init_auto instead
of Mpi::Init.
Fix a small unrelated issue noticed during testing: in
miniapps/multidomain/multidomain.cpp, do not open GLVis socket
connections when visualization is disabled.
stage of the method Create(). The previous implementation was
using MPI_Recv() with MPI_ANY_SOURCE which was causing issues
where one rank was receiving data sent from different calls to
Create().
In class GroupCommunicator, add const to the arguments of the
methods Create() and SetLTDofTable().
MPI_Session - the only communicator that makes sense with this class is
MPI_COMM_WORLD.
Replace `mfem::err.GetStream()` with `mfem::err`.
Remove #include "communication.hpp" from error.hpp.
general/globals.[ch].
Renamed class WrappedOStream to OutStream and use a different
implementation in order to support the case when the OutStream is passed
as a parameter of type std::ostream.
In class MPI_Session, added an MPI_Comm parameter to both constructors,
with a default value of MPI_COMM_WORLD.
Hide the global variable MFEM_COMM_WORLD and allow access (read and
write) with global functions: GetGlobalMPI_Comm() and
SetGlobalMPI_Comm() - in the future, we may want to perform some tasks
when changing the "global" MPI communicator.
In mfem_error(), use GetGlobalMPI_Comm() instead of MPI_COMM_WORLD.
Minor formatting changes and #include reordering.
while calling the original mode byGroup. The new mode, which is also the
new default, aggregates communications between the same two ranks that
may consist of data corresponding to multiple groups, thereby reducing
the total number of MPI send and receive calls. This optimization
addresses the slowdown observed in the last commit for small problem
sizes (per rank) while also providing small improvements for medium and
large problem sizes.
The communication mode of a GroupCommunicator can be specified with the
new second parameter of its constructor which has a default value
byNeighbor.
Add two new methods, mostly useful for debugging/diagnostics:
GroupCommunicator::PrintInfo() and HypreParMatrix::PrintCommPkg().
Andrew Ho