Improve #include logic.

default stream 0.

Optimize OperatorJacobiSmoother::Mult by fusing kernels as much
as possible.

.github/stale.yml

@@ -36,7 +36,7 @@ staleLabel: stale
 # Comment to post when marking an issue as stale. Set to `false` to disable
 markComment: >
   :warning: This issue or PR has been automatically marked as stale because it has not
-  had any activity in the last month. *If no activity occurs in the next week, it will
+  had any activity in the last month. *If no activity occurs in the next week, it will 
   be automatically closed.* Thank you for your contributions.
 
 # Comment to post when closing a stale issue. Set to `false` to disable

.gitlab-ci.yml

@@ -76,7 +76,7 @@ stages:
   - baseline_check
   - baseline_publish
 
-# The setup job in setup stage don't rely on MFEM git repo. It prepares a
+# The setup job in setup stage don’t rely on MFEM git repo. It prepares a
 # pipeline-wide working directory downloading/updating external repos.
 # TODO: updating tests and tpls is not necessary anymore since pipelines are
 # now using unique directories so repo are never shared with another pipeline.
@@ -226,7 +226,7 @@ setup:
     echo "${BASELINE_TEST}: Differences found, replacement file generated"
     cp ${_base_out} ${CI_PROJECT_DIR}/${ARTIFACTS_DIR}/${_base_out}.txt
   fi
-  # _base_diff won't even exist if there is no difference.
+  # _base_diff won’t even exist if there is no difference.
   if [[ -f ${_base_diff} ]]
   then
     echo "${BASELINE_TEST}: Relevant differences (filtered diff) ..."

CHANGELOG

@@ -5,141 +5,21 @@
                       | | | | | ||  _||  __/| | | | | |
                       |_| |_| |_||_|   \___||_| |_| |_|
 
-                               https://mfem.org
+                               http://mfem.org
 
 
-Version 4.2.1 (development)
+Version 4.1.1 (development)
 ===========================
 
-- Added interface to MUMPS direct solver. Its usage is demonstrated in ex25p.
-  See http://mumps.enseeiht.fr/ for more details. Supported versions >= 5.1.1.
+libCEED integration improvements
+--------------------------------
+- Add support for all types of (scalar) Coefficient.
 
-- Added three ESDIRK time integrators: implicit trapezoid rule, L-stable
-  ESDIRK-32, and A-stable ESDIRK-33.
+- Add support for VectorMassIntegrator and VectorDiffusionIntegrator.
 
-- Added a "scaled Jacobian" visualization option in the Mesh Explorer miniapp to
-  help identify elements with poor mesh quality.
-
-- Upgraded the Catch unit test framework from version 2.13.0 to version 2.13.2.
-
-- Implemented a filter method for the Navier miniapp to stabilize highly
-  turbulent flows in direct numerical simulation.
-
-
-Version 4.2, released on October 30, 2020
-=========================================
-
-High-Performance Computing
---------------------------
-- Added support for explicit vectorization in the high-performance templated
-  code, which can now take advantage of specific classes on the following
-  architectures:
-  * x86 (SSE/AVX/AVX2/AVX512),
-  * Power8 & Power9 (VSX),
-  * BG/Q (QPX).
-  These are disabled by default, but can be enabled with MFEM_USE_SIMD=YES.
-  See the new file linalg/simd.hpp and the new directory linalg/simd.
-
-- Added an Element Assembly mode compatible with GPU device execution for H1 and
-  L2 spaces in the mass, convection, diffusion, transpose, and the face DG trace
-  integrators. See option '-ea' in Example 9. When enabled, this assembly level
-  stores independent dense matrices for the elements, and independent dense
-  matrices for the faces in the DG case.
-
-- Added a Full Assembly mode compatible with GPU device execution. This assembly
-  level builds on top of the Element Assembly kernels to compute a global sparse
-  matrix. All integrators supported by element assembly are also supported by
-  full assembly. See the '-fa' option in Example 9.
-
-- Optimized the AMD/HIP kernel support and enabled HIP support in the libCEED
-  integration. This is now available via the "ceed-hip" device backend.
-
-- Improved the libCEED integration to support:
-  * AssemblyLevel::NONE for Mass, Diffusion, VectorMass, and VectorDiffusion
-    Integrators. This level computes the full operator evaluation "on the fly".
-  * VectorMassIntegrator and VectorDiffusionIntegrator.
-  * All types of (scalar) Coefficients.
-
-- Added partial assembly / device support for:
-  * H(div) bilinear forms and VectorFEDivergenceIntegrator.
-  * BlockOperator, see the updated Example 5.
-  * Complex operators, see the updated Example 22.
-  * Chebyshev accelerated polynomial smoother.
-  * Convergent diagonal preconditioning on non-conforming adaptively refined
-    meshes, see Example 6/6p.
-
-- Added CUDA support for:
-  * Sparse matrix-vector multiplication with cuSPARSE,
-  * SUNDIALS ODE integrators, see updated SUNDIALS modification of Example 9/9p.
-
-Linear and nonlinear solvers
-----------------------------
-- Added a new solver class for simple integration with NVIDIA's multigrid
-  library, AmgX. The AmgX class is designed to work as a standalone solver or
-  preconditioner for existing MFEM solvers. It uses MFEM's sparse matrix format
-  for serial runs and the HypreParMatrix format for parallel runs. The new
-  solver may be configured to run with one GPU per MPI rank or with more MPI
-  ranks than GPUs. In the latter case, matrices and vectors are consolidated to
-  ranks communicating with the GPUs and the solution is then broadcasted.
-
-  Although CUDA is required to build, the AmgX support is compatible with the
-  MFEM CPU device configuration. The examples/amgx folder illustrates how to
-  integrate AmgX in existing MFEM applications. The AmgX solver class is
-  partially based on: "AmgXWrapper: An interface between PETSc and the NVIDIA
-  AmgX library", by Pi-Yueh Chuang and Lorena A. Barba, doi:10.21105/joss.00280.
-
-- Added initial support for geometric h- and p-multigrid preconditioners for
-  matrix-based and matrix-free discretizations with basic GPU capability, see
-  Example 26/26p.
-
-- Added support for the CVODES package in SUNDIALS which provides ODE solvers
-  with sensitivity analysis capabilities. See the CVODESSolver class and the new
-  adjoint miniapps in the miniapps/adjoint directory.
-
-- Added an interface to the MKL CPardiso solver, an MPI-parallel sparse direct
-  solver developed by Intel. See Example 11p for an illustration of its usage.
-
-- Added support for the SLEPc eigensolver package, https://slepc.upv.es.
-
-- Upgraded SuperLU interface to use SuperLU_DIST 6.3.1. Added a simple SuperLU
-  example in the new directory examples/superlu.
-
-- Extended the KINSOL (SUNDIALS) nonlinear solver interface to support the
-  Jacobian-free Newton-Krylov method. A usage example is shown in Example 10p.
-
-- Block arrays of parallel matrices can now be merged into a single parallel
-  matrix with the function HypreParMatrixFromBlocks. This could be useful for
-  solving block systems with parallel direct solvers such as STRUMPACK.
-
-- Added wrappers for hypre's flexible GMRES solver and the new parallel ILU
-  preconditioner. The latter requires hypre version 2.19.0 or later.
-
-Discretization improvements
----------------------------
-- Extended GSLIB-FindPoints integration to support simplices and interpolation
-  of functions from L2, H(div) and H(curl) spaces.
-
-- Added support for computing asymptotic error estimates and convergence rates
-  for the whole de Rham sequence based on the new class ConvergenceStudy and new
-  member methods in GridFunction and ParGridFunction. See the rates.cpp file in
-  the tests/convergence directory for sample usage.
-
-- Extended the GetValue and GetVectorValue methods of GridFunction to support
-  evaluation on boundary elements and, in the continuous field case, arbitrary
-  mesh edges and faces. This requires passing an ElementTransformation argument.
-
-- Added support for matrix-free interpolation and restriction operators between
-  continuous H1 finite element spaces of different order on the same mesh or
-  with the same order on uniformly refined meshes.
-
-- The Coefficient classes based on a C-function pointer (FunctionCoefficient,
-  VectorFunctionCoefficient and MatrixFunctionCoefficient) now use the more
-  general std::function class template. This allows the classes to be backward
-  compatible (i.e. they can still work with C-functions) and, in addition,
-  support any "callable", e.g. lambda functions.
-
-- Non-conforming meshes are now supported with block nonlinear forms. See the
-  updated Example 19/19p.
+- Add support for AssemblyLevel::NONE for MassIntegrator, DiffusionIntegrator,
+  VectorMassIntegrator, and VectorDiffusionIntegrator. This level of assembly
+  fully applies on the fly the finite element operator.
 
 Meshing improvements
 --------------------
@@ -148,55 +28,188 @@ Meshing improvements
   always available. The interface has also been improved, see for example the
   Mesh Explorer miniapp.
 
-- Added support for finite difference-based gradient and Hessian approximation
-  in the TMOP mesh optimization algorithms. This improves the accuracy of the
-  Hessian for r-adaptivity using discrete fields, and allows use of skewness
-  and orientation based metrics.
-
 - Improved Gmsh reader (version 2.2), which now supports both high-order and
   periodic meshes. Segments, triangles, quadrilaterals, and tetrahedra are
   supported up to order 10. Wedges and hexahedra are supported up to order 9.
   For sample periodic meshes, see the periodic*.msh files in the data directory.
 
-- Added support for construction of (serial) non-conforming meshes. Hanging
-  nodes can be marked with Mesh::AddVertexParents when building the mesh with
-  the "init" constructor. The usage is demonstrated in a new meshing miniapp,
-  Polar NC, which generates meshes that are non-conforming from the start.
+- Added support for finite difference-based gradient and Hessian approximation
+  in the TMOP mesh optimization algorithms. This improves the accuracy of the
+  Hessian for r-adaptivity using discrete fields, and allows use of skewness
+  and orientation based metrics.
 
 - Added support for r-adaptivity with more than one discrete field. This allows
-  the user to specify different discrete functions for controlling the size,
-  aspect-ratio, orientation, and skew of elements in the mesh.
+  the user to specify different discrete functions for controlling the
+  size, aspect-ratio, orientation, and skew of elements in the mesh.
 
-- Additional TMOP improvements:
-  * Capability for approximate tangential mesh relaxation.
-  * Support and examples for using TMOP on mixed meshes.
-  * Complete integrator action accounting for spatial derivatives of discrete
-    and analytic targets.
+- Added TMOP capability for approximate tangential mesh relaxation. Added
+  support and examples for using TMOP on mixed meshes.
+
+- Added complete action of the TMOP Integrator to account for the spatial
+  derivatives of discrete and analytic targets.
+
+- Added support for initialization of (serial) non-conforming meshes. Hanging
+  nodes can be marked with Mesh::AddVertexParents when building the mesh with
+  the "init" constructor. The usage is demonstrated in a new meshing miniapp
+  (polar-nc) which generates meshes that are non-conforming from the start.
+
+Performance improvements
+------------------------
+- Added support for explicit vectorization in the high-performance templated
+  code, which can now take advantage of specific intrinsics classes on the
+  following architectures:
+    - x86 (SSE/AVX/AVX2/AVX512),
+    - Power8 & Power9 (VSX),
+    - BG/Q (QPX).
+  These are disabled by default, and can be enabled with MFEM_USE_SIMD=YES.
+  See the new file linalg/simd.hpp and the new directory linalg/simd.
+
+Improved GPU capabilities
+-------------------------
+- Added a new solver class for simple integration with NVIDIA's multigrid
+  library, AmgX. The AmgX class is designed to work as a standalone solver or
+  preconditioner for existing MFEM solvers. It uses MFEM's sparse matrix format
+  for serial runs and the HypreParMatrix format for parallel runs.
+
+  The new solver may be configured to run with one GPU per MPI rank or with more
+  MPI ranks than GPUs. In the latter case, matrices and vectors are consolidated
+  to ranks communicating with the GPUs and the solution is then broadcasted.
+
+  Although CUDA is required to build, the AmgX support is compatible with the
+  MFEM CPU device configuration. The examples/amgx folder illustrates how to
+  integrate AmgX in existing MFEM applications.
+
+  The AmgX solver class is partially based on: "AmgXWrapper: An interface
+  between PETSc and the NVIDIA AmgX library", by Pi-Yueh Chuang and Lorena
+  A. Barba, doi:10.21105/joss.00280.
+
+- Added support for Chebyshev accelerated polynomial smoother on GPU.
+
+- Optimized AMD/HIP kernel support.
+
+- Enabled HIP support in the libCEED integration, which is now available via the
+  "ceed-hip" device backend.
+
+- Added a Full Assembly mode compatible with Device kernel execution. This
+  assembly level builds on top of the current Element Assembly kernels to
+  compute a global sparse matrix. All integrators supported by element assembly
+  are also supported by full assembly. See the '-fa' option in Example 9.
+
+- Added CUDA support for sparse matrix-vector multiplication with cuSPARSE.
+
+- Added support for BlockOperator on GPU. See the updated Example 5.
+
+- Added partial assembly and GPU support for complex operators, including the
+  classes ComplexOperator, [Par]ComplexGridFunction, [Par]ComplexLinearForm, and
+  [Par]SesquilinearForm. See the updated Example 22.
+
+- Added CUDA support for SUNDIALS ODE integrators. See the updated SUNDIALS
+  modification of Example 9/9p.
+
+Discretization improvements
+---------------------------
+- Added support for matrix-free interpolation and restriction operators between
+  continuous H1 finite element spaces of different order on the same mesh or
+  with the same order on uniformly refined meshes.
+
+- Added support for simplices in GSLIB-FindPoints.
+
+- Added support for H1 and L2 element matrix assembly in the mass, convection,
+  diffusion, transpose, and the face DG trace integrators. This is compatible
+  with GPU device execution and is illustrated in Example 9/9p, see the option
+  '-ea'. When enabled, this level of assembly stores independent dense matrices
+  for the elements, and independent dense matrices for the faces in the DG case.
+
+- Added new partial assembly kernels for H(div) bilinear forms, as well as
+  VectorFEDivergenceIntegrator.
+
+- Improved the documentation of the GridFunction GetValue and GetVectorValue
+  methods. Expanded the GetValue and GetVectorValue methods which accept an
+  ElementTransformation argument to support evaluation on boundary elements
+  and, in the continuous field case, arbitrary mesh edges and faces.
+
+- Added new coefficient and vector coefficient classes for QuadratureFunctions.
+  Additionally, new LinearForm integrators were also added which make use of
+  these new QuadratureFunction coefficient classes.
+
+- Non-conforming meshes are now supported with block nonlinear forms. See the
+  updated Example 19/19p.
+
+- Added support face integrals on the boundaries of NURBS meshes.
+
+- Added support for interpolation of functions in L2, H(div) and H(curl)
+  spaces using GSLIB-FindPoints.
+
+- Added support for computing asymptotic error estimates and convergence rates
+  for the whole de Rham sequence based on the new class ConvergenceStudy and new
+  member methods in GridFunction and ParGridFunction. See the rates.cpp file in
+  the tests/convergence directory for sample usage.
+
+- The C-function based coefficient classes (FunctionCoefficient,
+  VectorFunctionCoefficient, and MatrixFunctionCoefficient) now use the more
+  general std::function class template. This allows the classes to be backward
+  compatible (i.e. they can still work with C-functions) and, in addition,
+  support any "callable", e.g. lambda functions.
+
+Linear and nonlinear solvers
+----------------------------
+- Added power method to iteratively estimate the largest eigenvalue and the
+  corresponding eigenvector of an operator.
+
+- Added initial support for h- and p-multigrid solvers and preconditioners for
+  matrix-based and matrix-free discretizations with basic GPU capability.
+
+- Added wrappers for Hypre's flexible GMRES solver and the new parallel ILU
+  preconditioner. The latter requires hypre version 2.19.0 or later.
+
+- Added a new IterativeSolverMonitor class that allows to monitor the residual
+  and solution during the solving process of an IterativeSolver after every
+  iteration.
+
+- Added support for the CVODES package in SUNDIALS which provides ODE
+  solvers with sensitivity analysis capabilities. See the CVODESSolver
+  class and the new adjoint miniapps below.
+
+- Block arrays of parallel matrices can now be merged into a single parallel
+  matrix with the function HypreParMatrixFromBlocks. This could be useful for
+  solving block systems with parallel direct solvers such as STRUMPACK.
+
+- In SLISolver, changed the residual inner product from (Br,r) to (Br,Br) so the
+  solver can work with non-SPD preconditioner B.
+
+- Added support for the SLEPc eigensolver package.
+
+- Added partially assembled convergent diagonal preconditioner for adaptively
+  refined meshes (i.e. non-conforming finite element spaces), see Example 6/6p.
+
+- Upgraded SuperLU interface to use SuperLU_DIST 6.3.1. Added a simple SuperLU
+  example in the new directory examples/superlu.
+
+- Extended the KINSOL (SUNDIALS) nonlinear solver interface to support the
+  Jacobian-free Newton-Krylov method. A usage example is shown in Example 10p.
+
+- Added an interface to the MKL CPardiso solver -- an MPI-parallel sparse direct
+  solver developed by Intel. See Example 11p for an illustration of its usage.
 
 New and updated examples and miniapps
 -------------------------------------
+- Added a new example, Example 25/25p, to demonstrate the use of a Perfectly
+  Matched Layer (PML) for the simulation of electromagnetic wave propagation.
+  The example defines and solves several indefinite Maxwell problems.
+
+- Added a new Example 26/26p to demonstrate the construction of a matrix-free
+  geometric and p-multigrid preconditioner for the Laplace problem.
+
+- Added a new example, Example 27/27p, to demonstrate the enforcement of various
+  boundary conditions with the Laplace operator. The example shows the procedure
+  for applying Dirichlet, Neumann (both homogeneous and inhomogeneous), Robin,
+  and periodic boundary conditions with either H1 or DG discretizations.
+
 - Added a new miniapp, Navier, that solves the time-dependent Navier-Stokes
   equations of incompressible fluid dynamics. See the miniapps/navier directory
   for more details.
 
-- Added 10 new example codes:
-  * Example 25/25p demonstrates the use of a Perfectly Matched Layer (PML) for
-    electromagnetic wave propagation (indefinite Maxwell).
-  * Example 26/26p shows how to construct matrix-free geometric and p-multigrid
-    preconditioner for the Laplace problem.
-  * Example 27/27p demonstrates the enforcement of Dirichlet, Neumann, Robin,
-    and periodic boundary conditions with either H1 or DG Laplace problems.
-  * Versions of Example 1/1p in examples/amgx demonstrating the use of AmgX,
-    to solve the Laplace problem with AMG preconditioning on GPUs.
-  * A version of Example 11p in examples/petsc demonstrating the use of SLEPc,
-    to solve the Laplace eigenproblem with shift-and-invert transformation.
-  * A version of Example 1 in examples/superlu demonstrating the use of SuperLU
-    to solve the Laplace problem.
-
-- Added a new Field Interpolation miniapp in miniapps/gslib that demonstrates
-  transfer of grid functions between different meshes using GSLIB-FindPoints.
-
-- Added 2 miniapps in the new miniapps/adjoint directory demonstrating how to
+- Added a new miniapps/adjoint directory with two miniapps demonstrating how to
   solve adjoint problems in MFEM using the CVODES package in SUNDIALS. Both of
   these miniapps require the MFEM_USE_SUNDIALS configuration option.
   * The cvsRoberts_ASAi_dns miniapp solves a backward adjoint problem for a
@@ -204,74 +217,82 @@ New and updated examples and miniapps
   * The adjoint_advection_diffusion miniapp solves a backward adjoint problem
     for an advection diffusion PDE, evaluating adjoint quadratures in parallel.
 
-- Added 4 additional meshing miniapps:
-  * The Minimal Surface miniapp solves Plateau's problem: the Dirichlet problem
-    for the minimal surface equation.
-  * The Twist miniapp demonstrates how to stitch together opposite surfaces of a
-    mesh to create a topologically periodic mesh.
-  * The Trimmer miniapp trims away parts of a mesh based on element attributes.
-  * Polar NC shows the construction of polar non-conforming meshes.
+- Ported Example 11p to SLEPc, to demonstrate solving the Laplace eigenvalue
+  equation with the shift-and-invert spectral transformation method.
 
-- Several examples and miniapps were updated to include:
-  * Full and element assembly support in Example 9/9p.
-  * Partial assembly with diagonal preconditioning in Examples 4/4p/5/5p/22/22p.
-  * Diagonal preconditioner in Example 6/6p for partial assembly with AMR.
-  * The option to plot a function in Mesh Explorer.
-  * A new test problem showing a mixed bilinear form for H1, H(curl), H(div) and
-    L2, with partial assembly support in Example 24/24p.
-  * Weak Dirichlet boundary conditions (Nitsche) to the NURBS miniapp.
+- Added a simple meshing miniapp, Twist, which demonstrates MFEM's strategy of
+  stitching together opposite surfaces of a mesh to create a topologically
+  periodic mesh.
 
-Data management and Visualization
----------------------------------
-- Added support for ADIOS2 for parallel I/O with ParaView visualization. See
-  Examples 5, 9, 12, 16. The classes adios2stream and ADIOS2DataCollection
-  provide the interface to generate ADIOS2 Binary Pack (BP4) directory datasets.
+- Added a new meshing miniapp, Minimal Surface, which solves Plateau's problem:
+  the Dirichlet problem for the minimal surface equation.
 
-- Added VTU output of boundary elements and attributes and parallel VTU (PVTU)
-  output of parallel meshes for visualization using ParaView.
+- Added a new meshing miniapp, Polar NC, which demonstrates the construction of
+  polar non-conforming meshes.
+
+- Added partial assembly support to Example 4/4p and Example 5/5p, with diagonal
+  preconditioning.
+
+- Added full assembly support in Example 9/9p.
+
+- Added a new test problem in Example 24/24p, demonstrating a mixed bilinear
+  form for H1, H(curl), H(div) and L_2, with partial assembly support.
+
+- Added weak Dirichlet boundary conditions (Nitsche) to the NURBS miniapp.
+
+- Added a simple mesh editing miniapp, Trimmer, which trims away portions of a
+  mesh based on element attributes. Any newly exposed boundary elements are
+  assigned attribute numbers related to the trimmed element attributes.
+
+- Added a new miniapp (field-interp) that demonstrates transfer of grid function
+  between different meshes using GSLIB-FindPoints.
+
+- Added diagonal preconditioner in Example 6/6p for partial assembly with AMR.
+
+- Added device support in Example 5/5p.
+
+- Added partial assembly and device support to Example 22/22p, with diagonal
+  preconditioning.
+
+- Added the option to plot a function in Mesh Explorer.
 
 Improved testing
 ----------------
+- Upgraded the Catch unit test framework from version 1.6.1 to version 2.13.0.
+
 - Added a GitLab pipeline that automates PR testing on supercomputing systems
   and Linux clusters at Lawrence Livermore National Lab (LLNL). This can be
   triggered only by LLNL developers, see .gitlab-ci.yml, the .gitlab directory
   and the updated CONTRIBUTING.md file.
 
-- Added additional testing for convergence, the parallel mesh I/O, and for the
-  libCEED integration in MFEM in the tests/ directory.
+- Add tests for the libCEED integration in MFEM.
 
-- Upgraded the Catch unit test framework from version 1.6.1 to version 2.13.0.
+- Added testing of the parallel mesh format in tests/par-mesh-format.
 
 Miscellaneous
 -------------
-- Renamed "Backend::DEBUG" to "Backend::DEBUG_DEVICE" to avoid conflicts,
-  as DEBUG is sometimes used as a macro.
+- Added support for ADIOS2 for parallel I/O with ParaView visualization. The
+  classes adios2stream and ADIOS2DataCollection are introduced in mfem as the
+  interfaces to generate ADIOS2 Binary Pack (BP4) directory datasets for the
+  entire spatial and temporal node data. Cell centered data is accessible by
+  ADIOS2 data readers (e.g. Python), but currently not yet implement as of
+  ParaView v5.8.1. In addition, ADIOS2 allows for setting a user-defined number
+  of data substreams/subfiles at scale. See examples 5, 9, 12, 16.
 
-- Added a new IterativeSolverMonitor class that allows to monitor the residual
-  and solution with an IterativeSolver after every iteration.
-
-- Added power method to iteratively estimate the largest eigenvalue and the
-  corresponding eigenvector of an operator.
-
-- Added support for face integrals on the boundaries of NURBS meshes.
-
-- In SLISolver, changed the residual inner product from (Br,r) to (Br,Br) so the
-  solver can work with non-SPD preconditioner B.
-
-- Added new coefficient and vector coefficient classes for QuadratureFunctions,
-  with new LinearForm integrators which use them.
+- Added VTU output of boundary elements and attributes and parallel VTU (PVTU)
+  output of parallel meshes for visualization using ParaView.
 
 - The integration order used in the ComputeLpError and ComputeElementLpError
   methods of class GridFunction has been increased.
 
+- Various other simplifications, extensions, and bugfixes in the code.
+
+- Renamed "Backend::DEBUG" to "Backend::DEBUG_DEVICE" to avoid conflicts,
+  as DEBUG is sometimes used as a macro.
+
 - Change the IntegrationRule inside VectorDiffusionIntegrator to use the same
   quadrature as DiffusionIntegrator.
 
-- The README.html files previously included in several source directories have
-  been removed. Use the corresponding pages at mfem.org instead.
-
-- Various other simplifications, extensions, and bugfixes in the code.
-
 
 Version 4.1, released on March 10, 2020
 =======================================
@@ -602,7 +623,7 @@ Other meshing improvements
   follows precisely the paper:
 
      D. Arnold, A. Mukherjee, and L. Pouly, "Locally Adapted Tetrahedral Meshes
-     Using Bisection", SIAM J. Sci. Comput. 22 (2000), 431-448.
+     Using Bisection", SIAM J. Sci. Comput. 22 (2000), 431–448.
 
   This guarantees that the shape regularity of the elements will be preserved
   under refinement.
@@ -747,7 +768,7 @@ Discretization improvements
 
 - New specialized time integrators: symplectic integrators of orders 1-4 for
   systems of first order ODEs derived from a Hamiltonian and generalized-alpha
-  ODE solver for the filtered Navier-Stokes equations with stabilization. See
+  ODE solver for the filtered Navier–Stokes equations with stabilization. See
   classes SIASolver and GeneralizedAlphaSolver in linalg/ode.hpp.
 
 - Inherit finite element classes from the new base class TensorBasisElement,
@@ -1237,7 +1258,7 @@ New and improved linear solvers
   it can only be enabled along with MFEM_USE_MPI. When MFEM is configured with
   MFEM_USE_SUPERLU, one also needs to alter the version of METIS, since SuperLU
   requires ParMETIS (which comes packaged with a serial version of METIS).  See
-  http://crd-legacy.lbl.gov/~xiaoye/SuperLU for SuperLU_DIST details.
+  http://http://crd-legacy.lbl.gov/~xiaoye/SuperLU for SuperLU_DIST details.
 
 - Added a wrapper for the KLU solver in SuiteSparse see
   http://faculty.cse.tamu.edu/davis/suitesparse.html for details of KLU.

CMakeLists.txt

@@ -14,7 +14,7 @@ set(USER_CONFIG "${CMAKE_CURRENT_SOURCE_DIR}/config/user.cmake" CACHE PATH
   "Path to optional user configuration file.")
 
 # Require C++11 and disable compiler-specific extensions
-set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD 11)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 set(CMAKE_CXX_EXTENSIONS OFF)
 
@@ -50,7 +50,7 @@ project(mfem NONE)
 # Current version of MFEM, see also `makefile`.
 #   mfem_VERSION = (string)
 #   MFEM_VERSION = (int)   [automatically derived from mfem_VERSION]
-set(${PROJECT_NAME}_VERSION 4.2.1)
+set(${PROJECT_NAME}_VERSION 4.1.1)
 
 # Prohibit in-source build
 if (${PROJECT_SOURCE_DIR} STREQUAL ${PROJECT_BINARY_DIR})
@@ -185,7 +185,7 @@ if (MFEM_USE_MPI)
     endif()
   endif()
 else()
-  set(PKGS_NEED_MPI SUPERLU MUMPS PETSC SLEPC STRUMPACK PUMI)
+  set(PKGS_NEED_MPI SUPERLU PETSC SLEPC STRUMPACK PUMI)
   foreach(PKG IN LISTS PKGS_NEED_MPI)
     if (MFEM_USE_${PKG})
       message(STATUS "Disabling package ${PKG} - requires MPI")
@@ -264,15 +264,6 @@ if (MFEM_USE_SUPERLU)
   endif()
 endif()
 
-# MUMPS can only be enabled in parallel
-if (MFEM_USE_MUMPS)
-  if (MFEM_USE_MPI)
-    find_package(MUMPS REQUIRED mumps_common pord)
-  else()
-    message(FATAL_ERROR " *** MUMPS requires that MPI be enabled.")
-  endif()
-endif()
-
 # STRUMPACK can only be enabled in parallel
 if (MFEM_USE_STRUMPACK)
   if (MFEM_USE_MPI)
@@ -391,7 +382,7 @@ endif()
 #    integers, the METIS header (with 32-bit indices, as used by mfem) needs to
 #    be before SuiteSparse.
 set(MFEM_TPLS MPI_CXX OPENMP BLAS LAPACK METIS HYPRE SuiteSparse SUNDIALS PETSC
-    SLEPC MESQUITE SuperLUDist MUMPS STRUMPACK AXOM CONDUIT Ginkgo GNUTLS GSLIB NETCDF
+    SLEPC MESQUITE SuperLUDist STRUMPACK AXOM CONDUIT Ginkgo GNUTLS GSLIB NETCDF
     MPFR PUMI HIOP POSIXCLOCKS MFEMBacktrace ZLIB OCCA CEED RAJA UMPIRE ADIOS2
     CUSPARSE MKL_CPARDISO AMGX)
 # Add all *_FOUND libraries in the variable TPL_LIBRARIES.

CONTRIBUTING.md

@@ -1,12 +1,12 @@
 <p align="center">
-<a href="https://mfem.org/"><img alt="mfem" src="https://mfem.org/img/logo-300.png"></a>
+<a href="http://mfem.org/"><img alt="mfem" src="http://mfem.org/img/logo-300.png"></a>
 </p>
 
 <p align="center">
 <a href="https://github.com/mfem/mfem/blob/master/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-BSD-brightgreen.svg"></a>
 <a href="https://travis-ci.org/mfem/mfem"><img alt="Build Status" src="https://travis-ci.org/mfem/mfem.svg?branch=master"></a>
 <a href="https://ci.appveyor.com/project/mfem/mfem"><img alt="Build Status" src="https://ci.appveyor.com/api/projects/status/19non9sqm6msi2wy?svg=true"></a>
-<a href="https://mfem.github.io/doxygen/html/index.html"><img alt="Doxygen" src="https://img.shields.io/badge/code-documented-brightgreen.svg"></a>
+<a href="http://mfem.github.io/doxygen/html/index.html"><img alt="Doxygen" src="https://img.shields.io/badge/code-documented-brightgreen.svg"></a>
 </p>
 
 
@@ -26,7 +26,7 @@ request (PR) toward the `mfem:master` branch to propose your contribution. If
 you are planning significant code changes or have questions, you may want to
 open an [issue](https://github.com/mfem/mfem/issues) before issuing a PR. In
 addition to technical contributions, we are also interested in your results and
-[simulation images](https://mfem.org/gallery/), which you can share via a pull
+[simulation images](http://mfem.org/gallery/), which you can share via a pull
 request in the [mfem/web](https://github.com/mfem/web) repo.
 
 See the [Quick Summary](#quick-summary) section for the main highlights of our
@@ -51,7 +51,7 @@ back to them before issuing pull requests:
 Contributing to MFEM requires knowledge of Git and, likely, finite elements. If
 you are new to Git, see the [GitHub learning
 resources](https://help.github.com/articles/git-and-github-learning-resources/).
-To learn more about the finite element method, see our [FEM page](https://mfem.org/fem).
+To learn more about the finite element method, see our [FEM page](http://mfem.org/fem).
 
 *By submitting a pull request, you are affirming the [Developer's Certificate of
 Origin](#developers-certificate-of-origin-11) at the end of this file.*
@@ -92,16 +92,18 @@ The MFEM source code has the following structure:
   .
   ├── config
   │   └── cmake
-  │       └── ...
+  │       └── modules
   ├── data
   ├── doc
+  │   └── web
+  │       └── examples
   ├── examples
   │   ├── amgx
   │   ├── ginkgo
   │   ├── hiop
   │   ├── petsc
   │   ├── pumi
-  │   ├── sundials
+  │   └── sundials
   |   └── superlu
   ├── fem
   │   └── libceed
@@ -124,8 +126,9 @@ The MFEM source code has the following structure:
       ├── convergence
       ├── par-mesh-format
       ├── scripts
-      └── unit
-          └── ...
+      ├── unit
+      │   ├── ...
+      └── ...
 ```
 
 #### Main directories and classes
@@ -135,24 +138,24 @@ classes implementing the finite element, mesh and linear algebra concepts
 respectively.
 
 - The main mesh classes are:
-  + [`Mesh`](https://mfem.github.io/doxygen/html/classmfem_1_1Mesh.html)
-  + [`NCMesh`](https://mfem.github.io/doxygen/html/classmfem_1_1NCMesh.html)
-  + [`Element`](https://mfem.github.io/doxygen/html/classmfem_1_1Element.html)
-  + [`ElementTransformation`](https://mfem.github.io/doxygen/html/classmfem_1_1ElementTransformation.html)
+  + [`Mesh`](http://mfem.github.io/doxygen/html/classmfem_1_1Mesh.html)
+  + [`NCMesh`](http://mfem.github.io/doxygen/html/classmfem_1_1NCMesh.html)
+  + [`Element`](http://mfem.github.io/doxygen/html/classmfem_1_1Element.html)
+  + [`ElementTransformation`](http://mfem.github.io/doxygen/html/classmfem_1_1ElementTransformation.html)
 
 - The main finite element classes are:
-  + [`FiniteElement`](https://mfem.github.io/doxygen/html/classmfem_1_1FiniteElement.html)
-  + [`FiniteElementCollection`](https://mfem.github.io/doxygen/html/classmfem_1_1FiniteElement.html)
-  + [`FiniteElementSpace`](https://mfem.github.io/doxygen/html/classmfem_1_1FiniteElementSpace.html)
-  + [`GridFunction`](https://mfem.github.io/doxygen/html/classmfem_1_1GridFunction.html)
-  + [`BilinearFormIntegrator`](https://mfem.github.io/doxygen/html/classmfem_1_1BilinearFormIntegrator.html) and [`LinearFormIntegrator`](https://mfem.github.io/doxygen/html/classmfem_1_1LinearFormIntegrator.html)
-  + [`LinearForm`](https://mfem.github.io/doxygen/html/classmfem_1_1LinearFormIntegrator.html), [`BilinearForm`](https://mfem.github.io/doxygen/html/classmfem_1_1BilinearForm.html) and [`MixedBilinearForm`](https://mfem.github.io/doxygen/html/classmfem_1_1MixedBilinearForm.html)
+  + [`FiniteElement`](http://mfem.github.io/doxygen/html/classmfem_1_1FiniteElement.html)
+  + [`FiniteElementCollection`](http://mfem.github.io/doxygen/html/classmfem_1_1FiniteElement.html)
+  + [`FiniteElementSpace`](http://mfem.github.io/doxygen/html/classmfem_1_1FiniteElementSpace.html)
+  + [`GridFunction`](http://mfem.github.io/doxygen/html/classmfem_1_1GridFunction.html)
+  + [`BilinearFormIntegrator`](http://mfem.github.io/doxygen/html/classmfem_1_1BilinearFormIntegrator.html) and [`LinearFormIntegrator`](http://mfem.github.io/doxygen/html/classmfem_1_1LinearFormIntegrator.html)
+  + [`LinearForm`](http://mfem.github.io/doxygen/html/classmfem_1_1LinearFormIntegrator.html), [`BilinearForm`](http://mfem.github.io/doxygen/html/classmfem_1_1BilinearForm.html) and [`MixedBilinearForm`](http://mfem.github.io/doxygen/html/classmfem_1_1MixedBilinearForm.html)
 
 - The main linear algebra classes and sources are
-  + [`Operator`](https://mfem.github.io/doxygen/html/classmfem_1_1Operator.html) and [`BilinearForm`](https://mfem.github.io/doxygen/html/classmfem_1_1BilinearForm.html)
-  + [`Vector`](https://mfem.github.io/doxygen/html/classmfem_1_1BilinearForm.html) and [`LinearForm`](https://mfem.github.io/doxygen/html/classmfem_1_1LinearForm.html)
-  + [`DenseMatrix`](https://mfem.github.io/doxygen/html/classmfem_1_1DenseMatrix.html) and [`SparseMatrix`](https://mfem.github.io/doxygen/html/classmfem_1_1SparseMatrix.html)
-  + Sparse [smoothers](https://mfem.github.io/doxygen/html/sparsesmoothers_8hpp.html) and linear [solvers](https://mfem.github.io/doxygen/html/solvers_8hpp.html)
+  + [`Operator`](http://mfem.github.io/doxygen/html/classmfem_1_1Operator.html) and [`BilinearForm`](http://mfem.github.io/doxygen/html/classmfem_1_1BilinearForm.html)
+  + [`Vector`](http://mfem.github.io/doxygen/html/classmfem_1_1BilinearForm.html) and [`LinearForm`](http://mfem.github.io/doxygen/html/classmfem_1_1LinearForm.html)
+  + [`DenseMatrix`](http://mfem.github.io/doxygen/html/classmfem_1_1DenseMatrix.html) and [`SparseMatrix`](http://mfem.github.io/doxygen/html/classmfem_1_1SparseMatrix.html)
+  + Sparse [smoothers](http://mfem.github.io/doxygen/html/sparsesmoothers_8hpp.html) and linear [solvers](http://mfem.github.io/doxygen/html/solvers_8hpp.html)
 
 #### Parallel implementation
 
@@ -162,13 +165,13 @@ shared geometric entities between different tasks. The parallel source files
 have a `p` prefix, e.g. `pmesh.cpp` vs. the serial `mesh.cpp`.
 
 - The main parallel classes are
-  + [`ParMesh`](https://mfem.github.io/doxygen/html/solvers_8hpp.html)
-  + [`ParNCMesh`](https://mfem.github.io/doxygen/html/classmfem_1_1ParMesh.html)
-  + [`ParFiniteElementSpace`](https://mfem.github.io/doxygen/html/classmfem_1_1ParFiniteElementSpace.html)
-  + [`ParGridFunction`](https://mfem.github.io/doxygen/html/classmfem_1_1ParGridFunction.html)
-  + [`ParBilinearForm`](https://mfem.github.io/doxygen/html/classmfem_1_1ParBilinearForm.html) and [`ParLinearForm`](https://mfem.github.io/doxygen/html/classmfem_1_1ParLinearForm.html)
-  + [`HypreParMatrix`](https://mfem.github.io/doxygen/html/classmfem_1_1HypreParMatrix.html) and [`HypreParVector`](https://mfem.github.io/doxygen/html/classmfem_1_1HypreParVector.html)
-  + [`HypreSolver`](https://mfem.github.io/doxygen/html/classmfem_1_1HypreSolver.html) and other [hypre classes](https://mfem.github.io/doxygen/html/hypre_8hpp.html)
+  + [`ParMesh`](http://mfem.github.io/doxygen/html/solvers_8hpp.html)
+  + [`ParNCMesh`](http://mfem.github.io/doxygen/html/classmfem_1_1ParMesh.html)
+  + [`ParFiniteElementSpace`](http://mfem.github.io/doxygen/html/classmfem_1_1ParFiniteElementSpace.html)
+  + [`ParGridFunction`](http://mfem.github.io/doxygen/html/classmfem_1_1ParGridFunction.html)
+  + [`ParBilinearForm`](http://mfem.github.io/doxygen/html/classmfem_1_1ParBilinearForm.html) and [`ParLinearForm`](http://mfem.github.io/doxygen/html/classmfem_1_1ParLinearForm.html)
+  + [`HypreParMatrix`](http://mfem.github.io/doxygen/html/classmfem_1_1HypreParMatrix.html) and [`HypreParVector`](http://mfem.github.io/doxygen/html/classmfem_1_1HypreParVector.html)
+  + [`HypreSolver`](http://mfem.github.io/doxygen/html/classmfem_1_1HypreSolver.html) and other [hypre classes](http://mfem.github.io/doxygen/html/hypre_8hpp.html)
 
 #### GPU and general device support
 
@@ -177,10 +180,10 @@ backends (CUDA, OCCA, RAJA, OpenMP, etc.) and an internal lightweight
 device/host memory manager.
 
 - The main device-relevant classes and sources are:
-  + [`Device`](https://mfem.github.io/doxygen/html/device_8hpp.html)
-  + [`MemoryManager`](https://mfem.github.io/doxygen/html/mem_manager_8hpp.html)
-  + the [`MFEM_FORALL`](https://mfem.github.io/doxygen/html/forall_8hpp.html) macro
-  + the [`cuda.hpp`](https://mfem.github.io/doxygen/html/cuda_8hpp.html) and [`occa.hpp`](https://mfem.github.io/doxygen/html/occa_8hpp.html) files
+  + [`Device`](http://mfem.github.io/doxygen/html/device_8hpp.html)
+  + [`MemoryManager`](http://mfem.github.io/doxygen/html/mem_manager_8hpp.html)
+  + the [`MFEM_FORALL`](http://mfem.github.io/doxygen/html/forall_8hpp.html) macro
+  + the [`cuda.hpp`](http://mfem.github.io/doxygen/html/cuda_8hpp.html) and [`occa.hpp`](http://mfem.github.io/doxygen/html/occa_8hpp.html) files
 
 #### Utilities, building and documentation
 - The `general/` directory contains C++ classes that serve as utilities for
@@ -189,7 +192,7 @@ device/host memory manager.
   Makefile and the CMake build options.
 - The `doc/` directory contains configuration for the Doxygen code documentation
   that can either be built locally or browsed online at
-  https://mfem.github.io/doxygen/html/index.html.
+  http://mfem.github.io/doxygen/html/index.html.
 
 #### Examples and tests
 - `examples` and `miniapps` respectively gather simple and more fully-featured
@@ -198,7 +201,7 @@ device/host memory manager.
 - The `tests/` directory contains a unit test suite and will later contain more
   tests that run example codes.
 
-See also the [code overview](https://mfem.org/code-overview/) section on the MFEM
+See also the [code overview](http://mfem.org/code-overview/) section on the MFEM
 website.
 
 ## GitHub Workflow
@@ -438,6 +441,7 @@ Before a PR can be merged, it should satisfy the following:
 - [ ] Run `make unittest` to make sure all unit tests pass.
 - [ ] Run the tests in `tests/scripts`.
 - [ ] (LLNL only) After merging:
+   - [ ] Regenerate `README.html` files from companion documentation pull requests.
    - [ ] Update internal tests to include the new features.
 
 ### Master/Next Workflow
@@ -510,10 +514,7 @@ MFEM uses a `master`/`next`-branch workflow as described below:
 - [ ] Check that continuous integration server configurations reflect the dependency version requirements of the new release
     - [ ] `.travis.yml`
     - [ ] `.appveyor.yml`
-- [ ] Update the `CHANGELOG` to organize all release contributions
-- [ ] Review the whole source code once over
-- [ ] Ask MFEM-based applications to test the pre-release branch
-- [ ] Test on additional platforms and compilers
+- [ ] (LLNL only) Make sure all `README.html` files in the source repo are up to date.
 - [ ] Tag the repository:
 
   ```
@@ -524,13 +525,14 @@ MFEM uses a `master`/`next`-branch workflow as described below:
 - [ ] Recreate the `next` branch as described in previous section.
 - [ ] Update and push documentation  to `mfem/doxygen`.
 - [ ] Update URL shortlinks:
-    - [ ] Create a shortlink at [http://bit.ly/](http://bit.ly/) for the release tarball, e.g. https://mfem.github.io/releases/mfem-3.1.tgz.
+    - [ ] Create a shortlink at [http://bit.ly/](http://bit.ly/) for the release tarball, e.g. http://mfem.github.io/releases/mfem-3.1.tgz.
     - [ ] (LLNL only) Add and commit the new shortlink in the `links` and `links-mfem` files of the internal `mfem/downloads` repo.
     - [ ] Add the new shortlinks to the MFEM packages in `spack`, `homebrew/science`, `VisIt`, etc.
 - [ ] Update website in `mfem/web` repo:
     - Update version and shortlinks in `src/index.md` and `src/download.md`.
     -  Use [cloc-1.62.pl](http://cloc.sourceforge.net/) and `ls -lh` to estimate the SLOC and the tarball size in `src/download.md`.
 
+
 ## LLNL Workflow
 
 ### Mirroring on Bitbucket

INSTALL

@@ -5,7 +5,7 @@
                       | | | | | ||  _||  __/| | | | | |
                       |_| |_| |_||_|   \___||_| |_| |_|
 
-                               https://mfem.org
+                               http://mfem.org
 
 The MFEM library has a serial and an MPI-based parallel version, which largely
 share the same code base. The only prerequisite for building the serial version
@@ -19,10 +19,10 @@ requires an MPI C++ compiler, as well as the following external libraries:
   http://glaros.dtc.umn.edu/gkhome/metis/metis/overview
 
 The hypre dependency can be downloaded as a tarball from GitHub or from the
-project webpage https://www.llnl.gov/casc/hypre. For example, the 2.20.0 release
+project webpage https://www.llnl.gov/casc/hypre. For example, the 2.16.0 release
 of hypre is available at
 
-   https://github.com/hypre-space/hypre/archive/v2.20.0.tar.gz
+   https://github.com/hypre-space/hypre/archive/v2.16.0.tar.gz
 
 The METIS dependency can be disabled but that is not generally recommended, see
 the option MFEM_USE_METIS.
@@ -62,7 +62,7 @@ following package managers:
 
 We also recommend downloading and building the MFEM-based GLVis visualization
 tool which can be used to visualize the meshes and solution in MFEM's examples
-and miniapps. See https://glvis.org and https://mfem.org/building.
+and miniapps. See http://glvis.org and http://mfem.org/building.
 
 Quick start with GNU make
 =========================
@@ -348,12 +348,6 @@ MFEM_USE_SUPERLU5 = YES/NO
    If SuperLU functionality is enabled, use the older 5.1.0 version rather than
    the more recent 6+ versions.
 
-MFEM_USE_MUMPS = YES/NO
-   Enable MFEM functionality based on the MUMPS library. Currently, this
-   option adds the class MUMPSSolver (a parallel sparse direct solver). 
-   When enabled, this option uses the MUMPS_* library options, see
-   below.
-   
 MFEM_USE_STRUMPACK = YES/NO
    Enable MFEM functionality based on the STRUMPACK sparse direct solver and
    preconditioner through the STRUMPACKSolver and STRUMPACKRowLocMatrix
@@ -564,12 +558,11 @@ The specific libraries and their options are:
   for CUDA.
   URL: http://computation.llnl.gov/projects/sundials/sundials-software
   Options: SUNDIALS_OPT, SUNDIALS_LIB.
-  Versions: SUNDIALS >= 5.0.0, SUNDIALS >= 5.4.0 for CUDA support.
+  Versions: SUNDIALS >= 5.0.0, SUNDIALS >= 5.4.0 for CUDA suppport.
 
 - Mesquite (optional), used when MFEM_USE_MESQUITE = YES.
   URL: http://trilinos.org/oldsite/packages/mesquite
   Options: MESQUITE_OPT, MESQUITE_LIB.
-  The Mesquite support is deprecated and will be removed in the future.
 
 - SuiteSparse (optional), used when MFEM_USE_SUITESPARSE = YES.
   URL: http://faculty.cse.tamu.edu/davis/suitesparse.html
@@ -579,17 +572,11 @@ The specific libraries and their options are:
 - SuperLU_DIST (optional), used when MFEM_USE_SUPERLU = YES. Note that
   SuperLU_DIST requires ParMETIS, which includes METIS 5 in its distribution.
   Both ParMETIS and the included METIS 5 should be built and installed in the
-  same location. If using SuperLU_Dist v5, set MFEM_USE_SUPERLU5=YES.
+  same location.
   URL: http://crd-legacy.lbl.gov/~xiaoye/SuperLU
   Options: SUPERLU_OPT, SUPERLU_LIB.
   Versions: SuperLU_DIST >= 5.1.0.
 
-- MUMPS (optional), used when MFEM_USE_MUMPS = YES. Note that MUMPS
-  requires LAPACK, SCALAPACK and a reordering package such as PORD or METIS.
-  URL: http://mumps.enseeiht.fr
-  Options: MUMPS_OPT, MUMPS_LIB.
-  Versions: MUMPS >= 5.1.1
-
 - STRUMPACK (optional), used when MFEM_USE_STRUMPACK = YES. Note that STRUMPACK
   requires the PT-Scotch and Scalapack libraries as well as ParMETIS, which
   includes METIS 5 in its distribution. Starting with STRUMPACK v2.2.0, ParMETIS
@@ -598,7 +585,7 @@ The specific libraries and their options are:
   2.0.0 or later.
   URL: http://portal.nersc.gov/project/sparse/strumpack
   Options: STRUMPACK_OPT, STRUMPACK_LIB.
-  Versions: STRUMPACK >= 3.0.0.
+  Versions: STRUMPACK >= 3.0.0, requires HYPRE < 2.16.0.
 
 - Ginkgo (optional), used when MFEM_USE_GINKGO = YES. Note that Ginkgo needs a
   C++ compiler that supports the C++-11 standard. For additional requirements
@@ -661,18 +648,16 @@ The specific libraries and their options are:
 
 - ADIOS2 (optional) used when MFEM_USE_ADIOS2 = YES.
   URL: https://adios2.readthedocs.io/
-  Versions: ADIOS >= 2.5.0.
 
 - PUMI (optional), used when MFEM_USE_PUMI = YES.
   URL: https://scorec.rpi.edu/pumi
        https://github.com/SCOREC/core
   Options: PUMI_OPT, PUMI_LIB.
-  Versions: PUMI == 2.2.3.
+  Versions: PUMI >= 2.2.3.
 
 - HiOp (optional), used when MFEM_USE_HIOP = YES.
   URL: https://github.com/LLNL/hiop
   Options: HIOP_OPT, HIOP_LIB.
-  Versions: HIOP >= 0.1.
 
 - GSLIB (optional), used when MFEM_USE_GSLIB = YES. The gslib library must be
   built prior to the MFEM build, as follows: download gslib-1.0.5, untar it at
@@ -849,7 +834,6 @@ MFEM_TIMER_TYPE - Set automatically, can be overwritten.
 MFEM_USE_MESQUITE
 MFEM_USE_SUITESPARSE
 MFEM_USE_SUPERLU
-MFEM_USE_MUMPS
 MFEM_USE_STRUMPACK
 MFEM_USE_GINKGO
 MFEM_USE_AMGX

README

@@ -5,7 +5,7 @@
                       | | | | | ||  _||  __/| | | | | |
                       |_| |_| |_||_|   \___||_| |_| |_|
 
-                               https://mfem.org
+                               http://mfem.org
 
 MFEM is a modular parallel C++ library for finite element methods. Its goal is
 to enable high-performance scalable finite element discretization research and
@@ -16,8 +16,8 @@ supercomputers.
 
 * Copyright and licensing information can be found in files LICENSE and NOTICE.
 
-* The best starting point for new users interested in MFEM's features is to
-  review the examples and miniapps at https://mfem.org/examples.
+* The best starting point for new users interested in MFEM's features is the
+  interactive documentation in examples/README.html.
 
 * Developers interested in contributing to the library, should read the
   instructions and documentation in the CONTRIBUTING.md file.
@@ -49,17 +49,17 @@ library. Nonlinear solvers (the Newton method), eigensolvers (LOBPCG), and
 several explicit and implicit Runge-Kutta time integrators are also available.
 
 MFEM supports MPI-based parallelism throughout the library, and can readily be
-used as a scalable unstructured finite element problem generator. Starting with
-version 4.0, MFEM offers support for GPU acceleration, and programming models,
-such as CUDA, HIP, OCCA, RAJA and OpenMP. MFEM-based applications require
-minimal changes to switch from a serial to a highly-performant MPI-parallel
-version of the code, where they can take advantage of the integrated linear
-solvers from the hypre library. Comprehensive support for other external
-packages, e.g. PETSc, SUNDIALS and libCEED is also included, giving access to
-additional linear and nonlinear solvers, preconditioners, time integrators, etc.
+used as a scalable unstructured finite element problem generator. As of version
+4.0, MFEM offers initial support for GPU acceleration, and programming models,
+such as CUDA, OCCA, RAJA and OpenMP. MFEM-based applications require minimal
+changes to switch from a serial to a high-performing MPI-parallel version of the
+code, where they can take advantage of the integrated linear solvers from the
+hypre library.  Comprehensive support for other external packages, e.g. PETSc
+and SUNDIALS is also included, giving access to many additional linear and
+nonlinear solvers, preconditioners, time integrators, etc.
 
 For examples of using MFEM, see the examples/ and miniapps/ directories, as well
-as the OpenGL visualization tool GLVis which is available at https://glvis.org.
+as the OpenGL visualization tool GLVis which is available at http://glvis.org.
 
 MFEM is distributed under the terms of the BSD-3 license. All new contributions
 must be made under this license. See LICENSE and NOTICE for details.

config/cmake/MFEMConfig.cmake.in

@@ -32,11 +32,9 @@ set(MFEM_USE_SUNDIALS @MFEM_USE_SUNDIALS@)
 set(MFEM_USE_MESQUITE @MFEM_USE_MESQUITE@)
 set(MFEM_USE_SUITESPARSE @MFEM_USE_SUITESPARSE@)
 set(MFEM_USE_SUPERLU @MFEM_USE_SUPERLU@)
-set(MFEM_USE_MUMPS @MFEM_USE_MUMPS@)
 set(MFEM_USE_STRUMPACK @MFEM_USE_STRUMPACK@)
 set(MFEM_USE_GINKGO @MFEM_USE_GINKGO@)
 set(MFEM_USE_AMGX @MFEM_USE_AMGX@)
-set(MFEM_USE_HIOP @MFEM_USE_HIOP@)
 set(MFEM_USE_GNUTLS @MFEM_USE_GNUTLS@)
 set(MFEM_USE_GSLIB @MFEM_USE_GSLIB@)
 set(MFEM_USE_NETCDF @MFEM_USE_NETCDF@)

config/cmake/config.hpp.in

@@ -86,9 +86,6 @@
 // Enable MFEM functionality based on the SuperLU_DIST library.
 #cmakedefine MFEM_USE_SUPERLU
 
-// Enable MFEM functionality based on the MUMPS library.
-#cmakedefine MFEM_USE_MUMPS
-
 // Enable MFEM functionality based on the STRUMPACK library.
 #cmakedefine MFEM_USE_STRUMPACK
 

config/cmake/modules/FindADIOS2.cmake

@@ -38,19 +38,19 @@ if(NOT ADIOS2_FOUND)
   endif()
 
   find_path(ADIOS2_INCLUDE_DIR adios2.h ${ADIOS2_INCLUDE_OPTS})
-
-  # adios2 version 2.5.0
+  
+  # adios2 version 2.5.0 
   find_library(ADIOS2_LIBRARY NAMES adios2 ${ADIOS2_LIBRARY_OPTS})
-
+  
   # adios2 version 2.6.0 and onwards
   if(NOT ADIOS2_LIBRARY)
-	find_library(ADIOS2_CXX11_MPI_LIBRARY NAMES adios2_cxx11_mpi ${ADIOS2_LIBRARY_OPTS})
-	find_library(ADIOS2_CXX11_LIBRARY NAMES adios2_cxx11 ${ADIOS2_LIBRARY_OPTS})
-	set(ADIOS2_LIBRARY ${ADIOS2_CXX11_MPI_LIBRARY} ${ADIOS2_CXX11_LIBRARY})
-	if(MFEM_USE_MPI)
-	  add_definitions(-DADIOS2_USE_MPI)
-	endif()
-  endif()
+  	find_library(ADIOS2_CXX11_MPI_LIBRARY NAMES adios2_cxx11_mpi ${ADIOS2_LIBRARY_OPTS})
+  	find_library(ADIOS2_CXX11_LIBRARY NAMES adios2_cxx11 ${ADIOS2_LIBRARY_OPTS})
+  	set(ADIOS2_LIBRARY ${ADIOS2_CXX11_MPI_LIBRARY} ${ADIOS2_CXX11_LIBRARY})
+  	if(MFEM_USE_MPI)
+  	  add_definitions(-DADIOS2_USE_MPI)
+  	endif()
+  endif() 
 
   include(FindPackageHandleStandardArgs)
   find_package_handle_standard_args(ADIOS2

config/cmake/modules/FindMUMPS.cmake

@@ -1,23 +0,0 @@
-# Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-# at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-# LICENSE and NOTICE for details. LLNL-CODE-806117.
-#
-# This file is part of the MFEM library. For more information and source code
-# availability visit https://mfem.org.
-#
-# MFEM is free software; you can redistribute it and/or modify it under the
-# terms of the BSD-3 license. We welcome feedback and contributions, see file
-# CONTRIBUTING.md for details.
-
-# Sets the following variables:
-#   - MUMPS_FOUND
-#   - MUMPS_INCLUDE_DIRS
-#   - MUMPS_LIBRARIES
-
-include(MfemCmakeUtilities)
-mfem_find_package(MUMPS MUMPS MUMPS_DIR
-  "include" dmumps_c.h "lib" dmumps
-  "Paths to headers required by MUMPS."
-  "Libraries required by MUMPS."
-  ADD_COMPONENT mumps_common "include" dmumps_c.h "lib" mumps_common
-  ADD_COMPONENT pord "include" dmumps_c.h "lib" pord)

config/cmake/modules/MfemCmakeUtilities.cmake

@@ -235,7 +235,7 @@ endfunction(mfem_find_component)
 #   code snippets. Additionally, a list of required/optional/alternative
 #   packages (given by ${Name}_REQUIRED_PACKAGES) are searched for and added to
 #   the ${Prefix}_INCLUDE_DIRS and ${Prefix}_LIBRARIES lists. The variable
-#   ${Name}_REQUIRED_LIBRARIES can be set to specify any additional libraries
+#   ${Name}_REQUIRED_LIBRARIES can be set to spcecify any additional libraries
 #   that are needed. This function defines the following CACHE variables:
 #
 #      ${Prefix}_FOUND
@@ -733,7 +733,7 @@ function(mfem_export_mk_files)
     set(shared_link_flag "-Wl,-rpath,")
   endif()
 
-  # Convert Boolean vars to YES/NO without writing the values to cache
+  # Convert Boolean vars to YES/NO without writting the values to cache
   set(CONFIG_MK_BOOL_VARS MFEM_USE_MPI MFEM_USE_METIS MFEM_USE_METIS_5
       MFEM_DEBUG MFEM_USE_EXCEPTIONS MFEM_USE_ZLIB MFEM_USE_LIBUNWIND
       MFEM_USE_LAPACK MFEM_THREAD_SAFE MFEM_USE_OPENMP MFEM_USE_LEGACY_OPENMP
@@ -820,7 +820,7 @@ function(mfem_export_mk_files)
       string(REGEX REPLACE "^SCOREC::" "" libname ${pumilib})
       string(FIND "${pumilib}" ".a" staticlib)
       string(FIND "${pumilib}" ".so" sharedlib)
-      find_library(lib ${libname} PATHS ${PUMI_DIR}/lib NO_DEFAULT_PATH)
+      find_library(lib ${libname} PATHS ${PUMI_DIR}/lib NO_DEFUALT_PATH)
       if (NOT "${sharedlib}" MATCHES "-1" OR
           NOT "${staticlib}" MATCHES "-1"   )
         set(MFEM_EXT_LIBS "${pumilib} ${MFEM_EXT_LIBS}")

config/config.hpp

@@ -42,9 +42,6 @@
 #ifdef MFEM_USE_SUPERLU
 #error Building with SuperLU_DIST (MFEM_USE_SUPERLU=YES) requires MPI (MFEM_USE_MPI=YES)
 #endif
-#ifdef MFEM_USE_MUMPS
-#error Building with MUMPS (MFEM_USE_MUMPS=YES) requires MPI (MFEM_USE_MPI=YES)
-#endif
 #ifdef MFEM_USE_STRUMPACK
 #error Building with STRUMPACK (MFEM_USE_STRUMPACK=YES) requires MPI (MFEM_USE_MPI=YES)
 #endif

config/config.hpp.in

@@ -95,10 +95,6 @@
 // #define MFEM_USE_SUPERLU
 // #define MFEM_USE_SUPERLU5
 
-// Enable MFEM functionality based on the MUMPS library.
-// #define MFEM_USE_MUMPS
-// #define MFEM_MUMPS_VERSION @MFEM_MUMPS_VERSION@
-
 // Enable MFEM functionality based on the STRUMPACK library.
 // #define MFEM_USE_STRUMPACK
 

config/config.mk.in

@@ -33,7 +33,6 @@ MFEM_USE_MESQUITE      = @MFEM_USE_MESQUITE@
 MFEM_USE_SUITESPARSE   = @MFEM_USE_SUITESPARSE@
 MFEM_USE_SUPERLU       = @MFEM_USE_SUPERLU@
 MFEM_USE_SUPERLU5      = @MFEM_USE_SUPERLU5@
-MFEM_USE_MUMPS         = @MFEM_USE_MUMPS@
 MFEM_USE_STRUMPACK     = @MFEM_USE_STRUMPACK@
 MFEM_USE_GINKGO        = @MFEM_USE_GINKGO@
 MFEM_USE_AMGX          = @MFEM_USE_AMGX@

config/defaults.cmake

@@ -34,7 +34,6 @@ option(MFEM_USE_MESQUITE "Enable MESQUITE usage" OFF)
 option(MFEM_USE_SUITESPARSE "Enable SuiteSparse usage" OFF)
 option(MFEM_USE_SUPERLU "Enable SuperLU_DIST usage" OFF)
 option(MFEM_USE_SUPERLU5 "Use the old SuperLU_DIST 5.1 version" OFF)
-option(MFEM_USE_MUMPS "Enable MUMPS usage" OFF)
 option(MFEM_USE_STRUMPACK "Enable STRUMPACK usage" OFF)
 option(MFEM_USE_GINKGO "Enable Ginkgo usage" OFF)
 option(MFEM_USE_AMGX "Enable AmgX usage" OFF)
@@ -119,15 +118,6 @@ set(SuperLUDist_DIR "${MFEM_DIR}/../SuperLU_DIST_6.3.1" CACHE PATH
 set(SuperLUDist_REQUIRED_PACKAGES "MPI" "BLAS" "ParMETIS" CACHE STRING
     "Additional packages required by SuperLU_DIST.")
 
-set(MUMPS_DIR "${MFEM_DIR}/../MUMPS_5.2.0" CACHE PATH
-    "Path to the MUMPS library.")
-# Packages required by MUMPS, depending on how it was compiled.
-set(MUMPS_REQUIRED_PACKAGES "MPI" "BLAS" "METIS" "ScaLAPACK" CACHE STRING
-    "Additional packages required by MUMPS.")    
-# If the MPI package does not find all required Fortran libraries:
-# set(MUMPS_REQUIRED_LIBRARIES "gfortran" "mpi_mpifh" CACHE STRING
-#     "Additional libraries required by MUMPS.")    
-
 set(STRUMPACK_DIR "${MFEM_DIR}/../STRUMPACK-build" CACHE PATH
     "Path to the STRUMPACK library.")
 # STRUMPACK may also depend on "OpenMP", depending on how it was compiled.

config/defaults.mk

@@ -53,8 +53,6 @@ HIP_CXX = hipcc
 # gfx900, gfx1010, etc.
 HIP_ARCH = gfx900
 HIP_FLAGS = --amdgpu-target=$(HIP_ARCH)
-HIP_XCOMPILER =
-HIP_XLINKER   = -Wl,
 
 ifneq ($(NOTMAC),)
    AR      = ar
@@ -123,7 +121,6 @@ MFEM_USE_MESQUITE      = NO
 MFEM_USE_SUITESPARSE   = NO
 MFEM_USE_SUPERLU       = NO
 MFEM_USE_SUPERLU5      = NO
-MFEM_USE_MUMPS         = NO
 MFEM_USE_STRUMPACK     = NO
 MFEM_USE_GINKGO        = NO
 MFEM_USE_AMGX          = NO
@@ -147,16 +144,6 @@ MFEM_USE_SIMD          = NO
 MFEM_USE_ADIOS2        = NO
 MFEM_USE_MKL_CPARDISO  = NO
 
-# MPI library compile and link flags
-# These settings are used only when building MFEM with MPI + HIP
-ifeq ($(MFEM_USE_MPI)$(MFEM_USE_HIP),YESYES)
-   # We determine MPI_DIR assuming $(MPICXX) is in $(MPI_DIR)/bin
-   MPI_DIR := $(patsubst %/,%,$(dir $(shell which $(MPICXX))))
-   MPI_DIR := $(patsubst %/,%,$(dir $(MPI_DIR)))
-   MPI_OPT = -I$(MPI_DIR)/include
-   MPI_LIB = -L$(MPI_DIR)/lib $(XLINKER)-rpath,$(MPI_DIR)/lib -lmpi
-endif
-
 # Compile and link options for zlib.
 ZLIB_DIR =
 ZLIB_OPT = $(if $(ZLIB_DIR),-I$(ZLIB_DIR)/include)
@@ -172,7 +159,7 @@ HYPRE_OPT = -I$(HYPRE_DIR)/include
 HYPRE_LIB = -L$(HYPRE_DIR)/lib -lHYPRE
 
 # METIS library configuration
-ifeq ($(MFEM_USE_SUPERLU)$(MFEM_USE_STRUMPACK)$(MFEM_USE_MUMPS),NONONO)
+ifeq ($(MFEM_USE_SUPERLU)$(MFEM_USE_STRUMPACK),NONO)
    ifeq ($(MFEM_USE_METIS_5),NO)
      METIS_DIR = @MFEM_DIR@/../metis-4.0
      METIS_OPT =
@@ -239,7 +226,7 @@ SUITESPARSE_LIB = -Wl,-rpath,$(SUITESPARSE_DIR)/lib -L$(SUITESPARSE_DIR)/lib\
 ifeq ($(MFEM_USE_SUPERLU5),YES)
    SUPERLU_DIR = @MFEM_DIR@/../SuperLU_DIST_5.1.0
    SUPERLU_OPT = -I$(SUPERLU_DIR)/include
-   SUPERLU_LIB = -Wl,-rpath,$(SUPERLU_DIR)/lib -L$(SUPERLU_DIR)/lib -lsuperlu_dist_5.1.0
+   SUPERLU_LIB = -Wl,-rpath,$(SUPERLU_DIR)/lib -L$(SUPERLU_DIR)/lib -lsuperlu_dist_5.1.0   
 else
    SUPERLU_DIR = @MFEM_DIR@/../SuperLU_DIST_6.3.1
    SUPERLU_OPT = -I$(SUPERLU_DIR)/include
@@ -253,12 +240,12 @@ SCOTCH_OPT = -I$(SCOTCH_DIR)/include
 SCOTCH_LIB = -L$(SCOTCH_DIR)/lib -lptscotch -lptscotcherr -lscotch -lscotcherr\
  -lpthread
 
-# SCALAPACK library configuration (required by STRUMPACK and MUMPS)
+# SCALAPACK library configuration (required by STRUMPACK)
 SCALAPACK_DIR = @MFEM_DIR@/../scalapack-2.0.2
 SCALAPACK_OPT = -I$(SCALAPACK_DIR)/SRC
 SCALAPACK_LIB = -L$(SCALAPACK_DIR)/lib -lscalapack $(LAPACK_LIB)
 
-# MPI Fortran library, needed e.g. by STRUMPACK or MUMPS
+# MPI Fortran library, needed e.g. by STRUMPACK
 # MPICH:
 MPI_FORTRAN_LIB = -lmpifort
 # OpenMPI:
@@ -266,12 +253,6 @@ MPI_FORTRAN_LIB = -lmpifort
 # Additional Fortan library:
 # MPI_FORTRAN_LIB += -lgfortran
 
-# MUMPS library configuration
-MUMPS_DIR = @MFEM_DIR@/../MUMPS_5.2.0
-MUMPS_OPT = -I$(MUMPS_DIR)/include
-MUMPS_LIB = -Wl,-rpath,$(MUMPS_DIR)/lib -L$(MUMPS_DIR)/lib -ldmumps\
- -lmumps_common -lpord $(SCALAPACK_LIB) $(LAPACK_LIB) $(MPI_FORTRAN_LIB)
-
 # STRUMPACK library configuration
 STRUMPACK_DIR = @MFEM_DIR@/../STRUMPACK-build
 STRUMPACK_OPT = -I$(STRUMPACK_DIR)/include $(SCOTCH_OPT)

config/get_mumps_version.cpp

@@ -1,33 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-
-#include "dmumps_c.h"
-#include <string>
-#include <iostream>
-#include <algorithm>
-
-// Macros to expand a macro as a string
-#define STR_EXPAND(s) #s
-#define STR(s) STR_EXPAND(s)
-
-int main()
-{
-#ifdef MUMPS_VERSION
-   const char *ptr = STR(MUMPS_VERSION);
-   std::string s(ptr);
-   s.erase(std::remove(s.begin(), s.end(), '"'), s.end());
-   s.erase(std::remove(s.begin(), s.end(), '.'), s.end());
-   std::cout << s << "\n";
-   return 0;
-#else
-   return -1;
-#endif
-}

config/makefile

@@ -42,10 +42,6 @@ GHV_FLAGS = $(subst @MFEM_DIR@,$(if $(MFEM_DIR),$(MFEM_DIR),..),$(HYPRE_OPT))
 SMX = $(if $(MFEM_USE_PUMI:NO=),MFEM_USE_SIMMETRIX)
 SMX_PATH = $(PUMI_DIR)/include/gmi_sim.h
 SMX_FILE = $(subst @MFEM_DIR@,$(if $(MFEM_DIR),$(MFEM_DIR),..),$(SMX_PATH))
-MUMPS = $(MFEM_USE_MUMPS:NO=)
-GMV_CXX ?= $(MFEM_CXX)
-GMV = get_mumps_version
-GMV_FLAGS = $(subst @MFEM_DIR@,$(if $(MFEM_DIR),$(MFEM_DIR),..),$(MUMPS_OPT))
 
 $(GHV): $(SRC)$(GHV).cpp
 	$(call mfem-info, Determining HYPRE version ...)
@@ -54,13 +50,6 @@ $(GHV).out: $(GHV)
 	./$(GHV) > $(GHV).out
 .INTERMEDIATE: $(GHV) $(GHV).out
 
-$(GMV): $(SRC)$(GMV).cpp
-	$(call mfem-info, Determining MUMPS version ...)
-	$(GMV_CXX) ${GMV_FLAGS} $(SRC)$(GMV).cpp -o $(GMV)
-$(GMV).out: $(GMV)
-	./$(GMV) > $(GMV).out
-.INTERMEDIATE: $(GMV) $(GMV).out
-
 get-hypre-version: $(GHV).out
 	$(eval MFEM_HYPRE_VERSION:=$(shell cat $(GHV).out))
 	$(if $(MFEM_HYPRE_VERSION),$(eval export MFEM_HYPRE_VERSION)\
@@ -73,18 +62,10 @@ check-smx:
 	$(call mfem-info, MFEM_USE_SIMMETRIX = $(MFEM_USE_SIMMETRIX))
 	$(eval export MFEM_USE_SIMMETRIX)
 
-get-mumps-version: $(GMV).out
-	$(eval MFEM_MUMPS_VERSION:=$(shell cat $(GMV).out))
-	$(if $(MFEM_MUMPS_VERSION),$(eval export MFEM_MUMPS_VERSION)\
-	$(info MUMPS version: $(MFEM_MUMPS_VERSION)),\
-	$(error Unable to determine MUMPS version))
-
-header: $(if $(MPI),get-hypre-version,) $(if $(SMX),check-smx,) \
- $(if $(MUMPS),get-mumps-version,)
+header: $(if $(MPI),get-hypre-version,) $(if $(SMX),check-smx)
 	$(call mfem-info, Writing $(CONFIG_HPP) ...)
 	@set -- && \
-	for def in $${MFEM_DEFINES} $(if $(MPI),MFEM_HYPRE_VERSION) $(SMX) \
-	           $(if $(MUMPS),MFEM_MUMPS_VERSION); do \
+	for def in $${MFEM_DEFINES} $(if $(MPI),MFEM_HYPRE_VERSION) $(SMX); do \
 	   eval var=\$$$$def && \
 	   if [ "NO" != "$${var}" ]; then \
 	      set -- "$$@" -e "s|// \(#define $${def} \)|\1|" && \

config/sample-runs.sh

@@ -48,10 +48,6 @@ groups_serial=(
    "miniapps/meshing"
    "mobius-strip.cpp klein-bottle.cpp extruder.cpp toroid.cpp
     mesh-optimizer.cpp minimal-surface.cpp"'
-'"convergence"
-   "Convergence tests:"
-   "tests/convergence"
-   "rates.cpp"'
 )
 # Parallel groups
 groups_parallel=(
@@ -85,7 +81,7 @@ groups_parallel=(
 '"convergence"
    "Convergence tests:"
    "tests/convergence"
-   "prates.cpp"'
+   "diffusion.cpp"'
 '"par-mesh-format"
    "Parallel mesh tests:"
    "tests/par-mesh-format"
@@ -122,7 +118,7 @@ groups_all=(
 '"convergence"
    "Convergence tests:"
    "tests/convergence"
-   "{,p}rates.cpp"'
+   "diffusion.cpp"'
 '"par-mesh-format"
    "Parallel mesh tests:"
    "tests/par-mesh-format"

doc/CMakeLists.txt

@@ -16,6 +16,7 @@ if (DOXYGEN_FOUND)
   configure_file(${CMAKE_CURRENT_SOURCE_DIR}/CodeDocumentation.conf.in
     ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.conf @ONLY)
 
+
   add_custom_target(doc
     COMMAND ${DOXYGEN_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.conf
     COMMAND echo "<meta http-equiv=\"REFRESH\" content=\"0;URL=CodeDocumentation/html/index.html\">" > ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.html
@@ -31,4 +32,5 @@ if (DOXYGEN_FOUND)
     COMMENT "Removing API documentation"
     VERBATIM)
 
+
 endif (DOXYGEN_FOUND)

doc/CodeDocumentation.conf.in

@@ -38,7 +38,7 @@ PROJECT_NAME           = "MFEM"
 # could be handy for archiving the generated documentation or if some version
 # control system is used.
 
-PROJECT_NUMBER         = v4.2.1
+PROJECT_NUMBER         = v4.1.1
 
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a
@@ -51,7 +51,7 @@ PROJECT_BRIEF          = "Finite element discretization library"
 # pixels and the maximum width should not exceed 200 pixels. Doxygen will copy
 # the logo to the output directory.
 
-PROJECT_LOGO           = logo-small.png
+PROJECT_LOGO           = web/logo-small.png
 
 # The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path
 # into which the generated documentation will be written. If a relative path is
@@ -766,14 +766,11 @@ INPUT                  = @MFEM_SOURCE_DIR@/doc/CodeDocumentation.dox \
                          @MFEM_SOURCE_DIR@/mesh \
                          @MFEM_SOURCE_DIR@/fem \
                          @MFEM_SOURCE_DIR@/examples \
-                         @MFEM_SOURCE_DIR@/examples/amgx \
-                         @MFEM_SOURCE_DIR@/examples/ginkgo \
-                         @MFEM_SOURCE_DIR@/examples/hiop \
                          @MFEM_SOURCE_DIR@/examples/petsc \
                          @MFEM_SOURCE_DIR@/examples/pumi \
+                         @MFEM_SOURCE_DIR@/examples/hiop \
                          @MFEM_SOURCE_DIR@/examples/sundials \
-                         @MFEM_SOURCE_DIR@/examples/superlu \
-                         @MFEM_SOURCE_DIR@/miniapps/adjoint \
+                         @MFEM_SOURCE_DIR@/miniapps/adjoint \			 
                          @MFEM_SOURCE_DIR@/miniapps/common \
                          @MFEM_SOURCE_DIR@/miniapps/electromagnetics \
                          @MFEM_SOURCE_DIR@/miniapps/gslib \
@@ -2337,7 +2334,7 @@ PLANTUML_INCLUDE_PATH  =
 # Minimum value: 0, maximum value: 10000, default value: 50.
 # This tag requires that the tag HAVE_DOT is set to YES.
 
-DOT_GRAPH_MAX_NODES    = 50
+DOT_GRAPH_MAX_NODES    = 10
 
 # The MAX_DOT_GRAPH_DEPTH tag can be used to set the maximum depth of the graphs
 # generated by dot. A depth value of 3 means that only nodes reachable from the

doc/CodeDocumentation.dox

@@ -62,7 +62,7 @@ namespace mfem {
  * - <a class="el" href="ex9p_8cpp_source.html">Example 9p</a>: parallel Discontinuous Galerkin (DG) time-dependent advection
  * - <a class="el" href="ex10_8cpp_source.html">Example 10</a>: time-dependent implicit nonlinear elasticity
  * - <a class="el" href="ex10p_8cpp_source.html">Example 10p</a>: parallel time-dependent implicit nonlinear elasticity
- * - <a class="el" href="ex11p_8cpp_source.html">Example 11p</a>: parallel Laplace eigensolver
+ * - <a class="el" href="examples_2ex11p_8cpp_source.html">Example 11p</a>: parallel Laplace eigensolver
  * - <a class="el" href="ex12p_8cpp_source.html">Example 12p</a>: parallel linear elasticity eigensolver
  * - <a class="el" href="ex13p_8cpp_source.html">Example 13p</a>: parallel Maxwell eigensolver
  * - <a class="el" href="ex14_8cpp_source.html">Example 14</a>: Discontinuous Galerkin (DG) for the Laplace problem
@@ -90,25 +90,20 @@ namespace mfem {
  * - <a class="el" href="ex25p_8cpp_source.html">Example 25p</a>: parallel simulation of electromagnetic wave propagation using a Perfectly Matched Layer (PML)
  * - <a class="el" href="ex26_8cpp_source.html">Example 26</a>: multigrid preconditioner for the Laplace problem using nodal H1 FEM
  * - <a class="el" href="ex26p_8cpp_source.html">Example 26p</a>: parallel multigrid preconditioner for the Laplace problem using nodal H1 FEM
- * - <a class="el" href="ex27_8cpp_source.html">Example 27</a>: boundary conditions for the Laplace problem
- * - <a class="el" href="ex27p_8cpp_source.html">Example 27p</a>: parallel boundary conditions for the Laplace problem
  *
- * <H4>AmgX Examples</H4>
+ * <H4>SUNDIALS Examples</H4>
  * - Variants of Examples
- *   <a class="el" href="examples_2amgx_2ex1_8cpp_source.html">1</a> and
- *   <a class="el" href="examples_2amgx_2ex1p_8cpp_source.html">1p</a>,
- *   demonstrating the use of MFEM's \link amgxsolver.hpp AmgX integration\endlink.
- *
- * <H4>Ginkgo Examples</H4>
- * - Variants of Example
- *   <a class="el" href="examples_2ginkgo_2ex1_8cpp_source.html">1</a>,
- *   demonstrating the use of MFEM's \link ginkgo.hpp Ginkgo integration\endlink.
- *
- * <H4>HiOp Examples</H4>
- * - Variants of Examples
- *   <a class="el" href="hiop_2ex9_8cpp_source.html">9</a> and
- *   <a class="el" href="hiop_2ex9p_8cpp_source.html">9p</a>,
- *   demonstrating the use of MFEM's \link hiop.hpp HiOp integration\endlink.
+ *   <a class="el" href="sundials_2ex9_8cpp_source.html">9</a>,
+ *   <a class="el" href="sundials_2ex9p_8cpp_source.html">9p</a>,
+ *   <a class="el" href="sundials_2ex10_8cpp_source.html">10</a>,
+ *   <a class="el" href="sundials_2ex10p_8cpp_source.html">10p</a>,
+ *   <a class="el" href="sundials_2ex16_8cpp_source.html">16</a>,
+ *   and
+ *   <a class="el" href="sundials_2ex16p_8cpp_source.html">16p</a>
+ *   demonstrating the use of MFEM's \link sundials.hpp SUNDIALS classes\endlink
+ * - CVODES adjoint examples:
+ *   <a class="el" href="cvsRoberts__ASAi__dns_8cpp_source.html">serial ODE system</a>,
+ *   <a class="el" href="adjoint__advection__diffusion_8cpp_source.html">parallel advection-diffusion</a>
  *
  * <H4>PETSc Examples</H4>
  * - Variants of Examples
@@ -120,8 +115,8 @@ namespace mfem {
  *   <a class="el" href="petsc_2ex6p_8cpp_source.html">6p</a>,
  *   <a class="el" href="petsc_2ex9p_8cpp_source.html">9p</a>,
  *   and
- *   <a class="el" href="petsc_2ex10p_8cpp_source.html">10p</a>,
- *   demonstrating the use of MFEM's \link petsc.hpp PETSc integration\endlink.
+ *   <a class="el" href="petsc_2ex10p_8cpp_source.html">10p</a>
+ *   demonstrating the use of MFEM's \link petsc.hpp PETSc classes\endlink
  *
  * <H4>PUMI Examples</H4>
  * - Variants of Examples
@@ -129,27 +124,19 @@ namespace mfem {
  *   <a class="el" href="examples_2pumi_2ex1p_8cpp_source.html">1p</a>,
  *   <a class="el" href="pumi_2ex2_8cpp_source.html">2</a>,
  *   and
- *   <a class="el" href="pumi_2ex6p_8cpp_source.html">6p</a>,
- *   demonstrating the use of MFEM's \link pumi.hpp PUMI integration\endlink.
+ *   <a class="el" href="pumi_2ex6p_8cpp_source.html">6p</a>
+ *   demonstrating the use of MFEM's \link pumi.hpp PUMI classes\endlink
  *
- * <H4>SUNDIALS Examples</H4>
+ * <H4>HiOp Examples</H4>
  * - Variants of Examples
- *   <a class="el" href="sundials_2ex9_8cpp_source.html">9</a>,
- *   <a class="el" href="sundials_2ex9p_8cpp_source.html">9p</a>,
- *   <a class="el" href="sundials_2ex10_8cpp_source.html">10</a>,
- *   <a class="el" href="sundials_2ex10p_8cpp_source.html">10p</a>,
- *   <a class="el" href="sundials_2ex16_8cpp_source.html">16</a>,
- *   and
- *   <a class="el" href="sundials_2ex16p_8cpp_source.html">16p</a>,
- *   demonstrating the use of MFEM's \link sundials.hpp SUNDIALS integration\endlink.
- * - CVODES adjoint miniapps:
- *   <a class="el" href="cvsRoberts__ASAi__dns_8cpp_source.html">serial ODE system</a>,
- *   <a class="el" href="adjoint__advection__diffusion_8cpp_source.html">parallel advection-diffusion</a>.
+ *   <a class="el" href="hiop_2ex9_8cpp_source.html">9</a> and
+ *   <a class="el" href="hiop_2ex9p_8cpp_source.html">9p</a>,
+ *   demonstrating the use of MFEM's \link hiop.hpp HiOp classes\endlink
  *
- * <H4>SuperLU Examples</H4>
+ * <H4>Ginkgo Examples</H4>
  * - Variants of Example
- *   <a class="el" href="examples_2superlu_2ex1p_8cpp_source.html">1p</a>,
- *   demonstrating the use of MFEM's \link superlu.hpp SuperLU integration\endlink.
+ *   <a class="el" href="ginkgo_2ex1_8cpp_source.html">1</a>
+ *   demonstrating the use of MFEM's \link ginkgo.hpp Ginkgo classes\endlink
  *
  * <H3>Miniapps</H3>
  * - <a class="el" href="volta_8cpp_source.html">Volta</a>: simple electrostatics simulation code
@@ -175,11 +162,11 @@ namespace mfem {
  * - <a class="el" href="lor-transfer_8cpp_source.html">LOR Transfer</a>: map functions between high-order and low-order refined spaces
  * - <a class="el" href="findpts_8cpp_source.html">Find Points</a>: evaluate grid function in physical space, <a class="el" href="findpts_8cpp_source.html">serial</a> and <a class="el" href="pfindpts_8cpp_source.html">parallel</a> versions
  * - <a class="el" href="field-diff_8cpp_source.html">Field Diff</a>: compare grid functions on different meshes
- * - <a class="el" href="field-interp_8cpp_source.html">Field Interp</a>: transfer a grid functions between meshes
+ * - <a class="el" href="field-interp_8cpp_source.html">Field Interp</a>: transfer a grid functions betwen meshes
  * - <a class="el" href="miniapps_2performance_2ex1_8cpp_source.html">HPC Example 1</a>: high-performance nodal H1 FEM for the Laplace problem
  * - <a class="el" href="miniapps_2performance_2ex1p_8cpp_source.html">HPC Example 1p</a>: high-performance parallel nodal H1 FEM for the Laplace problem
  *
- * See also the <a class="el" href="https://mfem.org/examples/">examples documentation</a> online.
+ * See also the <a class="el" href="../../../examples/README.html">README</a> in the <code>examples/</code> directory
  */
 
 }

doc/README

@@ -5,7 +5,7 @@
                       | | | | | ||  _||  __/| | | | | |
                       |_| |_| |_||_|   \___||_| |_| |_|
 
-                               https://mfem.org
+                               http://mfem.org
 
 This directory contains Doxygen configuration files for building HTML
 documentation of the MFEM sources (including the example codes and miniapps).
@@ -21,7 +21,7 @@ create the file CodeDocumentation.html, which can be viewed in any web browser.
 
 Alternatively, the (pre-build) documentation can be browsed directly online at
 
-   https://mfem.github.io/doxygen/html/index.html
+   http://mfem.github.io/doxygen/html/index.html
 
 
 Some building considerations:

doc/makefile

@@ -13,6 +13,7 @@ SHELL = /bin/bash
 MFEM_DIR ?= ..
 DOXYGEN_CONF = CodeDocumentation.conf
 
+
 # doxygen uses: graphviz, latex
 html: $(DOXYGEN_CONF)
 	@# Generate the html documentation

doc/web/base.css

@@ -0,0 +1,208 @@
+body {
+    padding-top: 70px;
+}
+
+/* hack to increase height of headers to make #anchors work with the top nav-bar,
+   (note: we only want this on pages with TOC, otherwise there are other problems) */
+div.col-md-9 h1[id]::before,
+div.col-md-9 h2[id]::before,
+div.col-md-9 h3[id]::before,
+div.col-md-9 h4[id]::before,
+div.col-md-9 h5[id]::before,
+div.col-md-9 h6[id]::before {
+    content: "";
+    display: block;
+    margin-top: -75px;
+    height: 75px;
+}
+
+img {
+    max-width: 100%;
+    height: auto;
+}
+
+pre {
+    overflow-x: auto;
+}
+
+pre code {
+    overflow-wrap: normal;
+    white-space: pre;
+    font-size: 13.5px;
+}
+
+label {
+    font-weight: normal;
+}
+
+ul.nav li.main {
+    font-weight: bold;
+}
+
+div.col-md-3 {
+    padding-right: 0;
+}
+
+div.col-md-9, div.col-md-12 {
+    padding-bottom: 100px;
+}
+
+div.source-links {
+    float: right;
+}
+
+img {
+    display: block;
+    margin-left: auto;
+    margin-right: auto;
+    margin-top: 25px;
+    margin-bottom: 25px;
+}
+
+img.floatright {
+    display: block;
+    float: right;
+    margin-top: 0;
+    margin-left: 25px;
+    margin-bottom: 25px;
+}
+
+img.teamphoto {
+    box-shadow: 4px 4px 4px #999;
+}
+
+p.centered {
+    margin-left: auto;
+    margin-right: auto;
+    text-align: center;
+}
+
+/*label {
+    display: inline-block;
+    margin-bottom: 5px;
+    font-weight: normal;
+}*/
+
+/*
+ * Side navigation
+ *
+ * Scrollspy and affixed enhanced navigation to highlight sections and secondary
+ * sections of docs content.
+ */
+
+/* By default it's not affixed in mobile views, so undo that */
+.bs-sidebar.affix {
+    position: static;
+}
+
+.bs-sidebar.well {
+    padding: 0;
+}
+
+/* First level of nav */
+.bs-sidenav {
+    /*margin-top: 10px;
+    margin-bottom: 10px;*/
+    padding-top:    10px;
+    padding-bottom: 10px;
+    border-radius: 5px;
+}
+
+/* All levels of nav */
+.bs-sidebar .nav > li > a {
+    display: block;
+    padding: 5px 20px;
+    z-index: 1;
+}
+.bs-sidebar .nav > li > a:hover,
+.bs-sidebar .nav > li > a:focus {
+    text-decoration: none;
+    border-right: 1px solid;
+}
+.bs-sidebar .nav > .active > a,
+.bs-sidebar .nav > .active:hover > a,
+.bs-sidebar .nav > .active:focus > a {
+    font-weight: bold;
+    background-color: transparent;
+    border-right: 1px solid;
+}
+
+/* Nav: second level (shown on .active) */
+.bs-sidebar .nav .nav {
+    display: none; /* Hide by default, but at >768px, show it */
+    margin-bottom: 8px;
+}
+.bs-sidebar .nav .nav > li > a {
+    padding-top:    3px;
+    padding-bottom: 3px;
+    padding-left: 30px;
+    font-size: 90%;
+}
+
+/* Show and affix the side nav when space allows it */
+@media (min-width: 992px) {
+    .bs-sidebar .nav > .active > ul {
+        display: block;
+    }
+    /* Widen the fixed sidebar */
+    .bs-sidebar.affix,
+    .bs-sidebar.affix-bottom {
+        width: 213px;
+    }
+    .bs-sidebar.affix {
+        position: fixed; /* Undo the static from mobile first approach */
+        top: 80px;
+    }
+    .bs-sidebar.affix-bottom {
+        position: absolute; /* Undo the static from mobile first approach */
+    }
+    .bs-sidebar.affix-bottom .bs-sidenav,
+    .bs-sidebar.affix .bs-sidenav {
+        margin-top: 0;
+        margin-bottom: 0;
+    }
+}
+@media (min-width: 1200px) {
+    /* Widen the fixed sidebar again */
+    .bs-sidebar.affix-bottom,
+    .bs-sidebar.affix {
+        width: 263px;
+    }
+}
+
+
+/* Sticky footer styles
+-------------------------------------------------- */
+html {
+  position: relative;
+  min-height: 100%;
+}
+body {
+  /* Margin bottom by footer height */
+  margin-bottom: 50px;
+}
+.footer {
+  position: absolute;
+  bottom: 0;
+  width: 100%;
+  /* Set the fixed height of the footer here */
+  height: 50px;
+  background-color: #eeeeee;
+  overflow: hidden;
+}
+
+
+/* Custom page CSS
+-------------------------------------------------- */
+/* Not required for template or sticky footer method. */
+
+.container .text-muted {
+  margin-top: 17px;
+  margin-bottom: 0;
+}
+.container .alignleft {
+  float: left;
+}
+.container .alignright {
+  float: right;
+}

examples/CMakeLists.txt

@@ -117,19 +117,19 @@ if (MFEM_USE_SUPERLU)
     ${MPIEXEC_POSTFLAGS})
 endif()
 
-# Include the examples/amgx directory if AmgX is enabled
-if (MFEM_USE_AMGX)
-  add_subdirectory(amgx)
+# Include the examples/sundials directory if SUNDIALS is enabled.
+if (MFEM_USE_SUNDIALS)
+  add_subdirectory(sundials)
 endif()
 
-# Include the examples/ginkgo directory if GINKGO is enabled.
+# Include the examples/sundials directory if SUNDIALS is enabled.
 if (MFEM_USE_GINKGO)
   add_subdirectory(ginkgo)
 endif()
 
-# Include the examples/hiop directory if HiOp is enabled
-if (MFEM_USE_HIOP)
-  add_subdirectory(hiop)
+# Include the examples/amgx directory if AmgX is enabled
+if (MFEM_USE_AMGX)
+  add_subdirectory(amgx)
 endif()
 
 # Include the examples/petsc directory if PETSc is enabled.
@@ -142,12 +142,6 @@ if (MFEM_USE_PUMI)
   add_subdirectory(pumi)
 endif()
 
-# Include the examples/sundials directory if SUNDIALS is enabled.
-if (MFEM_USE_SUNDIALS)
-  add_subdirectory(sundials)
-endif()
-
-# Include the examples/superlu directory if SUPERLU is enabled.
-if (MFEM_USE_SUPERLU)
-  add_subdirectory(superlu)
+if (MFEM_USE_HIOP)
+  add_subdirectory(hiop)
 endif()

examples/amgx/ex1.cpp

@@ -30,10 +30,6 @@
 #include <fstream>
 #include <iostream>
 
-#ifndef MFEM_USE_AMGX
-#error This example requires that MFEM is built with MFEM_USE_AMGX=YES
-#endif
-
 using namespace std;
 using namespace mfem;
 

examples/amgx/ex1p.cpp

@@ -31,10 +31,6 @@
 using namespace std;
 using namespace mfem;
 
-#ifndef MFEM_USE_AMGX
-#error This example requires that MFEM is built with MFEM_USE_AMGX=YES
-#endif
-
 int main(int argc, char *argv[])
 {
    // 1. Initialize MPI.

examples/ex17.cpp

@@ -403,6 +403,7 @@ VisMan::~VisMan()
    }
 }
 
+
 ostream &operator<<(ostream &v, void (*f)(VisMan&))
 {
    VisMan *vp = dynamic_cast<VisMan*>(&v);

examples/ex17p.cpp

@@ -441,6 +441,7 @@ VisMan::~VisMan()
    }
 }
 
+
 ostream &operator<<(ostream &v, void (*f)(VisMan&))
 {
    VisMan *vp = dynamic_cast<VisMan*>(&v);

examples/ex23.cpp

@@ -288,7 +288,6 @@ int main(int argc, char *argv[])
 
    GridFunction u_gf(&fespace);
    GridFunction dudt_gf(&fespace);
-
    // 6. Set the initial conditions for u. All boundaries are considered
    //    natural.
    FunctionCoefficient u_0(InitialSolution);

examples/ex25.cpp

@@ -13,16 +13,14 @@
 // Description:  This example code solves a simple electromagnetic wave
 //               propagation problem corresponding to the second order
 //               indefinite Maxwell equation
-//
 //                  (1/mu) * curl curl E - \omega^2 * epsilon E = f
-//
 //               with a Perfectly Matched Layer (PML).
 //
 //               The example demonstrates discretization with Nedelec finite
 //               elements in 2D or 3D, as well as the use of complex-valued
 //               bilinear and linear forms. Several test problems are included,
 //               with prob = 0-3 having known exact solutions, see "On perfectly
-//               matched layers for discontinuous Petrov-Galerkin methods" by
+//               matched layers for discontinuous Petrov–Galerkin methods" by
 //               Vaziri Astaneh, Keith, Demkowicz, Comput Mech 63, 2019.
 //
 //               We recommend viewing Example 22 before viewing this example.

examples/ex25p.cpp

@@ -13,16 +13,14 @@
 // Description:  This example code solves a simple electromagnetic wave
 //               propagation problem corresponding to the second order
 //               indefinite Maxwell equation
-//
 //                  (1/mu) * curl curl E - \omega^2 * epsilon E = f
-//
 //               with a Perfectly Matched Layer (PML).
 //
 //               The example demonstrates discretization with Nedelec finite
 //               elements in 2D or 3D, as well as the use of complex-valued
 //               bilinear and linear forms. Several test problems are included,
 //               with prob = 0-3 having known exact solutions, see "On perfectly
-//               matched layers for discontinuous Petrov-Galerkin methods" by
+//               matched layers for discontinuous Petrov–Galerkin methods" by
 //               Vaziri Astaneh, Keith, Demkowicz, Comput Mech 63, 2019.
 //
 //               We recommend viewing Example 22 before viewing this example.
@@ -164,8 +162,6 @@ int main(int argc, char *argv[])
    int iprob = 4;
    double freq = 5.0;
    bool herm_conv = true;
-   bool slu_solver  = false;
-   bool mumps_solver = false;
    bool visualization = 1;
 
    OptionsParser args(argc, argv);
@@ -187,26 +183,10 @@ int main(int argc, char *argv[])
                   "Frequency (in Hz).");
    args.AddOption(&herm_conv, "-herm", "--hermitian", "-no-herm",
                   "--no-hermitian", "Use convention for Hermitian operators.");
-#ifdef MFEM_USE_SUPERLU
-   args.AddOption(&slu_solver, "-slu", "--superlu", "-no-slu",
-                  "--no-superlu", "Use the SuperLU Solver.");
-#endif
-#ifdef MFEM_USE_MUMPS
-   args.AddOption(&mumps_solver, "-mumps", "--mumps-solver", "-no-mumps",
-                  "--no-mumps-solver", "Use the MUMPS Solver.");
-#endif
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
    args.Parse();
-   if (slu_solver && mumps_solver)
-   {
-      if (myid == 0)
-         cout << "WARNING: Both SuperLU and MUMPS have been selected,"
-              << " please choose either one." << endl
-              << "         Defaulting to SuperLU." << endl;
-      mumps_solver = false;
-   }
 
    if (iprob > 4) { iprob = 4; }
    prob = (prob_type)iprob;
@@ -448,7 +428,6 @@ int main(int argc, char *argv[])
 
    // 15. Solve using a direct or an iterative solver
 #ifdef MFEM_USE_SUPERLU
-   if (slu_solver)
    {
       // Transform to monolithic HypreParMatrix
       HypreParMatrix *A = Ah.As<ComplexHypreParMatrix>()->GetSystemMatrix();
@@ -461,19 +440,7 @@ int main(int argc, char *argv[])
       superlu.Mult(B, X);
       delete A;
    }
-#endif
-#ifdef MFEM_USE_MUMPS
-   if (mumps_solver)
-   {
-      HypreParMatrix *A = Ah.As<ComplexHypreParMatrix>()->GetSystemMatrix();
-      MUMPSSolver mumps;
-      mumps.SetPrintLevel(0);
-      mumps.SetMatrixSymType(MUMPSSolver::MatType::UNSYMMETRIC);
-      mumps.SetOperator(*A);
-      mumps.Mult(B,X);
-      delete A;
-   }
-#endif
+#else
    // 16a. Set up the parallel Bilinear form a(.,.) for the preconditioner
    //
    //    In Comp
@@ -481,7 +448,6 @@ int main(int argc, char *argv[])
    //
    //    In PML:   1/mu (abs(1/det(J) J^T J) Curl E, Curl F)
    //              + omega^2 * epsilon (abs(det(J) * (J^T J)^-1) * E, F)
-   if (!slu_solver && !mumps_solver)
    {
       ConstantCoefficient absomeg(pow(omega, 2) * epsilon);
       RestrictedCoefficient restr_absomeg(absomeg,attr);
@@ -531,6 +497,7 @@ int main(int argc, char *argv[])
       gmres.SetPreconditioner(BlockAMS);
       gmres.Mult(B, X);
    }
+#endif
 
    // 17. Recover the parallel grid function corresponding to X. This is the
    //     local finite element solution on each processor.

examples/ex26.cpp

@@ -204,9 +204,9 @@ int main(int argc, char *argv[])
    x = 0.0;
 
    // 8. Create the multigrid operator using the previously created
-   //    FiniteElementSpaceHierarchy and additional boundary information. This
-   //    operator is then used to create the MultigridSolver as a preconditioner
-   //    in the iterative solver.
+   //    FiniteElementSpaceHierarchy and additional boundary information. This operator
+   //    is then used to create the MultigridSolver as a preconditioner in the
+   //    iterative solver.
    Array<int> ess_bdr(mesh->bdr_attributes.Max());
    ess_bdr = 1;
 
@@ -224,8 +224,8 @@ int main(int argc, char *argv[])
    // 10. Recover the solution as a finite element grid function.
    M.RecoverFineFEMSolution(X, *b, x);
 
-   // 11. Save the refined mesh and the solution. This output can be viewed
-   //     later using GLVis: "glvis -m refined.mesh -g sol.gf".
+   // 11. Save the refined mesh and the solution. This output can be viewed later
+   //     using GLVis: "glvis -m refined.mesh -g sol.gf".
    ofstream mesh_ofs("refined.mesh");
    mesh_ofs.precision(8);
    fespaces.GetFinestFESpace().GetMesh()->Print(mesh_ofs);

examples/ex26p.cpp

@@ -245,9 +245,9 @@ int main(int argc, char *argv[])
    x = 0.0;
 
    // 10. Create the multigrid operator using the previously created parallel
-   //     FiniteElementSpaceHierarchy and additional boundary information. This
-   //     operator is then used to create the MultigridSolver as preconditioner
-   //     in the iterative solver.
+   //     FiniteElementSpaceHierarchy and additional boundary information. This operator
+   //     is then used to create the MultigridSolver as a preconditioner in the
+   //     iterative solver.
    Array<int> ess_bdr(pmesh->bdr_attributes.Max());
    if (pmesh->bdr_attributes.Size())
    {
@@ -274,8 +274,8 @@ int main(int argc, char *argv[])
    //     local finite element solution on each processor.
    M->RecoverFineFEMSolution(X, *b, x);
 
-   // 13. Save the refined mesh and the solution in parallel. This output can be
-   //     viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
+   // 13. Save the refined mesh and the solution in parallel. This output can
+   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
    {
       ostringstream mesh_name, sol_name;
       mesh_name << "mesh." << setfill('0') << setw(6) << myid;

examples/ex27.cpp

@@ -10,50 +10,50 @@
 // Description:  This example code demonstrates the use of MFEM to define a
 //               simple finite element discretization of the Laplace problem
 //               -Delta u = 0 with a variety of boundary conditions.
-//
 //               Specifically, we discretize using a FE space of the specified
-//               order using a continuous or discontinuous space. We then apply
-//               Dirichlet, Neumann (both homogeneous and inhomogeneous), Robin,
-//               and Periodic boundary conditions on different portions of a
-//               predefined mesh.
+//               order using a continuous or discontinuous space.  We then
+//               apply Dirichlet, Neumann (both homogeneous and inhomogeneous),
+//               Robin, and Periodic boundary conditions on different portions
+//               of a predefined mesh.
 //
-//               The predefined mesh consists of a rectangle with two holes
-//               removed (see below). The narrow ends of the mesh are connected
-//               to form a Periodic boundary condition. The lower edge (tagged
-//               with attribute 1) receives an inhomogeneous Neumann boundary
-//               condition. A Robin boundary condition is applied to upper edge
-//               (attribute 2). The circular hole on the left (attribute 3)
-//               enforces a Dirichlet boundary condition. Finally, a natural
-//               boundary condition, or homogeneous Neumann BC, is applied to
-//               the circular hole on the right (attribute 4).
+//               The predefined mesh consists of a rectangle with two
+//               holes removed (see below).  The narrow ends of the
+//               mesh are connected to form a Periodic boundary
+//               condition.  The lower edge (tagged with attribute 1)
+//               receives an inhomogeneous Neumann boundary condition.
+//               A Robin boundary condition is applied to upper edge
+//               (attribute 2).  The circular hole on the left
+//               (attribute 3) enforces a Dirichlet boundary
+//               condition.  Finally, a natural boundary condition, or
+//               homogeneous Neumann BC, is applied to the circular
+//               hole on the right (attribute 4).
 //
-//                    Attribute 3    ^ y  Attribute 2
-//                          \        |      /
-//                       +-----------+-----------+
-//                       |    \_     |     _     |
-//                       |    / \    |    / \    |
-//                    <--+---+---+---+---+---+---+--> x
-//                       |    \_/    |    \_/    |
-//                       |           |      \    |
-//                       +-----------+-----------+       (hole radii are
-//                            /      |        \            adjustable)
-//                    Attribute 1    v    Attribute 4
-//
-//               The boundary conditions are defined as (where u is the solution
-//               field):
+//                  Attribute 3    ^ y  Attribute 2
+//                        \        |      /
+//                     +-----------+-----------+
+//                     |    \_     |     _     |
+//                     |    / \    |    / \    |
+//                  <--+---+---+---+---+---+---+--> x
+//                     |    \_/    |    \_/    |
+//                     |           |      \    |
+//                     +-----------+-----------+       (hole radii are
+//                          /      |        \            adjustable)
+//                  Attribute 1    v    Attribute 4
 //
+//               The boundary conditions are defined as (where u is
+//               the solution field):
 //                  Dirichlet: u = d
 //                  Neumann:   n.Grad(u) = g
 //                  Robin:     n.Grad(u) + a u = b
 //
-//               The user can adjust the values of 'd', 'g', 'a', and 'b' with
-//               command line options.
+//               The user can adjust the values of 'd', 'g', 'a', and
+//               'b' with command line options.
 //
 //               This example highlights the differing implementations of
 //               boundary conditions with continuous and discontinuous Galerkin
 //               formulations of the Laplace problem.
 //
-//               We recommend viewing Examples 1 and 14 before viewing this
+//               We recommend viewing examples 1 and 14 before viewing this
 //               example.
 
 #include "mfem.hpp"
@@ -71,8 +71,8 @@ void n4Vec(const Vector &x, Vector &n) { n = x; n[0] -= 0.5; n /= -n.Norml2(); }
 Mesh * GenerateSerialMesh(int ref);
 
 // Compute the average value of alpha*n.Grad(sol) + beta*sol over the boundary
-// attributes marked in bdr_marker. Also computes the L2 norm of
-// alpha*n.Grad(sol) + beta*sol - gamma over the same boundary.
+// attributes marked in bdr_marker.  Also computes the L2 norm of
+// alpha*n.Grad(sol) + beta*sol - gamma  over the same boundary.
 double IntegrateBC(const GridFunction &sol, const Array<int> &bdr_marker,
                    double alpha, double beta, double gamma,
                    double &err);
@@ -152,9 +152,9 @@ int main(int argc, char *argv[])
    Mesh *mesh = GenerateSerialMesh(ser_ref_levels);
    int dim = mesh->Dimension();
 
-   // 3. Define a finite element space on the serial mesh. Here we use either
-   //    continuous Lagrange finite elements or discontinuous Galerkin finite
-   //    elements of the specified order.
+   // 3. Define a finite element space on the serial mesh.  Here we
+   //    use either continuous Lagrange finite elements or discontinuous
+   //    Galerkin finite elements of the specified order.
    FiniteElementCollection *fec =
       h1 ? (FiniteElementCollection*)new H1_FECollection(order, dim) :
       (FiniteElementCollection*)new DG_FECollection(order, dim);
@@ -162,10 +162,11 @@ int main(int argc, char *argv[])
    int size = fespace.GetTrueVSize();
    mfem::out << "Number of finite element unknowns: " << size << endl;
 
-   // 4. Create "marker arrays" to define the portions of boundary associated
-   //    with each type of boundary condition. These arrays have an entry
-   //    corresponding to each boundary attribute.  Placing a '1' in entry i
-   //    marks attribute i+1 as being active, '0' is inactive.
+   // 4. Create "marker arrays" to define the portions of the boundary
+   //    associated with each type of boundary condition.  These arrays
+   //    have an entry corresponding to each boundary attribute.
+   //    Placing a '1' in entry i marks attribute i+1 as being
+   //    active, '0' is inactive.
    Array<int> nbc_bdr(mesh->bdr_attributes.Max());
    Array<int> rbc_bdr(mesh->bdr_attributes.Max());
    Array<int> dbc_bdr(mesh->bdr_attributes.Max());
@@ -177,25 +178,25 @@ int main(int argc, char *argv[])
    Array<int> ess_tdof_list(0);
    if (h1 && mesh->bdr_attributes.Size())
    {
-      // For a continuous basis the linear system must be modified to enforce an
-      // essential (Dirichlet) boundary condition. In the DG case this is not
-      // necessary as the boundary condition will only be enforced weakly.
+      // For a continuous basis the linear system must be modifed to enforce
+      // an essential (Dirichlet) boundary condition.  In the DG case this is
+      // not necessary as the boundary condition will only be enforced weakly.
       fespace.GetEssentialTrueDofs(dbc_bdr, ess_tdof_list);
    }
 
-   // 5. Setup the various coefficients needed for the Laplace operator and the
-   //    various boundary conditions. In general these coefficients could be
-   //    functions of position but here we use only constants.
+   // 5. Setup the various coefficients needed for the Laplace operator and
+   //    the various boundary conditions.  In general these coefficients could
+   //    be functions of position but here we use only constants.
    ConstantCoefficient matCoef(mat_val);
    ConstantCoefficient dbcCoef(dbc_val);
    ConstantCoefficient nbcCoef(nbc_val);
    ConstantCoefficient rbcACoef(rbc_a_val);
    ConstantCoefficient rbcBCoef(rbc_b_val);
 
-   // Since the n.Grad(u) terms arise by integrating -Div(m Grad(u)) by parts we
-   // must introduce the coefficient 'm' into the boundary conditions.
-   // Therefore, in the case of the Neumann BC, we actually enforce m n.Grad(u)
-   // = m g rather than simply n.Grad(u) = g.
+   // Since the n.Grad(u) terms arise by integrating -Div(m Grad(u)) by parts
+   // we must introduce the coefficient 'm' into the boundary conditions.
+   // Therefore, in the case of the Neumann BC, we actually enforce
+   // m n.Grad(u) = m g rather than simply n.Grad(u) = g.
    ProductCoefficient m_nbcCoef(matCoef, nbcCoef);
    ProductCoefficient m_rbcACoef(matCoef, rbcACoef);
    ProductCoefficient m_rbcBCoef(matCoef, rbcBCoef);
@@ -217,7 +218,7 @@ int main(int argc, char *argv[])
    }
    else
    {
-      // Add the interfacial portion of the Laplace operator
+      // Add the interfacial portion of the Lapalce operator
       a.AddInteriorFaceIntegrator(new DGDiffusionIntegrator(matCoef,
                                                             sigma, kappa));
 
@@ -236,7 +237,7 @@ int main(int argc, char *argv[])
 
    if (h1)
    {
-      // Set the Dirichlet values in the solution vector
+      // Set the Dirchlet values in the solution vector
       u.ProjectBdrCoefficient(dbcCoef, dbc_bdr);
 
       // Add the desired value for n.Grad(u) on the Neumann boundary
@@ -247,7 +248,7 @@ int main(int argc, char *argv[])
    }
    else
    {
-      // Add the desired value for the Dirichlet boundary
+      // Add the desired value for the Dirchlet boundary
       b.AddBdrFaceIntegrator(new DGDirichletLFIntegrator(dbcCoef, matCoef,
                                                          sigma, kappa),
                              dbc_bdr);
@@ -291,12 +292,12 @@ int main(int argc, char *argv[])
    umf_solver.Mult(B, X);
 #endif
 
-   // 12. Recover the grid function corresponding to U. This is the local finite
-   //     element solution.
+   // 12. Recover the grid function corresponding to U. This is the
+   //     local finite element solution.
    a.RecoverFEMSolution(X, b, u);
 
-   // 13. Build a mass matrix to help solve for n.Grad(u) where 'n' is a surface
-   //     normal.
+   // 13. Build a mass matrix to help solve for n.Grad(u) where 'n' is
+   //     a surface normal.
    BilinearForm m(&fespace);
    m.AddDomainIntegrator(new MassIntegrator);
    m.Assemble();
@@ -310,8 +311,8 @@ int main(int argc, char *argv[])
              << "Verifying boundary conditions" << endl
              << "=============================" << endl;
    {
-      // Integrate the solution on the Dirichlet boundary and compare to the
-      // expected value.
+      // Integrate the solution on the Dirichlet boundary and compare
+      // to the expected value.
       double err, avg = IntegrateBC(u, dbc_bdr, 0.0, 1.0, dbc_val, err);
 
       bool hom_dbc = (dbc_val == 0.0);
@@ -322,8 +323,8 @@ int main(int argc, char *argv[])
                 << " error " << err << endl;
    }
    {
-      // Integrate n.Grad(u) on the inhomogeneous Neumann boundary and compare
-      // to the expected value.
+      // Integrate n.Grad(u) on the inhomogeneous Neumann boundary and
+      // compare to the expected value.
       double err, avg = IntegrateBC(u, nbc_bdr, 1.0, 0.0, nbc_val, err);
 
       bool hom_nbc = (nbc_val == 0.0);
@@ -334,8 +335,8 @@ int main(int argc, char *argv[])
                 << " error " << err << endl;
    }
    {
-      // Integrate n.Grad(u) on the homogeneous Neumann boundary and compare to
-      // the expected value of zero.
+      // Integrate n.Grad(u) on the homogeneous Neumann boundary and compare
+      // to the expected value of zero.
       Array<int> nbc0_bdr(mesh->bdr_attributes.Max());
       nbc0_bdr = 0;
       nbc0_bdr[3] = 1;
@@ -349,8 +350,8 @@ int main(int argc, char *argv[])
                 << " error " << err << endl;
    }
    {
-      // Integrate n.Grad(u) + a * u on the Robin boundary and compare to the
-      // expected value.
+      // Integrate n.Grad(u) + a * u on the Robin boundary and compare to
+      // the expected value.
       double err, avg = IntegrateBC(u, rbc_bdr, 1.0, rbc_a_val, rbc_b_val, err);
 
       bool hom_rbc = (rbc_b_val == 0.0);
@@ -726,8 +727,8 @@ double IntegrateBC(const GridFunction &x, const Array<int> &bdr,
       avg /= nrm;
    }
 
-   // Compute l2 norm of the error in the boundary condition (negative
-   // quadrature weights may produce negative 'err')
+   // Compute l2 norm of the error in the boundary condition
+   // (negative quadrature weights may produce negative 'err')
    err = (err >= 0.0) ? sqrt(err) : -sqrt(-err);
 
    // Return the average value of alpha * n.Grad(x) + beta * x