updting io benchmark

Boundary integral order bug [bdr-int-order-fix]

.gitignore

@@ -72,6 +72,10 @@ examples/deformed.*
 examples/velocity.*
 examples/elastic_energy.*
 examples/mode_*
+examples/ex5-p-*.bp
+examples/ex9-p-*.bp
+examples/ex12-p-*.bp
+examples/ex16-p-*.bp
 examples/ex16.mesh
 examples/ex16-mesh.*
 examples/ex16-init.*
@@ -165,6 +169,8 @@ miniapps/meshing/shaper
 miniapps/meshing/extruder
 miniapps/meshing/mesh-optimizer
 miniapps/meshing/pmesh-optimizer
+miniapps/meshing/minimal-surface
+miniapps/meshing/pminimal-surface
 
 miniapps/meshing/mobius-strip.mesh
 miniapps/meshing/klein-bottle.mesh
@@ -225,6 +231,14 @@ miniapps/gslib/field-diff
 miniapps/gslib/findpts
 miniapps/gslib/pfindpts
 
+miniapps/navier/navier_mms
+miniapps/navier/navier_kovasznay
+miniapps/navier/navier_tgv
+miniapps/navier/navier_shear
+miniapps/navier/navier_3dfoc
+miniapps/navier/tgv_out*.txt
+miniapps/navier/*_output
+
 # Unit test binary and outputs
 tests/unit/output_meshes
 tests/unit/unit_tests

CHANGELOG

@@ -23,19 +23,47 @@ Meshing improvements
   Hessian for r-adaptivity using discrete fields, and allows use of skewness
   and orientation based metrics.
 
+Improved GPU capabilities
+-------------------------
+- Added support for Chebyshev accelerated polynomial smoother on GPU.
+
+Discretization improvements
+---------------------------
+- Added support for matrix-free interpolation and restriction operators between
+  continuous H1 finite element spaces of different order on the same mesh or
+  with the same order on uniformly refined meshes.
+
+- Added support for simplices in GSLIB-FindPoints.
+
+Linear and nonlinear solvers
+----------------------------
+- Added power method to iteratively estimate the largest eigenvalue and the
+  corresponding eigenvector of an operator.
+
+- Added initial support for h- and p-multigrid solvers and preconditioners for
+  matrix-based and matrix-free discretizations with basic GPU capability.
+
 New and updated examples and miniapps
 -------------------------------------
-- Adding a simple meshing miniapp, Twist, which demonstrates MFEM's strategy of
-  stitching together opposite surfaces of a mesh to create a topologically
-  periodic mesh.
-
 - Added a new example, Example 25/25p, to demonstrate the use of a Perfectly
   Matched Layer (PML) for the simulation of electromagnetic wave propagation.
   The example defines and solves several indefinite Maxwell problems.
 
-Discretization improvements
----------------------------
-- Added support for simplices in GSLIB-FindPoints.
+- Added a new Example 26/26p to demonstrate the construction of a matrix-free
+  geometric and p-multigrid preconditioner for the Laplace problem.
+
+- Added a new example, Example 27/27p, to demonstrate the enforcement of
+  various boundary conditions with the Laplace operator.  The example shows the
+  procedures for applying Dirichlet, Neumann (both homogeneous and
+  inhomogeneous), Robin, and periodic boundary conditions with either H1 or DG
+  discretizations.
+
+- Added a simple meshing miniapp, Twist, which demonstrates MFEM's strategy of
+  stitching together opposite surfaces of a mesh to create a topologically
+  periodic mesh.
+
+- Added a new meshing miniapp, Minimal Surface, which solves Plateau's problem:
+  the Dirichlet problem for the minimal surface equation.
 
 Improved testing
 ----------------
@@ -49,6 +77,12 @@ Miscellaneous
 - In SLISolver, changed the residual inner product from (Br,r) to (Br,Br) so the
   solver can work with non-SPD preconditioner B.
 
+- Added support for ADIOS2 for parallel I/O with ParaView visualization. The
+  classes adios2stream and ADIOS2DataCollection are introduced in mfem as the
+  interfaces to generate ADIOS2 Binary Pack (BP4) directory datasets for the
+  entire spatial and temporal data. In addition, ADIOS2 allows for setting a
+  user-defined number of data substreams/subfiles. See examples 5, 9, 12, 16.
+
 
 Version 4.1, released on March 10, 2020
 =======================================

CMakeLists.txt

@@ -323,6 +323,11 @@ if (MFEM_USE_UMPIRE)
    find_package(UMPIRE REQUIRED)
 endif()
 
+# ADIOS2 for parallel I/O
+if (MFEM_USE_ADIOS2)
+   find_package(ADIOS2 REQUIRED)
+endif()
+
 # MFEM_TIMER_TYPE
 if (NOT DEFINED MFEM_TIMER_TYPE)
   if (APPLE)
@@ -348,7 +353,7 @@ endif()
 #    be before SuiteSparse.
 set(MFEM_TPLS MPI_CXX OPENMP BLAS LAPACK METIS HYPRE SuiteSparse SUNDIALS PETSC
     MESQUITE SuperLUDist STRUMPACK AXOM CONDUIT Ginkgo GNUTLS GSLIB NETCDF
-    MPFR PUMI HIOP POSIXCLOCKS MFEMBacktrace ZLIB OCCA CEED RAJA UMPIRE)
+    MPFR PUMI HIOP POSIXCLOCKS MFEMBacktrace ZLIB OCCA CEED RAJA UMPIRE ADIOS2)
 # Add all *_FOUND libraries in the variable TPL_LIBRARIES.
 set(TPL_LIBRARIES "")
 set(TPL_INCLUDE_DIRS "")

CONTRIBUTING.md

@@ -383,9 +383,13 @@ Before a PR can be merged, it should satisfy the following:
     - [ ] Is this a new feature users need to be aware of? New or updated example or miniapp?
     - [ ] Does it make sense to create a new section in the `CHANGELOG` to group with other related features?
 - [ ] Update `INSTALL`:
-    - [ ] Had a new optional library been added? (*Make sure the external library is compatible with our BSD license, e.g. it is not licensed under GPL!*)
+    - [ ] Had a new optional library been added? If so, what range of versions of this library are required? (*Make sure the external library is compatible with our BSD license, e.g. it is not licensed under GPL!*)
+    - [ ] Have the version ranges for any required or optional libraries changed?
     - [ ] Does `make` or `cmake` have a new target?
     - [ ] Did the requirements or the installation process change? *(rare)*
+- [ ] Update continuous integration server configurations if necessary (e.g. with new version requirements for each of MFEM's dependencies)
+    - [ ] `.travis.yml`
+    - [ ] `.appveyor.yml`
 - [ ] Update `.gitignore`:
     - [ ] Check if `make distclean; git status` shows any files that were generated from the source by the project (not an IDE) but we don't want to track in the repository.
     - [ ] Add new patterns (just for the new files above) and re-run the above test.
@@ -499,6 +503,10 @@ MFEM uses a `master`/`next`-branch workflow as described below:
     - [ ] `makefile`
     - [ ] `CMakeLists.txt`
     - [ ] `doc/CodeDocumentation.conf.in`
+- [ ] Check that version requirements for each of MFEM's dependencies are documented in `INSTALL` and up-to-date
+- [ ] Check that continuous integration server configurations reflect the dependency version requirements of the new release
+    - [ ] `.travis.yml`
+    - [ ] `.appveyor.yml`
 - [ ] (LLNL only) Make sure all `README.html` files in the source repo are up to date.
 - [ ] Tag the repository:
 

INSTALL

@@ -403,6 +403,11 @@ MFEM_USE_CONDUIT = YES/NO
    an installation of Conduit. If Conduit was built with HDF5 support, it also
    requires an installation of HDF5 (see also MFEM_USE_NETCDF).
 
+MFEM_USE_ADIOS2 = YES/NO
+   Enables support for ADIOS2, version 2 of the adaptable input output system
+   for scientific data management. In MFEM, ADIOS2 provides parallel I/O with
+   ParaView visualization.
+
 MFEM_USE_ZLIB = YES/NO
    Enables use of on-the-fly gzip compressed streams. With this feature enabled
    (YES), MFEM can compress its output files on-the-fly. In addition, it can
@@ -492,11 +497,13 @@ The specific libraries and their options are:
 - HYPRE, required for the parallel build, i.e. when MFEM_USE_MPI = YES.
   URL: https://github.com/hypre-space/hypre and https://www.llnl.gov/casc/hypre
   Options: HYPRE_OPT, HYPRE_LIB.
+  Versions: HYPRE >= 2.10.0b.
 
 - METIS, used when MFEM_USE_METIS = YES. If using METIS 5, set
   MFEM_USE_METIS_5 = YES (default is to use METIS 4).
   URL: http://glaros.dtc.umn.edu/gkhome/metis/metis/overview
   Options: METIS_OPT, METIS_LIB.
+  Versions: METIS 4.0.3 or 5.1.0.
 
 - LAPACK (optional), used when MFEM_USE_LAPACK = YES. Alternative, optimized
   implementations can also be used, e.g. the ATLAS project.
@@ -520,6 +527,7 @@ The specific libraries and their options are:
   both MPI and hypre.
   URL: http://computation.llnl.gov/projects/sundials/sundials-software
   Options: SUNDIALS_OPT, SUNDIALS_LIB.
+  Versions: SUNDIALS >= 5.0.0.
 
 - Mesquite (optional), used when MFEM_USE_MESQUITE = YES.
   URL: http://trilinos.org/oldsite/packages/mesquite
@@ -528,6 +536,7 @@ The specific libraries and their options are:
 - SuiteSparse (optional), used when MFEM_USE_SUITESPARSE = YES.
   URL: http://faculty.cse.tamu.edu/davis/suitesparse.html
   Options: SUITESPARSE_OPT, SUITESPARSE_LIB.
+  Versions: SuiteSparse >= 4.5.4, older versions may work too.
 
 - SuperLU_DIST (optional), used when MFEM_USE_SUPERLU = YES. Note that
   SuperLU_DIST requires ParMETIS, which includes METIS 5 in its distribution.
@@ -535,6 +544,7 @@ The specific libraries and their options are:
   same location.
   URL: http://crd-legacy.lbl.gov/~xiaoye/SuperLU
   Options: SUPERLU_OPT, SUPERLU_LIB.
+  Versions: SuperLU_DIST >= 5.1.0.
 
 - STRUMPACK (optional), used when MFEM_USE_STRUMPACK = YES. Note that STRUMPACK
   requires the PT-Scotch and Scalapack libraries as well as ParMETIS, which
@@ -544,6 +554,7 @@ The specific libraries and their options are:
   2.0.0 or later.
   URL: http://portal.nersc.gov/project/sparse/strumpack
   Options: STRUMPACK_OPT, STRUMPACK_LIB.
+  Versions: STRUMPACK >= 3.0.0, requires HYPRE < 2.16.0.
 
 - Ginkgo (optional), used when MFEM_USE_GINKGO = YES. Note that Ginkgo needs a
   C++ compiler that supports the C++-11 standard. For additional requirements
@@ -556,6 +567,7 @@ The specific libraries and their options are:
   one can get the library through the Homebrew package manager (http://brew.sh).
   URL: http://gnutls.org
   Options: GNUTLS_OPT, GNUTLS_LIB.
+  Versions: GnuTLS >= 2.12.0, older versions may work too.
 
 - NetCDF (optional), used when MFEM_USE_NETCDF = YES, required for reading Cubit
   mesh files. Also requires installation of HDF5 and ZLIB, as explained at the
@@ -563,6 +575,7 @@ The specific libraries and their options are:
   don't need the C++ or parallel versions.
   URL: www.unidata.ucar.edu/software/netcdf
   Options: NETCDF_OPT, NETCDF_LIB.
+  Versions: NetCDF >= 4.4.0.
 
 - PETSc (optional), used when MFEM_USE_PETSC = YES. Version 3.8 or higher of
   the PETSC dev branch is required. The MFEM and PETSc builds can share common
@@ -574,6 +587,7 @@ The specific libraries and their options are:
                 --with-shared-libraries=0
   URL: https://www.mcs.anl.gov/petsc
   Options: PETSC_OPT, PETSC_LIB.
+  Versions: PETSc >= 3.8.0.
 
 - Sidre (optional), part of LLNL's axom project, used when MFEM_USE_SIDRE = YES.
   Starting with MFEM v4.1, Axom version 0.3.1 or later is required.
@@ -581,16 +595,22 @@ The specific libraries and their options are:
        https://github.com/LLNL/conduit (Conduit)
        https://support.hdfgroup.org/HDF5 (HDF5)
   Options: SIDRE_OPT, SIDRE_LIB.
+  Versions: Axom >= 0.3.1.
 
 - Conduit (optional), used when MFEM_USE_CONDUIT = YES. Conduit Mesh Blueprint
   support requires Conduit >= v0.3.1 and VisIt >= v2.13.1 to read the output.
   URL: https://github.com/LLNL/conduit (Conduit)
        https://support.hdfgroup.org/HDF5 (HDF5)
   Options: CONDUIT_OPT, CONDUIT_LIB.
+  Versions: Conduit >= 0.3.1.
+
+- ADIOS2 (optional) used when MFEM_USE_ADIOS2 = YES.
+  URL: https://adios2.readthedocs.io/
 
 - PUMI (optional), used when MFEM_USE_PUMI = YES.
   URL: https://scorec.rpi.edu/pumi
   Options: PUMI_OPT, PUMI_LIB.
+  Versions: PUMI >= 2.2.0.
 
 - HiOp (optional), used when MFEM_USE_HIOP = YES.
   URL: https://github.com/LLNL/hiop
@@ -604,10 +624,12 @@ The specific libraries and their options are:
   MFEM_USE_GSLIB=YES.
   URL: https://github.com/gslib/gslib/archive/v1.0.5.tar.gz
   Options: GSLIB_OPT, GSLIB_LIB.
+  Versions: GSLIB >= 1.0.5.
 
 - CUDA (optional), used when MFEM_USE_CUDA = YES.
   URL: https://developer.nvidia.com/cuda-toolkit
   Options: CUDA_CXX, CUDA_ARCH, CUDA_OPT, CUDA_LIB.
+  Versions: CUDA >= 9.1, older versions may work too.
 
 - HIP (optional), used when MFEM_USE_HIP = YES.
   URL: https://rocm.github.io/ROCmInstall.html
@@ -616,21 +638,25 @@ The specific libraries and their options are:
 - OCCA (optional), used when MFEM_USE_OCCA = YES.
   URL: https://libocca.org
   Options: OCCA_DIR, OCCA_OPT, OCCA_LIB.
+  Versions: OCCA >= 1.0.9.
 
 - libCEED (optional), used when MFEM_USE_CEED = YES. Requires libCEED v0.6
   or later version, specifically, git-hash 3d05795 or later.
   URL: https://github.com/CEED/libCEED
        https://ceed.exascaleproject.org/libceed
   Options: CEED_DIR, CEED_OPT, CEED_LIB.
+  Versions: libCEED >= 0.6.
 
 - RAJA (optional), used when MFEM_USE_RAJA = YES.
   Beginning with MFEM v4.1, only RAJA v0.10.0+ is supported.
   URL: https://github.com/LLNL/RAJA
   Options: RAJA_DIR, RAJA_OPT, RAJA_LIB.
+  Versions: RAJA >= 0.10.0.
 
 - Umpire, used when MFEM_USE_UMPIRE = YES.
   URL: https://github.com/LLNL/Umpire
   Options: UMPIRE_DIR, UMPIRE_OPT, UMPIRE_LIB.
+  Versions: Umpire >= 2.0.0.
 
 - MPFR (optional), used when MFEM_USE_MPFR = YES.
   URL: http://mpfr.org, it depends on the GMP library: https://gmplib.org

config/cmake/MFEMConfig.cmake.in

@@ -47,6 +47,7 @@ set(MFEM_USE_OCCA @MFEM_USE_OCCA@)
 set(MFEM_USE_RAJA @MFEM_USE_RAJA@)
 set(MFEM_USE_CEED @MFEM_USE_CEED@)
 set(MFEM_USE_UMPIRE @MFEM_USE_UMPIRE@)
+set(MFEM_USE_ADIOS2 @MFEM_USE_ADIOS2@)
 
 set(MFEM_CXX_COMPILER "@CMAKE_CXX_COMPILER@")
 set(MFEM_CXX_FLAGS "@CMAKE_CXX_FLAGS@")

config/cmake/config.hpp.in

@@ -132,6 +132,9 @@
 // Enable MFEM functionality based on the Umpire library
 #cmakedefine MFEM_USE_UMPIRE
 
+// Enable MFEM functionality based on the ADIOS2 library
+#cmakedefine MFEM_USE_ADIOS2
+
 // Which library functions to use in class StopWatch for measuring time.
 // For a list of the available options, see INSTALL.
 // If not defined, an option is selected automatically.

config/cmake/modules/FindADIOS2.cmake

@@ -0,0 +1,52 @@
+#------------------------------------------------------------------------------#
+# Distributed under the OSI-approved Apache License, Version 2.0.  See
+# accompanying file Copyright.txt for details.
+#------------------------------------------------------------------------------#
+#
+# FindADIOS2
+# -----------
+#
+# Try to find the ADIOS2 library
+#
+# This module defines the following variables:
+#
+#   ADIOS2_FOUND        - System has ADIOS2
+#   ADIOS2_INCLUDE_DIRS - The ADIOS2 include directory
+#   ADIOS2_LIBRARIES    - Link these to use ADIOS2
+#
+# and the following imported targets:
+#   ADIOS2::ADIOS2 - The ADIOS2 compression library target
+#
+# You can also set the following variable to help guide the search:
+#   ADIOS2_DIR - The install prefix for ADIOS2 containing the
+#              include and lib folders
+#              Note: this can be set as a CMake variable or an
+#                    environment variable.  If specified as a CMake
+#                    variable, it will override any setting specified
+#                    as an environment variable.
+
+if(NOT ADIOS2_FOUND)
+  if((NOT ADIOS2_DIR) AND (NOT (ENV{ADIOS2_DIR} STREQUAL "")))
+    set(ADIOS2_DIR "$ENV{ADIOS2_DIR}")
+  endif()
+  if(ADIOS2_DIR)
+    set(ADIOS2_INCLUDE_OPTS HINTS ${ADIOS2_DIR}/include NO_DEFAULT_PATHS)
+    set(ADIOS2_LIBRARY_OPTS
+      HINTS ${ADIOS2_DIR}/lib ${ADIOS2_DIR}/lib64
+      NO_DEFAULT_PATHS
+    )
+  endif()
+
+  find_path(ADIOS2_INCLUDE_DIR adios2.h ${ADIOS2_INCLUDE_OPTS})
+  find_library(ADIOS2_LIBRARY NAMES adios2 ${ADIOS2_LIBRARY_OPTS}) 
+
+  include(FindPackageHandleStandardArgs)
+  find_package_handle_standard_args(ADIOS2
+    FOUND_VAR ADIOS2_FOUND
+    REQUIRED_VARS ADIOS2_LIBRARY ADIOS2_INCLUDE_DIR
+  )
+  if(ADIOS2_FOUND)
+    set(ADIOS2_INCLUDE_DIRS ${ADIOS2_INCLUDE_DIR})
+    set(ADIOS2_LIBRARIES ${ADIOS2_LIBRARY})
+  endif()
+endif()

config/config.hpp.in

@@ -147,6 +147,9 @@
 // Enable functionality based on the Umpire library.
 // #define MFEM_USE_UMPIRE
 
+// Enable IO functionality based on the ADIOS2 library.
+// #define MFEM_USE_ADIOS2
+
 // Version of HYPRE used for building MFEM.
 // #define MFEM_HYPRE_VERSION @MFEM_HYPRE_VERSION@
 

config/config.mk.in

@@ -49,6 +49,7 @@ MFEM_USE_RAJA          = @MFEM_USE_RAJA@
 MFEM_USE_OCCA          = @MFEM_USE_OCCA@
 MFEM_USE_CEED          = @MFEM_USE_CEED@
 MFEM_USE_UMPIRE        = @MFEM_USE_UMPIRE@
+MFEM_USE_ADIOS2        = @MFEM_USE_ADIOS2@
 
 # Compiler, compile options, and link options
 MFEM_CXX       = @MFEM_CXX@

config/defaults.cmake

@@ -49,6 +49,7 @@ option(MFEM_USE_OCCA "Enable OCCA" OFF)
 option(MFEM_USE_RAJA "Enable RAJA" OFF)
 option(MFEM_USE_CEED "Enable CEED" OFF)
 option(MFEM_USE_UMPIRE "Enable Umpire" OFF)
+option(MFEM_USE_ADIOS2 "Enable ADIOS2" OFF)
 
 set(MFEM_MPI_NP 4 CACHE STRING "Number of processes used for MPI tests")
 

config/defaults.mk

@@ -137,6 +137,7 @@ MFEM_USE_RAJA          = NO
 MFEM_USE_OCCA          = NO
 MFEM_USE_CEED          = NO
 MFEM_USE_UMPIRE        = NO
+MFEM_USE_ADIOS2        = NO
 
 # Compile and link options for zlib.
 ZLIB_DIR =

config/sample-runs.sh

@@ -47,7 +47,7 @@ groups_serial=(
    "Meshing miniapps:"
    "miniapps/meshing"
    "mobius-strip.cpp klein-bottle.cpp extruder.cpp toroid.cpp
-    mesh-optimizer.cpp"'
+    mesh-optimizer.cpp minimal-surface.cpp"'
 )
 # Parallel groups
 groups_parallel=(
@@ -72,7 +72,7 @@ groups_parallel=(
 '"meshing"
    "Meshing miniapps:"
    "miniapps/meshing"
-   "pmesh-optimizer.cpp"'
+   "pmesh-optimizer.cpp pminimal-surface.cpp"'
 '"electromagnetics"
    "Electromagnetics miniapps:"
    "miniapps/electromagnetics"
@@ -101,7 +101,7 @@ groups_all=(
    "Meshing miniapps:"
    "miniapps/meshing"
    "mobius-strip.cpp klein-bottle.cpp extruder.cpp toroid.cpp
-    {,p}mesh-optimizer.cpp"'
+    {,p}mesh-optimizer.cpp {,p}minimal-surface.cpp"'
 '"electromagnetics"
    "Electromagnetics miniapps:"
    "miniapps/electromagnetics"

doc/CodeDocumentation.conf.in

@@ -774,6 +774,7 @@ INPUT                  = @MFEM_SOURCE_DIR@/doc/CodeDocumentation.dox \
                          @MFEM_SOURCE_DIR@/miniapps/electromagnetics \
                          @MFEM_SOURCE_DIR@/miniapps/gslib \
                          @MFEM_SOURCE_DIR@/miniapps/meshing \
+                         @MFEM_SOURCE_DIR@/miniapps/navier \
                          @MFEM_SOURCE_DIR@/miniapps/nurbs \
                          @MFEM_SOURCE_DIR@/miniapps/performance \
                          @MFEM_SOURCE_DIR@/miniapps/tools \

doc/CodeDocumentation.dox

@@ -88,6 +88,8 @@ namespace mfem {
  * - <a class="el" href="ex24p_8cpp_source.html">Example 24p</a>: parallel mixed finite element spaces and interpolators
  * - <a class="el" href="ex25_8cpp_source.html">Example 25</a>: simulation of electromagnetic wave propagation using a Perfectly Matched Layer (PML)
  * - <a class="el" href="ex25p_8cpp_source.html">Example 25p</a>: parallel simulation of electromagnetic wave propagation using a Perfectly Matched Layer (PML)
+ * - <a class="el" href="ex26_8cpp_source.html">Example 26</a>: multigrid preconditioner for the Laplace problem using nodal H1 FEM 
+ * - <a class="el" href="ex26p_8cpp_source.html">Example 26p</a>: parallel multigrid preconditioner for the Laplace problem using nodal H1 FEM 
  *
  * <H4>SUNDIALS Examples</H4>
  * - Variants of Examples
@@ -142,6 +144,7 @@ namespace mfem {
  * - <a class="el" href="klein-bottle_8cpp_source.html">Klein Bottle</a>: generate three types of Klein bottle surfaces
  * - <a class="el" href="toroid_8cpp_source.html">Toroid</a>: generate simple toroidal meshes
  * - <a class="el" href="twist_8cpp_source.html">Twist</a>: generate simple periodic meshes
+ * - <a class="el" href="minimal-surface_8cpp_source.html">Minimal Surface</a>: compute minimal surfaces, <a class="el" href="minimal-surface_8cpp_source.html">serial</a> and <a class="el" href="pminimal-surface_8cpp_source.html">parallel</a> versions
  * - <a class="el" href="shaper_8cpp_source.html">Shaper</a>: resolve material interfaces by mesh refinement
  * - <a class="el" href="extruder_8cpp_source.html">Extruder</a>: extrude a low-dimensional mesh into a higher dimension
  * - <a class="el" href="mesh-explorer_8cpp_source.html">Mesh Explorer</a>: visualize and manipulate meshes
@@ -153,6 +156,7 @@ namespace mfem {
  * - <a class="el" href="lor-transfer_8cpp_source.html">LOR Transfer</a>: map functions between high-order and low-order refined spaces
  * - <a class="el" href="findpts_8cpp_source.html">Find Points</a>: evaluate grid function in physical space, <a class="el" href="findpts_8cpp_source.html">serial</a> and <a class="el" href="pfindpts_8cpp_source.html">parallel</a> versions
  * - <a class="el" href="field-diff_8cpp_source.html">Field Diff</a>: compare grid functions on different meshes
+ * - <a class="el" href="classmfem_1_1navier_1_1NavierSolver.html">Navier</a>: solve the transient incompressible Navier-Stokes equations
  * - <a class="el" href="miniapps_2performance_2ex1_8cpp_source.html">HPC Example 1</a>: high-performance nodal H1 FEM for the Laplace problem
  * - <a class="el" href="miniapps_2performance_2ex1p_8cpp_source.html">HPC Example 1p</a>: high-performance parallel nodal H1 FEM for the Laplace problem
  *

examples/CMakeLists.txt

@@ -32,6 +32,8 @@ list(APPEND ALL_EXE_SRCS
   ex23.cpp
   ex24.cpp
   ex25.cpp
+  ex26.cpp
+  ex27.cpp
   )
 
 if (MFEM_USE_MPI)
@@ -60,6 +62,10 @@ if (MFEM_USE_MPI)
     ex22p.cpp
     ex24p.cpp
     ex25p.cpp
+    ex26p.cpp
+    ex27p.cpp
+    pa_oper.cpp
+    io_benchmark.cpp
     )
 endif()
 
@@ -79,6 +85,8 @@ foreach(SRC_FILE ${ALL_EXE_SRCS})
     list(APPEND THIS_TEST_OPTIONS "-tf" "5")
   elseif(${TEST_NAME} MATCHES "ex15p*")
     list(APPEND THIS_TEST_OPTIONS "-e" "1")
+  elseif(${TEST_NAME} MATCHES "ex27p*")
+    list(APPEND THIS_TEST_OPTIONS "-dg")
   endif()
 
   if (NOT (${TEST_NAME} MATCHES ".*p$"))

examples/ex12p.cpp

@@ -33,7 +33,8 @@
 //               The example highlights the use of the LOBPCG eigenvalue solver
 //               together with the BoomerAMG preconditioner in HYPRE. Reusing a
 //               single GLVis visualization window for multiple eigenfunctions
-//               is also illustrated.
+//               and optional saving with ADIOS2 (adios2.readthedocs.io) streams
+//               are also illustrated.
 //
 //               We recommend viewing examples 2 and 11 before viewing this
 //               example.
@@ -60,6 +61,7 @@ int main(int argc, char *argv[])
    int seed = 66;
    bool visualization = 1;
    bool amg_elast = 0;
+   bool adios2 = false;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
@@ -77,6 +79,9 @@ int main(int argc, char *argv[])
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
+   args.AddOption(&adios2, "-adios2", "--adios2-streams", "-no-adios2",
+                  "--no-adios2-streams",
+                  "Save data using adios2 streams.");
    args.Parse();
    if (!args.Good())
    {
@@ -286,7 +291,28 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 13. Send the above data by socket to a GLVis server. Use the "n" and "b"
+   // 13. Optionally output a BP (binary pack) file using ADIOS2. This can be
+   //     visualized with the ParaView VTX reader.
+#ifdef MFEM_USE_ADIOS2
+   if (adios2)
+   {
+      std::string postfix(mesh_file);
+      postfix.erase(0, std::string("../data/").size() );
+      postfix += "_o" + std::to_string(order);
+
+      adios2stream adios2output("ex12-p-" + postfix + ".bp",
+                                adios2stream::openmode::out, MPI_COMM_WORLD);
+      pmesh->Print(adios2output);
+      for (int i=0; i<nev; i++)
+      {
+         x = lobpcg->GetEigenvector(i);
+         // x is a temporary that must be saved immediately
+         x.Save(adios2output, "mode_" + std::to_string(i));
+      }
+   }
+#endif
+
+   // 14. Send the above data by socket to a GLVis server. Use the "n" and "b"
    //     keys in GLVis to visualize the displacements.
    if (visualization)
    {
@@ -326,7 +352,7 @@ int main(int argc, char *argv[])
       mode_sock.close();
    }
 
-   // 14. Free the used memory.
+   // 15. Free the used memory.
    delete lobpcg;
    delete amg;
    delete M;

examples/ex16p.cpp

@@ -24,7 +24,8 @@
 //               class ConductionOperator defining C(u)), as well as their
 //               implicit time integration. Note that implementing the method
 //               ConductionOperator::ImplicitSolve is the only requirement for
-//               high-order implicit (SDIRK) time integration.
+//               high-order implicit (SDIRK) time integration. Optional saving
+//               with ADIOS2 (adios2.readthedocs.io) is also illustrated.
 //
 //               We recommend viewing examples 2, 9 and 10 before viewing this
 //               example.
@@ -108,6 +109,7 @@ int main(int argc, char *argv[])
    bool visualization = true;
    bool visit = false;
    int vis_steps = 5;
+   bool adios2 = false;
 
    int precision = 8;
    cout.precision(precision);
@@ -140,6 +142,9 @@ int main(int argc, char *argv[])
                   "Save data files for VisIt (visit.llnl.gov) visualization.");
    args.AddOption(&vis_steps, "-vs", "--visualization-steps",
                   "Visualize every n-th timestep.");
+   args.AddOption(&adios2, "-adios2", "--adios2-streams", "-no-adios2",
+                  "--no-adios2-streams",
+                  "Save data using adios2 streams.");
    args.Parse();
    if (!args.Good())
    {
@@ -248,6 +253,27 @@ int main(int argc, char *argv[])
       visit_dc.Save();
    }
 
+   // Optionally output a BP (binary pack) file using ADIOS2. This can be
+   // visualized with the ParaView VTX reader.
+#ifdef MFEM_USE_ADIOS2
+   ADIOS2DataCollection* adios2_dc = NULL;
+   if (adios2)
+   {
+      std::string postfix(mesh_file);
+      postfix.erase(0, std::string("../data/").size() );
+      postfix += "_o" + std::to_string(order);
+      postfix += "_solver" + std::to_string(ode_solver_type);
+      const std::string collection_name = "ex16-p-" + postfix + ".bp";
+
+      adios2_dc = new ADIOS2DataCollection(MPI_COMM_WORLD, collection_name, pmesh);
+      adios2_dc->SetParameter("SubStreams", std::to_string(num_procs/2) );
+      adios2_dc->RegisterField("temperature", &u_gf);
+      adios2_dc->SetCycle(0);
+      adios2_dc->SetTime(0.0);
+      adios2_dc->Save();
+   }
+#endif
+
    socketstream sout;
    if (visualization)
    {
@@ -317,10 +343,26 @@ int main(int argc, char *argv[])
             visit_dc.SetTime(t);
             visit_dc.Save();
          }
+
+#ifdef MFEM_USE_ADIOS2
+         if (adios2)
+         {
+            adios2_dc->SetCycle(ti);
+            adios2_dc->SetTime(t);
+            adios2_dc->Save();
+         }
+#endif
       }
       oper.SetParameters(u);
    }
 
+#ifdef MFEM_USE_ADIOS2
+   if (adios2)
+   {
+      delete adios2_dc;
+   }
+#endif
+
    // 11. Save the final solution in parallel. This output can be viewed later
    //     using GLVis: "glvis -np <np> -m ex16-mesh -g ex16-final".
    {

examples/ex1p.cpp

@@ -70,6 +70,7 @@ int main(int argc, char *argv[])
    bool pa = false;
    const char *device_config = "cpu";
    bool visualization = true;
+   int nfiles = 1;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
@@ -86,6 +87,7 @@ int main(int argc, char *argv[])
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
+   args.AddOption(&nfiles, "-nf", "--num-files", "Number of files to write.");
    args.Parse();
    if (!args.Good())
    {
@@ -158,7 +160,7 @@ int main(int argc, char *argv[])
    {
       fec = new H1_FECollection(order = 1, dim);
    }
-   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec, 1, 0);
    HYPRE_Int size = fespace->GlobalTrueVSize();
    if (myid == 0)
    {
@@ -237,20 +239,52 @@ int main(int argc, char *argv[])
    //     local finite element solution on each processor.
    a->RecoverFEMSolution(X, *b, x);
 
+   std::string filename("nranks_");
+   filename += to_string(num_procs);
+   filename += ".gf";
+   {
+      double t1;
+      t1 = MPI_Wtime();
+      x.Save(filename.c_str(), nfiles);
+      double t2 = MPI_Wtime();
+
+      if (myid == 0)
+      {
+         err << "elapsed write time: " << t2 - t1 << endl;
+      }
+   }
+   {
+      double t1;
+      t1 = MPI_Wtime();
+      ParGridFunction new_x(fespace, filename.c_str());
+      double t2 = MPI_Wtime();
+      if (myid == 0)
+      {
+         err << "elapsed read time: " << t2 - t1 << endl;
+      }
+      // new_x -= x;
+      // out << "GF difference: " << new_x.Norml1() << endl;
+   }
+
    // 15. Save the refined mesh and the solution in parallel. This output can
    //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
    {
       ostringstream mesh_name, sol_name;
-      mesh_name << "mesh." << setfill('0') << setw(6) << myid;
-      sol_name << "sol." << setfill('0') << setw(6) << myid;
-
-      ofstream mesh_ofs(mesh_name.str().c_str());
-      mesh_ofs.precision(8);
-      pmesh->Print(mesh_ofs);
+      //mesh_name << "mesh." << setfill('0') << setw(6) << myid;
+      sol_name << "sol." << num_procs << setfill('0') << setw(6) << myid;
 
+      //ofstream mesh_ofs(mesh_name.str().c_str());
+      //mesh_ofs.precision(8);
+      //pmesh->Print(mesh_ofs);
+      double t1 = MPI_Wtime();
       ofstream sol_ofs(sol_name.str().c_str());
       sol_ofs.precision(8);
       x.Save(sol_ofs);
+      double t2 = MPI_Wtime();
+      if (myid == 0)
+      {
+         err << t2 - t1 << endl;
+      }
    }
 
    // 16. Send the solution by socket to a GLVis server.

examples/ex26.cpp

@@ -0,0 +1,255 @@
+//                                MFEM Example 26
+//
+// Compile with: make ex26
+//
+// Sample runs:  ex26 -m ../data/star.mesh
+//               ex26 -m ../data/fichera.mesh
+//               ex26 -m ../data/beam-hex.mesh
+//
+// Device sample runs:
+//               ex26 -d cuda
+//               ex26 -d raja-cuda
+//               ex26 -d occa-cuda
+//               ex26 -d raja-omp
+//               ex26 -d occa-omp
+//               ex26 -d ceed-cpu
+//               ex26 -d ceed-cuda
+//               ex26 -m ../data/beam-hex.mesh -d cuda
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Laplace problem
+//               -Delta u = 1 with homogeneous Dirichlet boundary conditions
+//               as in Example 1.
+//
+//               It highlights on the creation of a hierarchy of discretization
+//               spaces with partial assembly and the construction of an
+//               efficient multigrid preconditioner for the iterative solver.
+//
+//               We recommend viewing Example 1 before viewing this example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Class for constructing a multigrid preconditioner for the diffusion operator.
+// This example multigrid preconditioner class demonstrates the creation of the
+// diffusion bilinear forms and operators using partial assembly for all spaces
+// in the FiniteElementSpaceHierarchy. The preconditioner uses a CG solver on
+// the coarsest level and second order Chebyshev accelerated smoothers on the
+// other levels.
+class DiffusionMultigrid : public Multigrid
+{
+private:
+   ConstantCoefficient one;
+
+public:
+   // Constructs a diffusion multigrid for the given FiniteElementSpaceHierarchy
+   // and the array of essential boundaries
+   DiffusionMultigrid(FiniteElementSpaceHierarchy& fespaces, Array<int>& ess_bdr)
+      : Multigrid(fespaces), one(1.0)
+   {
+      ConstructCoarseOperatorAndSolver(fespaces.GetFESpaceAtLevel(0), ess_bdr);
+
+      for (int level = 1; level < fespaces.GetNumLevels(); ++level)
+      {
+         ConstructOperatorAndSmoother(fespaces.GetFESpaceAtLevel(level), ess_bdr);
+      }
+   }
+
+private:
+   void ConstructBilinearForm(FiniteElementSpace& fespace, Array<int>& ess_bdr)
+   {
+      BilinearForm* form = new BilinearForm(&fespace);
+      form->SetAssemblyLevel(AssemblyLevel::PARTIAL);
+      form->AddDomainIntegrator(new DiffusionIntegrator(one));
+      form->Assemble();
+      bfs.Append(form);
+
+      essentialTrueDofs.Append(new Array<int>());
+      fespace.GetEssentialTrueDofs(ess_bdr, *essentialTrueDofs.Last());
+   }
+
+   void ConstructCoarseOperatorAndSolver(FiniteElementSpace& coarse_fespace,
+                                         Array<int>& ess_bdr)
+   {
+      ConstructBilinearForm(coarse_fespace, ess_bdr);
+
+      OperatorPtr opr;
+      opr.SetType(Operator::ANY_TYPE);
+      bfs.Last()->FormSystemMatrix(*essentialTrueDofs.Last(), opr);
+      opr.SetOperatorOwner(false);
+
+      CGSolver* pcg = new CGSolver();
+      pcg->SetPrintLevel(-1);
+      pcg->SetMaxIter(200);
+      pcg->SetRelTol(sqrt(1e-4));
+      pcg->SetAbsTol(0.0);
+      pcg->SetOperator(*opr.Ptr());
+
+      AddLevel(opr.Ptr(), pcg, true, true);
+   }
+
+   void ConstructOperatorAndSmoother(FiniteElementSpace& fespace,
+                                     Array<int>& ess_bdr)
+   {
+      ConstructBilinearForm(fespace, ess_bdr);
+
+      OperatorPtr opr;
+      opr.SetType(Operator::ANY_TYPE);
+      bfs.Last()->FormSystemMatrix(*essentialTrueDofs.Last(), opr);
+      opr.SetOperatorOwner(false);
+
+      Vector diag(fespace.GetTrueVSize());
+      bfs.Last()->AssembleDiagonal(diag);
+
+      Solver* smoother = new OperatorChebyshevSmoother(opr.Ptr(), diag,
+                                                       *essentialTrueDofs.Last(), 2);
+      AddLevel(opr.Ptr(), smoother, true, true);
+   }
+};
+
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options.
+   const char *mesh_file = "../data/star.mesh";
+   int geometric_refinements = 0;
+   int order_refinements = 2;
+   const char *device_config = "cpu";
+   bool visualization = true;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&geometric_refinements, "-gr", "--geometric-refinements",
+                  "Number of geometric refinements done prior to order refinements.");
+   args.AddOption(&order_refinements, "-or", "--order-refinements",
+                  "Number of order refinements. Finest level in the hierarchy has order 2^{or}.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   // 2. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   device.Print();
+
+   // 3. Read the mesh from the given mesh file. We can handle triangular,
+   //    quadrilateral, tetrahedral, hexahedral, surface and volume meshes with
+   //    the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 4. Refine the mesh to increase the resolution. In this example we do
+   //    'ref_levels' of uniform refinement. We choose 'ref_levels' to be the
+   //    largest number that gives a final mesh with no more than 50,000
+   //    elements.
+   {
+      int ref_levels =
+         (int)floor(log(5000./mesh->GetNE())/log(2.)/dim);
+      for (int l = 0; l < ref_levels; l++)
+      {
+         mesh->UniformRefinement();
+      }
+   }
+
+   // 5. Define a finite element space hierarchy on the mesh. Here we use
+   //    continuous Lagrange finite elements. We start with order 1 on the
+   //    coarse level and geometrically refine the spaces by the specified
+   //    amount. Afterwards, we increase the order of the finite elements
+   //    by a factor of 2 for each additional level.
+   FiniteElementCollection *fec = new H1_FECollection(1, dim);
+   FiniteElementSpace *coarse_fespace = new FiniteElementSpace(mesh, fec);
+   FiniteElementSpaceHierarchy fespaces(mesh, coarse_fespace, true, true);
+
+   Array<FiniteElementCollection*> collections;
+   collections.Append(fec);
+   for (int level = 0; level < geometric_refinements; ++level)
+   {
+      fespaces.AddUniformlyRefinedLevel();
+   }
+   for (int level = 0; level < order_refinements; ++level)
+   {
+      collections.Append(new H1_FECollection(std::pow(2, level+1), dim));
+      fespaces.AddOrderRefinedLevel(collections.Last());
+   }
+
+   cout << "Number of finite element unknowns: "
+        << fespaces.GetFinestFESpace().GetTrueVSize() << endl;
+
+   // 6. Set up the linear form b(.) which corresponds to the right-hand side of
+   //    the FEM linear system, which in this case is (1,phi_i) where phi_i are
+   //    the basis functions in the finite element fespace.
+   LinearForm *b = new LinearForm(&fespaces.GetFinestFESpace());
+   ConstantCoefficient one(1.0);
+   b->AddDomainIntegrator(new DomainLFIntegrator(one));
+   b->Assemble();
+
+   // 7. Define the solution vector x as a finite element grid function
+   //    corresponding to fespace. Initialize x with initial guess of zero,
+   //    which satisfies the boundary conditions.
+   GridFunction x(&fespaces.GetFinestFESpace());
+   x = 0.0;
+
+   // 8. Create the multigrid operator using the previously created
+   //    FiniteElementSpaceHierarchy and additional boundary information. This operator
+   //    is then used to create the MultigridSolver as a preconditioner in the
+   //    iterative solver.
+   Array<int> ess_bdr(mesh->bdr_attributes.Max());
+   ess_bdr = 1;
+
+   DiffusionMultigrid M(fespaces, ess_bdr);
+   M.SetCycleType(Multigrid::CycleType::VCYCLE, 1, 1);
+
+   OperatorPtr A;
+   Vector B, X;
+   M.FormFineLinearSystem(x, *b, A, X, B);
+   cout << "Size of linear system: " << A->Height() << endl;
+
+   // 9. Solve the linear system A X = B.
+   PCG(*A, M, B, X, 1, 2000, 1e-12, 0.0);
+
+   // 10. Recover the solution as a finite element grid function.
+   M.RecoverFineFEMSolution(X, *b, x);
+
+   // 11. Save the refined mesh and the solution. This output can be viewed later
+   //     using GLVis: "glvis -m refined.mesh -g sol.gf".
+   ofstream mesh_ofs("refined.mesh");
+   mesh_ofs.precision(8);
+   fespaces.GetFinestFESpace().GetMesh()->Print(mesh_ofs);
+   ofstream sol_ofs("sol.gf");
+   sol_ofs.precision(8);
+   x.Save(sol_ofs);
+
+   // 12. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *fespaces.GetFinestFESpace().GetMesh() << x <<
+               flush;
+   }
+
+   // 13. Free the used memory.
+   delete b;
+   for (int level = 0; level < collections.Size(); ++level)
+   {
+      delete collections[level];
+   }
+
+   return 0;
+}

examples/ex26p.cpp

@@ -0,0 +1,317 @@
+//                       MFEM Example 26 - Parallel Version
+//
+// Compile with: make ex26p
+//
+// Sample runs:  mpirun -np 4 ex26p -m ../data/star.mesh
+//               mpirun -np 4 ex26p -m ../data/fichera.mesh
+//               mpirun -np 4 ex26p -m ../data/beam-hex.mesh
+//
+// Device sample runs:
+//               mpirun -np 4 ex26p -d cuda
+//               mpirun -np 4 ex26p -d occa-cuda
+//               mpirun -np 4 ex26p -d raja-omp
+//               mpirun -np 4 ex26p -d ceed-cpu
+//               mpirun -np 4 ex26p -d ceed-cuda
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Laplace problem
+//               -Delta u = 1 with homogeneous Dirichlet boundary conditions
+//               as in Example 1.
+//
+//               It highlights on the creation of a hierarchy of discretization
+//               spaces with partial assembly and the construction of an
+//               efficient multigrid preconditioner for the iterative solver.
+//
+//               We recommend viewing Example 1 before viewing this example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Class for constructing a multigrid preconditioner for the diffusion operator.
+// This example multigrid preconditioner class demonstrates the creation of the
+// parallel diffusion bilinear forms and operators using partial assembly for
+// all spaces except the coarsest one in the ParFiniteElementSpaceHierarchy.
+// The multigrid uses a PCG solver preconditioned with AMG on the coarsest level
+// and second order Chebyshev accelerated smoothers on the other levels.
+class DiffusionMultigrid : public Multigrid
+{
+private:
+   ConstantCoefficient one;
+   HypreBoomerAMG* amg;
+
+public:
+   // Constructs a diffusion multigrid for the ParFiniteElementSpaceHierarchy
+   // and the array of essential boundaries
+   DiffusionMultigrid(ParFiniteElementSpaceHierarchy& fespaces,
+                      Array<int>& ess_bdr)
+      : Multigrid(fespaces), one(1.0)
+   {
+      ConstructCoarseOperatorAndSolver(fespaces.GetFESpaceAtLevel(0), ess_bdr);
+
+      for (int level = 1; level < fespaces.GetNumLevels(); ++level)
+      {
+         ConstructOperatorAndSmoother(fespaces.GetFESpaceAtLevel(level), ess_bdr);
+      }
+   }
+
+   virtual ~DiffusionMultigrid()
+   {
+      delete amg;
+   }
+
+private:
+   void ConstructBilinearForm(ParFiniteElementSpace& fespace, Array<int>& ess_bdr,
+                              bool partial_assembly)
+   {
+      ParBilinearForm* form = new ParBilinearForm(&fespace);
+      if (partial_assembly)
+      {
+         form->SetAssemblyLevel(AssemblyLevel::PARTIAL);
+      }
+      form->AddDomainIntegrator(new DiffusionIntegrator(one));
+      form->Assemble();
+      bfs.Append(form);
+
+      essentialTrueDofs.Append(new Array<int>());
+      fespace.GetEssentialTrueDofs(ess_bdr, *essentialTrueDofs.Last());
+   }
+
+   void ConstructCoarseOperatorAndSolver(ParFiniteElementSpace& coarse_fespace,
+                                         Array<int>& ess_bdr)
+   {
+      ConstructBilinearForm(coarse_fespace, ess_bdr, false);
+
+      HypreParMatrix* hypreCoarseMat = new HypreParMatrix();
+      bfs.Last()->FormSystemMatrix(*essentialTrueDofs.Last(), *hypreCoarseMat);
+
+      amg = new HypreBoomerAMG(*hypreCoarseMat);
+      amg->SetPrintLevel(-1);
+
+      CGSolver* pcg = new CGSolver(MPI_COMM_WORLD);
+      pcg->SetPrintLevel(-1);
+      pcg->SetMaxIter(10);
+      pcg->SetRelTol(sqrt(1e-4));
+      pcg->SetAbsTol(0.0);
+      pcg->SetOperator(*hypreCoarseMat);
+      pcg->SetPreconditioner(*amg);
+
+      AddLevel(hypreCoarseMat, pcg, true, true);
+   }
+
+   void ConstructOperatorAndSmoother(ParFiniteElementSpace& fespace,
+                                     Array<int>& ess_bdr)
+   {
+      ConstructBilinearForm(fespace, ess_bdr, true);
+
+      OperatorPtr opr;
+      opr.SetType(Operator::ANY_TYPE);
+      bfs.Last()->FormSystemMatrix(*essentialTrueDofs.Last(), opr);
+      opr.SetOperatorOwner(false);
+
+      Vector diag(fespace.GetTrueVSize());
+      bfs.Last()->AssembleDiagonal(diag);
+
+      Solver* smoother = new OperatorChebyshevSmoother(opr.Ptr(), diag,
+                                                       *essentialTrueDofs.Last(), 2, fespace.GetParMesh()->GetComm());
+
+      AddLevel(opr.Ptr(), smoother, true, true);
+   }
+};
+
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   const char *mesh_file = "../data/star.mesh";
+   int geometric_refinements = 0;
+   int order_refinements = 2;
+   const char *device_config = "cpu";
+   bool visualization = true;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&geometric_refinements, "-gr", "--geometric-refinements",
+                  "Number of geometric refinements done prior to order refinements.");
+   args.AddOption(&order_refinements, "-or", "--order-refinements",
+                  "Number of order refinements. Finest level in the hierarchy has order 2^{or}.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   // 3. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   if (myid == 0) { device.Print(); }
+
+   // 4. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
+   //    and volume meshes with the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 5. Refine the serial mesh on all processors to increase the resolution. In
+   //    this example we do 'ref_levels' of uniform refinement. We choose
+   //    'ref_levels' to be the largest number that gives a final mesh with no
+   //    more than 1,000 elements.
+   {
+      int ref_levels =
+         (int)floor(log(1000./mesh->GetNE())/log(2.)/dim);
+      for (int l = 0; l < ref_levels; l++)
+      {
+         mesh->UniformRefinement();
+      }
+   }
+
+   // 6. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   {
+      int par_ref_levels = 2;
+      for (int l = 0; l < par_ref_levels; l++)
+      {
+         pmesh->UniformRefinement();
+      }
+   }
+
+   // 7. Define a parallel finite element space hierarchy on the parallel mesh.
+   //    Here we use continuous Lagrange finite elements. We start with order 1
+   //    on the coarse level and geometrically refine the spaces by the specified
+   //    amount. Afterwards, we increase the order of the finite elements by a
+   //    factor of 2 for each additional level.
+   FiniteElementCollection *fec = new H1_FECollection(1, dim);
+   ParFiniteElementSpace *coarse_fespace = new ParFiniteElementSpace(pmesh, fec);
+
+   Array<FiniteElementCollection*> collections;
+   collections.Append(fec);
+   ParFiniteElementSpaceHierarchy* fespaces = new ParFiniteElementSpaceHierarchy(
+      pmesh, coarse_fespace, true, true);
+   for (int level = 0; level < geometric_refinements; ++level)
+   {
+      fespaces->AddUniformlyRefinedLevel();
+   }
+   for (int level = 0; level < order_refinements; ++level)
+   {
+      collections.Append(new H1_FECollection(std::pow(2, level+1), dim));
+      fespaces->AddOrderRefinedLevel(collections.Last());
+   }
+
+   HYPRE_Int size = fespaces->GetFinestFESpace().GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of finite element unknowns: " << size << endl;
+   }
+
+   // 8. Set up the parallel linear form b(.) which corresponds to the
+   //    right-hand side of the FEM linear system, which in this case is
+   //    (1,phi_i) where phi_i are the basis functions in fespace.
+   ParLinearForm *b = new ParLinearForm(&fespaces->GetFinestFESpace());
+   ConstantCoefficient one(1.0);
+   b->AddDomainIntegrator(new DomainLFIntegrator(one));
+   b->Assemble();
+
+   // 9. Define the solution vector x as a parallel finite element grid function
+   //    corresponding to fespace. Initialize x with initial guess of zero,
+   //    which satisfies the boundary conditions.
+   ParGridFunction x(&fespaces->GetFinestFESpace());
+   x = 0.0;
+
+   // 10. Create the multigrid operator using the previously created parallel
+   //     FiniteElementSpaceHierarchy and additional boundary information. This operator
+   //     is then used to create the MultigridSolver as a preconditioner in the
+   //     iterative solver.
+   Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+   if (pmesh->bdr_attributes.Size())
+   {
+      ess_bdr = 1;
+   }
+
+   DiffusionMultigrid* M = new DiffusionMultigrid(*fespaces, ess_bdr);
+   M->SetCycleType(Multigrid::CycleType::VCYCLE, 1, 1);
+
+   OperatorPtr A;
+   Vector X, B;
+   M->FormFineLinearSystem(x, *b, A, X, B);
+
+   // 11. Solve the linear system A X = B.
+   CGSolver cg(MPI_COMM_WORLD);
+   cg.SetRelTol(1e-12);
+   cg.SetMaxIter(2000);
+   cg.SetPrintLevel(1);
+   cg.SetOperator(*A);
+   cg.SetPreconditioner(*M);
+   cg.Mult(B, X);
+
+   // 12. Recover the parallel grid function corresponding to X. This is the
+   //     local finite element solution on each processor.
+   M->RecoverFineFEMSolution(X, *b, x);
+
+   // 13. Save the refined mesh and the solution in parallel. This output can
+   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
+   {
+      ostringstream mesh_name, sol_name;
+      mesh_name << "mesh." << setfill('0') << setw(6) << myid;
+      sol_name << "sol." << setfill('0') << setw(6) << myid;
+
+      ofstream mesh_ofs(mesh_name.str().c_str());
+      mesh_ofs.precision(8);
+      fespaces->GetFinestFESpace().GetParMesh()->Print(mesh_ofs);
+
+      ofstream sol_ofs(sol_name.str().c_str());
+      sol_ofs.precision(8);
+      x.Save(sol_ofs);
+   }
+
+   // 14. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *fespaces->GetFinestFESpace().GetParMesh()
+               << x << flush;
+   }
+
+   // 15. Free the used memory.
+   delete M;
+   delete b;
+   delete fespaces;
+   for (int level = 0; level < collections.Size(); ++level)
+   {
+      delete collections[level];
+   }
+
+   MPI_Finalize();
+
+   return 0;
+}

examples/ex27.cpp

@@ -0,0 +1,736 @@
+//                       MFEM Example 27 - Serial Version
+//
+// Compile with: make ex27
+//
+// Sample runs:  ex27
+//               ex27 -dg
+//               ex27 -dg -dbc 8 -nbc -2
+//               ex27 -rbc-a 1 -rbc-b 8
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Laplace problem
+//               -Delta u = 0 with a variety of boundary conditions.
+//               Specifically, we discretize using a FE space of the specified
+//               order using a continuous or discontinuous space.  We then
+//               apply Dirichlet, Neumann (both homogeneous and inhomogeneous),
+//               Robin, and Periodic boundary conditions on different portions
+//               of a predefined mesh.
+//
+//               The predefined mesh consists of a rectangle with two
+//               holes removed (see below).  The narrow ends of the
+//               mesh are connected to form a Periodic boundary
+//               condition.  The lower edge (tagged with attribute 1)
+//               receives an inhomogeneous Neumann boundary condition.
+//               A Robin boundary condition is applied to upper edge
+//               (attribute 2).  The circular hole on the left
+//               (attribute 3) enforces a Dirichlet boundary
+//               condition.  Finally, a natural boundary condition, or
+//               homogeneous Neumann BC, is applied to the circular
+//               hole on the right (attribute 4).
+//
+//                  Attribute 3    ^ y  Attribute 2
+//                        \        |      /
+//                     +-----------+-----------+
+//                     |    \_     |     _     |
+//                     |    / \    |    / \    |
+//                  <--+---+---+---+---+---+---+--> x
+//                     |    \_/    |    \_/    |
+//                     |           |      \    |
+//                     +-----------+-----------+       (hole radii are
+//                          /      |        \            adjustable)
+//                  Attribute 1    v    Attribute 4
+//
+//               The boundary conditions are defined as (where u is
+//               the solution field):
+//                  Dirichlet: u = d
+//                  Neumann:   n.Grad(u) = g
+//                  Robin:     n.Grad(u) + a u = b
+//
+//               The user can adjust the values of 'd', 'g', 'a', and
+//               'b' with command line options.
+//
+//               This example highlights the differing implementations of
+//               boundary conditions with continuous and discontinuous Galerkin
+//               formulations of the Laplace problem.
+//
+//               We recommend viewing examples 1 and 14 before viewing this
+//               example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+static double a_ = 0.2;
+
+// Normal to hole with boundary attribute 4
+void n4Vec(const Vector &x, Vector &n) { n = x; n[0] -= 0.5; n /= -n.Norml2(); }
+
+Mesh * GenerateSerialMesh(int ref);
+
+// Compute the average value of alpha*n.Grad(sol) + beta*sol over the boundary
+// attributes marked in bdr_marker.  Also computes the L2 norm of
+// alpha*n.Grad(sol) + beta*sol - gamma  over the same boundary.
+double IntegrateBC(const GridFunction &sol, const Array<int> &bdr_marker,
+                   double alpha, double beta, double gamma,
+                   double &err);
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options.
+   int ser_ref_levels = 2;
+   int order = 1;
+   double sigma = -1.0;
+   double kappa = -1.0;
+   bool h1 = true;
+   bool visualization = true;
+
+   double mat_val = 1.0;
+   double dbc_val = 0.0;
+   double nbc_val = 1.0;
+   double rbc_a_val = 1.0; // du/dn + a * u = b
+   double rbc_b_val = 1.0;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&h1, "-h1", "--continuous", "-dg", "--discontinuous",
+                  "Select continuous \"H1\" or discontinuous \"DG\" basis.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&sigma, "-s", "--sigma",
+                  "One of the two DG penalty parameters, typically +1/-1."
+                  " See the documentation of class DGDiffusionIntegrator.");
+   args.AddOption(&kappa, "-k", "--kappa",
+                  "One of the two DG penalty parameters, should be positive."
+                  " Negative values are replaced with (order+1)^2.");
+   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
+                  "Number of times to refine the mesh uniformly in serial.");
+   args.AddOption(&mat_val, "-mat", "--material-value",
+                  "Constant value for material coefficient "
+                  "in the Laplace operator.");
+   args.AddOption(&dbc_val, "-dbc", "--dirichlet-value",
+                  "Constant value for Dirichlet Boundary Condition.");
+   args.AddOption(&nbc_val, "-nbc", "--neumann-value",
+                  "Constant value for Neumann Boundary Condition.");
+   args.AddOption(&rbc_a_val, "-rbc-a", "--robin-a-value",
+                  "Constant 'a' value for Robin Boundary Condition: "
+                  "du/dn + a * u = b.");
+   args.AddOption(&rbc_b_val, "-rbc-b", "--robin-b-value",
+                  "Constant 'b' value for Robin Boundary Condition: "
+                  "du/dn + a * u = b.");
+   args.AddOption(&a_, "-a", "--radius",
+                  "Radius of holes in the mesh.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(mfem::out);
+      return 1;
+   }
+   if (kappa < 0 && !h1)
+   {
+      kappa = (order+1)*(order+1);
+   }
+   args.PrintOptions(mfem::out);
+
+   if (a_ < 0.01)
+   {
+      mfem::out << "Hole radius too small, resetting to 0.01.\n";
+      a_ = 0.01;
+   }
+   if (a_ > 0.49)
+   {
+      mfem::out << "Hole radius too large, resetting to 0.49.\n";
+      a_ = 0.49;
+   }
+
+   // 2. Construct the (serial) mesh and refine it if requested.
+   Mesh *mesh = GenerateSerialMesh(ser_ref_levels);
+   int dim = mesh->Dimension();
+
+   // 3. Define a finite element space on the serial mesh.  Here we
+   //    use either continuous Lagrange finite elements or discontinuous
+   //    Galerkin finite elements of the specified order.
+   FiniteElementCollection *fec =
+      h1 ? (FiniteElementCollection*)new H1_FECollection(order, dim) :
+      (FiniteElementCollection*)new DG_FECollection(order, dim);
+   FiniteElementSpace fespace(mesh, fec);
+   int size = fespace.GetTrueVSize();
+   mfem::out << "Number of finite element unknowns: " << size << endl;
+
+   // 4. Create "marker arrays" to define the portions of the boundary
+   //    associated with each type of boundary condition.  These arrays
+   //    have an entry corresponding to each boundary attribute.
+   //    Placing a '1' in entry i marks attribute i+1 as being
+   //    active, '0' is inactive.
+   Array<int> nbc_bdr(mesh->bdr_attributes.Max());
+   Array<int> rbc_bdr(mesh->bdr_attributes.Max());
+   Array<int> dbc_bdr(mesh->bdr_attributes.Max());
+
+   nbc_bdr = 0; nbc_bdr[0] = 1;
+   rbc_bdr = 0; rbc_bdr[1] = 1;
+   dbc_bdr = 0; dbc_bdr[2] = 1;
+
+   Array<int> ess_tdof_list(0);
+   if (h1 && mesh->bdr_attributes.Size())
+   {
+      // For a continuous basis the linear system must be modifed to enforce
+      // an essential (Dirichlet) boundary condition.  In the DG case this is
+      // not necessary as the boundary condition will only be enforced weakly.
+      fespace.GetEssentialTrueDofs(dbc_bdr, ess_tdof_list);
+   }
+
+   // 5. Setup the various coefficients needed for the Laplace operator and
+   //    the various boundary conditions.  In general these coefficients could
+   //    be functions of position but here we use only constants.
+   ConstantCoefficient matCoef(mat_val);
+   ConstantCoefficient dbcCoef(dbc_val);
+   ConstantCoefficient nbcCoef(nbc_val);
+   ConstantCoefficient rbcACoef(rbc_a_val);
+   ConstantCoefficient rbcBCoef(rbc_b_val);
+
+   // Since the n.Grad(u) terms arise by integrating -Div(m Grad(u)) by parts
+   // we must introduce the coefficient 'm' into the boundary conditions.
+   // Therefore, in the case of the Neumann BC, we actually enforce
+   // m n.Grad(u) = m g rather than simply n.Grad(u) = g.
+   ProductCoefficient m_nbcCoef(matCoef, nbcCoef);
+   ProductCoefficient m_rbcACoef(matCoef, rbcACoef);
+   ProductCoefficient m_rbcBCoef(matCoef, rbcBCoef);
+
+   // 6. Define the solution vector u as a finite element grid function
+   //    corresponding to fespace. Initialize u with initial guess of zero.
+   GridFunction u(&fespace);
+   u = 0.0;
+
+   // 7. Set up the bilinear form a(.,.) on the finite element space
+   //    corresponding to the Laplacian operator -Delta, by adding the Diffusion
+   //    domain integrator.
+   BilinearForm a(&fespace);
+   a.AddDomainIntegrator(new DiffusionIntegrator(matCoef));
+   if (h1)
+   {
+      // Add a Mass integrator on the Robin boundary
+      a.AddBoundaryIntegrator(new MassIntegrator(m_rbcACoef), rbc_bdr);
+   }
+   else
+   {
+      // Add the interfacial portion of the Lapalce operator
+      a.AddInteriorFaceIntegrator(new DGDiffusionIntegrator(matCoef,
+                                                            sigma, kappa));
+
+      // Counteract the n.Grad(u) term on the Dirichlet portion of the boundary
+      a.AddBdrFaceIntegrator(new DGDiffusionIntegrator(matCoef, sigma, kappa),
+                             dbc_bdr);
+
+      // Augment the n.Grad(u) term with a*u on the Robin portion of boundary
+      a.AddBdrFaceIntegrator(new BoundaryMassIntegrator(m_rbcACoef),
+                             rbc_bdr);
+   }
+   a.Assemble();
+
+   // 8. Assemble the linear form for the right hand side vector.
+   LinearForm b(&fespace);
+
+   if (h1)
+   {
+      // Set the Dirchlet values in the solution vector
+      u.ProjectBdrCoefficient(dbcCoef, dbc_bdr);
+
+      // Add the desired value for n.Grad(u) on the Neumann boundary
+      b.AddBoundaryIntegrator(new BoundaryLFIntegrator(m_nbcCoef), nbc_bdr);
+
+      // Add the desired value for n.Grad(u) + a*u on the Robin boundary
+      b.AddBoundaryIntegrator(new BoundaryLFIntegrator(m_rbcBCoef), rbc_bdr);
+   }
+   else
+   {
+      // Add the desired value for the Dirchlet boundary
+      b.AddBdrFaceIntegrator(new DGDirichletLFIntegrator(dbcCoef, matCoef,
+                                                         sigma, kappa),
+                             dbc_bdr);
+
+      // Add the desired value for n.Grad(u) on the Neumann boundary
+      b.AddBdrFaceIntegrator(new BoundaryLFIntegrator(m_nbcCoef),
+                             nbc_bdr);
+
+      // Add the desired value for n.Grad(u) + a*u on the Robin boundary
+      b.AddBdrFaceIntegrator(new BoundaryLFIntegrator(m_rbcBCoef),
+                             rbc_bdr);
+   }
+   b.Assemble();
+
+   // 9. Construct the linear system.
+   OperatorPtr A;
+   Vector B, X;
+   a.FormLinearSystem(ess_tdof_list, u, b, A, X, B);
+
+#ifndef MFEM_USE_SUITESPARSE
+   // 10. Define a simple symmetric Gauss-Seidel preconditioner and use it to
+   //     solve the system AX=B with PCG in the symmetric case, and GMRES in the
+   //     non-symmetric one.
+   {
+      GSSmoother M((SparseMatrix&)(*A));
+      if (sigma == -1.0)
+      {
+         PCG(*A, M, B, X, 1, 500, 1e-12, 0.0);
+      }
+      else
+      {
+         GMRES(*A, M, B, X, 1, 500, 10, 1e-12, 0.0);
+      }
+   }
+#else
+   // 11. If MFEM was compiled with SuiteSparse, use UMFPACK to solve the
+   //     system.
+   UMFPackSolver umf_solver;
+   umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+   umf_solver.SetOperator(*A);
+   umf_solver.Mult(B, X);
+#endif
+
+   // 12. Recover the grid function corresponding to U. This is the
+   //     local finite element solution.
+   a.RecoverFEMSolution(X, b, u);
+
+   // 13. Build a mass matrix to help solve for n.Grad(u) where 'n' is
+   //     a surface normal.
+   BilinearForm m(&fespace);
+   m.AddDomainIntegrator(new MassIntegrator);
+   m.Assemble();
+
+   ess_tdof_list.SetSize(0);
+   OperatorPtr M;
+   m.FormSystemMatrix(ess_tdof_list, M);
+
+   // 14. Compute the various boundary integrals.
+   mfem::out << endl
+             << "Verifying boundary conditions" << endl
+             << "=============================" << endl;
+   {
+      // Integrate the solution on the Dirichlet boundary and compare
+      // to the expected value.
+      double err, avg = IntegrateBC(u, dbc_bdr, 0.0, 1.0, dbc_val, err);
+
+      bool hom_dbc = (dbc_val == 0.0);
+      err /=  hom_dbc ? 1.0 : fabs(dbc_val);
+      mfem::out << "Average of solution on Gamma_dbc:\t"
+                << avg << ", \t"
+                << (hom_dbc ? "absolute" : "relative")
+                << " error " << err << endl;
+   }
+   {
+      // Integrate n.Grad(u) on the inhomogeneous Neumann boundary and
+      // compare to the expected value.
+      double err, avg = IntegrateBC(u, nbc_bdr, 1.0, 0.0, nbc_val, err);
+
+      bool hom_nbc = (nbc_val == 0.0);
+      err /=  hom_nbc ? 1.0 : fabs(nbc_val);
+      mfem::out << "Average of n.Grad(u) on Gamma_nbc:\t"
+                << avg << ", \t"
+                << (hom_nbc ? "absolute" : "relative")
+                << " error " << err << endl;
+   }
+   {
+      // Integrate n.Grad(u) on the homogeneous Neumann boundary and compare
+      // to the expected value of zero.
+      Array<int> nbc0_bdr(mesh->bdr_attributes.Max());
+      nbc0_bdr = 0;
+      nbc0_bdr[3] = 1;
+
+      double err, avg = IntegrateBC(u, nbc0_bdr, 1.0, 0.0, 0.0, err);
+
+      bool hom_nbc = true;
+      mfem::out << "Average of n.Grad(u) on Gamma_nbc0:\t"
+                << avg << ", \t"
+                << (hom_nbc ? "absolute" : "relative")
+                << " error " << err << endl;
+   }
+   {
+      // Integrate n.Grad(u) + a * u on the Robin boundary and compare to
+      // the expected value.
+      double err, avg = IntegrateBC(u, rbc_bdr, 1.0, rbc_a_val, rbc_b_val, err);
+
+      bool hom_rbc = (rbc_b_val == 0.0);
+      err /=  hom_rbc ? 1.0 : fabs(rbc_b_val);
+      mfem::out << "Average of n.Grad(u)+a*u on Gamma_rbc:\t"
+                << avg << ", \t"
+                << (hom_rbc ? "absolute" : "relative")
+                << " error " << err << endl;
+   }
+
+   // 15. Save the refined mesh and the solution. This output can be viewed
+   //     later using GLVis: "glvis -m refined.mesh -g sol.gf".
+   {
+      ofstream mesh_ofs("refined.mesh");
+      mesh_ofs.precision(8);
+      mesh->Print(mesh_ofs);
+      ofstream sol_ofs("sol.gf");
+      sol_ofs.precision(8);
+      u.Save(sol_ofs);
+   }
+
+   // 16. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      string title_str = h1 ? "H1" : "DG";
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *mesh << u
+               << "window_title '" << title_str << " Solution'"
+               << " keys 'mmc'" << flush;
+   }
+
+   // 17. Free the used memory.
+   delete fec;
+   delete mesh;
+
+   return 0;
+}
+
+void quad_trans(double u, double v, double &x, double &y, bool log = false)
+{
+   double a = a_; // Radius of disc
+
+   double d = 4.0 * a * (M_SQRT2 - 2.0 * a) * (1.0 - 2.0 * v);
+
+   double v0 = (1.0 + M_SQRT2) * (M_SQRT2 * a - 2.0 * v) *
+               ((4.0 - 3 * M_SQRT2) * a +
+                (8.0 * (M_SQRT2 - 1.0) * a - 2.0) * v) / d;
+
+   double r = 2.0 * ((M_SQRT2 - 1.0) * a * a * (1.0 - 4.0 *v) +
+                     2.0 * (1.0 + M_SQRT2 *
+                            (1.0 + 2.0 * (2.0 * a - M_SQRT2 - 1.0) * a)) * v * v
+                    ) / d;
+
+   double t = asin(v / r) * u / v;
+   if (log)
+   {
+      mfem::out << "u, v, r, v0, t "
+                << u << " " << v << " " << r << " " << v0 << " " << t
+                << endl;
+   }
+   x = r * sin(t);
+   y = r * cos(t) - v0;
+}
+
+void trans(const Vector &u, Vector &x)
+{
+   double tol = 1e-4;
+
+   if (u[1] > 0.5 - tol || u[1] < -0.5 + tol)
+   {
+      x = u;
+      return;
+   }
+   if (u[0] > 1.0 - tol || u[0] < -1.0 + tol || fabs(u[0]) < tol)
+   {
+      x = u;
+      return;
+   }
+
+   if (u[0] > 0.0)
+   {
+      if (u[1] > fabs(u[0] - 0.5))
+      {
+         quad_trans(u[0] - 0.5, u[1], x[0], x[1]);
+         x[0] += 0.5;
+         return;
+      }
+      if (u[1] < -fabs(u[0] - 0.5))
+      {
+         quad_trans(u[0] - 0.5, -u[1], x[0], x[1]);
+         x[0] += 0.5;
+         x[1] *= -1.0;
+         return;
+      }
+      if (u[0] - 0.5 > fabs(u[1]))
+      {
+         quad_trans(u[1], u[0] - 0.5, x[1], x[0]);
+         x[0] += 0.5;
+         return;
+      }
+      if (u[0] - 0.5 < -fabs(u[1]))
+      {
+         quad_trans(u[1], 0.5 - u[0], x[1], x[0]);
+         x[0] *= -1.0;
+         x[0] += 0.5;
+         return;
+      }
+   }
+   else
+   {
+      if (u[1] > fabs(u[0] + 0.5))
+      {
+         quad_trans(u[0] + 0.5, u[1], x[0], x[1]);
+         x[0] -= 0.5;
+         return;
+      }
+      if (u[1] < -fabs(u[0] + 0.5))
+      {
+         quad_trans(u[0] + 0.5, -u[1], x[0], x[1]);
+         x[0] -= 0.5;
+         x[1] *= -1.0;
+         return;
+      }
+      if (u[0] + 0.5 > fabs(u[1]))
+      {
+         quad_trans(u[1], u[0] + 0.5, x[1], x[0]);
+         x[0] -= 0.5;
+         return;
+      }
+      if (u[0] + 0.5 < -fabs(u[1]))
+      {
+         quad_trans(u[1], -0.5 - u[0], x[1], x[0]);
+         x[0] *= -1.0;
+         x[0] -= 0.5;
+         return;
+      }
+   }
+   x = u;
+}
+
+Mesh * GenerateSerialMesh(int ref)
+{
+   Mesh * mesh = new Mesh(2, 29, 16, 24, 2);
+
+   int vi[4];
+
+   for (int i=0; i<2; i++)
+   {
+      int o = 13 * i;
+      vi[0] = o + 0; vi[1] = o + 3; vi[2] = o + 4; vi[3] = o + 1;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 1; vi[1] = o + 4; vi[2] = o + 5; vi[3] = o + 2;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 5; vi[1] = o + 8; vi[2] = o + 9; vi[3] = o + 2;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 8; vi[1] = o + 12; vi[2] = o + 15; vi[3] = o + 9;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 11; vi[1] = o + 14; vi[2] = o + 15; vi[3] = o + 12;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 10; vi[1] = o + 13; vi[2] = o + 14; vi[3] = o + 11;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 6; vi[1] = o + 13; vi[2] = o + 10; vi[3] = o + 7;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 0; vi[1] = o + 6; vi[2] = o + 7; vi[3] = o + 3;
+      mesh->AddQuad(vi);
+   }
+
+   vi[0] =  0; vi[1] =  6; mesh->AddBdrSegment(vi, 1);
+   vi[0] =  6; vi[1] = 13; mesh->AddBdrSegment(vi, 1);
+   vi[0] = 13; vi[1] = 19; mesh->AddBdrSegment(vi, 1);
+   vi[0] = 19; vi[1] = 26; mesh->AddBdrSegment(vi, 1);
+
+   vi[0] = 28; vi[1] = 22; mesh->AddBdrSegment(vi, 2);
+   vi[0] = 22; vi[1] = 15; mesh->AddBdrSegment(vi, 2);
+   vi[0] = 15; vi[1] =  9; mesh->AddBdrSegment(vi, 2);
+   vi[0] =  9; vi[1] =  2; mesh->AddBdrSegment(vi, 2);
+
+   for (int i=0; i<2; i++)
+   {
+      int o = 13 * i;
+      vi[0] = o +  7; vi[1] = o +  3; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o + 10; vi[1] = o +  7; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o + 11; vi[1] = o + 10; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o + 12; vi[1] = o + 11; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o +  8; vi[1] = o + 12; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o +  5; vi[1] = o +  8; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o +  4; vi[1] = o +  5; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o +  3; vi[1] = o +  4; mesh->AddBdrSegment(vi, 3 + i);
+   }
+
+   double d[2];
+   double a = a_ / M_SQRT2;
+
+   d[0] = -1.0; d[1] = -0.5; mesh->AddVertex(d);
+   d[0] = -1.0; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] = -1.0; d[1] =  0.5; mesh->AddVertex(d);
+
+   d[0] = -0.5 - a; d[1] =   -a; mesh->AddVertex(d);
+   d[0] = -0.5 - a; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] = -0.5 - a; d[1] =    a; mesh->AddVertex(d);
+
+   d[0] = -0.5; d[1] = -0.5; mesh->AddVertex(d);
+   d[0] = -0.5; d[1] =   -a; mesh->AddVertex(d);
+   d[0] = -0.5; d[1] =    a; mesh->AddVertex(d);
+   d[0] = -0.5; d[1] =  0.5; mesh->AddVertex(d);
+
+   d[0] = -0.5 + a; d[1] =   -a; mesh->AddVertex(d);
+   d[0] = -0.5 + a; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] = -0.5 + a; d[1] =    a; mesh->AddVertex(d);
+
+   d[0] =  0.0; d[1] = -0.5; mesh->AddVertex(d);
+   d[0] =  0.0; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] =  0.0; d[1] =  0.5; mesh->AddVertex(d);
+
+   d[0] =  0.5 - a; d[1] =   -a; mesh->AddVertex(d);
+   d[0] =  0.5 - a; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] =  0.5 - a; d[1] =    a; mesh->AddVertex(d);
+
+   d[0] =  0.5; d[1] = -0.5; mesh->AddVertex(d);
+   d[0] =  0.5; d[1] =   -a; mesh->AddVertex(d);
+   d[0] =  0.5; d[1] =    a; mesh->AddVertex(d);
+   d[0] =  0.5; d[1] =  0.5; mesh->AddVertex(d);
+
+   d[0] =  0.5 + a; d[1] =   -a; mesh->AddVertex(d);
+   d[0] =  0.5 + a; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] =  0.5 + a; d[1] =    a; mesh->AddVertex(d);
+
+   d[0] =  1.0; d[1] = -0.5; mesh->AddVertex(d);
+   d[0] =  1.0; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] =  1.0; d[1] =  0.5; mesh->AddVertex(d);
+
+   mesh->FinalizeTopology();
+
+   mesh->SetCurvature(1, true);
+
+   // Stitch the ends of the stack together
+   {
+      Array<int> v2v(mesh->GetNV());
+      for (int i = 0; i < v2v.Size() - 3; i++)
+      {
+         v2v[i] = i;
+      }
+      // identify vertices on the narrow ends of the rectangle
+      v2v[v2v.Size() - 3] = 0;
+      v2v[v2v.Size() - 2] = 1;
+      v2v[v2v.Size() - 1] = 2;
+
+      // renumber elements
+      for (int i = 0; i < mesh->GetNE(); i++)
+      {
+         Element *el = mesh->GetElement(i);
+         int *v = el->GetVertices();
+         int nv = el->GetNVertices();
+         for (int j = 0; j < nv; j++)
+         {
+            v[j] = v2v[v[j]];
+         }
+      }
+      // renumber boundary elements
+      for (int i = 0; i < mesh->GetNBE(); i++)
+      {
+         Element *el = mesh->GetBdrElement(i);
+         int *v = el->GetVertices();
+         int nv = el->GetNVertices();
+         for (int j = 0; j < nv; j++)
+         {
+            v[j] = v2v[v[j]];
+         }
+      }
+      mesh->RemoveUnusedVertices();
+      mesh->RemoveInternalBoundaries();
+   }
+   mesh->SetCurvature(3, true);
+
+   for (int l = 0; l < ref; l++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   mesh->Transform(trans);
+
+   return mesh;
+}
+
+double IntegrateBC(const GridFunction &x, const Array<int> &bdr,
+                   double alpha, double beta, double gamma,
+                   double &err)
+{
+   double nrm = 0.0;
+   double avg = 0.0;
+   err = 0.0;
+
+   const bool a_is_zero = alpha == 0.0;
+   const bool b_is_zero = beta == 0.0;
+
+   const FiniteElementSpace &fes = *x.FESpace();
+   MFEM_ASSERT(fes.GetVDim() == 1, "");
+   Mesh &mesh = *fes.GetMesh();
+   Vector shape, loc_dofs, w_nor;
+   DenseMatrix dshape;
+   Array<int> dof_ids;
+   for (int i = 0; i < mesh.GetNBE(); i++)
+   {
+      if (bdr[mesh.GetBdrAttribute(i)-1] == 0) { continue; }
+
+      FaceElementTransformations *FTr = mesh.GetBdrFaceTransformations(i);
+      if (FTr == nullptr) { continue; }
+
+      const FiniteElement &fe = *fes.GetFE(FTr->Elem1No);
+      MFEM_ASSERT(fe.GetMapType() == FiniteElement::VALUE, "");
+      const int int_order = 2*fe.GetOrder() + 3;
+      const IntegrationRule &ir = IntRules.Get(FTr->FaceGeom, int_order);
+
+      fes.GetElementDofs(FTr->Elem1No, dof_ids);
+      x.GetSubVector(dof_ids, loc_dofs);
+      if (!a_is_zero)
+      {
+         const int sdim = FTr->Face->GetSpaceDim();
+         w_nor.SetSize(sdim);
+         dshape.SetSize(fe.GetDof(), sdim);
+      }
+      if (!b_is_zero)
+      {
+         shape.SetSize(fe.GetDof());
+      }
+      for (int j = 0; j < ir.GetNPoints(); j++)
+      {
+         const IntegrationPoint &ip = ir.IntPoint(j);
+         IntegrationPoint eip;
+         FTr->Loc1.Transform(ip, eip);
+         FTr->Face->SetIntPoint(&ip);
+         double face_weight = FTr->Face->Weight();
+         double val = 0.0;
+         if (!a_is_zero)
+         {
+            FTr->Elem1->SetIntPoint(&eip);
+            fe.CalcPhysDShape(*FTr->Elem1, dshape);
+            CalcOrtho(FTr->Face->Jacobian(), w_nor);
+            val += alpha * dshape.InnerProduct(w_nor, loc_dofs) / face_weight;
+         }
+         if (!b_is_zero)
+         {
+            fe.CalcShape(eip, shape);
+            val += beta * (shape * loc_dofs);
+         }
+
+         // Measure the length of the boundary
+         nrm += ip.weight * face_weight;
+
+         // Integrate alpha * n.Grad(x) + beta * x
+         avg += val * ip.weight * face_weight;
+
+         // Integrate |alpha * n.Grad(x) + beta * x - gamma|^2
+         val -= gamma;
+         err += (val*val) * ip.weight * face_weight;
+      }
+   }
+
+   // Normalize by the length of the boundary
+   if (std::abs(nrm) > 0.0)
+   {
+      err /= nrm;
+      avg /= nrm;
+   }
+
+   // Compute l2 norm of the error in the boundary condition
+   // (negative quadrature weights may produce negative 'err')
+   err = (err >= 0.0) ? sqrt(err) : -sqrt(-err);
+
+   // Return the average value of alpha * n.Grad(x) + beta * x
+   return avg;
+}

examples/ex27p.cpp

@@ -0,0 +1,773 @@
+//                       MFEM Example 27 - Parallel Version
+//
+// Compile with: make ex27p
+//
+// Sample runs:  mpirun -np 4 ex27p
+//               mpirun -np 4 ex27p -dg
+//               mpirun -np 4 ex27p -dg -dbc 8 -nbc -2
+//               mpirun -np 4 ex27p -rbc-a 1 -rbc-b 8
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Laplace problem
+//               -Delta u = 0 with a variety of boundary conditions.
+//               Specifically, we discretize using a FE space of the specified
+//               order using a continuous or discontinuous space.  We then
+//               apply Dirichlet, Neumann (both homogeneous and inhomogeneous),
+//               Robin, and Periodic boundary conditions on different portions
+//               of a predefined mesh.
+//
+//               The predefined mesh consists of a rectangle with two
+//               holes removed (see below).  The narrow ends of the
+//               mesh are connected to form a Periodic boundary
+//               condition.  The lower edge (tagged with attribute 1)
+//               receives an inhomogeneous Neumann boundary condition.
+//               A Robin boundary condition is applied to upper edge
+//               (attribute 2).  The circular hole on the left
+//               (attribute 3) enforces a Dirichlet boundary
+//               condition.  Finally, a natural boundary condition, or
+//               homogeneous Neumann BC, is applied to the circular
+//               hole on the right (attribute 4).
+//
+//                  Attribute 3    ^ y  Attribute 2
+//                        \        |      /
+//                     +-----------+-----------+
+//                     |    \_     |     _     |
+//                     |    / \    |    / \    |
+//                  <--+---+---+---+---+---+---+--> x
+//                     |    \_/    |    \_/    |
+//                     |           |      \    |
+//                     +-----------+-----------+       (hole radii are
+//                          /      |        \            adjustable)
+//                  Attribute 1    v    Attribute 4
+//
+//               The boundary conditions are defined as (where u is
+//               the solution field):
+//                  Dirichlet: u = d
+//                  Neumann:   n.Grad(u) = g
+//                  Robin:     n.Grad(u) + a u = b
+//
+//               The user can adjust the values of 'd', 'g', 'a', and
+//               'b' with command line options.
+//
+//               This example highlights the differing implementations of
+//               boundary conditions with continuous and discontinuous Galerkin
+//               formulations of the Laplace problem.
+//
+//               We recommend viewing examples 1 and 14 before viewing this
+//               example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+static double a_ = 0.2;
+
+// Normal to hole with boundary attribute 4
+void n4Vec(const Vector &x, Vector &n) { n = x; n[0] -= 0.5; n /= -n.Norml2(); }
+
+Mesh * GenerateSerialMesh(int ref);
+
+// Compute the average value of alpha*n.Grad(sol) + beta*sol over the boundary
+// attributes marked in bdr_marker.  Also computes the L2 norm of
+// alpha*n.Grad(sol) + beta*sol - gamma  over the same boundary.
+double IntegrateBC(const ParGridFunction &sol, const Array<int> &bdr_marker,
+                   double alpha, double beta, double gamma,
+                   double &err);
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   MPI_Session mpi;
+   if (!mpi.Root()) { mfem::out.Disable(); mfem::err.Disable(); }
+
+   // 2. Parse command-line options.
+   int ser_ref_levels = 2;
+   int par_ref_levels = 1;
+   int order = 1;
+   double sigma = -1.0;
+   double kappa = -1.0;
+   bool h1 = true;
+   bool visualization = true;
+
+   double mat_val = 1.0;
+   double dbc_val = 0.0;
+   double nbc_val = 1.0;
+   double rbc_a_val = 1.0; // du/dn + a * u = b
+   double rbc_b_val = 1.0;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&h1, "-h1", "--continuous", "-dg", "--discontinuous",
+                  "Select continuous \"H1\" or discontinuous \"DG\" basis.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&sigma, "-s", "--sigma",
+                  "One of the two DG penalty parameters, typically +1/-1."
+                  " See the documentation of class DGDiffusionIntegrator.");
+   args.AddOption(&kappa, "-k", "--kappa",
+                  "One of the two DG penalty parameters, should be positive."
+                  " Negative values are replaced with (order+1)^2.");
+   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
+                  "Number of times to refine the mesh uniformly in serial.");
+   args.AddOption(&par_ref_levels, "-rp", "--refine-parallel",
+                  "Number of times to refine the mesh uniformly in parallel.");
+   args.AddOption(&mat_val, "-mat", "--material-value",
+                  "Constant value for material coefficient "
+                  "in the Laplace operator.");
+   args.AddOption(&dbc_val, "-dbc", "--dirichlet-value",
+                  "Constant value for Dirichlet Boundary Condition.");
+   args.AddOption(&nbc_val, "-nbc", "--neumann-value",
+                  "Constant value for Neumann Boundary Condition.");
+   args.AddOption(&rbc_a_val, "-rbc-a", "--robin-a-value",
+                  "Constant 'a' value for Robin Boundary Condition: "
+                  "du/dn + a * u = b.");
+   args.AddOption(&rbc_b_val, "-rbc-b", "--robin-b-value",
+                  "Constant 'b' value for Robin Boundary Condition: "
+                  "du/dn + a * u = b.");
+   args.AddOption(&a_, "-a", "--radius",
+                  "Radius of holes in the mesh.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(mfem::out);
+      return 1;
+   }
+   if (kappa < 0 && !h1)
+   {
+      kappa = (order+1)*(order+1);
+   }
+   args.PrintOptions(mfem::out);
+
+   if (a_ < 0.01)
+   {
+      mfem::out << "Hole radius too small, resetting to 0.01.\n";
+      a_ = 0.01;
+   }
+   if (a_ > 0.49)
+   {
+      mfem::out << "Hole radius too large, resetting to 0.49.\n";
+      a_ = 0.49;
+   }
+
+   // 3. Construct the (serial) mesh and refine it if requested.
+   Mesh *mesh = GenerateSerialMesh(ser_ref_levels);
+   int dim = mesh->Dimension();
+
+   // 4. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted.
+   ParMesh pmesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   for (int l = 0; l < par_ref_levels; l++)
+   {
+      pmesh.UniformRefinement();
+   }
+
+   // 5. Define a parallel finite element space on the parallel mesh. Here we
+   //    use either continuous Lagrange finite elements or discontinuous
+   //    Galerkin finite elements of the specified order.
+   FiniteElementCollection *fec =
+      h1 ? (FiniteElementCollection*)new H1_FECollection(order, dim) :
+      (FiniteElementCollection*)new DG_FECollection(order, dim);
+   ParFiniteElementSpace fespace(&pmesh, fec);
+   HYPRE_Int size = fespace.GlobalTrueVSize();
+   mfem::out << "Number of finite element unknowns: " << size << endl;
+
+   // 6. Create "marker arrays" to define the portions of the boundary
+   //    associated with each type of boundary condition.  These arrays
+   //    have an entry corresponding to each boundary attribute.
+   //    Placing a '1' in entry i marks attribute i+1 as being
+   //    active, '0' is inactive.
+   Array<int> nbc_bdr(pmesh.bdr_attributes.Max());
+   Array<int> rbc_bdr(pmesh.bdr_attributes.Max());
+   Array<int> dbc_bdr(pmesh.bdr_attributes.Max());
+
+   nbc_bdr = 0; nbc_bdr[0] = 1;
+   rbc_bdr = 0; rbc_bdr[1] = 1;
+   dbc_bdr = 0; dbc_bdr[2] = 1;
+
+   Array<int> ess_tdof_list(0);
+   if (h1 && pmesh.bdr_attributes.Size())
+   {
+      // For a continuous basis the linear system must be modifed to enforce
+      // an essential (Dirichlet) boundary condition.  In the DG case this is
+      // not necessary as the boundary condition will only be enforced weakly.
+      fespace.GetEssentialTrueDofs(dbc_bdr, ess_tdof_list);
+   }
+
+   // 7. Setup the various coefficients needed for the Laplace operator and
+   //    the various boundary conditions.  In general these coefficients could
+   //    be functions of position but here we use only constants.
+   ConstantCoefficient matCoef(mat_val);
+   ConstantCoefficient dbcCoef(dbc_val);
+   ConstantCoefficient nbcCoef(nbc_val);
+   ConstantCoefficient rbcACoef(rbc_a_val);
+   ConstantCoefficient rbcBCoef(rbc_b_val);
+
+   // Since the n.Grad(u) terms arise by integrating -Div(m Grad(u)) by parts
+   // we must introduce the coefficient 'm' into the boundary conditions.
+   // Therefore, in the case of the Neumann BC, we actually enforce
+   // m n.Grad(u) = m g rather than simply n.Grad(u) = g.
+   ProductCoefficient m_nbcCoef(matCoef, nbcCoef);
+   ProductCoefficient m_rbcACoef(matCoef, rbcACoef);
+   ProductCoefficient m_rbcBCoef(matCoef, rbcBCoef);
+
+   // 8. Define the solution vector u as a parallel finite element grid function
+   //    corresponding to fespace. Initialize u with initial guess of zero.
+   ParGridFunction u(&fespace);
+   u = 0.0;
+
+   // 9. Set up the parallel bilinear form a(.,.) on the finite element space
+   //    corresponding to the Laplacian operator -Delta, by adding the Diffusion
+   //    domain integrator.
+   ParBilinearForm a(&fespace);
+   a.AddDomainIntegrator(new DiffusionIntegrator(matCoef));
+   if (h1)
+   {
+      // Add a Mass integrator on the Robin boundary
+      a.AddBoundaryIntegrator(new MassIntegrator(m_rbcACoef), rbc_bdr);
+   }
+   else
+   {
+      // Add the interfacial portion of the Lapalce operator
+      a.AddInteriorFaceIntegrator(new DGDiffusionIntegrator(matCoef,
+                                                            sigma, kappa));
+
+      // Counteract the n.Grad(u) term on the Dirichlet portion of the boundary
+      a.AddBdrFaceIntegrator(new DGDiffusionIntegrator(matCoef, sigma, kappa),
+                             dbc_bdr);
+
+      // Augment the n.Grad(u) term with a*u on the Robin portion of boundary
+      a.AddBdrFaceIntegrator(new BoundaryMassIntegrator(m_rbcACoef),
+                             rbc_bdr);
+   }
+   a.Assemble();
+
+   // 10. Assemble the parallel linear form for the right hand side vector.
+   ParLinearForm b(&fespace);
+
+   if (h1)
+   {
+      // Set the Dirchlet values in the solution vector
+      u.ProjectBdrCoefficient(dbcCoef, dbc_bdr);
+
+      // Add the desired value for n.Grad(u) on the Neumann boundary
+      b.AddBoundaryIntegrator(new BoundaryLFIntegrator(m_nbcCoef), nbc_bdr);
+
+      // Add the desired value for n.Grad(u) + a*u on the Robin boundary
+      b.AddBoundaryIntegrator(new BoundaryLFIntegrator(m_rbcBCoef), rbc_bdr);
+   }
+   else
+   {
+      // Add the desired value for the Dirchlet boundary
+      b.AddBdrFaceIntegrator(new DGDirichletLFIntegrator(dbcCoef, matCoef,
+                                                         sigma, kappa),
+                             dbc_bdr);
+
+      // Add the desired value for n.Grad(u) on the Neumann boundary
+      b.AddBdrFaceIntegrator(new BoundaryLFIntegrator(m_nbcCoef),
+                             nbc_bdr);
+
+      // Add the desired value for n.Grad(u) + a*u on the Robin boundary
+      b.AddBdrFaceIntegrator(new BoundaryLFIntegrator(m_rbcBCoef),
+                             rbc_bdr);
+   }
+   b.Assemble();
+
+   // 11. Construct the linear system.
+   OperatorPtr A;
+   Vector B, X;
+   a.FormLinearSystem(ess_tdof_list, u, b, A, X, B);
+
+   // 12. Solve the linear system A X = B.
+   HypreSolver *amg = new HypreBoomerAMG;
+   if (h1 || sigma == -1.0)
+   {
+      HyprePCG pcg(MPI_COMM_WORLD);
+      pcg.SetTol(1e-12);
+      pcg.SetMaxIter(200);
+      pcg.SetPrintLevel(2);
+      pcg.SetPreconditioner(*amg);
+      pcg.SetOperator(*A);
+      pcg.Mult(B, X);
+   }
+   else
+   {
+      GMRESSolver gmres(MPI_COMM_WORLD);
+      gmres.SetAbsTol(0.0);
+      gmres.SetRelTol(1e-12);
+      gmres.SetMaxIter(200);
+      gmres.SetKDim(10);
+      gmres.SetPrintLevel(1);
+      gmres.SetPreconditioner(*amg);
+      gmres.SetOperator(*A);
+      gmres.Mult(B, X);
+   }
+   delete amg;
+
+   // 13. Recover the parallel grid function corresponding to U. This is the
+   //     local finite element solution on each processor.
+   a.RecoverFEMSolution(X, b, u);
+
+   // 14. Build a mass matrix to help solve for n.Grad(u) where 'n' is
+   //     a surface normal.
+   ParBilinearForm m(&fespace);
+   m.AddDomainIntegrator(new MassIntegrator);
+   m.Assemble();
+
+   ess_tdof_list.SetSize(0);
+   OperatorPtr M;
+   m.FormSystemMatrix(ess_tdof_list, M);
+
+   // 15. Compute the various boundary integrals.
+   mfem::out << endl
+             << "Verifying boundary conditions" << endl
+             << "=============================" << endl;
+   {
+      // Integrate the solution on the Dirichlet boundary and compare
+      // to the expected value.
+      double err, avg = IntegrateBC(u, dbc_bdr, 0.0, 1.0, dbc_val, err);
+
+      bool hom_dbc = (dbc_val == 0.0);
+      err /=  hom_dbc ? 1.0 : fabs(dbc_val);
+      mfem::out << "Average of solution on Gamma_dbc:\t"
+                << avg << ", \t"
+                << (hom_dbc ? "absolute" : "relative")
+                << " error " << err << endl;
+   }
+   {
+      // Integrate n.Grad(u) on the inhomogeneous Neumann boundary and
+      // compare to the expected value.
+      double err, avg = IntegrateBC(u, nbc_bdr, 1.0, 0.0, nbc_val, err);
+
+      bool hom_nbc = (nbc_val == 0.0);
+      err /=  hom_nbc ? 1.0 : fabs(nbc_val);
+      mfem::out << "Average of n.Grad(u) on Gamma_nbc:\t"
+                << avg << ", \t"
+                << (hom_nbc ? "absolute" : "relative")
+                << " error " << err << endl;
+   }
+   {
+      // Integrate n.Grad(u) on the homogeneous Neumann boundary and compare
+      // to the expected value of zero.
+      Array<int> nbc0_bdr(pmesh.bdr_attributes.Max());
+      nbc0_bdr = 0;
+      nbc0_bdr[3] = 1;
+
+      double err, avg = IntegrateBC(u, nbc0_bdr, 1.0, 0.0, 0.0, err);
+
+      bool hom_nbc = true;
+      mfem::out << "Average of n.Grad(u) on Gamma_nbc0:\t"
+                << avg << ", \t"
+                << (hom_nbc ? "absolute" : "relative")
+                << " error " << err << endl;
+   }
+   {
+      // Integrate n.Grad(u) + a * u on the Robin boundary and compare to
+      // the expected value.
+      double err, avg = IntegrateBC(u, rbc_bdr, 1.0, rbc_a_val, rbc_b_val, err);
+
+      bool hom_rbc = (rbc_b_val == 0.0);
+      err /=  hom_rbc ? 1.0 : fabs(rbc_b_val);
+      mfem::out << "Average of n.Grad(u)+a*u on Gamma_rbc:\t"
+                << avg << ", \t"
+                << (hom_rbc ? "absolute" : "relative")
+                << " error " << err << endl;
+   }
+
+   // 16. Save the refined mesh and the solution in parallel. This output can
+   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
+   {
+      ostringstream mesh_name, sol_name;
+      mesh_name << "mesh." << setfill('0') << setw(6) << mpi.WorldRank();
+      sol_name << "sol." << setfill('0') << setw(6) << mpi.WorldRank();
+
+      ofstream mesh_ofs(mesh_name.str().c_str());
+      mesh_ofs.precision(8);
+      pmesh.Print(mesh_ofs);
+
+      ofstream sol_ofs(sol_name.str().c_str());
+      sol_ofs.precision(8);
+      u.Save(sol_ofs);
+   }
+
+   // 17. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      string title_str = h1 ? "H1" : "DG";
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << mpi.WorldSize()
+               << " " << mpi.WorldRank() << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << pmesh << u
+               << "window_title '" << title_str << " Solution'"
+               << " keys 'mmc'" << flush;
+   }
+
+   // 18. Free the used memory.
+   delete fec;
+
+   return 0;
+}
+
+void quad_trans(double u, double v, double &x, double &y, bool log = false)
+{
+   double a = a_; // Radius of disc
+
+   double d = 4.0 * a * (M_SQRT2 - 2.0 * a) * (1.0 - 2.0 * v);
+
+   double v0 = (1.0 + M_SQRT2) * (M_SQRT2 * a - 2.0 * v) *
+               ((4.0 - 3 * M_SQRT2) * a +
+                (8.0 * (M_SQRT2 - 1.0) * a - 2.0) * v) / d;
+
+   double r = 2.0 * ((M_SQRT2 - 1.0) * a * a * (1.0 - 4.0 *v) +
+                     2.0 * (1.0 + M_SQRT2 *
+                            (1.0 + 2.0 * (2.0 * a - M_SQRT2 - 1.0) * a)) * v * v
+                    ) / d;
+
+   double t = asin(v / r) * u / v;
+   if (log)
+   {
+      mfem::out << "u, v, r, v0, t "
+                << u << " " << v << " " << r << " " << v0 << " " << t
+                << endl;
+   }
+   x = r * sin(t);
+   y = r * cos(t) - v0;
+}
+
+void trans(const Vector &u, Vector &x)
+{
+   double tol = 1e-4;
+
+   if (u[1] > 0.5 - tol || u[1] < -0.5 + tol)
+   {
+      x = u;
+      return;
+   }
+   if (u[0] > 1.0 - tol || u[0] < -1.0 + tol || fabs(u[0]) < tol)
+   {
+      x = u;
+      return;
+   }
+
+   if (u[0] > 0.0)
+   {
+      if (u[1] > fabs(u[0] - 0.5))
+      {
+         quad_trans(u[0] - 0.5, u[1], x[0], x[1]);
+         x[0] += 0.5;
+         return;
+      }
+      if (u[1] < -fabs(u[0] - 0.5))
+      {
+         quad_trans(u[0] - 0.5, -u[1], x[0], x[1]);
+         x[0] += 0.5;
+         x[1] *= -1.0;
+         return;
+      }
+      if (u[0] - 0.5 > fabs(u[1]))
+      {
+         quad_trans(u[1], u[0] - 0.5, x[1], x[0]);
+         x[0] += 0.5;
+         return;
+      }
+      if (u[0] - 0.5 < -fabs(u[1]))
+      {
+         quad_trans(u[1], 0.5 - u[0], x[1], x[0]);
+         x[0] *= -1.0;
+         x[0] += 0.5;
+         return;
+      }
+   }
+   else
+   {
+      if (u[1] > fabs(u[0] + 0.5))
+      {
+         quad_trans(u[0] + 0.5, u[1], x[0], x[1]);
+         x[0] -= 0.5;
+         return;
+      }
+      if (u[1] < -fabs(u[0] + 0.5))
+      {
+         quad_trans(u[0] + 0.5, -u[1], x[0], x[1]);
+         x[0] -= 0.5;
+         x[1] *= -1.0;
+         return;
+      }
+      if (u[0] + 0.5 > fabs(u[1]))
+      {
+         quad_trans(u[1], u[0] + 0.5, x[1], x[0]);
+         x[0] -= 0.5;
+         return;
+      }
+      if (u[0] + 0.5 < -fabs(u[1]))
+      {
+         quad_trans(u[1], -0.5 - u[0], x[1], x[0]);
+         x[0] *= -1.0;
+         x[0] -= 0.5;
+         return;
+      }
+   }
+   x = u;
+}
+
+Mesh * GenerateSerialMesh(int ref)
+{
+   Mesh * mesh = new Mesh(2, 29, 16, 24, 2);
+
+   int vi[4];
+
+   for (int i=0; i<2; i++)
+   {
+      int o = 13 * i;
+      vi[0] = o + 0; vi[1] = o + 3; vi[2] = o + 4; vi[3] = o + 1;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 1; vi[1] = o + 4; vi[2] = o + 5; vi[3] = o + 2;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 5; vi[1] = o + 8; vi[2] = o + 9; vi[3] = o + 2;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 8; vi[1] = o + 12; vi[2] = o + 15; vi[3] = o + 9;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 11; vi[1] = o + 14; vi[2] = o + 15; vi[3] = o + 12;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 10; vi[1] = o + 13; vi[2] = o + 14; vi[3] = o + 11;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 6; vi[1] = o + 13; vi[2] = o + 10; vi[3] = o + 7;
+      mesh->AddQuad(vi);
+
+      vi[0] = o + 0; vi[1] = o + 6; vi[2] = o + 7; vi[3] = o + 3;
+      mesh->AddQuad(vi);
+   }
+
+   vi[0] =  0; vi[1] =  6; mesh->AddBdrSegment(vi, 1);
+   vi[0] =  6; vi[1] = 13; mesh->AddBdrSegment(vi, 1);
+   vi[0] = 13; vi[1] = 19; mesh->AddBdrSegment(vi, 1);
+   vi[0] = 19; vi[1] = 26; mesh->AddBdrSegment(vi, 1);
+
+   vi[0] = 28; vi[1] = 22; mesh->AddBdrSegment(vi, 2);
+   vi[0] = 22; vi[1] = 15; mesh->AddBdrSegment(vi, 2);
+   vi[0] = 15; vi[1] =  9; mesh->AddBdrSegment(vi, 2);
+   vi[0] =  9; vi[1] =  2; mesh->AddBdrSegment(vi, 2);
+
+   for (int i=0; i<2; i++)
+   {
+      int o = 13 * i;
+      vi[0] = o +  7; vi[1] = o +  3; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o + 10; vi[1] = o +  7; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o + 11; vi[1] = o + 10; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o + 12; vi[1] = o + 11; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o +  8; vi[1] = o + 12; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o +  5; vi[1] = o +  8; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o +  4; vi[1] = o +  5; mesh->AddBdrSegment(vi, 3 + i);
+      vi[0] = o +  3; vi[1] = o +  4; mesh->AddBdrSegment(vi, 3 + i);
+   }
+
+   double d[2];
+   double a = a_ / M_SQRT2;
+
+   d[0] = -1.0; d[1] = -0.5; mesh->AddVertex(d);
+   d[0] = -1.0; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] = -1.0; d[1] =  0.5; mesh->AddVertex(d);
+
+   d[0] = -0.5 - a; d[1] =   -a; mesh->AddVertex(d);
+   d[0] = -0.5 - a; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] = -0.5 - a; d[1] =    a; mesh->AddVertex(d);
+
+   d[0] = -0.5; d[1] = -0.5; mesh->AddVertex(d);
+   d[0] = -0.5; d[1] =   -a; mesh->AddVertex(d);
+   d[0] = -0.5; d[1] =    a; mesh->AddVertex(d);
+   d[0] = -0.5; d[1] =  0.5; mesh->AddVertex(d);
+
+   d[0] = -0.5 + a; d[1] =   -a; mesh->AddVertex(d);
+   d[0] = -0.5 + a; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] = -0.5 + a; d[1] =    a; mesh->AddVertex(d);
+
+   d[0] =  0.0; d[1] = -0.5; mesh->AddVertex(d);
+   d[0] =  0.0; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] =  0.0; d[1] =  0.5; mesh->AddVertex(d);
+
+   d[0] =  0.5 - a; d[1] =   -a; mesh->AddVertex(d);
+   d[0] =  0.5 - a; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] =  0.5 - a; d[1] =    a; mesh->AddVertex(d);
+
+   d[0] =  0.5; d[1] = -0.5; mesh->AddVertex(d);
+   d[0] =  0.5; d[1] =   -a; mesh->AddVertex(d);
+   d[0] =  0.5; d[1] =    a; mesh->AddVertex(d);
+   d[0] =  0.5; d[1] =  0.5; mesh->AddVertex(d);
+
+   d[0] =  0.5 + a; d[1] =   -a; mesh->AddVertex(d);
+   d[0] =  0.5 + a; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] =  0.5 + a; d[1] =    a; mesh->AddVertex(d);
+
+   d[0] =  1.0; d[1] = -0.5; mesh->AddVertex(d);
+   d[0] =  1.0; d[1] =  0.0; mesh->AddVertex(d);
+   d[0] =  1.0; d[1] =  0.5; mesh->AddVertex(d);
+
+   mesh->FinalizeTopology();
+
+   mesh->SetCurvature(1, true);
+
+   // Stitch the ends of the stack together
+   {
+      Array<int> v2v(mesh->GetNV());
+      for (int i = 0; i < v2v.Size() - 3; i++)
+      {
+         v2v[i] = i;
+      }
+      // identify vertices on the narrow ends of the rectangle
+      v2v[v2v.Size() - 3] = 0;
+      v2v[v2v.Size() - 2] = 1;
+      v2v[v2v.Size() - 1] = 2;
+
+      // renumber elements
+      for (int i = 0; i < mesh->GetNE(); i++)
+      {
+         Element *el = mesh->GetElement(i);
+         int *v = el->GetVertices();
+         int nv = el->GetNVertices();
+         for (int j = 0; j < nv; j++)
+         {
+            v[j] = v2v[v[j]];
+         }
+      }
+      // renumber boundary elements
+      for (int i = 0; i < mesh->GetNBE(); i++)
+      {
+         Element *el = mesh->GetBdrElement(i);
+         int *v = el->GetVertices();
+         int nv = el->GetNVertices();
+         for (int j = 0; j < nv; j++)
+         {
+            v[j] = v2v[v[j]];
+         }
+      }
+      mesh->RemoveUnusedVertices();
+      mesh->RemoveInternalBoundaries();
+   }
+   mesh->SetCurvature(3, true);
+
+   for (int l = 0; l < ref; l++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   mesh->Transform(trans);
+
+   return mesh;
+}
+
+double IntegrateBC(const ParGridFunction &x, const Array<int> &bdr,
+                   double alpha, double beta, double gamma,
+                   double &glb_err)
+{
+   double loc_vals[3];
+   double &nrm = loc_vals[0];
+   double &avg = loc_vals[1];
+   double &err = loc_vals[2];
+
+   nrm = 0.0;
+   avg = 0.0;
+   err = 0.0;
+
+   const bool a_is_zero = alpha == 0.0;
+   const bool b_is_zero = beta == 0.0;
+
+   const ParFiniteElementSpace &fes = *x.ParFESpace();
+   MFEM_ASSERT(fes.GetVDim() == 1, "");
+   ParMesh &mesh = *fes.GetParMesh();
+   Vector shape, loc_dofs, w_nor;
+   DenseMatrix dshape;
+   Array<int> dof_ids;
+   for (int i = 0; i < mesh.GetNBE(); i++)
+   {
+      if (bdr[mesh.GetBdrAttribute(i)-1] == 0) { continue; }
+
+      FaceElementTransformations *FTr = mesh.GetBdrFaceTransformations(i);
+      if (FTr == nullptr) { continue; }
+
+      const FiniteElement &fe = *fes.GetFE(FTr->Elem1No);
+      MFEM_ASSERT(fe.GetMapType() == FiniteElement::VALUE, "");
+      const int int_order = 2*fe.GetOrder() + 3;
+      const IntegrationRule &ir = IntRules.Get(FTr->FaceGeom, int_order);
+
+      fes.GetElementDofs(FTr->Elem1No, dof_ids);
+      x.GetSubVector(dof_ids, loc_dofs);
+      if (!a_is_zero)
+      {
+         const int sdim = FTr->Face->GetSpaceDim();
+         w_nor.SetSize(sdim);
+         dshape.SetSize(fe.GetDof(), sdim);
+      }
+      if (!b_is_zero)
+      {
+         shape.SetSize(fe.GetDof());
+      }
+      for (int j = 0; j < ir.GetNPoints(); j++)
+      {
+         const IntegrationPoint &ip = ir.IntPoint(j);
+         IntegrationPoint eip;
+         FTr->Loc1.Transform(ip, eip);
+         FTr->Face->SetIntPoint(&ip);
+         double face_weight = FTr->Face->Weight();
+         double val = 0.0;
+         if (!a_is_zero)
+         {
+            FTr->Elem1->SetIntPoint(&eip);
+            fe.CalcPhysDShape(*FTr->Elem1, dshape);
+            CalcOrtho(FTr->Face->Jacobian(), w_nor);
+            val += alpha * dshape.InnerProduct(w_nor, loc_dofs) / face_weight;
+         }
+         if (!b_is_zero)
+         {
+            fe.CalcShape(eip, shape);
+            val += beta * (shape * loc_dofs);
+         }
+
+         // Measure the length of the boundary
+         nrm += ip.weight * face_weight;
+
+         // Integrate alpha * n.Grad(x) + beta * x
+         avg += val * ip.weight * face_weight;
+
+         // Integrate |alpha * n.Grad(x) + beta * x - gamma|^2
+         val -= gamma;
+         err += (val*val) * ip.weight * face_weight;
+      }
+   }
+
+   double glb_vals[3];
+   MPI_Allreduce(loc_vals, glb_vals, 3, MPI_DOUBLE, MPI_SUM, fes.GetComm());
+
+   double glb_nrm = glb_vals[0];
+   double glb_avg = glb_vals[1];
+   glb_err = glb_vals[2];
+
+   // Normalize by the length of the boundary
+   if (std::abs(glb_nrm) > 0.0)
+   {
+      glb_err /= glb_nrm;
+      glb_avg /= glb_nrm;
+   }
+
+   // Compute l2 norm of the error in the boundary condition
+   // (negative quadrature weights may produce negative 'err')
+   glb_err = (glb_err >= 0.0) ? sqrt(glb_err) : -sqrt(-glb_err);
+
+   // Return the average value of alpha * n.Grad(x) + beta * x
+   return glb_avg;
+}

examples/ex2p.cpp

@@ -274,6 +274,13 @@ int main(int argc, char *argv[])
       pmesh->SetNodalFESpace(fespace);
    }
 
+   {
+      x.Save("ex2p.gf", 1);
+      ParGridFunction new_x(fespace, "ex2p.gf");
+      new_x -= x;
+      out << "GF difference: " << new_x.Norml1() << endl;
+   }
+
    // 16. Save in parallel the displaced mesh and the inverted solution (which
    //     gives the backward displacements to the original grid). This output
    //     can be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".

examples/ex5p.cpp

@@ -22,6 +22,8 @@
 //               The example demonstrates the use of the BlockMatrix class, as
 //               well as the collective saving of several grid functions in
 //               VisIt (visit.llnl.gov) and ParaView (paraview.org) formats.
+//               Optional saving with ADIOS2 (adios2.readthedocs.io) streams is
+//               also illustrated.
 //
 //               We recommend viewing examples 1-4 before viewing this example.
 
@@ -55,6 +57,7 @@ int main(int argc, char *argv[])
    int order = 1;
    bool par_format = false;
    bool visualization = 1;
+   bool adios2 = false;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
@@ -67,6 +70,9 @@ int main(int argc, char *argv[])
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
+   args.AddOption(&adios2, "-adios2", "--adios2-streams", "-no-adios2",
+                  "--no-adios2-streams",
+                  "Save data using adios2 streams.");
    args.Parse();
    if (!args.Good())
    {
@@ -337,7 +343,27 @@ int main(int argc, char *argv[])
    paraview_dc.RegisterField("pressure",p);
    paraview_dc.Save();
 
-   // 17. Send the solution by socket to a GLVis server.
+   // 17. Optionally output a BP (binary pack) file using ADIOS2. This can be
+   //     visualized with the ParaView VTX reader.
+#ifdef MFEM_USE_ADIOS2
+   if (adios2)
+   {
+      std::string postfix(mesh_file);
+      postfix.erase(0, std::string("../data/").size() );
+      postfix += "_o" + std::to_string(order);
+      const std::string collection_name = "ex5-p_" + postfix + ".bp";
+
+      ADIOS2DataCollection adios2_dc(MPI_COMM_WORLD, collection_name, pmesh);
+      adios2_dc.SetLevelsOfDetail(1);
+      adios2_dc.SetCycle(1);
+      adios2_dc.SetTime(0.0);
+      adios2_dc.RegisterField("velocity",u);
+      adios2_dc.RegisterField("pressure",p);
+      adios2_dc.Save();
+   }
+#endif
+
+   // 18. Send the solution by socket to a GLVis server.
    if (visualization)
    {
       char vishost[] = "localhost";
@@ -357,7 +383,7 @@ int main(int argc, char *argv[])
              << endl;
    }
 
-   // 18. Free the used memory.
+   // 19. Free the used memory.
    delete fform;
    delete gform;
    delete u;

examples/ex7.cpp

@@ -279,4 +279,11 @@ void SnapNodes(Mesh &mesh)
          nodes(nodes.FESpace()->DofToVDof(i, d)) = node(d);
       }
    }
+   if (mesh.Nonconforming())
+   {
+      // Snap hanging nodes to the master side.
+      Vector tnodes;
+      nodes.GetTrueDofs(tnodes);
+      nodes.SetFromTrueDofs(tnodes);
+   }
 }

examples/ex7p.cpp

@@ -348,4 +348,11 @@ void SnapNodes(Mesh &mesh)
          nodes(nodes.FESpace()->DofToVDof(i, d)) = node(d);
       }
    }
+   if (mesh.Nonconforming())
+   {
+      // Snap hanging nodes to the master side.
+      Vector tnodes;
+      nodes.GetTrueDofs(tnodes);
+      nodes.SetFromTrueDofs(tnodes);
+   }
 }

examples/ex9p.cpp

@@ -31,9 +31,10 @@
 //               and explicit ODE time integrators, the definition of periodic
 //               boundary conditions through periodic meshes, as well as the use
 //               of GLVis for persistent visualization of a time-evolving
-//               solution. The saving of time-dependent data files for external
-//               visualization with VisIt (visit.llnl.gov) and ParaView
-//               (paraview.org) is also illustrated.
+//               solution. Saving of time-dependent data files for visualization
+//               with VisIt (visit.llnl.gov) and ParaView (paraview.org), as
+//               well as the optional saving with ADIOS2 (adios2.readthedocs.io)
+//               are also illustrated.
 
 #include "mfem.hpp"
 #include <fstream>
@@ -167,6 +168,7 @@ int main(int argc, char *argv[])
    bool visualization = true;
    bool visit = false;
    bool paraview = false;
+   bool adios2 = false;
    bool binary = false;
    int vis_steps = 5;
 
@@ -208,6 +210,9 @@ int main(int argc, char *argv[])
    args.AddOption(&paraview, "-paraview", "--paraview-datafiles", "-no-paraview",
                   "--no-paraview-datafiles",
                   "Save data files for ParaView (paraview.org) visualization.");
+   args.AddOption(&adios2, "-adios2", "--adios2-streams", "-no-adios2",
+                  "--no-adios2-streams",
+                  "Save data using adios2 streams.");
    args.AddOption(&binary, "-binary", "--binary-datafiles", "-ascii",
                   "--ascii-datafiles",
                   "Use binary (Sidre) or ascii format for VisIt data files.");
@@ -394,6 +399,28 @@ int main(int argc, char *argv[])
       pd->Save();
    }
 
+   // Optionally output a BP (binary pack) file using ADIOS2. This can be
+   // visualized with the ParaView VTX reader.
+#ifdef MFEM_USE_ADIOS2
+   ADIOS2DataCollection *adios2_dc = NULL;
+   if (adios2)
+   {
+      std::string postfix(mesh_file);
+      postfix.erase(0, std::string("../data/").size() );
+      postfix += "_o" + std::to_string(order);
+      const std::string collection_name = "ex9-p-" + postfix + ".bp";
+
+      adios2_dc = new ADIOS2DataCollection(MPI_COMM_WORLD, collection_name, pmesh);
+      // output data substreams are half the number of mpi processes
+      adios2_dc->SetParameter("SubStreams", std::to_string(num_procs/2) );
+      // adios2_dc->SetLevelsOfDetail(2);
+      adios2_dc->RegisterField("solution", u);
+      adios2_dc->SetCycle(0);
+      adios2_dc->SetTime(0.0);
+      adios2_dc->Save();
+   }
+#endif
+
    socketstream sout;
    if (visualization)
    {
@@ -472,6 +499,16 @@ int main(int argc, char *argv[])
             pd->SetTime(t);
             pd->Save();
          }
+
+#ifdef MFEM_USE_ADIOS2
+         // transient solutions can be visualized with ParaView
+         if (adios2)
+         {
+            adios2_dc->SetCycle(ti);
+            adios2_dc->SetTime(t);
+            adios2_dc->Save();
+         }
+#endif
       }
    }
 
@@ -497,6 +534,12 @@ int main(int argc, char *argv[])
    delete pmesh;
    delete ode_solver;
    delete pd;
+#ifdef MFEM_USE_ADIOS2
+   if (adios2)
+   {
+      delete adios2_dc;
+   }
+#endif
    delete dc;
 
    MPI_Finalize();

examples/io_benchmark.cpp

@@ -0,0 +1,292 @@
+//                       MFEM Example 1 - Parallel Version
+//
+// Compile with: make ex1p
+//
+// Sample runs:  mpirun -np 4 ex1p -m ../data/square-disc.mesh
+//               mpirun -np 4 ex1p -m ../data/star.mesh
+//               mpirun -np 4 ex1p -m ../data/star-mixed.mesh
+//               mpirun -np 4 ex1p -m ../data/escher.mesh
+//               mpirun -np 4 ex1p -m ../data/fichera.mesh
+//               mpirun -np 4 ex1p -m ../data/fichera-mixed.mesh
+//               mpirun -np 4 ex1p -m ../data/toroid-wedge.mesh
+//               mpirun -np 4 ex1p -m ../data/square-disc-p2.vtk -o 2
+//               mpirun -np 4 ex1p -m ../data/square-disc-p3.mesh -o 3
+//               mpirun -np 4 ex1p -m ../data/square-disc-nurbs.mesh -o -1
+//               mpirun -np 4 ex1p -m ../data/star-mixed-p2.mesh -o 2
+//               mpirun -np 4 ex1p -m ../data/disc-nurbs.mesh -o -1
+//               mpirun -np 4 ex1p -m ../data/pipe-nurbs.mesh -o -1
+//               mpirun -np 4 ex1p -m ../data/ball-nurbs.mesh -o 2
+//               mpirun -np 4 ex1p -m ../data/fichera-mixed-p2.mesh -o 2
+//               mpirun -np 4 ex1p -m ../data/star-surf.mesh
+//               mpirun -np 4 ex1p -m ../data/square-disc-surf.mesh
+//               mpirun -np 4 ex1p -m ../data/inline-segment.mesh
+//               mpirun -np 4 ex1p -m ../data/amr-quad.mesh
+//               mpirun -np 4 ex1p -m ../data/amr-hex.mesh
+//               mpirun -np 4 ex1p -m ../data/mobius-strip.mesh
+//               mpirun -np 4 ex1p -m ../data/mobius-strip.mesh -o -1 -sc
+//
+// Device sample runs:
+//               mpirun -np 4 ex1p -pa -d cuda
+//               mpirun -np 4 ex1p -pa -d occa-cuda
+//               mpirun -np 4 ex1p -pa -d raja-omp
+//               mpirun -np 4 ex1p -pa -d ceed-cpu
+//               mpirun -np 4 ex1p -pa -d ceed-cuda
+//               mpirun -np 4 ex1p -m ../data/beam-tet.mesh -pa -d ceed-cpu
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Laplace problem
+//               -Delta u = 1 with homogeneous Dirichlet boundary conditions.
+//               Specifically, we discretize using a FE space of the specified
+//               order, or if order < 1 using an isoparametric/isogeometric
+//               space (i.e. quadratic for quadratic curvilinear mesh, NURBS for
+//               NURBS mesh, etc.)
+//
+//               The example highlights the use of mesh refinement, finite
+//               element grid functions, as well as linear and bilinear forms
+//               corresponding to the left-hand side and right-hand side of the
+//               discrete linear system. We also cover the explicit elimination
+//               of essential boundary conditions, static condensation, and the
+//               optional connection to the GLVis tool for visualization.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include "mpi.h"
+
+using namespace std;
+using namespace mfem;
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   // const char *mesh_file = "../data/star.mesh";
+   const char *mesh_file = "../data/square-disc.mesh";
+   int order = 1;
+   bool static_cond = false;
+   bool pa = false;
+   const char *device_config = "cpu";
+   bool visualization = false;
+   int nfiles = 1;
+   // const char *out_file = "0_0.gf";
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.AddOption(&nfiles, "-nf", "--num-files", "Number of files to write.");
+   // args.AddOption(&out_file, "-o", "--outfile",
+   //                "Name of file to write.");
+
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   // 3. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   if (myid == 0) { device.Print(); }
+
+   // 4. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
+   //    and volume meshes with the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 5. Refine the serial mesh on all processors to increase the resolution. In
+   //    this example we do 'ref_levels' of uniform refinement. We choose
+   //    'ref_levels' to be the largest number that gives a final mesh with no
+   //    more than 10,000 elements.
+   {
+      int ref_levels =
+         (int)floor(log(10000./mesh->GetNE())/log(2.)/dim);
+      for (int l = 0; l < ref_levels; l++)
+      {
+         mesh->UniformRefinement();
+      }
+   }
+
+   // 6. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   {
+      int par_ref_levels = 2;
+      for (int l = 0; l < par_ref_levels; l++)
+      {
+         pmesh->UniformRefinement();
+      }
+   }
+
+   // 7. Define a parallel finite element space on the parallel mesh. Here we
+   //    use continuous Lagrange finite elements of the specified order. If
+   //    order < 1, we instead use an isoparametric/isogeometric space.
+   FiniteElementCollection *fec;
+   if (order > 0)
+   {
+      fec = new H1_FECollection(order, dim);
+   }
+   else if (pmesh->GetNodes())
+   {
+      fec = pmesh->GetNodes()->OwnFEC();
+      if (myid == 0)
+      {
+         cout << "Using isoparametric FEs: " << fec->Name() << endl;
+      }
+   }
+   else
+   {
+      fec = new H1_FECollection(order = 1, dim);
+   }
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec, 1, 0);
+   HYPRE_Int size = fespace->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of finite element unknowns: " << size << endl;
+   }
+
+   // 8. Determine the list of true (i.e. parallel conforming) essential
+   //    boundary dofs. In this example, the boundary conditions are defined
+   //    by marking all the boundary attributes from the mesh as essential
+   //    (Dirichlet) and converting them to a list of true dofs.
+   Array<int> ess_tdof_list;
+   if (pmesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   // 9. Set up the parallel linear form b(.) which corresponds to the
+   //    right-hand side of the FEM linear system, which in this case is
+   //    (1,phi_i) where phi_i are the basis functions in fespace.
+   ParLinearForm *b = new ParLinearForm(fespace);
+   ConstantCoefficient one(1.0);
+   b->AddDomainIntegrator(new DomainLFIntegrator(one));
+   b->Assemble();
+
+   // 10. Define the solution vector x as a parallel finite element grid function
+   //     corresponding to fespace. Initialize x with initial guess of zero,
+   //     which satisfies the boundary conditions.
+   ParGridFunction x(fespace);
+   x = 0.0;
+
+   // 11. Set up the parallel bilinear form a(.,.) on the finite element space
+   //     corresponding to the Laplacian operator -Delta, by adding the Diffusion
+   //     domain integrator.
+   ParBilinearForm *a = new ParBilinearForm(fespace);
+   if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
+   a->AddDomainIntegrator(new DiffusionIntegrator(one));
+
+   // 12. Assemble the parallel bilinear form and the corresponding linear
+   //     system, applying any necessary transformations such as: parallel
+   //     assembly, eliminating boundary conditions, applying conforming
+   //     constraints for non-conforming AMR, static condensation, etc.
+   if (static_cond) { a->EnableStaticCondensation(); }
+   a->Assemble();
+
+   OperatorPtr A;
+   Vector B, X;
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+
+   // 13. Solve the linear system A X = B.
+   //     * With full assembly, use the BoomerAMG preconditioner from hypre.
+   //     * With partial assembly, use Jacobi smoothing, for now.
+   Solver *prec = NULL;
+   if (pa)
+   {
+      if (UsesTensorBasis(*fespace))
+      {
+         prec = new OperatorJacobiSmoother(*a, ess_tdof_list);
+      }
+   }
+   else
+   {
+      prec = new HypreBoomerAMG;
+   }
+   CGSolver cg(MPI_COMM_WORLD);
+   cg.SetRelTol(1e-12);
+   cg.SetMaxIter(2000);
+   cg.SetPrintLevel(1);
+   if (prec) { cg.SetPreconditioner(*prec); }
+   cg.SetOperator(*A);
+   cg.Mult(B, X);
+   delete prec;
+
+   // 14. Recover the parallel grid function corresponding to X. This is the
+   //     local finite element solution on each processor.
+   a->RecoverFEMSolution(X, *b, x);
+
+   std::string filename = to_string(num_procs) + "_" + to_string(nfiles) + "_";
+   {
+      double t1;
+      t1 = MPI_Wtime();
+      x.Save(filename.c_str(), nfiles);
+      double t2 = MPI_Wtime();
+
+      double write_time = t2 - t1;
+      double average_write_time;
+      MPI_Reduce(&write_time, &average_write_time, 1,
+               MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
+      if (myid == 0)
+      {
+         std::cout << "Average write time: " << average_write_time / num_procs << " for "
+                   << nfiles << " files and "  << num_procs << " ranks\n";
+      }
+   }
+   {
+      double t1;
+      t1 = MPI_Wtime();
+      ParGridFunction temp_gf(fespace, filename.c_str());
+      double t2 = MPI_Wtime();
+
+      double read_time = t2 - t1;
+      double average_read_time;
+      MPI_Reduce(&read_time, &average_read_time, 1,
+                 MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
+      if (myid == 0)
+      {
+         std::cout << "Average read time: " << average_read_time / num_procs << " for "
+                   << nfiles << " files and "  << num_procs << " ranks\n";
+      }
+   }
+
+   // 17. Free the used memory.
+   delete a;
+   delete b;
+   delete fespace;
+   if (order > 0) { delete fec; }
+   delete pmesh;
+
+   MPI_Finalize();
+
+   return 0;
+}

examples/makefile

@@ -22,9 +22,10 @@ MFEM_LIB_FILE = mfem_is_not_built
 -include $(CONFIG_MK)
 
 SEQ_EXAMPLES = ex1 ex2 ex3 ex4 ex5 ex6 ex7 ex8 ex9 ex10 ex14 ex15 ex16 ex17\
-  ex18 ex19 ex20 ex21 ex22 ex23 ex24 ex25
+  ex18 ex19 ex20 ex21 ex22 ex23 ex24 ex25 ex26 ex27
 PAR_EXAMPLES = ex1p ex2p ex3p ex4p ex5p ex6p ex7p ex8p ex9p ex10p ex11p ex12p\
- ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex21p ex22p ex24p ex25p
+  ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex21p ex22p ex24p ex25p\
+  ex26p ex27p
 
 ifeq ($(MFEM_USE_MPI),NO)
    EXAMPLES = $(SEQ_EXAMPLES)
@@ -103,6 +104,10 @@ ex15-test-seq: ex15
 	@$(call mfem-test,$<,, Serial example,-e 1)
 ex15p-test-par: ex15p
 	@$(call mfem-test,$<, $(RUN_MPI), Parallel example,-e 1)
+ex27-test-seq: ex27
+	@$(call mfem-test,$<,, Serial example,-dg)
+ex27p-test-par: ex27p
+	@$(call mfem-test,$<, $(RUN_MPI), Parallel example,-dg)
 # Testing: optional tests
 ifeq ($(MFEM_USE_STRUMPACK),YES)
 ex11p-test-strumpack: ex11p
@@ -128,6 +133,7 @@ clean-exec:
 	@rm -f sphere_refined.* sol.* sol_u.* sol_p.* sol_r.* sol_i.*
 	@rm -f ex9.mesh ex9-mesh.* ex9-init.* ex9-final.*
 	@rm -f deformed.* velocity.* elastic_energy.* mode_*
+	@rm -f ex5-p-*.bp ex9-p-*.bp ex12-p-*.bp ex16-p-*.bp
 	@rm -f ex16.mesh ex16-mesh.* ex16-init.* ex16-final.*
 	@rm -f vortex-mesh.* vortex.mesh vortex-?-init.* vortex-?-final.*
 	@rm -f deformation.* pressure.*

examples/pa_oper.cpp

@@ -0,0 +1,906 @@
+//                                MFEM Example 9
+//
+// Compile with: make serial_nogpu
+//
+// Description:  This code solves the time-dependent advection-diffusion
+//               equation:
+//               \frac(\partial u}{\partial t}
+//                                 = \mathbf{a} \cdot \Nabla u - \nu \Nabla^2 u
+//               where a is a given advection velocity, \nu is the diffusion
+//               parameter, and u0(x) = u(0,x) is a given initial condition.
+//
+//               The demonstrates explicit time marching with H1 elements of
+//               arbitrary order. Periodic boundary conditions are used through
+//               periodic meshes. GLVis can be used for visualization of a
+//               time-evolving solution.
+
+#include <fstream>
+#include <iostream>
+#include <algorithm>
+
+#include "mfem.hpp"
+#include "mpi.h"
+
+using namespace std;
+using namespace mfem;
+
+/** A time-dependent operator for the right-hand side of the ODE. The weak
+    form of du/dt = -a.grad(u) + nu Delta(u) is M du/dt = K u + b, where M and
+    K are the mass and advection-diffusion matrices, and b describes the flow
+    on the boundary. This can be written as a general ODE,
+    du/dt = M^{-1} (K u + b), and this class is used to evaluate the right-hand
+    side. */
+class AdvectionDiffusionEvolution : public mfem::TimeDependentOperator
+{
+public:
+   /// \param[in] M - bilinear form for mass matrix
+   /// \param[in] K - bilinear form for stiffness matrix
+   /// \param[in] b - load vector
+   AdvectionDiffusionEvolution(mfem::BilinearForm &M, mfem::BilinearForm &K,
+                               const mfem::Vector &b);
+
+   /// Perform the action of the operator: y = k = f(x, t), where k solves
+   /// Compute k = M^-1(Kx + l)
+   void Mult(const mfem::Vector &x, mfem::Vector &y) const override;
+
+   /// Solve the implicit equation: k = f(x + dt k, t), for the unknown k at
+   /// the current time t. 
+   void ImplicitSolve(const double dt, const mfem::Vector &x,
+                      mfem::Vector &k) override;
+
+   virtual ~AdvectionDiffusionEvolution();
+
+private:
+   mfem::BilinearForm &M, &K;
+   const mfem::Vector &b;
+   /// solver for inverting mass matrix for explicit time-marching
+   std::unique_ptr<mfem::Solver> M_prec;
+   mfem::CGSolver M_solver;
+   /// solver for implicit time-marching
+   mfem::GSSmoother prec;
+   mfem::GMRESSolver linear_solver;
+   mfem::NewtonSolver newton;
+
+   mutable mfem::Vector z;
+
+   /// pointer-to-implementation idiom
+   /// Hides implementation details of this operator
+   class SystemOperator;
+   /// Operator that combines the linear spatial discretization with
+   /// the load vector into one operator used for implicit solves
+   std::unique_ptr<SystemOperator> combined_oper;
+
+   /// sets the state and dt for the combined operator
+   /// \param[in] dt - time increment
+   /// \param[in] x - the current state
+   void setOperParameters(double dt, const mfem::Vector *x);
+
+};
+
+class PAJacobianOperator : public mfem::Operator
+{
+public:
+   PAJacobianOperator(mfem::ParBilinearForm &_mass,
+                      mfem::ParBilinearForm &_stiff);
+
+   /// Compute r = J@k = M@k + dt*K@k
+   /// \param[in] k - dx/dt
+   /// \param[out] r - J@k = M@k + dt*K@k
+   void Mult(const mfem::Vector &k, mfem::Vector &r) const override;
+      
+   /// Set current dt values - needed to compute action of Jacobian.
+   void setParameters(double dt);
+
+private:
+   mfem::ParBilinearForm &mass;
+   mfem::ParBilinearForm &stiff;
+
+   double dt;
+};
+
+class ParSystemOperator : public mfem::Operator
+{
+public:
+   /// Nonlinear operator of the form that combines the mass, res, stiff,
+   /// and load elements for implicit/explicit ODE integration
+   /// \param[in] ess_bdr - array of boundaries attributes marked essential
+   /// \param[in] mass - bilinear form for mass matrix (not owned)
+   /// \param[in] res - nonlinear residual operator (not owned)
+   /// \param[in] stiff - bilinear form for stiffness matrix (not owned)
+   /// \param[in] load - load vector (not owned)
+   /// \param[in] a - used to move the spatial residual to the rhs
+   ParSystemOperator(mfem::ParBilinearForm &_mass,
+                     mfem::ParBilinearForm &_stiff);
+
+   /// Compute r = M@k + K@(x+dt*k)
+   /// (with `@` denoting matrix-vector multiplication)
+   /// \param[in] k - dx/dt 
+   /// \param[out] r - the residual
+   /// \note the signs on each operator must be accounted for elsewhere
+   void Mult(const mfem::Vector &k, mfem::Vector &r) const override;
+
+   /// Compute J = M + dt * K
+   /// \param[in] k - dx/dt 
+   mfem::Operator &GetGradient(const mfem::Vector &k) const override;
+
+   /// Set current dt and x values - needed to compute action and Jacobian.
+   void setParameters(double _dt, const mfem::Vector *_x);
+
+   ~ParSystemOperator();
+
+private:
+   mfem::ParBilinearForm &mass;
+   mfem::ParBilinearForm &stiff;
+   mutable mfem::HypreParMatrix *jacobian, *stiff_jacobian;
+
+   double dt;
+   const mfem::Vector *x;
+
+   mutable mfem::Vector work, work2;
+
+   std::unique_ptr<PAJacobianOperator> pa_jac;
+
+};
+
+/** A time-dependent operator for the right-hand side of the ODE. The weak
+    form of du/dt = -a.grad(u) + nu Delta(u) is M du/dt = K u + b, where M and
+    K are the mass and advection-diffusion matrices, and b describes the flow
+    on the boundary. This can be written as a general ODE,
+    du/dt = M^{-1} (K u + b), and this class is used to evaluate the right-hand
+    side. */
+class ParAdvectionDiffusionEvolution : public mfem::TimeDependentOperator
+{
+public:
+   /// \param[in] M - parallel bilinear form for mass matrix
+   /// \param[in] K - parallel bilinear form for stiffness matrix
+   ParAdvectionDiffusionEvolution(mfem::ParBilinearForm &M,
+                                  mfem::ParBilinearForm &K);
+
+   /// Perform the action of the operator: y = k = f(x, t), where k solves
+   /// Compute k = M^-1(Kx + l)
+   void Mult(const mfem::Vector &x, mfem::Vector &y) const override;
+
+   /// Solve the implicit equation: k = f(x + dt k, t), for the unknown k at
+   /// the current time t. 
+   void ImplicitSolve(const double dt, const mfem::Vector &x,
+                      mfem::Vector &k) override;
+
+   virtual ~ParAdvectionDiffusionEvolution();
+
+private:
+   mfem::OperatorHandle M_;
+   mfem::ParBilinearForm &M, &K;
+   /// solver for inverting mass matrix for explicit time-marching
+   std::unique_ptr<mfem::Solver> M_prec;
+   mfem::CGSolver M_solver;
+   /// solver for implicit time-marching
+   mfem::Solver *prec;
+   mfem::GMRESSolver linear_solver;
+   mfem::NewtonSolver newton;
+
+   mfem::Vector diag;
+   mutable mfem::Vector z, work, work2;
+
+   /// pointer-to-implementation idiom
+   /// Hides implementation details of this operator
+   /// Operator that combines the linear spatial discretization with
+   /// the load vector into one operator used for implicit solves
+   std::unique_ptr<ParSystemOperator> combined_oper;
+
+   /// sets the state and dt for the combined operator
+   /// \param[in] dt - time increment
+   /// \param[in] x - the current state
+   void setOperParameters(double dt, const mfem::Vector *x);
+
+};
+
+// Choice for the problem setup. The fluid velocity, initial condition and
+// inflow boundary condition are chosen based on this parameter.
+int problem;
+
+// Velocity coefficient
+void velocity_function(const Vector &X, Vector &v);
+
+// Initial condition
+double u0_function(const Vector &X);
+
+// Inflow boundary condition
+double inflow_function(const Vector &X, const double t);
+
+// Mesh bounding box
+Vector bb_min, bb_max;
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   problem = 3;
+   const char *mesh_file = "../data/periodic-square.mesh";
+   int ser_ref_levels = 0;
+   int par_ref_levels = 0;
+   int order = 3;
+   const char *device_config = "cpu";
+   int ode_solver_type = 22;
+   double t_final = 3 * 2*M_PI;
+   double dt = 0.01;
+   bool glvis = false;
+   bool paraview = false;
+   int vis_steps = 5;
+
+   double nu_val = 0.001;
+
+   int precision = 8;
+   cout.precision(precision);
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&problem, "-p", "--problem",
+                  "Problem setup to use. See options in velocity_function().");
+   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
+                  "Number of times to refine the mesh uniformly in serial.");
+   args.AddOption(&par_ref_levels, "-rp", "--refine-parallel",
+                  "Number of times to refine the mesh uniformly in parallel.");
+   args.AddOption(&order, "-o", "--order",
+                  "Order (degree) of the finite elements.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
+   args.AddOption(&ode_solver_type, "-s", "--ode-solver",
+                  "ODE solver: 1 - Forward Euler,\n\t"
+                  "            2 - RK2 SSP, 3 - RK3 SSP, 4 - RK4, 6 - RK6.");
+   args.AddOption(&t_final, "-tf", "--t-final",
+                  "Final time; start time is 0.");
+   args.AddOption(&dt, "-dt", "--time-step",
+                  "Time step.");
+   args.AddOption(&glvis, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.AddOption(&paraview, "-paraview", "--paraview-datafiles", "-no-paraview",
+                  "--no-paraview-datafiles",
+                  "Save data files for ParaView (paraview.org) visualization.");
+   args.AddOption(&vis_steps, "-vs", "--visualization-steps",
+                  "Visualize every n-th timestep.");
+   args.AddOption(&nu_val, "-nu", "--nu-value",
+                  "Value for \nu, the parameter that controls diffusion.");
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      std::cout << "Num ranks: " << num_procs << "\n";
+      args.PrintOptions(cout);
+   }
+   Device device(device_config);
+   if (myid == 0) { device.Print(); }
+
+   // 3. Read the serial mesh from the given mesh file on all processors. We can
+   //    handle geometrically periodic meshes in this code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 5. Refine the mesh in serial to increase the resolution. In this example
+   //    we do 'ser_ref_levels' of uniform refinement, where 'ser_ref_levels' is
+   //    a command-line parameter. If the mesh is of NURBS type, we convert it
+   //    to a (piecewise-polynomial) high-order mesh.
+   for (int lev = 0; lev < ser_ref_levels; lev++)
+   {
+      mesh->UniformRefinement();
+   }
+   mesh->GetBoundingBox(bb_min, bb_max, max(order, 1));
+
+   // 6. Define the parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   for (int lev = 0; lev < par_ref_levels; lev++)
+   {
+      pmesh->UniformRefinement();
+   }
+
+   // 7. Define the finite element space of the given
+   //    polynomial order on the refined mesh.
+   H1_FECollection fec(order, dim, BasisType::GaussLobatto);
+   ParFiniteElementSpace *fes = new ParFiniteElementSpace(pmesh, &fec);
+
+   HYPRE_Int global_vSize = fes->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of unknowns: " << global_vSize << endl;
+   }
+
+   // 8. Set up and assemble the bilinear and linear forms corresponding to the
+   //    CG discretization.
+   /// negative to move the diffusion terms to the right side
+   ConstantCoefficient nu(-nu_val);
+   ConstantCoefficient one(1.0);
+   VectorFunctionCoefficient velocity(dim, velocity_function);
+   FunctionCoefficient u0(u0_function);
+
+   ParBilinearForm *m_pa = new ParBilinearForm(fes);
+   ParBilinearForm *k_pa = new ParBilinearForm(fes);
+   m_pa->SetAssemblyLevel(AssemblyLevel::PARTIAL);
+   k_pa->SetAssemblyLevel(AssemblyLevel::PARTIAL);
+
+   /// create mass matrix
+   m_pa->AddDomainIntegrator(new MassIntegrator(one));
+   /// add advection terms to stiffness matrix
+   k_pa->AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
+   /// add diffusion terms to stiffness matrix
+   k_pa->AddDomainIntegrator(new DiffusionIntegrator(nu));
+
+   m_pa->Assemble();
+   int skip_zeros = 0;
+   k_pa->Assemble(skip_zeros);
+   m_pa->Finalize();
+   k_pa->Finalize(skip_zeros);
+
+   ParBilinearForm *m = new ParBilinearForm(fes);
+   ParBilinearForm *k = new ParBilinearForm(fes);
+   /// create mass matrix
+   m->AddDomainIntegrator(new MassIntegrator);
+   /// add advection terms to stiffness matrix
+   k->AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
+   /// add diffusion terms to stiffness matrix
+   k->AddDomainIntegrator(new DiffusionIntegrator(nu));
+
+   m->Assemble();
+   k->Assemble(skip_zeros);
+   m->Finalize();
+   k->Finalize(skip_zeros);
+
+
+   ParGridFunction *u = new ParGridFunction(fes);
+   u->UseDevice(true);
+   u->ProjectCoefficient(u0);
+
+
+   HypreParVector *U = u->GetTrueDofs();
+
+   ParSystemOperator pso(*m, *k);
+   ParSystemOperator pso_pa(*m_pa, *k_pa);
+
+   pso.setParameters(dt, U);
+   pso_pa.setParameters(dt, U);
+
+   MPI_Barrier(MPI_COMM_WORLD);
+   mfem::Vector pso_r(U->Size());
+   double t1 = MPI_Wtime();
+   pso.Mult(*U, pso_r);
+   double t2 = MPI_Wtime();
+   double fa_mult_time = t2 - t1;
+   double average_fa_mult_time;
+   MPI_Reduce(&fa_mult_time, &average_fa_mult_time, 1,
+              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
+   if (myid == 0)
+      std::cout << "FA Mult time: " << average_fa_mult_time / num_procs << endl;
+
+   MPI_Barrier(MPI_COMM_WORLD);
+   mfem::Vector pso_pa_r(U->Size());
+   double t3 = MPI_Wtime();
+   pso_pa.Mult(*U, pso_pa_r);
+   double t4 = MPI_Wtime();
+   double pa_mult_time = t4 - t3;
+   double average_pa_mult_time;
+   MPI_Reduce(&pa_mult_time, &average_pa_mult_time, 1,
+              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
+   if (myid == 0)
+      std::cout << "FA Mult time: " << average_pa_mult_time / num_procs << endl;
+   
+   double local_mult_speedup = (t2-t1) / (t4-t3);
+   double global_mult_speedup;
+   MPI_Reduce(&local_mult_speedup, &global_mult_speedup, 1,
+              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
+
+   if (myid == 0)
+      std::cout << "PA mult speedup: " << global_mult_speedup / num_procs << endl;
+
+   mfem::Vector diff_r(pso_pa_r);
+   diff_r -= pso_r;
+   // std::cout << "r diff: " << diff_r.Norml2() << std::endl;
+
+   mfem::Operator &pso_jac = pso.GetGradient(*U);
+   mfem::Operator &pso_pa_jac = pso_pa.GetGradient(*U);
+
+   MPI_Barrier(MPI_COMM_WORLD);
+   mfem::Vector pso_jac_r(U->Size());
+   double t5 = MPI_Wtime();
+   pso_jac.Mult(*U, pso_jac_r);
+   double t6 = MPI_Wtime();
+   double fa_jac_mult_time = t6-t5;
+   double average_fa_jac_time;
+   MPI_Reduce(&fa_jac_mult_time, &average_fa_jac_time, 1,
+              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
+   if (myid == 0)
+      std::cout << "FA Jac Mult time: " << average_fa_jac_time / num_procs << endl;
+
+   MPI_Barrier(MPI_COMM_WORLD);
+   mfem::Vector pso_pa_jac_r(U->Size());
+   double t7 = MPI_Wtime();
+   pso_pa_jac.Mult(*U, pso_pa_jac_r);
+   double t8 = MPI_Wtime();
+   double pa_jac_mult_time = t8-t7;
+   double average_pa_jac_time;
+   MPI_Reduce(&pa_jac_mult_time, &average_pa_jac_time, 1,
+              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
+   if (myid == 0)
+      std::cout << "PA Jac Mult time: " << average_pa_jac_time / num_procs << endl;
+
+   double local_jac_speedup = (t6-t5) / (t8-t7);
+   double global_jac_speedup;
+   MPI_Reduce(&local_jac_speedup, &global_jac_speedup, 1,
+              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
+   if (myid == 0)
+      std::cout << "PA Jac mult speedup: " << global_jac_speedup / num_procs << endl;
+
+   // 13. Free the used memory.
+   delete U;
+   delete u;
+   delete k;
+   delete m;
+   delete fes;
+   delete pmesh;
+
+   MPI_Finalize();
+   return 0;
+}
+
+// Velocity coefficient
+void velocity_function(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+
+   // map to the reference [-1,1] domain
+   Vector X(dim);
+   for (int i = 0; i < dim; i++)
+   {
+      double center = (bb_min[i] + bb_max[i]) * 0.5;
+      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
+   }
+
+   switch (problem)
+   {
+      case 3:
+      {
+         // Translations in 1D, 2D, and 3D
+         switch (dim)
+         {
+            case 1: v(0) = 1.0; break;
+            case 2: v(0) = sqrt(2./3.); v(1) = sqrt(1./3.); break;
+            case 3: v(0) = sqrt(3./6.); v(1) = sqrt(2./6.); v(2) = sqrt(1./6.);
+               break;
+         }
+         break;
+      }
+      case 1:
+      case 2:
+      {
+         // Clockwise rotation in 2D around the origin
+         const double w = M_PI/2;
+         switch (dim)
+         {
+            case 1: v(0) = 1.0; break;
+            case 2: v(0) = w*X(1); v(1) = -w*X(0); break;
+            case 3: v(0) = w*X(1); v(1) = -w*X(0); v(2) = 0.0; break;
+         }
+         break;
+      }
+      case 0:
+      {
+         // Clockwise twisting rotation in 2D around the origin
+         const double w = M_PI/2;
+         double d = max((X(0)+1.)*(1.-X(0)),0.) * max((X(1)+1.)*(1.-X(1)),0.);
+         d = d*d;
+         switch (dim)
+         {
+            case 1: v(0) = 1.0; break;
+            case 2: v(0) = d*w*X(1); v(1) = -d*w*X(0); break;
+            case 3: v(0) = d*w*X(1); v(1) = -d*w*X(0); v(2) = 0.0; break;
+         }
+         break;
+      }
+   }
+}
+
+// Initial condition
+double u0_function(const Vector &x)
+{
+   int dim = x.Size();
+
+   // map to the reference [-1,1] domain
+   Vector X(dim);
+   for (int i = 0; i < dim; i++)
+   {
+      double center = (bb_min[i] + bb_max[i]) * 0.5;
+      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
+   }
+
+   switch (problem)
+   {
+      case 0:
+      case 1:
+      {
+         switch (dim)
+         {
+            case 1:
+               return exp(-40.*pow(X(0)-0.5,2));
+            case 2:
+            case 3:
+            {
+               double rx = 0.45, ry = 0.25, cx = 0., cy = -0.2, w = 10.;
+               if (dim == 3)
+               {
+                  const double s = (1. + 0.25*cos(2*M_PI*X(2)));
+                  rx *= s;
+                  ry *= s;
+               }
+               return ( erfc(w*(X(0)-cx-rx))*erfc(-w*(X(0)-cx+rx)) *
+                        erfc(w*(X(1)-cy-ry))*erfc(-w*(X(1)-cy+ry)) )/16;
+            }
+         }
+      }
+      case 2:
+      {
+         double x_ = X(0), y_ = X(1), rho, phi;
+         rho = hypot(x_, y_);
+         phi = atan2(y_, x_);
+         return pow(sin(M_PI*rho),2)*sin(3*phi);
+      }
+      case 3:
+      {
+         const double f = M_PI;
+         return sin(f*X(0))*sin(f*X(1));
+      }
+   }
+   return 0.0;
+}
+
+// Inflow boundary condition (zero for the problems considered in this example)
+double inflow_function(const Vector &x, const double t)
+{
+   switch (problem)
+   {
+      case 0:
+      case 1:
+      case 2:
+      case 3: return 0.0;
+   }
+   return 0.0;
+}
+
+class AdvectionDiffusionEvolution::SystemOperator : public mfem::Operator
+{
+public:
+   /// Nonlinear operator of the form that combines the mass, res, stiff,
+   /// and load elements for implicit/explicit ODE integration
+   /// \param[in] mass - bilinear form for mass matrix (not owned)
+   /// \param[in] res - nonlinear residual operator (not owned)
+   /// \param[in] stiff - bilinear form for stiffness matrix (not owned)
+   /// \param[in] load - load vector (not owned)
+   /// \param[in] a - used to move the spatial residual to the rhs
+   SystemOperator(BilinearForm &_mass, BilinearForm &_stiff,
+                  const mfem::Vector &b)
+      : Operator(_mass.Height()), mass(_mass), stiff(_stiff),
+        load(b), Jacobian(NULL), dt(0.0), x(NULL), work(height)
+      { }
+
+   /// Compute r = M@k + K@(x+dt*k) + l
+   /// (with `@` denoting matrix-vector multiplication)
+   /// \param[in] k - dx/dt 
+   /// \param[out] r - the residual
+   /// \note the signs on each operator must be accounted for elsewhere
+   void Mult(const mfem::Vector &k, mfem::Vector &r) const override
+   {
+      /// work = x+dt*k = x+dt*dx/dt = x+dx
+      add(1.0, *x, dt, k, work);
+      r = 0.0;
+      stiff.AddMult(work, r);
+      r += load;
+      mass.AddMult(k, r, -1.0);
+   }
+
+   /// Compute J = M + dt * K
+   /// \param[in] k - dx/dt 
+   mfem::Operator &GetGradient(const mfem::Vector &k) const override
+   {
+      delete Jacobian;
+      Jacobian = Add(-1.0, mass.SpMat(), dt, stiff.SpMat());
+      return *Jacobian;
+   }
+
+   /// Set current dt and x values - needed to compute action and Jacobian.
+   void setParameters(double _dt, const mfem::Vector *_x)
+   {
+      dt = _dt;
+      x = _x;
+   };
+
+   ~SystemOperator() {delete Jacobian;};
+
+private:
+   BilinearForm &mass;
+   BilinearForm &stiff;
+   const mfem::Vector &load;
+   mutable mfem::SparseMatrix *Jacobian;
+
+   double dt;
+   const mfem::Vector *x;
+
+   mutable mfem::Vector work, work2;
+
+};
+
+AdvectionDiffusionEvolution::AdvectionDiffusionEvolution(
+      BilinearForm &_M, BilinearForm &_K, const Vector &_b)
+   : TimeDependentOperator(_M.Height()), M(_M), K(_K), b(_b),
+     z(_M.Height())
+{
+   bool pa = M.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
+   Array<int> ess_tdof_list;
+   if (pa)
+   {
+      M_prec.reset(new OperatorJacobiSmoother(M, ess_tdof_list));
+      M_solver.SetOperator(M);
+   }
+   else
+   {
+      M_prec.reset(new DSmoother(M.SpMat()));
+      M_solver.SetOperator(M.SpMat());
+   }
+
+   combined_oper.reset(new SystemOperator(_M, _K, _b));
+
+   M_solver.SetPreconditioner(*M_prec);
+   M_solver.iterative_mode = false;
+   M_solver.SetRelTol(1e-9);
+   M_solver.SetAbsTol(0.0);
+   M_solver.SetMaxIter(100);
+   M_solver.SetPrintLevel(0);
+
+   linear_solver.iterative_mode = true;
+   linear_solver.SetRelTol(1e-12);
+   linear_solver.SetAbsTol(0.0);
+   linear_solver.SetMaxIter(100);
+   linear_solver.SetPrintLevel(0);
+   linear_solver.SetPreconditioner(prec);
+
+   newton.iterative_mode = false;
+   newton.SetRelTol(1e-9);
+   newton.SetAbsTol(0.0);
+   newton.SetMaxIter(100);
+   newton.SetPrintLevel(-1);
+   newton.SetSolver(linear_solver);
+   newton.SetOperator(*combined_oper);
+}
+
+void AdvectionDiffusionEvolution::Mult(const Vector &x, Vector &y) const
+{
+   // y = M^{-1} (K x + b)
+   K.Mult(x, z);
+   z += b;
+   M_solver.Mult(z, y);
+}
+
+void AdvectionDiffusionEvolution::ImplicitSolve(const double dt,
+                                                const Vector &x,
+                                                Vector &k)
+{
+   setOperParameters(dt, &x);
+   Vector zero; // empty vector is interpreted as zero r.h.s. by NewtonSolver
+   newton.Mult(zero, k);
+   MFEM_VERIFY(newton.GetConverged(), "Newton solver did not converge!");
+}
+
+void AdvectionDiffusionEvolution::setOperParameters(double dt,
+                                                    const mfem::Vector *x)
+{
+   combined_oper->setParameters(dt, x);
+}
+
+AdvectionDiffusionEvolution::~AdvectionDiffusionEvolution() {}
+
+
+PAJacobianOperator::PAJacobianOperator(ParBilinearForm &_mass, ParBilinearForm &_stiff)
+ : Operator(_mass.ParFESpace()->GetTrueVSize()), mass(_mass), stiff(_stiff),
+   dt(0.0) { }
+
+
+void PAJacobianOperator::Mult(const mfem::Vector &k, mfem::Vector &r) const
+{
+   r.UseDevice(true);
+   r = 0.0;
+   stiff.TrueAddMult(k, r, dt);
+   mass.TrueAddMult(k, r, -1.0);
+}
+   
+void PAJacobianOperator::setParameters(const double _dt)
+{
+   dt = _dt;
+};
+
+ParSystemOperator::ParSystemOperator(ParBilinearForm &_mass, ParBilinearForm &_stiff)
+   : Operator(_mass.ParFESpace()->GetTrueVSize()), mass(_mass), stiff(_stiff),
+   jacobian(NULL), stiff_jacobian(NULL), dt(0.0), x(NULL),
+   work(height)
+{
+   pa_jac.reset(new PAJacobianOperator(mass, stiff));
+}
+
+/// Compute r = M@k + K@(x+dt*k)
+/// (with `@` denoting matrix-vector multiplication)
+/// \param[in] k - dx/dt 
+/// \param[out] r - the residual
+/// \note the signs on each operator must be accounted for elsewhere
+void ParSystemOperator::Mult(const mfem::Vector &k, mfem::Vector &r) const
+{
+   r = 0.0;
+   work.UseDevice(true);
+   work = 0.0;
+   /// work = x+dt*k = x+dt*dx/dt = x+dx
+   if (x)
+   {
+      add(1.0, *x, dt, k, work);
+   }
+
+   stiff.TrueAddMult(work, r);
+   mass.TrueAddMult(k, r, -1.0);
+}
+
+/// Compute J = M + dt * K
+/// \param[in] k - dx/dt 
+mfem::Operator &ParSystemOperator::GetGradient(const mfem::Vector &k) const
+{
+   bool mass_pa = mass.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
+   bool stiff_pa = stiff.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
+   
+   if (mass_pa && stiff_pa)
+   {
+      return *pa_jac.get();
+   }
+   else
+   {
+      delete stiff_jacobian;
+      delete jacobian;
+      jacobian = mass.ParallelAssemble();
+      *jacobian *= -1.0; //alpha;
+      stiff_jacobian = stiff.ParallelAssemble();
+      jacobian->Add(dt, *stiff_jacobian);
+      return *jacobian;
+   }
+}
+
+/// Set current dt and x values - needed to compute action and Jacobian.
+void ParSystemOperator::setParameters(const double _dt, const mfem::Vector *_x)
+{
+   dt = _dt;
+   x = _x;
+   pa_jac->setParameters(_dt);
+};
+
+ParSystemOperator::~ParSystemOperator()
+{
+   delete jacobian;
+   delete stiff_jacobian;
+};
+
+ParAdvectionDiffusionEvolution::ParAdvectionDiffusionEvolution(
+      ParBilinearForm &_M, ParBilinearForm &_K)
+   : TimeDependentOperator(_M.ParFESpace()->GetTrueVSize()), M(_M), K(_K), z(_M.Height())
+{
+   bool mass_pa = M.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
+   bool stiff_pa = K.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
+
+   Array<int> ess_tdof_list;
+   M_solver = CGSolver(MPI_COMM_WORLD);
+   if (mass_pa)
+   {
+      M_prec.reset(new OperatorJacobiSmoother(M, ess_tdof_list));
+      M_solver.SetOperator(M);
+   }
+   else
+   {
+      M_.Reset(_M.ParallelAssemble(), true);
+
+      // M_prec.reset(new HypreSmoother());
+      // M_solver.SetOperator(M.As<HypreParMatrix>());
+      HypreParMatrix &M_mat = *M_.As<HypreParMatrix>();
+      // HypreParMatrix &K_mat = *K.As<HypreParMatrix>();
+      M_prec.reset(new HypreSmoother(M_mat, HypreSmoother::Jacobi));
+   }
+
+   combined_oper.reset(new ParSystemOperator(_M, _K));
+
+   M_solver.SetPreconditioner(*M_prec);
+   M_solver.iterative_mode = false;
+   M_solver.SetRelTol(1e-9);
+   M_solver.SetAbsTol(0.0);
+   M_solver.SetMaxIter(100);
+   M_solver.SetPrintLevel(0);
+
+   if (mass_pa && stiff_pa)
+   {
+      diag.UseDevice(true);
+      diag.SetSize(M.ParFESpace()->GetTrueVSize());
+      diag = 0.0;
+      work.UseDevice(true);
+      work2.UseDevice(true);
+      work.SetSize(M.ParFESpace()->GetTrueVSize());
+      work2.SetSize(M.ParFESpace()->GetTrueVSize());
+      work = 0.0;
+      work2 = 0.0;
+      M.AssembleDiagonal(work);
+
+      ParBilinearForm k(M.ParFESpace());
+      ConstantCoefficient nu(-0.01);
+      k.AddDomainIntegrator(new mfem::DiffusionIntegrator(nu));
+      k.SetAssemblyLevel(AssemblyLevel::PARTIAL);
+      k.Assemble(0);
+      k.Finalize(0);
+      k.AssembleDiagonal(work2);
+
+      double dt = 0.1;
+      add(-1.0, work, dt, work2, diag);
+
+      prec = new OperatorChebyshevSmoother(combined_oper.get(), diag,
+                                           ess_tdof_list, 5,
+                                           M.ParFESpace()->GetComm());
+   }
+   else
+   {
+      prec = new HypreSmoother();
+   }
+   
+   linear_solver = GMRESSolver(MPI_COMM_WORLD);
+   linear_solver.iterative_mode = true;
+   linear_solver.SetRelTol(1e-12);
+   linear_solver.SetAbsTol(0.0);
+   linear_solver.SetMaxIter(2000);
+   linear_solver.SetPrintLevel(0);
+   linear_solver.SetPreconditioner(*prec);
+   linear_solver.SetKDim(2000);
+
+   newton.iterative_mode = true;
+   newton.SetRelTol(1e-9);
+   newton.SetAbsTol(0.0);
+   newton.SetMaxIter(10);
+   newton.SetPrintLevel(-1);
+   newton.SetSolver(linear_solver);
+   newton.SetOperator(*combined_oper);
+}
+
+void ParAdvectionDiffusionEvolution::Mult(const Vector &x, Vector &y) const
+{
+   // y = M^{-1} (K x + b)
+   K.Mult(x, z);
+   M_solver.Mult(z, y);
+}
+
+void ParAdvectionDiffusionEvolution::ImplicitSolve(const double dt,
+                                                   const Vector &x,
+                                                   Vector &k)
+{
+   setOperParameters(dt, &x);
+   Vector zero; // empty vector is interpreted as zero r.h.s. by NewtonSolver
+   newton.Mult(zero, k);
+   MFEM_VERIFY(newton.GetConverged(), "Newton solver did not converge!");
+}
+
+void ParAdvectionDiffusionEvolution::setOperParameters(const double dt,
+                                                       const mfem::Vector *x)
+{
+   combined_oper->setParameters(dt, x);
+}
+
+ParAdvectionDiffusionEvolution::~ParAdvectionDiffusionEvolution() {delete prec;}

fem/CMakeLists.txt

@@ -36,9 +36,11 @@ set(SRCS
   intrules.cpp
   linearform.cpp
   lininteg.cpp
+  multigrid.cpp
   nonlinearform.cpp
   nonlinearform_ext.cpp
   nonlininteg.cpp
+  fespacehierarchy.cpp
   nonlininteg_vectorconvection.cpp
   quadinterpolator.cpp
   quadinterpolator_face.cpp
@@ -47,6 +49,7 @@ set(SRCS
   tmop.cpp
   tmop_tools.cpp
   gslib.cpp
+  transfer.cpp
   )
 
 set(HDRS
@@ -68,12 +71,14 @@ set(HDRS
   intrules.hpp
   linearform.hpp
   lininteg.hpp
+  multigrid.hpp
   nonlinearform.hpp
   nonlinearform_ext.hpp
   nonlininteg.hpp
   quadinterpolator.hpp
   quadinterpolator_face.hpp
   restriction.hpp
+  fespacehierarchy.hpp
   staticcond.hpp
   tbilinearform.hpp
   tbilininteg.hpp
@@ -86,6 +91,7 @@ set(HDRS
   tmop.hpp
   tmop_tools.hpp
   gslib.hpp
+  transfer.hpp
   )
 
 if (MFEM_USE_SIDRE)
@@ -98,6 +104,11 @@ if (MFEM_USE_CONDUIT)
   list(APPEND HDRS conduitdatacollection.hpp)
 endif()
 
+if (MFEM_USE_ADIOS2)
+  list(APPEND SRCS adios2datacollection.cpp)
+  list(APPEND HDRS adios2datacollection.hpp)
+endif()
+
 if (MFEM_USE_MPI)
   list(APPEND SRCS
     pbilinearform.cpp

fem/adios2datacollection.cpp

@@ -0,0 +1,90 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+//
+// Created on: Jan 7, 2020
+// Author: William F Godoy godoywf@ornl.gov
+// adios2: Adaptable Input/Output System https://github.com/ornladios/ADIOS2
+
+#include "adios2datacollection.hpp"
+
+namespace mfem
+{
+
+#ifdef MFEM_USE_MPI
+ADIOS2DataCollection::ADIOS2DataCollection(MPI_Comm comm,
+                                           const std::string& collection_name, Mesh* mesh,
+                                           const std::string engine_type) : DataCollection(collection_name, mesh),
+   stream( new adios2stream(name, adios2stream::openmode::out, comm, engine_type) )
+{
+   SetMesh(mesh);
+}
+#else
+ADIOS2DataCollection::ADIOS2DataCollection(
+   const std::string& collection_name, Mesh* mesh,
+   const std::string engine_type): DataCollection(collection_name, mesh),
+   stream( new adios2stream(name, adios2stream::openmode::out, engine_type) )
+{
+   SetMesh(mesh);
+}
+#endif
+
+ADIOS2DataCollection::~ADIOS2DataCollection()
+{
+   stream->Close();
+}
+
+void ADIOS2DataCollection::Save()
+{
+   stream->BeginStep();
+
+   // only save mesh once (moving mesh, not yet supported)
+   if (stream->CurrentStep() == 0)
+   {
+      if (mesh == nullptr)
+      {
+         const std::string error_message =
+            "MFEM ADIOS2DataCollection Save error: Mesh is null. Please call SetMesh before Save\n";
+         mfem_error(error_message.c_str());
+      }
+      stream->Print(*mesh);
+   }
+
+   // reduce footprint
+   if (myid == 0)
+   {
+      stream->SetTime(time);
+      stream->SetCycle(cycle);
+   }
+
+   for (const auto& field : field_map)
+   {
+      const std::string& variable_name = field.first;
+      field.second->Save(*stream.get(), variable_name);
+   }
+
+   stream->EndStep();
+}
+
+void ADIOS2DataCollection::SetParameter(const std::string key,
+                                        const std::string value) noexcept
+{
+   stream->SetParameter(key, value);
+}
+
+void ADIOS2DataCollection::SetLevelsOfDetail(const int levels_of_detail)
+noexcept
+{
+   stream->SetRefinementLevel(levels_of_detail);
+}
+
+} //end namespace mfem
+
+

fem/adios2datacollection.hpp

@@ -0,0 +1,88 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+//
+// Created on: Jan 7, 2020
+// Author: William F Godoy godoywf@ornl.gov
+// adios2: Adaptable Input/Output System https://github.com/ornladios/ADIOS2
+
+#ifndef MFEM_ADIOS2DATACOLLECTION
+#define MFEM_ADIOS2DATACOLLECTION
+
+#include "../config/config.hpp"
+#include "../general/adios2stream.hpp"
+#include "datacollection.hpp"
+
+#include <memory>  // std::unique_ptr
+#include <string>
+
+namespace mfem
+{
+
+class ADIOS2DataCollection : public DataCollection
+{
+public:
+#ifdef MFEM_USE_MPI
+   /**
+    * Parallel constructor. Important: scope of this object must be within
+    * MPI_Init and MPI_Finalize otherwise. The destructor will call the Close
+    * function. Either object must live in a try/catch block (inside try) or use
+    * raw pointers calling delete before MPI_Finalize.
+    * @param comm MPI communicator setting the datacollection domain
+    * @param collection_name unique name for saving data
+    * @param mesh can be set at the constructor level or later by calling
+    * SetMesh()
+    * @param engine_type adios2 engine type
+    */
+   ADIOS2DataCollection(MPI_Comm comm, const std::string& collection_name,
+                        Mesh* mesh = nullptr,
+                        const std::string engine_type = "BPFile");
+#else
+   /**
+    * Serial constructor
+    * @param collection_name unique name for saving data
+    * @param mesh can be set at the constructor level or later by calling
+    * SetMesh()
+    * @param engine_type adios2 engine type
+    * @throws std::invalid_argument (user input error) or std::runtime_error
+    *         (system error)
+    */
+   ADIOS2DataCollection(const std::string& collection_name, Mesh* mesh = nullptr,
+                        const std::string engine_type = "BPFile");
+#endif
+
+   virtual ~ADIOS2DataCollection();
+
+   /** Save the collection */
+   virtual void Save();
+
+   /**
+    * Pass a parameter unique to adios2datacollection
+    * For available parameters:
+    * See https://adios2.readthedocs.io/en/latest/engines/engines.html
+    * The most common is: key=SubStreams value=1 to nprocs (MPI processes)
+    * @param key parameter key
+    * @param value parameter value
+    */
+   void SetParameter(const std::string key, const std::string value) noexcept;
+
+   /**
+    * Sets the levels of detail for the global grid refinement
+    * @param levels_of_detail (default = 1)
+    */
+   void SetLevelsOfDetail(const int levels_of_detail) noexcept;
+
+private:
+   std::unique_ptr<adios2stream> stream;
+};
+
+}  // namespace mfem
+
+#endif /* MFEM_ADIOS2DATACOLLECTION */

fem/bilinearform.cpp

@@ -467,8 +467,17 @@ void BilinearForm::Assemble(int skip_zeros)
          const FiniteElement &be = *fes->GetBE(i);
          fes -> GetBdrElementVDofs (i, vdofs);
          eltrans = fes -> GetBdrElementTransformation (i);
-         bbfi[0]->AssembleElementMatrix(be, *eltrans, elmat);
-         for (int k = 1; k < bbfi.Size(); k++)
+         int k = 0;
+         for (; k < bbfi.Size(); k++)
+         {
+            if (bbfi_marker[k] &&
+                (*bbfi_marker[k])[bdr_attr-1] == 0) { continue; }
+
+            bbfi[k]->AssembleElementMatrix(be, *eltrans, elmat);
+            k++;
+            break;
+         }
+         for (; k < bbfi.Size(); k++)
          {
             if (bbfi_marker[k] &&
                 (*bbfi_marker[k])[bdr_attr-1] == 0) { continue; }

fem/bilinearform.hpp

@@ -151,8 +151,8 @@ public:
    /** This method must be called before assembly. */
    void SetAssemblyLevel(AssemblyLevel assembly_level);
 
-   /// Get the assembly level
-   AssemblyLevel GetAssemblyLevel() {return assembly;}
+   /// Returns the assembly level
+   AssemblyLevel GetAssemblyLevel() const { return assembly; }
 
    /** Enable the use of static condensation. For details see the description
        for class StaticCondensation in fem/staticcond.hpp This method should be
@@ -530,7 +530,7 @@ public:
 
    /// (DEPRECATED) Return the FE space associated with the BilinearForm.
    /** @deprecated Use FESpace() instead. */
-   FiniteElementSpace *GetFES() { return fes; }
+   MFEM_DEPRECATED FiniteElementSpace *GetFES() { return fes; }
 
    /// Return the FE space associated with the BilinearForm.
    FiniteElementSpace *FESpace() { return fes; }

fem/bilininteg.cpp

@@ -2135,17 +2135,17 @@ void VectorDiffusionIntegrator::AssembleElementMatrix(
    ElementTransformation &Trans,
    DenseMatrix &elmat)
 {
-   int dim = el.GetDim();
-   int dof = el.GetDof();
+   const int dim = el.GetDim();
+   const int dof = el.GetDof();
+   const int sdim = Trans.GetSpaceDim();
+   const bool square = (dim == sdim);
+   double w;
 
-   double norm;
+   elmat.SetSize(sdim * dof);
 
-   elmat.SetSize (dim * dof);
-
-   Jinv.  SetSize (dim);
-   dshape.SetSize (dof, dim);
-   gshape.SetSize (dof, dim);
-   pelmat.SetSize (dof);
+   dshape.SetSize(dof, dim);
+   dshapedxt.SetSize(dof, sdim);
+   pelmat.SetSize(dof);
 
    const IntegrationRule *ir = IntRule;
    if (ir == NULL)
@@ -2163,35 +2163,29 @@ void VectorDiffusionIntegrator::AssembleElementMatrix(
    }
 
    elmat = 0.0;
+   pelmat = 0.0;
 
    for (int i = 0; i < ir -> GetNPoints(); i++)
    {
       const IntegrationPoint &ip = ir->IntPoint(i);
-
       el.CalcDShape (ip, dshape);
-
       Trans.SetIntPoint (&ip);
-      norm = ip.weight * Trans.Weight();
-      CalcInverse (Trans.Jacobian(), Jinv);
-
-      Mult (dshape, Jinv, gshape);
-
-      MultAAt (gshape, pelmat);
-
-      if (Q)
+      w = Trans.Weight();
+      w = ip.weight / (square ? w : w*w*w);
+      // AdjugateJacobian = / adj(J),         if J is square
+      //                    \ adj(J^t.J).J^t, otherwise
+      Mult(dshape, Trans.AdjugateJacobian(), dshapedxt);
+      if (Q) {  w *= Q -> Eval (Trans, ip); }
+      AddMult_a_AAt(w, dshapedxt, pelmat);
+   }
+   for (int d = 0; d < sdim; d++)
+   {
+      for (int k = 0; k < dof; k++)
       {
-         norm *= Q -> Eval (Trans, ip);
-      }
-
-      pelmat *= norm;
-
-      for (int d = 0; d < dim; d++)
-      {
-         for (int k = 0; k < dof; k++)
-            for (int l = 0; l < dof; l++)
-            {
-               elmat (dof*d+k, dof*d+l) += pelmat (k, l);
-            }
+         for (int l = 0; l < dof; l++)
+         {
+            elmat(dof*d+k, dof*d+l) = pelmat(k, l);
+         }
       }
    }
 }

fem/bilininteg.hpp

@@ -453,6 +453,16 @@ public:
                                        ElementTransformation &Trans,
                                        DenseMatrix &elmat);
 
+   /// Support for use in BilinearForm. Can be used only when appropriate.
+   /** Appropriate use cases are classes derived from
+       MixedScalarVectorIntegrator where the trial and test spaces can be the
+       same. Examples of such classes are: MixedVectorDivergenceIntegrator,
+       MixedScalarWeakDivergenceIntegrator, etc. */
+   virtual void AssembleElementMatrix(const FiniteElement &fe,
+                                      ElementTransformation &Trans,
+                                      DenseMatrix &elmat)
+   { AssembleElementMatrix2(fe, fe, Trans, elmat); }
+
 protected:
 
    MixedScalarVectorIntegrator(VectorCoefficient &vq, bool _transpose = false,
@@ -2405,14 +2415,12 @@ protected:
    // PA extension
    const DofToQuad *maps;         ///< Not owned
    const GeometricFactors *geom;  ///< Not owned
-   int dim, ne, dofs1D, quad1D;
+   int dim, sdim, ne, dofs1D, quad1D;
    Vector pa_data;
 
 private:
-   DenseMatrix Jinv;
-   DenseMatrix dshape;
-   DenseMatrix gshape;
-   DenseMatrix pelmat;
+   DenseMatrix dshape, dshapedxt, pelmat;
+   DenseMatrix Jinv, gshape;
 
 public:
    VectorDiffusionIntegrator() { Q = NULL; }

fem/bilininteg_diffusion.cpp

@@ -81,12 +81,20 @@ static void OccaPADiffusionSetup3D(const int D1D,
 #endif // MFEM_USE_OCCA
 
 // PA Diffusion Assemble 2D kernel
+template<const int T_SDIM>
 static void PADiffusionSetup2D(const int Q1D,
                                const int NE,
                                const Array<double> &w,
                                const Vector &j,
                                const Vector &c,
-                               Vector &d)
+                               Vector &d);
+template<>
+void PADiffusionSetup2D<2>(const int Q1D,
+                           const int NE,
+                           const Array<double> &w,
+                           const Vector &j,
+                           const Vector &c,
+                           Vector &d)
 {
    const int NQ = Q1D*Q1D;
    const bool const_c = c.Size() == 1;
@@ -112,6 +120,48 @@ static void PADiffusionSetup2D(const int Q1D,
    });
 }
 
+// PA Diffusion Assemble 2D kernel with 3D node coords
+template<>
+void PADiffusionSetup2D<3>(const int Q1D,
+                           const int NE,
+                           const Array<double> &w,
+                           const Vector &j,
+                           const Vector &c,
+                           Vector &d)
+{
+   constexpr int DIM = 2;
+   constexpr int SDIM = 3;
+   const int NQ = Q1D*Q1D;
+   const bool const_c = c.Size() == 1;
+
+   auto W = w.Read();
+   auto J = Reshape(j.Read(), NQ, SDIM, DIM, NE);
+   auto C = const_c ? Reshape(c.Read(), 1, 1) : Reshape(c.Read(), NQ, NE);
+   auto D = Reshape(d.Write(), NQ, 3, NE);
+   MFEM_FORALL(e, NE,
+   {
+      for (int q = 0; q < NQ; ++q)
+      {
+         const double wq = W[q];
+         const double J11 = J(q,0,0,e);
+         const double J21 = J(q,1,0,e);
+         const double J31 = J(q,2,0,e);
+         const double J12 = J(q,0,1,e);
+         const double J22 = J(q,1,1,e);
+         const double J32 = J(q,2,1,e);
+         const double E = J11*J11 + J21*J21 + J31*J31;
+         const double G = J12*J12 + J22*J22 + J32*J32;
+         const double F = J11*J12 + J21*J22 + J31*J32;
+         const double iw = 1.0 / sqrt(E*G - F*F);
+         const double coeff = const_c ? C(0,0) : C(q,e);
+         const double alpha = wq * coeff * iw;
+         D(q,0,e) =  alpha * G; // 1,1
+         D(q,1,e) = -alpha * F; // 1,2
+         D(q,2,e) =  alpha * E; // 2,2
+      }
+   });
+}
+
 // PA Diffusion Assemble 3D kernel
 static void PADiffusionSetup3D(const int Q1D,
                                const int NE,
@@ -166,6 +216,7 @@ static void PADiffusionSetup3D(const int Q1D,
 }
 
 static void PADiffusionSetup(const int dim,
+                             const int sdim,
                              const int D1D,
                              const int Q1D,
                              const int NE,
@@ -183,8 +234,11 @@ static void PADiffusionSetup(const int dim,
          OccaPADiffusionSetup2D(D1D, Q1D, NE, W, J, C, D);
          return;
       }
+#else
+      MFEM_CONTRACT_VAR(D1D);
 #endif // MFEM_USE_OCCA
-      PADiffusionSetup2D(Q1D, NE, W, J, C, D);
+      if (sdim == 2) { PADiffusionSetup2D<2>(Q1D, NE, W, J, C, D); }
+      if (sdim == 3) { PADiffusionSetup2D<3>(Q1D, NE, W, J, C, D); }
    }
    if (dim == 3)
    {
@@ -217,6 +271,8 @@ void DiffusionIntegrator::SetupPA(const FiniteElementSpace &fes,
       InitCeedCoeff(Q, ptr);
       return CeedPADiffusionAssemble(fes, *ir, *ptr);
    }
+#else
+   MFEM_CONTRACT_VAR(force);
 #endif
    const int dims = el.GetDim();
    const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
@@ -224,6 +280,7 @@ void DiffusionIntegrator::SetupPA(const FiniteElementSpace &fes,
    dim = mesh->Dimension();
    ne = fes.GetNE();
    geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
+   const int sdim = mesh->SpaceDimension();
    maps = &el.GetDofToQuad(*ir, DofToQuad::TENSOR);
    dofs1D = maps->ndof;
    quad1D = maps->nqpt;
@@ -252,8 +309,8 @@ void DiffusionIntegrator::SetupPA(const FiniteElementSpace &fes,
          }
       }
    }
-   PADiffusionSetup(dim, dofs1D, quad1D, ne, ir->GetWeights(), geom->J, coeff,
-                    pa_data);
+   PADiffusionSetup(dim, sdim, dofs1D, quad1D, ne, ir->GetWeights(), geom->J,
+                    coeff, pa_data);
 }
 
 void DiffusionIntegrator::AssemblePA(const FiniteElementSpace &fes)
@@ -910,8 +967,6 @@ template<int T_D1D = 0, int T_Q1D = 0, int T_NBZ = 0>
 static void SmemPADiffusionApply2D(const int NE,
                                    const Array<double> &b_,
                                    const Array<double> &g_,
-                                   const Array<double> &bt_,
-                                   const Array<double> &gt_,
                                    const Vector &d_,
                                    const Vector &x_,
                                    Vector &y_,
@@ -1257,8 +1312,6 @@ template<int T_D1D = 0, int T_Q1D = 0>
 static void SmemPADiffusionApply3D(const int NE,
                                    const Array<double> &b_,
                                    const Array<double> &g_,
-                                   const Array<double> &bt_,
-                                   const Array<double> &gt_,
                                    const Vector &d_,
                                    const Vector &x_,
                                    Vector &y_,
@@ -1525,14 +1578,14 @@ static void PADiffusionApply(const int dim,
    {
       switch ((D1D << 4 ) | Q1D)
       {
-         case 0x22: return SmemPADiffusionApply2D<2,2,16>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x33: return SmemPADiffusionApply2D<3,3,16>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x44: return SmemPADiffusionApply2D<4,4,8>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x55: return SmemPADiffusionApply2D<5,5,8>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x66: return SmemPADiffusionApply2D<6,6,4>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x77: return SmemPADiffusionApply2D<7,7,4>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x88: return SmemPADiffusionApply2D<8,8,2>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x99: return SmemPADiffusionApply2D<9,9,2>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x22: return SmemPADiffusionApply2D<2,2,16>(NE,B,G,D,X,Y);
+         case 0x33: return SmemPADiffusionApply2D<3,3,16>(NE,B,G,D,X,Y);
+         case 0x44: return SmemPADiffusionApply2D<4,4,8>(NE,B,G,D,X,Y);
+         case 0x55: return SmemPADiffusionApply2D<5,5,8>(NE,B,G,D,X,Y);
+         case 0x66: return SmemPADiffusionApply2D<6,6,4>(NE,B,G,D,X,Y);
+         case 0x77: return SmemPADiffusionApply2D<7,7,4>(NE,B,G,D,X,Y);
+         case 0x88: return SmemPADiffusionApply2D<8,8,2>(NE,B,G,D,X,Y);
+         case 0x99: return SmemPADiffusionApply2D<9,9,2>(NE,B,G,D,X,Y);
          default:   return PADiffusionApply2D(NE,B,G,Bt,Gt,D,X,Y,D1D,Q1D);
       }
    }
@@ -1540,15 +1593,15 @@ static void PADiffusionApply(const int dim,
    {
       switch ((D1D << 4 ) | Q1D)
       {
-         case 0x23: return SmemPADiffusionApply3D<2,3>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x34: return SmemPADiffusionApply3D<3,4>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x45: return SmemPADiffusionApply3D<4,5>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x46: return SmemPADiffusionApply3D<4,6>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x56: return SmemPADiffusionApply3D<5,6>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x58: return SmemPADiffusionApply3D<5,8>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x67: return SmemPADiffusionApply3D<6,7>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x78: return SmemPADiffusionApply3D<7,8>(NE,B,G,Bt,Gt,D,X,Y);
-         case 0x89: return SmemPADiffusionApply3D<8,9>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x23: return SmemPADiffusionApply3D<2,3>(NE,B,G,D,X,Y);
+         case 0x34: return SmemPADiffusionApply3D<3,4>(NE,B,G,D,X,Y);
+         case 0x45: return SmemPADiffusionApply3D<4,5>(NE,B,G,D,X,Y);
+         case 0x46: return SmemPADiffusionApply3D<4,6>(NE,B,G,D,X,Y);
+         case 0x56: return SmemPADiffusionApply3D<5,6>(NE,B,G,D,X,Y);
+         case 0x58: return SmemPADiffusionApply3D<5,8>(NE,B,G,D,X,Y);
+         case 0x67: return SmemPADiffusionApply3D<6,7>(NE,B,G,D,X,Y);
+         case 0x78: return SmemPADiffusionApply3D<7,8>(NE,B,G,D,X,Y);
+         case 0x89: return SmemPADiffusionApply3D<8,9>(NE,B,G,D,X,Y);
          default:   return PADiffusionApply3D(NE,B,G,Bt,Gt,D,X,Y,D1D,Q1D);
       }
    }

fem/bilininteg_vecdiffusion.cpp

@@ -101,7 +101,6 @@ static void PAVectorDiffusionSetup3D(const int Q1D,
 }
 
 static void PAVectorDiffusionSetup(const int dim,
-                                   const int D1D,
                                    const int Q1D,
                                    const int NE,
                                    const Array<double> &W,
@@ -134,6 +133,7 @@ void VectorDiffusionIntegrator::AssemblePA(const FiniteElementSpace &fes)
    const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
    const int nq = ir->GetNPoints();
    dim = mesh->Dimension();
+   sdim = mesh->SpaceDimension();
    ne = fes.GetNE();
    geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
    maps = &el.GetDofToQuad(*ir, DofToQuad::TENSOR);
@@ -147,46 +147,83 @@ void VectorDiffusionIntegrator::AssemblePA(const FiniteElementSpace &fes)
       MFEM_VERIFY(cQ != NULL, "only ConstantCoefficient is supported!");
       coeff = cQ->constant;
    }
-   PAVectorDiffusionSetup(dim, dofs1D, quad1D, ne, ir->GetWeights(), geom->J,
-                          coeff, pa_data);
+   const Array<double> &w = ir->GetWeights();
+   const Vector &j = geom->J;
+   Vector &d = pa_data;
+   if (dim == 1) { MFEM_ABORT("dim==1 not supported in PAVectorDiffusionSetup"); }
+   if (dim == 2 && sdim == 3)
+   {
+      constexpr int DIM = 2;
+      constexpr int SDIM = 3;
+      const int NQ = quad1D*quad1D;
+      auto W = w.Read();
+      auto J = Reshape(j.Read(), NQ, SDIM, DIM, ne);
+      auto D = Reshape(d.Write(), NQ, SDIM, ne);
+      MFEM_FORALL(e, ne,
+      {
+         for (int q = 0; q < NQ; ++q)
+         {
+            const double wq = W[q];
+            const double J11 = J(q,0,0,e);
+            const double J21 = J(q,1,0,e);
+            const double J31 = J(q,2,0,e);
+            const double J12 = J(q,0,1,e);
+            const double J22 = J(q,1,1,e);
+            const double J32 = J(q,2,1,e);
+            const double E = J11*J11 + J21*J21 + J31*J31;
+            const double G = J12*J12 + J22*J22 + J32*J32;
+            const double F = J11*J12 + J21*J22 + J31*J32;
+            const double iw = 1.0 / sqrt(E*G - F*F);
+            const double alpha = wq * coeff * iw;
+            D(q,0,e) =  alpha * G; // 1,1
+            D(q,1,e) = -alpha * F; // 1,2
+            D(q,2,e) =  alpha * E; // 2,2
+         }
+      });
+   }
+   else
+   {
+      PAVectorDiffusionSetup(dim, quad1D, ne, w, j, coeff, d);
+   }
 }
 
 // PA Diffusion Apply 2D kernel
-template<int T_D1D = 0, int T_Q1D = 0> static
+template<int T_D1D = 0, int T_Q1D = 0, int T_VDIM = 0> static
 void PAVectorDiffusionApply2D(const int NE,
                               const Array<double> &b,
                               const Array<double> &g,
                               const Array<double> &bt,
                               const Array<double> &gt,
-                              const Vector &_op,
-                              const Vector &_x,
-                              Vector &_y,
+                              const Vector &d_,
+                              const Vector &x_,
+                              Vector &y_,
                               const int d1d = 0,
-                              const int q1d = 0)
+                              const int q1d = 0,
+                              const int vdim = 0)
 {
    const int D1D = T_D1D ? T_D1D : d1d;
    const int Q1D = T_Q1D ? T_Q1D : q1d;
-   constexpr int VDIM = 2;
+   const int VDIM = T_VDIM ? T_VDIM : vdim;
    MFEM_VERIFY(D1D <= MAX_D1D, "");
    MFEM_VERIFY(Q1D <= MAX_Q1D, "");
    auto B = Reshape(b.Read(), Q1D, D1D);
    auto G = Reshape(g.Read(), Q1D, D1D);
    auto Bt = Reshape(bt.Read(), D1D, Q1D);
    auto Gt = Reshape(gt.Read(), D1D, Q1D);
-   auto op = Reshape(_op.Read(), Q1D*Q1D, 3, NE);
-   auto x = Reshape(_x.Read(), D1D, D1D, VDIM, NE);
-   auto y = Reshape(_y.ReadWrite(), D1D, D1D, VDIM, NE);
+   auto D = Reshape(d_.Read(), Q1D*Q1D, 3, NE);
+   auto x = Reshape(x_.Read(), D1D, D1D, VDIM, NE);
+   auto y = Reshape(y_.ReadWrite(), D1D, D1D, VDIM, NE);
    MFEM_FORALL(e, NE,
    {
       const int D1D = T_D1D ? T_D1D : d1d;
       const int Q1D = T_Q1D ? T_Q1D : q1d;
-      // the following variables are evaluated at compile time
+      const int VDIM = T_VDIM ? T_VDIM : vdim;
       constexpr int max_D1D = T_D1D ? T_D1D : MAX_D1D;
       constexpr int max_Q1D = T_Q1D ? T_Q1D : MAX_Q1D;
 
-      for (int c = 0; c < VDIM; ++ c)
+      double grad[max_Q1D][max_Q1D][2];
+      for (int c = 0; c < VDIM; c++)
       {
-         double grad[max_Q1D][max_Q1D][2];
          for (int qy = 0; qy < Q1D; ++qy)
          {
             for (int qx = 0; qx < Q1D; ++qx)
@@ -229,14 +266,11 @@ void PAVectorDiffusionApply2D(const int NE,
             for (int qx = 0; qx < Q1D; ++qx)
             {
                const int q = qx + qy * Q1D;
-
-               const double O11 = op(q,0,e);
-               const double O12 = op(q,1,e);
-               const double O22 = op(q,2,e);
-
+               const double O11 = D(q,0,e);
+               const double O12 = D(q,1,e);
+               const double O22 = D(q,2,e);
                const double gradX = grad[qy][qx][0];
                const double gradY = grad[qy][qx][1];
-
                grad[qy][qx][0] = (O11 * gradX) + (O12 * gradY);
                grad[qy][qx][1] = (O12 * gradX) + (O22 * gradY);
             }
@@ -246,8 +280,8 @@ void PAVectorDiffusionApply2D(const int NE,
             double gradX[max_D1D][2];
             for (int dx = 0; dx < D1D; ++dx)
             {
-               gradX[dx][0] = 0;
-               gradX[dx][1] = 0;
+               gradX[dx][0] = 0.0;
+               gradX[dx][1] = 0.0;
             }
             for (int qx = 0; qx < Q1D; ++qx)
             {
@@ -469,35 +503,36 @@ void PAVectorDiffusionApply3D(const int NE,
    });
 }
 
-static void PAVectorDiffusionApply(const int dim,
-                                   const int D1D,
-                                   const int Q1D,
-                                   const int NE,
-                                   const Array<double> &B,
-                                   const Array<double> &G,
-                                   const Array<double> &Bt,
-                                   const Array<double> &Gt,
-                                   const Vector &op,
-                                   const Vector &x,
-                                   Vector &y)
-{
-   if (dim == 2)
-   {
-      return PAVectorDiffusionApply2D(NE,B,G,Bt,Gt,op,x,y,D1D,Q1D);
-   }
-   if (dim == 3)
-   {
-      return PAVectorDiffusionApply3D(NE,B,G,Bt,Gt,op,x,y,D1D,Q1D);
-   }
-   MFEM_ABORT("Unknown kernel.");
-}
-
 // PA Diffusion Apply kernel
 void VectorDiffusionIntegrator::AddMultPA(const Vector &x, Vector &y) const
 {
-   PAVectorDiffusionApply(dim, dofs1D, quad1D, ne,
-                          maps->B, maps->G, maps->Bt, maps->Gt,
-                          pa_data, x, y);
+   const int D1D = dofs1D;
+   const int Q1D = quad1D;
+   const Array<double> &B = maps->B;
+   const Array<double> &G = maps->G;
+   const Array<double> &Bt = maps->Bt;
+   const Array<double> &Gt = maps->Gt;
+   const Vector &D = pa_data;
+
+   if (dim == 2 && sdim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return PAVectorDiffusionApply2D<2,2,3>(ne,B,G,Bt,Gt,D,x,y);
+         case 0x33: return PAVectorDiffusionApply2D<3,3,3>(ne,B,G,Bt,Gt,D,x,y);
+         case 0x44: return PAVectorDiffusionApply2D<4,4,3>(ne,B,G,Bt,Gt,D,x,y);
+         case 0x55: return PAVectorDiffusionApply2D<5,5,3>(ne,B,G,Bt,Gt,D,x,y);
+         default:
+            return PAVectorDiffusionApply2D(ne,B,G,Bt,Gt,D,x,y,D1D,Q1D,sdim);
+      }
+   }
+   if (dim == 2 && sdim == 2)
+   { return PAVectorDiffusionApply2D(ne,B,G,Bt,Gt,D,x,y,D1D,Q1D,sdim); }
+
+   if (dim == 3 && sdim == 3)
+   { return PAVectorDiffusionApply3D(ne,B,G,Bt,Gt,D,x,y,D1D,Q1D); }
+
+   MFEM_ABORT("Unknown kernel.");
 }
 
 template<int T_D1D = 0, int T_Q1D = 0>

fem/coefficient.cpp

@@ -174,7 +174,24 @@ void VectorGridFunctionCoefficient::SetGridFunction(GridFunction *gf)
 void VectorGridFunctionCoefficient::Eval(Vector &V, ElementTransformation &T,
                                          const IntegrationPoint &ip)
 {
-   GridFunc->GetVectorValue(T.ElementNo, ip, V);
+   Mesh *mesh = GridFunc->FESpace()->GetMesh();
+   if (mesh->Dimension() == T.GetDimension())
+   {
+      GridFunc->GetVectorValue(T.ElementNo, ip, V);
+   }
+   else // Assuming T is a boundary element transformation
+   {
+      int el_id, el_info;
+      mesh->GetBdrElementAdjacentElement(T.ElementNo, el_id, el_info);
+      IntegrationPointTransformation loc_T;
+      mesh->GetLocalFaceTransformation(mesh->GetBdrElementType(T.ElementNo),
+                                       mesh->GetElementType(el_id),
+                                       loc_T.Transf,
+                                       el_info);
+      IntegrationPoint eip;
+      loc_T.Transform(ip, eip);
+      GridFunc->GetVectorValue(el_id, eip, V);
+   }
 }
 
 void VectorGridFunctionCoefficient::Eval(

fem/coefficient.hpp

@@ -138,7 +138,7 @@ public:
    /// (DEPRECATED) Define a time-independent coefficient from a C-function
    /** @deprecated Use the method where the C-function, @a f, uses a const
        Vector argument instead of Vector. */
-   FunctionCoefficient(double (*f)(Vector &))
+   MFEM_DEPRECATED FunctionCoefficient(double (*f)(Vector &))
    {
       Function = reinterpret_cast<double(*)(const Vector&)>(f);
       TDFunction = NULL;
@@ -147,7 +147,7 @@ public:
    /// (DEPRECATED) Define a time-dependent coefficient from a C-function
    /** @deprecated Use the method where the C-function, @a tdf, uses a const
        Vector argument instead of Vector. */
-   FunctionCoefficient(double (*tdf)(Vector &, double))
+   MFEM_DEPRECATED FunctionCoefficient(double (*tdf)(Vector &, double))
    {
       Function = NULL;
       TDFunction = reinterpret_cast<double(*)(const Vector&,double)>(tdf);

fem/eltrans.cpp

@@ -391,14 +391,10 @@ void IsoparametricTransformation::SetIdentityTransformation(
       nodes.IntPoint(j).Get(&PointMat(0,j), dim);
    }
    geom = GeomType;
-   space_dim = dim;
 }
 
 const DenseMatrix &IsoparametricTransformation::EvalJacobian()
 {
-   MFEM_ASSERT(space_dim == PointMat.Height(),
-               "the IsoparametricTransformation has not been finalized;"
-               " call FinilizeTransformation() after setup");
    MFEM_ASSERT((EvalState & JACOBIAN_MASK) == 0, "");
 
    dshape.SetSize(FElem->GetDof(), FElem->GetDim());
@@ -415,9 +411,6 @@ const DenseMatrix &IsoparametricTransformation::EvalJacobian()
 
 const DenseMatrix &IsoparametricTransformation::EvalHessian()
 {
-   MFEM_ASSERT(space_dim == PointMat.Height(),
-               "the IsoparametricTransformation has not been finalized;"
-               " call FinilizeTransformation() after setup");
    MFEM_ASSERT((EvalState & HESSIAN_MASK) == 0, "");
 
    int Dim = FElem->GetDim();
@@ -433,7 +426,7 @@ const DenseMatrix &IsoparametricTransformation::EvalHessian()
    return d2Fdx2;
 }
 
-int IsoparametricTransformation::OrderJ()
+int IsoparametricTransformation::OrderJ() const
 {
    switch (FElem->Space())
    {
@@ -442,12 +435,12 @@ int IsoparametricTransformation::OrderJ()
       case FunctionSpace::Qk:
          return (FElem->GetOrder());
       default:
-         mfem_error("IsoparametricTransformation::OrderJ()");
+         MFEM_ABORT("unsupported finite element");
    }
    return 0;
 }
 
-int IsoparametricTransformation::OrderW()
+int IsoparametricTransformation::OrderW() const
 {
    switch (FElem->Space())
    {
@@ -456,12 +449,12 @@ int IsoparametricTransformation::OrderW()
       case FunctionSpace::Qk:
          return (FElem->GetOrder() * FElem->GetDim() - 1);
       default:
-         mfem_error("IsoparametricTransformation::OrderW()");
+         MFEM_ABORT("unsupported finite element");
    }
    return 0;
 }
 
-int IsoparametricTransformation::OrderGrad(const FiniteElement *fe)
+int IsoparametricTransformation::OrderGrad(const FiniteElement *fe) const
 {
    if (FElem->Space() == fe->Space())
    {
@@ -474,9 +467,11 @@ int IsoparametricTransformation::OrderGrad(const FiniteElement *fe)
             return ((k-1)*(d-1)+(l-1));
          case FunctionSpace::Qk:
             return (k*(d-1)+(l-1));
+         default:
+            MFEM_ABORT("unsupported finite element");
       }
    }
-   mfem_error("IsoparametricTransformation::OrderGrad(...)");
+   MFEM_ABORT("incompatible finite elements");
    return 0;
 }
 

fem/eltrans.hpp

@@ -37,7 +37,6 @@ protected:
       HESSIAN_MASK  = 16
    };
    Geometry::Type geom;
-   int space_dim;
 
    // Evaluate the Jacobian of the transformation at the IntPoint and store it
    // in dFdx.
@@ -82,11 +81,11 @@ public:
    const DenseMatrix &InverseJacobian()
    { return (EvalState & INVERSE_MASK) ? invJ : EvalInverseJ(); }
 
-   virtual int Order() = 0;
-   virtual int OrderJ() = 0;
-   virtual int OrderW() = 0;
+   virtual int Order() const = 0;
+   virtual int OrderJ() const = 0;
+   virtual int OrderW() const = 0;
    /// Order of adj(J)^t.grad(fi)
-   virtual int OrderGrad(const FiniteElement *fe) = 0;
+   virtual int OrderGrad(const FiniteElement *fe) const = 0;
 
    /// Return the Geometry::Type of the reference element.
    Geometry::Type GetGeometryType() const { return geom; }
@@ -97,7 +96,7 @@ public:
    /// Get the dimension of the target (physical) space.
    /** We support 2D meshes embedded in 3D; in this case the function will
        return "3". */
-   int GetSpaceDim() const { return space_dim; }
+   virtual int GetSpaceDim() const = 0;
 
    /** @brief Transform a point @a pt from physical space to a point @a ip in
        reference space. */
@@ -307,19 +306,23 @@ public:
    void SetFE(const FiniteElement *FE) { FElem = FE; geom = FE->GetGeomType(); }
    const FiniteElement* GetFE() const { return FElem; }
 
-   /** @brief Read and write access to the underlying point matrix describing
-       the transformation. */
+   /// @brief Set the underlying point matrix describing the transformation.
    /** The dimensions of the matrix are space-dim x dof. The transformation is
        defined as
 
-           x=F(xh)=P.phi(xh),
+           x = F(xh) = P . phi(xh),
 
        where xh (x hat) is the reference point, x is the corresponding physical
        point, P is the point matrix, and phi(xh) is the column-vector of all
        basis functions evaluated at xh. The columns of P represent the control
        points in physical space defining the transformation. */
+   void SetPointMat(const DenseMatrix &pm) { PointMat = pm; }
+
+   /// Return the stored point matrix.
+   const DenseMatrix &GetPointMat() const { return PointMat; }
+
+   /// Write access to the stored point matrix. Use with caution.
    DenseMatrix &GetPointMat() { return PointMat; }
-   void FinalizeTransformation() { space_dim = PointMat.Height(); }
 
    void SetIdentityTransformation(Geometry::Type GeomType);
 
@@ -327,10 +330,12 @@ public:
    virtual void Transform(const IntegrationRule &, DenseMatrix &);
    virtual void Transform(const DenseMatrix &matrix, DenseMatrix &result);
 
-   virtual int Order() { return FElem->GetOrder(); }
-   virtual int OrderJ();
-   virtual int OrderW();
-   virtual int OrderGrad(const FiniteElement *fe);
+   virtual int Order() const { return FElem->GetOrder(); }
+   virtual int OrderJ() const;
+   virtual int OrderW() const;
+   virtual int OrderGrad(const FiniteElement *fe) const;
+
+   virtual int GetSpaceDim() const { return PointMat.Height(); }
 
    virtual int TransformBack(const Vector & v, IntegrationPoint & ip)
    {
@@ -339,6 +344,8 @@ public:
    }
 
    virtual ~IsoparametricTransformation() { }
+
+   MFEM_DEPRECATED void FinalizeTransformation() {}
 };
 
 class IntegrationPointTransformation

fem/fem.hpp

@@ -37,6 +37,9 @@
 #include "restriction.hpp"
 #include "quadinterpolator.hpp"
 #include "quadinterpolator_face.hpp"
+#include "transfer.hpp"
+#include "fespacehierarchy.hpp"
+#include "multigrid.hpp"
 
 #ifdef MFEM_USE_MPI
 #include "pfespace.hpp"
@@ -54,4 +57,8 @@
 #include "conduitdatacollection.hpp"
 #endif
 
+#ifdef MFEM_USE_ADIOS2
+#include "adios2datacollection.hpp"
+#endif
+
 #endif

fem/fespace.cpp

@@ -495,9 +495,10 @@ FiniteElementSpace::H2L_GlobalRestrictionMatrix (FiniteElementSpace *lfes)
 {
    SparseMatrix *R;
    DenseMatrix loc_restr;
-   Array<int> l_dofs, h_dofs;
+   Array<int> l_dofs, h_dofs, l_vdofs, h_vdofs;
 
-   R = new SparseMatrix (lfes -> GetNDofs(), ndofs);
+   int vdim = lfes->GetVDim();
+   R = new SparseMatrix (vdim * lfes -> GetNDofs(), vdim * ndofs);
 
    Geometry::Type cached_geom = Geometry::INVALID;
    const FiniteElement *h_fe = NULL;
@@ -520,7 +521,16 @@ FiniteElementSpace::H2L_GlobalRestrictionMatrix (FiniteElementSpace *lfes)
          cached_geom = geom;
       }
 
-      R -> SetSubMatrix (l_dofs, h_dofs, loc_restr, 1);
+      for (int vd = 0; vd < vdim; vd++)
+      {
+         l_dofs.Copy(l_vdofs);
+         lfes->DofsToVDofs(vd, l_vdofs);
+
+         h_dofs.Copy(h_vdofs);
+         this->DofsToVDofs(vd, h_vdofs);
+
+         R -> SetSubMatrix (l_vdofs, h_vdofs, loc_restr, 1);
+      }
    }
 
    R -> Finalize();
@@ -670,7 +680,6 @@ void FiniteElementSpace::BuildConformingInterpolation() const
             if (!slave_dofs.Size()) { continue; }
 
             slave.OrientedPointMatrix(T.GetPointMat());
-            T.FinalizeTransformation();
             fe->GetLocalInterpolation(T, I);
 
             // make each slave DOF dependent on all master DOFs
@@ -1026,8 +1035,7 @@ void FiniteElementSpace::GetLocalRefinementMatrices(
    localP.SetSize(ldof, ldof, nmat);
    for (int i = 0; i < nmat; i++)
    {
-      isotr.GetPointMat() = pmats(i);
-      isotr.FinalizeTransformation();
+      isotr.SetPointMat(pmats(i));
       fe->GetLocalInterpolation(isotr, localP(i));
    }
 }
@@ -1096,46 +1104,90 @@ void FiniteElementSpace::RefinementOperator
    Mesh* mesh = fespace->GetMesh();
    const CoarseFineTransformations &rtrans = mesh->GetRefinementTransforms();
 
-   Array<int> dofs, old_dofs, old_vdofs;
-
-   Array<char> processed(fespace->GetVSize());
-   processed = 0;
+   Array<int> dofs, vdofs, old_dofs, old_vdofs;
 
    int vdim = fespace->GetVDim();
    int old_ndofs = width / vdim;
 
+   Vector subY, subX;
+
    for (int k = 0; k < mesh->GetNE(); k++)
    {
       const Embedding &emb = rtrans.embeddings[k];
       const Geometry::Type geom = mesh->GetElementBaseGeometry(k);
       const DenseMatrix &lP = localP[geom](emb.matrix);
 
+      subY.SetSize(lP.Height());
+
       fespace->GetElementDofs(k, dofs);
       old_elem_dof->GetRow(emb.parent, old_dofs);
 
       for (int vd = 0; vd < vdim; vd++)
       {
+         dofs.Copy(vdofs);
+         fespace->DofsToVDofs(vd, vdofs);
          old_dofs.Copy(old_vdofs);
          fespace->DofsToVDofs(vd, old_vdofs, old_ndofs);
+         x.GetSubVector(old_vdofs, subX);
+         lP.Mult(subX, subY);
+         y.SetSubVector(vdofs, subY);
+      }
+   }
+}
 
-         for (int i = 0; i < dofs.Size(); i++)
+void FiniteElementSpace::RefinementOperator
+::MultTranspose(const Vector &x, Vector &y) const
+{
+   y = 0.0;
+
+   Mesh* mesh = fespace->GetMesh();
+   const CoarseFineTransformations &rtrans = mesh->GetRefinementTransforms();
+
+   Array<char> processed(fespace->GetVSize());
+   processed = 0;
+
+   Array<int> f_dofs, c_dofs, f_vdofs, c_vdofs;
+
+   int vdim = fespace->GetVDim();
+   int old_ndofs = width / vdim;
+
+   Vector subY, subX;
+
+   for (int k = 0; k < mesh->GetNE(); k++)
+   {
+      const Embedding &emb = rtrans.embeddings[k];
+      const Geometry::Type geom = mesh->GetElementBaseGeometry(k);
+      const DenseMatrix &lP = localP[geom](emb.matrix);
+
+      fespace->GetElementDofs(k, f_dofs);
+      old_elem_dof->GetRow(emb.parent, c_dofs);
+
+      subY.SetSize(lP.Width());
+
+      for (int vd = 0; vd < vdim; vd++)
+      {
+         f_dofs.Copy(f_vdofs);
+         fespace->DofsToVDofs(vd, f_vdofs);
+         c_dofs.Copy(c_vdofs);
+         fespace->DofsToVDofs(vd, c_vdofs, old_ndofs);
+
+         x.GetSubVector(f_vdofs, subX);
+
+         for (int p = 0; p < f_dofs.Size(); ++p)
          {
-            double rsign, osign;
-            int r = fespace->DofToVDof(dofs[i], vd);
-            r = DecodeDof(r, rsign);
-
-            if (!processed[r])
+            if (processed[DecodeDof(f_dofs[p])])
             {
-               double value = 0.0;
-               for (int j = 0; j < old_vdofs.Size(); j++)
-               {
-                  int o = DecodeDof(old_vdofs[j], osign);
-                  value += x[o] * lP(i, j) * osign;
-               }
-               y[r] = value * rsign;
-               processed[r] = 1;
+               subX[p] = 0.0;
             }
          }
+
+         lP.MultTranspose(subX, subY);
+         y.AddElementVector(c_vdofs, subY);
+      }
+
+      for (int p = 0; p < f_dofs.Size(); ++p)
+      {
+         processed[DecodeDof(f_dofs[p])] = 1;
       }
    }
 }
@@ -1171,8 +1223,7 @@ FiniteElementSpace::DerefinementOperator::DerefinementOperator(
       emb_tr.SetIdentityTransformation(geom);
       for (int i = 0; i < pmats.SizeK(); i++)
       {
-         emb_tr.GetPointMat() = pmats(i);
-         emb_tr.FinalizeTransformation();
+         emb_tr.SetPointMat(pmats(i));
          // Get the local interpolation matrix for this refinement type
          fine_fe->GetTransferMatrix(*coarse_fe, emb_tr, lP(i));
          // Get the local mass matrix for this refinement type
@@ -1292,9 +1343,7 @@ void FiniteElementSpace::GetLocalDerefinementMatrices(Geometry::Type geom,
    localR.SetSize(ldof, ldof, nmat);
    for (int i = 0; i < nmat; i++)
    {
-      isotr.GetPointMat() = pmats(i);
-      isotr.FinalizeTransformation();
-
+      isotr.SetPointMat(pmats(i));
       fe->GetLocalRestriction(isotr, localR(i));
    }
 }
@@ -1392,8 +1441,7 @@ void FiniteElementSpace::GetLocalRefinementMatrices(
    localP.SetSize(fine_fe->GetDof(), coarse_fe->GetDof(), nmat);
    for (int i = 0; i < nmat; i++)
    {
-      isotr.GetPointMat() = pmats(i);
-      isotr.FinalizeTransformation();
+      isotr.SetPointMat(pmats(i));
       fine_fe->GetTransferMatrix(*coarse_fe, isotr, localP(i));
    }
 }
@@ -2650,8 +2698,7 @@ L2ProjectionGridTransfer::L2Projection::L2Projection(
 
          // Create the transformation that embeds the fine low-order element
          // within the coarse high-order element in reference space
-         emb_tr.GetPointMat() = pmats(cf_tr.embeddings[ilor].matrix);
-         emb_tr.FinalizeTransformation();
+         emb_tr.SetPointMat(pmats(cf_tr.embeddings[ilor].matrix));
 
          int order = fe_lor->GetOrder() + fe_ho->GetOrder() + el_tr->OrderW();
          const IntegrationRule *ir = &IntRules.Get(geom, order);

fem/fespace.hpp

@@ -87,6 +87,7 @@ class FaceQuadratureInterpolator;
 class FiniteElementSpace
 {
    friend class InterpolationGridTransfer;
+   friend class PRefinementTransferOperator;
 
 protected:
    /// The mesh that FE space lives on (not owned).
@@ -158,7 +159,12 @@ protected:
 
    void BuildElementToDofTable() const;
 
-   /// Helper to remove encoded sign from a DOF
+   /// Helpers to remove encoded sign from a DOF
+   static inline int DecodeDof(int dof)
+   {
+      return (dof >= 0) ? dof : (-1 - dof);
+   }
+
    static inline int DecodeDof(int dof, double& sign)
    { return (dof >= 0) ? (sign = 1, dof) : (sign = -1, (-1 - dof)); }
 
@@ -196,6 +202,7 @@ protected:
       RefinementOperator(const FiniteElementSpace *fespace,
                          const FiniteElementSpace *coarse_fes);
       virtual void Mult(const Vector &x, Vector &y) const;
+      virtual void MultTranspose(const Vector &x, Vector &y) const;
       virtual ~RefinementOperator();
    };
 

fem/fespacehierarchy.cpp

@@ -0,0 +1,189 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "fespacehierarchy.hpp"
+#include "transfer.hpp"
+
+namespace mfem
+{
+
+FiniteElementSpaceHierarchy::FiniteElementSpaceHierarchy(Mesh* mesh,
+                                                         FiniteElementSpace* fespace,
+                                                         bool ownM, bool ownFES)
+{
+   meshes.Append(mesh);
+   fespaces.Append(fespace);
+   ownedMeshes.Append(ownM);
+   ownedFES.Append(ownFES);
+}
+
+FiniteElementSpaceHierarchy::~FiniteElementSpaceHierarchy()
+{
+   for (int i = 0; i < meshes.Size(); ++i)
+   {
+      if (ownedFES[i])
+      {
+         delete fespaces[i];
+      }
+      if (ownedMeshes[i])
+      {
+         delete meshes[i];
+      }
+   }
+
+   for (int i = 0; i < prolongations.Size(); ++i)
+   {
+      if (ownedProlongations[i])
+      {
+         delete prolongations[i];
+      }
+   }
+
+   fespaces.DeleteAll();
+   meshes.DeleteAll();
+   prolongations.DeleteAll();
+}
+
+int FiniteElementSpaceHierarchy::GetNumLevels() const { return meshes.Size(); }
+
+int FiniteElementSpaceHierarchy::GetFinestLevelIndex() const { return GetNumLevels() - 1; }
+
+void FiniteElementSpaceHierarchy::AddLevel(Mesh* mesh,
+                                           FiniteElementSpace* fespace,
+                                           Operator* prolongation,
+                                           bool ownM, bool ownFES,
+                                           bool ownP)
+{
+   meshes.Append(mesh);
+   fespaces.Append(fespace);
+   prolongations.Append(prolongation);
+   ownedMeshes.Append(ownM);
+   ownedFES.Append(ownFES);
+   ownedProlongations.Append(ownP);
+}
+
+void FiniteElementSpaceHierarchy::AddUniformlyRefinedLevel(int dim,
+                                                           int ordering)
+{
+   MFEM_VERIFY(GetNumLevels() > 0, "There is no level which can be refined");
+   Mesh* mesh = new Mesh(*GetFinestFESpace().GetMesh());
+   mesh->UniformRefinement();
+   FiniteElementSpace& coarseFEspace = GetFinestFESpace();
+   FiniteElementSpace* fineFEspace =
+      new FiniteElementSpace(mesh, coarseFEspace.FEColl(), dim, ordering);
+   Operator* P = new TransferOperator(coarseFEspace, *fineFEspace);
+   AddLevel(mesh, fineFEspace, P, true, true, true);
+}
+
+void FiniteElementSpaceHierarchy::AddOrderRefinedLevel(FiniteElementCollection*
+                                                       fec, int dim,
+                                                       int ordering)
+{
+   MFEM_VERIFY(GetNumLevels() > 0, "There is no level which can be refined");
+   Mesh* mesh = GetFinestFESpace().GetMesh();
+   FiniteElementSpace* newFEspace =
+      new FiniteElementSpace(mesh, fec, dim, ordering);
+   Operator* P = new TransferOperator(GetFinestFESpace(), *newFEspace);
+   AddLevel(mesh, newFEspace, P, false, true, true);
+}
+
+const FiniteElementSpace& FiniteElementSpaceHierarchy::GetFESpaceAtLevel(
+   int level) const
+{
+   MFEM_ASSERT(level < fespaces.Size(),
+               "FE space at given level does not exist.");
+   return *fespaces[level];
+}
+
+FiniteElementSpace& FiniteElementSpaceHierarchy::GetFESpaceAtLevel(int level)
+{
+   MFEM_ASSERT(level < fespaces.Size(),
+               "FE space at given level does not exist.");
+   return *fespaces[level];
+}
+
+const FiniteElementSpace& FiniteElementSpaceHierarchy::GetFinestFESpace() const
+{
+   return GetFESpaceAtLevel(GetFinestLevelIndex());
+}
+
+FiniteElementSpace& FiniteElementSpaceHierarchy::GetFinestFESpace()
+{
+   return GetFESpaceAtLevel(GetFinestLevelIndex());
+}
+
+Operator* FiniteElementSpaceHierarchy::GetProlongationAtLevel(int level) const
+{
+   MFEM_ASSERT(level < prolongations.Size(),
+               "Prolongation at given level does not exist.");
+   return prolongations[level];
+}
+
+#ifdef MFEM_USE_MPI
+
+ParFiniteElementSpaceHierarchy::ParFiniteElementSpaceHierarchy(ParMesh* mesh,
+                                                               ParFiniteElementSpace* fespace,
+                                                               bool ownM,
+                                                               bool ownFES)
+   : FiniteElementSpaceHierarchy(mesh, fespace, ownM, ownFES)
+{
+}
+
+void ParFiniteElementSpaceHierarchy::AddUniformlyRefinedLevel(int dim,
+                                                              int ordering)
+{
+   ParMesh* mesh = new ParMesh(*GetFinestFESpace().GetParMesh());
+   mesh->UniformRefinement();
+   ParFiniteElementSpace& coarseFEspace = GetFinestFESpace();
+   ParFiniteElementSpace* fineFEspace =
+      new ParFiniteElementSpace(mesh, coarseFEspace.FEColl(), dim, ordering);
+   Operator* P = new TrueTransferOperator(coarseFEspace, *fineFEspace);
+   AddLevel(mesh, fineFEspace, P, true, true, true);
+}
+
+void ParFiniteElementSpaceHierarchy::AddOrderRefinedLevel(
+   FiniteElementCollection* fec,
+   int dim, int ordering)
+{
+   ParMesh* mesh = GetFinestFESpace().GetParMesh();
+   ParFiniteElementSpace* newFEspace =
+      new ParFiniteElementSpace(mesh, fec, dim, ordering);
+   Operator* P = new TrueTransferOperator(GetFinestFESpace(), *newFEspace);
+   AddLevel(mesh, newFEspace, P, false, true, true);
+}
+
+const ParFiniteElementSpace&
+ParFiniteElementSpaceHierarchy::GetFESpaceAtLevel(int level) const
+{
+   return static_cast<const ParFiniteElementSpace&>(
+             FiniteElementSpaceHierarchy::GetFESpaceAtLevel(level));
+}
+
+ParFiniteElementSpace& ParFiniteElementSpaceHierarchy::GetFESpaceAtLevel(
+   int level)
+{
+   return static_cast<ParFiniteElementSpace&>(
+             FiniteElementSpaceHierarchy::GetFESpaceAtLevel(level));
+}
+
+const ParFiniteElementSpace& ParFiniteElementSpaceHierarchy::GetFinestFESpace()
+const
+{
+   return GetFESpaceAtLevel(GetFinestLevelIndex());
+}
+
+ParFiniteElementSpace& ParFiniteElementSpaceHierarchy::GetFinestFESpace()
+{
+   return GetFESpaceAtLevel(GetFinestLevelIndex());
+}
+#endif
+
+} // namespace mfem

fem/fespacehierarchy.hpp

@@ -0,0 +1,123 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#ifndef MFEM_FESPACEHIERARCHY
+#define MFEM_FESPACEHIERARCHY
+
+#include "fespace.hpp"
+
+#ifdef MFEM_USE_MPI
+#include "pfespace.hpp"
+#endif
+
+namespace mfem
+{
+
+/// Class bundling a hierarchy finite element spaces together with the
+/// corresponding prolongation operators
+class FiniteElementSpaceHierarchy
+{
+protected:
+   Array<Mesh*> meshes;
+   Array<FiniteElementSpace*> fespaces;
+   Array<Operator*> prolongations;
+   Array<bool> ownedMeshes;
+   Array<bool> ownedFES;
+   Array<bool> ownedProlongations;
+
+public:
+
+   /// @brief Constructs a space hierarchy with the given mesh and space on the
+   /// coarsest level.
+   /** The ownership of the mesh and space may be transferred to the
+       FiniteElementSpaceHierarchy by setting the according boolean variables. */
+   FiniteElementSpaceHierarchy(Mesh* mesh, FiniteElementSpace* fespace, bool ownM,
+                               bool ownFES);
+
+   /// Destructor deleting all meshes and spaces that are owned
+   virtual ~FiniteElementSpaceHierarchy();
+
+   /// Returns the number of levels in the hierarchy
+   int GetNumLevels() const;
+
+   /// Returns the index of the finest level
+   int GetFinestLevelIndex() const;
+
+   /// Adds one level to the hierarchy
+   void AddLevel(Mesh* mesh, FiniteElementSpace* fespace, Operator* prolongation,
+                 bool ownM, bool ownFES, bool ownP);
+
+   /// @brief Adds one level to the hierarchy by uniformly refining the mesh on the
+   /// previous level
+   virtual void AddUniformlyRefinedLevel(int dim = 1,
+                                         int ordering = Ordering::byVDIM);
+
+   /// @brief Adds one level to the hierarchy by using a different finite element
+   /// order defined through FiniteElementCollection
+   virtual void AddOrderRefinedLevel(FiniteElementCollection* fec, int dim = 1,
+                                     int ordering = Ordering::byVDIM);
+
+   /// Returns the finite element space at the given level
+   virtual const FiniteElementSpace& GetFESpaceAtLevel(int level) const;
+
+   /// Returns the finite element space at the given level
+   virtual FiniteElementSpace& GetFESpaceAtLevel(int level);
+
+   /// Returns the finite element space at the finest level
+   virtual const FiniteElementSpace& GetFinestFESpace() const;
+
+   /// Returns the finite element space at the finest level
+   virtual FiniteElementSpace& GetFinestFESpace();
+
+   /// @brief Returns the prolongation operator from the finite element space at
+   /// level to the finite element space at level + 1
+   Operator* GetProlongationAtLevel(int level) const;
+};
+
+#ifdef MFEM_USE_MPI
+class ParFiniteElementSpaceHierarchy : public FiniteElementSpaceHierarchy
+{
+public:
+   /// @brief Constructs a parallel space hierarchy with the given mesh and spaces
+   /// on level zero.
+   /** The ownership of the mesh and space may be transferred to the
+       ParFiniteElementSpaceHierarchy by setting the according boolean variables. */
+   ParFiniteElementSpaceHierarchy(ParMesh* mesh, ParFiniteElementSpace* fespace,
+                                  bool ownM,
+                                  bool ownFES);
+
+   /// @brief Adds one level to the hierarchy by uniformly refining the mesh on the
+   /// previous level
+   void AddUniformlyRefinedLevel(int dim = 1,
+                                 int ordering = Ordering::byVDIM) override;
+
+   /// @brief Adds one level to the hierarchy by using a different finite element
+   /// order defined through FiniteElementCollection
+   void AddOrderRefinedLevel(FiniteElementCollection* fec, int dim = 1,
+                             int ordering = Ordering::byVDIM) override;
+
+   /// Returns the finite element space at the given level
+   const ParFiniteElementSpace& GetFESpaceAtLevel(int level) const override;
+
+   /// Returns the finite element space at the given level
+   ParFiniteElementSpace& GetFESpaceAtLevel(int level) override;
+
+   /// Returns the finite element space at the finest level
+   const ParFiniteElementSpace& GetFinestFESpace() const override;
+
+   /// Returns the finite element space at the finest level
+   ParFiniteElementSpace& GetFinestFESpace() override;
+};
+#endif
+
+} // namespace mfem
+
+#endif

fem/geom.cpp

@@ -188,7 +188,6 @@ Geometry::Geometry()
       IsoparametricTransformation tri_T;
       tri_T.SetFE(&TriangleFE);
       GetPerfPointMat (TRIANGLE, tri_T.GetPointMat());
-      tri_T.FinalizeTransformation();
       tri_T.SetIntPoint(&GeomCenter[TRIANGLE]);
       *GeomToPerfGeomJac[TRIANGLE] = tri_T.Jacobian();
       CalcInverse(tri_T.Jacobian(), *PerfGeomToGeomJac[TRIANGLE]);
@@ -198,7 +197,6 @@ Geometry::Geometry()
       IsoparametricTransformation tet_T;
       tet_T.SetFE(&TetrahedronFE);
       GetPerfPointMat (TETRAHEDRON, tet_T.GetPointMat());
-      tet_T.FinalizeTransformation();
       tet_T.SetIntPoint(&GeomCenter[TETRAHEDRON]);
       *GeomToPerfGeomJac[TETRAHEDRON] = tet_T.Jacobian();
       CalcInverse(tet_T.Jacobian(), *PerfGeomToGeomJac[TETRAHEDRON]);
@@ -208,7 +206,6 @@ Geometry::Geometry()
       IsoparametricTransformation pri_T;
       pri_T.SetFE(&WedgeFE);
       GetPerfPointMat (PRISM, pri_T.GetPointMat());
-      pri_T.FinalizeTransformation();
       pri_T.SetIntPoint(&GeomCenter[PRISM]);
       *GeomToPerfGeomJac[PRISM] = pri_T.Jacobian();
       CalcInverse(pri_T.Jacobian(), *PerfGeomToGeomJac[PRISM]);

fem/gridfunc.cpp

@@ -2753,6 +2753,15 @@ void GridFunction::Save(std::ostream &out) const
    out.flush();
 }
 
+#ifdef MFEM_USE_ADIOS2
+void GridFunction::Save(adios2stream &out,
+                        const std::string& variable_name,
+                        const adios2stream::data_type type) const
+{
+   out.Save(*this, variable_name, type);
+}
+#endif
+
 void GridFunction::SaveVTK(std::ostream &out, const std::string &field_name,
                            int ref)
 {

fem/gridfunc.hpp

@@ -16,6 +16,9 @@
 #include "fespace.hpp"
 #include "coefficient.hpp"
 #include "bilininteg.hpp"
+#ifdef MFEM_USE_ADIOS2
+#include "../general/adios2stream.hpp"
+#endif
 #include <limits>
 #include <ostream>
 #include <string>
@@ -486,6 +489,13 @@ public:
    /// Save the GridFunction to an output stream.
    virtual void Save(std::ostream &out) const;
 
+#ifdef MFEM_USE_ADIOS2
+   /// Save the GridFunction to a binary output stream using adios2 bp format.
+   virtual void Save(adios2stream &out, const std::string& variable_name,
+                     const adios2stream::data_type
+                     type = adios2stream::data_type::point_data) const;
+#endif
+
    /** Write the GridFunction in VTK format. Note that Mesh::PrintVTK must be
        called first. The parameter ref > 0 must match the one used in
        Mesh::PrintVTK. */

fem/linearform.hpp

@@ -100,7 +100,7 @@ public:
 
    /// (DEPRECATED) Return the FE space associated with the LinearForm.
    /** @deprecated Use FESpace() instead. */
-   FiniteElementSpace *GetFES() { return fes; }
+   MFEM_DEPRECATED FiniteElementSpace *GetFES() { return fes; }
 
    /// Read+write access to the associated FiniteElementSpace.
    FiniteElementSpace *FESpace() { return fes; }

fem/lininteg.cpp

@@ -93,6 +93,37 @@ void BoundaryLFIntegrator::AssembleRHSElementVect(
    }
 }
 
+void BoundaryLFIntegrator::AssembleRHSElementVect(
+   const FiniteElement &el, FaceElementTransformations &Tr, Vector &elvect)
+{
+   int dof = el.GetDof();
+
+   shape.SetSize(dof);        // vector of size dof
+   elvect.SetSize(dof);
+   elvect = 0.0;
+
+   const IntegrationRule *ir = IntRule;
+   if (ir == NULL)
+   {
+      int intorder = oa * el.GetOrder() + ob;    // <------ user control
+      ir = &IntRules.Get(Tr.FaceGeom, intorder); // of integration order
+   }
+
+   for (int i = 0; i < ir->GetNPoints(); i++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(i);
+      IntegrationPoint eip;
+      Tr.Loc1.Transform(ip, eip);
+
+      Tr.Face->SetIntPoint (&ip);
+      double val = Tr.Face->Weight() * ip.weight * Q.Eval(*Tr.Face, ip);
+
+      el.CalcShape(eip, shape);
+
+      add(elvect, val, shape, elvect);
+   }
+}
+
 void BoundaryNormalLFIntegrator::AssembleRHSElementVect(
    const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
 {

fem/lininteg.hpp

@@ -126,7 +126,8 @@ class BoundaryLFIntegrator : public LinearFormIntegrator
    Coefficient &Q;
    int oa, ob;
 public:
-   /// Constructs a boundary integrator with a given Coefficient QG
+   /** @brief Constructs a boundary integrator with a given Coefficient @a QG.
+       Integration order will be @a a * basis_order + @a b. */
    BoundaryLFIntegrator(Coefficient &QG, int a = 1, int b = 1)
       : Q(QG), oa(a), ob(b) { }
 
@@ -135,8 +136,9 @@ public:
    virtual void AssembleRHSElementVect(const FiniteElement &el,
                                        ElementTransformation &Tr,
                                        Vector &elvect);
-
-   using LinearFormIntegrator::AssembleRHSElementVect;
+   virtual void AssembleRHSElementVect(const FiniteElement &el,
+                                       FaceElementTransformations &Tr,
+                                       Vector &elvect);
 };
 
 /// Class for boundary integration \f$ L(v) = (g \cdot n, v) \f$

fem/multigrid.cpp

@@ -0,0 +1,206 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "multigrid.hpp"
+
+namespace mfem
+{
+
+Multigrid::Multigrid(const FiniteElementSpaceHierarchy& fespaces_)
+   : fespaces(fespaces_), cycleType(CycleType::VCYCLE), preSmoothingSteps(1),
+     postSmoothingSteps(1)
+{}
+
+Multigrid::~Multigrid()
+{
+   for (int i = 0; i < operators.Size(); ++i)
+   {
+      if (ownedOperators[i])
+      {
+         delete operators[i];
+      }
+      if (ownedSmoothers[i])
+      {
+         delete smoothers[i];
+      }
+      delete X[i];
+      delete Y[i];
+      delete R[i];
+      delete Z[i];
+   }
+
+   operators.DeleteAll();
+   smoothers.DeleteAll();
+   X.DeleteAll();
+   Y.DeleteAll();
+   R.DeleteAll();
+   Z.DeleteAll();
+
+   for (int i = 0; i < bfs.Size(); ++i)
+   {
+      delete bfs[i];
+   }
+
+   bfs.DeleteAll();
+
+   for (int i = 0; i < essentialTrueDofs.Size(); ++i)
+   {
+      delete essentialTrueDofs[i];
+   }
+
+   essentialTrueDofs.DeleteAll();
+}
+
+void Multigrid::AddLevel(Operator* opr, Solver* smoother, bool ownOperator,
+                         bool ownSmoother)
+{
+   operators.Append(opr);
+   smoothers.Append(smoother);
+   ownedOperators.Append(ownOperator);
+   ownedSmoothers.Append(ownSmoother);
+   width = opr->Width();
+   height = opr->Height();
+
+   X.Append(new Vector(height));
+   *X.Last() = 0.0;
+   Y.Append(new Vector(height));
+   *Y.Last() = 0.0;
+   R.Append(new Vector(height));
+   *R.Last() = 0.0;
+   Z.Append(new Vector(height));
+   *Z.Last() = 0.0;
+}
+
+int Multigrid::NumLevels() const { return operators.Size(); }
+
+int Multigrid::GetFinestLevelIndex() const { return NumLevels() - 1; }
+
+const Operator* Multigrid::GetOperatorAtLevel(int level) const
+{
+   return operators[level];
+}
+
+Operator* Multigrid::GetOperatorAtLevel(int level)
+{
+   return operators[level];
+}
+
+const Operator* Multigrid::GetOperatorAtFinestLevel() const
+{
+   return GetOperatorAtLevel(operators.Size() - 1);
+}
+
+Operator* Multigrid::GetOperatorAtFinestLevel()
+{
+   return GetOperatorAtLevel(operators.Size() - 1);
+}
+
+Solver* Multigrid::GetSmootherAtLevel(int level) const
+{
+   return smoothers[level];
+}
+
+Solver* Multigrid::GetSmootherAtLevel(int level)
+{
+   return smoothers[level];
+}
+
+void Multigrid::SetCycleType(CycleType cycleType_, int preSmoothingSteps_,
+                             int postSmoothingSteps_)
+{
+   cycleType = cycleType_;
+   preSmoothingSteps = preSmoothingSteps_;
+   postSmoothingSteps = postSmoothingSteps_;
+}
+
+void Multigrid::Mult(const Vector& x, Vector& y) const
+{
+   MFEM_ASSERT(NumLevels() > 0, "");
+   *X.Last() = x;
+   *Y.Last() = 0.0;
+   Cycle(GetFinestLevelIndex());
+   y = *Y.Last();
+}
+
+void Multigrid::SetOperator(const Operator& op)
+{
+   MFEM_ABORT("SetOperator not supported in Multigrid");
+}
+
+void Multigrid::SmoothingStep(int level) const
+{
+   GetOperatorAtLevel(level)->Mult(*Y[level], *R[level]); // r = A x
+   subtract(*X[level], *R[level], *R[level]);             // r = b - A x
+   GetSmootherAtLevel(level)->Mult(*R[level], *Z[level]); // z = S r
+   add(*Y[level], 1.0, *Z[level], *Y[level]);             // x = x + S (b - A x)
+}
+
+void Multigrid::Cycle(int level) const
+{
+   if (level == 0)
+   {
+      GetSmootherAtLevel(level)->Mult(*X[level], *Y[level]);
+      return;
+   }
+
+   for (int i = 0; i < preSmoothingSteps; i++)
+   {
+      SmoothingStep(level);
+   }
+
+   // Compute residual
+   GetOperatorAtLevel(level)->Mult(*Y[level], *R[level]);
+   subtract(*X[level], *R[level], *R[level]);
+
+   // Restrict residual
+   fespaces.GetProlongationAtLevel(level - 1)->MultTranspose(*R[level],
+                                                             *X[level - 1]);
+
+   // Init zeros
+   *Y[level - 1] = 0.0;
+
+   // Corrections
+   int corrections = 1;
+   if (cycleType == CycleType::WCYCLE)
+   {
+      corrections = 2;
+   }
+   for (int correction = 0; correction < corrections; ++correction)
+   {
+      Cycle(level - 1);
+   }
+
+   // Prolongate
+   fespaces.GetProlongationAtLevel(level - 1)->Mult(*Y[level - 1], *R[level]);
+
+   // Add update
+   *Y[level] += *R[level];
+
+   // Post-smooth
+   for (int i = 0; i < postSmoothingSteps; i++)
+   {
+      SmoothingStep(level);
+   }
+}
+
+void Multigrid::FormFineLinearSystem(Vector& x, Vector& b, OperatorHandle& A,
+                                     Vector& X, Vector& B)
+{
+   bfs.Last()->FormLinearSystem(*essentialTrueDofs.Last(), x, b, A, X, B);
+}
+
+void Multigrid::RecoverFineFEMSolution(const Vector& X, const Vector& b,
+                                       Vector& x)
+{
+   bfs.Last()->RecoverFEMSolution(X, b, x);
+}
+
+} // namespace mfem

fem/multigrid.hpp

@@ -0,0 +1,119 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#ifndef MFEM_MULTIGRID
+#define MFEM_MULTIGRID
+
+#include "fespacehierarchy.hpp"
+#include "bilinearform.hpp"
+
+#include "../linalg/operator.hpp"
+#include "../linalg/handle.hpp"
+
+namespace mfem
+{
+
+/// Multigrid solver class
+class Multigrid : public Solver
+{
+public:
+   enum class CycleType
+   {
+      VCYCLE,
+      WCYCLE
+   };
+
+protected:
+   const FiniteElementSpaceHierarchy& fespaces;
+   Array<Array<int>*> essentialTrueDofs;
+   Array<BilinearForm*> bfs;
+
+private:
+   Array<Operator*> operators;
+   Array<Solver*> smoothers;
+
+   Array<bool> ownedOperators;
+   Array<bool> ownedSmoothers;
+
+   CycleType cycleType;
+   int preSmoothingSteps;
+   int postSmoothingSteps;
+
+   mutable Array<Vector*> X;
+   mutable Array<Vector*> Y;
+   mutable Array<Vector*> R;
+   mutable Array<Vector*> Z;
+
+public:
+   /// Constructs an empty multigrid for the given FiniteElementSpaceHierarchy
+   Multigrid(const FiniteElementSpaceHierarchy& fespaces_);
+
+   /// Destructor
+   virtual ~Multigrid();
+
+   /// Adds a level to the multigrid operator hierarchy.
+   /** The ownership of the operators and solvers/smoothers may be transferred
+       to the Multigrid by setting the according boolean variables. */
+   void AddLevel(Operator* opr, Solver* smoother, bool ownOperator,
+                 bool ownSmoother);
+
+   /// Returns the number of levels
+   int NumLevels() const;
+
+   /// Returns the index of the finest level
+   int GetFinestLevelIndex() const;
+
+   /// Returns operator at given level
+   const Operator* GetOperatorAtLevel(int level) const;
+
+   /// Returns operator at given level
+   Operator* GetOperatorAtLevel(int level);
+
+   /// Returns operator at finest level
+   const Operator* GetOperatorAtFinestLevel() const;
+
+   /// Returns operator at finest level
+   Operator* GetOperatorAtFinestLevel();
+
+   /// Returns smoother at given level
+   Solver* GetSmootherAtLevel(int level) const;
+
+   /// Returns smoother at given level
+   Solver* GetSmootherAtLevel(int level);
+
+   /// Set the cycle type and number of pre- and post-smoothing steps used by Mult
+   void SetCycleType(CycleType cycleType_, int preSmoothingSteps_,
+                     int postSmoothingSteps_);
+
+   /// Application of the multigrid as a preconditioner
+   virtual void Mult(const Vector& x, Vector& y) const override;
+
+   /// Not supported for multigrid
+   virtual void SetOperator(const Operator& op) override;
+
+   /// Form the linear system A X = B, corresponding to the operator on the finest level
+   void FormFineLinearSystem(Vector& x, Vector& b, OperatorHandle& A, Vector& X,
+                             Vector& B);
+
+   /// Recover the solution of a linear system formed with FormFineLinearSystem()
+   void RecoverFineFEMSolution(const Vector& X, const Vector& b, Vector& x);
+
+private:
+   /// Application of a smoothing step at particular level
+   void SmoothingStep(int level) const;
+
+   /// Application of a cycle at particular level
+   void Cycle(int level) const;
+};
+
+} // namespace mfem
+
+#endif

fem/nonlinearform.hpp

@@ -111,8 +111,8 @@ public:
        have zero entries at the essential true dofs. */
    void SetEssentialBC(const Array<int> &bdr_attr_is_ess, Vector *rhs = NULL);
 
-   /// (DEPRECATED) Specify essential boundary conditions.
-   /** @deprecated Use either SetEssentialBC() or SetEssentialTrueDofs(). */
+   /// Specify essential boundary conditions.
+   /** Use either SetEssentialBC() or SetEssentialTrueDofs() if possible. */
    void SetEssentialVDofs(const Array<int> &ess_vdofs_list);
 
    /// Specify essential boundary conditions.

fem/pbilinearform.cpp

@@ -283,7 +283,14 @@ const
    }
 
    X.Distribute(&x);
-   mat->Mult(X, Y);
+   if (ext)
+   {
+      ext->Mult(X, Y);
+   }
+   else
+   {
+      mat->Mult(X, Y);
+   }
    pfes->Dof_TrueDof_Matrix()->MultTranspose(a, Y, 1.0, y);
 }
 

fem/pfespace.cpp

@@ -2029,7 +2029,6 @@ int ParFiniteElementSpace
                if (!slave_dofs.Size()) { continue; }
 
                sf.OrientedPointMatrix(T.GetPointMat());
-               T.FinalizeTransformation();
                fe->GetLocalInterpolation(T, I);
 
                // make each slave DOF dependent on all master DOFs

fem/pfespace.hpp

@@ -240,7 +240,7 @@ public:
    int GetNRanks() const { return NRanks; }
    int GetMyRank() const { return MyRank; }
 
-   inline ParMesh *GetParMesh() { return pmesh; }
+   inline ParMesh *GetParMesh() const { return pmesh; }
 
    int GetDofSign(int i)
    { return NURBSext || Nonconforming() ? 1 : ldof_sign[VDofToDof(i)]; }

fem/pgridfunc.cpp

@@ -16,6 +16,7 @@
 #include "fem.hpp"
 #include <iostream>
 #include <limits>
+#include <string>
 #include "../general/forall.hpp"
 using namespace std;
 
@@ -78,6 +79,229 @@ ParGridFunction::ParGridFunction(ParMesh *pmesh, std::istream &input)
    fes = pfes;
 }
 
+ParGridFunction::ParGridFunction(ParFiniteElementSpace *pf,
+                                 const char *_filename)
+   : GridFunction(pf), pfes(pf)
+{
+   MPI_Comm fes_comm;
+   int fes_rank, n_fes_ranks;
+   fes_comm = pfes->GetComm();
+   MPI_Comm_size(fes_comm, &n_fes_ranks);
+   MPI_Comm_rank(fes_comm, &fes_rank);
+
+   std::string filename(_filename);
+   std::string file_prefix;
+   std::string file_ext;
+   {
+      size_t i = filename.rfind('.', filename.length());
+      if (i != string::npos)
+      {
+         file_prefix = (filename.substr(0, i));
+         file_ext = (filename.substr(i, filename.length() - i));
+      }
+   }
+
+   int nfiles = 1;
+   if (fes_rank == 0)
+   {
+      int n_rfes_ranks;
+      int tmp[2];
+      std::string mpi_filename;
+      size_t i = filename.rfind('.', filename.length());
+      if (i != string::npos)
+      {
+         mpi_filename = file_prefix + to_string(0) + file_ext;
+      }
+      else
+      {
+         mpi_filename = filename + to_string(0);
+      }
+      
+      MPI_File fh;
+      MPI_File_open(MPI_COMM_SELF, mpi_filename.c_str(), MPI_MODE_RDONLY,
+                                                      MPI_INFO_NULL, &fh);
+      MPI_File_read_at(fh, 0, tmp, 2, MPI_INT, MPI_STATUS_IGNORE);
+      MPI_File_close(&fh);
+      
+      n_rfes_ranks = tmp[0];
+      nfiles = tmp[1];
+
+      MFEM_ASSERT(n_fes_ranks == n_rfes_ranks,
+         "ParGridFunction::ParGridFunction(ParFiniteElementSpace *pf,"
+               " const char *_filename):\n"
+               "\tThe number of MPI ranks used to save the GridFunction is\n"
+               "\tnot the same as the number used to load it!");
+   }
+   MPI_Bcast(&nfiles, 1, MPI_INT, 0, fes_comm);
+
+   int color = fes_rank * nfiles / n_fes_ranks;
+   
+   MPI_Comm file_comm;
+   MPI_Comm_split(fes_comm, color, fes_rank, &file_comm);
+
+   int file_rank, n_file_ranks;
+   MPI_Comm_size(file_comm, &n_file_ranks);
+   MPI_Comm_rank(file_comm, &file_rank);
+
+   std::string mpi_filename;
+   {
+      size_t i = filename.rfind('.', filename.length());
+      if (i != string::npos) {
+         mpi_filename = file_prefix + std::to_string(color) + file_ext;
+      }
+      else
+      {
+         mpi_filename = filename + std::to_string(color);
+      }
+   }
+
+   MPI_File fh;
+   MPI_File_open(file_comm, mpi_filename.c_str(), MPI_MODE_RDONLY,
+                                                  MPI_INFO_NULL, &fh);
+
+   int *dof_counts = new int[5*n_file_ranks];
+   int **nv = new int*[n_file_ranks];
+   int **nvdofs = new int*[n_file_ranks];
+   int **nedofs = new int*[n_file_ranks];
+   int **nfdofs = new int*[n_file_ranks];
+   int **nrdofs = new int*[n_file_ranks];
+
+   for (int i = 0; i < n_file_ranks; ++i)
+   {
+      nv[i]     = &dof_counts[i*5+0];
+      nvdofs[i] = &dof_counts[i*5+1];
+      nedofs[i] = &dof_counts[i*5+2];
+      nfdofs[i] = &dof_counts[i*5+3];
+      nrdofs[i] = &dof_counts[i*5+4];
+   }
+
+   *nv[file_rank] = pfes->GetVSize();
+   *nvdofs[file_rank] = pfes->GetNVDofs();
+   *nedofs[file_rank] = pfes->GetNEDofs();
+   *nfdofs[file_rank] = pfes->GetNFDofs();
+
+   int vdim = pfes->GetVDim();
+   *nrdofs[file_rank] = *nv[file_rank] / vdim - *nvdofs[file_rank] -
+                                       *nedofs[file_rank] - *nfdofs[file_rank];
+
+   MPI_Allgather(MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, &dof_counts[0], 5,
+                 MPI_INT, file_comm);
+
+   double *data_ = HostWrite();
+
+   MPI_Offset header_offset = 0;
+   header_offset += 2 * sizeof(int);
+   MPI_Offset v_offset, e_offset, f_offset, r_offset;
+
+   int total_vdofs = 0, total_edofs = 0, total_fdofs = 0, total_rdofs = 0;
+   int total_scalar_dofs = 0;
+
+   for (int i = 0; i < n_file_ranks; ++i)
+   {
+      total_vdofs += *nvdofs[i];
+      total_edofs += *nedofs[i];
+      total_fdofs += *nfdofs[i];
+      total_rdofs += *nrdofs[i];
+      total_scalar_dofs += *nv[i];
+   }
+
+   total_scalar_dofs /= vdim;
+
+   if (pfes->GetOrdering() == Ordering::byNODES)
+   {
+      for (int d = 0; d < vdim; ++d)
+      {
+         int v_data_offset = 0 + *nv[file_rank] * d / vdim ;
+         int e_data_offset = v_data_offset + *nvdofs[file_rank];
+         int f_data_offset = e_data_offset + *nedofs[file_rank];
+         int r_data_offset = f_data_offset + *nfdofs[file_rank];
+
+         v_offset = header_offset;
+         e_offset = header_offset;
+         f_offset = header_offset;
+         r_offset = header_offset;
+         
+         v_offset += total_scalar_dofs * d * sizeof(double);
+         e_offset += (total_vdofs + total_scalar_dofs * d) * sizeof(double);
+         f_offset += (total_vdofs + total_edofs +
+                                    total_scalar_dofs * d) * sizeof(double);
+         r_offset += (total_vdofs + total_edofs + total_fdofs +
+                                    total_scalar_dofs * d) * sizeof(double);
+
+
+         for (int i = 0; i < file_rank; ++i)
+         {
+            v_offset += *nvdofs[i] * sizeof(double);
+            e_offset += *nedofs[i] * sizeof(double);
+            f_offset += *nfdofs[i] * sizeof(double);
+            r_offset += *nrdofs[i] * sizeof(double);
+         }
+
+         MPI_File_read_at_all(fh, v_offset, &data_[v_data_offset],
+                              *nvdofs[file_rank], MPI_DOUBLE,
+                              MPI_STATUS_IGNORE);
+         MPI_File_read_at_all(fh, e_offset, &data_[e_data_offset],
+                              *nedofs[file_rank], MPI_DOUBLE,
+                              MPI_STATUS_IGNORE);
+         MPI_File_read_at_all(fh, f_offset, &data_[f_data_offset],
+                              *nfdofs[file_rank], MPI_DOUBLE,
+                              MPI_STATUS_IGNORE);
+         MPI_File_read_at_all(fh, r_offset, &data_[r_data_offset],
+                              *nrdofs[file_rank], MPI_DOUBLE,
+                              MPI_STATUS_IGNORE);
+      }
+   }
+   else
+   {
+      v_offset = header_offset;
+      e_offset = v_offset + total_vdofs * vdim * sizeof(double);
+      f_offset = e_offset + total_edofs * vdim * sizeof(double);
+      r_offset = f_offset + total_fdofs * vdim * sizeof(double);
+
+      for (int i = 0; i < file_rank; ++i)
+      {
+         v_offset += *nvdofs[i] * sizeof(double) * vdim;
+         e_offset += *nedofs[i] * sizeof(double) * vdim;
+         f_offset += *nfdofs[i] * sizeof(double) * vdim;
+         r_offset += *nrdofs[i] * sizeof(double) * vdim;
+      }
+
+      int v_data_offset = 0;
+      int e_data_offset = v_data_offset + *nvdofs[file_rank] * vdim;
+      int f_data_offset = e_data_offset + *nedofs[file_rank] * vdim;
+      int r_data_offset = f_data_offset + *nfdofs[file_rank] * vdim;
+      
+      MPI_File_read_at_all(fh, v_offset, &data_[v_data_offset],
+                           *nvdofs[file_rank] * vdim, MPI_DOUBLE,
+                           MPI_STATUS_IGNORE);
+      MPI_File_read_at_all(fh, e_offset, &data_[e_data_offset],
+                           *nedofs[file_rank] * vdim, MPI_DOUBLE,
+                           MPI_STATUS_IGNORE);
+      MPI_File_read_at_all(fh, f_offset, &data_[f_data_offset],
+                           *nfdofs[file_rank] * vdim, MPI_DOUBLE,
+                           MPI_STATUS_IGNORE);
+      MPI_File_read_at_all(fh, r_offset, &data_[r_data_offset],
+                           *nrdofs[file_rank] * vdim, MPI_DOUBLE,
+                           MPI_STATUS_IGNORE);
+   }
+
+   MPI_File_close(&fh);
+   MPI_Comm_free(&file_comm);
+
+   for (int i = 0; i < size; i++)
+   {
+      if (pfes->GetDofSign(i) < 0) { data_[i] = -data_[i]; }
+   }
+
+   delete[] dof_counts;
+   delete[] nv;
+   delete[] nvdofs;
+   delete[] nedofs;
+   delete[] nfdofs;
+   delete[] nrdofs;
+}
+
+
 void ParGridFunction::Update()
 {
    face_nbr_data.Destroy();
@@ -498,6 +722,221 @@ void ParGridFunction::Save(std::ostream &out) const
    }
 }
 
+#ifdef MFEM_USE_ADIOS2
+void ParGridFunction::Save(adios2stream &out,
+                           const std::string& variable_name,
+                           const adios2stream::data_type type) const
+{
+   double *data_  = const_cast<double*>(HostRead());
+   for (int i = 0; i < size; i++)
+   {
+      if (pfes->GetDofSign(i) < 0) { data_[i] = -data_[i]; }
+   }
+
+   GridFunction::Save(out, variable_name, type);
+
+   for (int i = 0; i < size; i++)
+   {
+      if (pfes->GetDofSign(i) < 0) { data_[i] = -data_[i]; }
+   }
+}
+#endif
+
+void ParGridFunction::Save(const char *_filename, const int nfiles)
+{
+   MPI_Comm fes_comm;
+   int fes_rank, n_fes_ranks;
+   fes_comm = pfes->GetComm();
+
+   MPI_Comm_size(fes_comm, &n_fes_ranks);
+   MPI_Comm_rank(fes_comm, &fes_rank);
+
+   int color = fes_rank * nfiles / n_fes_ranks;
+   
+   MPI_Comm file_comm;
+   MPI_Comm_split(fes_comm, color, fes_rank, &file_comm);
+
+   int file_rank, n_file_ranks;
+   MPI_Comm_size(file_comm, &n_file_ranks);
+   MPI_Comm_rank(file_comm, &file_rank);
+
+   std::string filename(_filename);
+   std::string file_prefix;
+   std::string file_ext;
+   std::string mpi_filename;
+   {
+      size_t i = filename.rfind('.', filename.length());
+      if (i != string::npos)
+      {
+         file_prefix = (filename.substr(0, i));
+         file_ext = (filename.substr(i, filename.length() - i));
+         mpi_filename = file_prefix + std::to_string(color) + file_ext;
+      }
+      else
+      {
+         mpi_filename = filename + std::to_string(color);
+      }
+   }
+
+   MPI_File fh;
+   MPI_File_open(file_comm, mpi_filename.c_str(), MPI_MODE_CREATE |
+                                                  MPI_MODE_WRONLY,
+                                                  MPI_INFO_NULL, &fh);
+
+   int *dof_counts = new int[5*n_file_ranks];
+   int **nv = new int*[n_file_ranks];
+   int **nvdofs = new int*[n_file_ranks];
+   int **nedofs = new int*[n_file_ranks];
+   int **nfdofs = new int*[n_file_ranks];
+   int **nrdofs = new int*[n_file_ranks];
+
+   for (int i = 0; i < n_file_ranks; ++i)
+   {
+      nv[i]     = &dof_counts[i*5+0];
+      nvdofs[i] = &dof_counts[i*5+1];
+      nedofs[i] = &dof_counts[i*5+2];
+      nfdofs[i] = &dof_counts[i*5+3];
+      nrdofs[i] = &dof_counts[i*5+4];
+   }
+
+   *nv[file_rank] = pfes->GetVSize();
+   *nvdofs[file_rank] = pfes->GetNVDofs();
+   *nedofs[file_rank] = pfes->GetNEDofs();
+   *nfdofs[file_rank] = pfes->GetNFDofs();
+
+   int vdim = pfes->GetVDim();
+   *nrdofs[file_rank] = *nv[file_rank] / vdim - *nvdofs[file_rank] -
+                                       *nedofs[file_rank] - *nfdofs[file_rank];
+
+   MPI_Allgather(MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, &dof_counts[0], 5,
+                 MPI_INT, file_comm);
+
+   double *data_  = const_cast<double*>(HostRead());
+   for (int i = 0; i < size; i++)
+   {
+      if (pfes->GetDofSign(i) < 0) { data_[i] = -data_[i]; }
+   }
+
+   MPI_Offset header_offset = 0;
+
+   if (file_rank == 0)
+   {
+      int tmp[] = {n_fes_ranks, nfiles};
+      MPI_File_write_at(fh, header_offset, &tmp, 2, MPI_INT,
+                        MPI_STATUS_IGNORE);
+   }
+
+   header_offset += 2 * sizeof(int);
+
+   MPI_Offset v_offset, e_offset, f_offset, r_offset;
+
+   int total_vdofs = 0, total_edofs = 0, total_fdofs = 0, total_rdofs = 0;
+   int total_scalar_dofs = 0;
+
+   for (int i = 0; i < n_file_ranks; ++i)
+   {
+      total_vdofs += *nvdofs[i];
+      total_edofs += *nedofs[i];
+      total_fdofs += *nfdofs[i];
+      total_rdofs += *nrdofs[i];
+      total_scalar_dofs += *nv[i];
+   }
+
+   total_scalar_dofs /= vdim;
+
+   if (pfes->GetOrdering() == Ordering::byNODES)
+   {
+      for (int d = 0; d < vdim; ++d)
+      {
+         int v_data_offset = 0 + *nv[file_rank] * d / vdim ;
+         int e_data_offset = v_data_offset + *nvdofs[file_rank];
+         int f_data_offset = e_data_offset + *nedofs[file_rank];
+         int r_data_offset = f_data_offset + *nfdofs[file_rank];
+
+         v_offset = header_offset;
+         e_offset = header_offset;
+         f_offset = header_offset;
+         r_offset = header_offset;
+         
+         v_offset += total_scalar_dofs * d * sizeof(double);
+         e_offset += (total_vdofs + total_scalar_dofs * d) * sizeof(double);
+         f_offset += (total_vdofs + total_edofs +
+                                    total_scalar_dofs * d) * sizeof(double);
+         r_offset += (total_vdofs + total_edofs + total_fdofs +
+                                    total_scalar_dofs * d) * sizeof(double);
+
+         for (int i = 0; i < file_rank; ++i)
+         {
+            v_offset += *nvdofs[i] * sizeof(double);
+            e_offset += *nedofs[i] * sizeof(double);
+            f_offset += *nfdofs[i] * sizeof(double);
+            r_offset += *nrdofs[i] * sizeof(double);
+         }
+
+         MPI_File_write_at_all(fh, v_offset, &data_[v_data_offset],
+                               *nvdofs[file_rank], MPI_DOUBLE,
+                               MPI_STATUS_IGNORE);
+         MPI_File_write_at_all(fh, e_offset, &data_[e_data_offset],
+                               *nedofs[file_rank], MPI_DOUBLE,
+                               MPI_STATUS_IGNORE);
+         MPI_File_write_at_all(fh, f_offset, &data_[f_data_offset],
+                               *nfdofs[file_rank], MPI_DOUBLE,
+                               MPI_STATUS_IGNORE);
+         MPI_File_write_at_all(fh, r_offset, &data_[r_data_offset],
+                               *nrdofs[file_rank], MPI_DOUBLE,
+                               MPI_STATUS_IGNORE);
+      }
+   }
+   else
+   {
+      v_offset = header_offset;
+      e_offset = v_offset + total_vdofs * vdim * sizeof(double);
+      f_offset = e_offset + total_edofs * vdim * sizeof(double);
+      r_offset = f_offset + total_fdofs * vdim * sizeof(double);
+
+      for (int i = 0; i < file_rank; ++i)
+      {
+         v_offset += *nvdofs[i] * sizeof(double) * vdim;
+         e_offset += *nedofs[i] * sizeof(double) * vdim;
+         f_offset += *nfdofs[i] * sizeof(double) * vdim;
+         r_offset += *nrdofs[i] * sizeof(double) * vdim;
+      }
+
+      int v_data_offset = 0;
+      int e_data_offset = v_data_offset + *nvdofs[file_rank] * vdim;
+      int f_data_offset = e_data_offset + *nedofs[file_rank] * vdim;
+      int r_data_offset = f_data_offset + *nfdofs[file_rank] * vdim;
+      
+      MPI_File_write_at_all(fh, v_offset, &data_[v_data_offset],
+                            *nvdofs[file_rank] * vdim, MPI_DOUBLE,
+                            MPI_STATUS_IGNORE);
+      MPI_File_write_at_all(fh, e_offset, &data_[e_data_offset],
+                            *nedofs[file_rank] * vdim, MPI_DOUBLE,
+                            MPI_STATUS_IGNORE);
+      MPI_File_write_at_all(fh, f_offset, &data_[f_data_offset],
+                            *nfdofs[file_rank] * vdim, MPI_DOUBLE,
+                            MPI_STATUS_IGNORE);
+      MPI_File_write_at_all(fh, r_offset, &data_[r_data_offset],
+                            *nrdofs[file_rank] * vdim, MPI_DOUBLE,
+                            MPI_STATUS_IGNORE);
+   }
+
+   MPI_File_close(&fh);
+   MPI_Comm_free(&file_comm);
+   
+   for (int i = 0; i < size; i++)
+   {
+      if (pfes->GetDofSign(i) < 0) { data_[i] = -data_[i]; }
+   }
+
+   delete[] dof_counts;
+   delete[] nv;
+   delete[] nvdofs;
+   delete[] nedofs;
+   delete[] nfdofs;
+   delete[] nrdofs;
+}
+
 void ParGridFunction::SaveAsOne(std::ostream &out)
 {
    int i, p;

fem/pgridfunc.hpp

@@ -83,6 +83,13 @@ public:
        constructed. The new ParGridFunction assumes ownership of both. */
    ParGridFunction(ParMesh *pmesh, std::istream &input);
 
+   /// Construct a ParGridFunction by loading a ParGridFunction saved using
+   /// ParGridFunction::Save(char *filename, int nfiles).
+   /** The parallel space @a *pf and the space used by the GridFunction saved
+       in @a *filename should match. The number of ranks used when loading the
+       ParGridFunction must be the same as when it was saved. */
+   ParGridFunction(ParFiniteElementSpace *pf, const char *filename);
+
    /// Copy assignment. Only the data of the base class Vector is copied.
    /** It is assumed that this object and @a rhs use ParFiniteElementSpace%s
        that have the same size.
@@ -310,11 +317,34 @@ public:
                             GridFunction &flux,
                             bool wcoef = true, int subdomain = -1);
 
-   /** Save the local portion of the ParGridFunction. It differs from the
+   /** Save the local portion of the ParGridFunction. This differs from the
        serial GridFunction::Save in that it takes into account the signs of
        the local dofs. */
    virtual void Save(std::ostream &out) const;
 
+#ifdef MFEM_USE_ADIOS2
+   /** Save the local portion of the ParGridFunction. This differs from the
+       serial GridFunction::Save in that it takes into account the signs of
+       the local dofs. */
+   virtual void Save(
+      adios2stream &out, const std::string &variable_name,
+      const adios2stream::data_type type = adios2stream::data_type::point_data) const;
+#endif
+
+   /** Save the local grid functions to n number of files, where each file will
+       contain the grid functions from potentially multiple ranks. This is
+       similar to the syncIO approach from "Fu, Jing, et al. 'Scalable parallel
+       I/O alternatives for massively parallel partitioned solver systems.'
+       2010 IEEE International Symposium on Parallel & Distributed Processing,
+       Workshops and Phd Forum (IPDPSW). IEEE, 2010." 
+       @param[in] filename - filename for output files with extension
+       @param[in] nfiles - number of files to write using MPI-IO
+       @note - takes into account the signs of the local dofs.
+       @note - writes a binary file without the FESpace header; the saved file
+       should only be loaded by the accompanying constructor:
+       ParGridFunction(ParFiniteElementSpace *pf, const char *filename) */
+   void Save(const char *filename, const int nfiles = 1);
+
    /// Merge the local grid functions
    void SaveAsOne(std::ostream &out = mfem::out);
 

fem/quadinterpolator.cpp

@@ -319,6 +319,33 @@ void QuadratureInterpolator::Mult(
          }
       }
    }
+   else if (vdim == 3 && dim == 2)
+   {
+      switch (100*nd + nq)
+      {
+         // Q0
+         case 101: eval_func = &Eval2D<3,1,1>; break;
+         case 104: eval_func = &Eval2D<3,1,4>; break;
+         // Q1
+         case 404: eval_func = &Eval2D<3,4,4>; break;
+         case 409: eval_func = &Eval2D<3,4,9>; break;
+         // Q2
+         case 904: eval_func = &Eval2D<3,9,4>; break;
+         case 909: eval_func = &Eval2D<3,9,9>; break;
+         case 916: eval_func = &Eval2D<3,9,16>; break;
+         case 925: eval_func = &Eval2D<3,9,25>; break;
+         // Q3
+         case 1616: eval_func = &Eval2D<3,16,16>; break;
+         case 1625: eval_func = &Eval2D<3,16,25>; break;
+         case 1636: eval_func = &Eval2D<3,16,36>; break;
+         // Q4
+         case 2525: eval_func = &Eval2D<3,25,25>; break;
+         case 2536: eval_func = &Eval2D<3,25,36>; break;
+         case 2549: eval_func = &Eval2D<3,25,49>; break;
+         case 2564: eval_func = &Eval2D<3,25,64>; break;
+         default:   eval_func = &Eval2D<3>;
+      }
+   }
    else if (vdim == dim)
    {
       if (dim == 2)

fem/quadinterpolator.hpp

@@ -47,7 +47,7 @@ protected:
 
    static const int MAX_NQ2D = 100;
    static const int MAX_ND2D = 100;
-   static const int MAX_VDIM2D = 2;
+   static const int MAX_VDIM2D = 3;
 
    static const int MAX_NQ3D = 1000;
    static const int MAX_ND3D = 1000;

fem/restriction.cpp

@@ -223,6 +223,43 @@ void ElementRestriction::MultTransposeUnsigned(const Vector& x, Vector& y) const
    });
 }
 
+void ElementRestriction::BooleanMask(Vector& y) const
+{
+   // Assumes all elements have the same number of dofs
+   const int nd = dof;
+   const int vd = vdim;
+   const bool t = byvdim;
+
+   Array<char> processed(vd * ndofs);
+   processed = 0;
+
+   auto d_offsets = offsets.HostRead();
+   auto d_indices = indices.HostRead();
+   auto d_x = Reshape(processed.HostReadWrite(), t?vd:ndofs, t?ndofs:vd);
+   auto d_y = Reshape(y.HostWrite(), nd, vd, ne);
+   for (int i = 0; i < ndofs; ++i)
+   {
+      const int offset = d_offsets[i];
+      const int nextOffset = d_offsets[i+1];
+      for (int c = 0; c < vd; ++c)
+      {
+         for (int j = offset; j < nextOffset; ++j)
+         {
+            const int idx_j = d_indices[j];
+            if (d_x(t?c:i,t?i:c))
+            {
+               d_y(idx_j % nd, c, idx_j / nd) = 0.0;
+            }
+            else
+            {
+               d_y(idx_j % nd, c, idx_j / nd) = 1.0;
+               d_x(t?c:i,t?i:c) = 1;
+            }
+         }
+      }
+   }
+}
+
 /// Return the face degrees of freedom returned in Lexicographic order.
 void GetFaceDofs(const int dim, const int face_id,
                  const int dof1d, Array<int> &faceMap)

fem/restriction.hpp

@@ -49,6 +49,14 @@ public:
 
    /// Compute MultTranspose without applying signs based on DOF orientations.
    void MultTransposeUnsigned(const Vector &x, Vector &y) const;
+
+   /// @brief Fills the E-vector y with `boolean` values 0.0 and 1.0 such that each
+   /// each entry of the L-vector is uniquely represented in `y`.
+   /** This means, the sum of the E-vector `y` is equal to the sum of the
+       corresponding L-vector filled with ones. The boolean mask is required to
+       emulate SetSubVector and its transpose on GPUs. This method is running on
+       the host, since the `processed` array requires a large shared memory. */
+   void BooleanMask(Vector& y) const;
 };
 
 /// Operator that converts L2 FiniteElementSpace L-vectors to E-vectors.

fem/tmop.cpp

@@ -993,17 +993,21 @@ void DiscreteAdaptTC::SetSerialDiscreteTargetSpec(GridFunction &tspec_)
    tspec_sav = tspec;
 }
 
-void DiscreteAdaptTC::UpdateTargetSpecification(const Vector &new_x)
+void DiscreteAdaptTC::UpdateTargetSpecification(const Vector &new_x,
+                                                bool use_flag)
 {
+   if (use_flag && good_tspec) { return; }
+
    MFEM_VERIFY(tspec.Size() > 0, "Target specification is not set!");
    adapt_eval->ComputeAtNewPosition(new_x, tspec);
    tspec_sav = tspec;
+
+   good_tspec = use_flag;
 }
 
 void DiscreteAdaptTC::UpdateTargetSpecification(Vector &new_x,
                                                 Vector &IntData)
 {
-   MFEM_VERIFY(tspec.Size() > 0, "Target specification is not set!");
    adapt_eval->ComputeAtNewPosition(new_x, IntData);
 }
 
@@ -1070,14 +1074,17 @@ void DiscreteAdaptTC::ComputeElementTargets(int e_id, const FiniteElement &fe,
 }
 
 void DiscreteAdaptTC::UpdateGradientTargetSpecification(const Vector &x,
-                                                        const double dx)
+                                                        const double dx,
+                                                        bool use_flag)
 {
+   if (use_flag && good_tspec_grad) { return; }
+
    const int dim = tspec_fes->GetFE(0)->GetDim();
    const int cnt = x.Size()/dim;
 
-   if (tspec_perth.Size() != x.Size())
+   if (tspec_pert1h.Size() != x.Size())
    {
-      tspec_perth.SetSize(x.Size());
+      tspec_pert1h.SetSize(x.Size());
    }
 
    Vector TSpecTemp;
@@ -1086,16 +1093,20 @@ void DiscreteAdaptTC::UpdateGradientTargetSpecification(const Vector &x,
    {
       for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) += dx; }
 
-      TSpecTemp.SetDataAndSize(tspec_perth.GetData() + j*cnt, cnt);
+      TSpecTemp.SetDataAndSize(tspec_pert1h.GetData() + j*cnt, cnt);
       UpdateTargetSpecification(xtemp, TSpecTemp);
 
       for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) -= dx; }
    }
+
+   good_tspec_grad = use_flag;
 }
 
 void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
-                                                       const double dx)
+                                                       double dx, bool use_flag)
 {
+   if (use_flag && good_tspec_hess) { return; }
+
    const int dim = tspec_fes->GetFE(0)->GetDim();
    const int cnt = x.Size()/dim;
 
@@ -1142,6 +1153,8 @@ void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
          idx++;
       }
    }
+
+   good_tspec_hess = use_flag;
 }
 
 void AdaptivityEvaluator::SetSerialMetaInfo(const Mesh &m,
@@ -1180,19 +1193,8 @@ void TMOP_Integrator::EnableLimiting(const GridFunction &n0,
                                      const GridFunction &dist, Coefficient &w0,
                                      TMOP_LimiterFunction *lfunc)
 {
-   nodes0 = &n0;
-   coeff0 = &w0;
+   EnableLimiting(n0, w0, lfunc);
    lim_dist = &dist;
-
-   delete lim_func;
-   if (lfunc)
-   {
-      lim_func = lfunc;
-   }
-   else
-   {
-      lim_func = new TMOP_QuadraticLimiter;
-   }
 }
 void TMOP_Integrator::EnableLimiting(const GridFunction &n0, Coefficient &w0,
                                      TMOP_LimiterFunction *lfunc)
@@ -1810,6 +1812,111 @@ void TMOP_Integrator::EnableFiniteDifferences(const ParGridFunction &x)
 }
 #endif
 
+void TMOPComboIntegrator::EnableLimiting(const GridFunction &n0,
+                                         const GridFunction &dist,
+                                         Coefficient &w0,
+                                         TMOP_LimiterFunction *lfunc)
+{
+   MFEM_VERIFY(tmopi.Size() > 0, "No TMOP_Integrators were added.");
+
+   tmopi[0]->EnableLimiting(n0, dist, w0, lfunc);
+   for (int i = 1; i < tmopi.Size(); i++) { tmopi[i]->DisableLimiting(); }
+}
+
+void TMOPComboIntegrator::EnableLimiting(const GridFunction &n0,
+                                         Coefficient &w0,
+                                         TMOP_LimiterFunction *lfunc)
+{
+   MFEM_VERIFY(tmopi.Size() > 0, "No TMOP_Integrators were added.");
+
+   tmopi[0]->EnableLimiting(n0, w0, lfunc);
+   for (int i = 1; i < tmopi.Size(); i++) { tmopi[i]->DisableLimiting(); }
+}
+
+void TMOPComboIntegrator::SetLimitingNodes(const GridFunction &n0)
+{
+   MFEM_VERIFY(tmopi.Size() > 0, "No TMOP_Integrators were added.");
+
+   tmopi[0]->SetLimitingNodes(n0);
+   for (int i = 1; i < tmopi.Size(); i++) { tmopi[i]->DisableLimiting(); }
+}
+
+double TMOPComboIntegrator::GetElementEnergy(const FiniteElement &el,
+                                             ElementTransformation &T,
+                                             const Vector &elfun)
+{
+   double energy= 0.0;
+   for (int i = 0; i < tmopi.Size(); i++)
+   {
+      energy += tmopi[i]->GetElementEnergy(el, T, elfun);
+   }
+   return energy;
+}
+
+void TMOPComboIntegrator::AssembleElementVector(const FiniteElement &el,
+                                                ElementTransformation &T,
+                                                const Vector &elfun,
+                                                Vector &elvect)
+{
+   MFEM_VERIFY(tmopi.Size() > 0, "No TMOP_Integrators were added.");
+
+   tmopi[0]->AssembleElementVector(el, T, elfun, elvect);
+   for (int i = 1; i < tmopi.Size(); i++)
+   {
+      Vector elvect_i;
+      tmopi[i]->AssembleElementVector(el, T, elfun, elvect_i);
+      elvect += elvect_i;
+   }
+}
+
+void TMOPComboIntegrator::AssembleElementGrad(const FiniteElement &el,
+                                              ElementTransformation &T,
+                                              const Vector &elfun,
+                                              DenseMatrix &elmat)
+{
+   MFEM_VERIFY(tmopi.Size() > 0, "No TMOP_Integrators were added.");
+
+   tmopi[0]->AssembleElementGrad(el, T, elfun, elmat);
+   for (int i = 1; i < tmopi.Size(); i++)
+   {
+      DenseMatrix elmat_i;
+      tmopi[i]->AssembleElementGrad(el, T, elfun, elmat_i);
+      elmat += elmat_i;
+   }
+}
+
+void TMOPComboIntegrator::EnableNormalization(const GridFunction &x)
+{
+   const int cnt = tmopi.Size();
+   double total_integral = 0.0;
+   for (int i = 0; i < cnt; i++)
+   {
+      tmopi[i]->EnableNormalization(x);
+      total_integral += 1.0 / tmopi[i]->metric_normal;
+   }
+   for (int i = 0; i < cnt; i++)
+   {
+      tmopi[i]->metric_normal = 1.0 / total_integral;
+   }
+}
+
+#ifdef MFEM_USE_MPI
+void TMOPComboIntegrator::ParEnableNormalization(const ParGridFunction &x)
+{
+   const int cnt = tmopi.Size();
+   double total_integral = 0.0;
+   for (int i = 0; i < cnt; i++)
+   {
+      tmopi[i]->ParEnableNormalization(x);
+      total_integral += 1.0 / tmopi[i]->metric_normal;
+   }
+   for (int i = 0; i < cnt; i++)
+   {
+      tmopi[i]->metric_normal = 1.0 / total_integral;
+   }
+}
+#endif
+
 
 void InterpolateTMOP_QualityMetric(TMOP_QualityMetric &metric,
                                    const TargetConstructor &tc,

fem/tmop.hpp

@@ -710,7 +710,7 @@ protected:
    // Data is owned, updated by UpdateTargetSpecification.
    Vector tspec;             //eta(x)
    Vector tspec_sav;
-   Vector tspec_perth;       //eta(x+h)
+   Vector tspec_pert1h;      //eta(x+h)
    Vector tspec_pert2h;      //eta(x+2*h)
    Vector tspec_pertmix;     //eta(x+h,y+h)
 
@@ -718,6 +718,10 @@ protected:
    // positions corresponding to the values of tspec.
    const FiniteElementSpace *tspec_fes;
 
+   // These flags can be used by outside functions to avoid recomputing
+   // the tspec and tspec_perth fields again on the same mesh.
+   bool good_tspec, good_tspec_grad, good_tspec_hess;
+
    // Evaluation of the discrete target specification on different meshes.
    // Owned.
    AdaptivityEvaluator *adapt_eval;
@@ -725,7 +729,10 @@ protected:
 public:
    DiscreteAdaptTC(TargetType ttype)
       : TargetConstructor(ttype),
-        tspec(), tspec_fes(NULL), adapt_eval(NULL) { }
+        tspec(), tspec_sav(), tspec_pert1h(), tspec_pert2h(), tspec_pertmix(),
+        tspec_fes(NULL),
+        good_tspec(false), good_tspec_grad(false), good_tspec_hess(false),
+        adapt_eval(NULL) { }
 
    virtual ~DiscreteAdaptTC() { delete adapt_eval; }
 
@@ -734,9 +741,14 @@ public:
    virtual void SetParDiscreteTargetSpec(ParGridFunction &tspec_);
 #endif
 
+   /// Used in combination with the Update methods to avoid extra computations.
+   void ResetUpdateFlags()
+   { good_tspec = good_tspec_grad = good_tspec_hess = false; }
+
    /** Used to update the target specification after the mesh has changed. The
-       new mesh positions are given by new_x. */
-   void UpdateTargetSpecification(const Vector &new_x);
+       new mesh positions are given by new_x. If @a use_flags is true, repeated
+       calls won't do anything until ResetUpdateFlags() is called. */
+   void UpdateTargetSpecification(const Vector &new_x, bool use_flag = false);
 
    void UpdateTargetSpecification(Vector &new_x, Vector &IntData);
 
@@ -747,12 +759,17 @@ public:
    void RestoreTargetSpecificationAtNode(ElementTransformation &T, int nodenum);
 
    /** Used for finite-difference based computations. Computes the target
-       specifications after a mesh perturbation in x or y direction. */
-   void UpdateGradientTargetSpecification(const Vector &x, const double dx);
-
+       specifications after a mesh perturbation in x or y direction.
+       If @a use_flags is true, repeated calls won't do anything until
+       ResetUpdateFlags() is called. */
+   void UpdateGradientTargetSpecification(const Vector &x, double dx,
+                                          bool use_flag = false);
    /** Used for finite-difference based computations. Computes the target
-       specifications after two mesh perturbations in x and/or y direction. */
-   void UpdateHessianTargetSpecification(const Vector &x, const double dx);
+       specifications after two mesh perturbations in x and/or y direction.
+       If @a use_flags is true, repeated calls won't do anything until
+       ResetUpdateFlags() is called. */
+   void UpdateHessianTargetSpecification(const Vector &x, double dx,
+                                         bool use_flag = false);
 
    void SetAdaptivityEvaluator(AdaptivityEvaluator *ae)
    {
@@ -760,7 +777,7 @@ public:
       adapt_eval = ae;
    }
 
-   const Vector &GetTspecPert1H()   { return tspec_perth; }
+   const Vector &GetTspecPert1H()   { return tspec_pert1h; }
    const Vector &GetTspecPert2H()   { return tspec_pert2h; }
    const Vector &GetTspecPertMixH() { return tspec_pertmix; }
 
@@ -776,7 +793,6 @@ public:
 };
 
 class TMOPNewtonSolver;
-class TMOPDescentNewtonSolver;
 
 /** @brief A TMOP integrator class based on any given TMOP_QualityMetric and
     TargetConstructor.
@@ -789,7 +805,8 @@ class TMOP_Integrator : public NonlinearFormIntegrator
 {
 protected:
    friend class TMOPNewtonSolver;
-   friend class TMOPDescentNewtonSolver;
+   friend class TMOPComboIntegrator;
+
    TMOP_QualityMetric *metric;        // not owned
    const TargetConstructor *targetC;  // not owned
 
@@ -865,6 +882,11 @@ protected:
 #endif
    void ComputeMinJac(const Vector &x, const FiniteElementSpace &fes);
 
+   void DisableLimiting()
+   {
+      nodes0 = NULL; coeff0 = NULL; lim_dist = NULL; lim_func = NULL;
+   }
+
 public:
    /** @param[in] m  TMOP_QualityMetric that will be integrated (not owned).
        @param[in] tc Target-matrix construction algorithm to use (not owned). */
@@ -911,8 +933,8 @@ public:
 
    /** @brief Adds a limiting term to the integrator with limiting distance
        function (@a dist in the general version of the method) equal to 1. */
-   void EnableLimiting(const GridFunction &n0,
-                       Coefficient &w0, TMOP_LimiterFunction *lfunc = NULL);
+   void EnableLimiting(const GridFunction &n0, Coefficient &w0,
+                       TMOP_LimiterFunction *lfunc = NULL);
 
    /// Update the original/reference nodes used for limiting.
    void SetLimitingNodes(const GridFunction &n0) { nodes0 = &n0; }
@@ -933,7 +955,7 @@ public:
                                     ElementTransformation &T,
                                     const Vector &elfun, DenseMatrix &elmat);
 
-   DiscreteAdaptTC *GetDiscreteAdaptTC() { return discr_tc; }
+   DiscreteAdaptTC *GetDiscreteAdaptTC() const { return discr_tc; }
 
    /** @brief Computes the normalization factors of the metric and limiting
        integrals using the mesh position given by @a x. */
@@ -953,13 +975,59 @@ public:
    double GetFDh()    const { return dx; }
 };
 
+class TMOPComboIntegrator : public NonlinearFormIntegrator
+{
+protected:
+   // Integrators in the combination. Owned.
+   Array<TMOP_Integrator *> tmopi;
+
+public:
+   TMOPComboIntegrator() : tmopi(0) { }
+
+   ~TMOPComboIntegrator()
+   {
+      for (int i = 0; i < tmopi.Size(); i++) { delete tmopi[i]; }
+   }
+
+   /// Adds a new TMOP_Integrator to the combination.
+   void AddTMOPIntegrator(TMOP_Integrator *ti) { tmopi.Append(ti); }
+
+   Array<TMOP_Integrator *> GetTMOPIntegrators() const { return tmopi; }
+
+   /// Adds the limiting term to the first integrator. Disables it for the rest.
+   void EnableLimiting(const GridFunction &n0, const GridFunction &dist,
+                       Coefficient &w0, TMOP_LimiterFunction *lfunc = NULL);
+
+   /** @brief Adds the limiting term to the first integrator. Disables it for
+       the rest (@a dist in the general version of the method) equal to 1. */
+   void EnableLimiting(const GridFunction &n0, Coefficient &w0,
+                       TMOP_LimiterFunction *lfunc = NULL);
+
+   /// Update the original/reference nodes used for limiting.
+   void SetLimitingNodes(const GridFunction &n0);
+
+   virtual double GetElementEnergy(const FiniteElement &el,
+                                   ElementTransformation &T,
+                                   const Vector &elfun);
+   virtual void AssembleElementVector(const FiniteElement &el,
+                                      ElementTransformation &T,
+                                      const Vector &elfun, Vector &elvect);
+   virtual void AssembleElementGrad(const FiniteElement &el,
+                                    ElementTransformation &T,
+                                    const Vector &elfun, DenseMatrix &elmat);
+
+   /// Normalization factor that considers all integrators in the combination.
+   void EnableNormalization(const GridFunction &x);
+#ifdef MFEM_USE_MPI
+   void ParEnableNormalization(const ParGridFunction &x);
+#endif
+};
 
 /// Interpolates the @a metric's values at the nodes of @a metric_gf.
 /** Assumes that @a metric_gf's FiniteElementSpace is initialized. */
 void InterpolateTMOP_QualityMetric(TMOP_QualityMetric &metric,
                                    const TargetConstructor &tc,
                                    const Mesh &mesh, GridFunction &metric_gf);
-
 }
 
 #endif

fem/tmop_tools.cpp

@@ -61,10 +61,10 @@ void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
    ode_solver.Init(*oper);
 
    // Compute some time step [mesh_size / speed].
-   double min_h = std::numeric_limits<double>::infinity();
+   double h_min = std::numeric_limits<double>::infinity();
    for (int i = 0; i < m->GetNE(); i++)
    {
-      min_h = std::min(min_h, m->GetElementSize(i));
+      h_min = std::min(h_min, m->GetElementSize(i));
    }
    double v_max = 0.0;
    const int s = u.FESpace()->GetVSize() / 2;
@@ -73,26 +73,28 @@ void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
       const double vel = u(i) * u(i) + u(i+s) * u(i+s);
       v_max = std::max(v_max, vel);
    }
-   if (v_max == 0.0)
-   {
-      // No need to change the field.
-      return;
-   }
-   v_max = std::sqrt(v_max);
-   double dt = 0.5 * min_h / v_max;
-   double glob_dt = dt;
 #ifdef MFEM_USE_MPI
    if (pfes)
    {
-      MPI_Allreduce(&dt, &glob_dt, 1, MPI_DOUBLE, MPI_MIN, pfes->GetComm());
+      double v_loc = v_max, h_loc = h_min;
+      MPI_Allreduce(&v_loc, &v_max, 1, MPI_DOUBLE, MPI_MAX, pfes->GetComm());
+      MPI_Allreduce(&h_loc, &h_min, 1, MPI_DOUBLE, MPI_MIN, pfes->GetComm());
    }
 #endif
+   if (v_max == 0.0)
+   {
+      // No mesh motion --> no need to change the field.
+      delete oper;
+      return;
+   }
+   v_max = std::sqrt(v_max);
+   double dt = dt_scale * h_min / v_max;
 
    double t = 0.0;
    bool last_step = false;
    for (int ti = 1; !last_step; ti++)
    {
-      if (t + glob_dt >= 1.0)
+      if (t + dt >= 1.0)
       {
 #ifdef MFEM_DEBUG
          if (myid == 0)
@@ -100,10 +102,10 @@ void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
             mfem::out << "Remap took " << ti << " steps." << std::endl;
          }
 #endif
-         glob_dt = 1.0 - t;
+         dt = 1.0 - t;
          last_step = true;
       }
-      ode_solver.Step(new_field, t, glob_dt);
+      ode_solver.Step(new_field, t, dt);
    }
 
    // Trim the overshoots and undershoots.
@@ -410,28 +412,58 @@ double TMOPNewtonSolver::ComputeScalingFactor(const Vector &x,
 
 void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
 {
+   const NonlinearForm *nlf = dynamic_cast<const NonlinearForm *>(oper);
+   const Array<NonlinearFormIntegrator*> &integs = *nlf->GetDNFI();
+
+   // Reset the update flags of all TargetConstructors.
+   // This is done to avoid repeated updates of shared TargetConstructors.
+   TMOP_Integrator *ti  = NULL;
+   TMOPComboIntegrator *co = NULL;
+   DiscreteAdaptTC *dtc = NULL;
+   for (int i = 0; i < integs.Size(); i++)
+   {
+      ti = dynamic_cast<TMOP_Integrator *>(integs[i]);
+      if (ti)
+      {
+         dtc = ti->GetDiscreteAdaptTC();
+         if (dtc) { dtc->ResetUpdateFlags(); }
+      }
+      co = dynamic_cast<TMOPComboIntegrator *>(integs[i]);
+      if (co)
+      {
+         Array<TMOP_Integrator *> ati = co->GetTMOPIntegrators();
+         for (int j = 0; j < ati.Size(); j++)
+         {
+            dtc = ati[j]->GetDiscreteAdaptTC();
+            if (dtc) { dtc->ResetUpdateFlags(); }
+         }
+      }
+   }
+
    if (parallel)
    {
 #ifdef MFEM_USE_MPI
       const ParNonlinearForm *nlf =
          dynamic_cast<const ParNonlinearForm *>(oper);
-      const Array<NonlinearFormIntegrator*> &integs = *nlf->GetDNFI();
       const ParFiniteElementSpace *pfesc = nlf->ParFESpace();
       Vector x_loc(pfesc->GetVSize());
       pfesc->GetProlongationMatrix()->Mult(x, x_loc);
-      for (int i=0; i<integs.Size(); i++)
+      for (int i = 0; i < integs.Size(); i++)
       {
-         TMOP_Integrator *tmopi = dynamic_cast<TMOP_Integrator *>(integs[i]);
-         DiscreteAdaptTC *discrtc = tmopi->GetDiscreteAdaptTC();
-         tmopi->ComputeFDh(x_loc, *pfesc);
-         if (discrtc)
+         ti = dynamic_cast<TMOP_Integrator *>(integs[i]);
+         if (ti)
          {
-            discrtc->UpdateTargetSpecification(x_loc);
-            double dx = tmopi->GetFDh();
-            if (tmopi->GetFDFlag())
+            ti->ComputeFDh(x_loc, *pfesc);
+            UpdateDiscreteTC(*ti, x_loc);
+         }
+         co = dynamic_cast<TMOPComboIntegrator *>(integs[i]);
+         if (co)
+         {
+            Array<TMOP_Integrator *> ati = co->GetTMOPIntegrators();
+            for (int j = 0; j < ati.Size(); j++)
             {
-               discrtc->UpdateGradientTargetSpecification(x_loc, dx);
-               discrtc->UpdateHessianTargetSpecification(x_loc, dx);
+               ati[j]->ComputeFDh(x_loc, *pfesc);
+               UpdateDiscreteTC(*ati[j], x_loc);
             }
          }
       }
@@ -439,9 +471,6 @@ void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
    }
    else
    {
-      const NonlinearForm *nlf =
-         dynamic_cast<const NonlinearForm *>(oper);
-      const Array<NonlinearFormIntegrator*> &integs = *nlf->GetDNFI();
       const FiniteElementSpace *fesc = nlf->FESpace();
       const Operator *P = nlf->GetProlongation();
       Vector x_loc;
@@ -454,25 +483,45 @@ void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
       {
          x_loc = x;
       }
-      for (int i=0; i<integs.Size(); i++)
+      for (int i = 0; i < integs.Size(); i++)
       {
-         TMOP_Integrator *tmopi = dynamic_cast<TMOP_Integrator *>(integs[i]);
-         DiscreteAdaptTC *discrtc = tmopi->GetDiscreteAdaptTC();
-         tmopi->ComputeFDh(x_loc, *fesc);
-         if (discrtc)
+         ti = dynamic_cast<TMOP_Integrator *>(integs[i]);
+         if (ti)
          {
-            discrtc->UpdateTargetSpecification(x);
-            double dx = tmopi->GetFDh();
-            if (tmopi->GetFDFlag())
+            ti->ComputeFDh(x_loc, *fesc);
+            UpdateDiscreteTC(*ti, x_loc);
+         }
+         co = dynamic_cast<TMOPComboIntegrator *>(integs[i]);
+         if (co)
+         {
+            Array<TMOP_Integrator *> ati = co->GetTMOPIntegrators();
+            for (int j = 0; j < ati.Size(); j++)
             {
-               discrtc->UpdateGradientTargetSpecification(x_loc, dx);
-               discrtc->UpdateHessianTargetSpecification(x_loc, dx);
+               ati[j]->ComputeFDh(x_loc, *fesc);
+               UpdateDiscreteTC(*ati[j], x_loc);
             }
          }
       }
    }
 }
 
+void TMOPNewtonSolver::UpdateDiscreteTC(const TMOP_Integrator &ti,
+                                        const Vector &x_new) const
+{
+   const bool update_flag = true;
+   DiscreteAdaptTC *discrtc = ti.GetDiscreteAdaptTC();
+   if (discrtc)
+   {
+      discrtc->UpdateTargetSpecification(x_new, update_flag);
+      if (ti.GetFDFlag())
+      {
+         double dx = ti.GetFDh();
+         discrtc->UpdateGradientTargetSpecification(x_new, dx, update_flag);
+         discrtc->UpdateHessianTargetSpecification(x_new, dx, update_flag);
+      }
+   }
+}
+
 double TMOPDescentNewtonSolver::ComputeScalingFactor(const Vector &x,
                                                      const Vector &b) const
 {
@@ -572,71 +621,6 @@ double TMOPDescentNewtonSolver::ComputeScalingFactor(const Vector &x,
    return scale;
 }
 
-void TMOPDescentNewtonSolver::ProcessNewState(const Vector &x) const
-{
-   if (parallel)
-   {
-#ifdef MFEM_USE_MPI
-      const ParNonlinearForm *nlf =
-         dynamic_cast<const ParNonlinearForm *>(oper);
-      const Array<NonlinearFormIntegrator*> &integs = *nlf->GetDNFI();
-      const ParFiniteElementSpace *pfesc = nlf->ParFESpace();
-      Vector x_loc(pfesc->GetVSize());
-      pfesc->GetProlongationMatrix()->Mult(x, x_loc);
-      for (int i=0; i<integs.Size(); i++)
-      {
-         TMOP_Integrator *tmopi = dynamic_cast<TMOP_Integrator *>(integs[i]);
-         DiscreteAdaptTC *discrtc = tmopi->GetDiscreteAdaptTC();
-         tmopi->ComputeFDh(x_loc, *pfesc);
-         if (discrtc)
-         {
-            discrtc->UpdateTargetSpecification(x_loc);
-            double dx = tmopi->GetFDh();
-            if (tmopi->GetFDFlag())
-            {
-               discrtc->UpdateGradientTargetSpecification(x_loc, dx);
-               discrtc->UpdateHessianTargetSpecification(x_loc, dx);
-            }
-         }
-      }
-#endif
-   }
-   else
-   {
-      const NonlinearForm *nlf =
-         dynamic_cast<const NonlinearForm *>(oper);
-      const Array<NonlinearFormIntegrator*> &integs = *nlf->GetDNFI();
-      const FiniteElementSpace *fesc = nlf->FESpace();
-      const Operator *P = nlf->GetProlongation();
-      Vector x_loc;
-      if (P)
-      {
-         x_loc.SetSize(P->Height());
-         P->Mult(x,x_loc);
-      }
-      else
-      {
-         x_loc = x;
-      }
-      for (int i=0; i<integs.Size(); i++)
-      {
-         TMOP_Integrator *tmopi = dynamic_cast<TMOP_Integrator *>(integs[i]);
-         DiscreteAdaptTC *discrtc = tmopi->GetDiscreteAdaptTC();
-         tmopi->ComputeFDh(x_loc, *fesc);
-         if (discrtc)
-         {
-            discrtc->UpdateTargetSpecification(x);
-            double dx = tmopi->GetFDh();
-            if (tmopi->GetFDFlag())
-            {
-               discrtc->UpdateGradientTargetSpecification(x_loc, dx);
-               discrtc->UpdateHessianTargetSpecification(x_loc, dx);
-            }
-         }
-      }
-   }
-}
-
 #ifdef MFEM_USE_MPI
 // Metric values are visualized by creating an L2 finite element functions and
 // computing the metric values at the nodes.

fem/tmop_tools.hpp

@@ -28,8 +28,12 @@ private:
    Vector nodes0;
    Vector field0;
 
+   const double dt_scale;
+
 public:
-   AdvectorCG() : AdaptivityEvaluator(), ode_solver(), nodes0(), field0() { }
+   AdvectorCG(double timestep_scale = 0.5)
+      : AdaptivityEvaluator(),
+        ode_solver(), nodes0(), field0(), dt_scale(timestep_scale) { }
 
    virtual void SetInitialField(const Vector &init_nodes,
                                 const Vector &init_field);
@@ -105,12 +109,14 @@ public:
 
 class TMOPNewtonSolver : public NewtonSolver
 {
-private:
+protected:
    bool parallel;
 
    // Quadrature points that are checked for negative Jacobians etc.
    const IntegrationRule &ir;
 
+   void UpdateDiscreteTC(const TMOP_Integrator &ti, const Vector &x_new) const;
+
 public:
 #ifdef MFEM_USE_MPI
    TMOPNewtonSolver(MPI_Comm comm, const IntegrationRule &irule)
@@ -125,25 +131,17 @@ public:
 };
 
 /// Allows negative Jacobians. Used for untangling.
-class TMOPDescentNewtonSolver : public NewtonSolver
+class TMOPDescentNewtonSolver : public TMOPNewtonSolver
 {
-private:
-   bool parallel;
-
-   // Quadrature points that are checked for negative Jacobians etc.
-   const IntegrationRule &ir;
-
 public:
 #ifdef MFEM_USE_MPI
    TMOPDescentNewtonSolver(MPI_Comm comm, const IntegrationRule &irule)
-      : NewtonSolver(comm), parallel(true), ir(irule) { }
+      : TMOPNewtonSolver(comm, irule) { }
 #endif
    TMOPDescentNewtonSolver(const IntegrationRule &irule)
-      : NewtonSolver(), parallel(false), ir(irule) { }
+      : TMOPNewtonSolver(irule) { }
 
    virtual double ComputeScalingFactor(const Vector &x, const Vector &b) const;
-
-   virtual void ProcessNewState(const Vector &x) const;
 };
 
 void vis_tmop_metric_s(int order, TMOP_QualityMetric &qm,

fem/transfer.cpp

@@ -0,0 +1,553 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "transfer.hpp"
+#include "../general/forall.hpp"
+
+namespace mfem
+{
+
+TransferOperator::TransferOperator(const FiniteElementSpace& lFESpace_,
+                                   const FiniteElementSpace& hFESpace_)
+   : Operator(hFESpace_.GetVSize(), lFESpace_.GetVSize())
+{
+   if (lFESpace_.FEColl() == hFESpace_.FEColl())
+   {
+      OperatorPtr P(Operator::ANY_TYPE);
+      hFESpace_.GetTransferOperator(lFESpace_, P);
+      P.SetOperatorOwner(false);
+      opr = P.Ptr();
+   }
+   else if (lFESpace_.GetMesh()->GetNE() > 0
+            && hFESpace_.GetMesh()->GetNE() > 0
+            && dynamic_cast<const TensorBasisElement*>(lFESpace_.GetFE(0))
+            && dynamic_cast<const TensorBasisElement*>(hFESpace_.GetFE(0)))
+   {
+      opr = new TensorProductPRefinementTransferOperator(lFESpace_, hFESpace_);
+   }
+   else
+   {
+      opr = new PRefinementTransferOperator(lFESpace_, hFESpace_);
+   }
+}
+
+TransferOperator::~TransferOperator() { delete opr; }
+
+void TransferOperator::Mult(const Vector& x, Vector& y) const
+{
+   opr->Mult(x, y);
+}
+
+void TransferOperator::MultTranspose(const Vector& x, Vector& y) const
+{
+   opr->MultTranspose(x, y);
+}
+
+PRefinementTransferOperator::PRefinementTransferOperator(
+   const FiniteElementSpace& lFESpace_, const FiniteElementSpace& hFESpace_)
+   : Operator(hFESpace_.GetVSize(), lFESpace_.GetVSize()), lFESpace(lFESpace_),
+     hFESpace(hFESpace_)
+{
+}
+
+PRefinementTransferOperator::~PRefinementTransferOperator() {}
+
+void PRefinementTransferOperator::Mult(const Vector& x, Vector& y) const
+{
+   Mesh* mesh = hFESpace.GetMesh();
+   Array<int> l_dofs, h_dofs, l_vdofs, h_vdofs;
+   DenseMatrix loc_prol;
+   Vector subY, subX;
+
+   Geometry::Type cached_geom = Geometry::INVALID;
+   const FiniteElement* h_fe = NULL;
+   const FiniteElement* l_fe = NULL;
+   IsoparametricTransformation T;
+
+   int vdim = lFESpace.GetVDim();
+
+   for (int i = 0; i < mesh->GetNE(); i++)
+   {
+      hFESpace.GetElementDofs(i, h_dofs);
+      lFESpace.GetElementDofs(i, l_dofs);
+
+      const Geometry::Type geom = mesh->GetElementBaseGeometry(i);
+      if (geom != cached_geom)
+      {
+         h_fe = hFESpace.GetFE(i);
+         l_fe = lFESpace.GetFE(i);
+         T.SetIdentityTransformation(h_fe->GetGeomType());
+         h_fe->GetTransferMatrix(*l_fe, T, loc_prol);
+         subY.SetSize(loc_prol.Height());
+         cached_geom = geom;
+      }
+
+      for (int vd = 0; vd < vdim; vd++)
+      {
+         l_dofs.Copy(l_vdofs);
+         lFESpace.DofsToVDofs(vd, l_vdofs);
+         h_dofs.Copy(h_vdofs);
+         hFESpace.DofsToVDofs(vd, h_vdofs);
+         x.GetSubVector(l_vdofs, subX);
+         loc_prol.Mult(subX, subY);
+         y.SetSubVector(h_vdofs, subY);
+      }
+   }
+}
+
+void PRefinementTransferOperator::MultTranspose(const Vector& x,
+                                                Vector& y) const
+{
+   y = 0.0;
+
+   Mesh* mesh = hFESpace.GetMesh();
+   Array<int> l_dofs, h_dofs, l_vdofs, h_vdofs;
+   DenseMatrix loc_prol;
+   Vector subY, subX;
+
+   Array<char> processed(hFESpace.GetVSize());
+   processed = 0;
+
+   Geometry::Type cached_geom = Geometry::INVALID;
+   const FiniteElement* h_fe = NULL;
+   const FiniteElement* l_fe = NULL;
+   IsoparametricTransformation T;
+
+   int vdim = lFESpace.GetVDim();
+
+   for (int i = 0; i < mesh->GetNE(); i++)
+   {
+      hFESpace.GetElementDofs(i, h_dofs);
+      lFESpace.GetElementDofs(i, l_dofs);
+
+      const Geometry::Type geom = mesh->GetElementBaseGeometry(i);
+      if (geom != cached_geom)
+      {
+         h_fe = hFESpace.GetFE(i);
+         l_fe = lFESpace.GetFE(i);
+         T.SetIdentityTransformation(h_fe->GetGeomType());
+         h_fe->GetTransferMatrix(*l_fe, T, loc_prol);
+         loc_prol.Transpose();
+         subY.SetSize(loc_prol.Height());
+         cached_geom = geom;
+      }
+
+      for (int vd = 0; vd < vdim; vd++)
+      {
+         l_dofs.Copy(l_vdofs);
+         lFESpace.DofsToVDofs(vd, l_vdofs);
+         h_dofs.Copy(h_vdofs);
+         hFESpace.DofsToVDofs(vd, h_vdofs);
+
+         x.GetSubVector(h_vdofs, subX);
+         for (int p = 0; p < h_dofs.Size(); ++p)
+         {
+            if (processed[lFESpace.DecodeDof(h_dofs[p])])
+            {
+               subX[p] = 0.0;
+            }
+         }
+
+         loc_prol.Mult(subX, subY);
+         y.AddElementVector(l_vdofs, subY);
+      }
+
+      for (int p = 0; p < h_dofs.Size(); ++p)
+      {
+         processed[lFESpace.DecodeDof(h_dofs[p])] = 1;
+      }
+   }
+}
+
+TensorProductPRefinementTransferOperator::
+TensorProductPRefinementTransferOperator(
+   const FiniteElementSpace& lFESpace_,
+   const FiniteElementSpace& hFESpace_)
+   : Operator(hFESpace_.GetVSize(), lFESpace_.GetVSize()), lFESpace(lFESpace_),
+     hFESpace(hFESpace_)
+{
+   // Assuming the same element type
+   Mesh* mesh = lFESpace.GetMesh();
+   dim = mesh->Dimension();
+   if (mesh->GetNE() == 0)
+   {
+      return;
+   }
+   const FiniteElement& el = *lFESpace.GetFE(0);
+
+   const TensorBasisElement* ltel =
+      dynamic_cast<const TensorBasisElement*>(&el);
+   MFEM_VERIFY(ltel, "Low order FE space must be tensor product space");
+
+   const TensorBasisElement* htel =
+      dynamic_cast<const TensorBasisElement*>(hFESpace.GetFE(0));
+   MFEM_VERIFY(htel, "High order FE space must be tensor product space");
+   const Array<int>& hdofmap = htel->GetDofMap();
+
+   const IntegrationRule& ir = hFESpace.GetFE(0)->GetNodes();
+   IntegrationRule irLex = ir;
+
+   // The quadrature points, or equivalently, the dofs of the high order space
+   // must be sorted in lexicographical order
+   for (int i = 0; i < ir.GetNPoints(); ++i)
+   {
+      irLex.IntPoint(i) = ir.IntPoint(hdofmap[i]);
+   }
+
+   NE = lFESpace.GetNE();
+   const DofToQuad& maps = el.GetDofToQuad(irLex, DofToQuad::TENSOR);
+
+   D1D = maps.ndof;
+   Q1D = maps.nqpt;
+   B = maps.B;
+   Bt = maps.Bt;
+
+   elem_restrict_lex_l =
+      lFESpace.GetElementRestriction(ElementDofOrdering::LEXICOGRAPHIC);
+
+   MFEM_VERIFY(elem_restrict_lex_l,
+               "Low order ElementRestriction not available");
+
+   elem_restrict_lex_h =
+      hFESpace.GetElementRestriction(ElementDofOrdering::LEXICOGRAPHIC);
+
+   MFEM_VERIFY(elem_restrict_lex_h,
+               "High order ElementRestriction not available");
+
+   localL.SetSize(elem_restrict_lex_l->Height(), Device::GetMemoryType());
+   localH.SetSize(elem_restrict_lex_h->Height(), Device::GetMemoryType());
+   localL.UseDevice(true);
+   localH.UseDevice(true);
+
+   MFEM_VERIFY(dynamic_cast<const ElementRestriction*>(elem_restrict_lex_h),
+               "High order element restriction is of unsupported type");
+
+   mask.SetSize(localH.Size(), Device::GetMemoryType());
+   static_cast<const ElementRestriction*>(elem_restrict_lex_h)
+   ->BooleanMask(mask);
+   mask.UseDevice(true);
+}
+
+namespace TransferKernels
+{
+void Prolongation2D(const int NE, const int D1D, const int Q1D,
+                    const Vector& localL, Vector& localH,
+                    const Array<double>& B, const Vector& mask)
+{
+   auto x_ = Reshape(localL.Read(), D1D, D1D, NE);
+   auto y_ = Reshape(localH.ReadWrite(), Q1D, Q1D, NE);
+   auto B_ = Reshape(B.Read(), Q1D, D1D);
+   auto m_ = Reshape(mask.Read(), Q1D, Q1D, NE);
+
+   localH = 0.0;
+
+   MFEM_FORALL(e, NE,
+   {
+      for (int dy = 0; dy < D1D; ++dy)
+      {
+         double sol_x[MAX_Q1D];
+         for (int qy = 0; qy < Q1D; ++qy)
+         {
+            sol_x[qy] = 0.0;
+         }
+         for (int dx = 0; dx < D1D; ++dx)
+         {
+            const double s = x_(dx, dy, e);
+            for (int qx = 0; qx < Q1D; ++qx)
+            {
+               sol_x[qx] += B_(qx, dx) * s;
+            }
+         }
+         for (int qy = 0; qy < Q1D; ++qy)
+         {
+            const double d2q = B_(qy, dy);
+            for (int qx = 0; qx < Q1D; ++qx)
+            {
+               y_(qx, qy, e) += d2q * sol_x[qx];
+            }
+         }
+      }
+      for (int qy = 0; qy < Q1D; ++qy)
+      {
+         for (int qx = 0; qx < Q1D; ++qx)
+         {
+            y_(qx, qy, e) *= m_(qx, qy, e);
+         }
+      }
+   });
+}
+
+void Prolongation3D(const int NE, const int D1D, const int Q1D,
+                    const Vector& localL, Vector& localH,
+                    const Array<double>& B, const Vector& mask)
+{
+   auto x_ = Reshape(localL.Read(), D1D, D1D, D1D, NE);
+   auto y_ = Reshape(localH.ReadWrite(), Q1D, Q1D, Q1D, NE);
+   auto B_ = Reshape(B.Read(), Q1D, D1D);
+   auto m_ = Reshape(mask.Read(), Q1D, Q1D, Q1D, NE);
+
+   localH = 0.0;
+
+   MFEM_FORALL(e, NE,
+   {
+      for (int dz = 0; dz < D1D; ++dz)
+      {
+         double sol_xy[MAX_Q1D][MAX_Q1D];
+         for (int qy = 0; qy < Q1D; ++qy)
+         {
+            for (int qx = 0; qx < Q1D; ++qx)
+            {
+               sol_xy[qy][qx] = 0.0;
+            }
+         }
+         for (int dy = 0; dy < D1D; ++dy)
+         {
+            double sol_x[MAX_Q1D];
+            for (int qx = 0; qx < Q1D; ++qx)
+            {
+               sol_x[qx] = 0;
+            }
+            for (int dx = 0; dx < D1D; ++dx)
+            {
+               const double s = x_(dx, dy, dz, e);
+               for (int qx = 0; qx < Q1D; ++qx)
+               {
+                  sol_x[qx] += B_(qx, dx) * s;
+               }
+            }
+            for (int qy = 0; qy < Q1D; ++qy)
+            {
+               const double wy = B_(qy, dy);
+               for (int qx = 0; qx < Q1D; ++qx)
+               {
+                  sol_xy[qy][qx] += wy * sol_x[qx];
+               }
+            }
+         }
+         for (int qz = 0; qz < Q1D; ++qz)
+         {
+            const double wz = B_(qz, dz);
+            for (int qy = 0; qy < Q1D; ++qy)
+            {
+               for (int qx = 0; qx < Q1D; ++qx)
+               {
+                  y_(qx, qy, qz, e) += wz * sol_xy[qy][qx];
+               }
+            }
+         }
+      }
+      for (int qz = 0; qz < Q1D; ++qz)
+      {
+         for (int qy = 0; qy < Q1D; ++qy)
+         {
+            for (int qx = 0; qx < Q1D; ++qx)
+            {
+               y_(qx, qy, qz, e) *= m_(qx, qy, qz, e);
+            }
+         }
+      }
+   });
+}
+
+void Restriction2D(const int NE, const int D1D, const int Q1D,
+                   const Vector& localH, Vector& localL,
+                   const Array<double>& Bt, const Vector& mask)
+{
+   auto x_ = Reshape(localH.Read(), Q1D, Q1D, NE);
+   auto y_ = Reshape(localL.ReadWrite(), D1D, D1D, NE);
+   auto Bt_ = Reshape(Bt.Read(), D1D, Q1D);
+   auto m_ = Reshape(mask.Read(), Q1D, Q1D, NE);
+
+   localL = 0.0;
+
+   MFEM_FORALL(e, NE,
+   {
+      for (int qy = 0; qy < Q1D; ++qy)
+      {
+         double sol_x[MAX_D1D];
+         for (int dx = 0; dx < D1D; ++dx)
+         {
+            sol_x[dx] = 0.0;
+         }
+         for (int qx = 0; qx < Q1D; ++qx)
+         {
+            const double s = m_(qx, qy, e) * x_(qx, qy, e);
+            for (int dx = 0; dx < D1D; ++dx)
+            {
+               sol_x[dx] += Bt_(dx, qx) * s;
+            }
+         }
+         for (int dy = 0; dy < D1D; ++dy)
+         {
+            const double q2d = Bt_(dy, qy);
+            for (int dx = 0; dx < D1D; ++dx)
+            {
+               y_(dx, dy, e) += q2d * sol_x[dx];
+            }
+         }
+      }
+   });
+}
+void Restriction3D(const int NE, const int D1D, const int Q1D,
+                   const Vector& localH, Vector& localL,
+                   const Array<double>& Bt, const Vector& mask)
+{
+   auto x_ = Reshape(localH.Read(), Q1D, Q1D, Q1D, NE);
+   auto y_ = Reshape(localL.ReadWrite(), D1D, D1D, D1D, NE);
+   auto Bt_ = Reshape(Bt.Read(), D1D, Q1D);
+   auto m_ = Reshape(mask.Read(), Q1D, Q1D, Q1D, NE);
+
+   localL = 0.0;
+
+   MFEM_FORALL(e, NE,
+   {
+      for (int qz = 0; qz < Q1D; ++qz)
+      {
+         double sol_xy[MAX_D1D][MAX_D1D];
+         for (int dy = 0; dy < D1D; ++dy)
+         {
+            for (int dx = 0; dx < D1D; ++dx)
+            {
+               sol_xy[dy][dx] = 0;
+            }
+         }
+         for (int qy = 0; qy < Q1D; ++qy)
+         {
+            double sol_x[MAX_D1D];
+            for (int dx = 0; dx < D1D; ++dx)
+            {
+               sol_x[dx] = 0;
+            }
+            for (int qx = 0; qx < Q1D; ++qx)
+            {
+               const double s = m_(qx, qy, qz, e) * x_(qx, qy, qz, e);
+               for (int dx = 0; dx < D1D; ++dx)
+               {
+                  sol_x[dx] += Bt_(dx, qx) * s;
+               }
+            }
+            for (int dy = 0; dy < D1D; ++dy)
+            {
+               const double wy = Bt_(dy, qy);
+               for (int dx = 0; dx < D1D; ++dx)
+               {
+                  sol_xy[dy][dx] += wy * sol_x[dx];
+               }
+            }
+         }
+         for (int dz = 0; dz < D1D; ++dz)
+         {
+            const double wz = Bt_(dz, qz);
+            for (int dy = 0; dy < D1D; ++dy)
+            {
+               for (int dx = 0; dx < D1D; ++dx)
+               {
+                  y_(dx, dy, dz, e) += wz * sol_xy[dy][dx];
+               }
+            }
+         }
+      }
+   });
+}
+} // namespace TransferKernels
+
+TensorProductPRefinementTransferOperator::
+~TensorProductPRefinementTransferOperator()
+{
+}
+
+void TensorProductPRefinementTransferOperator::Mult(const Vector& x,
+                                                    Vector& y) const
+{
+   if (lFESpace.GetMesh()->GetNE() == 0)
+   {
+      return;
+   }
+
+   elem_restrict_lex_l->Mult(x, localL);
+   if (dim == 2)
+   {
+      TransferKernels::Prolongation2D(NE, D1D, Q1D, localL, localH, B, mask);
+   }
+   else if (dim == 3)
+   {
+      TransferKernels::Prolongation3D(NE, D1D, Q1D, localL, localH, B, mask);
+   }
+   else
+   {
+      MFEM_ABORT("TensorProductPRefinementTransferOperator::Mult not "
+                 "implemented for dim = "
+                 << dim);
+   }
+   elem_restrict_lex_h->MultTranspose(localH, y);
+}
+
+void TensorProductPRefinementTransferOperator::MultTranspose(const Vector& x,
+                                                             Vector& y) const
+{
+   if (lFESpace.GetMesh()->GetNE() == 0)
+   {
+      return;
+   }
+
+   elem_restrict_lex_h->Mult(x, localH);
+   if (dim == 2)
+   {
+      TransferKernels::Restriction2D(NE, D1D, Q1D, localH, localL, Bt, mask);
+   }
+   else if (dim == 3)
+   {
+      TransferKernels::Restriction3D(NE, D1D, Q1D, localH, localL, Bt, mask);
+   }
+   else
+   {
+      MFEM_ABORT("TensorProductPRefinementTransferOperator::MultTranspose not "
+                 "implemented for dim = "
+                 << dim);
+   }
+   elem_restrict_lex_l->MultTranspose(localL, y);
+}
+
+#ifdef MFEM_USE_MPI
+TrueTransferOperator::TrueTransferOperator(const
+                                           ParFiniteElementSpace& lFESpace_,
+                                           const ParFiniteElementSpace& hFESpace_)
+   : lFESpace(lFESpace_), hFESpace(hFESpace_)
+{
+   localTransferOperator = new TransferOperator(lFESpace_, hFESpace_);
+
+   tmpL.SetSize(lFESpace_.GetVSize());
+   tmpH.SetSize(hFESpace_.GetVSize());
+
+   hFESpace.GetRestrictionMatrix()->BuildTranspose();
+}
+
+TrueTransferOperator::~TrueTransferOperator()
+{
+   delete localTransferOperator;
+}
+
+void TrueTransferOperator::Mult(const Vector& x, Vector& y) const
+{
+   lFESpace.GetProlongationMatrix()->Mult(x, tmpL);
+   localTransferOperator->Mult(tmpL, tmpH);
+   hFESpace.GetRestrictionMatrix()->Mult(tmpH, y);
+}
+
+void TrueTransferOperator::MultTranspose(const Vector& x, Vector& y) const
+{
+   hFESpace.GetRestrictionMatrix()->MultTranspose(x, tmpH);
+   localTransferOperator->MultTranspose(tmpH, tmpL);
+   lFESpace.GetProlongationMatrix()->MultTranspose(tmpL, y);
+}
+#endif
+
+} // namespace mfem

fem/transfer.hpp

@@ -0,0 +1,162 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#ifndef MFEM_TRANSFER_HPP
+#define MFEM_TRANSFER_HPP
+
+#include "../linalg/linalg.hpp"
+#include "fespace.hpp"
+
+#ifdef MFEM_USE_MPI
+#include "pfespace.hpp"
+#endif
+
+namespace mfem
+{
+
+/// Matrix-free transfer operator between finite element spaces
+class TransferOperator : public Operator
+{
+private:
+   Operator* opr;
+
+public:
+   /// Constructs a transfer operator from \p lFESpace to \p hFESpace.
+   /** No matrices are assembled, only the action to a vector is being computed.
+       If both spaces' FE collection pointers are pointing to the same collection
+       we assume that the grid was refined while keeping the order constant. If
+       the FE collections are different, it is assumed that both spaces have are
+       using the same mesh. If the first element of the high-order space is a
+       `TensorBasisElement`, the optimized tensor-product transfers are used. If
+       not, the general transfers used. */
+   TransferOperator(const FiniteElementSpace& lFESpace,
+                    const FiniteElementSpace& hFESpace);
+
+   /// Destructor
+   virtual ~TransferOperator();
+
+   /// @brief Interpolation or prolongation of a vector \p x corresponding to the
+   /// coarse space to the vector \p y corresponding to the fine space.
+   virtual void Mult(const Vector& x, Vector& y) const override;
+
+   /// Restriction by applying the transpose of the Mult method.
+   /** The vector \p x corresponding to the fine space is restricted to the vector
+       \p y corresponding to the coarse space. */
+   virtual void MultTranspose(const Vector& x, Vector& y) const override;
+};
+
+/// Matrix-free transfer operator between finite element spaces on the same mesh
+class PRefinementTransferOperator : public Operator
+{
+private:
+   const FiniteElementSpace& lFESpace;
+   const FiniteElementSpace& hFESpace;
+
+public:
+   /// @brief Constructs a transfer operator from \p lFESpace to \p hFESpace
+   /// which have different FE collections.
+   /** No matrices are assembled, only the action to a vector is being computed.
+       The underlying finite elements need to implement the GetTransferMatrix
+       methods. */
+   PRefinementTransferOperator(const FiniteElementSpace& lFESpace_,
+                               const FiniteElementSpace& hFESpace_);
+
+   /// Destructor
+   virtual ~PRefinementTransferOperator();
+
+   /// @brief Interpolation or prolongation of a vector \p x corresponding to the
+   /// coarse space to the vector \p y corresponding to the fine space.
+   virtual void Mult(const Vector& x, Vector& y) const override;
+
+   /// Restriction by applying the transpose of the Mult method.
+   /** The vector \p x corresponding to the fine space is restricted to the vector
+   \p y corresponding to the coarse space. */
+   virtual void MultTranspose(const Vector& x, Vector& y) const override;
+};
+
+/// @brief Matrix-free transfer operator between finite element spaces on the same
+/// mesh exploiting the tensor product structure of the finite elements
+class TensorProductPRefinementTransferOperator : public Operator
+{
+private:
+   const FiniteElementSpace& lFESpace;
+   const FiniteElementSpace& hFESpace;
+   int dim;
+   int NE;
+   int D1D;
+   int Q1D;
+   Array<double> B;
+   Array<double> Bt;
+   const Operator* elem_restrict_lex_l;
+   const Operator* elem_restrict_lex_h;
+   Vector mask;
+   mutable Vector localL;
+   mutable Vector localH;
+
+public:
+   /// @brief Constructs a transfer operator from \p lFESpace to \p hFESpace which
+   /// have different FE collections.
+   /** No matrices are assembled, only the action to a vector is being computed.
+   The underlying finite elements need to be of the type `TensorBasisElement`. It
+   is also assumed that all the elements in the spaces are of the same type. */
+   TensorProductPRefinementTransferOperator(
+      const FiniteElementSpace& lFESpace_,
+      const FiniteElementSpace& hFESpace_);
+
+   /// Destructor
+   virtual ~TensorProductPRefinementTransferOperator();
+
+   /// @brief Interpolation or prolongation of a vector \p x corresponding to the
+   /// coarse space to the vector \p y corresponding to the fine space.
+   virtual void Mult(const Vector& x, Vector& y) const override;
+
+   /// Restriction by applying the transpose of the Mult method.
+   /** The vector \p x corresponding to the fine space is restricted to the vector
+   \p y corresponding to the coarse space. */
+   virtual void MultTranspose(const Vector& x, Vector& y) const override;
+};
+
+#ifdef MFEM_USE_MPI
+/// @brief Matrix-free transfer operator between finite element spaces working on
+/// true degrees of freedom
+class TrueTransferOperator : public Operator
+{
+private:
+   const ParFiniteElementSpace& lFESpace;
+   const ParFiniteElementSpace& hFESpace;
+   TransferOperator* localTransferOperator;
+   mutable Vector tmpL;
+   mutable Vector tmpH;
+
+public:
+   /// @brief Constructs a transfer operator working on true degrees of freedom from
+   /// from \p lFESpace to \p hFESpace
+   TrueTransferOperator(const ParFiniteElementSpace& lFESpace_,
+                        const ParFiniteElementSpace& hFESpace_);
+
+   /// Destructor
+   ~TrueTransferOperator();
+
+   /// @brief Interpolation or prolongation of a true dof vector \p x to a true dof
+   /// vector \p y.
+   /** The true dof vector \p x corresponding to the coarse space is restricted to
+       the true dof vector \p y corresponding to the fine space. */
+   virtual void Mult(const Vector& x, Vector& y) const override;
+
+   /// Restriction by applying the transpose of the Mult method.
+   /** The true dof vector \p x corresponding to the fine space is restricted to
+       the true dof vector \p y corresponding to the coarse space. */
+   virtual void MultTranspose(const Vector& x, Vector& y) const override;
+};
+#endif
+
+} // namespace mfem
+#endif

general/CMakeLists.txt

@@ -65,6 +65,11 @@ if (MFEM_USE_MPI)
   list(APPEND HDRS communication.hpp)
 endif()
 
+if (MFEM_USE_ADIOS2)
+  list(APPEND SRCS adios2stream.cpp)
+  list(APPEND HDRS adios2stream.hpp)
+endif()
+
 convert_filenames_to_full_paths(SRCS)
 convert_filenames_to_full_paths(HDRS)
 

general/adios2stream.cpp

@@ -0,0 +1,752 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+//
+// Created on: Jan 22, 2019
+// Author: William F Godoy godoywf@ornl.gov
+// adios2: Adaptable Input/Output System https://github.com/ornladios/ADIOS2
+
+#include "adios2stream.hpp"
+
+#include "../fem/geom.hpp"
+#include "../general/array.hpp"
+#include "../mesh/element.hpp"
+#include "../mesh/mesh.hpp"
+#include "../fem/gridfunc.hpp"
+
+#include <algorithm>
+
+namespace mfem
+{
+
+namespace
+{
+// these functions might be included in adios2 upstream next release
+template <class T>
+adios2::Variable<T> SafeDefineVariable(adios2::IO io,
+                                       const std::string& variable_name,
+                                       const adios2::Dims& shape = adios2::Dims(),
+                                       const adios2::Dims& start = adios2::Dims(),
+                                       const adios2::Dims& count = adios2::Dims())
+{
+   adios2::Variable<T> variable = io.InquireVariable<T>(variable_name);
+   if (variable)
+   {
+      if (variable.Count() != count &&
+          variable.ShapeID() == adios2::ShapeID::LocalArray)
+      {
+         variable.SetSelection({start, count});
+      }
+   }
+   else
+   {
+      variable = io.DefineVariable<T>(variable_name, shape, start, count);
+   }
+
+   return variable;
+}
+
+template <class T>
+adios2::Attribute<T> SafeDefineAttribute(adios2::IO io,
+                                         const std::string& attribute_name,
+                                         const T& value,
+                                         const std::string& variable_name = "",
+                                         const std::string separator = "/")
+{
+   adios2::Attribute<T> attribute = io.InquireAttribute<T>(attribute_name);
+   if (attribute)
+   {
+      return attribute;
+   }
+   return io.DefineAttribute<T>(attribute_name, value, variable_name, separator );
+}
+
+template <class T>
+adios2::Attribute<T> SafeDefineAttribute(adios2::IO io,
+                                         const std::string& attribute_name,
+                                         const T* values, const size_t size,
+                                         const std::string& variable_name = "",
+                                         const std::string separator = "/")
+{
+   adios2::Attribute<T> attribute = io.InquireAttribute<T>(attribute_name);
+   if (attribute)
+   {
+      return attribute;
+   }
+   return io.DefineAttribute<T>(attribute_name, values, size, variable_name,
+                                separator );
+}
+
+bool SetBoolParameter(const std::string key,
+                      const std::map<std::string, std::string>& parameters,
+                      const bool default_value) noexcept
+{
+   auto it = parameters.find(key);
+   if (it != parameters.end())
+   {
+      std::string value = it->second;
+      std::transform(value.begin(), value.end(), value.begin(), ::tolower);
+      if (value == "on" || value == "true")
+      {
+         return true;
+      }
+      else if ( value == "off" || value == "false")
+      {
+         return false;
+      }
+   }
+   return default_value;
+}
+
+} //end empty namespace
+
+// PUBLIC
+#ifdef MFEM_USE_MPI
+adios2stream::adios2stream(const std::string& name, const openmode mode,
+                           MPI_Comm comm, const std::string engineType)
+   : name(name),
+     adios2_openmode(mode),
+     adios(new adios2::ADIOS(comm)),
+     io(adios->DeclareIO(name))
+{
+   io.SetEngine(engineType);
+}
+#else
+adios2stream::adios2stream(const std::string& name, const openmode mode,
+                           const std::string engineType)
+   : name(name),
+     adios2_openmode(mode),
+     adios(new adios2::ADIOS()),
+     io(adios->DeclareIO(name))
+{
+   io.SetEngine(engineType);
+}
+#endif
+
+adios2stream::~adios2stream()
+{
+   if (engine)
+   {
+      SafeDefineAttribute<std::string>(io, "vtk.xml", VTKSchema() );
+      engine.Close();
+   }
+}
+
+void adios2stream::SetParameters(
+   const std::map<std::string, std::string>& parameters)
+{
+   io.SetParameters(parameters);
+   refine = SetBoolParameter("RefinedData", parameters, true);
+}
+
+void adios2stream::SetParameter(const std::string key,
+                                const std::string value) noexcept
+{
+   io.SetParameter(key, value);
+   if (key == "RefinedData")
+   {
+      refine = SetBoolParameter("RefinedData", io.Parameters(), true);
+   }
+}
+
+void adios2stream::BeginStep()
+{
+   if (!engine)
+   {
+      engine = io.Open(name, adios2::Mode::Write);
+   }
+   engine.BeginStep();
+   active_step = true;
+}
+
+void adios2stream::EndStep()
+{
+   if (!engine || !active_step)
+   {
+      const std::string message = "MFEM adios2stream error: calling EndStep "
+                                  "on uninitialized step (need BeginStep)";
+      mfem_error(message.c_str());
+   }
+
+   SafeDefineAttribute<std::string>(io, "vtk.xml", VTKSchema() );
+   engine.EndStep();
+   active_step = false;
+}
+
+void adios2stream::SetTime(const double time)
+{
+   adios2::Variable<double> var_time = SafeDefineVariable<double>(io, "TIME");
+   engine.Put(var_time, time);
+   transient = true;
+}
+
+void adios2stream::SetCycle(const int cycle)
+{
+   adios2::Variable<int> var_cycle = SafeDefineVariable<int>(io,"CYCLE");
+   engine.Put(var_cycle, cycle);
+}
+
+void adios2stream::SetRefinementLevel(const int level) noexcept
+{
+   refinement_level = level;
+}
+
+size_t adios2stream::CurrentStep() const
+{
+   return engine.CurrentStep();
+}
+
+void adios2stream::Close()
+{
+   if (engine)
+   {
+      if (!active_step)
+      {
+         SafeDefineAttribute<std::string>(io, "vtk.xml", VTKSchema() );
+      }
+      engine.Close();
+   }
+   if (adios)
+   {
+      adios.reset();
+   }
+}
+
+
+// PROTECTED (accessible by friend class Mesh)
+void adios2stream::Print(const Mesh& mesh, const mode print_mode)
+{
+   auto lf_DefineMeshMetadata = [this](Mesh& mesh)
+   {
+      // check types are constant
+      if (!IsConstantElementType(mesh.elements))
+      {
+         throw std::invalid_argument("MFEM::adios2stream ERROR: non-constant "
+                                     " element types not yet implemented\n");
+      }
+
+      // format info
+      SafeDefineAttribute<std::string>(io, "format", "MFEM ADIOS2 BP v0.1" );
+      SafeDefineAttribute<std::string>(io, "format/version", "0.1" );
+      std::string mesh_type = "Unknown";
+      std::vector<std::string> viz_tools;
+      viz_tools.reserve(2); //for now
+      if (mesh.NURBSext)
+      {
+         mesh_type = "MFEM NURBS";
+         viz_tools.push_back("NONE");
+      }
+      else if (mesh.ncmesh)
+      {
+         mesh_type = "MFEM mesh v1.1";
+         viz_tools.push_back("NONE");
+      }
+      else
+      {
+         mesh_type = "MFEM mesh v1.0";
+         viz_tools.push_back("Paraview: ADIOS2VTXReader");
+         viz_tools.push_back("VTK: vtkADIOS2VTXReader.h");
+      }
+
+      SafeDefineAttribute<std::string>(io, "format/mfem_mesh", mesh_type );
+      SafeDefineAttribute<std::string>(io, "format/viz_tools", viz_tools.data(),
+                                       viz_tools.size() );
+
+      // elements
+      const uint32_t dimension = static_cast<int32_t>(mesh.Dimension());
+      SafeDefineAttribute<uint32_t>(io, "dimension", dimension);
+      SafeDefineVariable<uint32_t>(io,"NumOfElements", {adios2::LocalValueDim});
+      SafeDefineVariable<uint32_t>(io, "types");
+      size_t nelements = 0;
+      size_t element_nvertices = 0;
+      size_t nvertices = 0;
+
+      if (refine)
+      {
+         for (int i = 0; i < mesh.GetNE(); ++i)
+         {
+            const Geometry::Type type = mesh.GetElementBaseGeometry(i);
+            RefinedGeometry* refined_geometry = GlobGeometryRefiner.Refine(type,
+                                                                           refinement_level, 1);
+            if (refined_geometry == nullptr)
+            {
+               mfem_error("ERROR: could not refine geometry in call to Save with adios2stream \n");
+            }
+
+
+            element_nvertices = static_cast<size_t>(Geometries.GetVertices(
+                                                       type)->GetNPoints());
+            nelements += refined_geometry->RefGeoms.Size() / element_nvertices;
+            nvertices  += refined_geometry->RefPts.GetNPoints();
+         }
+
+         refined_mesh_nelements = nelements;
+         refined_mesh_nvertices = nvertices;
+      }
+      else
+      {
+         nelements = static_cast<size_t>(mesh.GetNE());
+         element_nvertices = static_cast<size_t>(mesh.elements[0]->GetNVertices());
+      }
+      SafeDefineVariable<uint64_t>(io, "connectivity", {}, {}, {nelements, element_nvertices+1});
+
+      // vertices
+      SafeDefineVariable<uint32_t>(io,"NumOfVertices", {adios2::LocalValueDim});
+      if (refine)
+      {
+         SafeDefineVariable<double>( io, "vertices", {}, {}, {nvertices, static_cast<size_t>(dimension)});
+      }
+      else
+      {
+         const GridFunction* grid_function = mesh.GetNodes();
+         if (grid_function == nullptr)
+         {
+            const size_t nVertices = static_cast<size_t>(mesh.GetNV());
+            const size_t spaceDim = static_cast<size_t>(mesh.SpaceDimension());
+            // similar to Ordering::byVDIM
+            SafeDefineVariable<double>( io, "vertices", {}, {}, {nVertices, spaceDim});
+         }
+         else
+         {
+            const size_t size = static_cast<size_t>(grid_function->Size());
+            const FiniteElementSpace* fes = grid_function->FESpace();
+            const size_t components = static_cast<size_t>(fes->GetVDim());
+            const size_t tuples = size /components;
+            SafeDefineVariable<double>(io, "vertices", {}, {}, {tuples, components} );
+
+            if (fes->GetOrdering() == Ordering::byNODES)
+            {
+               ordering_by_node = true;
+            }
+         }
+      }
+   };
+
+   auto lf_PrintRefinedMeshData = [this](Mesh& mesh)
+   {
+      // elements and vertices
+      engine.Put("NumOfElements", static_cast<uint32_t>(refined_mesh_nelements));
+      engine.Put("NumOfVertices", static_cast<uint32_t>(refined_mesh_nvertices));
+
+      const uint32_t vtkType =
+         GLVISToVTKType(static_cast<int>(mesh.elements[0]->GetGeometryType()));
+      engine.Put("types", vtkType);
+
+      adios2::Variable<double> var_vertices = io.InquireVariable<double>("vertices");
+      adios2::Variable<double>::Span span_vertices = engine.Put<double>(var_vertices);
+
+      adios2::Variable<uint64_t> var_connectivity =
+         io.InquireVariable<uint64_t>("connectivity");
+      adios2::Variable<uint64_t>::Span span_connectivity = engine.Put<uint64_t>
+                                                           (var_connectivity);
+      size_t span_vertices_offset = 0;
+      size_t span_connectivity_offset = 0;
+      // use for setting absolute node id for each element
+      size_t point_id = 0;
+      DenseMatrix pmatrix;
+
+      for (int e = 0; e < mesh.GetNE(); ++e)
+      {
+         const Geometry::Type type = mesh.GetElementBaseGeometry(e);
+         RefinedGeometry* refined_geometry = GlobGeometryRefiner.Refine(type,
+                                                                        refinement_level, 1);
+         // vertices
+         mesh.GetElementTransformation(e)->Transform(refined_geometry->RefPts, pmatrix);
+         for (int i = 0; i < pmatrix.Width(); ++i)
+         {
+            for (int j = 0; j < pmatrix.Height(); ++j)
+            {
+               span_vertices[span_vertices_offset +  i*pmatrix.Height() + j] = pmatrix(j, i);
+            }
+         }
+         span_vertices_offset += static_cast<size_t>(pmatrix.Width()*pmatrix.Height());
+
+         // connectivity
+         const int nv = Geometries.GetVertices(type)->GetNPoints();
+         const Array<int> &element_vertices = refined_geometry->RefGeoms;
+
+         for (int v = 0; v < element_vertices.Size();)
+         {
+            span_connectivity[span_connectivity_offset] = static_cast<uint64_t>(nv);
+            ++span_connectivity_offset;
+
+            for (int k =0; k < nv; k++, v++ )
+            {
+               span_connectivity[span_connectivity_offset] = static_cast<uint64_t>
+                                                             (point_id + element_vertices[v]);
+               ++span_connectivity_offset;
+            }
+         }
+
+         point_id += static_cast<size_t>(refined_geometry->RefPts.GetNPoints());
+      }
+
+      for (int e = 0; e < mesh.GetNE(); ++e)
+      {
+         const Geometry::Type type = mesh.GetElementBaseGeometry(e);
+         RefinedGeometry* refined_geometry = GlobGeometryRefiner.Refine(type,
+                                                                        refinement_level, 1);
+      }
+   };
+
+   auto lf_PrintMeshData = [&](Mesh& mesh)
+   {
+      if (refine)
+      {
+         lf_PrintRefinedMeshData(mesh);
+         return;
+      }
+
+      // elements
+      engine.Put("NumOfElements", static_cast<uint32_t>(mesh.GetNE()));
+
+      const uint32_t vtkType =
+         GLVISToVTKType(static_cast<int>(mesh.elements[0]->GetGeometryType()));
+      engine.Put("types", vtkType);
+
+      adios2::Variable<uint64_t> varConnectivity =
+         io.InquireVariable<uint64_t>("connectivity");
+      // zero-copy access to adios2 buffer to put non-contiguous to contiguous memory
+      adios2::Variable<uint64_t>::Span spanConnectivity =
+         engine.Put<uint64_t>(varConnectivity);
+
+      size_t elementPosition = 0;
+      for (int e = 0; e < mesh.GetNE(); ++e)
+      {
+         const int nVertices = mesh.elements[e]->GetNVertices();
+         spanConnectivity[elementPosition] = nVertices;
+         for (int v = 0; v < nVertices; ++v)
+         {
+            spanConnectivity[elementPosition + v + 1] =
+               mesh.elements[e]->GetVertices()[v];
+         }
+         elementPosition += nVertices + 1;
+      }
+
+      // vertices
+      engine.Put("NumOfVertices", static_cast<uint32_t>(mesh.GetNV()));
+
+      if (mesh.GetNodes() == nullptr)
+      {
+         adios2::Variable<double> varVertices = io.InquireVariable<double>("vertices");
+         // zero-copy access to adios2 buffer to put non-contiguous to contiguous memory
+         adios2::Variable<double>::Span spanVertices = engine.Put(varVertices);
+
+         for (int v = 0; v < mesh.GetNV(); ++v)
+         {
+            const int space_dim = mesh.SpaceDimension();
+            for (int coord = 0; coord < space_dim; ++coord)
+            {
+               spanVertices[v * space_dim + coord] = mesh.vertices[v](coord);
+            }
+         }
+      }
+      else
+      {
+         const GridFunction* grid_function = mesh.GetNodes();
+         if (ordering_by_node)
+         {
+            adios2::Variable<double> varVertices = io.InquireVariable<double>("vertices");
+            // zero-copy access to adios2 buffer to put non-contiguous to contiguous memory
+            adios2::Variable<double>::Span spanVertices = engine.Put(varVertices);
+
+            const size_t size = static_cast<size_t>(grid_function->Size());
+            const FiniteElementSpace* fes = grid_function->FESpace();
+            const size_t components = static_cast<size_t>(fes->GetVDim());
+            const size_t tuples = size /components;
+
+            const double* data = grid_function->GetData();
+
+            for (size_t i = 0; i < tuples; ++i)
+            {
+               for (size_t j = 0; j < components; ++j)
+               {
+                  spanVertices[i*components + j] = data[j*tuples + i];
+               }
+            }
+         }
+         else
+         {
+            grid_function->Print(*this, "vertices");
+         }
+      }
+   };
+
+   // BODY OF FUNCTION STARTS HERE
+   try
+   {
+      Mesh ref_mesh(mesh);
+      lf_DefineMeshMetadata(ref_mesh);
+
+      if (!engine) // if Engine is closed
+      {
+         engine = io.Open(name, adios2::Mode::Write);
+      }
+
+      lf_PrintMeshData(ref_mesh);
+
+      if (print_mode == mode::sync)
+      {
+         engine.PerformPuts();
+      }
+   }
+   catch (std::exception& e)
+   {
+      const std::string warning =
+         "MFEM: adios2stream exception caught, invalid bp dataset: " + name +
+         "," + e.what();
+      mfem_warning( warning.c_str());
+   }
+}
+
+void adios2stream::Save(const GridFunction& grid_function,
+                        const std::string& variable_name, const data_type type)
+{
+   auto lf_SafeDefine = [&](const std::string& variable_name,
+                            const size_t tuples, const size_t components,
+                            const Ordering::Type ordering,
+                            const std::string& fespace_name)
+   {
+      adios2::Variable<double> var = io.InquireVariable<double>(variable_name);
+      if (!var)
+      {
+         if (components == 1 && type == adios2stream::data_type::point_data)
+         {
+            io.DefineVariable<double>(variable_name, {}, {}, {tuples*components});
+         }
+         else
+         {
+            const adios2::Dims count = (ordering == Ordering::byNODES) ?
+                                       adios2::Dims{components, tuples} :
+                                       adios2::Dims{tuples, components};
+            io.DefineVariable<double>(variable_name, {}, {}, count);
+         }
+         SafeDefineAttribute<std::string>(io, "FiniteElementSpace",
+                                          fespace_name, variable_name);
+
+      }
+   };
+
+   // BODY OF FUNCTION STARTS HERE
+   const std::map<std::string, std::string> parameters = io.Parameters();
+   const bool full_data = SetBoolParameter("FullData", parameters, false);
+
+   if (!full_data && !refine)
+   {
+      return;
+   }
+
+   const FiniteElementSpace* fes = grid_function.FESpace();
+
+   if (refine)
+   {
+      const Mesh *mesh = fes->GetMesh();
+      const size_t components = static_cast<size_t>(grid_function.VectorDim());
+      // const size_t tuples = static_cast<size_t>(mesh->GetNV());
+      const size_t tuples = refined_mesh_nvertices;
+
+      lf_SafeDefine(variable_name, tuples, components,
+                    Ordering::byVDIM, std::string(fes->FEColl()->Name()));
+      if (type == adios2stream::data_type::point_data)
+      {
+         point_data_variables.insert(variable_name);
+      }
+
+      RefinedGeometry* refined_geometry;
+      DenseMatrix transform;
+
+      // zero-copy access to adios2 buffer to put non-contiguous to contiguous memory
+      adios2::Variable<double> variable = io.InquireVariable<double>(variable_name);
+      adios2::Variable<double>::Span span = engine.Put<double>(variable);
+
+      size_t offset = 0;
+      if (components == 1)
+      {
+         Vector scalar;
+
+         const int nelements = mesh->GetNE();
+         for (int e = 0; e < nelements; ++e)
+         {
+            refined_geometry = GlobGeometryRefiner.Refine(
+                                  mesh->GetElementBaseGeometry(e), refinement_level, 1);
+
+            grid_function.GetValues(e, refined_geometry->RefPts, scalar, transform);
+
+            const int size = scalar.Size();
+
+            for (int i = 0; i < size; ++i)
+            {
+               const double value = scalar(i);
+               span.at(offset+i) = value;
+            }
+            offset += static_cast<size_t>(size);
+         }
+      }
+      else
+      {
+         DenseMatrix vector;
+         for (int e = 0; e < mesh->GetNE(); ++e)
+         {
+            refined_geometry = GlobGeometryRefiner.Refine(
+                                  mesh->GetElementBaseGeometry(e), refinement_level, 1);
+            grid_function.GetVectorValues(e, refined_geometry->RefPts, vector, transform);
+
+            for (int i = 0; i < vector.Width(); ++i)
+            {
+               for (int j = 0; j < vector.Height(); ++j)
+               {
+                  span[offset +  i*vector.Height() + j] = vector(j, i);
+               }
+            }
+            offset += static_cast<size_t>(vector.Width()*vector.Height());
+         }
+      }
+   }
+
+   if (full_data)
+   {
+      const size_t size = static_cast<size_t>(grid_function.Size());
+      const size_t components = static_cast<size_t>(fes->GetVDim());
+      const size_t tuples = size /components;
+      lf_SafeDefine(variable_name +"/full", tuples, components,
+                    fes->GetOrdering(),
+                    std::string(fes->FEColl()->Name()) );
+      // calls Vector::Print
+      grid_function.Print(*this, variable_name+"/full");
+
+      if (!refine && type == adios2stream::data_type::point_data)
+      {
+         point_data_variables.insert(variable_name+"/full");
+      }
+   }
+}
+
+// PRIVATE
+int32_t adios2stream::GLVISToVTKType(
+   const int glvisType) const noexcept
+{
+   uint32_t vtkType = 0;
+   switch (glvisType)
+   {
+      case Geometry::Type::POINT:
+         vtkType = 1;
+         break;
+      case Geometry::Type::SEGMENT:
+         vtkType = 3;
+         break;
+      case Geometry::Type::TRIANGLE:
+         vtkType = 5;
+         break;
+      case Geometry::Type::SQUARE:
+         // vtkType = 8;
+         vtkType = 9;
+         break;
+      case Geometry::Type::TETRAHEDRON:
+         vtkType = 10;
+         break;
+      case Geometry::Type::CUBE:
+         // vtkType = 11;
+         vtkType = 12;
+         break;
+      case Geometry::Type::PRISM:
+         vtkType = 13;
+         break;
+      default:
+         vtkType = 0;
+         break;
+   }
+   return vtkType;
+}
+
+bool adios2stream::IsConstantElementType(const Array<Element*>& elements ) const
+noexcept
+{
+   bool isConstType = true;
+   const Geometry::Type type = elements[0]->GetGeometryType();
+
+   for (int e = 1; e < elements.Size(); ++e)
+   {
+      if (type != elements[e]->GetGeometryType())
+      {
+         isConstType = false;
+         break;
+      }
+   }
+   return isConstType;
+}
+
+std::string adios2stream::VTKSchema() const noexcept
+{
+   std::string vtkSchema = R"(
+<?xml version="1.0"?>
+<VTKFile type="UnstructuredGrid" version="0.1" byte_order="LittleEndian">
+  <UnstructuredGrid>
+    <Piece NumberOfPoints="NumOfVertices" NumberOfCells="NumOfElements">
+      <Points>
+        <DataArray Name="vertices" />)";
+
+   vtkSchema += R"(
+      </Points>
+      <Cells>
+        <DataArray Name="connectivity" />
+        <DataArray Name="types" />
+      </Cells>
+      <PointData>)";
+
+   if (point_data_variables.empty())
+   {
+      vtkSchema += "\n";
+   }
+   else
+   {
+      for (const std::string& point_datum : point_data_variables )
+      {
+         vtkSchema += "        <DataArray Name=\"" + point_datum +"\"/>\n";
+      }
+   }
+
+   if (transient)
+   {
+      vtkSchema += "        <DataArray Name=\"TIME\">\n";
+      vtkSchema += "          TIME\n";
+      vtkSchema += "        </DataArray>\n";
+   }
+
+   vtkSchema += R"(
+       </PointData>
+       </Piece>
+     </UnstructuredGrid>
+   </VTKFile>)";
+
+   return vtkSchema;
+}
+
+adios2::Mode adios2stream::ToADIOS2Mode(const adios2stream::openmode mode) const
+noexcept
+{
+   adios2::Mode adios2Mode = adios2::Mode::Undefined;
+   switch (mode)
+   {
+      case adios2stream::openmode::out:
+         adios2Mode = adios2::Mode::Write;
+         break;
+      case adios2stream::openmode::in:
+         adios2Mode = adios2::Mode::Read;
+         break;
+      default:
+         const std::string message = "MFEM adios2stream ERROR: only "
+                                     "openmode::out and openmode::in "
+                                     " are valid, in call to adios2stream constructor";
+         mfem_error(message.c_str());
+   }
+   return adios2Mode;
+}
+
+}  // end namespace mfem

general/adios2stream.hpp

@@ -0,0 +1,233 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+//
+// Created on: Jan 22, 2019
+// Author: William F Godoy godoywf@ornl.gov
+// adios2: Adaptable Input/Output System https://github.com/ornladios/ADIOS2
+
+#ifndef MFEM_ADIOS2STREAM
+#define MFEM_ADIOS2STREAM
+
+#include "../config/config.hpp"
+
+#include <map>
+#include <memory>  // std::unique_ptr
+#include <string>
+#include <set>
+
+#ifdef MFEM_USE_MPI
+#include <mpi.h>
+#endif
+
+#include <adios2.h>
+
+namespace mfem
+{
+
+// forward declaring classes to avoid circular references
+class Vector;
+class GridFunction;
+class Mesh;
+class ADIOS2DataCollection;
+
+template <class T>
+class Array;
+class Element;
+
+class adios2stream
+{
+   friend class Vector;
+   friend class GridFunction;
+   friend class Mesh;
+   friend class ADIOS2DataCollection;
+
+public:
+   /**
+    * Open modes for adios2stream (from std::fstream)
+    * out: write
+    * in:  read
+    * app: append
+    */
+   enum class openmode { out, in, app };
+
+   /** Print and Save modes, deferred is done at Close or EndStep, sync is immediate */
+   enum class mode {sync, deferred};
+
+   enum class data_type {none, point_data, cell_data};
+
+#ifdef MFEM_USE_MPI
+
+   /**
+    * adios2stream MPI constructor, allows for passing parameters in source
+    * code (compile-time) only.
+    * @param name stream name
+    * @param mode adios2stream::openmode::in (Read), adios2stream::openmode::out
+    * (Write)
+    * @param comm MPI communicator establishing domain for fstream
+    * @param engine_type available adios2 engine, default is BPFile
+    *        see https://adios2.readthedocs.io/en/latest/engines/engines.html
+    * @throws std::invalid_argument (user input error) or std::runtime_error
+    *         (system error)
+    */
+   adios2stream(const std::string& name, const openmode mode, MPI_Comm comm,
+                const std::string engine_type = "BPFile");
+#else
+   /**
+    * adios2stream Non-MPI serial constructor, allows for passing parameters in
+    * source code (compile-time) only.
+    * @param name stream name
+    * @param mode adios2stream::openmode::in (Read), adios2stream::openmode::out
+    * (Write)
+    * @param engine_type available adios2 engine, default is BPFile
+    * @throws std::invalid_argument (user input error) or std::runtime_error
+    *         (system error)
+    */
+   adios2stream(const std::string& name, const openmode mode,
+                const std::string engine_type = "BPFile");
+#endif
+
+   /** calls Close if stream is valid basically follows C++ RAII **/
+   virtual ~adios2stream();
+
+   /**
+    * Set parameters for a particular adios2stream Engine
+    * See https://adios2.readthedocs.io/en/latest/engines/engines.html
+    * @param parameters map of key/value string elements
+    */
+   void SetParameters(const std::map<std::string, std::string>& parameters =
+                         std::map<std::string, std::string>());
+
+   /**
+    * Single parameter version of SetParameters passing a key/value pair
+    * See https://adios2.readthedocs.io/en/latest/engines/engines.html
+    * @param key input parameter key
+    * @param value input parameter value
+    */
+   void SetParameter(const std::string key, const std::string value) noexcept;
+
+   /** Begins an I/O step */
+   void BeginStep();
+
+   /** Ends the current step, by default transports the data */
+   void EndStep();
+
+   /**
+    * Associates a physical time with the current I/O step as TIME variable
+    * @param time input physical time
+    */
+   void SetTime(const double time);
+
+   /**
+    * Associates a current time step (cycle) with the current I/O step as CYCLE variable
+    * @param cycle physical time
+    */
+   void SetCycle(const int cycle);
+
+   /**
+    * Input to the Global Geometry Refiner
+    * @param level input level
+    */
+   void SetRefinementLevel(const int level) noexcept;
+
+   /** Return the current step between BeginStep and EndStep */
+   size_t CurrentStep() const;
+
+   /** Finished interaction with adios2stream and flushes the data */
+   void Close();
+
+protected:
+
+   /**
+    * Called from friend class Mesh (which is called from ParMesh)
+    * @param mesh input Mesh object to print
+    * @param print_mode sync: one at a time, deferred: collected (pre-fetch)
+    */
+   void Print(const Mesh& mesh, const adios2stream::mode print_mode = mode::sync);
+
+   void Save(const GridFunction& grid_function, const std::string& variable_name,
+             const data_type type);
+
+private:
+   /** placeholder for engine name */
+   const std::string name;
+
+   /** placeholder for engine openmode */
+   const openmode adios2_openmode;
+
+   /** main adios2 object that owns all the io and engine components */
+   std::unique_ptr<adios2::ADIOS> adios;
+
+   /** io object to set parameters, variables and engines */
+   adios2::IO io;
+
+   /** heavy object doing system-level I/O operations */
+   adios2::Engine engine;
+
+   /** true: transient problem (SetTime is called) */
+   bool transient = false;
+
+   /** true : engine step is active after engine.BeginStep(),
+     *  false: inactive after engine.EndStep() */
+   bool active_step = false;
+
+   /** true: mesh is defined, false: not yet */
+   bool is_mesh_defined = false;
+
+   /** ordering of the nodes to be passed to the schema as an attribute
+    *  true: XXX YYY ZZZ, false: XYZ, XYZ, XYZ
+    *  if true it must swap the vertices to Ordering::byDIM*/
+   bool ordering_by_node = false;
+
+   /** true: refine solution at Save */
+   bool refine = true;
+
+   /** refinement level at Save and Print */
+   int refinement_level = 1;
+
+   /** save for point data */
+   size_t refined_mesh_nvertices = 0;
+
+   /** save for cell data */
+   size_t refined_mesh_nelements = 0;
+
+   /** saves the variable names representing point data */
+   std::set<std::string> point_data_variables;
+
+   /**
+    * Map glvis element types to VTK element types
+    * @param glvisType input
+    * @return VTK element type
+    */
+   int32_t GLVISToVTKType(const int glvisType) const noexcept;
+
+   /** sets the current vtk_schema from point data arrays to be parsed
+    *  in VTK for Paraview visualization */
+   std::string VTKSchema() const noexcept;
+
+   /**
+    * Checks if array of elements contains only constant types
+    * @param elements array input to check
+    * @return true: types are constant, false: mixed types
+    */
+   bool IsConstantElementType(const Array<Element*>& elements ) const noexcept;
+
+   /**
+    * Maps to appropriate adios2::Mode from out, in to write, read
+    * @param mode
+    * @return
+    */
+   adios2::Mode ToADIOS2Mode(const adios2stream::openmode mode) const noexcept;
+
+};
+
+}  // end namespace mfem
+
+#endif /* MFEM_ADIOS2STREAM */

general/globals.hpp

@@ -114,4 +114,16 @@ void SetGlobalMPI_Comm(MPI_Comm comm);
 #define MFEM_THREAD_LOCAL thread_local
 
 
+// MFEM_DEPRECATED macro to mark obsolete functions and methods
+// see https://stackoverflow.com/questions/295120/c-mark-as-deprecated
+#if defined(__GNUC__) || defined(__clang__)
+#define MFEM_DEPRECATED __attribute__((deprecated))
+#elif defined(_MSC_VER)
+#define MFEM_DEPRECATED __declspec(deprecated)
+#else
+#pragma message("WARNING: You need to implement MFEM_DEPRECATED for this compiler")
+#define MFEM_DEPRECATED
+#endif
+
+
 #endif

general/version.cpp

@@ -147,6 +147,9 @@ const char *GetConfigStr()
 #endif
 #ifdef MFEM_USE_OCCA
       "MFEM_USE_OCCA\n"
+#endif
+#ifdef MFEM_USE_ADIOS2
+      "MFEM_USE_ADIOS2\n"
 #endif
       "MFEM_TIMER_TYPE = " EXPAND_AND_QUOTE(MFEM_TIMER_TYPE)
       ;

linalg/hypre.cpp

@@ -1590,6 +1590,8 @@ void HypreParMatrix::Destroy()
    }
 }
 
+#if MFEM_HYPRE_VERSION < 21400
+
 HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
                     double beta,  const HypreParMatrix &B)
 {
@@ -1607,6 +1609,39 @@ HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
    return C;
 }
 
+HypreParMatrix * ParAdd(const HypreParMatrix *A, const HypreParMatrix *B)
+{
+   hypre_ParCSRMatrix * C = internal::hypre_ParCSRMatrixAdd(*A,*B);
+
+   hypre_MatvecCommPkgCreate(C);
+
+   return new HypreParMatrix(C);
+}
+
+#else
+
+HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
+                    double beta,  const HypreParMatrix &B)
+{
+   hypre_ParCSRMatrix *C;
+   hypre_ParcsrAdd(alpha, A, beta, B, &C);
+   hypre_MatvecCommPkgCreate(C);
+
+   return new HypreParMatrix(C);
+}
+
+HypreParMatrix * ParAdd(const HypreParMatrix *A, const HypreParMatrix *B)
+{
+   hypre_ParCSRMatrix *C;
+   hypre_ParcsrAdd(1.0, *A, 1.0, *B, &C);
+
+   hypre_MatvecCommPkgCreate(C);
+
+   return new HypreParMatrix(C);
+}
+
+#endif
+
 HypreParMatrix * ParMult(const HypreParMatrix *A, const HypreParMatrix *B,
                          bool own_matrix)
 {
@@ -1624,15 +1659,6 @@ HypreParMatrix * ParMult(const HypreParMatrix *A, const HypreParMatrix *B,
    return C;
 }
 
-HypreParMatrix * ParAdd(const HypreParMatrix *A, const HypreParMatrix *B)
-{
-   hypre_ParCSRMatrix * C = internal::hypre_ParCSRMatrixAdd(*A,*B);
-
-   hypre_MatvecCommPkgCreate(C);
-
-   return new HypreParMatrix(C);
-}
-
 HypreParMatrix * RAP(const HypreParMatrix *A, const HypreParMatrix *P)
 {
    HYPRE_Int P_owns_its_col_starts =

linalg/operator.cpp

@@ -543,4 +543,59 @@ void RectangularConstrainedOperator::Mult(const Vector &x, Vector &y) const
    }
 }
 
+double PowerMethod::EstimateLargestEigenvalue(Operator& opr, Vector& v0,
+                                              int numSteps, double tolerance, int seed)
+{
+   v1.SetSize(v0.Size());
+   v0.Randomize(seed);
+
+   double eigenvalue = 1.0;
+
+   for (int iter = 0; iter < numSteps; ++iter)
+   {
+      double normV0;
+
+#ifdef MFEM_USE_MPI
+      if (comm != MPI_COMM_NULL)
+      {
+         normV0 = InnerProduct(comm, v0, v0);
+      }
+      else
+      {
+         normV0 = InnerProduct(v0, v0);
+      }
+#else
+      normV0 = InnerProduct(v0, v0);
+#endif
+
+      v0 /= sqrt(normV0);
+      opr.Mult(v0, v1);
+
+      double eigenvalueNew;
+#ifdef MFEM_USE_MPI
+      if (comm != MPI_COMM_NULL)
+      {
+         eigenvalueNew = InnerProduct(comm, v0, v1);
+      }
+      else
+      {
+         eigenvalueNew = InnerProduct(v0, v1);
+      }
+#else
+      eigenvalueNew = InnerProduct(v0, v1);
+#endif
+      double diff = std::abs((eigenvalueNew - eigenvalue) / eigenvalue);
+
+      eigenvalue = eigenvalueNew;
+      std::swap(v0, v1);
+
+      if (diff < tolerance)
+      {
+         break;
+      }
+   }
+
+   return eigenvalue;
+}
+
 }

linalg/operator.hpp

@@ -762,6 +762,39 @@ public:
    virtual ~RectangularConstrainedOperator() { if (own_A) { delete A; } }
 };
 
+/** @brief PowerMethod helper class to estimate the largest eigenvalue of an
+           operator using the iterative power method. */
+class PowerMethod
+{
+   Vector v1;
+#ifdef MFEM_USE_MPI
+   MPI_Comm comm;
+#endif
+
+public:
+
+#ifdef MFEM_USE_MPI
+   PowerMethod() : comm(MPI_COMM_NULL) {}
+#else
+   PowerMethod() {}
+#endif
+
+#ifdef MFEM_USE_MPI
+   PowerMethod(MPI_Comm _comm) : comm(_comm) {}
+#endif
+
+   /// @brief Returns an estimate of the largest eigenvalue of the operator \p opr
+   /// using the iterative power method.
+   /** \p v0 is being used as the vector for the iterative process and will contain
+       the eigenvector corresponding to the largest eigenvalue after convergence.
+       The maximum number of iterations may set with \p numSteps, the relative
+       tolerance with \p tolerance and the seed of the random initialization of
+       \p v0 with \p seed. */
+   double EstimateLargestEigenvalue(Operator& opr, Vector& v0,
+                                    int numSteps = 10, double tolerance = 1e-8,
+                                    int seed = 12345);
+};
+
 }
 
 #endif

linalg/solvers.cpp

@@ -170,6 +170,174 @@ void OperatorJacobiSmoother::Mult(const Vector &x, Vector &y) const
    MFEM_FORALL(i, N, Y[i] += DI[i] * R[i]; );
 }
 
+OperatorChebyshevSmoother::OperatorChebyshevSmoother(Operator* oper_,
+                                                     const Vector &d,
+                                                     const Array<int>& ess_tdofs,
+                                                     int order_, double max_eig_estimate_)
+   :
+   Solver(d.Size()),
+   order(order_),
+   max_eig_estimate(max_eig_estimate_),
+   N(d.Size()),
+   dinv(N),
+   diag(d),
+   coeffs(order),
+   ess_tdof_list(ess_tdofs),
+   residual(N),
+   oper(oper_) { Setup(); }
+
+#ifdef MFEM_USE_MPI
+OperatorChebyshevSmoother::OperatorChebyshevSmoother(Operator* oper_,
+                                                     const Vector &d,
+                                                     const Array<int>& ess_tdofs,
+                                                     int order_, MPI_Comm comm, int power_iterations, double power_tolerance)
+#else
+OperatorChebyshevSmoother::OperatorChebyshevSmoother(Operator* oper_,
+                                                     const Vector &d,
+                                                     const Array<int>& ess_tdofs,
+                                                     int order_, int power_iterations, double power_tolerance)
+#endif
+   : Solver(d.Size()),
+     order(order_),
+     N(d.Size()),
+     dinv(N),
+     diag(d),
+     coeffs(order),
+     ess_tdof_list(ess_tdofs),
+     residual(N),
+     oper(oper_)
+{
+   OperatorJacobiSmoother invDiagOperator(diag, ess_tdofs, 1.0);
+   ProductOperator diagPrecond(&invDiagOperator, oper, false, false);
+
+#ifdef MFEM_USE_MPI
+   PowerMethod powerMethod(comm);
+#else
+   PowerMethod powerMethod;
+#endif
+   Vector ev(oper->Width());
+   max_eig_estimate = powerMethod.EstimateLargestEigenvalue(diagPrecond, ev,
+                                                            power_iterations, power_tolerance);
+
+   Setup();
+}
+
+void OperatorChebyshevSmoother::Setup()
+{
+   // Invert diagonal
+   residual.UseDevice(true);
+   auto D = diag.Read();
+   auto X = dinv.Write();
+   MFEM_FORALL(i, N, X[i] = 1.0 / D[i]; );
+   auto I = ess_tdof_list.Read();
+   MFEM_FORALL(i, ess_tdof_list.Size(), X[I[i]] = 1.0; );
+
+   // Set up Chebyshev coefficients
+   // For reference, see e.g., Parallel multigrid smoothing: polynomial versus
+   // Gauss-Seidel by Adams et al.
+   double upper_bound = 1.2 * max_eig_estimate;
+   double lower_bound = 0.3 * max_eig_estimate;
+   double theta = 0.5 * (upper_bound + lower_bound);
+   double delta = 0.5 * (upper_bound - lower_bound);
+
+   switch (order-1)
+   {
+      case 0:
+      {
+         coeffs[0] = 1.0 / theta;
+         break;
+      }
+      case 1:
+      {
+         double tmp_0 = 1.0/(pow(delta, 2) - 2*pow(theta, 2));
+         coeffs[0] = -4*theta*tmp_0;
+         coeffs[1] = 2*tmp_0;
+         break;
+      }
+      case 2:
+      {
+         double tmp_0 = 3*pow(delta, 2);
+         double tmp_1 = pow(theta, 2);
+         double tmp_2 = 1.0/(-4*pow(theta, 3) + theta*tmp_0);
+         coeffs[0] = tmp_2*(tmp_0 - 12*tmp_1);
+         coeffs[1] = 12/(tmp_0 - 4*tmp_1);
+         coeffs[2] = -4*tmp_2;
+         break;
+      }
+      case 3:
+      {
+         double tmp_0 = pow(delta, 2);
+         double tmp_1 = pow(theta, 2);
+         double tmp_2 = 8*tmp_0;
+         double tmp_3 = 1.0/(pow(delta, 4) + 8*pow(theta, 4) - tmp_1*tmp_2);
+         coeffs[0] = tmp_3*(32*pow(theta, 3) - 16*theta*tmp_0);
+         coeffs[1] = tmp_3*(-48*tmp_1 + tmp_2);
+         coeffs[2] = 32*theta*tmp_3;
+         coeffs[3] = -8*tmp_3;
+         break;
+      }
+      case 4:
+      {
+         double tmp_0 = 5*pow(delta, 4);
+         double tmp_1 = pow(theta, 4);
+         double tmp_2 = pow(theta, 2);
+         double tmp_3 = pow(delta, 2);
+         double tmp_4 = 60*tmp_3;
+         double tmp_5 = 20*tmp_3;
+         double tmp_6 = 1.0/(16*pow(theta, 5) - pow(theta, 3)*tmp_5 + theta*tmp_0);
+         double tmp_7 = 160*tmp_2;
+         double tmp_8 = 1.0/(tmp_0 + 16*tmp_1 - tmp_2*tmp_5);
+         coeffs[0] = tmp_6*(tmp_0 + 80*tmp_1 - tmp_2*tmp_4);
+         coeffs[1] = tmp_8*(tmp_4 - tmp_7);
+         coeffs[2] = tmp_6*(-tmp_5 + tmp_7);
+         coeffs[3] = -80*tmp_8;
+         coeffs[4] = 16*tmp_6;
+         break;
+      }
+      default:
+         MFEM_ABORT("Chebyshev smoother not implemented for order = " << order);
+   }
+}
+
+void OperatorChebyshevSmoother::Mult(const Vector& x, Vector &y) const
+{
+   if (iterative_mode)
+   {
+      MFEM_ABORT("Chebyshev smoother not implemented for iterative mode");
+   }
+
+   if (!oper)
+   {
+      MFEM_ABORT("Chebyshev smoother requires operator");
+   }
+
+   residual = x;
+   helperVector.SetSize(x.Size());
+
+   y.UseDevice(true);
+   y = 0.0;
+
+   for (int k = 0; k < order; ++k)
+   {
+      // Apply
+      if (k > 0)
+      {
+         oper->Mult(residual, helperVector);
+         residual = helperVector;
+      }
+
+      // Scale residual by inverse diagonal
+      const int n = N;
+      auto Dinv = dinv.Read();
+      auto R = residual.ReadWrite();
+      MFEM_FORALL(i, n, R[i] *= Dinv[i]; );
+
+      // Add weighted contribution to y
+      auto Y = y.ReadWrite();
+      auto C = coeffs.Read();
+      MFEM_FORALL(i, n, Y[i] += C[k] * R[i]; );
+   }
+}
 
 void SLISolver::UpdateVectors()
 {
@@ -382,13 +550,24 @@ void CGSolver::Mult(const Vector &b, Vector &x) const
    }
    nom0 = nom = Dot(d, r);
    MFEM_ASSERT(IsFinite(nom), "nom = " << nom);
-
    if (print_level == 1 || print_level == 3)
    {
       mfem::out << "   Iteration : " << setw(3) << 0 << "  (B r, r) = "
                 << nom << (print_level == 3 ? " ...\n" : "\n");
    }
 
+   if (nom < 0.0)
+   {
+      if (print_level >= 0)
+      {
+         mfem::out << "PCG: The preconditioner is not positive definite. (Br, r) = "
+                   << nom << '\n';
+      }
+      converged = 0;
+      final_iter = 0;
+      final_norm = nom;
+      return;
+   }
    r0 = std::max(nom*rel_tol*rel_tol, abs_tol*abs_tol);
    if (nom <= r0)
    {
@@ -436,6 +615,17 @@ void CGSolver::Mult(const Vector &b, Vector &x) const
          betanom = Dot(r, r);
       }
       MFEM_ASSERT(IsFinite(betanom), "betanom = " << betanom);
+      if (betanom < 0.0)
+      {
+         if (print_level >= 0)
+         {
+            mfem::out << "PCG: The preconditioner is not positive definite. (Br, r) = "
+                      << betanom << '\n';
+         }
+         converged = 0;
+         final_iter = i;
+         break;
+      }
 
       if (print_level == 1)
       {

linalg/solvers.hpp

@@ -115,6 +115,65 @@ private:
    const Operator *oper;
 };
 
+/// Chebyshev accelerated smoothing with given vector, no matrix necessary
+/** Potentially useful with tensorized operators, for example. This is just a
+    very basic Chebyshev iteration, if you want tolerances, iteration control,
+    etc. wrap this with SLISolver. */
+class OperatorChebyshevSmoother : public Solver
+{
+public:
+   /** Application is by *inverse* of the given vector. It is assumed the
+       underlying operator acts as the identity on entries in ess_tdof_list,
+       corresponding to (assembled) DIAG_ONE policy or ConstrainedOperator in
+       the matrix-free setting. The estimated largest eigenvalue of the
+       diagonally preconditoned operator must be provided via
+       max_eig_estimate. */
+   OperatorChebyshevSmoother(Operator* oper_, const Vector &d,
+                             const Array<int>& ess_tdof_list,
+                             int order, double max_eig_estimate);
+
+   /** Application is by *inverse* of the given vector. It is assumed the
+       underlying operator acts as the identity on entries in ess_tdof_list,
+       corresponding to (assembled) DIAG_ONE policy or ConstrainedOperator in
+       the matrix-free setting. The largest eigenvalue of the diagonally
+       preconditoned operator is estimated internally via a power method. The
+       accuracy of the estimated eigenvalue may be controlled via
+       power_iterations and power_tolerance. */
+#ifdef MFEM_USE_MPI
+   OperatorChebyshevSmoother(Operator* oper_, const Vector &d,
+                             const Array<int>& ess_tdof_list,
+                             int order, MPI_Comm comm = MPI_COMM_NULL, int power_iterations = 10,
+                             double power_tolerance = 1e-8);
+#else
+   OperatorChebyshevSmoother(Operator* oper_, const Vector &d,
+                             const Array<int>& ess_tdof_list,
+                             int order, int power_iterations = 10, double power_tolerance = 1e-8);
+#endif
+
+   ~OperatorChebyshevSmoother() {}
+
+   void Mult(const Vector&x, Vector &y) const;
+
+   void SetOperator(const Operator &op_)
+   {
+      oper = &op_;
+   }
+
+   void Setup();
+
+private:
+   const int order;
+   double max_eig_estimate;
+   const int N;
+   Vector dinv;
+   const Vector &diag;
+   Array<double> coeffs;
+   const Array<int>& ess_tdof_list;
+   mutable Vector residual;
+   mutable Vector helperVector;
+   const Operator* oper;
+};
+
 
 /// Stationary linear iteration: x <- x + B (b - A x)
 class SLISolver : public IterativeSolver

linalg/vector.cpp

@@ -670,6 +670,16 @@ void Vector::Print(std::ostream &out, int width) const
    out << '\n';
 }
 
+#ifdef MFEM_USE_ADIOS2
+void Vector::Print(adios2stream &out,
+                   const std::string& variable_name) const
+{
+   if (!size) { return; }
+   data.Read(MemoryClass::HOST, size);
+   out.engine.Put(variable_name, &data[0] );
+}
+#endif
+
 void Vector::Print_HYPRE(std::ostream &out) const
 {
    int i;

linalg/vector.hpp

@@ -13,6 +13,9 @@
 #define MFEM_VECTOR
 
 #include "../general/array.hpp"
+#ifdef MFEM_USE_ADIOS2
+#include "../general/adios2stream.hpp"
+#endif
 #include "../general/globals.hpp"
 #include "../general/mem_manager.hpp"
 #include "../general/device.hpp"
@@ -298,6 +301,13 @@ public:
    /// Prints vector to stream out.
    void Print(std::ostream &out = mfem::out, int width = 8) const;
 
+#ifdef MFEM_USE_ADIOS2
+   /// Prints vector to stream out.
+   /// @param out adios2stream output
+   /// @param variable_name variable name associated with current Vector
+   void Print(adios2stream & out, const std::string& variable_name) const;
+#endif
+
    /// Prints vector to stream out in HYPRE_Vector format.
    void Print_HYPRE(std::ostream &out) const;
 

makefile

@@ -117,7 +117,7 @@ EXAMPLE_SUBDIRS = sundials petsc pumi hiop ginkgo
 EXAMPLE_DIRS := examples $(addprefix examples/,$(EXAMPLE_SUBDIRS))
 EXAMPLE_TEST_DIRS := examples
 
-MINIAPP_SUBDIRS = common electromagnetics meshing performance tools toys nurbs gslib
+MINIAPP_SUBDIRS = common electromagnetics meshing navier performance tools toys nurbs gslib
 MINIAPP_DIRS := $(addprefix miniapps/,$(MINIAPP_SUBDIRS))
 MINIAPP_TEST_DIRS := $(filter-out %/common,$(MINIAPP_DIRS))
 MINIAPP_USE_COMMON := $(addprefix miniapps/,electromagnetics tools toys)
@@ -392,6 +392,9 @@ endif
 # Source dirs in logical order
 DIRS = general linalg mesh fem fem/libceed
 SOURCE_FILES = $(foreach dir,$(DIRS),$(wildcard $(SRC)$(dir)/*.cpp))
+ADIOS2_FILES = $(SRC)general/adios2stream.h $(SRC)general/adios2stream.cpp \
+ $(SRC)fem/adios2datacollection.hpp $(SRC)fem/adios2datacollection.cpp
+SOURCE_FILES := $(filter-out $(ADIOS2_FILES),$(SOURCE_FILES))
 RELSRC_FILES = $(patsubst $(SRC)%,%,$(SOURCE_FILES))
 OBJECT_FILES = $(patsubst $(SRC)%,$(BLD)%,$(SOURCE_FILES:.cpp=.o))
 OKL_DIRS = fem

mesh/element.hpp

@@ -75,7 +75,7 @@ public:
    virtual const int *GetEdgeVertices(int) const = 0;
 
    /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   virtual int GetNFaces(int &nFaceVertices) const = 0;
+   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const = 0;
 
    virtual int GetNFaces() const = 0;
 

mesh/hexahedron.hpp

@@ -52,7 +52,7 @@ public:
    { return geom_t::Edges[ei]; }
 
    /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   virtual int GetNFaces(int &nFaceVertices) const
+   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
    { nFaceVertices = 4; return 6; }
 
    virtual int GetNFaces() const { return 6; }

mesh/mesh.cpp

@@ -360,7 +360,6 @@ void Mesh::GetElementTransformation(int i, IsoparametricTransformation *ElTr)
       }
       ElTr->SetFE(Nodes->FESpace()->GetFE(i));
    }
-   ElTr->FinalizeTransformation();
 }
 
 void Mesh::GetElementTransformation(int i, const Vector &nodes,
@@ -402,7 +401,6 @@ void Mesh::GetElementTransformation(int i, const Vector &nodes,
       }
       ElTr->SetFE(Nodes->FESpace()->GetFE(i));
    }
-   ElTr->FinalizeTransformation();
 }
 
 ElementTransformation *Mesh::GetElementTransformation(int i)
@@ -470,7 +468,6 @@ void Mesh::GetBdrElementTransformation(int i, IsoparametricTransformation* ElTr)
          ElTr->SetFE(face_el);
       }
    }
-   ElTr->FinalizeTransformation();
 }
 
 void Mesh::GetFaceTransformation(int FaceNo, IsoparametricTransformation *FTr)
@@ -535,7 +532,6 @@ void Mesh::GetFaceTransformation(int FaceNo, IsoparametricTransformation *FTr)
          FTr->SetFE(face_el);
       }
    }
-   FTr->FinalizeTransformation();
 }
 
 ElementTransformation *Mesh::GetFaceTransformation(int FaceNo)
@@ -597,7 +593,6 @@ void Mesh::GetEdgeTransformation(int EdgeNo, IsoparametricTransformation *EdTr)
          MFEM_ABORT("Not implemented.");
       }
    }
-   EdTr->FinalizeTransformation();
 }
 
 ElementTransformation *Mesh::GetEdgeTransformation(int EdgeNo)
@@ -619,7 +614,6 @@ void Mesh::GetLocalPtToSegTransformation(
    locpm(0, 0) = SegVert->IntPoint(i/64).x;
    //  (i/64) is the local face no. in the segment
    //  (i%64) is the orientation of the point (not used)
-   Transf.FinalizeTransformation();
 }
 
 void Mesh::GetLocalSegToTriTransformation(
@@ -639,7 +633,6 @@ void Mesh::GetLocalSegToTriTransformation(
       locpm(0, so[j]) = TriVert->IntPoint(tv[j]).x;
       locpm(1, so[j]) = TriVert->IntPoint(tv[j]).y;
    }
-   Transf.FinalizeTransformation();
 }
 
 void Mesh::GetLocalSegToQuadTransformation(
@@ -659,7 +652,6 @@ void Mesh::GetLocalSegToQuadTransformation(
       locpm(0, so[j]) = QuadVert->IntPoint(qv[j]).x;
       locpm(1, so[j]) = QuadVert->IntPoint(qv[j]).y;
    }
-   Transf.FinalizeTransformation();
 }
 
 void Mesh::GetLocalTriToTetTransformation(
@@ -683,7 +675,6 @@ void Mesh::GetLocalTriToTetTransformation(
       locpm(1, j) = vert.y;
       locpm(2, j) = vert.z;
    }
-   Transf.FinalizeTransformation();
 }
 
 void Mesh::GetLocalTriToWdgTransformation(
@@ -709,7 +700,6 @@ void Mesh::GetLocalTriToWdgTransformation(
       locpm(1, j) = vert.y;
       locpm(2, j) = vert.z;
    }
-   Transf.FinalizeTransformation();
 }
 
 void Mesh::GetLocalQuadToHexTransformation(
@@ -731,7 +721,6 @@ void Mesh::GetLocalQuadToHexTransformation(
       locpm(1, j) = vert.y;
       locpm(2, j) = vert.z;
    }
-   Transf.FinalizeTransformation();
 }
 
 void Mesh::GetLocalQuadToWdgTransformation(
@@ -755,7 +744,6 @@ void Mesh::GetLocalQuadToWdgTransformation(
       locpm(1, j) = vert.y;
       locpm(2, j) = vert.z;
    }
-   Transf.FinalizeTransformation();
 }
 
 const GeometricFactors* Mesh::GetGeometricFactors(const IntegrationRule& ir,
@@ -939,8 +927,7 @@ void Mesh::ApplyLocalSlaveTransformation(IsoparametricTransformation &transf,
 #endif
    MFEM_ASSERT(fi.NCFace >= 0, "");
    transf.Transform(*nc_faces_info[fi.NCFace].PointMatrix, composition);
-   transf.GetPointMat() = composition;
-   transf.FinalizeTransformation();
+   transf.SetPointMat(composition);
 }
 
 FaceElementTransformations *Mesh::GetBdrFaceTransformations(int BdrElemNo)
@@ -4846,10 +4833,12 @@ void Mesh::GetPointMatrix(int i, DenseMatrix &pointmat) const
 
    pointmat.SetSize(spaceDim, nv);
    for (k = 0; k < spaceDim; k++)
+   {
       for (j = 0; j < nv; j++)
       {
          pointmat(k, j) = vertices[v[j]](k);
       }
+   }
 }
 
 void Mesh::GetBdrPointMatrix(int i,DenseMatrix &pointmat) const
@@ -8616,6 +8605,13 @@ void Mesh::PrintTopo(std::ostream &out,const Array<int> &e_to_k) const
    out << "\nvertices\n" << NumOfVertices << '\n';
 }
 
+#ifdef MFEM_USE_ADIOS2
+void Mesh::Print(adios2stream &out) const
+{
+   out.Print(*this);
+}
+#endif
+
 void Mesh::PrintVTK(std::ostream &out)
 {
    out <<
@@ -10428,6 +10424,7 @@ GeometricFactors::GeometricFactors(const Mesh *mesh, const IntegrationRule &ir,
    const GridFunction *nodes = mesh->GetNodes();
    const FiniteElementSpace *fespace = nodes->FESpace();
    const FiniteElement *fe = fespace->GetFE(0);
+   const int dim  = fe->GetDim();
    const int vdim = fespace->GetVDim();
    const int NE   = fespace->GetNE();
    const int ND   = fe->GetDof();
@@ -10445,7 +10442,7 @@ GeometricFactors::GeometricFactors(const Mesh *mesh, const IntegrationRule &ir,
    }
    if (flags & GeometricFactors::JACOBIANS)
    {
-      J.SetSize(vdim*vdim*NQ*NE);
+      J.SetSize(dim*vdim*NQ*NE);
       eval_flags |= QuadratureInterpolator::DERIVATIVES;
    }
    if (flags & GeometricFactors::DETERMINANTS)

mesh/mesh.hpp

@@ -23,6 +23,9 @@
 #include "../fem/eltrans.hpp"
 #include "../fem/coefficient.hpp"
 #include "../general/zstr.hpp"
+#ifdef MFEM_USE_ADIOS2
+#include "../general/adios2stream.hpp"
+#endif
 #include <iostream>
 
 namespace mfem
@@ -55,6 +58,10 @@ class Mesh
    friend class NCMesh;
    friend class NURBSExtension;
 
+#ifdef MFEM_USE_ADIOS2
+   friend class adios2stream;
+#endif
+
 protected:
    int Dim;
    int spaceDim;
@@ -1195,6 +1202,10 @@ public:
    /// \see mfem::ofgzstream() for on-the-fly compression of ascii outputs
    virtual void Print(std::ostream &out = mfem::out) const { Printer(out); }
 
+   /// Print the mesh to the given stream using the adios2 bp format
+#ifdef MFEM_USE_ADIOS2
+   virtual void Print(adios2stream &out) const;
+#endif
    /// Print the mesh in VTK format (linear and quadratic meshes only).
    /// \see mfem::ofgzstream() for on-the-fly compression of ascii outputs
    void PrintVTK(std::ostream &out);

mesh/pmesh.cpp

@@ -1695,7 +1695,6 @@ void ParMesh::GetFaceNbrElementTransformation(
          MFEM_ABORT("Nodes are not ParGridFunction!");
       }
    }
-   ElTr->FinalizeTransformation();
 }
 
 void ParMesh::DeleteFaceNbrData()
@@ -2355,7 +2354,6 @@ ElementTransformation* ParMesh::GetGhostFaceTransformation(
 #endif
       FaceTransformation.SetFE(face_el);
    }
-   FaceTransformation.FinalizeTransformation();
    return &FaceTransformation;
 }
 
@@ -4223,6 +4221,13 @@ void ParMesh::Print(std::ostream &out) const
    }
 }
 
+#ifdef MFEM_USE_ADIOS2
+void ParMesh::Print(adios2stream &out) const
+{
+   Mesh::Print(out);
+}
+#endif
+
 static void dump_element(const Element* elem, Array<int> &data)
 {
    data.Append(elem->GetGeometryType());

mesh/pmesh.hpp

@@ -309,6 +309,12 @@ public:
        as boundary (for visualization purposes) using the mfem v1.0 format. */
    virtual void Print(std::ostream &out = mfem::out) const;
 
+#ifdef MFEM_USE_ADIOS2
+   /** Print the part of the mesh in the calling processor using adios2 bp
+       format. */
+   virtual void Print(adios2stream &out) const;
+#endif
+
    /** Print the part of the mesh in the calling processor adding the interface
        as boundary (for visualization purposes) using Netgen/Truegrid format .*/
    virtual void PrintXG(std::ostream &out = mfem::out) const;
@@ -348,6 +354,10 @@ public:
 #ifdef MFEM_USE_PUMI
    friend class ParPumiMesh;
 #endif
+
+#ifdef MFEM_USE_ADIOS2
+   friend class adios2stream;
+#endif
 };
 
 }

mesh/point.hpp

@@ -47,7 +47,7 @@ public:
    virtual const int *GetEdgeVertices(int ei) const { return NULL; }
 
    /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   virtual int GetNFaces(int &nFaceVertices) const
+   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
    { nFaceVertices = 0; return 0; }
 
    virtual int GetNFaces() const { return 0; }

mesh/quadrilateral.hpp

@@ -54,7 +54,7 @@ public:
    { return geom_t::Edges[ei]; }
 
    /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   virtual int GetNFaces(int &nFaceVertices) const
+   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
    { nFaceVertices = 0; return 0; }
 
    virtual int GetNFaces() const { return 0; }

mesh/segment.hpp

@@ -53,7 +53,7 @@ public:
    virtual const int *GetEdgeVertices(int ei) const { return NULL; }
 
    /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   virtual int GetNFaces(int &nFaceVertices) const
+   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
    { nFaceVertices = 0; return 0; }
 
    virtual int GetNFaces() const { return 0; }

mesh/tetrahedron.hpp

@@ -101,7 +101,7 @@ public:
    { return geom_t::Edges[ei]; }
 
    /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   virtual int GetNFaces(int &nFaceVertices) const
+   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
    { nFaceVertices = 3; return 4; }
 
    virtual int GetNFaces() const { return 4; }

mesh/triangle.hpp

@@ -81,7 +81,7 @@ public:
    { return geom_t::Edges[ei]; }
 
    /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   virtual int GetNFaces(int &nFaceVertices) const
+   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
    { nFaceVertices = 0; return 0; }
 
    virtual int GetNFaces() const { return 0; }

mesh/vertex.hpp

@@ -13,6 +13,7 @@
 #define MFEM_VERTEX
 
 #include "../config/config.hpp"
+#include "../general/globals.hpp"
 
 namespace mfem
 {
@@ -44,7 +45,7 @@ public:
 
    /// (DEPRECATED) Set the coordinates of the Vertex.
    /** @deprecated This old version of SetCoords is not always memory safe. */
-   void SetCoords(const double *p)
+   MFEM_DEPRECATED void SetCoords(const double *p)
    { coord[0] = p[0]; coord[1] = p[1]; coord[2] = p[2]; }
 
    /// Sets vertex location based on given point p

mesh/wedge.hpp

@@ -55,7 +55,7 @@ public:
    { return geom_t::Edges[ei]; }
 
    /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   virtual int GetNFaces(int &nFaceVertices) const;
+   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const;
 
    virtual int GetNFaces() const { return 5; }
 

mfem.hpp

@@ -24,6 +24,9 @@
 #include "general/stable3d.hpp"
 #include "general/table.hpp"
 #include "general/tic_toc.hpp"
+#ifdef MFEM_USE_ADIOS2
+#include "general/adios2stream.hpp"
+#endif
 #include "general/isockstream.hpp"
 #include "general/osockstream.hpp"
 #include "general/socketstream.hpp"