Extended PA for some vector mass and curl integrators to handle diagonal matrix coefficients. Added unit tests.

Implemented PA for MixedVectorCurlIntegrator and MixedVectorWeakCurlIntegrator. Added unit tests for these integrators.
2020-03-31 11:56:02 -07:00 · 2020-03-30 18:02:46 -07:00
127 changed files with 1608 additions and 15227 deletions
@@ -72,10 +72,6 @@ examples/deformed.*
 examples/velocity.*
 examples/elastic_energy.*
 examples/mode_*
-examples/ex5-p-*.bp
-examples/ex9-p-*.bp
-examples/ex12-p-*.bp
-examples/ex16-p-*.bp
 examples/ex16.mesh
 examples/ex16-mesh.*
 examples/ex16-init.*
@@ -169,8 +165,6 @@ miniapps/meshing/shaper
 miniapps/meshing/extruder
 miniapps/meshing/mesh-optimizer
 miniapps/meshing/pmesh-optimizer
-miniapps/meshing/minimal-surface
-miniapps/meshing/pminimal-surface

 miniapps/meshing/mobius-strip.mesh
 miniapps/meshing/klein-bottle.mesh
@@ -231,14 +225,6 @@ miniapps/gslib/field-diff
 miniapps/gslib/findpts
 miniapps/gslib/pfindpts

-miniapps/navier/navier_mms
-miniapps/navier/navier_kovasznay
-miniapps/navier/navier_tgv
-miniapps/navier/navier_shear
-miniapps/navier/navier_3dfoc
-miniapps/navier/tgv_out*.txt
-miniapps/navier/*_output
-
 # Unit test binary and outputs
 tests/unit/output_meshes
 tests/unit/unit_tests
@@ -23,47 +23,19 @@ Meshing improvements
  Hessian for r-adaptivity using discrete fields, and allows use of skewness
  and orientation based metrics.

-Improved GPU capabilities
-------------------------
- Added support for Chebyshev accelerated polynomial smoother on GPU.
-
-Discretization improvements
---------------------------
- Added support for matrix-free interpolation and restriction operators between
-  continuous H1 finite element spaces of different order on the same mesh or
-  with the same order on uniformly refined meshes.
-
- Added support for simplices in GSLIB-FindPoints.
-
-Linear and nonlinear solvers
----------------------------
- Added power method to iteratively estimate the largest eigenvalue and the
-  corresponding eigenvector of an operator.
-
- Added initial support for h- and p-multigrid solvers and preconditioners for
-  matrix-based and matrix-free discretizations with basic GPU capability.
-
 New and updated examples and miniapps
 -------------------------------------
+- Adding a simple meshing miniapp, Twist, which demonstrates MFEM's strategy of
+  stitching together opposite surfaces of a mesh to create a topologically
+  periodic mesh.
+
 - Added a new example, Example 25/25p, to demonstrate the use of a Perfectly
  Matched Layer (PML) for the simulation of electromagnetic wave propagation.
  The example defines and solves several indefinite Maxwell problems.

- Added a new Example 26/26p to demonstrate the construction of a matrix-free
-  geometric and p-multigrid preconditioner for the Laplace problem.
-
- Added a new example, Example 27/27p, to demonstrate the enforcement of
-  various boundary conditions with the Laplace operator.  The example shows the
-  procedures for applying Dirichlet, Neumann (both homogeneous and
-  inhomogeneous), Robin, and periodic boundary conditions with either H1 or DG
-  discretizations.
-
- Added a simple meshing miniapp, Twist, which demonstrates MFEM's strategy of
-  stitching together opposite surfaces of a mesh to create a topologically
-  periodic mesh.
-
- Added a new meshing miniapp, Minimal Surface, which solves Plateau's problem:
-  the Dirichlet problem for the minimal surface equation.
+Discretization improvements
+---------------------------
+- Added support for simplices in GSLIB-FindPoints.

 Improved testing
 ----------------
@@ -77,12 +49,6 @@ Miscellaneous
 - In SLISolver, changed the residual inner product from (Br,r) to (Br,Br) so the
  solver can work with non-SPD preconditioner B.

- Added support for ADIOS2 for parallel I/O with ParaView visualization. The
-  classes adios2stream and ADIOS2DataCollection are introduced in mfem as the
-  interfaces to generate ADIOS2 Binary Pack (BP4) directory datasets for the
-  entire spatial and temporal data. In addition, ADIOS2 allows for setting a
-  user-defined number of data substreams/subfiles. See examples 5, 9, 12, 16.
-

 Version 4.1, released on March 10, 2020
 =======================================
@@ -323,11 +323,6 @@ if (MFEM_USE_UMPIRE)
   find_package(UMPIRE REQUIRED)
 endif()

-# ADIOS2 for parallel I/O
-if (MFEM_USE_ADIOS2)
-   find_package(ADIOS2 REQUIRED)
-endif()
-
 # MFEM_TIMER_TYPE
 if (NOT DEFINED MFEM_TIMER_TYPE)
  if (APPLE)
@@ -353,7 +348,7 @@ endif()
 #    be before SuiteSparse.
 set(MFEM_TPLS MPI_CXX OPENMP BLAS LAPACK METIS HYPRE SuiteSparse SUNDIALS PETSC
    MESQUITE SuperLUDist STRUMPACK AXOM CONDUIT Ginkgo GNUTLS GSLIB NETCDF
-    MPFR PUMI HIOP POSIXCLOCKS MFEMBacktrace ZLIB OCCA CEED RAJA UMPIRE ADIOS2)
+    MPFR PUMI HIOP POSIXCLOCKS MFEMBacktrace ZLIB OCCA CEED RAJA UMPIRE)
 # Add all *_FOUND libraries in the variable TPL_LIBRARIES.
 set(TPL_LIBRARIES "")
 set(TPL_INCLUDE_DIRS "")
@@ -383,13 +383,9 @@ Before a PR can be merged, it should satisfy the following:
    - [ ] Is this a new feature users need to be aware of? New or updated example or miniapp?
    - [ ] Does it make sense to create a new section in the `CHANGELOG` to group with other related features?
 - [ ] Update `INSTALL`:
-    - [ ] Had a new optional library been added? If so, what range of versions of this library are required? (*Make sure the external library is compatible with our BSD license, e.g. it is not licensed under GPL!*)
-    - [ ] Have the version ranges for any required or optional libraries changed?
+    - [ ] Had a new optional library been added? (*Make sure the external library is compatible with our BSD license, e.g. it is not licensed under GPL!*)
    - [ ] Does `make` or `cmake` have a new target?
    - [ ] Did the requirements or the installation process change? *(rare)*
- [ ] Update continuous integration server configurations if necessary (e.g. with new version requirements for each of MFEM's dependencies)
-    - [ ] `.travis.yml`
-    - [ ] `.appveyor.yml`
 - [ ] Update `.gitignore`:
    - [ ] Check if `make distclean; git status` shows any files that were generated from the source by the project (not an IDE) but we don't want to track in the repository.
    - [ ] Add new patterns (just for the new files above) and re-run the above test.
@@ -503,10 +499,6 @@ MFEM uses a `master`/`next`-branch workflow as described below:
    - [ ] `makefile`
    - [ ] `CMakeLists.txt`
    - [ ] `doc/CodeDocumentation.conf.in`
- [ ] Check that version requirements for each of MFEM's dependencies are documented in `INSTALL` and up-to-date
- [ ] Check that continuous integration server configurations reflect the dependency version requirements of the new release
-    - [ ] `.travis.yml`
-    - [ ] `.appveyor.yml`
 - [ ] (LLNL only) Make sure all `README.html` files in the source repo are up to date.
 - [ ] Tag the repository:

@@ -403,11 +403,6 @@ MFEM_USE_CONDUIT = YES/NO
   an installation of Conduit. If Conduit was built with HDF5 support, it also
   requires an installation of HDF5 (see also MFEM_USE_NETCDF).

-MFEM_USE_ADIOS2 = YES/NO
-   Enables support for ADIOS2, version 2 of the adaptable input output system
-   for scientific data management. In MFEM, ADIOS2 provides parallel I/O with
-   ParaView visualization.
-
 MFEM_USE_ZLIB = YES/NO
   Enables use of on-the-fly gzip compressed streams. With this feature enabled
   (YES), MFEM can compress its output files on-the-fly. In addition, it can
@@ -497,13 +492,11 @@ The specific libraries and their options are:
 - HYPRE, required for the parallel build, i.e. when MFEM_USE_MPI = YES.
  URL: https://github.com/hypre-space/hypre and https://www.llnl.gov/casc/hypre
  Options: HYPRE_OPT, HYPRE_LIB.
-  Versions: HYPRE >= 2.10.0b.

 - METIS, used when MFEM_USE_METIS = YES. If using METIS 5, set
  MFEM_USE_METIS_5 = YES (default is to use METIS 4).
  URL: http://glaros.dtc.umn.edu/gkhome/metis/metis/overview
  Options: METIS_OPT, METIS_LIB.
-  Versions: METIS 4.0.3 or 5.1.0.

 - LAPACK (optional), used when MFEM_USE_LAPACK = YES. Alternative, optimized
  implementations can also be used, e.g. the ATLAS project.
@@ -527,7 +520,6 @@ The specific libraries and their options are:
  both MPI and hypre.
  URL: http://computation.llnl.gov/projects/sundials/sundials-software
  Options: SUNDIALS_OPT, SUNDIALS_LIB.
-  Versions: SUNDIALS >= 5.0.0.

 - Mesquite (optional), used when MFEM_USE_MESQUITE = YES.
  URL: http://trilinos.org/oldsite/packages/mesquite
@@ -536,7 +528,6 @@ The specific libraries and their options are:
 - SuiteSparse (optional), used when MFEM_USE_SUITESPARSE = YES.
  URL: http://faculty.cse.tamu.edu/davis/suitesparse.html
  Options: SUITESPARSE_OPT, SUITESPARSE_LIB.
-  Versions: SuiteSparse >= 4.5.4, older versions may work too.

 - SuperLU_DIST (optional), used when MFEM_USE_SUPERLU = YES. Note that
  SuperLU_DIST requires ParMETIS, which includes METIS 5 in its distribution.
@@ -544,7 +535,6 @@ The specific libraries and their options are:
  same location.
  URL: http://crd-legacy.lbl.gov/~xiaoye/SuperLU
  Options: SUPERLU_OPT, SUPERLU_LIB.
-  Versions: SuperLU_DIST >= 5.1.0.

 - STRUMPACK (optional), used when MFEM_USE_STRUMPACK = YES. Note that STRUMPACK
  requires the PT-Scotch and Scalapack libraries as well as ParMETIS, which
@@ -554,7 +544,6 @@ The specific libraries and their options are:
  2.0.0 or later.
  URL: http://portal.nersc.gov/project/sparse/strumpack
  Options: STRUMPACK_OPT, STRUMPACK_LIB.
-  Versions: STRUMPACK >= 3.0.0, requires HYPRE < 2.16.0.

 - Ginkgo (optional), used when MFEM_USE_GINKGO = YES. Note that Ginkgo needs a
  C++ compiler that supports the C++-11 standard. For additional requirements
@@ -567,7 +556,6 @@ The specific libraries and their options are:
  one can get the library through the Homebrew package manager (http://brew.sh).
  URL: http://gnutls.org
  Options: GNUTLS_OPT, GNUTLS_LIB.
-  Versions: GnuTLS >= 2.12.0, older versions may work too.

 - NetCDF (optional), used when MFEM_USE_NETCDF = YES, required for reading Cubit
  mesh files. Also requires installation of HDF5 and ZLIB, as explained at the
@@ -575,7 +563,6 @@ The specific libraries and their options are:
  don't need the C++ or parallel versions.
  URL: www.unidata.ucar.edu/software/netcdf
  Options: NETCDF_OPT, NETCDF_LIB.
-  Versions: NetCDF >= 4.4.0.

 - PETSc (optional), used when MFEM_USE_PETSC = YES. Version 3.8 or higher of
  the PETSC dev branch is required. The MFEM and PETSc builds can share common
@@ -587,7 +574,6 @@ The specific libraries and their options are:
                --with-shared-libraries=0
  URL: https://www.mcs.anl.gov/petsc
  Options: PETSC_OPT, PETSC_LIB.
-  Versions: PETSc >= 3.8.0.

 - Sidre (optional), part of LLNL's axom project, used when MFEM_USE_SIDRE = YES.
  Starting with MFEM v4.1, Axom version 0.3.1 or later is required.
@@ -595,22 +581,16 @@ The specific libraries and their options are:
       https://github.com/LLNL/conduit (Conduit)
       https://support.hdfgroup.org/HDF5 (HDF5)
  Options: SIDRE_OPT, SIDRE_LIB.
-  Versions: Axom >= 0.3.1.

 - Conduit (optional), used when MFEM_USE_CONDUIT = YES. Conduit Mesh Blueprint
  support requires Conduit >= v0.3.1 and VisIt >= v2.13.1 to read the output.
  URL: https://github.com/LLNL/conduit (Conduit)
       https://support.hdfgroup.org/HDF5 (HDF5)
  Options: CONDUIT_OPT, CONDUIT_LIB.
-  Versions: Conduit >= 0.3.1.
-
- ADIOS2 (optional) used when MFEM_USE_ADIOS2 = YES.
-  URL: https://adios2.readthedocs.io/

 - PUMI (optional), used when MFEM_USE_PUMI = YES.
  URL: https://scorec.rpi.edu/pumi
  Options: PUMI_OPT, PUMI_LIB.
-  Versions: PUMI >= 2.2.0.

 - HiOp (optional), used when MFEM_USE_HIOP = YES.
  URL: https://github.com/LLNL/hiop
@@ -624,12 +604,10 @@ The specific libraries and their options are:
  MFEM_USE_GSLIB=YES.
  URL: https://github.com/gslib/gslib/archive/v1.0.5.tar.gz
  Options: GSLIB_OPT, GSLIB_LIB.
-  Versions: GSLIB >= 1.0.5.

 - CUDA (optional), used when MFEM_USE_CUDA = YES.
  URL: https://developer.nvidia.com/cuda-toolkit
  Options: CUDA_CXX, CUDA_ARCH, CUDA_OPT, CUDA_LIB.
-  Versions: CUDA >= 9.1, older versions may work too.

 - HIP (optional), used when MFEM_USE_HIP = YES.
  URL: https://rocm.github.io/ROCmInstall.html
@@ -638,25 +616,21 @@ The specific libraries and their options are:
 - OCCA (optional), used when MFEM_USE_OCCA = YES.
  URL: https://libocca.org
  Options: OCCA_DIR, OCCA_OPT, OCCA_LIB.
-  Versions: OCCA >= 1.0.9.

 - libCEED (optional), used when MFEM_USE_CEED = YES. Requires libCEED v0.6
  or later version, specifically, git-hash 3d05795 or later.
  URL: https://github.com/CEED/libCEED
       https://ceed.exascaleproject.org/libceed
  Options: CEED_DIR, CEED_OPT, CEED_LIB.
-  Versions: libCEED >= 0.6.

 - RAJA (optional), used when MFEM_USE_RAJA = YES.
  Beginning with MFEM v4.1, only RAJA v0.10.0+ is supported.
  URL: https://github.com/LLNL/RAJA
  Options: RAJA_DIR, RAJA_OPT, RAJA_LIB.
-  Versions: RAJA >= 0.10.0.

 - Umpire, used when MFEM_USE_UMPIRE = YES.
  URL: https://github.com/LLNL/Umpire
  Options: UMPIRE_DIR, UMPIRE_OPT, UMPIRE_LIB.
-  Versions: Umpire >= 2.0.0.

 - MPFR (optional), used when MFEM_USE_MPFR = YES.
  URL: http://mpfr.org, it depends on the GMP library: https://gmplib.org
@@ -47,7 +47,6 @@ set(MFEM_USE_OCCA @MFEM_USE_OCCA@)
 set(MFEM_USE_RAJA @MFEM_USE_RAJA@)
 set(MFEM_USE_CEED @MFEM_USE_CEED@)
 set(MFEM_USE_UMPIRE @MFEM_USE_UMPIRE@)
-set(MFEM_USE_ADIOS2 @MFEM_USE_ADIOS2@)

 set(MFEM_CXX_COMPILER "@CMAKE_CXX_COMPILER@")
 set(MFEM_CXX_FLAGS "@CMAKE_CXX_FLAGS@")
@@ -132,9 +132,6 @@
 // Enable MFEM functionality based on the Umpire library
 #cmakedefine MFEM_USE_UMPIRE

-// Enable MFEM functionality based on the ADIOS2 library
-#cmakedefine MFEM_USE_ADIOS2
-
 // Which library functions to use in class StopWatch for measuring time.
 // For a list of the available options, see INSTALL.
 // If not defined, an option is selected automatically.
@@ -1,52 +0,0 @@
-#------------------------------------------------------------------------------#
-# Distributed under the OSI-approved Apache License, Version 2.0.  See
-# accompanying file Copyright.txt for details.
-#------------------------------------------------------------------------------#
-#
-# FindADIOS2
-# -----------
-#
-# Try to find the ADIOS2 library
-#
-# This module defines the following variables:
-#
-#   ADIOS2_FOUND        - System has ADIOS2
-#   ADIOS2_INCLUDE_DIRS - The ADIOS2 include directory
-#   ADIOS2_LIBRARIES    - Link these to use ADIOS2
-#
-# and the following imported targets:
-#   ADIOS2::ADIOS2 - The ADIOS2 compression library target
-#
-# You can also set the following variable to help guide the search:
-#   ADIOS2_DIR - The install prefix for ADIOS2 containing the
-#              include and lib folders
-#              Note: this can be set as a CMake variable or an
-#                    environment variable.  If specified as a CMake
-#                    variable, it will override any setting specified
-#                    as an environment variable.
-
-if(NOT ADIOS2_FOUND)
-  if((NOT ADIOS2_DIR) AND (NOT (ENV{ADIOS2_DIR} STREQUAL "")))
-    set(ADIOS2_DIR "$ENV{ADIOS2_DIR}")
-  endif()
-  if(ADIOS2_DIR)
-    set(ADIOS2_INCLUDE_OPTS HINTS ${ADIOS2_DIR}/include NO_DEFAULT_PATHS)
-    set(ADIOS2_LIBRARY_OPTS
-      HINTS ${ADIOS2_DIR}/lib ${ADIOS2_DIR}/lib64
-      NO_DEFAULT_PATHS
-    )
-  endif()
-
-  find_path(ADIOS2_INCLUDE_DIR adios2.h ${ADIOS2_INCLUDE_OPTS})
-  find_library(ADIOS2_LIBRARY NAMES adios2 ${ADIOS2_LIBRARY_OPTS}) 
-
-  include(FindPackageHandleStandardArgs)
-  find_package_handle_standard_args(ADIOS2
-    FOUND_VAR ADIOS2_FOUND
-    REQUIRED_VARS ADIOS2_LIBRARY ADIOS2_INCLUDE_DIR
-  )
-  if(ADIOS2_FOUND)
-    set(ADIOS2_INCLUDE_DIRS ${ADIOS2_INCLUDE_DIR})
-    set(ADIOS2_LIBRARIES ${ADIOS2_LIBRARY})
-  endif()
-endif()
@@ -147,9 +147,6 @@
 // Enable functionality based on the Umpire library.
 // #define MFEM_USE_UMPIRE

-// Enable IO functionality based on the ADIOS2 library.
-// #define MFEM_USE_ADIOS2
-
 // Version of HYPRE used for building MFEM.
 // #define MFEM_HYPRE_VERSION @MFEM_HYPRE_VERSION@

@@ -49,7 +49,6 @@ MFEM_USE_RAJA          = @MFEM_USE_RAJA@
 MFEM_USE_OCCA          = @MFEM_USE_OCCA@
 MFEM_USE_CEED          = @MFEM_USE_CEED@
 MFEM_USE_UMPIRE        = @MFEM_USE_UMPIRE@
-MFEM_USE_ADIOS2        = @MFEM_USE_ADIOS2@

 # Compiler, compile options, and link options
 MFEM_CXX       = @MFEM_CXX@
@@ -49,7 +49,6 @@ option(MFEM_USE_OCCA "Enable OCCA" OFF)
 option(MFEM_USE_RAJA "Enable RAJA" OFF)
 option(MFEM_USE_CEED "Enable CEED" OFF)
 option(MFEM_USE_UMPIRE "Enable Umpire" OFF)
-option(MFEM_USE_ADIOS2 "Enable ADIOS2" OFF)

 set(MFEM_MPI_NP 4 CACHE STRING "Number of processes used for MPI tests")

@@ -137,7 +137,6 @@ MFEM_USE_RAJA          = NO
 MFEM_USE_OCCA          = NO
 MFEM_USE_CEED          = NO
 MFEM_USE_UMPIRE        = NO
-MFEM_USE_ADIOS2        = NO

 # Compile and link options for zlib.
 ZLIB_DIR =
@@ -47,7 +47,7 @@ groups_serial=(
   "Meshing miniapps:"
   "miniapps/meshing"
   "mobius-strip.cpp klein-bottle.cpp extruder.cpp toroid.cpp
-    mesh-optimizer.cpp minimal-surface.cpp"'
+    mesh-optimizer.cpp"'
 )
 # Parallel groups
 groups_parallel=(
@@ -72,7 +72,7 @@ groups_parallel=(
 '"meshing"
   "Meshing miniapps:"
   "miniapps/meshing"
-   "pmesh-optimizer.cpp pminimal-surface.cpp"'
+   "pmesh-optimizer.cpp"'
 '"electromagnetics"
   "Electromagnetics miniapps:"
   "miniapps/electromagnetics"
@@ -101,7 +101,7 @@ groups_all=(
   "Meshing miniapps:"
   "miniapps/meshing"
   "mobius-strip.cpp klein-bottle.cpp extruder.cpp toroid.cpp
-    {,p}mesh-optimizer.cpp {,p}minimal-surface.cpp"'
+    {,p}mesh-optimizer.cpp"'
 '"electromagnetics"
   "Electromagnetics miniapps:"
   "miniapps/electromagnetics"
@@ -774,7 +774,6 @@ INPUT                  = @MFEM_SOURCE_DIR@/doc/CodeDocumentation.dox \
                         @MFEM_SOURCE_DIR@/miniapps/electromagnetics \
                         @MFEM_SOURCE_DIR@/miniapps/gslib \
                         @MFEM_SOURCE_DIR@/miniapps/meshing \
-                         @MFEM_SOURCE_DIR@/miniapps/navier \
                         @MFEM_SOURCE_DIR@/miniapps/nurbs \
                         @MFEM_SOURCE_DIR@/miniapps/performance \
                         @MFEM_SOURCE_DIR@/miniapps/tools \
@@ -88,8 +88,6 @@ namespace mfem {
 * - <a class="el" href="ex24p_8cpp_source.html">Example 24p</a>: parallel mixed finite element spaces and interpolators
 * - <a class="el" href="ex25_8cpp_source.html">Example 25</a>: simulation of electromagnetic wave propagation using a Perfectly Matched Layer (PML)
 * - <a class="el" href="ex25p_8cpp_source.html">Example 25p</a>: parallel simulation of electromagnetic wave propagation using a Perfectly Matched Layer (PML)
- * - <a class="el" href="ex26_8cpp_source.html">Example 26</a>: multigrid preconditioner for the Laplace problem using nodal H1 FEM 
- * - <a class="el" href="ex26p_8cpp_source.html">Example 26p</a>: parallel multigrid preconditioner for the Laplace problem using nodal H1 FEM 
 *
 * <H4>SUNDIALS Examples</H4>
 * - Variants of Examples
@@ -144,7 +142,6 @@ namespace mfem {
 * - <a class="el" href="klein-bottle_8cpp_source.html">Klein Bottle</a>: generate three types of Klein bottle surfaces
 * - <a class="el" href="toroid_8cpp_source.html">Toroid</a>: generate simple toroidal meshes
 * - <a class="el" href="twist_8cpp_source.html">Twist</a>: generate simple periodic meshes
- * - <a class="el" href="minimal-surface_8cpp_source.html">Minimal Surface</a>: compute minimal surfaces, <a class="el" href="minimal-surface_8cpp_source.html">serial</a> and <a class="el" href="pminimal-surface_8cpp_source.html">parallel</a> versions
 * - <a class="el" href="shaper_8cpp_source.html">Shaper</a>: resolve material interfaces by mesh refinement
 * - <a class="el" href="extruder_8cpp_source.html">Extruder</a>: extrude a low-dimensional mesh into a higher dimension
 * - <a class="el" href="mesh-explorer_8cpp_source.html">Mesh Explorer</a>: visualize and manipulate meshes
@@ -156,7 +153,6 @@ namespace mfem {
 * - <a class="el" href="lor-transfer_8cpp_source.html">LOR Transfer</a>: map functions between high-order and low-order refined spaces
 * - <a class="el" href="findpts_8cpp_source.html">Find Points</a>: evaluate grid function in physical space, <a class="el" href="findpts_8cpp_source.html">serial</a> and <a class="el" href="pfindpts_8cpp_source.html">parallel</a> versions
 * - <a class="el" href="field-diff_8cpp_source.html">Field Diff</a>: compare grid functions on different meshes
- * - <a class="el" href="classmfem_1_1navier_1_1NavierSolver.html">Navier</a>: solve the transient incompressible Navier-Stokes equations
 * - <a class="el" href="miniapps_2performance_2ex1_8cpp_source.html">HPC Example 1</a>: high-performance nodal H1 FEM for the Laplace problem
 * - <a class="el" href="miniapps_2performance_2ex1p_8cpp_source.html">HPC Example 1p</a>: high-performance parallel nodal H1 FEM for the Laplace problem
 *
@@ -32,8 +32,6 @@ list(APPEND ALL_EXE_SRCS
  ex23.cpp
  ex24.cpp
  ex25.cpp
-  ex26.cpp
-  ex27.cpp
  )

 if (MFEM_USE_MPI)
@@ -62,10 +60,6 @@ if (MFEM_USE_MPI)
    ex22p.cpp
    ex24p.cpp
    ex25p.cpp
-    ex26p.cpp
-    ex27p.cpp
-    pa_oper.cpp
-    io_benchmark.cpp
    )
 endif()

@@ -85,8 +79,6 @@ foreach(SRC_FILE ${ALL_EXE_SRCS})
    list(APPEND THIS_TEST_OPTIONS "-tf" "5")
  elseif(${TEST_NAME} MATCHES "ex15p*")
    list(APPEND THIS_TEST_OPTIONS "-e" "1")
-  elseif(${TEST_NAME} MATCHES "ex27p*")
-    list(APPEND THIS_TEST_OPTIONS "-dg")
  endif()

  if (NOT (${TEST_NAME} MATCHES ".*p$"))
@@ -33,8 +33,7 @@
 //               The example highlights the use of the LOBPCG eigenvalue solver
 //               together with the BoomerAMG preconditioner in HYPRE. Reusing a
 //               single GLVis visualization window for multiple eigenfunctions
-//               and optional saving with ADIOS2 (adios2.readthedocs.io) streams
-//               are also illustrated.
+//               is also illustrated.
 //
 //               We recommend viewing examples 2 and 11 before viewing this
 //               example.
@@ -61,7 +60,6 @@ int main(int argc, char *argv[])
   int seed = 66;
   bool visualization = 1;
   bool amg_elast = 0;
-   bool adios2 = false;

   OptionsParser args(argc, argv);
   args.AddOption(&mesh_file, "-m", "--mesh",
@@ -79,9 +77,6 @@ int main(int argc, char *argv[])
   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                  "--no-visualization",
                  "Enable or disable GLVis visualization.");
-   args.AddOption(&adios2, "-adios2", "--adios2-streams", "-no-adios2",
-                  "--no-adios2-streams",
-                  "Save data using adios2 streams.");
   args.Parse();
   if (!args.Good())
   {
@@ -291,28 +286,7 @@ int main(int argc, char *argv[])
      }
   }

-   // 13. Optionally output a BP (binary pack) file using ADIOS2. This can be
-   //     visualized with the ParaView VTX reader.
-#ifdef MFEM_USE_ADIOS2
-   if (adios2)
-   {
-      std::string postfix(mesh_file);
-      postfix.erase(0, std::string("../data/").size() );
-      postfix += "_o" + std::to_string(order);
-
-      adios2stream adios2output("ex12-p-" + postfix + ".bp",
-                                adios2stream::openmode::out, MPI_COMM_WORLD);
-      pmesh->Print(adios2output);
-      for (int i=0; i<nev; i++)
-      {
-         x = lobpcg->GetEigenvector(i);
-         // x is a temporary that must be saved immediately
-         x.Save(adios2output, "mode_" + std::to_string(i));
-      }
-   }
-#endif
-
-   // 14. Send the above data by socket to a GLVis server. Use the "n" and "b"
+   // 13. Send the above data by socket to a GLVis server. Use the "n" and "b"
   //     keys in GLVis to visualize the displacements.
   if (visualization)
   {
@@ -352,7 +326,7 @@ int main(int argc, char *argv[])
      mode_sock.close();
   }

-   // 15. Free the used memory.
+   // 14. Free the used memory.
   delete lobpcg;
   delete amg;
   delete M;
@@ -24,8 +24,7 @@
 //               class ConductionOperator defining C(u)), as well as their
 //               implicit time integration. Note that implementing the method
 //               ConductionOperator::ImplicitSolve is the only requirement for
-//               high-order implicit (SDIRK) time integration. Optional saving
-//               with ADIOS2 (adios2.readthedocs.io) is also illustrated.
+//               high-order implicit (SDIRK) time integration.
 //
 //               We recommend viewing examples 2, 9 and 10 before viewing this
 //               example.
@@ -109,7 +108,6 @@ int main(int argc, char *argv[])
   bool visualization = true;
   bool visit = false;
   int vis_steps = 5;
-   bool adios2 = false;

   int precision = 8;
   cout.precision(precision);
@@ -142,9 +140,6 @@ int main(int argc, char *argv[])
                  "Save data files for VisIt (visit.llnl.gov) visualization.");
   args.AddOption(&vis_steps, "-vs", "--visualization-steps",
                  "Visualize every n-th timestep.");
-   args.AddOption(&adios2, "-adios2", "--adios2-streams", "-no-adios2",
-                  "--no-adios2-streams",
-                  "Save data using adios2 streams.");
   args.Parse();
   if (!args.Good())
   {
@@ -253,27 +248,6 @@ int main(int argc, char *argv[])
      visit_dc.Save();
   }

-   // Optionally output a BP (binary pack) file using ADIOS2. This can be
-   // visualized with the ParaView VTX reader.
-#ifdef MFEM_USE_ADIOS2
-   ADIOS2DataCollection* adios2_dc = NULL;
-   if (adios2)
-   {
-      std::string postfix(mesh_file);
-      postfix.erase(0, std::string("../data/").size() );
-      postfix += "_o" + std::to_string(order);
-      postfix += "_solver" + std::to_string(ode_solver_type);
-      const std::string collection_name = "ex16-p-" + postfix + ".bp";
-
-      adios2_dc = new ADIOS2DataCollection(MPI_COMM_WORLD, collection_name, pmesh);
-      adios2_dc->SetParameter("SubStreams", std::to_string(num_procs/2) );
-      adios2_dc->RegisterField("temperature", &u_gf);
-      adios2_dc->SetCycle(0);
-      adios2_dc->SetTime(0.0);
-      adios2_dc->Save();
-   }
-#endif
-
   socketstream sout;
   if (visualization)
   {
@@ -343,26 +317,10 @@ int main(int argc, char *argv[])
            visit_dc.SetTime(t);
            visit_dc.Save();
         }
-
-#ifdef MFEM_USE_ADIOS2
-         if (adios2)
-         {
-            adios2_dc->SetCycle(ti);
-            adios2_dc->SetTime(t);
-            adios2_dc->Save();
-         }
-#endif
      }
      oper.SetParameters(u);
   }

-#ifdef MFEM_USE_ADIOS2
-   if (adios2)
-   {
-      delete adios2_dc;
-   }
-#endif
-
   // 11. Save the final solution in parallel. This output can be viewed later
   //     using GLVis: "glvis -np <np> -m ex16-mesh -g ex16-final".
   {
@@ -70,7 +70,6 @@ int main(int argc, char *argv[])
   bool pa = false;
   const char *device_config = "cpu";
   bool visualization = true;
-   int nfiles = 1;

   OptionsParser args(argc, argv);
   args.AddOption(&mesh_file, "-m", "--mesh",
@@ -87,7 +86,6 @@ int main(int argc, char *argv[])
   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                  "--no-visualization",
                  "Enable or disable GLVis visualization.");
-   args.AddOption(&nfiles, "-nf", "--num-files", "Number of files to write.");
   args.Parse();
   if (!args.Good())
   {
@@ -160,7 +158,7 @@ int main(int argc, char *argv[])
   {
      fec = new H1_FECollection(order = 1, dim);
   }
-   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec, 1, 0);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
   HYPRE_Int size = fespace->GlobalTrueVSize();
   if (myid == 0)
   {
@@ -239,52 +237,20 @@ int main(int argc, char *argv[])
   //     local finite element solution on each processor.
   a->RecoverFEMSolution(X, *b, x);

-   std::string filename("nranks_");
-   filename += to_string(num_procs);
-   filename += ".gf";
-   {
-      double t1;
-      t1 = MPI_Wtime();
-      x.Save(filename.c_str(), nfiles);
-      double t2 = MPI_Wtime();
-
-      if (myid == 0)
-      {
-         err << "elapsed write time: " << t2 - t1 << endl;
-      }
-   }
-   {
-      double t1;
-      t1 = MPI_Wtime();
-      ParGridFunction new_x(fespace, filename.c_str());
-      double t2 = MPI_Wtime();
-      if (myid == 0)
-      {
-         err << "elapsed read time: " << t2 - t1 << endl;
-      }
-      // new_x -= x;
-      // out << "GF difference: " << new_x.Norml1() << endl;
-   }
-
   // 15. Save the refined mesh and the solution in parallel. This output can
   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
   {
      ostringstream mesh_name, sol_name;
-      //mesh_name << "mesh." << setfill('0') << setw(6) << myid;
-      sol_name << "sol." << num_procs << setfill('0') << setw(6) << myid;
+      mesh_name << "mesh." << setfill('0') << setw(6) << myid;
+      sol_name << "sol." << setfill('0') << setw(6) << myid;
+
+      ofstream mesh_ofs(mesh_name.str().c_str());
+      mesh_ofs.precision(8);
+      pmesh->Print(mesh_ofs);

-      //ofstream mesh_ofs(mesh_name.str().c_str());
-      //mesh_ofs.precision(8);
-      //pmesh->Print(mesh_ofs);
-      double t1 = MPI_Wtime();
      ofstream sol_ofs(sol_name.str().c_str());
      sol_ofs.precision(8);
      x.Save(sol_ofs);
-      double t2 = MPI_Wtime();
-      if (myid == 0)
-      {
-         err << t2 - t1 << endl;
-      }
   }

   // 16. Send the solution by socket to a GLVis server.
@@ -1,255 +0,0 @@
-//                                MFEM Example 26
-//
-// Compile with: make ex26
-//
-// Sample runs:  ex26 -m ../data/star.mesh
-//               ex26 -m ../data/fichera.mesh
-//               ex26 -m ../data/beam-hex.mesh
-//
-// Device sample runs:
-//               ex26 -d cuda
-//               ex26 -d raja-cuda
-//               ex26 -d occa-cuda
-//               ex26 -d raja-omp
-//               ex26 -d occa-omp
-//               ex26 -d ceed-cpu
-//               ex26 -d ceed-cuda
-//               ex26 -m ../data/beam-hex.mesh -d cuda
-//
-// Description:  This example code demonstrates the use of MFEM to define a
-//               simple finite element discretization of the Laplace problem
-//               -Delta u = 1 with homogeneous Dirichlet boundary conditions
-//               as in Example 1.
-//
-//               It highlights on the creation of a hierarchy of discretization
-//               spaces with partial assembly and the construction of an
-//               efficient multigrid preconditioner for the iterative solver.
-//
-//               We recommend viewing Example 1 before viewing this example.
-
-#include "mfem.hpp"
-#include <fstream>
-#include <iostream>
-
-using namespace std;
-using namespace mfem;
-
-// Class for constructing a multigrid preconditioner for the diffusion operator.
-// This example multigrid preconditioner class demonstrates the creation of the
-// diffusion bilinear forms and operators using partial assembly for all spaces
-// in the FiniteElementSpaceHierarchy. The preconditioner uses a CG solver on
-// the coarsest level and second order Chebyshev accelerated smoothers on the
-// other levels.
-class DiffusionMultigrid : public Multigrid
-{
-private:
-   ConstantCoefficient one;
-
-public:
-   // Constructs a diffusion multigrid for the given FiniteElementSpaceHierarchy
-   // and the array of essential boundaries
-   DiffusionMultigrid(FiniteElementSpaceHierarchy& fespaces, Array<int>& ess_bdr)
-      : Multigrid(fespaces), one(1.0)
-   {
-      ConstructCoarseOperatorAndSolver(fespaces.GetFESpaceAtLevel(0), ess_bdr);
-
-      for (int level = 1; level < fespaces.GetNumLevels(); ++level)
-      {
-         ConstructOperatorAndSmoother(fespaces.GetFESpaceAtLevel(level), ess_bdr);
-      }
-   }
-
-private:
-   void ConstructBilinearForm(FiniteElementSpace& fespace, Array<int>& ess_bdr)
-   {
-      BilinearForm* form = new BilinearForm(&fespace);
-      form->SetAssemblyLevel(AssemblyLevel::PARTIAL);
-      form->AddDomainIntegrator(new DiffusionIntegrator(one));
-      form->Assemble();
-      bfs.Append(form);
-
-      essentialTrueDofs.Append(new Array<int>());
-      fespace.GetEssentialTrueDofs(ess_bdr, *essentialTrueDofs.Last());
-   }
-
-   void ConstructCoarseOperatorAndSolver(FiniteElementSpace& coarse_fespace,
-                                         Array<int>& ess_bdr)
-   {
-      ConstructBilinearForm(coarse_fespace, ess_bdr);
-
-      OperatorPtr opr;
-      opr.SetType(Operator::ANY_TYPE);
-      bfs.Last()->FormSystemMatrix(*essentialTrueDofs.Last(), opr);
-      opr.SetOperatorOwner(false);
-
-      CGSolver* pcg = new CGSolver();
-      pcg->SetPrintLevel(-1);
-      pcg->SetMaxIter(200);
-      pcg->SetRelTol(sqrt(1e-4));
-      pcg->SetAbsTol(0.0);
-      pcg->SetOperator(*opr.Ptr());
-
-      AddLevel(opr.Ptr(), pcg, true, true);
-   }
-
-   void ConstructOperatorAndSmoother(FiniteElementSpace& fespace,
-                                     Array<int>& ess_bdr)
-   {
-      ConstructBilinearForm(fespace, ess_bdr);
-
-      OperatorPtr opr;
-      opr.SetType(Operator::ANY_TYPE);
-      bfs.Last()->FormSystemMatrix(*essentialTrueDofs.Last(), opr);
-      opr.SetOperatorOwner(false);
-
-      Vector diag(fespace.GetTrueVSize());
-      bfs.Last()->AssembleDiagonal(diag);
-
-      Solver* smoother = new OperatorChebyshevSmoother(opr.Ptr(), diag,
-                                                       *essentialTrueDofs.Last(), 2);
-      AddLevel(opr.Ptr(), smoother, true, true);
-   }
-};
-
-
-int main(int argc, char *argv[])
-{
-   // 1. Parse command-line options.
-   const char *mesh_file = "../data/star.mesh";
-   int geometric_refinements = 0;
-   int order_refinements = 2;
-   const char *device_config = "cpu";
-   bool visualization = true;
-
-   OptionsParser args(argc, argv);
-   args.AddOption(&mesh_file, "-m", "--mesh",
-                  "Mesh file to use.");
-   args.AddOption(&geometric_refinements, "-gr", "--geometric-refinements",
-                  "Number of geometric refinements done prior to order refinements.");
-   args.AddOption(&order_refinements, "-or", "--order-refinements",
-                  "Number of order refinements. Finest level in the hierarchy has order 2^{or}.");
-   args.AddOption(&device_config, "-d", "--device",
-                  "Device configuration string, see Device::Configure().");
-   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
-                  "--no-visualization",
-                  "Enable or disable GLVis visualization.");
-   args.Parse();
-   if (!args.Good())
-   {
-      args.PrintUsage(cout);
-      return 1;
-   }
-   args.PrintOptions(cout);
-
-   // 2. Enable hardware devices such as GPUs, and programming models such as
-   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
-   Device device(device_config);
-   device.Print();
-
-   // 3. Read the mesh from the given mesh file. We can handle triangular,
-   //    quadrilateral, tetrahedral, hexahedral, surface and volume meshes with
-   //    the same code.
-   Mesh *mesh = new Mesh(mesh_file, 1, 1);
-   int dim = mesh->Dimension();
-
-   // 4. Refine the mesh to increase the resolution. In this example we do
-   //    'ref_levels' of uniform refinement. We choose 'ref_levels' to be the
-   //    largest number that gives a final mesh with no more than 50,000
-   //    elements.
-   {
-      int ref_levels =
-         (int)floor(log(5000./mesh->GetNE())/log(2.)/dim);
-      for (int l = 0; l < ref_levels; l++)
-      {
-         mesh->UniformRefinement();
-      }
-   }
-
-   // 5. Define a finite element space hierarchy on the mesh. Here we use
-   //    continuous Lagrange finite elements. We start with order 1 on the
-   //    coarse level and geometrically refine the spaces by the specified
-   //    amount. Afterwards, we increase the order of the finite elements
-   //    by a factor of 2 for each additional level.
-   FiniteElementCollection *fec = new H1_FECollection(1, dim);
-   FiniteElementSpace *coarse_fespace = new FiniteElementSpace(mesh, fec);
-   FiniteElementSpaceHierarchy fespaces(mesh, coarse_fespace, true, true);
-
-   Array<FiniteElementCollection*> collections;
-   collections.Append(fec);
-   for (int level = 0; level < geometric_refinements; ++level)
-   {
-      fespaces.AddUniformlyRefinedLevel();
-   }
-   for (int level = 0; level < order_refinements; ++level)
-   {
-      collections.Append(new H1_FECollection(std::pow(2, level+1), dim));
-      fespaces.AddOrderRefinedLevel(collections.Last());
-   }
-
-   cout << "Number of finite element unknowns: "
-        << fespaces.GetFinestFESpace().GetTrueVSize() << endl;
-
-   // 6. Set up the linear form b(.) which corresponds to the right-hand side of
-   //    the FEM linear system, which in this case is (1,phi_i) where phi_i are
-   //    the basis functions in the finite element fespace.
-   LinearForm *b = new LinearForm(&fespaces.GetFinestFESpace());
-   ConstantCoefficient one(1.0);
-   b->AddDomainIntegrator(new DomainLFIntegrator(one));
-   b->Assemble();
-
-   // 7. Define the solution vector x as a finite element grid function
-   //    corresponding to fespace. Initialize x with initial guess of zero,
-   //    which satisfies the boundary conditions.
-   GridFunction x(&fespaces.GetFinestFESpace());
-   x = 0.0;
-
-   // 8. Create the multigrid operator using the previously created
-   //    FiniteElementSpaceHierarchy and additional boundary information. This operator
-   //    is then used to create the MultigridSolver as a preconditioner in the
-   //    iterative solver.
-   Array<int> ess_bdr(mesh->bdr_attributes.Max());
-   ess_bdr = 1;
-
-   DiffusionMultigrid M(fespaces, ess_bdr);
-   M.SetCycleType(Multigrid::CycleType::VCYCLE, 1, 1);
-
-   OperatorPtr A;
-   Vector B, X;
-   M.FormFineLinearSystem(x, *b, A, X, B);
-   cout << "Size of linear system: " << A->Height() << endl;
-
-   // 9. Solve the linear system A X = B.
-   PCG(*A, M, B, X, 1, 2000, 1e-12, 0.0);
-
-   // 10. Recover the solution as a finite element grid function.
-   M.RecoverFineFEMSolution(X, *b, x);
-
-   // 11. Save the refined mesh and the solution. This output can be viewed later
-   //     using GLVis: "glvis -m refined.mesh -g sol.gf".
-   ofstream mesh_ofs("refined.mesh");
-   mesh_ofs.precision(8);
-   fespaces.GetFinestFESpace().GetMesh()->Print(mesh_ofs);
-   ofstream sol_ofs("sol.gf");
-   sol_ofs.precision(8);
-   x.Save(sol_ofs);
-
-   // 12. Send the solution by socket to a GLVis server.
-   if (visualization)
-   {
-      char vishost[] = "localhost";
-      int  visport   = 19916;
-      socketstream sol_sock(vishost, visport);
-      sol_sock.precision(8);
-      sol_sock << "solution\n" << *fespaces.GetFinestFESpace().GetMesh() << x <<
-               flush;
-   }
-
-   // 13. Free the used memory.
-   delete b;
-   for (int level = 0; level < collections.Size(); ++level)
-   {
-      delete collections[level];
-   }
-
-   return 0;
-}
@@ -1,317 +0,0 @@
-//                       MFEM Example 26 - Parallel Version
-//
-// Compile with: make ex26p
-//
-// Sample runs:  mpirun -np 4 ex26p -m ../data/star.mesh
-//               mpirun -np 4 ex26p -m ../data/fichera.mesh
-//               mpirun -np 4 ex26p -m ../data/beam-hex.mesh
-//
-// Device sample runs:
-//               mpirun -np 4 ex26p -d cuda
-//               mpirun -np 4 ex26p -d occa-cuda
-//               mpirun -np 4 ex26p -d raja-omp
-//               mpirun -np 4 ex26p -d ceed-cpu
-//               mpirun -np 4 ex26p -d ceed-cuda
-//
-// Description:  This example code demonstrates the use of MFEM to define a
-//               simple finite element discretization of the Laplace problem
-//               -Delta u = 1 with homogeneous Dirichlet boundary conditions
-//               as in Example 1.
-//
-//               It highlights on the creation of a hierarchy of discretization
-//               spaces with partial assembly and the construction of an
-//               efficient multigrid preconditioner for the iterative solver.
-//
-//               We recommend viewing Example 1 before viewing this example.
-
-#include "mfem.hpp"
-#include <fstream>
-#include <iostream>
-
-using namespace std;
-using namespace mfem;
-
-// Class for constructing a multigrid preconditioner for the diffusion operator.
-// This example multigrid preconditioner class demonstrates the creation of the
-// parallel diffusion bilinear forms and operators using partial assembly for
-// all spaces except the coarsest one in the ParFiniteElementSpaceHierarchy.
-// The multigrid uses a PCG solver preconditioned with AMG on the coarsest level
-// and second order Chebyshev accelerated smoothers on the other levels.
-class DiffusionMultigrid : public Multigrid
-{
-private:
-   ConstantCoefficient one;
-   HypreBoomerAMG* amg;
-
-public:
-   // Constructs a diffusion multigrid for the ParFiniteElementSpaceHierarchy
-   // and the array of essential boundaries
-   DiffusionMultigrid(ParFiniteElementSpaceHierarchy& fespaces,
-                      Array<int>& ess_bdr)
-      : Multigrid(fespaces), one(1.0)
-   {
-      ConstructCoarseOperatorAndSolver(fespaces.GetFESpaceAtLevel(0), ess_bdr);
-
-      for (int level = 1; level < fespaces.GetNumLevels(); ++level)
-      {
-         ConstructOperatorAndSmoother(fespaces.GetFESpaceAtLevel(level), ess_bdr);
-      }
-   }
-
-   virtual ~DiffusionMultigrid()
-   {
-      delete amg;
-   }
-
-private:
-   void ConstructBilinearForm(ParFiniteElementSpace& fespace, Array<int>& ess_bdr,
-                              bool partial_assembly)
-   {
-      ParBilinearForm* form = new ParBilinearForm(&fespace);
-      if (partial_assembly)
-      {
-         form->SetAssemblyLevel(AssemblyLevel::PARTIAL);
-      }
-      form->AddDomainIntegrator(new DiffusionIntegrator(one));
-      form->Assemble();
-      bfs.Append(form);
-
-      essentialTrueDofs.Append(new Array<int>());
-      fespace.GetEssentialTrueDofs(ess_bdr, *essentialTrueDofs.Last());
-   }
-
-   void ConstructCoarseOperatorAndSolver(ParFiniteElementSpace& coarse_fespace,
-                                         Array<int>& ess_bdr)
-   {
-      ConstructBilinearForm(coarse_fespace, ess_bdr, false);
-
-      HypreParMatrix* hypreCoarseMat = new HypreParMatrix();
-      bfs.Last()->FormSystemMatrix(*essentialTrueDofs.Last(), *hypreCoarseMat);
-
-      amg = new HypreBoomerAMG(*hypreCoarseMat);
-      amg->SetPrintLevel(-1);
-
-      CGSolver* pcg = new CGSolver(MPI_COMM_WORLD);
-      pcg->SetPrintLevel(-1);
-      pcg->SetMaxIter(10);
-      pcg->SetRelTol(sqrt(1e-4));
-      pcg->SetAbsTol(0.0);
-      pcg->SetOperator(*hypreCoarseMat);
-      pcg->SetPreconditioner(*amg);
-
-      AddLevel(hypreCoarseMat, pcg, true, true);
-   }
-
-   void ConstructOperatorAndSmoother(ParFiniteElementSpace& fespace,
-                                     Array<int>& ess_bdr)
-   {
-      ConstructBilinearForm(fespace, ess_bdr, true);
-
-      OperatorPtr opr;
-      opr.SetType(Operator::ANY_TYPE);
-      bfs.Last()->FormSystemMatrix(*essentialTrueDofs.Last(), opr);
-      opr.SetOperatorOwner(false);
-
-      Vector diag(fespace.GetTrueVSize());
-      bfs.Last()->AssembleDiagonal(diag);
-
-      Solver* smoother = new OperatorChebyshevSmoother(opr.Ptr(), diag,
-                                                       *essentialTrueDofs.Last(), 2, fespace.GetParMesh()->GetComm());
-
-      AddLevel(opr.Ptr(), smoother, true, true);
-   }
-};
-
-
-int main(int argc, char *argv[])
-{
-   // 1. Initialize MPI.
-   int num_procs, myid;
-   MPI_Init(&argc, &argv);
-   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
-   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
-
-   // 2. Parse command-line options.
-   const char *mesh_file = "../data/star.mesh";
-   int geometric_refinements = 0;
-   int order_refinements = 2;
-   const char *device_config = "cpu";
-   bool visualization = true;
-
-   OptionsParser args(argc, argv);
-   args.AddOption(&mesh_file, "-m", "--mesh",
-                  "Mesh file to use.");
-   args.AddOption(&geometric_refinements, "-gr", "--geometric-refinements",
-                  "Number of geometric refinements done prior to order refinements.");
-   args.AddOption(&order_refinements, "-or", "--order-refinements",
-                  "Number of order refinements. Finest level in the hierarchy has order 2^{or}.");
-   args.AddOption(&device_config, "-d", "--device",
-                  "Device configuration string, see Device::Configure().");
-   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
-                  "--no-visualization",
-                  "Enable or disable GLVis visualization.");
-   args.Parse();
-   if (!args.Good())
-   {
-      if (myid == 0)
-      {
-         args.PrintUsage(cout);
-      }
-      MPI_Finalize();
-      return 1;
-   }
-   if (myid == 0)
-   {
-      args.PrintOptions(cout);
-   }
-
-   // 3. Enable hardware devices such as GPUs, and programming models such as
-   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
-   Device device(device_config);
-   if (myid == 0) { device.Print(); }
-
-   // 4. Read the (serial) mesh from the given mesh file on all processors.  We
-   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
-   //    and volume meshes with the same code.
-   Mesh *mesh = new Mesh(mesh_file, 1, 1);
-   int dim = mesh->Dimension();
-
-   // 5. Refine the serial mesh on all processors to increase the resolution. In
-   //    this example we do 'ref_levels' of uniform refinement. We choose
-   //    'ref_levels' to be the largest number that gives a final mesh with no
-   //    more than 1,000 elements.
-   {
-      int ref_levels =
-         (int)floor(log(1000./mesh->GetNE())/log(2.)/dim);
-      for (int l = 0; l < ref_levels; l++)
-      {
-         mesh->UniformRefinement();
-      }
-   }
-
-   // 6. Define a parallel mesh by a partitioning of the serial mesh. Refine
-   //    this mesh further in parallel to increase the resolution. Once the
-   //    parallel mesh is defined, the serial mesh can be deleted.
-   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
-   delete mesh;
-   {
-      int par_ref_levels = 2;
-      for (int l = 0; l < par_ref_levels; l++)
-      {
-         pmesh->UniformRefinement();
-      }
-   }
-
-   // 7. Define a parallel finite element space hierarchy on the parallel mesh.
-   //    Here we use continuous Lagrange finite elements. We start with order 1
-   //    on the coarse level and geometrically refine the spaces by the specified
-   //    amount. Afterwards, we increase the order of the finite elements by a
-   //    factor of 2 for each additional level.
-   FiniteElementCollection *fec = new H1_FECollection(1, dim);
-   ParFiniteElementSpace *coarse_fespace = new ParFiniteElementSpace(pmesh, fec);
-
-   Array<FiniteElementCollection*> collections;
-   collections.Append(fec);
-   ParFiniteElementSpaceHierarchy* fespaces = new ParFiniteElementSpaceHierarchy(
-      pmesh, coarse_fespace, true, true);
-   for (int level = 0; level < geometric_refinements; ++level)
-   {
-      fespaces->AddUniformlyRefinedLevel();
-   }
-   for (int level = 0; level < order_refinements; ++level)
-   {
-      collections.Append(new H1_FECollection(std::pow(2, level+1), dim));
-      fespaces->AddOrderRefinedLevel(collections.Last());
-   }
-
-   HYPRE_Int size = fespaces->GetFinestFESpace().GlobalTrueVSize();
-   if (myid == 0)
-   {
-      cout << "Number of finite element unknowns: " << size << endl;
-   }
-
-   // 8. Set up the parallel linear form b(.) which corresponds to the
-   //    right-hand side of the FEM linear system, which in this case is
-   //    (1,phi_i) where phi_i are the basis functions in fespace.
-   ParLinearForm *b = new ParLinearForm(&fespaces->GetFinestFESpace());
-   ConstantCoefficient one(1.0);
-   b->AddDomainIntegrator(new DomainLFIntegrator(one));
-   b->Assemble();
-
-   // 9. Define the solution vector x as a parallel finite element grid function
-   //    corresponding to fespace. Initialize x with initial guess of zero,
-   //    which satisfies the boundary conditions.
-   ParGridFunction x(&fespaces->GetFinestFESpace());
-   x = 0.0;
-
-   // 10. Create the multigrid operator using the previously created parallel
-   //     FiniteElementSpaceHierarchy and additional boundary information. This operator
-   //     is then used to create the MultigridSolver as a preconditioner in the
-   //     iterative solver.
-   Array<int> ess_bdr(pmesh->bdr_attributes.Max());
-   if (pmesh->bdr_attributes.Size())
-   {
-      ess_bdr = 1;
-   }
-
-   DiffusionMultigrid* M = new DiffusionMultigrid(*fespaces, ess_bdr);
-   M->SetCycleType(Multigrid::CycleType::VCYCLE, 1, 1);
-
-   OperatorPtr A;
-   Vector X, B;
-   M->FormFineLinearSystem(x, *b, A, X, B);
-
-   // 11. Solve the linear system A X = B.
-   CGSolver cg(MPI_COMM_WORLD);
-   cg.SetRelTol(1e-12);
-   cg.SetMaxIter(2000);
-   cg.SetPrintLevel(1);
-   cg.SetOperator(*A);
-   cg.SetPreconditioner(*M);
-   cg.Mult(B, X);
-
-   // 12. Recover the parallel grid function corresponding to X. This is the
-   //     local finite element solution on each processor.
-   M->RecoverFineFEMSolution(X, *b, x);
-
-   // 13. Save the refined mesh and the solution in parallel. This output can
-   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
-   {
-      ostringstream mesh_name, sol_name;
-      mesh_name << "mesh." << setfill('0') << setw(6) << myid;
-      sol_name << "sol." << setfill('0') << setw(6) << myid;
-
-      ofstream mesh_ofs(mesh_name.str().c_str());
-      mesh_ofs.precision(8);
-      fespaces->GetFinestFESpace().GetParMesh()->Print(mesh_ofs);
-
-      ofstream sol_ofs(sol_name.str().c_str());
-      sol_ofs.precision(8);
-      x.Save(sol_ofs);
-   }
-
-   // 14. Send the solution by socket to a GLVis server.
-   if (visualization)
-   {
-      char vishost[] = "localhost";
-      int  visport   = 19916;
-      socketstream sol_sock(vishost, visport);
-      sol_sock << "parallel " << num_procs << " " << myid << "\n";
-      sol_sock.precision(8);
-      sol_sock << "solution\n" << *fespaces->GetFinestFESpace().GetParMesh()
-               << x << flush;
-   }
-
-   // 15. Free the used memory.
-   delete M;
-   delete b;
-   delete fespaces;
-   for (int level = 0; level < collections.Size(); ++level)
-   {
-      delete collections[level];
-   }
-
-   MPI_Finalize();
-
-   return 0;
-}
@@ -1,736 +0,0 @@
-//                       MFEM Example 27 - Serial Version
-//
-// Compile with: make ex27
-//
-// Sample runs:  ex27
-//               ex27 -dg
-//               ex27 -dg -dbc 8 -nbc -2
-//               ex27 -rbc-a 1 -rbc-b 8
-//
-// Description:  This example code demonstrates the use of MFEM to define a
-//               simple finite element discretization of the Laplace problem
-//               -Delta u = 0 with a variety of boundary conditions.
-//               Specifically, we discretize using a FE space of the specified
-//               order using a continuous or discontinuous space.  We then
-//               apply Dirichlet, Neumann (both homogeneous and inhomogeneous),
-//               Robin, and Periodic boundary conditions on different portions
-//               of a predefined mesh.
-//
-//               The predefined mesh consists of a rectangle with two
-//               holes removed (see below).  The narrow ends of the
-//               mesh are connected to form a Periodic boundary
-//               condition.  The lower edge (tagged with attribute 1)
-//               receives an inhomogeneous Neumann boundary condition.
-//               A Robin boundary condition is applied to upper edge
-//               (attribute 2).  The circular hole on the left
-//               (attribute 3) enforces a Dirichlet boundary
-//               condition.  Finally, a natural boundary condition, or
-//               homogeneous Neumann BC, is applied to the circular
-//               hole on the right (attribute 4).
-//
-//                  Attribute 3    ^ y  Attribute 2
-//                        \        |      /
-//                     +-----------+-----------+
-//                     |    \_     |     _     |
-//                     |    / \    |    / \    |
-//                  <--+---+---+---+---+---+---+--> x
-//                     |    \_/    |    \_/    |
-//                     |           |      \    |
-//                     +-----------+-----------+       (hole radii are
-//                          /      |        \            adjustable)
-//                  Attribute 1    v    Attribute 4
-//
-//               The boundary conditions are defined as (where u is
-//               the solution field):
-//                  Dirichlet: u = d
-//                  Neumann:   n.Grad(u) = g
-//                  Robin:     n.Grad(u) + a u = b
-//
-//               The user can adjust the values of 'd', 'g', 'a', and
-//               'b' with command line options.
-//
-//               This example highlights the differing implementations of
-//               boundary conditions with continuous and discontinuous Galerkin
-//               formulations of the Laplace problem.
-//
-//               We recommend viewing examples 1 and 14 before viewing this
-//               example.
-
-#include "mfem.hpp"
-#include <fstream>
-#include <iostream>
-
-using namespace std;
-using namespace mfem;
-
-static double a_ = 0.2;
-
-// Normal to hole with boundary attribute 4
-void n4Vec(const Vector &x, Vector &n) { n = x; n[0] -= 0.5; n /= -n.Norml2(); }
-
-Mesh * GenerateSerialMesh(int ref);
-
-// Compute the average value of alpha*n.Grad(sol) + beta*sol over the boundary
-// attributes marked in bdr_marker.  Also computes the L2 norm of
-// alpha*n.Grad(sol) + beta*sol - gamma  over the same boundary.
-double IntegrateBC(const GridFunction &sol, const Array<int> &bdr_marker,
-                   double alpha, double beta, double gamma,
-                   double &err);
-
-int main(int argc, char *argv[])
-{
-   // 1. Parse command-line options.
-   int ser_ref_levels = 2;
-   int order = 1;
-   double sigma = -1.0;
-   double kappa = -1.0;
-   bool h1 = true;
-   bool visualization = true;
-
-   double mat_val = 1.0;
-   double dbc_val = 0.0;
-   double nbc_val = 1.0;
-   double rbc_a_val = 1.0; // du/dn + a * u = b
-   double rbc_b_val = 1.0;
-
-   OptionsParser args(argc, argv);
-   args.AddOption(&h1, "-h1", "--continuous", "-dg", "--discontinuous",
-                  "Select continuous \"H1\" or discontinuous \"DG\" basis.");
-   args.AddOption(&order, "-o", "--order",
-                  "Finite element order (polynomial degree) or -1 for"
-                  " isoparametric space.");
-   args.AddOption(&sigma, "-s", "--sigma",
-                  "One of the two DG penalty parameters, typically +1/-1."
-                  " See the documentation of class DGDiffusionIntegrator.");
-   args.AddOption(&kappa, "-k", "--kappa",
-                  "One of the two DG penalty parameters, should be positive."
-                  " Negative values are replaced with (order+1)^2.");
-   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
-                  "Number of times to refine the mesh uniformly in serial.");
-   args.AddOption(&mat_val, "-mat", "--material-value",
-                  "Constant value for material coefficient "
-                  "in the Laplace operator.");
-   args.AddOption(&dbc_val, "-dbc", "--dirichlet-value",
-                  "Constant value for Dirichlet Boundary Condition.");
-   args.AddOption(&nbc_val, "-nbc", "--neumann-value",
-                  "Constant value for Neumann Boundary Condition.");
-   args.AddOption(&rbc_a_val, "-rbc-a", "--robin-a-value",
-                  "Constant 'a' value for Robin Boundary Condition: "
-                  "du/dn + a * u = b.");
-   args.AddOption(&rbc_b_val, "-rbc-b", "--robin-b-value",
-                  "Constant 'b' value for Robin Boundary Condition: "
-                  "du/dn + a * u = b.");
-   args.AddOption(&a_, "-a", "--radius",
-                  "Radius of holes in the mesh.");
-   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
-                  "--no-visualization",
-                  "Enable or disable GLVis visualization.");
-   args.Parse();
-   if (!args.Good())
-   {
-      args.PrintUsage(mfem::out);
-      return 1;
-   }
-   if (kappa < 0 && !h1)
-   {
-      kappa = (order+1)*(order+1);
-   }
-   args.PrintOptions(mfem::out);
-
-   if (a_ < 0.01)
-   {
-      mfem::out << "Hole radius too small, resetting to 0.01.\n";
-      a_ = 0.01;
-   }
-   if (a_ > 0.49)
-   {
-      mfem::out << "Hole radius too large, resetting to 0.49.\n";
-      a_ = 0.49;
-   }
-
-   // 2. Construct the (serial) mesh and refine it if requested.
-   Mesh *mesh = GenerateSerialMesh(ser_ref_levels);
-   int dim = mesh->Dimension();
-
-   // 3. Define a finite element space on the serial mesh.  Here we
-   //    use either continuous Lagrange finite elements or discontinuous
-   //    Galerkin finite elements of the specified order.
-   FiniteElementCollection *fec =
-      h1 ? (FiniteElementCollection*)new H1_FECollection(order, dim) :
-      (FiniteElementCollection*)new DG_FECollection(order, dim);
-   FiniteElementSpace fespace(mesh, fec);
-   int size = fespace.GetTrueVSize();
-   mfem::out << "Number of finite element unknowns: " << size << endl;
-
-   // 4. Create "marker arrays" to define the portions of the boundary
-   //    associated with each type of boundary condition.  These arrays
-   //    have an entry corresponding to each boundary attribute.
-   //    Placing a '1' in entry i marks attribute i+1 as being
-   //    active, '0' is inactive.
-   Array<int> nbc_bdr(mesh->bdr_attributes.Max());
-   Array<int> rbc_bdr(mesh->bdr_attributes.Max());
-   Array<int> dbc_bdr(mesh->bdr_attributes.Max());
-
-   nbc_bdr = 0; nbc_bdr[0] = 1;
-   rbc_bdr = 0; rbc_bdr[1] = 1;
-   dbc_bdr = 0; dbc_bdr[2] = 1;
-
-   Array<int> ess_tdof_list(0);
-   if (h1 && mesh->bdr_attributes.Size())
-   {
-      // For a continuous basis the linear system must be modifed to enforce
-      // an essential (Dirichlet) boundary condition.  In the DG case this is
-      // not necessary as the boundary condition will only be enforced weakly.
-      fespace.GetEssentialTrueDofs(dbc_bdr, ess_tdof_list);
-   }
-
-   // 5. Setup the various coefficients needed for the Laplace operator and
-   //    the various boundary conditions.  In general these coefficients could
-   //    be functions of position but here we use only constants.
-   ConstantCoefficient matCoef(mat_val);
-   ConstantCoefficient dbcCoef(dbc_val);
-   ConstantCoefficient nbcCoef(nbc_val);
-   ConstantCoefficient rbcACoef(rbc_a_val);
-   ConstantCoefficient rbcBCoef(rbc_b_val);
-
-   // Since the n.Grad(u) terms arise by integrating -Div(m Grad(u)) by parts
-   // we must introduce the coefficient 'm' into the boundary conditions.
-   // Therefore, in the case of the Neumann BC, we actually enforce
-   // m n.Grad(u) = m g rather than simply n.Grad(u) = g.
-   ProductCoefficient m_nbcCoef(matCoef, nbcCoef);
-   ProductCoefficient m_rbcACoef(matCoef, rbcACoef);
-   ProductCoefficient m_rbcBCoef(matCoef, rbcBCoef);
-
-   // 6. Define the solution vector u as a finite element grid function
-   //    corresponding to fespace. Initialize u with initial guess of zero.
-   GridFunction u(&fespace);
-   u = 0.0;
-
-   // 7. Set up the bilinear form a(.,.) on the finite element space
-   //    corresponding to the Laplacian operator -Delta, by adding the Diffusion
-   //    domain integrator.
-   BilinearForm a(&fespace);
-   a.AddDomainIntegrator(new DiffusionIntegrator(matCoef));
-   if (h1)
-   {
-      // Add a Mass integrator on the Robin boundary
-      a.AddBoundaryIntegrator(new MassIntegrator(m_rbcACoef), rbc_bdr);
-   }
-   else
-   {
-      // Add the interfacial portion of the Lapalce operator
-      a.AddInteriorFaceIntegrator(new DGDiffusionIntegrator(matCoef,
-                                                            sigma, kappa));
-
-      // Counteract the n.Grad(u) term on the Dirichlet portion of the boundary
-      a.AddBdrFaceIntegrator(new DGDiffusionIntegrator(matCoef, sigma, kappa),
-                             dbc_bdr);
-
-      // Augment the n.Grad(u) term with a*u on the Robin portion of boundary
-      a.AddBdrFaceIntegrator(new BoundaryMassIntegrator(m_rbcACoef),
-                             rbc_bdr);
-   }
-   a.Assemble();
-
-   // 8. Assemble the linear form for the right hand side vector.
-   LinearForm b(&fespace);
-
-   if (h1)
-   {
-      // Set the Dirchlet values in the solution vector
-      u.ProjectBdrCoefficient(dbcCoef, dbc_bdr);
-
-      // Add the desired value for n.Grad(u) on the Neumann boundary
-      b.AddBoundaryIntegrator(new BoundaryLFIntegrator(m_nbcCoef), nbc_bdr);
-
-      // Add the desired value for n.Grad(u) + a*u on the Robin boundary
-      b.AddBoundaryIntegrator(new BoundaryLFIntegrator(m_rbcBCoef), rbc_bdr);
-   }
-   else
-   {
-      // Add the desired value for the Dirchlet boundary
-      b.AddBdrFaceIntegrator(new DGDirichletLFIntegrator(dbcCoef, matCoef,
-                                                         sigma, kappa),
-                             dbc_bdr);
-
-      // Add the desired value for n.Grad(u) on the Neumann boundary
-      b.AddBdrFaceIntegrator(new BoundaryLFIntegrator(m_nbcCoef),
-                             nbc_bdr);
-
-      // Add the desired value for n.Grad(u) + a*u on the Robin boundary
-      b.AddBdrFaceIntegrator(new BoundaryLFIntegrator(m_rbcBCoef),
-                             rbc_bdr);
-   }
-   b.Assemble();
-
-   // 9. Construct the linear system.
-   OperatorPtr A;
-   Vector B, X;
-   a.FormLinearSystem(ess_tdof_list, u, b, A, X, B);
-
-#ifndef MFEM_USE_SUITESPARSE
-   // 10. Define a simple symmetric Gauss-Seidel preconditioner and use it to
-   //     solve the system AX=B with PCG in the symmetric case, and GMRES in the
-   //     non-symmetric one.
-   {
-      GSSmoother M((SparseMatrix&)(*A));
-      if (sigma == -1.0)
-      {
-         PCG(*A, M, B, X, 1, 500, 1e-12, 0.0);
-      }
-      else
-      {
-         GMRES(*A, M, B, X, 1, 500, 10, 1e-12, 0.0);
-      }
-   }
-#else
-   // 11. If MFEM was compiled with SuiteSparse, use UMFPACK to solve the
-   //     system.
-   UMFPackSolver umf_solver;
-   umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
-   umf_solver.SetOperator(*A);
-   umf_solver.Mult(B, X);
-#endif
-
-   // 12. Recover the grid function corresponding to U. This is the
-   //     local finite element solution.
-   a.RecoverFEMSolution(X, b, u);
-
-   // 13. Build a mass matrix to help solve for n.Grad(u) where 'n' is
-   //     a surface normal.
-   BilinearForm m(&fespace);
-   m.AddDomainIntegrator(new MassIntegrator);
-   m.Assemble();
-
-   ess_tdof_list.SetSize(0);
-   OperatorPtr M;
-   m.FormSystemMatrix(ess_tdof_list, M);
-
-   // 14. Compute the various boundary integrals.
-   mfem::out << endl
-             << "Verifying boundary conditions" << endl
-             << "=============================" << endl;
-   {
-      // Integrate the solution on the Dirichlet boundary and compare
-      // to the expected value.
-      double err, avg = IntegrateBC(u, dbc_bdr, 0.0, 1.0, dbc_val, err);
-
-      bool hom_dbc = (dbc_val == 0.0);
-      err /=  hom_dbc ? 1.0 : fabs(dbc_val);
-      mfem::out << "Average of solution on Gamma_dbc:\t"
-                << avg << ", \t"
-                << (hom_dbc ? "absolute" : "relative")
-                << " error " << err << endl;
-   }
-   {
-      // Integrate n.Grad(u) on the inhomogeneous Neumann boundary and
-      // compare to the expected value.
-      double err, avg = IntegrateBC(u, nbc_bdr, 1.0, 0.0, nbc_val, err);
-
-      bool hom_nbc = (nbc_val == 0.0);
-      err /=  hom_nbc ? 1.0 : fabs(nbc_val);
-      mfem::out << "Average of n.Grad(u) on Gamma_nbc:\t"
-                << avg << ", \t"
-                << (hom_nbc ? "absolute" : "relative")
-                << " error " << err << endl;
-   }
-   {
-      // Integrate n.Grad(u) on the homogeneous Neumann boundary and compare
-      // to the expected value of zero.
-      Array<int> nbc0_bdr(mesh->bdr_attributes.Max());
-      nbc0_bdr = 0;
-      nbc0_bdr[3] = 1;
-
-      double err, avg = IntegrateBC(u, nbc0_bdr, 1.0, 0.0, 0.0, err);
-
-      bool hom_nbc = true;
-      mfem::out << "Average of n.Grad(u) on Gamma_nbc0:\t"
-                << avg << ", \t"
-                << (hom_nbc ? "absolute" : "relative")
-                << " error " << err << endl;
-   }
-   {
-      // Integrate n.Grad(u) + a * u on the Robin boundary and compare to
-      // the expected value.
-      double err, avg = IntegrateBC(u, rbc_bdr, 1.0, rbc_a_val, rbc_b_val, err);
-
-      bool hom_rbc = (rbc_b_val == 0.0);
-      err /=  hom_rbc ? 1.0 : fabs(rbc_b_val);
-      mfem::out << "Average of n.Grad(u)+a*u on Gamma_rbc:\t"
-                << avg << ", \t"
-                << (hom_rbc ? "absolute" : "relative")
-                << " error " << err << endl;
-   }
-
-   // 15. Save the refined mesh and the solution. This output can be viewed
-   //     later using GLVis: "glvis -m refined.mesh -g sol.gf".
-   {
-      ofstream mesh_ofs("refined.mesh");
-      mesh_ofs.precision(8);
-      mesh->Print(mesh_ofs);
-      ofstream sol_ofs("sol.gf");
-      sol_ofs.precision(8);
-      u.Save(sol_ofs);
-   }
-
-   // 16. Send the solution by socket to a GLVis server.
-   if (visualization)
-   {
-      string title_str = h1 ? "H1" : "DG";
-      char vishost[] = "localhost";
-      int  visport   = 19916;
-      socketstream sol_sock(vishost, visport);
-      sol_sock.precision(8);
-      sol_sock << "solution\n" << *mesh << u
-               << "window_title '" << title_str << " Solution'"
-               << " keys 'mmc'" << flush;
-   }
-
-   // 17. Free the used memory.
-   delete fec;
-   delete mesh;
-
-   return 0;
-}
-
-void quad_trans(double u, double v, double &x, double &y, bool log = false)
-{
-   double a = a_; // Radius of disc
-
-   double d = 4.0 * a * (M_SQRT2 - 2.0 * a) * (1.0 - 2.0 * v);
-
-   double v0 = (1.0 + M_SQRT2) * (M_SQRT2 * a - 2.0 * v) *
-               ((4.0 - 3 * M_SQRT2) * a +
-                (8.0 * (M_SQRT2 - 1.0) * a - 2.0) * v) / d;
-
-   double r = 2.0 * ((M_SQRT2 - 1.0) * a * a * (1.0 - 4.0 *v) +
-                     2.0 * (1.0 + M_SQRT2 *
-                            (1.0 + 2.0 * (2.0 * a - M_SQRT2 - 1.0) * a)) * v * v
-                    ) / d;
-
-   double t = asin(v / r) * u / v;
-   if (log)
-   {
-      mfem::out << "u, v, r, v0, t "
-                << u << " " << v << " " << r << " " << v0 << " " << t
-                << endl;
-   }
-   x = r * sin(t);
-   y = r * cos(t) - v0;
-}
-
-void trans(const Vector &u, Vector &x)
-{
-   double tol = 1e-4;
-
-   if (u[1] > 0.5 - tol || u[1] < -0.5 + tol)
-   {
-      x = u;
-      return;
-   }
-   if (u[0] > 1.0 - tol || u[0] < -1.0 + tol || fabs(u[0]) < tol)
-   {
-      x = u;
-      return;
-   }
-
-   if (u[0] > 0.0)
-   {
-      if (u[1] > fabs(u[0] - 0.5))
-      {
-         quad_trans(u[0] - 0.5, u[1], x[0], x[1]);
-         x[0] += 0.5;
-         return;
-      }
-      if (u[1] < -fabs(u[0] - 0.5))
-      {
-         quad_trans(u[0] - 0.5, -u[1], x[0], x[1]);
-         x[0] += 0.5;
-         x[1] *= -1.0;
-         return;
-      }
-      if (u[0] - 0.5 > fabs(u[1]))
-      {
-         quad_trans(u[1], u[0] - 0.5, x[1], x[0]);
-         x[0] += 0.5;
-         return;
-      }
-      if (u[0] - 0.5 < -fabs(u[1]))
-      {
-         quad_trans(u[1], 0.5 - u[0], x[1], x[0]);
-         x[0] *= -1.0;
-         x[0] += 0.5;
-         return;
-      }
-   }
-   else
-   {
-      if (u[1] > fabs(u[0] + 0.5))
-      {
-         quad_trans(u[0] + 0.5, u[1], x[0], x[1]);
-         x[0] -= 0.5;
-         return;
-      }
-      if (u[1] < -fabs(u[0] + 0.5))
-      {
-         quad_trans(u[0] + 0.5, -u[1], x[0], x[1]);
-         x[0] -= 0.5;
-         x[1] *= -1.0;
-         return;
-      }
-      if (u[0] + 0.5 > fabs(u[1]))
-      {
-         quad_trans(u[1], u[0] + 0.5, x[1], x[0]);
-         x[0] -= 0.5;
-         return;
-      }
-      if (u[0] + 0.5 < -fabs(u[1]))
-      {
-         quad_trans(u[1], -0.5 - u[0], x[1], x[0]);
-         x[0] *= -1.0;
-         x[0] -= 0.5;
-         return;
-      }
-   }
-   x = u;
-}
-
-Mesh * GenerateSerialMesh(int ref)
-{
-   Mesh * mesh = new Mesh(2, 29, 16, 24, 2);
-
-   int vi[4];
-
-   for (int i=0; i<2; i++)
-   {
-      int o = 13 * i;
-      vi[0] = o + 0; vi[1] = o + 3; vi[2] = o + 4; vi[3] = o + 1;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 1; vi[1] = o + 4; vi[2] = o + 5; vi[3] = o + 2;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 5; vi[1] = o + 8; vi[2] = o + 9; vi[3] = o + 2;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 8; vi[1] = o + 12; vi[2] = o + 15; vi[3] = o + 9;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 11; vi[1] = o + 14; vi[2] = o + 15; vi[3] = o + 12;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 10; vi[1] = o + 13; vi[2] = o + 14; vi[3] = o + 11;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 6; vi[1] = o + 13; vi[2] = o + 10; vi[3] = o + 7;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 0; vi[1] = o + 6; vi[2] = o + 7; vi[3] = o + 3;
-      mesh->AddQuad(vi);
-   }
-
-   vi[0] =  0; vi[1] =  6; mesh->AddBdrSegment(vi, 1);
-   vi[0] =  6; vi[1] = 13; mesh->AddBdrSegment(vi, 1);
-   vi[0] = 13; vi[1] = 19; mesh->AddBdrSegment(vi, 1);
-   vi[0] = 19; vi[1] = 26; mesh->AddBdrSegment(vi, 1);
-
-   vi[0] = 28; vi[1] = 22; mesh->AddBdrSegment(vi, 2);
-   vi[0] = 22; vi[1] = 15; mesh->AddBdrSegment(vi, 2);
-   vi[0] = 15; vi[1] =  9; mesh->AddBdrSegment(vi, 2);
-   vi[0] =  9; vi[1] =  2; mesh->AddBdrSegment(vi, 2);
-
-   for (int i=0; i<2; i++)
-   {
-      int o = 13 * i;
-      vi[0] = o +  7; vi[1] = o +  3; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o + 10; vi[1] = o +  7; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o + 11; vi[1] = o + 10; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o + 12; vi[1] = o + 11; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o +  8; vi[1] = o + 12; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o +  5; vi[1] = o +  8; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o +  4; vi[1] = o +  5; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o +  3; vi[1] = o +  4; mesh->AddBdrSegment(vi, 3 + i);
-   }
-
-   double d[2];
-   double a = a_ / M_SQRT2;
-
-   d[0] = -1.0; d[1] = -0.5; mesh->AddVertex(d);
-   d[0] = -1.0; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] = -1.0; d[1] =  0.5; mesh->AddVertex(d);
-
-   d[0] = -0.5 - a; d[1] =   -a; mesh->AddVertex(d);
-   d[0] = -0.5 - a; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] = -0.5 - a; d[1] =    a; mesh->AddVertex(d);
-
-   d[0] = -0.5; d[1] = -0.5; mesh->AddVertex(d);
-   d[0] = -0.5; d[1] =   -a; mesh->AddVertex(d);
-   d[0] = -0.5; d[1] =    a; mesh->AddVertex(d);
-   d[0] = -0.5; d[1] =  0.5; mesh->AddVertex(d);
-
-   d[0] = -0.5 + a; d[1] =   -a; mesh->AddVertex(d);
-   d[0] = -0.5 + a; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] = -0.5 + a; d[1] =    a; mesh->AddVertex(d);
-
-   d[0] =  0.0; d[1] = -0.5; mesh->AddVertex(d);
-   d[0] =  0.0; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] =  0.0; d[1] =  0.5; mesh->AddVertex(d);
-
-   d[0] =  0.5 - a; d[1] =   -a; mesh->AddVertex(d);
-   d[0] =  0.5 - a; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] =  0.5 - a; d[1] =    a; mesh->AddVertex(d);
-
-   d[0] =  0.5; d[1] = -0.5; mesh->AddVertex(d);
-   d[0] =  0.5; d[1] =   -a; mesh->AddVertex(d);
-   d[0] =  0.5; d[1] =    a; mesh->AddVertex(d);
-   d[0] =  0.5; d[1] =  0.5; mesh->AddVertex(d);
-
-   d[0] =  0.5 + a; d[1] =   -a; mesh->AddVertex(d);
-   d[0] =  0.5 + a; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] =  0.5 + a; d[1] =    a; mesh->AddVertex(d);
-
-   d[0] =  1.0; d[1] = -0.5; mesh->AddVertex(d);
-   d[0] =  1.0; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] =  1.0; d[1] =  0.5; mesh->AddVertex(d);
-
-   mesh->FinalizeTopology();
-
-   mesh->SetCurvature(1, true);
-
-   // Stitch the ends of the stack together
-   {
-      Array<int> v2v(mesh->GetNV());
-      for (int i = 0; i < v2v.Size() - 3; i++)
-      {
-         v2v[i] = i;
-      }
-      // identify vertices on the narrow ends of the rectangle
-      v2v[v2v.Size() - 3] = 0;
-      v2v[v2v.Size() - 2] = 1;
-      v2v[v2v.Size() - 1] = 2;
-
-      // renumber elements
-      for (int i = 0; i < mesh->GetNE(); i++)
-      {
-         Element *el = mesh->GetElement(i);
-         int *v = el->GetVertices();
-         int nv = el->GetNVertices();
-         for (int j = 0; j < nv; j++)
-         {
-            v[j] = v2v[v[j]];
-         }
-      }
-      // renumber boundary elements
-      for (int i = 0; i < mesh->GetNBE(); i++)
-      {
-         Element *el = mesh->GetBdrElement(i);
-         int *v = el->GetVertices();
-         int nv = el->GetNVertices();
-         for (int j = 0; j < nv; j++)
-         {
-            v[j] = v2v[v[j]];
-         }
-      }
-      mesh->RemoveUnusedVertices();
-      mesh->RemoveInternalBoundaries();
-   }
-   mesh->SetCurvature(3, true);
-
-   for (int l = 0; l < ref; l++)
-   {
-      mesh->UniformRefinement();
-   }
-
-   mesh->Transform(trans);
-
-   return mesh;
-}
-
-double IntegrateBC(const GridFunction &x, const Array<int> &bdr,
-                   double alpha, double beta, double gamma,
-                   double &err)
-{
-   double nrm = 0.0;
-   double avg = 0.0;
-   err = 0.0;
-
-   const bool a_is_zero = alpha == 0.0;
-   const bool b_is_zero = beta == 0.0;
-
-   const FiniteElementSpace &fes = *x.FESpace();
-   MFEM_ASSERT(fes.GetVDim() == 1, "");
-   Mesh &mesh = *fes.GetMesh();
-   Vector shape, loc_dofs, w_nor;
-   DenseMatrix dshape;
-   Array<int> dof_ids;
-   for (int i = 0; i < mesh.GetNBE(); i++)
-   {
-      if (bdr[mesh.GetBdrAttribute(i)-1] == 0) { continue; }
-
-      FaceElementTransformations *FTr = mesh.GetBdrFaceTransformations(i);
-      if (FTr == nullptr) { continue; }
-
-      const FiniteElement &fe = *fes.GetFE(FTr->Elem1No);
-      MFEM_ASSERT(fe.GetMapType() == FiniteElement::VALUE, "");
-      const int int_order = 2*fe.GetOrder() + 3;
-      const IntegrationRule &ir = IntRules.Get(FTr->FaceGeom, int_order);
-
-      fes.GetElementDofs(FTr->Elem1No, dof_ids);
-      x.GetSubVector(dof_ids, loc_dofs);
-      if (!a_is_zero)
-      {
-         const int sdim = FTr->Face->GetSpaceDim();
-         w_nor.SetSize(sdim);
-         dshape.SetSize(fe.GetDof(), sdim);
-      }
-      if (!b_is_zero)
-      {
-         shape.SetSize(fe.GetDof());
-      }
-      for (int j = 0; j < ir.GetNPoints(); j++)
-      {
-         const IntegrationPoint &ip = ir.IntPoint(j);
-         IntegrationPoint eip;
-         FTr->Loc1.Transform(ip, eip);
-         FTr->Face->SetIntPoint(&ip);
-         double face_weight = FTr->Face->Weight();
-         double val = 0.0;
-         if (!a_is_zero)
-         {
-            FTr->Elem1->SetIntPoint(&eip);
-            fe.CalcPhysDShape(*FTr->Elem1, dshape);
-            CalcOrtho(FTr->Face->Jacobian(), w_nor);
-            val += alpha * dshape.InnerProduct(w_nor, loc_dofs) / face_weight;
-         }
-         if (!b_is_zero)
-         {
-            fe.CalcShape(eip, shape);
-            val += beta * (shape * loc_dofs);
-         }
-
-         // Measure the length of the boundary
-         nrm += ip.weight * face_weight;
-
-         // Integrate alpha * n.Grad(x) + beta * x
-         avg += val * ip.weight * face_weight;
-
-         // Integrate |alpha * n.Grad(x) + beta * x - gamma|^2
-         val -= gamma;
-         err += (val*val) * ip.weight * face_weight;
-      }
-   }
-
-   // Normalize by the length of the boundary
-   if (std::abs(nrm) > 0.0)
-   {
-      err /= nrm;
-      avg /= nrm;
-   }
-
-   // Compute l2 norm of the error in the boundary condition
-   // (negative quadrature weights may produce negative 'err')
-   err = (err >= 0.0) ? sqrt(err) : -sqrt(-err);
-
-   // Return the average value of alpha * n.Grad(x) + beta * x
-   return avg;
-}
@@ -1,773 +0,0 @@
-//                       MFEM Example 27 - Parallel Version
-//
-// Compile with: make ex27p
-//
-// Sample runs:  mpirun -np 4 ex27p
-//               mpirun -np 4 ex27p -dg
-//               mpirun -np 4 ex27p -dg -dbc 8 -nbc -2
-//               mpirun -np 4 ex27p -rbc-a 1 -rbc-b 8
-//
-// Description:  This example code demonstrates the use of MFEM to define a
-//               simple finite element discretization of the Laplace problem
-//               -Delta u = 0 with a variety of boundary conditions.
-//               Specifically, we discretize using a FE space of the specified
-//               order using a continuous or discontinuous space.  We then
-//               apply Dirichlet, Neumann (both homogeneous and inhomogeneous),
-//               Robin, and Periodic boundary conditions on different portions
-//               of a predefined mesh.
-//
-//               The predefined mesh consists of a rectangle with two
-//               holes removed (see below).  The narrow ends of the
-//               mesh are connected to form a Periodic boundary
-//               condition.  The lower edge (tagged with attribute 1)
-//               receives an inhomogeneous Neumann boundary condition.
-//               A Robin boundary condition is applied to upper edge
-//               (attribute 2).  The circular hole on the left
-//               (attribute 3) enforces a Dirichlet boundary
-//               condition.  Finally, a natural boundary condition, or
-//               homogeneous Neumann BC, is applied to the circular
-//               hole on the right (attribute 4).
-//
-//                  Attribute 3    ^ y  Attribute 2
-//                        \        |      /
-//                     +-----------+-----------+
-//                     |    \_     |     _     |
-//                     |    / \    |    / \    |
-//                  <--+---+---+---+---+---+---+--> x
-//                     |    \_/    |    \_/    |
-//                     |           |      \    |
-//                     +-----------+-----------+       (hole radii are
-//                          /      |        \            adjustable)
-//                  Attribute 1    v    Attribute 4
-//
-//               The boundary conditions are defined as (where u is
-//               the solution field):
-//                  Dirichlet: u = d
-//                  Neumann:   n.Grad(u) = g
-//                  Robin:     n.Grad(u) + a u = b
-//
-//               The user can adjust the values of 'd', 'g', 'a', and
-//               'b' with command line options.
-//
-//               This example highlights the differing implementations of
-//               boundary conditions with continuous and discontinuous Galerkin
-//               formulations of the Laplace problem.
-//
-//               We recommend viewing examples 1 and 14 before viewing this
-//               example.
-
-#include "mfem.hpp"
-#include <fstream>
-#include <iostream>
-
-using namespace std;
-using namespace mfem;
-
-static double a_ = 0.2;
-
-// Normal to hole with boundary attribute 4
-void n4Vec(const Vector &x, Vector &n) { n = x; n[0] -= 0.5; n /= -n.Norml2(); }
-
-Mesh * GenerateSerialMesh(int ref);
-
-// Compute the average value of alpha*n.Grad(sol) + beta*sol over the boundary
-// attributes marked in bdr_marker.  Also computes the L2 norm of
-// alpha*n.Grad(sol) + beta*sol - gamma  over the same boundary.
-double IntegrateBC(const ParGridFunction &sol, const Array<int> &bdr_marker,
-                   double alpha, double beta, double gamma,
-                   double &err);
-
-int main(int argc, char *argv[])
-{
-   // 1. Initialize MPI.
-   MPI_Session mpi;
-   if (!mpi.Root()) { mfem::out.Disable(); mfem::err.Disable(); }
-
-   // 2. Parse command-line options.
-   int ser_ref_levels = 2;
-   int par_ref_levels = 1;
-   int order = 1;
-   double sigma = -1.0;
-   double kappa = -1.0;
-   bool h1 = true;
-   bool visualization = true;
-
-   double mat_val = 1.0;
-   double dbc_val = 0.0;
-   double nbc_val = 1.0;
-   double rbc_a_val = 1.0; // du/dn + a * u = b
-   double rbc_b_val = 1.0;
-
-   OptionsParser args(argc, argv);
-   args.AddOption(&h1, "-h1", "--continuous", "-dg", "--discontinuous",
-                  "Select continuous \"H1\" or discontinuous \"DG\" basis.");
-   args.AddOption(&order, "-o", "--order",
-                  "Finite element order (polynomial degree) or -1 for"
-                  " isoparametric space.");
-   args.AddOption(&sigma, "-s", "--sigma",
-                  "One of the two DG penalty parameters, typically +1/-1."
-                  " See the documentation of class DGDiffusionIntegrator.");
-   args.AddOption(&kappa, "-k", "--kappa",
-                  "One of the two DG penalty parameters, should be positive."
-                  " Negative values are replaced with (order+1)^2.");
-   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
-                  "Number of times to refine the mesh uniformly in serial.");
-   args.AddOption(&par_ref_levels, "-rp", "--refine-parallel",
-                  "Number of times to refine the mesh uniformly in parallel.");
-   args.AddOption(&mat_val, "-mat", "--material-value",
-                  "Constant value for material coefficient "
-                  "in the Laplace operator.");
-   args.AddOption(&dbc_val, "-dbc", "--dirichlet-value",
-                  "Constant value for Dirichlet Boundary Condition.");
-   args.AddOption(&nbc_val, "-nbc", "--neumann-value",
-                  "Constant value for Neumann Boundary Condition.");
-   args.AddOption(&rbc_a_val, "-rbc-a", "--robin-a-value",
-                  "Constant 'a' value for Robin Boundary Condition: "
-                  "du/dn + a * u = b.");
-   args.AddOption(&rbc_b_val, "-rbc-b", "--robin-b-value",
-                  "Constant 'b' value for Robin Boundary Condition: "
-                  "du/dn + a * u = b.");
-   args.AddOption(&a_, "-a", "--radius",
-                  "Radius of holes in the mesh.");
-   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
-                  "--no-visualization",
-                  "Enable or disable GLVis visualization.");
-   args.Parse();
-   if (!args.Good())
-   {
-      args.PrintUsage(mfem::out);
-      return 1;
-   }
-   if (kappa < 0 && !h1)
-   {
-      kappa = (order+1)*(order+1);
-   }
-   args.PrintOptions(mfem::out);
-
-   if (a_ < 0.01)
-   {
-      mfem::out << "Hole radius too small, resetting to 0.01.\n";
-      a_ = 0.01;
-   }
-   if (a_ > 0.49)
-   {
-      mfem::out << "Hole radius too large, resetting to 0.49.\n";
-      a_ = 0.49;
-   }
-
-   // 3. Construct the (serial) mesh and refine it if requested.
-   Mesh *mesh = GenerateSerialMesh(ser_ref_levels);
-   int dim = mesh->Dimension();
-
-   // 4. Define a parallel mesh by a partitioning of the serial mesh. Refine
-   //    this mesh further in parallel to increase the resolution. Once the
-   //    parallel mesh is defined, the serial mesh can be deleted.
-   ParMesh pmesh(MPI_COMM_WORLD, *mesh);
-   delete mesh;
-   for (int l = 0; l < par_ref_levels; l++)
-   {
-      pmesh.UniformRefinement();
-   }
-
-   // 5. Define a parallel finite element space on the parallel mesh. Here we
-   //    use either continuous Lagrange finite elements or discontinuous
-   //    Galerkin finite elements of the specified order.
-   FiniteElementCollection *fec =
-      h1 ? (FiniteElementCollection*)new H1_FECollection(order, dim) :
-      (FiniteElementCollection*)new DG_FECollection(order, dim);
-   ParFiniteElementSpace fespace(&pmesh, fec);
-   HYPRE_Int size = fespace.GlobalTrueVSize();
-   mfem::out << "Number of finite element unknowns: " << size << endl;
-
-   // 6. Create "marker arrays" to define the portions of the boundary
-   //    associated with each type of boundary condition.  These arrays
-   //    have an entry corresponding to each boundary attribute.
-   //    Placing a '1' in entry i marks attribute i+1 as being
-   //    active, '0' is inactive.
-   Array<int> nbc_bdr(pmesh.bdr_attributes.Max());
-   Array<int> rbc_bdr(pmesh.bdr_attributes.Max());
-   Array<int> dbc_bdr(pmesh.bdr_attributes.Max());
-
-   nbc_bdr = 0; nbc_bdr[0] = 1;
-   rbc_bdr = 0; rbc_bdr[1] = 1;
-   dbc_bdr = 0; dbc_bdr[2] = 1;
-
-   Array<int> ess_tdof_list(0);
-   if (h1 && pmesh.bdr_attributes.Size())
-   {
-      // For a continuous basis the linear system must be modifed to enforce
-      // an essential (Dirichlet) boundary condition.  In the DG case this is
-      // not necessary as the boundary condition will only be enforced weakly.
-      fespace.GetEssentialTrueDofs(dbc_bdr, ess_tdof_list);
-   }
-
-   // 7. Setup the various coefficients needed for the Laplace operator and
-   //    the various boundary conditions.  In general these coefficients could
-   //    be functions of position but here we use only constants.
-   ConstantCoefficient matCoef(mat_val);
-   ConstantCoefficient dbcCoef(dbc_val);
-   ConstantCoefficient nbcCoef(nbc_val);
-   ConstantCoefficient rbcACoef(rbc_a_val);
-   ConstantCoefficient rbcBCoef(rbc_b_val);
-
-   // Since the n.Grad(u) terms arise by integrating -Div(m Grad(u)) by parts
-   // we must introduce the coefficient 'm' into the boundary conditions.
-   // Therefore, in the case of the Neumann BC, we actually enforce
-   // m n.Grad(u) = m g rather than simply n.Grad(u) = g.
-   ProductCoefficient m_nbcCoef(matCoef, nbcCoef);
-   ProductCoefficient m_rbcACoef(matCoef, rbcACoef);
-   ProductCoefficient m_rbcBCoef(matCoef, rbcBCoef);
-
-   // 8. Define the solution vector u as a parallel finite element grid function
-   //    corresponding to fespace. Initialize u with initial guess of zero.
-   ParGridFunction u(&fespace);
-   u = 0.0;
-
-   // 9. Set up the parallel bilinear form a(.,.) on the finite element space
-   //    corresponding to the Laplacian operator -Delta, by adding the Diffusion
-   //    domain integrator.
-   ParBilinearForm a(&fespace);
-   a.AddDomainIntegrator(new DiffusionIntegrator(matCoef));
-   if (h1)
-   {
-      // Add a Mass integrator on the Robin boundary
-      a.AddBoundaryIntegrator(new MassIntegrator(m_rbcACoef), rbc_bdr);
-   }
-   else
-   {
-      // Add the interfacial portion of the Lapalce operator
-      a.AddInteriorFaceIntegrator(new DGDiffusionIntegrator(matCoef,
-                                                            sigma, kappa));
-
-      // Counteract the n.Grad(u) term on the Dirichlet portion of the boundary
-      a.AddBdrFaceIntegrator(new DGDiffusionIntegrator(matCoef, sigma, kappa),
-                             dbc_bdr);
-
-      // Augment the n.Grad(u) term with a*u on the Robin portion of boundary
-      a.AddBdrFaceIntegrator(new BoundaryMassIntegrator(m_rbcACoef),
-                             rbc_bdr);
-   }
-   a.Assemble();
-
-   // 10. Assemble the parallel linear form for the right hand side vector.
-   ParLinearForm b(&fespace);
-
-   if (h1)
-   {
-      // Set the Dirchlet values in the solution vector
-      u.ProjectBdrCoefficient(dbcCoef, dbc_bdr);
-
-      // Add the desired value for n.Grad(u) on the Neumann boundary
-      b.AddBoundaryIntegrator(new BoundaryLFIntegrator(m_nbcCoef), nbc_bdr);
-
-      // Add the desired value for n.Grad(u) + a*u on the Robin boundary
-      b.AddBoundaryIntegrator(new BoundaryLFIntegrator(m_rbcBCoef), rbc_bdr);
-   }
-   else
-   {
-      // Add the desired value for the Dirchlet boundary
-      b.AddBdrFaceIntegrator(new DGDirichletLFIntegrator(dbcCoef, matCoef,
-                                                         sigma, kappa),
-                             dbc_bdr);
-
-      // Add the desired value for n.Grad(u) on the Neumann boundary
-      b.AddBdrFaceIntegrator(new BoundaryLFIntegrator(m_nbcCoef),
-                             nbc_bdr);
-
-      // Add the desired value for n.Grad(u) + a*u on the Robin boundary
-      b.AddBdrFaceIntegrator(new BoundaryLFIntegrator(m_rbcBCoef),
-                             rbc_bdr);
-   }
-   b.Assemble();
-
-   // 11. Construct the linear system.
-   OperatorPtr A;
-   Vector B, X;
-   a.FormLinearSystem(ess_tdof_list, u, b, A, X, B);
-
-   // 12. Solve the linear system A X = B.
-   HypreSolver *amg = new HypreBoomerAMG;
-   if (h1 || sigma == -1.0)
-   {
-      HyprePCG pcg(MPI_COMM_WORLD);
-      pcg.SetTol(1e-12);
-      pcg.SetMaxIter(200);
-      pcg.SetPrintLevel(2);
-      pcg.SetPreconditioner(*amg);
-      pcg.SetOperator(*A);
-      pcg.Mult(B, X);
-   }
-   else
-   {
-      GMRESSolver gmres(MPI_COMM_WORLD);
-      gmres.SetAbsTol(0.0);
-      gmres.SetRelTol(1e-12);
-      gmres.SetMaxIter(200);
-      gmres.SetKDim(10);
-      gmres.SetPrintLevel(1);
-      gmres.SetPreconditioner(*amg);
-      gmres.SetOperator(*A);
-      gmres.Mult(B, X);
-   }
-   delete amg;
-
-   // 13. Recover the parallel grid function corresponding to U. This is the
-   //     local finite element solution on each processor.
-   a.RecoverFEMSolution(X, b, u);
-
-   // 14. Build a mass matrix to help solve for n.Grad(u) where 'n' is
-   //     a surface normal.
-   ParBilinearForm m(&fespace);
-   m.AddDomainIntegrator(new MassIntegrator);
-   m.Assemble();
-
-   ess_tdof_list.SetSize(0);
-   OperatorPtr M;
-   m.FormSystemMatrix(ess_tdof_list, M);
-
-   // 15. Compute the various boundary integrals.
-   mfem::out << endl
-             << "Verifying boundary conditions" << endl
-             << "=============================" << endl;
-   {
-      // Integrate the solution on the Dirichlet boundary and compare
-      // to the expected value.
-      double err, avg = IntegrateBC(u, dbc_bdr, 0.0, 1.0, dbc_val, err);
-
-      bool hom_dbc = (dbc_val == 0.0);
-      err /=  hom_dbc ? 1.0 : fabs(dbc_val);
-      mfem::out << "Average of solution on Gamma_dbc:\t"
-                << avg << ", \t"
-                << (hom_dbc ? "absolute" : "relative")
-                << " error " << err << endl;
-   }
-   {
-      // Integrate n.Grad(u) on the inhomogeneous Neumann boundary and
-      // compare to the expected value.
-      double err, avg = IntegrateBC(u, nbc_bdr, 1.0, 0.0, nbc_val, err);
-
-      bool hom_nbc = (nbc_val == 0.0);
-      err /=  hom_nbc ? 1.0 : fabs(nbc_val);
-      mfem::out << "Average of n.Grad(u) on Gamma_nbc:\t"
-                << avg << ", \t"
-                << (hom_nbc ? "absolute" : "relative")
-                << " error " << err << endl;
-   }
-   {
-      // Integrate n.Grad(u) on the homogeneous Neumann boundary and compare
-      // to the expected value of zero.
-      Array<int> nbc0_bdr(pmesh.bdr_attributes.Max());
-      nbc0_bdr = 0;
-      nbc0_bdr[3] = 1;
-
-      double err, avg = IntegrateBC(u, nbc0_bdr, 1.0, 0.0, 0.0, err);
-
-      bool hom_nbc = true;
-      mfem::out << "Average of n.Grad(u) on Gamma_nbc0:\t"
-                << avg << ", \t"
-                << (hom_nbc ? "absolute" : "relative")
-                << " error " << err << endl;
-   }
-   {
-      // Integrate n.Grad(u) + a * u on the Robin boundary and compare to
-      // the expected value.
-      double err, avg = IntegrateBC(u, rbc_bdr, 1.0, rbc_a_val, rbc_b_val, err);
-
-      bool hom_rbc = (rbc_b_val == 0.0);
-      err /=  hom_rbc ? 1.0 : fabs(rbc_b_val);
-      mfem::out << "Average of n.Grad(u)+a*u on Gamma_rbc:\t"
-                << avg << ", \t"
-                << (hom_rbc ? "absolute" : "relative")
-                << " error " << err << endl;
-   }
-
-   // 16. Save the refined mesh and the solution in parallel. This output can
-   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
-   {
-      ostringstream mesh_name, sol_name;
-      mesh_name << "mesh." << setfill('0') << setw(6) << mpi.WorldRank();
-      sol_name << "sol." << setfill('0') << setw(6) << mpi.WorldRank();
-
-      ofstream mesh_ofs(mesh_name.str().c_str());
-      mesh_ofs.precision(8);
-      pmesh.Print(mesh_ofs);
-
-      ofstream sol_ofs(sol_name.str().c_str());
-      sol_ofs.precision(8);
-      u.Save(sol_ofs);
-   }
-
-   // 17. Send the solution by socket to a GLVis server.
-   if (visualization)
-   {
-      string title_str = h1 ? "H1" : "DG";
-      char vishost[] = "localhost";
-      int  visport   = 19916;
-      socketstream sol_sock(vishost, visport);
-      sol_sock << "parallel " << mpi.WorldSize()
-               << " " << mpi.WorldRank() << "\n";
-      sol_sock.precision(8);
-      sol_sock << "solution\n" << pmesh << u
-               << "window_title '" << title_str << " Solution'"
-               << " keys 'mmc'" << flush;
-   }
-
-   // 18. Free the used memory.
-   delete fec;
-
-   return 0;
-}
-
-void quad_trans(double u, double v, double &x, double &y, bool log = false)
-{
-   double a = a_; // Radius of disc
-
-   double d = 4.0 * a * (M_SQRT2 - 2.0 * a) * (1.0 - 2.0 * v);
-
-   double v0 = (1.0 + M_SQRT2) * (M_SQRT2 * a - 2.0 * v) *
-               ((4.0 - 3 * M_SQRT2) * a +
-                (8.0 * (M_SQRT2 - 1.0) * a - 2.0) * v) / d;
-
-   double r = 2.0 * ((M_SQRT2 - 1.0) * a * a * (1.0 - 4.0 *v) +
-                     2.0 * (1.0 + M_SQRT2 *
-                            (1.0 + 2.0 * (2.0 * a - M_SQRT2 - 1.0) * a)) * v * v
-                    ) / d;
-
-   double t = asin(v / r) * u / v;
-   if (log)
-   {
-      mfem::out << "u, v, r, v0, t "
-                << u << " " << v << " " << r << " " << v0 << " " << t
-                << endl;
-   }
-   x = r * sin(t);
-   y = r * cos(t) - v0;
-}
-
-void trans(const Vector &u, Vector &x)
-{
-   double tol = 1e-4;
-
-   if (u[1] > 0.5 - tol || u[1] < -0.5 + tol)
-   {
-      x = u;
-      return;
-   }
-   if (u[0] > 1.0 - tol || u[0] < -1.0 + tol || fabs(u[0]) < tol)
-   {
-      x = u;
-      return;
-   }
-
-   if (u[0] > 0.0)
-   {
-      if (u[1] > fabs(u[0] - 0.5))
-      {
-         quad_trans(u[0] - 0.5, u[1], x[0], x[1]);
-         x[0] += 0.5;
-         return;
-      }
-      if (u[1] < -fabs(u[0] - 0.5))
-      {
-         quad_trans(u[0] - 0.5, -u[1], x[0], x[1]);
-         x[0] += 0.5;
-         x[1] *= -1.0;
-         return;
-      }
-      if (u[0] - 0.5 > fabs(u[1]))
-      {
-         quad_trans(u[1], u[0] - 0.5, x[1], x[0]);
-         x[0] += 0.5;
-         return;
-      }
-      if (u[0] - 0.5 < -fabs(u[1]))
-      {
-         quad_trans(u[1], 0.5 - u[0], x[1], x[0]);
-         x[0] *= -1.0;
-         x[0] += 0.5;
-         return;
-      }
-   }
-   else
-   {
-      if (u[1] > fabs(u[0] + 0.5))
-      {
-         quad_trans(u[0] + 0.5, u[1], x[0], x[1]);
-         x[0] -= 0.5;
-         return;
-      }
-      if (u[1] < -fabs(u[0] + 0.5))
-      {
-         quad_trans(u[0] + 0.5, -u[1], x[0], x[1]);
-         x[0] -= 0.5;
-         x[1] *= -1.0;
-         return;
-      }
-      if (u[0] + 0.5 > fabs(u[1]))
-      {
-         quad_trans(u[1], u[0] + 0.5, x[1], x[0]);
-         x[0] -= 0.5;
-         return;
-      }
-      if (u[0] + 0.5 < -fabs(u[1]))
-      {
-         quad_trans(u[1], -0.5 - u[0], x[1], x[0]);
-         x[0] *= -1.0;
-         x[0] -= 0.5;
-         return;
-      }
-   }
-   x = u;
-}
-
-Mesh * GenerateSerialMesh(int ref)
-{
-   Mesh * mesh = new Mesh(2, 29, 16, 24, 2);
-
-   int vi[4];
-
-   for (int i=0; i<2; i++)
-   {
-      int o = 13 * i;
-      vi[0] = o + 0; vi[1] = o + 3; vi[2] = o + 4; vi[3] = o + 1;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 1; vi[1] = o + 4; vi[2] = o + 5; vi[3] = o + 2;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 5; vi[1] = o + 8; vi[2] = o + 9; vi[3] = o + 2;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 8; vi[1] = o + 12; vi[2] = o + 15; vi[3] = o + 9;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 11; vi[1] = o + 14; vi[2] = o + 15; vi[3] = o + 12;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 10; vi[1] = o + 13; vi[2] = o + 14; vi[3] = o + 11;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 6; vi[1] = o + 13; vi[2] = o + 10; vi[3] = o + 7;
-      mesh->AddQuad(vi);
-
-      vi[0] = o + 0; vi[1] = o + 6; vi[2] = o + 7; vi[3] = o + 3;
-      mesh->AddQuad(vi);
-   }
-
-   vi[0] =  0; vi[1] =  6; mesh->AddBdrSegment(vi, 1);
-   vi[0] =  6; vi[1] = 13; mesh->AddBdrSegment(vi, 1);
-   vi[0] = 13; vi[1] = 19; mesh->AddBdrSegment(vi, 1);
-   vi[0] = 19; vi[1] = 26; mesh->AddBdrSegment(vi, 1);
-
-   vi[0] = 28; vi[1] = 22; mesh->AddBdrSegment(vi, 2);
-   vi[0] = 22; vi[1] = 15; mesh->AddBdrSegment(vi, 2);
-   vi[0] = 15; vi[1] =  9; mesh->AddBdrSegment(vi, 2);
-   vi[0] =  9; vi[1] =  2; mesh->AddBdrSegment(vi, 2);
-
-   for (int i=0; i<2; i++)
-   {
-      int o = 13 * i;
-      vi[0] = o +  7; vi[1] = o +  3; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o + 10; vi[1] = o +  7; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o + 11; vi[1] = o + 10; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o + 12; vi[1] = o + 11; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o +  8; vi[1] = o + 12; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o +  5; vi[1] = o +  8; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o +  4; vi[1] = o +  5; mesh->AddBdrSegment(vi, 3 + i);
-      vi[0] = o +  3; vi[1] = o +  4; mesh->AddBdrSegment(vi, 3 + i);
-   }
-
-   double d[2];
-   double a = a_ / M_SQRT2;
-
-   d[0] = -1.0; d[1] = -0.5; mesh->AddVertex(d);
-   d[0] = -1.0; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] = -1.0; d[1] =  0.5; mesh->AddVertex(d);
-
-   d[0] = -0.5 - a; d[1] =   -a; mesh->AddVertex(d);
-   d[0] = -0.5 - a; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] = -0.5 - a; d[1] =    a; mesh->AddVertex(d);
-
-   d[0] = -0.5; d[1] = -0.5; mesh->AddVertex(d);
-   d[0] = -0.5; d[1] =   -a; mesh->AddVertex(d);
-   d[0] = -0.5; d[1] =    a; mesh->AddVertex(d);
-   d[0] = -0.5; d[1] =  0.5; mesh->AddVertex(d);
-
-   d[0] = -0.5 + a; d[1] =   -a; mesh->AddVertex(d);
-   d[0] = -0.5 + a; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] = -0.5 + a; d[1] =    a; mesh->AddVertex(d);
-
-   d[0] =  0.0; d[1] = -0.5; mesh->AddVertex(d);
-   d[0] =  0.0; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] =  0.0; d[1] =  0.5; mesh->AddVertex(d);
-
-   d[0] =  0.5 - a; d[1] =   -a; mesh->AddVertex(d);
-   d[0] =  0.5 - a; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] =  0.5 - a; d[1] =    a; mesh->AddVertex(d);
-
-   d[0] =  0.5; d[1] = -0.5; mesh->AddVertex(d);
-   d[0] =  0.5; d[1] =   -a; mesh->AddVertex(d);
-   d[0] =  0.5; d[1] =    a; mesh->AddVertex(d);
-   d[0] =  0.5; d[1] =  0.5; mesh->AddVertex(d);
-
-   d[0] =  0.5 + a; d[1] =   -a; mesh->AddVertex(d);
-   d[0] =  0.5 + a; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] =  0.5 + a; d[1] =    a; mesh->AddVertex(d);
-
-   d[0] =  1.0; d[1] = -0.5; mesh->AddVertex(d);
-   d[0] =  1.0; d[1] =  0.0; mesh->AddVertex(d);
-   d[0] =  1.0; d[1] =  0.5; mesh->AddVertex(d);
-
-   mesh->FinalizeTopology();
-
-   mesh->SetCurvature(1, true);
-
-   // Stitch the ends of the stack together
-   {
-      Array<int> v2v(mesh->GetNV());
-      for (int i = 0; i < v2v.Size() - 3; i++)
-      {
-         v2v[i] = i;
-      }
-      // identify vertices on the narrow ends of the rectangle
-      v2v[v2v.Size() - 3] = 0;
-      v2v[v2v.Size() - 2] = 1;
-      v2v[v2v.Size() - 1] = 2;
-
-      // renumber elements
-      for (int i = 0; i < mesh->GetNE(); i++)
-      {
-         Element *el = mesh->GetElement(i);
-         int *v = el->GetVertices();
-         int nv = el->GetNVertices();
-         for (int j = 0; j < nv; j++)
-         {
-            v[j] = v2v[v[j]];
-         }
-      }
-      // renumber boundary elements
-      for (int i = 0; i < mesh->GetNBE(); i++)
-      {
-         Element *el = mesh->GetBdrElement(i);
-         int *v = el->GetVertices();
-         int nv = el->GetNVertices();
-         for (int j = 0; j < nv; j++)
-         {
-            v[j] = v2v[v[j]];
-         }
-      }
-      mesh->RemoveUnusedVertices();
-      mesh->RemoveInternalBoundaries();
-   }
-   mesh->SetCurvature(3, true);
-
-   for (int l = 0; l < ref; l++)
-   {
-      mesh->UniformRefinement();
-   }
-
-   mesh->Transform(trans);
-
-   return mesh;
-}
-
-double IntegrateBC(const ParGridFunction &x, const Array<int> &bdr,
-                   double alpha, double beta, double gamma,
-                   double &glb_err)
-{
-   double loc_vals[3];
-   double &nrm = loc_vals[0];
-   double &avg = loc_vals[1];
-   double &err = loc_vals[2];
-
-   nrm = 0.0;
-   avg = 0.0;
-   err = 0.0;
-
-   const bool a_is_zero = alpha == 0.0;
-   const bool b_is_zero = beta == 0.0;
-
-   const ParFiniteElementSpace &fes = *x.ParFESpace();
-   MFEM_ASSERT(fes.GetVDim() == 1, "");
-   ParMesh &mesh = *fes.GetParMesh();
-   Vector shape, loc_dofs, w_nor;
-   DenseMatrix dshape;
-   Array<int> dof_ids;
-   for (int i = 0; i < mesh.GetNBE(); i++)
-   {
-      if (bdr[mesh.GetBdrAttribute(i)-1] == 0) { continue; }
-
-      FaceElementTransformations *FTr = mesh.GetBdrFaceTransformations(i);
-      if (FTr == nullptr) { continue; }
-
-      const FiniteElement &fe = *fes.GetFE(FTr->Elem1No);
-      MFEM_ASSERT(fe.GetMapType() == FiniteElement::VALUE, "");
-      const int int_order = 2*fe.GetOrder() + 3;
-      const IntegrationRule &ir = IntRules.Get(FTr->FaceGeom, int_order);
-
-      fes.GetElementDofs(FTr->Elem1No, dof_ids);
-      x.GetSubVector(dof_ids, loc_dofs);
-      if (!a_is_zero)
-      {
-         const int sdim = FTr->Face->GetSpaceDim();
-         w_nor.SetSize(sdim);
-         dshape.SetSize(fe.GetDof(), sdim);
-      }
-      if (!b_is_zero)
-      {
-         shape.SetSize(fe.GetDof());
-      }
-      for (int j = 0; j < ir.GetNPoints(); j++)
-      {
-         const IntegrationPoint &ip = ir.IntPoint(j);
-         IntegrationPoint eip;
-         FTr->Loc1.Transform(ip, eip);
-         FTr->Face->SetIntPoint(&ip);
-         double face_weight = FTr->Face->Weight();
-         double val = 0.0;
-         if (!a_is_zero)
-         {
-            FTr->Elem1->SetIntPoint(&eip);
-            fe.CalcPhysDShape(*FTr->Elem1, dshape);
-            CalcOrtho(FTr->Face->Jacobian(), w_nor);
-            val += alpha * dshape.InnerProduct(w_nor, loc_dofs) / face_weight;
-         }
-         if (!b_is_zero)
-         {
-            fe.CalcShape(eip, shape);
-            val += beta * (shape * loc_dofs);
-         }
-
-         // Measure the length of the boundary
-         nrm += ip.weight * face_weight;
-
-         // Integrate alpha * n.Grad(x) + beta * x
-         avg += val * ip.weight * face_weight;
-
-         // Integrate |alpha * n.Grad(x) + beta * x - gamma|^2
-         val -= gamma;
-         err += (val*val) * ip.weight * face_weight;
-      }
-   }
-
-   double glb_vals[3];
-   MPI_Allreduce(loc_vals, glb_vals, 3, MPI_DOUBLE, MPI_SUM, fes.GetComm());
-
-   double glb_nrm = glb_vals[0];
-   double glb_avg = glb_vals[1];
-   glb_err = glb_vals[2];
-
-   // Normalize by the length of the boundary
-   if (std::abs(glb_nrm) > 0.0)
-   {
-      glb_err /= glb_nrm;
-      glb_avg /= glb_nrm;
-   }
-
-   // Compute l2 norm of the error in the boundary condition
-   // (negative quadrature weights may produce negative 'err')
-   glb_err = (glb_err >= 0.0) ? sqrt(glb_err) : -sqrt(-glb_err);
-
-   // Return the average value of alpha * n.Grad(x) + beta * x
-   return glb_avg;
-}
@@ -274,13 +274,6 @@ int main(int argc, char *argv[])
      pmesh->SetNodalFESpace(fespace);
   }

-   {
-      x.Save("ex2p.gf", 1);
-      ParGridFunction new_x(fespace, "ex2p.gf");
-      new_x -= x;
-      out << "GF difference: " << new_x.Norml1() << endl;
-   }
-
   // 16. Save in parallel the displaced mesh and the inverted solution (which
   //     gives the backward displacements to the original grid). This output
   //     can be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
@@ -22,8 +22,6 @@
 //               The example demonstrates the use of the BlockMatrix class, as
 //               well as the collective saving of several grid functions in
 //               VisIt (visit.llnl.gov) and ParaView (paraview.org) formats.
-//               Optional saving with ADIOS2 (adios2.readthedocs.io) streams is
-//               also illustrated.
 //
 //               We recommend viewing examples 1-4 before viewing this example.

@@ -57,7 +55,6 @@ int main(int argc, char *argv[])
   int order = 1;
   bool par_format = false;
   bool visualization = 1;
-   bool adios2 = false;

   OptionsParser args(argc, argv);
   args.AddOption(&mesh_file, "-m", "--mesh",
@@ -70,9 +67,6 @@ int main(int argc, char *argv[])
   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                  "--no-visualization",
                  "Enable or disable GLVis visualization.");
-   args.AddOption(&adios2, "-adios2", "--adios2-streams", "-no-adios2",
-                  "--no-adios2-streams",
-                  "Save data using adios2 streams.");
   args.Parse();
   if (!args.Good())
   {
@@ -343,27 +337,7 @@ int main(int argc, char *argv[])
   paraview_dc.RegisterField("pressure",p);
   paraview_dc.Save();

-   // 17. Optionally output a BP (binary pack) file using ADIOS2. This can be
-   //     visualized with the ParaView VTX reader.
-#ifdef MFEM_USE_ADIOS2
-   if (adios2)
-   {
-      std::string postfix(mesh_file);
-      postfix.erase(0, std::string("../data/").size() );
-      postfix += "_o" + std::to_string(order);
-      const std::string collection_name = "ex5-p_" + postfix + ".bp";
-
-      ADIOS2DataCollection adios2_dc(MPI_COMM_WORLD, collection_name, pmesh);
-      adios2_dc.SetLevelsOfDetail(1);
-      adios2_dc.SetCycle(1);
-      adios2_dc.SetTime(0.0);
-      adios2_dc.RegisterField("velocity",u);
-      adios2_dc.RegisterField("pressure",p);
-      adios2_dc.Save();
-   }
-#endif
-
-   // 18. Send the solution by socket to a GLVis server.
+   // 17. Send the solution by socket to a GLVis server.
   if (visualization)
   {
      char vishost[] = "localhost";
@@ -383,7 +357,7 @@ int main(int argc, char *argv[])
             << endl;
   }

-   // 19. Free the used memory.
+   // 18. Free the used memory.
   delete fform;
   delete gform;
   delete u;
@@ -279,11 +279,4 @@ void SnapNodes(Mesh &mesh)
         nodes(nodes.FESpace()->DofToVDof(i, d)) = node(d);
      }
   }
-   if (mesh.Nonconforming())
-   {
-      // Snap hanging nodes to the master side.
-      Vector tnodes;
-      nodes.GetTrueDofs(tnodes);
-      nodes.SetFromTrueDofs(tnodes);
-   }
 }
@@ -348,11 +348,4 @@ void SnapNodes(Mesh &mesh)
         nodes(nodes.FESpace()->DofToVDof(i, d)) = node(d);
      }
   }
-   if (mesh.Nonconforming())
-   {
-      // Snap hanging nodes to the master side.
-      Vector tnodes;
-      nodes.GetTrueDofs(tnodes);
-      nodes.SetFromTrueDofs(tnodes);
-   }
 }
@@ -31,10 +31,9 @@
 //               and explicit ODE time integrators, the definition of periodic
 //               boundary conditions through periodic meshes, as well as the use
 //               of GLVis for persistent visualization of a time-evolving
-//               solution. Saving of time-dependent data files for visualization
-//               with VisIt (visit.llnl.gov) and ParaView (paraview.org), as
-//               well as the optional saving with ADIOS2 (adios2.readthedocs.io)
-//               are also illustrated.
+//               solution. The saving of time-dependent data files for external
+//               visualization with VisIt (visit.llnl.gov) and ParaView
+//               (paraview.org) is also illustrated.

 #include "mfem.hpp"
 #include <fstream>
@@ -168,7 +167,6 @@ int main(int argc, char *argv[])
   bool visualization = true;
   bool visit = false;
   bool paraview = false;
-   bool adios2 = false;
   bool binary = false;
   int vis_steps = 5;

@@ -210,9 +208,6 @@ int main(int argc, char *argv[])
   args.AddOption(&paraview, "-paraview", "--paraview-datafiles", "-no-paraview",
                  "--no-paraview-datafiles",
                  "Save data files for ParaView (paraview.org) visualization.");
-   args.AddOption(&adios2, "-adios2", "--adios2-streams", "-no-adios2",
-                  "--no-adios2-streams",
-                  "Save data using adios2 streams.");
   args.AddOption(&binary, "-binary", "--binary-datafiles", "-ascii",
                  "--ascii-datafiles",
                  "Use binary (Sidre) or ascii format for VisIt data files.");
@@ -399,28 +394,6 @@ int main(int argc, char *argv[])
      pd->Save();
   }

-   // Optionally output a BP (binary pack) file using ADIOS2. This can be
-   // visualized with the ParaView VTX reader.
-#ifdef MFEM_USE_ADIOS2
-   ADIOS2DataCollection *adios2_dc = NULL;
-   if (adios2)
-   {
-      std::string postfix(mesh_file);
-      postfix.erase(0, std::string("../data/").size() );
-      postfix += "_o" + std::to_string(order);
-      const std::string collection_name = "ex9-p-" + postfix + ".bp";
-
-      adios2_dc = new ADIOS2DataCollection(MPI_COMM_WORLD, collection_name, pmesh);
-      // output data substreams are half the number of mpi processes
-      adios2_dc->SetParameter("SubStreams", std::to_string(num_procs/2) );
-      // adios2_dc->SetLevelsOfDetail(2);
-      adios2_dc->RegisterField("solution", u);
-      adios2_dc->SetCycle(0);
-      adios2_dc->SetTime(0.0);
-      adios2_dc->Save();
-   }
-#endif
-
   socketstream sout;
   if (visualization)
   {
@@ -499,16 +472,6 @@ int main(int argc, char *argv[])
            pd->SetTime(t);
            pd->Save();
         }
-
-#ifdef MFEM_USE_ADIOS2
-         // transient solutions can be visualized with ParaView
-         if (adios2)
-         {
-            adios2_dc->SetCycle(ti);
-            adios2_dc->SetTime(t);
-            adios2_dc->Save();
-         }
-#endif
      }
   }

@@ -534,12 +497,6 @@ int main(int argc, char *argv[])
   delete pmesh;
   delete ode_solver;
   delete pd;
-#ifdef MFEM_USE_ADIOS2
-   if (adios2)
-   {
-      delete adios2_dc;
-   }
-#endif
   delete dc;

   MPI_Finalize();
@@ -1,292 +0,0 @@
-//                       MFEM Example 1 - Parallel Version
-//
-// Compile with: make ex1p
-//
-// Sample runs:  mpirun -np 4 ex1p -m ../data/square-disc.mesh
-//               mpirun -np 4 ex1p -m ../data/star.mesh
-//               mpirun -np 4 ex1p -m ../data/star-mixed.mesh
-//               mpirun -np 4 ex1p -m ../data/escher.mesh
-//               mpirun -np 4 ex1p -m ../data/fichera.mesh
-//               mpirun -np 4 ex1p -m ../data/fichera-mixed.mesh
-//               mpirun -np 4 ex1p -m ../data/toroid-wedge.mesh
-//               mpirun -np 4 ex1p -m ../data/square-disc-p2.vtk -o 2
-//               mpirun -np 4 ex1p -m ../data/square-disc-p3.mesh -o 3
-//               mpirun -np 4 ex1p -m ../data/square-disc-nurbs.mesh -o -1
-//               mpirun -np 4 ex1p -m ../data/star-mixed-p2.mesh -o 2
-//               mpirun -np 4 ex1p -m ../data/disc-nurbs.mesh -o -1
-//               mpirun -np 4 ex1p -m ../data/pipe-nurbs.mesh -o -1
-//               mpirun -np 4 ex1p -m ../data/ball-nurbs.mesh -o 2
-//               mpirun -np 4 ex1p -m ../data/fichera-mixed-p2.mesh -o 2
-//               mpirun -np 4 ex1p -m ../data/star-surf.mesh
-//               mpirun -np 4 ex1p -m ../data/square-disc-surf.mesh
-//               mpirun -np 4 ex1p -m ../data/inline-segment.mesh
-//               mpirun -np 4 ex1p -m ../data/amr-quad.mesh
-//               mpirun -np 4 ex1p -m ../data/amr-hex.mesh
-//               mpirun -np 4 ex1p -m ../data/mobius-strip.mesh
-//               mpirun -np 4 ex1p -m ../data/mobius-strip.mesh -o -1 -sc
-//
-// Device sample runs:
-//               mpirun -np 4 ex1p -pa -d cuda
-//               mpirun -np 4 ex1p -pa -d occa-cuda
-//               mpirun -np 4 ex1p -pa -d raja-omp
-//               mpirun -np 4 ex1p -pa -d ceed-cpu
-//               mpirun -np 4 ex1p -pa -d ceed-cuda
-//               mpirun -np 4 ex1p -m ../data/beam-tet.mesh -pa -d ceed-cpu
-//
-// Description:  This example code demonstrates the use of MFEM to define a
-//               simple finite element discretization of the Laplace problem
-//               -Delta u = 1 with homogeneous Dirichlet boundary conditions.
-//               Specifically, we discretize using a FE space of the specified
-//               order, or if order < 1 using an isoparametric/isogeometric
-//               space (i.e. quadratic for quadratic curvilinear mesh, NURBS for
-//               NURBS mesh, etc.)
-//
-//               The example highlights the use of mesh refinement, finite
-//               element grid functions, as well as linear and bilinear forms
-//               corresponding to the left-hand side and right-hand side of the
-//               discrete linear system. We also cover the explicit elimination
-//               of essential boundary conditions, static condensation, and the
-//               optional connection to the GLVis tool for visualization.
-
-#include "mfem.hpp"
-#include <fstream>
-#include <iostream>
-#include "mpi.h"
-
-using namespace std;
-using namespace mfem;
-
-int main(int argc, char *argv[])
-{
-   // 1. Initialize MPI.
-   int num_procs, myid;
-   MPI_Init(&argc, &argv);
-   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
-   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
-
-   // 2. Parse command-line options.
-   // const char *mesh_file = "../data/star.mesh";
-   const char *mesh_file = "../data/square-disc.mesh";
-   int order = 1;
-   bool static_cond = false;
-   bool pa = false;
-   const char *device_config = "cpu";
-   bool visualization = false;
-   int nfiles = 1;
-   // const char *out_file = "0_0.gf";
-
-   OptionsParser args(argc, argv);
-   args.AddOption(&mesh_file, "-m", "--mesh",
-                  "Mesh file to use.");
-   args.AddOption(&order, "-o", "--order",
-                  "Finite element order (polynomial degree) or -1 for"
-                  " isoparametric space.");
-   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
-                  "--no-static-condensation", "Enable static condensation.");
-   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
-                  "--no-partial-assembly", "Enable Partial Assembly.");
-   args.AddOption(&device_config, "-d", "--device",
-                  "Device configuration string, see Device::Configure().");
-   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
-                  "--no-visualization",
-                  "Enable or disable GLVis visualization.");
-   args.AddOption(&nfiles, "-nf", "--num-files", "Number of files to write.");
-   // args.AddOption(&out_file, "-o", "--outfile",
-   //                "Name of file to write.");
-
-   args.Parse();
-   if (!args.Good())
-   {
-      if (myid == 0)
-      {
-         args.PrintUsage(cout);
-      }
-      MPI_Finalize();
-      return 1;
-   }
-   if (myid == 0)
-   {
-      args.PrintOptions(cout);
-   }
-
-   // 3. Enable hardware devices such as GPUs, and programming models such as
-   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
-   Device device(device_config);
-   if (myid == 0) { device.Print(); }
-
-   // 4. Read the (serial) mesh from the given mesh file on all processors.  We
-   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
-   //    and volume meshes with the same code.
-   Mesh *mesh = new Mesh(mesh_file, 1, 1);
-   int dim = mesh->Dimension();
-
-   // 5. Refine the serial mesh on all processors to increase the resolution. In
-   //    this example we do 'ref_levels' of uniform refinement. We choose
-   //    'ref_levels' to be the largest number that gives a final mesh with no
-   //    more than 10,000 elements.
-   {
-      int ref_levels =
-         (int)floor(log(10000./mesh->GetNE())/log(2.)/dim);
-      for (int l = 0; l < ref_levels; l++)
-      {
-         mesh->UniformRefinement();
-      }
-   }
-
-   // 6. Define a parallel mesh by a partitioning of the serial mesh. Refine
-   //    this mesh further in parallel to increase the resolution. Once the
-   //    parallel mesh is defined, the serial mesh can be deleted.
-   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
-   delete mesh;
-   {
-      int par_ref_levels = 2;
-      for (int l = 0; l < par_ref_levels; l++)
-      {
-         pmesh->UniformRefinement();
-      }
-   }
-
-   // 7. Define a parallel finite element space on the parallel mesh. Here we
-   //    use continuous Lagrange finite elements of the specified order. If
-   //    order < 1, we instead use an isoparametric/isogeometric space.
-   FiniteElementCollection *fec;
-   if (order > 0)
-   {
-      fec = new H1_FECollection(order, dim);
-   }
-   else if (pmesh->GetNodes())
-   {
-      fec = pmesh->GetNodes()->OwnFEC();
-      if (myid == 0)
-      {
-         cout << "Using isoparametric FEs: " << fec->Name() << endl;
-      }
-   }
-   else
-   {
-      fec = new H1_FECollection(order = 1, dim);
-   }
-   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec, 1, 0);
-   HYPRE_Int size = fespace->GlobalTrueVSize();
-   if (myid == 0)
-   {
-      cout << "Number of finite element unknowns: " << size << endl;
-   }
-
-   // 8. Determine the list of true (i.e. parallel conforming) essential
-   //    boundary dofs. In this example, the boundary conditions are defined
-   //    by marking all the boundary attributes from the mesh as essential
-   //    (Dirichlet) and converting them to a list of true dofs.
-   Array<int> ess_tdof_list;
-   if (pmesh->bdr_attributes.Size())
-   {
-      Array<int> ess_bdr(pmesh->bdr_attributes.Max());
-      ess_bdr = 1;
-      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
-   }
-
-   // 9. Set up the parallel linear form b(.) which corresponds to the
-   //    right-hand side of the FEM linear system, which in this case is
-   //    (1,phi_i) where phi_i are the basis functions in fespace.
-   ParLinearForm *b = new ParLinearForm(fespace);
-   ConstantCoefficient one(1.0);
-   b->AddDomainIntegrator(new DomainLFIntegrator(one));
-   b->Assemble();
-
-   // 10. Define the solution vector x as a parallel finite element grid function
-   //     corresponding to fespace. Initialize x with initial guess of zero,
-   //     which satisfies the boundary conditions.
-   ParGridFunction x(fespace);
-   x = 0.0;
-
-   // 11. Set up the parallel bilinear form a(.,.) on the finite element space
-   //     corresponding to the Laplacian operator -Delta, by adding the Diffusion
-   //     domain integrator.
-   ParBilinearForm *a = new ParBilinearForm(fespace);
-   if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
-   a->AddDomainIntegrator(new DiffusionIntegrator(one));
-
-   // 12. Assemble the parallel bilinear form and the corresponding linear
-   //     system, applying any necessary transformations such as: parallel
-   //     assembly, eliminating boundary conditions, applying conforming
-   //     constraints for non-conforming AMR, static condensation, etc.
-   if (static_cond) { a->EnableStaticCondensation(); }
-   a->Assemble();
-
-   OperatorPtr A;
-   Vector B, X;
-   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
-
-   // 13. Solve the linear system A X = B.
-   //     * With full assembly, use the BoomerAMG preconditioner from hypre.
-   //     * With partial assembly, use Jacobi smoothing, for now.
-   Solver *prec = NULL;
-   if (pa)
-   {
-      if (UsesTensorBasis(*fespace))
-      {
-         prec = new OperatorJacobiSmoother(*a, ess_tdof_list);
-      }
-   }
-   else
-   {
-      prec = new HypreBoomerAMG;
-   }
-   CGSolver cg(MPI_COMM_WORLD);
-   cg.SetRelTol(1e-12);
-   cg.SetMaxIter(2000);
-   cg.SetPrintLevel(1);
-   if (prec) { cg.SetPreconditioner(*prec); }
-   cg.SetOperator(*A);
-   cg.Mult(B, X);
-   delete prec;
-
-   // 14. Recover the parallel grid function corresponding to X. This is the
-   //     local finite element solution on each processor.
-   a->RecoverFEMSolution(X, *b, x);
-
-   std::string filename = to_string(num_procs) + "_" + to_string(nfiles) + "_";
-   {
-      double t1;
-      t1 = MPI_Wtime();
-      x.Save(filename.c_str(), nfiles);
-      double t2 = MPI_Wtime();
-
-      double write_time = t2 - t1;
-      double average_write_time;
-      MPI_Reduce(&write_time, &average_write_time, 1,
-               MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
-      if (myid == 0)
-      {
-         std::cout << "Average write time: " << average_write_time / num_procs << " for "
-                   << nfiles << " files and "  << num_procs << " ranks\n";
-      }
-   }
-   {
-      double t1;
-      t1 = MPI_Wtime();
-      ParGridFunction temp_gf(fespace, filename.c_str());
-      double t2 = MPI_Wtime();
-
-      double read_time = t2 - t1;
-      double average_read_time;
-      MPI_Reduce(&read_time, &average_read_time, 1,
-                 MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
-      if (myid == 0)
-      {
-         std::cout << "Average read time: " << average_read_time / num_procs << " for "
-                   << nfiles << " files and "  << num_procs << " ranks\n";
-      }
-   }
-
-   // 17. Free the used memory.
-   delete a;
-   delete b;
-   delete fespace;
-   if (order > 0) { delete fec; }
-   delete pmesh;
-
-   MPI_Finalize();
-
-   return 0;
-}
@@ -22,10 +22,9 @@ MFEM_LIB_FILE = mfem_is_not_built
 -include $(CONFIG_MK)

 SEQ_EXAMPLES = ex1 ex2 ex3 ex4 ex5 ex6 ex7 ex8 ex9 ex10 ex14 ex15 ex16 ex17\
-  ex18 ex19 ex20 ex21 ex22 ex23 ex24 ex25 ex26 ex27
+  ex18 ex19 ex20 ex21 ex22 ex23 ex24 ex25
 PAR_EXAMPLES = ex1p ex2p ex3p ex4p ex5p ex6p ex7p ex8p ex9p ex10p ex11p ex12p\
-  ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex21p ex22p ex24p ex25p\
-  ex26p ex27p
+ ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex21p ex22p ex24p ex25p

 ifeq ($(MFEM_USE_MPI),NO)
   EXAMPLES = $(SEQ_EXAMPLES)
@@ -104,10 +103,6 @@ ex15-test-seq: ex15
 	@$(call mfem-test,$<,, Serial example,-e 1)
 ex15p-test-par: ex15p
 	@$(call mfem-test,$<, $(RUN_MPI), Parallel example,-e 1)
-ex27-test-seq: ex27
-	@$(call mfem-test,$<,, Serial example,-dg)
-ex27p-test-par: ex27p
-	@$(call mfem-test,$<, $(RUN_MPI), Parallel example,-dg)
 # Testing: optional tests
 ifeq ($(MFEM_USE_STRUMPACK),YES)
 ex11p-test-strumpack: ex11p
@@ -133,7 +128,6 @@ clean-exec:
 	@rm -f sphere_refined.* sol.* sol_u.* sol_p.* sol_r.* sol_i.*
 	@rm -f ex9.mesh ex9-mesh.* ex9-init.* ex9-final.*
 	@rm -f deformed.* velocity.* elastic_energy.* mode_*
-	@rm -f ex5-p-*.bp ex9-p-*.bp ex12-p-*.bp ex16-p-*.bp
 	@rm -f ex16.mesh ex16-mesh.* ex16-init.* ex16-final.*
 	@rm -f vortex-mesh.* vortex.mesh vortex-?-init.* vortex-?-final.*
 	@rm -f deformation.* pressure.*
@@ -1,906 +0,0 @@
-//                                MFEM Example 9
-//
-// Compile with: make serial_nogpu
-//
-// Description:  This code solves the time-dependent advection-diffusion
-//               equation:
-//               \frac(\partial u}{\partial t}
-//                                 = \mathbf{a} \cdot \Nabla u - \nu \Nabla^2 u
-//               where a is a given advection velocity, \nu is the diffusion
-//               parameter, and u0(x) = u(0,x) is a given initial condition.
-//
-//               The demonstrates explicit time marching with H1 elements of
-//               arbitrary order. Periodic boundary conditions are used through
-//               periodic meshes. GLVis can be used for visualization of a
-//               time-evolving solution.
-
-#include <fstream>
-#include <iostream>
-#include <algorithm>
-
-#include "mfem.hpp"
-#include "mpi.h"
-
-using namespace std;
-using namespace mfem;
-
-/** A time-dependent operator for the right-hand side of the ODE. The weak
-    form of du/dt = -a.grad(u) + nu Delta(u) is M du/dt = K u + b, where M and
-    K are the mass and advection-diffusion matrices, and b describes the flow
-    on the boundary. This can be written as a general ODE,
-    du/dt = M^{-1} (K u + b), and this class is used to evaluate the right-hand
-    side. */
-class AdvectionDiffusionEvolution : public mfem::TimeDependentOperator
-{
-public:
-   /// \param[in] M - bilinear form for mass matrix
-   /// \param[in] K - bilinear form for stiffness matrix
-   /// \param[in] b - load vector
-   AdvectionDiffusionEvolution(mfem::BilinearForm &M, mfem::BilinearForm &K,
-                               const mfem::Vector &b);
-
-   /// Perform the action of the operator: y = k = f(x, t), where k solves
-   /// Compute k = M^-1(Kx + l)
-   void Mult(const mfem::Vector &x, mfem::Vector &y) const override;
-
-   /// Solve the implicit equation: k = f(x + dt k, t), for the unknown k at
-   /// the current time t. 
-   void ImplicitSolve(const double dt, const mfem::Vector &x,
-                      mfem::Vector &k) override;
-
-   virtual ~AdvectionDiffusionEvolution();
-
-private:
-   mfem::BilinearForm &M, &K;
-   const mfem::Vector &b;
-   /// solver for inverting mass matrix for explicit time-marching
-   std::unique_ptr<mfem::Solver> M_prec;
-   mfem::CGSolver M_solver;
-   /// solver for implicit time-marching
-   mfem::GSSmoother prec;
-   mfem::GMRESSolver linear_solver;
-   mfem::NewtonSolver newton;
-
-   mutable mfem::Vector z;
-
-   /// pointer-to-implementation idiom
-   /// Hides implementation details of this operator
-   class SystemOperator;
-   /// Operator that combines the linear spatial discretization with
-   /// the load vector into one operator used for implicit solves
-   std::unique_ptr<SystemOperator> combined_oper;
-
-   /// sets the state and dt for the combined operator
-   /// \param[in] dt - time increment
-   /// \param[in] x - the current state
-   void setOperParameters(double dt, const mfem::Vector *x);
-
-};
-
-class PAJacobianOperator : public mfem::Operator
-{
-public:
-   PAJacobianOperator(mfem::ParBilinearForm &_mass,
-                      mfem::ParBilinearForm &_stiff);
-
-   /// Compute r = J@k = M@k + dt*K@k
-   /// \param[in] k - dx/dt
-   /// \param[out] r - J@k = M@k + dt*K@k
-   void Mult(const mfem::Vector &k, mfem::Vector &r) const override;
-      
-   /// Set current dt values - needed to compute action of Jacobian.
-   void setParameters(double dt);
-
-private:
-   mfem::ParBilinearForm &mass;
-   mfem::ParBilinearForm &stiff;
-
-   double dt;
-};
-
-class ParSystemOperator : public mfem::Operator
-{
-public:
-   /// Nonlinear operator of the form that combines the mass, res, stiff,
-   /// and load elements for implicit/explicit ODE integration
-   /// \param[in] ess_bdr - array of boundaries attributes marked essential
-   /// \param[in] mass - bilinear form for mass matrix (not owned)
-   /// \param[in] res - nonlinear residual operator (not owned)
-   /// \param[in] stiff - bilinear form for stiffness matrix (not owned)
-   /// \param[in] load - load vector (not owned)
-   /// \param[in] a - used to move the spatial residual to the rhs
-   ParSystemOperator(mfem::ParBilinearForm &_mass,
-                     mfem::ParBilinearForm &_stiff);
-
-   /// Compute r = M@k + K@(x+dt*k)
-   /// (with `@` denoting matrix-vector multiplication)
-   /// \param[in] k - dx/dt 
-   /// \param[out] r - the residual
-   /// \note the signs on each operator must be accounted for elsewhere
-   void Mult(const mfem::Vector &k, mfem::Vector &r) const override;
-
-   /// Compute J = M + dt * K
-   /// \param[in] k - dx/dt 
-   mfem::Operator &GetGradient(const mfem::Vector &k) const override;
-
-   /// Set current dt and x values - needed to compute action and Jacobian.
-   void setParameters(double _dt, const mfem::Vector *_x);
-
-   ~ParSystemOperator();
-
-private:
-   mfem::ParBilinearForm &mass;
-   mfem::ParBilinearForm &stiff;
-   mutable mfem::HypreParMatrix *jacobian, *stiff_jacobian;
-
-   double dt;
-   const mfem::Vector *x;
-
-   mutable mfem::Vector work, work2;
-
-   std::unique_ptr<PAJacobianOperator> pa_jac;
-
-};
-
-/** A time-dependent operator for the right-hand side of the ODE. The weak
-    form of du/dt = -a.grad(u) + nu Delta(u) is M du/dt = K u + b, where M and
-    K are the mass and advection-diffusion matrices, and b describes the flow
-    on the boundary. This can be written as a general ODE,
-    du/dt = M^{-1} (K u + b), and this class is used to evaluate the right-hand
-    side. */
-class ParAdvectionDiffusionEvolution : public mfem::TimeDependentOperator
-{
-public:
-   /// \param[in] M - parallel bilinear form for mass matrix
-   /// \param[in] K - parallel bilinear form for stiffness matrix
-   ParAdvectionDiffusionEvolution(mfem::ParBilinearForm &M,
-                                  mfem::ParBilinearForm &K);
-
-   /// Perform the action of the operator: y = k = f(x, t), where k solves
-   /// Compute k = M^-1(Kx + l)
-   void Mult(const mfem::Vector &x, mfem::Vector &y) const override;
-
-   /// Solve the implicit equation: k = f(x + dt k, t), for the unknown k at
-   /// the current time t. 
-   void ImplicitSolve(const double dt, const mfem::Vector &x,
-                      mfem::Vector &k) override;
-
-   virtual ~ParAdvectionDiffusionEvolution();
-
-private:
-   mfem::OperatorHandle M_;
-   mfem::ParBilinearForm &M, &K;
-   /// solver for inverting mass matrix for explicit time-marching
-   std::unique_ptr<mfem::Solver> M_prec;
-   mfem::CGSolver M_solver;
-   /// solver for implicit time-marching
-   mfem::Solver *prec;
-   mfem::GMRESSolver linear_solver;
-   mfem::NewtonSolver newton;
-
-   mfem::Vector diag;
-   mutable mfem::Vector z, work, work2;
-
-   /// pointer-to-implementation idiom
-   /// Hides implementation details of this operator
-   /// Operator that combines the linear spatial discretization with
-   /// the load vector into one operator used for implicit solves
-   std::unique_ptr<ParSystemOperator> combined_oper;
-
-   /// sets the state and dt for the combined operator
-   /// \param[in] dt - time increment
-   /// \param[in] x - the current state
-   void setOperParameters(double dt, const mfem::Vector *x);
-
-};
-
-// Choice for the problem setup. The fluid velocity, initial condition and
-// inflow boundary condition are chosen based on this parameter.
-int problem;
-
-// Velocity coefficient
-void velocity_function(const Vector &X, Vector &v);
-
-// Initial condition
-double u0_function(const Vector &X);
-
-// Inflow boundary condition
-double inflow_function(const Vector &X, const double t);
-
-// Mesh bounding box
-Vector bb_min, bb_max;
-
-int main(int argc, char *argv[])
-{
-   // 1. Initialize MPI.
-   int num_procs, myid;
-   MPI_Init(&argc, &argv);
-   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
-   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
-
-   // 2. Parse command-line options.
-   problem = 3;
-   const char *mesh_file = "../data/periodic-square.mesh";
-   int ser_ref_levels = 0;
-   int par_ref_levels = 0;
-   int order = 3;
-   const char *device_config = "cpu";
-   int ode_solver_type = 22;
-   double t_final = 3 * 2*M_PI;
-   double dt = 0.01;
-   bool glvis = false;
-   bool paraview = false;
-   int vis_steps = 5;
-
-   double nu_val = 0.001;
-
-   int precision = 8;
-   cout.precision(precision);
-
-   OptionsParser args(argc, argv);
-   args.AddOption(&mesh_file, "-m", "--mesh",
-                  "Mesh file to use.");
-   args.AddOption(&problem, "-p", "--problem",
-                  "Problem setup to use. See options in velocity_function().");
-   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
-                  "Number of times to refine the mesh uniformly in serial.");
-   args.AddOption(&par_ref_levels, "-rp", "--refine-parallel",
-                  "Number of times to refine the mesh uniformly in parallel.");
-   args.AddOption(&order, "-o", "--order",
-                  "Order (degree) of the finite elements.");
-   args.AddOption(&device_config, "-d", "--device",
-                  "Device configuration string, see Device::Configure().");
-   args.AddOption(&ode_solver_type, "-s", "--ode-solver",
-                  "ODE solver: 1 - Forward Euler,\n\t"
-                  "            2 - RK2 SSP, 3 - RK3 SSP, 4 - RK4, 6 - RK6.");
-   args.AddOption(&t_final, "-tf", "--t-final",
-                  "Final time; start time is 0.");
-   args.AddOption(&dt, "-dt", "--time-step",
-                  "Time step.");
-   args.AddOption(&glvis, "-vis", "--visualization", "-no-vis",
-                  "--no-visualization",
-                  "Enable or disable GLVis visualization.");
-   args.AddOption(&paraview, "-paraview", "--paraview-datafiles", "-no-paraview",
-                  "--no-paraview-datafiles",
-                  "Save data files for ParaView (paraview.org) visualization.");
-   args.AddOption(&vis_steps, "-vs", "--visualization-steps",
-                  "Visualize every n-th timestep.");
-   args.AddOption(&nu_val, "-nu", "--nu-value",
-                  "Value for \nu, the parameter that controls diffusion.");
-   args.Parse();
-   if (!args.Good())
-   {
-      if (myid == 0)
-      {
-         args.PrintUsage(cout);
-      }
-      MPI_Finalize();
-      return 1;
-   }
-   if (myid == 0)
-   {
-      std::cout << "Num ranks: " << num_procs << "\n";
-      args.PrintOptions(cout);
-   }
-   Device device(device_config);
-   if (myid == 0) { device.Print(); }
-
-   // 3. Read the serial mesh from the given mesh file on all processors. We can
-   //    handle geometrically periodic meshes in this code.
-   Mesh *mesh = new Mesh(mesh_file, 1, 1);
-   int dim = mesh->Dimension();
-
-   // 5. Refine the mesh in serial to increase the resolution. In this example
-   //    we do 'ser_ref_levels' of uniform refinement, where 'ser_ref_levels' is
-   //    a command-line parameter. If the mesh is of NURBS type, we convert it
-   //    to a (piecewise-polynomial) high-order mesh.
-   for (int lev = 0; lev < ser_ref_levels; lev++)
-   {
-      mesh->UniformRefinement();
-   }
-   mesh->GetBoundingBox(bb_min, bb_max, max(order, 1));
-
-   // 6. Define the parallel mesh by a partitioning of the serial mesh. Refine
-   //    this mesh further in parallel to increase the resolution. Once the
-   //    parallel mesh is defined, the serial mesh can be deleted.
-   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
-   delete mesh;
-   for (int lev = 0; lev < par_ref_levels; lev++)
-   {
-      pmesh->UniformRefinement();
-   }
-
-   // 7. Define the finite element space of the given
-   //    polynomial order on the refined mesh.
-   H1_FECollection fec(order, dim, BasisType::GaussLobatto);
-   ParFiniteElementSpace *fes = new ParFiniteElementSpace(pmesh, &fec);
-
-   HYPRE_Int global_vSize = fes->GlobalTrueVSize();
-   if (myid == 0)
-   {
-      cout << "Number of unknowns: " << global_vSize << endl;
-   }
-
-   // 8. Set up and assemble the bilinear and linear forms corresponding to the
-   //    CG discretization.
-   /// negative to move the diffusion terms to the right side
-   ConstantCoefficient nu(-nu_val);
-   ConstantCoefficient one(1.0);
-   VectorFunctionCoefficient velocity(dim, velocity_function);
-   FunctionCoefficient u0(u0_function);
-
-   ParBilinearForm *m_pa = new ParBilinearForm(fes);
-   ParBilinearForm *k_pa = new ParBilinearForm(fes);
-   m_pa->SetAssemblyLevel(AssemblyLevel::PARTIAL);
-   k_pa->SetAssemblyLevel(AssemblyLevel::PARTIAL);
-
-   /// create mass matrix
-   m_pa->AddDomainIntegrator(new MassIntegrator(one));
-   /// add advection terms to stiffness matrix
-   k_pa->AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
-   /// add diffusion terms to stiffness matrix
-   k_pa->AddDomainIntegrator(new DiffusionIntegrator(nu));
-
-   m_pa->Assemble();
-   int skip_zeros = 0;
-   k_pa->Assemble(skip_zeros);
-   m_pa->Finalize();
-   k_pa->Finalize(skip_zeros);
-
-   ParBilinearForm *m = new ParBilinearForm(fes);
-   ParBilinearForm *k = new ParBilinearForm(fes);
-   /// create mass matrix
-   m->AddDomainIntegrator(new MassIntegrator);
-   /// add advection terms to stiffness matrix
-   k->AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
-   /// add diffusion terms to stiffness matrix
-   k->AddDomainIntegrator(new DiffusionIntegrator(nu));
-
-   m->Assemble();
-   k->Assemble(skip_zeros);
-   m->Finalize();
-   k->Finalize(skip_zeros);
-
-
-   ParGridFunction *u = new ParGridFunction(fes);
-   u->UseDevice(true);
-   u->ProjectCoefficient(u0);
-
-
-   HypreParVector *U = u->GetTrueDofs();
-
-   ParSystemOperator pso(*m, *k);
-   ParSystemOperator pso_pa(*m_pa, *k_pa);
-
-   pso.setParameters(dt, U);
-   pso_pa.setParameters(dt, U);
-
-   MPI_Barrier(MPI_COMM_WORLD);
-   mfem::Vector pso_r(U->Size());
-   double t1 = MPI_Wtime();
-   pso.Mult(*U, pso_r);
-   double t2 = MPI_Wtime();
-   double fa_mult_time = t2 - t1;
-   double average_fa_mult_time;
-   MPI_Reduce(&fa_mult_time, &average_fa_mult_time, 1,
-              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
-   if (myid == 0)
-      std::cout << "FA Mult time: " << average_fa_mult_time / num_procs << endl;
-
-   MPI_Barrier(MPI_COMM_WORLD);
-   mfem::Vector pso_pa_r(U->Size());
-   double t3 = MPI_Wtime();
-   pso_pa.Mult(*U, pso_pa_r);
-   double t4 = MPI_Wtime();
-   double pa_mult_time = t4 - t3;
-   double average_pa_mult_time;
-   MPI_Reduce(&pa_mult_time, &average_pa_mult_time, 1,
-              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
-   if (myid == 0)
-      std::cout << "FA Mult time: " << average_pa_mult_time / num_procs << endl;
-   
-   double local_mult_speedup = (t2-t1) / (t4-t3);
-   double global_mult_speedup;
-   MPI_Reduce(&local_mult_speedup, &global_mult_speedup, 1,
-              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
-
-   if (myid == 0)
-      std::cout << "PA mult speedup: " << global_mult_speedup / num_procs << endl;
-
-   mfem::Vector diff_r(pso_pa_r);
-   diff_r -= pso_r;
-   // std::cout << "r diff: " << diff_r.Norml2() << std::endl;
-
-   mfem::Operator &pso_jac = pso.GetGradient(*U);
-   mfem::Operator &pso_pa_jac = pso_pa.GetGradient(*U);
-
-   MPI_Barrier(MPI_COMM_WORLD);
-   mfem::Vector pso_jac_r(U->Size());
-   double t5 = MPI_Wtime();
-   pso_jac.Mult(*U, pso_jac_r);
-   double t6 = MPI_Wtime();
-   double fa_jac_mult_time = t6-t5;
-   double average_fa_jac_time;
-   MPI_Reduce(&fa_jac_mult_time, &average_fa_jac_time, 1,
-              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
-   if (myid == 0)
-      std::cout << "FA Jac Mult time: " << average_fa_jac_time / num_procs << endl;
-
-   MPI_Barrier(MPI_COMM_WORLD);
-   mfem::Vector pso_pa_jac_r(U->Size());
-   double t7 = MPI_Wtime();
-   pso_pa_jac.Mult(*U, pso_pa_jac_r);
-   double t8 = MPI_Wtime();
-   double pa_jac_mult_time = t8-t7;
-   double average_pa_jac_time;
-   MPI_Reduce(&pa_jac_mult_time, &average_pa_jac_time, 1,
-              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
-   if (myid == 0)
-      std::cout << "PA Jac Mult time: " << average_pa_jac_time / num_procs << endl;
-
-   double local_jac_speedup = (t6-t5) / (t8-t7);
-   double global_jac_speedup;
-   MPI_Reduce(&local_jac_speedup, &global_jac_speedup, 1,
-              MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
-   if (myid == 0)
-      std::cout << "PA Jac mult speedup: " << global_jac_speedup / num_procs << endl;
-
-   // 13. Free the used memory.
-   delete U;
-   delete u;
-   delete k;
-   delete m;
-   delete fes;
-   delete pmesh;
-
-   MPI_Finalize();
-   return 0;
-}
-
-// Velocity coefficient
-void velocity_function(const Vector &x, Vector &v)
-{
-   int dim = x.Size();
-
-   // map to the reference [-1,1] domain
-   Vector X(dim);
-   for (int i = 0; i < dim; i++)
-   {
-      double center = (bb_min[i] + bb_max[i]) * 0.5;
-      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
-   }
-
-   switch (problem)
-   {
-      case 3:
-      {
-         // Translations in 1D, 2D, and 3D
-         switch (dim)
-         {
-            case 1: v(0) = 1.0; break;
-            case 2: v(0) = sqrt(2./3.); v(1) = sqrt(1./3.); break;
-            case 3: v(0) = sqrt(3./6.); v(1) = sqrt(2./6.); v(2) = sqrt(1./6.);
-               break;
-         }
-         break;
-      }
-      case 1:
-      case 2:
-      {
-         // Clockwise rotation in 2D around the origin
-         const double w = M_PI/2;
-         switch (dim)
-         {
-            case 1: v(0) = 1.0; break;
-            case 2: v(0) = w*X(1); v(1) = -w*X(0); break;
-            case 3: v(0) = w*X(1); v(1) = -w*X(0); v(2) = 0.0; break;
-         }
-         break;
-      }
-      case 0:
-      {
-         // Clockwise twisting rotation in 2D around the origin
-         const double w = M_PI/2;
-         double d = max((X(0)+1.)*(1.-X(0)),0.) * max((X(1)+1.)*(1.-X(1)),0.);
-         d = d*d;
-         switch (dim)
-         {
-            case 1: v(0) = 1.0; break;
-            case 2: v(0) = d*w*X(1); v(1) = -d*w*X(0); break;
-            case 3: v(0) = d*w*X(1); v(1) = -d*w*X(0); v(2) = 0.0; break;
-         }
-         break;
-      }
-   }
-}
-
-// Initial condition
-double u0_function(const Vector &x)
-{
-   int dim = x.Size();
-
-   // map to the reference [-1,1] domain
-   Vector X(dim);
-   for (int i = 0; i < dim; i++)
-   {
-      double center = (bb_min[i] + bb_max[i]) * 0.5;
-      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
-   }
-
-   switch (problem)
-   {
-      case 0:
-      case 1:
-      {
-         switch (dim)
-         {
-            case 1:
-               return exp(-40.*pow(X(0)-0.5,2));
-            case 2:
-            case 3:
-            {
-               double rx = 0.45, ry = 0.25, cx = 0., cy = -0.2, w = 10.;
-               if (dim == 3)
-               {
-                  const double s = (1. + 0.25*cos(2*M_PI*X(2)));
-                  rx *= s;
-                  ry *= s;
-               }
-               return ( erfc(w*(X(0)-cx-rx))*erfc(-w*(X(0)-cx+rx)) *
-                        erfc(w*(X(1)-cy-ry))*erfc(-w*(X(1)-cy+ry)) )/16;
-            }
-         }
-      }
-      case 2:
-      {
-         double x_ = X(0), y_ = X(1), rho, phi;
-         rho = hypot(x_, y_);
-         phi = atan2(y_, x_);
-         return pow(sin(M_PI*rho),2)*sin(3*phi);
-      }
-      case 3:
-      {
-         const double f = M_PI;
-         return sin(f*X(0))*sin(f*X(1));
-      }
-   }
-   return 0.0;
-}
-
-// Inflow boundary condition (zero for the problems considered in this example)
-double inflow_function(const Vector &x, const double t)
-{
-   switch (problem)
-   {
-      case 0:
-      case 1:
-      case 2:
-      case 3: return 0.0;
-   }
-   return 0.0;
-}
-
-class AdvectionDiffusionEvolution::SystemOperator : public mfem::Operator
-{
-public:
-   /// Nonlinear operator of the form that combines the mass, res, stiff,
-   /// and load elements for implicit/explicit ODE integration
-   /// \param[in] mass - bilinear form for mass matrix (not owned)
-   /// \param[in] res - nonlinear residual operator (not owned)
-   /// \param[in] stiff - bilinear form for stiffness matrix (not owned)
-   /// \param[in] load - load vector (not owned)
-   /// \param[in] a - used to move the spatial residual to the rhs
-   SystemOperator(BilinearForm &_mass, BilinearForm &_stiff,
-                  const mfem::Vector &b)
-      : Operator(_mass.Height()), mass(_mass), stiff(_stiff),
-        load(b), Jacobian(NULL), dt(0.0), x(NULL), work(height)
-      { }
-
-   /// Compute r = M@k + K@(x+dt*k) + l
-   /// (with `@` denoting matrix-vector multiplication)
-   /// \param[in] k - dx/dt 
-   /// \param[out] r - the residual
-   /// \note the signs on each operator must be accounted for elsewhere
-   void Mult(const mfem::Vector &k, mfem::Vector &r) const override
-   {
-      /// work = x+dt*k = x+dt*dx/dt = x+dx
-      add(1.0, *x, dt, k, work);
-      r = 0.0;
-      stiff.AddMult(work, r);
-      r += load;
-      mass.AddMult(k, r, -1.0);
-   }
-
-   /// Compute J = M + dt * K
-   /// \param[in] k - dx/dt 
-   mfem::Operator &GetGradient(const mfem::Vector &k) const override
-   {
-      delete Jacobian;
-      Jacobian = Add(-1.0, mass.SpMat(), dt, stiff.SpMat());
-      return *Jacobian;
-   }
-
-   /// Set current dt and x values - needed to compute action and Jacobian.
-   void setParameters(double _dt, const mfem::Vector *_x)
-   {
-      dt = _dt;
-      x = _x;
-   };
-
-   ~SystemOperator() {delete Jacobian;};
-
-private:
-   BilinearForm &mass;
-   BilinearForm &stiff;
-   const mfem::Vector &load;
-   mutable mfem::SparseMatrix *Jacobian;
-
-   double dt;
-   const mfem::Vector *x;
-
-   mutable mfem::Vector work, work2;
-
-};
-
-AdvectionDiffusionEvolution::AdvectionDiffusionEvolution(
-      BilinearForm &_M, BilinearForm &_K, const Vector &_b)
-   : TimeDependentOperator(_M.Height()), M(_M), K(_K), b(_b),
-     z(_M.Height())
-{
-   bool pa = M.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
-   Array<int> ess_tdof_list;
-   if (pa)
-   {
-      M_prec.reset(new OperatorJacobiSmoother(M, ess_tdof_list));
-      M_solver.SetOperator(M);
-   }
-   else
-   {
-      M_prec.reset(new DSmoother(M.SpMat()));
-      M_solver.SetOperator(M.SpMat());
-   }
-
-   combined_oper.reset(new SystemOperator(_M, _K, _b));
-
-   M_solver.SetPreconditioner(*M_prec);
-   M_solver.iterative_mode = false;
-   M_solver.SetRelTol(1e-9);
-   M_solver.SetAbsTol(0.0);
-   M_solver.SetMaxIter(100);
-   M_solver.SetPrintLevel(0);
-
-   linear_solver.iterative_mode = true;
-   linear_solver.SetRelTol(1e-12);
-   linear_solver.SetAbsTol(0.0);
-   linear_solver.SetMaxIter(100);
-   linear_solver.SetPrintLevel(0);
-   linear_solver.SetPreconditioner(prec);
-
-   newton.iterative_mode = false;
-   newton.SetRelTol(1e-9);
-   newton.SetAbsTol(0.0);
-   newton.SetMaxIter(100);
-   newton.SetPrintLevel(-1);
-   newton.SetSolver(linear_solver);
-   newton.SetOperator(*combined_oper);
-}
-
-void AdvectionDiffusionEvolution::Mult(const Vector &x, Vector &y) const
-{
-   // y = M^{-1} (K x + b)
-   K.Mult(x, z);
-   z += b;
-   M_solver.Mult(z, y);
-}
-
-void AdvectionDiffusionEvolution::ImplicitSolve(const double dt,
-                                                const Vector &x,
-                                                Vector &k)
-{
-   setOperParameters(dt, &x);
-   Vector zero; // empty vector is interpreted as zero r.h.s. by NewtonSolver
-   newton.Mult(zero, k);
-   MFEM_VERIFY(newton.GetConverged(), "Newton solver did not converge!");
-}
-
-void AdvectionDiffusionEvolution::setOperParameters(double dt,
-                                                    const mfem::Vector *x)
-{
-   combined_oper->setParameters(dt, x);
-}
-
-AdvectionDiffusionEvolution::~AdvectionDiffusionEvolution() {}
-
-
-PAJacobianOperator::PAJacobianOperator(ParBilinearForm &_mass, ParBilinearForm &_stiff)
- : Operator(_mass.ParFESpace()->GetTrueVSize()), mass(_mass), stiff(_stiff),
-   dt(0.0) { }
-
-
-void PAJacobianOperator::Mult(const mfem::Vector &k, mfem::Vector &r) const
-{
-   r.UseDevice(true);
-   r = 0.0;
-   stiff.TrueAddMult(k, r, dt);
-   mass.TrueAddMult(k, r, -1.0);
-}
-   
-void PAJacobianOperator::setParameters(const double _dt)
-{
-   dt = _dt;
-};
-
-ParSystemOperator::ParSystemOperator(ParBilinearForm &_mass, ParBilinearForm &_stiff)
-   : Operator(_mass.ParFESpace()->GetTrueVSize()), mass(_mass), stiff(_stiff),
-   jacobian(NULL), stiff_jacobian(NULL), dt(0.0), x(NULL),
-   work(height)
-{
-   pa_jac.reset(new PAJacobianOperator(mass, stiff));
-}
-
-/// Compute r = M@k + K@(x+dt*k)
-/// (with `@` denoting matrix-vector multiplication)
-/// \param[in] k - dx/dt 
-/// \param[out] r - the residual
-/// \note the signs on each operator must be accounted for elsewhere
-void ParSystemOperator::Mult(const mfem::Vector &k, mfem::Vector &r) const
-{
-   r = 0.0;
-   work.UseDevice(true);
-   work = 0.0;
-   /// work = x+dt*k = x+dt*dx/dt = x+dx
-   if (x)
-   {
-      add(1.0, *x, dt, k, work);
-   }
-
-   stiff.TrueAddMult(work, r);
-   mass.TrueAddMult(k, r, -1.0);
-}
-
-/// Compute J = M + dt * K
-/// \param[in] k - dx/dt 
-mfem::Operator &ParSystemOperator::GetGradient(const mfem::Vector &k) const
-{
-   bool mass_pa = mass.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
-   bool stiff_pa = stiff.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
-   
-   if (mass_pa && stiff_pa)
-   {
-      return *pa_jac.get();
-   }
-   else
-   {
-      delete stiff_jacobian;
-      delete jacobian;
-      jacobian = mass.ParallelAssemble();
-      *jacobian *= -1.0; //alpha;
-      stiff_jacobian = stiff.ParallelAssemble();
-      jacobian->Add(dt, *stiff_jacobian);
-      return *jacobian;
-   }
-}
-
-/// Set current dt and x values - needed to compute action and Jacobian.
-void ParSystemOperator::setParameters(const double _dt, const mfem::Vector *_x)
-{
-   dt = _dt;
-   x = _x;
-   pa_jac->setParameters(_dt);
-};
-
-ParSystemOperator::~ParSystemOperator()
-{
-   delete jacobian;
-   delete stiff_jacobian;
-};
-
-ParAdvectionDiffusionEvolution::ParAdvectionDiffusionEvolution(
-      ParBilinearForm &_M, ParBilinearForm &_K)
-   : TimeDependentOperator(_M.ParFESpace()->GetTrueVSize()), M(_M), K(_K), z(_M.Height())
-{
-   bool mass_pa = M.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
-   bool stiff_pa = K.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
-
-   Array<int> ess_tdof_list;
-   M_solver = CGSolver(MPI_COMM_WORLD);
-   if (mass_pa)
-   {
-      M_prec.reset(new OperatorJacobiSmoother(M, ess_tdof_list));
-      M_solver.SetOperator(M);
-   }
-   else
-   {
-      M_.Reset(_M.ParallelAssemble(), true);
-
-      // M_prec.reset(new HypreSmoother());
-      // M_solver.SetOperator(M.As<HypreParMatrix>());
-      HypreParMatrix &M_mat = *M_.As<HypreParMatrix>();
-      // HypreParMatrix &K_mat = *K.As<HypreParMatrix>();
-      M_prec.reset(new HypreSmoother(M_mat, HypreSmoother::Jacobi));
-   }
-
-   combined_oper.reset(new ParSystemOperator(_M, _K));
-
-   M_solver.SetPreconditioner(*M_prec);
-   M_solver.iterative_mode = false;
-   M_solver.SetRelTol(1e-9);
-   M_solver.SetAbsTol(0.0);
-   M_solver.SetMaxIter(100);
-   M_solver.SetPrintLevel(0);
-
-   if (mass_pa && stiff_pa)
-   {
-      diag.UseDevice(true);
-      diag.SetSize(M.ParFESpace()->GetTrueVSize());
-      diag = 0.0;
-      work.UseDevice(true);
-      work2.UseDevice(true);
-      work.SetSize(M.ParFESpace()->GetTrueVSize());
-      work2.SetSize(M.ParFESpace()->GetTrueVSize());
-      work = 0.0;
-      work2 = 0.0;
-      M.AssembleDiagonal(work);
-
-      ParBilinearForm k(M.ParFESpace());
-      ConstantCoefficient nu(-0.01);
-      k.AddDomainIntegrator(new mfem::DiffusionIntegrator(nu));
-      k.SetAssemblyLevel(AssemblyLevel::PARTIAL);
-      k.Assemble(0);
-      k.Finalize(0);
-      k.AssembleDiagonal(work2);
-
-      double dt = 0.1;
-      add(-1.0, work, dt, work2, diag);
-
-      prec = new OperatorChebyshevSmoother(combined_oper.get(), diag,
-                                           ess_tdof_list, 5,
-                                           M.ParFESpace()->GetComm());
-   }
-   else
-   {
-      prec = new HypreSmoother();
-   }
-   
-   linear_solver = GMRESSolver(MPI_COMM_WORLD);
-   linear_solver.iterative_mode = true;
-   linear_solver.SetRelTol(1e-12);
-   linear_solver.SetAbsTol(0.0);
-   linear_solver.SetMaxIter(2000);
-   linear_solver.SetPrintLevel(0);
-   linear_solver.SetPreconditioner(*prec);
-   linear_solver.SetKDim(2000);
-
-   newton.iterative_mode = true;
-   newton.SetRelTol(1e-9);
-   newton.SetAbsTol(0.0);
-   newton.SetMaxIter(10);
-   newton.SetPrintLevel(-1);
-   newton.SetSolver(linear_solver);
-   newton.SetOperator(*combined_oper);
-}
-
-void ParAdvectionDiffusionEvolution::Mult(const Vector &x, Vector &y) const
-{
-   // y = M^{-1} (K x + b)
-   K.Mult(x, z);
-   M_solver.Mult(z, y);
-}
-
-void ParAdvectionDiffusionEvolution::ImplicitSolve(const double dt,
-                                                   const Vector &x,
-                                                   Vector &k)
-{
-   setOperParameters(dt, &x);
-   Vector zero; // empty vector is interpreted as zero r.h.s. by NewtonSolver
-   newton.Mult(zero, k);
-   MFEM_VERIFY(newton.GetConverged(), "Newton solver did not converge!");
-}
-
-void ParAdvectionDiffusionEvolution::setOperParameters(const double dt,
-                                                       const mfem::Vector *x)
-{
-   combined_oper->setParameters(dt, x);
-}
-
-ParAdvectionDiffusionEvolution::~ParAdvectionDiffusionEvolution() {delete prec;}
@@ -36,11 +36,9 @@ set(SRCS
  intrules.cpp
  linearform.cpp
  lininteg.cpp
-  multigrid.cpp
  nonlinearform.cpp
  nonlinearform_ext.cpp
  nonlininteg.cpp
-  fespacehierarchy.cpp
  nonlininteg_vectorconvection.cpp
  quadinterpolator.cpp
  quadinterpolator_face.cpp
@@ -49,7 +47,6 @@ set(SRCS
  tmop.cpp
  tmop_tools.cpp
  gslib.cpp
-  transfer.cpp
  )

 set(HDRS
@@ -71,14 +68,12 @@ set(HDRS
  intrules.hpp
  linearform.hpp
  lininteg.hpp
-  multigrid.hpp
  nonlinearform.hpp
  nonlinearform_ext.hpp
  nonlininteg.hpp
  quadinterpolator.hpp
  quadinterpolator_face.hpp
  restriction.hpp
-  fespacehierarchy.hpp
  staticcond.hpp
  tbilinearform.hpp
  tbilininteg.hpp
@@ -91,7 +86,6 @@ set(HDRS
  tmop.hpp
  tmop_tools.hpp
  gslib.hpp
-  transfer.hpp
  )

 if (MFEM_USE_SIDRE)
@@ -104,11 +98,6 @@ if (MFEM_USE_CONDUIT)
  list(APPEND HDRS conduitdatacollection.hpp)
 endif()

-if (MFEM_USE_ADIOS2)
-  list(APPEND SRCS adios2datacollection.cpp)
-  list(APPEND HDRS adios2datacollection.hpp)
-endif()
-
 if (MFEM_USE_MPI)
  list(APPEND SRCS
    pbilinearform.cpp
@@ -1,90 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-//
-// Created on: Jan 7, 2020
-// Author: William F Godoy godoywf@ornl.gov
-// adios2: Adaptable Input/Output System https://github.com/ornladios/ADIOS2
-
-#include "adios2datacollection.hpp"
-
-namespace mfem
-{
-
-#ifdef MFEM_USE_MPI
-ADIOS2DataCollection::ADIOS2DataCollection(MPI_Comm comm,
-                                           const std::string& collection_name, Mesh* mesh,
-                                           const std::string engine_type) : DataCollection(collection_name, mesh),
-   stream( new adios2stream(name, adios2stream::openmode::out, comm, engine_type) )
-{
-   SetMesh(mesh);
-}
-#else
-ADIOS2DataCollection::ADIOS2DataCollection(
-   const std::string& collection_name, Mesh* mesh,
-   const std::string engine_type): DataCollection(collection_name, mesh),
-   stream( new adios2stream(name, adios2stream::openmode::out, engine_type) )
-{
-   SetMesh(mesh);
-}
-#endif
-
-ADIOS2DataCollection::~ADIOS2DataCollection()
-{
-   stream->Close();
-}
-
-void ADIOS2DataCollection::Save()
-{
-   stream->BeginStep();
-
-   // only save mesh once (moving mesh, not yet supported)
-   if (stream->CurrentStep() == 0)
-   {
-      if (mesh == nullptr)
-      {
-         const std::string error_message =
-            "MFEM ADIOS2DataCollection Save error: Mesh is null. Please call SetMesh before Save\n";
-         mfem_error(error_message.c_str());
-      }
-      stream->Print(*mesh);
-   }
-
-   // reduce footprint
-   if (myid == 0)
-   {
-      stream->SetTime(time);
-      stream->SetCycle(cycle);
-   }
-
-   for (const auto& field : field_map)
-   {
-      const std::string& variable_name = field.first;
-      field.second->Save(*stream.get(), variable_name);
-   }
-
-   stream->EndStep();
-}
-
-void ADIOS2DataCollection::SetParameter(const std::string key,
-                                        const std::string value) noexcept
-{
-   stream->SetParameter(key, value);
-}
-
-void ADIOS2DataCollection::SetLevelsOfDetail(const int levels_of_detail)
-noexcept
-{
-   stream->SetRefinementLevel(levels_of_detail);
-}
-
-} //end namespace mfem
-
-
@@ -1,88 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-//
-// Created on: Jan 7, 2020
-// Author: William F Godoy godoywf@ornl.gov
-// adios2: Adaptable Input/Output System https://github.com/ornladios/ADIOS2
-
-#ifndef MFEM_ADIOS2DATACOLLECTION
-#define MFEM_ADIOS2DATACOLLECTION
-
-#include "../config/config.hpp"
-#include "../general/adios2stream.hpp"
-#include "datacollection.hpp"
-
-#include <memory>  // std::unique_ptr
-#include <string>
-
-namespace mfem
-{
-
-class ADIOS2DataCollection : public DataCollection
-{
-public:
-#ifdef MFEM_USE_MPI
-   /**
-    * Parallel constructor. Important: scope of this object must be within
-    * MPI_Init and MPI_Finalize otherwise. The destructor will call the Close
-    * function. Either object must live in a try/catch block (inside try) or use
-    * raw pointers calling delete before MPI_Finalize.
-    * @param comm MPI communicator setting the datacollection domain
-    * @param collection_name unique name for saving data
-    * @param mesh can be set at the constructor level or later by calling
-    * SetMesh()
-    * @param engine_type adios2 engine type
-    */
-   ADIOS2DataCollection(MPI_Comm comm, const std::string& collection_name,
-                        Mesh* mesh = nullptr,
-                        const std::string engine_type = "BPFile");
-#else
-   /**
-    * Serial constructor
-    * @param collection_name unique name for saving data
-    * @param mesh can be set at the constructor level or later by calling
-    * SetMesh()
-    * @param engine_type adios2 engine type
-    * @throws std::invalid_argument (user input error) or std::runtime_error
-    *         (system error)
-    */
-   ADIOS2DataCollection(const std::string& collection_name, Mesh* mesh = nullptr,
-                        const std::string engine_type = "BPFile");
-#endif
-
-   virtual ~ADIOS2DataCollection();
-
-   /** Save the collection */
-   virtual void Save();
-
-   /**
-    * Pass a parameter unique to adios2datacollection
-    * For available parameters:
-    * See https://adios2.readthedocs.io/en/latest/engines/engines.html
-    * The most common is: key=SubStreams value=1 to nprocs (MPI processes)
-    * @param key parameter key
-    * @param value parameter value
-    */
-   void SetParameter(const std::string key, const std::string value) noexcept;
-
-   /**
-    * Sets the levels of detail for the global grid refinement
-    * @param levels_of_detail (default = 1)
-    */
-   void SetLevelsOfDetail(const int levels_of_detail) noexcept;
-
-private:
-   std::unique_ptr<adios2stream> stream;
-};
-
-}  // namespace mfem
-
-#endif /* MFEM_ADIOS2DATACOLLECTION */
@@ -467,17 +467,8 @@ void BilinearForm::Assemble(int skip_zeros)
         const FiniteElement &be = *fes->GetBE(i);
         fes -> GetBdrElementVDofs (i, vdofs);
         eltrans = fes -> GetBdrElementTransformation (i);
-         int k = 0;
-         for (; k < bbfi.Size(); k++)
-         {
-            if (bbfi_marker[k] &&
-                (*bbfi_marker[k])[bdr_attr-1] == 0) { continue; }
-
-            bbfi[k]->AssembleElementMatrix(be, *eltrans, elmat);
-            k++;
-            break;
-         }
-         for (; k < bbfi.Size(); k++)
+         bbfi[0]->AssembleElementMatrix(be, *eltrans, elmat);
+         for (int k = 1; k < bbfi.Size(); k++)
         {
            if (bbfi_marker[k] &&
                (*bbfi_marker[k])[bdr_attr-1] == 0) { continue; }
@@ -151,8 +151,8 @@ public:
   /** This method must be called before assembly. */
   void SetAssemblyLevel(AssemblyLevel assembly_level);

-   /// Returns the assembly level
-   AssemblyLevel GetAssemblyLevel() const { return assembly; }
+   /// Get the assembly level
+   AssemblyLevel GetAssemblyLevel() {return assembly;}

   /** Enable the use of static condensation. For details see the description
       for class StaticCondensation in fem/staticcond.hpp This method should be
@@ -530,7 +530,7 @@ public:

   /// (DEPRECATED) Return the FE space associated with the BilinearForm.
   /** @deprecated Use FESpace() instead. */
-   MFEM_DEPRECATED FiniteElementSpace *GetFES() { return fes; }
+   FiniteElementSpace *GetFES() { return fes; }

   /// Return the FE space associated with the BilinearForm.
   FiniteElementSpace *FESpace() { return fes; }
@@ -2135,17 +2135,17 @@ void VectorDiffusionIntegrator::AssembleElementMatrix(
   ElementTransformation &Trans,
   DenseMatrix &elmat)
 {
-   const int dim = el.GetDim();
-   const int dof = el.GetDof();
-   const int sdim = Trans.GetSpaceDim();
-   const bool square = (dim == sdim);
-   double w;
+   int dim = el.GetDim();
+   int dof = el.GetDof();

-   elmat.SetSize(sdim * dof);
+   double norm;

-   dshape.SetSize(dof, dim);
-   dshapedxt.SetSize(dof, sdim);
-   pelmat.SetSize(dof);
+   elmat.SetSize (dim * dof);
+
+   Jinv.  SetSize (dim);
+   dshape.SetSize (dof, dim);
+   gshape.SetSize (dof, dim);
+   pelmat.SetSize (dof);

   const IntegrationRule *ir = IntRule;
   if (ir == NULL)
@@ -2163,29 +2163,35 @@ void VectorDiffusionIntegrator::AssembleElementMatrix(
   }

   elmat = 0.0;
-   pelmat = 0.0;

   for (int i = 0; i < ir -> GetNPoints(); i++)
   {
      const IntegrationPoint &ip = ir->IntPoint(i);
+
      el.CalcDShape (ip, dshape);
+
      Trans.SetIntPoint (&ip);
-      w = Trans.Weight();
-      w = ip.weight / (square ? w : w*w*w);
-      // AdjugateJacobian = / adj(J),         if J is square
-      //                    \ adj(J^t.J).J^t, otherwise
-      Mult(dshape, Trans.AdjugateJacobian(), dshapedxt);
-      if (Q) {  w *= Q -> Eval (Trans, ip); }
-      AddMult_a_AAt(w, dshapedxt, pelmat);
-   }
-   for (int d = 0; d < sdim; d++)
-   {
-      for (int k = 0; k < dof; k++)
+      norm = ip.weight * Trans.Weight();
+      CalcInverse (Trans.Jacobian(), Jinv);
+
+      Mult (dshape, Jinv, gshape);
+
+      MultAAt (gshape, pelmat);
+
+      if (Q)
      {
-         for (int l = 0; l < dof; l++)
-         {
-            elmat(dof*d+k, dof*d+l) = pelmat(k, l);
-         }
+         norm *= Q -> Eval (Trans, ip);
+      }
+
+      pelmat *= norm;
+
+      for (int d = 0; d < dim; d++)
+      {
+         for (int k = 0; k < dof; k++)
+            for (int l = 0; l < dof; l++)
+            {
+               elmat (dof*d+k, dof*d+l) += pelmat (k, l);
+            }
      }
   }
 }
@@ -453,16 +453,6 @@ public:
                                       ElementTransformation &Trans,
                                       DenseMatrix &elmat);

-   /// Support for use in BilinearForm. Can be used only when appropriate.
-   /** Appropriate use cases are classes derived from
-       MixedScalarVectorIntegrator where the trial and test spaces can be the
-       same. Examples of such classes are: MixedVectorDivergenceIntegrator,
-       MixedScalarWeakDivergenceIntegrator, etc. */
-   virtual void AssembleElementMatrix(const FiniteElement &fe,
-                                      ElementTransformation &Trans,
-                                      DenseMatrix &elmat)
-   { AssembleElementMatrix2(fe, fe, Trans, elmat); }
-
 protected:

   MixedScalarVectorIntegrator(VectorCoefficient &vq, bool _transpose = false,
@@ -1655,6 +1645,19 @@ protected:
   {
      trial_fe.CalcPhysCurlShape(Trans, shape);
   }
+
+   virtual void AssemblePA(const FiniteElementSpace &trial_fes,
+                           const FiniteElementSpace &test_fes);
+
+   virtual void AddMultPA(const Vector&, Vector&) const;
+
+private:
+   // PA extension
+   Vector pa_data;
+   const DofToQuad *mapsO;         ///< Not owned. DOF-to-quad map, open.
+   const DofToQuad *mapsC;         ///< Not owned. DOF-to-quad map, closed.
+   const GeometricFactors *geom;   ///< Not owned
+   int dim, ne, dofs1D, quad1D, testType, trialType, coeffDim;
 };

 /** Class for integrating the bilinear form a(u,v) := (Q u, curl v) in 3D and
@@ -1694,6 +1697,19 @@ protected:
   {
      test_fe.CalcPhysCurlShape(Trans, shape);
   }
+
+   virtual void AssemblePA(const FiniteElementSpace &trial_fes,
+                           const FiniteElementSpace &test_fes);
+
+   virtual void AddMultPA(const Vector&, Vector&) const;
+
+private:
+   // PA extension
+   Vector pa_data;
+   const DofToQuad *mapsO;         ///< Not owned. DOF-to-quad map, open.
+   const DofToQuad *mapsC;         ///< Not owned. DOF-to-quad map, closed.
+   const GeometricFactors *geom;   ///< Not owned
+   int dim, ne, dofs1D, quad1D, testType, trialType, coeffDim;
 };

 /** Class for integrating the bilinear form a(u,v) := - (Q u, grad v) in either
@@ -2415,12 +2431,14 @@ protected:
   // PA extension
   const DofToQuad *maps;         ///< Not owned
   const GeometricFactors *geom;  ///< Not owned
-   int dim, sdim, ne, dofs1D, quad1D;
+   int dim, ne, dofs1D, quad1D;
   Vector pa_data;

 private:
-   DenseMatrix dshape, dshapedxt, pelmat;
-   DenseMatrix Jinv, gshape;
+   DenseMatrix Jinv;
+   DenseMatrix dshape;
+   DenseMatrix gshape;
+   DenseMatrix pelmat;

 public:
   VectorDiffusionIntegrator() { Q = NULL; }
@@ -81,20 +81,12 @@ static void OccaPADiffusionSetup3D(const int D1D,
 #endif // MFEM_USE_OCCA

 // PA Diffusion Assemble 2D kernel
-template<const int T_SDIM>
 static void PADiffusionSetup2D(const int Q1D,
                               const int NE,
                               const Array<double> &w,
                               const Vector &j,
                               const Vector &c,
-                               Vector &d);
-template<>
-void PADiffusionSetup2D<2>(const int Q1D,
-                           const int NE,
-                           const Array<double> &w,
-                           const Vector &j,
-                           const Vector &c,
-                           Vector &d)
+                               Vector &d)
 {
   const int NQ = Q1D*Q1D;
   const bool const_c = c.Size() == 1;
@@ -120,48 +112,6 @@ void PADiffusionSetup2D<2>(const int Q1D,
   });
 }

-// PA Diffusion Assemble 2D kernel with 3D node coords
-template<>
-void PADiffusionSetup2D<3>(const int Q1D,
-                           const int NE,
-                           const Array<double> &w,
-                           const Vector &j,
-                           const Vector &c,
-                           Vector &d)
-{
-   constexpr int DIM = 2;
-   constexpr int SDIM = 3;
-   const int NQ = Q1D*Q1D;
-   const bool const_c = c.Size() == 1;
-
-   auto W = w.Read();
-   auto J = Reshape(j.Read(), NQ, SDIM, DIM, NE);
-   auto C = const_c ? Reshape(c.Read(), 1, 1) : Reshape(c.Read(), NQ, NE);
-   auto D = Reshape(d.Write(), NQ, 3, NE);
-   MFEM_FORALL(e, NE,
-   {
-      for (int q = 0; q < NQ; ++q)
-      {
-         const double wq = W[q];
-         const double J11 = J(q,0,0,e);
-         const double J21 = J(q,1,0,e);
-         const double J31 = J(q,2,0,e);
-         const double J12 = J(q,0,1,e);
-         const double J22 = J(q,1,1,e);
-         const double J32 = J(q,2,1,e);
-         const double E = J11*J11 + J21*J21 + J31*J31;
-         const double G = J12*J12 + J22*J22 + J32*J32;
-         const double F = J11*J12 + J21*J22 + J31*J32;
-         const double iw = 1.0 / sqrt(E*G - F*F);
-         const double coeff = const_c ? C(0,0) : C(q,e);
-         const double alpha = wq * coeff * iw;
-         D(q,0,e) =  alpha * G; // 1,1
-         D(q,1,e) = -alpha * F; // 1,2
-         D(q,2,e) =  alpha * E; // 2,2
-      }
-   });
-}
-
 // PA Diffusion Assemble 3D kernel
 static void PADiffusionSetup3D(const int Q1D,
                               const int NE,
@@ -216,7 +166,6 @@ static void PADiffusionSetup3D(const int Q1D,
 }

 static void PADiffusionSetup(const int dim,
-                             const int sdim,
                             const int D1D,
                             const int Q1D,
                             const int NE,
@@ -234,11 +183,8 @@ static void PADiffusionSetup(const int dim,
         OccaPADiffusionSetup2D(D1D, Q1D, NE, W, J, C, D);
         return;
      }
-#else
-      MFEM_CONTRACT_VAR(D1D);
 #endif // MFEM_USE_OCCA
-      if (sdim == 2) { PADiffusionSetup2D<2>(Q1D, NE, W, J, C, D); }
-      if (sdim == 3) { PADiffusionSetup2D<3>(Q1D, NE, W, J, C, D); }
+      PADiffusionSetup2D(Q1D, NE, W, J, C, D);
   }
   if (dim == 3)
   {
@@ -271,8 +217,6 @@ void DiffusionIntegrator::SetupPA(const FiniteElementSpace &fes,
      InitCeedCoeff(Q, ptr);
      return CeedPADiffusionAssemble(fes, *ir, *ptr);
   }
-#else
-   MFEM_CONTRACT_VAR(force);
 #endif
   const int dims = el.GetDim();
   const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
@@ -280,7 +224,6 @@ void DiffusionIntegrator::SetupPA(const FiniteElementSpace &fes,
   dim = mesh->Dimension();
   ne = fes.GetNE();
   geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
-   const int sdim = mesh->SpaceDimension();
   maps = &el.GetDofToQuad(*ir, DofToQuad::TENSOR);
   dofs1D = maps->ndof;
   quad1D = maps->nqpt;
@@ -309,8 +252,8 @@ void DiffusionIntegrator::SetupPA(const FiniteElementSpace &fes,
         }
      }
   }
-   PADiffusionSetup(dim, sdim, dofs1D, quad1D, ne, ir->GetWeights(), geom->J,
-                    coeff, pa_data);
+   PADiffusionSetup(dim, dofs1D, quad1D, ne, ir->GetWeights(), geom->J, coeff,
+                    pa_data);
 }

 void DiffusionIntegrator::AssemblePA(const FiniteElementSpace &fes)
@@ -967,6 +910,8 @@ template<int T_D1D = 0, int T_Q1D = 0, int T_NBZ = 0>
 static void SmemPADiffusionApply2D(const int NE,
                                   const Array<double> &b_,
                                   const Array<double> &g_,
+                                   const Array<double> &bt_,
+                                   const Array<double> &gt_,
                                   const Vector &d_,
                                   const Vector &x_,
                                   Vector &y_,
@@ -1312,6 +1257,8 @@ template<int T_D1D = 0, int T_Q1D = 0>
 static void SmemPADiffusionApply3D(const int NE,
                                   const Array<double> &b_,
                                   const Array<double> &g_,
+                                   const Array<double> &bt_,
+                                   const Array<double> &gt_,
                                   const Vector &d_,
                                   const Vector &x_,
                                   Vector &y_,
@@ -1578,14 +1525,14 @@ static void PADiffusionApply(const int dim,
   {
      switch ((D1D << 4 ) | Q1D)
      {
-         case 0x22: return SmemPADiffusionApply2D<2,2,16>(NE,B,G,D,X,Y);
-         case 0x33: return SmemPADiffusionApply2D<3,3,16>(NE,B,G,D,X,Y);
-         case 0x44: return SmemPADiffusionApply2D<4,4,8>(NE,B,G,D,X,Y);
-         case 0x55: return SmemPADiffusionApply2D<5,5,8>(NE,B,G,D,X,Y);
-         case 0x66: return SmemPADiffusionApply2D<6,6,4>(NE,B,G,D,X,Y);
-         case 0x77: return SmemPADiffusionApply2D<7,7,4>(NE,B,G,D,X,Y);
-         case 0x88: return SmemPADiffusionApply2D<8,8,2>(NE,B,G,D,X,Y);
-         case 0x99: return SmemPADiffusionApply2D<9,9,2>(NE,B,G,D,X,Y);
+         case 0x22: return SmemPADiffusionApply2D<2,2,16>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x33: return SmemPADiffusionApply2D<3,3,16>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x44: return SmemPADiffusionApply2D<4,4,8>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x55: return SmemPADiffusionApply2D<5,5,8>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x66: return SmemPADiffusionApply2D<6,6,4>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x77: return SmemPADiffusionApply2D<7,7,4>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x88: return SmemPADiffusionApply2D<8,8,2>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x99: return SmemPADiffusionApply2D<9,9,2>(NE,B,G,Bt,Gt,D,X,Y);
         default:   return PADiffusionApply2D(NE,B,G,Bt,Gt,D,X,Y,D1D,Q1D);
      }
   }
@@ -1593,15 +1540,15 @@ static void PADiffusionApply(const int dim,
   {
      switch ((D1D << 4 ) | Q1D)
      {
-         case 0x23: return SmemPADiffusionApply3D<2,3>(NE,B,G,D,X,Y);
-         case 0x34: return SmemPADiffusionApply3D<3,4>(NE,B,G,D,X,Y);
-         case 0x45: return SmemPADiffusionApply3D<4,5>(NE,B,G,D,X,Y);
-         case 0x46: return SmemPADiffusionApply3D<4,6>(NE,B,G,D,X,Y);
-         case 0x56: return SmemPADiffusionApply3D<5,6>(NE,B,G,D,X,Y);
-         case 0x58: return SmemPADiffusionApply3D<5,8>(NE,B,G,D,X,Y);
-         case 0x67: return SmemPADiffusionApply3D<6,7>(NE,B,G,D,X,Y);
-         case 0x78: return SmemPADiffusionApply3D<7,8>(NE,B,G,D,X,Y);
-         case 0x89: return SmemPADiffusionApply3D<8,9>(NE,B,G,D,X,Y);
+         case 0x23: return SmemPADiffusionApply3D<2,3>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x34: return SmemPADiffusionApply3D<3,4>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x45: return SmemPADiffusionApply3D<4,5>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x46: return SmemPADiffusionApply3D<4,6>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x56: return SmemPADiffusionApply3D<5,6>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x58: return SmemPADiffusionApply3D<5,8>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x67: return SmemPADiffusionApply3D<6,7>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x78: return SmemPADiffusionApply3D<7,8>(NE,B,G,Bt,Gt,D,X,Y);
+         case 0x89: return SmemPADiffusionApply3D<8,9>(NE,B,G,Bt,Gt,D,X,Y);
         default:   return PADiffusionApply3D(NE,B,G,Bt,Gt,D,X,Y,D1D,Q1D);
      }
   }
@@ -101,6 +101,7 @@ static void PAVectorDiffusionSetup3D(const int Q1D,
 }

 static void PAVectorDiffusionSetup(const int dim,
+                                   const int D1D,
                                   const int Q1D,
                                   const int NE,
                                   const Array<double> &W,
@@ -133,7 +134,6 @@ void VectorDiffusionIntegrator::AssemblePA(const FiniteElementSpace &fes)
   const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
   const int nq = ir->GetNPoints();
   dim = mesh->Dimension();
-   sdim = mesh->SpaceDimension();
   ne = fes.GetNE();
   geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
   maps = &el.GetDofToQuad(*ir, DofToQuad::TENSOR);
@@ -147,83 +147,46 @@ void VectorDiffusionIntegrator::AssemblePA(const FiniteElementSpace &fes)
      MFEM_VERIFY(cQ != NULL, "only ConstantCoefficient is supported!");
      coeff = cQ->constant;
   }
-   const Array<double> &w = ir->GetWeights();
-   const Vector &j = geom->J;
-   Vector &d = pa_data;
-   if (dim == 1) { MFEM_ABORT("dim==1 not supported in PAVectorDiffusionSetup"); }
-   if (dim == 2 && sdim == 3)
-   {
-      constexpr int DIM = 2;
-      constexpr int SDIM = 3;
-      const int NQ = quad1D*quad1D;
-      auto W = w.Read();
-      auto J = Reshape(j.Read(), NQ, SDIM, DIM, ne);
-      auto D = Reshape(d.Write(), NQ, SDIM, ne);
-      MFEM_FORALL(e, ne,
-      {
-         for (int q = 0; q < NQ; ++q)
-         {
-            const double wq = W[q];
-            const double J11 = J(q,0,0,e);
-            const double J21 = J(q,1,0,e);
-            const double J31 = J(q,2,0,e);
-            const double J12 = J(q,0,1,e);
-            const double J22 = J(q,1,1,e);
-            const double J32 = J(q,2,1,e);
-            const double E = J11*J11 + J21*J21 + J31*J31;
-            const double G = J12*J12 + J22*J22 + J32*J32;
-            const double F = J11*J12 + J21*J22 + J31*J32;
-            const double iw = 1.0 / sqrt(E*G - F*F);
-            const double alpha = wq * coeff * iw;
-            D(q,0,e) =  alpha * G; // 1,1
-            D(q,1,e) = -alpha * F; // 1,2
-            D(q,2,e) =  alpha * E; // 2,2
-         }
-      });
-   }
-   else
-   {
-      PAVectorDiffusionSetup(dim, quad1D, ne, w, j, coeff, d);
-   }
+   PAVectorDiffusionSetup(dim, dofs1D, quad1D, ne, ir->GetWeights(), geom->J,
+                          coeff, pa_data);
 }

 // PA Diffusion Apply 2D kernel
-template<int T_D1D = 0, int T_Q1D = 0, int T_VDIM = 0> static
+template<int T_D1D = 0, int T_Q1D = 0> static
 void PAVectorDiffusionApply2D(const int NE,
                              const Array<double> &b,
                              const Array<double> &g,
                              const Array<double> &bt,
                              const Array<double> &gt,
-                              const Vector &d_,
-                              const Vector &x_,
-                              Vector &y_,
+                              const Vector &_op,
+                              const Vector &_x,
+                              Vector &_y,
                              const int d1d = 0,
-                              const int q1d = 0,
-                              const int vdim = 0)
+                              const int q1d = 0)
 {
   const int D1D = T_D1D ? T_D1D : d1d;
   const int Q1D = T_Q1D ? T_Q1D : q1d;
-   const int VDIM = T_VDIM ? T_VDIM : vdim;
+   constexpr int VDIM = 2;
   MFEM_VERIFY(D1D <= MAX_D1D, "");
   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
   auto B = Reshape(b.Read(), Q1D, D1D);
   auto G = Reshape(g.Read(), Q1D, D1D);
   auto Bt = Reshape(bt.Read(), D1D, Q1D);
   auto Gt = Reshape(gt.Read(), D1D, Q1D);
-   auto D = Reshape(d_.Read(), Q1D*Q1D, 3, NE);
-   auto x = Reshape(x_.Read(), D1D, D1D, VDIM, NE);
-   auto y = Reshape(y_.ReadWrite(), D1D, D1D, VDIM, NE);
+   auto op = Reshape(_op.Read(), Q1D*Q1D, 3, NE);
+   auto x = Reshape(_x.Read(), D1D, D1D, VDIM, NE);
+   auto y = Reshape(_y.ReadWrite(), D1D, D1D, VDIM, NE);
   MFEM_FORALL(e, NE,
   {
      const int D1D = T_D1D ? T_D1D : d1d;
      const int Q1D = T_Q1D ? T_Q1D : q1d;
-      const int VDIM = T_VDIM ? T_VDIM : vdim;
+      // the following variables are evaluated at compile time
      constexpr int max_D1D = T_D1D ? T_D1D : MAX_D1D;
      constexpr int max_Q1D = T_Q1D ? T_Q1D : MAX_Q1D;

-      double grad[max_Q1D][max_Q1D][2];
-      for (int c = 0; c < VDIM; c++)
+      for (int c = 0; c < VDIM; ++ c)
      {
+         double grad[max_Q1D][max_Q1D][2];
         for (int qy = 0; qy < Q1D; ++qy)
         {
            for (int qx = 0; qx < Q1D; ++qx)
@@ -266,11 +229,14 @@ void PAVectorDiffusionApply2D(const int NE,
            for (int qx = 0; qx < Q1D; ++qx)
            {
               const int q = qx + qy * Q1D;
-               const double O11 = D(q,0,e);
-               const double O12 = D(q,1,e);
-               const double O22 = D(q,2,e);
+
+               const double O11 = op(q,0,e);
+               const double O12 = op(q,1,e);
+               const double O22 = op(q,2,e);
+
               const double gradX = grad[qy][qx][0];
               const double gradY = grad[qy][qx][1];
+
               grad[qy][qx][0] = (O11 * gradX) + (O12 * gradY);
               grad[qy][qx][1] = (O12 * gradX) + (O22 * gradY);
            }
@@ -280,8 +246,8 @@ void PAVectorDiffusionApply2D(const int NE,
            double gradX[max_D1D][2];
            for (int dx = 0; dx < D1D; ++dx)
            {
-               gradX[dx][0] = 0.0;
-               gradX[dx][1] = 0.0;
+               gradX[dx][0] = 0;
+               gradX[dx][1] = 0;
            }
            for (int qx = 0; qx < Q1D; ++qx)
            {
@@ -503,36 +469,35 @@ void PAVectorDiffusionApply3D(const int NE,
   });
 }

+static void PAVectorDiffusionApply(const int dim,
+                                   const int D1D,
+                                   const int Q1D,
+                                   const int NE,
+                                   const Array<double> &B,
+                                   const Array<double> &G,
+                                   const Array<double> &Bt,
+                                   const Array<double> &Gt,
+                                   const Vector &op,
+                                   const Vector &x,
+                                   Vector &y)
+{
+   if (dim == 2)
+   {
+      return PAVectorDiffusionApply2D(NE,B,G,Bt,Gt,op,x,y,D1D,Q1D);
+   }
+   if (dim == 3)
+   {
+      return PAVectorDiffusionApply3D(NE,B,G,Bt,Gt,op,x,y,D1D,Q1D);
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
 // PA Diffusion Apply kernel
 void VectorDiffusionIntegrator::AddMultPA(const Vector &x, Vector &y) const
 {
-   const int D1D = dofs1D;
-   const int Q1D = quad1D;
-   const Array<double> &B = maps->B;
-   const Array<double> &G = maps->G;
-   const Array<double> &Bt = maps->Bt;
-   const Array<double> &Gt = maps->Gt;
-   const Vector &D = pa_data;
-
-   if (dim == 2 && sdim == 3)
-   {
-      switch ((dofs1D << 4 ) | quad1D)
-      {
-         case 0x22: return PAVectorDiffusionApply2D<2,2,3>(ne,B,G,Bt,Gt,D,x,y);
-         case 0x33: return PAVectorDiffusionApply2D<3,3,3>(ne,B,G,Bt,Gt,D,x,y);
-         case 0x44: return PAVectorDiffusionApply2D<4,4,3>(ne,B,G,Bt,Gt,D,x,y);
-         case 0x55: return PAVectorDiffusionApply2D<5,5,3>(ne,B,G,Bt,Gt,D,x,y);
-         default:
-            return PAVectorDiffusionApply2D(ne,B,G,Bt,Gt,D,x,y,D1D,Q1D,sdim);
-      }
-   }
-   if (dim == 2 && sdim == 2)
-   { return PAVectorDiffusionApply2D(ne,B,G,Bt,Gt,D,x,y,D1D,Q1D,sdim); }
-
-   if (dim == 3 && sdim == 3)
-   { return PAVectorDiffusionApply3D(ne,B,G,Bt,Gt,D,x,y,D1D,Q1D); }
-
-   MFEM_ABORT("Unknown kernel.");
+   PAVectorDiffusionApply(dim, dofs1D, quad1D, ne,
+                          maps->B, maps->G, maps->Bt, maps->Gt,
+                          pa_data, x, y);
 }

 template<int T_D1D = 0, int T_Q1D = 0>
@@ -174,24 +174,7 @@ void VectorGridFunctionCoefficient::SetGridFunction(GridFunction *gf)
 void VectorGridFunctionCoefficient::Eval(Vector &V, ElementTransformation &T,
                                         const IntegrationPoint &ip)
 {
-   Mesh *mesh = GridFunc->FESpace()->GetMesh();
-   if (mesh->Dimension() == T.GetDimension())
-   {
-      GridFunc->GetVectorValue(T.ElementNo, ip, V);
-   }
-   else // Assuming T is a boundary element transformation
-   {
-      int el_id, el_info;
-      mesh->GetBdrElementAdjacentElement(T.ElementNo, el_id, el_info);
-      IntegrationPointTransformation loc_T;
-      mesh->GetLocalFaceTransformation(mesh->GetBdrElementType(T.ElementNo),
-                                       mesh->GetElementType(el_id),
-                                       loc_T.Transf,
-                                       el_info);
-      IntegrationPoint eip;
-      loc_T.Transform(ip, eip);
-      GridFunc->GetVectorValue(el_id, eip, V);
-   }
+   GridFunc->GetVectorValue(T.ElementNo, ip, V);
 }

 void VectorGridFunctionCoefficient::Eval(
@@ -138,7 +138,7 @@ public:
   /// (DEPRECATED) Define a time-independent coefficient from a C-function
   /** @deprecated Use the method where the C-function, @a f, uses a const
       Vector argument instead of Vector. */
-   MFEM_DEPRECATED FunctionCoefficient(double (*f)(Vector &))
+   FunctionCoefficient(double (*f)(Vector &))
   {
      Function = reinterpret_cast<double(*)(const Vector&)>(f);
      TDFunction = NULL;
@@ -147,7 +147,7 @@ public:
   /// (DEPRECATED) Define a time-dependent coefficient from a C-function
   /** @deprecated Use the method where the C-function, @a tdf, uses a const
       Vector argument instead of Vector. */
-   MFEM_DEPRECATED FunctionCoefficient(double (*tdf)(Vector &, double))
+   FunctionCoefficient(double (*tdf)(Vector &, double))
   {
      Function = NULL;
      TDFunction = reinterpret_cast<double(*)(const Vector&,double)>(tdf);
@@ -391,10 +391,14 @@ void IsoparametricTransformation::SetIdentityTransformation(
      nodes.IntPoint(j).Get(&PointMat(0,j), dim);
   }
   geom = GeomType;
+   space_dim = dim;
 }

 const DenseMatrix &IsoparametricTransformation::EvalJacobian()
 {
+   MFEM_ASSERT(space_dim == PointMat.Height(),
+               "the IsoparametricTransformation has not been finalized;"
+               " call FinilizeTransformation() after setup");
   MFEM_ASSERT((EvalState & JACOBIAN_MASK) == 0, "");

   dshape.SetSize(FElem->GetDof(), FElem->GetDim());
@@ -411,6 +415,9 @@ const DenseMatrix &IsoparametricTransformation::EvalJacobian()

 const DenseMatrix &IsoparametricTransformation::EvalHessian()
 {
+   MFEM_ASSERT(space_dim == PointMat.Height(),
+               "the IsoparametricTransformation has not been finalized;"
+               " call FinilizeTransformation() after setup");
   MFEM_ASSERT((EvalState & HESSIAN_MASK) == 0, "");

   int Dim = FElem->GetDim();
@@ -426,7 +433,7 @@ const DenseMatrix &IsoparametricTransformation::EvalHessian()
   return d2Fdx2;
 }

-int IsoparametricTransformation::OrderJ() const
+int IsoparametricTransformation::OrderJ()
 {
   switch (FElem->Space())
   {
@@ -435,12 +442,12 @@ int IsoparametricTransformation::OrderJ() const
      case FunctionSpace::Qk:
         return (FElem->GetOrder());
      default:
-         MFEM_ABORT("unsupported finite element");
+         mfem_error("IsoparametricTransformation::OrderJ()");
   }
   return 0;
 }

-int IsoparametricTransformation::OrderW() const
+int IsoparametricTransformation::OrderW()
 {
   switch (FElem->Space())
   {
@@ -449,12 +456,12 @@ int IsoparametricTransformation::OrderW() const
      case FunctionSpace::Qk:
         return (FElem->GetOrder() * FElem->GetDim() - 1);
      default:
-         MFEM_ABORT("unsupported finite element");
+         mfem_error("IsoparametricTransformation::OrderW()");
   }
   return 0;
 }

-int IsoparametricTransformation::OrderGrad(const FiniteElement *fe) const
+int IsoparametricTransformation::OrderGrad(const FiniteElement *fe)
 {
   if (FElem->Space() == fe->Space())
   {
@@ -467,11 +474,9 @@ int IsoparametricTransformation::OrderGrad(const FiniteElement *fe) const
            return ((k-1)*(d-1)+(l-1));
         case FunctionSpace::Qk:
            return (k*(d-1)+(l-1));
-         default:
-            MFEM_ABORT("unsupported finite element");
      }
   }
-   MFEM_ABORT("incompatible finite elements");
+   mfem_error("IsoparametricTransformation::OrderGrad(...)");
   return 0;
 }

@@ -37,6 +37,7 @@ protected:
      HESSIAN_MASK  = 16
   };
   Geometry::Type geom;
+   int space_dim;

   // Evaluate the Jacobian of the transformation at the IntPoint and store it
   // in dFdx.
@@ -81,11 +82,11 @@ public:
   const DenseMatrix &InverseJacobian()
   { return (EvalState & INVERSE_MASK) ? invJ : EvalInverseJ(); }

-   virtual int Order() const = 0;
-   virtual int OrderJ() const = 0;
-   virtual int OrderW() const = 0;
+   virtual int Order() = 0;
+   virtual int OrderJ() = 0;
+   virtual int OrderW() = 0;
   /// Order of adj(J)^t.grad(fi)
-   virtual int OrderGrad(const FiniteElement *fe) const = 0;
+   virtual int OrderGrad(const FiniteElement *fe) = 0;

   /// Return the Geometry::Type of the reference element.
   Geometry::Type GetGeometryType() const { return geom; }
@@ -96,7 +97,7 @@ public:
   /// Get the dimension of the target (physical) space.
   /** We support 2D meshes embedded in 3D; in this case the function will
       return "3". */
-   virtual int GetSpaceDim() const = 0;
+   int GetSpaceDim() const { return space_dim; }

   /** @brief Transform a point @a pt from physical space to a point @a ip in
       reference space. */
@@ -306,23 +307,19 @@ public:
   void SetFE(const FiniteElement *FE) { FElem = FE; geom = FE->GetGeomType(); }
   const FiniteElement* GetFE() const { return FElem; }

-   /// @brief Set the underlying point matrix describing the transformation.
+   /** @brief Read and write access to the underlying point matrix describing
+       the transformation. */
   /** The dimensions of the matrix are space-dim x dof. The transformation is
       defined as

-           x = F(xh) = P . phi(xh),
+           x=F(xh)=P.phi(xh),

       where xh (x hat) is the reference point, x is the corresponding physical
       point, P is the point matrix, and phi(xh) is the column-vector of all
       basis functions evaluated at xh. The columns of P represent the control
       points in physical space defining the transformation. */
-   void SetPointMat(const DenseMatrix &pm) { PointMat = pm; }
-
-   /// Return the stored point matrix.
-   const DenseMatrix &GetPointMat() const { return PointMat; }
-
-   /// Write access to the stored point matrix. Use with caution.
   DenseMatrix &GetPointMat() { return PointMat; }
+   void FinalizeTransformation() { space_dim = PointMat.Height(); }

   void SetIdentityTransformation(Geometry::Type GeomType);

@@ -330,12 +327,10 @@ public:
   virtual void Transform(const IntegrationRule &, DenseMatrix &);
   virtual void Transform(const DenseMatrix &matrix, DenseMatrix &result);

-   virtual int Order() const { return FElem->GetOrder(); }
-   virtual int OrderJ() const;
-   virtual int OrderW() const;
-   virtual int OrderGrad(const FiniteElement *fe) const;
-
-   virtual int GetSpaceDim() const { return PointMat.Height(); }
+   virtual int Order() { return FElem->GetOrder(); }
+   virtual int OrderJ();
+   virtual int OrderW();
+   virtual int OrderGrad(const FiniteElement *fe);

   virtual int TransformBack(const Vector & v, IntegrationPoint & ip)
   {
@@ -344,8 +339,6 @@ public:
   }

   virtual ~IsoparametricTransformation() { }
-
-   MFEM_DEPRECATED void FinalizeTransformation() {}
 };

 class IntegrationPointTransformation
@@ -37,9 +37,6 @@
 #include "restriction.hpp"
 #include "quadinterpolator.hpp"
 #include "quadinterpolator_face.hpp"
-#include "transfer.hpp"
-#include "fespacehierarchy.hpp"
-#include "multigrid.hpp"

 #ifdef MFEM_USE_MPI
 #include "pfespace.hpp"
@@ -57,8 +54,4 @@
 #include "conduitdatacollection.hpp"
 #endif

-#ifdef MFEM_USE_ADIOS2
-#include "adios2datacollection.hpp"
-#endif
-
 #endif
@@ -495,10 +495,9 @@ FiniteElementSpace::H2L_GlobalRestrictionMatrix (FiniteElementSpace *lfes)
 {
   SparseMatrix *R;
   DenseMatrix loc_restr;
-   Array<int> l_dofs, h_dofs, l_vdofs, h_vdofs;
+   Array<int> l_dofs, h_dofs;

-   int vdim = lfes->GetVDim();
-   R = new SparseMatrix (vdim * lfes -> GetNDofs(), vdim * ndofs);
+   R = new SparseMatrix (lfes -> GetNDofs(), ndofs);

   Geometry::Type cached_geom = Geometry::INVALID;
   const FiniteElement *h_fe = NULL;
@@ -521,16 +520,7 @@ FiniteElementSpace::H2L_GlobalRestrictionMatrix (FiniteElementSpace *lfes)
         cached_geom = geom;
      }

-      for (int vd = 0; vd < vdim; vd++)
-      {
-         l_dofs.Copy(l_vdofs);
-         lfes->DofsToVDofs(vd, l_vdofs);
-
-         h_dofs.Copy(h_vdofs);
-         this->DofsToVDofs(vd, h_vdofs);
-
-         R -> SetSubMatrix (l_vdofs, h_vdofs, loc_restr, 1);
-      }
+      R -> SetSubMatrix (l_dofs, h_dofs, loc_restr, 1);
   }

   R -> Finalize();
@@ -680,6 +670,7 @@ void FiniteElementSpace::BuildConformingInterpolation() const
            if (!slave_dofs.Size()) { continue; }

            slave.OrientedPointMatrix(T.GetPointMat());
+            T.FinalizeTransformation();
            fe->GetLocalInterpolation(T, I);

            // make each slave DOF dependent on all master DOFs
@@ -1035,7 +1026,8 @@ void FiniteElementSpace::GetLocalRefinementMatrices(
   localP.SetSize(ldof, ldof, nmat);
   for (int i = 0; i < nmat; i++)
   {
-      isotr.SetPointMat(pmats(i));
+      isotr.GetPointMat() = pmats(i);
+      isotr.FinalizeTransformation();
      fe->GetLocalInterpolation(isotr, localP(i));
   }
 }
@@ -1104,90 +1096,46 @@ void FiniteElementSpace::RefinementOperator
   Mesh* mesh = fespace->GetMesh();
   const CoarseFineTransformations &rtrans = mesh->GetRefinementTransforms();

-   Array<int> dofs, vdofs, old_dofs, old_vdofs;
+   Array<int> dofs, old_dofs, old_vdofs;
+
+   Array<char> processed(fespace->GetVSize());
+   processed = 0;

   int vdim = fespace->GetVDim();
   int old_ndofs = width / vdim;

-   Vector subY, subX;
-
   for (int k = 0; k < mesh->GetNE(); k++)
   {
      const Embedding &emb = rtrans.embeddings[k];
      const Geometry::Type geom = mesh->GetElementBaseGeometry(k);
      const DenseMatrix &lP = localP[geom](emb.matrix);

-      subY.SetSize(lP.Height());
-
      fespace->GetElementDofs(k, dofs);
      old_elem_dof->GetRow(emb.parent, old_dofs);

      for (int vd = 0; vd < vdim; vd++)
      {
-         dofs.Copy(vdofs);
-         fespace->DofsToVDofs(vd, vdofs);
         old_dofs.Copy(old_vdofs);
         fespace->DofsToVDofs(vd, old_vdofs, old_ndofs);
-         x.GetSubVector(old_vdofs, subX);
-         lP.Mult(subX, subY);
-         y.SetSubVector(vdofs, subY);
-      }
-   }
-}

-void FiniteElementSpace::RefinementOperator
-::MultTranspose(const Vector &x, Vector &y) const
-{
-   y = 0.0;
-
-   Mesh* mesh = fespace->GetMesh();
-   const CoarseFineTransformations &rtrans = mesh->GetRefinementTransforms();
-
-   Array<char> processed(fespace->GetVSize());
-   processed = 0;
-
-   Array<int> f_dofs, c_dofs, f_vdofs, c_vdofs;
-
-   int vdim = fespace->GetVDim();
-   int old_ndofs = width / vdim;
-
-   Vector subY, subX;
-
-   for (int k = 0; k < mesh->GetNE(); k++)
-   {
-      const Embedding &emb = rtrans.embeddings[k];
-      const Geometry::Type geom = mesh->GetElementBaseGeometry(k);
-      const DenseMatrix &lP = localP[geom](emb.matrix);
-
-      fespace->GetElementDofs(k, f_dofs);
-      old_elem_dof->GetRow(emb.parent, c_dofs);
-
-      subY.SetSize(lP.Width());
-
-      for (int vd = 0; vd < vdim; vd++)
-      {
-         f_dofs.Copy(f_vdofs);
-         fespace->DofsToVDofs(vd, f_vdofs);
-         c_dofs.Copy(c_vdofs);
-         fespace->DofsToVDofs(vd, c_vdofs, old_ndofs);
-
-         x.GetSubVector(f_vdofs, subX);
-
-         for (int p = 0; p < f_dofs.Size(); ++p)
+         for (int i = 0; i < dofs.Size(); i++)
         {
-            if (processed[DecodeDof(f_dofs[p])])
+            double rsign, osign;
+            int r = fespace->DofToVDof(dofs[i], vd);
+            r = DecodeDof(r, rsign);
+
+            if (!processed[r])
            {
-               subX[p] = 0.0;
+               double value = 0.0;
+               for (int j = 0; j < old_vdofs.Size(); j++)
+               {
+                  int o = DecodeDof(old_vdofs[j], osign);
+                  value += x[o] * lP(i, j) * osign;
+               }
+               y[r] = value * rsign;
+               processed[r] = 1;
            }
         }
-
-         lP.MultTranspose(subX, subY);
-         y.AddElementVector(c_vdofs, subY);
-      }
-
-      for (int p = 0; p < f_dofs.Size(); ++p)
-      {
-         processed[DecodeDof(f_dofs[p])] = 1;
      }
   }
 }
@@ -1223,7 +1171,8 @@ FiniteElementSpace::DerefinementOperator::DerefinementOperator(
      emb_tr.SetIdentityTransformation(geom);
      for (int i = 0; i < pmats.SizeK(); i++)
      {
-         emb_tr.SetPointMat(pmats(i));
+         emb_tr.GetPointMat() = pmats(i);
+         emb_tr.FinalizeTransformation();
         // Get the local interpolation matrix for this refinement type
         fine_fe->GetTransferMatrix(*coarse_fe, emb_tr, lP(i));
         // Get the local mass matrix for this refinement type
@@ -1343,7 +1292,9 @@ void FiniteElementSpace::GetLocalDerefinementMatrices(Geometry::Type geom,
   localR.SetSize(ldof, ldof, nmat);
   for (int i = 0; i < nmat; i++)
   {
-      isotr.SetPointMat(pmats(i));
+      isotr.GetPointMat() = pmats(i);
+      isotr.FinalizeTransformation();
+
      fe->GetLocalRestriction(isotr, localR(i));
   }
 }
@@ -1441,7 +1392,8 @@ void FiniteElementSpace::GetLocalRefinementMatrices(
   localP.SetSize(fine_fe->GetDof(), coarse_fe->GetDof(), nmat);
   for (int i = 0; i < nmat; i++)
   {
-      isotr.SetPointMat(pmats(i));
+      isotr.GetPointMat() = pmats(i);
+      isotr.FinalizeTransformation();
      fine_fe->GetTransferMatrix(*coarse_fe, isotr, localP(i));
   }
 }
@@ -2698,7 +2650,8 @@ L2ProjectionGridTransfer::L2Projection::L2Projection(

         // Create the transformation that embeds the fine low-order element
         // within the coarse high-order element in reference space
-         emb_tr.SetPointMat(pmats(cf_tr.embeddings[ilor].matrix));
+         emb_tr.GetPointMat() = pmats(cf_tr.embeddings[ilor].matrix);
+         emb_tr.FinalizeTransformation();

         int order = fe_lor->GetOrder() + fe_ho->GetOrder() + el_tr->OrderW();
         const IntegrationRule *ir = &IntRules.Get(geom, order);
@@ -87,7 +87,6 @@ class FaceQuadratureInterpolator;
 class FiniteElementSpace
 {
   friend class InterpolationGridTransfer;
-   friend class PRefinementTransferOperator;

 protected:
   /// The mesh that FE space lives on (not owned).
@@ -159,12 +158,7 @@ protected:

   void BuildElementToDofTable() const;

-   /// Helpers to remove encoded sign from a DOF
-   static inline int DecodeDof(int dof)
-   {
-      return (dof >= 0) ? dof : (-1 - dof);
-   }
-
+   /// Helper to remove encoded sign from a DOF
   static inline int DecodeDof(int dof, double& sign)
   { return (dof >= 0) ? (sign = 1, dof) : (sign = -1, (-1 - dof)); }

@@ -202,7 +196,6 @@ protected:
      RefinementOperator(const FiniteElementSpace *fespace,
                         const FiniteElementSpace *coarse_fes);
      virtual void Mult(const Vector &x, Vector &y) const;
-      virtual void MultTranspose(const Vector &x, Vector &y) const;
      virtual ~RefinementOperator();
   };

@@ -1,189 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-
-#include "fespacehierarchy.hpp"
-#include "transfer.hpp"
-
-namespace mfem
-{
-
-FiniteElementSpaceHierarchy::FiniteElementSpaceHierarchy(Mesh* mesh,
-                                                         FiniteElementSpace* fespace,
-                                                         bool ownM, bool ownFES)
-{
-   meshes.Append(mesh);
-   fespaces.Append(fespace);
-   ownedMeshes.Append(ownM);
-   ownedFES.Append(ownFES);
-}
-
-FiniteElementSpaceHierarchy::~FiniteElementSpaceHierarchy()
-{
-   for (int i = 0; i < meshes.Size(); ++i)
-   {
-      if (ownedFES[i])
-      {
-         delete fespaces[i];
-      }
-      if (ownedMeshes[i])
-      {
-         delete meshes[i];
-      }
-   }
-
-   for (int i = 0; i < prolongations.Size(); ++i)
-   {
-      if (ownedProlongations[i])
-      {
-         delete prolongations[i];
-      }
-   }
-
-   fespaces.DeleteAll();
-   meshes.DeleteAll();
-   prolongations.DeleteAll();
-}
-
-int FiniteElementSpaceHierarchy::GetNumLevels() const { return meshes.Size(); }
-
-int FiniteElementSpaceHierarchy::GetFinestLevelIndex() const { return GetNumLevels() - 1; }
-
-void FiniteElementSpaceHierarchy::AddLevel(Mesh* mesh,
-                                           FiniteElementSpace* fespace,
-                                           Operator* prolongation,
-                                           bool ownM, bool ownFES,
-                                           bool ownP)
-{
-   meshes.Append(mesh);
-   fespaces.Append(fespace);
-   prolongations.Append(prolongation);
-   ownedMeshes.Append(ownM);
-   ownedFES.Append(ownFES);
-   ownedProlongations.Append(ownP);
-}
-
-void FiniteElementSpaceHierarchy::AddUniformlyRefinedLevel(int dim,
-                                                           int ordering)
-{
-   MFEM_VERIFY(GetNumLevels() > 0, "There is no level which can be refined");
-   Mesh* mesh = new Mesh(*GetFinestFESpace().GetMesh());
-   mesh->UniformRefinement();
-   FiniteElementSpace& coarseFEspace = GetFinestFESpace();
-   FiniteElementSpace* fineFEspace =
-      new FiniteElementSpace(mesh, coarseFEspace.FEColl(), dim, ordering);
-   Operator* P = new TransferOperator(coarseFEspace, *fineFEspace);
-   AddLevel(mesh, fineFEspace, P, true, true, true);
-}
-
-void FiniteElementSpaceHierarchy::AddOrderRefinedLevel(FiniteElementCollection*
-                                                       fec, int dim,
-                                                       int ordering)
-{
-   MFEM_VERIFY(GetNumLevels() > 0, "There is no level which can be refined");
-   Mesh* mesh = GetFinestFESpace().GetMesh();
-   FiniteElementSpace* newFEspace =
-      new FiniteElementSpace(mesh, fec, dim, ordering);
-   Operator* P = new TransferOperator(GetFinestFESpace(), *newFEspace);
-   AddLevel(mesh, newFEspace, P, false, true, true);
-}
-
-const FiniteElementSpace& FiniteElementSpaceHierarchy::GetFESpaceAtLevel(
-   int level) const
-{
-   MFEM_ASSERT(level < fespaces.Size(),
-               "FE space at given level does not exist.");
-   return *fespaces[level];
-}
-
-FiniteElementSpace& FiniteElementSpaceHierarchy::GetFESpaceAtLevel(int level)
-{
-   MFEM_ASSERT(level < fespaces.Size(),
-               "FE space at given level does not exist.");
-   return *fespaces[level];
-}
-
-const FiniteElementSpace& FiniteElementSpaceHierarchy::GetFinestFESpace() const
-{
-   return GetFESpaceAtLevel(GetFinestLevelIndex());
-}
-
-FiniteElementSpace& FiniteElementSpaceHierarchy::GetFinestFESpace()
-{
-   return GetFESpaceAtLevel(GetFinestLevelIndex());
-}
-
-Operator* FiniteElementSpaceHierarchy::GetProlongationAtLevel(int level) const
-{
-   MFEM_ASSERT(level < prolongations.Size(),
-               "Prolongation at given level does not exist.");
-   return prolongations[level];
-}
-
-#ifdef MFEM_USE_MPI
-
-ParFiniteElementSpaceHierarchy::ParFiniteElementSpaceHierarchy(ParMesh* mesh,
-                                                               ParFiniteElementSpace* fespace,
-                                                               bool ownM,
-                                                               bool ownFES)
-   : FiniteElementSpaceHierarchy(mesh, fespace, ownM, ownFES)
-{
-}
-
-void ParFiniteElementSpaceHierarchy::AddUniformlyRefinedLevel(int dim,
-                                                              int ordering)
-{
-   ParMesh* mesh = new ParMesh(*GetFinestFESpace().GetParMesh());
-   mesh->UniformRefinement();
-   ParFiniteElementSpace& coarseFEspace = GetFinestFESpace();
-   ParFiniteElementSpace* fineFEspace =
-      new ParFiniteElementSpace(mesh, coarseFEspace.FEColl(), dim, ordering);
-   Operator* P = new TrueTransferOperator(coarseFEspace, *fineFEspace);
-   AddLevel(mesh, fineFEspace, P, true, true, true);
-}
-
-void ParFiniteElementSpaceHierarchy::AddOrderRefinedLevel(
-   FiniteElementCollection* fec,
-   int dim, int ordering)
-{
-   ParMesh* mesh = GetFinestFESpace().GetParMesh();
-   ParFiniteElementSpace* newFEspace =
-      new ParFiniteElementSpace(mesh, fec, dim, ordering);
-   Operator* P = new TrueTransferOperator(GetFinestFESpace(), *newFEspace);
-   AddLevel(mesh, newFEspace, P, false, true, true);
-}
-
-const ParFiniteElementSpace&
-ParFiniteElementSpaceHierarchy::GetFESpaceAtLevel(int level) const
-{
-   return static_cast<const ParFiniteElementSpace&>(
-             FiniteElementSpaceHierarchy::GetFESpaceAtLevel(level));
-}
-
-ParFiniteElementSpace& ParFiniteElementSpaceHierarchy::GetFESpaceAtLevel(
-   int level)
-{
-   return static_cast<ParFiniteElementSpace&>(
-             FiniteElementSpaceHierarchy::GetFESpaceAtLevel(level));
-}
-
-const ParFiniteElementSpace& ParFiniteElementSpaceHierarchy::GetFinestFESpace()
-const
-{
-   return GetFESpaceAtLevel(GetFinestLevelIndex());
-}
-
-ParFiniteElementSpace& ParFiniteElementSpaceHierarchy::GetFinestFESpace()
-{
-   return GetFESpaceAtLevel(GetFinestLevelIndex());
-}
-#endif
-
-} // namespace mfem
@@ -1,123 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-
-#ifndef MFEM_FESPACEHIERARCHY
-#define MFEM_FESPACEHIERARCHY
-
-#include "fespace.hpp"
-
-#ifdef MFEM_USE_MPI
-#include "pfespace.hpp"
-#endif
-
-namespace mfem
-{
-
-/// Class bundling a hierarchy finite element spaces together with the
-/// corresponding prolongation operators
-class FiniteElementSpaceHierarchy
-{
-protected:
-   Array<Mesh*> meshes;
-   Array<FiniteElementSpace*> fespaces;
-   Array<Operator*> prolongations;
-   Array<bool> ownedMeshes;
-   Array<bool> ownedFES;
-   Array<bool> ownedProlongations;
-
-public:
-
-   /// @brief Constructs a space hierarchy with the given mesh and space on the
-   /// coarsest level.
-   /** The ownership of the mesh and space may be transferred to the
-       FiniteElementSpaceHierarchy by setting the according boolean variables. */
-   FiniteElementSpaceHierarchy(Mesh* mesh, FiniteElementSpace* fespace, bool ownM,
-                               bool ownFES);
-
-   /// Destructor deleting all meshes and spaces that are owned
-   virtual ~FiniteElementSpaceHierarchy();
-
-   /// Returns the number of levels in the hierarchy
-   int GetNumLevels() const;
-
-   /// Returns the index of the finest level
-   int GetFinestLevelIndex() const;
-
-   /// Adds one level to the hierarchy
-   void AddLevel(Mesh* mesh, FiniteElementSpace* fespace, Operator* prolongation,
-                 bool ownM, bool ownFES, bool ownP);
-
-   /// @brief Adds one level to the hierarchy by uniformly refining the mesh on the
-   /// previous level
-   virtual void AddUniformlyRefinedLevel(int dim = 1,
-                                         int ordering = Ordering::byVDIM);
-
-   /// @brief Adds one level to the hierarchy by using a different finite element
-   /// order defined through FiniteElementCollection
-   virtual void AddOrderRefinedLevel(FiniteElementCollection* fec, int dim = 1,
-                                     int ordering = Ordering::byVDIM);
-
-   /// Returns the finite element space at the given level
-   virtual const FiniteElementSpace& GetFESpaceAtLevel(int level) const;
-
-   /// Returns the finite element space at the given level
-   virtual FiniteElementSpace& GetFESpaceAtLevel(int level);
-
-   /// Returns the finite element space at the finest level
-   virtual const FiniteElementSpace& GetFinestFESpace() const;
-
-   /// Returns the finite element space at the finest level
-   virtual FiniteElementSpace& GetFinestFESpace();
-
-   /// @brief Returns the prolongation operator from the finite element space at
-   /// level to the finite element space at level + 1
-   Operator* GetProlongationAtLevel(int level) const;
-};
-
-#ifdef MFEM_USE_MPI
-class ParFiniteElementSpaceHierarchy : public FiniteElementSpaceHierarchy
-{
-public:
-   /// @brief Constructs a parallel space hierarchy with the given mesh and spaces
-   /// on level zero.
-   /** The ownership of the mesh and space may be transferred to the
-       ParFiniteElementSpaceHierarchy by setting the according boolean variables. */
-   ParFiniteElementSpaceHierarchy(ParMesh* mesh, ParFiniteElementSpace* fespace,
-                                  bool ownM,
-                                  bool ownFES);
-
-   /// @brief Adds one level to the hierarchy by uniformly refining the mesh on the
-   /// previous level
-   void AddUniformlyRefinedLevel(int dim = 1,
-                                 int ordering = Ordering::byVDIM) override;
-
-   /// @brief Adds one level to the hierarchy by using a different finite element
-   /// order defined through FiniteElementCollection
-   void AddOrderRefinedLevel(FiniteElementCollection* fec, int dim = 1,
-                             int ordering = Ordering::byVDIM) override;
-
-   /// Returns the finite element space at the given level
-   const ParFiniteElementSpace& GetFESpaceAtLevel(int level) const override;
-
-   /// Returns the finite element space at the given level
-   ParFiniteElementSpace& GetFESpaceAtLevel(int level) override;
-
-   /// Returns the finite element space at the finest level
-   const ParFiniteElementSpace& GetFinestFESpace() const override;
-
-   /// Returns the finite element space at the finest level
-   ParFiniteElementSpace& GetFinestFESpace() override;
-};
-#endif
-
-} // namespace mfem
-
-#endif
@@ -188,6 +188,7 @@ Geometry::Geometry()
      IsoparametricTransformation tri_T;
      tri_T.SetFE(&TriangleFE);
      GetPerfPointMat (TRIANGLE, tri_T.GetPointMat());
+      tri_T.FinalizeTransformation();
      tri_T.SetIntPoint(&GeomCenter[TRIANGLE]);
      *GeomToPerfGeomJac[TRIANGLE] = tri_T.Jacobian();
      CalcInverse(tri_T.Jacobian(), *PerfGeomToGeomJac[TRIANGLE]);
@@ -197,6 +198,7 @@ Geometry::Geometry()
      IsoparametricTransformation tet_T;
      tet_T.SetFE(&TetrahedronFE);
      GetPerfPointMat (TETRAHEDRON, tet_T.GetPointMat());
+      tet_T.FinalizeTransformation();
      tet_T.SetIntPoint(&GeomCenter[TETRAHEDRON]);
      *GeomToPerfGeomJac[TETRAHEDRON] = tet_T.Jacobian();
      CalcInverse(tet_T.Jacobian(), *PerfGeomToGeomJac[TETRAHEDRON]);
@@ -206,6 +208,7 @@ Geometry::Geometry()
      IsoparametricTransformation pri_T;
      pri_T.SetFE(&WedgeFE);
      GetPerfPointMat (PRISM, pri_T.GetPointMat());
+      pri_T.FinalizeTransformation();
      pri_T.SetIntPoint(&GeomCenter[PRISM]);
      *GeomToPerfGeomJac[PRISM] = pri_T.Jacobian();
      CalcInverse(pri_T.Jacobian(), *PerfGeomToGeomJac[PRISM]);
@@ -2753,15 +2753,6 @@ void GridFunction::Save(std::ostream &out) const
   out.flush();
 }

-#ifdef MFEM_USE_ADIOS2
-void GridFunction::Save(adios2stream &out,
-                        const std::string& variable_name,
-                        const adios2stream::data_type type) const
-{
-   out.Save(*this, variable_name, type);
-}
-#endif
-
 void GridFunction::SaveVTK(std::ostream &out, const std::string &field_name,
                           int ref)
 {
@@ -16,9 +16,6 @@
 #include "fespace.hpp"
 #include "coefficient.hpp"
 #include "bilininteg.hpp"
-#ifdef MFEM_USE_ADIOS2
-#include "../general/adios2stream.hpp"
-#endif
 #include <limits>
 #include <ostream>
 #include <string>
@@ -489,13 +486,6 @@ public:
   /// Save the GridFunction to an output stream.
   virtual void Save(std::ostream &out) const;

-#ifdef MFEM_USE_ADIOS2
-   /// Save the GridFunction to a binary output stream using adios2 bp format.
-   virtual void Save(adios2stream &out, const std::string& variable_name,
-                     const adios2stream::data_type
-                     type = adios2stream::data_type::point_data) const;
-#endif
-
   /** Write the GridFunction in VTK format. Note that Mesh::PrintVTK must be
       called first. The parameter ref > 0 must match the one used in
       Mesh::PrintVTK. */
@@ -100,7 +100,7 @@ public:

   /// (DEPRECATED) Return the FE space associated with the LinearForm.
   /** @deprecated Use FESpace() instead. */
-   MFEM_DEPRECATED FiniteElementSpace *GetFES() { return fes; }
+   FiniteElementSpace *GetFES() { return fes; }

   /// Read+write access to the associated FiniteElementSpace.
   FiniteElementSpace *FESpace() { return fes; }
@@ -93,37 +93,6 @@ void BoundaryLFIntegrator::AssembleRHSElementVect(
   }
 }

-void BoundaryLFIntegrator::AssembleRHSElementVect(
-   const FiniteElement &el, FaceElementTransformations &Tr, Vector &elvect)
-{
-   int dof = el.GetDof();
-
-   shape.SetSize(dof);        // vector of size dof
-   elvect.SetSize(dof);
-   elvect = 0.0;
-
-   const IntegrationRule *ir = IntRule;
-   if (ir == NULL)
-   {
-      int intorder = oa * el.GetOrder() + ob;    // <------ user control
-      ir = &IntRules.Get(Tr.FaceGeom, intorder); // of integration order
-   }
-
-   for (int i = 0; i < ir->GetNPoints(); i++)
-   {
-      const IntegrationPoint &ip = ir->IntPoint(i);
-      IntegrationPoint eip;
-      Tr.Loc1.Transform(ip, eip);
-
-      Tr.Face->SetIntPoint (&ip);
-      double val = Tr.Face->Weight() * ip.weight * Q.Eval(*Tr.Face, ip);
-
-      el.CalcShape(eip, shape);
-
-      add(elvect, val, shape, elvect);
-   }
-}
-
 void BoundaryNormalLFIntegrator::AssembleRHSElementVect(
   const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
 {
@@ -126,8 +126,7 @@ class BoundaryLFIntegrator : public LinearFormIntegrator
   Coefficient &Q;
   int oa, ob;
 public:
-   /** @brief Constructs a boundary integrator with a given Coefficient @a QG.
-       Integration order will be @a a * basis_order + @a b. */
+   /// Constructs a boundary integrator with a given Coefficient QG
   BoundaryLFIntegrator(Coefficient &QG, int a = 1, int b = 1)
      : Q(QG), oa(a), ob(b) { }

@@ -136,9 +135,8 @@ public:
   virtual void AssembleRHSElementVect(const FiniteElement &el,
                                       ElementTransformation &Tr,
                                       Vector &elvect);
-   virtual void AssembleRHSElementVect(const FiniteElement &el,
-                                       FaceElementTransformations &Tr,
-                                       Vector &elvect);
+
+   using LinearFormIntegrator::AssembleRHSElementVect;
 };

 /// Class for boundary integration \f$ L(v) = (g \cdot n, v) \f$
@@ -1,206 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-
-#include "multigrid.hpp"
-
-namespace mfem
-{
-
-Multigrid::Multigrid(const FiniteElementSpaceHierarchy& fespaces_)
-   : fespaces(fespaces_), cycleType(CycleType::VCYCLE), preSmoothingSteps(1),
-     postSmoothingSteps(1)
-{}
-
-Multigrid::~Multigrid()
-{
-   for (int i = 0; i < operators.Size(); ++i)
-   {
-      if (ownedOperators[i])
-      {
-         delete operators[i];
-      }
-      if (ownedSmoothers[i])
-      {
-         delete smoothers[i];
-      }
-      delete X[i];
-      delete Y[i];
-      delete R[i];
-      delete Z[i];
-   }
-
-   operators.DeleteAll();
-   smoothers.DeleteAll();
-   X.DeleteAll();
-   Y.DeleteAll();
-   R.DeleteAll();
-   Z.DeleteAll();
-
-   for (int i = 0; i < bfs.Size(); ++i)
-   {
-      delete bfs[i];
-   }
-
-   bfs.DeleteAll();
-
-   for (int i = 0; i < essentialTrueDofs.Size(); ++i)
-   {
-      delete essentialTrueDofs[i];
-   }
-
-   essentialTrueDofs.DeleteAll();
-}
-
-void Multigrid::AddLevel(Operator* opr, Solver* smoother, bool ownOperator,
-                         bool ownSmoother)
-{
-   operators.Append(opr);
-   smoothers.Append(smoother);
-   ownedOperators.Append(ownOperator);
-   ownedSmoothers.Append(ownSmoother);
-   width = opr->Width();
-   height = opr->Height();
-
-   X.Append(new Vector(height));
-   *X.Last() = 0.0;
-   Y.Append(new Vector(height));
-   *Y.Last() = 0.0;
-   R.Append(new Vector(height));
-   *R.Last() = 0.0;
-   Z.Append(new Vector(height));
-   *Z.Last() = 0.0;
-}
-
-int Multigrid::NumLevels() const { return operators.Size(); }
-
-int Multigrid::GetFinestLevelIndex() const { return NumLevels() - 1; }
-
-const Operator* Multigrid::GetOperatorAtLevel(int level) const
-{
-   return operators[level];
-}
-
-Operator* Multigrid::GetOperatorAtLevel(int level)
-{
-   return operators[level];
-}
-
-const Operator* Multigrid::GetOperatorAtFinestLevel() const
-{
-   return GetOperatorAtLevel(operators.Size() - 1);
-}
-
-Operator* Multigrid::GetOperatorAtFinestLevel()
-{
-   return GetOperatorAtLevel(operators.Size() - 1);
-}
-
-Solver* Multigrid::GetSmootherAtLevel(int level) const
-{
-   return smoothers[level];
-}
-
-Solver* Multigrid::GetSmootherAtLevel(int level)
-{
-   return smoothers[level];
-}
-
-void Multigrid::SetCycleType(CycleType cycleType_, int preSmoothingSteps_,
-                             int postSmoothingSteps_)
-{
-   cycleType = cycleType_;
-   preSmoothingSteps = preSmoothingSteps_;
-   postSmoothingSteps = postSmoothingSteps_;
-}
-
-void Multigrid::Mult(const Vector& x, Vector& y) const
-{
-   MFEM_ASSERT(NumLevels() > 0, "");
-   *X.Last() = x;
-   *Y.Last() = 0.0;
-   Cycle(GetFinestLevelIndex());
-   y = *Y.Last();
-}
-
-void Multigrid::SetOperator(const Operator& op)
-{
-   MFEM_ABORT("SetOperator not supported in Multigrid");
-}
-
-void Multigrid::SmoothingStep(int level) const
-{
-   GetOperatorAtLevel(level)->Mult(*Y[level], *R[level]); // r = A x
-   subtract(*X[level], *R[level], *R[level]);             // r = b - A x
-   GetSmootherAtLevel(level)->Mult(*R[level], *Z[level]); // z = S r
-   add(*Y[level], 1.0, *Z[level], *Y[level]);             // x = x + S (b - A x)
-}
-
-void Multigrid::Cycle(int level) const
-{
-   if (level == 0)
-   {
-      GetSmootherAtLevel(level)->Mult(*X[level], *Y[level]);
-      return;
-   }
-
-   for (int i = 0; i < preSmoothingSteps; i++)
-   {
-      SmoothingStep(level);
-   }
-
-   // Compute residual
-   GetOperatorAtLevel(level)->Mult(*Y[level], *R[level]);
-   subtract(*X[level], *R[level], *R[level]);
-
-   // Restrict residual
-   fespaces.GetProlongationAtLevel(level - 1)->MultTranspose(*R[level],
-                                                             *X[level - 1]);
-
-   // Init zeros
-   *Y[level - 1] = 0.0;
-
-   // Corrections
-   int corrections = 1;
-   if (cycleType == CycleType::WCYCLE)
-   {
-      corrections = 2;
-   }
-   for (int correction = 0; correction < corrections; ++correction)
-   {
-      Cycle(level - 1);
-   }
-
-   // Prolongate
-   fespaces.GetProlongationAtLevel(level - 1)->Mult(*Y[level - 1], *R[level]);
-
-   // Add update
-   *Y[level] += *R[level];
-
-   // Post-smooth
-   for (int i = 0; i < postSmoothingSteps; i++)
-   {
-      SmoothingStep(level);
-   }
-}
-
-void Multigrid::FormFineLinearSystem(Vector& x, Vector& b, OperatorHandle& A,
-                                     Vector& X, Vector& B)
-{
-   bfs.Last()->FormLinearSystem(*essentialTrueDofs.Last(), x, b, A, X, B);
-}
-
-void Multigrid::RecoverFineFEMSolution(const Vector& X, const Vector& b,
-                                       Vector& x)
-{
-   bfs.Last()->RecoverFEMSolution(X, b, x);
-}
-
-} // namespace mfem
@@ -1,119 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-
-#ifndef MFEM_MULTIGRID
-#define MFEM_MULTIGRID
-
-#include "fespacehierarchy.hpp"
-#include "bilinearform.hpp"
-
-#include "../linalg/operator.hpp"
-#include "../linalg/handle.hpp"
-
-namespace mfem
-{
-
-/// Multigrid solver class
-class Multigrid : public Solver
-{
-public:
-   enum class CycleType
-   {
-      VCYCLE,
-      WCYCLE
-   };
-
-protected:
-   const FiniteElementSpaceHierarchy& fespaces;
-   Array<Array<int>*> essentialTrueDofs;
-   Array<BilinearForm*> bfs;
-
-private:
-   Array<Operator*> operators;
-   Array<Solver*> smoothers;
-
-   Array<bool> ownedOperators;
-   Array<bool> ownedSmoothers;
-
-   CycleType cycleType;
-   int preSmoothingSteps;
-   int postSmoothingSteps;
-
-   mutable Array<Vector*> X;
-   mutable Array<Vector*> Y;
-   mutable Array<Vector*> R;
-   mutable Array<Vector*> Z;
-
-public:
-   /// Constructs an empty multigrid for the given FiniteElementSpaceHierarchy
-   Multigrid(const FiniteElementSpaceHierarchy& fespaces_);
-
-   /// Destructor
-   virtual ~Multigrid();
-
-   /// Adds a level to the multigrid operator hierarchy.
-   /** The ownership of the operators and solvers/smoothers may be transferred
-       to the Multigrid by setting the according boolean variables. */
-   void AddLevel(Operator* opr, Solver* smoother, bool ownOperator,
-                 bool ownSmoother);
-
-   /// Returns the number of levels
-   int NumLevels() const;
-
-   /// Returns the index of the finest level
-   int GetFinestLevelIndex() const;
-
-   /// Returns operator at given level
-   const Operator* GetOperatorAtLevel(int level) const;
-
-   /// Returns operator at given level
-   Operator* GetOperatorAtLevel(int level);
-
-   /// Returns operator at finest level
-   const Operator* GetOperatorAtFinestLevel() const;
-
-   /// Returns operator at finest level
-   Operator* GetOperatorAtFinestLevel();
-
-   /// Returns smoother at given level
-   Solver* GetSmootherAtLevel(int level) const;
-
-   /// Returns smoother at given level
-   Solver* GetSmootherAtLevel(int level);
-
-   /// Set the cycle type and number of pre- and post-smoothing steps used by Mult
-   void SetCycleType(CycleType cycleType_, int preSmoothingSteps_,
-                     int postSmoothingSteps_);
-
-   /// Application of the multigrid as a preconditioner
-   virtual void Mult(const Vector& x, Vector& y) const override;
-
-   /// Not supported for multigrid
-   virtual void SetOperator(const Operator& op) override;
-
-   /// Form the linear system A X = B, corresponding to the operator on the finest level
-   void FormFineLinearSystem(Vector& x, Vector& b, OperatorHandle& A, Vector& X,
-                             Vector& B);
-
-   /// Recover the solution of a linear system formed with FormFineLinearSystem()
-   void RecoverFineFEMSolution(const Vector& X, const Vector& b, Vector& x);
-
-private:
-   /// Application of a smoothing step at particular level
-   void SmoothingStep(int level) const;
-
-   /// Application of a cycle at particular level
-   void Cycle(int level) const;
-};
-
-} // namespace mfem
-
-#endif
@@ -111,8 +111,8 @@ public:
       have zero entries at the essential true dofs. */
   void SetEssentialBC(const Array<int> &bdr_attr_is_ess, Vector *rhs = NULL);

-   /// Specify essential boundary conditions.
-   /** Use either SetEssentialBC() or SetEssentialTrueDofs() if possible. */
+   /// (DEPRECATED) Specify essential boundary conditions.
+   /** @deprecated Use either SetEssentialBC() or SetEssentialTrueDofs(). */
   void SetEssentialVDofs(const Array<int> &ess_vdofs_list);

   /// Specify essential boundary conditions.
@@ -283,14 +283,7 @@ const
   }

   X.Distribute(&x);
-   if (ext)
-   {
-      ext->Mult(X, Y);
-   }
-   else
-   {
-      mat->Mult(X, Y);
-   }
+   mat->Mult(X, Y);
   pfes->Dof_TrueDof_Matrix()->MultTranspose(a, Y, 1.0, y);
 }

@@ -2029,6 +2029,7 @@ int ParFiniteElementSpace
               if (!slave_dofs.Size()) { continue; }

               sf.OrientedPointMatrix(T.GetPointMat());
+               T.FinalizeTransformation();
               fe->GetLocalInterpolation(T, I);

               // make each slave DOF dependent on all master DOFs
@@ -240,7 +240,7 @@ public:
   int GetNRanks() const { return NRanks; }
   int GetMyRank() const { return MyRank; }

-   inline ParMesh *GetParMesh() const { return pmesh; }
+   inline ParMesh *GetParMesh() { return pmesh; }

   int GetDofSign(int i)
   { return NURBSext || Nonconforming() ? 1 : ldof_sign[VDofToDof(i)]; }
@@ -16,7 +16,6 @@
 #include "fem.hpp"
 #include <iostream>
 #include <limits>
-#include <string>
 #include "../general/forall.hpp"
 using namespace std;

@@ -79,229 +78,6 @@ ParGridFunction::ParGridFunction(ParMesh *pmesh, std::istream &input)
   fes = pfes;
 }

-ParGridFunction::ParGridFunction(ParFiniteElementSpace *pf,
-                                 const char *_filename)
-   : GridFunction(pf), pfes(pf)
-{
-   MPI_Comm fes_comm;
-   int fes_rank, n_fes_ranks;
-   fes_comm = pfes->GetComm();
-   MPI_Comm_size(fes_comm, &n_fes_ranks);
-   MPI_Comm_rank(fes_comm, &fes_rank);
-
-   std::string filename(_filename);
-   std::string file_prefix;
-   std::string file_ext;
-   {
-      size_t i = filename.rfind('.', filename.length());
-      if (i != string::npos)
-      {
-         file_prefix = (filename.substr(0, i));
-         file_ext = (filename.substr(i, filename.length() - i));
-      }
-   }
-
-   int nfiles = 1;
-   if (fes_rank == 0)
-   {
-      int n_rfes_ranks;
-      int tmp[2];
-      std::string mpi_filename;
-      size_t i = filename.rfind('.', filename.length());
-      if (i != string::npos)
-      {
-         mpi_filename = file_prefix + to_string(0) + file_ext;
-      }
-      else
-      {
-         mpi_filename = filename + to_string(0);
-      }
-      
-      MPI_File fh;
-      MPI_File_open(MPI_COMM_SELF, mpi_filename.c_str(), MPI_MODE_RDONLY,
-                                                      MPI_INFO_NULL, &fh);
-      MPI_File_read_at(fh, 0, tmp, 2, MPI_INT, MPI_STATUS_IGNORE);
-      MPI_File_close(&fh);
-      
-      n_rfes_ranks = tmp[0];
-      nfiles = tmp[1];
-
-      MFEM_ASSERT(n_fes_ranks == n_rfes_ranks,
-         "ParGridFunction::ParGridFunction(ParFiniteElementSpace *pf,"
-               " const char *_filename):\n"
-               "\tThe number of MPI ranks used to save the GridFunction is\n"
-               "\tnot the same as the number used to load it!");
-   }
-   MPI_Bcast(&nfiles, 1, MPI_INT, 0, fes_comm);
-
-   int color = fes_rank * nfiles / n_fes_ranks;
-   
-   MPI_Comm file_comm;
-   MPI_Comm_split(fes_comm, color, fes_rank, &file_comm);
-
-   int file_rank, n_file_ranks;
-   MPI_Comm_size(file_comm, &n_file_ranks);
-   MPI_Comm_rank(file_comm, &file_rank);
-
-   std::string mpi_filename;
-   {
-      size_t i = filename.rfind('.', filename.length());
-      if (i != string::npos) {
-         mpi_filename = file_prefix + std::to_string(color) + file_ext;
-      }
-      else
-      {
-         mpi_filename = filename + std::to_string(color);
-      }
-   }
-
-   MPI_File fh;
-   MPI_File_open(file_comm, mpi_filename.c_str(), MPI_MODE_RDONLY,
-                                                  MPI_INFO_NULL, &fh);
-
-   int *dof_counts = new int[5*n_file_ranks];
-   int **nv = new int*[n_file_ranks];
-   int **nvdofs = new int*[n_file_ranks];
-   int **nedofs = new int*[n_file_ranks];
-   int **nfdofs = new int*[n_file_ranks];
-   int **nrdofs = new int*[n_file_ranks];
-
-   for (int i = 0; i < n_file_ranks; ++i)
-   {
-      nv[i]     = &dof_counts[i*5+0];
-      nvdofs[i] = &dof_counts[i*5+1];
-      nedofs[i] = &dof_counts[i*5+2];
-      nfdofs[i] = &dof_counts[i*5+3];
-      nrdofs[i] = &dof_counts[i*5+4];
-   }
-
-   *nv[file_rank] = pfes->GetVSize();
-   *nvdofs[file_rank] = pfes->GetNVDofs();
-   *nedofs[file_rank] = pfes->GetNEDofs();
-   *nfdofs[file_rank] = pfes->GetNFDofs();
-
-   int vdim = pfes->GetVDim();
-   *nrdofs[file_rank] = *nv[file_rank] / vdim - *nvdofs[file_rank] -
-                                       *nedofs[file_rank] - *nfdofs[file_rank];
-
-   MPI_Allgather(MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, &dof_counts[0], 5,
-                 MPI_INT, file_comm);
-
-   double *data_ = HostWrite();
-
-   MPI_Offset header_offset = 0;
-   header_offset += 2 * sizeof(int);
-   MPI_Offset v_offset, e_offset, f_offset, r_offset;
-
-   int total_vdofs = 0, total_edofs = 0, total_fdofs = 0, total_rdofs = 0;
-   int total_scalar_dofs = 0;
-
-   for (int i = 0; i < n_file_ranks; ++i)
-   {
-      total_vdofs += *nvdofs[i];
-      total_edofs += *nedofs[i];
-      total_fdofs += *nfdofs[i];
-      total_rdofs += *nrdofs[i];
-      total_scalar_dofs += *nv[i];
-   }
-
-   total_scalar_dofs /= vdim;
-
-   if (pfes->GetOrdering() == Ordering::byNODES)
-   {
-      for (int d = 0; d < vdim; ++d)
-      {
-         int v_data_offset = 0 + *nv[file_rank] * d / vdim ;
-         int e_data_offset = v_data_offset + *nvdofs[file_rank];
-         int f_data_offset = e_data_offset + *nedofs[file_rank];
-         int r_data_offset = f_data_offset + *nfdofs[file_rank];
-
-         v_offset = header_offset;
-         e_offset = header_offset;
-         f_offset = header_offset;
-         r_offset = header_offset;
-         
-         v_offset += total_scalar_dofs * d * sizeof(double);
-         e_offset += (total_vdofs + total_scalar_dofs * d) * sizeof(double);
-         f_offset += (total_vdofs + total_edofs +
-                                    total_scalar_dofs * d) * sizeof(double);
-         r_offset += (total_vdofs + total_edofs + total_fdofs +
-                                    total_scalar_dofs * d) * sizeof(double);
-
-
-         for (int i = 0; i < file_rank; ++i)
-         {
-            v_offset += *nvdofs[i] * sizeof(double);
-            e_offset += *nedofs[i] * sizeof(double);
-            f_offset += *nfdofs[i] * sizeof(double);
-            r_offset += *nrdofs[i] * sizeof(double);
-         }
-
-         MPI_File_read_at_all(fh, v_offset, &data_[v_data_offset],
-                              *nvdofs[file_rank], MPI_DOUBLE,
-                              MPI_STATUS_IGNORE);
-         MPI_File_read_at_all(fh, e_offset, &data_[e_data_offset],
-                              *nedofs[file_rank], MPI_DOUBLE,
-                              MPI_STATUS_IGNORE);
-         MPI_File_read_at_all(fh, f_offset, &data_[f_data_offset],
-                              *nfdofs[file_rank], MPI_DOUBLE,
-                              MPI_STATUS_IGNORE);
-         MPI_File_read_at_all(fh, r_offset, &data_[r_data_offset],
-                              *nrdofs[file_rank], MPI_DOUBLE,
-                              MPI_STATUS_IGNORE);
-      }
-   }
-   else
-   {
-      v_offset = header_offset;
-      e_offset = v_offset + total_vdofs * vdim * sizeof(double);
-      f_offset = e_offset + total_edofs * vdim * sizeof(double);
-      r_offset = f_offset + total_fdofs * vdim * sizeof(double);
-
-      for (int i = 0; i < file_rank; ++i)
-      {
-         v_offset += *nvdofs[i] * sizeof(double) * vdim;
-         e_offset += *nedofs[i] * sizeof(double) * vdim;
-         f_offset += *nfdofs[i] * sizeof(double) * vdim;
-         r_offset += *nrdofs[i] * sizeof(double) * vdim;
-      }
-
-      int v_data_offset = 0;
-      int e_data_offset = v_data_offset + *nvdofs[file_rank] * vdim;
-      int f_data_offset = e_data_offset + *nedofs[file_rank] * vdim;
-      int r_data_offset = f_data_offset + *nfdofs[file_rank] * vdim;
-      
-      MPI_File_read_at_all(fh, v_offset, &data_[v_data_offset],
-                           *nvdofs[file_rank] * vdim, MPI_DOUBLE,
-                           MPI_STATUS_IGNORE);
-      MPI_File_read_at_all(fh, e_offset, &data_[e_data_offset],
-                           *nedofs[file_rank] * vdim, MPI_DOUBLE,
-                           MPI_STATUS_IGNORE);
-      MPI_File_read_at_all(fh, f_offset, &data_[f_data_offset],
-                           *nfdofs[file_rank] * vdim, MPI_DOUBLE,
-                           MPI_STATUS_IGNORE);
-      MPI_File_read_at_all(fh, r_offset, &data_[r_data_offset],
-                           *nrdofs[file_rank] * vdim, MPI_DOUBLE,
-                           MPI_STATUS_IGNORE);
-   }
-
-   MPI_File_close(&fh);
-   MPI_Comm_free(&file_comm);
-
-   for (int i = 0; i < size; i++)
-   {
-      if (pfes->GetDofSign(i) < 0) { data_[i] = -data_[i]; }
-   }
-
-   delete[] dof_counts;
-   delete[] nv;
-   delete[] nvdofs;
-   delete[] nedofs;
-   delete[] nfdofs;
-   delete[] nrdofs;
-}
-
-
 void ParGridFunction::Update()
 {
   face_nbr_data.Destroy();
@@ -722,221 +498,6 @@ void ParGridFunction::Save(std::ostream &out) const
   }
 }

-#ifdef MFEM_USE_ADIOS2
-void ParGridFunction::Save(adios2stream &out,
-                           const std::string& variable_name,
-                           const adios2stream::data_type type) const
-{
-   double *data_  = const_cast<double*>(HostRead());
-   for (int i = 0; i < size; i++)
-   {
-      if (pfes->GetDofSign(i) < 0) { data_[i] = -data_[i]; }
-   }
-
-   GridFunction::Save(out, variable_name, type);
-
-   for (int i = 0; i < size; i++)
-   {
-      if (pfes->GetDofSign(i) < 0) { data_[i] = -data_[i]; }
-   }
-}
-#endif
-
-void ParGridFunction::Save(const char *_filename, const int nfiles)
-{
-   MPI_Comm fes_comm;
-   int fes_rank, n_fes_ranks;
-   fes_comm = pfes->GetComm();
-
-   MPI_Comm_size(fes_comm, &n_fes_ranks);
-   MPI_Comm_rank(fes_comm, &fes_rank);
-
-   int color = fes_rank * nfiles / n_fes_ranks;
-   
-   MPI_Comm file_comm;
-   MPI_Comm_split(fes_comm, color, fes_rank, &file_comm);
-
-   int file_rank, n_file_ranks;
-   MPI_Comm_size(file_comm, &n_file_ranks);
-   MPI_Comm_rank(file_comm, &file_rank);
-
-   std::string filename(_filename);
-   std::string file_prefix;
-   std::string file_ext;
-   std::string mpi_filename;
-   {
-      size_t i = filename.rfind('.', filename.length());
-      if (i != string::npos)
-      {
-         file_prefix = (filename.substr(0, i));
-         file_ext = (filename.substr(i, filename.length() - i));
-         mpi_filename = file_prefix + std::to_string(color) + file_ext;
-      }
-      else
-      {
-         mpi_filename = filename + std::to_string(color);
-      }
-   }
-
-   MPI_File fh;
-   MPI_File_open(file_comm, mpi_filename.c_str(), MPI_MODE_CREATE |
-                                                  MPI_MODE_WRONLY,
-                                                  MPI_INFO_NULL, &fh);
-
-   int *dof_counts = new int[5*n_file_ranks];
-   int **nv = new int*[n_file_ranks];
-   int **nvdofs = new int*[n_file_ranks];
-   int **nedofs = new int*[n_file_ranks];
-   int **nfdofs = new int*[n_file_ranks];
-   int **nrdofs = new int*[n_file_ranks];
-
-   for (int i = 0; i < n_file_ranks; ++i)
-   {
-      nv[i]     = &dof_counts[i*5+0];
-      nvdofs[i] = &dof_counts[i*5+1];
-      nedofs[i] = &dof_counts[i*5+2];
-      nfdofs[i] = &dof_counts[i*5+3];
-      nrdofs[i] = &dof_counts[i*5+4];
-   }
-
-   *nv[file_rank] = pfes->GetVSize();
-   *nvdofs[file_rank] = pfes->GetNVDofs();
-   *nedofs[file_rank] = pfes->GetNEDofs();
-   *nfdofs[file_rank] = pfes->GetNFDofs();
-
-   int vdim = pfes->GetVDim();
-   *nrdofs[file_rank] = *nv[file_rank] / vdim - *nvdofs[file_rank] -
-                                       *nedofs[file_rank] - *nfdofs[file_rank];
-
-   MPI_Allgather(MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, &dof_counts[0], 5,
-                 MPI_INT, file_comm);
-
-   double *data_  = const_cast<double*>(HostRead());
-   for (int i = 0; i < size; i++)
-   {
-      if (pfes->GetDofSign(i) < 0) { data_[i] = -data_[i]; }
-   }
-
-   MPI_Offset header_offset = 0;
-
-   if (file_rank == 0)
-   {
-      int tmp[] = {n_fes_ranks, nfiles};
-      MPI_File_write_at(fh, header_offset, &tmp, 2, MPI_INT,
-                        MPI_STATUS_IGNORE);
-   }
-
-   header_offset += 2 * sizeof(int);
-
-   MPI_Offset v_offset, e_offset, f_offset, r_offset;
-
-   int total_vdofs = 0, total_edofs = 0, total_fdofs = 0, total_rdofs = 0;
-   int total_scalar_dofs = 0;
-
-   for (int i = 0; i < n_file_ranks; ++i)
-   {
-      total_vdofs += *nvdofs[i];
-      total_edofs += *nedofs[i];
-      total_fdofs += *nfdofs[i];
-      total_rdofs += *nrdofs[i];
-      total_scalar_dofs += *nv[i];
-   }
-
-   total_scalar_dofs /= vdim;
-
-   if (pfes->GetOrdering() == Ordering::byNODES)
-   {
-      for (int d = 0; d < vdim; ++d)
-      {
-         int v_data_offset = 0 + *nv[file_rank] * d / vdim ;
-         int e_data_offset = v_data_offset + *nvdofs[file_rank];
-         int f_data_offset = e_data_offset + *nedofs[file_rank];
-         int r_data_offset = f_data_offset + *nfdofs[file_rank];
-
-         v_offset = header_offset;
-         e_offset = header_offset;
-         f_offset = header_offset;
-         r_offset = header_offset;
-         
-         v_offset += total_scalar_dofs * d * sizeof(double);
-         e_offset += (total_vdofs + total_scalar_dofs * d) * sizeof(double);
-         f_offset += (total_vdofs + total_edofs +
-                                    total_scalar_dofs * d) * sizeof(double);
-         r_offset += (total_vdofs + total_edofs + total_fdofs +
-                                    total_scalar_dofs * d) * sizeof(double);
-
-         for (int i = 0; i < file_rank; ++i)
-         {
-            v_offset += *nvdofs[i] * sizeof(double);
-            e_offset += *nedofs[i] * sizeof(double);
-            f_offset += *nfdofs[i] * sizeof(double);
-            r_offset += *nrdofs[i] * sizeof(double);
-         }
-
-         MPI_File_write_at_all(fh, v_offset, &data_[v_data_offset],
-                               *nvdofs[file_rank], MPI_DOUBLE,
-                               MPI_STATUS_IGNORE);
-         MPI_File_write_at_all(fh, e_offset, &data_[e_data_offset],
-                               *nedofs[file_rank], MPI_DOUBLE,
-                               MPI_STATUS_IGNORE);
-         MPI_File_write_at_all(fh, f_offset, &data_[f_data_offset],
-                               *nfdofs[file_rank], MPI_DOUBLE,
-                               MPI_STATUS_IGNORE);
-         MPI_File_write_at_all(fh, r_offset, &data_[r_data_offset],
-                               *nrdofs[file_rank], MPI_DOUBLE,
-                               MPI_STATUS_IGNORE);
-      }
-   }
-   else
-   {
-      v_offset = header_offset;
-      e_offset = v_offset + total_vdofs * vdim * sizeof(double);
-      f_offset = e_offset + total_edofs * vdim * sizeof(double);
-      r_offset = f_offset + total_fdofs * vdim * sizeof(double);
-
-      for (int i = 0; i < file_rank; ++i)
-      {
-         v_offset += *nvdofs[i] * sizeof(double) * vdim;
-         e_offset += *nedofs[i] * sizeof(double) * vdim;
-         f_offset += *nfdofs[i] * sizeof(double) * vdim;
-         r_offset += *nrdofs[i] * sizeof(double) * vdim;
-      }
-
-      int v_data_offset = 0;
-      int e_data_offset = v_data_offset + *nvdofs[file_rank] * vdim;
-      int f_data_offset = e_data_offset + *nedofs[file_rank] * vdim;
-      int r_data_offset = f_data_offset + *nfdofs[file_rank] * vdim;
-      
-      MPI_File_write_at_all(fh, v_offset, &data_[v_data_offset],
-                            *nvdofs[file_rank] * vdim, MPI_DOUBLE,
-                            MPI_STATUS_IGNORE);
-      MPI_File_write_at_all(fh, e_offset, &data_[e_data_offset],
-                            *nedofs[file_rank] * vdim, MPI_DOUBLE,
-                            MPI_STATUS_IGNORE);
-      MPI_File_write_at_all(fh, f_offset, &data_[f_data_offset],
-                            *nfdofs[file_rank] * vdim, MPI_DOUBLE,
-                            MPI_STATUS_IGNORE);
-      MPI_File_write_at_all(fh, r_offset, &data_[r_data_offset],
-                            *nrdofs[file_rank] * vdim, MPI_DOUBLE,
-                            MPI_STATUS_IGNORE);
-   }
-
-   MPI_File_close(&fh);
-   MPI_Comm_free(&file_comm);
-   
-   for (int i = 0; i < size; i++)
-   {
-      if (pfes->GetDofSign(i) < 0) { data_[i] = -data_[i]; }
-   }
-
-   delete[] dof_counts;
-   delete[] nv;
-   delete[] nvdofs;
-   delete[] nedofs;
-   delete[] nfdofs;
-   delete[] nrdofs;
-}
-
 void ParGridFunction::SaveAsOne(std::ostream &out)
 {
   int i, p;
@@ -83,13 +83,6 @@ public:
       constructed. The new ParGridFunction assumes ownership of both. */
   ParGridFunction(ParMesh *pmesh, std::istream &input);

-   /// Construct a ParGridFunction by loading a ParGridFunction saved using
-   /// ParGridFunction::Save(char *filename, int nfiles).
-   /** The parallel space @a *pf and the space used by the GridFunction saved
-       in @a *filename should match. The number of ranks used when loading the
-       ParGridFunction must be the same as when it was saved. */
-   ParGridFunction(ParFiniteElementSpace *pf, const char *filename);
-
   /// Copy assignment. Only the data of the base class Vector is copied.
   /** It is assumed that this object and @a rhs use ParFiniteElementSpace%s
       that have the same size.
@@ -317,34 +310,11 @@ public:
                            GridFunction &flux,
                            bool wcoef = true, int subdomain = -1);

-   /** Save the local portion of the ParGridFunction. This differs from the
+   /** Save the local portion of the ParGridFunction. It differs from the
       serial GridFunction::Save in that it takes into account the signs of
       the local dofs. */
   virtual void Save(std::ostream &out) const;

-#ifdef MFEM_USE_ADIOS2
-   /** Save the local portion of the ParGridFunction. This differs from the
-       serial GridFunction::Save in that it takes into account the signs of
-       the local dofs. */
-   virtual void Save(
-      adios2stream &out, const std::string &variable_name,
-      const adios2stream::data_type type = adios2stream::data_type::point_data) const;
-#endif
-
-   /** Save the local grid functions to n number of files, where each file will
-       contain the grid functions from potentially multiple ranks. This is
-       similar to the syncIO approach from "Fu, Jing, et al. 'Scalable parallel
-       I/O alternatives for massively parallel partitioned solver systems.'
-       2010 IEEE International Symposium on Parallel & Distributed Processing,
-       Workshops and Phd Forum (IPDPSW). IEEE, 2010." 
-       @param[in] filename - filename for output files with extension
-       @param[in] nfiles - number of files to write using MPI-IO
-       @note - takes into account the signs of the local dofs.
-       @note - writes a binary file without the FESpace header; the saved file
-       should only be loaded by the accompanying constructor:
-       ParGridFunction(ParFiniteElementSpace *pf, const char *filename) */
-   void Save(const char *filename, const int nfiles = 1);
-
   /// Merge the local grid functions
   void SaveAsOne(std::ostream &out = mfem::out);

@@ -319,33 +319,6 @@ void QuadratureInterpolator::Mult(
         }
      }
   }
-   else if (vdim == 3 && dim == 2)
-   {
-      switch (100*nd + nq)
-      {
-         // Q0
-         case 101: eval_func = &Eval2D<3,1,1>; break;
-         case 104: eval_func = &Eval2D<3,1,4>; break;
-         // Q1
-         case 404: eval_func = &Eval2D<3,4,4>; break;
-         case 409: eval_func = &Eval2D<3,4,9>; break;
-         // Q2
-         case 904: eval_func = &Eval2D<3,9,4>; break;
-         case 909: eval_func = &Eval2D<3,9,9>; break;
-         case 916: eval_func = &Eval2D<3,9,16>; break;
-         case 925: eval_func = &Eval2D<3,9,25>; break;
-         // Q3
-         case 1616: eval_func = &Eval2D<3,16,16>; break;
-         case 1625: eval_func = &Eval2D<3,16,25>; break;
-         case 1636: eval_func = &Eval2D<3,16,36>; break;
-         // Q4
-         case 2525: eval_func = &Eval2D<3,25,25>; break;
-         case 2536: eval_func = &Eval2D<3,25,36>; break;
-         case 2549: eval_func = &Eval2D<3,25,49>; break;
-         case 2564: eval_func = &Eval2D<3,25,64>; break;
-         default:   eval_func = &Eval2D<3>;
-      }
-   }
   else if (vdim == dim)
   {
      if (dim == 2)
@@ -47,7 +47,7 @@ protected:

   static const int MAX_NQ2D = 100;
   static const int MAX_ND2D = 100;
-   static const int MAX_VDIM2D = 3;
+   static const int MAX_VDIM2D = 2;

   static const int MAX_NQ3D = 1000;
   static const int MAX_ND3D = 1000;
@@ -223,43 +223,6 @@ void ElementRestriction::MultTransposeUnsigned(const Vector& x, Vector& y) const
   });
 }

-void ElementRestriction::BooleanMask(Vector& y) const
-{
-   // Assumes all elements have the same number of dofs
-   const int nd = dof;
-   const int vd = vdim;
-   const bool t = byvdim;
-
-   Array<char> processed(vd * ndofs);
-   processed = 0;
-
-   auto d_offsets = offsets.HostRead();
-   auto d_indices = indices.HostRead();
-   auto d_x = Reshape(processed.HostReadWrite(), t?vd:ndofs, t?ndofs:vd);
-   auto d_y = Reshape(y.HostWrite(), nd, vd, ne);
-   for (int i = 0; i < ndofs; ++i)
-   {
-      const int offset = d_offsets[i];
-      const int nextOffset = d_offsets[i+1];
-      for (int c = 0; c < vd; ++c)
-      {
-         for (int j = offset; j < nextOffset; ++j)
-         {
-            const int idx_j = d_indices[j];
-            if (d_x(t?c:i,t?i:c))
-            {
-               d_y(idx_j % nd, c, idx_j / nd) = 0.0;
-            }
-            else
-            {
-               d_y(idx_j % nd, c, idx_j / nd) = 1.0;
-               d_x(t?c:i,t?i:c) = 1;
-            }
-         }
-      }
-   }
-}
-
 /// Return the face degrees of freedom returned in Lexicographic order.
 void GetFaceDofs(const int dim, const int face_id,
                 const int dof1d, Array<int> &faceMap)
@@ -49,14 +49,6 @@ public:

   /// Compute MultTranspose without applying signs based on DOF orientations.
   void MultTransposeUnsigned(const Vector &x, Vector &y) const;
-
-   /// @brief Fills the E-vector y with `boolean` values 0.0 and 1.0 such that each
-   /// each entry of the L-vector is uniquely represented in `y`.
-   /** This means, the sum of the E-vector `y` is equal to the sum of the
-       corresponding L-vector filled with ones. The boolean mask is required to
-       emulate SetSubVector and its transpose on GPUs. This method is running on
-       the host, since the `processed` array requires a large shared memory. */
-   void BooleanMask(Vector& y) const;
 };

 /// Operator that converts L2 FiniteElementSpace L-vectors to E-vectors.
@@ -993,21 +993,17 @@ void DiscreteAdaptTC::SetSerialDiscreteTargetSpec(GridFunction &tspec_)
   tspec_sav = tspec;
 }

-void DiscreteAdaptTC::UpdateTargetSpecification(const Vector &new_x,
-                                                bool use_flag)
+void DiscreteAdaptTC::UpdateTargetSpecification(const Vector &new_x)
 {
-   if (use_flag && good_tspec) { return; }
-
   MFEM_VERIFY(tspec.Size() > 0, "Target specification is not set!");
   adapt_eval->ComputeAtNewPosition(new_x, tspec);
   tspec_sav = tspec;
-
-   good_tspec = use_flag;
 }

 void DiscreteAdaptTC::UpdateTargetSpecification(Vector &new_x,
                                                Vector &IntData)
 {
+   MFEM_VERIFY(tspec.Size() > 0, "Target specification is not set!");
   adapt_eval->ComputeAtNewPosition(new_x, IntData);
 }

@@ -1074,17 +1070,14 @@ void DiscreteAdaptTC::ComputeElementTargets(int e_id, const FiniteElement &fe,
 }

 void DiscreteAdaptTC::UpdateGradientTargetSpecification(const Vector &x,
-                                                        const double dx,
-                                                        bool use_flag)
+                                                        const double dx)
 {
-   if (use_flag && good_tspec_grad) { return; }
-
   const int dim = tspec_fes->GetFE(0)->GetDim();
   const int cnt = x.Size()/dim;

-   if (tspec_pert1h.Size() != x.Size())
+   if (tspec_perth.Size() != x.Size())
   {
-      tspec_pert1h.SetSize(x.Size());
+      tspec_perth.SetSize(x.Size());
   }

   Vector TSpecTemp;
@@ -1093,20 +1086,16 @@ void DiscreteAdaptTC::UpdateGradientTargetSpecification(const Vector &x,
   {
      for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) += dx; }

-      TSpecTemp.SetDataAndSize(tspec_pert1h.GetData() + j*cnt, cnt);
+      TSpecTemp.SetDataAndSize(tspec_perth.GetData() + j*cnt, cnt);
      UpdateTargetSpecification(xtemp, TSpecTemp);

      for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) -= dx; }
   }
-
-   good_tspec_grad = use_flag;
 }

 void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
-                                                       double dx, bool use_flag)
+                                                       const double dx)
 {
-   if (use_flag && good_tspec_hess) { return; }
-
   const int dim = tspec_fes->GetFE(0)->GetDim();
   const int cnt = x.Size()/dim;

@@ -1153,8 +1142,6 @@ void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
         idx++;
      }
   }
-
-   good_tspec_hess = use_flag;
 }

 void AdaptivityEvaluator::SetSerialMetaInfo(const Mesh &m,
@@ -1193,8 +1180,19 @@ void TMOP_Integrator::EnableLimiting(const GridFunction &n0,
                                     const GridFunction &dist, Coefficient &w0,
                                     TMOP_LimiterFunction *lfunc)
 {
-   EnableLimiting(n0, w0, lfunc);
+   nodes0 = &n0;
+   coeff0 = &w0;
   lim_dist = &dist;
+
+   delete lim_func;
+   if (lfunc)
+   {
+      lim_func = lfunc;
+   }
+   else
+   {
+      lim_func = new TMOP_QuadraticLimiter;
+   }
 }
 void TMOP_Integrator::EnableLimiting(const GridFunction &n0, Coefficient &w0,
                                     TMOP_LimiterFunction *lfunc)
@@ -1812,111 +1810,6 @@ void TMOP_Integrator::EnableFiniteDifferences(const ParGridFunction &x)
 }
 #endif

-void TMOPComboIntegrator::EnableLimiting(const GridFunction &n0,
-                                         const GridFunction &dist,
-                                         Coefficient &w0,
-                                         TMOP_LimiterFunction *lfunc)
-{
-   MFEM_VERIFY(tmopi.Size() > 0, "No TMOP_Integrators were added.");
-
-   tmopi[0]->EnableLimiting(n0, dist, w0, lfunc);
-   for (int i = 1; i < tmopi.Size(); i++) { tmopi[i]->DisableLimiting(); }
-}
-
-void TMOPComboIntegrator::EnableLimiting(const GridFunction &n0,
-                                         Coefficient &w0,
-                                         TMOP_LimiterFunction *lfunc)
-{
-   MFEM_VERIFY(tmopi.Size() > 0, "No TMOP_Integrators were added.");
-
-   tmopi[0]->EnableLimiting(n0, w0, lfunc);
-   for (int i = 1; i < tmopi.Size(); i++) { tmopi[i]->DisableLimiting(); }
-}
-
-void TMOPComboIntegrator::SetLimitingNodes(const GridFunction &n0)
-{
-   MFEM_VERIFY(tmopi.Size() > 0, "No TMOP_Integrators were added.");
-
-   tmopi[0]->SetLimitingNodes(n0);
-   for (int i = 1; i < tmopi.Size(); i++) { tmopi[i]->DisableLimiting(); }
-}
-
-double TMOPComboIntegrator::GetElementEnergy(const FiniteElement &el,
-                                             ElementTransformation &T,
-                                             const Vector &elfun)
-{
-   double energy= 0.0;
-   for (int i = 0; i < tmopi.Size(); i++)
-   {
-      energy += tmopi[i]->GetElementEnergy(el, T, elfun);
-   }
-   return energy;
-}
-
-void TMOPComboIntegrator::AssembleElementVector(const FiniteElement &el,
-                                                ElementTransformation &T,
-                                                const Vector &elfun,
-                                                Vector &elvect)
-{
-   MFEM_VERIFY(tmopi.Size() > 0, "No TMOP_Integrators were added.");
-
-   tmopi[0]->AssembleElementVector(el, T, elfun, elvect);
-   for (int i = 1; i < tmopi.Size(); i++)
-   {
-      Vector elvect_i;
-      tmopi[i]->AssembleElementVector(el, T, elfun, elvect_i);
-      elvect += elvect_i;
-   }
-}
-
-void TMOPComboIntegrator::AssembleElementGrad(const FiniteElement &el,
-                                              ElementTransformation &T,
-                                              const Vector &elfun,
-                                              DenseMatrix &elmat)
-{
-   MFEM_VERIFY(tmopi.Size() > 0, "No TMOP_Integrators were added.");
-
-   tmopi[0]->AssembleElementGrad(el, T, elfun, elmat);
-   for (int i = 1; i < tmopi.Size(); i++)
-   {
-      DenseMatrix elmat_i;
-      tmopi[i]->AssembleElementGrad(el, T, elfun, elmat_i);
-      elmat += elmat_i;
-   }
-}
-
-void TMOPComboIntegrator::EnableNormalization(const GridFunction &x)
-{
-   const int cnt = tmopi.Size();
-   double total_integral = 0.0;
-   for (int i = 0; i < cnt; i++)
-   {
-      tmopi[i]->EnableNormalization(x);
-      total_integral += 1.0 / tmopi[i]->metric_normal;
-   }
-   for (int i = 0; i < cnt; i++)
-   {
-      tmopi[i]->metric_normal = 1.0 / total_integral;
-   }
-}
-
-#ifdef MFEM_USE_MPI
-void TMOPComboIntegrator::ParEnableNormalization(const ParGridFunction &x)
-{
-   const int cnt = tmopi.Size();
-   double total_integral = 0.0;
-   for (int i = 0; i < cnt; i++)
-   {
-      tmopi[i]->ParEnableNormalization(x);
-      total_integral += 1.0 / tmopi[i]->metric_normal;
-   }
-   for (int i = 0; i < cnt; i++)
-   {
-      tmopi[i]->metric_normal = 1.0 / total_integral;
-   }
-}
-#endif
-

 void InterpolateTMOP_QualityMetric(TMOP_QualityMetric &metric,
                                   const TargetConstructor &tc,
@@ -710,7 +710,7 @@ protected:
   // Data is owned, updated by UpdateTargetSpecification.
   Vector tspec;             //eta(x)
   Vector tspec_sav;
-   Vector tspec_pert1h;      //eta(x+h)
+   Vector tspec_perth;       //eta(x+h)
   Vector tspec_pert2h;      //eta(x+2*h)
   Vector tspec_pertmix;     //eta(x+h,y+h)

@@ -718,10 +718,6 @@ protected:
   // positions corresponding to the values of tspec.
   const FiniteElementSpace *tspec_fes;

-   // These flags can be used by outside functions to avoid recomputing
-   // the tspec and tspec_perth fields again on the same mesh.
-   bool good_tspec, good_tspec_grad, good_tspec_hess;
-
   // Evaluation of the discrete target specification on different meshes.
   // Owned.
   AdaptivityEvaluator *adapt_eval;
@@ -729,10 +725,7 @@ protected:
 public:
   DiscreteAdaptTC(TargetType ttype)
      : TargetConstructor(ttype),
-        tspec(), tspec_sav(), tspec_pert1h(), tspec_pert2h(), tspec_pertmix(),
-        tspec_fes(NULL),
-        good_tspec(false), good_tspec_grad(false), good_tspec_hess(false),
-        adapt_eval(NULL) { }
+        tspec(), tspec_fes(NULL), adapt_eval(NULL) { }

   virtual ~DiscreteAdaptTC() { delete adapt_eval; }

@@ -741,14 +734,9 @@ public:
   virtual void SetParDiscreteTargetSpec(ParGridFunction &tspec_);
 #endif

-   /// Used in combination with the Update methods to avoid extra computations.
-   void ResetUpdateFlags()
-   { good_tspec = good_tspec_grad = good_tspec_hess = false; }
-
   /** Used to update the target specification after the mesh has changed. The
-       new mesh positions are given by new_x. If @a use_flags is true, repeated
-       calls won't do anything until ResetUpdateFlags() is called. */
-   void UpdateTargetSpecification(const Vector &new_x, bool use_flag = false);
+       new mesh positions are given by new_x. */
+   void UpdateTargetSpecification(const Vector &new_x);

   void UpdateTargetSpecification(Vector &new_x, Vector &IntData);

@@ -759,17 +747,12 @@ public:
   void RestoreTargetSpecificationAtNode(ElementTransformation &T, int nodenum);

   /** Used for finite-difference based computations. Computes the target
-       specifications after a mesh perturbation in x or y direction.
-       If @a use_flags is true, repeated calls won't do anything until
-       ResetUpdateFlags() is called. */
-   void UpdateGradientTargetSpecification(const Vector &x, double dx,
-                                          bool use_flag = false);
+       specifications after a mesh perturbation in x or y direction. */
+   void UpdateGradientTargetSpecification(const Vector &x, const double dx);
+
   /** Used for finite-difference based computations. Computes the target
-       specifications after two mesh perturbations in x and/or y direction.
-       If @a use_flags is true, repeated calls won't do anything until
-       ResetUpdateFlags() is called. */
-   void UpdateHessianTargetSpecification(const Vector &x, double dx,
-                                         bool use_flag = false);
+       specifications after two mesh perturbations in x and/or y direction. */
+   void UpdateHessianTargetSpecification(const Vector &x, const double dx);

   void SetAdaptivityEvaluator(AdaptivityEvaluator *ae)
   {
@@ -777,7 +760,7 @@ public:
      adapt_eval = ae;
   }

-   const Vector &GetTspecPert1H()   { return tspec_pert1h; }
+   const Vector &GetTspecPert1H()   { return tspec_perth; }
   const Vector &GetTspecPert2H()   { return tspec_pert2h; }
   const Vector &GetTspecPertMixH() { return tspec_pertmix; }

@@ -793,6 +776,7 @@ public:
 };

 class TMOPNewtonSolver;
+class TMOPDescentNewtonSolver;

 /** @brief A TMOP integrator class based on any given TMOP_QualityMetric and
    TargetConstructor.
@@ -805,8 +789,7 @@ class TMOP_Integrator : public NonlinearFormIntegrator
 {
 protected:
   friend class TMOPNewtonSolver;
-   friend class TMOPComboIntegrator;
-
+   friend class TMOPDescentNewtonSolver;
   TMOP_QualityMetric *metric;        // not owned
   const TargetConstructor *targetC;  // not owned

@@ -882,11 +865,6 @@ protected:
 #endif
   void ComputeMinJac(const Vector &x, const FiniteElementSpace &fes);

-   void DisableLimiting()
-   {
-      nodes0 = NULL; coeff0 = NULL; lim_dist = NULL; lim_func = NULL;
-   }
-
 public:
   /** @param[in] m  TMOP_QualityMetric that will be integrated (not owned).
       @param[in] tc Target-matrix construction algorithm to use (not owned). */
@@ -933,8 +911,8 @@ public:

   /** @brief Adds a limiting term to the integrator with limiting distance
       function (@a dist in the general version of the method) equal to 1. */
-   void EnableLimiting(const GridFunction &n0, Coefficient &w0,
-                       TMOP_LimiterFunction *lfunc = NULL);
+   void EnableLimiting(const GridFunction &n0,
+                       Coefficient &w0, TMOP_LimiterFunction *lfunc = NULL);

   /// Update the original/reference nodes used for limiting.
   void SetLimitingNodes(const GridFunction &n0) { nodes0 = &n0; }
@@ -955,7 +933,7 @@ public:
                                    ElementTransformation &T,
                                    const Vector &elfun, DenseMatrix &elmat);

-   DiscreteAdaptTC *GetDiscreteAdaptTC() const { return discr_tc; }
+   DiscreteAdaptTC *GetDiscreteAdaptTC() { return discr_tc; }

   /** @brief Computes the normalization factors of the metric and limiting
       integrals using the mesh position given by @a x. */
@@ -975,59 +953,13 @@ public:
   double GetFDh()    const { return dx; }
 };

-class TMOPComboIntegrator : public NonlinearFormIntegrator
-{
-protected:
-   // Integrators in the combination. Owned.
-   Array<TMOP_Integrator *> tmopi;
-
-public:
-   TMOPComboIntegrator() : tmopi(0) { }
-
-   ~TMOPComboIntegrator()
-   {
-      for (int i = 0; i < tmopi.Size(); i++) { delete tmopi[i]; }
-   }
-
-   /// Adds a new TMOP_Integrator to the combination.
-   void AddTMOPIntegrator(TMOP_Integrator *ti) { tmopi.Append(ti); }
-
-   Array<TMOP_Integrator *> GetTMOPIntegrators() const { return tmopi; }
-
-   /// Adds the limiting term to the first integrator. Disables it for the rest.
-   void EnableLimiting(const GridFunction &n0, const GridFunction &dist,
-                       Coefficient &w0, TMOP_LimiterFunction *lfunc = NULL);
-
-   /** @brief Adds the limiting term to the first integrator. Disables it for
-       the rest (@a dist in the general version of the method) equal to 1. */
-   void EnableLimiting(const GridFunction &n0, Coefficient &w0,
-                       TMOP_LimiterFunction *lfunc = NULL);
-
-   /// Update the original/reference nodes used for limiting.
-   void SetLimitingNodes(const GridFunction &n0);
-
-   virtual double GetElementEnergy(const FiniteElement &el,
-                                   ElementTransformation &T,
-                                   const Vector &elfun);
-   virtual void AssembleElementVector(const FiniteElement &el,
-                                      ElementTransformation &T,
-                                      const Vector &elfun, Vector &elvect);
-   virtual void AssembleElementGrad(const FiniteElement &el,
-                                    ElementTransformation &T,
-                                    const Vector &elfun, DenseMatrix &elmat);
-
-   /// Normalization factor that considers all integrators in the combination.
-   void EnableNormalization(const GridFunction &x);
-#ifdef MFEM_USE_MPI
-   void ParEnableNormalization(const ParGridFunction &x);
-#endif
-};

 /// Interpolates the @a metric's values at the nodes of @a metric_gf.
 /** Assumes that @a metric_gf's FiniteElementSpace is initialized. */
 void InterpolateTMOP_QualityMetric(TMOP_QualityMetric &metric,
                                   const TargetConstructor &tc,
                                   const Mesh &mesh, GridFunction &metric_gf);
+
 }

 #endif
@@ -61,10 +61,10 @@ void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
   ode_solver.Init(*oper);

   // Compute some time step [mesh_size / speed].
-   double h_min = std::numeric_limits<double>::infinity();
+   double min_h = std::numeric_limits<double>::infinity();
   for (int i = 0; i < m->GetNE(); i++)
   {
-      h_min = std::min(h_min, m->GetElementSize(i));
+      min_h = std::min(min_h, m->GetElementSize(i));
   }
   double v_max = 0.0;
   const int s = u.FESpace()->GetVSize() / 2;
@@ -73,28 +73,26 @@ void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
      const double vel = u(i) * u(i) + u(i+s) * u(i+s);
      v_max = std::max(v_max, vel);
   }
-#ifdef MFEM_USE_MPI
-   if (pfes)
-   {
-      double v_loc = v_max, h_loc = h_min;
-      MPI_Allreduce(&v_loc, &v_max, 1, MPI_DOUBLE, MPI_MAX, pfes->GetComm());
-      MPI_Allreduce(&h_loc, &h_min, 1, MPI_DOUBLE, MPI_MIN, pfes->GetComm());
-   }
-#endif
   if (v_max == 0.0)
   {
-      // No mesh motion --> no need to change the field.
-      delete oper;
+      // No need to change the field.
      return;
   }
   v_max = std::sqrt(v_max);
-   double dt = dt_scale * h_min / v_max;
+   double dt = 0.5 * min_h / v_max;
+   double glob_dt = dt;
+#ifdef MFEM_USE_MPI
+   if (pfes)
+   {
+      MPI_Allreduce(&dt, &glob_dt, 1, MPI_DOUBLE, MPI_MIN, pfes->GetComm());
+   }
+#endif

   double t = 0.0;
   bool last_step = false;
   for (int ti = 1; !last_step; ti++)
   {
-      if (t + dt >= 1.0)
+      if (t + glob_dt >= 1.0)
      {
 #ifdef MFEM_DEBUG
         if (myid == 0)
@@ -102,10 +100,10 @@ void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
            mfem::out << "Remap took " << ti << " steps." << std::endl;
         }
 #endif
-         dt = 1.0 - t;
+         glob_dt = 1.0 - t;
         last_step = true;
      }
-      ode_solver.Step(new_field, t, dt);
+      ode_solver.Step(new_field, t, glob_dt);
   }

   // Trim the overshoots and undershoots.
@@ -412,58 +410,28 @@ double TMOPNewtonSolver::ComputeScalingFactor(const Vector &x,

 void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
 {
-   const NonlinearForm *nlf = dynamic_cast<const NonlinearForm *>(oper);
-   const Array<NonlinearFormIntegrator*> &integs = *nlf->GetDNFI();
-
-   // Reset the update flags of all TargetConstructors.
-   // This is done to avoid repeated updates of shared TargetConstructors.
-   TMOP_Integrator *ti  = NULL;
-   TMOPComboIntegrator *co = NULL;
-   DiscreteAdaptTC *dtc = NULL;
-   for (int i = 0; i < integs.Size(); i++)
-   {
-      ti = dynamic_cast<TMOP_Integrator *>(integs[i]);
-      if (ti)
-      {
-         dtc = ti->GetDiscreteAdaptTC();
-         if (dtc) { dtc->ResetUpdateFlags(); }
-      }
-      co = dynamic_cast<TMOPComboIntegrator *>(integs[i]);
-      if (co)
-      {
-         Array<TMOP_Integrator *> ati = co->GetTMOPIntegrators();
-         for (int j = 0; j < ati.Size(); j++)
-         {
-            dtc = ati[j]->GetDiscreteAdaptTC();
-            if (dtc) { dtc->ResetUpdateFlags(); }
-         }
-      }
-   }
-
   if (parallel)
   {
 #ifdef MFEM_USE_MPI
      const ParNonlinearForm *nlf =
         dynamic_cast<const ParNonlinearForm *>(oper);
+      const Array<NonlinearFormIntegrator*> &integs = *nlf->GetDNFI();
      const ParFiniteElementSpace *pfesc = nlf->ParFESpace();
      Vector x_loc(pfesc->GetVSize());
      pfesc->GetProlongationMatrix()->Mult(x, x_loc);
-      for (int i = 0; i < integs.Size(); i++)
+      for (int i=0; i<integs.Size(); i++)
      {
-         ti = dynamic_cast<TMOP_Integrator *>(integs[i]);
-         if (ti)
+         TMOP_Integrator *tmopi = dynamic_cast<TMOP_Integrator *>(integs[i]);
+         DiscreteAdaptTC *discrtc = tmopi->GetDiscreteAdaptTC();
+         tmopi->ComputeFDh(x_loc, *pfesc);
+         if (discrtc)
         {
-            ti->ComputeFDh(x_loc, *pfesc);
-            UpdateDiscreteTC(*ti, x_loc);
-         }
-         co = dynamic_cast<TMOPComboIntegrator *>(integs[i]);
-         if (co)
-         {
-            Array<TMOP_Integrator *> ati = co->GetTMOPIntegrators();
-            for (int j = 0; j < ati.Size(); j++)
+            discrtc->UpdateTargetSpecification(x_loc);
+            double dx = tmopi->GetFDh();
+            if (tmopi->GetFDFlag())
            {
-               ati[j]->ComputeFDh(x_loc, *pfesc);
-               UpdateDiscreteTC(*ati[j], x_loc);
+               discrtc->UpdateGradientTargetSpecification(x_loc, dx);
+               discrtc->UpdateHessianTargetSpecification(x_loc, dx);
            }
         }
      }
@@ -471,6 +439,9 @@ void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
   }
   else
   {
+      const NonlinearForm *nlf =
+         dynamic_cast<const NonlinearForm *>(oper);
+      const Array<NonlinearFormIntegrator*> &integs = *nlf->GetDNFI();
      const FiniteElementSpace *fesc = nlf->FESpace();
      const Operator *P = nlf->GetProlongation();
      Vector x_loc;
@@ -483,45 +454,25 @@ void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
      {
         x_loc = x;
      }
-      for (int i = 0; i < integs.Size(); i++)
+      for (int i=0; i<integs.Size(); i++)
      {
-         ti = dynamic_cast<TMOP_Integrator *>(integs[i]);
-         if (ti)
+         TMOP_Integrator *tmopi = dynamic_cast<TMOP_Integrator *>(integs[i]);
+         DiscreteAdaptTC *discrtc = tmopi->GetDiscreteAdaptTC();
+         tmopi->ComputeFDh(x_loc, *fesc);
+         if (discrtc)
         {
-            ti->ComputeFDh(x_loc, *fesc);
-            UpdateDiscreteTC(*ti, x_loc);
-         }
-         co = dynamic_cast<TMOPComboIntegrator *>(integs[i]);
-         if (co)
-         {
-            Array<TMOP_Integrator *> ati = co->GetTMOPIntegrators();
-            for (int j = 0; j < ati.Size(); j++)
+            discrtc->UpdateTargetSpecification(x);
+            double dx = tmopi->GetFDh();
+            if (tmopi->GetFDFlag())
            {
-               ati[j]->ComputeFDh(x_loc, *fesc);
-               UpdateDiscreteTC(*ati[j], x_loc);
+               discrtc->UpdateGradientTargetSpecification(x_loc, dx);
+               discrtc->UpdateHessianTargetSpecification(x_loc, dx);
            }
         }
      }
   }
 }

-void TMOPNewtonSolver::UpdateDiscreteTC(const TMOP_Integrator &ti,
-                                        const Vector &x_new) const
-{
-   const bool update_flag = true;
-   DiscreteAdaptTC *discrtc = ti.GetDiscreteAdaptTC();
-   if (discrtc)
-   {
-      discrtc->UpdateTargetSpecification(x_new, update_flag);
-      if (ti.GetFDFlag())
-      {
-         double dx = ti.GetFDh();
-         discrtc->UpdateGradientTargetSpecification(x_new, dx, update_flag);
-         discrtc->UpdateHessianTargetSpecification(x_new, dx, update_flag);
-      }
-   }
-}
-
 double TMOPDescentNewtonSolver::ComputeScalingFactor(const Vector &x,
                                                     const Vector &b) const
 {
@@ -621,6 +572,71 @@ double TMOPDescentNewtonSolver::ComputeScalingFactor(const Vector &x,
   return scale;
 }

+void TMOPDescentNewtonSolver::ProcessNewState(const Vector &x) const
+{
+   if (parallel)
+   {
+#ifdef MFEM_USE_MPI
+      const ParNonlinearForm *nlf =
+         dynamic_cast<const ParNonlinearForm *>(oper);
+      const Array<NonlinearFormIntegrator*> &integs = *nlf->GetDNFI();
+      const ParFiniteElementSpace *pfesc = nlf->ParFESpace();
+      Vector x_loc(pfesc->GetVSize());
+      pfesc->GetProlongationMatrix()->Mult(x, x_loc);
+      for (int i=0; i<integs.Size(); i++)
+      {
+         TMOP_Integrator *tmopi = dynamic_cast<TMOP_Integrator *>(integs[i]);
+         DiscreteAdaptTC *discrtc = tmopi->GetDiscreteAdaptTC();
+         tmopi->ComputeFDh(x_loc, *pfesc);
+         if (discrtc)
+         {
+            discrtc->UpdateTargetSpecification(x_loc);
+            double dx = tmopi->GetFDh();
+            if (tmopi->GetFDFlag())
+            {
+               discrtc->UpdateGradientTargetSpecification(x_loc, dx);
+               discrtc->UpdateHessianTargetSpecification(x_loc, dx);
+            }
+         }
+      }
+#endif
+   }
+   else
+   {
+      const NonlinearForm *nlf =
+         dynamic_cast<const NonlinearForm *>(oper);
+      const Array<NonlinearFormIntegrator*> &integs = *nlf->GetDNFI();
+      const FiniteElementSpace *fesc = nlf->FESpace();
+      const Operator *P = nlf->GetProlongation();
+      Vector x_loc;
+      if (P)
+      {
+         x_loc.SetSize(P->Height());
+         P->Mult(x,x_loc);
+      }
+      else
+      {
+         x_loc = x;
+      }
+      for (int i=0; i<integs.Size(); i++)
+      {
+         TMOP_Integrator *tmopi = dynamic_cast<TMOP_Integrator *>(integs[i]);
+         DiscreteAdaptTC *discrtc = tmopi->GetDiscreteAdaptTC();
+         tmopi->ComputeFDh(x_loc, *fesc);
+         if (discrtc)
+         {
+            discrtc->UpdateTargetSpecification(x);
+            double dx = tmopi->GetFDh();
+            if (tmopi->GetFDFlag())
+            {
+               discrtc->UpdateGradientTargetSpecification(x_loc, dx);
+               discrtc->UpdateHessianTargetSpecification(x_loc, dx);
+            }
+         }
+      }
+   }
+}
+
 #ifdef MFEM_USE_MPI
 // Metric values are visualized by creating an L2 finite element functions and
 // computing the metric values at the nodes.
@@ -28,12 +28,8 @@ private:
   Vector nodes0;
   Vector field0;

-   const double dt_scale;
-
 public:
-   AdvectorCG(double timestep_scale = 0.5)
-      : AdaptivityEvaluator(),
-        ode_solver(), nodes0(), field0(), dt_scale(timestep_scale) { }
+   AdvectorCG() : AdaptivityEvaluator(), ode_solver(), nodes0(), field0() { }

   virtual void SetInitialField(const Vector &init_nodes,
                                const Vector &init_field);
@@ -109,14 +105,12 @@ public:

 class TMOPNewtonSolver : public NewtonSolver
 {
-protected:
+private:
   bool parallel;

   // Quadrature points that are checked for negative Jacobians etc.
   const IntegrationRule &ir;

-   void UpdateDiscreteTC(const TMOP_Integrator &ti, const Vector &x_new) const;
-
 public:
 #ifdef MFEM_USE_MPI
   TMOPNewtonSolver(MPI_Comm comm, const IntegrationRule &irule)
@@ -131,17 +125,25 @@ public:
 };

 /// Allows negative Jacobians. Used for untangling.
-class TMOPDescentNewtonSolver : public TMOPNewtonSolver
+class TMOPDescentNewtonSolver : public NewtonSolver
 {
+private:
+   bool parallel;
+
+   // Quadrature points that are checked for negative Jacobians etc.
+   const IntegrationRule &ir;
+
 public:
 #ifdef MFEM_USE_MPI
   TMOPDescentNewtonSolver(MPI_Comm comm, const IntegrationRule &irule)
-      : TMOPNewtonSolver(comm, irule) { }
+      : NewtonSolver(comm), parallel(true), ir(irule) { }
 #endif
   TMOPDescentNewtonSolver(const IntegrationRule &irule)
-      : TMOPNewtonSolver(irule) { }
+      : NewtonSolver(), parallel(false), ir(irule) { }

   virtual double ComputeScalingFactor(const Vector &x, const Vector &b) const;
+
+   virtual void ProcessNewState(const Vector &x) const;
 };

 void vis_tmop_metric_s(int order, TMOP_QualityMetric &qm,
@@ -1,553 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-
-#include "transfer.hpp"
-#include "../general/forall.hpp"
-
-namespace mfem
-{
-
-TransferOperator::TransferOperator(const FiniteElementSpace& lFESpace_,
-                                   const FiniteElementSpace& hFESpace_)
-   : Operator(hFESpace_.GetVSize(), lFESpace_.GetVSize())
-{
-   if (lFESpace_.FEColl() == hFESpace_.FEColl())
-   {
-      OperatorPtr P(Operator::ANY_TYPE);
-      hFESpace_.GetTransferOperator(lFESpace_, P);
-      P.SetOperatorOwner(false);
-      opr = P.Ptr();
-   }
-   else if (lFESpace_.GetMesh()->GetNE() > 0
-            && hFESpace_.GetMesh()->GetNE() > 0
-            && dynamic_cast<const TensorBasisElement*>(lFESpace_.GetFE(0))
-            && dynamic_cast<const TensorBasisElement*>(hFESpace_.GetFE(0)))
-   {
-      opr = new TensorProductPRefinementTransferOperator(lFESpace_, hFESpace_);
-   }
-   else
-   {
-      opr = new PRefinementTransferOperator(lFESpace_, hFESpace_);
-   }
-}
-
-TransferOperator::~TransferOperator() { delete opr; }
-
-void TransferOperator::Mult(const Vector& x, Vector& y) const
-{
-   opr->Mult(x, y);
-}
-
-void TransferOperator::MultTranspose(const Vector& x, Vector& y) const
-{
-   opr->MultTranspose(x, y);
-}
-
-PRefinementTransferOperator::PRefinementTransferOperator(
-   const FiniteElementSpace& lFESpace_, const FiniteElementSpace& hFESpace_)
-   : Operator(hFESpace_.GetVSize(), lFESpace_.GetVSize()), lFESpace(lFESpace_),
-     hFESpace(hFESpace_)
-{
-}
-
-PRefinementTransferOperator::~PRefinementTransferOperator() {}
-
-void PRefinementTransferOperator::Mult(const Vector& x, Vector& y) const
-{
-   Mesh* mesh = hFESpace.GetMesh();
-   Array<int> l_dofs, h_dofs, l_vdofs, h_vdofs;
-   DenseMatrix loc_prol;
-   Vector subY, subX;
-
-   Geometry::Type cached_geom = Geometry::INVALID;
-   const FiniteElement* h_fe = NULL;
-   const FiniteElement* l_fe = NULL;
-   IsoparametricTransformation T;
-
-   int vdim = lFESpace.GetVDim();
-
-   for (int i = 0; i < mesh->GetNE(); i++)
-   {
-      hFESpace.GetElementDofs(i, h_dofs);
-      lFESpace.GetElementDofs(i, l_dofs);
-
-      const Geometry::Type geom = mesh->GetElementBaseGeometry(i);
-      if (geom != cached_geom)
-      {
-         h_fe = hFESpace.GetFE(i);
-         l_fe = lFESpace.GetFE(i);
-         T.SetIdentityTransformation(h_fe->GetGeomType());
-         h_fe->GetTransferMatrix(*l_fe, T, loc_prol);
-         subY.SetSize(loc_prol.Height());
-         cached_geom = geom;
-      }
-
-      for (int vd = 0; vd < vdim; vd++)
-      {
-         l_dofs.Copy(l_vdofs);
-         lFESpace.DofsToVDofs(vd, l_vdofs);
-         h_dofs.Copy(h_vdofs);
-         hFESpace.DofsToVDofs(vd, h_vdofs);
-         x.GetSubVector(l_vdofs, subX);
-         loc_prol.Mult(subX, subY);
-         y.SetSubVector(h_vdofs, subY);
-      }
-   }
-}
-
-void PRefinementTransferOperator::MultTranspose(const Vector& x,
-                                                Vector& y) const
-{
-   y = 0.0;
-
-   Mesh* mesh = hFESpace.GetMesh();
-   Array<int> l_dofs, h_dofs, l_vdofs, h_vdofs;
-   DenseMatrix loc_prol;
-   Vector subY, subX;
-
-   Array<char> processed(hFESpace.GetVSize());
-   processed = 0;
-
-   Geometry::Type cached_geom = Geometry::INVALID;
-   const FiniteElement* h_fe = NULL;
-   const FiniteElement* l_fe = NULL;
-   IsoparametricTransformation T;
-
-   int vdim = lFESpace.GetVDim();
-
-   for (int i = 0; i < mesh->GetNE(); i++)
-   {
-      hFESpace.GetElementDofs(i, h_dofs);
-      lFESpace.GetElementDofs(i, l_dofs);
-
-      const Geometry::Type geom = mesh->GetElementBaseGeometry(i);
-      if (geom != cached_geom)
-      {
-         h_fe = hFESpace.GetFE(i);
-         l_fe = lFESpace.GetFE(i);
-         T.SetIdentityTransformation(h_fe->GetGeomType());
-         h_fe->GetTransferMatrix(*l_fe, T, loc_prol);
-         loc_prol.Transpose();
-         subY.SetSize(loc_prol.Height());
-         cached_geom = geom;
-      }
-
-      for (int vd = 0; vd < vdim; vd++)
-      {
-         l_dofs.Copy(l_vdofs);
-         lFESpace.DofsToVDofs(vd, l_vdofs);
-         h_dofs.Copy(h_vdofs);
-         hFESpace.DofsToVDofs(vd, h_vdofs);
-
-         x.GetSubVector(h_vdofs, subX);
-         for (int p = 0; p < h_dofs.Size(); ++p)
-         {
-            if (processed[lFESpace.DecodeDof(h_dofs[p])])
-            {
-               subX[p] = 0.0;
-            }
-         }
-
-         loc_prol.Mult(subX, subY);
-         y.AddElementVector(l_vdofs, subY);
-      }
-
-      for (int p = 0; p < h_dofs.Size(); ++p)
-      {
-         processed[lFESpace.DecodeDof(h_dofs[p])] = 1;
-      }
-   }
-}
-
-TensorProductPRefinementTransferOperator::
-TensorProductPRefinementTransferOperator(
-   const FiniteElementSpace& lFESpace_,
-   const FiniteElementSpace& hFESpace_)
-   : Operator(hFESpace_.GetVSize(), lFESpace_.GetVSize()), lFESpace(lFESpace_),
-     hFESpace(hFESpace_)
-{
-   // Assuming the same element type
-   Mesh* mesh = lFESpace.GetMesh();
-   dim = mesh->Dimension();
-   if (mesh->GetNE() == 0)
-   {
-      return;
-   }
-   const FiniteElement& el = *lFESpace.GetFE(0);
-
-   const TensorBasisElement* ltel =
-      dynamic_cast<const TensorBasisElement*>(&el);
-   MFEM_VERIFY(ltel, "Low order FE space must be tensor product space");
-
-   const TensorBasisElement* htel =
-      dynamic_cast<const TensorBasisElement*>(hFESpace.GetFE(0));
-   MFEM_VERIFY(htel, "High order FE space must be tensor product space");
-   const Array<int>& hdofmap = htel->GetDofMap();
-
-   const IntegrationRule& ir = hFESpace.GetFE(0)->GetNodes();
-   IntegrationRule irLex = ir;
-
-   // The quadrature points, or equivalently, the dofs of the high order space
-   // must be sorted in lexicographical order
-   for (int i = 0; i < ir.GetNPoints(); ++i)
-   {
-      irLex.IntPoint(i) = ir.IntPoint(hdofmap[i]);
-   }
-
-   NE = lFESpace.GetNE();
-   const DofToQuad& maps = el.GetDofToQuad(irLex, DofToQuad::TENSOR);
-
-   D1D = maps.ndof;
-   Q1D = maps.nqpt;
-   B = maps.B;
-   Bt = maps.Bt;
-
-   elem_restrict_lex_l =
-      lFESpace.GetElementRestriction(ElementDofOrdering::LEXICOGRAPHIC);
-
-   MFEM_VERIFY(elem_restrict_lex_l,
-               "Low order ElementRestriction not available");
-
-   elem_restrict_lex_h =
-      hFESpace.GetElementRestriction(ElementDofOrdering::LEXICOGRAPHIC);
-
-   MFEM_VERIFY(elem_restrict_lex_h,
-               "High order ElementRestriction not available");
-
-   localL.SetSize(elem_restrict_lex_l->Height(), Device::GetMemoryType());
-   localH.SetSize(elem_restrict_lex_h->Height(), Device::GetMemoryType());
-   localL.UseDevice(true);
-   localH.UseDevice(true);
-
-   MFEM_VERIFY(dynamic_cast<const ElementRestriction*>(elem_restrict_lex_h),
-               "High order element restriction is of unsupported type");
-
-   mask.SetSize(localH.Size(), Device::GetMemoryType());
-   static_cast<const ElementRestriction*>(elem_restrict_lex_h)
-   ->BooleanMask(mask);
-   mask.UseDevice(true);
-}
-
-namespace TransferKernels
-{
-void Prolongation2D(const int NE, const int D1D, const int Q1D,
-                    const Vector& localL, Vector& localH,
-                    const Array<double>& B, const Vector& mask)
-{
-   auto x_ = Reshape(localL.Read(), D1D, D1D, NE);
-   auto y_ = Reshape(localH.ReadWrite(), Q1D, Q1D, NE);
-   auto B_ = Reshape(B.Read(), Q1D, D1D);
-   auto m_ = Reshape(mask.Read(), Q1D, Q1D, NE);
-
-   localH = 0.0;
-
-   MFEM_FORALL(e, NE,
-   {
-      for (int dy = 0; dy < D1D; ++dy)
-      {
-         double sol_x[MAX_Q1D];
-         for (int qy = 0; qy < Q1D; ++qy)
-         {
-            sol_x[qy] = 0.0;
-         }
-         for (int dx = 0; dx < D1D; ++dx)
-         {
-            const double s = x_(dx, dy, e);
-            for (int qx = 0; qx < Q1D; ++qx)
-            {
-               sol_x[qx] += B_(qx, dx) * s;
-            }
-         }
-         for (int qy = 0; qy < Q1D; ++qy)
-         {
-            const double d2q = B_(qy, dy);
-            for (int qx = 0; qx < Q1D; ++qx)
-            {
-               y_(qx, qy, e) += d2q * sol_x[qx];
-            }
-         }
-      }
-      for (int qy = 0; qy < Q1D; ++qy)
-      {
-         for (int qx = 0; qx < Q1D; ++qx)
-         {
-            y_(qx, qy, e) *= m_(qx, qy, e);
-         }
-      }
-   });
-}
-
-void Prolongation3D(const int NE, const int D1D, const int Q1D,
-                    const Vector& localL, Vector& localH,
-                    const Array<double>& B, const Vector& mask)
-{
-   auto x_ = Reshape(localL.Read(), D1D, D1D, D1D, NE);
-   auto y_ = Reshape(localH.ReadWrite(), Q1D, Q1D, Q1D, NE);
-   auto B_ = Reshape(B.Read(), Q1D, D1D);
-   auto m_ = Reshape(mask.Read(), Q1D, Q1D, Q1D, NE);
-
-   localH = 0.0;
-
-   MFEM_FORALL(e, NE,
-   {
-      for (int dz = 0; dz < D1D; ++dz)
-      {
-         double sol_xy[MAX_Q1D][MAX_Q1D];
-         for (int qy = 0; qy < Q1D; ++qy)
-         {
-            for (int qx = 0; qx < Q1D; ++qx)
-            {
-               sol_xy[qy][qx] = 0.0;
-            }
-         }
-         for (int dy = 0; dy < D1D; ++dy)
-         {
-            double sol_x[MAX_Q1D];
-            for (int qx = 0; qx < Q1D; ++qx)
-            {
-               sol_x[qx] = 0;
-            }
-            for (int dx = 0; dx < D1D; ++dx)
-            {
-               const double s = x_(dx, dy, dz, e);
-               for (int qx = 0; qx < Q1D; ++qx)
-               {
-                  sol_x[qx] += B_(qx, dx) * s;
-               }
-            }
-            for (int qy = 0; qy < Q1D; ++qy)
-            {
-               const double wy = B_(qy, dy);
-               for (int qx = 0; qx < Q1D; ++qx)
-               {
-                  sol_xy[qy][qx] += wy * sol_x[qx];
-               }
-            }
-         }
-         for (int qz = 0; qz < Q1D; ++qz)
-         {
-            const double wz = B_(qz, dz);
-            for (int qy = 0; qy < Q1D; ++qy)
-            {
-               for (int qx = 0; qx < Q1D; ++qx)
-               {
-                  y_(qx, qy, qz, e) += wz * sol_xy[qy][qx];
-               }
-            }
-         }
-      }
-      for (int qz = 0; qz < Q1D; ++qz)
-      {
-         for (int qy = 0; qy < Q1D; ++qy)
-         {
-            for (int qx = 0; qx < Q1D; ++qx)
-            {
-               y_(qx, qy, qz, e) *= m_(qx, qy, qz, e);
-            }
-         }
-      }
-   });
-}
-
-void Restriction2D(const int NE, const int D1D, const int Q1D,
-                   const Vector& localH, Vector& localL,
-                   const Array<double>& Bt, const Vector& mask)
-{
-   auto x_ = Reshape(localH.Read(), Q1D, Q1D, NE);
-   auto y_ = Reshape(localL.ReadWrite(), D1D, D1D, NE);
-   auto Bt_ = Reshape(Bt.Read(), D1D, Q1D);
-   auto m_ = Reshape(mask.Read(), Q1D, Q1D, NE);
-
-   localL = 0.0;
-
-   MFEM_FORALL(e, NE,
-   {
-      for (int qy = 0; qy < Q1D; ++qy)
-      {
-         double sol_x[MAX_D1D];
-         for (int dx = 0; dx < D1D; ++dx)
-         {
-            sol_x[dx] = 0.0;
-         }
-         for (int qx = 0; qx < Q1D; ++qx)
-         {
-            const double s = m_(qx, qy, e) * x_(qx, qy, e);
-            for (int dx = 0; dx < D1D; ++dx)
-            {
-               sol_x[dx] += Bt_(dx, qx) * s;
-            }
-         }
-         for (int dy = 0; dy < D1D; ++dy)
-         {
-            const double q2d = Bt_(dy, qy);
-            for (int dx = 0; dx < D1D; ++dx)
-            {
-               y_(dx, dy, e) += q2d * sol_x[dx];
-            }
-         }
-      }
-   });
-}
-void Restriction3D(const int NE, const int D1D, const int Q1D,
-                   const Vector& localH, Vector& localL,
-                   const Array<double>& Bt, const Vector& mask)
-{
-   auto x_ = Reshape(localH.Read(), Q1D, Q1D, Q1D, NE);
-   auto y_ = Reshape(localL.ReadWrite(), D1D, D1D, D1D, NE);
-   auto Bt_ = Reshape(Bt.Read(), D1D, Q1D);
-   auto m_ = Reshape(mask.Read(), Q1D, Q1D, Q1D, NE);
-
-   localL = 0.0;
-
-   MFEM_FORALL(e, NE,
-   {
-      for (int qz = 0; qz < Q1D; ++qz)
-      {
-         double sol_xy[MAX_D1D][MAX_D1D];
-         for (int dy = 0; dy < D1D; ++dy)
-         {
-            for (int dx = 0; dx < D1D; ++dx)
-            {
-               sol_xy[dy][dx] = 0;
-            }
-         }
-         for (int qy = 0; qy < Q1D; ++qy)
-         {
-            double sol_x[MAX_D1D];
-            for (int dx = 0; dx < D1D; ++dx)
-            {
-               sol_x[dx] = 0;
-            }
-            for (int qx = 0; qx < Q1D; ++qx)
-            {
-               const double s = m_(qx, qy, qz, e) * x_(qx, qy, qz, e);
-               for (int dx = 0; dx < D1D; ++dx)
-               {
-                  sol_x[dx] += Bt_(dx, qx) * s;
-               }
-            }
-            for (int dy = 0; dy < D1D; ++dy)
-            {
-               const double wy = Bt_(dy, qy);
-               for (int dx = 0; dx < D1D; ++dx)
-               {
-                  sol_xy[dy][dx] += wy * sol_x[dx];
-               }
-            }
-         }
-         for (int dz = 0; dz < D1D; ++dz)
-         {
-            const double wz = Bt_(dz, qz);
-            for (int dy = 0; dy < D1D; ++dy)
-            {
-               for (int dx = 0; dx < D1D; ++dx)
-               {
-                  y_(dx, dy, dz, e) += wz * sol_xy[dy][dx];
-               }
-            }
-         }
-      }
-   });
-}
-} // namespace TransferKernels
-
-TensorProductPRefinementTransferOperator::
-~TensorProductPRefinementTransferOperator()
-{
-}
-
-void TensorProductPRefinementTransferOperator::Mult(const Vector& x,
-                                                    Vector& y) const
-{
-   if (lFESpace.GetMesh()->GetNE() == 0)
-   {
-      return;
-   }
-
-   elem_restrict_lex_l->Mult(x, localL);
-   if (dim == 2)
-   {
-      TransferKernels::Prolongation2D(NE, D1D, Q1D, localL, localH, B, mask);
-   }
-   else if (dim == 3)
-   {
-      TransferKernels::Prolongation3D(NE, D1D, Q1D, localL, localH, B, mask);
-   }
-   else
-   {
-      MFEM_ABORT("TensorProductPRefinementTransferOperator::Mult not "
-                 "implemented for dim = "
-                 << dim);
-   }
-   elem_restrict_lex_h->MultTranspose(localH, y);
-}
-
-void TensorProductPRefinementTransferOperator::MultTranspose(const Vector& x,
-                                                             Vector& y) const
-{
-   if (lFESpace.GetMesh()->GetNE() == 0)
-   {
-      return;
-   }
-
-   elem_restrict_lex_h->Mult(x, localH);
-   if (dim == 2)
-   {
-      TransferKernels::Restriction2D(NE, D1D, Q1D, localH, localL, Bt, mask);
-   }
-   else if (dim == 3)
-   {
-      TransferKernels::Restriction3D(NE, D1D, Q1D, localH, localL, Bt, mask);
-   }
-   else
-   {
-      MFEM_ABORT("TensorProductPRefinementTransferOperator::MultTranspose not "
-                 "implemented for dim = "
-                 << dim);
-   }
-   elem_restrict_lex_l->MultTranspose(localL, y);
-}
-
-#ifdef MFEM_USE_MPI
-TrueTransferOperator::TrueTransferOperator(const
-                                           ParFiniteElementSpace& lFESpace_,
-                                           const ParFiniteElementSpace& hFESpace_)
-   : lFESpace(lFESpace_), hFESpace(hFESpace_)
-{
-   localTransferOperator = new TransferOperator(lFESpace_, hFESpace_);
-
-   tmpL.SetSize(lFESpace_.GetVSize());
-   tmpH.SetSize(hFESpace_.GetVSize());
-
-   hFESpace.GetRestrictionMatrix()->BuildTranspose();
-}
-
-TrueTransferOperator::~TrueTransferOperator()
-{
-   delete localTransferOperator;
-}
-
-void TrueTransferOperator::Mult(const Vector& x, Vector& y) const
-{
-   lFESpace.GetProlongationMatrix()->Mult(x, tmpL);
-   localTransferOperator->Mult(tmpL, tmpH);
-   hFESpace.GetRestrictionMatrix()->Mult(tmpH, y);
-}
-
-void TrueTransferOperator::MultTranspose(const Vector& x, Vector& y) const
-{
-   hFESpace.GetRestrictionMatrix()->MultTranspose(x, tmpH);
-   localTransferOperator->MultTranspose(tmpH, tmpL);
-   lFESpace.GetProlongationMatrix()->MultTranspose(tmpL, y);
-}
-#endif
-
-} // namespace mfem
@@ -1,162 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-
-#ifndef MFEM_TRANSFER_HPP
-#define MFEM_TRANSFER_HPP
-
-#include "../linalg/linalg.hpp"
-#include "fespace.hpp"
-
-#ifdef MFEM_USE_MPI
-#include "pfespace.hpp"
-#endif
-
-namespace mfem
-{
-
-/// Matrix-free transfer operator between finite element spaces
-class TransferOperator : public Operator
-{
-private:
-   Operator* opr;
-
-public:
-   /// Constructs a transfer operator from \p lFESpace to \p hFESpace.
-   /** No matrices are assembled, only the action to a vector is being computed.
-       If both spaces' FE collection pointers are pointing to the same collection
-       we assume that the grid was refined while keeping the order constant. If
-       the FE collections are different, it is assumed that both spaces have are
-       using the same mesh. If the first element of the high-order space is a
-       `TensorBasisElement`, the optimized tensor-product transfers are used. If
-       not, the general transfers used. */
-   TransferOperator(const FiniteElementSpace& lFESpace,
-                    const FiniteElementSpace& hFESpace);
-
-   /// Destructor
-   virtual ~TransferOperator();
-
-   /// @brief Interpolation or prolongation of a vector \p x corresponding to the
-   /// coarse space to the vector \p y corresponding to the fine space.
-   virtual void Mult(const Vector& x, Vector& y) const override;
-
-   /// Restriction by applying the transpose of the Mult method.
-   /** The vector \p x corresponding to the fine space is restricted to the vector
-       \p y corresponding to the coarse space. */
-   virtual void MultTranspose(const Vector& x, Vector& y) const override;
-};
-
-/// Matrix-free transfer operator between finite element spaces on the same mesh
-class PRefinementTransferOperator : public Operator
-{
-private:
-   const FiniteElementSpace& lFESpace;
-   const FiniteElementSpace& hFESpace;
-
-public:
-   /// @brief Constructs a transfer operator from \p lFESpace to \p hFESpace
-   /// which have different FE collections.
-   /** No matrices are assembled, only the action to a vector is being computed.
-       The underlying finite elements need to implement the GetTransferMatrix
-       methods. */
-   PRefinementTransferOperator(const FiniteElementSpace& lFESpace_,
-                               const FiniteElementSpace& hFESpace_);
-
-   /// Destructor
-   virtual ~PRefinementTransferOperator();
-
-   /// @brief Interpolation or prolongation of a vector \p x corresponding to the
-   /// coarse space to the vector \p y corresponding to the fine space.
-   virtual void Mult(const Vector& x, Vector& y) const override;
-
-   /// Restriction by applying the transpose of the Mult method.
-   /** The vector \p x corresponding to the fine space is restricted to the vector
-   \p y corresponding to the coarse space. */
-   virtual void MultTranspose(const Vector& x, Vector& y) const override;
-};
-
-/// @brief Matrix-free transfer operator between finite element spaces on the same
-/// mesh exploiting the tensor product structure of the finite elements
-class TensorProductPRefinementTransferOperator : public Operator
-{
-private:
-   const FiniteElementSpace& lFESpace;
-   const FiniteElementSpace& hFESpace;
-   int dim;
-   int NE;
-   int D1D;
-   int Q1D;
-   Array<double> B;
-   Array<double> Bt;
-   const Operator* elem_restrict_lex_l;
-   const Operator* elem_restrict_lex_h;
-   Vector mask;
-   mutable Vector localL;
-   mutable Vector localH;
-
-public:
-   /// @brief Constructs a transfer operator from \p lFESpace to \p hFESpace which
-   /// have different FE collections.
-   /** No matrices are assembled, only the action to a vector is being computed.
-   The underlying finite elements need to be of the type `TensorBasisElement`. It
-   is also assumed that all the elements in the spaces are of the same type. */
-   TensorProductPRefinementTransferOperator(
-      const FiniteElementSpace& lFESpace_,
-      const FiniteElementSpace& hFESpace_);
-
-   /// Destructor
-   virtual ~TensorProductPRefinementTransferOperator();
-
-   /// @brief Interpolation or prolongation of a vector \p x corresponding to the
-   /// coarse space to the vector \p y corresponding to the fine space.
-   virtual void Mult(const Vector& x, Vector& y) const override;
-
-   /// Restriction by applying the transpose of the Mult method.
-   /** The vector \p x corresponding to the fine space is restricted to the vector
-   \p y corresponding to the coarse space. */
-   virtual void MultTranspose(const Vector& x, Vector& y) const override;
-};
-
-#ifdef MFEM_USE_MPI
-/// @brief Matrix-free transfer operator between finite element spaces working on
-/// true degrees of freedom
-class TrueTransferOperator : public Operator
-{
-private:
-   const ParFiniteElementSpace& lFESpace;
-   const ParFiniteElementSpace& hFESpace;
-   TransferOperator* localTransferOperator;
-   mutable Vector tmpL;
-   mutable Vector tmpH;
-
-public:
-   /// @brief Constructs a transfer operator working on true degrees of freedom from
-   /// from \p lFESpace to \p hFESpace
-   TrueTransferOperator(const ParFiniteElementSpace& lFESpace_,
-                        const ParFiniteElementSpace& hFESpace_);
-
-   /// Destructor
-   ~TrueTransferOperator();
-
-   /// @brief Interpolation or prolongation of a true dof vector \p x to a true dof
-   /// vector \p y.
-   /** The true dof vector \p x corresponding to the coarse space is restricted to
-       the true dof vector \p y corresponding to the fine space. */
-   virtual void Mult(const Vector& x, Vector& y) const override;
-
-   /// Restriction by applying the transpose of the Mult method.
-   /** The true dof vector \p x corresponding to the fine space is restricted to
-       the true dof vector \p y corresponding to the coarse space. */
-   virtual void MultTranspose(const Vector& x, Vector& y) const override;
-};
-#endif
-
-} // namespace mfem
-#endif
@@ -65,11 +65,6 @@ if (MFEM_USE_MPI)
  list(APPEND HDRS communication.hpp)
 endif()

-if (MFEM_USE_ADIOS2)
-  list(APPEND SRCS adios2stream.cpp)
-  list(APPEND HDRS adios2stream.hpp)
-endif()
-
 convert_filenames_to_full_paths(SRCS)
 convert_filenames_to_full_paths(HDRS)

@@ -1,752 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-//
-// Created on: Jan 22, 2019
-// Author: William F Godoy godoywf@ornl.gov
-// adios2: Adaptable Input/Output System https://github.com/ornladios/ADIOS2
-
-#include "adios2stream.hpp"
-
-#include "../fem/geom.hpp"
-#include "../general/array.hpp"
-#include "../mesh/element.hpp"
-#include "../mesh/mesh.hpp"
-#include "../fem/gridfunc.hpp"
-
-#include <algorithm>
-
-namespace mfem
-{
-
-namespace
-{
-// these functions might be included in adios2 upstream next release
-template <class T>
-adios2::Variable<T> SafeDefineVariable(adios2::IO io,
-                                       const std::string& variable_name,
-                                       const adios2::Dims& shape = adios2::Dims(),
-                                       const adios2::Dims& start = adios2::Dims(),
-                                       const adios2::Dims& count = adios2::Dims())
-{
-   adios2::Variable<T> variable = io.InquireVariable<T>(variable_name);
-   if (variable)
-   {
-      if (variable.Count() != count &&
-          variable.ShapeID() == adios2::ShapeID::LocalArray)
-      {
-         variable.SetSelection({start, count});
-      }
-   }
-   else
-   {
-      variable = io.DefineVariable<T>(variable_name, shape, start, count);
-   }
-
-   return variable;
-}
-
-template <class T>
-adios2::Attribute<T> SafeDefineAttribute(adios2::IO io,
-                                         const std::string& attribute_name,
-                                         const T& value,
-                                         const std::string& variable_name = "",
-                                         const std::string separator = "/")
-{
-   adios2::Attribute<T> attribute = io.InquireAttribute<T>(attribute_name);
-   if (attribute)
-   {
-      return attribute;
-   }
-   return io.DefineAttribute<T>(attribute_name, value, variable_name, separator );
-}
-
-template <class T>
-adios2::Attribute<T> SafeDefineAttribute(adios2::IO io,
-                                         const std::string& attribute_name,
-                                         const T* values, const size_t size,
-                                         const std::string& variable_name = "",
-                                         const std::string separator = "/")
-{
-   adios2::Attribute<T> attribute = io.InquireAttribute<T>(attribute_name);
-   if (attribute)
-   {
-      return attribute;
-   }
-   return io.DefineAttribute<T>(attribute_name, values, size, variable_name,
-                                separator );
-}
-
-bool SetBoolParameter(const std::string key,
-                      const std::map<std::string, std::string>& parameters,
-                      const bool default_value) noexcept
-{
-   auto it = parameters.find(key);
-   if (it != parameters.end())
-   {
-      std::string value = it->second;
-      std::transform(value.begin(), value.end(), value.begin(), ::tolower);
-      if (value == "on" || value == "true")
-      {
-         return true;
-      }
-      else if ( value == "off" || value == "false")
-      {
-         return false;
-      }
-   }
-   return default_value;
-}
-
-} //end empty namespace
-
-// PUBLIC
-#ifdef MFEM_USE_MPI
-adios2stream::adios2stream(const std::string& name, const openmode mode,
-                           MPI_Comm comm, const std::string engineType)
-   : name(name),
-     adios2_openmode(mode),
-     adios(new adios2::ADIOS(comm)),
-     io(adios->DeclareIO(name))
-{
-   io.SetEngine(engineType);
-}
-#else
-adios2stream::adios2stream(const std::string& name, const openmode mode,
-                           const std::string engineType)
-   : name(name),
-     adios2_openmode(mode),
-     adios(new adios2::ADIOS()),
-     io(adios->DeclareIO(name))
-{
-   io.SetEngine(engineType);
-}
-#endif
-
-adios2stream::~adios2stream()
-{
-   if (engine)
-   {
-      SafeDefineAttribute<std::string>(io, "vtk.xml", VTKSchema() );
-      engine.Close();
-   }
-}
-
-void adios2stream::SetParameters(
-   const std::map<std::string, std::string>& parameters)
-{
-   io.SetParameters(parameters);
-   refine = SetBoolParameter("RefinedData", parameters, true);
-}
-
-void adios2stream::SetParameter(const std::string key,
-                                const std::string value) noexcept
-{
-   io.SetParameter(key, value);
-   if (key == "RefinedData")
-   {
-      refine = SetBoolParameter("RefinedData", io.Parameters(), true);
-   }
-}
-
-void adios2stream::BeginStep()
-{
-   if (!engine)
-   {
-      engine = io.Open(name, adios2::Mode::Write);
-   }
-   engine.BeginStep();
-   active_step = true;
-}
-
-void adios2stream::EndStep()
-{
-   if (!engine || !active_step)
-   {
-      const std::string message = "MFEM adios2stream error: calling EndStep "
-                                  "on uninitialized step (need BeginStep)";
-      mfem_error(message.c_str());
-   }
-
-   SafeDefineAttribute<std::string>(io, "vtk.xml", VTKSchema() );
-   engine.EndStep();
-   active_step = false;
-}
-
-void adios2stream::SetTime(const double time)
-{
-   adios2::Variable<double> var_time = SafeDefineVariable<double>(io, "TIME");
-   engine.Put(var_time, time);
-   transient = true;
-}
-
-void adios2stream::SetCycle(const int cycle)
-{
-   adios2::Variable<int> var_cycle = SafeDefineVariable<int>(io,"CYCLE");
-   engine.Put(var_cycle, cycle);
-}
-
-void adios2stream::SetRefinementLevel(const int level) noexcept
-{
-   refinement_level = level;
-}
-
-size_t adios2stream::CurrentStep() const
-{
-   return engine.CurrentStep();
-}
-
-void adios2stream::Close()
-{
-   if (engine)
-   {
-      if (!active_step)
-      {
-         SafeDefineAttribute<std::string>(io, "vtk.xml", VTKSchema() );
-      }
-      engine.Close();
-   }
-   if (adios)
-   {
-      adios.reset();
-   }
-}
-
-
-// PROTECTED (accessible by friend class Mesh)
-void adios2stream::Print(const Mesh& mesh, const mode print_mode)
-{
-   auto lf_DefineMeshMetadata = [this](Mesh& mesh)
-   {
-      // check types are constant
-      if (!IsConstantElementType(mesh.elements))
-      {
-         throw std::invalid_argument("MFEM::adios2stream ERROR: non-constant "
-                                     " element types not yet implemented\n");
-      }
-
-      // format info
-      SafeDefineAttribute<std::string>(io, "format", "MFEM ADIOS2 BP v0.1" );
-      SafeDefineAttribute<std::string>(io, "format/version", "0.1" );
-      std::string mesh_type = "Unknown";
-      std::vector<std::string> viz_tools;
-      viz_tools.reserve(2); //for now
-      if (mesh.NURBSext)
-      {
-         mesh_type = "MFEM NURBS";
-         viz_tools.push_back("NONE");
-      }
-      else if (mesh.ncmesh)
-      {
-         mesh_type = "MFEM mesh v1.1";
-         viz_tools.push_back("NONE");
-      }
-      else
-      {
-         mesh_type = "MFEM mesh v1.0";
-         viz_tools.push_back("Paraview: ADIOS2VTXReader");
-         viz_tools.push_back("VTK: vtkADIOS2VTXReader.h");
-      }
-
-      SafeDefineAttribute<std::string>(io, "format/mfem_mesh", mesh_type );
-      SafeDefineAttribute<std::string>(io, "format/viz_tools", viz_tools.data(),
-                                       viz_tools.size() );
-
-      // elements
-      const uint32_t dimension = static_cast<int32_t>(mesh.Dimension());
-      SafeDefineAttribute<uint32_t>(io, "dimension", dimension);
-      SafeDefineVariable<uint32_t>(io,"NumOfElements", {adios2::LocalValueDim});
-      SafeDefineVariable<uint32_t>(io, "types");
-      size_t nelements = 0;
-      size_t element_nvertices = 0;
-      size_t nvertices = 0;
-
-      if (refine)
-      {
-         for (int i = 0; i < mesh.GetNE(); ++i)
-         {
-            const Geometry::Type type = mesh.GetElementBaseGeometry(i);
-            RefinedGeometry* refined_geometry = GlobGeometryRefiner.Refine(type,
-                                                                           refinement_level, 1);
-            if (refined_geometry == nullptr)
-            {
-               mfem_error("ERROR: could not refine geometry in call to Save with adios2stream \n");
-            }
-
-
-            element_nvertices = static_cast<size_t>(Geometries.GetVertices(
-                                                       type)->GetNPoints());
-            nelements += refined_geometry->RefGeoms.Size() / element_nvertices;
-            nvertices  += refined_geometry->RefPts.GetNPoints();
-         }
-
-         refined_mesh_nelements = nelements;
-         refined_mesh_nvertices = nvertices;
-      }
-      else
-      {
-         nelements = static_cast<size_t>(mesh.GetNE());
-         element_nvertices = static_cast<size_t>(mesh.elements[0]->GetNVertices());
-      }
-      SafeDefineVariable<uint64_t>(io, "connectivity", {}, {}, {nelements, element_nvertices+1});
-
-      // vertices
-      SafeDefineVariable<uint32_t>(io,"NumOfVertices", {adios2::LocalValueDim});
-      if (refine)
-      {
-         SafeDefineVariable<double>( io, "vertices", {}, {}, {nvertices, static_cast<size_t>(dimension)});
-      }
-      else
-      {
-         const GridFunction* grid_function = mesh.GetNodes();
-         if (grid_function == nullptr)
-         {
-            const size_t nVertices = static_cast<size_t>(mesh.GetNV());
-            const size_t spaceDim = static_cast<size_t>(mesh.SpaceDimension());
-            // similar to Ordering::byVDIM
-            SafeDefineVariable<double>( io, "vertices", {}, {}, {nVertices, spaceDim});
-         }
-         else
-         {
-            const size_t size = static_cast<size_t>(grid_function->Size());
-            const FiniteElementSpace* fes = grid_function->FESpace();
-            const size_t components = static_cast<size_t>(fes->GetVDim());
-            const size_t tuples = size /components;
-            SafeDefineVariable<double>(io, "vertices", {}, {}, {tuples, components} );
-
-            if (fes->GetOrdering() == Ordering::byNODES)
-            {
-               ordering_by_node = true;
-            }
-         }
-      }
-   };
-
-   auto lf_PrintRefinedMeshData = [this](Mesh& mesh)
-   {
-      // elements and vertices
-      engine.Put("NumOfElements", static_cast<uint32_t>(refined_mesh_nelements));
-      engine.Put("NumOfVertices", static_cast<uint32_t>(refined_mesh_nvertices));
-
-      const uint32_t vtkType =
-         GLVISToVTKType(static_cast<int>(mesh.elements[0]->GetGeometryType()));
-      engine.Put("types", vtkType);
-
-      adios2::Variable<double> var_vertices = io.InquireVariable<double>("vertices");
-      adios2::Variable<double>::Span span_vertices = engine.Put<double>(var_vertices);
-
-      adios2::Variable<uint64_t> var_connectivity =
-         io.InquireVariable<uint64_t>("connectivity");
-      adios2::Variable<uint64_t>::Span span_connectivity = engine.Put<uint64_t>
-                                                           (var_connectivity);
-      size_t span_vertices_offset = 0;
-      size_t span_connectivity_offset = 0;
-      // use for setting absolute node id for each element
-      size_t point_id = 0;
-      DenseMatrix pmatrix;
-
-      for (int e = 0; e < mesh.GetNE(); ++e)
-      {
-         const Geometry::Type type = mesh.GetElementBaseGeometry(e);
-         RefinedGeometry* refined_geometry = GlobGeometryRefiner.Refine(type,
-                                                                        refinement_level, 1);
-         // vertices
-         mesh.GetElementTransformation(e)->Transform(refined_geometry->RefPts, pmatrix);
-         for (int i = 0; i < pmatrix.Width(); ++i)
-         {
-            for (int j = 0; j < pmatrix.Height(); ++j)
-            {
-               span_vertices[span_vertices_offset +  i*pmatrix.Height() + j] = pmatrix(j, i);
-            }
-         }
-         span_vertices_offset += static_cast<size_t>(pmatrix.Width()*pmatrix.Height());
-
-         // connectivity
-         const int nv = Geometries.GetVertices(type)->GetNPoints();
-         const Array<int> &element_vertices = refined_geometry->RefGeoms;
-
-         for (int v = 0; v < element_vertices.Size();)
-         {
-            span_connectivity[span_connectivity_offset] = static_cast<uint64_t>(nv);
-            ++span_connectivity_offset;
-
-            for (int k =0; k < nv; k++, v++ )
-            {
-               span_connectivity[span_connectivity_offset] = static_cast<uint64_t>
-                                                             (point_id + element_vertices[v]);
-               ++span_connectivity_offset;
-            }
-         }
-
-         point_id += static_cast<size_t>(refined_geometry->RefPts.GetNPoints());
-      }
-
-      for (int e = 0; e < mesh.GetNE(); ++e)
-      {
-         const Geometry::Type type = mesh.GetElementBaseGeometry(e);
-         RefinedGeometry* refined_geometry = GlobGeometryRefiner.Refine(type,
-                                                                        refinement_level, 1);
-      }
-   };
-
-   auto lf_PrintMeshData = [&](Mesh& mesh)
-   {
-      if (refine)
-      {
-         lf_PrintRefinedMeshData(mesh);
-         return;
-      }
-
-      // elements
-      engine.Put("NumOfElements", static_cast<uint32_t>(mesh.GetNE()));
-
-      const uint32_t vtkType =
-         GLVISToVTKType(static_cast<int>(mesh.elements[0]->GetGeometryType()));
-      engine.Put("types", vtkType);
-
-      adios2::Variable<uint64_t> varConnectivity =
-         io.InquireVariable<uint64_t>("connectivity");
-      // zero-copy access to adios2 buffer to put non-contiguous to contiguous memory
-      adios2::Variable<uint64_t>::Span spanConnectivity =
-         engine.Put<uint64_t>(varConnectivity);
-
-      size_t elementPosition = 0;
-      for (int e = 0; e < mesh.GetNE(); ++e)
-      {
-         const int nVertices = mesh.elements[e]->GetNVertices();
-         spanConnectivity[elementPosition] = nVertices;
-         for (int v = 0; v < nVertices; ++v)
-         {
-            spanConnectivity[elementPosition + v + 1] =
-               mesh.elements[e]->GetVertices()[v];
-         }
-         elementPosition += nVertices + 1;
-      }
-
-      // vertices
-      engine.Put("NumOfVertices", static_cast<uint32_t>(mesh.GetNV()));
-
-      if (mesh.GetNodes() == nullptr)
-      {
-         adios2::Variable<double> varVertices = io.InquireVariable<double>("vertices");
-         // zero-copy access to adios2 buffer to put non-contiguous to contiguous memory
-         adios2::Variable<double>::Span spanVertices = engine.Put(varVertices);
-
-         for (int v = 0; v < mesh.GetNV(); ++v)
-         {
-            const int space_dim = mesh.SpaceDimension();
-            for (int coord = 0; coord < space_dim; ++coord)
-            {
-               spanVertices[v * space_dim + coord] = mesh.vertices[v](coord);
-            }
-         }
-      }
-      else
-      {
-         const GridFunction* grid_function = mesh.GetNodes();
-         if (ordering_by_node)
-         {
-            adios2::Variable<double> varVertices = io.InquireVariable<double>("vertices");
-            // zero-copy access to adios2 buffer to put non-contiguous to contiguous memory
-            adios2::Variable<double>::Span spanVertices = engine.Put(varVertices);
-
-            const size_t size = static_cast<size_t>(grid_function->Size());
-            const FiniteElementSpace* fes = grid_function->FESpace();
-            const size_t components = static_cast<size_t>(fes->GetVDim());
-            const size_t tuples = size /components;
-
-            const double* data = grid_function->GetData();
-
-            for (size_t i = 0; i < tuples; ++i)
-            {
-               for (size_t j = 0; j < components; ++j)
-               {
-                  spanVertices[i*components + j] = data[j*tuples + i];
-               }
-            }
-         }
-         else
-         {
-            grid_function->Print(*this, "vertices");
-         }
-      }
-   };
-
-   // BODY OF FUNCTION STARTS HERE
-   try
-   {
-      Mesh ref_mesh(mesh);
-      lf_DefineMeshMetadata(ref_mesh);
-
-      if (!engine) // if Engine is closed
-      {
-         engine = io.Open(name, adios2::Mode::Write);
-      }
-
-      lf_PrintMeshData(ref_mesh);
-
-      if (print_mode == mode::sync)
-      {
-         engine.PerformPuts();
-      }
-   }
-   catch (std::exception& e)
-   {
-      const std::string warning =
-         "MFEM: adios2stream exception caught, invalid bp dataset: " + name +
-         "," + e.what();
-      mfem_warning( warning.c_str());
-   }
-}
-
-void adios2stream::Save(const GridFunction& grid_function,
-                        const std::string& variable_name, const data_type type)
-{
-   auto lf_SafeDefine = [&](const std::string& variable_name,
-                            const size_t tuples, const size_t components,
-                            const Ordering::Type ordering,
-                            const std::string& fespace_name)
-   {
-      adios2::Variable<double> var = io.InquireVariable<double>(variable_name);
-      if (!var)
-      {
-         if (components == 1 && type == adios2stream::data_type::point_data)
-         {
-            io.DefineVariable<double>(variable_name, {}, {}, {tuples*components});
-         }
-         else
-         {
-            const adios2::Dims count = (ordering == Ordering::byNODES) ?
-                                       adios2::Dims{components, tuples} :
-                                       adios2::Dims{tuples, components};
-            io.DefineVariable<double>(variable_name, {}, {}, count);
-         }
-         SafeDefineAttribute<std::string>(io, "FiniteElementSpace",
-                                          fespace_name, variable_name);
-
-      }
-   };
-
-   // BODY OF FUNCTION STARTS HERE
-   const std::map<std::string, std::string> parameters = io.Parameters();
-   const bool full_data = SetBoolParameter("FullData", parameters, false);
-
-   if (!full_data && !refine)
-   {
-      return;
-   }
-
-   const FiniteElementSpace* fes = grid_function.FESpace();
-
-   if (refine)
-   {
-      const Mesh *mesh = fes->GetMesh();
-      const size_t components = static_cast<size_t>(grid_function.VectorDim());
-      // const size_t tuples = static_cast<size_t>(mesh->GetNV());
-      const size_t tuples = refined_mesh_nvertices;
-
-      lf_SafeDefine(variable_name, tuples, components,
-                    Ordering::byVDIM, std::string(fes->FEColl()->Name()));
-      if (type == adios2stream::data_type::point_data)
-      {
-         point_data_variables.insert(variable_name);
-      }
-
-      RefinedGeometry* refined_geometry;
-      DenseMatrix transform;
-
-      // zero-copy access to adios2 buffer to put non-contiguous to contiguous memory
-      adios2::Variable<double> variable = io.InquireVariable<double>(variable_name);
-      adios2::Variable<double>::Span span = engine.Put<double>(variable);
-
-      size_t offset = 0;
-      if (components == 1)
-      {
-         Vector scalar;
-
-         const int nelements = mesh->GetNE();
-         for (int e = 0; e < nelements; ++e)
-         {
-            refined_geometry = GlobGeometryRefiner.Refine(
-                                  mesh->GetElementBaseGeometry(e), refinement_level, 1);
-
-            grid_function.GetValues(e, refined_geometry->RefPts, scalar, transform);
-
-            const int size = scalar.Size();
-
-            for (int i = 0; i < size; ++i)
-            {
-               const double value = scalar(i);
-               span.at(offset+i) = value;
-            }
-            offset += static_cast<size_t>(size);
-         }
-      }
-      else
-      {
-         DenseMatrix vector;
-         for (int e = 0; e < mesh->GetNE(); ++e)
-         {
-            refined_geometry = GlobGeometryRefiner.Refine(
-                                  mesh->GetElementBaseGeometry(e), refinement_level, 1);
-            grid_function.GetVectorValues(e, refined_geometry->RefPts, vector, transform);
-
-            for (int i = 0; i < vector.Width(); ++i)
-            {
-               for (int j = 0; j < vector.Height(); ++j)
-               {
-                  span[offset +  i*vector.Height() + j] = vector(j, i);
-               }
-            }
-            offset += static_cast<size_t>(vector.Width()*vector.Height());
-         }
-      }
-   }
-
-   if (full_data)
-   {
-      const size_t size = static_cast<size_t>(grid_function.Size());
-      const size_t components = static_cast<size_t>(fes->GetVDim());
-      const size_t tuples = size /components;
-      lf_SafeDefine(variable_name +"/full", tuples, components,
-                    fes->GetOrdering(),
-                    std::string(fes->FEColl()->Name()) );
-      // calls Vector::Print
-      grid_function.Print(*this, variable_name+"/full");
-
-      if (!refine && type == adios2stream::data_type::point_data)
-      {
-         point_data_variables.insert(variable_name+"/full");
-      }
-   }
-}
-
-// PRIVATE
-int32_t adios2stream::GLVISToVTKType(
-   const int glvisType) const noexcept
-{
-   uint32_t vtkType = 0;
-   switch (glvisType)
-   {
-      case Geometry::Type::POINT:
-         vtkType = 1;
-         break;
-      case Geometry::Type::SEGMENT:
-         vtkType = 3;
-         break;
-      case Geometry::Type::TRIANGLE:
-         vtkType = 5;
-         break;
-      case Geometry::Type::SQUARE:
-         // vtkType = 8;
-         vtkType = 9;
-         break;
-      case Geometry::Type::TETRAHEDRON:
-         vtkType = 10;
-         break;
-      case Geometry::Type::CUBE:
-         // vtkType = 11;
-         vtkType = 12;
-         break;
-      case Geometry::Type::PRISM:
-         vtkType = 13;
-         break;
-      default:
-         vtkType = 0;
-         break;
-   }
-   return vtkType;
-}
-
-bool adios2stream::IsConstantElementType(const Array<Element*>& elements ) const
-noexcept
-{
-   bool isConstType = true;
-   const Geometry::Type type = elements[0]->GetGeometryType();
-
-   for (int e = 1; e < elements.Size(); ++e)
-   {
-      if (type != elements[e]->GetGeometryType())
-      {
-         isConstType = false;
-         break;
-      }
-   }
-   return isConstType;
-}
-
-std::string adios2stream::VTKSchema() const noexcept
-{
-   std::string vtkSchema = R"(
-<?xml version="1.0"?>
-<VTKFile type="UnstructuredGrid" version="0.1" byte_order="LittleEndian">
-  <UnstructuredGrid>
-    <Piece NumberOfPoints="NumOfVertices" NumberOfCells="NumOfElements">
-      <Points>
-        <DataArray Name="vertices" />)";
-
-   vtkSchema += R"(
-      </Points>
-      <Cells>
-        <DataArray Name="connectivity" />
-        <DataArray Name="types" />
-      </Cells>
-      <PointData>)";
-
-   if (point_data_variables.empty())
-   {
-      vtkSchema += "\n";
-   }
-   else
-   {
-      for (const std::string& point_datum : point_data_variables )
-      {
-         vtkSchema += "        <DataArray Name=\"" + point_datum +"\"/>\n";
-      }
-   }
-
-   if (transient)
-   {
-      vtkSchema += "        <DataArray Name=\"TIME\">\n";
-      vtkSchema += "          TIME\n";
-      vtkSchema += "        </DataArray>\n";
-   }
-
-   vtkSchema += R"(
-       </PointData>
-       </Piece>
-     </UnstructuredGrid>
-   </VTKFile>)";
-
-   return vtkSchema;
-}
-
-adios2::Mode adios2stream::ToADIOS2Mode(const adios2stream::openmode mode) const
-noexcept
-{
-   adios2::Mode adios2Mode = adios2::Mode::Undefined;
-   switch (mode)
-   {
-      case adios2stream::openmode::out:
-         adios2Mode = adios2::Mode::Write;
-         break;
-      case adios2stream::openmode::in:
-         adios2Mode = adios2::Mode::Read;
-         break;
-      default:
-         const std::string message = "MFEM adios2stream ERROR: only "
-                                     "openmode::out and openmode::in "
-                                     " are valid, in call to adios2stream constructor";
-         mfem_error(message.c_str());
-   }
-   return adios2Mode;
-}
-
-}  // end namespace mfem
@@ -1,233 +0,0 @@
-// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
-// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
-// LICENSE and NOTICE for details. LLNL-CODE-806117.
-//
-// This file is part of the MFEM library. For more information and source code
-// availability visit https://mfem.org.
-//
-// MFEM is free software; you can redistribute it and/or modify it under the
-// terms of the BSD-3 license. We welcome feedback and contributions, see file
-// CONTRIBUTING.md for details.
-//
-// Created on: Jan 22, 2019
-// Author: William F Godoy godoywf@ornl.gov
-// adios2: Adaptable Input/Output System https://github.com/ornladios/ADIOS2
-
-#ifndef MFEM_ADIOS2STREAM
-#define MFEM_ADIOS2STREAM
-
-#include "../config/config.hpp"
-
-#include <map>
-#include <memory>  // std::unique_ptr
-#include <string>
-#include <set>
-
-#ifdef MFEM_USE_MPI
-#include <mpi.h>
-#endif
-
-#include <adios2.h>
-
-namespace mfem
-{
-
-// forward declaring classes to avoid circular references
-class Vector;
-class GridFunction;
-class Mesh;
-class ADIOS2DataCollection;
-
-template <class T>
-class Array;
-class Element;
-
-class adios2stream
-{
-   friend class Vector;
-   friend class GridFunction;
-   friend class Mesh;
-   friend class ADIOS2DataCollection;
-
-public:
-   /**
-    * Open modes for adios2stream (from std::fstream)
-    * out: write
-    * in:  read
-    * app: append
-    */
-   enum class openmode { out, in, app };
-
-   /** Print and Save modes, deferred is done at Close or EndStep, sync is immediate */
-   enum class mode {sync, deferred};
-
-   enum class data_type {none, point_data, cell_data};
-
-#ifdef MFEM_USE_MPI
-
-   /**
-    * adios2stream MPI constructor, allows for passing parameters in source
-    * code (compile-time) only.
-    * @param name stream name
-    * @param mode adios2stream::openmode::in (Read), adios2stream::openmode::out
-    * (Write)
-    * @param comm MPI communicator establishing domain for fstream
-    * @param engine_type available adios2 engine, default is BPFile
-    *        see https://adios2.readthedocs.io/en/latest/engines/engines.html
-    * @throws std::invalid_argument (user input error) or std::runtime_error
-    *         (system error)
-    */
-   adios2stream(const std::string& name, const openmode mode, MPI_Comm comm,
-                const std::string engine_type = "BPFile");
-#else
-   /**
-    * adios2stream Non-MPI serial constructor, allows for passing parameters in
-    * source code (compile-time) only.
-    * @param name stream name
-    * @param mode adios2stream::openmode::in (Read), adios2stream::openmode::out
-    * (Write)
-    * @param engine_type available adios2 engine, default is BPFile
-    * @throws std::invalid_argument (user input error) or std::runtime_error
-    *         (system error)
-    */
-   adios2stream(const std::string& name, const openmode mode,
-                const std::string engine_type = "BPFile");
-#endif
-
-   /** calls Close if stream is valid basically follows C++ RAII **/
-   virtual ~adios2stream();
-
-   /**
-    * Set parameters for a particular adios2stream Engine
-    * See https://adios2.readthedocs.io/en/latest/engines/engines.html
-    * @param parameters map of key/value string elements
-    */
-   void SetParameters(const std::map<std::string, std::string>& parameters =
-                         std::map<std::string, std::string>());
-
-   /**
-    * Single parameter version of SetParameters passing a key/value pair
-    * See https://adios2.readthedocs.io/en/latest/engines/engines.html
-    * @param key input parameter key
-    * @param value input parameter value
-    */
-   void SetParameter(const std::string key, const std::string value) noexcept;
-
-   /** Begins an I/O step */
-   void BeginStep();
-
-   /** Ends the current step, by default transports the data */
-   void EndStep();
-
-   /**
-    * Associates a physical time with the current I/O step as TIME variable
-    * @param time input physical time
-    */
-   void SetTime(const double time);
-
-   /**
-    * Associates a current time step (cycle) with the current I/O step as CYCLE variable
-    * @param cycle physical time
-    */
-   void SetCycle(const int cycle);
-
-   /**
-    * Input to the Global Geometry Refiner
-    * @param level input level
-    */
-   void SetRefinementLevel(const int level) noexcept;
-
-   /** Return the current step between BeginStep and EndStep */
-   size_t CurrentStep() const;
-
-   /** Finished interaction with adios2stream and flushes the data */
-   void Close();
-
-protected:
-
-   /**
-    * Called from friend class Mesh (which is called from ParMesh)
-    * @param mesh input Mesh object to print
-    * @param print_mode sync: one at a time, deferred: collected (pre-fetch)
-    */
-   void Print(const Mesh& mesh, const adios2stream::mode print_mode = mode::sync);
-
-   void Save(const GridFunction& grid_function, const std::string& variable_name,
-             const data_type type);
-
-private:
-   /** placeholder for engine name */
-   const std::string name;
-
-   /** placeholder for engine openmode */
-   const openmode adios2_openmode;
-
-   /** main adios2 object that owns all the io and engine components */
-   std::unique_ptr<adios2::ADIOS> adios;
-
-   /** io object to set parameters, variables and engines */
-   adios2::IO io;
-
-   /** heavy object doing system-level I/O operations */
-   adios2::Engine engine;
-
-   /** true: transient problem (SetTime is called) */
-   bool transient = false;
-
-   /** true : engine step is active after engine.BeginStep(),
-     *  false: inactive after engine.EndStep() */
-   bool active_step = false;
-
-   /** true: mesh is defined, false: not yet */
-   bool is_mesh_defined = false;
-
-   /** ordering of the nodes to be passed to the schema as an attribute
-    *  true: XXX YYY ZZZ, false: XYZ, XYZ, XYZ
-    *  if true it must swap the vertices to Ordering::byDIM*/
-   bool ordering_by_node = false;
-
-   /** true: refine solution at Save */
-   bool refine = true;
-
-   /** refinement level at Save and Print */
-   int refinement_level = 1;
-
-   /** save for point data */
-   size_t refined_mesh_nvertices = 0;
-
-   /** save for cell data */
-   size_t refined_mesh_nelements = 0;
-
-   /** saves the variable names representing point data */
-   std::set<std::string> point_data_variables;
-
-   /**
-    * Map glvis element types to VTK element types
-    * @param glvisType input
-    * @return VTK element type
-    */
-   int32_t GLVISToVTKType(const int glvisType) const noexcept;
-
-   /** sets the current vtk_schema from point data arrays to be parsed
-    *  in VTK for Paraview visualization */
-   std::string VTKSchema() const noexcept;
-
-   /**
-    * Checks if array of elements contains only constant types
-    * @param elements array input to check
-    * @return true: types are constant, false: mixed types
-    */
-   bool IsConstantElementType(const Array<Element*>& elements ) const noexcept;
-
-   /**
-    * Maps to appropriate adios2::Mode from out, in to write, read
-    * @param mode
-    * @return
-    */
-   adios2::Mode ToADIOS2Mode(const adios2stream::openmode mode) const noexcept;
-
-};
-
-}  // end namespace mfem
-
-#endif /* MFEM_ADIOS2STREAM */
@@ -114,16 +114,4 @@ void SetGlobalMPI_Comm(MPI_Comm comm);
 #define MFEM_THREAD_LOCAL thread_local


-// MFEM_DEPRECATED macro to mark obsolete functions and methods
-// see https://stackoverflow.com/questions/295120/c-mark-as-deprecated
-#if defined(__GNUC__) || defined(__clang__)
-#define MFEM_DEPRECATED __attribute__((deprecated))
-#elif defined(_MSC_VER)
-#define MFEM_DEPRECATED __declspec(deprecated)
-#else
-#pragma message("WARNING: You need to implement MFEM_DEPRECATED for this compiler")
-#define MFEM_DEPRECATED
-#endif
-
-
 #endif
@@ -147,9 +147,6 @@ const char *GetConfigStr()
 #endif
 #ifdef MFEM_USE_OCCA
      "MFEM_USE_OCCA\n"
-#endif
-#ifdef MFEM_USE_ADIOS2
-      "MFEM_USE_ADIOS2\n"
 #endif
      "MFEM_TIMER_TYPE = " EXPAND_AND_QUOTE(MFEM_TIMER_TYPE)
      ;
@@ -1590,8 +1590,6 @@ void HypreParMatrix::Destroy()
   }
 }

-#if MFEM_HYPRE_VERSION < 21400
-
 HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
                    double beta,  const HypreParMatrix &B)
 {
@@ -1609,39 +1607,6 @@ HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
   return C;
 }

-HypreParMatrix * ParAdd(const HypreParMatrix *A, const HypreParMatrix *B)
-{
-   hypre_ParCSRMatrix * C = internal::hypre_ParCSRMatrixAdd(*A,*B);
-
-   hypre_MatvecCommPkgCreate(C);
-
-   return new HypreParMatrix(C);
-}
-
-#else
-
-HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
-                    double beta,  const HypreParMatrix &B)
-{
-   hypre_ParCSRMatrix *C;
-   hypre_ParcsrAdd(alpha, A, beta, B, &C);
-   hypre_MatvecCommPkgCreate(C);
-
-   return new HypreParMatrix(C);
-}
-
-HypreParMatrix * ParAdd(const HypreParMatrix *A, const HypreParMatrix *B)
-{
-   hypre_ParCSRMatrix *C;
-   hypre_ParcsrAdd(1.0, *A, 1.0, *B, &C);
-
-   hypre_MatvecCommPkgCreate(C);
-
-   return new HypreParMatrix(C);
-}
-
-#endif
-
 HypreParMatrix * ParMult(const HypreParMatrix *A, const HypreParMatrix *B,
                         bool own_matrix)
 {
@@ -1659,6 +1624,15 @@ HypreParMatrix * ParMult(const HypreParMatrix *A, const HypreParMatrix *B,
   return C;
 }

+HypreParMatrix * ParAdd(const HypreParMatrix *A, const HypreParMatrix *B)
+{
+   hypre_ParCSRMatrix * C = internal::hypre_ParCSRMatrixAdd(*A,*B);
+
+   hypre_MatvecCommPkgCreate(C);
+
+   return new HypreParMatrix(C);
+}
+
 HypreParMatrix * RAP(const HypreParMatrix *A, const HypreParMatrix *P)
 {
   HYPRE_Int P_owns_its_col_starts =
@@ -543,59 +543,4 @@ void RectangularConstrainedOperator::Mult(const Vector &x, Vector &y) const
   }
 }

-double PowerMethod::EstimateLargestEigenvalue(Operator& opr, Vector& v0,
-                                              int numSteps, double tolerance, int seed)
-{
-   v1.SetSize(v0.Size());
-   v0.Randomize(seed);
-
-   double eigenvalue = 1.0;
-
-   for (int iter = 0; iter < numSteps; ++iter)
-   {
-      double normV0;
-
-#ifdef MFEM_USE_MPI
-      if (comm != MPI_COMM_NULL)
-      {
-         normV0 = InnerProduct(comm, v0, v0);
-      }
-      else
-      {
-         normV0 = InnerProduct(v0, v0);
-      }
-#else
-      normV0 = InnerProduct(v0, v0);
-#endif
-
-      v0 /= sqrt(normV0);
-      opr.Mult(v0, v1);
-
-      double eigenvalueNew;
-#ifdef MFEM_USE_MPI
-      if (comm != MPI_COMM_NULL)
-      {
-         eigenvalueNew = InnerProduct(comm, v0, v1);
-      }
-      else
-      {
-         eigenvalueNew = InnerProduct(v0, v1);
-      }
-#else
-      eigenvalueNew = InnerProduct(v0, v1);
-#endif
-      double diff = std::abs((eigenvalueNew - eigenvalue) / eigenvalue);
-
-      eigenvalue = eigenvalueNew;
-      std::swap(v0, v1);
-
-      if (diff < tolerance)
-      {
-         break;
-      }
-   }
-
-   return eigenvalue;
-}
-
 }
@@ -762,39 +762,6 @@ public:
   virtual ~RectangularConstrainedOperator() { if (own_A) { delete A; } }
 };

-/** @brief PowerMethod helper class to estimate the largest eigenvalue of an
-           operator using the iterative power method. */
-class PowerMethod
-{
-   Vector v1;
-#ifdef MFEM_USE_MPI
-   MPI_Comm comm;
-#endif
-
-public:
-
-#ifdef MFEM_USE_MPI
-   PowerMethod() : comm(MPI_COMM_NULL) {}
-#else
-   PowerMethod() {}
-#endif
-
-#ifdef MFEM_USE_MPI
-   PowerMethod(MPI_Comm _comm) : comm(_comm) {}
-#endif
-
-   /// @brief Returns an estimate of the largest eigenvalue of the operator \p opr
-   /// using the iterative power method.
-   /** \p v0 is being used as the vector for the iterative process and will contain
-       the eigenvector corresponding to the largest eigenvalue after convergence.
-       The maximum number of iterations may set with \p numSteps, the relative
-       tolerance with \p tolerance and the seed of the random initialization of
-       \p v0 with \p seed. */
-   double EstimateLargestEigenvalue(Operator& opr, Vector& v0,
-                                    int numSteps = 10, double tolerance = 1e-8,
-                                    int seed = 12345);
-};
-
 }

 #endif
@@ -170,174 +170,6 @@ void OperatorJacobiSmoother::Mult(const Vector &x, Vector &y) const
   MFEM_FORALL(i, N, Y[i] += DI[i] * R[i]; );
 }

-OperatorChebyshevSmoother::OperatorChebyshevSmoother(Operator* oper_,
-                                                     const Vector &d,
-                                                     const Array<int>& ess_tdofs,
-                                                     int order_, double max_eig_estimate_)
-   :
-   Solver(d.Size()),
-   order(order_),
-   max_eig_estimate(max_eig_estimate_),
-   N(d.Size()),
-   dinv(N),
-   diag(d),
-   coeffs(order),
-   ess_tdof_list(ess_tdofs),
-   residual(N),
-   oper(oper_) { Setup(); }
-
-#ifdef MFEM_USE_MPI
-OperatorChebyshevSmoother::OperatorChebyshevSmoother(Operator* oper_,
-                                                     const Vector &d,
-                                                     const Array<int>& ess_tdofs,
-                                                     int order_, MPI_Comm comm, int power_iterations, double power_tolerance)
-#else
-OperatorChebyshevSmoother::OperatorChebyshevSmoother(Operator* oper_,
-                                                     const Vector &d,
-                                                     const Array<int>& ess_tdofs,
-                                                     int order_, int power_iterations, double power_tolerance)
-#endif
-   : Solver(d.Size()),
-     order(order_),
-     N(d.Size()),
-     dinv(N),
-     diag(d),
-     coeffs(order),
-     ess_tdof_list(ess_tdofs),
-     residual(N),
-     oper(oper_)
-{
-   OperatorJacobiSmoother invDiagOperator(diag, ess_tdofs, 1.0);
-   ProductOperator diagPrecond(&invDiagOperator, oper, false, false);
-
-#ifdef MFEM_USE_MPI
-   PowerMethod powerMethod(comm);
-#else
-   PowerMethod powerMethod;
-#endif
-   Vector ev(oper->Width());
-   max_eig_estimate = powerMethod.EstimateLargestEigenvalue(diagPrecond, ev,
-                                                            power_iterations, power_tolerance);
-
-   Setup();
-}
-
-void OperatorChebyshevSmoother::Setup()
-{
-   // Invert diagonal
-   residual.UseDevice(true);
-   auto D = diag.Read();
-   auto X = dinv.Write();
-   MFEM_FORALL(i, N, X[i] = 1.0 / D[i]; );
-   auto I = ess_tdof_list.Read();
-   MFEM_FORALL(i, ess_tdof_list.Size(), X[I[i]] = 1.0; );
-
-   // Set up Chebyshev coefficients
-   // For reference, see e.g., Parallel multigrid smoothing: polynomial versus
-   // Gauss-Seidel by Adams et al.
-   double upper_bound = 1.2 * max_eig_estimate;
-   double lower_bound = 0.3 * max_eig_estimate;
-   double theta = 0.5 * (upper_bound + lower_bound);
-   double delta = 0.5 * (upper_bound - lower_bound);
-
-   switch (order-1)
-   {
-      case 0:
-      {
-         coeffs[0] = 1.0 / theta;
-         break;
-      }
-      case 1:
-      {
-         double tmp_0 = 1.0/(pow(delta, 2) - 2*pow(theta, 2));
-         coeffs[0] = -4*theta*tmp_0;
-         coeffs[1] = 2*tmp_0;
-         break;
-      }
-      case 2:
-      {
-         double tmp_0 = 3*pow(delta, 2);
-         double tmp_1 = pow(theta, 2);
-         double tmp_2 = 1.0/(-4*pow(theta, 3) + theta*tmp_0);
-         coeffs[0] = tmp_2*(tmp_0 - 12*tmp_1);
-         coeffs[1] = 12/(tmp_0 - 4*tmp_1);
-         coeffs[2] = -4*tmp_2;
-         break;
-      }
-      case 3:
-      {
-         double tmp_0 = pow(delta, 2);
-         double tmp_1 = pow(theta, 2);
-         double tmp_2 = 8*tmp_0;
-         double tmp_3 = 1.0/(pow(delta, 4) + 8*pow(theta, 4) - tmp_1*tmp_2);
-         coeffs[0] = tmp_3*(32*pow(theta, 3) - 16*theta*tmp_0);
-         coeffs[1] = tmp_3*(-48*tmp_1 + tmp_2);
-         coeffs[2] = 32*theta*tmp_3;
-         coeffs[3] = -8*tmp_3;
-         break;
-      }
-      case 4:
-      {
-         double tmp_0 = 5*pow(delta, 4);
-         double tmp_1 = pow(theta, 4);
-         double tmp_2 = pow(theta, 2);
-         double tmp_3 = pow(delta, 2);
-         double tmp_4 = 60*tmp_3;
-         double tmp_5 = 20*tmp_3;
-         double tmp_6 = 1.0/(16*pow(theta, 5) - pow(theta, 3)*tmp_5 + theta*tmp_0);
-         double tmp_7 = 160*tmp_2;
-         double tmp_8 = 1.0/(tmp_0 + 16*tmp_1 - tmp_2*tmp_5);
-         coeffs[0] = tmp_6*(tmp_0 + 80*tmp_1 - tmp_2*tmp_4);
-         coeffs[1] = tmp_8*(tmp_4 - tmp_7);
-         coeffs[2] = tmp_6*(-tmp_5 + tmp_7);
-         coeffs[3] = -80*tmp_8;
-         coeffs[4] = 16*tmp_6;
-         break;
-      }
-      default:
-         MFEM_ABORT("Chebyshev smoother not implemented for order = " << order);
-   }
-}
-
-void OperatorChebyshevSmoother::Mult(const Vector& x, Vector &y) const
-{
-   if (iterative_mode)
-   {
-      MFEM_ABORT("Chebyshev smoother not implemented for iterative mode");
-   }
-
-   if (!oper)
-   {
-      MFEM_ABORT("Chebyshev smoother requires operator");
-   }
-
-   residual = x;
-   helperVector.SetSize(x.Size());
-
-   y.UseDevice(true);
-   y = 0.0;
-
-   for (int k = 0; k < order; ++k)
-   {
-      // Apply
-      if (k > 0)
-      {
-         oper->Mult(residual, helperVector);
-         residual = helperVector;
-      }
-
-      // Scale residual by inverse diagonal
-      const int n = N;
-      auto Dinv = dinv.Read();
-      auto R = residual.ReadWrite();
-      MFEM_FORALL(i, n, R[i] *= Dinv[i]; );
-
-      // Add weighted contribution to y
-      auto Y = y.ReadWrite();
-      auto C = coeffs.Read();
-      MFEM_FORALL(i, n, Y[i] += C[k] * R[i]; );
-   }
-}

 void SLISolver::UpdateVectors()
 {
@@ -550,24 +382,13 @@ void CGSolver::Mult(const Vector &b, Vector &x) const
   }
   nom0 = nom = Dot(d, r);
   MFEM_ASSERT(IsFinite(nom), "nom = " << nom);
+
   if (print_level == 1 || print_level == 3)
   {
      mfem::out << "   Iteration : " << setw(3) << 0 << "  (B r, r) = "
                << nom << (print_level == 3 ? " ...\n" : "\n");
   }

-   if (nom < 0.0)
-   {
-      if (print_level >= 0)
-      {
-         mfem::out << "PCG: The preconditioner is not positive definite. (Br, r) = "
-                   << nom << '\n';
-      }
-      converged = 0;
-      final_iter = 0;
-      final_norm = nom;
-      return;
-   }
   r0 = std::max(nom*rel_tol*rel_tol, abs_tol*abs_tol);
   if (nom <= r0)
   {
@@ -615,17 +436,6 @@ void CGSolver::Mult(const Vector &b, Vector &x) const
         betanom = Dot(r, r);
      }
      MFEM_ASSERT(IsFinite(betanom), "betanom = " << betanom);
-      if (betanom < 0.0)
-      {
-         if (print_level >= 0)
-         {
-            mfem::out << "PCG: The preconditioner is not positive definite. (Br, r) = "
-                      << betanom << '\n';
-         }
-         converged = 0;
-         final_iter = i;
-         break;
-      }

      if (print_level == 1)
      {
@@ -115,65 +115,6 @@ private:
   const Operator *oper;
 };

-/// Chebyshev accelerated smoothing with given vector, no matrix necessary
-/** Potentially useful with tensorized operators, for example. This is just a
-    very basic Chebyshev iteration, if you want tolerances, iteration control,
-    etc. wrap this with SLISolver. */
-class OperatorChebyshevSmoother : public Solver
-{
-public:
-   /** Application is by *inverse* of the given vector. It is assumed the
-       underlying operator acts as the identity on entries in ess_tdof_list,
-       corresponding to (assembled) DIAG_ONE policy or ConstrainedOperator in
-       the matrix-free setting. The estimated largest eigenvalue of the
-       diagonally preconditoned operator must be provided via
-       max_eig_estimate. */
-   OperatorChebyshevSmoother(Operator* oper_, const Vector &d,
-                             const Array<int>& ess_tdof_list,
-                             int order, double max_eig_estimate);
-
-   /** Application is by *inverse* of the given vector. It is assumed the
-       underlying operator acts as the identity on entries in ess_tdof_list,
-       corresponding to (assembled) DIAG_ONE policy or ConstrainedOperator in
-       the matrix-free setting. The largest eigenvalue of the diagonally
-       preconditoned operator is estimated internally via a power method. The
-       accuracy of the estimated eigenvalue may be controlled via
-       power_iterations and power_tolerance. */
-#ifdef MFEM_USE_MPI
-   OperatorChebyshevSmoother(Operator* oper_, const Vector &d,
-                             const Array<int>& ess_tdof_list,
-                             int order, MPI_Comm comm = MPI_COMM_NULL, int power_iterations = 10,
-                             double power_tolerance = 1e-8);
-#else
-   OperatorChebyshevSmoother(Operator* oper_, const Vector &d,
-                             const Array<int>& ess_tdof_list,
-                             int order, int power_iterations = 10, double power_tolerance = 1e-8);
-#endif
-
-   ~OperatorChebyshevSmoother() {}
-
-   void Mult(const Vector&x, Vector &y) const;
-
-   void SetOperator(const Operator &op_)
-   {
-      oper = &op_;
-   }
-
-   void Setup();
-
-private:
-   const int order;
-   double max_eig_estimate;
-   const int N;
-   Vector dinv;
-   const Vector &diag;
-   Array<double> coeffs;
-   const Array<int>& ess_tdof_list;
-   mutable Vector residual;
-   mutable Vector helperVector;
-   const Operator* oper;
-};
-

 /// Stationary linear iteration: x <- x + B (b - A x)
 class SLISolver : public IterativeSolver
@@ -670,16 +670,6 @@ void Vector::Print(std::ostream &out, int width) const
   out << '\n';
 }

-#ifdef MFEM_USE_ADIOS2
-void Vector::Print(adios2stream &out,
-                   const std::string& variable_name) const
-{
-   if (!size) { return; }
-   data.Read(MemoryClass::HOST, size);
-   out.engine.Put(variable_name, &data[0] );
-}
-#endif
-
 void Vector::Print_HYPRE(std::ostream &out) const
 {
   int i;
@@ -13,9 +13,6 @@
 #define MFEM_VECTOR

 #include "../general/array.hpp"
-#ifdef MFEM_USE_ADIOS2
-#include "../general/adios2stream.hpp"
-#endif
 #include "../general/globals.hpp"
 #include "../general/mem_manager.hpp"
 #include "../general/device.hpp"
@@ -301,13 +298,6 @@ public:
   /// Prints vector to stream out.
   void Print(std::ostream &out = mfem::out, int width = 8) const;

-#ifdef MFEM_USE_ADIOS2
-   /// Prints vector to stream out.
-   /// @param out adios2stream output
-   /// @param variable_name variable name associated with current Vector
-   void Print(adios2stream & out, const std::string& variable_name) const;
-#endif
-
   /// Prints vector to stream out in HYPRE_Vector format.
   void Print_HYPRE(std::ostream &out) const;

@@ -117,7 +117,7 @@ EXAMPLE_SUBDIRS = sundials petsc pumi hiop ginkgo
 EXAMPLE_DIRS := examples $(addprefix examples/,$(EXAMPLE_SUBDIRS))
 EXAMPLE_TEST_DIRS := examples

-MINIAPP_SUBDIRS = common electromagnetics meshing navier performance tools toys nurbs gslib
+MINIAPP_SUBDIRS = common electromagnetics meshing performance tools toys nurbs gslib
 MINIAPP_DIRS := $(addprefix miniapps/,$(MINIAPP_SUBDIRS))
 MINIAPP_TEST_DIRS := $(filter-out %/common,$(MINIAPP_DIRS))
 MINIAPP_USE_COMMON := $(addprefix miniapps/,electromagnetics tools toys)
@@ -392,9 +392,6 @@ endif
 # Source dirs in logical order
 DIRS = general linalg mesh fem fem/libceed
 SOURCE_FILES = $(foreach dir,$(DIRS),$(wildcard $(SRC)$(dir)/*.cpp))
-ADIOS2_FILES = $(SRC)general/adios2stream.h $(SRC)general/adios2stream.cpp \
- $(SRC)fem/adios2datacollection.hpp $(SRC)fem/adios2datacollection.cpp
-SOURCE_FILES := $(filter-out $(ADIOS2_FILES),$(SOURCE_FILES))
 RELSRC_FILES = $(patsubst $(SRC)%,%,$(SOURCE_FILES))
 OBJECT_FILES = $(patsubst $(SRC)%,$(BLD)%,$(SOURCE_FILES:.cpp=.o))
 OKL_DIRS = fem
@@ -75,7 +75,7 @@ public:
   virtual const int *GetEdgeVertices(int) const = 0;

   /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const = 0;
+   virtual int GetNFaces(int &nFaceVertices) const = 0;

   virtual int GetNFaces() const = 0;

@@ -52,7 +52,7 @@ public:
   { return geom_t::Edges[ei]; }

   /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
+   virtual int GetNFaces(int &nFaceVertices) const
   { nFaceVertices = 4; return 6; }

   virtual int GetNFaces() const { return 6; }
@@ -360,6 +360,7 @@ void Mesh::GetElementTransformation(int i, IsoparametricTransformation *ElTr)
      }
      ElTr->SetFE(Nodes->FESpace()->GetFE(i));
   }
+   ElTr->FinalizeTransformation();
 }

 void Mesh::GetElementTransformation(int i, const Vector &nodes,
@@ -401,6 +402,7 @@ void Mesh::GetElementTransformation(int i, const Vector &nodes,
      }
      ElTr->SetFE(Nodes->FESpace()->GetFE(i));
   }
+   ElTr->FinalizeTransformation();
 }

 ElementTransformation *Mesh::GetElementTransformation(int i)
@@ -468,6 +470,7 @@ void Mesh::GetBdrElementTransformation(int i, IsoparametricTransformation* ElTr)
         ElTr->SetFE(face_el);
      }
   }
+   ElTr->FinalizeTransformation();
 }

 void Mesh::GetFaceTransformation(int FaceNo, IsoparametricTransformation *FTr)
@@ -532,6 +535,7 @@ void Mesh::GetFaceTransformation(int FaceNo, IsoparametricTransformation *FTr)
         FTr->SetFE(face_el);
      }
   }
+   FTr->FinalizeTransformation();
 }

 ElementTransformation *Mesh::GetFaceTransformation(int FaceNo)
@@ -593,6 +597,7 @@ void Mesh::GetEdgeTransformation(int EdgeNo, IsoparametricTransformation *EdTr)
         MFEM_ABORT("Not implemented.");
      }
   }
+   EdTr->FinalizeTransformation();
 }

 ElementTransformation *Mesh::GetEdgeTransformation(int EdgeNo)
@@ -614,6 +619,7 @@ void Mesh::GetLocalPtToSegTransformation(
   locpm(0, 0) = SegVert->IntPoint(i/64).x;
   //  (i/64) is the local face no. in the segment
   //  (i%64) is the orientation of the point (not used)
+   Transf.FinalizeTransformation();
 }

 void Mesh::GetLocalSegToTriTransformation(
@@ -633,6 +639,7 @@ void Mesh::GetLocalSegToTriTransformation(
      locpm(0, so[j]) = TriVert->IntPoint(tv[j]).x;
      locpm(1, so[j]) = TriVert->IntPoint(tv[j]).y;
   }
+   Transf.FinalizeTransformation();
 }

 void Mesh::GetLocalSegToQuadTransformation(
@@ -652,6 +659,7 @@ void Mesh::GetLocalSegToQuadTransformation(
      locpm(0, so[j]) = QuadVert->IntPoint(qv[j]).x;
      locpm(1, so[j]) = QuadVert->IntPoint(qv[j]).y;
   }
+   Transf.FinalizeTransformation();
 }

 void Mesh::GetLocalTriToTetTransformation(
@@ -675,6 +683,7 @@ void Mesh::GetLocalTriToTetTransformation(
      locpm(1, j) = vert.y;
      locpm(2, j) = vert.z;
   }
+   Transf.FinalizeTransformation();
 }

 void Mesh::GetLocalTriToWdgTransformation(
@@ -700,6 +709,7 @@ void Mesh::GetLocalTriToWdgTransformation(
      locpm(1, j) = vert.y;
      locpm(2, j) = vert.z;
   }
+   Transf.FinalizeTransformation();
 }

 void Mesh::GetLocalQuadToHexTransformation(
@@ -721,6 +731,7 @@ void Mesh::GetLocalQuadToHexTransformation(
      locpm(1, j) = vert.y;
      locpm(2, j) = vert.z;
   }
+   Transf.FinalizeTransformation();
 }

 void Mesh::GetLocalQuadToWdgTransformation(
@@ -744,6 +755,7 @@ void Mesh::GetLocalQuadToWdgTransformation(
      locpm(1, j) = vert.y;
      locpm(2, j) = vert.z;
   }
+   Transf.FinalizeTransformation();
 }

 const GeometricFactors* Mesh::GetGeometricFactors(const IntegrationRule& ir,
@@ -927,7 +939,8 @@ void Mesh::ApplyLocalSlaveTransformation(IsoparametricTransformation &transf,
 #endif
   MFEM_ASSERT(fi.NCFace >= 0, "");
   transf.Transform(*nc_faces_info[fi.NCFace].PointMatrix, composition);
-   transf.SetPointMat(composition);
+   transf.GetPointMat() = composition;
+   transf.FinalizeTransformation();
 }

 FaceElementTransformations *Mesh::GetBdrFaceTransformations(int BdrElemNo)
@@ -4833,12 +4846,10 @@ void Mesh::GetPointMatrix(int i, DenseMatrix &pointmat) const

   pointmat.SetSize(spaceDim, nv);
   for (k = 0; k < spaceDim; k++)
-   {
      for (j = 0; j < nv; j++)
      {
         pointmat(k, j) = vertices[v[j]](k);
      }
-   }
 }

 void Mesh::GetBdrPointMatrix(int i,DenseMatrix &pointmat) const
@@ -8605,13 +8616,6 @@ void Mesh::PrintTopo(std::ostream &out,const Array<int> &e_to_k) const
   out << "\nvertices\n" << NumOfVertices << '\n';
 }

-#ifdef MFEM_USE_ADIOS2
-void Mesh::Print(adios2stream &out) const
-{
-   out.Print(*this);
-}
-#endif
-
 void Mesh::PrintVTK(std::ostream &out)
 {
   out <<
@@ -10424,7 +10428,6 @@ GeometricFactors::GeometricFactors(const Mesh *mesh, const IntegrationRule &ir,
   const GridFunction *nodes = mesh->GetNodes();
   const FiniteElementSpace *fespace = nodes->FESpace();
   const FiniteElement *fe = fespace->GetFE(0);
-   const int dim  = fe->GetDim();
   const int vdim = fespace->GetVDim();
   const int NE   = fespace->GetNE();
   const int ND   = fe->GetDof();
@@ -10442,7 +10445,7 @@ GeometricFactors::GeometricFactors(const Mesh *mesh, const IntegrationRule &ir,
   }
   if (flags & GeometricFactors::JACOBIANS)
   {
-      J.SetSize(dim*vdim*NQ*NE);
+      J.SetSize(vdim*vdim*NQ*NE);
      eval_flags |= QuadratureInterpolator::DERIVATIVES;
   }
   if (flags & GeometricFactors::DETERMINANTS)
@@ -23,9 +23,6 @@
 #include "../fem/eltrans.hpp"
 #include "../fem/coefficient.hpp"
 #include "../general/zstr.hpp"
-#ifdef MFEM_USE_ADIOS2
-#include "../general/adios2stream.hpp"
-#endif
 #include <iostream>

 namespace mfem
@@ -58,10 +55,6 @@ class Mesh
   friend class NCMesh;
   friend class NURBSExtension;

-#ifdef MFEM_USE_ADIOS2
-   friend class adios2stream;
-#endif
-
 protected:
   int Dim;
   int spaceDim;
@@ -1202,10 +1195,6 @@ public:
   /// \see mfem::ofgzstream() for on-the-fly compression of ascii outputs
   virtual void Print(std::ostream &out = mfem::out) const { Printer(out); }

-   /// Print the mesh to the given stream using the adios2 bp format
-#ifdef MFEM_USE_ADIOS2
-   virtual void Print(adios2stream &out) const;
-#endif
   /// Print the mesh in VTK format (linear and quadratic meshes only).
   /// \see mfem::ofgzstream() for on-the-fly compression of ascii outputs
   void PrintVTK(std::ostream &out);
@@ -1695,6 +1695,7 @@ void ParMesh::GetFaceNbrElementTransformation(
         MFEM_ABORT("Nodes are not ParGridFunction!");
      }
   }
+   ElTr->FinalizeTransformation();
 }

 void ParMesh::DeleteFaceNbrData()
@@ -2354,6 +2355,7 @@ ElementTransformation* ParMesh::GetGhostFaceTransformation(
 #endif
      FaceTransformation.SetFE(face_el);
   }
+   FaceTransformation.FinalizeTransformation();
   return &FaceTransformation;
 }

@@ -4221,13 +4223,6 @@ void ParMesh::Print(std::ostream &out) const
   }
 }

-#ifdef MFEM_USE_ADIOS2
-void ParMesh::Print(adios2stream &out) const
-{
-   Mesh::Print(out);
-}
-#endif
-
 static void dump_element(const Element* elem, Array<int> &data)
 {
   data.Append(elem->GetGeometryType());
@@ -309,12 +309,6 @@ public:
       as boundary (for visualization purposes) using the mfem v1.0 format. */
   virtual void Print(std::ostream &out = mfem::out) const;

-#ifdef MFEM_USE_ADIOS2
-   /** Print the part of the mesh in the calling processor using adios2 bp
-       format. */
-   virtual void Print(adios2stream &out) const;
-#endif
-
   /** Print the part of the mesh in the calling processor adding the interface
       as boundary (for visualization purposes) using Netgen/Truegrid format .*/
   virtual void PrintXG(std::ostream &out = mfem::out) const;
@@ -354,10 +348,6 @@ public:
 #ifdef MFEM_USE_PUMI
   friend class ParPumiMesh;
 #endif
-
-#ifdef MFEM_USE_ADIOS2
-   friend class adios2stream;
-#endif
 };

 }
@@ -47,7 +47,7 @@ public:
   virtual const int *GetEdgeVertices(int ei) const { return NULL; }

   /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
+   virtual int GetNFaces(int &nFaceVertices) const
   { nFaceVertices = 0; return 0; }

   virtual int GetNFaces() const { return 0; }
@@ -54,7 +54,7 @@ public:
   { return geom_t::Edges[ei]; }

   /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
+   virtual int GetNFaces(int &nFaceVertices) const
   { nFaceVertices = 0; return 0; }

   virtual int GetNFaces() const { return 0; }
@@ -53,7 +53,7 @@ public:
   virtual const int *GetEdgeVertices(int ei) const { return NULL; }

   /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
+   virtual int GetNFaces(int &nFaceVertices) const
   { nFaceVertices = 0; return 0; }

   virtual int GetNFaces() const { return 0; }
@@ -101,7 +101,7 @@ public:
   { return geom_t::Edges[ei]; }

   /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
+   virtual int GetNFaces(int &nFaceVertices) const
   { nFaceVertices = 3; return 4; }

   virtual int GetNFaces() const { return 4; }
@@ -81,7 +81,7 @@ public:
   { return geom_t::Edges[ei]; }

   /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const
+   virtual int GetNFaces(int &nFaceVertices) const
   { nFaceVertices = 0; return 0; }

   virtual int GetNFaces() const { return 0; }
@@ -13,7 +13,6 @@
 #define MFEM_VERTEX

 #include "../config/config.hpp"
-#include "../general/globals.hpp"

 namespace mfem
 {
@@ -45,7 +44,7 @@ public:

   /// (DEPRECATED) Set the coordinates of the Vertex.
   /** @deprecated This old version of SetCoords is not always memory safe. */
-   MFEM_DEPRECATED void SetCoords(const double *p)
+   void SetCoords(const double *p)
   { coord[0] = p[0]; coord[1] = p[1]; coord[2] = p[2]; }

   /// Sets vertex location based on given point p
@@ -55,7 +55,7 @@ public:
   { return geom_t::Edges[ei]; }

   /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
-   MFEM_DEPRECATED virtual int GetNFaces(int &nFaceVertices) const;
+   virtual int GetNFaces(int &nFaceVertices) const;

   virtual int GetNFaces() const { return 5; }

--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Dylan Copeland	b51732c827	Extended PA for some vector mass and curl integrators to handle diagonal matrix coefficients. Added unit tests.	2020-03-31 11:56:02 -07:00
Dylan Copeland	d838c6b33b	Implemented PA for MixedVectorCurlIntegrator and MixedVectorWeakCurlIntegrator. Added unit tests for these integrators.	2020-03-30 18:02:46 -07:00