Merge remote-tracking branch 'origin/master' into entity-sets-dev

# Conflicts: # mesh/pncmesh.hpp
2021-12-07 19:11:39 -08:00 · 2021-12-07 15:01:43 -08:00 · 2021-07-06 17:12:03 -07:00 · 2021-07-06 17:11:43 -07:00 · 2020-02-20 11:20:03 -08:00 · 2020-02-20 11:18:15 -08:00
241 changed files with 10303 additions and 10612 deletions
@@ -259,8 +259,6 @@ miniapps/performance/sol.*

 miniapps/shifted/distance
 miniapps/shifted/ParaViewDistance
-miniapps/shifted/extrapolate
-miniapps/shifted/ParaViewExtrapolate
 miniapps/shifted/diffusion
 miniapps/shifted/diffusion.mesh
 miniapps/shifted/diffusion.gf
@@ -45,5 +45,5 @@ variables:
    - echo ${MFEM_DATA_DIR}
    - echo ${SPEC}
    # Next script uses 'THREADS': leaving it empty --> it uses 'make all -j'
-    - lalloc 1 -W 30 -q pdebug --atsdisable tests/gitlab/build_and_test --spec "${SPEC}" --data-dir "${MFEM_DATA_DIR}" --data
+    - lalloc 1 -W 30 -q pdebug tests/gitlab/build_and_test --spec "${SPEC}" --data-dir "${MFEM_DATA_DIR}" --data
  needs: [setup]
@@ -19,11 +19,6 @@ opt_mpi_cuda_xl_16_1_1_8:
    SPEC: "%xl@16.1.1.8 +mpi +cuda cuda_arch=70"
  extends: .build_and_test_on_lassen

-opt_mpi_cuda_hypre_cuda_xl:
-  variables:
-    SPEC: "%xl@16.1.1.8 +mpi +cuda cuda_arch=70 ^hypre+cuda~shared cuda_arch=70"
-  extends: .build_and_test_on_lassen
-
 # Jobs report
 report_job_success:
  stage: report
@@ -59,7 +59,7 @@ opt_par_gcc_6_1_0_sundials:

 opt_par_gcc_6_1_0_petsc:
  variables:
-    SPEC: "%gcc@6.1.0 +petsc ^petsc+mumps~superlu-dist"
+    SPEC: "%gcc@6.1.0 +petsc ^petsc+mumps"
  extends: .build_and_test_on_quartz

 opt_par_gcc_6_1_0_pumi:
@@ -29,9 +29,9 @@ mkdir _${BASELINE_TEST} && cd _${BASELINE_TEST}

 # run
 if [[ "${MACHINE_NAME}" == "quartz" || "${MACHINE_NAME}" == "ruby" ]]; then
-  salloc --nodes=1 -p pdebug ../runtest ../../mfem "${BASELINE_TEST} ${TPLS_DIR}"
+  srun --nodes=1 -p pdebug ../runtest ../../mfem "${BASELINE_TEST} ${TPLS_DIR}"
 elif [[ ${MACHINE_NAME} == "corona" ]]; then
-  salloc --nodes=1 -t 60 -p pbatch ../runtest ../../mfem "${BASELINE_TEST} ${TPLS_DIR}"
+  srun --nodes=1 -t 60 -p mi60 ../runtest ../../mfem "${BASELINE_TEST} ${TPLS_DIR}"
 elif [[ ${MACHINE_NAME} == "lassen" ]]; then
  lalloc 1 -q pdebug ../runtest ../../mfem "${BASELINE_TEST} ${TPLS_DIR}"
 else
@@ -10,27 +10,10 @@

 Version 4.3.1 (development)
 ===========================
-
- Add hipSPARSE support for sparse mat-vec multiplications.
-
- Added support for using the HYPRE library built with HIP support. Similar to
-  the HYPRE + CUDA support added earlier, most of the MFEM examples and miniapps
-  work transparently with HYPRE + HIP builds. This includes the BoomerAMG, AMS,
-  and ADS solvers.
-
- More explicit and consistent formating of the output of iterative solvers
-  with the new IterativeSolver::PrintLevel options. See linalg/solvers.hpp.
-
- Added a miniapp for PDE-based extrapolation of finite element functions. See
-  miniapps/shifted/extrapolate.cpp.
-
- Added support for automatic differentiation. Users can select between native
-  implementation and external library implementation during configuration. One
-  parallel and two serial examples are implemented in the miniapps/autodiff/
-  directory.
-
- GridFunctionCoefficient (and the related vector, gradient, divergence, and
-  curl classes) now work properly with LORDiscretization and LORSolver.
+- Added support for automatic differentiation. Users can select between 
+  native implementation and external library implementation at the 
+  configuration phase. A parallel and two serial examples are implemented
+  in the autodiff miniapp directory.

 - Added support for mesh preprocessing to resolve fine scale problem data
  before simulation. This feature uses adaptive mesh refinement to control the
@@ -88,19 +71,13 @@ Version 4.3.1 (development)

 - Added initial TMOP-based capabilities for surface fitting and tangential
  relaxation in the mesh-optimizer and pmesh-optimizer miniapps.
-
+ 
 - Added ParMesh Adjaceny Set (adjset) creation support to the Conduit Mesh
  Blueprint MFEM wrapper functions in ConduitDataCollection.

 - `HypreParVector` and `Vector` now support move semantics, and the copy
  constructor for `HypreParVector` now copies the local vector data.

- The HPC versions of ex1 and ex1p (in miniapps/performance) now support
-  runtime selection of either 2D or 3D meshes.
-
- Added ParaView visualization of `QuadratureFunction` fields, through both
-  `QuadratureFunction::SaveVTU` and `ParaViewDataCollection::RegisterQField`.
-

 Version 4.3, released on July 29, 2021
 ======================================
@@ -141,12 +118,6 @@ Discretization improvements

 - Added support for nonscalar coefficient with VectorDiffusionIntegrator.

- Added support for Partial Assembly with Discontinuous Galerkin methods on
-  nonconforming meshes.
-
- Added a simpler interface to request face information: see
-  `Mesh::FaceInformation` and `Mesh::GetFaceInformation`.
-
 Linear and nonlinear solvers
 ----------------------------
 - Added support for AMG preconditioners on GPUs based on the hypre library
@@ -16,6 +16,9 @@ set(USER_CONFIG "${CMAKE_CURRENT_SOURCE_DIR}/config/user.cmake" CACHE PATH

 # Require C++11 and disable compiler-specific extensions
 set(CMAKE_CXX_STANDARD 11)
+if (MFEM_USE_GINKGO)
+  set(CMAKE_CXX_STANDARD 14)
+endif()
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
 set(CMAKE_CXX_EXTENSIONS OFF)

@@ -81,9 +84,6 @@ if (MFEM_USE_STRUMPACK)
  # Just needed to find the MPI_Fortran libraries to link with
  set(XSDK_ENABLE_Fortran ON)
 endif()
-if (MFEM_USE_GINKGO AND ("${CMAKE_CXX_STANDARD}" LESS "14"))
-  set(CMAKE_CXX_STANDARD 14)
-endif()

 # Include xSDK default CMake file.
 include("${CMAKE_CURRENT_SOURCE_DIR}/config/XSDKDefaults.cmake")
@@ -180,19 +180,6 @@ else()
  set(MFEM_DEBUG OFF)
 endif()

-# AMD HIP
-if (MFEM_USE_HIP)
-  if (HIP_ARCH)
-    message(STATUS "Using HIP architecture: ${HIP_ARCH}")
-    set(GPU_TARGETS "${HIP_ARCH}" CACHE STRING "HIP targets to compile for")
-  endif()
-  if (ROCM_PATH)
-    list(INSERT CMAKE_PREFIX_PATH 0 ${ROCM_PATH})
-  endif()
-  find_package(HIP REQUIRED)
-  find_package(HIPSPARSE REQUIRED)
-endif()
-
 # MPI -> hypre; PETSc (optional)
 if (MFEM_USE_MPI)
  find_package(MPI REQUIRED)
@@ -367,7 +354,7 @@ endif()
 if (MFEM_USE_PUMI)
  # If PUMI_DIR was specified, only link to that directory,
  # i.e. don't link to another installation in /usr/lib by mistake
-  find_package(SCOREC 2.2.6 REQUIRED OPTIONAL_COMPONENTS gmi_sim
+  find_package(SCOREC 2.1.0 REQUIRED OPTIONAL_COMPONENTS gmi_sim
    CONFIG PATHS ${PUMI_DIR} NO_DEFAULT_PATH)
  if (SCOREC_FOUND)
    # Define a header file with the MFEM_USE_SIMMETRIX preprocessor variable
@@ -416,6 +403,18 @@ if (MFEM_USE_CALIPER)
  find_package(Caliper REQUIRED)
 endif()

+# AMD HIP
+if (MFEM_USE_HIP)
+  find_package(HIP REQUIRED)
+  if (HIP_ARCH)
+    message(STATUS "Using HIP architecture: ${HIP_ARCH}")
+    list(APPEND HIP_HIPCC_FLAGS "--amdgpu-target=${HIP_ARCH}")
+    if (MFEM_USE_GINKGO)
+      list(APPEND HIP_HIPCC_FLAGS "-std=c++14")
+    endif()
+  endif()
+endif()
+
 # ADIOS2 for parallel I/O
 if (MFEM_USE_ADIOS2)
   find_package(ADIOS2 REQUIRED)
@@ -455,11 +454,10 @@ endif()
 #    With newer versions of SuiteSparse which include METIS header using 64-bit
 #    integers, the METIS header (with 32-bit indices, as used by mfem) needs to
 #    be before SuiteSparse.
-set(MFEM_TPLS OPENMP HYPRE BLAS LAPACK SuperLUDist METIS SuiteSparse SUNDIALS
-    PETSC SLEPC MESQUITE MUMPS STRUMPACK AXOM FMS CONDUIT Ginkgo GNUTLS GSLIB
-    NETCDF MPFR PUMI HIOP POSIXCLOCKS MFEMBacktrace ZLIB OCCA CEED RAJA UMPIRE
-    ADIOS2 CUSPARSE MKL_CPARDISO AMGX CALIPER CODIPACK BENCHMARK PARELAG
-    MPI_CXX HIP HIPSPARSE)
+set(MFEM_TPLS OPENMP HYPRE BLAS LAPACK SuperLUDist METIS SuiteSparse SUNDIALS PETSC
+    SLEPC MESQUITE MUMPS STRUMPACK AXOM FMS CONDUIT Ginkgo GNUTLS GSLIB NETCDF
+    MPFR PUMI HIOP POSIXCLOCKS MFEMBacktrace ZLIB OCCA CEED RAJA UMPIRE ADIOS2
+    CUSPARSE MKL_CPARDISO AMGX CALIPER CODIPACK BENCHMARK PARELAG MPI_CXX)

 # Add all *_FOUND libraries in the variable TPL_LIBRARIES.
 set(TPL_LIBRARIES "")
@@ -502,6 +500,8 @@ endforeach()

 if (MFEM_USE_CUDA)
  set_source_files_properties(${SOURCES} PROPERTIES LANGUAGE CUDA)
+elseif(MFEM_USE_HIP)
+  set_source_files_properties(${SOURCES} PROPERTIES HIP_SOURCE_PROPERTY_FORMAT TRUE)
 endif()

 add_subdirectory(config)
@@ -1,133 +0,0 @@
-# MFEM Code of Conduct
-
-## Our Pledge
-
-We as members, contributors, and leaders pledge to make participation in our
-community a harassment-free experience for everyone, regardless of age, body
-size, visible or invisible disability, ethnicity, sex characteristics, gender
-identity and expression, level of experience, education, socio-economic status,
-nationality, personal appearance, race, caste, color, religion, or sexual
-identity and orientation.
-
-We pledge to act and interact in ways that contribute to an open, welcoming,
-diverse, inclusive, and healthy community.
-
-## Our Standards
-
-Examples of behavior that contributes to a positive environment for our
-community include:
-
-* Demonstrating empathy and kindness toward other people
-* Being respectful of differing opinions, viewpoints, and experiences
-* Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes,
-  and learning from the experience
-* Focusing on what is best not just for us as individuals, but for the overall
-  community
-
-Examples of unacceptable behavior include:
-
-* The use of sexualized language or imagery, and sexual attention or advances of
-  any kind
-* Trolling, insulting or derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or email address,
-  without their explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
-  professional setting
-
-## Enforcement Responsibilities
-
-Community leaders are responsible for clarifying and enforcing our standards of
-acceptable behavior and will take appropriate and fair corrective action in
-response to any behavior that they deem inappropriate, threatening, offensive,
-or harmful.
-
-Community leaders have the right and responsibility to remove, edit, or reject
-comments, commits, code, wiki edits, issues, and other contributions that are
-not aligned to this Code of Conduct, and will communicate reasons for moderation
-decisions when appropriate.
-
-## Scope
-
-This Code of Conduct applies within all community spaces, and also applies when
-an individual is officially representing the community in public spaces.
-Examples of representing our community include using an official e-mail address,
-posting via an official social media account, or acting as an appointed
-representative at an online or offline event.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported to the community leaders responsible for enforcement at mfem@llnl.gov.
-All complaints will be reviewed and investigated promptly and fairly.
-
-All community leaders are obligated to respect the privacy and security of the
-reporter of any incident. Anyone involved in the reported behavior will recuse
-themselves from the investigation and decision making about the resolution of
-the complaint.
-
-## Enforcement Guidelines
-
-Community leaders will follow these Community Impact Guidelines in determining
-the consequences for any action they deem in violation of this Code of Conduct:
-
-### 1. Correction
-
-**Community Impact**: Use of inappropriate language or other behavior deemed
-unprofessional or unwelcome in the community.
-
-**Consequence**: A private, written warning from community leaders, providing
-clarity around the nature of the violation and an explanation of why the
-behavior was inappropriate. A public apology may be requested.
-
-### 2. Warning
-
-**Community Impact**: A violation through a single incident or series of
-actions.
-
-**Consequence**: A warning with consequences for continued behavior. No
-interaction with the people involved, including unsolicited interaction with
-those enforcing the Code of Conduct, for a specified period of time. This
-includes avoiding interactions in community spaces as well as external channels
-like social media. Violating these terms may lead to a temporary or permanent
-ban.
-
-### 3. Temporary Ban
-
-**Community Impact**: A serious violation of community standards, including
-sustained inappropriate behavior.
-
-**Consequence**: A temporary ban from any sort of interaction or public
-communication with the community for a specified period of time. No public or
-private interaction with the people involved, including unsolicited interaction
-with those enforcing the Code of Conduct, is allowed during this period.
-Violating these terms may lead to a permanent ban.
-
-### 4. Permanent Ban
-
-**Community Impact**: Demonstrating a pattern of violation of community
-standards, including sustained inappropriate behavior, harassment of an
-individual, or aggression toward or disparagement of classes of individuals.
-
-**Consequence**: A permanent ban from any sort of public interaction within the
-community.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage],
-version 2.1, available at
-[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
-
-Community Impact Guidelines were inspired by
-[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
-
-For answers to common questions about this code of conduct, see the FAQ at
-[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
-[https://www.contributor-covenant.org/translations][translations].
-
-[homepage]: https://www.contributor-covenant.org
-[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
-[Mozilla CoC]: https://github.com/mozilla/diversity
-[FAQ]: https://www.contributor-covenant.org/faq
-[translations]: https://www.contributor-covenant.org/translations
@@ -21,9 +21,6 @@ documentation; new examples and miniapps; HPC performance improvements; etc.
 MFEM is distributed under the terms of the BSD-3 license. All new contributions
 must be made under this license.

-Note also that MFEM has a [Code of Conduct](CODE_OF_CONDUCT.md). By participating
-in the MFEM community, you agree to abide by its rules.
-
 If you plan on contributing to MFEM, consider reviewing the
 [issue tracker](https://github.com/mfem/mfem/issues) first to check if a thread
 already exists for your desired feature or the bug you ran into. Use a pull
@@ -37,7 +37,7 @@ as CUDA, HIP, OCCA, OpenMP and RAJA.
  https://developer.nvidia.com/cuda-toolkit

 - HIP support requires an AMD GPU and an installation of the ROCm software stack
-  https://rocmdocs.amd.com
+  https://rocm.github.io/ROCmInstall.html#installing-from-amd-rocm-repositories

 - OCCA support requires the OCCA library
  https://libocca.org
@@ -58,8 +58,7 @@ following package managers:

 - Spack, https://github.com/spack/spack
 - OpenHPC, http://openhpc.community
- Conda-forge, https://conda-forge.org (pre-built binaries linked with
-  OpenMPI/MPICH, hypre, and METIS)
+- Conda-forge, https://conda-forge.org (pre-built binaries linked with OpenMPI/MPICH, hypre, and METIS)
 - Homebrew/Science, https://github.com/Homebrew/homebrew-science (deprecated)

 We also recommend downloading and building the MFEM-based GLVis visualization
@@ -79,7 +78,7 @@ Parallel build:

 CUDA build:
   make cuda -j 4
-   (build for a specific compute capability: 'make cuda -j 4 CUDA_ARCH=sm_70')
+   (build for a specific compute capability: 'make cuda -j 4 CUDA_ARCH=sm_30')

 HIP build:
   make hip -j 4
@@ -567,7 +566,7 @@ The specific libraries and their options are:
  Options: HYPRE_OPT, HYPRE_LIB.
  Versions: HYPRE >= 2.10.0b  (HYPRE built without CUDA)
            HYPRE >= 2.20.0   (HYPRE built with '--enable-mixedint')
-            HYPRE >= 2.22.1   (HYPRE built with CUDA or HIP)
+            HYPRE >= 2.22.1   (HYPRE built with CUDA)

 - METIS, used when MFEM_USE_METIS = YES. If using METIS 5, set
  MFEM_USE_METIS_5 = YES (default is to use METIS 4).
@@ -640,8 +639,7 @@ The specific libraries and their options are:
  C++ compiler that supports the C++-14 standard. For additional requirements
  and dependencies of specific modules, see the Ginkgo webpage below.
  URL: https://ginkgo-project.github.io
-  Options: GINKGO_OPT, GINKGO_LIB, GINKGO_DIR, GINKGO_BUILD_TYPE (Release or
-           Debug).
+  Options: GINKGO_OPT, GINKGO_LIB, GINKGO_DIR, GINKGO_BUILD_TYPE (Release or Debug).
  Versions: Ginkgo >= 1.4.0.

 - AmgX (optional), used when MFEM_USE_AMGX = YES.
@@ -706,7 +704,7 @@ The specific libraries and their options are:
  URL: https://scorec.rpi.edu/pumi
       https://github.com/SCOREC/core
  Options: PUMI_OPT, PUMI_LIB.
-  Versions: PUMI >= 2.2.6.
+  Versions: PUMI == 2.2.3.

 - HiOp (optional), used when MFEM_USE_HIOP = YES.
  URL: https://github.com/LLNL/hiop
@@ -739,7 +737,7 @@ The specific libraries and their options are:
  Versions: CUDA >= 10.1.168.

 - HIP (optional), used when MFEM_USE_HIP = YES.
-  URL: https://rocmdocs.amd.com
+  URL: https://rocm.github.io/ROCmInstall.html
  Options: HIP_CXX, HIP_ARCH, HIP_OPT, HIP_LIB.

 - OCCA (optional), used when MFEM_USE_OCCA = YES.
@@ -926,7 +924,6 @@ MFEM_USE_HIOP
 MFEM_USE_CODIPACK
 MFEM_USE_ADFORWARD
 MFEM_USE_CUDA
-MFEM_USE_HIP
 MFEM_USE_OCCA
 MFEM_USE_CEED
 MFEM_USE_RAJA
@@ -12,9 +12,6 @@ to enable high-performance scalable finite element discretization research and
 application development on a wide variety of platforms, ranging from laptops to
 supercomputers.

-We welcome contributions and feedback from the community. Please see the file
-CONTRIBUTING.md for additional details about our development process.
-
 * For building instructions, see the file INSTALL, or type "make help".

 * Copyright and licensing information can be found in files LICENSE and NOTICE.
@@ -22,6 +19,9 @@ CONTRIBUTING.md for additional details about our development process.
 * The best starting point for new users interested in MFEM's features is to
  review the examples and miniapps at https://mfem.org/examples.

+* Developers interested in contributing to the library, should read the
+  instructions and documentation in the CONTRIBUTING.md file.
+
 Conceptually, MFEM can be viewed as a finite element toolbox that provides the
 building blocks for developing finite element algorithms in a manner similar to
 that of MATLAB for linear algebra methods. In particular, MFEM provides support
@@ -0,0 +1,692 @@
+###############################################################################
+# FindHIP.cmake
+###############################################################################
+include(CheckCXXCompilerFlag)
+###############################################################################
+# SET: Variable defaults
+###############################################################################
+# User defined flags
+set(HIP_HIPCC_FLAGS "" CACHE STRING "Semicolon delimited flags for HIPCC")
+set(HIP_HCC_FLAGS "" CACHE STRING "Semicolon delimited flags for HCC")
+set(HIP_CLANG_FLAGS "" CACHE STRING "Semicolon delimited flags for CLANG")
+set(HIP_NVCC_FLAGS "" CACHE STRING "Semicolon delimted flags for NVCC")
+mark_as_advanced(HIP_HIPCC_FLAGS HIP_HCC_FLAGS HIP_CLANG_FLAGS HIP_NVCC_FLAGS)
+
+set(_hip_configuration_types ${CMAKE_CONFIGURATION_TYPES} ${CMAKE_BUILD_TYPE} Debug MinSizeRel Release RelWithDebInfo)
+list(REMOVE_DUPLICATES _hip_configuration_types)
+foreach(config ${_hip_configuration_types})
+    string(TOUPPER ${config} config_upper)
+    set(HIP_HIPCC_FLAGS_${config_upper} "" CACHE STRING "Semicolon delimited flags for HIPCC")
+    set(HIP_HCC_FLAGS_${config_upper} "" CACHE STRING "Semicolon delimited flags for HCC")
+    set(HIP_CLANG_FLAGS_${config_upper} "" CACHE STRING "Semicolon delimited flags for CLANG")
+    set(HIP_NVCC_FLAGS_${config_upper} "" CACHE STRING "Semicolon delimited flags for NVCC")
+    mark_as_advanced(HIP_HIPCC_FLAGS_${config_upper} HIP_HCC_FLAGS_${config_upper} HIP_CLANG_FLAGS_${config_upper} HIP_NVCC_FLAGS_${config_upper})
+endforeach()
+option(HIP_HOST_COMPILATION_CPP "Host code compilation mode" ON)
+option(HIP_VERBOSE_BUILD "Print out the commands run while compiling the HIP source file.  With the Makefile generator this defaults to VERBOSE variable specified on the command line, but can be forced on with this option." OFF)
+mark_as_advanced(HIP_HOST_COMPILATION_CPP)
+
+###############################################################################
+# FIND: HIP and associated helper binaries
+###############################################################################
+
+get_filename_component(_IMPORT_PREFIX "${CMAKE_CURRENT_LIST_DIR}/../" REALPATH)
+
+# HIP is supported on Linux only
+if(UNIX AND NOT APPLE AND NOT CYGWIN)
+    # Search for HIP installation
+    if(NOT HIP_ROOT_DIR)
+        # Search in user specified path first
+        find_path(
+            HIP_ROOT_DIR
+            NAMES bin/hipconfig
+            PATHS
+            "$ENV{ROCM_PATH}/hip"
+            ENV HIP_PATH
+            ${_IMPORT_PREFIX}
+            /opt/rocm/hip
+            DOC "HIP installed location"
+            NO_DEFAULT_PATH
+            )
+        if(NOT EXISTS ${HIP_ROOT_DIR})
+            if(HIP_FIND_REQUIRED)
+                message(FATAL_ERROR "Specify HIP_ROOT_DIR")
+            elseif(NOT HIP_FIND_QUIETLY)
+                message("HIP_ROOT_DIR not found or specified")
+            endif()
+        endif()
+        # And push it back to the cache
+        set(HIP_ROOT_DIR ${HIP_ROOT_DIR} CACHE PATH "HIP installed location" FORCE)
+    endif()
+
+    # Find HIPCC executable
+    find_program(
+        HIP_HIPCC_EXECUTABLE
+        NAMES hipcc
+        PATHS
+        "${HIP_ROOT_DIR}"
+        ENV ROCM_PATH
+        ENV HIP_PATH
+        /opt/rocm
+        /opt/rocm/hip
+        PATH_SUFFIXES bin
+        NO_DEFAULT_PATH
+        )
+    if(NOT HIP_HIPCC_EXECUTABLE)
+        # Now search in default paths
+        find_program(HIP_HIPCC_EXECUTABLE hipcc)
+    endif()
+    mark_as_advanced(HIP_HIPCC_EXECUTABLE)
+
+    # Find HIPCONFIG executable
+    find_program(
+        HIP_HIPCONFIG_EXECUTABLE
+        NAMES hipconfig
+        PATHS
+        "${HIP_ROOT_DIR}"
+        ENV ROCM_PATH
+        ENV HIP_PATH
+        /opt/rocm
+        /opt/rocm/hip
+        PATH_SUFFIXES bin
+        NO_DEFAULT_PATH
+        )
+    if(NOT HIP_HIPCONFIG_EXECUTABLE)
+        # Now search in default paths
+        find_program(HIP_HIPCONFIG_EXECUTABLE hipconfig)
+    endif()
+    mark_as_advanced(HIP_HIPCONFIG_EXECUTABLE)
+
+    # Find HIPCC_CMAKE_LINKER_HELPER executable
+    find_program(
+        HIP_HIPCC_CMAKE_LINKER_HELPER
+        NAMES hipcc_cmake_linker_helper
+        PATHS
+        "${HIP_ROOT_DIR}"
+        ENV ROCM_PATH
+        ENV HIP_PATH
+        /opt/rocm
+        /opt/rocm/hip
+        PATH_SUFFIXES bin
+        NO_DEFAULT_PATH
+        )
+    if(NOT HIP_HIPCC_CMAKE_LINKER_HELPER)
+        # Now search in default paths
+        find_program(HIP_HIPCC_CMAKE_LINKER_HELPER hipcc_cmake_linker_helper)
+    endif()
+    mark_as_advanced(HIP_HIPCC_CMAKE_LINKER_HELPER)
+
+    if(HIP_HIPCONFIG_EXECUTABLE AND NOT HIP_VERSION)
+        # Compute the version
+        execute_process(
+            COMMAND ${HIP_HIPCONFIG_EXECUTABLE} --version
+            OUTPUT_VARIABLE _hip_version
+            ERROR_VARIABLE _hip_error
+            OUTPUT_STRIP_TRAILING_WHITESPACE
+            ERROR_STRIP_TRAILING_WHITESPACE
+            )
+        if(NOT _hip_error)
+            set(HIP_VERSION ${_hip_version} CACHE STRING "Version of HIP as computed from hipcc")
+        else()
+            set(HIP_VERSION "0.0.0" CACHE STRING "Version of HIP as computed by FindHIP()")
+        endif()
+        mark_as_advanced(HIP_VERSION)
+    endif()
+    if(HIP_VERSION)
+        string(REPLACE "." ";" _hip_version_list "${HIP_VERSION}")
+        list(GET _hip_version_list 0 HIP_VERSION_MAJOR)
+        list(GET _hip_version_list 1 HIP_VERSION_MINOR)
+        list(GET _hip_version_list 2 HIP_VERSION_PATCH)
+        set(HIP_VERSION_STRING "${HIP_VERSION}")
+    endif()
+
+    if(HIP_HIPCONFIG_EXECUTABLE AND NOT HIP_PLATFORM)
+        # Compute the platform
+        execute_process(
+            COMMAND ${HIP_HIPCONFIG_EXECUTABLE} --platform
+            OUTPUT_VARIABLE _hip_platform
+            OUTPUT_STRIP_TRAILING_WHITESPACE
+            )
+        set(HIP_PLATFORM ${_hip_platform} CACHE STRING "HIP platform as computed by hipconfig")
+        mark_as_advanced(HIP_PLATFORM)
+    endif()
+
+    if(HIP_HIPCONFIG_EXECUTABLE AND NOT HIP_COMPILER)
+        # Compute the compiler
+        execute_process(
+            COMMAND ${HIP_HIPCONFIG_EXECUTABLE} --compiler
+            OUTPUT_VARIABLE _hip_compiler
+            OUTPUT_STRIP_TRAILING_WHITESPACE
+            )
+        set(HIP_COMPILER ${_hip_compiler} CACHE STRING "HIP compiler as computed by hipconfig")
+        mark_as_advanced(HIP_COMPILER)
+    endif()
+
+    if(HIP_HIPCONFIG_EXECUTABLE AND NOT HIP_RUNTIME)
+        # Compute the runtime
+        execute_process(
+            COMMAND ${HIP_HIPCONFIG_EXECUTABLE} --runtime
+            OUTPUT_VARIABLE _hip_runtime
+            OUTPUT_STRIP_TRAILING_WHITESPACE
+            )
+        set(HIP_RUNTIME ${_hip_runtime} CACHE STRING "HIP runtime as computed by hipconfig")
+        mark_as_advanced(HIP_RUNTIME)
+    endif()
+endif()
+
+include(FindPackageHandleStandardArgs)
+find_package_handle_standard_args(
+    HIP
+    REQUIRED_VARS
+    HIP_ROOT_DIR
+    HIP_HIPCC_EXECUTABLE
+    HIP_HIPCONFIG_EXECUTABLE
+    HIP_PLATFORM
+    HIP_COMPILER
+    HIP_RUNTIME
+    VERSION_VAR HIP_VERSION
+    )
+
+###############################################################################
+# Set HIP CMAKE Flags
+###############################################################################
+# Copy the invocation styles from CXX to HIP
+set(CMAKE_HIP_ARCHIVE_CREATE ${CMAKE_CXX_ARCHIVE_CREATE})
+set(CMAKE_HIP_ARCHIVE_APPEND ${CMAKE_CXX_ARCHIVE_APPEND})
+set(CMAKE_HIP_ARCHIVE_FINISH ${CMAKE_CXX_ARCHIVE_FINISH})
+set(CMAKE_SHARED_LIBRARY_SONAME_HIP_FLAG ${CMAKE_SHARED_LIBRARY_SONAME_CXX_FLAG})
+set(CMAKE_SHARED_LIBRARY_CREATE_HIP_FLAGS ${CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS})
+set(CMAKE_SHARED_LIBRARY_HIP_FLAGS ${CMAKE_SHARED_LIBRARY_CXX_FLAGS})
+#set(CMAKE_SHARED_LIBRARY_LINK_HIP_FLAGS ${CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS})
+set(CMAKE_SHARED_LIBRARY_RUNTIME_HIP_FLAG ${CMAKE_SHARED_LIBRARY_RUNTIME_CXX_FLAG})
+set(CMAKE_SHARED_LIBRARY_RUNTIME_HIP_FLAG_SEP ${CMAKE_SHARED_LIBRARY_RUNTIME_CXX_FLAG_SEP})
+set(CMAKE_SHARED_LIBRARY_LINK_STATIC_HIP_FLAGS ${CMAKE_SHARED_LIBRARY_LINK_STATIC_CXX_FLAGS})
+set(CMAKE_SHARED_LIBRARY_LINK_DYNAMIC_HIP_FLAGS ${CMAKE_SHARED_LIBRARY_LINK_DYNAMIC_CXX_FLAGS})
+
+set(HIP_CLANG_PARALLEL_BUILD_COMPILE_OPTIONS "")
+set(HIP_CLANG_PARALLEL_BUILD_LINK_OPTIONS "")
+
+if("${HIP_COMPILER}" STREQUAL "nvcc")
+    # Set the CMake Flags to use the nvcc Compiler.
+    set(CMAKE_HIP_CREATE_SHARED_LIBRARY "${HIP_HIPCC_CMAKE_LINKER_HELPER} <CMAKE_SHARED_LIBRARY_CXX_FLAGS> <LANGUAGE_COMPILE_FLAGS> <LINK_FLAGS> <CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS> <SONAME_FLAG><TARGET_SONAME> -o <TARGET> <OBJECTS> <LINK_LIBRARIES>")
+    set(CMAKE_HIP_CREATE_SHARED_MODULE "${HIP_HIPCC_CMAKE_LINKER_HELPER} <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> <SONAME_FLAG><TARGET_SONAME> -o <TARGET> <LINK_LIBRARIES> -shared" )
+    set(CMAKE_HIP_LINK_EXECUTABLE "${HIP_HIPCC_CMAKE_LINKER_HELPER} <FLAGS> <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>")
+elseif("${HIP_COMPILER}" STREQUAL "hcc")
+    # Set the CMake Flags to use the hcc Compiler.
+    set(CMAKE_HIP_CREATE_SHARED_LIBRARY "${HIP_HIPCC_CMAKE_LINKER_HELPER} ${HCC_HOME} <CMAKE_SHARED_LIBRARY_CXX_FLAGS> <LANGUAGE_COMPILE_FLAGS> <LINK_FLAGS> <CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS> <SONAME_FLAG><TARGET_SONAME> -o <TARGET> <OBJECTS> <LINK_LIBRARIES>")
+    set(CMAKE_HIP_CREATE_SHARED_MODULE "${HIP_HIPCC_CMAKE_LINKER_HELPER} ${HCC_HOME} <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> <SONAME_FLAG><TARGET_SONAME> -o <TARGET> <LINK_LIBRARIES> -shared" )
+    set(CMAKE_HIP_LINK_EXECUTABLE "${HIP_HIPCC_CMAKE_LINKER_HELPER} ${HCC_HOME} <FLAGS> <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>")
+elseif("${HIP_COMPILER}" STREQUAL "clang")
+    #Number of parallel jobs by default is 1
+    if(NOT DEFINED HIP_CLANG_NUM_PARALLEL_JOBS)
+      set(HIP_CLANG_NUM_PARALLEL_JOBS 1)
+    endif()
+    #Add support for parallel build and link
+    if(${CMAKE_CXX_COMPILER_ID} STREQUAL "Clang")
+      check_cxx_compiler_flag("-parallel-jobs=1" HIP_CLANG_SUPPORTS_PARALLEL_JOBS)
+    endif()
+    if(HIP_CLANG_NUM_PARALLEL_JOBS GREATER 1)
+      if(${HIP_CLANG_SUPPORTS_PARALLEL_JOBS})
+        set(HIP_CLANG_PARALLEL_BUILD_COMPILE_OPTIONS "-Wno-format-nonliteral -parallel-jobs=${HIP_CLANG_NUM_PARALLEL_JOBS}")
+        set(HIP_CLANG_PARALLEL_BUILD_LINK_OPTIONS "-parallel-jobs=${HIP_CLANG_NUM_PARALLEL_JOBS}")
+      else()
+        message("clang compiler doesn't support parallel jobs")
+      endif()
+    endif()
+
+    # Set the CMake Flags to use the HIP-Clang Compiler.
+    set(CMAKE_HIP_CREATE_SHARED_LIBRARY "${HIP_HIPCC_CMAKE_LINKER_HELPER} ${HIP_CLANG_PATH} ${HIP_CLANG_PARALLEL_BUILD_LINK_OPTIONS} <CMAKE_SHARED_LIBRARY_CXX_FLAGS> <LANGUAGE_COMPILE_FLAGS> <LINK_FLAGS> <CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS> <SONAME_FLAG><TARGET_SONAME> -o <TARGET> <OBJECTS> <LINK_LIBRARIES>")
+    set(CMAKE_HIP_CREATE_SHARED_MODULE "${HIP_HIPCC_CMAKE_LINKER_HELPER} ${HIP_CLANG_PATH} ${HIP_CLANG_PARALLEL_BUILD_LINK_OPTIONS} <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> <SONAME_FLAG><TARGET_SONAME> -o <TARGET> <LINK_LIBRARIES> -shared" )
+    set(CMAKE_HIP_LINK_EXECUTABLE "${HIP_HIPCC_CMAKE_LINKER_HELPER} ${HIP_CLANG_PATH} ${HIP_CLANG_PARALLEL_BUILD_LINK_OPTIONS} <FLAGS> <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>")
+
+    if("${HIP_RUNTIME}" STREQUAL "rocclr")
+      if(TARGET host)
+        message(STATUS "host interface - found")
+        set(HIP_HOST_INTERFACE host)
+      endif()
+    endif()
+endif()
+
+###############################################################################
+# MACRO: Locate helper files
+###############################################################################
+macro(HIP_FIND_HELPER_FILE _name _extension)
+    set(_hip_full_name "${_name}.${_extension}")
+    get_filename_component(CMAKE_CURRENT_LIST_DIR "${CMAKE_CURRENT_LIST_FILE}" PATH)
+    set(HIP_${_name} "${CMAKE_CURRENT_LIST_DIR}/FindHIP/${_hip_full_name}")
+    if(NOT EXISTS "${HIP_${_name}}")
+        set(error_message "${_hip_full_name} not found in ${CMAKE_CURRENT_LIST_DIR}/FindHIP")
+        if(HIP_FIND_REQUIRED)
+            message(FATAL_ERROR "${error_message}")
+        else()
+            if(NOT HIP_FIND_QUIETLY)
+                message(STATUS "${error_message}")
+            endif()
+        endif()
+    endif()
+    # Set this variable as internal, so the user isn't bugged with it.
+    set(HIP_${_name} ${HIP_${_name}} CACHE INTERNAL "Location of ${_full_name}" FORCE)
+endmacro()
+
+###############################################################################
+hip_find_helper_file(run_make2cmake cmake)
+hip_find_helper_file(run_hipcc cmake)
+###############################################################################
+
+###############################################################################
+# MACRO: Reset compiler flags
+###############################################################################
+macro(HIP_RESET_FLAGS)
+    unset(HIP_HIPCC_FLAGS)
+    unset(HIP_HCC_FLAGS)
+    unset(HIP_CLANG_FLAGS)
+    unset(HIP_NVCC_FLAGS)
+    foreach(config ${_hip_configuration_types})
+        string(TOUPPER ${config} config_upper)
+        unset(HIP_HIPCC_FLAGS_${config_upper})
+        unset(HIP_HCC_FLAGS_${config_upper})
+        unset(HIP_CLANG_FLAGS_${config_upper})
+        unset(HIP_NVCC_FLAGS_${config_upper})
+    endforeach()
+endmacro()
+
+###############################################################################
+# MACRO: Separate the options from the sources
+###############################################################################
+macro(HIP_GET_SOURCES_AND_OPTIONS _sources _cmake_options _hipcc_options _hcc_options _clang_options _nvcc_options)
+    set(${_sources})
+    set(${_cmake_options})
+    set(${_hipcc_options})
+    set(${_hcc_options})
+    set(${_clang_options})
+    set(${_nvcc_options})
+    set(_hipcc_found_options FALSE)
+    set(_hcc_found_options FALSE)
+    set(_clang_found_options FALSE)
+    set(_nvcc_found_options FALSE)
+    foreach(arg ${ARGN})
+        if("x${arg}" STREQUAL "xHIPCC_OPTIONS")
+            set(_hipcc_found_options TRUE)
+            set(_hcc_found_options FALSE)
+            set(_clang_found_options FALSE)
+            set(_nvcc_found_options FALSE)
+        elseif("x${arg}" STREQUAL "xHCC_OPTIONS")
+            set(_hipcc_found_options FALSE)
+            set(_hcc_found_options TRUE)
+            set(_clang_found_options FALSE)
+            set(_nvcc_found_options FALSE)
+        elseif("x${arg}" STREQUAL "xCLANG_OPTIONS")
+            set(_hipcc_found_options FALSE)
+            set(_hcc_found_options FALSE)
+            set(_clang_found_options TRUE)
+            set(_nvcc_found_options FALSE)
+        elseif("x${arg}" STREQUAL "xNVCC_OPTIONS")
+            set(_hipcc_found_options FALSE)
+            set(_hcc_found_options FALSE)
+            set(_clang_found_options FALSE)
+            set(_nvcc_found_options TRUE)
+        elseif(
+                "x${arg}" STREQUAL "xEXCLUDE_FROM_ALL" OR
+                "x${arg}" STREQUAL "xSTATIC" OR
+                "x${arg}" STREQUAL "xSHARED" OR
+                "x${arg}" STREQUAL "xMODULE"
+                )
+            list(APPEND ${_cmake_options} ${arg})
+        else()
+            if(_hipcc_found_options)
+                list(APPEND ${_hipcc_options} ${arg})
+            elseif(_hcc_found_options)
+                list(APPEND ${_hcc_options} ${arg})
+            elseif(_clang_found_options)
+                list(APPEND ${_clang_options} ${arg})
+            elseif(_nvcc_found_options)
+                list(APPEND ${_nvcc_options} ${arg})
+            else()
+                # Assume this is a file
+                list(APPEND ${_sources} ${arg})
+            endif()
+        endif()
+    endforeach()
+endmacro()
+
+###############################################################################
+# MACRO: Add include directories to pass to the hipcc command
+###############################################################################
+set(HIP_HIPCC_INCLUDE_ARGS_USER "")
+macro(HIP_INCLUDE_DIRECTORIES)
+    foreach(dir ${ARGN})
+        list(APPEND HIP_HIPCC_INCLUDE_ARGS_USER $<$<BOOL:${dir}>:-I${dir}>)
+    endforeach()
+endmacro()
+
+###############################################################################
+# FUNCTION: Helper to avoid clashes of files with the same basename but different paths
+###############################################################################
+function(HIP_COMPUTE_BUILD_PATH path build_path)
+    # Convert to cmake style paths
+    file(TO_CMAKE_PATH "${path}" bpath)
+    if(IS_ABSOLUTE "${bpath}")
+        string(FIND "${bpath}" "${CMAKE_CURRENT_BINARY_DIR}" _binary_dir_pos)
+        if(_binary_dir_pos EQUAL 0)
+            file(RELATIVE_PATH bpath "${CMAKE_CURRENT_BINARY_DIR}" "${bpath}")
+        else()
+            file(RELATIVE_PATH bpath "${CMAKE_CURRENT_SOURCE_DIR}" "${bpath}")
+        endif()
+    endif()
+
+    # Remove leading /
+    string(REGEX REPLACE "^[/]+" "" bpath "${bpath}")
+    # Avoid absolute paths by removing ':'
+    string(REPLACE ":" "_" bpath "${bpath}")
+    # Avoid relative paths that go up the tree
+    string(REPLACE "../" "__/" bpath "${bpath}")
+    # Avoid spaces
+    string(REPLACE " " "_" bpath "${bpath}")
+    # Strip off the filename
+    get_filename_component(bpath "${bpath}" PATH)
+
+    set(${build_path} "${bpath}" PARENT_SCOPE)
+endfunction()
+
+###############################################################################
+# MACRO: Parse OPTIONS from ARGN & set variables prefixed by _option_prefix
+###############################################################################
+macro(HIP_PARSE_HIPCC_OPTIONS _option_prefix)
+    set(_hip_found_config)
+    foreach(arg ${ARGN})
+        # Determine if we are dealing with a per-configuration flag
+        foreach(config ${_hip_configuration_types})
+            string(TOUPPER ${config} config_upper)
+            if(arg STREQUAL "${config_upper}")
+                set(_hip_found_config _${arg})
+                # Clear arg to prevent it from being processed anymore
+                set(arg)
+            endif()
+        endforeach()
+        if(arg)
+            list(APPEND ${_option_prefix}${_hip_found_config} "${arg}")
+        endif()
+    endforeach()
+endmacro()
+
+###############################################################################
+# MACRO: Try and include dependency file if it exists
+###############################################################################
+macro(HIP_INCLUDE_HIPCC_DEPENDENCIES dependency_file)
+    set(HIP_HIPCC_DEPEND)
+    set(HIP_HIPCC_DEPEND_REGENERATE FALSE)
+
+    # Create the dependency file if it doesn't exist
+    if(NOT EXISTS ${dependency_file})
+        file(WRITE ${dependency_file} "# Generated by: FindHIP.cmake. Do not edit.\n")
+    endif()
+    # Include the dependency file
+    include(${dependency_file})
+
+    # Verify the existence of all the included files
+    if(HIP_HIPCC_DEPEND)
+        foreach(f ${HIP_HIPCC_DEPEND})
+            if(NOT EXISTS ${f})
+                # If they aren't there, regenerate the file again
+                set(HIP_HIPCC_DEPEND_REGENERATE TRUE)
+            endif()
+        endforeach()
+    else()
+        # No dependencies, so regenerate the file
+        set(HIP_HIPCC_DEPEND_REGENERATE TRUE)
+    endif()
+
+    # Regenerate the dependency file if needed
+    if(HIP_HIPCC_DEPEND_REGENERATE)
+        set(HIP_HIPCC_DEPEND ${dependency_file})
+        file(WRITE ${dependency_file} "# Generated by: FindHIP.cmake. Do not edit.\n")
+    endif()
+endmacro()
+
+###############################################################################
+# MACRO: Prepare cmake commands for the target
+###############################################################################
+macro(HIP_PREPARE_TARGET_COMMANDS _target _format _generated_files _source_files)
+    set(_hip_flags "")
+    string(TOUPPER "${CMAKE_BUILD_TYPE}" _hip_build_configuration)
+    if(HIP_HOST_COMPILATION_CPP)
+        set(HIP_C_OR_CXX CXX)
+    else()
+        set(HIP_C_OR_CXX C)
+    endif()
+    set(generated_extension ${CMAKE_${HIP_C_OR_CXX}_OUTPUT_EXTENSION})
+
+    # Initialize list of includes with those specified by the user. Append with
+    # ones specified to cmake directly.
+    set(HIP_HIPCC_INCLUDE_ARGS ${HIP_HIPCC_INCLUDE_ARGS_USER})
+
+    # Add the include directories
+    set(include_directories_generator "$<TARGET_PROPERTY:${_target},INCLUDE_DIRECTORIES>")
+    list(APPEND HIP_HIPCC_INCLUDE_ARGS "$<$<BOOL:${include_directories_generator}>:-I$<JOIN:${include_directories_generator}, -I>>")
+
+    get_directory_property(_hip_include_directories INCLUDE_DIRECTORIES)
+    list(REMOVE_DUPLICATES _hip_include_directories)
+    if(_hip_include_directories)
+        foreach(dir ${_hip_include_directories})
+            list(APPEND HIP_HIPCC_INCLUDE_ARGS $<$<BOOL:${dir}>:-I${dir}>)
+        endforeach()
+    endif()
+
+    HIP_GET_SOURCES_AND_OPTIONS(_hip_sources _hip_cmake_options _hipcc_options _hcc_options _clang_options _nvcc_options ${ARGN})
+    HIP_PARSE_HIPCC_OPTIONS(HIP_HIPCC_FLAGS ${_hipcc_options})
+    HIP_PARSE_HIPCC_OPTIONS(HIP_HCC_FLAGS ${_hcc_options})
+    HIP_PARSE_HIPCC_OPTIONS(HIP_CLANG_FLAGS ${_clang_options})
+    HIP_PARSE_HIPCC_OPTIONS(HIP_NVCC_FLAGS ${_nvcc_options})
+
+    # Add the compile definitions
+    set(compile_definition_generator "$<TARGET_PROPERTY:${_target},COMPILE_DEFINITIONS>")
+    list(APPEND HIP_HIPCC_FLAGS "$<$<BOOL:${compile_definition_generator}>:-D$<JOIN:${compile_definition_generator}, -D>>")
+
+    # Check if we are building shared library.
+    set(_hip_build_shared_libs FALSE)
+    list(FIND _hip_cmake_options SHARED _hip_found_SHARED)
+    list(FIND _hip_cmake_options MODULE _hip_found_MODULE)
+    if(_hip_found_SHARED GREATER -1 OR _hip_found_MODULE GREATER -1)
+        set(_hip_build_shared_libs TRUE)
+    endif()
+    list(FIND _hip_cmake_options STATIC _hip_found_STATIC)
+    if(_hip_found_STATIC GREATER -1)
+        set(_hip_build_shared_libs FALSE)
+    endif()
+
+    # If we are building a shared library, add extra flags to HIP_HIPCC_FLAGS
+    if(_hip_build_shared_libs)
+        list(APPEND HIP_HCC_FLAGS "-fPIC")
+        list(APPEND HIP_CLANG_FLAGS "-fPIC")
+        list(APPEND HIP_NVCC_FLAGS "--shared -Xcompiler '-fPIC'")
+    endif()
+
+    # Set host compiler
+    set(HIP_HOST_COMPILER "${CMAKE_${HIP_C_OR_CXX}_COMPILER}")
+
+    # Set compiler flags
+    set(_HIP_HOST_FLAGS "set(CMAKE_HOST_FLAGS ${CMAKE_${HIP_C_OR_CXX}_FLAGS})")
+    set(_HIP_HIPCC_FLAGS "set(HIP_HIPCC_FLAGS ${HIP_HIPCC_FLAGS})")
+    set(_HIP_HCC_FLAGS "set(HIP_HCC_FLAGS ${HIP_HCC_FLAGS})")
+    set(_HIP_CLANG_FLAGS "set(HIP_CLANG_FLAGS ${HIP_CLANG_FLAGS})")
+    set(_HIP_NVCC_FLAGS "set(HIP_NVCC_FLAGS ${HIP_NVCC_FLAGS})")
+    foreach(config ${_hip_configuration_types})
+        string(TOUPPER ${config} config_upper)
+        set(_HIP_HOST_FLAGS "${_HIP_HOST_FLAGS}\nset(CMAKE_HOST_FLAGS_${config_upper} ${CMAKE_${HIP_C_OR_CXX}_FLAGS_${config_upper}})")
+        set(_HIP_HIPCC_FLAGS "${_HIP_HIPCC_FLAGS}\nset(HIP_HIPCC_FLAGS_${config_upper} ${HIP_HIPCC_FLAGS_${config_upper}})")
+        set(_HIP_HCC_FLAGS "${_HIP_HCC_FLAGS}\nset(HIP_HCC_FLAGS_${config_upper} ${HIP_HCC_FLAGS_${config_upper}})")
+        set(_HIP_CLANG_FLAGS "${_HIP_CLANG_FLAGS}\nset(HIP_CLANG_FLAGS_${config_upper} ${HIP_CLANG_FLAGS_${config_upper}})")
+        set(_HIP_NVCC_FLAGS "${_HIP_NVCC_FLAGS}\nset(HIP_NVCC_FLAGS_${config_upper} ${HIP_NVCC_FLAGS_${config_upper}})")
+    endforeach()
+
+    # Reset the output variable
+    set(_hip_generated_files "")
+    set(_hip_source_files "")
+
+    # Iterate over all arguments and create custom commands for all source files
+    foreach(file ${ARGN})
+        # Ignore any file marked as a HEADER_FILE_ONLY
+        get_source_file_property(_is_header ${file} HEADER_FILE_ONLY)
+        # Allow per source file overrides of the format. Also allows compiling non .cu files.
+        get_source_file_property(_hip_source_format ${file} HIP_SOURCE_PROPERTY_FORMAT)
+        if((${file} MATCHES "\\.cu$" OR _hip_source_format) AND NOT _is_header)
+            set(host_flag FALSE)
+        else()
+            set(host_flag TRUE)
+        endif()
+
+        if(NOT host_flag)
+            # Determine output directory
+            HIP_COMPUTE_BUILD_PATH("${file}" hip_build_path)
+            set(hip_compile_output_dir "${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${_target}.dir/${hip_build_path}")
+
+            get_filename_component(basename ${file} NAME)
+            set(generated_file_path "${hip_compile_output_dir}/${CMAKE_CFG_INTDIR}")
+            set(generated_file_basename "${_target}_generated_${basename}${generated_extension}")
+
+            # Set file names
+            set(generated_file "${generated_file_path}/${generated_file_basename}")
+            set(cmake_dependency_file "${hip_compile_output_dir}/${generated_file_basename}.depend")
+            set(custom_target_script_pregen "${hip_compile_output_dir}/${generated_file_basename}.cmake.pre-gen")
+            set(custom_target_script "${hip_compile_output_dir}/${generated_file_basename}.cmake")
+
+            # Set properties for object files
+            set_source_files_properties("${generated_file}"
+                PROPERTIES
+                EXTERNAL_OBJECT true # This is an object file not to be compiled, but only be linked
+                )
+
+            # Don't add CMAKE_CURRENT_SOURCE_DIR if the path is already an absolute path
+            get_filename_component(file_path "${file}" PATH)
+            if(IS_ABSOLUTE "${file_path}")
+                set(source_file "${file}")
+            else()
+                set(source_file "${CMAKE_CURRENT_SOURCE_DIR}/${file}")
+            endif()
+
+            # Bring in the dependencies
+            HIP_INCLUDE_HIPCC_DEPENDENCIES(${cmake_dependency_file})
+
+            # Configure the build script
+            configure_file("${HIP_run_hipcc}" "${custom_target_script_pregen}" @ONLY)
+            file(GENERATE
+                OUTPUT "${custom_target_script}"
+                INPUT "${custom_target_script_pregen}"
+                )
+            set(main_dep DEPENDS ${source_file})
+            if(CMAKE_GENERATOR MATCHES "Makefiles")
+                set(verbose_output "$(VERBOSE)")
+            elseif(HIP_VERBOSE_BUILD)
+                set(verbose_output ON)
+            else()
+                set(verbose_output OFF)
+            endif()
+
+            # Create up the comment string
+            file(RELATIVE_PATH generated_file_relative_path "${CMAKE_BINARY_DIR}" "${generated_file}")
+            set(hip_build_comment_string "Building HIPCC object ${generated_file_relative_path}")
+
+            # Build the generated file and dependency file
+            add_custom_command(
+                OUTPUT ${generated_file}
+                # These output files depend on the source_file and the contents of cmake_dependency_file
+                ${main_dep}
+                DEPENDS ${HIP_HIPCC_DEPEND}
+                DEPENDS ${custom_target_script}
+                # Make sure the output directory exists before trying to write to it.
+                COMMAND ${CMAKE_COMMAND} -E make_directory "${generated_file_path}"
+                COMMAND ${CMAKE_COMMAND} ARGS
+                -D verbose:BOOL=${verbose_output}
+                -D build_configuration:STRING=${_hip_build_configuration}
+                -D "generated_file:STRING=${generated_file}"
+                -P "${custom_target_script}"
+                WORKING_DIRECTORY "${hip_compile_output_dir}"
+                COMMENT "${hip_build_comment_string}"
+                )
+
+            # Make sure the build system knows the file is generated
+            set_source_files_properties(${generated_file} PROPERTIES GENERATED TRUE)
+            list(APPEND _hip_generated_files ${generated_file})
+            list(APPEND _hip_source_files ${file})
+        endif()
+    endforeach()
+
+    # Set the return parameter
+    set(${_generated_files} ${_hip_generated_files})
+    set(${_source_files} ${_hip_source_files})
+endmacro()
+
+###############################################################################
+# HIP_ADD_EXECUTABLE
+###############################################################################
+macro(HIP_ADD_EXECUTABLE hip_target)
+    # Separate the sources from the options
+    HIP_GET_SOURCES_AND_OPTIONS(_sources _cmake_options _hipcc_options _hcc_options _clang_options _nvcc_options ${ARGN})
+    HIP_PREPARE_TARGET_COMMANDS(${hip_target} OBJ _generated_files _source_files ${_sources} HIPCC_OPTIONS ${_hipcc_options} HCC_OPTIONS ${_hcc_options} CLANG_OPTIONS ${_clang_options} NVCC_OPTIONS ${_nvcc_options})
+    if(_source_files)
+        list(REMOVE_ITEM _sources ${_source_files})
+    endif()
+    if("${HIP_COMPILER}" STREQUAL "hcc")
+        if("x${HCC_HOME}" STREQUAL "x")
+            if (DEFINED ENV{ROCM_PATH})
+                set(HCC_HOME "$ENV{ROCM_PATH}/hcc")
+            elseif(DEFINED ENV{HIP_PATH})
+                set(HCC_HOME "$ENV{HIP_PATH}/../hcc")
+            else()
+                set(HCC_HOME "/opt/rocm/hcc")
+            endif()
+        endif()
+        set(CMAKE_HIP_LINK_EXECUTABLE "${HIP_HIPCC_CMAKE_LINKER_HELPER} ${HCC_HOME} <FLAGS> <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>")
+    elseif("${HIP_COMPILER}" STREQUAL "clang")
+        if("x${HIP_CLANG_PATH}" STREQUAL "x")
+            if(DEFINED ENV{HIP_CLANG_PATH})
+                set(HIP_CLANG_PATH $ENV{HIP_CLANG_PATH})
+            elseif(DEFINED ENV{ROCM_PATH})
+                set(HIP_CLANG_PATH "$ENV{ROCM_PATH}/llvm/bin")
+            elseif(DEFINED ENV{HIP_PATH})
+                set(HIP_CLANG_PATH "$ENV{HIP_PATH}/../llvm/bin")
+            else()
+                set(HIP_CLANG_PATH "/opt/rocm/llvm/bin")
+            endif()
+        endif()
+        set(CMAKE_HIP_LINK_EXECUTABLE "${HIP_HIPCC_CMAKE_LINKER_HELPER} ${HIP_CLANG_PATH} ${HIP_CLANG_PARALLEL_BUILD_LINK_OPTIONS} <FLAGS> <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>")
+    else()
+        set(CMAKE_HIP_LINK_EXECUTABLE "${HIP_HIPCC_CMAKE_LINKER_HELPER} <FLAGS> <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>")
+    endif()
+    if ("${_sources}" STREQUAL "")
+        add_executable(${hip_target} ${_cmake_options} ${_generated_files} "")
+    else()
+        add_executable(${hip_target} ${_cmake_options} ${_generated_files} ${_sources})
+    endif()
+    set_target_properties(${hip_target} PROPERTIES LINKER_LANGUAGE HIP)
+    # Link with host
+    if (HIP_HOST_INTERFACE)
+        # hip rt should be rocclr, compiler should be clang
+        target_link_libraries(${hip_target} ${HIP_HOST_INTERFACE})
+    endif()
+endmacro()
+
+###############################################################################
+# HIP_ADD_LIBRARY
+###############################################################################
+macro(HIP_ADD_LIBRARY hip_target)
+    # Separate the sources from the options
+    HIP_GET_SOURCES_AND_OPTIONS(_sources _cmake_options _hipcc_options _hcc_options _clang_options _nvcc_options ${ARGN})
+    HIP_PREPARE_TARGET_COMMANDS(${hip_target} OBJ _generated_files _source_files ${_sources} ${_cmake_options} HIPCC_OPTIONS ${_hipcc_options} HCC_OPTIONS ${_hcc_options} CLANG_OPTIONS ${_clang_options} NVCC_OPTIONS ${_nvcc_options})
+    if(_source_files)
+        list(REMOVE_ITEM _sources ${_source_files})
+    endif()
+    if ("${_sources}" STREQUAL "")
+        add_library(${hip_target} ${_cmake_options} ${_generated_files} "")
+    else()
+        add_library(${hip_target} ${_cmake_options} ${_generated_files} ${_sources})
+    endif()
+    set_target_properties(${hip_target} PROPERTIES LINKER_LANGUAGE ${HIP_C_OR_CXX})
+    # Link with host
+    if (HIP_HOST_INTERFACE)
+        # hip rt should be rocclr, compiler should be clang
+        target_link_libraries(${hip_target} ${HIP_HOST_INTERFACE})
+    endif()
+endmacro()
+
+# vim: ts=4:sw=4:expandtab:smartindent
@@ -0,0 +1,182 @@
+###############################################################################
+# Runs commands using HIPCC
+###############################################################################
+
+###############################################################################
+# This file runs the hipcc commands to produce the desired output file
+# along with the dependency file needed by CMake to compute dependencies.
+#
+# Input variables:
+#
+# verbose:BOOL=<>               OFF: Be as quiet as possible (default)
+#                               ON : Describe each step
+# build_configuration:STRING=<> Build configuration. Defaults to Debug.
+# generated_file:STRING=<>      File to generate. Mandatory argument.
+
+if(NOT build_configuration)
+    set(build_configuration Debug)
+endif()
+if(NOT generated_file)
+    message(FATAL_ERROR "You must specify generated_file on the command line")
+endif()
+
+# Set these up as variables to make reading the generated file easier
+set(HIP_HIPCC_EXECUTABLE "@HIP_HIPCC_EXECUTABLE@") # path
+set(HIP_HIPCONFIG_EXECUTABLE "@HIP_HIPCONFIG_EXECUTABLE@") #path
+set(HIP_HOST_COMPILER "@HIP_HOST_COMPILER@") # path
+set(CMAKE_COMMAND "@CMAKE_COMMAND@") # path
+set(HIP_run_make2cmake "@HIP_run_make2cmake@") # path
+set(HCC_HOME "@HCC_HOME@") #path
+set(HIP_CLANG_PATH "@HIP_CLANG_PATH@") #path
+set(HIP_CLANG_PARALLEL_BUILD_COMPILE_OPTIONS "@HIP_CLANG_PARALLEL_BUILD_COMPILE_OPTIONS@")
+
+@HIP_HOST_FLAGS@
+@_HIP_HIPCC_FLAGS@
+@_HIP_HCC_FLAGS@
+@_HIP_CLANG_FLAGS@
+@_HIP_NVCC_FLAGS@
+#Needed to bring the HIP_HIPCC_INCLUDE_ARGS variable in scope
+set(HIP_HIPCC_INCLUDE_ARGS @HIP_HIPCC_INCLUDE_ARGS@) # list
+
+set(cmake_dependency_file "@cmake_dependency_file@") # path
+set(source_file "@source_file@") # path
+set(host_flag "@host_flag@") # bool
+
+# Determine compiler and compiler flags
+execute_process(COMMAND ${HIP_HIPCONFIG_EXECUTABLE} --platform OUTPUT_VARIABLE HIP_PLATFORM OUTPUT_STRIP_TRAILING_WHITESPACE)
+execute_process(COMMAND ${HIP_HIPCONFIG_EXECUTABLE} --compiler OUTPUT_VARIABLE HIP_COMPILER OUTPUT_STRIP_TRAILING_WHITESPACE)
+execute_process(COMMAND ${HIP_HIPCONFIG_EXECUTABLE} --runtime OUTPUT_VARIABLE HIP_RUNTIME OUTPUT_STRIP_TRAILING_WHITESPACE)
+if(NOT host_flag)
+    set(__CC ${HIP_HIPCC_EXECUTABLE})
+    if("${HIP_PLATFORM}" STREQUAL "amd")
+        if("${HIP_COMPILER}" STREQUAL "hcc")
+            if(NOT "x${HCC_HOME}" STREQUAL "x")
+                set(ENV{HCC_HOME} ${HCC_HOME})
+            endif()
+            set(__CC_FLAGS ${HIP_HIPCC_FLAGS} ${HIP_HCC_FLAGS} ${HIP_HIPCC_FLAGS_${build_configuration}} ${HIP_HCC_FLAGS_${build_configuration}})
+        elseif("${HIP_COMPILER}" STREQUAL "clang")
+            if(NOT "x${HIP_CLANG_PATH}" STREQUAL "x")
+                set(ENV{HIP_CLANG_PATH} ${HIP_CLANG_PATH})
+            endif()
+            # Temporarily include HIP_HCC_FLAGS for HIP-Clang for PyTorch builds
+            set(__CC_FLAGS ${HIP_CLANG_PARALLEL_BUILD_COMPILE_OPTIONS} ${HIP_HIPCC_FLAGS} ${HIP_HCC_FLAGS} ${HIP_CLANG_FLAGS} ${HIP_HIPCC_FLAGS_${build_configuration}} ${HIP_HCC_FLAGS_${build_configuration}} ${HIP_CLANG_FLAGS_${build_configuration}})
+        endif()
+    else()
+        set(__CC_FLAGS ${HIP_HIPCC_FLAGS} ${HIP_NVCC_FLAGS} ${HIP_HIPCC_FLAGS_${build_configuration}} ${HIP_NVCC_FLAGS_${build_configuration}})
+    endif()
+else()
+    set(__CC ${HIP_HOST_COMPILER})
+    set(__CC_FLAGS ${CMAKE_HOST_FLAGS} ${CMAKE_HOST_FLAGS_${build_configuration}})
+endif()
+set(__CC_INCLUDES ${HIP_HIPCC_INCLUDE_ARGS})
+
+# hip_execute_process - Executes a command with optional command echo and status message.
+#   status     - Status message to print if verbose is true
+#   command    - COMMAND argument from the usual execute_process argument structure
+#   ARGN       - Remaining arguments are the command with arguments
+#   HIP_result - Return value from running the command
+macro(hip_execute_process status command)
+    set(_command ${command})
+    if(NOT "x${_command}" STREQUAL "xCOMMAND")
+        message(FATAL_ERROR "Malformed call to hip_execute_process.  Missing COMMAND as second argument. (command = ${command})")
+    endif()
+    if(verbose)
+        execute_process(COMMAND "${CMAKE_COMMAND}" -E echo -- ${status})
+        # Build command string to print
+        set(hip_execute_process_string)
+        foreach(arg ${ARGN})
+            # Escape quotes if any
+            string(REPLACE "\"" "\\\"" arg ${arg})
+            # Surround args with spaces with quotes
+            if(arg MATCHES " ")
+                list(APPEND hip_execute_process_string "\"${arg}\"")
+            else()
+                list(APPEND hip_execute_process_string ${arg})
+            endif()
+        endforeach()
+        # Echo the command
+        execute_process(COMMAND ${CMAKE_COMMAND} -E echo ${hip_execute_process_string})
+    endif()
+    # Run the command
+    execute_process(COMMAND ${ARGN} RESULT_VARIABLE HIP_result)
+endmacro()
+
+# Delete the target file
+hip_execute_process(
+    "Removing ${generated_file}"
+    COMMAND "${CMAKE_COMMAND}" -E remove "${generated_file}"
+    )
+
+# Generate the dependency file
+hip_execute_process(
+    "Generating dependency file: ${cmake_dependency_file}.pre"
+    COMMAND "${__CC}"
+    -M
+    "${source_file}"
+    -o "${cmake_dependency_file}.pre"
+    ${__CC_FLAGS}
+    ${__CC_INCLUDES}
+    )
+
+if(HIP_result)
+    message(FATAL_ERROR "Error generating ${generated_file}")
+endif()
+
+# Generate the cmake readable dependency file to a temp file
+hip_execute_process(
+    "Generating temporary cmake readable file: ${cmake_dependency_file}.tmp"
+    COMMAND "${CMAKE_COMMAND}"
+    -D "input_file:FILEPATH=${cmake_dependency_file}.pre"
+    -D "output_file:FILEPATH=${cmake_dependency_file}.tmp"
+    -D "verbose=${verbose}"
+    -P "${HIP_run_make2cmake}"
+    )
+
+if(HIP_result)
+    message(FATAL_ERROR "Error generating ${generated_file}")
+endif()
+
+# Copy the file if it is different
+hip_execute_process(
+    "Copy if different ${cmake_dependency_file}.tmp to ${cmake_dependency_file}"
+    COMMAND "${CMAKE_COMMAND}" -E copy_if_different "${cmake_dependency_file}.tmp" "${cmake_dependency_file}"
+    )
+
+if(HIP_result)
+    message(FATAL_ERROR "Error generating ${generated_file}")
+endif()
+
+# Delete the temporary file
+hip_execute_process(
+    "Removing ${cmake_dependency_file}.tmp and ${cmake_dependency_file}.pre"
+    COMMAND "${CMAKE_COMMAND}" -E remove "${cmake_dependency_file}.tmp" "${cmake_dependency_file}.pre"
+    )
+
+if(HIP_result)
+    message(FATAL_ERROR "Error generating ${generated_file}")
+endif()
+
+# Generate the output file
+hip_execute_process(
+    "Generating ${generated_file}"
+    COMMAND "${__CC}"
+    -c
+    "${source_file}"
+    -o "${generated_file}"
+    ${__CC_FLAGS}
+    ${__CC_INCLUDES}
+    )
+
+if(HIP_result)
+    # Make sure that we delete the output file
+    hip_execute_process(
+        "Removing ${generated_file}"
+        COMMAND "${CMAKE_COMMAND}" -E remove "${generated_file}"
+        )
+    message(FATAL_ERROR "Error generating file ${generated_file}")
+else()
+    if(verbose)
+        message("Generated ${generated_file} successfully.")
+    endif()
+endif()
+# vim: ts=4:sw=4:expandtab:smartindent
@@ -0,0 +1,50 @@
+###############################################################################
+# Computes dependencies using HIPCC
+###############################################################################
+
+###############################################################################
+# This file converts dependency files generated using hipcc to a format that
+# cmake can understand.
+
+# Input variables:
+#
+# input_file:STRING=<> Dependency file to parse. Required argument
+# output_file:STRING=<> Output file to generate. Required argument
+
+if(NOT input_file OR NOT output_file)
+    message(FATAL_ERROR "You must specify input_file and output_file on the command line")
+endif()
+
+file(READ ${input_file} depend_text)
+
+if (NOT "${depend_text}" STREQUAL "")
+    string(REPLACE " /" "\n/" depend_text ${depend_text})
+    string(REGEX REPLACE "^.*:" "" depend_text ${depend_text})
+    string(REGEX REPLACE "[ \\\\]*\n" ";" depend_text ${depend_text})
+
+    set(dependency_list "")
+
+    foreach(file ${depend_text})
+        string(REGEX REPLACE "^ +" "" file ${file})
+        if(NOT EXISTS "${file}")
+            message(WARNING " Removing non-existent dependency file: ${file}")
+            set(file "")
+        endif()
+
+        if(NOT IS_DIRECTORY "${file}")
+            get_filename_component(file_absolute "${file}" ABSOLUTE)
+            list(APPEND dependency_list "${file_absolute}")
+        endif()
+    endforeach()
+endif()
+
+# Remove the duplicate entries and sort them.
+list(REMOVE_DUPLICATES dependency_list)
+list(SORT dependency_list)
+
+foreach(file ${dependency_list})
+    set(hip_hipcc_depend "${hip_hipcc_depend} \"${file}\"\n")
+endforeach()
+
+file(WRITE ${output_file} "# Generated by: FindHIP.cmake. Do not edit.\nSET(HIP_HIPCC_DEPEND\n ${hip_hipcc_depend})\n\n")
+# vim: ts=4:sw=4:expandtab:smartindent
@@ -14,33 +14,10 @@
 #   - HYPRE_LIBRARIES
 #   - HYPRE_INCLUDE_DIRS
 #   - HYPRE_VERSION
-#   - HYPRE_USING_HIP (internal)
-
-if (HYPRE_FOUND)
-  if (HYPRE_USING_HIP)
-    find_package(rocsparse REQUIRED)
-    find_package(rocrand REQUIRED)
-  endif()
-  return()
-endif()

 include(MfemCmakeUtilities)
 mfem_find_package(HYPRE HYPRE HYPRE_DIR "include" "HYPRE.h" "lib" "HYPRE"
-  "Paths to headers required by HYPRE." "Libraries required by HYPRE."
-  CHECK_BUILD HYPRE_USING_HIP FALSE
-  "
-#undef HYPRE_USING_HIP
-#include <HYPRE_config.h>
-
-#ifndef HYPRE_USING_HIP
-#error HYPRE is built without HIP.
-#endif
-
-int main()
-{
-   return 0;
-}
-")
+  "Paths to headers required by HYPRE." "Libraries required by HYPRE.")

 if (HYPRE_FOUND AND (NOT HYPRE_VERSION))
  try_run(HYPRE_VERSION_RUN_RESULT HYPRE_VERSION_COMPILE_RESULT
@@ -56,12 +33,3 @@ if (HYPRE_FOUND AND (NOT HYPRE_VERSION))
    message(FATAL_ERROR "Unable to determine HYPRE version.")
  endif()
 endif()
-
-if (HYPRE_FOUND AND HYPRE_USING_HIP)
-  find_package(rocsparse REQUIRED)
-  find_package(rocrand REQUIRED)
-  list(APPEND HYPRE_LIBRARIES ${rocsparse_LIBRARIES} ${rocrand_LIBRARIES})
-  set(HYPRE_LIBRARIES ${HYPRE_LIBRARIES} CACHE STRING
-      "HYPRE libraries + dependencies." FORCE)
-  message(STATUS "Updated HYPRE_LIBRARIES: ${HYPRE_LIBRARIES}")
-endif()
@@ -46,7 +46,8 @@ endfunction()
 # Wrapper for add_executable that calls the HIP wrapper if applicable
 macro(mfem_add_executable NAME)
  if (MFEM_USE_HIP)
-    add_executable(${NAME} ${ARGN})
+    hip_add_executable(${NAME} ${ARGN})
+    set_target_properties(${NAME} PROPERTIES LINKER_LANGUAGE CXX)
  else()
    add_executable(${NAME} ${ARGN})
  endif()
@@ -55,7 +56,7 @@ endmacro()
 # Wrapper for add_library that calls the HIP wrapper if applicable
 macro(mfem_add_library NAME)
  if (MFEM_USE_HIP)
-    add_library(${NAME} ${ARGN})
+    hip_add_library(${NAME} ${ARGN})
  else()
    add_library(${NAME} ${ARGN})
  endif()
@@ -91,6 +92,8 @@ macro(add_mfem_examples EXE_SRCS)
    # If CUDA is enabled, tag source files to be compiled with nvcc.
    if (MFEM_USE_CUDA)
      set_source_files_properties(${SRC_FILE} PROPERTIES LANGUAGE CUDA)
+    elseif(MFEM_USE_HIP)
+      set_source_files_properties(${SRC_FILE} PROPERTIES HIP_SOURCE_PROPERTY_FORMAT TRUE)
    endif()

    get_filename_component(SRC_FILENAME ${SRC_FILE} NAME)
@@ -156,6 +159,8 @@ macro(add_mfem_miniapp MFEM_EXE_NAME)
      endforeach()
      set(EXTRA_OPTIONS_LIST ${LIST_})
    endif()
+  elseif(MFEM_USE_HIP)
+    set_source_files_properties(${MAIN_LIST} ${EXTRA_SOURCES_LIST} PROPERTIES HIP_SOURCE_PROPERTY_FORMAT TRUE)
  endif()

  # Actually add the executable
@@ -529,15 +534,12 @@ function(mfem_find_package Name Prefix DirVar IncSuffixes Header LibSuffixes
            if (NOT ImportConfig)
              set(ImportConfig RELEASE)
            endif()
-            set(ImportConfigSuffix "_${ImportConfig}")
            get_target_property(ImpConfigs ${TargetName} IMPORTED_CONFIGURATIONS)
            list(FIND ImpConfigs ${ImportConfig} _Index)
-            if ((_Index EQUAL -1) OR ("${ImportConfig}" STREQUAL "NO_CONFIG"))
-              set(ImportConfig "NO_CONFIG")
-              set(ImportConfigSuffix "")
-              # message(FATAL_ERROR " *** ${ReqPack}: configuration "
-              #   "${ImportConfig} not found. Set ${ReqPack}_IMPORT_CONFIG "
-              #   "from the list: ${ImpConfigs}.")
+            if (_Index EQUAL -1)
+              message(FATAL_ERROR " *** ${ReqPack}: configuration "
+                "${ImportConfig} not found. Set ${ReqPack}_IMPORT_CONFIG "
+                "from the list: ${ImpConfigs}.")
            endif()
          endif()
          # Set _Pack_LIBS
@@ -549,8 +551,8 @@ function(mfem_find_package Name Prefix DirVar IncSuffixes Header LibSuffixes
            endif()
          else()
            # Set _Pack_LIBS from the target properties for ImportConfig
-            foreach (_prop IMPORTED_LOCATION${ImportConfigSuffix}
-                IMPORTED_LINK_INTERFACE_LIBRARIES${ImportConfigSuffix})
+            foreach (_prop IMPORTED_LOCATION_${ImportConfig}
+                IMPORTED_LINK_INTERFACE_LIBRARIES_${ImportConfig})
              get_target_property(_value ${TargetName} ${_prop})
              if (_value)
                list(APPEND _Pack_LIBS ${_value})
@@ -562,7 +564,7 @@ function(mfem_find_package Name Prefix DirVar IncSuffixes Header LibSuffixes
            endif()
          endif()
          # Set _Pack_INCS
-          foreach (_prop INCLUDE_DIRECTORIES INTERFACE_INCLUDE_DIRECTORIES)
+          foreach (_prop INCLUDE_DIRECTORIES)
            get_target_property(_value ${TargetName} ${_prop})
            if (_value)
              list(APPEND _Pack_INCS ${_value})
@@ -740,133 +742,6 @@ function(mfem_find_library Name Prefix Lib LibDoc CheckVar CheckSrc)
 endfunction(mfem_find_library)


-#
-# Extract compile and link options needed by the given target.
-#
-function(mfem_get_target_options Target CompileOptsVar LinkOptsVar)
-
-  if (NOT TARGET ${Target})
-    return()
-  endif()
-
-  # CMAKE_SHARED_LIBRARY_RUNTIME_C_FLAG -> '-Wl,-rpath,'
-  set(shared_link_flag ${CMAKE_SHARED_LIBRARY_RUNTIME_C_FLAG})
-  if (NOT shared_link_flag)
-    set(shared_link_flag "-Wl,-rpath,")
-  endif()
-
-  set(tgt "${Target}")
-  unset(CompileOpts)
-  unset(LinkOpts)
-  get_target_property(IsImported ${tgt} IMPORTED)
-  # message(STATUS "${tgt}[IMPORTED]: ${IsImported}")
-  # Generally, the possible target types are: STATIC_LIBRARY, MODULE_LIBRARY,
-  # SHARED_LIBRARY, INTERFACE_LIBRARY, EXECUTABLE.
-  get_target_property(type ${tgt} TYPE)
-  # message(STATUS "${tgt}[TYPE]: ${type}")
-  unset(ImportConfig)
-  get_target_property(ImportConfigs ${tgt} IMPORTED_CONFIGURATIONS)
-  if (ImportConfigs)
-    list(GET ImportConfigs 0 ImportConfig)
-  endif()
-  if (NOT ImportConfig)
-    set(ImportConfig RELEASE)
-  endif()
-  # message(STATUS "${tgt}[ImportConfig]: ${ImportConfig}")
-  # List all properties with: cmake --help-property-list
-  get_target_property(Defs ${tgt} INTERFACE_COMPILE_DEFINITIONS)
-  if (Defs)
-    list(REMOVE_DUPLICATES Defs)
-    foreach(Def ${Defs})
-      list(APPEND CompileOpts "-D${Def}")
-    endforeach()
-  endif()
-  get_target_property(Opts ${tgt} INTERFACE_COMPILE_OPTIONS)
-  if (Opts)
-    foreach(Opt ${Opts})
-      list(APPEND CompileOpts "${Opt}")
-    endforeach()
-  endif()
-  get_target_property(Dirs ${tgt} INTERFACE_INCLUDE_DIRECTORIES)
-  if (Dirs)
-    list(REMOVE_DUPLICATES Dirs)
-    foreach(Dir ${Dirs})
-      list(APPEND CompileOpts "-I\"${Dir}\"")
-    endforeach()
-  endif()
-  get_target_property(SysDirs ${tgt} INTERFACE_SYSTEM_INCLUDE_DIRECTORIES)
-  if (SysDirs)
-    list(REMOVE_DUPLICATES SysDirs)
-    foreach(SysDir ${SysDirs})
-      list(APPEND CompileOpts "-isystem \"${SysDir}\"")
-    endforeach()
-  endif()
-  if ("${type}" STREQUAL "STATIC_LIBRARY")
-    get_target_property(Location ${tgt} LOCATION)
-    if (Location)
-      list(APPEND LinkOpts "\"${Location}\"")
-    else()
-      message(STATUS " *** Warning: [${tgt}] LOCATION not defined!")
-    endif()
-  elseif ("${type}" STREQUAL "SHARED_LIBRARY")
-    get_target_property(Location ${tgt} LOCATION)
-    if (Location)
-      get_filename_component(Dir ${Location} DIRECTORY)
-      get_filename_component(NameWE ${Location} NAME_WE)
-      string(REGEX REPLACE "^lib" "" LibName ${NameWE})
-      list(APPEND LinkOpts
-        "-L\"${Dir}\""
-        "${shared_link_flag}\"${Dir}\""
-        "-l${LibName}")
-    else()
-      message(STATUS " *** Warning: [${tgt}] LOCATION not defined!")
-    endif()
-  elseif ("${type}" STREQUAL "INTERFACE_LIBRARY")
-    get_target_property(Libs ${tgt} INTERFACE_LINK_LIBRARIES)
-    if (Libs)
-      foreach(Lib ${Libs})
-        if (NOT (TARGET ${Lib}))
-          list(APPEND LinkOpts "${Lib}")
-        else()
-          mfem_get_target_options(${Lib} COpts LOpts)
-          list(APPEND CompileOpts ${COpts})
-          list(APPEND LinkOpts ${LOpts})
-        endif()
-      endforeach()
-    endif()
-    # Other properties we may need to handle:
-    # INTERFACE_LINK_DEPENDS
-    # INTERFACE_LINK_DIRECTORIES
-    # INTERFACE_LINK_OPTIONS
-  else()
-    message(STATUS " *** Warning: [${tgt}] uses target type '${type}'"
-      " which is not supported!")
-  endif()
-
-  # Other potentially relevant properties:
-  # - For all target types:
-  # IMPORTED_LIBNAME
-  # IMPORTED_LIBNAME_${ImportConfig}
-  # INTERFACE_AUTOUIC_OPTIONS
-  # INTERFACE_COMPILE_FEATURES
-  # INTERFACE_POSITION_INDEPENDENT_CODE
-  # INTERFACE_SOURCES
-  # INTERFACE_SYSTEM_INCLUDE_DIRECTORIES)
-  # - For non-"INTERFACE_LIBRARY" target types only:
-  # IMPORTED_LOCATION
-  # IMPORTED_LOCATION_${ImportConfig}
-  # IMPORTED_LINK_INTERFACE_LIBRARIES
-  # IMPORTED_LINK_INTERFACE_LIBRARIES_${ImportConfig}
-  # LINK_FLAGS
-  # LINK_FLAGS_${ImportConfig}
-  # LOCATION_${ImportConfig})
-
-  set(${CompileOptsVar} "${CompileOpts}" PARENT_SCOPE)
-  set(${LinkOptsVar} "${LinkOpts}" PARENT_SCOPE)
-
-endfunction(mfem_get_target_options)
-
-
 #
 #   Function that creates 'config.mk' from 'config.mk.in' for the both the
 #   build- and the install-locations and define install rules for 'config.mk'
@@ -885,15 +760,13 @@ function(mfem_export_mk_files)
  # Convert Boolean vars to YES/NO without writing the values to cache
  set(CONFIG_MK_BOOL_VARS MFEM_USE_MPI MFEM_USE_METIS MFEM_USE_METIS_5
      MFEM_DEBUG MFEM_USE_EXCEPTIONS MFEM_USE_ZLIB MFEM_USE_LIBUNWIND
-      MFEM_USE_LAPACK MFEM_THREAD_SAFE MFEM_USE_LEGACY_OPENMP MFEM_USE_OPENMP
+      MFEM_USE_LAPACK MFEM_THREAD_SAFE MFEM_USE_OPENMP MFEM_USE_LEGACY_OPENMP
      MFEM_USE_MEMALLOC MFEM_USE_SUNDIALS MFEM_USE_MESQUITE MFEM_USE_SUITESPARSE
-      MFEM_USE_SUPERLU MFEM_USE_SUPERLU5 MFEM_USE_MUMPS MFEM_USE_STRUMPACK
-      MFEM_USE_GINKGO MFEM_USE_AMGX MFEM_USE_GNUTLS MFEM_USE_NETCDF
-      MFEM_USE_PETSC MFEM_USE_SLEPC MFEM_USE_MPFR MFEM_USE_SIDRE MFEM_USE_FMS
-      MFEM_USE_CONDUIT MFEM_USE_PUMI MFEM_USE_HIOP MFEM_USE_GSLIB MFEM_USE_CUDA
-      MFEM_USE_HIP MFEM_USE_RAJA MFEM_USE_OCCA MFEM_USE_CEED MFEM_USE_CALIPER
-      MFEM_USE_UMPIRE MFEM_USE_SIMD MFEM_USE_ADIOS2 MFEM_USE_MKL_CPARDISO
-      MFEM_USE_ADFORWARD MFEM_USE_CODIPACK MFEM_USE_BENCHMARK MFEM_USE_PARELAG)
+      MFEM_USE_SUPERLU MFEM_USE_STRUMPACK MFEM_USE_GINKGO MFEM_USE_AMGX
+      MFEM_USE_GNUTLS MFEM_USE_GSLIB MFEM_USE_NETCDF MFEM_USE_PETSC
+      MFEM_USE_SLEPC MFEM_USE_MPFR MFEM_USE_SIDRE MFEM_USE_CONDUIT MFEM_USE_PUMI
+      MFEM_USE_CUDA MFEM_USE_OCCA MFEM_USE_RAJA MFEM_USE_UMPIRE MFEM_USE_SIMD
+      MFEM_USE_ADIOS2 MFEM_USE_BENCHMARK MFEM_USE_PARELAG)
  foreach(var ${CONFIG_MK_BOOL_VARS})
    if (${var})
      set(${var} YES)
@@ -995,18 +868,8 @@ function(mfem_export_mk_files)
    get_filename_component(suffix ${lib} EXT)
    # handle interfaces (e.g., SCOREC::apf)
    if ("${lib}" MATCHES "SCOREC::.*" OR "${lib}" MATCHES "Ginkgo::.*")
-    elseif (TARGET "${lib}")
-      mfem_get_target_options(${lib} CompileOpts LinkOpts)
-      # Removing duplicates may lead to issues:
-      # list(REMOVE_DUPLICATES CompileOpts)
-      # list(REMOVE_DUPLICATES LinkOpts)
-      string(REPLACE ";" " " COpts "${CompileOpts}")
-      string(REPLACE ";" " " LOpts "${LinkOpts}")
-      # message(STATUS "${lib}[COpts]: '${COpts}'")
-      # message(STATUS "${lib}[LOpts]: '${LOpts}'")
-      set(MFEM_TPLFLAGS "${MFEM_TPLFLAGS} ${COpts}")
-      set(MFEM_EXT_LIBS "${MFEM_EXT_LIBS} ${LOpts}")
-      # message(FATAL_ERROR "***** interface lib found ... exiting *****")
+    elseif (NOT "${lib}" MATCHES "SCOREC::.*" AND "${lib}" MATCHES ".*::.*")
+      message(FATAL_ERROR "***** interface lib found ... exiting *****")
      # handle static and shared libs
    elseif ("${suffix}" STREQUAL "${CMAKE_SHARED_LIBRARY_SUFFIX}")
      get_filename_component(dir ${lib} DIRECTORY)
@@ -91,7 +91,7 @@ MFEM_MPIEXEC_NP = @MFEM_MPIEXEC_NP@
 MFEM_MPI_NP     = @MFEM_MPI_NP@

 # The NVCC compiler cannot link with -x=cu
-MFEM_LINK_FLAGS := $(filter-out -x=cu -xhip, $(MFEM_FLAGS))
+MFEM_LINK_FLAGS := $(filter-out -x=cu, $(MFEM_FLAGS))

 # Optional extra configuration
@MFEM_CONFIG_EXTRA@
@@ -50,7 +50,6 @@ option(MFEM_USE_CONDUIT "Enable Conduit usage" OFF)
 option(MFEM_USE_PUMI "Enable PUMI" OFF)
 option(MFEM_USE_HIOP "Enable HiOp" OFF)
 option(MFEM_USE_CUDA "Enable CUDA" OFF)
-option(MFEM_USE_HIP "Enable HIP" OFF)
 option(MFEM_USE_OCCA "Enable OCCA" OFF)
 option(MFEM_USE_RAJA "Enable RAJA" OFF)
 option(MFEM_USE_CEED "Enable CEED" OFF)
@@ -108,7 +107,6 @@ if (MFEM_USE_CUDA)
   set(HYPRE_REQUIRED_LIBRARIES "-lcusparse" "-lcurand" CACHE STRING
       "Libraries that HYPRE depends on.")
 endif()
-# HIP dependency for HYPRE is handled in FindHYPRE.cmake.

 set(METIS_DIR "${MFEM_DIR}/../metis-4.0" CACHE PATH "Path to the METIS library.")

@@ -228,11 +226,6 @@ set(MKL_LIBRARY_DIR "" CACHE STRING "Custom library subdirectory")

 set(OCCA_DIR "${MFEM_DIR}/../occa" CACHE PATH "Path to OCCA")
 set(RAJA_DIR "${MFEM_DIR}/../raja" CACHE PATH "Path to RAJA")
-# If RAJA is built with external CAMP:
-# set(RAJA_REQUIRED_PACKAGES "camp"
-#     CACHE STRING "Packages that RAJA depends on.")
-# set(camp_DIR "${MFEM_DIR}/../camp/lib/cmake/camp"
-#     CACHE PATH "Path to CAMP CMake files.")
 set(CEED_DIR "${MFEM_DIR}/../libCEED" CACHE PATH "Path to libCEED")
 set(UMPIRE_DIR "${MFEM_DIR}/../umpire" CACHE PATH "Path to Umpire")
 set(CALIPER_DIR "${MFEM_DIR}/../caliper" CACHE PATH "Path to Caliper")
@@ -252,8 +245,7 @@ set(BLAS_LIBRARIES "" CACHE STRING "The BLAS library.")
 set(LAPACK_INCLUDE_DIRS "" CACHE STRING "Path to LAPACK headers.")
 set(LAPACK_LIBRARIES "" CACHE STRING "The LAPACK library.")

-set(CODIPACK_INCLUDE_DIRS "${MFEM_DIR}/../CoDiPack/include" CACHE STRING
-    "Path to CoDiPack headers.")
+set(CODIPACK_INCLUDE_DIRS "${MFEM_DIR}/../CoDiPack/inlude" CACHE STRING "Path to CoDiPack headers.")
 set(CODIPACK_LIBRARIES "")

 # Some useful variables:
@@ -172,20 +172,6 @@ ifeq ($(MFEM_USE_MPI)$(MFEM_USE_HIP),YESYES)
   MPI_LIB = -L$(MPI_DIR)/lib $(XLINKER)-rpath,$(MPI_DIR)/lib -lmpi
 endif

-# ROCM/HIP directory such that ROCM/HIP libraries like rocsparse and rocrand are
-# found in $(HIP_DIR)/lib, usually as links. Typically, this directoory is of
-# the form /opt/rocm-X.Y.Z which is called ROCM_PATH by hipconfig.
-ifeq ($(MFEM_USE_HIP),YES)
-   HIP_DIR := $(patsubst %/,%,$(dir $(shell which $(HIP_CXX))))
-   HIP_DIR := $(patsubst %/,%,$(dir $(HIP_DIR)))
-   ifeq (,$(wildcard $(HIP_DIR)/lib/librocsparse.*))
-      HIP_DIR := $(shell hipconfig --rocmpath 2> /dev/null)
-      ifeq (,$(wildcard $(HIP_DIR)/lib/librocsparse.*))
-         $(error Unable to determine HIP_DIR. Please set it manually.)
-      endif
-   endif
-endif
-
 # Compile and link options for zlib.
 ZLIB_DIR =
 ZLIB_OPT = $(if $(ZLIB_DIR),-I$(ZLIB_DIR)/include)
@@ -203,11 +189,6 @@ ifeq (YES,$(MFEM_USE_CUDA))
   # This is only necessary when hypre is built with cuda:
   HYPRE_LIB += -lcusparse -lcurand
 endif
-ifeq (YES,$(MFEM_USE_HIP))
-   # This is only necessary when hypre is built with hip:
-   HYPRE_LIB += -L$(HIP_DIR)/lib $(XLINKER)-rpath,$(HIP_DIR)/lib\
- -lrocsparse -lrocrand
-endif

 # METIS library configuration
 ifeq ($(MFEM_USE_SUPERLU)$(MFEM_USE_STRUMPACK)$(MFEM_USE_MUMPS),NONONO)
@@ -449,9 +430,9 @@ GSLIB_LIB = -L$(GSLIB_DIR)/lib -lgs
 CUDA_OPT =
 CUDA_LIB = -lcusparse

-# HIP library configuration
+# HIP library configuration (currently not needed)
 HIP_OPT =
-HIP_LIB = -L$(HIP_DIR)/lib $(XLINKER)-rpath,$(HIP_DIR)/lib -lhipsparse
+HIP_LIB =

 # OCCA library configuration
 OCCA_DIR = @MFEM_DIR@/../occa
@@ -58,14 +58,14 @@ ifneq (,$(filter test%,$(MAKECMDGOALS)))
   MAKEFLAGS += -k
 endif
 # Test runs of the examples/miniapps with parameters - check exit code:
-# 0 means success, 242 means the test was skipped, anything else means error
+# 0 means success, 255 means the test was skipped, anything else means error
 mfem-test = \
   printf "   $(3) [$(2) $(1) ... ]: "; \
   $(call $(TIMEFUN),$(TIMECMD),$(2) ./$(1) $(if $(5),,-no-vis )$(4) \
     > $(1).stderr 2>&1); \
   err="$$3"; \
   if [ "$$3" = 0 ]; then $(PRINT_OK); \
-   else if [ "$$3" = 242 ]; then $(PRINT_SKIP); err=0; \
+   else if [ "$$3" = 255 ]; then $(PRINT_SKIP); err=0; \
   else $(PRINT_FAILED); cat $(1).stderr; fi; fi; \
   rm -f $(1).stderr; exit $$err

@@ -76,7 +76,7 @@ mfem-test-file = \
   $(call $(TIMEFUN),$(TIMECMD),$(2) ./$(1) -no-vis > $(1).stderr 2>&1); \
   err="$$3"; \
   if [ "$$3" = 0 ] && [ -e $(4) ]; then $(PRINT_OK); \
-   else if [ "$$3" = 242 ] && [ -e $(4) ]; then $(PRINT_SKIP); err=0; \
+   else if [ "$$3" = 255 ] && [ -e $(4) ]; then $(PRINT_SKIP); err=0; \
   else $(PRINT_FAILED); cat $(1).stderr; err=64; fi; fi; \
   rm -f $(1).stderr; exit $$err

@@ -149,7 +149,7 @@ void InitialDeformation(const Vector &x, Vector &y);

 void InitialVelocity(const Vector &x, Vector &v);

-void visualize(ostream &os, Mesh *mesh, GridFunction *deformed_nodes,
+void visualize(ostream &out, Mesh *mesh, GridFunction *deformed_nodes,
               GridFunction *field, const char *field_name = NULL,
               bool init_vis = false);

@@ -376,10 +376,10 @@ int main(int argc, char *argv[])
 }


-void visualize(ostream &os, Mesh *mesh, GridFunction *deformed_nodes,
+void visualize(ostream &out, Mesh *mesh, GridFunction *deformed_nodes,
               GridFunction *field, const char *field_name, bool init_vis)
 {
-   if (!os)
+   if (!out)
   {
      return;
   }
@@ -389,25 +389,24 @@ void visualize(ostream &os, Mesh *mesh, GridFunction *deformed_nodes,

   mesh->SwapNodes(nodes, owns_nodes);

-   os << "solution\n" << *mesh << *field;
+   out << "solution\n" << *mesh << *field;

   mesh->SwapNodes(nodes, owns_nodes);

   if (init_vis)
   {
-      os << "window_size 800 800\n";
-      os << "window_title '" << field_name << "'\n";
+      out << "window_size 800 800\n";
+      out << "window_title '" << field_name << "'\n";
      if (mesh->SpaceDimension() == 2)
      {
-         os << "view 0 0\n"; // view from top
-         os << "keys jl\n";  // turn off perspective and light
+         out << "view 0 0\n"; // view from top
+         out << "keys jl\n";  // turn off perspective and light
      }
-      os << "keys cm\n";         // show colorbar and mesh
-      // update value-range; keep mesh-extents fixed
-      os << "autoscale value\n";
-      os << "pause\n";
+      out << "keys cm\n";         // show colorbar and mesh
+      out << "autoscale value\n"; // update value-range; keep mesh-extents fixed
+      out << "pause\n";
   }
-   os << flush;
+   out << flush;
 }


@@ -154,8 +154,7 @@ void InitialDeformation(const Vector &x, Vector &y);

 void InitialVelocity(const Vector &x, Vector &v);

-void visualize(ostream &os, ParMesh *mesh,
-               ParGridFunction *deformed_nodes,
+void visualize(ostream &out, ParMesh *mesh, ParGridFunction *deformed_nodes,
               ParGridFunction *field, const char *field_name = NULL,
               bool init_vis = false);

@@ -439,11 +438,10 @@ int main(int argc, char *argv[])
   return 0;
 }

-void visualize(ostream &os, ParMesh *mesh,
-               ParGridFunction *deformed_nodes,
+void visualize(ostream &out, ParMesh *mesh, ParGridFunction *deformed_nodes,
               ParGridFunction *field, const char *field_name, bool init_vis)
 {
-   if (!os)
+   if (!out)
   {
      return;
   }
@@ -453,27 +451,25 @@ void visualize(ostream &os, ParMesh *mesh,

   mesh->SwapNodes(nodes, owns_nodes);

-   os << "parallel " << mesh->GetNRanks()
-      << " " << mesh->GetMyRank() << "\n";
-   os << "solution\n" << *mesh << *field;
+   out << "parallel " << mesh->GetNRanks() << " " << mesh->GetMyRank() << "\n";
+   out << "solution\n" << *mesh << *field;

   mesh->SwapNodes(nodes, owns_nodes);

   if (init_vis)
   {
-      os << "window_size 800 800\n";
-      os << "window_title '" << field_name << "'\n";
+      out << "window_size 800 800\n";
+      out << "window_title '" << field_name << "'\n";
      if (mesh->SpaceDimension() == 2)
      {
-         os << "view 0 0\n"; // view from top
-         os << "keys jl\n";  // turn off perspective and light
+         out << "view 0 0\n"; // view from top
+         out << "keys jl\n";  // turn off perspective and light
      }
-      os << "keys cm\n";         // show colorbar and mesh
-      // update value-range; keep mesh-extents fixed
-      os << "autoscale value\n";
-      os << "pause\n";
+      out << "keys cm\n";         // show colorbar and mesh
+      out << "autoscale value\n"; // update value-range; keep mesh-extents fixed
+      out << "pause\n";
   }
-   os << flush;
+   out << flush;
 }


@@ -32,7 +32,7 @@ private:
   mutable DenseTensor flux;
   mutable Vector z;

-   void GetFlux(const DenseMatrix &state_, DenseTensor &flux_) const;
+   void GetFlux(const DenseMatrix &state, DenseTensor &flux) const;

 public:
   FE_Evolution(FiniteElementSpace &vfes_,
@@ -256,26 +256,26 @@ inline double ComputeMaxCharSpeed(const Vector &state, const int dim)
 }

 // Compute the flux at solution nodes.
-void FE_Evolution::GetFlux(const DenseMatrix &x_, DenseTensor &flux_) const
+void FE_Evolution::GetFlux(const DenseMatrix &x, DenseTensor &flux) const
 {
-   const int flux_dof = flux_.SizeI();
-   const int flux_dim = flux_.SizeJ();
+   const int dof = flux.SizeI();
+   const int dim = flux.SizeJ();

-   for (int i = 0; i < flux_dof; i++)
+   for (int i = 0; i < dof; i++)
   {
-      for (int k = 0; k < num_equation; k++) { state(k) = x_(i, k); }
-      ComputeFlux(state, flux_dim, f);
+      for (int k = 0; k < num_equation; k++) { state(k) = x(i, k); }
+      ComputeFlux(state, dim, f);

-      for (int d = 0; d < flux_dim; d++)
+      for (int d = 0; d < dim; d++)
      {
         for (int k = 0; k < num_equation; k++)
         {
-            flux_(i, d, k) = f(k, d);
+            flux(i, d, k) = f(k, d);
         }
      }

      // Update max char speed
-      const double mcs = ComputeMaxCharSpeed(state, flux_dim);
+      const double mcs = ComputeMaxCharSpeed(state, dim);
      if (mcs > max_char_speed) { max_char_speed = mcs; }
   }
 }
@@ -171,7 +171,7 @@ public:
 };

 // Visualization driver
-void visualize(ostream &os, Mesh *mesh, GridFunction *deformed_nodes,
+void visualize(ostream &out, Mesh *mesh, GridFunction *deformed_nodes,
               GridFunction *field, const char *field_name = NULL,
               bool init_vis = false);

@@ -542,10 +542,10 @@ RubberOperator::~RubberOperator()


 // Inline visualization
-void visualize(ostream &os, Mesh *mesh, GridFunction *deformed_nodes,
+void visualize(ostream &out, Mesh *mesh, GridFunction *deformed_nodes,
               GridFunction *field, const char *field_name, bool init_vis)
 {
-   if (!os)
+   if (!out)
   {
      return;
   }
@@ -555,25 +555,23 @@ void visualize(ostream &os, Mesh *mesh, GridFunction *deformed_nodes,

   mesh->SwapNodes(nodes, owns_nodes);

-   os << "solution\n" << *mesh << *field;
+   out << "solution\n" << *mesh << *field;

   mesh->SwapNodes(nodes, owns_nodes);

   if (init_vis)
   {
-      os << "window_size 800 800\n";
-      os << "window_title '" << field_name << "'\n";
+      out << "window_size 800 800\n";
+      out << "window_title '" << field_name << "'\n";
      if (mesh->SpaceDimension() == 2)
      {
-         os << "view 0 0\n"; // view from top
-         // turn off perspective and light, +anti-aliasing
-         os << "keys jlA\n";
+         out << "view 0 0\n"; // view from top
+         out << "keys jlA\n"; // turn off perspective and light, +anti-aliasing
      }
-      os << "keys cmA\n"; // show colorbar and mesh, +anti-aliasing
-      // update value-range; keep mesh-extents fixed
-      os << "autoscale value\n";
+      out << "keys cmA\n";        // show colorbar and mesh, +anti-aliasing
+      out << "autoscale value\n"; // update value-range; keep mesh-extents fixed
   }
-   os << flush;
+   out << flush;
 }

 void ReferenceConfiguration(const Vector &x, Vector &y)
@@ -185,8 +185,7 @@ public:
 };

 // Visualization driver
-void visualize(ostream &os, ParMesh *mesh,
-               ParGridFunction *deformed_nodes,
+void visualize(ostream &out, ParMesh *mesh, ParGridFunction *deformed_nodes,
               ParGridFunction *field, const char *field_name = NULL,
               bool init_vis = false);

@@ -197,10 +196,10 @@ void InitialDeformation(const Vector &x, Vector &y);

 int main(int argc, char *argv[])
 {
-#ifdef HYPRE_USING_GPU
+#ifdef HYPRE_USING_CUDA
   cout << "\nAs of mfem-4.3 and hypre-2.22.0 (July 2021) this example\n"
-        << "is NOT supported with the GPU version of hypre.\n\n";
-   return 242;
+        << "is NOT supported with the CUDA version of hypre.\n\n";
+   return 255;
 #endif

   // 1. Initialize MPI
@@ -487,8 +486,8 @@ void JacobianPreconditioner::SetOperator(const Operator &op)

      if (!spaces[0]->GetParMesh()->Nonconforming())
      {
-#if !defined(HYPRE_USING_GPU)
-         // Not available yet when hypre is built with GPU support
+#ifndef HYPRE_USING_CUDA
+         // Not available yet when hypre is built with CUDA
         stiff_prec_amg->SetElasticityOptions(spaces[0]);
 #endif
      }
@@ -618,11 +617,10 @@ RubberOperator::~RubberOperator()


 // Inline visualization
-void visualize(ostream &os, ParMesh *mesh,
-               ParGridFunction *deformed_nodes,
+void visualize(ostream &out, ParMesh *mesh, ParGridFunction *deformed_nodes,
               ParGridFunction *field, const char *field_name, bool init_vis)
 {
-   if (!os)
+   if (!out)
   {
      return;
   }
@@ -632,27 +630,24 @@ void visualize(ostream &os, ParMesh *mesh,

   mesh->SwapNodes(nodes, owns_nodes);

-   os << "parallel " << mesh->GetNRanks() << " " << mesh->GetMyRank() <<
-      "\n";
-   os << "solution\n" << *mesh << *field;
+   out << "parallel " << mesh->GetNRanks() << " " << mesh->GetMyRank() << "\n";
+   out << "solution\n" << *mesh << *field;

   mesh->SwapNodes(nodes, owns_nodes);

   if (init_vis)
   {
-      os << "window_size 800 800\n";
-      os << "window_title '" << field_name << "'\n";
+      out << "window_size 800 800\n";
+      out << "window_title '" << field_name << "'\n";
      if (mesh->SpaceDimension() == 2)
      {
-         os << "view 0 0\n"; // view from top
-         // turn off perspective and light, +anti-aliasing
-         os << "keys jlA\n";
+         out << "view 0 0\n"; // view from top
+         out << "keys jlA\n"; // turn off perspective and light, +anti-aliasing
      }
-      os << "keys cmA\n"; // show colorbar and mesh, +anti-aliasing
-      // update value-range; keep mesh-extents fixed
-      os << "autoscale value\n";
+      out << "keys cmA\n";        // show colorbar and mesh, +anti-aliasing
+      out << "autoscale value\n"; // update value-range; keep mesh-extents fixed
   }
-   os << flush;
+   out << flush;
 }

 void ReferenceConfiguration(const Vector &x, Vector &y)
@@ -0,0 +1,266 @@
+//                                MFEM Example 1
+//
+// Compile with: make ex1
+//
+// Sample runs:  ex1 -m ../data/square-disc.mesh
+//               ex1 -m ../data/star.mesh
+//               ex1 -m ../data/escher.mesh
+//               ex1 -m ../data/fichera.mesh
+//               ex1 -m ../data/square-disc-p2.vtk -o 2
+//               ex1 -m ../data/square-disc-p3.mesh -o 3
+//               ex1 -m ../data/square-disc-nurbs.mesh -o -1
+//               ex1 -m ../data/disc-nurbs.mesh -o -1
+//               ex1 -m ../data/pipe-nurbs.mesh -o -1
+//               ex1 -m ../data/star-surf.mesh
+//               ex1 -m ../data/square-disc-surf.mesh
+//               ex1 -m ../data/inline-segment.mesh
+//               ex1 -m ../data/amr-quad.mesh
+//               ex1 -m ../data/amr-hex.mesh
+//               ex1 -m ../data/fichera-amr.mesh
+//               ex1 -m ../data/mobius-strip.mesh
+//               ex1 -m ../data/mobius-strip.mesh -o -1 -sc
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Laplace problem
+//               -Delta u = 1 with homogeneous Dirichlet boundary conditions.
+//               Specifically, we discretize using a FE space of the specified
+//               order, or if order < 1 using an isoparametric/isogeometric
+//               space (i.e. quadratic for quadratic curvilinear mesh, NURBS for
+//               NURBS mesh, etc.)
+//
+//               The example highlights the use of mesh refinement, finite
+//               element grid functions, as well as linear and bilinear forms
+//               corresponding to the left-hand side and right-hand side of the
+//               discrete linear system. We also cover the explicit elimination
+//               of essential boundary conditions, static condensation, and the
+//               optional connection to the GLVis tool for visualization.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options.
+   const char *mesh_file = "./star-set.mesh";
+   int order = 1;
+   int rs = -1;
+   int ra = 0;
+   int bt = EntitySets::INVALID;
+   const char *bs = "Origin";
+   bool static_cond = false;
+   bool visualization = 1;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&rs, "-rs", "--refine-serial",
+                  "Number of serial refinement levels");
+   args.AddOption(&ra, "-ra", "--refine-adaptive",
+                  "Number of adaptive refinement levels");
+   args.AddOption(&bt, "-bt", "--bc-entity-type",
+                  "");
+   args.AddOption(&bs, "-bs", "--bc-entity-set-name",
+                  "");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   // 2. Read the mesh from the given mesh file. We can handle triangular,
+   //    quadrilateral, tetrahedral, hexahedral, surface and volume meshes with
+   //    the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 3. Refine the mesh to increase the resolution. In this example we do
+   //    'ref_levels' of uniform refinement. We choose 'ref_levels' to be the
+   //    largest number that gives a final mesh with no more than 50,000
+   //    elements.
+   {
+      int ref_levels = ( rs >= 0 ) ? rs :
+                       (int)floor(log(50000./mesh->GetNE())/log(2.)/dim);
+      for (int l = 0; l < ref_levels; l++)
+      {
+         mesh->UniformRefinement();
+      }
+   }
+   if ( mesh->ent_sets )
+   {
+      cout << "mesh->ent_sets is non NULL" << endl;
+      mesh->ent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "mesh->ent_sets is NULL" << endl;
+   }
+
+   if ( ra > 0 )
+   {
+      cout << "calling EnsureNCMesh" << endl;
+      mesh->EnsureNCMesh();
+      cout << "back from EnsureNCMesh" << endl;
+   }
+   if ( mesh->ent_sets )
+   {
+      cout << "mesh->ent_sets is non NULL" << endl;
+   }
+   else
+   {
+      cout << "mesh->ent_sets is NULL" << endl;
+   }
+   cout << "Calling RandomRefinement " << ra << " times." << endl;
+   for (int l = 0; l < ra; l++)
+   {
+      mesh->RandomRefinement(0.2);
+   }
+   cout << "Done with refinement" << endl;
+   if ( mesh->ent_sets )
+   {
+      cout << "mesh->ent_sets is non NULL" << endl;
+      mesh->ent_sets->PrintSetInfo(cout);
+   }
+   if ( mesh->ncmesh )
+   {
+      mesh->ncmesh->PrintStats(cout);
+
+      ofstream ofsV("vp.out");
+      ofstream ofsE("ce.out");
+      mesh->ncmesh->PrintVertexParents(ofsV);
+      mesh->ncmesh->PrintCoarseElements(ofsE);
+   }
+
+   // 4. Define a finite element space on the mesh. Here we use continuous
+   //    Lagrange finite elements of the specified order. If order < 1, we
+   //    instead use an isoparametric/isogeometric space.
+   FiniteElementCollection *fec;
+   if (order > 0)
+   {
+      fec = new H1_FECollection(order, dim);
+   }
+   else if (mesh->GetNodes())
+   {
+      fec = mesh->GetNodes()->OwnFEC();
+      cout << "Using isoparametric FEs: " << fec->Name() << endl;
+   }
+   else
+   {
+      fec = new H1_FECollection(order = 1, dim);
+   }
+   FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
+   cout << "Number of finite element unknowns: "
+        << fespace->GetTrueVSize() << endl;
+
+   // 5. Determine the list of true (i.e. conforming) essential boundary dofs.
+   //    In this example, the boundary conditions are defined by marking all
+   //    the boundary attributes from the mesh as essential (Dirichlet) and
+   //    converting them to a list of true dofs.
+   Array<int> ess_tdof_list;
+   if ( bt == EntitySets::INVALID )
+   {
+      if (mesh->bdr_attributes.Size())
+      {
+         Array<int> ess_bdr(mesh->bdr_attributes.Max());
+         ess_bdr = 1;
+         fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+      }
+   }
+   else
+   {
+      fespace->GetEssentialTrueDofs((EntitySets::EntityType)bt, bs,
+                                    ess_tdof_list);
+   }
+
+   cout << "Number of Dirichlet dofs: " << ess_tdof_list.Size() << endl;
+
+   // 6. Set up the linear form b(.) which corresponds to the right-hand side of
+   //    the FEM linear system, which in this case is (1,phi_i) where phi_i are
+   //    the basis functions in the finite element fespace.
+   LinearForm *b = new LinearForm(fespace);
+   ConstantCoefficient one(1.0);
+   b->AddDomainIntegrator(new DomainLFIntegrator(one));
+   b->Assemble();
+
+   // 7. Define the solution vector x as a finite element grid function
+   //    corresponding to fespace. Initialize x with initial guess of zero,
+   //    which satisfies the boundary conditions.
+   GridFunction x(fespace);
+   x = 0.0;
+
+   // 8. Set up the bilinear form a(.,.) on the finite element space
+   //    corresponding to the Laplacian operator -Delta, by adding the Diffusion
+   //    domain integrator.
+   BilinearForm *a = new BilinearForm(fespace);
+   a->AddDomainIntegrator(new DiffusionIntegrator(one));
+
+   // 9. Assemble the bilinear form and the corresponding linear system,
+   //    applying any necessary transformations such as: eliminating boundary
+   //    conditions, applying conforming constraints for non-conforming AMR,
+   //    static condensation, etc.
+   if (static_cond) { a->EnableStaticCondensation(); }
+   a->Assemble();
+
+   SparseMatrix A;
+   Vector B, X;
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+
+   cout << "Size of linear system: " << A.Height() << endl;
+
+#ifndef MFEM_USE_SUITESPARSE
+   // 10. Define a simple symmetric Gauss-Seidel preconditioner and use it to
+   //     solve the system A X = B with PCG.
+   GSSmoother M(A);
+   PCG(A, M, B, X, 1, 200, 1e-12, 0.0);
+#else
+   // 10. If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.
+   UMFPackSolver umf_solver;
+   umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+   umf_solver.SetOperator(A);
+   umf_solver.Mult(B, X);
+#endif
+
+   // 11. Recover the solution as a finite element grid function.
+   a->RecoverFEMSolution(X, *b, x);
+
+   // 12. Save the refined mesh and the solution. This output can be viewed later
+   //     using GLVis: "glvis -m refined.mesh -g sol.gf".
+   ofstream mesh_ofs("refined.mesh");
+   mesh_ofs.precision(8);
+   mesh->Print(mesh_ofs);
+   ofstream sol_ofs("sol.gf");
+   sol_ofs.precision(8);
+   x.Save(sol_ofs);
+
+   // 13. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *mesh << x << flush;
+   }
+
+   // 14. Free the used memory.
+   delete a;
+   delete b;
+   delete fespace;
+   if (order > 0) { delete fec; }
+   delete mesh;
+
+   return 0;
+}
@@ -0,0 +1,388 @@
+//                       MFEM Example 1 - Parallel Version
+//
+// Compile with: make ex1p
+//
+// Sample runs:  mpirun -np 4 ex1p -m ../data/square-disc.mesh
+//               mpirun -np 4 ex1p -m ../data/star.mesh
+//               mpirun -np 4 ex1p -m ../data/escher.mesh
+//               mpirun -np 4 ex1p -m ../data/fichera.mesh
+//               mpirun -np 4 ex1p -m ../data/square-disc-p2.vtk -o 2
+//               mpirun -np 4 ex1p -m ../data/square-disc-p3.mesh -o 3
+//               mpirun -np 4 ex1p -m ../data/square-disc-nurbs.mesh -o -1
+//               mpirun -np 4 ex1p -m ../data/disc-nurbs.mesh -o -1
+//               mpirun -np 4 ex1p -m ../data/pipe-nurbs.mesh -o -1
+//               mpirun -np 4 ex1p -m ../data/ball-nurbs.mesh -o 2
+//               mpirun -np 4 ex1p -m ../data/star-surf.mesh
+//               mpirun -np 4 ex1p -m ../data/square-disc-surf.mesh
+//               mpirun -np 4 ex1p -m ../data/inline-segment.mesh
+//               mpirun -np 4 ex1p -m ../data/amr-quad.mesh
+//               mpirun -np 4 ex1p -m ../data/amr-hex.mesh
+//               mpirun -np 4 ex1p -m ../data/mobius-strip.mesh
+//               mpirun -np 4 ex1p -m ../data/mobius-strip.mesh -o -1 -sc
+//
+//               The following are examples of using EntitySets to define
+//               homogeneous Dirichlet boundary condition.  These examples
+//               require a modified mesh file and a specialized version of
+//               example 1 called "ex1p_es".
+//               mpirun -np 4 ex1p_es -m ./fichera-set.mesh -bt 0 -bs Origin
+//               mpirun -np 4 ex1p_es -m ./fichera-set.mesh -bt 1 -bs Axes
+//               mpirun -np 4 ex1p_es -m ./fichera-set.mesh
+//                                    -bt 1 -bs "Negative Axes"
+//               mpirun -np 4 ex1p_es -m ./fichera-set.mesh
+//                                    -bt 2 -bs "Interior Corner"
+//               mpirun -np 4 ex1p_es -m ./fichera-set.mesh
+//                                    -bt 2 -bs "Exterior Corner"
+//               mpirun -np 4 ex1p_es -m ./fichera-set.mesh
+//                                    -bt 3 -bs "Interior Corner"
+//               mpirun -np 4 ex1p_es -m ./fichera-set.mesh
+//                                    -bt 3 -bs "Exterior Corner"
+//               mpirun -np 4 ex1p_es -m ./fichera-set.mesh -bt 3 -bs "Steps"
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Laplace problem
+//               -Delta u = 1 with homogeneous Dirichlet boundary conditions.
+//               Specifically, we discretize using a FE space of the specified
+//               order, or if order < 1 using an isoparametric/isogeometric
+//               space (i.e. quadratic for quadratic curvilinear mesh, NURBS for
+//               NURBS mesh, etc.)
+//
+//               The example highlights the use of mesh refinement, finite
+//               element grid functions, as well as linear and bilinear forms
+//               corresponding to the left-hand side and right-hand side of the
+//               discrete linear system. We also cover the explicit elimination
+//               of essential boundary conditions, static condensation, and the
+//               optional connection to the GLVis tool for visualization.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   const char *mesh_file = "./star-set.mesh";
+   int order = 1;
+   int rs = -1;
+   int rp = 2;
+   int ra = 0;
+   int bt = EntitySets::INVALID;
+   const char *bs = "Origin";
+   bool static_cond = false;
+   bool visualization = 1;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&rs, "-rs", "--refine-serial",
+                  "Number of serial refinement levels");
+   args.AddOption(&rp, "-rp", "--refine-parallel",
+                  "Number of parallel refinement levels");
+   args.AddOption(&ra, "-ra", "--refine-adaptive",
+                  "Number of adaptive refinement levels");
+   args.AddOption(&bt, "-bt", "--bc-entity-type",
+                  "");
+   args.AddOption(&bs, "-bs", "--bc-entity-set-name",
+                  "");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
+   //    and volume meshes with the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 4. Refine the serial mesh on all processors to increase the resolution. In
+   //    this example we do 'ref_levels' of uniform refinement. We choose
+   //    'ref_levels' to be the largest number that gives a final mesh with no
+   //    more than 10,000 elements.
+   {
+      int ref_levels = ( rs >= 0 ) ? rs :
+                       (int)floor(log(10000./mesh->GetNE())/log(2.)/dim);
+      for (int l = 0; l < ref_levels; l++)
+      {
+         if ( myid == 0 ) { cout << "Uniform refinement in serial..."; }
+         mesh->UniformRefinement();
+      }
+      MPI_Barrier(MPI_COMM_WORLD);
+      if ( myid == 0 && rs > 0 ) { cout << "Done" << endl; }
+   }
+   if ( mesh->ent_sets )
+   {
+      cout << "mesh->ent_sets is non NULL" << endl;
+      mesh->ent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "mesh->ent_sets is NULL" << endl;
+   }
+   /*
+     At this point we have a serial mesh containing an EntitySets
+     object which stores the current node/edge/face/element indices
+     for each entity in each set.  This data is duplicated on each MPI
+     rank.
+    */
+   if ( ra > 0 )
+   {
+      cout << "calling EnsureNCMesh" << endl;
+      mesh->EnsureNCMesh();
+      cout << "back from EnsureNCMesh" << endl;
+   }
+   if ( mesh->ent_sets )
+   {
+      cout << "mesh->ent_sets is non NULL" << endl;
+      mesh->ent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "mesh->ent_sets is NULL" << endl;
+   }
+   /*
+     We now have an NCEntitySets object which stores the node indices
+     describing each enity in each node/edge/face set and the element
+     indices for the elements in each element set.  This data is
+     duplicated on each MPI rank.
+    */
+
+   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted.
+   cout << "creating ParMesh from serial mesh" << endl;
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   cout << "done creating ParMesh from serial mesh" << endl;
+   delete mesh;
+   if ( pmesh->pent_sets )
+   {
+      cout << "pmesh->pent_sets is non NULL" << endl;
+      pmesh->pent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "pmesh->pent_sets is NULL" << endl;
+   }
+   /*
+     We now have a ParEntitySets object which marshals the data stored
+     in EntitySets objects.  The data has now been pruned so that each
+     rank only contains indices of local entities.
+
+     The NCEntitySets object remains unchanged...
+
+     If we have an NC mesh a different path is taken and the
+     EntitySets are ignored.
+
+     1) ParNCMesh is created from NCMesh
+       a) Creates a ParNCEntitySets object from ncmesh (every rank contains
+          information to find every entity)
+     2) ParNCMesh is pruned which involves renumbering elements and vertices
+     3) ParMesh is initialized from ParNCMesh
+     4) ParNCMesh::OnMeshUpdated is called
+     5) Mesh::GenerateNCFaceInfo is called
+    */
+   {
+      int par_ref_levels = rp;
+      for (int l = 0; l < par_ref_levels; l++)
+      {
+         if ( myid == 0 ) { cout << "Uniform refinement in parallel..."; }
+         pmesh->UniformRefinement();
+      }
+      MPI_Barrier(MPI_COMM_WORLD);
+      if ( myid == 0 && rs > 0 ) { cout << "Done" << endl; }
+   }
+   /*
+     RandomRefinement will end up calling
+     ParMesh::NonconformingRefinement which will create a new ParMesh
+     object using the ParNCMesh object and then call
+     ParMesh::OnMeshUpdated on this new mesh.
+    */
+
+   for (int l = 0; l < ra; l++)
+   {
+      pmesh->RandomRefinement(0.2);
+   }
+   if ( ra > 0 )
+   {
+      if ( pmesh->pent_sets )
+      {
+         cout << "pmesh->pent_sets is non NULL post random refinement" << endl;
+         pmesh->pent_sets->PrintSetInfo(cout);
+      }
+      else
+      {
+         cout << "pmesh->pent_sets is NULL post random refinement" << endl;
+      }
+   }
+
+   // 6. Define a parallel finite element space on the parallel mesh. Here we
+   //    use continuous Lagrange finite elements of the specified order. If
+   //    order < 1, we instead use an isoparametric/isogeometric space.
+   FiniteElementCollection *fec;
+   if (order > 0)
+   {
+      fec = new H1_FECollection(order, dim);
+   }
+   else if (pmesh->GetNodes())
+   {
+      fec = pmesh->GetNodes()->OwnFEC();
+      if (myid == 0)
+      {
+         cout << "Using isoparametric FEs: " << fec->Name() << endl;
+      }
+   }
+   else
+   {
+      fec = new H1_FECollection(order = 1, dim);
+   }
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   HYPRE_Int size = fespace->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of finite element unknowns: " << size << endl;
+   }
+
+   // 7. Determine the list of true (i.e. parallel conforming) essential
+   //    boundary dofs. In this example, the boundary conditions are defined
+   //    by marking all the boundary attributes from the mesh as essential
+   //    (Dirichlet) and converting them to a list of true dofs.
+   Array<int> ess_tdof_list;
+   if ( bt == EntitySets::INVALID )
+   {
+      if (pmesh->bdr_attributes.Size())
+      {
+         Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+         ess_bdr = 1;
+         fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+      }
+   }
+   else
+   {
+      fespace->GetEssentialTrueDofs((EntitySets::EntityType)bt, bs,
+                                    ess_tdof_list);
+   }
+   for (int i=0; i<num_procs; i++)
+   {
+      if (myid == i)
+      {
+         cout << "Number of Dirichlet dofs on proc " << i << ": "
+              << ess_tdof_list.Size() << endl;
+      }
+   }
+
+   // 8. Set up the parallel linear form b(.) which corresponds to the
+   //    right-hand side of the FEM linear system, which in this case is
+   //    (1,phi_i) where phi_i are the basis functions in fespace.
+   ParLinearForm *b = new ParLinearForm(fespace);
+   ConstantCoefficient one(1.0);
+   b->AddDomainIntegrator(new DomainLFIntegrator(one));
+   b->Assemble();
+
+   // 9. Define the solution vector x as a parallel finite element grid function
+   //    corresponding to fespace. Initialize x with initial guess of zero,
+   //    which satisfies the boundary conditions.
+   ParGridFunction x(fespace);
+   x = 0.0;
+
+   // 10. Set up the parallel bilinear form a(.,.) on the finite element space
+   //     corresponding to the Laplacian operator -Delta, by adding the Diffusion
+   //     domain integrator.
+   ParBilinearForm *a = new ParBilinearForm(fespace);
+   a->AddDomainIntegrator(new DiffusionIntegrator(one));
+
+   // 11. Assemble the parallel bilinear form and the corresponding linear
+   //     system, applying any necessary transformations such as: parallel
+   //     assembly, eliminating boundary conditions, applying conforming
+   //     constraints for non-conforming AMR, static condensation, etc.
+   if (static_cond) { a->EnableStaticCondensation(); }
+   a->Assemble();
+
+   HypreParMatrix A;
+   Vector B, X;
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+
+   if (myid == 0)
+   {
+      cout << "Size of linear system: " << A.GetGlobalNumRows() << endl;
+   }
+
+   // 12. Define and apply a parallel PCG solver for AX=B with the BoomerAMG
+   //     preconditioner from hypre.
+   HypreSolver *amg = new HypreBoomerAMG(A);
+   HyprePCG *pcg = new HyprePCG(A);
+   pcg->SetTol(1e-12);
+   pcg->SetMaxIter(200);
+   pcg->SetPrintLevel(2);
+   pcg->SetPreconditioner(*amg);
+   pcg->Mult(B, X);
+
+   // 13. Recover the parallel grid function corresponding to X. This is the
+   //     local finite element solution on each processor.
+   a->RecoverFEMSolution(X, *b, x);
+
+   // 14. Save the refined mesh and the solution in parallel. This output can
+   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
+   {
+      ostringstream mesh_name, sol_name;
+      mesh_name << "mesh." << setfill('0') << setw(6) << myid;
+      sol_name << "sol." << setfill('0') << setw(6) << myid;
+
+      ofstream mesh_ofs(mesh_name.str().c_str());
+      mesh_ofs.precision(8);
+      pmesh->Print(mesh_ofs);
+
+      ofstream sol_ofs(sol_name.str().c_str());
+      sol_ofs.precision(8);
+      x.Save(sol_ofs);
+   }
+
+   // 15. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << x << flush;
+   }
+
+   // 16. Free the used memory.
+   delete pcg;
+   delete amg;
+   delete a;
+   delete b;
+   delete fespace;
+   if (order > 0) { delete fec; }
+   delete pmesh;
+
+   MPI_Finalize();
+
+   return 0;
+}
@@ -75,7 +75,7 @@ Mesh * GenerateSerialMesh(int ref);
 // alpha*n.Grad(sol) + beta*sol - gamma over the same boundary.
 double IntegrateBC(const GridFunction &sol, const Array<int> &bdr_marker,
                   double alpha, double beta, double gamma,
-                   double &error);
+                   double &err);

 int main(int argc, char *argv[])
 {
@@ -302,26 +302,26 @@ int main(int argc, char *argv[])
   {
      // Integrate the solution on the Dirichlet boundary and compare to the
      // expected value.
-      double error, avg = IntegrateBC(u, dbc_bdr, 0.0, 1.0, dbc_val, error);
+      double err, avg = IntegrateBC(u, dbc_bdr, 0.0, 1.0, dbc_val, err);

      bool hom_dbc = (dbc_val == 0.0);
-      error /=  hom_dbc ? 1.0 : fabs(dbc_val);
+      err /=  hom_dbc ? 1.0 : fabs(dbc_val);
      mfem::out << "Average of solution on Gamma_dbc:\t"
                << avg << ", \t"
                << (hom_dbc ? "absolute" : "relative")
-                << " error " << error << endl;
+                << " error " << err << endl;
   }
   {
      // Integrate n.Grad(u) on the inhomogeneous Neumann boundary and compare
      // to the expected value.
-      double error, avg = IntegrateBC(u, nbc_bdr, 1.0, 0.0, nbc_val, error);
+      double err, avg = IntegrateBC(u, nbc_bdr, 1.0, 0.0, nbc_val, err);

      bool hom_nbc = (nbc_val == 0.0);
-      error /=  hom_nbc ? 1.0 : fabs(nbc_val);
+      err /=  hom_nbc ? 1.0 : fabs(nbc_val);
      mfem::out << "Average of n.Grad(u) on Gamma_nbc:\t"
                << avg << ", \t"
                << (hom_nbc ? "absolute" : "relative")
-                << " error " << error << endl;
+                << " error " << err << endl;
   }
   {
      // Integrate n.Grad(u) on the homogeneous Neumann boundary and compare to
@@ -330,26 +330,25 @@ int main(int argc, char *argv[])
      nbc0_bdr = 0;
      nbc0_bdr[3] = 1;

-      double error, avg = IntegrateBC(u, nbc0_bdr, 1.0, 0.0, 0.0, error);
+      double err, avg = IntegrateBC(u, nbc0_bdr, 1.0, 0.0, 0.0, err);

      bool hom_nbc = true;
      mfem::out << "Average of n.Grad(u) on Gamma_nbc0:\t"
                << avg << ", \t"
                << (hom_nbc ? "absolute" : "relative")
-                << " error " << error << endl;
+                << " error " << err << endl;
   }
   {
      // Integrate n.Grad(u) + a * u on the Robin boundary and compare to the
      // expected value.
-      double error;
-      double avg = IntegrateBC(u, rbc_bdr, 1.0, rbc_a_val, rbc_b_val, error);
+      double err, avg = IntegrateBC(u, rbc_bdr, 1.0, rbc_a_val, rbc_b_val, err);

      bool hom_rbc = (rbc_b_val == 0.0);
-      error /=  hom_rbc ? 1.0 : fabs(rbc_b_val);
+      err /=  hom_rbc ? 1.0 : fabs(rbc_b_val);
      mfem::out << "Average of n.Grad(u)+a*u on Gamma_rbc:\t"
                << avg << ", \t"
                << (hom_rbc ? "absolute" : "relative")
-                << " error " << error << endl;
+                << " error " << err << endl;
   }

   // 14. Save the refined mesh and the solution. This output can be viewed
@@ -638,11 +637,11 @@ Mesh * GenerateSerialMesh(int ref)

 double IntegrateBC(const GridFunction &x, const Array<int> &bdr,
                   double alpha, double beta, double gamma,
-                   double &error)
+                   double &err)
 {
   double nrm = 0.0;
   double avg = 0.0;
-   error = 0.0;
+   err = 0.0;

   const bool a_is_zero = alpha == 0.0;
   const bool b_is_zero = beta == 0.0;
@@ -706,20 +705,20 @@ double IntegrateBC(const GridFunction &x, const Array<int> &bdr,

         // Integrate |alpha * n.Grad(x) + beta * x - gamma|^2
         val -= gamma;
-         error += (val*val) * ip.weight * face_weight;
+         err += (val*val) * ip.weight * face_weight;
      }
   }

   // Normalize by the length of the boundary
   if (std::abs(nrm) > 0.0)
   {
-      error /= nrm;
+      err /= nrm;
      avg /= nrm;
   }

   // Compute l2 norm of the error in the boundary condition (negative
-   // quadrature weights may produce negative 'error')
-   error = (error >= 0.0) ? sqrt(error) : -sqrt(-error);
+   // quadrature weights may produce negative 'err')
+   err = (err >= 0.0) ? sqrt(err) : -sqrt(-err);

   // Return the average value of alpha * n.Grad(x) + beta * x
   return avg;
@@ -75,7 +75,7 @@ Mesh * GenerateSerialMesh(int ref);
 // alpha*n.Grad(sol) + beta*sol - gamma over the same boundary.
 double IntegrateBC(const ParGridFunction &sol, const Array<int> &bdr_marker,
                   double alpha, double beta, double gamma,
-                   double &error);
+                   double &err);

 int main(int argc, char *argv[])
 {
@@ -321,26 +321,26 @@ int main(int argc, char *argv[])
   {
      // Integrate the solution on the Dirichlet boundary and compare to the
      // expected value.
-      double error, avg = IntegrateBC(u, dbc_bdr, 0.0, 1.0, dbc_val, error);
+      double err, avg = IntegrateBC(u, dbc_bdr, 0.0, 1.0, dbc_val, err);

      bool hom_dbc = (dbc_val == 0.0);
-      error /=  hom_dbc ? 1.0 : fabs(dbc_val);
+      err /=  hom_dbc ? 1.0 : fabs(dbc_val);
      mfem::out << "Average of solution on Gamma_dbc:\t"
                << avg << ", \t"
                << (hom_dbc ? "absolute" : "relative")
-                << " error " << error << endl;
+                << " error " << err << endl;
   }
   {
      // Integrate n.Grad(u) on the inhomogeneous Neumann boundary and compare
      // to the expected value.
-      double error, avg = IntegrateBC(u, nbc_bdr, 1.0, 0.0, nbc_val, error);
+      double err, avg = IntegrateBC(u, nbc_bdr, 1.0, 0.0, nbc_val, err);

      bool hom_nbc = (nbc_val == 0.0);
-      error /=  hom_nbc ? 1.0 : fabs(nbc_val);
+      err /=  hom_nbc ? 1.0 : fabs(nbc_val);
      mfem::out << "Average of n.Grad(u) on Gamma_nbc:\t"
                << avg << ", \t"
                << (hom_nbc ? "absolute" : "relative")
-                << " error " << error << endl;
+                << " error " << err << endl;
   }
   {
      // Integrate n.Grad(u) on the homogeneous Neumann boundary and compare to
@@ -349,26 +349,25 @@ int main(int argc, char *argv[])
      nbc0_bdr = 0;
      nbc0_bdr[3] = 1;

-      double error, avg = IntegrateBC(u, nbc0_bdr, 1.0, 0.0, 0.0, error);
+      double err, avg = IntegrateBC(u, nbc0_bdr, 1.0, 0.0, 0.0, err);

      bool hom_nbc = true;
      mfem::out << "Average of n.Grad(u) on Gamma_nbc0:\t"
                << avg << ", \t"
                << (hom_nbc ? "absolute" : "relative")
-                << " error " << error << endl;
+                << " error " << err << endl;
   }
   {
      // Integrate n.Grad(u) + a * u on the Robin boundary and compare to the
      // expected value.
-      double error, avg = IntegrateBC(u, rbc_bdr, 1.0, rbc_a_val, rbc_b_val,
-                                      error);
+      double err, avg = IntegrateBC(u, rbc_bdr, 1.0, rbc_a_val, rbc_b_val, err);

      bool hom_rbc = (rbc_b_val == 0.0);
-      error /=  hom_rbc ? 1.0 : fabs(rbc_b_val);
+      err /=  hom_rbc ? 1.0 : fabs(rbc_b_val);
      mfem::out << "Average of n.Grad(u)+a*u on Gamma_rbc:\t"
                << avg << ", \t"
                << (hom_rbc ? "absolute" : "relative")
-                << " error " << error << endl;
+                << " error " << err << endl;
   }

   // 15. Save the refined mesh and the solution in parallel. This output can be
@@ -668,11 +667,11 @@ double IntegrateBC(const ParGridFunction &x, const Array<int> &bdr,
   double loc_vals[3];
   double &nrm = loc_vals[0];
   double &avg = loc_vals[1];
-   double &error = loc_vals[2];
+   double &err = loc_vals[2];

   nrm = 0.0;
   avg = 0.0;
-   error = 0.0;
+   err = 0.0;

   const bool a_is_zero = alpha == 0.0;
   const bool b_is_zero = beta == 0.0;
@@ -736,7 +735,7 @@ double IntegrateBC(const ParGridFunction &x, const Array<int> &bdr,

         // Integrate |alpha * n.Grad(x) + beta * x - gamma|^2
         val -= gamma;
-         error += (val*val) * ip.weight * face_weight;
+         err += (val*val) * ip.weight * face_weight;
      }
   }

@@ -755,7 +754,7 @@ double IntegrateBC(const ParGridFunction &x, const Array<int> &bdr,
   }

   // Compute l2 norm of the error in the boundary condition (negative
-   // quadrature weights may produce negative 'error')
+   // quadrature weights may produce negative 'err')
   glb_err = (glb_err >= 0.0) ? sqrt(glb_err) : -sqrt(-glb_err);

   // Return the average value of alpha * n.Grad(x) + beta * x
@@ -81,10 +81,10 @@ Mesh * build_trapezoid_mesh(double offset)

 int main(int argc, char *argv[])
 {
-#ifdef HYPRE_USING_GPU
+#ifdef HYPRE_USING_CUDA
   cout << "\nAs of mfem-4.3 and hypre-2.22.0 (July 2021) this example\n"
-        << "is NOT supported with the GPU version of hypre.\n\n";
-   return 242;
+        << "is NOT supported with the CUDA version of hypre.\n\n";
+   return 255;
 #endif

   // 1. Initialize MPI.
@@ -167,9 +167,9 @@ int main(int argc, char *argv[])

   // 13. Compute error in the solution and its flux
   FunctionCoefficient uCoef(uExact);
-   double error = x.ComputeL2Error(uCoef);
+   double err = x.ComputeL2Error(uCoef);

-   cout << "|u - u_h|_2 = " << error << endl;
+   cout << "|u - u_h|_2 = " << err << endl;

   FiniteElementSpace flux_fespace(mesh, &fec, 3);
   GridFunction flux(&flux_fespace);
@@ -197,9 +197,9 @@ int main(int argc, char *argv[])

   // 15. Compute error in the solution and its flux
   FunctionCoefficient uCoef(uExact);
-   double error = x.ComputeL2Error(uCoef);
+   double err = x.ComputeL2Error(uCoef);

-   if (myid == 0) { cout << "|u - u_h|_2 = " << error << endl; }
+   if (myid == 0) { cout << "|u - u_h|_2 = " << err << endl; }

   ParFiniteElementSpace flux_fespace(&pmesh, &fec, 3);
   ParGridFunction flux(&flux_fespace);
@@ -255,10 +255,10 @@ int main(int argc, char *argv[])

   // 15. Compute and print the L^2 norm of the error.
   {
-      double error = x.ComputeL2Error(E);
+      double err = x.ComputeL2Error(E);
      if (myid == 0)
      {
-         cout << "\n|| E_h - E ||_{L^2} = " << error << '\n' << endl;
+         cout << "\n|| E_h - E ||_{L^2} = " << err << '\n' << endl;
      }
   }

@@ -0,0 +1,411 @@
+//                       MFEM Example 3 - Parallel Version
+//
+// Compile with: make ex3p
+//
+// Sample runs:  mpirun -np 4 ex3p -m ../data/star.mesh
+//               mpirun -np 4 ex3p -m ../data/square-disc.mesh -o 2
+//               mpirun -np 4 ex3p -m ../data/beam-tet.mesh
+//               mpirun -np 4 ex3p -m ../data/beam-hex.mesh
+//               mpirun -np 4 ex3p -m ../data/escher.mesh
+//               mpirun -np 4 ex3p -m ../data/fichera.mesh
+//               mpirun -np 4 ex3p -m ../data/fichera-q2.vtk
+//               mpirun -np 4 ex3p -m ../data/fichera-q3.mesh
+//               mpirun -np 4 ex3p -m ../data/square-disc-nurbs.mesh
+//               mpirun -np 4 ex3p -m ../data/beam-hex-nurbs.mesh
+//               mpirun -np 4 ex3p -m ../data/amr-quad.mesh -o 2
+//               mpirun -np 4 ex3p -m ../data/amr-hex.mesh
+//               mpirun -np 4 ex3p -m ../data/star-surf.mesh -o 2
+//               mpirun -np 4 ex3p -m ../data/mobius-strip.mesh -o 2 -f 0.1
+//               mpirun -np 4 ex3p -m ../data/klein-bottle.mesh -o 2 -f 0.1
+//
+// Description:  This example code solves a simple electromagnetic diffusion
+//               problem corresponding to the second order definite Maxwell
+//               equation curl curl E + E = f with boundary condition
+//               E x n = <given tangential field>. Here, we use a given exact
+//               solution E and compute the corresponding r.h.s. f.
+//               We discretize with Nedelec finite elements in 2D or 3D.
+//
+//               The example demonstrates the use of H(curl) finite element
+//               spaces with the curl-curl and the (vector finite element) mass
+//               bilinear form, as well as the computation of discretization
+//               error when the exact solution is known. Static condensation is
+//               also illustrated.
+//
+//               We recommend viewing examples 1-2 before viewing this example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Exact solution, E, and r.h.s., f. See below for implementation.
+//void E_exact(const Vector &, Vector &);
+//void f_exact(const Vector &, Vector &);
+//double freq = 1.0, kappa;
+void f_const(const Vector &, Vector &);
+
+int dim;
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   const char *mesh_file = "../data/beam-tet.mesh";
+   int order = 1;
+   int rs = -1;
+   int rp = 2;
+   int ra = 0;
+   int bt = EntitySets::INVALID;
+   const char *bs = "Origin";
+   bool static_cond = false;
+   bool visualization = 1;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree).");
+   /*
+   args.AddOption(&freq, "-f", "--frequency", "Set the frequency for the exact"
+                  " solution.");
+   */
+   args.AddOption(&rs, "-rs", "--refine-serial",
+                  "Number of serial refinement levels");
+   args.AddOption(&rp, "-rp", "--refine-parallel",
+                  "Number of parallel refinement levels");
+   args.AddOption(&ra, "-ra", "--refine-adaptive",
+                  "Number of adaptive refinement levels");
+   args.AddOption(&bt, "-bt", "--bc-entity-type",
+                  "");
+   args.AddOption(&bs, "-bs", "--bc-entity-set-name",
+                  "");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+   // kappa = freq * M_PI;
+
+   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
+   //    and volume meshes with the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 4. Refine the serial mesh on all processors to increase the resolution. In
+   //    this example we do 'ref_levels' of uniform refinement. We choose
+   //    'ref_levels' to be the largest number that gives a final mesh with no
+   //    more than 1,000 elements.
+   {
+      int ref_levels = ( rs >= 0 ) ? rs :
+                       (int)floor(log(1000./mesh->GetNE())/log(2.)/dim);
+      for (int l = 0; l < ref_levels; l++)
+      {
+         if ( myid == 0 ) { cout << "Uniform refinement in serial..."; }
+         mesh->UniformRefinement();
+      }
+      MPI_Barrier(MPI_COMM_WORLD);
+      if ( myid == 0 && rs > 0 ) { cout << "Done" << endl; }
+   }
+   if ( mesh->ent_sets )
+   {
+      cout << "mesh->ent_sets is non NULL" << endl;
+      mesh->ent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "mesh->ent_sets is NULL" << endl;
+   }
+   /*
+     At this point we have a serial mesh containing an EntitySets
+     object which stores the current node/edge/face/element indices
+     for each entity in each set.  This data is duplicated on each MPI
+     rank.
+    */
+   if ( ra > 0 )
+   {
+      cout << "calling EnsureNCMesh" << endl;
+      mesh->EnsureNCMesh();
+      cout << "back from EnsureNCMesh" << endl;
+   }
+   if ( mesh->ent_sets )
+   {
+      cout << "mesh->ent_sets is non NULL" << endl;
+      mesh->ent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "mesh->ent_sets is NULL" << endl;
+   }
+   /*
+     We now have an NCEntitySets object which stores the node indices
+     describing each enity in each node/edge/face set and the element
+     indices for the elements in each element set.  This data is
+     duplicated on each MPI rank.
+    */
+
+   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted. Tetrahedral
+   //    meshes need to be reoriented before we can define high-order Nedelec
+   //    spaces on them.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   if ( pmesh->pent_sets )
+   {
+      cout << "pmesh->pent_sets is non NULL" << endl;
+      pmesh->pent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "pmesh->pent_sets is NULL" << endl;
+   }
+   {
+      int par_ref_levels = rp;
+      for (int l = 0; l < par_ref_levels; l++)
+      {
+         if ( myid == 0 ) { cout << "Uniform refinement in parallel..."; }
+         pmesh->UniformRefinement();
+      }
+      MPI_Barrier(MPI_COMM_WORLD);
+      if ( myid == 0 && rs > 0 ) { cout << "Done" << endl; }
+   }
+   pmesh->ReorientTetMesh();
+   pmesh->ent_sets->PrintSetInfo(cout);
+
+   for (int l = 0; l < ra; l++)
+   {
+      pmesh->RandomRefinement(0.2);
+   }
+   if ( ra > 0 )
+   {
+      if ( pmesh->pent_sets )
+      {
+         cout << "pmesh->pent_sets is non NULL post random refinement" << endl;
+         pmesh->pent_sets->PrintSetInfo(cout);
+      }
+      else
+      {
+         cout << "pmesh->pent_sets is NULL post random refinement" << endl;
+      }
+   }
+
+   // 6. Define a parallel finite element space on the parallel mesh. Here we
+   //    use the Nedelec finite elements of the specified order.
+   FiniteElementCollection *fec = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   HYPRE_Int size = fespace->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of finite element unknowns: " << size << endl;
+   }
+
+   // 7. Determine the list of true (i.e. parallel conforming) essential
+   //    boundary dofs. In this example, the boundary conditions are defined
+   //    by marking all the boundary attributes from the mesh as essential
+   //    (Dirichlet) and converting them to a list of true dofs.
+   Array<int> ess_tdof_list;
+   if ( bt == EntitySets::INVALID )
+   {
+      if (pmesh->bdr_attributes.Size())
+      {
+         Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+         ess_bdr = 1;
+         fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+      }
+   }
+   else
+   {
+      fespace->GetEssentialTrueDofs((EntitySets::EntityType)bt, bs,
+                                    ess_tdof_list);
+   }
+   if (myid == 0)
+   {
+      cout << "Number of Dirichlet dofs: " << ess_tdof_list.Size() << endl;
+   }
+
+
+   // 8. Set up the parallel linear form b(.) which corresponds to the
+   //    right-hand side of the FEM linear system, which in this case is
+   //    (f,phi_i) where f is given by the function f_exact and phi_i are the
+   //    basis functions in the finite element fespace.
+   VectorFunctionCoefficient f(sdim, f_const);
+   ParLinearForm *b = new ParLinearForm(fespace);
+   b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
+   b->Assemble();
+
+   // 9. Define the solution vector x as a parallel finite element grid function
+   //    corresponding to fespace. Initialize x by projecting the exact
+   //    solution. Note that only values from the boundary edges will be used
+   //    when eliminating the non-homogeneous boundary condition to modify the
+   //    r.h.s. vector b.
+   ParGridFunction x(fespace);
+   // VectorFunctionCoefficient E(sdim, E_exact);
+   // x.ProjectCoefficient(E);
+   x = 0.0;
+
+   // 10. Set up the parallel bilinear form corresponding to the EM diffusion
+   //     operator curl muinv curl + sigma I, by adding the curl-curl and the
+   //     mass domain integrators.
+   Coefficient *muinv = new ConstantCoefficient(1.0);
+   Coefficient *sigma = new ConstantCoefficient(1.0);
+   ParBilinearForm *a = new ParBilinearForm(fespace);
+   a->AddDomainIntegrator(new CurlCurlIntegrator(*muinv));
+   a->AddDomainIntegrator(new VectorFEMassIntegrator(*sigma));
+
+   // 11. Assemble the parallel bilinear form and the corresponding linear
+   //     system, applying any necessary transformations such as: parallel
+   //     assembly, eliminating boundary conditions, applying conforming
+   //     constraints for non-conforming AMR, static condensation, etc.
+   if (static_cond) { a->EnableStaticCondensation(); }
+   a->Assemble();
+
+   HypreParMatrix A;
+   Vector B, X;
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+
+   if (myid == 0)
+   {
+      cout << "Size of linear system: " << A.GetGlobalNumRows() << endl;
+   }
+
+   // 12. Define and apply a parallel PCG solver for AX=B with the AMS
+   //     preconditioner from hypre.
+   ParFiniteElementSpace *prec_fespace =
+      (a->StaticCondensationIsEnabled() ? a->SCParFESpace() : fespace);
+   HypreSolver *ams = new HypreAMS(A, prec_fespace);
+   HyprePCG *pcg = new HyprePCG(A);
+   pcg->SetTol(1e-12);
+   pcg->SetMaxIter(500);
+   pcg->SetPrintLevel(2);
+   pcg->SetPreconditioner(*ams);
+   pcg->Mult(B, X);
+
+   // 13. Recover the parallel grid function corresponding to X. This is the
+   //     local finite element solution on each processor.
+   a->RecoverFEMSolution(X, *b, x);
+
+   /*
+   // 14. Compute and print the L^2 norm of the error.
+   {
+      double err = x.ComputeL2Error(E);
+      if (myid == 0)
+      {
+         cout << "\n|| E_h - E ||_{L^2} = " << err << '\n' << endl;
+      }
+   }
+   */
+   // 15. Save the refined mesh and the solution in parallel. This output can
+   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
+   {
+      ostringstream mesh_name, sol_name;
+      mesh_name << "mesh." << setfill('0') << setw(6) << myid;
+      sol_name << "sol." << setfill('0') << setw(6) << myid;
+
+      ofstream mesh_ofs(mesh_name.str().c_str());
+      mesh_ofs.precision(8);
+      pmesh->Print(mesh_ofs);
+
+      ofstream sol_ofs(sol_name.str().c_str());
+      sol_ofs.precision(8);
+      x.Save(sol_ofs);
+   }
+
+   // 16. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << x << flush;
+   }
+
+   // 17. Free the used memory.
+   delete pcg;
+   delete ams;
+   delete a;
+   delete sigma;
+   delete muinv;
+   delete b;
+   delete fespace;
+   delete fec;
+   delete pmesh;
+
+   MPI_Finalize();
+
+   return 0;
+}
+
+/*
+void E_exact(const Vector &x, Vector &E)
+{
+   if (dim == 3)
+   {
+      E(0) = sin(kappa * x(1));
+      E(1) = sin(kappa * x(2));
+      E(2) = sin(kappa * x(0));
+   }
+   else
+   {
+      E(0) = sin(kappa * x(1));
+      E(1) = sin(kappa * x(0));
+      if (x.Size() == 3) { E(2) = 0.0; }
+   }
+}
+
+void f_exact(const Vector &x, Vector &f)
+{
+   if (dim == 3)
+   {
+      f(0) = (1. + kappa * kappa) * sin(kappa * x(1));
+      f(1) = (1. + kappa * kappa) * sin(kappa * x(2));
+      f(2) = (1. + kappa * kappa) * sin(kappa * x(0));
+   }
+   else
+   {
+      f(0) = (1. + kappa * kappa) * sin(kappa * x(1));
+      f(1) = (1. + kappa * kappa) * sin(kappa * x(0));
+      if (x.Size() == 3) { f(2) = 0.0; }
+   }
+}
+*/
+void f_const(const Vector &x, Vector &f)
+{
+   if (dim == 3)
+   {
+      f(0) = 1.0;
+      f(1) = 1.0;
+      f(2) = 1.0;
+   }
+   else
+   {
+      f(0) = 1.0;
+      f(1) = 1.0;
+      if (x.Size() == 3) { f(2) = 0.0; }
+   }
+}
@@ -256,10 +256,10 @@ int main(int argc, char *argv[])

   // 15. Compute and print the L^2 norm of the error.
   {
-      double error = x.ComputeL2Error(F);
+      double err = x.ComputeL2Error(F);
      if (myid == 0)
      {
-         cout << "\n|| F_h - F ||_{L^2} = " << error << '\n' << endl;
+         cout << "\n|| F_h - F ||_{L^2} = " << err << '\n' << endl;
      }
   }

@@ -0,0 +1,438 @@
+//                       MFEM Example 4 - Parallel Version
+//
+// Compile with: make ex4p
+//
+// Sample runs:  mpirun -np 4 ex4p -m ../data/square-disc.mesh
+//               mpirun -np 4 ex4p -m ../data/star.mesh
+//               mpirun -np 4 ex4p -m ../data/beam-tet.mesh
+//               mpirun -np 4 ex4p -m ../data/beam-hex.mesh
+//               mpirun -np 4 ex4p -m ../data/escher.mesh -o 2 -sc
+//               mpirun -np 4 ex4p -m ../data/fichera.mesh -o 2 -hb
+//               mpirun -np 4 ex4p -m ../data/fichera-q2.vtk
+//               mpirun -np 4 ex4p -m ../data/fichera-q3.mesh -o 2 -sc
+//               mpirun -np 4 ex4p -m ../data/square-disc-nurbs.mesh -o 3
+//               mpirun -np 4 ex4p -m ../data/beam-hex-nurbs.mesh -o 3
+//               mpirun -np 4 ex4p -m ../data/periodic-square.mesh -no-bc
+//               mpirun -np 4 ex4p -m ../data/periodic-cube.mesh -no-bc
+//               mpirun -np 4 ex4p -m ../data/amr-quad.mesh
+//               mpirun -np 4 ex4p -m ../data/amr-hex.mesh -o 2 -sc
+//               mpirun -np 4 ex4p -m ../data/amr-hex.mesh -o 2 -hb
+//               mpirun -np 4 ex4p -m ../data/star-surf.mesh -o 3 -hb
+//
+// Description:  This example code solves a simple 2D/3D H(div) diffusion
+//               problem corresponding to the second order definite equation
+//               -grad(alpha div F) + beta F = f with boundary condition F dot n
+//               = <given normal field>. Here, we use a given exact solution F
+//               and compute the corresponding r.h.s. f.  We discretize with
+//               Raviart-Thomas finite elements.
+//
+//               The example demonstrates the use of H(div) finite element
+//               spaces with the grad-div and H(div) vector finite element mass
+//               bilinear form, as well as the computation of discretization
+//               error when the exact solution is known. Bilinear form
+//               hybridization and static condensation are also illustrated.
+//
+//               We recommend viewing examples 1-3 before viewing this example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Exact solution, F, and r.h.s., f. See below for implementation.
+//void F_exact(const Vector &, Vector &);
+//void f_exact(const Vector &, Vector &);
+//double freq = 1.0, kappa;
+void f_const(const Vector &, Vector &);
+
+int dim;
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   const char *mesh_file = "../data/star.mesh";
+   int order = 1;
+   int rs = -1;
+   int rp = 2;
+   int ra = 0;
+   int bt = EntitySets::INVALID;
+   const char *bs = "Origin";
+   bool set_bc = true;
+   bool static_cond = false;
+   bool hybridization = false;
+   bool visualization = 1;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree).");
+   args.AddOption(&set_bc, "-bc", "--impose-bc", "-no-bc", "--dont-impose-bc",
+                  "Impose or not essential boundary conditions.");
+   args.AddOption(&rs, "-rs", "--refine-serial",
+                  "Number of serial refinement levels");
+   args.AddOption(&rp, "-rp", "--refine-parallel",
+                  "Number of parallel refinement levels");
+   args.AddOption(&ra, "-ra", "--refine-adaptive",
+                  "Number of adaptive refinement levels");
+   args.AddOption(&bt, "-bt", "--bc-entity-type",
+                  "");
+   args.AddOption(&bs, "-bs", "--bc-entity-set-name",
+                  "");
+   // args.AddOption(&freq, "-f", "--frequency", "Set the frequency for the exact"
+   //                 " solution.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&hybridization, "-hb", "--hybridization", "-no-hb",
+                  "--no-hybridization", "Enable hybridization.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+   // kappa = freq * M_PI;
+
+   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
+   //    and volume, as well as periodic meshes with the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 4. Refine the serial mesh on all processors to increase the resolution. In
+   //    this example we do 'ref_levels' of uniform refinement. We choose
+   //    'ref_levels' to be the largest number that gives a final mesh with no
+   //    more than 1,000 elements.
+   {
+      int ref_levels = ( rs >= 0 ) ? rs :
+                       (int)floor(log(1000./mesh->GetNE())/log(2.)/dim);
+      for (int l = 0; l < ref_levels; l++)
+      {
+         if ( myid == 0 ) { cout << "Uniform refinement in serial..."; }
+         mesh->UniformRefinement();
+      }
+      MPI_Barrier(MPI_COMM_WORLD);
+      if ( myid == 0 && rs > 0 ) { cout << "Done" << endl; }
+   }
+   if ( mesh->ent_sets )
+   {
+      cout << "mesh->ent_sets is non NULL" << endl;
+      mesh->ent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "mesh->ent_sets is NULL" << endl;
+   }
+   /*
+     At this point we have a serial mesh containing an EntitySets
+     object which stores the current node/edge/face/element indices
+     for each entity in each set.  This data is duplicated on each MPI
+     rank.
+    */
+   if ( ra > 0 )
+   {
+      cout << "calling EnsureNCMesh" << endl;
+      mesh->EnsureNCMesh();
+      cout << "back from EnsureNCMesh" << endl;
+   }
+   if ( mesh->ent_sets )
+   {
+      cout << "mesh->ent_sets is non NULL" << endl;
+      mesh->ent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "mesh->ent_sets is NULL" << endl;
+   }
+   /*
+     We now have an NCEntitySets object which stores the node indices
+     describing each enity in each node/edge/face set and the element
+     indices for the elements in each element set.  This data is
+     duplicated on each MPI rank.
+    */
+
+   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted. Tetrahedral
+   //    meshes need to be reoriented before we can define high-order Nedelec
+   //    spaces on them (this is needed in the ADS solver below).
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   if ( pmesh->pent_sets )
+   {
+      cout << "pmesh->pent_sets is non NULL" << endl;
+      pmesh->pent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "pmesh->pent_sets is NULL" << endl;
+   }
+   {
+      int par_ref_levels = rp;
+      for (int l = 0; l < par_ref_levels; l++)
+      {
+         if ( myid == 0 ) { cout << "Uniform refinement in parallel..."; }
+         pmesh->UniformRefinement();
+      }
+   }
+   pmesh->ReorientTetMesh();
+
+   for (int l = 0; l < ra; l++)
+   {
+      pmesh->RandomRefinement(0.2);
+   }
+   if ( ra > 0 )
+   {
+      if ( pmesh->pent_sets )
+      {
+         cout << "pmesh->pent_sets is non NULL post random refinement" << endl;
+         pmesh->pent_sets->PrintSetInfo(cout);
+      }
+      else
+      {
+         cout << "pmesh->pent_sets is NULL post random refinement" << endl;
+      }
+   }
+
+   // 6. Define a parallel finite element space on the parallel mesh. Here we
+   //    use the Raviart-Thomas finite elements of the specified order.
+   FiniteElementCollection *fec = new RT_FECollection(order-1, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   HYPRE_Int size = fespace->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of finite element unknowns: " << size << endl;
+   }
+
+   // 7. Determine the list of true (i.e. parallel conforming) essential
+   //    boundary dofs. In this example, the boundary conditions are defined
+   //    by marking all the boundary attributes from the mesh as essential
+   //    (Dirichlet) and converting them to a list of true dofs.
+   Array<int> ess_tdof_list;
+   if ( bt == EntitySets::INVALID )
+   {
+      if (pmesh->bdr_attributes.Size())
+      {
+         Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+         ess_bdr = set_bc ? 1 : 0;
+         fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+      }
+   }
+   else
+   {
+      fespace->GetEssentialTrueDofs((EntitySets::EntityType)bt, bs,
+                                    ess_tdof_list);
+   }
+   if (myid == 0)
+   {
+      cout << "Number of Dirichlet dofs: " << ess_tdof_list.Size() << endl;
+   }
+
+   // 8. Set up the parallel linear form b(.) which corresponds to the
+   //    right-hand side of the FEM linear system, which in this case is
+   //    (f,phi_i) where f is given by the function f_exact and phi_i are the
+   //    basis functions in the finite element fespace.
+   VectorFunctionCoefficient f(sdim, f_const);
+   ParLinearForm *b = new ParLinearForm(fespace);
+   b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
+   b->Assemble();
+
+   // 9. Define the solution vector x as a parallel finite element grid function
+   //    corresponding to fespace. Initialize x by projecting the exact
+   //    solution. Note that only values from the boundary faces will be used
+   //    when eliminating the non-homogeneous boundary condition to modify the
+   //    r.h.s. vector b.
+   ParGridFunction x(fespace);
+   // VectorFunctionCoefficient F(sdim, F_exact);
+   // x.ProjectCoefficient(F);
+   x = 0.0;
+
+   // 10. Set up the parallel bilinear form corresponding to the H(div)
+   //     diffusion operator grad alpha div + beta I, by adding the div-div and
+   //     the mass domain integrators.
+   Coefficient *alpha = new ConstantCoefficient(1.0);
+   Coefficient *beta  = new ConstantCoefficient(1.0);
+   ParBilinearForm *a = new ParBilinearForm(fespace);
+   a->AddDomainIntegrator(new DivDivIntegrator(*alpha));
+   a->AddDomainIntegrator(new VectorFEMassIntegrator(*beta));
+
+   // 11. Assemble the parallel bilinear form and the corresponding linear
+   //     system, applying any necessary transformations such as: parallel
+   //     assembly, eliminating boundary conditions, applying conforming
+   //     constraints for non-conforming AMR, static condensation,
+   //     hybridization, etc.
+   FiniteElementCollection *hfec = NULL;
+   ParFiniteElementSpace *hfes = NULL;
+   if (static_cond)
+   {
+      a->EnableStaticCondensation();
+   }
+   else if (hybridization)
+   {
+      hfec = new DG_Interface_FECollection(order-1, dim);
+      hfes = new ParFiniteElementSpace(pmesh, hfec);
+      a->EnableHybridization(hfes, new NormalTraceJumpIntegrator(),
+                             ess_tdof_list);
+   }
+   a->Assemble();
+
+   HypreParMatrix A;
+   Vector B, X;
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+
+   HYPRE_Int glob_size = A.GetGlobalNumRows();
+   if (myid == 0)
+   {
+      cout << "Size of linear system: " << glob_size << endl;
+   }
+
+   // 12. Define and apply a parallel PCG solver for A X = B with the 2D AMS or
+   //     the 3D ADS preconditioners from hypre. If using hybridization, the
+   //     system is preconditioned with hypre's BoomerAMG.
+   HypreSolver *prec = NULL;
+   CGSolver *pcg = new CGSolver(A.GetComm());
+   pcg->SetOperator(A);
+   pcg->SetRelTol(1e-12);
+   pcg->SetMaxIter(500);
+   pcg->SetPrintLevel(1);
+   if (hybridization) { prec = new HypreBoomerAMG(A); }
+   else
+   {
+      ParFiniteElementSpace *prec_fespace =
+         (a->StaticCondensationIsEnabled() ? a->SCParFESpace() : fespace);
+      if (dim == 2)   { prec = new HypreAMS(A, prec_fespace); }
+      else            { prec = new HypreADS(A, prec_fespace); }
+   }
+   pcg->SetPreconditioner(*prec);
+   pcg->Mult(B, X);
+
+   // 13. Recover the parallel grid function corresponding to X. This is the
+   //     local finite element solution on each processor.
+   a->RecoverFEMSolution(X, *b, x);
+   /*
+   // 14. Compute and print the L^2 norm of the error.
+   {
+      double err = x.ComputeL2Error(F);
+      if (myid == 0)
+      {
+         cout << "\n|| F_h - F ||_{L^2} = " << err << '\n' << endl;
+      }
+   }
+   */
+   // 15. Save the refined mesh and the solution in parallel. This output can
+   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
+   {
+      ostringstream mesh_name, sol_name;
+      mesh_name << "mesh." << setfill('0') << setw(6) << myid;
+      sol_name << "sol." << setfill('0') << setw(6) << myid;
+
+      ofstream mesh_ofs(mesh_name.str().c_str());
+      mesh_ofs.precision(8);
+      pmesh->Print(mesh_ofs);
+
+      ofstream sol_ofs(sol_name.str().c_str());
+      sol_ofs.precision(8);
+      x.Save(sol_ofs);
+   }
+
+   // 16. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << x << flush;
+   }
+
+   // 17. Free the used memory.
+   delete pcg;
+   delete prec;
+   delete hfes;
+   delete hfec;
+   delete a;
+   delete alpha;
+   delete beta;
+   delete b;
+   delete fespace;
+   delete fec;
+   delete pmesh;
+
+   MPI_Finalize();
+
+   return 0;
+}
+
+/*
+// The exact solution (for non-surface meshes)
+void F_exact(const Vector &p, Vector &F)
+{
+   int dim = p.Size();
+
+   double x = p(0);
+   double y = p(1);
+   // double z = (dim == 3) ? p(2) : 0.0;
+
+   F(0) = cos(kappa*x)*sin(kappa*y);
+   F(1) = cos(kappa*y)*sin(kappa*x);
+   if (dim == 3)
+   {
+      F(2) = 0.0;
+   }
+}
+
+// The right hand side
+void f_exact(const Vector &p, Vector &f)
+{
+   int dim = p.Size();
+
+   double x = p(0);
+   double y = p(1);
+   // double z = (dim == 3) ? p(2) : 0.0;
+
+   double temp = 1 + 2*kappa*kappa;
+
+   f(0) = temp*cos(kappa*x)*sin(kappa*y);
+   f(1) = temp*cos(kappa*y)*sin(kappa*x);
+   if (dim == 3)
+   {
+      f(2) = 0;
+   }
+}
+*/
+void f_const(const Vector &x, Vector &f)
+{
+   if (dim == 3)
+   {
+      f(0) = 1.0;
+      f(1) = 1.0;
+      f(2) = 1.0;
+   }
+   else
+   {
+      f(0) = 1.0;
+      f(1) = 1.0;
+      if (x.Size() == 3) { f(2) = 0.0; }
+   }
+}
@@ -197,7 +197,7 @@ int main(int argc, char *argv[])
      SparseMatrix &M(mVarf->SpMat());
      SparseMatrix &B(bVarf->SpMat());
      B *= -1.;
-      B.EnsureMultTranspose();
+      if (Device::IsEnabled()) { B.BuildTranspose(); }
      Bt = new TransposeOperator(&B);

      darcyOp.SetBlock(0,0, &M);
@@ -0,0 +1,325 @@
+//                       MFEM Example 6 - Parallel Version
+//
+// Compile with: make ex6p
+//
+// Sample runs:  mpirun -np 4 ex6p -m ../data/square-disc.mesh -o 1
+//               mpirun -np 4 ex6p -m ../data/square-disc.mesh -o 2
+//               mpirun -np 4 ex6p -m ../data/square-disc-nurbs.mesh -o 2
+//               mpirun -np 4 ex6p -m ../data/star.mesh -o 3
+//               mpirun -np 4 ex6p -m ../data/escher.mesh -o 2
+//               mpirun -np 4 ex6p -m ../data/fichera.mesh -o 2
+//               mpirun -np 4 ex6p -m ../data/disc-nurbs.mesh -o 2
+//               mpirun -np 4 ex6p -m ../data/ball-nurbs.mesh
+//               mpirun -np 4 ex6p -m ../data/pipe-nurbs.mesh
+//               mpirun -np 4 ex6p -m ../data/star-surf.mesh -o 2
+//               mpirun -np 4 ex6p -m ../data/square-disc-surf.mesh -o 2
+//               mpirun -np 4 ex6p -m ../data/amr-quad.mesh
+//
+// Description:  This is a version of Example 1 with a simple adaptive mesh
+//               refinement loop. The problem being solved is again the Laplace
+//               equation -Delta u = 1 with homogeneous Dirichlet boundary
+//               conditions. The problem is solved on a sequence of meshes which
+//               are locally refined in a conforming (triangles, tetrahedrons)
+//               or non-conforming (quadrilaterals, hexahedra) manner according
+//               to a simple ZZ error estimator.
+//
+//               The example demonstrates MFEM's capability to work with both
+//               conforming and nonconforming refinements, in 2D and 3D, on
+//               linear, curved and surface meshes. Interpolation of functions
+//               from coarse to fine meshes, as well as persistent GLVis
+//               visualization are also illustrated.
+//
+//               We recommend viewing Example 1 before viewing this example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+static int max_dofs = 100000;
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   const char *mesh_file = "./star-set.mesh";
+   int order = 1;
+   int bt = EntitySets::INVALID;
+   const char *bs = "";
+   bool visualization = true;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree).");
+   args.AddOption(&max_dofs, "-md", "--max-dofs",
+                  "Maximum number of degrees of freedom.");
+   args.AddOption(&bt, "-bt", "--bc-entity-type",
+                  "");
+   args.AddOption(&bs, "-bs", "--bc-entity-set-name",
+                  "");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
+   //    and volume meshes with the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 4. Refine the serial mesh on all processors to increase the resolution.
+   //    Also project a NURBS mesh to a piecewise-quadratic curved mesh. Make
+   //    sure that the mesh is non-conforming.
+   if (mesh->NURBSext)
+   {
+      mesh->UniformRefinement();
+      mesh->SetCurvature(2);
+   }
+   mesh->EnsureNCMesh();
+   if ( mesh->ent_sets )
+   {
+      cout << "mesh->ent_sets is non NULL" << endl;
+      mesh->ent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "mesh->ent_sets is NULL" << endl;
+   }
+
+   // 5. Define a parallel mesh by partitioning the serial mesh.
+   //    Once the parallel mesh is defined, the serial mesh can be deleted.
+   ParMesh pmesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   if ( pmesh.pent_sets )
+   {
+      cout << "pmesh->pent_sets is non NULL" << endl;
+      pmesh.pent_sets->PrintSetInfo(cout);
+   }
+   else
+   {
+      cout << "pmesh->pent_sets is NULL" << endl;
+   }
+
+   // 6. Define a finite element space on the mesh. The polynomial order is
+   //    one (linear) by default, but this can be changed on the command line.
+   H1_FECollection fec(order, dim);
+   ParFiniteElementSpace fespace(&pmesh, &fec);
+
+   Array<int> ess_tdof_list;
+   if ( bt == EntitySets::INVALID )
+   {
+      if (pmesh.bdr_attributes.Size())
+      {
+         Array<int> ess_bdr(pmesh.bdr_attributes.Max());
+         ess_bdr = 1;
+         fespace.GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+      }
+   }
+   else
+   {
+      fespace.GetEssentialTrueDofs((EntitySets::EntityType)bt, bs,
+                                   ess_tdof_list);
+   }
+
+   // 7. As in Example 1p, we set up bilinear and linear forms corresponding to
+   //    the Laplace problem -\Delta u = 1. We don't assemble the discrete
+   //    problem yet, this will be done in the main loop.
+   ParBilinearForm a(&fespace);
+   ParLinearForm b(&fespace);
+
+   ConstantCoefficient one(1.0);
+
+   BilinearFormIntegrator *integ = new DiffusionIntegrator(one);
+   a.AddDomainIntegrator(integ);
+   b.AddDomainIntegrator(new DomainLFIntegrator(one));
+
+   // 8. The solution vector x and the associated finite element grid function
+   //    will be maintained over the AMR iterations. We initialize it to zero.
+   ParGridFunction x(&fespace);
+   x = 0;
+
+   // 9. Connect to GLVis.
+   char vishost[] = "localhost";
+   int  visport   = 19916;
+
+   socketstream sout;
+   if (visualization)
+   {
+      sout.open(vishost, visport);
+      if (!sout)
+      {
+         if (myid == 0)
+         {
+            cout << "Unable to connect to GLVis server at "
+                 << vishost << ':' << visport << endl;
+            cout << "GLVis visualization disabled.\n";
+         }
+         visualization = false;
+      }
+
+      sout.precision(8);
+   }
+
+   // 10. Set up an error estimator. Here we use the Zienkiewicz-Zhu estimator
+   //     with L2 projection in the smoothing step to better handle hanging
+   //     nodes and parallel partitioning. We need to supply a space for the
+   //     discontinuous flux (L2) and a space for the smoothed flux (H(div) is
+   //     used here).
+   L2_FECollection flux_fec(order, dim);
+   ParFiniteElementSpace flux_fes(&pmesh, &flux_fec, sdim);
+   RT_FECollection smooth_flux_fec(order-1, dim);
+   ParFiniteElementSpace smooth_flux_fes(&pmesh, &smooth_flux_fec);
+   // Another possible option for the smoothed flux space:
+   // H1_FECollection smooth_flux_fec(order, dim);
+   // ParFiniteElementSpace smooth_flux_fes(&pmesh, &smooth_flux_fec, dim);
+   L2ZienkiewiczZhuEstimator estimator(*integ, x, flux_fes, smooth_flux_fes);
+
+   // 11. A refiner selects and refines elements based on a refinement strategy.
+   //     The strategy here is to refine elements with errors larger than a
+   //     fraction of the maximum element error. Other strategies are possible.
+   //     The refiner will call the given error estimator.
+   ThresholdRefiner refiner(estimator);
+   refiner.SetTotalErrorFraction(0.7);
+
+   // 12. The main AMR loop. In each iteration we solve the problem on the
+   //     current mesh, visualize the solution, and refine the mesh.
+   // const int max_dofs = 100000;
+   for (int it = 0; ; it++)
+   {
+      HYPRE_Int global_dofs = fespace.GlobalTrueVSize();
+      if (myid == 0)
+      {
+         cout << "\nAMR iteration " << it << endl;
+         cout << "Number of unknowns: " << global_dofs << endl;
+      }
+
+      // 13. Assemble the stiffness matrix and the right-hand side. Note that
+      //     MFEM doesn't care at this point that the mesh is nonconforming
+      //     and parallel. The FE space is considered 'cut' along hanging
+      //     edges/faces, and also across processor boundaries.
+      a.Assemble();
+      b.Assemble();
+
+      // 14. Create the parallel linear system: eliminate boundary conditions,
+      //     constrain hanging nodes and nodes across processor boundaries.
+      //     The system will be solved for true (unconstrained/unique) DOFs only.
+      // Array<int> ess_tdof_list;
+      if ( bt == EntitySets::INVALID )
+      {
+         if (pmesh.bdr_attributes.Size())
+         {
+            Array<int> ess_bdr(pmesh.bdr_attributes.Max());
+            ess_bdr = 1;
+            fespace.GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+         }
+      }
+      else
+      {
+         fespace.GetEssentialTrueDofs((EntitySets::EntityType)bt, bs,
+                                      ess_tdof_list);
+      }
+
+      HypreParMatrix A;
+      Vector B, X;
+      const int copy_interior = 1;
+      a.FormLinearSystem(ess_tdof_list, x, b, A, X, B, copy_interior);
+
+      // 15. Define and apply a parallel PCG solver for AX=B with the BoomerAMG
+      //     preconditioner from hypre.
+      HypreBoomerAMG amg;
+      amg.SetPrintLevel(0);
+      CGSolver pcg(A.GetComm());
+      pcg.SetPreconditioner(amg);
+      pcg.SetOperator(A);
+      pcg.SetRelTol(1e-6);
+      pcg.SetMaxIter(200);
+      pcg.SetPrintLevel(3); // print the first and the last iterations only
+      pcg.Mult(B, X);
+
+      // 16. Extract the parallel grid function corresponding to the finite element
+      //     approximation X. This is the local solution on each processor.
+      a.RecoverFEMSolution(X, b, x);
+
+      // 17. Send the solution by socket to a GLVis server.
+      if (visualization)
+      {
+         sout << "parallel " << num_procs << " " << myid << "\n";
+         sout << "solution\n" << pmesh << x << flush;
+      }
+
+      if (global_dofs > max_dofs)
+      {
+         if (myid == 0)
+         {
+            cout << "Reached the maximum number of dofs. Stop." << endl;
+         }
+         break;
+      }
+
+      // 18. Call the refiner to modify the mesh. The refiner calls the error
+      //     estimator to obtain element errors, then it selects elements to be
+      //     refined and finally it modifies the mesh. The Stop() method can be
+      //     used to determine if a stopping criterion was met.
+      refiner.Apply(pmesh);
+      if (refiner.Stop())
+      {
+         if (myid == 0)
+         {
+            cout << "Stopping criterion satisfied. Stop." << endl;
+         }
+         break;
+      }
+
+      // 19. Update the finite element space (recalculate the number of DOFs,
+      //     etc.) and create a grid function update matrix. Apply the matrix
+      //     to any GridFunctions over the space. In this case, the update
+      //     matrix is an interpolation matrix so the updated GridFunction will
+      //     still represent the same function as before refinement.
+      fespace.Update();
+      x.Update();
+
+      // 20. Load balance the mesh, and update the space and solution. Currently
+      //     available only for nonconforming meshes.
+      if (pmesh.Nonconforming())
+      {
+         pmesh.Rebalance();
+
+         // Update the space and the GridFunction. This time the update matrix
+         // redistributes the GridFunction among the processors.
+         fespace.Update();
+         x.Update();
+      }
+
+      // 21. Inform also the bilinear and linear forms that the space has
+      //     changed.
+      a.Update();
+      b.Update();
+   }
+
+   MPI_Finalize();
+   return 0;
+}
@@ -282,10 +282,10 @@ int main(int argc, char *argv[])
   delete b;

   // 12. Compute and print the L^2 norm of the error.
-   double error = x.ComputeL2Error(sol_coef);
+   double err = x.ComputeL2Error(sol_coef);
   if (myid == 0)
   {
-      cout << "\nL2 norm of error: " << error << endl;
+      cout << "\nL2 norm of error: " << err << endl;
   }

   // 13. Save the refined mesh and the solution. This output can be viewed
@@ -27,7 +27,6 @@
 //    ex9 -pa -m ../data/periodic-cube.mesh -d cuda
 //    ex9 -ea -m ../data/periodic-cube.mesh -d cuda
 //    ex9 -fa -m ../data/periodic-cube.mesh -d cuda
-//    ex9 -pa -m ../data/amr-quad.mesh -p 1 -r 2 -dt 0.002 -tf 9 -d cuda
 //
 // Description:  This example code solves the time-dependent advection equation
 //               du/dt + v.grad(u) = 0, where v is a given fluid velocity, and
@@ -28,7 +28,6 @@
 //    mpirun -np 4 ex9p -pa -m ../data/periodic-cube.mesh -d cuda
 //    mpirun -np 4 ex9p -ea -m ../data/periodic-cube.mesh -d cuda
 //    mpirun -np 4 ex9p -fa -m ../data/periodic-cube.mesh -d cuda
-//    mpirun -np 4 ex9p -pa -m ../data/amr-quad.mesh -p 1 -rp 1 -dt 0.002 -tf 9 -d cuda
 //
 // Description:  This example code solves the time-dependent advection equation
 //               du/dt + v.grad(u) = 0, where v is a given fluid velocity, and
@@ -0,0 +1,162 @@
+MFEM mesh v1.0
+
+#
+# MFEM Geometry Types (see mesh/geom.hpp):
+#
+# POINT       = 0
+# SEGMENT     = 1
+# TRIANGLE    = 2
+# SQUARE      = 3
+# TETRAHEDRON = 4
+# CUBE        = 5
+# PRISM       = 6
+#
+
+dimension
+3
+
+elements
+14
+1 4 13 15 21 25
+1 4 12 13 15 21
+1 4 13 21 22 25
+1 4 15 24 21 25
+1 4 13 15 25 16
+1 5  0  1  4  3  9 10 13 12
+1 5  8  9 12 11 17 18 21 20
+1 5  2  3  6  5 11 12 15 14
+1 6  3  4  6 12 13 15
+1 6  4  7  6 13 16 15
+1 6 12 13 21  9 10 18
+1 6 13 22 21 10 19 18
+1 6 11 14 20 12 15 21
+1 6 15 21 24 14 20 23
+
+boundary
+30
+ 1 3  5  6  3  2
+ 2 2  3  6  4
+ 2 2  4  6  7
+ 3 3  3  4  1  0
+ 4 3 11 12  9  8
+ 5 3  2  3 12 11
+ 6 3  0  1 10  9
+ 7 2  9 10 18
+ 7 2 10 19 18
+ 8 3  8  9 18 17
+ 9 3  1  4 13 10
+10 3  4  7 16 13
+11 2 13 16 25
+11 2 13 25 22
+12 3 10 13 22 19
+13 3  7  6 15 16
+14 3  6  5 14 15
+15 3 15 14 23 24
+16 2 16 15 25
+16 2 15 24 25
+17 3  5  2 11 14
+18 3  3  0  9 12
+19 3 11  8 17 20
+20 2 11 20 14
+20 2 14 20 23
+21 3 17 18 21 20
+22 3 18 19 22 21
+23 2 21 22 25
+23 2 21 25 24
+24 3 20 21 24 23
+
+vertices
+26
+3
+0 -1 -1
+1 -1 -1
+-1 0 -1
+0 0 -1
+1 0 -1
+-1 1 -1
+0 1 -1
+1 1 -1
+-1 -1 0
+0 -1 0
+1 -1 0
+-1 0 0
+0 0 0
+1 0 0
+-1 1 0
+0 1 0
+1 1 0
+-1 -1 1
+0 -1 1
+1 -1 1
+-1 0 1
+0 0 1
+1 0 1
+-1 1 1
+0 1 1
+1 1 1
+
+MFEM sets v1.0
+
+vertex_sets
+1
+
+Origin
+1
+12
+
+edge_sets
+2
+
+Axes
+3
+12 13
+12 15
+12 21
+
+Negative Axes
+3
+12 9
+12 11
+12 3
+
+face_sets
+2
+
+Interior Corner
+3
+3 11 12  9 8
+3  2  3 12 11
+3  3  0  9 12
+
+Exterior Corner
+15
+2 13 16 25
+2 13 25 22
+2 16 15 25
+2 15 24 25
+2 21 22 25
+2 21 25 24
+3 10 13 22 19
+3 4 7 16 13
+3 1 4 13 10
+3 7 6 15 16
+3 6 5 14 15
+3 15 14 23 24
+3 20 21 24 23
+3 18 19 22 21
+3 17 18 21 20
+
+element_sets
+3
+
+Interior Corner
+3
+5 6 7
+
+Exterior Corner
+5
+0 1 2 3 4
+
+Steps
+3
+6 8 9
@@ -0,0 +1,145 @@
+MFEM mesh v1.0
+
+#
+# MFEM Geometry Types (see mesh/geom.hpp):
+#
+# POINT       = 0
+# SEGMENT     = 1
+# TRIANGLE    = 2
+# SQUARE      = 3
+# TETRAHEDRON = 4
+# CUBE        = 5
+#
+
+dimension
+3
+
+elements
+7
+1 5 0 1 4 3 9 10 13 12
+1 5 3 4 7 6 12 13 16 15
+1 5 2 3 6 5 11 12 15 14
+1 5 8 9 12 11 17 18 21 20
+1 5 9 10 13 12 18 19 22 21
+1 5 12 13 16 15 21 22 25 24
+1 5 11 12 15 14 20 21 24 23
+
+boundary
+24
+1 3 5 6 3 2
+2 3 6 7 4 3
+3 3 3 4 1 0
+4 3 11 12 9 8
+5 3 2 3 12 11
+6 3 0 1 10 9
+7 3 9 10 19 18
+8 3 8 9 18 17
+9 3 1 4 13 10
+10 3 4 7 16 13
+11 3 13 16 25 22
+12 3 10 13 22 19
+13 3 7 6 15 16
+14 3 6 5 14 15
+15 3 15 14 23 24
+16 3 16 15 24 25
+17 3 5 2 11 14
+18 3 3 0 9 12
+19 3 11 8 17 20
+20 3 14 11 20 23
+21 3 17 18 21 20
+22 3 18 19 22 21
+23 3 21 22 25 24
+24 3 20 21 24 23
+
+vertices
+26
+3
+0 -1 -1
+1 -1 -1
+-1 0 -1
+0 0 -1
+1 0 -1
+-1 1 -1
+0 1 -1
+1 1 -1
+-1 -1 0
+0 -1 0
+1 -1 0
+-1 0 0
+0 0 0
+1 0 0
+-1 1 0
+0 1 0
+1 1 0
+-1 -1 1
+0 -1 1
+1 -1 1
+-1 0 1
+0 0 1
+1 0 1
+-1 1 1
+0 1 1
+1 1 1
+
+MFEM sets v1.0
+
+vertex_sets
+1
+
+Origin
+1
+12
+
+edge_sets
+2
+
+Axes
+3
+12 13
+12 15
+12 21
+
+Negative Axes
+3
+12 9
+12 11
+12 3
+
+face_sets
+2
+
+Interior Corner
+3
+3 11 12  9 8
+3  2  3 12 11
+3  3  0  9 12
+
+Exterior Corner
+12
+3 13 16 25 22
+3 16 15 24 25
+3 21 22 25 24
+3 10 13 22 19
+3 4 7 16 13
+3 1 4 13 10
+3 7 6 15 16
+3 6 5 14 15
+3 15 14 23 24
+3 20 21 24 23
+3 18 19 22 21
+3 17 18 21 20
+
+element_sets
+3
+
+Interior Corner
+3
+0 2 3
+
+Exterior Corner
+1
+5
+
+Steps
+2
+1 3
@@ -31,10 +31,6 @@
 //               also illustrated. The example also shows how to form a linear
 //               system using a PETSc matrix and solve with a PETSc solver.
 //
-//               The example also show how to use the non-overlapping feature of
-//               the ParBilinearForm class to obtain the linear operator in
-//               a format suitable for the BDDC preconditioner in PETSc.
-//
 //               We recommend viewing Example 1 before viewing this example.

 #include "mfem.hpp"
@@ -65,15 +61,10 @@ int main(int argc, char *argv[])
   bool use_petsc = true;
   const char *petscrc_file = "";
   bool use_nonoverlapping = false;
-   int ser_ref_levels = -1, par_ref_levels = 1;

   OptionsParser args(argc, argv);
   args.AddOption(&mesh_file, "-m", "--mesh",
                  "Mesh file to use.");
-   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
-                  "Number of times to refine the mesh uniformly in serial.");
-   args.AddOption(&par_ref_levels, "-rp", "--refine-parallel",
-                  "Number of times to refine the mesh uniformly in parallel.");
   args.AddOption(&order, "-o", "--order",
                  "Finite element order (polynomial degree).");
   args.AddOption(&amg_elast, "-elast", "--amg-for-elasticity", "-sys",
@@ -140,8 +131,8 @@ int main(int argc, char *argv[])
   //    'ref_levels' to be the largest number that gives a final mesh with no
   //    more than 1,000 elements.
   {
-      int ref_levels = ser_ref_levels >= 0 ? ser_ref_levels :
-                       (int)floor(log(1000./mesh->GetNE())/log(2.)/dim);
+      int ref_levels =
+         (int)floor(log(1000./mesh->GetNE())/log(2.)/dim);
      for (int l = 0; l < ref_levels; l++)
      {
         mesh->UniformRefinement();
@@ -154,6 +145,7 @@ int main(int argc, char *argv[])
   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
   delete mesh;
   {
+      int par_ref_levels = 1;
      for (int l = 0; l < par_ref_levels; l++)
      {
         pmesh->UniformRefinement();
@@ -304,20 +296,12 @@ int main(int argc, char *argv[])
      PetscPreconditioner *prec = NULL;
      if (use_nonoverlapping)
      {
-         // Compute dofs belonging to the natural boundary
-         Array<int> nat_tdof_list, nat_bdr(pmesh->bdr_attributes.Max());
-         nat_bdr = 1;
-         nat_bdr[0] = 0;
-         fespace->GetEssentialTrueDofs(nat_bdr, nat_tdof_list);
-
         // Auxiliary class for BDDC customization
         PetscBDDCSolverParams opts;
         // Inform the solver about the finite element space
         opts.SetSpace(fespace);
         // Inform the solver about essential dofs
         opts.SetEssBdrDofs(&ess_tdof_list);
-         // Inform the solver about natural dofs
-         opts.SetNatBdrDofs(&nat_tdof_list);
         // Create a BDDC solver with parameters
         prec = new PetscBDDCSolver(A,opts);
         pcg->SetPreconditioner(*prec);
@@ -77,7 +77,6 @@ EX1_ARGS_P            := -m ../../data/amr-quad.mesh --usepetsc --petscopts rc_e
 EX1_ARGS_CUDA         := -m ../../data/star.mesh --usepetsc --partial-assembly --device cuda --petscopts rc_ex1p_cuda
 EX1_ARGS_CUDAAMG      := -m ../../data/star.mesh --usepetsc --device cuda --petscopts rc_ex1p_cudaamg
 EX2_ARGS              := -m ../../data/beam-quad.mesh --usepetsc --petscopts rc_ex2p
-EX2_ARGS_BDDC         := -m ../../data/beam-tri.mesh --usepetsc --nonoverlapping --petscopts rc_ex2p_bddc
 EX3_ARGS              := -m ../../data/klein-bottle.mesh -o 2 -f 0.1 --usepetsc --petscopts rc_ex3p_bddc --nonoverlapping
 EX4_ARGS              := -m ../../data/klein-bottle.mesh -o 2 --usepetsc --petscopts rc_ex4p_bddc --nonoverlapping
 EX4_HYB_ARGS          := -m ../../data/klein-bottle.mesh -o 2 --usepetsc --petscopts rc_ex4p_bddc --nonoverlapping --hybridization
@@ -108,7 +107,6 @@ ifeq ($(MFEM_USE_CUDA),YES)
 endif
 ex2p-test-par: ex2p
 	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX2_ARGS))
-	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX2_ARGS_BDDC))
 ex3p-test-par: ex3p
 	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX3_ARGS))
 ex4p-test-par: ex4p
@@ -1,25 +0,0 @@
-# Sample options for BDDC
-
-ksp_converged_reason
-ksp_view
-pc_type bddc
-
-# Turn on diagnostic for errors
-#-pc_bddc_check_level 1
-
-# This is an H1 problem, local problems may be singular
-# Turn on automatic corner selection
-pc_bddc_corner_selection
-
-# Advanced customization
-
-# Deluxe scaling
-pc_bddc_use_deluxe_scaling
-
-# Adaptive primal space (requires PETSc configured with MUMPS or PARDISO support)
-#-pc_bddc_adaptive_threshold 1.2 # tolerance for eigenvalue selection
-#-pc_bddc_adaptive_userdefined # preserve RBMs
-#-pc_bddc_monolithic # treat all displacements components at once -> smaller primal spaces, larger eigenvalue problems
-
-# Select solver for coarse problem
-# -pc_bddc_coarse_pc_type cholesky
@@ -145,7 +145,7 @@ int main(int argc, char *argv[])
   // Perform Uniform refinement
   if (ref_levels > 1)
   {
-      auto uniInput = ma::configureUniformRefine(pumi_mesh, ref_levels);
+      ma::Input* uniInput = ma::configureUniformRefine(pumi_mesh, ref_levels);

      if (geom_order > 1)
      {
@@ -150,7 +150,7 @@ int main(int argc, char *argv[])

   if (ref_levels > 1)
   {
-      auto uniInput = ma::configureUniformRefine(pumi_mesh, ref_levels);
+      ma::Input* uniInput = ma::configureUniformRefine(pumi_mesh, ref_levels);

      if ( geom_order > 1)
      {
@@ -345,7 +345,9 @@ int main(int argc, char *argv[])
      apf::destroyField(ipfield);

      // 18. Perform MesAdapt.
-      auto erinput = ma::configure(pumi_mesh, sizefield);
+      ma::Input* erinput = ma::configure(pumi_mesh, sizefield);
+      erinput->shouldFixShape = true;
+      erinput->maximumIterations = 2;
      if ( geom_order > 1)
      {
         crv::adapt(erinput);
@@ -0,0 +1,158 @@
+MFEM mesh v1.0
+
+#
+# MFEM Geometry Types (see mesh/geom.hpp):
+#
+# POINT       = 0
+# SEGMENT     = 1
+# TRIANGLE    = 2
+# SQUARE      = 3
+# TETRAHEDRON = 4
+# CUBE        = 5
+# PRISM       = 6
+#
+
+dimension
+2
+
+elements
+30
+1 3 0 11 26 14
+1 3 0 14 27 17
+1 3 0 17 28 20
+1 3 0 20 29 23
+1 3 0 23 30 11
+1 2 11 1 26
+1 2 1 12 26
+1 3 26 12 3 13
+1 2 26 13 2
+1 2 14 26 2
+1 2 14 2 27
+1 2 2 15 27
+1 3 27 15 5 16
+1 2 27 16 4
+1 2 17 27 4
+1 2 17 4 28
+1 2 4 18 28
+1 3 28 18 7 19
+1 2 28 19 6
+1 2 20 28 6
+1 2 20 6 29
+1 2 6 21 29
+1 3 29 21 9 22
+1 2 29 22 8
+1 2 23 29 8
+1 2 23 8 30
+1 2 8 24 30
+1 3 30 24 10 25
+1 2 30 25 1
+1 2 11 30 1
+
+boundary
+20
+1 1 13 2
+1 1 12 3
+1 1 16 4
+1 1 15 5
+1 1 19 6
+1 1 18 7
+1 1 22 8
+1 1 21 9
+1 1 25 1
+1 1 24 10
+1 1 3 13
+1 1 1 12
+1 1 5 16
+1 1 2 15
+1 1 7 19
+1 1 4 18
+1 1 9 22
+1 1 6 21
+1 1 10 25
+1 1 8 24
+
+vertices
+31
+2
+0 0
+1 0
+0.309017 0.951057
+1.30902 0.951057
+-0.809017 0.587785
+-0.5 1.53884
+-0.809017 -0.587785
+-1.61803 0
+0.309017 -0.951057
+-0.5 -1.53884
+1.30902 -0.951057
+0.5 0
+1.15451 0.475529
+0.809019 0.951057
+0.154508 0.475529
+-0.0954915 1.24495
+-0.654508 1.06331
+-0.404508 0.293893
+-1.21352 0.293893
+-1.21352 -0.293892
+-0.404508 -0.293893
+-0.654508 -1.06331
+-0.0954915 -1.24495
+0.154508 -0.475529
+0.809019 -0.951057
+1.15451 -0.475529
+0.654509 0.475529
+-0.25 0.769421
+-0.809016 0
+-0.25 -0.76942
+0.654509 -0.475529
+
+MFEM sets v1.0
+
+vertex_sets
+3
+
+Origin
+1
+0
+
+Tent
+5
+1 2 4 6 8
+
+Gazebo
+5
+3 5 7 9 10
+
+edge_sets
+2
+
+Columbine
+5
+1 11
+2 14
+4 17
+6 20
+8 23
+
+Lily
+5
+0 11
+0 14
+0 17
+0 20
+0 23
+
+element_sets
+3
+
+Flying Squirrel
+3
+7 17 27
+
+Sea Lion
+4
+12 17 22 27
+
+Pinwheel
+5
+8 13 18 23 28
@@ -0,0 +1,143 @@
+MFEM mesh v1.0
+
+#
+# MFEM Geometry Types (see mesh/geom.hpp):
+#
+# POINT       = 0
+# SEGMENT     = 1
+# TRIANGLE    = 2
+# SQUARE      = 3
+# TETRAHEDRON = 4
+# CUBE        = 5
+#
+
+dimension
+2
+
+elements
+20
+1 3 0 11 26 14
+1 3 0 14 27 17
+1 3 0 17 28 20
+1 3 0 20 29 23
+1 3 0 23 30 11
+1 3 11 1 12 26
+1 3 26 12 3 13
+1 3 14 26 13 2
+1 3 14 2 15 27
+1 3 27 15 5 16
+1 3 17 27 16 4
+1 3 17 4 18 28
+1 3 28 18 7 19
+1 3 20 28 19 6
+1 3 20 6 21 29
+1 3 29 21 9 22
+1 3 23 29 22 8
+1 3 23 8 24 30
+1 3 30 24 10 25
+1 3 11 30 25 1
+
+boundary
+20
+1 1 13 2
+1 1 12 3
+1 1 16 4
+1 1 15 5
+1 1 19 6
+1 1 18 7
+1 1 22 8
+1 1 21 9
+1 1 25 1
+1 1 24 10
+1 1 3 13
+1 1 1 12
+1 1 5 16
+1 1 2 15
+1 1 7 19
+1 1 4 18
+1 1 9 22
+1 1 6 21
+1 1 10 25
+1 1 8 24
+
+vertices
+31
+2
+0 0
+1 0
+0.309017 0.951057
+1.30902 0.951057
+-0.809017 0.587785
+-0.5 1.53884
+-0.809017 -0.587785
+-1.61803 0
+0.309017 -0.951057
+-0.5 -1.53884
+1.30902 -0.951057
+0.5 0
+1.15451 0.475529
+0.809019 0.951057
+0.154508 0.475529
+-0.0954915 1.24495
+-0.654508 1.06331
+-0.404508 0.293893
+-1.21352 0.293893
+-1.21352 -0.293892
+-0.404508 -0.293893
+-0.654508 -1.06331
+-0.0954915 -1.24495
+0.154508 -0.475529
+0.809019 -0.951057
+1.15451 -0.475529
+0.654509 0.475529
+-0.25 0.769421
+-0.809016 0
+-0.25 -0.76942
+0.654509 -0.475529
+
+MFEM sets v1.0
+
+vertex_sets
+3
+
+Origin
+1
+0
+
+Tent
+5
+1 2 4 6 8
+
+Gazebo
+5
+3 5 7 9 10
+
+edge_sets
+2
+
+Columbine
+5
+1 11
+2 14
+4 17
+6 20
+8 23
+
+Lily
+5
+0 11
+0 14
+0 17
+0 20
+0 23
+
+element_sets
+2
+
+Flying Squirrel
+3
+6 12 18
+
+Sea Lion
+4
+9 12 15 18
@@ -333,9 +333,9 @@ void BilinearForm::AssembleElementMatrix(
 }

 void BilinearForm::AssembleElementMatrix(
-   int i, const DenseMatrix &elmat, Array<int> &vdofs_, int skip_zeros)
+   int i, const DenseMatrix &elmat, Array<int> &vdofs, int skip_zeros)
 {
-   fes->GetElementVDofs(i, vdofs_);
+   fes->GetElementVDofs(i, vdofs);
   if (static_cond)
   {
      static_cond->AssembleMatrix(i, elmat);
@@ -346,7 +346,7 @@ void BilinearForm::AssembleElementMatrix(
      {
         AllocMat();
      }
-      mat->AddSubMatrix(vdofs_, vdofs_, elmat, skip_zeros);
+      mat->AddSubMatrix(vdofs, vdofs, elmat, skip_zeros);
      if (hybridization)
      {
         hybridization->AssembleMatrix(i, elmat);
@@ -361,9 +361,9 @@ void BilinearForm::AssembleBdrElementMatrix(
 }

 void BilinearForm::AssembleBdrElementMatrix(
-   int i, const DenseMatrix &elmat, Array<int> &vdofs_, int skip_zeros)
+   int i, const DenseMatrix &elmat, Array<int> &vdofs, int skip_zeros)
 {
-   fes->GetBdrElementVDofs(i, vdofs_);
+   fes->GetBdrElementVDofs(i, vdofs);
   if (static_cond)
   {
      static_cond->AssembleBdrMatrix(i, elmat);
@@ -374,7 +374,7 @@ void BilinearForm::AssembleBdrElementMatrix(
      {
         AllocMat();
      }
-      mat->AddSubMatrix(vdofs_, vdofs_, elmat, skip_zeros);
+      mat->AddSubMatrix(vdofs, vdofs, elmat, skip_zeros);
      if (hybridization)
      {
         hybridization->AssembleBdrMatrix(i, elmat);
@@ -965,14 +965,14 @@ void BilinearForm::EliminateEssentialBCDiag (const Array<int> &bdr_attr_is_ess,
   }
 }

-void BilinearForm::EliminateVDofs(const Array<int> &vdofs_,
+void BilinearForm::EliminateVDofs(const Array<int> &vdofs,
                                  const Vector &sol, Vector &rhs,
                                  DiagonalPolicy dpolicy)
 {
-   vdofs_.HostRead();
-   for (int i = 0; i < vdofs_.Size(); i++)
+   vdofs.HostRead();
+   for (int i = 0; i < vdofs.Size(); i++)
   {
-      int vdof = vdofs_[i];
+      int vdof = vdofs[i];
      if ( vdof >= 0 )
      {
         mat -> EliminateRowCol (vdof, sol(vdof), rhs, dpolicy);
@@ -984,7 +984,7 @@ void BilinearForm::EliminateVDofs(const Array<int> &vdofs_,
   }
 }

-void BilinearForm::EliminateVDofs(const Array<int> &vdofs_,
+void BilinearForm::EliminateVDofs(const Array<int> &vdofs,
                                  DiagonalPolicy dpolicy)
 {
   if (mat_e == NULL)
@@ -992,9 +992,9 @@ void BilinearForm::EliminateVDofs(const Array<int> &vdofs_,
      mat_e = new SparseMatrix(height);
   }

-   for (int i = 0; i < vdofs_.Size(); i++)
+   for (int i = 0; i < vdofs.Size(); i++)
   {
-      int vdof = vdofs_[i];
+      int vdof = vdofs[i];
      if ( vdof >= 0 )
      {
         mat -> EliminateRowCol (vdof, *mat_e, dpolicy);
@@ -1046,10 +1046,10 @@ void BilinearForm::EliminateEssentialBCFromDofsDiag (const Array<int> &ess_dofs,
 }

 void BilinearForm::EliminateVDofsInRHS(
-   const Array<int> &vdofs_, const Vector &x, Vector &b)
+   const Array<int> &vdofs, const Vector &x, Vector &b)
 {
   mat_e->AddMult(x, b, -1.);
-   mat->PartMult(vdofs_, x, b);
+   mat->PartMult(vdofs, x, b);
 }

 void BilinearForm::Mult(const Vector &x, Vector &y) const
@@ -1653,16 +1653,16 @@ void MixedBilinearForm::AssembleElementMatrix(
 }

 void MixedBilinearForm::AssembleElementMatrix(
-   int i, const DenseMatrix &elmat, Array<int> &trial_vdofs_,
-   Array<int> &test_vdofs_, int skip_zeros)
+   int i, const DenseMatrix &elmat, Array<int> &trial_vdofs,
+   Array<int> &test_vdofs, int skip_zeros)
 {
-   trial_fes->GetElementVDofs(i, trial_vdofs_);
-   test_fes->GetElementVDofs(i, test_vdofs_);
+   trial_fes->GetElementVDofs(i, trial_vdofs);
+   test_fes->GetElementVDofs(i, test_vdofs);
   if (mat == NULL)
   {
      mat = new SparseMatrix(height, width);
   }
-   mat->AddSubMatrix(test_vdofs_, trial_vdofs_, elmat, skip_zeros);
+   mat->AddSubMatrix(test_vdofs, trial_vdofs, elmat, skip_zeros);
 }

 void MixedBilinearForm::AssembleBdrElementMatrix(
@@ -1672,16 +1672,16 @@ void MixedBilinearForm::AssembleBdrElementMatrix(
 }

 void MixedBilinearForm::AssembleBdrElementMatrix(
-   int i, const DenseMatrix &elmat, Array<int> &trial_vdofs_,
-   Array<int> &test_vdofs_, int skip_zeros)
+   int i, const DenseMatrix &elmat, Array<int> &trial_vdofs,
+   Array<int> &test_vdofs, int skip_zeros)
 {
-   trial_fes->GetBdrElementVDofs(i, trial_vdofs_);
-   test_fes->GetBdrElementVDofs(i, test_vdofs_);
+   trial_fes->GetBdrElementVDofs(i, trial_vdofs);
+   test_fes->GetBdrElementVDofs(i, test_vdofs);
   if (mat == NULL)
   {
      mat = new SparseMatrix(height, width);
   }
-   mat->AddSubMatrix(test_vdofs_, trial_vdofs_, elmat, skip_zeros);
+   mat->AddSubMatrix(test_vdofs, trial_vdofs, elmat, skip_zeros);
 }

 void MixedBilinearForm::EliminateTrialDofs (
@@ -514,10 +514,6 @@ void EABilinearFormExtension::Assemble()

   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
   const int integratorCount = integrators.Size();
-   if ( integratorCount == 0 )
-   {
-      ea_data = 0.0;
-   }
   for (int i = 0; i < integratorCount; ++i)
   {
      integrators[i]->AssembleEA(*a->FESpace(), ea_data, i);
@@ -659,7 +659,7 @@ void GradientIntegrator::AssembleElementMatrix2(
   const FiniteElement &trial_fe, const FiniteElement &test_fe,
   ElementTransformation &Trans,  DenseMatrix &elmat)
 {
-   dim = test_fe.GetDim();
+   int dim = test_fe.GetDim();
   int trial_dof = trial_fe.GetDof();
   int test_dof = test_fe.GetDof();
   double c;
@@ -726,7 +726,7 @@ void DiffusionIntegrator::AssembleElementMatrix
  DenseMatrix &elmat )
 {
   int nd = el.GetDof();
-   dim = el.GetDim();
+   int dim = el.GetDim();
   int spaceDim = Trans.GetSpaceDim();
   bool square = (dim == spaceDim);
   double w;
@@ -802,7 +802,7 @@ void DiffusionIntegrator::AssembleElementMatrix2(
 {
   int tr_nd = trial_fe.GetDof();
   int te_nd = test_fe.GetDof();
-   dim = trial_fe.GetDim();
+   int dim = trial_fe.GetDim();
   int spaceDim = Trans.GetSpaceDim();
   bool square = (dim == spaceDim);
   double w;
@@ -885,7 +885,7 @@ void DiffusionIntegrator::AssembleElementVector(
   Vector &elvect)
 {
   int nd = el.GetDof();
-   dim = el.GetDim();
+   int dim = el.GetDim();
   int spaceDim = Tr.GetSpaceDim();
   double w;

@@ -966,7 +966,7 @@ void DiffusionIntegrator::ComputeElementFlux
 ( const FiniteElement &el, ElementTransformation &Trans,
  Vector &u, const FiniteElement &fluxelem, Vector &flux, bool with_coef )
 {
-   int nd, spaceDim, fnd;
+   int i, j, nd, dim, spaceDim, fnd;

   nd = el.GetDof();
   dim = el.GetDim();
@@ -1005,7 +1005,7 @@ void DiffusionIntegrator::ComputeElementFlux
   fnd = ir.GetNPoints();
   flux.SetSize( fnd * spaceDim );

-   for (int i = 0; i < fnd; i++)
+   for (i = 0; i < fnd; i++)
   {
      const IntegrationPoint &ip = ir.IntPoint(i);
      el.CalcDShape(ip, dshape);
@@ -1023,7 +1023,7 @@ void DiffusionIntegrator::ComputeElementFlux
            {
               vecdxt *= Q->Eval(Trans,ip);
            }
-            for (int j = 0; j < spaceDim; j++)
+            for (j = 0; j < spaceDim; j++)
            {
               flux(fnd*j+i) = vecdxt(j);
            }
@@ -1043,7 +1043,7 @@ void DiffusionIntegrator::ComputeElementFlux
                  pointflux[j] = D[j] * vecdxt[j];
               }
            }
-            for (int j = 0; j < spaceDim; j++)
+            for (j = 0; j < spaceDim; j++)
            {
               flux(fnd*j+i) = pointflux(j);
            }
@@ -1051,7 +1051,7 @@ void DiffusionIntegrator::ComputeElementFlux
      }
      else
      {
-         for (int j = 0; j < spaceDim; j++)
+         for (j = 0; j < spaceDim; j++)
         {
            flux(fnd*j+i) = vecdxt(j);
         }
@@ -1064,7 +1064,7 @@ double DiffusionIntegrator::ComputeFluxEnergy
  Vector &flux, Vector* d_energy)
 {
   int nd = fluxelem.GetDof();
-   dim = fluxelem.GetDim();
+   int dim = fluxelem.GetDim();
   int spaceDim = Trans.GetSpaceDim();

 #ifdef MFEM_THREAD_SAFE
@@ -1295,7 +1295,7 @@ void ConvectionIntegrator::AssembleElementMatrix(
   const FiniteElement &el, ElementTransformation &Trans, DenseMatrix &elmat)
 {
   int nd = el.GetDof();
-   dim = el.GetDim();
+   int dim = el.GetDim();

 #ifdef MFEM_THREAD_SAFE
   DenseMatrix dshape, adjJ, Q_ir;
@@ -1864,7 +1864,7 @@ void CurlCurlIntegrator::AssembleElementMatrix
  DenseMatrix &elmat )
 {
   int nd = el.GetDof();
-   dim = el.GetDim();
+   int dim = el.GetDim();
   int dimc = (dim == 3) ? 3 : 1;
   double w;

@@ -1961,7 +1961,7 @@ double CurlCurlIntegrator::ComputeFluxEnergy(const FiniteElement &fluxelem,
                                             Vector &flux, Vector *d_energy)
 {
   int nd = fluxelem.GetDof();
-   dim = fluxelem.GetDim();
+   int dim = fluxelem.GetDim();

 #ifdef MFEM_THREAD_SAFE
   DenseMatrix vshape;
@@ -2420,7 +2420,7 @@ void VectorDivergenceIntegrator::AssembleElementMatrix2(
   ElementTransformation &Trans,
   DenseMatrix &elmat)
 {
-   dim  = trial_fe.GetDim();
+   int dim  = trial_fe.GetDim();
   int trial_dof = trial_fe.GetDof();
   int test_dof = test_fe.GetDof();
   double c;
@@ -2523,9 +2523,9 @@ void VectorDiffusionIntegrator::AssembleElementMatrix(
   ElementTransformation &Trans,
   DenseMatrix &elmat)
 {
+   const int dim = el.GetDim();
   const int dof = el.GetDof();
-   dim = el.GetDim();
-   sdim = Trans.GetSpaceDim();
+   const int sdim = Trans.GetSpaceDim();

   // If vdim is not set, set it to the space dimension;
   vdim = (vdim <= 0) ? sdim : vdim;
@@ -2579,12 +2579,12 @@ void VectorDiffusionIntegrator::AssembleElementMatrix(
      else if (MQ)
      {
         MQ->Eval(mcoeff, Trans, ip);
-         for (int ii = 0; ii < vdim; ++ii)
+         for (int i = 0; i < vdim; ++i)
         {
-            for (int jj = 0; jj < vdim; ++jj)
+            for (int j = 0; j < vdim; ++j)
            {
-               Mult_a_AAt(w*mcoeff(ii,jj), dshapedxt, pelmat);
-               elmat.AddMatrix(pelmat, dof*ii, dof*jj);
+               Mult_a_AAt(w*mcoeff(i,j), dshapedxt, pelmat);
+               elmat.AddMatrix(pelmat, dof*i, dof*j);
            }
         }
      }
@@ -2604,9 +2604,9 @@ void VectorDiffusionIntegrator::AssembleElementVector(
   const FiniteElement &el, ElementTransformation &Tr,
   const Vector &elfun, Vector &elvect)
 {
+   const int dim = el.GetDim();
   const int dof = el.GetDof();
-   dim = el.GetDim();
-   sdim = Tr.GetSpaceDim();
+   const int sdim = Tr.GetSpaceDim();

   // If vdim is not set, set it to the space dimension;
   vdim = (vdim <= 0) ? sdim : vdim;
@@ -2665,13 +2665,13 @@ void VectorDiffusionIntegrator::AssembleElementVector(
      else if (MQ)
      {
         MQ->Eval(mcoeff, Tr, ip);
-         for (int ii = 0; ii < vdim; ++ii)
+         for (int i = 0; i < vdim; ++i)
         {
-            Vector vec_out(mat_out.GetColumn(ii), dof);
-            for (int jj = 0; jj < vdim; ++jj)
+            Vector vec_out(mat_out.GetColumn(i), dof);
+            for (int j = 0; j < vdim; ++j)
            {
-               pelmat *= w*mcoeff(ii,jj);
-               const Vector vec_in(mat_in.GetColumn(jj), dof);
+               pelmat *= w*mcoeff(i,j);
+               const Vector vec_in(mat_in.GetColumn(j), dof);
               pelmat.Mult(vec_in, vec_out);
            }
         }
@@ -2759,14 +2759,14 @@ void ElasticityIntegrator::AssembleElementMatrix(
                  elmat (dof*d+k, dof*d+l) += (M * w) * pelmat(k, l);
               }
         }
-         for (int ii = 0; ii < dim; ii++)
-            for (int jj = 0; jj < dim; jj++)
+         for (int i = 0; i < dim; i++)
+            for (int j = 0; j < dim; j++)
            {
-               for (int kk = 0; kk < dof; kk++)
-                  for (int ll = 0; ll < dof; ll++)
+               for (int k = 0; k < dof; k++)
+                  for (int l = 0; l < dof; l++)
                  {
-                     elmat(dof*ii+kk, dof*jj+ll) +=
-                        (M * w) * gshape(kk, jj) * gshape(ll, ii);
+                     elmat(dof*i+k, dof*j+l) +=
+                        (M * w) * gshape(k, j) * gshape(l, i);
                  }
            }
      }
@@ -2947,7 +2947,7 @@ void DGTraceIntegrator::AssembleFaceMatrix(const FiniteElement &el1,
                                           FaceElementTransformations &Trans,
                                           DenseMatrix &elmat)
 {
-   int ndof1, ndof2;
+   int dim, ndof1, ndof2;

   double un, a, b, w;

@@ -644,13 +644,13 @@ protected:

   inline virtual void CalcVShape(const FiniteElement & vector_fe,
                                  ElementTransformation &Trans,
-                                  DenseMatrix & shape_)
-   { vector_fe.CalcVShape(Trans, shape_); }
+                                  DenseMatrix & shape)
+   { vector_fe.CalcVShape(Trans, shape); }

   inline virtual void CalcShape(const FiniteElement & scalar_fe,
                                 ElementTransformation &Trans,
-                                 Vector & shape_)
-   { scalar_fe.CalcPhysShape(Trans, shape_); }
+                                 Vector & shape)
+   { scalar_fe.CalcPhysShape(Trans, shape); }

   VectorCoefficient *VQ;
   bool transpose;
@@ -2258,7 +2258,7 @@ public:
      : vdim(q.GetVDim()), Q_order(qo), Q(NULL), VQ(NULL), MQ(&q) { }

   int GetVDim() const { return vdim; }
-   void SetVDim(int vdim_) { vdim = vdim_; }
+   void SetVDim(int vdim) { this->vdim = vdim; }

   virtual void AssembleElementMatrix(const FiniteElement &el,
                                      ElementTransformation &Trans,
@@ -182,26 +182,22 @@ void DGTraceIntegrator::SetupPA(const FiniteElementSpace &fes, FaceType type)
      auto C = Reshape(vel.HostWrite(), dim, nq, nf);
      Vector Vq(dim);
      int f_ind = 0;
-      for (int f = 0; f < mesh->GetNumFacesWithGhost(); ++f)
+      for (int f = 0; f < fes.GetNF(); ++f)
      {
-         Mesh::FaceInformation face = mesh->GetFaceInformation(f);
-         if (face.IsNonconformingCoarse())
+         int e1, e2;
+         int inf1, inf2;
+         fes.GetMesh()->GetFaceElements(f, &e1, &e2);
+         fes.GetMesh()->GetFaceInfos(f, &inf1, &inf2);
+         int face_id = inf1 / 64;
+         if ((type==FaceType::Interior && (e2>=0 || (e2<0 && inf2>=0))) ||
+             (type==FaceType::Boundary && e2<0 && inf2<0) )
         {
-            // We skip nonconforming coarse faces as they are treated
-            // by the corresponding nonconforming fine faces.
-            continue;
-         }
-         else if ( face.IsOfFaceType(type) )
-         {
-            const int mask = FaceElementTransformations::HAVE_ELEM1 |
-                             FaceElementTransformations::HAVE_LOC1;
            FaceElementTransformations &T =
-               *fes.GetMesh()->GetFaceElementTransformations(f, mask);
+               *fes.GetMesh()->GetFaceElementTransformations(f);
            for (int q = 0; q < nq; ++q)
            {
               // Convert to lexicographic ordering
-               int iq = ToLexOrdering(dim, face.element[0].local_face_id,
-                                      quad1D, q);
+               int iq = ToLexOrdering(dim, face_id, quad1D, q);
               T.SetAllIntPoints(&ir->IntPoint(q));
               const IntegrationPoint &eip1 = T.GetElement1IntPoint();
               u->Eval(Vq, *T.Elem1, eip1);
@@ -246,31 +242,29 @@ void DGTraceIntegrator::SetupPA(const FiniteElementSpace &fes, FaceType type)
      auto n = Reshape(geom->normal.HostRead(), nq, dim, nf);
      auto C = Reshape(r.HostWrite(), nq, nf);
      int f_ind = 0;
-      for (int f = 0; f < mesh->GetNumFacesWithGhost(); ++f)
+      for (int f = 0; f < fes.GetNF(); ++f)
      {
-         Mesh::FaceInformation face = mesh->GetFaceInformation(f);
-         if (face.IsNonconformingCoarse())
-         {
-            // We skip nonconforming coarse faces as they are treated
-            // by the corresponding nonconforming fine faces.
-            continue;
-         }
-         else if ( face.IsOfFaceType(type) )
+         int e1, e2;
+         int inf1, inf2;
+         fes.GetMesh()->GetFaceElements(f, &e1, &e2);
+         fes.GetMesh()->GetFaceInfos(f, &inf1, &inf2);
+         int face_id = inf1 / 64;
+         if ((type==FaceType::Interior && (e2>=0 || (e2<0 && inf2>=0))) ||
+             (type==FaceType::Boundary && e2<0 && inf2<0) )
         {
            FaceElementTransformations &T =
               *fes.GetMesh()->GetFaceElementTransformations(f);
            for (int q = 0; q < nq; ++q)
            {
               // Convert to lexicographic ordering
-               int iq = ToLexOrdering(dim, face.element[0].local_face_id,
-                                      quad1D, q);
+               int iq = ToLexOrdering(dim, face_id, quad1D, q);

               T.SetAllIntPoints(&ir->IntPoint(q));
               const IntegrationPoint &eip1 = T.GetElement1IntPoint();
               const IntegrationPoint &eip2 = T.GetElement2IntPoint();
               double r;

-               if ( face.IsBoundary() )
+               if (inf2 < 0)
               {
                  r = rho->Eval(*T.Elem1, eip1);
               }
@@ -41,7 +41,6 @@ void MassIntegrator::AssemblePA(const FiniteElementSpace &fes)
      ceedOp = new ceed::PAMassIntegrator(fes, *ir, Q);
      return;
   }
-   int map_type = el.GetMapType();
   dim = mesh->Dimension();
   ne = fes.GetMesh()->GetNE();
   nq = ir->GetNPoints();
@@ -94,7 +93,6 @@ void MassIntegrator::AssemblePA(const FiniteElementSpace &fes)
      const int NE = ne;
      const int Q1D = quad1D;
      const bool const_c = coeff.Size() == 1;
-      const bool by_val = map_type == FiniteElement::VALUE;
      const auto W = Reshape(ir->GetWeights().Read(), Q1D,Q1D);
      const auto J = Reshape(geom->J.Read(), Q1D,Q1D,2,2,NE);
      const auto C = const_c ? Reshape(coeff.Read(), 1,1,1) :
@@ -112,7 +110,7 @@ void MassIntegrator::AssemblePA(const FiniteElementSpace &fes)
               const double J22 = J(qx,qy,1,1,e);
               const double detJ = (J11*J22)-(J21*J12);
               const double coeff = const_c ? C(0,0,0) : C(qx,qy,e);
-               v(qx,qy,e) =  W(qx,qy) * coeff * (by_val ? detJ : 1.0/detJ);
+               v(qx,qy,e) =  W(qx,qy) * coeff * detJ;
            }
         }
      });
@@ -122,7 +120,6 @@ void MassIntegrator::AssemblePA(const FiniteElementSpace &fes)
      const int NE = ne;
      const int Q1D = quad1D;
      const bool const_c = coeff.Size() == 1;
-      const bool by_val = map_type == FiniteElement::VALUE;
      const auto W = Reshape(ir->GetWeights().Read(), Q1D,Q1D,Q1D);
      const auto J = Reshape(geom->J.Read(), Q1D,Q1D,Q1D,3,3,NE);
      const auto C = const_c ? Reshape(coeff.Read(), 1,1,1,1) :
@@ -149,7 +146,7 @@ void MassIntegrator::AssemblePA(const FiniteElementSpace &fes)
                  /* */               J21 * (J12 * J33 - J32 * J13) +
                  /* */               J31 * (J12 * J23 - J22 * J13);
                  const double coeff = const_c ? C(0,0,0,0) : C(qx,qy,qz,e);
-                  v(qx,qy,qz,e) = W(qx,qy,qz) * coeff * (by_val ? detJ : 1.0/detJ);
+                  v(qx,qy,qz,e) = W(qx,qy,qz) * coeff * detJ;
               }
            }
         }
@@ -667,7 +667,7 @@ AlgebraicSpaceHierarchy::AlgebraicSpaceHierarchy(FiniteElementSpace &fes)
      const SparseMatrix *R = fespaces[ilevel+1]->GetRestrictionMatrix();
      if (R)
      {
-         R->EnsureMultTranspose();
+         R->BuildTranspose();
         R_tr[ilevel] = new TransposeOperator(*R);
      }
      else
@@ -21,33 +21,6 @@ namespace mfem

 using namespace std;

-// Given an ElementTransformation and IntegrationPoint in a refined mesh,
-// return the ElementTransformation of the parent coarse element, and set
-// coarse_ip to the location of the original ip within the coarse element.
-ElementTransformation *RefinedToCoarse(
-   Mesh &coarse_mesh, const ElementTransformation &T,
-   const IntegrationPoint &ip, IntegrationPoint &coarse_ip)
-{
-   Mesh &fine_mesh = *T.mesh;
-   // Get the element transformation of the coarse element containing the
-   // fine element.
-   int fine_element = T.ElementNo;
-   const CoarseFineTransformations &cf = fine_mesh.GetRefinementTransforms();
-   int coarse_element = cf.embeddings[fine_element].parent;
-   ElementTransformation *coarse_T = coarse_mesh.GetElementTransformation(
-                                        coarse_element);
-   // Transform the integration point from fine element coordinates to coarse
-   // element coordinates.
-   Geometry::Type geom = T.GetGeometryType();
-   IntegrationPointTransformation fine_to_coarse;
-   IsoparametricTransformation &emb_tr = fine_to_coarse.Transf;
-   emb_tr.SetIdentityTransformation(geom);
-   emb_tr.SetPointMat(cf.point_matrices[geom](cf.embeddings[fine_element].matrix));
-   fine_to_coarse.Transform(ip, coarse_ip);
-   coarse_T->SetIntPoint(&coarse_ip);
-   return coarse_T;
-}
-
 double PWConstCoefficient::Eval(ElementTransformation & T,
                                const IntegrationPoint & ip)
 {
@@ -122,17 +95,7 @@ double FunctionCoefficient::Eval(ElementTransformation & T,
 double GridFunctionCoefficient::Eval (ElementTransformation &T,
                                      const IntegrationPoint &ip)
 {
-   Mesh *gf_mesh = GridF->FESpace()->GetMesh();
-   if (T.mesh == gf_mesh)
-   {
-      return GridF->GetValue(T, ip, Component);
-   }
-   else
-   {
-      IntegrationPoint coarse_ip;
-      ElementTransformation *coarse_T = RefinedToCoarse(*gf_mesh, T, ip, coarse_ip);
-      return GridF->GetValue(*coarse_T, coarse_ip, Component);
-   }
+   return GridF -> GetValue (T, ip, Component);
 }

 void TransformedCoefficient::SetTime(double t)
@@ -342,30 +305,13 @@ void VectorGridFunctionCoefficient::SetGridFunction(const GridFunction *gf)
 void VectorGridFunctionCoefficient::Eval(Vector &V, ElementTransformation &T,
                                         const IntegrationPoint &ip)
 {
-   Mesh *gf_mesh = GridFunc->FESpace()->GetMesh();
-   if (T.mesh == gf_mesh)
-   {
-      GridFunc->GetVectorValue(T, ip, V);
-   }
-   else
-   {
-      IntegrationPoint coarse_ip;
-      ElementTransformation *coarse_T = RefinedToCoarse(*gf_mesh, T, ip, coarse_ip);
-      GridFunc->GetVectorValue(*coarse_T, coarse_ip, V);
-   }
+   GridFunc->GetVectorValue(T, ip, V);
 }

 void VectorGridFunctionCoefficient::Eval(
   DenseMatrix &M, ElementTransformation &T, const IntegrationRule &ir)
 {
-   if (T.mesh == GridFunc->FESpace()->GetMesh())
-   {
-      GridFunc->GetVectorValues(T, ir, M);
-   }
-   else
-   {
-      VectorCoefficient::Eval(M, T, ir);
-   }
+   GridFunc->GetVectorValues(T, ir, M);
 }

 GradientGridFunctionCoefficient::GradientGridFunctionCoefficient (
@@ -385,30 +331,13 @@ void GradientGridFunctionCoefficient::SetGridFunction(const GridFunction *gf)
 void GradientGridFunctionCoefficient::Eval(Vector &V, ElementTransformation &T,
                                           const IntegrationPoint &ip)
 {
-   Mesh *gf_mesh = GridFunc->FESpace()->GetMesh();
-   if (T.mesh == gf_mesh)
-   {
-      GridFunc->GetGradient(T, V);
-   }
-   else
-   {
-      IntegrationPoint coarse_ip;
-      ElementTransformation *coarse_T = RefinedToCoarse(*gf_mesh, T, ip, coarse_ip);
-      GridFunc->GetGradient(*coarse_T, V);
-   }
+   GridFunc->GetGradient(T, V);
 }

 void GradientGridFunctionCoefficient::Eval(
   DenseMatrix &M, ElementTransformation &T, const IntegrationRule &ir)
 {
-   if (T.mesh == GridFunc->FESpace()->GetMesh())
-   {
-      GridFunc->GetGradients(T, ir, M);
-   }
-   else
-   {
-      VectorCoefficient::Eval(M, T, ir);
-   }
+   GridFunc->GetGradients(T, ir, M);
 }

 CurlGridFunctionCoefficient::CurlGridFunctionCoefficient(
@@ -434,17 +363,7 @@ void CurlGridFunctionCoefficient::SetGridFunction(const GridFunction *gf)
 void CurlGridFunctionCoefficient::Eval(Vector &V, ElementTransformation &T,
                                       const IntegrationPoint &ip)
 {
-   Mesh *gf_mesh = GridFunc->FESpace()->GetMesh();
-   if (T.mesh == gf_mesh)
-   {
-      GridFunc->GetCurl(T, V);
-   }
-   else
-   {
-      IntegrationPoint coarse_ip;
-      ElementTransformation *coarse_T = RefinedToCoarse(*gf_mesh, T, ip, coarse_ip);
-      GridFunc->GetCurl(*coarse_T, V);
-   }
+   GridFunc->GetCurl(T, V);
 }

 DivergenceGridFunctionCoefficient::DivergenceGridFunctionCoefficient (
@@ -456,17 +375,7 @@ DivergenceGridFunctionCoefficient::DivergenceGridFunctionCoefficient (
 double DivergenceGridFunctionCoefficient::Eval(ElementTransformation &T,
                                               const IntegrationPoint &ip)
 {
-   Mesh *gf_mesh = GridFunc->FESpace()->GetMesh();
-   if (T.mesh == gf_mesh)
-   {
-      return GridFunc->GetDivergence(T);
-   }
-   else
-   {
-      IntegrationPoint coarse_ip;
-      ElementTransformation *coarse_T = RefinedToCoarse(*gf_mesh, T, ip, coarse_ip);
-      return GridFunc->GetDivergence(*coarse_T);
-   }
+   return GridFunc->GetDivergence(T);
 }

 void VectorDeltaCoefficient::SetTime(double t)
@@ -574,7 +574,6 @@ public:
   /// Evaluate the coefficient.
   virtual void Eval(Vector &V, ElementTransformation &T,
                     const IntegrationPoint &ip);
-   using VectorCoefficient::Eval;
 };

 /// A general vector function coefficient
@@ -1605,22 +1604,22 @@ public:
   void SetTime(double t);

   /// Reset the first vector coefficient
-   void SetACoef(VectorCoefficient &A_) { ACoef = &A_; }
+   void SetACoef(VectorCoefficient &A) { ACoef = &A; }
   /// Return the first vector coefficient
   VectorCoefficient * GetACoef() const { return ACoef; }

   /// Reset the second vector coefficient
-   void SetBCoef(VectorCoefficient &B_) { BCoef = &B_; }
+   void SetBCoef(VectorCoefficient &B) { BCoef = &B; }
   /// Return the second vector coefficient
   VectorCoefficient * GetBCoef() const { return BCoef; }

   /// Reset the factor in front of the first vector coefficient
-   void SetAlphaCoef(Coefficient &A_) { alphaCoef = &A_; }
+   void SetAlphaCoef(Coefficient &A) { alphaCoef = &A; }
   /// Return the factor in front of the first vector coefficient
   Coefficient * GetAlphaCoef() const { return alphaCoef; }

   /// Reset the factor in front of the second vector coefficient
-   void SetBetaCoef(Coefficient &B_) { betaCoef = &B_; }
+   void SetBetaCoef(Coefficient &B) { betaCoef = &B; }
   /// Return the factor in front of the second vector coefficient
   Coefficient * GetBetaCoef() const { return betaCoef; }

@@ -1243,7 +1243,7 @@ ParSesquilinearForm::FormLinearSystem(const Array<int> &ess_tdof_list,
         HypreParMatrix * Ah;
         A_i.Get(Ah);
         hypre_ParCSRMatrix *Aih = *Ah;
-#if !defined(HYPRE_USING_GPU)
+#ifndef HYPRE_USING_CUDA
         ess_tdof_list.HostRead();
         for (int k = 0; k < n; k++)
         {
@@ -1256,7 +1256,7 @@ ParSesquilinearForm::FormLinearSystem(const Array<int> &ess_tdof_list,
            ess_tdof_list.GetMemory().Read(MemoryClass::DEVICE, n);
         const int *d_diag_i = Aih->diag->i;
         double *d_diag_data = Aih->diag->data;
-         MFEM_GPU_FORALL(k, n,
+         CuWrap1D(n, [=] MFEM_DEVICE (int k)
         {
            const int j = d_ess_tdof_list[k];
            d_diag_data[d_diag_i[j]] = 0.0;
@@ -330,7 +330,7 @@ ConduitDataCollection::BlueprintMeshToMesh(const Node &n_mesh,
         }
         else
         {
-            Node &n_bndry_conn_conv =
+            Node &(n_bndry_conn_conv) =
               n_conv["topologies"][bndry_topo_name]["elements/connectivity"];
            n_bndry_conn.to_int_array(n_bndry_conn_conv);
            bndry_indices = (n_bndry_conn_conv).value();
@@ -11,7 +11,6 @@

 #include "fem.hpp"
 #include "../mesh/nurbs.hpp"
-#include "../mesh/vtk.hpp"
 #include "../general/binaryio.hpp"
 #include "../general/text.hpp"
 #include "picojson.h"
@@ -188,7 +187,7 @@ void DataCollection::SetPrefixPath(const std::string& prefix)
   }
 }

-void DataCollection::Load(int cycle_)
+void DataCollection::Load(int cycle)
 {
   MFEM_ABORT("this method is not implemented");
 }
@@ -788,44 +787,53 @@ void ParaViewDataCollection::Load(int )

 std::string ParaViewDataCollection::GenerateCollectionPath()
 {
-   return prefix_path + DataCollection::GetCollectionName();
+   std::string out = "";
+   out = prefix_path + DataCollection::GetCollectionName();
+   return out;
 }

 std::string ParaViewDataCollection::GeneratePVTUPath()
 {
-   return "Cycle" + to_padded_string(cycle,pad_digits_cycle);
+   std::string out = "Cycle" + to_padded_string(cycle,pad_digits_cycle);
+   return out;
 }

 std::string ParaViewDataCollection::GenerateVTUPath()
 {
-   return GeneratePVTUPath();
+   std::string out = GeneratePVTUPath();
+   return out;
 }

 std::string ParaViewDataCollection::GeneratePVDFileName()
 {
-   return GetCollectionName() + ".pvd";
+   std::string out = GetCollectionName()+".pvd";
+   return out;
 }

-std::string ParaViewDataCollection::GeneratePVTUFileName(
-   const std::string &prefix)
+std::string ParaViewDataCollection::GeneratePVTUFileName()
 {
-   return prefix + ".pvtu";
+   std::string out = "data.pvtu";
+   return out;
 }

-std::string ParaViewDataCollection::GenerateVTUFileName(
-   const std::string &prefix, int rank)
+std::string ParaViewDataCollection::GenerateVTUFileName()
 {
-   return prefix + to_padded_string(rank, pad_digits_rank) + ".vtu";
+   std::string out = "proc" + to_padded_string(myid,pad_digits_rank)+".vtu";
+   return out;
+}
+std::string ParaViewDataCollection::GenerateVTUFileName(int crank)
+{
+   std::string out = "proc" + to_padded_string(crank,pad_digits_rank)+".vtu";
+   return out;
 }

 void ParaViewDataCollection::Save()
 {
   // add a new collection to the PDV file

-   std::string col_path = GenerateCollectionPath();
   // check if the directories are created
   {
-      std::string path = col_path + "/" + GenerateVTUPath();
+      std::string path = GenerateCollectionPath()+"/"+GenerateVTUPath();
      int err = create_directory(path, mesh, myid);
      if (err)
      {
@@ -842,7 +850,8 @@ void ParaViewDataCollection::Save()

   if (myid == 0 && !pvd_stream.is_open())
   {
-      std::string pvdname = col_path + "/" + GeneratePVDFileName();
+      std::string dpath=GenerateCollectionPath();
+      std::string pvdname=dpath+"/"+GeneratePVDFileName();

      bool write_header = true;
      std::ifstream pvd_in;
@@ -906,87 +915,80 @@ void ParaViewDataCollection::Save()
      }
   }

-   std::string vtu_prefix = col_path + "/" + GenerateVTUPath() + "/";
-
-   // Save the local part of the mesh and grid functions fields to the local
-   // VTU file
+   // define the vtu file
   {
-      std::ofstream out(vtu_prefix + GenerateVTUFileName("proc", myid));
+      std::string fname = GenerateCollectionPath()+"/"+GenerateVTUPath()+"/"
+                          +GenerateVTUFileName();
+      std::fstream out(fname, std::ios::out);
      out.precision(precision);
-      SaveDataVTU(out, levels_of_detail);
+      SaveDataVTU(out,levels_of_detail);
+      out.close();
   }

-   // Save the local part of the quadrature function fields
-   for (const auto &qfield : q_field_map)
+   // define the pvtu file only on process 0
+   if (myid==0)
   {
-      const std::string &field_name = qfield.first;
-      std::ofstream out(vtu_prefix + GenerateVTUFileName(field_name, myid));
-      qfield.second->SaveVTU(out, pv_data_format, compression);
-   }
+      std::string fname = GenerateCollectionPath()+"/"+GeneratePVTUPath()+"/"
+                          +GeneratePVTUFileName();
+      std::fstream out(fname, std::ios::out);

-   // MPI rank 0 also creates a "PVTU" file that points to all of the separately
-   // written VTU files.
-   // This file path is then appended to the PVD file.
-   if (myid == 0)
-   {
-      // Create the main PVTU file
+      out << "<?xml version=\"1.0\"?>\n";
+      out << "<VTKFile type=\"PUnstructuredGrid\"";
+      out << " version =\"0.1\" byte_order=\"" << VTKByteOrder() << "\">\n";
+      out << "<PUnstructuredGrid GhostLevel=\"0\">\n";
+
+      out << "<PPoints>\n";
+      out << "\t<PDataArray type=\"" << GetDataTypeString() << "\" ";
+      out << " Name=\"Points\" NumberOfComponents=\"3\""
+          << " format=\"" << GetDataFormatString() << "\"/>\n";
+      out << "</PPoints>\n";
+
+      out << "<PCells>\n";
+      out << "\t<PDataArray type=\"Int32\" ";
+      out << " Name=\"connectivity\" NumberOfComponents=\"1\""
+          << " format=\"" << GetDataFormatString() << "\"/>\n";
+      out << "\t<PDataArray type=\"Int32\" ";
+      out << " Name=\"offsets\"      NumberOfComponents=\"1\""
+          << " format=\"" << GetDataFormatString() << "\"/>\n";
+      out << "\t<PDataArray type=\"UInt8\" ";
+      out << " Name=\"types\"        NumberOfComponents=\"1\""
+          << " format=\"" << GetDataFormatString() << "\"/>\n";
+      out << "</PCells>\n";
+
+      out << "<PPointData>\n";
+      for (FieldMapIterator it=field_map.begin(); it!=field_map.end(); ++it)
      {
-         std::ofstream pvtu_out(vtu_prefix + GeneratePVTUFileName("data"));
-         WritePVTUHeader(pvtu_out);
-
-         // Grid function fields
-         pvtu_out << "<PPointData>\n";
-         for (auto &field_it : field_map)
-         {
-            int vec_dim = field_it.second->VectorDim();
-            pvtu_out << "<PDataArray type=\"" << GetDataTypeString()
-                     << "\" Name=\"" << field_it.first
-                     << "\" NumberOfComponents=\"" << vec_dim << "\" "
-                     << "format=\"" << GetDataFormatString() << "\" />\n";
-         }
-         pvtu_out << "</PPointData>\n";
-         // Element attributes
-         pvtu_out << "<PCellData>\n";
-         pvtu_out << "\t<PDataArray type=\"Int32\" Name=\"" << "attribute"
-                  << "\" NumberOfComponents=\"1\""
-                  << " format=\"" << GetDataFormatString() << "\"/>\n";
-         pvtu_out << "</PCellData>\n";
-
-         WritePVTUFooter(pvtu_out, "proc");
+         int vec_dim=it->second->VectorDim();
+         out << "<PDataArray type=\"" << GetDataTypeString()
+             << "\" Name=\"" << it->first
+             << "\" NumberOfComponents=\"" << vec_dim << "\" "
+             << "format=\"" << GetDataFormatString() << "\" />\n";
      }
+      out << "</PPointData>\n";

-      // Add the latest PVTU to the PVD
-      pvd_stream << "<DataSet timestep=\"" << GetTime()
-                 << "\" group=\"\" part=\"" << 0 << "\" file=\""
-                 << GeneratePVTUPath() + "/" + GeneratePVTUFileName("data")
-                 << "\" name=\"mesh\"/>\n";
+      // CELL DATA
+      out << "<PCellData>\n";
+      out << "\t<PDataArray type=\"Int32\" Name=\"" << "attribute"
+          << "\" NumberOfComponents=\"1\""
+          << " format=\"" << GetDataFormatString() << "\"/>\n";
+      out << "</PCellData>\n";

-      // Create PVTU files for each quadrature field and add them to the PVD
-      // file
-      for (auto &q_field : q_field_map)
+      for (int ii=0; ii<num_procs; ii++)
      {
-         const std::string &q_field_name = q_field.first;
-         std::string q_fname = GeneratePVTUPath() + "/"
-                               + GeneratePVTUFileName(q_field_name);
-
-         std::ofstream pvtu_out(col_path + "/" + q_fname);
-         WritePVTUHeader(pvtu_out);
-         int vec_dim = q_field.second->GetVDim();
-         pvtu_out << "<PPointData>\n";
-         pvtu_out << "<PDataArray type=\"" << GetDataTypeString()
-                  << "\" Name=\"" << q_field_name
-                  << "\" NumberOfComponents=\"" << vec_dim << "\" "
-                  << "format=\"" << GetDataFormatString() << "\" />\n";
-         pvtu_out << "</PPointData>\n";
-         WritePVTUFooter(pvtu_out, q_field_name);
-
-         pvd_stream << "<DataSet timestep=\"" << GetTime()
-                    << "\" group=\"\" part=\"" << 0 << "\" file=\""
-                    << q_fname << "\" name=\"" << q_field_name << "\"/>\n";
+         // this one is generated without the path
+         std::string nfname=GenerateVTUFileName(ii);
+         out << "<Piece Source=\"" << nfname << "\"/>\n";
      }
+      out << "</PUnstructuredGrid>\n";
+      out << "</VTKFile>\n";
+      out.close();
+
+      fname = GeneratePVTUPath()+"/"+GeneratePVTUFileName();
+      // add the pvtu file to the pvd_stream
+      pvd_stream << "<DataSet timestep=\"" << GetTime();  // GetCycle();
+      pvd_stream << "\" group=\"\" part=\"" << 0 << "\" file=\"";
+      pvd_stream << fname << "\"/>\n";
      pvd_stream.flush();
-      // Move the insertion point before the closing collection tag, so that
-      // the PVD file is valid even when writing incrementally.
      std::fstream::pos_type pos = pvd_stream.tellp();
      pvd_stream << "</Collection>\n";
      pvd_stream << "</VTKFile>" << std::endl;
@@ -994,44 +996,6 @@ void ParaViewDataCollection::Save()
   }
 }

-void ParaViewDataCollection::WritePVTUHeader(std::ostream &out)
-{
-   out << "<?xml version=\"1.0\"?>\n";
-   out << "<VTKFile type=\"PUnstructuredGrid\"";
-   out << " version =\"0.1\" byte_order=\"" << VTKByteOrder() << "\">\n";
-   out << "<PUnstructuredGrid GhostLevel=\"0\">\n";
-
-   out << "<PPoints>\n";
-   out << "\t<PDataArray type=\"" << GetDataTypeString() << "\" ";
-   out << " Name=\"Points\" NumberOfComponents=\"3\""
-       << " format=\"" << GetDataFormatString() << "\"/>\n";
-   out << "</PPoints>\n";
-
-   out << "<PCells>\n";
-   out << "\t<PDataArray type=\"Int32\" ";
-   out << " Name=\"connectivity\" NumberOfComponents=\"1\""
-       << " format=\"" << GetDataFormatString() << "\"/>\n";
-   out << "\t<PDataArray type=\"Int32\" ";
-   out << " Name=\"offsets\"      NumberOfComponents=\"1\""
-       << " format=\"" << GetDataFormatString() << "\"/>\n";
-   out << "\t<PDataArray type=\"UInt8\" ";
-   out << " Name=\"types\"        NumberOfComponents=\"1\""
-       << " format=\"" << GetDataFormatString() << "\"/>\n";
-   out << "</PCells>\n";
-}
-
-void ParaViewDataCollection::WritePVTUFooter(std::ostream &out,
-                                             const std::string &vtu_prefix)
-{
-   for (int ii=0; ii<num_procs; ii++)
-   {
-      std::string vtu_filename = GenerateVTUFileName(vtu_prefix, ii);
-      out << "<Piece Source=\"" << vtu_filename << "\"/>\n";
-   }
-   out << "</PUnstructuredGrid>\n";
-   out << "</VTKFile>\n";
-}
-
 void ParaViewDataCollection::SaveDataVTU(std::ostream &out, int ref)
 {
   out << "<VTKFile type=\"UnstructuredGrid\"";
@@ -1051,6 +1015,16 @@ void ParaViewDataCollection::SaveDataVTU(std::ostream &out, int ref)
   {
      SaveGFieldVTU(out,ref,it);
   }
+   // iterate over all quadrature functions
+   // if the Quadrature functions are dumped as cell data
+   // the cycle should be moved before the grid functions
+   // and the PrintVTU CellData section should be open in the mesh dump
+   for (QFieldMapIterator it=q_field_map.begin(); it!=q_field_map.end(); ++it)
+   {
+      // save the quadrature functions
+      // this one is not implemented yet
+      SaveQFieldVTU(out,ref,it);
+   }
   out << "</PointData>\n";
   // close the mesh
   out << "</Piece>\n"; // close the piece open in the PrintVTU method
@@ -1058,21 +1032,27 @@ void ParaViewDataCollection::SaveDataVTU(std::ostream &out, int ref)
   out << "</VTKFile>" << std::endl;
 }

+void ParaViewDataCollection::SaveQFieldVTU(std::ostream &out, int ref,
+                                           const QFieldMapIterator& it )
+{
+   MFEM_WARNING("SaveQFieldVTU is not currently implemented - field name:"<<it->second);
+}
+
 void ParaViewDataCollection::SaveGFieldVTU(std::ostream &out, int ref_,
-                                           const FieldMapIterator &it)
+                                           const FieldMapIterator& it)
 {
   RefinedGeometry *RefG;
   Vector val;
   DenseMatrix vval, pmat;
   std::vector<char> buf;
   int vec_dim = it->second->VectorDim();
-   out << "<DataArray type=\"" << GetDataTypeString()
-       << "\" Name=\"" << it->first;
-   out << "\" NumberOfComponents=\"" << vec_dim << "\""
-       << " format=\"" << GetDataFormatString() << "\" >" << '\n';
   if (vec_dim == 1)
   {
      // scalar data
+      out << "<DataArray type=\"" << GetDataTypeString()
+          << "\" Name=\"" << it->first;
+      out << "\" NumberOfComponents=\"1\" format=\""
+          << GetDataFormatString() << "\" >\n";
      for (int i = 0; i < mesh->GetNE(); i++)
      {
         RefG = GlobGeometryRefiner.Refine(
@@ -1080,23 +1060,51 @@ void ParaViewDataCollection::SaveGFieldVTU(std::ostream &out, int ref_,
         it->second->GetValues(i, RefG->RefPts, val, pmat);
         for (int j = 0; j < val.Size(); j++)
         {
-            WriteBinaryOrASCII(out, buf, val(j), "\n", pv_data_format);
+            if (pv_data_format == VTKFormat::ASCII)
+            {
+               out << ZeroSubnormal(val(j)) << '\n';
+            }
+            else if (pv_data_format == VTKFormat::BINARY)
+            {
+               bin_io::AppendBytes(buf, val(j));
+            }
+            else
+            {
+               bin_io::AppendBytes<float>(buf, float(val(j)));
+            }
         }
      }
   }
   else
   {
      // vector data
+      out << "<DataArray type=\"" << GetDataTypeString()
+          << "\" Name=\"" << it->first;
+      out << "\" NumberOfComponents=\"" << vec_dim << "\""
+          << " format=\"" << GetDataFormatString() << "\" >" << '\n';
      for (int i = 0; i < mesh->GetNE(); i++)
      {
         RefG = GlobGeometryRefiner.Refine(
                   mesh->GetElementBaseGeometry(i), ref_, 1);
+
         it->second->GetVectorValues(i, RefG->RefPts, vval, pmat);
+
         for (int jj = 0; jj < vval.Width(); jj++)
         {
            for (int ii = 0; ii < vval.Height(); ii++)
            {
-               WriteBinaryOrASCII(out, buf, vval(ii,jj), " ", pv_data_format);
+               if (pv_data_format == VTKFormat::ASCII)
+               {
+                  out << ZeroSubnormal(vval(ii,jj)) << ' ';
+               }
+               else if (pv_data_format == VTKFormat::BINARY)
+               {
+                  bin_io::AppendBytes(buf, vval(ii,jj));
+               }
+               else
+               {
+                  bin_io::AppendBytes<float>(buf, float(vval(ii,jj)));
+               }
            }
            if (pv_data_format == VTKFormat::ASCII) { out << '\n'; }
         }
@@ -258,8 +258,8 @@ public:
   { q_field_map.Deregister(field_name, own_data); }

   /// Check if a grid function is part of the collection
-   bool HasField(const std::string& field_name) const
-   { return field_map.Has(field_name); }
+   bool HasField(const std::string& name) const
+   { return field_map.Has(name); }

   /// Get a pointer to a grid function in the collection.
   /** Returns NULL if @a field_name is not in the collection. */
@@ -491,20 +491,19 @@ private:
   bool restart_mode;

 protected:
-   void WritePVTUHeader(std::ostream &out);
-   void WritePVTUFooter(std::ostream &out, const std::string &vtu_prefix);
   void SaveDataVTU(std::ostream &out, int ref);
   void SaveGFieldVTU(std::ostream& out, int ref_, const FieldMapIterator& it);
+   void SaveQFieldVTU(std::ostream &out, int ref, const QFieldMapIterator& it);
   const char *GetDataFormatString() const;
   const char *GetDataTypeString() const;

-   std::string GenerateCollectionPath();
-   std::string GenerateVTUFileName(const std::string &prefix, int rank);
-   std::string GenerateVTUPath();
-   std::string GeneratePVDFileName();
-   std::string GeneratePVTUFileName(const std::string &prefix);
-   std::string GeneratePVTUPath();
-
+   std::string  GenerateCollectionPath();
+   std::string  GenerateVTUFileName();
+   std::string  GenerateVTUFileName(int rank);
+   std::string  GenerateVTUPath();
+   std::string  GeneratePVDFileName();
+   std::string  GeneratePVTUFileName();
+   std::string  GeneratePVTUPath();

 public:
   /// Constructor. The collection name is used when saving the data.
@@ -243,7 +243,6 @@ public:
   void TransformDual(double *v) const;

   void InvTransformDual(double *v) const;
-   using DofTransformation::InvTransformDual;
 };

 /// DoF transformation implementation for the Nedelec basis on tetrahedra
@@ -21,8 +21,7 @@ ElementTransformation::ElementTransformation()
     EvalState(0),
     geom(Geometry::INVALID),
     Attribute(-1),
-     ElementNo(-1),
-     mesh(nullptr)
+     ElementNo(-1)
 { }

 double ElementTransformation::EvalWeight()
@@ -75,12 +75,6 @@ public:

   int Attribute, ElementNo, ElementType;

-   /// The Mesh object containing the element.
-   /** If the element transformation belongs to a mesh, this will point to the
-       containing Mesh object. ElementNo will be the number of the element in
-       this Mesh. This will be NULL if the element does not belong to a mesh. */
-   class Mesh *mesh;
-
   ElementTransformation();

   /** @brief Force the reevaluation of the Jacobian in the next call. */
@@ -234,7 +234,7 @@ protected:
   bool own_flux_fes; ///< Ownership flag for flux_space and smooth_flux_space.

   /// Initialize with the integrator, solution, and flux finite element spaces.
-   void Init(BilinearFormIntegrator &integ_,
+   void Init(BilinearFormIntegrator &integ,
             ParGridFunction &sol,
             ParFiniteElementSpace *flux_fes,
             ParFiniteElementSpace *smooth_flux_fes)
@@ -242,7 +242,7 @@ protected:
      current_sequence = -1;
      local_norm_p = 1;
      total_error = 0.0;
-      integ = &integ_;
+      this->integ = &integ;
      solution = &sol;
      flux_space = flux_fes;
      smooth_flux_space = smooth_flux_fes;
@@ -1538,7 +1538,7 @@ void VectorFiniteElement::LocalRestriction_ND(


 Poly_1D::Basis::Basis(const int p, const double *nodes, EvalType etype)
-   : etype(etype), auxiliary_basis(NULL), scale_integrated(false)
+   : etype(etype), auxiliary_basis(NULL)
 {
   switch (etype)
   {
@@ -1838,29 +1838,11 @@ void Poly_1D::Basis::EvalIntegrated(const Vector &d_aux, Vector &u) const
   MFEM_VERIFY(etype == Integrated,
               "EvalIntegrated is only valid for Integrated basis type");
   int p = d_aux.Size() - 1;
-   // See Gerritsma, M. (2010).  "Edge functions for spectral element methods",
-   // in Lecture Notes in Computational Science and Engineering, 199--207.
   u[0] = -d_aux[0];
   for (int j=1; j<p; ++j)
   {
      u[j] = u[j-1] - d_aux[j];
   }
-   // If scale_integrated is true, the degrees of freedom represent mean values,
-   // otherwise they represent subcell integrals. Generally, scale_integrated
-   // should be true for MapType::VALUE, and false for other map types.
-   if (scale_integrated)
-   {
-      Vector &aux_nodes = auxiliary_basis->x;
-      for (int j=0; j<aux_nodes.Size()-1; ++j)
-      {
-         u[j] *= aux_nodes[j+1] - aux_nodes[j];
-      }
-   }
-}
-
-void Poly_1D::Basis::ScaleIntegrated(bool scale_integrated_)
-{
-   scale_integrated = scale_integrated_;
 }

 Poly_1D::Basis::~Basis()
@@ -2397,18 +2379,6 @@ NodalTensorFiniteElement::NodalTensorFiniteElement(const int dims,
   lex_ordering = dof_map;
 }

-void NodalTensorFiniteElement::SetMapType(const int map_type)
-{
-   ScalarFiniteElement::SetMapType(map_type);
-   // If we are using the "integrated" basis, the basis functions should be
-   // scaled for MapType::VALUE, and not scaled for MapType::INTEGRAL. This
-   // ensures spectral equivalence of the mass matrix with its low-order-refined
-   // counterpart (cf. LORDiscretization)
-   if (basis1d.IsIntegratedType())
-   {
-      basis1d.ScaleIntegrated(map_type == VALUE);
-   }
-}

 VectorTensorFiniteElement::VectorTensorFiniteElement(const int dims,
                                                     const int d,
@@ -657,7 +657,7 @@ public:
   /** @brief Set the FiniteElement::MapType of the element to either VALUE or
       INTEGRAL. Also sets the FiniteElement::DerivType to GRAD if the
       FiniteElement::MapType is VALUE. */
-   virtual void SetMapType(int M)
+   void SetMapType(int M)
   {
      MFEM_VERIFY(M == VALUE || M == INTEGRAL, "unknown MapType");
      map_type = M;
@@ -955,68 +955,41 @@ public:
 };


-/// @brief Class for computing 1D special polynomials and their associated basis
+/// Class for computing 1D special polynomials and their associated basis
 /// functions
 class Poly_1D
 {
 public:
-   /// One-dimensional basis evaluation type
   enum EvalType
   {
-      ChangeOfBasis = 0, ///< Use change of basis, O(p^2) Evals
-      Barycentric   = 1, ///< Use barycentric Lagrangian interpolation, O(p) Evals
-      Positive      = 2, ///< Fast evaluation of Bernstein polynomials
-      Integrated    = 3, ///< Integrated indicator functions (cf. Gerritsma)
-      NumEvalTypes  = 4  ///< Keep count of the number of eval types
+      ChangeOfBasis = 0, // Use change of basis, O(p^2) Evals
+      Barycentric   = 1, // Use barycentric Lagrangian interpolation, O(p) Evals
+      Positive      = 2, // Fast evaluation of Bernstein polynomials
+      Integrated    = 3, // Integrated indicator functions (cf. Gerritsma)
+      NumEvalTypes  = 4  // Keep count of the number of eval types
   };

-   /// @brief Class for evaluating 1D nodal, positive (Bernstein), or integrated
-   /// (Gerritsma) bases.
   class Basis
   {
   private:
-      EvalType etype; ///< Determines how the basis functions should be evaluated.
+      int etype;
      DenseMatrixInverse Ai;
      mutable Vector x, w;
-      /// The following data members are used for "integrated basis type", which
-      /// is defined in terms of nodal basis of one degree higher.
-      ///@{
+      // The following data members are used for "integrated basis type", which
+      // is defined in terms of nodal basis of one degree higher.
      mutable Vector u_aux, d_aux, d2_aux;
-      ///@}
-      /// @brief An auxiliary nodal basis used to evaluate the integrated basis.
-      /// This member variable is NULL whenever etype != Integrated.
-      Basis *auxiliary_basis;
-      /// Should the integrated basis functions be scaled? See ScaleIntegrated.
-      bool scale_integrated;
+      Basis *auxiliary_basis; // Non-NULL only for etype == Integrated

   public:
-      /// Create a nodal or positive (Bernstein) basis of degree @a p
+      /// Create a nodal or positive (Bernstein) basis
      Basis(const int p, const double *nodes, EvalType etype = Barycentric);
-      /// Evaluate the basis functions at point @a x in [0,1]
      void Eval(const double x, Vector &u) const;
-      /// @brief Evaluate the basis functions and their derivatives at point @a
-      /// x in [0,1]
      void Eval(const double x, Vector &u, Vector &d) const;
-      /// @brief Evaluate the basis functions and their first two derivatives at
-      /// point @a x in [0,1]
      void Eval(const double x, Vector &u, Vector &d, Vector &d2) const;
-      /// @brief Evaluate the "integrated" basis type using pre-computed closed
-      /// basis derivatives.
-      ///
-      /// This basis is given by the negative partial sum of the corresponding
-      /// closed basis derivatives. The closed basis derivatives are given by @a
-      /// d, and the result is stored in @a i.
+      /// Evaluate the "integrated" basis, which is given by the negative
+      /// partial sum of the corresponding closed basis derivatives. The closed
+      /// basis derivatives are given by @a d, and the result is stored in @a i.
      void EvalIntegrated(const Vector &d, Vector &i) const;
-      /// @brief Set whether the "integrated" basis should be scaled by the
-      /// subcell sizes. Has no effect for non-integrated bases.
-      ///
-      /// Generally, this should be true for mfem::FiniteElement::MapType VALUE
-      /// and false for all other map types. If this option is enabled, the
-      /// basis functions will be scaled by the widths of the subintervals, so
-      /// that the basis functions represent mean values. Otherwise, the basis
-      /// functions represent integrated values.
-      void ScaleIntegrated(bool scale_integrated_);
-      /// Returns true if the basis is "integrated", false otherwise.
      bool IsIntegratedType() const { return etype == Integrated; }
      ~Basis();
   };
@@ -1220,8 +1193,6 @@ public:
             ScalarFiniteElement::GetTensorDofToQuad(*this, ir, mode);
   }

-   virtual void SetMapType(const int map_type_);
-
   virtual void GetTransferMatrix(const FiniteElement &fe,
                                  ElementTransformation &Trans,
                                  DenseMatrix &I) const
@@ -1897,21 +1897,21 @@ const int *H1_FECollection::DofOrderForOrientation(Geometry::Type GeomType,

 FiniteElementCollection *H1_FECollection::GetTraceCollection() const
 {
-   int tr_p = H1_dof[Geometry::SEGMENT] + 1;
-   int tr_dim = -1;
+   int p = H1_dof[Geometry::SEGMENT] + 1;
+   int dim = -1;
   if (!strncmp(h1_name, "H1_", 3))
   {
-      tr_dim = atoi(h1_name + 3);
+      dim = atoi(h1_name + 3);
   }
   else if (!strncmp(h1_name, "H1Pos_", 6))
   {
-      tr_dim = atoi(h1_name + 6);
+      dim = atoi(h1_name + 6);
   }
   else if (!strncmp(h1_name, "H1@", 3))
   {
-      tr_dim = atoi(h1_name + 5);
+      dim = atoi(h1_name + 5);
   }
-   return (dim < 0) ? NULL : new H1_Trace_FECollection(tr_p, tr_dim, b_type);
+   return (dim < 0) ? NULL : new H1_Trace_FECollection(p, dim, b_type);
 }

 const int *H1_FECollection::GetDofMap(Geometry::Type GeomType) const
@@ -2374,7 +2374,7 @@ RT_FECollection::RT_FECollection(const int p, const int dim,
   InitFaces(p, dim, map_type, signs);
 }

-void RT_FECollection::InitFaces(const int p, const int dim_,
+void RT_FECollection::InitFaces(const int p, const int dim,
                                const int map_type,
                                const bool signs)
 {
@@ -2404,7 +2404,7 @@ void RT_FECollection::InitFaces(const int p, const int dim_,
      QuadDofOrd[i] = NULL;
   }

-   if (dim_ == 2)
+   if (dim == 2)
   {
      L2_SegmentElement *l2_seg = new L2_SegmentElement(p, ob_type);
      l2_seg->SetMapType(map_type);
@@ -2419,7 +2419,7 @@ void RT_FECollection::InitFaces(const int p, const int dim_,
         SegDofOrd[1][i] = signs ? (-1 - (p - i)) : (p - i);
      }
   }
-   else if (dim_ == 3)
+   else if (dim == 3)
   {
      L2_TriangleElement *l2_tri = new L2_TriangleElement(p, ob_type);
      l2_tri->SetMapType(map_type);
@@ -2453,9 +2453,9 @@ void RT_FECollection::InitFaces(const int p, const int dim_,
            TriDofOrd[5][o] = -1-(TriDof-((pp2-i)*(pp1-i))/2+j);  // (0,2,1)
            if (!signs)
            {
-               for (int kk = 1; kk < 6; kk += 2)
+               for (int k = 1; k < 6; k += 2)
               {
-                  TriDofOrd[kk][o] = -1 - TriDofOrd[kk][o];
+                  TriDofOrd[k][o] = -1 - TriDofOrd[k][o];
               }
            }
         }
@@ -2529,19 +2529,18 @@ const int *RT_FECollection::DofOrderForOrientation(Geometry::Type GeomType,

 FiniteElementCollection *RT_FECollection::GetTraceCollection() const
 {
-   int tr_dim, tr_p;
+   int dim, p;
   if (!strncmp(rt_name, "RT_", 3))
   {
-      tr_dim = atoi(rt_name + 3);
-      tr_p = atoi(rt_name + 7);
+      dim = atoi(rt_name + 3);
+      p = atoi(rt_name + 7);
   }
   else // rt_name = RT@.._.D_P*
   {
-      tr_dim = atoi(rt_name + 6);
-      tr_p = atoi(rt_name + 10);
+      dim = atoi(rt_name + 6);
+      p = atoi(rt_name + 10);
   }
-   return new RT_Trace_FECollection(tr_p, tr_dim, FiniteElement::INTEGRAL,
-                                    ob_type);
+   return new RT_Trace_FECollection(p, dim, FiniteElement::INTEGRAL, ob_type);
 }

 RT_FECollection::~RT_FECollection()
@@ -2815,22 +2814,22 @@ const int *ND_FECollection::DofOrderForOrientation(Geometry::Type GeomType,

 FiniteElementCollection *ND_FECollection::GetTraceCollection() const
 {
-   int tr_p, tr_dim, tr_cb_type, tr_ob_type;
+   int p, dim, cb_type, ob_type;

-   tr_p = ND_dof[Geometry::SEGMENT];
+   p = ND_dof[Geometry::SEGMENT];
   if (nd_name[2] == '_') // ND_
   {
-      tr_dim = atoi(nd_name + 3);
-      tr_cb_type = BasisType::GaussLobatto;
-      tr_ob_type = BasisType::GaussLegendre;
+      dim = atoi(nd_name + 3);
+      cb_type = BasisType::GaussLobatto;
+      ob_type = BasisType::GaussLegendre;
   }
   else // ND@
   {
-      tr_dim = atoi(nd_name + 6);
-      tr_cb_type = BasisType::GetType(nd_name[3]);
-      tr_ob_type = BasisType::GetType(nd_name[4]);
+      dim = atoi(nd_name + 6);
+      cb_type = BasisType::GetType(nd_name[3]);
+      ob_type = BasisType::GetType(nd_name[4]);
   }
-   return new ND_Trace_FECollection(tr_p, tr_dim, tr_cb_type, tr_ob_type);
+   return new ND_Trace_FECollection(p, dim, cb_type, ob_type);
 }

 ND_FECollection::~ND_FECollection()
@@ -1117,7 +1117,7 @@ public:
   { return (GeomType == GeomType_) ? Local_Element : NULL; }
   virtual int DofForGeometry(Geometry::Type GeomType_) const
   { return (GeomType == GeomType_) ? Local_Element->GetDof() : 0; }
-   virtual const int *DofOrderForOrientation(Geometry::Type GeomType_,
+   virtual const int *DofOrderForOrientation(Geometry::Type GeomType,
                                             int Or) const
   { return NULL; }
   virtual const char *Name() const { return d_name; }
@@ -70,14 +70,14 @@ FiniteElementSpace::FiniteElementSpace()
 { }

 FiniteElementSpace::FiniteElementSpace(const FiniteElementSpace &orig,
-                                       Mesh *mesh_,
-                                       const FiniteElementCollection *fec_)
+                                       Mesh *mesh,
+                                       const FiniteElementCollection *fec)
   : VDoFTrans(orig.vdim, orig.ordering)
 {
-   mesh_ = mesh_ ? mesh_ : orig.mesh;
-   fec_ = fec_ ? fec_ : orig.fec;
+   mesh = mesh ? mesh : orig.mesh;
+   fec = fec ? fec : orig.fec;

-   NURBSExtension *nurbs_ext = NULL;
+   NURBSExtension *NURBSext = NULL;
   if (orig.NURBSext && orig.NURBSext != orig.mesh->NURBSext)
   {
 #ifdef MFEM_USE_MPI
@@ -85,16 +85,16 @@ FiniteElementSpace::FiniteElementSpace(const FiniteElementSpace &orig,
         dynamic_cast<ParNURBSExtension *>(orig.NURBSext);
      if (pNURBSext)
      {
-         nurbs_ext = new ParNURBSExtension(*pNURBSext);
+         NURBSext = new ParNURBSExtension(*pNURBSext);
      }
      else
 #endif
      {
-         nurbs_ext = new NURBSExtension(*orig.NURBSext);
+         NURBSext = new NURBSExtension(*orig.NURBSext);
      }
   }

-   Constructor(mesh_, nurbs_ext, fec_, orig.vdim, orig.ordering);
+   Constructor(mesh, NURBSext, fec, orig.vdim, orig.ordering);
 }

 void FiniteElementSpace::CopyProlongationAndRestriction(
@@ -192,74 +192,74 @@ int FiniteElementSpace::GetElementOrderImpl(int i) const
   return elem_order.Size() ? elem_order[i] : fec->GetOrder();
 }

-void FiniteElementSpace::GetVDofs(int vd, Array<int>& dofs, int ndofs_) const
+void FiniteElementSpace::GetVDofs(int vd, Array<int>& dofs, int ndofs) const
 {
-   if (ndofs_ < 0) { ndofs_ = this->ndofs; }
+   if (ndofs < 0) { ndofs = this->ndofs; }

   if (ordering == Ordering::byNODES)
   {
      for (int i = 0; i < dofs.Size(); i++)
      {
-         dofs[i] = Ordering::Map<Ordering::byNODES>(ndofs_, vdim, i, vd);
+         dofs[i] = Ordering::Map<Ordering::byNODES>(ndofs, vdim, i, vd);
      }
   }
   else
   {
      for (int i = 0; i < dofs.Size(); i++)
      {
-         dofs[i] = Ordering::Map<Ordering::byVDIM>(ndofs_, vdim, i, vd);
+         dofs[i] = Ordering::Map<Ordering::byVDIM>(ndofs, vdim, i, vd);
      }
   }
 }

-void FiniteElementSpace::DofsToVDofs (Array<int> &dofs, int ndofs_) const
+void FiniteElementSpace::DofsToVDofs (Array<int> &dofs, int ndofs) const
 {
   if (vdim == 1) { return; }
-   if (ndofs_ < 0) { ndofs_ = this->ndofs; }
+   if (ndofs < 0) { ndofs = this->ndofs; }

   if (ordering == Ordering::byNODES)
   {
-      Ordering::DofsToVDofs<Ordering::byNODES>(ndofs_, vdim, dofs);
+      Ordering::DofsToVDofs<Ordering::byNODES>(ndofs, vdim, dofs);
   }
   else
   {
-      Ordering::DofsToVDofs<Ordering::byVDIM>(ndofs_, vdim, dofs);
+      Ordering::DofsToVDofs<Ordering::byVDIM>(ndofs, vdim, dofs);
   }
 }

-void FiniteElementSpace::DofsToVDofs(int vd, Array<int> &dofs, int ndofs_) const
+void FiniteElementSpace::DofsToVDofs(int vd, Array<int> &dofs, int ndofs) const
 {
   if (vdim == 1) { return; }
-   if (ndofs_ < 0) { ndofs_ = this->ndofs; }
+   if (ndofs < 0) { ndofs = this->ndofs; }

   if (ordering == Ordering::byNODES)
   {
      for (int i = 0; i < dofs.Size(); i++)
      {
-         dofs[i] = Ordering::Map<Ordering::byNODES>(ndofs_, vdim, dofs[i], vd);
+         dofs[i] = Ordering::Map<Ordering::byNODES>(ndofs, vdim, dofs[i], vd);
      }
   }
   else
   {
      for (int i = 0; i < dofs.Size(); i++)
      {
-         dofs[i] = Ordering::Map<Ordering::byVDIM>(ndofs_, vdim, dofs[i], vd);
+         dofs[i] = Ordering::Map<Ordering::byVDIM>(ndofs, vdim, dofs[i], vd);
      }
   }
 }

-int FiniteElementSpace::DofToVDof(int dof, int vd, int ndofs_) const
+int FiniteElementSpace::DofToVDof(int dof, int vd, int ndofs) const
 {
   if (vdim == 1) { return dof; }
-   if (ndofs_ < 0) { ndofs_ = this->ndofs; }
+   if (ndofs < 0) { ndofs = this->ndofs; }

   if (ordering == Ordering::byNODES)
   {
-      return Ordering::Map<Ordering::byNODES>(ndofs_, vdim, dof, vd);
+      return Ordering::Map<Ordering::byNODES>(ndofs, vdim, dof, vd);
   }
   else
   {
-      return Ordering::Map<Ordering::byVDIM>(ndofs_, vdim, dof, vd);
+      return Ordering::Map<Ordering::byVDIM>(ndofs, vdim, dof, vd);
   }
 }

@@ -561,6 +561,155 @@ void FiniteElementSpace::GetEssentialVDofs(const Array<int> &bdr_attr_is_ess,
   }
 }

+void FiniteElementSpace::GetEssentialVDofs(EntitySets::EntityType type,
+                                           int set_index,
+                                           Array<int> &ess_vdofs,
+                                           int component) const
+{
+   Array<int> vdofs, dofs;
+
+   ess_vdofs.SetSize(GetVSize());
+   ess_vdofs = 0;
+
+   MFEM_VERIFY(mesh->ent_sets != NULL, "Mesh object contains no "
+               "entity set information");
+   if (!mesh->ent_sets->SetExists(type, set_index))
+   {
+      ostringstream oss; oss << "Entity set of type \""
+                             << EntitySets::GetTypeName(type)
+                             << "\" and index " << set_index
+                             << " was not found.";
+
+      MFEM_VERIFY(false, oss.str().c_str());
+   }
+
+   set<int>::iterator it;
+   for (it=(*mesh->ent_sets)(type, set_index).begin();
+        it!=(*mesh->ent_sets)(type, set_index).end(); it++)
+   {
+      int ent_index = *it;
+      cout << "collecting vdofs for entity " << ent_index << "->";
+      if (component < 0)
+      {
+         switch (type)
+         {
+            case EntitySets::VERTEX:
+               GetVertexVDofs(ent_index, vdofs);
+               break;
+            case EntitySets::EDGE:
+               GetEdgeVDofs(ent_index, vdofs);
+               break;
+            case EntitySets::FACE:
+               GetFaceVDofs(ent_index, vdofs);
+               break;
+            case EntitySets::ELEMENT:
+               GetElementVDofs(ent_index, vdofs);
+               break;
+            default:
+               mfem_error("GetEssentialVDofs: Invalid entity type");
+         }
+         vdofs.Print(cout);
+         mark_dofs(vdofs, ess_vdofs);
+      }
+      else
+      {
+         switch (type)
+         {
+            case EntitySets::VERTEX:
+               GetVertexDofs(ent_index, dofs);
+               break;
+            case EntitySets::EDGE:
+               GetEdgeDofs(ent_index, dofs);
+               break;
+            case EntitySets::FACE:
+               GetFaceDofs(ent_index, dofs);
+               break;
+            case EntitySets::ELEMENT:
+               GetElementDofs(ent_index, dofs);
+               break;
+            default:
+               mfem_error("GetEssentialDofs: Invalid entity type");
+         }
+         for (int d = 0; d < dofs.Size(); d++)
+         { dofs[d] = DofToVDof(dofs[d], component); }
+         mark_dofs(dofs, ess_vdofs);
+      }
+   }
+
+   if (mesh->ncmesh)
+   {
+      Array<int> es_verts, es_edges, es_faces;
+      mesh->ncmesh->GetEntitySetClosure(type, set_index,
+                                        es_verts, es_edges, es_faces);
+      cout << "returned from get closure" << endl;
+      for (int i = 0; i < es_verts.Size(); i++)
+      {
+         if (es_verts[i] < GetNV())
+         {
+            if (component < 0)
+            {
+               GetVertexVDofs(es_verts[i], vdofs);
+               mark_dofs(vdofs, ess_vdofs);
+            }
+            else
+            {
+               GetVertexDofs(es_verts[i], dofs);
+               for (int d = 0; d < dofs.Size(); d++)
+               { dofs[d] = DofToVDof(dofs[d], component); }
+               mark_dofs(dofs, ess_vdofs);
+            }
+         }
+      }
+      for (int i = 0; i < es_edges.Size(); i++)
+      {
+         if (es_edges[i] < GetMesh()->GetNEdges())
+         {
+            if (component < 0)
+            {
+               GetEdgeVDofs(es_edges[i], vdofs);
+               mark_dofs(vdofs, ess_vdofs);
+            }
+            else
+            {
+               GetEdgeDofs(es_edges[i], dofs);
+               for (int d = 0; d < dofs.Size(); d++)
+               { dofs[d] = DofToVDof(dofs[d], component); }
+               mark_dofs(dofs, ess_vdofs);
+            }
+         }
+      }
+      for (int i = 0; i < es_faces.Size(); i++)
+      {
+         if (es_faces[i] < GetMesh()->GetNFaces())
+         {
+            if (component < 0)
+            {
+               GetFaceVDofs(es_faces[i], vdofs);
+               mark_dofs(vdofs, ess_vdofs);
+            }
+            else
+            {
+               GetFaceDofs(es_faces[i], dofs);
+               for (int d = 0; d < dofs.Size(); d++)
+               { dofs[d] = DofToVDof(dofs[d], component); }
+               mark_dofs(dofs, ess_vdofs);
+            }
+         }
+      }
+   }
+}
+
+void FiniteElementSpace::GetEssentialVDofs(EntitySets::EntityType type,
+                                           const string & set_name,
+                                           Array<int> &ess_vdofs,
+                                           int component) const
+{
+   MFEM_VERIFY(mesh->ent_sets != NULL, "Mesh object contains no "
+               "entity set information");
+   GetEssentialVDofs(type, mesh->ent_sets->GetSetIndex(type, set_name),
+                     ess_vdofs, component);
+}
+
 void FiniteElementSpace::GetEssentialTrueDofs(const Array<int> &bdr_attr_is_ess,
                                              Array<int> &ess_tdof_list,
                                              int component)
@@ -579,6 +728,36 @@ void FiniteElementSpace::GetEssentialTrueDofs(const Array<int> &bdr_attr_is_ess,
   MarkerToList(ess_tdofs, ess_tdof_list);
 }

+void FiniteElementSpace::GetEssentialTrueDofs(EntitySets::EntityType type,
+                                              int set_index,
+                                              Array<int> &ess_tdof_list,
+                                              int component)
+{
+   Array<int> ess_vdofs, ess_tdofs;
+   GetEssentialVDofs(type, set_index, ess_vdofs, component);
+   const SparseMatrix *R = GetConformingRestriction();
+   if (!R)
+   {
+      ess_tdofs.MakeRef(ess_vdofs);
+   }
+   else
+   {
+      R->BooleanMult(ess_vdofs, ess_tdofs);
+   }
+   MarkerToList(ess_tdofs, ess_tdof_list);
+}
+
+void FiniteElementSpace::GetEssentialTrueDofs(EntitySets::EntityType type,
+                                              const string & set_name,
+                                              Array<int> &ess_tdof_list,
+                                              int component)
+{
+   MFEM_VERIFY(mesh->ent_sets != NULL, "Mesh object contains no "
+               "entity set information");
+   GetEssentialTrueDofs(type, mesh->ent_sets->GetSetIndex(type, set_name),
+                        ess_tdof_list, component);
+}
+
 void FiniteElementSpace::GetBoundaryTrueDofs(Array<int> &boundary_dofs,
                                             int component)
 {
@@ -713,8 +892,8 @@ FiniteElementSpace::H2L_GlobalRestrictionMatrix (FiniteElementSpace *lfes)
   DenseMatrix loc_restr;
   Array<int> l_dofs, h_dofs, l_vdofs, h_vdofs;

-   int lvdim = lfes->GetVDim();
-   R = new SparseMatrix (lvdim * lfes -> GetNDofs(), lvdim * ndofs);
+   int vdim = lfes->GetVDim();
+   R = new SparseMatrix (vdim * lfes -> GetNDofs(), vdim * ndofs);

   Geometry::Type cached_geom = Geometry::INVALID;
   const FiniteElement *h_fe = NULL;
@@ -737,7 +916,7 @@ FiniteElementSpace::H2L_GlobalRestrictionMatrix (FiniteElementSpace *lfes)
         cached_geom = geom;
      }

-      for (int vd = 0; vd < lvdim; vd++)
+      for (int vd = 0; vd < vdim; vd++)
      {
         l_dofs.Copy(l_vdofs);
         lfes->DofsToVDofs(vd, l_vdofs);
@@ -894,10 +1073,9 @@ int FiniteElementSpace::GetDegenerateFaceDofs(int index, Array<int> &dofs,
 int FiniteElementSpace::GetNumBorderDofs(Geometry::Type geom, int order) const
 {
   // return the number of vertex and edge DOFs that precede inner DOFs
-   const int nv = fec->GetNumDof(Geometry::POINT, order);
-   const int ne = fec->GetNumDof(Geometry::SEGMENT, order);
-
-   return Geometry::NumVerts[geom] * (geom == Geometry::SEGMENT ? nv : (nv + ne));
+   int nv = fec->GetNumDof(Geometry::POINT, order);
+   int ne = fec->GetNumDof(Geometry::SEGMENT, order);
+   return Geometry::NumVerts[geom] * (nv + ne);
 }

 int FiniteElementSpace::GetEntityDofs(int entity, int index, Array<int> &dofs,
@@ -935,12 +1113,6 @@ void FiniteElementSpace::BuildConformingInterpolation() const
   if (cP_is_set) { return; }
   cP_is_set = true;

-   if (FEColl()->GetContType() == FiniteElementCollection::DISCONTINUOUS)
-   {
-      cP = cR = cR_hp = NULL; // will be treated as identities
-      return;
-   }
-
   Array<int> master_dofs, slave_dofs, highest_dofs;

   IsoparametricTransformation T;
@@ -1061,7 +1233,6 @@ void FiniteElementSpace::BuildConformingInterpolation() const
            // get lowest order variant DOFs and FE
            int p = GetEntityDofs(entity, i, master_dofs, geom, 0);
            const auto *master_fe = fec->GetFE(geom, p);
-            if (!master_fe) { break; }

            // constrain all higher order DOFs: interpolate lowest order function
            for (int variant = 1; ; variant++)
@@ -1200,7 +1371,7 @@ void FiniteElementSpace::BuildConformingInterpolation() const
      if (cR_hp) { MakeVDimMatrix(*cR_hp); }
   }

-   cP->EnsureMultTranspose();
+   if (Device::IsEnabled()) { cP->BuildTranspose(); }
 }

 void FiniteElementSpace::MakeVDimMatrix(SparseMatrix &mat) const
@@ -1308,14 +1479,7 @@ const FaceRestriction *FiniteElementSpace::GetFaceRestriction(
      FaceRestriction *res;
      if (is_dg_space)
      {
-         if (Conforming())
-         {
-            res = new L2FaceRestriction(*this, e_ordering, type, m);
-         }
-         else
-         {
-            res = new NCL2FaceRestriction(*this, e_ordering, type, m);
-         }
+         res = new L2FaceRestriction(*this, e_ordering, type, m);
      }
      else
      {
@@ -1555,11 +1719,11 @@ void FiniteElementSpace::RefinementOperator
      old_DoFTrans[i] = NULL;
   }

-   const FiniteElementCollection *fec_ref = fespace->FEColl();
-   if (dynamic_cast<const ND_FECollection*>(fec_ref))
+   const FiniteElementCollection *fec = fespace->FEColl();
+   if (dynamic_cast<const ND_FECollection*>(fec))
   {
      const FiniteElement * nd_tri =
-         fec_ref->FiniteElementForGeometry(Geometry::TRIANGLE);
+         fec->FiniteElementForGeometry(Geometry::TRIANGLE);
      if (nd_tri)
      {
         old_DoFTrans[Geometry::TRIANGLE] =
@@ -1567,7 +1731,7 @@ void FiniteElementSpace::RefinementOperator
      }

      const FiniteElement * nd_tet =
-         fec_ref->FiniteElementForGeometry(Geometry::TETRAHEDRON);
+         fec->FiniteElementForGeometry(Geometry::TETRAHEDRON);
      if (nd_tet)
      {
         old_DoFTrans[Geometry::TETRAHEDRON] =
@@ -1579,21 +1743,20 @@ void FiniteElementSpace::RefinementOperator
 void FiniteElementSpace::RefinementOperator
 ::Mult(const Vector &x, Vector &y) const
 {
-   Mesh* mesh_ref = fespace->GetMesh();
-   const CoarseFineTransformations &trans_ref =
-      mesh_ref->GetRefinementTransforms();
+   Mesh* mesh = fespace->GetMesh();
+   const CoarseFineTransformations &rtrans = mesh->GetRefinementTransforms();

   Array<int> dofs, vdofs, old_dofs, old_vdofs, old_Fo;

-   int rvdim = fespace->GetVDim();
-   int old_ndofs = width / rvdim;
+   int vdim = fespace->GetVDim();
+   int old_ndofs = width / vdim;

   Vector subY, subX;

-   for (int k = 0; k < mesh_ref->GetNE(); k++)
+   for (int k = 0; k < mesh->GetNE(); k++)
   {
-      const Embedding &emb = trans_ref.embeddings[k];
-      const Geometry::Type geom = mesh_ref->GetElementBaseGeometry(k);
+      const Embedding &emb = rtrans.embeddings[k];
+      const Geometry::Type geom = mesh->GetElementBaseGeometry(k);
      const DenseMatrix &lP = localP[geom](emb.matrix);

      subY.SetSize(lP.Height());
@@ -1603,7 +1766,7 @@ void FiniteElementSpace::RefinementOperator

      if (!doftrans)
      {
-         for (int vd = 0; vd < rvdim; vd++)
+         for (int vd = 0; vd < vdim; vd++)
         {
            dofs.Copy(vdofs);
            fespace->DofsToVDofs(vd, vdofs);
@@ -1628,7 +1791,7 @@ void FiniteElementSpace::RefinementOperator
            doftrans = vdoftrans->GetDofTransformation();
         }

-         for (int vd = 0; vd < rvdim; vd++)
+         for (int vd = 0; vd < vdim; vd++)
         {
            dofs.Copy(vdofs);
            fespace->DofsToVDofs(vd, vdofs);
@@ -1654,24 +1817,23 @@ void FiniteElementSpace::RefinementOperator
 {
   y = 0.0;

-   Mesh* mesh_ref = fespace->GetMesh();
-   const CoarseFineTransformations &trans_ref =
-      mesh_ref->GetRefinementTransforms();
+   Mesh* mesh = fespace->GetMesh();
+   const CoarseFineTransformations &rtrans = mesh->GetRefinementTransforms();

   Array<char> processed(fespace->GetVSize());
   processed = 0;

   Array<int> f_dofs, c_dofs, f_vdofs, c_vdofs, old_Fo;

-   int rvdim = fespace->GetVDim();
-   int old_ndofs = width / rvdim;
+   int vdim = fespace->GetVDim();
+   int old_ndofs = width / vdim;

-   Vector subY, subX, subYt;
+   Vector subY, subX, subYt, subXt;

-   for (int k = 0; k < mesh_ref->GetNE(); k++)
+   for (int k = 0; k < mesh->GetNE(); k++)
   {
-      const Embedding &emb = trans_ref.embeddings[k];
-      const Geometry::Type geom = mesh_ref->GetElementBaseGeometry(k);
+      const Embedding &emb = rtrans.embeddings[k];
+      const Geometry::Type geom = mesh->GetElementBaseGeometry(k);
      const DenseMatrix &lP = localP[geom](emb.matrix);

      DofTransformation * doftrans = fespace->GetElementDofs(k, f_dofs);
@@ -1681,7 +1843,7 @@ void FiniteElementSpace::RefinementOperator
      {
         subY.SetSize(lP.Width());

-         for (int vd = 0; vd < rvdim; vd++)
+         for (int vd = 0; vd < vdim; vd++)
         {
            f_dofs.Copy(f_vdofs);
            fespace->DofsToVDofs(vd, f_vdofs);
@@ -1718,7 +1880,7 @@ void FiniteElementSpace::RefinementOperator
            doftrans = vdoftrans->GetDofTransformation();
         }

-         for (int vd = 0; vd < rvdim; vd++)
+         for (int vd = 0; vd < vdim; vd++)
         {
            f_dofs.Copy(f_vdofs);
            fespace->DofsToVDofs(vd, f_vdofs);
@@ -1977,16 +2139,15 @@ void FiniteElementSpace::DerefinementOperator
   Array<int> c_vdofs, f_vdofs;
   Vector loc_x, loc_y;
   DenseMatrix loc_x_mat, loc_y_mat;
-   const int fine_vdim = fine_fes->GetVDim();
-   const int coarse_ndofs = height/fine_vdim;
+   const int vdim = fine_fes->GetVDim();
+   const int coarse_ndofs = height/vdim;
   for (int coarse_el = 0; coarse_el < coarse_to_fine.Size(); coarse_el++)
   {
      coarse_elem_dof->GetRow(coarse_el, c_vdofs);
      fine_fes->DofsToVDofs(c_vdofs, coarse_ndofs);
      loc_y.SetSize(c_vdofs.Size());
      loc_y = 0.0;
-      loc_y_mat.UseExternalData(loc_y.GetData(), c_vdofs.Size()/fine_vdim,
-                                fine_vdim);
+      loc_y_mat.UseExternalData(loc_y.GetData(), c_vdofs.Size()/vdim, vdim);
      const int ref_type = coarse_to_ref_type[coarse_el];
      const Geometry::Type geom = ref_type_to_geom[ref_type];
      const int *fine_elems = coarse_to_fine.GetRow(coarse_el);
@@ -1997,8 +2158,7 @@ void FiniteElementSpace::DerefinementOperator
         const DenseMatrix &lR = localR[geom](lR_offset+s);
         fine_fes->GetElementVDofs(fine_elems[s], f_vdofs);
         x.GetSubVector(f_vdofs, loc_x);
-         loc_x_mat.UseExternalData(loc_x.GetData(), f_vdofs.Size()/fine_vdim,
-                                   fine_vdim);
+         loc_x_mat.UseExternalData(loc_x.GetData(), f_vdofs.Size()/vdim, vdim);
         AddMult(lR, loc_x_mat, loc_y_mat);
      }
      y.SetSubVector(c_vdofs, loc_y);
@@ -2130,14 +2290,14 @@ void FiniteElementSpace::GetLocalRefinementMatrices(
   }
 }

-void FiniteElementSpace::Constructor(Mesh *mesh_, NURBSExtension *NURBSext_,
-                                     const FiniteElementCollection *fec_,
-                                     int vdim_, int ordering_)
+void FiniteElementSpace::Constructor(Mesh *mesh, NURBSExtension *NURBSext,
+                                     const FiniteElementCollection *fec,
+                                     int vdim, int ordering)
 {
-   mesh = mesh_;
-   fec = fec_;
-   vdim = vdim_;
-   ordering = (Ordering::Type) ordering_;
+   this->mesh = mesh;
+   this->fec = fec;
+   this->vdim = vdim;
+   this->ordering = (Ordering::Type) ordering;

   elem_dof = NULL;
   elem_fos = NULL;
@@ -2150,19 +2310,19 @@ void FiniteElementSpace::Constructor(Mesh *mesh_, NURBSExtension *NURBSext_,
   Th.SetType(Operator::ANY_TYPE);

   const NURBSFECollection *nurbs_fec =
-      dynamic_cast<const NURBSFECollection *>(fec_);
+      dynamic_cast<const NURBSFECollection *>(fec);
   if (nurbs_fec)
   {
-      MFEM_VERIFY(mesh_->NURBSext, "NURBS FE space requires a NURBS mesh.");
+      MFEM_VERIFY(mesh->NURBSext, "NURBS FE space requires a NURBS mesh.");

-      if (NURBSext_ == NULL)
+      if (NURBSext == NULL)
      {
-         NURBSext = mesh_->NURBSext;
+         this->NURBSext = mesh->NURBSext;
         own_ext = 0;
      }
      else
      {
-         NURBSext = NURBSext_;
+         this->NURBSext = NURBSext;
         own_ext = 1;
      }
      UpdateNURBS();
@@ -2173,7 +2333,7 @@ void FiniteElementSpace::Constructor(Mesh *mesh_, NURBSExtension *NURBSext_,
   }
   else
   {
-      NURBSext = NULL;
+      this->NURBSext = NULL;
      own_ext = 0;
      Construct();
   }
@@ -3521,7 +3681,7 @@ FiniteElementCollection *FiniteElementSpace::Load(Mesh *m, std::istream &input)
   input >> ord;

   NURBSFECollection *nurbs_fec = dynamic_cast<NURBSFECollection*>(r_fec);
-   NURBSExtension *nurbs_ext = NULL;
+   NURBSExtension *NURBSext = NULL;
   if (fes_format == 90) // original format, v0.9
   {
      if (nurbs_fec)
@@ -3531,7 +3691,7 @@ FiniteElementCollection *FiniteElementSpace::Load(Mesh *m, std::istream &input)
         if (order != m->NURBSext->GetOrder() &&
             order != NURBSFECollection::VariableOrder)
         {
-            nurbs_ext = new NURBSExtension(m->NURBSext, order);
+            NURBSext = new NURBSExtension(m->NURBSext, order);
         }
      }
   }
@@ -3548,18 +3708,18 @@ FiniteElementCollection *FiniteElementSpace::Load(Mesh *m, std::istream &input)
            MFEM_VERIFY(nurbs_fec,
                        buff << ": NURBS FE collection is required!");
            MFEM_VERIFY(m->NURBSext, buff << ": NURBS mesh is required!");
-            MFEM_VERIFY(!nurbs_ext, buff << ": order redefinition!");
+            MFEM_VERIFY(!NURBSext, buff << ": order redefinition!");
            if (buff == "NURBS_order")
            {
               int order;
               input >> order;
-               nurbs_ext = new NURBSExtension(m->NURBSext, order);
+               NURBSext = new NURBSExtension(m->NURBSext, order);
            }
            else
            {
               Array<int> orders;
               orders.Load(m->NURBSext->GetNKV(), input);
-               nurbs_ext = new NURBSExtension(m->NURBSext, orders);
+               NURBSext = new NURBSExtension(m->NURBSext, orders);
            }
         }
         else if (buff == "NURBS_periodic")
@@ -3567,13 +3727,13 @@ FiniteElementCollection *FiniteElementSpace::Load(Mesh *m, std::istream &input)
            Array<int> master, slave;
            master.Load(input);
            slave.Load(input);
-            nurbs_ext->ConnectBoundaries(master,slave);
+            NURBSext->ConnectBoundaries(master,slave);
         }
         else if (buff == "NURBS_weights")
         {
-            MFEM_VERIFY(nurbs_ext, "NURBS_weights: NURBS_orders have to be "
+            MFEM_VERIFY(NURBSext, "NURBS_weights: NURBS_orders have to be "
                        "specified before NURBS_weights!");
-            nurbs_ext->GetWeights().Load(input, nurbs_ext->GetNDof());
+            NURBSext->GetWeights().Load(input, NURBSext->GetNDof());
         }
         else if (buff == "element_orders")
         {
@@ -3592,7 +3752,7 @@ FiniteElementCollection *FiniteElementSpace::Load(Mesh *m, std::istream &input)
      }
   }

-   Constructor(m, nurbs_ext, r_fec, vdim, ord);
+   Constructor(m, NURBSext, r_fec, vdim, ord);

   return r_fec;
 }
@@ -778,6 +778,19 @@ public:
                                  Array<int> &ess_vdofs,
                                  int component = -1) const;

+   /** Mark degrees of freedom associated with the entity set with the
+       specified entity type and set index. */
+   virtual void GetEssentialVDofs(EntitySets::EntityType type, int set_index,
+                                  Array<int> &ess_vdofs,
+                                  int component = -1) const;
+
+   /** Mark degrees of freedom associated with the entity set with the
+       specified entity type and set index. */
+   virtual void GetEssentialVDofs(EntitySets::EntityType type,
+                                  const std::string & set_name,
+                                  Array<int> &ess_vdofs,
+                                  int component = -1) const;
+
   /** @brief Get a list of essential true dofs, ess_tdof_list, corresponding to the
       boundary attributes marked in the array bdr_attr_is_ess.
       For spaces with 'vdim' > 1, the 'component' parameter can be used
@@ -786,6 +799,19 @@ public:
                                     Array<int> &ess_tdof_list,
                                     int component = -1);

+   /** Get a list of essential true dofs, ess_tdof_list, corresponding to the
+       entity set specified by the given entity type and set index. */
+   virtual void GetEssentialTrueDofs(EntitySets::EntityType type, int set_index,
+                                     Array<int> &ess_tdof_list,
+                                     int component = -1);
+
+   /** Get a list of essential true dofs, ess_tdof_list, corresponding to the
+       entity set specified by the given entity type and set name. */
+   virtual void GetEssentialTrueDofs(EntitySets::EntityType type,
+                                     const std::string & set_name,
+                                     Array<int> &ess_tdof_list,
+                                     int component = -1);
+
   /** @brief Get a list of all boundary true dofs, @a boundary_dofs. For spaces
       with 'vdim' > 1, the 'component' parameter can be used to restricts the
       marked tDOFs to the specified component. Equivalent to
@@ -112,7 +112,7 @@ FmsFieldToGridFunction(FmsMesh fms_mesh, FmsField f, Mesh *mesh,
   // NOTE: transplanted from the FmsMeshToMesh function
   //       We should do this work once and save it.
   //--------------------------------------------------
-   FmsInt dim, n_elem, space_dim;
+   FmsInt dim, n_vert, n_elem, space_dim;

   // Find the first component that has coordinates - that will be the new mfem
   // mesh.
@@ -144,6 +144,7 @@ FmsFieldToGridFunction(FmsMesh fms_mesh, FmsField f, Mesh *mesh,
         n_ents[et] += num_ents;
      }
   }
+   n_vert = n_ents[FMS_VERTEX];
   //--------------------------------------------------

   // Interrogate the field.
@@ -1337,9 +1337,9 @@ RefinedGeometry * GeometryRefiner::Refine(Geometry::Type Geom,
                  // (ii,jj,kk) are coordinates in the reference tetrahedron,
                  // transform to coordinates (i,j,k) in the auxiliary
                  // tetrahedron defined by (0,0,0), (0,0,1), (1,1,1), (0,1,1)
-                  i = jj;
-                  j = jj+kk;
-                  k = ii+jj+kk;
+                  int i = jj;
+                  int j = jj+kk;
+                  int k = ii+jj+kk;
                  l = i + (j + k * (n+1)) * (n+1);
                  // map from linear Cartesian hex index in the auxiliary tet
                  // to lexicographic in the reference tet
@@ -323,11 +323,10 @@ int GridFunction::VectorDim() const
   const FiniteElement *fe;
   if (!fes->GetNE())
   {
-      const FiniteElementCollection *fe_coll = fes->FEColl();
+      const FiniteElementCollection *fec = fes->FEColl();
      static const Geometry::Type geoms[3] =
      { Geometry::SEGMENT, Geometry::TRIANGLE, Geometry::TETRAHEDRON };
-      fe = fe_coll->
-           FiniteElementForGeometry(geoms[fes->GetMesh()->Dimension()-1]);
+      fe = fec->FiniteElementForGeometry(geoms[fes->GetMesh()->Dimension()-1]);
   }
   else
   {
@@ -606,11 +605,11 @@ const
      ET->SetIntPoint(&ip);
      FElem->CalcPhysHessian(*ET, DofHes);

-      for (int j = 0; j < size; j++)
+      for (int i = 0; i < size; i++)
      {
         for (int d = 0; d < dof; d++)
         {
-            hess(k,j) += DofHes(d,j) * loc_data[d];
+            hess(k,i) += DofHes(d,i) * loc_data[d];
         }
      }
   }
@@ -3949,129 +3948,6 @@ std::ostream &operator<<(std::ostream &out, const QuadratureFunction &qf)
   return out;
 }

-void QuadratureFunction::SaveVTU(std::ostream &out, VTKFormat format,
-                                 int compression_level) const
-{
-   out << R"(<VTKFile type="UnstructuredGrid" version="0.1")";
-   if (compression_level != 0)
-   {
-      out << R"( compressor="vtkZLibDataCompressor")";
-   }
-   out << " byte_order=\"" << VTKByteOrder() << "\">\n";
-   out << "<UnstructuredGrid>\n";
-
-   const char *fmt_str = (format == VTKFormat::ASCII) ? "ascii" : "binary";
-   const char *type_str = (format != VTKFormat::BINARY32) ? "Float64" : "Float32";
-   std::vector<char> buf;
-
-   int np = qspace->GetSize();
-   int ne = qspace->GetNE();
-   int sdim = qspace->GetMesh()->SpaceDimension();
-
-   // For quadrature functions, each point is a vertex cell, so number of cells
-   // is equal to number of points
-   out << "<Piece NumberOfPoints=\"" << np
-       << "\" NumberOfCells=\"" << np << "\">\n";
-
-   // print out the points
-   out << "<Points>\n";
-   out << "<DataArray type=\"" << type_str
-       << "\" NumberOfComponents=\"3\" format=\"" << fmt_str << "\">\n";
-
-   Vector pt(sdim);
-   for (int i = 0; i < ne; i++)
-   {
-      ElementTransformation &T = *qspace->GetMesh()->GetElementTransformation(i);
-      const IntegrationRule &ir = GetElementIntRule(i);
-      for (int j = 0; j < ir.Size(); j++)
-      {
-         T.Transform(ir[j], pt);
-         WriteBinaryOrASCII(out, buf, pt[0], " ", format);
-         if (sdim > 1) { WriteBinaryOrASCII(out, buf, pt[1], " ", format); }
-         else { WriteBinaryOrASCII(out, buf, 0.0, " ", format); }
-         if (sdim > 2) { WriteBinaryOrASCII(out, buf, pt[2], "", format); }
-         else { WriteBinaryOrASCII(out, buf, 0.0, "", format); }
-         if (format == VTKFormat::ASCII) { out << '\n'; }
-      }
-   }
-   if (format != VTKFormat::ASCII)
-   {
-      WriteBase64WithSizeAndClear(out, buf, compression_level);
-   }
-   out << "</DataArray>\n";
-   out << "</Points>\n";
-
-   // Write cells (each cell is just a vertex)
-   out << "<Cells>\n";
-   // Connectivity
-   out << R"(<DataArray type="Int32" Name="connectivity" format=")"
-       << fmt_str << "\">\n";
-
-   for (int i=0; i<np; ++i) { WriteBinaryOrASCII(out, buf, i, "\n", format); }
-   if (format != VTKFormat::ASCII)
-   {
-      WriteBase64WithSizeAndClear(out, buf, compression_level);
-   }
-   out << "</DataArray>\n";
-   // Offsets
-   out << R"(<DataArray type="Int32" Name="offsets" format=")"
-       << fmt_str << "\">\n";
-   for (int i=0; i<np; ++i) { WriteBinaryOrASCII(out, buf, i, "\n", format); }
-   if (format != VTKFormat::ASCII)
-   {
-      WriteBase64WithSizeAndClear(out, buf, compression_level);
-   }
-   out << "</DataArray>\n";
-   // Types
-   out << R"(<DataArray type="UInt8" Name="types" format=")"
-       << fmt_str << "\">\n";
-   for (int i = 0; i < np; i++)
-   {
-      uint8_t vtk_cell_type = VTKGeometry::POINT;
-      WriteBinaryOrASCII(out, buf, vtk_cell_type, "\n", format);
-   }
-   if (format != VTKFormat::ASCII)
-   {
-      WriteBase64WithSizeAndClear(out, buf, compression_level);
-   }
-   out << "</DataArray>\n";
-   out << "</Cells>\n";
-
-   out << "<PointData>\n";
-   out << "<DataArray type=\"" << type_str << "\" Name=\"u\" format=\""
-       << fmt_str << "\" NumberOfComponents=\"" << vdim << "\">\n";
-   for (int i = 0; i < ne; i++)
-   {
-      DenseMatrix vals;
-      GetElementValues(i, vals);
-      for (int j = 0; j < vals.Size(); ++j)
-      {
-         for (int vd = 0; vd < vdim; ++vd)
-         {
-            WriteBinaryOrASCII(out, buf, vals(vd, j), " ", format);
-         }
-         if (format == VTKFormat::ASCII) { out << '\n'; }
-      }
-   }
-   if (format != VTKFormat::ASCII)
-   {
-      WriteBase64WithSizeAndClear(out, buf, compression_level);
-   }
-   out << "</DataArray>\n";
-   out << "</PointData>\n";
-
-   out << "</Piece>\n";
-   out << "</UnstructuredGrid>\n";
-   out << "</VTKFile>" << std::endl;
-}
-
-void QuadratureFunction::SaveVTU(const std::string &filename, VTKFormat format,
-                                 int compression_level) const
-{
-   std::ofstream f(filename + ".vtu");
-   SaveVTU(f, format, compression_level);
-}
-

 double ZZErrorEstimator(BilinearFormIntegrator &blfi,
                        GridFunction &u,
@@ -902,22 +902,6 @@ public:

   /// Write the QuadratureFunction to the stream @a out.
   void Save(std::ostream &out) const;
-
-   /// @brief Write the QuadratureFunction to @a out in VTU (ParaView) format.
-   ///
-   /// The data will be uncompressed if @a compression_level is zero, or if the
-   /// format is VTKFormat::ASCII. Otherwise, zlib compression will be used for
-   /// binary data.
-   void SaveVTU(std::ostream &out, VTKFormat format=VTKFormat::ASCII,
-                int compression_level=0) const;
-
-   /// @brief Save the QuadratureFunction to a VTU (ParaView) file.
-   ///
-   /// The extension ".vtu" will be appended to @a filename.
-   /// @sa SaveVTU(std::ostream &out, VTKFormat format=VTKFormat::ASCII,
-   ///             int compression_level=0)
-   void SaveVTU(const std::string &filename, VTKFormat format=VTKFormat::ASCII,
-                int compression_level=0) const;
 };

 /// Overload operator<< for std::ostream and QuadratureFunction.
@@ -78,8 +78,6 @@ void FindPointsGSLIB::Setup(Mesh &m, const double bb_t, const double newt_tol,
   MFEM_VERIFY(m.GetNodes() != NULL, "Mesh nodes are required.");
   MFEM_VERIFY(m.GetNumGeometries(m.Dimension()) == 1,
               "Mixed meshes are not currently supported in FindPointsGSLIB.");
-   MFEM_VERIFY(!(m.GetNodes()->FESpace()->IsVariableOrder()),
-               "Variable order mesh is not currently supported.");

   // call FreeData if FindPointsGSLIB::Setup has been called already
   if (setupflag) { FreeData(); }
@@ -592,8 +590,7 @@ void FindPointsGSLIB::Interpolate(const GridFunction &field_in,
   const L2_FECollection *fec_l2 = dynamic_cast<const L2_FECollection *>(fec_in);

   if (fec_h1 && gf_order == mesh_order &&
-       fec_h1->GetBasisType() == BasisType::GaussLobatto &&
-       !field_in.FESpace()->IsVariableOrder())
+       fec_h1->GetBasisType() == BasisType::GaussLobatto)
   {
      InterpolateH1(field_in, field_out);
      return;
@@ -860,8 +857,6 @@ void OversetFindPointsGSLIB::Setup(Mesh &m, const int meshid,
   MFEM_VERIFY(m.GetNodes() != NULL, "Mesh nodes are required.");
   MFEM_VERIFY(m.GetNumGeometries(m.Dimension()) == 1,
               "Mixed meshes are not currently supported in FindPointsGSLIB.");
-   MFEM_VERIFY(!(m.GetNodes()->FESpace()->IsVariableOrder()),
-               "Variable order mesh is not currently supported.");

   // FreeData if OversetFindPointsGSLIB::Setup has been called already
   if (setupflag) { FreeData(); }
@@ -776,7 +776,7 @@ void Hybridization::MultAfInv(const Vector &b, const Vector &lambda, Vector &bf,
         if (vdof_marker[vdof]) { el_vals(j) = 0.0; }
         else { vdof_marker[vdof] = true; }
      }
-      bf_i.MakeRef(bf, hat_offsets[i], vdofs.Size());
+      bf_i.SetDataAndSize(&bf[hat_offsets[i]], vdofs.Size());
      if (mode == 1)
      {
         el_vals -= bf_i;
@@ -821,15 +821,7 @@ void Hybridization::ReduceRHS(const Vector &b, Vector &b_r) const
   else
   {
      Vector bl(pC ? pC->Height() : Ct->Width());
-      if (pC)
-      {
-         pC->Mult(bf, bl);
-      }
-      else
-      {
-         Ct->EnsureMultTranspose();
-         Ct->MultTranspose(bf, bl);
-      }
+      pC ? pC->Mult(bf, bl) : Ct->MultTranspose(bf, bl);
      b_r.SetSize(pH.Ptr()->Height());
      (P_pc ? P_pc : c_pfes->GetProlongationMatrix())->MultTranspose(bl, b_r);
   }
@@ -106,6 +106,8 @@ void LinearForm::Assemble()
   DofTransformation *doftrans;
   Vector elemvect;

+   int i;
+
   Vector::operator=(0.0);

   // The above operation is executed on device because of UseDevice().
@@ -125,7 +127,7 @@ void LinearForm::Assemble()
         }
      }

-      for (int i = 0; i < fes -> GetNE(); i++)
+      for (i = 0; i < fes -> GetNE(); i++)
      {
         int elem_attr = fes->GetMesh()->GetAttribute(i);
         for (int k = 0; k < domain_integs.Size(); k++)
@@ -173,7 +175,7 @@ void LinearForm::Assemble()
         }
      }

-      for (int i = 0; i < fes -> GetNBE(); i++)
+      for (i = 0; i < fes -> GetNBE(); i++)
      {
         const int bdr_attr = mesh->GetBdrAttribute(i);
         if (bdr_attr_marker[bdr_attr-1] == 0) { continue; }
@@ -221,7 +223,7 @@ void LinearForm::Assemble()
         }
      }

-      for (int i = 0; i < mesh->GetNBE(); i++)
+      for (i = 0; i < mesh->GetNBE(); i++)
      {
         const int bdr_attr = mesh->GetBdrAttribute(i);
         if (bdr_attr_marker[bdr_attr-1] == 0) { continue; }
@@ -251,7 +253,7 @@ void LinearForm::Assemble()

      for (int k = 0; k < interior_face_integs.Size(); k++)
      {
-         for (int i = 0; i < mesh->GetNumFaces(); i++)
+         for (i = 0; i < mesh->GetNumFaces(); i++)
         {
            FaceElementTransformations *tr = NULL;
            tr = mesh->GetInteriorFaceTransformations (i);
@@ -67,11 +67,11 @@ void LORBase::ResetIntegrationRules(GetIntegratorsFn get_integrators)

 LORBase::FESpaceType LORBase::GetFESpaceType() const
 {
-   const FiniteElementCollection *fec_ho = fes_ho.FEColl();
-   if (dynamic_cast<const H1_FECollection*>(fec_ho)) { return H1; }
-   else if (dynamic_cast<const ND_FECollection*>(fec_ho)) { return ND; }
-   else if (dynamic_cast<const RT_FECollection*>(fec_ho)) { return RT; }
-   else if (dynamic_cast<const L2_FECollection*>(fec_ho)) { return L2; }
+   const FiniteElementCollection *fec = fes_ho.FEColl();
+   if (dynamic_cast<const H1_FECollection*>(fec)) { return H1; }
+   else if (dynamic_cast<const ND_FECollection*>(fec)) { return ND; }
+   else if (dynamic_cast<const RT_FECollection*>(fec)) { return RT; }
+   else if (dynamic_cast<const L2_FECollection*>(fec)) { return L2; }
   else { MFEM_ABORT("Bad LOR space type."); }
   return INVALID;
 }
@@ -87,9 +87,9 @@ void LORBase::ConstructLocalDofPermutation(Array<int> &perm_) const
   FESpaceType type = GetFESpaceType();
   MFEM_VERIFY(type != H1 && type != L2, "");

-   auto get_dof_map = [](FiniteElementSpace &fes_, int i)
+   auto get_dof_map = [](FiniteElementSpace &fes, int i)
   {
-      const FiniteElement *fe = fes_.GetFE(i);
+      const FiniteElement *fe = fes.GetFE(i);
      auto tfe = dynamic_cast<const TensorBasisElement*>(fe);
      MFEM_ASSERT(tfe != NULL, "");
      return tfe->GetDofMap();
@@ -151,8 +151,8 @@ public:
   void SetEssentialVDofs(const Array<int> &ess_vdofs_list);

   /// Specify essential boundary conditions.
-   void SetEssentialTrueDofs(const Array<int> &ess_tdof_list_)
-   { ess_tdof_list_.Copy(this->ess_tdof_list); }
+   void SetEssentialTrueDofs(const Array<int> &ess_tdof_list)
+   { ess_tdof_list.Copy(this->ess_tdof_list); }

   /// Return a (read-only) list of all essential true dofs.
   const Array<int> &GetEssentialTrueDofs() const { return ess_tdof_list; }
@@ -746,7 +746,7 @@ void VectorConvectionNLFIntegrator::AssembleElementVector(
   Vector &elvect)
 {
   const int nd = el.GetDof();
-   dim = el.GetDim();
+   const int dim = el.GetDim();

   shape.SetSize(nd);
   dshape.SetSize(nd, dim);
@@ -783,7 +783,7 @@ void VectorConvectionNLFIntegrator::AssembleElementGrad(
   DenseMatrix &elmat)
 {
   const int nd = el.GetDof();
-   dim = el.GetDim();
+   const int dim = el.GetDim();

   shape.SetSize(nd);
   dshape.SetSize(nd, dim);
@@ -826,9 +826,9 @@ void VectorConvectionNLFIntegrator::AssembleElementGrad(
      dshape.Mult(vec2, vec3);
      MultVWt(shape, vec3, elmat_comp);

-      for (int ii = 0; ii < dim; ii++)
+      for (int i = 0; i < dim; i++)
      {
-         elmat.AddMatrix(elmat_comp, ii * nd, ii * nd);
+         elmat.AddMatrix(elmat_comp, i * nd, i * nd);
      }

      MultVVt(shape, elmat_comp);
@@ -837,11 +837,11 @@ void VectorConvectionNLFIntegrator::AssembleElementGrad(
      {
         w *= Q->Eval(trans, ip);
      }
-      for (int ii = 0; ii < dim; ii++)
+      for (int i = 0; i < dim; i++)
      {
-         for (int jj = 0; jj < dim; jj++)
+         for (int j = 0; j < dim; j++)
         {
-            elmat.AddMatrix(w * gradEF(ii, jj), elmat_comp, ii * nd, jj * nd);
+            elmat.AddMatrix(w * gradEF(i, j), elmat_comp, i * nd, j * nd);
         }
      }
   }
@@ -889,9 +889,9 @@ void ConvectiveVectorConvectionNLFIntegrator::AssembleElementGrad(
      dshape.Mult(vec2, vec3); // (u^n \cdot grad u^{n+1})
      MultVWt(shape, vec3, elmat_comp); // (u^n \cdot grad u^{n+1},v)

-      for (int ii = 0; ii < dim; ii++)
+      for (int i = 0; i < dim; i++)
      {
-         elmat.AddMatrix(elmat_comp, ii * nd, ii * nd);
+         elmat.AddMatrix(elmat_comp, i * nd, i * nd);
      }
   }
 }
@@ -944,10 +944,10 @@ void SkewSymmetricVectorConvectionNLFIntegrator::AssembleElementGrad(
      MultVWt(shape, vec3, elmat_comp); // (u^n \cdot grad u^{n+1},v)
      elmat_comp_T.Transpose(elmat_comp);

-      for (int ii = 0; ii < dim; ii++)
+      for (int i = 0; i < dim; i++)
      {
-         elmat.AddMatrix(.5, elmat_comp, ii * nd, ii * nd);
-         elmat.AddMatrix(-.5, elmat_comp_T, ii * nd, ii * nd);
+         elmat.AddMatrix(.5, elmat_comp, i * nd, i * nd);
+         elmat.AddMatrix(-.5, elmat_comp_T, i * nd, i * nd);
      }
   }
 }
@@ -317,26 +317,26 @@ ParallelEliminateEssentialBC(const Array<int> &bdr_attr_is_ess,
 void ParBilinearForm::TrueAddMult(const Vector &x, Vector &y, const double a)
 const
 {
-   if (Xaux.ParFESpace() != pfes)
+   if (X.ParFESpace() != pfes)
   {
-      Xaux.SetSpace(pfes);
-      Yaux.SetSpace(pfes);
+      X.SetSpace(pfes);
+      Y.SetSpace(pfes);
      Ytmp.SetSize(pfes->GetTrueVSize());
   }

-   Xaux.Distribute(&x);
+   X.Distribute(&x);
   if (ext)
   {
-      ext->Mult(Xaux, Yaux);
+      ext->Mult(X, Y);
   }
   else
   {
      MFEM_VERIFY(interior_face_integs.Size() == 0,
                  "the case of interior face integrators is not"
                  " implemented");
-      mat->Mult(Xaux, Yaux);
+      mat->Mult(X, Y);
   }
-   pfes->GetProlongationMatrix()->MultTranspose(Yaux, Ytmp);
+   pfes->GetProlongationMatrix()->MultTranspose(Y, Ytmp);
   y.Add(a,Ytmp);
 }

@@ -373,7 +373,6 @@ void ParBilinearForm::FormLinearSystem(
      P.MultTranspose(b, true_B);
      R.Mult(x, true_X);
      p_mat.EliminateBC(p_mat_e, ess_tdof_list, true_X, true_B);
-      R.EnsureMultTranspose();
      R.MultTranspose(true_B, b);
      hybridization->ReduceRHS(true_B, B);
      X.SetSize(B.Size());
@@ -538,15 +537,15 @@ void ParMixedBilinearForm::ParallelAssemble(OperatorHandle &A)
 void ParMixedBilinearForm::TrueAddMult(const Vector &x, Vector &y,
                                       const double a) const
 {
-   if (Xaux.ParFESpace() != trial_pfes)
+   if (X.ParFESpace() != trial_pfes)
   {
-      Xaux.SetSpace(trial_pfes);
-      Yaux.SetSpace(test_pfes);
+      X.SetSpace(trial_pfes);
+      Y.SetSpace(test_pfes);
   }

-   Xaux.Distribute(&x);
-   mat->Mult(Xaux, Yaux);
-   test_pfes->Dof_TrueDof_Matrix()->MultTranspose(a, Yaux, 1.0, y);
+   X.Distribute(&x);
+   mat->Mult(X, Y);
+   test_pfes->Dof_TrueDof_Matrix()->MultTranspose(a, Y, 1.0, y);
 }

 void ParMixedBilinearForm::FormRectangularSystemMatrix(
@@ -32,7 +32,7 @@ protected:
   ParFiniteElementSpace *pfes; ///< Points to the same object as #fes

   /// Auxiliary objects used in TrueAddMult().
-   mutable ParGridFunction Xaux, Yaux;
+   mutable ParGridFunction X, Y;
   mutable Vector Ytmp;

   OperatorHandle p_mat, p_mat_e;
@@ -211,7 +211,7 @@ protected:
   /// Points to the same object as #test_fes
   ParFiniteElementSpace *test_pfes;
   /// Auxiliary objects used in TrueAddMult().
-   mutable ParGridFunction Xaux, Yaux;
+   mutable ParGridFunction X, Y;

   /// Matrix and eliminated matrix
   OperatorHandle p_mat, p_mat_e;
@@ -23,6 +23,8 @@
 #include <limits>
 #include <list>

+using namespace std;
+
 namespace mfem
 {

@@ -232,7 +234,7 @@ void ParFiniteElementSpace::PrintPartitionStats()
 }

 void ParFiniteElementSpace::GetGroupComm(
-   GroupCommunicator &gc, int ldof_type, Array<int> *g_ldof_sign)
+   GroupCommunicator &gc, int ldof_type, Array<int> *ldof_sign)
 {
   int gr;
   int ng = pmesh->GetNGroups();
@@ -257,10 +259,10 @@ void ParFiniteElementSpace::GetGroupComm(
      }
   }

-   if (g_ldof_sign)
+   if (ldof_sign)
   {
-      g_ldof_sign->SetSize(GetNDofs());
-      *g_ldof_sign = 1;
+      ldof_sign->SetSize(GetNDofs());
+      *ldof_sign = 1;
   }

   // count the number of ldofs in all groups (excluding the local group 0)
@@ -333,9 +335,9 @@ void ParFiniteElementSpace::GetGroupComm(
               if (ind[l] < 0)
               {
                  dofs[l] = m + (-1-ind[l]);
-                  if (g_ldof_sign)
+                  if (ldof_sign)
                  {
-                     (*g_ldof_sign)[dofs[l]] = -1;
+                     (*ldof_sign)[dofs[l]] = -1;
                  }
               }
               else
@@ -371,9 +373,9 @@ void ParFiniteElementSpace::GetGroupComm(
               if (ind[l] < 0)
               {
                  dofs[l] = m + (-1-ind[l]);
-                  if (g_ldof_sign)
+                  if (ldof_sign)
                  {
-                     (*g_ldof_sign)[dofs[l]] = -1;
+                     (*ldof_sign)[dofs[l]] = -1;
                  }
               }
               else
@@ -409,9 +411,9 @@ void ParFiniteElementSpace::GetGroupComm(
               if (ind[l] < 0)
               {
                  dofs[l] = m + (-1-ind[l]);
-                  if (g_ldof_sign)
+                  if (ldof_sign)
                  {
-                     (*g_ldof_sign)[dofs[l]] = -1;
+                     (*ldof_sign)[dofs[l]] = -1;
                  }
               }
               else
@@ -556,25 +558,11 @@ const FaceRestriction *ParFiniteElementSpace::GetFaceRestriction(
      FaceRestriction *res;
      if (is_dg_space)
      {
-         if (Conforming())
-         {
-            res = new ParL2FaceRestriction(*this, e_ordering, type, m);
-         }
-         else
-         {
-            res = new ParNCL2FaceRestriction(*this, e_ordering, type, m);
-         }
+         res = new ParL2FaceRestriction(*this, e_ordering, type, m);
      }
      else
      {
-         if (Conforming())
-         {
-            res = new H1FaceRestriction(*this, e_ordering, type);
-         }
-         else
-         {
-            res = new ParNCH1FaceRestriction(*this, e_ordering, type);
-         }
+         res = new H1FaceRestriction(*this, e_ordering, type);
      }
      L2F[key] = res;
      return res;
@@ -764,10 +752,10 @@ void ParFiniteElementSpace::Build_Dof_TrueDof_Matrix() const // matrix P
      diag_counter = offd_counter = 0;
      for (int i = 0; i < ldof; i++)
      {
-         int ltdof_i = GetLocalTDofNumber(i);
-         if (ltdof_i >= 0)
+         int ltdof = GetLocalTDofNumber(i);
+         if (ltdof >= 0)
         {
-            j_diag[diag_counter++] = ltdof_i;
+            j_diag[diag_counter++] = ltdof;
         }
         else
         {
@@ -884,10 +872,10 @@ void ParFiniteElementSpace::Build_Dof_TrueDof_Matrix() const // matrix P
      int offd_col_counter = 0;
      for (int i = 0; i < ldof; i++)
      {
-         int ltdofi = GetLocalTDofNumber(i);
-         if (ltdofi >= 0)
+         int ltdof = GetLocalTDofNumber(i);
+         if (ltdof >= 0)
         {
-            j_diag[diag_counter]   = ltdofi;
+            j_diag[diag_counter]   = ltdof;
            d_diag[diag_counter++] = 1.0;
         }
         else
@@ -1024,9 +1012,36 @@ void ParFiniteElementSpace::GetEssentialVDofs(const Array<int> &bdr_attr_is_ess,
 {
   FiniteElementSpace::GetEssentialVDofs(bdr_attr_is_ess, ess_dofs, component);

-   // Make sure that processors without boundary elements mark
-   // their boundary dofs (if they have any).
-   Synchronize(ess_dofs);
+   if (Conforming())
+   {
+      // Make sure that processors without boundary elements mark
+      // their boundary dofs (if they have any).
+      Synchronize(ess_dofs);
+   }
+}
+
+void ParFiniteElementSpace::GetEssentialVDofs(EntitySets::EntityType type,
+                                              int set_index,
+                                              Array<int> &ess_dofs,
+                                              int component) const
+{
+   FiniteElementSpace::GetEssentialVDofs(type, set_index, ess_dofs, component);
+
+   if (Conforming())
+   {
+      // Make sure that processors without boundary elements mark
+      // their boundary dofs (if they have any).
+      Synchronize(ess_dofs);
+   }
+}
+
+void ParFiniteElementSpace::GetEssentialVDofs(EntitySets::EntityType type,
+                                              const string & set_name,
+                                              Array<int> &ess_vdofs,
+                                              int component) const
+{
+   GetEssentialVDofs(type, pmesh->ent_sets->GetSetIndex(type, set_name),
+                     ess_vdofs, component);
 }

 void ParFiniteElementSpace::GetEssentialTrueDofs(const Array<int>
@@ -1052,13 +1067,33 @@ void ParFiniteElementSpace::GetEssentialTrueDofs(const Array<int>
   {
      if (bool(ted[i]) != bool(true_ess_dofs2[i])) { counter++; }
   }
-   MFEM_VERIFY(counter == 0, "internal MFEM error: counter = " << counter
-               << ", rank = " << MyRank);
+   MFEM_VERIFY(counter == 0, "internal MFEM error: counter = " << counter);
 #endif

   MarkerToList(true_ess_dofs, ess_tdof_list);
 }

+void ParFiniteElementSpace::GetEssentialTrueDofs(EntitySets::EntityType type,
+                                                 int set_index,
+                                                 Array<int> &ess_tdof_list,
+                                                 int component)
+{
+   Array<int> ess_dofs, true_ess_dofs;
+
+   GetEssentialVDofs(type, set_index, ess_dofs, component);
+   GetRestrictionMatrix()->BooleanMult(ess_dofs, true_ess_dofs);
+   MarkerToList(true_ess_dofs, ess_tdof_list);
+}
+
+void ParFiniteElementSpace::GetEssentialTrueDofs(EntitySets::EntityType type,
+                                                 const string & set_name,
+                                                 Array<int> &ess_tdof_list,
+                                                 int component)
+{
+   GetEssentialTrueDofs(type, pmesh->ent_sets->GetSetIndex(type, set_name),
+                        ess_tdof_list, component);
+}
+
 int ParFiniteElementSpace::GetLocalTDofNumber(int ldof) const
 {
   if (Nonconforming())
@@ -1341,12 +1376,12 @@ void ParFiniteElementSpace::ExchangeFaceNbrData()
   for (int fn = 0, j = 0; fn < num_face_nbrs; fn++)
   {
      int  num_ldofs = send_face_nbr_ldof.RowSize(fn);
-      int *ldofs_fn  = send_face_nbr_ldof.GetRow(fn);
+      int *ldofs     = send_face_nbr_ldof.GetRow(fn);
      int  j_end     = send_I[send_el_off[fn+1]];

      for (int i = 0; i < num_ldofs; i++)
      {
-         int ldof = (ldofs_fn[i] >= 0 ? ldofs_fn[i] : -1-ldofs_fn[i]);
+         int ldof = (ldofs[i] >= 0 ? ldofs[i] : -1-ldofs[i]);
         ldof_marker[ldof] = i;
      }

@@ -2037,7 +2072,7 @@ public:
   const RowInfo::List& GetRows() const { return rows; }

   void SetNCMesh(ParNCMesh* pnc) { pncmesh = pnc; }
-   void SetFEC(const FiniteElementCollection* fec_) { this->fec = fec_; }
+   void SetFEC(const FiniteElementCollection* fec) { this->fec = fec; }

   typedef std::map<int, NeighborRowMessage> Map;

@@ -2300,7 +2335,7 @@ void ParFiniteElementSpace
 #endif

 int ParFiniteElementSpace
-::BuildParallelConformingInterpolation(HypreParMatrix **P_, SparseMatrix **R_,
+::BuildParallelConformingInterpolation(HypreParMatrix **P, SparseMatrix **R,
                                       Array<HYPRE_BigInt> &dof_offs,
                                       Array<HYPRE_BigInt> &tdof_offs,
                                       Array<int> *dof_tdof,
@@ -2457,10 +2492,10 @@ int ParFiniteElementSpace
   HYPRE_BigInt my_tdof_offset =
      tdof_offs[HYPRE_AssumedPartitionCheck() ? 0 : MyRank];

-   if (R_)
+   if (R)
   {
      // initialize the restriction matrix (also parallel but block-diagonal)
-      *R_ = new SparseMatrix(num_true_dofs*vdim, ndofs*vdim);
+      *R = new SparseMatrix(num_true_dofs*vdim, ndofs*vdim);
   }
   if (dof_tdof)
   {
@@ -2498,7 +2533,7 @@ int ParFiniteElementSpace
            int vdof = dof*vdim_factor + vd*dof_stride;
            int vtdof = tdof*vdim_factor + vd*tdof_stride;

-            if (R_) { (*R_)->Add(vtdof, vdof, 1.0); }
+            if (R) { (*R)->Add(vtdof, vdof, 1.0); }
            if (dof_tdof) { (*dof_tdof)[vdof] = vtdof; }
         }
         tdof++;
@@ -2511,7 +2546,7 @@ int ParFiniteElementSpace
   n_msgs_sent += send_msg.back().size();
 #endif

-   if (R_) { (*R_)->Finalize(); }
+   if (R) { (*R)->Finalize(); }

   // *** STEP 4: main loop ***

@@ -2620,10 +2655,10 @@ int ParFiniteElementSpace
 #endif
   }

-   if (P_)
+   if (P)
   {
-      *P_ = MakeVDimHypreMatrix(pmatrix, ndofs, num_true_dofs,
-                                dof_offs, tdof_offs);
+      *P = MakeVDimHypreMatrix(pmatrix, ndofs, num_true_dofs,
+                               dof_offs, tdof_offs);
   }

   // clean up possible remaining messages in the queue to avoid receiving
@@ -2812,13 +2847,13 @@ ParFiniteElementSpace::RebalanceMatrix(int old_ndofs,
                             ? old_dof_offsets[0] : old_dof_offsets[MyRank];

   // send old DOFs of elements we used to own
-   ParNCMesh* old_pncmesh = pmesh->pncmesh;
-   old_pncmesh->SendRebalanceDofs(old_ndofs, *old_elem_dof, old_offset, this);
+   ParNCMesh* pncmesh = pmesh->pncmesh;
+   pncmesh->SendRebalanceDofs(old_ndofs, *old_elem_dof, old_offset, this);

   Array<int> dofs;
   int vsize = GetVSize();

-   const Array<int> &old_index = old_pncmesh->GetRebalanceOldIndex();
+   const Array<int> &old_index = pncmesh->GetRebalanceOldIndex();
   MFEM_VERIFY(old_index.Size() == pmesh->GetNE(),
               "Mesh::Rebalance was not called before "
               "ParFiniteElementSpace::RebalanceMatrix");
@@ -2852,7 +2887,7 @@ ParFiniteElementSpace::RebalanceMatrix(int old_ndofs,
   // receive old DOFs for elements we obtained from others in Rebalance
   Array<int> new_elements;
   Array<long> old_remote_dofs;
-   old_pncmesh->RecvRebalanceDofs(new_elements, old_remote_dofs);
+   pncmesh->RecvRebalanceDofs(new_elements, old_remote_dofs);

   // create the offdiagonal part of the matrix
   HYPRE_BigInt* i_offd = make_i_array<HYPRE_BigInt>(vsize);
@@ -2950,7 +2985,7 @@ ParFiniteElementSpace::ParallelDerefinementMatrix(int old_ndofs,
   Array<int> dofs, old_dofs, old_vdofs;
   Vector row;

-   ParNCMesh* old_pncmesh = pmesh->pncmesh;
+   ParNCMesh* pncmesh = pmesh->pncmesh;

   int ldof[Geometry::NumGeom];
   for (int i = 0; i < Geometry::NumGeom; i++)
@@ -2963,9 +2998,8 @@ ParFiniteElementSpace::ParallelDerefinementMatrix(int old_ndofs,
      ldof[geom] = fec->FiniteElementForGeometry(geom)->GetDof();
   }

-   const CoarseFineTransformations &dtrans =
-      old_pncmesh->GetDerefinementTransforms();
-   const Array<int> &old_ranks = old_pncmesh->GetDerefineOldRanks();
+   const CoarseFineTransformations &dtrans = pncmesh->GetDerefinementTransforms();
+   const Array<int> &old_ranks = pncmesh->GetDerefineOldRanks();

   std::map<int, DerefDofMessage> messages;

@@ -2980,7 +3014,7 @@ ParFiniteElementSpace::ParallelDerefinementMatrix(int old_ndofs,

      int fine_rank = old_ranks[k];
      int coarse_rank = (emb.parent < 0) ? (-1 - emb.parent)
-                        : old_pncmesh->ElementRank(emb.parent);
+                        : pncmesh->ElementRank(emb.parent);

      if (coarse_rank != MyRank && fine_rank == MyRank)
      {
@@ -3030,7 +3064,7 @@ ParFiniteElementSpace::ParallelDerefinementMatrix(int old_ndofs,
      const Embedding &emb = dtrans.embeddings[k];
      if (emb.parent < 0) { continue; }

-      int coarse_rank = old_pncmesh->ElementRank(emb.parent);
+      int coarse_rank = pncmesh->ElementRank(emb.parent);
      int fine_rank = old_ranks[k];

      if (coarse_rank == MyRank && fine_rank == MyRank)
@@ -3080,7 +3114,7 @@ ParFiniteElementSpace::ParallelDerefinementMatrix(int old_ndofs,
      const Embedding &emb = dtrans.embeddings[k];
      if (emb.parent < 0) { continue; }

-      int coarse_rank = old_pncmesh->ElementRank(emb.parent);
+      int coarse_rank = pncmesh->ElementRank(emb.parent);
      int fine_rank = old_ranks[k];

      if (coarse_rank == MyRank && fine_rank != MyRank)
@@ -3161,14 +3195,14 @@ ParFiniteElementSpace::ParallelDerefinementMatrix(int old_ndofs,
      offd->SortColumnIndices();
   }

-   HypreParMatrix* new_R;
-   new_R = new HypreParMatrix(MyComm, dof_offsets[nrk], old_dof_offsets[nrk],
-                              dof_offsets, old_dof_offsets, diag, offd, cmap,
-                              true);
+   HypreParMatrix* R;
+   R = new HypreParMatrix(MyComm, dof_offsets[nrk], old_dof_offsets[nrk],
+                          dof_offsets, old_dof_offsets, diag, offd, cmap,
+                          true);

-   new_R->SetOwnerFlags(new_R->OwnsDiag(), new_R->OwnsOffd(), 1);
+   R->SetOwnerFlags(R->OwnsDiag(), R->OwnsOffd(), 1);

-   return new_R;
+   return R;
 }

 void ParFiniteElementSpace::Destroy()
@@ -3552,18 +3586,18 @@ DeviceConformingProlongationOperator::DeviceConformingProlongationOperator(
      shr_buf.UseDevice(true);
      shr_buf_offsets = nbr_ltdof.GetIMemory();
      {
-         Array<int> shared_ltdof(nbr_ltdof.GetJ(), nb_connections);
-         Array<int> unique_ltdof(shared_ltdof);
+         Array<int> shr_ltdof(nbr_ltdof.GetJ(), nb_connections);
+         Array<int> unique_ltdof(shr_ltdof);
         unique_ltdof.Sort();
         unique_ltdof.Unique();
         // Note: the next loop modifies the J array of nbr_ltdof
-         for (int i = 0; i < shared_ltdof.Size(); i++)
+         for (int i = 0; i < shr_ltdof.Size(); i++)
         {
-            shared_ltdof[i] = unique_ltdof.FindSorted(shared_ltdof[i]);
-            MFEM_ASSERT(shared_ltdof[i] != -1, "internal error");
+            shr_ltdof[i] = unique_ltdof.FindSorted(shr_ltdof[i]);
+            MFEM_ASSERT(shr_ltdof[i] != -1, "internal error");
         }
         Table unique_shr;
-         Transpose(shared_ltdof, unique_shr, unique_ltdof.Size());
+         Transpose(shr_ltdof, unique_shr, unique_ltdof.Size());
         unq_ltdof = Array<int>(unique_ltdof, unique_ltdof.Size());
         unq_shr_i = Array<int>(unique_shr.GetI(), unique_shr.Size()+1);
         unq_shr_j = Array<int>(unique_shr.GetJ(), unique_shr.Size_of_connections());
@@ -355,12 +355,38 @@ public:
                                  Array<int> &ess_dofs,
                                  int component = -1) const;

+   /** Mark degrees of freedom associated with the entity set with the
+       specified entity type and set index. */
+   virtual void GetEssentialVDofs(EntitySets::EntityType type, int set_index,
+                                  Array<int> &ess_vdofs,
+                                  int component = -1) const;
+
+   /** Mark degrees of freedom associated with the entity set with the
+       specified entity type and set index. */
+   virtual void GetEssentialVDofs(EntitySets::EntityType type,
+                                  const std::string & set_name,
+                                  Array<int> &ess_vdofs,
+                                  int component = -1) const;
+
   /** Get a list of essential true dofs, ess_tdof_list, corresponding to the
       boundary attributes marked in the array bdr_attr_is_ess. */
   virtual void GetEssentialTrueDofs(const Array<int> &bdr_attr_is_ess,
                                     Array<int> &ess_tdof_list,
                                     int component = -1);

+   /** Get a list of essential true dofs, ess_tdof_list, corresponding to the
+       entity set specified by the given entity type and set index. */
+   virtual void GetEssentialTrueDofs(EntitySets::EntityType type, int set_index,
+                                     Array<int> &ess_tdof_list,
+                                     int component = -1);
+
+   /** Get a list of essential true dofs, ess_tdof_list, corresponding to the
+       entity set specified by the given entity type and set name. */
+   virtual void GetEssentialTrueDofs(EntitySets::EntityType type,
+                                     const std::string & set_name,
+                                     Array<int> &ess_tdof_list,
+                                     int component = -1);
+
   /** If the given ldof is owned by the current processor, return its local
       tdof number, otherwise return -1 */
   int GetLocalTDofNumber(int ldof) const;
@@ -90,8 +90,8 @@ void ParNonlinearForm::Mult(const Vector &x, Vector &y) const

   const int N = ess_tdof_list.Size();
   const auto idx = ess_tdof_list.Read();
-   auto Y_RW = y.ReadWrite();
-   MFEM_FORALL(i, N, Y_RW[idx[i]] = 0.0; );
+   auto Y = y.ReadWrite();
+   MFEM_FORALL(i, N, Y[idx[i]] = 0.0; );
 }

 const SparseMatrix &ParNonlinearForm::GetLocalGradient(const Vector &x) const
@@ -23,348 +23,34 @@ namespace mfem

 class ParFiniteElementSpace;

-/// Operator that extracts Face degrees of freedom for NCMesh in parallel.
-/** Objects of this type are typically created and owned by
-    ParFiniteElementSpace objects, see
-    ParFiniteElementSpace::GetFaceRestriction(). */
-class ParNCH1FaceRestriction : public H1FaceRestriction
-{
-protected:
-   const FaceType type;
-   InterpolationManager interpolations;
-   mutable Vector x_interp;
-
-public:
-   /** @brief Constructs an ParNCH1FaceRestriction.
-
-       @param[in] fes      The ParFiniteElementSpace on which this operates
-       @param[in] ordering Request a specific ordering
-       @param[in] type     Request internal or boundary faces dofs */
-   ParNCH1FaceRestriction(const ParFiniteElementSpace &fes,
-                          ElementDofOrdering ordering,
-                          FaceType type);
-
-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector.
-
-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     face_dofs x vdim x nf
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
-   void Mult(const Vector &x, Vector &y) const override;
-
-   /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
-       L-Vector.
-
-       @param[in]  x The face E-Vector degrees of freedom with the given format:
-                     face_dofs x vdim x nf
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs should be ordered according to the given
-                     ElementDofOrdering.
-       @param[in,out] y The L-vector degrees of freedom. */
-   void AddMultTranspose(const Vector &x, Vector &y) const override;
-
-private:
-   /** @brief Compute the scatter indices: L-vector to E-vector, the offsets
-       for the gathering: E-vector to L-vector, and the interpolators from
-       coarse to fine face for master non-comforming faces.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeScatterIndicesAndOffsets(const ElementDofOrdering ordering,
-                                        const FaceType type);
-
-   /** @brief Compute the gather indices: E-vector to L-vector.
-
-       Note: Requires the gather offsets to be computed.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeGatherIndices(const ElementDofOrdering ordering,
-                             const FaceType type);
-};
-
 /// Operator that extracts Face degrees of freedom in parallel.
-/** Objects of this type are typically created and owned by
-    ParFiniteElementSpace objects, see
-    ParFiniteElementSpace::GetFaceRestriction(). */
-class ParL2FaceRestriction : virtual public L2FaceRestriction
+/** Objects of this type are typically created and owned by FiniteElementSpace
+    objects, see FiniteElementSpace::GetFaceRestriction(). */
+class ParL2FaceRestriction : public L2FaceRestriction
 {
-protected:
-   /** @brief Constructs an ParL2FaceRestriction.
-
-       @param[in] fes      The ParFiniteElementSpace on which this operates
-       @param[in] ordering Request a specific ordering
-       @param[in] type     Request internal or boundary faces dofs
-       @param[in] m        Request the face dofs for elem1, or both elem1 and
-                           elem2
-       @param[in] build    Request the ParL2FaceRestriction to compute the
-                           scatter/gather indices. False should only be used
-                           when inheriting from ParL2FaceRestriction. */
-   ParL2FaceRestriction(const ParFiniteElementSpace& fes,
-                        ElementDofOrdering ordering,
-                        FaceType type,
-                        L2FaceValues m,
-                        bool build);
-
 public:
-   /** @brief Constructs an ParL2FaceRestriction.
-
-       @param[in] fes      The ParFiniteElementSpace on which this operates
-       @param[in] ordering Request a specific ordering
-       @param[in] type     Request internal or boundary faces dofs
-       @param[in] m        Request the face dofs for elem1, or both elem1 and
-                           elem2 */
-   ParL2FaceRestriction(const ParFiniteElementSpace& fes,
-                        ElementDofOrdering ordering,
+   ParL2FaceRestriction(const ParFiniteElementSpace&, ElementDofOrdering,
                        FaceType type,
                        L2FaceValues m = L2FaceValues::DoubleValued);
-
-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector.
-
-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     if L2FacesValues::DoubleValued (face_dofs x vdim x 2 x nf),
-                     if L2FacesValues::SingleValued (face_dofs x vdim x nf),
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
-   void Mult(const Vector &x, Vector &y) const override;
-
+   void Mult(const Vector &x, Vector &y) const;
   /** Fill the I array of SparseMatrix corresponding to the sparsity pattern
-       given by this ParL2FaceRestriction.
-
-       @param[in,out] mat The sparse matrix for which we want to initialize the
-                          row offsets.
-       @param[in] keep_nbr_block When set to true the SparseMatrix will
-                                 include the rows (in addition to the columns)
-                                 corresponding to face-neighbor dofs. The
-                                 default behavior is to disregard those rows. */
-   void FillI(SparseMatrix &mat,
-              const bool keep_nbr_block = false) const override;
-
+       given by this L2FaceRestriction. */
+   virtual void FillI(SparseMatrix &mat, const bool keep_nbr_block = false) const;
   /** Fill the I array of SparseMatrix corresponding to the sparsity pattern
-       given by this ParL2FaceRestriction. @a mat contains the interior dofs
+       given by this L2FaceRestriction. @a mat contains the interior dofs
       contribution, the @a face_mat contains the shared dofs contribution.*/
-   void FillI(SparseMatrix &mat,
-              SparseMatrix &face_mat) const;
-
+   virtual void FillI(SparseMatrix &mat, SparseMatrix &face_mat) const;
   /** Fill the J and Data arrays of SparseMatrix corresponding to the sparsity
-       pattern given by this ParL2FaceRestriction, and the values of ea_data.
+       pattern given by this L2FaceRestriction, and the values of ea_data.
       @a mat contains the interior dofs contribution, the @a face_mat contains
       the shared dofs contribution.*/
-   void FillJAndData(const Vector &ea_data,
-                     SparseMatrix &mat,
-                     SparseMatrix &face_mat) const;
+   virtual void FillJAndData(const Vector &ea_data,
+                             SparseMatrix &mat,
+                             SparseMatrix &face_mat) const;

-   /** @brief Fill the J and Data arrays of the SparseMatrix corresponding to
-       the sparsity pattern given by this ParL2FaceRestriction, and the values of
-       fea_data.
-
-       @param[in] fea_data The dense matrices representing the local operators
-                           on each face. The format is:
-                           face_dofs x face_dofs x 2 x nf.
-                           On each face the first local matrix corresponds to
-                           the contribution of elem1 on elem2, and the second to
-                           the contribution of elem2 on elem1.
-       @param[in,out] mat The sparse matrix that is getting filled.
-       @param[in] keep_nbr_block When set to true the SparseMatrix will
-                                 include the rows (in addition to the columns)
-                                 corresponding to face-neighbor dofs. The
-                                 default behavior is to disregard those rows. */
-   void FillJAndData(const Vector &fea_data,
-                     SparseMatrix &mat,
-                     const bool keep_nbr_block = false) const override;
-
-private:
-   /** @brief Compute the scatter indices: L-vector to E-vector, and the offsets
-       for the gathering: E-vector to L-vector.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeScatterIndicesAndOffsets(const ElementDofOrdering ordering,
-                                        const FaceType type);
-
-   /** @brief Compute the gather indices: E-vector to L-vector.
-
-       Note: Requires the gather offsets to be computed.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeGatherIndices(const ElementDofOrdering ordering,
-                             const FaceType type);
-
-public:
-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector. Should only be used with conforming faces and when:
-       m == L2FacesValues::DoubleValued
-
-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     face_dofs x vdim x 2 x nf
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
-   void DoubleValuedConformingMult(const Vector& x, Vector& y) const override;
-};
-
-/// Operator that extracts Face degrees of freedom for NCMesh in parallel.
-/** Objects of this type are typically created and owned by
-    ParFiniteElementSpace objects, see
-    ParFiniteElementSpace::GetFaceRestriction(). */
-class ParNCL2FaceRestriction
-   : public NCL2FaceRestriction, public ParL2FaceRestriction
-{
-public:
-   /** @brief Constructs an ParNCL2FaceRestriction.
-
-       @param[in] fes      The ParFiniteElementSpace on which this operates
-       @param[in] ordering Request a specific ordering
-       @param[in] type     Request internal or boundary faces dofs
-       @param[in] m        Request the face dofs for elem1, or both elem1 and
-                           elem2 */
-   ParNCL2FaceRestriction(const ParFiniteElementSpace& fes,
-                          ElementDofOrdering ordering,
-                          FaceType type,
-                          L2FaceValues m = L2FaceValues::DoubleValued);
-
-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector.
-
-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     if L2FacesValues::DoubleValued (face_dofs x vdim x 2 x nf),
-                     if L2FacesValues::SingleValued (face_dofs x vdim x nf),
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
-   void Mult(const Vector &x, Vector &y) const override;
-
-   /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
-       L-Vector.
-
-       @param[in]  x The face E-Vector degrees of freedom with the given format:
-                     if L2FacesValues::DoubleValued (face_dofs x vdim x 2 x nf),
-                     if L2FacesValues::SingleValued (face_dofs x vdim x nf),
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs should be ordered according to the given
-                     ElementDofOrdering
-       @param[in,out] y The L-vector degrees of freedom. */
-   void AddMultTranspose(const Vector &x, Vector &y) const override;
-
-   /** @brief Fill the I array of SparseMatrix corresponding to the sparsity
-       pattern given by this ParNCL2FaceRestriction.
-
-       @param[in,out] mat The sparse matrix for which we want to initialize the
-                          row offsets.
-       @param[in] keep_nbr_block When set to true the SparseMatrix will
-                                 include the rows (in addition to the columns)
-                                 corresponding to face-neighbor dofs. The
-                                 default behavior is to disregard those rows.
-
-       @warning This method is not implemented yet. */
-   void FillI(SparseMatrix &mat,
-              const bool keep_nbr_block = false) const override;
-
-   /** Fill the I array of SparseMatrix corresponding to the sparsity pattern
-       given by this ParNCL2FaceRestriction. @a mat contains the interior dofs
-       contribution, the @a face_mat contains the shared dofs contribution.
-
-       @warning This method is not implemented yet. */
-   void FillI(SparseMatrix &mat,
-              SparseMatrix &face_mat) const;
-
-   /** Fill the J and Data arrays of SparseMatrix corresponding to the sparsity
-       pattern given by this ParNCL2FaceRestriction, and the values of ea_data.
-       @a mat contains the interior dofs contribution, the @a face_mat contains
-       the shared dofs contribution.
-
-       @warning This method is not implemented yet. */
-   void FillJAndData(const Vector &fea_data,
-                     SparseMatrix &mat,
-                     SparseMatrix &face_mat) const;
-
-   /** @brief Fill the J and Data arrays of the SparseMatrix corresponding to
-       the sparsity pattern given by this ParNCL2FaceRestriction, and the values
-       of ea_data.
-
-       @param[in] fea_data The dense matrices representing the local operators
-                           on each face. The format is:
-                           face_dofs x face_dofs x 2 x nf.
-                           On each face the first local matrix corresponds to
-                           the contribution of elem1 on elem2, and the second to
-                           the contribution of elem2 on elem1.
-       @param[in,out] mat The sparse matrix that is getting filled.
-       @param[in] keep_nbr_block When set to true the SparseMatrix will
-                                 include the rows (in addition to the columns)
-                                 corresponding to face-neighbor dofs. The
-                                 default behavior is to disregard those rows.
-
-       @warning This method is not implemented yet. */
-   void FillJAndData(const Vector &fea_data,
-                     SparseMatrix &mat,
-                     const bool keep_nbr_block = false) const override;
-
-private:
-   /** @brief Compute the scatter indices: L-vector to E-vector, the offsets
-       for the gathering: E-vector to L-vector, and the interpolators from
-       coarse to fine face for master non-comforming faces.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeScatterIndicesAndOffsets(const ElementDofOrdering ordering,
-                                        const FaceType type);
-
-   /** @brief Compute the gather indices: E-vector to L-vector.
-
-       Note: Requires the gather offsets to be computed.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeGatherIndices(const ElementDofOrdering ordering,
-                             const FaceType type);
-
-public:
-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector. Should only be used with nonconforming faces and when:
-       L2FaceValues m == L2FaceValues::SingleValued
-
-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     (face_dofs x vdim x nf),
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
-   void SingleValuedNonconformingMult(const Vector& x, Vector& y) const;
-
-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector. Should only be used with nonconforming faces and when:
-       L2FaceValues m == L2FaceValues::DoubleValued
-
-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     (face_dofs x vdim x 2 x nf),
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
-   void DoubleValuedNonconformingMult(const Vector& x, Vector& y) const override;
+   virtual void FillJAndData(const Vector &ea_data,
+                             SparseMatrix &mat,
+                             const bool keep_nbr_block = false) const;
 };

 }
@@ -65,13 +65,13 @@ static void Values2D(const int NE,
      MFEM_SHARED double sm0[NBZ][MDQ*MDQ];
      MFEM_SHARED double sm1[NBZ][MDQ*MDQ];

-      kernels::internal::LoadB<MD1,MQ1>(D1D,Q1D,b,sB);
-
      ConstDeviceMatrix B(sB, D1D,Q1D);
      DeviceMatrix DD(sm0[tidz], MD1, MD1);
      DeviceMatrix DQ(sm1[tidz], MD1, MQ1);
      DeviceMatrix QQ(sm0[tidz], MQ1, MQ1);

+      kernels::internal::LoadB<MD1,MQ1>(D1D,Q1D,b,sB);
+
      for (int c = 0; c < VDIM; c++)
      {
         kernels::internal::LoadX(e,D1D,c,x,DD);
@@ -126,14 +126,14 @@ static void Values3D(const int NE,
      MFEM_SHARED double sm0[MDQ*MDQ*MDQ];
      MFEM_SHARED double sm1[MDQ*MDQ*MDQ];

-      kernels::internal::LoadB<MD1,MQ1>(D1D,Q1D,b,sB);
-
      ConstDeviceMatrix B(sB, D1D,Q1D);
      DeviceCube DDD(sm0, MD1,MD1,MD1);
      DeviceCube DDQ(sm1, MD1,MD1,MQ1);
      DeviceCube DQQ(sm0, MD1,MQ1,MQ1);
      DeviceCube QQQ(sm1, MQ1,MQ1,MQ1);

+      kernels::internal::LoadB<MD1,MQ1>(D1D,Q1D,b,sB);
+
      for (int c = 0; c < VDIM; c++)
      {
         kernels::internal::LoadX(e,D1D,c,x,DDD);
@@ -21,21 +21,18 @@ namespace mfem
 static void GetSigns(const FiniteElementSpace &fes, const FaceType type,
                     Array<bool> &signs)
 {
-   const Mesh &mesh = *fes.GetMesh();
-   const int dim = mesh.SpaceDimension();
+   const int dim = fes.GetMesh()->SpaceDimension();
+   int e1, e2;
+   int inf1, inf2;
   int face_id;
   int f_ind = 0;
-   for (int f = 0; f < mesh.GetNumFacesWithGhost(); ++f)
+   for (int f = 0; f < fes.GetNF(); ++f)
   {
-      Mesh::FaceInformation face = mesh.GetFaceInformation(f);
-      face_id = face.element[0].local_face_id;
-      if (face.IsNonconformingCoarse())
-      {
-         // We skip nonconforming coarse-fine faces as they are treated
-         // by the corresponding nonconforming fine-coarse faces.
-         continue;
-      }
-      else if ( face.IsOfFaceType(type) )
+      fes.GetMesh()->GetFaceElements(f, &e1, &e2);
+      fes.GetMesh()->GetFaceInfos(f, &inf1, &inf2);
+      face_id = inf1 / 64;
+      if ( (type==FaceType::Interior && (e2>=0 || (e2<0 && inf2>=0))) ||
+           (type==FaceType::Boundary && e2<0 && inf2<0) )
      {
         if (dim==2)
         {
@@ -41,7 +41,7 @@ protected:
   const int nedofs;
   Array<int> offsets;
   Array<int> indices;
-   Array<int> gather_map;
+   Array<int> gatherMap;

 public:
   ElementRestriction(const FiniteElementSpace&, ElementDofOrdering);
@@ -172,719 +172,125 @@ class H1FaceRestriction : public FaceRestriction
 {
 protected:
   const FiniteElementSpace &fes;
-   const int nf; // Number of faces of the requested type
+   const int nf;
   const int vdim;
   const bool byvdim;
-   const int face_dofs; // Number of dofs on each face
-   const int elem_dofs; // Number of dofs in each element
-   const int nfdofs; // Total number of face E-vector dofs
-   const int ndofs; // Total number of dofs
-   Array<int> scatter_indices; // Scattering indices for element 1 on each face
-   Array<int> gather_offsets; // offsets for the gathering indices of each dof
-   Array<int> gather_indices; // gathering indices for each dof
+   const int ndofs;
+   const int dof;
+   const int nfdofs;
+   Array<int> scatter_indices;
+   Array<int> offsets;
+   Array<int> gather_indices;

-   /** @brief Construct an H1FaceRestriction.
-
-       @param[in] fes      The FiniteElementSpace on which this operates
-       @param[in] ordering Request a specific element ordering
-       @param[in] type     Request internal or boundary faces dofs
-       @param[in] build    Request the NCL2FaceRestriction to compute the
-                           scatter/gather indices. False should only be used
-                           when inheriting from H1FaceRestriction.
-   */
-   H1FaceRestriction(const FiniteElementSpace& fes,
-                     const ElementDofOrdering ordering,
-                     const FaceType type,
-                     bool build);
 public:
-   /** @brief Construct an H1FaceRestriction.
+   /** @brief Constructor for a H1FaceRestriction.

-       @param[in] fes      The FiniteElementSpace on which this operates
-       @param[in] ordering Request a specific element ordering
-       @param[in] type     Request internal or boundary faces dofs */
+       @param[in] fes The FiniteElementSpace on which this H1FaceRestriction
+                      operates.
+       @param[in] ordering The requested output ordering of the
+                           H1FaceRestriction, either Native or Lexicographic.
+       @param[in] type The requested type of faces on which this operator
+                       extracts the degrees of freedom, either Interior or
+                       Boundary.
+   */
   H1FaceRestriction(const FiniteElementSpace& fes,
                     const ElementDofOrdering ordering,
                     const FaceType type);

-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector.
+   /** @brief Extract the face degrees of freedom from @a x into @a y.

-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     face_dofs x vdim x nf
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
+       @param[in]  x The L-vector of degrees of freedom.
+       @param[out] y The degrees of freedom on the face, corresponding to a face
+                     E-vector.
+   */
   void Mult(const Vector &x, Vector &y) const override;

-   /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
-       L-Vector.
+   /** @brief Add the face degrees of freedom @a x to the element degrees of
+       freedom @a y.

-       @param[in]  x The face E-Vector degrees of freedom with the given format:
-                     face_dofs x vdim x nf
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs should be ordered according to the given
-                     ElementDofOrdering
-       @param[in,out] y The L-vector degrees of freedom. */
+       @param[in]     x The face degrees of freedom on the face.
+       @param[in,out] y The L-vector of degrees of freedom to which we add the
+                        face degrees of freedom.
+   */
   void AddMultTranspose(const Vector &x, Vector &y) const override;
-
-private:
-   /** @brief Compute the scatter indices: L-vector to E-vector, and the offsets
-       for the gathering: E-vector to L-vector.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeScatterIndicesAndOffsets(const ElementDofOrdering ordering,
-                                        const FaceType type);
-
-   /** @brief Compute the gather indices: E-vector to L-vector.
-
-       Note: Requires the gather offsets to be computed.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeGatherIndices(const ElementDofOrdering ordering,
-                             const FaceType type);
-
-protected:
-   mutable Array<int> face_map; // Used in the computation of GetFaceDofs
-
-   /** @brief Verify that H1FaceRestriction is build from an H1 FESpace.
-
-       @param[in] ordering The FESpace element ordering.
-   */
-   void CheckFESpace(const ElementDofOrdering ordering);
-
-   /** @brief Set the scattering indices of elem1, and increment the offsets for
-       the face described by the @a face.
-
-       @param[in] face The face information of the current face.
-       @param[in] face_index The interior/boundary face index.
-       @param[in] ordering Request a specific element ordering.
-    */
-   void SetFaceDofsScatterIndices(const Mesh::FaceInformation &face,
-                                  const int face_index,
-                                  const ElementDofOrdering ordering);
-
-   /** @brief Set the gathering indices of elem1 for the interior face described
-       by the @a face.
-
-       @param[in] face The face information of the current face.
-       @param[in] face_index The interior/boundary face index.
-       @param[in] ordering Request a specific element ordering.
-    */
-   void SetFaceDofsGatherIndices(const Mesh::FaceInformation &face,
-                                 const int face_index,
-                                 const ElementDofOrdering ordering);
 };

-/// Operator that extracts Face degrees of freedom for L2 spaces.
+/// Operator that extracts Face degrees of freedom on L2 FiniteElementSpaces.
 /** Objects of this type are typically created and owned by FiniteElementSpace
    objects, see FiniteElementSpace::GetFaceRestriction(). */
 class L2FaceRestriction : public FaceRestriction
 {
 protected:
   const FiniteElementSpace &fes;
-   const int nf; // Number of faces of the requested type
-   const int ne; // Number of elements
-   const int vdim; // vdim
+   const int nf;
+   const int ne;
+   const int vdim;
   const bool byvdim;
-   const int face_dofs; // Number of dofs on each face
-   const int elem_dofs; // Number of dofs in each element
-   const int nfdofs; // Total number of dofs on the faces
-   const int ndofs; // Total number of dofs
-   const FaceType type;
+   const int ndofs;
+   const int dof;
+   const int elemDofs;
   const L2FaceValues m;
-   Array<int> scatter_indices1; // Scattering indices for element 1 on each face
-   Array<int> scatter_indices2; // Scattering indices for element 2 on each face
-   Array<int> gather_offsets; // offsets for the gathering indices of each dof
-   Array<int> gather_indices; // gathering indices for each dof
+   const int nfdofs;
+   Array<int> scatter_indices1;
+   Array<int> scatter_indices2;
+   Array<int> offsets;
+   Array<int> gather_indices;

-   /** @brief Constructs an L2FaceRestriction.
-
-       @param[in] fes      The FiniteElementSpace on which this operates
-       @param[in] ordering Request a specific ordering
-       @param[in] type     Request internal or boundary faces dofs
-       @param[in] m        Request the face dofs for elem1, or both elem1 and
-                           elem2
-       @param[in] build    Request the NCL2FaceRestriction to compute the
-                           scatter/gather indices. False should only be used
-                           when inheriting from L2FaceRestriction.
-   */
-   L2FaceRestriction(const FiniteElementSpace& fes,
-                     const ElementDofOrdering ordering,
-                     const FaceType type,
-                     const L2FaceValues m,
-                     bool build);
-
-public:
-   /** @brief Constructs an L2FaceRestriction.
-
-       @param[in] fes      The FiniteElementSpace on which this operates
-       @param[in] ordering Request a specific ordering
-       @param[in] type     Request internal or boundary faces dofs
-       @param[in] m        Request the face dofs for elem1, or both elem1 and
-                           elem2 */
-   L2FaceRestriction(const FiniteElementSpace& fes,
-                     const ElementDofOrdering ordering,
-                     const FaceType type,
+   L2FaceRestriction(const FiniteElementSpace&,
+                     const FaceType,
                     const L2FaceValues m = L2FaceValues::DoubleValued);

-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector.
+public:
+   L2FaceRestriction(const FiniteElementSpace&,
+                     const ElementDofOrdering,
+                     const FaceType,
+                     const L2FaceValues m = L2FaceValues::DoubleValued);

-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     if L2FacesValues::DoubleValued (face_dofs x vdim x 2 x nf)
-                     if L2FacesValues::SingleValued (face_dofs x vdim x nf)
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
+   /** @brief Extract the face degrees of freedom from @a x into @a y.
+
+       @param[in]  x The L-vector of degrees of freedom.
+       @param[out] y The degrees of freedom on the face, corresponding to a face
+                     E-vector.
+   */
   void Mult(const Vector &x, Vector &y) const override;

-   /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
-       L-Vector.
+   /** @brief Add the face degrees of freedom @a x to the element degrees of
+       freedom @a y.

-       @param[in]  x The face E-Vector degrees of freedom with the given format:
-                     if L2FacesValues::DoubleValued (face_dofs x vdim x 2 x nf)
-                     if L2FacesValues::SingleValued (face_dofs x vdim x nf)
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs should be ordered according to the given
-                     ElementDofOrdering
-       @param[in,out] y The L-vector degrees of freedom. */
+       @param[in]     x The face degrees of freedom on the face.
+       @param[in,out] y The L-vector of degrees of freedom to which we add the
+                        face degrees of freedom.
+   */
   void AddMultTranspose(const Vector &x, Vector &y) const override;

-   /** @brief Fill the I array of SparseMatrix corresponding to the sparsity
-       pattern given by this L2FaceRestriction.
-
-       @param[in,out] mat The sparse matrix for which we want to initialize the
-                          row offsets.
-       @param[in] keep_nbr_block When set to true the SparseMatrix will
-                                 include the rows (in addition to the columns)
-                                 corresponding to face-neighbor dofs. The
-                                 default behavior is to disregard those rows. */
+   /** Fill the I array of SparseMatrix corresponding to the sparsity pattern
+       given by this L2FaceRestriction. */
   virtual void FillI(SparseMatrix &mat, const bool keep_nbr_block = false) const;

-   /** @brief Fill the J and Data arrays of the SparseMatrix corresponding to
-       the sparsity pattern given by this L2FaceRestriction, and the values of
-       fea_data.
-
-       @param[in] fea_data The dense matrices representing the local operators
-                           on each face. The format is:
-                           face_dofs x face_dofs x 2 x nf
-                           On each face the first local matrix corresponds to
-                           the contribution of elem1 on elem2, and the second to
-                           the contribution of elem2 on elem1.
-       @param[in,out] mat The sparse matrix that is getting filled.
-       @param[in] keep_nbr_block When set to true the SparseMatrix will
-                                 include the rows (in addition to the columns)
-                                 corresponding to face-neighbor dofs. The
-                                 default behavior is to disregard those rows. */
-   virtual void FillJAndData(const Vector &fea_data,
+   /** Fill the J and Data arrays of SparseMatrix corresponding to the sparsity
+       pattern given by this L2FaceRestriction, and the values of ea_data. */
+   virtual void FillJAndData(const Vector &ea_data,
                             SparseMatrix &mat,
                             const bool keep_nbr_block = false) const;

-   /** @brief This methods adds the DG face matrices to the element matrices.
-
-       @param[in] fea_data The dense matrices representing the local operators
-                           on each face. The format is:
-                           face_dofs x face_dofs x 2 x nf
-                           On each face the first and second local matrices
-                           correspond to the contributions of elem1 and elem2 on
-                           themselves respectively.
-       @param[in,out] ea_data The dense matrices representing the element local
-                              contributions for each element to which will be
-                              added the face contributions.
-                              The format is: dofs x dofs x ne, where dofs is the
-                              number of dofs per element and ne the number of
-                              elements. */
-   virtual void AddFaceMatricesToElementMatrices(const Vector &fea_data,
-                                                 Vector &ea_data) const;
-
-private:
-   /** @brief Compute the scatter indices: L-vector to E-vector, and the offsets
-       for the gathering: E-vector to L-vector.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeScatterIndicesAndOffsets(const ElementDofOrdering ordering,
-                                        const FaceType type);
-
-   /** @brief Compute the gather indices: E-vector to L-vector.
-
-       Note: Requires the gather offsets to be computed.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeGatherIndices(const ElementDofOrdering ordering,
-                             const FaceType type);
-
-protected:
-   mutable Array<int> face_map; // Used in the computation of GetFaceDofs
-
-   /** @brief Verify that L2FaceRestriction is build from an L2 FESpace.
-
-       @param[in] ordering The FESpace element ordering.
-   */
-   void CheckFESpace(const ElementDofOrdering ordering);
-
-   /** @brief Set the scattering indices of elem1, and increment the offsets for
-       the face described by the @a face. The ordering of the face dofs of elem1
-       is lexicographic relative to elem1.
-
-       @param[in] face The face information of the current face.
-       @param[in] face_index The interior/boundary face index.
-    */
-   void SetFaceDofsScatterIndices1(const Mesh::FaceInformation &face,
-                                   const int face_index);
-
-   /** @brief Permute and set the scattering indices of elem2, and increment the
-       offsets for the face described by the @a face. The permutation orders the
-       dofs of elem2 lexicographically as the ones of elem1.
-
-       @param[in] face The face information of the current face.
-       @param[in] face_index The interior/boundary face index.
-    */
-   void PermuteAndSetFaceDofsScatterIndices2(const Mesh::FaceInformation &face,
-                                             const int face_index);
-
-   /** @brief Permute and set the scattering indices of elem2 for the shared
-       face described by the @a face. The permutation orders the dofs of elem2 as
-       the ones of elem1.
-
-       @param[in] face The face information of the current face.
-       @param[in] face_index The interior/boundary face index.
-    */
-   void PermuteAndSetSharedFaceDofsScatterIndices2(
-      const Mesh::FaceInformation &face,
-      const int face_index);
-
-   /** @brief Set the scattering indices of elem2 for the boundary face
-       described by the @a face.
-
-       @param[in] face The face information of the current face.
-       @param[in] face_index The interior/boundary face index.
-    */
-   void SetBoundaryDofsScatterIndices2(const Mesh::FaceInformation &face,
-                                       const int face_index);
-
-   /** @brief Set the gathering indices of elem1 for the interior face described
-       by the @a face.
-
-       Note: This function modifies the offsets.
-
-       @param[in] face The face information of the current face.
-       @param[in] face_index The interior/boundary face index.
-    */
-   void SetFaceDofsGatherIndices1(const Mesh::FaceInformation &face,
-                                  const int face_index);
-
-   /** @brief Permute and set the gathering indices of elem2 for the interior
-       face described by the @a face. The permutation orders the dofs of elem2 as
-       the ones of elem1.
-
-       Note: This function modifies the offsets.
-
-       @param[in] face The face information of the current face.
-       @param[in] face_index The interior/boundary face index.
-    */
-   void PermuteAndSetFaceDofsGatherIndices2(const Mesh::FaceInformation &face,
-                                            const int face_index);
-
-public:
-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector. Should only be used with conforming faces and when:
-       m == L2FacesValues::SingleValued
-
-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     face_dofs x vdim x nf
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
-   void SingleValuedConformingMult(const Vector& x, Vector& y) const;
-
-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector. Should only be used with conforming faces and when:
-       m == L2FacesValues::DoubleValued
-
-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     face_dofs x vdim x 2 x nf
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
-   virtual void DoubleValuedConformingMult(const Vector& x, Vector& y) const;
-
-   /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
-       L-Vector. Should only be used with conforming faces and when:
-       m == L2FacesValues::SingleValued
-
-       @param[in]  x The face E-Vector degrees of freedom with the given format:
-                     face_dofs x vdim x nf
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs should be ordered according to the given
-                     ElementDofOrdering
-       @param[in,out] y The L-vector degrees of freedom. */
-   void SingleValuedConformingAddMultTranspose(const Vector& x, Vector& y) const;
-
-   /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
-       L-Vector. Should only be used with conforming faces and when:
-       m == L2FacesValues::DoubleValued
-
-       @param[in]  x The face E-Vector degrees of freedom with the given format:
-                     face_dofs x vdim x 2 x nf
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs should be ordered according to the given
-                     ElementDofOrdering
-       @param[in,out] y The L-vector degrees of freedom. */
-   void DoubleValuedConformingAddMultTranspose(const Vector& x, Vector& y) const;
+   /// This methods adds the DG face matrices to the element matrices.
+   void AddFaceMatricesToElementMatrices(Vector &fea_data,
+                                         Vector &ea_data) const;
 };

-/** This struct stores which side is the master nonconforming side and the
-    index of the interpolator, see InterpolationManager class below. */
-struct InterpConfig
-{
-   uint32_t is_non_conforming : 1;
-   uint32_t master_side : 1;
-   uint32_t index : 30;
-
-   // default constructor, create a conforming face with index 0.
-   InterpConfig() = default;
-
-   // Non-conforming face
-   InterpConfig(int master_side, int nc_index)
-      : is_non_conforming(1), master_side(master_side), index(nc_index)
-   { }
-
-   InterpConfig(const InterpConfig&) = default;
-
-   InterpConfig &operator=(const InterpConfig &rhs) = default;
-};
-
-/** @brief This class manages the storage and computation of the interpolations
-    from master (coarse) face to slave (fine) face.
-*/
-class InterpolationManager
-{
-protected:
-   const FiniteElementSpace &fes;
-   const ElementDofOrdering ordering;
-   Array<InterpConfig> interp_config; // interpolator index for each face
-   Vector interpolators; // face_dofs x face_dofs x num_interpolators
-   int nc_cpt; // Counter for interpolators, and used as index.
-
-   /** The interpolators are associated to a key of containing the address of
-       PointMatrix and a local face identifier. */
-   using Key = std::pair<const DenseMatrix*,int>;
-   /// The temporary map used to store the different interpolators.
-   using Map = std::map<Key, std::pair<int,const DenseMatrix*>>;
-   Map interp_map; // The temporary map that stores the interpolators.
-
-public:
-   InterpolationManager() = delete;
-
-   /** @brief main constructor.
-
-       @param[in] fes      The FiniteElementSpace on which this operates
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs
-    */
-   InterpolationManager(const FiniteElementSpace &fes,
-                        ElementDofOrdering ordering,
-                        FaceType type);
-
-   /** @brief Register the face with @a face and index @a face_index as a
-       conforming face for the interpolation of the degrees of freedom.
-
-       @param[in] face The face information of the current face.
-       @param[in] face_index The interior/boundary face index.
-    */
-   void RegisterFaceConformingInterpolation(const Mesh::FaceInformation &face,
-                                            int face_index);
-
-   /** @brief Register the face with @a face and index @a face_index as a
-       conforming face for the interpolation of the degrees of freedom.
-
-       @param[in] face The face information of the current face.
-       @param[in] face_index The interior/boundary face index.
-    */
-   void RegisterFaceCoarseToFineInterpolation(const Mesh::FaceInformation &face,
-                                              int face_index);
-
-   /** @brief Transform the interpolation matrix map into a contiguous memory
-       structure. */
-   void LinearizeInterpolatorMapIntoVector();
-
-   /// @brief Return the total number of interpolators.
-   int GetNumInterpolators() const
-   {
-      return nc_cpt;
-   }
-
-   /** @brief Return an mfem::Vector containing the interpolators in the
-       following format: face_dofs x face_dofs x num_interpolators. */
-   const Vector& GetInterpolators() const
-   {
-      return interpolators;
-   }
-
-   /** @brief Return an array containing the interpolation configuration for
-       each face registered with RegisterFaceConformingInterpolation and
-       RegisterFaceCoarseToFineInterpolation. */
-   const Array<InterpConfig>& GetFaceInterpConfig() const
-   {
-      return interp_config;
-   }
-
-private:
-   /** @brief Returns the interpolation operator from a master (coarse) face to
-       a slave (fine) face.
-
-       @param[in] face The face information of the current face.
-       @param[in] ptMat The PointMatrix describing the position and orientation
-                        of the fine face in the coarse face. This PointMatrix is
-                        usually obtained from the mesh through the method
-                        GetNCFacesPtMat.
-       @param[in] ordering  Request a specific element ordering.
-       @return The dense matrix corresponding to the interpolation of the face
-               degrees of freedom of the master (coarse) face to the slave
-               (fine) face. */
-   const DenseMatrix* GetCoarseToFineInterpolation(
-      const Mesh::FaceInformation &face,
-      const DenseMatrix* ptMat);
-};
-
-/** @brief Operator that extracts face degrees of freedom for L2 nonconforming
-    spaces.
-
-    In order to support face restrictions on nonconforming meshes, this
-    operator interpolates master (coarse) face degrees of freedom onto the
-    slave (fine) face. This allows face integrators to treat nonconforming
-    faces just as regular conforming faces. */
-class NCL2FaceRestriction : virtual public L2FaceRestriction
-{
-protected:
-   InterpolationManager interpolations;
-   mutable Vector x_interp;
-
-   /** @brief Constructs an NCL2FaceRestriction, this is a specialization of a
-       L2FaceRestriction for nonconforming meshes.
-
-       @param[in] fes      The FiniteElementSpace on which this operates
-       @param[in] ordering Request a specific ordering
-       @param[in] type     Request internal or boundary faces dofs
-       @param[in] m        Request the face dofs for elem1, or both elem1 and
-                           elem2
-       @param[in] build    Request the NCL2FaceRestriction to compute the
-                           scatter/gather indices. False should only be used
-                           when inheriting from NCL2FaceRestriction.
-   */
-   NCL2FaceRestriction(const FiniteElementSpace& fes,
-                       const ElementDofOrdering ordering,
-                       const FaceType type,
-                       const L2FaceValues m,
-                       bool build);
-public:
-   /** @brief Constructs an NCL2FaceRestriction, this is a specialization of a
-       L2FaceRestriction for nonconforming meshes.
-
-       @param[in] fes      The FiniteElementSpace on which this operates
-       @param[in] ordering Request a specific ordering
-       @param[in] type     Request internal or boundary faces dofs
-       @param[in] m        Request the face dofs for elem1, or both elem1 and
-                           elem2
-   */
-   NCL2FaceRestriction(const FiniteElementSpace& fes,
-                       const ElementDofOrdering ordering,
-                       const FaceType type,
-                       const L2FaceValues m = L2FaceValues::DoubleValued);
-
-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector.
-
-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     if L2FacesValues::DoubleValued (face_dofs x vdim x 2 x nf),
-                     if L2FacesValues::SingleValued (face_dofs x vdim x nf),
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
-   void Mult(const Vector &x, Vector &y) const override;
-
-   /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
-       L-Vector.
-
-       @param[in]  x The face E-Vector degrees of freedom with the given format:
-                     if L2FacesValues::DoubleValued (face_dofs x vdim x 2 x nf),
-                     if L2FacesValues::SingleValued (face_dofs x vdim x nf),
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs should be ordered according to the given
-                     ElementDofOrdering
-       @param[in,out] y The L-vector degrees of freedom. */
-   void AddMultTranspose(const Vector &x, Vector &y) const override;
-
-   /** @brief Fill the I array of SparseMatrix corresponding to the sparsity
-       pattern given by this NCL2FaceRestriction.
-
-       @param[in,out] mat The sparse matrix for which we want to initialize the
-                          row offsets.
-       @param[in] keep_nbr_block When set to true the SparseMatrix will
-                                 include the rows (in addition to the columns)
-                                 corresponding to face-neighbor dofs. The
-                                 default behavior is to disregard those rows.
-
-       @warning This method is not implemented yet. */
-   void FillI(SparseMatrix &mat,
-              const bool keep_nbr_block = false) const override;
-
-   /** @brief Fill the J and Data arrays of the SparseMatrix corresponding to
-       the sparsity pattern given by this NCL2FaceRestriction, and the values of
-       ea_data.
-
-       @param[in] fea_data The dense matrices representing the local operators
-                           on each face. The format is:
-                           face_dofs x face_dofs x 2 x nf.
-                           On each face the first local matrix corresponds to
-                           the contribution of elem1 on elem2, and the second to
-                           the contribution of elem2 on elem1.
-       @param[in,out] mat The sparse matrix that is getting filled.
-       @param[in] keep_nbr_block When set to true the SparseMatrix will
-                                 include the rows (in addition to the columns)
-                                 corresponding to face-neighbor dofs. The
-                                 default behavior is to disregard those rows.
-
-       @warning This method is not implemented yet. */
-   void FillJAndData(const Vector &fea_data,
-                     SparseMatrix &mat,
-                     const bool keep_nbr_block = false) const override;
-
-   /** @brief This methods adds the DG face matrices to the element matrices.
-
-       @param[in] fea_data The dense matrices representing the local operators
-                           on each face. The format is:
-                           face_dofs x face_dofs x 2 x nf.
-                           On each face the first and second local matrices
-                           correspond to the contributions of elem1 and elem2 on
-                           themselves respectively.
-       @param[in,out] ea_data The dense matrices representing the element local
-                              contributions for each element to which will be
-                              added the face contributions.
-                              The format is: dofs x dofs x ne, where dofs is the
-                              number of dofs per element and ne the number of
-                              elements.
-
-       @warning This method is not implemented yet. */
-   void AddFaceMatricesToElementMatrices(const Vector &fea_data,
-                                         Vector &ea_data) const override;
-
-private:
-   /** @brief Compute the scatter indices: L-vector to E-vector, the offsets
-       for the gathering: E-vector to L-vector, and the interpolators from
-       coarse to fine face for master non-comforming faces.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeScatterIndicesAndOffsets(const ElementDofOrdering ordering,
-                                        const FaceType type);
-
-   /** @brief Compute the gather indices: E-vector to L-vector.
-
-       Note: Requires the gather offsets to be computed.
-
-       @param[in] ordering Request a specific element ordering.
-       @param[in] type     Request internal or boundary faces dofs.
-   */
-   void ComputeGatherIndices(const ElementDofOrdering ordering,
-                             const FaceType type);
-
-public:
-   /** @brief Scatter the degrees of freedom, i.e. goes from L-Vector to
-       face E-Vector. Should only be used with nonconforming faces and when:
-       L2FaceValues m == L2FaceValues::DoubleValued
-
-       @param[in]  x The L-vector degrees of freedom.
-       @param[out] y The face E-Vector degrees of freedom with the given format:
-                     (face_dofs x vdim x 2 x nf),
-                     where nf is the number of interior or boundary faces
-                     requested by @a type in the constructor.
-                     The face_dofs are ordered according to the given
-                     ElementDofOrdering. */
-   virtual void DoubleValuedNonconformingMult(const Vector& x, Vector& y) const;
-
-   /** @brief Apply a change of basis from fine element basis to coarse element
-       basis for the coarse face dofs. Should only be used when:
-       L2FaceValues m == L2FaceValues::SingleValued
-
-       @param[in] x The dofs vector that needs coarse dofs to be express in term
-                    of the coarse basis, the result is stored in x_interp.
-   */
-   void SingleValuedNonconformingTransposeInterpolation(const Vector& x) const;
-
-   /** @brief Apply a change of basis from fine element basis to coarse element
-       basis for the coarse face dofs. Should only be used when:
-       L2FaceValues m == L2FaceValues::DoubleValued
-
-       @param[in] x The dofs vector that needs coarse dofs to be express in term
-                    of the coarse basis, the result is stored in x_interp.
-   */
-   void DoubleValuedNonconformingTransposeInterpolation(const Vector& x) const;
-};
-
-/** @brief Return the face map that extracts the degrees of freedom for the
-    requested local face of a quad or hex, returned in Lexicographic order.
-
-    @param[in] dim The dimension of the space
-    @param[in] face_id The local face identifier
-    @param[in] dof1d The 1D number of degrees of freedom for each dimension
-    @param[out] face_map The map that maps each face dof to an element dof
-*/
+// Return the face degrees of freedom returned in Lexicographic order.
 void GetFaceDofs(const int dim, const int face_id,
-                 const int dof1d, Array<int> &face_map);
+                 const int dof1d, Array<int> &faceMap);

-/** @brief Convert a dof face index from Native ordering to lexicographic
-    ordering for quads and hexes.
-
-    @param[in] dim The dimension of the element, 2 for quad, 3 for hex
-    @param[in] face_id The local face identifier
-    @param[in] size1d The 1D number of degrees of freedom for each dimension
-    @param[in] index The native index on the face
-    @return The lexicographic index on the face
-*/
+// Convert from Native ordering to lexicographic ordering
 int ToLexOrdering(const int dim, const int face_id, const int size1d,
                  const int index);

-/** @brief Compute the dof face index of elem2 corresponding to the given dof
-    face index.
-
-    @param[in] dim The dimension of the element, 2 for quad, 3 for hex
-    @param[in] face_id1 The local face identifier of elem1
-    @param[in] face_id2 The local face identifier of elem2
-    @param[in] orientation The orientation of elem2 relative to elem1 on the
-                           face
-    @param[in] size1d The 1D number of degrees of freedom for each dimension
-    @param[in] index The dof index on elem1
-    @return The dof index on elem2 facing the dof on elem1
-*/
+// Permute dofs or quads on a face for e2 to match with the ordering of e1
 int PermuteFaceL2(const int dim, const int face_id1,
                  const int face_id2, const int orientation,
                  const int size1d, const int index);

 }

-#endif // MFEM_RESTRICTION
+#endif //MFEM_RESTRICTION
@@ -284,7 +284,7 @@ void StaticCondensation::Finalize()
 }

 void StaticCondensation::EliminateReducedTrueDofs(
-   const Array<int> &ess_rtdof_list_, Matrix::DiagonalPolicy dpolicy)
+   const Array<int> &ess_rtdof_list, Matrix::DiagonalPolicy dpolicy)
 {
   if (!Parallel() || S) // not parallel or not finalized
   {
@@ -292,16 +292,16 @@ void StaticCondensation::EliminateReducedTrueDofs(
      {
         S_e = new SparseMatrix(S->Height());
      }
-      for (int i = 0; i < ess_rtdof_list_.Size(); i++)
+      for (int i = 0; i < ess_rtdof_list.Size(); i++)
      {
-         S->EliminateRowCol(ess_rtdof_list_[i], *S_e, dpolicy);
+         S->EliminateRowCol(ess_rtdof_list[i], *S_e, dpolicy);
      }
   }
   else // parallel and finalized
   {
 #ifdef MFEM_USE_MPI
      MFEM_ASSERT(pS_e.Ptr() == NULL, "essential b.c. already eliminated");
-      pS_e.EliminateRowsCols(pS, ess_rtdof_list_);
+      pS_e.EliminateRowsCols(pS, ess_rtdof_list);
 #endif
   }
 }
@@ -199,14 +199,14 @@ public:

   /** Restrict a list of true FE space dofs to a list of reduced/trace true FE
       space dofs. */
-   void ConvertListToReducedTrueDofs(const Array<int> &ess_tdof_list_,
-                                     Array<int> &ess_rtdof_list_) const
+   void ConvertListToReducedTrueDofs(const Array<int> &ess_tdof_list,
+                                     Array<int> &ess_rtdof_list) const
   {
      Array<int> ess_tdof_marker, ess_rtdof_marker;
-      FiniteElementSpace::ListToMarker(ess_tdof_list_, fes->GetTrueVSize(),
+      FiniteElementSpace::ListToMarker(ess_tdof_list, fes->GetTrueVSize(),
                                       ess_tdof_marker);
      ConvertMarkerToReducedTrueDofs(ess_tdof_marker, ess_rtdof_marker);
-      FiniteElementSpace::MarkerToList(ess_rtdof_marker, ess_rtdof_list_);
+      FiniteElementSpace::MarkerToList(ess_rtdof_marker, ess_rtdof_list);
   }

   /** Given a solution of the reduced system 'sc_sol' and the RHS 'b' for the
@@ -274,18 +274,18 @@ public:
   {
      y = 0.0;

-      solVecLayout_type solVecLayoutLoc(this->solVecLayout);
-      solFESpace solFESLoc(this->solFES);
+      solVecLayout_type solVecLayout(this->solVecLayout);
+      solFESpace solFES(this->solFES);

      TTensor3<dofs,vdim,BE,vcomplex_t> xy_dof;

      const int NE = mesh.GetNE();
      for (int el = 0; el < NE; el += TE)
      {
-         solFESLoc.SetElement(el);
+         solFES.SetElement(el);

-         solFESLoc.VectorExtract(solVecLayoutLoc, x, xy_dof.layout, xy_dof);
-         solFESLoc.VectorAssemble(xy_dof.layout, xy_dof, solVecLayoutLoc, y);
+         solFES.VectorExtract(solVecLayout, x, xy_dof.layout, xy_dof);
+         solFES.VectorAssemble(xy_dof.layout, xy_dof, solVecLayout, y);
      }
   }

@@ -346,8 +346,8 @@ public:
   {
      typedef TTensor3<dofs,vdim,BE,vcomplex_t> vdof_data_t;

-      solVecLayout_t solVecLayoutLoc(this->solVecLayout);
-      solFESpace solFESLoc(this->solFES);
+      solVecLayout_t solVecLayout(this->solVecLayout);
+      solFESpace solFES(this->solFES);

      const int NE = mesh.GetNE();
      // TODO: How do we make sure that this array is aligned properly, AND
@@ -358,8 +358,8 @@ public:
      sx.MakeDataOwner();
      for (int el = 0; el < NE; el += TE)
      {
-         solFESLoc.SetElement(el);
-         solFESLoc.VectorExtract(solVecLayoutLoc, x, vdof_data_t::layout, vsx);
+         solFES.SetElement(el);
+         solFES.VectorExtract(solVecLayout, x, vdof_data_t::layout, vsx);
         vsx += vdof_data_t::size;
      }
   }
@@ -397,9 +397,9 @@ public:
   void AssembleMatrix(SparseMatrix &M) const
   {
      Trans_t T(mesh, meshEval);
-      solFESpace solFESLoc(this->solFES);
-      solShapeEval solEvalLoc(this->solEval);
-      solVecLayout_t solVecLayoutLoc(this->solVecLayout);
+      solFESpace solFES(this->solFES);
+      solShapeEval solEval(this->solEval);
+      solVecLayout_t solVecLayout(this->solVecLayout);
      coeff_eval_t wQ(int_rule, coeff);

      const int NE = mesh.GetNE();
@@ -429,12 +429,12 @@ public:
            TMatrix<dofs,dofs,vcomplex_t> M_loc;
            S_spec::ElementMatrix::Compute(
               asm_qpt_data[k].layout, asm_qpt_data[k], M_loc.layout, M_loc,
-               solEvalLoc);
+               solEval);

-            solFESLoc.SetElement(el_k);
+            solFES.SetElement(el_k);
            for (int bi = 0; bi < vdim; bi++)
            {
-               solFESLoc.AssembleBlock(bi, bi, solVecLayoutLoc, M_loc, M);
+               solFES.AssembleBlock(bi, bi, solVecLayout, M_loc, M);
            }
         }
      }
@@ -445,7 +445,7 @@ public:
   void AssembleMatrix(DenseTensor &M) const
   {
      Trans_t T(mesh, meshEval);
-      solShapeEval solEvalLoc(this->solEval);
+      solShapeEval solEval(this->solEval);
      coeff_eval_t wQ(int_rule, coeff);

      const int NE = mesh.GetNE();
@@ -476,7 +476,7 @@ public:
            TMatrix<dofs,dofs,vcomplex_t> M_loc;
            S_spec::ElementMatrix::Compute(
               asm_qpt_data[k].layout, asm_qpt_data[k], M_loc.layout, M_loc,
-               solEvalLoc);
+               solEval);

            for (int s = 0; s < SS && el_k+s < NE; s++)
            {
@@ -498,7 +498,7 @@ public:
   void AssembleBilinearForm(BilinearForm &a) const
   {
      Trans_t T(mesh, meshEval);
-      solShapeEval solEvalLoc(this->solEval);
+      solShapeEval solEval(this->solEval);
      coeff_eval_t wQ(int_rule, coeff);

      Array<int> vdofs;
@@ -533,7 +533,7 @@ public:
            TMatrix<dofs,dofs,vcomplex_t> M_loc;
            S_spec::ElementMatrix::Compute(
               asm_qpt_data[k].layout, asm_qpt_data[k], M_loc.layout, M_loc,
-               solEvalLoc);
+               solEval);

            if (dof_map) // switch from tensor-product ordering
            {
@@ -600,19 +600,19 @@ public:
      // For now, when vdim > 1, assume block-diagonal matrix with the same
      // diagonal block for all components.
      // M is assumed to be (dof x dof x NE).
-      solVecLayout_t solVecLayoutLoc(this->solVecLayout);
+      solVecLayout_t solVecLayout(this->solVecLayout);
      const int NE = mesh.GetNE();
      for (int el = 0; el < NE; el++)
      {
         TTensor3<dofs,vdim,1,AutoSIMD<complex_t,1,1> > x_dof, y_dof;

         solFES.SetElement(el);
-         solFES.VectorExtract(solVecLayoutLoc, x, x_dof.layout, x_dof);
+         solFES.VectorExtract(solVecLayout, x, x_dof.layout, x_dof);
         Mult_AB<false>(TMatrix<dofs,dofs>::layout,
                        M(el).Data(),
                        x_dof.layout.merge_23(), x_dof,
                        y_dof.layout.merge_23(), y_dof);
-         solFES.VectorAssemble(y_dof.layout, y_dof, solVecLayoutLoc, y);
+         solFES.VectorAssemble(y_dof.layout, y_dof, solVecLayout, y);
      }
   }
 };
@@ -1133,7 +1133,7 @@ protected:

   // Evaluation of the discrete target specification on different meshes.
   // Owned.
-   AdaptivityEvaluator *adapt_lim_eval;
+   AdaptivityEvaluator *adapt_eval;

   void SetDiscreteTargetBase(const GridFunction &tspec_);
   void SetTspecAtIndex(int idx, const GridFunction &tspec_);
@@ -1156,7 +1156,7 @@ public:
 #endif
        amr_el(-1), lim_min_size(-0.1),
        good_tspec(false), good_tspec_grad(false), good_tspec_hess(false),
-        adapt_lim_eval(NULL) { }
+        adapt_eval(NULL) { }

   virtual ~DiscreteAdaptTC();

@@ -1232,8 +1232,8 @@ public:

   void SetAdaptivityEvaluator(AdaptivityEvaluator *ae)
   {
-      if (adapt_lim_eval) { delete adapt_lim_eval; }
-      adapt_lim_eval = ae;
+      if (adapt_eval) { delete adapt_eval; }
+      adapt_eval = ae;
   }

   const Vector &GetTspecPert1H()   { return tspec_pert1h; }
@@ -1315,15 +1315,15 @@ protected:
   int integ_order;

   // Weight Coefficient multiplying the quality metric term.
-   Coefficient *metric_coeff; // not owned, if NULL -> metric_coeff is 1.
+   Coefficient *coeff1; // not owned, if NULL -> coeff1 is 1.
   // Normalization factor for the metric term.
   double metric_normal;

   // Nodes and weight Coefficient used for "limiting" the TMOP_Integrator.
   // These are both NULL when there is no limiting.
-   // The class doesn't own lim_nodes0 and lim_coeff.
-   const GridFunction *lim_nodes0;
-   Coefficient *lim_coeff;
+   // The class doesn't own nodes0 and coeff0.
+   const GridFunction *nodes0;
+   Coefficient *coeff0;
   // Limiting reference distance. Not owned.
   const GridFunction *lim_dist;
   // Limiting function. Owned.
@@ -1332,21 +1332,20 @@ protected:
   double lim_normal;

   // Adaptive limiting.
-   const GridFunction *adapt_lim_gf0;    // Not owned.
+   const GridFunction *zeta_0;       // Not owned.
 #ifdef MFEM_USE_MPI
-   const ParGridFunction *adapt_lim_pgf0;
+   const ParGridFunction *pzeta_0;
 #endif
-   GridFunction *adapt_lim_gf;           // Owned. Updated by adapt_lim_eval.
-   Coefficient *adapt_lim_coeff;         // Not owned.
-   AdaptivityEvaluator *adapt_lim_eval;  // Not owned.
+   GridFunction *zeta;               // Owned. Updated by adapt_eval.
+   Coefficient *coeff_zeta;          // Not owned.
+   AdaptivityEvaluator *adapt_eval;  // Not owned.

   // Surface fitting.
-   GridFunction *surf_fit_gf,
-                *surf_fit_gf_bar;       // Owned, Updated by surf_fit_eval.
-   const Array<bool> *surf_fit_marker;  // Not owned.
-   Coefficient *surf_fit_coeff;         // Not owned.
-   AdaptivityEvaluator *surf_fit_eval;  // Not owned.
-   double surf_fit_normal;
+   GridFunction *sigma, *sigma_bar;  // Owned. Updated by sigma_eval.
+   const Array<bool> *sigma_marker;  // Not owned.
+   Coefficient *coeff_sigma;         // Not owned.
+   AdaptivityEvaluator *sigma_eval;  // Not owned.
+   double sigma_normal;

   DiscreteAdaptTC *discr_tc;

@@ -1417,7 +1416,7 @@ protected:

   void ComputeNormalizationEnergies(const GridFunction &x,
                                     double &metric_energy, double &lim_energy,
-                                     double &surf_fit_gf_energy);
+                                     double &sigma_energy);

   void AssembleElementVectorExact(const FiniteElement &el,
                                   ElementTransformation &T,
@@ -1472,7 +1471,7 @@ protected:

   void DisableLimiting()
   {
-      lim_nodes0 = NULL; lim_coeff = NULL; lim_dist = NULL;
+      nodes0 = NULL; coeff0 = NULL; lim_dist = NULL;
      delete lim_func; lim_func = NULL;
   }

@@ -1532,14 +1531,12 @@ public:
   TMOP_Integrator(TMOP_QualityMetric *m, TargetConstructor *tc,
                   TMOP_QualityMetric *hm)
      : h_metric(hm), metric(m), targetC(tc), IntegRules(NULL),
-        integ_order(-1), metric_coeff(NULL), metric_normal(1.0),
-        lim_nodes0(NULL), lim_coeff(NULL),
+        integ_order(-1), coeff1(NULL), metric_normal(1.0),
+        nodes0(NULL), coeff0(NULL),
        lim_dist(NULL), lim_func(NULL), lim_normal(1.0),
-        adapt_lim_gf0(NULL), adapt_lim_gf(NULL), adapt_lim_coeff(NULL),
-        adapt_lim_eval(NULL),
-        surf_fit_gf(NULL), surf_fit_gf_bar(NULL), surf_fit_marker(NULL),
-        surf_fit_coeff(NULL),
-        surf_fit_eval(NULL), surf_fit_normal(1.0),
+        zeta_0(NULL), zeta(NULL), coeff_zeta(NULL), adapt_eval(NULL),
+        sigma(NULL), sigma_bar(NULL), sigma_marker(NULL), coeff_sigma(NULL),
+        sigma_eval(NULL), sigma_normal(1.0),
        discr_tc(dynamic_cast<DiscreteAdaptTC *>(tc)),
        fdflag(false), dxscale(1.0e3), fd_call_flag(false), exact_action(false)
   { PA.enabled = false; }
@@ -1551,7 +1548,7 @@ public:

   /// Release the device memory of large PA allocations. This will copy device
   /// memory back to the host before releasing.
-   void ReleasePADeviceMemory(bool copy_to_host = true);
+   void ReleasePADeviceMemory();

   /// Prescribe a set of integration rules; relevant for mixed meshes.
   /** This function has priority over SetIntRule(), if both are called. */
@@ -1567,7 +1564,7 @@ public:

       Note that the Coefficient is evaluated in the physical configuration and
       not in the target configuration which may be undefined. */
-   void SetCoefficient(Coefficient &w1) { metric_coeff = &w1; }
+   void SetCoefficient(Coefficient &w1) { coeff1 = &w1; }

   /** @brief Limiting of the mesh displacements (general version).

@@ -1634,7 +1631,7 @@ public:
   void GetSurfaceFittingErrors(double &err_avg, double &err_max);

   /// Update the original/reference nodes used for limiting.
-   void SetLimitingNodes(const GridFunction &n0) { lim_nodes0 = &n0; }
+   void SetLimitingNodes(const GridFunction &n0) { nodes0 = &n0; }

   /** @brief Computes the integral of W(Jacobian(Trt)) over a target zone.
       @param[in] el     Type of FiniteElement.
@@ -38,13 +38,13 @@ void TMOP_Integrator::AssembleGradPA(const Vector &xe,
   if (PA.dim == 2)
   {
      AssembleGradPA_2D(xe);
-      if (lim_coeff) { AssembleGradPA_C0_2D(xe); }
+      if (coeff0) { AssembleGradPA_C0_2D(xe); }
   }

   if (PA.dim == 3)
   {
      AssembleGradPA_3D(xe);
-      if (lim_coeff) { AssembleGradPA_C0_3D(xe); }
+      if (coeff0) { AssembleGradPA_C0_3D(xe); }
   }
 }

@@ -53,8 +53,8 @@ void TMOP_Integrator::AssemblePA_Limiting()
   const MemoryType mt = (pa_mt == MemoryType::DEFAULT) ?
                         Device::GetDeviceMemoryType() : pa_mt;
   // Return immediately if limiting is not enabled
-   if (lim_coeff == nullptr) { return; }
-   MFEM_VERIFY(lim_nodes0, "internal error");
+   if (coeff0 == nullptr) { return; }
+   MFEM_VERIFY(nodes0, "internal error");

   MFEM_VERIFY(PA.enabled, "AssemblePA_Limiting but PA is not enabled!");
   MFEM_VERIFY(lim_func, "No TMOP_LimiterFunction specification!")
@@ -68,14 +68,14 @@ void TMOP_Integrator::AssemblePA_Limiting()

   const ElementDofOrdering ordering = ElementDofOrdering::LEXICOGRAPHIC;

-   // H0 for lim_coeff, (dim x dim) Q-vector
+   // H0 for coeff0, (dim x dim) Q-vector
   PA.H0.UseDevice(true);
   PA.H0.SetSize(PA.dim * PA.dim * PA.nq * NE, mt);

-   // lim_coeff -> PA.C0 (Q-vector)
+   // coeff0 -> PA.C0 (Q-vector)
   PA.C0.UseDevice(true);
   if (ConstantCoefficient* cQ =
-          dynamic_cast<ConstantCoefficient*>(lim_coeff))
+          dynamic_cast<ConstantCoefficient*>(coeff0))
   {
      PA.C0.SetSize(1, Device::GetMemoryType());
      PA.C0.HostWrite();
@@ -90,17 +90,17 @@ void TMOP_Integrator::AssemblePA_Limiting()
         ElementTransformation& T = *fes->GetElementTransformation(e);
         for (int q = 0; q < ir.GetNPoints(); ++q)
         {
-            C0(q,e) = lim_coeff->Eval(T, ir.IntPoint(q));
+            C0(q,e) = coeff0->Eval(T, ir.IntPoint(q));
         }
      }
   }

-   // lim_nodes0 -> PA.X0 (E-vector)
-   MFEM_VERIFY(lim_nodes0->FESpace() == fes, "");
+   // nodes0 -> PA.X0 (E-vector)
+   MFEM_VERIFY(nodes0->FESpace() == fes, "");
   const Operator *n0_R = fes->GetElementRestriction(ordering);
   PA.X0.SetSize(n0_R->Height(), Device::GetMemoryType());
   PA.X0.UseDevice(true);
-   n0_R->Mult(*lim_nodes0, PA.X0);
+   n0_R->Mult(*nodes0, PA.X0);

   // Limiting distances: lim_dist -> PA.LD (E-vector)
   // TODO: remove the hack for the case lim_dist == NULL.
@@ -217,8 +217,8 @@ void TMOP_Integrator::AssemblePA(const FiniteElementSpace &fes)
   PA.Jtr_needs_update = true;
   PA.Jtr_debug_grad = false;

-   // Limiting: lim_coeff -> PA.C0, lim_nodes0 -> PA.X0, lim_dist -> PA.LD, PA.H0
-   if (lim_coeff) { AssemblePA_Limiting(); }
+   // Limiting: coeff0 -> PA.C0, nodes0 -> PA.X0, lim_dist -> PA.LD, PA.H0
+   if (coeff0) { AssemblePA_Limiting(); }
 }

 void TMOP_Integrator::AssembleGradDiagonalPA(Vector &de) const
@@ -236,13 +236,13 @@ void TMOP_Integrator::AssembleGradDiagonalPA(Vector &de) const
   if (PA.dim == 2)
   {
      AssembleDiagonalPA_2D(de);
-      if (lim_coeff) { AssembleDiagonalPA_C0_2D(de); }
+      if (coeff0) { AssembleDiagonalPA_C0_2D(de); }
   }

   if (PA.dim == 3)
   {
      AssembleDiagonalPA_3D(de);
-      if (lim_coeff) { AssembleDiagonalPA_C0_3D(de); }
+      if (coeff0) { AssembleDiagonalPA_C0_3D(de); }
   }
 }

@@ -258,13 +258,13 @@ void TMOP_Integrator::AddMultPA(const Vector &xe, Vector &ye) const
   if (PA.dim == 2)
   {
      AddMultPA_2D(xe,ye);
-      if (lim_coeff) { AddMultPA_C0_2D(xe,ye); }
+      if (coeff0) { AddMultPA_C0_2D(xe,ye); }
   }

   if (PA.dim == 3)
   {
      AddMultPA_3D(xe,ye);
-      if (lim_coeff) { AddMultPA_C0_3D(xe,ye); }
+      if (coeff0) { AddMultPA_C0_3D(xe,ye); }
   }
 }

@@ -283,13 +283,13 @@ void TMOP_Integrator::AddMultGradPA(const Vector &re, Vector &ce) const
   if (PA.dim == 2)
   {
      AddMultGradPA_2D(re,ce);
-      if (lim_coeff) { AddMultGradPA_C0_2D(re,ce); }
+      if (coeff0) { AddMultGradPA_C0_2D(re,ce); }
   }

   if (PA.dim == 3)
   {
      AddMultGradPA_3D(re,ce);
-      if (lim_coeff) { AddMultGradPA_C0_3D(re,ce); }
+      if (coeff0) { AddMultGradPA_C0_3D(re,ce); }
   }
 }

@@ -307,13 +307,13 @@ double TMOP_Integrator::GetLocalStateEnergyPA(const Vector &xe) const
   if (PA.dim == 2)
   {
      energy = GetLocalStateEnergyPA_2D(xe);
-      if (lim_coeff) { energy += GetLocalStateEnergyPA_C0_2D(xe); }
+      if (coeff0) { energy += GetLocalStateEnergyPA_C0_2D(xe); }
   }

   if (PA.dim == 3)
   {
      energy = GetLocalStateEnergyPA_3D(xe);
-      if (lim_coeff) { energy += GetLocalStateEnergyPA_C0_3D(xe); }
+      if (coeff0) { energy += GetLocalStateEnergyPA_C0_3D(xe); }
   }

   return energy;
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Stowell, Mark L	872822da4f	Merge remote-tracking branch 'origin/master' into entity-sets-dev	2021-12-07 19:11:39 -08:00
Stowell, Mark L	fd5505407d	Merge remote-tracking branch 'origin/master' into entity-sets-dev # Conflicts: # mesh/pncmesh.hpp	2021-12-07 15:01:43 -08:00
Stowell, Mark L	ccf27151f5	Remove a warning	2021-07-06 17:12:03 -07:00
Stowell, Mark L	ae9f953b20	Merge remote-tracking branch 'origin/master' into entity-sets-dev # Conflicts: # fem/fespace.cpp # fem/fespace.hpp # general/sets.hpp # general/stable3d.hpp # mesh/mesh.cpp # mesh/mesh_readers.cpp # mesh/ncmesh.cpp # mesh/ncmesh.hpp # mesh/pmesh.cpp # mesh/pncmesh.cpp	2021-07-06 17:11:43 -07:00
Stowell, Mark L	7667f4c3f1	make style	2020-02-20 11:20:03 -08:00
Stowell, Mark L	092bf6c883	Using the new triangle face support	2020-02-20 11:18:15 -08:00
Stowell, Mark L	fe921e8c83	Merge remote-tracking branch 'origin/master' into entity-sets-dev # Conflicts: # mesh/mesh.cpp # mesh/ncmesh.cpp # mesh/ncmesh.hpp # mesh/pncmesh.cpp	2020-02-20 11:02:35 -08:00
Stowell, Mark L	1dbdfe33e0	Merge remote-tracking branch 'origin/master' into entity-sets-dev	2019-05-01 10:07:25 -07:00
Stowell, Mark L	f23d0a333d	Merge remote-tracking branch 'origin/master' into entity-sets-dev	2019-04-12 14:46:26 -07:00
Stowell, Mark L	a7ceecdcca	Merge remote-tracking branch 'origin/master' into entity-sets-dev	2019-04-09 14:46:06 -07:00
Stowell, Mark L	5ed4c7b407	Merge remote-tracking branch 'origin/master' into entity-sets-dev # Conflicts: # mesh/ncmesh.cpp	2019-04-01 11:07:50 -07:00
Stowell, Mark L	99ae185c6a	Adding user defined AMR limit	2019-03-17 17:46:56 -07:00
Stowell, Mark L	d29134d0e7	Adding checks for non-null pointers before accessing same	2019-03-17 17:46:13 -07:00
Stowell, Mark L	180ee2a9c6	Adding AMR test code	2019-03-17 10:45:22 -07:00
Stowell, Mark L	207efd476d	Merge remote-tracking branch 'origin/master' into entity-sets-dev	2019-01-20 00:02:39 -08:00
Stowell, Mark L	88f5ec5fec	Merge remote-tracking branch 'origin/master' into entity-sets-dev	2018-12-23 17:26:43 -08:00
Stowell, Mark L	0ee86ac277	Merge remote-tracking branch 'origin/master' into entity-sets-dev	2018-11-26 14:59:10 -08:00
Stowell, Mark L	4268ec6a55	Post merge bugfix	2018-11-12 21:08:45 -08:00
Stowell, Mark L	dfa9340302	Test meshes for mixed meshes with entity sets	2018-11-11 14:20:08 -08:00
Stowell, Mark L	c2004e4eb4	Merge remote-tracking branch 'origin/master' into entity-sets-dev	2018-11-08 17:59:05 -08:00
Stowell, Mark L	2afa90dd44	make style	2018-11-08 09:21:36 -08:00
Stowell, Mark L	dbb0d57f10	Starting mixed mesh refinement in entity sets	2018-11-08 09:20:44 -08:00
Stowell, Mark L	13f205f111	Merge remote-tracking branch 'origin/master' into entity-sets-dev # Conflicts: # mesh/pmesh.cpp # mesh/pncmesh.hpp	2018-11-07 22:59:17 -08:00
Stowell, Mark L	7f0c88bfaa	Merge remote-tracking branch 'origin/master' into entity-sets-dev # Conflicts: # mesh/mesh.cpp	2018-10-10 21:56:22 -07:00
Stowell, Mark L	b2a89fefbc	no message	2018-09-25 13:23:08 -07:00
Stowell, Mark L	3ac0168600	Adding "Tier" to STable3D::Print output	2018-09-08 17:34:20 -07:00
Stowell, Mark L	68db9e6ea9	Cleaning up merge conflicts	2018-09-08 17:22:45 -07:00
Stowell, Mark L	3d6227f2cf	Merge remote-tracking branch 'origin/master' into entity-sets-dev # Conflicts: # mesh/mesh.hpp # mesh/mesh_readers.cpp # mesh/ncmesh.cpp # mesh/ncmesh.hpp # mesh/pncmesh.cpp # mesh/pncmesh.hpp	2018-09-08 15:09:02 -07:00
Stowell, Mark L	67bfaa60d4	file format change	2017-08-22 19:58:53 -07:00
Stowell, Mark L	c434761551	Style and debugging changes	2017-08-22 19:58:23 -07:00
Stowell, Mark L	710f6bd8e6	Adding two example codes for testing	2017-08-14 15:54:58 -07:00
Stowell, Mark L	9175575dcb	Added CollectFaceVertices to entity set closure method.	2017-08-14 10:23:46 -07:00
Stowell, Mark L	3de54b7d06	Merge remote-tracking branch 'origin/master' into entity-sets-dev # Conflicts: # fem/pfespace.cpp	2017-08-11 14:47:52 -07:00
Stowell, Mark L	1a721e699a	Closure seems to work on 2D meshes though I suspect it is still incomplete	2017-07-21 16:02:58 -07:00
Stowell, Mark L	607cf3c355	Run through "make style"	2017-07-21 09:12:32 -07:00
Stowell, Mark L	b32c0d9430	Nearly working. There is still an issue related to missing dofs from neighboring processors.	2017-07-20 16:03:52 -07:00
Stowell, Mark L	dd6f843634	This commit is a mess but I need to move the code to another platform for debugging	2017-07-17 15:04:21 -07:00
Stowell, Mark L	911fb9925d	Changed internal data structure to std::set<int> rather from std::vector<int>	2017-07-08 22:27:17 -07:00
Stowell, Mark L	58f39f31fe	Small updates to serial test driver.	2017-07-07 15:40:19 -07:00
Stowell, Mark L	40da819bc6	Adding an NCEntitySet class to store and manage NCMesh specific data.	2017-07-07 15:39:51 -07:00
Stowell, Mark L	802c6f11f1	Moving NCMesh specific data related to Entity Sets into a separate class to keep NCMesh clean.	2017-07-07 15:39:13 -07:00
Stowell, Mark L	08369c3086	Test programs for validating EntitySet behavior	2017-06-29 15:25:39 -07:00
Stowell, Mark L	4002955677	Preliminary support for Entity Sets in NCMesh class. This contains a lot of debugging output which will be removed once the parallel implementation is finished.	2017-06-29 15:25:03 -07:00
Stowell, Mark L	4283d54ad9	Fixing Mesh::Swap so that it is aware of the EntitySet member data in the Mesh class	2017-06-29 15:23:32 -07:00
Stowell, Mark L	c6e3427808	Storing the coarse set information for use with the NCMesh class	2017-06-29 15:22:02 -07:00
Stowell, Mark L	7a927ae3b6	Merge remote-tracking branch 'origin/master' into entity-sets-dev	2017-06-22 15:13:12 -07:00
Tzanio	aa334eb386	Sets of mesh entities feature developed internally at LLNL. This pull request introduces two new classes; EntitySet and ParEntitySet. These are intended to provide additional flexibility to the way MFEM tags groups of mesh entities. These groups can be used for applying sources, boundary conditions, post processing, etc.. MFEM currently assigns an integer attribute to each element or boundary element. These attributes are then used by looping over all elements and searching for attributes of interest. This is fine for large groups of elements but for small groups this involves a large loop to locate a small number of things. EntitySets provide a way to loop over small groups of entities more efficiently. They also generalize to groups of vertices, edges, faces, or elements which can be very convenient. The current implementation works for sets defined on quadrilateral or hexahedral meshes in serial and parallel, with or without uniform refinement. Triangle and tetrahedral meshes have not been tested. Support for non-conforming meshes has not been started. If I can convince others of the value of these entity sets then we can add support for these other mesh types. Based on the following commits by Mark Stowell: - Adding element access operators. - Declaring "Size()" methods as "const". - Added a comment for clarification. - First draft of EntitySets class. - Run through astyle. - Adding methods to lookup VDofs and TrueDofs by set information. - Adding a method to compute a Vertex to Edge table. - Adding a non-const operator[] method. - Adding a Load method. - Testing the EntitySets::Load method. - Mesh::NumOfEdges is not set even when the edge_vertex Table has been created. - Adding Vertex to Face table. - Testing face sets. - Switched to using the STable3D face_tbl rather than creating a vertex to face table. - Switched to using the Vertex to Vertex table. - Adding support for QuadUniformRefinement with entity sets. - Added copy constructor - Adding support for entity sets in ParFiniteElementSpace - Initial changes to support entity sets in parallel. - Testing QuadUniformRefinement in parallel - Adding test codes for entity sets. - Adding parallel version of entity set code. - Adding test meshes - Improveing the way edge_vertex is protecgted in the mesh class. - Adding support for parallel uniform refinement of quad meshes. - Adding a method to return the number of rows in a table. - Adding a RowIterator like the one in DSTable. - Adding a fourth integer to the STable3D so that we don't lose the fourth value supplied to Push4. - Changed the behavior when an entry is not found to match the behavior of DSTable and Table which return -1. Previously this object would issue an abort upon failure. - Building faces array if face sets are read from disk. - Adding a face_vertex table similar to the edge_vertex table. - Adding support for face sets. This does not quite work yet but I don't want to lose these changes. - Adding face sets for testing. - Added a Print method to STable3D for debugging purposes. - Fixed face sets in parallel with uniform refinement. - Adding options to select the traditional boundary condition or one based on entity sets. - Cleaned up the mesh file by adding more interesting sets rather than test sets. - Adding code to display information about the entity sets.	2017-05-02 17:33:45 -07:00