Merge branch 'vector-remove-implicit-conversion' into metis-and-implicit-conv

Fix nvcc warnings when HiOp integration is enabled

.gitignore

@@ -275,6 +275,7 @@ miniapps/tools/load-dc
 miniapps/tools/convert-dc
 miniapps/tools/lor-transfer
 miniapps/tools/get-values
+miniapps/tools/check-tmop-metric
 
 miniapps/toys/automata
 miniapps/toys/life

CHANGELOG

@@ -8,35 +8,52 @@
                                https://mfem.org
 
 
-Version 4.4.1 (development)
+Version 4.5.1 (development)
 ===========================
 
+
+Version 4.5, released on October 22, 2022
+=========================================
+
 Meshing improvements
 --------------------
-- Added support for mixed meshes and pyramids in GSLIB-FindPoints.
-
 - Added new SubMesh and ParSubMesh classes that can be used to extract a subset
   of a given Mesh. These classes have the same functionality as Mesh and ParMesh
   and work with all existing MFEM interfaces like finite element spaces etc.
 
+- Added a method, ParMesh::GetSerialMesh(), that reconstructs a partitioned
+  parallel mesh on a given single rank. Also, added ParMesh::PrintAsSerial(),
+  which saves the reconstructed serial mesh to a C++ stream on rank 0.
+
+- Added more 3D TMOP metrics, as well as specialized metrics for mesh
+  untangling and worst-case quality improvement.
+
+- Added a new method, Mesh::NodesUpdated, which should be called after the mesh
+  node coordinates have changed, e.g. after the mesh has moved. This is
+  necessary, for example, with device assembly of linear and bilinear forms.
+
+- Added support for mixed meshes and pyramids in GSLIB-FindPoints.
+
 Discretization improvements
 ---------------------------
-- Added support for assembling low-order-refined matrices using a GPU-enabled
-  "batched" algorithm. The lor_solvers and plor_solvers now fully support GPU
-  acceleration.
-
-- Added support for partial assembly and fully matrix-free operators on mixed
-  meshes (different element types and p-adaptivity) through libCEED, including
-  device acceleration, e.g. with NVIDIA and AMD GPUs. The p-adaptivity is
-  currently limited by MFEM capabilities, i.e. 2D serial meshes. All mixed
-  element topologies are supported in serial and parallel: segment, triangle,
-  square, tetrahedron, cube, prism, and pyramid.
-
 - Added full assembly and device support for several LinearForm integrators:
   * DomainLF: (f, v)
   * VectorDomainLF: ((f1,...,fn), (v1,...,vn))
   * DomainLFGrad: (f, grad(v))
   * VectorDomainLFGrad: ((f1x,f1y,f1z,...,fnx,fny,fnz), grad(v1,...,vn))
+  The device assembly of linear forms has to be explicitly enabled by calling
+  LinearForm::UseFastAssembly(true), otherwise the legacy linear form assembly
+  is used by default.
+
+- Added support for assembling low-order-refined matrices using a GPU-enabled
+  "batched" algorithm. The lor_solvers and plor_solvers now fully support GPU
+  acceleration with arbitrary user-supplied coefficients.
+
+- Added a new class FaceQuadratureSpace that allows for the construction of
+  QuadratureFunctions on the interior or boundary faces of a mesh.
+
+- Added a class CoefficientVector for efficient access of variable coefficient
+  values at quadrature points (in particular for GPU/device kernels).
 
 - Added WhiteGaussianNoiseDomainLFIntegrator: a LinearFormIntegrator class for
   spatial Gaussian white noise.
@@ -44,13 +61,26 @@ Discretization improvements
 - Added a new Zienkiewicz-Zhu patch recovery-based a posteriori error estimator.
   See fem/estimators.hpp.
 
+- Various fixes and improvements in LinearFormExtension.
+
 Linear and nonlinear solvers
 ----------------------------
-
-- Added a new class DGMassInverse that performs a local elementwise CG
+- Added a new class DGMassInverse that performs a local element-wise CG
   iteration to solve systems involving the discontinuous Galerkin mass matrix,
   including support for device/GPU acceleration.
 
+- Added more flexibility to the constrained solver classes:
+  * PenaltyConstrainedSolver now allows for a vector of penalty parameters
+    (necessary for penalty contact)
+  * PenaltyConstrainedSolver and EliminationSolver can use GMRES or PCG
+  * All constraint solver classes can take a user-defined preconditioner
+
+ - Added functions to toggle additional options for the SuperLU_Dist and Hypre
+   preconditioners (ParaSails, Euclid, ILU).
+
+- Added boundary elimination with device support for `SparseMatrix` and
+  `HypreParMatrix`.
+
 New and updated examples and miniapps
 -------------------------------------
 - Added a new elasticity miniapp, Hooke, that showcases a low-level approach of
@@ -59,12 +89,28 @@ New and updated examples and miniapps
   automatic differentiation tools like a native dual number implementation or a
   third party library such as Enzyme. See miniapps/elasticity for more details.
 
+- Added example for body-fitted volumetric and shape integration using the
+  Algoim library in miniapps/shifted.
+
 - Add a new example code, Example 33/33p, to demonstrate the solution of
   spectral fractional PDEs with MFEM.
 
 Integrations, testing and documentation
 ---------------------------------------
 - Added a Dockerfile for a simple MFEM container, see config/docker/README.md.
+  More sophisticated developer containers are available in the new repo
+  https://github.com/mfem/containers.
+
+- Added support for the LLVM-based automatic differentiation tool Enzyme, see
+  https://github.com/EnzymeAD/Enzyme. Build system flags and a convenience
+  header are provided. The functionality and interaction are demonstrated in a
+  new miniapp in miniapps/elasticity.
+
+- Added support for partial assembly and fully matrix-free operators on mixed
+  meshes (different element types and p-adaptivity) through libCEED, including
+  device acceleration, e.g. with NVIDIA and AMD GPUs. The p-adaptivity is
+  currently limited to 2D serial meshes. All mixed element topologies are
+  supported in both serial and parallel.
 
 - Added support for ParMoonolith, https://bitbucket.org/zulianp/par_moonolith,
   which provides parallel non-conforming, non-matching, variational, volumetric
@@ -72,28 +118,39 @@ Integrations, testing and documentation
   between arbitrarily distributed and unrelated finite element meshes in a
   variationally consistent way.
 
-- Added support for the LLVM-based automatic differentiation tool Enzyme, see
-  https://github.com/EnzymeAD/Enzyme. Build system flags and a convenience
-  header are provided. The functionality and interaction are demonstrated in a
-  new miniapp in miniapps/elasticity.
+- Fully encapsulated SUNDIALS `N_Vector` object within the `SundialsNVector`
+  class by removing deprecated (e.g. `HypreParVector::ToNVector`) and
+  non-deprecated (e.g. `Vector::ToNVector`) functions in other classes.
 
-- Added example for body-fitted volumetric and shape integration using the
-  Algoim library.
+- New benchmark for the different assembly levels inspired by the CEED
+  Bake-Off Problems, see tests/benchmarks/bench_assembly_levels.cpp.
 
 - Added Windows 2022 CI testing with GitHub actions.
 
 Miscellaneous
 -------------
-- Various other simplifications, extensions, and bugfixes in the code.
+- The method SparseMatrix::EnsureMultTranspose() is now automatically called
+  by the methods AddMultTranspose(), MultTranspose(), and AbsMultTranspose().
+  Added a method with the same name to class HypreParMatrix which is also called
+  automatically by the HypreParMatrix::MultTranspose() methods.
 
-- Added boundary elimination with device support for `SparseMatrix` and
-  `HypreParMatrix`.
+- Updated various MemoryUsage methods to return 'std::size_t' instead of 'long'
+  since the latter is 32-bit in Win64 builds.
 
 - When using `AssemblyLevel::FULL`, `FABilinearFormExtension::FormSystemMatrix`
   outputs an `OperatorHandle` containing a `SparseMatrix` in serial, and an
   `HypreParMatrix` in parallel (instead of a `ConstrainedOperator`).
 
-- Added TMOP metrics for mesh untangling and worst-case quality improvement.
+- In various places in the library, replace the use of 'long' with 'long long'
+  to better support Win64 builds where 'long' is 32-bit and 'long long' is
+  64-bit. On Linux and MacOS, both types are typically 64-bit.
+
+- The behavior of GridFunction::GetTrueVector() has been changed to not return
+  an empty true vector.
+
+- Added support for ordering search points byVDIM in FindPointsGSLIB.
+
+- Various other simplifications, extensions, and bugfixes in the code.
 
 
 Version 4.4, released on March 21, 2022
@@ -127,11 +184,6 @@ Meshing improvements
 - Added a simpler interface to access mesh face information, see FaceInformation
   and GetFaceInformation in the Mesh class.
 
-- Added the method ParMesh::GetSerialMesh() that reconstructs a partitioned
-  parallel mesh on a given single rank. Also, added the method
-  ParMesh::PrintAsSerial() that saves the reconstructed serial mesh to a C++
-  stream on rank 0.
-
 - Gmsh meshes where all elements have zero physical tag (the default Gmsh output
   format if no physical groups are defined) are now successfully loaded, and
   elements are reassigned attribute number 1.
@@ -221,9 +273,6 @@ Integrations, testing and documentation
 - Switched from Artistic Style (astyle) version 2.05.1 to version 3.1 for code
   formatting. See the "make style" target.
 
-- New benchmark for the different assembly levels inspired by the CEED
-  Bake-Off Problems, see tests/benchmarks/bench_assembly_levels.cpp.
-
 Miscellaneous
 -------------
 - Added a simple singleton class, Mpi, as a replacement for MPI_Session. New
@@ -234,13 +283,6 @@ Miscellaneous
 
 - Fixed several MinGW build issues on Windows.
 
-- In various places in the library, replace the use of 'long' with 'long long'
-  to better support Win64 builds where 'long' is 32-bit and 'long long' is
-  64-bit. On Linux and MacOS, both types are typically 64-bit.
-
-- Update various "MemoryUsage" methods to return 'std::size_t' instead of 'long'
-  since the latter is 32-bit in Win64 builds.
-
 - Added 'double' atomicAdd implementation for previous versions of CUDA.
 
 - HypreParVector and Vector now support C++ move semantics, and the copy

CMakeLists.txt

@@ -51,7 +51,7 @@ project(mfem NONE)
 # Current version of MFEM, see also `makefile`.
 #   mfem_VERSION = (string)
 #   MFEM_VERSION = (int)   [automatically derived from mfem_VERSION]
-set(${PROJECT_NAME}_VERSION 4.4.1)
+set(${PROJECT_NAME}_VERSION 4.5.1)
 
 # Prohibit in-source build
 if (${PROJECT_SOURCE_DIR} STREQUAL ${PROJECT_BINARY_DIR})
@@ -81,6 +81,10 @@ if (MFEM_USE_STRUMPACK)
   # Just needed to find the MPI_Fortran libraries to link with
   set(XSDK_ENABLE_Fortran ON)
 endif()
+# SUNDIALS >= 6.4.0 requires C++14:
+if (MFEM_USE_SUNDIALS AND ("${CMAKE_CXX_STANDARD}" LESS "14"))
+  set(CMAKE_CXX_STANDARD 14)
+endif()
 if (MFEM_USE_GINKGO AND ("${CMAKE_CXX_STANDARD}" LESS "14"))
   set(CMAKE_CXX_STANDARD 14)
 endif()
@@ -137,7 +141,7 @@ if (MFEM_USE_CUDA)
    set(CUSPARSE_FOUND TRUE)
    set(CUSPARSE_LIBRARIES "cusparse")
    set(CUBLAS_FOUND TRUE)
-   set(CUSBLAS_LIBRARIES "cublas")
+   set(CUBLAS_LIBRARIES "cublas")
 endif()
 
 if (XSDK_ENABLE_C)
@@ -200,10 +204,10 @@ if (MFEM_USE_MPI)
   find_package(MPI REQUIRED)
   set(MPI_CXX_INCLUDE_DIRS ${MPI_CXX_INCLUDE_PATH})
   if (MFEM_MPIEXEC)
-    set(MPIEXEC ${MFEM_MPIEXEC})
+    string(REPLACE " " ";" MPIEXEC ${MFEM_MPIEXEC})
   endif()
   if (MFEM_MPIEXEC_NP)
-    set(MPIEXEC_NUMPROC_FLAG ${MFEM_MPIEXEC_NP})
+    string(REPLACE " " ";" MPIEXEC_NUMPROC_FLAG ${MFEM_MPIEXEC_NP})
   endif()
   # Parallel MFEM depends on hypre
   find_package(HYPRE REQUIRED)
@@ -477,6 +481,15 @@ if (NOT DEFINED MFEM_TIMER_TYPE)
   endif()
 endif()
 
+# Without this, CMake 3.21.1 (and 3.20.2) run into CMake Errors like the following:
+# CMake Error at config/cmake/modules/MfemCmakeUtilities.cmake:60 (add_library):
+#   Target "mfem" links to target "Threads::Threads" but the target was not
+#   found.  Perhaps a find_package() call is missing for an IMPORTED target, or
+#   an ALIAS target is missing?
+# Call Stack (most recent call first):
+#   CMakeLists.txt:474 (mfem_add_library)
+find_package(Threads REQUIRED)
+
 # List all possible libraries in order of dependencies.
 # [METIS < SuiteSparse]:
 #    With newer versions of SuiteSparse which include METIS header using 64-bit

INSTALL

@@ -7,6 +7,10 @@
 
                                https://mfem.org
 
+This file provides a detailed description of how to build and install the MFEM
+library. For a simple build, see the step-by-step instructions on the website
+at https://mfem.org/building.
+
 The MFEM library has a serial and an MPI-based parallel version, which largely
 share the same code base. The only prerequisite for building the serial version
 of MFEM is a (modern) C++ compiler, such as g++. The parallel version of MFEM
@@ -19,7 +23,7 @@ requires an MPI C++ compiler, as well as the following external libraries:
   https://github.com/mfem/tpls
 
   Note: We recommend our mirror of metis-4.0.3/5.1.0 above because the METIS
-  webpage, https://glaros.dtc.umn.edu/gkhome/metis/metis/overview, is often down
+  webpage, http://glaros.dtc.umn.edu/gkhome/metis/metis/overview, is often down
   and we don't support yet the new repo https://github.com/KarypisLab/METIS.
 
 The hypre dependency can be downloaded as a tarball from GitHub or from the
@@ -758,12 +762,12 @@ The specific libraries and their options are:
   Options: GSLIB_OPT, GSLIB_LIB.
   Versions: GSLIB >= 1.0.7.
 
-- ALGOIM (optional), used when MFE_USE_ALGOIM=YES. The library provides only
+- ALGOIM (optional), used when MFEM_USE_ALGOIM=YES. The library provides only
   headers so it just needs to be downloaded at the same level as MFEM. Download
   the specific version we use as:
   "git clone https://github.com/algoim/algoim.git;
    git checkout 9c9ca0ef094d8ab0390ed36367a1151b459bbe0a"
-  ALGOIM depends on BLITZ and rhe library must be built prior to the MFEM build.
+  ALGOIM depends on BLITZ and the library must be built prior to the MFEM build.
   Download v1.0.2, untar it at the same level as MFEM and create a symbolic link:
   "ln -s blitz-1.0.2 blitz".
   Build Blitz using CMake as:

config/cmake/modules/FindHDF5.cmake

@@ -0,0 +1,57 @@
+# Copyright (c) 2010-2022, Lawrence Livermore National Security, LLC. Produced
+# at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+# LICENSE and NOTICE for details. LLNL-CODE-806117.
+#
+# This file is part of the MFEM library. For more information and source code
+# availability visit https://mfem.org.
+#
+# MFEM is free software; you can redistribute it and/or modify it under the
+# terms of the BSD-3 license. We welcome feedback and contributions, see file
+# CONTRIBUTING.md for details.
+
+# Defines the following variables:
+#   - HDF5_FOUND - If HDF5 was found
+#   - HDF5_LIBRARIES - The HDF5 libraries
+#   - HDF5_INCLUDE_DIRS - The HDF5 include directories
+
+# First Check for HDF5_DIR
+if(NOT HDF5_DIR)
+    MESSAGE(FATAL_ERROR "Could not find HDF5. HDF5 support needs explicit HDF5_DIR")
+endif()
+
+# Find includes
+find_path( HDF5_INCLUDE_DIRS hdf5.h
+           PATHS  ${HDF5_DIR}/include/
+           NO_DEFAULT_PATH
+           NO_CMAKE_ENVIRONMENT_PATH
+           NO_CMAKE_PATH
+           NO_SYSTEM_ENVIRONMENT_PATH
+           NO_CMAKE_SYSTEM_PATH)
+
+find_library( __HDF5_LIBRARY NAMES hdf5 libhdf5 libhdf5_D libhdf5_debug
+              PATHS ${HDF5_DIR}/lib
+              NO_DEFAULT_PATH
+              NO_CMAKE_ENVIRONMENT_PATH
+              NO_CMAKE_PATH
+              NO_SYSTEM_ENVIRONMENT_PATH
+              NO_CMAKE_SYSTEM_PATH)
+
+find_library( __HDF5_HL_LIBRARY NAMES hdf5_hl libhdf5_hl libhdf5_hl_D libhdf5_hl_debug
+              PATHS ${HDF5_DIR}/lib
+              NO_DEFAULT_PATH
+              NO_CMAKE_ENVIRONMENT_PATH
+              NO_CMAKE_PATH
+              NO_SYSTEM_ENVIRONMENT_PATH
+              NO_CMAKE_SYSTEM_PATH)
+
+set(HDF5_LIBRARIES ${__HDF5_HL_LIBRARY} ${__HDF5_LIBRARY})
+
+include(FindPackageHandleStandardArgs)
+
+# Handle the QUIETLY and REQUIRED arguments and set HDF5_FOUND to TRUE if all
+# listed variables are TRUE
+find_package_handle_standard_args(HDF5  DEFAULT_MSG
+                                  HDF5_INCLUDE_DIRS
+                                  __HDF5_LIBRARY
+                                  __HDF5_HL_LIBRARY
+                                  HDF5_LIBRARIES )

config/cmake/modules/FindUMPIRE.cmake

@@ -14,6 +14,6 @@
 #   - UMPIRE_LIBRARIES
 #   - UMPIRE_INCLUDE_DIRS
 
-include(MfemCmakeUtilities)
-mfem_find_package(UMPIRE UMPIRE UMPIRE_DIR "include" "umpire/Umpire.hpp" "lib" "umpire"
-  "Paths to headers required by UMPIRE." "Libraries required by UMPIRE.")
+find_package(umpire REQUIRED CONFIG)
+set(UMPIRE_FOUND ${umpire_FOUND})
+set(UMPIRE_LIBRARIES "umpire")

config/config.hpp

@@ -31,9 +31,11 @@
 
 // Windows specific options
 #ifdef _WIN32
+#ifndef _USE_MATH_DEFINES
 // Macro needed to get defines like M_PI from <cmath>. (Visual Studio C++ only?)
 #define _USE_MATH_DEFINES
 #endif
+#endif
 // On Cygwin the option -std=c++11 prevents the definition of M_PI. Defining
 // the following macro allows us to get M_PI and some needed functions, e.g.
 // posix_memalign(), strdup(), strerror_r().

config/defaults.mk

@@ -251,12 +251,16 @@ POSIX_CLOCKS_LIB = -lrt
 # SUNDIALS library configuration
 # For sundials_nvecmpiplusx and nvecparallel remember to build with MPI_ENABLE=ON
 # and modify cmake variables for hypre for sundials
-SUNDIALS_DIR    = @MFEM_DIR@/../sundials-5.0.0/instdir
-SUNDIALS_OPT    = -I$(SUNDIALS_DIR)/include
-SUNDIALS_LIBDIR = $(wildcard $(SUNDIALS_DIR)/lib*)
-SUNDIALS_LIB    = $(XLINKER)-rpath,$(SUNDIALS_LIBDIR) -L$(SUNDIALS_LIBDIR)\
+SUNDIALS_DIR = @MFEM_DIR@/../sundials-5.0.0/instdir
+# SUNDIALS >= 6.4.0 requires C++14:
+ifeq ($(MFEM_USE_SUNDIALS),YES)
+   BASE_FLAGS = -std=c++14
+endif
+SUNDIALS_OPT = -I$(SUNDIALS_DIR)/include
+SUNDIALS_LIB = $(XLINKER)-rpath,$(SUNDIALS_DIR)/lib64\
+ $(XLINKER)-rpath,$(SUNDIALS_DIR)/lib\
+ -L$(SUNDIALS_DIR)/lib64 -L$(SUNDIALS_DIR)/lib\
  -lsundials_arkode -lsundials_cvodes -lsundials_nvecserial -lsundials_kinsol
-
 ifeq ($(MFEM_USE_MPI),YES)
    SUNDIALS_LIB += -lsundials_nvecparallel -lsundials_nvecmpiplusx
 endif

config/sample-runs.sh

@@ -554,15 +554,14 @@ function go()
    local cmd_line="${1##+( )}"
    cmd_line="${cmd_line%%+( )}"
    shopt -u extglob
-   eval local cmd=(${cmd_line})
    local res=""
    echo $sep
    echo "<${group}>" "${cmd_line}"
    echo $sep
    if [ "${timing}" == "yes" ]; then
-      timed_run "${cmd[@]}"
+      timed_run eval "${cmd_line}"
    else
-      "${cmd[@]}"
+      eval "${cmd_line}"
    fi
    if [ "$?" -eq 0 ]; then
       res="${green}  OK  ${none}"

config/vcpkg/ports/metis-mfem/vcpkg.json

@@ -3,5 +3,5 @@
   "version-string": "5.1.0",
   "port-version": 0,
   "description": "Serial Graph Partitioning and Fill-reducing Matrix Ordering",
-  "homepage": "https://glaros.dtc.umn.edu/gkhome/metis/metis/overview"
+  "homepage": "http://glaros.dtc.umn.edu/gkhome/metis/metis/overview"
 }

doc/CodeDocumentation.conf.in

@@ -38,7 +38,7 @@ PROJECT_NAME           = "MFEM"
 # could be handy for archiving the generated documentation or if some version
 # control system is used.
 
-PROJECT_NUMBER         = v4.4.1
+PROJECT_NUMBER         = v4.5.1
 
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a

examples/ex1.cpp

@@ -30,7 +30,7 @@
 //
 // Device sample runs:
 //               ex1 -pa -d cuda
-//             * ex1 -fa -d cuda
+//               ex1 -fa -d cuda
 //               ex1 -pa -d raja-cuda
 //             * ex1 -pa -d raja-hip
 //               ex1 -pa -d occa-cuda
@@ -192,7 +192,14 @@ int main(int argc, char *argv[])
    //    domain integrator.
    BilinearForm a(&fespace);
    if (pa) { a.SetAssemblyLevel(AssemblyLevel::PARTIAL); }
-   if (fa) { a.SetAssemblyLevel(AssemblyLevel::FULL); }
+   if (fa)
+   {
+      a.SetAssemblyLevel(AssemblyLevel::FULL);
+      // Sort the matrix column indices when running on GPU or with OpenMP (i.e.
+      // when Device::IsEnabled() returns true). This makes the results
+      // bit-for-bit deterministic at the cost of somewhat longer run time.
+      a.EnableSparseMatrixSorting(Device::IsEnabled());
+   }
    a.AddDomainIntegrator(new DiffusionIntegrator(one));
 
    // 10. Assemble the bilinear form and the corresponding linear system,

examples/ex1p.cpp

@@ -30,7 +30,7 @@
 //
 // Device sample runs:
 //               mpirun -np 4 ex1p -pa -d cuda
-//             * mpirun -np 4 ex1p -fa -d cuda
+//               mpirun -np 4 ex1p -fa -d cuda
 //               mpirun -np 4 ex1p -pa -d occa-cuda
 //               mpirun -np 4 ex1p -pa -d raja-omp
 //               mpirun -np 4 ex1p -pa -d ceed-cpu
@@ -219,7 +219,14 @@ int main(int argc, char *argv[])
    //     Diffusion domain integrator.
    ParBilinearForm a(&fespace);
    if (pa) { a.SetAssemblyLevel(AssemblyLevel::PARTIAL); }
-   if (fa) { a.SetAssemblyLevel(AssemblyLevel::FULL); }
+   if (fa)
+   {
+      a.SetAssemblyLevel(AssemblyLevel::FULL);
+      // Sort the matrix column indices when running on GPU or with OpenMP (i.e.
+      // when Device::IsEnabled() returns true). This makes the results
+      // bit-for-bit deterministic at the cost of somewhat longer run time.
+      a.EnableSparseMatrixSorting(Device::IsEnabled());
+   }
    a.AddDomainIntegrator(new DiffusionIntegrator(one));
 
    // 12. Assemble the parallel bilinear form and the corresponding linear

examples/ex5.cpp

@@ -197,7 +197,6 @@ int main(int argc, char *argv[])
       SparseMatrix &M(mVarf->SpMat());
       SparseMatrix &B(bVarf->SpMat());
       B *= -1.;
-      B.EnsureMultTranspose();
       Bt = new TransposeOperator(&B);
 
       darcyOp.SetBlock(0,0, &M);

examples/petsc/ex10p.cpp

@@ -187,7 +187,6 @@ int main(int argc, char *argv[])
 {
    // 1. Initialize MPI and HYPRE.
    Mpi::Init(argc, argv);
-   int num_procs = Mpi::WorldSize();
    int myid = Mpi::WorldRank();
    Hypre::Init();
 

examples/petsc/ex2p.cpp

@@ -248,7 +248,10 @@ int main(int argc, char *argv[])
    //     constraints for non-conforming AMR, static condensation, etc.
    if (myid == 0) { cout << "matrix ... " << flush; }
    if (static_cond) { a->EnableStaticCondensation(); }
-   a->Assemble();
+   // Here we want to try out block-size aware AMG solver in PETSc.
+   // For that to work properly, we need a fully-compliant block-size
+   // structure and we do not skip zeros when assembling.
+   a->Assemble(use_petsc ? 0 : 1);
 
    Vector B, X;
    if (!use_petsc)
@@ -294,13 +297,14 @@ int main(int argc, char *argv[])
          cout << "done." << endl;
          cout << "Size of linear system: " << A.M() << endl;
       }
-      PetscPCGSolver *pcg = new PetscPCGSolver(A);
+      // Tell PETSc the matrix has a block structure
+      A.SetBlockSize(dim);
 
-      // The preconditioner for the PCG solver defined below is specified in the
-      // PETSc config file, rc_ex2p, since a Krylov solver in PETSc can also
-      // customize its preconditioner.
+      // The preconditioner for the PCG solver can be specified in the
+      // PETSc config file
+      PetscPCGSolver *pcg = new PetscPCGSolver(A);
       PetscPreconditioner *prec = NULL;
-      if (use_nonoverlapping)
+      if (use_nonoverlapping) // Specialized BDDC construction
       {
          // Compute dofs belonging to the natural boundary
          Array<int> nat_tdof_list, nat_bdr(pmesh->bdr_attributes.Max());

examples/petsc/makefile

@@ -78,6 +78,7 @@ EX1_ARGS_CUDA         := -m ../../data/star.mesh --usepetsc --partial-assembly -
 EX1_ARGS_CUDAAMG      := -m ../../data/star.mesh --usepetsc --device cuda --petscopts rc_ex1p_cudaamg
 EX2_ARGS              := -m ../../data/beam-quad.mesh --usepetsc --petscopts rc_ex2p
 EX2_ARGS_BDDC         := -m ../../data/beam-tri.mesh --usepetsc --nonoverlapping --petscopts rc_ex2p_bddc
+EX2_ARGS_ASM          := -m ../../data/beam-quad.mesh --usepetsc --petscopts rc_ex2p_asm
 EX3_ARGS              := -m ../../data/klein-bottle.mesh -o 2 -f 0.1 --usepetsc --petscopts rc_ex3p_bddc --nonoverlapping
 EX4_ARGS              := -m ../../data/klein-bottle.mesh -o 2 --usepetsc --petscopts rc_ex4p_bddc --nonoverlapping
 EX4_HYB_ARGS          := -m ../../data/klein-bottle.mesh -o 2 --usepetsc --petscopts rc_ex4p_bddc --nonoverlapping --hybridization
@@ -109,6 +110,7 @@ endif
 ex2p-test-par: ex2p
 	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX2_ARGS))
 	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX2_ARGS_BDDC))
+	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX2_ARGS_ASM))
 ex3p-test-par: ex3p
 	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX3_ARGS))
 ex4p-test-par: ex4p

examples/petsc/rc_ex1p

@@ -14,4 +14,5 @@
 -mg_levels_esteig_ksp_type cg
 -mg_levels_esteig_ksp_max_it 10
 -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
+-pc_gamg_use_sa_esteig 0
 -mg_levels_pc_type sor

examples/petsc/rc_ex2p

@@ -1,8 +1,7 @@
 -ksp_converged_reason
 
 # GAMG is still not used at its best,
-# since we are not exploiting the
-# block size (Ordering::byVDIM) and the RBMs
+# since we are not exploiting the RBMs
 
 -ksp_view
 -pc_type gamg

examples/petsc/rc_ex2p_asm

@@ -0,0 +1,10 @@
+# Additive Schwarz with Overlap
+# This is not a good solver for elasticity
+# These options are here only to describe
+# the setup of the solver
+-ksp_converged_reason
+-ksp_view
+-ksp_max_it 10
+-pc_type asm
+-pc_asm_overlap 1
+-sub_pc_type icc

examples/sundials/ex10.cpp

@@ -210,6 +210,9 @@ void visualize(ostream &os, Mesh *mesh, GridFunction *deformed_nodes,
 
 int main(int argc, char *argv[])
 {
+   // 0. Initialize SUNDIALS.
+   Sundials::Init();
+
    // 1. Parse command-line options.
    const char *mesh_file = "../../data/beam-quad.mesh";
    int ref_levels = 2;

examples/sundials/ex10p.cpp

@@ -215,10 +215,11 @@ void visualize(ostream &os, ParMesh *mesh, ParGridFunction *deformed_nodes,
 
 int main(int argc, char *argv[])
 {
-   // 1. Initialize MPI and HYPRE.
+   // 1. Initialize MPI, HYPRE, and SUNDIALS.
    Mpi::Init(argc, argv);
    int myid = Mpi::WorldRank();
    Hypre::Init();
+   Sundials::Init();
 
    // 2. Parse command-line options.
    const char *mesh_file = "../../data/beam-quad.mesh";

examples/sundials/ex16.cpp

@@ -109,6 +109,9 @@ double InitialTemperature(const Vector &x);
 
 int main(int argc, char *argv[])
 {
+   // 0. Initialize SUNDIALS.
+   Sundials::Init();
+
    // 1. Parse command-line options.
    const char *mesh_file = "../../data/star.mesh";
    int ref_levels = 2;
@@ -290,7 +293,10 @@ int main(int argc, char *argv[])
          arkode->Init(oper);
          arkode->SetSStolerances(reltol, abstol);
          arkode->SetMaxStep(dt);
-         if (ode_solver_type == 11) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
+         if (ode_solver_type == 11)
+         {
+            arkode->SetERKTableNum(ARKODE_FEHLBERG_13_7_8);
+         }
          ode_solver = arkode; break;
       case 12:
          arkode = new ARKStepSolver(ARKStepSolver::IMPLICIT);

examples/sundials/ex16p.cpp

@@ -101,11 +101,12 @@ double InitialTemperature(const Vector &x);
 
 int main(int argc, char *argv[])
 {
-   // 1. Initialize MPI and HYPRE.
+   // 1. Initialize MPI, HYPRE, and SUNDIALS.
    Mpi::Init(argc, argv);
    int num_procs = Mpi::WorldSize();
    int myid = Mpi::WorldRank();
    Hypre::Init();
+   Sundials::Init();
 
    // 2. Parse command-line options.
    const char *mesh_file = "../../data/star.mesh";
@@ -327,7 +328,10 @@ int main(int argc, char *argv[])
          arkode->Init(oper);
          arkode->SetSStolerances(reltol, abstol);
          arkode->SetMaxStep(dt);
-         if (ode_solver_type == 11) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
+         if (ode_solver_type == 11)
+         {
+            arkode->SetERKTableNum(ARKODE_FEHLBERG_13_7_8);
+         }
          ode_solver = arkode; break;
       case 12:
          arkode = new ARKStepSolver(MPI_COMM_WORLD, ARKStepSolver::IMPLICIT);

examples/sundials/ex9.cpp

@@ -140,6 +140,9 @@ public:
 
 int main(int argc, char *argv[])
 {
+   // 0. Initialize SUNDIALS.
+   Sundials::Init();
+
    // 1. Parse command-line options.
    problem = 0;
    const char *mesh_file = "../../data/periodic-hexagon.mesh";
@@ -408,7 +411,7 @@ int main(int argc, char *argv[])
          arkode->Init(adv);
          arkode->SetSStolerances(reltol, abstol);
          arkode->SetMaxStep(dt);
-         arkode->SetERKTableNum(FEHLBERG_13_7_8);
+         arkode->SetERKTableNum(ARKODE_FEHLBERG_13_7_8);
          ode_solver = arkode; break;
    }
 

examples/sundials/ex9p.cpp

@@ -152,11 +152,12 @@ public:
 
 int main(int argc, char *argv[])
 {
-   // 1. Initialize MPI and HYPRE.
+   // 1. Initialize MPI, HYPRE, and SUNDIALS.
    Mpi::Init(argc, argv);
    int num_procs = Mpi::WorldSize();
    int myid = Mpi::WorldRank();
    Hypre::Init();
+   Sundials::Init();
 
    // 2. Parse command-line options.
    problem = 0;
@@ -487,7 +488,10 @@ int main(int argc, char *argv[])
          arkode->Init(adv);
          arkode->SetSStolerances(reltol, abstol);
          arkode->SetMaxStep(dt);
-         if (ode_solver_type == 9) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
+         if (ode_solver_type == 9)
+         {
+            arkode->SetERKTableNum(ARKODE_FEHLBERG_13_7_8);
+         }
          ode_solver = arkode; break;
    }
 

examples/superlu/CMakeLists.txt

@@ -35,7 +35,7 @@ add_mfem_examples(SUPERLU_EXAMPLES_SRCS ${PFX} "" test_superlu)
 if (MFEM_ENABLE_TESTING)
   # Command line options for the tests.
   # Example 1: Test SuperLU on the simple Poisson problem
-  set(EX1_COMMON_OPTS -m ../../data/star.mesh -p 2)
+  set(EX1_COMMON_OPTS -m ../../data/star.mesh)
   set(EX1P_TEST_OPTS ${EX1_COMMON_OPTS})
 
   # Add the tests: one test per source file.

fem/bilinearform.cpp

@@ -124,6 +124,7 @@ void BilinearForm::SetAssemblyLevel(AssemblyLevel assembly_level)
       case AssemblyLevel::LEGACY:
          break;
       case AssemblyLevel::FULL:
+         SetDiagonalPolicy( DIAG_ONE ); // Only diagonal policy supported on device
          ext = new FABilinearFormExtension(this);
          break;
       case AssemblyLevel::ELEMENT:
@@ -136,7 +137,7 @@ void BilinearForm::SetAssemblyLevel(AssemblyLevel assembly_level)
          ext = new MFBilinearFormExtension(this);
          break;
       default:
-         mfem_error("Unknown assembly level");
+         MFEM_ABORT("BilinearForm: unknown assembly level");
    }
 }
 

fem/bilinearform.hpp

@@ -26,7 +26,8 @@ namespace mfem
 {
 
 /** @brief Enumeration defining the assembly level for bilinear and nonlinear
-    form classes derived from Operator. */
+    form classes derived from Operator. For more details, see
+    https://mfem.org/howto/assembly_levels */
 enum class AssemblyLevel
 {
    /// In the case of a BilinearForm LEGACY corresponds to a fully assembled
@@ -79,6 +80,9 @@ protected:
    /** @brief Extension for supporting Full Assembly (FA), Element Assembly (EA),
        Partial Assembly (PA), or Matrix Free assembly (MF). */
    BilinearFormExtension *ext;
+   /** Indicates if the sparse matrix is sorted after assembly when using
+       Full Assembly (FA). */
+   bool sort_sparse_matrix = false;
 
    /** @brief Indicates the Mesh::sequence corresponding to the current state of
        the BilinearForm. */
@@ -177,9 +181,24 @@ public:
        - AssemblyLevel::ELEMENT
        - AssemblyLevel::NONE
 
-       This method must be called before assembly. */
+       If used, this method must be called before assembly. */
    void SetAssemblyLevel(AssemblyLevel assembly_level);
 
+   /** @brief Force the sparse matrix column indices to be sorted when using
+       AssemblyLevel::FULL.
+
+       When assembling on device the assembly algorithm uses atomic operations
+       to insert values in the sparse matrix, which can result in different
+       column index orderings across runs. Calling this method with @a enable_it
+       set to @a true forces a sorting algorithm to be called at the end of the
+       assembly procedure to ensure sorted column indices (and therefore
+       deterministic results).
+   */
+   void EnableSparseMatrixSorting(bool enable_it)
+   {
+      sort_sparse_matrix = enable_it;
+   }
+
    /// Returns the assembly level
    AssemblyLevel GetAssemblyLevel() const { return assembly; }
 

fem/bilinearform_ext.cpp

@@ -160,7 +160,7 @@ void MFBilinearFormExtension::Mult(const Vector &x, Vector &y) const
          {
             intFaceIntegrators[i]->AddMultMF(int_face_X, int_face_Y);
          }
-         int_face_restrict_lex->AddMultTranspose(int_face_Y, y);
+         int_face_restrict_lex->AddMultTransposeInPlace(int_face_Y, y);
       }
    }
 
@@ -176,7 +176,7 @@ void MFBilinearFormExtension::Mult(const Vector &x, Vector &y) const
          {
             bdrFaceIntegrators[i]->AddMultMF(bdr_face_X, bdr_face_Y);
          }
-         bdr_face_restrict_lex->AddMultTranspose(bdr_face_Y, y);
+         bdr_face_restrict_lex->AddMultTransposeInPlace(bdr_face_Y, y);
       }
    }
 }
@@ -217,7 +217,7 @@ void MFBilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
          {
             intFaceIntegrators[i]->AddMultTransposeMF(int_face_X, int_face_Y);
          }
-         int_face_restrict_lex->AddMultTranspose(int_face_Y, y);
+         int_face_restrict_lex->AddMultTransposeInPlace(int_face_Y, y);
       }
    }
 
@@ -233,7 +233,7 @@ void MFBilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
          {
             bdrFaceIntegrators[i]->AddMultTransposeMF(bdr_face_X, bdr_face_Y);
          }
-         bdr_face_restrict_lex->AddMultTranspose(bdr_face_Y, y);
+         bdr_face_restrict_lex->AddMultTransposeInPlace(bdr_face_Y, y);
       }
    }
 }
@@ -418,7 +418,7 @@ void PABilinearFormExtension::Mult(const Vector &x, Vector &y) const
          {
             intFaceIntegrators[i]->AddMultPA(int_face_X, int_face_Y);
          }
-         int_face_restrict_lex->AddMultTranspose(int_face_Y, y);
+         int_face_restrict_lex->AddMultTransposeInPlace(int_face_Y, y);
       }
    }
 
@@ -434,7 +434,7 @@ void PABilinearFormExtension::Mult(const Vector &x, Vector &y) const
          {
             bdrFaceIntegrators[i]->AddMultPA(bdr_face_X, bdr_face_Y);
          }
-         bdr_face_restrict_lex->AddMultTranspose(bdr_face_Y, y);
+         bdr_face_restrict_lex->AddMultTransposeInPlace(bdr_face_Y, y);
       }
    }
 }
@@ -475,7 +475,7 @@ void PABilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
          {
             intFaceIntegrators[i]->AddMultTransposePA(int_face_X, int_face_Y);
          }
-         int_face_restrict_lex->AddMultTranspose(int_face_Y, y);
+         int_face_restrict_lex->AddMultTransposeInPlace(int_face_Y, y);
       }
    }
 
@@ -491,7 +491,7 @@ void PABilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
          {
             bdrFaceIntegrators[i]->AddMultTransposePA(bdr_face_X, bdr_face_Y);
          }
-         bdr_face_restrict_lex->AddMultTranspose(bdr_face_Y, y);
+         bdr_face_restrict_lex->AddMultTransposeInPlace(bdr_face_Y, y);
       }
    }
 }
@@ -668,7 +668,7 @@ void EABilinearFormExtension::Mult(const Vector &x, Vector &y) const
             Y(j, 0, f) += res;
          });
          // Apply the Interior Face Restriction transposed
-         int_face_restrict_lex->AddMultTranspose(int_face_Y, y);
+         int_face_restrict_lex->AddMultTransposeInPlace(int_face_Y, y);
       }
    }
 
@@ -699,7 +699,7 @@ void EABilinearFormExtension::Mult(const Vector &x, Vector &y) const
             Y(j, f) += res;
          });
          // Apply the Boundary Face Restriction transposed
-         bdr_face_restrict_lex->AddMultTranspose(bdr_face_Y, y);
+         bdr_face_restrict_lex->AddMultTransposeInPlace(bdr_face_Y, y);
       }
    }
 }
@@ -796,7 +796,7 @@ void EABilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
             Y(j, 0, f) += res;
          });
          // Apply the Interior Face Restriction transposed
-         int_face_restrict_lex->AddMultTranspose(int_face_Y, y);
+         int_face_restrict_lex->AddMultTransposeInPlace(int_face_Y, y);
       }
    }
 
@@ -827,7 +827,7 @@ void EABilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
             Y(j, f) += res;
          });
          // Apply the Boundary Face Restriction transposed
-         bdr_face_restrict_lex->AddMultTranspose(bdr_face_Y, y);
+         bdr_face_restrict_lex->AddMultTransposeInPlace(bdr_face_Y, y);
       }
    }
 }
@@ -955,6 +955,10 @@ void FABilinearFormExtension::Assemble()
       }
       a->mat = mat;
    }
+   if ( a->sort_sparse_matrix )
+   {
+      a->mat->SortColumnIndices();
+   }
 }
 
 

fem/bilininteg.cpp

@@ -2003,6 +2003,83 @@ void CurlCurlIntegrator::AssembleElementMatrix
    }
 }
 
+void CurlCurlIntegrator::AssembleElementMatrix2(const FiniteElement &trial_fe,
+                                                const FiniteElement &test_fe,
+                                                ElementTransformation &Trans,
+                                                DenseMatrix &elmat)
+{
+   int tr_nd = trial_fe.GetDof();
+   int te_nd = test_fe.GetDof();
+   dim = trial_fe.GetDim();
+   int dimc = trial_fe.GetCurlDim();
+   double w;
+
+#ifdef MFEM_THREAD_SAFE
+   Vector D;
+   DenseMatrix curlshape(tr_nd,dimc), curlshape_dFt(tr_nd,dimc), M;
+   DenseMatrix te_curlshape(te_nd,dimc), te_curlshape_dFt(te_nd,dimc);
+#else
+   curlshape.SetSize(tr_nd,dimc);
+   curlshape_dFt.SetSize(tr_nd,dimc);
+   te_curlshape.SetSize(te_nd,dimc);
+   te_curlshape_dFt.SetSize(te_nd,dimc);
+#endif
+   elmat.SetSize(te_nd, tr_nd);
+
+   if (MQ) { M.SetSize(dimc); }
+   if (DQ) { D.SetSize(dimc); }
+
+   const IntegrationRule *ir = IntRule;
+   if (ir == NULL)
+   {
+      int order;
+      if (trial_fe.Space() == FunctionSpace::Pk)
+      {
+         order = test_fe.GetOrder() + trial_fe.GetOrder() - 2;
+      }
+      else
+      {
+         order = test_fe.GetOrder() + trial_fe.GetOrder() + trial_fe.GetDim() - 1;
+      }
+      ir = &IntRules.Get(trial_fe.GetGeomType(), order);
+   }
+
+   elmat = 0.0;
+   for (int i = 0; i < ir->GetNPoints(); i++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(i);
+
+      Trans.SetIntPoint(&ip);
+
+      w = ip.weight * Trans.Weight();
+      trial_fe.CalcPhysCurlShape(Trans, curlshape_dFt);
+      test_fe.CalcPhysCurlShape(Trans, te_curlshape_dFt);
+
+      if (MQ)
+      {
+         MQ->Eval(M, Trans, ip);
+         M *= w;
+         Mult(te_curlshape_dFt, M, te_curlshape);
+         AddMultABt(te_curlshape, curlshape_dFt, elmat);
+      }
+      else if (DQ)
+      {
+         DQ->Eval(D, Trans, ip);
+         D *= w;
+         AddMultADBt(te_curlshape_dFt,D,curlshape_dFt,elmat);
+      }
+      else
+      {
+         if (Q)
+         {
+            w *= Q->Eval(Trans, ip);
+         }
+         curlshape_dFt *= w;
+         AddMultABt(te_curlshape_dFt, curlshape_dFt, elmat);
+      }
+   }
+}
+
 void CurlCurlIntegrator
 ::ComputeElementFlux(const FiniteElement &el, ElementTransformation &Trans,
                      Vector &u, const FiniteElement &fluxelem, Vector &flux,
@@ -2240,6 +2317,84 @@ double VectorCurlCurlIntegrator::GetElementEnergy(
    return 0.5 * energy;
 }
 
+void MixedCurlIntegrator::AssembleElementMatrix2(
+   const FiniteElement &trial_fe, const FiniteElement &test_fe,
+   ElementTransformation &Trans, DenseMatrix &elmat)
+{
+   int dim = trial_fe.GetDim();
+   int trial_dof = trial_fe.GetDof();
+   int test_dof = test_fe.GetDof();
+   int dimc = (dim == 3) ? 3 : 1;
+
+   MFEM_VERIFY(trial_fe.GetMapType() == mfem::FiniteElement::H_CURL ||
+               (dim == 2 && trial_fe.GetMapType() == mfem::FiniteElement::VALUE),
+               "Trial finite element must be either 2D/3D H(Curl) or 2D H1");
+   MFEM_VERIFY(test_fe.GetMapType() == mfem::FiniteElement::VALUE ||
+               test_fe.GetMapType() == mfem::FiniteElement::INTEGRAL,
+               "Test finite element must be in H1/L2");
+
+   bool spaceH1 = (trial_fe.GetMapType() == mfem::FiniteElement::VALUE);
+
+   if (spaceH1)
+   {
+      dshape.SetSize(trial_dof,dim);
+      curlshape.SetSize(dim*trial_dof,1);
+      dimc = dim;
+   }
+   else
+   {
+      curlshape.SetSize(trial_dof,dimc);
+      elmat_comp.SetSize(test_dof, trial_dof);
+   }
+   elmat.SetSize(dimc * test_dof, trial_dof);
+   shape.SetSize(test_dof);
+   elmat = 0.0;
+
+   double c;
+   Vector d_col;
+   const IntegrationRule *ir = IntRule;
+
+   if (ir == NULL)
+   {
+      int order = trial_fe.GetOrder() + test_fe.GetOrder() + Trans.OrderJ();
+      ir = &IntRules.Get(trial_fe.GetGeomType(), order);
+   }
+
+   for (int i = 0; i < ir->GetNPoints(); i++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(i);
+      Trans.SetIntPoint(&ip);
+      if (spaceH1)
+      {
+         trial_fe.CalcPhysDShape(Trans, dshape);
+         dshape.GradToCurl(curlshape);
+      }
+      else
+      {
+         trial_fe.CalcPhysCurlShape(Trans, curlshape);
+      }
+      test_fe.CalcPhysShape(Trans, shape);
+      c = ip.weight*Trans.Weight();
+      if (Q)
+      {
+         c *= Q->Eval(Trans, ip);
+      }
+      shape *= c;
+
+      for (int d = 0; d < dimc; ++d)
+      {
+         double * curldata = &(curlshape.GetData())[d*trial_dof];
+         for (int jj = 0; jj < trial_dof; ++jj)
+         {
+            for (int ii = 0; ii < test_dof; ++ii)
+            {
+               elmat(d * test_dof + ii, jj) += shape(ii) * curldata[jj];
+            }
+         }
+      }
+   }
+}
+
 
 void VectorFEMassIntegrator::AssembleElementMatrix(
    const FiniteElement &el,
@@ -2586,6 +2741,54 @@ void DivDivIntegrator::AssembleElementMatrix(
    }
 }
 
+void DivDivIntegrator::AssembleElementMatrix2(
+   const FiniteElement &trial_fe,
+   const FiniteElement &test_fe,
+   ElementTransformation &Trans,
+   DenseMatrix &elmat)
+{
+   int tr_nd = trial_fe.GetDof();
+   int te_nd = test_fe.GetDof();
+   double c;
+
+#ifdef MFEM_THREAD_SAFE
+   Vector divshape(tr_nd);
+   Vector te_divshape(te_nd);
+#else
+   divshape.SetSize(tr_nd);
+   te_divshape.SetSize(te_nd);
+#endif
+   elmat.SetSize(te_nd,tr_nd);
+
+   const IntegrationRule *ir = IntRule;
+   if (ir == NULL)
+   {
+      int order = 2 * max(test_fe.GetOrder(),
+                          trial_fe.GetOrder()) - 2; // <--- OK for RTk
+      ir = &IntRules.Get(test_fe.GetGeomType(), order);
+   }
+
+   elmat = 0.0;
+
+   for (int i = 0; i < ir -> GetNPoints(); i++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(i);
+
+      trial_fe.CalcDivShape(ip,divshape);
+      test_fe.CalcDivShape(ip,te_divshape);
+
+      Trans.SetIntPoint (&ip);
+      c = ip.weight / Trans.Weight();
+
+      if (Q)
+      {
+         c *= Q -> Eval (Trans, ip);
+      }
+
+      te_divshape *= c;
+      AddMultVWt(te_divshape, divshape, elmat);
+   }
+}
 
 void VectorDiffusionIntegrator::AssembleElementMatrix(
    const FiniteElement &el,
@@ -3780,7 +3983,7 @@ void NormalTraceJumpIntegrator::AssembleFaceMatrix(
       for (i = 0; i < ndof1; i++)
          for (j = 0; j < face_ndof; j++)
          {
-            elmat(i, j) -= shape1_n(i) * face_shape(j);
+            elmat(i, j) += shape1_n(i) * face_shape(j);
          }
       if (ndof2)
       {
@@ -3788,7 +3991,7 @@ void NormalTraceJumpIntegrator::AssembleFaceMatrix(
          for (i = 0; i < ndof2; i++)
             for (j = 0; j < face_ndof; j++)
             {
-               elmat(ndof1+i, j) += shape2_n(i) * face_shape(j);
+               elmat(ndof1+i, j) -= shape2_n(i) * face_shape(j);
             }
       }
    }

fem/bilininteg.hpp

@@ -215,10 +215,10 @@ public:
                              function by any coefficients describing the
                              integrator.
        @param[in] ir  If passed (the default value is NULL), the implementation
-                        of the method will ignore the integration rule provided
-                        by the @a fluxelem parameter and, instead, compute the
-                        discrete flux at the points specified by the integration
-                        rule @a ir.
+                      of the method will ignore the integration rule provided
+                      by the @a fluxelem parameter and, instead, compute the
+                      discrete flux at the points specified by the integration
+                      rule @a ir.
     */
    virtual void ComputeElementFlux(const FiniteElement &el,
                                    ElementTransformation &Trans,
@@ -2525,6 +2525,7 @@ private:
 #ifndef MFEM_THREAD_SAFE
    Vector D;
    DenseMatrix curlshape, curlshape_dFt, M;
+   DenseMatrix te_curlshape, te_curlshape_dFt;
    DenseMatrix vshape, projcurl;
 #endif
 
@@ -2558,6 +2559,11 @@ public:
                                       ElementTransformation &Trans,
                                       DenseMatrix &elmat);
 
+   virtual void AssembleElementMatrix2(const FiniteElement &trial_fe,
+                                       const FiniteElement &test_fe,
+                                       ElementTransformation &Trans,
+                                       DenseMatrix &elmat);
+
    virtual void ComputeElementFlux(const FiniteElement &el,
                                    ElementTransformation &Trans,
                                    Vector &u, const FiniteElement &fluxelem,
@@ -2603,6 +2609,35 @@ public:
                                    const Vector &elfun);
 };
 
+/** Class for integrating the bilinear form a(u,v) := (Q curl u, v) where Q is
+    an optional scalar coefficient, and v is a vector with components v_i in
+    the L2 or H1 space. This integrator handles 3 cases:
+    (a) u ∈ H(curl) in 3D, v is a 3D vector with components v_i in L^2 or H^1
+    (b) u ∈ H(curl) in 2D, v is a scalar field in L^2 or H^1
+    (c) u is a scalar field in H^1, i.e, curl u := [0 1;-1 0]grad u and v is a
+        2D vector field with components v_i in L^2 or H^1 space.
+    Note: Case (b) can also be handled by MixedScalarCurlIntegrator  */
+class MixedCurlIntegrator : public BilinearFormIntegrator
+{
+protected:
+   Coefficient *Q;
+
+private:
+   Vector shape;
+   DenseMatrix dshape;
+   DenseMatrix curlshape;
+   DenseMatrix elmat_comp;
+public:
+   MixedCurlIntegrator() : Q{NULL} { }
+   MixedCurlIntegrator(Coefficient *q_) :  Q{q_} { }
+   MixedCurlIntegrator(Coefficient &q) :  Q{&q} { }
+
+   virtual void AssembleElementMatrix2(const FiniteElement &trial_fe,
+                                       const FiniteElement &test_fe,
+                                       ElementTransformation &Trans,
+                                       DenseMatrix &elmat);
+};
+
 /** Integrator for (Q u, v), where Q is an optional coefficient (of type scalar,
     vector (diagonal matrix), or matrix), trial function u is in H(Curl) or
     H(Div), and test function v is in H(Curl), H(Div), or v=(v1,...,vn), where
@@ -2726,7 +2761,7 @@ protected:
 
 private:
 #ifndef MFEM_THREAD_SAFE
-   Vector divshape;
+   Vector divshape, te_divshape;
 #endif
 
    // PA extension
@@ -2744,6 +2779,12 @@ public:
    virtual void AssembleElementMatrix(const FiniteElement &el,
                                       ElementTransformation &Trans,
                                       DenseMatrix &elmat);
+
+   virtual void AssembleElementMatrix2(const FiniteElement &trial_fe,
+                                       const FiniteElement &test_fe,
+                                       ElementTransformation &Trans,
+                                       DenseMatrix &elmat);
+
    const Coefficient *GetCoefficient() const { return Q; }
 };
 
@@ -3018,8 +3059,8 @@ public:
 
 /** Integrator for the DG form:
 
-    - < {(Q grad(u)).n}, [v] > + sigma < [u], {(Q grad(v)).n} >
-    + kappa < {h^{-1} Q} [u], [v] >,
+        - < {(Q grad(u)).n}, [v] > + sigma < [u], {(Q grad(v)).n} >
+        + kappa < {h^{-1} Q} [u], [v] >
 
     where Q is a scalar or matrix diffusion coefficient and u, v are the trial
     and test spaces, respectively. The parameters sigma and kappa determine the

fem/bilininteg_hcurl.cpp

@@ -2710,7 +2710,7 @@ void CurlCurlIntegrator::AssembleDiagonalPA(Vector& diag)
    }
 }
 
-// Apply to x corresponding to DOF's in H^1 (trial), whose gradients are
+// Apply to x corresponding to DOFs in H^1 (trial), whose gradients are
 // integrated against H(curl) test functions corresponding to y.
 void PAHcurlH1Apply3D(const int D1D,
                       const int Q1D,
@@ -2900,7 +2900,7 @@ void PAHcurlH1Apply3D(const int D1D,
    }); // end of element loop
 }
 
-// Apply to x corresponding to DOF's in H(curl), integrated
+// Apply to x corresponding to DOFs in H(curl), integrated
 // against gradients of H^1 functions corresponding to y.
 void PAHcurlH1ApplyTranspose3D(const int D1D,
                                const int Q1D,
@@ -3099,7 +3099,7 @@ void PAHcurlH1ApplyTranspose3D(const int D1D,
    }); // end of element loop
 }
 
-// Apply to x corresponding to DOF's in H^1 (trial), whose gradients are
+// Apply to x corresponding to DOFs in H^1 (trial), whose gradients are
 // integrated against H(curl) test functions corresponding to y.
 void PAHcurlH1Apply2D(const int D1D,
                       const int Q1D,
@@ -3223,7 +3223,7 @@ void PAHcurlH1Apply2D(const int D1D,
    }); // end of element loop
 }
 
-// Apply to x corresponding to DOF's in H(curl), integrated
+// Apply to x corresponding to DOFs in H(curl), integrated
 // against gradients of H^1 functions corresponding to y.
 void PAHcurlH1ApplyTranspose2D(const int D1D,
                                const int Q1D,
@@ -3543,7 +3543,7 @@ void MixedVectorCurlIntegrator::AssemblePA(const FiniteElementSpace &trial_fes,
    }
 }
 
-// Apply to x corresponding to DOF's in H(curl) (trial), whose curl is
+// Apply to x corresponding to DOFs in H(curl) (trial), whose curl is
 // integrated against H(curl) test functions corresponding to y.
 template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
 static void PAHcurlL2Apply3D(const int D1D,
@@ -3906,7 +3906,7 @@ static void PAHcurlL2Apply3D(const int D1D,
    }); // end of element loop
 }
 
-// Apply to x corresponding to DOF's in H(curl) (trial), whose curl is
+// Apply to x corresponding to DOFs in H(curl) (trial), whose curl is
 // integrated against H(curl) test functions corresponding to y.
 template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
 static void SmemPAHcurlL2Apply3D(const int D1D,
@@ -4216,7 +4216,7 @@ static void SmemPAHcurlL2Apply3D(const int D1D,
    ForallWrap<3>(true, NE, device_kernel, host_kernel, Q1D, Q1D, Q1D);
 }
 
-// Apply to x corresponding to DOF's in H(curl) (trial), whose curl is
+// Apply to x corresponding to DOFs in H(curl) (trial), whose curl is
 // integrated against H(div) test functions corresponding to y.
 template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
 static void PAHcurlHdivApply3D(const int D1D,
@@ -4572,7 +4572,7 @@ static void PAHcurlHdivApply3D(const int D1D,
    }); // end of element loop
 }
 
-// Apply to x corresponding to DOF's in H(div) (test), integrated against the
+// Apply to x corresponding to DOFs in H(div) (test), integrated against the
 // curl of H(curl) trial functions corresponding to y.
 template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
 static void PAHcurlHdivApply3DTranspose(const int D1D,
@@ -5067,7 +5067,7 @@ void MixedVectorWeakCurlIntegrator::AssemblePA(const FiniteElementSpace
    }
 }
 
-// Apply to x corresponding to DOF's in H(curl) (trial), integrated against curl
+// Apply to x corresponding to DOFs in H(curl) (trial), integrated against curl
 // of H(curl) test functions corresponding to y.
 template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
 static void PAHcurlL2Apply3DTranspose(const int D1D,

fem/bilininteg_hdiv.cpp

@@ -1797,7 +1797,7 @@ VectorFEDivergenceIntegrator::AssemblePA(const FiniteElementSpace &trial_fes,
    }
 }
 
-// Apply to x corresponding to DOF's in H(div) (trial), whose divergence is
+// Apply to x corresponding to DOFs in H(div) (trial), whose divergence is
 // integrated against L_2 test functions corresponding to y.
 static void PAHdivL2Apply3D(const int D1D,
                             const int Q1D,
@@ -1960,7 +1960,7 @@ static void PAHdivL2Apply3D(const int D1D,
    }); // end of element loop
 }
 
-// Apply to x corresponding to DOF's in H(div) (trial), whose divergence is
+// Apply to x corresponding to DOFs in H(div) (trial), whose divergence is
 // integrated against L_2 test functions corresponding to y.
 static void PAHdivL2Apply2D(const int D1D,
                             const int Q1D,

fem/ceed/solvers/algebraic.cpp

@@ -676,7 +676,6 @@ AlgebraicSpaceHierarchy::AlgebraicSpaceHierarchy(FiniteElementSpace &fes)
       const SparseMatrix *R = fespaces[ilevel+1]->GetRestrictionMatrix();
       if (R)
       {
-         R->EnsureMultTranspose();
          R_tr[ilevel] = new TransposeOperator(*R);
       }
       else
@@ -822,11 +821,11 @@ HypreParMatrix *ParAlgebraicCoarseSpace::GetProlongationHypreParMatrix()
 
    ParMesh *pmesh = dynamic_cast<ParMesh*>(mesh);
    MFEM_VERIFY(pmesh != NULL, "");
-   Array<HYPRE_Int> dof_offsets, tdof_offsets, tdof_nb_offsets;
-   Array<HYPRE_Int> *offsets[2] = {&dof_offsets, &tdof_offsets};
+   Array<HYPRE_BigInt> dof_offsets, tdof_offsets, tdof_nb_offsets;
+   Array<HYPRE_BigInt> *offsets[2] = {&dof_offsets, &tdof_offsets};
    int lsize = P->Height();
    int ltsize = P->Width();
-   HYPRE_Int loc_sizes[2] = {lsize, ltsize};
+   HYPRE_BigInt loc_sizes[2] = {lsize, ltsize};
    pmesh->GenerateOffsets(2, loc_sizes, offsets);
 
    MPI_Comm comm = pmesh->GetComm();
@@ -870,12 +869,12 @@ HypreParMatrix *ParAlgebraicCoarseSpace::GetProlongationHypreParMatrix()
    HYPRE_Int *j_offd = Memory<HYPRE_Int>(lsize-ltsize);
    int offd_counter;
 
-   HYPRE_Int *cmap   = Memory<HYPRE_Int>(lsize-ltsize);
+   HYPRE_BigInt *cmap   = Memory<HYPRE_BigInt>(lsize-ltsize);
 
-   HYPRE_Int *col_starts = tdof_offsets;
-   HYPRE_Int *row_starts = dof_offsets;
+   HYPRE_BigInt *col_starts = tdof_offsets;
+   HYPRE_BigInt *row_starts = dof_offsets;
 
-   Array<Pair<HYPRE_Int, int> > cmap_j_offd(lsize-ltsize);
+   Array<Pair<HYPRE_BigInt, int> > cmap_j_offd(lsize-ltsize);
 
    i_diag[0] = i_offd[0] = 0;
    diag_counter = offd_counter = 0;
@@ -909,7 +908,7 @@ HypreParMatrix *ParAlgebraicCoarseSpace::GetProlongationHypreParMatrix()
       i_offd[i_ldof+1] = offd_counter;
    }
 
-   SortPairs<HYPRE_Int, int>(cmap_j_offd, offd_counter);
+   SortPairs<HYPRE_BigInt, int>(cmap_j_offd, offd_counter);
 
    for (int i = 0; i < offd_counter; i++)
    {

fem/coefficient.hpp

@@ -2239,7 +2239,7 @@ public:
    /// @sa CoefficientVector for a description of the @a compress argument.
    void Project(MatrixCoefficient &coeff, bool transpose=false);
 
-   /// @brief Project the tranpose of @a coeff.
+   /// @brief Project the transpose of @a coeff.
    ///
    /// @sa Project(MatrixCoefficient&, QuadratureSpace&, bool, bool)
    void ProjectTranspose(MatrixCoefficient &coeff);

fem/estimators.cpp

@@ -159,7 +159,7 @@ void KellyErrorEstimator::ComputeEstimates()
    // the FaceInfo class [1]. Also, the FaceElementTransformations
    // documentation [2] may be helpful to grasp what is going on. Note
    // that the FaceElementTransformations also works in the non-
-   // conforming case to transfer the gauss points from the slave to
+   // conforming case to transfer the Gauss points from the slave to
    // the master element.
    // [1]
    // https://github.com/mfem/mfem/blob/02d0bfe9c18ce049c3c93a6a4208080fcfc96991/mesh/mesh.hpp#L94
@@ -417,7 +417,7 @@ void KellyErrorEstimator::ComputeEstimates()
          Vector val(flux_space->GetVDim());
          flux->GetVectorValue(FT->Elem2No, ip, val);
 
-         // Evaluate gauss point
+         // Evaluate Gauss point
          Vector normal(mesh->SpaceDimension());
          FT->Face->SetIntPoint(&fip);
          if (mesh->Dimension() == mesh->SpaceDimension())

fem/fe/fe_nd.cpp

@@ -314,7 +314,7 @@ void ND_HexahedronElement::CalcVShape(const IntegrationPoint &ip,
 #ifdef MFEM_THREAD_SAFE
    Vector shape_cx(p + 1), shape_ox(p), shape_cy(p + 1), shape_oy(p);
    Vector shape_cz(p + 1), shape_oz(p);
-   Vector dshape_cx, dshape_cy, dshape_cz;
+   Vector dshape_cx(p + 1), dshape_cy(p + 1), dshape_cz(p + 1);
 #endif
 
    if (obasis1d.IsIntegratedType())
@@ -656,7 +656,7 @@ void ND_QuadrilateralElement::CalcVShape(const IntegrationPoint &ip,
 
 #ifdef MFEM_THREAD_SAFE
    Vector shape_cx(p + 1), shape_ox(p), shape_cy(p + 1), shape_oy(p);
-   Vector dshape_cx, dshape_cy;
+   Vector dshape_cx(p + 1), dshape_cy(p + 1);
 #endif
 
    if (obasis1d.IsIntegratedType())

fem/fe/fe_rt.cpp

@@ -145,7 +145,7 @@ void RT_QuadrilateralElement::CalcVShape(const IntegrationPoint &ip,
 
 #ifdef MFEM_THREAD_SAFE
    Vector shape_cx(pp1 + 1), shape_ox(pp1), shape_cy(pp1 + 1), shape_oy(pp1);
-   Vector dshape_cx, dshape_cy;
+   Vector dshape_cx(pp1 + 1), dshape_cy(pp1 + 1);
 #endif
 
    if (obasis1d.IsIntegratedType())
@@ -473,7 +473,7 @@ void RT_HexahedronElement::CalcVShape(const IntegrationPoint &ip,
 #ifdef MFEM_THREAD_SAFE
    Vector shape_cx(pp1 + 1), shape_ox(pp1), shape_cy(pp1 + 1), shape_oy(pp1);
    Vector shape_cz(pp1 + 1), shape_oz(pp1);
-   Vector dshape_cx, dshape_cy, dshape_cz;
+   Vector dshape_cx(pp1 + 1), dshape_cy(pp1 + 1), dshape_cz(pp1 + 1);
 #endif
 
    if (obasis1d.IsIntegratedType())

fem/fespace.cpp

@@ -1199,8 +1199,6 @@ void FiniteElementSpace::BuildConformingInterpolation() const
       MakeVDimMatrix(*cR);
       if (cR_hp) { MakeVDimMatrix(*cR_hp); }
    }
-
-   cP->EnsureMultTranspose();
 }
 
 void FiniteElementSpace::MakeVDimMatrix(SparseMatrix &mat) const
@@ -1258,7 +1256,7 @@ int FiniteElementSpace::GetNConformingDofs() const
    return P ? (P->Width() / vdim) : ndofs;
 }
 
-const Operator *FiniteElementSpace::GetElementRestriction(
+const ElementRestrictionOperator *FiniteElementSpace::GetElementRestriction(
    ElementDofOrdering e_ordering) const
 {
    // Check if we have a discontinuous space using the FE collection:
@@ -1273,7 +1271,7 @@ const Operator *FiniteElementSpace::GetElementRestriction(
          // The output E-vector layout is: ND x VDIM x NE.
          L2E_nat.Reset(new L2ElementRestriction(*this));
       }
-      return L2E_nat.Ptr();
+      return L2E_nat.Is<ElementRestrictionOperator>();
    }
    if (e_ordering == ElementDofOrdering::LEXICOGRAPHIC)
    {
@@ -1281,14 +1279,14 @@ const Operator *FiniteElementSpace::GetElementRestriction(
       {
          L2E_lex.Reset(new ElementRestriction(*this, e_ordering));
       }
-      return L2E_lex.Ptr();
+      return L2E_lex.Is<ElementRestrictionOperator>();
    }
    // e_ordering == ElementDofOrdering::NATIVE
    if (L2E_nat.Ptr() == NULL)
    {
       L2E_nat.Reset(new ElementRestriction(*this, e_ordering));
    }
-   return L2E_nat.Ptr();
+   return L2E_nat.Is<ElementRestrictionOperator>();
 }
 
 const FaceRestriction *FiniteElementSpace::GetFaceRestriction(

fem/fespace.hpp

@@ -516,7 +516,8 @@ public:
        L2ElementRestriction class.
 
        The returned Operator is owned by the FiniteElementSpace. */
-   const Operator *GetElementRestriction(ElementDofOrdering e_ordering) const;
+   const ElementRestrictionOperator *GetElementRestriction(
+      ElementDofOrdering e_ordering) const;
 
    /// Return an Operator that converts L-vectors to E-vectors on each face.
    virtual const FaceRestriction *GetFaceRestriction(
@@ -530,7 +531,12 @@ public:
        Operator returned by GetElementRestriction().
 
        All elements will use the same IntegrationRule, @a ir as the target
-       quadrature points. */
+       quadrature points.
+
+       @note The returned pointer is shared. A good practice, before using it,
+       is to set all its properties to their expected values, as other parts of
+       the code may also change them. That is, it's good to call
+       SetOutputLayout() and DisableTensorProducts() before interpolating. */
    const QuadratureInterpolator *GetQuadratureInterpolator(
       const IntegrationRule &ir) const;
 
@@ -541,12 +547,22 @@ public:
        Operator returned by GetElementRestriction().
 
        The target quadrature points in the elements are described by the given
-       QuadratureSpace, @a qs. */
+       QuadratureSpace, @a qs.
+
+       @note The returned pointer is shared. A good practice, before using it,
+       is to set all its properties to their expected values, as other parts of
+       the code may also change them. That is, it's good to call
+       SetOutputLayout() and DisableTensorProducts() before interpolating. */
    const QuadratureInterpolator *GetQuadratureInterpolator(
       const QuadratureSpace &qs) const;
 
    /** @brief Return a FaceQuadratureInterpolator that interpolates E-vectors to
-       quadrature point values and/or derivatives (Q-vectors). */
+       quadrature point values and/or derivatives (Q-vectors).
+
+       @note The returned pointer is shared. A good practice, before using it,
+       is to set all its properties to their expected values, as other parts of
+       the code may also change them. That is, it's good to call
+       SetOutputLayout() and DisableTensorProducts() before interpolating. */
    const FaceQuadratureInterpolator *GetFaceQuadratureInterpolator(
       const IntegrationRule &ir, FaceType type) const;
 

fem/gridfunc.cpp

@@ -189,6 +189,8 @@ void GridFunction::Update()
    {
       SetSize(fes->GetVSize());
    }
+
+   if (t_vec.Size() > 0) { SetTrueVector(); }
 }
 
 void GridFunction::SetSpace(FiniteElementSpace *f)
@@ -4090,7 +4092,7 @@ void TensorProductLegendre(int dim,                // input
    }
    else
    {
-      // Bounding box is not reorientated no need to change orientation
+      // Bounding box is not reoriented no need to change orientation
       x = x_in;
    }
 

fem/gridfunc.hpp

@@ -129,19 +129,21 @@ public:
    int CurlDim() const;
 
    /// Read only access to the (optional) internal true-dof Vector.
-   /** Note that the returned Vector may be empty, if not previously allocated
-       or set. */
-   const Vector &GetTrueVector() const { return t_vec; }
+   const Vector &GetTrueVector() const
+   {
+      MFEM_VERIFY(t_vec.Size() > 0, "SetTrueVector() before GetTrueVector()");
+      return t_vec;
+   }
    /// Read and write access to the (optional) internal true-dof Vector.
-   /** Note that the returned Vector may be empty, if not previously allocated
-       or set. */
-   Vector &GetTrueVector() { return t_vec; }
+   /** Note that @a t_vec is set if it is not allocated or set already.*/
+   Vector &GetTrueVector()
+   { if (t_vec.Size() == 0) { SetTrueVector(); } return t_vec; }
 
    /// Extract the true-dofs from the GridFunction.
    void GetTrueDofs(Vector &tv) const;
 
    /// Shortcut for calling GetTrueDofs() with GetTrueVector() as argument.
-   void SetTrueVector() { GetTrueDofs(GetTrueVector()); }
+   void SetTrueVector() { GetTrueDofs(t_vec); }
 
    /// Set the GridFunction from the given true-dof vector.
    virtual void SetFromTrueDofs(const Vector &tv);

fem/gslib.cpp

@@ -168,7 +168,8 @@ void FindPointsGSLIB::Setup(Mesh &m, const double bb_t, const double newt_tol,
    setupflag = true;
 }
 
-void FindPointsGSLIB::FindPoints(const Vector &point_pos)
+void FindPointsGSLIB::FindPoints(const Vector &point_pos,
+                                 int point_pos_ordering)
 {
    MFEM_VERIFY(setupflag, "Use FindPointsGSLIB::Setup before finding points.");
    points_cnt = point_pos.Size() / dim;
@@ -178,14 +179,24 @@ void FindPointsGSLIB::FindPoints(const Vector &point_pos)
    gsl_ref.SetSize(points_cnt * dim);
    gsl_dist.SetSize(points_cnt);
 
+   const double *xv_base[dim];
+   unsigned xv_stride[dim];
+   for (int d = 0; d < dim; d++)
+   {
+      if (point_pos_ordering == Ordering::byNODES)
+      {
+         xv_base[d] = point_pos.GetData() + d*points_cnt;
+         xv_stride[d] = sizeof(double);
+      }
+      else
+      {
+         xv_base[d] = point_pos.GetData() + d;
+         xv_stride[d] = dim*sizeof(double);
+      }
+   }
+
    if (dim == 2)
    {
-      const double *xv_base[2];
-      xv_base[0] = point_pos.GetData();
-      xv_base[1] = point_pos.GetData() + points_cnt;
-      unsigned xv_stride[2];
-      xv_stride[0] = sizeof(double);
-      xv_stride[1] = sizeof(double);
       findpts_2(gsl_code.GetData(), sizeof(unsigned int),
                 gsl_proc.GetData(), sizeof(unsigned int),
                 gsl_elem.GetData(), sizeof(unsigned int),
@@ -195,14 +206,6 @@ void FindPointsGSLIB::FindPoints(const Vector &point_pos)
    }
    else
    {
-      const double *xv_base[3];
-      xv_base[0] = point_pos.GetData();
-      xv_base[1] = point_pos.GetData() + points_cnt;
-      xv_base[2] = point_pos.GetData() + 2*points_cnt;
-      unsigned xv_stride[3];
-      xv_stride[0] = sizeof(double);
-      xv_stride[1] = sizeof(double);
-      xv_stride[2] = sizeof(double);
       findpts_3(gsl_code.GetData(), sizeof(unsigned int),
                 gsl_proc.GetData(), sizeof(unsigned int),
                 gsl_elem.GetData(), sizeof(unsigned int),
@@ -227,27 +230,29 @@ void FindPointsGSLIB::FindPoints(const Vector &point_pos)
 }
 
 void FindPointsGSLIB::FindPoints(Mesh &m, const Vector &point_pos,
-                                 const double bb_t, const double newt_tol,
-                                 const int npt_max)
+                                 int point_pos_ordering, const double bb_t,
+                                 const double newt_tol, const int npt_max)
 {
    if (!setupflag || (mesh != &m) )
    {
       Setup(m, bb_t, newt_tol, npt_max);
    }
-   FindPoints(point_pos);
+   FindPoints(point_pos, point_pos_ordering);
 }
 
 void FindPointsGSLIB::Interpolate(const Vector &point_pos,
-                                  const GridFunction &field_in, Vector &field_out)
+                                  const GridFunction &field_in, Vector &field_out,
+                                  int point_pos_ordering)
 {
-   FindPoints(point_pos);
+   FindPoints(point_pos, point_pos_ordering);
    Interpolate(field_in, field_out);
 }
 
 void FindPointsGSLIB::Interpolate(Mesh &m, const Vector &point_pos,
-                                  const GridFunction &field_in, Vector &field_out)
+                                  const GridFunction &field_in, Vector &field_out,
+                                  int point_pos_ordering)
 {
-   FindPoints(m, point_pos);
+   FindPoints(m, point_pos, point_pos_ordering);
    Interpolate(field_in, field_out);
 }
 
@@ -860,7 +865,9 @@ void FindPointsGSLIB::Interpolate(const GridFunction &field_in,
       {
          for (int i = 0; i < indl2.Size(); i++)
          {
-            int idx = indl2[i] + j*points_cnt;
+            int idx = field_in.FESpace()->GetOrdering() == Ordering::byNODES ?
+                      indl2[i] + j*points_cnt:
+                      indl2[i]*ncomp + j;
             field_out(idx) = field_out_l2(idx);
          }
       }
@@ -885,7 +892,17 @@ void FindPointsGSLIB::InterpolateH1(const GridFunction &field_in,
    {
       const int dataptrin  = i*points_fld,
                 dataptrout = i*points_cnt;
-      field_in_scalar.NewDataAndSize(field_in.GetData()+dataptrin, points_fld);
+      if (field_in.FESpace()->GetOrdering() == Ordering::byNODES)
+      {
+         field_in_scalar.NewDataAndSize(field_in.GetData()+dataptrin, points_fld);
+      }
+      else
+      {
+         for (int j = 0; j < points_fld; j++)
+         {
+            field_in_scalar(j) = field_in(i + j*ncomp);
+         }
+      }
       GetNodalValues(&field_in_scalar, node_vals);
 
       if (dim==2)
@@ -907,6 +924,17 @@ void FindPointsGSLIB::InterpolateH1(const GridFunction &field_in,
                         points_cnt, node_vals.GetData(), fdata3D);
       }
    }
+   if (field_in.FESpace()->GetOrdering() == Ordering::byVDIM)
+   {
+      Vector field_out_temp = field_out;
+      for (int i = 0; i < ncomp; i++)
+      {
+         for (int j = 0; j < points_cnt; j++)
+         {
+            field_out(i + j*ncomp) = field_out_temp(j + i*points_cnt);
+         }
+      }
+   }
 }
 
 void FindPointsGSLIB::InterpolateGeneral(const GridFunction &field_in,
@@ -929,9 +957,19 @@ void FindPointsGSLIB::InterpolateGeneral(const GridFunction &field_in,
          if (dim == 3) { ip.z = gsl_mfem_ref(index*dim + 2); }
          Vector localval(ncomp);
          field_in.GetVectorValue(gsl_mfem_elem[index], ip, localval);
-         for (int i = 0; i < ncomp; i++)
+         if (field_in.FESpace()->GetOrdering() == Ordering::byNODES)
          {
-            field_out(index + i*npt) = localval(i);
+            for (int i = 0; i < ncomp; i++)
+            {
+               field_out(index + i*npt) = localval(i);
+            }
+         }
+         else //byVDIM
+         {
+            for (int i = 0; i < ncomp; i++)
+            {
+               field_out(index*ncomp + i) = localval(i);
+            }
          }
       }
    }
@@ -1044,7 +1082,9 @@ void FindPointsGSLIB::InterpolateGeneral(const GridFunction &field_in,
             sdpt = (struct send_pt *)sendpt->ptr;
             for (int index = 0; index < sendpt->n; index++)
             {
-               int idx = sdpt->index + j*nptorig;
+               int idx = field_in.FESpace()->GetOrdering() == Ordering::byNODES ?
+                         sdpt->index + j*nptorig :
+                         sdpt->index*ncomp + j;
                field_out(idx) = sdpt->ival;
                ++sdpt;
             }
@@ -1139,7 +1179,8 @@ void OversetFindPointsGSLIB::Setup(Mesh &m, const int meshid,
 }
 
 void OversetFindPointsGSLIB::FindPoints(const Vector &point_pos,
-                                        Array<unsigned int> &point_id)
+                                        Array<unsigned int> &point_id,
+                                        int point_pos_ordering)
 {
    MFEM_VERIFY(setupflag, "Use OversetFindPointsGSLIB::Setup before "
                "finding points.");
@@ -1153,14 +1194,24 @@ void OversetFindPointsGSLIB::FindPoints(const Vector &point_pos,
    gsl_ref.SetSize(points_cnt * dim);
    gsl_dist.SetSize(points_cnt);
 
+   const double *xv_base[dim];
+   unsigned xv_stride[dim];
+   for (int d = 0; d < dim; d++)
+   {
+      if (point_pos_ordering == Ordering::byNODES)
+      {
+         xv_base[d] = point_pos.GetData() + d*points_cnt;
+         xv_stride[d] = sizeof(double);
+      }
+      else
+      {
+         xv_base[d] = point_pos.GetData() + d;
+         xv_stride[d] = dim*sizeof(double);
+      }
+   }
+
    if (dim == 2)
    {
-      const double *xv_base[2];
-      xv_base[0] = point_pos.GetData();
-      xv_base[1] = point_pos.GetData() + points_cnt;
-      unsigned xv_stride[2];
-      xv_stride[0] = sizeof(double);
-      xv_stride[1] = sizeof(double);
       findptsms_2(gsl_code.GetData(), sizeof(unsigned int),
                   gsl_proc.GetData(), sizeof(unsigned int),
                   gsl_elem.GetData(), sizeof(unsigned int),
@@ -1172,14 +1223,6 @@ void OversetFindPointsGSLIB::FindPoints(const Vector &point_pos,
    }
    else
    {
-      const double *xv_base[3];
-      xv_base[0] = point_pos.GetData();
-      xv_base[1] = point_pos.GetData() + points_cnt;
-      xv_base[2] = point_pos.GetData() + 2*points_cnt;
-      unsigned xv_stride[3];
-      xv_stride[0] = sizeof(double);
-      xv_stride[1] = sizeof(double);
-      xv_stride[2] = sizeof(double);
       findptsms_3(gsl_code.GetData(), sizeof(unsigned int),
                   gsl_proc.GetData(), sizeof(unsigned int),
                   gsl_elem.GetData(), sizeof(unsigned int),
@@ -1208,9 +1251,10 @@ void OversetFindPointsGSLIB::FindPoints(const Vector &point_pos,
 void OversetFindPointsGSLIB::Interpolate(const Vector &point_pos,
                                          Array<unsigned int> &point_id,
                                          const GridFunction &field_in,
-                                         Vector &field_out)
+                                         Vector &field_out,
+                                         int point_pos_ordering)
 {
-   FindPoints(point_pos, point_id);
+   FindPoints(point_pos, point_id, point_pos_ordering);
    Interpolate(field_in, field_out);
 }
 

fem/gslib.hpp

@@ -121,8 +121,9 @@ public:
    void Setup(Mesh &m, const double bb_t = 0.1,
               const double newt_tol = 1.0e-12,
               const int npt_max = 256);
-   /** Searches positions given in physical space by @a point_pos. These positions
-       must by ordered by nodes: (XXX...,YYY...,ZZZ).
+   /** Searches positions given in physical space by @a point_pos.
+       These positions can be ordered byNodes: (XXX...,YYY...,ZZZ) or
+       byVDim: (XYZ,XYZ,....XYZ) specified by @a point_pos_ordering.
        This function populates the following member variables:
        #gsl_code        Return codes for each point: inside element (0),
                         element boundary (1), not found (2).
@@ -140,9 +141,11 @@ public:
                         Defaults to 0 for points that were not found.
        #gsl_dist        Distance between the sought and the found point
                         in physical space. */
-   void FindPoints(const Vector &point_pos);
+   void FindPoints(const Vector &point_pos,
+                   int point_pos_ordering = Ordering::byNODES);
    /// Setup FindPoints and search positions
    void FindPoints(Mesh &m, const Vector &point_pos,
+                   int point_pos_ordering = Ordering::byNODES,
                    const double bb_t = 0.1,
                    const double newt_tol = 1.0e-12,  const int npt_max = 256);
 
@@ -154,12 +157,18 @@ public:
        @param[out] field_out  Interpolated values. For points that are not found
                               the value is set to #default_interp_value. */
    virtual void Interpolate(const GridFunction &field_in, Vector &field_out);
-   /** Search positions and interpolate */
+   /** Search positions and interpolate. The ordering (byNODES or byVDIM) of
+       the output values in @a field_out corresponds to the ordering used
+       in the input GridFunction @a field_in. */
    void Interpolate(const Vector &point_pos, const GridFunction &field_in,
-                    Vector &field_out);
-   /** Setup FindPoints, search positions and interpolate */
+                    Vector &field_out,
+                    int point_pos_ordering = Ordering::byNODES);
+   /** Setup FindPoints, search positions and interpolate. The ordering (byNODES
+       or byVDIM) of the output values in @a field_out corresponds to the
+       ordering used in the input GridFunction @a field_in. */
    void Interpolate(Mesh &m, const Vector &point_pos,
-                    const GridFunction &field_in, Vector &field_out);
+                    const GridFunction &field_in, Vector &field_out,
+                    int point_pos_ordering = Ordering::byNODES);
 
    /// Average type to be used for L2 functions in-case a point is located at
    /// an element boundary where the function might be multi-valued.
@@ -247,15 +256,20 @@ public:
    /** Searches positions given in physical space by @a point_pos. All output
        Arrays and Vectors are expected to have the correct size.
 
-       @param[in]  point_pos  Positions to be found. Must by ordered by nodes
-                              (XXX...,YYY...,ZZZ).
-       @param[in]  point_id   Index of the mesh that the point belongs to
-                              (corresponding to @a meshid in Setup). */
-   void FindPoints(const Vector &point_pos, Array<unsigned int> &point_id);
+       @param[in]  point_pos           Positions to be found.
+       @param[in]  point_id            Index of the mesh that the point belongs
+                                       to (corresponding to @a meshid in Setup).
+       @param[in]  point_pos_ordering  Ordering of the points:
+                                       byNodes: (XXX...,YYY...,ZZZ) or
+                                       byVDim: (XYZ,XYZ,....XYZ) */
+   void FindPoints(const Vector &point_pos,
+                   Array<unsigned int> &point_id,
+                   int point_pos_ordering = Ordering::byNODES);
 
    /** Search positions and interpolate */
    void Interpolate(const Vector &point_pos, Array<unsigned int> &point_id,
-                    const GridFunction &field_in, Vector &field_out);
+                    const GridFunction &field_in, Vector &field_out,
+                    int point_pos_ordering = Ordering::byNODES);
    using FindPointsGSLIB::Interpolate;
 };
 

fem/hybridization.cpp

@@ -827,7 +827,6 @@ void Hybridization::ReduceRHS(const Vector &b, Vector &b_r) const
       }
       else
       {
-         Ct->EnsureMultTranspose();
          Ct->MultTranspose(bf, bl);
       }
       b_r.SetSize(pH.Ptr()->Height());

fem/linearform.cpp

@@ -19,12 +19,13 @@ namespace mfem
 LinearForm::LinearForm(FiniteElementSpace *f, LinearForm *lf)
    : Vector(f->GetVSize())
 {
-   // Linear forms are stored on the device
-   UseDevice(true);
-
-   fes = f;
    ext = nullptr;
    extern_lfs = 1;
+   fast_assembly = false;
+   fes = f;
+
+   // Linear forms are stored on the device
+   UseDevice(true);
 
    // Copy the pointers to the integrators
    domain_integs = lf->domain_integs;
@@ -102,9 +103,10 @@ void LinearForm::AddInteriorFaceIntegrator(LinearFormIntegrator *lfi)
 
 bool LinearForm::SupportsDevice()
 {
-   // return false for NURBS meshs, so we don’t convert it to non-NURBS
+   // return false for NURBS meshes, so we don’t convert it to non-NURBS
    // through Assemble, AssembleDevice, GetGeometricFactors and EnsureNodes
-   if (fes->GetMesh()->NURBSext != nullptr) { return false; }
+   const Mesh &mesh = *fes->GetMesh();
+   if (mesh.NURBSext != nullptr) { return false; }
 
    // scan integrators to verify that all can use device assembly
    auto IntegratorsSupportDevice = [](const Array<LinearFormIntegrator*> &integ)
@@ -123,25 +125,23 @@ bool LinearForm::SupportsDevice()
 
    if (boundary_integs.Size() > 0)
    {
-      // Make sure every boundary element corresponds to a boundary face
-      for (int be = 0; be < fes->GetNBE(); ++be)
-      {
-         const int f = fes->GetMesh()->GetBdrElementEdgeIndex(be);
-         const auto face_info = fes->GetMesh()->GetFaceInformation(f);
-         if (!face_info.IsBoundary())
-         {
-            return false;
-         }
-      }
       // Make sure there are no boundary faces that are not boundary elements
       if (fes->GetNFbyType(FaceType::Boundary) != fes->GetNBE())
       {
          return false;
       }
+      // Make sure every boundary element corresponds to a boundary face
+      for (int be = 0; be < fes->GetNBE(); ++be)
+      {
+         const int f = mesh.GetBdrElementEdgeIndex(be);
+         const auto face_info = mesh.GetFaceInformation(f);
+         if (!face_info.IsBoundary())
+         {
+            return false;
+         }
+      }
    }
 
-   const Mesh &mesh = *fes->GetMesh();
-
    // no support for elements with varying polynomial orders
    if (fes->IsVariableOrder()) { return false; }
 
@@ -155,25 +155,30 @@ bool LinearForm::SupportsDevice()
    return true;
 }
 
-void LinearForm::Assemble(bool use_device)
+void LinearForm::UseFastAssembly(bool use_fa)
+{
+   fast_assembly = use_fa;
+
+   if (fast_assembly && SupportsDevice() && !ext)
+   {
+      ext = new LinearFormExtension(this);
+   }
+}
+
+void LinearForm::Assemble()
 {
    Array<int> vdofs;
    ElementTransformation *eltrans;
    DofTransformation *doftrans;
    Vector elemvect;
 
-   if (!ext && use_device && SupportsDevice())
-   {
-      ext = new LinearFormExtension(this);
-   }
-
    Vector::operator=(0.0);
 
    // The above operation is executed on device because of UseDevice().
    // The first use of AddElementVector() below will move it back to host
    // because both 'vdofs' and 'elemvect' are on host.
 
-   if (ext) { return ext->Assemble(); }
+   if (fast_assembly && ext) { return ext->Assemble(); }
 
    if (domain_integs.Size())
    {

fem/linearform.hpp

@@ -30,12 +30,16 @@ protected:
    FiniteElementSpace *fes;
 
    /** @brief Extension for supporting different assembly levels. */
-   LinearFormExtension *ext;
+   LinearFormExtension *ext = nullptr;
+
+   /// @brief Should we use the device-compatible fast assembly algorithm (false
+   /// by default)
+   bool fast_assembly = false;
 
    /** @brief Indicates the LinearFormIntegrator%s stored in #domain_integs,
        #domain_delta_integs, #boundary_integs, and #boundary_face_integs are
        owned by another LinearForm. */
-   int extern_lfs;
+   int extern_lfs = 0;
 
    /// Set of Domain Integrators to be applied.
    Array<LinearFormIntegrator*> domain_integs;
@@ -82,7 +86,7 @@ public:
    /// Creates linear form associated with FE space @a *f.
    /** The pointer @a f is not owned by the newly constructed object. */
    LinearForm(FiniteElementSpace *f) : Vector(f->GetVSize())
-   { fes = f; ext = nullptr; extern_lfs = 0; UseDevice(true); }
+   { fes = f; UseDevice(true); }
 
    /** @brief Create a LinearForm on the FiniteElementSpace @a f, using the
        same integrators as the LinearForm @a lf.
@@ -97,7 +101,8 @@ public:
    /** The associated FiniteElementSpace can be set later using one of the
        methods: Update(FiniteElementSpace *) or
        Update(FiniteElementSpace *, Vector &, int). */
-   LinearForm() { fes = NULL; ext = nullptr; extern_lfs = 0; UseDevice(true); }
+   LinearForm()
+   { fes = NULL; UseDevice(true); }
 
    /// Construct a LinearForm using previously allocated array @a data.
    /** The LinearForm does not assume ownership of @a data which is assumed to
@@ -105,7 +110,7 @@ public:
        for externally allocated array, the pointer @a data can be NULL. The data
        array can be replaced later using the method SetData(). */
    LinearForm(FiniteElementSpace *f, double *data) : Vector(data, f->GetVSize())
-   { fes = f; ext = nullptr; extern_lfs = 0; }
+   { fes = f; }
 
    /// Copy assignment. Only the data of the base class Vector is copied.
    /** It is assumed that this object and @a rhs use FiniteElementSpace%s that
@@ -185,13 +190,20 @@ public:
        corresponding pointer (to Array<int>) will be NULL. */
    Array<Array<int>*> *GetFLFI_Marker() { return &boundary_face_integs_marker; }
 
+   /// @brief Which assembly algorithm to use: the new device-compatible fast
+   /// assembly (true), or the legacy CPU-only algorithm (false).
+   /** If not set, the default value is false.  If used, this method must be
+       called before assembly. */
+   void UseFastAssembly(bool use_fa);
+
    /// Assembles the linear form i.e. sums over all domain/bdr integrators.
-   /// When @a use_device is set to true and the linearform assembly is
-   /// compatible with device execution, it will be executed on the device.
-   void Assemble(bool use_device = true);
+   /** When @ref UseFastAssembly "UseFastAssembly(true)" has been called and the
+       linearform assembly is compatible with device execution, it will be
+       executed on the device. */
+   void Assemble();
 
    /// Return true if assembly on device is supported, false otherwise.
-   bool SupportsDevice();
+   virtual bool SupportsDevice();
 
    /// Assembles delta functions of the linear form
    void AssembleDelta();

fem/linearform_ext.cpp

@@ -25,7 +25,8 @@ void LinearFormExtension::Assemble()
                "match the number of vector dofs!");
 
    const Array<Array<int>*> &domain_integs_marker = *lf->GetDLFI_Marker();
-   const int mesh_attributes_size = fes.GetMesh()->attributes.Size();
+   const int mesh_attributes_max = fes.GetMesh()->attributes.Size() ?
+                                   fes.GetMesh()->attributes.Max() : 0;
    const Array<LinearFormIntegrator*> &domain_integs = *lf->GetDLFI();
 
    for (int k = 0; k < domain_integs.Size(); ++k)
@@ -39,7 +40,7 @@ void LinearFormExtension::Assemble()
       if (has_markers_k)
       {
          // Element attribute marker should be of length mesh->attributes
-         MFEM_VERIFY(mesh_attributes_size == domain_integs_marker_k->Size(),
+         MFEM_VERIFY(mesh_attributes_max == domain_integs_marker_k->Size(),
                      "invalid element marker for domain linear form "
                      "integrator #" << k << ", counting from zero");
       }
@@ -59,11 +60,13 @@ void LinearFormExtension::Assemble()
       // Assemble the linear form
       b = 0.0;
       domain_integs[k]->AssembleDevice(fes, markers, b);
-      elem_restrict_lex->MultTranspose(b, *lf);
+      if (k == 0) { elem_restrict_lex->MultTranspose(b, *lf); }
+      else { elem_restrict_lex->AddMultTranspose(b, *lf); }
    }
 
    const Array<Array<int>*> &boundary_integs_marker = lf->boundary_integs_marker;
-   const int bdr_attributes_size = fes.GetMesh()->bdr_attributes.Size();
+   const int bdr_attributes_max = fes.GetMesh()->bdr_attributes.Size() ?
+                                  fes.GetMesh()->bdr_attributes.Max() : 0;
    const Array<LinearFormIntegrator*> &boundary_integs = lf->boundary_integs;
 
    for (int k = 0; k < boundary_integs.Size(); ++k)
@@ -77,7 +80,7 @@ void LinearFormExtension::Assemble()
       if (has_markers_k)
       {
          // Element attribute marker should be of length mesh->attributes
-         MFEM_VERIFY(bdr_attributes_size == boundary_integs_marker_k->Size(),
+         MFEM_VERIFY(bdr_attributes_max == boundary_integs_marker_k->Size(),
                      "invalid boundary marker for boundary linear form "
                      "integrator #" << k << ", counting from zero");
       }

fem/linearform_ext.hpp

@@ -34,7 +34,7 @@ class LinearFormExtension
    LinearForm *lf;
 
    /// Operator that converts FiniteElementSpace L-vectors to E-vectors.
-   const Operator *elem_restrict_lex; // Not owned
+   const ElementRestrictionOperator *elem_restrict_lex; // Not owned
 
    /// Operator that converts L-vectors to boundary E-vectors.
    const FaceRestriction *bdr_restrict_lex; // Not owned

fem/lininteg_boundary.cpp

@@ -216,40 +216,10 @@ void BoundaryLFIntegrator::AssembleDevice(const FiniteElementSpace &fes,
    const int qorder = oa * fe.GetOrder() + ob;
    const Geometry::Type gtype = fe.GetGeomType();
    const IntegrationRule &ir = IntRule ? *IntRule : IntRules.Get(gtype, qorder);
-   const int nq = ir.GetNPoints();
    Mesh &mesh = *fes.GetMesh();
-   const int nbe = mesh.GetNFbyType(FaceType::Boundary);
-   const int dim = mesh.Dimension();
-   const DofToQuad &maps = fe.GetDofToQuad(ir, DofToQuad::TENSOR);
 
-   Vector coeff;
-   if (ConstantCoefficient *cQ =
-          dynamic_cast<ConstantCoefficient*>(&Q))
-   {
-      coeff.SetSize(1);
-      coeff(0) = cQ->constant;
-   }
-   else
-   {
-      coeff.SetSize(nq * nbe);
-      auto C = Reshape(coeff.HostWrite(), nq, nbe);
-      int f_ind = 0;
-      for (int f = 0; f < fes.GetNF(); ++f)
-      {
-         const Mesh::FaceInformation face = mesh.GetFaceInformation(f);
-         if (!face.IsBoundary()) { continue; }
-         ElementTransformation &Tr = *mesh.GetFaceElementTransformations(f);
-         for (int q = 0; q < nq; ++q)
-         {
-            int iq = ToLexOrdering(dim, face.element[0].local_face_id,
-                                   maps.nqpt, q);
-            const IntegrationPoint &ip = ir[iq];
-            Tr.SetIntPoint(&ip);
-            C(q,f_ind) = Q.Eval(Tr, ip);
-         }
-         f_ind++;
-      }
-   }
+   FaceQuadratureSpace qs(mesh, ir, FaceType::Boundary);
+   CoefficientVector coeff(Q, qs, CoefficientStorage::COMPRESSED);
    BLFEvalAssemble(fes, ir, markers, coeff, false, b);
 }
 
@@ -261,41 +231,10 @@ void BoundaryNormalLFIntegrator::AssembleDevice(const FiniteElementSpace &fes,
    const int qorder = oa * fe.GetOrder() + ob;
    const Geometry::Type gtype = fe.GetGeomType();
    const IntegrationRule &ir = IntRule ? *IntRule : IntRules.Get(gtype, qorder);
-   const int nq = ir.GetNPoints();
    Mesh &mesh = *fes.GetMesh();
-   const int nbe = mesh.GetNFbyType(FaceType::Boundary);
-   const int dim = mesh.Dimension();
-   const DofToQuad &maps = fe.GetDofToQuad(ir, DofToQuad::TENSOR);
 
-   Vector coeff;
-   if (const auto *cQ = dynamic_cast<VectorConstantCoefficient*>(&Q))
-   {
-      coeff = cQ->GetVec();
-   }
-   else
-   {
-
-      Vector coeff_val(dim);
-      coeff.SetSize(dim * nq * nbe);
-      auto C = Reshape(coeff.HostWrite(), dim, nq, nbe);
-      int f_ind = 0;
-      for (int f = 0; f < fes.GetNF(); ++f)
-      {
-         const Mesh::FaceInformation face = mesh.GetFaceInformation(f);
-         if (!face.IsBoundary()) { continue; }
-         ElementTransformation &Tr = *mesh.GetFaceElementTransformations(f);
-         for (int q = 0; q < nq; ++q)
-         {
-            int iq = ToLexOrdering(dim, face.element[0].local_face_id,
-                                   maps.nqpt, q);
-            const IntegrationPoint &ip = ir[iq];
-            Tr.SetIntPoint(&ip);
-            Q.Eval(coeff_val, Tr, ip);
-            for (int d=0; d<dim; ++d) { C(d,q,f_ind) = coeff_val[d]; }
-         }
-         f_ind++;
-      }
-   }
+   FaceQuadratureSpace qs(mesh, ir, FaceType::Boundary);
+   CoefficientVector coeff(Q, qs, CoefficientStorage::COMPRESSED);
    BLFEvalAssemble(fes, ir, markers, coeff, true, b);
 }
 

fem/lininteg_domain.cpp

@@ -19,13 +19,13 @@ namespace mfem
 template<int T_D1D = 0, int T_Q1D = 0> static
 void DLFEvalAssemble2D(const int vdim, const int ne, const int d, const int q,
                        const int map_type, const int *markers, const double *b,
-                       const double *j, const double *weights,
+                       const double *detj, const double *weights,
                        const Vector &coeff, double *y)
 {
    const auto F = coeff.Read();
    const auto M = Reshape(markers, ne);
    const auto B = Reshape(b, q, d);
-   const auto J = Reshape(j, q, q, 2,2, ne);
+   const auto DETJ = Reshape(detj, q, q, ne);
    const auto W = Reshape(weights, q, q);
    const bool cst = coeff.Size() == vdim;
    const auto C = cst ? Reshape(F,vdim,1,1,1) : Reshape(F,vdim,q,q,ne);
@@ -55,19 +55,7 @@ void DLFEvalAssemble2D(const int vdim, const int ne, const int d, const int q,
          {
             MFEM_FOREACH_THREAD(y,y,q)
             {
-               double detJ;
-               if (map_type == FiniteElement::VALUE)
-               {
-                  const double J11 = J(x,y,0,0,e);
-                  const double J21 = J(x,y,1,0,e);
-                  const double J12 = J(x,y,0,1,e);
-                  const double J22 = J(x,y,1,1,e);
-                  detJ = J11 * J22 - J21 * J12;
-               }
-               else
-               {
-                  detJ = 1.0;
-               }
+               const double detJ = (map_type == FiniteElement::VALUE) ? DETJ(x,y,e) : 1.0;
                const double coeff_val = cst ? cst_val : C(c,x,y,e);
                QQ(y,x) = W(x,y) * coeff_val * detJ;
             }
@@ -100,13 +88,13 @@ void DLFEvalAssemble2D(const int vdim, const int ne, const int d, const int q,
 template<int T_D1D = 0, int T_Q1D = 0> static
 void DLFEvalAssemble3D(const int vdim, const int ne, const int d, const int q,
                        const int map_type, const int *markers, const double *b,
-                       const double *j, const double *weights,
+                       const double *detj, const double *weights,
                        const Vector &coeff, double *y)
 {
    const auto F = coeff.Read();
    const auto M = Reshape(markers, ne);
    const auto B = Reshape(b, q,d);
-   const auto J = Reshape(j, q,q,q, 3,3, ne);
+   const auto DETJ = Reshape(detj, q, q, q, ne);
    const auto W = Reshape(weights, q,q,q);
    const bool cst_coeff = coeff.Size() == vdim;
    const auto C = cst_coeff ? Reshape(F,vdim,1,1,1,1):Reshape(F,vdim,q,q,q,ne);
@@ -138,26 +126,7 @@ void DLFEvalAssemble3D(const int vdim, const int ne, const int d, const int q,
             {
                for (int z = 0; z < q; ++z)
                {
-                  double detJ;
-                  if (map_type == FiniteElement::VALUE)
-                  {
-                     const double J11 = J(x,y,z,0,0,e);
-                     const double J21 = J(x,y,z,1,0,e);
-                     const double J31 = J(x,y,z,2,0,e);
-                     const double J12 = J(x,y,z,0,1,e);
-                     const double J22 = J(x,y,z,1,1,e);
-                     const double J32 = J(x,y,z,2,1,e);
-                     const double J13 = J(x,y,z,0,2,e);
-                     const double J23 = J(x,y,z,1,2,e);
-                     const double J33 = J(x,y,z,2,2,e);
-                     detJ = J11 * (J22 * J33 - J32 * J23) -
-                     /* */  J21 * (J12 * J33 - J32 * J13) +
-                     /* */  J31 * (J12 * J23 - J22 * J13);
-                  }
-                  else
-                  {
-                     detJ = 1.0;
-                  }
+                  const double detJ = (map_type == FiniteElement::VALUE) ? DETJ(x,y,z,e) : 1.0;
                   const double coeff_val = cst_coeff ? cst_val : C(c,x,y,z,e);
                   QQQ(z,y,x) = W(x,y,z) * coeff_val * detJ;
                }
@@ -222,7 +191,7 @@ static void DLFEvalAssemble(const FiniteElementSpace &fes,
    const MemoryType mt = Device::GetDeviceMemoryType();
    const DofToQuad &maps = el.GetDofToQuad(*ir, DofToQuad::TENSOR);
    const int d = maps.ndof, q = maps.nqpt;
-   constexpr int flags = GeometricFactors::JACOBIANS;
+   constexpr int flags = GeometricFactors::DETERMINANTS;
    const GeometricFactors *geom = mesh->GetGeometricFactors(*ir, flags, mt);
    const int map_type = fes.GetFE(0)->GetMapType();
    decltype(&DLFEvalAssemble2D<>) ker =
@@ -260,10 +229,10 @@ static void DLFEvalAssemble(const FiniteElementSpace &fes,
    const int ne = fes.GetMesh()->GetNE();
    const int *M = markers.Read();
    const double *B = maps.B.Read();
-   const double *J = geom->J.Read();
+   const double *detJ = geom->detJ.Read();
    const double *W = ir->GetWeights().Read();
    double *Y = y.ReadWrite();
-   ker(vdim, ne, d, q, map_type, M, B, J, W, coeff, Y);
+   ker(vdim, ne, d, q, map_type, M, B, detJ, W, coeff, Y);
 }
 
 void DomainLFIntegrator::AssembleDevice(const FiniteElementSpace &fes,

fem/lor/lor_batched.cpp

@@ -91,9 +91,7 @@ void BatchedLORAssembly::FormLORVertexCoordinates(FiniteElementSpace &fes_ho,
    Vector nodal_evec(nodal_restriction->Height());
    nodal_restriction->Mult(*nodal_gf, nodal_evec);
 
-   IntegrationRules irs(0, Quadrature1D::GaussLobatto);
-   Geometry::Type geom = mesh_ho.GetElementGeometry(0);
-   const IntegrationRule &ir = irs.Get(geom, 2*nd1d - 3);
+   IntegrationRule ir = GetCollocatedIntRule(fes_ho);
 
    // Map from nodal E-vector to Q-vector at the LOR vertex points
    X_vert.SetSize(dim*ndof_per_el*nel_ho);
@@ -493,4 +491,12 @@ BatchedLORAssembly::BatchedLORAssembly(FiniteElementSpace &fes_ho_)
    FormLORVertexCoordinates(fes_ho, X_vert);
 }
 
+IntegrationRule GetCollocatedIntRule(FiniteElementSpace &fes)
+{
+   IntegrationRules irs(0, Quadrature1D::GaussLobatto);
+   const Geometry::Type geom = fes.GetMesh()->GetElementGeometry(0);
+   const int nd1d = fes.GetMaxElementOrder() + 1;
+   return irs.Get(geom, 2*nd1d - 3);
+}
+
 } // namespace mfem

fem/lor/lor_batched.hpp

@@ -13,6 +13,7 @@
 #define MFEM_LOR_BATCHED
 
 #include "lor.hpp"
+#include "../qspace.hpp"
 
 namespace mfem
 {
@@ -143,6 +144,25 @@ static T *GetIntegrator(BilinearForm &a)
    return nullptr;
 }
 
+IntegrationRule GetCollocatedIntRule(FiniteElementSpace &fes);
+
+template <typename INTEGRATOR>
+void ProjectLORCoefficient(BilinearForm &a, CoefficientVector &coeff_vector)
+{
+   INTEGRATOR *i = GetIntegrator<INTEGRATOR>(a);
+   if (i)
+   {
+      // const_cast since Coefficient::Eval is not const...
+      auto *coeff = const_cast<Coefficient*>(i->GetCoefficient());
+      if (coeff) { coeff_vector.Project(*coeff); }
+      else { coeff_vector.SetConstant(1.0); }
+   }
+   else
+   {
+      coeff_vector.SetConstant(0.0);
+   }
+}
+
 /// Abstract base class for the batched LOR assembly kernels.
 class BatchedLORKernel
 {
@@ -151,12 +171,18 @@ protected:
    Vector &X_vert; ///< Mesh coordinate vector.
    Vector &sparse_ij; ///< Local element sparsity matrix data.
    Array<int> &sparse_mapping; ///< Local element sparsity pattern.
+   IntegrationRule ir; ///< Collocated integration rule.
+   QuadratureSpace qs; ///< Quadrature space for coefficients.
+   CoefficientVector c1; ///< Coefficient of first integrator.
+   CoefficientVector c2; ///< Coefficient of second integrator.
    BatchedLORKernel(FiniteElementSpace &fes_ho_,
                     Vector &X_vert_,
                     Vector &sparse_ij_,
                     Array<int> &sparse_mapping_)
       : fes_ho(fes_ho_), X_vert(X_vert_), sparse_ij(sparse_ij_),
-        sparse_mapping(sparse_mapping_)
+        sparse_mapping(sparse_mapping_), ir(GetCollocatedIntRule(fes_ho)),
+        qs(*fes_ho.GetMesh(), ir), c1(qs, CoefficientStorage::COMPRESSED),
+        c2(qs, CoefficientStorage::COMPRESSED)
    { }
 };
 

fem/lor/lor_h1.cpp

@@ -30,8 +30,14 @@ void BatchedLOR_H1::Assemble2D()
    static constexpr int nnz_per_row = 9;
    static constexpr int sz_local_mat = nv*nv;
 
-   const double DQ = diffusion_coeff;
-   const double MQ = mass_coeff;
+   const bool const_mq = c1.Size() == 1;
+   const auto MQ = const_mq
+                   ? Reshape(c1.Read(), 1, 1, 1)
+                   : Reshape(c1.Read(), nd1d, nd1d, nel_ho);
+   const bool const_dq = c2.Size() == 1;
+   const auto DQ = const_dq
+                   ? Reshape(c2.Read(), 1, 1, 1)
+                   : Reshape(c2.Read(), nd1d, nd1d, nel_ho);
 
    sparse_ij.SetSize(nnz_per_row*ndof_per_el*nel_ho);
    auto V = Reshape(sparse_ij.Write(), nnz_per_row, nd1d, nd1d, nel_ho);
@@ -97,6 +103,8 @@ void BatchedLOR_H1::Assemble2D()
             {
                for (int iqy=0; iqy<2; ++iqy)
                {
+                  const double mq = const_mq ? MQ(0,0,0) : MQ(kx+iqx, ky+iqy, iel_ho);
+                  const double dq = const_dq ? DQ(0,0,0) : DQ(kx+iqx, ky+iqy, iel_ho);
                   for (int jy=0; jy<2; ++jy)
                   {
                      const double bjy = (jy == iqy) ? 1.0 : 0.0;
@@ -133,9 +141,9 @@ void BatchedLOR_H1::Assemble2D()
                               val += dix*djx*Q(0,iqy,iqx);
                               val += (dix*djy + diy*djx)*Q(1,iqy,iqx);
                               val += diy*djy*Q(2,iqy,iqx);
-                              val *= DQ;
+                              val *= dq;
 
-                              val += MQ*bix*biy*bjx*bjy*Q(3,iqy,iqx);
+                              val += mq*bix*biy*bjx*bjy*Q(3,iqy,iqx);
 
                               local_mat(ii_loc, jj_loc) += val;
                            }
@@ -201,10 +209,6 @@ template <int ORDER>
 void BatchedLOR_H1::Assemble3D()
 {
    const int nel_ho = fes_ho.GetNE();
-
-   const double DQ = diffusion_coeff;
-   const double MQ = mass_coeff;
-
    static constexpr int nv = 8;
    static constexpr int dim = 3;
    static constexpr int ddm2 = (dim*(dim+1))/2;
@@ -217,6 +221,15 @@ void BatchedLOR_H1::Assemble3D()
    static constexpr int sz_mass_B = sz_mass_A*2;
    static constexpr int sz_local_mat = nv*nv;
 
+   const bool const_mq = c1.Size() == 1;
+   const auto MQ = const_mq
+                   ? Reshape(c1.Read(), 1, 1, 1, 1)
+                   : Reshape(c1.Read(), nd1d, nd1d, nd1d, nel_ho);
+   const bool const_dq = c2.Size() == 1;
+   const auto DQ = const_dq
+                   ? Reshape(c2.Read(), 1, 1, 1, 1)
+                   : Reshape(c2.Read(), nd1d, nd1d, nd1d, nel_ho);
+
    sparse_ij.SetSize(nel_ho*ndof_per_el*nnz_per_row);
    auto V = Reshape(sparse_ij.Write(), nnz_per_row, nd1d, nd1d, nd1d, nel_ho);
 
@@ -288,7 +301,6 @@ void BatchedLOR_H1::Assemble3D()
                      //MFEM_UNROLL(2)
                      for (int iqx=0; iqx<2; ++iqx)
                      {
-
                         const double x = iqx;
                         const double y = iqy;
                         const double z = iqz;
@@ -335,6 +347,9 @@ void BatchedLOR_H1::Assemble3D()
                            //MFEM_UNROLL(2)
                            for (int iqz=0; iqz<2; ++iqz)
                            {
+                              const double mq = const_mq ? MQ(0,0,0,0) : MQ(kx+iqx, ky+iqy, kz+iqz, iel_ho);
+                              const double dq = const_dq ? DQ(0,0,0,0) : DQ(kx+iqx, ky+iqy, kz+iqz, iel_ho);
+
                               const double biz = (iz == iqz) ? 1.0 : 0.0;
                               const double giz = (iz == 0) ? -1.0 : 1.0;
 
@@ -351,18 +366,18 @@ void BatchedLOR_H1::Assemble3D()
                               const double J23 = J32;
                               const double J33 = Q(5,iqz,iqy,iqx);
 
-                              grad_A(0,0,iqy,iz,jz,iqx) += J11*biz*bjz;
-                              grad_A(1,0,iqy,iz,jz,iqx) += J21*biz*bjz;
-                              grad_A(2,0,iqy,iz,jz,iqx) += J31*giz*bjz;
-                              grad_A(0,1,iqy,iz,jz,iqx) += J12*biz*bjz;
-                              grad_A(1,1,iqy,iz,jz,iqx) += J22*biz*bjz;
-                              grad_A(2,1,iqy,iz,jz,iqx) += J32*giz*bjz;
-                              grad_A(0,2,iqy,iz,jz,iqx) += J13*biz*gjz;
-                              grad_A(1,2,iqy,iz,jz,iqx) += J23*biz*gjz;
-                              grad_A(2,2,iqy,iz,jz,iqx) += J33*giz*gjz;
+                              grad_A(0,0,iqy,iz,jz,iqx) += dq*J11*biz*bjz;
+                              grad_A(1,0,iqy,iz,jz,iqx) += dq*J21*biz*bjz;
+                              grad_A(2,0,iqy,iz,jz,iqx) += dq*J31*giz*bjz;
+                              grad_A(0,1,iqy,iz,jz,iqx) += dq*J12*biz*bjz;
+                              grad_A(1,1,iqy,iz,jz,iqx) += dq*J22*biz*bjz;
+                              grad_A(2,1,iqy,iz,jz,iqx) += dq*J32*giz*bjz;
+                              grad_A(0,2,iqy,iz,jz,iqx) += dq*J13*biz*gjz;
+                              grad_A(1,2,iqy,iz,jz,iqx) += dq*J23*biz*gjz;
+                              grad_A(2,2,iqy,iz,jz,iqx) += dq*J33*giz*gjz;
 
                               double wdetJ = Q(6,iqz,iqy,iqx);
-                              mass_A(iqy,iz,jz,iqx) += wdetJ*biz*bjz;
+                              mass_A(iqy,iz,jz,iqx) += mq*wdetJ*biz*bjz;
                            }
                            //MFEM_UNROLL(2)
                            for (int jy=0; jy<2; ++jy)
@@ -426,9 +441,7 @@ void BatchedLOR_H1::Assemble3D()
                                     val += bix*bjx*grad_B(2,2,iy,jy,iz,jz,iqx);
                                     val += bix*bjx*grad_B(1,2,iy,jy,iz,jz,iqx);
 
-                                    val *= DQ;
-
-                                    val += MQ*bix*bjx*mass_B(iy,jy,iz,jz,iqx);
+                                    val += bix*bjx*mass_B(iy,jy,iz,jz,iqx);
 
                                     local_mat(ii_loc, jj_loc) += val;
                                  }
@@ -531,29 +544,8 @@ BatchedLOR_H1::BatchedLOR_H1(BilinearForm &a,
                              Array<int> &sparse_mapping_)
    : BatchedLORKernel(fes_ho_, X_vert_, sparse_ij_, sparse_mapping_)
 {
-   MassIntegrator *mass = GetIntegrator<MassIntegrator>(a);
-   DiffusionIntegrator *diffusion = GetIntegrator<DiffusionIntegrator>(a);
-
-   if (mass != nullptr)
-   {
-      auto *coeff = dynamic_cast<const ConstantCoefficient*>(mass->GetCoefficient());
-      mass_coeff = coeff ? coeff->constant : 1.0;
-   }
-   else
-   {
-      mass_coeff = 0.0;
-   }
-
-   if (diffusion != nullptr)
-   {
-      auto *coeff = dynamic_cast<const ConstantCoefficient*>
-                    (diffusion->GetCoefficient());
-      diffusion_coeff = coeff ? coeff->constant : 1.0;
-   }
-   else
-   {
-      diffusion_coeff = 0.0;
-   }
+   ProjectLORCoefficient<MassIntegrator>(a, c1);
+   ProjectLORCoefficient<DiffusionIntegrator>(a, c2);
 }
 
 } // namespace mfem

fem/lor/lor_h1.hpp

@@ -21,9 +21,6 @@ namespace mfem
 // classes BatchedLORAssembly and BatchedLORKernel .
 class BatchedLOR_H1 : BatchedLORKernel
 {
-protected:
-   // TODO: for now only supporting constant coefficients
-   double mass_coeff, diffusion_coeff;
 public:
    template <int ORDER> void Assemble2D();
    template <int ORDER> void Assemble3D();

fem/lor/lor_nd.cpp

@@ -33,8 +33,14 @@ void BatchedLOR_ND::Assemble2D()
    static constexpr int nnz_per_row = 7;
    static constexpr int sz_local_mat = ne*ne;
 
-   const double DQ = curl_curl_coeff;
-   const double MQ = mass_coeff;
+   const bool const_mq = c1.Size() == 1;
+   const auto MQ = const_mq
+                   ? Reshape(c1.Read(), 1, 1, 1)
+                   : Reshape(c1.Read(), op1, op1, nel_ho);
+   const bool const_dq = c2.Size() == 1;
+   const auto DQ = const_dq
+                   ? Reshape(c2.Read(), 1, 1, 1)
+                   : Reshape(c2.Read(), op1, op1, nel_ho);
 
    sparse_ij.SetSize(nnz_per_row*ndof_per_el*nel_ho);
    auto V = Reshape(sparse_ij.Write(), nnz_per_row, o*op1, dim, nel_ho);
@@ -106,6 +112,8 @@ void BatchedLOR_ND::Assemble2D()
             {
                for (int iqy=0; iqy<2; ++iqy)
                {
+                  const double mq = const_mq ? MQ(0,0,0) : MQ(kx+iqx, ky+iqy, iel_ho);
+                  const double dq = const_dq ? DQ(0,0,0) : DQ(kx+iqx, ky+iqy, iel_ho);
                   // Loop over x,y components. c=0 => x, c=1 => y
                   for (int cj=0; cj<dim; ++cj)
                   {
@@ -136,8 +144,8 @@ void BatchedLOR_ND::Assemble2D()
                               val += byi*bxj*Q(1,iqy,iqx);
                               val += bxi*byj*Q(1,iqy,iqx);
                               val += byi*byj*Q(2,iqy,iqx);
-                              val *= MQ;
-                              val += DQ*curl_i*curl_j*Q(3,iqy,iqx);
+                              val *= mq;
+                              val += dq*curl_i*curl_j*Q(3,iqy,iqx);
 
                               local_mat(ii_loc, jj_loc) += val;
                            }
@@ -224,8 +232,14 @@ void BatchedLOR_ND::Assemble3D()
    static constexpr int nnz_per_row = 33;
    static constexpr int sz_local_mat = ne*ne;
 
-   const double DQ = curl_curl_coeff;
-   const double MQ = mass_coeff;
+   const bool const_mq = c1.Size() == 1;
+   const auto MQ = const_mq
+                   ? Reshape(c1.Read(), 1, 1, 1, 1)
+                   : Reshape(c1.Read(), op1, op1, op1, nel_ho);
+   const bool const_dq = c2.Size() == 1;
+   const auto DQ = const_dq
+                   ? Reshape(c2.Read(), 1, 1, 1, 1)
+                   : Reshape(c2.Read(), op1, op1, op1, nel_ho);
 
    sparse_ij.SetSize(nnz_per_row*ndof_per_el*nel_ho);
    auto V = Reshape(sparse_ij.Write(), nnz_per_row, o*op1*op1, dim, nel_ho);
@@ -318,6 +332,8 @@ void BatchedLOR_ND::Assemble3D()
                   {
                      for (int iqx=0; iqx<2; ++iqx)
                      {
+                        const double mq = const_mq ? MQ(0,0,0,0) : MQ(kx+iqx, ky+iqy, kz+iqz, iel_ho);
+                        const double dq = const_dq ? DQ(0,0,0,0) : DQ(kx+iqx, ky+iqy, kz+iqz, iel_ho);
                         // Loop over x,y,z components. 0 => x, 1 => y, 2 => z
                         for (int cj=0; cj<dim; ++cj)
                         {
@@ -391,7 +407,7 @@ void BatchedLOR_ND::Assemble3D()
                                     basis_basis += Q(4,iqz,iqy,iqx)*(basis_i[1]*basis_j[2] + basis_i[2]*basis_j[1]);
                                     basis_basis += Q(5,iqz,iqy,iqx)*basis_i[2]*basis_j[2];
 
-                                    const double val = DQ*curl_curl + MQ*basis_basis;
+                                    const double val = dq*curl_curl + mq*basis_basis;
 
                                     local_mat(ii_loc, jj_loc) += val;
                                  }
@@ -572,28 +588,8 @@ BatchedLOR_ND::BatchedLOR_ND(BilinearForm &a,
                              Array<int> &sparse_mapping_)
    : BatchedLORKernel(fes_ho_, X_vert_, sparse_ij_, sparse_mapping_)
 {
-   VectorFEMassIntegrator *mass = GetIntegrator<VectorFEMassIntegrator>(a);
-   if (mass != nullptr)
-   {
-      auto *coeff = dynamic_cast<const ConstantCoefficient*>(mass->GetCoefficient());
-      mass_coeff = coeff ? coeff->constant : 1.0;
-   }
-   else
-   {
-      mass_coeff = 0.0;
-   }
-
-   CurlCurlIntegrator *diffusion = GetIntegrator<CurlCurlIntegrator>(a);
-   if (diffusion != nullptr)
-   {
-      auto *coeff = dynamic_cast<const ConstantCoefficient*>
-                    (diffusion->GetCoefficient());
-      curl_curl_coeff = coeff ? coeff->constant : 1.0;
-   }
-   else
-   {
-      curl_curl_coeff = 0.0;
-   }
+   ProjectLORCoefficient<VectorFEMassIntegrator>(a, c1);
+   ProjectLORCoefficient<CurlCurlIntegrator>(a, c2);
 }
 
 } // namespace mfem

fem/lor/lor_nd.hpp

@@ -21,8 +21,6 @@ namespace mfem
 // classes BatchedLORAssembly and BatchedLORKernel .
 class BatchedLOR_ND : BatchedLORKernel
 {
-protected:
-   double mass_coeff, curl_curl_coeff;
 public:
    template <int ORDER> void Assemble2D();
    template <int ORDER> void Assemble3D();

fem/lor/lor_rt.cpp

@@ -33,8 +33,14 @@ void BatchedLOR_RT::Assemble2D()
    static constexpr int nnz_per_row = 7;
    static constexpr int sz_local_mat = ne*ne;
 
-   const double DQ = div_div_coeff;
-   const double MQ = mass_coeff;
+   const bool const_mq = c1.Size() == 1;
+   const auto MQ = const_mq
+                   ? Reshape(c1.Read(), 1, 1, 1)
+                   : Reshape(c1.Read(), op1, op1, nel_ho);
+   const bool const_dq = c2.Size() == 1;
+   const auto DQ = const_dq
+                   ? Reshape(c2.Read(), 1, 1, 1)
+                   : Reshape(c2.Read(), op1, op1, nel_ho);
 
    sparse_ij.SetSize(nnz_per_row*ndof_per_el*nel_ho);
    auto V = Reshape(sparse_ij.Write(), nnz_per_row, o*op1, dim, nel_ho);
@@ -102,6 +108,8 @@ void BatchedLOR_RT::Assemble2D()
             {
                for (int iqy=0; iqy<2; ++iqy)
                {
+                  const double mq = const_mq ? MQ(0,0,0) : MQ(kx+iqx, ky+iqy, iel_ho);
+                  const double dq = const_dq ? DQ(0,0,0) : DQ(kx+iqx, ky+iqy, iel_ho);
                   // Loop over x,y components. c=0 => x, c=1 => y
                   for (int cj=0; cj<dim; ++cj)
                   {
@@ -132,8 +140,8 @@ void BatchedLOR_RT::Assemble2D()
                               val += byi*bxj*Q(1,iqy,iqx);
                               val += bxi*byj*Q(1,iqy,iqx);
                               val += byi*byj*Q(2,iqy,iqx);
-                              val *= MQ;
-                              val += DQ*div_j*div_i*Q(3,iqy,iqx);
+                              val *= mq;
+                              val += dq*div_j*div_i*Q(3,iqy,iqx);
 
                               local_mat(ii_loc, jj_loc) += val;
                            }
@@ -241,8 +249,14 @@ void BatchedLOR_RT::Assemble3D()
    static constexpr int nnz_per_row = 11;
    static constexpr int sz_local_mat = nf*nf;
 
-   const double DQ = div_div_coeff;
-   const double MQ = mass_coeff;
+   const bool const_mq = c1.Size() == 1;
+   const auto MQ = const_mq
+                   ? Reshape(c1.Read(), 1, 1, 1, 1)
+                   : Reshape(c1.Read(), op1, op1, op1, nel_ho);
+   const bool const_dq = c2.Size() == 1;
+   const auto DQ = const_dq
+                   ? Reshape(c2.Read(), 1, 1, 1, 1)
+                   : Reshape(c2.Read(), op1, op1, op1, nel_ho);
 
    sparse_ij.SetSize(nnz_per_row*ndof_per_el*nel_ho);
    auto V = Reshape(sparse_ij.Write(), nnz_per_row, o*o*op1, dim, nel_ho);
@@ -323,6 +337,8 @@ void BatchedLOR_RT::Assemble3D()
                   {
                      for (int iqx=0; iqx<2; ++iqx)
                      {
+                        const double mq = const_mq ? MQ(0,0,0,0) : MQ(kx+iqx, ky+iqy, kz+iqz, iel_ho);
+                        const double dq = const_dq ? DQ(0,0,0,0) : DQ(kx+iqx, ky+iqy, kz+iqz, iel_ho);
                         // Loop over x,y,z components. 0 => x, 1 => y, 2 => z
                         for (int cj=0; cj<dim; ++cj)
                         {
@@ -376,7 +392,7 @@ void BatchedLOR_RT::Assemble3D()
                                     basis_basis += Q(4,iqz,iqy,iqx)*(basis_i[1]*basis_j[2] + basis_i[2]*basis_j[1]);
                                     basis_basis += Q(5,iqz,iqy,iqx)*basis_i[2]*basis_j[2];
 
-                                    const double val = DQ*div_div + MQ*basis_basis;
+                                    const double val = dq*div_div + mq*basis_basis;
                                     // const double val = 1.0;
 
                                     local_mat(ii_loc, jj_loc) += val;
@@ -556,26 +572,8 @@ BatchedLOR_RT::BatchedLOR_RT(BilinearForm &a,
                              Array<int> &sparse_mapping_)
    : BatchedLORKernel(fes_ho_, X_vert_, sparse_ij_, sparse_mapping_)
 {
-   if (VectorFEMassIntegrator *mass = GetIntegrator<VectorFEMassIntegrator>(a))
-   {
-      auto *coeff = dynamic_cast<const ConstantCoefficient*>(mass->GetCoefficient());
-      mass_coeff = coeff ? coeff->constant : 1.0;
-   }
-   else
-   {
-      mass_coeff = 0.0;
-   }
-
-   if (DivDivIntegrator *divdiv = GetIntegrator<DivDivIntegrator>(a))
-   {
-      auto *coeff = dynamic_cast<const ConstantCoefficient*>
-                    (divdiv->GetCoefficient());
-      div_div_coeff = coeff ? coeff->constant : 1.0;
-   }
-   else
-   {
-      div_div_coeff = 0.0;
-   }
+   ProjectLORCoefficient<VectorFEMassIntegrator>(a, c1);
+   ProjectLORCoefficient<DivDivIntegrator>(a, c2);
 }
 
 } // namespace mfem

fem/lor/lor_rt.hpp

@@ -21,8 +21,6 @@ namespace mfem
 // classes BatchedLORAssembly and BatchedLORKernel .
 class BatchedLOR_RT : BatchedLORKernel
 {
-protected:
-   double mass_coeff, div_div_coeff;
 public:
    template <int ORDER> void Assemble2D();
    template <int ORDER> void Assemble3D();

fem/moonolith/mortarintegrator.hpp

@@ -19,7 +19,7 @@ namespace mfem
 
 /*!
  * @brief Interface for mortar element assembly.
- * The MortarIntegrator interface is used for performing Pertrov-Galerkin
+ * The MortarIntegrator interface is used for performing Petrov-Galerkin
  * finite element assembly on intersections between elements.
  * The quadrature rules are to be generated by a cut algorithm (e.g.,
  * mfem::Cut). The quadrature rules are defined in the respective trial and test

fem/pbilinearform.cpp

@@ -403,7 +403,6 @@ void ParBilinearForm::FormLinearSystem(
       P.MultTranspose(b, true_B);
       R.Mult(x, true_X);
       p_mat.EliminateBC(p_mat_e, ess_tdof_list, true_X, true_B);
-      R.EnsureMultTranspose();
       R.MultTranspose(true_B, b);
       hybridization->ReduceRHS(true_B, B);
       X.SetSize(B.Size());

fem/pfespace.cpp

@@ -959,10 +959,6 @@ void ParFiniteElementSpace::Build_Dof_TrueDof_Matrix() const // matrix P
    SparseMatrix Pdiag;
    P->GetDiag(Pdiag);
    R = Transpose(Pdiag);
-
-   // The following call ensures that the action of the transpose of P is
-   // performed fast when HYPRE is built for GPUs.
-   P->EnsureMultTranspose();
 }
 
 HypreParMatrix *ParFiniteElementSpace::GetPartialConformingInterpolation()
@@ -2610,7 +2606,7 @@ int ParFiniteElementSpace
       if (dump < 10)
       {
          char fname[100];
-         sprintf(fname, "dofs%02d.txt", MyRank);
+         snprintf(fname, 100, "dofs%02d.txt", MyRank);
          std::ofstream f(fname);
          DebugDumpDOFs(f, deps, dof_group, dof_owner, finalized);
          dump++;
@@ -2628,10 +2624,6 @@ int ParFiniteElementSpace
    {
       *P_ = MakeVDimHypreMatrix(pmatrix, ndofs, num_true_dofs,
                                 dof_offs, tdof_offs);
-
-      // The following call ensures that the action of the transpose of *P_ is
-      // performed fast when HYPRE is built for GPUs.
-      (*P_)->EnsureMultTranspose();
    }
 
    // clean up possible remaining messages in the queue to avoid receiving

fem/plinearform.cpp

@@ -43,18 +43,9 @@ void ParLinearForm::MakeRef(ParFiniteElementSpace *pf, Vector &v, int v_offset)
    pfes = pf;
 }
 
-void ParLinearForm::Assemble(bool use_device)
+void ParLinearForm::Assemble()
 {
-   bool all_supports_device = use_device;
-
-   if (use_device)
-   {
-      bool supports_device = SupportsDevice();
-      MPI_Allreduce(&supports_device, &all_supports_device, 1,
-                    MPI_C_BOOL, MPI_LAND, pfes->GetComm());
-   }
-
-   LinearForm::Assemble(all_supports_device);
+   LinearForm::Assemble();
 
    if (interior_face_integs.Size())
    {
@@ -63,6 +54,14 @@ void ParLinearForm::Assemble(bool use_device)
    }
 }
 
+bool ParLinearForm::SupportsDevice()
+{
+   bool parallel;
+   bool local = LinearForm::SupportsDevice();
+   MPI_Allreduce(&local, &parallel, 1, MPI_C_BOOL, MPI_LAND, pfes->GetComm());
+   return parallel;
+}
+
 void ParLinearForm::AssembleSharedFaces()
 {
    Array<int> vdofs;

fem/plinearform.hpp

@@ -114,9 +114,13 @@ public:
    void MakeRef(ParFiniteElementSpace *pf, Vector &v, int v_offset);
 
    /// Assembles the ParLinearForm i.e. sums over all domain/bdr integrators.
-   /// When @a use_device is set to true and all the ParLinearForm assembly is
-   /// compatible with device execution, it will be executed on the device.
-   void Assemble(bool use_device = true);
+   /** When @ref LinearForm::UseFastAssembly "UseFastAssembly(true)" has been
+       called and the linear form assembly is compatible with device execution,
+       the assembly will be executed on the device. */
+   void Assemble();
+
+   /// Return true if assembly on device is supported, false otherwise.
+   virtual bool SupportsDevice();
 
    void AssembleSharedFaces();
 

fem/prestriction.cpp

@@ -42,118 +42,108 @@ ParNCH1FaceRestriction::ParNCH1FaceRestriction(const ParFiniteElementSpace &fes,
 
 void ParNCH1FaceRestriction::Mult(const Vector &x, Vector &y) const
 {
-   if (nf==0) { return; }
+   H1FaceRestriction::Mult(x, y);
+   NonconformingInterpolation(y);
+}
+
+void ParNCH1FaceRestriction::NonconformingInterpolation(Vector& y) const
+{
    // Assumes all elements have the same number of dofs
    const int nface_dofs = face_dofs;
    const int vd = vdim;
-   const bool t = byvdim;
-
-   if ( type==FaceType::Boundary )
+   auto d_y = Reshape(y.ReadWrite(), nface_dofs, vd, nf);
+   auto &nc_interp_config = interpolations.GetNCFaceInterpConfig();
+   const int num_nc_faces = nc_interp_config.Size();
+   if ( num_nc_faces == 0 ) { return; }
+   auto interp_config_ptr = nc_interp_config.Read();
+   const int nc_size = interpolations.GetNumInterpolators();
+   auto d_interp = Reshape(interpolations.GetInterpolators().Read(),
+                           nface_dofs, nface_dofs, nc_size);
+   static constexpr int max_nd = 16*16;
+   MFEM_VERIFY(nface_dofs<=max_nd, "Too many degrees of freedom.");
+   MFEM_FORALL_3D(nc_face, num_nc_faces, nface_dofs, 1, 1,
    {
-      auto d_indices = scatter_indices.Read();
-      auto d_x = Reshape(x.Read(), t?vd:ndofs, t?ndofs:vd);
-      auto d_y = Reshape(y.Write(), nface_dofs, vd, nf);
-      MFEM_FORALL(i, nfdofs,
+      MFEM_SHARED double dof_values[max_nd];
+      const NCInterpConfig conf = interp_config_ptr[nc_face];
+      if ( conf.is_non_conforming && conf.master_side == 0 )
       {
-         const int dof = i % nface_dofs;
-         const int face = i / nface_dofs;
-         const int idx = d_indices[i];
-         for (int c = 0; c < vd; ++c)
-         {
-            d_y(dof, c, face) = d_x(t?c:idx, t?idx:c);
-         }
-      });
-   }
-   else // type==FaceType::Interior
-   {
-      auto d_indices = scatter_indices.Read();
-      auto d_x = Reshape(x.Read(), t?vd:ndofs, t?ndofs:vd);
-      auto d_y = Reshape(y.Write(), nface_dofs, vd, nf);
-      auto interp_config_ptr = interpolations.GetFaceInterpConfig().Read();
-      auto interpolators = interpolations.GetInterpolators().Read();
-      const int nc_size = interpolations.GetNumInterpolators();
-      auto d_interp = Reshape(interpolators, nface_dofs, nface_dofs, nc_size);
-      static constexpr int max_nd = 1024;
-      MFEM_VERIFY(nface_dofs<=max_nd, "Too many degrees of freedom.");
-      MFEM_FORALL_3D(face, nf, nface_dofs, 1, 1,
-      {
-         MFEM_SHARED double dof_values[max_nd];
-         const InterpConfig conf = interp_config_ptr[face];
-         const int master_side = conf.master_side;
          const int interp_index = conf.index;
-         const int side = 0;
-         if ( !conf.is_non_conforming || side!=master_side )
+         const int face = conf.face_index;
+         for (int c = 0; c < vd; ++c)
          {
             MFEM_FOREACH_THREAD(dof,x,nface_dofs)
             {
-               const int i = face*nface_dofs + dof;
-               const int idx = d_indices[i];
-               for (int c = 0; c < vd; ++c)
-               {
-                  d_y(dof, c, face) = d_x(t?c:idx, t?idx:c);
-               }
+               dof_values[dof] = d_y(dof, c, face);
             }
-         }
-         else // Interpolation from coarse to fine
-         {
-            for (int c = 0; c < vd; ++c)
+            MFEM_SYNC_THREAD;
+            MFEM_FOREACH_THREAD(dof_out,x,nface_dofs)
             {
-               // Load the face dofs in shared memory
-               MFEM_FOREACH_THREAD(dof,x,nface_dofs)
+               double res = 0.0;
+               for (int dof_in = 0; dof_in<nface_dofs; dof_in++)
                {
-                  const int i = face*nface_dofs + dof;
-                  const int idx = d_indices[i];
-                  dof_values[dof] = d_x(t?c:idx, t?idx:c);
+                  res += d_interp(dof_out, dof_in, interp_index)*dof_values[dof_in];
                }
-               MFEM_SYNC_THREAD;
-               // Apply the interpolation to the face dofs
-               MFEM_FOREACH_THREAD(dof_out,x,nface_dofs)
-               {
-                  double res = 0.0;
-                  for (int dof_in = 0; dof_in<nface_dofs; dof_in++)
-                  {
-                     res += d_interp(dof_out, dof_in, interp_index)*
-                            dof_values[dof_in];
-                  }
-                  d_y(dof_out, c, face) = res;
-               }
-               MFEM_SYNC_THREAD;
+               d_y(dof_out, c, face) = res;
             }
+            MFEM_SYNC_THREAD;
          }
-      });
-   }
+      }
+   });
 }
 
 void ParNCH1FaceRestriction::AddMultTranspose(const Vector &x, Vector &y) const
 {
    if (nf==0) { return; }
+   NonconformingTransposeInterpolation(x);
+   H1FaceRestriction::AddMultTranspose(x_interp, y);
+}
+
+void ParNCH1FaceRestriction::AddMultTransposeInPlace(Vector &x, Vector &y) const
+{
+   if (nf==0) { return; }
+   NonconformingTransposeInterpolationInPlace(x);
+   H1FaceRestriction::AddMultTranspose(x, y);
+}
+
+void ParNCH1FaceRestriction::NonconformingTransposeInterpolation(
+   const Vector& x) const
+{
    if (x_interp.Size()==0)
    {
       x_interp.SetSize(x.Size());
    }
    x_interp = x;
+   NonconformingTransposeInterpolationInPlace(x_interp);
+}
+
+void ParNCH1FaceRestriction::NonconformingTransposeInterpolationInPlace(
+   Vector& x) const
+{
    // Assumes all elements have the same number of dofs
    const int nface_dofs = face_dofs;
    const int vd = vdim;
-   const bool t = byvdim;
    if ( type==FaceType::Interior )
    {
       // Interpolation from slave to master face dofs
-      auto d_x = Reshape(x_interp.ReadWrite(), nface_dofs, vd, nf);
-      auto interp_config_ptr = interpolations.GetFaceInterpConfig().Read();
-      auto interpolators = interpolations.GetInterpolators().Read();
+      auto d_x = Reshape(x.ReadWrite(), nface_dofs, vd, nf);
+      auto &nc_interp_config = interpolations.GetNCFaceInterpConfig();
+      const int num_nc_faces = nc_interp_config.Size();
+      if ( num_nc_faces == 0 ) { return; }
+      auto interp_config_ptr = nc_interp_config.Read();
       const int nc_size = interpolations.GetNumInterpolators();
-      auto d_interp = Reshape(interpolators, nface_dofs, nface_dofs, nc_size);
+      auto d_interp = Reshape(interpolations.GetInterpolators().Read(),
+                              nface_dofs, nface_dofs, nc_size);
       static constexpr int max_nd = 1024;
       MFEM_VERIFY(nface_dofs<=max_nd, "Too many degrees of freedom.");
-      MFEM_FORALL_3D(face, nf, nface_dofs, 1, 1,
+      MFEM_FORALL_3D(nc_face, num_nc_faces, nface_dofs, 1, 1,
       {
          MFEM_SHARED double dof_values[max_nd];
-         const InterpConfig conf = interp_config_ptr[face];
+         const NCInterpConfig conf = interp_config_ptr[nc_face];
          const int master_side = conf.master_side;
-         const int interp_index = conf.index;
          if ( conf.is_non_conforming && master_side==0 )
          {
+            const int interp_index = conf.index;
+            const int face = conf.face_index;
             // Interpolation from fine to coarse
             for (int c = 0; c < vd; ++c)
             {
@@ -176,27 +166,6 @@ void ParNCH1FaceRestriction::AddMultTranspose(const Vector &x, Vector &y) const
          }
       });
    }
-
-   // Gathering of face dofs into element dofs
-   auto d_offsets = gather_offsets.Read();
-   auto d_indices = gather_indices.Read();
-   auto d_x = Reshape(x_interp.Read(), nface_dofs, vd, nf);
-   auto d_y = Reshape(y.ReadWrite(), t?vd:ndofs, t?ndofs:vd);
-   MFEM_FORALL(i, ndofs,
-   {
-      const int offset = d_offsets[i];
-      const int next_offset = d_offsets[i + 1];
-      for (int c = 0; c < vd; ++c)
-      {
-         double dof_value = 0;
-         for (int j = offset; j < next_offset; ++j)
-         {
-            int idx_j = d_indices[j];
-            dof_value +=  d_x(idx_j % nface_dofs, c, idx_j / nface_dofs);
-         }
-         d_y(t?c:i,t?i:c) += dof_value;
-      }
-   });
 }
 
 void ParNCH1FaceRestriction::ComputeScatterIndicesAndOffsets(
@@ -264,6 +233,7 @@ void ParNCH1FaceRestriction::ComputeScatterIndicesAndOffsets(
 
    // Transform the interpolation matrix map into a contiguous memory structure.
    interpolations.LinearizeInterpolatorMapIntoVector();
+   interpolations.InitializeNCInterpConfig();
 }
 
 void ParNCH1FaceRestriction::ComputeGatherIndices(
@@ -775,110 +745,8 @@ void ParNCL2FaceRestriction::SingleValuedNonconformingMult(
 void ParNCL2FaceRestriction::DoubleValuedNonconformingMult(
    const Vector& x, Vector& y) const
 {
-   MFEM_ASSERT(
-      m == L2FaceValues::DoubleValued,
-      "This method should be called when m == L2FaceValues::DoubleValued.");
-   const ParFiniteElementSpace &pfes =
-      static_cast<const ParFiniteElementSpace&>(this->fes);
-   ParGridFunction x_gf;
-   x_gf.MakeRef(const_cast<ParFiniteElementSpace*>(&pfes),
-                const_cast<Vector&>(x), 0);
-   x_gf.ExchangeFaceNbrData();
-
-   // Assumes all elements have the same number of dofs
-   const int nface_dofs = face_dofs;
-   const int vd = vdim;
-   const bool t = byvdim;
-   const int threshold = ndofs;
-   const int nsdofs = pfes.GetFaceNbrVSize();
-   auto d_indices1 = scatter_indices1.Read();
-   auto d_indices2 = scatter_indices2.Read();
-   auto d_x = Reshape(x.Read(), t?vd:ndofs, t?ndofs:vd);
-   auto d_x_shared = Reshape(x_gf.FaceNbrData().Read(),
-                             t?vd:nsdofs, t?nsdofs:vd);
-   auto d_y = Reshape(y.Write(), nface_dofs, vd, 2, nf);
-   auto interp_config_ptr = interpolations.GetFaceInterpConfig().Read();
-   auto interpolators = interpolations.GetInterpolators().Read();
-   const int nc_size = interpolations.GetNumInterpolators();
-   auto d_interp = Reshape(interpolators, nface_dofs, nface_dofs, nc_size);
-   static constexpr int max_nd = 1024;
-   MFEM_VERIFY(nface_dofs<=max_nd, "Too many degrees of freedom.");
-   MFEM_FORALL_3D(face, nf, nface_dofs, 1, 1,
-   {
-      MFEM_SHARED double dof_values[max_nd];
-      const InterpConfig conf = interp_config_ptr[face];
-      const int master_side = conf.master_side;
-      const int interp_index = conf.index;
-      for (int side = 0; side < 2; side++)
-      {
-         if ( !conf.is_non_conforming || side!=master_side )
-         {
-            // No interpolation
-            MFEM_FOREACH_THREAD(dof,x,nface_dofs)
-            {
-               const int i = face*nface_dofs + dof;
-               const int idx = side==0 ? d_indices1[i] : d_indices2[i];
-               if (idx>-1 && idx<threshold) // local interior face
-               {
-                  for (int c = 0; c < vd; ++c)
-                  {
-                     d_y(dof, c, side, face) = d_x(t?c:idx, t?idx:c);
-                  }
-               }
-               else if (idx>=threshold) // shared interior face
-               {
-                  const int sidx = idx-threshold;
-                  for (int c = 0; c < vd; ++c)
-                  {
-                     d_y(dof, c, side, face) = d_x_shared(t?c:sidx, t?sidx:c);
-                  }
-               }
-               else // true boundary
-               {
-                  for (int c = 0; c < vd; ++c)
-                  {
-                     d_y(dof, c, side, face) = 0.0;
-                  }
-               }
-            }
-         }
-         else // Interpolation from coarse to fine
-         {
-            for (int c = 0; c < vd; ++c)
-            {
-               MFEM_FOREACH_THREAD(dof,x,nface_dofs)
-               {
-                  const int i = face*nface_dofs + dof;
-                  const int idx = side==0 ? d_indices1[i] : d_indices2[i];
-                  if (idx>-1 && idx<threshold) // local interior face
-                  {
-                     dof_values[dof] = d_x(t?c:idx, t?idx:c);
-                  }
-                  else if (idx>=threshold) // shared interior face
-                  {
-                     const int sidx = idx-threshold;
-                     dof_values[dof] = d_x_shared(t?c:sidx, t?sidx:c);
-                  }
-                  else // true boundary
-                  {
-                     dof_values[dof] = 0.0;
-                  }
-               }
-               MFEM_SYNC_THREAD;
-               MFEM_FOREACH_THREAD(dof_out,x,nface_dofs)
-               {
-                  double res = 0.0;
-                  for (int dof_in = 0; dof_in<nface_dofs; dof_in++)
-                  {
-                     res += d_interp(dof_out, dof_in, interp_index)*dof_values[dof_in];
-                  }
-                  d_y(dof_out, c, side, face) = res;
-               }
-               MFEM_SYNC_THREAD;
-            }
-         }
-      }
-   });
+   ParL2FaceRestriction::DoubleValuedConformingMult(x, y);
+   NCL2FaceRestriction::DoubleValuedNonconformingInterpolation(y);
 }
 
 void ParNCL2FaceRestriction::Mult(const Vector& x, Vector& y) const
@@ -935,6 +803,35 @@ void ParNCL2FaceRestriction::AddMultTranspose(const Vector &x, Vector &y) const
    }
 }
 
+void ParNCL2FaceRestriction::AddMultTransposeInPlace(Vector& x, Vector& y) const
+{
+   if (nf==0) { return; }
+   if (type==FaceType::Interior)
+   {
+      if ( m==L2FaceValues::DoubleValued )
+      {
+         DoubleValuedNonconformingTransposeInterpolationInPlace(x);
+         DoubleValuedConformingAddMultTranspose(x, y);
+      }
+      else if ( m==L2FaceValues::SingleValued )
+      {
+         SingleValuedNonconformingTransposeInterpolationInPlace(x);
+         SingleValuedConformingAddMultTranspose(x, y);
+      }
+   }
+   else
+   {
+      if ( m==L2FaceValues::DoubleValued )
+      {
+         DoubleValuedConformingAddMultTranspose(x, y);
+      }
+      else if ( m==L2FaceValues::SingleValued )
+      {
+         SingleValuedConformingAddMultTranspose(x, y);
+      }
+   }
+}
+
 void ParNCL2FaceRestriction::FillI(SparseMatrix &mat,
                                    const bool keep_nbr_block) const
 {
@@ -1042,6 +939,7 @@ void ParNCL2FaceRestriction::ComputeScatterIndicesAndOffsets(
 
    // Transform the interpolation matrix map into a contiguous memory structure.
    interpolations.LinearizeInterpolatorMapIntoVector();
+   interpolations.InitializeNCInterpConfig();
 }
 
 void ParNCL2FaceRestriction::ComputeGatherIndices(

fem/prestriction.hpp

@@ -68,6 +68,21 @@ public:
        @param[in,out] y The L-vector degrees of freedom. */
    void AddMultTranspose(const Vector &x, Vector &y) const override;
 
+   /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
+       L-Vector.
+
+       @param[in,out]  x The face E-Vector degrees of freedom with the given format:
+                         face_dofs x vdim x nf
+                         where nf is the number of interior or boundary faces
+                         requested by @a type in the constructor.
+                         The face_dofs should be ordered according to the given
+                         ElementDofOrdering.
+       @param[in,out] y The L-vector degrees of freedom.
+
+      @note This method is an optimization of AddMultTranspose where the @a x
+      Vector is used and modified to avoid memory allocation and memcpy. */
+   void AddMultTransposeInPlace(Vector &x, Vector &y) const override;
+
 private:
    /** @brief Compute the scatter indices: L-vector to E-vector, the offsets
        for the gathering: E-vector to L-vector, and the interpolators from
@@ -88,6 +103,31 @@ private:
    */
    void ComputeGatherIndices(const ElementDofOrdering ordering,
                              const FaceType type);
+
+public: // For nvcc
+   /** @brief Apply a change of basis from coarse element basis to fine element
+       basis for the coarse face dofs.
+
+       @param[in,out] x The dofs vector that needs coarse dofs to be express in
+                        term of the fine basis.
+   */
+   void NonconformingInterpolation(Vector& x) const;
+
+   /** @brief Apply a change of basis from fine element basis to coarse element
+       basis for the coarse face dofs.
+
+       @param[in] x The dofs vector that needs coarse dofs to be express in term
+                    of the coarse basis, the result is stored in x_interp.
+   */
+   void NonconformingTransposeInterpolation(const Vector& x) const;
+
+   /** @brief Apply a change of basis from fine element basis to coarse element
+       basis for the coarse face dofs.
+
+       @param[in] x The dofs vector that needs coarse dofs to be express in term
+                    of the coarse basis, the result is stored in x_interp.
+   */
+   void NonconformingTransposeInterpolationInPlace(Vector& x) const;
 };
 
 /// Operator that extracts Face degrees of freedom in parallel.
@@ -265,6 +305,21 @@ public:
        @param[in,out] y The L-vector degrees of freedom. */
    void AddMultTranspose(const Vector &x, Vector &y) const override;
 
+   /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
+       L-Vector.
+
+       @param[in,out]  x The face E-Vector degrees of freedom with the given format:
+                         if L2FacesValues::DoubleValued (face_dofs x vdim x 2 x nf),
+                         if L2FacesValues::SingleValued (face_dofs x vdim x nf),
+                         where nf is the number of interior or boundary faces
+                         requested by @a type in the constructor.
+                         The face_dofs should be ordered according to the given
+                         ElementDofOrdering
+       @param[in,out] y The L-vector degrees of freedom.
+
+       @note @a x is used for computation. */
+   void AddMultTransposeInPlace(Vector &x, Vector &y) const override;
+
    /** @brief Fill the I array of SparseMatrix corresponding to the sparsity
        pattern given by this ParNCL2FaceRestriction.
 

fem/qfunction.cpp

@@ -44,31 +44,6 @@ QuadratureFunction::QuadratureFunction(Mesh *mesh, std::istream &in)
    Load(in, vdim*qspace->GetSize());
 }
 
-void QuadratureFunction::SetSpace(QuadratureSpaceBase *qspace_, int vdim_)
-{
-   if (qspace_ != qspace)
-   {
-      if (own_qspace) { delete qspace; }
-      qspace = qspace_;
-      own_qspace = false;
-   }
-   vdim = (vdim_ < 0) ? vdim : vdim_;
-   SetSize(vdim*qspace->GetSize());
-}
-
-void QuadratureFunction::SetSpace(
-   QuadratureSpaceBase *qspace_, double *qf_data, int vdim_)
-{
-   if (qspace_ != qspace)
-   {
-      if (own_qspace) { delete qspace; }
-      qspace = qspace_;
-      own_qspace = false;
-   }
-   vdim = (vdim_ < 0) ? vdim : vdim_;
-   NewDataAndSize(qf_data, vdim*qspace->GetSize());
-}
-
 void QuadratureFunction::Save(std::ostream &os) const
 {
    GetSpace()->Save(os);
@@ -96,7 +71,8 @@ void QuadratureFunction::ProjectGridFunction(const GridFunction &gf)
       R->Mult(gf, e_vec);
 
       // Use quadrature interpolator to go from E-vector to Q-vector
-      const QuadratureInterpolator *qi = gf_fes.GetQuadratureInterpolator(*qs_elem);
+      const QuadratureInterpolator *qi =
+         gf_fes.GetQuadratureInterpolator(*qs_elem);
       qi->SetOutputLayout(QVectorLayout::byVDIM);
       qi->DisableTensorProducts(!use_tensor_products);
       qi->Values(e_vec, *this);

fem/qfunction.hpp

@@ -262,6 +262,34 @@ inline void QuadratureFunction::GetValues(
    }
 }
 
+
+inline void QuadratureFunction::SetSpace(QuadratureSpaceBase *qspace_,
+                                         int vdim_)
+{
+   if (qspace_ != qspace)
+   {
+      if (own_qspace) { delete qspace; }
+      qspace = qspace_;
+      own_qspace = false;
+   }
+   vdim = (vdim_ < 0) ? vdim : vdim_;
+   SetSize(vdim*qspace->GetSize());
+}
+
+inline void QuadratureFunction::SetSpace(
+   QuadratureSpaceBase *qspace_, double *qf_data, int vdim_)
+{
+   if (qspace_ != qspace)
+   {
+      if (own_qspace) { delete qspace; }
+      qspace = qspace_;
+      own_qspace = false;
+   }
+   vdim = (vdim_ < 0) ? vdim : vdim_;
+   NewDataAndSize(qf_data, vdim*qspace->GetSize());
+}
+
+
 } // namespace mfem
 
 #endif

fem/qspace.cpp

@@ -16,7 +16,7 @@ namespace mfem
 
 QuadratureSpaceBase::QuadratureSpaceBase(Mesh &mesh_, Geometry::Type geom,
                                          const IntegrationRule &ir)
-   : mesh(mesh_)
+   : mesh(mesh_), order(ir.GetOrder())
 {
    for (int g = 0; g < Geometry::NumGeom; g++)
    {

fem/restriction.cpp

@@ -147,7 +147,8 @@ void ElementRestriction::MultUnsigned(const Vector& x, Vector& y) const
    });
 }
 
-void ElementRestriction::MultTranspose(const Vector& x, Vector& y) const
+template <bool ADD>
+void ElementRestriction::AddMultTranspose(const Vector& x, Vector& y) const
 {
    // Assumes all elements have the same number of dofs
    const int nd = dof;
@@ -156,7 +157,7 @@ void ElementRestriction::MultTranspose(const Vector& x, Vector& y) const
    auto d_offsets = offsets.Read();
    auto d_indices = indices.Read();
    auto d_x = Reshape(x.Read(), nd, vd, ne);
-   auto d_y = Reshape(y.Write(), t?vd:ndofs, t?ndofs:vd);
+   auto d_y = Reshape(ADD ? y.ReadWrite() : y.Write(), t?vd:ndofs, t?ndofs:vd);
    MFEM_FORALL(i, ndofs,
    {
       const int offset = d_offsets[i];
@@ -170,11 +171,24 @@ void ElementRestriction::MultTranspose(const Vector& x, Vector& y) const
             dof_value += ((d_indices[j] >= 0) ? d_x(idx_j % nd, c, idx_j / nd) :
                           -d_x(idx_j % nd, c, idx_j / nd));
          }
-         d_y(t?c:i,t?i:c) = dof_value;
+         if (ADD) { d_y(t?c:i,t?i:c) += dof_value; }
+         else { d_y(t?c:i,t?i:c) = dof_value; }
       }
    });
 }
 
+void ElementRestriction::MultTranspose(const Vector& x, Vector& y) const
+{
+   constexpr bool ADD = false;
+   AddMultTranspose<ADD>(x, y);
+}
+
+void ElementRestriction::AddMultTranspose(const Vector& x, Vector& y) const
+{
+   constexpr bool ADD = true;
+   AddMultTranspose<ADD>(x, y);
+}
+
 void ElementRestriction::MultTransposeUnsigned(const Vector& x, Vector& y) const
 {
    // Assumes all elements have the same number of dofs
@@ -506,13 +520,14 @@ void L2ElementRestriction::Mult(const Vector &x, Vector &y) const
    });
 }
 
-void L2ElementRestriction::MultTranspose(const Vector &x, Vector &y) const
+template <bool ADD>
+void L2ElementRestriction::AddMultTranspose(const Vector &x, Vector &y) const
 {
    const int nd = ndof;
    const int vd = vdim;
    const bool t = byvdim;
    auto d_x = Reshape(x.Read(), nd, vd, ne);
-   auto d_y = Reshape(y.Write(), t?vd:ndofs, t?ndofs:vd);
+   auto d_y = Reshape(ADD ? y.ReadWrite() : y.Write(), t?vd:ndofs, t?ndofs:vd);
    MFEM_FORALL(i, ndofs,
    {
       const int idx = i;
@@ -520,11 +535,24 @@ void L2ElementRestriction::MultTranspose(const Vector &x, Vector &y) const
       const int e = idx / nd;
       for (int c = 0; c < vd; ++c)
       {
-         d_y(t?c:idx,t?idx:c) = d_x(dof, c, e);
+         if (ADD) { d_y(t?c:idx,t?idx:c) += d_x(dof, c, e); }
+         else { d_y(t?c:idx,t?idx:c) = d_x(dof, c, e); }
       }
    });
 }
 
+void L2ElementRestriction::MultTranspose(const Vector &x, Vector &y) const
+{
+   constexpr bool ADD = false;
+   AddMultTranspose<ADD>(x, y);
+}
+
+void L2ElementRestriction::AddMultTranspose(const Vector &x, Vector &y) const
+{
+   constexpr bool ADD = true;
+   AddMultTranspose<ADD>(x, y);
+}
+
 void L2ElementRestriction::FillI(SparseMatrix &mat) const
 {
    const int elem_dofs = ndof;
@@ -1774,6 +1802,31 @@ void InterpolationManager::LinearizeInterpolatorMapIntoVector()
    interp_map.clear();
 }
 
+void InterpolationManager::InitializeNCInterpConfig()
+{
+   // Count nonconforming faces
+   int num_nc_faces = 0;
+   for (int i = 0; i < interp_config.Size(); i++)
+   {
+      if ( interp_config[i].is_non_conforming )
+      {
+         num_nc_faces++;
+      }
+   }
+   // Set nc_interp_config
+   nc_interp_config.SetSize(num_nc_faces);
+   int nc_index = 0;
+   for (int i = 0; i < interp_config.Size(); i++)
+   {
+      auto & config = interp_config[i];
+      if ( config.is_non_conforming )
+      {
+         nc_interp_config[nc_index] = NCInterpConfig(i, config);
+         nc_index++;
+      }
+   }
+}
+
 NCL2FaceRestriction::NCL2FaceRestriction(const FiniteElementSpace &fes,
                                          const ElementDofOrdering ordering,
                                          const FaceType type,
@@ -1801,64 +1854,53 @@ NCL2FaceRestriction::NCL2FaceRestriction(const FiniteElementSpace &fes,
 
 void NCL2FaceRestriction::DoubleValuedNonconformingMult(
    const Vector& x, Vector& y) const
+{
+   DoubleValuedConformingMult(x, y);
+   DoubleValuedNonconformingInterpolation(y);
+}
+
+void NCL2FaceRestriction::DoubleValuedNonconformingInterpolation(
+   Vector& y) const
 {
    // Assumes all elements have the same number of dofs
    const int nface_dofs = face_dofs;
    const int vd = vdim;
-   const bool t = byvdim;
-   auto d_indices1 = scatter_indices1.Read();
-   auto d_indices2 = scatter_indices2.Read();
-   auto d_x = Reshape(x.Read(), t?vd:ndofs, t?ndofs:vd);
-   auto d_y = Reshape(y.Write(), nface_dofs, vd, 2, nf);
-   auto interp_config_ptr = interpolations.GetFaceInterpConfig().Read();
+   auto d_y = Reshape(y.ReadWrite(), nface_dofs, vd, 2, nf);
+   auto &nc_interp_config = interpolations.GetNCFaceInterpConfig();
+   const int num_nc_faces = nc_interp_config.Size();
+   if ( num_nc_faces == 0 ) { return; }
+   auto interp_config_ptr = nc_interp_config.Read();
    const int nc_size = interpolations.GetNumInterpolators();
    auto d_interp = Reshape(interpolations.GetInterpolators().Read(),
                            nface_dofs, nface_dofs, nc_size);
    static constexpr int max_nd = 16*16;
    MFEM_VERIFY(nface_dofs<=max_nd, "Too many degrees of freedom.");
-   MFEM_FORALL_3D(face, nf, nface_dofs, 1, 1,
+   MFEM_FORALL_3D(nc_face, num_nc_faces, nface_dofs, 1, 1,
    {
       MFEM_SHARED double dof_values[max_nd];
-      const InterpConfig conf = interp_config_ptr[face];
-      const int master_side = conf.master_side;
-      const int interp_index = conf.index;
-      for (int side = 0; side < 2; side++)
+      const NCInterpConfig conf = interp_config_ptr[nc_face];
+      if ( conf.is_non_conforming )
       {
-         if ( !conf.is_non_conforming || side!=master_side )
+         const int master_side = conf.master_side;
+         const int interp_index = conf.index;
+         const int face = conf.face_index;
+         for (int c = 0; c < vd; ++c)
          {
-            // No interpolation needed
             MFEM_FOREACH_THREAD(dof,x,nface_dofs)
             {
-               const int i = face*nface_dofs + dof;
-               const int idx = side==0 ? d_indices1[i] : d_indices2[i];
-               for (int c = 0; c < vd; ++c)
-               {
-                  d_y(dof, c, side, face) = d_x(t?c:idx, t?idx:c);
-               }
+               dof_values[dof] = d_y(dof, c, master_side, face);
             }
-         }
-         else // Interpolation from coarse to fine
-         {
-            for (int c = 0; c < vd; ++c)
+            MFEM_SYNC_THREAD;
+            MFEM_FOREACH_THREAD(dof_out,x,nface_dofs)
             {
-               MFEM_FOREACH_THREAD(dof,x,nface_dofs)
+               double res = 0.0;
+               for (int dof_in = 0; dof_in<nface_dofs; dof_in++)
                {
-                  const int i = face*nface_dofs + dof;
-                  const int idx = side==0 ? d_indices1[i] : d_indices2[i];
-                  dof_values[dof] = d_x(t?c:idx, t?idx:c);
+                  res += d_interp(dof_out, dof_in, interp_index)*dof_values[dof_in];
                }
-               MFEM_SYNC_THREAD;
-               MFEM_FOREACH_THREAD(dof_out,x,nface_dofs)
-               {
-                  double res = 0.0;
-                  for (int dof_in = 0; dof_in<nface_dofs; dof_in++)
-                  {
-                     res += d_interp(dof_out, dof_in, interp_index)*dof_values[dof_in];
-                  }
-                  d_y(dof_out, c, side, face) = res;
-               }
-               MFEM_SYNC_THREAD;
+               d_y(dof_out, c, master_side, face) = res;
             }
+            MFEM_SYNC_THREAD;
          }
       }
    });
@@ -1892,23 +1934,34 @@ void NCL2FaceRestriction::SingleValuedNonconformingTransposeInterpolation(
       x_interp.SetSize(x.Size());
    }
    x_interp = x;
+   SingleValuedNonconformingTransposeInterpolationInPlace(x_interp);
+}
+
+
+void NCL2FaceRestriction::SingleValuedNonconformingTransposeInterpolationInPlace(
+   Vector& x) const
+{
    // Assumes all elements have the same number of dofs
    const int nface_dofs = face_dofs;
    const int vd = vdim;
    // Interpolation
    auto d_x = Reshape(x_interp.ReadWrite(), nface_dofs, vd, nf);
-   auto interp_config_ptr = interpolations.GetFaceInterpConfig().Read();
+   auto &nc_interp_config = interpolations.GetNCFaceInterpConfig();
+   const int num_nc_faces = nc_interp_config.Size();
+   if ( num_nc_faces == 0 ) { return; }
+   auto interp_config_ptr = nc_interp_config.Read();
    auto interpolators = interpolations.GetInterpolators().Read();
    const int nc_size = interpolations.GetNumInterpolators();
    auto d_interp = Reshape(interpolators, nface_dofs, nface_dofs, nc_size);
    static constexpr int max_nd = 16*16;
    MFEM_VERIFY(nface_dofs<=max_nd, "Too many degrees of freedom.");
-   MFEM_FORALL_3D(face, nf, nface_dofs, 1, 1,
+   MFEM_FORALL_3D(nc_face, num_nc_faces, nface_dofs, 1, 1,
    {
       MFEM_SHARED double dof_values[max_nd];
-      const InterpConfig conf = interp_config_ptr[face];
+      const NCInterpConfig conf = interp_config_ptr[nc_face];
       const int master_side = conf.master_side;
       const int interp_index = conf.index;
+      const int face = conf.face_index;
       if ( conf.is_non_conforming && master_side==0 )
       {
          // Interpolation from fine to coarse
@@ -1945,23 +1998,33 @@ void NCL2FaceRestriction::DoubleValuedNonconformingTransposeInterpolation(
       x_interp.SetSize(x.Size());
    }
    x_interp = x;
+   DoubleValuedNonconformingTransposeInterpolationInPlace(x_interp);
+}
+
+void NCL2FaceRestriction::DoubleValuedNonconformingTransposeInterpolationInPlace(
+   Vector& x) const
+{
    // Assumes all elements have the same number of dofs
    const int nface_dofs = face_dofs;
    const int vd = vdim;
    // Interpolation
-   auto d_x = Reshape(x_interp.ReadWrite(), nface_dofs, vd, 2, nf);
-   auto interp_config_ptr = interpolations.GetFaceInterpConfig().Read();
+   auto d_x = Reshape(x.ReadWrite(), nface_dofs, vd, 2, nf);
+   auto &nc_interp_config = interpolations.GetNCFaceInterpConfig();
+   const int num_nc_faces = nc_interp_config.Size();
+   if ( num_nc_faces == 0 ) { return; }
+   auto interp_config_ptr = nc_interp_config.Read();
    auto interpolators = interpolations.GetInterpolators().Read();
    const int nc_size = interpolations.GetNumInterpolators();
    auto d_interp = Reshape(interpolators, nface_dofs, nface_dofs, nc_size);
    static constexpr int max_nd = 16*16;
    MFEM_VERIFY(nface_dofs<=max_nd, "Too many degrees of freedom.");
-   MFEM_FORALL_3D(face, nf, nface_dofs, 1, 1,
+   MFEM_FORALL_3D(nc_face, num_nc_faces, nface_dofs, 1, 1,
    {
       MFEM_SHARED double dof_values[max_nd];
-      const InterpConfig conf = interp_config_ptr[face];
+      const NCInterpConfig conf = interp_config_ptr[nc_face];
       const int master_side = conf.master_side;
       const int interp_index = conf.index;
+      const int face = conf.face_index;
       if ( conf.is_non_conforming )
       {
          // Interpolation from fine to coarse
@@ -2016,6 +2079,35 @@ void NCL2FaceRestriction::AddMultTranspose(const Vector& x, Vector& y) const
    }
 }
 
+void NCL2FaceRestriction::AddMultTransposeInPlace(Vector& x, Vector& y) const
+{
+   if (nf==0) { return; }
+   if (type==FaceType::Interior)
+   {
+      if ( m==L2FaceValues::DoubleValued )
+      {
+         DoubleValuedNonconformingTransposeInterpolationInPlace(x);
+         DoubleValuedConformingAddMultTranspose(x, y);
+      }
+      else if ( m==L2FaceValues::SingleValued )
+      {
+         SingleValuedNonconformingTransposeInterpolationInPlace(x);
+         SingleValuedConformingAddMultTranspose(x, y);
+      }
+   }
+   else
+   {
+      if ( m==L2FaceValues::DoubleValued )
+      {
+         DoubleValuedConformingAddMultTranspose(x, y);
+      }
+      else if ( m==L2FaceValues::SingleValued )
+      {
+         SingleValuedConformingAddMultTranspose(x, y);
+      }
+   }
+}
+
 void NCL2FaceRestriction::FillI(SparseMatrix &mat,
                                 const bool keep_nbr_block) const
 {
@@ -2116,6 +2208,7 @@ void NCL2FaceRestriction::ComputeScatterIndicesAndOffsets(
 
    // Transform the interpolation matrix map into a contiguous memory structure.
    interpolations.LinearizeInterpolatorMapIntoVector();
+   interpolations.InitializeNCInterpConfig();
 }
 
 void NCL2FaceRestriction::ComputeGatherIndices(

fem/restriction.hpp

@@ -21,10 +21,19 @@ namespace mfem
 class FiniteElementSpace;
 enum class ElementDofOrdering;
 
+/// Abstract base class that defines an interface for element restrictions.
+class ElementRestrictionOperator : public Operator
+{
+public:
+   /// @brief Add the E-vector degrees of freedom @a x to the L-vector degrees
+   /// of freedom @a y.
+   virtual void AddMultTranspose(const Vector &x, Vector &y) const = 0;
+};
+
 /// Operator that converts FiniteElementSpace L-vectors to E-vectors.
 /** Objects of this type are typically created and owned by FiniteElementSpace
     objects, see FiniteElementSpace::GetElementRestriction(). */
-class ElementRestriction : public Operator
+class ElementRestriction : public ElementRestrictionOperator
 {
 private:
    /** This number defines the maximum number of elements any dof can belong to
@@ -58,6 +67,7 @@ public:
    ElementRestriction(const FiniteElementSpace&, ElementDofOrdering);
    void Mult(const Vector &x, Vector &y) const;
    void MultTranspose(const Vector &x, Vector &y) const;
+   void AddMultTranspose(const Vector &x, Vector &y) const;
 
    /// Compute Mult without applying signs based on DOF orientations.
    void MultUnsigned(const Vector &x, Vector &y) const;
@@ -85,6 +95,11 @@ public:
    /** Fill the J and Data arrays of SparseMatrix corresponding to the sparsity
        pattern given by this ElementRestriction, and the values of ea_data. */
    void FillJAndData(const Vector &ea_data, SparseMatrix &mat) const;
+   /// @private Not part of the public interface (device kernel limitation).
+   ///
+   /// Performs either MultTranspose or AddMultTranspose depending on the
+   /// boolean template parameter @a ADD.
+   template <bool ADD> void AddMultTranspose(const Vector &x, Vector &y) const;
 };
 
 /// Operator that converts L2 FiniteElementSpace L-vectors to E-vectors.
@@ -92,7 +107,7 @@ public:
     objects, see FiniteElementSpace::GetElementRestriction(). L-vectors
     corresponding to grid functions in L2 finite element spaces differ from
     E-vectors only in the ordering of the degrees of freedom. */
-class L2ElementRestriction : public Operator
+class L2ElementRestriction : public ElementRestrictionOperator
 {
    const int ne;
    const int vdim;
@@ -103,12 +118,18 @@ public:
    L2ElementRestriction(const FiniteElementSpace&);
    void Mult(const Vector &x, Vector &y) const;
    void MultTranspose(const Vector &x, Vector &y) const;
+   void AddMultTranspose(const Vector &x, Vector &y) const;
    /** Fill the I array of SparseMatrix corresponding to the sparsity pattern
        given by this ElementRestriction. */
    void FillI(SparseMatrix &mat) const;
    /** Fill the J and Data arrays of SparseMatrix corresponding to the sparsity
        pattern given by this L2FaceRestriction, and the values of ea_data. */
    void FillJAndData(const Vector &ea_data, SparseMatrix &mat) const;
+   /// @private Not part of the public interface (device kernel limitation).
+   ///
+   /// Performs either MultTranspose or AddMultTranspose depending on the
+   /// boolean template parameter @a ADD.
+   template <bool ADD> void AddMultTranspose(const Vector &x, Vector &y) const;
 };
 
 /** An enum type to specify if only e1 value is requested (SingleValued) or both
@@ -162,6 +183,22 @@ public:
    */
    virtual void AddMultTranspose(const Vector &x, Vector &y) const = 0;
 
+   /** @brief Add the face degrees of freedom @a x to the element degrees of
+       freedom @a y. Perform the same computation as AddMultTranspose, but
+       @a x is invalid after calling this method.
+
+       @param[in,out]     x The face degrees of freedom on the face.
+       @param[in,out] y The L-vector of degrees of freedom to which we add the
+                        face degrees of freedom.
+
+      @note This method is an optimization of AddMultTranspose where the @a x
+      Vector is used and modified to avoid memory allocation and memcpy.
+   */
+   virtual void AddMultTransposeInPlace(Vector &x, Vector &y) const
+   {
+      AddMultTranspose(x, y);
+   }
+
    /** @brief Set the face degrees of freedom in the element degrees of freedom
        @a y to the values given in @a x.
 
@@ -229,6 +266,8 @@ public:
                      ElementDofOrdering. */
    void Mult(const Vector &x, Vector &y) const override;
 
+   using FaceRestriction::AddMultTransposeInPlace;
+
    /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
        L-Vector.
 
@@ -358,6 +397,8 @@ public:
                      ElementDofOrdering. */
    void Mult(const Vector &x, Vector &y) const override;
 
+   using FaceRestriction::AddMultTranspose;
+
    /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
        L-Vector.
 
@@ -585,6 +626,39 @@ struct InterpConfig
    InterpConfig &operator=(const InterpConfig &rhs) = default;
 };
 
+/** This struct stores which side is the master nonconforming side and the
+    index of the interpolator, see InterpolationManager class below. */
+struct NCInterpConfig
+{
+   int face_index;
+   uint32_t is_non_conforming : 1;
+   uint32_t master_side : 1;
+   uint32_t index : 30;
+
+   // default constructor.
+   NCInterpConfig() = default;
+
+   // Non-conforming face
+   NCInterpConfig(int face_index, int master_side, int nc_index)
+      : face_index(face_index),
+        is_non_conforming(1),
+        master_side(master_side),
+        index(nc_index)
+   { }
+
+   // Non-conforming face
+   NCInterpConfig(int face_index, InterpConfig & config)
+      : face_index(face_index),
+        is_non_conforming(config.is_non_conforming),
+        master_side(config.master_side),
+        index(config.index)
+   { }
+
+   NCInterpConfig(const NCInterpConfig&) = default;
+
+   NCInterpConfig &operator=(const NCInterpConfig &rhs) = default;
+};
+
 /** @brief This class manages the storage and computation of the interpolations
     from master (coarse) face to slave (fine) face.
 */
@@ -594,6 +668,7 @@ protected:
    const FiniteElementSpace &fes;
    const ElementDofOrdering ordering;
    Array<InterpConfig> interp_config; // interpolator index for each face
+   Array<NCInterpConfig> nc_interp_config; // interpolator index for each ncface
    Vector interpolators; // face_dofs x face_dofs x num_interpolators
    int nc_cpt; // Counter for interpolators, and used as index.
 
@@ -639,6 +714,8 @@ public:
        structure. */
    void LinearizeInterpolatorMapIntoVector();
 
+   void InitializeNCInterpConfig();
+
    /// @brief Return the total number of interpolators.
    int GetNumInterpolators() const
    {
@@ -660,6 +737,14 @@ public:
       return interp_config;
    }
 
+   /** @brief Return an array containing the interpolation configuration for
+       each face registered with RegisterFaceConformingInterpolation and
+       RegisterFaceCoarseToFineInterpolation. */
+   const Array<NCInterpConfig>& GetNCFaceInterpConfig() const
+   {
+      return nc_interp_config;
+   }
+
 private:
    /** @brief Returns the interpolation operator from a master (coarse) face to
        a slave (fine) face.
@@ -749,6 +834,22 @@ public:
        @param[in,out] y The L-vector degrees of freedom. */
    void AddMultTranspose(const Vector &x, Vector &y) const override;
 
+   /** @brief Gather the degrees of freedom, i.e. goes from face E-Vector to
+       L-Vector.
+
+       @param[in,out]  x The face E-Vector degrees of freedom with the given format:
+                         if L2FacesValues::DoubleValued (face_dofs x vdim x 2 x nf),
+                         if L2FacesValues::SingleValued (face_dofs x vdim x nf),
+                         where nf is the number of interior or boundary faces
+                         requested by @a type in the constructor.
+                         The face_dofs should be ordered according to the given
+                         ElementDofOrdering
+       @param[in,out] y The L-vector degrees of freedom.
+
+      @note This method is an optimization of AddMultTranspose where the @a x
+      Vector is used and modified to avoid memory allocation and memcpy. */
+   void AddMultTransposeInPlace(Vector &x, Vector &y) const override;
+
    /** @brief Fill the I array of SparseMatrix corresponding to the sparsity
        pattern given by this NCL2FaceRestriction.
 
@@ -838,6 +939,14 @@ public:
                      ElementDofOrdering. */
    virtual void DoubleValuedNonconformingMult(const Vector& x, Vector& y) const;
 
+   /** @brief Apply a change of basis from coarse element basis to fine element
+       basis for the coarse face dofs.
+
+       @param[in,out] x The dofs vector that needs coarse dofs to be express in
+                        term of the fine basis.
+   */
+   void DoubleValuedNonconformingInterpolation(Vector& x) const;
+
    /** @brief Apply a change of basis from fine element basis to coarse element
        basis for the coarse face dofs. Should only be used when:
        L2FaceValues m == L2FaceValues::SingleValued
@@ -847,6 +956,15 @@ public:
    */
    void SingleValuedNonconformingTransposeInterpolation(const Vector& x) const;
 
+   /** @brief Apply a change of basis from fine element basis to coarse element
+       basis for the coarse face dofs. Should only be used when:
+       L2FaceValues m == L2FaceValues::SingleValued
+
+       @param[in,out] x The dofs vector that needs coarse dofs to be express in
+                        term of the coarse basis, the result is stored in x.
+   */
+   void SingleValuedNonconformingTransposeInterpolationInPlace(Vector& x) const;
+
    /** @brief Apply a change of basis from fine element basis to coarse element
        basis for the coarse face dofs. Should only be used when:
        L2FaceValues m == L2FaceValues::DoubleValued
@@ -855,6 +973,16 @@ public:
                     of the coarse basis, the result is stored in x_interp.
    */
    void DoubleValuedNonconformingTransposeInterpolation(const Vector& x) const;
+
+   /** @brief Apply a change of basis from fine element basis to coarse element
+       basis for the coarse face dofs. Should only be used when:
+       L2FaceValues m == L2FaceValues::DoubleValued
+
+       @param[in,out] x The dofs vector that needs coarse dofs to be express in
+                        term of the coarse basis, the result is stored in
+                        x.
+   */
+   void DoubleValuedNonconformingTransposeInterpolationInPlace(Vector& x) const;
 };
 
 /** @brief Return the face map that extracts the degrees of freedom for the

fem/tmop.cpp

@@ -20,6 +20,16 @@ namespace mfem
 
 // Target-matrix optimization paradigm (TMOP) mesh quality metrics.
 
+double TMOP_Combo_QualityMetric::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   double metric = 0.;
+   for (int i = 0; i < tmop_q_arr.Size(); i++)
+   {
+      metric += wt_arr[i]*tmop_q_arr[i]->EvalWMatrixForm(Jpt);
+   }
+   return metric;
+}
+
 double TMOP_Combo_QualityMetric::EvalW(const DenseMatrix &Jpt) const
 {
    double metric = 0.;
@@ -232,6 +242,11 @@ double TMOP_Metric_aspratio3D::EvalW(const DenseMatrix &Jpt) const
           ) / 3.0;
 }
 
+double TMOP_Metric_002::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   return 0.5 * Jpt.FNorm2() / Jpt.Det() - 1.0;
+}
+
 double TMOP_Metric_002::EvalW(const DenseMatrix &Jpt) const
 {
    ie.SetJacobian(Jpt.GetData());
@@ -516,12 +531,23 @@ void TMOP_Metric_056::AssembleH(const DenseMatrix &Jpt,
    ie.Assemble_ddI2b(weight*(0.5 - 0.5/ie.Get_I2()), A.GetData());
 }
 
+double TMOP_Metric_058::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   // mu_58 = |J^t J|^2 / det(J)^2 - 2|J|^2 / det(J) + 2
+   DenseMatrix JtJ(2);
+   MultAAt(Jpt, JtJ);
+   JtJ.Transpose();
+   double det = Jpt.Det();
+
+   return JtJ.FNorm2()/(det*det) - 2*Jpt.FNorm2()/det + 2.0;
+}
+
 double TMOP_Metric_058::EvalW(const DenseMatrix &Jpt) const
 {
    // mu_58 = I1b*(I1b - 2)
    ie.SetJacobian(Jpt.GetData());
    const double I1b = ie.Get_I1b();
-   return I1b*(I1b - 1.0);
+   return I1b*(I1b - 2.0);
 }
 
 void TMOP_Metric_058::EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const
@@ -668,8 +694,18 @@ void TMOP_Metric_252::AssembleH(const DenseMatrix &Jpt,
    ie.Assemble_ddI2b(weight*(c - 0.5*c*c), A.GetData());
 }
 
+double TMOP_Metric_301::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   // mu_301 = 1/3 |J| |J^-1| - 1.
+   ie.SetJacobian(Jpt.GetData());
+   DenseMatrix inv(3);
+   CalcInverse(Jpt, inv);
+   return Jpt.FNorm() * inv.FNorm() / 3.0 - 1.0;
+}
+
 double TMOP_Metric_301::EvalW(const DenseMatrix &Jpt) const
 {
+   // mu_301 = 1/3 sqrt(I1b * I2b) - 1
    ie.SetJacobian(Jpt.GetData());
    return std::sqrt(ie.Get_I1b()*ie.Get_I2b())/3. - 1.;
 }
@@ -718,6 +754,15 @@ void TMOP_Metric_301::AssembleH(const DenseMatrix &Jpt,
    ie.Assemble_TProd(a/(2*I1b_I2b), d_I1b_I2b_data, A.GetData());
 }
 
+double TMOP_Metric_302::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   // mu_301 = |J|^2 |J^{-1}|^2 / 9 - 1.
+   ie.SetJacobian(Jpt.GetData());
+   DenseMatrix inv(3);
+   CalcInverse(Jpt, inv);
+   return Jpt.FNorm2() * inv.FNorm2() / 9.0 - 1.0;
+}
+
 double TMOP_Metric_302::EvalW(const DenseMatrix &Jpt) const
 {
    // mu_2 = |J|^2 |J^{-1}|^2 / 9 - 1
@@ -752,14 +797,24 @@ void TMOP_Metric_302::AssembleH(const DenseMatrix &Jpt,
    ie.Assemble_ddI1b(c1*ie.Get_I2b(), A.GetData());
 }
 
+double TMOP_Metric_303::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   // mu_303 = |J|^2 / 3 / det(J)^(2/3) - 1.
+   ie.SetJacobian(Jpt.GetData());
+   return Jpt.FNorm2() / 3.0 / pow(Jpt.Det(), 2.0 / 3.0) - 1.0;
+}
+
 double TMOP_Metric_303::EvalW(const DenseMatrix &Jpt) const
 {
+   // mu_303 = |J|^2 / 3 / det(J)^(2/3) - 1 = I1b/3 - 1.
    ie.SetJacobian(Jpt.GetData());
    return ie.Get_I1b()/3.0 - 1.0;
 }
 
 void TMOP_Metric_303::EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const
 {
+   // mu_304 = I1b/3 - 1.
+   // P      = dI1b/3.
    ie.SetJacobian(Jpt.GetData());
    P.Set(1./3., ie.Get_dI1b());
 }
@@ -769,11 +824,47 @@ void TMOP_Metric_303::AssembleH(const DenseMatrix &Jpt,
                                 const double weight,
                                 DenseMatrix &A) const
 {
+   // P  = dI1b/3.
+   // dP = ddI1b/3.
    ie.SetJacobian(Jpt.GetData());
    ie.SetDerivativeMatrix(DS.Height(), DS.GetData());
    ie.Assemble_ddI1b(weight/3., A.GetData());
 }
 
+double TMOP_Metric_304::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   // mu_304 = |J|^3 / 3^(3/2) / det(J) - 1
+   const double fnorm = Jpt.FNorm();
+   return fnorm * fnorm * fnorm / pow(3.0, 1.5) / Jpt.Det() - 1.0;
+}
+
+double TMOP_Metric_304::EvalW(const DenseMatrix &Jpt) const
+{
+   // mu_304 = (I1b/3)^3/2 - 1.
+   ie.SetJacobian(Jpt.GetData());
+   return pow(ie.Get_I1b()/3.0, 1.5) - 1.0;
+}
+
+void TMOP_Metric_304::EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const
+{
+   // mu_304 = (I1b/3)^3/2 - 1.
+   // P      = 3/2 * (I1b/3)^1/2 * dI1b / 3 = 1/2 * (I1b/3)^1/2 * dI1b.
+   ie.SetJacobian(Jpt.GetData());
+   P.Set(0.5 * sqrt(ie.Get_I1b()/3.0), ie.Get_dI1b());
+}
+
+void TMOP_Metric_304::AssembleH(const DenseMatrix &Jpt, const DenseMatrix &DS,
+                                const double weight, DenseMatrix &A) const
+{
+   // P  = 1/2 * (I1b/3)^1/2 * dI1b.
+   // dP = 1/12 * (I1b/3)^(-1/2) * (dI1b x dI1b) + 1/2 * (I1b/3)^1/2 * ddI1b.
+   ie.SetJacobian(Jpt.GetData());
+   ie.SetDerivativeMatrix(DS.Height(), DS.GetData());
+   ie.Assemble_TProd(weight / 12.0 / sqrt(ie.Get_I1b()/3.0),
+                     ie.Get_dI1b(), A.GetData());
+   ie.Assemble_ddI1b(weight / 2.0 * sqrt(ie.Get_I1b()/3.0), A.GetData());
+}
+
 double TMOP_Metric_311::EvalW(const DenseMatrix &Jpt) const
 {
    // mu_311 = (det(J) - 1)^2 - det(J) + (det(J)^2 + eps)^{1/2}
@@ -868,6 +959,12 @@ void TMOP_Metric_315::AssembleH(const DenseMatrix &Jpt,
    ie.Assemble_ddI3b(2*weight*(ie.Get_I3b() - 1.0), A.GetData());
 }
 
+double TMOP_Metric_316::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   // mu_316 = 0.5 (det(J) + 1/det(J)) - 1.
+   return 0.5 * (Jpt.Det() + 1.0 / Jpt.Det()) - 1.0;
+}
+
 double TMOP_Metric_316::EvalW(const DenseMatrix &Jpt) const
 {
    // mu_316 = mu_16_3D = 0.5*(I3b + 1/I3b) - 1
@@ -898,6 +995,16 @@ void TMOP_Metric_316::AssembleH(const DenseMatrix &Jpt,
    ie.Assemble_ddI3b(weight*(0.5 - 0.5/ie.Get_I3()), A.GetData());
 }
 
+double TMOP_Metric_321::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   // mu_321 = |J - J^-t|^2.
+   ie.SetJacobian(Jpt.GetData());
+   DenseMatrix invt(3);
+   CalcInverseTranspose(Jpt, invt);
+   invt.Add(-1.0, Jpt);
+   return invt.FNorm2();
+}
+
 double TMOP_Metric_321::EvalW(const DenseMatrix &Jpt) const
 {
    // mu_321 = mu_21_3D = |J - J^{-t}|^2
@@ -946,6 +1053,119 @@ void TMOP_Metric_321::AssembleH(const DenseMatrix &Jpt,
    ie.Assemble_TProd(-3*c0*c3, ie.Get_dI3b(), A.GetData());
 }
 
+double TMOP_Metric_322::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   // mu_322 = 1 / (6 det(J)) |J - adj(J)^t|^2
+   DenseMatrix adj_J_t(3);
+   CalcAdjugateTranspose(Jpt, adj_J_t);
+   adj_J_t *= -1.0;
+   adj_J_t.Add(1.0, Jpt);
+   return 1.0 / 6.0 / Jpt.Det() * adj_J_t.FNorm2();
+}
+
+double TMOP_Metric_322::EvalW(const DenseMatrix &Jpt) const
+{
+   // mu_322 = 1 / (6 det(J)) |J - adj(J)^t|^2
+   //        = 1 / (6 det(J)) |J|^2 + 1/6 det(J) |J^{-1}|^2 - 1
+   //        = I1b / (I3b^-1/3) / 6 + I2b (I3b^1/3) / 6 - 1
+   ie.SetJacobian(Jpt.GetData());
+
+   return ie.Get_I1b() / pow(ie.Get_I3b(), 1.0/3.0) / 6.0 +
+          ie.Get_I2b() * pow(ie.Get_I3b(), 1.0/3.0) / 6.0 - 1.0;
+}
+
+void TMOP_Metric_322::EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const
+{
+   // mu_322 = I1b (I3b^-1/3) / 6 + I2b (I3b^1/3) / 6 - 1
+   // P      =   1/6 (I3b^-1/3) dI1b - 1/18 I1b (I3b^-4/3) dI3b
+   //          + 1/6 (I3b^1/3) dI2b  + 1/18 I2b (I3b^-2/3) dI3b
+   ie.SetJacobian(Jpt.GetData());
+   P.Set(1.0/6.0 * pow(ie.Get_I3b(), -1.0/3.0),
+         ie.Get_dI1b());
+   P.Add(-1.0/18.0 * ie.Get_I1b() * pow(ie.Get_I3b(), -4.0/3.0),
+         ie.Get_dI3b());
+   P.Add(1.0/6.0 * pow(ie.Get_I3b(), 1.0/3.0),
+         ie.Get_dI2b());
+   P.Add(1.0/18.0 * ie.Get_I2b() * pow(ie.Get_I3b(), -2.0/3.0),
+         ie.Get_dI3b());
+}
+
+void TMOP_Metric_322::AssembleH(const DenseMatrix &Jpt, const DenseMatrix &DS,
+                                const double weight, DenseMatrix &A) const
+{
+   //  P =   1/6 (I3b^-1/3) dI1b - 1/18 I1b (I3b^-4/3) dI3b
+   //      + 1/6 (I3b^1/3) dI2b  + 1/18 I2b (I3b^-2/3) dI3b
+   // dP =   1/6 (I3b^-1/3) ddI1b - 1/18 (I3b^-4/3) (dI1b x dI3b)
+   //      - 1/18 I1b (I3b^-4/3) ddI3b
+   //      - 1/18 (I3b^-4/3) (dI3b x dI1b)
+   //      + 2/27 I1b (I3b^-7/3) (dI3b x dI3b)
+   //      + 1/6 (I3b^1/3) ddI2b + 1/18 (I3b^-2/3) (dI2b x dI3b)
+   //      + 1/18 I2b (I3b^-2/3) ddI3b
+   //      + 1/18 (I3b^-2/3) (dI3b x dI2b)
+   //      - 1/27 I2b (I3b^-5/3) (dI3b x dI3b)
+   ie.SetJacobian(Jpt.GetData());
+   ie.SetDerivativeMatrix(DS.Height(), DS.GetData());
+   const double p13 = weight * pow(ie.Get_I3b(),  1.0/3.0),
+                m13 = weight * pow(ie.Get_I3b(), -1.0/3.0),
+                m23 = weight * pow(ie.Get_I3b(), -2.0/3.0),
+                m43 = weight * pow(ie.Get_I3b(), -4.0/3.0),
+                m53 = weight * pow(ie.Get_I3b(), -5.0/3.0),
+                m73 = weight * pow(ie.Get_I3b(), -7.0/3.0);
+   ie.Assemble_ddI1b(1.0/6.0 * m13, A.GetData());
+   // Combines - 1/18 (I3b^-4/3) (dI1b x dI3b) - 1/18 (I3b^-4/3) (dI3b x dI1b).
+   ie.Assemble_TProd(-1.0/18.0 * m43,
+                     ie.Get_dI1b(), ie.Get_dI3b(), A.GetData());
+   ie.Assemble_ddI3b(-1.0/18.0 * ie.Get_I1b() * m43, A.GetData());
+   ie.Assemble_TProd(2.0/27.0 * ie.Get_I1b() * m73,
+                     ie.Get_dI3b(), A.GetData());
+   ie.Assemble_ddI2b(1.0/6.0 * p13, A.GetData());
+   // Combines + 1/18 (I3b^-2/3) (dI2b x dI3b) + 1/18 (I3b^-2/3) (dI3b x dI2b).
+   ie.Assemble_TProd(1.0/18.0 * m23,
+                     ie.Get_dI2b(), ie.Get_dI3b(), A.GetData());
+   ie.Assemble_ddI3b(1.0/18.0 * ie.Get_I2b() * m23, A.GetData());
+   ie.Assemble_TProd(-1.0/27.0 * ie.Get_I2b() * m53,
+                     ie.Get_dI3b(), A.GetData());
+}
+
+double TMOP_Metric_323::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   // mu_323 = |J|^3 - 3 sqrt(3) ln(det(J)) - 3 sqrt(3).
+   double fnorm = Jpt.FNorm();
+   return fnorm * fnorm * fnorm - 3.0 * sqrt(3.0) * (log(Jpt.Det()) + 1.0);
+}
+
+double TMOP_Metric_323::EvalW(const DenseMatrix &Jpt) const
+{
+   // mu_323 = I1^3/2 - 3 sqrt(3) ln(I3b) - 3 sqrt(3).
+   ie.SetJacobian(Jpt.GetData());
+   return pow(ie.Get_I1(), 1.5) - 3.0 * sqrt(3.0) * (log(ie.Get_I3b()) + 1.0);
+}
+
+void TMOP_Metric_323::EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const
+{
+   // mu_323 = I1^3/2 - 3 sqrt(3) ln(I3b) - 3 sqrt(3).
+   // P      = 3/2 (I1^1/2) dI1 - 3 sqrt(3) (I3b^-1) dI3b.
+   ie.SetJacobian(Jpt.GetData());
+   P.Set(1.5 * sqrt(ie.Get_I1()), ie.Get_dI1());
+   P.Add(- 3.0 * sqrt(3.0) / ie.Get_I3b(), ie.Get_dI3b());
+}
+
+void TMOP_Metric_323::AssembleH(const DenseMatrix &Jpt, const DenseMatrix &DS,
+                                const double weight, DenseMatrix &A) const
+{
+   // P  =   3/2 (I1^1/2) dI1 - 3 sqrt(3) (I3b^-1) dI3b
+   // dP =   3/2 (I1^1/2) ddI1 + 3/4 (I1^-1/2) (dI1 x dI1)
+   //      - 3 sqrt(3) (I3b^-1) ddI3b + 3 sqrt(3) (I3b^-2) (dI3b x dI3b)
+   ie.SetJacobian(Jpt.GetData());
+   ie.SetDerivativeMatrix(DS.Height(), DS.GetData());
+   ie.Assemble_ddI1(weight * 1.5 * sqrt(ie.Get_I1()), A.GetData());
+   ie.Assemble_TProd(weight * 0.75 / sqrt(ie.Get_I1()),
+                     ie.Get_dI1(), A.GetData());
+   ie.Assemble_ddI3b(- weight * 3.0 * sqrt(3.0) / ie.Get_I3b(), A.GetData());
+   ie.Assemble_TProd(weight * 3.0 * sqrt(3.0) / ie.Get_I3b() / ie.Get_I3b(),
+                     ie.Get_dI3b(), A.GetData());
+}
+
 double TMOP_Metric_352::EvalW(const DenseMatrix &Jpt) const
 {
    // mu_352 = 0.5*(det(J) - 1)^2 / (det(J) - tau0)
@@ -989,6 +1209,45 @@ void TMOP_Metric_352::AssembleH(const DenseMatrix &Jpt,
    ie.Assemble_ddI3b(weight*(c - 0.5*c*c), A.GetData());
 }
 
+
+
+
+double TMOP_Metric_360::EvalWMatrixForm(const DenseMatrix &Jpt) const
+{
+   // mu_360 = |J|^3 / 3^(3/2) - det(J)
+   const double fnorm = Jpt.FNorm();
+   return fnorm * fnorm * fnorm / pow(3.0, 1.5) - Jpt.Det();
+}
+
+double TMOP_Metric_360::EvalW(const DenseMatrix &Jpt) const
+{
+   // mu_360 = (I1/3)^(3/2) - I3b.
+   ie.SetJacobian(Jpt.GetData());
+   return pow(ie.Get_I1()/3.0, 1.5) - ie.Get_I3b();
+}
+
+void TMOP_Metric_360::EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const
+{
+   // mu_360 = (I1/3)^(3/2) - I3b.
+   // P      = 3/2 * (I1/3)^1/2 * dI1 / 3 - dI3b
+   //        = 1/2 * (I1/3)^1/2 * dI1 - dI3b.
+   ie.SetJacobian(Jpt.GetData());
+   Add(0.5 * sqrt(ie.Get_I1()/3.0), ie.Get_dI1(), -1.0, ie.Get_dI3b(), P);
+}
+
+void TMOP_Metric_360::AssembleH(const DenseMatrix &Jpt, const DenseMatrix &DS,
+                                const double weight, DenseMatrix &A) const
+{
+   // P  = 1/2 * (I1/3)^1/2 * dI1 - dI3b.
+   // dP = 1/12 * (I1/3)^(-1/2) * (dI1 x dI1) + 1/2 * (I1/3)^1/2 * ddI1 - ddI3b
+   ie.SetJacobian(Jpt.GetData());
+   ie.SetDerivativeMatrix(DS.Height(), DS.GetData());
+   ie.Assemble_TProd(weight / 12.0 / sqrt(ie.Get_I1()/3.0),
+                     ie.Get_dI1(), A.GetData());
+   ie.Assemble_ddI1(weight / 2.0 * sqrt(ie.Get_I1()/3.0), A.GetData());
+   ie.Assemble_ddI3b(-weight, A.GetData());
+}
+
 double TMOP_AMetric_011::EvalW(const DenseMatrix &Jpt) const
 {
    MFEM_VERIFY(Jtr != NULL,
@@ -1378,10 +1637,6 @@ void DiscreteAdaptTC::FinalizeParDiscreteTargetSpec(const ParGridFunction &t)
 
    ParFiniteElementSpace *ptspec_fes = t.ParFESpace();
 
-   adapt_eval->SetParMetaInfo(*ptspec_fes->GetParMesh(),
-                              *ptspec_fes->FEColl(), ncomp);
-   adapt_eval->SetInitialField(*ptspec_fes->GetMesh()->GetNodes(), tspec);
-
    tspec_sav = tspec;
 
    delete tspec_fesv;
@@ -1395,6 +1650,9 @@ void DiscreteAdaptTC::FinalizeParDiscreteTargetSpec(const ParGridFunction &t)
    delete tspec_pgf;
    tspec_pgf = new ParGridFunction(ptspec_fesv, tspec);
    tspec_gf = tspec_pgf;
+
+   adapt_eval->SetParMetaInfo(*ptspec_fes->GetParMesh(), *ptspec_fesv);
+   adapt_eval->SetInitialField(*ptspec_fes->GetMesh()->GetNodes(), tspec);
 }
 
 void DiscreteAdaptTC::ParUpdateAfterMeshTopologyChange()
@@ -1411,8 +1669,7 @@ void DiscreteAdaptTC::ParUpdateAfterMeshTopologyChange()
    tspec.SetDataAndSize(tspec_pgf->GetData(), tspec_pgf->Size());
    tspec_sav = tspec;
 
-   adapt_eval->SetParMetaInfo(*ptspec_fesv->GetParMesh(),
-                              *ptspec_fesv->FEColl(), ncomp);
+   adapt_eval->SetParMetaInfo(*ptspec_fesv->GetParMesh(), *ptspec_fesv);
    adapt_eval->SetInitialField(*ptspec_fesv->GetMesh()->GetNodes(), tspec);
 }
 
@@ -1431,6 +1688,8 @@ void DiscreteAdaptTC::SetTspecAtIndex(int idx, const ParGridFunction &tspec_)
 
 void DiscreteAdaptTC::SetParDiscreteTargetSize(const ParGridFunction &tspec_)
 {
+   MFEM_VERIFY(tspec_.FESpace()->GetOrdering() == Ordering::byNODES,
+               "Discrete target size should be ordered byNodes.");
    if (sizeidx > -1) { SetTspecAtIndex(sizeidx, tspec_); return; }
    sizeidx = ncomp;
    SetDiscreteTargetBase(tspec_);
@@ -1439,6 +1698,8 @@ void DiscreteAdaptTC::SetParDiscreteTargetSize(const ParGridFunction &tspec_)
 
 void DiscreteAdaptTC::SetParDiscreteTargetSkew(const ParGridFunction &tspec_)
 {
+   MFEM_VERIFY(tspec_.FESpace()->GetOrdering() == Ordering::byNODES,
+               "Discrete target skewness should be ordered byNodes.");
    if (skewidx > -1) { SetTspecAtIndex(skewidx, tspec_); return; }
    skewidx = ncomp;
    SetDiscreteTargetBase(tspec_);
@@ -1447,6 +1708,8 @@ void DiscreteAdaptTC::SetParDiscreteTargetSkew(const ParGridFunction &tspec_)
 
 void DiscreteAdaptTC::SetParDiscreteTargetAspectRatio(const ParGridFunction &ar)
 {
+   MFEM_VERIFY(ar.FESpace()->GetOrdering() == Ordering::byNODES,
+               "Discrete target aspect ratio should be ordered byNodes.");
    if (aspectratioidx > -1) { SetTspecAtIndex(aspectratioidx, ar); return; }
    aspectratioidx = ncomp;
    SetDiscreteTargetBase(ar);
@@ -1455,6 +1718,8 @@ void DiscreteAdaptTC::SetParDiscreteTargetAspectRatio(const ParGridFunction &ar)
 
 void DiscreteAdaptTC::SetParDiscreteTargetOrientation(const ParGridFunction &o)
 {
+   MFEM_VERIFY(o.FESpace()->GetOrdering() == Ordering::byNODES,
+               "Discrete target orientation should be ordered byNodes.");
    if (orientationidx > -1) { SetTspecAtIndex(orientationidx, o); return; }
    orientationidx = ncomp;
    SetDiscreteTargetBase(o);
@@ -1506,6 +1771,8 @@ void DiscreteAdaptTC::SetTspecAtIndex(int idx, const GridFunction &tspec_)
 
 void DiscreteAdaptTC::SetSerialDiscreteTargetSize(const GridFunction &tspec_)
 {
+   MFEM_VERIFY(tspec_.FESpace()->GetOrdering() == Ordering::byNODES,
+               "Discrete target size should be ordered byNodes.");
    if (sizeidx > -1) { SetTspecAtIndex(sizeidx, tspec_); return; }
    sizeidx = ncomp;
    SetDiscreteTargetBase(tspec_);
@@ -1514,6 +1781,8 @@ void DiscreteAdaptTC::SetSerialDiscreteTargetSize(const GridFunction &tspec_)
 
 void DiscreteAdaptTC::SetSerialDiscreteTargetSkew(const GridFunction &tspec_)
 {
+   MFEM_VERIFY(tspec_.FESpace()->GetOrdering() == Ordering::byNODES,
+               "Discrete target skewness should be ordered byNodes.");
    if (skewidx > -1) { SetTspecAtIndex(skewidx, tspec_); return; }
    skewidx = ncomp;
    SetDiscreteTargetBase(tspec_);
@@ -1522,6 +1791,8 @@ void DiscreteAdaptTC::SetSerialDiscreteTargetSkew(const GridFunction &tspec_)
 
 void DiscreteAdaptTC::SetSerialDiscreteTargetAspectRatio(const GridFunction &ar)
 {
+   MFEM_VERIFY(ar.FESpace()->GetOrdering() == Ordering::byNODES,
+               "Discrete target aspect ratio should be ordered byNodes.");
    if (aspectratioidx > -1) { SetTspecAtIndex(aspectratioidx, ar); return; }
    aspectratioidx = ncomp;
    SetDiscreteTargetBase(ar);
@@ -1530,6 +1801,8 @@ void DiscreteAdaptTC::SetSerialDiscreteTargetAspectRatio(const GridFunction &ar)
 
 void DiscreteAdaptTC::SetSerialDiscreteTargetOrientation(const GridFunction &o)
 {
+   MFEM_VERIFY(o.FESpace()->GetOrdering() == Ordering::byNODES,
+               "Discrete target orientation should be ordered byNodes.");
    if (orientationidx > -1) { SetTspecAtIndex(orientationidx, o); return; }
    orientationidx = ncomp;
    SetDiscreteTargetBase(o);
@@ -1542,18 +1815,19 @@ void DiscreteAdaptTC::FinalizeSerialDiscreteTargetSpec(const GridFunction &t)
    MFEM_VERIFY(ncomp > 0, "No target specifications have been set!");
 
    const FiniteElementSpace *tspec_fes = t.FESpace();
-   adapt_eval->SetSerialMetaInfo(*tspec_fes->GetMesh(),
-                                 *tspec_fes->FEColl(), ncomp);
-   adapt_eval->SetInitialField(*tspec_fes->GetMesh()->GetNodes(), tspec);
 
    tspec_sav = tspec;
 
    delete tspec_fesv;
    tspec_fesv = new FiniteElementSpace(tspec_fes->GetMesh(),
-                                       tspec_fes->FEColl(), ncomp);
+                                       tspec_fes->FEColl(), ncomp,
+                                       Ordering::byNODES);
 
    delete tspec_gf;
    tspec_gf = new GridFunction(tspec_fesv, tspec);
+
+   adapt_eval->SetSerialMetaInfo(*tspec_fes->GetMesh(), *tspec_fesv);
+   adapt_eval->SetInitialField(*tspec_fes->GetMesh()->GetNodes(), tspec);
 }
 
 void DiscreteAdaptTC::GetDiscreteTargetSpec(GridFunction &tspec_, int idx)
@@ -1577,8 +1851,7 @@ void DiscreteAdaptTC::UpdateAfterMeshTopologyChange()
    tspec.SetDataAndSize(tspec_gf->GetData(), tspec_gf->Size());
    tspec_sav = tspec;
 
-   adapt_eval->SetSerialMetaInfo(*tspec_fesv->GetMesh(),
-                                 *tspec_fesv->FEColl(), ncomp);
+   adapt_eval->SetSerialMetaInfo(*tspec_fesv->GetMesh(), *tspec_fesv);
    adapt_eval->SetInitialField(*tspec_fesv->GetMesh()->GetNodes(), tspec);
 }
 
@@ -1589,21 +1862,23 @@ void DiscreteAdaptTC::SetSerialDiscreteTargetSpec(const GridFunction &tspec_)
 
 
 void DiscreteAdaptTC::UpdateTargetSpecification(const Vector &new_x,
-                                                bool use_flag)
+                                                bool use_flag,
+                                                int new_x_ordering)
 {
    if (use_flag && good_tspec) { return; }
 
    MFEM_VERIFY(tspec.Size() > 0, "Target specification is not set!");
-   adapt_eval->ComputeAtNewPosition(new_x, tspec);
+   adapt_eval->ComputeAtNewPosition(new_x, tspec, new_x_ordering);
    tspec_sav = tspec;
 
    good_tspec = use_flag;
 }
 
 void DiscreteAdaptTC::UpdateTargetSpecification(Vector &new_x,
-                                                Vector &IntData)
+                                                Vector &IntData,
+                                                int new_x_ordering)
 {
-   adapt_eval->ComputeAtNewPosition(new_x, IntData);
+   adapt_eval->ComputeAtNewPosition(new_x, IntData, new_x_ordering);
 }
 
 void DiscreteAdaptTC::UpdateTargetSpecificationAtNode(const FiniteElement &el,
@@ -2233,9 +2508,10 @@ void DiscreteAdaptTC::ComputeElementTargetsGradient(const IntegrationRule &ir,
    Jtrcomp.Clear();
 }
 
-void DiscreteAdaptTC::UpdateGradientTargetSpecification(const Vector &x,
-                                                        const double dx,
-                                                        bool use_flag)
+void DiscreteAdaptTC:: UpdateGradientTargetSpecification(const Vector &x,
+                                                         const double dx,
+                                                         bool use_flag,
+                                                         int x_ordering)
 {
    if (use_flag && good_tspec_grad) { return; }
 
@@ -2248,19 +2524,28 @@ void DiscreteAdaptTC::UpdateGradientTargetSpecification(const Vector &x,
    Vector xtemp = x;
    for (int j = 0; j < dim; j++)
    {
-      for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) += dx; }
+      for (int i = 0; i < cnt; i++)
+      {
+         int idx = x_ordering == Ordering::byNODES ? j*cnt + i : i*dim + j;
+         xtemp(idx) += dx;
+      }
 
       TSpecTemp.NewDataAndSize(tspec_pert1h.GetData() + j*cnt*ncomp, cnt*ncomp);
-      UpdateTargetSpecification(xtemp, TSpecTemp);
+      UpdateTargetSpecification(xtemp, TSpecTemp, x_ordering);
 
-      for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) -= dx; }
+      for (int i = 0; i < cnt; i++)
+      {
+         int idx = x_ordering == Ordering::byNODES ? j*cnt + i : i*dim + j;
+         xtemp(idx) -= dx;
+      }
    }
 
    good_tspec_grad = use_flag;
 }
 
 void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
-                                                       double dx, bool use_flag)
+                                                       double dx, bool use_flag,
+                                                       int x_ordering)
 {
 
    if (use_flag && good_tspec_hess) { return; }
@@ -2278,12 +2563,20 @@ void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
    // T(x+2h)
    for (int j = 0; j < dim; j++)
    {
-      for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) += 2*dx; }
+      for (int i = 0; i < cnt; i++)
+      {
+         int idx = x_ordering == Ordering::byNODES ? j*cnt + i : i*dim + j;
+         xtemp(idx) += 2*dx;
+      }
 
       TSpecTemp.NewDataAndSize(tspec_pert2h.GetData() + j*cnt*ncomp, cnt*ncomp);
-      UpdateTargetSpecification(xtemp, TSpecTemp);
+      UpdateTargetSpecification(xtemp, TSpecTemp, x_ordering);
 
-      for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) -= 2*dx; }
+      for (int i = 0; i < cnt; i++)
+      {
+         int idx = x_ordering == Ordering::byNODES ? j*cnt + i : i*dim + j;
+         xtemp(idx) -= 2*dx;
+      }
    }
 
    // T(x+h,y+h)
@@ -2294,17 +2587,21 @@ void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
       {
          for (int i = 0; i < cnt; i++)
          {
-            xtemp(k1*cnt+i) += dx;
-            xtemp(k2*cnt+i) += dx;
+            int idx1 = x_ordering == Ordering::byNODES ? k1*cnt+i : i*dim + k1;
+            int idx2 = x_ordering == Ordering::byNODES ? k2*cnt+i : i*dim + k2;
+            xtemp(idx1) += dx;
+            xtemp(idx2) += dx;
          }
 
          TSpecTemp.NewDataAndSize(tspec_pertmix.GetData() + j*cnt*ncomp, cnt*ncomp);
-         UpdateTargetSpecification(xtemp, TSpecTemp);
+         UpdateTargetSpecification(xtemp, TSpecTemp, x_ordering);
 
          for (int i = 0; i < cnt; i++)
          {
-            xtemp(k1*cnt+i) -= dx;
-            xtemp(k2*cnt+i) -= dx;
+            int idx1 = x_ordering == Ordering::byNODES ? k1*cnt+i : i*dim + k1;
+            int idx2 = x_ordering == Ordering::byNODES ? k2*cnt+i : i*dim + k2;
+            xtemp(idx1) -= dx;
+            xtemp(idx2) -= dx;
          }
          j++;
       }
@@ -2324,28 +2621,24 @@ DiscreteAdaptTC::~DiscreteAdaptTC()
 }
 
 void AdaptivityEvaluator::SetSerialMetaInfo(const Mesh &m,
-                                            const FiniteElementCollection &fec,
-                                            int num_comp)
+                                            const FiniteElementSpace &f)
 {
    delete fes;
    delete mesh;
    mesh = new Mesh(m, true);
-   fes = new FiniteElementSpace(mesh, &fec, num_comp);
-   dim = fes->GetFE(0)->GetDim();
-   ncomp = num_comp;
+   fes = new FiniteElementSpace(mesh, f.FEColl(),
+                                f.GetVDim(), f.GetOrdering());
 }
 
 #ifdef MFEM_USE_MPI
 void AdaptivityEvaluator::SetParMetaInfo(const ParMesh &m,
-                                         const FiniteElementCollection &fec,
-                                         int num_comp)
+                                         const ParFiniteElementSpace &f)
 {
    delete pfes;
    delete pmesh;
    pmesh = new ParMesh(m, true);
-   pfes  = new ParFiniteElementSpace(pmesh, &fec, num_comp);
-   dim = pfes->GetFE(0)->GetDim();
-   ncomp = num_comp;
+   pfes  = new ParFiniteElementSpace(pmesh, f.FEColl(),
+                                     f.GetVDim(), f.GetOrdering());
 }
 #endif
 
@@ -2429,8 +2722,8 @@ void TMOP_Integrator::EnableAdaptiveLimiting(const GridFunction &z0,
    adapt_lim_coeff = &coeff;
    adapt_lim_eval = &ae;
 
-   adapt_lim_eval->SetSerialMetaInfo(*adapt_lim_gf->FESpace()->GetMesh(),
-                                     *adapt_lim_gf->FESpace()->FEColl(), 1);
+   adapt_lim_eval->SetSerialMetaInfo(*z0.FESpace()->GetMesh(),
+                                     *z0.FESpace());
    adapt_lim_eval->SetInitialField
    (*adapt_lim_gf->FESpace()->GetMesh()->GetNodes(), *adapt_lim_gf);
 }
@@ -2448,7 +2741,7 @@ void TMOP_Integrator::EnableAdaptiveLimiting(const ParGridFunction &z0,
    adapt_lim_eval = &ae;
 
    adapt_lim_eval->SetParMetaInfo(*z0.ParFESpace()->GetParMesh(),
-                                  *z0.ParFESpace()->FEColl(), 1);
+                                  *z0.ParFESpace());
    adapt_lim_eval->SetInitialField
    (*adapt_lim_gf->FESpace()->GetMesh()->GetNodes(), *adapt_lim_gf);
 }
@@ -2466,7 +2759,7 @@ void TMOP_Integrator::EnableSurfaceFitting(const GridFunction &s0,
    surf_fit_eval = &ae;
 
    surf_fit_eval->SetSerialMetaInfo(*s0.FESpace()->GetMesh(),
-                                    *s0.FESpace()->FEColl(), 1);
+                                    *s0.FESpace());
    surf_fit_eval->SetInitialField
    (*surf_fit_gf->FESpace()->GetMesh()->GetNodes(), *surf_fit_gf);
 }
@@ -2484,7 +2777,7 @@ void TMOP_Integrator::EnableSurfaceFitting(const ParGridFunction &s0,
    surf_fit_eval = &ae;
 
    surf_fit_eval->SetParMetaInfo(*s0.ParFESpace()->GetParMesh(),
-                                 *s0.ParFESpace()->FEColl(), 1);
+                                 *s0.ParFESpace());
    surf_fit_eval->SetInitialField
    (*surf_fit_gf->FESpace()->GetMesh()->GetNodes(), *surf_fit_gf);
 }
@@ -2525,7 +2818,7 @@ void TMOP_Integrator::UpdateAfterMeshTopologyChange()
    {
       adapt_lim_gf->Update();
       adapt_lim_eval->SetSerialMetaInfo(*adapt_lim_gf->FESpace()->GetMesh(),
-                                        *adapt_lim_gf->FESpace()->FEColl(), 1);
+                                        *adapt_lim_gf->FESpace());
       adapt_lim_eval->SetInitialField
       (*adapt_lim_gf->FESpace()->GetMesh()->GetNodes(), *adapt_lim_gf);
    }
@@ -2538,7 +2831,7 @@ void TMOP_Integrator::ParUpdateAfterMeshTopologyChange()
    {
       adapt_lim_gf->Update();
       adapt_lim_eval->SetParMetaInfo(*adapt_lim_pgf0->ParFESpace()->GetParMesh(),
-                                     *adapt_lim_pgf0->ParFESpace()->FEColl(), 1);
+                                     *adapt_lim_pgf0->ParFESpace());
       adapt_lim_eval->SetInitialField
       (*adapt_lim_gf->FESpace()->GetMesh()->GetNodes(), *adapt_lim_gf);
    }
@@ -2917,7 +3210,6 @@ void TMOP_Integrator::AssembleElementVectorExact(const FiniteElement &el,
       }
    }
 
-
    Vector d_detW_dx(dim);
    Vector d_Winv_dx(dim);
 
@@ -3662,19 +3954,23 @@ void TMOP_Integrator::ComputeMinJac(const Vector &x,
    dx = detv_avg_min / dxscale;
 }
 
-void TMOP_Integrator::UpdateAfterMeshPositionChange(const Vector &new_x)
+void TMOP_Integrator::UpdateAfterMeshPositionChange(const Vector &new_x,
+                                                    int new_x_ordering)
 {
    if (discr_tc)
    {
       PA.Jtr_needs_update = true;
    }
    // Update adapt_lim_gf if adaptive limiting is enabled.
-   if (adapt_lim_gf) { adapt_lim_eval->ComputeAtNewPosition(new_x, *adapt_lim_gf); }
+   if (adapt_lim_gf)
+   {
+      adapt_lim_eval->ComputeAtNewPosition(new_x, *adapt_lim_gf, new_x_ordering);
+   }
 
    // Update surf_fit_gf if surface fitting is enabled.
    if (surf_fit_gf)
    {
-      surf_fit_eval->ComputeAtNewPosition(new_x, *surf_fit_gf);
+      surf_fit_eval->ComputeAtNewPosition(new_x, *surf_fit_gf, new_x_ordering);
    }
 }
 
@@ -3701,9 +3997,9 @@ void TMOP_Integrator::EnableFiniteDifferences(const GridFunction &x)
    ComputeFDh(x,*fes);
    if (discr_tc)
    {
-      discr_tc->UpdateTargetSpecification(x);
-      discr_tc->UpdateGradientTargetSpecification(x, dx);
-      discr_tc->UpdateHessianTargetSpecification(x, dx);
+      discr_tc->UpdateTargetSpecification(x, false, fes->GetOrdering());
+      discr_tc->UpdateGradientTargetSpecification(x, dx, false, fes->GetOrdering());
+      discr_tc->UpdateHessianTargetSpecification(x, dx, false, fes->GetOrdering());
    }
 }
 
@@ -3715,9 +4011,9 @@ void TMOP_Integrator::EnableFiniteDifferences(const ParGridFunction &x)
    ComputeFDh(x,*pfes);
    if (discr_tc)
    {
-      discr_tc->UpdateTargetSpecification(x);
-      discr_tc->UpdateGradientTargetSpecification(x, dx);
-      discr_tc->UpdateHessianTargetSpecification(x, dx);
+      discr_tc->UpdateTargetSpecification(x, false, pfes->GetOrdering());
+      discr_tc->UpdateGradientTargetSpecification(x, dx, false, pfes->GetOrdering());
+      discr_tc->UpdateHessianTargetSpecification(x, dx, false, pfes->GetOrdering());
    }
 }
 #endif

fem/tmop.hpp

@@ -26,8 +26,8 @@ protected:
    const DenseMatrix *Jtr; /**< Jacobian of the reference-element to
                                 target-element transformation. */
 
-   /** @brief The method SetTransformation() is hidden for TMOP_QualityMetric%s,
-       because it is not used. */
+   /** @brief The method HyperelasticModel::SetTransformation() is hidden
+       for TMOP_QualityMetric%s, because it is not used. */
    void SetTransformation(ElementTransformation &) { }
 
 public:
@@ -42,7 +42,13 @@ public:
        Jpt. */
    virtual void SetTargetJacobian(const DenseMatrix &Jtr_) { Jtr = &Jtr_; }
 
-   /** @brief Evaluate the strain energy density function, W = W(Jpt).
+   /** @brief Evaluates the metric in matrix form (opposed to invariant form).
+       Used for validating the invariant evaluations. */
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const
+   { return -1.0; /* not implemented -> checks would fail. */ }
+
+   /** @brief Evaluate the strain energy density function, W = W(Jpt), by using
+       the 2D or 3D matrix invariants, see linalg/invariants.hpp.
        @param[in] Jpt  Represents the target->physical transformation
                        Jacobian matrix. */
    virtual double EvalW(const DenseMatrix &Jpt) const = 0;
@@ -64,13 +70,11 @@ public:
 
        Computes weight * d(dW_dxi)_d(xj) at the current point, for all i and j,
        where x1 ... xn are the FE dofs. This function is usually defined using
-       the matrix invariants and their derivatives.
-   */
+       the matrix invariants and their derivatives. */
    virtual void AssembleH(const DenseMatrix &Jpt, const DenseMatrix &DS,
                           const double weight, DenseMatrix &A) const = 0;
 
-   /** @brief Return the metric ID.
-    */
+   /** @brief Return the metric ID. */
    virtual int Id() const { return 0; }
 };
 
@@ -96,6 +100,8 @@ public:
       }
    }
 
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
    virtual double EvalW(const DenseMatrix &Jpt) const;
 
    virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
@@ -113,7 +119,7 @@ public:
 /// and mu_hat = (mu/2phi(tau,ep)) where
 /// 2phi(tau,ep) = 1, when                                 when BarrierType = None,
 ///             = 2*(tau - min(alpha*min(tau)-detT_ep,0)), when BarrierType = Shifted
-///             = tau^2 + sqrt(tau^2 + ep^2),              when BarrierType = Pseuso
+///             = tau^2 + sqrt(tau^2 + ep^2),              when BarrierType = Pseudo
 /// where tau = det(T), and max(mu_hat) and min(tau) are computed over the
 /// entire mesh.
 /// Ultimately, this metric can be used for mesh untangling with the BarrierType
@@ -272,7 +278,10 @@ protected:
    mutable InvariantsEvaluator2D<double> ie;
 
 public:
-   // W = 0.5|J|^2 / det(J) - 1.
+   // W = 0.5 |J|^2 / det(J) - 1.
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = 0.5 I1b - 1.
    virtual double EvalW(const DenseMatrix &Jpt) const;
 
    virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
@@ -428,7 +437,9 @@ protected:
 
 public:
    // W = |J^t J|^2 / det(J)^2 - 2|J|^2 / det(J) + 2
-   //   = I1b (I1b - 2).
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = I1b (I1b - 2).
    virtual double EvalW(const DenseMatrix &Jpt) const;
 
    virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
@@ -570,14 +581,17 @@ public:
                           const double weight, DenseMatrix &A) const;
 };
 
-/// 3D barrier Shape (S) metric.
+/// 3D barrier Shape (S) metric, well-posed (polyconvex & invex).
 class TMOP_Metric_301 : public TMOP_QualityMetric
 {
 protected:
    mutable InvariantsEvaluator3D<double> ie;
 
 public:
-   // W = |J| |J^-1| / 3 - 1.
+   // W = 1/3 |J| |J^-1| - 1.
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = 1/3 sqrt(I1b * I2b) - 1
    virtual double EvalW(const DenseMatrix &Jpt) const;
 
    virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
@@ -586,14 +600,17 @@ public:
                           const double weight, DenseMatrix &A) const;
 };
 
-/// 3D barrier Shape (S) metric.
+/// 3D barrier Shape (S) metric, well-posed (polyconvex & invex).
 class TMOP_Metric_302 : public TMOP_QualityMetric
 {
 protected:
    mutable InvariantsEvaluator3D<double> ie;
 
 public:
-   // W = |J|^2 |J^-1|^2 / 9 - 1.
+   // W = |J|^2 |J^{-1}|^2 / 9 - 1.
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = I1b * I2b / 9 - 1.
    virtual double EvalW(const DenseMatrix &Jpt) const;
 
    virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
@@ -604,14 +621,17 @@ public:
    virtual int Id() const { return 302; }
 };
 
-/// 3D barrier Shape (S) metric.
+/// 3D barrier Shape (S) metric, well-posed (polyconvex & invex).
 class TMOP_Metric_303 : public TMOP_QualityMetric
 {
 protected:
    mutable InvariantsEvaluator3D<double> ie;
 
 public:
-   // W = |J|^2 / (3 * det(J)^(2/3)) - 1.
+   // W = |J|^2 / 3 / det(J)^(2/3) - 1.
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = I1b / 3 - 1.
    virtual double EvalW(const DenseMatrix &Jpt) const;
 
    virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
@@ -622,6 +642,27 @@ public:
    virtual int Id() const { return 303; }
 };
 
+/// 3D barrier Shape (S) metric, well-posed (polyconvex & invex).
+class TMOP_Metric_304 : public TMOP_QualityMetric
+{
+protected:
+   mutable InvariantsEvaluator3D<double> ie;
+
+public:
+   // W = |J|^3 / 3^(3/2) / det(J) - 1.
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = (I1b/3)^3/2 - 1.
+   virtual double EvalW(const DenseMatrix &Jpt) const;
+
+   virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
+
+   virtual void AssembleH(const DenseMatrix &Jpt, const DenseMatrix &DS,
+                          const double weight, DenseMatrix &A) const;
+
+   virtual int Id() const { return 304; }
+};
+
 /// 3D Size (V) untangling metric.
 class TMOP_Metric_311 : public TMOP_QualityMetric
 {
@@ -662,7 +703,7 @@ public:
    virtual int Id() const { return 313; }
 };
 
-/// 3D non-barrier Size (V) metric.
+/// 3D non-barrier metric without a type.
 class TMOP_Metric_315 : public TMOP_QualityMetric
 {
 protected:
@@ -680,16 +721,17 @@ public:
    virtual int Id() const { return 315; }
 };
 
-/// 3D barrier Size (V) metric.
+/// 3D barrier metric without a type.
 class TMOP_Metric_316 : public TMOP_QualityMetric
 {
 protected:
    mutable InvariantsEvaluator3D<double> ie;
 
 public:
-   // W = 0.5( sqrt(det(J)) - 1 / sqrt(det(J)) )^2
-   //   = 0.5( det(J) - 1 )^2 / det(J)
-   //   = 0.5( det(J) + 1/det(J) ) - 1.
+   // W = 0.5 (det(J) + 1/det(J)) - 1.
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = 0.5 (I3b + 1/I3b) - 1.
    virtual double EvalW(const DenseMatrix &Jpt) const;
 
    virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
@@ -698,7 +740,7 @@ public:
                           const double weight, DenseMatrix &A) const;
 };
 
-/// 3D barrier Shape+Size (VS) metric.
+/// 3D barrier Shape+Size (VS) metric, well-posed (invex).
 class TMOP_Metric_321 : public TMOP_QualityMetric
 {
 protected:
@@ -706,6 +748,9 @@ protected:
 
 public:
    // W = |J - J^-t|^2.
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = I1 + I2/I3 - 6.
    virtual double EvalW(const DenseMatrix &Jpt) const;
 
    virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
@@ -716,6 +761,48 @@ public:
    virtual int Id() const { return 321; }
 };
 
+/// 3D barrier Shape+Size (VS) metric, well-posed (invex).
+class TMOP_Metric_322 : public TMOP_QualityMetric
+{
+protected:
+   mutable InvariantsEvaluator3D<double> ie;
+
+public:
+   // W = |J - adjJ^-t|^2.
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = I1b / (I3b^-1/3) / 6 + I2b (I3b^1/3) / 6 - 1
+   virtual double EvalW(const DenseMatrix &Jpt) const;
+
+   virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
+
+   virtual void AssembleH(const DenseMatrix &Jpt, const DenseMatrix &DS,
+                          const double weight, DenseMatrix &A) const;
+
+   virtual int Id() const { return 322; }
+};
+
+/// 3D barrier Shape+Size (VS) metric, well-posed (invex).
+class TMOP_Metric_323 : public TMOP_QualityMetric
+{
+protected:
+   mutable InvariantsEvaluator3D<double> ie;
+
+public:
+   // W = |J|^3 - 3 sqrt(3) ln(det(J)) - 3 sqrt(3).
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = I1^3/2 - 3 sqrt(3) ln(I3b) - 3 sqrt(3).
+   virtual double EvalW(const DenseMatrix &Jpt) const;
+
+   virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
+
+   virtual void AssembleH(const DenseMatrix &Jpt, const DenseMatrix &DS,
+                          const double weight, DenseMatrix &A) const;
+
+   virtual int Id() const { return 323; }
+};
+
 /// 3D barrier Shape+Size (VS) metric (polyconvex).
 class TMOP_Metric_328 : public TMOP_Combo_QualityMetric
 {
@@ -760,7 +847,7 @@ public:
    virtual ~TMOP_Metric_332() { delete sh_metric; delete sz_metric; }
 };
 
-/// 3D barrier Shape+Size (VS) metric (polyconvex).
+/// 3D barrier Shape+Size (VS) metric, well-posed (polyconvex).
 class TMOP_Metric_333 : public TMOP_Combo_QualityMetric
 {
 protected:
@@ -781,7 +868,7 @@ public:
    virtual ~TMOP_Metric_333() { delete sh_metric; delete sz_metric; }
 };
 
-/// 3D barrier Shape+Size (VS) metric (polyconvex).
+/// 3D barrier Shape+Size (VS) metric, well-posed (polyconvex).
 class TMOP_Metric_334 : public TMOP_Combo_QualityMetric
 {
 protected:
@@ -799,10 +886,37 @@ public:
       AddQualityMetric(sz_metric, gamma_);
    }
 
+   virtual int Id() const { return 334; }
+   double GetGamma() const { return gamma; }
+
    virtual ~TMOP_Metric_334() { delete sh_metric; delete sz_metric; }
 };
 
-/// Shifted barrier form of 3D metric 16 (volume, ideal barrier metric), 3D
+/// 3D barrier Shape+Size (VS) metric, well-posed (polyconvex).
+class TMOP_Metric_347 : public TMOP_Combo_QualityMetric
+{
+protected:
+   mutable InvariantsEvaluator2D<double> ie;
+   double gamma;
+   TMOP_QualityMetric *sh_metric, *sz_metric;
+
+public:
+   TMOP_Metric_347(double gamma_) : gamma(gamma_),
+      sh_metric(new TMOP_Metric_304),
+      sz_metric(new TMOP_Metric_316)
+   {
+      // (1-gamma) mu_304 + gamma mu_316
+      AddQualityMetric(sh_metric, 1.-gamma_);
+      AddQualityMetric(sz_metric, gamma_);
+   }
+
+   virtual int Id() const { return 347; }
+   double GetGamma() const { return gamma; }
+
+   virtual ~TMOP_Metric_347() { delete sh_metric; delete sz_metric; }
+};
+
+/// 3D shifted barrier form of metric 316 (not typed).
 class TMOP_Metric_352 : public TMOP_QualityMetric
 {
 protected:
@@ -821,6 +935,27 @@ public:
                           const double weight, DenseMatrix &A) const;
 };
 
+/// 3D non-barrier Shape (S) metric.
+class TMOP_Metric_360 : public TMOP_QualityMetric
+{
+protected:
+   mutable InvariantsEvaluator3D<double> ie;
+
+public:
+   // W = |J|^3 / 3^(3/2) - det(J).
+   virtual double EvalWMatrixForm(const DenseMatrix &Jpt) const;
+
+   // W = (I1b/3)^3/2 - 1.
+   virtual double EvalW(const DenseMatrix &Jpt) const;
+
+   virtual void EvalP(const DenseMatrix &Jpt, DenseMatrix &P) const;
+
+   virtual void AssembleH(const DenseMatrix &Jpt, const DenseMatrix &DS,
+                          const double weight, DenseMatrix &A) const;
+
+   virtual int Id() const { return 360; }
+};
+
 /// A-metrics
 /// 2D barrier Shape (S) metric (polyconvex).
 class TMOP_AMetric_011 : public TMOP_QualityMetric
@@ -987,8 +1122,6 @@ protected:
    ParFiniteElementSpace *pfes;
 #endif
 
-   int dim, ncomp;
-
 public:
    AdaptivityEvaluator() : mesh(NULL), fes(NULL)
    {
@@ -999,15 +1132,15 @@ public:
    }
    virtual ~AdaptivityEvaluator();
 
-   /** Specifies the Mesh and FiniteElementCollection of the solution that will
+   /** Specifies the Mesh and FiniteElementSpace of the solution that will
        be evaluated. The given mesh will be copied into the internal object. */
    void SetSerialMetaInfo(const Mesh &m,
-                          const FiniteElementCollection &fec, int num_comp);
+                          const FiniteElementSpace &f);
 
 #ifdef MFEM_USE_MPI
    /// Parallel version of SetSerialMetaInfo.
    void SetParMetaInfo(const ParMesh &m,
-                       const FiniteElementCollection &fec, int num_comp);
+                       const ParFiniteElementSpace &f);
 #endif
 
    // TODO use GridFunctions to make clear it's on the ldofs?
@@ -1015,7 +1148,8 @@ public:
                                 const Vector &init_field) = 0;
 
    virtual void ComputeAtNewPosition(const Vector &new_nodes,
-                                     Vector &new_field) = 0;
+                                     Vector &new_field,
+                                     int new_nodes_ordering = Ordering::byNODES) = 0;
 
    void ClearGeometricFactors();
 };
@@ -1216,7 +1350,7 @@ protected:
    // Discrete target specification.
    // Data is owned, updated by UpdateTargetSpecification.
    int ncomp, sizeidx, skewidx, aspectratioidx, orientationidx;
-   Vector tspec;             //eta(x)
+   Vector tspec;             //eta(x) - we enforce Ordering::byNODES
    Vector tspec_sav;
    Vector tspec_pert1h;      //eta(x+h)
    Vector tspec_pert2h;      //eta(x+2*h)
@@ -1330,9 +1464,11 @@ public:
    /** Used to update the target specification after the mesh has changed. The
        new mesh positions are given by new_x. If @a use_flags is true, repeated
        calls won't do anything until ResetUpdateFlags() is called. */
-   void UpdateTargetSpecification(const Vector &new_x, bool use_flag = false);
+   void UpdateTargetSpecification(const Vector &new_x, bool use_flag = false,
+                                  int new_x_ordering=Ordering::byNODES);
 
-   void UpdateTargetSpecification(Vector &new_x, Vector &IntData);
+   void UpdateTargetSpecification(Vector &new_x, Vector &IntData,
+                                  int new_x_ordering=Ordering::byNODES);
 
    void UpdateTargetSpecificationAtNode(const FiniteElement &el,
                                         ElementTransformation &T,
@@ -1346,13 +1482,15 @@ public:
        If @a use_flags is true, repeated calls won't do anything until
        ResetUpdateFlags() is called. */
    void UpdateGradientTargetSpecification(const Vector &x, double dx,
-                                          bool use_flag = false);
+                                          bool use_flag = false,
+                                          int x_ordering = Ordering::byNODES);
    /** Used for finite-difference based computations. Computes the target
        specifications after two mesh perturbations in x and/or y direction.
        If @a use_flags is true, repeated calls won't do anything until
        ResetUpdateFlags() is called. */
    void UpdateHessianTargetSpecification(const Vector &x, double dx,
-                                         bool use_flag = false);
+                                         bool use_flag = false,
+                                         int x_ordering = Ordering::byNODES);
 
    void SetAdaptivityEvaluator(AdaptivityEvaluator *ae)
    {
@@ -1587,7 +1725,8 @@ protected:
    void ComputeFDh(const Vector &x, const FiniteElementSpace &fes);
    void ComputeMinJac(const Vector &x, const FiniteElementSpace &fes);
 
-   void UpdateAfterMeshPositionChange(const Vector &new_x);
+   void UpdateAfterMeshPositionChange(const Vector &new_x,
+                                      int new_x_ordering = Ordering::byNODES);
 
    void DisableLimiting()
    {

fem/tmop/tmop_pa.cpp

@@ -192,7 +192,7 @@ void TMOP_Integrator::AssemblePA(const FiniteElementSpace &fes)
    const FiniteElement &fe = *fes.GetFE(0);
    PA.ir = &EnergyIntegrationRule(fe);
    const IntegrationRule &ir = *PA.ir;
-   MFEM_ASSERT(fes.GetOrdering() == Ordering::byNODES,
+   MFEM_VERIFY(fes.GetOrdering() == Ordering::byNODES,
                "PA Only supports Ordering::byNODES!");
 
    const int nq = PA.nq = ir.GetNPoints();

fem/tmop_tools.cpp

@@ -27,17 +27,43 @@ void AdvectorCG::SetInitialField(const Vector &init_nodes,
 }
 
 void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
-                                      Vector &new_field)
+                                      Vector &new_field,
+                                      int new_nodes_ordering)
 {
+   FiniteElementSpace *space = fes;
+#ifdef MFEM_USE_MPI
+   if (pfes) { space = pfes; }
+#endif
+   int fes_ordering = space->GetOrdering(),
+       ncomp = space->GetVDim();
+
    // TODO: Implement for AMR meshes.
-   const int pnt_cnt = new_field.Size()/ncomp;
+   const int pnt_cnt = field0.Size() / ncomp;
 
    new_field = field0;
    Vector new_field_temp;
    for (int i = 0; i < ncomp; i++)
    {
-      new_field_temp.MakeRef(new_field, i*pnt_cnt, pnt_cnt);
+      if (fes_ordering == Ordering::byNODES)
+      {
+         new_field_temp.MakeRef(new_field, i*pnt_cnt, pnt_cnt);
+      }
+      else
+      {
+         new_field_temp.SetSize(pnt_cnt);
+         for (int j = 0; j < pnt_cnt; j++)
+         {
+            new_field_temp(j) = new_field(i + j*ncomp);
+         }
+      }
       ComputeAtNewPositionScalar(new_nodes, new_field_temp);
+      if (fes_ordering == Ordering::byVDIM)
+      {
+         for (int j = 0; j < pnt_cnt; j++)
+         {
+            new_field(i + j*ncomp) = new_field_temp(j);
+         }
+      }
    }
 
    field0 = new_field;
@@ -98,7 +124,7 @@ void AdvectorCG::ComputeAtNewPositionScalar(const Vector &new_nodes,
    for (int i = 0; i < s; i++)
    {
       double vel = 0.;
-      for (int j = 0; j < dim; j++)
+      for (int j = 0; j < m->Dimension(); j++)
       {
          vel += u(i+j*s)*u(i+j*s);
       }
@@ -190,11 +216,7 @@ void SerialAdvectorCGOper::Mult(const Vector &ind, Vector &di_dt) const
    // Move the mesh.
    const double t = GetTime();
    add(x0, t, u, x_now);
-
-   if (al == AssemblyLevel::PARTIAL)
-   {
-      K.FESpace()->GetMesh()->DeleteGeometricFactors();
-   }
+   K.FESpace()->GetMesh()->NodesUpdated();
 
    // Assemble on the new mesh.
    K.BilinearForm::operator=(0.0);
@@ -263,11 +285,7 @@ void ParAdvectorCGOper::Mult(const Vector &ind, Vector &di_dt) const
    // Move the mesh.
    const double t = GetTime();
    add(x0, t, u, x_now);
-
-   if (al == AssemblyLevel::PARTIAL)
-   {
-      K.ParFESpace()->GetParMesh()->DeleteGeometricFactors();
-   }
+   K.ParFESpace()->GetParMesh()->NodesUpdated();
 
    // Assemble on the new mesh.
    K.BilinearForm::operator=(0.0);
@@ -347,14 +365,13 @@ void InterpolatorFP::SetInitialField(const Vector &init_nodes,
 
    field0_gf.SetSpace(f);
    field0_gf = init_field;
-
-   dim = f->GetFE(0)->GetDim();
 }
 
 void InterpolatorFP::ComputeAtNewPosition(const Vector &new_nodes,
-                                          Vector &new_field)
+                                          Vector &new_field,
+                                          int new_nodes_ordering)
 {
-   finder->Interpolate(new_nodes, field0_gf, new_field);
+   finder->Interpolate(new_nodes, field0_gf, new_field, new_nodes_ordering);
 }
 
 #endif
@@ -701,13 +718,13 @@ void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
          ti = dynamic_cast<TMOP_Integrator *>(integs[i]);
          if (ti)
          {
-            ti->UpdateAfterMeshPositionChange(x_loc);
+            ti->UpdateAfterMeshPositionChange(x_loc, pfesc->GetOrdering());
             ti->ComputeFDh(x_loc, *pfesc);
             if (compute_metric_quantile_flag)
             {
                ti->ComputeUntangleMetricQuantiles(x_loc, *pfesc);
             }
-            UpdateDiscreteTC(*ti, x_loc);
+            UpdateDiscreteTC(*ti, x_loc, pfesc->GetOrdering());
          }
          co = dynamic_cast<TMOPComboIntegrator *>(integs[i]);
          if (co)
@@ -715,13 +732,13 @@ void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
             Array<TMOP_Integrator *> ati = co->GetTMOPIntegrators();
             for (int j = 0; j < ati.Size(); j++)
             {
-               ati[j]->UpdateAfterMeshPositionChange(x_loc);
+               ati[j]->UpdateAfterMeshPositionChange(x_loc, pfesc->GetOrdering());
                ati[j]->ComputeFDh(x_loc, *pfesc);
                if (compute_metric_quantile_flag)
                {
                   ati[j]->ComputeUntangleMetricQuantiles(x_loc, *pfesc);
                }
-               UpdateDiscreteTC(*ati[j], x_loc);
+               UpdateDiscreteTC(*ati[j], x_loc, pfesc->GetOrdering());
             }
          }
       }
@@ -746,13 +763,13 @@ void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
          ti = dynamic_cast<TMOP_Integrator *>(integs[i]);
          if (ti)
          {
-            ti->UpdateAfterMeshPositionChange(x_loc);
+            ti->UpdateAfterMeshPositionChange(x_loc, fesc->GetOrdering());
             ti->ComputeFDh(x_loc, *fesc);
             if (compute_metric_quantile_flag)
             {
                ti->ComputeUntangleMetricQuantiles(x_loc, *fesc);
             }
-            UpdateDiscreteTC(*ti, x_loc);
+            UpdateDiscreteTC(*ti, x_loc, fesc->GetOrdering());
          }
          co = dynamic_cast<TMOPComboIntegrator *>(integs[i]);
          if (co)
@@ -760,13 +777,13 @@ void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
             Array<TMOP_Integrator *> ati = co->GetTMOPIntegrators();
             for (int j = 0; j < ati.Size(); j++)
             {
-               ati[j]->UpdateAfterMeshPositionChange(x_loc);
+               ati[j]->UpdateAfterMeshPositionChange(x_loc, fesc->GetOrdering());
                ati[j]->ComputeFDh(x_loc, *fesc);
                if (compute_metric_quantile_flag)
                {
                   ati[j]->ComputeUntangleMetricQuantiles(x_loc, *fesc);
                }
-               UpdateDiscreteTC(*ati[j], x_loc);
+               UpdateDiscreteTC(*ati[j], x_loc, fesc->GetOrdering());
             }
          }
       }
@@ -809,18 +826,19 @@ void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
 }
 
 void TMOPNewtonSolver::UpdateDiscreteTC(const TMOP_Integrator &ti,
-                                        const Vector &x_new) const
+                                        const Vector &x_new,
+                                        int x_ordering) const
 {
    const bool update_flag = true;
    DiscreteAdaptTC *discrtc = ti.GetDiscreteAdaptTC();
    if (discrtc)
    {
-      discrtc->UpdateTargetSpecification(x_new, update_flag);
+      discrtc->UpdateTargetSpecification(x_new, update_flag, x_ordering);
       if (ti.GetFDFlag())
       {
          double dx = ti.GetFDh();
-         discrtc->UpdateGradientTargetSpecification(x_new, dx, update_flag);
-         discrtc->UpdateHessianTargetSpecification(x_new, dx, update_flag);
+         discrtc->UpdateGradientTargetSpecification(x_new, dx, update_flag, x_ordering);
+         discrtc->UpdateHessianTargetSpecification(x_new, dx, update_flag, x_ordering);
       }
    }
 }

fem/tmop_tools.hpp

@@ -42,7 +42,8 @@ public:
                                 const Vector &init_field);
 
    virtual void ComputeAtNewPosition(const Vector &new_nodes,
-                                     Vector &new_field);
+                                     Vector &new_field,
+                                     int new_nodes_ordering = Ordering::byNODES);
 
    /// Set the memory type used for large memory allocations. This memory type
    /// is used when constructing the AdvectorCGOper but currently only for the
@@ -57,7 +58,6 @@ private:
    Vector nodes0;
    GridFunction field0_gf;
    FindPointsGSLIB *finder;
-   int dim;
 public:
    InterpolatorFP() : finder(NULL) { }
 
@@ -65,7 +65,8 @@ public:
                                 const Vector &init_field);
 
    virtual void ComputeAtNewPosition(const Vector &new_nodes,
-                                     Vector &new_field);
+                                     Vector &new_field,
+                                     int new_nodes_ordering = Ordering::byNODES);
 
    ~InterpolatorFP()
    {
@@ -157,7 +158,8 @@ protected:
       return ir;
    }
 
-   void UpdateDiscreteTC(const TMOP_Integrator &ti, const Vector &x_new) const;
+   void UpdateDiscreteTC(const TMOP_Integrator &ti, const Vector &x_new,
+                         int x_ordering = Ordering::byNODES) const;
 
    double ComputeMinDet(const Vector &x_loc,
                         const FiniteElementSpace &fes) const;

fem/transfer.cpp

@@ -1451,13 +1451,11 @@ TrueTransferOperator::TrueTransferOperator(const FiniteElementSpace& lFESpace_,
    {
       tmpL.SetSize(lFESpace_.GetVSize());
       tmpH.SetSize(hFESpace_.GetVSize());
-      R->EnsureMultTranspose();
    }
    // P can be null and R not null
    else if (R)
    {
       tmpH.SetSize(hFESpace_.GetVSize());
-      R->EnsureMultTranspose();
    }
 }
 

general/mem_manager.cpp

@@ -310,10 +310,11 @@ inline uintptr_t MmuLengthP(const void *ptr, const size_t bytes)
 /// The protected access error, used for the host
 static void MmuError(int, siginfo_t *si, void*)
 {
+   constexpr size_t buf_size = 64;
    fflush(0);
-   char str[64];
+   char str[buf_size];
    const void *ptr = si->si_addr;
-   sprintf(str, "Error while accessing address %p!", ptr);
+   snprintf(str, buf_size, "Error while accessing address %p!", ptr);
    mfem::out << std::endl << "An illegal memory access was made!";
    MFEM_ABORT(str);
 }
@@ -1654,9 +1655,9 @@ int MemoryManager::CompareHostAndDevice_(void *h_ptr, size_t size,
                  mm.GetAliasDevicePtr(h_ptr, size, false) :
                  mm.GetDevicePtr(h_ptr, size, false);
    char *h_buf = new char[size];
-#ifdef MFEM_USE_CUDA
+#if defined(MFEM_USE_CUDA)
    CuMemcpyDtoH(h_buf, d_ptr, size);
-#elif MFE_USE_HIP
+#elif defined(MFEM_USE_HIP)
    HipMemcpyDtoH(h_buf, d_ptr, size);
 #else
    std::memcpy(h_buf, d_ptr, size);

general/mem_manager.hpp

@@ -119,9 +119,9 @@ MemoryClass operator*(MemoryClass mc1, MemoryClass mc2);
     - Pointer arithmetic is not supported, MakeAlias() should be used instead.
     - Const Memory object does not allow modification of the content
       (unlike e.g. a const pointer).
-    - Move constructor and assignement will transfer ownership flags, and
+    - Move constructor and assignment will transfer ownership flags, and
       Reset() the moved Memory object.
-    - Copy constructor and assignement copy flags. This may result in two Memory
+    - Copy constructor and assignment copy flags. This may result in two Memory
       objects owning the data which is an invalid state. This invalid state MUST
       be resolved by users manually using SetHostPtrOwner(),
       SetDevicePtrOwner(), or ClearOwnerFlags(). It is also possible to call
@@ -1141,7 +1141,7 @@ inline void Memory<T>::SyncAlias(const Memory &base, int alias_size) const
 template <typename T>
 inline MemoryType Memory<T>::GetMemoryType() const
 {
-   if (!(flags & VALID_DEVICE)) { return h_mt; }
+   if (h_ptr == nullptr || !(flags & VALID_DEVICE)) { return h_mt; }
    return MemoryManager::GetDeviceMemoryType_(h_ptr, flags & ALIAS);
 }
 

general/zstr.hpp

@@ -5,7 +5,7 @@
 
 // Original version, https://github.com/mateidavid/zstr, distributed under MIT
 // license. This file is a combination of the zstr.hpp and strict_fstream.hpp
-// files in the original src/ directory with additional MFEM modifactions.
+// files in the original src/ directory with additional MFEM modifications.
 
 // The MIT License (MIT)
 //
@@ -60,9 +60,9 @@ namespace strict_fstream
 {
 
 // Overloaded error checks to handle POSIX and GNU strerror_r
-inline char* check_strerror_r(int r, char* buff, int err)
+inline char* check_strerror_r(int r, char* buff, size_t buff_size)
 {
-   if (r) { sprintf(buff, "unknown error: %d", err); }
+   if (r) { snprintf(buff, buff_size, "unknown error: %d", r); }
    return buff;
 }
 

linalg/CMakeLists.txt

@@ -14,6 +14,7 @@ list(APPEND SRCS
   blockmatrix.cpp
   blockoperator.cpp
   blockvector.cpp
+  complex_densemat.cpp
   complex_operator.cpp
   constraints.cpp
   densemat.cpp
@@ -33,6 +34,7 @@ list(APPEND HDRS
   blockmatrix.hpp
   blockoperator.hpp
   blockvector.hpp
+  complex_densemat.hpp
   complex_operator.hpp
   constraints.hpp
   densemat.hpp

linalg/complex_densemat.hpp

@@ -0,0 +1,246 @@
+// Copyright (c) 2010-2022, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#ifndef MFEM_COMPLEX_DENSEMAT
+#define MFEM_COMPLEX_DENSEMAT
+
+#include "complex_operator.hpp"
+#include <complex>
+
+namespace mfem
+{
+
+/** @brief Specialization of the ComplexOperator built from a pair of
+    Dense Matrices.  The purpose of this specialization is to support
+    the inverse of a ComplexDenseMatrix and various MatMat operations
+    See ComplexOperator documentation for more information.
+ */
+class ComplexDenseMatrix : public ComplexOperator
+{
+
+public:
+   ComplexDenseMatrix(DenseMatrix * A_Real, DenseMatrix * A_Imag,
+                      bool ownReal, bool ownImag, Convention convention = HERMITIAN)
+      : ComplexOperator(A_Real, A_Imag, ownReal, ownImag, convention)
+   { }
+
+   virtual DenseMatrix & real();
+   virtual DenseMatrix & imag();
+
+   virtual const DenseMatrix & real() const;
+   virtual const DenseMatrix & imag() const;
+
+   /** Combine the blocks making up this complex operator into a single
+       DenseMatrix. Note that this combined operator requires roughly
+       twice the memory of the block structured operator. */
+   DenseMatrix * GetSystemMatrix() const;
+
+   virtual Type GetType() const { return Complex_DenseMat; }
+
+   ComplexDenseMatrix * ComputeInverse();
+
+};
+
+/// Matrix matrix multiplication.  A = B * C.
+ComplexDenseMatrix * Mult(const ComplexDenseMatrix &B,
+                          const ComplexDenseMatrix &C);
+
+/// Multiply the complex conjugate transpose of a matrix A with a matrix B. A^H*B
+ComplexDenseMatrix * MultAtB(const ComplexDenseMatrix &A,
+                             const ComplexDenseMatrix &B);
+
+
+/** Abstract class that can compute factorization of external data and perform various
+    operations with the factored data. */
+class ComplexFactors
+{
+protected:
+
+   // returns a new complex array
+   std::complex<double> * RealToComplex(int m, const double * x_r,
+                                        const double * x_i) const;
+   // copies the given complex array to real and imag arrays
+   void ComplexToReal(int m, const std::complex<double> * x, double * x_r,
+                      double * x_i) const;
+
+public:
+
+   double *data_r = nullptr;
+   double *data_i = nullptr;
+   std::complex<double> * data = nullptr;
+
+   ComplexFactors() { }
+
+   ComplexFactors(double *data_r_, double *data_i_)
+      : data_r(data_r_), data_i(data_i_) { }
+
+   void SetComplexData(int m);
+
+   void ResetComplexData(int m)
+   {
+      delete [] data; data = nullptr;
+      SetComplexData(m);
+   }
+
+   virtual bool Factor(int m, double TOL = 0.0)
+   {
+      mfem_error("ComplexFactors::ComplexFactors(...)");
+      return false;
+   }
+
+   virtual std::complex<double> Det(int m) const
+   {
+      mfem_error("Factors::Det(...)");
+      return 0.;
+   }
+
+   virtual void Solve(int m, int n, double *X_r, double * X_i) const
+   {
+      mfem_error("Factors::Solve(...)");
+   }
+
+   virtual void GetInverseMatrix(int m, double *X_r, double * X_i) const
+   {
+      mfem_error("Factors::GetInverseMatrix(...)");
+   }
+
+   virtual ~ComplexFactors()
+   {
+      delete [] data; data = nullptr;
+   }
+};
+
+/** Class that computes factorization of external data and perform various
+    operations with the factored data. */
+class ComplexLUFactors : public ComplexFactors
+{
+public:
+   int *ipiv;
+#ifdef MFEM_USE_LAPACK
+   static const int ipiv_base = 1;
+#else
+   static const int ipiv_base = 0;
+#endif
+
+   /** With this constructor, the (public) data and ipiv members should be set
+       explicitly before calling class methods. */
+   ComplexLUFactors(): ComplexFactors() { }
+
+   ComplexLUFactors(double *data_r_,double * data_i, int *ipiv_)
+      : ComplexFactors(data_r_, data_i), ipiv(ipiv_) { }
+   /**
+    * @brief Compute the LU factorization of the current matrix
+    *
+    * Factorize the current matrix of size (m x m) overwriting it with the
+    * LU factors. The factorization is such that L.U = P.A, where A is the
+    * original matrix and P is a permutation matrix represented by ipiv.
+    *
+    * @param [in] m size of the square matrix
+    * @param [in] TOL optional fuzzy comparison tolerance. Defaults to 0.0.
+    *
+    * @return status set to true if successful, otherwise, false.
+    */
+   virtual bool Factor(int m, double TOL = 0.0);
+
+   /** Assuming L.U = P.A factored data of size (m x m), compute |A|
+       from the diagonal values of U and the permutation information. */
+   virtual std::complex<double> Det(int m) const;
+
+   /** Assuming L.U = P.A factored data of size (m x m), compute X <- A X,
+       for a matrix X of size (m x n). */
+   void Mult(int m, int n, double *X_r, double * X_i) const;
+
+   void Mult(int m, int n, std::complex<double> *X) const;
+
+   /** Assuming L.U = P.A factored data of size (m x m), compute
+       X <- L^{-1} P X, for a matrix X of size (m x n). */
+   void LSolve(int m, int n, double *X_r, double *X_i) const;
+
+   /** Assuming L.U = P.A factored data of size (m x m), compute
+       X <- U^{-1} X, for a matrix X of size (m x n). */
+   void USolve(int m, int n, double *X_r, double *X_i) const;
+
+   /** Assuming L.U = P.A factored data of size (m x m), compute X <- A^{-1} X,
+       for a matrix X of size (m x n). */
+   virtual void Solve(int m, int n, double *X_r, double *X_i) const;
+
+   /** Assuming L.U = P.A factored data of size (m x m), compute X <- X A^{-1},
+       for a matrix X of size (n x m). */
+   void RightSolve(int m, int n, double *X_r, double *X_i) const;
+
+   /// Assuming L.U = P.A factored data of size (m x m), compute X <- A^{-1}.
+   virtual void GetInverseMatrix(int m, double *X_r, double * X_i) const;
+};
+
+
+/** Class that can compute Cholesky factorizations of external data of an
+    Hermitian positive matrix and perform various operations with the factored data. */
+class ComplexCholeskyFactors : public ComplexFactors
+{
+public:
+
+   /** With this constructor, the (public) data should be set
+       explicitly before calling class methods. */
+   ComplexCholeskyFactors() : ComplexFactors() { }
+
+   ComplexCholeskyFactors(double *data_r_, double * data_i_)
+      : ComplexFactors(data_r_, data_i_) { }
+
+   /**
+    * @brief Compute the Cholesky factorization of the current matrix
+    *
+    * Factorize the current matrix of size (m x m) overwriting it with the
+    * Cholesky factors. The factorization is such that LL^H = A, where A is the
+    * original matrix
+    *
+    * @param [in] m size of the square matrix
+    * @param [in] TOL optional fuzzy comparison tolerance. Defaults to 0.0.
+    *
+    * @return status set to true if successful, otherwise, false.
+    */
+   virtual bool Factor(int m, double TOL = 0.0);
+
+   /** Assuming LL^H = A factored data of size (m x m), compute |A|
+       from the diagonal values of L */
+   virtual std::complex<double> Det(int m) const;
+
+   /** Assuming L.L^H = A factored data of size (m x m), compute X <- L X,
+       for a matrix X of size (m x n). */
+   void LMult(int m, int n, double *X_r, double * X_i) const;
+
+   /** Assuming L.L^H = A factored data of size (m x m), compute X <- L^t X,
+       for a matrix X of size (m x n). */
+   void UMult(int m, int n, double *X_r, double *X_i) const;
+
+   /** Assuming L L^H = A factored data of size (m x m), compute
+       X <- L^{-1} X, for a matrix X of size (m x n). */
+   void LSolve(int m, int n, double *X_r, double * X_i) const;
+
+   /** Assuming L L^H = A factored data of size (m x m), compute
+       X <- L^{-t} X, for a matrix X of size (m x n). */
+   void USolve(int m, int n, double *X_r, double *X_i) const;
+
+   /** Assuming L.L^H = A factored data of size (m x m), compute X <- A^{-1} X,
+       for a matrix X of size (m x n). */
+   virtual void Solve(int m, int n, double *X_r, double * X_i) const;
+
+   /** Assuming L.L^H = A factored data of size (m x m), compute X <- X A^{-1},
+       for a matrix X of size (n x m). */
+   void RightSolve(int m, int n, double *X_r, double *X_i) const;
+
+   /// Assuming L.L^H = A factored data of size (m x m), compute X <- A^{-1}.
+   virtual void GetInverseMatrix(int m, double *X_r, double * X_i) const;
+
+};
+
+} // namespace mfem
+
+#endif // MFEM_COMPLEX_DENSEMAT

linalg/constraints.cpp

@@ -319,6 +319,10 @@ void EliminationSolver::Mult(const Vector& rhs, Vector& sol) const
    {
       prec = BuildPreconditioner();
    }
+   else
+   {
+      prec->SetOperator(*h_explicit_operator);
+   }
    if (!krylov)
    {
       krylov = BuildKrylov();
@@ -356,23 +360,21 @@ void EliminationSolver::Mult(const Vector& rhs, Vector& sol) const
    sol += rtilde;
 }
 
-void PenaltyConstrainedSolver::Initialize(HypreParMatrix& A, HypreParMatrix& B)
+void PenaltyConstrainedSolver::Initialize(HypreParMatrix& A, HypreParMatrix& B,
+                                          HypreParMatrix& D)
 {
-   HypreParMatrix * hBT = B.Transpose();
-   HypreParMatrix * hBTB = ParMult(hBT, &B, true);
+   HypreParMatrix * hBTB = RAP(&D, &B);
    // this matrix doesn't get cleanly deleted?
    // (hypre comm pkg)
-   (*hBTB) *= penalty;
    penalized_mat = ParAdd(&A, hBTB);
    delete hBTB;
-   delete hBT;
 }
 
 PenaltyConstrainedSolver::PenaltyConstrainedSolver(
    HypreParMatrix& A, SparseMatrix& B, double penalty_)
    :
    ConstrainedSolver(A.GetComm(), A, B),
-   penalty(penalty_),
+   penalty(B.Height()),
    constraintB(B),
    krylov(nullptr),
    prec(nullptr)
@@ -399,19 +401,45 @@ PenaltyConstrainedSolver::PenaltyConstrainedSolver(
                      row_starts, col_starts, &B);
    hB.CopyRowStarts();
    hB.CopyColStarts();
-   Initialize(A, hB);
+   penalty=penalty_;
+   SparseMatrix D(penalty);
+   HypreParMatrix hD(hB.GetComm(), hB.M(), hB.RowPart(), &D);
+   hD.CopyRowStarts();
+   hD.CopyColStarts();
+   Initialize(A, hB, hD);
 }
 
 PenaltyConstrainedSolver::PenaltyConstrainedSolver(
    HypreParMatrix& A, HypreParMatrix& B, double penalty_)
    :
    ConstrainedSolver(A.GetComm(), A, B),
+   penalty(B.Height()),
+   constraintB(B),
+   krylov(nullptr),
+   prec(nullptr)
+{
+   penalty=penalty_;
+   SparseMatrix D(penalty);
+   HypreParMatrix hD(B.GetComm(), B.M(), B.RowPart(), &D);
+   hD.CopyRowStarts();
+   hD.CopyColStarts();
+   Initialize(A, B, hD);
+}
+
+PenaltyConstrainedSolver::PenaltyConstrainedSolver(
+   HypreParMatrix& A, HypreParMatrix& B, Vector& penalty_)
+   :
+   ConstrainedSolver(A.GetComm(), A, B),
    penalty(penalty_),
    constraintB(B),
    krylov(nullptr),
    prec(nullptr)
 {
-   Initialize(A, B);
+   SparseMatrix D(penalty_);
+   HypreParMatrix hD(B.GetComm(), B.M(), B.RowPart(), &D);
+   hD.CopyRowStarts();
+   hD.CopyColStarts();
+   Initialize(A, B, hD);
 }
 
 PenaltyConstrainedSolver::~PenaltyConstrainedSolver()
@@ -427,6 +455,10 @@ void PenaltyConstrainedSolver::Mult(const Vector& b, Vector& x) const
    {
       prec = BuildPreconditioner();
    }
+   else
+   {
+      prec->SetOperator(*penalized_mat);
+   }
    if (!krylov)
    {
       krylov = BuildKrylov();
@@ -438,9 +470,11 @@ void PenaltyConstrainedSolver::Mult(const Vector& b, Vector& x) const
    Vector penalized_rhs(b);
    if (constraint_rhs.Size() > 0)
    {
+      Vector temp_rhs(constraint_rhs.Size());
+      SparseMatrix D(penalty);
+      D.Mult(constraint_rhs, temp_rhs);
       Vector temp(x.Size());
-      constraintB.MultTranspose(constraint_rhs, temp);
-      temp *= penalty;
+      constraintB.MultTranspose(temp_rhs, temp);
       penalized_rhs += temp;
    }
 
@@ -581,6 +615,8 @@ void SchurConstrainedSolver::LagrangeSystemMult(const Vector& x,
 
    gmres->Mult(x, y);
    final_iter = gmres->GetNumIterations();
+   converged = gmres->GetConverged();
+   final_norm = gmres->GetFinalNorm();
    delete gmres;
 }
 
@@ -588,6 +624,7 @@ void SchurConstrainedSolver::LagrangeSystemMult(const Vector& x,
 SchurConstrainedHypreSolver::SchurConstrainedHypreSolver(MPI_Comm comm,
                                                          HypreParMatrix& hA_,
                                                          HypreParMatrix& hB_,
+                                                         Solver * prec,
                                                          int dimension,
                                                          bool reorder)
    :
@@ -595,13 +632,20 @@ SchurConstrainedHypreSolver::SchurConstrainedHypreSolver(MPI_Comm comm,
    hA(hA_),
    hB(hB_)
 {
-   auto h_primal_pc = new HypreBoomerAMG(hA);
-   h_primal_pc->SetPrintLevel(0);
-   if (dimension > 0)
+   if (prec == nullptr)
    {
-      h_primal_pc->SetSystemsOptions(dimension, reorder);
+      auto h_primal_pc = new HypreBoomerAMG(hA);
+      h_primal_pc->SetPrintLevel(0);
+      if (dimension > 0)
+      {
+         h_primal_pc->SetSystemsOptions(dimension, reorder);
+      }
+      primal_pc = h_primal_pc;
+   }
+   else
+   {
+      primal_pc = prec;
    }
-   primal_pc = h_primal_pc;
 
    HypreParMatrix * scaledB = new HypreParMatrix(hB);
    Vector diagA;

linalg/constraints.hpp

@@ -235,6 +235,9 @@ public:
    void SetOperator(const Operator& op) override
    { MFEM_ABORT("Operator cannot be reset!"); }
 
+   void SetPreconditioner(Solver& precond) override
+   { prec = &precond; }
+
 protected:
    /// Internal utility routine; assembles eliminated matrix explicitly
    void BuildExplicitOperator();
@@ -281,6 +284,33 @@ private:
    bool reorder;
 };
 
+/** EliminationSolver using GMRES and HypreBoomerAMG */
+class EliminationGMRESSolver : public EliminationSolver
+{
+public:
+   EliminationGMRESSolver(HypreParMatrix& A, SparseMatrix& B,
+                          Array<int>& constraint_rowstarts,
+                          int dimension_=0, bool reorder_=false) :
+      EliminationSolver(A, B, constraint_rowstarts),
+      dimension(dimension_), reorder(reorder_)
+   { }
+
+protected:
+   virtual Solver* BuildPreconditioner() const override
+   {
+      HypreBoomerAMG * h_prec = new HypreBoomerAMG(*h_explicit_operator);
+      h_prec->SetPrintLevel(0);
+      if (dimension > 0) { h_prec->SetSystemsOptions(dimension, reorder); }
+      return h_prec;
+   }
+
+   virtual IterativeSolver* BuildKrylov() const override
+   { return new GMRESSolver(GetComm()); }
+
+private:
+   int dimension;
+   bool reorder;
+};
 
 /** @brief Solve constrained system with penalty method; see ConstrainedSolver.
 
@@ -295,6 +325,9 @@ public:
    PenaltyConstrainedSolver(HypreParMatrix& A, HypreParMatrix& B,
                             double penalty_);
 
+   PenaltyConstrainedSolver(HypreParMatrix& A, HypreParMatrix& B,
+                            Vector& penalty_);
+
    ~PenaltyConstrainedSolver();
 
    void Mult(const Vector& x, Vector& y) const override;
@@ -302,15 +335,22 @@ public:
    void SetOperator(const Operator& op) override
    { MFEM_ABORT("Operator cannot be reset!"); }
 
+   void SetPreconditioner(Solver& precond) override
+   { prec = &precond; }
+
 protected:
-   void Initialize(HypreParMatrix& A, HypreParMatrix& B);
+   /// Initialize the matrix A + B*D*B^T for the constrained linear system
+   /// A - original matrix (N x N square matrix)
+   /// B - constraint matrix (M x N rectangular matrix)
+   /// D - diagonal matrix of penalty values (M x M square matrix)
+   void Initialize(HypreParMatrix& A, HypreParMatrix& B, HypreParMatrix& D);
 
    /// Build preconditioner for penalized system
    virtual Solver* BuildPreconditioner() const = 0;
    /// Select krylov solver for penalized system
    virtual IterativeSolver* BuildKrylov() const = 0;
 
-   double penalty;
+   Vector penalty;
    Operator& constraintB;
    HypreParMatrix * penalized_mat;
    mutable IterativeSolver * krylov;
@@ -334,6 +374,12 @@ public:
       dimension_(dimension), reorder_(reorder)
    { }
 
+   PenaltyPCGSolver(HypreParMatrix& A, HypreParMatrix& B, Vector& penalty_,
+                    int dimension=0, bool reorder=false) :
+      PenaltyConstrainedSolver(A, B, penalty_),
+      dimension_(dimension), reorder_(reorder)
+   { }
+
 protected:
    virtual Solver* BuildPreconditioner() const override
    {
@@ -351,6 +397,45 @@ private:
    bool reorder_;
 };
 
+/** Uses GMRES and a HypreBoomerAMG preconditioner for the penalized system. */
+class PenaltyGMRESSolver : public PenaltyConstrainedSolver
+{
+public:
+   PenaltyGMRESSolver(HypreParMatrix& A, SparseMatrix& B, double penalty_,
+                      int dimension=0, bool reorder=false) :
+      PenaltyConstrainedSolver(A, B, penalty_),
+      dimension_(dimension), reorder_(reorder)
+   { }
+
+   PenaltyGMRESSolver(HypreParMatrix& A, HypreParMatrix& B, double penalty_,
+                      int dimension=0, bool reorder=false) :
+      PenaltyConstrainedSolver(A, B, penalty_),
+      dimension_(dimension), reorder_(reorder)
+   { }
+
+   PenaltyGMRESSolver(HypreParMatrix& A, HypreParMatrix& B, Vector& penalty_,
+                      int dimension=0, bool reorder=false) :
+      PenaltyConstrainedSolver(A, B, penalty_),
+      dimension_(dimension), reorder_(reorder)
+   { }
+
+protected:
+   virtual Solver* BuildPreconditioner() const override
+   {
+      HypreBoomerAMG* h_prec = new HypreBoomerAMG(*penalized_mat);
+      h_prec->SetPrintLevel(0);
+      if (dimension_ > 0) { h_prec->SetSystemsOptions(dimension_, reorder_); }
+      return h_prec;
+   }
+
+   virtual IterativeSolver* BuildKrylov() const override
+   { return new GMRESSolver(GetComm()); }
+
+private:
+   int dimension_;
+   bool reorder_;
+};
+
 #endif
 
 /** @brief Solve constrained system by solving original mixed system;
@@ -409,8 +494,8 @@ class SchurConstrainedHypreSolver : public SchurConstrainedSolver
 {
 public:
    SchurConstrainedHypreSolver(MPI_Comm comm, HypreParMatrix& hA_,
-                               HypreParMatrix& hB_, int dimension=0,
-                               bool reorder=false);
+                               HypreParMatrix& hB_, Solver * prec = nullptr,
+                               int dimension=0, bool reorder=false);
    virtual ~SchurConstrainedHypreSolver();
 
 private:

linalg/densemat.cpp

@@ -550,6 +550,15 @@ void DenseMatrix::Add(const double c, const DenseMatrix &A)
    }
 }
 
+void DenseMatrix::Add(const double c, const double *A)
+{
+   const int s = Width()*Height();
+   for (int i = 0; i < s; i++)
+   {
+      data[i] += c*A[i];
+   }
+}
+
 DenseMatrix &DenseMatrix::operator=(double c)
 {
    const int s = Height()*Width();
@@ -1450,10 +1459,10 @@ void DenseMatrix::GradToCurl(DenseMatrix &curl)
          int j = i+n;
 
          // curl of (Ui,0)
-         curl(i,0) = -y;
+         curl(i,0) = y;
 
          // curl of (0,Ui)
-         curl(j,0) =  x;
+         curl(j,0) = -x;
       }
    }
    else
@@ -1718,6 +1727,336 @@ void DenseMatrix::AddMatrix(double a, const DenseMatrix &A, int ro, int co)
    }
 }
 
+void DenseMatrix::GetSubMatrix(const Array<int> & idx, DenseMatrix & A) const
+{
+   int k = idx.Size();
+   int idx_max = idx.Max();
+   MFEM_VERIFY(idx.Min() >=0 && idx_max < this->height && idx_max < this->width,
+               "DenseMatrix::GetSubMatrix: Index out of bounds");
+   A.SetSize(k);
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = idx[i];
+      for (int j = 0; j<k; j++)
+      {
+         jj = idx[j];
+         adata[i+j*k] = this->data[ii+jj*height];
+      }
+   }
+}
+
+void DenseMatrix::GetSubMatrix(const Array<int> & idx_i,
+                               const Array<int> & idx_j, DenseMatrix & A) const
+{
+   int k = idx_i.Size();
+   int l = idx_j.Size();
+
+   MFEM_VERIFY(idx_i.Min() >=0 && idx_i.Max() < this->height,
+               "DenseMatrix::GetSubMatrix: Row index out of bounds");
+   MFEM_VERIFY(idx_j.Min() >=0 && idx_j.Max() < this->width,
+               "DenseMatrix::GetSubMatrix: Col index out of bounds");
+
+   A.SetSize(k,l);
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = idx_i[i];
+      for (int j = 0; j<l; j++)
+      {
+         jj = idx_j[j];
+         adata[i+j*k] = this->data[ii+jj*height];
+      }
+   }
+}
+
+void DenseMatrix::GetSubMatrix(int ibeg, int iend, DenseMatrix & A)
+{
+   MFEM_VERIFY(iend >= ibeg, "DenseMatrix::GetSubMatrix: Inconsistent range");
+   MFEM_VERIFY(ibeg >=0,
+               "DenseMatrix::GetSubMatrix: Negative index");
+   MFEM_VERIFY(iend <= this->height && iend <= this->width,
+               "DenseMatrix::GetSubMatrix: Index bigger than upper bound");
+
+   int k = iend - ibeg;
+   A.SetSize(k);
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = ibeg + i;
+      for (int j = 0; j<k; j++)
+      {
+         jj = ibeg + j;
+         adata[i+j*k] = this->data[ii+jj*height];
+      }
+   }
+}
+
+void DenseMatrix::GetSubMatrix(int ibeg, int iend, int jbeg, int jend,
+                               DenseMatrix & A)
+{
+   MFEM_VERIFY(iend >= ibeg,
+               "DenseMatrix::GetSubMatrix: Inconsistent row range");
+   MFEM_VERIFY(jend >= jbeg,
+               "DenseMatrix::GetSubMatrix: Inconsistent col range");
+   MFEM_VERIFY(ibeg >=0,
+               "DenseMatrix::GetSubMatrix: Negative row index");
+   MFEM_VERIFY(jbeg >=0,
+               "DenseMatrix::GetSubMatrix: Negative row index");
+   MFEM_VERIFY(iend <= this->height,
+               "DenseMatrix::GetSubMatrix: Index bigger than row upper bound");
+   MFEM_VERIFY(jend <= this->width,
+               "DenseMatrix::GetSubMatrix: Index bigger than col upper bound");
+
+   int k = iend - ibeg;
+   int l = jend - jbeg;
+   A.SetSize(k,l);
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = ibeg + i;
+      for (int j = 0; j<l; j++)
+      {
+         jj = jbeg + j;
+         adata[i+j*k] = this->data[ii+jj*height];
+      }
+   }
+}
+
+void DenseMatrix::SetSubMatrix(const Array<int> & idx, const DenseMatrix & A)
+{
+   int k = idx.Size();
+   MFEM_VERIFY(A.Height() == k && A.Width() == k,
+               "DenseMatrix::SetSubMatrix:Inconsistent matrix dimensions");
+
+   int idx_max = idx.Max();
+
+   MFEM_VERIFY(idx.Min() >=0,
+               "DenseMatrix::SetSubMatrix: Negative index");
+   MFEM_VERIFY(idx_max < this->height,
+               "DenseMatrix::SetSubMatrix: Index bigger than row upper bound");
+   MFEM_VERIFY(idx_max < this->width,
+               "DenseMatrix::SetSubMatrix: Index bigger than col upper bound");
+
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = idx[i];
+      for (int j = 0; j<k; j++)
+      {
+         jj = idx[j];
+         this->data[ii+jj*height] = adata[i+j*k];
+      }
+   }
+}
+
+void DenseMatrix::SetSubMatrix(const Array<int> & idx_i,
+                               const Array<int> & idx_j, const DenseMatrix & A)
+{
+   int k = idx_i.Size();
+   int l = idx_j.Size();
+   MFEM_VERIFY(k == A.Height() && l == A.Width(),
+               "DenseMatrix::SetSubMatrix:Inconsistent matrix dimensions");
+   MFEM_VERIFY(idx_i.Min() >=0,
+               "DenseMatrix::SetSubMatrix: Negative row index");
+   MFEM_VERIFY(idx_j.Min() >=0,
+               "DenseMatrix::SetSubMatrix: Negative col index");
+   MFEM_VERIFY(idx_i.Max() < this->height,
+               "DenseMatrix::SetSubMatrix: Index bigger than row upper bound");
+   MFEM_VERIFY(idx_j.Max() < this->width,
+               "DenseMatrix::SetSubMatrix: Index bigger than col upper bound");
+
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = idx_i[i];
+      for (int j = 0; j<l; j++)
+      {
+         jj = idx_j[j];
+         this->data[ii+jj*height] = adata[i+j*k];
+      }
+   }
+}
+
+void DenseMatrix::SetSubMatrix(int ibeg, const DenseMatrix & A)
+{
+   int k = A.Height();
+
+   MFEM_VERIFY(A.Width() == k, "DenseMatrix::SetSubmatrix: A is not square");
+   MFEM_VERIFY(ibeg >=0,
+               "DenseMatrix::SetSubmatrix: Negative index");
+   MFEM_VERIFY(ibeg + k <= this->height,
+               "DenseMatrix::SetSubmatrix: index bigger than row upper bound");
+   MFEM_VERIFY(ibeg + k <= this->width,
+               "DenseMatrix::SetSubmatrix: index bigger than col upper bound");
+
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = ibeg + i;
+      for (int j = 0; j<k; j++)
+      {
+         jj = ibeg + j;
+         this->data[ii+jj*height] = adata[i+j*k];
+      }
+   }
+}
+
+void DenseMatrix::SetSubMatrix(int ibeg, int jbeg, const DenseMatrix & A)
+{
+   int k = A.Height();
+   int l = A.Width();
+
+   MFEM_VERIFY(ibeg>=0,
+               "DenseMatrix::SetSubmatrix: Negative row index");
+   MFEM_VERIFY(jbeg>=0,
+               "DenseMatrix::SetSubmatrix: Negative col index");
+   MFEM_VERIFY(ibeg + k <= this->height,
+               "DenseMatrix::SetSubmatrix: Index bigger than row upper bound");
+   MFEM_VERIFY(jbeg + l <= this->width,
+               "DenseMatrix::SetSubmatrix: Index bigger than col upper bound");
+
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = ibeg + i;
+      for (int j = 0; j<l; j++)
+      {
+         jj = jbeg + j;
+         this->data[ii+jj*height] = adata[i+j*k];
+      }
+   }
+}
+
+void DenseMatrix::AddSubMatrix(const Array<int> & idx, const DenseMatrix & A)
+{
+   int k = idx.Size();
+   MFEM_VERIFY(A.Height() == k && A.Width() == k,
+               "DenseMatrix::AddSubMatrix:Inconsistent matrix dimensions");
+
+   int idx_max = idx.Max();
+
+   MFEM_VERIFY(idx.Min() >=0, "DenseMatrix::AddSubMatrix: Negative index");
+   MFEM_VERIFY(idx_max < this->height,
+               "DenseMatrix::AddSubMatrix: Index bigger than row upper bound");
+   MFEM_VERIFY(idx_max < this->width,
+               "DenseMatrix::AddSubMatrix: Index bigger than col upper bound");
+
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = idx[i];
+      for (int j = 0; j<k; j++)
+      {
+         jj = idx[j];
+         this->data[ii+jj*height] += adata[i+j*k];
+      }
+   }
+}
+
+void DenseMatrix::AddSubMatrix(const Array<int> & idx_i,
+                               const Array<int> & idx_j, const DenseMatrix & A)
+{
+   int k = idx_i.Size();
+   int l = idx_j.Size();
+   MFEM_VERIFY(k == A.Height() && l == A.Width(),
+               "DenseMatrix::AddSubMatrix:Inconsistent matrix dimensions");
+
+   MFEM_VERIFY(idx_i.Min() >=0,
+               "DenseMatrix::AddSubMatrix: Negative row index");
+   MFEM_VERIFY(idx_j.Min() >=0,
+               "DenseMatrix::AddSubMatrix: Negative col index");
+   MFEM_VERIFY(idx_i.Max() < this->height,
+               "DenseMatrix::AddSubMatrix: Index bigger than row upper bound");
+   MFEM_VERIFY(idx_j.Max() < this->width,
+               "DenseMatrix::AddSubMatrix: Index bigger than col upper bound");
+
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = idx_i[i];
+      for (int j = 0; j<l; j++)
+      {
+         jj = idx_j[j];
+         this->data[ii+jj*height] += adata[i+j*k];
+      }
+   }
+}
+
+void DenseMatrix::AddSubMatrix(int ibeg, const DenseMatrix & A)
+{
+   int k = A.Height();
+   MFEM_VERIFY(A.Width() == k, "DenseMatrix::AddSubmatrix: A is not square");
+
+   MFEM_VERIFY(ibeg>=0,
+               "DenseMatrix::AddSubmatrix: Negative index");
+   MFEM_VERIFY(ibeg + k <= this->Height(),
+               "DenseMatrix::AddSubmatrix: Index bigger than row upper bound");
+   MFEM_VERIFY(ibeg + k <= this->Width(),
+               "DenseMatrix::AddSubmatrix: Index bigger than col upper bound");
+
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = ibeg + i;
+      for (int j = 0; j<k; j++)
+      {
+         jj = ibeg + j;
+         this->data[ii+jj*height] += adata[i+j*k];
+      }
+   }
+}
+
+void DenseMatrix::AddSubMatrix(int ibeg, int jbeg, const DenseMatrix & A)
+{
+   int k = A.Height();
+   int l = A.Width();
+
+   MFEM_VERIFY(ibeg>=0,
+               "DenseMatrix::AddSubmatrix: Negative row index");
+   MFEM_VERIFY(jbeg>=0,
+               "DenseMatrix::AddSubmatrix: Negative col index");
+   MFEM_VERIFY(ibeg + k <= this->height,
+               "DenseMatrix::AddSubmatrix: Index bigger than row upper bound");
+   MFEM_VERIFY(jbeg + l <= this->width,
+               "DenseMatrix::AddSubmatrix: Index bigger than col upper bound");
+
+   double * adata = A.Data();
+
+   int ii, jj;
+   for (int i = 0; i<k; i++)
+   {
+      ii = ibeg + i;
+      for (int j = 0; j<l; j++)
+      {
+         jj = jbeg + j;
+         this->data[ii+jj*height] += adata[i+j*k];
+      }
+   }
+}
+
 void DenseMatrix::AddToVector(int offset, Vector &v) const
 {
    const int n = height * width;
@@ -2857,6 +3196,22 @@ void AddMult_a_VVt(const double a, const Vector &v, DenseMatrix &VVt)
    }
 }
 
+void RAP(const DenseMatrix &A, const DenseMatrix &P, DenseMatrix & RAP)
+{
+   DenseMatrix RA(P.Width(),A.Width());
+   MultAtB(P,A,RA);
+   RAP.SetSize(RA.Height(), P.Width());
+   Mult(RA,P, RAP);
+}
+
+void RAP(const DenseMatrix &Rt, const DenseMatrix &A,
+         const DenseMatrix &P, DenseMatrix & RAP)
+{
+   DenseMatrix RA(Rt.Width(),A.Width());
+   MultAtB(Rt,A,RA);
+   RAP.SetSize(RA.Height(), P.Width());
+   Mult(RA,P, RAP);
+}
 
 bool LUFactors::Factor(int m, double TOL)
 {

linalg/densemat.hpp

@@ -207,9 +207,13 @@ public:
       Set(alpha, A.GetData());
    }
 
-   /// Adds the matrix A multiplied by the number c to the matrix
+   /// Adds the matrix A multiplied by the number c to the matrix.
    void Add(const double c, const DenseMatrix &A);
 
+   /// Adds the matrix A multiplied by the number c to the matrix,
+   /// assuming A has the same dimensions and uses column-major layout.
+   void Add(const double c, const double *A);
+
    /// Sets the matrix elements equal to constant c
    DenseMatrix &operator=(double c);
 
@@ -355,6 +359,55 @@ public:
    /// Perform (ro+i,co+j)+=a*A(i,j) for 0<=i<A.Height, 0<=j<A.Width
    void AddMatrix(double a, const DenseMatrix &A, int ro, int co);
 
+   /** Get the square submatrix which corresponds to the given indices @a idx.
+       Note: the @a A matrix will be resized to accommodate the data */
+   void GetSubMatrix(const Array<int> & idx, DenseMatrix & A) const;
+
+   /** Get the rectangular submatrix which corresponds to the given indices
+      @a idx_i and @a idx_j. Note: the @a A matrix will be resized to
+      accommodate the data */
+   void GetSubMatrix(const Array<int> & idx_i, const Array<int> & idx_j,
+                     DenseMatrix & A) const;
+
+   /** Get the square submatrix which corresponds to the range
+       [ @a ibeg, @a iend ). Note: the @a A matrix will be resized
+        to accommodate the data */
+   void GetSubMatrix(int ibeg, int iend, DenseMatrix & A);
+
+   /** Get the square submatrix which corresponds to the range
+      i ∈ [ @a ibeg, @a iend ) and j ∈ [ @a jbeg, @a jend )
+      Note: the @a A matrix will be resized to accommodate the data */
+   void GetSubMatrix(int ibeg, int iend, int jbeg, int jend, DenseMatrix & A);
+
+   /// Set (*this)(idx[i],idx[j]) = A(i,j)
+   void SetSubMatrix(const Array<int> & idx, const DenseMatrix & A);
+
+   /// Set (*this)(idx_i[i],idx_j[j]) = A(i,j)
+   void SetSubMatrix(const Array<int> & idx_i, const Array<int> & idx_j,
+                     const DenseMatrix & A);
+
+   /** Set a submatrix of (this) to the given matrix @a A
+       with row and column offset @a ibeg */
+   void SetSubMatrix(int ibeg, const DenseMatrix & A);
+
+   /** Set a submatrix of (this) to the given matrix @a A
+       with row and column offset @a ibeg and @a jbeg respectively */
+   void SetSubMatrix(int ibeg, int jbeg, const DenseMatrix & A);
+
+   /// (*this)(idx[i],idx[j]) += A(i,j)
+   void AddSubMatrix(const Array<int> & idx, const DenseMatrix & A);
+
+   /// (*this)(idx_i[i],idx_j[j]) += A(i,j)
+   void AddSubMatrix(const Array<int> & idx_i, const Array<int> & idx_j,
+                     const DenseMatrix & A);
+
+   /** Add the submatrix @a A to this with row and column offset @a ibeg */
+   void AddSubMatrix(int ibeg, const DenseMatrix & A);
+
+   /** Add the submatrix @a A to this with row and column offsets
+       @a ibeg and @a jbeg respectively */
+   void AddSubMatrix(int ibeg, int jbeg, const DenseMatrix & A);
+
    /// Add the matrix 'data' to the Vector 'v' at the given 'offset'
    void AddToVector(int offset, Vector &v) const;
    /// Get the matrix 'data' from the Vector 'v' at the given 'offset'
@@ -368,7 +421,7 @@ public:
 
    /** Count the number of entries in the matrix for which isfinite
        is false, i.e. the entry is a NaN or +/-Inf. */
-   int CheckFinite() const { return mfem::CheckFinite(data, height*width); }
+   int CheckFinite() const { return mfem::CheckFinite(HostRead(), height*width); }
 
    /// Prints matrix to stream out.
    virtual void Print(std::ostream &out = mfem::out, int width_ = 4) const;
@@ -523,6 +576,14 @@ void AddMult_a_VWt(const double a, const Vector &v, const Vector &w,
 /// VVt += a * v v^t
 void AddMult_a_VVt(const double a, const Vector &v, DenseMatrix &VVt);
 
+/** Computes matrix P^t * A * P. Note: The @a RAP matrix will be resized
+    to accommodate the data */
+void RAP(const DenseMatrix &A, const DenseMatrix &P, DenseMatrix & RAP);
+
+/** Computes the matrix Rt^t * A * P. Note: The @a RAP matrix will be resized
+    to accommodate the data */
+void RAP(const DenseMatrix &Rt, const DenseMatrix &A,
+         const DenseMatrix &P, DenseMatrix & RAP);
 
 /** Abstract class that can compute factorization of external data and perform various
     operations with the factored data. */
@@ -887,7 +948,7 @@ class Table;
 class DenseTensor
 {
 private:
-   DenseMatrix Mk;
+   mutable DenseMatrix Mk;
    Memory<double> tdata;
    int nk;
 
@@ -966,7 +1027,12 @@ public:
       return Mk;
    }
    const DenseMatrix &operator()(int k) const
-   { return const_cast<DenseTensor&>(*this)(k); }
+   {
+      MFEM_ASSERT_INDEX_IN_RANGE(k, 0, SizeK());
+      Mk.data = Memory<double>(const_cast<double*>(GetData(k)), SizeI()*SizeJ(),
+                               false);
+      return Mk;
+   }
 
    double &operator()(int i, int j, int k)
    {
@@ -990,6 +1056,12 @@ public:
       return tdata+k*Mk.Height()*Mk.Width();
    }
 
+   const double *GetData(int k) const
+   {
+      MFEM_ASSERT_INDEX_IN_RANGE(k, 0, SizeK());
+      return tdata+k*Mk.Height()*Mk.Width();
+   }
+
    double *Data() { return tdata; }
 
    const double *Data() const { return tdata; }

linalg/hiop.hpp

@@ -92,6 +92,8 @@ public:
     *  internally in HiOp. */
    virtual bool get_starting_point(const hiop::size_type &n, double *x0);
 
+   using hiop::hiopInterfaceBase::get_starting_point;
+
    virtual bool get_vars_info(const hiop::size_type &n, double *xlow, double* xupp,
                               NonlinearityType* type);
 
@@ -136,6 +138,8 @@ public:
                           const hiop::index_type *idx_cons,
                           const double *x, bool new_x, double *cons);
 
+   using hiop::hiopInterfaceBase::eval_cons;
+
    /** Evaluates the Jacobian of the subset of constraints indicated by
     *  idx_cons. The idx_cons is assumed to be of size num_cons.
     *  Example: if cons[c] = C(x)[idx_cons[c]] where c = 0 .. num_cons-1, then
@@ -152,6 +156,8 @@ public:
                               const hiop::index_type *idx_cons,
                               const double *x, bool new_x, double *Jac);
 
+   using hiop::hiopInterfaceDenseConstraints::eval_Jac_cons;
+
    /** Specifies column partitioning for distributed memory vectors.
     *  Process p owns vector entries with indices cols[p] to cols[p+1]-1,
     *  where p = 0 .. nranks-1. The cols array is of size nranks + 1.

linalg/hypre.cpp

@@ -22,10 +22,6 @@
 #include <cmath>
 #include <cstdlib>
 
-#ifdef MFEM_USE_SUNDIALS
-#include <nvector/nvector_parallel.h>
-#endif
-
 using namespace std;
 
 namespace mfem
@@ -413,15 +409,6 @@ HypreParVector::~HypreParVector()
    }
 }
 
-#ifdef MFEM_USE_SUNDIALS
-
-N_Vector HypreParVector::ToNVector()
-{
-   return N_VMake_Parallel(GetComm(), Size(), GlobalSize(), GetData());
-}
-
-#endif // MFEM_USE_SUNDIALS
-
 
 double InnerProduct(HypreParVector *x, HypreParVector *y)
 {
@@ -950,7 +937,6 @@ HypreParMatrix::HypreParMatrix(
 #ifdef HYPRE_USING_GPU
    hypre_CSRMatrixMemoryLocation(A->diag) = HYPRE_MEMORY_HOST;
 #endif
-   hypre_CSRMatrixSetRownnz(A->diag);
 
    hypre_CSRMatrixSetDataOwner(A->offd, hypre_arrays);
    hypre_CSRMatrixI(A->offd) = offd_i;
@@ -960,7 +946,6 @@ HypreParMatrix::HypreParMatrix(
 #ifdef HYPRE_USING_GPU
    hypre_CSRMatrixMemoryLocation(A->offd) = HYPRE_MEMORY_HOST;
 #endif
-   hypre_CSRMatrixSetRownnz(A->offd);
 
    hypre_ParCSRMatrixColMapOffd(A) = offd_col_map;
    // Prevent hypre from destroying A->col_map_offd, own A->col_map_offd
@@ -992,6 +977,9 @@ HypreParMatrix::HypreParMatrix(
       offdOwner = HypreCsrToMem(A->offd, host_mt, false, mem_offd);
    }
    HypreRead();
+
+   hypre_CSRMatrixSetRownnz(A->diag);
+   hypre_CSRMatrixSetRownnz(A->offd);
 }
 
 // Constructor from a CSR matrix on rank 0 (4 arguments, v2)
@@ -1136,7 +1124,6 @@ HypreParMatrix::HypreParMatrix(MPI_Comm comm, int id, int np,
 #ifdef HYPRE_USING_GPU
    hypre_CSRMatrixMemoryLocation(A->diag) = HYPRE_MEMORY_HOST;
 #endif
-   hypre_CSRMatrixSetRownnz(A->diag);
 
    hypre_CSRMatrixSetDataOwner(A->offd,0);
    hypre_CSRMatrixI(A->offd)    = i_offd;
@@ -1145,7 +1132,6 @@ HypreParMatrix::HypreParMatrix(MPI_Comm comm, int id, int np,
 #ifdef HYPRE_USING_GPU
    hypre_CSRMatrixMemoryLocation(A->offd) = HYPRE_MEMORY_HOST;
 #endif
-   hypre_CSRMatrixSetRownnz(A->offd);
 
    hypre_ParCSRMatrixColMapOffd(A) = cmap;
    // Prevent hypre from destroying A->col_map_offd, own A->col_map_offd
@@ -1168,6 +1154,9 @@ HypreParMatrix::HypreParMatrix(MPI_Comm comm, int id, int np,
    diagOwner = HypreCsrToMem(A->diag, host_mt, true, mem_diag);
    offdOwner = HypreCsrToMem(A->offd, host_mt, true, mem_offd);
    HypreRead();
+
+   hypre_CSRMatrixSetRownnz(A->diag);
+   hypre_CSRMatrixSetRownnz(A->offd);
 }
 
 // General rectangular constructor with diagonal and off-diagonal constructed
@@ -1621,7 +1610,7 @@ HypreParMatrix * HypreParMatrix::Transpose() const
    hypre_ParCSRMatrixTranspose(A, &At, 1);
    hypre_ParCSRMatrixSetNumNonzeros(At);
 
-   hypre_MatvecCommPkgCreate(At);
+   if (!hypre_ParCSRMatrixCommPkg(At)) { hypre_MatvecCommPkgCreate(At); }
 
    if ( M() == N() )
    {
@@ -1721,6 +1710,25 @@ void HypreParMatrix::EnsureMultTranspose() const
 #endif
 }
 
+void HypreParMatrix::ResetTranspose() const
+{
+#if (MFEM_HYPRE_VERSION == 22500 && HYPRE_DEVELOP_NUMBER >= 1) || \
+    (MFEM_HYPRE_VERSION > 22500)
+#ifdef HYPRE_USING_GPU
+   if (A->diagT)
+   {
+      hypre_CSRMatrixDestroy(A->diagT);
+      A->diagT = NULL;
+   }
+   if (A->offdT)
+   {
+      hypre_CSRMatrixDestroy(A->offdT);
+      A->offdT = NULL;
+   }
+#endif
+#endif
+}
+
 HYPRE_Int HypreParMatrix::Mult(HypreParVector &x, HypreParVector &y,
                                double a, double b) const
 {
@@ -1843,13 +1851,7 @@ void HypreParMatrix::MultTranspose(double a, const Vector &x,
       }
    }
 
-#if (MFEM_HYPRE_VERSION == 22500 && HYPRE_DEVELOP_NUMBER >= 1) || \
-    (MFEM_HYPRE_VERSION > 22500)
-#ifdef HYPRE_USING_GPU
-   MFEM_VERIFY(A->diagT != NULL,
-               "Transpose action requires EnsureMultTranspose()");
-#endif
-#endif
+   EnsureMultTranspose();
 
    hypre_ParCSRMatrixMatvecT(a, A, *Y, b, *X);
 
@@ -1866,13 +1868,7 @@ HYPRE_Int HypreParMatrix::Mult(HYPRE_ParVector x, HYPRE_ParVector y,
 HYPRE_Int HypreParMatrix::MultTranspose(HypreParVector & x, HypreParVector & y,
                                         double a, double b) const
 {
-#if (MFEM_HYPRE_VERSION == 22500 && HYPRE_DEVELOP_NUMBER >= 1) || \
-    (MFEM_HYPRE_VERSION > 22500)
-#ifdef HYPRE_USING_GPU
-   MFEM_VERIFY(A->diagT != NULL,
-               "Transpose action requires EnsureMultTranspose()");
-#endif
-#endif
+   EnsureMultTranspose();
    x.HypreRead();
    (b == 0.0) ? y.HypreWrite() : y.HypreReadWrite();
    return hypre_ParCSRMatrixMatvecT(a, A, x, b, y);
@@ -2222,7 +2218,7 @@ void HypreParMatrix::Threshold(double threshold)
    {
       hypre_CSRMatrixReorder(hypre_ParCSRMatrixDiag(A));
    }
-   hypre_MatvecCommPkgCreate(A);
+   if (!hypre_ParCSRMatrixCommPkg(A)) { hypre_MatvecCommPkgCreate(A); }
    height = GetNumRows();
    width = GetNumCols();
 }
@@ -2544,7 +2540,7 @@ void HypreParMatrix::Read(MPI_Comm comm, const char *fname)
    hypre_ParCSRMatrixReadIJ(comm, fname, &base_i, &base_j, &A);
    hypre_ParCSRMatrixSetNumNonzeros(A);
 
-   hypre_MatvecCommPkgCreate(A);
+   if (!hypre_ParCSRMatrixCommPkg(A)) { hypre_MatvecCommPkgCreate(A); }
 
    height = GetNumRows();
    width = GetNumCols();
@@ -2565,7 +2561,7 @@ void HypreParMatrix::Read_IJMatrix(MPI_Comm comm, const char *fname)
 
    hypre_ParCSRMatrixSetNumNonzeros(A);
 
-   hypre_MatvecCommPkgCreate(A);
+   if (!hypre_ParCSRMatrixCommPkg(A)) { hypre_MatvecCommPkgCreate(A); }
 
    height = GetNumRows();
    width = GetNumCols();
@@ -2801,7 +2797,7 @@ HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
                                       const_cast<HypreParMatrix &>(B));
    MFEM_VERIFY(C_hypre, "error in hypre_ParCSRMatrixAdd");
 
-   hypre_MatvecCommPkgCreate(C_hypre);
+   if (!hypre_ParCSRMatrixCommPkg(C_hypre)) { hypre_MatvecCommPkgCreate(C_hypre); }
    HypreParMatrix *C = new HypreParMatrix(C_hypre);
    *C = 0.0;
    C->Add(alpha, A);
@@ -2814,7 +2810,7 @@ HypreParMatrix * ParAdd(const HypreParMatrix *A, const HypreParMatrix *B)
 {
    hypre_ParCSRMatrix * C = internal::hypre_ParCSRMatrixAdd(*A,*B);
 
-   hypre_MatvecCommPkgCreate(C);
+   if (!hypre_ParCSRMatrixCommPkg(C)) { hypre_MatvecCommPkgCreate(C); }
 
    return new HypreParMatrix(C);
 }
@@ -2830,7 +2826,7 @@ HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
 #else
    hypre_ParCSRMatrixAdd(alpha, A, beta, B, &C);
 #endif
-   hypre_MatvecCommPkgCreate(C);
+   if (!hypre_ParCSRMatrixCommPkg(C)) { hypre_MatvecCommPkgCreate(C); }
 
    return new HypreParMatrix(C);
 }
@@ -2843,6 +2839,7 @@ HypreParMatrix * ParAdd(const HypreParMatrix *A, const HypreParMatrix *B)
 #else
    hypre_ParCSRMatrixAdd(1.0, *A, 1.0, *B, &C);
 #endif
+   if (!hypre_ParCSRMatrixCommPkg(C)) { hypre_MatvecCommPkgCreate(C); }
 
    return new HypreParMatrix(C);
 }
@@ -2860,7 +2857,7 @@ HypreParMatrix * ParMult(const HypreParMatrix *A, const HypreParMatrix *B,
 #endif
    hypre_ParCSRMatrixSetNumNonzeros(ab);
 
-   hypre_MatvecCommPkgCreate(ab);
+   if (!hypre_ParCSRMatrixCommPkg(ab)) { hypre_MatvecCommPkgCreate(ab); }
    HypreParMatrix *C = new HypreParMatrix(ab);
    if (own_matrix)
    {
@@ -4570,6 +4567,31 @@ void HypreParaSails::SetOperator(const Operator &op)
    auxX.Delete(); auxX.Reset();
 }
 
+void HypreParaSails::SetParams(double threshold, int max_levels)
+{
+   HYPRE_ParaSailsSetParams(sai_precond, threshold, max_levels);
+}
+
+void HypreParaSails::SetFilter(double filter)
+{
+   HYPRE_ParaSailsSetFilter(sai_precond, filter);
+}
+
+void HypreParaSails::SetLoadBal(double loadbal)
+{
+   HYPRE_ParaSailsSetLoadbal(sai_precond, loadbal);
+}
+
+void HypreParaSails::SetReuse(int reuse)
+{
+   HYPRE_ParaSailsSetReuse(sai_precond, reuse);
+}
+
+void HypreParaSails::SetLogging(int logging)
+{
+   HYPRE_ParaSailsSetLogging(sai_precond, logging);
+}
+
 void HypreParaSails::SetSymmetry(int sym)
 {
    HYPRE_ParaSailsSetSym(sai_precond, sym);
@@ -4612,6 +4634,31 @@ void HypreEuclid::SetDefaultOptions()
    HYPRE_EuclidSetRowScale(euc_precond, euc_ro_sc);
 }
 
+void HypreEuclid::SetLevel(int level)
+{
+   HYPRE_EuclidSetLevel(euc_precond, level);
+}
+
+void HypreEuclid::SetStats(int stats)
+{
+   HYPRE_EuclidSetStats(euc_precond, stats);
+}
+
+void HypreEuclid::SetMemory(int mem)
+{
+   HYPRE_EuclidSetMem(euc_precond, mem);
+}
+
+void HypreEuclid::SetBJ(int bj)
+{
+   HYPRE_EuclidSetBJ(euc_precond, bj);
+}
+
+void HypreEuclid::SetRowScale(int row_scale)
+{
+   HYPRE_EuclidSetRowScale(euc_precond, row_scale);
+}
+
 void HypreEuclid::ResetEuclidPrecond(MPI_Comm comm)
 {
    // Euclid does not seem to offer access to its current configuration, so we
@@ -4701,6 +4748,26 @@ void HypreILU::SetLevelOfFill(HYPRE_Int lev_fill)
    HYPRE_ILUSetLevelOfFill(ilu_precond, lev_fill);
 }
 
+void HypreILU::SetType(HYPRE_Int ilu_type)
+{
+   HYPRE_ILUSetType(ilu_precond, ilu_type);
+}
+
+void HypreILU::SetMaxIter(HYPRE_Int max_iter)
+{
+   HYPRE_ILUSetMaxIter(ilu_precond, max_iter);
+}
+
+void HypreILU::SetTol(HYPRE_Real tol)
+{
+   HYPRE_ILUSetTol(ilu_precond, tol);
+}
+
+void HypreILU::SetLocalReordering(HYPRE_Int reorder_type)
+{
+   HYPRE_ILUSetLocalReordering(ilu_precond, reorder_type);
+}
+
 void HypreILU::SetPrintLevel(HYPRE_Int print_level)
 {
    HYPRE_ILUSetPrintLevel(ilu_precond, print_level);
@@ -5238,6 +5305,8 @@ void HypreAMS::MakeSolver(int sdim, int cycle_type)
    int amg_Pmax         = 4;
 #endif
 
+   space_dim = sdim;
+   ams_cycle_type = cycle_type;
    HYPRE_AMSCreate(&ams);
 
    HYPRE_AMSSetDimension(ams, sdim); // 2D H(div) and 3D H(curl) problems
@@ -5432,11 +5501,136 @@ void HypreAMS::Init(ParFiniteElementSpace *edge_fespace)
    MakeGradientAndInterpolation(edge_fespace, cycle_type);
 }
 
+void HypreAMS::ResetAMSPrecond()
+{
+#if MFEM_HYPRE_VERSION >= 22600
+   /* Read options from ams */
+   auto *ams_data = (hypre_AMSData *)ams;
+
+   /* Space dimension */
+   HYPRE_Int dim = hypre_AMSDataDimension(ams_data);
+
+   /* Vertex space data */
+   hypre_ParCSRMatrix *hy_G = hypre_AMSDataDiscreteGradient(ams_data);
+
+   HYPRE_Int beta_is_zero = hypre_AMSDataBetaIsZero(ams_data);
+
+   /* Vector vertex space data */
+   hypre_ParCSRMatrix *hy_Pi hypre_AMSDataPiInterpolation(ams_data);
+   hypre_ParCSRMatrix *hy_Pix = ams_data->Pix;
+   hypre_ParCSRMatrix *hy_Piy = ams_data->Piy;
+   hypre_ParCSRMatrix *hy_Piz = ams_data->Piz;
+   HYPRE_Int owns_Pi = hypre_AMSDataOwnsPiInterpolation(ams_data);
+   if (owns_Pi)
+   {
+      ams_data->owns_Pi = 0; // we're stealing Pi
+   }
+
+   /* Coordinates of the vertices */
+   hypre_ParVector *hy_x = hypre_AMSDataVertexCoordinateX(ams_data);
+   hypre_ParVector *hy_y = hypre_AMSDataVertexCoordinateY(ams_data);
+   hypre_ParVector *hy_z = hypre_AMSDataVertexCoordinateZ(ams_data);
+
+   /* Solver options */
+   HYPRE_Int maxit = hypre_AMSDataMaxIter(ams_data);
+   HYPRE_Real tol = hypre_AMSDataTol(ams_data);
+   HYPRE_Int cycle_type = hypre_AMSDataCycleType(ams_data);
+   HYPRE_Int ams_print_level = hypre_AMSDataPrintLevel(ams_data);
+
+   /* Smoothing and AMG options */
+   HYPRE_Int A_relax_type = hypre_AMSDataARelaxType(ams_data);
+   HYPRE_Int A_relax_times = hypre_AMSDataARelaxTimes(ams_data);
+   HYPRE_Real A_relax_weight = hypre_AMSDataARelaxWeight(ams_data);
+   HYPRE_Real A_omega = hypre_AMSDataAOmega(ams_data);
+   HYPRE_Int A_cheby_order = hypre_AMSDataAChebyOrder(ams_data);
+   HYPRE_Real A_cheby_fraction = hypre_AMSDataAChebyFraction(ams_data);
+
+   HYPRE_Int B_Pi_coarsen_type = hypre_AMSDataPoissonAlphaAMGCoarsenType(ams_data);
+   HYPRE_Int B_Pi_agg_levels = hypre_AMSDataPoissonAlphaAMGAggLevels(ams_data);
+   HYPRE_Int B_Pi_relax_type = hypre_AMSDataPoissonAlphaAMGRelaxType(ams_data);
+   HYPRE_Int B_Pi_coarse_relax_type = ams_data->B_Pi_coarse_relax_type;
+   HYPRE_Real B_Pi_theta = hypre_AMSDataPoissonAlphaAMGStrengthThreshold(ams_data);
+   HYPRE_Int B_Pi_interp_type = ams_data->B_Pi_interp_type;
+   HYPRE_Int B_Pi_Pmax = ams_data->B_Pi_Pmax;
+
+   HYPRE_Int B_G_coarsen_type = hypre_AMSDataPoissonBetaAMGCoarsenType(ams_data);
+   HYPRE_Int B_G_agg_levels = hypre_AMSDataPoissonBetaAMGAggLevels(ams_data);
+   HYPRE_Int B_G_relax_type = hypre_AMSDataPoissonBetaAMGRelaxType(ams_data);
+   HYPRE_Int B_G_coarse_relax_type = ams_data->B_G_coarse_relax_type;
+   HYPRE_Real B_G_theta = hypre_AMSDataPoissonBetaAMGStrengthThreshold(ams_data);
+   HYPRE_Int B_G_interp_type = ams_data->B_G_interp_type;
+   HYPRE_Int B_G_Pmax = ams_data->B_G_Pmax;
+
+   HYPRE_AMSDestroy(ams);
+   HYPRE_AMSCreate(&ams);
+   ams_data = (hypre_AMSData *)ams;
+
+   HYPRE_AMSSetDimension(ams, dim); // 2D H(div) and 3D H(curl) problems
+   HYPRE_AMSSetTol(ams, tol);
+   HYPRE_AMSSetMaxIter(ams, maxit); // use as a preconditioner
+   HYPRE_AMSSetCycleType(ams, cycle_type);
+   HYPRE_AMSSetPrintLevel(ams, ams_print_level);
+
+   HYPRE_AMSSetCoordinateVectors(ams, hy_x, hy_y, hy_z);
+
+   HYPRE_AMSSetDiscreteGradient(ams, hy_G);
+   HYPRE_AMSSetCoordinateVectors(ams, hy_x, hy_y, hy_z);
+   HYPRE_AMSSetInterpolations(ams, hy_Pi, hy_Pix, hy_Piy, hy_Piz);
+   ams_data->owns_Pi = owns_Pi;
+
+   // set additional AMS options
+   HYPRE_AMSSetSmoothingOptions(ams, A_relax_type, A_relax_times, A_relax_weight,
+                                A_omega);
+
+   hypre_AMSDataAChebyOrder(ams_data) = A_cheby_order;
+   hypre_AMSDataAChebyFraction(ams_data) = A_cheby_fraction;
+
+   HYPRE_AMSSetAlphaAMGOptions(ams, B_Pi_coarsen_type, B_Pi_agg_levels,
+                               B_Pi_relax_type,
+                               B_Pi_theta, B_Pi_interp_type, B_Pi_Pmax);
+   HYPRE_AMSSetBetaAMGOptions(ams, B_G_coarsen_type, B_G_agg_levels,
+                              B_G_relax_type,
+                              B_G_theta, B_G_interp_type, B_G_Pmax);
+
+   HYPRE_AMSSetAlphaAMGCoarseRelaxType(ams, B_Pi_coarse_relax_type);
+   HYPRE_AMSSetBetaAMGCoarseRelaxType(ams, B_G_coarse_relax_type);
+
+   ams_data->beta_is_zero = beta_is_zero;
+
+#else
+   HYPRE_AMSDestroy(ams);
+
+   MakeSolver(space_dim, ams_cycle_type);
+
+   HYPRE_AMSSetPrintLevel(ams, print_level);
+   if (singular) { HYPRE_AMSSetBetaPoissonMatrix(ams, NULL); }
+
+   HYPRE_AMSSetDiscreteGradient(ams, *G);
+   if (x != nullptr)
+   {
+      HYPRE_AMSSetCoordinateVectors(ams,
+                                    x ? (HYPRE_ParVector)(*x) : nullptr,
+                                    y ? (HYPRE_ParVector)(*y) : nullptr,
+                                    z ? (HYPRE_ParVector)(*z) : nullptr);
+   }
+   else
+   {
+      HYPRE_AMSSetInterpolations(ams,
+                                 Pi ? (HYPRE_ParCSRMatrix) *Pi : nullptr,
+                                 Pix ? (HYPRE_ParCSRMatrix) *Pix : nullptr,
+                                 Piy ? (HYPRE_ParCSRMatrix) *Piy : nullptr,
+                                 Piz ? (HYPRE_ParCSRMatrix) *Piz : nullptr);
+   }
+#endif
+}
+
 void HypreAMS::SetOperator(const Operator &op)
 {
    const HypreParMatrix *new_A = dynamic_cast<const HypreParMatrix *>(&op);
    MFEM_VERIFY(new_A, "new Operator must be a HypreParMatrix!");
 
+   if (A) { ResetAMSPrecond(); }
+
    // update base classes: Operator, Solver, HypreSolver
    height = new_A->Height();
    width  = new_A->Width();
@@ -5468,6 +5662,7 @@ HypreAMS::~HypreAMS()
 void HypreAMS::SetPrintLevel(int print_lvl)
 {
    HYPRE_AMSSetPrintLevel(ams, print_lvl);
+   print_level = print_lvl;
 }
 
 HypreADS::HypreADS(ParFiniteElementSpace *face_fespace)
@@ -5495,17 +5690,15 @@ HypreADS::HypreADS(
    MFEM_ASSERT(x != NULL, "");
    MFEM_ASSERT(y != NULL, "");
    MFEM_ASSERT(z != NULL, "");
-   int cycle_type = 11;
-   int ams_cycle_type = 14;
 
-   MakeSolver(cycle_type, ams_cycle_type);
+   MakeSolver();
 
    HYPRE_ADSSetCoordinateVectors(ads, *x, *y, *z);
    HYPRE_ADSSetDiscreteCurl(ads, *C);
    HYPRE_ADSSetDiscreteGradient(ads, *G);
 }
 
-void HypreADS::MakeSolver(int cycle_type, int ams_cycle_type)
+void HypreADS::MakeSolver()
 {
    int rlx_sweeps       = 1;
    double rlx_weight    = 1.0;
@@ -5549,10 +5742,7 @@ void HypreADS::MakeSolver(int cycle_type, int ams_cycle_type)
    error_mode = IGNORE_HYPRE_ERRORS;
 }
 
-void HypreADS::MakeDiscreteMatrices(
-   ParFiniteElementSpace *face_fespace,
-   int cycle_type,
-   int ams_cycle_type)
+void HypreADS::MakeDiscreteMatrices(ParFiniteElementSpace *face_fespace)
 {
    const FiniteElementCollection *face_fec = face_fespace->FEColl();
    bool trace_space =
@@ -5748,10 +5938,41 @@ void HypreADS::MakeDiscreteMatrices(
 
 void HypreADS::Init(ParFiniteElementSpace *face_fespace)
 {
-   int cycle_type = 11;
-   int ams_cycle_type = 14;
-   MakeSolver(cycle_type, ams_cycle_type);
-   MakeDiscreteMatrices(face_fespace, cycle_type, ams_cycle_type);
+   MakeSolver();
+   MakeDiscreteMatrices(face_fespace);
+}
+
+void HypreADS::ResetADSPrecond()
+{
+   HYPRE_ADSDestroy(ads);
+
+   MakeSolver();
+
+   HYPRE_ADSSetPrintLevel(ads, print_level);
+
+   HYPRE_ADSSetDiscreteCurl(ads, *C);
+   HYPRE_ADSSetDiscreteGradient(ads, *G);
+   if (x != nullptr)
+   {
+      MFEM_VERIFY(x && y && z, "");
+      HYPRE_ADSSetCoordinateVectors(ads, *x, *y, *z);
+   }
+   else
+   {
+      HYPRE_ParCSRMatrix HY_RT_Pi, HY_RT_Pix, HY_RT_Piy, HY_RT_Piz;
+      HY_RT_Pi  = (RT_Pi)  ? (HYPRE_ParCSRMatrix) *RT_Pi  : NULL;
+      HY_RT_Pix = (RT_Pix) ? (HYPRE_ParCSRMatrix) *RT_Pix : NULL;
+      HY_RT_Piy = (RT_Piy) ? (HYPRE_ParCSRMatrix) *RT_Piy : NULL;
+      HY_RT_Piz = (RT_Piz) ? (HYPRE_ParCSRMatrix) *RT_Piz : NULL;
+      HYPRE_ParCSRMatrix HY_ND_Pi, HY_ND_Pix, HY_ND_Piy, HY_ND_Piz;
+      HY_ND_Pi  = (ND_Pi)  ? (HYPRE_ParCSRMatrix) *ND_Pi  : NULL;
+      HY_ND_Pix = (ND_Pix) ? (HYPRE_ParCSRMatrix) *ND_Pix : NULL;
+      HY_ND_Piy = (ND_Piy) ? (HYPRE_ParCSRMatrix) *ND_Piy : NULL;
+      HY_ND_Piz = (ND_Piz) ? (HYPRE_ParCSRMatrix) *ND_Piz : NULL;
+      HYPRE_ADSSetInterpolations(ads,
+                                 HY_RT_Pi, HY_RT_Pix, HY_RT_Piy, HY_RT_Piz,
+                                 HY_ND_Pi, HY_ND_Pix, HY_ND_Piy, HY_ND_Piz);
+   }
 }
 
 void HypreADS::SetOperator(const Operator &op)
@@ -5759,6 +5980,8 @@ void HypreADS::SetOperator(const Operator &op)
    const HypreParMatrix *new_A = dynamic_cast<const HypreParMatrix *>(&op);
    MFEM_VERIFY(new_A, "new Operator must be a HypreParMatrix!");
 
+   if (A) { ResetADSPrecond(); }
+
    // update base classes: Operator, Solver, HypreSolver
    height = new_A->Height();
    width  = new_A->Width();
@@ -5797,6 +6020,7 @@ HypreADS::~HypreADS()
 void HypreADS::SetPrintLevel(int print_lvl)
 {
    HYPRE_ADSSetPrintLevel(ads, print_lvl);
+   print_level = print_lvl;
 }
 
 HypreLOBPCG::HypreMultiVector::HypreMultiVector(int n, HypreParVector & v,

linalg/hypre.hpp

@@ -327,13 +327,6 @@ public:
 
    /// Calls hypre's destroy function
    ~HypreParVector();
-
-#ifdef MFEM_USE_SUNDIALS
-   /// (DEPRECATED) Return a new wrapper SUNDIALS N_Vector of type SUNDIALS_NVEC_PARALLEL.
-   /** @deprecated The returned N_Vector must be destroyed by the caller. */
-   MFEM_DEPRECATED virtual N_Vector ToNVector();
-   using Vector::ToNVector;
-#endif
 };
 
 /// Returns the inner product of x and y
@@ -661,24 +654,50 @@ public:
    /// Ensure the action of the transpose is performed fast.
    /** When HYPRE is built for GPUs, this method will construct and store the
        transposes of the 'diag' and 'offd' CSR matrices. When HYPRE is not built
-       for GPUs, this method is a no-op. */
+       for GPUs, this method is a no-op.
+
+       This method is automatically called by MultTranspose().
+
+       If the matrix is modified the old transpose blocks can be deleted by
+       calling ResetTranspose(). */
    void EnsureMultTranspose() const;
 
+   /** @brief Reset (destroy) the internal transpose matrix that is created by
+       EnsureMultTranspose() and MultTranspose().
+
+       If the matrix is modified, this method should be called to delete the
+       out-of-date transpose that is stored internally. */
+   void ResetTranspose() const;
+
    /// Computes y = alpha * A * x + beta * y
    HYPRE_Int Mult(HypreParVector &x, HypreParVector &y,
                   double alpha = 1.0, double beta = 0.0) const;
    /// Computes y = alpha * A * x + beta * y
    HYPRE_Int Mult(HYPRE_ParVector x, HYPRE_ParVector y,
                   double alpha = 1.0, double beta = 0.0) const;
+
    /// Computes y = alpha * A^t * x + beta * y
+   /** If the matrix is modified, call ResetTranspose() and optionally
+       EnsureMultTranspose() to make sure this method uses the correct updated
+       transpose. */
    HYPRE_Int MultTranspose(HypreParVector &x, HypreParVector &y,
                            double alpha = 1.0, double beta = 0.0) const;
 
    void Mult(double a, const Vector &x, double b, Vector &y) const;
+
+   /// Computes y = alpha * A^t * x + beta * y
+   /** If the matrix is modified, call ResetTranspose() and optionally
+       EnsureMultTranspose() to make sure this method uses the correct updated
+       transpose. */
    void MultTranspose(double a, const Vector &x, double b, Vector &y) const;
 
    virtual void Mult(const Vector &x, Vector &y) const
    { Mult(1.0, x, 0.0, y); }
+
+   /// Computes y = A^t * x
+   /** If the matrix is modified, call ResetTranspose() and optionally
+       EnsureMultTranspose() to make sure this method uses the correct updated
+       transpose. */
    virtual void MultTranspose(const Vector &x, Vector &y) const
    { MultTranspose(1.0, x, 0.0, y); }
 
@@ -1223,6 +1242,12 @@ public:
       num_iterations = internal::to_int(num_it);
    }
 
+   void GetFinalResidualNorm(double &final_res_norm) const
+   {
+      HYPRE_ParCSRPCGGetFinalRelativeResidualNorm(pcg_solver,
+                                                  &final_res_norm);
+   }
+
    /// The typecast to HYPRE_Solver returns the internal pcg_solver
    virtual operator HYPRE_Solver() const { return pcg_solver; }
 
@@ -1274,6 +1299,19 @@ public:
    /// non-hypre setting
    void SetZeroInitialIterate() { iterative_mode = false; }
 
+   void GetNumIterations(int &num_iterations) const
+   {
+      HYPRE_Int num_it;
+      HYPRE_ParCSRGMRESGetNumIterations(gmres_solver, &num_it);
+      num_iterations = internal::to_int(num_it);
+   }
+
+   void GetFinalResidualNorm(double &final_res_norm) const
+   {
+      HYPRE_ParCSRGMRESGetFinalRelativeResidualNorm(gmres_solver,
+                                                    &final_res_norm);
+   }
+
    /// The typecast to HYPRE_Solver returns the internal gmres_solver
    virtual operator HYPRE_Solver() const  { return gmres_solver; }
 
@@ -1324,6 +1362,19 @@ public:
    /// non-hypre setting
    void SetZeroInitialIterate() { iterative_mode = false; }
 
+   void GetNumIterations(int &num_iterations) const
+   {
+      HYPRE_Int num_it;
+      HYPRE_ParCSRFlexGMRESGetNumIterations(fgmres_solver, &num_it);
+      num_iterations = internal::to_int(num_it);
+   }
+
+   void GetFinalResidualNorm(double &final_res_norm) const
+   {
+      HYPRE_ParCSRFlexGMRESGetFinalRelativeResidualNorm(fgmres_solver,
+                                                        &final_res_norm);
+   }
+
    /// The typecast to HYPRE_Solver returns the internal fgmres_solver
    virtual operator HYPRE_Solver() const  { return fgmres_solver; }
 
@@ -1400,6 +1451,11 @@ public:
 
    virtual void SetOperator(const Operator &op);
 
+   void SetParams(double threshold, int max_levels);
+   void SetFilter(double filter);
+   void SetLoadBal(double loadbal);
+   void SetReuse(int reuse);
+   void SetLogging(int logging);
    void SetSymmetry(int sym);
 
    /// The typecast to HYPRE_Solver returns the internal sai_precond
@@ -1439,6 +1495,12 @@ public:
 
    HypreEuclid(const HypreParMatrix &A);
 
+   void SetLevel(int level);
+   void SetStats(int stats);
+   void SetMemory(int mem);
+   void SetBJ(int bj);
+   void SetRowScale(int row_scale);
+
    virtual void SetOperator(const Operator &op);
 
    /// The typecast to HYPRE_Solver returns the internal euc_precond
@@ -1490,6 +1552,11 @@ public:
    /// Set the fill level for ILU(k); the default is k=1.
    void SetLevelOfFill(HYPRE_Int lev_fill);
 
+   void SetType(HYPRE_Int ilu_type);
+   void SetMaxIter(HYPRE_Int max_iter);
+   void SetTol(HYPRE_Real tol);
+   void SetLocalReordering(HYPRE_Int reorder_type);
+
    /// Set the print level: 0 = none, 1 = setup, 2 = solve, 3 = setup+solve
    void SetPrintLevel(HYPRE_Int print_level);
 
@@ -1686,6 +1753,19 @@ private:
    /// Nedelec interpolation matrix and its components
    HypreParMatrix *Pi, *Pix, *Piy, *Piz;
 
+   /// AMS cycle type
+   int ams_cycle_type = 0;
+   /// Spatial dimension of the underlying mesh
+   int space_dim = 0;
+   /// Flag set if `SetSingularProblem` is called, needed in `ResetAMSPrecond`
+   bool singular = false;
+   /// Flag set if `SetPrintLevel` is called, needed in `ResetAMSPrecond`
+   int print_level = 1;
+
+   // Recreates another AMS solver with the same options when SetOperator is
+   // called multiple times.
+   void ResetAMSPrecond();
+
 public:
    /// @brief Construct the AMS solver on the given edge finite element space.
    ///
@@ -1708,7 +1788,11 @@ public:
    void SetPrintLevel(int print_lvl);
 
    /// Set this option when solving a curl-curl problem with zero mass term
-   void SetSingularProblem() { HYPRE_AMSSetBetaPoissonMatrix(ams, NULL); }
+   void SetSingularProblem()
+   {
+      HYPRE_AMSSetBetaPoissonMatrix(ams, NULL);
+      singular = true;
+   }
 
    /// The typecast to HYPRE_Solver returns the internal ams object
    virtual operator HYPRE_Solver() const { return ams; }
@@ -1728,15 +1812,13 @@ private:
    /// Construct ADS solver from finite element space
    void Init(ParFiniteElementSpace *face_fespace);
 
-   /// Create the hypre solver object and set the default options, given the
-   /// cycle type @a cycle_type and AMS cycle type @a ams_cycle_type.
-   void MakeSolver(int cycle_type, int ams_cycle_type);
+   /// Create the hypre solver object and set the default options, using the
+   /// cycle type cycle_type and AMS cycle type ams_cycle_type data members.
+   void MakeSolver();
 
    /// Construct the discrete curl, gradient and interpolation matrices
    /// associated with @a face_fespace, and add them to the solver.
-   void MakeDiscreteMatrices(ParFiniteElementSpace *face_fespace,
-                             int cycle_type,
-                             int ams_cycle_type);
+   void MakeDiscreteMatrices(ParFiniteElementSpace *face_fespace);
 
    HYPRE_Solver ads;
 
@@ -1751,6 +1833,16 @@ private:
    /// Raviart-Thomas interpolation matrix and its components
    HypreParMatrix *RT_Pi, *RT_Pix, *RT_Piy, *RT_Piz;
 
+   /// ADS cycle type
+   const int cycle_type = 11;
+   /// AMS cycle type
+   const int ams_cycle_type = 14;
+   /// ADS print level
+   int print_level = 1;
+
+   // Recreates another ADS solver with the same options when SetOperator is
+   // called multiple times.
+   void ResetADSPrecond();
 public:
    HypreADS(ParFiniteElementSpace *face_fespace);
 

linalg/invariants.hpp

@@ -497,7 +497,10 @@ protected:
    // Transformation Jacobian
    const scalar_t *J;
 
-   // Invariants:
+   // Invatiants: I1b = det(J)^{-2/3} * ||J||_F^2
+   //             I2b = det(J)^{ 2/3} * ||J^{-1}||_F^2
+   //             I3b = det(J)
+   // Computed as:
    //    I_1 = ||J||_F^2, \bar{I}_1 = det(J)^{-2/3}*I_1,
    //    I_2 = (1/2)*(||J||_F^4-||J J^t||_F^2) = (1/2)*(I_1^2-||J J^t||_F^2),
    //    \bar{I}_2 = det(J)^{-4/3}*I_2,

linalg/linalg.hpp

@@ -19,6 +19,7 @@
 #include "matrix.hpp"
 #include "sparsemat.hpp"
 #include "complex_operator.hpp"
+#include "complex_densemat.hpp"
 #include "blockvector.hpp"
 #include "blockmatrix.hpp"
 #include "blockoperator.hpp"

linalg/operator.hpp

@@ -267,6 +267,7 @@ public:
       Complex_Operator, ///< ID for class ComplexOperator.
       MFEM_ComplexSparseMat, ///< ID for class ComplexSparseMatrix.
       Complex_Hypre_ParCSR,   ///< ID for class ComplexHypreParMatrix.
+      Complex_DenseMat,  ///< ID for class ComplexDenseMatrix
       MFEM_Block_Matrix,     ///< ID for class BlockMatrix.
       MFEM_Block_Operator   ///< ID for the base class BlockOperator.
    };

linalg/petsc.cpp

@@ -460,6 +460,11 @@ PetscInt PetscParVector::GlobalSize() const
    return N;
 }
 
+void PetscParVector::SetBlockSize(PetscInt bs)
+{
+   ierr = VecSetBlockSize(x,bs); PCHKERRQ(x,ierr);
+}
+
 PetscParVector::PetscParVector(MPI_Comm comm, const Vector &x_,
                                bool copy) : Vector()
 {
@@ -942,6 +947,12 @@ PetscInt PetscParMatrix::NNZ() const
    return (PetscInt)info.nz_used;
 }
 
+void PetscParMatrix::SetBlockSize(PetscInt rbs, PetscInt cbs)
+{
+   if (cbs < 0) { cbs = rbs; }
+   ierr = MatSetBlockSizes(A,rbs,cbs); PCHKERRQ(A,ierr);
+}
+
 void PetscParMatrix::Init()
 {
    A = NULL;
@@ -1324,7 +1335,12 @@ void PetscParMatrix::ConvertOperator(MPI_Comm comm, const Operator &op, Mat* A,
       PetscBool ismatis;
 #endif
 
+#if PETSC_VERSION_LT(3,18,0)
       ierr = PetscObjectTypeCompare((PetscObject)(pA->A),MATTRANSPOSEMAT,&istrans);
+#else
+      ierr = PetscObjectTypeCompare((PetscObject)(pA->A),MATTRANSPOSEVIRTUAL,
+                                    &istrans);
+#endif
       CCHKERRQ(pA->GetComm(),ierr);
       if (!istrans)
       {
@@ -2839,37 +2855,43 @@ void PetscBCHandler::ZeroBC(const Vector &x, Vector &y)
 // PetscLinearSolver methods
 
 PetscLinearSolver::PetscLinearSolver(MPI_Comm comm, const std::string &prefix,
-                                     bool wrapin)
-   : PetscSolver(), Solver(), wrap(wrapin)
+                                     bool wrapin, bool iter_mode)
+   : PetscSolver(), Solver(0,iter_mode), wrap(wrapin)
 {
    KSP ksp;
    ierr = KSPCreate(comm,&ksp); CCHKERRQ(comm,ierr);
    obj  = (PetscObject)ksp;
    ierr = PetscObjectGetClassId(obj,&cid); PCHKERRQ(obj,ierr);
    ierr = KSPSetOptionsPrefix(ksp, prefix.c_str()); PCHKERRQ(ksp, ierr);
+   ierr = KSPSetInitialGuessNonzero(ksp, (PetscBool)iterative_mode);
+   PCHKERRQ(ksp, ierr);
 }
 
 PetscLinearSolver::PetscLinearSolver(const PetscParMatrix &A,
-                                     const std::string &prefix)
-   : PetscSolver(), Solver(), wrap(false)
+                                     const std::string &prefix, bool iter_mode)
+   : PetscSolver(), Solver(0,iter_mode), wrap(false)
 {
    KSP ksp;
    ierr = KSPCreate(A.GetComm(),&ksp); CCHKERRQ(A.GetComm(),ierr);
    obj  = (PetscObject)ksp;
    ierr = PetscObjectGetClassId(obj,&cid); PCHKERRQ(obj,ierr);
    ierr = KSPSetOptionsPrefix(ksp, prefix.c_str()); PCHKERRQ(ksp, ierr);
+   ierr = KSPSetInitialGuessNonzero(ksp, (PetscBool)iterative_mode);
+   PCHKERRQ(ksp, ierr);
    SetOperator(A);
 }
 
 PetscLinearSolver::PetscLinearSolver(const HypreParMatrix &A, bool wrapin,
-                                     const std::string &prefix)
-   : PetscSolver(), Solver(), wrap(wrapin)
+                                     const std::string &prefix, bool iter_mode)
+   : PetscSolver(), Solver(0,iter_mode), wrap(wrapin)
 {
    KSP ksp;
    ierr = KSPCreate(A.GetComm(),&ksp); CCHKERRQ(A.GetComm(),ierr);
    obj  = (PetscObject)ksp;
    ierr = PetscObjectGetClassId(obj, &cid); PCHKERRQ(obj, ierr);
    ierr = KSPSetOptionsPrefix(ksp, prefix.c_str()); PCHKERRQ(ksp, ierr);
+   ierr = KSPSetInitialGuessNonzero(ksp, (PetscBool)iterative_mode);
+   PCHKERRQ(ksp, ierr);
    SetOperator(A);
 }
 
@@ -3078,12 +3100,12 @@ void PetscLinearSolver::MultKernel(const Vector &b, Vector &x, bool trans) const
       }
    }
    B->PlaceMemory(b.GetMemory());
-   X->PlaceMemory(x.GetMemory(),iterative_mode);
 
    Customize();
 
-   ierr = KSPSetInitialGuessNonzero(ksp, (PetscBool)iterative_mode);
-   PCHKERRQ(ksp, ierr);
+   PetscBool flg;
+   ierr = KSPGetInitialGuessNonzero(ksp, &flg);
+   X->PlaceMemory(x.GetMemory(),flg);
 
    // Solve the system.
    if (trans)
@@ -3118,8 +3140,9 @@ PetscLinearSolver::~PetscLinearSolver()
 
 // PetscPCGSolver methods
 
-PetscPCGSolver::PetscPCGSolver(MPI_Comm comm, const std::string &prefix)
-   : PetscLinearSolver(comm,prefix)
+PetscPCGSolver::PetscPCGSolver(MPI_Comm comm, const std::string &prefix,
+                               bool iter_mode)
+   : PetscLinearSolver(comm,prefix,iter_mode)
 {
    KSP ksp = (KSP)obj;
    ierr = KSPSetType(ksp,KSPCG); PCHKERRQ(ksp,ierr);
@@ -3127,8 +3150,9 @@ PetscPCGSolver::PetscPCGSolver(MPI_Comm comm, const std::string &prefix)
    ierr = KSPSetNormType(ksp,KSP_NORM_NATURAL); PCHKERRQ(ksp,ierr);
 }
 
-PetscPCGSolver::PetscPCGSolver(PetscParMatrix& A, const std::string &prefix)
-   : PetscLinearSolver(A,prefix)
+PetscPCGSolver::PetscPCGSolver(PetscParMatrix& A, const std::string &prefix,
+                               bool iter_mode)
+   : PetscLinearSolver(A,prefix,iter_mode)
 {
    KSP ksp = (KSP)obj;
    ierr = KSPSetType(ksp,KSPCG); PCHKERRQ(ksp,ierr);
@@ -3137,8 +3161,8 @@ PetscPCGSolver::PetscPCGSolver(PetscParMatrix& A, const std::string &prefix)
 }
 
 PetscPCGSolver::PetscPCGSolver(HypreParMatrix& A, bool wrap,
-                               const std::string &prefix)
-   : PetscLinearSolver(A,wrap,prefix)
+                               const std::string &prefix, bool iter_mode)
+   : PetscLinearSolver(A,wrap,prefix,iter_mode)
 {
    KSP ksp = (KSP)obj;
    ierr = KSPSetType(ksp,KSPCG); PCHKERRQ(ksp,ierr);

linalg/petsc.hpp

@@ -239,16 +239,24 @@ public:
    /// Set constant values
    PetscParVector& operator= (PetscScalar d);
 
+   /** @brief Set block size of a vector.
+
+       @note This will error if the local size of the vector is not a multiple
+       of the block size @a bs.
+       @note This is a logically collective operation, so all processes need
+       to call it. */
+   void SetBlockSize(PetscInt bs);
+
    /** @brief Set values in a vector.
 
-       @note any process can insert in any location
-       @note This is a collective operation, so all process needs to call it  */
+       @note Any process can insert in any location.
+       @note This is a collective operation, so all processes need to call it.  */
    PetscParVector& SetValues(const Array<PetscInt>&, const Array<PetscScalar>&);
 
    /** @brief Add values in a vector.
 
-       @note any process can add to any location
-       @note This is a collective operation, so all process needs to call it  */
+       @note Any process can add to any location.
+       @note This is a collective operation, so all processes need to call it.  */
    PetscParVector& AddValues(const Array<PetscInt>&, const Array<PetscScalar>&);
 
    /// Define operators for PETSc vectors.
@@ -526,6 +534,14 @@ public:
    /** @brief Eliminate only the rows from the matrix */
    void EliminateRows(const Array<int> &rows);
 
+   /** @brief Set row and column block sizes of a matrix.
+
+       @note This will error if the local sizes of the matrix are not a
+       multiple of the block sizes.
+       @note This is a logically collective operation, so all processes need
+       to call it.  */
+   void SetBlockSize(PetscInt rbs,PetscInt cbs=-1);
+
    /// Makes this object a reference to another PetscParMatrix
    void MakeRef(const PetscParMatrix &master);
 
@@ -728,16 +744,16 @@ private:
 
 public:
    PetscLinearSolver(MPI_Comm comm, const std::string &prefix = std::string(),
-                     bool wrap = true);
+                     bool wrap = true, bool iter_mode = false);
    PetscLinearSolver(const PetscParMatrix &A,
-                     const std::string &prefix = std::string());
+                     const std::string &prefix = std::string(), bool iter_mode = false);
    /// Constructs a solver using a HypreParMatrix.
    /** If @a wrap is true, then the MatMult ops of HypreParMatrix are wrapped.
        No preconditioner can be automatically constructed from PETSc. If
        @a wrap is false, the HypreParMatrix is converted into a the AIJ
        PETSc format, which is suitable for most preconditioning methods. */
    PetscLinearSolver(const HypreParMatrix &A, bool wrap = true,
-                     const std::string &prefix = std::string());
+                     const std::string &prefix = std::string(), bool iter_mode = false);
    virtual ~PetscLinearSolver();
 
    /// Sets the operator to be used for mat-vec operations and
@@ -764,10 +780,12 @@ public:
 class PetscPCGSolver : public PetscLinearSolver
 {
 public:
-   PetscPCGSolver(MPI_Comm comm, const std::string &prefix = std::string());
-   PetscPCGSolver(PetscParMatrix &A, const std::string &prefix = std::string());
-   PetscPCGSolver(HypreParMatrix &A,bool wrap=true,
-                  const std::string &prefix = std::string());
+   PetscPCGSolver(MPI_Comm comm, const std::string &prefix = std::string(),
+                  bool iter_mode = false);
+   PetscPCGSolver(PetscParMatrix &A, const std::string &prefix = std::string(),
+                  bool iter_mode = false);
+   PetscPCGSolver(HypreParMatrix &A, bool wrap = true,
+                  const std::string &prefix = std::string(), bool iter_mode = false);
 };
 
 

linalg/sparsemat.cpp

@@ -15,6 +15,7 @@
 #include "../general/forall.hpp"
 #include "../general/table.hpp"
 #include "../general/sort_pairs.hpp"
+#include "../general/backends.hpp"
 
 #include <iostream>
 #include <iomanip>
@@ -462,25 +463,108 @@ void SparseMatrix::SortColumnIndices()
       return;
    }
 
-   const int * Ip=HostReadI();
-   HostReadWriteJ();
-   HostReadWriteData();
-
-   Array<Pair<int,double> > row;
-   for (int j = 0, i = 0; i < height; i++)
+#ifdef MFEM_USE_CUDA_OR_HIP
+   if ( Device::Allows( Backend::CUDA_MASK ))
    {
-      int end = Ip[i+1];
-      row.SetSize(end - j);
-      for (int k = 0; k < row.Size(); k++)
+#if defined(MFEM_USE_CUDA)
+      size_t pBufferSizeInBytes = 0;
+      void *pBuffer = NULL;
+
+      const int n = Height();
+      const int m = Width();
+      const int nnzA = J.Capacity();
+      double * d_a_sorted = ReadWriteData();
+      const int * d_ia = ReadI();
+      int * d_ja_sorted = ReadWriteJ();
+      csru2csrInfo_t sortInfoA;
+
+      cusparseMatDescr_t matA_descr;
+      cusparseCreateMatDescr( &matA_descr );
+      cusparseSetMatIndexBase( matA_descr, CUSPARSE_INDEX_BASE_ZERO );
+      cusparseSetMatType( matA_descr, CUSPARSE_MATRIX_TYPE_GENERAL );
+
+      cusparseCreateCsru2csrInfo( &sortInfoA );
+
+      cusparseDcsru2csr_bufferSizeExt( handle, n, m, nnzA, d_a_sorted, d_ia,
+                                       d_ja_sorted, sortInfoA,
+                                       &pBufferSizeInBytes);
+
+      CuMemAlloc( &pBuffer, pBufferSizeInBytes );
+
+      cusparseDcsru2csr( handle, n, m, nnzA, matA_descr, d_a_sorted, d_ia,
+                         d_ja_sorted, sortInfoA, pBuffer);
+
+      cusparseDestroyCsru2csrInfo( sortInfoA );
+      cusparseDestroyMatDescr( matA_descr );
+
+      CuMemFree( pBuffer );
+#endif
+   }
+   else if ( Device::Allows( Backend::HIP_MASK ))
+   {
+#if defined(MFEM_USE_HIP)
+      size_t pBufferSizeInBytes = 0;
+      void *pBuffer = NULL;
+      int *P = NULL;
+
+      const int n = Height();
+      const int m = Width();
+      const int nnzA = J.Capacity();
+      double * d_a_sorted = ReadWriteData();
+      const int * d_ia = ReadI();
+      int * d_ja_sorted = ReadWriteJ();
+
+      hipsparseMatDescr_t descrA;
+      hipsparseCreateMatDescr( &descrA );
+      // FIXME: There is not in-place version of csr sort in hipSPARSE currently, so we make
+      //        a temporary copy of the data for gthr, sort that, and then copy the sorted values
+      //        back to the array being returned. Where there is an in-place version available,
+      //        we should use it.
+      Array< double > a_tmp( nnzA );
+      double *d_a_tmp = a_tmp.Write();
+
+      hipsparseXcsrsort_bufferSizeExt(handle, n, m, nnzA, d_ia, d_ja_sorted,
+                                      &pBufferSizeInBytes);
+
+      HipMemAlloc( &pBuffer, pBufferSizeInBytes );
+      HipMemAlloc( &P, nnzA * sizeof(int) );
+
+      hipsparseCreateIdentityPermutation(handle, nnzA, P);
+      hipsparseXcsrsort(handle, n, m, nnzA, descrA, d_ia, d_ja_sorted, P, pBuffer);
+
+      hipsparseDgthr(handle, nnzA, d_a_sorted, d_a_tmp, P,
+                     HIPSPARSE_INDEX_BASE_ZERO);
+
+      A.CopyFrom( a_tmp.GetMemory(), nnzA );
+      hipsparseDestroyMatDescr( descrA );
+
+      HipMemFree( pBuffer );
+      HipMemFree( P );
+#endif
+   }
+   else
+#endif // MFEM_USE_CUDA_OR_HIP
+   {
+      const int * Ip=HostReadI();
+      HostReadWriteJ();
+      HostReadWriteData();
+
+      Array<Pair<int,double> > row;
+      for (int j = 0, i = 0; i < height; i++)
       {
-         row[k].one = J[j+k];
-         row[k].two = A[j+k];
-      }
-      row.Sort();
-      for (int k = 0; k < row.Size(); k++, j++)
-      {
-         J[j] = row[k].one;
-         A[j] = row[k].two;
+         int end = Ip[i+1];
+         row.SetSize(end - j);
+         for (int k = 0; k < row.Size(); k++)
+         {
+            row[k].one = J[j+k];
+            row[k].two = A[j+k];
+         }
+         row.Sort();
+         for (int k = 0; k < row.Size(); k++, j++)
+         {
+            J[j] = row[k].one;
+            A[j] = row[k].two;
+         }
       }
    }
    isSorted = true;
@@ -849,15 +933,13 @@ void SparseMatrix::AddMultTranspose(const Vector &x, Vector &y,
       return;
    }
 
+   EnsureMultTranspose();
    if (At)
    {
       At->AddMult(x, y, a);
    }
    else
    {
-      MFEM_VERIFY(!Device::Allows(~Backend::CPU_MASK), "transpose action with "
-                  "this backend is not enabled; see EnsureMultTranspose() for "
-                  "details.");
       for (int i = 0; i < height; i++)
       {
          const double xi = a * x[i];
@@ -1064,15 +1146,13 @@ void SparseMatrix::AbsMultTranspose(const Vector &x, Vector &y) const
       return;
    }
 
+   EnsureMultTranspose();
    if (At)
    {
       At->AbsMult(x, y);
    }
    else
    {
-      MFEM_VERIFY(!Device::Allows(~Backend::CPU_MASK), "transpose action with "
-                  "this backend is not enabled; see EnsureMultTranspose() for "
-                  "details.");
       for (int i = 0; i < height; i++)
       {
          const double xi = x[i];

linalg/sparsemat.hpp

@@ -348,9 +348,15 @@ public:
    void AddMult(const Vector &x, Vector &y, const double a = 1.0) const;
 
    /// Multiply a vector with the transposed matrix. y = At * x
+   /** If the matrix is modified, call ResetTranspose() and optionally
+       EnsureMultTranspose() to make sure this method uses the correct updated
+       transpose. */
    void MultTranspose(const Vector &x, Vector &y) const;
 
    /// y += At * x (default)  or  y += a * At * x
+   /** If the matrix is modified, call ResetTranspose() and optionally
+       EnsureMultTranspose() to make sure this method uses the correct updated
+       transpose. */
    void AddMultTranspose(const Vector &x, Vector &y,
                          const double a = 1.0) const;
 
@@ -362,16 +368,16 @@ public:
        MultTranspose(), and AbsMultTranspose().
 
        Warning: any changes in this matrix will invalidate the internal
-       transpose. To rebuild the transpose, call ResetTranspose() followed by a
-       call to this method. If the internal transpose is already built, this
-       method has no effect.
+       transpose. To rebuild the transpose, call ResetTranspose() followed by
+       (optionally) a call to this method. If the internal transpose is already
+       built, this method has no effect.
 
        When any non-serial-CPU backend is enabled, i.e. the call
        Device::Allows(~ Backend::CPU_MASK) returns true, the above methods
        require the internal transpose to be built. If that is not the case (i.e.
-       the internal transpose is not built), these methods will raise an error
-       with an appropriate message pointing to EnsureMultTranspose(). When using
-       any backend from Backend::CPU_MASK, calling this method is optional.
+       the internal transpose is not built), these methods will automatically
+       call EnsureMultTranspose(). When using any backend from
+       Backend::CPU_MASK, calling this method is optional.
 
        This method can only be used when the sparse matrix is finalized.
 
@@ -414,6 +420,9 @@ public:
    void AbsMult(const Vector &x, Vector &y) const;
 
    /// y = |At| * x, using entry-wise absolute values of the transpose of matrix A
+   /** If the matrix is modified, call ResetTranspose() and optionally
+       EnsureMultTranspose() to make sure this method uses the correct updated
+       transpose. */
    void AbsMultTranspose(const Vector &x, Vector &y) const;
 
    /// Compute y^t A x

linalg/sundials.cpp

@@ -22,7 +22,6 @@
 #include <nvector/nvector_serial.h>
 #ifdef MFEM_USE_CUDA
 #include <nvector/nvector_cuda.h>
-#include <sunmemory/sunmemory_cuda.h>
 #endif
 #ifdef MFEM_USE_MPI
 #include <nvector/nvector_mpiplusx.h>
@@ -38,111 +37,273 @@
 
 using namespace std;
 
+#if (SUNDIALS_VERSION_MAJOR < 6)
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED N_Vector N_VNewEmpty_Serial(sunindextype vec_length, SUNContext)
+{
+   return N_VNewEmpty_Serial(vec_length);
+}
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED SUNMatrix SUNMatNewEmpty(SUNContext)
+{
+   return SUNMatNewEmpty();
+}
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED SUNLinearSolver SUNLinSolNewEmpty(SUNContext)
+{
+   return SUNLinSolNewEmpty();
+}
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED SUNLinearSolver SUNLinSol_SPGMR(N_Vector y, int pretype,
+                                                int maxl, SUNContext)
+{
+   return SUNLinSol_SPGMR(y, pretype, maxl);
+}
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED SUNLinearSolver SUNLinSol_SPFGMR(N_Vector y, int pretype,
+                                                 int maxl, SUNContext)
+{
+   return SUNLinSol_SPFGMR(y, pretype, maxl);
+}
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED void* CVodeCreate(int lmm, SUNContext)
+{
+   return CVodeCreate(lmm);
+}
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED void* ARKStepCreate(ARKRhsFn fe, ARKRhsFn fi, realtype t0,
+                                    N_Vector y0, SUNContext)
+{
+   return ARKStepCreate(fe, fi, t0, y0);
+}
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED void* KINCreate(SUNContext)
+{
+   return KINCreate();
+}
+
+#ifdef MFEM_USE_MPI
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED N_Vector N_VNewEmpty_Parallel(MPI_Comm comm,
+                                              sunindextype local_length,
+                                              sunindextype global_length,
+                                              SUNContext)
+{
+   return N_VNewEmpty_Parallel(comm, local_length, global_length);
+}
+
+#endif // MFEM_USE_MPI
+
+#ifdef MFEM_USE_CUDA
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED N_Vector N_VNewWithMemHelp_Cuda(sunindextype length,
+                                                booleantype use_managed_mem,
+                                                SUNMemoryHelper helper,
+                                                SUNContext)
+{
+   return N_VNewWithMemHelp_Cuda(length, use_managed_mem, helper);
+}
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED SUNMemoryHelper SUNMemoryHelper_NewEmpty(SUNContext)
+{
+   return SUNMemoryHelper_NewEmpty();
+}
+
+#endif // MFEM_USE_CUDA
+
+#if defined(MFEM_USE_MPI) && defined(MFEM_USE_CUDA)
+
+/// (DEPRECATED) Wrapper function for backwards compatibility with SUNDIALS
+/// version < 6
+MFEM_DEPRECATED N_Vector N_VMake_MPIPlusX(MPI_Comm comm, N_Vector local_vector,
+                                          SUNContext)
+{
+   return N_VMake_MPIPlusX(comm, local_vector);
+}
+
+#endif // MFEM_USE_MPI && MFEM_USE_CUDA
+
+#endif // SUNDIALS_VERSION_MAJOR < 6
+
+
 namespace mfem
 {
 
-// ---------------------------------------------------------------------------
-// SUNMemory interface class (private)
-// ---------------------------------------------------------------------------
-
-#ifdef MFEM_USE_CUDA
-class SundialsMemHelper
+void Sundials::Init()
 {
-protected:
-   /// The actual SUNDIALS object
-   SUNMemoryHelper h;
+   Sundials::Instance();
+}
 
-   friend class SundialsNVector;
+Sundials &Sundials::Instance()
+{
+   static Sundials sundials;
+   return sundials;
+}
 
-public:
-   SundialsMemHelper()
-   {
-      /* Allocate helper */
-      h = SUNMemoryHelper_NewEmpty();
+SUNContext &Sundials::GetContext()
+{
+   return Sundials::Instance().context;
+}
 
-      /* Set the ops */
-      h->ops->alloc     = SundialsMemHelper_Alloc;
-      h->ops->dealloc   = SundialsMemHelper_Dealloc;
-#ifdef MFEM_USE_CUDA
-      h->ops->copy      = SUNMemoryHelper_Copy_Cuda;
-      h->ops->copyasync = SUNMemoryHelper_CopyAsync_Cuda;
+SundialsMemHelper &Sundials::GetMemHelper()
+{
+   return Sundials::Instance().memHelper;
+}
+
+#if (SUNDIALS_VERSION_MAJOR >= 6)
+
+Sundials::Sundials()
+{
+#ifdef MFEM_USE_MPI
+   MPI_Comm communicator = MPI_COMM_WORLD;
+   int return_val = SUNContext_Create((void*) &communicator, &context);
+#else
+   int return_val = SUNContext_Create(nullptr, &context);
 #endif
+   MFEM_VERIFY(return_val == 0, "Call to SUNContext_Create failed");
+   SundialsMemHelper actual_helper(context);
+   memHelper = std::move(actual_helper);
+}
+
+Sundials::~Sundials()
+{
+   SUNContext_Free(&context);
+}
+
+#else // SUNDIALS_VERSION_MAJOR >= 6
+
+Sundials::Sundials()
+{
+   // Do nothing
+}
+
+Sundials::~Sundials()
+{
+   // Do nothing
+}
+
+#endif // SUNDIALS_VERSION_MAJOR >= 6
+
+#ifdef MFEM_USE_CUDA
+SundialsMemHelper::SundialsMemHelper(SUNContext context)
+{
+   /* Allocate helper */
+   h = SUNMemoryHelper_NewEmpty(context);
+
+   /* Set the ops */
+   h->ops->alloc     = SundialsMemHelper_Alloc;
+   h->ops->dealloc   = SundialsMemHelper_Dealloc;
+   h->ops->copy      = SUNMemoryHelper_Copy_Cuda;
+   h->ops->copyasync = SUNMemoryHelper_CopyAsync_Cuda;
+}
+
+SundialsMemHelper::SundialsMemHelper(SundialsMemHelper&& that_helper)
+{
+   this->h = that_helper.h;
+   that_helper.h = nullptr;
+}
+
+SundialsMemHelper& SundialsMemHelper::operator=(SundialsMemHelper&& rhs)
+{
+   this->h = rhs.h;
+   rhs.h = nullptr;
+   return *this;
+}
+
+int SundialsMemHelper::SundialsMemHelper_Alloc(SUNMemoryHelper helper,
+                                               SUNMemory* memptr, size_t memsize,
+                                               SUNMemoryType mem_type
+#if (SUNDIALS_VERSION_MAJOR >= 6)
+                                               , void*
+#endif
+                                              )
+{
+   int length = memsize/sizeof(double);
+   SUNMemory sunmem = SUNMemoryNewEmpty();
+
+   sunmem->ptr = NULL;
+   sunmem->own = SUNTRUE;
+
+   if (mem_type == SUNMEMTYPE_HOST)
+   {
+      Memory<double> mem(length, Device::GetHostMemoryType());
+      mem.SetHostPtrOwner(false);
+      sunmem->ptr  = mfem::HostReadWrite(mem, length);
+      sunmem->type = SUNMEMTYPE_HOST;
+      mem.Delete();
+   }
+   else if (mem_type == SUNMEMTYPE_DEVICE || mem_type == SUNMEMTYPE_UVM)
+   {
+      Memory<double> mem(length, Device::GetDeviceMemoryType());
+      mem.SetDevicePtrOwner(false);
+      sunmem->ptr  = mfem::ReadWrite(mem, length);
+      sunmem->type = mem_type;
+      mem.Delete();
+   }
+   else
+   {
+      free(sunmem);
+      return -1;
    }
 
-   ~SundialsMemHelper()
+   *memptr = sunmem;
+   return 0;
+}
+
+int SundialsMemHelper::SundialsMemHelper_Dealloc(SUNMemoryHelper helper,
+                                                 SUNMemory sunmem
+#if (SUNDIALS_VERSION_MAJOR >= 6)
+                                                 , void*
+#endif
+                                                )
+{
+   if (sunmem->ptr && sunmem->own && !mm.IsKnown(sunmem->ptr))
    {
-      SUNMemoryHelper_Destroy(h);
-   }
-
-   /// Typecasting to SUNDIALS' SUNMemoryHelper type
-   operator SUNMemoryHelper() const { return h; }
-
-   static int SundialsMemHelper_Alloc(SUNMemoryHelper helper,
-                                      SUNMemory* memptr,
-                                      size_t memsize,
-                                      SUNMemoryType mem_type)
-   {
-      int length = memsize/sizeof(double);
-      SUNMemory sunmem = SUNMemoryNewEmpty();
-
-      sunmem->ptr = NULL;
-      sunmem->own = SUNTRUE;
-
-      if (mem_type == SUNMEMTYPE_HOST)
+      if (sunmem->type == SUNMEMTYPE_HOST)
       {
-         Memory<double> mem(length, Device::GetHostMemoryType());
-         mem.SetHostPtrOwner(false);
-         sunmem->ptr  = mfem::HostReadWrite(mem, length);
-         sunmem->type = SUNMEMTYPE_HOST;
+         Memory<double> mem(static_cast<double*>(sunmem->ptr), 1,
+                            Device::GetHostMemoryType(), true);
          mem.Delete();
       }
-      else if (mem_type == SUNMEMTYPE_DEVICE || mem_type == SUNMEMTYPE_UVM)
+      else if (sunmem->type == SUNMEMTYPE_DEVICE || sunmem->type == SUNMEMTYPE_UVM)
       {
-         Memory<double> mem(length, Device::GetDeviceMemoryType());
-         mem.SetDevicePtrOwner(false);
-         sunmem->ptr  = mfem::ReadWrite(mem, length);
-         sunmem->type = mem_type;
+         Memory<double> mem(static_cast<double*>(sunmem->ptr), 1,
+                            Device::GetDeviceMemoryType(), true);
          mem.Delete();
       }
       else
       {
-         free(sunmem);
+         MFEM_ABORT("Invalid SUNMEMTYPE");
          return -1;
       }
-
-      *memptr = sunmem;
-      return 0;
    }
+   free(sunmem);
+   return 0;
+}
 
-   static int SundialsMemHelper_Dealloc(SUNMemoryHelper helper, SUNMemory sunmem)
-   {
-      if (sunmem->ptr && sunmem->own && !mm.IsKnown(sunmem->ptr))
-      {
-         if (sunmem->type == SUNMEMTYPE_HOST)
-         {
-            Memory<double> mem(static_cast<double*>(sunmem->ptr), 1,
-                               Device::GetHostMemoryType(), true);
-            mem.Delete();
-         }
-         else if (sunmem->type == SUNMEMTYPE_DEVICE || sunmem->type == SUNMEMTYPE_UVM)
-         {
-            Memory<double> mem(static_cast<double*>(sunmem->ptr), 1,
-                               Device::GetDeviceMemoryType(), true);
-            mem.Delete();
-         }
-         else
-         {
-            MFEM_ABORT("Invalid SUNMEMTYPE");
-            return -1;
-         }
-      }
-      free(sunmem);
-      return 0;
-   }
-};
-
-SundialsMemHelper sunmemHelper;
-#endif
+#endif // MFEM_USE_CUDA
 
 
 // ---------------------------------------------------------------------------
@@ -367,14 +528,15 @@ N_Vector SundialsNVector::MakeNVector(bool use_device)
 #ifdef MFEM_USE_CUDA
    if (use_device)
    {
-      x = N_VNewWithMemHelp_Cuda(0, UseManagedMemory(), sunmemHelper);
+      x = N_VNewWithMemHelp_Cuda(0, UseManagedMemory(), Sundials::GetMemHelper(),
+                                 Sundials::GetContext());
    }
    else
    {
-      x = N_VNewEmpty_Serial(0);
+      x = N_VNewEmpty_Serial(0, Sundials::GetContext());
    }
 #else
-   x = N_VNewEmpty_Serial(0);
+   x = N_VNewEmpty_Serial(0, Sundials::GetContext());
 #endif
 
    MFEM_VERIFY(x, "Error in SundialsNVector::MakeNVector.");
@@ -397,14 +559,16 @@ N_Vector SundialsNVector::MakeNVector(MPI_Comm comm, bool use_device)
       if (use_device)
       {
          x = N_VMake_MPIPlusX(comm, N_VNewWithMemHelp_Cuda(0, UseManagedMemory(),
-                                                           sunmemHelper));
+                                                           Sundials::GetMemHelper(),
+                                                           Sundials::GetContext()),
+                              Sundials::GetContext());
       }
       else
       {
-         x = N_VNewEmpty_Parallel(comm, 0, 0);
+         x = N_VNewEmpty_Parallel(comm, 0, 0, Sundials::GetContext());
       }
 #else
-      x = N_VNewEmpty_Parallel(comm, 0, 0);
+      x = N_VNewEmpty_Parallel(comm, 0, 0, Sundials::GetContext());
 #endif // MFEM_USE_CUDA
    }
 
@@ -594,7 +758,7 @@ void CVODESolver::Init(TimeDependentOperator &f_)
 #endif
 
       // Create CVODE
-      sundials_mem = CVodeCreate(lmm_type);
+      sundials_mem = CVodeCreate(lmm_type, Sundials::GetContext());
       MFEM_VERIFY(sundials_mem, "error in CVodeCreate()");
 
       // Initialize CVODE
@@ -651,7 +815,7 @@ void CVODESolver::UseMFEMLinearSolver()
    if (LSA != NULL) { SUNLinSolFree(LSA); LSA = NULL; }
 
    // Wrap linear solver as SUNLinearSolver and SUNMatrix
-   LSA = SUNLinSolNewEmpty();
+   LSA = SUNLinSolNewEmpty(Sundials::GetContext());
    MFEM_VERIFY(LSA, "error in SUNLinSolNewEmpty()");
 
    LSA->content      = this;
@@ -659,7 +823,7 @@ void CVODESolver::UseMFEMLinearSolver()
    LSA->ops->solve   = CVODESolver::LinSysSolve;
    LSA->ops->free    = LSFree;
 
-   A = SUNMatNewEmpty();
+   A = SUNMatNewEmpty(Sundials::GetContext());
    MFEM_VERIFY(A, "error in SUNMatNewEmpty()");
 
    A->content      = this;
@@ -682,7 +846,7 @@ void CVODESolver::UseSundialsLinearSolver()
    if (LSA != NULL) { SUNLinSolFree(LSA); LSA = NULL; }
 
    // Create linear solver
-   LSA = SUNLinSol_SPGMR(*Y, PREC_NONE, 0);
+   LSA = SUNLinSol_SPGMR(*Y, PREC_NONE, 0, Sundials::GetContext());
    MFEM_VERIFY(LSA, "error in SUNLinSol_SPGMR()");
 
    // Attach linear solver
@@ -941,7 +1105,7 @@ void CVODESSolver::UseMFEMLinearSolverB()
    if (LSB != NULL) { SUNLinSolFree(LSB); LSB = NULL; }
 
    // Wrap linear solver as SUNLinearSolver and SUNMatrix
-   LSB = SUNLinSolNewEmpty();
+   LSB = SUNLinSolNewEmpty(Sundials::GetContext());
    MFEM_VERIFY(LSB, "error in SUNLinSolNewEmpty()");
 
    LSB->content         = this;
@@ -949,7 +1113,7 @@ void CVODESSolver::UseMFEMLinearSolverB()
    LSB->ops->solve      = CVODESSolver::LinSysSolveB; // JW change
    LSB->ops->free       = LSFree;
 
-   AB = SUNMatNewEmpty();
+   AB = SUNMatNewEmpty(Sundials::GetContext());
    MFEM_VERIFY(AB, "error in SUNMatNewEmpty()");
 
    AB->content      = this;
@@ -973,7 +1137,7 @@ void CVODESSolver::UseSundialsLinearSolverB()
    if (LSB != NULL) { SUNLinSolFree(LSB); LSB = NULL; }
 
    // Set default linear solver (Newton is the default Nonlinear Solver)
-   LSB = SUNLinSol_SPGMR(*yB, PREC_NONE, 0);
+   LSB = SUNLinSol_SPGMR(*yB, PREC_NONE, 0, Sundials::GetContext());
    MFEM_VERIFY(LSB, "error in SUNLinSol_SPGMR()");
 
    /* Attach the matrix and linear solver */
@@ -1359,16 +1523,18 @@ void ARKStepSolver::Init(TimeDependentOperator &f_)
       // Create ARKStep memory
       if (rk_type == IMPLICIT)
       {
-         sundials_mem = ARKStepCreate(NULL, ARKStepSolver::RHS1, t, *Y);
+         sundials_mem = ARKStepCreate(NULL, ARKStepSolver::RHS1, t, *Y,
+                                      Sundials::GetContext());
       }
       else if (rk_type == EXPLICIT)
       {
-         sundials_mem = ARKStepCreate(ARKStepSolver::RHS1, NULL, t, *Y);
+         sundials_mem = ARKStepCreate(ARKStepSolver::RHS1, NULL, t, *Y,
+                                      Sundials::GetContext());
       }
       else
       {
          sundials_mem = ARKStepCreate(ARKStepSolver::RHS1, ARKStepSolver::RHS2,
-                                      t, *Y);
+                                      t, *Y, Sundials::GetContext());
       }
       MFEM_VERIFY(sundials_mem, "error in ARKStepCreate()");
 
@@ -1435,7 +1601,7 @@ void ARKStepSolver::UseMFEMLinearSolver()
    if (LSA != NULL) { SUNLinSolFree(LSA); LSA = NULL; }
 
    // Wrap linear solver as SUNLinearSolver and SUNMatrix
-   LSA = SUNLinSolNewEmpty();
+   LSA = SUNLinSolNewEmpty(Sundials::GetContext());
    MFEM_VERIFY(LSA, "error in SUNLinSolNewEmpty()");
 
    LSA->content      = this;
@@ -1443,7 +1609,7 @@ void ARKStepSolver::UseMFEMLinearSolver()
    LSA->ops->solve   = ARKStepSolver::LinSysSolve;
    LSA->ops->free    = LSFree;
 
-   A = SUNMatNewEmpty();
+   A = SUNMatNewEmpty(Sundials::GetContext());
    MFEM_VERIFY(A, "error in SUNMatNewEmpty()");
 
    A->content      = this;
@@ -1466,7 +1632,7 @@ void ARKStepSolver::UseSundialsLinearSolver()
    if (LSA != NULL) { SUNLinSolFree(LSA); LSA = NULL; }
 
    // Create linear solver
-   LSA = SUNLinSol_SPGMR(*Y, PREC_NONE, 0);
+   LSA = SUNLinSol_SPGMR(*Y, PREC_NONE, 0, Sundials::GetContext());
    MFEM_VERIFY(LSA, "error in SUNLinSol_SPGMR()");
 
    // Attach linear solver
@@ -1481,7 +1647,7 @@ void ARKStepSolver::UseMFEMMassLinearSolver(int tdep)
    if (LSM != NULL) { SUNLinSolFree(LSM); LSM = NULL; }
 
    // Wrap linear solver as SUNLinearSolver and SUNMatrix
-   LSM = SUNLinSolNewEmpty();
+   LSM = SUNLinSolNewEmpty(Sundials::GetContext());
    MFEM_VERIFY(LSM, "error in SUNLinSolNewEmpty()");
 
    LSM->content      = this;
@@ -1489,7 +1655,7 @@ void ARKStepSolver::UseMFEMMassLinearSolver(int tdep)
    LSM->ops->solve   = ARKStepSolver::MassSysSolve;
    LSA->ops->free    = LSFree;
 
-   M = SUNMatNewEmpty();
+   M = SUNMatNewEmpty(Sundials::GetContext());
    MFEM_VERIFY(M, "error in SUNMatNewEmpty()");
 
    M->content      = this;
@@ -1513,7 +1679,7 @@ void ARKStepSolver::UseSundialsMassLinearSolver(int tdep)
    if (LSM != NULL) { SUNLinSolFree(LSM); LSM = NULL; }
 
    // Create linear solver
-   LSM = SUNLinSol_SPGMR(*Y, PREC_NONE, 0);
+   LSM = SUNLinSol_SPGMR(*Y, PREC_NONE, 0, Sundials::GetContext());
    MFEM_VERIFY(LSM, "error in SUNLinSol_SPGMR()");
 
    // Attach linear solver
@@ -1549,21 +1715,22 @@ void ARKStepSolver::SetOrder(int order)
    MFEM_VERIFY(flag == ARK_SUCCESS, "error in ARKStepSetOrder()");
 }
 
-void ARKStepSolver::SetERKTableNum(int table_num)
+void ARKStepSolver::SetERKTableNum(ARKODE_ERKTableID table_id)
 {
-   flag = ARKStepSetTableNum(sundials_mem, -1, table_num);
+   flag = ARKStepSetTableNum(sundials_mem, ARKODE_DIRK_NONE, table_id);
    MFEM_VERIFY(flag == ARK_SUCCESS, "error in ARKStepSetTableNum()");
 }
 
-void ARKStepSolver::SetIRKTableNum(int table_num)
+void ARKStepSolver::SetIRKTableNum(ARKODE_DIRKTableID table_id)
 {
-   flag = ARKStepSetTableNum(sundials_mem, table_num, -1);
+   flag = ARKStepSetTableNum(sundials_mem, table_id, ARKODE_ERK_NONE);
    MFEM_VERIFY(flag == ARK_SUCCESS, "error in ARKStepSetTableNum()");
 }
 
-void ARKStepSolver::SetIMEXTableNum(int etable_num, int itable_num)
+void ARKStepSolver::SetIMEXTableNum(ARKODE_ERKTableID etable_id,
+                                    ARKODE_DIRKTableID itable_id)
 {
-   flag = ARKStepSetTableNum(sundials_mem, itable_num, etable_num);
+   flag = ARKStepSetTableNum(sundials_mem, itable_id, etable_id);
    MFEM_VERIFY(flag == ARK_SUCCESS, "error in ARKStepSetTableNum()");
 }
 
@@ -1836,7 +2003,7 @@ void KINSolver::SetOperator(const Operator &op)
 #endif
 
       // Create the solver memory
-      sundials_mem = KINCreate();
+      sundials_mem = KINCreate(Sundials::GetContext());
       MFEM_VERIFY(sundials_mem, "Error in KINCreate().");
 
       // Set number of acceleration vectors
@@ -1865,7 +2032,7 @@ void KINSolver::SetOperator(const Operator &op)
          if (A != NULL) { SUNMatDestroy(A); A = NULL; }
          if (LSA != NULL) { SUNLinSolFree(LSA); LSA = NULL; }
 
-         LSA = SUNLinSol_SPGMR(*Y, PREC_NONE, 0);
+         LSA = SUNLinSol_SPGMR(*Y, PREC_NONE, 0, Sundials::GetContext());
          MFEM_VERIFY(LSA, "error in SUNLinSol_SPGMR()");
 
          flag = KINSetLinearSolver(sundials_mem, LSA, NULL);
@@ -1897,7 +2064,7 @@ void KINSolver::SetSolver(Solver &solver)
       if (LSA != NULL) { SUNLinSolFree(LSA); LSA = NULL; }
 
       // Wrap KINSolver as SUNLinearSolver and SUNMatrix
-      LSA = SUNLinSolNewEmpty();
+      LSA = SUNLinSolNewEmpty(Sundials::GetContext());
       MFEM_VERIFY(LSA, "error in SUNLinSolNewEmpty()");
 
       LSA->content      = this;
@@ -1905,7 +2072,7 @@ void KINSolver::SetSolver(Solver &solver)
       LSA->ops->solve   = KINSolver::LinSysSolve;
       LSA->ops->free    = LSFree;
 
-      A = SUNMatNewEmpty();
+      A = SUNMatNewEmpty(Sundials::GetContext());
       MFEM_VERIFY(A, "error in SUNMatNewEmpty()");
 
       A->content      = this;
@@ -1934,7 +2101,8 @@ void KINSolver::SetJFNKSolver(Solver &solver)
    if (LSA != NULL) { SUNLinSolFree(LSA); LSA = NULL; }
 
    // Setup FGMRES
-   LSA = SUNLinSol_SPFGMR(*Y, prec ? PREC_RIGHT : PREC_NONE, maxli);
+   LSA = SUNLinSol_SPFGMR(*Y, prec ? PREC_RIGHT : PREC_NONE, maxli,
+                          Sundials::GetContext());
    MFEM_VERIFY(LSA, "error in SUNLinSol_SPFGMR()");
 
    flag = SUNLinSol_SPFGMRSetMaxRestarts(LSA, maxlrs);

linalg/sundials.hpp

@@ -37,15 +37,137 @@
 #include <arkode/arkode_arkstep.h>
 #include <cvodes/cvodes.h>
 #include <kinsol/kinsol.h>
+#ifdef MFEM_USE_CUDA
+#include <sunmemory/sunmemory_cuda.h>
+#endif
 
 #include <functional>
 
+#if (SUNDIALS_VERSION_MAJOR < 6)
+
+/// (DEPRECATED) Map SUNDIALS version >= 6 datatypes and constants to
+/// version < 6 for backwards compatibility
+using ARKODE_ERKTableID = int;
+using ARKODE_DIRKTableID = int;
+constexpr ARKODE_ERKTableID ARKODE_ERK_NONE = -1;
+constexpr ARKODE_DIRKTableID ARKODE_DIRK_NONE = -1;
+constexpr ARKODE_ERKTableID ARKODE_FEHLBERG_13_7_8 = FEHLBERG_13_7_8;
+
+/// (DEPRECATED) There is no SUNContext in SUNDIALS version < 6 so set it to
+/// arbitrary type for more compact backwards compatibility
+using SUNContext = void*;
+
+#endif // SUNDIALS_VERSION_MAJOR < 6
+
 namespace mfem
 {
 
+#ifdef MFEM_USE_CUDA
+
 // ---------------------------------------------------------------------------
-// Base class for interfacing with SUNDIALS packages
+// SUNMemory interface class (used when CUDA is enabled)
 // ---------------------------------------------------------------------------
+class SundialsMemHelper
+{
+   /// The actual SUNDIALS object
+   SUNMemoryHelper h;
+
+public:
+
+   /// Default constructor -- object must be moved to
+   SundialsMemHelper() = default;
+
+   /// Require a SUNContext as an argument (rather than calling Sundials::GetContext)
+   /// to avoid undefined behavior during the construction of the Sundials singleton.
+   SundialsMemHelper(SUNContext context);
+
+   /// Implement move assignment
+   SundialsMemHelper(SundialsMemHelper&& that_helper);
+
+   /// Disable copy construction
+   SundialsMemHelper(const SundialsMemHelper& that_helper) = delete;
+
+   ~SundialsMemHelper() { if (h) { SUNMemoryHelper_Destroy(h); } }
+
+   /// Disable copy assignment
+   SundialsMemHelper& operator=(const SundialsMemHelper&) = delete;
+
+   /// Implement move assignment
+   SundialsMemHelper& operator=(SundialsMemHelper&& rhs);
+
+   /// Typecasting to SUNDIALS' SUNMemoryHelper type
+   operator SUNMemoryHelper() const { return h; }
+
+   static int SundialsMemHelper_Alloc(SUNMemoryHelper helper, SUNMemory* memptr,
+                                      size_t memsize, SUNMemoryType mem_type
+#if (SUNDIALS_VERSION_MAJOR >= 6)
+                                      , void* queue
+#endif
+                                     );
+
+   static int SundialsMemHelper_Dealloc(SUNMemoryHelper helper, SUNMemory sunmem
+#if (SUNDIALS_VERSION_MAJOR >= 6)
+                                        , void* queue
+#endif
+                                       );
+
+};
+
+#else // MFEM_USE_CUDA
+
+// ---------------------------------------------------------------------------
+// Dummy SUNMemory interface class (used when CUDA is not enabled)
+// ---------------------------------------------------------------------------
+class SundialsMemHelper
+{
+public:
+
+   SundialsMemHelper() = default;
+
+   SundialsMemHelper(SUNContext context)
+   {
+      // Do nothing
+   }
+};
+
+#endif // MFEM_USE_CUDA
+
+
+/// Singleton class for SUNContext and SundialsMemHelper objects
+class Sundials
+{
+public:
+
+   /// Disable copy construction
+   Sundials(Sundials &other) = delete;
+
+   /// Disable copy assignment
+   void operator=(const Sundials &other) = delete;
+
+   /// Initializes SUNContext and SundialsMemHelper objects. Should be called at
+   /// the beginning of the calling program (after Mpi::Init if applicable)
+   static void Init();
+
+   /// Provides access to the SUNContext object
+   static SUNContext &GetContext();
+
+   /// Provides access to the SundialsMemHelper object
+   static SundialsMemHelper &GetMemHelper();
+
+private:
+   /// Returns a reference to the singleton instance of the class.
+   static Sundials &Instance();
+
+   /// Constructor called by Sundials::Instance (does nothing for version < 6)
+   Sundials();
+
+   /// Destructor called at end of calling program (does nothing for version < 6)
+   ~Sundials();
+
+   SUNContext context;
+   SundialsMemHelper memHelper;
+};
+
 
 /// Vector interface for SUNDIALS N_Vectors.
 class SundialsNVector : public Vector
@@ -676,19 +798,19 @@ public:
 
    /// Choose a specific Butcher table for an explicit RK method.
    /** See ARKODE documentation for all possible options, stability regions, etc.
-       For example, table_num = BOGACKI_SHAMPINE_4_2_3 is 4-stage 3rd order. */
-   void SetERKTableNum(int table_num);
+       For example, table_id = BOGACKI_SHAMPINE_4_2_3 is 4-stage 3rd order. */
+   void SetERKTableNum(ARKODE_ERKTableID table_id);
 
    /// Choose a specific Butcher table for a diagonally implicit RK method.
    /** See ARKODE documentation for all possible options, stability regions, etc.
-       For example, table_num = CASH_5_3_4 is 5-stage 4th order. */
-   void SetIRKTableNum(int table_num);
+       For example, table_id = CASH_5_3_4 is 5-stage 4th order. */
+   void SetIRKTableNum(ARKODE_DIRKTableID table_id);
 
    /// Choose a specific Butcher table for an IMEX RK method.
    /** See ARKODE documentation for all possible options, stability regions, etc.
-       For example, etable_num = ARK548L2SA_DIRK_8_4_5 and
-       itable_num = ARK548L2SA_ERK_8_4_5 is 8-stage 5th order. */
-   void SetIMEXTableNum(int etable_num, int itable_num);
+       For example, etable_id = ARK548L2SA_DIRK_8_4_5 and
+       itable_id = ARK548L2SA_ERK_8_4_5 is 8-stage 5th order. */
+   void SetIMEXTableNum(ARKODE_ERKTableID etable_id, ARKODE_DIRKTableID itable_id);
 
    /// Use a fixed time step size (disable temporal adaptivity).
    /** Use of this function is not recommended, since there is no assurance of