Some comments to make it clear what some newer parts are doing.

Change number of CG iterations in HypreSmoother [hypre-smoother-cg-iter-dev]

.travis.yml

@@ -11,8 +11,6 @@
 
 language: cpp
 
-sudo: false
-
 stages:
   - checks
   - tests
@@ -370,8 +368,10 @@ script:
    # Compiler
    - if [ $MPI == "YES" ]; then
         export MYCXX=mpic++;
+        export MAKE_CXX_FLAG=MPICXX=$MYCXX;
      else
         export MYCXX="$CXX";
+        export MAKE_CXX_FLAG=CXX=$MYCXX;
      fi
 
    # Print the compiler version
@@ -384,12 +384,9 @@ script:
      if [ "$CODECOV" == "YES" ]; then
         CPPFLAGS="--coverage -g";
      fi;
-     if [ "$CXX" == "clang++" ]; then
-        export MFEM_PERF_SW=clang;
-     fi
 
    # Configure the library
-   - make config MFEM_USE_MPI=$MPI MFEM_DEBUG=$DEBUG MFEM_CXX="$MYCXX"
+   - make config MFEM_USE_MPI=$MPI MFEM_DEBUG=$DEBUG $MAKE_CXX_FLAG
         MFEM_MPI_NP=$NPROCS CPPFLAGS="$CPPFLAGS"
    # Show the configuration
    - make info

CHANGELOG

@@ -23,8 +23,21 @@ Meshing improvements
   Hessian for r-adaptivity using discrete fields, and allows use of skewness
   and orientation based metrics.
 
-Improved GPU capabilities
--------------------------
+- Added support for r-adaptivity with more than one discrete field. This allows
+  the user to specify different discrete functions for controlling the 
+  size, aspect-ratio, orientation, and skew of elements in the mesh. 
+
+Performance improvements
+------------------------
+- Added support for explicit vectorization in the high-performance templated
+  code, which can now take advantage of specific intrinsics classes on the
+  following architectures:
+    - x86 (SSE/AVX/AVX2/AVX512),
+    - Power8 & Power9 (VSX),
+    - BG/Q (QPX).
+  These are now enabled by default, and can be disabled with MFEM_USE_SIMD=NO.
+  See the new file linalg/simd.hpp and the new directory linalg/simd.
+
 - Added support for Chebyshev accelerated polynomial smoother on GPU.
 
 Discretization improvements
@@ -35,6 +48,24 @@ Discretization improvements
 
 - Added support for simplices in GSLIB-FindPoints.
 
+- Added support for H1 and L2 element matrix assembly in the mass, convection,
+  diffusion, transpose, and the face DG trace integrators. This is compatible
+  with GPU device execution and is illustrated in Example 9/9p, see the option
+  '-ea'. When enabled, this level of assembly stores independent dense matrices
+  for the elements, and independent dense matrices for the faces in the DG case.
+
+- Added new partial assembly kernels for H(div) bilinear forms, as well as
+  VectorFEDivergenceIntegrator.
+
+- Improved the documentation of the GridFunction GetValue and GetVectorValue
+  methods. Expanded the GetValue and GetVectorValue methods which accept an
+  ElementTransformation argument to support evaluation on boundary elements
+  and, in the continuous field case, arbitrary mesh edges and faces.
+
+- Added new coefficient and vector coefficient classes for QuadratureFunctions.
+  Additionaly, new LinearForm integrators were also added which make use of
+  these new QuadratureFunction coefficient classes.
+
 Linear and nonlinear solvers
 ----------------------------
 - Added power method to iteratively estimate the largest eigenvalue and the
@@ -43,6 +74,17 @@ Linear and nonlinear solvers
 - Added initial support for h- and p-multigrid solvers and preconditioners for
   matrix-based and matrix-free discretizations with basic GPU capability.
 
+- Added a new IterativeSolverMonitor class that allows to monitor the residual
+  and solution during the solving process of an IterativeSolver after every
+  iteration.
+
+- Block arrays of parallel matrices can now be merged into a single parallel
+  matrix with the function HypreParMatrixFromBlocks. This could be useful for
+  solving block systems with parallel direct solvers such as STRUMPACK.
+
+- In SLISolver, changed the residual inner product from (Br,r) to (Br,Br) so the
+  solver can work with non-SPD preconditioner B.
+
 New and updated examples and miniapps
 -------------------------------------
 - Added a new example, Example 25/25p, to demonstrate the use of a Perfectly
@@ -65,6 +107,12 @@ New and updated examples and miniapps
 - Added a new meshing miniapp, Minimal Surface, which solves Plateau's problem:
   the Dirichlet problem for the minimal surface equation.
 
+- Added partial assembly support to examples 4/4p and 5/5p, with diagonal
+  preconditioning.
+
+- Added a new test problem in example 24/24p, demonstrating a mixed bilinear
+  form for H(div) and L_2, with partial assembly support.
+
 Improved testing
 ----------------
 - Added a GitLab pipeline that automates PR testing on supercomputing systems
@@ -74,15 +122,17 @@ Improved testing
 
 Miscellaneous
 -------------
-- In SLISolver, changed the residual inner product from (Br,r) to (Br,Br) so the
-  solver can work with non-SPD preconditioner B.
-
 - Added support for ADIOS2 for parallel I/O with ParaView visualization. The
   classes adios2stream and ADIOS2DataCollection are introduced in mfem as the
   interfaces to generate ADIOS2 Binary Pack (BP4) directory datasets for the
   entire spatial and temporal data. In addition, ADIOS2 allows for setting a
   user-defined number of data substreams/subfiles. See examples 5, 9, 12, 16.
 
+- The integration order used in the ComputeLpError and ComputeElementLpError
+  methods of class GridFunction has been increased.
+
+- Various other simplifications, extensions, and bugfixes in the code.
+
 
 Version 4.1, released on March 10, 2020
 =======================================

INSTALL

@@ -396,6 +396,12 @@ MFEM_USE_SIDRE = YES/NO
    blueprint specification. When enabled, this option requires installation of
    HDF5 (see also MFEM_USE_NETCDF), Conduit and LLNL's axom project.
 
+MFEM_USE_SIMD = YES/NO
+   Enables the high performance templated classes to use architecture dependent
+   SIMD intrinsics instead of the generic implementation of class AutoSIMD in
+   linalg/simd/auto.hpp. This option should be combined with suitable
+   compiler options, such as -march=native, to enable optimal vectorization.
+
 MFEM_USE_CONDUIT = YES/NO
    Enables support for converting MFEM Mesh and Grid Function objects to and
    from Conduit Mesh Blueprint Descriptions (https://github.com/LLNL/conduit/)
@@ -426,6 +432,8 @@ MFEM_USE_PUMI = YES/NO
    data management system that is capable of handling general non-manifold
    models and effectively supports automated adaptive analysis. PUMI enables
    support for parallel unstructured mesh modifications in MFEM.
+   The develop branch of PUMI repository (https://github.com/SCOREC/core)
+   should be used for most updated features.
 
 MFEM_USE_UMPIRE = YES/NO
    Enables support for Umpire, a resource management library that allows the
@@ -609,8 +617,9 @@ The specific libraries and their options are:
 
 - PUMI (optional), used when MFEM_USE_PUMI = YES.
   URL: https://scorec.rpi.edu/pumi
+       https://github.com/SCOREC/core
   Options: PUMI_OPT, PUMI_LIB.
-  Versions: PUMI >= 2.2.0.
+  Versions: PUMI >= 2.2.3.
 
 - HiOp (optional), used when MFEM_USE_HIOP = YES.
   URL: https://github.com/LLNL/hiop

config/cmake/MFEMConfig.cmake.in

@@ -47,6 +47,7 @@ set(MFEM_USE_OCCA @MFEM_USE_OCCA@)
 set(MFEM_USE_RAJA @MFEM_USE_RAJA@)
 set(MFEM_USE_CEED @MFEM_USE_CEED@)
 set(MFEM_USE_UMPIRE @MFEM_USE_UMPIRE@)
+set(MFEM_USE_SIMD @MFEM_USE_SIMD@)
 set(MFEM_USE_ADIOS2 @MFEM_USE_ADIOS2@)
 
 set(MFEM_CXX_COMPILER "@CMAKE_CXX_COMPILER@")

config/cmake/config.hpp.in

@@ -107,6 +107,9 @@
 // Enable MFEM functionality based on the Sidre library
 #cmakedefine MFEM_USE_SIDRE
 
+// Enable the use of SIMD in the high performance templated classes
+#cmakedefine MFEM_USE_SIMD
+
 // Enable MFEM functionality based on Conduit
 #cmakedefine MFEM_USE_CONDUIT
 

config/cmake/modules/MfemCmakeUtilities.cmake

@@ -733,7 +733,7 @@ function(mfem_export_mk_files)
       MFEM_USE_SUPERLU MFEM_USE_STRUMPACK MFEM_USE_GNUTLS
       MFEM_USE_GSLIB MFEM_USE_NETCDF MFEM_USE_PETSC MFEM_USE_MPFR MFEM_USE_SIDRE
       MFEM_USE_CONDUIT MFEM_USE_PUMI MFEM_USE_CUDA MFEM_USE_OCCA MFEM_USE_RAJA
-      MFEM_USE_UMPIRE)
+      MFEM_USE_UMPIRE MFEM_USE_SIMD MFEM_USE_ADIOS2)
   foreach(var ${CONFIG_MK_BOOL_VARS})
     if (${var})
       set(${var} YES)
@@ -743,6 +743,7 @@ function(mfem_export_mk_files)
   endforeach()
   # TODO: Add support for MFEM_USE_CUDA=YES
   set(MFEM_CXX ${CMAKE_CXX_COMPILER})
+  set(MFEM_HOST_CXX ${MFEM_CXX})
   set(MFEM_CPPFLAGS "")
   string(STRIP "${CMAKE_CXX_FLAGS_${BUILD_TYPE}} ${CMAKE_CXX_FLAGS}"
          MFEM_CXXFLAGS)

config/config.hpp.in

@@ -106,6 +106,9 @@
 // Enable Sidre support
 // #define MFEM_USE_SIDRE
 
+// Enable the use of SIMD in the high performance templated classes
+// #define MFEM_USE_SIMD
+
 // Enable Conduit support
 // #define MFEM_USE_CONDUIT
 

config/config.mk.in

@@ -49,10 +49,12 @@ MFEM_USE_RAJA          = @MFEM_USE_RAJA@
 MFEM_USE_OCCA          = @MFEM_USE_OCCA@
 MFEM_USE_CEED          = @MFEM_USE_CEED@
 MFEM_USE_UMPIRE        = @MFEM_USE_UMPIRE@
+MFEM_USE_SIMD          = @MFEM_USE_SIMD@
 MFEM_USE_ADIOS2        = @MFEM_USE_ADIOS2@
 
 # Compiler, compile options, and link options
 MFEM_CXX       = @MFEM_CXX@
+MFEM_HOST_CXX  = @MFEM_HOST_CXX@
 MFEM_CPPFLAGS  = @MFEM_CPPFLAGS@
 MFEM_CXXFLAGS  = @MFEM_CXXFLAGS@
 MFEM_TPLFLAGS  = @MFEM_TPLFLAGS@

config/defaults.cmake

@@ -49,6 +49,7 @@ option(MFEM_USE_OCCA "Enable OCCA" OFF)
 option(MFEM_USE_RAJA "Enable RAJA" OFF)
 option(MFEM_USE_CEED "Enable CEED" OFF)
 option(MFEM_USE_UMPIRE "Enable Umpire" OFF)
+option(MFEM_USE_SIMD "Enable use of SIMD intrinsics" ON)
 option(MFEM_USE_ADIOS2 "Enable ADIOS2" OFF)
 
 set(MFEM_MPI_NP 4 CACHE STRING "Number of processes used for MPI tests")

config/defaults.mk

@@ -137,6 +137,7 @@ MFEM_USE_RAJA          = NO
 MFEM_USE_OCCA          = NO
 MFEM_USE_CEED          = NO
 MFEM_USE_UMPIRE        = NO
+MFEM_USE_SIMD          = YES
 MFEM_USE_ADIOS2        = NO
 
 # Compile and link options for zlib.

config/tconfig.hpp

@@ -29,8 +29,20 @@
 #define MFEM_ALWAYS_INLINE
 #endif
 
+// --- MFEM_VECTORIZE_LOOP (disabled)
+#if (__cplusplus >= 201103L) && !defined(MFEM_DEBUG) && defined(__GNUC__)
+//#define MFEM_VECTORIZE_LOOP _Pragma("GCC ivdep")
+#define MFEM_VECTORIZE_LOOP
+#else
+#define MFEM_VECTORIZE_LOOP
+#endif
+
+// MFEM_TEMPLATE_BLOCK_SIZE is the block size used by the template matrix-matrix
+// multiply, Mult_AB, defined in tmatrix.hpp. This parameter will generally
+// require tuning to determine good value. It is probably highly influenced by
+// the SIMD width when Mult_AB is used with a SIMD type like AutoSIMD.
 #define MFEM_TEMPLATE_BLOCK_SIZE 4
-#define MFEM_SIMD_SIZE 32
+
 #define MFEM_TEMPLATE_ENABLE_SERIALIZE
 
 // #define MFEM_TEMPLATE_ELTRANS_HAS_NODE_DOFS
@@ -38,11 +50,6 @@
 // #define MFEM_TEMPLATE_FIELD_EVAL_DATA_HAS_DOFS
 #define MFEM_TEMPLATE_INTRULE_COEFF_PRECOMP
 
-// derived macros
-#define MFEM_ROUNDUP(val,base) ((((val)+(base)-1)/(base))*(base))
-#define MFEM_ALIGN_SIZE(size,type) \
-   MFEM_ROUNDUP(size,(MFEM_SIMD_SIZE)/sizeof(type))
-
 #ifdef MFEM_COUNT_FLOPS
 namespace mfem
 {

examples/ex14p.cpp

@@ -35,6 +35,38 @@
 using namespace std;
 using namespace mfem;
 
+class CustomSolverMonitor : public IterativeSolverMonitor
+{
+public:
+   CustomSolverMonitor(const ParMesh *m,
+                       ParGridFunction *f) :
+      pmesh(m),
+      pgf(f) {}
+
+   void MonitorSolution(int i, double norm, const Vector &x, bool final)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      int  num_procs, myid;
+
+      MPI_Comm_size(pmesh->GetComm(),&num_procs);
+      MPI_Comm_rank(pmesh->GetComm(),&myid);
+
+      pgf->SetFromTrueDofs(x);
+
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << *pgf
+               << "window_title 'Iteration no " << i << "'"
+               << "keys rRjlc\n" << flush;
+   }
+
+private:
+   const ParMesh *pmesh;
+   ParGridFunction *pgf;
+};
+
 int main(int argc, char *argv[])
 {
    // 1. Initialize MPI.
@@ -188,6 +220,7 @@ int main(int argc, char *argv[])
    }
    else
    {
+      CustomSolverMonitor monitor(pmesh, &x);
       GMRESSolver gmres(MPI_COMM_WORLD);
       gmres.SetAbsTol(0.0);
       gmres.SetRelTol(1e-12);
@@ -196,6 +229,7 @@ int main(int argc, char *argv[])
       gmres.SetPrintLevel(1);
       gmres.SetOperator(*A);
       gmres.SetPreconditioner(*amg);
+      gmres.SetMonitor(monitor);
       gmres.Mult(*B, *X);
    }
    delete amg;

examples/ex18.hpp

@@ -418,7 +418,7 @@ void FaceIntegrator::AssembleFaceVector(const FiniteElement &el1,
    {
       intorder++;
    }
-   const IntegrationRule *ir = &IntRules.Get(Tr.FaceGeom, intorder);
+   const IntegrationRule *ir = &IntRules.Get(Tr.GetGeometryType(), intorder);
 
    for (int i = 0; i < ir->GetNPoints(); i++)
    {
@@ -435,10 +435,10 @@ void FaceIntegrator::AssembleFaceVector(const FiniteElement &el1,
       elfun1_mat.MultTranspose(shape1, funval1);
       elfun2_mat.MultTranspose(shape2, funval2);
 
-      Tr.Face->SetIntPoint(&ip);
+      Tr.SetIntPoint(&ip);
 
       // Get the normal vector and the flux on the face
-      CalcOrtho(Tr.Face->Jacobian(), nor);
+      CalcOrtho(Tr.Jacobian(), nor);
       const double mcs = rsolver.Eval(funval1, funval2, nor, fluxN);
 
       // Update max char speed

examples/ex19.cpp

@@ -38,6 +38,42 @@
 using namespace std;
 using namespace mfem;
 
+class GeneralResidualMonitor : public IterativeSolverMonitor
+{
+public:
+   GeneralResidualMonitor(const std::string& prefix_, int print_lvl)
+      : prefix(prefix_)
+   {
+      print_level = print_lvl;
+   }
+
+   virtual void MonitorResidual(int it, double norm, const Vector &r, bool final);
+
+private:
+   const std::string prefix;
+   int print_level;
+   mutable double norm0;
+};
+
+void GeneralResidualMonitor::MonitorResidual(int it, double norm,
+                                             const Vector &r, bool final)
+{
+   if (print_level == 1 || (print_level == 3 && (final || it == 0)))
+   {
+      mfem::out << prefix << " iteration " << setw(2) << it
+                << " : ||r|| = " << norm;
+      if (it > 0)
+      {
+         mfem::out << ",  ||r||/||r_0|| = " << norm/norm0;
+      }
+      else
+      {
+         norm0 = norm;
+      }
+      mfem::out << '\n';
+   }
+}
+
 // Custom block preconditioner for the Jacobian of the incompressible nonlinear
 // elasticity operator. It has the form
 //
@@ -103,9 +139,11 @@ protected:
 
    // Newton solver for the hyperelastic operator
    NewtonSolver newton_solver;
+   GeneralResidualMonitor newton_monitor;
 
    // Solver for the Jacobian solve in the Newton method
    Solver *j_solver;
+   GeneralResidualMonitor j_monitor;
 
    // Preconditioner for the Jacobian
    Solver *j_prec;
@@ -410,7 +448,8 @@ RubberOperator::RubberOperator(Array<FiniteElementSpace *> &fes,
                                int iter,
                                Coefficient &c_mu)
    : Operator(fes[0]->GetVSize() + fes[1]->GetVSize()),
-     newton_solver(), mu(c_mu), block_offsets(offsets)
+     newton_solver(), newton_monitor("Newton", 1),
+     j_monitor("  GMRES", 3), mu(c_mu), block_offsets(offsets)
 {
    Array<Vector *> rhs(2);
    rhs = NULL; // Set all entries in the array
@@ -446,7 +485,8 @@ RubberOperator::RubberOperator(Array<FiniteElementSpace *> &fes,
    j_gmres->SetRelTol(1e-12);
    j_gmres->SetAbsTol(1e-12);
    j_gmres->SetMaxIter(300);
-   j_gmres->SetPrintLevel(0);
+   j_gmres->SetPrintLevel(-1);
+   j_gmres->SetMonitor(j_monitor);
    j_gmres->SetPreconditioner(*j_prec);
    j_solver = j_gmres;
 
@@ -454,7 +494,8 @@ RubberOperator::RubberOperator(Array<FiniteElementSpace *> &fes,
    newton_solver.iterative_mode = true;
    newton_solver.SetSolver(*j_solver);
    newton_solver.SetOperator(*this);
-   newton_solver.SetPrintLevel(1);
+   newton_solver.SetPrintLevel(-1);
+   newton_solver.SetMonitor(newton_monitor);
    newton_solver.SetRelTol(rel_tol);
    newton_solver.SetAbsTol(abs_tol);
    newton_solver.SetMaxIter(iter);

examples/ex19p.cpp

@@ -38,6 +38,56 @@
 using namespace std;
 using namespace mfem;
 
+class GeneralResidualMonitor : public IterativeSolverMonitor
+{
+public:
+   GeneralResidualMonitor(MPI_Comm comm, const std::string& prefix_,
+                          int print_lvl)
+      : prefix(prefix_)
+   {
+#ifndef MFEM_USE_MPI
+      print_level = print_lvl;
+#else
+      int rank;
+      MPI_Comm_rank(comm, &rank);
+      if (rank == 0)
+      {
+         print_level = print_lvl;
+      }
+      else
+      {
+         print_level = -1;
+      }
+#endif
+   }
+
+   virtual void MonitorResidual(int it, double norm, const Vector &r, bool final);
+
+private:
+   const std::string prefix;
+   int print_level;
+   mutable double norm0;
+};
+
+void GeneralResidualMonitor::MonitorResidual(int it, double norm,
+                                             const Vector &r, bool final)
+{
+   if (print_level == 1 || (print_level == 3 && (final || it == 0)))
+   {
+      mfem::out << prefix << " iteration " << setw(2) << it
+                << " : ||r|| = " << norm;
+      if (it > 0)
+      {
+         mfem::out << ",  ||r||/||r_0|| = " << norm/norm0;
+      }
+      else
+      {
+         norm0 = norm;
+      }
+      mfem::out << '\n';
+   }
+}
+
 // Custom block preconditioner for the Jacobian of the incompressible nonlinear
 // elasticity operator. It has the form
 //
@@ -103,9 +153,11 @@ protected:
 
    // Newton solver for the hyperelastic operator
    NewtonSolver newton_solver;
+   GeneralResidualMonitor newton_monitor;
 
    // Solver for the Jacobian solve in the Newton method
    Solver *j_solver;
+   GeneralResidualMonitor j_monitor;
 
    // Preconditioner for the Jacobian
    Solver *j_prec;
@@ -459,7 +511,10 @@ RubberOperator::RubberOperator(Array<ParFiniteElementSpace *> &fes,
                                int iter,
                                Coefficient &c_mu)
    : Operator(fes[0]->TrueVSize() + fes[1]->TrueVSize()),
-     newton_solver(fes[0]->GetComm()), mu(c_mu), block_trueOffsets(trueOffsets)
+     newton_solver(fes[0]->GetComm()),
+     newton_monitor(fes[0]->GetComm(), "Newton", 1),
+     j_monitor(fes[0]->GetComm(), "  GMRES", 3),
+     mu(c_mu), block_trueOffsets(trueOffsets)
 {
    Array<Vector *> rhs(2);
    rhs = NULL; // Set all entries in the array
@@ -499,7 +554,8 @@ RubberOperator::RubberOperator(Array<ParFiniteElementSpace *> &fes,
    j_gmres->SetRelTol(1e-12);
    j_gmres->SetAbsTol(1e-12);
    j_gmres->SetMaxIter(300);
-   j_gmres->SetPrintLevel(0);
+   j_gmres->SetPrintLevel(-1);
+   j_gmres->SetMonitor(j_monitor);
    j_gmres->SetPreconditioner(*j_prec);
    j_solver = j_gmres;
 
@@ -507,7 +563,8 @@ RubberOperator::RubberOperator(Array<ParFiniteElementSpace *> &fes,
    newton_solver.iterative_mode = true;
    newton_solver.SetSolver(*j_solver);
    newton_solver.SetOperator(*this);
-   newton_solver.SetPrintLevel(1);
+   newton_solver.SetPrintLevel(-1);
+   newton_solver.SetMonitor(newton_monitor);
    newton_solver.SetRelTol(rel_tol);
    newton_solver.SetAbsTol(abs_tol);
    newton_solver.SetMaxIter(iter);

examples/ex24.cpp

@@ -6,6 +6,7 @@
 //               ex24 -m ../data/square-disc.mesh -o 2
 //               ex24 -m ../data/beam-tet.mesh
 //               ex24 -m ../data/beam-hex.mesh -o 2 -pa
+//               ex24 -m ../data/beam-hex.mesh -o 2 -pa -p 1
 //               ex24 -m ../data/escher.mesh
 //               ex24 -m ../data/escher.mesh -o 2
 //               ex24 -m ../data/fichera.mesh
@@ -23,11 +24,15 @@
 //               ex24 -m ../data/beam-hex.mesh -pa -d cuda
 //
 // Description:  This example code illustrates usage of mixed finite element
-//               spaces. Using two different approaches, we project a gradient
-//               of a function in H^1 to H(curl). Other spaces and example
-//               computations are to be added in the future.
+//               spaces, with two variants:
 //
-//               We recommend viewing examples 1 and 3 before viewing this
+//               1) (grad p, u) for p in H^1 tested against u in H(curl)
+//               2) (div v, q) for v in H(div) tested against q in L_2
+//
+//               Using different approaches, we project the gradient or
+//               divergence to the appropriate space.
+//
+//               We recommend viewing examples 1, 3, and 5 before viewing this
 //               example.
 
 #include "mfem.hpp"
@@ -39,6 +44,7 @@ using namespace mfem;
 
 double p_exact(const Vector &x);
 void gradp_exact(const Vector &, Vector &);
+double div_gradp_exact(const Vector &x);
 
 int dim;
 
@@ -47,6 +53,7 @@ int main(int argc, char *argv[])
    // 1. Parse command-line options.
    const char *mesh_file = "../data/beam-hex.mesh";
    int order = 1;
+   int prob = 0;
    bool static_cond = false;
    bool pa = false;
    const char *device_config = "cpu";
@@ -57,6 +64,8 @@ int main(int argc, char *argv[])
                   "Mesh file to use.");
    args.AddOption(&order, "-o", "--order",
                   "Finite element order (polynomial degree).");
+   args.AddOption(&prob, "-p", "--problem-type",
+                  "Choose between 0: H(Curl) or 1: H(Div)");
    args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
                   "--no-static-condensation", "Enable static condensation.");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
@@ -100,72 +109,107 @@ int main(int argc, char *argv[])
    }
    mesh->ReorientTetMesh();
 
-   // 5. Define a parallel finite element space on the parallel mesh. Here we
-   //    use the Nedelec finite elements of the specified order.
-   FiniteElementCollection *fec = new ND_FECollection(order, dim);
-   FiniteElementCollection *H1fec = new H1_FECollection(order, dim);
-   FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
-   FiniteElementSpace *H1fespace = new FiniteElementSpace(mesh, H1fec);
+   // 5. Define a finite element space on the mesh. Here we use Nedelec or
+   //    Raviart-Thomas finite elements of the specified order.
+   FiniteElementCollection *trial_fec = NULL;
+   FiniteElementCollection *test_fec = NULL;
 
-   int size = fespace->GetTrueVSize();
-   int H1size = H1fespace->GetTrueVSize();
-   cout << "Number of Nedelec finite element unknowns: " << size << endl;
-   cout << "Number of H1 finite element unknowns: " << H1size << endl;
-
-   // 6. Define the solution vector x as a parallel finite element grid function
-   //    corresponding to fespace. Initialize x by projecting the exact
-   //    solution. Note that only values from the boundary edges will be used
-   //    when eliminating the non-homogeneous boundary condition to modify the
-   //    r.h.s. vector b.
-   GridFunction x(fespace);
-   FunctionCoefficient p_coef(p_exact);
-   GridFunction p(H1fespace);
-   p.ProjectCoefficient(p_coef);
-   p.SetTrueVector();
-   p.SetFromTrueVector();
-
-   VectorFunctionCoefficient gradp_coef(sdim, gradp_exact);
-
-   // 7. Set up the bilinear forms.
-   Coefficient *muinv = new ConstantCoefficient(1.0);
-   Coefficient *sigma = new ConstantCoefficient(1.0);
-   BilinearForm *a = new BilinearForm(fespace);
-   MixedBilinearForm *a_NDH1 = new MixedBilinearForm(H1fespace, fespace);
-   if (pa)
+   if (prob == 0)
    {
-      a->SetAssemblyLevel(AssemblyLevel::PARTIAL);
-      a_NDH1->SetAssemblyLevel(AssemblyLevel::PARTIAL);
-   }
-
-   // First approach: L2 projection
-   a->AddDomainIntegrator(new VectorFEMassIntegrator(*sigma));
-   a_NDH1->AddDomainIntegrator(new MixedVectorGradientIntegrator(*muinv));
-
-   // 8. Assemble the parallel bilinear form and the corresponding linear
-   //    system, applying any necessary transformations such as: parallel
-   //    assembly, eliminating boundary conditions, applying conforming
-   //    constraints for non-conforming AMR, static condensation, etc.
-   if (static_cond) { a->EnableStaticCondensation(); }
-
-   a->Assemble();
-   if (!pa) { a->Finalize(); }
-
-   a_NDH1->Assemble();
-   if (!pa) { a_NDH1->Finalize(); }
-
-   if (pa)
-   {
-      a_NDH1->Mult(p, x);
+      trial_fec = new H1_FECollection(order, dim);
+      test_fec = new ND_FECollection(order, dim);
    }
    else
    {
-      SparseMatrix& NDH1 = a_NDH1->SpMat();
-      NDH1.Mult(p, x);
+      trial_fec = new RT_FECollection(order - 1, dim);
+      test_fec = new L2_FECollection(order - 1, dim);
+   }
+
+   FiniteElementSpace trial_fes(mesh, trial_fec);
+   FiniteElementSpace test_fes(mesh, test_fec);
+
+   int trial_size = trial_fes.GetTrueVSize();
+   int test_size = test_fes.GetTrueVSize();
+
+   if (prob == 0)
+   {
+      cout << "Number of Nedelec finite element unknowns: " << test_size << endl;
+      cout << "Number of H1 finite element unknowns: " << trial_size << endl;
+   }
+   else
+   {
+      cout << "Number of Raviart-Thomas finite element unknowns: "
+           << trial_size << endl;
+      cout << "Number of L2 finite element unknowns: " << test_size << endl;
+   }
+
+   // 6. Define the solution vector as a finite element grid function
+   //    corresponding to the trial fespace.
+   GridFunction gftest(&test_fes);
+   GridFunction gftrial(&trial_fes);
+   GridFunction x(&test_fes);
+   FunctionCoefficient p_coef(p_exact);
+   VectorFunctionCoefficient gradp_coef(sdim, gradp_exact);
+   FunctionCoefficient divgradp_coef(div_gradp_exact);
+
+   if (prob == 0)
+   {
+      gftrial.ProjectCoefficient(p_coef);
+   }
+   else
+   {
+      gftrial.ProjectCoefficient(gradp_coef);
+   }
+
+   gftrial.SetTrueVector();
+   gftrial.SetFromTrueVector();
+
+   // 7. Set up the bilinear forms for L2 projection.
+   ConstantCoefficient one(1.0);
+   BilinearForm a(&test_fes);
+   MixedBilinearForm a_mixed(&trial_fes, &test_fes);
+   if (pa)
+   {
+      a.SetAssemblyLevel(AssemblyLevel::PARTIAL);
+      a_mixed.SetAssemblyLevel(AssemblyLevel::PARTIAL);
+   }
+
+   if (prob == 0)
+   {
+      a.AddDomainIntegrator(new VectorFEMassIntegrator(one));
+      a_mixed.AddDomainIntegrator(new MixedVectorGradientIntegrator(one));
+   }
+   else
+   {
+      a.AddDomainIntegrator(new MassIntegrator(one));
+      a_mixed.AddDomainIntegrator(new VectorFEDivergenceIntegrator(one));
+   }
+
+   // 8. Assemble the bilinear form and the corresponding linear system,
+   //    applying any necessary transformations such as: eliminating boundary
+   //    conditions, applying conforming constraints for non-conforming AMR,
+   //    static condensation, etc.
+   if (static_cond) { a.EnableStaticCondensation(); }
+
+   a.Assemble();
+   if (!pa) { a.Finalize(); }
+
+   a_mixed.Assemble();
+   if (!pa) { a_mixed.Finalize(); }
+
+   if (pa)
+   {
+      a_mixed.Mult(gftrial, x);
+   }
+   else
+   {
+      SparseMatrix& mixed = a_mixed.SpMat();
+      mixed.Mult(gftrial, x);
    }
 
    // 9. Define and apply a PCG solver for Ax = b with Jacobi preconditioner.
    {
-      GridFunction rhs(fespace);
+      GridFunction rhs(&test_fes);
       rhs = x;
       x = 0.0;
 
@@ -176,15 +220,15 @@ int main(int argc, char *argv[])
       if (pa)
       {
          Array<int> ess_tdof_list; // empty
-         OperatorJacobiSmoother Jacobi(*a, ess_tdof_list);
+         OperatorJacobiSmoother Jacobi(a, ess_tdof_list);
 
-         cg.SetOperator(*a);
+         cg.SetOperator(a);
          cg.SetPreconditioner(Jacobi);
          cg.Mult(rhs, x);
       }
       else
       {
-         SparseMatrix& Amat = a->SpMat();
+         SparseMatrix& Amat = a.SpMat();
          DSmoother Jacobi(Amat);
 
          cg.SetOperator(Amat);
@@ -193,33 +237,68 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 10. Second approach: compute the same solution by applying
-   //     GradientInterpolator in H(curl).
-   DiscreteLinearOperator grad(H1fespace, fespace);
-   grad.AddDomainInterpolator(new GradientInterpolator());
-   grad.Assemble();
+   // 10. Compute the same field by applying a DiscreteInterpolator.
+   GridFunction discreteInterpolant(&test_fes);
+   DiscreteLinearOperator dlo(&trial_fes, &test_fes);
+   if (prob == 0)
+   {
+      dlo.AddDomainInterpolator(new GradientInterpolator());
+   }
+   else
+   {
+      dlo.AddDomainInterpolator(new DivergenceInterpolator());
+   }
 
-   GridFunction gradp(fespace);
-   grad.Mult(p, gradp);
+   dlo.Assemble();
+   dlo.Mult(gftrial, discreteInterpolant);
 
-   // 11. Compute the projection of the exact grad p.
-   GridFunction exact_gradp(fespace);
-   exact_gradp.ProjectCoefficient(gradp_coef);
-   exact_gradp.SetTrueVector();
-   exact_gradp.SetFromTrueVector();
+   // 11. Compute the projection of the exact field.
+   GridFunction exact_proj(&test_fes);
+   if (prob == 0)
+   {
+      exact_proj.ProjectCoefficient(gradp_coef);
+   }
+   else
+   {
+      exact_proj.ProjectCoefficient(divgradp_coef);
+   }
 
-   // 12. Compute and print the L^2 norm of the error.
+   exact_proj.SetTrueVector();
+   exact_proj.SetFromTrueVector();
+
+   // 12. Compute and print the L_2 norm of the error.
+   if (prob == 0)
    {
       double errSol = x.ComputeL2Error(gradp_coef);
-      double errInterp = gradp.ComputeL2Error(gradp_coef);
-      double errProj = exact_gradp.ComputeL2Error(gradp_coef);
+      double errInterp = discreteInterpolant.ComputeL2Error(gradp_coef);
+      double errProj = exact_proj.ComputeL2Error(gradp_coef);
 
       cout << "\n Solution of (E_h,v) = (grad p_h,v) for E_h and v in H(curl): "
-           "|| E_h - grad p ||_{L^2} = " << errSol << '\n' << endl;
+           "|| E_h - grad p ||_{L_2} = " << errSol << '\n' << endl;
       cout << " Gradient interpolant E_h = grad p_h in H(curl): || E_h - grad p"
-           "||_{L^2} = " << errInterp << '\n' << endl;
+           "||_{L_2} = " << errInterp << '\n' << endl;
       cout << " Projection E_h of exact grad p in H(curl): || E_h - grad p "
-           "||_{L^2} = " << errProj << '\n' << endl;
+           "||_{L_2} = " << errProj << '\n' << endl;
+   }
+   else
+   {
+      int order_quad = max(2, 2*order+1);
+      const IntegrationRule *irs[Geometry::NumGeom];
+      for (int i=0; i < Geometry::NumGeom; ++i)
+      {
+         irs[i] = &(IntRules.Get(i, order_quad));
+      }
+
+      double errSol = x.ComputeL2Error(divgradp_coef, irs);
+      double errInterp = discreteInterpolant.ComputeL2Error(divgradp_coef, irs);
+      double errProj = exact_proj.ComputeL2Error(divgradp_coef, irs);
+
+      cout << "\n Solution of (f_h,q) = (div v_h,q) for f_h and q in L_2: "
+           "|| f_h - div v ||_{L_2} = " << errSol << '\n' << endl;
+      cout << " Divergence interpolant f_h = div v_h in L_2: || f_h - div v"
+           "||_{L_2} = " << errInterp << '\n' << endl;
+      cout << " Projection f_h of exact div v in L_2: || f_h - div v "
+           "||_{L_2} = " << errProj << '\n' << endl;
    }
 
    // 13. Save the refined mesh and the solution. This output can be viewed
@@ -242,14 +321,8 @@ int main(int argc, char *argv[])
    }
 
    // 15. Free the used memory.
-   delete a;
-   delete a_NDH1;
-   delete sigma;
-   delete muinv;
-   delete fespace;
-   delete H1fespace;
-   delete fec;
-   delete H1fec;
+   delete trial_fec;
+   delete test_fec;
    delete mesh;
 
    return 0;
@@ -284,3 +357,17 @@ void gradp_exact(const Vector &x, Vector &f)
       if (x.Size() == 3) { f(2) = 0.0; }
    }
 }
+
+double div_gradp_exact(const Vector &x)
+{
+   if (dim == 3)
+   {
+      return -3.0 * sin(x(0)) * sin(x(1)) * sin(x(2));
+   }
+   else if (dim == 2)
+   {
+      return -2.0 * sin(x(0)) * sin(x(1));
+   }
+
+   return 0.0;
+}

examples/ex24p.cpp

@@ -6,6 +6,7 @@
 //               mpirun -np 4 ex24p -m ../data/square-disc.mesh -o 2
 //               mpirun -np 4 ex24p -m ../data/beam-tet.mesh
 //               mpirun -np 4 ex24p -m ../data/beam-hex.mesh -o 2 -pa
+//               mpirun -np 4 ex24p -m ../data/beam-hex.mesh -o 2 -p 1 -pa
 //               mpirun -np 4 ex24p -m ../data/escher.mesh
 //               mpirun -np 4 ex24p -m ../data/escher.mesh -o 2
 //               mpirun -np 4 ex24p -m ../data/fichera.mesh
@@ -23,11 +24,15 @@
 //               mpirun -np 4 ex24p -m ../data/beam-hex.mesh -pa -d cuda
 //
 // Description:  This example code illustrates usage of mixed finite element
-//               spaces. Using two different approaches, we project a gradient
-//               of a function in H^1 to H(curl). Other spaces and example
-//               computations are to be added in the future.
+//               spaces, with two variants:
 //
-//               We recommend viewing examples 1 and 3 before viewing this
+//               1) (grad p, u) for p in H^1 tested against u in H(curl)
+//               2) (div v, q) for v in H(div) tested against q in L_2
+//
+//               Using different approaches, we project the gradient or
+//               divergence to the appropriate space.
+//
+//               We recommend viewing examples 1, 3, and 5 before viewing this
 //               example.
 
 #include "mfem.hpp"
@@ -39,6 +44,7 @@ using namespace mfem;
 
 double p_exact(const Vector &x);
 void gradp_exact(const Vector &, Vector &);
+double div_gradp_exact(const Vector &x);
 
 int dim;
 
@@ -53,6 +59,7 @@ int main(int argc, char *argv[])
    // 2. Parse command-line options.
    const char *mesh_file = "../data/beam-hex.mesh";
    int order = 1;
+   int prob = 0;
    bool static_cond = false;
    bool pa = false;
    const char *device_config = "cpu";
@@ -63,6 +70,8 @@ int main(int argc, char *argv[])
                   "Mesh file to use.");
    args.AddOption(&order, "-o", "--order",
                   "Finite element order (polynomial degree).");
+   args.AddOption(&prob, "-p", "--problem-type",
+                  "Choose between 0: H(Curl) or 1: H(Div)");
    args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
                   "--no-static-condensation", "Enable static condensation.");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
@@ -129,80 +138,115 @@ int main(int argc, char *argv[])
    pmesh->ReorientTetMesh();
 
    // 7. Define a parallel finite element space on the parallel mesh. Here we
-   //    use the Nedelec finite elements of the specified order.
-   FiniteElementCollection *fec = new ND_FECollection(order, dim);
-   FiniteElementCollection *H1fec = new H1_FECollection(order, dim);
-   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
-   ParFiniteElementSpace *H1fespace = new ParFiniteElementSpace(pmesh, H1fec);
-   HYPRE_Int size = fespace->GlobalTrueVSize();
-   HYPRE_Int H1size = H1fespace->GlobalTrueVSize();
+   //    use Nedelec or Raviart-Thomas finite elements of the specified order.
+   FiniteElementCollection *trial_fec = NULL;
+   FiniteElementCollection *test_fec = NULL;
+
+   if (prob == 0)
+   {
+      trial_fec = new H1_FECollection(order, dim);
+      test_fec = new ND_FECollection(order, dim);
+   }
+   else
+   {
+      trial_fec = new RT_FECollection(order - 1, dim);
+      test_fec = new L2_FECollection(order - 1, dim);
+   }
+
+   ParFiniteElementSpace trial_fes(pmesh, trial_fec);
+   ParFiniteElementSpace test_fes(pmesh, test_fec);
+
+   HYPRE_Int trial_size = trial_fes.GlobalTrueVSize();
+   HYPRE_Int test_size = test_fes.GlobalTrueVSize();
+
    if (myid == 0)
    {
-      cout << "Number of Nedelec finite element unknowns: " << size << endl;
-      cout << "Number of H1 finite element unknowns: " << H1size << endl;
+      if (prob == 0)
+      {
+         cout << "Number of Nedelec finite element unknowns: " << test_size << endl;
+         cout << "Number of H1 finite element unknowns: " << trial_size << endl;
+      }
+      else
+      {
+         cout << "Number of Raviart-Thomas finite element unknowns: "
+              << trial_size << endl;
+         cout << "Number of L2 finite element unknowns: " << test_size << endl;
+      }
    }
 
-   // 8. Define the solution vector x as a parallel finite element grid function
-   //    corresponding to fespace. Initialize x by projecting the exact
-   //    solution. Note that only values from the boundary edges will be used
-   //    when eliminating the non-homogeneous boundary condition to modify the
-   //    r.h.s. vector b.
-   ParGridFunction x(fespace);
+   // 8. Define the solution vector as a parallel finite element grid function
+   //    corresponding to the trial fespace.
+   ParGridFunction gftest(&test_fes);
+   ParGridFunction gftrial(&trial_fes);
+   ParGridFunction x(&test_fes);
    FunctionCoefficient p_coef(p_exact);
-   ParGridFunction p(H1fespace);
-   p.ProjectCoefficient(p_coef);
-   p.SetTrueVector();
-   p.SetFromTrueVector();
-
    VectorFunctionCoefficient gradp_coef(sdim, gradp_exact);
+   FunctionCoefficient divgradp_coef(div_gradp_exact);
 
-   // 9. Set up the parallel bilinear forms.
-   Coefficient *muinv = new ConstantCoefficient(1.0);
-   Coefficient *sigma = new ConstantCoefficient(1.0);
-   ParBilinearForm *a = new ParBilinearForm(fespace);
-   ParMixedBilinearForm *a_NDH1 = new ParMixedBilinearForm(H1fespace, fespace);
+   if (prob == 0)
+   {
+      gftrial.ProjectCoefficient(p_coef);
+   }
+   else
+   {
+      gftrial.ProjectCoefficient(gradp_coef);
+   }
+
+   gftrial.SetTrueVector();
+   gftrial.SetFromTrueVector();
+
+   // 9. Set up the parallel bilinear forms for L2 projection.
+   ConstantCoefficient one(1.0);
+   ParBilinearForm a(&test_fes);
+   ParMixedBilinearForm a_mixed(&trial_fes, &test_fes);
    if (pa)
    {
-      a->SetAssemblyLevel(AssemblyLevel::PARTIAL);
-      a_NDH1->SetAssemblyLevel(AssemblyLevel::PARTIAL);
+      a.SetAssemblyLevel(AssemblyLevel::PARTIAL);
+      a_mixed.SetAssemblyLevel(AssemblyLevel::PARTIAL);
    }
 
-   // First approach: L2 projection
-   a->AddDomainIntegrator(new VectorFEMassIntegrator(*sigma));
-   a_NDH1->AddDomainIntegrator(new MixedVectorGradientIntegrator(*muinv));
+   if (prob == 0)
+   {
+      a.AddDomainIntegrator(new VectorFEMassIntegrator(one));
+      a_mixed.AddDomainIntegrator(new MixedVectorGradientIntegrator(one));
+   }
+   else
+   {
+      a.AddDomainIntegrator(new MassIntegrator(one));
+      a_mixed.AddDomainIntegrator(new VectorFEDivergenceIntegrator(one));
+   }
 
    // 10. Assemble the parallel bilinear form and the corresponding linear
    //     system, applying any necessary transformations such as: parallel
    //     assembly, eliminating boundary conditions, applying conforming
    //     constraints for non-conforming AMR, static condensation, etc.
-   if (static_cond) { a->EnableStaticCondensation(); }
+   if (static_cond) { a.EnableStaticCondensation(); }
 
-   a->Assemble();
-   if (!pa) { a->Finalize(); }
+   a.Assemble();
+   if (!pa) { a.Finalize(); }
 
-   a_NDH1->Assemble();
-   if (!pa) { a_NDH1->Finalize(); }
+   a_mixed.Assemble();
+   if (!pa) { a_mixed.Finalize(); }
 
-   Vector B(fespace->GetTrueVSize());
-   Vector X(fespace->GetTrueVSize());
+   Vector B(test_fes.GetTrueVSize());
+   Vector X(test_fes.GetTrueVSize());
 
    if (pa)
    {
-      ParLinearForm *b = new ParLinearForm(fespace); // used as a vector
-      a_NDH1->Mult(p, *b); // process-local multiplication
-      b->ParallelAssemble(B);
-      delete b;
+      ParLinearForm b(&test_fes); // used as a vector
+      a_mixed.Mult(gftrial, b); // process-local multiplication
+      b.ParallelAssemble(B);
    }
    else
    {
-      HypreParMatrix *NDH1 = a_NDH1->ParallelAssemble();
+      HypreParMatrix *mixed = a_mixed.ParallelAssemble();
 
-      Vector P(H1fespace->GetTrueVSize());
-      p.GetTrueDofs(P);
+      Vector P(trial_fes.GetTrueVSize());
+      gftrial.GetTrueDofs(P);
 
-      NDH1->Mult(P,B);
+      mixed->Mult(P,B);
 
-      delete NDH1;
+      delete mixed;
    }
 
    // 11. Define and apply a parallel PCG solver for AX=B with Jacobi
@@ -212,9 +256,9 @@ int main(int argc, char *argv[])
       Array<int> ess_tdof_list; // empty
 
       OperatorPtr A;
-      a->FormSystemMatrix(ess_tdof_list, A);
+      a.FormSystemMatrix(ess_tdof_list, A);
 
-      OperatorJacobiSmoother Jacobi(*a, ess_tdof_list);
+      OperatorJacobiSmoother Jacobi(a, ess_tdof_list);
 
       CGSolver cg(MPI_COMM_WORLD);
       cg.SetRelTol(1e-12);
@@ -227,7 +271,7 @@ int main(int argc, char *argv[])
    }
    else
    {
-      HypreParMatrix *Amat = a->ParallelAssemble();
+      HypreParMatrix *Amat = a.ParallelAssemble();
       HypreDiagScale Jacobi(*Amat);
       HyprePCG pcg(*Amat);
       pcg.SetTol(1e-12);
@@ -242,35 +286,73 @@ int main(int argc, char *argv[])
 
    x.SetFromTrueDofs(X);
 
-   // 12. Second approach: compute the same solution by applying
-   //     GradientInterpolator in H(curl).
-   ParDiscreteLinearOperator grad(H1fespace, fespace);
-   grad.AddDomainInterpolator(new GradientInterpolator());
-   grad.Assemble();
+   // 12. Compute the same field by applying a DiscreteInterpolator.
+   ParGridFunction discreteInterpolant(&test_fes);
+   ParDiscreteLinearOperator dlo(&trial_fes, &test_fes);
+   if (prob == 0)
+   {
+      dlo.AddDomainInterpolator(new GradientInterpolator());
+   }
+   else
+   {
+      dlo.AddDomainInterpolator(new DivergenceInterpolator());
+   }
 
-   ParGridFunction gradp(fespace);
-   grad.Mult(p, gradp);
+   dlo.Assemble();
+   dlo.Mult(gftrial, discreteInterpolant);
 
-   // 13. Compute the projection of the exact grad p.
-   ParGridFunction exact_gradp(fespace);
-   exact_gradp.ProjectCoefficient(gradp_coef);
-   exact_gradp.SetTrueVector();
-   exact_gradp.SetFromTrueVector();
+   // 13. Compute the projection of the exact field.
+   ParGridFunction exact_proj(&test_fes);
+   if (prob == 0)
+   {
+      exact_proj.ProjectCoefficient(gradp_coef);
+   }
+   else
+   {
+      exact_proj.ProjectCoefficient(divgradp_coef);
+   }
 
-   // 14. Compute and print the L^2 norm of the error.
+   exact_proj.SetTrueVector();
+   exact_proj.SetFromTrueVector();
+
+   // 14. Compute and print the L_2 norm of the error.
+   if (prob == 0)
    {
       double errSol = x.ComputeL2Error(gradp_coef);
-      double errInterp = gradp.ComputeL2Error(gradp_coef);
-      double errProj = exact_gradp.ComputeL2Error(gradp_coef);
+      double errInterp = discreteInterpolant.ComputeL2Error(gradp_coef);
+      double errProj = exact_proj.ComputeL2Error(gradp_coef);
 
       if (myid == 0)
       {
-         cout << "\n Solution of (E_h,v) = (grad p_h,v) for E_h and v in "
-              "H(curl): || E_h - grad p ||_{L^2} = " << errSol << '\n' << endl;
-         cout << " Gradient interpolant E_h = grad p_h in H(curl): || E_h - "
-              "grad p ||_{L^2} = " << errInterp << '\n' << endl;
+         cout << "\n Solution of (E_h,v) = (grad p_h,v) for E_h and v in H(curl): "
+              "|| E_h - grad p ||_{L_2} = " << errSol << '\n' << endl;
+         cout << " Gradient interpolant E_h = grad p_h in H(curl): || E_h - grad p"
+              "||_{L_2} = " << errInterp << '\n' << endl;
          cout << " Projection E_h of exact grad p in H(curl): || E_h - grad p "
-              "||_{L^2} = " << errProj << '\n' << endl;
+              "||_{L_2} = " << errProj << '\n' << endl;
+      }
+   }
+   else
+   {
+      int order_quad = max(2, 2*order+1);
+      const IntegrationRule *irs[Geometry::NumGeom];
+      for (int i=0; i < Geometry::NumGeom; ++i)
+      {
+         irs[i] = &(IntRules.Get(i, order_quad));
+      }
+
+      double errSol = x.ComputeL2Error(divgradp_coef, irs);
+      double errInterp = discreteInterpolant.ComputeL2Error(divgradp_coef, irs);
+      double errProj = exact_proj.ComputeL2Error(divgradp_coef, irs);
+
+      if (myid == 0)
+      {
+         cout << "\n Solution of (f_h,q) = (div v_h,q) for f_h and q in L_2: "
+              "|| f_h - div v ||_{L_2} = " << errSol << '\n' << endl;
+         cout << " Divergence interpolant f_h = div v_h in L_2: || f_h - div v"
+              "||_{L_2} = " << errInterp << '\n' << endl;
+         cout << " Projection f_h of exact div v in L_2: || f_h - div v "
+              "||_{L_2} = " << errProj << '\n' << endl;
       }
    }
 
@@ -302,14 +384,8 @@ int main(int argc, char *argv[])
    }
 
    // 17. Free the used memory.
-   delete a;
-   delete a_NDH1;
-   delete sigma;
-   delete muinv;
-   delete fespace;
-   delete H1fespace;
-   delete fec;
-   delete H1fec;
+   delete trial_fec;
+   delete test_fec;
    delete pmesh;
 
    MPI_Finalize();
@@ -346,3 +422,17 @@ void gradp_exact(const Vector &x, Vector &f)
       if (x.Size() == 3) { f(2) = 0.0; }
    }
 }
+
+double div_gradp_exact(const Vector &x)
+{
+   if (dim == 3)
+   {
+      return -3.0 * sin(x(0)) * sin(x(1)) * sin(x(2));
+   }
+   else if (dim == 2)
+   {
+      return -2.0 * sin(x(0)) * sin(x(1));
+   }
+
+   return 0.0;
+}

examples/ex4.cpp

@@ -6,6 +6,7 @@
 //               ex4 -m ../data/star.mesh
 //               ex4 -m ../data/beam-tet.mesh
 //               ex4 -m ../data/beam-hex.mesh
+//               ex4 -m ../data/beam-hex.mesh -o 2 -pa
 //               ex4 -m ../data/escher.mesh
 //               ex4 -m ../data/fichera.mesh -o 2 -hb
 //               ex4 -m ../data/fichera-q2.vtk
@@ -20,6 +21,12 @@
 //               ex4 -m ../data/fichera-amr.mesh -o 2 -sc
 //               ex4 -m ../data/star-surf.mesh -o 1
 //
+// Device sample runs:
+//               ex4 -m ../data/star.mesh -pa -d cuda
+//               ex4 -m ../data/star.mesh -pa -d raja-cuda
+//               ex4 -m ../data/star.mesh -pa -d raja-omp
+//               ex4 -m ../data/beam-hex.mesh -pa -d cuda
+//
 // Description:  This example code solves a simple 2D/3D H(div) diffusion
 //               problem corresponding to the second order definite equation
 //               -grad(alpha div F) + beta F = f with boundary condition F dot n
@@ -55,6 +62,8 @@ int main(int argc, char *argv[])
    bool set_bc = true;
    bool static_cond = false;
    bool hybridization = false;
+   bool pa = false;
+   const char *device_config = "cpu";
    bool visualization = 1;
 
    OptionsParser args(argc, argv);
@@ -70,6 +79,10 @@ int main(int argc, char *argv[])
                   "--no-static-condensation", "Enable static condensation.");
    args.AddOption(&hybridization, "-hb", "--hybridization", "-no-hb",
                   "--no-hybridization", "Enable hybridization.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -82,14 +95,19 @@ int main(int argc, char *argv[])
    args.PrintOptions(cout);
    kappa = freq * M_PI;
 
-   // 2. Read the mesh from the given mesh file. We can handle triangular,
+   // 2. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   device.Print();
+
+   // 3. Read the mesh from the given mesh file. We can handle triangular,
    //    quadrilateral, tetrahedral, hexahedral, surface and volume, as well as
    //    periodic meshes with the same code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
    int sdim = mesh->SpaceDimension();
 
-   // 3. Refine the mesh to increase the resolution. In this example we do
+   // 4. Refine the mesh to increase the resolution. In this example we do
    //    'ref_levels' of uniform refinement. We choose 'ref_levels' to be the
    //    largest number that gives a final mesh with no more than 25,000
    //    elements.
@@ -102,14 +120,14 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 4. Define a finite element space on the mesh. Here we use the
+   // 5. Define a finite element space on the mesh. Here we use the
    //    Raviart-Thomas finite elements of the specified order.
    FiniteElementCollection *fec = new RT_FECollection(order-1, dim);
    FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
    cout << "Number of finite element unknowns: "
         << fespace->GetTrueVSize() << endl;
 
-   // 5. Determine the list of true (i.e. conforming) essential boundary dofs.
+   // 6. Determine the list of true (i.e. conforming) essential boundary dofs.
    //    In this example, the boundary conditions are defined by marking all
    //    the boundary attributes from the mesh as essential (Dirichlet) and
    //    converting them to a list of true dofs.
@@ -121,7 +139,7 @@ int main(int argc, char *argv[])
       fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
    }
 
-   // 6. Set up the linear form b(.) which corresponds to the right-hand side
+   // 7. Set up the linear form b(.) which corresponds to the right-hand side
    //    of the FEM linear system, which in this case is (f,phi_i) where f is
    //    given by the function f_exact and phi_i are the basis functions in the
    //    finite element fespace.
@@ -130,7 +148,7 @@ int main(int argc, char *argv[])
    b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
    b->Assemble();
 
-   // 7. Define the solution vector x as a finite element grid function
+   // 8. Define the solution vector x as a finite element grid function
    //    corresponding to fespace. Initialize x by projecting the exact
    //    solution. Note that only values from the boundary faces will be used
    //    when eliminating the non-homogeneous boundary condition to modify the
@@ -139,16 +157,17 @@ int main(int argc, char *argv[])
    VectorFunctionCoefficient F(sdim, F_exact);
    x.ProjectCoefficient(F);
 
-   // 8. Set up the bilinear form corresponding to the H(div) diffusion operator
+   // 9. Set up the bilinear form corresponding to the H(div) diffusion operator
    //    grad alpha div + beta I, by adding the div-div and the mass domain
    //    integrators.
    Coefficient *alpha = new ConstantCoefficient(1.0);
    Coefficient *beta  = new ConstantCoefficient(1.0);
    BilinearForm *a = new BilinearForm(fespace);
+   if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    a->AddDomainIntegrator(new DivDivIntegrator(*alpha));
    a->AddDomainIntegrator(new VectorFEMassIntegrator(*beta));
 
-   // 9. Assemble the bilinear form and the corresponding linear system,
+   // 10. Assemble the bilinear form and the corresponding linear system,
    //    applying any necessary transformations such as: eliminating boundary
    //    conditions, applying conforming constraints for non-conforming AMR,
    //    static condensation, hybridization, etc.
@@ -167,32 +186,47 @@ int main(int argc, char *argv[])
    }
    a->Assemble();
 
-   SparseMatrix A;
+   OperatorPtr A;
    Vector B, X;
    a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
 
-   cout << "Size of linear system: " << A.Height() << endl;
+   cout << "Size of linear system: " << A->Height() << endl;
 
+   // 11. Solve the linear system A X = B.
+   if (!pa)
+   {
 #ifndef MFEM_USE_SUITESPARSE
-   // 10. Define a simple symmetric Gauss-Seidel preconditioner and use it to
-   //     solve the system A X = B with PCG.
-   GSSmoother M(A);
-   PCG(A, M, B, X, 1, 10000, 1e-20, 0.0);
+      // Use a simple symmetric Gauss-Seidel preconditioner with PCG.
+      GSSmoother M((SparseMatrix&)(*A));
+      PCG(*A, M, B, X, 1, 10000, 1e-20, 0.0);
 #else
-   // 10. If compiled with SuiteSparse support, use UMFPACK to solve the system.
-   UMFPackSolver umf_solver;
-   umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
-   umf_solver.SetOperator(A);
-   umf_solver.Mult(B, X);
+      // If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.
+      UMFPackSolver umf_solver;
+      umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+      umf_solver.SetOperator(*A);
+      umf_solver.Mult(B, X);
 #endif
+   }
+   else // Jacobi preconditioning in partial assembly mode
+   {
+      if (UsesTensorBasis(*fespace))
+      {
+         OperatorJacobiSmoother M(*a, ess_tdof_list);
+         PCG(*A, M, B, X, 1, 10000, 1e-20, 0.0);
+      }
+      else
+      {
+         CG(*A, B, X, 1, 10000, 1e-20, 0.0);
+      }
+   }
 
-   // 11. Recover the solution as a finite element grid function.
+   // 12. Recover the solution as a finite element grid function.
    a->RecoverFEMSolution(X, *b, x);
 
-   // 12. Compute and print the L^2 norm of the error.
+   // 13. Compute and print the L^2 norm of the error.
    cout << "\n|| F_h - F ||_{L^2} = " << x.ComputeL2Error(F) << '\n' << endl;
 
-   // 13. Save the refined mesh and the solution. This output can be viewed
+   // 14. Save the refined mesh and the solution. This output can be viewed
    //     later using GLVis: "glvis -m refined.mesh -g sol.gf".
    {
       ofstream mesh_ofs("refined.mesh");
@@ -203,7 +237,7 @@ int main(int argc, char *argv[])
       x.Save(sol_ofs);
    }
 
-   // 14. Send the solution by socket to a GLVis server.
+   // 15. Send the solution by socket to a GLVis server.
    if (visualization)
    {
       char vishost[] = "localhost";
@@ -213,7 +247,7 @@ int main(int argc, char *argv[])
       sol_sock << "solution\n" << *mesh << x << flush;
    }
 
-   // 15. Free the used memory.
+   // 16. Free the used memory.
    delete hfes;
    delete hfec;
    delete a;
@@ -235,7 +269,7 @@ void F_exact(const Vector &p, Vector &F)
 
    double x = p(0);
    double y = p(1);
-   // double z = (dim == 3) ? p(2) : 0.0;
+   // double z = (dim == 3) ? p(2) : 0.0; // Uncomment if F is changed to depend on z
 
    F(0) = cos(kappa*x)*sin(kappa*y);
    F(1) = cos(kappa*y)*sin(kappa*x);
@@ -252,7 +286,7 @@ void f_exact(const Vector &p, Vector &f)
 
    double x = p(0);
    double y = p(1);
-   // double z = (dim == 3) ? p(2) : 0.0;
+   // double z = (dim == 3) ? p(2) : 0.0; // Uncomment if f is changed to depend on z
 
    double temp = 1 + 2*kappa*kappa;
 

examples/ex4p.cpp

@@ -6,6 +6,7 @@
 //               mpirun -np 4 ex4p -m ../data/star.mesh
 //               mpirun -np 4 ex4p -m ../data/beam-tet.mesh
 //               mpirun -np 4 ex4p -m ../data/beam-hex.mesh
+//               mpirun -np 4 ex4p -m ../data/beam-hex.mesh -o 2 -pa
 //               mpirun -np 4 ex4p -m ../data/escher.mesh -o 2 -sc
 //               mpirun -np 4 ex4p -m ../data/fichera.mesh -o 2 -hb
 //               mpirun -np 4 ex4p -m ../data/fichera-q2.vtk
@@ -19,6 +20,12 @@
 //               mpirun -np 4 ex4p -m ../data/amr-hex.mesh -o 2 -hb
 //               mpirun -np 4 ex4p -m ../data/star-surf.mesh -o 3 -hb
 //
+// Device sample runs:
+//               mpirun -np 4 ex4p -m ../data/star.mesh -pa -d cuda
+//               mpirun -np 4 ex4p -m ../data/star.mesh -pa -d raja-cuda
+//               mpirun -np 4 ex4p -m ../data/star.mesh -pa -d raja-omp
+//               mpirun -np 4 ex4p -m ../data/beam-hex.mesh -pa -d cuda
+//
 // Description:  This example code solves a simple 2D/3D H(div) diffusion
 //               problem corresponding to the second order definite equation
 //               -grad(alpha div F) + beta F = f with boundary condition F dot n
@@ -60,6 +67,8 @@ int main(int argc, char *argv[])
    bool set_bc = true;
    bool static_cond = false;
    bool hybridization = false;
+   bool pa = false;
+   const char *device_config = "cpu";
    bool visualization = 1;
 
    OptionsParser args(argc, argv);
@@ -75,6 +84,10 @@ int main(int argc, char *argv[])
                   "--no-static-condensation", "Enable static condensation.");
    args.AddOption(&hybridization, "-hb", "--hybridization", "-no-hb",
                   "--no-hybridization", "Enable hybridization.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -94,14 +107,19 @@ int main(int argc, char *argv[])
    }
    kappa = freq * M_PI;
 
-   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   // 3. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   if (myid == 0) { device.Print(); }
+
+   // 4. Read the (serial) mesh from the given mesh file on all processors.  We
    //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
    //    and volume, as well as periodic meshes with the same code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
    int sdim = mesh->SpaceDimension();
 
-   // 4. Refine the serial mesh on all processors to increase the resolution. In
+   // 5. Refine the serial mesh on all processors to increase the resolution. In
    //    this example we do 'ref_levels' of uniform refinement. We choose
    //    'ref_levels' to be the largest number that gives a final mesh with no
    //    more than 1,000 elements.
@@ -114,7 +132,7 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   // 6. Define a parallel mesh by a partitioning of the serial mesh. Refine
    //    this mesh further in parallel to increase the resolution. Once the
    //    parallel mesh is defined, the serial mesh can be deleted. Tetrahedral
    //    meshes need to be reoriented before we can define high-order Nedelec
@@ -130,7 +148,7 @@ int main(int argc, char *argv[])
    }
    pmesh->ReorientTetMesh();
 
-   // 6. Define a parallel finite element space on the parallel mesh. Here we
+   // 7. Define a parallel finite element space on the parallel mesh. Here we
    //    use the Raviart-Thomas finite elements of the specified order.
    FiniteElementCollection *fec = new RT_FECollection(order-1, dim);
    ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
@@ -140,7 +158,7 @@ int main(int argc, char *argv[])
       cout << "Number of finite element unknowns: " << size << endl;
    }
 
-   // 7. Determine the list of true (i.e. parallel conforming) essential
+   // 8. Determine the list of true (i.e. parallel conforming) essential
    //    boundary dofs. In this example, the boundary conditions are defined
    //    by marking all the boundary attributes from the mesh as essential
    //    (Dirichlet) and converting them to a list of true dofs.
@@ -152,7 +170,7 @@ int main(int argc, char *argv[])
       fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
    }
 
-   // 8. Set up the parallel linear form b(.) which corresponds to the
+   // 9. Set up the parallel linear form b(.) which corresponds to the
    //    right-hand side of the FEM linear system, which in this case is
    //    (f,phi_i) where f is given by the function f_exact and phi_i are the
    //    basis functions in the finite element fespace.
@@ -161,7 +179,7 @@ int main(int argc, char *argv[])
    b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
    b->Assemble();
 
-   // 9. Define the solution vector x as a parallel finite element grid function
+   // 10. Define the solution vector x as a parallel finite element grid function
    //    corresponding to fespace. Initialize x by projecting the exact
    //    solution. Note that only values from the boundary faces will be used
    //    when eliminating the non-homogeneous boundary condition to modify the
@@ -170,16 +188,17 @@ int main(int argc, char *argv[])
    VectorFunctionCoefficient F(sdim, F_exact);
    x.ProjectCoefficient(F);
 
-   // 10. Set up the parallel bilinear form corresponding to the H(div)
+   // 11. Set up the parallel bilinear form corresponding to the H(div)
    //     diffusion operator grad alpha div + beta I, by adding the div-div and
    //     the mass domain integrators.
    Coefficient *alpha = new ConstantCoefficient(1.0);
    Coefficient *beta  = new ConstantCoefficient(1.0);
    ParBilinearForm *a = new ParBilinearForm(fespace);
+   if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    a->AddDomainIntegrator(new DivDivIntegrator(*alpha));
    a->AddDomainIntegrator(new VectorFEMassIntegrator(*beta));
 
-   // 11. Assemble the parallel bilinear form and the corresponding linear
+   // 12. Assemble the parallel bilinear form and the corresponding linear
    //     system, applying any necessary transformations such as: parallel
    //     assembly, eliminating boundary conditions, applying conforming
    //     constraints for non-conforming AMR, static condensation,
@@ -199,41 +218,43 @@ int main(int argc, char *argv[])
    }
    a->Assemble();
 
-   HypreParMatrix A;
+   OperatorPtr A;
    Vector B, X;
    a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
 
-   HYPRE_Int glob_size = A.GetGlobalNumRows();
-   if (myid == 0)
+   if (myid == 0 && !pa)
    {
-      cout << "Size of linear system: " << glob_size << endl;
+      cout << "Size of linear system: "
+           << A.As<HypreParMatrix>()->GetGlobalNumRows() << endl;
    }
 
-   // 12. Define and apply a parallel PCG solver for A X = B with the 2D AMS or
+   // 13. Define and apply a parallel PCG solver for A X = B with the 2D AMS or
    //     the 3D ADS preconditioners from hypre. If using hybridization, the
-   //     system is preconditioned with hypre's BoomerAMG.
-   HypreSolver *prec = NULL;
-   CGSolver *pcg = new CGSolver(A.GetComm());
-   pcg->SetOperator(A);
+   //     system is preconditioned with hypre's BoomerAMG. In the partial
+   //     assembly case, use Jacobi preconditioning.
+   Solver *prec = NULL;
+   CGSolver *pcg = new CGSolver(MPI_COMM_WORLD);
+   pcg->SetOperator(*A);
    pcg->SetRelTol(1e-12);
-   pcg->SetMaxIter(500);
+   pcg->SetMaxIter(2000);
    pcg->SetPrintLevel(1);
-   if (hybridization) { prec = new HypreBoomerAMG(A); }
+   if (hybridization) { prec = new HypreBoomerAMG(*A.As<HypreParMatrix>()); }
+   else if (pa) { prec = new OperatorJacobiSmoother(*a, ess_tdof_list); }
    else
    {
       ParFiniteElementSpace *prec_fespace =
          (a->StaticCondensationIsEnabled() ? a->SCParFESpace() : fespace);
-      if (dim == 2)   { prec = new HypreAMS(A, prec_fespace); }
-      else            { prec = new HypreADS(A, prec_fespace); }
+      if (dim == 2)   { prec = new HypreAMS(*A.As<HypreParMatrix>(), prec_fespace); }
+      else            { prec = new HypreADS(*A.As<HypreParMatrix>(), prec_fespace); }
    }
    pcg->SetPreconditioner(*prec);
    pcg->Mult(B, X);
 
-   // 13. Recover the parallel grid function corresponding to X. This is the
+   // 14. Recover the parallel grid function corresponding to X. This is the
    //     local finite element solution on each processor.
    a->RecoverFEMSolution(X, *b, x);
 
-   // 14. Compute and print the L^2 norm of the error.
+   // 15. Compute and print the L^2 norm of the error.
    {
       double err = x.ComputeL2Error(F);
       if (myid == 0)
@@ -242,7 +263,7 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 15. Save the refined mesh and the solution in parallel. This output can
+   // 16. Save the refined mesh and the solution in parallel. This output can
    //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
    {
       ostringstream mesh_name, sol_name;
@@ -258,7 +279,7 @@ int main(int argc, char *argv[])
       x.Save(sol_ofs);
    }
 
-   // 16. Send the solution by socket to a GLVis server.
+   // 17. Send the solution by socket to a GLVis server.
    if (visualization)
    {
       char vishost[] = "localhost";
@@ -269,7 +290,7 @@ int main(int argc, char *argv[])
       sol_sock << "solution\n" << *pmesh << x << flush;
    }
 
-   // 17. Free the used memory.
+   // 18. Free the used memory.
    delete pcg;
    delete prec;
    delete hfes;
@@ -295,7 +316,7 @@ void F_exact(const Vector &p, Vector &F)
 
    double x = p(0);
    double y = p(1);
-   // double z = (dim == 3) ? p(2) : 0.0;
+   // double z = (dim == 3) ? p(2) : 0.0; // Uncomment if F is changed to depend on z
 
    F(0) = cos(kappa*x)*sin(kappa*y);
    F(1) = cos(kappa*y)*sin(kappa*x);
@@ -312,7 +333,7 @@ void f_exact(const Vector &p, Vector &f)
 
    double x = p(0);
    double y = p(1);
-   // double z = (dim == 3) ? p(2) : 0.0;
+   // double z = (dim == 3) ? p(2) : 0.0; // Uncomment if f is changed to depend on z
 
    double temp = 1 + 2*kappa*kappa;
 

examples/ex5.cpp

@@ -4,8 +4,10 @@
 //
 // Sample runs:  ex5 -m ../data/square-disc.mesh
 //               ex5 -m ../data/star.mesh
+//               ex5 -m ../data/star.mesh -pa
 //               ex5 -m ../data/beam-tet.mesh
 //               ex5 -m ../data/beam-hex.mesh
+//               ex5 -m ../data/beam-hex.mesh -pa
 //               ex5 -m ../data/escher.mesh
 //               ex5 -m ../data/fichera.mesh
 //
@@ -47,6 +49,7 @@ int main(int argc, char *argv[])
    // 1. Parse command-line options.
    const char *mesh_file = "../data/star.mesh";
    int order = 1;
+   bool pa = false;
    bool visualization = 1;
 
    OptionsParser args(argc, argv);
@@ -54,6 +57,8 @@ int main(int argc, char *argv[])
                   "Mesh file to use.");
    args.AddOption(&order, "-o", "--order",
                   "Finite element order (polynomial degree).");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -146,22 +151,39 @@ int main(int argc, char *argv[])
    BilinearForm *mVarf(new BilinearForm(R_space));
    MixedBilinearForm *bVarf(new MixedBilinearForm(R_space, W_space));
 
+   if (pa) { mVarf->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    mVarf->AddDomainIntegrator(new VectorFEMassIntegrator(k));
    mVarf->Assemble();
-   mVarf->Finalize();
-   SparseMatrix &M(mVarf->SpMat());
+   if (!pa) { mVarf->Finalize(); }
 
+   if (pa) { bVarf->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    bVarf->AddDomainIntegrator(new VectorFEDivergenceIntegrator);
    bVarf->Assemble();
-   bVarf->Finalize();
-   SparseMatrix & B(bVarf->SpMat());
-   B *= -1.;
-   SparseMatrix *BT = Transpose(B);
+   if (!pa) { bVarf->Finalize(); }
 
-   BlockMatrix darcyMatrix(block_offsets);
-   darcyMatrix.SetBlock(0,0, &M);
-   darcyMatrix.SetBlock(0,1, BT);
-   darcyMatrix.SetBlock(1,0, &B);
+   BlockOperator darcyOp(block_offsets);
+
+   TransposeOperator *Bt = NULL;
+
+   if (pa)
+   {
+      Bt = new TransposeOperator(bVarf);
+
+      darcyOp.SetBlock(0,0, mVarf);
+      darcyOp.SetBlock(0,1, Bt, -1.0);
+      darcyOp.SetBlock(1,0, bVarf, -1.0);
+   }
+   else
+   {
+      SparseMatrix &M(mVarf->SpMat());
+      SparseMatrix &B(bVarf->SpMat());
+      B *= -1.;
+      Bt = new TransposeOperator(&B);
+
+      darcyOp.SetBlock(0,0, &M);
+      darcyOp.SetBlock(0,1, Bt);
+      darcyOp.SetBlock(1,0, &B);
+   }
 
    // 9. Construct the operators for preconditioner
    //
@@ -170,27 +192,57 @@ int main(int argc, char *argv[])
    //
    //     Here we use Symmetric Gauss-Seidel to approximate the inverse of the
    //     pressure Schur Complement
-   SparseMatrix *MinvBt = Transpose(B);
-   Vector Md(M.Height());
-   M.GetDiag(Md);
-   for (int i = 0; i < Md.Size(); i++)
-   {
-      MinvBt->ScaleRow(i, 1./Md(i));
-   }
-   SparseMatrix *S = Mult(B, *MinvBt);
+   SparseMatrix *MinvBt = NULL;
+   Vector Md(mVarf->Height());
 
+   BlockDiagonalPreconditioner darcyPrec(block_offsets);
    Solver *invM, *invS;
-   invM = new DSmoother(M);
+   SparseMatrix *S = NULL;
+
+   if (pa)
+   {
+      mVarf->AssembleDiagonal(Md);
+      Vector invMd(mVarf->Height());
+      for (int i=0; i<mVarf->Height(); ++i)
+      {
+         invMd(i) = 1.0 / Md(i);
+      }
+
+      Vector BMBt_diag(bVarf->Height());
+      bVarf->AssembleDiagonal_ADAt(invMd, BMBt_diag);
+
+      Array<int> ess_tdof_list;  // empty
+
+      invM = new OperatorJacobiSmoother(Md, ess_tdof_list);
+      invS = new OperatorJacobiSmoother(BMBt_diag, ess_tdof_list);
+   }
+   else
+   {
+      SparseMatrix &M(mVarf->SpMat());
+      M.GetDiag(Md);
+
+      SparseMatrix &B(bVarf->SpMat());
+      MinvBt = Transpose(B);
+
+      for (int i = 0; i < Md.Size(); i++)
+      {
+         MinvBt->ScaleRow(i, 1./Md(i));
+      }
+
+      S = Mult(B, *MinvBt);
+
+      invM = new DSmoother(M);
+
 #ifndef MFEM_USE_SUITESPARSE
-   invS = new GSSmoother(*S);
+      invS = new GSSmoother(*S);
 #else
-   invS = new UMFPackSolver(*S);
+      invS = new UMFPackSolver(*S);
 #endif
+   }
 
    invM->iterative_mode = false;
    invS->iterative_mode = false;
 
-   BlockDiagonalPreconditioner darcyPrec(block_offsets);
    darcyPrec.SetDiagonalBlock(0, invM);
    darcyPrec.SetDiagonalBlock(1, invS);
 
@@ -206,7 +258,7 @@ int main(int argc, char *argv[])
    solver.SetAbsTol(atol);
    solver.SetRelTol(rtol);
    solver.SetMaxIter(maxIter);
-   solver.SetOperator(darcyMatrix);
+   solver.SetOperator(darcyOp);
    solver.SetPreconditioner(darcyPrec);
    solver.SetPrintLevel(1);
    x = 0.0;
@@ -295,8 +347,8 @@ int main(int argc, char *argv[])
    delete invM;
    delete invS;
    delete S;
+   delete Bt;
    delete MinvBt;
-   delete BT;
    delete mVarf;
    delete bVarf;
    delete W_space;

examples/ex5p.cpp

@@ -4,8 +4,10 @@
 //
 // Sample runs:  mpirun -np 4 ex5p -m ../data/square-disc.mesh
 //               mpirun -np 4 ex5p -m ../data/star.mesh
+//               mpirun -np 4 ex5p -m ../data/star.mesh -r 2 -pa
 //               mpirun -np 4 ex5p -m ../data/beam-tet.mesh
 //               mpirun -np 4 ex5p -m ../data/beam-hex.mesh
+//               mpirun -np 4 ex5p -m ../data/beam-hex.mesh -pa
 //               mpirun -np 4 ex5p -m ../data/escher.mesh
 //               mpirun -np 4 ex5p -m ../data/fichera.mesh
 //
@@ -54,19 +56,25 @@ int main(int argc, char *argv[])
 
    // 2. Parse command-line options.
    const char *mesh_file = "../data/star.mesh";
+   int ref_levels = -1;
    int order = 1;
    bool par_format = false;
+   bool pa = false;
    bool visualization = 1;
    bool adios2 = false;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
                   "Mesh file to use.");
+   args.AddOption(&ref_levels, "-r", "--refine",
+                  "Number of times to refine the mesh uniformly.");
    args.AddOption(&order, "-o", "--order",
                   "Finite element order (polynomial degree).");
    args.AddOption(&par_format, "-pf", "--parallel-format", "-sf",
                   "--serial-format",
                   "Format to use when saving the results for VisIt.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -97,10 +105,13 @@ int main(int argc, char *argv[])
    // 4. Refine the serial mesh on all processors to increase the resolution. In
    //    this example we do 'ref_levels' of uniform refinement. We choose
    //    'ref_levels' to be the largest number that gives a final mesh with no
-   //    more than 10,000 elements.
+   //    more than 10,000 elements, unless the user specifies it as input.
    {
-      int ref_levels =
-         (int)floor(log(10000./mesh->GetNE())/log(2.)/dim);
+      if (ref_levels == -1)
+      {
+         ref_levels = (int)floor(log(10000./mesh->GetNE())/log(2.)/dim);
+      }
+
       for (int l = 0; l < ref_levels; l++)
       {
          mesh->UniformRefinement();
@@ -196,25 +207,47 @@ int main(int argc, char *argv[])
    ParBilinearForm *mVarf(new ParBilinearForm(R_space));
    ParMixedBilinearForm *bVarf(new ParMixedBilinearForm(R_space, W_space));
 
-   HypreParMatrix *M, *B;
+   HypreParMatrix *M = NULL;
+   HypreParMatrix *B = NULL;
 
+   if (pa) { mVarf->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    mVarf->AddDomainIntegrator(new VectorFEMassIntegrator(k));
    mVarf->Assemble();
-   mVarf->Finalize();
-   M = mVarf->ParallelAssemble();
+   if (!pa) { mVarf->Finalize(); }
 
+   if (pa) { bVarf->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    bVarf->AddDomainIntegrator(new VectorFEDivergenceIntegrator);
    bVarf->Assemble();
-   bVarf->Finalize();
-   B = bVarf->ParallelAssemble();
-   (*B) *= -1;
-
-   HypreParMatrix *BT = B->Transpose();
+   if (!pa) { bVarf->Finalize(); }
 
    BlockOperator *darcyOp = new BlockOperator(block_trueOffsets);
-   darcyOp->SetBlock(0,0, M);
-   darcyOp->SetBlock(0,1, BT);
-   darcyOp->SetBlock(1,0, B);
+
+   Array<int> empty_tdof_list;  // empty
+   OperatorPtr opM, opB;
+
+   TransposeOperator *Bt = NULL;
+
+   if (pa)
+   {
+      mVarf->FormSystemMatrix(empty_tdof_list, opM);
+      bVarf->FormRectangularSystemMatrix(empty_tdof_list, empty_tdof_list, opB);
+      Bt = new TransposeOperator(opB.Ptr());
+
+      darcyOp->SetBlock(0,0, opM.Ptr());
+      darcyOp->SetBlock(0,1, Bt, -1.0);
+      darcyOp->SetBlock(1,0, opB.Ptr(), -1.0);
+   }
+   else
+   {
+      M = mVarf->ParallelAssemble();
+      B = bVarf->ParallelAssemble();
+      (*B) *= -1;
+      Bt = new TransposeOperator(B);
+
+      darcyOp->SetBlock(0,0, M);
+      darcyOp->SetBlock(0,1, Bt);
+      darcyOp->SetBlock(1,0, B);
+   }
 
    // 11. Construct the operators for preconditioner
    //
@@ -223,17 +256,43 @@ int main(int argc, char *argv[])
    //
    //     Here we use Symmetric Gauss-Seidel to approximate the inverse of the
    //     pressure Schur Complement.
-   HypreParMatrix *MinvBt = B->Transpose();
-   HypreParVector *Md = new HypreParVector(MPI_COMM_WORLD, M->GetGlobalNumRows(),
-                                           M->GetRowStarts());
-   M->GetDiag(*Md);
+   HypreParMatrix *MinvBt = NULL;
+   HypreParVector *Md = NULL;
+   HypreParMatrix *S = NULL;
+   Vector Md_PA;
+   Solver *invM, *invS;
 
-   MinvBt->InvScaleRows(*Md);
-   HypreParMatrix *S = ParMult(B, MinvBt);
+   if (pa)
+   {
+      Md_PA.SetSize(R_space->GetTrueVSize());
+      mVarf->AssembleDiagonal(Md_PA);
+      Vector invMd(Md_PA.Size());
+      for (int i=0; i<Md_PA.Size(); ++i)
+      {
+         invMd(i) = 1.0 / Md_PA(i);
+      }
 
-   HypreSolver *invM, *invS;
-   invM = new HypreDiagScale(*M);
-   invS = new HypreBoomerAMG(*S);
+      Vector BMBt_diag(W_space->GetTrueVSize());
+      bVarf->AssembleDiagonal_ADAt(invMd, BMBt_diag);
+
+      Array<int> ess_tdof_list;  // empty
+
+      invM = new OperatorJacobiSmoother(Md_PA, ess_tdof_list);
+      invS = new OperatorJacobiSmoother(BMBt_diag, ess_tdof_list);
+   }
+   else
+   {
+      Md = new HypreParVector(MPI_COMM_WORLD, M->GetGlobalNumRows(),
+                              M->GetRowStarts());
+      M->GetDiag(*Md);
+
+      MinvBt = B->Transpose();
+      MinvBt->InvScaleRows(*Md);
+      S = ParMult(B, MinvBt);
+
+      invM = new HypreDiagScale(*M);
+      invS = new HypreBoomerAMG(*S);
+   }
 
    invM->iterative_mode = false;
    invS->iterative_mode = false;
@@ -245,7 +304,7 @@ int main(int argc, char *argv[])
 
    // 12. Solve the linear system with MINRES.
    //     Check the norm of the unpreconditioned residual.
-   int maxIter(500);
+   int maxIter(pa ? 1000 : 500);
    double rtol(1.e-6);
    double atol(1.e-10);
 
@@ -395,7 +454,7 @@ int main(int argc, char *argv[])
    delete S;
    delete Md;
    delete MinvBt;
-   delete BT;
+   delete Bt;
    delete B;
    delete M;
    delete mVarf;

examples/ex9.cpp

@@ -19,6 +19,7 @@
 //
 // Device sample runs:
 //    ex9 -pa
+//    ex9 -ea
 //    ex9 -pa -m ../data/periodic-cube.mesh
 //    ex9 -pa -m ../data/periodic-cube.mesh -d cuda
 //
@@ -142,6 +143,7 @@ int main(int argc, char *argv[])
    int ref_levels = 2;
    int order = 3;
    bool pa = false;
+   bool ea = false;
    const char *device_config = "cpu";
    int ode_solver_type = 4;
    double t_final = 10.0;
@@ -166,6 +168,8 @@ int main(int argc, char *argv[])
                   "Order (degree) of the finite elements.");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
                   "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&ea, "-ea", "--element-assembly", "-no-ea",
+                  "--no-element-assembly", "Enable Element Assembly.");
    args.AddOption(&device_config, "-d", "--device",
                   "Device configuration string, see Device::Configure().");
    args.AddOption(&ode_solver_type, "-s", "--ode-solver",
@@ -269,6 +273,11 @@ int main(int argc, char *argv[])
       m.SetAssemblyLevel(AssemblyLevel::PARTIAL);
       k.SetAssemblyLevel(AssemblyLevel::PARTIAL);
    }
+   else if (ea)
+   {
+      m.SetAssemblyLevel(AssemblyLevel::ELEMENT);
+      k.SetAssemblyLevel(AssemblyLevel::ELEMENT);
+   }
    m.AddDomainIntegrator(new MassIntegrator);
    k.AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
    k.AddInteriorFaceIntegrator(
@@ -429,8 +438,9 @@ FE_Evolution::FE_Evolution(BilinearForm &_M, BilinearForm &_K, const Vector &_b)
    : TimeDependentOperator(_M.Height()), M(_M), K(_K), b(_b), z(_M.Height())
 {
    bool pa = M.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
+   bool ea = M.GetAssemblyLevel() == AssemblyLevel::ELEMENT;
    Array<int> ess_tdof_list;
-   if (pa)
+   if (pa || ea)
    {
       M_prec = new OperatorJacobiSmoother(M, ess_tdof_list);
       M_solver.SetOperator(M);

examples/ex9p.cpp

@@ -19,6 +19,7 @@
 //
 // Device sample runs:
 //    mpirun -np 4 ex9p -pa
+//    mpirun -np 4 ex9p -ea
 //    mpirun -np 4 ex9p -pa -m ../data/periodic-cube.mesh
 //    mpirun -np 4 ex9p -pa -m ../data/periodic-cube.mesh -d cuda
 //
@@ -161,6 +162,7 @@ int main(int argc, char *argv[])
    int par_ref_levels = 0;
    int order = 3;
    bool pa = false;
+   bool ea = false;
    const char *device_config = "cpu";
    int ode_solver_type = 4;
    double t_final = 10.0;
@@ -188,6 +190,8 @@ int main(int argc, char *argv[])
                   "Order (degree) of the finite elements.");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
                   "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&ea, "-ea", "--element-assembly", "-no-ea",
+                  "--no-element-assembly", "Enable Element Assembly.");
    args.AddOption(&device_config, "-d", "--device",
                   "Device configuration string, see Device::Configure().");
    args.AddOption(&ode_solver_type, "-s", "--ode-solver",
@@ -319,6 +323,11 @@ int main(int argc, char *argv[])
       m->SetAssemblyLevel(AssemblyLevel::PARTIAL);
       k->SetAssemblyLevel(AssemblyLevel::PARTIAL);
    }
+   else if (ea)
+   {
+      m->SetAssemblyLevel(AssemblyLevel::ELEMENT);
+      k->SetAssemblyLevel(AssemblyLevel::ELEMENT);
+   }
    m->AddDomainIntegrator(new MassIntegrator);
    k->AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
    k->AddInteriorFaceIntegrator(
@@ -556,8 +565,9 @@ FE_Evolution::FE_Evolution(ParBilinearForm &_M, ParBilinearForm &_K,
      z(_M.Height())
 {
    bool pa = _M.GetAssemblyLevel()==AssemblyLevel::PARTIAL;
+   bool ea = _M.GetAssemblyLevel()==AssemblyLevel::ELEMENT;
 
-   if (pa)
+   if (pa || ea)
    {
       M.Reset(&_M, false);
       K.Reset(&_K, false);
@@ -571,7 +581,7 @@ FE_Evolution::FE_Evolution(ParBilinearForm &_M, ParBilinearForm &_K,
    M_solver.SetOperator(*M);
 
    Array<int> ess_tdof_list;
-   if (pa)
+   if (pa || ea)
    {
       M_prec = new OperatorJacobiSmoother(_M, ess_tdof_list);
       dg_solver = NULL;

examples/pumi/ex1.cpp

@@ -32,6 +32,13 @@
 //               is used for the Finite Element order and "-go" is used for the
 //               geometry order. Note that they can be used independently, i.e.
 //               "-o 8 -go 3" solves for 8th order FE on a third order geometry.
+//
+// NOTE:         Model/Mesh files for this example are in the (large) data file
+//               repository of MFEM here https://github.com/mfem/data under the
+//               folder named "pumi", which consists of the following sub-folders:
+//               a) geom -->  model files
+//               b) parallel --> parallel pumi mesh files
+//               c) serial --> serial pumi mesh files
 
 #include "mfem.hpp"
 #include <fstream>

examples/pumi/ex1p.cpp

@@ -36,6 +36,14 @@
 //               option "-o" is used for the Finite Element order and "-go" for
 //               the geometry order. Note that they can be used independently:
 //               "-o 8 -go 3" solves for 8th order FE on third order geometry.
+//
+// NOTE:         Model/Mesh files for this example are in the (large) data file
+//               repository of MFEM here https://github.com/mfem/data under the
+//               folder named "pumi", which consists of the following sub-folders:
+//               a) geom -->  model files
+//               b) parallel --> parallel pumi mesh files
+//               c) serial --> serial pumi mesh files
+
 
 #include "mfem.hpp"
 #include <fstream>

examples/pumi/ex2.cpp

@@ -43,6 +43,14 @@
 //               also illustrated.
 //
 //               We recommend viewing Example 1 before viewing this example.
+//
+// NOTE:         Model/Mesh files for this example are in the (large) data file
+//               repository of MFEM here https://github.com/mfem/data under the
+//               folder named "pumi", which consists of the following sub-folders:
+//               a) geom -->  model files
+//               b) parallel --> parallel pumi mesh files
+//               c) serial --> serial pumi mesh files
+
 
 #include "mfem.hpp"
 #include <fstream>

examples/pumi/ex6p.cpp

@@ -1,7 +1,7 @@
 //                       MFEM Example 6 - Parallel Version
 //                              PUMI Modification
 //
-// Compile with: make ex1p
+// Compile with: make ex6p
 //
 // Sample runs:  mpirun -np 8 ex6p
 //
@@ -18,6 +18,13 @@
 //               is added to modify the "adapt_ratio" which is the fraction of
 //               allowable error that scales the output size field of the error
 //               estimator.
+//
+// NOTE:         Model/Mesh files for this example are in the (large) data file
+//               repository of MFEM here https://github.com/mfem/data under the
+//               folder named "pumi", which consists of the following sub-folders:
+//               a) geom -->  model files
+//               b) parallel --> parallel pumi mesh files
+//               c) serial --> serial pumi mesh files
 
 #include "mfem.hpp"
 #include <fstream>
@@ -332,7 +339,6 @@ int main(int argc, char *argv[])
 
       apf::destroyField(Tmag_field);
       apf::destroyField(ipfield);
-      apf::destroyNumbering(pumi_mesh->findNumbering("LocalVertexNumbering"));
 
       // 18. Perform MesAdapt.
       ma::Input* erinput = ma::configure(pumi_mesh, sizefield);

fem/CMakeLists.txt

@@ -13,13 +13,20 @@ set(SRCS
   bilinearform.cpp
   bilinearform_ext.cpp
   bilininteg.cpp
-  bilininteg_convection.cpp
-  bilininteg_dgtrace.cpp
-  bilininteg_diffusion.cpp
+  bilininteg_convection_pa.cpp
+  bilininteg_convection_ea.cpp
+  bilininteg_dgtrace_pa.cpp
+  bilininteg_dgtrace_ea.cpp
+  bilininteg_diffusion_pa.cpp
+  bilininteg_diffusion_ea.cpp
   bilininteg_divergence.cpp
   bilininteg_hcurl.cpp
+  bilininteg_hdiv.cpp
+  bilininteg_vectorfe.cpp
   bilininteg_gradient.cpp
-  bilininteg_mass.cpp
+  bilininteg_mass_pa.cpp
+  bilininteg_mass_ea.cpp
+  bilininteg_transpose_ea.cpp
   bilininteg_vecdiffusion.cpp
   bilininteg_vecmass.cpp
   coefficient.cpp

fem/adios2datacollection.cpp

@@ -15,6 +15,8 @@
 
 #include "adios2datacollection.hpp"
 
+#ifdef MFEM_USE_ADIOS2
+
 namespace mfem
 {
 
@@ -87,4 +89,4 @@ noexcept
 
 } //end namespace mfem
 
-
+#endif // MFEM_USE_ADIOS2

fem/adios2datacollection.hpp

@@ -17,6 +17,9 @@
 #define MFEM_ADIOS2DATACOLLECTION
 
 #include "../config/config.hpp"
+
+#ifdef MFEM_USE_ADIOS2
+
 #include "../general/adios2stream.hpp"
 #include "datacollection.hpp"
 
@@ -85,4 +88,6 @@ private:
 
 }  // namespace mfem
 
+#endif // MFEM_USE_ADIOS2
+
 #endif /* MFEM_ADIOS2DATACOLLECTION */

fem/bilinearform.cpp

@@ -126,8 +126,7 @@ void BilinearForm::SetAssemblyLevel(AssemblyLevel assembly_level)
          // Use the original BilinearForm implementation for now
          break;
       case AssemblyLevel::ELEMENT:
-         mfem_error("Element assembly not supported yet... stay tuned!");
-         // ext = new EABilinearFormExtension(this);
+         ext = new EABilinearFormExtension(this);
          break;
       case AssemblyLevel::PARTIAL:
          ext = new PABilinearFormExtension(this);
@@ -1432,6 +1431,54 @@ void MixedBilinearForm::Assemble (int skip_zeros)
    }
 }
 
+void MixedBilinearForm::AssembleDiagonal_ADAt(const Vector &D,
+                                              Vector &diag) const
+{
+   if (ext)
+   {
+      MFEM_ASSERT(diag.Size() == test_fes->GetTrueVSize(),
+                  "Vector for holding diagonal has wrong size!");
+      MFEM_ASSERT(D.Size() == trial_fes->GetTrueVSize(),
+                  "Vector for holding diagonal has wrong size!");
+      const Operator *P_trial = trial_fes->GetProlongationMatrix();
+      const Operator *P_test = test_fes->GetProlongationMatrix();
+      if (!IsIdentityProlongation(P_trial))
+      {
+         Vector local_D(P_trial->Height());
+         P_trial->Mult(D, local_D);
+
+         if (!IsIdentityProlongation(P_test))
+         {
+            Vector local_diag(P_test->Height());
+            ext->AssembleDiagonal_ADAt(local_D, local_diag);
+            P_test->MultTranspose(local_diag, diag);
+         }
+         else
+         {
+            ext->AssembleDiagonal_ADAt(local_D, diag);
+         }
+      }
+      else
+      {
+         if (!IsIdentityProlongation(P_test))
+         {
+            Vector local_diag(P_test->Height());
+            ext->AssembleDiagonal_ADAt(D, local_diag);
+            P_test->MultTranspose(local_diag, diag);
+         }
+         else
+         {
+            ext->AssembleDiagonal_ADAt(D, diag);
+         }
+      }
+   }
+   else
+   {
+      MFEM_ABORT("Not implemented. Maybe assemble your bilinear form into a "
+                 "matrix and use SparseMatrix functions?");
+   }
+}
+
 void MixedBilinearForm::ConformingAssemble()
 {
    if (assembly != AssemblyLevel::FULL)

fem/bilinearform.hpp

@@ -705,6 +705,10 @@ public:
 
    void Assemble(int skip_zeros = 1);
 
+   /** @brief Assemble the diagonal of ADA^T into diag, where A is this mixed
+       bilinear form and D is a diagonal. */
+   void AssembleDiagonal_ADAt(const Vector &D, Vector &diag) const;
+
    /// Get the input finite element space prolongation matrix
    virtual const Operator *GetProlongation() const
    { return trial_fes->GetProlongationMatrix(); }

fem/bilinearform_ext.cpp

@@ -47,7 +47,7 @@ PABilinearFormExtension::PABilinearFormExtension(BilinearForm *form)
    bdr_face_restrict_lex = NULL;
 }
 
-void PABilinearFormExtension::SetupRestrictionOperators()
+void PABilinearFormExtension::SetupRestrictionOperators(const L2FaceValues m)
 {
    ElementDofOrdering ordering = UsesTensorBasis(*a->FESpace())?
                                  ElementDofOrdering::LEXICOGRAPHIC:
@@ -65,7 +65,8 @@ void PABilinearFormExtension::SetupRestrictionOperators()
    if (int_face_restrict_lex == NULL && a->GetFBFI()->Size() > 0)
    {
       int_face_restrict_lex = trialFes->GetFaceRestriction(
-                                 ElementDofOrdering::LEXICOGRAPHIC, FaceType::Interior);
+                                 ElementDofOrdering::LEXICOGRAPHIC,
+                                 FaceType::Interior);
       faceIntX.SetSize(int_face_restrict_lex->Height(), Device::GetMemoryType());
       faceIntY.SetSize(int_face_restrict_lex->Height(), Device::GetMemoryType());
       faceIntY.UseDevice(true); // ensure 'faceIntY = 0.0' is done on device
@@ -74,7 +75,9 @@ void PABilinearFormExtension::SetupRestrictionOperators()
    if (bdr_face_restrict_lex == NULL && a->GetBFBFI()->Size() > 0)
    {
       bdr_face_restrict_lex = trialFes->GetFaceRestriction(
-                                 ElementDofOrdering::LEXICOGRAPHIC, FaceType::Boundary);
+                                 ElementDofOrdering::LEXICOGRAPHIC,
+                                 FaceType::Boundary,
+                                 m);
       faceBdrX.SetSize(bdr_face_restrict_lex->Height(), Device::GetMemoryType());
       faceBdrY.SetSize(bdr_face_restrict_lex->Height(), Device::GetMemoryType());
       faceBdrY.UseDevice(true); // ensure 'faceBoundY = 0.0' is done on device
@@ -83,7 +86,7 @@ void PABilinearFormExtension::SetupRestrictionOperators()
 
 void PABilinearFormExtension::Assemble()
 {
-   SetupRestrictionOperators();
+   SetupRestrictionOperators(L2FaceValues::DoubleValued);
 
    Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
    const int integratorCount = integrators.Size();
@@ -287,6 +290,311 @@ void PABilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
    }
 }
 
+// Data and methods for element-assembled bilinear forms
+EABilinearFormExtension::EABilinearFormExtension(BilinearForm *form)
+   : PABilinearFormExtension(form)
+{
+}
+
+void EABilinearFormExtension::Assemble()
+{
+   SetupRestrictionOperators(L2FaceValues::SingleValued);
+
+   ne = trialFes->GetMesh()->GetNE();
+   elemDofs = trialFes->GetFE(0)->GetDof();
+
+   ea_data.SetSize(ne*elemDofs*elemDofs, Device::GetMemoryType());
+   ea_data.UseDevice(true);
+   ea_data = 0.0;
+
+   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
+   const int integratorCount = integrators.Size();
+   for (int i = 0; i < integratorCount; ++i)
+   {
+      integrators[i]->AssembleEA(*a->FESpace(), ea_data);
+   }
+
+   faceDofs = trialFes ->
+              GetTraceElement(0, trialFes->GetMesh()->GetFaceBaseGeometry(0)) ->
+              GetDof();
+
+   Array<BilinearFormIntegrator*> &intFaceIntegrators = *a->GetFBFI();
+   const int intFaceIntegratorCount = intFaceIntegrators.Size();
+   if (intFaceIntegratorCount>0)
+   {
+      nf_int = trialFes->GetNFbyType(FaceType::Interior);
+      ea_data_int.SetSize(2*nf_int*faceDofs*faceDofs, Device::GetMemoryType());
+      ea_data_ext.SetSize(2*nf_int*faceDofs*faceDofs, Device::GetMemoryType());
+      ea_data_int = 0.0;
+      ea_data_ext = 0.0;
+   }
+   for (int i = 0; i < intFaceIntegratorCount; ++i)
+   {
+      intFaceIntegrators[i]->AssembleEAInteriorFaces(*a->FESpace(),
+                                                     ea_data_int,
+                                                     ea_data_ext);
+   }
+
+   Array<BilinearFormIntegrator*> &bdrFaceIntegrators = *a->GetBFBFI();
+   const int boundFaceIntegratorCount = bdrFaceIntegrators.Size();
+   if (boundFaceIntegratorCount>0)
+   {
+      nf_bdr = trialFes->GetNFbyType(FaceType::Boundary);
+      ea_data_bdr.SetSize(nf_bdr*faceDofs*faceDofs, Device::GetMemoryType());
+      ea_data_bdr = 0.0;
+   }
+   for (int i = 0; i < boundFaceIntegratorCount; ++i)
+   {
+      bdrFaceIntegrators[i]->AssembleEABoundaryFaces(*a->FESpace(),ea_data_bdr);
+   }
+}
+
+void EABilinearFormExtension::Mult(const Vector &x, Vector &y) const
+{
+   // Apply the Element Restriction
+   const bool useRestrict = !DeviceCanUseCeed() && elem_restrict;
+   if (!useRestrict)
+   {
+      y.UseDevice(true); // typically this is a large vector, so store on device
+      y = 0.0;
+   }
+   else
+   {
+      elem_restrict->Mult(x, localX);
+      localY = 0.0;
+   }
+   // Apply the Element Matrices
+   const int NDOFS = elemDofs;
+   auto X = Reshape(useRestrict?localX.Read():x.Read(), NDOFS, ne);
+   auto Y = Reshape(useRestrict?localY.ReadWrite():y.ReadWrite(), NDOFS, ne);
+   auto A = Reshape(ea_data.Read(), NDOFS, NDOFS, ne);
+   MFEM_FORALL(glob_j, ne*NDOFS,
+   {
+      const int e = glob_j/NDOFS;
+      const int j = glob_j%NDOFS;
+      double res = 0.0;
+      for (int i = 0; i < NDOFS; i++)
+      {
+         res += A(i, j, e)*X(i, e);
+      }
+      Y(j, e) += res;
+   });
+   // Apply the Element Restriction transposed
+   if (useRestrict)
+   {
+      elem_restrict->MultTranspose(localY, y);
+   }
+
+   // Treatment of interior faces
+   Array<BilinearFormIntegrator*> &intFaceIntegrators = *a->GetFBFI();
+   const int iFISz = intFaceIntegrators.Size();
+   if (int_face_restrict_lex && iFISz>0)
+   {
+      // Apply the Interior Face Restriction
+      int_face_restrict_lex->Mult(x, faceIntX);
+      if (faceIntX.Size()>0)
+      {
+         faceIntY = 0.0;
+         // Apply the interior face matrices
+         const int NDOFS = faceDofs;
+         auto X = Reshape(faceIntX.Read(), NDOFS, 2, nf_int);
+         auto Y = Reshape(faceIntY.ReadWrite(), NDOFS, 2, nf_int);
+         auto A_int = Reshape(ea_data_int.Read(), NDOFS, NDOFS, 2, nf_int);
+         MFEM_FORALL(glob_j, nf_int*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_int(i, j, 0, f)*X(i, 0, f);
+            }
+            Y(j, 0, f) += res;
+            res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_int(i, j, 1, f)*X(i, 1, f);
+            }
+            Y(j, 1, f) += res;
+         });
+         auto A_ext = Reshape(ea_data_ext.Read(), NDOFS, NDOFS, 2, nf_int);
+         MFEM_FORALL(glob_j, nf_int*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_ext(i, j, 0, f)*X(i, 0, f);
+            }
+            Y(j, 1, f) += res;
+            res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_ext(i, j, 1, f)*X(i, 1, f);
+            }
+            Y(j, 0, f) += res;
+         });
+         // Apply the Interior Face Restriction transposed
+         int_face_restrict_lex->MultTranspose(faceIntY, y);
+      }
+   }
+
+   // Treatment of boundary faces
+   Array<BilinearFormIntegrator*> &bdrFaceIntegrators = *a->GetBFBFI();
+   const int bFISz = bdrFaceIntegrators.Size();
+   if (bdr_face_restrict_lex && bFISz>0)
+   {
+      // Apply the Boundary Face Restriction
+      bdr_face_restrict_lex->Mult(x, faceBdrX);
+      if (faceBdrX.Size()>0)
+      {
+         faceBdrY = 0.0;
+         // Apply the boundary face matrices
+         const int NDOFS = faceDofs;
+         auto X = Reshape(faceBdrX.Read(), NDOFS, nf_bdr);
+         auto Y = Reshape(faceBdrY.ReadWrite(), NDOFS, nf_bdr);
+         auto A = Reshape(ea_data_bdr.Read(), NDOFS, NDOFS, nf_bdr);
+         MFEM_FORALL(glob_j, nf_bdr*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A(i, j, f)*X(i, f);
+            }
+            Y(j, f) += res;
+         });
+         // Apply the Boundary Face Restriction transposed
+         bdr_face_restrict_lex->MultTranspose(faceBdrY, y);
+      }
+   }
+}
+
+void EABilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
+{
+   // Apply the Element Restriction
+   const bool useRestrict = DeviceCanUseCeed() || !elem_restrict;
+   if (!useRestrict)
+   {
+      y.UseDevice(true); // typically this is a large vector, so store on device
+      y = 0.0;
+   }
+   else
+   {
+      elem_restrict->Mult(x, localX);
+      localY = 0.0;
+   }
+   // Apply the Element Matrices transposed
+   const int NDOFS = elemDofs;
+   auto X = Reshape(useRestrict?localX.Read():x.Read(), NDOFS, ne);
+   auto Y = Reshape(useRestrict?localY.ReadWrite():y.ReadWrite(), NDOFS, ne);
+   auto A = Reshape(ea_data.Read(), NDOFS, NDOFS, ne);
+   MFEM_FORALL(glob_j, ne*NDOFS,
+   {
+      const int e = glob_j/NDOFS;
+      const int j = glob_j%NDOFS;
+      double res = 0.0;
+      for (int i = 0; i < NDOFS; i++)
+      {
+         res += A(j, i, e)*X(i, e);
+      }
+      Y(j, e) += res;
+   });
+   // Apply the Element Restriction transposed
+   if (useRestrict)
+   {
+      elem_restrict->MultTranspose(localY, y);
+   }
+
+   // Treatment of interior faces
+   Array<BilinearFormIntegrator*> &intFaceIntegrators = *a->GetFBFI();
+   const int iFISz = intFaceIntegrators.Size();
+   if (int_face_restrict_lex && iFISz>0)
+   {
+      // Apply the Interior Face Restriction
+      int_face_restrict_lex->Mult(x, faceIntX);
+      if (faceIntX.Size()>0)
+      {
+         faceIntY = 0.0;
+         // Apply the interior face matrices transposed
+         const int NDOFS = faceDofs;
+         auto X = Reshape(faceIntX.Read(), NDOFS, 2, nf_int);
+         auto Y = Reshape(faceIntY.ReadWrite(), NDOFS, 2, nf_int);
+         auto A_int = Reshape(ea_data_int.Read(), NDOFS, NDOFS, 2, nf_int);
+         MFEM_FORALL(glob_j, nf_int*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_int(j, i, 0, f)*X(i, 0, f);
+            }
+            Y(j, 0, f) += res;
+            res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_int(j, i, 1, f)*X(i, 1, f);
+            }
+            Y(j, 1, f) += res;
+         });
+         auto A_ext = Reshape(ea_data_ext.Read(), NDOFS, NDOFS, 2, nf_int);
+         MFEM_FORALL(glob_j, nf_int*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_ext(j, i, 0, f)*X(i, 0, f);
+            }
+            Y(j, 1, f) += res;
+            res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_ext(j, i, 1, f)*X(i, 1, f);
+            }
+            Y(j, 0, f) += res;
+         });
+         // Apply the Interior Face Restriction transposed
+         int_face_restrict_lex->MultTranspose(faceIntY, y);
+      }
+   }
+
+   // Treatment of boundary faces
+   Array<BilinearFormIntegrator*> &bdrFaceIntegrators = *a->GetBFBFI();
+   const int bFISz = bdrFaceIntegrators.Size();
+   if (bdr_face_restrict_lex && bFISz>0)
+   {
+      // Apply the Boundary Face Restriction
+      bdr_face_restrict_lex->Mult(x, faceBdrX);
+      if (faceBdrX.Size()>0)
+      {
+         faceBdrY = 0.0;
+         // Apply the boundary face matrices transposed
+         const int NDOFS = faceDofs;
+         auto X = Reshape(faceBdrX.Read(), NDOFS, nf_bdr);
+         auto Y = Reshape(faceBdrY.ReadWrite(), NDOFS, nf_bdr);
+         auto A = Reshape(ea_data_bdr.Read(), NDOFS, NDOFS, nf_bdr);
+         MFEM_FORALL(glob_j, nf_bdr*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A(j, i, f)*X(i, f);
+            }
+            Y(j, f) += res;
+         });
+         // Apply the Boundary Face Restriction transposed
+         bdr_face_restrict_lex->MultTranspose(faceBdrY, y);
+      }
+   }
+}
+
 MixedBilinearFormExtension::MixedBilinearFormExtension(MixedBilinearForm *form)
    : Operator(form->Height(), form->Width()), a(form)
 {
@@ -487,4 +795,68 @@ void PAMixedBilinearFormExtension::AddMultTranspose(const Vector &x, Vector &y,
    }
 }
 
+void PAMixedBilinearFormExtension::AssembleDiagonal_ADAt(const Vector &D,
+                                                         Vector &diag) const
+{
+   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
+
+   const int iSz = integrators.Size();
+
+   if (elem_restrict_trial)
+   {
+      const ElementRestriction* H1elem_restrict_trial =
+         dynamic_cast<const ElementRestriction*>(elem_restrict_trial);
+      if (H1elem_restrict_trial)
+      {
+         H1elem_restrict_trial->MultUnsigned(D, localTrial);
+      }
+      else
+      {
+         elem_restrict_trial->Mult(D, localTrial);
+      }
+   }
+
+   if (elem_restrict_test)
+   {
+      localTest = 0.0;
+      for (int i = 0; i < iSz; ++i)
+      {
+         if (elem_restrict_trial)
+         {
+            integrators[i]->AssembleDiagonalPA_ADAt(localTrial, localTest);
+         }
+         else
+         {
+            integrators[i]->AssembleDiagonalPA_ADAt(D, localTest);
+         }
+      }
+      const ElementRestriction* H1elem_restrict_test =
+         dynamic_cast<const ElementRestriction*>(elem_restrict_test);
+      if (H1elem_restrict_test)
+      {
+         H1elem_restrict_test->MultTransposeUnsigned(localTest, diag);
+      }
+      else
+      {
+         elem_restrict_test->MultTranspose(localTest, diag);
+      }
+   }
+   else
+   {
+      diag.UseDevice(true); // typically this is a large vector, so store on device
+      diag = 0.0;
+      for (int i = 0; i < iSz; ++i)
+      {
+         if (elem_restrict_trial)
+         {
+            integrators[i]->AssembleDiagonalPA_ADAt(localTrial, diag);
+         }
+         else
+         {
+            integrators[i]->AssembleDiagonalPA_ADAt(D, diag);
+         }
+      }
+   }
+}
+
 } // namespace mfem

fem/bilinearform_ext.hpp

@@ -78,26 +78,6 @@ public:
    ~FABilinearFormExtension() {}
 };
 
-/// Data and methods for element-assembled bilinear forms
-class EABilinearFormExtension : public BilinearFormExtension
-{
-public:
-   EABilinearFormExtension(BilinearForm *form)
-      : BilinearFormExtension(form) { }
-
-   /// TODO
-   void Assemble() {}
-   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A) {}
-   void FormLinearSystem(const Array<int> &ess_tdof_list,
-                         Vector &x, Vector &b,
-                         OperatorHandle &A, Vector &X, Vector &B,
-                         int copy_interior = 0) {}
-   void Mult(const Vector &x, Vector &y) const {}
-   void MultTranspose(const Vector &x, Vector &y) const {}
-   void Update() {}
-   ~EABilinearFormExtension() {}
-};
-
 /// Data and methods for partially-assembled bilinear forms
 class PABilinearFormExtension : public BilinearFormExtension
 {
@@ -113,7 +93,6 @@ protected:
 public:
    PABilinearFormExtension(BilinearForm*);
 
-   void SetupRestrictionOperators();
    void Assemble();
    void AssembleDiagonal(Vector &diag) const;
    void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A);
@@ -121,12 +100,32 @@ public:
                          Vector &x, Vector &b,
                          OperatorHandle &A, Vector &X, Vector &B,
                          int copy_interior = 0);
-
    void Mult(const Vector &x, Vector &y) const;
    void MultTranspose(const Vector &x, Vector &y) const;
    void Update();
+
+protected:
+   void SetupRestrictionOperators(const L2FaceValues m);
 };
 
+/// Data and methods for element-assembled bilinear forms
+class EABilinearFormExtension : public PABilinearFormExtension
+{
+protected:
+   int ne;
+   int elemDofs;
+   Vector ea_data;
+   int nf_int, nf_bdr;
+   int faceDofs;
+   Vector ea_data_int, ea_data_ext, ea_data_bdr;
+
+public:
+   EABilinearFormExtension(BilinearForm *form);
+
+   void Assemble();
+   void Mult(const Vector &x, Vector &y) const;
+   void MultTranspose(const Vector &x, Vector &y) const;
+};
 
 /// Data and methods for matrix-free bilinear forms
 class MFBilinearFormExtension : public BilinearFormExtension
@@ -186,6 +185,8 @@ public:
    virtual void AddMultTranspose(const Vector &x, Vector &y,
                                  const double c=1.0) const = 0;
 
+   virtual void AssembleDiagonal_ADAt(const Vector &D, Vector &diag) const = 0;
+
    virtual void Update() = 0;
 };
 
@@ -236,6 +237,9 @@ public:
    void MultTranspose(const Vector &x, Vector &y) const;
    /// y += c*A^T*x
    void AddMultTranspose(const Vector &x, Vector &y, const double c=1.0) const;
+   /// Assemble the diagonal of ADA^T for a diagonal vector D.
+   void AssembleDiagonal_ADAt(const Vector &D, Vector &diag) const;
+
    /// Update internals for when a new MixedBilinearForm is given to this class
    void Update();
 };

fem/bilininteg.cpp

@@ -47,7 +47,37 @@ void BilinearFormIntegrator::AssemblePABoundaryFaces(const FiniteElementSpace&)
 
 void BilinearFormIntegrator::AssembleDiagonalPA(Vector &)
 {
-   MFEM_ABORT("BilinearFormIntegrator::AssembleDiagonalPA(...)\n"
+   mfem_error ("BilinearFormIntegrator::AssembleDiagonalPA(...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::AssembleEA(const FiniteElementSpace &fes,
+                                        Vector &emat)
+{
+   mfem_error ("BilinearFormIntegrator::AssembleEA(...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::AssembleEAInteriorFaces(const FiniteElementSpace
+                                                     &fes,
+                                                     Vector &ea_data_int,
+                                                     Vector &ea_data_ext)
+{
+   mfem_error ("BilinearFormIntegrator::AssembleEAInteriorFaces(...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::AssembleEABoundaryFaces(const FiniteElementSpace
+                                                     &fes,
+                                                     Vector &ea_data_bdr)
+{
+   mfem_error ("BilinearFormIntegrator::AssembleEABoundaryFaces(...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::AssembleDiagonalPA_ADAt(const Vector &, Vector &)
+{
+   MFEM_ABORT("BilinearFormIntegrator::AssembleDiagonalPA_ADAt(...)\n"
               "   is not implemented for this class.");
 }
 
@@ -889,7 +919,7 @@ void BoundaryMassIntegrator::AssembleFaceMatrix(
    {
       int order = 2 * el1.GetOrder();
 
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    elmat = 0.0;
@@ -900,11 +930,11 @@ void BoundaryMassIntegrator::AssembleFaceMatrix(
       Trans.Loc1.Transform(ip, eip);
       el1.CalcShape(eip, shape);
 
-      Trans.Face->SetIntPoint(&ip);
-      w = Trans.Face->Weight() * ip.weight;
+      Trans.SetIntPoint(&ip);
+      w = Trans.Weight() * ip.weight;
       if (Q)
       {
-         w *= Q -> Eval(*Trans.Face, ip);
+         w *= Q -> Eval(Trans, ip);
       }
 
       AddMult_a_VVt(w, shape, elmat);
@@ -1974,7 +2004,7 @@ void VectorFEMassIntegrator::AssembleElementMatrix2(
       D.SetSize(VQ ? VQ->GetVDim() : 0);
       K.SetSize(MQ ? MQ->GetVDim() : 0, MQ ? MQ->GetVDim() : 0);
 #endif
-      DenseMatrix tmp(trial_vshape.Height(), K.Width());
+      DenseMatrix tmp(test_vshape.Height(), K.Width());
 
       elmat.SetSize (test_dof, trial_dof);
 
@@ -2535,7 +2565,7 @@ void DGTraceIntegrator::AssembleFaceMatrix(const FiniteElement &el1,
       {
          order++;
       }
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    for (int p = 0; p < ir->GetNPoints(); p++)
@@ -2549,8 +2579,7 @@ void DGTraceIntegrator::AssembleFaceMatrix(const FiniteElement &el1,
       }
       el1.CalcShape(eip1, shape1);
 
-      Trans.Face->SetIntPoint(&ip);
-      Trans.Elem1->SetIntPoint(&eip1);
+      Trans.SetIntPoint(&ip);
 
       u->Eval(vu, *Trans.Elem1, eip1);
 
@@ -2560,7 +2589,7 @@ void DGTraceIntegrator::AssembleFaceMatrix(const FiniteElement &el1,
       }
       else
       {
-         CalcOrtho(Trans.Face->Jacobian(), nor);
+         CalcOrtho(Trans.Jacobian(), nor);
       }
 
       un = vu * nor;
@@ -2575,7 +2604,6 @@ void DGTraceIntegrator::AssembleFaceMatrix(const FiniteElement &el1,
          double rho_p;
          if (un >= 0.0 && ndof2)
          {
-            Trans.Elem2->SetIntPoint(&eip2);
             rho_p = rho->Eval(*Trans.Elem2, eip2);
          }
          else
@@ -2691,7 +2719,7 @@ void DGDiffusionIntegrator::AssembleFaceMatrix(
       {
          order = 2*el1.GetOrder();
       }
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    // assemble: < {(Q \nabla u).n},[v] >      --> elmat
@@ -2702,19 +2730,18 @@ void DGDiffusionIntegrator::AssembleFaceMatrix(
       IntegrationPoint eip1, eip2;
 
       Trans.Loc1.Transform(ip, eip1);
-      Trans.Face->SetIntPoint(&ip);
+      Trans.SetIntPoint(&ip);
       if (dim == 1)
       {
          nor(0) = 2*eip1.x - 1.0;
       }
       else
       {
-         CalcOrtho(Trans.Face->Jacobian(), nor);
+         CalcOrtho(Trans.Jacobian(), nor);
       }
 
       el1.CalcShape(eip1, shape1);
       el1.CalcDShape(eip1, dshape1);
-      Trans.Elem1->SetIntPoint(&eip1);
       w = ip.weight/Trans.Elem1->Weight();
       if (ndof2)
       {
@@ -2763,7 +2790,6 @@ void DGDiffusionIntegrator::AssembleFaceMatrix(
          Trans.Loc2.Transform(ip, eip2);
          el2.CalcShape(eip2, shape2);
          el2.CalcDShape(eip2, dshape2);
-         Trans.Elem2->SetIntPoint(&eip2);
          w = ip.weight/2/Trans.Elem2->Weight();
          if (!MQ)
          {
@@ -2973,7 +2999,7 @@ void DGElasticityIntegrator::AssembleFaceMatrix(
    {
       // a simple choice for the integration order; is this OK?
       const int order = 2 * max(el1.GetOrder(), ndofs2 ? el2.GetOrder() : 0);
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    for (int pind = 0; pind < ir->GetNPoints(); ++pind)
@@ -2981,8 +3007,7 @@ void DGElasticityIntegrator::AssembleFaceMatrix(
       const IntegrationPoint &ip = ir->IntPoint(pind);
       IntegrationPoint eip1, eip2; // integration point in the reference space
       Trans.Loc1.Transform(ip, eip1);
-      Trans.Face->SetIntPoint(&ip);
-      Trans.Elem1->SetIntPoint(&eip1);
+      Trans.SetIntPoint(&ip);
 
       el1.CalcShape(eip1, shape1);
       el1.CalcDShape(eip1, dshape1);
@@ -2996,14 +3021,13 @@ void DGElasticityIntegrator::AssembleFaceMatrix(
       }
       else
       {
-         CalcOrtho(Trans.Face->Jacobian(), nor);
+         CalcOrtho(Trans.Jacobian(), nor);
       }
 
       double w, wLM;
       if (ndofs2)
       {
          Trans.Loc2.Transform(ip, eip2);
-         Trans.Elem2->SetIntPoint(&eip2);
          el2.CalcShape(eip2, shape2);
          el2.CalcDShape(eip2, dshape2);
          CalcAdjugate(Trans.Elem2->Jacobian(), adjJ);
@@ -3133,9 +3157,9 @@ void TraceJumpIntegrator::AssembleFaceMatrix(
       order += trial_face_fe.GetOrder();
       if (trial_face_fe.GetMapType() == FiniteElement::VALUE)
       {
-         order += Trans.Face->OrderW();
+         order += Trans.OrderW();
       }
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    for (int p = 0; p < ir->GetNPoints(); p++)
@@ -3143,23 +3167,21 @@ void TraceJumpIntegrator::AssembleFaceMatrix(
       const IntegrationPoint &ip = ir->IntPoint(p);
       IntegrationPoint eip1, eip2;
       // Trace finite element shape function
-      Trans.Face->SetIntPoint(&ip);
+      Trans.SetIntPoint(&ip);
       trial_face_fe.CalcShape(ip, face_shape);
       // Side 1 finite element shape function
       Trans.Loc1.Transform(ip, eip1);
       test_fe1.CalcShape(eip1, shape1);
-      Trans.Elem1->SetIntPoint(&eip1);
       if (ndof2)
       {
          // Side 2 finite element shape function
          Trans.Loc2.Transform(ip, eip2);
          test_fe2.CalcShape(eip2, shape2);
-         Trans.Elem2->SetIntPoint(&eip2);
       }
       w = ip.weight;
       if (trial_face_fe.GetMapType() == FiniteElement::VALUE)
       {
-         w *= Trans.Face->Weight();
+         w *= Trans.Weight();
       }
       face_shape *= w;
       for (i = 0; i < ndof1; i++)
@@ -3224,7 +3246,7 @@ void NormalTraceJumpIntegrator::AssembleFaceMatrix(
          order = test_fe1.GetOrder() - 1;
       }
       order += trial_face_fe.GetOrder();
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    for (int p = 0; p < ir->GetNPoints(); p++)

fem/bilininteg.hpp

@@ -57,6 +57,9 @@ public:
    /// Assemble diagonal and add it to Vector @a diag.
    virtual void AssembleDiagonalPA(Vector &diag);
 
+   /// Assemble diagonal of ADA^T (A is this integrator) and add it to @a diag.
+   virtual void AssembleDiagonalPA_ADAt(const Vector &D, Vector &diag);
+
    /// Method for partially assembled action.
    /** Perform the action of integrator on the input @a x and add the result to
        the output @a y. Both @a x and @a y are E-vectors, i.e. they represent
@@ -75,6 +78,22 @@ public:
        called. */
    virtual void AddMultTransposePA(const Vector &x, Vector &y) const;
 
+   /// Method defining element assembly.
+   /** The result of the element assembly is added and stored in the @a emat
+       Vector. */
+   virtual void AssembleEA(const FiniteElementSpace &fes, Vector &emat);
+   /** Used with BilinearFormIntegrators that have different spaces. */
+   // virtual void AssembleEA(const FiniteElementSpace &trial_fes,
+   //                         const FiniteElementSpace &test_fes,
+   //                         Vector &emat);
+
+   virtual void AssembleEAInteriorFaces(const FiniteElementSpace &fes,
+                                        Vector &ea_data_int,
+                                        Vector &ea_data_ext);
+
+   virtual void AssembleEABoundaryFaces(const FiniteElementSpace &fes,
+                                        Vector &ea_data_bdr);
+
    /// Given a particular Finite Element computes the element matrix elmat.
    virtual void AssembleElementMatrix(const FiniteElement &el,
                                       ElementTransformation &Trans,
@@ -234,6 +253,15 @@ public:
       bfi->AddMultTransposePA(x, y);
    }
 
+   virtual void AssembleEA(const FiniteElementSpace &fes, Vector &emat);
+
+   virtual void AssembleEAInteriorFaces(const FiniteElementSpace &fes,
+                                        Vector &ea_data_int,
+                                        Vector &ea_data_ext);
+
+   virtual void AssembleEABoundaryFaces(const FiniteElementSpace &fes,
+                                        Vector &ea_data_bdr);
+
    virtual ~TransposeIntegrator() { if (own_bfi) { delete bfi; } }
 };
 
@@ -1885,6 +1913,8 @@ public:
 
    virtual void AssemblePA(const FiniteElementSpace &fes);
 
+   virtual void AssembleEA(const FiniteElementSpace &fes, Vector &emat);
+
    virtual void AssembleDiagonalPA(Vector &diag);
 
    virtual void AddMultPA(const Vector&, Vector&) const;
@@ -1958,6 +1988,8 @@ public:
 
    virtual void AssemblePA(const FiniteElementSpace &fes);
 
+   virtual void AssembleEA(const FiniteElementSpace &fes, Vector &emat);
+
    virtual void AssembleDiagonalPA(Vector &diag);
 
    virtual void AddMultPA(const Vector&, Vector&) const;
@@ -2011,6 +2043,8 @@ public:
 
    virtual void AssemblePA(const FiniteElementSpace&);
 
+   virtual void AssembleEA(const FiniteElementSpace &fes, Vector &emat);
+
    virtual void AddMultPA(const Vector&, Vector&) const;
 
    static const IntegrationRule &GetRule(const FiniteElement &el,
@@ -2110,11 +2144,25 @@ class VectorFEDivergenceIntegrator : public BilinearFormIntegrator
 protected:
    Coefficient *Q;
 
+   using BilinearFormIntegrator::AssemblePA;
+   virtual void AssemblePA(const FiniteElementSpace &trial_fes,
+                           const FiniteElementSpace &test_fes);
+
+   virtual void AddMultPA(const Vector&, Vector&) const;
+   virtual void AddMultTransposePA(const Vector&, Vector&) const;
+
 private:
 #ifndef MFEM_THREAD_SAFE
    Vector divshape, shape;
 #endif
 
+   // PA extension
+   Vector pa_data;
+   const DofToQuad *mapsO;         ///< Not owned. DOF-to-quad map, open.
+   const DofToQuad *L2mapsO;       ///< Not owned. DOF-to-quad map, open.
+   const DofToQuad *mapsC;         ///< Not owned. DOF-to-quad map, closed.
+   int dim, ne, dofs1D, L2dofs1D, quad1D;
+
 public:
    VectorFEDivergenceIntegrator() { Q = NULL; }
    VectorFEDivergenceIntegrator(Coefficient &q) { Q = &q; }
@@ -2125,6 +2173,8 @@ public:
                                        const FiniteElement &test_fe,
                                        ElementTransformation &Trans,
                                        DenseMatrix &elmat);
+
+   virtual void AssembleDiagonalPA_ADAt(const Vector &D, Vector &diag);
 };
 
 
@@ -2308,7 +2358,7 @@ protected:
    const DofToQuad *mapsO;         ///< Not owned. DOF-to-quad map, open.
    const DofToQuad *mapsC;         ///< Not owned. DOF-to-quad map, closed.
    const GeometricFactors *geom;   ///< Not owned
-   int dim, ne, nq, dofs1D, quad1D;
+   int dim, ne, nq, dofs1D, quad1D, fetype;
 
 public:
    VectorFEMassIntegrator() { Init(NULL, NULL, NULL); }
@@ -2387,11 +2437,23 @@ class DivDivIntegrator: public BilinearFormIntegrator
 protected:
    Coefficient *Q;
 
+   using BilinearFormIntegrator::AssemblePA;
+   virtual void AssemblePA(const FiniteElementSpace &fes);
+   virtual void AddMultPA(const Vector &x, Vector &y) const;
+   virtual void AssembleDiagonalPA(Vector& diag);
+
 private:
 #ifndef MFEM_THREAD_SAFE
    Vector divshape;
 #endif
 
+   // PA extension
+   Vector pa_data;
+   const DofToQuad *mapsO;         ///< Not owned. DOF-to-quad map, open.
+   const DofToQuad *mapsC;         ///< Not owned. DOF-to-quad map, closed.
+   const GeometricFactors *geom;   ///< Not owned
+   int dim, ne, dofs1D, quad1D;
+
 public:
    DivDivIntegrator() { Q = NULL; }
    DivDivIntegrator(Coefficient &q) : Q(&q) { }
@@ -2544,6 +2606,13 @@ public:
 
    virtual void AddMultPA(const Vector&, Vector&) const;
 
+   virtual void AssembleEAInteriorFaces(const FiniteElementSpace& fes,
+                                        Vector &ea_data_int,
+                                        Vector &ea_data_ext);
+
+   virtual void AssembleEABoundaryFaces(const FiniteElementSpace& fes,
+                                        Vector &ea_data_bdr);
+
    static const IntegrationRule &GetRule(Geometry::Type geom, int order,
                                          FaceElementTransformations &T);
 

fem/bilininteg_convection_ea.cpp

@@ -0,0 +1,258 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "../general/forall.hpp"
+#include "bilininteg.hpp"
+#include "gridfunc.hpp"
+
+namespace mfem
+{
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAConvectionAssemble1D(const int NE,
+                                   const Array<double> &b,
+                                   const Array<double> &g,
+                                   const Vector &padata,
+                                   Vector &eadata,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(b.Read(), Q1D, D1D);
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_Gi[MQ1];
+      double r_Bj[MQ1];
+      for (int q = 0; q < Q1D; q++)
+      {
+         r_Gi[q] = G(q,MFEM_THREAD_ID(x));
+         r_Bj[q] = B(q,MFEM_THREAD_ID(y));
+      }
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(j1,y,D1D)
+         {
+            double val = 0.0;
+            for (int k1 = 0; k1 < Q1D; ++k1)
+            {
+               val += r_Bj[k1] * D(k1, e) * r_Gi[k1];
+            }
+            A(i1, j1, e) = val;
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAConvectionAssemble2D(const int NE,
+                                   const Array<double> &b,
+                                   const Array<double> &g,
+                                   const Vector &padata,
+                                   Vector &eadata,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(b.Read(), Q1D, D1D);
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, 2, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      double r_G[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+            r_G[q][d] = G(q,d);
+         }
+      }
+      MFEM_SHARED double s_D[MQ1][MQ1][2];
+      MFEM_FOREACH_THREAD(k1,x,Q1D)
+      {
+         MFEM_FOREACH_THREAD(k2,y,Q1D)
+         {
+            s_D[k1][k2][0] = D(k1,k2,0,e);
+            s_D[k1][k2][1] = D(k1,k2,1,e);
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            for (int j1 = 0; j1 < D1D; ++j1)
+            {
+               for (int j2 = 0; j2 < D1D; ++j2)
+               {
+                  double val = 0.0;
+                  for (int k1 = 0; k1 < Q1D; ++k1)
+                  {
+                     for (int k2 = 0; k2 < Q1D; ++k2)
+                     {
+                        val += (r_G[k1][i1] * r_B[k2][i2] * s_D[k1][k2][0]
+                                + r_B[k1][i1] * r_G[k2][i2] * s_D[k1][k2][1])
+                               * r_B[k1][j1]* r_B[k2][j2];
+                     }
+                  }
+                  A(i1, i2, j1, j2, e) = val;
+               }
+            }
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAConvectionAssemble3D(const int NE,
+                                   const Array<double> &b,
+                                   const Array<double> &g,
+                                   const Vector &padata,
+                                   Vector &eadata,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(b.Read(), Q1D, D1D);
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, Q1D, 3, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, D1D,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      double r_G[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+            r_G[q][d] = G(q,d);
+         }
+      }
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            MFEM_FOREACH_THREAD(i3,z,D1D)
+            {
+               for (int j1 = 0; j1 < D1D; ++j1)
+               {
+                  for (int j2 = 0; j2 < D1D; ++j2)
+                  {
+                     for (int j3 = 0; j3 < D1D; ++j3)
+                     {
+                        double val = 0.0;
+                        for (int k1 = 0; k1 < Q1D; ++k1)
+                        {
+                           for (int k2 = 0; k2 < Q1D; ++k2)
+                           {
+                              for (int k3 = 0; k3 < Q1D; ++k3)
+                              {
+                                 double D0 = D(k1,k2,k3,0,e);
+                                 double D1 = D(k1,k2,k3,1,e);
+                                 double D2 = D(k1,k2,k3,2,e);
+                                 val += (r_G[k1][i1] * r_B[k2][i2] * r_B[k3][i3] * D0
+                                         + r_B[k1][i1] * r_G[k2][i2] * r_B[k3][i3] * D1
+                                         + r_B[k1][i1] * r_B[k2][i2] * r_G[k3][i3] * D2)
+                                        * r_B[k1][j1] * r_B[k2][j2] * r_B[k3][j3];
+                              }
+                           }
+                        }
+                        A(i1, i2, i3, j1, j2, j3, e) = val;
+                     }
+                  }
+               }
+            }
+         }
+      }
+   });
+}
+
+void ConvectionIntegrator::AssembleEA(const FiniteElementSpace &fes,
+                                      Vector &ea_data)
+{
+   AssemblePA(fes);
+   const int ne = fes.GetMesh()->GetNE();
+   const Array<double> &B = maps->B;
+   const Array<double> &G = maps->G;
+   if (dim == 1)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EAConvectionAssemble1D<2,2>(ne,B,G,pa_data,ea_data);
+         case 0x33: return EAConvectionAssemble1D<3,3>(ne,B,G,pa_data,ea_data);
+         case 0x44: return EAConvectionAssemble1D<4,4>(ne,B,G,pa_data,ea_data);
+         case 0x55: return EAConvectionAssemble1D<5,5>(ne,B,G,pa_data,ea_data);
+         case 0x66: return EAConvectionAssemble1D<6,6>(ne,B,G,pa_data,ea_data);
+         case 0x77: return EAConvectionAssemble1D<7,7>(ne,B,G,pa_data,ea_data);
+         case 0x88: return EAConvectionAssemble1D<8,8>(ne,B,G,pa_data,ea_data);
+         case 0x99: return EAConvectionAssemble1D<9,9>(ne,B,G,pa_data,ea_data);
+         default:   return EAConvectionAssemble1D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 2)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EAConvectionAssemble2D<2,2>(ne,B,G,pa_data,ea_data);
+         case 0x33: return EAConvectionAssemble2D<3,3>(ne,B,G,pa_data,ea_data);
+         case 0x44: return EAConvectionAssemble2D<4,4>(ne,B,G,pa_data,ea_data);
+         case 0x55: return EAConvectionAssemble2D<5,5>(ne,B,G,pa_data,ea_data);
+         case 0x66: return EAConvectionAssemble2D<6,6>(ne,B,G,pa_data,ea_data);
+         case 0x77: return EAConvectionAssemble2D<7,7>(ne,B,G,pa_data,ea_data);
+         case 0x88: return EAConvectionAssemble2D<8,8>(ne,B,G,pa_data,ea_data);
+         case 0x99: return EAConvectionAssemble2D<9,9>(ne,B,G,pa_data,ea_data);
+         default:   return EAConvectionAssemble2D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x23: return EAConvectionAssemble3D<2,3>(ne,B,G,pa_data,ea_data);
+         case 0x34: return EAConvectionAssemble3D<3,4>(ne,B,G,pa_data,ea_data);
+         case 0x45: return EAConvectionAssemble3D<4,5>(ne,B,G,pa_data,ea_data);
+         case 0x56: return EAConvectionAssemble3D<5,6>(ne,B,G,pa_data,ea_data);
+         case 0x67: return EAConvectionAssemble3D<6,7>(ne,B,G,pa_data,ea_data);
+         case 0x78: return EAConvectionAssemble3D<7,8>(ne,B,G,pa_data,ea_data);
+         case 0x89: return EAConvectionAssemble3D<8,9>(ne,B,G,pa_data,ea_data);
+         default:   return EAConvectionAssemble3D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
+}

fem/bilininteg_dgtrace_ea.cpp

@@ -0,0 +1,414 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "../general/forall.hpp"
+#include "bilininteg.hpp"
+#include "gridfunc.hpp"
+
+namespace mfem
+{
+
+static void EADGTraceAssemble1DInt(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_int,
+                                   Vector &eadata_ext)
+{
+   auto D = Reshape(padata.Read(), 2, 2, NF);
+   auto A_int = Reshape(eadata_int.ReadWrite(), 2, NF);
+   auto A_ext = Reshape(eadata_ext.ReadWrite(), 2, NF);
+   MFEM_FORALL(f, NF,
+   {
+      double val_int0, val_int1, val_ext01, val_ext10;
+      val_int0  = D(0, 0, f);
+      val_ext10 = D(1, 0, f);
+      val_ext01 = D(0, 1, f);
+      val_int1  = D(1, 1, f);
+      A_int(0, f) += val_int0;
+      A_int(1, f) += val_int1;
+      A_ext(0, f) += val_ext01;
+      A_ext(1, f) += val_ext10;
+   });
+}
+
+static void EADGTraceAssemble1DBdr(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_bdr)
+{
+   auto D = Reshape(padata.Read(), 2, 2, NF);
+   auto A_bdr = Reshape(eadata_bdr.ReadWrite(), NF);
+   MFEM_FORALL(f, NF,
+   {
+      A_bdr(f) += D(0, 0, f);
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADGTraceAssemble2DInt(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_int,
+                                   Vector &eadata_ext,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, 2, 2, NF);
+   auto A_int = Reshape(eadata_int.ReadWrite(), D1D, D1D, 2, NF);
+   auto A_ext = Reshape(eadata_ext.ReadWrite(), D1D, D1D, 2, NF);
+   MFEM_FORALL_3D(f, NF, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(j1,y,D1D)
+         {
+            double val_int0 = 0.0;
+            double val_int1 = 0.0;
+            double val_ext01 = 0.0;
+            double val_ext10 = 0.0;
+            for (int k1 = 0; k1 < Q1D; ++k1)
+            {
+               val_int0  += B(k1,i1) * B(k1,j1) * D(k1, 0, 0, f);
+               val_ext01 += B(k1,i1) * B(k1,j1) * D(k1, 0, 1, f);
+               val_ext10 += B(k1,i1) * B(k1,j1) * D(k1, 1, 0, f);
+               val_int1  += B(k1,i1) * B(k1,j1) * D(k1, 1, 1, f);
+            }
+            A_int(i1, j1, 0, f) += val_int0;
+            A_int(i1, j1, 1, f) += val_int1;
+            A_ext(i1, j1, 0, f) += val_ext01;
+            A_ext(i1, j1, 1, f) += val_ext10;
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADGTraceAssemble2DBdr(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_bdr,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, 2, 2, NF);
+   auto A_bdr = Reshape(eadata_bdr.ReadWrite(), D1D, D1D, NF);
+   MFEM_FORALL_3D(f, NF, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(j1,y,D1D)
+         {
+            double val_bdr = 0.0;
+            for (int k1 = 0; k1 < Q1D; ++k1)
+            {
+               val_bdr  += B(k1,i1) * B(k1,j1) * D(k1, 0, 0, f);
+            }
+            A_bdr(i1, j1, f) += val_bdr;
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADGTraceAssemble3DInt(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_int,
+                                   Vector &eadata_ext,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, 2, 2, NF);
+   auto A_int = Reshape(eadata_int.ReadWrite(), D1D, D1D, D1D, D1D, 2, NF);
+   auto A_ext = Reshape(eadata_ext.ReadWrite(), D1D, D1D, D1D, D1D, 2, NF);
+   MFEM_FORALL_3D(f, NF, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+         }
+      }
+      MFEM_SHARED double s_D[MQ1][MQ1][2][2];
+      for (int i=0; i < 2; i++)
+      {
+         for (int j=0; j < 2; j++)
+         {
+            MFEM_FOREACH_THREAD(k1,x,Q1D)
+            {
+               MFEM_FOREACH_THREAD(k2,y,Q1D)
+               {
+                  s_D[k1][k2][i][j] = D(k1,k2,i,j,f);
+               }
+            }
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            for (int j1 = 0; j1 < D1D; ++j1)
+            {
+               for (int j2 = 0; j2 < D1D; ++j2)
+               {
+                  double val_int0 = 0.0;
+                  double val_int1 = 0.0;
+                  double val_ext01 = 0.0;
+                  double val_ext10 = 0.0;
+                  for (int k1 = 0; k1 < Q1D; ++k1)
+                  {
+                     for (int k2 = 0; k2 < Q1D; ++k2)
+                     {
+                        val_int0 += r_B[k1][i1] * r_B[k1][j1]
+                                    * r_B[k2][i2] * r_B[k2][j2]
+                                    * s_D[k1][k2][0][0];
+                        val_int1 += r_B[k1][i1] * r_B[k1][j1]
+                                    * r_B[k2][i2] * r_B[k2][j2]
+                                    * s_D[k1][k2][1][1];
+                        val_ext01+= r_B[k1][i1] * r_B[k1][j1]
+                                    * r_B[k2][i2] * r_B[k2][j2]
+                                    * s_D[k1][k2][0][1];
+                        val_ext10+= r_B[k1][i1] * r_B[k1][j1]
+                                    * r_B[k2][i2] * r_B[k2][j2]
+                                    * s_D[k1][k2][1][0];
+                     }
+                  }
+                  A_int(i1, i2, j1, j2, 0, f) += val_int0;
+                  A_int(i1, i2, j1, j2, 1, f) += val_int1;
+                  A_ext(i1, i2, j1, j2, 0, f) += val_ext01;
+                  A_ext(i1, i2, j1, j2, 1, f) += val_ext10;
+               }
+            }
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADGTraceAssemble3DBdr(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_bdr,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, 2, 2, NF);
+   auto A_bdr = Reshape(eadata_bdr.ReadWrite(), D1D, D1D, D1D, D1D, NF);
+   MFEM_FORALL_3D(f, NF, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+         }
+      }
+      MFEM_SHARED double s_D[MQ1][MQ1][2][2];
+      for (int i=0; i < 2; i++)
+      {
+         for (int j=0; j < 2; j++)
+         {
+            MFEM_FOREACH_THREAD(k1,x,Q1D)
+            {
+               MFEM_FOREACH_THREAD(k2,y,Q1D)
+               {
+                  s_D[k1][k2][i][j] = D(k1,k2,i,j,f);
+               }
+            }
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            for (int j1 = 0; j1 < D1D; ++j1)
+            {
+               for (int j2 = 0; j2 < D1D; ++j2)
+               {
+                  double val_bdr = 0.0;
+                  for (int k1 = 0; k1 < Q1D; ++k1)
+                  {
+                     for (int k2 = 0; k2 < Q1D; ++k2)
+                     {
+                        val_bdr += r_B[k1][i1] * r_B[k1][j1]
+                                   * r_B[k2][i2] * r_B[k2][j2]
+                                   * s_D[k1][k2][0][0];
+                     }
+                  }
+                  A_bdr(i1, i2, j1, j2, f) += val_bdr;
+               }
+            }
+         }
+      }
+   });
+}
+
+void DGTraceIntegrator::AssembleEAInteriorFaces(const FiniteElementSpace& fes,
+                                                Vector &ea_data_int,
+                                                Vector &ea_data_ext)
+{
+   SetupPA(fes, FaceType::Interior);
+   nf = fes.GetNFbyType(FaceType::Interior);
+   if (nf==0) { return; }
+   const Array<double> &B = maps->B;
+   if (dim == 1)
+   {
+      return EADGTraceAssemble1DInt(nf,B,pa_data,ea_data_int,ea_data_ext);
+   }
+   else if (dim == 2)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22:
+            return EADGTraceAssemble2DInt<2,2>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x33:
+            return EADGTraceAssemble2DInt<3,3>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x44:
+            return EADGTraceAssemble2DInt<4,4>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x55:
+            return EADGTraceAssemble2DInt<5,5>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x66:
+            return EADGTraceAssemble2DInt<6,6>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x77:
+            return EADGTraceAssemble2DInt<7,7>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x88:
+            return EADGTraceAssemble2DInt<8,8>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x99:
+            return EADGTraceAssemble2DInt<9,9>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         default:
+            return EADGTraceAssemble2DInt(nf,B,pa_data,ea_data_int,
+                                          ea_data_ext,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x23:
+            return EADGTraceAssemble3DInt<2,3>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x34:
+            return EADGTraceAssemble3DInt<3,4>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x45:
+            return EADGTraceAssemble3DInt<4,5>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x56:
+            return EADGTraceAssemble3DInt<5,6>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x67:
+            return EADGTraceAssemble3DInt<6,7>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x78:
+            return EADGTraceAssemble3DInt<7,8>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x89:
+            return EADGTraceAssemble3DInt<8,9>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         default:
+            return EADGTraceAssemble3DInt(nf,B,pa_data,ea_data_int,
+                                          ea_data_ext,dofs1D,quad1D);
+      }
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
+void DGTraceIntegrator::AssembleEABoundaryFaces(const FiniteElementSpace& fes,
+                                                Vector &ea_data_bdr)
+{
+   SetupPA(fes, FaceType::Boundary);
+   nf = fes.GetNFbyType(FaceType::Boundary);
+   if (nf==0) { return; }
+   const Array<double> &B = maps->B;
+   if (dim == 1)
+   {
+      return EADGTraceAssemble1DBdr(nf,B,pa_data,ea_data_bdr);
+   }
+   else if (dim == 2)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EADGTraceAssemble2DBdr<2,2>(nf,B,pa_data,ea_data_bdr);
+         case 0x33: return EADGTraceAssemble2DBdr<3,3>(nf,B,pa_data,ea_data_bdr);
+         case 0x44: return EADGTraceAssemble2DBdr<4,4>(nf,B,pa_data,ea_data_bdr);
+         case 0x55: return EADGTraceAssemble2DBdr<5,5>(nf,B,pa_data,ea_data_bdr);
+         case 0x66: return EADGTraceAssemble2DBdr<6,6>(nf,B,pa_data,ea_data_bdr);
+         case 0x77: return EADGTraceAssemble2DBdr<7,7>(nf,B,pa_data,ea_data_bdr);
+         case 0x88: return EADGTraceAssemble2DBdr<8,8>(nf,B,pa_data,ea_data_bdr);
+         case 0x99: return EADGTraceAssemble2DBdr<9,9>(nf,B,pa_data,ea_data_bdr);
+         default:
+            return EADGTraceAssemble2DBdr(nf,B,pa_data,ea_data_bdr,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x23: return EADGTraceAssemble3DBdr<2,3>(nf,B,pa_data,ea_data_bdr);
+         case 0x34: return EADGTraceAssemble3DBdr<3,4>(nf,B,pa_data,ea_data_bdr);
+         case 0x45: return EADGTraceAssemble3DBdr<4,5>(nf,B,pa_data,ea_data_bdr);
+         case 0x56: return EADGTraceAssemble3DBdr<5,6>(nf,B,pa_data,ea_data_bdr);
+         case 0x67: return EADGTraceAssemble3DBdr<6,7>(nf,B,pa_data,ea_data_bdr);
+         case 0x78: return EADGTraceAssemble3DBdr<7,8>(nf,B,pa_data,ea_data_bdr);
+         case 0x89: return EADGTraceAssemble3DBdr<8,9>(nf,B,pa_data,ea_data_bdr);
+         default:
+            return EADGTraceAssemble3DBdr(nf,B,pa_data,ea_data_bdr,dofs1D,quad1D);
+      }
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
+}

fem/bilininteg_diffusion_ea.cpp

@@ -0,0 +1,275 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "../general/forall.hpp"
+#include "bilininteg.hpp"
+#include "gridfunc.hpp"
+
+namespace mfem
+{
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADiffusionAssemble1D(const int NE,
+                                  const Array<double> &b,
+                                  const Array<double> &g,
+                                  const Vector &padata,
+                                  Vector &eadata,
+                                  const int d1d = 0,
+                                  const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_Gi[MQ1];
+      double r_Gj[MQ1];
+      for (int q = 0; q < Q1D; q++)
+      {
+         r_Gi[q] = G(q,MFEM_THREAD_ID(x));
+         r_Gj[q] = G(q,MFEM_THREAD_ID(y));
+      }
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(j1,y,D1D)
+         {
+            double val = 0.0;
+            for (int k1 = 0; k1 < Q1D; ++k1)
+            {
+               val += r_Gj[k1] * D(k1, e) * r_Gi[k1];
+            }
+            A(i1, j1, e) = val;
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADiffusionAssemble2D(const int NE,
+                                  const Array<double> &b,
+                                  const Array<double> &g,
+                                  const Vector &padata,
+                                  Vector &eadata,
+                                  const int d1d = 0,
+                                  const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(b.Read(), Q1D, D1D);
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, 3, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      double r_G[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+            r_G[q][d] = G(q,d);
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            for (int j1 = 0; j1 < D1D; ++j1)
+            {
+               for (int j2 = 0; j2 < D1D; ++j2)
+               {
+                  double val = 0.0;
+                  for (int k1 = 0; k1 < Q1D; ++k1)
+                  {
+                     for (int k2 = 0; k2 < Q1D; ++k2)
+                     {
+                        double bgi = r_G[k1][i1] * r_B[k2][i2];
+                        double gbi = r_B[k1][i1] * r_G[k2][i2];
+                        double bgj = r_G[k1][j1] * r_B[k2][j2];
+                        double gbj = r_B[k1][j1] * r_G[k2][j2];
+                        double D00 = D(k1,k2,0,e);
+                        double D10 = D(k1,k2,1,e);
+                        double D01 = D10;
+                        double D11 = D(k1,k2,2,e);
+                        val += bgi * D00 * bgj
+                               + gbi * D01 * bgj
+                               + bgi * D10 * gbj
+                               + gbi * D11 * gbj;
+                     }
+                  }
+                  A(i1, i2, j1, j2, e) = val;
+               }
+            }
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADiffusionAssemble3D(const int NE,
+                                  const Array<double> &g,
+                                  const Array<double> &b,
+                                  const Vector &padata,
+                                  Vector &eadata,
+                                  const int d1d = 0,
+                                  const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(b.Read(), Q1D, D1D);
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, Q1D, 6, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, D1D,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      double r_G[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+            r_G[q][d] = G(q,d);
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            MFEM_FOREACH_THREAD(i3,z,D1D)
+            {
+               for (int j1 = 0; j1 < D1D; ++j1)
+               {
+                  for (int j2 = 0; j2 < D1D; ++j2)
+                  {
+                     for (int j3 = 0; j3 < D1D; ++j3)
+                     {
+                        double val = 0.0;
+                        for (int k1 = 0; k1 < Q1D; ++k1)
+                        {
+                           for (int k2 = 0; k2 < Q1D; ++k2)
+                           {
+                              for (int k3 = 0; k3 < Q1D; ++k3)
+                              {
+                                 double bbgi = r_G[k1][i1] * r_B[k2][i2] * r_B[k3][i3];
+                                 double bgbi = r_B[k1][i1] * r_G[k2][i2] * r_B[k3][i3];
+                                 double gbbi = r_B[k1][i1] * r_B[k2][i2] * r_G[k3][i3];
+                                 double bbgj = r_G[k1][j1] * r_B[k2][j2] * r_B[k3][j3];
+                                 double bgbj = r_B[k1][j1] * r_G[k2][j2] * r_B[k3][j3];
+                                 double gbbj = r_B[k1][j1] * r_B[k2][j2] * r_G[k3][j3];
+                                 double D00 = D(k1,k2,k3,0,e);
+                                 double D10 = D(k1,k2,k3,1,e);
+                                 double D20 = D(k1,k2,k3,2,e);
+                                 double D01 = D10;
+                                 double D11 = D(k1,k2,k3,3,e);
+                                 double D21 = D(k1,k2,k3,4,e);
+                                 double D02 = D20;
+                                 double D12 = D21;
+                                 double D22 = D(k1,k2,k3,5,e);
+                                 val += bbgi * D00 * bbgj
+                                        + bgbi * D10 * bbgj
+                                        + gbbi * D20 * bbgj
+                                        + bbgi * D01 * bgbj
+                                        + bgbi * D11 * bgbj
+                                        + gbbi * D21 * bgbj
+                                        + bbgi * D02 * gbbj
+                                        + bgbi * D12 * gbbj
+                                        + gbbi * D22 * gbbj;
+                              }
+                           }
+                        }
+                        A(i1, i2, i3, j1, j2, j3, e) = val;
+                     }
+                  }
+               }
+            }
+         }
+      }
+   });
+}
+
+void DiffusionIntegrator::AssembleEA(const FiniteElementSpace &fes,
+                                     Vector &ea_data)
+{
+   AssemblePA(fes);
+   const int ne = fes.GetMesh()->GetNE();
+   const Array<double> &B = maps->B;
+   const Array<double> &G = maps->G;
+   if (dim == 1)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EADiffusionAssemble1D<2,2>(ne,B,G,pa_data,ea_data);
+         case 0x33: return EADiffusionAssemble1D<3,3>(ne,B,G,pa_data,ea_data);
+         case 0x44: return EADiffusionAssemble1D<4,4>(ne,B,G,pa_data,ea_data);
+         case 0x55: return EADiffusionAssemble1D<5,5>(ne,B,G,pa_data,ea_data);
+         case 0x66: return EADiffusionAssemble1D<6,6>(ne,B,G,pa_data,ea_data);
+         case 0x77: return EADiffusionAssemble1D<7,7>(ne,B,G,pa_data,ea_data);
+         case 0x88: return EADiffusionAssemble1D<8,8>(ne,B,G,pa_data,ea_data);
+         case 0x99: return EADiffusionAssemble1D<9,9>(ne,B,G,pa_data,ea_data);
+         default:   return EADiffusionAssemble1D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 2)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EADiffusionAssemble2D<2,2>(ne,B,G,pa_data,ea_data);
+         case 0x33: return EADiffusionAssemble2D<3,3>(ne,B,G,pa_data,ea_data);
+         case 0x44: return EADiffusionAssemble2D<4,4>(ne,B,G,pa_data,ea_data);
+         case 0x55: return EADiffusionAssemble2D<5,5>(ne,B,G,pa_data,ea_data);
+         case 0x66: return EADiffusionAssemble2D<6,6>(ne,B,G,pa_data,ea_data);
+         case 0x77: return EADiffusionAssemble2D<7,7>(ne,B,G,pa_data,ea_data);
+         case 0x88: return EADiffusionAssemble2D<8,8>(ne,B,G,pa_data,ea_data);
+         case 0x99: return EADiffusionAssemble2D<9,9>(ne,B,G,pa_data,ea_data);
+         default:   return EADiffusionAssemble2D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x23: return EADiffusionAssemble3D<2,3>(ne,B,G,pa_data,ea_data);
+         case 0x34: return EADiffusionAssemble3D<3,4>(ne,B,G,pa_data,ea_data);
+         case 0x45: return EADiffusionAssemble3D<4,5>(ne,B,G,pa_data,ea_data);
+         case 0x56: return EADiffusionAssemble3D<5,6>(ne,B,G,pa_data,ea_data);
+         case 0x67: return EADiffusionAssemble3D<6,7>(ne,B,G,pa_data,ea_data);
+         case 0x78: return EADiffusionAssemble3D<7,8>(ne,B,G,pa_data,ea_data);
+         case 0x89: return EADiffusionAssemble3D<8,9>(ne,B,G,pa_data,ea_data);
+         default:   return EADiffusionAssemble3D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
+}

fem/bilininteg_hcurl.cpp

@@ -24,12 +24,12 @@ constexpr int HCURL_MAX_D1D = 5;
 constexpr int HCURL_MAX_Q1D = 6;
 
 // PA H(curl) Mass Assemble 2D kernel
-static void PAHcurlSetup2D(const int Q1D,
-                           const int NE,
-                           const Array<double> &w,
-                           const Vector &j,
-                           Vector &_coeff,
-                           Vector &op)
+void PAHcurlSetup2D(const int Q1D,
+                    const int NE,
+                    const Array<double> &w,
+                    const Vector &j,
+                    Vector &_coeff,
+                    Vector &op)
 {
    const int NQ = Q1D*Q1D;
    auto W = w.Read();
@@ -55,12 +55,12 @@ static void PAHcurlSetup2D(const int Q1D,
 }
 
 // PA H(curl) Mass Assemble 3D kernel
-static void PAHcurlSetup3D(const int Q1D,
-                           const int NE,
-                           const Array<double> &w,
-                           const Vector &j,
-                           Vector &_coeff,
-                           Vector &op)
+void PAHcurlSetup3D(const int Q1D,
+                    const int NE,
+                    const Array<double> &w,
+                    const Vector &j,
+                    Vector &_coeff,
+                    Vector &op)
 {
    const int NQ = Q1D*Q1D*Q1D;
    auto W = w.Read();
@@ -106,78 +106,16 @@ static void PAHcurlSetup3D(const int Q1D,
    });
 }
 
-void VectorFEMassIntegrator::AssemblePA(const FiniteElementSpace &fes)
-{
-   // Assumes tensor-product elements
-   Mesh *mesh = fes.GetMesh();
-   const FiniteElement *fel = fes.GetFE(0);
-
-   const VectorTensorFiniteElement *el =
-      dynamic_cast<const VectorTensorFiniteElement*>(fel);
-   MFEM_VERIFY(el != NULL, "Only VectorTensorFiniteElement is supported!");
-
-   const IntegrationRule *ir
-      = IntRule ? IntRule : &MassIntegrator::GetRule(*el, *el,
-                                                     *mesh->GetElementTransformation(0));
-   const int dims = el->GetDim();
-   MFEM_VERIFY(dims == 2 || dims == 3, "");
-
-   const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
-   const int nq = ir->GetNPoints();
-   dim = mesh->Dimension();
-   MFEM_VERIFY(dim == 2 || dim == 3, "");
-
-   ne = fes.GetNE();
-   geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
-   mapsC = &el->GetDofToQuad(*ir, DofToQuad::TENSOR);
-   mapsO = &el->GetDofToQuadOpen(*ir, DofToQuad::TENSOR);
-   dofs1D = mapsC->ndof;
-   quad1D = mapsC->nqpt;
-
-   MFEM_VERIFY(dofs1D == mapsO->ndof + 1 && quad1D == mapsO->nqpt, "");
-
-   pa_data.SetSize(symmDims * nq * ne, Device::GetMemoryType());
-
-   Vector coeff(ne * nq);
-   coeff = 1.0;
-   if (Q)
-   {
-      for (int e=0; e<ne; ++e)
-      {
-         ElementTransformation *tr = mesh->GetElementTransformation(e);
-         for (int p=0; p<nq; ++p)
-         {
-            coeff[p + (e * nq)] = Q->Eval(*tr, ir->IntPoint(p));
-         }
-      }
-   }
-
-   if (el->GetDerivType() == mfem::FiniteElement::CURL && dim == 3)
-   {
-      PAHcurlSetup3D(quad1D, ne, ir->GetWeights(), geom->J,
-                     coeff, pa_data);
-   }
-   else if (el->GetDerivType() == mfem::FiniteElement::CURL && dim == 2)
-   {
-      PAHcurlSetup2D(quad1D, ne, ir->GetWeights(), geom->J,
-                     coeff, pa_data);
-   }
-   else
-   {
-      MFEM_ABORT("Unknown kernel.");
-   }
-}
-
-static void PAHcurlMassApply2D(const int D1D,
-                               const int Q1D,
-                               const int NE,
-                               const Array<double> &_Bo,
-                               const Array<double> &_Bc,
-                               const Array<double> &_Bot,
-                               const Array<double> &_Bct,
-                               const Vector &_op,
-                               const Vector &_x,
-                               Vector &_y)
+void PAHcurlMassApply2D(const int D1D,
+                        const int Q1D,
+                        const int NE,
+                        const Array<double> &_Bo,
+                        const Array<double> &_Bc,
+                        const Array<double> &_Bot,
+                        const Array<double> &_Bct,
+                        const Vector &_op,
+                        const Vector &_x,
+                        Vector &_y)
 {
    constexpr static int VDIM = 2;
 
@@ -294,13 +232,13 @@ static void PAHcurlMassApply2D(const int D1D,
    }); // end of element loop
 }
 
-static void PAHcurlMassAssembleDiagonal2D(const int D1D,
-                                          const int Q1D,
-                                          const int NE,
-                                          const Array<double> &_Bo,
-                                          const Array<double> &_Bc,
-                                          const Vector &_op,
-                                          Vector &_diag)
+void PAHcurlMassAssembleDiagonal2D(const int D1D,
+                                   const int Q1D,
+                                   const int NE,
+                                   const Array<double> &_Bo,
+                                   const Array<double> &_Bc,
+                                   const Vector &_op,
+                                   Vector &_diag)
 {
    constexpr static int VDIM = 2;
 
@@ -348,15 +286,17 @@ static void PAHcurlMassAssembleDiagonal2D(const int D1D,
    }); // end of element loop
 }
 
-template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
-static void PAHcurlMassAssembleDiagonal3D(const int D1D,
-                                          const int Q1D,
-                                          const int NE,
-                                          const Array<double> &_Bo,
-                                          const Array<double> &_Bc,
-                                          const Vector &_op,
-                                          Vector &_diag)
+void PAHcurlMassAssembleDiagonal3D(const int D1D,
+                                   const int Q1D,
+                                   const int NE,
+                                   const Array<double> &_Bo,
+                                   const Array<double> &_Bc,
+                                   const Vector &_op,
+                                   Vector &_diag)
 {
+   constexpr static int MAX_D1D = HCURL_MAX_D1D;
+   constexpr static int MAX_Q1D = HCURL_MAX_Q1D;
+
    MFEM_VERIFY(D1D <= MAX_D1D, "Error: D1D > MAX_D1D");
    MFEM_VERIFY(Q1D <= MAX_Q1D, "Error: Q1D > MAX_Q1D");
    constexpr static int VDIM = 3;
@@ -416,28 +356,20 @@ static void PAHcurlMassAssembleDiagonal3D(const int D1D,
    }); // end of element loop
 }
 
-void VectorFEMassIntegrator::AssembleDiagonalPA(Vector& diag)
+void PAHcurlMassApply3D(const int D1D,
+                        const int Q1D,
+                        const int NE,
+                        const Array<double> &_Bo,
+                        const Array<double> &_Bc,
+                        const Array<double> &_Bot,
+                        const Array<double> &_Bct,
+                        const Vector &_op,
+                        const Vector &_x,
+                        Vector &_y)
 {
-   if (dim == 3)
-      PAHcurlMassAssembleDiagonal3D(dofs1D, quad1D, ne,
-                                    mapsO->B, mapsC->B, pa_data, diag);
-   else
-      PAHcurlMassAssembleDiagonal2D(dofs1D, quad1D, ne,
-                                    mapsO->B, mapsC->B, pa_data, diag);
-}
+   constexpr static int MAX_D1D = HCURL_MAX_D1D;
+   constexpr static int MAX_Q1D = HCURL_MAX_Q1D;
 
-template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
-static void PAHcurlMassApply3D(const int D1D,
-                               const int Q1D,
-                               const int NE,
-                               const Array<double> &_Bo,
-                               const Array<double> &_Bc,
-                               const Array<double> &_Bot,
-                               const Array<double> &_Bct,
-                               const Vector &_op,
-                               const Vector &_x,
-                               Vector &_y)
-{
    MFEM_VERIFY(D1D <= MAX_D1D, "Error: D1D > MAX_D1D");
    MFEM_VERIFY(Q1D <= MAX_Q1D, "Error: Q1D > MAX_Q1D");
    constexpr static int VDIM = 3;
@@ -615,20 +547,6 @@ static void PAHcurlMassApply3D(const int D1D,
    }); // end of element loop
 }
 
-void VectorFEMassIntegrator::AddMultPA(const Vector &x, Vector &y) const
-{
-   if (dim == 3)
-   {
-      PAHcurlMassApply3D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
-                         mapsC->Bt, pa_data, x, y);
-   }
-   else
-   {
-      PAHcurlMassApply2D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
-                         mapsC->Bt, pa_data, x, y);
-   }
-}
-
 // PA H(curl) curl-curl assemble 2D kernel
 static void PACurlCurlSetup2D(const int Q1D,
                               const int NE,
@@ -1678,92 +1596,25 @@ void CurlCurlIntegrator::AssembleDiagonalPA(Vector& diag)
    }
 }
 
-void MixedVectorGradientIntegrator::AssemblePA(const FiniteElementSpace
-                                               &trial_fes,
-                                               const FiniteElementSpace &test_fes)
-{
-   // Assumes tensor-product elements, with a vector test space and H^1 trial space.
-   Mesh *mesh = trial_fes.GetMesh();
-   const FiniteElement *trial_fel = trial_fes.GetFE(0);
-   const FiniteElement *test_fel = test_fes.GetFE(0);
-
-   const NodalTensorFiniteElement *trial_el =
-      dynamic_cast<const NodalTensorFiniteElement*>(trial_fel);
-   MFEM_VERIFY(trial_el != NULL, "Only NodalTensorFiniteElement is supported!");
-
-   const VectorTensorFiniteElement *test_el =
-      dynamic_cast<const VectorTensorFiniteElement*>(test_fel);
-   MFEM_VERIFY(test_el != NULL, "Only VectorTensorFiniteElement is supported!");
-
-   const IntegrationRule *ir
-      = IntRule ? IntRule : &MassIntegrator::GetRule(*trial_el, *trial_el,
-                                                     *mesh->GetElementTransformation(0));
-   const int dims = trial_el->GetDim();
-   MFEM_VERIFY(dims == 2 || dims == 3, "");
-
-   const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
-   const int nq = ir->GetNPoints();
-   dim = mesh->Dimension();
-   MFEM_VERIFY(dim == 2 || dim == 3, "");
-
-   MFEM_VERIFY(trial_el->GetOrder() == test_el->GetOrder(), "");
-
-   ne = trial_fes.GetNE();
-   geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
-   mapsC = &test_el->GetDofToQuad(*ir, DofToQuad::TENSOR);
-   mapsO = &test_el->GetDofToQuadOpen(*ir, DofToQuad::TENSOR);
-   dofs1D = mapsC->ndof;
-   quad1D = mapsC->nqpt;
-
-   MFEM_VERIFY(dofs1D == mapsO->ndof + 1 && quad1D == mapsO->nqpt, "");
-
-   pa_data.SetSize(symmDims * nq * ne, Device::GetMemoryType());
-
-   Vector coeff(ne * nq);
-   coeff = 1.0;
-   if (Q)
-   {
-      for (int e=0; e<ne; ++e)
-      {
-         ElementTransformation *tr = mesh->GetElementTransformation(e);
-         for (int p=0; p<nq; ++p)
-         {
-            coeff[p + (e * nq)] = Q->Eval(*tr, ir->IntPoint(p));
-         }
-      }
-   }
-
-   // Use the same setup functions as VectorFEMassIntegrator.
-   if (test_el->GetDerivType() == mfem::FiniteElement::CURL && dim == 3)
-   {
-      PAHcurlSetup3D(quad1D, ne, ir->GetWeights(), geom->J,
-                     coeff, pa_data);
-   }
-   else if (test_el->GetDerivType() == mfem::FiniteElement::CURL && dim == 2)
-   {
-      PAHcurlSetup2D(quad1D, ne, ir->GetWeights(), geom->J,
-                     coeff, pa_data);
-   }
-   else
-   {
-      MFEM_ABORT("Unknown kernel.");
-   }
-}
-
 // Apply to x corresponding to DOF's in H^1 (trial), whose gradients are integrated
 // against H(curl) test functions corresponding to y.
-template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
-static void PAHcurlH1Apply3D(const int D1D,
-                             const int Q1D,
-                             const int NE,
-                             const Array<double> &_Bc,
-                             const Array<double> &_Gc,
-                             const Array<double> &_Bot,
-                             const Array<double> &_Bct,
-                             const Vector &_op,
-                             const Vector &_x,
-                             Vector &_y)
+void PAHcurlH1Apply3D(const int D1D,
+                      const int Q1D,
+                      const int NE,
+                      const Array<double> &_Bc,
+                      const Array<double> &_Gc,
+                      const Array<double> &_Bot,
+                      const Array<double> &_Bct,
+                      const Vector &_op,
+                      const Vector &_x,
+                      Vector &_y)
 {
+   constexpr static int MAX_D1D = HCURL_MAX_D1D;
+   constexpr static int MAX_Q1D = HCURL_MAX_Q1D;
+
+   MFEM_VERIFY(D1D <= MAX_D1D, "Error: D1D > MAX_D1D");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "Error: Q1D > MAX_Q1D");
+
    constexpr static int VDIM = 3;
 
    auto Bc = Reshape(_Bc.Read(), Q1D, D1D);
@@ -1937,16 +1788,16 @@ static void PAHcurlH1Apply3D(const int D1D,
 
 // Apply to x corresponding to DOF's in H^1 (trial), whose gradients are integrated
 // against H(curl) test functions corresponding to y.
-static void PAHcurlH1Apply2D(const int D1D,
-                             const int Q1D,
-                             const int NE,
-                             const Array<double> &_Bc,
-                             const Array<double> &_Gc,
-                             const Array<double> &_Bot,
-                             const Array<double> &_Bct,
-                             const Vector &_op,
-                             const Vector &_x,
-                             Vector &_y)
+void PAHcurlH1Apply2D(const int D1D,
+                      const int Q1D,
+                      const int NE,
+                      const Array<double> &_Bc,
+                      const Array<double> &_Gc,
+                      const Array<double> &_Bot,
+                      const Array<double> &_Bct,
+                      const Vector &_op,
+                      const Vector &_x,
+                      Vector &_y)
 {
    constexpr static int VDIM = 2;
 
@@ -2057,18 +1908,4 @@ static void PAHcurlH1Apply2D(const int D1D,
    }); // end of element loop
 }
 
-void MixedVectorGradientIntegrator::AddMultPA(const Vector &x, Vector &y) const
-{
-   if (dim == 3)
-      PAHcurlH1Apply3D(dofs1D, quad1D, ne, mapsC->B, mapsC->G,
-                       mapsO->Bt, mapsC->Bt, pa_data, x, y);
-   else if (dim == 2)
-      PAHcurlH1Apply2D(dofs1D, quad1D, ne, mapsC->B, mapsC->G,
-                       mapsO->Bt, mapsC->Bt, pa_data, x, y);
-   else
-   {
-      MFEM_ABORT("Unsupported dimension!");
-   }
-}
-
 } // namespace mfem

fem/bilininteg_mass_ea.cpp

@@ -0,0 +1,255 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "../general/forall.hpp"
+#include "bilininteg.hpp"
+#include "gridfunc.hpp"
+
+namespace mfem
+{
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAMassAssemble1D(const int NE,
+                             const Array<double> &basis,
+                             const Vector &padata,
+                             Vector &eadata,
+                             const int d1d = 0,
+                             const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, NE);
+   auto M = Reshape(eadata.Write(), D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_Bi[MQ1];
+      double r_Bj[MQ1];
+      for (int q = 0; q < Q1D; q++)
+      {
+         r_Bi[q] = B(q,MFEM_THREAD_ID(x));
+         r_Bj[q] = B(q,MFEM_THREAD_ID(y));
+      }
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(j1,y,D1D)
+         {
+            double val = 0.0;
+            for (int k1 = 0; k1 < Q1D; ++k1)
+            {
+               val += r_Bi[k1] * r_Bj[k1] * D(k1, e);
+            }
+            M(i1, j1, e) = val;
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAMassAssemble2D(const int NE,
+                             const Array<double> &basis,
+                             const Vector &padata,
+                             Vector &eadata,
+                             const int d1d = 0,
+                             const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, NE);
+   auto M = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+         }
+      }
+      MFEM_SHARED double s_D[MQ1][MQ1];
+      MFEM_FOREACH_THREAD(k1,x,Q1D)
+      {
+         MFEM_FOREACH_THREAD(k2,y,Q1D)
+         {
+            s_D[k1][k2] = D(k1,k2,e);
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            for (int j1 = 0; j1 < D1D; ++j1)
+            {
+               for (int j2 = 0; j2 < D1D; ++j2)
+               {
+                  double val = 0.0;
+                  for (int k1 = 0; k1 < Q1D; ++k1)
+                  {
+                     for (int k2 = 0; k2 < Q1D; ++k2)
+                     {
+                        val += r_B[k1][i1] * r_B[k1][j1]
+                               * r_B[k2][i2] * r_B[k2][j2]
+                               * s_D[k1][k2];
+                     }
+                  }
+                  M(i1, i2, j1, j2, e) = val;
+               }
+            }
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAMassAssemble3D(const int NE,
+                             const Array<double> &basis,
+                             const Vector &padata,
+                             Vector &eadata,
+                             const int d1d = 0,
+                             const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, Q1D, NE);
+   auto M = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, D1D,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+         }
+      }
+      MFEM_SHARED double s_D[MQ1][MQ1][MQ1];
+      MFEM_FOREACH_THREAD(k1,x,Q1D)
+      {
+         MFEM_FOREACH_THREAD(k2,y,Q1D)
+         {
+            MFEM_FOREACH_THREAD(k3,z,Q1D)
+            {
+               s_D[k1][k2][k3] = D(k1,k2,k3,e);
+            }
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            MFEM_FOREACH_THREAD(i3,z,D1D)
+            {
+               for (int j1 = 0; j1 < D1D; ++j1)
+               {
+                  for (int j2 = 0; j2 < D1D; ++j2)
+                  {
+                     for (int j3 = 0; j3 < D1D; ++j3)
+                     {
+                        double val = 0.0;
+                        for (int k1 = 0; k1 < Q1D; ++k1)
+                        {
+                           for (int k2 = 0; k2 < Q1D; ++k2)
+                           {
+                              for (int k3 = 0; k3 < Q1D; ++k3)
+                              {
+                                 val += r_B[k1][i1] * r_B[k1][j1]
+                                        * r_B[k2][i2] * r_B[k2][j2]
+                                        * r_B[k3][i3] * r_B[k3][j3]
+                                        * s_D[k1][k2][k3];
+                              }
+                           }
+                        }
+                        M(i1, i2, i3, j1, j2, j3, e) = val;
+                     }
+                  }
+               }
+            }
+         }
+      }
+   });
+}
+
+void MassIntegrator::AssembleEA(const FiniteElementSpace &fes,
+                                Vector &ea_data)
+{
+   AssemblePA(fes);
+   const int ne = fes.GetMesh()->GetNE();
+   const Array<double> &B = maps->B;
+   if (dim == 1)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EAMassAssemble1D<2,2>(ne,B,pa_data,ea_data);
+         case 0x33: return EAMassAssemble1D<3,3>(ne,B,pa_data,ea_data);
+         case 0x44: return EAMassAssemble1D<4,4>(ne,B,pa_data,ea_data);
+         case 0x55: return EAMassAssemble1D<5,5>(ne,B,pa_data,ea_data);
+         case 0x66: return EAMassAssemble1D<6,6>(ne,B,pa_data,ea_data);
+         case 0x77: return EAMassAssemble1D<7,7>(ne,B,pa_data,ea_data);
+         case 0x88: return EAMassAssemble1D<8,8>(ne,B,pa_data,ea_data);
+         case 0x99: return EAMassAssemble1D<9,9>(ne,B,pa_data,ea_data);
+         default:   return EAMassAssemble1D(ne,B,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 2)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EAMassAssemble2D<2,2>(ne,B,pa_data,ea_data);
+         case 0x33: return EAMassAssemble2D<3,3>(ne,B,pa_data,ea_data);
+         case 0x44: return EAMassAssemble2D<4,4>(ne,B,pa_data,ea_data);
+         case 0x55: return EAMassAssemble2D<5,5>(ne,B,pa_data,ea_data);
+         case 0x66: return EAMassAssemble2D<6,6>(ne,B,pa_data,ea_data);
+         case 0x77: return EAMassAssemble2D<7,7>(ne,B,pa_data,ea_data);
+         case 0x88: return EAMassAssemble2D<8,8>(ne,B,pa_data,ea_data);
+         case 0x99: return EAMassAssemble2D<9,9>(ne,B,pa_data,ea_data);
+         default:   return EAMassAssemble2D(ne,B,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x23: return EAMassAssemble3D<2,3>(ne,B,pa_data,ea_data);
+         case 0x34: return EAMassAssemble3D<3,4>(ne,B,pa_data,ea_data);
+         case 0x45: return EAMassAssemble3D<4,5>(ne,B,pa_data,ea_data);
+         case 0x56: return EAMassAssemble3D<5,6>(ne,B,pa_data,ea_data);
+         case 0x67: return EAMassAssemble3D<6,7>(ne,B,pa_data,ea_data);
+         case 0x78: return EAMassAssemble3D<7,8>(ne,B,pa_data,ea_data);
+         case 0x89: return EAMassAssemble3D<8,9>(ne,B,pa_data,ea_data);
+         default:   return EAMassAssemble3D(ne,B,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
+}

fem/bilininteg_transpose_ea.cpp

@@ -0,0 +1,103 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "../general/forall.hpp"
+#include "bilininteg.hpp"
+
+namespace mfem
+{
+
+void TransposeIntegrator::AssembleEA(const FiniteElementSpace &fes,
+                                     Vector &ea_data)
+{
+   Vector ea_data_tmp(ea_data.Size());
+   ea_data_tmp = 0.0;
+   bfi->AssembleEA(fes, ea_data_tmp);
+   const int ne = fes.GetNE();
+   if (ne == 0) { return; }
+   const int dofs = fes.GetFE(0)->GetDof();
+   auto A = Reshape(ea_data_tmp.Write(), dofs, dofs, ne);
+   auto AT = Reshape(ea_data.Write(), dofs, dofs, ne);
+   MFEM_FORALL(e, ne,
+   {
+      for (int i = 0; i < dofs; i++)
+      {
+         for (int j = 0; j < dofs; j++)
+         {
+            const double a = A(i, j, e);
+            AT(j, i, e) += a;
+         }
+      }
+   });
+}
+
+void TransposeIntegrator::AssembleEAInteriorFaces(const FiniteElementSpace& fes,
+                                                  Vector &ea_data_int,
+                                                  Vector &ea_data_ext)
+{
+   const int nf = fes.GetNFbyType(FaceType::Interior);
+   if (nf == 0) { return; }
+   Vector ea_data_int_tmp(ea_data_int.Size());
+   Vector ea_data_ext_tmp(ea_data_ext.Size());
+   ea_data_int_tmp = 0.0;
+   ea_data_ext_tmp = 0.0;
+   bfi->AssembleEAInteriorFaces(fes, ea_data_int_tmp, ea_data_ext_tmp);
+   const int faceDofs = fes.GetTraceElement(0,
+                                            fes.GetMesh()->GetFaceBaseGeometry(0))->GetDof();
+   auto A_int = Reshape(ea_data_int_tmp.Read(), faceDofs, faceDofs, 2, nf);
+   auto A_ext = Reshape(ea_data_ext_tmp.Read(), faceDofs, faceDofs, 2, nf);
+   auto AT_int = Reshape(ea_data_int.ReadWrite(), faceDofs, faceDofs, 2, nf);
+   auto AT_ext = Reshape(ea_data_ext.ReadWrite(), faceDofs, faceDofs, 2, nf);
+   MFEM_FORALL(f, nf,
+   {
+      for (int i = 0; i < faceDofs; i++)
+      {
+         for (int j = 0; j < faceDofs; j++)
+         {
+            const double a_int0 = A_int(i, j, 0, f);
+            const double a_int1 = A_int(i, j, 1, f);
+            const double a_ext0 = A_ext(i, j, 0, f);
+            const double a_ext1 = A_ext(i, j, 1, f);
+            AT_int(j, i, 0, f) += a_int0;
+            AT_int(j, i, 1, f) += a_int1;
+            AT_ext(j, i, 0, f) += a_ext1;
+            AT_ext(j, i, 1, f) += a_ext0;
+         }
+      }
+   });
+}
+
+void TransposeIntegrator::AssembleEABoundaryFaces(const FiniteElementSpace& fes,
+                                                  Vector &ea_data_bdr)
+{
+   const int nf = fes.GetNFbyType(FaceType::Boundary);
+   if (nf == 0) { return; }
+   Vector ea_data_bdr_tmp(ea_data_bdr.Size());
+   ea_data_bdr_tmp = 0.0;
+   bfi->AssembleEABoundaryFaces(fes, ea_data_bdr_tmp);
+   const int faceDofs = fes.GetTraceElement(0,
+                                            fes.GetMesh()->GetFaceBaseGeometry(0))->GetDof();
+   auto A_bdr = Reshape(ea_data_bdr_tmp.Read(), faceDofs, faceDofs, nf);
+   auto AT_bdr = Reshape(ea_data_bdr.ReadWrite(), faceDofs, faceDofs, nf);
+   MFEM_FORALL(f, nf,
+   {
+      for (int i = 0; i < faceDofs; i++)
+      {
+         for (int j = 0; j < faceDofs; j++)
+         {
+            const double a_bdr = A_bdr(i, j, f);
+            AT_bdr(j, i, f) += a_bdr;
+         }
+      }
+   });
+}
+
+}

fem/bilininteg_vectorfe.cpp

@@ -0,0 +1,379 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "bilininteg.hpp"
+
+namespace mfem
+{
+
+void PAHcurlSetup2D(const int Q1D,
+                    const int NE,
+                    const Array<double> &w,
+                    const Vector &j,
+                    Vector &_coeff,
+                    Vector &op);
+
+void PAHcurlSetup3D(const int Q1D,
+                    const int NE,
+                    const Array<double> &w,
+                    const Vector &j,
+                    Vector &_coeff,
+                    Vector &op);
+
+void PAHcurlMassAssembleDiagonal2D(const int D1D,
+                                   const int Q1D,
+                                   const int NE,
+                                   const Array<double> &_Bo,
+                                   const Array<double> &_Bc,
+                                   const Vector &_op,
+                                   Vector &_diag);
+
+void PAHcurlMassAssembleDiagonal3D(const int D1D,
+                                   const int Q1D,
+                                   const int NE,
+                                   const Array<double> &_Bo,
+                                   const Array<double> &_Bc,
+                                   const Vector &_op,
+                                   Vector &_diag);
+
+void PAHcurlMassApply2D(const int D1D,
+                        const int Q1D,
+                        const int NE,
+                        const Array<double> &_Bo,
+                        const Array<double> &_Bc,
+                        const Array<double> &_Bot,
+                        const Array<double> &_Bct,
+                        const Vector &_op,
+                        const Vector &_x,
+                        Vector &_y);
+
+void PAHcurlMassApply3D(const int D1D,
+                        const int Q1D,
+                        const int NE,
+                        const Array<double> &_Bo,
+                        const Array<double> &_Bc,
+                        const Array<double> &_Bot,
+                        const Array<double> &_Bct,
+                        const Vector &_op,
+                        const Vector &_x,
+                        Vector &_y);
+
+void PAHdivSetup2D(const int Q1D,
+                   const int NE,
+                   const Array<double> &w,
+                   const Vector &j,
+                   Vector &_coeff,
+                   Vector &op);
+
+void PAHdivSetup3D(const int Q1D,
+                   const int NE,
+                   const Array<double> &w,
+                   const Vector &j,
+                   Vector &_coeff,
+                   Vector &op);
+
+void PAHcurlH1Apply2D(const int D1D,
+                      const int Q1D,
+                      const int NE,
+                      const Array<double> &_Bc,
+                      const Array<double> &_Gc,
+                      const Array<double> &_Bot,
+                      const Array<double> &_Bct,
+                      const Vector &_op,
+                      const Vector &_x,
+                      Vector &_y);
+
+void PAHcurlH1Apply3D(const int D1D,
+                      const int Q1D,
+                      const int NE,
+                      const Array<double> &_Bc,
+                      const Array<double> &_Gc,
+                      const Array<double> &_Bot,
+                      const Array<double> &_Bct,
+                      const Vector &_op,
+                      const Vector &_x,
+                      Vector &_y);
+
+void PAHdivMassAssembleDiagonal2D(const int D1D,
+                                  const int Q1D,
+                                  const int NE,
+                                  const Array<double> &_Bo,
+                                  const Array<double> &_Bc,
+                                  const Vector &_op,
+                                  Vector &_diag);
+
+void PAHdivMassAssembleDiagonal3D(const int D1D,
+                                  const int Q1D,
+                                  const int NE,
+                                  const Array<double> &_Bo,
+                                  const Array<double> &_Bc,
+                                  const Vector &_op,
+                                  Vector &_diag);
+
+void PAHdivMassApply2D(const int D1D,
+                       const int Q1D,
+                       const int NE,
+                       const Array<double> &_Bo,
+                       const Array<double> &_Bc,
+                       const Array<double> &_Bot,
+                       const Array<double> &_Bct,
+                       const Vector &_op,
+                       const Vector &_x,
+                       Vector &_y);
+
+void PAHdivMassApply3D(const int D1D,
+                       const int Q1D,
+                       const int NE,
+                       const Array<double> &_Bo,
+                       const Array<double> &_Bc,
+                       const Array<double> &_Bot,
+                       const Array<double> &_Bct,
+                       const Vector &_op,
+                       const Vector &_x,
+                       Vector &_y);
+
+void VectorFEMassIntegrator::AssemblePA(const FiniteElementSpace &fes)
+{
+   // Assumes tensor-product elements
+   Mesh *mesh = fes.GetMesh();
+   const FiniteElement *fel = fes.GetFE(0);
+
+   const VectorTensorFiniteElement *el =
+      dynamic_cast<const VectorTensorFiniteElement*>(fel);
+   MFEM_VERIFY(el != NULL, "Only VectorTensorFiniteElement is supported!");
+
+   const IntegrationRule *ir
+      = IntRule ? IntRule : &MassIntegrator::GetRule(*el, *el,
+                                                     *mesh->GetElementTransformation(0));
+   const int dims = el->GetDim();
+   MFEM_VERIFY(dims == 2 || dims == 3, "");
+
+   const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
+   const int nq = ir->GetNPoints();
+   dim = mesh->Dimension();
+   MFEM_VERIFY(dim == 2 || dim == 3, "");
+
+   ne = fes.GetNE();
+   geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
+   mapsC = &el->GetDofToQuad(*ir, DofToQuad::TENSOR);
+   mapsO = &el->GetDofToQuadOpen(*ir, DofToQuad::TENSOR);
+   dofs1D = mapsC->ndof;
+   quad1D = mapsC->nqpt;
+
+   MFEM_VERIFY(dofs1D == mapsO->ndof + 1 && quad1D == mapsO->nqpt, "");
+
+   pa_data.SetSize(symmDims * nq * ne, Device::GetMemoryType());
+
+   Vector coeff(ne * nq);
+   coeff = 1.0;
+   if (Q)
+   {
+      for (int e=0; e<ne; ++e)
+      {
+         ElementTransformation *tr = mesh->GetElementTransformation(e);
+         for (int p=0; p<nq; ++p)
+         {
+            coeff[p + (e * nq)] = Q->Eval(*tr, ir->IntPoint(p));
+         }
+      }
+   }
+
+   fetype = el->GetDerivType();
+
+   if (el->GetDerivType() == mfem::FiniteElement::CURL && dim == 3)
+   {
+      PAHcurlSetup3D(quad1D, ne, ir->GetWeights(), geom->J,
+                     coeff, pa_data);
+   }
+   else if (el->GetDerivType() == mfem::FiniteElement::CURL && dim == 2)
+   {
+      PAHcurlSetup2D(quad1D, ne, ir->GetWeights(), geom->J,
+                     coeff, pa_data);
+   }
+   else if (el->GetDerivType() == mfem::FiniteElement::DIV && dim == 3)
+   {
+      PAHdivSetup3D(quad1D, ne, ir->GetWeights(), geom->J,
+                    coeff, pa_data);
+   }
+   else if (el->GetDerivType() == mfem::FiniteElement::DIV && dim == 2)
+   {
+      PAHdivSetup2D(quad1D, ne, ir->GetWeights(), geom->J,
+                    coeff, pa_data);
+   }
+   else
+   {
+      MFEM_ABORT("Unknown kernel.");
+   }
+}
+
+void VectorFEMassIntegrator::AssembleDiagonalPA(Vector& diag)
+{
+   if (dim == 3)
+   {
+      if (fetype == mfem::FiniteElement::CURL)
+      {
+         PAHcurlMassAssembleDiagonal3D(dofs1D, quad1D, ne,
+                                       mapsO->B, mapsC->B, pa_data, diag);
+      }
+      else if (fetype == mfem::FiniteElement::DIV)
+      {
+         PAHdivMassAssembleDiagonal3D(dofs1D, quad1D, ne,
+                                      mapsO->B, mapsC->B, pa_data, diag);
+      }
+      else
+      {
+         MFEM_ABORT("Unknown kernel.");
+      }
+   }
+   else
+   {
+      if (fetype == mfem::FiniteElement::CURL)
+      {
+         PAHcurlMassAssembleDiagonal2D(dofs1D, quad1D, ne,
+                                       mapsO->B, mapsC->B, pa_data, diag);
+      }
+      else if (fetype == mfem::FiniteElement::DIV)
+      {
+         PAHdivMassAssembleDiagonal2D(dofs1D, quad1D, ne,
+                                      mapsO->B, mapsC->B, pa_data, diag);
+      }
+      else
+      {
+         MFEM_ABORT("Unknown kernel.");
+      }
+   }
+}
+
+void VectorFEMassIntegrator::AddMultPA(const Vector &x, Vector &y) const
+{
+   if (dim == 3)
+   {
+      if (fetype == mfem::FiniteElement::CURL)
+      {
+         PAHcurlMassApply3D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
+                            mapsC->Bt, pa_data, x, y);
+      }
+      else if (fetype == mfem::FiniteElement::DIV)
+      {
+         PAHdivMassApply3D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
+                           mapsC->Bt, pa_data, x, y);
+      }
+      else
+      {
+         MFEM_ABORT("Unknown kernel.");
+      }
+   }
+   else
+   {
+      if (fetype == mfem::FiniteElement::CURL)
+      {
+         PAHcurlMassApply2D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
+                            mapsC->Bt, pa_data, x, y);
+      }
+      else if (fetype == mfem::FiniteElement::DIV)
+      {
+         PAHdivMassApply2D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
+                           mapsC->Bt, pa_data, x, y);
+      }
+      else
+      {
+         MFEM_ABORT("Unknown kernel.");
+      }
+   }
+}
+
+void MixedVectorGradientIntegrator::AssemblePA(const FiniteElementSpace
+                                               &trial_fes,
+                                               const FiniteElementSpace &test_fes)
+{
+   // Assumes tensor-product elements, with a vector test space and H^1 trial space.
+   Mesh *mesh = trial_fes.GetMesh();
+   const FiniteElement *trial_fel = trial_fes.GetFE(0);
+   const FiniteElement *test_fel = test_fes.GetFE(0);
+
+   const NodalTensorFiniteElement *trial_el =
+      dynamic_cast<const NodalTensorFiniteElement*>(trial_fel);
+   MFEM_VERIFY(trial_el != NULL, "Only NodalTensorFiniteElement is supported!");
+
+   const VectorTensorFiniteElement *test_el =
+      dynamic_cast<const VectorTensorFiniteElement*>(test_fel);
+   MFEM_VERIFY(test_el != NULL, "Only VectorTensorFiniteElement is supported!");
+
+   const IntegrationRule *ir
+      = IntRule ? IntRule : &MassIntegrator::GetRule(*trial_el, *trial_el,
+                                                     *mesh->GetElementTransformation(0));
+   const int dims = trial_el->GetDim();
+   MFEM_VERIFY(dims == 2 || dims == 3, "");
+
+   const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
+   const int nq = ir->GetNPoints();
+   dim = mesh->Dimension();
+   MFEM_VERIFY(dim == 2 || dim == 3, "");
+
+   MFEM_VERIFY(trial_el->GetOrder() == test_el->GetOrder(), "");
+
+   ne = trial_fes.GetNE();
+   geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
+   mapsC = &test_el->GetDofToQuad(*ir, DofToQuad::TENSOR);
+   mapsO = &test_el->GetDofToQuadOpen(*ir, DofToQuad::TENSOR);
+   dofs1D = mapsC->ndof;
+   quad1D = mapsC->nqpt;
+
+   MFEM_VERIFY(dofs1D == mapsO->ndof + 1 && quad1D == mapsO->nqpt, "");
+
+   pa_data.SetSize(symmDims * nq * ne, Device::GetMemoryType());
+
+   Vector coeff(ne * nq);
+   coeff = 1.0;
+   if (Q)
+   {
+      for (int e=0; e<ne; ++e)
+      {
+         ElementTransformation *tr = mesh->GetElementTransformation(e);
+         for (int p=0; p<nq; ++p)
+         {
+            coeff[p + (e * nq)] = Q->Eval(*tr, ir->IntPoint(p));
+         }
+      }
+   }
+
+   // Use the same setup functions as VectorFEMassIntegrator.
+   if (test_el->GetDerivType() == mfem::FiniteElement::CURL && dim == 3)
+   {
+      PAHcurlSetup3D(quad1D, ne, ir->GetWeights(), geom->J,
+                     coeff, pa_data);
+   }
+   else if (test_el->GetDerivType() == mfem::FiniteElement::CURL && dim == 2)
+   {
+      PAHcurlSetup2D(quad1D, ne, ir->GetWeights(), geom->J,
+                     coeff, pa_data);
+   }
+   else
+   {
+      MFEM_ABORT("Unknown kernel.");
+   }
+}
+
+void MixedVectorGradientIntegrator::AddMultPA(const Vector &x, Vector &y) const
+{
+   if (dim == 3)
+      PAHcurlH1Apply3D(dofs1D, quad1D, ne, mapsC->B, mapsC->G,
+                       mapsO->Bt, mapsC->Bt, pa_data, x, y);
+   else if (dim == 2)
+      PAHcurlH1Apply2D(dofs1D, quad1D, ne, mapsC->B, mapsC->G,
+                       mapsO->Bt, mapsC->Bt, pa_data, x, y);
+   else
+   {
+      MFEM_ABORT("Unsupported dimension!");
+   }
+}
+
+} // namespace mfem

fem/coefficient.cpp

@@ -49,7 +49,7 @@ double FunctionCoefficient::Eval(ElementTransformation & T,
 double GridFunctionCoefficient::Eval (ElementTransformation &T,
                                       const IntegrationPoint &ip)
 {
-   return GridF -> GetValue (T.ElementNo, ip, Component);
+   return GridF -> GetValue (T, ip, Component);
 }
 
 double TransformedCoefficient::Eval(ElementTransformation &T,
@@ -160,13 +160,13 @@ void VectorArrayCoefficient::Eval(Vector &V, ElementTransformation &T,
 }
 
 VectorGridFunctionCoefficient::VectorGridFunctionCoefficient (
-   GridFunction *gf)
+   const GridFunction *gf)
    : VectorCoefficient ((gf) ? gf -> VectorDim() : 0)
 {
    GridFunc = gf;
 }
 
-void VectorGridFunctionCoefficient::SetGridFunction(GridFunction *gf)
+void VectorGridFunctionCoefficient::SetGridFunction(const GridFunction *gf)
 {
    GridFunc = gf; vdim = (gf) ? gf -> VectorDim() : 0;
 }
@@ -174,24 +174,7 @@ void VectorGridFunctionCoefficient::SetGridFunction(GridFunction *gf)
 void VectorGridFunctionCoefficient::Eval(Vector &V, ElementTransformation &T,
                                          const IntegrationPoint &ip)
 {
-   Mesh *mesh = GridFunc->FESpace()->GetMesh();
-   if (mesh->Dimension() == T.GetDimension())
-   {
-      GridFunc->GetVectorValue(T.ElementNo, ip, V);
-   }
-   else // Assuming T is a boundary element transformation
-   {
-      int el_id, el_info;
-      mesh->GetBdrElementAdjacentElement(T.ElementNo, el_id, el_info);
-      IntegrationPointTransformation loc_T;
-      mesh->GetLocalFaceTransformation(mesh->GetBdrElementType(T.ElementNo),
-                                       mesh->GetElementType(el_id),
-                                       loc_T.Transf,
-                                       el_info);
-      IntegrationPoint eip;
-      loc_T.Transform(ip, eip);
-      GridFunc->GetVectorValue(el_id, eip, V);
-   }
+   GridFunc->GetVectorValue(T, ip, V);
 }
 
 void VectorGridFunctionCoefficient::Eval(
@@ -201,14 +184,14 @@ void VectorGridFunctionCoefficient::Eval(
 }
 
 GradientGridFunctionCoefficient::GradientGridFunctionCoefficient (
-   GridFunction *gf)
+   const GridFunction *gf)
    : VectorCoefficient((gf) ?
                        gf -> FESpace() -> GetMesh() -> SpaceDimension() : 0)
 {
    GridFunc = gf;
 }
 
-void GradientGridFunctionCoefficient::SetGridFunction(GridFunction *gf)
+void GradientGridFunctionCoefficient::SetGridFunction(const GridFunction *gf)
 {
    GridFunc = gf; vdim = (gf) ?
                          gf -> FESpace() -> GetMesh() -> SpaceDimension() : 0;
@@ -227,14 +210,14 @@ void GradientGridFunctionCoefficient::Eval(
 }
 
 CurlGridFunctionCoefficient::CurlGridFunctionCoefficient (
-   GridFunction *gf)
+   const GridFunction *gf)
    : VectorCoefficient ((gf) ?
                         gf -> FESpace() -> GetMesh() -> SpaceDimension() : 0)
 {
    GridFunc = gf;
 }
 
-void CurlGridFunctionCoefficient::SetGridFunction(GridFunction *gf)
+void CurlGridFunctionCoefficient::SetGridFunction(const GridFunction *gf)
 {
    GridFunc = gf; vdim = (gf) ?
                          gf -> FESpace() -> GetMesh() -> SpaceDimension() : 0;
@@ -247,7 +230,7 @@ void CurlGridFunctionCoefficient::Eval(Vector &V, ElementTransformation &T,
 }
 
 DivergenceGridFunctionCoefficient::DivergenceGridFunctionCoefficient (
-   GridFunction *gf) : Coefficient()
+   const GridFunction *gf) : Coefficient()
 {
    GridFunc = gf;
 }
@@ -775,4 +758,61 @@ double ComputeGlobalLpNorm(double p, VectorCoefficient &coeff, ParMesh &pmesh,
 }
 #endif
 
+VectorQuadratureFunctionCoefficient::VectorQuadratureFunctionCoefficient(
+   QuadratureFunction &qf)
+   : VectorCoefficient(qf.GetVDim()), QuadF(qf), index(0) { }
+
+void VectorQuadratureFunctionCoefficient::SetComponent(int _index, int _length)
+{
+   MFEM_VERIFY(_index >= 0, "Index must be >= 0");
+   MFEM_VERIFY(_index < QuadF.GetVDim(),
+               "Index must be < QuadratureFunction length");
+   index = _index;
+
+   MFEM_VERIFY(_length > 0, "Length must be > 0");
+   MFEM_VERIFY(_length <= QuadF.GetVDim() - index,
+               "Length must be <= (QuadratureFunction length - index)");
+
+   vdim = _length;
+}
+
+void VectorQuadratureFunctionCoefficient::Eval(Vector &V,
+                                               ElementTransformation &T,
+                                               const IntegrationPoint &ip)
+{
+   QuadF.HostRead();
+
+   if (index == 0 && vdim == QuadF.GetVDim())
+   {
+      QuadF.GetElementValues(T.ElementNo, ip.index, V);
+   }
+   else
+   {
+      Vector temp;
+      QuadF.GetElementValues(T.ElementNo, ip.index, temp);
+      V.SetSize(vdim);
+      for (int i = 0; i < vdim; i++)
+      {
+         V(i) = temp(index + i);
+      }
+   }
+
+   return;
+}
+
+QuadratureFunctionCoefficient::QuadratureFunctionCoefficient(
+   QuadratureFunction &qf) : QuadF(qf)
+{
+   MFEM_VERIFY(qf.GetVDim() == 1, "QuadratureFunction's vdim must be 1");
+}
+
+double QuadratureFunctionCoefficient::Eval(ElementTransformation &T,
+                                           const IntegrationPoint &ip)
+{
+   QuadF.HostRead();
+   Vector temp(1);
+   QuadF.GetElementValues(T.ElementNo, ip.index, temp);
+   return temp[0];
+}
+
 }

fem/coefficient.hpp

@@ -164,18 +164,18 @@ class GridFunction;
 class GridFunctionCoefficient : public Coefficient
 {
 private:
-   GridFunction *GridF;
+   const GridFunction *GridF;
    int Component;
 
 public:
    GridFunctionCoefficient() : GridF(NULL), Component(1) { }
    /** Construct GridFunctionCoefficient from a given GridFunction, and
        optionally specify a component to use if it is a vector GridFunction. */
-   GridFunctionCoefficient (GridFunction *gf, int comp = 1)
+   GridFunctionCoefficient (const GridFunction *gf, int comp = 1)
    { GridF = gf; Component = comp; }
 
-   void SetGridFunction(GridFunction *gf) { GridF = gf; }
-   GridFunction * GetGridFunction() const { return GridF; }
+   void SetGridFunction(const GridFunction *gf) { GridF = gf; }
+   const GridFunction * GetGridFunction() const { return GridF; }
 
    virtual double Eval(ElementTransformation &T,
                        const IntegrationPoint &ip);
@@ -400,14 +400,14 @@ public:
 class VectorGridFunctionCoefficient : public VectorCoefficient
 {
 protected:
-   GridFunction *GridFunc;
+   const GridFunction *GridFunc;
 
 public:
    VectorGridFunctionCoefficient() : VectorCoefficient(0), GridFunc(NULL) { }
-   VectorGridFunctionCoefficient(GridFunction *gf);
+   VectorGridFunctionCoefficient(const GridFunction *gf);
 
-   void SetGridFunction(GridFunction *gf);
-   GridFunction * GetGridFunction() const { return GridFunc; }
+   void SetGridFunction(const GridFunction *gf);
+   const GridFunction * GetGridFunction() const { return GridFunc; }
 
    virtual void Eval(Vector &V, ElementTransformation &T,
                      const IntegrationPoint &ip);
@@ -422,13 +422,13 @@ public:
 class GradientGridFunctionCoefficient : public VectorCoefficient
 {
 protected:
-   GridFunction *GridFunc;
+   const GridFunction *GridFunc;
 
 public:
-   GradientGridFunctionCoefficient(GridFunction *gf);
+   GradientGridFunctionCoefficient(const GridFunction *gf);
 
-   void SetGridFunction(GridFunction *gf);
-   GridFunction * GetGridFunction() const { return GridFunc; }
+   void SetGridFunction(const GridFunction *gf);
+   const GridFunction * GetGridFunction() const { return GridFunc; }
 
    virtual void Eval(Vector &V, ElementTransformation &T,
                      const IntegrationPoint &ip);
@@ -443,13 +443,13 @@ public:
 class CurlGridFunctionCoefficient : public VectorCoefficient
 {
 protected:
-   GridFunction *GridFunc;
+   const GridFunction *GridFunc;
 
 public:
-   CurlGridFunctionCoefficient(GridFunction *gf);
+   CurlGridFunctionCoefficient(const GridFunction *gf);
 
-   void SetGridFunction(GridFunction *gf);
-   GridFunction * GetGridFunction() const { return GridFunc; }
+   void SetGridFunction(const GridFunction *gf);
+   const GridFunction * GetGridFunction() const { return GridFunc; }
 
    using VectorCoefficient::Eval;
    virtual void Eval(Vector &V, ElementTransformation &T,
@@ -462,13 +462,13 @@ public:
 class DivergenceGridFunctionCoefficient : public Coefficient
 {
 protected:
-   GridFunction *GridFunc;
+   const GridFunction *GridFunc;
 
 public:
-   DivergenceGridFunctionCoefficient(GridFunction *gf);
+   DivergenceGridFunctionCoefficient(const GridFunction *gf);
 
-   void SetGridFunction(GridFunction *gf) { GridFunc = gf; }
-   GridFunction * GetGridFunction() const { return GridFunc; }
+   void SetGridFunction(const GridFunction *gf) { GridFunc = gf; }
+   const GridFunction * GetGridFunction() const { return GridFunc; }
 
    virtual double Eval(ElementTransformation &T,
                        const IntegrationPoint &ip);
@@ -946,6 +946,54 @@ public:
                      const IntegrationPoint &ip);
 };
 
+class QuadratureFunction;
+
+/** @brief Vector quadrature function coefficient which requires that the
+   quadrature rules used for this vector coefficient be the same as those that
+   live within the supplied QuadratureFunction. */
+class VectorQuadratureFunctionCoefficient : public VectorCoefficient
+{
+private:
+   const QuadratureFunction &QuadF; //do not own
+   int index;
+
+public:
+   /// Constructor with a quadrature function as input
+   VectorQuadratureFunctionCoefficient(QuadratureFunction &qf);
+
+   /** Set the starting index within the QuadFunc that'll be used to
+      project outwards as well as the corresponding length. The projected length
+      should have the bounds of 1 <= length <= (length QuadFunc - index). */
+   void SetComponent(int _index, int _length);
+
+   const QuadratureFunction& GetQuadFunction() const { return QuadF; }
+
+   using VectorCoefficient::Eval;
+   virtual void Eval(Vector &V, ElementTransformation &T,
+                     const IntegrationPoint &ip);
+
+   virtual ~VectorQuadratureFunctionCoefficient() { }
+};
+
+/** @brief Quadrature function coefficient which requires that the quadrature
+   rules used for this coefficient be the same as those that live within the
+   supplied QuadratureFunction. */
+class QuadratureFunctionCoefficient : public Coefficient
+{
+private:
+   const QuadratureFunction &QuadF;
+
+public:
+   /// Constructor with a quadrature function as input
+   QuadratureFunctionCoefficient(QuadratureFunction &qf);
+
+   const QuadratureFunction& GetQuadFunction() const { return QuadF; }
+
+   virtual double Eval(ElementTransformation &T, const IntegrationPoint &ip);
+
+   virtual ~QuadratureFunctionCoefficient() { }
+};
+
 /** Compute the Lp norm of a function f.
     \f$ \| f \|_{Lp} = ( \int_\Omega | f |^p d\Omega)^{1/p} \f$ */
 double ComputeLpNorm(double p, Coefficient &coeff, Mesh &mesh,

fem/datacollection.cpp

@@ -739,12 +739,6 @@ ParaViewDataCollection::ParaViewDataCollection(const std::string&
 #endif
 }
 
-void ParaViewDataCollection::RegisterField(const std::string& field_name,
-                                           mfem::GridFunction *gf)
-{
-   DataCollection::RegisterField(field_name,gf);
-}
-
 void ParaViewDataCollection::SetLevelsOfDetail(int levels_of_detail_)
 {
    levels_of_detail = levels_of_detail_;
@@ -815,7 +809,7 @@ void ParaViewDataCollection::Save()
    // the directory is created
 
    // create pvd file if needed
-   if (!pvd_stream.is_open())
+   if (myid == 0 && !pvd_stream.is_open())
    {
       std::string dpath=GenerateCollectionPath();
       std::string pvdname=dpath+"/"+GeneratePVDFileName();

fem/datacollection.hpp

@@ -501,10 +501,6 @@ public:
    ParaViewDataCollection(const std::string& collection_name,
                           mfem::Mesh *mesh_ = NULL);
 
-   /// Add a grid function to the collection
-   virtual void RegisterField(const std::string& field_name,
-                              mfem::GridFunction *gf) override;
-
    /// Set refinement levels - every element is uniformly split based on
    /// levels_of_detail_
    void SetLevelsOfDetail(int levels_of_detail_);

fem/eltrans.cpp

@@ -19,6 +19,7 @@ namespace mfem
 ElementTransformation::ElementTransformation()
    : IntPoint(static_cast<IntegrationPoint *>(NULL)),
      EvalState(0),
+     geom(Geometry::INVALID),
      Attribute(-1),
      ElementNo(-1)
 { }
@@ -551,4 +552,76 @@ void IntegrationPointTransformation::Transform (const IntegrationRule &ir1,
    }
 }
 
+void FaceElementTransformations::SetIntPoint(const IntegrationPoint *ip)
+{
+   IsoparametricTransformation::SetIntPoint(ip);
+
+   if (Elem1)
+   {
+      Loc1.Transform(*ip, eip1);
+      Elem1->SetIntPoint(&eip1);
+   }
+   if (Elem2)
+   {
+      Loc2.Transform(*ip, eip2);
+      Elem2->SetIntPoint(&eip2);
+   }
+}
+
+ElementTransformation &
+FaceElementTransformations::GetElement1Transformation()
+{
+   MFEM_VERIFY(mask & 1 && Elem1 != NULL, "The ElementTransformation "
+               "for the element has not been configured for side 1.");
+   return *Elem1;
+}
+
+ElementTransformation &
+FaceElementTransformations::GetElement2Transformation()
+{
+   MFEM_VERIFY(mask & 2 && Elem2 != NULL, "The ElementTransformation "
+               "for the element has not been configured for side 2.");
+   return *Elem2;
+}
+
+IntegrationPointTransformation &
+FaceElementTransformations::GetIntPoint1Transformation()
+{
+   MFEM_VERIFY(mask & 4, "The IntegrationPointTransformation "
+               "for the element has not been configured for side 1.");
+   return Loc1;
+}
+
+IntegrationPointTransformation &
+FaceElementTransformations::GetIntPoint2Transformation()
+{
+   MFEM_VERIFY(mask & 8, "The IntegrationPointTransformation "
+               "for the element has not been configured for side 2.");
+   return Loc2;
+}
+
+void FaceElementTransformations::Transform(const IntegrationPoint &ip,
+                                           Vector &trans)
+{
+   MFEM_VERIFY(mask & 16, "The ElementTransformation "
+               "for the face has not been configured.");
+   IsoparametricTransformation::Transform(ip, trans);
+}
+
+void FaceElementTransformations::Transform(const IntegrationRule &ir,
+                                           DenseMatrix &tr)
+{
+   MFEM_VERIFY(mask & 16, "The ElementTransformation "
+               "for the face has not been configured.");
+   IsoparametricTransformation::Transform(ir, tr);
+}
+
+void FaceElementTransformations::Transform(const DenseMatrix &matrix,
+                                           DenseMatrix &result)
+{
+   MFEM_VERIFY(mask & 16, "The ElementTransformation "
+               "for the face has not been configured.");
+   IsoparametricTransformation::Transform(matrix, result);
+}
+
 }

fem/eltrans.hpp

@@ -48,7 +48,29 @@ protected:
    const DenseMatrix &EvalInverseJ();
 
 public:
-   int Attribute, ElementNo;
+
+   /** This enumeration declares the values stored in
+       ElementTransformation::ElementType and indicates which group of objects
+       the index stored in ElementTransformation::ElementNo refers:
+
+       | ElementType | Range of ElementNo
+       +-------------+-------------------------
+       | ELEMENT     | [0, Mesh::GetNE()     )
+       | BDR_ELEMENT | [0, Mesh::GetNBE()    )
+       | EDGE        | [0, Mesh::GetNEdges() )
+       | FACE        | [0, Mesh::GetNFaces() )
+       | BDR_FACE    | [0, Mesh::GetNBE()    )
+   */
+   enum
+   {
+      ELEMENT     = 1,
+      BDR_ELEMENT = 2,
+      EDGE        = 3,
+      FACE        = 4,
+      BDR_FACE    = 5
+   };
+
+   int Attribute, ElementNo, ElementType;
 
    ElementTransformation();
 
@@ -356,12 +378,58 @@ public:
    void Transform (const IntegrationRule  &, IntegrationRule  &);
 };
 
-class FaceElementTransformations
+class FaceElementTransformations : public IsoparametricTransformation
 {
+private:
+   int mask;
+
+   IntegrationPoint eip1, eip2;
+
 public:
-   int Elem1No, Elem2No, FaceGeom;
-   ElementTransformation *Elem1, *Elem2, *Face;
+   int Elem1No, Elem2No;
+   Geometry::Type &FaceGeom; ///< @deprecated Use GetGeometryType instead
+   ElementTransformation *Elem1, *Elem2;
+   ElementTransformation *Face; ///< @deprecated No longer necessary
    IntegrationPointTransformation Loc1, Loc2;
+
+   FaceElementTransformations() : FaceGeom(geom), Face(this) {}
+
+   /** @brief Method to set the geometry type of the face.
+
+       @note This method is designed to be used when
+       [Par]Mesh::GetFaceTransformation will not be called i.e. when the face
+       transformation will not be needed but the neighboring element
+       transformations will be.  Using this method to override the GeometryType
+       should only be done with great care.
+   */
+   void SetGeometryType(Geometry::Type g) { geom = g; }
+
+   /// Set the mask indicating which portions of the object have been setup
+   /** The argument @a m is a bitmask used in
+       Mesh::GetFaceElementTransformations to indicate which portions of the
+       FaceElement Transformations object have been configured.
+
+       mask &  1: Elem1 is configured
+       mask &  2: Elem2 is configured
+       mask &  4: Loc1 is configured
+       mask &  8: Loc2 is configured
+       mask & 16: The Face transformation itself is configured
+   */
+   void SetConfigurationMask(int m) { mask = m; }
+   int  GetConfigurationMask() const { return mask; }
+
+   /** @brief Set the integration point in the Face and the two neighboring
+       elements, if present. */
+   void SetIntPoint(const IntegrationPoint *ip);
+
+   virtual void Transform(const IntegrationPoint &, Vector &);
+   virtual void Transform(const IntegrationRule &, DenseMatrix &);
+   virtual void Transform(const DenseMatrix &matrix, DenseMatrix &result);
+
+   ElementTransformation & GetElement1Transformation();
+   ElementTransformation & GetElement2Transformation();
+   IntegrationPointTransformation & GetIntPoint1Transformation();
+   IntegrationPointTransformation & GetIntPoint2Transformation();
 };
 
 /*                 Elem1(Loc1(x)) = Face(x) = Elem2(Loc2(x))

fem/estimators.cpp

@@ -50,4 +50,21 @@ void L2ZienkiewiczZhuEstimator::ComputeEstimates()
 
 #endif // MFEM_USE_MPI
 
+void LpErrorEstimator::ComputeEstimates()
+{
+   MFEM_VERIFY(coef != NULL || vcoef != NULL,
+               "LpErrorEstimator has no coefficient!  Call SetCoef first.");
+
+   error_estimates.SetSize(sol->FESpace()->GetMesh()->GetNE());
+   if (coef)
+   {
+      sol->ComputeElementLpErrors(local_norm_p, *coef, error_estimates);
+   }
+   else
+   {
+      sol->ComputeElementLpErrors(local_norm_p, *vcoef, error_estimates);
+   }
+   current_sequence = sol->FESpace()->GetMesh()->GetSequence();
+}
+
 } // namespace mfem

fem/estimators.hpp

@@ -304,6 +304,88 @@ public:
 
 #endif // MFEM_USE_MPI
 
+/** @brief The LpErrorEstimator class compares the solution to a known
+    coefficient.
+
+    This class can be used, for example, to adapt a mesh to a non-trivial
+    initial condition in a time-dependent simulation. It can also be used to
+    force refinement in the neighborhood of small features before switching to a
+    more traditional error estimator.
+
+    The LpErrorEstimator supports either scalar or vector coefficients and works
+    both in serial and in parallel.
+*/
+class LpErrorEstimator : public ErrorEstimator
+{
+protected:
+   long current_sequence;
+   int local_norm_p;
+   Vector error_estimates;
+
+   Coefficient * coef;
+   VectorCoefficient * vcoef;
+   GridFunction * sol;
+
+   /// Check if the mesh of the solution was modified.
+   bool MeshIsModified()
+   {
+      long mesh_sequence = sol->FESpace()->GetMesh()->GetSequence();
+      MFEM_ASSERT(mesh_sequence >= current_sequence, "");
+      return (mesh_sequence > current_sequence);
+   }
+
+   /// Compute the element error estimates.
+   void ComputeEstimates();
+
+public:
+   /** @brief Construct a new LpErrorEstimator object for a scalar field.
+       @param p    Integer which selects which Lp norm to use.
+       @param sol  The GridFunction representation of the scalar field.
+       Note: the coefficient must be set before use with the SetCoef method.
+   */
+   LpErrorEstimator(int p, GridFunction &sol)
+      : current_sequence(-1), local_norm_p(p),
+        error_estimates(0), coef(NULL), vcoef(NULL), sol(&sol) { }
+
+   /** @brief Construct a new LpErrorEstimator object for a scalar field.
+       @param p    Integer which selects which Lp norm to use.
+       @param coef The scalar Coefficient to compare to the solution.
+       @param sol  The GridFunction representation of the scalar field.
+   */
+   LpErrorEstimator(int p, Coefficient &coef, GridFunction &sol)
+      : current_sequence(-1), local_norm_p(p),
+        error_estimates(0), coef(&coef), vcoef(NULL), sol(&sol) { }
+
+   /** @brief Construct a new LpErrorEstimator object for a vector field.
+       @param p    Integer which selects which Lp norm to use.
+       @param coef The vector VectorCoefficient to compare to the solution.
+       @param sol  The GridFunction representation of the vector field.
+   */
+   LpErrorEstimator(int p, VectorCoefficient &coef, GridFunction &sol)
+      : current_sequence(-1), local_norm_p(p),
+        error_estimates(0), coef(NULL), vcoef(&coef), sol(&sol) { }
+
+   /** @brief Set the exponent, p, of the Lp norm used for computing the local
+       element errors. */
+   void SetLocalErrorNormP(int p) { local_norm_p = p; }
+
+   void SetCoef(Coefficient &A) { coef = &A; }
+   void SetCoef(VectorCoefficient &A) { vcoef = &A; }
+
+   /// Reset the error estimator.
+   virtual void Reset() { current_sequence = -1; }
+
+   /// Get a Vector with all element errors.
+   virtual const Vector &GetLocalErrors()
+   {
+      if (MeshIsModified()) { ComputeEstimates(); }
+      return error_estimates;
+   }
+
+   /// Destructor
+   virtual ~LpErrorEstimator() {}
+};
+
 } // namespace mfem
 
 #endif // MFEM_ERROR_ESTIMATORS

fem/fe.cpp

@@ -10287,12 +10287,12 @@ const double RT_QuadrilateralElement::nk[8] =
 RT_QuadrilateralElement::RT_QuadrilateralElement(const int p,
                                                  const int cb_type,
                                                  const int ob_type)
-   : VectorFiniteElement(2, Geometry::SQUARE, 2*(p + 1)*(p + 2), p + 1,
-                         H_DIV, FunctionSpace::Qk),
-     cbasis1d(poly1d.GetBasis(p + 1, VerifyClosed(cb_type))),
-     obasis1d(poly1d.GetBasis(p, VerifyOpen(ob_type))),
-     dof_map(Dof), dof2nk(Dof)
+   : VectorTensorFiniteElement(2, 2*(p + 1)*(p + 2), p + 1, cb_type, ob_type,
+                               H_DIV, DofMapType::L2_DOF_MAP),
+     dof2nk(Dof)
 {
+   dof_map.SetSize(Dof);
+
    const double *cp = poly1d.ClosedPoints(p + 1, cb_type);
    const double *op = poly1d.OpenPoints(p, ob_type);
    const int dof2 = Dof/2;
@@ -10498,12 +10498,12 @@ const double RT_HexahedronElement::nk[18] =
 RT_HexahedronElement::RT_HexahedronElement(const int p,
                                            const int cb_type,
                                            const int ob_type)
-   : VectorFiniteElement(3, Geometry::CUBE, 3*(p + 1)*(p + 1)*(p + 2), p + 1,
-                         H_DIV, FunctionSpace::Qk),
-     cbasis1d(poly1d.GetBasis(p + 1, VerifyClosed(cb_type))),
-     obasis1d(poly1d.GetBasis(p, VerifyOpen(ob_type))),
-     dof_map(Dof), dof2nk(Dof)
+   : VectorTensorFiniteElement(3, 3*(p + 1)*(p + 1)*(p + 2), p + 1, cb_type,
+                               ob_type, H_DIV, DofMapType::L2_DOF_MAP),
+     dof2nk(Dof)
 {
+   dof_map.SetSize(Dof);
+
    const double *cp = poly1d.ClosedPoints(p + 1, cb_type);
    const double *op = poly1d.OpenPoints(p, ob_type);
    const int dof3 = Dof/3;
@@ -11537,7 +11537,8 @@ const DofToQuad &VectorTensorFiniteElement::GetTensorDofToQuad(
 {
    MFEM_VERIFY(mode == DofToQuad::TENSOR, "invalid mode requested");
 
-   for (int i = 0; i < closed ? dof2quad_array.Size() : dof2quad_array_open.Size();
+   for (int i = 0;
+        i < (closed ? dof2quad_array.Size() : dof2quad_array_open.Size());
         i++)
    {
       const DofToQuad &d2q = closed ? *dof2quad_array[i] : *dof2quad_array_open[i];
@@ -11590,6 +11591,14 @@ const DofToQuad &VectorTensorFiniteElement::GetTensorDofToQuad(
    return *d2q;
 }
 
+VectorTensorFiniteElement::~VectorTensorFiniteElement()
+{
+   for (int i = 0; i < dof2quad_array_open.Size(); i++)
+   {
+      delete dof2quad_array_open[i];
+   }
+}
+
 const double ND_QuadrilateralElement::tk[8] =
 { 1.,0.,  0.,1., -1.,0., 0.,-1. };
 

fem/fe.hpp

@@ -1930,6 +1930,8 @@ public:
    const DofToQuad &GetTensorDofToQuad(const IntegrationRule &ir,
                                        DofToQuad::Mode mode,
                                        const bool closed) const;
+
+   ~VectorTensorFiniteElement();
 };
 
 class H1_SegmentElement : public NodalTensorFiniteElement
@@ -2430,17 +2432,16 @@ public:
 };
 
 
-class RT_QuadrilateralElement : public VectorFiniteElement
+class RT_QuadrilateralElement : public VectorTensorFiniteElement
 {
 private:
    static const double nk[8];
 
-   Poly_1D::Basis &cbasis1d, &obasis1d;
 #ifndef MFEM_THREAD_SAFE
    mutable Vector shape_cx, shape_ox, shape_cy, shape_oy;
    mutable Vector dshape_cx, dshape_cy;
 #endif
-   Array<int> dof_map, dof2nk;
+   Array<int> dof2nk;
 
 public:
    RT_QuadrilateralElement(const int p,
@@ -2486,16 +2487,15 @@ public:
 };
 
 
-class RT_HexahedronElement : public VectorFiniteElement
+class RT_HexahedronElement : public VectorTensorFiniteElement
 {
    static const double nk[18];
 
-   Poly_1D::Basis &cbasis1d, &obasis1d;
 #ifndef MFEM_THREAD_SAFE
    mutable Vector shape_cx, shape_ox, shape_cy, shape_oy, shape_cz, shape_oz;
    mutable Vector dshape_cx, dshape_cy, dshape_cz;
 #endif
-   Array<int> dof_map, dof2nk;
+   Array<int> dof2nk;
 
 public:
    RT_HexahedronElement(const int p,

fem/fe_coll.hpp

@@ -40,6 +40,15 @@ protected:
                               const int face_info);
 
 public:
+   /** @brief Enumeration for ContType: defines the continuity of the field
+       across element interfaces.
+   */
+   enum { CONTINUOUS,   ///< Field is continuous across element interfaces
+          TANGENTIAL,   ///< Tangential components of vector field
+          NORMAL,       ///< Normal component of vector field
+          DISCONTINUOUS ///< Field is discontinuous across element interfaces
+        };
+
    virtual const FiniteElement *
    FiniteElementForGeometry(Geometry::Type GeomType) const = 0;
 
@@ -52,6 +61,8 @@ public:
 
    virtual const char * Name() const { return "Undefined"; }
 
+   virtual int GetContType() const = 0;
+
    int HasFaceDofs(Geometry::Type GeomType) const;
 
    virtual const FiniteElement *TraceFiniteElementForGeometry(
@@ -102,6 +113,7 @@ public:
    virtual const int *DofOrderForOrientation(Geometry::Type GeomType,
                                              int Or) const;
    virtual const char *Name() const { return h1_name; }
+   virtual int GetContType() const { return CONTINUOUS; }
    FiniteElementCollection *GetTraceCollection() const;
 
    int GetBasisType() const { return b_type; }
@@ -174,6 +186,8 @@ public:
                                              int Or) const;
    virtual const char *Name() const { return d_name; }
 
+   virtual int GetContType() const { return DISCONTINUOUS; }
+
    virtual const FiniteElement *TraceFiniteElementForGeometry(
       Geometry::Type GeomType) const
    {
@@ -221,6 +235,7 @@ public:
    virtual const int *DofOrderForOrientation(Geometry::Type GeomType,
                                              int Or) const;
    virtual const char *Name() const { return rt_name; }
+   virtual int GetContType() const { return NORMAL; }
    FiniteElementCollection *GetTraceCollection() const;
 
    virtual ~RT_FECollection();
@@ -270,6 +285,7 @@ public:
    virtual const int *DofOrderForOrientation(Geometry::Type GeomType,
                                              int Or) const;
    virtual const char *Name() const { return nd_name; }
+   virtual int GetContType() const { return TANGENTIAL; }
    FiniteElementCollection *GetTraceCollection() const;
 
    virtual ~ND_FECollection();
@@ -334,6 +350,8 @@ public:
 
    virtual const char *Name() const { return name; }
 
+   virtual int GetContType() const { return CONTINUOUS; }
+
    FiniteElementCollection *GetTraceCollection() const;
 
    virtual ~NURBSFECollection();
@@ -363,6 +381,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "Linear"; }
+
+   virtual int GetContType() const { return CONTINUOUS; }
 };
 
 /// Piecewise-(bi)quadratic continuous finite elements.
@@ -389,6 +409,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "Quadratic"; }
+
+   virtual int GetContType() const { return CONTINUOUS; }
 };
 
 /// Version of QuadraticFECollection with positive basis functions.
@@ -410,6 +432,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "QuadraticPos"; }
+
+   virtual int GetContType() const { return CONTINUOUS; }
 };
 
 /// Piecewise-(bi)cubic continuous finite elements.
@@ -437,6 +461,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "Cubic"; }
+
+   virtual int GetContType() const { return CONTINUOUS; }
 };
 
 /// Crouzeix-Raviart nonconforming elements in 2D.
@@ -458,6 +484,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "CrouzeixRaviart"; }
+
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Piecewise-linear nonconforming finite elements in 3D.
@@ -481,6 +509,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "LinearNonConf3D"; }
+
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 
@@ -504,6 +534,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RT0_2D"; }
+
+   virtual int GetContType() const { return NORMAL; }
 };
 
 /** Second order Raviart-Thomas finite elements in 2D. This class is kept only
@@ -526,6 +558,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RT1_2D"; }
+
+   virtual int GetContType() const { return NORMAL; }
 };
 
 /** Third order Raviart-Thomas finite elements in 2D. This class is kept only
@@ -548,6 +582,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RT2_2D"; }
+
+   virtual int GetContType() const { return NORMAL; }
 };
 
 /** Piecewise-constant discontinuous finite elements in 2D. This class is kept
@@ -569,6 +605,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "Const2D"; }
+
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /** Piecewise-linear discontinuous finite elements in 2D. This class is kept
@@ -591,6 +629,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "LinearDiscont2D"; }
+
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Version of LinearDiscont2DFECollection with dofs in the Gaussian points.
@@ -613,6 +653,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "GaussLinearDiscont2D"; }
+
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Linear (P1) finite elements on quadrilaterals.
@@ -628,6 +670,7 @@ public:
    virtual const int *DofOrderForOrientation(Geometry::Type GeomType,
                                              int Or) const;
    virtual const char * Name() const { return "P1OnQuad"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /** Piecewise-quadratic discontinuous finite elements in 2D. This class is kept
@@ -650,6 +693,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "QuadraticDiscont2D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Version of QuadraticDiscont2DFECollection with positive basis functions.
@@ -667,6 +711,7 @@ public:
                                              int Or) const
    { return NULL; }
    virtual const char * Name() const { return "QuadraticPosDiscont2D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Version of QuadraticDiscont2DFECollection with dofs in the Gaussian points.
@@ -689,6 +734,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "GaussQuadraticDiscont2D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /** Piecewise-cubic discontinuous finite elements in 2D. This class is kept
@@ -711,6 +757,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "CubicDiscont2D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /** Piecewise-constant discontinuous finite elements in 3D. This class is kept
@@ -734,6 +781,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "Const3D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /** Piecewise-linear discontinuous finite elements in 3D. This class is kept
@@ -756,6 +804,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "LinearDiscont3D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /** Piecewise-quadratic discontinuous finite elements in 3D. This class is kept
@@ -778,6 +827,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "QuadraticDiscont3D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Finite element collection on a macro-element.
@@ -803,6 +853,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RefinedLinear"; }
+   virtual int GetContType() const { return CONTINUOUS; }
 };
 
 /** Lowest order Nedelec finite elements in 3D. This class is kept only for
@@ -825,6 +876,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "ND1_3D"; }
+   virtual int GetContType() const { return TANGENTIAL; }
 };
 
 /** First order Raviart-Thomas finite elements in 3D. This class is kept only
@@ -848,6 +900,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RT0_3D"; }
+   virtual int GetContType() const { return NORMAL; }
 };
 
 /** Second order Raviart-Thomas finite elements in 3D. This class is kept only
@@ -870,6 +923,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RT1_3D"; }
+   virtual int GetContType() const { return NORMAL; }
 };
 
 /// Discontinuous collection defined locally by a given finite element.
@@ -894,6 +948,7 @@ public:
    virtual const char *Name() const { return d_name; }
 
    virtual ~Local_FECollection() { delete Local_Element; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 }

fem/fespace.cpp

@@ -2615,7 +2615,9 @@ const Operator &InterpolationGridTransfer::BackwardOperator()
 
 L2ProjectionGridTransfer::L2Projection::L2Projection(
    const FiniteElementSpace &fes_ho_, const FiniteElementSpace &fes_lor_)
-   : fes_ho(fes_ho_), fes_lor(fes_lor_)
+   : Operator(fes_lor_.GetVSize(), fes_ho_.GetVSize()),
+     fes_ho(fes_ho_),
+     fes_lor(fes_lor_)
 {
    Mesh *mesh_ho = fes_ho.GetMesh();
    MFEM_VERIFY(mesh_ho->GetNumGeometries(mesh_ho->Dimension()) <= 1,

fem/fespace.hpp

@@ -906,7 +906,8 @@ protected:
       const L2Projection &l2proj;
 
    public:
-      L2Prolongation(const L2Projection &l2proj_) : l2proj(l2proj_) { }
+      L2Prolongation(const L2Projection &l2proj_)
+         : Operator(l2proj_.Width(), l2proj_.Height()), l2proj(l2proj_) { }
       void Mult(const Vector &x, Vector &y) const
       {
          l2proj.Prolongate(x, y);

fem/gridfunc.cpp

@@ -236,7 +236,6 @@ void GridFunction::MakeTRef(FiniteElementSpace *f, Vector &tv, int tv_offset)
    }
 }
 
-
 void GridFunction::SumFluxAndCount(BilinearFormIntegrator &blfi,
                                    GridFunction &flux,
                                    Array<int>& count,
@@ -617,17 +616,354 @@ int GridFunction::GetFaceValues(int i, int side, const IntegrationRule &ir,
    return dir;
 }
 
+void GridFunction::GetVectorValues(int i, const IntegrationRule &ir,
+                                   DenseMatrix &vals, DenseMatrix &tr) const
+{
+   ElementTransformation *Tr = fes->GetElementTransformation(i);
+   Tr->Transform(ir, tr);
+
+   GetVectorValues(*Tr, ir, vals);
+}
+
+void be_to_bfe(Geometry::Type geom, int o, const IntegrationPoint &ip,
+               IntegrationPoint &fip)
+{
+   if (geom == Geometry::TRIANGLE)
+   {
+      if (o == 2)
+      {
+         fip.x = 1.0 - ip.x - ip.y;
+         fip.y = ip.x;
+      }
+      else if (o == 4)
+      {
+         fip.x = ip.y;
+         fip.y = 1.0 - ip.x - ip.y;
+      }
+      else
+      {
+         fip.x = ip.x;
+         fip.y = ip.y;
+      }
+      fip.z = ip.z;
+   }
+   else
+   {
+      if (o == 2)
+      {
+         fip.x = ip.y;
+         fip.y = 1.0 - ip.x;
+      }
+      else if (o == 4)
+      {
+         fip.x = 1.0 - ip.x;
+         fip.y = 1.0 - ip.y;
+      }
+      else if (o == 6)
+      {
+         fip.x = 1.0 - ip.y;
+         fip.y = ip.x;
+      }
+      else
+      {
+         fip.x = ip.x;
+         fip.y = ip.y;
+      }
+      fip.z = ip.z;
+   }
+   fip.weight = ip.weight;
+   fip.index  = ip.index;
+}
+
+double GridFunction::GetValue(ElementTransformation &T,
+                              const IntegrationPoint &ip,
+                              int comp, Vector *tr) const
+{
+   if (tr)
+   {
+      T.SetIntPoint(&ip);
+      T.Transform(ip, *tr);
+   }
+
+   const FiniteElement * fe = NULL;
+   Array<int> dofs;
+
+   switch (T.ElementType)
+   {
+      case ElementTransformation::ELEMENT:
+         fe = fes->GetFE(T.ElementNo);
+         fes->GetElementDofs(T.ElementNo, dofs);
+         break;
+      case ElementTransformation::EDGE:
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            fe = fes->GetEdgeElement(T.ElementNo);
+            fes->GetEdgeDofs(T.ElementNo, dofs);
+         }
+         else
+         {
+            MFEM_ABORT("GridFunction::GetValue: Field continuity type \""
+                       << fes->FEColl()->GetContType() << "\" not supported "
+                       << "on mesh edges.");
+            return NAN;
+         }
+         break;
+      case ElementTransformation::FACE:
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            fe = fes->GetFaceElement(T.ElementNo);
+            fes->GetFaceDofs(T.ElementNo, dofs);
+         }
+         else
+         {
+            MFEM_ABORT("GridFunction::GetValue: Field continuity type \""
+                       << fes->FEColl()->GetContType() << "\" not supported "
+                       << "on mesh faces.");
+            return NAN;
+         }
+         break;
+      case ElementTransformation::BDR_ELEMENT:
+      {
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            // This is a continuous field so we can evaluate it on the boundary.
+            fe = fes->GetBE(T.ElementNo);
+            fes->GetBdrElementDofs(T.ElementNo, dofs);
+         }
+         else
+         {
+            // This is a discontinuous field which cannot be evaluated on the
+            // boundary so we'll evaluate it in the neighboring element.
+            FaceElementTransformations * FET =
+               fes->GetMesh()->GetBdrFaceTransformations(T.ElementNo);
+
+            // Boundary elements and Boundary Faces may have different
+            // orientations so adjust the integration point if necessary.
+            int o = 0;
+            if (fes->GetMesh()->Dimension() == 3)
+            {
+               int f;
+               fes->GetMesh()->GetBdrElementFace(T.ElementNo, &f, &o);
+            }
+
+            IntegrationPoint fip;
+            be_to_bfe(FET->GetGeometryType(), o, ip, fip);
+
+            FET->SetIntPoint(&fip);
+            ElementTransformation & T1 = FET->GetElement1Transformation();
+            return GetValue(T1, T1.GetIntPoint(), comp);
+         }
+         break;
+      }
+      case ElementTransformation::BDR_FACE:
+      {
+         FaceElementTransformations * FET =
+            dynamic_cast<FaceElementTransformations *>(&T);
+
+         // Evaluate in neighboring element for both continuous and
+         // discontinuous fields.
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+         return GetValue(T1, T1.GetIntPoint(), comp);
+      }
+      default:
+      {
+         MFEM_ABORT("GridFunction::GetValue: Unsupported element type \""
+                    << T.ElementType << "\"");
+         return NAN;
+      }
+   }
+
+   fes->DofsToVDofs(comp-1, dofs);
+   Vector DofVal(dofs.Size()), LocVec;
+   if (fe->GetMapType() == FiniteElement::VALUE)
+   {
+      fe->CalcShape(ip, DofVal);
+   }
+   else
+   {
+      fe->CalcPhysShape(T, DofVal);
+   }
+   GetSubVector(dofs, LocVec);
+
+   return (DofVal * LocVec);
+}
+
+void GridFunction::GetValues(ElementTransformation &T,
+                             const IntegrationRule &ir,
+                             Vector &vals, int comp,
+                             DenseMatrix *tr) const
+{
+   if (tr)
+   {
+      T.Transform(ir, *tr);
+   }
+
+   int nip = ir.GetNPoints();
+   vals.SetSize(nip);
+   for (int j = 0; j < nip; j++)
+   {
+      const IntegrationPoint &ip = ir.IntPoint(j);
+      T.SetIntPoint(&ip);
+      vals[j] = GetValue(T, ip, comp);
+   }
+}
+
+void GridFunction::GetVectorValue(ElementTransformation &T,
+                                  const IntegrationPoint &ip,
+                                  Vector &val, Vector *tr) const
+{
+   if (tr)
+   {
+      T.SetIntPoint(&ip);
+      T.Transform(ip, *tr);
+   }
+
+   Array<int> vdofs;
+   const FiniteElement *fe = NULL;
+
+   switch (T.ElementType)
+   {
+      case ElementTransformation::ELEMENT:
+         fes->GetElementVDofs(T.ElementNo, vdofs);
+         fe = fes->GetFE(T.ElementNo);
+         break;
+      case ElementTransformation::EDGE:
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            fe = fes->GetEdgeElement(T.ElementNo);
+            fes->GetEdgeVDofs(T.ElementNo, vdofs);
+         }
+         else
+         {
+            MFEM_ABORT("GridFunction::GetVectorValue: Field continuity type \""
+                       << fes->FEColl()->GetContType() << "\" not supported "
+                       << "on mesh edges.");
+            return;
+         }
+         break;
+      case ElementTransformation::FACE:
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            fe = fes->GetFaceElement(T.ElementNo);
+            fes->GetFaceVDofs(T.ElementNo, vdofs);
+         }
+         else
+         {
+            MFEM_ABORT("GridFunction::GetVectorValue: Field continuity type \""
+                       << fes->FEColl()->GetContType() << "\" not supported "
+                       << "on mesh faces.");
+            return;
+         }
+         break;
+      case ElementTransformation::BDR_ELEMENT:
+      {
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            // This is a continuous field so we can evaluate it on the boundary.
+            fes->GetBdrElementVDofs(T.ElementNo, vdofs);
+            fe = fes->GetBE(T.ElementNo);
+         }
+         else
+         {
+            // This is a discontinuous vector field which cannot be evaluated on
+            // the boundary so we'll evaluate it in the neighboring element.
+            FaceElementTransformations * FET =
+               fes->GetMesh()->GetBdrFaceTransformations(T.ElementNo);
+
+            // Boundary elements and Boundary Faces may have different
+            // orientations so adjust the integration point if necessary.
+            int o = 0;
+            if (fes->GetMesh()->Dimension() == 3)
+            {
+               int f;
+               fes->GetMesh()->GetBdrElementFace(T.ElementNo, &f, &o);
+            }
+
+            IntegrationPoint fip;
+            be_to_bfe(FET->GetGeometryType(), o, ip, fip);
+
+            FET->SetIntPoint(&fip);
+            ElementTransformation & T1 = FET->GetElement1Transformation();
+            return GetVectorValue(T1, T1.GetIntPoint(), val);
+         }
+         break;
+      }
+      case ElementTransformation::BDR_FACE:
+      {
+         FaceElementTransformations * FET =
+            dynamic_cast<FaceElementTransformations *>(&T);
+
+         // Evaluate in neighboring element for both continuous and
+         // discontinuous fields.
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+         return GetVectorValue(T1, T1.GetIntPoint(), val);
+      }
+      default:
+      {
+         MFEM_ABORT("GridFunction::GetVectorValue: Unsupported element type \""
+                    << T.ElementType << "\"");
+         if (val.Size() > 0) { val = NAN; }
+         return;
+      }
+   }
+
+   int dof = fe->GetDof();
+   Vector loc_data;
+   GetSubVector(vdofs, loc_data);
+   if (fe->GetRangeType() == FiniteElement::SCALAR)
+   {
+      Vector shape(dof);
+      if (fe->GetMapType() == FiniteElement::VALUE)
+      {
+         fe->CalcShape(ip, shape);
+      }
+      else
+      {
+         fe->CalcPhysShape(T, shape);
+      }
+      int vdim = fes->GetVDim();
+      val.SetSize(vdim);
+      for (int k = 0; k < vdim; k++)
+      {
+         val(k) = shape * ((const double *)loc_data + dof * k);
+      }
+   }
+   else
+   {
+      int spaceDim = fes->GetMesh()->SpaceDimension();
+      DenseMatrix vshape(dof, spaceDim);
+      fe->CalcVShape(T, vshape);
+      val.SetSize(spaceDim);
+      vshape.MultTranspose(loc_data, val);
+   }
+}
+
 void GridFunction::GetVectorValues(ElementTransformation &T,
                                    const IntegrationRule &ir,
-                                   DenseMatrix &vals) const
+                                   DenseMatrix &vals,
+                                   DenseMatrix *tr) const
 {
+   if (tr)
+   {
+      T.Transform(ir, *tr);
+   }
+
    const FiniteElement *FElem = fes->GetFE(T.ElementNo);
    int dof = FElem->GetDof();
+
    Array<int> vdofs;
    fes->GetElementVDofs(T.ElementNo, vdofs);
+
    Vector loc_data;
    GetSubVector(vdofs, loc_data);
    int nip = ir.GetNPoints();
+
    if (FElem->GetRangeType() == FiniteElement::SCALAR)
    {
       MFEM_ASSERT(FElem->GetMapType() == FiniteElement::VALUE,
@@ -639,6 +975,7 @@ void GridFunction::GetVectorValues(ElementTransformation &T,
       {
          const IntegrationPoint &ip = ir.IntPoint(j);
          FElem->CalcShape(ip, shape);
+
          for (int k = 0; k < vdim; k++)
          {
             vals(k,j) = shape * ((const double *)loc_data + dof * k);
@@ -649,28 +986,22 @@ void GridFunction::GetVectorValues(ElementTransformation &T,
    {
       int spaceDim = fes->GetMesh()->SpaceDimension();
       DenseMatrix vshape(dof, spaceDim);
+
       vals.SetSize(spaceDim, nip);
       Vector val_j;
+
       for (int j = 0; j < nip; j++)
       {
          const IntegrationPoint &ip = ir.IntPoint(j);
          T.SetIntPoint(&ip);
          FElem->CalcVShape(T, vshape);
+
          vals.GetColumnReference(j, val_j);
          vshape.MultTranspose(loc_data, val_j);
       }
    }
 }
 
-void GridFunction::GetVectorValues(int i, const IntegrationRule &ir,
-                                   DenseMatrix &vals, DenseMatrix &tr) const
-{
-   ElementTransformation *Tr = fes->GetElementTransformation(i);
-   Tr->Transform(ir, tr);
-
-   GetVectorValues(*Tr, ir, vals);
-}
-
 int GridFunction::GetFaceVectorValues(
    int i, int side, const IntegrationRule &ir,
    DenseMatrix &vals, DenseMatrix &tr) const
@@ -702,13 +1033,13 @@ int GridFunction::GetFaceVectorValues(
    {
       Transf = fes->GetMesh()->GetFaceElementTransformations(i, 4);
       Transf->Loc1.Transform(ir, eir);
-      GetVectorValues(Transf->Elem1No, eir, vals, tr);
+      GetVectorValues(*Transf->Elem1, eir, vals, &tr);
    }
    else
    {
       Transf = fes->GetMesh()->GetFaceElementTransformations(i, 8);
       Transf->Loc2.Transform(ir, eir);
-      GetVectorValues(Transf->Elem2No, eir, vals, tr);
+      GetVectorValues(*Transf->Elem2, eir, vals, &tr);
    }
 
    return di;
@@ -1716,6 +2047,8 @@ void GridFunction::ProjectCoefficient(
    ElementTransformation *T = NULL;
    const FiniteElement *fe = NULL;
 
+   fes->BuildDofToArrays(); // ensures GetElementForDof(), GetLocalDofForDof() initialized.
+
    for (int i = 0; i < dofs.Size(); i++)
    {
       int dof = dofs[i], j = fes->GetElementForDof(dof);
@@ -1757,6 +2090,8 @@ void GridFunction::ProjectCoefficient(
 
    Vector val;
 
+   fes->BuildDofToArrays(); // ensures GetElementForDof(), GetLocalDofForDof() initialized.
+
    for (int i = 0; i < dofs.Size(); i++)
    {
       int dof = dofs[i], j = fes->GetElementForDof(dof);
@@ -2009,7 +2344,7 @@ double GridFunction::ComputeL2Error(
       fdof = fe->GetDof();
       transf = fes->GetElementTransformation(i);
       shape.SetSize(fdof);
-      intorder = 2*fe->GetOrder() + 1; // <----------
+      intorder = 2*fe->GetOrder() + 3; // <----------
       const IntegrationRule *ir;
       if (irs)
       {
@@ -2064,7 +2399,7 @@ double GridFunction::ComputeL2Error(
    {
       if (elems != NULL && (*elems)[i] == 0) { continue; }
       fe = fes->GetFE(i);
-      int intorder = 2*fe->GetOrder() + 1; // <----------
+      int intorder = 2*fe->GetOrder() + 3; // <----------
       const IntegrationRule *ir;
       if (irs)
       {
@@ -2168,7 +2503,7 @@ double GridFunction::ComputeH1Error(
             }
          intorder = 2 * intorder;  // <-------------
          const IntegrationRule &ir =
-            IntRules.Get(face_elem_transf->FaceGeom, intorder);
+            IntRules.Get(face_elem_transf->GetGeometryType(), intorder);
          err_val.SetSize(ir.GetNPoints());
          ell_coeff_val.SetSize(ir.GetNPoints());
          // side 1
@@ -2225,7 +2560,7 @@ double GridFunction::ComputeH1Error(
             }
          }
          face_elem_transf = mesh->GetFaceElementTransformations(i, 16);
-         transf = face_elem_transf->Face;
+         transf = face_elem_transf;
          for (j = 0; j < ir.GetNPoints(); j++)
          {
             const IntegrationPoint &ip = ir.IntPoint(j);
@@ -2259,7 +2594,7 @@ double GridFunction::ComputeMaxError(
       fdof = fe->GetDof();
       transf = fes->GetElementTransformation(i);
       shape.SetSize(fdof);
-      intorder = 2*fe->GetOrder() + 1; // <----------
+      intorder = 2*fe->GetOrder() + 3; // <----------
       const IntegrationRule *ir;
       if (irs)
       {
@@ -2425,7 +2760,7 @@ double GridFunction::ComputeLpError(const double p, Coefficient &exsol,
       }
       else
       {
-         int intorder = 2*fe->GetOrder() + 1; // <----------
+         int intorder = 2*fe->GetOrder() + 3; // <----------
          ir = &(IntRules.Get(fe->GetGeomType(), intorder));
       }
       GetValues(i, *ir, vals);
@@ -2472,10 +2807,13 @@ double GridFunction::ComputeLpError(const double p, Coefficient &exsol,
 }
 
 void GridFunction::ComputeElementLpErrors(const double p, Coefficient &exsol,
-                                          GridFunction &error,
+                                          Vector &error,
                                           Coefficient *weight,
                                           const IntegrationRule *irs[]) const
 {
+   MFEM_ASSERT(error.Size() == fes->GetNE(),
+               "Incorrect size for result vector");
+
    error = 0.0;
    const FiniteElement *fe;
    ElementTransformation *T;
@@ -2491,7 +2829,7 @@ void GridFunction::ComputeElementLpErrors(const double p, Coefficient &exsol,
       }
       else
       {
-         int intorder = 2*fe->GetOrder() + 1; // <----------
+         int intorder = 2*fe->GetOrder() + 3; // <----------
          ir = &(IntRules.Get(fe->GetGeomType(), intorder));
       }
       GetValues(i, *ir, vals);
@@ -2555,7 +2893,7 @@ double GridFunction::ComputeLpError(const double p, VectorCoefficient &exsol,
       }
       else
       {
-         int intorder = 2*fe->GetOrder() + 1; // <----------
+         int intorder = 2*fe->GetOrder() + 3; // <----------
          ir = &(IntRules.Get(fe->GetGeomType(), intorder));
       }
       T = fes->GetElementTransformation(i);
@@ -2627,11 +2965,14 @@ double GridFunction::ComputeLpError(const double p, VectorCoefficient &exsol,
 
 void GridFunction::ComputeElementLpErrors(const double p,
                                           VectorCoefficient &exsol,
-                                          GridFunction &error,
+                                          Vector &error,
                                           Coefficient *weight,
                                           VectorCoefficient *v_weight,
                                           const IntegrationRule *irs[]) const
 {
+   MFEM_ASSERT(error.Size() == fes->GetNE(),
+               "Incorrect size for result vector");
+
    error = 0.0;
    const FiniteElement *fe;
    ElementTransformation *T;
@@ -2648,7 +2989,7 @@ void GridFunction::ComputeElementLpErrors(const double p,
       }
       else
       {
-         int intorder = 2*fe->GetOrder() + 1; // <----------
+         int intorder = 2*fe->GetOrder() + 3; // <----------
          ir = &(IntRules.Get(fe->GetGeomType(), intorder));
       }
       T = fes->GetElementTransformation(i);
@@ -2658,15 +2999,15 @@ void GridFunction::ComputeElementLpErrors(const double p,
       loc_errs.SetSize(vals.Width());
       if (!v_weight)
       {
-         // compute the lengths of the errors at the integration points
-         // thus the vector norm is rotationally invariant
+         // compute the lengths of the errors at the integration points thus the
+         // vector norm is rotationally invariant
          vals.Norm2(loc_errs);
       }
       else
       {
          v_weight->Eval(exact_vals, *T, *ir);
-         // column-wise dot product of the vector error (in vals) and the
-         // vector weight (in exact_vals)
+         // column-wise dot product of the vector error (in vals) and the vector
+         // weight (in exact_vals)
          for (int j = 0; j < vals.Width(); j++)
          {
             double err = 0.0;
@@ -2798,7 +3139,9 @@ void GridFunction::SaveVTK(std::ostream &out, const std::string &field_name,
          RefG = GlobGeometryRefiner.Refine(
                    mesh->GetElementBaseGeometry(i), ref, 1);
 
-         GetVectorValues(i, RefG->RefPts, vval, pmat);
+         // GetVectorValues(i, RefG->RefPts, vval, pmat);
+         ElementTransformation * T = mesh->GetElementTransformation(i);
+         GetVectorValues(*T, RefG->RefPts, vval, &pmat);
 
          for (int j = 0; j < vval.Width(); j++)
          {

fem/gridfunc.hpp

@@ -144,17 +144,133 @@ public:
    /// Returns the values in the vertices of i'th element for dimension vdim.
    void GetNodalValues(int i, Array<double> &nval, int vdim = 1) const;
 
+   /** @name Element index Get Value Methods
+
+       These methods take an element index and return the interpolated value of
+       the field at a given reference point within the element.
+
+       @warning These methods retrieve and use the ElementTransformation object
+       from the mfem::Mesh. This can alter the state of the element
+       transformation object and can also lead to unexpected results when the
+       ElementTransformation object is already in use such as when these methods
+       are called from within an integration loop. Consider using
+       GetValue(ElementTransformation &T, ...) instead.
+   */
+   ///@{
+   /** Return a scalar value from within the given element. */
    virtual double GetValue(int i, const IntegrationPoint &ip,
                            int vdim = 1) const;
 
+   /** Return a vector value from within the given element. */
    void GetVectorValue(int i, const IntegrationPoint &ip, Vector &val) const;
+   ///@}
 
+   /** @name Element Index Get Values Methods
+
+       These are convenience methods for repeatedly calling GetValue for
+       multiple points within a given element. The GetValues methods are
+       optimized and should perform better than repeatedly calling GetValue. The
+       GetVectorValues method simply calls GetVectorValue repeatedly.
+
+       @warning These methods retrieve and use the ElementTransformation object
+       from the mfem::Mesh. This can alter the state of the element
+       transformation object and can also lead to unexpected results when the
+       ElementTransformation object is already in use such as when these methods
+       are called from within an integration loop. Consider using
+       GetValues(ElementTransformation &T, ...) instead.
+   */
+   ///@{
+   /** Compute a collection of scalar values from within the element indicated
+       by the index i. */
    void GetValues(int i, const IntegrationRule &ir, Vector &vals,
                   int vdim = 1) const;
 
+   /** Compute a collection of vector values from within the element indicated
+       by the index i. */
    void GetValues(int i, const IntegrationRule &ir, Vector &vals,
                   DenseMatrix &tr, int vdim = 1) const;
 
+   void GetVectorValues(int i, const IntegrationRule &ir,
+                        DenseMatrix &vals, DenseMatrix &tr) const;
+   ///@}
+
+   /** @name ElementTransformation Get Value Methods
+
+       These member functions are designed for use within
+       GridFunctionCoefficient objects. These can be used with
+       ElementTransformation objects coming from either
+       Mesh::GetElementTransformation() or Mesh::GetBdrElementTransformation().
+
+       @note These methods do not reset the ElementTransformation object so they
+       should be safe to use within integration loops or other contexts where
+       the ElementTransformation is already in use.
+   */
+   ///@{
+   /** Return a scalar value from within the element indicated by the
+       ElementTransformation Object. */
+   double GetValue(ElementTransformation &T, const IntegrationPoint &ip,
+                   int comp = 0, Vector *tr = NULL) const;
+
+   /** Return a vector value from within the element indicated by the
+       ElementTransformation Object. */
+   void GetVectorValue(ElementTransformation &T, const IntegrationPoint &ip,
+                       Vector &val, Vector *tr = NULL) const;
+   ///@}
+
+   /** @name ElementTransformation Get Values Methods
+
+       These are convenience methods for repeatedly calling GetValue for
+       multiple points within a given element. They work by calling either the
+       ElementTransformation or FaceElementTransformations versions described
+       above. Consequently, these methods should not be expected to run faster
+       than calling the above methods in an external loop.
+
+       @note These methods do not reset the ElementTransformation object so they
+       should be safe to use within integration loops or other contexts where
+       the ElementTransformation is already in use.
+
+       @note These methods can also be used with FaceElementTransformations
+       objects.
+    */
+   ///@{
+   /** Compute a collection of scalar values from within the element indicated
+       by the ElementTransformation object. */
+   void GetValues(ElementTransformation &T, const IntegrationRule &ir,
+                  Vector &vals, int comp = 0, DenseMatrix *tr = NULL) const;
+
+   /** Compute a collection of vector values from within the element indicated
+       by the ElementTransformation object. */
+   void GetVectorValues(ElementTransformation &T, const IntegrationRule &ir,
+                        DenseMatrix &vals, DenseMatrix *tr = NULL) const;
+   ///@}
+
+   /** @name Face Index Get Values Methods
+
+       These methods are designed to work with Discontinuous Galerkin basis
+       functions. They compute field values on the interface between elements,
+       or on boundary elements, by interpolating the field in a neighboring
+       element. The \a side argument indices which neighboring element should be
+       used: 0, 1, or 2 (automatically chosen).
+
+       @warning These methods retrieve and use the FaceElementTransformations
+       object from the mfem::Mesh. This can alter the state of the face element
+       transformations object and can also lead to unexpected results when the
+       FaceElementTransformations object is already in use such as when these
+       methods are called from within an integration loop. Consider using
+       GetValues(ElementTransformation &T, ...) instead.
+    */
+   ///@{
+   /** Compute a collection of scalar values from within the face
+       indicated by the index i. */
+   int GetFaceValues(int i, int side, const IntegrationRule &ir, Vector &vals,
+                     DenseMatrix &tr, int vdim = 1) const;
+
+   /** Compute a collection of vector values from within the face
+       indicated by the index i. */
+   int GetFaceVectorValues(int i, int side, const IntegrationRule &ir,
+                           DenseMatrix &vals, DenseMatrix &tr) const;
+   ///@}
+
    void GetLaplacians(int i, const IntegrationRule &ir, Vector &laps,
                       int vdim = 1) const;
 
@@ -167,18 +283,6 @@ public:
    void GetHessians(int i, const IntegrationRule &ir, DenseMatrix &hess,
                     DenseMatrix &tr, int vdim = 1) const;
 
-   int GetFaceValues(int i, int side, const IntegrationRule &ir, Vector &vals,
-                     DenseMatrix &tr, int vdim = 1) const;
-
-   void GetVectorValues(ElementTransformation &T, const IntegrationRule &ir,
-                        DenseMatrix &vals) const;
-
-   void GetVectorValues(int i, const IntegrationRule &ir,
-                        DenseMatrix &vals, DenseMatrix &tr) const;
-
-   int GetFaceVectorValues(int i, int side, const IntegrationRule &ir,
-                           DenseMatrix &vals, DenseMatrix &tr) const;
-
    void GetValuesFrom(const GridFunction &orig_func);
 
    void GetBdrValuesFrom(const GridFunction &orig_func);
@@ -236,12 +340,10 @@ public:
 
    virtual void ProjectCoefficient(Coefficient &coeff);
 
-   // call fes -> BuildDofToArrays() before using this projection
    void ProjectCoefficient(Coefficient &coeff, Array<int> &dofs, int vd = 0);
 
    void ProjectCoefficient(VectorCoefficient &vcoeff);
 
-   // call fes -> BuildDofToArrays() before using this projection
    void ProjectCoefficient(VectorCoefficient &vcoeff, Array<int> &dofs);
 
    void ProjectCoefficient(Coefficient *coeff[]);
@@ -365,28 +467,28 @@ public:
                                  const IntegrationRule *irs[] = NULL) const;
 
    /** Compute the Lp error in each element of the mesh and store the results in
-       the GridFunction @a error. The result should be an L2 GridFunction of
-       order zero using map type VALUE. */
+       the Vector @a error. The result should be of length number of elements,
+       for example an L2 GridFunction of order zero using map type VALUE. */
    virtual void ComputeElementLpErrors(const double p, Coefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        Coefficient *weight = NULL,
                                        const IntegrationRule *irs[] = NULL
                                       ) const;
 
    virtual void ComputeElementL1Errors(Coefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        const IntegrationRule *irs[] = NULL
                                       ) const
    { ComputeElementLpErrors(1.0, exsol, error, NULL, irs); }
 
    virtual void ComputeElementL2Errors(Coefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        const IntegrationRule *irs[] = NULL
                                       ) const
    { ComputeElementLpErrors(2.0, exsol, error, NULL, irs); }
 
    virtual void ComputeElementMaxErrors(Coefficient &exsol,
-                                        GridFunction &error,
+                                        Vector &error,
                                         const IntegrationRule *irs[] = NULL
                                        ) const
    { ComputeElementLpErrors(infinity(), exsol, error, NULL, irs); }
@@ -400,29 +502,29 @@ public:
                                  const IntegrationRule *irs[] = NULL) const;
 
    /** Compute the Lp error in each element of the mesh and store the results in
-       the GridFunction @ error. The result should be an L2 GridFunction of
-       order zero using map type VALUE. */
+       the Vector @ error. The result should be of length number of elements,
+       for example an L2 GridFunction of order zero using map type VALUE. */
    virtual void ComputeElementLpErrors(const double p, VectorCoefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        Coefficient *weight = NULL,
                                        VectorCoefficient *v_weight = NULL,
                                        const IntegrationRule *irs[] = NULL
                                       ) const;
 
    virtual void ComputeElementL1Errors(VectorCoefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        const IntegrationRule *irs[] = NULL
                                       ) const
    { ComputeElementLpErrors(1.0, exsol, error, NULL, NULL, irs); }
 
    virtual void ComputeElementL2Errors(VectorCoefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        const IntegrationRule *irs[] = NULL
                                       ) const
    { ComputeElementLpErrors(2.0, exsol, error, NULL, NULL, irs); }
 
    virtual void ComputeElementMaxErrors(VectorCoefficient &exsol,
-                                        GridFunction &error,
+                                        Vector &error,
                                         const IntegrationRule *irs[] = NULL
                                        ) const
    { ComputeElementLpErrors(infinity(), exsol, error, NULL, NULL, irs); }
@@ -633,6 +735,16 @@ public:
     */
    inline void GetElementValues(int idx, Vector &values) const;
 
+   /// Return the quadrature function values at an integration point.
+   /** The result is stored in the Vector @a values as a reference to the
+       global values. */
+   inline void GetElementValues(int idx, const int ip_num, Vector &values);
+
+   /// Return the quadrature function values at an integration point.
+   /** The result is stored in the Vector @a values as a copy to the
+       global values. */
+   inline void GetElementValues(int idx, const int ip_num, Vector &values) const;
+
    /// Return all values associated with mesh element @a idx in a DenseMatrix.
    /** The result is stored in the DenseMatrix @a values as a reference to the
        global values.
@@ -737,6 +849,25 @@ inline void QuadratureFunction::GetElementValues(int idx, Vector &values) const
    }
 }
 
+inline void QuadratureFunction::GetElementValues(int idx, const int ip_num,
+                                                 Vector &values)
+{
+   const int s_offset = qspace->element_offsets[idx] * vdim + ip_num * vdim;
+   values.NewDataAndSize(data + s_offset, vdim);
+}
+
+inline void QuadratureFunction::GetElementValues(int idx, const int ip_num,
+                                                 Vector &values) const
+{
+   const int s_offset = qspace->element_offsets[idx] * vdim + ip_num * vdim;
+   values.SetSize(vdim);
+   const double *q = data + s_offset;
+   for (int i = 0; i < values.Size(); i++)
+   {
+      values(i) = *(q++);
+   }
+}
+
 inline void QuadratureFunction::GetElementValues(int idx, DenseMatrix &values)
 {
    const int s_offset = qspace->element_offsets[idx];

fem/gslib.cpp

@@ -29,12 +29,14 @@ namespace mfem
 {
 
 FindPointsGSLIB::FindPointsGSLIB()
-   : mesh(NULL), ir_simplex(NULL), gsl_mesh(), fdata2D(NULL), fdata3D(NULL),
-     dim(-1)
+   : mesh(NULL), ir_simplex(NULL), fdata2D(NULL), fdata3D(NULL),
+     dim(-1), gsl_mesh(), gsl_ref(), gsl_dist(), setupflag(false)
 {
    gsl_comm = new comm;
 #ifdef MFEM_USE_MPI
-   MPI_Init(NULL, NULL);
+   int initialized;
+   MPI_Initialized(&initialized);
+   if (!initialized) { MPI_Init(NULL, NULL); }
    MPI_Comm comm = MPI_COMM_WORLD;;
    comm_init(gsl_comm, comm);
 #else
@@ -50,28 +52,29 @@ FindPointsGSLIB::~FindPointsGSLIB()
 
 #ifdef MFEM_USE_MPI
 FindPointsGSLIB::FindPointsGSLIB(MPI_Comm _comm)
-   : mesh(NULL), ir_simplex(NULL), gsl_mesh(), fdata2D(NULL), fdata3D(NULL),
-     dim(-1)
+   : mesh(NULL), ir_simplex(NULL), fdata2D(NULL), fdata3D(NULL),
+     dim(-1), gsl_mesh(), gsl_ref(), gsl_dist(), setupflag(false)
 {
    gsl_comm = new comm;
    comm_init(gsl_comm, _comm);
 }
 #endif
 
-void FindPointsGSLIB::Setup(Mesh &m, double bb_t, double newt_tol, int npt_max)
+void FindPointsGSLIB::Setup(Mesh &m, const double bb_t, const double newt_tol,
+                            const int npt_max)
 {
    MFEM_VERIFY(m.GetNodes() != NULL, "Mesh nodes are required.");
    MFEM_VERIFY(m.GetNumGeometries(m.Dimension()) == 1,
                "Mixed meshes are not currently supported in FindPointsGSLIB.");
 
+   // call FreeData if FindPointsGSLIB::Setup has been called already
+   if (setupflag) { FreeData(); }
+
    mesh = &m;
    dim  = mesh->Dimension();
    const FiniteElement *fe = mesh->GetNodalFESpace()->GetFE(0);
    unsigned dof1D = fe->GetOrder() + 1;
-   int NE      = mesh->GetNE(),
-       dof_cnt = fe->GetDof(),
-       pts_cnt = NE * dof_cnt,
-       gt      = fe->GetGeomType();
+   const int gt   = fe->GetGeomType();
 
    if (gt == Geometry::TRIANGLE || gt == Geometry::TETRAHEDRON ||
        gt == Geometry::PRISM)
@@ -87,8 +90,8 @@ void FindPointsGSLIB::Setup(Mesh &m, double bb_t, double newt_tol, int npt_max)
       MFEM_ABORT("Element type not currently supported in FindPointsGSLIB.");
    }
 
-   pts_cnt = gsl_mesh.Size()/dim;
-   int NEtot = pts_cnt/(int)pow(dof1D, dim);
+   const int pts_cnt = gsl_mesh.Size()/dim,
+             NEtot = pts_cnt/(int)pow(dof1D, dim);
 
    if (dim == 2)
    {
@@ -107,6 +110,7 @@ void FindPointsGSLIB::Setup(Mesh &m, double bb_t, double newt_tol, int npt_max)
       fdata3D = findpts_setup_3(gsl_comm, elx, nr, NEtot, mr, bb_t,
                                 pts_cnt, pts_cnt, npt_max, newt_tol);
    }
+   setupflag = true;
 }
 
 void FindPointsGSLIB::FindPoints(const Vector &point_pos,
@@ -115,6 +119,7 @@ void FindPointsGSLIB::FindPoints(const Vector &point_pos,
                                  Array<unsigned int> &elem_ids,
                                  Vector &ref_pos, Vector &dist)
 {
+   MFEM_VERIFY(setupflag, "Use FindPointsGSLIB::Setup before finding points.");
    const int points_cnt = point_pos.Size() / dim;
    if (dim == 2)
    {
@@ -150,36 +155,92 @@ void FindPointsGSLIB::FindPoints(const Vector &point_pos,
    }
 }
 
+void FindPointsGSLIB::FindPoints(const Vector &point_pos)
+{
+   const int points_cnt = point_pos.Size() / dim;
+   gsl_code.SetSize(points_cnt);
+   gsl_proc.SetSize(points_cnt);
+   gsl_elem.SetSize(points_cnt);
+   gsl_ref.SetSize(points_cnt * dim);
+   gsl_dist.SetSize(points_cnt);
+
+   FindPoints(point_pos, gsl_code, gsl_proc, gsl_elem, gsl_ref, gsl_dist);
+}
+
+void FindPointsGSLIB::FindPoints(Mesh &m, const Vector &point_pos,
+                                 const double bb_t, const double newt_tol,
+                                 const int npt_max)
+{
+   if (!setupflag || (mesh != &m) )
+   {
+      Setup(m, bb_t, newt_tol, npt_max);
+   }
+   FindPoints(point_pos);
+}
+
 void FindPointsGSLIB::Interpolate(Array<unsigned int> &codes,
                                   Array<unsigned int> &proc_ids,
                                   Array<unsigned int> &elem_ids,
                                   Vector &ref_pos, const GridFunction &field_in,
                                   Vector &field_out)
 {
-   Vector node_vals;
-   GetNodeValues(field_in, node_vals);
 
-   const int points_cnt = ref_pos.Size() / dim;
-   if (dim==2)
+   FiniteElementSpace ind_fes(mesh, field_in.FESpace()->FEColl());
+   GridFunction field_in_scalar(&ind_fes);
+   Vector node_vals;
+
+   const int ncomp      = field_in.FESpace()->GetVDim(),
+             points_fld = field_in.Size() / ncomp,
+             points_cnt = codes.Size();
+
+   for (int i = 0; i < ncomp; i++)
    {
-      findpts_eval_2(field_out.GetData(), sizeof(double),
-                     codes.GetData(), sizeof(unsigned int),
-                     proc_ids.GetData(), sizeof(unsigned int),
-                     elem_ids.GetData(), sizeof(unsigned int),
-                     ref_pos.GetData(), sizeof(double) * dim,
-                     points_cnt, node_vals.GetData(), fdata2D);
-   }
-   else
-   {
-      findpts_eval_3(field_out.GetData(), sizeof(double),
-                     codes.GetData(), sizeof(unsigned int),
-                     proc_ids.GetData(), sizeof(unsigned int),
-                     elem_ids.GetData(), sizeof(unsigned int),
-                     ref_pos.GetData(), sizeof(double) * dim,
-                     points_cnt, node_vals.GetData(), fdata3D);
+      const int dataptrin  = i*points_fld,
+                dataptrout = i*points_cnt;
+      field_in_scalar.NewDataAndSize(field_in.GetData()+dataptrin, points_fld);
+      GetNodeValues(field_in_scalar, node_vals);
+
+      if (dim==2)
+      {
+         findpts_eval_2(field_out.GetData()+dataptrout, sizeof(double),
+                        codes.GetData(),       sizeof(unsigned int),
+                        proc_ids.GetData(),    sizeof(unsigned int),
+                        elem_ids.GetData(),    sizeof(unsigned int),
+                        ref_pos.GetData(),     sizeof(double) * dim,
+                        points_cnt, node_vals.GetData(), fdata2D);
+      }
+      else
+      {
+         findpts_eval_3(field_out.GetData()+dataptrout, sizeof(double),
+                        codes.GetData(),       sizeof(unsigned int),
+                        proc_ids.GetData(),    sizeof(unsigned int),
+                        elem_ids.GetData(),    sizeof(unsigned int),
+                        ref_pos.GetData(),     sizeof(double) * dim,
+                        points_cnt, node_vals.GetData(), fdata3D);
+      }
    }
 }
 
+void FindPointsGSLIB::Interpolate(const GridFunction &field_in,
+                                  Vector &field_out)
+{
+   Interpolate(gsl_code, gsl_proc, gsl_elem, gsl_ref, field_in, field_out);
+}
+
+void FindPointsGSLIB::Interpolate(const Vector &point_pos,
+                                  const GridFunction &field_in, Vector &field_out)
+{
+   FindPoints(point_pos);
+   Interpolate(gsl_code, gsl_proc, gsl_elem, gsl_ref, field_in, field_out);
+}
+
+void FindPointsGSLIB::Interpolate(Mesh &m, const Vector &point_pos,
+                                  const GridFunction &field_in, Vector &field_out)
+{
+   FindPoints(m, point_pos);
+   Interpolate(gsl_code, gsl_proc, gsl_elem, gsl_ref, field_in, field_out);
+}
+
 void FindPointsGSLIB::FreeData()
 {
    if (dim == 2)
@@ -190,7 +251,13 @@ void FindPointsGSLIB::FreeData()
    {
       findpts_free_3(fdata3D);
    }
+   setupflag = false;
+   gsl_code.DeleteAll();
+   gsl_proc.DeleteAll();
+   gsl_elem.DeleteAll();
    gsl_mesh.Destroy();
+   gsl_ref.Destroy();
+   gsl_dist.Destroy();
 }
 
 void FindPointsGSLIB::GetNodeValues(const GridFunction &gf_in,
@@ -292,7 +359,7 @@ void FindPointsGSLIB::GetSimplexNodalCoordinates()
    const GridFunction *nodes = mesh->GetNodes();
    Mesh *meshsplit           = NULL;
    const int NE              = mesh->GetNE();
-   int NEsplit;
+   int NEsplit = -1;
 
    // Split the reference element into a reference submesh of quads or hexes.
    if (gt == Geometry::TRIANGLE)
@@ -386,6 +453,7 @@ void FindPointsGSLIB::GetSimplexNodalCoordinates()
       }
       meshsplit->FinalizeHexMesh(1, 1, true);
    }
+   else { MFEM_ABORT("Unsupported geometry type."); }
 
    // Curve the reference submesh.
    H1_FECollection fec(fe->GetOrder(), dim);

fem/gslib.hpp

@@ -29,10 +29,12 @@ class FindPointsGSLIB
 protected:
    Mesh *mesh;
    IntegrationRule *ir_simplex;
-   Vector gsl_mesh;
    struct findpts_data_2 *fdata2D;
    struct findpts_data_3 *fdata3D;
    int dim;
+   Array<unsigned int> gsl_code, gsl_proc, gsl_elem;
+   Vector gsl_mesh, gsl_ref, gsl_dist;
+   bool setupflag;
 
    struct comm *gsl_comm;
 
@@ -59,7 +61,8 @@ public:
        @param[in] newt_tol  Newton tolerance for the gslib search methods.
        @param[in] npt_max   Number of points for simultaneous iteration. This
                             alters performance and memory footprint. */
-   void Setup(Mesh &m, double bb_t, double newt_tol, int npt_max);
+   void Setup(Mesh &m, const double bb_t = 0.1, const double newt_tol = 1.0e-12,
+              const int npt_max = 256);
 
    /** Searches positions given in physical space by @a point_pos. All output
        Arrays and Vectors are expected to have the correct size.
@@ -73,11 +76,15 @@ public:
        @param[out] ref_pos    Reference coordinates of the found point. Ordered
                               by vdim (XYZ,XYZ,XYZ...).
                               Note: the gslib reference frame is [-1,1].
-       @param[out] dist       Distance between the seeked and the found point
+       @param[out] dist       Distance between the sought and the found point
                               in physical space. */
    void FindPoints(const Vector &point_pos, Array<unsigned int> &codes,
                    Array<unsigned int> &proc_ids, Array<unsigned int> &elem_ids,
                    Vector &ref_pos, Vector &dist);
+   void FindPoints(const Vector &point_pos);
+   /// Setup FindPoints and search positions
+   void FindPoints(Mesh &m, const Vector &point_pos, const double bb_t = 0.1,
+                   const double newt_tol = 1.0e-12,  const int npt_max = 256);
 
    /** Interpolation of field values at prescribed reference space positions.
 
@@ -96,11 +103,31 @@ public:
    void Interpolate(Array<unsigned int> &codes, Array<unsigned int> &proc_ids,
                     Array<unsigned int> &elem_ids, Vector &ref_pos,
                     const GridFunction &field_in, Vector &field_out);
+   void Interpolate(const GridFunction &field_in, Vector &field_out);
+   /** Search positions and interpolate */
+   void Interpolate(const Vector &point_pos, const GridFunction &field_in,
+                    Vector &field_out);
+   /** Setup FindPoints, search positions and interpolate */
+   void Interpolate(Mesh &m, const Vector &point_pos,
+                    const GridFunction &field_in, Vector &field_out);
 
    /** Cleans up memory allocated internally by gslib.
        Note that in parallel, this must be called before MPI_Finalize(), as
        it calls MPI_Comm_free() for internal gslib communicators. */
    void FreeData();
+
+   /// Return code for each point searched by FindPoints: inside element (0), on
+   /// element boundary (1), or not found (2).
+   const Array<unsigned int> &GetCode() const { return gsl_code; }
+   /// Return element number for each point found by FindPoints.
+   const Array<unsigned int> &GetElem() const { return gsl_elem; }
+   /// Return MPI rank on which each point was found by FindPoints.
+   const Array<unsigned int> &GetProc() const { return gsl_proc; }
+   /// Return reference coordinates for each point found by FindPoints.
+   const Vector &GetReferencePosition() const { return gsl_ref;  }
+   /// Return distance Distance between the sought and the found point
+   /// in physical space, for each point found by FindPoints.
+   const Vector &GetDist()              const { return gsl_dist; }
 };
 
 } // namespace mfem

fem/libceed/ceed.hpp

@@ -40,7 +40,7 @@ struct CeedConstCoeff
 
 struct CeedGridCoeff
 {
-   GridFunction* coeff;
+   const GridFunction* coeff;
    CeedBasis basis;
    CeedElemRestriction restr;
    CeedVector coeffVector;

fem/lininteg.cpp

@@ -307,7 +307,7 @@ void VectorBoundaryLFIntegrator::AssembleRHSElementVect(
    if (ir == NULL)
    {
       int intorder = 2*el.GetOrder();
-      ir = &IntRules.Get(Tr.FaceGeom, intorder);
+      ir = &IntRules.Get(Tr.GetGeometryType(), intorder);
    }
 
    for (int i = 0; i < ir->GetNPoints(); i++)
@@ -316,9 +316,11 @@ void VectorBoundaryLFIntegrator::AssembleRHSElementVect(
       IntegrationPoint eip;
       Tr.Loc1.Transform(ip, eip);
 
-      Tr.Face->SetIntPoint(&ip);
-      Q.Eval(vec, *Tr.Face, ip);
-      vec *= Tr.Face->Weight() * ip.weight;
+      Tr.SetIntPoint(&ip);
+
+      // Use Tr transformation in case Q depends on boundary attribute
+      Q.Eval(vec, Tr, ip);
+      vec *= Tr.Weight() * ip.weight;
       el.CalcShape(eip, shape);
       for (int k = 0; k < vdim; k++)
       {
@@ -510,7 +512,7 @@ void BoundaryFlowIntegrator::AssembleRHSElementVect(
       {
          order++;
       }
-      ir = &IntRules.Get(Tr.FaceGeom, order);
+      ir = &IntRules.Get(Tr.GetGeometryType(), order);
    }
 
    shape.SetSize(ndof);
@@ -524,8 +526,10 @@ void BoundaryFlowIntegrator::AssembleRHSElementVect(
       Tr.Loc1.Transform(ip, eip);
       el.CalcShape(eip, shape);
 
-      Tr.Face->SetIntPoint(&ip);
+      Tr.SetIntPoint(&ip);
 
+      // Use Tr.Elem1 transformation for u so that it matches the coefficient
+      // used with the ConvectionIntegrator and/or the DGTraceIntegrator.
       u->Eval(vu, *Tr.Elem1, eip);
 
       if (dim == 1)
@@ -534,12 +538,12 @@ void BoundaryFlowIntegrator::AssembleRHSElementVect(
       }
       else
       {
-         CalcOrtho(Tr.Face->Jacobian(), nor);
+         CalcOrtho(Tr.Jacobian(), nor);
       }
 
       un = vu * nor;
       w = 0.5*alpha*un - beta*fabs(un);
-      w *= ip.weight*f->Eval(*Tr.Elem1, eip);
+      w *= ip.weight*f->Eval(Tr, ip);
       elvect.Add(w, shape);
    }
 }
@@ -582,7 +586,7 @@ void DGDirichletLFIntegrator::AssembleRHSElementVect(
    {
       // a simple choice for the integration order; is this OK?
       int order = 2*el.GetOrder();
-      ir = &IntRules.Get(Tr.FaceGeom, order);
+      ir = &IntRules.Get(Tr.GetGeometryType(), order);
    }
 
    for (int p = 0; p < ir->GetNPoints(); p++)
@@ -591,33 +595,33 @@ void DGDirichletLFIntegrator::AssembleRHSElementVect(
       IntegrationPoint eip;
 
       Tr.Loc1.Transform(ip, eip);
-      Tr.Face->SetIntPoint(&ip);
+      Tr.SetIntPoint(&ip);
       if (dim == 1)
       {
          nor(0) = 2*eip.x - 1.0;
       }
       else
       {
-         CalcOrtho(Tr.Face->Jacobian(), nor);
+         CalcOrtho(Tr.Jacobian(), nor);
       }
 
       el.CalcShape(eip, shape);
       el.CalcDShape(eip, dshape);
-      Tr.Elem1->SetIntPoint(&eip);
+
       // compute uD through the face transformation
-      w = ip.weight * uD->Eval(*Tr.Face, ip) / Tr.Elem1->Weight();
+      w = ip.weight * uD->Eval(Tr, ip) / Tr.Elem1->Weight();
       if (!MQ)
       {
          if (Q)
          {
-            w *= Q->Eval(*Tr.Elem1, eip);
+            w *= Q->Eval(Tr, ip);
          }
          ni.Set(w, nor);
       }
       else
       {
          nh.Set(w, nor);
-         MQ->Eval(mq, *Tr.Elem1, eip);
+         MQ->Eval(mq, Tr, ip);
          mq.MultTranspose(nh, ni);
       }
       CalcAdjugate(Tr.Elem1->Jacobian(), adjJ);
@@ -676,7 +680,7 @@ void DGElasticityDirichletLFIntegrator::AssembleRHSElementVect(
    if (ir == NULL)
    {
       const int order = 2*el.GetOrder(); // <-----
-      ir = &IntRules.Get(Tr.FaceGeom, order);
+      ir = &IntRules.Get(Tr.GetGeometryType(), order);
    }
 
    for (int pi = 0; pi < ir->GetNPoints(); ++pi)
@@ -684,11 +688,10 @@ void DGElasticityDirichletLFIntegrator::AssembleRHSElementVect(
       const IntegrationPoint &ip = ir->IntPoint(pi);
       IntegrationPoint eip;
       Tr.Loc1.Transform(ip, eip);
-      Tr.Face->SetIntPoint(&ip);
-      Tr.Elem1->SetIntPoint(&eip);
+      Tr.SetIntPoint(&ip);
 
       // Evaluate the Dirichlet b.c. using the face transformation.
-      uD.Eval(u_dir, *Tr.Face, ip);
+      uD.Eval(u_dir, Tr, ip);
 
       el.CalcShape(eip, shape);
       el.CalcDShape(eip, dshape);
@@ -702,7 +705,7 @@ void DGElasticityDirichletLFIntegrator::AssembleRHSElementVect(
       }
       else
       {
-         CalcOrtho(Tr.Face->Jacobian(), nor);
+         CalcOrtho(Tr.Jacobian(), nor);
       }
 
       double wL, wM, jcoef;
@@ -768,4 +771,58 @@ void DGElasticityDirichletLFIntegrator::AssembleRHSElementVect(
    }
 }
 
+void VectorQuadratureLFIntegrator::AssembleRHSElementVect(
+   const FiniteElement &fe, ElementTransformation &Tr, Vector &elvect)
+{
+   const IntegrationRule *ir =
+      &vqfc.GetQuadFunction().GetSpace()->GetElementIntRule(Tr.ElementNo);
+
+   const int nqp = ir->GetNPoints();
+   const int vdim = vqfc.GetVDim();
+   const int ndofs = fe.GetDof();
+   Vector shape(ndofs);
+   Vector temp(vdim);
+   elvect.SetSize(vdim * ndofs);
+   elvect = 0.0;
+   for (int q = 0; q < nqp; q++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(q);
+      Tr.SetIntPoint(&ip);
+      const double w = Tr.Weight() * ip.weight;
+      vqfc.Eval(temp, Tr, ip);
+      fe.CalcShape(ip, shape);
+      for (int ind = 0; ind < vdim; ind++)
+      {
+         for (int nd = 0; nd < ndofs; nd++)
+         {
+            elvect(nd + ind * ndofs) += w * shape(nd) * temp(ind);
+         }
+      }
+   }
+}
+
+void QuadratureLFIntegrator::AssembleRHSElementVect(const FiniteElement &fe,
+                                                    ElementTransformation &Tr,
+                                                    Vector &elvect)
+{
+   const IntegrationRule *ir =
+      &qfc.GetQuadFunction().GetSpace()->GetElementIntRule(Tr.ElementNo);
+
+   const int nqp = ir->GetNPoints();
+   const int ndofs = fe.GetDof();
+   Vector shape(ndofs);
+   elvect.SetSize(ndofs);
+   elvect = 0.0;
+   for (int q = 0; q < nqp; q++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(q);
+      Tr.SetIntPoint (&ip);
+      const double w = Tr.Weight() * ip.weight;
+      double temp = qfc.Eval(Tr, ip);
+      fe.CalcShape(ip, shape);
+      shape *= (w * temp);
+      elvect += shape;
+   }
+}
+
 }

fem/lininteg.hpp

@@ -36,7 +36,7 @@ public:
                                        FaceElementTransformations &Tr,
                                        Vector &elvect);
 
-   void SetIntRule(const IntegrationRule *ir) { IntRule = ir; }
+   virtual void SetIntRule(const IntegrationRule *ir) { IntRule = ir; }
    const IntegrationRule* GetIntRule() { return IntRule; }
 
    virtual ~LinearFormIntegrator() { }
@@ -426,6 +426,69 @@ public:
                                        Vector &elvect);
 };
 
+/** Class for domain integration of L(v) := (f, v), where
+    f=(f1,...,fn) and v=(v1,...,vn). that makes use of
+    VectorQuadratureFunctionCoefficient*/
+class VectorQuadratureLFIntegrator : public LinearFormIntegrator
+{
+private:
+   VectorQuadratureFunctionCoefficient &vqfc;
+
+public:
+   VectorQuadratureLFIntegrator(VectorQuadratureFunctionCoefficient &vqfc,
+                                const IntegrationRule *ir)
+      : LinearFormIntegrator(ir), vqfc(vqfc)
+   {
+      if (ir)
+      {
+         MFEM_WARNING("Integration rule not used in this class. "
+                      "The QuadratureFunction integration rules are used instead");
+      }
+   }
+
+   using LinearFormIntegrator::AssembleRHSElementVect;
+   virtual void AssembleRHSElementVect(const FiniteElement &fe,
+                                       ElementTransformation &Tr,
+                                       Vector &elvect);
+
+   virtual void SetIntRule(const IntegrationRule *ir)
+   {
+      MFEM_WARNING("Integration rule not used in this class. "
+                   "The QuadratureFunction integration rules are used instead");
+   }
+};
+
+/** Class for domain integration L(v) := (f, v) that makes use
+    of QuadratureFunctionCoefficient. */
+class QuadratureLFIntegrator : public LinearFormIntegrator
+{
+private:
+   QuadratureFunctionCoefficient &qfc;
+
+public:
+   QuadratureLFIntegrator(QuadratureFunctionCoefficient &qfc,
+                          const IntegrationRule *ir)
+      : LinearFormIntegrator(ir), qfc(qfc)
+   {
+      if (ir)
+      {
+         MFEM_WARNING("Integration rule not used in this class. "
+                      "The QuadratureFunction integration rules are used instead");
+      }
+   }
+
+   using LinearFormIntegrator::AssembleRHSElementVect;
+   virtual void AssembleRHSElementVect(const FiniteElement &fe,
+                                       ElementTransformation &Tr,
+                                       Vector &elvect);
+
+   virtual void SetIntRule(const IntegrationRule *ir)
+   {
+      MFEM_WARNING("Integration rule not used in this class. "
+                   "The QuadratureFunction integration rules are used instead");
+   }
+};
+
 }
 
 #endif

fem/pgridfunc.cpp

@@ -232,7 +232,8 @@ void ParGridFunction::ExchangeFaceNbrData()
    auto d_send_data = send_data.Write();
    MFEM_FORALL(i, send_data.Size(),
    {
-      d_send_data[i] = d_data[d_send_ldof[i]];
+      const int ldof = d_send_ldof[i];
+      d_send_data[i] = d_data[ldof >= 0 ? ldof : -1-ldof];
    });
 
    bool mpi_gpu_aware = Device::GetGPUAwareMPI();

fem/restriction.cpp

@@ -168,6 +168,27 @@ void ElementRestriction::Mult(const Vector& x, Vector& y) const
    });
 }
 
+void ElementRestriction::MultUnsigned(const Vector& x, Vector& y) const
+{
+   // Assumes all elements have the same number of dofs
+   const int nd = dof;
+   const int vd = vdim;
+   const bool t = byvdim;
+   auto d_x = Reshape(x.Read(), t?vd:ndofs, t?ndofs:vd);
+   auto d_y = Reshape(y.Write(), nd, vd, ne);
+   auto d_gatherMap = gatherMap.Read();
+
+   MFEM_FORALL(i, dof*ne,
+   {
+      const int gid = d_gatherMap[i];
+      const int j = gid >= 0 ? gid : -1-gid;
+      for (int c = 0; c < vd; ++c)
+      {
+         d_y(i % nd, c, i / nd) = d_x(t?c:j, t?j:c);
+      }
+   });
+}
+
 void ElementRestriction::MultTranspose(const Vector& x, Vector& y) const
 {
    // Assumes all elements have the same number of dofs
@@ -966,27 +987,51 @@ void L2FaceRestriction::MultTranspose(const Vector& x, Vector& y) const
    const int dofs = nfdofs;
    auto d_offsets = offsets.Read();
    auto d_indices = gather_indices.Read();
-   auto d_x = Reshape(x.Read(), nd, vd, 2, nf);
-   auto d_y = Reshape(y.Write(), t?vd:ndofs, t?ndofs:vd);
-   MFEM_FORALL(i, ndofs,
+
+   if (m == L2FaceValues::DoubleValued)
    {
-      const int offset = d_offsets[i];
-      const int nextOffset = d_offsets[i + 1];
-      for (int c = 0; c < vd; ++c)
+      auto d_x = Reshape(x.Read(), nd, vd, 2, nf);
+      auto d_y = Reshape(y.Write(), t?vd:ndofs, t?ndofs:vd);
+      MFEM_FORALL(i, ndofs,
       {
-         double dofValue = 0;
-         for (int j = offset; j < nextOffset; ++j)
+         const int offset = d_offsets[i];
+         const int nextOffset = d_offsets[i + 1];
+         for (int c = 0; c < vd; ++c)
          {
-            int idx_j = d_indices[j];
-            bool isE1 = idx_j < dofs;
-            idx_j = isE1 ? idx_j : idx_j - dofs;
-            dofValue +=  isE1 ?
-            d_x(idx_j % nd, c, 0, idx_j / nd)
-            :d_x(idx_j % nd, c, 1, idx_j / nd);
+            double dofValue = 0;
+            for (int j = offset; j < nextOffset; ++j)
+            {
+               int idx_j = d_indices[j];
+               bool isE1 = idx_j < dofs;
+               idx_j = isE1 ? idx_j : idx_j - dofs;
+               dofValue +=  isE1 ?
+               d_x(idx_j % nd, c, 0, idx_j / nd)
+               :d_x(idx_j % nd, c, 1, idx_j / nd);
+            }
+            d_y(t?c:i,t?i:c) += dofValue;
          }
-         d_y(t?c:i,t?i:c) += dofValue;
-      }
-   });
+      });
+   }
+   else
+   {
+      auto d_x = Reshape(x.Read(), nd, vd, nf);
+      auto d_y = Reshape(y.Write(), t?vd:ndofs, t?ndofs:vd);
+      MFEM_FORALL(i, ndofs,
+      {
+         const int offset = d_offsets[i];
+         const int nextOffset = d_offsets[i + 1];
+         for (int c = 0; c < vd; ++c)
+         {
+            double dofValue = 0;
+            for (int j = offset; j < nextOffset; ++j)
+            {
+               int idx_j = d_indices[j];
+               dofValue +=  d_x(idx_j % nd, c, idx_j / nd);
+            }
+            d_y(t?c:i,t?i:c) += dofValue;
+         }
+      });
+   }
 }
 
 int ToLexOrdering(const int dim, const int face_id, const int size1d,

fem/restriction.hpp

@@ -47,6 +47,8 @@ public:
    void Mult(const Vector &x, Vector &y) const;
    void MultTranspose(const Vector &x, Vector &y) const;
 
+   /// Compute Mult without applying signs based on DOF orientations.
+   void MultUnsigned(const Vector &x, Vector &y) const;
    /// Compute MultTranspose without applying signs based on DOF orientations.
    void MultTransposeUnsigned(const Vector &x, Vector &y) const;
 

fem/tbilinearform.hpp

@@ -13,6 +13,7 @@
 #define MFEM_TEMPLATE_BILINEAR_FORM
 
 #include "../config/tconfig.hpp"
+#include "../linalg/simd.hpp"
 #include "../linalg/ttensor.hpp"
 #include "bilinearform.hpp"
 #include "tevaluator.hpp"
@@ -30,9 +31,13 @@ namespace mfem
 template <typename meshType, typename solFESpace,
           typename IR, typename IntegratorType,
           typename solVecLayout_t = ScalarLayout,
-          typename complex_t = double, typename real_t = double>
+          typename complex_t = double, typename real_t = double,
+          typename impl_traits_t = AutoSIMDTraits<complex_t,real_t> >
 class TBilinearForm : public Operator
 {
+public:
+   typedef impl_traits_t impl_traits_type;
+
 protected:
    typedef complex_t complex_type;
    typedef real_t    real_type;
@@ -48,26 +53,36 @@ protected:
    static const int dofs = solFE_type::dofs;
    static const int vdim = solVecLayout_t::vec_dim;
    static const int qpts = IR::qpts;
+   static const int AB   = impl_traits_t::align_bytes;
+   static const int SS   = impl_traits_t::simd_size;
+   static const int BE   = impl_traits_t::batch_size;
+   static const int TE   = SS*BE;
+
+   typedef typename impl_traits_t::vcomplex_t vcomplex_t;
+   typedef typename impl_traits_t::vreal_t    vreal_t;
 
    typedef IntegratorType integ_t;
    typedef typename integ_t::coefficient_type coeff_t;
-   typedef typename integ_t::template kernel<sdim,dim,complex_t>::type kernel_t;
+   typedef typename integ_t::template kernel<sdim,dim,vcomplex_t>::type kernel_t;
    typedef typename kernel_t::template p_asm_data<qpts>::type p_assembled_t;
    typedef typename kernel_t::template f_asm_data<qpts>::type f_assembled_t;
+   typedef typename kernel_t::template
+   CoefficientEval<IR,coeff_t,impl_traits_t>::Type coeff_eval_t;
+
 
    typedef TElementTransformation<meshType,IR,real_t> Trans_t;
-   template <int NE> struct T_result
+   struct T_result
    {
       static const int EvalOps =
          Trans_t::template Get<coeff_t,kernel_t>::EvalOps;
-      typedef typename Trans_t::template Result<EvalOps,NE> Type;
+      typedef typename Trans_t::template Result<EvalOps,impl_traits_t> Type;
    };
 
    typedef FieldEvaluator<solFESpace,solVecLayout_t,IR,
            complex_t,real_t> solFieldEval;
-   template <int BE> struct S_spec
+   struct S_spec
    {
-      typedef typename solFieldEval::template Spec<kernel_t,BE> Spec;
+      typedef typename solFieldEval::template Spec<kernel_t,impl_traits_t> Spec;
       typedef typename Spec::DataType DataType;
       typedef typename Spec::ElementMatrix ElementMatrix;
    };
@@ -86,7 +101,7 @@ protected:
 
    coeff_t coeff;
 
-   p_assembled_t *assembled_data;
+   Memory<p_assembled_t> assembled_data;
 
    const FiniteElementSpace &in_fes;
 
@@ -101,13 +116,17 @@ public:
         solVecLayout(sol_fes),
         int_rule(),
         coeff(integ.coeff),
-        assembled_data(NULL),
+        assembled_data(),
         in_fes(sol_fes)
-   { }
+   {
+      assembled_data.Reset(AB == 64 ? MemoryType::HOST_64 :
+                           AB == 32 ? MemoryType::HOST_32 :
+                           MemoryType::HOST);
+   }
 
    virtual ~TBilinearForm()
    {
-      delete [] assembled_data;
+      assembled_data.Delete();
    }
 
    /// Get the input finite element space prolongation matrix
@@ -119,10 +138,9 @@ public:
 
    virtual void Mult(const Vector &x, Vector &y) const
    {
-      if (assembled_data)
+      if (!assembled_data.Empty())
       {
-         const int num_elem = 1;
-         MultAssembled<num_elem>(x, y);
+         MultAssembled(x, y);
       }
       else
       {
@@ -135,10 +153,6 @@ public:
    {
       y = 0.0;
 
-      const int BE = 1; // batch-size of elements
-      typedef typename kernel_t::template
-      CoefficientEval<IR,coeff_t,BE>::Type coeff_eval_t;
-
       // For better performance, create stack copies of solFES, and solEval
       // inside 'solFEval'. The element-transformation 'T' also copies the
       // meshFES, meshEval, etc internally.
@@ -149,26 +163,29 @@ public:
       coeff_eval_t wQ(int_rule, coeff);
 
       const int NE = mesh.GetNE();
-      for (int el = 0; el < NE; el++)
+      for (int el = 0; el < NE; el += TE)
       {
 #if 0
-         typename S_spec<BE>::DataType R;
+         typename S_spec::DataType R;
          solFEval.Eval(el, R);
 
-         typename T_result<BE>::Type F;
+         typename T_result::Type F;
          T.Eval(el, F);
 #else
-         typename T_result<BE>::Type F;
+         typename T_result::Type F;
          T.Eval(el, F);
 
-         typename S_spec<BE>::DataType R;
+         typename S_spec::DataType R;
          solFEval.Eval(el, R);
 #endif
 
          typename coeff_eval_t::result_t res;
          wQ.Eval(F, res);
 
-         kernel_t::Action(0, F, wQ, res, R);
+         for (int k = 0; k < BE; k++)
+         {
+            kernel_t::Action(k, F, wQ, res, R);
+         }
 
          solFEval.template Assemble<true>(R);
       }
@@ -177,21 +194,18 @@ public:
    // Partial assembly of quadrature point data
    void Assemble()
    {
-      const int BE = 1; // batch-size of elements
-      typedef typename kernel_t::template
-      CoefficientEval<IR,coeff_t,BE>::Type coeff_eval_t;
-
       Trans_t T(mesh, meshEval);
       coeff_eval_t wQ(int_rule, coeff);
 
       const int NE = mesh.GetNE();
-      if (!assembled_data)
+      if (assembled_data.Empty())
       {
-         assembled_data = new p_assembled_t[NE];
+         const int size = ((NE+TE-1)/TE)*BE;
+         assembled_data.New(size, assembled_data.GetMemoryType());
       }
-      for (int el = 0; el < NE; el++) // BE == 1
+      for (int el = 0; el < NE; el += TE)
       {
-         typename T_result<BE>::Type F;
+         typename T_result::Type F;
          T.Eval(el, F);
 
          typename coeff_eval_t::result_t res;
@@ -199,28 +213,26 @@ public:
 
          for (int k = 0; k < BE; k++)
          {
-            kernel_t::Assemble(k, F, wQ, res, assembled_data[el+k]);
+            kernel_t::Assemble(k, F, wQ, res, assembled_data[el/SS+k]);
          }
       }
    }
 
-   template <int num_elem>
    inline MFEM_ALWAYS_INLINE
    void ElementAddMultAssembled(int el, solFieldEval &solFEval) const
    {
-      typename S_spec<num_elem>::DataType R;
+      typename S_spec::DataType R;
       solFEval.Eval(el, R);
 
-      for (int k = 0; k < num_elem; k++)
+      for (int k = 0; k < BE; k++)
       {
-         kernel_t::MultAssembled(k, assembled_data[el+k], R);
+         kernel_t::MultAssembled(k, assembled_data[el/SS+k], R);
       }
 
       solFEval.template Assemble<true>(R);
    }
 
    // complex_t = double
-   template <int num_elem>
    void MultAssembled(const Vector &x, Vector &y) const
    {
       y = 0.0;
@@ -229,14 +241,9 @@ public:
                             x.GetData(), y.GetData());
 
       const int NE = mesh.GetNE();
-      const int bNE = NE-NE%num_elem;
-      for (int el = 0; el < bNE; el += num_elem)
+      for (int el = 0; el < NE; el += TE)
       {
-         ElementAddMultAssembled<num_elem>(el, solFEval);
-      }
-      for (int el = bNE; el < NE; el++)
-      {
-         ElementAddMultAssembled<1>(el, solFEval);
+         ElementAddMultAssembled(el, solFEval);
       }
    }
 
@@ -249,10 +256,10 @@ public:
       solVecLayout_type solVecLayout(this->solVecLayout);
       solFESpace solFES(this->solFES);
 
-      TTensor3<dofs,vdim,1,complex_t> xy_dof;
+      TTensor3<dofs,vdim,BE,vcomplex_t> xy_dof;
 
       const int NE = mesh.GetNE();
-      for (int el = 0; el < NE; el++)
+      for (int el = 0; el < NE; el += TE)
       {
          solFES.SetElement(el);
 
@@ -266,17 +273,21 @@ public:
    {
       typedef typename meshType::FESpace_type meshFESpace;
       meshFESpace meshFES(mesh.t_fes);
-      typedef TTensor3<meshFE_type::dofs,sdim,1,real_t> lnodes_t;
+      typedef TTensor3<meshFE_type::dofs,sdim,BE,vreal_t> lnodes_t;
 
       const int NE = mesh.GetNE();
-      sNodes.SetSize(lnodes_t::size*NE);
-      real_t *lNodes = sNodes.GetData();
-      for (int el = 0; el < NE; el++)
+      // TODO: How do we make sure that this array is aligned properly, AND
+      //       the compiler knows that it is aligned? => ALIGN_32|ALIGN_64 when ready
+      const int NVE = (NE+TE-1)/TE;
+      vreal_t *vsNodes = new vreal_t[lnodes_t::size*NVE];
+      sNodes.NewDataAndSize(vsNodes[0].vec, (lnodes_t::size*SS)*NVE);
+      sNodes.MakeDataOwner();
+      for (int el = 0; el < NE; el += TE)
       {
          meshFES.SetElement(el);
          meshFES.VectorExtract(mesh.node_layout, mesh.Nodes,
-                               lnodes_t::layout, lNodes);
-         lNodes += lnodes_t::size;
+                               lnodes_t::layout, vsNodes);
+         vsNodes += lnodes_t::size;
       }
    }
 
@@ -284,45 +295,51 @@ public:
    // real_t = double
    void AssembleFromSerializedNodes(const Vector &sNodes)
    {
-      const int  BE = 1; // batch-size of elements
-      typedef typename kernel_t::template
-      CoefficientEval<IR,coeff_t,BE>::Type coeff_eval_t;
-
-      Trans_t T(this->mesh, this->meshEval);
+      Trans_t T(mesh, meshEval);
       coeff_eval_t wQ(int_rule, coeff);
 
       const int NE = mesh.GetNE();
-      if (!assembled_data)
+      if (assembled_data.Empty())
       {
-         assembled_data = new p_assembled_t[NE];
+         const int size = ((NE+TE-1)/TE)*BE;
+         assembled_data.New(size, assembled_data.GetMemoryType());
       }
-      for (int el = 0; el < NE; el++)
+      const vreal_t *vsNodes = (const vreal_t*)(sNodes.GetData());
+      for (int el = 0; el < NE; el += TE)
       {
-         typename T_result<BE>::Type F;
-         T.EvalSerialized(el, sNodes.GetData(), F);
+         typename T_result::Type F;
+         T.EvalSerialized(el, vsNodes, F);
 
          typename coeff_eval_t::result_t res;
          wQ.Eval(F, res);
 
-         kernel_t::Assemble(0, F, wQ, res, assembled_data[el]);
+         for (int k = 0; k < BE; k++)
+         {
+            kernel_t::Assemble(k, F, wQ, res, assembled_data[el/SS+k]);
+         }
       }
    }
 
    // complex_t = double
    void Serialize(const Vector &x, Vector &sx) const
    {
+      typedef TTensor3<dofs,vdim,BE,vcomplex_t> vdof_data_t;
+
       solVecLayout_t solVecLayout(this->solVecLayout);
-      typedef TTensor3<dofs,vdim,1,complex_t> vdof_data_t;
       solFESpace solFES(this->solFES);
 
       const int NE = mesh.GetNE();
-      sx.SetSize(vdim*dofs*NE);
-      complex_t *loc_sx = sx.GetData();
-      for (int el = 0; el < NE; el++)
+      // TODO: How do we make sure that this array is aligned properly, AND
+      //       the compiler knows that it is aligned? => ALIGN_32|ALIGN_64 when ready
+      const int NVE = (NE+TE-1)/TE;
+      vreal_t *vsx = new vreal_t[vdof_data_t::size*NVE];
+      sx.NewDataAndSize(vsx[0].vec, (vdof_data_t::size*SS)*NVE);
+      sx.MakeDataOwner();
+      for (int el = 0; el < NE; el += TE)
       {
          solFES.SetElement(el);
-         solFES.VectorExtract(solVecLayout, x, vdof_data_t::layout, loc_sx);
-         loc_sx += vdim*dofs;
+         solFES.VectorExtract(solVecLayout, x, vdof_data_t::layout, vsx);
+         vsx += vdof_data_t::size;
       }
    }
 
@@ -333,19 +350,23 @@ public:
       solFieldEval solFEval(solFES, solEval, solVecLayout, NULL, NULL);
 
       const int NE = mesh.GetNE();
-      const complex_t *loc_sx = sx.GetData();
-      complex_t *loc_sy = sy.GetData();
-      for (int el = 0; el < NE; el++)
+      const vreal_t *vsx = (const vreal_t*)(sx.GetData());
+      vreal_t *vsy = (vreal_t*)(sy.GetData());
+
+      for (int el = 0; el < NE; el += TE)
       {
-         typename S_spec<1>::DataType R;
-         solFEval.EvalSerialized(loc_sx, R);
+         typename S_spec::DataType R;
+         solFEval.EvalSerialized(vsx, R);
 
-         kernel_t::MultAssembled(0, assembled_data[el], R);
+         for (int k = 0; k < BE; k++)
+         {
+            kernel_t::MultAssembled(k, assembled_data[el/SS+k], R);
+         }
 
-         solFEval.template AssembleSerialized<false>(R, loc_sy);
+         solFEval.template AssembleSerialized<false>(R, vsy);
 
-         loc_sx += vdim*dofs;
-         loc_sy += vdim*dofs;
+         vsx += vdim*dofs*BE;
+         vsy += vdim*dofs*BE;
       }
    }
 #endif // MFEM_TEMPLATE_ENABLE_SERIALIZE
@@ -354,10 +375,6 @@ public:
    // complex_t = double
    void AssembleMatrix(SparseMatrix &M) const
    {
-      const int BE = 1; // batch-size of elements
-      typedef typename kernel_t::template
-      CoefficientEval<IR,coeff_t,BE>::Type coeff_eval_t;
-
       Trans_t T(mesh, meshEval);
       solFESpace solFES(this->solFES);
       solShapeEval solEval(this->solEval);
@@ -365,29 +382,39 @@ public:
       coeff_eval_t wQ(int_rule, coeff);
 
       const int NE = mesh.GetNE();
-      for (int el = 0; el < NE; el++)
+      for (int el = 0; el < NE; el += TE)
       {
-         f_assembled_t asm_qpt_data;
+         f_assembled_t asm_qpt_data[BE];
          {
-            typename T_result<BE>::Type F;
+            typename T_result::Type F;
             T.Eval(el, F);
 
             typename coeff_eval_t::result_t res;
             wQ.Eval(F, res);
 
-            kernel_t::Assemble(0, F, wQ, res, asm_qpt_data);
+            for (int k = 0; k < BE; k++)
+            {
+               kernel_t::Assemble(k, F, wQ, res, asm_qpt_data[k]);
+            }
          }
 
          // For now, when vdim > 1, assume block-diagonal matrix with the same
          // diagonal block for all components.
-         TMatrix<dofs,dofs> M_loc;
-         S_spec<BE>::ElementMatrix::Compute(
-            asm_qpt_data.layout, asm_qpt_data, M_loc.layout, M_loc, solEval);
-
-         solFES.SetElement(el);
-         for (int bi = 0; bi < vdim; bi++)
+         for (int k = 0; k < BE; k++)
          {
-            solFES.AssembleBlock(bi, bi, solVecLayout, M_loc, M);
+            const int el_k = el+SS*k;
+            if (el_k >= NE) { break; }
+
+            TMatrix<dofs,dofs,vcomplex_t> M_loc;
+            S_spec::ElementMatrix::Compute(
+               asm_qpt_data[k].layout, asm_qpt_data[k], M_loc.layout, M_loc,
+               solEval);
+
+            solFES.SetElement(el_k);
+            for (int bi = 0; bi < vdim; bi++)
+            {
+               solFES.AssembleBlock(bi, bi, solVecLayout, M_loc, M);
+            }
          }
       }
    }
@@ -396,37 +423,52 @@ public:
    // complex_t = double
    void AssembleMatrix(DenseTensor &M) const
    {
-      const int BE = 1; // batch-size of elements
-      typedef typename kernel_t::template
-      CoefficientEval<IR,coeff_t,BE>::Type coeff_eval_t;
-
       Trans_t T(mesh, meshEval);
       solShapeEval solEval(this->solEval);
       coeff_eval_t wQ(int_rule, coeff);
 
       const int NE = mesh.GetNE();
-      for (int el = 0; el < NE; el++)
+      for (int el = 0; el < NE; el += TE)
       {
-         f_assembled_t asm_qpt_data;
+         f_assembled_t asm_qpt_data[BE];
          {
-            typename T_result<BE>::Type F;
+            typename T_result::Type F;
             T.Eval(el, F);
 
             typename coeff_eval_t::result_t res;
             wQ.Eval(F, res);
 
-            kernel_t::Assemble(0, F, wQ, res, asm_qpt_data);
+            for (int k = 0; k < BE; k++)
+            {
+               kernel_t::Assemble(k, F, wQ, res, asm_qpt_data[k]);
+            }
          }
 
          // For now, when vdim > 1, assume block-diagonal matrix with the same
          // diagonal block for all components.
          // M is assumed to be (dof x dof x NE).
-         TMatrix<dofs,dofs> M_loc;
-         S_spec<BE>::ElementMatrix::Compute(
-            asm_qpt_data.layout, asm_qpt_data, M_loc.layout, M_loc, solEval);
+         for (int k = 0; k < BE; k++)
+         {
+            const int el_k = el+SS*k;
+            if (el_k >= NE) { break; }
 
-         complex_t *M_data = M.GetData(el);
-         M_loc.template AssignTo<AssignOp::Set>(M_data);
+            TMatrix<dofs,dofs,vcomplex_t> M_loc;
+            S_spec::ElementMatrix::Compute(
+               asm_qpt_data[k].layout, asm_qpt_data[k], M_loc.layout, M_loc,
+               solEval);
+
+            for (int s = 0; s < SS && el_k+s < NE; s++)
+            {
+               complex_t *M_data = M.GetData(el_k+s);
+               for (int j = 0; j < dofs; j++)
+               {
+                  for (int i = 0; i < dofs; i++)
+                  {
+                     M_data[j+dofs*i] = M_loc(i,j)[s];
+                  }
+               }
+            }
+         }
       }
    }
 
@@ -434,10 +476,6 @@ public:
    // complex_t = double
    void AssembleBilinearForm(BilinearForm &a) const
    {
-      const int BE = 1; // batch-size of elements
-      typedef typename kernel_t::template
-      CoefficientEval<IR,coeff_t,BE>::Type coeff_eval_t;
-
       Trans_t T(mesh, meshEval);
       solShapeEval solEval(this->solEval);
       coeff_eval_t wQ(int_rule, coeff);
@@ -448,55 +486,87 @@ public:
       DenseMatrix M_loc_perm(dofs*vdim,dofs*vdim); // initialized with zeros
 
       const int NE = mesh.GetNE();
-      for (int el = 0; el < NE; el++)
+      for (int el = 0; el < NE; el += TE)
       {
-         f_assembled_t asm_qpt_data;
+         f_assembled_t asm_qpt_data[BE];
          {
-            typename T_result<BE>::Type F;
+            typename T_result::Type F;
             T.Eval(el, F);
 
             typename coeff_eval_t::result_t res;
             wQ.Eval(F, res);
 
-            kernel_t::Assemble(0, F, wQ, res, asm_qpt_data);
+            for (int k = 0; k < BE; k++)
+            {
+               kernel_t::Assemble(k, F, wQ, res, asm_qpt_data[k]);
+            }
          }
 
          // For now, when vdim > 1, assume block-diagonal matrix with the same
          // diagonal block for all components.
-         TMatrix<dofs,dofs> M_loc;
-         S_spec<BE>::ElementMatrix::Compute(
-            asm_qpt_data.layout, asm_qpt_data, M_loc.layout, M_loc, solEval);
-
-         if (dof_map) // switch from tensor-product ordering
+         for (int k = 0; k < BE; k++)
          {
-            for (int i = 0; i < dofs; i++)
+            const int el_k = el+SS*k;
+            if (el_k >= NE) { break; }
+
+            TMatrix<dofs,dofs,vcomplex_t> M_loc;
+            S_spec::ElementMatrix::Compute(
+               asm_qpt_data[k].layout, asm_qpt_data[k], M_loc.layout, M_loc,
+               solEval);
+
+            if (dof_map) // switch from tensor-product ordering
             {
-               for (int j = 0; j < dofs; j++)
+               for (int s = 0; s < SS && el_k+s < NE; s++)
                {
-                  M_loc_perm(dof_map_[i],dof_map_[j]) = M_loc(i,j);
+                  for (int i = 0; i < dofs; i++)
+                  {
+                     for (int j = 0; j < dofs; j++)
+                     {
+                        M_loc_perm(dof_map_[i],dof_map_[j]) = M_loc(i,j)[s];
+                     }
+                  }
+                  for (int bi = 1; bi < vdim; bi++)
+                  {
+                     M_loc_perm.CopyMN(M_loc_perm, dofs, dofs, 0, 0,
+                                       bi*dofs, bi*dofs);
+                  }
+                  a.AssembleElementMatrix(el_k+s, M_loc_perm, vdofs);
                }
             }
-            for (int bi = 1; bi < vdim; bi++)
+            else if (SS == 1)
             {
-               M_loc_perm.CopyMN(M_loc_perm, dofs, dofs, 0, 0,
-                                 bi*dofs, bi*dofs);
-            }
-            a.AssembleElementMatrix(el, M_loc_perm, vdofs);
-         }
-         else
-         {
-            DenseMatrix DM(M_loc.data, dofs, dofs);
-            if (vdim == 1)
-            {
-               a.AssembleElementMatrix(el, DM, vdofs);
+               DenseMatrix DM(M_loc.data[0].vec, dofs, dofs);
+               if (vdim == 1)
+               {
+                  a.AssembleElementMatrix(el_k, DM, vdofs);
+               }
+               else
+               {
+                  for (int bi = 0; bi < vdim; bi++)
+                  {
+                     M_loc_perm.CopyMN(DM, dofs, dofs, 0, 0, bi*dofs, bi*dofs);
+                  }
+                  a.AssembleElementMatrix(el_k, M_loc_perm, vdofs);
+               }
             }
             else
             {
-               for (int bi = 0; bi < vdim; bi++)
+               for (int s = 0; s < SS && el_k+s < NE; s++)
                {
-                  M_loc_perm.CopyMN(DM, dofs, dofs, 0, 0, bi*dofs, bi*dofs);
+                  for (int i = 0; i < dofs; i++)
+                  {
+                     for (int j = 0; j < dofs; j++)
+                     {
+                        M_loc_perm(i,j) = M_loc(i,j)[s];
+                     }
+                  }
+                  for (int bi = 1; bi < vdim; bi++)
+                  {
+                     M_loc_perm.CopyMN(M_loc_perm, dofs, dofs, 0, 0,
+                                       bi*dofs, bi*dofs);
+                  }
+                  a.AssembleElementMatrix(el_k+s, M_loc_perm, vdofs);
                }
-               a.AssembleElementMatrix(el, M_loc_perm, vdofs);
             }
          }
       }
@@ -513,7 +583,7 @@ public:
       const int NE = mesh.GetNE();
       for (int el = 0; el < NE; el++)
       {
-         TTensor3<dofs,vdim,1,complex_t> x_dof, y_dof;
+         TTensor3<dofs,vdim,1,AutoSIMD<complex_t,1,1> > x_dof, y_dof;
 
          solFES.SetElement(el);
          solFES.VectorExtract(solVecLayout, x, x_dof.layout, x_dof);

fem/tbilininteg.hpp

@@ -65,10 +65,10 @@ struct TMassKernel
    template <int qpts>
    struct f_asm_data { typedef TVector<qpts,complex_t> type; };
 
-   template <typename IR, typename coeff_t, int NE>
+   template <typename IR, typename coeff_t, typename impl_traits_t>
    struct CoefficientEval
    {
-      typedef typename IntRuleCoefficient<IR,coeff_t,NE>::Type Type;
+      typedef typename IntRuleCoefficient<IR,coeff_t,impl_traits_t>::Type Type;
    };
 
    // Method used for un-assembled (matrix free) action.
@@ -180,10 +180,10 @@ struct TDiffusionKernel<1,1,complex_t>
    template <int qpts>
    struct f_asm_data { typedef TTensor3<qpts,1,1,complex_t> type; };
 
-   template <typename IR, typename coeff_t, int NE>
+   template <typename IR, typename coeff_t, typename impl_traits_t>
    struct CoefficientEval
    {
-      typedef typename IntRuleCoefficient<IR,coeff_t,NE>::Type Type;
+      typedef typename IntRuleCoefficient<IR,coeff_t,impl_traits_t>::Type Type;
    };
 
    // Method used for un-assembled (matrix free) action.
@@ -293,10 +293,10 @@ struct TDiffusionKernel<2,2,complex_t>
    template <int qpts>
    struct f_asm_data { typedef TTensor3<qpts,2,2,complex_t> type; };
 
-   template <typename IR, typename coeff_t, int NE>
+   template <typename IR, typename coeff_t, typename impl_traits_t>
    struct CoefficientEval
    {
-      typedef typename IntRuleCoefficient<IR,coeff_t,NE>::Type Type;
+      typedef typename IntRuleCoefficient<IR,coeff_t,impl_traits_t>::Type Type;
    };
 
    // Method used for un-assembled (matrix free) action.
@@ -434,10 +434,10 @@ struct TDiffusionKernel<3,3,complex_t>
    template <int qpts>
    struct f_asm_data { typedef TTensor3<qpts,3,3,complex_t> type; };
 
-   template <typename IR, typename coeff_t, int NE>
+   template <typename IR, typename coeff_t, typename impl_traits_t>
    struct CoefficientEval
    {
-      typedef typename IntRuleCoefficient<IR,coeff_t,NE>::Type Type;
+      typedef typename IntRuleCoefficient<IR,coeff_t,impl_traits_t>::Type Type;
    };
 
    // Method used for un-assembled (matrix free) action.

fem/tcoefficient.hpp

@@ -81,11 +81,15 @@ protected:
       {
          const int qpts = T_result_t::x_type::layout_type::dim_1;
          const int ne   = T_result_t::x_type::layout_type::dim_3;
+         const int vs   = sizeof(T.x[0])/sizeof(T.x[0][0]);
          for (int k = 0; k < ne; k++)
          {
             for (int i = 0; i < qpts; i++)
             {
-               c[l.ind(i,k)] = F.Eval1D(T.x(i,0,k));
+               for (int s = 0; s < vs; s++)
+               {
+                  c[l.ind(i,k)][s] = F.Eval1D(T.x(i,0,k)[s]);
+               }
             }
          }
       }
@@ -98,11 +102,15 @@ protected:
       {
          const int qpts = T_result_t::x_type::layout_type::dim_1;
          const int ne   = T_result_t::x_type::layout_type::dim_3;
+         const int vs   = sizeof(T.x[0])/sizeof(T.x[0][0]);
          for (int k = 0; k < ne; k++)
          {
             for (int i = 0; i < qpts; i++)
             {
-               c[l.ind(i,k)] = F.Eval2D(T.x(i,0,k), T.x(i,1,k));
+               for (int s = 0; s < vs; s++)
+               {
+                  c[l.ind(i,k)][s] = F.Eval2D(T.x(i,0,k)[s], T.x(i,1,k)[s]);
+               }
             }
          }
       }
@@ -115,11 +123,16 @@ protected:
       {
          const int qpts = T_result_t::x_type::layout_type::dim_1;
          const int ne   = T_result_t::x_type::layout_type::dim_3;
+         const int vs   = sizeof(T.x[0])/sizeof(T.x[0][0]);
          for (int k = 0; k < ne; k++)
          {
             for (int i = 0; i < qpts; i++)
             {
-               c[l.ind(i,k)] = F.Eval3D(T.x(i,0,k), T.x(i,1,k), T.x(i,2,k));
+               for (int s = 0; s < vs; s++)
+               {
+                  c[l.ind(i,k)][s] =
+                     F.Eval3D(T.x(i,0,k)[s], T.x(i,1,k)[s], T.x(i,2,k)[s]);
+               }
             }
          }
       }
@@ -170,9 +183,16 @@ public:
    void Eval(const T_result_t &T, const c_layout_t &l, c_data_t &c)
    {
       const int ne = T_result_t::ne;
+      const int vs = sizeof(T.attrib[0])/sizeof(T.attrib[0][0]);
+      MFEM_STATIC_ASSERT(vs == sizeof(c[0])/sizeof(c[0][0]), "");
       for (int i = 0; i < ne; i++)
       {
-         TAssign<AssignOp::Set>(l.ind2(i), c, constants(T.attrib[i]-1));
+         typename c_data_t::data_type ci;
+         for (int s = 0; s < vs; s++)
+         {
+            ci[s] = constants(T.attrib[i][s]-1);
+         }
+         TAssign<AssignOp::Set>(l.ind2(i), c, ci);
       }
    }
 };
@@ -243,12 +263,13 @@ public:
 
 /// Auxiliary class that is used to simplify the evaluation of a coefficient and
 /// scaling it by the weights of a quadrature rule.
-template <typename IR, typename coeff_t, int NE>
+template <typename IR, typename coeff_t, typename impl_traits_t>
 struct IntRuleCoefficient
 {
    static const int qpts = IR::qpts;
-   static const int ne   = NE;
+   static const int ne   = impl_traits_t::batch_size;
    typedef typename coeff_t::complex_type complex_type;
+   typedef typename impl_traits_t::vcomplex_t vcomplex_t;
 
    template <bool is_const, bool dummy> struct Aux;
 
@@ -277,7 +298,7 @@ struct IntRuleCoefficient
    // non-constant coefficient
    template <bool dummy> struct Aux<false,dummy>
    {
-      typedef TMatrix<qpts,ne,complex_type> result_t;
+      typedef TMatrix<qpts,ne,vcomplex_t> result_t;
 #ifdef MFEM_TEMPLATE_INTRULE_COEFF_PRECOMP
       TMatrix<qpts,1,typename IR::real_type> w;
 #else
@@ -312,7 +333,7 @@ struct IntRuleCoefficient
       }
 
       inline MFEM_ALWAYS_INLINE
-      const complex_type &get(const result_t &res, int i, int k) const
+      const vcomplex_t &get(const result_t &res, int i, int k) const
       {
          return res(i,k);
       }

fem/teltrans.hpp

@@ -64,9 +64,9 @@ public:
    // Templated struct Result, used to specify the type result that is computed
    // by the TElementTransformation::Eval() method and stored in this structure.
    // The template parameter EvalOps is a sum (bitwise or) of constants from
-   // the enum EvalOperations. The parameter NE is the number of elements to be
-   // processed in the Eval() method.
-   template<int EvalOps, int NE> struct Result;
+   // the enum EvalOperations. The type impl_traits_t specifies additional
+   // parameters and types to be used by the Eval() method.
+   template<int EvalOps, typename impl_traits_t> struct Result;
 
    static const int dim  = Mesh_t::dim;
    static const int sdim = Mesh_t::space_dim;
@@ -85,13 +85,17 @@ protected:
 
    const Element* const *elements;
 
-   template <int NE>
+   template <typename vint_t, int NE>
    inline MFEM_ALWAYS_INLINE
-   void SetAttributes(int el, int (&attrib)[NE]) const
+   void SetAttributes(int el, vint_t (&attrib)[NE]) const
    {
+      const int vsize = sizeof(vint_t)/sizeof(attrib[0][0]);
       for (int i = 0; i < NE; i++)
       {
-         attrib[i] = elements[el+i]->GetAttribute();
+         for (int j = 0; j < vsize; i++)
+         {
+            attrib[i][j] = elements[el+j+i*vsize]->GetAttribute();
+         }
       }
    }
 
@@ -106,25 +110,30 @@ public:
    { }
 
    // Evaluate coordinates and/or Jacobian matrices at quadrature points.
-   template<int EvalOps, int NE>
+   template<int EvalOps, typename impl_traits_t>
    inline MFEM_ALWAYS_INLINE
-   void Eval(int el, Result<EvalOps,NE> &F)
+   void Eval(int el, Result<EvalOps,impl_traits_t> &F)
    {
       F.Eval(el, *this);
    }
 
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
-   template<int EvalOps, int NE>
+   template<int EvalOps, typename impl_traits_t>
    inline MFEM_ALWAYS_INLINE
-   void EvalSerialized(int el, const real_t *nodeData, Result<EvalOps,NE> &F)
+   void EvalSerialized(int el, const typename impl_traits_t::vreal_t *nodeData,
+                       Result<EvalOps,impl_traits_t> &F)
    {
       F.EvalSerialized(el, *this, nodeData);
    }
 #endif
 
-   template <int NE> struct Result<0,NE> // 0 = EvalNone
+   // Specialization of the Result<> class
+
+   // Case EvalOps = 0 = EvalNone
+   template <typename it_t> struct Result<0,it_t>
    {
-      static const int ne = NE;
+      static const int ne = it_t::batch_size;
+      typedef typename it_t::vreal_t vreal_t;
       // x_type x;
       // Jt_type Jt;
       // int attrib[NE];
@@ -137,20 +146,23 @@ public:
       }
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
       inline MFEM_ALWAYS_INLINE
-      void EvalSerialized(int el, T_type &T, const real_t *nodeData) { }
+      void EvalSerialized(int el, T_type &T, const vreal_t *nodeData) { }
 #endif
    };
-   template <int NE> struct Result<1,NE> // 1 = EvalCoordinates
+
+   // Case EvalOps = 1 = EvalCoordinates
+   template <typename it_t> struct Result<1,it_t>
    {
-      static const int ne = NE;
+      static const int ne = it_t::batch_size;
+      typedef typename it_t::vreal_t vreal_t;
 #ifdef MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES
-      typedef TTensor3<qpts,sdim,NE,real_t,true> x_type;
+      typedef TTensor3<qpts,sdim,NE,vreal_t,true> x_type;
 #else
-      typedef TTensor3<qpts,sdim,NE,real_t/*,true*/> x_type;
+      typedef TTensor3<qpts,sdim,ne,vreal_t/*,true*/> x_type;
 #endif
       x_type x;
 
-      typedef TTensor3<dofs,sdim,NE,real_t> nodes_dof_t;
+      typedef TTensor3<dofs,sdim,ne,vreal_t> nodes_dof_t;
 #ifdef MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES
       nodes_dof_t nodes_dof;
 #endif
@@ -159,8 +171,8 @@ public:
       void Eval(int el, T_type &T)
       {
 #ifdef MFEM_TEMPLATE_ELTRANS_HAS_NODE_DOFS
-         MFEM_STATIC_ASSERT(NE == 1, "only NE == 1 is supported");
-         TTensor3<dofs,sdim,1,real_t> &nodes_dof = T.nodes_dof;
+         MFEM_STATIC_ASSERT(ne == 1, "only ne == 1 is supported");
+         TTensor3<dofs,sdim,1,vreal_t> &nodes_dof = T.nodes_dof;
 #elif !defined(MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES)
          nodes_dof_t nodes_dof;
 #endif
@@ -173,25 +185,30 @@ public:
 
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
       inline MFEM_ALWAYS_INLINE
-      void EvalSerialized(int el, T_type &T, const real_t *nodeData)
+      void EvalSerialized(int el, T_type &T, const vreal_t *nodeData)
       {
+         const int SS = sizeof(nodeData[0])/sizeof(nodeData[0][0]);
+         MFEM_ASSERT(el % (SS*ne) == 0, "invalid element index: " << el);
          T.evaluator.Calc(nodes_dof_t::layout.merge_23(),
-                          &nodeData[el*nodes_dof_t::size],
+                          &nodeData[el/SS*nodes_dof_t::size],
                           x.layout.merge_23(), x);
       }
 #endif
    };
-   template <int NE> struct Result<2,NE> // 2 = EvalJacobians
+
+   // Case EvalOps = 2 = EvalJacobians
+   template <typename it_t> struct Result<2,it_t>
    {
-      static const int ne = NE;
+      static const int ne = it_t::batch_size;
+      typedef typename it_t::vreal_t vreal_t;
 #ifdef MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES
-      typedef TTensor4<qpts,dim,sdim,NE,real_t,true> Jt_type;
+      typedef TTensor4<qpts,dim,sdim,ne,vreal_t,true> Jt_type;
 #else
-      typedef TTensor4<qpts,dim,sdim,NE,real_t/*,true*/> Jt_type;
+      typedef TTensor4<qpts,dim,sdim,ne,vreal_t/*,true*/> Jt_type;
 #endif
       Jt_type Jt;
 
-      typedef TTensor3<dofs,sdim,NE,real_t> nodes_dof_t;
+      typedef TTensor3<dofs,sdim,ne,vreal_t> nodes_dof_t;
 #ifdef MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES
       nodes_dof_t nodes_dof;
 #endif
@@ -200,8 +217,8 @@ public:
       void Eval(int el, T_type &T)
       {
 #ifdef MFEM_TEMPLATE_ELTRANS_HAS_NODE_DOFS
-         MFEM_STATIC_ASSERT(NE == 1, "only NE == 1 is supported");
-         TTensor3<dofs,sdim,1,real_t> &nodes_dof = T.nodes_dof;
+         MFEM_STATIC_ASSERT(ne == 1, "only ne == 1 is supported");
+         TTensor3<dofs,sdim,1,vreal_t> &nodes_dof = T.nodes_dof;
 #elif !defined(MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES)
          nodes_dof_t nodes_dof;
 #endif
@@ -214,27 +231,32 @@ public:
 
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
       inline MFEM_ALWAYS_INLINE
-      void EvalSerialized(int el, T_type &T, const real_t *nodeData)
+      void EvalSerialized(int el, T_type &T, const vreal_t *nodeData)
       {
+         const int SS = sizeof(nodeData[0])/sizeof(nodeData[0][0]);
+         MFEM_ASSERT(el % (SS*ne) == 0, "invalid element index: " << el);
          T.evaluator.CalcGrad(nodes_dof_t::layout.merge_23(),
-                              &nodeData[el*nodes_dof_t::size],
+                              &nodeData[el/SS*nodes_dof_t::size],
                               Jt.layout.merge_34(), Jt);
       }
 #endif
    };
-   template <int NE> struct Result<3,NE> // 3 = EvalCoordinates|EvalJacobians
+
+   // Case EvalOps = 3 = EvalCoordinates|EvalJacobians
+   template <typename it_t> struct Result<3,it_t>
    {
-      static const int ne = NE;
-      typedef TTensor3<qpts,sdim,NE,real_t,true> x_type;
+      static const int ne = it_t::batch_size;
+      typedef typename it_t::vreal_t vreal_t;
+      typedef TTensor3<qpts,sdim,ne,vreal_t,true> x_type;
       x_type x;
 #ifdef MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES
-      typedef TTensor4<qpts,dim,sdim,NE,real_t,true> Jt_type;
+      typedef TTensor4<qpts,dim,sdim,ne,vreal_t,true> Jt_type;
 #else
-      typedef TTensor4<qpts,dim,sdim,NE,real_t/*,true*/> Jt_type;
+      typedef TTensor4<qpts,dim,sdim,ne,vreal_t/*,true*/> Jt_type;
 #endif
       Jt_type Jt;
 
-      typedef TTensor3<dofs,sdim,NE,real_t> nodes_dof_t;
+      typedef TTensor3<dofs,sdim,ne,vreal_t> nodes_dof_t;
 #ifdef MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES
       nodes_dof_t nodes_dof;
 #endif
@@ -243,8 +265,8 @@ public:
       void Eval(int el, T_type &T)
       {
 #ifdef MFEM_TEMPLATE_ELTRANS_HAS_NODE_DOFS
-         MFEM_STATIC_ASSERT(NE == 1, "only NE == 1 is supported");
-         TTensor3<dofs,sdim,1,real_t> &nodes_dof = T.nodes_dof;
+         MFEM_STATIC_ASSERT(ne == 1, "only ne == 1 is supported");
+         TTensor3<dofs,sdim,1,vreal_t> &nodes_dof = T.nodes_dof;
 #elif !defined(MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES)
          nodes_dof_t nodes_dof;
 #endif
@@ -259,39 +281,45 @@ public:
 
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
       inline MFEM_ALWAYS_INLINE
-      void EvalSerialized(int el, T_type &T, const real_t *nodeData)
+      void EvalSerialized(int el, T_type &T, const vreal_t *nodeData)
       {
+         const int SS = sizeof(nodeData[0])/sizeof(nodeData[0][0]);
+         MFEM_ASSERT(el % (SS*ne) == 0, "invalid element index: " << el);
          T.evaluator.Calc(nodes_dof_t::layout.merge_23(),
-                          &nodeData[el*nodes_dof_t::size],
+                          &nodeData[el/SS*nodes_dof_t::size],
                           x.layout.merge_23(), x);
          T.evaluator.CalcGrad(nodes_dof_t::layout.merge_23(),
-                              &nodeData[el*nodes_dof_t::size],
+                              &nodeData[el/SS*nodes_dof_t::size],
                               Jt.layout.merge_34(), Jt);
       }
 #endif
    };
-   template <int NE> struct Result<6,NE> // 6 = EvalJacobians|LoadAttributes
+
+   // Case EvalOps = 6 = EvalJacobians|LoadAttributes
+   template <typename it_t> struct Result<6,it_t>
    {
-      static const int ne = NE;
+      static const int ne = it_t::batch_size;
+      typedef typename it_t::vreal_t vreal_t;
+      typedef typename it_t::vint_t  vint_t;
 #ifdef MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES
-      typedef TTensor4<qpts,dim,sdim,NE,real_t,true> Jt_type;
+      typedef TTensor4<qpts,dim,sdim,ne,vreal_t,true> Jt_type;
 #else
-      typedef TTensor4<qpts,dim,sdim,NE,real_t/*,true*/> Jt_type;
+      typedef TTensor4<qpts,dim,sdim,ne,vreal_t/*,true*/> Jt_type;
 #endif
       Jt_type Jt;
 
-      typedef TTensor3<dofs,sdim,NE,real_t> nodes_dof_t;
+      typedef TTensor3<dofs,sdim,ne,vreal_t> nodes_dof_t;
 #ifdef MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES
       nodes_dof_t nodes_dof;
 #endif
-      int attrib[NE];
+      vint_t attrib[ne];
 
       inline MFEM_ALWAYS_INLINE
       void Eval(int el, T_type &T)
       {
 #ifdef MFEM_TEMPLATE_ELTRANS_HAS_NODE_DOFS
-         MFEM_STATIC_ASSERT(NE == 1, "only NE == 1 is supported");
-         TTensor3<dofs,sdim,1,real_t> &nodes_dof = T.nodes_dof;
+         MFEM_STATIC_ASSERT(ne == 1, "only ne == 1 is supported");
+         TTensor3<dofs,sdim,1,vreal_t> &nodes_dof = T.nodes_dof;
 #elif !defined(MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES)
          nodes_dof_t nodes_dof;
 #endif
@@ -305,26 +333,31 @@ public:
 
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
       inline MFEM_ALWAYS_INLINE
-      void EvalSerialized(int el, T_type &T, const real_t *nodeData)
+      void EvalSerialized(int el, T_type &T, const vreal_t *nodeData)
       {
+         const int SS = sizeof(nodeData[0])/sizeof(nodeData[0][0]);
+         MFEM_ASSERT(el % (SS*ne) == 0, "invalid element index: " << el);
          T.evaluator.CalcGrad(nodes_dof_t::layout.merge_23(),
-                              &nodeData[el*nodes_dof_t::size],
+                              &nodeData[el/SS*nodes_dof_t::size],
                               Jt.layout.merge_34(), Jt);
          T.SetAttributes(el, attrib);
       }
 #endif
    };
-   template <int NE> struct Result<10,NE> // 10 = EvalJacobians|LoadElementIdxs
+
+   // Case EvalOps = 10 = EvalJacobians|LoadElementIdxs
+   template <typename it_t> struct Result<10,it_t>
    {
-      static const int ne = NE;
+      static const int ne = it_t::batch_size;
+      typedef typename it_t::vreal_t vreal_t;
 #ifdef MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES
-      typedef TTensor4<qpts,dim,sdim,NE,real_t,true> Jt_type;
+      typedef TTensor4<qpts,dim,sdim,ne,vreal_t,true> Jt_type;
 #else
-      typedef TTensor4<qpts,dim,sdim,NE,real_t/*,true*/> Jt_type;
+      typedef TTensor4<qpts,dim,sdim,ne,vreal_t/*,true*/> Jt_type;
 #endif
       Jt_type Jt;
 
-      typedef TTensor3<dofs,sdim,NE,real_t> nodes_dof_t;
+      typedef TTensor3<dofs,sdim,ne,vreal_t> nodes_dof_t;
 #ifdef MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES
       nodes_dof_t nodes_dof;
 #endif
@@ -334,8 +367,8 @@ public:
       void Eval(int el, T_type &T)
       {
 #ifdef MFEM_TEMPLATE_ELTRANS_HAS_NODE_DOFS
-         MFEM_STATIC_ASSERT(NE == 1, "only NE == 1 is supported");
-         TTensor3<dofs,sdim,1,real_t> &nodes_dof = T.nodes_dof;
+         MFEM_STATIC_ASSERT(ne == 1, "only ne == 1 is supported");
+         TTensor3<dofs,sdim,1,vreal_t> &nodes_dof = T.nodes_dof;
 #elif !defined(MFEM_TEMPLATE_ELTRANS_RESULT_HAS_NODES)
          nodes_dof_t nodes_dof;
 #endif
@@ -349,10 +382,12 @@ public:
 
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
       inline MFEM_ALWAYS_INLINE
-      void EvalSerialized(int el, T_type &T, const real_t *nodeData)
+      void EvalSerialized(int el, T_type &T, const vreal_t *nodeData)
       {
+         const int SS = sizeof(nodeData[0])/sizeof(nodeData[0][0]);
+         MFEM_ASSERT(el % (SS*ne) == 0, "invalid element index: " << el);
          T.evaluator.CalcGrad(nodes_dof_t::layout.merge_23(),
-                              &nodeData[el*nodes_dof_t::size],
+                              &nodeData[el/SS*nodes_dof_t::size],
                               Jt.layout.merge_34(), Jt);
          first_elem_idx = el;
       }

fem/tevaluator.hpp

@@ -58,7 +58,7 @@ public:
    // dof_layout is (DOF x NumComp) and qpt_layout is (NIP x NumComp).
    template <typename dof_layout_t, typename dof_data_t,
              typename qpt_layout_t, typename qpt_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Calc(const dof_layout_t &dof_layout, const dof_data_t &dof_data,
              const qpt_layout_t &qpt_layout, qpt_data_t &qpt_data) const
    {
@@ -81,7 +81,7 @@ public:
    template <bool Add,
              typename qpt_layout_t, typename qpt_data_t,
              typename dof_layout_t, typename dof_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcT(const qpt_layout_t &qpt_layout, const qpt_data_t &qpt_data,
               const dof_layout_t &dof_layout, dof_data_t &dof_data) const
    {
@@ -103,7 +103,7 @@ public:
    // dof_layout is (DOF x NumComp) and grad_layout is (NIP x DIM x NumComp).
    template <typename dof_layout_t, typename dof_data_t,
              typename grad_layout_t, typename grad_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcGrad(const dof_layout_t  &dof_layout,
                  const dof_data_t    &dof_data,
                  const grad_layout_t &grad_layout,
@@ -129,7 +129,7 @@ public:
    template <bool Add,
              typename grad_layout_t, typename grad_data_t,
              typename dof_layout_t, typename dof_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcGradT(const grad_layout_t &grad_layout,
                   const grad_data_t   &grad_data,
                   const dof_layout_t  &dof_layout,
@@ -154,7 +154,7 @@ public:
    // qpt_layout is (NIP x NumComp), M_layout is (DOF x DOF x NumComp)
    template <typename qpt_layout_t, typename qpt_data_t,
              typename M_layout_t, typename M_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Assemble(const qpt_layout_t &qpt_layout, const qpt_data_t &qpt_data,
                  const M_layout_t &M_layout, M_data_t &M_data) const
    {
@@ -178,14 +178,15 @@ public:
    // D_layout is (DOF x DOF x NumComp).
    template <typename qpt_layout_t, typename qpt_data_t,
              typename D_layout_t, typename D_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void AssembleGradGrad(const qpt_layout_t &qpt_layout,
                          const qpt_data_t   &qpt_data,
                          const D_layout_t   &D_layout,
                          D_data_t           &D_data) const
    {
       const int NC = qpt_layout_t::dim_4;
-      TTensor4<NIP,DIM,DOF,NC> F;
+      typedef typename qpt_data_t::data_type entry_type;
+      TTensor4<NIP,DIM,DOF,NC,entry_type> F;
       for (int k = 0; k < NC; k++)
       {
          // Next loop performs a batch of matrix-matrix products of size
@@ -224,7 +225,7 @@ public:
    // dof_layout is (DOF x NumComp) and qpt_layout is (NIP x NumComp).
    template <typename dof_layout_t, typename dof_data_t,
              typename qpt_layout_t, typename qpt_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Calc(const dof_layout_t &dof_layout, const dof_data_t &dof_data,
              const qpt_layout_t &qpt_layout, qpt_data_t &qpt_data) const
    {
@@ -238,7 +239,7 @@ public:
    template <bool Add,
              typename qpt_layout_t, typename qpt_data_t,
              typename dof_layout_t, typename dof_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcT(const qpt_layout_t &qpt_layout, const qpt_data_t &qpt_data,
               const dof_layout_t &dof_layout, dof_data_t &dof_data) const
    {
@@ -251,7 +252,7 @@ public:
    // dof_layout is (DOF x NumComp) and grad_layout is (NIP x DIM x NumComp).
    template <typename dof_layout_t, typename dof_data_t,
              typename grad_layout_t, typename grad_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcGrad(const dof_layout_t  &dof_layout,
                  const dof_data_t    &dof_data,
                  const grad_layout_t &grad_layout,
@@ -268,7 +269,7 @@ public:
    template <bool Add,
              typename grad_layout_t, typename grad_data_t,
              typename dof_layout_t, typename dof_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcGradT(const grad_layout_t &grad_layout,
                   const grad_data_t   &grad_data,
                   const dof_layout_t  &dof_layout,
@@ -285,7 +286,7 @@ public:
    // qpt_layout is (NIP x NumComp), M_layout is (DOF x DOF x NumComp)
    template <typename qpt_layout_t, typename qpt_data_t,
              typename M_layout_t, typename M_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Assemble(const qpt_layout_t &qpt_layout, const qpt_data_t &qpt_data,
                  const M_layout_t &M_layout, M_data_t &M_data) const
    {
@@ -309,7 +310,7 @@ public:
    // D_layout is (DOF x DOF x NumComp).
    template <typename qpt_layout_t, typename qpt_data_t,
              typename D_layout_t, typename D_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void AssembleGradGrad(const qpt_layout_t &qpt_layout,
                          const qpt_data_t   &qpt_data,
                          const D_layout_t   &D_layout,
@@ -348,13 +349,14 @@ public:
    template <bool Dx, bool Dy,
              typename dof_layout_t, typename dof_data_t,
              typename qpt_layout_t, typename qpt_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Calc(const dof_layout_t &dof_layout, const dof_data_t &dof_data,
              const qpt_layout_t &qpt_layout, qpt_data_t &qpt_data) const
    {
       const int NC = dof_layout_t::dim_2;
+      typedef typename qpt_data_t::data_type entry_type;
       // DOF x DOF x NC --> NIP x DOF x NC --> NIP x NIP x NC
-      TTensor3<NIP,DOF,NC> A;
+      TTensor3<NIP,DOF,NC,entry_type> A;
 
       // (1) A_{i,j,k} = \sum_s B_1d_{i,s} dof_data_{s,j,k}
       Mult_2_1<false>(B_1d.layout, Dx ? G_1d : B_1d,
@@ -370,7 +372,7 @@ public:
    // dof_layout is (TDOF x NumComp) and qpt_layout is (TNIP x NumComp).
    template <typename dof_layout_t, typename dof_data_t,
              typename qpt_layout_t, typename qpt_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Calc(const dof_layout_t &dof_layout, const dof_data_t &dof_data,
              const qpt_layout_t &qpt_layout, qpt_data_t &qpt_data) const
    {
@@ -380,13 +382,14 @@ public:
    template <bool Dx, bool Dy, bool Add,
              typename qpt_layout_t, typename qpt_data_t,
              typename dof_layout_t, typename dof_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcT(const qpt_layout_t &qpt_layout, const qpt_data_t &qpt_data,
               const dof_layout_t &dof_layout, dof_data_t &dof_data) const
    {
       const int NC = dof_layout_t::dim_2;
+      typedef typename qpt_data_t::data_type entry_type;
       // NIP x NIP X NC --> NIP x DOF x NC --> DOF x DOF x NC
-      TTensor3<NIP,DOF,NC> A;
+      TTensor3<NIP,DOF,NC,entry_type> A;
 
       // (1) A_{i,j,k} = \sum_s B_1d_{s,j} qpt_data_{i,s,k}
       Mult_1_2<false>(B_1d.layout, Dy ? G_1d : B_1d,
@@ -403,7 +406,7 @@ public:
    template <bool Add,
              typename qpt_layout_t, typename qpt_data_t,
              typename dof_layout_t, typename dof_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcT(const qpt_layout_t &qpt_layout, const qpt_data_t &qpt_data,
               const dof_layout_t &dof_layout, dof_data_t &dof_data) const
    {
@@ -414,7 +417,7 @@ public:
    // dof_layout is (TDOF x NumComp) and grad_layout is (TNIP x DIM x NumComp).
    template <typename dof_layout_t, typename dof_data_t,
              typename grad_layout_t, typename grad_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcGrad(const dof_layout_t  &dof_layout,
                  const dof_data_t    &dof_data,
                  const grad_layout_t &grad_layout,
@@ -432,7 +435,7 @@ public:
    template <bool Add,
              typename grad_layout_t, typename grad_data_t,
              typename dof_layout_t, typename dof_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcGradT(const grad_layout_t &grad_layout,
                   const grad_data_t   &grad_data,
                   const dof_layout_t  &dof_layout,
@@ -448,11 +451,12 @@ public:
    // qpt_layout is (TNIP x NumComp), M_layout is (TDOF x TDOF x NumComp)
    template <typename qpt_layout_t, typename qpt_data_t,
              typename M_layout_t, typename M_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Assemble(const qpt_layout_t &qpt_layout, const qpt_data_t &qpt_data,
                  const M_layout_t &M_layout, M_data_t &M_data) const
    {
       const int NC = qpt_layout_t::dim_2;
+      typedef typename qpt_data_t::data_type entry_type;
 
       // Using TensorAssemble: <I,NIP,J> --> <DOF,I,DOF,J>
 
@@ -469,7 +473,7 @@ public:
          TTensor3<DOF,NIP,DOF*NC>::layout, A,
          M_layout.merge_23().template split_12<DOF,DOF,DOF,DOF*NC>(), M_data);
 #elif 1
-      TTensor4<DOF,NIP,DOF,NC> A;
+      TTensor4<DOF,NIP,DOF,NC,entry_type> A;
       // qpt_data<NIP1,NIP2,NC> --> A<DOF2,NIP1,DOF2,NC>
       TensorAssemble<false>(
          Bt_1d.layout, Bt_1d, B_1d.layout, B_1d,
@@ -510,14 +514,15 @@ public:
    template <int D1, int D2, bool Add,
              typename qpt_layout_t, typename qpt_data_t,
              typename D_layout_t, typename D_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Assemble(const qpt_layout_t &qpt_layout,
                  const qpt_data_t   &qpt_data,
                  const D_layout_t   &D_layout,
                  D_data_t           &D_data) const
    {
       const int NC = qpt_layout_t::dim_2;
-      TTensor4<DOF,NIP,DOF,NC> A;
+      typedef typename qpt_data_t::data_type entry_type;
+      TTensor4<DOF,NIP,DOF,NC,entry_type> A;
 
       // Using TensorAssemble: <I,NIP,J> --> <DOF,I,DOF,J>
 
@@ -531,7 +536,7 @@ public:
       TensorAssemble<Add>(
          Bt_1d.layout, D1 == 1 ? Bt_1d : Gt_1d,
          B_1d.layout, D2 == 1 ? B_1d : G_1d,
-         TTensor3<DOF,NIP,DOF*NC>::layout, A,
+         A.layout.merge_34(), A,
          D_layout.merge_23().template split_12<DOF,DOF,DOF,DOF*NC>(), D_data);
    }
 
@@ -540,7 +545,7 @@ public:
    // D_layout is (TDOF x TDOF x NumComp).
    template <typename qpt_layout_t, typename qpt_data_t,
              typename D_layout_t, typename D_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void AssembleGradGrad(const qpt_layout_t &qpt_layout,
                          const qpt_data_t   &qpt_data,
                          const D_layout_t   &D_layout,
@@ -624,13 +629,14 @@ public:
    template <bool Dx, bool Dy, bool Dz,
              typename dof_layout_t, typename dof_data_t,
              typename qpt_layout_t, typename qpt_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Calc(const dof_layout_t &dof_layout, const dof_data_t &dof_data,
              const qpt_layout_t &qpt_layout, qpt_data_t &qpt_data) const
    {
       const int NC = dof_layout_t::dim_2;
-      TVector<NIP*DOF*DOF*NC> QDD;
-      TVector<NIP*NIP*DOF*NC> QQD;
+      typedef typename qpt_data_t::data_type entry_type;
+      TVector<NIP*DOF*DOF*NC,entry_type> QDD;
+      TVector<NIP*NIP*DOF*NC,entry_type> QQD;
 
       // QDD_{i,jj,k} = \sum_s B_1d_{i,s} dof_data_{s,jj,k}
       Mult_2_1<false>(B_1d.layout, Dx ? G_1d : B_1d,
@@ -650,7 +656,7 @@ public:
    // dof_layout is (TDOF x NumComp) and qpt_layout is (TNIP x NumComp).
    template <typename dof_layout_t, typename dof_data_t,
              typename qpt_layout_t, typename qpt_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Calc(const dof_layout_t &dof_layout, const dof_data_t &dof_data,
              const qpt_layout_t &qpt_layout, qpt_data_t &qpt_data) const
    {
@@ -660,13 +666,14 @@ public:
    template <bool Dx, bool Dy, bool Dz, bool Add,
              typename qpt_layout_t, typename qpt_data_t,
              typename dof_layout_t, typename dof_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcT(const qpt_layout_t &qpt_layout, const qpt_data_t &qpt_data,
               const dof_layout_t &dof_layout, dof_data_t &dof_data) const
    {
       const int NC = dof_layout_t::dim_2;
-      TVector<NIP*DOF*DOF*NC> QDD;
-      TVector<NIP*NIP*DOF*NC> QQD;
+      typedef typename qpt_data_t::data_type entry_type;
+      TVector<NIP*DOF*DOF*NC,entry_type> QDD;
+      TVector<NIP*NIP*DOF*NC,entry_type> QQD;
 
       // QQD_{ii,j,k} = \sum_s B_1d_{s,j} qpt_data_{ii,s,k}
       Mult_1_2<false>(B_1d.layout, Dz ? G_1d : B_1d,
@@ -687,7 +694,7 @@ public:
    template <bool Add,
              typename qpt_layout_t, typename qpt_data_t,
              typename dof_layout_t, typename dof_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcT(const qpt_layout_t &qpt_layout, const qpt_data_t &qpt_data,
               const dof_layout_t &dof_layout, dof_data_t &dof_data) const
    {
@@ -698,7 +705,7 @@ public:
    // dof_layout is (TDOF x NumComp) and grad_layout is (TNIP x DIM x NumComp).
    template <typename dof_layout_t, typename dof_data_t,
              typename grad_layout_t, typename grad_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcGrad(const dof_layout_t  &dof_layout,
                  const dof_data_t    &dof_data,
                  const grad_layout_t &grad_layout,
@@ -720,7 +727,7 @@ public:
    template <bool Add,
              typename grad_layout_t, typename grad_data_t,
              typename dof_layout_t, typename dof_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void CalcGradT(const grad_layout_t &grad_layout,
                   const grad_data_t   &grad_data,
                   const dof_layout_t  &dof_layout,
@@ -738,13 +745,14 @@ public:
    // qpt_layout is (TNIP x NumComp), M_layout is (TDOF x TDOF x NumComp)
    template <typename qpt_layout_t, typename qpt_data_t,
              typename M_layout_t, typename M_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Assemble(const qpt_layout_t &qpt_layout, const qpt_data_t &qpt_data,
                  const M_layout_t &M_layout, M_data_t &M_data) const
    {
       const int NC = qpt_layout_t::dim_2;
-      TTensor4<DOF,NIP*NIP,DOF,NC> A1;
-      TTensor4<DOF,DOF*NIP,DOF,DOF*NC> A2;
+      typedef typename qpt_data_t::data_type entry_type;
+      TTensor4<DOF,NIP*NIP,DOF,NC,entry_type> A1;
+      TTensor4<DOF,DOF*NIP,DOF,DOF*NC,entry_type> A2;
 
       // Using TensorAssemble: <I,NIP,J> --> <DOF,I,DOF,J>
 
@@ -788,15 +796,16 @@ public:
    template <int D1, int D2, bool Add,
              typename qpt_layout_t, typename qpt_data_t,
              typename D_layout_t, typename D_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Assemble(const qpt_layout_t &qpt_layout,
                  const qpt_data_t   &qpt_data,
                  const D_layout_t   &D_layout,
                  D_data_t           &D_data) const
    {
       const int NC = qpt_layout_t::dim_2;
-      TTensor4<DOF,NIP*NIP,DOF,NC> A1;
-      TTensor4<DOF,DOF*NIP,DOF,DOF*NC> A2;
+      typedef typename qpt_data_t::data_type entry_type;
+      TTensor4<DOF,NIP*NIP,DOF,NC,entry_type> A1;
+      TTensor4<DOF,DOF*NIP,DOF,DOF*NC,entry_type> A2;
 
       // Using TensorAssemble: <I,NIP,J> --> <DOF,I,DOF,J>
 
@@ -824,7 +833,7 @@ public:
 #if 0
    template <typename qpt_layout_t, typename qpt_data_t,
              typename D_layout_t, typename D_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void Assemble(int D1, int D2,
                  const qpt_layout_t &qpt_layout,
                  const qpt_data_t   &qpt_data,
@@ -864,7 +873,7 @@ public:
    // D_layout is (TDOF x TDOF x NumComp).
    template <typename qpt_layout_t, typename qpt_data_t,
              typename D_layout_t, typename D_data_t>
-   MFEM_ALWAYS_INLINE
+   inline MFEM_ALWAYS_INLINE
    void AssembleGradGrad(const qpt_layout_t &qpt_layout,
                          const qpt_data_t   &qpt_data,
                          const D_layout_t   &D_layout,
@@ -1055,7 +1064,7 @@ public:
    void GetValues(int el, const val_layout_t &l, val_data_t &vals)
    {
       const int ne = val_layout_t::dim_3;
-      TTensor3<dofs,vdim,ne,complex_type> val_dofs;
+      TTensor3<dofs,vdim,ne,typename val_data_t::data_type> val_dofs;
       SetElement(el);
       fespace.VectorExtract(vec_layout, data_in, val_dofs.layout, val_dofs);
       shapeEval.Calc(val_dofs.layout.merge_23(), val_dofs, l.merge_23(), vals);
@@ -1067,7 +1076,7 @@ public:
    void GetGradients(int el, const grad_layout_t &l, grad_data_t &grad)
    {
       const int ne = grad_layout_t::dim_4;
-      TTensor3<dofs,vdim,ne,complex_type> val_dofs;
+      TTensor3<dofs,vdim,ne,typename grad_data_t::data_type> val_dofs;
       SetElement(el);
       fespace.VectorExtract(vec_layout, data_in, val_dofs.layout, val_dofs);
       shapeEval.CalcGrad(val_dofs.layout.merge_23(), val_dofs,
@@ -1112,14 +1121,16 @@ public:
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
    template <typename DataType>
    inline MFEM_ALWAYS_INLINE
-   void EvalSerialized(const complex_t *loc_dofs, DataType &F)
+   void EvalSerialized(const typename DataType::vcomplex_t *loc_dofs,
+                       DataType &F)
    {
       Action<DataType::InData,true>::EvalSerialized(*this, loc_dofs, F);
    }
 
    template <bool Add, typename DataType>
    inline MFEM_ALWAYS_INLINE
-   void AssembleSerialized(const DataType &F, complex_t *loc_dofs)
+   void AssembleSerialized(const DataType &F,
+                           typename DataType::vcomplex_t *loc_dofs)
    {
       Action<DataType::OutData,true>::
       template AssembleSerialized<Add>(*this, F, loc_dofs);
@@ -1138,56 +1149,61 @@ public:
 
    // Auxiliary templated struct AData, used by the Eval() and Assemble()
    // methods. The template parameter IOData is "bitwise or" of constants from
-   // the enum InOutData. The parameter NE is the number of elements to be
-   // processed in the Eval() and Assemble() methods.
-   template<int IOData, int NE> struct AData;
+   // the enum InOutData. The type impl_traits_t specifies parameters and types
+   // to be used in the Eval() and Assemble() methods.
+   template<int IOData, typename impl_traits_t> struct AData;
 
-   template <int NE> struct AData<0,NE> // 0 = None
+   template <typename it_t> struct AData<0,it_t> // 0 = None
    {
       // Do we need this?
    };
 
-   template <int NE> struct AData<1,NE> // 1 = Values
+   template <typename it_t> struct AData<1,it_t> // 1 = Values
    {
+      static const int ne = it_t::batch_size;
+      typedef typename it_t::vcomplex_t vcomplex_t;
 #ifdef MFEM_TEMPLATE_FIELD_EVAL_DATA_HAS_DOFS
-      typedef TTensor3<dofs,vdim,NE,complex_t,true> val_dofs_t;
+      typedef TTensor3<dofs,vdim,ne,vcomplex_t,true> val_dofs_t;
       val_dofs_t val_dofs;
 #else
-      typedef TTensor3<dofs,vdim,NE,complex_t> val_dofs_t;
+      typedef TTensor3<dofs,vdim,ne,vcomplex_t> val_dofs_t;
 #endif
-      TTensor3<qpts,vdim,NE,complex_t>      val_qpts;
+      TTensor3<qpts,vdim,ne,vcomplex_t>      val_qpts;
    };
 
-   template <int NE> struct AData<2,NE> // 2 = Gradients
+   template <typename it_t> struct AData<2,it_t> // 2 = Gradients
    {
+      static const int ne = it_t::batch_size;
+      typedef typename it_t::vcomplex_t vcomplex_t;
 #ifdef MFEM_TEMPLATE_FIELD_EVAL_DATA_HAS_DOFS
-      typedef TTensor3<dofs,vdim,NE,complex_t,true> val_dofs_t;
+      typedef TTensor3<dofs,vdim,ne,vcomplex_t,true> val_dofs_t;
       val_dofs_t val_dofs;
 #else
-      typedef TTensor3<dofs,vdim,NE,complex_t> val_dofs_t;
+      typedef TTensor3<dofs,vdim,ne,vcomplex_t> val_dofs_t;
 #endif
-      TTensor4<qpts,dim,vdim,NE,complex_t>      grad_qpts;
+      TTensor4<qpts,dim,vdim,ne,vcomplex_t>      grad_qpts;
    };
 
-   template <int NE> struct AData<3,NE> // 3 = Values+Gradients
+   template <typename it_t> struct AData<3,it_t> // 3 = Values+Gradients
    {
+      static const int ne = it_t::batch_size;
+      typedef typename it_t::vcomplex_t vcomplex_t;
 #ifdef MFEM_TEMPLATE_FIELD_EVAL_DATA_HAS_DOFS
-      typedef TTensor3<dofs,vdim,NE,complex_t,true> val_dofs_t;
+      typedef TTensor3<dofs,vdim,ne,vcomplex_t,true> val_dofs_t;
       val_dofs_t val_dofs;
 #else
-      typedef TTensor3<dofs,vdim,NE,complex_t> val_dofs_t;
+      typedef TTensor3<dofs,vdim,ne,vcomplex_t> val_dofs_t;
 #endif
-      TTensor3<qpts,    vdim,NE,complex_t,true>  val_qpts;
-      TTensor4<qpts,dim,vdim,NE,complex_t>      grad_qpts;
+      TTensor3<qpts,    vdim,ne,vcomplex_t,true>  val_qpts;
+      TTensor4<qpts,dim,vdim,ne,vcomplex_t>      grad_qpts;
    };
 
    // This struct is similar to struct AData, adding separate static data
    // members for the input (InData) and output (OutData) data types.
-   template <int IData, int OData, int NE>
-   struct BData : public AData<IData|OData,NE>
+   template <int IData, int OData, typename it_t>
+   struct BData : public AData<IData|OData,it_t>
    {
       typedef T_type eval_type;
-      static const int ne = NE;
       static const int InData = IData;
       static const int OutData = OData;
    };
@@ -1238,7 +1254,9 @@ public:
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
       template <typename AData_t>
       static inline MFEM_ALWAYS_INLINE
-      void EvalSerialized(T_type &T, const complex_t *loc_dofs, AData_t &D)
+      void EvalSerialized(T_type &T,
+                          const typename AData_t::vcomplex_t *loc_dofs,
+                          AData_t &D)
       {
          T.shapeEval.Calc(AData_t::val_dofs_t::layout.merge_23(), loc_dofs,
                           D.val_qpts.layout.merge_23(), D.val_qpts);
@@ -1246,7 +1264,8 @@ public:
 
       template <bool Add, typename AData_t>
       static inline MFEM_ALWAYS_INLINE
-      void AssembleSerialized(T_type &T, const AData_t &D, complex_t *loc_dofs)
+      void AssembleSerialized(T_type &T, const AData_t &D,
+                              typename AData_t::vcomplex_t *loc_dofs)
       {
          T.shapeEval.template CalcT<Add>(
             D.val_qpts.layout.merge_23(), D.val_qpts,
@@ -1291,7 +1310,9 @@ public:
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
       template <typename AData_t>
       static inline MFEM_ALWAYS_INLINE
-      void EvalSerialized(T_type &T, const complex_t *loc_dofs, AData_t &D)
+      void EvalSerialized(T_type &T,
+                          const typename AData_t::vcomplex_t *loc_dofs,
+                          AData_t &D)
       {
          T.shapeEval.CalcGrad(AData_t::val_dofs_t::layout.merge_23(), loc_dofs,
                               D.grad_qpts.layout.merge_34(), D.grad_qpts);
@@ -1299,7 +1320,8 @@ public:
 
       template <bool Add, typename AData_t>
       static inline MFEM_ALWAYS_INLINE
-      void AssembleSerialized(T_type &T, const AData_t &D, complex_t *loc_dofs)
+      void AssembleSerialized(T_type &T, const AData_t &D,
+                              typename AData_t::vcomplex_t *loc_dofs)
       {
          T.shapeEval.template CalcGradT<Add>(
             D.grad_qpts.layout.merge_34(), D.grad_qpts,
@@ -1349,7 +1371,9 @@ public:
 #ifdef MFEM_TEMPLATE_ENABLE_SERIALIZE
       template <typename AData_t>
       static inline MFEM_ALWAYS_INLINE
-      void EvalSerialized(T_type &T, const complex_t *loc_dofs, AData_t &D)
+      void EvalSerialized(T_type &T,
+                          const typename AData_t::vcomplex_t *loc_dofs,
+                          AData_t &D)
       {
          T.shapeEval.Calc(AData_t::val_dofs_t::layout.merge_23(), loc_dofs,
                           D.val_qpts.layout.merge_23(), D.val_qpts);
@@ -1359,7 +1383,8 @@ public:
 
       template <bool Add, typename AData_t>
       static inline MFEM_ALWAYS_INLINE
-      void AssembleSerialized(T_type &T, const AData_t &D, complex_t *loc_dofs)
+      void AssembleSerialized(T_type &T, const AData_t &D,
+                              typename AData_t::vcomplex_t *loc_dofs)
       {
          T.shapeEval.template CalcT<Add>(
             D.val_qpts.layout.merge_23(), D.val_qpts,
@@ -1373,12 +1398,13 @@ public:
 
    // This struct implements element matrix computation for some combinations
    // of input (InOps) and output (OutOps) operations.
-   template <int InOps, int OutOps, int NE> struct TElementMatrix;
+   template <int InOps, int OutOps, typename it_t> struct TElementMatrix;
 
-   template <int NE> struct TElementMatrix<1,1,NE> // 1,1 = Values,Values
+   // Case 1,1 = Values,Values
+   template <typename it_t> struct TElementMatrix<1,1,it_t>
    {
       // qpt_layout_t is (nip), M_layout_t is (dof x dof)
-      // NE = 1 is assumed
+      // it_t::batch_size = 1 is assumed
       template <typename qpt_layout_t, typename qpt_data_t,
                 typename M_layout_t, typename M_data_t>
       static inline MFEM_ALWAYS_INLINE
@@ -1390,10 +1416,11 @@ public:
       }
    };
 
-   template <int NE> struct TElementMatrix<2,2,NE> // 2,2 = Gradients,Gradients
+   // Case 2,2 = Gradients,Gradients
+   template <typename it_t> struct TElementMatrix<2,2,it_t>
    {
       // qpt_layout_t is (nip x dim x dim), M_layout_t is (dof x dof)
-      // NE = 1 is assumed
+      // it_t::batch_size = 1 is assumed
       template <typename qpt_layout_t, typename qpt_data_t,
                 typename M_layout_t, typename M_data_t>
       static inline MFEM_ALWAYS_INLINE
@@ -1405,15 +1432,15 @@ public:
       }
    };
 
-   template <typename kernel_t, int NE> struct Spec
+   template <typename kernel_t, typename impl_traits_t> struct Spec
    {
       static const int InData =
          Values*kernel_t::in_values + Gradients*kernel_t::in_gradients;
       static const int OutData =
          Values*kernel_t::out_values + Gradients*kernel_t::out_gradients;
 
-      typedef BData<InData,OutData,NE>          DataType;
-      typedef TElementMatrix<InData,OutData,NE> ElementMatrix;
+      typedef BData<InData,OutData,impl_traits_t>          DataType;
+      typedef TElementMatrix<InData,OutData,impl_traits_t> ElementMatrix;
    };
 };
 

fem/tfespace.hpp

@@ -114,18 +114,23 @@ class TFiniteElementSpace_simple
 public:
    typedef FE        FE_type;
    typedef IndexType index_type;
+   static const int dofs = FE::dofs;
 
 protected:
    index_type ind;
+   int num_elems, remain_elems;
 
 public:
    TFiniteElementSpace_simple(const FE &fe, const FiniteElementSpace &fes)
-      : ind(fe, fes) { }
+      : ind(fe, fes), num_elems(fes.GetNE()), remain_elems(num_elems) { }
 
    // default copy constructor
 
-   void SetElement(int el) { ind.SetElement(el); }
+   int GetNE() const { return num_elems; }
 
+   void SetElement(int el) { ind.SetElement(el); remain_elems = num_elems-el; }
+
+#if 0
    // Multi-element Extract:
    // Extract dofs for multiple elements starting with the current element.
    // The number of elements to extract is given by the second dimension of
@@ -137,6 +142,7 @@ public:
                 const dof_layout_t    &dof_layout,
                 dof_data_t            &dof_data) const
    {
+      const int SS = sizeof(dof_data[0])/sizeof(dof_data[0][0]);
       const int NE = dof_layout_t::dim_2;
       MFEM_STATIC_ASSERT(FE::dofs == dof_layout_t::dim_1,
                          "invalid number of dofs");
@@ -144,8 +150,11 @@ public:
       {
          for (int i = 0; i < FE::dofs; i++)
          {
-            Assign<Op>(dof_data[dof_layout.ind(i,j)],
-                       glob_dof_data[ind.map(i,j)]);
+            for (int s = 0; s < SS; s++)
+            {
+               Assign<Op>(dof_data[dof_layout.ind(i,j)][s],
+                          glob_dof_data[ind.map(i,s+SS*j)]);
+            }
          }
       }
    }
@@ -169,6 +178,7 @@ public:
                  const dof_data_t   &dof_data,
                  glob_dof_data_t    &glob_dof_data) const
    {
+      const int SS = sizeof(dof_data[0])/sizeof(dof_data[0][0]);
       const int NE = dof_layout_t::dim_2;
       MFEM_STATIC_ASSERT(FE::dofs == dof_layout_t::dim_1,
                          "invalid number of dofs");
@@ -176,8 +186,11 @@ public:
       {
          for (int i = 0; i < FE::dofs; i++)
          {
-            Assign<Op>(glob_dof_data[ind.map(i,j)],
-                       dof_data[dof_layout.ind(i,j)]);
+            for (int s = 0; s < SS; s++)
+            {
+               Assign<Op>(glob_dof_data[ind.map(i,s+SS*j)],
+                          dof_data[dof_layout.ind(i,j)][s]);
+            }
          }
       }
    }
@@ -191,6 +204,7 @@ public:
    {
       Assemble<AssignOp::Add>(dof_layout, dof_data, glob_dof_data);
    }
+#endif
 
    // Multi-element VectorExtract: vdof_layout is (DOFS x NumComp x NumElems).
    template <AssignOp::Type Op,
@@ -202,21 +216,39 @@ public:
                       const vdof_layout_t    &vdof_layout,
                       vdof_data_t            &vdof_data) const
    {
+      const int SS = sizeof(vdof_data[0])/sizeof(vdof_data[0][0]);
       const int NC = vdof_layout_t::dim_2;
       const int NE = vdof_layout_t::dim_3;
       MFEM_STATIC_ASSERT(FE::dofs == vdof_layout_t::dim_1,
                          "invalid number of dofs");
       MFEM_ASSERT(NC == vl.NumComponents(), "invalid number of components");
+      const int TE = std::min(SS*NE, remain_elems);
+      // const int TE = SS*NE;
       for (int k = 0; k < NC; k++)
       {
+#if 0
          for (int j = 0; j < NE; j++)
          {
             for (int i = 0; i < FE::dofs; i++)
             {
-               Assign<Op>(vdof_data[vdof_layout.ind(i,k,j)],
-                          glob_vdof_data[vl.ind(ind.map(i,j), k)]);
+               for (int s = 0; s < SS; s++)
+               {
+                  Assign<Op>(vdof_data[vdof_layout.ind(i,k,j)][s],
+                             glob_vdof_data[vl.ind(ind.map(i,s+SS*j), k)]);
+               }
             }
          }
+#else
+         for (int js = 0; js < TE; js++)
+         {
+            for (int i = 0; i < FE::dofs; i++)
+            {
+               const int s = js % SS, j = js / SS;
+               Assign<Op>(vdof_data[vdof_layout.ind(i,k,j)][s],
+                          glob_vdof_data[vl.ind(ind.map(i,js), k)]);
+            }
+         }
+#endif
       }
    }
 
@@ -241,21 +273,39 @@ public:
                        const vec_layout_t  &vl,
                        glob_vdof_data_t    &glob_vdof_data) const
    {
+      const int SS = sizeof(vdof_data[0])/sizeof(vdof_data[0][0]);
       const int NC = vdof_layout_t::dim_2;
       const int NE = vdof_layout_t::dim_3;
       MFEM_STATIC_ASSERT(FE::dofs == vdof_layout_t::dim_1,
                          "invalid number of dofs");
       MFEM_ASSERT(NC == vl.NumComponents(), "invalid number of components");
+      const int TE = std::min(SS*NE, remain_elems);
+      // const int TE = SS*NE;
       for (int k = 0; k < NC; k++)
       {
+#if 0
          for (int j = 0; j < NE; j++)
          {
             for (int i = 0; i < FE::dofs; i++)
             {
-               Assign<Op>(glob_vdof_data[vl.ind(ind.map(i,j), k)],
-                          vdof_data[vdof_layout.ind(i,k,j)]);
+               for (int s = 0; s < SS; s++)
+               {
+                  Assign<Op>(glob_vdof_data[vl.ind(ind.map(i,s+SS*j), k)],
+                             vdof_data[vdof_layout.ind(i,k,j)][s]);
+               }
             }
          }
+#else
+         for (int js = 0; js < TE; js++)
+         {
+            for (int i = 0; i < FE::dofs; i++)
+            {
+               const int s = js % SS, j = js / SS;
+               Assign<Op>(glob_vdof_data[vl.ind(ind.map(i,js), k)],
+                          vdof_data[vdof_layout.ind(i,k,j)][s]);
+            }
+         }
+#endif
       }
    }
 
@@ -282,21 +332,24 @@ public:
                           const vdof_layout_t    &vdof_layout,
                           vdof_data_t            &vdof_data) const
    {
+      const int SS = sizeof(vdof_data[0])/sizeof(vdof_data[0][0]);
       const int NC = vdof_layout_t::dim_2;
       const int NE = vdof_layout_t::dim_3;
+      const int TE = std::min(SS*NE, remain_elems);
       MFEM_STATIC_ASSERT(FE::dofs == vdof_layout_t::dim_1,
                          "invalid number of dofs");
       MFEM_ASSERT(first_comp + NC <= vl.NumComponents(),
                   "invalid number of components");
       for (int k = 0; k < NC; k++)
       {
-         for (int j = 0; j < NE; j++)
+         for (int js = 0; js < TE; js++)
          {
             for (int i = 0; i < FE::dofs; i++)
             {
+               const int s = js % SS, j = js / SS;
                Assign<AssignOp::Set>(
-                  vdof_data[vdof_layout.ind(i,k,j)],
-                  glob_vdof_data[vl.ind(ind.map(i,j), first_comp+k)]);
+                  vdof_data[vdof_layout.ind(i,k,j)][s],
+                  glob_vdof_data[vl.ind(ind.map(i,js), first_comp+k)]);
             }
          }
       }
@@ -314,55 +367,69 @@ public:
                            const vec_layout_t  &vl,
                            glob_vdof_data_t    &glob_vdof_data) const
    {
+      const int SS = sizeof(vdof_data[0])/sizeof(vdof_data[0][0]);
       const int NC = vdof_layout_t::dim_2;
       const int NE = vdof_layout_t::dim_3;
+      const int TE = std::min(SS*NE, remain_elems);
       MFEM_STATIC_ASSERT(FE::dofs == vdof_layout_t::dim_1,
                          "invalid number of dofs");
       MFEM_ASSERT(first_comp + NC <= vl.NumComponents(),
                   "invalid number of components");
       for (int k = 0; k < NC; k++)
       {
-         for (int j = 0; j < NE; j++)
+         for (int js = 0; js < TE; js++)
          {
             for (int i = 0; i < FE::dofs; i++)
             {
+               const int s = js % SS, j = js / SS;
                Assign<AssignOp::Add>(
-                  glob_vdof_data[vl.ind(ind.map(i,j), first_comp+k)],
-                  vdof_data[vdof_layout.ind(i,k,j)]);
+                  glob_vdof_data[vl.ind(ind.map(i,js), first_comp+k)],
+                  vdof_data[vdof_layout.ind(i,k,j)][s]);
             }
          }
       }
    }
 
-   void Assemble(const TMatrix<FE::dofs,FE::dofs,double> &m,
+   template <typename vcomplex_t>
+   void Assemble(const TMatrix<FE::dofs,FE::dofs,vcomplex_t> &m,
                  SparseMatrix &M) const
    {
+      const int SS = sizeof(m[0])/sizeof(m[0][0]);
+      const int TE = std::min(SS, remain_elems);
       MFEM_FLOPS_ADD(FE::dofs*FE::dofs);
-      for (int i = 0; i < FE::dofs; i++)
+      for (int s = 0; s < TE; s++)
       {
-         M.SetColPtr(ind.map(i,0));
-         for (int j = 0; j < FE::dofs; j++)
+         for (int i = 0; i < FE::dofs; i++)
          {
-            M._Add_(ind.map(j,0), m(i,j));
+            M.SetColPtr(ind.map(i,s));
+            for (int j = 0; j < FE::dofs; j++)
+            {
+               M._Add_(ind.map(j,s), m(i,j)[s]);
+            }
+            M.ClearColPtr();
          }
-         M.ClearColPtr();
       }
    }
 
-   template <typename vec_layout_t>
+   template <typename vec_layout_t, typename vcomplex_t>
    void AssembleBlock(int block_i, int block_j, const vec_layout_t &vl,
-                      const TMatrix<FE::dofs,FE::dofs,double> &m,
+                      const TMatrix<FE::dofs,FE::dofs,vcomplex_t> &m,
                       SparseMatrix &M) const
    {
+      const int SS = sizeof(m[0])/sizeof(m[0][0]);
+      const int TE = std::min(SS, remain_elems);
       MFEM_FLOPS_ADD(FE::dofs*FE::dofs);
-      for (int i = 0; i < FE::dofs; i++)
+      for (int s = 0; s < TE; s++)
       {
-         M.SetColPtr(vl.ind(ind.map(i,0), block_i));
-         for (int j = 0; j < FE::dofs; j++)
+         for (int i = 0; i < FE::dofs; i++)
          {
-            M._Add_(vl.ind(ind.map(j,0), block_j), m(i,j));
+            M.SetColPtr(vl.ind(ind.map(i,s), block_i));
+            for (int j = 0; j < FE::dofs; j++)
+            {
+               M._Add_(vl.ind(ind.map(j,s), block_j), m(i,j)[s]);
+            }
+            M.ClearColPtr();
          }
-         M.ClearColPtr();
       }
    }
 };

fem/tmop.cpp

@@ -941,6 +941,7 @@ void AnalyticAdaptTC::ComputeElementTargets(int e_id, const FiniteElement &fe,
          IsoparametricTransformation Tpr;
          Tpr.SetFE(&fe);
          Tpr.ElementNo = e_id;
+         Tpr.ElementType = ElementTransformation::ELEMENT;
          Tpr.GetPointMat().Transpose(point_mat);
 
          for (int i = 0; i < ir.GetNPoints(); i++)
@@ -957,42 +958,176 @@ void AnalyticAdaptTC::ComputeElementTargets(int e_id, const FiniteElement &fe,
 }
 
 #ifdef MFEM_USE_MPI
-void DiscreteAdaptTC::SetParDiscreteTargetSpec(ParGridFunction &tspec_)
+void DiscreteAdaptTC::FinalizeParDiscreteTargetSpec(const ParGridFunction
+                                                    &tspec_)
 {
-   tspec.SetSize(tspec_.Size());
-   tspec = tspec_;
-   tspec_fes   = tspec_.FESpace();
+   MFEM_VERIFY(adapt_eval, "SetAdaptivityEvaluator() has not been called!")
+   MFEM_VERIFY(ncomp > 0, "No target specifications have been set!");
 
-   if (!adapt_eval) { MFEM_ABORT("Set adaptivity evaluator\n"); }
+   ParFiniteElementSpace *ptspec_fes = tspec_.ParFESpace();
 
-   adapt_eval->SetParMetaInfo(*tspec_.ParFESpace()->GetParMesh(),
-                              *tspec_.FESpace()->FEColl(),
-                              tspec_.FESpace()->GetVDim());
-
-   adapt_eval->SetInitialField
-   (*tspec_.FESpace()->GetMesh()->GetNodes(), tspec);
+   adapt_eval->SetParMetaInfo(*ptspec_fes->GetParMesh(),
+                              *ptspec_fes->FEColl(), ncomp);
+   adapt_eval->SetInitialField(*tspec_fes->GetMesh()->GetNodes(), tspec);
 
    tspec_sav = tspec;
+
+   delete tspec_fesv;
+   tspec_fesv = new FiniteElementSpace(tspec_fes->GetMesh(),
+                                       tspec_fes->FEColl(), ncomp);
+}
+
+void DiscreteAdaptTC::SetTspecAtIndex(int idx, const ParGridFunction &tspec_)
+{
+   const int vdim     = tspec_.FESpace()->GetVDim(),
+             dof_cnt  = tspec_.Size()/vdim;
+   for (int i = 0; i < dof_cnt*vdim; i++)
+   {
+      tspec(i+idx*dof_cnt) = tspec_(i);
+   }
+
+   FinalizeParDiscreteTargetSpec(tspec_);
+}
+
+void DiscreteAdaptTC::SetParDiscreteTargetSize(const ParGridFunction &tspec_)
+{
+   if (sizeidx > -1) { SetTspecAtIndex(sizeidx, tspec_); return; }
+   sizeidx = ncomp;
+   SetDiscreteTargetBase(tspec_);
+   FinalizeParDiscreteTargetSpec(tspec_);
+}
+
+void DiscreteAdaptTC::SetParDiscreteTargetSkew(const ParGridFunction &tspec_)
+{
+   if (skewidx > -1) { SetTspecAtIndex(skewidx, tspec_); return; }
+   skewidx = ncomp;
+   SetDiscreteTargetBase(tspec_);
+   FinalizeParDiscreteTargetSpec(tspec_);
+}
+
+void DiscreteAdaptTC::SetParDiscreteTargetAspectRatio(const ParGridFunction
+                                                      &tspec_)
+{
+   if (aspectratioidx > -1) { SetTspecAtIndex(aspectratioidx, tspec_); return; }
+   aspectratioidx = ncomp;
+   SetDiscreteTargetBase(tspec_);
+   FinalizeParDiscreteTargetSpec(tspec_);
+}
+
+void DiscreteAdaptTC::SetParDiscreteTargetOrientation(const ParGridFunction
+                                                      &tspec_)
+{
+   if (orientationidx > -1) { SetTspecAtIndex(orientationidx, tspec_); return; }
+   orientationidx = ncomp;
+   SetDiscreteTargetBase(tspec_);
+   FinalizeParDiscreteTargetSpec(tspec_);
+}
+
+void DiscreteAdaptTC::SetParDiscreteTargetSpec(const ParGridFunction &tspec_)
+{
+   SetParDiscreteTargetSize(tspec_);
+   FinalizeParDiscreteTargetSpec(tspec_);
 }
 #endif
 
-void DiscreteAdaptTC::SetSerialDiscreteTargetSpec(GridFunction &tspec_)
+void DiscreteAdaptTC::SetDiscreteTargetBase(const GridFunction &tspec_)
 {
-   tspec.SetSize(tspec_.Size());
-   tspec = tspec_;
-   tspec_fes   = tspec_.FESpace();
+   const int vdim     = tspec_.FESpace()->GetVDim(),
+             dof_cnt  = tspec_.Size()/vdim;
 
-   if (!adapt_eval) { MFEM_ABORT("Set adaptivity evaluator\n"); }
+   ncomp += vdim;
 
-   adapt_eval->SetSerialMetaInfo(*tspec_.FESpace()->GetMesh(),
-                                 *tspec_.FESpace()->FEColl(),
-                                 tspec_.FESpace()->GetVDim());
-   adapt_eval->SetInitialField
-   (*tspec_.FESpace()->GetMesh()->GetNodes(), tspec);
+   delete tspec_fes;
+   tspec_fes = new FiniteElementSpace(tspec_.FESpace()->GetMesh(),
+                                      tspec_.FESpace()->FEColl(), 1);
+
+   // need to append data to tspec
+   // make a copy of tspec->tspec_temp, increase its size, and
+   // copy data from tspec_temp -> tspec, then add new entries
+   Vector tspec_temp = tspec;
+   tspec.SetSize(ncomp*dof_cnt);
+
+   for (int i = 0; i < tspec_temp.Size(); i++)
+   {
+      tspec(i) = tspec_temp(i);
+   }
+
+   for (int i = 0; i < dof_cnt*vdim; i++)
+   {
+      tspec(i+(ncomp-vdim)*dof_cnt) = tspec_(i);
+   }
+}
+
+void DiscreteAdaptTC::SetTspecAtIndex(int idx, const GridFunction &tspec_)
+{
+   const int vdim     = tspec_.FESpace()->GetVDim(),
+             dof_cnt  = tspec_.Size()/vdim;
+   for (int i = 0; i < dof_cnt*vdim; i++)
+   {
+      tspec(i+idx*dof_cnt) = tspec_(i);
+   }
+
+   FinalizeSerialDiscreteTargetSpec();
+}
+
+void DiscreteAdaptTC::SetSerialDiscreteTargetSize(const GridFunction &tspec_)
+{
+
+   if (sizeidx > -1) { SetTspecAtIndex(sizeidx, tspec_); return; }
+   sizeidx = ncomp;
+   SetDiscreteTargetBase(tspec_);
+   FinalizeSerialDiscreteTargetSpec();
+}
+
+void DiscreteAdaptTC::SetSerialDiscreteTargetSkew(const GridFunction &tspec_)
+{
+   if (skewidx > -1) { SetTspecAtIndex(skewidx, tspec_); return; }
+   skewidx = ncomp;
+   SetDiscreteTargetBase(tspec_);
+   FinalizeSerialDiscreteTargetSpec();
+}
+
+void DiscreteAdaptTC::SetSerialDiscreteTargetAspectRatio(
+   const GridFunction &tspec_)
+{
+   if (aspectratioidx > -1) { SetTspecAtIndex(aspectratioidx, tspec_); return; }
+   aspectratioidx = ncomp;
+   SetDiscreteTargetBase(tspec_);
+   FinalizeSerialDiscreteTargetSpec();
+}
+
+void DiscreteAdaptTC::SetSerialDiscreteTargetOrientation(
+   const GridFunction &tspec_)
+{
+   if (orientationidx > -1) { SetTspecAtIndex(orientationidx, tspec_); return; }
+   orientationidx = ncomp;
+   SetDiscreteTargetBase(tspec_);
+   FinalizeSerialDiscreteTargetSpec();
+}
+
+void DiscreteAdaptTC::FinalizeSerialDiscreteTargetSpec()
+{
+   MFEM_VERIFY(adapt_eval, "SetAdaptivityEvaluator() has not been called!")
+   MFEM_VERIFY(ncomp > 0, "No target specifications have been set!");
+
+   adapt_eval->SetSerialMetaInfo(*tspec_fes->GetMesh(),
+                                 *tspec_fes->FEColl(), ncomp);
+   adapt_eval->SetInitialField(*tspec_fes->GetMesh()->GetNodes(), tspec);
 
    tspec_sav = tspec;
+
+   delete tspec_fesv;
+   tspec_fesv = new FiniteElementSpace(tspec_fes->GetMesh(),
+                                       tspec_fes->FEColl(), ncomp);
 }
 
+void DiscreteAdaptTC::SetSerialDiscreteTargetSpec(const GridFunction &tspec_)
+{
+   SetSerialDiscreteTargetSize(tspec_);
+   FinalizeSerialDiscreteTargetSpec();
+}
+
+
 void DiscreteAdaptTC::UpdateTargetSpecification(const Vector &new_x,
                                                 bool use_flag)
 {
@@ -1020,8 +1155,12 @@ void DiscreteAdaptTC::UpdateTargetSpecificationAtNode(const FiniteElement &el,
 
    Array<int> dofs;
    tspec_fes->GetElementDofs(T.ElementNo, dofs);
-   int cnt = tspec.Size();
-   tspec(dofs[dofidx]) = IntData(dofs[dofidx]+dir*cnt);
+   const int cnt = tspec.Size()/ncomp; //dofs per scalar-field
+
+   for (int i = 0; i < ncomp; i++)
+   {
+      tspec(dofs[dofidx]+i*cnt) = IntData(dofs[dofidx] + i*cnt + dir*cnt*ncomp);
+   }
 }
 
 void DiscreteAdaptTC::RestoreTargetSpecificationAtNode(ElementTransformation &T,
@@ -1031,7 +1170,11 @@ void DiscreteAdaptTC::RestoreTargetSpecificationAtNode(ElementTransformation &T,
 
    Array<int> dofs;
    tspec_fes->GetElementDofs(T.ElementNo, dofs);
-   tspec(dofs[dofidx]) = tspec_sav(dofs[dofidx]);
+   const int cnt = tspec.Size()/ncomp;
+   for (int i = 0; i < ncomp; i++)
+   {
+      tspec(dofs[dofidx] + i*cnt) = tspec_sav(dofs[dofidx] + i*cnt);
+   }
 }
 
 void DiscreteAdaptTC::ComputeElementTargets(int e_id, const FiniteElement &fe,
@@ -1039,37 +1182,176 @@ void DiscreteAdaptTC::ComputeElementTargets(int e_id, const FiniteElement &fe,
                                             const Vector &elfun,
                                             DenseTensor &Jtr) const
 {
-   MFEM_VERIFY(tspec_fes, "A call to SetDiscreteTargerSpec() is needed.");
+   MFEM_VERIFY(tspec_fesv, "No target specifications have been set.");
 
    switch (target_type)
    {
       case IDEAL_SHAPE_GIVEN_SIZE:
+      case GIVEN_SHAPE_AND_SIZE:
       {
          const DenseMatrix &Wideal =
             Geometries.GetGeomToPerfGeomJac(fe.GetGeomType());
          const int dim = Wideal.Height(),
-                   ntspec_dofs = tspec_fes->GetFE(0)->GetDof();
+                   ndofs = tspec_fes->GetFE(0)->GetDof(),
+                   ntspec_dofs = ndofs*ncomp;
+
+         Vector shape(ndofs), tspec_vals(ntspec_dofs), par_vals,
+                par_vals_c1(ndofs), par_vals_c2(ndofs), par_vals_c3(ndofs);
 
-         Vector shape(ntspec_dofs), tspec_vals(ntspec_dofs);
          Array<int> dofs;
-         tspec_fes->GetElementDofs(e_id, dofs);
+         DenseMatrix D_rho(dim), Q_phi(dim), R_theta(dim);
+         tspec_fesv->GetElementVDofs(e_id, dofs);
          tspec.GetSubVector(dofs, tspec_vals);
 
-         const double min_size = tspec_vals.Min();
-         MFEM_ASSERT(min_size > 0.0,
-                     "Non-positive size propagated in the target definition.");
-
          for (int i = 0; i < ir.GetNPoints(); i++)
          {
             const IntegrationPoint &ip = ir.IntPoint(i);
             tspec_fes->GetFE(e_id)->CalcShape(ip, shape);
-            const double size = std::max(shape * tspec_vals, min_size);
-            Jtr(i).Set(std::pow(size / Wideal.Det(), 1.0/dim), Wideal);
+            Jtr(i) = Wideal; //Initialize to identity
+
+            if (sizeidx != -1) //Set size
+            {
+               par_vals.SetDataAndSize(tspec_vals.GetData()+sizeidx*ndofs, ndofs);
+               const double min_size = par_vals.Min();
+               MFEM_VERIFY(min_size > 0.0,
+                           "Non-positive size propagated in the target definition.");
+               const double size = std::max(shape * par_vals, min_size);
+               Jtr(i).Set(std::pow(size, 1.0/dim), Jtr(i));
+            } //Done size
+
+            if (target_type == IDEAL_SHAPE_GIVEN_SIZE) { continue; }
+
+            if (aspectratioidx != -1) //Set aspect ratio
+            {
+               if (dim == 2)
+               {
+                  par_vals.SetDataAndSize(tspec_vals.GetData()+
+                                          aspectratioidx*ndofs, ndofs);
+
+                  const double aspectratio = shape * par_vals;
+                  D_rho = 0.;
+                  D_rho(0,0) = 1./pow(aspectratio,0.5);
+                  D_rho(1,1) = pow(aspectratio,0.5);
+               }
+               else
+               {
+                  par_vals.SetDataAndSize(tspec_vals.GetData()+
+                                          aspectratioidx*ndofs, ndofs*3);
+                  par_vals_c1.SetData(par_vals.GetData());
+                  par_vals_c2.SetData(par_vals.GetData()+ndofs);
+                  par_vals_c3.SetData(par_vals.GetData()+2*ndofs);
+
+                  const double rho1 = shape * par_vals_c1;
+                  const double rho2 = shape * par_vals_c2;
+                  const double rho3 = shape * par_vals_c3;
+                  D_rho = 0.;
+                  D_rho(0,0) = pow(rho1,2./3.);
+                  D_rho(1,1) = pow(rho2,2./3.);
+                  D_rho(2,2) = pow(rho3,2./3.);
+               }
+
+               DenseMatrix Temp = Jtr(i);
+               Mult(D_rho, Temp, Jtr(i));
+            } //Done aspect ratio
+
+            if (skewidx != -1) //Set skew
+            {
+               if (dim == 2)
+               {
+                  par_vals.SetDataAndSize(tspec_vals.GetData()+
+                                          skewidx*ndofs, ndofs);
+
+                  const double skew = shape * par_vals;
+
+                  Q_phi = 0.;
+                  Q_phi(0,0) = 1.;
+                  Q_phi(0,1) = cos(skew);
+                  Q_phi(1,1) = sin(skew);
+               }
+               else
+               {
+                  par_vals.SetDataAndSize(tspec_vals.GetData()+
+                                          skewidx*ndofs, ndofs*3);
+                  par_vals_c1.SetData(par_vals.GetData());
+                  par_vals_c2.SetData(par_vals.GetData()+ndofs);
+                  par_vals_c3.SetData(par_vals.GetData()+2*ndofs);
+
+                  const double phi12  = shape * par_vals_c1;
+                  const double phi13  = shape * par_vals_c2;
+                  const double chi = shape * par_vals_c3;
+
+                  Q_phi = 0.;
+                  Q_phi(0,0) = 1.;
+                  Q_phi(0,1) = cos(phi12);
+                  Q_phi(0,2) = cos(phi13);
+
+                  Q_phi(1,1) = sin(phi12);
+                  Q_phi(1,2) = sin(phi13)*cos(chi);
+
+                  Q_phi(2,2) = sin(phi13)*sin(chi);
+               }
+
+               DenseMatrix Temp = Jtr(i);
+               Mult(Q_phi, Temp, Jtr(i));
+            } // done skew
+
+            if (orientationidx != -1) //Set orientation
+            {
+               if (dim == 2)
+               {
+                  par_vals.SetDataAndSize(tspec_vals.GetData()+
+                                          orientationidx*ndofs, ndofs);
+
+                  const double theta = shape * par_vals;
+                  R_theta(0,0) =  cos(theta);
+                  R_theta(0,1) = -sin(theta);
+                  R_theta(1,0) =  sin(theta);
+                  R_theta(1,1) =  cos(theta);
+               }
+               else
+               {
+                  par_vals.SetDataAndSize(tspec_vals.GetData()+
+                                          orientationidx*ndofs, ndofs*3);
+                  par_vals_c1.SetData(par_vals.GetData());
+                  par_vals_c2.SetData(par_vals.GetData()+ndofs);
+                  par_vals_c3.SetData(par_vals.GetData()+2*ndofs);
+
+                  const double theta = shape * par_vals_c1;
+                  const double psi   = shape * par_vals_c2;
+                  const double beta  = shape * par_vals_c3;
+
+                  DenseMatrix R_tp(dim), R_beta(dim), R_theta(dim);
+                  double ct = cos(theta), st = sin(theta),
+                         cp = cos(psi),   sp = sin(psi);
+                  R_tp(0,0) = ct*sp;
+                  R_tp(1,0) = st*sp;
+                  R_tp(2,0) = cp;
+
+                  R_tp(0,1) = -(ct*st*sp*sp)/(1+cp);
+                  R_tp(1,1) = cp+(pow(ct,2.)*pow(sp,2.))/(1+cp);
+                  R_tp(2,1) = -st*sp;
+
+                  R_tp(0,2) = -cp-(pow(st,2.)*pow(sp,2.))/(1+cp);
+                  R_tp(1,2) = -R_tp(0,1);
+                  R_tp(2,2) =  ct*sp;
+
+                  R_beta = 0.;
+                  R_beta(0,0) = 1.;
+                  R_beta(1,1) =  cos(beta);
+                  R_beta(1,2) = -sin(beta);
+                  R_beta(2,1) =  sin(beta);
+                  R_beta(2,2) =  cos(beta);
+
+                  Mult(R_tp, R_beta, R_theta);
+               }
+               DenseMatrix Temp = Jtr(i);
+               Mult(R_theta, Temp, Jtr(i));
+            } // done orientation
          }
          break;
       }
       default:
-         MFEM_ABORT("Incompatible target type for analytic adaptation!");
+         MFEM_ABORT("Incompatible target type for discrete adaptation!");
    }
 }
 
@@ -1079,13 +1361,10 @@ void DiscreteAdaptTC::UpdateGradientTargetSpecification(const Vector &x,
 {
    if (use_flag && good_tspec_grad) { return; }
 
-   const int dim = tspec_fes->GetFE(0)->GetDim();
-   const int cnt = x.Size()/dim;
+   const int dim = tspec_fes->GetFE(0)->GetDim(),
+             cnt = x.Size()/dim;
 
-   if (tspec_pert1h.Size() != x.Size())
-   {
-      tspec_pert1h.SetSize(x.Size());
-   }
+   tspec_pert1h.SetSize(x.Size()*ncomp);
 
    Vector TSpecTemp;
    Vector xtemp = x;
@@ -1093,7 +1372,7 @@ void DiscreteAdaptTC::UpdateGradientTargetSpecification(const Vector &x,
    {
       for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) += dx; }
 
-      TSpecTemp.SetDataAndSize(tspec_pert1h.GetData() + j*cnt, cnt);
+      TSpecTemp.NewDataAndSize(tspec_pert1h.GetData() + j*cnt*ncomp, cnt*ncomp);
       UpdateTargetSpecification(xtemp, TSpecTemp);
 
       for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) -= dx; }
@@ -1105,16 +1384,15 @@ void DiscreteAdaptTC::UpdateGradientTargetSpecification(const Vector &x,
 void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
                                                        double dx, bool use_flag)
 {
+
    if (use_flag && good_tspec_hess) { return; }
 
-   const int dim = tspec_fes->GetFE(0)->GetDim();
-   const int cnt = x.Size()/dim;
+   const int dim    = tspec_fes->GetFE(0)->GetDim(),
+             cnt    = x.Size()/dim,
+             totmix = 1+2*(dim-2);
 
-   if (tspec_pert2h.Size() != x.Size())
-   {
-      tspec_pert2h.SetSize(x.Size());
-      tspec_pertmix.SetSize(cnt*(1+2*(dim-2)));
-   }
+   tspec_pert2h.SetSize(cnt*dim*ncomp);
+   tspec_pertmix.SetSize(cnt*totmix*ncomp);
 
    Vector TSpecTemp;
    Vector xtemp = x;
@@ -1124,14 +1402,14 @@ void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
    {
       for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) += 2*dx; }
 
-      TSpecTemp.SetDataAndSize(tspec_pert2h.GetData() + j*cnt, cnt);
+      TSpecTemp.NewDataAndSize(tspec_pert2h.GetData() + j*cnt*ncomp, cnt*ncomp);
       UpdateTargetSpecification(xtemp, TSpecTemp);
 
       for (int i = 0; i < cnt; i++) { xtemp(j*cnt+i) -= 2*dx; }
    }
 
    // T(x+h,y+h)
-   int idx = 0;
+   int j = 0;
    for (int k1 = 0; k1 < dim; k1++)
    {
       for (int k2 = 0; (k1 != k2) && (k2 < dim); k2++)
@@ -1142,7 +1420,7 @@ void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
             xtemp(k2*cnt+i) += dx;
          }
 
-         TSpecTemp.SetDataAndSize(tspec_pertmix.GetData() + idx*cnt, cnt);
+         TSpecTemp.NewDataAndSize(tspec_pertmix.GetData() + j*cnt*ncomp, cnt*ncomp);
          UpdateTargetSpecification(xtemp, TSpecTemp);
 
          for (int i = 0; i < cnt; i++)
@@ -1150,7 +1428,7 @@ void DiscreteAdaptTC::UpdateHessianTargetSpecification(const Vector &x,
             xtemp(k1*cnt+i) -= dx;
             xtemp(k2*cnt+i) -= dx;
          }
-         idx++;
+         j++;
       }
    }
 
@@ -1165,6 +1443,8 @@ void AdaptivityEvaluator::SetSerialMetaInfo(const Mesh &m,
    delete mesh;
    mesh = new Mesh(m, true);
    fes = new FiniteElementSpace(mesh, &fec, num_comp);
+   dim = fes->GetFE(0)->GetDim();
+   ncomp = num_comp;
 }
 
 #ifdef MFEM_USE_MPI
@@ -1176,6 +1456,8 @@ void AdaptivityEvaluator::SetParMetaInfo(const ParMesh &m,
    delete pmesh;
    pmesh = new ParMesh(m, true);
    pfes  = new ParFiniteElementSpace(pmesh, &fec, num_comp);
+   dim = pfes->GetFE(0)->GetDim();
+   ncomp = num_comp;
 }
 #endif
 
@@ -1267,6 +1549,7 @@ double TMOP_Integrator::GetElementEnergy(const FiniteElement &el,
       Tpr = new IsoparametricTransformation;
       Tpr->SetFE(&el);
       Tpr->ElementNo = T.ElementNo;
+      Tpr->ElementType = ElementTransformation::ELEMENT;
       Tpr->Attribute = T.Attribute;
       Tpr->GetPointMat().Transpose(PMatI); // PointMat = PMatI^T
    }
@@ -1391,6 +1674,7 @@ void TMOP_Integrator::AssembleElementVectorExact(const FiniteElement &el,
       Tpr = new IsoparametricTransformation;
       Tpr->SetFE(&el);
       Tpr->ElementNo = T.ElementNo;
+      Tpr->ElementType = ElementTransformation::ELEMENT;
       Tpr->Attribute = T.Attribute;
       Tpr->GetPointMat().Transpose(PMatI); // PointMat = PMatI^T
    }
@@ -1484,6 +1768,7 @@ void TMOP_Integrator::AssembleElementGradExact(const FiniteElement &el,
       Tpr = new IsoparametricTransformation;
       Tpr->SetFE(&el);
       Tpr->ElementNo = T.ElementNo;
+      Tpr->ElementType = ElementTransformation::ELEMENT;
       Tpr->Attribute = T.Attribute;
       Tpr->GetPointMat().Transpose(PMatI);
    }

fem/tmop.hpp

@@ -560,6 +560,8 @@ protected:
    ParFiniteElementSpace *pfes;
 #endif
 
+   int dim, ncomp;
+
 public:
    AdaptivityEvaluator() : mesh(NULL), fes(NULL)
    {
@@ -708,15 +710,20 @@ class DiscreteAdaptTC : public TargetConstructor
 protected:
    // Discrete target specification.
    // Data is owned, updated by UpdateTargetSpecification.
+   int ncomp, sizeidx, skewidx, aspectratioidx, orientationidx;
    Vector tspec;             //eta(x)
    Vector tspec_sav;
    Vector tspec_pert1h;      //eta(x+h)
    Vector tspec_pert2h;      //eta(x+2*h)
    Vector tspec_pertmix;     //eta(x+h,y+h)
+   // The order inside these perturbation vectors (e.g. in 2D) is
+   // eta1(x+h,y), eta2(x+h,y) ... etan(x+h,y), eta1(x,y+h), eta2(x,y+h) ...
+   // same for tspec_pert2h and tspec_pertmix.
 
    // Note: do not use the Nodes of this space as they may not be on the
    // positions corresponding to the values of tspec.
    const FiniteElementSpace *tspec_fes;
+   const FiniteElementSpace *tspec_fesv;
 
    // These flags can be used by outside functions to avoid recomputing
    // the tspec and tspec_perth fields again on the same mesh.
@@ -726,20 +733,55 @@ protected:
    // Owned.
    AdaptivityEvaluator *adapt_eval;
 
+   void SetDiscreteTargetBase(const GridFunction &tspec_);
+   void SetTspecAtIndex(int idx, const GridFunction &tspec_);
+   void FinalizeSerialDiscreteTargetSpec();
+#ifdef MFEM_USE_MPI
+   void SetTspecAtIndex(int idx, const ParGridFunction &tspec_);
+   void FinalizeParDiscreteTargetSpec(const ParGridFunction &tspec_);
+#endif
+
 public:
    DiscreteAdaptTC(TargetType ttype)
       : TargetConstructor(ttype),
+        ncomp(0),
+        sizeidx(-1), skewidx(-1), aspectratioidx(-1), orientationidx(-1),
         tspec(), tspec_sav(), tspec_pert1h(), tspec_pert2h(), tspec_pertmix(),
-        tspec_fes(NULL),
+        tspec_fes(NULL), tspec_fesv(NULL),
         good_tspec(false), good_tspec_grad(false), good_tspec_hess(false),
         adapt_eval(NULL) { }
 
-   virtual ~DiscreteAdaptTC() { delete adapt_eval; }
+   virtual ~DiscreteAdaptTC()
+   {
+      delete adapt_eval;
+      delete tspec_fes;
+      delete tspec_fesv;
+   }
 
-   virtual void SetSerialDiscreteTargetSpec(GridFunction &tspec_);
+   /** @name Target specification methods.
+       The following methods are used to specify geometric parameters of the
+       targets when these parameters are given by discrete FE functions.
+       Note that every GridFunction given to the Set methods must use a
+       H1_FECollection of the same order. The number of components must
+       correspond to the type of geometric parameter and dimension.
+
+       @param[in] tspec_  Input values of a geometric parameter. Note that
+                          the methods in this class support only functions that
+                          use H1_FECollection collection of the same order. */
+   ///@{
+   virtual void SetSerialDiscreteTargetSpec(const GridFunction &tspec_);
+   virtual void SetSerialDiscreteTargetSize(const GridFunction &tspec_);
+   virtual void SetSerialDiscreteTargetSkew(const GridFunction &tspec_);
+   virtual void SetSerialDiscreteTargetAspectRatio(const GridFunction &tspec_);
+   virtual void SetSerialDiscreteTargetOrientation(const GridFunction &tspec_);
 #ifdef MFEM_USE_MPI
-   virtual void SetParDiscreteTargetSpec(ParGridFunction &tspec_);
+   virtual void SetParDiscreteTargetSpec(const ParGridFunction &tspec_);
+   virtual void SetParDiscreteTargetSize(const ParGridFunction &tspec_);
+   virtual void SetParDiscreteTargetSkew(const ParGridFunction &tspec_);
+   virtual void SetParDiscreteTargetAspectRatio(const ParGridFunction &tspec_);
+   virtual void SetParDiscreteTargetOrientation(const ParGridFunction &tspec_);
 #endif
+   ///@}
 
    /// Used in combination with the Update methods to avoid extra computations.
    void ResetUpdateFlags()
@@ -756,6 +798,7 @@ public:
                                         ElementTransformation &T,
                                         int nodenum, int idir,
                                         const Vector &IntData);
+
    void RestoreTargetSpecificationAtNode(ElementTransformation &T, int nodenum);
 
    /** Used for finite-difference based computations. Computes the target

fem/tmop_tools.cpp

@@ -29,13 +29,26 @@ void AdvectorCG::SetInitialField(const Vector &init_nodes,
 void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
                                       Vector &new_field)
 {
-#if defined(MFEM_DEBUG) || defined(MFEM_USE_MPI)
-   int myid = 0;
-#endif
-   Mesh *m = mesh;
+   // TODO: Implement for AMR meshes.
+   const int pnt_cnt = new_field.Size()/ncomp;
 
+   new_field = field0;
+
+   for (int i = 0; i < ncomp; i++)
+   {
+      Vector new_field_temp(new_field.GetData()+i*pnt_cnt, pnt_cnt);
+      ComputeAtNewPositionScalar(new_nodes, new_field_temp);
+   }
+
+   field0 = new_field;
+   nodes0 = new_nodes;
+}
+
+void AdvectorCG::ComputeAtNewPositionScalar(const Vector &new_nodes,
+                                            Vector &new_field)
+{
+   Mesh *m = mesh;
 #ifdef MFEM_USE_MPI
-   if (pfes) { MPI_Comm_rank(pfes->GetComm(), &myid); }
    if (pmesh) { m = pmesh; }
 #endif
 
@@ -44,17 +57,29 @@ void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
    // This will be used to move the positions.
    GridFunction *mesh_nodes = m->GetNodes();
    *mesh_nodes = nodes0;
-   new_field = field0;
+   double minv = new_field.Min(), maxv = new_field.Max();
 
    // Velocity of the positions.
    GridFunction u(mesh_nodes->FESpace());
    subtract(new_nodes, nodes0, u);
 
+   // Define a scalar FE space for the solution, and the advection operator.
    TimeDependentOperator *oper = NULL;
-   // This must be the fes of the ind, associated with the object's mesh.
-   if (fes)  { oper = new SerialAdvectorCGOper(nodes0, u, *fes); }
+   FiniteElementSpace *fess = NULL;
 #ifdef MFEM_USE_MPI
-   else if (pfes) { oper = new ParAdvectorCGOper(nodes0, u, *pfes); }
+   ParFiniteElementSpace *pfess = NULL;
+#endif
+   if (fes)
+   {
+      fess = new FiniteElementSpace(fes->GetMesh(), fes->FEColl(), 1);
+      oper = new SerialAdvectorCGOper(nodes0, u, *fess);
+   }
+#ifdef MFEM_USE_MPI
+   else if (pfes)
+   {
+      pfess = new ParFiniteElementSpace(pfes->GetParMesh(), pfes->FEColl(), 1);
+      oper  = new ParAdvectorCGOper(nodes0, u, *pfess);
+   }
 #endif
    MFEM_VERIFY(oper != NULL,
                "No FE space has been given to the AdaptivityEvaluator.");
@@ -67,12 +92,18 @@ void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
       h_min = std::min(h_min, m->GetElementSize(i));
    }
    double v_max = 0.0;
-   const int s = u.FESpace()->GetVSize() / 2;
+   const int s = new_field.Size();
+
    for (int i = 0; i < s; i++)
    {
-      const double vel = u(i) * u(i) + u(i+s) * u(i+s);
+      double vel = 0.;
+      for (int j = 0; j < dim; j++)
+      {
+         vel += u(i+j*s)*u(i+j*s);
+      }
       v_max = std::max(v_max, vel);
    }
+
 #ifdef MFEM_USE_MPI
    if (pfes)
    {
@@ -81,12 +112,17 @@ void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
       MPI_Allreduce(&h_loc, &h_min, 1, MPI_DOUBLE, MPI_MIN, pfes->GetComm());
    }
 #endif
-   if (v_max == 0.0)
+
+   if (v_max == 0.0) // No need to change the field.
    {
-      // No mesh motion --> no need to change the field.
       delete oper;
+      delete fess;
+#ifdef MFEM_USE_MPI
+      delete pfess;
+#endif
       return;
    }
+
    v_max = std::sqrt(v_max);
    double dt = dt_scale * h_min / v_max;
 
@@ -96,30 +132,34 @@ void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
    {
       if (t + dt >= 1.0)
       {
-#ifdef MFEM_DEBUG
-         if (myid == 0)
-         {
-            mfem::out << "Remap took " << ti << " steps." << std::endl;
-         }
-#endif
          dt = 1.0 - t;
          last_step = true;
       }
       ode_solver.Step(new_field, t, dt);
    }
 
-   // Trim the overshoots and undershoots.
-   const double minv = field0.Min(), maxv = field0.Max();
-   for (int i = 0; i < new_field.Size(); i++)
+   double glob_minv = minv,
+          glob_maxv = maxv;
+#ifdef MFEM_USE_MPI
+   if (pfes)
    {
-      if (new_field(i) < minv) { new_field(i) = minv; }
-      if (new_field(i) > maxv) { new_field(i) = maxv; }
+      MPI_Allreduce(&minv, &glob_minv, 1, MPI_DOUBLE, MPI_MIN, pfes->GetComm());
+      MPI_Allreduce(&maxv, &glob_maxv, 1, MPI_DOUBLE, MPI_MAX, pfes->GetComm());
+   }
+#endif
+
+   // Trim the overshoots and undershoots.
+   for (int i = 0; i < s; i++)
+   {
+      if (new_field(i) < glob_minv) { new_field(i) = glob_minv; }
+      if (new_field(i) > glob_maxv) { new_field(i) = glob_maxv; }
    }
 
-   nodes0 = new_nodes;
-   field0 = new_field;
-
    delete oper;
+   delete fess;
+#ifdef MFEM_USE_MPI
+   delete pfess;
+#endif
 }
 
 SerialAdvectorCGOper::SerialAdvectorCGOper(const Vector &x_start,
@@ -235,6 +275,12 @@ void InterpolatorFP::SetInitialField(const Vector &init_nodes,
    const double newton_tol  = 1.0e-12;
    const int npts_at_once   = 256;
 
+   if (finder)
+   {
+      finder->FreeData();
+      delete finder;
+   }
+
    FiniteElementSpace *f = fes;
 #ifdef MFEM_USE_MPI
    if (pfes)

fem/tmop_tools.hpp

@@ -27,9 +27,9 @@ private:
    RK4Solver ode_solver;
    Vector nodes0;
    Vector field0;
-
    const double dt_scale;
 
+   void ComputeAtNewPositionScalar(const Vector &new_nodes, Vector &new_field);
 public:
    AdvectorCG(double timestep_scale = 0.5)
       : AdaptivityEvaluator(),
@@ -53,6 +53,8 @@ private:
    Vector pos_r_out, dist_p_out;
    int dim;
 public:
+   InterpolatorFP() : finder(NULL) { }
+
    virtual void SetInitialField(const Vector &init_nodes,
                                 const Vector &init_field);
 

fem/transfer.cpp

@@ -520,7 +520,9 @@ void TensorProductPRefinementTransferOperator::MultTranspose(const Vector& x,
 TrueTransferOperator::TrueTransferOperator(const
                                            ParFiniteElementSpace& lFESpace_,
                                            const ParFiniteElementSpace& hFESpace_)
-   : lFESpace(lFESpace_), hFESpace(hFESpace_)
+   : Operator(hFESpace_.GetTrueVSize(), lFESpace_.GetTrueVSize()),
+     lFESpace(lFESpace_),
+     hFESpace(hFESpace_)
 {
    localTransferOperator = new TransferOperator(lFESpace_, hFESpace_);
 

general/adios2stream.cpp

@@ -15,6 +15,8 @@
 
 #include "adios2stream.hpp"
 
+#ifdef MFEM_USE_ADIOS2
+
 #include "../fem/geom.hpp"
 #include "../general/array.hpp"
 #include "../mesh/element.hpp"
@@ -750,3 +752,5 @@ noexcept
 }
 
 }  // end namespace mfem
+
+#endif // MFEM_USE_ADIOS2

general/adios2stream.hpp

@@ -18,6 +18,8 @@
 
 #include "../config/config.hpp"
 
+#ifdef MFEM_USE_ADIOS2
+
 #include <map>
 #include <memory>  // std::unique_ptr
 #include <string>
@@ -230,4 +232,6 @@ private:
 
 }  // end namespace mfem
 
+#endif // MFEM_USE_ADIOS2
+
 #endif /* MFEM_ADIOS2STREAM */

general/mem_manager.cpp

@@ -26,8 +26,10 @@
 #include <signal.h>
 #include <sys/mman.h>
 #define mfem_memalign(p,a,s) posix_memalign(p,a,s)
+#define mfem_aligned_free free
 #else
 #define mfem_memalign(p,a,s) (((*(p))=_aligned_malloc((s),(a))),*(p)?0:errno)
+#define mfem_aligned_free _aligned_free
 #endif
 
 #ifdef MFEM_USE_UMPIRE
@@ -212,7 +214,7 @@ public:
    Aligned32HostMemorySpace(): HostMemorySpace() { }
    void Alloc(void **ptr, size_t bytes)
    { if (mfem_memalign(ptr, 32, bytes) != 0) { throw ::std::bad_alloc(); } }
-   void Dealloc(void *ptr) { std::free(ptr); }
+   void Dealloc(void *ptr) { mfem_aligned_free(ptr); }
 };
 
 /// The aligned 64 host memory space
@@ -222,6 +224,7 @@ public:
    Aligned64HostMemorySpace(): HostMemorySpace() { }
    void Alloc(void **ptr, size_t bytes)
    { if (mfem_memalign(ptr, 64, bytes) != 0) { throw ::std::bad_alloc(); } }
+   void Dealloc(void *ptr) { mfem_aligned_free(ptr); }
 };
 
 #ifndef _WIN32
@@ -666,12 +669,11 @@ void *MemoryManager::Register_(void *ptr, void *h_tmp, size_t bytes,
 {
    MFEM_CONTRACT_VAR(alias);
    MFEM_ASSERT(exists, "Internal error!");
-   MFEM_ASSERT(IsHostMemory(mt), "Internal error!");
    MFEM_ASSERT(!alias, "Cannot register an alias!");
    const bool is_host_mem = IsHostMemory(mt);
    const MemType dual_mt = GetDualMemoryType_(mt);
-   const MemType h_mt = mt;
-   const MemType d_mt = dual_mt;
+   const MemType h_mt = is_host_mem ? mt : dual_mt;
+   const MemType d_mt = is_host_mem ? dual_mt : mt;
    MFEM_VERIFY_TYPES(h_mt, d_mt);
 
    if (ptr == nullptr && h_tmp == nullptr)
@@ -693,10 +695,11 @@ void *MemoryManager::Register_(void *ptr, void *h_tmp, size_t bytes,
    else // DEVICE TYPES
    {
       h_ptr = h_tmp;
-      if (h_tmp == nullptr) { ctrl->Host(h_mt)->Alloc(&h_ptr, bytes); }
+      if (own && h_tmp == nullptr) { ctrl->Host(h_mt)->Alloc(&h_ptr, bytes); }
       mm.InsertDevice(ptr, h_ptr, bytes, h_mt, d_mt);
-      flags = (own ? flags | Mem::OWNS_DEVICE : flags & ~Mem::OWNS_DEVICE) |
-              Mem::OWNS_HOST | Mem::VALID_DEVICE;
+      flags = own ? flags | Mem::OWNS_DEVICE : flags & ~Mem::OWNS_DEVICE;
+      flags = own ? flags | Mem::OWNS_HOST   : flags & ~Mem::OWNS_HOST;
+      flags |= Mem::VALID_DEVICE;
    }
    CheckHostMemoryType_(h_mt, h_ptr);
    return h_ptr;

general/mem_manager.hpp

@@ -16,6 +16,7 @@
 #include "error.hpp"
 #include <cstring> // std::memcpy
 #include <type_traits> // std::is_const
+#include <cstddef> // std::max_align_t
 
 namespace mfem
 {
@@ -283,6 +284,28 @@ public:
        @note The current memory is NOT deleted by this method. */
    inline void Wrap(T *ptr, int size, MemoryType mt, bool own);
 
+   /** Wrap an externally pair of allocated pointers, @a h_ptr and @ d_ptr,
+       of the given host MemoryType @a h_mt. */
+   /** The new memory object will have the device MemoryType set as valid.
+
+       The given @a h_ptr and @a d_ptr must be allocated appropriately for the
+       given host MemoryType and its associated device MemoryType:
+          - MANAGED => MANAGED,
+          - HOST_DEBUG => DEVICE_DEBUG,
+          - HOST_UMPIRE => DEVICE_UMPIRE,
+          - HOST, HOST_32, HOST_64 => DEVICE.
+
+       The parameter @a own determines whether both @a h_ptr and @a d_ptr will
+       be deleted when the method Delete() is called.
+
+       @note Ownership can also be controled by using the folowing methods:
+         - ClearOwnerFlags,
+         - SetHostPtrOwner,
+         - SetDevicePtrOwner.
+
+       @note The current memory is NOT deleted by this method. */
+   inline void Wrap(T *h_ptr, T *d_ptr, int size, MemoryType h_mt, bool own);
+
    /// Create a memory object that points inside the memory object @a base.
    /** The new Memory object uses the same MemoryType(s) as @a base.
 
@@ -413,6 +436,38 @@ public:
    /** This method can be useful for debugging. It is explicitly instantiated
        for Memory<T> with T = int and T = double. */
    inline int CompareHostAndDevice(int size) const;
+
+private:
+   // GCC 4.8 workaround: max_align_t is not in std.
+   static constexpr std::size_t def_align_bytes_()
+   {
+      using namespace std;
+      return alignof(max_align_t);
+   }
+   static constexpr std::size_t def_align_bytes = def_align_bytes_();
+   static constexpr std::size_t new_align_bytes =
+      alignof(T) > def_align_bytes ? alignof(T) : def_align_bytes;
+
+   template <std::size_t align_bytes, bool dummy = true> struct Alloc
+   {
+      static inline T *New(std::size_t)
+      {
+#if __cplusplus < 201703L
+         // Generate an error in debug mode
+         MFEM_ASSERT(false, "overaligned type cannot use MemoryType::HOST");
+         return nullptr;
+#else
+         return new T[size];
+#endif
+      }
+   };
+
+#if __cplusplus < 201703L
+   template<bool dummy> struct Alloc<def_align_bytes,dummy>
+   {
+      static inline T *New(std::size_t size) { return new T[size]; }
+   };
+#endif
 };
 
 
@@ -625,7 +680,7 @@ inline void Memory<T>::New(int size)
    capacity = size;
    flags = OWNS_HOST | VALID_HOST;
    h_mt = MemoryManager::host_mem_type;
-   h_ptr = (h_mt == MemoryType::HOST) ? new T[size] :
+   h_ptr = (h_mt == MemoryType::HOST) ? Alloc<new_align_bytes>::New(size) :
            (T*)MemoryManager::New_(nullptr, size*sizeof(T), h_mt, flags);
 }
 
@@ -637,8 +692,9 @@ inline void Memory<T>::New(int size, MemoryType mt)
    const bool mt_host = mt == MemoryType::HOST;
    if (mt_host) { flags = OWNS_HOST | VALID_HOST; }
    h_mt = IsHostMemory(mt) ? mt : MemoryManager::GetDualMemoryType_(mt);
-   T *h_tmp = (h_mt == MemoryType::HOST) ? new T[size] : nullptr;
-   h_ptr = (mt_host) ? h_tmp: (T*)MemoryManager::New_(h_tmp, bytes, mt, flags);
+   T *h_tmp = (h_mt == MemoryType::HOST) ?
+              Alloc<new_align_bytes>::New(size) : nullptr;
+   h_ptr = (mt_host) ? h_tmp : (T*)MemoryManager::New_(h_tmp, bytes, mt, flags);
 }
 
 template <typename T>
@@ -682,6 +738,19 @@ inline void Memory<T>::Wrap(T *ptr, int size, MemoryType mt, bool own)
                                         own, false, flags);
 }
 
+template <typename T>
+inline void Memory<T>::Wrap(T *ptr, T *d_ptr, int size, MemoryType mt, bool own)
+{
+   h_mt = mt;
+   flags = 0;
+   h_ptr = ptr;
+   capacity = size;
+   MFEM_ASSERT(IsHostMemory(h_mt),"");
+   const size_t bytes = size*sizeof(T);
+   const MemoryType d_mt = MemoryManager::GetDualMemoryType_(h_mt);
+   MemoryManager::Register_(d_ptr, h_ptr, bytes, d_mt, own, false, flags);
+}
+
 template <typename T>
 inline void Memory<T>::MakeAlias(const Memory &base, int offset, int size)
 {

general/tassign.hpp

@@ -43,70 +43,110 @@ template <>
 struct AssignOp_Impl<AssignOp::Set>
 {
    template <typename lvalue_t, typename rvalue_t>
-   MFEM_HOST_DEVICE
    static inline lvalue_t &Assign(lvalue_t &a, const rvalue_t &b)
    {
       return (a = b);
    }
+
+   template <typename lvalue_t, typename rvalue_t>
+   MFEM_HOST_DEVICE
+   static inline lvalue_t &AssignHD(lvalue_t &a, const rvalue_t &b)
+   {
+      return (a = b);
+   }
 };
 
 template <>
 struct AssignOp_Impl<AssignOp::Add>
 {
    template <typename lvalue_t, typename rvalue_t>
-   MFEM_HOST_DEVICE
    static inline lvalue_t &Assign(lvalue_t &a, const rvalue_t &b)
    {
       MFEM_FLOPS_ADD(1);
       return (a += b);
    }
+
+   template <typename lvalue_t, typename rvalue_t>
+   MFEM_HOST_DEVICE
+   static inline lvalue_t &AssignHD(lvalue_t &a, const rvalue_t &b)
+   {
+      MFEM_FLOPS_ADD(1);
+      return (a += b);
+   }
 };
 
 template <>
 struct AssignOp_Impl<AssignOp::Mult>
 {
    template <typename lvalue_t, typename rvalue_t>
-   MFEM_HOST_DEVICE
    static inline lvalue_t &Assign(lvalue_t &a, const rvalue_t &b)
    {
       MFEM_FLOPS_ADD(1);
       return (a *= b);
    }
+
+   template <typename lvalue_t, typename rvalue_t>
+   MFEM_HOST_DEVICE
+   static inline lvalue_t &AssignHD(lvalue_t &a, const rvalue_t &b)
+   {
+      MFEM_FLOPS_ADD(1);
+      return (a *= b);
+   }
 };
 
 template <>
 struct AssignOp_Impl<AssignOp::Div>
 {
    template <typename lvalue_t, typename rvalue_t>
-   MFEM_HOST_DEVICE
    static inline lvalue_t &Assign(lvalue_t &a, const rvalue_t &b)
    {
       MFEM_FLOPS_ADD(1);
       return (a /= b);
    }
+
+   template <typename lvalue_t, typename rvalue_t>
+   MFEM_HOST_DEVICE
+   static inline lvalue_t &AssignHD(lvalue_t &a, const rvalue_t &b)
+   {
+      MFEM_FLOPS_ADD(1);
+      return (a /= b);
+   }
 };
 
 template <>
 struct AssignOp_Impl<AssignOp::rDiv>
 {
    template <typename lvalue_t, typename rvalue_t>
-   MFEM_HOST_DEVICE
    static inline lvalue_t &Assign(lvalue_t &a, const rvalue_t &b)
    {
       MFEM_FLOPS_ADD(1);
       return (a = b/a);
    }
+
+   template <typename lvalue_t, typename rvalue_t>
+   MFEM_HOST_DEVICE
+   static inline lvalue_t &AssignHD(lvalue_t &a, const rvalue_t &b)
+   {
+      MFEM_FLOPS_ADD(1);
+      return (a = b/a);
+   }
 };
 
 } // namespace mfem::internal
 
 template <AssignOp::Type Op, typename lvalue_t, typename rvalue_t>
-MFEM_HOST_DEVICE
 inline lvalue_t &Assign(lvalue_t &a, const rvalue_t &b)
 {
    return internal::AssignOp_Impl<Op>::Assign(a, b);
 }
 
+template <AssignOp::Type Op, typename lvalue_t, typename rvalue_t>
+MFEM_HOST_DEVICE
+inline lvalue_t &AssignHD(lvalue_t &a, const rvalue_t &b)
+{
+   return internal::AssignOp_Impl<Op>::AssignHD(a, b);
+}
+
 } // namespace mfem
 
 #endif // MFEM_TEMPLATE_ASSIGN

general/version.cpp

@@ -148,6 +148,9 @@ const char *GetConfigStr()
 #ifdef MFEM_USE_OCCA
       "MFEM_USE_OCCA\n"
 #endif
+#ifdef MFEM_USE_SIMD
+      "MFEM_USE_SIMD\n"
+#endif
 #ifdef MFEM_USE_ADIOS2
       "MFEM_USE_ADIOS2\n"
 #endif

general/zstr.hpp

@@ -175,7 +175,7 @@ struct static_method_holder
          is_p->peek();
          peek_failed = is_p->fail();
       }
-      catch (std::ios_base::failure &e) {}
+      catch (std::ios_base::failure&) {}
       if (peek_failed)
       {
          throw Exception(std::string("strict_fstream: open('")

linalg/densemat.cpp

@@ -3043,9 +3043,6 @@ void LUFactors::RightSolve(int m, int n, double *X) const
    }
 #else
    // compiling without LAPACK
-   const double *data = this->data;
-   const int *ipiv = this->ipiv;
-
    // X <- X U^{-1}
    x = X;
    for (int k = 0; k < n; k++)
@@ -3080,7 +3077,7 @@ void LUFactors::RightSolve(int m, int n, double *X) const
    x = X;
    for (int k = 0; k < n; k++)
    {
-      for (int i = 0; i < m; i++)
+      for (int i = m-1; i >= 0; --i)
       {
          Swap<double>(x[i*n], x[(ipiv[i]-ipiv_base)*n]);
       }

linalg/hypre.cpp

@@ -21,55 +21,6 @@
 #include <cmath>
 #include <cstdlib>
 
-// Define macro wrappers for hypre_TAlloc, hypre_CTAlloc and hypre_TFree:
-// mfem_hypre_TAlloc, mfem_hypre_CTAlloc, and mfem_hypre_TFree, respectively.
-// Note: the same macros are defined in hypre_parcsr.cpp.
-#if MFEM_HYPRE_VERSION < 21400
-
-#define mfem_hypre_TAlloc(type, size) hypre_TAlloc(type, size)
-#define mfem_hypre_CTAlloc(type, size) hypre_CTAlloc(type, size)
-#define mfem_hypre_TFree(ptr) hypre_TFree(ptr)
-
-#else // MFEM_HYPRE_VERSION >= 21400
-
-#define mfem_hypre_TAlloc(type, size) \
-   hypre_TAlloc(type, size, HYPRE_MEMORY_HOST)
-#define mfem_hypre_CTAlloc(type, size) \
-   hypre_CTAlloc(type, size, HYPRE_MEMORY_HOST)
-#define mfem_hypre_TFree(ptr) hypre_TFree(ptr, HYPRE_MEMORY_HOST)
-
-// Notes regarding allocation and deallocation of hypre objects in 2.14.0
-//-----------------------------------------------------------------------
-//
-// 1. hypre_CSRMatrix: i, j, data, and rownnz use HYPRE_MEMORY_SHARED while the
-//    hypre_CSRMatrix structure uses HYPRE_MEMORY_HOST.
-//
-//    Note: the function HYPRE_CSRMatrixCreate creates the i array using
-//          HYPRE_MEMORY_HOST!
-//    Note: the functions hypre_CSRMatrixAdd and hypre_CSRMatrixMultiply create
-//          C_i using HYPRE_MEMORY_HOST!
-//
-// 2. hypre_Vector: data uses HYPRE_MEMORY_SHARED while the hypre_Vector
-//    structure uses HYPRE_MEMORY_HOST.
-//
-// 3. hypre_ParVector: the structure hypre_ParVector uses HYPRE_MEMORY_HOST;
-//    partitioning uses HYPRE_MEMORY_HOST.
-//
-// 4. hypre_ParCSRMatrix: the structure hypre_ParCSRMatrix uses
-//    HYPRE_MEMORY_HOST; col_map_offd, row_starts, col_starts, rowindices,
-//    rowvalues also use HYPRE_MEMORY_HOST.
-//
-//    Note: the function hypre_ParCSRMatrixToCSRMatrixAll allocates matrix_i
-//          using HYPRE_MEMORY_HOST!
-//
-// 5. The goal for the MFEM wrappers of hypre objects is to support only the
-//    standard hypre build case, i.e. when hypre is build without device support
-//    and all memory types correspond to host memory. In this case memory
-//    allocated with operator new can be used by hypre but (as usual) it must
-//    not be owned by hypre.
-
-#endif // #if MFEM_HYPRE_VERSION < 21400
-
 using namespace std;
 
 namespace mfem
@@ -1713,6 +1664,293 @@ HypreParMatrix * RAP(const HypreParMatrix * Rt, const HypreParMatrix *A,
    return new HypreParMatrix(rap);
 }
 
+// Helper function for HypreParMatrixFromBlocks. Note that scalability to
+// extremely large processor counts is limited by the use of MPI_Allgather.
+void GatherBlockOffsetData(MPI_Comm comm, const int rank, const int nprocs,
+                           const int num_loc, Array<int> &offsets,
+                           std::vector<int> &all_num_loc, const int numBlocks,
+                           std::vector<std::vector<int>> &blockProcOffsets,
+                           std::vector<int> &procOffsets,
+                           std::vector<std::vector<int>> &procBlockOffsets,
+                           int &firstLocal, int &globalNum)
+{
+   std::vector<std::vector<int>> all_block_num_loc(numBlocks);
+
+   MPI_Allgather(&num_loc, 1, MPI_INT, all_num_loc.data(), 1, MPI_INT, comm);
+
+   for (int j = 0; j < numBlocks; ++j)
+   {
+      all_block_num_loc[j].resize(nprocs);
+      blockProcOffsets[j].resize(nprocs);
+
+      const int blockNumRows = offsets[j + 1] - offsets[j];
+      MPI_Allgather(&blockNumRows, 1, MPI_INT, all_block_num_loc[j].data(), 1,
+                    MPI_INT, comm);
+      blockProcOffsets[j][0] = 0;
+      for (int i = 0; i < nprocs - 1; ++i)
+      {
+         blockProcOffsets[j][i + 1] = blockProcOffsets[j][i]
+                                      + all_block_num_loc[j][i];
+      }
+   }
+
+   firstLocal = 0;
+   globalNum = 0;
+   procOffsets[0] = 0;
+   for (int i = 0; i < nprocs; ++i)
+   {
+      globalNum += all_num_loc[i];
+      if (i < rank)
+      {
+         firstLocal += all_num_loc[i];
+      }
+
+      if (i < nprocs - 1)
+      {
+         procOffsets[i + 1] = procOffsets[i] + all_num_loc[i];
+      }
+
+      procBlockOffsets[i].resize(numBlocks);
+      procBlockOffsets[i][0] = 0;
+      for (int j = 1; j < numBlocks; ++j)
+      {
+         procBlockOffsets[i][j] = procBlockOffsets[i][j - 1]
+                                  + all_block_num_loc[j - 1][i];
+      }
+   }
+}
+
+HypreParMatrix * HypreParMatrixFromBlocks(Array2D<HypreParMatrix*> &blocks,
+                                          Array2D<double> *blockCoeff)
+{
+   const int numBlockRows = blocks.NumRows();
+   const int numBlockCols = blocks.NumCols();
+
+   MFEM_VERIFY(numBlockRows > 0 &&
+               numBlockCols > 0, "Invalid input to HypreParMatrixFromBlocks");
+
+   if (blockCoeff != NULL)
+   {
+      MFEM_VERIFY(numBlockRows == blockCoeff->NumRows() &&
+                  numBlockCols == blockCoeff->NumCols(),
+                  "Invalid input to HypreParMatrixFromBlocks");
+   }
+
+   Array<int> rowOffsets(numBlockRows+1);
+   Array<int> colOffsets(numBlockCols+1);
+
+   int nonNullBlockRow0 = -1;
+   for (int j=0; j<numBlockCols; ++j)
+   {
+      if (blocks(0,j) != NULL)
+      {
+         nonNullBlockRow0 = j;
+         break;
+      }
+   }
+
+   MFEM_VERIFY(nonNullBlockRow0 >= 0, "Null row of blocks");
+   MPI_Comm comm = blocks(0,nonNullBlockRow0)->GetComm();
+
+   // Set offsets based on the number of rows or columns in each block.
+   rowOffsets = 0;
+   colOffsets = 0;
+   for (int i=0; i<numBlockRows; ++i)
+   {
+      for (int j=0; j<numBlockCols; ++j)
+      {
+         if (blocks(i,j) != NULL)
+         {
+            const int nrows = blocks(i,j)->NumRows();
+            const int ncols = blocks(i,j)->NumCols();
+
+            MFEM_VERIFY(nrows > 0 &&
+                        ncols > 0, "Invalid block in HypreParMatrixFromBlocks");
+
+            if (rowOffsets[i+1] == 0)
+            {
+               rowOffsets[i+1] = nrows;
+            }
+            else
+            {
+               MFEM_VERIFY(rowOffsets[i+1] == nrows,
+                           "Inconsistent blocks in HypreParMatrixFromBlocks");
+            }
+
+            if (colOffsets[j+1] == 0)
+            {
+               colOffsets[j+1] = ncols;
+            }
+            else
+            {
+               MFEM_VERIFY(colOffsets[j+1] == ncols,
+                           "Inconsistent blocks in HypreParMatrixFromBlocks");
+            }
+         }
+      }
+
+      MFEM_VERIFY(rowOffsets[i+1] > 0, "Invalid input blocks");
+      rowOffsets[i+1] += rowOffsets[i];
+   }
+
+   for (int j=0; j<numBlockCols; ++j)
+   {
+      MFEM_VERIFY(colOffsets[j+1] > 0, "Invalid input blocks");
+      colOffsets[j+1] += colOffsets[j];
+   }
+
+   const int num_loc_rows = rowOffsets[numBlockRows];
+   const int num_loc_cols = colOffsets[numBlockCols];
+
+   int nprocs, rank;
+   MPI_Comm_rank(comm, &rank);
+   MPI_Comm_size(comm, &nprocs);
+
+   std::vector<int> all_num_loc_rows(nprocs);
+   std::vector<int> all_num_loc_cols(nprocs);
+   std::vector<int> procRowOffsets(nprocs);
+   std::vector<int> procColOffsets(nprocs);
+   std::vector<std::vector<int>> blockRowProcOffsets(numBlockRows);
+   std::vector<std::vector<int>> blockColProcOffsets(numBlockCols);
+   std::vector<std::vector<int>> procBlockRowOffsets(nprocs);
+   std::vector<std::vector<int>> procBlockColOffsets(nprocs);
+
+   int first_loc_row, glob_nrows, first_loc_col, glob_ncols;
+   GatherBlockOffsetData(comm, rank, nprocs, num_loc_rows, rowOffsets,
+                         all_num_loc_rows, numBlockRows, blockRowProcOffsets,
+                         procRowOffsets, procBlockRowOffsets, first_loc_row,
+                         glob_nrows);
+
+   GatherBlockOffsetData(comm, rank, nprocs, num_loc_cols, colOffsets,
+                         all_num_loc_cols, numBlockCols, blockColProcOffsets,
+                         procColOffsets, procBlockColOffsets, first_loc_col,
+                         glob_ncols);
+
+   std::vector<int> opI(num_loc_rows + 1);
+   std::vector<int> cnt(num_loc_rows);
+
+   for (int i = 0; i < num_loc_rows; ++i)
+   {
+      opI[i] = 0;
+      cnt[i] = 0;
+   }
+
+   opI[num_loc_rows] = 0;
+
+   Array2D<hypre_CSRMatrix *> csr_blocks(numBlockRows, numBlockCols);
+
+   // Loop over all blocks, to determine nnz for each row.
+   for (int i = 0; i < numBlockRows; ++i)
+   {
+      for (int j = 0; j < numBlockCols; ++j)
+      {
+         if (blocks(i, j) == NULL)
+         {
+            csr_blocks(i, j) = NULL;
+         }
+         else
+         {
+            {
+               hypre_ParCSRMatrix *parcsr_op = (hypre_ParCSRMatrix*)
+                                               const_cast<HypreParMatrix&>
+                                               (*(blocks(i, j)));
+               MFEM_ASSERT(parcsr_op != NULL, "const_cast failed");
+               csr_blocks(i, j) = hypre_MergeDiagAndOffd(parcsr_op);
+            }
+
+            for (int k = 0; k < csr_blocks(i, j)->num_rows; ++k)
+            {
+               opI[rowOffsets[i] + k + 1] +=
+                  csr_blocks(i, j)->i[k + 1] - csr_blocks(i, j)->i[k];
+            }
+         }
+      }
+   }
+
+   // Now opI[i] is nnz for row i-1. Do a partial sum to get offsets.
+   for (int i = 0; i < num_loc_rows; ++i)
+   {
+      opI[i + 1] += opI[i];
+   }
+
+   const int nnz = opI[num_loc_rows];
+
+   std::vector<HYPRE_Int> opJ(nnz);
+   std::vector<double> data(nnz);
+
+   // Loop over all blocks, to set matrix data.
+   for (int i = 0; i < numBlockRows; ++i)
+   {
+      for (int j = 0; j < numBlockCols; ++j)
+      {
+         if (csr_blocks(i, j) != NULL)
+         {
+            const int nrows = csr_blocks(i, j)->num_rows;
+            const double cij = blockCoeff ? (*blockCoeff)(i, j) : 1.0;
+
+            for (int k = 0; k < nrows; ++k)
+            {
+               const int rowg = rowOffsets[i] + k; // process-local row
+               const int nnz_k = csr_blocks(i,j)->i[k+1]-csr_blocks(i,j)->i[k];
+               const int osk = csr_blocks(i, j)->i[k];
+
+               for (int l = 0; l < nnz_k; ++l)
+               {
+                  // Find the column process offset for the block.
+                  const int bcol = csr_blocks(i, j)->j[osk + l];
+                  int bcolproc = 0;
+
+                  for (int p = 1; p < nprocs; ++p)
+                  {
+                     if (blockColProcOffsets[j][p] > bcol)
+                     {
+                        bcolproc = p - 1;
+                        break;
+                     }
+                  }
+                  if (blockColProcOffsets[j][nprocs - 1] <= bcol)
+                  {
+                     bcolproc = nprocs - 1;
+                  }
+
+                  opJ[opI[rowg] + cnt[rowg]] = procColOffsets[bcolproc] +
+                                               procBlockColOffsets[bcolproc][j]
+                                               + bcol
+                                               - blockColProcOffsets[j][bcolproc];
+                  data[opI[rowg] + cnt[rowg]] = cij * csr_blocks(i, j)->data[osk + l];
+                  cnt[rowg]++;
+               }
+            }
+         }
+      }
+   }
+
+   for (int i = 0; i < numBlockRows; ++i)
+   {
+      for (int j = 0; j < numBlockCols; ++j)
+      {
+         if (csr_blocks(i, j) != NULL)
+         {
+            hypre_CSRMatrixDestroy(csr_blocks(i, j));
+         }
+      }
+   }
+
+   std::vector<HYPRE_Int> rowStarts2(2);
+   rowStarts2[0] = first_loc_row;
+   rowStarts2[1] = first_loc_row + all_num_loc_rows[rank];
+
+   std::vector<HYPRE_Int> colStarts2(2);
+   colStarts2[0] = first_loc_col;
+   colStarts2[1] = first_loc_col + all_num_loc_cols[rank];
+
+   return new HypreParMatrix(comm, num_loc_rows, glob_nrows, glob_ncols,
+                             (int *)opI.data(), (HYPRE_Int *)opJ.data(),
+                             (double *)data.data(),
+                             (HYPRE_Int *)rowStarts2.data(),
+                             (HYPRE_Int *)colStarts2.data());
+}
+
 void EliminateBC(HypreParMatrix &A, HypreParMatrix &Ae,
                  const Array<int> &ess_dof_list,
                  const Vector &X, Vector &B)
@@ -1892,6 +2130,7 @@ HypreSmoother::HypreSmoother() : Solver()
 
    l1_norms = NULL;
    pos_l1_norms = false;
+   eig_est_cg_iter = 10;
    B = X = V = Z = NULL;
    X0 = X1 = NULL;
    fir_coeffs = NULL;
@@ -1899,7 +2138,7 @@ HypreSmoother::HypreSmoother() : Solver()
 
 HypreSmoother::HypreSmoother(HypreParMatrix &_A, int _type,
                              int _relax_times, double _relax_weight, double _omega,
-                             int _poly_order, double _poly_fraction)
+                             int _poly_order, double _poly_fraction, int _eig_est_cg_iter)
 {
    type = _type;
    relax_times = _relax_times;
@@ -1907,6 +2146,7 @@ HypreSmoother::HypreSmoother(HypreParMatrix &_A, int _type,
    omega = _omega;
    poly_order = _poly_order;
    poly_fraction = _poly_fraction;
+   eig_est_cg_iter = _eig_est_cg_iter;
 
    l1_norms = NULL;
    pos_l1_norms = false;
@@ -1929,10 +2169,12 @@ void HypreSmoother::SetSOROptions(double _relax_weight, double _omega)
    omega = _omega;
 }
 
-void HypreSmoother::SetPolyOptions(int _poly_order, double _poly_fraction)
+void HypreSmoother::SetPolyOptions(int _poly_order, double _poly_fraction,
+                                   int _eig_est_cg_iter)
 {
    poly_order = _poly_order;
    poly_fraction = _poly_fraction;
+   eig_est_cg_iter = _eig_est_cg_iter;
 }
 
 void HypreSmoother::SetTaubinOptions(double _lambda, double _mu,
@@ -2016,15 +2258,31 @@ void HypreSmoother::SetOperator(const Operator &op)
    if (type == 16)
    {
       poly_scale = 1;
-      hypre_ParCSRMaxEigEstimateCG(*A, poly_scale, 10,
-                                   &max_eig_est, &min_eig_est);
+      if (eig_est_cg_iter > 0)
+      {
+         hypre_ParCSRMaxEigEstimateCG(*A, poly_scale, eig_est_cg_iter,
+                                      &max_eig_est, &min_eig_est);
+      }
+      else
+      {
+         min_eig_est = 0;
+         hypre_ParCSRMaxEigEstimate(*A, poly_scale, &max_eig_est);
+      }
       Z = new HypreParVector(*A);
    }
    else if (type == 1001 || type == 1002)
    {
       poly_scale = 0;
-      hypre_ParCSRMaxEigEstimateCG(*A, poly_scale, 10,
-                                   &max_eig_est, &min_eig_est);
+      if (eig_est_cg_iter > 0)
+      {
+         hypre_ParCSRMaxEigEstimateCG(*A, poly_scale, eig_est_cg_iter,
+                                      &max_eig_est, &min_eig_est);
+      }
+      else
+      {
+         min_eig_est = 0;
+         hypre_ParCSRMaxEigEstimate(*A, poly_scale, &max_eig_est);
+      }
 
       // The Taubin and FIR polynomials are defined on [0, 2]
       max_eig_est /= 2;

linalg/hypre.hpp

@@ -570,6 +570,17 @@ HypreParMatrix * RAP(const HypreParMatrix *A, const HypreParMatrix *P);
 HypreParMatrix * RAP(const HypreParMatrix * Rt, const HypreParMatrix *A,
                      const HypreParMatrix *P);
 
+/// Returns a merged hypre matrix constructed from hypre matrix blocks.
+/** It is assumed that all block matrices use the same communicator, and the
+    block sizes are consistent in rows and columns. Rows and columns are
+    renumbered but not redistributed in parallel, e.g. the block rows owned by
+    each process remain on that process in the resulting matrix. Some blocks can
+    be NULL. Each block and the entire system can be rectangular. Scalability to
+    extremely large processor counts is limited by global MPI communication, see
+    GatherBlockOffsetData in hypre.cpp. */
+HypreParMatrix * HypreParMatrixFromBlocks(Array2D<HypreParMatrix*> &blocks,
+                                          Array2D<double> *blockCoeff=NULL);
+
 /** Eliminate essential BC specified by 'ess_dof_list' from the solution X to
     the r.h.s. B. Here A is a matrix with eliminated BC, while Ae is such that
     (A+Ae) is the original (Neumann) matrix before elimination. */
@@ -615,6 +626,8 @@ protected:
    double *l1_norms;
    /// If set, take absolute values of the computed l1_norms
    bool pos_l1_norms;
+   /// Number of CG iterations to determine eigenvalue estimates
+   int eig_est_cg_iter;
    /// Maximal eigenvalue estimate for polynomial smoothing
    double max_eig_est;
    /// Minimal eigenvalue estimate for polynomial smoothing
@@ -645,14 +658,17 @@ public:
    HypreSmoother(HypreParMatrix &_A, int type = l1GS,
                  int relax_times = 1, double relax_weight = 1.0,
                  double omega = 1.0, int poly_order = 2,
-                 double poly_fraction = .3);
+                 double poly_fraction = .3, int eig_est_cg_iter = 10);
 
    /// Set the relaxation type and number of sweeps
    void SetType(HypreSmoother::Type type, int relax_times = 1);
    /// Set SOR-related parameters
    void SetSOROptions(double relax_weight, double omega);
    /// Set parameters for polynomial smoothing
-   void SetPolyOptions(int poly_order, double poly_fraction);
+   /** By default, 10 iterations of CG are used to estimate the eigenvalues.
+       Setting eig_est_cg_iter = 0 uses hypre's hypre_ParCSRMaxEigEstimate() instead. */
+   void SetPolyOptions(int poly_order, double poly_fraction,
+                       int eig_est_cg_iter = 10);
    /// Set parameters for Taubin's lambda-mu method
    void SetTaubinOptions(double lambda, double mu, int iter);
 

linalg/hypre_parcsr.cpp

@@ -17,26 +17,6 @@
 #include "hypre_parcsr.hpp"
 #include <limits>
 
-// Define macro wrappers for hypre_TAlloc, hypre_CTAlloc and hypre_TFree:
-// mfem_hypre_TAlloc, mfem_hypre_CTAlloc, and mfem_hypre_TFree, respectively.
-// Note: the same macros are defined in hypre.cpp.
-#if MFEM_HYPRE_VERSION < 21400
-
-#define mfem_hypre_TAlloc(type, size) hypre_TAlloc(type, size)
-#define mfem_hypre_CTAlloc(type, size) hypre_CTAlloc(type, size)
-#define mfem_hypre_TFree(ptr) hypre_TFree(ptr)
-
-#else // MFEM_HYPRE_VERSION >= 21400
-
-// See the notes about hypre 2.14.0 in hypre.cpp
-#define mfem_hypre_TAlloc(type, size) \
-   hypre_TAlloc(type, size, HYPRE_MEMORY_HOST)
-#define mfem_hypre_CTAlloc(type, size) \
-   hypre_CTAlloc(type, size, HYPRE_MEMORY_HOST)
-#define mfem_hypre_TFree(ptr) hypre_TFree(ptr, HYPRE_MEMORY_HOST)
-
-#endif // #if MFEM_HYPRE_VERSION < 21400
-
 namespace mfem
 {
 namespace internal

linalg/hypre_parcsr.hpp

@@ -21,6 +21,56 @@
 
 #include "_hypre_parcsr_mv.h"
 
+// Define macro wrappers for hypre_TAlloc, hypre_CTAlloc and hypre_TFree:
+// mfem_hypre_TAlloc, mfem_hypre_CTAlloc, and mfem_hypre_TFree, respectively.
+// Note: these macros are used in hypre.cpp, hypre_parcsr.cpp, and perhaps
+// other locations in the future.
+#if MFEM_HYPRE_VERSION < 21400
+
+#define mfem_hypre_TAlloc(type, size) hypre_TAlloc(type, size)
+#define mfem_hypre_CTAlloc(type, size) hypre_CTAlloc(type, size)
+#define mfem_hypre_TFree(ptr) hypre_TFree(ptr)
+
+#else // MFEM_HYPRE_VERSION >= 21400
+
+#define mfem_hypre_TAlloc(type, size) \
+   hypre_TAlloc(type, size, HYPRE_MEMORY_HOST)
+#define mfem_hypre_CTAlloc(type, size) \
+   hypre_CTAlloc(type, size, HYPRE_MEMORY_HOST)
+#define mfem_hypre_TFree(ptr) hypre_TFree(ptr, HYPRE_MEMORY_HOST)
+
+// Notes regarding allocation and deallocation of hypre objects in 2.14.0
+//-----------------------------------------------------------------------
+//
+// 1. hypre_CSRMatrix: i, j, data, and rownnz use HYPRE_MEMORY_SHARED while the
+//    hypre_CSRMatrix structure uses HYPRE_MEMORY_HOST.
+//
+//    Note: the function HYPRE_CSRMatrixCreate creates the i array using
+//          HYPRE_MEMORY_HOST!
+//    Note: the functions hypre_CSRMatrixAdd and hypre_CSRMatrixMultiply create
+//          C_i using HYPRE_MEMORY_HOST!
+//
+// 2. hypre_Vector: data uses HYPRE_MEMORY_SHARED while the hypre_Vector
+//    structure uses HYPRE_MEMORY_HOST.
+//
+// 3. hypre_ParVector: the structure hypre_ParVector uses HYPRE_MEMORY_HOST;
+//    partitioning uses HYPRE_MEMORY_HOST.
+//
+// 4. hypre_ParCSRMatrix: the structure hypre_ParCSRMatrix uses
+//    HYPRE_MEMORY_HOST; col_map_offd, row_starts, col_starts, rowindices,
+//    rowvalues also use HYPRE_MEMORY_HOST.
+//
+//    Note: the function hypre_ParCSRMatrixToCSRMatrixAll allocates matrix_i
+//          using HYPRE_MEMORY_HOST!
+//
+// 5. The goal for the MFEM wrappers of hypre objects is to support only the
+//    standard hypre build case, i.e. when hypre is build without device support
+//    and all memory types correspond to host memory. In this case memory
+//    allocated with operator new can be used by hypre but (as usual) it must
+//    not be owned by hypre.
+
+#endif // #if MFEM_HYPRE_VERSION < 21400
+
 namespace mfem
 {
 

linalg/kernels.hpp

@@ -120,7 +120,7 @@ void Symmetrize(const int size, T *data)
 template<int dim, typename T>
 MFEM_HOST_DEVICE inline T Det(const T *data)
 {
-   return TDet<T>(ColumnMajorLayout2D<dim,dim>(), data);
+   return TDetHD<T>(ColumnMajorLayout2D<dim,dim>(), data);
 }
 
 /** @brief Return the inverse a matrix with given @a size and @a data into the
@@ -130,8 +130,8 @@ MFEM_HOST_DEVICE inline
 void CalcInverse(const T *data, T *inv_data)
 {
    typedef ColumnMajorLayout2D<dim,dim> layout_t;
-   const T det = TAdjDet<T>(layout_t(), data, layout_t(), inv_data);
-   TAssign<AssignOp::Mult>(layout_t(), inv_data, static_cast<T>(1.0)/det);
+   const T det = TAdjDetHD<T>(layout_t(), data, layout_t(), inv_data);
+   TAssignHD<AssignOp::Mult>(layout_t(), inv_data, static_cast<T>(1.0)/det);
 }
 
 /** @brief Compute C = A + alpha*B, where the matrices A, B and C are of size @a

linalg/operator.cpp

@@ -543,6 +543,35 @@ void RectangularConstrainedOperator::Mult(const Vector &x, Vector &y) const
    }
 }
 
+void RectangularConstrainedOperator::MultTranspose(const Vector &x,
+                                                   Vector &y) const
+{
+   const int trial_csz = trial_constraints.Size();
+   const int test_csz = test_constraints.Size();
+   if (test_csz == 0)
+   {
+      A->MultTranspose(x, y);
+   }
+   else
+   {
+      z = x;
+
+      auto idx = test_constraints.Read();
+      // Use read+write access - we are modifying sub-vector of z
+      auto d_z = z.ReadWrite();
+      MFEM_FORALL(i, test_csz, d_z[idx[i]] = 0.0;);
+
+      A->MultTranspose(z, y);
+   }
+
+   if (trial_csz != 0)
+   {
+      auto idx = trial_constraints.Read();
+      auto d_y = y.ReadWrite();
+      MFEM_FORALL(i, trial_csz, d_y[idx[i]] = 0.0;);
+   }
+}
+
 double PowerMethod::EstimateLargestEigenvalue(Operator& opr, Vector& v0,
                                               int numSteps, double tolerance, int seed)
 {

linalg/operator.hpp

@@ -759,6 +759,7 @@ public:
        where the "_i" subscripts denote all the nonessential (boundary) trial
        indices and the "_j" subscript denotes the essential test indices */
    virtual void Mult(const Vector &x, Vector &y) const;
+   virtual void MultTranspose(const Vector &x, Vector &y) const;
    virtual ~RectangularConstrainedOperator() { if (own_A) { delete A; } }
 };
 

linalg/simd.hpp

@@ -0,0 +1,100 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#ifndef MFEM_SIMD_HPP
+#define MFEM_SIMD_HPP
+
+#include "../config/tconfig.hpp"
+
+// --- AutoSIMD + specializations with intrinsics
+#include "simd/auto.hpp"
+#ifdef MFEM_USE_SIMD
+#if defined(__VSX__)
+#include "simd/vsx.hpp"
+#elif defined (__bgq__)
+#include "simd/qpx.hpp"
+#elif defined(__x86_64__) || defined(_M_X64) || defined(_M_IX86)
+#include "simd/x86.hpp"
+#elif !defined(_MSC_VER)
+#warning Unknown SIMD architecture
+#else
+#pragma message("warning: Unknown SIMD architecture")
+#endif
+#endif
+
+// MFEM_SIMD_BYTES is the default SIMD size used by MFEM, see e.g. class
+// TBilinearForm and the default traits class AutoSIMDTraits.
+// MFEM_ALIGN_BYTES determines the padding used in TVector when its 'align'
+// template parameter is set to true -- it ensues that the size of such TVector
+// types is a multiple of MFEM_ALIGN_BYTES. MFEM_ALIGN_BYTES must be a multiple
+// of MFEM_SIMD_BYTES.
+#if !defined(MFEM_USE_SIMD)
+#define MFEM_SIMD_BYTES 8
+#define MFEM_ALIGN_BYTES 32
+#elif defined(__AVX512F__)
+#define MFEM_SIMD_BYTES 64
+#define MFEM_ALIGN_BYTES 64
+#elif defined(__AVX__) || defined(__VECTOR4DOUBLE__)
+#define MFEM_SIMD_BYTES 32
+#define MFEM_ALIGN_BYTES 32
+#elif defined(__SSE2__) || defined(__VSX__)
+#define MFEM_SIMD_BYTES 16
+#define MFEM_ALIGN_BYTES 32
+#else
+#define MFEM_SIMD_BYTES 8
+#define MFEM_ALIGN_BYTES 32
+#endif
+
+// derived macros
+#define MFEM_ROUNDUP(val,base) ((((val)+(base)-1)/(base))*(base))
+#define MFEM_ALIGN_SIZE(size,type) \
+   MFEM_ROUNDUP(size,(MFEM_ALIGN_BYTES)/sizeof(type))
+
+namespace mfem
+{
+
+template<typename complex_t, typename real_t>
+struct AutoSIMDTraits
+{
+   static const int block_size = MFEM_TEMPLATE_BLOCK_SIZE;
+
+   // Alignment for arrays of vcomplex_t and vreal_t
+   static const int align_bytes = MFEM_SIMD_BYTES;
+
+   static const int batch_size = 1;
+
+   static const int simd_size = MFEM_SIMD_BYTES/sizeof(real_t);
+
+   typedef AutoSIMD<complex_t, simd_size, MFEM_SIMD_BYTES> vcomplex_t;
+   typedef AutoSIMD<real_t, simd_size, MFEM_SIMD_BYTES> vreal_t;
+   typedef AutoSIMD<int, simd_size, simd_size*sizeof(int)> vint_t;
+};
+
+template<typename complex_t, typename real_t>
+struct NoSIMDTraits
+{
+   static const int block_size = MFEM_TEMPLATE_BLOCK_SIZE;
+
+   // Alignment for arrays of vcomplex_t and vreal_t
+   static const int align_bytes = sizeof(real_t);
+
+   static const int batch_size = 1;
+
+   static const int simd_size = 1;
+
+   typedef AutoSIMD<complex_t, simd_size, align_bytes> vcomplex_t;
+   typedef AutoSIMD<real_t, simd_size, align_bytes> vreal_t;
+   typedef AutoSIMD<int, simd_size, simd_size*sizeof(int)> vint_t;
+};
+
+} // mfem namespace
+
+#endif // MFEM_SIMD_HPP

linalg/simd/auto.hpp

@@ -0,0 +1,273 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#ifndef MFEM_SIMD_AUTO_HPP
+#define MFEM_SIMD_AUTO_HPP
+
+#include "../../config/tconfig.hpp"
+
+namespace mfem
+{
+
+// Use this macro as a workaround for astyle formatting issue with 'alignas'
+#define MFEM_AUTOSIMD_ALIGN__ alignas(align_bytes_)
+
+template <typename scalar_t, int S, int align_bytes_>
+struct MFEM_AUTOSIMD_ALIGN__ AutoSIMD
+{
+   typedef scalar_t scalar_type;
+   static const int size = S;
+   static const int align_bytes = align_bytes_;
+
+   scalar_t vec[size];
+
+   inline MFEM_ALWAYS_INLINE scalar_t &operator[](int i)
+   {
+      return vec[i];
+   }
+
+   inline MFEM_ALWAYS_INLINE const scalar_t &operator[](int i) const
+   {
+      return vec[i];
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator=(const AutoSIMD &v)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] = v[i]; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator=(const scalar_t &e)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] = e; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator+=(const AutoSIMD &v)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] += v[i]; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator+=(const scalar_t &e)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] += e; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator-=(const AutoSIMD &v)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] -= v[i]; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator-=(const scalar_t &e)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] -= e; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator*=(const AutoSIMD &v)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] *= v[i]; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator*=(const scalar_t &e)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] *= e; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator/=(const AutoSIMD &v)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] /= v[i]; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator/=(const scalar_t &e)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] /= e; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator-() const
+   {
+      AutoSIMD r;
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { r[i] = -vec[i]; }
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator+(const AutoSIMD &v) const
+   {
+      AutoSIMD r;
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { r[i] = vec[i] + v[i]; }
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator+(const scalar_t &e) const
+   {
+      AutoSIMD r;
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { r[i] = vec[i] + e; }
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator-(const AutoSIMD &v) const
+   {
+      AutoSIMD r;
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { r[i] = vec[i] - v[i]; }
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator-(const scalar_t &e) const
+   {
+      AutoSIMD r;
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { r[i] = vec[i] - e; }
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator*(const AutoSIMD &v) const
+   {
+      AutoSIMD r;
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { r[i] = vec[i] * v[i]; }
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator*(const scalar_t &e) const
+   {
+      AutoSIMD r;
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { r[i] = vec[i] * e; }
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator/(const AutoSIMD &v) const
+   {
+      AutoSIMD r;
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { r[i] = vec[i] / v[i]; }
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator/(const scalar_t &e) const
+   {
+      AutoSIMD r;
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { r[i] = vec[i] / e; }
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &fma(const AutoSIMD &v, const AutoSIMD &w)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] += v[i] * w[i]; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &fma(const AutoSIMD &v, const scalar_t &e)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] += v[i] * e; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &fma(const scalar_t &e, const AutoSIMD &v)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] += e * v[i]; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &mul(const AutoSIMD &v, const AutoSIMD &w)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] = v[i] * w[i]; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &mul(const AutoSIMD &v, const scalar_t &e)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] = v[i] * e; }
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &mul(const scalar_t &e, const AutoSIMD &v)
+   {
+      MFEM_VECTORIZE_LOOP
+      for (int i = 0; i < size; i++) { vec[i] = e * v[i]; }
+      return *this;
+   }
+};
+
+template <typename scalar_t, int S, int A>
+inline MFEM_ALWAYS_INLINE
+AutoSIMD<scalar_t,S,A> operator+(const scalar_t &e,
+                                 const AutoSIMD<scalar_t,S,A> &v)
+{
+   AutoSIMD<scalar_t,S,A> r;
+   MFEM_VECTORIZE_LOOP
+   for (int i = 0; i < S; i++) { r[i] = e + v[i]; }
+   return r;
+}
+
+template <typename scalar_t, int S, int A>
+inline MFEM_ALWAYS_INLINE
+AutoSIMD<scalar_t,S,A> operator-(const scalar_t &e,
+                                 const AutoSIMD<scalar_t,S,A> &v)
+{
+   AutoSIMD<scalar_t,S,A> r;
+   MFEM_VECTORIZE_LOOP
+   for (int i = 0; i < S; i++) { r[i] = e - v[i]; }
+   return r;
+}
+
+template <typename scalar_t, int S, int A>
+inline MFEM_ALWAYS_INLINE
+AutoSIMD<scalar_t,S,A> operator*(const scalar_t &e,
+                                 const AutoSIMD<scalar_t,S,A> &v)
+{
+   AutoSIMD<scalar_t,S,A> r;
+   MFEM_VECTORIZE_LOOP
+   for (int i = 0; i < S; i++) { r[i] = e * v[i]; }
+   return r;
+}
+
+template <typename scalar_t, int S, int A>
+inline MFEM_ALWAYS_INLINE
+AutoSIMD<scalar_t,S,A> operator/(const scalar_t &e,
+                                 const AutoSIMD<scalar_t,S,A> &v)
+{
+   AutoSIMD<scalar_t,S,A> r;
+   MFEM_VECTORIZE_LOOP
+   for (int i = 0; i < S; i++) { r[i] = e / v[i]; }
+   return r;
+}
+
+} // namespace mfem
+
+#endif // MFEM_SIMD_AUTO_HPP

linalg/simd/m128.hpp

@@ -0,0 +1,254 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#ifndef MFEM_SIMD_M128_HPP
+#define MFEM_SIMD_M128_HPP
+
+#ifdef __SSE2__
+
+#include "../../config/tconfig.hpp"
+#if defined(__x86_64__)
+#include <x86intrin.h>
+#else // assuming MSVC with _M_X64 or _M_IX86
+#include <intrin.h>
+#endif
+
+namespace mfem
+{
+
+template <typename, int, int> struct AutoSIMD;
+
+template <> struct AutoSIMD<double,2,16>
+{
+   typedef double scalar_type;
+   static constexpr int size = 2;
+   static constexpr int align_bytes = 16;
+
+   union
+   {
+      __m128d m128d;
+      double vec[size];
+   };
+
+   inline MFEM_ALWAYS_INLINE double &operator[](int i)
+   {
+      return vec[i];
+   }
+
+   inline MFEM_ALWAYS_INLINE const double &operator[](int i) const
+   {
+      return vec[i];
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator=(const AutoSIMD &v)
+   {
+      m128d = v.m128d;
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator=(const double &e)
+   {
+      m128d = _mm_set1_pd(e);
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator+=(const AutoSIMD &v)
+   {
+      m128d = _mm_add_pd(m128d,v.m128d);
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator+=(const double &e)
+   {
+      m128d = _mm_add_pd(m128d,_mm_set1_pd(e));
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator-=(const AutoSIMD &v)
+   {
+      m128d = _mm_sub_pd(m128d,v.m128d);
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator-=(const double &e)
+   {
+      m128d = _mm_sub_pd(m128d,_mm_set1_pd(e));
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator*=(const AutoSIMD &v)
+   {
+      m128d = _mm_mul_pd(m128d,v.m128d);
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator*=(const double &e)
+   {
+      m128d = _mm_mul_pd(m128d,_mm_set1_pd(e));
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator/=(const AutoSIMD &v)
+   {
+      m128d = _mm_div_pd(m128d,v.m128d);
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &operator/=(const double &e)
+   {
+      m128d = _mm_div_pd(m128d,_mm_set1_pd(e));
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator-() const
+   {
+      AutoSIMD r;
+      r.m128d = _mm_xor_pd(_mm_set1_pd(-0.0), m128d);
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator+(const AutoSIMD &v) const
+   {
+      AutoSIMD r;
+      r.m128d = _mm_add_pd(m128d,v.m128d);
+      return r;
+   }
+
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator+(const double &e) const
+   {
+      AutoSIMD r;
+      r.m128d = _mm_add_pd(m128d, _mm_set1_pd(e));
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator-(const AutoSIMD &v) const
+   {
+      AutoSIMD r;
+      r.m128d = _mm_sub_pd(m128d,v.m128d);
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator-(const double &e) const
+   {
+      AutoSIMD r;
+      r.m128d = _mm_sub_pd(m128d, _mm_set1_pd(e));
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator*(const AutoSIMD &v) const
+   {
+      AutoSIMD r;
+      r.m128d = _mm_mul_pd(m128d,v.m128d);
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator*(const double &e) const
+   {
+      AutoSIMD r;
+      r.m128d = _mm_mul_pd(m128d, _mm_set1_pd(e));
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator/(const AutoSIMD &v) const
+   {
+      AutoSIMD r;
+      r.m128d = _mm_div_pd(m128d,v.m128d);
+      return r;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD operator/(const double &e) const
+   {
+      AutoSIMD r;
+      r.m128d = _mm_div_pd(m128d, _mm_set1_pd(e));
+      return r;
+   }
+
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &fma(const AutoSIMD &v, const AutoSIMD &w)
+   {
+      m128d = _mm_add_pd(_mm_mul_pd(w.m128d,v.m128d),m128d);
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &fma(const AutoSIMD &v, const double &e)
+   {
+      m128d = _mm_add_pd(_mm_mul_pd(_mm_set1_pd(e),v.m128d),m128d);
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &fma(const double &e, const AutoSIMD &v)
+   {
+      m128d = _mm_add_pd(_mm_mul_pd(v.m128d,_mm_set1_pd(e)),m128d);
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &mul(const AutoSIMD &v, const AutoSIMD &w)
+   {
+      m128d = _mm_mul_pd(v.m128d,w.m128d);
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &mul(const AutoSIMD &v, const double &e)
+   {
+      m128d = _mm_mul_pd(v.m128d,_mm_set1_pd(e));
+      return *this;
+   }
+
+   inline MFEM_ALWAYS_INLINE AutoSIMD &mul(const double &e, const AutoSIMD &v)
+   {
+      m128d = _mm_mul_pd(_mm_set1_pd(e),v.m128d);
+      return *this;
+   }
+};
+
+inline MFEM_ALWAYS_INLINE
+AutoSIMD<double,2,16> operator+(const double &e,
+                                const AutoSIMD<double,2,16> &v)
+{
+   AutoSIMD<double,2,16> r;
+   r.m128d = _mm_add_pd(_mm_set1_pd(e),v.m128d);
+   return r;
+}
+
+inline MFEM_ALWAYS_INLINE
+AutoSIMD<double,2,16> operator-(const double &e,
+                                const AutoSIMD<double,2,16> &v)
+{
+   AutoSIMD<double,2,16> r;
+   r.m128d = _mm_sub_pd(_mm_set1_pd(e),v.m128d);
+   return r;
+}
+
+inline MFEM_ALWAYS_INLINE
+AutoSIMD<double,2,16> operator*(const double &e,
+                                const AutoSIMD<double,2,16> &v)
+{
+   AutoSIMD<double,2,16> r;
+   r.m128d = _mm_mul_pd(_mm_set1_pd(e),v.m128d);
+   return r;
+}
+
+inline MFEM_ALWAYS_INLINE
+AutoSIMD<double,2,16> operator/(const double &e,
+                                const AutoSIMD<double,2,16> &v)
+{
+   AutoSIMD<double,2,16> r;
+   r.m128d = _mm_div_pd(_mm_set1_pd(e),v.m128d);
+   return r;
+}
+
+} // namespace mfem
+
+#endif // __SSE2__
+
+#endif // MFEM_SIMD_M128_HPP
+