changes in the configuration files

Fixing GetValue methods in parallel [bugfix/gf-getvalue-dev]

.gitignore

@@ -29,6 +29,8 @@ config/sample-runs-build.log
 doc/CodeDocumentation.conf
 doc/CodeDocumentation.html
 doc/CodeDocumentation
+doc/undoc.log
+doc/warnings.log
 
 # Temporary files created by the tests.
 *.stderr
@@ -167,6 +169,7 @@ miniapps/meshing/twist
 miniapps/meshing/mesh-explorer
 miniapps/meshing/shaper
 miniapps/meshing/extruder
+miniapps/meshing/trimmer
 miniapps/meshing/mesh-optimizer
 miniapps/meshing/pmesh-optimizer
 miniapps/meshing/minimal-surface
@@ -180,6 +183,7 @@ miniapps/meshing/mesh-explorer.mesh
 miniapps/meshing/partitioning.txt
 miniapps/meshing/shaper.mesh
 miniapps/meshing/extruder.mesh
+miniapps/meshing/trimmer.mesh
 miniapps/meshing/optimized*
 miniapps/meshing/perturbed*
 

.travis.yml

@@ -11,8 +11,6 @@
 
 language: cpp
 
-sudo: false
-
 stages:
   - checks
   - tests
@@ -370,8 +368,10 @@ script:
    # Compiler
    - if [ $MPI == "YES" ]; then
         export MYCXX=mpic++;
+        export MAKE_CXX_FLAG=MPICXX=$MYCXX;
      else
         export MYCXX="$CXX";
+        export MAKE_CXX_FLAG=CXX=$MYCXX;
      fi
 
    # Print the compiler version
@@ -384,12 +384,9 @@ script:
      if [ "$CODECOV" == "YES" ]; then
         CPPFLAGS="--coverage -g";
      fi;
-     if [ "$CXX" == "clang++" ]; then
-        export MFEM_PERF_SW=clang;
-     fi
 
    # Configure the library
-   - make config MFEM_USE_MPI=$MPI MFEM_DEBUG=$DEBUG MFEM_CXX="$MYCXX"
+   - make config MFEM_USE_MPI=$MPI MFEM_DEBUG=$DEBUG $MAKE_CXX_FLAG
         MFEM_MPI_NP=$NPROCS CPPFLAGS="$CPPFLAGS"
    # Show the configuration
    - make info

CHANGELOG

@@ -23,6 +23,27 @@ Meshing improvements
   Hessian for r-adaptivity using discrete fields, and allows use of skewness
   and orientation based metrics.
 
+- Added support for r-adaptivity with more than one discrete field. This allows
+  the user to specify different discrete functions for controlling the
+  size, aspect-ratio, orientation, and skew of elements in the mesh.
+
+- Added TMOP capability for approximate tangential mesh relaxation.
+
+- Added support for reading periodic meshes in Gmsh format (version 2.2). See
+  for example the periodic-annulus-sector and periodic-torus-sector files in
+  the data directory.
+
+Performance improvements
+------------------------
+- Added support for explicit vectorization in the high-performance templated
+  code, which can now take advantage of specific intrinsics classes on the
+  following architectures:
+    - x86 (SSE/AVX/AVX2/AVX512),
+    - Power8 & Power9 (VSX),
+    - BG/Q (QPX).
+  These are now enabled by default, and can be disabled with MFEM_USE_SIMD=NO.
+  See the new file linalg/simd.hpp and the new directory linalg/simd.
+
 Improved GPU capabilities
 -------------------------
 - Added support for Chebyshev accelerated polynomial smoother on GPU.
@@ -35,6 +56,26 @@ Discretization improvements
 
 - Added support for simplices in GSLIB-FindPoints.
 
+- Added support for H1 and L2 element matrix assembly in the mass, convection,
+  diffusion, transpose, and the face DG trace integrators. This is compatible
+  with GPU device execution and is illustrated in Example 9/9p, see the option
+  '-ea'. When enabled, this level of assembly stores independent dense matrices
+  for the elements, and independent dense matrices for the faces in the DG case.
+
+- Added new partial assembly kernels for H(div) bilinear forms, as well as
+  VectorFEDivergenceIntegrator.
+
+- Improved the documentation of the GridFunction GetValue and GetVectorValue
+  methods. Expanded the GetValue and GetVectorValue methods which accept an
+  ElementTransformation argument to support evaluation on boundary elements
+  and, in the continuous field case, arbitrary mesh edges and faces.
+
+- Added new coefficient and vector coefficient classes for QuadratureFunctions.
+  Additionaly, new LinearForm integrators were also added which make use of
+  these new QuadratureFunction coefficient classes.
+
+- Added support face integrals on the boundaries of NURBS meshes.
+
 Linear and nonlinear solvers
 ----------------------------
 - Added power method to iteratively estimate the largest eigenvalue and the
@@ -43,6 +84,17 @@ Linear and nonlinear solvers
 - Added initial support for h- and p-multigrid solvers and preconditioners for
   matrix-based and matrix-free discretizations with basic GPU capability.
 
+- Added a new IterativeSolverMonitor class that allows to monitor the residual
+  and solution during the solving process of an IterativeSolver after every
+  iteration.
+
+- Block arrays of parallel matrices can now be merged into a single parallel
+  matrix with the function HypreParMatrixFromBlocks. This could be useful for
+  solving block systems with parallel direct solvers such as STRUMPACK.
+
+- In SLISolver, changed the residual inner product from (Br,r) to (Br,Br) so the
+  solver can work with non-SPD preconditioner B.
+
 New and updated examples and miniapps
 -------------------------------------
 - Added a new example, Example 25/25p, to demonstrate the use of a Perfectly
@@ -52,11 +104,14 @@ New and updated examples and miniapps
 - Added a new Example 26/26p to demonstrate the construction of a matrix-free
   geometric and p-multigrid preconditioner for the Laplace problem.
 
-- Added a new example, Example 27/27p, to demonstrate the enforcement of
-  various boundary conditions with the Laplace operator.  The example shows the
-  procedures for applying Dirichlet, Neumann (both homogeneous and
-  inhomogeneous), Robin, and periodic boundary conditions with either H1 or DG
-  discretizations.
+- Added a new example, Example 27/27p, to demonstrate the enforcement of various
+  boundary conditions with the Laplace operator. The example shows the procedure
+  for applying Dirichlet, Neumann (both homogeneous and inhomogeneous), Robin,
+  and periodic boundary conditions with either H1 or DG discretizations.
+
+- Added a new miniapp, Navier, that solves the time-dependent Navier-Stokes
+  equations of incompressible fluid dynamics. See the miniapps/navier directory
+  for more details.
 
 - Added a simple meshing miniapp, Twist, which demonstrates MFEM's strategy of
   stitching together opposite surfaces of a mesh to create a topologically
@@ -65,6 +120,18 @@ New and updated examples and miniapps
 - Added a new meshing miniapp, Minimal Surface, which solves Plateau's problem:
   the Dirichlet problem for the minimal surface equation.
 
+- Added partial assembly support to examples 4/4p and 5/5p, with diagonal
+  preconditioning.
+
+- Added a new test problem in example 24/24p, demonstrating a mixed bilinear
+  form for H(div) and L_2, with partial assembly support.
+
+- Added weak Dirichlet boundary conditions (Nitsche) to the NURBS miniapp.
+
+- Added a simple mesh editing miniapp, Trimmer, which trims away portions of a
+  mesh based on element attributes. Any newly exposed boundary elements are
+  assigned attribute numbers related to the trimmed element attributes.
+
 Improved testing
 ----------------
 - Added a GitLab pipeline that automates PR testing on supercomputing systems
@@ -74,15 +141,17 @@ Improved testing
 
 Miscellaneous
 -------------
-- In SLISolver, changed the residual inner product from (Br,r) to (Br,Br) so the
-  solver can work with non-SPD preconditioner B.
-
 - Added support for ADIOS2 for parallel I/O with ParaView visualization. The
   classes adios2stream and ADIOS2DataCollection are introduced in mfem as the
   interfaces to generate ADIOS2 Binary Pack (BP4) directory datasets for the
   entire spatial and temporal data. In addition, ADIOS2 allows for setting a
   user-defined number of data substreams/subfiles. See examples 5, 9, 12, 16.
 
+- The integration order used in the ComputeLpError and ComputeElementLpError
+  methods of class GridFunction has been increased.
+
+- Various other simplifications, extensions, and bugfixes in the code.
+
 
 Version 4.1, released on March 10, 2020
 =======================================

CMakeLists.txt

@@ -292,6 +292,25 @@ if (MFEM_USE_HIOP)
   # find_package updates HIOP_FOUND, HIOP_INCLUDE_DIRS, HIOP_LIBRARIES
 endif()
 
+# ADEPT package
+if (MFEM_USE_ADEPT)
+        find_package(ADEPT REQUIRED)
+        # find_package updates ADEPT_FOUND, ADEPT_INCLUDE_DIRS, ADEPT_LIBRARIES
+endif()
+
+# CODIPACK package
+if (MFEM_USE_CODIPACK)
+	find_package(CODIPACK REQUIRED)
+	# find_package updates CODIPACK_FOUND, CODIPACK_INCLUDE_DIRS, CODIPACK_LIBRARIES
+endif()
+
+# FADBAD++ package
+if (MFEM_USE_FADBADPP)
+	find_package(FADBADPP REQUIRED)
+        # find_package updates FADBADPP_FOUND, FADBADPP_INCLUDE_DIRS, FADBADPP_LIBRARIES
+endif()
+
+
 # CUDA
 if (MFEM_USE_CUDA)
    set(CMAKE_CUDA_STANDARD 11)
@@ -353,7 +372,7 @@ endif()
 #    be before SuiteSparse.
 set(MFEM_TPLS MPI_CXX OPENMP BLAS LAPACK METIS HYPRE SuiteSparse SUNDIALS PETSC
     MESQUITE SuperLUDist STRUMPACK AXOM CONDUIT Ginkgo GNUTLS GSLIB NETCDF
-    MPFR PUMI HIOP POSIXCLOCKS MFEMBacktrace ZLIB OCCA CEED RAJA UMPIRE ADIOS2)
+    MPFR PUMI HIOP POSIXCLOCKS MFEMBacktrace ZLIB OCCA CEED RAJA UMPIRE ADIOS2 ADEPT CODIPACK FADBADPP) 
 # Add all *_FOUND libraries in the variable TPL_LIBRARIES.
 set(TPL_LIBRARIES "")
 set(TPL_INCLUDE_DIRS "")

INSTALL

@@ -396,6 +396,12 @@ MFEM_USE_SIDRE = YES/NO
    blueprint specification. When enabled, this option requires installation of
    HDF5 (see also MFEM_USE_NETCDF), Conduit and LLNL's axom project.
 
+MFEM_USE_SIMD = YES/NO
+   Enables the high performance templated classes to use architecture dependent
+   SIMD intrinsics instead of the generic implementation of class AutoSIMD in
+   linalg/simd/auto.hpp. This option should be combined with suitable
+   compiler options, such as -march=native, to enable optimal vectorization.
+
 MFEM_USE_CONDUIT = YES/NO
    Enables support for converting MFEM Mesh and Grid Function objects to and
    from Conduit Mesh Blueprint Descriptions (https://github.com/LLNL/conduit/)
@@ -426,6 +432,8 @@ MFEM_USE_PUMI = YES/NO
    data management system that is capable of handling general non-manifold
    models and effectively supports automated adaptive analysis. PUMI enables
    support for parallel unstructured mesh modifications in MFEM.
+   The develop branch of PUMI repository (https://github.com/SCOREC/core)
+   should be used for most updated features.
 
 MFEM_USE_UMPIRE = YES/NO
    Enables support for Umpire, a resource management library that allows the
@@ -609,8 +617,9 @@ The specific libraries and their options are:
 
 - PUMI (optional), used when MFEM_USE_PUMI = YES.
   URL: https://scorec.rpi.edu/pumi
+       https://github.com/SCOREC/core
   Options: PUMI_OPT, PUMI_LIB.
-  Versions: PUMI >= 2.2.0.
+  Versions: PUMI >= 2.2.3.
 
 - HiOp (optional), used when MFEM_USE_HIOP = YES.
   URL: https://github.com/LLNL/hiop

config/cmake/MFEMConfig.cmake.in

@@ -47,6 +47,7 @@ set(MFEM_USE_OCCA @MFEM_USE_OCCA@)
 set(MFEM_USE_RAJA @MFEM_USE_RAJA@)
 set(MFEM_USE_CEED @MFEM_USE_CEED@)
 set(MFEM_USE_UMPIRE @MFEM_USE_UMPIRE@)
+set(MFEM_USE_SIMD @MFEM_USE_SIMD@)
 set(MFEM_USE_ADIOS2 @MFEM_USE_ADIOS2@)
 
 set(MFEM_CXX_COMPILER "@CMAKE_CXX_COMPILER@")

config/cmake/config.hpp.in

@@ -107,6 +107,9 @@
 // Enable MFEM functionality based on the Sidre library
 #cmakedefine MFEM_USE_SIDRE
 
+// Enable the use of SIMD in the high performance templated classes
+#cmakedefine MFEM_USE_SIMD
+
 // Enable MFEM functionality based on Conduit
 #cmakedefine MFEM_USE_CONDUIT
 
@@ -150,4 +153,11 @@
 // library.
 #cmakedefine MFEM_USE_SIMMETRIX
 
+#cmakedefine MFEM_USE_ADEPT
+
+#cmakedefine MFEM_USE_CODIPACK
+
+#cmakedefine MFEM_USE_FADBADPP
+
+
 #endif // MFEM_CONFIG_HEADER

config/cmake/modules/FindADEPT.cmake

@@ -0,0 +1,23 @@
+# Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at the
+# Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights reserved.
+# See file COPYRIGHT for details.
+#
+# This file is part of the MFEM library. For more information and source code
+# availability see http://mfem.org.
+#
+# MFEM is free software; you can redistribute it and/or modify it under the
+# terms of the GNU Lesser General Public License (as published by the Free
+# Software Foundation) version 2.1 dated February 1999.
+
+# Sets the following variables:
+#   - ADEPT_FOUND
+#   - ADEPT_INCLUDE_DIRS
+#   - ADEPT_LIBRARIES
+
+include(MfemCmakeUtilities)
+mfem_find_package(ADEPT ADEPT ADEPT_DIR
+  "include" "adept.hpp"
+  "lib" "libadept.so"
+  "Paths to headers required by ADEPT."
+  "Libraries required by ADEPT.")
+

config/cmake/modules/FindCODIPACK.cmake

@@ -0,0 +1,23 @@
+# Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at the
+# Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights reserved.
+# See file COPYRIGHT for details.
+#
+# This file is part of the MFEM library. For more information and source code
+# availability see http://mfem.org.
+#
+# MFEM is free software; you can redistribute it and/or modify it under the
+# terms of the GNU Lesser General Public License (as published by the Free
+# Software Foundation) version 2.1 dated February 1999.
+
+# Sets the following variables:
+#   - CODIPACK_FOUND
+#   - CODIPACK_INCLUDE_DIRS
+#   - CODIPACK_LIBRARIES
+
+include(MfemCmakeUtilities)
+mfem_find_package(CODIPACK CODIPACK CODIPACK_DIR
+  "include" "codi.hpp"
+  "lib" ""
+  "Paths to headers required by CODIPACK."
+  "Libraries required by CODIPACK.")
+

config/cmake/modules/FindFADBADPP.cmake

@@ -0,0 +1,23 @@
+# Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at the
+# Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights reserved.
+# See file COPYRIGHT for details.
+#
+# This file is part of the MFEM library. For more information and source code
+# availability see http://mfem.org.
+#
+# MFEM is free software; you can redistribute it and/or modify it under the
+# terms of the GNU Lesser General Public License (as published by the Free
+# Software Foundation) version 2.1 dated February 1999.
+
+# Sets the following variables:
+#   - FADBADPP_FOUND
+#   - FADBADPP_INCLUDE_DIRS
+#   - FADBADPP_LIBRARIES
+
+include(MfemCmakeUtilities)
+mfem_find_package(FADBADPP FADBADPP FADBADPP_DIR
+  "include" "fadiff.h"
+  "lib" ""
+  "Paths to headers required by FADBADPP."
+  "Libraries required by FADBADPP.")
+

config/cmake/modules/MfemCmakeUtilities.cmake

@@ -733,7 +733,7 @@ function(mfem_export_mk_files)
       MFEM_USE_SUPERLU MFEM_USE_STRUMPACK MFEM_USE_GNUTLS
       MFEM_USE_GSLIB MFEM_USE_NETCDF MFEM_USE_PETSC MFEM_USE_MPFR MFEM_USE_SIDRE
       MFEM_USE_CONDUIT MFEM_USE_PUMI MFEM_USE_CUDA MFEM_USE_OCCA MFEM_USE_RAJA
-      MFEM_USE_UMPIRE)
+      MFEM_USE_UMPIRE MFEM_USE_SIMD MFEM_USE_ADIOS2)
   foreach(var ${CONFIG_MK_BOOL_VARS})
     if (${var})
       set(${var} YES)
@@ -743,6 +743,7 @@ function(mfem_export_mk_files)
   endforeach()
   # TODO: Add support for MFEM_USE_CUDA=YES
   set(MFEM_CXX ${CMAKE_CXX_COMPILER})
+  set(MFEM_HOST_CXX ${MFEM_CXX})
   set(MFEM_CPPFLAGS "")
   string(STRIP "${CMAKE_CXX_FLAGS_${BUILD_TYPE}} ${CMAKE_CXX_FLAGS}"
          MFEM_CXXFLAGS)

config/config.hpp.in

@@ -106,6 +106,9 @@
 // Enable Sidre support
 // #define MFEM_USE_SIDRE
 
+// Enable the use of SIMD in the high performance templated classes
+// #define MFEM_USE_SIMD
+
 // Enable Conduit support
 // #define MFEM_USE_CONDUIT
 

config/config.mk.in

@@ -49,10 +49,12 @@ MFEM_USE_RAJA          = @MFEM_USE_RAJA@
 MFEM_USE_OCCA          = @MFEM_USE_OCCA@
 MFEM_USE_CEED          = @MFEM_USE_CEED@
 MFEM_USE_UMPIRE        = @MFEM_USE_UMPIRE@
+MFEM_USE_SIMD          = @MFEM_USE_SIMD@
 MFEM_USE_ADIOS2        = @MFEM_USE_ADIOS2@
 
 # Compiler, compile options, and link options
 MFEM_CXX       = @MFEM_CXX@
+MFEM_HOST_CXX  = @MFEM_HOST_CXX@
 MFEM_CPPFLAGS  = @MFEM_CPPFLAGS@
 MFEM_CXXFLAGS  = @MFEM_CXXFLAGS@
 MFEM_TPLFLAGS  = @MFEM_TPLFLAGS@

config/defaults.cmake

@@ -49,7 +49,11 @@ option(MFEM_USE_OCCA "Enable OCCA" OFF)
 option(MFEM_USE_RAJA "Enable RAJA" OFF)
 option(MFEM_USE_CEED "Enable CEED" OFF)
 option(MFEM_USE_UMPIRE "Enable Umpire" OFF)
+option(MFEM_USE_SIMD "Enable use of SIMD intrinsics" ON)
 option(MFEM_USE_ADIOS2 "Enable ADIOS2" OFF)
+option(MFEM_USE_ADEPT "Enable AD using ADEPT"   OFF)
+option(MFEM_USE_CODIPACK "Enable AD using CoDiPack" OFF)
+option(MFEM_USE_FADBADPP "Enable AD using FADBAD++" OFF)
 
 set(MFEM_MPI_NP 4 CACHE STRING "Number of processes used for MPI tests")
 
@@ -182,6 +186,16 @@ set(BLAS_LIBRARIES "" CACHE STRING "The BLAS library.")
 set(LAPACK_INCLUDE_DIRS "" CACHE STRING "Path to LAPACK headers.")
 set(LAPACK_LIBRARIES "" CACHE STRING "The LAPACK library.")
 
+set(ADEPT_INCLUDE_DIRS "${MFEM_DIR}/../adept-1.1/include" CACHE STRING "Path to ADEPT headers.")
+set(ADEPT_LIBRARIES    "-L${MFEM_DIR}/../adept-1.1/lib -ladept" CACHE STRING "The ADEPT library.")
+
+set(CODIPACK_INCLUDE_DIRS "${MFEM_DIR}/../CoDiPack/include" CACHE STRING "Path to CoDiPack headers.")
+set(CODIPACK_LIBRARIES    "")
+
+set(FADBADPP_INCLUDE_DIRS "${MFEM_DIR}/../FADBAD++" CACHE STRING "Path to FADBAD++ headers.")
+set(FADBADPP_LIBRARIES    "")
+
+
 # Some useful variables:
 set(CMAKE_SKIP_PREPROCESSED_SOURCE_RULES ON) # Skip *.i rules
 set(CMAKE_SKIP_ASSEMBLY_SOURCE_RULES  ON)    # Skip *.s rules

config/defaults.mk

@@ -137,6 +137,7 @@ MFEM_USE_RAJA          = NO
 MFEM_USE_OCCA          = NO
 MFEM_USE_CEED          = NO
 MFEM_USE_UMPIRE        = NO
+MFEM_USE_SIMD          = YES
 MFEM_USE_ADIOS2        = NO
 
 # Compile and link options for zlib.

config/tconfig.hpp

@@ -29,8 +29,20 @@
 #define MFEM_ALWAYS_INLINE
 #endif
 
+// --- MFEM_VECTORIZE_LOOP (disabled)
+#if (__cplusplus >= 201103L) && !defined(MFEM_DEBUG) && defined(__GNUC__)
+//#define MFEM_VECTORIZE_LOOP _Pragma("GCC ivdep")
+#define MFEM_VECTORIZE_LOOP
+#else
+#define MFEM_VECTORIZE_LOOP
+#endif
+
+// MFEM_TEMPLATE_BLOCK_SIZE is the block size used by the template matrix-matrix
+// multiply, Mult_AB, defined in tmatrix.hpp. This parameter will generally
+// require tuning to determine good value. It is probably highly influenced by
+// the SIMD width when Mult_AB is used with a SIMD type like AutoSIMD.
 #define MFEM_TEMPLATE_BLOCK_SIZE 4
-#define MFEM_SIMD_SIZE 32
+
 #define MFEM_TEMPLATE_ENABLE_SERIALIZE
 
 // #define MFEM_TEMPLATE_ELTRANS_HAS_NODE_DOFS
@@ -38,11 +50,6 @@
 // #define MFEM_TEMPLATE_FIELD_EVAL_DATA_HAS_DOFS
 #define MFEM_TEMPLATE_INTRULE_COEFF_PRECOMP
 
-// derived macros
-#define MFEM_ROUNDUP(val,base) ((((val)+(base)-1)/(base))*(base))
-#define MFEM_ALIGN_SIZE(size,type) \
-   MFEM_ROUNDUP(size,(MFEM_SIMD_SIZE)/sizeof(type))
-
 #ifdef MFEM_COUNT_FLOPS
 namespace mfem
 {

config/user.cmake

@@ -0,0 +1,204 @@
+# Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+# at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+# LICENSE and NOTICE for details. LLNL-CODE-806117.
+#
+# This file is part of the MFEM library. For more information and source code
+# availability visit https://mfem.org.
+#
+# MFEM is free software; you can redistribute it and/or modify it under the
+# terms of the BSD-3 license. We welcome feedback and contributions, see file
+# CONTRIBUTING.md for details.
+
+# See the file INSTALL for description of the configuration options.
+
+# Default options. To replace these, copy this file to user.cmake and modify it.
+
+if (NOT CMAKE_BUILD_TYPE)
+set(CMAKE_BUILD_TYPE "Debug" CACHE STRING
+      "Build type: Debug, Release, RelWithDebInfo, or MinSizeRel." FORCE)
+endif()
+
+# MFEM options. Set to mimic the default "defaults.mk" file.
+option(MFEM_USE_MPI "Enable MPI parallel build" ON)
+option(MFEM_USE_METIS "Enable METIS usage" ${MFEM_USE_MPI})
+option(MFEM_USE_EXCEPTIONS "Enable the use of exceptions" OFF)
+option(MFEM_USE_ZLIB "Enable zlib for compressed data streams." OFF)
+option(MFEM_USE_LIBUNWIND "Enable backtrace for errors." ON)
+option(MFEM_USE_LAPACK "Enable LAPACK usage" ON)
+option(MFEM_THREAD_SAFE "Enable thread safety" OFF)
+option(MFEM_USE_OPENMP "Enable the OpenMP backend" OFF)
+option(MFEM_USE_LEGACY_OPENMP "Enable legacy OpenMP usage" OFF)
+option(MFEM_USE_MEMALLOC "Enable the internal MEMALLOC option." ON)
+option(MFEM_USE_SUNDIALS "Enable SUNDIALS usage" OFF)
+option(MFEM_USE_MESQUITE "Enable MESQUITE usage" OFF)
+option(MFEM_USE_SUITESPARSE "Enable SuiteSparse usage" ON)
+option(MFEM_USE_SUPERLU "Enable SuperLU_DIST usage" OFF)
+option(MFEM_USE_STRUMPACK "Enable STRUMPACK usage" OFF)
+option(MFEM_USE_GINKGO "Enable Ginkgo usage" OFF)
+option(MFEM_USE_GNUTLS "Enable GNUTLS usage" OFF)
+option(MFEM_USE_GSLIB "Enable GSLIB usage" OFF)
+option(MFEM_USE_NETCDF "Enable NETCDF usage" OFF)
+option(MFEM_USE_PETSC "Enable PETSc support." ON)
+option(MFEM_USE_MPFR "Enable MPFR usage." OFF)
+option(MFEM_USE_SIDRE "Enable Axom/Sidre usage" OFF)
+option(MFEM_USE_CONDUIT "Enable Conduit usage" OFF)
+option(MFEM_USE_PUMI "Enable PUMI" OFF)
+option(MFEM_USE_HIOP "Enable HiOp" OFF)
+option(MFEM_USE_CUDA "Enable CUDA" OFF)
+option(MFEM_USE_OCCA "Enable OCCA" OFF)
+option(MFEM_USE_RAJA "Enable RAJA" OFF)
+option(MFEM_USE_CEED "Enable CEED" OFF)
+option(MFEM_USE_UMPIRE "Enable Umpire" OFF)
+option(MFEM_USE_SIMD "Enable use of SIMD intrinsics" ON)
+option(MFEM_USE_ADIOS2 "Enable ADIOS2" OFF)
+option(MFEM_USE_ADEPT "Enable AD using ADEPT"   OFF)
+option(MFEM_USE_CODIPACK "Enable AD using CoDiPack" ON)
+option(MFEM_USE_FADBADPP "Enable AD using FADBAD++" OFF)
+
+
+
+set(MFEM_MPI_NP 4 CACHE STRING "Number of processes used for MPI tests")
+
+# Allow a user to disable testing, examples, and/or miniapps at CONFIGURE TIME
+# if they don't want/need them (e.g. if MFEM is "just a dependency" and all they
+# need is the library, building all that stuff adds unnecessary overhead). Note
+# that the examples or miniapps can always be built using the targets 'examples'
+# or 'miniapps', respectively.
+option(MFEM_ENABLE_TESTING "Enable the ctest framework for testing" ON)
+option(MFEM_ENABLE_EXAMPLES "Build all of the examples" OFF)
+option(MFEM_ENABLE_MINIAPPS "Build all of the miniapps" OFF)
+
+# Setting CXX/MPICXX on the command line or in user.cmake will overwrite the
+# autodetected C++ compiler.
+# set(CXX g++)
+# set(MPICXX mpicxx)
+
+# Set the target CUDA architecture
+set(CUDA_ARCH "sm_60" CACHE STRING "Target CUDA architecture.")
+
+set(MFEM_DIR ${CMAKE_CURRENT_SOURCE_DIR})
+
+# The *_DIR paths below will be the first place searched for the corresponding
+# headers and library. If these fail, then standard cmake search is performed.
+# Note: if the variables are already in the cache, they are not overwritten.
+
+set(HYPRE_DIR "/home/blaz/develop/common/dbg/petsc_3.12.5/" CACHE PATH
+    "Path to the hypre library.")
+# If hypre was compiled to depend on BLAS and LAPACK:
+# set(HYPRE_REQUIRED_PACKAGES "BLAS" "LAPACK" CACHE STRING
+#     "Packages that HYPRE depends on.")
+
+set(METIS_DIR "/home/blaz/develop/common/dbg/petsc_3.12.5/" CACHE PATH "Path to the METIS library.")
+
+set(LIBUNWIND_DIR "" CACHE PATH "Path to Libunwind.")
+
+set(SUNDIALS_DIR "/home/blaz/develop/common/dbg/SUNDIALS_5.2.0/" CACHE PATH
+    "Path to the SUNDIALS library.")
+# The following may be necessary, if SUNDIALS was built with KLU:
+# set(SUNDIALS_REQUIRED_PACKAGES "SuiteSparse/KLU/AMD/BTF/COLAMD/config"
+#     CACHE STRING "Additional packages required by SUNDIALS.")
+
+set(MESQUITE_DIR "${MFEM_DIR}/../mesquite-2.99" CACHE PATH
+    "Path to the Mesquite library.")
+
+set(SuiteSparse_DIR "/home/blaz/develop/common/dbg/petsc_3.12.5/" CACHE PATH
+    "Path to the SuiteSparse library.")
+set(SuiteSparse_REQUIRED_PACKAGES "BLAS" "METIS"
+    CACHE STRING "Additional packages required by SuiteSparse.")
+
+set(ParMETIS_DIR "/home/blaz/develop/common/dbg/petsc_3.12.5/" CACHE PATH
+    "Path to the ParMETIS library.")
+set(ParMETIS_REQUIRED_PACKAGES "METIS" CACHE STRING
+    "Additional packages required by ParMETIS.")
+
+set(SuperLUDist_DIR "${MFEM_DIR}/../SuperLU_DIST_5.1.0" CACHE PATH
+    "Path to the SuperLU_DIST library.")
+# SuperLU_DIST may also depend on "OpenMP", depending on how it was compiled.
+set(SuperLUDist_REQUIRED_PACKAGES "MPI" "BLAS" "ParMETIS" CACHE STRING
+    "Additional packages required by SuperLU_DIST.")
+
+set(STRUMPACK_DIR "/home/blaz/develop/common/dbg/petsc_3.12.5/" CACHE PATH
+    "Path to the STRUMPACK library.")
+# STRUMPACK may also depend on "OpenMP", depending on how it was compiled.
+# Starting with v2.2.0 of STRUMPACK, ParMETIS and Scotch are optional.
+set(STRUMPACK_REQUIRED_PACKAGES "MPI" "MPI_Fortran" "ParMETIS" "METIS"
+    "ScaLAPACK" "Scotch/ptscotch/ptscotcherr/scotch/scotcherr" CACHE STRING
+    "Additional packages required by STRUMPACK.")
+# If the MPI package does not find all required Fortran libraries:
+# set(STRUMPACK_REQUIRED_LIBRARIES "gfortran" "mpi_mpifh" CACHE STRING
+#     "Additional libraries required by STRUMPACK.")
+
+# The Scotch library, required by STRUMPACK <= v2.1.0, optional in STRUMPACK >=
+# v2.2.0.
+set(Scotch_DIR "${MFEM_DIR}/../scotch_6.0.4" CACHE PATH
+    "Path to the Scotch and PT-Scotch libraries.")
+set(Scotch_REQUIRED_PACKAGES "Threads" CACHE STRING
+    "Additional packages required by Scotch.")
+# Tell the "Threads" package/module to prefer pthreads.
+set(CMAKE_THREAD_PREFER_PTHREAD TRUE)
+set(Threads_LIB_VARS CMAKE_THREAD_LIBS_INIT)
+
+# The ScaLAPACK library, required by STRUMPACK
+set(ScaLAPACK_DIR "/home/blaz/develop/common/dbg/petsc_3.12.5/"
+    CACHE PATH "Path to the configuration file scalapack-config.cmake")
+set(ScaLAPACK_TARGET_NAMES scalapack)
+# set(ScaLAPACK_TARGET_FORCE)
+# set(ScaLAPACK_IMPORT_CONFIG DEBUG)
+
+set(Ginkgo_DIR "${MFEM_DIR}/../ginkgo" CACHE PATH "Path to the Ginkgo library.")
+
+set(GNUTLS_DIR "" CACHE PATH "Path to the GnuTLS library.")
+
+set(GSLIB_DIR "" CACHE PATH "Path to the GSLIB library.")
+
+set(NETCDF_DIR "" CACHE PATH "Path to the NetCDF library.")
+# May need to add "HDF5" as requirement.
+set(NetCDF_REQUIRED_PACKAGES "" CACHE STRING
+    "Additional packages required by NetCDF.")
+
+set(PETSC_DIR "/home/blaz/develop/common/dbg/petsc_3.12.5/" CACHE PATH
+    "Path to the PETSc main directory.")
+set(PETSC_ARCH "" CACHE STRING "PETSc build architecture.")
+
+set(MPFR_DIR "" CACHE PATH "Path to the MPFR library.")
+
+set(CONDUIT_DIR "${MFEM_DIR}/../conduit" CACHE PATH
+    "Path to the Conduit library.")
+
+set(AXOM_DIR "${MFEM_DIR}/../axom" CACHE PATH "Path to the Axom library.")
+# May need to add "Boost" as requirement.
+set(Axom_REQUIRED_PACKAGES "Conduit/relay/blueprint" CACHE STRING
+    "Additional packages required by Axom.")
+
+set(PUMI_DIR "${MFEM_DIR}/../pumi-2.1.0" CACHE STRING
+    "Directory where PUMI is installed")
+
+set(HIOP_DIR "${MFEM_DIR}/../hiop/install" CACHE STRING
+    "Directory where HiOp is installed")
+set(HIOP_REQUIRED_PACKAGES "BLAS" "LAPACK" CACHE STRING
+    "Packages that HiOp depends on.")
+
+set(OCCA_DIR "${MFEM_DIR}/../occa" CACHE PATH "Path to OCCA")
+set(RAJA_DIR "${MFEM_DIR}/../raja" CACHE PATH "Path to RAJA")
+set(CEED_DIR "${MFEM_DIR}/../libCEED" CACHE PATH "Path to libCEED")
+set(UMPIRE_DIR "${MFEM_DIR}/../umpire" CACHE PATH "Path to Umpire")
+
+set(BLAS_INCLUDE_DIRS "" CACHE STRING "Path to BLAS headers.")
+set(BLAS_LIBRARIES "-L/home/blaz/develop/common/lib -lblas" CACHE STRING "The BLAS library.")
+set(LAPACK_INCLUDE_DIRS "" CACHE STRING "Path to LAPACK headers.")
+set(LAPACK_LIBRARIES "-L/home/blaz/develop/common/lib -llapack" CACHE STRING "The LAPACK library.")
+
+
+set(ADEPT_INCLUDE_DIRS "/home/blaz/develop/common/dbg/adept-1.1/include" CACHE STRING "Path to ADEPT headers.")
+set(ADEPT_LIBRARIES    "/home/blaz/develop/common/dbg/adept-1.1/lib/libadept.so" CACHE STRING "The ADEPT library.")
+
+set(CODIPACK_INCLUDE_DIRS "/home/blaz/develop/common/CoDiPack/include" CACHE STRING "Path to CoDiPack headers.")
+set(CODIPACK_LIBRARIES    "")
+
+set(FADBADPP_INCLUDE_DIRS "/home/blaz/develop/common/FADBAD++" CACHE STRING "Path to FADBAD++ headers.")
+set(FADBADPP_LIBRARIES    "")
+
+# Some useful variables:
+set(CMAKE_SKIP_PREPROCESSED_SOURCE_RULES ON) # Skip *.i rules
+set(CMAKE_SKIP_ASSEMBLY_SOURCE_RULES  ON)    # Skip *.s rules
+# set(CMAKE_VERBOSE_MAKEFILE ON CACHE BOOL "Verbose makefiles.")

data/periodic-annulus-sector.geo

@@ -0,0 +1,37 @@
+SetFactory("OpenCASCADE");
+
+R1 = 1.0;
+R2 = 2.0;
+
+Point(1) = {0.0, 0, 0, 1.0};
+Point(2) = {R1, 0, 0, 1.0};
+Point(3) = {R2, 0, 0, 1.0};
+Point(4) = {R1*Cos(Pi/3), R1*Sin(Pi/3), 0, 1.0};
+Point(5) = {R2*Cos(Pi/3), R2*Sin(Pi/3), 0, 1.0};
+Line(1) = {2, 3};
+Line(2) = {4, 5};
+Circle(3) = {2, 1, 4};
+Circle(4) = {3, 1, 5};
+Curve Loop(5) = {1, 4, -2, -3};
+Plane Surface(1) = {5};
+
+Transfinite Curve{1} = 7;
+Transfinite Curve{2} = 7;
+Transfinite Curve{3} = 4;
+Transfinite Curve{4} = 10;
+
+// Set a rotation periodicity constraint:
+Periodic Line{1} = {2} Rotate{{0,0,1}, {0,0,0}, -Pi/3};
+
+// Tag surfaces and volumes with positive integers
+Physical Curve(1) = {3};
+Physical Curve(2) = {4};
+Physical Curve(3) = {1};
+Physical Curve(4) = {2};
+Physical Surface(1) = {1};
+
+// Generate 2D mesh
+Mesh 2;
+Mesh.MshFileVersion = 2.2;
+
+Save "periodic-annulus-sector.msh";

data/periodic-annulus-sector.msh

@@ -0,0 +1,185 @@
+$MeshFormat
+2.2 0 8
+$EndMeshFormat
+$Nodes
+55
+1 1 0 0
+2 2 0 0
+3 0.5000000000000001 0.8660254037844386 0
+4 1 1.732050807568877 0
+5 1.166666666666667 0 0
+6 1.333333333333333 0 0
+7 1.5 0 0
+8 1.666666666666667 0 0
+9 1.833333333333333 0 0
+10 0.5833333333333335 1.010362971081845 0
+11 0.6666666666666667 1.154700538379251 0
+12 0.7500000000000002 1.299038105676658 0
+13 0.8333333333333335 1.443375672974064 0
+14 0.9166666666666669 1.587713240271471 0
+15 0.9396926207859085 0.3420201433256683 0
+16 0.7660444431189786 0.6427876096865386 0
+17 1.986476715483886 0.2321858282504602 0
+18 1.946089741159648 0.4612317414848793 0
+19 1.879385241571817 0.6840402866513365 0
+20 1.787265280646825 0.8975983604009234 0
+21 1.670975622825874 1.09901795614161 0
+22 1.532088886237958 1.285575219373077 0
+23 1.372483275737469 1.454747283146095 0
+24 1.194317183405575 1.604246385510085 0
+25 1.425989114816062 0.1915326920916892 0
+26 0.8788667344146573 1.13917645290495 0
+27 1.630372059110754 0.7154531062316609 0
+28 1.436395769298814 1.053728612482506 0
+29 1.081023776188756 0.6241293681829633 0
+30 1.168737372335971 1.428012728596308 0
+31 1.821063986059922 0.298149890497067 0
+32 1.234707097211386 0.3469796339295647 0
+33 1.377747393186519 0.6200150626754309 0
+34 1.457047681210906 0.3890895843559762 0
+35 0.917846726184522 0.8957978954532204 0
+36 1.218335619030348 0.9017812086952638 0
+37 1.066623110765233 1.061857005744772 0
+38 1.587029716281926 0.1355955181472859 0
+39 1.744445799211916 0.1441515753740107 0
+40 1.25 0.1443375672974065 0
+41 1.453660070628011 0.8435769396609902 0
+42 1.741367044061892 0.499612708014486 0
+43 1.30550638526547 1.257610469847477 0
+44 1.118213276932792 0.1666674689105279 0
+45 0.9109440214958271 1.306610291787315 0
+46 0.9970618258753989 1.438658589955562 0
+47 0.7499999999999998 1.010362971081845 0
+48 0.7034449005273667 0.8850673702175776 0
+49 1.605449512513618 0.9269067082200894 0
+50 1.561654019115059 0.5298592532912715 0
+51 1.229782222487711 1.096820457143683 0
+52 1.617066998712459 0.3090202662210922 0
+53 1.079645953234324 1.246963713711438 0
+54 1.877063966817811 0.1348974588243076 0
+55 1.055356609656722 1.558136350380461 0
+$EndNodes
+$Elements
+108
+1 1 2 3 1 1 5
+2 1 2 3 1 5 6
+3 1 2 3 1 6 7
+4 1 2 3 1 7 8
+5 1 2 3 1 8 9
+6 1 2 3 1 9 2
+7 1 2 4 2 3 10
+8 1 2 4 2 10 11
+9 1 2 4 2 11 12
+10 1 2 4 2 12 13
+11 1 2 4 2 13 14
+12 1 2 4 2 14 4
+13 1 2 1 3 1 15
+14 1 2 1 3 15 16
+15 1 2 1 3 16 3
+16 1 2 2 4 2 17
+17 1 2 2 4 17 18
+18 1 2 2 4 18 19
+19 1 2 2 4 19 20
+20 1 2 2 4 20 21
+21 1 2 2 4 21 22
+22 1 2 2 4 22 23
+23 1 2 2 4 23 24
+24 1 2 2 4 24 4
+25 2 2 1 1 32 40 25
+26 2 2 1 1 25 34 32
+27 2 2 1 1 33 41 36
+28 2 2 1 1 38 52 25
+29 2 2 1 1 33 36 29
+30 2 2 1 1 26 47 35
+31 2 2 1 1 35 37 26
+32 2 2 1 1 25 52 34
+33 2 2 1 1 32 44 40
+34 2 2 1 1 15 32 29
+35 2 2 1 1 15 29 16
+36 2 2 1 1 36 41 28
+37 2 2 1 1 32 33 29
+38 2 2 1 1 50 52 42
+39 2 2 1 1 32 34 33
+40 2 2 1 1 42 52 31
+41 2 2 1 1 43 53 51
+42 2 2 1 1 27 41 33
+43 2 2 1 1 26 53 45
+44 2 2 1 1 18 31 17
+45 2 2 1 1 29 35 16
+46 2 2 1 1 29 36 35
+47 2 2 1 1 24 30 23
+48 2 2 1 1 30 53 43
+49 2 2 1 1 17 54 2
+50 2 2 1 1 4 55 24
+51 2 2 1 1 28 51 36
+52 2 2 1 1 47 48 35
+53 2 2 1 1 36 37 35
+54 2 2 1 1 37 53 26
+55 2 2 1 1 22 28 21
+56 2 2 1 1 20 27 19
+57 2 2 1 1 33 50 27
+58 2 2 1 1 15 44 32
+59 2 2 1 1 18 42 31
+60 2 2 1 1 30 43 23
+61 2 2 1 1 35 48 16
+62 2 2 1 1 31 54 17
+63 2 2 1 1 9 39 8
+64 2 2 1 1 8 38 7
+65 2 2 1 1 7 25 6
+66 2 2 1 1 22 43 28
+67 2 2 1 1 23 43 22
+68 2 2 1 1 39 54 31
+69 2 2 1 1 19 42 18
+70 2 2 1 1 24 55 30
+71 2 2 1 1 27 42 19
+72 2 2 1 1 13 46 14
+73 2 2 1 1 51 53 37
+74 2 2 1 1 39 52 38
+75 2 2 1 1 6 40 5
+76 2 2 1 1 34 52 50
+77 2 2 1 1 12 45 13
+78 2 2 1 1 30 55 46
+79 2 2 1 1 10 47 11
+80 2 2 1 1 8 39 38
+81 2 2 1 1 28 49 21
+82 2 2 1 1 7 38 25
+83 2 2 1 1 41 49 28
+84 2 2 1 1 20 49 27
+85 2 2 1 1 11 26 12
+86 2 2 1 1 27 49 41
+87 2 2 1 1 31 52 39
+88 2 2 1 1 25 40 6
+89 2 2 1 1 2 54 9
+90 2 2 1 1 14 55 4
+91 2 2 1 1 45 53 46
+92 2 2 1 1 45 46 13
+93 2 2 1 1 5 44 1
+94 2 2 1 1 21 49 20
+95 2 2 1 1 46 53 30
+96 2 2 1 1 3 48 10
+97 2 2 1 1 34 50 33
+98 2 2 1 1 36 51 37
+99 2 2 1 1 26 45 12
+100 2 2 1 1 11 47 26
+101 2 2 1 1 27 50 42
+102 2 2 1 1 40 44 5
+103 2 2 1 1 43 51 28
+104 2 2 1 1 10 48 47
+105 2 2 1 1 9 54 39
+106 2 2 1 1 46 55 14
+107 2 2 1 1 1 44 15
+108 2 2 1 1 16 48 3
+$EndElements
+$Periodic
+1
+1 1 2
+Affine 0.5000000000000001 0.8660254037844386 0 0 -0.8660254037844386 0.5000000000000001 0 0 0 0 1 0 0 0 0 1
+7
+9 14
+6 11
+8 13
+5 10
+7 12
+2 4
+1 3
+$EndPeriodic

data/periodic-torus-sector.geo

@@ -0,0 +1,25 @@
+SetFactory("OpenCASCADE");
+
+R = 1.5;
+r = 0.5;
+
+Torus(1) = {0,0,0, R, r, Pi/3};
+
+pts() = PointsOf{ Volume{1}; };
+
+Characteristic Length{ pts() } = 0.25;
+
+// Set a rotation periodicity constraint:
+Periodic Surface{3} = {2} Rotate{{0,0,1}, {0,0,0}, Pi/3};
+
+// Tag surfaces and volumes with positive integers
+Physical Surface(1) = {1};
+Physical Surface(2) = {2};
+Physical Surface(3) = {3};
+Physical Volume(1) = {1};
+
+// Generate 3D mesh
+Mesh 3;
+
+Mesh.MshFileVersion = 2.2;
+Save "periodic-torus-sector.msh";

data/square-disc-nurbs-patch.mesh

@@ -0,0 +1,155 @@
+MFEM NURBS mesh v1.0
+
+dimension
+2
+
+elements
+5
+1 3 0 3 7 4
+1 3 3 2 6 7
+1 3 2 1 5 6
+1 3 1 0 4 5
+1 3 2 8 9 1
+
+boundary
+10
+1 1 0 3
+2 1 3 2
+2 1 1 0
+2 1 2 8
+2 1 9 1
+3 1 7 4
+3 1 6 7
+3 1 5 6
+3 1 4 5
+4 1 8 9
+
+edges
+15
+0 0 4
+0 3 7
+0 1 5
+0 2 6
+1 0 3
+1 4 7
+2 3 2
+2 7 6
+2 1 0
+2 5 4
+1 2 1
+1 6 5
+1 8 9
+3 2 8
+3 1 9
+
+vertices
+10
+
+patches
+
+knotvectors
+2
+2 3 0 0 0 1 1 1
+2 4 0 0 0 0.5 1 1 1
+
+dimension
+2
+
+controlpoints_cartesian
+-5 5 1
+-5 3.92523e-16 1
+-5 -5 1
+-2.47593 2.47593 1
+-4.95187 6.06429e-16 0.707107
+-2.47593 -2.47593 1
+-0.424264 0.424264 1
+-0.848528 1.03915e-16 0.707107
+-0.424264 -0.424264 1
+-0.353553 0.353553 1
+-0.707107 8.65956e-17 0.707107
+-0.353553 -0.353553 1
+
+knotvectors
+2
+2 3 0 0 0 1 1 1
+2 4 0 0 0 0.5 1 1 1
+
+dimension
+2
+
+controlpoints_cartesian
+-5 -5 1
+-1.17757e-15 -5 1
+5 -5 1
+-2.47593 -2.47593 1
+-9.09644e-16 -4.95187 0.707107
+2.47593 -2.47593 1
+-0.424264 -0.424264 1
+-1.55872e-16 -0.848528 0.707107
+0.424264 -0.424264 1
+-0.353553 -0.353553 1
+-1.29893e-16 -0.707107 0.707107
+0.353553 -0.353553 1
+
+knotvectors
+2
+2 3 0 0 0 1 1 1
+2 4 0 0 0 0.5 1 1 1
+
+dimension
+2
+
+controlpoints_cartesian
+5 -5 1
+5 -1.17757e-15 1
+5 5 1
+2.47593 -2.47593 1
+4.95187 -1.21286e-15 0.707107
+2.47593 2.47593 1
+0.424264 -0.424264 1
+0.848528 -2.07829e-16 0.707107
+0.424264 0.424264 1
+0.353553 -0.353553 1
+0.707107 -1.73191e-16 0.707107
+0.353553 0.353553 1
+
+knotvectors
+2
+2 3 0 0 0 1 1 1
+2 4 0 0 0 0.5 1 1 1
+
+dimension
+2
+
+controlpoints_cartesian
+5 5 1
+3.92523e-16 5 1
+-5 5 1
+2.47593 2.47593 1
+3.03215e-16 4.95187 0.707107
+-2.47593 2.47593 1
+0.424264 0.424264 1
+5.19574e-17 0.848528 0.707107
+-0.424264 0.424264 1
+0.353553 0.353553 1
+4.32978e-17 0.707107 0.707107
+-0.353553 0.353553 1
+
+knotvectors
+2
+2 3 0 0 0 1 1 1
+2 3 0 0 0 1 1 1
+
+dimension
+2
+
+controlpoints_cartesian
+5 -5 1
+10 -5 1
+15 -5 1
+5 0 1
+10 0 1
+15 0 1
+5 5 1
+10 5 1
+15 5 1

doc/CMakeLists.txt

@@ -16,36 +16,21 @@ if (DOXYGEN_FOUND)
   configure_file(${CMAKE_CURRENT_SOURCE_DIR}/CodeDocumentation.conf.in
     ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.conf @ONLY)
 
-  if (UNIX)
-    # Only create symlinks if UNIX operating system
-    add_custom_target(doc
-      COMMAND ${DOXYGEN_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.conf
-      COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.html
-      COMMAND ${CMAKE_COMMAND} -E create_symlink
-      ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation/html/index.html
-      ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.html
-      BYPRODUCTS ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation/html/index.html
-      WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
-      COMMENT "Generating API documentation with Doxygen to CodeDocumentation.html"
-      VERBATIM)
 
-    add_custom_target(clean-doc
-      COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.html
-      COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation
-      COMMENT "Removing API documentation"
-      VERBATIM)
+  add_custom_target(doc
+    COMMAND ${DOXYGEN_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.conf
+    COMMAND echo "<meta http-equiv=\"REFRESH\" content=\"0;URL=CodeDocumentation/html/index.html\">" > ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.html
+    BYPRODUCTS ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation/html/index.html
+    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
+    COMMENT "Generating API documentation with Doxygen to CodeDocumentation.html"
+    VERBATIM)
+
+  add_custom_target(clean-doc
+    COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.html
+    COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_CURRENT_BINARY_DIR}/warnings.log
+    COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation
+    COMMENT "Removing API documentation"
+    VERBATIM)
 
-  else (UNIX)
-    add_custom_target(doc
-      COMMAND ${DOXYGEN_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation.conf
-      BYPRODUCTS ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation/html/index.html
-      WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
-      COMMENT "Generating API documentation with Doxygen to CodeDocumentation/html/index.html"
-      VERBATIM)
 
-    add_custom_target(clean-doc
-      COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_CURRENT_BINARY_DIR}/CodeDocumentation
-      COMMENT "Removing API documentation"
-      VERBATIM)
-  endif (UNIX)
 endif (DOXYGEN_FOUND)

doc/CodeDocumentation.conf.in

@@ -51,7 +51,7 @@ PROJECT_BRIEF          = "Finite element discretization library"
 # pixels and the maximum width should not exceed 200 pixels. Doxygen will copy
 # the logo to the output directory.
 
-PROJECT_LOGO           =
+PROJECT_LOGO           = web/logo-small.png
 
 # The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path
 # into which the generated documentation will be written. If a relative path is
@@ -746,7 +746,7 @@ WARN_FORMAT            = "$file:$line: $text"
 # messages should be written. If left blank the output is written to standard
 # error (stderr).
 
-WARN_LOGFILE           =
+WARN_LOGFILE           = warnings.log
 
 #---------------------------------------------------------------------------
 # Configuration options related to the input files
@@ -1470,7 +1470,7 @@ MATHJAX_FORMAT         = HTML-CSS
 # The default value is: http://cdn.mathjax.org/mathjax/latest.
 # This tag requires that the tag USE_MATHJAX is set to YES.
 
-MATHJAX_RELPATH        = https://cdn.llnl.gov/mathjax/2.7.2
+MATHJAX_RELPATH        = http://cdn.mathjax.org/mathjax/latest
 
 # The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax
 # extension names that should be enabled during MathJax rendering. For example

doc/CodeDocumentation.dox

@@ -149,6 +149,7 @@ namespace mfem {
  * - <a class="el" href="extruder_8cpp_source.html">Extruder</a>: extrude a low-dimensional mesh into a higher dimension
  * - <a class="el" href="mesh-explorer_8cpp_source.html">Mesh Explorer</a>: visualize and manipulate meshes
  * - <a class="el" href="mesh-optimizer_8cpp_source.html">Mesh Optimizer</a>: optimize high-order meshes, <a class="el" href="mesh-optimizer_8cpp_source.html">serial</a> and <a class="el" href="pmesh-optimizer_8cpp_source.html">parallel</a> versions
+ * - <a class="el" href="trimmer_8cpp_source.html">Trimmer</a>: trim elements from existing meshes
  * - <a class="el" href="display-basis_8cpp_source.html">Display Basis</a>: visualize finite element basis functions
  * - <a class="el" href="get-values_8cpp_source.html">Get Values</a>: extract field values via DataCollection classes
  * - <a class="el" href="load-dc_8cpp_source.html">Load DC</a>: visualize fields saved via DataCollection classes

doc/makefile

@@ -9,18 +9,25 @@
 # terms of the BSD-3 license. We welcome feedback and contributions, see file
 # CONTRIBUTING.md for details.
 
+SHELL = /bin/bash
 MFEM_DIR ?= ..
 DOXYGEN_CONF = CodeDocumentation.conf
 
+
 # doxygen uses: graphviz, latex
 html: $(DOXYGEN_CONF)
-	doxygen $(DOXYGEN_CONF)
-	rm -f CodeDocumentation.html
-	ln -s CodeDocumentation/html/index.html CodeDocumentation.html
+	@# Generate the html documentation
+	@doxygen $(DOXYGEN_CONF)
+	@echo "<meta http-equiv=\"REFRESH\" content=\"0;URL=CodeDocumentation/html/index.html\">" > CodeDocumentation.html
+	@cat warnings.log
+	@# Generate the log of undocumented methods
+	@( cat $(DOXYGEN_CONF) ; echo "GENERATE_HTML=NO" ; echo "EXTRACT_ALL=NO" ; echo "WARN_LOGFILE=undoc.log" ; echo "QUIET=YES" ) | doxygen - &> /dev/null
 
 clean:
 	rm -rf $(DOXYGEN_CONF) CodeDocumentation CodeDocumentation.html *~
+	rm -rf  undoc.log warnings.log
 
 $(DOXYGEN_CONF): $(MFEM_DIR)/doc/$(DOXYGEN_CONF).in
-	sed -e 's%@MFEM_SOURCE_DIR@%$(MFEM_DIR)%g' $(<) \
+	@sed -e 's%@MFEM_SOURCE_DIR@%$(MFEM_DIR)%g' $(<) \
 	  > $(DOXYGEN_CONF)
+

examples/CMakeLists.txt

@@ -34,6 +34,8 @@ list(APPEND ALL_EXE_SRCS
   ex25.cpp
   ex26.cpp
   ex27.cpp
+  ex51.cpp
+  ex71.cpp
   )
 
 if (MFEM_USE_MPI)
@@ -64,6 +66,7 @@ if (MFEM_USE_MPI)
     ex25p.cpp
     ex26p.cpp
     ex27p.cpp
+    ex71p.cpp
     )
 endif()
 

examples/ex1.cpp

@@ -9,6 +9,8 @@
 //               ex1 -m ../data/fichera.mesh
 //               ex1 -m ../data/fichera-mixed.mesh
 //               ex1 -m ../data/toroid-wedge.mesh
+//               ex1 -m ../data/periodic-annulus-sector.msh
+//               ex1 -m ../data/periodic-torus-sector.msh
 //               ex1 -m ../data/square-disc-p2.vtk -o 2
 //               ex1 -m ../data/square-disc-p3.mesh -o 3
 //               ex1 -m ../data/square-disc-nurbs.mesh -o -1

examples/ex11p.cpp

@@ -8,6 +8,8 @@
 //               mpirun -np 4 ex11p -m ../data/escher.mesh
 //               mpirun -np 4 ex11p -m ../data/fichera.mesh
 //               mpirun -np 4 ex11p -m ../data/fichera-mixed.mesh
+//               mpirun -np 4 ex11p -m ../data/periodic-annulus-sector.msh
+//               mpirun -np 4 ex11p -m ../data/periodic-torus-sector.msh -rs 1
 //               mpirun -np 4 ex11p -m ../data/toroid-wedge.mesh -o 2
 //               mpirun -np 4 ex11p -m ../data/square-disc-p2.vtk -o 2
 //               mpirun -np 4 ex11p -m ../data/square-disc-p3.mesh -o 3

examples/ex14p.cpp

@@ -35,6 +35,38 @@
 using namespace std;
 using namespace mfem;
 
+class CustomSolverMonitor : public IterativeSolverMonitor
+{
+public:
+   CustomSolverMonitor(const ParMesh *m,
+                       ParGridFunction *f) :
+      pmesh(m),
+      pgf(f) {}
+
+   void MonitorSolution(int i, double norm, const Vector &x, bool final)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      int  num_procs, myid;
+
+      MPI_Comm_size(pmesh->GetComm(),&num_procs);
+      MPI_Comm_rank(pmesh->GetComm(),&myid);
+
+      pgf->SetFromTrueDofs(x);
+
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << *pgf
+               << "window_title 'Iteration no " << i << "'"
+               << "keys rRjlc\n" << flush;
+   }
+
+private:
+   const ParMesh *pmesh;
+   ParGridFunction *pgf;
+};
+
 int main(int argc, char *argv[])
 {
    // 1. Initialize MPI.
@@ -188,6 +220,7 @@ int main(int argc, char *argv[])
    }
    else
    {
+      CustomSolverMonitor monitor(pmesh, &x);
       GMRESSolver gmres(MPI_COMM_WORLD);
       gmres.SetAbsTol(0.0);
       gmres.SetRelTol(1e-12);
@@ -196,6 +229,7 @@ int main(int argc, char *argv[])
       gmres.SetPrintLevel(1);
       gmres.SetOperator(*A);
       gmres.SetPreconditioner(*amg);
+      gmres.SetMonitor(monitor);
       gmres.Mult(*B, *X);
    }
    delete amg;

examples/ex18.hpp

@@ -88,8 +88,6 @@ private:
    Vector funval2;
    Vector nor;
    Vector fluxN;
-   IntegrationPoint eip1;
-   IntegrationPoint eip2;
 
 public:
    FaceIntegrator(RiemannSolver &rsolver_, const int dim);
@@ -418,27 +416,24 @@ void FaceIntegrator::AssembleFaceVector(const FiniteElement &el1,
    {
       intorder++;
    }
-   const IntegrationRule *ir = &IntRules.Get(Tr.FaceGeom, intorder);
+   const IntegrationRule *ir = &IntRules.Get(Tr.GetGeometryType(), intorder);
 
    for (int i = 0; i < ir->GetNPoints(); i++)
    {
       const IntegrationPoint &ip = ir->IntPoint(i);
 
-      Tr.Loc1.Transform(ip, eip1);
-      Tr.Loc2.Transform(ip, eip2);
+      Tr.SetAllIntPoints(&ip); // set face and element int. points
 
       // Calculate basis functions on both elements at the face
-      el1.CalcShape(eip1, shape1);
-      el2.CalcShape(eip2, shape2);
+      el1.CalcShape(Tr.GetElement1IntPoint(), shape1);
+      el2.CalcShape(Tr.GetElement2IntPoint(), shape2);
 
       // Interpolate elfun at the point
       elfun1_mat.MultTranspose(shape1, funval1);
       elfun2_mat.MultTranspose(shape2, funval2);
 
-      Tr.Face->SetIntPoint(&ip);
-
       // Get the normal vector and the flux on the face
-      CalcOrtho(Tr.Face->Jacobian(), nor);
+      CalcOrtho(Tr.Jacobian(), nor);
       const double mcs = rsolver.Eval(funval1, funval2, nor, fluxN);
 
       // Update max char speed

examples/ex19.cpp

@@ -38,6 +38,42 @@
 using namespace std;
 using namespace mfem;
 
+class GeneralResidualMonitor : public IterativeSolverMonitor
+{
+public:
+   GeneralResidualMonitor(const std::string& prefix_, int print_lvl)
+      : prefix(prefix_)
+   {
+      print_level = print_lvl;
+   }
+
+   virtual void MonitorResidual(int it, double norm, const Vector &r, bool final);
+
+private:
+   const std::string prefix;
+   int print_level;
+   mutable double norm0;
+};
+
+void GeneralResidualMonitor::MonitorResidual(int it, double norm,
+                                             const Vector &r, bool final)
+{
+   if (print_level == 1 || (print_level == 3 && (final || it == 0)))
+   {
+      mfem::out << prefix << " iteration " << setw(2) << it
+                << " : ||r|| = " << norm;
+      if (it > 0)
+      {
+         mfem::out << ",  ||r||/||r_0|| = " << norm/norm0;
+      }
+      else
+      {
+         norm0 = norm;
+      }
+      mfem::out << '\n';
+   }
+}
+
 // Custom block preconditioner for the Jacobian of the incompressible nonlinear
 // elasticity operator. It has the form
 //
@@ -103,9 +139,11 @@ protected:
 
    // Newton solver for the hyperelastic operator
    NewtonSolver newton_solver;
+   GeneralResidualMonitor newton_monitor;
 
    // Solver for the Jacobian solve in the Newton method
    Solver *j_solver;
+   GeneralResidualMonitor j_monitor;
 
    // Preconditioner for the Jacobian
    Solver *j_prec;
@@ -410,7 +448,8 @@ RubberOperator::RubberOperator(Array<FiniteElementSpace *> &fes,
                                int iter,
                                Coefficient &c_mu)
    : Operator(fes[0]->GetVSize() + fes[1]->GetVSize()),
-     newton_solver(), mu(c_mu), block_offsets(offsets)
+     newton_solver(), newton_monitor("Newton", 1),
+     j_monitor("  GMRES", 3), mu(c_mu), block_offsets(offsets)
 {
    Array<Vector *> rhs(2);
    rhs = NULL; // Set all entries in the array
@@ -446,7 +485,8 @@ RubberOperator::RubberOperator(Array<FiniteElementSpace *> &fes,
    j_gmres->SetRelTol(1e-12);
    j_gmres->SetAbsTol(1e-12);
    j_gmres->SetMaxIter(300);
-   j_gmres->SetPrintLevel(0);
+   j_gmres->SetPrintLevel(-1);
+   j_gmres->SetMonitor(j_monitor);
    j_gmres->SetPreconditioner(*j_prec);
    j_solver = j_gmres;
 
@@ -454,7 +494,8 @@ RubberOperator::RubberOperator(Array<FiniteElementSpace *> &fes,
    newton_solver.iterative_mode = true;
    newton_solver.SetSolver(*j_solver);
    newton_solver.SetOperator(*this);
-   newton_solver.SetPrintLevel(1);
+   newton_solver.SetPrintLevel(-1);
+   newton_solver.SetMonitor(newton_monitor);
    newton_solver.SetRelTol(rel_tol);
    newton_solver.SetAbsTol(abs_tol);
    newton_solver.SetMaxIter(iter);

examples/ex19p.cpp

@@ -38,6 +38,56 @@
 using namespace std;
 using namespace mfem;
 
+class GeneralResidualMonitor : public IterativeSolverMonitor
+{
+public:
+   GeneralResidualMonitor(MPI_Comm comm, const std::string& prefix_,
+                          int print_lvl)
+      : prefix(prefix_)
+   {
+#ifndef MFEM_USE_MPI
+      print_level = print_lvl;
+#else
+      int rank;
+      MPI_Comm_rank(comm, &rank);
+      if (rank == 0)
+      {
+         print_level = print_lvl;
+      }
+      else
+      {
+         print_level = -1;
+      }
+#endif
+   }
+
+   virtual void MonitorResidual(int it, double norm, const Vector &r, bool final);
+
+private:
+   const std::string prefix;
+   int print_level;
+   mutable double norm0;
+};
+
+void GeneralResidualMonitor::MonitorResidual(int it, double norm,
+                                             const Vector &r, bool final)
+{
+   if (print_level == 1 || (print_level == 3 && (final || it == 0)))
+   {
+      mfem::out << prefix << " iteration " << setw(2) << it
+                << " : ||r|| = " << norm;
+      if (it > 0)
+      {
+         mfem::out << ",  ||r||/||r_0|| = " << norm/norm0;
+      }
+      else
+      {
+         norm0 = norm;
+      }
+      mfem::out << '\n';
+   }
+}
+
 // Custom block preconditioner for the Jacobian of the incompressible nonlinear
 // elasticity operator. It has the form
 //
@@ -103,9 +153,11 @@ protected:
 
    // Newton solver for the hyperelastic operator
    NewtonSolver newton_solver;
+   GeneralResidualMonitor newton_monitor;
 
    // Solver for the Jacobian solve in the Newton method
    Solver *j_solver;
+   GeneralResidualMonitor j_monitor;
 
    // Preconditioner for the Jacobian
    Solver *j_prec;
@@ -459,7 +511,10 @@ RubberOperator::RubberOperator(Array<ParFiniteElementSpace *> &fes,
                                int iter,
                                Coefficient &c_mu)
    : Operator(fes[0]->TrueVSize() + fes[1]->TrueVSize()),
-     newton_solver(fes[0]->GetComm()), mu(c_mu), block_trueOffsets(trueOffsets)
+     newton_solver(fes[0]->GetComm()),
+     newton_monitor(fes[0]->GetComm(), "Newton", 1),
+     j_monitor(fes[0]->GetComm(), "  GMRES", 3),
+     mu(c_mu), block_trueOffsets(trueOffsets)
 {
    Array<Vector *> rhs(2);
    rhs = NULL; // Set all entries in the array
@@ -499,7 +554,8 @@ RubberOperator::RubberOperator(Array<ParFiniteElementSpace *> &fes,
    j_gmres->SetRelTol(1e-12);
    j_gmres->SetAbsTol(1e-12);
    j_gmres->SetMaxIter(300);
-   j_gmres->SetPrintLevel(0);
+   j_gmres->SetPrintLevel(-1);
+   j_gmres->SetMonitor(j_monitor);
    j_gmres->SetPreconditioner(*j_prec);
    j_solver = j_gmres;
 
@@ -507,7 +563,8 @@ RubberOperator::RubberOperator(Array<ParFiniteElementSpace *> &fes,
    newton_solver.iterative_mode = true;
    newton_solver.SetSolver(*j_solver);
    newton_solver.SetOperator(*this);
-   newton_solver.SetPrintLevel(1);
+   newton_solver.SetPrintLevel(-1);
+   newton_solver.SetMonitor(newton_monitor);
    newton_solver.SetRelTol(rel_tol);
    newton_solver.SetAbsTol(abs_tol);
    newton_solver.SetMaxIter(iter);

examples/ex1p.cpp

@@ -9,6 +9,8 @@
 //               mpirun -np 4 ex1p -m ../data/fichera.mesh
 //               mpirun -np 4 ex1p -m ../data/fichera-mixed.mesh
 //               mpirun -np 4 ex1p -m ../data/toroid-wedge.mesh
+//               mpirun -np 4 ex1p -m ../data/periodic-annulus-sector.msh
+//               mpirun -np 4 ex1p -m ../data/periodic-torus-sector.msh
 //               mpirun -np 4 ex1p -m ../data/square-disc-p2.vtk -o 2
 //               mpirun -np 4 ex1p -m ../data/square-disc-p3.mesh -o 3
 //               mpirun -np 4 ex1p -m ../data/square-disc-nurbs.mesh -o -1

examples/ex24.cpp

@@ -6,6 +6,7 @@
 //               ex24 -m ../data/square-disc.mesh -o 2
 //               ex24 -m ../data/beam-tet.mesh
 //               ex24 -m ../data/beam-hex.mesh -o 2 -pa
+//               ex24 -m ../data/beam-hex.mesh -o 2 -pa -p 1
 //               ex24 -m ../data/escher.mesh
 //               ex24 -m ../data/escher.mesh -o 2
 //               ex24 -m ../data/fichera.mesh
@@ -23,11 +24,15 @@
 //               ex24 -m ../data/beam-hex.mesh -pa -d cuda
 //
 // Description:  This example code illustrates usage of mixed finite element
-//               spaces. Using two different approaches, we project a gradient
-//               of a function in H^1 to H(curl). Other spaces and example
-//               computations are to be added in the future.
+//               spaces, with two variants:
 //
-//               We recommend viewing examples 1 and 3 before viewing this
+//               1) (grad p, u) for p in H^1 tested against u in H(curl)
+//               2) (div v, q) for v in H(div) tested against q in L_2
+//
+//               Using different approaches, we project the gradient or
+//               divergence to the appropriate space.
+//
+//               We recommend viewing examples 1, 3, and 5 before viewing this
 //               example.
 
 #include "mfem.hpp"
@@ -39,6 +44,7 @@ using namespace mfem;
 
 double p_exact(const Vector &x);
 void gradp_exact(const Vector &, Vector &);
+double div_gradp_exact(const Vector &x);
 
 int dim;
 
@@ -47,6 +53,7 @@ int main(int argc, char *argv[])
    // 1. Parse command-line options.
    const char *mesh_file = "../data/beam-hex.mesh";
    int order = 1;
+   int prob = 0;
    bool static_cond = false;
    bool pa = false;
    const char *device_config = "cpu";
@@ -57,6 +64,8 @@ int main(int argc, char *argv[])
                   "Mesh file to use.");
    args.AddOption(&order, "-o", "--order",
                   "Finite element order (polynomial degree).");
+   args.AddOption(&prob, "-p", "--problem-type",
+                  "Choose between 0: H(Curl) or 1: H(Div)");
    args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
                   "--no-static-condensation", "Enable static condensation.");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
@@ -100,72 +109,107 @@ int main(int argc, char *argv[])
    }
    mesh->ReorientTetMesh();
 
-   // 5. Define a parallel finite element space on the parallel mesh. Here we
-   //    use the Nedelec finite elements of the specified order.
-   FiniteElementCollection *fec = new ND_FECollection(order, dim);
-   FiniteElementCollection *H1fec = new H1_FECollection(order, dim);
-   FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
-   FiniteElementSpace *H1fespace = new FiniteElementSpace(mesh, H1fec);
+   // 5. Define a finite element space on the mesh. Here we use Nedelec or
+   //    Raviart-Thomas finite elements of the specified order.
+   FiniteElementCollection *trial_fec = NULL;
+   FiniteElementCollection *test_fec = NULL;
 
-   int size = fespace->GetTrueVSize();
-   int H1size = H1fespace->GetTrueVSize();
-   cout << "Number of Nedelec finite element unknowns: " << size << endl;
-   cout << "Number of H1 finite element unknowns: " << H1size << endl;
-
-   // 6. Define the solution vector x as a parallel finite element grid function
-   //    corresponding to fespace. Initialize x by projecting the exact
-   //    solution. Note that only values from the boundary edges will be used
-   //    when eliminating the non-homogeneous boundary condition to modify the
-   //    r.h.s. vector b.
-   GridFunction x(fespace);
-   FunctionCoefficient p_coef(p_exact);
-   GridFunction p(H1fespace);
-   p.ProjectCoefficient(p_coef);
-   p.SetTrueVector();
-   p.SetFromTrueVector();
-
-   VectorFunctionCoefficient gradp_coef(sdim, gradp_exact);
-
-   // 7. Set up the bilinear forms.
-   Coefficient *muinv = new ConstantCoefficient(1.0);
-   Coefficient *sigma = new ConstantCoefficient(1.0);
-   BilinearForm *a = new BilinearForm(fespace);
-   MixedBilinearForm *a_NDH1 = new MixedBilinearForm(H1fespace, fespace);
-   if (pa)
+   if (prob == 0)
    {
-      a->SetAssemblyLevel(AssemblyLevel::PARTIAL);
-      a_NDH1->SetAssemblyLevel(AssemblyLevel::PARTIAL);
-   }
-
-   // First approach: L2 projection
-   a->AddDomainIntegrator(new VectorFEMassIntegrator(*sigma));
-   a_NDH1->AddDomainIntegrator(new MixedVectorGradientIntegrator(*muinv));
-
-   // 8. Assemble the parallel bilinear form and the corresponding linear
-   //    system, applying any necessary transformations such as: parallel
-   //    assembly, eliminating boundary conditions, applying conforming
-   //    constraints for non-conforming AMR, static condensation, etc.
-   if (static_cond) { a->EnableStaticCondensation(); }
-
-   a->Assemble();
-   if (!pa) { a->Finalize(); }
-
-   a_NDH1->Assemble();
-   if (!pa) { a_NDH1->Finalize(); }
-
-   if (pa)
-   {
-      a_NDH1->Mult(p, x);
+      trial_fec = new H1_FECollection(order, dim);
+      test_fec = new ND_FECollection(order, dim);
    }
    else
    {
-      SparseMatrix& NDH1 = a_NDH1->SpMat();
-      NDH1.Mult(p, x);
+      trial_fec = new RT_FECollection(order - 1, dim);
+      test_fec = new L2_FECollection(order - 1, dim);
+   }
+
+   FiniteElementSpace trial_fes(mesh, trial_fec);
+   FiniteElementSpace test_fes(mesh, test_fec);
+
+   int trial_size = trial_fes.GetTrueVSize();
+   int test_size = test_fes.GetTrueVSize();
+
+   if (prob == 0)
+   {
+      cout << "Number of Nedelec finite element unknowns: " << test_size << endl;
+      cout << "Number of H1 finite element unknowns: " << trial_size << endl;
+   }
+   else
+   {
+      cout << "Number of Raviart-Thomas finite element unknowns: "
+           << trial_size << endl;
+      cout << "Number of L2 finite element unknowns: " << test_size << endl;
+   }
+
+   // 6. Define the solution vector as a finite element grid function
+   //    corresponding to the trial fespace.
+   GridFunction gftest(&test_fes);
+   GridFunction gftrial(&trial_fes);
+   GridFunction x(&test_fes);
+   FunctionCoefficient p_coef(p_exact);
+   VectorFunctionCoefficient gradp_coef(sdim, gradp_exact);
+   FunctionCoefficient divgradp_coef(div_gradp_exact);
+
+   if (prob == 0)
+   {
+      gftrial.ProjectCoefficient(p_coef);
+   }
+   else
+   {
+      gftrial.ProjectCoefficient(gradp_coef);
+   }
+
+   gftrial.SetTrueVector();
+   gftrial.SetFromTrueVector();
+
+   // 7. Set up the bilinear forms for L2 projection.
+   ConstantCoefficient one(1.0);
+   BilinearForm a(&test_fes);
+   MixedBilinearForm a_mixed(&trial_fes, &test_fes);
+   if (pa)
+   {
+      a.SetAssemblyLevel(AssemblyLevel::PARTIAL);
+      a_mixed.SetAssemblyLevel(AssemblyLevel::PARTIAL);
+   }
+
+   if (prob == 0)
+   {
+      a.AddDomainIntegrator(new VectorFEMassIntegrator(one));
+      a_mixed.AddDomainIntegrator(new MixedVectorGradientIntegrator(one));
+   }
+   else
+   {
+      a.AddDomainIntegrator(new MassIntegrator(one));
+      a_mixed.AddDomainIntegrator(new VectorFEDivergenceIntegrator(one));
+   }
+
+   // 8. Assemble the bilinear form and the corresponding linear system,
+   //    applying any necessary transformations such as: eliminating boundary
+   //    conditions, applying conforming constraints for non-conforming AMR,
+   //    static condensation, etc.
+   if (static_cond) { a.EnableStaticCondensation(); }
+
+   a.Assemble();
+   if (!pa) { a.Finalize(); }
+
+   a_mixed.Assemble();
+   if (!pa) { a_mixed.Finalize(); }
+
+   if (pa)
+   {
+      a_mixed.Mult(gftrial, x);
+   }
+   else
+   {
+      SparseMatrix& mixed = a_mixed.SpMat();
+      mixed.Mult(gftrial, x);
    }
 
    // 9. Define and apply a PCG solver for Ax = b with Jacobi preconditioner.
    {
-      GridFunction rhs(fespace);
+      GridFunction rhs(&test_fes);
       rhs = x;
       x = 0.0;
 
@@ -176,15 +220,15 @@ int main(int argc, char *argv[])
       if (pa)
       {
          Array<int> ess_tdof_list; // empty
-         OperatorJacobiSmoother Jacobi(*a, ess_tdof_list);
+         OperatorJacobiSmoother Jacobi(a, ess_tdof_list);
 
-         cg.SetOperator(*a);
+         cg.SetOperator(a);
          cg.SetPreconditioner(Jacobi);
          cg.Mult(rhs, x);
       }
       else
       {
-         SparseMatrix& Amat = a->SpMat();
+         SparseMatrix& Amat = a.SpMat();
          DSmoother Jacobi(Amat);
 
          cg.SetOperator(Amat);
@@ -193,33 +237,68 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 10. Second approach: compute the same solution by applying
-   //     GradientInterpolator in H(curl).
-   DiscreteLinearOperator grad(H1fespace, fespace);
-   grad.AddDomainInterpolator(new GradientInterpolator());
-   grad.Assemble();
+   // 10. Compute the same field by applying a DiscreteInterpolator.
+   GridFunction discreteInterpolant(&test_fes);
+   DiscreteLinearOperator dlo(&trial_fes, &test_fes);
+   if (prob == 0)
+   {
+      dlo.AddDomainInterpolator(new GradientInterpolator());
+   }
+   else
+   {
+      dlo.AddDomainInterpolator(new DivergenceInterpolator());
+   }
 
-   GridFunction gradp(fespace);
-   grad.Mult(p, gradp);
+   dlo.Assemble();
+   dlo.Mult(gftrial, discreteInterpolant);
 
-   // 11. Compute the projection of the exact grad p.
-   GridFunction exact_gradp(fespace);
-   exact_gradp.ProjectCoefficient(gradp_coef);
-   exact_gradp.SetTrueVector();
-   exact_gradp.SetFromTrueVector();
+   // 11. Compute the projection of the exact field.
+   GridFunction exact_proj(&test_fes);
+   if (prob == 0)
+   {
+      exact_proj.ProjectCoefficient(gradp_coef);
+   }
+   else
+   {
+      exact_proj.ProjectCoefficient(divgradp_coef);
+   }
 
-   // 12. Compute and print the L^2 norm of the error.
+   exact_proj.SetTrueVector();
+   exact_proj.SetFromTrueVector();
+
+   // 12. Compute and print the L_2 norm of the error.
+   if (prob == 0)
    {
       double errSol = x.ComputeL2Error(gradp_coef);
-      double errInterp = gradp.ComputeL2Error(gradp_coef);
-      double errProj = exact_gradp.ComputeL2Error(gradp_coef);
+      double errInterp = discreteInterpolant.ComputeL2Error(gradp_coef);
+      double errProj = exact_proj.ComputeL2Error(gradp_coef);
 
       cout << "\n Solution of (E_h,v) = (grad p_h,v) for E_h and v in H(curl): "
-           "|| E_h - grad p ||_{L^2} = " << errSol << '\n' << endl;
+           "|| E_h - grad p ||_{L_2} = " << errSol << '\n' << endl;
       cout << " Gradient interpolant E_h = grad p_h in H(curl): || E_h - grad p"
-           "||_{L^2} = " << errInterp << '\n' << endl;
+           "||_{L_2} = " << errInterp << '\n' << endl;
       cout << " Projection E_h of exact grad p in H(curl): || E_h - grad p "
-           "||_{L^2} = " << errProj << '\n' << endl;
+           "||_{L_2} = " << errProj << '\n' << endl;
+   }
+   else
+   {
+      int order_quad = max(2, 2*order+1);
+      const IntegrationRule *irs[Geometry::NumGeom];
+      for (int i=0; i < Geometry::NumGeom; ++i)
+      {
+         irs[i] = &(IntRules.Get(i, order_quad));
+      }
+
+      double errSol = x.ComputeL2Error(divgradp_coef, irs);
+      double errInterp = discreteInterpolant.ComputeL2Error(divgradp_coef, irs);
+      double errProj = exact_proj.ComputeL2Error(divgradp_coef, irs);
+
+      cout << "\n Solution of (f_h,q) = (div v_h,q) for f_h and q in L_2: "
+           "|| f_h - div v ||_{L_2} = " << errSol << '\n' << endl;
+      cout << " Divergence interpolant f_h = div v_h in L_2: || f_h - div v"
+           "||_{L_2} = " << errInterp << '\n' << endl;
+      cout << " Projection f_h of exact div v in L_2: || f_h - div v "
+           "||_{L_2} = " << errProj << '\n' << endl;
    }
 
    // 13. Save the refined mesh and the solution. This output can be viewed
@@ -242,14 +321,8 @@ int main(int argc, char *argv[])
    }
 
    // 15. Free the used memory.
-   delete a;
-   delete a_NDH1;
-   delete sigma;
-   delete muinv;
-   delete fespace;
-   delete H1fespace;
-   delete fec;
-   delete H1fec;
+   delete trial_fec;
+   delete test_fec;
    delete mesh;
 
    return 0;
@@ -284,3 +357,17 @@ void gradp_exact(const Vector &x, Vector &f)
       if (x.Size() == 3) { f(2) = 0.0; }
    }
 }
+
+double div_gradp_exact(const Vector &x)
+{
+   if (dim == 3)
+   {
+      return -3.0 * sin(x(0)) * sin(x(1)) * sin(x(2));
+   }
+   else if (dim == 2)
+   {
+      return -2.0 * sin(x(0)) * sin(x(1));
+   }
+
+   return 0.0;
+}

examples/ex24p.cpp

@@ -6,6 +6,7 @@
 //               mpirun -np 4 ex24p -m ../data/square-disc.mesh -o 2
 //               mpirun -np 4 ex24p -m ../data/beam-tet.mesh
 //               mpirun -np 4 ex24p -m ../data/beam-hex.mesh -o 2 -pa
+//               mpirun -np 4 ex24p -m ../data/beam-hex.mesh -o 2 -p 1 -pa
 //               mpirun -np 4 ex24p -m ../data/escher.mesh
 //               mpirun -np 4 ex24p -m ../data/escher.mesh -o 2
 //               mpirun -np 4 ex24p -m ../data/fichera.mesh
@@ -23,11 +24,15 @@
 //               mpirun -np 4 ex24p -m ../data/beam-hex.mesh -pa -d cuda
 //
 // Description:  This example code illustrates usage of mixed finite element
-//               spaces. Using two different approaches, we project a gradient
-//               of a function in H^1 to H(curl). Other spaces and example
-//               computations are to be added in the future.
+//               spaces, with two variants:
 //
-//               We recommend viewing examples 1 and 3 before viewing this
+//               1) (grad p, u) for p in H^1 tested against u in H(curl)
+//               2) (div v, q) for v in H(div) tested against q in L_2
+//
+//               Using different approaches, we project the gradient or
+//               divergence to the appropriate space.
+//
+//               We recommend viewing examples 1, 3, and 5 before viewing this
 //               example.
 
 #include "mfem.hpp"
@@ -39,6 +44,7 @@ using namespace mfem;
 
 double p_exact(const Vector &x);
 void gradp_exact(const Vector &, Vector &);
+double div_gradp_exact(const Vector &x);
 
 int dim;
 
@@ -53,6 +59,7 @@ int main(int argc, char *argv[])
    // 2. Parse command-line options.
    const char *mesh_file = "../data/beam-hex.mesh";
    int order = 1;
+   int prob = 0;
    bool static_cond = false;
    bool pa = false;
    const char *device_config = "cpu";
@@ -63,6 +70,8 @@ int main(int argc, char *argv[])
                   "Mesh file to use.");
    args.AddOption(&order, "-o", "--order",
                   "Finite element order (polynomial degree).");
+   args.AddOption(&prob, "-p", "--problem-type",
+                  "Choose between 0: H(Curl) or 1: H(Div)");
    args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
                   "--no-static-condensation", "Enable static condensation.");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
@@ -129,80 +138,115 @@ int main(int argc, char *argv[])
    pmesh->ReorientTetMesh();
 
    // 7. Define a parallel finite element space on the parallel mesh. Here we
-   //    use the Nedelec finite elements of the specified order.
-   FiniteElementCollection *fec = new ND_FECollection(order, dim);
-   FiniteElementCollection *H1fec = new H1_FECollection(order, dim);
-   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
-   ParFiniteElementSpace *H1fespace = new ParFiniteElementSpace(pmesh, H1fec);
-   HYPRE_Int size = fespace->GlobalTrueVSize();
-   HYPRE_Int H1size = H1fespace->GlobalTrueVSize();
+   //    use Nedelec or Raviart-Thomas finite elements of the specified order.
+   FiniteElementCollection *trial_fec = NULL;
+   FiniteElementCollection *test_fec = NULL;
+
+   if (prob == 0)
+   {
+      trial_fec = new H1_FECollection(order, dim);
+      test_fec = new ND_FECollection(order, dim);
+   }
+   else
+   {
+      trial_fec = new RT_FECollection(order - 1, dim);
+      test_fec = new L2_FECollection(order - 1, dim);
+   }
+
+   ParFiniteElementSpace trial_fes(pmesh, trial_fec);
+   ParFiniteElementSpace test_fes(pmesh, test_fec);
+
+   HYPRE_Int trial_size = trial_fes.GlobalTrueVSize();
+   HYPRE_Int test_size = test_fes.GlobalTrueVSize();
+
    if (myid == 0)
    {
-      cout << "Number of Nedelec finite element unknowns: " << size << endl;
-      cout << "Number of H1 finite element unknowns: " << H1size << endl;
+      if (prob == 0)
+      {
+         cout << "Number of Nedelec finite element unknowns: " << test_size << endl;
+         cout << "Number of H1 finite element unknowns: " << trial_size << endl;
+      }
+      else
+      {
+         cout << "Number of Raviart-Thomas finite element unknowns: "
+              << trial_size << endl;
+         cout << "Number of L2 finite element unknowns: " << test_size << endl;
+      }
    }
 
-   // 8. Define the solution vector x as a parallel finite element grid function
-   //    corresponding to fespace. Initialize x by projecting the exact
-   //    solution. Note that only values from the boundary edges will be used
-   //    when eliminating the non-homogeneous boundary condition to modify the
-   //    r.h.s. vector b.
-   ParGridFunction x(fespace);
+   // 8. Define the solution vector as a parallel finite element grid function
+   //    corresponding to the trial fespace.
+   ParGridFunction gftest(&test_fes);
+   ParGridFunction gftrial(&trial_fes);
+   ParGridFunction x(&test_fes);
    FunctionCoefficient p_coef(p_exact);
-   ParGridFunction p(H1fespace);
-   p.ProjectCoefficient(p_coef);
-   p.SetTrueVector();
-   p.SetFromTrueVector();
-
    VectorFunctionCoefficient gradp_coef(sdim, gradp_exact);
+   FunctionCoefficient divgradp_coef(div_gradp_exact);
 
-   // 9. Set up the parallel bilinear forms.
-   Coefficient *muinv = new ConstantCoefficient(1.0);
-   Coefficient *sigma = new ConstantCoefficient(1.0);
-   ParBilinearForm *a = new ParBilinearForm(fespace);
-   ParMixedBilinearForm *a_NDH1 = new ParMixedBilinearForm(H1fespace, fespace);
+   if (prob == 0)
+   {
+      gftrial.ProjectCoefficient(p_coef);
+   }
+   else
+   {
+      gftrial.ProjectCoefficient(gradp_coef);
+   }
+
+   gftrial.SetTrueVector();
+   gftrial.SetFromTrueVector();
+
+   // 9. Set up the parallel bilinear forms for L2 projection.
+   ConstantCoefficient one(1.0);
+   ParBilinearForm a(&test_fes);
+   ParMixedBilinearForm a_mixed(&trial_fes, &test_fes);
    if (pa)
    {
-      a->SetAssemblyLevel(AssemblyLevel::PARTIAL);
-      a_NDH1->SetAssemblyLevel(AssemblyLevel::PARTIAL);
+      a.SetAssemblyLevel(AssemblyLevel::PARTIAL);
+      a_mixed.SetAssemblyLevel(AssemblyLevel::PARTIAL);
    }
 
-   // First approach: L2 projection
-   a->AddDomainIntegrator(new VectorFEMassIntegrator(*sigma));
-   a_NDH1->AddDomainIntegrator(new MixedVectorGradientIntegrator(*muinv));
+   if (prob == 0)
+   {
+      a.AddDomainIntegrator(new VectorFEMassIntegrator(one));
+      a_mixed.AddDomainIntegrator(new MixedVectorGradientIntegrator(one));
+   }
+   else
+   {
+      a.AddDomainIntegrator(new MassIntegrator(one));
+      a_mixed.AddDomainIntegrator(new VectorFEDivergenceIntegrator(one));
+   }
 
    // 10. Assemble the parallel bilinear form and the corresponding linear
    //     system, applying any necessary transformations such as: parallel
    //     assembly, eliminating boundary conditions, applying conforming
    //     constraints for non-conforming AMR, static condensation, etc.
-   if (static_cond) { a->EnableStaticCondensation(); }
+   if (static_cond) { a.EnableStaticCondensation(); }
 
-   a->Assemble();
-   if (!pa) { a->Finalize(); }
+   a.Assemble();
+   if (!pa) { a.Finalize(); }
 
-   a_NDH1->Assemble();
-   if (!pa) { a_NDH1->Finalize(); }
+   a_mixed.Assemble();
+   if (!pa) { a_mixed.Finalize(); }
 
-   Vector B(fespace->GetTrueVSize());
-   Vector X(fespace->GetTrueVSize());
+   Vector B(test_fes.GetTrueVSize());
+   Vector X(test_fes.GetTrueVSize());
 
    if (pa)
    {
-      ParLinearForm *b = new ParLinearForm(fespace); // used as a vector
-      a_NDH1->Mult(p, *b); // process-local multiplication
-      b->ParallelAssemble(B);
-      delete b;
+      ParLinearForm b(&test_fes); // used as a vector
+      a_mixed.Mult(gftrial, b); // process-local multiplication
+      b.ParallelAssemble(B);
    }
    else
    {
-      HypreParMatrix *NDH1 = a_NDH1->ParallelAssemble();
+      HypreParMatrix *mixed = a_mixed.ParallelAssemble();
 
-      Vector P(H1fespace->GetTrueVSize());
-      p.GetTrueDofs(P);
+      Vector P(trial_fes.GetTrueVSize());
+      gftrial.GetTrueDofs(P);
 
-      NDH1->Mult(P,B);
+      mixed->Mult(P,B);
 
-      delete NDH1;
+      delete mixed;
    }
 
    // 11. Define and apply a parallel PCG solver for AX=B with Jacobi
@@ -212,9 +256,9 @@ int main(int argc, char *argv[])
       Array<int> ess_tdof_list; // empty
 
       OperatorPtr A;
-      a->FormSystemMatrix(ess_tdof_list, A);
+      a.FormSystemMatrix(ess_tdof_list, A);
 
-      OperatorJacobiSmoother Jacobi(*a, ess_tdof_list);
+      OperatorJacobiSmoother Jacobi(a, ess_tdof_list);
 
       CGSolver cg(MPI_COMM_WORLD);
       cg.SetRelTol(1e-12);
@@ -227,7 +271,7 @@ int main(int argc, char *argv[])
    }
    else
    {
-      HypreParMatrix *Amat = a->ParallelAssemble();
+      HypreParMatrix *Amat = a.ParallelAssemble();
       HypreDiagScale Jacobi(*Amat);
       HyprePCG pcg(*Amat);
       pcg.SetTol(1e-12);
@@ -242,35 +286,73 @@ int main(int argc, char *argv[])
 
    x.SetFromTrueDofs(X);
 
-   // 12. Second approach: compute the same solution by applying
-   //     GradientInterpolator in H(curl).
-   ParDiscreteLinearOperator grad(H1fespace, fespace);
-   grad.AddDomainInterpolator(new GradientInterpolator());
-   grad.Assemble();
+   // 12. Compute the same field by applying a DiscreteInterpolator.
+   ParGridFunction discreteInterpolant(&test_fes);
+   ParDiscreteLinearOperator dlo(&trial_fes, &test_fes);
+   if (prob == 0)
+   {
+      dlo.AddDomainInterpolator(new GradientInterpolator());
+   }
+   else
+   {
+      dlo.AddDomainInterpolator(new DivergenceInterpolator());
+   }
 
-   ParGridFunction gradp(fespace);
-   grad.Mult(p, gradp);
+   dlo.Assemble();
+   dlo.Mult(gftrial, discreteInterpolant);
 
-   // 13. Compute the projection of the exact grad p.
-   ParGridFunction exact_gradp(fespace);
-   exact_gradp.ProjectCoefficient(gradp_coef);
-   exact_gradp.SetTrueVector();
-   exact_gradp.SetFromTrueVector();
+   // 13. Compute the projection of the exact field.
+   ParGridFunction exact_proj(&test_fes);
+   if (prob == 0)
+   {
+      exact_proj.ProjectCoefficient(gradp_coef);
+   }
+   else
+   {
+      exact_proj.ProjectCoefficient(divgradp_coef);
+   }
 
-   // 14. Compute and print the L^2 norm of the error.
+   exact_proj.SetTrueVector();
+   exact_proj.SetFromTrueVector();
+
+   // 14. Compute and print the L_2 norm of the error.
+   if (prob == 0)
    {
       double errSol = x.ComputeL2Error(gradp_coef);
-      double errInterp = gradp.ComputeL2Error(gradp_coef);
-      double errProj = exact_gradp.ComputeL2Error(gradp_coef);
+      double errInterp = discreteInterpolant.ComputeL2Error(gradp_coef);
+      double errProj = exact_proj.ComputeL2Error(gradp_coef);
 
       if (myid == 0)
       {
-         cout << "\n Solution of (E_h,v) = (grad p_h,v) for E_h and v in "
-              "H(curl): || E_h - grad p ||_{L^2} = " << errSol << '\n' << endl;
-         cout << " Gradient interpolant E_h = grad p_h in H(curl): || E_h - "
-              "grad p ||_{L^2} = " << errInterp << '\n' << endl;
+         cout << "\n Solution of (E_h,v) = (grad p_h,v) for E_h and v in H(curl): "
+              "|| E_h - grad p ||_{L_2} = " << errSol << '\n' << endl;
+         cout << " Gradient interpolant E_h = grad p_h in H(curl): || E_h - grad p"
+              "||_{L_2} = " << errInterp << '\n' << endl;
          cout << " Projection E_h of exact grad p in H(curl): || E_h - grad p "
-              "||_{L^2} = " << errProj << '\n' << endl;
+              "||_{L_2} = " << errProj << '\n' << endl;
+      }
+   }
+   else
+   {
+      int order_quad = max(2, 2*order+1);
+      const IntegrationRule *irs[Geometry::NumGeom];
+      for (int i=0; i < Geometry::NumGeom; ++i)
+      {
+         irs[i] = &(IntRules.Get(i, order_quad));
+      }
+
+      double errSol = x.ComputeL2Error(divgradp_coef, irs);
+      double errInterp = discreteInterpolant.ComputeL2Error(divgradp_coef, irs);
+      double errProj = exact_proj.ComputeL2Error(divgradp_coef, irs);
+
+      if (myid == 0)
+      {
+         cout << "\n Solution of (f_h,q) = (div v_h,q) for f_h and q in L_2: "
+              "|| f_h - div v ||_{L_2} = " << errSol << '\n' << endl;
+         cout << " Divergence interpolant f_h = div v_h in L_2: || f_h - div v"
+              "||_{L_2} = " << errInterp << '\n' << endl;
+         cout << " Projection f_h of exact div v in L_2: || f_h - div v "
+              "||_{L_2} = " << errProj << '\n' << endl;
       }
    }
 
@@ -302,14 +384,8 @@ int main(int argc, char *argv[])
    }
 
    // 17. Free the used memory.
-   delete a;
-   delete a_NDH1;
-   delete sigma;
-   delete muinv;
-   delete fespace;
-   delete H1fespace;
-   delete fec;
-   delete H1fec;
+   delete trial_fec;
+   delete test_fec;
    delete pmesh;
 
    MPI_Finalize();
@@ -346,3 +422,17 @@ void gradp_exact(const Vector &x, Vector &f)
       if (x.Size() == 3) { f(2) = 0.0; }
    }
 }
+
+double div_gradp_exact(const Vector &x)
+{
+   if (dim == 3)
+   {
+      return -3.0 * sin(x(0)) * sin(x(1)) * sin(x(2));
+   }
+   else if (dim == 2)
+   {
+      return -2.0 * sin(x(0)) * sin(x(1));
+   }
+
+   return 0.0;
+}

examples/ex4.cpp

@@ -6,6 +6,7 @@
 //               ex4 -m ../data/star.mesh
 //               ex4 -m ../data/beam-tet.mesh
 //               ex4 -m ../data/beam-hex.mesh
+//               ex4 -m ../data/beam-hex.mesh -o 2 -pa
 //               ex4 -m ../data/escher.mesh
 //               ex4 -m ../data/fichera.mesh -o 2 -hb
 //               ex4 -m ../data/fichera-q2.vtk
@@ -20,6 +21,12 @@
 //               ex4 -m ../data/fichera-amr.mesh -o 2 -sc
 //               ex4 -m ../data/star-surf.mesh -o 1
 //
+// Device sample runs:
+//               ex4 -m ../data/star.mesh -pa -d cuda
+//               ex4 -m ../data/star.mesh -pa -d raja-cuda
+//               ex4 -m ../data/star.mesh -pa -d raja-omp
+//               ex4 -m ../data/beam-hex.mesh -pa -d cuda
+//
 // Description:  This example code solves a simple 2D/3D H(div) diffusion
 //               problem corresponding to the second order definite equation
 //               -grad(alpha div F) + beta F = f with boundary condition F dot n
@@ -55,6 +62,8 @@ int main(int argc, char *argv[])
    bool set_bc = true;
    bool static_cond = false;
    bool hybridization = false;
+   bool pa = false;
+   const char *device_config = "cpu";
    bool visualization = 1;
 
    OptionsParser args(argc, argv);
@@ -70,6 +79,10 @@ int main(int argc, char *argv[])
                   "--no-static-condensation", "Enable static condensation.");
    args.AddOption(&hybridization, "-hb", "--hybridization", "-no-hb",
                   "--no-hybridization", "Enable hybridization.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -82,14 +95,19 @@ int main(int argc, char *argv[])
    args.PrintOptions(cout);
    kappa = freq * M_PI;
 
-   // 2. Read the mesh from the given mesh file. We can handle triangular,
+   // 2. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   device.Print();
+
+   // 3. Read the mesh from the given mesh file. We can handle triangular,
    //    quadrilateral, tetrahedral, hexahedral, surface and volume, as well as
    //    periodic meshes with the same code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
    int sdim = mesh->SpaceDimension();
 
-   // 3. Refine the mesh to increase the resolution. In this example we do
+   // 4. Refine the mesh to increase the resolution. In this example we do
    //    'ref_levels' of uniform refinement. We choose 'ref_levels' to be the
    //    largest number that gives a final mesh with no more than 25,000
    //    elements.
@@ -102,14 +120,14 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 4. Define a finite element space on the mesh. Here we use the
+   // 5. Define a finite element space on the mesh. Here we use the
    //    Raviart-Thomas finite elements of the specified order.
    FiniteElementCollection *fec = new RT_FECollection(order-1, dim);
    FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
    cout << "Number of finite element unknowns: "
         << fespace->GetTrueVSize() << endl;
 
-   // 5. Determine the list of true (i.e. conforming) essential boundary dofs.
+   // 6. Determine the list of true (i.e. conforming) essential boundary dofs.
    //    In this example, the boundary conditions are defined by marking all
    //    the boundary attributes from the mesh as essential (Dirichlet) and
    //    converting them to a list of true dofs.
@@ -121,7 +139,7 @@ int main(int argc, char *argv[])
       fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
    }
 
-   // 6. Set up the linear form b(.) which corresponds to the right-hand side
+   // 7. Set up the linear form b(.) which corresponds to the right-hand side
    //    of the FEM linear system, which in this case is (f,phi_i) where f is
    //    given by the function f_exact and phi_i are the basis functions in the
    //    finite element fespace.
@@ -130,7 +148,7 @@ int main(int argc, char *argv[])
    b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
    b->Assemble();
 
-   // 7. Define the solution vector x as a finite element grid function
+   // 8. Define the solution vector x as a finite element grid function
    //    corresponding to fespace. Initialize x by projecting the exact
    //    solution. Note that only values from the boundary faces will be used
    //    when eliminating the non-homogeneous boundary condition to modify the
@@ -139,16 +157,17 @@ int main(int argc, char *argv[])
    VectorFunctionCoefficient F(sdim, F_exact);
    x.ProjectCoefficient(F);
 
-   // 8. Set up the bilinear form corresponding to the H(div) diffusion operator
+   // 9. Set up the bilinear form corresponding to the H(div) diffusion operator
    //    grad alpha div + beta I, by adding the div-div and the mass domain
    //    integrators.
    Coefficient *alpha = new ConstantCoefficient(1.0);
    Coefficient *beta  = new ConstantCoefficient(1.0);
    BilinearForm *a = new BilinearForm(fespace);
+   if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    a->AddDomainIntegrator(new DivDivIntegrator(*alpha));
    a->AddDomainIntegrator(new VectorFEMassIntegrator(*beta));
 
-   // 9. Assemble the bilinear form and the corresponding linear system,
+   // 10. Assemble the bilinear form and the corresponding linear system,
    //    applying any necessary transformations such as: eliminating boundary
    //    conditions, applying conforming constraints for non-conforming AMR,
    //    static condensation, hybridization, etc.
@@ -167,32 +186,47 @@ int main(int argc, char *argv[])
    }
    a->Assemble();
 
-   SparseMatrix A;
+   OperatorPtr A;
    Vector B, X;
    a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
 
-   cout << "Size of linear system: " << A.Height() << endl;
+   cout << "Size of linear system: " << A->Height() << endl;
 
+   // 11. Solve the linear system A X = B.
+   if (!pa)
+   {
 #ifndef MFEM_USE_SUITESPARSE
-   // 10. Define a simple symmetric Gauss-Seidel preconditioner and use it to
-   //     solve the system A X = B with PCG.
-   GSSmoother M(A);
-   PCG(A, M, B, X, 1, 10000, 1e-20, 0.0);
+      // Use a simple symmetric Gauss-Seidel preconditioner with PCG.
+      GSSmoother M((SparseMatrix&)(*A));
+      PCG(*A, M, B, X, 1, 10000, 1e-20, 0.0);
 #else
-   // 10. If compiled with SuiteSparse support, use UMFPACK to solve the system.
-   UMFPackSolver umf_solver;
-   umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
-   umf_solver.SetOperator(A);
-   umf_solver.Mult(B, X);
+      // If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.
+      UMFPackSolver umf_solver;
+      umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+      umf_solver.SetOperator(*A);
+      umf_solver.Mult(B, X);
 #endif
+   }
+   else // Jacobi preconditioning in partial assembly mode
+   {
+      if (UsesTensorBasis(*fespace))
+      {
+         OperatorJacobiSmoother M(*a, ess_tdof_list);
+         PCG(*A, M, B, X, 1, 10000, 1e-20, 0.0);
+      }
+      else
+      {
+         CG(*A, B, X, 1, 10000, 1e-20, 0.0);
+      }
+   }
 
-   // 11. Recover the solution as a finite element grid function.
+   // 12. Recover the solution as a finite element grid function.
    a->RecoverFEMSolution(X, *b, x);
 
-   // 12. Compute and print the L^2 norm of the error.
+   // 13. Compute and print the L^2 norm of the error.
    cout << "\n|| F_h - F ||_{L^2} = " << x.ComputeL2Error(F) << '\n' << endl;
 
-   // 13. Save the refined mesh and the solution. This output can be viewed
+   // 14. Save the refined mesh and the solution. This output can be viewed
    //     later using GLVis: "glvis -m refined.mesh -g sol.gf".
    {
       ofstream mesh_ofs("refined.mesh");
@@ -203,7 +237,7 @@ int main(int argc, char *argv[])
       x.Save(sol_ofs);
    }
 
-   // 14. Send the solution by socket to a GLVis server.
+   // 15. Send the solution by socket to a GLVis server.
    if (visualization)
    {
       char vishost[] = "localhost";
@@ -213,7 +247,7 @@ int main(int argc, char *argv[])
       sol_sock << "solution\n" << *mesh << x << flush;
    }
 
-   // 15. Free the used memory.
+   // 16. Free the used memory.
    delete hfes;
    delete hfec;
    delete a;
@@ -235,7 +269,7 @@ void F_exact(const Vector &p, Vector &F)
 
    double x = p(0);
    double y = p(1);
-   // double z = (dim == 3) ? p(2) : 0.0;
+   // double z = (dim == 3) ? p(2) : 0.0; // Uncomment if F is changed to depend on z
 
    F(0) = cos(kappa*x)*sin(kappa*y);
    F(1) = cos(kappa*y)*sin(kappa*x);
@@ -252,7 +286,7 @@ void f_exact(const Vector &p, Vector &f)
 
    double x = p(0);
    double y = p(1);
-   // double z = (dim == 3) ? p(2) : 0.0;
+   // double z = (dim == 3) ? p(2) : 0.0; // Uncomment if f is changed to depend on z
 
    double temp = 1 + 2*kappa*kappa;
 

examples/ex4p.cpp

@@ -6,6 +6,7 @@
 //               mpirun -np 4 ex4p -m ../data/star.mesh
 //               mpirun -np 4 ex4p -m ../data/beam-tet.mesh
 //               mpirun -np 4 ex4p -m ../data/beam-hex.mesh
+//               mpirun -np 4 ex4p -m ../data/beam-hex.mesh -o 2 -pa
 //               mpirun -np 4 ex4p -m ../data/escher.mesh -o 2 -sc
 //               mpirun -np 4 ex4p -m ../data/fichera.mesh -o 2 -hb
 //               mpirun -np 4 ex4p -m ../data/fichera-q2.vtk
@@ -15,10 +16,17 @@
 //               mpirun -np 4 ex4p -m ../data/periodic-square.mesh -no-bc
 //               mpirun -np 4 ex4p -m ../data/periodic-cube.mesh -no-bc
 //               mpirun -np 4 ex4p -m ../data/amr-quad.mesh
+//               mpirun -np 3 ex4p -m ../data/amr-quad.mesh -o 2 -hb
 //               mpirun -np 4 ex4p -m ../data/amr-hex.mesh -o 2 -sc
 //               mpirun -np 4 ex4p -m ../data/amr-hex.mesh -o 2 -hb
 //               mpirun -np 4 ex4p -m ../data/star-surf.mesh -o 3 -hb
 //
+// Device sample runs:
+//               mpirun -np 4 ex4p -m ../data/star.mesh -pa -d cuda
+//               mpirun -np 4 ex4p -m ../data/star.mesh -pa -d raja-cuda
+//               mpirun -np 4 ex4p -m ../data/star.mesh -pa -d raja-omp
+//               mpirun -np 4 ex4p -m ../data/beam-hex.mesh -pa -d cuda
+//
 // Description:  This example code solves a simple 2D/3D H(div) diffusion
 //               problem corresponding to the second order definite equation
 //               -grad(alpha div F) + beta F = f with boundary condition F dot n
@@ -60,6 +68,8 @@ int main(int argc, char *argv[])
    bool set_bc = true;
    bool static_cond = false;
    bool hybridization = false;
+   bool pa = false;
+   const char *device_config = "cpu";
    bool visualization = 1;
 
    OptionsParser args(argc, argv);
@@ -75,6 +85,10 @@ int main(int argc, char *argv[])
                   "--no-static-condensation", "Enable static condensation.");
    args.AddOption(&hybridization, "-hb", "--hybridization", "-no-hb",
                   "--no-hybridization", "Enable hybridization.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -94,14 +108,19 @@ int main(int argc, char *argv[])
    }
    kappa = freq * M_PI;
 
-   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   // 3. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   if (myid == 0) { device.Print(); }
+
+   // 4. Read the (serial) mesh from the given mesh file on all processors.  We
    //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
    //    and volume, as well as periodic meshes with the same code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
    int sdim = mesh->SpaceDimension();
 
-   // 4. Refine the serial mesh on all processors to increase the resolution. In
+   // 5. Refine the serial mesh on all processors to increase the resolution. In
    //    this example we do 'ref_levels' of uniform refinement. We choose
    //    'ref_levels' to be the largest number that gives a final mesh with no
    //    more than 1,000 elements.
@@ -114,7 +133,7 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   // 6. Define a parallel mesh by a partitioning of the serial mesh. Refine
    //    this mesh further in parallel to increase the resolution. Once the
    //    parallel mesh is defined, the serial mesh can be deleted. Tetrahedral
    //    meshes need to be reoriented before we can define high-order Nedelec
@@ -130,7 +149,7 @@ int main(int argc, char *argv[])
    }
    pmesh->ReorientTetMesh();
 
-   // 6. Define a parallel finite element space on the parallel mesh. Here we
+   // 7. Define a parallel finite element space on the parallel mesh. Here we
    //    use the Raviart-Thomas finite elements of the specified order.
    FiniteElementCollection *fec = new RT_FECollection(order-1, dim);
    ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
@@ -140,7 +159,7 @@ int main(int argc, char *argv[])
       cout << "Number of finite element unknowns: " << size << endl;
    }
 
-   // 7. Determine the list of true (i.e. parallel conforming) essential
+   // 8. Determine the list of true (i.e. parallel conforming) essential
    //    boundary dofs. In this example, the boundary conditions are defined
    //    by marking all the boundary attributes from the mesh as essential
    //    (Dirichlet) and converting them to a list of true dofs.
@@ -152,7 +171,7 @@ int main(int argc, char *argv[])
       fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
    }
 
-   // 8. Set up the parallel linear form b(.) which corresponds to the
+   // 9. Set up the parallel linear form b(.) which corresponds to the
    //    right-hand side of the FEM linear system, which in this case is
    //    (f,phi_i) where f is given by the function f_exact and phi_i are the
    //    basis functions in the finite element fespace.
@@ -161,7 +180,7 @@ int main(int argc, char *argv[])
    b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
    b->Assemble();
 
-   // 9. Define the solution vector x as a parallel finite element grid function
+   // 10. Define the solution vector x as a parallel finite element grid function
    //    corresponding to fespace. Initialize x by projecting the exact
    //    solution. Note that only values from the boundary faces will be used
    //    when eliminating the non-homogeneous boundary condition to modify the
@@ -170,16 +189,17 @@ int main(int argc, char *argv[])
    VectorFunctionCoefficient F(sdim, F_exact);
    x.ProjectCoefficient(F);
 
-   // 10. Set up the parallel bilinear form corresponding to the H(div)
+   // 11. Set up the parallel bilinear form corresponding to the H(div)
    //     diffusion operator grad alpha div + beta I, by adding the div-div and
    //     the mass domain integrators.
    Coefficient *alpha = new ConstantCoefficient(1.0);
    Coefficient *beta  = new ConstantCoefficient(1.0);
    ParBilinearForm *a = new ParBilinearForm(fespace);
+   if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    a->AddDomainIntegrator(new DivDivIntegrator(*alpha));
    a->AddDomainIntegrator(new VectorFEMassIntegrator(*beta));
 
-   // 11. Assemble the parallel bilinear form and the corresponding linear
+   // 12. Assemble the parallel bilinear form and the corresponding linear
    //     system, applying any necessary transformations such as: parallel
    //     assembly, eliminating boundary conditions, applying conforming
    //     constraints for non-conforming AMR, static condensation,
@@ -199,41 +219,43 @@ int main(int argc, char *argv[])
    }
    a->Assemble();
 
-   HypreParMatrix A;
+   OperatorPtr A;
    Vector B, X;
    a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
 
-   HYPRE_Int glob_size = A.GetGlobalNumRows();
-   if (myid == 0)
+   if (myid == 0 && !pa)
    {
-      cout << "Size of linear system: " << glob_size << endl;
+      cout << "Size of linear system: "
+           << A.As<HypreParMatrix>()->GetGlobalNumRows() << endl;
    }
 
-   // 12. Define and apply a parallel PCG solver for A X = B with the 2D AMS or
+   // 13. Define and apply a parallel PCG solver for A X = B with the 2D AMS or
    //     the 3D ADS preconditioners from hypre. If using hybridization, the
-   //     system is preconditioned with hypre's BoomerAMG.
-   HypreSolver *prec = NULL;
-   CGSolver *pcg = new CGSolver(A.GetComm());
-   pcg->SetOperator(A);
+   //     system is preconditioned with hypre's BoomerAMG. In the partial
+   //     assembly case, use Jacobi preconditioning.
+   Solver *prec = NULL;
+   CGSolver *pcg = new CGSolver(MPI_COMM_WORLD);
+   pcg->SetOperator(*A);
    pcg->SetRelTol(1e-12);
-   pcg->SetMaxIter(500);
+   pcg->SetMaxIter(2000);
    pcg->SetPrintLevel(1);
-   if (hybridization) { prec = new HypreBoomerAMG(A); }
+   if (hybridization) { prec = new HypreBoomerAMG(*A.As<HypreParMatrix>()); }
+   else if (pa) { prec = new OperatorJacobiSmoother(*a, ess_tdof_list); }
    else
    {
       ParFiniteElementSpace *prec_fespace =
          (a->StaticCondensationIsEnabled() ? a->SCParFESpace() : fespace);
-      if (dim == 2)   { prec = new HypreAMS(A, prec_fespace); }
-      else            { prec = new HypreADS(A, prec_fespace); }
+      if (dim == 2)   { prec = new HypreAMS(*A.As<HypreParMatrix>(), prec_fespace); }
+      else            { prec = new HypreADS(*A.As<HypreParMatrix>(), prec_fespace); }
    }
    pcg->SetPreconditioner(*prec);
    pcg->Mult(B, X);
 
-   // 13. Recover the parallel grid function corresponding to X. This is the
+   // 14. Recover the parallel grid function corresponding to X. This is the
    //     local finite element solution on each processor.
    a->RecoverFEMSolution(X, *b, x);
 
-   // 14. Compute and print the L^2 norm of the error.
+   // 15. Compute and print the L^2 norm of the error.
    {
       double err = x.ComputeL2Error(F);
       if (myid == 0)
@@ -242,7 +264,7 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 15. Save the refined mesh and the solution in parallel. This output can
+   // 16. Save the refined mesh and the solution in parallel. This output can
    //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
    {
       ostringstream mesh_name, sol_name;
@@ -258,7 +280,7 @@ int main(int argc, char *argv[])
       x.Save(sol_ofs);
    }
 
-   // 16. Send the solution by socket to a GLVis server.
+   // 17. Send the solution by socket to a GLVis server.
    if (visualization)
    {
       char vishost[] = "localhost";
@@ -269,7 +291,7 @@ int main(int argc, char *argv[])
       sol_sock << "solution\n" << *pmesh << x << flush;
    }
 
-   // 17. Free the used memory.
+   // 18. Free the used memory.
    delete pcg;
    delete prec;
    delete hfes;
@@ -295,7 +317,7 @@ void F_exact(const Vector &p, Vector &F)
 
    double x = p(0);
    double y = p(1);
-   // double z = (dim == 3) ? p(2) : 0.0;
+   // double z = (dim == 3) ? p(2) : 0.0; // Uncomment if F is changed to depend on z
 
    F(0) = cos(kappa*x)*sin(kappa*y);
    F(1) = cos(kappa*y)*sin(kappa*x);
@@ -312,7 +334,7 @@ void f_exact(const Vector &p, Vector &f)
 
    double x = p(0);
    double y = p(1);
-   // double z = (dim == 3) ? p(2) : 0.0;
+   // double z = (dim == 3) ? p(2) : 0.0; // Uncomment if f is changed to depend on z
 
    double temp = 1 + 2*kappa*kappa;
 

examples/ex5.cpp

@@ -4,8 +4,10 @@
 //
 // Sample runs:  ex5 -m ../data/square-disc.mesh
 //               ex5 -m ../data/star.mesh
+//               ex5 -m ../data/star.mesh -pa
 //               ex5 -m ../data/beam-tet.mesh
 //               ex5 -m ../data/beam-hex.mesh
+//               ex5 -m ../data/beam-hex.mesh -pa
 //               ex5 -m ../data/escher.mesh
 //               ex5 -m ../data/fichera.mesh
 //
@@ -47,6 +49,7 @@ int main(int argc, char *argv[])
    // 1. Parse command-line options.
    const char *mesh_file = "../data/star.mesh";
    int order = 1;
+   bool pa = false;
    bool visualization = 1;
 
    OptionsParser args(argc, argv);
@@ -54,6 +57,8 @@ int main(int argc, char *argv[])
                   "Mesh file to use.");
    args.AddOption(&order, "-o", "--order",
                   "Finite element order (polynomial degree).");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -146,22 +151,39 @@ int main(int argc, char *argv[])
    BilinearForm *mVarf(new BilinearForm(R_space));
    MixedBilinearForm *bVarf(new MixedBilinearForm(R_space, W_space));
 
+   if (pa) { mVarf->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    mVarf->AddDomainIntegrator(new VectorFEMassIntegrator(k));
    mVarf->Assemble();
-   mVarf->Finalize();
-   SparseMatrix &M(mVarf->SpMat());
+   if (!pa) { mVarf->Finalize(); }
 
+   if (pa) { bVarf->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    bVarf->AddDomainIntegrator(new VectorFEDivergenceIntegrator);
    bVarf->Assemble();
-   bVarf->Finalize();
-   SparseMatrix & B(bVarf->SpMat());
-   B *= -1.;
-   SparseMatrix *BT = Transpose(B);
+   if (!pa) { bVarf->Finalize(); }
 
-   BlockMatrix darcyMatrix(block_offsets);
-   darcyMatrix.SetBlock(0,0, &M);
-   darcyMatrix.SetBlock(0,1, BT);
-   darcyMatrix.SetBlock(1,0, &B);
+   BlockOperator darcyOp(block_offsets);
+
+   TransposeOperator *Bt = NULL;
+
+   if (pa)
+   {
+      Bt = new TransposeOperator(bVarf);
+
+      darcyOp.SetBlock(0,0, mVarf);
+      darcyOp.SetBlock(0,1, Bt, -1.0);
+      darcyOp.SetBlock(1,0, bVarf, -1.0);
+   }
+   else
+   {
+      SparseMatrix &M(mVarf->SpMat());
+      SparseMatrix &B(bVarf->SpMat());
+      B *= -1.;
+      Bt = new TransposeOperator(&B);
+
+      darcyOp.SetBlock(0,0, &M);
+      darcyOp.SetBlock(0,1, Bt);
+      darcyOp.SetBlock(1,0, &B);
+   }
 
    // 9. Construct the operators for preconditioner
    //
@@ -170,27 +192,57 @@ int main(int argc, char *argv[])
    //
    //     Here we use Symmetric Gauss-Seidel to approximate the inverse of the
    //     pressure Schur Complement
-   SparseMatrix *MinvBt = Transpose(B);
-   Vector Md(M.Height());
-   M.GetDiag(Md);
-   for (int i = 0; i < Md.Size(); i++)
-   {
-      MinvBt->ScaleRow(i, 1./Md(i));
-   }
-   SparseMatrix *S = Mult(B, *MinvBt);
+   SparseMatrix *MinvBt = NULL;
+   Vector Md(mVarf->Height());
 
+   BlockDiagonalPreconditioner darcyPrec(block_offsets);
    Solver *invM, *invS;
-   invM = new DSmoother(M);
+   SparseMatrix *S = NULL;
+
+   if (pa)
+   {
+      mVarf->AssembleDiagonal(Md);
+      Vector invMd(mVarf->Height());
+      for (int i=0; i<mVarf->Height(); ++i)
+      {
+         invMd(i) = 1.0 / Md(i);
+      }
+
+      Vector BMBt_diag(bVarf->Height());
+      bVarf->AssembleDiagonal_ADAt(invMd, BMBt_diag);
+
+      Array<int> ess_tdof_list;  // empty
+
+      invM = new OperatorJacobiSmoother(Md, ess_tdof_list);
+      invS = new OperatorJacobiSmoother(BMBt_diag, ess_tdof_list);
+   }
+   else
+   {
+      SparseMatrix &M(mVarf->SpMat());
+      M.GetDiag(Md);
+
+      SparseMatrix &B(bVarf->SpMat());
+      MinvBt = Transpose(B);
+
+      for (int i = 0; i < Md.Size(); i++)
+      {
+         MinvBt->ScaleRow(i, 1./Md(i));
+      }
+
+      S = Mult(B, *MinvBt);
+
+      invM = new DSmoother(M);
+
 #ifndef MFEM_USE_SUITESPARSE
-   invS = new GSSmoother(*S);
+      invS = new GSSmoother(*S);
 #else
-   invS = new UMFPackSolver(*S);
+      invS = new UMFPackSolver(*S);
 #endif
+   }
 
    invM->iterative_mode = false;
    invS->iterative_mode = false;
 
-   BlockDiagonalPreconditioner darcyPrec(block_offsets);
    darcyPrec.SetDiagonalBlock(0, invM);
    darcyPrec.SetDiagonalBlock(1, invS);
 
@@ -206,7 +258,7 @@ int main(int argc, char *argv[])
    solver.SetAbsTol(atol);
    solver.SetRelTol(rtol);
    solver.SetMaxIter(maxIter);
-   solver.SetOperator(darcyMatrix);
+   solver.SetOperator(darcyOp);
    solver.SetPreconditioner(darcyPrec);
    solver.SetPrintLevel(1);
    x = 0.0;
@@ -295,8 +347,8 @@ int main(int argc, char *argv[])
    delete invM;
    delete invS;
    delete S;
+   delete Bt;
    delete MinvBt;
-   delete BT;
    delete mVarf;
    delete bVarf;
    delete W_space;

examples/ex51.cpp

@@ -0,0 +1,629 @@
+//                                MFEM Example 1
+//
+// Compile with: make ex1
+//
+// Sample runs:  ex1 -m ../data/square-disc.mesh
+//               ex1 -m ../data/star.mesh
+//               ex1 -m ../data/star-mixed.mesh
+//               ex1 -m ../data/escher.mesh
+//               ex1 -m ../data/fichera.mesh
+//               ex1 -m ../data/fichera-mixed.mesh
+//               ex1 -m ../data/toroid-wedge.mesh
+//               ex1 -m ../data/square-disc-p2.vtk -o 2
+//               ex1 -m ../data/square-disc-p3.mesh -o 3
+//               ex1 -m ../data/square-disc-nurbs.mesh -o -1
+//               ex1 -m ../data/star-mixed-p2.mesh -o 2
+//               ex1 -m ../data/disc-nurbs.mesh -o -1
+//               ex1 -m ../data/pipe-nurbs.mesh -o -1
+//               ex1 -m ../data/fichera-mixed-p2.mesh -o 2
+//               ex1 -m ../data/star-surf.mesh
+//               ex1 -m ../data/square-disc-surf.mesh
+//               ex1 -m ../data/inline-segment.mesh
+//               ex1 -m ../data/amr-quad.mesh
+//               ex1 -m ../data/amr-hex.mesh
+//               ex1 -m ../data/fichera-amr.mesh
+//               ex1 -m ../data/mobius-strip.mesh
+//               ex1 -m ../data/mobius-strip.mesh -o -1 -sc
+//
+// Device sample runs:
+//               ex1 -pa -d cuda
+//               ex1 -pa -d raja-cuda
+//               ex1 -pa -d occa-cuda
+//               ex1 -pa -d raja-omp
+//               ex1 -pa -d occa-omp
+//               ex1 -pa -d ceed-cpu
+//               ex1 -pa -d ceed-cuda
+//               ex1 -m ../data/beam-hex.mesh -pa -d cuda
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Laplace problem
+//               -Delta u = 1 with homogeneous Dirichlet boundary conditions.
+//               Specifically, we discretize using a FE space of the specified
+//               order, or if order < 1 using an isoparametric/isogeometric
+//               space (i.e. quadratic for quadratic curvilinear mesh, NURBS for
+//               NURBS mesh, etc.)
+//
+//               The example highlights the use of mesh refinement, finite
+//               element grid functions, as well as linear and bilinear forms
+//               corresponding to the left-hand side and right-hand side of the
+//               discrete linear system. We also cover the explicit elimination
+//               of essential boundary conditions, static condensation, and the
+//               optional connection to the GLVis tool for visualization.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+
+#include "../fem/adnonlininteg.hpp"
+
+
+using namespace std;
+
+namespace mfem{
+    
+class VolNonlinearForm: public NonlinearFormIntegrator
+{
+protected:
+    double eta;
+    double beta;
+public:
+    VolNonlinearForm(double eta_, double beta_){
+        eta=eta_;
+        beta=beta_;}
+    virtual ~VolNonlinearForm(){ }
+    
+    double Project(double inp)
+    {
+        // tanh projection - Wang&Lazarov&Sigmund2011 
+        double a=std::tanh(eta*beta);
+        double b=std::tanh(beta*(1.0-eta));
+        double c=std::tanh(beta*(inp-eta));
+        double rez=(a+c)/(a+b);
+        return rez;
+    }
+
+    double ProjGrad(double inp)
+    {
+        double c=std::tanh(beta*(inp-eta));
+        double a=std::tanh(eta*beta);
+        double b=std::tanh(beta*(1.0-eta));
+        double rez=beta*(1.0-c*c)/(a+b);
+        return rez;
+    }
+
+    
+    double ProjSec(double inp)
+    {
+        double c=std::tanh(beta*(inp-eta));
+        double a=std::tanh(eta*beta);
+        double b=std::tanh(beta*(1.0-eta));
+        double rez=-2.0*beta*beta*c*(1.0-c*c)/(a+b);
+        return rez;
+    }
+    
+    
+    virtual double GetElementEnergy(const mfem::FiniteElement & el,
+                                           mfem::ElementTransformation & trans,
+                                           const mfem::Vector & elfun) override
+    {
+        double energy=0.0;
+        int ndof = el.GetDof();
+        int ndim = el.GetDim();
+        
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+        
+        mfem::Vector shapef(ndof);
+        
+        double w;
+        for (int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            el.CalcShape(ip,shapef);
+            w= Project(shapef*elfun);
+            w= ip.weight * trans.Weight() * w;
+            energy = energy + w;
+        }
+        return energy;
+    }
+    
+    virtual void AssembleElementVector(const mfem::FiniteElement & el,
+                                       mfem::ElementTransformation & trans,
+                                       const mfem::Vector & elfun,
+                                       mfem::Vector & elvect) override
+    {
+        
+        
+        int ndof = el.GetDof();
+    
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+        
+        elvect.SetSize(ndof);
+        elvect=0.0;
+        
+        mfem::Vector shapef(ndof);
+        double w;
+        for (int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            el.CalcShape(ip,shapef);
+            w= ProjGrad(shapef*elfun);
+            w= ip.weight * trans.Weight() * w;
+            elvect.Add(w,shapef);
+        }
+        
+    }
+    
+    
+    virtual void AssembleElementGrad(const mfem::FiniteElement & el,
+                                     mfem::ElementTransformation & trans,
+                                     const mfem::Vector & elfun,
+                                     mfem::DenseMatrix & elmat) override
+    {
+        int ndof = el.GetDof();
+    
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+        
+        elmat.SetSize(ndof);
+        elmat=0.0;
+        
+        mfem::Vector shapef(ndof);
+        double w;
+        for (int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            el.CalcShape(ip,shapef);
+            w= ProjSec(shapef*elfun);
+            w= ip.weight * trans.Weight() * w;
+            AddMult_a_VVt(w, shapef, elmat);
+        }
+        
+   }
+    
+};
+
+
+class VolNonlinearFormADH:public ADNonlinearFormIntegratorH
+{
+private:
+    double eta;
+    double beta;
+    
+    template<typename DType>
+    DType Project(DType inp)
+    {
+        // tanh projection - Wang&Lazarov&Sigmund2011 
+        double a=std::tanh(eta*beta);
+        double b=std::tanh(beta*(1.0-eta));
+        DType c=tanh(beta*(inp-eta));
+        DType rez=(a+c)/(a+b);
+        return rez;
+    }
+    
+public:
+    
+    VolNonlinearFormADH(double eta_, double beta_){
+        eta=eta_;
+        beta=beta_;
+    }
+    
+    
+    virtual ADFType ElementEnergy(const mfem::FiniteElement & el,
+                                       mfem::ElementTransformation & trans,
+                                       const ADFVector & elfun) override
+    {
+        ADFType rez=MyElementEnergy<ADFType,ADFVector>(el,trans,elfun);
+        return rez;
+    }
+    
+    virtual ADSType ElementEnergy(const mfem::FiniteElement & el,
+                                       mfem::ElementTransformation & trans,
+                                       const ADSVector & elfun) override
+    {
+        return MyElementEnergy<ADSType,ADSVector>(el,trans,elfun);
+    }
+    
+    template<typename MDType, typename MVType>
+    MDType MyElementEnergy(const mfem::FiniteElement & el,
+                            mfem::ElementTransformation & trans,
+                            const MVType & elfun)
+    {
+        MDType energy=MDType();
+        int ndof = el.GetDof();
+        
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+        
+        mfem::Vector shapef(ndof);
+        
+        MDType w;
+        for (int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            el.CalcShape(ip,shapef);
+            w= Project(elfun*shapef);
+            w= ip.weight * trans.Weight() * w;
+            energy = energy + w;
+        }
+        return energy;
+    }
+    
+    
+    virtual double ElementEnergy(const mfem::FiniteElement & el, 
+                                 mfem::ElementTransformation & Tr, 
+                                 const mfem::Vector & elfun) override
+    {
+        return GetElementEnergy(el,Tr,elfun);
+    }
+    
+    virtual double GetElementEnergy(const mfem::FiniteElement & el,
+                                           mfem::ElementTransformation & trans,
+                                           const mfem::Vector & elfun) override
+    {
+        double rez;
+        rez=MyElementEnergy<double,mfem::Vector>(el,trans,elfun);
+        return rez;
+    
+    }
+};
+
+
+class VolQIntegratorJ: public ADQIntegratorJ
+{
+  private:
+    template<typename DType>
+    DType Project(double eta, double beta, DType inp)
+    {
+        // tanh projection - Wang&Lazarov&Sigmund2011
+        double a=std::tanh(eta*beta);
+        double b=std::tanh(beta*(1.0-eta));
+        DType c=tanh(beta*(inp-eta));
+        DType rez=(a+c)/(a+b);
+        return rez;
+    }
+
+    template<typename DType>
+    DType ProjGrad(double eta, double beta, DType inp)
+    {
+        DType c=tanh(beta*(inp-eta));
+        DType a=tanh(eta*beta);
+        DType b=tanh(beta*(1.0-eta));
+        DType rez=beta*(1.0-c*c)/(a+b);
+        return rez;
+    }
+
+
+  public:
+    VolQIntegratorJ(){}
+    virtual ~VolQIntegratorJ(){}
+
+
+    template<typename MVType>
+    void MyQIntegratorDU(const mfem::Vector& vparam, MVType& uu, MVType& rr)
+    {
+        //implement all evaluations executed at integration point
+        double eta=vparam[0];
+        double beta=vparam[1];
+        rr.SetSize(1); //return the derivative of the projected value
+        rr[0]=ProjGrad(eta,beta,uu[0]);
+        return;
+    }
+
+    virtual void QIntegratorDU(const mfem::Vector& vparam, mfem::Vector& uu, mfem::Vector& rr) override
+    {
+        MyQIntegratorDU<mfem::Vector>(vparam,uu,rr);
+    }
+
+    virtual void QIntegratorDU(const mfem::Vector& vparam, ADFVector& uu, ADFVector& rr) override
+    {
+        MyQIntegratorDU<ADFVector>(vparam,uu,rr);
+    }
+
+
+    virtual double QIntegrator(const Vector &vparam, const Vector &uu) override
+    {
+        //implement all evaluations executed at integration point
+        double eta=vparam[0];
+        double beta=vparam[1];
+        double rez=Project(eta,beta,uu[0]);
+        return rez;
+    }
+
+};
+
+
+class VolNonlinearFormQJ: public NonlinearFormIntegrator
+{
+protected:
+    double eta;
+    double beta;
+    mfem::Vector vparam;
+    VolQIntegratorJ qint;
+public:
+    VolNonlinearFormQJ(double eta_, double beta_){
+        eta=eta_;
+        beta=beta_;
+        vparam.SetSize(2);
+        vparam[0]=eta;
+        vparam[1]=beta;
+    }
+
+    virtual ~VolNonlinearFormQJ(){ }
+
+    virtual double GetElementEnergy(const mfem::FiniteElement & el,
+                                           mfem::ElementTransformation & trans,
+                                           const mfem::Vector & elfun) override
+    {
+        double energy=0.0;
+        int ndof = el.GetDof();
+
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+
+        mfem::Vector shapef(ndof);
+        mfem::Vector uu(1);
+
+        double w;
+        for (int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            el.CalcShape(ip,shapef);
+            uu[0]= shapef*elfun;
+            w= qint.QIntegrator(vparam,uu);
+            w= ip.weight * trans.Weight() * w;
+            energy = energy + w;
+        }
+        return energy;
+    }
+
+    virtual void AssembleElementVector(const mfem::FiniteElement & el,
+                                       mfem::ElementTransformation & trans,
+                                       const mfem::Vector & elfun,
+                                       mfem::Vector & elvect) override
+    {
+        int ndof = el.GetDof();
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+
+        elvect.SetSize(ndof);
+        elvect=0.0;
+
+        mfem::Vector shapef(ndof);
+        mfem::Vector uu(1);
+        mfem::Vector rr(1);
+
+        double w;
+
+        for (int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            el.CalcShape(ip,shapef);
+            uu[0]=shapef*elfun;
+            qint.QIntegratorDU(vparam,uu,rr);
+            w= ip.weight * trans.Weight() * rr[0];
+            elvect.Add(w,shapef);
+        }
+    }
+
+    virtual void AssembleElementGrad(const mfem::FiniteElement & el,
+                                     mfem::ElementTransformation & trans,
+                                     const mfem::Vector & elfun,
+                                     mfem::DenseMatrix & elmat) override
+    {
+        int ndof = el.GetDof();
+
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+
+        elmat.SetSize(ndof);
+        elmat=0.0;
+
+        mfem::DenseMatrix jac(1,1);
+
+        mfem::Vector shapef(ndof);
+        mfem::Vector uu(1);
+
+
+        double w;
+        for (int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            el.CalcShape(ip,shapef);
+            uu[0]=shapef*elfun;
+            qint.QIntegratorDD(vparam,uu,jac);
+            w= ip.weight * trans.Weight() * jac(0,0);
+            AddMult_a_VVt(w, shapef, elmat);
+        }
+
+   }
+
+};
+
+
+}
+
+
+
+double TFunc(const mfem::Vector& a){
+    double sca=4.0;
+    double rez=(std::sin(sca*a[0])*std::sin(sca*a[1])*std::sin(sca*a[2]))*0.5+0.5;
+    return rez;
+}
+
+
+int main(int argc, char *argv[])
+{
+   
+    // 1. Parse command-line options.
+   const char *mesh_file = "../data/star.mesh";
+   int order = 1;
+   bool static_cond = false;
+   bool pa = false;
+   const char *device_config = "cpu";
+   bool visualization = true;
+
+   mfem::OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   // 2. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   mfem::Device device(device_config);
+   device.Print();
+
+   // 3. Read the mesh from the given mesh file. We can handle triangular,
+   //    quadrilateral, tetrahedral, hexahedral, surface and volume meshes with
+   //    the same code.
+   mfem::Mesh *mesh = new mfem::Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+   
+   
+   // 4. Refine the mesh to increase the resolution. In this example we do
+   //    'ref_levels' of uniform refinement. We choose 'ref_levels' to be the
+   //    largest number that gives a final mesh with no more than 50,000
+   //    elements.
+   
+   {
+      int ref_levels =
+         (int)floor(log(50000./mesh->GetNE())/log(2.)/dim);
+      ref_levels=1;
+      for (int l = 0; l < ref_levels; l++)
+      {
+         mesh->UniformRefinement();
+      }
+   }
+   
+
+   // 5. Define a finite element space on the mesh. Here we use continuous
+   //    Lagrange finite elements of the specified order. If order < 1, we
+   //    instead use an isoparametric/isogeometric space.
+   mfem::FiniteElementCollection *fec;
+   if (order > 0)
+   {
+      fec = new mfem::H1_FECollection(order, dim);
+   }
+   else if (mesh->GetNodes())
+   {
+      fec = mesh->GetNodes()->OwnFEC();
+      cout << "Using isoparametric FEs: " << fec->Name() << endl;
+   }
+   else
+   {
+      fec = new mfem::H1_FECollection(order = 1, dim);
+   }
+   mfem::FiniteElementSpace *fespace = new mfem::FiniteElementSpace(mesh, fec);
+   cout << "Number of finite element unknowns: "
+        << fespace->GetTrueVSize() << endl;
+   
+   mfem::NonlinearForm* nf0=new mfem::NonlinearForm(fespace);
+   mfem::NonlinearForm* nf1=new mfem::NonlinearForm(fespace);
+
+   mfem::FunctionCoefficient ifun(TFunc);
+   //create an input for the NonlinearForm
+   mfem::GridFunction* igf = new mfem::GridFunction(fespace);
+   igf->ProjectCoefficient(ifun);
+   
+   std::cout << "Size of the grid function igf:"<<igf->Size()<<std::endl;
+   
+   
+   mfem::Vector* resv0=new mfem::Vector(fespace->GetTrueVSize());
+   mfem::Vector* resv1=new mfem::Vector(fespace->GetTrueVSize());
+   mfem::Vector* stat=new mfem::Vector(fespace->GetTrueVSize());
+   
+   igf->GetTrueDofs(*stat);
+   
+   
+   //compute the energy - the total volume above 0.5
+   nf0->AddDomainIntegrator(new mfem::VolNonlinearForm(0.5,8.0));
+   //nf1->AddDomainIntegrator(new mfem::VolNonlinearFormADH(0.5,8.0));
+   nf1->AddDomainIntegrator(new mfem::VolNonlinearFormQJ(0.5,8.0));
+   
+   double vol0=nf0->GetEnergy(*stat);
+   double vol1=nf1->GetEnergy(*stat);
+   std::cout<<"The total volume is:("<<vol0<<","<<vol1<<")"<<std::endl;
+   nf0->Mult(*stat,*resv0);
+   nf1->Mult(*stat,*resv1);
+   //project back the gradients to a grid function
+   mfem::GridFunction* ggf0=new mfem::GridFunction(fespace);
+   ggf0->SetFromTrueDofs(*resv0);
+   mfem::GridFunction* ggf1=new mfem::GridFunction(fespace);
+   ggf1->SetFromTrueDofs(*resv1);
+   
+   
+   resv0->Add(-1.0,*resv1);
+   std::cout<<"Norm|v_1-v_0|="<<resv0->Norml2()<<std::endl;
+   
+   mfem::Operator& grad0(nf0->GetGradient(*stat));
+   mfem::SparseMatrix* spmat0=dynamic_cast<mfem::SparseMatrix*>(&grad0);
+   mfem::Operator& grad1(nf1->GetGradient(*stat));
+   mfem::SparseMatrix* spmat1=dynamic_cast<mfem::SparseMatrix*>(&grad1);
+   std::cout<<"Norm mat1="<<spmat0->MaxNorm()<<" mat2="<<spmat1->MaxNorm()<<std::endl;
+   spmat0->Add(-1.0,*spmat1);
+   std::cout<<"Norm diff"<<spmat0->MaxNorm()<<std::endl;
+   {
+       std::fstream mstr; 
+       mstr.open("mat.dat",std::ios::out);
+       spmat0->PrintMatlab(mstr);
+       mstr.close();
+   }
+   
+   mfem::ParaViewDataCollection *dacol=new mfem::ParaViewDataCollection("IGF_OUT",mesh);
+   dacol->SetLevelsOfDetail(2);
+   dacol->SetCycle(1);
+   dacol->SetTime(0.0); // set the time
+   dacol->RegisterField("density",igf);
+   dacol->RegisterField("grads0",ggf0);
+   dacol->RegisterField("grads1",ggf1);
+   dacol->Save();
+   delete dacol;
+   
+   delete ggf0;
+   delete ggf1;
+   delete stat;
+   delete resv0;
+   delete resv1;
+   delete igf;     
+   delete nf0;
+   delete nf1;
+   delete fespace;
+   delete fec;
+   delete mesh;
+   
+   return 0;
+}

examples/ex5p.cpp

@@ -4,8 +4,10 @@
 //
 // Sample runs:  mpirun -np 4 ex5p -m ../data/square-disc.mesh
 //               mpirun -np 4 ex5p -m ../data/star.mesh
+//               mpirun -np 4 ex5p -m ../data/star.mesh -r 2 -pa
 //               mpirun -np 4 ex5p -m ../data/beam-tet.mesh
 //               mpirun -np 4 ex5p -m ../data/beam-hex.mesh
+//               mpirun -np 4 ex5p -m ../data/beam-hex.mesh -pa
 //               mpirun -np 4 ex5p -m ../data/escher.mesh
 //               mpirun -np 4 ex5p -m ../data/fichera.mesh
 //
@@ -54,19 +56,25 @@ int main(int argc, char *argv[])
 
    // 2. Parse command-line options.
    const char *mesh_file = "../data/star.mesh";
+   int ref_levels = -1;
    int order = 1;
    bool par_format = false;
+   bool pa = false;
    bool visualization = 1;
    bool adios2 = false;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
                   "Mesh file to use.");
+   args.AddOption(&ref_levels, "-r", "--refine",
+                  "Number of times to refine the mesh uniformly.");
    args.AddOption(&order, "-o", "--order",
                   "Finite element order (polynomial degree).");
    args.AddOption(&par_format, "-pf", "--parallel-format", "-sf",
                   "--serial-format",
                   "Format to use when saving the results for VisIt.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -97,10 +105,13 @@ int main(int argc, char *argv[])
    // 4. Refine the serial mesh on all processors to increase the resolution. In
    //    this example we do 'ref_levels' of uniform refinement. We choose
    //    'ref_levels' to be the largest number that gives a final mesh with no
-   //    more than 10,000 elements.
+   //    more than 10,000 elements, unless the user specifies it as input.
    {
-      int ref_levels =
-         (int)floor(log(10000./mesh->GetNE())/log(2.)/dim);
+      if (ref_levels == -1)
+      {
+         ref_levels = (int)floor(log(10000./mesh->GetNE())/log(2.)/dim);
+      }
+
       for (int l = 0; l < ref_levels; l++)
       {
          mesh->UniformRefinement();
@@ -196,25 +207,47 @@ int main(int argc, char *argv[])
    ParBilinearForm *mVarf(new ParBilinearForm(R_space));
    ParMixedBilinearForm *bVarf(new ParMixedBilinearForm(R_space, W_space));
 
-   HypreParMatrix *M, *B;
+   HypreParMatrix *M = NULL;
+   HypreParMatrix *B = NULL;
 
+   if (pa) { mVarf->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    mVarf->AddDomainIntegrator(new VectorFEMassIntegrator(k));
    mVarf->Assemble();
-   mVarf->Finalize();
-   M = mVarf->ParallelAssemble();
+   if (!pa) { mVarf->Finalize(); }
 
+   if (pa) { bVarf->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    bVarf->AddDomainIntegrator(new VectorFEDivergenceIntegrator);
    bVarf->Assemble();
-   bVarf->Finalize();
-   B = bVarf->ParallelAssemble();
-   (*B) *= -1;
-
-   HypreParMatrix *BT = B->Transpose();
+   if (!pa) { bVarf->Finalize(); }
 
    BlockOperator *darcyOp = new BlockOperator(block_trueOffsets);
-   darcyOp->SetBlock(0,0, M);
-   darcyOp->SetBlock(0,1, BT);
-   darcyOp->SetBlock(1,0, B);
+
+   Array<int> empty_tdof_list;  // empty
+   OperatorPtr opM, opB;
+
+   TransposeOperator *Bt = NULL;
+
+   if (pa)
+   {
+      mVarf->FormSystemMatrix(empty_tdof_list, opM);
+      bVarf->FormRectangularSystemMatrix(empty_tdof_list, empty_tdof_list, opB);
+      Bt = new TransposeOperator(opB.Ptr());
+
+      darcyOp->SetBlock(0,0, opM.Ptr());
+      darcyOp->SetBlock(0,1, Bt, -1.0);
+      darcyOp->SetBlock(1,0, opB.Ptr(), -1.0);
+   }
+   else
+   {
+      M = mVarf->ParallelAssemble();
+      B = bVarf->ParallelAssemble();
+      (*B) *= -1;
+      Bt = new TransposeOperator(B);
+
+      darcyOp->SetBlock(0,0, M);
+      darcyOp->SetBlock(0,1, Bt);
+      darcyOp->SetBlock(1,0, B);
+   }
 
    // 11. Construct the operators for preconditioner
    //
@@ -223,17 +256,43 @@ int main(int argc, char *argv[])
    //
    //     Here we use Symmetric Gauss-Seidel to approximate the inverse of the
    //     pressure Schur Complement.
-   HypreParMatrix *MinvBt = B->Transpose();
-   HypreParVector *Md = new HypreParVector(MPI_COMM_WORLD, M->GetGlobalNumRows(),
-                                           M->GetRowStarts());
-   M->GetDiag(*Md);
+   HypreParMatrix *MinvBt = NULL;
+   HypreParVector *Md = NULL;
+   HypreParMatrix *S = NULL;
+   Vector Md_PA;
+   Solver *invM, *invS;
 
-   MinvBt->InvScaleRows(*Md);
-   HypreParMatrix *S = ParMult(B, MinvBt);
+   if (pa)
+   {
+      Md_PA.SetSize(R_space->GetTrueVSize());
+      mVarf->AssembleDiagonal(Md_PA);
+      Vector invMd(Md_PA.Size());
+      for (int i=0; i<Md_PA.Size(); ++i)
+      {
+         invMd(i) = 1.0 / Md_PA(i);
+      }
 
-   HypreSolver *invM, *invS;
-   invM = new HypreDiagScale(*M);
-   invS = new HypreBoomerAMG(*S);
+      Vector BMBt_diag(W_space->GetTrueVSize());
+      bVarf->AssembleDiagonal_ADAt(invMd, BMBt_diag);
+
+      Array<int> ess_tdof_list;  // empty
+
+      invM = new OperatorJacobiSmoother(Md_PA, ess_tdof_list);
+      invS = new OperatorJacobiSmoother(BMBt_diag, ess_tdof_list);
+   }
+   else
+   {
+      Md = new HypreParVector(MPI_COMM_WORLD, M->GetGlobalNumRows(),
+                              M->GetRowStarts());
+      M->GetDiag(*Md);
+
+      MinvBt = B->Transpose();
+      MinvBt->InvScaleRows(*Md);
+      S = ParMult(B, MinvBt);
+
+      invM = new HypreDiagScale(*M);
+      invS = new HypreBoomerAMG(*S);
+   }
 
    invM->iterative_mode = false;
    invS->iterative_mode = false;
@@ -245,7 +304,7 @@ int main(int argc, char *argv[])
 
    // 12. Solve the linear system with MINRES.
    //     Check the norm of the unpreconditioned residual.
-   int maxIter(500);
+   int maxIter(pa ? 1000 : 500);
    double rtol(1.e-6);
    double atol(1.e-10);
 
@@ -395,7 +454,7 @@ int main(int argc, char *argv[])
    delete S;
    delete Md;
    delete MinvBt;
-   delete BT;
+   delete Bt;
    delete B;
    delete M;
    delete mVarf;

examples/ex71.cpp

@@ -0,0 +1,349 @@
+//                       MFEM Example 71 - Serial Version
+//
+// Compile with: make ex71
+//
+// Sample runs:
+//    ex71 -m ../data/beam-quad.mesh
+//    ex71 -m ../data/beam-tri.mesh
+//    ex71 -m ../data/beam-hex.mesh
+//    ex71 -m ../data/beam-tet.mesh
+//    ex71 -m ../data/beam-wedge.mesh
+//
+// Description:  This examples solves a quasi-static nonlinear
+//               pLaplacian problem with zero Dirichlet boundary
+//				 conditions applied on all defined boundaries
+//
+//               The example demonstrates the use of nonlinear operators
+//				 combined with automatic differentiation (AD). The definitions
+//				 of the integrators are written in the ex71.hpp.
+//				 Selecting integrator=0 will use handcoded integrator.
+//				 Selecting integrator=1 will utilize AD integrator.
+//				 The AD integrator can be modifief to use ADQFunctionJ
+//				 or ADQFunctionH by overwritting the class type of qint,
+//				 i.e., pLapIntegrandJ or pLapIntegrandH.
+//
+//				 qint (the integrand) is a function which is evaluated
+//				 at every integration point. For implementations utilizing
+//				 ADQFunctionJ, the user has to implement the function and the
+//				 residual evaluation - all virtual methods. The Jacobian of
+//				 the residual is evaluated using AD
+//
+//				 For implementations utilizing ADQFunctionH, the user has
+//				 to implement only the function evaluation (preferebaly as
+//				 a template) and the first derivative (the residual) and the
+//				 second derivatives (the Hessian) are evaluated using AD.
+//
+//               We recommend viewing examples 1 and 19, before viewing this
+//               example.
+
+#include "ex71.hpp"
+
+#undef MFEM_USE_SUITESPARSE
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options
+   const char *mesh_file = "../data/beam-tet.mesh";
+   int ser_ref_levels = 3;
+   int order = 1;
+   bool visualization = true;
+   double newton_rel_tol = 1e-4;
+   double newton_abs_tol = 1e-6;
+   int newton_iter = 500;
+   int print_level = 0;
+   double pp = 2.0;
+   int integrator=0;
+   mfem::StopWatch* timer=new mfem::StopWatch();
+
+   mfem::OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
+                  "Number of times to refine the mesh uniformly in serial.");
+   args.AddOption(&order, "-o", "--order",
+                  "Order (degree) of the finite elements.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.AddOption(&newton_rel_tol, "-rel", "--relative-tolerance",
+                  "Relative tolerance for the Newton solve.");
+   args.AddOption(&newton_abs_tol, "-abs", "--absolute-tolerance",
+                  "Absolute tolerance for the Newton solve.");
+   args.AddOption(&newton_iter, "-it", "--newton-iterations",
+                  "Maximum iterations for the Newton solve.");
+   args.AddOption(&pp, "-pp", "--power-parameter",
+                  "Power parameter (>=2.0) for the p-Laplacian.");
+   args.AddOption((&print_level),"-prt","--print-level",
+                  "Print level.");
+   args.AddOption(&integrator, "-int","--integrator",
+                  "Integrator 0: standard; 1: AD uaing energy; 2: AD using gradients");
+
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(std::cout);
+      return 1;
+   }
+   args.PrintOptions(std::cout);
+
+
+   // 2. Read the (serial) mesh from the given mesh file.
+   mfem::Mesh *mesh = new mfem::Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 3. Refine the mesh in serial to increase the resolution. In this example
+   //    we do 'ser_ref_levels' of uniform refinement, where 'ser_ref_levels' is
+   //    a command-line parameter.
+   for (int lev = 0; lev < ser_ref_levels; lev++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   // 4. Define the power parameter for the p-Laplacian and all other
+   //    coefficients
+   mfem::ConstantCoefficient c_pp(pp);
+   mfem::ConstantCoefficient load(1.000000000);
+   mfem::ConstantCoefficient c_ee(0.000000001);
+
+   // 5. Define the finite element spaces for the solution
+   mfem::H1_FECollection fec(order,dim);
+   mfem::FiniteElementSpace fespace(mesh,&fec,1,mfem::Ordering::byVDIM);
+   int glob_size=fespace.GetTrueVSize();
+   std::cout << "Number of finite element unknowns: " << glob_size << std::endl;
+
+   // 6. Define the Dirichlet conditions
+   mfem::Array<int> ess_bdr(mesh->bdr_attributes.Max());
+   ess_bdr = 1;
+
+   // 7. Define the nonlinear form
+   mfem::NonlinearForm* nf=new mfem::NonlinearForm(&fespace);
+
+   // 8. Define the solution vector x
+   mfem::GridFunction x(&fespace);
+   x = 0.0;
+   mfem::Vector tv(fespace.GetTrueVSize());
+   mfem::Vector sv(fespace.GetTrueVSize());
+   tv=0.0;
+   sv=0.0;
+
+   // 9. Define ParaView DataCollection
+   mfem::ParaViewDataCollection *dacol=new mfem::ParaViewDataCollection("pLap",mesh);
+   dacol->SetLevelsOfDetail(order);
+   dacol->RegisterField("sol",&x);
+
+
+   // 11. Set domain integrators - start with linear diffusion
+   {
+       // the default power coefficient is 2.0
+       mfem::ConstantCoefficient lpp(2.0);
+       if(integrator==0)
+       {
+            nf->AddDomainIntegrator(new mfem::pLaplace(lpp,c_ee,load));
+       }else
+       if(integrator==1)
+       {
+           nf->AddDomainIntegrator(new mfem::pLaplaceAD(lpp,c_ee,load));
+       }
+       nf->SetEssentialBC(ess_bdr);
+       // compute the energy
+       double energy=nf->GetEnergy(tv);
+       std::cout<<"[2] The total energy of the system is E="<<energy<<std::endl;
+       // time the assembly
+       timer->Clear();
+       timer->Start();
+       mfem::Operator &op=nf->GetGradient(sv);
+       timer->Stop();
+       std::cout<<"[2] The assembly time is: "<<timer->RealTime()<<std::endl;
+       mfem::Solver *prec;
+#ifdef MFEM_USE_SUITESPARSE
+       prec=new mfem::UMFPackSolver();
+#else
+       prec=new mfem::GSSmoother();
+#endif
+       mfem::CGSolver *j_pcg = new mfem::CGSolver();
+       j_pcg->SetRelTol(1e-7);
+       j_pcg->SetAbsTol(1e-15);
+       j_pcg->SetMaxIter(500);
+       j_pcg->SetPrintLevel(print_level);
+       j_pcg->SetPreconditioner(*prec);
+
+       mfem::NewtonSolver* ns;
+       ns=new mfem::NewtonSolver();
+       ns->iterative_mode = true;
+       ns->SetSolver(*j_pcg);
+       ns->SetOperator(*nf);
+       ns->SetPrintLevel(print_level);
+       ns->SetRelTol(1e-6);
+       ns->SetAbsTol(1e-12);
+       ns->SetMaxIter(10);
+       //solve the problem
+       timer->Clear();
+       timer->Start();
+       ns->Mult(tv, sv);
+       timer->Stop();
+       std::cout<<"Time for the NewtonSolver: "<<timer->RealTime()<<std::endl;
+
+       energy=nf->GetEnergy(sv);
+       std::cout<<"[pp=2] The total energy of the system is E="<<energy<<std::endl;
+
+       delete ns;
+       delete j_pcg;
+       delete prec;
+
+       x.SetFromTrueDofs(sv);
+       dacol->SetTime(2.0);
+       dacol->SetCycle(2);
+       dacol->Save();
+   }
+
+   // 12. Continue with powers higher than 2
+   for(int i=3;i<pp;i++)
+   {
+       delete nf;
+       nf=new mfem::NonlinearForm(&fespace);
+       mfem::ConstantCoefficient lpp((double)i);
+       if(integrator==0)
+       {
+            nf->AddDomainIntegrator(new mfem::pLaplace(lpp,c_ee,load));
+       }else
+       if(integrator==1)
+       {
+           nf->AddDomainIntegrator(new mfem::pLaplaceAD(lpp,c_ee,load));
+       }
+       nf->SetEssentialBC(ess_bdr);
+       // compute the energy
+       double energy=nf->GetEnergy(sv);
+       std::cout<<"[pp="<<i<<"] The total energy of the system is E="<<energy<<std::endl;
+       // time the assembly
+       timer->Clear();
+       timer->Start();
+       mfem::Operator &op=nf->GetGradient(sv);
+       timer->Stop();
+       std::cout<<"[pp="<<i<<"] The assembly time is: "<<timer->RealTime()<<std::endl;
+       mfem::Solver *prec;
+#ifdef MFEM_USE_SUITESPARSE
+       prec=new mfem::UMFPackSolver();
+#else
+       prec=new mfem::GSSmoother();
+#endif
+       mfem::CGSolver *j_pcg = new mfem::CGSolver();
+       j_pcg->SetRelTol(1e-7);
+       j_pcg->SetAbsTol(1e-15);
+       j_pcg->SetMaxIter(500);
+       j_pcg->SetPrintLevel(print_level);
+       j_pcg->SetPreconditioner(*prec);
+
+       mfem::NewtonSolver* ns;
+       ns=new mfem::NewtonSolver();
+       ns->iterative_mode = true;
+       ns->SetSolver(*j_pcg);
+       ns->SetOperator(*nf);
+       ns->SetPrintLevel(print_level);
+       ns->SetRelTol(1e-6);
+       ns->SetAbsTol(1e-12);
+       ns->SetMaxIter(10);
+       //solve the problem
+       timer->Clear();
+       timer->Start();
+       ns->Mult(tv, sv);
+       timer->Stop();
+       std::cout<<"Time for the NewtonSolver: "<<timer->RealTime()<<std::endl;
+
+       energy=nf->GetEnergy(sv);
+       std::cout<<"[pp="<<i<<"] The total energy of the system is E="<<energy<<std::endl;
+
+       delete ns;
+       delete j_pcg;
+       delete prec;
+
+       x.SetFromTrueDofs(sv);
+       dacol->SetTime(i);
+       dacol->SetCycle(i);
+       dacol->Save();
+   }
+
+   // 13. Continue with the final power
+   if( std::abs(pp-2.0) > std::numeric_limits<double>::epsilon())
+   {
+       delete nf;
+       nf=new mfem::NonlinearForm(&fespace);
+       if(integrator==0)
+       {
+            nf->AddDomainIntegrator(new mfem::pLaplace(c_pp,c_ee,load));
+       }else
+       if(integrator==1)
+       {
+           nf->AddDomainIntegrator(new mfem::pLaplaceAD(c_pp,c_ee,load));
+       }
+       nf->SetEssentialBC(ess_bdr);
+       // compute the energy
+       double energy=nf->GetEnergy(sv);
+       std::cout<<"[pp="<<pp<<"] The total energy of the system is E="<<energy<<std::endl;
+       // time the assembly
+       timer->Clear();
+       timer->Start();
+       mfem::Operator &op=nf->GetGradient(sv);
+       timer->Stop();
+       std::cout<<"[pp="<<pp<<"] The assembly time is: "<<timer->RealTime()<<std::endl;
+       mfem::Solver *prec;
+#ifdef MFEM_USE_SUITESPARSE
+       prec=new mfem::UMFPackSolver();
+#else
+       prec=new mfem::GSSmoother();
+#endif
+       mfem::CGSolver *j_pcg = new mfem::CGSolver();
+       j_pcg->SetRelTol(1e-7);
+       j_pcg->SetAbsTol(1e-15);
+       j_pcg->SetMaxIter(500);
+       j_pcg->SetPrintLevel(print_level);
+       j_pcg->SetPreconditioner(*prec);
+
+       mfem::NewtonSolver* ns;
+       ns=new mfem::NewtonSolver();
+       ns->iterative_mode = true;
+       ns->SetSolver(*j_pcg);
+       ns->SetOperator(*nf);
+       ns->SetPrintLevel(print_level);
+       ns->SetRelTol(1e-6);
+       ns->SetAbsTol(1e-12);
+       ns->SetMaxIter(10);
+       //solve the problem
+       timer->Clear();
+       timer->Start();
+       ns->Mult(tv, sv);
+       timer->Stop();
+       std::cout<<"Time for the NewtonSolver: "<<timer->RealTime()<<std::endl;
+
+       energy=nf->GetEnergy(sv);
+       std::cout<<"[pp="<<pp<<"] The total energy of the system is E="<<energy<<std::endl;
+
+       delete ns;
+       delete j_pcg;
+       delete prec;
+
+       x.SetFromTrueDofs(sv);
+       dacol->SetTime(pp);
+       if(pp<2.0)
+       {
+           dacol->SetCycle(std::floor(pp));
+       }
+       else
+       {
+           dacol->SetCycle(std::ceil(pp));
+       }
+       dacol->Save();
+   }
+
+
+
+   // 19. Free the used memory
+   delete dacol;
+   delete nf;
+   delete mesh;
+   delete timer;
+
+
+   return 0;
+}
+
+

examples/ex71.hpp

@@ -0,0 +1,619 @@
+// shared implementation ex71p/ex71 for the AD integrands and
+// the handconded integrators
+
+
+#ifndef EXAMPLE71_H
+#define EXAMPLE71_H
+
+#include "mfem.hpp"
+#include <memory>
+#include <iostream>
+#include <fstream>
+
+
+namespace mfem {
+
+class pLapIntegrandJ: public ADQFunctionJ
+{
+private:
+
+    template<typename DType, typename MVType>
+    void MyQFunctionDU(const mfem::Vector& vparam, MVType& uu, MVType& rr)
+    {
+        double pp=vparam[0];
+        double ee=vparam[1];
+        double ff=vparam[2];
+
+        DType norm2=uu[0]*uu[0]+uu[1]*uu[1]+uu[2]*uu[2];
+        DType tvar=pow(ee*ee+norm2,(pp-2.0)/2.0);
+
+        rr[0]=tvar*uu[0];
+        rr[1]=tvar*uu[1];
+        rr[2]=tvar*uu[2];
+        rr[3]=-ff;
+
+    }
+
+public:
+    pLapIntegrandJ():ADQFunctionJ(4){} //the residual vector rr has size of 4 elements
+
+    ~pLapIntegrandJ(){}
+
+    virtual double QFunction(const mfem::Vector &vparam,const mfem::Vector &uu) override
+    {
+        double pp=vparam[0];
+        double ee=vparam[1];
+        double ff=vparam[2];
+
+        double  u=uu[3];
+        double  norm2=uu[0]*uu[0]+uu[1]*uu[1]+uu[2]*uu[2];
+
+        double rez= pow(ee*ee+norm2,pp/2.0)/pp-ff*u;
+        return rez;
+    }
+
+
+    virtual void QFunctionDU(const mfem::Vector& vparam, mfem::Vector& uu, mfem::Vector& rr) override
+    {
+        MyQFunctionDU<double,mfem::Vector>(vparam,uu,rr);
+    }
+
+    virtual void QFunctionDU(const mfem::Vector &vparam, ADFVector &uu, ADFVector &rr) override
+    {
+        MyQFunctionDU<ADFType,ADFVector>(vparam,uu,rr);
+    }
+};
+
+
+class pLapIntegrandH: public ADQFunctionH
+{
+private:
+
+    //MVType - vector type taking one of the following
+    // mfem::Vector - scalar double
+    // ADFVector    - scalar ADFType
+    // ADSVector    - scalar ADSType
+    template<typename DType, typename MVType>
+    DType MyQFunction(const mfem::Vector& vparam, MVType& uu)
+    {
+        double pp=vparam[0];
+        double ee=vparam[1];
+        double ff=vparam[2];
+
+        DType  u=uu[3];
+        DType  norm2=uu[0]*uu[0]+uu[1]*uu[1]+uu[2]*uu[2];
+
+        DType rez= pow(ee*ee+norm2,pp/2.0)/pp-ff*u;
+        return rez;
+    }
+
+    public:
+    pLapIntegrandH(){}
+
+    virtual ~pLapIntegrandH(){}
+
+    virtual double QFunction(const mfem::Vector &vparam,const mfem::Vector &uu) override
+    {
+        double rez=MyQFunction<double,const mfem::Vector>(vparam,uu);
+        return rez;
+    }
+
+    virtual ADFType QFunction(const mfem::Vector &vparam, ADFVector& uu) override
+    {
+        ADFType rez=MyQFunction<ADFType,ADFVector>(vparam,uu);
+        return rez;
+    }
+
+    virtual ADSType QFunction(const mfem::Vector &vparam, ADSVector& uu) override
+    {
+        ADSType rez=MyQFunction<ADSType,ADSVector>(vparam,uu);
+        return rez;
+    }
+
+};
+
+
+class pLaplaceAD: public mfem::NonlinearFormIntegrator
+{
+protected:
+    mfem::Coefficient* pp;
+    mfem::Coefficient* coeff;
+    mfem::Coefficient* load;
+
+    pLapIntegrandJ qint;
+public:
+    pLaplaceAD()
+    {
+        coeff=nullptr;
+        pp=nullptr;
+    }
+
+    pLaplaceAD(mfem::Coefficient& pp_):pp(&pp_), coeff(nullptr), load(nullptr)
+    {
+
+    }
+
+    pLaplaceAD(mfem::Coefficient &pp_,mfem::Coefficient& q, mfem::Coefficient& ld_): pp(&pp_), coeff(&q), load(&ld_)
+    {
+
+    }
+
+    virtual ~pLaplaceAD()
+    {
+
+    }
+
+    virtual double GetElementEnergy(const mfem::FiniteElement &el, mfem::ElementTransformation &trans, const mfem::Vector &elfun) override
+    {
+        double energy=0.0;
+        int ndof = el.GetDof();
+        int ndim = el.GetDim();
+        int spaceDim = trans.GetSpaceDim();
+        bool square = (ndim == spaceDim);
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+
+        mfem::Vector shapef(ndof);
+        mfem::DenseMatrix dshape_iso(ndof,ndim);
+        mfem::DenseMatrix dshape_xyz(ndof,spaceDim);
+        mfem::Vector grad(spaceDim);
+
+        mfem::Vector vparam(3);//[power, epsilon, load]
+        mfem::Vector uu(4);//[diff_x,diff_y,diff_z,u]
+
+        uu=0.0;
+        vparam[0]=2.0;  //default power
+        vparam[1]=1e-8; //default epsilon
+        vparam[2]=1.0;  //default load
+
+        double w;
+        double detJ;
+
+        for(int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            w = trans.Weight();
+            detJ = (square ? w : w*w);
+            w = ip.weight *w;
+
+            el.CalcDShape(ip,dshape_iso);
+            el.CalcShape(ip,shapef);
+            // AdjugateJacobian = / adj(J),         if J is square
+            //                    \ adj(J^t.J).J^t, otherwise
+            mfem::Mult(dshape_iso, trans.AdjugateJacobian(), dshape_xyz);
+            // dshape_xyz should be devided by detJ for obtaining the real value
+            // calculate the gradient
+            dshape_xyz.MultTranspose(elfun,grad);
+
+            //set the power
+            if(pp!=nullptr)
+            {
+                vparam[0]=pp->Eval(trans,ip);
+            }
+
+            //set the coefficient ensuring possitiveness of the tangent matrix
+            if(coeff!=nullptr)
+            {
+                vparam[1]=coeff->Eval(trans,ip);
+            }
+            //add the contribution from the load
+            if(load!=nullptr)
+            {
+                vparam[2]=load->Eval(trans,ip);
+            }
+            //fill the values of vector uu
+            for(int jj=0;jj<spaceDim;jj++)
+            {
+                uu[jj]=grad[jj]/detJ;
+            }
+            uu[3]=shapef*elfun;
+
+            energy = energy + w * (qint.QFunction(vparam,uu));
+
+        }
+        return energy;
+    }
+
+    virtual void AssembleElementVector(const mfem::FiniteElement & el,
+                                       mfem::ElementTransformation & trans,
+                                       const mfem::Vector & elfun,
+                                       mfem::Vector & elvect) override
+    {
+            int ndof = el.GetDof();
+            int ndim = el.GetDim();
+            int spaceDim = trans.GetSpaceDim();
+            bool square = (ndim == spaceDim);
+            const mfem::IntegrationRule *ir = NULL;
+            int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+            ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+
+            mfem::Vector shapef(ndof);
+            mfem::DenseMatrix dshape_iso(ndof,ndim);
+            mfem::DenseMatrix dshape_xyz(ndof,spaceDim);
+            mfem::Vector lvec(ndof);
+            elvect.SetSize(ndof);
+            elvect=0.0;
+
+            mfem::DenseMatrix B(ndof,4); //[diff_x,diff_y,diff_z, shape]
+            mfem::Vector vparam(3);//[power, epsilon, load]
+            mfem::Vector uu(4);//[diff_x,diff_y,diff_z,u]
+            mfem::Vector du(4);
+            B=0.0;
+            uu=0.0;
+            //initialize the parameters - keep the same order
+            //utilized in the pLapIntegrator definition
+            vparam[0]=2.0;  //default power
+            vparam[1]=1e-8; //default epsilon
+            vparam[2]=1.0;  //default load
+
+            double w;
+            double detJ;
+
+            for (int i = 0; i < ir -> GetNPoints(); i++)
+            {
+                const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+                trans.SetIntPoint(&ip);
+                w = trans.Weight();
+                detJ = (square ? w : w*w);
+                w = ip.weight * w;
+
+                el.CalcDShape(ip,dshape_iso);
+                el.CalcShape(ip,shapef);
+                mfem::Mult(dshape_iso, trans.InverseJacobian(), dshape_xyz);
+
+                //set the matrix B
+                for(int jj=0;jj<spaceDim;jj++)
+                {
+                    B.SetCol(jj,dshape_xyz.GetColumn(jj));
+                }
+                B.SetCol(3,shapef);
+
+
+                //set the power
+                if(pp!=nullptr)
+                {
+                    vparam[0]=pp->Eval(trans,ip);
+                }
+                //set the coefficient ensuring possitiveness of the tangent matrix
+                if(coeff!=nullptr)
+                {
+                    vparam[1]=coeff->Eval(trans,ip);
+                }
+                //add the contribution from the load
+                if(load!=nullptr)
+                {
+                    vparam[2]=load->Eval(trans,ip);
+                }
+
+                //calculate uu
+                B.MultTranspose(elfun,uu);
+                //calculate derivative of the energy with respect to uu
+                qint.QFunctionDU(vparam,uu,du);
+
+                B.Mult(du,lvec);
+                elvect.Add( w, lvec);
+            }// end integration loop
+    }
+
+    virtual void AssembleElementGrad(const mfem::FiniteElement & el,
+                                         mfem::ElementTransformation & trans,
+                                         const mfem::Vector & elfun, mfem::DenseMatrix & elmat) override
+    {
+        int ndof = el.GetDof();
+        int ndim = el.GetDim();
+        int spaceDim = trans.GetSpaceDim();
+        bool square = (ndim == spaceDim);
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+
+        mfem::Vector shapef(ndof);
+        mfem::DenseMatrix dshape_iso(ndof,ndim);
+        mfem::DenseMatrix dshape_xyz(ndof,spaceDim);
+        elmat.SetSize(ndof,ndof);
+        elmat=0.0;
+
+        mfem::DenseMatrix B(ndof,4); //[diff_x,diff_y,diff_z, shape]
+        mfem::DenseMatrix A(ndof,4);
+        mfem::Vector vparam(3);//[power, epsilon, load]
+        mfem::Vector uu(4);//[diff_x,diff_y,diff_z,u]
+        mfem::DenseMatrix duu(4,4);
+        B=0.0;
+        uu=0.0;
+        //initialize the parameters - keep the same order
+        //utilized in the pLapIntegrator definition
+        vparam[0]=2.0;  //default power
+        vparam[1]=1e-8; //default epsilon
+        vparam[2]=1.0;  //default load
+
+        double w;
+        double detJ;
+
+        for (int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            w = trans.Weight();
+            detJ = (square ? w : w*w);
+            w = ip.weight * w;
+
+            el.CalcDShape(ip,dshape_iso);
+            el.CalcShape(ip,shapef);
+            mfem::Mult(dshape_iso, trans.InverseJacobian(), dshape_xyz);
+
+            //set the matrix B
+            for(int jj=0;jj<spaceDim;jj++)
+            {
+                B.SetCol(jj,dshape_xyz.GetColumn(jj));
+            }
+            B.SetCol(3,shapef);
+
+
+            //set the power
+            if(pp!=nullptr)
+            {
+                vparam[0]=pp->Eval(trans,ip);
+            }
+            //set the coefficient ensuring possitiveness of the tangent matrix
+            if(coeff!=nullptr)
+            {
+                vparam[1]=coeff->Eval(trans,ip);
+            }
+            //add the contribution from the load
+            if(load!=nullptr)
+            {
+                vparam[2]=load->Eval(trans,ip);
+            }
+
+            //calculate uu
+            B.MultTranspose(elfun,uu);
+            //calculate derivative of the energy with respect to uu
+            qint.QFunctionDD(vparam,uu,duu);
+
+            mfem::Mult(B,duu,A);
+            mfem::AddMult_a_ABt(w,A,B,elmat);
+
+        }//end integration loop
+    }
+
+
+};
+
+
+class pLaplace: public mfem::NonlinearFormIntegrator
+{
+protected:
+    mfem::Coefficient* pp;
+    mfem::Coefficient* coeff;
+    mfem::Coefficient* load;
+public:
+    pLaplace()
+    {
+        coeff=nullptr;
+        pp=nullptr;
+    }
+
+    pLaplace(mfem::Coefficient& pp_):pp(&pp_), coeff(nullptr), load(nullptr)
+    {
+
+    }
+
+    pLaplace(mfem::Coefficient &pp_,mfem::Coefficient& q, mfem::Coefficient& ld_): pp(&pp_), coeff(&q), load(&ld_)
+    {
+
+    }
+
+    virtual ~pLaplace()
+    {
+
+    }
+
+    virtual double GetElementEnergy(const mfem::FiniteElement &el, mfem::ElementTransformation &trans, const mfem::Vector &elfun) override
+    {
+        double energy=0.0;
+        int ndof = el.GetDof();
+        int ndim = el.GetDim();
+        int spaceDim = trans.GetSpaceDim();
+        bool square = (ndim == spaceDim);
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+
+        mfem::Vector shapef(ndof);
+        mfem::DenseMatrix dshape_iso(ndof,ndim);
+        mfem::DenseMatrix dshape_xyz(ndof,spaceDim);
+        mfem::Vector grad(spaceDim);
+
+        double w;
+        double detJ;
+        double nrgrad2;
+        double ppp=2.0;
+        double eee=0.0;
+
+        for(int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            w = trans.Weight();
+            detJ = (square ? w : w*w);
+            w = ip.weight *w;
+
+            el.CalcDShape(ip,dshape_iso);
+            el.CalcShape(ip,shapef);
+            // AdjugateJacobian = / adj(J),         if J is square
+            //                    \ adj(J^t.J).J^t, otherwise
+            mfem::Mult(dshape_iso, trans.AdjugateJacobian(), dshape_xyz);
+            // dshape_xyz should be devided by detJ for obtaining the real value
+            // calculate the gradient
+            dshape_xyz.MultTranspose(elfun,grad);
+            nrgrad2=grad*grad/(detJ*detJ);
+
+            //set the power
+            if(pp!=nullptr)
+            {
+                ppp=pp->Eval(trans,ip);
+            }
+
+            //set the coefficient ensuring possitiveness of the tangent matrix
+            if(coeff!=nullptr)
+            {
+                eee=coeff->Eval(trans,ip);
+            }
+
+            energy = energy + w * std::pow( nrgrad2 + eee * eee , ppp / 2.0 ) / ppp;
+
+            //add the contribution from the load
+            if(load!=nullptr)
+            {
+                energy = energy - w *  (shapef*elfun) * load->Eval(trans,ip);
+            }
+        }
+        return energy;
+    }
+
+    virtual void  AssembleElementVector(const mfem::FiniteElement & el,
+                                                mfem::ElementTransformation & trans,
+                                                const mfem::Vector & elfun,
+                                                mfem::Vector & elvect) override
+    {
+            int ndof = el.GetDof();
+            int ndim = el.GetDim();
+            int spaceDim = trans.GetSpaceDim();
+            bool square = (ndim == spaceDim);
+            const mfem::IntegrationRule *ir = NULL;
+            int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+            ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+
+            mfem::Vector shapef(ndof);
+            mfem::DenseMatrix dshape_iso(ndof,ndim);
+            mfem::DenseMatrix dshape_xyz(ndof,spaceDim);
+            mfem::Vector grad(spaceDim);
+            mfem::Vector lvec(ndof);
+            elvect.SetSize(ndof);
+            elvect=0.0;
+
+            double w;
+            double detJ;
+            double nrgrad;
+            double aa;
+            double ppp=2.0;
+            double eee=0.0;
+
+            for (int i = 0; i < ir -> GetNPoints(); i++)
+            {
+                const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+                trans.SetIntPoint(&ip);
+                w = trans.Weight();
+                detJ = (square ? w : w*w);
+                w = ip.weight * w;//w;
+
+                el.CalcDShape(ip,dshape_iso);
+                el.CalcShape(ip,shapef);
+                // AdjugateJacobian = / adj(J),         if J is square
+                //                    \ adj(J^t.J).J^t, otherwise
+                mfem::Mult(dshape_iso, trans.AdjugateJacobian(), dshape_xyz);
+                // dshape_xyz should be devided by detJ for obtaining the real value
+
+                //calculate the gradient
+                dshape_xyz.MultTranspose(elfun,grad);
+                nrgrad=grad.Norml2()/detJ;
+                //grad is not scaled so far, i.e., grad=grad/detJ
+
+                //set the power
+                if(pp!=nullptr)
+                {
+                    ppp=pp->Eval(trans,ip);
+                }
+
+                //set the coefficient ensuring possitiveness of the tangent matrix
+                if(coeff!=nullptr)
+                {
+                    eee=coeff->Eval(trans,ip);
+                }
+
+                aa = nrgrad * nrgrad + eee * eee;
+                aa=std::pow( aa , ( ppp - 2.0 ) / 2.0 );
+                dshape_xyz.Mult(grad,lvec);
+                elvect.Add( w * aa / ( detJ * detJ ), lvec);
+
+
+                //add loading
+                if(load!=nullptr)
+                {
+                    elvect.Add(-w*load->Eval(trans,ip),shapef);
+                }
+            }// end integration loop
+    }
+
+    virtual void AssembleElementGrad(const mfem::FiniteElement & el,
+                                         mfem::ElementTransformation & trans,
+                                         const mfem::Vector & elfun, mfem::DenseMatrix & elmat) override
+    {
+        int ndof = el.GetDof();
+        int ndim = el.GetDim();
+        int spaceDim = trans.GetSpaceDim();
+        bool square = (ndim == spaceDim);
+        const mfem::IntegrationRule *ir = NULL;
+        int order = 2 * trans.OrderGrad(&el) - 1; // correct order?
+        ir = &mfem::IntRules.Get(el.GetGeomType(), order);
+
+        mfem::DenseMatrix dshape_iso(ndof,ndim);
+        mfem::DenseMatrix dshape_xyz(ndof,spaceDim);
+        mfem::Vector grad(spaceDim);
+        mfem::Vector lvec(ndof);
+        elmat.SetSize(ndof,ndof);
+        elmat=0.0;
+
+        double w;
+        double detJ;
+        double nrgrad;
+        double aa0;
+        double aa1;
+        double ppp=2.0;
+        double eee=0.0;
+
+        for (int i = 0; i < ir -> GetNPoints(); i++)
+        {
+            const mfem::IntegrationPoint &ip = ir->IntPoint(i);
+            trans.SetIntPoint(&ip);
+            w = trans.Weight();
+            detJ = (square ? w : w*w);
+            w = ip.weight * w;
+
+            el.CalcDShape(ip,dshape_iso);
+            // AdjugateJacobian = / adj(J),         if J is square
+            //                    \ adj(J^t.J).J^t, otherwise
+            mfem::Mult(dshape_iso, trans.AdjugateJacobian(), dshape_xyz);
+            // dshape_xyz should be devided by detJ for obtaining the real value
+            // grad is not scaled so far,i.e., grad=grad/detJ
+
+            //set the power
+            if(pp!=nullptr)
+            {
+                ppp=pp->Eval(trans,ip);
+            }
+            //set the coefficient ensuring possitiveness of the tangent matrix
+            if(coeff!=nullptr)
+            {
+                eee=coeff->Eval(trans,ip);
+            }
+
+            //calculate the gradient
+            dshape_xyz.MultTranspose(elfun,grad);
+            nrgrad = grad.Norml2() / detJ;
+            aa0 = nrgrad * nrgrad + eee * eee;
+            aa1 = std::pow( aa0 , ( ppp - 2.0 ) / 2.0 );
+            aa0 = ( ppp - 2.0 ) * std::pow(aa0, ( ppp - 4.0 ) / 2.0 );
+            dshape_xyz.Mult(grad,lvec);
+            w = w / ( detJ * detJ );
+            mfem::AddMult_a_VVt( w * aa0 / ( detJ * detJ ), lvec, elmat);
+            mfem::AddMult_a_AAt( w * aa1 , dshape_xyz, elmat);
+
+        }//end integration loop
+    }
+};
+
+}
+#endif

examples/ex71p.cpp

@@ -0,0 +1,378 @@
+//                       MFEM Example 71 - Parallel Version
+//
+// Compile with: make ex71p
+//
+// Sample runs:
+//   mpirun -np 2 ex71p -m ../data/beam-quad.mesh
+//   mpirun -np 2 ex71p -m ../data/beam-tri.mesh
+//   mpirun -np 2 ex71p -m ../data/beam-hex.mesh
+//   mpirun -np 2 ex71p -m ../data/beam-tet.mesh
+//   mpirun -np 2 ex71p -m ../data/beam-wedge.mesh
+//
+// Description:  This examples solves a quasi-static nonlinear
+//               pLaplacian problem with zero Dirichlet boundary
+//				 conditions applied on all defined boundaries
+//
+//               The example demonstrates the use of nonlinear operators
+//				 combined with automatic differentiation (AD). The definitions
+//				 of the integrators are written in the ex71.hpp.
+//				 Selecting integrator=0 will use handcoded integrator.
+//				 Selecting integrator=1 will utilize AD integrator.
+//				 The AD integrator can be modifief to use ADQFunctionJ
+//				 or ADQFunctionH by overwritting the class type of qint,
+//				 i.e., pLapIntegrandJ or pLapIntegrandH.
+//
+//				 qint (the integrand) is a function which is evaluated
+//				 at every integration point. For implementations utilizing
+//				 ADQFunctionJ, the user has to implement the function and the
+//				 residual evaluation - all virtual methods. The Jacobian of
+//				 the residual is evaluated using AD
+//
+//				 For implementations utilizing ADQFunctionH, the user has
+//				 to implement only the function evaluation (preferebaly as
+//				 a template) and the first derivative (the residual) and the
+//				 second derivatives (the Hessian) are evaluated using AD.
+//
+//               We recommend viewing examples 1 and 19, before viewing this
+//               example.
+
+
+#include "ex71.hpp"
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI
+   int num_procs, myrank;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
+
+   // 2. Parse command-line options
+   const char *mesh_file = "../data/beam-tet.mesh";
+   int ser_ref_levels = 0;
+   int par_ref_levels = 0;
+   int order = 2;
+   bool visualization = true;
+   double newton_rel_tol = 1e-4;
+   double newton_abs_tol = 1e-6;
+   int newton_iter = 500;
+   int print_level = 0;
+   double pp = 2.0;
+   int integrator=0;
+   mfem::StopWatch* timer=new mfem::StopWatch();
+
+   mfem::OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
+                  "Number of times to refine the mesh uniformly in serial.");
+   args.AddOption(&par_ref_levels, "-rp", "--refine-parallel",
+                  "Number of times to refine the mesh uniformly in parallel.");
+   args.AddOption(&order, "-o", "--order",
+                  "Order (degree) of the finite elements.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.AddOption(&newton_rel_tol, "-rel", "--relative-tolerance",
+                  "Relative tolerance for the Newton solve.");
+   args.AddOption(&newton_abs_tol, "-abs", "--absolute-tolerance",
+                  "Absolute tolerance for the Newton solve.");
+   args.AddOption(&newton_iter, "-it", "--newton-iterations",
+                  "Maximum iterations for the Newton solve.");
+   args.AddOption(&pp, "-pp", "--power-parameter",
+                  "Power parameter (>=2.0) for the p-Laplacian.");
+   args.AddOption((&print_level),"-prt","--print-level",
+                  "Print level.");
+   args.AddOption(&integrator, "-int","--integrator",
+                  "Integrator 0: standard; 1: AD uaing energy; 2: AD using gradients");
+
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myrank == 0)
+      {
+         args.PrintUsage(std::cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myrank == 0)
+   {
+      args.PrintOptions(std::cout);
+   }
+
+   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral and hexahedral meshes
+   //    with the same code.
+   mfem::Mesh *mesh = new mfem::Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 4. Refine the mesh in serial to increase the resolution. In this example
+   //    we do 'ser_ref_levels' of uniform refinement, where 'ser_ref_levels' is
+   //    a command-line parameter.
+   for (int lev = 0; lev < ser_ref_levels; lev++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted.
+   mfem::ParMesh *pmesh = new mfem::ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   for (int lev = 0; lev < par_ref_levels; lev++)
+   {
+      pmesh->UniformRefinement();
+   }
+
+   // 6. Define the power parameter for the p-Laplacian and all other
+   //    coefficients
+   mfem::ConstantCoefficient c_pp(pp);
+   mfem::ConstantCoefficient load(1.000000000);
+   mfem::ConstantCoefficient c_ee(0.000000001);
+
+   // 7. Define the finite element spaces for the solution
+   mfem::H1_FECollection fec(order,dim);
+   mfem::ParFiniteElementSpace fespace(pmesh,&fec,1,mfem::Ordering::byVDIM);
+   HYPRE_Int glob_size=fespace.GlobalTrueVSize();
+   if (myrank == 0)
+   {
+        std::cout << "Number of finite element unknowns: " << glob_size << std::endl;
+   }
+
+   // 8. Define the Dirichlet conditions
+   mfem::Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+   ess_bdr = 1;
+
+   // 9. Define the nonlinear form
+   mfem::ParNonlinearForm* nf=new mfem::ParNonlinearForm(&fespace);
+
+   // 10. Define the solution vector x as a parallel finite element grid function
+   //     corresponding to fespace. Initialize x with initial guess of zero,
+   //     which satisfies the boundary conditions.
+   mfem::ParGridFunction x(&fespace);
+   x = 0.0;
+   mfem::HypreParVector* tv=x.GetTrueDofs();
+   mfem::HypreParVector* sv=x.GetTrueDofs();
+
+   // 11. Define ParaView DataCollection
+   mfem::ParaViewDataCollection *dacol=new mfem::ParaViewDataCollection("pLap",pmesh);
+   dacol->SetLevelsOfDetail(order);
+   dacol->RegisterField("sol",&x);
+
+
+   // 11. Set domain integrators - start with linear diffusion
+   {
+       // the default power coefficient is 2.0
+       mfem::ConstantCoefficient lpp(2.0);
+       if(integrator==0)
+       {
+            nf->AddDomainIntegrator(new mfem::pLaplace(lpp,c_ee,load));
+       }else
+       if(integrator==1)
+       {
+           nf->AddDomainIntegrator(new mfem::pLaplaceAD(lpp,c_ee,load));
+       }
+       nf->SetEssentialBC(ess_bdr);
+       // compute the energy
+       double energy=nf->GetEnergy(*tv);
+       if(myrank==0){
+           std::cout<<"[2] The total energy of the system is E="<<energy<<std::endl;}
+       // time the assembly
+       timer->Clear();
+       timer->Start();
+       mfem::Operator &op=nf->GetGradient(*sv);
+       timer->Stop();
+       if(myrank==0){
+           std::cout<<"[2] The assembly time is: "<<timer->RealTime()<<std::endl;}
+       mfem::Solver *prec=new mfem::HypreBoomerAMG();
+       mfem::GMRESSolver *j_gmres = new mfem::GMRESSolver(MPI_COMM_WORLD);
+       j_gmres->SetRelTol(1e-7);
+       j_gmres->SetAbsTol(1e-15);
+       j_gmres->SetMaxIter(300);
+       j_gmres->SetPrintLevel(print_level);
+       j_gmres->SetPreconditioner(*prec);
+
+       mfem::NewtonSolver* ns;
+       ns=new mfem::NewtonSolver(MPI_COMM_WORLD);
+       ns->iterative_mode = true;
+       ns->SetSolver(*j_gmres);
+       ns->SetOperator(*nf);
+       ns->SetPrintLevel(print_level);
+       ns->SetRelTol(1e-6);
+       ns->SetAbsTol(1e-12);
+       ns->SetMaxIter(3);
+       //solve the problem
+       timer->Clear();
+       timer->Start();
+       ns->Mult(*tv, *sv);
+       timer->Stop();
+       if(myrank==0){
+           std::cout<<"Time for the NewtonSolver: "<<timer->RealTime()<<std::endl;}
+
+       energy=nf->GetEnergy(*sv);
+       if(myrank==0){
+                 std::cout<<"[pp=2] The total energy of the system is E="<<energy<<std::endl;}
+
+       delete ns;
+       delete j_gmres;
+       delete prec;
+
+       x.SetFromTrueDofs(*sv);
+       dacol->SetTime(2.0);
+       dacol->SetCycle(2);
+       dacol->Save();
+   }
+
+   // 12. Continue with powers higher than 2
+   for(int i=3;i<pp;i++)
+   {
+       delete nf;
+       nf=new mfem::ParNonlinearForm(&fespace);
+       mfem::ConstantCoefficient lpp((double)i);
+       if(integrator==0)
+       {
+            nf->AddDomainIntegrator(new mfem::pLaplace(lpp,c_ee,load));
+       }else
+       if(integrator==1)
+       {
+           nf->AddDomainIntegrator(new mfem::pLaplaceAD(lpp,c_ee,load));
+       }
+       nf->SetEssentialBC(ess_bdr);
+       // compute the energy
+       double energy=nf->GetEnergy(*sv);
+       if(myrank==0){
+           std::cout<<"[pp="<<i<<"] The total energy of the system is E="<<energy<<std::endl;}
+       // time the assembly
+       timer->Clear();
+       timer->Start();
+       mfem::Operator &op=nf->GetGradient(*sv);
+       timer->Stop();
+       if(myrank==0){
+           std::cout<<"[pp="<<i<<"] The assembly time is: "<<timer->RealTime()<<std::endl;}
+       mfem::Solver *prec=new mfem::HypreBoomerAMG();
+       mfem::GMRESSolver *j_gmres = new mfem::GMRESSolver(MPI_COMM_WORLD);
+       j_gmres->SetRelTol(1e-7);
+       j_gmres->SetAbsTol(1e-15);
+       j_gmres->SetMaxIter(300);
+       j_gmres->SetPrintLevel(print_level);
+       j_gmres->SetPreconditioner(*prec);
+
+       mfem::NewtonSolver* ns;
+       ns=new mfem::NewtonSolver(MPI_COMM_WORLD);
+       ns->iterative_mode = true;
+       ns->SetSolver(*j_gmres);
+       ns->SetOperator(*nf);
+       ns->SetPrintLevel(print_level);
+       ns->SetRelTol(1e-6);
+       ns->SetAbsTol(1e-12);
+       ns->SetMaxIter(3);
+       //solve the problem
+       timer->Clear();
+       timer->Start();
+       ns->Mult(*tv, *sv);
+       timer->Stop();
+       if(myrank==0){
+           std::cout<<"Time for the NewtonSolver: "<<timer->RealTime()<<std::endl;}
+
+       energy=nf->GetEnergy(*sv);
+       if(myrank==0){
+                 std::cout<<"[pp="<<i<<"] The total energy of the system is E="<<energy<<std::endl;}
+
+       delete ns;
+       delete j_gmres;
+       delete prec;
+
+       x.SetFromTrueDofs(*sv);
+       dacol->SetTime(i);
+       dacol->SetCycle(i);
+       dacol->Save();
+   }
+
+   // 13. Continue with the final power
+   if( std::abs(pp-2.0) > std::numeric_limits<double>::epsilon())
+   {
+       delete nf;
+       nf=new mfem::ParNonlinearForm(&fespace);
+       if(integrator==0)
+       {
+            nf->AddDomainIntegrator(new mfem::pLaplace(c_pp,c_ee,load));
+       }else
+       if(integrator==1)
+       {
+           nf->AddDomainIntegrator(new mfem::pLaplaceAD(c_pp,c_ee,load));
+       }
+       nf->SetEssentialBC(ess_bdr);
+       // compute the energy
+       double energy=nf->GetEnergy(*sv);
+       if(myrank==0){
+           std::cout<<"[pp="<<pp<<"] The total energy of the system is E="<<energy<<std::endl;}
+       // time the assembly
+       timer->Clear();
+       timer->Start();
+       mfem::Operator &op=nf->GetGradient(*sv);
+       timer->Stop();
+       if(myrank==0){
+           std::cout<<"[pp="<<pp<<"] The assembly time is: "<<timer->RealTime()<<std::endl;}
+       mfem::Solver *prec=new mfem::HypreBoomerAMG();
+       mfem::GMRESSolver *j_gmres = new mfem::GMRESSolver(MPI_COMM_WORLD);
+       j_gmres->SetRelTol(1e-8);
+       j_gmres->SetAbsTol(1e-15);
+       j_gmres->SetMaxIter(300);
+       j_gmres->SetPrintLevel(print_level);
+       j_gmres->SetPreconditioner(*prec);
+
+       mfem::NewtonSolver* ns;
+       ns=new mfem::NewtonSolver(MPI_COMM_WORLD);
+       ns->iterative_mode = true;
+       ns->SetSolver(*j_gmres);
+       ns->SetOperator(*nf);
+       ns->SetPrintLevel(print_level);
+       ns->SetRelTol(1e-6);
+       ns->SetAbsTol(1e-12);
+       ns->SetMaxIter(3);
+       //solve the problem
+       timer->Clear();
+       timer->Start();
+       ns->Mult(*tv, *sv);
+       timer->Stop();
+       if(myrank==0){
+           std::cout<<"Time for the NewtonSolver: "<<timer->RealTime()<<std::endl;}
+
+       energy=nf->GetEnergy(*sv);
+       if(myrank==0){
+                 std::cout<<"[pp="<<pp<<"] The total energy of the system is E="<<energy<<std::endl;}
+
+       delete ns;
+       delete j_gmres;
+       delete prec;
+
+       x.SetFromTrueDofs(*sv);
+       dacol->SetTime(pp);
+       if(pp<2.0)
+       {
+           dacol->SetCycle(std::floor(pp));
+       }
+       else
+       {
+           dacol->SetCycle(std::ceil(pp));
+       }
+       dacol->Save();
+   }
+
+
+
+   // 19. Free the used memory
+   delete dacol;
+   delete sv;
+   delete tv;
+   delete nf;
+   delete pmesh;
+   delete timer;
+
+   MPI_Finalize();
+
+   return 0;
+}
+
+

examples/ex9.cpp

@@ -19,6 +19,7 @@
 //
 // Device sample runs:
 //    ex9 -pa
+//    ex9 -ea
 //    ex9 -pa -m ../data/periodic-cube.mesh
 //    ex9 -pa -m ../data/periodic-cube.mesh -d cuda
 //
@@ -142,6 +143,7 @@ int main(int argc, char *argv[])
    int ref_levels = 2;
    int order = 3;
    bool pa = false;
+   bool ea = false;
    const char *device_config = "cpu";
    int ode_solver_type = 4;
    double t_final = 10.0;
@@ -166,6 +168,8 @@ int main(int argc, char *argv[])
                   "Order (degree) of the finite elements.");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
                   "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&ea, "-ea", "--element-assembly", "-no-ea",
+                  "--no-element-assembly", "Enable Element Assembly.");
    args.AddOption(&device_config, "-d", "--device",
                   "Device configuration string, see Device::Configure().");
    args.AddOption(&ode_solver_type, "-s", "--ode-solver",
@@ -269,6 +273,11 @@ int main(int argc, char *argv[])
       m.SetAssemblyLevel(AssemblyLevel::PARTIAL);
       k.SetAssemblyLevel(AssemblyLevel::PARTIAL);
    }
+   else if (ea)
+   {
+      m.SetAssemblyLevel(AssemblyLevel::ELEMENT);
+      k.SetAssemblyLevel(AssemblyLevel::ELEMENT);
+   }
    m.AddDomainIntegrator(new MassIntegrator);
    k.AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
    k.AddInteriorFaceIntegrator(
@@ -429,8 +438,9 @@ FE_Evolution::FE_Evolution(BilinearForm &_M, BilinearForm &_K, const Vector &_b)
    : TimeDependentOperator(_M.Height()), M(_M), K(_K), b(_b), z(_M.Height())
 {
    bool pa = M.GetAssemblyLevel() == AssemblyLevel::PARTIAL;
+   bool ea = M.GetAssemblyLevel() == AssemblyLevel::ELEMENT;
    Array<int> ess_tdof_list;
-   if (pa)
+   if (pa || ea)
    {
       M_prec = new OperatorJacobiSmoother(M, ess_tdof_list);
       M_solver.SetOperator(M);

examples/ex9p.cpp

@@ -16,9 +16,11 @@
 //    mpirun -np 4 ex9p -m ../data/disc-nurbs.mesh -p 2 -rp 1 -dt 0.005 -tf 9
 //    mpirun -np 4 ex9p -m ../data/periodic-square.mesh -p 3 -rp 2 -dt 0.0025 -tf 9 -vs 20
 //    mpirun -np 4 ex9p -m ../data/periodic-cube.mesh -p 0 -o 2 -rp 1 -dt 0.01 -tf 8
+//    mpirun -np 3 ex9p -m ../data/amr-hex.mesh -p 1 -rs 1 -rp 0 -dt 0.005 -tf 0.5
 //
 // Device sample runs:
 //    mpirun -np 4 ex9p -pa
+//    mpirun -np 4 ex9p -ea
 //    mpirun -np 4 ex9p -pa -m ../data/periodic-cube.mesh
 //    mpirun -np 4 ex9p -pa -m ../data/periodic-cube.mesh -d cuda
 //
@@ -161,6 +163,7 @@ int main(int argc, char *argv[])
    int par_ref_levels = 0;
    int order = 3;
    bool pa = false;
+   bool ea = false;
    const char *device_config = "cpu";
    int ode_solver_type = 4;
    double t_final = 10.0;
@@ -188,6 +191,8 @@ int main(int argc, char *argv[])
                   "Order (degree) of the finite elements.");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
                   "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&ea, "-ea", "--element-assembly", "-no-ea",
+                  "--no-element-assembly", "Enable Element Assembly.");
    args.AddOption(&device_config, "-d", "--device",
                   "Device configuration string, see Device::Configure().");
    args.AddOption(&ode_solver_type, "-s", "--ode-solver",
@@ -319,6 +324,11 @@ int main(int argc, char *argv[])
       m->SetAssemblyLevel(AssemblyLevel::PARTIAL);
       k->SetAssemblyLevel(AssemblyLevel::PARTIAL);
    }
+   else if (ea)
+   {
+      m->SetAssemblyLevel(AssemblyLevel::ELEMENT);
+      k->SetAssemblyLevel(AssemblyLevel::ELEMENT);
+   }
    m->AddDomainIntegrator(new MassIntegrator);
    k->AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
    k->AddInteriorFaceIntegrator(
@@ -556,8 +566,9 @@ FE_Evolution::FE_Evolution(ParBilinearForm &_M, ParBilinearForm &_K,
      z(_M.Height())
 {
    bool pa = _M.GetAssemblyLevel()==AssemblyLevel::PARTIAL;
+   bool ea = _M.GetAssemblyLevel()==AssemblyLevel::ELEMENT;
 
-   if (pa)
+   if (pa || ea)
    {
       M.Reset(&_M, false);
       K.Reset(&_K, false);
@@ -571,7 +582,7 @@ FE_Evolution::FE_Evolution(ParBilinearForm &_M, ParBilinearForm &_K,
    M_solver.SetOperator(*M);
 
    Array<int> ess_tdof_list;
-   if (pa)
+   if (pa || ea)
    {
       M_prec = new OperatorJacobiSmoother(_M, ess_tdof_list);
       dg_solver = NULL;

examples/pumi/ex1.cpp

@@ -32,6 +32,13 @@
 //               is used for the Finite Element order and "-go" is used for the
 //               geometry order. Note that they can be used independently, i.e.
 //               "-o 8 -go 3" solves for 8th order FE on a third order geometry.
+//
+// NOTE:         Model/Mesh files for this example are in the (large) data file
+//               repository of MFEM here https://github.com/mfem/data under the
+//               folder named "pumi", which consists of the following sub-folders:
+//               a) geom -->  model files
+//               b) parallel --> parallel pumi mesh files
+//               c) serial --> serial pumi mesh files
 
 #include "mfem.hpp"
 #include <fstream>

examples/pumi/ex1p.cpp

@@ -36,6 +36,14 @@
 //               option "-o" is used for the Finite Element order and "-go" for
 //               the geometry order. Note that they can be used independently:
 //               "-o 8 -go 3" solves for 8th order FE on third order geometry.
+//
+// NOTE:         Model/Mesh files for this example are in the (large) data file
+//               repository of MFEM here https://github.com/mfem/data under the
+//               folder named "pumi", which consists of the following sub-folders:
+//               a) geom -->  model files
+//               b) parallel --> parallel pumi mesh files
+//               c) serial --> serial pumi mesh files
+
 
 #include "mfem.hpp"
 #include <fstream>

examples/pumi/ex2.cpp

@@ -43,6 +43,14 @@
 //               also illustrated.
 //
 //               We recommend viewing Example 1 before viewing this example.
+//
+// NOTE:         Model/Mesh files for this example are in the (large) data file
+//               repository of MFEM here https://github.com/mfem/data under the
+//               folder named "pumi", which consists of the following sub-folders:
+//               a) geom -->  model files
+//               b) parallel --> parallel pumi mesh files
+//               c) serial --> serial pumi mesh files
+
 
 #include "mfem.hpp"
 #include <fstream>

examples/pumi/ex6p.cpp

@@ -1,7 +1,7 @@
 //                       MFEM Example 6 - Parallel Version
 //                              PUMI Modification
 //
-// Compile with: make ex1p
+// Compile with: make ex6p
 //
 // Sample runs:  mpirun -np 8 ex6p
 //
@@ -18,6 +18,13 @@
 //               is added to modify the "adapt_ratio" which is the fraction of
 //               allowable error that scales the output size field of the error
 //               estimator.
+//
+// NOTE:         Model/Mesh files for this example are in the (large) data file
+//               repository of MFEM here https://github.com/mfem/data under the
+//               folder named "pumi", which consists of the following sub-folders:
+//               a) geom -->  model files
+//               b) parallel --> parallel pumi mesh files
+//               c) serial --> serial pumi mesh files
 
 #include "mfem.hpp"
 #include <fstream>
@@ -332,7 +339,6 @@ int main(int argc, char *argv[])
 
       apf::destroyField(Tmag_field);
       apf::destroyField(ipfield);
-      apf::destroyNumbering(pumi_mesh->findNumbering("LocalVertexNumbering"));
 
       // 18. Perform MesAdapt.
       ma::Input* erinput = ma::configure(pumi_mesh, sizefield);

fem/CMakeLists.txt

@@ -13,13 +13,20 @@ set(SRCS
   bilinearform.cpp
   bilinearform_ext.cpp
   bilininteg.cpp
-  bilininteg_convection.cpp
-  bilininteg_dgtrace.cpp
-  bilininteg_diffusion.cpp
+  bilininteg_convection_pa.cpp
+  bilininteg_convection_ea.cpp
+  bilininteg_dgtrace_pa.cpp
+  bilininteg_dgtrace_ea.cpp
+  bilininteg_diffusion_pa.cpp
+  bilininteg_diffusion_ea.cpp
   bilininteg_divergence.cpp
   bilininteg_hcurl.cpp
+  bilininteg_hdiv.cpp
+  bilininteg_vectorfe.cpp
   bilininteg_gradient.cpp
-  bilininteg_mass.cpp
+  bilininteg_mass_pa.cpp
+  bilininteg_mass_ea.cpp
+  bilininteg_transpose_ea.cpp
   bilininteg_vecdiffusion.cpp
   bilininteg_vecmass.cpp
   coefficient.cpp
@@ -49,6 +56,7 @@ set(SRCS
   tmop.cpp
   tmop_tools.cpp
   gslib.cpp
+  adnonlininteg.cpp
   transfer.cpp
   )
 
@@ -91,6 +99,7 @@ set(HDRS
   tmop.hpp
   tmop_tools.hpp
   gslib.hpp
+  adnonlininteg.hpp
   transfer.hpp
   )
 

fem/adios2datacollection.cpp

@@ -15,6 +15,8 @@
 
 #include "adios2datacollection.hpp"
 
+#ifdef MFEM_USE_ADIOS2
+
 namespace mfem
 {
 
@@ -87,4 +89,4 @@ noexcept
 
 } //end namespace mfem
 
-
+#endif // MFEM_USE_ADIOS2

fem/adios2datacollection.hpp

@@ -17,6 +17,9 @@
 #define MFEM_ADIOS2DATACOLLECTION
 
 #include "../config/config.hpp"
+
+#ifdef MFEM_USE_ADIOS2
+
 #include "../general/adios2stream.hpp"
 #include "datacollection.hpp"
 
@@ -85,4 +88,6 @@ private:
 
 }  // namespace mfem
 
+#endif // MFEM_USE_ADIOS2
+
 #endif /* MFEM_ADIOS2DATACOLLECTION */

fem/adnonlininteg.cpp

@@ -0,0 +1,408 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "fem.hpp"
+#include "../general/forall.hpp"
+#include "adnonlininteg.hpp"
+
+namespace mfem
+{
+
+
+void ADQFunctionJ::QFunctionDD(const Vector &vparam, const Vector &uu, DenseMatrix &jac)
+{
+#if defined MFEM_USE_ADEPT
+    //use ADEPT package
+    adept::Stack* p_stack=adept::active_stack();
+    p_stack->deactivate();
+
+    int n=uu.Size();
+    jac.SetSize(m,n);
+    jac=0.0;
+    m_stack.activate();
+    {
+        ADFVector aduu(uu);
+        ADFVector rr(m); //residual vector
+        m_stack.new_recording();
+        this->QFunctionDU(vparam,aduu,rr);
+        m_stack.independent(aduu.GetData(), n);//independent variables
+        m_stack.dependent(rr.GetData(), m);//dependent variables
+        m_stack.jacobian(jac.Data());
+    }
+    m_stack.deactivate();
+#elif defined MFEM_USE_CODIPACK
+#if defined MFEM_USE_ADFORWARD
+    //use CoDipack
+    int n=uu.Size();
+    jac.SetSize(m,n);
+    jac=0.0;
+    {
+        ADFVector aduu(n);
+        ADFVector rr(m);
+        for(int i=0;i<n;i++)
+        {
+            aduu[i]=uu[i];
+            aduu[i].setGradient(0.0);
+        }
+
+        for(int ii=0;ii<n;ii++){
+            aduu[ii].setGradient(1.0);
+            this->QFunctionDU(vparam,aduu,rr);
+            for(int jj=0;jj<m;jj++)
+            {
+                jac(jj,ii)=rr[jj].getGradient();
+            }
+            aduu[ii].setGradient(0.0);
+        }
+
+    }
+#else
+    //use CoDiPack in reverse mode
+    int n=uu.Size();
+    jac.SetSize(m,n);
+    jac=0.0;
+    {
+        ADFVector aduu(n);
+        ADFVector rr(m);
+        for(int i=0;i<n;i++)
+        {
+            aduu[i]=uu[i];
+        }
+
+        ADFType::TapeType& tape= ADFType::getGlobalTape();
+        typename ADFType::TapeType::Position pos=tape.getPosition();
+
+        tape.setActive();
+        for(int ii=0;ii<n;ii++){ tape.registerInput(aduu[ii]); }
+        this->QFunctionDU(vparam,aduu,rr);
+        for(int ii=0;ii<m;ii++){ tape.registerOutput(rr[ii]); }
+        tape.setPassive();
+
+        for(int jj=0;jj<m;jj++){
+            rr[jj].setGradient(1.0);
+            tape.evaluate();
+            for(int ii=0;ii<n;ii++){
+                jac(jj,ii)=aduu[ii].getGradient();
+            }
+            rr[jj].setGradient(0.0);
+        }
+        tape.reset(pos);
+    }
+#endif
+#elif defined MFEM_USE_FADBADPP
+    //use FADBAD++
+    #ifdef MFEM_USE_ADFORWARD
+        int n=uu.Size();
+        jac.SetSize(m,n);
+        jac=0.0;
+        {
+            ADFVector aduu(uu);
+            ADFVector rr(m);
+
+            for(int ii=0;ii<n;ii++){
+                aduu[ii].diff(ii,n);
+            }
+            this->QFunctionDU(vparam,aduu,rr);
+            for(int ii=0;ii<n;ii++){
+                for(int jj=0;jj<m;jj++)
+                {
+                    jac(jj,ii)=rr[jj].d(ii);
+                }
+            }
+        }
+    #else
+        int n=uu.Size();
+        jac.SetSize(m,n);
+        jac=0.0;
+        {
+            ADFVector aduu(uu);
+            ADFVector rr(m);
+            this->QFunctionDU(vparam,aduu,rr);
+            for(int ii=0;ii<m;ii++)
+            {
+                rr[ii].diff(ii,m);
+            }
+            for(int ii=0;ii<n;ii++){
+                for(int jj=0;jj<m;jj++)
+                {
+                    jac(jj,ii)=aduu[ii].d(jj);
+                }
+            }
+
+        }
+    #endif
+#else
+    //use native AD package
+   int n=uu.Size();
+   jac.SetSize(m,n);
+   jac=0.0;
+   {
+       ADFVector aduu(uu); //all dual numbers are initialized to zero
+       ADFVector rr(m);
+
+       for(int ii=0;ii<n;ii++){
+           aduu[ii].dual(1.0);
+           this->QFunctionDU(vparam,aduu,rr);
+           for(int jj=0;jj<m;jj++)
+           {
+               jac(jj,ii)=rr[jj].dual();
+           }
+           aduu[ii].dual(0.0);
+       }
+   }
+#endif
+}
+
+void ADQFunctionH::QFunctionDU(const Vector &vparam, Vector &uu, Vector &rr)
+{
+
+#if defined MFEM_USE_CODIPACK
+    int n=uu.Size();
+    rr.SetSize(n);
+    ADFVector aduu(n);
+    ADFType   rez;
+    for(int ii=0;ii<n;ii++)
+    {
+        aduu[ii].setValue(uu[ii]);
+        aduu[ii].setGradient(0.0);
+    }
+    for(int ii=0;ii<n;ii++)
+    {
+        aduu[ii].setGradient(1.0);
+        rez=this->QFunction(vparam,aduu);
+        rr[ii]=rez.getGradient();
+        aduu[ii].setGradient(0.0);
+    }
+#elif defined MFEM_USE_FADBADPP
+    int n=uu.Size();
+    rr.SetSize(n);
+    ADFVector aduu(uu);
+    ADFType   rez;
+    rez=this->QFunction(vparam,aduu);
+    rez.diff(0,1);
+    for(int ii=0;ii<n;ii++)
+    {
+        rr[ii]=aduu[ii].d(0);
+    }
+#else
+    int n=uu.Size();
+    rr.SetSize(n);
+    ADFVector aduu(uu);
+    ADFType   rez;
+    for(int ii=0;ii<n;ii++)
+    {
+        aduu[ii].dual(1.0);
+        rez=this->QFunction(vparam,aduu);
+        rr[ii]=rez.dual();
+        aduu[ii].dual(0.0);
+    }
+#endif
+}
+
+void ADQFunctionH::QFunctionDD(const Vector &vparam, const Vector &uu, DenseMatrix &jac)
+{
+
+#if defined MFEM_USE_CODIPACK
+#if defined MFEM_USE_ADFORWARD
+    int n=uu.Size();
+    jac.SetSize(n);
+    jac=0.0;
+    {
+        ADSVector aduu(n);
+        for(int ii = 0;  ii < n ; ii++)
+        {
+            aduu[ii].value().value()=uu[ii];
+            aduu[ii].value().gradient()=0.0;
+            aduu[ii].gradient().value()=0.0;
+            aduu[ii].gradient().gradient()=0.0;
+        }
+
+        for(int ii = 0; ii < n ; ii++)
+        {
+            aduu[ii].value().gradient()=1.0;
+            for(int jj=0; jj<(ii+1); jj++)
+            {
+                aduu[ii].gradient().value()=1.0;
+                ADSType rez= this->QFunction(vparam,aduu);
+                jac(ii,jj)=rez.gradient().gradient();
+                jac(jj,ii)=jac(ii,jj);
+                aduu[jj].gradient().value()=0.0;
+            }
+            aduu[ii].value().gradient()=0.0;
+        }
+    }
+#else
+    int n=uu.Size();
+    jac.SetSize(n);
+    jac=0.0;
+    {
+        ADSVector aduu(n);
+        for(int ii=0;ii < n ; ii++)
+        {
+            aduu[ii].value().value()=uu[ii];
+        }
+
+        ADSType rez;
+        ADSType::TapeType& tape = ADSType::getGlobalTape();
+        typename ADSType::TapeType::Position pos;
+
+        for(int ii = 0; ii < n ; ii++)
+        {
+            pos=tape.getPosition();
+            tape.setActive();
+
+            for(int jj=0;jj < n; jj++) {
+                if(jj==ii) {aduu[jj].value().gradient()=1.0;}
+                else {aduu[jj].value().gradient()=0.0;}
+                tape.registerInput(aduu[jj]);
+            }
+
+            rez=this->QFunction(vparam,aduu);
+            tape.registerOutput(rez);
+            tape.setPassive();
+
+            rez.gradient().value()=1.0;
+            tape.evaluate();
+
+            for(int jj=0; jj<(ii+1); jj++)
+            {
+                jac(ii,jj)=aduu[jj].gradient().gradient();
+                jac(jj,ii)=jac(ii,jj);
+            }
+            tape.reset(pos);
+        }
+    }
+#endif
+#elif defined MFEM_USE_FADBADPP
+    int n=uu.Size();
+    jac.SetSize(n);
+    jac=0.0;
+    {
+        ADSVector aduu(n);
+        for(int ii = 0;  ii < n ; ii++)
+        {
+            aduu[ii]=uu[ii];
+            aduu[ii].x().diff(ii,n);
+        }
+        ADSType rez= this->QFunction(vparam,aduu);
+        rez.diff(0,1);
+        for(int ii = 0; ii < n ; ii++)
+        {
+            for(int jj=0; jj<ii; jj++)
+            {
+                jac(ii,jj)=aduu[ii].d(0).d(jj);
+                jac(jj,ii)=aduu[jj].d(0).d(ii);
+            }
+            jac(ii,ii)=aduu[ii].d(0).d(ii);
+        }
+    }
+#else
+    int n=uu.Size();
+    jac.SetSize(n);
+    jac=0.0;
+    {
+        ADSVector aduu(n);
+        for(int ii = 0;  ii < n ; ii++)
+        {
+            aduu[ii].real(ADFType(uu[ii],0.0));
+            aduu[ii].dual(ADFType(0.0,0.0));
+        }
+
+        for(int ii = 0; ii < n ; ii++)
+        {
+            aduu[ii].real(ADFType(uu[ii],1.0));
+            for(int jj=0; jj<(ii+1); jj++)
+            {
+                aduu[jj].dual(ADFType(1.0,0.0));
+                ADSType rez= this->QFunction(vparam,aduu);
+                jac(ii,jj)=rez.dual().dual();
+                jac(jj,ii)=rez.dual().dual();
+                aduu[jj].dual(ADFType(0.0,0.0));
+            }
+            aduu[ii].real(ADFType(uu[ii],0.0));
+        }
+    }
+#endif
+}
+
+
+
+
+
+double ADNonlinearFormIntegratorH::GetElementEnergy(const mfem::FiniteElement & el,
+                                    mfem::ElementTransformation & Tr, 
+                                    const mfem::Vector & elfun)
+{
+    return this->ElementEnergy(el,Tr,elfun);
+}
+
+void ADNonlinearFormIntegratorH::AssembleElementVector(const mfem::FiniteElement & el,
+                                            mfem::ElementTransformation & Tr, 
+                                            const mfem::Vector & elfun, mfem::Vector & elvect)
+{
+    
+    int ndof = el.GetDof();
+    elvect.SetSize(ndof);
+    
+    {
+        ADFVector adelfun(elfun);
+        //all dual numbers in adelfun are initialized to 0.0
+        for(int ii = 0; ii < adelfun.Size(); ii++)
+        {
+            //set the dual for the ii^th element to 1.0
+            adelfun[ii].dual(1.0);
+            ADFType rez= this->ElementEnergy(el,Tr, adelfun);
+            elvect[ii]=rez.dual();
+            //return it back to zero
+            adelfun[ii].dual(0.0);
+        }
+    }
+    
+}
+
+void ADNonlinearFormIntegratorH::AssembleElementGrad(const mfem::FiniteElement & el,
+                                          mfem::ElementTransformation & Tr, 
+                                          const mfem::Vector & elfun, 
+                                          mfem::DenseMatrix & elmat)
+{
+    
+    int ndof = el.GetDof();
+    elmat.SetSize(ndof);
+    elmat=0.0;
+    {
+        ADSVector adelfun(ndof);
+        for(int ii = 0; ii < ndof; ii++)
+        {
+            adelfun[ii].real(ADFType(elfun[ii],0.0));
+            adelfun[ii].dual(ADFType(0.0,0.0));
+        }
+        
+        for(int ii = 0; ii < adelfun.Size(); ii++)
+        {
+            adelfun[ii].real(ADFType(elfun[ii],1.0));
+            for(int jj = 0; jj < (ii+1); jj++)
+            {
+                adelfun[jj].dual(ADFType(1.0,0.0));
+                ADSType rez= this->ElementEnergy(el,Tr, adelfun);
+                elmat(ii,jj)=rez.dual().dual();
+                elmat(jj,ii)=rez.dual().dual();
+                adelfun[jj].dual(ADFType(0.0,0.0));
+            }
+            adelfun[ii].real(ADFType(elfun[ii],0.0));
+        }
+        
+    }
+    
+}
+
+    
+} //end namespace mfem
+

fem/adnonlininteg.hpp

@@ -0,0 +1,204 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+
+
+#ifndef MFEM_ADNONLININTEG
+#define MFEM_ADNONLININTEG
+
+#include "../config/config.hpp"
+#include "fe.hpp"
+#include "coefficient.hpp"
+#include "fespace.hpp"
+#include "nonlininteg.hpp"
+#include "../linalg/tadvector.hpp"
+#include "../linalg/taddensemat.hpp"
+#include "../linalg/fdual.hpp"
+
+#if defined MFEM_USE_ADEPT
+#include <adept.h>
+#elif defined  MFEM_USE_CODIPACK
+#include <codi.hpp>
+#elif defined  MFEM_USE_FADBADPP
+#include <fadiff.h>
+#include <badiff.h>
+#endif
+
+//define Forward AD mode
+//#define MFEM_USE_ADFORWARD
+
+namespace mfem
+{
+
+class ADQFunctionJ
+{
+  private:
+    int m;  //dimension of the residual vector
+            //the Jacobian will have dimensions [m,length(uu)]
+  protected:protected:
+#ifdef MFEM_USE_ADEPT
+    adept::Stack  m_stack;
+#endif
+
+  public:
+#if defined MFEM_USE_ADEPT
+    typedef adept::adouble 			   ADFType;
+    typedef TADVector<ADFType>         ADFVector;
+    typedef TADDenseMatrix<ADFType>    ADFDenseMatrix;
+#elif defined MFEM_USE_CODIPACK
+    #if defined MFEM_USE_ADFORWARD
+        typedef codi::RealForward 		   ADFType;
+        typedef TADVector<ADFType>         ADFVector;
+        typedef TADDenseMatrix<ADFType>    ADFDenseMatrix;
+    #else
+        typedef codi::RealRevers 		   ADFType;
+        typedef TADVector<ADFType>         ADFVector;
+        typedef TADDenseMatrix<ADFType>    ADFDenseMatrix;
+    #endif
+#elif defined MFEM_USE_FADBADPP
+    #ifdef MFEM_USE_ADFORWARD
+        typedef fadbad::F<double>		   ADFType;
+        typedef TADVector<ADFType>         ADFVector;
+        typedef TADDenseMatrix<ADFType>    ADFDenseMatrix;
+    #else
+        typedef fadbad::B<double>		   ADFType;
+        typedef TADVector<ADFType>         ADFVector;
+        typedef TADDenseMatrix<ADFType>    ADFDenseMatrix;
+    #endif
+#else
+    typedef mfem::ad::FDual<double>    ADFType;
+    typedef TADVector<ADFType>         ADFVector;
+    typedef TADDenseMatrix<ADFType>    ADFDenseMatrix;
+#endif
+
+#ifdef MFEM_USE_ADEPT
+    ADQFunctionJ(int m_=1):m_stack(false)
+    {
+        m=m_;
+    }
+#else
+    ADQFunctionJ(int m_=1){ m=m_;}
+#endif
+
+    virtual ~ADQFunctionJ(){}
+
+    virtual double QFunction(const mfem::Vector& vparam, const mfem::Vector& uu)=0;
+    virtual void QFunctionDU(const mfem::Vector& vparam, ADFVector& uu, ADFVector& rr)=0;
+    virtual void QFunctionDU(const mfem::Vector& vparam, mfem::Vector& uu, mfem::Vector& rr)=0;
+
+    void QFunctionDD(const mfem::Vector& vparam, const mfem::Vector& uu, mfem::DenseMatrix& jj);
+
+};
+
+
+class ADQFunctionH
+{
+  public:
+#if defined MFEM_USE_CODIPACK
+#if defined MFEM_USE_ADFORWARD
+    //use forward mode for both the first and the second derivatives
+    typedef codi::RealForwardGen<double> ADFType;
+    typedef TADVector<ADFType>        	 ADFVector;
+    typedef TADDenseMatrix<ADFType>    	 ADFDenseMatrix;
+
+    typedef codi::RealForwardGen<ADFType> 	ADSType;
+    typedef TADVector<ADSType>         		ADSVector;
+    typedef TADDenseMatrix<ADSType>    		ADSDenseMatrix;
+#else
+    //use mixed forward and reverse mode
+    typedef codi::RealForwardGen<double> ADFType;
+    typedef TADVector<ADFType>        	 ADFVector;
+    typedef TADDenseMatrix<ADFType>    	 ADFDenseMatrix;
+
+    typedef codi::RealReverseGen<ADFType>   ADSType;
+    typedef TADVector<ADSType>         		ADSVector;
+    typedef TADDenseMatrix<ADSType>    		ADSDenseMatrix;
+#endif
+#elif defined MFEM_USE_FADBADPP
+        typedef fadbad::B<double> 			 ADFType;
+        typedef TADVector<ADFType>        	 ADFVector;
+        typedef TADDenseMatrix<ADFType>    	 ADFDenseMatrix;
+
+        typedef fadbad::B<fadbad::F<double>>    ADSType;
+        typedef TADVector<ADSType>         		ADSVector;
+        typedef TADDenseMatrix<ADSType>    		ADSDenseMatrix;
+#else
+    typedef mfem::ad::FDual<double>    ADFType;
+    typedef TADVector<ADFType>         ADFVector;
+    typedef TADDenseMatrix<ADFType>    ADFDenseMatrix;
+
+    typedef mfem::ad::FDual<ADFType>   ADSType;
+    typedef TADVector<ADSType>         ADSVector;
+    typedef TADDenseMatrix<ADSType>    ADSDenseMatrix;
+#endif
+
+    ADQFunctionH(){}
+
+    virtual ~ADQFunctionH(){}
+
+    virtual double QFunction(const mfem::Vector& vparam, const mfem::Vector& uu)=0;
+    virtual ADFType QFunction(const mfem::Vector& vparam, ADFVector& uu)=0;
+    virtual ADSType QFunction(const mfem::Vector &vparam, ADSVector& uu)=0;
+
+    virtual void QFunctionDU(const mfem::Vector& vparam, mfem::Vector& uu, mfem::Vector& rr);
+    void QFunctionDD(const mfem::Vector& vparam, const mfem::Vector& uu, mfem::DenseMatrix& jj);
+};
+
+
+
+class ADNonlinearFormIntegratorH: public NonlinearFormIntegrator
+{
+public:
+    
+    typedef mfem::ad::FDual<double>    ADFType;
+    typedef TADVector<ADFType>         ADFVector;
+    typedef TADDenseMatrix<ADFType>    ADFDenseMatrix;
+    
+    typedef mfem::ad::FDual<ADFType>   ADSType;
+    typedef TADVector<ADSType>         ADSVector;
+    typedef TADDenseMatrix<ADSType>    ADSDenseMatrix;
+    
+    ADNonlinearFormIntegratorH(){}
+    
+    virtual ~ADNonlinearFormIntegratorH(){}
+    
+    virtual ADSType ElementEnergy(const mfem::FiniteElement & el,
+                                 mfem::ElementTransformation & Tr,
+                                 const ADSVector & elfun)=0;
+                                 
+    virtual ADFType ElementEnergy(const mfem::FiniteElement & el,
+                                 mfem::ElementTransformation & Tr,
+                                 const ADFVector & elfun)=0;
+                                
+    virtual double ElementEnergy(const mfem::FiniteElement & el, 
+                                 mfem::ElementTransformation & Tr,
+                                 const mfem::Vector & elfun)=0;
+                                    
+                                    
+    virtual double GetElementEnergy(const mfem::FiniteElement & el, 
+                                    mfem::ElementTransformation & Tr, 
+                                    const mfem::Vector & elfun) override;
+    
+    virtual void AssembleElementVector(const mfem::FiniteElement & el,
+                                            mfem::ElementTransformation & Tr, 
+                                            const mfem::Vector & elfun, mfem::Vector & elvect) override;
+    
+    virtual void AssembleElementGrad(const mfem::FiniteElement & el, 
+                                          mfem::ElementTransformation & Tr, 
+                                          const mfem::Vector & elfun, 
+                                          mfem::DenseMatrix & elmat) override;
+};
+
+
+}
+
+#endif    
+

fem/bilinearform.cpp

@@ -126,8 +126,7 @@ void BilinearForm::SetAssemblyLevel(AssemblyLevel assembly_level)
          // Use the original BilinearForm implementation for now
          break;
       case AssemblyLevel::ELEMENT:
-         mfem_error("Element assembly not supported yet... stay tuned!");
-         // ext = new EABilinearFormExtension(this);
+         ext = new EABilinearFormExtension(this);
          break;
       case AssemblyLevel::PARTIAL:
          ext = new PABilinearFormExtension(this);
@@ -1432,6 +1431,54 @@ void MixedBilinearForm::Assemble (int skip_zeros)
    }
 }
 
+void MixedBilinearForm::AssembleDiagonal_ADAt(const Vector &D,
+                                              Vector &diag) const
+{
+   if (ext)
+   {
+      MFEM_ASSERT(diag.Size() == test_fes->GetTrueVSize(),
+                  "Vector for holding diagonal has wrong size!");
+      MFEM_ASSERT(D.Size() == trial_fes->GetTrueVSize(),
+                  "Vector for holding diagonal has wrong size!");
+      const Operator *P_trial = trial_fes->GetProlongationMatrix();
+      const Operator *P_test = test_fes->GetProlongationMatrix();
+      if (!IsIdentityProlongation(P_trial))
+      {
+         Vector local_D(P_trial->Height());
+         P_trial->Mult(D, local_D);
+
+         if (!IsIdentityProlongation(P_test))
+         {
+            Vector local_diag(P_test->Height());
+            ext->AssembleDiagonal_ADAt(local_D, local_diag);
+            P_test->MultTranspose(local_diag, diag);
+         }
+         else
+         {
+            ext->AssembleDiagonal_ADAt(local_D, diag);
+         }
+      }
+      else
+      {
+         if (!IsIdentityProlongation(P_test))
+         {
+            Vector local_diag(P_test->Height());
+            ext->AssembleDiagonal_ADAt(D, local_diag);
+            P_test->MultTranspose(local_diag, diag);
+         }
+         else
+         {
+            ext->AssembleDiagonal_ADAt(D, diag);
+         }
+      }
+   }
+   else
+   {
+      MFEM_ABORT("Not implemented. Maybe assemble your bilinear form into a "
+                 "matrix and use SparseMatrix functions?");
+   }
+}
+
 void MixedBilinearForm::ConformingAssemble()
 {
    if (assembly != AssemblyLevel::FULL)

fem/bilinearform.hpp

@@ -25,8 +25,8 @@
 namespace mfem
 {
 
-/// Enumeration defining the assembly level for bilinear and nonlinear form
-/// classes derived from Operator.
+/** @brief Enumeration defining the assembly level for bilinear and nonlinear
+    form classes derived from Operator. */
 enum class AssemblyLevel
 {
    /// Fully assembled form, i.e. a global sparse matrix in MFEM, Hypre or PETSC
@@ -44,15 +44,19 @@ enum class AssemblyLevel
 };
 
 
-/** Class for bilinear form - "Matrix" with associated FE space and
-    BLFIntegrators. */
+/** @brief A "square matrix" operator for the associated FE space and
+    BLFIntegrators The sum of all the BLFIntegrators can be used form the matrix
+    M. This class also supports other assembly levels specified via the
+    SetAssemblyLevel() function. */
 class BilinearForm : public Matrix
 {
 protected:
-   /// Sparse matrix to be associated with the form. Owned.
+   /// Sparse matrix \f$ M \f$ to be associated with the form. Owned.
    SparseMatrix *mat;
 
-   /// Matrix used to eliminate b.c. Owned.
+   /** @brief Sparse Matrix \f$ M_e \f$ used to store the eliminations
+        from the b.c.  Owned.
+       \f$ M + M_e = M_{original} \f$ */
    SparseMatrix *mat_e;
 
    /// FE space on which the form lives. Not owned.
@@ -62,12 +66,12 @@ protected:
    AssemblyLevel assembly;
    /// Element batch size used in the form action (1, 8, num_elems, etc.)
    int batch;
-   /** Extension for supporting Full Assembly (FA), Element Assembly (EA),
+   /** @brief Extension for supporting Full Assembly (FA), Element Assembly (EA),
        Partial Assembly (PA), or Matrix Free assembly (MF). */
    BilinearFormExtension *ext;
 
-   /// Indicates the Mesh::sequence corresponding to the current state of the
-   /// BilinearForm.
+   /** @brief Indicates the Mesh::sequence corresponding to the current state of
+       the BilinearForm. */
    long sequence;
 
    /** @brief Indicates the BilinearFormIntegrator%s stored in #dbfi, #bbfi,
@@ -147,35 +151,43 @@ public:
    /// Get the size of the BilinearForm as a square matrix.
    int Size() const { return height; }
 
-   /// Set the desired assembly level. The default is AssemblyLevel::FULL.
-   /** This method must be called before assembly. */
+   /// Set the desired assembly level.
+   /** Valid choices are:
+
+       - AssemblyLevel::FULL  (default)
+       - AssemblyLevel::PARTIAL
+       - AssemblyLevel::ELEMENT
+       - AssemblyLevel::NONE
+
+       This method must be called before assembly. */
    void SetAssemblyLevel(AssemblyLevel assembly_level);
 
    /// Returns the assembly level
    AssemblyLevel GetAssemblyLevel() const { return assembly; }
 
-   /** Enable the use of static condensation. For details see the description
-       for class StaticCondensation in fem/staticcond.hpp This method should be
-       called before assembly. If the number of unknowns after static
+   /** @brief Enable the use of static condensation. For details see the
+       description for class StaticCondensation in fem/staticcond.hpp This method
+       should be called before assembly. If the number of unknowns after static
        condensation is not reduced, it is not enabled. */
    void EnableStaticCondensation();
 
-   /** Check if static condensation was actually enabled by a previous call to
-       EnableStaticCondensation(). */
+   /** @brief Check if static condensation was actually enabled by a previous
+       call to EnableStaticCondensation(). */
    bool StaticCondensationIsEnabled() const { return static_cond; }
 
    /// Return the trace FE space associated with static condensation.
    FiniteElementSpace *SCFESpace() const
    { return static_cond ? static_cond->GetTraceFESpace() : NULL; }
 
-   /** Enable hybridization; for details see the description for class
+   /// Enable hybridization.
+   /** For details see the description for class
        Hybridization in fem/hybridization.hpp. This method should be called
        before assembly. */
    void EnableHybridization(FiniteElementSpace *constr_space,
                             BilinearFormIntegrator *constr_integ,
                             const Array<int> &ess_tdof_list);
 
-   /** For scalar FE spaces, precompute the sparsity pattern of the matrix
+   /** @brief For scalar FE spaces, precompute the sparsity pattern of the matrix
        (assuming dense element matrices) based on the types of integrators
        present in the bilinear form. */
    void UsePrecomputedSparsity(int ps = 1) { precompute_sparsity = ps; }
@@ -194,15 +206,16 @@ public:
    /// Use the sparsity of @a A to allocate the internal SparseMatrix.
    void UseSparsity(SparseMatrix &A);
 
-   /** Pre-allocate the internal SparseMatrix before assembly. If the flag
-       'precompute sparsity' is set, the matrix is allocated in CSR format (i.e.
+   /// Pre-allocate the internal SparseMatrix before assembly.
+   /**  If the flag 'precompute sparsity'
+       is set, the matrix is allocated in CSR format (i.e.
        finalized) and the entries are initialized with zeros. */
    void AllocateMatrix() { if (mat == NULL) { AllocMat(); } }
 
-   /// Access all integrators added with AddDomainIntegrator().
+   /// Access all the integrators added with AddDomainIntegrator().
    Array<BilinearFormIntegrator*> *GetDBFI() { return &dbfi; }
 
-   /// Access all integrators added with AddBoundaryIntegrator().
+   /// Access all the integrators added with AddBoundaryIntegrator().
    Array<BilinearFormIntegrator*> *GetBBFI() { return &bbfi; }
    /** @brief Access all boundary markers added with AddBoundaryIntegrator().
        If no marker was specified when the integrator was added, the
@@ -219,64 +232,85 @@ public:
        corresponding pointer (to Array<int>) will be NULL. */
    Array<Array<int>*> *GetBFBFI_Marker() { return &bfbfi_marker; }
 
+   /// Returns a reference to: \f$ M_{ij} \f$
    const double &operator()(int i, int j) { return (*mat)(i,j); }
 
-   /// Returns reference to a_{ij}.
+   /// Returns a reference to: \f$ M_{ij} \f$
    virtual double &Elem(int i, int j);
 
-   /// Returns constant reference to a_{ij}.
+   /// Returns constant reference to: \f$ M_{ij} \f$
    virtual const double &Elem(int i, int j) const;
 
-   /// Matrix vector multiplication.
+   /// Matrix vector multiplication:  \f$ y = M x \f$
    virtual void Mult(const Vector &x, Vector &y) const;
 
+   /** @brief Matrix vector multiplication with the original uneliminated
+       matrix.  The original matrix is \f$ M + M_e \f$ so we have:
+       \f$ y = M x + M_e x \f$ */
    void FullMult(const Vector &x, Vector &y) const
    { mat->Mult(x, y); mat_e->AddMult(x, y); }
 
+   /// Add the matrix vector multiple to a vector:  \f$ y += a M x \f$
    virtual void AddMult(const Vector &x, Vector &y, const double a = 1.0) const
    { mat -> AddMult (x, y, a); }
 
+   /** @brief Add the original uneliminated matrix vector multiple to a vector.
+       The original matrix is \f$ M + Me \f$ so we have:
+       \f$ y += M x + M_e x \f$ */
    void FullAddMult(const Vector &x, Vector &y) const
    { mat->AddMult(x, y); mat_e->AddMult(x, y); }
 
+   /// Add the matrix transpose vector multiplication:  \f$ y += a M^T x \f$
    virtual void AddMultTranspose(const Vector & x, Vector & y,
                                  const double a = 1.0) const
    { mat->AddMultTranspose(x, y, a); }
 
+   /** @brief Add the original uneliminated matrix transpose vector
+       multiple to a vector. The original matrix is \f$ M + M_e \f$
+       so we have: \f$ y += M^T x + {M_e}^T x \f$ */
    void FullAddMultTranspose(const Vector & x, Vector & y) const
    { mat->AddMultTranspose(x, y); mat_e->AddMultTranspose(x, y); }
 
+   /// Matrix transpose vector multiplication:  \f$ y = M^T x \f$
    virtual void MultTranspose(const Vector & x, Vector & y) const
    { y = 0.0; AddMultTranspose (x, y); }
 
+   /// Compute \f$ y^T M x \f$
    double InnerProduct(const Vector &x, const Vector &y) const
    { return mat->InnerProduct (x, y); }
 
-   /// Returns a pointer to (approximation) of the matrix inverse.
+   /// Returns a pointer to (approximation) of the matrix inverse:  \f$ M^{-1} \f$
    virtual MatrixInverse *Inverse() const;
 
    /// Finalizes the matrix initialization.
    virtual void Finalize(int skip_zeros = 1);
 
-   /// Returns a reference to the sparse matrix
+   /// Returns a const reference to the sparse matrix.
    const SparseMatrix &SpMat() const
    {
       MFEM_VERIFY(mat, "mat is NULL and can't be dereferenced");
       return *mat;
    }
+
+   /// Returns a reference to the sparse matrix:  \f$ M \f$
    SparseMatrix &SpMat()
    {
       MFEM_VERIFY(mat, "mat is NULL and can't be dereferenced");
       return *mat;
    }
+
+   /**  @brief Nullifies the internal matrix \f$ M \f$ and returns a pointer
+        to it.  Used for transfering ownership. */
    SparseMatrix *LoseMat() { SparseMatrix *tmp = mat; mat = NULL; return tmp; }
 
-   /// Returns a reference to the sparse matrix of eliminated b.c.
+   /// Returns a const reference to the sparse matrix of eliminated b.c.: \f$ M_e \f$
    const SparseMatrix &SpMatElim() const
    {
       MFEM_VERIFY(mat_e, "mat_e is NULL and can't be dereferenced");
       return *mat_e;
    }
+
+   /// Returns a reference to the sparse matrix of eliminated b.c.: \f$ M_e \f$
    SparseMatrix &SpMatElim()
    {
       MFEM_VERIFY(mat_e, "mat_e is NULL and can't be dereferenced");
@@ -311,6 +345,7 @@ public:
    void AddBdrFaceIntegrator(BilinearFormIntegrator *bfi,
                              Array<int> &bdr_marker);
 
+   /// Sets all sparse values of \f$ M \f$ and \f$ M_e \f$ to 'a'.
    void operator=(const double a)
    {
       if (mat != NULL) { *mat = a; }
@@ -328,10 +363,10 @@ public:
        for an AMR mesh. */
    void AssembleDiagonal(Vector &diag) const;
 
-   /// Get the finite element space prolongation matrix
+   /// Get the finite element space prolongation operator.
    virtual const Operator *GetProlongation() const
    { return fes->GetConformingProlongation(); }
-   /// Get the finite element space restriction matrix
+   /// Get the finite element space restriction operator
    virtual const Operator *GetRestriction() const
    { return fes->GetConformingRestriction(); }
    /// Get the output finite element space prolongation matrix
@@ -491,10 +526,12 @@ public:
                                  double value);
 
    /// Eliminate the given @a vdofs. NOTE: here, @a vdofs is a list of DOFs.
+   /** In this case the eliminations are applied to the internal \f$ M \f$
+       and @a rhs without storing the elimination matrix \f$ M_e \f$. */
    void EliminateVDofs(const Array<int> &vdofs, const Vector &sol, Vector &rhs,
                        DiagonalPolicy dpolicy = DIAG_ONE);
 
-   /// Eliminate the given @a vdofs, storing the eliminated part internally.
+   /// Eliminate the given @a vdofs, storing the eliminated part internally in \f$ M_e \f$.
    /** This method works in conjunction with EliminateVDofsInRHS() and allows
        elimination of boundary conditions in multiple right-hand sides. In this
        method, @a vdofs is a list of DOFs. */
@@ -523,9 +560,11 @@ public:
    void EliminateVDofsInRHS(const Array<int> &vdofs, const Vector &x,
                             Vector &b);
 
+   /// Compute inner product for full uneliminated matrix \f$ y^T M x + y^T M_e x \f$
    double FullInnerProduct(const Vector &x, const Vector &y) const
    { return mat->InnerProduct(x, y) + mat_e->InnerProduct(x, y); }
 
+   /// Update the @a FiniteElementSpace and delete all data associated with the old one.
    virtual void Update(FiniteElementSpace *nfes = NULL);
 
    /// (DEPRECATED) Return the FE space associated with the BilinearForm.
@@ -537,7 +576,13 @@ public:
    /// Read-only access to the associated FiniteElementSpace.
    const FiniteElementSpace *FESpace() const { return fes; }
 
-   /// Sets diagonal policy used upon construction of the linear system
+   /// Sets diagonal policy used upon construction of the linear system.
+   /** Policies include:
+
+       - DIAG_ZERO (Set the diagonal values to zero)
+       - DIAG_ONE  (Set the diagonal values to one)
+       - DIAG_KEEP (Keep the diagonal values)
+   */
    void SetDiagonalPolicy(DiagonalPolicy policy);
 
    /// Indicate that integrators are not owned by the BilinearForm
@@ -550,16 +595,16 @@ public:
 
 /**
    Class for assembling of bilinear forms `a(u,v)` defined on different
-   trial and test spaces. The assembled matrix `A` is such that
+   trial and test spaces. The assembled matrix `M` is such that
 
-       a(u,v) = V^t A U
+       a(u,v) = V^t M U
 
    where `U` and `V` are the vectors representing the functions `u` and `v`,
    respectively.  The first argument, `u`, of `a(,)` is in the trial space
    and the second argument, `v`, is in the test space. Thus,
 
-       # of rows of A = dimension of the test space and
-       # of cols of A = dimension of the trial space.
+       # of rows of M = dimension of the test space and
+       # of cols of M = dimension of the trial space.
 
    Both trial and test spaces should be defined on the same mesh.
 */
@@ -628,11 +673,15 @@ public:
                      FiniteElementSpace *te_fes,
                      MixedBilinearForm *mbf);
 
+   /// Returns a reference to: \f$ M_{ij} \f$
    virtual double &Elem(int i, int j);
 
+   /// Returns a reference to: \f$ M_{ij} \f$
    virtual const double &Elem(int i, int j) const;
 
+   /// Matrix multiplication: \f$ y = M x \f$
    virtual void Mult(const Vector & x, Vector & y) const;
+
    virtual void AddMult(const Vector & x, Vector & y,
                         const double a = 1.0) const;
 
@@ -642,6 +691,7 @@ public:
 
    virtual MatrixInverse *Inverse() const;
 
+   /// Finalizes the matrix initialization.
    virtual void Finalize(int skip_zeros = 1);
 
    /** Extract the associated matrix as SparseMatrix blocks. The number of
@@ -649,8 +699,14 @@ public:
        test and trial spaces, respectively. */
    void GetBlocks(Array2D<SparseMatrix *> &blocks) const;
 
+   /// Returns a const reference to the sparse matrix:  \f$ M \f$
    const SparseMatrix &SpMat() const { return *mat; }
+
+   /// Returns a reference to the sparse matrix:  \f$ M \f$
    SparseMatrix &SpMat() { return *mat; }
+
+   /**  @brief Nullifies the internal matrix \f$ M \f$ and returns a pointer
+        to it.  Used for transfering ownership. */
    SparseMatrix *LoseMat() { SparseMatrix *tmp = mat; mat = NULL; return tmp; }
 
    /// Adds a domain integrator. Assumes ownership of @a bfi.
@@ -697,6 +753,7 @@ public:
        corresponding pointer (to Array<int>) will be NULL. */
    Array<Array<int>*> *GetBTFBFI_Marker() { return &btfbfi_marker; }
 
+   /// Sets all sparse values of \f$ M \f$ to @a a.
    void operator=(const double a) { *mat = a; }
 
    /// Set the desired assembly level. The default is AssemblyLevel::FULL.
@@ -705,6 +762,10 @@ public:
 
    void Assemble(int skip_zeros = 1);
 
+   /** @brief Assemble the diagonal of ADA^T into diag, where A is this mixed
+       bilinear form and D is a diagonal. */
+   void AssembleDiagonal_ADAt(const Vector &D, Vector &diag) const;
+
    /// Get the input finite element space prolongation matrix
    virtual const Operator *GetProlongation() const
    { return trial_fes->GetProlongationMatrix(); }

fem/bilinearform_ext.cpp

@@ -47,7 +47,7 @@ PABilinearFormExtension::PABilinearFormExtension(BilinearForm *form)
    bdr_face_restrict_lex = NULL;
 }
 
-void PABilinearFormExtension::SetupRestrictionOperators()
+void PABilinearFormExtension::SetupRestrictionOperators(const L2FaceValues m)
 {
    ElementDofOrdering ordering = UsesTensorBasis(*a->FESpace())?
                                  ElementDofOrdering::LEXICOGRAPHIC:
@@ -65,7 +65,8 @@ void PABilinearFormExtension::SetupRestrictionOperators()
    if (int_face_restrict_lex == NULL && a->GetFBFI()->Size() > 0)
    {
       int_face_restrict_lex = trialFes->GetFaceRestriction(
-                                 ElementDofOrdering::LEXICOGRAPHIC, FaceType::Interior);
+                                 ElementDofOrdering::LEXICOGRAPHIC,
+                                 FaceType::Interior);
       faceIntX.SetSize(int_face_restrict_lex->Height(), Device::GetMemoryType());
       faceIntY.SetSize(int_face_restrict_lex->Height(), Device::GetMemoryType());
       faceIntY.UseDevice(true); // ensure 'faceIntY = 0.0' is done on device
@@ -74,7 +75,9 @@ void PABilinearFormExtension::SetupRestrictionOperators()
    if (bdr_face_restrict_lex == NULL && a->GetBFBFI()->Size() > 0)
    {
       bdr_face_restrict_lex = trialFes->GetFaceRestriction(
-                                 ElementDofOrdering::LEXICOGRAPHIC, FaceType::Boundary);
+                                 ElementDofOrdering::LEXICOGRAPHIC,
+                                 FaceType::Boundary,
+                                 m);
       faceBdrX.SetSize(bdr_face_restrict_lex->Height(), Device::GetMemoryType());
       faceBdrY.SetSize(bdr_face_restrict_lex->Height(), Device::GetMemoryType());
       faceBdrY.UseDevice(true); // ensure 'faceBoundY = 0.0' is done on device
@@ -83,7 +86,7 @@ void PABilinearFormExtension::SetupRestrictionOperators()
 
 void PABilinearFormExtension::Assemble()
 {
-   SetupRestrictionOperators();
+   SetupRestrictionOperators(L2FaceValues::DoubleValued);
 
    Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
    const int integratorCount = integrators.Size();
@@ -287,6 +290,311 @@ void PABilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
    }
 }
 
+// Data and methods for element-assembled bilinear forms
+EABilinearFormExtension::EABilinearFormExtension(BilinearForm *form)
+   : PABilinearFormExtension(form)
+{
+}
+
+void EABilinearFormExtension::Assemble()
+{
+   SetupRestrictionOperators(L2FaceValues::SingleValued);
+
+   ne = trialFes->GetMesh()->GetNE();
+   elemDofs = trialFes->GetFE(0)->GetDof();
+
+   ea_data.SetSize(ne*elemDofs*elemDofs, Device::GetMemoryType());
+   ea_data.UseDevice(true);
+   ea_data = 0.0;
+
+   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
+   const int integratorCount = integrators.Size();
+   for (int i = 0; i < integratorCount; ++i)
+   {
+      integrators[i]->AssembleEA(*a->FESpace(), ea_data);
+   }
+
+   faceDofs = trialFes ->
+              GetTraceElement(0, trialFes->GetMesh()->GetFaceBaseGeometry(0)) ->
+              GetDof();
+
+   Array<BilinearFormIntegrator*> &intFaceIntegrators = *a->GetFBFI();
+   const int intFaceIntegratorCount = intFaceIntegrators.Size();
+   if (intFaceIntegratorCount>0)
+   {
+      nf_int = trialFes->GetNFbyType(FaceType::Interior);
+      ea_data_int.SetSize(2*nf_int*faceDofs*faceDofs, Device::GetMemoryType());
+      ea_data_ext.SetSize(2*nf_int*faceDofs*faceDofs, Device::GetMemoryType());
+      ea_data_int = 0.0;
+      ea_data_ext = 0.0;
+   }
+   for (int i = 0; i < intFaceIntegratorCount; ++i)
+   {
+      intFaceIntegrators[i]->AssembleEAInteriorFaces(*a->FESpace(),
+                                                     ea_data_int,
+                                                     ea_data_ext);
+   }
+
+   Array<BilinearFormIntegrator*> &bdrFaceIntegrators = *a->GetBFBFI();
+   const int boundFaceIntegratorCount = bdrFaceIntegrators.Size();
+   if (boundFaceIntegratorCount>0)
+   {
+      nf_bdr = trialFes->GetNFbyType(FaceType::Boundary);
+      ea_data_bdr.SetSize(nf_bdr*faceDofs*faceDofs, Device::GetMemoryType());
+      ea_data_bdr = 0.0;
+   }
+   for (int i = 0; i < boundFaceIntegratorCount; ++i)
+   {
+      bdrFaceIntegrators[i]->AssembleEABoundaryFaces(*a->FESpace(),ea_data_bdr);
+   }
+}
+
+void EABilinearFormExtension::Mult(const Vector &x, Vector &y) const
+{
+   // Apply the Element Restriction
+   const bool useRestrict = !DeviceCanUseCeed() && elem_restrict;
+   if (!useRestrict)
+   {
+      y.UseDevice(true); // typically this is a large vector, so store on device
+      y = 0.0;
+   }
+   else
+   {
+      elem_restrict->Mult(x, localX);
+      localY = 0.0;
+   }
+   // Apply the Element Matrices
+   const int NDOFS = elemDofs;
+   auto X = Reshape(useRestrict?localX.Read():x.Read(), NDOFS, ne);
+   auto Y = Reshape(useRestrict?localY.ReadWrite():y.ReadWrite(), NDOFS, ne);
+   auto A = Reshape(ea_data.Read(), NDOFS, NDOFS, ne);
+   MFEM_FORALL(glob_j, ne*NDOFS,
+   {
+      const int e = glob_j/NDOFS;
+      const int j = glob_j%NDOFS;
+      double res = 0.0;
+      for (int i = 0; i < NDOFS; i++)
+      {
+         res += A(i, j, e)*X(i, e);
+      }
+      Y(j, e) += res;
+   });
+   // Apply the Element Restriction transposed
+   if (useRestrict)
+   {
+      elem_restrict->MultTranspose(localY, y);
+   }
+
+   // Treatment of interior faces
+   Array<BilinearFormIntegrator*> &intFaceIntegrators = *a->GetFBFI();
+   const int iFISz = intFaceIntegrators.Size();
+   if (int_face_restrict_lex && iFISz>0)
+   {
+      // Apply the Interior Face Restriction
+      int_face_restrict_lex->Mult(x, faceIntX);
+      if (faceIntX.Size()>0)
+      {
+         faceIntY = 0.0;
+         // Apply the interior face matrices
+         const int NDOFS = faceDofs;
+         auto X = Reshape(faceIntX.Read(), NDOFS, 2, nf_int);
+         auto Y = Reshape(faceIntY.ReadWrite(), NDOFS, 2, nf_int);
+         auto A_int = Reshape(ea_data_int.Read(), NDOFS, NDOFS, 2, nf_int);
+         MFEM_FORALL(glob_j, nf_int*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_int(i, j, 0, f)*X(i, 0, f);
+            }
+            Y(j, 0, f) += res;
+            res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_int(i, j, 1, f)*X(i, 1, f);
+            }
+            Y(j, 1, f) += res;
+         });
+         auto A_ext = Reshape(ea_data_ext.Read(), NDOFS, NDOFS, 2, nf_int);
+         MFEM_FORALL(glob_j, nf_int*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_ext(i, j, 0, f)*X(i, 0, f);
+            }
+            Y(j, 1, f) += res;
+            res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_ext(i, j, 1, f)*X(i, 1, f);
+            }
+            Y(j, 0, f) += res;
+         });
+         // Apply the Interior Face Restriction transposed
+         int_face_restrict_lex->MultTranspose(faceIntY, y);
+      }
+   }
+
+   // Treatment of boundary faces
+   Array<BilinearFormIntegrator*> &bdrFaceIntegrators = *a->GetBFBFI();
+   const int bFISz = bdrFaceIntegrators.Size();
+   if (bdr_face_restrict_lex && bFISz>0)
+   {
+      // Apply the Boundary Face Restriction
+      bdr_face_restrict_lex->Mult(x, faceBdrX);
+      if (faceBdrX.Size()>0)
+      {
+         faceBdrY = 0.0;
+         // Apply the boundary face matrices
+         const int NDOFS = faceDofs;
+         auto X = Reshape(faceBdrX.Read(), NDOFS, nf_bdr);
+         auto Y = Reshape(faceBdrY.ReadWrite(), NDOFS, nf_bdr);
+         auto A = Reshape(ea_data_bdr.Read(), NDOFS, NDOFS, nf_bdr);
+         MFEM_FORALL(glob_j, nf_bdr*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A(i, j, f)*X(i, f);
+            }
+            Y(j, f) += res;
+         });
+         // Apply the Boundary Face Restriction transposed
+         bdr_face_restrict_lex->MultTranspose(faceBdrY, y);
+      }
+   }
+}
+
+void EABilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
+{
+   // Apply the Element Restriction
+   const bool useRestrict = DeviceCanUseCeed() || !elem_restrict;
+   if (!useRestrict)
+   {
+      y.UseDevice(true); // typically this is a large vector, so store on device
+      y = 0.0;
+   }
+   else
+   {
+      elem_restrict->Mult(x, localX);
+      localY = 0.0;
+   }
+   // Apply the Element Matrices transposed
+   const int NDOFS = elemDofs;
+   auto X = Reshape(useRestrict?localX.Read():x.Read(), NDOFS, ne);
+   auto Y = Reshape(useRestrict?localY.ReadWrite():y.ReadWrite(), NDOFS, ne);
+   auto A = Reshape(ea_data.Read(), NDOFS, NDOFS, ne);
+   MFEM_FORALL(glob_j, ne*NDOFS,
+   {
+      const int e = glob_j/NDOFS;
+      const int j = glob_j%NDOFS;
+      double res = 0.0;
+      for (int i = 0; i < NDOFS; i++)
+      {
+         res += A(j, i, e)*X(i, e);
+      }
+      Y(j, e) += res;
+   });
+   // Apply the Element Restriction transposed
+   if (useRestrict)
+   {
+      elem_restrict->MultTranspose(localY, y);
+   }
+
+   // Treatment of interior faces
+   Array<BilinearFormIntegrator*> &intFaceIntegrators = *a->GetFBFI();
+   const int iFISz = intFaceIntegrators.Size();
+   if (int_face_restrict_lex && iFISz>0)
+   {
+      // Apply the Interior Face Restriction
+      int_face_restrict_lex->Mult(x, faceIntX);
+      if (faceIntX.Size()>0)
+      {
+         faceIntY = 0.0;
+         // Apply the interior face matrices transposed
+         const int NDOFS = faceDofs;
+         auto X = Reshape(faceIntX.Read(), NDOFS, 2, nf_int);
+         auto Y = Reshape(faceIntY.ReadWrite(), NDOFS, 2, nf_int);
+         auto A_int = Reshape(ea_data_int.Read(), NDOFS, NDOFS, 2, nf_int);
+         MFEM_FORALL(glob_j, nf_int*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_int(j, i, 0, f)*X(i, 0, f);
+            }
+            Y(j, 0, f) += res;
+            res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_int(j, i, 1, f)*X(i, 1, f);
+            }
+            Y(j, 1, f) += res;
+         });
+         auto A_ext = Reshape(ea_data_ext.Read(), NDOFS, NDOFS, 2, nf_int);
+         MFEM_FORALL(glob_j, nf_int*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_ext(j, i, 0, f)*X(i, 0, f);
+            }
+            Y(j, 1, f) += res;
+            res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A_ext(j, i, 1, f)*X(i, 1, f);
+            }
+            Y(j, 0, f) += res;
+         });
+         // Apply the Interior Face Restriction transposed
+         int_face_restrict_lex->MultTranspose(faceIntY, y);
+      }
+   }
+
+   // Treatment of boundary faces
+   Array<BilinearFormIntegrator*> &bdrFaceIntegrators = *a->GetBFBFI();
+   const int bFISz = bdrFaceIntegrators.Size();
+   if (bdr_face_restrict_lex && bFISz>0)
+   {
+      // Apply the Boundary Face Restriction
+      bdr_face_restrict_lex->Mult(x, faceBdrX);
+      if (faceBdrX.Size()>0)
+      {
+         faceBdrY = 0.0;
+         // Apply the boundary face matrices transposed
+         const int NDOFS = faceDofs;
+         auto X = Reshape(faceBdrX.Read(), NDOFS, nf_bdr);
+         auto Y = Reshape(faceBdrY.ReadWrite(), NDOFS, nf_bdr);
+         auto A = Reshape(ea_data_bdr.Read(), NDOFS, NDOFS, nf_bdr);
+         MFEM_FORALL(glob_j, nf_bdr*NDOFS,
+         {
+            const int f = glob_j/NDOFS;
+            const int j = glob_j%NDOFS;
+            double res = 0.0;
+            for (int i = 0; i < NDOFS; i++)
+            {
+               res += A(j, i, f)*X(i, f);
+            }
+            Y(j, f) += res;
+         });
+         // Apply the Boundary Face Restriction transposed
+         bdr_face_restrict_lex->MultTranspose(faceBdrY, y);
+      }
+   }
+}
+
 MixedBilinearFormExtension::MixedBilinearFormExtension(MixedBilinearForm *form)
    : Operator(form->Height(), form->Width()), a(form)
 {
@@ -487,4 +795,68 @@ void PAMixedBilinearFormExtension::AddMultTranspose(const Vector &x, Vector &y,
    }
 }
 
+void PAMixedBilinearFormExtension::AssembleDiagonal_ADAt(const Vector &D,
+                                                         Vector &diag) const
+{
+   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
+
+   const int iSz = integrators.Size();
+
+   if (elem_restrict_trial)
+   {
+      const ElementRestriction* H1elem_restrict_trial =
+         dynamic_cast<const ElementRestriction*>(elem_restrict_trial);
+      if (H1elem_restrict_trial)
+      {
+         H1elem_restrict_trial->MultUnsigned(D, localTrial);
+      }
+      else
+      {
+         elem_restrict_trial->Mult(D, localTrial);
+      }
+   }
+
+   if (elem_restrict_test)
+   {
+      localTest = 0.0;
+      for (int i = 0; i < iSz; ++i)
+      {
+         if (elem_restrict_trial)
+         {
+            integrators[i]->AssembleDiagonalPA_ADAt(localTrial, localTest);
+         }
+         else
+         {
+            integrators[i]->AssembleDiagonalPA_ADAt(D, localTest);
+         }
+      }
+      const ElementRestriction* H1elem_restrict_test =
+         dynamic_cast<const ElementRestriction*>(elem_restrict_test);
+      if (H1elem_restrict_test)
+      {
+         H1elem_restrict_test->MultTransposeUnsigned(localTest, diag);
+      }
+      else
+      {
+         elem_restrict_test->MultTranspose(localTest, diag);
+      }
+   }
+   else
+   {
+      diag.UseDevice(true); // typically this is a large vector, so store on device
+      diag = 0.0;
+      for (int i = 0; i < iSz; ++i)
+      {
+         if (elem_restrict_trial)
+         {
+            integrators[i]->AssembleDiagonalPA_ADAt(localTrial, diag);
+         }
+         else
+         {
+            integrators[i]->AssembleDiagonalPA_ADAt(D, diag);
+         }
+      }
+   }
+}
+
 } // namespace mfem

fem/bilinearform_ext.hpp

@@ -22,9 +22,12 @@ namespace mfem
 class BilinearForm;
 class MixedBilinearForm;
 
-
-/** @brief Class extending the BilinearForm class to support the different
-    AssemblyLevel%s. */
+/// Class extending the BilinearForm class to support different AssemblyLevels.
+/**  FA - Full Assembly
+     PA - Partial Assembly
+     EA - Element Assembly
+     MF - Matrix Free
+*/
 class BilinearFormExtension : public Operator
 {
 protected:
@@ -42,6 +45,7 @@ public:
    /// Get the finite element space restriction matrix
    virtual const Operator *GetRestriction() const;
 
+   /// Assemble at the level given for the BilinearFormExtension subclass
    virtual void Assemble() = 0;
 
    virtual void AssembleDiagonal(Vector &diag) const
@@ -58,7 +62,8 @@ public:
    virtual void Update() = 0;
 };
 
-/// Data and methods for fully-assembled bilinear forms
+/** @brief Data and methods for fully-assembled bilinear forms.
+    Not yet implemented!  Use the BilinearForm Class instead. */
 class FABilinearFormExtension : public BilinearFormExtension
 {
 public:
@@ -78,26 +83,6 @@ public:
    ~FABilinearFormExtension() {}
 };
 
-/// Data and methods for element-assembled bilinear forms
-class EABilinearFormExtension : public BilinearFormExtension
-{
-public:
-   EABilinearFormExtension(BilinearForm *form)
-      : BilinearFormExtension(form) { }
-
-   /// TODO
-   void Assemble() {}
-   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A) {}
-   void FormLinearSystem(const Array<int> &ess_tdof_list,
-                         Vector &x, Vector &b,
-                         OperatorHandle &A, Vector &X, Vector &B,
-                         int copy_interior = 0) {}
-   void Mult(const Vector &x, Vector &y) const {}
-   void MultTranspose(const Vector &x, Vector &y) const {}
-   void Update() {}
-   ~EABilinearFormExtension() {}
-};
-
 /// Data and methods for partially-assembled bilinear forms
 class PABilinearFormExtension : public BilinearFormExtension
 {
@@ -113,7 +98,6 @@ protected:
 public:
    PABilinearFormExtension(BilinearForm*);
 
-   void SetupRestrictionOperators();
    void Assemble();
    void AssembleDiagonal(Vector &diag) const;
    void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A);
@@ -121,14 +105,34 @@ public:
                          Vector &x, Vector &b,
                          OperatorHandle &A, Vector &X, Vector &B,
                          int copy_interior = 0);
-
    void Mult(const Vector &x, Vector &y) const;
    void MultTranspose(const Vector &x, Vector &y) const;
    void Update();
+
+protected:
+   void SetupRestrictionOperators(const L2FaceValues m);
 };
 
+/// Data and methods for element-assembled bilinear forms
+class EABilinearFormExtension : public PABilinearFormExtension
+{
+protected:
+   int ne;
+   int elemDofs;
+   Vector ea_data;
+   int nf_int, nf_bdr;
+   int faceDofs;
+   Vector ea_data_int, ea_data_ext, ea_data_bdr;
 
-/// Data and methods for matrix-free bilinear forms
+public:
+   EABilinearFormExtension(BilinearForm *form);
+
+   void Assemble();
+   void Mult(const Vector &x, Vector &y) const;
+   void MultTranspose(const Vector &x, Vector &y) const;
+};
+
+/// Data and methods for matrix-free bilinear forms NOT YET IMPLEMENTED.
 class MFBilinearFormExtension : public BilinearFormExtension
 {
 public:
@@ -148,8 +152,12 @@ public:
    ~MFBilinearFormExtension() {}
 };
 
-/** @brief Class extending the MixedBilinearForm class to support the different
-    AssemblyLevel%s. */
+/// Class extending the MixedBilinearForm class to support different AssemblyLevels.
+/**  FA - Full Assembly
+     PA - Partial Assembly
+     EA - Element Assembly
+     MF - Matrix Free
+*/
 class MixedBilinearFormExtension : public Operator
 {
 protected:
@@ -186,6 +194,8 @@ public:
    virtual void AddMultTranspose(const Vector &x, Vector &y,
                                  const double c=1.0) const = 0;
 
+   virtual void AssembleDiagonal_ADAt(const Vector &D, Vector &diag) const = 0;
+
    virtual void Update() = 0;
 };
 
@@ -236,6 +246,9 @@ public:
    void MultTranspose(const Vector &x, Vector &y) const;
    /// y += c*A^T*x
    void AddMultTranspose(const Vector &x, Vector &y, const double c=1.0) const;
+   /// Assemble the diagonal of ADA^T for a diagonal vector D.
+   void AssembleDiagonal_ADAt(const Vector &D, Vector &diag) const;
+
    /// Update internals for when a new MixedBilinearForm is given to this class
    void Update();
 };

fem/bilininteg.cpp

@@ -47,7 +47,37 @@ void BilinearFormIntegrator::AssemblePABoundaryFaces(const FiniteElementSpace&)
 
 void BilinearFormIntegrator::AssembleDiagonalPA(Vector &)
 {
-   MFEM_ABORT("BilinearFormIntegrator::AssembleDiagonalPA(...)\n"
+   mfem_error ("BilinearFormIntegrator::AssembleDiagonalPA(...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::AssembleEA(const FiniteElementSpace &fes,
+                                        Vector &emat)
+{
+   mfem_error ("BilinearFormIntegrator::AssembleEA(...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::AssembleEAInteriorFaces(const FiniteElementSpace
+                                                     &fes,
+                                                     Vector &ea_data_int,
+                                                     Vector &ea_data_ext)
+{
+   mfem_error ("BilinearFormIntegrator::AssembleEAInteriorFaces(...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::AssembleEABoundaryFaces(const FiniteElementSpace
+                                                     &fes,
+                                                     Vector &ea_data_bdr)
+{
+   mfem_error ("BilinearFormIntegrator::AssembleEABoundaryFaces(...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::AssembleDiagonalPA_ADAt(const Vector &, Vector &)
+{
+   MFEM_ABORT("BilinearFormIntegrator::AssembleDiagonalPA_ADAt(...)\n"
               "   is not implemented for this class.");
 }
 
@@ -889,22 +919,25 @@ void BoundaryMassIntegrator::AssembleFaceMatrix(
    {
       int order = 2 * el1.GetOrder();
 
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    elmat = 0.0;
    for (int i = 0; i < ir->GetNPoints(); i++)
    {
       const IntegrationPoint &ip = ir->IntPoint(i);
-      IntegrationPoint eip;
-      Trans.Loc1.Transform(ip, eip);
+
+      // Set the integration point in the face and the neighboring element
+      Trans.SetAllIntPoints(&ip);
+
+      // Access the neighboring element's integration point
+      const IntegrationPoint &eip = Trans.GetElement1IntPoint();
       el1.CalcShape(eip, shape);
 
-      Trans.Face->SetIntPoint(&ip);
-      w = Trans.Face->Weight() * ip.weight;
+      w = Trans.Weight() * ip.weight;
       if (Q)
       {
-         w *= Q -> Eval(*Trans.Face, ip);
+         w *= Q -> Eval(Trans, ip);
       }
 
       AddMult_a_VVt(w, shape, elmat);
@@ -1974,7 +2007,7 @@ void VectorFEMassIntegrator::AssembleElementMatrix2(
       D.SetSize(VQ ? VQ->GetVDim() : 0);
       K.SetSize(MQ ? MQ->GetVDim() : 0, MQ ? MQ->GetVDim() : 0);
 #endif
-      DenseMatrix tmp(trial_vshape.Height(), K.Width());
+      DenseMatrix tmp(test_vshape.Height(), K.Width());
 
       elmat.SetSize (test_dof, trial_dof);
 
@@ -2535,22 +2568,22 @@ void DGTraceIntegrator::AssembleFaceMatrix(const FiniteElement &el1,
       {
          order++;
       }
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    for (int p = 0; p < ir->GetNPoints(); p++)
    {
       const IntegrationPoint &ip = ir->IntPoint(p);
-      IntegrationPoint eip1, eip2;
-      Trans.Loc1.Transform(ip, eip1);
-      if (ndof2)
-      {
-         Trans.Loc2.Transform(ip, eip2);
-      }
-      el1.CalcShape(eip1, shape1);
 
-      Trans.Face->SetIntPoint(&ip);
-      Trans.Elem1->SetIntPoint(&eip1);
+      // Set the integration point in the face and the neighboring elements
+      Trans.SetAllIntPoints(&ip);
+
+      // Access the neighboring elements' integration points
+      // Note: eip2 will only contain valid data if Elem2 exists
+      const IntegrationPoint &eip1 = Trans.GetElement1IntPoint();
+      const IntegrationPoint &eip2 = Trans.GetElement2IntPoint();
+
+      el1.CalcShape(eip1, shape1);
 
       u->Eval(vu, *Trans.Elem1, eip1);
 
@@ -2560,7 +2593,7 @@ void DGTraceIntegrator::AssembleFaceMatrix(const FiniteElement &el1,
       }
       else
       {
-         CalcOrtho(Trans.Face->Jacobian(), nor);
+         CalcOrtho(Trans.Jacobian(), nor);
       }
 
       un = vu * nor;
@@ -2575,7 +2608,6 @@ void DGTraceIntegrator::AssembleFaceMatrix(const FiniteElement &el1,
          double rho_p;
          if (un >= 0.0 && ndof2)
          {
-            Trans.Elem2->SetIntPoint(&eip2);
             rho_p = rho->Eval(*Trans.Elem2, eip2);
          }
          else
@@ -2691,7 +2723,7 @@ void DGDiffusionIntegrator::AssembleFaceMatrix(
       {
          order = 2*el1.GetOrder();
       }
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    // assemble: < {(Q \nabla u).n},[v] >      --> elmat
@@ -2699,22 +2731,26 @@ void DGDiffusionIntegrator::AssembleFaceMatrix(
    for (int p = 0; p < ir->GetNPoints(); p++)
    {
       const IntegrationPoint &ip = ir->IntPoint(p);
-      IntegrationPoint eip1, eip2;
 
-      Trans.Loc1.Transform(ip, eip1);
-      Trans.Face->SetIntPoint(&ip);
+      // Set the integration point in the face and the neighboring elements
+      Trans.SetAllIntPoints(&ip);
+
+      // Access the neighboring elements' integration points
+      // Note: eip2 will only contain valid data if Elem2 exists
+      const IntegrationPoint &eip1 = Trans.GetElement1IntPoint();
+      const IntegrationPoint &eip2 = Trans.GetElement2IntPoint();
+
       if (dim == 1)
       {
          nor(0) = 2*eip1.x - 1.0;
       }
       else
       {
-         CalcOrtho(Trans.Face->Jacobian(), nor);
+         CalcOrtho(Trans.Jacobian(), nor);
       }
 
       el1.CalcShape(eip1, shape1);
       el1.CalcDShape(eip1, dshape1);
-      Trans.Elem1->SetIntPoint(&eip1);
       w = ip.weight/Trans.Elem1->Weight();
       if (ndof2)
       {
@@ -2760,10 +2796,8 @@ void DGDiffusionIntegrator::AssembleFaceMatrix(
 
       if (ndof2)
       {
-         Trans.Loc2.Transform(ip, eip2);
          el2.CalcShape(eip2, shape2);
          el2.CalcDShape(eip2, dshape2);
-         Trans.Elem2->SetIntPoint(&eip2);
          w = ip.weight/2/Trans.Elem2->Weight();
          if (!MQ)
          {
@@ -2973,16 +3007,20 @@ void DGElasticityIntegrator::AssembleFaceMatrix(
    {
       // a simple choice for the integration order; is this OK?
       const int order = 2 * max(el1.GetOrder(), ndofs2 ? el2.GetOrder() : 0);
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    for (int pind = 0; pind < ir->GetNPoints(); ++pind)
    {
       const IntegrationPoint &ip = ir->IntPoint(pind);
-      IntegrationPoint eip1, eip2; // integration point in the reference space
-      Trans.Loc1.Transform(ip, eip1);
-      Trans.Face->SetIntPoint(&ip);
-      Trans.Elem1->SetIntPoint(&eip1);
+
+      // Set the integration point in the face and the neighboring elements
+      Trans.SetAllIntPoints(&ip);
+
+      // Access the neighboring elements' integration points
+      // Note: eip2 will only contain valid data if Elem2 exists
+      const IntegrationPoint &eip1 = Trans.GetElement1IntPoint();
+      const IntegrationPoint &eip2 = Trans.GetElement2IntPoint();
 
       el1.CalcShape(eip1, shape1);
       el1.CalcDShape(eip1, dshape1);
@@ -2996,14 +3034,12 @@ void DGElasticityIntegrator::AssembleFaceMatrix(
       }
       else
       {
-         CalcOrtho(Trans.Face->Jacobian(), nor);
+         CalcOrtho(Trans.Jacobian(), nor);
       }
 
       double w, wLM;
       if (ndofs2)
       {
-         Trans.Loc2.Transform(ip, eip2);
-         Trans.Elem2->SetIntPoint(&eip2);
          el2.CalcShape(eip2, shape2);
          el2.CalcDShape(eip2, dshape2);
          CalcAdjugate(Trans.Elem2->Jacobian(), adjJ);
@@ -3133,33 +3169,36 @@ void TraceJumpIntegrator::AssembleFaceMatrix(
       order += trial_face_fe.GetOrder();
       if (trial_face_fe.GetMapType() == FiniteElement::VALUE)
       {
-         order += Trans.Face->OrderW();
+         order += Trans.OrderW();
       }
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    for (int p = 0; p < ir->GetNPoints(); p++)
    {
       const IntegrationPoint &ip = ir->IntPoint(p);
-      IntegrationPoint eip1, eip2;
+
+      // Set the integration point in the face and the neighboring elements
+      Trans.SetAllIntPoints(&ip);
+
+      // Access the neighboring elements' integration points
+      // Note: eip2 will only contain valid data if Elem2 exists
+      const IntegrationPoint &eip1 = Trans.GetElement1IntPoint();
+      const IntegrationPoint &eip2 = Trans.GetElement2IntPoint();
+
       // Trace finite element shape function
-      Trans.Face->SetIntPoint(&ip);
       trial_face_fe.CalcShape(ip, face_shape);
       // Side 1 finite element shape function
-      Trans.Loc1.Transform(ip, eip1);
       test_fe1.CalcShape(eip1, shape1);
-      Trans.Elem1->SetIntPoint(&eip1);
       if (ndof2)
       {
          // Side 2 finite element shape function
-         Trans.Loc2.Transform(ip, eip2);
          test_fe2.CalcShape(eip2, shape2);
-         Trans.Elem2->SetIntPoint(&eip2);
       }
       w = ip.weight;
       if (trial_face_fe.GetMapType() == FiniteElement::VALUE)
       {
-         w *= Trans.Face->Weight();
+         w *= Trans.Weight();
       }
       face_shape *= w;
       for (i = 0; i < ndof1; i++)
@@ -3224,7 +3263,7 @@ void NormalTraceJumpIntegrator::AssembleFaceMatrix(
          order = test_fe1.GetOrder() - 1;
       }
       order += trial_face_fe.GetOrder();
-      ir = &IntRules.Get(Trans.FaceGeom, order);
+      ir = &IntRules.Get(Trans.GetGeometryType(), order);
    }
 
    for (int p = 0; p < ir->GetNPoints(); p++)

fem/bilininteg.hpp

@@ -57,6 +57,9 @@ public:
    /// Assemble diagonal and add it to Vector @a diag.
    virtual void AssembleDiagonalPA(Vector &diag);
 
+   /// Assemble diagonal of ADA^T (A is this integrator) and add it to @a diag.
+   virtual void AssembleDiagonalPA_ADAt(const Vector &D, Vector &diag);
+
    /// Method for partially assembled action.
    /** Perform the action of integrator on the input @a x and add the result to
        the output @a y. Both @a x and @a y are E-vectors, i.e. they represent
@@ -75,6 +78,22 @@ public:
        called. */
    virtual void AddMultTransposePA(const Vector &x, Vector &y) const;
 
+   /// Method defining element assembly.
+   /** The result of the element assembly is added and stored in the @a emat
+       Vector. */
+   virtual void AssembleEA(const FiniteElementSpace &fes, Vector &emat);
+   /** Used with BilinearFormIntegrators that have different spaces. */
+   // virtual void AssembleEA(const FiniteElementSpace &trial_fes,
+   //                         const FiniteElementSpace &test_fes,
+   //                         Vector &emat);
+
+   virtual void AssembleEAInteriorFaces(const FiniteElementSpace &fes,
+                                        Vector &ea_data_int,
+                                        Vector &ea_data_ext);
+
+   virtual void AssembleEABoundaryFaces(const FiniteElementSpace &fes,
+                                        Vector &ea_data_bdr);
+
    /// Given a particular Finite Element computes the element matrix elmat.
    virtual void AssembleElementMatrix(const FiniteElement &el,
                                       ElementTransformation &Trans,
@@ -180,6 +199,8 @@ public:
    virtual ~BilinearFormIntegrator() { }
 };
 
+/** Wraps a given @a BilinearFormIntegrator and transposes the resulting element
+    matrices. See for example ex9, ex9p. */
 class TransposeIntegrator : public BilinearFormIntegrator
 {
 private:
@@ -234,6 +255,15 @@ public:
       bfi->AddMultTransposePA(x, y);
    }
 
+   virtual void AssembleEA(const FiniteElementSpace &fes, Vector &emat);
+
+   virtual void AssembleEAInteriorFaces(const FiniteElementSpace &fes,
+                                        Vector &ea_data_int,
+                                        Vector &ea_data_ext);
+
+   virtual void AssembleEABoundaryFaces(const FiniteElementSpace &fes,
+                                        Vector &ea_data_bdr);
+
    virtual ~TransposeIntegrator() { if (own_bfi) { delete bfi; } }
 };
 
@@ -1535,7 +1565,7 @@ public:
 };
 
 /** Class for integrating the bilinear form a(u,v) := (-V u, Grad v) in 2D or 3D
-    and where V is a vector coefficient, u is in H1 and v is in H1. */
+    and where V is a vector coefficient, u is in H1 or L2 and v is in H1. */
 class MixedScalarWeakDivergenceIntegrator : public MixedScalarVectorIntegrator
 {
 public:
@@ -1885,6 +1915,8 @@ public:
 
    virtual void AssemblePA(const FiniteElementSpace &fes);
 
+   virtual void AssembleEA(const FiniteElementSpace &fes, Vector &emat);
+
    virtual void AssembleDiagonalPA(Vector &diag);
 
    virtual void AddMultPA(const Vector&, Vector&) const;
@@ -1958,6 +1990,8 @@ public:
 
    virtual void AssemblePA(const FiniteElementSpace &fes);
 
+   virtual void AssembleEA(const FiniteElementSpace &fes, Vector &emat);
+
    virtual void AssembleDiagonalPA(Vector &diag);
 
    virtual void AddMultPA(const Vector&, Vector&) const;
@@ -1969,6 +2003,7 @@ public:
    void SetupPA(const FiniteElementSpace &fes, const bool force = false);
 };
 
+/** Mass integrator (u, v) restricted to the boundary of a domain */
 class BoundaryMassIntegrator : public MassIntegrator
 {
 public:
@@ -2011,6 +2046,8 @@ public:
 
    virtual void AssemblePA(const FiniteElementSpace&);
 
+   virtual void AssembleEA(const FiniteElementSpace &fes, Vector &emat);
+
    virtual void AddMultPA(const Vector&, Vector&) const;
 
    static const IntegrationRule &GetRule(const FiniteElement &el,
@@ -2110,11 +2147,25 @@ class VectorFEDivergenceIntegrator : public BilinearFormIntegrator
 protected:
    Coefficient *Q;
 
+   using BilinearFormIntegrator::AssemblePA;
+   virtual void AssemblePA(const FiniteElementSpace &trial_fes,
+                           const FiniteElementSpace &test_fes);
+
+   virtual void AddMultPA(const Vector&, Vector&) const;
+   virtual void AddMultTransposePA(const Vector&, Vector&) const;
+
 private:
 #ifndef MFEM_THREAD_SAFE
    Vector divshape, shape;
 #endif
 
+   // PA extension
+   Vector pa_data;
+   const DofToQuad *mapsO;         ///< Not owned. DOF-to-quad map, open.
+   const DofToQuad *L2mapsO;       ///< Not owned. DOF-to-quad map, open.
+   const DofToQuad *mapsC;         ///< Not owned. DOF-to-quad map, closed.
+   int dim, ne, dofs1D, L2dofs1D, quad1D;
+
 public:
    VectorFEDivergenceIntegrator() { Q = NULL; }
    VectorFEDivergenceIntegrator(Coefficient &q) { Q = &q; }
@@ -2125,6 +2176,8 @@ public:
                                        const FiniteElement &test_fe,
                                        ElementTransformation &Trans,
                                        DenseMatrix &elmat);
+
+   virtual void AssembleDiagonalPA_ADAt(const Vector &D, Vector &diag);
 };
 
 
@@ -2308,7 +2361,7 @@ protected:
    const DofToQuad *mapsO;         ///< Not owned. DOF-to-quad map, open.
    const DofToQuad *mapsC;         ///< Not owned. DOF-to-quad map, closed.
    const GeometricFactors *geom;   ///< Not owned
-   int dim, ne, nq, dofs1D, quad1D;
+   int dim, ne, nq, dofs1D, quad1D, fetype;
 
 public:
    VectorFEMassIntegrator() { Init(NULL, NULL, NULL); }
@@ -2387,11 +2440,23 @@ class DivDivIntegrator: public BilinearFormIntegrator
 protected:
    Coefficient *Q;
 
+   using BilinearFormIntegrator::AssemblePA;
+   virtual void AssemblePA(const FiniteElementSpace &fes);
+   virtual void AddMultPA(const Vector &x, Vector &y) const;
+   virtual void AssembleDiagonalPA(Vector& diag);
+
 private:
 #ifndef MFEM_THREAD_SAFE
    Vector divshape;
 #endif
 
+   // PA extension
+   Vector pa_data;
+   const DofToQuad *mapsO;         ///< Not owned. DOF-to-quad map, open.
+   const DofToQuad *mapsC;         ///< Not owned. DOF-to-quad map, closed.
+   const GeometricFactors *geom;   ///< Not owned
+   int dim, ne, dofs1D, quad1D;
+
 public:
    DivDivIntegrator() { Q = NULL; }
    DivDivIntegrator(Coefficient &q) : Q(&q) { }
@@ -2544,6 +2609,13 @@ public:
 
    virtual void AddMultPA(const Vector&, Vector&) const;
 
+   virtual void AssembleEAInteriorFaces(const FiniteElementSpace& fes,
+                                        Vector &ea_data_int,
+                                        Vector &ea_data_ext);
+
+   virtual void AssembleEABoundaryFaces(const FiniteElementSpace& fes,
+                                        Vector &ea_data_bdr);
+
    static const IntegrationRule &GetRule(Geometry::Type geom, int order,
                                          FaceElementTransformations &T);
 

fem/bilininteg_convection_ea.cpp

@@ -0,0 +1,258 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "../general/forall.hpp"
+#include "bilininteg.hpp"
+#include "gridfunc.hpp"
+
+namespace mfem
+{
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAConvectionAssemble1D(const int NE,
+                                   const Array<double> &b,
+                                   const Array<double> &g,
+                                   const Vector &padata,
+                                   Vector &eadata,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(b.Read(), Q1D, D1D);
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_Gi[MQ1];
+      double r_Bj[MQ1];
+      for (int q = 0; q < Q1D; q++)
+      {
+         r_Gi[q] = G(q,MFEM_THREAD_ID(x));
+         r_Bj[q] = B(q,MFEM_THREAD_ID(y));
+      }
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(j1,y,D1D)
+         {
+            double val = 0.0;
+            for (int k1 = 0; k1 < Q1D; ++k1)
+            {
+               val += r_Bj[k1] * D(k1, e) * r_Gi[k1];
+            }
+            A(i1, j1, e) = val;
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAConvectionAssemble2D(const int NE,
+                                   const Array<double> &b,
+                                   const Array<double> &g,
+                                   const Vector &padata,
+                                   Vector &eadata,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(b.Read(), Q1D, D1D);
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, 2, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      double r_G[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+            r_G[q][d] = G(q,d);
+         }
+      }
+      MFEM_SHARED double s_D[MQ1][MQ1][2];
+      MFEM_FOREACH_THREAD(k1,x,Q1D)
+      {
+         MFEM_FOREACH_THREAD(k2,y,Q1D)
+         {
+            s_D[k1][k2][0] = D(k1,k2,0,e);
+            s_D[k1][k2][1] = D(k1,k2,1,e);
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            for (int j1 = 0; j1 < D1D; ++j1)
+            {
+               for (int j2 = 0; j2 < D1D; ++j2)
+               {
+                  double val = 0.0;
+                  for (int k1 = 0; k1 < Q1D; ++k1)
+                  {
+                     for (int k2 = 0; k2 < Q1D; ++k2)
+                     {
+                        val += (r_G[k1][i1] * r_B[k2][i2] * s_D[k1][k2][0]
+                                + r_B[k1][i1] * r_G[k2][i2] * s_D[k1][k2][1])
+                               * r_B[k1][j1]* r_B[k2][j2];
+                     }
+                  }
+                  A(i1, i2, j1, j2, e) = val;
+               }
+            }
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAConvectionAssemble3D(const int NE,
+                                   const Array<double> &b,
+                                   const Array<double> &g,
+                                   const Vector &padata,
+                                   Vector &eadata,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(b.Read(), Q1D, D1D);
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, Q1D, 3, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, D1D,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      double r_G[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+            r_G[q][d] = G(q,d);
+         }
+      }
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            MFEM_FOREACH_THREAD(i3,z,D1D)
+            {
+               for (int j1 = 0; j1 < D1D; ++j1)
+               {
+                  for (int j2 = 0; j2 < D1D; ++j2)
+                  {
+                     for (int j3 = 0; j3 < D1D; ++j3)
+                     {
+                        double val = 0.0;
+                        for (int k1 = 0; k1 < Q1D; ++k1)
+                        {
+                           for (int k2 = 0; k2 < Q1D; ++k2)
+                           {
+                              for (int k3 = 0; k3 < Q1D; ++k3)
+                              {
+                                 double D0 = D(k1,k2,k3,0,e);
+                                 double D1 = D(k1,k2,k3,1,e);
+                                 double D2 = D(k1,k2,k3,2,e);
+                                 val += (r_G[k1][i1] * r_B[k2][i2] * r_B[k3][i3] * D0
+                                         + r_B[k1][i1] * r_G[k2][i2] * r_B[k3][i3] * D1
+                                         + r_B[k1][i1] * r_B[k2][i2] * r_G[k3][i3] * D2)
+                                        * r_B[k1][j1] * r_B[k2][j2] * r_B[k3][j3];
+                              }
+                           }
+                        }
+                        A(i1, i2, i3, j1, j2, j3, e) = val;
+                     }
+                  }
+               }
+            }
+         }
+      }
+   });
+}
+
+void ConvectionIntegrator::AssembleEA(const FiniteElementSpace &fes,
+                                      Vector &ea_data)
+{
+   AssemblePA(fes);
+   const int ne = fes.GetMesh()->GetNE();
+   const Array<double> &B = maps->B;
+   const Array<double> &G = maps->G;
+   if (dim == 1)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EAConvectionAssemble1D<2,2>(ne,B,G,pa_data,ea_data);
+         case 0x33: return EAConvectionAssemble1D<3,3>(ne,B,G,pa_data,ea_data);
+         case 0x44: return EAConvectionAssemble1D<4,4>(ne,B,G,pa_data,ea_data);
+         case 0x55: return EAConvectionAssemble1D<5,5>(ne,B,G,pa_data,ea_data);
+         case 0x66: return EAConvectionAssemble1D<6,6>(ne,B,G,pa_data,ea_data);
+         case 0x77: return EAConvectionAssemble1D<7,7>(ne,B,G,pa_data,ea_data);
+         case 0x88: return EAConvectionAssemble1D<8,8>(ne,B,G,pa_data,ea_data);
+         case 0x99: return EAConvectionAssemble1D<9,9>(ne,B,G,pa_data,ea_data);
+         default:   return EAConvectionAssemble1D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 2)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EAConvectionAssemble2D<2,2>(ne,B,G,pa_data,ea_data);
+         case 0x33: return EAConvectionAssemble2D<3,3>(ne,B,G,pa_data,ea_data);
+         case 0x44: return EAConvectionAssemble2D<4,4>(ne,B,G,pa_data,ea_data);
+         case 0x55: return EAConvectionAssemble2D<5,5>(ne,B,G,pa_data,ea_data);
+         case 0x66: return EAConvectionAssemble2D<6,6>(ne,B,G,pa_data,ea_data);
+         case 0x77: return EAConvectionAssemble2D<7,7>(ne,B,G,pa_data,ea_data);
+         case 0x88: return EAConvectionAssemble2D<8,8>(ne,B,G,pa_data,ea_data);
+         case 0x99: return EAConvectionAssemble2D<9,9>(ne,B,G,pa_data,ea_data);
+         default:   return EAConvectionAssemble2D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x23: return EAConvectionAssemble3D<2,3>(ne,B,G,pa_data,ea_data);
+         case 0x34: return EAConvectionAssemble3D<3,4>(ne,B,G,pa_data,ea_data);
+         case 0x45: return EAConvectionAssemble3D<4,5>(ne,B,G,pa_data,ea_data);
+         case 0x56: return EAConvectionAssemble3D<5,6>(ne,B,G,pa_data,ea_data);
+         case 0x67: return EAConvectionAssemble3D<6,7>(ne,B,G,pa_data,ea_data);
+         case 0x78: return EAConvectionAssemble3D<7,8>(ne,B,G,pa_data,ea_data);
+         case 0x89: return EAConvectionAssemble3D<8,9>(ne,B,G,pa_data,ea_data);
+         default:   return EAConvectionAssemble3D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
+}

fem/bilininteg_dgtrace_ea.cpp

@@ -0,0 +1,414 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "../general/forall.hpp"
+#include "bilininteg.hpp"
+#include "gridfunc.hpp"
+
+namespace mfem
+{
+
+static void EADGTraceAssemble1DInt(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_int,
+                                   Vector &eadata_ext)
+{
+   auto D = Reshape(padata.Read(), 2, 2, NF);
+   auto A_int = Reshape(eadata_int.ReadWrite(), 2, NF);
+   auto A_ext = Reshape(eadata_ext.ReadWrite(), 2, NF);
+   MFEM_FORALL(f, NF,
+   {
+      double val_int0, val_int1, val_ext01, val_ext10;
+      val_int0  = D(0, 0, f);
+      val_ext10 = D(1, 0, f);
+      val_ext01 = D(0, 1, f);
+      val_int1  = D(1, 1, f);
+      A_int(0, f) += val_int0;
+      A_int(1, f) += val_int1;
+      A_ext(0, f) += val_ext01;
+      A_ext(1, f) += val_ext10;
+   });
+}
+
+static void EADGTraceAssemble1DBdr(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_bdr)
+{
+   auto D = Reshape(padata.Read(), 2, 2, NF);
+   auto A_bdr = Reshape(eadata_bdr.ReadWrite(), NF);
+   MFEM_FORALL(f, NF,
+   {
+      A_bdr(f) += D(0, 0, f);
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADGTraceAssemble2DInt(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_int,
+                                   Vector &eadata_ext,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, 2, 2, NF);
+   auto A_int = Reshape(eadata_int.ReadWrite(), D1D, D1D, 2, NF);
+   auto A_ext = Reshape(eadata_ext.ReadWrite(), D1D, D1D, 2, NF);
+   MFEM_FORALL_3D(f, NF, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(j1,y,D1D)
+         {
+            double val_int0 = 0.0;
+            double val_int1 = 0.0;
+            double val_ext01 = 0.0;
+            double val_ext10 = 0.0;
+            for (int k1 = 0; k1 < Q1D; ++k1)
+            {
+               val_int0  += B(k1,i1) * B(k1,j1) * D(k1, 0, 0, f);
+               val_ext01 += B(k1,i1) * B(k1,j1) * D(k1, 0, 1, f);
+               val_ext10 += B(k1,i1) * B(k1,j1) * D(k1, 1, 0, f);
+               val_int1  += B(k1,i1) * B(k1,j1) * D(k1, 1, 1, f);
+            }
+            A_int(i1, j1, 0, f) += val_int0;
+            A_int(i1, j1, 1, f) += val_int1;
+            A_ext(i1, j1, 0, f) += val_ext01;
+            A_ext(i1, j1, 1, f) += val_ext10;
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADGTraceAssemble2DBdr(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_bdr,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, 2, 2, NF);
+   auto A_bdr = Reshape(eadata_bdr.ReadWrite(), D1D, D1D, NF);
+   MFEM_FORALL_3D(f, NF, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(j1,y,D1D)
+         {
+            double val_bdr = 0.0;
+            for (int k1 = 0; k1 < Q1D; ++k1)
+            {
+               val_bdr  += B(k1,i1) * B(k1,j1) * D(k1, 0, 0, f);
+            }
+            A_bdr(i1, j1, f) += val_bdr;
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADGTraceAssemble3DInt(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_int,
+                                   Vector &eadata_ext,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, 2, 2, NF);
+   auto A_int = Reshape(eadata_int.ReadWrite(), D1D, D1D, D1D, D1D, 2, NF);
+   auto A_ext = Reshape(eadata_ext.ReadWrite(), D1D, D1D, D1D, D1D, 2, NF);
+   MFEM_FORALL_3D(f, NF, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+         }
+      }
+      MFEM_SHARED double s_D[MQ1][MQ1][2][2];
+      for (int i=0; i < 2; i++)
+      {
+         for (int j=0; j < 2; j++)
+         {
+            MFEM_FOREACH_THREAD(k1,x,Q1D)
+            {
+               MFEM_FOREACH_THREAD(k2,y,Q1D)
+               {
+                  s_D[k1][k2][i][j] = D(k1,k2,i,j,f);
+               }
+            }
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            for (int j1 = 0; j1 < D1D; ++j1)
+            {
+               for (int j2 = 0; j2 < D1D; ++j2)
+               {
+                  double val_int0 = 0.0;
+                  double val_int1 = 0.0;
+                  double val_ext01 = 0.0;
+                  double val_ext10 = 0.0;
+                  for (int k1 = 0; k1 < Q1D; ++k1)
+                  {
+                     for (int k2 = 0; k2 < Q1D; ++k2)
+                     {
+                        val_int0 += r_B[k1][i1] * r_B[k1][j1]
+                                    * r_B[k2][i2] * r_B[k2][j2]
+                                    * s_D[k1][k2][0][0];
+                        val_int1 += r_B[k1][i1] * r_B[k1][j1]
+                                    * r_B[k2][i2] * r_B[k2][j2]
+                                    * s_D[k1][k2][1][1];
+                        val_ext01+= r_B[k1][i1] * r_B[k1][j1]
+                                    * r_B[k2][i2] * r_B[k2][j2]
+                                    * s_D[k1][k2][0][1];
+                        val_ext10+= r_B[k1][i1] * r_B[k1][j1]
+                                    * r_B[k2][i2] * r_B[k2][j2]
+                                    * s_D[k1][k2][1][0];
+                     }
+                  }
+                  A_int(i1, i2, j1, j2, 0, f) += val_int0;
+                  A_int(i1, i2, j1, j2, 1, f) += val_int1;
+                  A_ext(i1, i2, j1, j2, 0, f) += val_ext01;
+                  A_ext(i1, i2, j1, j2, 1, f) += val_ext10;
+               }
+            }
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADGTraceAssemble3DBdr(const int NF,
+                                   const Array<double> &basis,
+                                   const Vector &padata,
+                                   Vector &eadata_bdr,
+                                   const int d1d = 0,
+                                   const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, 2, 2, NF);
+   auto A_bdr = Reshape(eadata_bdr.ReadWrite(), D1D, D1D, D1D, D1D, NF);
+   MFEM_FORALL_3D(f, NF, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+         }
+      }
+      MFEM_SHARED double s_D[MQ1][MQ1][2][2];
+      for (int i=0; i < 2; i++)
+      {
+         for (int j=0; j < 2; j++)
+         {
+            MFEM_FOREACH_THREAD(k1,x,Q1D)
+            {
+               MFEM_FOREACH_THREAD(k2,y,Q1D)
+               {
+                  s_D[k1][k2][i][j] = D(k1,k2,i,j,f);
+               }
+            }
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            for (int j1 = 0; j1 < D1D; ++j1)
+            {
+               for (int j2 = 0; j2 < D1D; ++j2)
+               {
+                  double val_bdr = 0.0;
+                  for (int k1 = 0; k1 < Q1D; ++k1)
+                  {
+                     for (int k2 = 0; k2 < Q1D; ++k2)
+                     {
+                        val_bdr += r_B[k1][i1] * r_B[k1][j1]
+                                   * r_B[k2][i2] * r_B[k2][j2]
+                                   * s_D[k1][k2][0][0];
+                     }
+                  }
+                  A_bdr(i1, i2, j1, j2, f) += val_bdr;
+               }
+            }
+         }
+      }
+   });
+}
+
+void DGTraceIntegrator::AssembleEAInteriorFaces(const FiniteElementSpace& fes,
+                                                Vector &ea_data_int,
+                                                Vector &ea_data_ext)
+{
+   SetupPA(fes, FaceType::Interior);
+   nf = fes.GetNFbyType(FaceType::Interior);
+   if (nf==0) { return; }
+   const Array<double> &B = maps->B;
+   if (dim == 1)
+   {
+      return EADGTraceAssemble1DInt(nf,B,pa_data,ea_data_int,ea_data_ext);
+   }
+   else if (dim == 2)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22:
+            return EADGTraceAssemble2DInt<2,2>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x33:
+            return EADGTraceAssemble2DInt<3,3>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x44:
+            return EADGTraceAssemble2DInt<4,4>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x55:
+            return EADGTraceAssemble2DInt<5,5>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x66:
+            return EADGTraceAssemble2DInt<6,6>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x77:
+            return EADGTraceAssemble2DInt<7,7>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x88:
+            return EADGTraceAssemble2DInt<8,8>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x99:
+            return EADGTraceAssemble2DInt<9,9>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         default:
+            return EADGTraceAssemble2DInt(nf,B,pa_data,ea_data_int,
+                                          ea_data_ext,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x23:
+            return EADGTraceAssemble3DInt<2,3>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x34:
+            return EADGTraceAssemble3DInt<3,4>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x45:
+            return EADGTraceAssemble3DInt<4,5>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x56:
+            return EADGTraceAssemble3DInt<5,6>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x67:
+            return EADGTraceAssemble3DInt<6,7>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x78:
+            return EADGTraceAssemble3DInt<7,8>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         case 0x89:
+            return EADGTraceAssemble3DInt<8,9>(nf,B,pa_data,ea_data_int,
+                                               ea_data_ext);
+         default:
+            return EADGTraceAssemble3DInt(nf,B,pa_data,ea_data_int,
+                                          ea_data_ext,dofs1D,quad1D);
+      }
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
+void DGTraceIntegrator::AssembleEABoundaryFaces(const FiniteElementSpace& fes,
+                                                Vector &ea_data_bdr)
+{
+   SetupPA(fes, FaceType::Boundary);
+   nf = fes.GetNFbyType(FaceType::Boundary);
+   if (nf==0) { return; }
+   const Array<double> &B = maps->B;
+   if (dim == 1)
+   {
+      return EADGTraceAssemble1DBdr(nf,B,pa_data,ea_data_bdr);
+   }
+   else if (dim == 2)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EADGTraceAssemble2DBdr<2,2>(nf,B,pa_data,ea_data_bdr);
+         case 0x33: return EADGTraceAssemble2DBdr<3,3>(nf,B,pa_data,ea_data_bdr);
+         case 0x44: return EADGTraceAssemble2DBdr<4,4>(nf,B,pa_data,ea_data_bdr);
+         case 0x55: return EADGTraceAssemble2DBdr<5,5>(nf,B,pa_data,ea_data_bdr);
+         case 0x66: return EADGTraceAssemble2DBdr<6,6>(nf,B,pa_data,ea_data_bdr);
+         case 0x77: return EADGTraceAssemble2DBdr<7,7>(nf,B,pa_data,ea_data_bdr);
+         case 0x88: return EADGTraceAssemble2DBdr<8,8>(nf,B,pa_data,ea_data_bdr);
+         case 0x99: return EADGTraceAssemble2DBdr<9,9>(nf,B,pa_data,ea_data_bdr);
+         default:
+            return EADGTraceAssemble2DBdr(nf,B,pa_data,ea_data_bdr,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x23: return EADGTraceAssemble3DBdr<2,3>(nf,B,pa_data,ea_data_bdr);
+         case 0x34: return EADGTraceAssemble3DBdr<3,4>(nf,B,pa_data,ea_data_bdr);
+         case 0x45: return EADGTraceAssemble3DBdr<4,5>(nf,B,pa_data,ea_data_bdr);
+         case 0x56: return EADGTraceAssemble3DBdr<5,6>(nf,B,pa_data,ea_data_bdr);
+         case 0x67: return EADGTraceAssemble3DBdr<6,7>(nf,B,pa_data,ea_data_bdr);
+         case 0x78: return EADGTraceAssemble3DBdr<7,8>(nf,B,pa_data,ea_data_bdr);
+         case 0x89: return EADGTraceAssemble3DBdr<8,9>(nf,B,pa_data,ea_data_bdr);
+         default:
+            return EADGTraceAssemble3DBdr(nf,B,pa_data,ea_data_bdr,dofs1D,quad1D);
+      }
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
+}

fem/bilininteg_diffusion_ea.cpp

@@ -0,0 +1,275 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "../general/forall.hpp"
+#include "bilininteg.hpp"
+#include "gridfunc.hpp"
+
+namespace mfem
+{
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADiffusionAssemble1D(const int NE,
+                                  const Array<double> &b,
+                                  const Array<double> &g,
+                                  const Vector &padata,
+                                  Vector &eadata,
+                                  const int d1d = 0,
+                                  const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_Gi[MQ1];
+      double r_Gj[MQ1];
+      for (int q = 0; q < Q1D; q++)
+      {
+         r_Gi[q] = G(q,MFEM_THREAD_ID(x));
+         r_Gj[q] = G(q,MFEM_THREAD_ID(y));
+      }
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(j1,y,D1D)
+         {
+            double val = 0.0;
+            for (int k1 = 0; k1 < Q1D; ++k1)
+            {
+               val += r_Gj[k1] * D(k1, e) * r_Gi[k1];
+            }
+            A(i1, j1, e) = val;
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADiffusionAssemble2D(const int NE,
+                                  const Array<double> &b,
+                                  const Array<double> &g,
+                                  const Vector &padata,
+                                  Vector &eadata,
+                                  const int d1d = 0,
+                                  const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(b.Read(), Q1D, D1D);
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, 3, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      double r_G[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+            r_G[q][d] = G(q,d);
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            for (int j1 = 0; j1 < D1D; ++j1)
+            {
+               for (int j2 = 0; j2 < D1D; ++j2)
+               {
+                  double val = 0.0;
+                  for (int k1 = 0; k1 < Q1D; ++k1)
+                  {
+                     for (int k2 = 0; k2 < Q1D; ++k2)
+                     {
+                        double bgi = r_G[k1][i1] * r_B[k2][i2];
+                        double gbi = r_B[k1][i1] * r_G[k2][i2];
+                        double bgj = r_G[k1][j1] * r_B[k2][j2];
+                        double gbj = r_B[k1][j1] * r_G[k2][j2];
+                        double D00 = D(k1,k2,0,e);
+                        double D10 = D(k1,k2,1,e);
+                        double D01 = D10;
+                        double D11 = D(k1,k2,2,e);
+                        val += bgi * D00 * bgj
+                               + gbi * D01 * bgj
+                               + bgi * D10 * gbj
+                               + gbi * D11 * gbj;
+                     }
+                  }
+                  A(i1, i2, j1, j2, e) = val;
+               }
+            }
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EADiffusionAssemble3D(const int NE,
+                                  const Array<double> &g,
+                                  const Array<double> &b,
+                                  const Vector &padata,
+                                  Vector &eadata,
+                                  const int d1d = 0,
+                                  const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(b.Read(), Q1D, D1D);
+   auto G = Reshape(g.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, Q1D, 6, NE);
+   auto A = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, D1D,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      double r_G[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+            r_G[q][d] = G(q,d);
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            MFEM_FOREACH_THREAD(i3,z,D1D)
+            {
+               for (int j1 = 0; j1 < D1D; ++j1)
+               {
+                  for (int j2 = 0; j2 < D1D; ++j2)
+                  {
+                     for (int j3 = 0; j3 < D1D; ++j3)
+                     {
+                        double val = 0.0;
+                        for (int k1 = 0; k1 < Q1D; ++k1)
+                        {
+                           for (int k2 = 0; k2 < Q1D; ++k2)
+                           {
+                              for (int k3 = 0; k3 < Q1D; ++k3)
+                              {
+                                 double bbgi = r_G[k1][i1] * r_B[k2][i2] * r_B[k3][i3];
+                                 double bgbi = r_B[k1][i1] * r_G[k2][i2] * r_B[k3][i3];
+                                 double gbbi = r_B[k1][i1] * r_B[k2][i2] * r_G[k3][i3];
+                                 double bbgj = r_G[k1][j1] * r_B[k2][j2] * r_B[k3][j3];
+                                 double bgbj = r_B[k1][j1] * r_G[k2][j2] * r_B[k3][j3];
+                                 double gbbj = r_B[k1][j1] * r_B[k2][j2] * r_G[k3][j3];
+                                 double D00 = D(k1,k2,k3,0,e);
+                                 double D10 = D(k1,k2,k3,1,e);
+                                 double D20 = D(k1,k2,k3,2,e);
+                                 double D01 = D10;
+                                 double D11 = D(k1,k2,k3,3,e);
+                                 double D21 = D(k1,k2,k3,4,e);
+                                 double D02 = D20;
+                                 double D12 = D21;
+                                 double D22 = D(k1,k2,k3,5,e);
+                                 val += bbgi * D00 * bbgj
+                                        + bgbi * D10 * bbgj
+                                        + gbbi * D20 * bbgj
+                                        + bbgi * D01 * bgbj
+                                        + bgbi * D11 * bgbj
+                                        + gbbi * D21 * bgbj
+                                        + bbgi * D02 * gbbj
+                                        + bgbi * D12 * gbbj
+                                        + gbbi * D22 * gbbj;
+                              }
+                           }
+                        }
+                        A(i1, i2, i3, j1, j2, j3, e) = val;
+                     }
+                  }
+               }
+            }
+         }
+      }
+   });
+}
+
+void DiffusionIntegrator::AssembleEA(const FiniteElementSpace &fes,
+                                     Vector &ea_data)
+{
+   AssemblePA(fes);
+   const int ne = fes.GetMesh()->GetNE();
+   const Array<double> &B = maps->B;
+   const Array<double> &G = maps->G;
+   if (dim == 1)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EADiffusionAssemble1D<2,2>(ne,B,G,pa_data,ea_data);
+         case 0x33: return EADiffusionAssemble1D<3,3>(ne,B,G,pa_data,ea_data);
+         case 0x44: return EADiffusionAssemble1D<4,4>(ne,B,G,pa_data,ea_data);
+         case 0x55: return EADiffusionAssemble1D<5,5>(ne,B,G,pa_data,ea_data);
+         case 0x66: return EADiffusionAssemble1D<6,6>(ne,B,G,pa_data,ea_data);
+         case 0x77: return EADiffusionAssemble1D<7,7>(ne,B,G,pa_data,ea_data);
+         case 0x88: return EADiffusionAssemble1D<8,8>(ne,B,G,pa_data,ea_data);
+         case 0x99: return EADiffusionAssemble1D<9,9>(ne,B,G,pa_data,ea_data);
+         default:   return EADiffusionAssemble1D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 2)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EADiffusionAssemble2D<2,2>(ne,B,G,pa_data,ea_data);
+         case 0x33: return EADiffusionAssemble2D<3,3>(ne,B,G,pa_data,ea_data);
+         case 0x44: return EADiffusionAssemble2D<4,4>(ne,B,G,pa_data,ea_data);
+         case 0x55: return EADiffusionAssemble2D<5,5>(ne,B,G,pa_data,ea_data);
+         case 0x66: return EADiffusionAssemble2D<6,6>(ne,B,G,pa_data,ea_data);
+         case 0x77: return EADiffusionAssemble2D<7,7>(ne,B,G,pa_data,ea_data);
+         case 0x88: return EADiffusionAssemble2D<8,8>(ne,B,G,pa_data,ea_data);
+         case 0x99: return EADiffusionAssemble2D<9,9>(ne,B,G,pa_data,ea_data);
+         default:   return EADiffusionAssemble2D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x23: return EADiffusionAssemble3D<2,3>(ne,B,G,pa_data,ea_data);
+         case 0x34: return EADiffusionAssemble3D<3,4>(ne,B,G,pa_data,ea_data);
+         case 0x45: return EADiffusionAssemble3D<4,5>(ne,B,G,pa_data,ea_data);
+         case 0x56: return EADiffusionAssemble3D<5,6>(ne,B,G,pa_data,ea_data);
+         case 0x67: return EADiffusionAssemble3D<6,7>(ne,B,G,pa_data,ea_data);
+         case 0x78: return EADiffusionAssemble3D<7,8>(ne,B,G,pa_data,ea_data);
+         case 0x89: return EADiffusionAssemble3D<8,9>(ne,B,G,pa_data,ea_data);
+         default:   return EADiffusionAssemble3D(ne,B,G,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
+}

fem/bilininteg_hcurl.cpp

@@ -24,12 +24,12 @@ constexpr int HCURL_MAX_D1D = 5;
 constexpr int HCURL_MAX_Q1D = 6;
 
 // PA H(curl) Mass Assemble 2D kernel
-static void PAHcurlSetup2D(const int Q1D,
-                           const int NE,
-                           const Array<double> &w,
-                           const Vector &j,
-                           Vector &_coeff,
-                           Vector &op)
+void PAHcurlSetup2D(const int Q1D,
+                    const int NE,
+                    const Array<double> &w,
+                    const Vector &j,
+                    Vector &_coeff,
+                    Vector &op)
 {
    const int NQ = Q1D*Q1D;
    auto W = w.Read();
@@ -55,12 +55,12 @@ static void PAHcurlSetup2D(const int Q1D,
 }
 
 // PA H(curl) Mass Assemble 3D kernel
-static void PAHcurlSetup3D(const int Q1D,
-                           const int NE,
-                           const Array<double> &w,
-                           const Vector &j,
-                           Vector &_coeff,
-                           Vector &op)
+void PAHcurlSetup3D(const int Q1D,
+                    const int NE,
+                    const Array<double> &w,
+                    const Vector &j,
+                    Vector &_coeff,
+                    Vector &op)
 {
    const int NQ = Q1D*Q1D*Q1D;
    auto W = w.Read();
@@ -106,78 +106,16 @@ static void PAHcurlSetup3D(const int Q1D,
    });
 }
 
-void VectorFEMassIntegrator::AssemblePA(const FiniteElementSpace &fes)
-{
-   // Assumes tensor-product elements
-   Mesh *mesh = fes.GetMesh();
-   const FiniteElement *fel = fes.GetFE(0);
-
-   const VectorTensorFiniteElement *el =
-      dynamic_cast<const VectorTensorFiniteElement*>(fel);
-   MFEM_VERIFY(el != NULL, "Only VectorTensorFiniteElement is supported!");
-
-   const IntegrationRule *ir
-      = IntRule ? IntRule : &MassIntegrator::GetRule(*el, *el,
-                                                     *mesh->GetElementTransformation(0));
-   const int dims = el->GetDim();
-   MFEM_VERIFY(dims == 2 || dims == 3, "");
-
-   const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
-   const int nq = ir->GetNPoints();
-   dim = mesh->Dimension();
-   MFEM_VERIFY(dim == 2 || dim == 3, "");
-
-   ne = fes.GetNE();
-   geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
-   mapsC = &el->GetDofToQuad(*ir, DofToQuad::TENSOR);
-   mapsO = &el->GetDofToQuadOpen(*ir, DofToQuad::TENSOR);
-   dofs1D = mapsC->ndof;
-   quad1D = mapsC->nqpt;
-
-   MFEM_VERIFY(dofs1D == mapsO->ndof + 1 && quad1D == mapsO->nqpt, "");
-
-   pa_data.SetSize(symmDims * nq * ne, Device::GetMemoryType());
-
-   Vector coeff(ne * nq);
-   coeff = 1.0;
-   if (Q)
-   {
-      for (int e=0; e<ne; ++e)
-      {
-         ElementTransformation *tr = mesh->GetElementTransformation(e);
-         for (int p=0; p<nq; ++p)
-         {
-            coeff[p + (e * nq)] = Q->Eval(*tr, ir->IntPoint(p));
-         }
-      }
-   }
-
-   if (el->GetDerivType() == mfem::FiniteElement::CURL && dim == 3)
-   {
-      PAHcurlSetup3D(quad1D, ne, ir->GetWeights(), geom->J,
-                     coeff, pa_data);
-   }
-   else if (el->GetDerivType() == mfem::FiniteElement::CURL && dim == 2)
-   {
-      PAHcurlSetup2D(quad1D, ne, ir->GetWeights(), geom->J,
-                     coeff, pa_data);
-   }
-   else
-   {
-      MFEM_ABORT("Unknown kernel.");
-   }
-}
-
-static void PAHcurlMassApply2D(const int D1D,
-                               const int Q1D,
-                               const int NE,
-                               const Array<double> &_Bo,
-                               const Array<double> &_Bc,
-                               const Array<double> &_Bot,
-                               const Array<double> &_Bct,
-                               const Vector &_op,
-                               const Vector &_x,
-                               Vector &_y)
+void PAHcurlMassApply2D(const int D1D,
+                        const int Q1D,
+                        const int NE,
+                        const Array<double> &_Bo,
+                        const Array<double> &_Bc,
+                        const Array<double> &_Bot,
+                        const Array<double> &_Bct,
+                        const Vector &_op,
+                        const Vector &_x,
+                        Vector &_y)
 {
    constexpr static int VDIM = 2;
 
@@ -294,13 +232,13 @@ static void PAHcurlMassApply2D(const int D1D,
    }); // end of element loop
 }
 
-static void PAHcurlMassAssembleDiagonal2D(const int D1D,
-                                          const int Q1D,
-                                          const int NE,
-                                          const Array<double> &_Bo,
-                                          const Array<double> &_Bc,
-                                          const Vector &_op,
-                                          Vector &_diag)
+void PAHcurlMassAssembleDiagonal2D(const int D1D,
+                                   const int Q1D,
+                                   const int NE,
+                                   const Array<double> &_Bo,
+                                   const Array<double> &_Bc,
+                                   const Vector &_op,
+                                   Vector &_diag)
 {
    constexpr static int VDIM = 2;
 
@@ -348,15 +286,17 @@ static void PAHcurlMassAssembleDiagonal2D(const int D1D,
    }); // end of element loop
 }
 
-template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
-static void PAHcurlMassAssembleDiagonal3D(const int D1D,
-                                          const int Q1D,
-                                          const int NE,
-                                          const Array<double> &_Bo,
-                                          const Array<double> &_Bc,
-                                          const Vector &_op,
-                                          Vector &_diag)
+void PAHcurlMassAssembleDiagonal3D(const int D1D,
+                                   const int Q1D,
+                                   const int NE,
+                                   const Array<double> &_Bo,
+                                   const Array<double> &_Bc,
+                                   const Vector &_op,
+                                   Vector &_diag)
 {
+   constexpr static int MAX_D1D = HCURL_MAX_D1D;
+   constexpr static int MAX_Q1D = HCURL_MAX_Q1D;
+
    MFEM_VERIFY(D1D <= MAX_D1D, "Error: D1D > MAX_D1D");
    MFEM_VERIFY(Q1D <= MAX_Q1D, "Error: Q1D > MAX_Q1D");
    constexpr static int VDIM = 3;
@@ -416,28 +356,20 @@ static void PAHcurlMassAssembleDiagonal3D(const int D1D,
    }); // end of element loop
 }
 
-void VectorFEMassIntegrator::AssembleDiagonalPA(Vector& diag)
+void PAHcurlMassApply3D(const int D1D,
+                        const int Q1D,
+                        const int NE,
+                        const Array<double> &_Bo,
+                        const Array<double> &_Bc,
+                        const Array<double> &_Bot,
+                        const Array<double> &_Bct,
+                        const Vector &_op,
+                        const Vector &_x,
+                        Vector &_y)
 {
-   if (dim == 3)
-      PAHcurlMassAssembleDiagonal3D(dofs1D, quad1D, ne,
-                                    mapsO->B, mapsC->B, pa_data, diag);
-   else
-      PAHcurlMassAssembleDiagonal2D(dofs1D, quad1D, ne,
-                                    mapsO->B, mapsC->B, pa_data, diag);
-}
+   constexpr static int MAX_D1D = HCURL_MAX_D1D;
+   constexpr static int MAX_Q1D = HCURL_MAX_Q1D;
 
-template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
-static void PAHcurlMassApply3D(const int D1D,
-                               const int Q1D,
-                               const int NE,
-                               const Array<double> &_Bo,
-                               const Array<double> &_Bc,
-                               const Array<double> &_Bot,
-                               const Array<double> &_Bct,
-                               const Vector &_op,
-                               const Vector &_x,
-                               Vector &_y)
-{
    MFEM_VERIFY(D1D <= MAX_D1D, "Error: D1D > MAX_D1D");
    MFEM_VERIFY(Q1D <= MAX_Q1D, "Error: Q1D > MAX_Q1D");
    constexpr static int VDIM = 3;
@@ -615,20 +547,6 @@ static void PAHcurlMassApply3D(const int D1D,
    }); // end of element loop
 }
 
-void VectorFEMassIntegrator::AddMultPA(const Vector &x, Vector &y) const
-{
-   if (dim == 3)
-   {
-      PAHcurlMassApply3D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
-                         mapsC->Bt, pa_data, x, y);
-   }
-   else
-   {
-      PAHcurlMassApply2D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
-                         mapsC->Bt, pa_data, x, y);
-   }
-}
-
 // PA H(curl) curl-curl assemble 2D kernel
 static void PACurlCurlSetup2D(const int Q1D,
                               const int NE,
@@ -1678,92 +1596,25 @@ void CurlCurlIntegrator::AssembleDiagonalPA(Vector& diag)
    }
 }
 
-void MixedVectorGradientIntegrator::AssemblePA(const FiniteElementSpace
-                                               &trial_fes,
-                                               const FiniteElementSpace &test_fes)
-{
-   // Assumes tensor-product elements, with a vector test space and H^1 trial space.
-   Mesh *mesh = trial_fes.GetMesh();
-   const FiniteElement *trial_fel = trial_fes.GetFE(0);
-   const FiniteElement *test_fel = test_fes.GetFE(0);
-
-   const NodalTensorFiniteElement *trial_el =
-      dynamic_cast<const NodalTensorFiniteElement*>(trial_fel);
-   MFEM_VERIFY(trial_el != NULL, "Only NodalTensorFiniteElement is supported!");
-
-   const VectorTensorFiniteElement *test_el =
-      dynamic_cast<const VectorTensorFiniteElement*>(test_fel);
-   MFEM_VERIFY(test_el != NULL, "Only VectorTensorFiniteElement is supported!");
-
-   const IntegrationRule *ir
-      = IntRule ? IntRule : &MassIntegrator::GetRule(*trial_el, *trial_el,
-                                                     *mesh->GetElementTransformation(0));
-   const int dims = trial_el->GetDim();
-   MFEM_VERIFY(dims == 2 || dims == 3, "");
-
-   const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
-   const int nq = ir->GetNPoints();
-   dim = mesh->Dimension();
-   MFEM_VERIFY(dim == 2 || dim == 3, "");
-
-   MFEM_VERIFY(trial_el->GetOrder() == test_el->GetOrder(), "");
-
-   ne = trial_fes.GetNE();
-   geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
-   mapsC = &test_el->GetDofToQuad(*ir, DofToQuad::TENSOR);
-   mapsO = &test_el->GetDofToQuadOpen(*ir, DofToQuad::TENSOR);
-   dofs1D = mapsC->ndof;
-   quad1D = mapsC->nqpt;
-
-   MFEM_VERIFY(dofs1D == mapsO->ndof + 1 && quad1D == mapsO->nqpt, "");
-
-   pa_data.SetSize(symmDims * nq * ne, Device::GetMemoryType());
-
-   Vector coeff(ne * nq);
-   coeff = 1.0;
-   if (Q)
-   {
-      for (int e=0; e<ne; ++e)
-      {
-         ElementTransformation *tr = mesh->GetElementTransformation(e);
-         for (int p=0; p<nq; ++p)
-         {
-            coeff[p + (e * nq)] = Q->Eval(*tr, ir->IntPoint(p));
-         }
-      }
-   }
-
-   // Use the same setup functions as VectorFEMassIntegrator.
-   if (test_el->GetDerivType() == mfem::FiniteElement::CURL && dim == 3)
-   {
-      PAHcurlSetup3D(quad1D, ne, ir->GetWeights(), geom->J,
-                     coeff, pa_data);
-   }
-   else if (test_el->GetDerivType() == mfem::FiniteElement::CURL && dim == 2)
-   {
-      PAHcurlSetup2D(quad1D, ne, ir->GetWeights(), geom->J,
-                     coeff, pa_data);
-   }
-   else
-   {
-      MFEM_ABORT("Unknown kernel.");
-   }
-}
-
 // Apply to x corresponding to DOF's in H^1 (trial), whose gradients are integrated
 // against H(curl) test functions corresponding to y.
-template<int MAX_D1D = HCURL_MAX_D1D, int MAX_Q1D = HCURL_MAX_Q1D>
-static void PAHcurlH1Apply3D(const int D1D,
-                             const int Q1D,
-                             const int NE,
-                             const Array<double> &_Bc,
-                             const Array<double> &_Gc,
-                             const Array<double> &_Bot,
-                             const Array<double> &_Bct,
-                             const Vector &_op,
-                             const Vector &_x,
-                             Vector &_y)
+void PAHcurlH1Apply3D(const int D1D,
+                      const int Q1D,
+                      const int NE,
+                      const Array<double> &_Bc,
+                      const Array<double> &_Gc,
+                      const Array<double> &_Bot,
+                      const Array<double> &_Bct,
+                      const Vector &_op,
+                      const Vector &_x,
+                      Vector &_y)
 {
+   constexpr static int MAX_D1D = HCURL_MAX_D1D;
+   constexpr static int MAX_Q1D = HCURL_MAX_Q1D;
+
+   MFEM_VERIFY(D1D <= MAX_D1D, "Error: D1D > MAX_D1D");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "Error: Q1D > MAX_Q1D");
+
    constexpr static int VDIM = 3;
 
    auto Bc = Reshape(_Bc.Read(), Q1D, D1D);
@@ -1937,16 +1788,16 @@ static void PAHcurlH1Apply3D(const int D1D,
 
 // Apply to x corresponding to DOF's in H^1 (trial), whose gradients are integrated
 // against H(curl) test functions corresponding to y.
-static void PAHcurlH1Apply2D(const int D1D,
-                             const int Q1D,
-                             const int NE,
-                             const Array<double> &_Bc,
-                             const Array<double> &_Gc,
-                             const Array<double> &_Bot,
-                             const Array<double> &_Bct,
-                             const Vector &_op,
-                             const Vector &_x,
-                             Vector &_y)
+void PAHcurlH1Apply2D(const int D1D,
+                      const int Q1D,
+                      const int NE,
+                      const Array<double> &_Bc,
+                      const Array<double> &_Gc,
+                      const Array<double> &_Bot,
+                      const Array<double> &_Bct,
+                      const Vector &_op,
+                      const Vector &_x,
+                      Vector &_y)
 {
    constexpr static int VDIM = 2;
 
@@ -2057,18 +1908,4 @@ static void PAHcurlH1Apply2D(const int D1D,
    }); // end of element loop
 }
 
-void MixedVectorGradientIntegrator::AddMultPA(const Vector &x, Vector &y) const
-{
-   if (dim == 3)
-      PAHcurlH1Apply3D(dofs1D, quad1D, ne, mapsC->B, mapsC->G,
-                       mapsO->Bt, mapsC->Bt, pa_data, x, y);
-   else if (dim == 2)
-      PAHcurlH1Apply2D(dofs1D, quad1D, ne, mapsC->B, mapsC->G,
-                       mapsO->Bt, mapsC->Bt, pa_data, x, y);
-   else
-   {
-      MFEM_ABORT("Unsupported dimension!");
-   }
-}
-
 } // namespace mfem

fem/bilininteg_mass_ea.cpp

@@ -0,0 +1,255 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "../general/forall.hpp"
+#include "bilininteg.hpp"
+#include "gridfunc.hpp"
+
+namespace mfem
+{
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAMassAssemble1D(const int NE,
+                             const Array<double> &basis,
+                             const Vector &padata,
+                             Vector &eadata,
+                             const int d1d = 0,
+                             const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, NE);
+   auto M = Reshape(eadata.Write(), D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_Bi[MQ1];
+      double r_Bj[MQ1];
+      for (int q = 0; q < Q1D; q++)
+      {
+         r_Bi[q] = B(q,MFEM_THREAD_ID(x));
+         r_Bj[q] = B(q,MFEM_THREAD_ID(y));
+      }
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(j1,y,D1D)
+         {
+            double val = 0.0;
+            for (int k1 = 0; k1 < Q1D; ++k1)
+            {
+               val += r_Bi[k1] * r_Bj[k1] * D(k1, e);
+            }
+            M(i1, j1, e) = val;
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAMassAssemble2D(const int NE,
+                             const Array<double> &basis,
+                             const Vector &padata,
+                             Vector &eadata,
+                             const int d1d = 0,
+                             const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, NE);
+   auto M = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, 1,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+         }
+      }
+      MFEM_SHARED double s_D[MQ1][MQ1];
+      MFEM_FOREACH_THREAD(k1,x,Q1D)
+      {
+         MFEM_FOREACH_THREAD(k2,y,Q1D)
+         {
+            s_D[k1][k2] = D(k1,k2,e);
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            for (int j1 = 0; j1 < D1D; ++j1)
+            {
+               for (int j2 = 0; j2 < D1D; ++j2)
+               {
+                  double val = 0.0;
+                  for (int k1 = 0; k1 < Q1D; ++k1)
+                  {
+                     for (int k2 = 0; k2 < Q1D; ++k2)
+                     {
+                        val += r_B[k1][i1] * r_B[k1][j1]
+                               * r_B[k2][i2] * r_B[k2][j2]
+                               * s_D[k1][k2];
+                     }
+                  }
+                  M(i1, i2, j1, j2, e) = val;
+               }
+            }
+         }
+      }
+   });
+}
+
+template<int T_D1D = 0, int T_Q1D = 0>
+static void EAMassAssemble3D(const int NE,
+                             const Array<double> &basis,
+                             const Vector &padata,
+                             Vector &eadata,
+                             const int d1d = 0,
+                             const int q1d = 0)
+{
+   const int D1D = T_D1D ? T_D1D : d1d;
+   const int Q1D = T_Q1D ? T_Q1D : q1d;
+   MFEM_VERIFY(D1D <= MAX_D1D, "");
+   MFEM_VERIFY(Q1D <= MAX_Q1D, "");
+   auto B = Reshape(basis.Read(), Q1D, D1D);
+   auto D = Reshape(padata.Read(), Q1D, Q1D, Q1D, NE);
+   auto M = Reshape(eadata.Write(), D1D, D1D, D1D, D1D, D1D, D1D, NE);
+   MFEM_FORALL_3D(e, NE, D1D, D1D, D1D,
+   {
+      const int D1D = T_D1D ? T_D1D : d1d;
+      const int Q1D = T_Q1D ? T_Q1D : q1d;
+      constexpr int MD1 = T_D1D ? T_D1D : MAX_D1D;
+      constexpr int MQ1 = T_Q1D ? T_Q1D : MAX_Q1D;
+      double r_B[MQ1][MD1];
+      for (int d = 0; d < D1D; d++)
+      {
+         for (int q = 0; q < Q1D; q++)
+         {
+            r_B[q][d] = B(q,d);
+         }
+      }
+      MFEM_SHARED double s_D[MQ1][MQ1][MQ1];
+      MFEM_FOREACH_THREAD(k1,x,Q1D)
+      {
+         MFEM_FOREACH_THREAD(k2,y,Q1D)
+         {
+            MFEM_FOREACH_THREAD(k3,z,Q1D)
+            {
+               s_D[k1][k2][k3] = D(k1,k2,k3,e);
+            }
+         }
+      }
+      MFEM_SYNC_THREAD;
+      MFEM_FOREACH_THREAD(i1,x,D1D)
+      {
+         MFEM_FOREACH_THREAD(i2,y,D1D)
+         {
+            MFEM_FOREACH_THREAD(i3,z,D1D)
+            {
+               for (int j1 = 0; j1 < D1D; ++j1)
+               {
+                  for (int j2 = 0; j2 < D1D; ++j2)
+                  {
+                     for (int j3 = 0; j3 < D1D; ++j3)
+                     {
+                        double val = 0.0;
+                        for (int k1 = 0; k1 < Q1D; ++k1)
+                        {
+                           for (int k2 = 0; k2 < Q1D; ++k2)
+                           {
+                              for (int k3 = 0; k3 < Q1D; ++k3)
+                              {
+                                 val += r_B[k1][i1] * r_B[k1][j1]
+                                        * r_B[k2][i2] * r_B[k2][j2]
+                                        * r_B[k3][i3] * r_B[k3][j3]
+                                        * s_D[k1][k2][k3];
+                              }
+                           }
+                        }
+                        M(i1, i2, i3, j1, j2, j3, e) = val;
+                     }
+                  }
+               }
+            }
+         }
+      }
+   });
+}
+
+void MassIntegrator::AssembleEA(const FiniteElementSpace &fes,
+                                Vector &ea_data)
+{
+   AssemblePA(fes);
+   const int ne = fes.GetMesh()->GetNE();
+   const Array<double> &B = maps->B;
+   if (dim == 1)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EAMassAssemble1D<2,2>(ne,B,pa_data,ea_data);
+         case 0x33: return EAMassAssemble1D<3,3>(ne,B,pa_data,ea_data);
+         case 0x44: return EAMassAssemble1D<4,4>(ne,B,pa_data,ea_data);
+         case 0x55: return EAMassAssemble1D<5,5>(ne,B,pa_data,ea_data);
+         case 0x66: return EAMassAssemble1D<6,6>(ne,B,pa_data,ea_data);
+         case 0x77: return EAMassAssemble1D<7,7>(ne,B,pa_data,ea_data);
+         case 0x88: return EAMassAssemble1D<8,8>(ne,B,pa_data,ea_data);
+         case 0x99: return EAMassAssemble1D<9,9>(ne,B,pa_data,ea_data);
+         default:   return EAMassAssemble1D(ne,B,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 2)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x22: return EAMassAssemble2D<2,2>(ne,B,pa_data,ea_data);
+         case 0x33: return EAMassAssemble2D<3,3>(ne,B,pa_data,ea_data);
+         case 0x44: return EAMassAssemble2D<4,4>(ne,B,pa_data,ea_data);
+         case 0x55: return EAMassAssemble2D<5,5>(ne,B,pa_data,ea_data);
+         case 0x66: return EAMassAssemble2D<6,6>(ne,B,pa_data,ea_data);
+         case 0x77: return EAMassAssemble2D<7,7>(ne,B,pa_data,ea_data);
+         case 0x88: return EAMassAssemble2D<8,8>(ne,B,pa_data,ea_data);
+         case 0x99: return EAMassAssemble2D<9,9>(ne,B,pa_data,ea_data);
+         default:   return EAMassAssemble2D(ne,B,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   else if (dim == 3)
+   {
+      switch ((dofs1D << 4 ) | quad1D)
+      {
+         case 0x23: return EAMassAssemble3D<2,3>(ne,B,pa_data,ea_data);
+         case 0x34: return EAMassAssemble3D<3,4>(ne,B,pa_data,ea_data);
+         case 0x45: return EAMassAssemble3D<4,5>(ne,B,pa_data,ea_data);
+         case 0x56: return EAMassAssemble3D<5,6>(ne,B,pa_data,ea_data);
+         case 0x67: return EAMassAssemble3D<6,7>(ne,B,pa_data,ea_data);
+         case 0x78: return EAMassAssemble3D<7,8>(ne,B,pa_data,ea_data);
+         case 0x89: return EAMassAssemble3D<8,9>(ne,B,pa_data,ea_data);
+         default:   return EAMassAssemble3D(ne,B,pa_data,ea_data,dofs1D,quad1D);
+      }
+   }
+   MFEM_ABORT("Unknown kernel.");
+}
+
+}

fem/bilininteg_transpose_ea.cpp

@@ -0,0 +1,103 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "../general/forall.hpp"
+#include "bilininteg.hpp"
+
+namespace mfem
+{
+
+void TransposeIntegrator::AssembleEA(const FiniteElementSpace &fes,
+                                     Vector &ea_data)
+{
+   Vector ea_data_tmp(ea_data.Size());
+   ea_data_tmp = 0.0;
+   bfi->AssembleEA(fes, ea_data_tmp);
+   const int ne = fes.GetNE();
+   if (ne == 0) { return; }
+   const int dofs = fes.GetFE(0)->GetDof();
+   auto A = Reshape(ea_data_tmp.Write(), dofs, dofs, ne);
+   auto AT = Reshape(ea_data.Write(), dofs, dofs, ne);
+   MFEM_FORALL(e, ne,
+   {
+      for (int i = 0; i < dofs; i++)
+      {
+         for (int j = 0; j < dofs; j++)
+         {
+            const double a = A(i, j, e);
+            AT(j, i, e) += a;
+         }
+      }
+   });
+}
+
+void TransposeIntegrator::AssembleEAInteriorFaces(const FiniteElementSpace& fes,
+                                                  Vector &ea_data_int,
+                                                  Vector &ea_data_ext)
+{
+   const int nf = fes.GetNFbyType(FaceType::Interior);
+   if (nf == 0) { return; }
+   Vector ea_data_int_tmp(ea_data_int.Size());
+   Vector ea_data_ext_tmp(ea_data_ext.Size());
+   ea_data_int_tmp = 0.0;
+   ea_data_ext_tmp = 0.0;
+   bfi->AssembleEAInteriorFaces(fes, ea_data_int_tmp, ea_data_ext_tmp);
+   const int faceDofs = fes.GetTraceElement(0,
+                                            fes.GetMesh()->GetFaceBaseGeometry(0))->GetDof();
+   auto A_int = Reshape(ea_data_int_tmp.Read(), faceDofs, faceDofs, 2, nf);
+   auto A_ext = Reshape(ea_data_ext_tmp.Read(), faceDofs, faceDofs, 2, nf);
+   auto AT_int = Reshape(ea_data_int.ReadWrite(), faceDofs, faceDofs, 2, nf);
+   auto AT_ext = Reshape(ea_data_ext.ReadWrite(), faceDofs, faceDofs, 2, nf);
+   MFEM_FORALL(f, nf,
+   {
+      for (int i = 0; i < faceDofs; i++)
+      {
+         for (int j = 0; j < faceDofs; j++)
+         {
+            const double a_int0 = A_int(i, j, 0, f);
+            const double a_int1 = A_int(i, j, 1, f);
+            const double a_ext0 = A_ext(i, j, 0, f);
+            const double a_ext1 = A_ext(i, j, 1, f);
+            AT_int(j, i, 0, f) += a_int0;
+            AT_int(j, i, 1, f) += a_int1;
+            AT_ext(j, i, 0, f) += a_ext1;
+            AT_ext(j, i, 1, f) += a_ext0;
+         }
+      }
+   });
+}
+
+void TransposeIntegrator::AssembleEABoundaryFaces(const FiniteElementSpace& fes,
+                                                  Vector &ea_data_bdr)
+{
+   const int nf = fes.GetNFbyType(FaceType::Boundary);
+   if (nf == 0) { return; }
+   Vector ea_data_bdr_tmp(ea_data_bdr.Size());
+   ea_data_bdr_tmp = 0.0;
+   bfi->AssembleEABoundaryFaces(fes, ea_data_bdr_tmp);
+   const int faceDofs = fes.GetTraceElement(0,
+                                            fes.GetMesh()->GetFaceBaseGeometry(0))->GetDof();
+   auto A_bdr = Reshape(ea_data_bdr_tmp.Read(), faceDofs, faceDofs, nf);
+   auto AT_bdr = Reshape(ea_data_bdr.ReadWrite(), faceDofs, faceDofs, nf);
+   MFEM_FORALL(f, nf,
+   {
+      for (int i = 0; i < faceDofs; i++)
+      {
+         for (int j = 0; j < faceDofs; j++)
+         {
+            const double a_bdr = A_bdr(i, j, f);
+            AT_bdr(j, i, f) += a_bdr;
+         }
+      }
+   });
+}
+
+}

fem/bilininteg_vectorfe.cpp

@@ -0,0 +1,379 @@
+// Copyright (c) 2010-2020, Lawrence Livermore National Security, LLC. Produced
+// at the Lawrence Livermore National Laboratory. All Rights reserved. See files
+// LICENSE and NOTICE for details. LLNL-CODE-806117.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability visit https://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the BSD-3 license. We welcome feedback and contributions, see file
+// CONTRIBUTING.md for details.
+
+#include "bilininteg.hpp"
+
+namespace mfem
+{
+
+void PAHcurlSetup2D(const int Q1D,
+                    const int NE,
+                    const Array<double> &w,
+                    const Vector &j,
+                    Vector &_coeff,
+                    Vector &op);
+
+void PAHcurlSetup3D(const int Q1D,
+                    const int NE,
+                    const Array<double> &w,
+                    const Vector &j,
+                    Vector &_coeff,
+                    Vector &op);
+
+void PAHcurlMassAssembleDiagonal2D(const int D1D,
+                                   const int Q1D,
+                                   const int NE,
+                                   const Array<double> &_Bo,
+                                   const Array<double> &_Bc,
+                                   const Vector &_op,
+                                   Vector &_diag);
+
+void PAHcurlMassAssembleDiagonal3D(const int D1D,
+                                   const int Q1D,
+                                   const int NE,
+                                   const Array<double> &_Bo,
+                                   const Array<double> &_Bc,
+                                   const Vector &_op,
+                                   Vector &_diag);
+
+void PAHcurlMassApply2D(const int D1D,
+                        const int Q1D,
+                        const int NE,
+                        const Array<double> &_Bo,
+                        const Array<double> &_Bc,
+                        const Array<double> &_Bot,
+                        const Array<double> &_Bct,
+                        const Vector &_op,
+                        const Vector &_x,
+                        Vector &_y);
+
+void PAHcurlMassApply3D(const int D1D,
+                        const int Q1D,
+                        const int NE,
+                        const Array<double> &_Bo,
+                        const Array<double> &_Bc,
+                        const Array<double> &_Bot,
+                        const Array<double> &_Bct,
+                        const Vector &_op,
+                        const Vector &_x,
+                        Vector &_y);
+
+void PAHdivSetup2D(const int Q1D,
+                   const int NE,
+                   const Array<double> &w,
+                   const Vector &j,
+                   Vector &_coeff,
+                   Vector &op);
+
+void PAHdivSetup3D(const int Q1D,
+                   const int NE,
+                   const Array<double> &w,
+                   const Vector &j,
+                   Vector &_coeff,
+                   Vector &op);
+
+void PAHcurlH1Apply2D(const int D1D,
+                      const int Q1D,
+                      const int NE,
+                      const Array<double> &_Bc,
+                      const Array<double> &_Gc,
+                      const Array<double> &_Bot,
+                      const Array<double> &_Bct,
+                      const Vector &_op,
+                      const Vector &_x,
+                      Vector &_y);
+
+void PAHcurlH1Apply3D(const int D1D,
+                      const int Q1D,
+                      const int NE,
+                      const Array<double> &_Bc,
+                      const Array<double> &_Gc,
+                      const Array<double> &_Bot,
+                      const Array<double> &_Bct,
+                      const Vector &_op,
+                      const Vector &_x,
+                      Vector &_y);
+
+void PAHdivMassAssembleDiagonal2D(const int D1D,
+                                  const int Q1D,
+                                  const int NE,
+                                  const Array<double> &_Bo,
+                                  const Array<double> &_Bc,
+                                  const Vector &_op,
+                                  Vector &_diag);
+
+void PAHdivMassAssembleDiagonal3D(const int D1D,
+                                  const int Q1D,
+                                  const int NE,
+                                  const Array<double> &_Bo,
+                                  const Array<double> &_Bc,
+                                  const Vector &_op,
+                                  Vector &_diag);
+
+void PAHdivMassApply2D(const int D1D,
+                       const int Q1D,
+                       const int NE,
+                       const Array<double> &_Bo,
+                       const Array<double> &_Bc,
+                       const Array<double> &_Bot,
+                       const Array<double> &_Bct,
+                       const Vector &_op,
+                       const Vector &_x,
+                       Vector &_y);
+
+void PAHdivMassApply3D(const int D1D,
+                       const int Q1D,
+                       const int NE,
+                       const Array<double> &_Bo,
+                       const Array<double> &_Bc,
+                       const Array<double> &_Bot,
+                       const Array<double> &_Bct,
+                       const Vector &_op,
+                       const Vector &_x,
+                       Vector &_y);
+
+void VectorFEMassIntegrator::AssemblePA(const FiniteElementSpace &fes)
+{
+   // Assumes tensor-product elements
+   Mesh *mesh = fes.GetMesh();
+   const FiniteElement *fel = fes.GetFE(0);
+
+   const VectorTensorFiniteElement *el =
+      dynamic_cast<const VectorTensorFiniteElement*>(fel);
+   MFEM_VERIFY(el != NULL, "Only VectorTensorFiniteElement is supported!");
+
+   const IntegrationRule *ir
+      = IntRule ? IntRule : &MassIntegrator::GetRule(*el, *el,
+                                                     *mesh->GetElementTransformation(0));
+   const int dims = el->GetDim();
+   MFEM_VERIFY(dims == 2 || dims == 3, "");
+
+   const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
+   const int nq = ir->GetNPoints();
+   dim = mesh->Dimension();
+   MFEM_VERIFY(dim == 2 || dim == 3, "");
+
+   ne = fes.GetNE();
+   geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
+   mapsC = &el->GetDofToQuad(*ir, DofToQuad::TENSOR);
+   mapsO = &el->GetDofToQuadOpen(*ir, DofToQuad::TENSOR);
+   dofs1D = mapsC->ndof;
+   quad1D = mapsC->nqpt;
+
+   MFEM_VERIFY(dofs1D == mapsO->ndof + 1 && quad1D == mapsO->nqpt, "");
+
+   pa_data.SetSize(symmDims * nq * ne, Device::GetMemoryType());
+
+   Vector coeff(ne * nq);
+   coeff = 1.0;
+   if (Q)
+   {
+      for (int e=0; e<ne; ++e)
+      {
+         ElementTransformation *tr = mesh->GetElementTransformation(e);
+         for (int p=0; p<nq; ++p)
+         {
+            coeff[p + (e * nq)] = Q->Eval(*tr, ir->IntPoint(p));
+         }
+      }
+   }
+
+   fetype = el->GetDerivType();
+
+   if (el->GetDerivType() == mfem::FiniteElement::CURL && dim == 3)
+   {
+      PAHcurlSetup3D(quad1D, ne, ir->GetWeights(), geom->J,
+                     coeff, pa_data);
+   }
+   else if (el->GetDerivType() == mfem::FiniteElement::CURL && dim == 2)
+   {
+      PAHcurlSetup2D(quad1D, ne, ir->GetWeights(), geom->J,
+                     coeff, pa_data);
+   }
+   else if (el->GetDerivType() == mfem::FiniteElement::DIV && dim == 3)
+   {
+      PAHdivSetup3D(quad1D, ne, ir->GetWeights(), geom->J,
+                    coeff, pa_data);
+   }
+   else if (el->GetDerivType() == mfem::FiniteElement::DIV && dim == 2)
+   {
+      PAHdivSetup2D(quad1D, ne, ir->GetWeights(), geom->J,
+                    coeff, pa_data);
+   }
+   else
+   {
+      MFEM_ABORT("Unknown kernel.");
+   }
+}
+
+void VectorFEMassIntegrator::AssembleDiagonalPA(Vector& diag)
+{
+   if (dim == 3)
+   {
+      if (fetype == mfem::FiniteElement::CURL)
+      {
+         PAHcurlMassAssembleDiagonal3D(dofs1D, quad1D, ne,
+                                       mapsO->B, mapsC->B, pa_data, diag);
+      }
+      else if (fetype == mfem::FiniteElement::DIV)
+      {
+         PAHdivMassAssembleDiagonal3D(dofs1D, quad1D, ne,
+                                      mapsO->B, mapsC->B, pa_data, diag);
+      }
+      else
+      {
+         MFEM_ABORT("Unknown kernel.");
+      }
+   }
+   else
+   {
+      if (fetype == mfem::FiniteElement::CURL)
+      {
+         PAHcurlMassAssembleDiagonal2D(dofs1D, quad1D, ne,
+                                       mapsO->B, mapsC->B, pa_data, diag);
+      }
+      else if (fetype == mfem::FiniteElement::DIV)
+      {
+         PAHdivMassAssembleDiagonal2D(dofs1D, quad1D, ne,
+                                      mapsO->B, mapsC->B, pa_data, diag);
+      }
+      else
+      {
+         MFEM_ABORT("Unknown kernel.");
+      }
+   }
+}
+
+void VectorFEMassIntegrator::AddMultPA(const Vector &x, Vector &y) const
+{
+   if (dim == 3)
+   {
+      if (fetype == mfem::FiniteElement::CURL)
+      {
+         PAHcurlMassApply3D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
+                            mapsC->Bt, pa_data, x, y);
+      }
+      else if (fetype == mfem::FiniteElement::DIV)
+      {
+         PAHdivMassApply3D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
+                           mapsC->Bt, pa_data, x, y);
+      }
+      else
+      {
+         MFEM_ABORT("Unknown kernel.");
+      }
+   }
+   else
+   {
+      if (fetype == mfem::FiniteElement::CURL)
+      {
+         PAHcurlMassApply2D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
+                            mapsC->Bt, pa_data, x, y);
+      }
+      else if (fetype == mfem::FiniteElement::DIV)
+      {
+         PAHdivMassApply2D(dofs1D, quad1D, ne, mapsO->B, mapsC->B, mapsO->Bt,
+                           mapsC->Bt, pa_data, x, y);
+      }
+      else
+      {
+         MFEM_ABORT("Unknown kernel.");
+      }
+   }
+}
+
+void MixedVectorGradientIntegrator::AssemblePA(const FiniteElementSpace
+                                               &trial_fes,
+                                               const FiniteElementSpace &test_fes)
+{
+   // Assumes tensor-product elements, with a vector test space and H^1 trial space.
+   Mesh *mesh = trial_fes.GetMesh();
+   const FiniteElement *trial_fel = trial_fes.GetFE(0);
+   const FiniteElement *test_fel = test_fes.GetFE(0);
+
+   const NodalTensorFiniteElement *trial_el =
+      dynamic_cast<const NodalTensorFiniteElement*>(trial_fel);
+   MFEM_VERIFY(trial_el != NULL, "Only NodalTensorFiniteElement is supported!");
+
+   const VectorTensorFiniteElement *test_el =
+      dynamic_cast<const VectorTensorFiniteElement*>(test_fel);
+   MFEM_VERIFY(test_el != NULL, "Only VectorTensorFiniteElement is supported!");
+
+   const IntegrationRule *ir
+      = IntRule ? IntRule : &MassIntegrator::GetRule(*trial_el, *trial_el,
+                                                     *mesh->GetElementTransformation(0));
+   const int dims = trial_el->GetDim();
+   MFEM_VERIFY(dims == 2 || dims == 3, "");
+
+   const int symmDims = (dims * (dims + 1)) / 2; // 1x1: 1, 2x2: 3, 3x3: 6
+   const int nq = ir->GetNPoints();
+   dim = mesh->Dimension();
+   MFEM_VERIFY(dim == 2 || dim == 3, "");
+
+   MFEM_VERIFY(trial_el->GetOrder() == test_el->GetOrder(), "");
+
+   ne = trial_fes.GetNE();
+   geom = mesh->GetGeometricFactors(*ir, GeometricFactors::JACOBIANS);
+   mapsC = &test_el->GetDofToQuad(*ir, DofToQuad::TENSOR);
+   mapsO = &test_el->GetDofToQuadOpen(*ir, DofToQuad::TENSOR);
+   dofs1D = mapsC->ndof;
+   quad1D = mapsC->nqpt;
+
+   MFEM_VERIFY(dofs1D == mapsO->ndof + 1 && quad1D == mapsO->nqpt, "");
+
+   pa_data.SetSize(symmDims * nq * ne, Device::GetMemoryType());
+
+   Vector coeff(ne * nq);
+   coeff = 1.0;
+   if (Q)
+   {
+      for (int e=0; e<ne; ++e)
+      {
+         ElementTransformation *tr = mesh->GetElementTransformation(e);
+         for (int p=0; p<nq; ++p)
+         {
+            coeff[p + (e * nq)] = Q->Eval(*tr, ir->IntPoint(p));
+         }
+      }
+   }
+
+   // Use the same setup functions as VectorFEMassIntegrator.
+   if (test_el->GetDerivType() == mfem::FiniteElement::CURL && dim == 3)
+   {
+      PAHcurlSetup3D(quad1D, ne, ir->GetWeights(), geom->J,
+                     coeff, pa_data);
+   }
+   else if (test_el->GetDerivType() == mfem::FiniteElement::CURL && dim == 2)
+   {
+      PAHcurlSetup2D(quad1D, ne, ir->GetWeights(), geom->J,
+                     coeff, pa_data);
+   }
+   else
+   {
+      MFEM_ABORT("Unknown kernel.");
+   }
+}
+
+void MixedVectorGradientIntegrator::AddMultPA(const Vector &x, Vector &y) const
+{
+   if (dim == 3)
+      PAHcurlH1Apply3D(dofs1D, quad1D, ne, mapsC->B, mapsC->G,
+                       mapsO->Bt, mapsC->Bt, pa_data, x, y);
+   else if (dim == 2)
+      PAHcurlH1Apply2D(dofs1D, quad1D, ne, mapsC->B, mapsC->G,
+                       mapsO->Bt, mapsC->Bt, pa_data, x, y);
+   else
+   {
+      MFEM_ABORT("Unsupported dimension!");
+   }
+}
+
+} // namespace mfem

fem/coefficient.cpp

@@ -49,7 +49,7 @@ double FunctionCoefficient::Eval(ElementTransformation & T,
 double GridFunctionCoefficient::Eval (ElementTransformation &T,
                                       const IntegrationPoint &ip)
 {
-   return GridF -> GetValue (T.ElementNo, ip, Component);
+   return GridF -> GetValue (T, ip, Component);
 }
 
 double TransformedCoefficient::Eval(ElementTransformation &T,
@@ -160,13 +160,13 @@ void VectorArrayCoefficient::Eval(Vector &V, ElementTransformation &T,
 }
 
 VectorGridFunctionCoefficient::VectorGridFunctionCoefficient (
-   GridFunction *gf)
+   const GridFunction *gf)
    : VectorCoefficient ((gf) ? gf -> VectorDim() : 0)
 {
    GridFunc = gf;
 }
 
-void VectorGridFunctionCoefficient::SetGridFunction(GridFunction *gf)
+void VectorGridFunctionCoefficient::SetGridFunction(const GridFunction *gf)
 {
    GridFunc = gf; vdim = (gf) ? gf -> VectorDim() : 0;
 }
@@ -174,24 +174,7 @@ void VectorGridFunctionCoefficient::SetGridFunction(GridFunction *gf)
 void VectorGridFunctionCoefficient::Eval(Vector &V, ElementTransformation &T,
                                          const IntegrationPoint &ip)
 {
-   Mesh *mesh = GridFunc->FESpace()->GetMesh();
-   if (mesh->Dimension() == T.GetDimension())
-   {
-      GridFunc->GetVectorValue(T.ElementNo, ip, V);
-   }
-   else // Assuming T is a boundary element transformation
-   {
-      int el_id, el_info;
-      mesh->GetBdrElementAdjacentElement(T.ElementNo, el_id, el_info);
-      IntegrationPointTransformation loc_T;
-      mesh->GetLocalFaceTransformation(mesh->GetBdrElementType(T.ElementNo),
-                                       mesh->GetElementType(el_id),
-                                       loc_T.Transf,
-                                       el_info);
-      IntegrationPoint eip;
-      loc_T.Transform(ip, eip);
-      GridFunc->GetVectorValue(el_id, eip, V);
-   }
+   GridFunc->GetVectorValue(T, ip, V);
 }
 
 void VectorGridFunctionCoefficient::Eval(
@@ -201,14 +184,14 @@ void VectorGridFunctionCoefficient::Eval(
 }
 
 GradientGridFunctionCoefficient::GradientGridFunctionCoefficient (
-   GridFunction *gf)
+   const GridFunction *gf)
    : VectorCoefficient((gf) ?
                        gf -> FESpace() -> GetMesh() -> SpaceDimension() : 0)
 {
    GridFunc = gf;
 }
 
-void GradientGridFunctionCoefficient::SetGridFunction(GridFunction *gf)
+void GradientGridFunctionCoefficient::SetGridFunction(const GridFunction *gf)
 {
    GridFunc = gf; vdim = (gf) ?
                          gf -> FESpace() -> GetMesh() -> SpaceDimension() : 0;
@@ -226,18 +209,24 @@ void GradientGridFunctionCoefficient::Eval(
    GridFunc->GetGradients(T, ir, M);
 }
 
-CurlGridFunctionCoefficient::CurlGridFunctionCoefficient (
-   GridFunction *gf)
-   : VectorCoefficient ((gf) ?
-                        gf -> FESpace() -> GetMesh() -> SpaceDimension() : 0)
+CurlGridFunctionCoefficient::CurlGridFunctionCoefficient(
+   const GridFunction *gf)
+   : VectorCoefficient(0)
 {
-   GridFunc = gf;
+   SetGridFunction(gf);
 }
 
-void CurlGridFunctionCoefficient::SetGridFunction(GridFunction *gf)
+void CurlGridFunctionCoefficient::SetGridFunction(const GridFunction *gf)
 {
-   GridFunc = gf; vdim = (gf) ?
-                         gf -> FESpace() -> GetMesh() -> SpaceDimension() : 0;
+   if (gf)
+   {
+      int sdim = gf -> FESpace() -> GetMesh() -> SpaceDimension();
+      MFEM_VERIFY(sdim == 2 || sdim == 3,
+                  "CurlGridFunctionCoefficient "
+                  "only defind for spaces of dimension 2 or 3.");
+   }
+   GridFunc = gf;
+   vdim = (gf) ? (2 * gf -> FESpace() -> GetMesh() -> SpaceDimension() - 3) : 0;
 }
 
 void CurlGridFunctionCoefficient::Eval(Vector &V, ElementTransformation &T,
@@ -247,7 +236,7 @@ void CurlGridFunctionCoefficient::Eval(Vector &V, ElementTransformation &T,
 }
 
 DivergenceGridFunctionCoefficient::DivergenceGridFunctionCoefficient (
-   GridFunction *gf) : Coefficient()
+   const GridFunction *gf) : Coefficient()
 {
    GridFunc = gf;
 }
@@ -433,13 +422,43 @@ double DeterminantCoefficient::Eval(ElementTransformation &T,
    return ma.Det();
 }
 
-VectorSumCoefficient::VectorSumCoefficient(VectorCoefficient &A,
-                                           VectorCoefficient &B,
-                                           double _alpha, double _beta)
-   : VectorCoefficient(A.GetVDim()), a(&A), b(&B), alpha(_alpha), beta(_beta),
-     va(A.GetVDim())
+VectorSumCoefficient::VectorSumCoefficient(int dim)
+   : VectorCoefficient(dim),
+     ACoef(NULL), BCoef(NULL),
+     A(dim), B(dim),
+     alphaCoef(NULL), betaCoef(NULL),
+     alpha(1.0), beta(1.0)
 {
-   MFEM_ASSERT(A.GetVDim() == B.GetVDim(),
+   A = 0.0; B = 0.0;
+}
+
+VectorSumCoefficient::VectorSumCoefficient(VectorCoefficient &_A,
+                                           VectorCoefficient &_B,
+                                           double _alpha, double _beta)
+   : VectorCoefficient(_A.GetVDim()),
+     ACoef(&_A), BCoef(&_B),
+     A(_A.GetVDim()), B(_A.GetVDim()),
+     alphaCoef(NULL), betaCoef(NULL),
+     alpha(_alpha), beta(_beta)
+{
+   MFEM_ASSERT(_A.GetVDim() == _B.GetVDim(),
+               "VectorSumCoefficient:  "
+               "Arguments must have the same dimension.");
+}
+
+VectorSumCoefficient::VectorSumCoefficient(VectorCoefficient &_A,
+                                           VectorCoefficient &_B,
+                                           Coefficient &_alpha,
+                                           Coefficient &_beta)
+   : VectorCoefficient(_A.GetVDim()),
+     ACoef(&_A), BCoef(&_B),
+     A(_A.GetVDim()),
+     B(_A.GetVDim()),
+     alphaCoef(&_alpha),
+     betaCoef(&_beta),
+     alpha(0.0), beta(0.0)
+{
+   MFEM_ASSERT(_A.GetVDim() == _B.GetVDim(),
                "VectorSumCoefficient:  "
                "Arguments must have the same dimension.");
 }
@@ -447,26 +466,47 @@ VectorSumCoefficient::VectorSumCoefficient(VectorCoefficient &A,
 void VectorSumCoefficient::Eval(Vector &V, ElementTransformation &T,
                                 const IntegrationPoint &ip)
 {
-   b->Eval(V, T, ip);
-   if ( beta != 1.0 ) { V *= beta; }
-   a->Eval(va, T, ip);
-   V.Add(alpha, va);
+   V.SetSize(A.Size());
+   if (    ACoef) { ACoef->Eval(A, T, ip); }
+   if (    BCoef) { BCoef->Eval(B, T, ip); }
+   if (alphaCoef) { alpha = alphaCoef->Eval(T, ip); }
+   if ( betaCoef) { beta  = betaCoef->Eval(T, ip); }
+   add(alpha, A, beta, B, V);
 }
 
+ScalarVectorProductCoefficient::ScalarVectorProductCoefficient(
+   double A,
+   VectorCoefficient &B)
+   : VectorCoefficient(B.GetVDim()), aConst(A), a(NULL), b(&B)
+{}
+
 ScalarVectorProductCoefficient::ScalarVectorProductCoefficient(
    Coefficient &A,
    VectorCoefficient &B)
-   : VectorCoefficient(B.GetVDim()), a(&A), b(&B)
+   : VectorCoefficient(B.GetVDim()), aConst(0.0), a(&A), b(&B)
 {}
 
 void ScalarVectorProductCoefficient::Eval(Vector &V, ElementTransformation &T,
                                           const IntegrationPoint &ip)
 {
-   double sa = a->Eval(T, ip);
+   double sa = (a == NULL) ? aConst : a->Eval(T, ip);
    b->Eval(V, T, ip);
    V *= sa;
 }
 
+NormalizedVectorCoefficient::NormalizedVectorCoefficient(VectorCoefficient &A,
+                                                         double _tol)
+   : VectorCoefficient(A.GetVDim()), a(&A), tol(_tol)
+{}
+
+void NormalizedVectorCoefficient::Eval(Vector &V, ElementTransformation &T,
+                                       const IntegrationPoint &ip)
+{
+   a->Eval(V, T, ip);
+   double nv = V.Norml2();
+   V *= (nv > tol) ? (1.0/nv) : 0.0;
+}
+
 VectorCrossProductCoefficient::VectorCrossProductCoefficient(
    VectorCoefficient &A,
    VectorCoefficient &B)
@@ -488,17 +528,18 @@ void VectorCrossProductCoefficient::Eval(Vector &V, ElementTransformation &T,
    V[2] = va[0] * vb[1] - va[1] * vb[0];
 }
 
-MatVecCoefficient::MatVecCoefficient(MatrixCoefficient &A,
-                                     VectorCoefficient &B)
+MatrixVectorProductCoefficient::MatrixVectorProductCoefficient(
+   MatrixCoefficient &A, VectorCoefficient &B)
    : VectorCoefficient(A.GetHeight()), a(&A), b(&B),
      ma(A.GetHeight(), A.GetWidth()), vb(B.GetVDim())
 {
    MFEM_ASSERT(A.GetWidth() == B.GetVDim(),
-               "MatVecCoefficient:  Arguments have incompatible dimensions.");
+               "MatrixVectorProductCoefficient:  "
+               "Arguments have incompatible dimensions.");
 }
 
-void MatVecCoefficient::Eval(Vector &V, ElementTransformation &T,
-                             const IntegrationPoint &ip)
+void MatrixVectorProductCoefficient::Eval(Vector &V, ElementTransformation &T,
+                                          const IntegrationPoint &ip)
 {
    a->Eval(ma, T, ip);
    b->Eval(vb, T, ip);
@@ -534,17 +575,23 @@ void MatrixSumCoefficient::Eval(DenseMatrix &M, ElementTransformation &T,
    M.Add(alpha, ma);
 }
 
+ScalarMatrixProductCoefficient::ScalarMatrixProductCoefficient(
+   double A,
+   MatrixCoefficient &B)
+   : MatrixCoefficient(B.GetHeight(), B.GetWidth()), aConst(A), a(NULL), b(&B)
+{}
+
 ScalarMatrixProductCoefficient::ScalarMatrixProductCoefficient(
    Coefficient &A,
    MatrixCoefficient &B)
-   : MatrixCoefficient(B.GetHeight(), B.GetWidth()), a(&A), b(&B)
+   : MatrixCoefficient(B.GetHeight(), B.GetWidth()), aConst(0.0), a(&A), b(&B)
 {}
 
 void ScalarMatrixProductCoefficient::Eval(DenseMatrix &M,
                                           ElementTransformation &T,
                                           const IntegrationPoint &ip)
 {
-   double sa = a->Eval(T, ip);
+   double sa = (a == NULL) ? aConst : a->Eval(T, ip);
    b->Eval(M, T, ip);
    M *= sa;
 }
@@ -598,6 +645,30 @@ void OuterProductCoefficient::Eval(DenseMatrix &M, ElementTransformation &T,
    }
 }
 
+CrossCrossCoefficient::CrossCrossCoefficient(Coefficient &A,
+                                             VectorCoefficient &K)
+   : MatrixCoefficient(K.GetVDim(), K.GetVDim()), aConst(0.0), a(&A), k(&K),
+     vk(K.GetVDim())
+{}
+
+void CrossCrossCoefficient::Eval(DenseMatrix &M, ElementTransformation &T,
+                                 const IntegrationPoint &ip)
+{
+   k->Eval(vk, T, ip);
+   M.SetSize(vk.Size(), vk.Size());
+   M = 0.0;
+   double k2 = vk*vk;
+   for (int i=0; i<vk.Size(); i++)
+   {
+      M(i, i) = k2;
+      for (int j=0; j<vk.Size(); j++)
+      {
+         M(i, j) -= vk[i] * vk[j];
+      }
+   }
+   M *= ((a == NULL ) ? aConst : a->Eval(T, ip) );
+}
+
 double LpNormLoop(double p, Coefficient &coeff, Mesh &mesh,
                   const IntegrationRule *irs[])
 {
@@ -775,4 +846,61 @@ double ComputeGlobalLpNorm(double p, VectorCoefficient &coeff, ParMesh &pmesh,
 }
 #endif
 
+VectorQuadratureFunctionCoefficient::VectorQuadratureFunctionCoefficient(
+   QuadratureFunction &qf)
+   : VectorCoefficient(qf.GetVDim()), QuadF(qf), index(0) { }
+
+void VectorQuadratureFunctionCoefficient::SetComponent(int _index, int _length)
+{
+   MFEM_VERIFY(_index >= 0, "Index must be >= 0");
+   MFEM_VERIFY(_index < QuadF.GetVDim(),
+               "Index must be < QuadratureFunction length");
+   index = _index;
+
+   MFEM_VERIFY(_length > 0, "Length must be > 0");
+   MFEM_VERIFY(_length <= QuadF.GetVDim() - index,
+               "Length must be <= (QuadratureFunction length - index)");
+
+   vdim = _length;
+}
+
+void VectorQuadratureFunctionCoefficient::Eval(Vector &V,
+                                               ElementTransformation &T,
+                                               const IntegrationPoint &ip)
+{
+   QuadF.HostRead();
+
+   if (index == 0 && vdim == QuadF.GetVDim())
+   {
+      QuadF.GetElementValues(T.ElementNo, ip.index, V);
+   }
+   else
+   {
+      Vector temp;
+      QuadF.GetElementValues(T.ElementNo, ip.index, temp);
+      V.SetSize(vdim);
+      for (int i = 0; i < vdim; i++)
+      {
+         V(i) = temp(index + i);
+      }
+   }
+
+   return;
+}
+
+QuadratureFunctionCoefficient::QuadratureFunctionCoefficient(
+   QuadratureFunction &qf) : QuadF(qf)
+{
+   MFEM_VERIFY(qf.GetVDim() == 1, "QuadratureFunction's vdim must be 1");
+}
+
+double QuadratureFunctionCoefficient::Eval(ElementTransformation &T,
+                                           const IntegrationPoint &ip)
+{
+   QuadF.HostRead();
+   Vector temp(1);
+   QuadF.GetElementValues(T.ElementNo, ip.index, temp);
+   return temp[0];
+}
+
 }

fem/datacollection.cpp

@@ -739,12 +739,6 @@ ParaViewDataCollection::ParaViewDataCollection(const std::string&
 #endif
 }
 
-void ParaViewDataCollection::RegisterField(const std::string& field_name,
-                                           mfem::GridFunction *gf)
-{
-   DataCollection::RegisterField(field_name,gf);
-}
-
 void ParaViewDataCollection::SetLevelsOfDetail(int levels_of_detail_)
 {
    levels_of_detail = levels_of_detail_;
@@ -815,7 +809,7 @@ void ParaViewDataCollection::Save()
    // the directory is created
 
    // create pvd file if needed
-   if (!pvd_stream.is_open())
+   if (myid == 0 && !pvd_stream.is_open())
    {
       std::string dpath=GenerateCollectionPath();
       std::string pvdname=dpath+"/"+GeneratePVDFileName();

fem/datacollection.hpp

@@ -501,10 +501,6 @@ public:
    ParaViewDataCollection(const std::string& collection_name,
                           mfem::Mesh *mesh_ = NULL);
 
-   /// Add a grid function to the collection
-   virtual void RegisterField(const std::string& field_name,
-                              mfem::GridFunction *gf) override;
-
    /// Set refinement levels - every element is uniformly split based on
    /// levels_of_detail_
    void SetLevelsOfDetail(int levels_of_detail_);

fem/eltrans.cpp

@@ -19,6 +19,7 @@ namespace mfem
 ElementTransformation::ElementTransformation()
    : IntPoint(static_cast<IntegrationPoint *>(NULL)),
      EvalState(0),
+     geom(Geometry::INVALID),
      Attribute(-1),
      ElementNo(-1)
 { }
@@ -551,4 +552,155 @@ void IntegrationPointTransformation::Transform (const IntegrationRule &ir1,
    }
 }
 
+void FaceElementTransformations::SetIntPoint(const IntegrationPoint *face_ip)
+{
+   IsoparametricTransformation::SetIntPoint(face_ip);
+
+   if (mask & 4)
+   {
+      Loc1.Transform(*face_ip, eip1);
+      if (Elem1)
+      {
+         Elem1->SetIntPoint(&eip1);
+      }
+   }
+   if (mask & 8)
+   {
+      Loc2.Transform(*face_ip, eip2);
+      if (Elem2)
+      {
+         Elem2->SetIntPoint(&eip2);
+      }
+   }
+}
+
+ElementTransformation &
+FaceElementTransformations::GetElement1Transformation()
+{
+   MFEM_VERIFY(mask & HAVE_ELEM1 && Elem1 != NULL, "The ElementTransformation "
+               "for the element has not been configured for side 1.");
+   return *Elem1;
+}
+
+ElementTransformation &
+FaceElementTransformations::GetElement2Transformation()
+{
+   MFEM_VERIFY(mask & HAVE_ELEM2 && Elem2 != NULL, "The ElementTransformation "
+               "for the element has not been configured for side 2.");
+   return *Elem2;
+}
+
+IntegrationPointTransformation &
+FaceElementTransformations::GetIntPoint1Transformation()
+{
+   MFEM_VERIFY(mask & HAVE_LOC1, "The IntegrationPointTransformation "
+               "for the element has not been configured for side 1.");
+   return Loc1;
+}
+
+IntegrationPointTransformation &
+FaceElementTransformations::GetIntPoint2Transformation()
+{
+   MFEM_VERIFY(mask & HAVE_LOC2, "The IntegrationPointTransformation "
+               "for the element has not been configured for side 2.");
+   return Loc2;
+}
+
+void FaceElementTransformations::Transform(const IntegrationPoint &ip,
+                                           Vector &trans)
+{
+   MFEM_VERIFY(mask & HAVE_FACE, "The ElementTransformation "
+               "for the face has not been configured.");
+   IsoparametricTransformation::Transform(ip, trans);
+}
+
+void FaceElementTransformations::Transform(const IntegrationRule &ir,
+                                           DenseMatrix &tr)
+{
+   MFEM_VERIFY(mask & HAVE_FACE, "The ElementTransformation "
+               "for the face has not been configured.");
+   IsoparametricTransformation::Transform(ir, tr);
+}
+
+void FaceElementTransformations::Transform(const DenseMatrix &matrix,
+                                           DenseMatrix &result)
+{
+   MFEM_VERIFY(mask & HAVE_FACE, "The ElementTransformation "
+               "for the face has not been configured.");
+   IsoparametricTransformation::Transform(matrix, result);
+}
+
+double FaceElementTransformations::CheckConsistency(int print_level,
+                                                    std::ostream &out)
+{
+   // Check that the face vertices are mapped to the same physical location
+   // when using the following three transformations:
+   // - the face transformation, *this
+   // - Loc1 + Elem1
+   // - Loc2 + Elem2, if present.
+
+   const bool have_face = (mask & 16);
+   const bool have_el1 = (mask & 1) && (mask & 4);
+   const bool have_el2 = (mask & 2) && (mask & 8) && (Elem2No >= 0);
+   if (int(have_face) + int(have_el1) + int(have_el2) < 2)
+   {
+      // need at least two different transformations to perform a check
+      return 0.0;
+   }
+
+   const IntegrationRule &v_ir = *Geometries.GetVertices(GetGeometryType());
+
+   double max_dist = 0.0;
+   Vector dist(v_ir.GetNPoints());
+   DenseMatrix coords_base, coords_el;
+   IntegrationRule v_eir(v_ir.GetNPoints());
+   if (have_face)
+   {
+      Transform(v_ir, coords_base);
+      if (print_level > 0)
+      {
+         out << "\nface vertex coordinates (from face transform):\n"
+             << "----------------------------------------------\n";
+         coords_base.PrintT(out, coords_base.Height());
+      }
+   }
+   if (have_el1)
+   {
+      Loc1.Transform(v_ir, v_eir);
+      Elem1->Transform(v_eir, coords_el);
+      if (print_level > 0)
+      {
+         out << "\nface vertex coordinates (from element 1 transform):\n"
+             << "---------------------------------------------------\n";
+         coords_el.PrintT(out, coords_el.Height());
+      }
+      if (have_face)
+      {
+         coords_el -= coords_base;
+         coords_el.Norm2(dist);
+         max_dist = std::max(max_dist, dist.Normlinf());
+      }
+      else
+      {
+         coords_base = coords_el;
+      }
+   }
+   if (have_el2)
+   {
+      Loc2.Transform(v_ir, v_eir);
+      Elem2->Transform(v_eir, coords_el);
+      if (print_level > 0)
+      {
+         out << "\nface vertex coordinates (from element 2 transform):\n"
+             << "---------------------------------------------------\n";
+         coords_el.PrintT(out, coords_el.Height());
+      }
+      coords_el -= coords_base;
+      coords_el.Norm2(dist);
+      max_dist = std::max(max_dist, dist.Normlinf());
+   }
+
+   return max_dist;
+}
+
 }

fem/eltrans.hpp

@@ -38,9 +38,12 @@ protected:
    };
    Geometry::Type geom;
 
-   // Evaluate the Jacobian of the transformation at the IntPoint and store it
-   // in dFdx.
+   /** @brief Evaluate the Jacobian of the transformation at the IntPoint and
+       store it in dFdx. */
    virtual const DenseMatrix &EvalJacobian() = 0;
+
+   /** @brief Evaluate the Hessian of the transformation at the IntPoint and
+       store it in d2Fdx2. */
    virtual const DenseMatrix &EvalHessian() = 0;
 
    double EvalWeight();
@@ -48,18 +51,53 @@ protected:
    const DenseMatrix &EvalInverseJ();
 
 public:
-   int Attribute, ElementNo;
+
+   /** This enumeration declares the values stored in
+       ElementTransformation::ElementType and indicates which group of objects
+       the index stored in ElementTransformation::ElementNo refers:
+
+       | ElementType | Range of ElementNo
+       +-------------+-------------------------
+       | ELEMENT     | [0, Mesh::GetNE()     )
+       | BDR_ELEMENT | [0, Mesh::GetNBE()    )
+       | EDGE        | [0, Mesh::GetNEdges() )
+       | FACE        | [0, Mesh::GetNFaces() )
+       | BDR_FACE    | [0, Mesh::GetNBE()    )
+   */
+   enum
+   {
+      ELEMENT     = 1,
+      BDR_ELEMENT = 2,
+      EDGE        = 3,
+      FACE        = 4,
+      BDR_FACE    = 5
+   };
+
+   int Attribute, ElementNo, ElementType;
 
    ElementTransformation();
 
+   /** @brief Set the integration point @a ip that weights and Jacobians will
+       be evaluated at. */
    void SetIntPoint(const IntegrationPoint *ip)
    { IntPoint = ip; EvalState = 0; }
+
+   /** @brief Get a const reference to the currently set integration point.  This
+       will return NULL if no integration point is set. */
    const IntegrationPoint &GetIntPoint() { return *IntPoint; }
 
+   /** @brief Transform integration point from reference coordinates to
+       physical coordinates and store them in the vector. */
    virtual void Transform(const IntegrationPoint &, Vector &) = 0;
+
+   /** @brief Transform all the integration points from the integration rule
+       from reference coordinates to physical
+       coordinates and store them as column vectors in the matrix. */
    virtual void Transform(const IntegrationRule &, DenseMatrix &) = 0;
 
-   /// Transform columns of 'matrix', store result in 'result'.
+   /** @brief Transform all the integration points from the column vectors
+       of @a matrix from reference coordinates to physical
+       coordinates and store them as column vectors in @a result. */
    virtual void Transform(const DenseMatrix &matrix, DenseMatrix &result) = 0;
 
    /** @brief Return the Jacobian matrix of the transformation at the currently
@@ -70,27 +108,44 @@ public:
    const DenseMatrix &Jacobian()
    { return (EvalState & JACOBIAN_MASK) ? dFdx : EvalJacobian(); }
 
+
+   /** @brief Return the Hessian matrix of the transformation at the currently
+       set IntegrationPoint, using the method SetIntPoint(). */
    const DenseMatrix &Hessian()
    { return (EvalState & HESSIAN_MASK) ? d2Fdx2 : EvalHessian(); }
 
+   /** @brief Return the weight of the Jacobian matrix of the transformation
+       at the currently set IntegrationPoint.
+       The Weight evaluates to \f$ \sqrt{\lvert J^T J \rvert} \f$. */
    double Weight() { return (EvalState & WEIGHT_MASK) ? Wght : EvalWeight(); }
 
+   /** @brief Return the adjugate of the Jacobian matrix of the transformation
+        at the currently set IntegrationPoint. */
    const DenseMatrix &AdjugateJacobian()
    { return (EvalState & ADJUGATE_MASK) ? adjJ : EvalAdjugateJ(); }
 
+   /** @brief Return the inverse of the Jacobian matrix of the transformation
+        at the currently set IntegrationPoint. */
    const DenseMatrix &InverseJacobian()
    { return (EvalState & INVERSE_MASK) ? invJ : EvalInverseJ(); }
 
+   /// Return the order of the current element we are using for the transformation.
    virtual int Order() const = 0;
+
+   /// Return the order of the elements of the Jacobian of the transformation.
    virtual int OrderJ() const = 0;
+
+   /** @brief Return the order of the determinant of the Jacobian (weight)
+       of the transformation. */
    virtual int OrderW() const = 0;
-   /// Order of adj(J)^t.grad(fi)
+
+   /// Return the order of \f$ adj(J)^T \nabla fi \f$
    virtual int OrderGrad(const FiniteElement *fe) const = 0;
 
    /// Return the Geometry::Type of the reference element.
    Geometry::Type GetGeometryType() const { return geom; }
 
-   /// Return the dimension of the reference element.
+   /// Return the topological dimension of the reference element.
    int GetDimension() const { return Geometry::Dimension[geom]; }
 
    /// Get the dimension of the target (physical) space.
@@ -286,7 +341,7 @@ public:
    virtual int Transform(const Vector &pt, IntegrationPoint &ip);
 };
 
-
+/// A standard isoparametric element transformation
 class IsoparametricTransformation : public ElementTransformation
 {
 private:
@@ -296,26 +351,29 @@ private:
    const FiniteElement *FElem;
    DenseMatrix PointMat; // dim x dof
 
-   // Evaluate the Jacobian of the transformation at the IntPoint and store it
-   // in dFdx.
+   /** @brief Evaluate the Jacobian of the transformation at the IntPoint and
+       store it in dFdx. */
    virtual const DenseMatrix &EvalJacobian();
    // Evaluate the Hessian of the transformation at the IntPoint and store it
    // in d2Fdx2.
    virtual const DenseMatrix &EvalHessian();
 public:
+   /// Set the element that will be used to compute the transformations
    void SetFE(const FiniteElement *FE) { FElem = FE; geom = FE->GetGeomType(); }
+
+   /// Get the current element used to compute the transformations
    const FiniteElement* GetFE() const { return FElem; }
 
    /// @brief Set the underlying point matrix describing the transformation.
    /** The dimensions of the matrix are space-dim x dof. The transformation is
        defined as
+           \f$ x = F( \hat x ) = P \phi( \hat x ) \f$
 
-           x = F(xh) = P . phi(xh),
-
-       where xh (x hat) is the reference point, x is the corresponding physical
-       point, P is the point matrix, and phi(xh) is the column-vector of all
-       basis functions evaluated at xh. The columns of P represent the control
-       points in physical space defining the transformation. */
+       where \f$ \hat x \f$  is the reference point, @a x is the corresponding
+       physical point, @a P is the point matrix, and \f$ \phi( \hat x ) \f$ is
+       the column-vector of all basis functions evaluated at \f$ \hat x \f$ .
+       The columns of @a P represent the control points in physical space
+       defining the transformation. */
    void SetPointMat(const DenseMatrix &pm) { PointMat = pm; }
 
    /// Return the stored point matrix.
@@ -324,19 +382,44 @@ public:
    /// Write access to the stored point matrix. Use with caution.
    DenseMatrix &GetPointMat() { return PointMat; }
 
+   /// Set the FiniteElement Geometry for the reference elements being used.
    void SetIdentityTransformation(Geometry::Type GeomType);
 
+   /** @brief Transform integration point from reference coordinates to
+       physical coordinates and store them in the vector. */
    virtual void Transform(const IntegrationPoint &, Vector &);
+
+   /** @brief Transform all the integration points from the integration rule
+       from reference coordinates to physical
+      coordinates and store them as column vectors in the matrix. */
    virtual void Transform(const IntegrationRule &, DenseMatrix &);
+
+   /** @brief Transform all the integration points from the column vectors
+       of @a matrix from reference coordinates to physical
+       coordinates and store them as column vectors in @a result. */
    virtual void Transform(const DenseMatrix &matrix, DenseMatrix &result);
 
+   /// Return the order of the current element we are using for the transformation.
    virtual int Order() const { return FElem->GetOrder(); }
+
+   /// Return the order of the elements of the Jacobian of the transformation.
    virtual int OrderJ() const;
+
+   /** @brief Return the order of the determinant of the Jacobian (weight)
+       of the transformation. */
    virtual int OrderW() const;
+
+   /// Return the order of \f$ adj(J)^T \nabla fi \f$
    virtual int OrderGrad(const FiniteElement *fe) const;
 
    virtual int GetSpaceDim() const { return PointMat.Height(); }
 
+   /** @brief Transform a point @a pt from physical space to a point @a ip in
+       reference space. */
+   /** Attempt to find the IntegrationPoint that is transformed into the given
+       point in physical space. If the inversion fails a non-zero value is
+       returned. This method is not 100 percent reliable for non-linear
+       transformations. */
    virtual int TransformBack(const Vector & v, IntegrationPoint & ip)
    {
       InverseElementTransformation inv_tr(this);
@@ -356,15 +439,157 @@ public:
    void Transform (const IntegrationRule  &, IntegrationRule  &);
 };
 
-class FaceElementTransformations
+/** @brief A specialized ElementTransformation class representing a face and
+    its two neighboring elements.
+
+    This class can be used as a container for the element transformation data
+    needed for integrating discontinuous fields on element interfaces in a
+    Discontinuous Galerkin (DG) context.
+
+    The secondary purpose of this class is to enable the
+    GridFunction::GetValue function, and various related functions, to properly
+    evaluate fields with limited continuity on boundary elements.
+*/
+class FaceElementTransformations : public IsoparametricTransformation
 {
+private:
+
+   // Bitwise OR of ConfigMasks
+   int mask;
+
+   IntegrationPoint eip1, eip2;
+
+protected: // interface for Mesh to be able to configure this object.
+
+   friend class Mesh;
+#ifdef MFEM_USE_MPI
+   friend class ParMesh;
+#endif
+
+   /// Set the mask indicating which portions of the object have been setup
+   /** The argument @a m is a bitmask used in
+       Mesh::GetFaceElementTransformations to indicate which portions of the
+       FaceElementTransformations object have been configured.
+
+       mask &  1: Elem1 is configured
+       mask &  2: Elem2 is configured
+       mask &  4: Loc1 is configured
+       mask &  8: Loc2 is configured
+       mask & 16: The Face transformation itself is configured
+   */
+   void SetConfigurationMask(int m) { mask = m; }
+
 public:
-   int Elem1No, Elem2No, FaceGeom;
-   ElementTransformation *Elem1, *Elem2, *Face;
+
+   enum ConfigMasks
+   {
+      HAVE_ELEM1 =  1, ///< Element on side 1 is configured
+      HAVE_ELEM2 =  2, ///< Element on side 2 is configured
+      HAVE_LOC1  =  4, ///< Point transformation for side 1 is configured
+      HAVE_LOC2  =  8, ///< Point transformation for side 2 is configured
+      HAVE_FACE  = 16  ///< Face transformation is configured
+   };
+
+   int Elem1No, Elem2No;
+   Geometry::Type &FaceGeom; ///< @deprecated Use GetGeometryType instead
+   ElementTransformation *Elem1, *Elem2;
+   ElementTransformation *Face; ///< @deprecated No longer necessary
    IntegrationPointTransformation Loc1, Loc2;
+
+   FaceElementTransformations() : FaceGeom(geom), Face(this) {}
+
+   /** @brief Method to set the geometry type of the face.
+
+       @note This method is designed to be used when
+       [Par]Mesh::GetFaceTransformation will not be called i.e. when the face
+       transformation will not be needed but the neighboring element
+       transformations will be.  Using this method to override the GeometryType
+       should only be done with great care.
+   */
+   void SetGeometryType(Geometry::Type g) { geom = g; }
+
+   /** @brief Return the mask defining the configuration state.
+
+       The mask value indicates which portions of FaceElementTransformations
+       object have been configured.
+
+       mask &  1: Elem1 is configured
+       mask &  2: Elem2 is configured
+       mask &  4: Loc1 is configured
+       mask &  8: Loc2 is configured
+       mask & 16: The Face transformation itself is configured
+   */
+   int GetConfigurationMask() const { return mask; }
+
+   /** @brief Set the integration point in the Face and the two neighboring
+       elements, if present.
+
+       The point @a face_ip must be in the reference coordinate system of the
+       face.
+   */
+   void SetIntPoint(const IntegrationPoint *face_ip);
+
+   /** @brief Set the integration point in the Face and the two neighboring
+       elements, if present.
+
+       This is a more expressive member function name than SetIntPoint, which
+       in this special case, does the same thing. This function can be used for
+       greater code clarity.
+   */
+   inline void SetAllIntPoints(const IntegrationPoint *face_ip)
+   { FaceElementTransformations::SetIntPoint(face_ip); }
+
+   /** @brief Get a const reference to the integration point in neighboring
+       element 1 corresponding to the currently set integration point on the
+       face.
+
+       This IntegrationPoint object will only contain up-to-date data if
+       SetIntPoint or SetAllIntPoints has been called with the latest
+       integration point for the face and the appropriate point transformation
+       has been configured. */
+   const IntegrationPoint &GetElement1IntPoint() { return eip1; }
+
+   /** @brief Get a const reference to the integration point in neighboring
+       element 2 corresponding to the currently set integration point on the
+       face.
+
+       This IntegrationPoint object will only contain up-to-date data if
+       SetIntPoint or SetAllIntPoints has been called with the latest
+       integration point for the face and the appropriate point transformation
+       has been configured. */
+   const IntegrationPoint &GetElement2IntPoint() { return eip2; }
+
+   virtual void Transform(const IntegrationPoint &, Vector &);
+   virtual void Transform(const IntegrationRule &, DenseMatrix &);
+   virtual void Transform(const DenseMatrix &matrix, DenseMatrix &result);
+
+   ElementTransformation & GetElement1Transformation();
+   ElementTransformation & GetElement2Transformation();
+   IntegrationPointTransformation & GetIntPoint1Transformation();
+   IntegrationPointTransformation & GetIntPoint2Transformation();
+
+   /** @brief Check for self-consistency: compares the result of mapping the
+       reference face vertices to physical coordinates using the three
+       transformations: face, element 1, and element 2.
+
+       @param[in] print_level  If set to a positive number, print the physical
+                               coordinates of the face vertices computed through
+                               all available transformations: face, element 1,
+                               and/or element 2.
+       @param[in,out] out      The output stream to use for printing.
+
+       @returns A maximal distance between physical coordinates of face vertices
+                that should coincide. A successful check should return a small
+                number relative to the mesh extents. If less than 2 of the three
+                transformations are set, returns 0.
+
+       @warning This check will generally fail on periodic boundary faces.
+   */
+   double CheckConsistency(int print_level = 0,
+                           std::ostream &out = mfem::out);
 };
 
-/*                 Elem1(Loc1(x)) = Face(x) = Elem2(Loc2(x))
+/**                Elem1(Loc1(x)) = Face(x) = Elem2(Loc2(x))
 
 
                                 Physical Space

fem/estimators.cpp

@@ -50,4 +50,21 @@ void L2ZienkiewiczZhuEstimator::ComputeEstimates()
 
 #endif // MFEM_USE_MPI
 
+void LpErrorEstimator::ComputeEstimates()
+{
+   MFEM_VERIFY(coef != NULL || vcoef != NULL,
+               "LpErrorEstimator has no coefficient!  Call SetCoef first.");
+
+   error_estimates.SetSize(sol->FESpace()->GetMesh()->GetNE());
+   if (coef)
+   {
+      sol->ComputeElementLpErrors(local_norm_p, *coef, error_estimates);
+   }
+   else
+   {
+      sol->ComputeElementLpErrors(local_norm_p, *vcoef, error_estimates);
+   }
+   current_sequence = sol->FESpace()->GetMesh()->GetSequence();
+}
+
 } // namespace mfem

fem/estimators.hpp

@@ -45,6 +45,7 @@ public:
    /// Force recomputation of the estimates on the next call to GetLocalErrors.
    virtual void Reset() = 0;
 
+   /// Destruct the error estimator
    virtual ~ErrorEstimator() { }
 };
 
@@ -66,6 +67,14 @@ public:
 /** @brief The ZienkiewiczZhuEstimator class implements the Zienkiewicz-Zhu
     error estimation procedure.
 
+    Zienkiewicz, O.C. and Zhu, J.Z., The superconvergent patch recovery
+    and a posteriori error estimates. Part 1: The recovery technique.
+    Int. J. Num. Meth. Engng. 33, 1331-1364 (1992).
+
+    Zienkiewicz, O.C. and Zhu, J.Z., The superconvergent patch recovery
+    and a posteriori error estimates. Part 2: Error estimates and adaptivity.
+    Int. J. Num. Meth. Engng. 33, 1365-1382 (1992).
+
     The required BilinearFormIntegrator must implement the methods
     ComputeElementFlux() and ComputeFluxEnergy().
  */
@@ -217,6 +226,7 @@ protected:
       class when needed.*/
    bool own_flux_fes; ///< Ownership flag for flux_space and smooth_flux_space.
 
+   /// Initialize with the integrator, solution, and flux finite element spaces.
    void Init(BilinearFormIntegrator &integ,
              ParGridFunction &sol,
              ParFiniteElementSpace *flux_fes,
@@ -304,6 +314,88 @@ public:
 
 #endif // MFEM_USE_MPI
 
+/** @brief The LpErrorEstimator class compares the solution to a known
+    coefficient.
+
+    This class can be used, for example, to adapt a mesh to a non-trivial
+    initial condition in a time-dependent simulation. It can also be used to
+    force refinement in the neighborhood of small features before switching to a
+    more traditional error estimator.
+
+    The LpErrorEstimator supports either scalar or vector coefficients and works
+    both in serial and in parallel.
+*/
+class LpErrorEstimator : public ErrorEstimator
+{
+protected:
+   long current_sequence;
+   int local_norm_p;
+   Vector error_estimates;
+
+   Coefficient * coef;
+   VectorCoefficient * vcoef;
+   GridFunction * sol;
+
+   /// Check if the mesh of the solution was modified.
+   bool MeshIsModified()
+   {
+      long mesh_sequence = sol->FESpace()->GetMesh()->GetSequence();
+      MFEM_ASSERT(mesh_sequence >= current_sequence, "");
+      return (mesh_sequence > current_sequence);
+   }
+
+   /// Compute the element error estimates.
+   void ComputeEstimates();
+
+public:
+   /** @brief Construct a new LpErrorEstimator object for a scalar field.
+       @param p    Integer which selects which Lp norm to use.
+       @param sol  The GridFunction representation of the scalar field.
+       Note: the coefficient must be set before use with the SetCoef method.
+   */
+   LpErrorEstimator(int p, GridFunction &sol)
+      : current_sequence(-1), local_norm_p(p),
+        error_estimates(0), coef(NULL), vcoef(NULL), sol(&sol) { }
+
+   /** @brief Construct a new LpErrorEstimator object for a scalar field.
+       @param p    Integer which selects which Lp norm to use.
+       @param coef The scalar Coefficient to compare to the solution.
+       @param sol  The GridFunction representation of the scalar field.
+   */
+   LpErrorEstimator(int p, Coefficient &coef, GridFunction &sol)
+      : current_sequence(-1), local_norm_p(p),
+        error_estimates(0), coef(&coef), vcoef(NULL), sol(&sol) { }
+
+   /** @brief Construct a new LpErrorEstimator object for a vector field.
+       @param p    Integer which selects which Lp norm to use.
+       @param coef The vector VectorCoefficient to compare to the solution.
+       @param sol  The GridFunction representation of the vector field.
+   */
+   LpErrorEstimator(int p, VectorCoefficient &coef, GridFunction &sol)
+      : current_sequence(-1), local_norm_p(p),
+        error_estimates(0), coef(NULL), vcoef(&coef), sol(&sol) { }
+
+   /** @brief Set the exponent, p, of the Lp norm used for computing the local
+       element errors. */
+   void SetLocalErrorNormP(int p) { local_norm_p = p; }
+
+   void SetCoef(Coefficient &A) { coef = &A; }
+   void SetCoef(VectorCoefficient &A) { vcoef = &A; }
+
+   /// Reset the error estimator.
+   virtual void Reset() { current_sequence = -1; }
+
+   /// Get a Vector with all element errors.
+   virtual const Vector &GetLocalErrors()
+   {
+      if (MeshIsModified()) { ComputeEstimates(); }
+      return error_estimates;
+   }
+
+   /// Destructor
+   virtual ~LpErrorEstimator() {}
+};
+
 } // namespace mfem
 
 #endif // MFEM_ERROR_ESTIMATORS

fem/fe_coll.cpp

@@ -311,10 +311,10 @@ GetEdge(int &nv, v_t &v, int &ne, int &e, int &eo, const int edge_info)
    eo = edge_info%64;
    MFEM_ASSERT(0 <= e && e < g_consts::NumEdges, "");
    MFEM_ASSERT(0 <= eo && eo < e_consts::NumOrient, "");
-   v[0] = g_consts::Edges[e][0];
-   v[1] = g_consts::Edges[e][1];
-   v[0] = e_consts::Orient[eo][v[0]];
-   v[1] = e_consts::Orient[eo][v[1]];
+   v[0] = e_consts::Orient[eo][0];
+   v[1] = e_consts::Orient[eo][1];
+   v[0] = g_consts::Edges[e][v[0]];
+   v[1] = g_consts::Edges[e][v[1]];
 }
 
 template <Geometry::Type geom, Geometry::Type f_geom,

fem/fe_coll.hpp

@@ -19,10 +19,10 @@
 namespace mfem
 {
 
-/** Collection of finite elements from the same family in multiple dimensions.
-    This class is used to match the degrees of freedom of a FiniteElementSpace
-    between elements, and to provide the finite element restriction from an
-    element to its boundary. */
+/** @brief Collection of finite elements from the same family in multiple
+    dimensions. This class is used to match the degrees of freedom of a
+    FiniteElementSpace between elements, and to provide the finite element
+    restriction from an element to its boundary. */
 class FiniteElementCollection
 {
 protected:
@@ -40,6 +40,14 @@ protected:
                               const int face_info);
 
 public:
+   /** @brief Enumeration for ContType: defines the continuity of the field
+       across element interfaces. */
+   enum { CONTINUOUS,   ///< Field is continuous across element interfaces
+          TANGENTIAL,   ///< Tangential components of vector field
+          NORMAL,       ///< Normal component of vector field
+          DISCONTINUOUS ///< Field is discontinuous across element interfaces
+        };
+
    virtual const FiniteElement *
    FiniteElementForGeometry(Geometry::Type GeomType) const = 0;
 
@@ -52,6 +60,8 @@ public:
 
    virtual const char * Name() const { return "Undefined"; }
 
+   virtual int GetContType() const = 0;
+
    int HasFaceDofs(Geometry::Type GeomType) const;
 
    virtual const FiniteElement *TraceFiniteElementForGeometry(
@@ -66,15 +76,81 @@ public:
 
    /** @brief Factory method: return a newly allocated FiniteElementCollection
        according to the given name. */
+   /**
+   | FEC Name | Space | Order | BasisType | FiniteElement::MapT | Notes |
+   | :------: | :---: | :---: | :-------: | :-----: | :---: |
+   | H1_[DIM]_[ORDER] | H1 | * | 1 | VALUE | H1 nodal elements |
+   | H1@[BTYPE]_[DIM]_[ORDER] | H1 | * | * | VALUE | H1 nodal elements |
+   | H1Pos_[DIM]_[ORDER] | H1 | * | 1 | VALUE | H1 nodal elements |
+   | H1Pos_Trace_[DIM]_[ORDER] | H^{1/2} | * | 2 | VALUE | H^{1/2}-conforming trace elements for H1 defined on the interface between mesh elements (faces,edges,vertices) |
+   | H1_Trace_[DIM]_[ORDER] | H^{1/2} | * | 1 | VALUE | H^{1/2}-conforming trace elements for H1 defined on the interface between mesh elements (faces,edges,vertices) |
+   | H1_Trace@[BTYPE]_[DIM]_[ORDER] | H^{1/2} | * | 1 | VALUE | H^{1/2}-conforming trace elements for H1 defined on the interface between mesh elements (faces,edges,vertices) |
+   | ND_[DIM]_[ORDER] | H(curl) | * | 1 / 0 | H_CURL | Nedelec vector elements |
+   | ND@[CBTYPE][OBTYPE]_[DIM]_[ORDER] | H(curl) | * | * / * | H_CURL | Nedelec vector elements |
+   | ND_Trace_[DIM]_[ORDER] | H^{1/2} | * | 1 / 0  | H_CURL | H^{1/2}-conforming trace elements for H(curl) defined on the interface between mesh elements (faces) |
+   | ND_Trace@[CBTYPE][OBTYPE]_[DIM]_[ORDER] | H^{1/2} | * | 1 / 0 | H_CURL | H^{1/2}-conforming trace elements for H(curl) defined on the interface between mesh elements (faces) |
+   | RT_[DIM]_[ORDER] | H(div) | * | 1 / 0 | H_DIV | Raviart-Thomas vector elements |
+   | RT@[CBTYPE][OBTYPE]_[DIM]_[ORDER] | H(div) | * | * / * | H_DIV | Raviart-Thomas vector elements |
+   | RT_Trace_[DIM]_[ORDER] | H^{1/2} | * | 1 / 0 | INTEGRAL | H^{1/2}-conforming trace elements for H(div) defined on the interface between mesh elements (faces) |
+   | RT_ValTrace_[DIM]_[ORDER] | H^{1/2} | * | 1 / 0 | VALUE | H^{1/2}-conforming trace elements for H(div) defined on the interface between mesh elements (faces) |
+   | RT_Trace@[BTYPE]_[DIM]_[ORDER] | H^{1/2} | * | 1 / 0 | INTEGRAL | H^{1/2}-conforming trace elements for H(div) defined on the interface between mesh elements (faces) |
+   | RT_ValTrace@[BTYPE]_[DIM]_[ORDER] |  H^{1/2} | * | 1 / 0 | VALUE | H^{1/2}-conforming trace elements for H(div) defined on the interface between mesh elements (faces) |
+   | L2_[DIM]_[ORDER] | L2 | * | 0 | VALUE | Discontinous L2 elements |
+   | L2_T[BTYPE]_[DIM]_[ORDER] | L2 | * | 0 | VALUE | Discontinous L2 elements |
+   | L2Int_[DIM]_[ORDER] | L2 | * | 0 | INTEGRAL | Discontinous L2 elements |
+   | L2Int_T[BTYPE]_[DIM]_[ORDER] | L2 | * | 0 | INTEGRAL | Discontinous L2 elements |
+   | DG_Iface_[DIM]_[ORDER] | - | * | 0 | VALUE | Discontinuous elements on the interface between mesh elements (faces) |
+   | DG_Iface@[BTYPE]_[DIM]_[ORDER] | - | * | 0 | VALUE | Discontinuous elements on the interface between mesh elements (faces) |
+   | DG_IntIface_[DIM]_[ORDER] | - | * | 0 | INTEGRAL | Discontinuous elements on the interface between mesh elements (faces) |
+   | DG_IntIface@[BTYPE]_[DIM]_[ORDER] | - | * | 0 | INTEGRAL | Discontinuous elements on the interface between mesh elements (faces) |
+   | NURBS[ORDER] | - | * | - | VALUE | Non-Uniform Rational B-Splines (NURBS) elements |
+   | LinearNonConf3D | - | 1 | 1 | VALUE | Piecewise-linear nonconforming finite elements in 3D |
+   | CrouzeixRaviart | - | - | - | - | Crouzeix-Raviart nonconforming elements in 2D |
+   | Local_[FENAME] | - | - | - | - | Special collection that builds a local version out of the FENAME collection |
+   |-|-|-|-|-|-|
+   | Linear | H1 | 1 | 1 | VALUE | Left in for backward compatibility, consider using H1_ |
+   | Quadratic | H1 | 2 | 1 | VALUE | Left in for backward compatibility, consider using H1_ |
+   | QuadraticPos | H1 | 2 | 2 | VALUE | Left in for backward compatibility, consider using H1_ |
+   | Cubic | H1 | 2 | 1 | VALUE | Left in for backward compatibility, consider using H1_ |
+   | Const2D | L2 | 0 | 1 | VALUE | Left in for backward compatibility, consider using L2_ |
+   | Const3D | L2 | 0 | 1 | VALUE | Left in for backward compatibility, consider using L2_ |
+   | LinearDiscont2D | L2 | 1 | 1 | VALUE | Left in for backward compatibility, consider using L2_ |
+   | GaussLinearDiscont2D | L2 | 1 | 0 | VALUE | Left in for backward compatibility, consider using L2_ |
+   | P1OnQuad | H1 | 1 | 1 | VALUE | Linear P1 element with 3 nodes on a square |
+   | QuadraticDiscont2D | L2 | 2 | 1 | VALUE | Left in for backward compatibility, consider using L2_ |
+   | QuadraticPosDiscont2D | L2 | 2 | 2 | VALUE | Left in for backward compatibility, consider using L2_ |
+   | GaussQuadraticDiscont2D | L2 | 2 | 0 | VALUE | Left in for backward compatibility, consider using L2_ |
+   | CubicDiscont2D | L2 | 3 | 1 | VALUE | Left in for backward compatibility, consider using L2_ |
+   | LinearDiscont3D | L2 | 1 | 1 | VALUE | Left in for backward compatibility, consider using L2_ |
+   | QuadraticDiscont3D | L2 | 2 | 1 | VALUE | Left in for backward compatibility, consider using L2_ |
+   | ND1_3D | H(Curl) | 1 | 1 / 0 | H_CURL | Left in for backward compatibility, consider using ND_ |
+   | RT0_2D | H(Div) | 1 | 1 / 0 | H_DIV | Left in for backward compatibility, consider using RT_ |
+   | RT1_2D | H(Div) | 2 | 1 / 0 | H_DIV | Left in for backward compatibility, consider using RT_ |
+   | RT2_2D | H(Div) | 3 | 1 / 0 | H_DIV | Left in for backward compatibility, consider using RT_ |
+   | RT0_3D | H(Div) | 1 | 1 / 0 | H_DIV | Left in for backward compatibility, consider using RT_ |
+   | RT1_3D | H(Div) | 2 | 1 / 0 | H_DIV | Left in for backward compatibility, consider using RT_ |
+
+   | Tag | Description |
+   | :------: | :--------: |
+   | [DIM]    | Dimension of the elements (1D, 2D, 3D) |
+   | [ORDER]  | Approximation order of the elements (P0, P1, P2, ...) |
+   | [BTYPE]  | BasisType of the element (0-GaussLegendre, 1 - GaussLobatto, 2-Bernstein, 3-OpenUniform, 4-CloseUniform, 5-OpenHalfUniform) |
+   | [OBTYPE] | Open BasisType of the element for elements which have both types |
+   | [CBTYPE] | Closed BasisType of the element for elements which have both types |
+
+   [FENAME]  Is a special case for the Local FEC which generates a local version of a given
+   FEC.  It is selected from one of (BiCubic2DFiniteElement, Quad_Q3, Nedelec1HexFiniteElement,
+      Hex_ND1, H1_[DIM]_[ORDER],H1Pos_[DIM]_[ORDER], L2_[DIM]_[ORDER] )
+   */
    static FiniteElementCollection *New(const char *name);
 
    /** @brief Get the local dofs for a given sub-manifold.
 
-      Return the local dofs for a SDim-dimensional sub-manifold (0D - vertex,
-      1D - edge, 2D - face) including those on its boundary. The local index of
-      the sub-manifold (inside Geom) and its orientation are given by the
-      parameter Info = 64 * SubIndex + SubOrientation. Naturally, it is assumed
-      that 0 <= SDim <= Dim(Geom). */
+      Return the local dofs for a SDim-dimensional sub-manifold (0D - vertex, 1D
+      - edge, 2D - face) including those on its boundary. The local index of the
+      sub-manifold (inside Geom) and its orientation are given by the parameter
+      Info = 64 * SubIndex + SubOrientation. Naturally, it is assumed that 0 <=
+      SDim <= Dim(Geom). */
    void SubDofOrder(Geometry::Type Geom, int SDim, int Info,
                     Array<int> &dofs) const;
 };
@@ -102,6 +178,7 @@ public:
    virtual const int *DofOrderForOrientation(Geometry::Type GeomType,
                                              int Or) const;
    virtual const char *Name() const { return h1_name; }
+   virtual int GetContType() const { return CONTINUOUS; }
    FiniteElementCollection *GetTraceCollection() const;
 
    int GetBasisType() const { return b_type; }
@@ -111,8 +188,8 @@ public:
    virtual ~H1_FECollection();
 };
 
-/** Arbitrary order H1-conforming (continuous) finite elements with positive
-    basis functions. */
+/** @brief Arbitrary order H1-conforming (continuous) finite elements with
+    positive basis functions. */
 class H1Pos_FECollection : public H1_FECollection
 {
 public:
@@ -120,6 +197,7 @@ public:
       : H1_FECollection(p, dim, BasisType::Positive) { }
 };
 
+
 /** Arbitrary order H1-conforming (continuous) serendipity finite elements;
     Current implementation works in 2D only; 3D version is in development. */
 class H1Ser_FECollection : public H1_FECollection
@@ -129,9 +207,9 @@ public:
       : H1_FECollection(p, dim, BasisType::Serendipity) { };
 };
 
-/** Arbitrary order "H^{1/2}-conforming" trace finite elements defined on the
-    interface between mesh elements (faces,edges,vertices); these are the trace
-    FEs of the H1-conforming FEs. */
+/** @brief Arbitrary order "H^{1/2}-conforming" trace finite elements defined on
+    the interface between mesh elements (faces,edges,vertices); these are the
+    trace FEs of the H1-conforming FEs. */
 class H1_Trace_FECollection : public H1_FECollection
 {
 public:
@@ -174,6 +252,8 @@ public:
                                              int Or) const;
    virtual const char *Name() const { return d_name; }
 
+   virtual int GetContType() const { return DISCONTINUOUS; }
+
    virtual const FiniteElement *TraceFiniteElementForGeometry(
       Geometry::Type GeomType) const
    {
@@ -221,14 +301,15 @@ public:
    virtual const int *DofOrderForOrientation(Geometry::Type GeomType,
                                              int Or) const;
    virtual const char *Name() const { return rt_name; }
+   virtual int GetContType() const { return NORMAL; }
    FiniteElementCollection *GetTraceCollection() const;
 
    virtual ~RT_FECollection();
 };
 
-/** Arbitrary order "H^{-1/2}-conforming" face finite elements defined on the
-    interface between mesh elements (faces); these are the normal trace FEs of
-    the H(div)-conforming FEs. */
+/** @brief Arbitrary order "H^{-1/2}-conforming" face finite elements defined on
+    the interface between mesh elements (faces); these are the normal trace FEs
+    of the H(div)-conforming FEs. */
 class RT_Trace_FECollection : public RT_FECollection
 {
 public:
@@ -270,14 +351,15 @@ public:
    virtual const int *DofOrderForOrientation(Geometry::Type GeomType,
                                              int Or) const;
    virtual const char *Name() const { return nd_name; }
+   virtual int GetContType() const { return TANGENTIAL; }
    FiniteElementCollection *GetTraceCollection() const;
 
    virtual ~ND_FECollection();
 };
 
-/** Arbitrary order H(curl)-trace finite elements defined on the interface
-    between mesh elements (faces,edges); these are the tangential trace FEs of
-    the H(curl)-conforming FEs. */
+/** @brief Arbitrary order H(curl)-trace finite elements defined on the
+    interface between mesh elements (faces,edges); these are the tangential
+    trace FEs of the H(curl)-conforming FEs. */
 class ND_Trace_FECollection : public ND_FECollection
 {
 public:
@@ -334,13 +416,15 @@ public:
 
    virtual const char *Name() const { return name; }
 
+   virtual int GetContType() const { return CONTINUOUS; }
+
    FiniteElementCollection *GetTraceCollection() const;
 
    virtual ~NURBSFECollection();
 };
 
 
-/// Piecewise-(bi)linear continuous finite elements.
+/// Piecewise-(bi/tri)linear continuous finite elements.
 class LinearFECollection : public FiniteElementCollection
 {
 private:
@@ -363,6 +447,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "Linear"; }
+
+   virtual int GetContType() const { return CONTINUOUS; }
 };
 
 /// Piecewise-(bi)quadratic continuous finite elements.
@@ -389,6 +475,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "Quadratic"; }
+
+   virtual int GetContType() const { return CONTINUOUS; }
 };
 
 /// Version of QuadraticFECollection with positive basis functions.
@@ -410,6 +498,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "QuadraticPos"; }
+
+   virtual int GetContType() const { return CONTINUOUS; }
 };
 
 /// Piecewise-(bi)cubic continuous finite elements.
@@ -437,6 +527,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "Cubic"; }
+
+   virtual int GetContType() const { return CONTINUOUS; }
 };
 
 /// Crouzeix-Raviart nonconforming elements in 2D.
@@ -458,6 +550,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "CrouzeixRaviart"; }
+
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Piecewise-linear nonconforming finite elements in 3D.
@@ -481,11 +575,13 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "LinearNonConf3D"; }
+
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 
-/** First order Raviart-Thomas finite elements in 2D. This class is kept only
-    for backward compatibility, consider using RT_FECollection instead. */
+/** @brief First order Raviart-Thomas finite elements in 2D. This class is kept
+    only for backward compatibility, consider using RT_FECollection instead. */
 class RT0_2DFECollection : public FiniteElementCollection
 {
 private:
@@ -504,10 +600,12 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RT0_2D"; }
+
+   virtual int GetContType() const { return NORMAL; }
 };
 
-/** Second order Raviart-Thomas finite elements in 2D. This class is kept only
-    for backward compatibility, consider using RT_FECollection instead. */
+/** @brief Second order Raviart-Thomas finite elements in 2D. This class is kept
+    only for backward compatibility, consider using RT_FECollection instead. */
 class RT1_2DFECollection : public FiniteElementCollection
 {
 private:
@@ -526,10 +624,12 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RT1_2D"; }
+
+   virtual int GetContType() const { return NORMAL; }
 };
 
-/** Third order Raviart-Thomas finite elements in 2D. This class is kept only
-    for backward compatibility, consider using RT_FECollection instead. */
+/** @brief Third order Raviart-Thomas finite elements in 2D. This class is kept
+    only for backward compatibility, consider using RT_FECollection instead. */
 class RT2_2DFECollection : public FiniteElementCollection
 {
 private:
@@ -548,10 +648,13 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RT2_2D"; }
+
+   virtual int GetContType() const { return NORMAL; }
 };
 
-/** Piecewise-constant discontinuous finite elements in 2D. This class is kept
-    only for backward compatibility, consider using L2_FECollection instead. */
+/** @brief Piecewise-constant discontinuous finite elements in 2D. This class is
+    kept only for backward compatibility, consider using L2_FECollection
+    instead. */
 class Const2DFECollection : public FiniteElementCollection
 {
 private:
@@ -569,10 +672,13 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "Const2D"; }
+
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
-/** Piecewise-linear discontinuous finite elements in 2D. This class is kept
-    only for backward compatibility, consider using L2_FECollection instead. */
+/** @brief Piecewise-linear discontinuous finite elements in 2D. This class is
+    kept only for backward compatibility, consider using L2_FECollection
+    instead. */
 class LinearDiscont2DFECollection : public FiniteElementCollection
 {
 private:
@@ -591,6 +697,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "LinearDiscont2D"; }
+
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Version of LinearDiscont2DFECollection with dofs in the Gaussian points.
@@ -613,6 +721,8 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "GaussLinearDiscont2D"; }
+
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Linear (P1) finite elements on quadrilaterals.
@@ -628,10 +738,12 @@ public:
    virtual const int *DofOrderForOrientation(Geometry::Type GeomType,
                                              int Or) const;
    virtual const char * Name() const { return "P1OnQuad"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
-/** Piecewise-quadratic discontinuous finite elements in 2D. This class is kept
-    only for backward compatibility, consider using L2_FECollection instead. */
+/** @brief Piecewise-quadratic discontinuous finite elements in 2D. This class
+    is kept only for backward compatibility, consider using L2_FECollection
+    instead. */
 class QuadraticDiscont2DFECollection : public FiniteElementCollection
 {
 private:
@@ -650,6 +762,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "QuadraticDiscont2D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Version of QuadraticDiscont2DFECollection with positive basis functions.
@@ -667,6 +780,7 @@ public:
                                              int Or) const
    { return NULL; }
    virtual const char * Name() const { return "QuadraticPosDiscont2D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Version of QuadraticDiscont2DFECollection with dofs in the Gaussian points.
@@ -689,10 +803,12 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "GaussQuadraticDiscont2D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
-/** Piecewise-cubic discontinuous finite elements in 2D. This class is kept
-    only for backward compatibility, consider using L2_FECollection instead. */
+/** @brief Piecewise-cubic discontinuous finite elements in 2D. This class is
+    kept only for backward compatibility, consider using L2_FECollection
+    instead. */
 class CubicDiscont2DFECollection : public FiniteElementCollection
 {
 private:
@@ -711,10 +827,12 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "CubicDiscont2D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
-/** Piecewise-constant discontinuous finite elements in 3D. This class is kept
-    only for backward compatibility, consider using L2_FECollection instead. */
+/** @brief Piecewise-constant discontinuous finite elements in 3D. This class is
+    kept only for backward compatibility, consider using L2_FECollection
+    instead. */
 class Const3DFECollection : public FiniteElementCollection
 {
 private:
@@ -734,10 +852,12 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "Const3D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
-/** Piecewise-linear discontinuous finite elements in 3D. This class is kept
-    only for backward compatibility, consider using L2_FECollection instead. */
+/** @brief Piecewise-linear discontinuous finite elements in 3D. This class is
+    kept only for backward compatibility, consider using L2_FECollection
+    instead. */
 class LinearDiscont3DFECollection : public FiniteElementCollection
 {
 private:
@@ -756,10 +876,12 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "LinearDiscont3D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
-/** Piecewise-quadratic discontinuous finite elements in 3D. This class is kept
-    only for backward compatibility, consider using L2_FECollection instead. */
+/** @brief Piecewise-quadratic discontinuous finite elements in 3D. This class
+    is kept only for backward compatibility, consider using L2_FECollection
+    instead. */
 class QuadraticDiscont3DFECollection : public FiniteElementCollection
 {
 private:
@@ -778,6 +900,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "QuadraticDiscont3D"; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 /// Finite element collection on a macro-element.
@@ -803,10 +926,12 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RefinedLinear"; }
+   virtual int GetContType() const { return CONTINUOUS; }
 };
 
-/** Lowest order Nedelec finite elements in 3D. This class is kept only for
-    backward compatibility, consider using the new ND_FECollection instead. */
+/** @brief Lowest order Nedelec finite elements in 3D. This class is kept only
+    for backward compatibility, consider using the new ND_FECollection
+    instead. */
 class ND1_3DFECollection : public FiniteElementCollection
 {
 private:
@@ -825,10 +950,11 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "ND1_3D"; }
+   virtual int GetContType() const { return TANGENTIAL; }
 };
 
-/** First order Raviart-Thomas finite elements in 3D. This class is kept only
-    for backward compatibility, consider using RT_FECollection instead. */
+/** @brief First order Raviart-Thomas finite elements in 3D. This class is kept
+    only for backward compatibility, consider using RT_FECollection instead. */
 class RT0_3DFECollection : public FiniteElementCollection
 {
 private:
@@ -848,10 +974,11 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RT0_3D"; }
+   virtual int GetContType() const { return NORMAL; }
 };
 
-/** Second order Raviart-Thomas finite elements in 3D. This class is kept only
-    for backward compatibility, consider using RT_FECollection instead. */
+/** @brief Second order Raviart-Thomas finite elements in 3D. This class is kept
+    only for backward compatibility, consider using RT_FECollection instead. */
 class RT1_3DFECollection : public FiniteElementCollection
 {
 private:
@@ -870,6 +997,7 @@ public:
                                              int Or) const;
 
    virtual const char * Name() const { return "RT1_3D"; }
+   virtual int GetContType() const { return NORMAL; }
 };
 
 /// Discontinuous collection defined locally by a given finite element.
@@ -894,6 +1022,7 @@ public:
    virtual const char *Name() const { return d_name; }
 
    virtual ~Local_FECollection() { delete Local_Element; }
+   virtual int GetContType() const { return DISCONTINUOUS; }
 };
 
 }

fem/fem.hpp

@@ -34,6 +34,7 @@
 #include "tmop.hpp"
 #include "tmop_tools.hpp"
 #include "gslib.hpp"
+#include "adnonlininteg.hpp"
 #include "restriction.hpp"
 #include "quadinterpolator.hpp"
 #include "quadinterpolator_face.hpp"

fem/fespace.cpp

@@ -60,7 +60,7 @@ FiniteElementSpace::FiniteElementSpace()
    : mesh(NULL), fec(NULL), vdim(0), ordering(Ordering::byNODES),
      ndofs(0), nvdofs(0), nedofs(0), nfdofs(0), nbdofs(0),
      fdofs(NULL), bdofs(NULL),
-     elem_dof(NULL), bdrElem_dof(NULL),
+     elem_dof(NULL), bdrElem_dof(NULL), face_dof(NULL),
      NURBSext(NULL), own_ext(false),
      cP(NULL), cR(NULL), cP_is_set(false),
      Th(Operator::ANY_TYPE),
@@ -233,6 +233,54 @@ void FiniteElementSpace::BuildElementToDofTable() const
    elem_dof = el_dof;
 }
 
+void FiniteElementSpace::BuildBdrElementToDofTable() const
+{
+   if (bdrElem_dof) { return; }
+
+   Table *bel_dof = new Table;
+   Array<int> dofs;
+   bel_dof->MakeI(mesh->GetNBE());
+   for (int i = 0; i < mesh->GetNBE(); i++)
+   {
+      GetBdrElementDofs(i, dofs);
+      bel_dof->AddColumnsInRow(i, dofs.Size());
+   }
+   bel_dof->MakeJ();
+   for (int i = 0; i < mesh->GetNBE(); i++)
+   {
+      GetBdrElementDofs(i, dofs);
+      bel_dof->AddConnections(i, (int *)dofs, dofs.Size());
+   }
+   bel_dof->ShiftUpI();
+   bdrElem_dof = bel_dof;
+}
+
+void FiniteElementSpace::BuildFaceToDofTable() const
+{
+   // Here, "face" == (dim-1)-dimensional mesh entity.
+
+   if (face_dof) { return; }
+
+   if (NURBSext) { BuildNURBSFaceToDofTable(); return; }
+
+   Table *fc_dof = new Table;
+   Array<int> dofs;
+   fc_dof->MakeI(mesh->GetNumFaces());
+   for (int i = 0; i < fc_dof->Size(); i++)
+   {
+      GetFaceDofs(i, dofs);
+      fc_dof->AddColumnsInRow(i, dofs.Size());
+   }
+   fc_dof->MakeJ();
+   for (int i = 0; i < fc_dof->Size(); i++)
+   {
+      GetFaceDofs(i, dofs);
+      fc_dof->AddConnections(i, (int *)dofs, dofs.Size());
+   }
+   fc_dof->ShiftUpI();
+   face_dof = fc_dof;
+}
+
 void FiniteElementSpace::RebuildElementToDofTable()
 {
    delete elem_dof;
@@ -1456,6 +1504,7 @@ void FiniteElementSpace::Constructor(Mesh *mesh, NURBSExtension *NURBSext,
    this->ordering = (Ordering::Type) ordering;
 
    elem_dof = NULL;
+   face_dof = NULL;
    sequence = mesh->GetSequence();
    Th.SetType(Operator::ANY_TYPE);
 
@@ -1505,6 +1554,8 @@ NURBSExtension *FiniteElementSpace::StealNURBSext()
 
 void FiniteElementSpace::UpdateNURBS()
 {
+   MFEM_VERIFY(NURBSext, "NURBSExt not defined.");
+
    nvdofs = 0;
    nedofs = 0;
    nfdofs = 0;
@@ -1512,6 +1563,10 @@ void FiniteElementSpace::UpdateNURBS()
    fdofs = NULL;
    bdofs = NULL;
 
+   delete face_dof;
+   face_dof = NULL;
+   face_to_be.DeleteAll();
+
    dynamic_cast<const NURBSFECollection *>(fec)->Reset();
 
    ndofs = NURBSext->GetNDof();
@@ -1519,6 +1574,55 @@ void FiniteElementSpace::UpdateNURBS()
    bdrElem_dof = NURBSext->GetBdrElementDofTable();
 }
 
+void FiniteElementSpace::BuildNURBSFaceToDofTable() const
+{
+   if (face_dof) { return; }
+
+   const int dim = mesh->Dimension();
+
+   // Find bdr to face mapping
+   face_to_be.SetSize(GetNF());
+   face_to_be = -1;
+   for (int b = 0; b < GetNBE(); b++)
+   {
+      int f = mesh->GetBdrElementEdgeIndex(b);
+      face_to_be[f] = b;
+   }
+
+   // Loop over faces in correct order, to prevent a sort
+   // Sort will destroy orientation info in ordering of dofs
+   Array<Connection> face_dof_list;
+   Array<int> row;
+   for (int f = 0; f < GetNF(); f++)
+   {
+      int b = face_to_be[f];
+      if (b == -1) { continue; }
+      // FIXME: this assumes the boundary element and the face element have the
+      //        same orientation.
+      if (dim > 1)
+      {
+         const Element *fe = mesh->GetFace(f);
+         const Element *be = mesh->GetBdrElement(b);
+         const int nv = be->GetNVertices();
+         const int *fv = fe->GetVertices();
+         const int *bv = be->GetVertices();
+         for (int i = 0; i < nv; i++)
+         {
+            MFEM_VERIFY(fv[i] == bv[i],
+                        "non-matching face and boundary elements detected!");
+         }
+      }
+      GetBdrElementDofs(b, row);
+      Connection conn(f,0);
+      for (int i = 0; i < row.Size(); i++)
+      {
+         conn.to = row[i];
+         face_dof_list.Append(conn);
+      }
+   }
+   face_dof = new Table(GetNF(), face_dof_list);
+}
+
 void FiniteElementSpace::Construct()
 {
    // This method should be used only for non-NURBS spaces.
@@ -1526,6 +1630,7 @@ void FiniteElementSpace::Construct()
 
    elem_dof = NULL;
    bdrElem_dof = NULL;
+   face_dof = NULL;
 
    ndofs = 0;
    nedofs = nfdofs = nbdofs = 0;
@@ -1788,59 +1893,68 @@ void FiniteElementSpace::GetBdrElementDofs(int i, Array<int> &dofs) const
 
 void FiniteElementSpace::GetFaceDofs(int i, Array<int> &dofs) const
 {
-   int j, k, nv, ne, nf, nd, dim = mesh->Dimension();
-   Array<int> V, E, Eo;
-   const int *ind;
+   // If face_dof is already built, use it.
+   // If it is not and we have a NURBS space, build the face_dof and use it.
+   if (face_dof || (NURBSext && (BuildNURBSFaceToDofTable(), true)))
+   {
+      face_dof->GetRow(i, dofs);
+   }
+   else
+   {
+      int j, k, nv, ne, nf, nd, dim = mesh->Dimension();
+      Array<int> V, E, Eo;
+      const int *ind;
 
-   // for 1D, 2D and 3D faces
-   nv = fec->DofForGeometry(Geometry::POINT);
-   ne = (dim > 1) ? fec->DofForGeometry(Geometry::SEGMENT) : 0;
-   if (nv > 0)
-   {
-      mesh->GetFaceVertices(i, V);
-   }
-   if (ne > 0)
-   {
-      mesh->GetFaceEdges(i, E, Eo);
-   }
-   nf = (fdofs) ? (fdofs[i+1]-fdofs[i]) : (0);
-   nd = V.Size() * nv + E.Size() * ne + nf;
-   dofs.SetSize(nd);
-   if (nv > 0)
-   {
-      for (k = 0; k < V.Size(); k++)
+      // for 1D, 2D and 3D faces
+      nv = fec->DofForGeometry(Geometry::POINT);
+      ne = (dim > 1) ? fec->DofForGeometry(Geometry::SEGMENT) : 0;
+      if (nv > 0)
       {
-         for (j = 0; j < nv; j++)
-         {
-            dofs[k*nv+j] = V[k]*nv+j;
-         }
+         mesh->GetFaceVertices(i, V);
       }
-   }
-   nv *= V.Size();
-   if (ne > 0)
-   {
-      for (k = 0; k < E.Size(); k++)
+      if (ne > 0)
       {
-         ind = fec->DofOrderForOrientation(Geometry::SEGMENT, Eo[k]);
-         for (j = 0; j < ne; j++)
+         mesh->GetFaceEdges(i, E, Eo);
+      }
+      nf = (fdofs) ? (fdofs[i+1]-fdofs[i]) : (0);
+      nd = V.Size() * nv + E.Size() * ne + nf;
+      dofs.SetSize(nd);
+      if (nv > 0)
+      {
+         for (k = 0; k < V.Size(); k++)
          {
-            if (ind[j] < 0)
+            for (j = 0; j < nv; j++)
             {
-               dofs[nv+k*ne+j] = -1 - ( nvdofs+E[k]*ne+(-1-ind[j]) );
-            }
-            else
-            {
-               dofs[nv+k*ne+j] = nvdofs+E[k]*ne+ind[j];
+               dofs[k*nv+j] = V[k]*nv+j;
             }
          }
       }
-   }
-   ne = nv + ne * E.Size();
-   if (nf > 0)
-   {
-      for (j = nvdofs+nedofs+fdofs[i], k = 0; k < nf; j++, k++)
+      nv *= V.Size();
+      if (ne > 0)
       {
-         dofs[ne+k] = j;
+         for (k = 0; k < E.Size(); k++)
+         {
+            ind = fec->DofOrderForOrientation(Geometry::SEGMENT, Eo[k]);
+            for (j = 0; j < ne; j++)
+            {
+               if (ind[j] < 0)
+               {
+                  dofs[nv+k*ne+j] = -1 - ( nvdofs+E[k]*ne+(-1-ind[j]) );
+               }
+               else
+               {
+                  dofs[nv+k*ne+j] = nvdofs+E[k]*ne+ind[j];
+               }
+            }
+         }
+      }
+      ne = nv + ne * E.Size();
+      if (nf > 0)
+      {
+         for (j = nvdofs+nedofs+fdofs[i], k = 0; k < nf; j++, k++)
+         {
+            dofs[ne+k] = j;
+         }
       }
    }
 }
@@ -1969,14 +2083,21 @@ const FiniteElement *FiniteElementSpace::GetFaceElement(int i) const
          fe = fec->FiniteElementForGeometry(mesh->GetFaceBaseGeometry(i));
    }
 
-   // if (NURBSext)
-   //    NURBSext->LoadFaceElement(i, fe);
+   if (NURBSext)
+   {
+      // Ensure 'face_to_be' is built:
+      if (!face_dof) { BuildNURBSFaceToDofTable(); }
+      MFEM_ASSERT(face_to_be[i] >= 0,
+                  "NURBS mesh: only boundary faces are supported!");
+      NURBSext->LoadBE(face_to_be[i], fe);
+   }
 
    return fe;
 }
 
 const FiniteElement *FiniteElementSpace::GetEdgeElement(int i) const
 {
+   MFEM_ASSERT(mesh->Dimension() > 1, "No edges with a mesh dimension < 2");
    return fec->FiniteElementForGeometry(Geometry::SEGMENT);
 }
 
@@ -2024,11 +2145,14 @@ void FiniteElementSpace::Destroy()
    if (NURBSext)
    {
       if (own_ext) { delete NURBSext; }
+      delete face_dof;
+      face_to_be.DeleteAll();
    }
    else
    {
       delete elem_dof;
       delete bdrElem_dof;
+      delete face_dof;
 
       delete [] bdofs;
       delete [] fdofs;
@@ -2615,7 +2739,9 @@ const Operator &InterpolationGridTransfer::BackwardOperator()
 
 L2ProjectionGridTransfer::L2Projection::L2Projection(
    const FiniteElementSpace &fes_ho_, const FiniteElementSpace &fes_lor_)
-   : fes_ho(fes_ho_), fes_lor(fes_lor_)
+   : Operator(fes_lor_.GetVSize(), fes_ho_.GetVSize()),
+     fes_ho(fes_ho_),
+     fes_lor(fes_lor_)
 {
    Mesh *mesh_ho = fes_ho.GetMesh();
    MFEM_VERIFY(mesh_ho->GetNumGeometries(mesh_ho->Dimension()) <= 1,

fem/fespace.hpp

@@ -111,7 +111,9 @@ protected:
    int *fdofs, *bdofs;
 
    mutable Table *elem_dof; // if NURBS FE space, not owned; otherwise, owned.
-   Table *bdrElem_dof; // used only with NURBS FE spaces; not owned.
+   mutable Table *bdrElem_dof; // not owned only if NURBS FE space.
+   mutable Table *face_dof; // owned
+   mutable Array<int> face_to_be; // used only with NURBS FE spaces; owned.
 
    Array<int> dof_elem_array, dof_ldof_array;
 
@@ -158,6 +160,14 @@ protected:
    void Destroy();
 
    void BuildElementToDofTable() const;
+   void BuildBdrElementToDofTable() const;
+   void BuildFaceToDofTable() const;
+
+   /** @brief  Generates partial face_dof table for a NURBS space.
+
+       The table is only defined for exterior faces that coincide with a
+       boundary. */
+   void BuildNURBSFaceToDofTable() const;
 
    /// Helpers to remove encoded sign from a DOF
    static inline int DecodeDof(int dof)
@@ -206,7 +216,7 @@ protected:
       virtual ~RefinementOperator();
    };
 
-   // Derefinement operator, used by the friend class InterpolationGridTransfer.
+   /// Derefinement operator, used by the friend class InterpolationGridTransfer.
    class DerefinementOperator : public Operator
    {
       const FiniteElementSpace *fine_fes; // Not owned.
@@ -225,12 +235,12 @@ protected:
       virtual ~DerefinementOperator();
    };
 
-   // This method makes the same assumptions as the method:
-   //    void GetLocalRefinementMatrices(
-   //       const FiniteElementSpace &coarse_fes, Geometry::Type geom,
-   //       DenseTensor &localP) const
-   // which is defined below. It also assumes that the coarse fes and this have
-   // the same vector dimension, vdim.
+   /** This method makes the same assumptions as the method:
+       void GetLocalRefinementMatrices(
+           const FiniteElementSpace &coarse_fes, Geometry::Type geom,
+           DenseTensor &localP) const
+       which is defined below. It also assumes that the coarse fes and this have
+       the same vector dimension, vdim. */
    SparseMatrix *RefinementMatrix_main(const int coarse_ndofs,
                                        const Table &coarse_elem_dof,
                                        const DenseTensor localP[]) const;
@@ -248,11 +258,13 @@ protected:
    /// Calculate GridFunction restriction matrix after mesh derefinement.
    SparseMatrix* DerefinementMatrix(int old_ndofs, const Table* old_elem_dof);
 
-   // This method assumes that this->mesh is a refinement of coarse_fes->mesh
-   // and that the CoarseFineTransformations of this->mesh are set accordingly.
-   // Another assumption is that the FEs of this use the same MapType as the FEs
-   // of coarse_fes. Finally, it assumes that the spaces this and coarse_fes are
-   // NOT variable-order spaces.
+   /** @brief Return in @a localP the local refinement matrices that map
+       between fespaces after mesh refinement. */
+   /** This method assumes that this->mesh is a refinement of coarse_fes->mesh
+       and that the CoarseFineTransformations of this->mesh are set accordingly.
+       Another assumption is that the FEs of this use the same MapType as the FEs
+       of coarse_fes. Finally, it assumes that the spaces this and coarse_fes are
+       NOT variable-order spaces. */
    void GetLocalRefinementMatrices(const FiniteElementSpace &coarse_fes,
                                    Geometry::Type geom,
                                    DenseTensor &localP) const;
@@ -467,11 +479,11 @@ public:
    /// Returns indexes of degrees of freedom for i'th boundary element.
    virtual void GetBdrElementDofs(int i, Array<int> &dofs) const;
 
-   /** Returns the indexes of the degrees of freedom for i'th face
+   /** @brief eturns the indexes of the degrees of freedom for i'th face
        including the dofs for the edges and the vertices of the face. */
    virtual void GetFaceDofs(int i, Array<int> &dofs) const;
 
-   /** Returns the indexes of the degrees of freedom for i'th edge
+   /** @brief Returns the indexes of the degrees of freedom for i'th edge
        including the dofs for the vertices of the edge. */
    void GetEdgeDofs(int i, Array<int> &dofs) const;
 
@@ -526,28 +538,59 @@ public:
        is preserved. */
    void ReorderElementToDofTable();
 
+   /** @brief Return a reference to the internal Table that stores the lists of
+       scalar dofs, for each mesh element, as returned by GetElementDofs(). */
+   const Table &GetElementToDofTable() const { return *elem_dof; }
+
+   /** @brief Return a reference to the internal Table that stores the lists of
+       scalar dofs, for each boundary mesh element, as returned by
+       GetBdrElementDofs(). */
+   const Table &GetBdrElementToDofTable() const
+   { if (!bdrElem_dof) { BuildBdrElementToDofTable(); } return *bdrElem_dof; }
+
+   /** @brief Return a reference to the internal Table that stores the lists of
+       scalar dofs, for each face in the mesh, as returned by GetFaceDofs(). In
+       this context, "face" refers to a (dim-1)-dimensional mesh entity. */
+   /** @note In the case of a NURBS space, the rows corresponding to interior
+       faces will be empty. */
+   const Table &GetFaceToDofTable() const
+   { if (!face_dof) { BuildFaceToDofTable(); } return *face_dof; }
+
+   /** @brief Initialize internal data that enables the use of the methods
+       GetElementForDof() and GetLocalDofForDof(). */
    void BuildDofToArrays();
 
-   const Table &GetElementToDofTable() const { return *elem_dof; }
-   const Table &GetBdrElementToDofTable() const { return *bdrElem_dof; }
-
+   /// Return the index of the first element that contains dof @a i.
+   /** This method can be called only after setup is performed using the method
+       BuildDofToArrays(). */
    int GetElementForDof(int i) const { return dof_elem_array[i]; }
+   /// Return the local dof index in the first element that contains dof @a i.
+   /** This method can be called only after setup is performed using the method
+       BuildDofToArrays(). */
    int GetLocalDofForDof(int i) const { return dof_ldof_array[i]; }
 
-   /// Returns pointer to the FiniteElement associated with i'th element.
+   /** @brief Returns pointer to the FiniteElement in the FiniteElementCollection
+        associated with i'th element in the mesh object. */
    const FiniteElement *GetFE(int i) const;
 
-   /// Returns pointer to the FiniteElement for the i'th boundary element.
+   /** @brief Returns pointer to the FiniteElement in the FiniteElementCollection
+        associated with i'th boundary face in the mesh object. */
    const FiniteElement *GetBE(int i) const;
 
+   /** @brief Returns pointer to the FiniteElement in the FiniteElementCollection
+        associated with i'th face in the mesh object.  Faces in this case refer
+        to the MESHDIM-1 primitive so in 2D they are segments and in 1D they are
+        points.*/
    const FiniteElement *GetFaceElement(int i) const;
 
+   /** @brief Returns pointer to the FiniteElement in the FiniteElementCollection
+        associated with i'th edge in the mesh object. */
    const FiniteElement *GetEdgeElement(int i) const;
 
    /// Return the trace element from element 'i' to the given 'geom_type'
    const FiniteElement *GetTraceElement(int i, Geometry::Type geom_type) const;
 
-   /** Mark degrees of freedom associated with boundary elements with
+   /** @brief Mark degrees of freedom associated with boundary elements with
        the specified boundary attributes (marked in 'bdr_attr_is_ess').
        For spaces with 'vdim' > 1, the 'component' parameter can be used
        to restricts the marked vDOFs to the specified component. */
@@ -555,7 +598,7 @@ public:
                                   Array<int> &ess_vdofs,
                                   int component = -1) const;
 
-   /** Get a list of essential true dofs, ess_tdof_list, corresponding to the
+   /** @brief Get a list of essential true dofs, ess_tdof_list, corresponding to the
        boundary attributes marked in the array bdr_attr_is_ess.
        For spaces with 'vdim' > 1, the 'component' parameter can be used
        to restricts the marked tDOFs to the specified component. */
@@ -566,19 +609,19 @@ public:
    /// Convert a Boolean marker array to a list containing all marked indices.
    static void MarkerToList(const Array<int> &marker, Array<int> &list);
 
-   /** Convert an array of indices (list) to a Boolean marker array where all
+   /** @brief Convert an array of indices (list) to a Boolean marker array where all
        indices in the list are marked with the given value and the rest are set
        to zero. */
    static void ListToMarker(const Array<int> &list, int marker_size,
                             Array<int> &marker, int mark_val = -1);
 
-   /** For a partially conforming FE space, convert a marker array (nonzero
+   /** @brief For a partially conforming FE space, convert a marker array (nonzero
        entries are true) on the partially conforming dofs to a marker array on
        the conforming dofs. A conforming dofs is marked iff at least one of its
        dependent dofs is marked. */
    void ConvertToConformingVDofs(const Array<int> &dofs, Array<int> &cdofs);
 
-   /** For a partially conforming FE space, convert a marker array (nonzero
+   /** @brief For a partially conforming FE space, convert a marker array (nonzero
        entries are true) on the conforming dofs to a marker array on the
        (partially conforming) dofs. A dof is marked iff it depends on a marked
        conforming dofs, where dependency is defined by the ConformingRestriction
@@ -586,15 +629,15 @@ public:
        conforming dof. */
    void ConvertFromConformingVDofs(const Array<int> &cdofs, Array<int> &dofs);
 
-   /** Generate the global restriction matrix from a discontinuous
+   /** @brief Generate the global restriction matrix from a discontinuous
        FE space to the continuous FE space of the same polynomial degree. */
    SparseMatrix *D2C_GlobalRestrictionMatrix(FiniteElementSpace *cfes);
 
-   /** Generate the global restriction matrix from a discontinuous
+   /** @brief Generate the global restriction matrix from a discontinuous
        FE space to the piecewise constant FE space. */
    SparseMatrix *D2Const_GlobalRestrictionMatrix(FiniteElementSpace *cfes);
 
-   /** Construct the restriction matrix from the FE space given by
+   /** @brief Construct the restriction matrix from the FE space given by
        (*this) to the lower degree FE space given by (*lfes) which
        is defined on the same mesh. */
    SparseMatrix *H2L_GlobalRestrictionMatrix(FiniteElementSpace *lfes);
@@ -631,7 +674,7 @@ public:
    virtual void GetTrueTransferOperator(const FiniteElementSpace &coarse_fes,
                                         OperatorHandle &T) const;
 
-   /** Reflect changes in the mesh: update number of DOFs, etc. Also, calculate
+   /** @brief Reflect changes in the mesh: update number of DOFs, etc. Also, calculate
        GridFunction transformation operator (unless want_transform is false).
        Safe to call multiple times, does nothing if space already up to date. */
    virtual void Update(bool want_transform = true);
@@ -669,6 +712,7 @@ public:
       return dynamic_cast<const L2_FECollection*>(fec) != NULL;
    }
 
+   /// Save finite element space to output stream @a out.
    void Save(std::ostream &out) const;
 
    /** @brief Read a FiniteElementSpace from a stream. The returned
@@ -906,7 +950,8 @@ protected:
       const L2Projection &l2proj;
 
    public:
-      L2Prolongation(const L2Projection &l2proj_) : l2proj(l2proj_) { }
+      L2Prolongation(const L2Projection &l2proj_)
+         : Operator(l2proj_.Width(), l2proj_.Height()), l2proj(l2proj_) { }
       void Mult(const Vector &x, Vector &y) const
       {
          l2proj.Prolongate(x, y);

fem/gridfunc.cpp

@@ -236,7 +236,6 @@ void GridFunction::MakeTRef(FiniteElementSpace *f, Vector &tv, int tv_offset)
    }
 }
 
-
 void GridFunction::SumFluxAndCount(BilinearFormIntegrator &blfi,
                                    GridFunction &flux,
                                    Array<int>& count,
@@ -617,17 +616,358 @@ int GridFunction::GetFaceValues(int i, int side, const IntegrationRule &ir,
    return dir;
 }
 
+void GridFunction::GetVectorValues(int i, const IntegrationRule &ir,
+                                   DenseMatrix &vals, DenseMatrix &tr) const
+{
+   ElementTransformation *Tr = fes->GetElementTransformation(i);
+   Tr->Transform(ir, tr);
+
+   GetVectorValues(*Tr, ir, vals);
+}
+
+void be_to_bfe(Geometry::Type geom, int o, const IntegrationPoint &ip,
+               IntegrationPoint &fip)
+{
+   if (geom == Geometry::TRIANGLE)
+   {
+      if (o == 2)
+      {
+         fip.x = 1.0 - ip.x - ip.y;
+         fip.y = ip.x;
+      }
+      else if (o == 4)
+      {
+         fip.x = ip.y;
+         fip.y = 1.0 - ip.x - ip.y;
+      }
+      else
+      {
+         fip.x = ip.x;
+         fip.y = ip.y;
+      }
+      fip.z = ip.z;
+   }
+   else
+   {
+      if (o == 2)
+      {
+         fip.x = ip.y;
+         fip.y = 1.0 - ip.x;
+      }
+      else if (o == 4)
+      {
+         fip.x = 1.0 - ip.x;
+         fip.y = 1.0 - ip.y;
+      }
+      else if (o == 6)
+      {
+         fip.x = 1.0 - ip.y;
+         fip.y = ip.x;
+      }
+      else
+      {
+         fip.x = ip.x;
+         fip.y = ip.y;
+      }
+      fip.z = ip.z;
+   }
+   fip.weight = ip.weight;
+   fip.index  = ip.index;
+}
+
+double GridFunction::GetValue(ElementTransformation &T,
+                              const IntegrationPoint &ip,
+                              int comp, Vector *tr) const
+{
+   if (tr)
+   {
+      T.SetIntPoint(&ip);
+      T.Transform(ip, *tr);
+   }
+
+   const FiniteElement * fe = NULL;
+   Array<int> dofs;
+
+   switch (T.ElementType)
+   {
+      case ElementTransformation::ELEMENT:
+         fe = fes->GetFE(T.ElementNo);
+         fes->GetElementDofs(T.ElementNo, dofs);
+         break;
+      case ElementTransformation::EDGE:
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            fe = fes->GetEdgeElement(T.ElementNo);
+            fes->GetEdgeDofs(T.ElementNo, dofs);
+         }
+         else
+         {
+            MFEM_ABORT("GridFunction::GetValue: Field continuity type \""
+                       << fes->FEColl()->GetContType() << "\" not supported "
+                       << "on mesh edges.");
+            return NAN;
+         }
+         break;
+      case ElementTransformation::FACE:
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            fe = fes->GetFaceElement(T.ElementNo);
+            fes->GetFaceDofs(T.ElementNo, dofs);
+         }
+         else
+         {
+            MFEM_ABORT("GridFunction::GetValue: Field continuity type \""
+                       << fes->FEColl()->GetContType() << "\" not supported "
+                       << "on mesh faces.");
+            return NAN;
+         }
+         break;
+      case ElementTransformation::BDR_ELEMENT:
+      {
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            // This is a continuous field so we can evaluate it on the boundary.
+            fe = fes->GetBE(T.ElementNo);
+            fes->GetBdrElementDofs(T.ElementNo, dofs);
+         }
+         else
+         {
+            // This is a discontinuous field which cannot be evaluated on the
+            // boundary so we'll evaluate it in the neighboring element.
+            FaceElementTransformations * FET =
+               fes->GetMesh()->GetBdrFaceTransformations(T.ElementNo);
+
+            // Boundary elements and Boundary Faces may have different
+            // orientations so adjust the integration point if necessary.
+            int o = 0;
+            if (fes->GetMesh()->Dimension() == 3)
+            {
+               int f;
+               fes->GetMesh()->GetBdrElementFace(T.ElementNo, &f, &o);
+            }
+
+            IntegrationPoint fip;
+            be_to_bfe(FET->GetGeometryType(), o, ip, fip);
+
+            // Compute and set the point in element 1 from fip
+            FET->SetAllIntPoints(&fip);
+            ElementTransformation & T1 = FET->GetElement1Transformation();
+            return GetValue(T1, T1.GetIntPoint(), comp);
+         }
+      }
+      break;
+      case ElementTransformation::BDR_FACE:
+      {
+         FaceElementTransformations * FET =
+            dynamic_cast<FaceElementTransformations *>(&T);
+
+         // Evaluate in neighboring element for both continuous and
+         // discontinuous fields (the integration point in T1 should have
+         // already been set).
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+         return GetValue(T1, T1.GetIntPoint(), comp);
+      }
+      default:
+      {
+         MFEM_ABORT("GridFunction::GetValue: Unsupported element type \""
+                    << T.ElementType << "\"");
+         return NAN;
+      }
+   }
+
+   fes->DofsToVDofs(comp-1, dofs);
+   Vector DofVal(dofs.Size()), LocVec;
+   if (fe->GetMapType() == FiniteElement::VALUE)
+   {
+      fe->CalcShape(ip, DofVal);
+   }
+   else
+   {
+      fe->CalcPhysShape(T, DofVal);
+   }
+   GetSubVector(dofs, LocVec);
+
+   return (DofVal * LocVec);
+}
+
+void GridFunction::GetValues(ElementTransformation &T,
+                             const IntegrationRule &ir,
+                             Vector &vals, int comp,
+                             DenseMatrix *tr) const
+{
+   if (tr)
+   {
+      T.Transform(ir, *tr);
+   }
+
+   int nip = ir.GetNPoints();
+   vals.SetSize(nip);
+   for (int j = 0; j < nip; j++)
+   {
+      const IntegrationPoint &ip = ir.IntPoint(j);
+      T.SetIntPoint(&ip);
+      vals[j] = GetValue(T, ip, comp);
+   }
+}
+
+void GridFunction::GetVectorValue(ElementTransformation &T,
+                                  const IntegrationPoint &ip,
+                                  Vector &val, Vector *tr) const
+{
+   if (tr)
+   {
+      T.SetIntPoint(&ip);
+      T.Transform(ip, *tr);
+   }
+
+   Array<int> vdofs;
+   const FiniteElement *fe = NULL;
+
+   switch (T.ElementType)
+   {
+      case ElementTransformation::ELEMENT:
+         fes->GetElementVDofs(T.ElementNo, vdofs);
+         fe = fes->GetFE(T.ElementNo);
+         break;
+      case ElementTransformation::EDGE:
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            fe = fes->GetEdgeElement(T.ElementNo);
+            fes->GetEdgeVDofs(T.ElementNo, vdofs);
+         }
+         else
+         {
+            MFEM_ABORT("GridFunction::GetVectorValue: Field continuity type \""
+                       << fes->FEColl()->GetContType() << "\" not supported "
+                       << "on mesh edges.");
+            return;
+         }
+         break;
+      case ElementTransformation::FACE:
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            fe = fes->GetFaceElement(T.ElementNo);
+            fes->GetFaceVDofs(T.ElementNo, vdofs);
+         }
+         else
+         {
+            MFEM_ABORT("GridFunction::GetVectorValue: Field continuity type \""
+                       << fes->FEColl()->GetContType() << "\" not supported "
+                       << "on mesh faces.");
+            return;
+         }
+         break;
+      case ElementTransformation::BDR_ELEMENT:
+      {
+         if (fes->FEColl()->GetContType() ==
+             FiniteElementCollection::CONTINUOUS)
+         {
+            // This is a continuous field so we can evaluate it on the boundary.
+            fes->GetBdrElementVDofs(T.ElementNo, vdofs);
+            fe = fes->GetBE(T.ElementNo);
+         }
+         else
+         {
+            // This is a discontinuous vector field which cannot be evaluated on
+            // the boundary so we'll evaluate it in the neighboring element.
+            FaceElementTransformations * FET =
+               fes->GetMesh()->GetBdrFaceTransformations(T.ElementNo);
+
+            // Boundary elements and Boundary Faces may have different
+            // orientations so adjust the integration point if necessary.
+            int o = 0;
+            if (fes->GetMesh()->Dimension() == 3)
+            {
+               int f;
+               fes->GetMesh()->GetBdrElementFace(T.ElementNo, &f, &o);
+            }
+
+            IntegrationPoint fip;
+            be_to_bfe(FET->GetGeometryType(), o, ip, fip);
+
+            // Compute and set the point in element 1 from fip
+            FET->SetAllIntPoints(&fip);
+            ElementTransformation & T1 = FET->GetElement1Transformation();
+            return GetVectorValue(T1, T1.GetIntPoint(), val);
+         }
+      }
+      break;
+      case ElementTransformation::BDR_FACE:
+      {
+         FaceElementTransformations * FET =
+            dynamic_cast<FaceElementTransformations *>(&T);
+
+         // Evaluate in neighboring element for both continuous and
+         // discontinuous fields (the integration point in T1 should have
+         // already been set).
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+         return GetVectorValue(T1, T1.GetIntPoint(), val);
+      }
+      default:
+      {
+         MFEM_ABORT("GridFunction::GetVectorValue: Unsupported element type \""
+                    << T.ElementType << "\"");
+         if (val.Size() > 0) { val = NAN; }
+         return;
+      }
+   }
+
+   int dof = fe->GetDof();
+   Vector loc_data;
+   GetSubVector(vdofs, loc_data);
+   if (fe->GetRangeType() == FiniteElement::SCALAR)
+   {
+      Vector shape(dof);
+      if (fe->GetMapType() == FiniteElement::VALUE)
+      {
+         fe->CalcShape(ip, shape);
+      }
+      else
+      {
+         fe->CalcPhysShape(T, shape);
+      }
+      int vdim = fes->GetVDim();
+      val.SetSize(vdim);
+      for (int k = 0; k < vdim; k++)
+      {
+         val(k) = shape * ((const double *)loc_data + dof * k);
+      }
+   }
+   else
+   {
+      int spaceDim = fes->GetMesh()->SpaceDimension();
+      DenseMatrix vshape(dof, spaceDim);
+      fe->CalcVShape(T, vshape);
+      val.SetSize(spaceDim);
+      vshape.MultTranspose(loc_data, val);
+   }
+}
+
 void GridFunction::GetVectorValues(ElementTransformation &T,
                                    const IntegrationRule &ir,
-                                   DenseMatrix &vals) const
+                                   DenseMatrix &vals,
+                                   DenseMatrix *tr) const
 {
+   if (tr)
+   {
+      T.Transform(ir, *tr);
+   }
+
    const FiniteElement *FElem = fes->GetFE(T.ElementNo);
    int dof = FElem->GetDof();
+
    Array<int> vdofs;
    fes->GetElementVDofs(T.ElementNo, vdofs);
+
    Vector loc_data;
    GetSubVector(vdofs, loc_data);
    int nip = ir.GetNPoints();
+
    if (FElem->GetRangeType() == FiniteElement::SCALAR)
    {
       MFEM_ASSERT(FElem->GetMapType() == FiniteElement::VALUE,
@@ -639,6 +979,7 @@ void GridFunction::GetVectorValues(ElementTransformation &T,
       {
          const IntegrationPoint &ip = ir.IntPoint(j);
          FElem->CalcShape(ip, shape);
+
          for (int k = 0; k < vdim; k++)
          {
             vals(k,j) = shape * ((const double *)loc_data + dof * k);
@@ -649,28 +990,22 @@ void GridFunction::GetVectorValues(ElementTransformation &T,
    {
       int spaceDim = fes->GetMesh()->SpaceDimension();
       DenseMatrix vshape(dof, spaceDim);
+
       vals.SetSize(spaceDim, nip);
       Vector val_j;
+
       for (int j = 0; j < nip; j++)
       {
          const IntegrationPoint &ip = ir.IntPoint(j);
          T.SetIntPoint(&ip);
          FElem->CalcVShape(T, vshape);
+
          vals.GetColumnReference(j, val_j);
          vshape.MultTranspose(loc_data, val_j);
       }
    }
 }
 
-void GridFunction::GetVectorValues(int i, const IntegrationRule &ir,
-                                   DenseMatrix &vals, DenseMatrix &tr) const
-{
-   ElementTransformation *Tr = fes->GetElementTransformation(i);
-   Tr->Transform(ir, tr);
-
-   GetVectorValues(*Tr, ir, vals);
-}
-
 int GridFunction::GetFaceVectorValues(
    int i, int side, const IntegrationRule &ir,
    DenseMatrix &vals, DenseMatrix &tr) const
@@ -700,15 +1035,15 @@ int GridFunction::GetFaceVectorValues(
    }
    if (di == 0)
    {
-      Transf = fes->GetMesh()->GetFaceElementTransformations(i, 4);
+      Transf = fes->GetMesh()->GetFaceElementTransformations(i, 5);
       Transf->Loc1.Transform(ir, eir);
-      GetVectorValues(Transf->Elem1No, eir, vals, tr);
+      GetVectorValues(*Transf->Elem1, eir, vals, &tr);
    }
    else
    {
-      Transf = fes->GetMesh()->GetFaceElementTransformations(i, 8);
+      Transf = fes->GetMesh()->GetFaceElementTransformations(i, 10);
       Transf->Loc2.Transform(ir, eir);
-      GetVectorValues(Transf->Elem2No, eir, vals, tr);
+      GetVectorValues(*Transf->Elem2, eir, vals, &tr);
    }
 
    return di;
@@ -1007,107 +1342,262 @@ void GridFunction::GetVectorGradientHat(
    MultAtB(loc_data_mat, dshape, gh);
 }
 
-double GridFunction::GetDivergence(ElementTransformation &tr) const
+double GridFunction::GetDivergence(ElementTransformation &T) const
 {
-   double div_v;
-   int elNo = tr.ElementNo;
-   const FiniteElement *FElem = fes->GetFE(elNo);
-   if (FElem->GetRangeType() == FiniteElement::SCALAR)
+   switch (T.ElementType)
    {
-      MFEM_ASSERT(FElem->GetMapType() == FiniteElement::VALUE,
-                  "invalid FE map type");
-      DenseMatrix grad_hat;
-      GetVectorGradientHat(tr, grad_hat);
-      const DenseMatrix &Jinv = tr.InverseJacobian();
-      div_v = 0.0;
-      for (int i = 0; i < Jinv.Width(); i++)
+      case ElementTransformation::ELEMENT:
       {
-         for (int j = 0; j < Jinv.Height(); j++)
+         int elNo = T.ElementNo;
+         const FiniteElement *fe = fes->GetFE(elNo);
+         if (fe->GetRangeType() == FiniteElement::SCALAR)
          {
-            div_v += grad_hat(i, j) * Jinv(j, i);
+            MFEM_ASSERT(fe->GetMapType() == FiniteElement::VALUE,
+                        "invalid FE map type");
+            DenseMatrix grad_hat;
+            GetVectorGradientHat(T, grad_hat);
+            const DenseMatrix &Jinv = T.InverseJacobian();
+            double div_v = 0.0;
+            for (int i = 0; i < Jinv.Width(); i++)
+            {
+               for (int j = 0; j < Jinv.Height(); j++)
+               {
+                  div_v += grad_hat(i, j) * Jinv(j, i);
+               }
+            }
+            return div_v;
+         }
+         else
+         {
+            // Assuming RT-type space
+            Array<int> dofs;
+            fes->GetElementDofs(elNo, dofs);
+            Vector loc_data, divshape(fe->GetDof());
+            GetSubVector(dofs, loc_data);
+            fe->CalcDivShape(T.GetIntPoint(), divshape);
+            return (loc_data * divshape) / T.Weight();
          }
       }
+      break;
+      case ElementTransformation::BDR_ELEMENT:
+      {
+         // In order to properly capture the derivative of the normal component
+         // of the field (as well as the transverse divergence of the
+         // tangential compoents) we must evaluate it in the neighboring
+         // element.
+         FaceElementTransformations * FET =
+            fes->GetMesh()->GetBdrFaceTransformations(T.ElementNo);
+
+         // Boundary elements and Boundary Faces may have different
+         // orientations so adjust the integration point if necessary.
+         int o = 0;
+         if (fes->GetMesh()->Dimension() == 3)
+         {
+            int f;
+            fes->GetMesh()->GetBdrElementFace(T.ElementNo, &f, &o);
+         }
+
+         IntegrationPoint fip;
+         be_to_bfe(FET->GetGeometryType(), o, T.GetIntPoint(), fip);
+
+         // Compute and set the point in element 1 from fip
+         FET->SetAllIntPoints(&fip);
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+
+         return GetDivergence(T1);
+      }
+      break;
+      case ElementTransformation::BDR_FACE:
+      {
+         // This must be a DG context so this dynamic cast must succeed.
+         FaceElementTransformations * FET =
+            dynamic_cast<FaceElementTransformations *>(&T);
+
+         // Evaluate in neighboring element (the integration point in T1 should
+         // have already been set).
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+         return GetDivergence(T1);
+      }
+      break;
+      default:
+      {
+         MFEM_ABORT("GridFunction::GetDivergence: Unsupported element type \""
+                    << T.ElementType << "\"");
+      }
    }
-   else
-   {
-      // Assuming RT-type space
-      Array<int> dofs;
-      fes->GetElementDofs(elNo, dofs);
-      Vector loc_data, divshape(FElem->GetDof());
-      GetSubVector(dofs, loc_data);
-      FElem->CalcDivShape(tr.GetIntPoint(), divshape);
-      div_v = (loc_data * divshape) / tr.Weight();
-   }
-   return div_v;
+   return 0.0; // never reached
 }
 
-void GridFunction::GetCurl(ElementTransformation &tr, Vector &curl) const
+void GridFunction::GetCurl(ElementTransformation &T, Vector &curl) const
 {
-   int elNo = tr.ElementNo;
-   const FiniteElement *FElem = fes->GetFE(elNo);
-   if (FElem->GetRangeType() == FiniteElement::SCALAR)
+   switch (T.ElementType)
    {
-      MFEM_ASSERT(FElem->GetMapType() == FiniteElement::VALUE,
-                  "invalid FE map type");
-      DenseMatrix grad_hat;
-      GetVectorGradientHat(tr, grad_hat);
-      const DenseMatrix &Jinv = tr.InverseJacobian();
-      DenseMatrix grad(grad_hat.Height(), Jinv.Width()); // vdim x FElem->Dim
-      Mult(grad_hat, Jinv, grad);
-      MFEM_ASSERT(grad.Height() == grad.Width(), "");
-      if (grad.Height() == 3)
+      case ElementTransformation::ELEMENT:
       {
-         curl.SetSize(3);
-         curl(0) = grad(2,1) - grad(1,2);
-         curl(1) = grad(0,2) - grad(2,0);
-         curl(2) = grad(1,0) - grad(0,1);
+         int elNo = T.ElementNo;
+         const FiniteElement *fe = fes->GetFE(elNo);
+         if (fe->GetRangeType() == FiniteElement::SCALAR)
+         {
+            MFEM_ASSERT(fe->GetMapType() == FiniteElement::VALUE,
+                        "invalid FE map type");
+            DenseMatrix grad_hat;
+            GetVectorGradientHat(T, grad_hat);
+            const DenseMatrix &Jinv = T.InverseJacobian();
+            // Dimensions of grad are vdim x FElem->Dim
+            DenseMatrix grad(grad_hat.Height(), Jinv.Width());
+            Mult(grad_hat, Jinv, grad);
+            MFEM_ASSERT(grad.Height() == grad.Width(), "");
+            if (grad.Height() == 3)
+            {
+               curl.SetSize(3);
+               curl(0) = grad(2,1) - grad(1,2);
+               curl(1) = grad(0,2) - grad(2,0);
+               curl(2) = grad(1,0) - grad(0,1);
+            }
+            else if (grad.Height() == 2)
+            {
+               curl.SetSize(1);
+               curl(0) = grad(1,0) - grad(0,1);
+            }
+         }
+         else
+         {
+            // Assuming ND-type space
+            Array<int> dofs;
+            fes->GetElementDofs(elNo, dofs);
+            Vector loc_data;
+            GetSubVector(dofs, loc_data);
+            DenseMatrix curl_shape(fe->GetDof(), fe->GetDim() == 3 ? 3 : 1);
+            fe->CalcCurlShape(T.GetIntPoint(), curl_shape);
+            curl.SetSize(curl_shape.Width());
+            if (curl_shape.Width() == 3)
+            {
+               double curl_hat[3];
+               curl_shape.MultTranspose(loc_data, curl_hat);
+               T.Jacobian().Mult(curl_hat, curl);
+            }
+            else
+            {
+               curl_shape.MultTranspose(loc_data, curl);
+            }
+            curl /= T.Weight();
+         }
       }
-      else if (grad.Height() == 2)
+      break;
+      case ElementTransformation::BDR_ELEMENT:
       {
-         curl.SetSize(1);
-         curl(0) = grad(1,0) - grad(0,1);
+         // In order to capture the tangential components of the curl we
+         // must evaluate it in the neighboring element.
+         FaceElementTransformations * FET =
+            fes->GetMesh()->GetBdrFaceTransformations(T.ElementNo);
+
+         // Boundary elements and Boundary Faces may have different
+         // orientations so adjust the integration point if necessary.
+         int o = 0;
+         if (fes->GetMesh()->Dimension() == 3)
+         {
+            int f;
+            fes->GetMesh()->GetBdrElementFace(T.ElementNo, &f, &o);
+         }
+
+         IntegrationPoint fip;
+         be_to_bfe(FET->GetGeometryType(), o, T.GetIntPoint(), fip);
+
+         // Compute and set the point in element 1 from fip
+         FET->SetAllIntPoints(&fip);
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+
+         GetCurl(T1, curl);
       }
-   }
-   else
-   {
-      // Assuming ND-type space
-      Array<int> dofs;
-      fes->GetElementDofs(elNo, dofs);
-      Vector loc_data;
-      GetSubVector(dofs, loc_data);
-      DenseMatrix curl_shape(FElem->GetDof(), FElem->GetDim() == 3 ? 3 : 1);
-      FElem->CalcCurlShape(tr.GetIntPoint(), curl_shape);
-      curl.SetSize(curl_shape.Width());
-      if (curl_shape.Width() == 3)
+      break;
+      case ElementTransformation::BDR_FACE:
       {
-         double curl_hat[3];
-         curl_shape.MultTranspose(loc_data, curl_hat);
-         tr.Jacobian().Mult(curl_hat, curl);
+         // This must be a DG context so this dynamic cast must succeed.
+         FaceElementTransformations * FET =
+            dynamic_cast<FaceElementTransformations *>(&T);
+
+         // Evaluate in neighboring element (the integration point in T1 should
+         // have already been set).
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+         GetCurl(T1, curl);
       }
-      else
+      break;
+      default:
       {
-         curl_shape.MultTranspose(loc_data, curl);
+         MFEM_ABORT("GridFunction::GetCurl: Unsupported element type \""
+                    << T.ElementType << "\"");
       }
-      curl /= tr.Weight();
    }
 }
 
-void GridFunction::GetGradient(ElementTransformation &tr, Vector &grad) const
+void GridFunction::GetGradient(ElementTransformation &T, Vector &grad) const
 {
-   int elNo = tr.ElementNo;
-   const FiniteElement *fe = fes->GetFE(elNo);
-   MFEM_ASSERT(fe->GetMapType() == FiniteElement::VALUE, "invalid FE map type");
-   int dim = fe->GetDim(), dof = fe->GetDof();
-   DenseMatrix dshape(dof, dim);
-   Vector lval, gh(dim);
-   Array<int> dofs;
+   switch (T.ElementType)
+   {
+      case ElementTransformation::ELEMENT:
+      {
+         const FiniteElement * fe = fes->GetFE(T.ElementNo);
+         MFEM_ASSERT(fe->GetMapType() == FiniteElement::VALUE,
+                     "invalid FE map type");
+         int spaceDim = fes->GetMesh()->SpaceDimension();
+         int dim = fe->GetDim(), dof = fe->GetDof();
+         DenseMatrix dshape(dof, dim);
+         Vector lval, gh(dim);
+         Array<int> dofs;
 
-   grad.SetSize(dim);
-   fes->GetElementDofs(elNo, dofs);
-   GetSubVector(dofs, lval);
-   fe->CalcDShape(tr.GetIntPoint(), dshape);
-   dshape.MultTranspose(lval, gh);
-   tr.InverseJacobian().MultTranspose(gh, grad);
+         grad.SetSize(spaceDim);
+         fes->GetElementDofs(T.ElementNo, dofs);
+         GetSubVector(dofs, lval);
+         fe->CalcDShape(T.GetIntPoint(), dshape);
+         dshape.MultTranspose(lval, gh);
+         T.InverseJacobian().MultTranspose(gh, grad);
+      }
+      break;
+      case ElementTransformation::BDR_ELEMENT:
+      {
+         // In order to properly capture the normal component of the gradient
+         // as well as its tangential components we must evaluate it in the
+         // neighboring element.
+         FaceElementTransformations * FET =
+            fes->GetMesh()->GetBdrFaceTransformations(T.ElementNo);
+
+         // Boundary elements and Boundary Faces may have different
+         // orientations so adjust the integration point if necessary.
+         int o = 0;
+         if (fes->GetMesh()->Dimension() == 3)
+         {
+            int f;
+            fes->GetMesh()->GetBdrElementFace(T.ElementNo, &f, &o);
+         }
+
+         IntegrationPoint fip;
+         be_to_bfe(FET->GetGeometryType(), o, T.GetIntPoint(), fip);
+
+         // Compute and set the point in element 1 from fip
+         FET->SetAllIntPoints(&fip);
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+
+         GetGradient(T1, grad);
+      }
+      break;
+      case ElementTransformation::BDR_FACE:
+      {
+         // This must be a DG context so this dynamic cast must succeed.
+         FaceElementTransformations * FET =
+            dynamic_cast<FaceElementTransformations *>(&T);
+
+         // Evaluate in neighboring element (the integration point in T1 should
+         // have already been set).
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+         GetGradient(T1, grad);
+      }
+      break;
+      default:
+      {
+         MFEM_ABORT("GridFunction::GetGradient: Unsupported element type \""
+                    << T.ElementType << "\"");
+      }
+   }
 }
 
 void GridFunction::GetGradients(ElementTransformation &tr,
@@ -1136,15 +1626,65 @@ void GridFunction::GetGradients(ElementTransformation &tr,
 }
 
 void GridFunction::GetVectorGradient(
-   ElementTransformation &tr, DenseMatrix &grad) const
+   ElementTransformation &T, DenseMatrix &grad) const
 {
-   MFEM_ASSERT(fes->GetFE(tr.ElementNo)->GetMapType() == FiniteElement::VALUE,
-               "invalid FE map type");
-   DenseMatrix grad_hat;
-   GetVectorGradientHat(tr, grad_hat);
-   const DenseMatrix &Jinv = tr.InverseJacobian();
-   grad.SetSize(grad_hat.Height(), Jinv.Width());
-   Mult(grad_hat, Jinv, grad);
+   switch (T.ElementType)
+   {
+      case ElementTransformation::ELEMENT:
+      {
+         MFEM_ASSERT(fes->GetFE(T.ElementNo)->GetMapType() ==
+                     FiniteElement::VALUE, "invalid FE map type");
+         DenseMatrix grad_hat;
+         GetVectorGradientHat(T, grad_hat);
+         const DenseMatrix &Jinv = T.InverseJacobian();
+         grad.SetSize(grad_hat.Height(), Jinv.Width());
+         Mult(grad_hat, Jinv, grad);
+      }
+      break;
+      case ElementTransformation::BDR_ELEMENT:
+      {
+         // In order to capture the normal component of the gradient we
+         // must evaluate it in the neighboring element.
+         FaceElementTransformations * FET =
+            fes->GetMesh()->GetBdrFaceTransformations(T.ElementNo);
+
+         // Boundary elements and Boundary Faces may have different
+         // orientations so adjust the integration point if necessary.
+         int o = 0;
+         if (fes->GetMesh()->Dimension() == 3)
+         {
+            int f;
+            fes->GetMesh()->GetBdrElementFace(T.ElementNo, &f, &o);
+         }
+
+         IntegrationPoint fip;
+         be_to_bfe(FET->GetGeometryType(), o, T.GetIntPoint(), fip);
+
+         // Compute and set the point in element 1 from fip
+         FET->SetAllIntPoints(&fip);
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+
+         GetVectorGradient(T1, grad);
+      }
+      break;
+      case ElementTransformation::BDR_FACE:
+      {
+         // This must be a DG context so this dynamic cast must succeed.
+         FaceElementTransformations * FET =
+            dynamic_cast<FaceElementTransformations *>(&T);
+
+         // Evaluate in neighboring element (the integration point in T1 should
+         // have already been set).
+         ElementTransformation & T1 = FET->GetElement1Transformation();
+         GetVectorGradient(T1, grad);
+      }
+      break;
+      default:
+      {
+         MFEM_ABORT("GridFunction::GetVectorGradient: "
+                    "Unsupported element type \"" << T.ElementType << "\"");
+      }
+   }
 }
 
 void GridFunction::GetElementAverages(GridFunction &avgs) const
@@ -1716,6 +2256,8 @@ void GridFunction::ProjectCoefficient(
    ElementTransformation *T = NULL;
    const FiniteElement *fe = NULL;
 
+   fes->BuildDofToArrays(); // ensures GetElementForDof(), GetLocalDofForDof() initialized.
+
    for (int i = 0; i < dofs.Size(); i++)
    {
       int dof = dofs[i], j = fes->GetElementForDof(dof);
@@ -1757,6 +2299,8 @@ void GridFunction::ProjectCoefficient(
 
    Vector val;
 
+   fes->BuildDofToArrays(); // ensures GetElementForDof(), GetLocalDofForDof() initialized.
+
    for (int i = 0; i < dofs.Size(); i++)
    {
       int dof = dofs[i], j = fes->GetElementForDof(dof);
@@ -2009,7 +2553,7 @@ double GridFunction::ComputeL2Error(
       fdof = fe->GetDof();
       transf = fes->GetElementTransformation(i);
       shape.SetSize(fdof);
-      intorder = 2*fe->GetOrder() + 1; // <----------
+      intorder = 2*fe->GetOrder() + 3; // <----------
       const IntegrationRule *ir;
       if (irs)
       {
@@ -2064,7 +2608,7 @@ double GridFunction::ComputeL2Error(
    {
       if (elems != NULL && (*elems)[i] == 0) { continue; }
       fe = fes->GetFE(i);
-      int intorder = 2*fe->GetOrder() + 1; // <----------
+      int intorder = 2*fe->GetOrder() + 3; // <----------
       const IntegrationRule *ir;
       if (irs)
       {
@@ -2168,7 +2712,7 @@ double GridFunction::ComputeH1Error(
             }
          intorder = 2 * intorder;  // <-------------
          const IntegrationRule &ir =
-            IntRules.Get(face_elem_transf->FaceGeom, intorder);
+            IntRules.Get(face_elem_transf->GetGeometryType(), intorder);
          err_val.SetSize(ir.GetNPoints());
          ell_coeff_val.SetSize(ir.GetNPoints());
          // side 1
@@ -2225,7 +2769,7 @@ double GridFunction::ComputeH1Error(
             }
          }
          face_elem_transf = mesh->GetFaceElementTransformations(i, 16);
-         transf = face_elem_transf->Face;
+         transf = face_elem_transf;
          for (j = 0; j < ir.GetNPoints(); j++)
          {
             const IntegrationPoint &ip = ir.IntPoint(j);
@@ -2259,7 +2803,7 @@ double GridFunction::ComputeMaxError(
       fdof = fe->GetDof();
       transf = fes->GetElementTransformation(i);
       shape.SetSize(fdof);
-      intorder = 2*fe->GetOrder() + 1; // <----------
+      intorder = 2*fe->GetOrder() + 3; // <----------
       const IntegrationRule *ir;
       if (irs)
       {
@@ -2425,7 +2969,7 @@ double GridFunction::ComputeLpError(const double p, Coefficient &exsol,
       }
       else
       {
-         int intorder = 2*fe->GetOrder() + 1; // <----------
+         int intorder = 2*fe->GetOrder() + 3; // <----------
          ir = &(IntRules.Get(fe->GetGeomType(), intorder));
       }
       GetValues(i, *ir, vals);
@@ -2472,10 +3016,13 @@ double GridFunction::ComputeLpError(const double p, Coefficient &exsol,
 }
 
 void GridFunction::ComputeElementLpErrors(const double p, Coefficient &exsol,
-                                          GridFunction &error,
+                                          Vector &error,
                                           Coefficient *weight,
                                           const IntegrationRule *irs[]) const
 {
+   MFEM_ASSERT(error.Size() == fes->GetNE(),
+               "Incorrect size for result vector");
+
    error = 0.0;
    const FiniteElement *fe;
    ElementTransformation *T;
@@ -2491,7 +3038,7 @@ void GridFunction::ComputeElementLpErrors(const double p, Coefficient &exsol,
       }
       else
       {
-         int intorder = 2*fe->GetOrder() + 1; // <----------
+         int intorder = 2*fe->GetOrder() + 3; // <----------
          ir = &(IntRules.Get(fe->GetGeomType(), intorder));
       }
       GetValues(i, *ir, vals);
@@ -2555,7 +3102,7 @@ double GridFunction::ComputeLpError(const double p, VectorCoefficient &exsol,
       }
       else
       {
-         int intorder = 2*fe->GetOrder() + 1; // <----------
+         int intorder = 2*fe->GetOrder() + 3; // <----------
          ir = &(IntRules.Get(fe->GetGeomType(), intorder));
       }
       T = fes->GetElementTransformation(i);
@@ -2627,11 +3174,14 @@ double GridFunction::ComputeLpError(const double p, VectorCoefficient &exsol,
 
 void GridFunction::ComputeElementLpErrors(const double p,
                                           VectorCoefficient &exsol,
-                                          GridFunction &error,
+                                          Vector &error,
                                           Coefficient *weight,
                                           VectorCoefficient *v_weight,
                                           const IntegrationRule *irs[]) const
 {
+   MFEM_ASSERT(error.Size() == fes->GetNE(),
+               "Incorrect size for result vector");
+
    error = 0.0;
    const FiniteElement *fe;
    ElementTransformation *T;
@@ -2648,7 +3198,7 @@ void GridFunction::ComputeElementLpErrors(const double p,
       }
       else
       {
-         int intorder = 2*fe->GetOrder() + 1; // <----------
+         int intorder = 2*fe->GetOrder() + 3; // <----------
          ir = &(IntRules.Get(fe->GetGeomType(), intorder));
       }
       T = fes->GetElementTransformation(i);
@@ -2658,15 +3208,15 @@ void GridFunction::ComputeElementLpErrors(const double p,
       loc_errs.SetSize(vals.Width());
       if (!v_weight)
       {
-         // compute the lengths of the errors at the integration points
-         // thus the vector norm is rotationally invariant
+         // compute the lengths of the errors at the integration points thus the
+         // vector norm is rotationally invariant
          vals.Norm2(loc_errs);
       }
       else
       {
          v_weight->Eval(exact_vals, *T, *ir);
-         // column-wise dot product of the vector error (in vals) and the
-         // vector weight (in exact_vals)
+         // column-wise dot product of the vector error (in vals) and the vector
+         // weight (in exact_vals)
          for (int j = 0; j < vals.Width(); j++)
          {
             double err = 0.0;
@@ -2798,7 +3348,9 @@ void GridFunction::SaveVTK(std::ostream &out, const std::string &field_name,
          RefG = GlobGeometryRefiner.Refine(
                    mesh->GetElementBaseGeometry(i), ref, 1);
 
-         GetVectorValues(i, RefG->RefPts, vval, pmat);
+         // GetVectorValues(i, RefG->RefPts, vval, pmat);
+         ElementTransformation * T = mesh->GetElementTransformation(i);
+         GetVectorValues(*T, RefG->RefPts, vval, &pmat);
 
          for (int j = 0; j < vval.Width(); j++)
          {

fem/gridfunc.hpp

@@ -144,17 +144,134 @@ public:
    /// Returns the values in the vertices of i'th element for dimension vdim.
    void GetNodalValues(int i, Array<double> &nval, int vdim = 1) const;
 
+   /** @name Element index Get Value Methods
+
+       These methods take an element index and return the interpolated value of
+       the field at a given reference point within the element.
+
+       @warning These methods retrieve and use the ElementTransformation object
+       from the mfem::Mesh. This can alter the state of the element
+       transformation object and can also lead to unexpected results when the
+       ElementTransformation object is already in use such as when these methods
+       are called from within an integration loop. Consider using
+       GetValue(ElementTransformation &T, ...) instead.
+   */
+   ///@{
+   /** Return a scalar value from within the given element. */
    virtual double GetValue(int i, const IntegrationPoint &ip,
                            int vdim = 1) const;
 
+   /** Return a vector value from within the given element. */
    void GetVectorValue(int i, const IntegrationPoint &ip, Vector &val) const;
+   ///@}
 
+   /** @name Element Index Get Values Methods
+
+       These are convenience methods for repeatedly calling GetValue for
+       multiple points within a given element. The GetValues methods are
+       optimized and should perform better than repeatedly calling GetValue. The
+       GetVectorValues method simply calls GetVectorValue repeatedly.
+
+       @warning These methods retrieve and use the ElementTransformation object
+       from the mfem::Mesh. This can alter the state of the element
+       transformation object and can also lead to unexpected results when the
+       ElementTransformation object is already in use such as when these methods
+       are called from within an integration loop. Consider using
+       GetValues(ElementTransformation &T, ...) instead.
+   */
+   ///@{
+   /** Compute a collection of scalar values from within the element indicated
+       by the index i. */
    void GetValues(int i, const IntegrationRule &ir, Vector &vals,
                   int vdim = 1) const;
 
+   /** Compute a collection of vector values from within the element indicated
+       by the index i. */
    void GetValues(int i, const IntegrationRule &ir, Vector &vals,
                   DenseMatrix &tr, int vdim = 1) const;
 
+   void GetVectorValues(int i, const IntegrationRule &ir,
+                        DenseMatrix &vals, DenseMatrix &tr) const;
+   ///@}
+
+   /** @name ElementTransformation Get Value Methods
+
+       These member functions are designed for use within
+       GridFunctionCoefficient objects. These can be used with
+       ElementTransformation objects coming from either
+       Mesh::GetElementTransformation() or Mesh::GetBdrElementTransformation().
+
+       @note These methods do not reset the ElementTransformation object so they
+       should be safe to use within integration loops or other contexts where
+       the ElementTransformation is already in use.
+   */
+   ///@{
+   /** Return a scalar value from within the element indicated by the
+       ElementTransformation Object. */
+   virtual double GetValue(ElementTransformation &T, const IntegrationPoint &ip,
+                           int comp = 0, Vector *tr = NULL) const;
+
+   /** Return a vector value from within the element indicated by the
+       ElementTransformation Object. */
+   virtual void GetVectorValue(ElementTransformation &T,
+                               const IntegrationPoint &ip,
+                               Vector &val, Vector *tr = NULL) const;
+   ///@}
+
+   /** @name ElementTransformation Get Values Methods
+
+       These are convenience methods for repeatedly calling GetValue for
+       multiple points within a given element. They work by calling either the
+       ElementTransformation or FaceElementTransformations versions described
+       above. Consequently, these methods should not be expected to run faster
+       than calling the above methods in an external loop.
+
+       @note These methods do not reset the ElementTransformation object so they
+       should be safe to use within integration loops or other contexts where
+       the ElementTransformation is already in use.
+
+       @note These methods can also be used with FaceElementTransformations
+       objects.
+    */
+   ///@{
+   /** Compute a collection of scalar values from within the element indicated
+       by the ElementTransformation object. */
+   void GetValues(ElementTransformation &T, const IntegrationRule &ir,
+                  Vector &vals, int comp = 0, DenseMatrix *tr = NULL) const;
+
+   /** Compute a collection of vector values from within the element indicated
+       by the ElementTransformation object. */
+   void GetVectorValues(ElementTransformation &T, const IntegrationRule &ir,
+                        DenseMatrix &vals, DenseMatrix *tr = NULL) const;
+   ///@}
+
+   /** @name Face Index Get Values Methods
+
+       These methods are designed to work with Discontinuous Galerkin basis
+       functions. They compute field values on the interface between elements,
+       or on boundary elements, by interpolating the field in a neighboring
+       element. The \a side argument indices which neighboring element should be
+       used: 0, 1, or 2 (automatically chosen).
+
+       @warning These methods retrieve and use the FaceElementTransformations
+       object from the mfem::Mesh. This can alter the state of the face element
+       transformations object and can also lead to unexpected results when the
+       FaceElementTransformations object is already in use such as when these
+       methods are called from within an integration loop. Consider using
+       GetValues(ElementTransformation &T, ...) instead.
+    */
+   ///@{
+   /** Compute a collection of scalar values from within the face
+       indicated by the index i. */
+   int GetFaceValues(int i, int side, const IntegrationRule &ir, Vector &vals,
+                     DenseMatrix &tr, int vdim = 1) const;
+
+   /** Compute a collection of vector values from within the face
+       indicated by the index i. */
+   int GetFaceVectorValues(int i, int side, const IntegrationRule &ir,
+                           DenseMatrix &vals, DenseMatrix &tr) const;
+   ///@}
+
    void GetLaplacians(int i, const IntegrationRule &ir, Vector &laps,
                       int vdim = 1) const;
 
@@ -167,18 +284,6 @@ public:
    void GetHessians(int i, const IntegrationRule &ir, DenseMatrix &hess,
                     DenseMatrix &tr, int vdim = 1) const;
 
-   int GetFaceValues(int i, int side, const IntegrationRule &ir, Vector &vals,
-                     DenseMatrix &tr, int vdim = 1) const;
-
-   void GetVectorValues(ElementTransformation &T, const IntegrationRule &ir,
-                        DenseMatrix &vals) const;
-
-   void GetVectorValues(int i, const IntegrationRule &ir,
-                        DenseMatrix &vals, DenseMatrix &tr) const;
-
-   int GetFaceVectorValues(int i, int side, const IntegrationRule &ir,
-                           DenseMatrix &vals, DenseMatrix &tr) const;
-
    void GetValuesFrom(const GridFunction &orig_func);
 
    void GetBdrValuesFrom(const GridFunction &orig_func);
@@ -236,12 +341,10 @@ public:
 
    virtual void ProjectCoefficient(Coefficient &coeff);
 
-   // call fes -> BuildDofToArrays() before using this projection
    void ProjectCoefficient(Coefficient &coeff, Array<int> &dofs, int vd = 0);
 
    void ProjectCoefficient(VectorCoefficient &vcoeff);
 
-   // call fes -> BuildDofToArrays() before using this projection
    void ProjectCoefficient(VectorCoefficient &vcoeff, Array<int> &dofs);
 
    void ProjectCoefficient(Coefficient *coeff[]);
@@ -365,28 +468,28 @@ public:
                                  const IntegrationRule *irs[] = NULL) const;
 
    /** Compute the Lp error in each element of the mesh and store the results in
-       the GridFunction @a error. The result should be an L2 GridFunction of
-       order zero using map type VALUE. */
+       the Vector @a error. The result should be of length number of elements,
+       for example an L2 GridFunction of order zero using map type VALUE. */
    virtual void ComputeElementLpErrors(const double p, Coefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        Coefficient *weight = NULL,
                                        const IntegrationRule *irs[] = NULL
                                       ) const;
 
    virtual void ComputeElementL1Errors(Coefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        const IntegrationRule *irs[] = NULL
                                       ) const
    { ComputeElementLpErrors(1.0, exsol, error, NULL, irs); }
 
    virtual void ComputeElementL2Errors(Coefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        const IntegrationRule *irs[] = NULL
                                       ) const
    { ComputeElementLpErrors(2.0, exsol, error, NULL, irs); }
 
    virtual void ComputeElementMaxErrors(Coefficient &exsol,
-                                        GridFunction &error,
+                                        Vector &error,
                                         const IntegrationRule *irs[] = NULL
                                        ) const
    { ComputeElementLpErrors(infinity(), exsol, error, NULL, irs); }
@@ -400,29 +503,29 @@ public:
                                  const IntegrationRule *irs[] = NULL) const;
 
    /** Compute the Lp error in each element of the mesh and store the results in
-       the GridFunction @ error. The result should be an L2 GridFunction of
-       order zero using map type VALUE. */
+       the Vector @ error. The result should be of length number of elements,
+       for example an L2 GridFunction of order zero using map type VALUE. */
    virtual void ComputeElementLpErrors(const double p, VectorCoefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        Coefficient *weight = NULL,
                                        VectorCoefficient *v_weight = NULL,
                                        const IntegrationRule *irs[] = NULL
                                       ) const;
 
    virtual void ComputeElementL1Errors(VectorCoefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        const IntegrationRule *irs[] = NULL
                                       ) const
    { ComputeElementLpErrors(1.0, exsol, error, NULL, NULL, irs); }
 
    virtual void ComputeElementL2Errors(VectorCoefficient &exsol,
-                                       GridFunction &error,
+                                       Vector &error,
                                        const IntegrationRule *irs[] = NULL
                                       ) const
    { ComputeElementLpErrors(2.0, exsol, error, NULL, NULL, irs); }
 
    virtual void ComputeElementMaxErrors(VectorCoefficient &exsol,
-                                        GridFunction &error,
+                                        Vector &error,
                                         const IntegrationRule *irs[] = NULL
                                        ) const
    { ComputeElementLpErrors(infinity(), exsol, error, NULL, NULL, irs); }
@@ -496,11 +599,13 @@ public:
                      type = adios2stream::data_type::point_data) const;
 #endif
 
-   /** Write the GridFunction in VTK format. Note that Mesh::PrintVTK must be
-       called first. The parameter ref > 0 must match the one used in
+   /** @brief Write the GridFunction in VTK format. Note that Mesh::PrintVTK
+       must be called first. The parameter ref > 0 must match the one used in
        Mesh::PrintVTK. */
    void SaveVTK(std::ostream &out, const std::string &field_name, int ref);
 
+   /** @brief Write the GridFunction in STL format. Note that the mesh dimension
+       must be 2 and that quad elements will be broken into two triangles.*/
    void SaveSTL(std::ostream &out, int TimesToRefine = 1);
 
    /// Destroys grid function.
@@ -633,6 +738,16 @@ public:
     */
    inline void GetElementValues(int idx, Vector &values) const;
 
+   /// Return the quadrature function values at an integration point.
+   /** The result is stored in the Vector @a values as a reference to the
+       global values. */
+   inline void GetElementValues(int idx, const int ip_num, Vector &values);
+
+   /// Return the quadrature function values at an integration point.
+   /** The result is stored in the Vector @a values as a copy to the
+       global values. */
+   inline void GetElementValues(int idx, const int ip_num, Vector &values) const;
+
    /// Return all values associated with mesh element @a idx in a DenseMatrix.
    /** The result is stored in the DenseMatrix @a values as a reference to the
        global values.
@@ -737,6 +852,25 @@ inline void QuadratureFunction::GetElementValues(int idx, Vector &values) const
    }
 }
 
+inline void QuadratureFunction::GetElementValues(int idx, const int ip_num,
+                                                 Vector &values)
+{
+   const int s_offset = qspace->element_offsets[idx] * vdim + ip_num * vdim;
+   values.NewDataAndSize(data + s_offset, vdim);
+}
+
+inline void QuadratureFunction::GetElementValues(int idx, const int ip_num,
+                                                 Vector &values) const
+{
+   const int s_offset = qspace->element_offsets[idx] * vdim + ip_num * vdim;
+   values.SetSize(vdim);
+   const double *q = data + s_offset;
+   for (int i = 0; i < values.Size(); i++)
+   {
+      values(i) = *(q++);
+   }
+}
+
 inline void QuadratureFunction::GetElementValues(int idx, DenseMatrix &values)
 {
    const int s_offset = qspace->element_offsets[idx];

fem/gslib.cpp

@@ -29,12 +29,14 @@ namespace mfem
 {
 
 FindPointsGSLIB::FindPointsGSLIB()
-   : mesh(NULL), ir_simplex(NULL), gsl_mesh(), fdata2D(NULL), fdata3D(NULL),
-     dim(-1)
+   : mesh(NULL), ir_simplex(NULL), fdata2D(NULL), fdata3D(NULL),
+     dim(-1), gsl_mesh(), gsl_ref(), gsl_dist(), setupflag(false)
 {
    gsl_comm = new comm;
 #ifdef MFEM_USE_MPI
-   MPI_Init(NULL, NULL);
+   int initialized;
+   MPI_Initialized(&initialized);
+   if (!initialized) { MPI_Init(NULL, NULL); }
    MPI_Comm comm = MPI_COMM_WORLD;;
    comm_init(gsl_comm, comm);
 #else
@@ -50,28 +52,29 @@ FindPointsGSLIB::~FindPointsGSLIB()
 
 #ifdef MFEM_USE_MPI
 FindPointsGSLIB::FindPointsGSLIB(MPI_Comm _comm)
-   : mesh(NULL), ir_simplex(NULL), gsl_mesh(), fdata2D(NULL), fdata3D(NULL),
-     dim(-1)
+   : mesh(NULL), ir_simplex(NULL), fdata2D(NULL), fdata3D(NULL),
+     dim(-1), gsl_mesh(), gsl_ref(), gsl_dist(), setupflag(false)
 {
    gsl_comm = new comm;
    comm_init(gsl_comm, _comm);
 }
 #endif
 
-void FindPointsGSLIB::Setup(Mesh &m, double bb_t, double newt_tol, int npt_max)
+void FindPointsGSLIB::Setup(Mesh &m, const double bb_t, const double newt_tol,
+                            const int npt_max)
 {
    MFEM_VERIFY(m.GetNodes() != NULL, "Mesh nodes are required.");
    MFEM_VERIFY(m.GetNumGeometries(m.Dimension()) == 1,
                "Mixed meshes are not currently supported in FindPointsGSLIB.");
 
+   // call FreeData if FindPointsGSLIB::Setup has been called already
+   if (setupflag) { FreeData(); }
+
    mesh = &m;
    dim  = mesh->Dimension();
    const FiniteElement *fe = mesh->GetNodalFESpace()->GetFE(0);
    unsigned dof1D = fe->GetOrder() + 1;
-   int NE      = mesh->GetNE(),
-       dof_cnt = fe->GetDof(),
-       pts_cnt = NE * dof_cnt,
-       gt      = fe->GetGeomType();
+   const int gt   = fe->GetGeomType();
 
    if (gt == Geometry::TRIANGLE || gt == Geometry::TETRAHEDRON ||
        gt == Geometry::PRISM)
@@ -87,8 +90,8 @@ void FindPointsGSLIB::Setup(Mesh &m, double bb_t, double newt_tol, int npt_max)
       MFEM_ABORT("Element type not currently supported in FindPointsGSLIB.");
    }
 
-   pts_cnt = gsl_mesh.Size()/dim;
-   int NEtot = pts_cnt/(int)pow(dof1D, dim);
+   const int pts_cnt = gsl_mesh.Size()/dim,
+             NEtot = pts_cnt/(int)pow(dof1D, dim);
 
    if (dim == 2)
    {
@@ -107,6 +110,7 @@ void FindPointsGSLIB::Setup(Mesh &m, double bb_t, double newt_tol, int npt_max)
       fdata3D = findpts_setup_3(gsl_comm, elx, nr, NEtot, mr, bb_t,
                                 pts_cnt, pts_cnt, npt_max, newt_tol);
    }
+   setupflag = true;
 }
 
 void FindPointsGSLIB::FindPoints(const Vector &point_pos,
@@ -115,6 +119,7 @@ void FindPointsGSLIB::FindPoints(const Vector &point_pos,
                                  Array<unsigned int> &elem_ids,
                                  Vector &ref_pos, Vector &dist)
 {
+   MFEM_VERIFY(setupflag, "Use FindPointsGSLIB::Setup before finding points.");
    const int points_cnt = point_pos.Size() / dim;
    if (dim == 2)
    {
@@ -150,36 +155,93 @@ void FindPointsGSLIB::FindPoints(const Vector &point_pos,
    }
 }
 
+void FindPointsGSLIB::FindPoints(const Vector &point_pos)
+{
+   const int points_cnt = point_pos.Size() / dim;
+   gsl_code.SetSize(points_cnt);
+   gsl_proc.SetSize(points_cnt);
+   gsl_elem.SetSize(points_cnt);
+   gsl_ref.SetSize(points_cnt * dim);
+   gsl_dist.SetSize(points_cnt);
+
+   FindPoints(point_pos, gsl_code, gsl_proc, gsl_elem, gsl_ref, gsl_dist);
+}
+
+void FindPointsGSLIB::FindPoints(Mesh &m, const Vector &point_pos,
+                                 const double bb_t, const double newt_tol,
+                                 const int npt_max)
+{
+   if (!setupflag || (mesh != &m) )
+   {
+      Setup(m, bb_t, newt_tol, npt_max);
+   }
+   FindPoints(point_pos);
+}
+
 void FindPointsGSLIB::Interpolate(Array<unsigned int> &codes,
                                   Array<unsigned int> &proc_ids,
                                   Array<unsigned int> &elem_ids,
                                   Vector &ref_pos, const GridFunction &field_in,
                                   Vector &field_out)
 {
-   Vector node_vals;
-   GetNodeValues(field_in, node_vals);
 
-   const int points_cnt = ref_pos.Size() / dim;
-   if (dim==2)
+   FiniteElementSpace ind_fes(mesh, field_in.FESpace()->FEColl());
+   GridFunction field_in_scalar(&ind_fes);
+   Vector node_vals;
+
+   const int ncomp      = field_in.FESpace()->GetVDim(),
+             points_fld = field_in.Size() / ncomp,
+             points_cnt = codes.Size();
+   field_out.SetSize(points_cnt*ncomp);
+
+   for (int i = 0; i < ncomp; i++)
    {
-      findpts_eval_2(field_out.GetData(), sizeof(double),
-                     codes.GetData(), sizeof(unsigned int),
-                     proc_ids.GetData(), sizeof(unsigned int),
-                     elem_ids.GetData(), sizeof(unsigned int),
-                     ref_pos.GetData(), sizeof(double) * dim,
-                     points_cnt, node_vals.GetData(), fdata2D);
-   }
-   else
-   {
-      findpts_eval_3(field_out.GetData(), sizeof(double),
-                     codes.GetData(), sizeof(unsigned int),
-                     proc_ids.GetData(), sizeof(unsigned int),
-                     elem_ids.GetData(), sizeof(unsigned int),
-                     ref_pos.GetData(), sizeof(double) * dim,
-                     points_cnt, node_vals.GetData(), fdata3D);
+      const int dataptrin  = i*points_fld,
+                dataptrout = i*points_cnt;
+      field_in_scalar.NewDataAndSize(field_in.GetData()+dataptrin, points_fld);
+      GetNodeValues(field_in_scalar, node_vals);
+
+      if (dim==2)
+      {
+         findpts_eval_2(field_out.GetData()+dataptrout, sizeof(double),
+                        codes.GetData(),       sizeof(unsigned int),
+                        proc_ids.GetData(),    sizeof(unsigned int),
+                        elem_ids.GetData(),    sizeof(unsigned int),
+                        ref_pos.GetData(),     sizeof(double) * dim,
+                        points_cnt, node_vals.GetData(), fdata2D);
+      }
+      else
+      {
+         findpts_eval_3(field_out.GetData()+dataptrout, sizeof(double),
+                        codes.GetData(),       sizeof(unsigned int),
+                        proc_ids.GetData(),    sizeof(unsigned int),
+                        elem_ids.GetData(),    sizeof(unsigned int),
+                        ref_pos.GetData(),     sizeof(double) * dim,
+                        points_cnt, node_vals.GetData(), fdata3D);
+      }
    }
 }
 
+void FindPointsGSLIB::Interpolate(const GridFunction &field_in,
+                                  Vector &field_out)
+{
+   Interpolate(gsl_code, gsl_proc, gsl_elem, gsl_ref, field_in, field_out);
+}
+
+void FindPointsGSLIB::Interpolate(const Vector &point_pos,
+                                  const GridFunction &field_in, Vector &field_out)
+{
+   FindPoints(point_pos);
+   Interpolate(gsl_code, gsl_proc, gsl_elem, gsl_ref, field_in, field_out);
+}
+
+void FindPointsGSLIB::Interpolate(Mesh &m, const Vector &point_pos,
+                                  const GridFunction &field_in, Vector &field_out)
+{
+   FindPoints(m, point_pos);
+   Interpolate(gsl_code, gsl_proc, gsl_elem, gsl_ref, field_in, field_out);
+}
+
 void FindPointsGSLIB::FreeData()
 {
    if (dim == 2)
@@ -190,7 +252,13 @@ void FindPointsGSLIB::FreeData()
    {
       findpts_free_3(fdata3D);
    }
+   setupflag = false;
+   gsl_code.DeleteAll();
+   gsl_proc.DeleteAll();
+   gsl_elem.DeleteAll();
    gsl_mesh.Destroy();
+   gsl_ref.Destroy();
+   gsl_dist.Destroy();
 }
 
 void FindPointsGSLIB::GetNodeValues(const GridFunction &gf_in,
@@ -292,7 +360,7 @@ void FindPointsGSLIB::GetSimplexNodalCoordinates()
    const GridFunction *nodes = mesh->GetNodes();
    Mesh *meshsplit           = NULL;
    const int NE              = mesh->GetNE();
-   int NEsplit;
+   int NEsplit = -1;
 
    // Split the reference element into a reference submesh of quads or hexes.
    if (gt == Geometry::TRIANGLE)
@@ -386,6 +454,7 @@ void FindPointsGSLIB::GetSimplexNodalCoordinates()
       }
       meshsplit->FinalizeHexMesh(1, 1, true);
    }
+   else { MFEM_ABORT("Unsupported geometry type."); }
 
    // Curve the reference submesh.
    H1_FECollection fec(fe->GetOrder(), dim);

fem/gslib.hpp

@@ -29,10 +29,12 @@ class FindPointsGSLIB
 protected:
    Mesh *mesh;
    IntegrationRule *ir_simplex;
-   Vector gsl_mesh;
    struct findpts_data_2 *fdata2D;
    struct findpts_data_3 *fdata3D;
    int dim;
+   Array<unsigned int> gsl_code, gsl_proc, gsl_elem;
+   Vector gsl_mesh, gsl_ref, gsl_dist;
+   bool setupflag;
 
    struct comm *gsl_comm;
 
@@ -59,7 +61,8 @@ public:
        @param[in] newt_tol  Newton tolerance for the gslib search methods.
        @param[in] npt_max   Number of points for simultaneous iteration. This
                             alters performance and memory footprint. */
-   void Setup(Mesh &m, double bb_t, double newt_tol, int npt_max);
+   void Setup(Mesh &m, const double bb_t = 0.1, const double newt_tol = 1.0e-12,
+              const int npt_max = 256);
 
    /** Searches positions given in physical space by @a point_pos. All output
        Arrays and Vectors are expected to have the correct size.
@@ -73,11 +76,15 @@ public:
        @param[out] ref_pos    Reference coordinates of the found point. Ordered
                               by vdim (XYZ,XYZ,XYZ...).
                               Note: the gslib reference frame is [-1,1].
-       @param[out] dist       Distance between the seeked and the found point
+       @param[out] dist       Distance between the sought and the found point
                               in physical space. */
    void FindPoints(const Vector &point_pos, Array<unsigned int> &codes,
                    Array<unsigned int> &proc_ids, Array<unsigned int> &elem_ids,
                    Vector &ref_pos, Vector &dist);
+   void FindPoints(const Vector &point_pos);
+   /// Setup FindPoints and search positions
+   void FindPoints(Mesh &m, const Vector &point_pos, const double bb_t = 0.1,
+                   const double newt_tol = 1.0e-12,  const int npt_max = 256);
 
    /** Interpolation of field values at prescribed reference space positions.
 
@@ -96,11 +103,31 @@ public:
    void Interpolate(Array<unsigned int> &codes, Array<unsigned int> &proc_ids,
                     Array<unsigned int> &elem_ids, Vector &ref_pos,
                     const GridFunction &field_in, Vector &field_out);
+   void Interpolate(const GridFunction &field_in, Vector &field_out);
+   /** Search positions and interpolate */
+   void Interpolate(const Vector &point_pos, const GridFunction &field_in,
+                    Vector &field_out);
+   /** Setup FindPoints, search positions and interpolate */
+   void Interpolate(Mesh &m, const Vector &point_pos,
+                    const GridFunction &field_in, Vector &field_out);
 
    /** Cleans up memory allocated internally by gslib.
        Note that in parallel, this must be called before MPI_Finalize(), as
        it calls MPI_Comm_free() for internal gslib communicators. */
    void FreeData();
+
+   /// Return code for each point searched by FindPoints: inside element (0), on
+   /// element boundary (1), or not found (2).
+   const Array<unsigned int> &GetCode() const { return gsl_code; }
+   /// Return element number for each point found by FindPoints.
+   const Array<unsigned int> &GetElem() const { return gsl_elem; }
+   /// Return MPI rank on which each point was found by FindPoints.
+   const Array<unsigned int> &GetProc() const { return gsl_proc; }
+   /// Return reference coordinates for each point found by FindPoints.
+   const Vector &GetReferencePosition() const { return gsl_ref;  }
+   /// Return distance Distance between the sought and the found point
+   /// in physical space, for each point found by FindPoints.
+   const Vector &GetDist()              const { return gsl_dist; }
 };
 
 } // namespace mfem

fem/libceed/ceed.hpp

@@ -40,7 +40,7 @@ struct CeedConstCoeff
 
 struct CeedGridCoeff
 {
-   GridFunction* coeff;
+   const GridFunction* coeff;
    CeedBasis basis;
    CeedElemRestriction restr;
    CeedVector coeffVector;

fem/linearform.hpp

@@ -19,7 +19,7 @@
 namespace mfem
 {
 
-/// Class for linear form - Vector with associated FE space and LFIntegrators.
+/// Vector with associated FE space and LinearFormIntegrators.
 class LinearForm : public Vector
 {
 protected:

fem/lininteg.cpp

@@ -63,6 +63,53 @@ void DomainLFIntegrator::AssembleDeltaElementVect(
    elvect *= delta->EvalDelta(Trans, Trans.GetIntPoint());
 }
 
+void DomainLFGradIntegrator::AssembleRHSElementVect(
+   const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
+{
+   int dof = el.GetDof();
+   int spaceDim = Tr.GetSpaceDim();
+
+   dshape.SetSize(dof, spaceDim);
+
+   elvect.SetSize(dof);
+   elvect = 0.0;
+
+   const IntegrationRule *ir = IntRule;
+   if (ir == NULL)
+   {
+      int intorder = 2 * el.GetOrder();
+      ir = &IntRules.Get(el.GetGeomType(), intorder);
+   }
+
+   for (int i = 0; i < ir->GetNPoints(); i++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(i);
+
+      Tr.SetIntPoint(&ip);
+      el.CalcPhysDShape(Tr, dshape);
+
+      Q.Eval(Qvec, Tr, ip);
+      Qvec *= ip.weight * Tr.Weight();
+
+      dshape.AddMult(Qvec, elvect);
+   }
+}
+
+void DomainLFGradIntegrator::AssembleDeltaElementVect(
+   const FiniteElement &fe, ElementTransformation &Trans, Vector &elvect)
+{
+   MFEM_ASSERT(vec_delta != NULL,"coefficient must be VectorDeltaCoefficient");
+   int dof = fe.GetDof();
+   int spaceDim = Trans.GetSpaceDim();
+
+   dshape.SetSize(dof, spaceDim);
+   fe.CalcPhysDShape(Trans, dshape);
+
+   vec_delta->EvalDelta(Qvec, Trans, Trans.GetIntPoint());
+
+   elvect.SetSize(dof);
+   dshape.Mult(Qvec, elvect);
+}
 
 void BoundaryLFIntegrator::AssembleRHSElementVect(
    const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
@@ -112,10 +159,13 @@ void BoundaryLFIntegrator::AssembleRHSElementVect(
    for (int i = 0; i < ir->GetNPoints(); i++)
    {
       const IntegrationPoint &ip = ir->IntPoint(i);
-      IntegrationPoint eip;
-      Tr.Loc1.Transform(ip, eip);
 
-      Tr.Face->SetIntPoint (&ip);
+      // Set the integration point in the face and the neighboring element
+      Tr.SetAllIntPoints(&ip);
+
+      // Access the neighboring element's integration point
+      const IntegrationPoint &eip = Tr.GetElement1IntPoint();
+
       double val = Tr.Face->Weight() * ip.weight * Q.Eval(*Tr.Face, ip);
 
       el.CalcShape(eip, shape);
@@ -255,7 +305,6 @@ void VectorDomainLFIntegrator::AssembleDeltaElementVect(
    MultVWt(shape, Qvec, elvec_as_mat);
 }
 
-
 void VectorBoundaryLFIntegrator::AssembleRHSElementVect(
    const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
 {
@@ -307,18 +356,22 @@ void VectorBoundaryLFIntegrator::AssembleRHSElementVect(
    if (ir == NULL)
    {
       int intorder = 2*el.GetOrder();
-      ir = &IntRules.Get(Tr.FaceGeom, intorder);
+      ir = &IntRules.Get(Tr.GetGeometryType(), intorder);
    }
 
    for (int i = 0; i < ir->GetNPoints(); i++)
    {
       const IntegrationPoint &ip = ir->IntPoint(i);
-      IntegrationPoint eip;
-      Tr.Loc1.Transform(ip, eip);
 
-      Tr.Face->SetIntPoint(&ip);
-      Q.Eval(vec, *Tr.Face, ip);
-      vec *= Tr.Face->Weight() * ip.weight;
+      // Set the integration point in the face and the neighboring element
+      Tr.SetAllIntPoints(&ip);
+
+      // Access the neighboring element's integration point
+      const IntegrationPoint &eip = Tr.GetElement1IntPoint();
+
+      // Use Tr transformation in case Q depends on boundary attribute
+      Q.Eval(vec, Tr, ip);
+      vec *= Tr.Weight() * ip.weight;
       el.CalcShape(eip, shape);
       for (int k = 0; k < vdim; k++)
       {
@@ -330,7 +383,6 @@ void VectorBoundaryLFIntegrator::AssembleRHSElementVect(
    }
 }
 
-
 void VectorFEDomainLFIntegrator::AssembleRHSElementVect(
    const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
 {
@@ -360,7 +412,6 @@ void VectorFEDomainLFIntegrator::AssembleRHSElementVect(
 
       QF.Eval (vec, Tr, ip);
       vec *= ip.weight * Tr.Weight();
-
       vshape.AddMult (vec, elvect);
    }
 }
@@ -381,6 +432,125 @@ void VectorFEDomainLFIntegrator::AssembleDeltaElementVect(
    vshape.Mult(vec, elvect);
 }
 
+void VectorFEDomainLFCurlIntegrator::AssembleRHSElementVect(
+   const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
+{
+   int dof = el.GetDof();
+   int spaceDim = Tr.GetSpaceDim();
+   int n=(spaceDim == 3)? spaceDim : 1;
+   curlshape.SetSize(dof,n);
+   vec.SetSize(n);
+
+   elvect.SetSize(dof);
+   elvect = 0.0;
+
+   const IntegrationRule *ir = IntRule;
+   if (ir == NULL)
+   {
+      int intorder = 2*el.GetOrder();
+      ir = &IntRules.Get(el.GetGeomType(), intorder);
+   }
+
+   for (int i = 0; i < ir->GetNPoints(); i++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(i);
+
+      Tr.SetIntPoint (&ip);
+      el.CalcPhysCurlShape(Tr, curlshape);
+
+      switch (spaceDim)
+      {
+         case 3:
+            MFEM_VERIFY(QF, "VectorFunctionCoefficient not provided");
+            QF->Eval(vec, Tr, ip);
+            break;
+         case 2:
+            MFEM_VERIFY(Q, "FunctionCoefficient (Scalar) not provided");
+            vec[0] = Q->Eval(Tr, ip);
+            break;
+         default:
+            break; // This should be unreachable
+      }
+      vec *= ip.weight * Tr.Weight();
+      curlshape.AddMult (vec, elvect);
+   }
+}
+
+void VectorFEDomainLFCurlIntegrator::AssembleDeltaElementVect(
+   const FiniteElement &fe, ElementTransformation &Trans, Vector &elvect)
+{
+   int spaceDim = Trans.GetSpaceDim();
+   switch (spaceDim)
+   {
+      case 3:
+         MFEM_ASSERT(vec_delta != NULL,
+                     "coefficient must be VectorDeltaCoefficient");
+         break;
+      case 2:
+         MFEM_ASSERT(delta != NULL,
+                     "coefficient must be DeltaCoefficient");
+         break;
+      default:
+         break; // This should be unreachable
+   }
+   int dof = fe.GetDof();
+   int n=(spaceDim == 3)? spaceDim : 1;
+   curlshape.SetSize(dof, n);
+   elvect.SetSize(dof);
+   fe.CalcPhysCurlShape(Trans, curlshape);
+
+   switch (spaceDim)
+   {
+      case 3:
+         vec_delta->EvalDelta(vec, Trans, Trans.GetIntPoint());
+         curlshape.Mult(vec, elvect);
+         break;
+      case 2:
+         curlshape.GetColumn(0,elvect);
+         elvect *= delta->EvalDelta(Trans, Trans.GetIntPoint());
+         break;
+      default:
+         break; // This should be unreachable
+   }
+}
+
+void VectorFEDomainLFDivIntegrator::AssembleRHSElementVect(
+   const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
+{
+   int dof = el.GetDof();
+
+   divshape.SetSize(dof);       // vector of size dof
+   elvect.SetSize(dof);
+   elvect = 0.0;
+
+   const IntegrationRule *ir = IntRule;
+   if (ir == NULL)
+   {
+      int intorder = 2 * el.GetOrder();
+      ir = &IntRules.Get(el.GetGeomType(), intorder);
+   }
+
+   for (int i = 0; i < ir->GetNPoints(); i++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(i);
+
+      Tr.SetIntPoint (&ip);
+      double val = Tr.Weight() * Q.Eval(Tr, ip);
+      el.CalcPhysDivShape(Tr, divshape);
+
+      add(elvect, ip.weight * val, divshape, elvect);
+   }
+}
+
+void VectorFEDomainLFDivIntegrator::AssembleDeltaElementVect(
+   const FiniteElement &fe, ElementTransformation &Trans, Vector &elvect)
+{
+   MFEM_ASSERT(delta != NULL, "coefficient must be DeltaCoefficient");
+   elvect.SetSize(fe.GetDof());
+   fe.CalcPhysDivShape(Trans, elvect);
+   elvect *= delta->EvalDelta(Trans, Trans.GetIntPoint());
+}
+
 void VectorBoundaryFluxLFIntegrator::AssembleRHSElementVect(
    const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
 {
@@ -446,7 +616,6 @@ void VectorFEBoundaryFluxLFIntegrator::AssembleRHSElementVect(
    }
 }
 
-
 void VectorFEBoundaryTangentLFIntegrator::AssembleRHSElementVect(
    const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
 {
@@ -481,7 +650,6 @@ void VectorFEBoundaryTangentLFIntegrator::AssembleRHSElementVect(
    }
 }
 
-
 void BoundaryFlowIntegrator::AssembleRHSElementVect(
    const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
 {
@@ -510,7 +678,7 @@ void BoundaryFlowIntegrator::AssembleRHSElementVect(
       {
          order++;
       }
-      ir = &IntRules.Get(Tr.FaceGeom, order);
+      ir = &IntRules.Get(Tr.GetGeometryType(), order);
    }
 
    shape.SetSize(ndof);
@@ -520,12 +688,16 @@ void BoundaryFlowIntegrator::AssembleRHSElementVect(
    for (int p = 0; p < ir->GetNPoints(); p++)
    {
       const IntegrationPoint &ip = ir->IntPoint(p);
-      IntegrationPoint eip;
-      Tr.Loc1.Transform(ip, eip);
+
+      // Set the integration point in the face and the neighboring element
+      Tr.SetAllIntPoints(&ip);
+
+      // Access the neighboring element's integration point
+      const IntegrationPoint &eip = Tr.GetElement1IntPoint();
       el.CalcShape(eip, shape);
 
-      Tr.Face->SetIntPoint(&ip);
-
+      // Use Tr.Elem1 transformation for u so that it matches the coefficient
+      // used with the ConvectionIntegrator and/or the DGTraceIntegrator.
       u->Eval(vu, *Tr.Elem1, eip);
 
       if (dim == 1)
@@ -534,17 +706,16 @@ void BoundaryFlowIntegrator::AssembleRHSElementVect(
       }
       else
       {
-         CalcOrtho(Tr.Face->Jacobian(), nor);
+         CalcOrtho(Tr.Jacobian(), nor);
       }
 
       un = vu * nor;
       w = 0.5*alpha*un - beta*fabs(un);
-      w *= ip.weight*f->Eval(*Tr.Elem1, eip);
+      w *= ip.weight*f->Eval(Tr, ip);
       elvect.Add(w, shape);
    }
 }
 
-
 void DGDirichletLFIntegrator::AssembleRHSElementVect(
    const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
 {
@@ -582,30 +753,33 @@ void DGDirichletLFIntegrator::AssembleRHSElementVect(
    {
       // a simple choice for the integration order; is this OK?
       int order = 2*el.GetOrder();
-      ir = &IntRules.Get(Tr.FaceGeom, order);
+      ir = &IntRules.Get(Tr.GetGeometryType(), order);
    }
 
    for (int p = 0; p < ir->GetNPoints(); p++)
    {
       const IntegrationPoint &ip = ir->IntPoint(p);
-      IntegrationPoint eip;
 
-      Tr.Loc1.Transform(ip, eip);
-      Tr.Face->SetIntPoint(&ip);
+      // Set the integration point in the face and the neighboring element
+      Tr.SetAllIntPoints(&ip);
+
+      // Access the neighboring element's integration point
+      const IntegrationPoint &eip = Tr.GetElement1IntPoint();
+
       if (dim == 1)
       {
          nor(0) = 2*eip.x - 1.0;
       }
       else
       {
-         CalcOrtho(Tr.Face->Jacobian(), nor);
+         CalcOrtho(Tr.Jacobian(), nor);
       }
 
       el.CalcShape(eip, shape);
       el.CalcDShape(eip, dshape);
-      Tr.Elem1->SetIntPoint(&eip);
+
       // compute uD through the face transformation
-      w = ip.weight * uD->Eval(*Tr.Face, ip) / Tr.Elem1->Weight();
+      w = ip.weight * uD->Eval(Tr, ip) / Tr.Elem1->Weight();
       if (!MQ)
       {
          if (Q)
@@ -633,7 +807,6 @@ void DGDirichletLFIntegrator::AssembleRHSElementVect(
    }
 }
 
-
 void DGElasticityDirichletLFIntegrator::AssembleRHSElementVect(
    const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
 {
@@ -676,19 +849,21 @@ void DGElasticityDirichletLFIntegrator::AssembleRHSElementVect(
    if (ir == NULL)
    {
       const int order = 2*el.GetOrder(); // <-----
-      ir = &IntRules.Get(Tr.FaceGeom, order);
+      ir = &IntRules.Get(Tr.GetGeometryType(), order);
    }
 
    for (int pi = 0; pi < ir->GetNPoints(); ++pi)
    {
       const IntegrationPoint &ip = ir->IntPoint(pi);
-      IntegrationPoint eip;
-      Tr.Loc1.Transform(ip, eip);
-      Tr.Face->SetIntPoint(&ip);
-      Tr.Elem1->SetIntPoint(&eip);
+
+      // Set the integration point in the face and the neighboring element
+      Tr.SetAllIntPoints(&ip);
+
+      // Access the neighboring element's integration point
+      const IntegrationPoint &eip = Tr.GetElement1IntPoint();
 
       // Evaluate the Dirichlet b.c. using the face transformation.
-      uD.Eval(u_dir, *Tr.Face, ip);
+      uD.Eval(u_dir, Tr, ip);
 
       el.CalcShape(eip, shape);
       el.CalcDShape(eip, dshape);
@@ -702,7 +877,7 @@ void DGElasticityDirichletLFIntegrator::AssembleRHSElementVect(
       }
       else
       {
-         CalcOrtho(Tr.Face->Jacobian(), nor);
+         CalcOrtho(Tr.Jacobian(), nor);
       }
 
       double wL, wM, jcoef;
@@ -768,4 +943,58 @@ void DGElasticityDirichletLFIntegrator::AssembleRHSElementVect(
    }
 }
 
+void VectorQuadratureLFIntegrator::AssembleRHSElementVect(
+   const FiniteElement &fe, ElementTransformation &Tr, Vector &elvect)
+{
+   const IntegrationRule *ir =
+      &vqfc.GetQuadFunction().GetSpace()->GetElementIntRule(Tr.ElementNo);
+
+   const int nqp = ir->GetNPoints();
+   const int vdim = vqfc.GetVDim();
+   const int ndofs = fe.GetDof();
+   Vector shape(ndofs);
+   Vector temp(vdim);
+   elvect.SetSize(vdim * ndofs);
+   elvect = 0.0;
+   for (int q = 0; q < nqp; q++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(q);
+      Tr.SetIntPoint(&ip);
+      const double w = Tr.Weight() * ip.weight;
+      vqfc.Eval(temp, Tr, ip);
+      fe.CalcShape(ip, shape);
+      for (int ind = 0; ind < vdim; ind++)
+      {
+         for (int nd = 0; nd < ndofs; nd++)
+         {
+            elvect(nd + ind * ndofs) += w * shape(nd) * temp(ind);
+         }
+      }
+   }
+}
+
+void QuadratureLFIntegrator::AssembleRHSElementVect(const FiniteElement &fe,
+                                                    ElementTransformation &Tr,
+                                                    Vector &elvect)
+{
+   const IntegrationRule *ir =
+      &qfc.GetQuadFunction().GetSpace()->GetElementIntRule(Tr.ElementNo);
+
+   const int nqp = ir->GetNPoints();
+   const int ndofs = fe.GetDof();
+   Vector shape(ndofs);
+   elvect.SetSize(ndofs);
+   elvect = 0.0;
+   for (int q = 0; q < nqp; q++)
+   {
+      const IntegrationPoint &ip = ir->IntPoint(q);
+      Tr.SetIntPoint (&ip);
+      const double w = Tr.Weight() * ip.weight;
+      double temp = qfc.Eval(Tr, ip);
+      fe.CalcShape(ip, shape);
+      shape *= (w * temp);
+      elvect += shape;
+   }
+}
+
 }

fem/lininteg.hpp

@@ -36,7 +36,7 @@ public:
                                        FaceElementTransformations &Tr,
                                        Vector &elvect);
 
-   void SetIntRule(const IntegrationRule *ir) { IntRule = ir; }
+   virtual void SetIntRule(const IntegrationRule *ir) { IntRule = ir; }
    const IntegrationRule* GetIntRule() { return IntRule; }
 
    virtual ~LinearFormIntegrator() { }
@@ -119,6 +119,33 @@ public:
    using LinearFormIntegrator::AssembleRHSElementVect;
 };
 
+/// Class for domain integrator L(v) := (f, grad v)
+class DomainLFGradIntegrator : public DeltaLFIntegrator
+{
+private:
+   Vector shape, Qvec;
+   VectorCoefficient &Q;
+   DenseMatrix dshape;
+
+public:
+   /// Constructs the domain integrator (Q, grad v)
+   DomainLFGradIntegrator(VectorCoefficient &QF)
+      : DeltaLFIntegrator(QF), Q(QF) { }
+
+   /** Given a particular Finite Element and a transformation (Tr)
+       computes the element right hand side element vector, elvect. */
+   virtual void AssembleRHSElementVect(const FiniteElement &el,
+                                       ElementTransformation &Tr,
+                                       Vector &elvect);
+
+   virtual void AssembleDeltaElementVect(const FiniteElement &fe,
+                                         ElementTransformation &Trans,
+                                         Vector &elvect);
+
+   using LinearFormIntegrator::AssembleRHSElementVect;
+};
+
+
 /// Class for boundary integration L(v) := (g, v)
 class BoundaryLFIntegrator : public LinearFormIntegrator
 {
@@ -252,6 +279,56 @@ public:
    using LinearFormIntegrator::AssembleRHSElementVect;
 };
 
+/// \f$ (Q, curl v)_{\Omega} \f$ for Nedelec Elements)
+class VectorFEDomainLFCurlIntegrator : public DeltaLFIntegrator
+{
+private:
+   VectorCoefficient *QF=nullptr;
+   Coefficient *Q=nullptr;
+   DenseMatrix curlshape;
+   Vector vec;
+
+public:
+   /// Constructs the domain integrator (Q, curl v)
+   VectorFEDomainLFCurlIntegrator(VectorCoefficient &F)
+      : DeltaLFIntegrator(F), QF(&F) { }
+   VectorFEDomainLFCurlIntegrator(Coefficient &F)
+      : DeltaLFIntegrator(F), Q(&F) { }
+
+   virtual void AssembleRHSElementVect(const FiniteElement &el,
+                                       ElementTransformation &Tr,
+                                       Vector &elvect);
+
+   virtual void AssembleDeltaElementVect(const FiniteElement &fe,
+                                         ElementTransformation &Trans,
+                                         Vector &elvect);
+
+   using LinearFormIntegrator::AssembleRHSElementVect;
+};
+
+/// \f$ (Q, div v)_{\Omega} \f$ for RT Elements)
+class VectorFEDomainLFDivIntegrator : public DeltaLFIntegrator
+{
+private:
+   Vector divshape;
+   Coefficient &Q;
+public:
+   /// Constructs the domain integrator (Q, div v)
+   VectorFEDomainLFDivIntegrator(Coefficient &QF)
+      : DeltaLFIntegrator(QF), Q(QF) { }
+
+   /** Given a particular Finite Element and a transformation (Tr)
+       computes the element right hand side element vector, elvect. */
+   virtual void AssembleRHSElementVect(const FiniteElement &el,
+                                       ElementTransformation &Tr,
+                                       Vector &elvect);
+
+   virtual void AssembleDeltaElementVect(const FiniteElement &fe,
+                                         ElementTransformation &Trans,
+                                         Vector &elvect);
+
+   using LinearFormIntegrator::AssembleRHSElementVect;
+};
 
 /** \f$ (f, v \cdot n)_{\partial\Omega} \f$ for vector test function
     v=(v1,...,vn) where all vi are in the same scalar FE space and f is a
@@ -283,7 +360,7 @@ class VectorFEBoundaryFluxLFIntegrator : public LinearFormIntegrator
 private:
    Coefficient *F;
    Vector shape;
-   int oa, ob; // these contol the quadrature order, see DomainLFIntegrator
+   int oa, ob; // these control the quadrature order, see DomainLFIntegrator
 
 public:
    VectorFEBoundaryFluxLFIntegrator(int a = 1, int b = -1)
@@ -426,6 +503,69 @@ public:
                                        Vector &elvect);
 };
 
+/** Class for domain integration of L(v) := (f, v), where
+    f=(f1,...,fn) and v=(v1,...,vn). that makes use of
+    VectorQuadratureFunctionCoefficient*/
+class VectorQuadratureLFIntegrator : public LinearFormIntegrator
+{
+private:
+   VectorQuadratureFunctionCoefficient &vqfc;
+
+public:
+   VectorQuadratureLFIntegrator(VectorQuadratureFunctionCoefficient &vqfc,
+                                const IntegrationRule *ir)
+      : LinearFormIntegrator(ir), vqfc(vqfc)
+   {
+      if (ir)
+      {
+         MFEM_WARNING("Integration rule not used in this class. "
+                      "The QuadratureFunction integration rules are used instead");
+      }
+   }
+
+   using LinearFormIntegrator::AssembleRHSElementVect;
+   virtual void AssembleRHSElementVect(const FiniteElement &fe,
+                                       ElementTransformation &Tr,
+                                       Vector &elvect);
+
+   virtual void SetIntRule(const IntegrationRule *ir)
+   {
+      MFEM_WARNING("Integration rule not used in this class. "
+                   "The QuadratureFunction integration rules are used instead");
+   }
+};
+
+/** Class for domain integration L(v) := (f, v) that makes use
+    of QuadratureFunctionCoefficient. */
+class QuadratureLFIntegrator : public LinearFormIntegrator
+{
+private:
+   QuadratureFunctionCoefficient &qfc;
+
+public:
+   QuadratureLFIntegrator(QuadratureFunctionCoefficient &qfc,
+                          const IntegrationRule *ir)
+      : LinearFormIntegrator(ir), qfc(qfc)
+   {
+      if (ir)
+      {
+         MFEM_WARNING("Integration rule not used in this class. "
+                      "The QuadratureFunction integration rules are used instead");
+      }
+   }
+
+   using LinearFormIntegrator::AssembleRHSElementVect;
+   virtual void AssembleRHSElementVect(const FiniteElement &fe,
+                                       ElementTransformation &Tr,
+                                       Vector &elvect);
+
+   virtual void SetIntRule(const IntegrationRule *ir)
+   {
+      MFEM_WARNING("Integration rule not used in this class. "
+                   "The QuadratureFunction integration rules are used instead");
+   }
+};
+
 }
 
 #endif