Merge remote-tracking branch 'origin/master' into complex-strumpack-dev

Okina GPU [okina]

.gitignore

@@ -169,6 +169,7 @@ miniapps/performance/sol.*
 miniapps/tools/display-basis
 miniapps/tools/load-dc
 miniapps/tools/convert-dc
+miniapps/tools/lor-transfer
 
 miniapps/nurbs/ex1
 miniapps/nurbs/ex1p

CHANGELOG

@@ -8,12 +8,69 @@
                                http://mfem.org
 
 
-Version 3.4.1 (development)
-===========================
+Version 4.0-RC1, Apr 11, 2019
+=============================
+
+Requirements and Limitations
+----------------------------
+- This is a release candidate for mfem-4.0.
+- Use at your own risk -- not everything will work, the API may change.
+- We are looking for feedback from friendly users.
+- Unlike previous MFEM releases, this version requires a C++11 compiler.
+
+- GPU-related limitations:
+  * NVCC is not supported in the CMake build system yet.
+  * Element batching is currently ignored.
+  * Full-assembly (on device), element assembly, and matrix-free bilinear forms
+    are not supported yet.
+  * FunctionCoefficients do not currently work on GPUs.
+  * Partial assembly kernels are not implemented yet for simplices.
+
+GPU support
+-----------
+- Added initial support for hardware devices, such as GPUs, and programming
+  models, such as CUDA, OCCA, RAJA and OpenMP.
+
+- The GPU/device support is based on MFEM's new backends and kernels working
+  seamlessly with a new lightweight device/host memory manager. The kernels can
+  be implemented either in OCCA, or as a simple wrapper around for-loops, which
+  can then be dispatched to RAJA and native backends. See the files forall.hpp
+  and mem_manager.hpp in the general/ directory.
+
+- Several of the MFEM example codes (ex1, ex1p, ex6, and ex6p) can now take
+  advantage of GPU acceleration with the backend selectable at runtime. Many of
+  the linear algebra and finite element operations (e.g. partially assembled
+  bilinear forms) have been extended to take advantage of kernel acceleration by
+  simply replacing loops with the MFEM_FORALL() macro.
+
+- In addition to pure CUDA, the library currently supports OCCA, RAJA and OpenMP
+  kernels, which could be mixed and matched in different parts of the same
+  application. We plan on adding support for more programming models and devices
+  in the future, without the need for significant modifications in user code.
+  The list of current backends is: "occa-cuda", "raja-cuda", "cuda", "occa-omp",
+  "raja-omp", "omp", "occa-cpu", "raja-cpu", and "cpu".
+
+Discretization improvements
+---------------------------
+- Added support for a general "low-order refined"-to-"high-order" transfer of
+  GridFunction data from a "low-order refined" (LOR) space defined on a refined
+  mesh to a "high-order" (HO) finite element space defined on a coarse mesh. See
+  the new classes InterpolationGridTransfer and L2ProjectionGridTransfer and the
+  new LOR Transfer miniapp: miniapps/tools/lor-transfer.cpp.
+
+- Added support for derefinement of vector (RT + ND) spaces.
+
+- Added element flux, and flux energy computation in class ElasticityIntegrator,
+  allowing for the use of Zienkiewicz-Zhu type error estimators with the
+  integrator. For an illustration of this addition, see the new Example 22.
+
+- Added a variety of coefficients which are sums or products of existing
+  coefficients as well as grid function coefficients which return the
+  divergence, gradient, or curl of their GridFunctions.
 
 Support for wedge elements and meshes with mixed element types
 --------------------------------------------------------------
-- Added support for wedge shaped mesh elements of arbitrary order (with Geometry
+- Added support for wedge-shaped mesh elements of arbitrary order (with Geometry
   type PRISM) which have two triangular faces and three quadrilateral faces.
   Several examples of such meshes can be found in the data/ directory.
 
@@ -39,13 +96,18 @@ Other meshing improvements
   follows precisely the paper:
 
      D. Arnold, A. Mukherjee, and L. Pouly, "Locally Adapted Tetrahedral Meshes
-     Using Bisection", SIAM J. Sci. Comput., 22(2), 431–448.
+     Using Bisection", SIAM J. Sci. Comput. 22 (2000), 431–448.
 
   This guarantees that the shape regularity of the elements will be preserved
   under refinement.
 
 - Added support for parallel communication groups on non-conforming meshes.
 
+- Improved parallel partitioning of non-conforming meshes. If the coarse mesh
+  elements are ordered as a sequence of face-neighbors, the parallel partitions
+  are now guaranteed to be continuous. To that end, inline quadrilateral and
+  hexahedral meshes are now by default ordered along a space-filling curve.
+
 - A boundary in a NURBS mesh can now be connected with another boundary. Such a
   periodic NURBS mesh is a simple way to impose periodic boundary conditions.
 
@@ -53,24 +115,8 @@ Other meshing improvements
   Cubit meshes.
 
 - The TMOP mesh optimization algorithms were extended to support user-defined
-  space-dependent limiting terms. Improved the TMOP objective functions by
-  more accurate normalization of the different terms.
-  
-Discretization improvements
----------------------------
-- Added support for derefinement of vector (RT + ND) spaces.
-
-- Added element flux, and flux energy computation in class ElasticityIntegrator,
-  allowing for the use of Zienkiewicz-Zhu type error estimators with the
-  integrator. For an illustration of this addition, see the new Example 22.
-
-- Added a variety of coefficients which are sums or products of existing
-  coefficients as well as grid function coefficients which return the
-  divergence, gradient, or curl of their GridFunctions.
-
-New and improved solvers and preconditioners
---------------------------------------------
-- Added support for parallel ILU preconditioning via hypre's Euclid solver.
+  space-dependent limiting terms. Improved the TMOP objective functions by more
+  accurate normalization of the different terms.
 
 New and updated examples and miniapps
 -------------------------------------
@@ -80,6 +126,9 @@ New and updated examples and miniapps
 - Added a new meshing miniapp, Extruder, that demonstrates the capability to
   produce 3D meshes by extruding 2D meshes.
 
+- Added a simple miniapp, LOR Transfer, for visualizing the actions of the
+  transfer operators between a high-order and a low-order refined spaces.
+
 - Added a new example, Example 20/20p, that solves a system of 1D ODEs derived
   from a Hamiltonian. The example demonstrates the use of the variable order,
   symplectic integration algorithm implemented in class SIAVSolver.
@@ -87,10 +136,22 @@ New and updated examples and miniapps
 - Added a new example, Example 22/22p, that illustrates the use of AMR to solve
   a linear elasticity problem. This is an extension of Example 2/2p.
 
+New and improved solvers and preconditioners
+--------------------------------------------
+- Added support for parallel ILU preconditioning via hypre's Euclid solver.
+
+- Added support for STRUMPACK v3 with a small API change in the class
+  STRUMPACKSolver, see "API changes" below.
+
 Miscellaneous
 -------------
 - Added unit tests based on the Catch++ library.
 
+- Renamed the option MFEM_USE_OPENMP to MFEM_USE_LEGACY_OPENMP. This legacy
+  option is deprecated and planned for removal in a future release. The original
+  option name, MFEM_USE_OPENMP, is now used to enable the new OpenMP backends in
+  the new kernels.
+
 - Altered the way FGMRES counts its iterations so that it matches GMRES.
 
 - Various other simplifications, extensions, and bugfixes in the code.
@@ -115,6 +176,8 @@ API changes
 - Removed the virtual method Element::GetRefinementFlag, it is only used by the
   derived class Tetrahedron.
 - Added new methods: Array::CopyTo, Tetrahedron::Init.
+- In class STRUMPACKSolver, the method SetMC64Job() was replaced by the new
+  methods: DisableMatching(), EnableMatching(), and EnableParallelMatching().
 
 
 Version 3.4, released on May 29, 2018

CMakeLists.txt

@@ -13,6 +13,11 @@ cmake_minimum_required(VERSION 2.8.11)
 set(USER_CONFIG "${CMAKE_CURRENT_SOURCE_DIR}/config/user.cmake" CACHE PATH
   "Path to optional user configuration file.")
 
+# Require C++11 and disable compiler-specific extensions
+set(CMAKE_CXX_STANDARD 11)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(CMAKE_CXX_EXTENSIONS OFF)
+
 # Load user settings before the defaults - this way the defaults will not
 # overwrite the user set options. If the user has not set all options, we still
 # have the defaults.
@@ -170,12 +175,11 @@ if (MFEM_USE_LAPACK)
 endif()
 
 # OpenMP
-if (MFEM_USE_OPENMP)
-  if (MFEM_THREAD_SAFE)
-    find_package(OpenMP REQUIRED)
-  else()
-    message(FATAL_ERROR " *** MFEM_USE_OPENMP requires MFEM_THREAD_SAFE=ON.")
+if (MFEM_USE_OPENMP OR MFEM_USE_LEGACY_OPENMP)
+  if (NOT MFEM_THREAD_SAFE AND MFEM_USE_LEGACY_OPENMP)
+    message(FATAL_ERROR " *** MFEM_USE_LEGACY_OPENMP requires MFEM_THREAD_SAFE=ON.")
   endif()
+  find_package(OpenMP REQUIRED)
 endif()
 
 # SuiteSparse (before SUNDIALS which may depend on KLU)

INSTALL

@@ -21,6 +21,24 @@ requires an MPI C++ compiler, as well as the following external libraries:
 The METIS dependency can be disabled but that is not generally recommended, see
 the option MFEM_USE_METIS.
 
+MFEM also includes support for devices such as GPUs, and programming models such
+as CUDA, OCCA, OpenMP and RAJA.
+
+- Starting with version 4.0, MFEM requires a C++11 compiler
+
+- CUDA support requires an NVIDIA GPU and an installation of the CUDA Toolkit
+  https://developer.nvidia.com/cuda-toolkit
+
+- OCCA support requires the OCCA library
+  https://libocca.org
+
+- OpenMP support requires a compiler implementing the OpenMP API
+  https://www.openmp.org
+
+- RAJA support requires installation of the RAJA performance portability layer
+  with (optionally) support for CUDA and OpenMP
+  https://github.com/LLNL/RAJA
+
 The library supports two build systems: one based on GNU make, and a second one
 based on CMake. Both build systems are described below. Some hints for building
 without GNU make or CMake can be found at the end of this file.
@@ -47,6 +65,10 @@ Parallel build:
    (build hypre 2.10.0b in ../hypre-2.10.0b relative to mfem/)
    make parallel -j 4
 
+CUDA build:
+   make cuda -j 4
+   (build for a specific compute capability: 'make cuda -j 4 CUDA_ARCH=sm_30')
+
 Example codes (serial/parallel, depending on the build):
    cd examples
    make -j 4
@@ -57,7 +79,6 @@ Build everything (library, examples and miniapps) with current configuration:
 Quick-check the build by running Example 1/1p (optional):
    make check
 
-
 Quick start with CMake
 ======================
 Serial build:
@@ -132,6 +153,10 @@ are also defined:
    make parallel -> Builds parallel optimized version of the library
    make debug    -> Builds serial debug version of the library
    make pdebug   -> Builds parallel debug version of the library
+   make cuda     -> Builds serial cuda optimized version of the library
+   make pcuda    -> Builds parallel cuda optimized version of the library
+   make cudebug  -> Builds serial cuda debug version of the library
+   make pcudebug -> Builds parallel cuda debug version of the library
 
 Note that any of the above shortcuts accept configuration options, either at the
 command line or through a user configuration file.
@@ -193,8 +218,9 @@ Configuration options (GNU make)
 See the configuration file config/defaults.mk for the default settings.
 
 Compilers:
-   CXX    - C++ compiler, serial build
-   MPICXX - MPI C++ compiler, parallel build
+   CXX      - C++ compiler, serial build
+   MPICXX   - MPI C++ compiler, parallel build
+   CUDA_CXX - The CUDA compiler, 'nvcc'
 
 Compiler options:
    OPTIM_FLAGS - Options for optimized build
@@ -230,7 +256,7 @@ MFEM_DEBUG = YES/NO
    and consistency checks that may simplify bug-hunting.
 
 MFEM_USE_EXCEPTIONS = YES/NO
-   Enable the use of exceptions. In particular, modifies the default bahavior
+   Enable the use of exceptions. In particular, modifies the default behavior
    when errors are encountered: throw an exception, instead of aborting.
 
 MFEM_USE_LIBUNWIND = YES/NO
@@ -250,9 +276,12 @@ MFEM_THREAD_SAFE = YES/NO
    Use thread-safe implementation for some classes/methods. This comes at the
    cost of extra memory allocation and de-allocation.
 
-MFEM_USE_OPENMP = YES/NO
+MFEM_USE_LEGACY_OPENMP = YES/NO
    Enable (basic) experimental OpenMP support. Requires MFEM_THREAD_SAFE.
 
+MFEM_USE_OPENMP = YES/NO
+   Enable the OpenMP backend.
+
 MFEM_USE_MEMALLOC = YES/NO
    Internal MFEM option: enable batch allocation for some small objects.
    Recommended value is YES.
@@ -362,6 +391,29 @@ MFEM_USE_PUMI = YES/NO
    models and effectively supports automated adaptive analysis. PUMI enables
    support for parallel unstructured mesh modifications in MFEM.
 
+MFEM_USE_MM = YES/NO
+   Enables support for the MFEM's memory manager (MM), which is required to
+   support devices with different memory spaces.
+
+MFEM_USE_CUDA = YES/NO
+   Enables support for CUDA devices in MFEM. CUDA is a parallel computing
+   platform and programming model for general computing on graphical processing
+   units (GPUs). This option requires MFEM_USE_MM. The variable CUDA_ARCH is
+   used to specify the CUDA compute capability used during compilation (by
+   default, CUDA_ARCH=sm_60). When enabled, this option uses the CUDA_* build
+   options, see below.
+
+MFEM_USE_RAJA = YES/NO
+   Enable support for the RAJA performance portability layer in MFEM. RAJA
+   provides a portable abstraction for loops, supporting different programming
+   model backends. When using the RAJA CUDA backend, MFEM_USE_MM is required.
+
+MFEM_USE_OCCA = YES/NO
+   Enables support for the OCCA library in MFEM. OCCA is an open-source library
+   which aims to make it easy to program different types of devices (e.g. CPU,
+   GPU, FPGA) by providing an unified API for interacting with JIT-compiled
+   backends. When using the OCCA CUDA backend, MFEM_USE_MM is required.
+
 MFEM_BUILD_TAG = (any value)
    An optional tag to characterize the build. Exported to config/config.mk.
    Can be used to identify the MFEM build from other makefiles.
@@ -397,7 +449,8 @@ The specific libraries and their options are:
        http://math-atlas.sourceforge.net (ATLAS)
   Options: LAPACK_OPT (currently not used/needed), LAPACK_LIB.
 
-- OpenMP (optional), usually part of compiler, used when MFEM_USE_OPENMP = YES.
+- OpenMP (optional), usually part of compiler, used when either MFEM_USE_OPENMP
+  or MFEM_USE_LEGACY_OPENMP is set to YES.
   Options: OPENMP_OPT, OPENMP_LIB.
 
 - High-resolution POSIX clocks: when using MFEM_TIMER_TYPE = 2, it may be
@@ -429,7 +482,8 @@ The specific libraries and their options are:
 
 - STRUMPACK (optional), used when MFEM_USE_STRUMPACK = YES. Note that STRUMPACK
   requires the PT-Scotch and Scalapack libraries as well as ParMETIS, which
-  includes METIS 5 in its distribution.
+  includes METIS 5 in its distribution. Starting with STRUMPACK v2.2.0, ParMETIS
+  and PT-Scotch are optional dependencies.
   The support for STRUMPACK was added in MFEM v3.3.2 and it requires STRUMPACK
   2.0.0 or later.
   URL: http://portal.nersc.gov/project/sparse/strumpack
@@ -475,6 +529,18 @@ The specific libraries and their options are:
   URL: https://scorec.rpi.edu/pumi
   Options: PUMI_OPT, PUMI_LIB.
 
+- CUDA, used when MFEM_USE_CUDA = YES.
+  URL: https://developer.nvidia.com/cuda-toolkit
+  Options: CUDA_CXX, CUDA_ARCH, CUDA_OPT, CUDA_LIB.
+
+- OCCA, used when MFEM_USE_OCCA = YES.
+  URL: https://libocca.org
+  Options: OCCA_DIR, OCCA_OPT, OCCA_LIB.
+
+- RAJA, used when MFEM_USE_RAJA = YES.
+  URL: https://github.com/LLNL/RAJA
+  Options: RAJA_DIR, RAJA_OPT, RAJA_LIB.
+
 - MPFR (optional), used when MFEM_USE_MPFR = YES.
   URL: http://mpfr.org, it depends on the GMP library: https://gmplib.org
   Options: MPFR_OPT, MPFR_LIB.
@@ -596,6 +662,7 @@ MFEM_USE_METIS - Set to ${MFEM_USE_MPI}, can be overwritten.
 MFEM_USE_LIBUNWIND
 MFEM_USE_LAPACK
 MFEM_THREAD_SAFE
+MFEM_USE_LEGACY_OPENMP
 MFEM_USE_OPENMP
 MFEM_USE_MEMALLOC
 MFEM_TIMER_TYPE - Set automatically, can be overwritten.
@@ -609,6 +676,13 @@ MFEM_USE_MPFR
 MFEM_USE_GZSTREAM
 MFEM_USE_PUMI
 
+The following GNU make options are not supported with CMake yet:
+
+MFEM_USE_CUDA
+MFEM_USE_OCCA
+MFEM_USE_RAJA
+MFEM_USE_MM
+
 The following options are CMake specific:
 
 MFEM_ENABLE_TESTING  - Enable the ctest framework for testing.

config/cmake/MFEMConfig.cmake.in

@@ -25,6 +25,7 @@ set(MFEM_USE_LIBUNWIND @MFEM_USE_LIBUNWIND@)
 set(MFEM_USE_LAPACK @MFEM_USE_LAPACK@)
 set(MFEM_THREAD_SAFE @MFEM_THREAD_SAFE@)
 set(MFEM_USE_OPENMP @MFEM_USE_OPENMP@)
+set(MFEM_USE_LEGACY_OPENMP @MFEM_USE_LEGACY_OPENMP@)
 set(MFEM_USE_MEMALLOC @MFEM_USE_MEMALLOC@)
 set(MFEM_TIMER_TYPE @MFEM_TIMER_TYPE@)
 set(MFEM_USE_SUNDIALS @MFEM_USE_SUNDIALS@)

config/cmake/config.hpp.in

@@ -62,9 +62,12 @@
 // allocation and de-allocation.
 #cmakedefine MFEM_THREAD_SAFE
 
-// Enable experimental OpenMP support. Requires MFEM_THREAD_SAFE.
+// Enable the OpenMP backend.
 #cmakedefine MFEM_USE_OPENMP
 
+// [Deprecated] Enable experimental OpenMP support. Requires MFEM_THREAD_SAFE.
+#cmakedefine MFEM_USE_LEGACY_OPENMP
+
 // Enable MFEM functionality based on the Mesquite library.
 #cmakedefine MFEM_USE_MESQUITE
 

config/cmake/modules/MfemCmakeUtilities.cmake

@@ -714,11 +714,11 @@ function(mfem_export_mk_files)
   # Convert Boolean vars to YES/NO without writting the values to cache
   set(CONFIG_MK_BOOL_VARS MFEM_USE_MPI MFEM_USE_METIS MFEM_USE_METIS_5
       MFEM_DEBUG MFEM_USE_EXCEPTIONS MFEM_USE_GZSTREAM MFEM_USE_LIBUNWIND
-      MFEM_USE_LAPACK MFEM_THREAD_SAFE MFEM_USE_OPENMP MFEM_USE_MEMALLOC
-      MFEM_USE_SUNDIALS MFEM_USE_MESQUITE MFEM_USE_SUITESPARSE MFEM_USE_SUPERLU
-      MFEM_USE_STRUMPACK MFEM_USE_GECKO MFEM_USE_GNUTLS MFEM_USE_NETCDF
-      MFEM_USE_PETSC MFEM_USE_MPFR MFEM_USE_SIDRE MFEM_USE_CONDUIT
-      MFEM_USE_PUMI)
+      MFEM_USE_LAPACK MFEM_THREAD_SAFE MFEM_USE_OPENMP MFEM_USE_LEGACY_OPENMP
+      MFEM_USE_MEMALLOC MFEM_USE_SUNDIALS MFEM_USE_MESQUITE MFEM_USE_SUITESPARSE
+      MFEM_USE_SUPERLU MFEM_USE_STRUMPACK MFEM_USE_GECKO MFEM_USE_GNUTLS
+      MFEM_USE_NETCDF MFEM_USE_PETSC MFEM_USE_MPFR MFEM_USE_SIDRE
+      MFEM_USE_CONDUIT MFEM_USE_PUMI)
   foreach(var ${CONFIG_MK_BOOL_VARS})
     if (${var})
       set(${var} YES)

config/config.hpp

@@ -15,6 +15,9 @@
 //
 // Otherwise, use the local file: _config.hpp.
 
+#ifndef MFEM_CONFIG_HPP
+#define MFEM_CONFIG_HPP
+
 #ifdef MFEM_BUILD_DIR
 #define MFEM_QUOTE(a) #a
 #define MFEM_MAKE_PATH(x,y) MFEM_QUOTE(x/y)
@@ -52,3 +55,5 @@
 #error Building with PUMI (MFEM_USE_PUMI=YES) requires MPI (MFEM_USE_MPI=YES)
 #endif
 #endif // MFEM_USE_MPI not defined
+
+#endif // MFEM_CONFIG_HPP

config/config.hpp.in

@@ -30,6 +30,12 @@
 #define MFEM_VERSION_MINOR (((MFEM_VERSION)/100)%100)
 #define MFEM_VERSION_PATCH ((MFEM_VERSION)%100)
 
+// The absolute path of the MFEM source prefix
+// #define MFEM_SOURCE_DIR "@MFEM_SOURCE_DIR@"
+
+// The absolute path of the MFEM installation prefix
+// #define MFEM_INSTALL_DIR "@MFEM_INSTALL_DIR@"
+
 // Description of the git commit used to build MFEM.
 // #define MFEM_GIT_STRING "@MFEM_GIT_STRING@"
 
@@ -62,9 +68,12 @@
 // allocation and de-allocation.
 // #define MFEM_THREAD_SAFE
 
-// Enable experimental OpenMP support. Requires MFEM_THREAD_SAFE.
+// Enable the OpenMP backend.
 // #define MFEM_USE_OPENMP
 
+// [Deprecated] Enable experimental OpenMP support. Requires MFEM_THREAD_SAFE.
+// #define MFEM_USE_LEGACY_OPENMP
+
 // Internal MFEM option: enable group/batch allocation for some small objects.
 // #define MFEM_USE_MEMALLOC
 
@@ -112,6 +121,19 @@
 // Enable MFEM functionality based on the PUMI library
 // #define MFEM_USE_PUMI
 
+// Build the GPU/CUDA-enabled version of the MFEM library.
+// Requires a CUDA compiler (nvcc).
+// #define MFEM_USE_CUDA
+
+// Enable functionality based on the RAJA library.
+// #define MFEM_USE_RAJA
+
+// Enable functionality based on the OCCA library.
+// #define MFEM_USE_OCCA
+
+// Enable MFEM's internal Memory Manager (needed e.g. for MFEM_USE_CUDA)
+// #define MFEM_USE_MM
+
 // Version of HYPRE used for building MFEM.
 // #define MFEM_HYPRE_VERSION @MFEM_HYPRE_VERSION@
 

config/config.mk.in

@@ -10,34 +10,41 @@
 # Software Foundation) version 2.1 dated February 1999.
 
 # Variables corresponding to defines in config.hpp (YES, NO, or value)
-MFEM_VERSION         = @MFEM_VERSION@
-MFEM_VERSION_STRING  = @MFEM_VERSION_STRING@
-MFEM_GIT_STRING      = @MFEM_GIT_STRING@
-MFEM_USE_MPI         = @MFEM_USE_MPI@
-MFEM_USE_METIS       = @MFEM_USE_METIS@
-MFEM_USE_METIS_5     = @MFEM_USE_METIS_5@
-MFEM_DEBUG           = @MFEM_DEBUG@
-MFEM_USE_EXCEPTIONS  = @MFEM_USE_EXCEPTIONS@
-MFEM_USE_GZSTREAM    = @MFEM_USE_GZSTREAM@
-MFEM_USE_LIBUNWIND   = @MFEM_USE_LIBUNWIND@
-MFEM_USE_LAPACK      = @MFEM_USE_LAPACK@
-MFEM_THREAD_SAFE     = @MFEM_THREAD_SAFE@
-MFEM_USE_OPENMP      = @MFEM_USE_OPENMP@
-MFEM_USE_MEMALLOC    = @MFEM_USE_MEMALLOC@
-MFEM_TIMER_TYPE      = @MFEM_TIMER_TYPE@
-MFEM_USE_SUNDIALS    = @MFEM_USE_SUNDIALS@
-MFEM_USE_MESQUITE    = @MFEM_USE_MESQUITE@
-MFEM_USE_SUITESPARSE = @MFEM_USE_SUITESPARSE@
-MFEM_USE_SUPERLU     = @MFEM_USE_SUPERLU@
-MFEM_USE_STRUMPACK   = @MFEM_USE_STRUMPACK@
-MFEM_USE_GECKO       = @MFEM_USE_GECKO@
-MFEM_USE_GNUTLS      = @MFEM_USE_GNUTLS@
-MFEM_USE_NETCDF      = @MFEM_USE_NETCDF@
-MFEM_USE_PETSC       = @MFEM_USE_PETSC@
-MFEM_USE_MPFR        = @MFEM_USE_MPFR@
-MFEM_USE_SIDRE       = @MFEM_USE_SIDRE@
-MFEM_USE_CONDUIT     = @MFEM_USE_CONDUIT@
-MFEM_USE_PUMI        = @MFEM_USE_PUMI@
+MFEM_VERSION           = @MFEM_VERSION@
+MFEM_VERSION_STRING    = @MFEM_VERSION_STRING@
+MFEM_SOURCE_DIR        = @MFEM_SOURCE_DIR@
+MFEM_INSTALL_DIR       = @MFEM_INSTALL_DIR@
+MFEM_GIT_STRING        = @MFEM_GIT_STRING@
+MFEM_USE_MPI           = @MFEM_USE_MPI@
+MFEM_USE_METIS         = @MFEM_USE_METIS@
+MFEM_USE_METIS_5       = @MFEM_USE_METIS_5@
+MFEM_DEBUG             = @MFEM_DEBUG@
+MFEM_USE_EXCEPTIONS    = @MFEM_USE_EXCEPTIONS@
+MFEM_USE_GZSTREAM      = @MFEM_USE_GZSTREAM@
+MFEM_USE_LIBUNWIND     = @MFEM_USE_LIBUNWIND@
+MFEM_USE_LAPACK        = @MFEM_USE_LAPACK@
+MFEM_THREAD_SAFE       = @MFEM_THREAD_SAFE@
+MFEM_USE_LEGACY_OPENMP = @MFEM_USE_LEGACY_OPENMP@
+MFEM_USE_OPENMP        = @MFEM_USE_OPENMP@
+MFEM_USE_MEMALLOC      = @MFEM_USE_MEMALLOC@
+MFEM_TIMER_TYPE        = @MFEM_TIMER_TYPE@
+MFEM_USE_SUNDIALS      = @MFEM_USE_SUNDIALS@
+MFEM_USE_MESQUITE      = @MFEM_USE_MESQUITE@
+MFEM_USE_SUITESPARSE   = @MFEM_USE_SUITESPARSE@
+MFEM_USE_SUPERLU       = @MFEM_USE_SUPERLU@
+MFEM_USE_STRUMPACK     = @MFEM_USE_STRUMPACK@
+MFEM_USE_GECKO         = @MFEM_USE_GECKO@
+MFEM_USE_GNUTLS        = @MFEM_USE_GNUTLS@
+MFEM_USE_NETCDF        = @MFEM_USE_NETCDF@
+MFEM_USE_PETSC         = @MFEM_USE_PETSC@
+MFEM_USE_MPFR          = @MFEM_USE_MPFR@
+MFEM_USE_SIDRE         = @MFEM_USE_SIDRE@
+MFEM_USE_CONDUIT       = @MFEM_USE_CONDUIT@
+MFEM_USE_PUMI          = @MFEM_USE_PUMI@
+MFEM_USE_CUDA          = @MFEM_USE_CUDA@
+MFEM_USE_RAJA          = @MFEM_USE_RAJA@
+MFEM_USE_OCCA          = @MFEM_USE_OCCA@
+MFEM_USE_MM            = @MFEM_USE_MM@
 
 # Compiler, compile options, and link options
 MFEM_CXX       = @MFEM_CXX@
@@ -65,5 +72,8 @@ MFEM_MPIEXEC    = @MFEM_MPIEXEC@
 MFEM_MPIEXEC_NP = @MFEM_MPIEXEC_NP@
 MFEM_MPI_NP     = @MFEM_MPI_NP@
 
+# The NVCC compiler cannot link with -x=cu
+MFEM_LINK_FLAGS := $(filter-out -x=cu, $(MFEM_FLAGS))
+
 # Optional extra configuration
 @MFEM_CONFIG_EXTRA@

config/defaults.cmake

@@ -26,7 +26,8 @@ option(MFEM_USE_GZSTREAM "Enable gzstream for compressed data streams." OFF)
 option(MFEM_USE_LIBUNWIND "Enable backtrace for errors." OFF)
 option(MFEM_USE_LAPACK "Enable LAPACK usage" OFF)
 option(MFEM_THREAD_SAFE "Enable thread safety" OFF)
-option(MFEM_USE_OPENMP "Enable OpenMP usage" OFF)
+option(MFEM_USE_OPENMP "Enable the OpenMP backend" OFF)
+option(MFEM_USE_LEGACY_OPENMP "Enable legacy OpenMP usage" OFF)
 option(MFEM_USE_MEMALLOC "Enable the internal MEMALLOC option." ON)
 option(MFEM_USE_SUNDIALS "Enable SUNDIALS usage" OFF)
 option(MFEM_USE_MESQUITE "Enable MESQUITE usage" OFF)
@@ -102,6 +103,7 @@ set(SuperLUDist_REQUIRED_PACKAGES "MPI" "BLAS" "ParMETIS" CACHE STRING
 set(STRUMPACK_DIR "${MFEM_DIR}/../STRUMPACK-build" CACHE PATH
     "Path to the STRUMPACK library.")
 # STRUMPACK may also depend on "OpenMP", depending on how it was compiled.
+# Starting with v2.2.0 of STRUMPACK, ParMETIS and Scotch are optional.
 set(STRUMPACK_REQUIRED_PACKAGES "MPI" "MPI_Fortran" "ParMETIS" "METIS"
     "ScaLAPACK" "Scotch/ptscotch/ptscotcherr/scotch/scotcherr" CACHE STRING
     "Additional packages required by STRUMPACK.")
@@ -109,7 +111,8 @@ set(STRUMPACK_REQUIRED_PACKAGES "MPI" "MPI_Fortran" "ParMETIS" "METIS"
 # set(STRUMPACK_REQUIRED_LIBRARIES "gfortran" "mpi_mpifh" CACHE STRING
 #     "Additional libraries required by STRUMPACK.")
 
-# The Scotch library, required by STRUMPACK
+# The Scotch library, required by STRUMPACK <= v2.1.0, optional in STRUMPACK >=
+# v2.2.0.
 set(Scotch_DIR "${MFEM_DIR}/../scotch_6.0.4" CACHE PATH
     "Path to the Scotch and PT-Scotch libraries.")
 set(Scotch_REQUIRED_PACKAGES "Threads" CACHE STRING

config/defaults.mk

@@ -21,8 +21,13 @@ NOTMAC := $(subst Darwin,,$(shell uname -s))
 CXX = g++
 MPICXX = mpicxx
 
-OPTIM_FLAGS = -O3
-DEBUG_FLAGS = -g -Wall
+BASE_FLAGS  = -std=c++11
+OPTIM_FLAGS = -O3 $(BASE_FLAGS)
+DEBUG_FLAGS = -g $(XCOMPILER)-Wall $(BASE_FLAGS)
+
+# Prefixes for passing flags to the compiler and linker when using CXX or MPICXX
+CXX_XCOMPILER =
+CXX_XLINKER   = -Wl,
 
 # Destination location of make install
 # PREFIX = $(HOME)/mfem
@@ -33,33 +38,41 @@ INSTALL = /usr/bin/install
 STATIC = YES
 SHARED = NO
 
+# CUDA configuration options
+CUDA_CXX = nvcc
+CUDA_ARCH = sm_60
+CUDA_FLAGS = -x=cu --expt-extended-lambda -arch=$(CUDA_ARCH)
+# Prefixes for passing flags to the host compiler and linker when using CUDA_CXX
+CUDA_XCOMPILER = -Xcompiler=
+CUDA_XLINKER   = -Xlinker=
+
 ifneq ($(NOTMAC),)
    AR      = ar
    ARFLAGS = cruv
    RANLIB  = ranlib
-   PICFLAG = -fPIC
+   PICFLAG = $(XCOMPILER)-fPIC
    SO_EXT  = so
    SO_VER  = so.$(MFEM_VERSION_STRING)
-   BUILD_SOFLAGS = -shared -Wl,-soname,libmfem.$(SO_VER)
-   BUILD_RPATH = -Wl,-rpath,$(BUILD_REAL_DIR)
+   BUILD_SOFLAGS = -shared $(XLINKER)-soname,libmfem.$(SO_VER)
+   BUILD_RPATH = $(XLINKER)-rpath,$(BUILD_REAL_DIR)
    INSTALL_SOFLAGS = $(BUILD_SOFLAGS)
-   INSTALL_RPATH = -Wl,-rpath,@MFEM_LIB_DIR@
+   INSTALL_RPATH = $(XLINKER)-rpath,@MFEM_LIB_DIR@
 else
    # Silence "has no symbols" warnings on Mac OS X
    AR      = ar
    ARFLAGS = Scruv
    RANLIB  = ranlib -no_warning_for_no_symbols
-   PICFLAG = -fPIC
+   PICFLAG = $(XCOMPILER)-fPIC
    SO_EXT  = dylib
    SO_VER  = $(MFEM_VERSION_STRING).dylib
-   MAKE_SOFLAGS = -Wl,-dylib,-install_name,$(1)/libmfem.$(SO_VER),\
+   MAKE_SOFLAGS = $(XLINKER)-dylib,-install_name,$(1)/libmfem.$(SO_VER),\
       -compatibility_version,$(MFEM_VERSION_STRING),\
       -current_version,$(MFEM_VERSION_STRING),\
       -undefined,dynamic_lookup
    BUILD_SOFLAGS = $(subst $1 ,,$(call MAKE_SOFLAGS,$(BUILD_REAL_DIR)))
-   BUILD_RPATH = -Wl,-undefined,dynamic_lookup
+   BUILD_RPATH = $(XLINKER)-undefined,dynamic_lookup
    INSTALL_SOFLAGS = $(subst $1 ,,$(call MAKE_SOFLAGS,$(MFEM_LIB_DIR)))
-   INSTALL_RPATH = -Wl,-undefined,dynamic_lookup
+   INSTALL_RPATH = $(XLINKER)-undefined,dynamic_lookup
 endif
 
 # Set CXXFLAGS to overwrite the default selection of DEBUG_FLAGS/OPTIM_FLAGS
@@ -82,31 +95,36 @@ MFEM_MPI_NP = 4
 # config.hpp. The values below are the defaults for generating the actual values
 # in config.mk and config.hpp.
 
-MFEM_USE_MPI         = NO
-MFEM_USE_METIS       = $(MFEM_USE_MPI)
-MFEM_USE_METIS_5     = NO
-MFEM_DEBUG           = NO
-MFEM_USE_EXCEPTIONS  = NO
-MFEM_USE_GZSTREAM    = NO
-MFEM_USE_LIBUNWIND   = NO
-MFEM_USE_LAPACK      = NO
-MFEM_THREAD_SAFE     = NO
-MFEM_USE_OPENMP      = NO
-MFEM_USE_MEMALLOC    = YES
-MFEM_TIMER_TYPE      = $(if $(NOTMAC),2,4)
-MFEM_USE_SUNDIALS    = NO
-MFEM_USE_MESQUITE    = NO
-MFEM_USE_SUITESPARSE = NO
-MFEM_USE_SUPERLU     = NO
-MFEM_USE_STRUMPACK   = NO
-MFEM_USE_GECKO       = NO
-MFEM_USE_GNUTLS      = NO
-MFEM_USE_NETCDF      = NO
-MFEM_USE_PETSC       = NO
-MFEM_USE_MPFR        = NO
-MFEM_USE_SIDRE       = NO
-MFEM_USE_CONDUIT     = NO
-MFEM_USE_PUMI        = NO
+MFEM_USE_MPI           = NO
+MFEM_USE_METIS         = $(MFEM_USE_MPI)
+MFEM_USE_METIS_5       = NO
+MFEM_DEBUG             = NO
+MFEM_USE_EXCEPTIONS    = NO
+MFEM_USE_GZSTREAM      = NO
+MFEM_USE_LIBUNWIND     = NO
+MFEM_USE_LAPACK        = NO
+MFEM_THREAD_SAFE       = NO
+MFEM_USE_OPENMP        = NO
+MFEM_USE_LEGACY_OPENMP = NO
+MFEM_USE_MEMALLOC      = YES
+MFEM_TIMER_TYPE        = $(if $(NOTMAC),2,4)
+MFEM_USE_SUNDIALS      = NO
+MFEM_USE_MESQUITE      = NO
+MFEM_USE_SUITESPARSE   = NO
+MFEM_USE_SUPERLU       = NO
+MFEM_USE_STRUMPACK     = NO
+MFEM_USE_GECKO         = NO
+MFEM_USE_GNUTLS        = NO
+MFEM_USE_NETCDF        = NO
+MFEM_USE_PETSC         = NO
+MFEM_USE_MPFR          = NO
+MFEM_USE_SIDRE         = NO
+MFEM_USE_CONDUIT       = NO
+MFEM_USE_PUMI          = NO
+MFEM_USE_CUDA          = NO
+MFEM_USE_RAJA          = NO
+MFEM_USE_OCCA          = NO
+MFEM_USE_MM            = NO
 
 # Compile and link options for zlib.
 ZLIB_DIR =
@@ -136,6 +154,8 @@ ifeq ($(MFEM_USE_SUPERLU)$(MFEM_USE_STRUMPACK),NONO)
 else
    # ParMETIS: currently needed by SuperLU or STRUMPACK. We assume that METIS 5
    # (included with ParMETIS) is installed in the same location.
+   # Starting with STRUMPACK v2.2.0, ParMETIS is an optional dependency while
+   # METIS is still required.
    METIS_DIR = @MFEM_DIR@/../parmetis-4.0.3
    METIS_OPT = -I$(METIS_DIR)/include
    METIS_LIB = -L$(METIS_DIR)/lib -lparmetis -lmetis
@@ -147,7 +167,7 @@ LAPACK_OPT =
 LAPACK_LIB = $(if $(NOTMAC),-llapack -lblas,-framework Accelerate)
 
 # OpenMP configuration
-OPENMP_OPT = -fopenmp
+OPENMP_OPT = $(XCOMPILER)-fopenmp
 OPENMP_LIB =
 
 # Used when MFEM_TIMER_TYPE = 2
@@ -183,7 +203,8 @@ SUPERLU_DIR = @MFEM_DIR@/../SuperLU_DIST_5.1.0
 SUPERLU_OPT = -I$(SUPERLU_DIR)/SRC
 SUPERLU_LIB = -Wl,-rpath,$(SUPERLU_DIR)/SRC -L$(SUPERLU_DIR)/SRC -lsuperlu_dist
 
-# SCOTCH library configuration (required by STRUMPACK)
+# SCOTCH library configuration (required by STRUMPACK <= v2.1.0, optional in
+# STRUMPACK >= v2.2.0)
 SCOTCH_DIR = @MFEM_DIR@/../scotch_6.0.4
 SCOTCH_OPT = -I$(SCOTCH_DIR)/include
 SCOTCH_LIB = -L$(SCOTCH_DIR)/lib -lptscotch -lptscotcherr -lscotch -lscotcherr\
@@ -279,6 +300,25 @@ PUMI_OPT = -I$(PUMI_DIR)/include
 PUMI_LIB = -L$(PUMI_DIR)/lib -lpumi -lcrv -lma -lmds -lapf -lpcu -lgmi -lparma\
    -llion -lmth -lapf_zoltan -lspr
 
+# CUDA library configuration. Since we compile and link with nvcc (when CUDA is
+# enabled) we only need to explicitly link with the CUDA driver, libcuda.*,
+# which is usually in a system path.
+CUDA_OPT =
+CUDA_LIB = $(if $(NOTMAC),,-L/usr/local/cuda/lib) -lcuda
+
+# OCCA library configuration
+OCCA_DIR ?= @MFEM_DIR@/../occa
+OCCA_OPT = -I$(OCCA_DIR)/include
+OCCA_LIB = $(XLINKER)-rpath,$(OCCA_DIR)/lib -L$(OCCA_DIR)/lib -locca
+
+# RAJA library configuration
+RAJA_DIR ?= @MFEM_DIR@/../raja
+RAJA_OPT = -I$(RAJA_DIR)/include
+ifdef CUB_DIR
+   RAJA_OPT += -I$(CUB_DIR)
+endif
+RAJA_LIB = $(XLINKER)-rpath,$(RAJA_DIR)/lib -L$(RAJA_DIR)/lib -lRAJA
+
 # If YES, enable some informational messages
 VERBOSE = NO
 

data/fichera.mesh

@@ -19,10 +19,10 @@ elements
 1 5 0 1 4 3 9 10 13 12
 1 5 3 4 7 6 12 13 16 15
 1 5 2 3 6 5 11 12 15 14
-1 5 8 9 12 11 17 18 21 20
-1 5 9 10 13 12 18 19 22 21
-1 5 12 13 16 15 21 22 25 24
 1 5 11 12 15 14 20 21 24 23
+1 5 12 13 16 15 21 22 25 24
+1 5 9 10 13 12 18 19 22 21
+1 5 8 9 12 11 17 18 21 20
 
 boundary
 24

doc/CodeDocumentation.dox

@@ -124,6 +124,7 @@ namespace mfem {
  * - <a class="el" href="display-basis_8cpp_source.html">Display Basis</a>: visualize finite element basis functions
  * - <a class="el" href="load-dc_8cpp_source.html">Load DC</a>: visualize fields saved via DataCollection classes
  * - <a class="el" href="convert-dc_8cpp_source.html">Convert DC</a>: convert between diffirent DataCollection formats
+ * - <a class="el" href="lor-transfer_8cpp_source.html">LOR Transfer</a>: map functions between high-order and low-order refined spaces
  * - <a class="el" href="miniapps_2performance_2ex1_8cpp_source.html">HPC Example 1</a>: high-performance nodal H1 FEM for the Laplace problem
  * - <a class="el" href="miniapps_2performance_2ex1p_8cpp_source.html">HPC Example 1p</a>: high-performance parallel nodal H1 FEM for the Laplace problem
  *

examples/CMakeLists.txt

@@ -27,6 +27,7 @@ list(APPEND ALL_EXE_SRCS
   ex18.cpp
   ex19.cpp
   ex20.cpp
+  ex21.cpp
   ex22.cpp
   )
 
@@ -52,6 +53,7 @@ if (MFEM_USE_MPI)
     ex18p.cpp
     ex19p.cpp
     ex20p.cpp
+    ex21p.cpp
     ex22p.cpp
     )
 endif()
@@ -86,6 +88,15 @@ foreach(SRC_FILE ${ALL_EXE_SRCS})
   endif()
 endforeach()
 
+# If STRUMPACK is enabled, add a test run that uses it.
+if (MFEM_USE_STRUMPACK)
+  add_test(NAME ex11p_strumpack_np=4
+    COMMAND ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} ${MFEM_MPI_NP}
+    ${MPIEXEC_PREFLAGS}
+    $<TARGET_FILE:ex11p> "-no-vis" "--strumpack"
+    ${MPIEXEC_POSTFLAGS})
+endif()
+
 # Include the examples/sundials directory if SUNDIALS is enabled.
 if (MFEM_USE_SUNDIALS)
   add_subdirectory(sundials)

examples/ex1.cpp

@@ -25,6 +25,14 @@
 //               ex1 -m ../data/mobius-strip.mesh
 //               ex1 -m ../data/mobius-strip.mesh -o -1 -sc
 //
+// Device sample runs:
+//             > ex1 -pa -d cuda
+//             > ex1 -pa -d raja-cuda
+//             > ex1 -pa -d occa-cuda
+//             > ex1 -pa -d raja-omp
+//             > ex1 -pa -d occa-omp
+//             > ex1 -m ../data/beam-hex.mesh -pa -d cuda
+//
 // Description:  This example code demonstrates the use of MFEM to define a
 //               simple finite element discretization of the Laplace problem
 //               -Delta u = 1 with homogeneous Dirichlet boundary conditions.
@@ -53,7 +61,9 @@ int main(int argc, char *argv[])
    const char *mesh_file = "../data/star.mesh";
    int order = 1;
    bool static_cond = false;
-   bool visualization = 1;
+   bool pa = false;
+   const char *device = "cpu";
+   bool visualization = true;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
@@ -63,6 +73,10 @@ int main(int argc, char *argv[])
                   " isoparametric space.");
    args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
                   "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -134,48 +148,65 @@ int main(int argc, char *argv[])
    b->AddDomainIntegrator(new DomainLFIntegrator(one));
    b->Assemble();
 
-   // 7. Define the solution vector x as a finite element grid function
+   // 7. Set device config parameters from the command line options and switch
+   //    to working on the device.
+   Device::Configure(device);
+   Device::Print();
+   Device::Enable();
+
+   // 8. Define the solution vector x as a finite element grid function
    //    corresponding to fespace. Initialize x with initial guess of zero,
    //    which satisfies the boundary conditions.
    GridFunction x(fespace);
    x = 0.0;
 
-   // 8. Set up the bilinear form a(.,.) on the finite element space
+   // 9. Set up the bilinear form a(.,.) on the finite element space
    //    corresponding to the Laplacian operator -Delta, by adding the Diffusion
    //    domain integrator.
    BilinearForm *a = new BilinearForm(fespace);
+   if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    a->AddDomainIntegrator(new DiffusionIntegrator(one));
 
-   // 9. Assemble the bilinear form and the corresponding linear system,
-   //    applying any necessary transformations such as: eliminating boundary
-   //    conditions, applying conforming constraints for non-conforming AMR,
-   //    static condensation, etc.
+   // 10. Assemble the bilinear form and the corresponding linear system,
+   //     applying any necessary transformations such as: eliminating boundary
+   //     conditions, applying conforming constraints for non-conforming AMR,
+   //     static condensation, etc.
    if (static_cond) { a->EnableStaticCondensation(); }
    a->Assemble();
 
-   SparseMatrix A;
+   OperatorPtr A;
    Vector B, X;
    a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
 
-   cout << "Size of linear system: " << A.Height() << endl;
+   cout << "Size of linear system: " << A->Height() << endl;
 
+   // 11. Solve the linear system A X = B.
+   if (!pa)
+   {
 #ifndef MFEM_USE_SUITESPARSE
-   // 10. Define a simple symmetric Gauss-Seidel preconditioner and use it to
-   //     solve the system A X = B with PCG.
-   GSSmoother M(A);
-   PCG(A, M, B, X, 1, 200, 1e-12, 0.0);
+      // Use a simple symmetric Gauss-Seidel preconditioner with PCG.
+      GSSmoother M((SparseMatrix&)(*A));
+      PCG(*A, M, B, X, 1, 200, 1e-12, 0.0);
 #else
-   // 10. If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.
-   UMFPackSolver umf_solver;
-   umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
-   umf_solver.SetOperator(A);
-   umf_solver.Mult(B, X);
+      // If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.
+      UMFPackSolver umf_solver;
+      umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+      umf_solver.SetOperator(*A);
+      umf_solver.Mult(B, X);
 #endif
+   }
+   else // No preconditioning for now in partial assembly mode.
+   {
+      CG(*A, B, X, 1, 2000, 1e-12, 0.0);
+   }
 
-   // 11. Recover the solution as a finite element grid function.
+   // 12. Recover the solution as a finite element grid function.
    a->RecoverFEMSolution(X, *b, x);
 
-   // 12. Save the refined mesh and the solution. This output can be viewed later
+   // 13. Switch back to the host.
+   Device::Disable();
+
+   // 14. Save the refined mesh and the solution. This output can be viewed later
    //     using GLVis: "glvis -m refined.mesh -g sol.gf".
    ofstream mesh_ofs("refined.mesh");
    mesh_ofs.precision(8);
@@ -184,7 +215,7 @@ int main(int argc, char *argv[])
    sol_ofs.precision(8);
    x.Save(sol_ofs);
 
-   // 13. Send the solution by socket to a GLVis server.
+   // 15. Send the solution by socket to a GLVis server.
    if (visualization)
    {
       char vishost[] = "localhost";
@@ -194,7 +225,7 @@ int main(int argc, char *argv[])
       sol_sock << "solution\n" << *mesh << x << flush;
    }
 
-   // 14. Free the used memory.
+   // 16. Free the used memory.
    delete a;
    delete b;
    delete fespace;

examples/ex11p.cpp

@@ -261,8 +261,7 @@ int main(int argc, char *argv[])
          strumpack->SetPrintSolveStatistics(false);
          strumpack->SetKrylovSolver(strumpack::KrylovSolver::DIRECT);
          strumpack->SetReorderingStrategy(strumpack::ReorderingStrategy::METIS);
-         strumpack->SetMC64Job(strumpack::MC64Job::NONE);
-         // strumpack->SetSymmetricPattern(true);
+         strumpack->DisableMatching();
          strumpack->SetOperator(*Arow);
          strumpack->SetFromCommandLine();
          precond = strumpack;

examples/ex1p.cpp

@@ -25,6 +25,11 @@
 //               mpirun -np 4 ex1p -m ../data/mobius-strip.mesh
 //               mpirun -np 4 ex1p -m ../data/mobius-strip.mesh -o -1 -sc
 //
+// Device sample runs:
+//             > mpirun -np 4 ex1p -pa -d cuda
+//             > mpirun -np 4 ex1p -pa -d occa-cuda
+//             > mpirun -np 4 ex1p -pa -d raja-omp
+//
 // Description:  This example code demonstrates the use of MFEM to define a
 //               simple finite element discretization of the Laplace problem
 //               -Delta u = 1 with homogeneous Dirichlet boundary conditions.
@@ -59,7 +64,9 @@ int main(int argc, char *argv[])
    const char *mesh_file = "../data/star.mesh";
    int order = 1;
    bool static_cond = false;
-   bool visualization = 1;
+   bool pa = false;
+   const char *device = "cpu";
+   bool visualization = true;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
@@ -69,6 +76,10 @@ int main(int argc, char *argv[])
                   " isoparametric space.");
    args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
                   "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -166,49 +177,58 @@ int main(int argc, char *argv[])
    b->AddDomainIntegrator(new DomainLFIntegrator(one));
    b->Assemble();
 
-   // 9. Define the solution vector x as a parallel finite element grid function
-   //    corresponding to fespace. Initialize x with initial guess of zero,
-   //    which satisfies the boundary conditions.
+   // 9. Set device config parameters from the command line options and switch
+   //    to working on the device.
+   Device::Configure(device);
+   if (myid == 0) { Device::Print(); }
+   Device::Enable();
+
+   // 10. Define the solution vector x as a parallel finite element grid function
+   //     corresponding to fespace. Initialize x with initial guess of zero,
+   //     which satisfies the boundary conditions.
    ParGridFunction x(fespace);
    x = 0.0;
 
-   // 10. Set up the parallel bilinear form a(.,.) on the finite element space
+   // 11. Set up the parallel bilinear form a(.,.) on the finite element space
    //     corresponding to the Laplacian operator -Delta, by adding the Diffusion
    //     domain integrator.
    ParBilinearForm *a = new ParBilinearForm(fespace);
+   if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    a->AddDomainIntegrator(new DiffusionIntegrator(one));
 
-   // 11. Assemble the parallel bilinear form and the corresponding linear
+   // 12. Assemble the parallel bilinear form and the corresponding linear
    //     system, applying any necessary transformations such as: parallel
    //     assembly, eliminating boundary conditions, applying conforming
    //     constraints for non-conforming AMR, static condensation, etc.
    if (static_cond) { a->EnableStaticCondensation(); }
    a->Assemble();
 
-   HypreParMatrix A;
+   OperatorPtr A;
    Vector B, X;
    a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
 
-   if (myid == 0)
-   {
-      cout << "Size of linear system: " << A.GetGlobalNumRows() << endl;
-   }
+   // 13. Solve the linear system A X = B.
+   //     * With full assembly, use the BoomerAMG preconditioner from hypre.
+   //     * With partial assembly, use no preconditioner, for now.
+   Solver *prec = NULL;
+   if (!pa) { prec = new HypreBoomerAMG; }
+   CGSolver cg(MPI_COMM_WORLD);
+   cg.SetRelTol(1e-12);
+   cg.SetMaxIter(2000);
+   cg.SetPrintLevel(1);
+   if (prec) { cg.SetPreconditioner(*prec); }
+   cg.SetOperator(*A);
+   cg.Mult(B, X);
+   delete prec;
 
-   // 12. Define and apply a parallel PCG solver for AX=B with the BoomerAMG
-   //     preconditioner from hypre.
-   HypreSolver *amg = new HypreBoomerAMG(A);
-   HyprePCG *pcg = new HyprePCG(A);
-   pcg->SetTol(1e-12);
-   pcg->SetMaxIter(200);
-   pcg->SetPrintLevel(2);
-   pcg->SetPreconditioner(*amg);
-   pcg->Mult(B, X);
-
-   // 13. Recover the parallel grid function corresponding to X. This is the
+   // 14. Recover the parallel grid function corresponding to X. This is the
    //     local finite element solution on each processor.
    a->RecoverFEMSolution(X, *b, x);
 
-   // 14. Save the refined mesh and the solution in parallel. This output can
+   // 15. Switch back to the host.
+   Device::Disable();
+
+   // 16. Save the refined mesh and the solution in parallel. This output can
    //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
    {
       ostringstream mesh_name, sol_name;
@@ -224,7 +244,7 @@ int main(int argc, char *argv[])
       x.Save(sol_ofs);
    }
 
-   // 15. Send the solution by socket to a GLVis server.
+   // 17. Send the solution by socket to a GLVis server.
    if (visualization)
    {
       char vishost[] = "localhost";
@@ -235,9 +255,7 @@ int main(int argc, char *argv[])
       sol_sock << "solution\n" << *pmesh << x << flush;
    }
 
-   // 16. Free the used memory.
-   delete pcg;
-   delete amg;
+   // 18. Free the used memory.
    delete a;
    delete b;
    delete fespace;

examples/ex21.cpp

@@ -0,0 +1,477 @@
+//                               MFEM Example 21
+//
+// Compile with: make ex21
+//
+// Sample runs:  ex21 -m ../data/inline-segment.mesh -o 3
+//               ex21 -m ../data/inline-tri.mesh -o 3
+//               ex21 -m ../data/inline-quad.mesh -o 3
+//               ex21 -m ../data/inline-quad.mesh -o 3 -p 1
+//               ex21 -m ../data/inline-quad.mesh -o 3 -p 2
+//               ex21 -m ../data/inline-tet.mesh -o 2
+//               ex21 -m ../data/inline-hex.mesh -o 2
+//               ex21 -m ../data/inline-hex.mesh -o 2 -p 1
+//               ex21 -m ../data/inline-hex.mesh -o 2 -p 2
+//               ex21 -m ../data/star.mesh -o 2 -sigma 10.0
+//
+// Description:  This example code demonstrates the use of MFEM to define and
+//               solve simple complex-valued linear systems.  We implement three
+//               variants of a damped harmonic oscillator:
+//
+//               1) A scalar H1 field
+//                  -Div(a Grad u) - omega^2 b u + i omega c u = 0
+//
+//               2) A vector H(Curl) field
+//                  Curl(a Curl u) - omega^2 b u + i omega c u = 0
+//
+//               3) A vector H(Div) field
+//                  -Grad(a Div u) - omega^2 b u + i omega c u = 0
+//
+//               In each case the field is driven by a forced oscillation, with
+//               angular frequency omega, imposed at the boundary or a portion
+//               of the boundary.
+//
+//               In electromagnetics the coefficients are typically named the
+//               permeability, mu = 1/a, permittivity, epsilon = b, and
+//               conductivity, sigma = c.  The user can specify these constants
+//               using either set of names.
+//
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+static double mu_ = 1.0;
+static double epsilon_ = 1.0;
+static double sigma_ = 20.0;
+static double omega_ = 10.0;
+
+double u0_real_exact(const Vector &);
+double u0_imag_exact(const Vector &);
+
+void u1_real_exact(const Vector &, Vector &);
+void u1_imag_exact(const Vector &, Vector &);
+
+void u2_real_exact(const Vector &, Vector &);
+void u2_imag_exact(const Vector &, Vector &);
+
+bool check_for_inline_mesh(const char * mesh_file);
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options.
+   const char *mesh_file = "../data/inline-quad.mesh";
+   int ref_levels = 0;
+   int order = 1;
+   int prob = 0;
+   double freq = -1.0;
+   double a_coef = 0.0;
+   bool visualization = 1;
+   bool herm_conv = true;
+   bool exact_sol = true;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&ref_levels, "-r", "--refine",
+                  "Number of times to refine the mesh uniformly.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree).");
+   args.AddOption(&prob, "-p", "--problem-type",
+                  "Choose from 0: H_1, 1: H(Curl), or 2: H(Div) "
+                  "damped harmonic oscillator.");
+   args.AddOption(&a_coef, "-a", "--stiffness-coef",
+                  "Stiffness coefficient (spring constant or 1/mu).");
+   args.AddOption(&epsilon_, "-b", "--mass-coef",
+                  "Mass coefficient (or epsilon).");
+   args.AddOption(&sigma_, "-c", "--damping-coef",
+                  "Damping coefficient (or sigma).");
+   args.AddOption(&mu_, "-mu", "--permeability",
+                  "Permeability of free space (or 1/(spring constant)).");
+   args.AddOption(&epsilon_, "-eps", "--permittivity",
+                  "Permittivity of free space (or mass constant).");
+   args.AddOption(&sigma_, "-sigma", "--conductivity",
+                  "Conductivity (or damping constant).");
+   args.AddOption(&freq, "-f", "--frequency",
+                  "Frequency (in Hz).");
+   args.AddOption(&herm_conv, "-herm", "--hermitian", "-no-herm",
+                  "--no-hermitian", "Use convention for Hermitian operators.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   if ( a_coef != 0.0 )
+   {
+      mu_ = 1.0 / a_coef;
+   }
+   if ( freq > 0.0 )
+   {
+      omega_ = 2.0 * M_PI * freq;
+   }
+
+   exact_sol = check_for_inline_mesh(mesh_file);
+   if (exact_sol)
+   {
+      cout << "Identified an 'inline' mesh" << endl;
+   }
+
+   ComplexOperator::Convention conv =
+      herm_conv ? ComplexOperator::HERMITIAN : ComplexOperator::BLOCK_SYMMETRIC;
+
+   // 2. Read the mesh from the given mesh file. We can handle triangular,
+   //    quadrilateral, tetrahedral, hexahedral, surface and volume meshes
+   //    with the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 3. Refine the mesh to increase resolution. In this example we do
+   //    'ref_levels' of uniform refinement where the user specifies
+   //    the number of levels with the '-r' option.
+   for (int l = 0; l < ref_levels; l++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   // 4. Define a finite element space on the mesh. Here we use continuous
+   //    Lagrange, Nedelec, or Raviart-Thomas finite elements of the specified
+   //    order.
+   if (dim == 1 && prob != 0 )
+   {
+      cout << "Switching to problem type 0, H1 basis functions, "
+           << "for 1 dimensional mesh." << endl;
+      prob = 0;
+   }
+
+   FiniteElementCollection *fec;
+   switch (prob)
+   {
+      case 0:  fec = new H1_FECollection(order, dim);      break;
+      case 1:  fec = new ND_FECollection(order, dim);      break;
+      case 2:  fec = new RT_FECollection(order - 1, dim);  break;
+   }
+   FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
+   cout << "Number of finite element unknowns: " << fespace->GetTrueVSize()
+        << endl;
+
+   // 5. Determine the list of true (i.e. conforming) essential boundary dofs.
+   //    In this example, the boundary conditions are defined based on the type
+   //    of mesh and the problem type.
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr;
+   if (mesh->bdr_attributes.Size())
+   {
+      ess_bdr.SetSize(mesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      if (exact_sol)
+      {
+         switch (prob)
+         {
+            case 0:   ess_bdr = 0; ess_bdr[0] = 1;  break;
+            default:  ess_bdr = 1; ess_bdr[2] = 0;  break;
+         }
+      }
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   // 6. Set up the linear form b(.) which corresponds to the
+   //    right-hand side of the FEM linear system.
+   ComplexLinearForm b(fespace, conv);
+   b.Vector::operator=(0.0);
+
+   // 7. Define the solution vector u as a finite element grid function
+   //    corresponding to fespace. Initialize u with initial guess of 1+0i
+   //    or the exact solution if it is known.
+   ComplexGridFunction u(fespace);
+   ComplexGridFunction * u_exact = NULL;
+   if (exact_sol) { u_exact = new ComplexGridFunction(fespace); }
+
+   FunctionCoefficient u0_r(u0_real_exact);
+   FunctionCoefficient u0_i(u0_imag_exact);
+   VectorFunctionCoefficient u1_r(dim, u1_real_exact);
+   VectorFunctionCoefficient u1_i(dim, u1_imag_exact);
+   VectorFunctionCoefficient u2_r(dim, u2_real_exact);
+   VectorFunctionCoefficient u2_i(dim, u2_imag_exact);
+
+   ConstantCoefficient zeroCoef(0.0);
+   ConstantCoefficient oneCoef(1.0);
+
+   Vector zeroVec(dim); zeroVec = 0.0;
+   Vector  oneVec(dim);  oneVec = 0.0; oneVec[(prob==2)?(dim-1):0] = 1.0;
+   VectorConstantCoefficient zeroVecCoef(zeroVec);
+   VectorConstantCoefficient oneVecCoef(oneVec);
+
+   switch (prob)
+   {
+      case 0:
+         u.ProjectBdrCoefficient(oneCoef, zeroCoef, ess_bdr);
+         if (exact_sol) { u_exact->ProjectCoefficient(u0_r, u0_i); }
+         break;
+      case 1:
+         u.ProjectBdrCoefficientTangent(oneVecCoef, zeroVecCoef, ess_bdr);
+         if (exact_sol) { u_exact->ProjectCoefficient(u1_r, u1_i); }
+         break;
+      case 2:
+         u.ProjectBdrCoefficientNormal(oneVecCoef, zeroVecCoef, ess_bdr);
+         if (exact_sol) { u_exact->ProjectCoefficient(u2_r, u2_i); }
+         break;
+   }
+
+   if (visualization && exact_sol)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock_r(vishost, visport);
+      socketstream sol_sock_i(vishost, visport);
+      sol_sock_r.precision(8);
+      sol_sock_i.precision(8);
+      sol_sock_r << "solution\n" << *mesh << u_exact->real()
+                 << "window_title 'Exact Real Part'" << flush;
+      sol_sock_i << "solution\n" << *mesh << u_exact->imag()
+                 << "window_title 'Exact Imaginary Part'" << flush;
+   }
+
+   // 8. Set up the sesquilinear form a(.,.) on the finite element
+   //    space corresponding to the damped harmonic oscillator operator
+   //    of the appropriate type:
+   //
+   //    0) A scalar H1 field
+   //       -Div(a Grad) - omega^2 b + i omega c
+   //
+   //    1) A vector H(Curl) field
+   //       Curl(a Curl) - omega^2 b + i omega c
+   //
+   //    2) A vector H(Div) field
+   //       -Grad(a Div) - omega^2 b + i omega c
+   //
+   ConstantCoefficient stiffnessCoef(1.0/mu_);
+   ConstantCoefficient massCoef(-omega_ * omega_ * epsilon_);
+   ConstantCoefficient lossCoef(omega_ * sigma_);
+   ConstantCoefficient negMassCoef(omega_ * omega_ * epsilon_);
+
+   SesquilinearForm *a = new SesquilinearForm(fespace, conv);
+   switch (prob)
+   {
+      case 0:
+         a->AddDomainIntegrator(new DiffusionIntegrator(stiffnessCoef),
+                                NULL);
+         a->AddDomainIntegrator(new MassIntegrator(massCoef),
+                                new MassIntegrator(lossCoef));
+         break;
+      case 1:
+         a->AddDomainIntegrator(new CurlCurlIntegrator(stiffnessCoef),
+                                NULL);
+         a->AddDomainIntegrator(new VectorFEMassIntegrator(massCoef),
+                                new VectorFEMassIntegrator(lossCoef));
+         break;
+      case 2:
+         a->AddDomainIntegrator(new DivDivIntegrator(stiffnessCoef),
+                                NULL);
+         a->AddDomainIntegrator(new VectorFEMassIntegrator(massCoef),
+                                new VectorFEMassIntegrator(lossCoef));
+         break;
+   }
+
+   // 9. Assemble the bilinear form and the corresponding linear
+   //    system, applying any necessary transformations such as:
+   //    assembly, eliminating boundary conditions, applying conforming
+   //    constraints for non-conforming AMR, etc.
+   a->Assemble();
+
+   OperatorHandle A;
+   Vector B, U;
+
+   a->FormLinearSystem(ess_tdof_list, u, b, A, U, B);
+   u = 0.0;
+   U = 0.0;
+
+   {
+      ComplexSparseMatrix * Asp =
+         dynamic_cast<ComplexSparseMatrix*>(A.Ptr());
+
+      cout << "Size of linear system: "
+           << 2 * Asp->real().Width() << endl << endl;
+   }
+
+   // 10. Define and apply a GMRES solver for AU=B.
+   {
+      GMRESSolver gmres;
+      gmres.SetOperator(*A.Ptr());
+      gmres.SetRelTol(1e-12);
+      gmres.SetMaxIter(1000);
+      gmres.SetPrintLevel(1);
+      gmres.Mult(B, U);
+   }
+
+   // 11. Recover the solution as a finite element grid function and
+   //     compute the errors if the exact solution is known.
+   a->RecoverFEMSolution(U, b, u);
+
+   if (exact_sol)
+   {
+      double err_r = -1.0;
+      double err_i = -1.0;
+
+      switch (prob)
+      {
+         case 0:
+            err_r = u.real().ComputeL2Error(u0_r);
+            err_i = u.imag().ComputeL2Error(u0_i);
+            break;
+         case 1:
+            err_r = u.real().ComputeL2Error(u1_r);
+            err_i = u.imag().ComputeL2Error(u1_i);
+            break;
+         case 2:
+            err_r = u.real().ComputeL2Error(u2_r);
+            err_i = u.imag().ComputeL2Error(u2_i);
+            break;
+      }
+
+      cout << endl;
+      cout << "|| Re (u_h - u) ||_{L^2} = " << err_r << endl;
+      cout << "|| Im (u_h - u) ||_{L^2} = " << err_i << endl;
+      cout << endl;
+   }
+
+   // 12. Save the refined mesh and the solution. This output can be
+   //     viewed later using GLVis: "glvis -m mesh -g sol".
+   {
+      ofstream mesh_ofs("refined.mesh");
+      mesh_ofs.precision(8);
+      mesh->Print(mesh_ofs);
+
+      ofstream sol_r_ofs("sol_r.gf");
+      ofstream sol_i_ofs("sol_i.gf");
+      sol_r_ofs.precision(8);
+      sol_i_ofs.precision(8);
+      u.real().Save(sol_r_ofs);
+      u.imag().Save(sol_i_ofs);
+   }
+
+   // 13. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock_r(vishost, visport);
+      socketstream sol_sock_i(vishost, visport);
+      sol_sock_r.precision(8);
+      sol_sock_i.precision(8);
+      sol_sock_r << "solution\n" << *mesh << u.real()
+                 << "window_title 'Comp Real Part'" << flush;
+      sol_sock_i << "solution\n" << *mesh << u.imag()
+                 << "window_title 'Comp Imaginary Part'" << flush;
+   }
+   if (visualization && exact_sol)
+   {
+      *u_exact -= u;
+
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock_r(vishost, visport);
+      socketstream sol_sock_i(vishost, visport);
+      sol_sock_r.precision(8);
+      sol_sock_i.precision(8);
+      sol_sock_r << "solution\n" << *mesh << u_exact->real()
+                 << "window_title 'Exact-Comp Real Part'" << flush;
+      sol_sock_i << "solution\n" << *mesh << u_exact->imag()
+                 << "window_title 'Exact-Comp Imaginary Part'" << flush;
+   }
+   if (visualization)
+   {
+      GridFunction u_t(fespace);
+      u_t = u.real();
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *mesh << u_t
+               << "window_title 'Harmonic Solution (t = 0.0 T)'"
+               << "pause\n" << flush;
+
+      cout << "GLVis visualization paused."
+           << " Press space (in the GLVis window) to resume it.\n";
+      int num_frames = 32;
+      int i = 0;
+      while (sol_sock)
+      {
+         double t = (double)(i % num_frames) / num_frames;
+         ostringstream oss;
+         oss << "Harmonic Solution (t = " << t << " T)";
+
+         add(cos( 2.0 * M_PI * t), u.real(),
+             sin(-2.0 * M_PI * t), u.imag(), u_t);
+         sol_sock << "solution\n" << *mesh << u_t
+                  << "window_title '" << oss.str() << "'" << flush;
+         i++;
+      }
+   }
+
+   // 14. Free the used memory.
+   delete a;
+   delete u_exact;
+   delete fespace;
+   delete fec;
+   delete mesh;
+
+   return 0;
+}
+
+bool check_for_inline_mesh(const char * mesh_file)
+{
+   string file(mesh_file);
+   size_t p0 = file.find_last_of("/");
+   string s0 = file.substr((p0==string::npos)?0:(p0+1),7);
+   return s0 == "inline-";
+}
+
+complex<double> u0_exact(const Vector &x)
+{
+   int dim = x.Size();
+   complex<double> i(0.0, 1.0);
+   complex<double> alpha = (epsilon_ * omega_ - i * sigma_);
+   complex<double> kappa = std::sqrt(mu_ * omega_* alpha);
+   return std::exp(-i * kappa * x[dim - 1]);
+}
+
+double u0_real_exact(const Vector &x)
+{
+   return u0_exact(x).real();
+}
+
+double u0_imag_exact(const Vector &x)
+{
+   return u0_exact(x).imag();
+}
+
+void u1_real_exact(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+   v.SetSize(dim); v = 0.0; v[0] = u0_real_exact(x);
+}
+
+void u1_imag_exact(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+   v.SetSize(dim); v = 0.0; v[0] = u0_imag_exact(x);
+}
+
+void u2_real_exact(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+   v.SetSize(dim); v = 0.0; v[dim-1] = u0_real_exact(x);
+}
+
+void u2_imag_exact(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+   v.SetSize(dim); v = 0.0; v[dim-1] = u0_imag_exact(x);
+}

examples/ex21p.cpp

@@ -0,0 +1,658 @@
+//                       MFEM Example 21 - Parallel Version
+//
+// Compile with: make ex21p
+//
+// Sample runs:  mpirun -np 4 ex21p -m ../data/inline-segment.mesh -o 3
+//               mpirun -np 4 ex21p -m ../data/inline-tri.mesh -o 3
+//               mpirun -np 4 ex21p -m ../data/inline-quad.mesh -o 3
+//               mpirun -np 4 ex21p -m ../data/inline-quad.mesh -o 3 -p 1
+//               mpirun -np 4 ex21p -m ../data/inline-quad.mesh -o 3 -p 2
+//               mpirun -np 4 ex21p -m ../data/inline-tet.mesh -o 2
+//               mpirun -np 4 ex21p -m ../data/inline-hex.mesh -o 2
+//               mpirun -np 4 ex21p -m ../data/inline-hex.mesh -o 2 -p 1
+//               mpirun -np 4 ex21p -m ../data/inline-hex.mesh -o 2 -p 2
+//               mpirun -np 4 ex21p -m ../data/star.mesh -o 2 -sigma 10.0
+//
+// Description:  This example code demonstrates the use of MFEM to define and
+//               solve simple complex-valued linear systems.  We implement three
+//               variants of a damped harmonic oscillator:
+//
+//               1) A scalar H1 field
+//                  -Div(a Grad u) - omega^2 b u + i omega c u = 0
+//
+//               2) A vector H(Curl) field
+//                  Curl(a Curl u) - omega^2 b u + i omega c u = 0
+//
+//               3) A vector H(Div) field
+//                  -Grad(a Div u) - omega^2 b u + i omega c u = 0
+//
+//               In each case the field is driven by a forced oscillation, with
+//               angular frequency omega, imposed at the boundary or a portion
+//               of the boundary.
+//
+//               In electromagnetics the coefficients are typically named the
+//               permeability, mu = 1/a, permittivity, epsilon = b, and
+//               conductivity, sigma = c.  The user can specify these constants
+//               using either set of names.
+//
+//#define MFEM_STRUMPACK_SRC
+
+#include <fstream>
+#include <iostream>
+#include "mfem.hpp"
+
+using namespace std;
+using namespace mfem;
+
+static double mu_ = 1.0;
+static double epsilon_ = 1.0;
+static double sigma_ = 20.0;
+static double omega_ = 10.0;
+
+double u0_real_exact(const Vector &);
+double u0_imag_exact(const Vector &);
+
+void u1_real_exact(const Vector &, Vector &);
+void u1_imag_exact(const Vector &, Vector &);
+
+void u2_real_exact(const Vector &, Vector &);
+void u2_imag_exact(const Vector &, Vector &);
+
+bool check_for_inline_mesh(const char * mesh_file);
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm comm = MPI_COMM_WORLD;
+   MPI_Comm_size(comm, &num_procs);
+   MPI_Comm_rank(comm, &myid);
+
+   // 2. Parse command-line options.
+   const char *mesh_file = "../data/inline-quad.mesh";
+   int ser_ref_levels = 1;
+   int par_ref_levels = 1;
+   int order = 1;
+   int prob = 0;
+   double freq = -1.0;
+   double a_coef = 0.0;
+   bool visualization = 1;
+   bool herm_conv = true;
+   bool exact_sol = true;
+#ifdef MFEM_USE_STRUMPACK
+   bool strumpack = false;
+#endif
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
+                  "Number of times to refine the mesh uniformly in serial.");
+   args.AddOption(&par_ref_levels, "-rp", "--refine-parallel",
+                  "Number of times to refine the mesh uniformly in parallel.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree).");
+   args.AddOption(&prob, "-p", "--problem-type",
+                  "Choose from 0: H_1, 1: H(Curl), or 2: H(Div) "
+                  "damped harmonic oscillator.");
+   args.AddOption(&a_coef, "-a", "--stiffness-coef",
+                  "Stiffness coefficient (spring constant or 1/mu).");
+   args.AddOption(&epsilon_, "-b", "--mass-coef",
+                  "Mass coefficient (or epsilon).");
+   args.AddOption(&sigma_, "-c", "--damping-coef",
+                  "Damping coefficient (or sigma).");
+   args.AddOption(&mu_, "-mu", "--permeability",
+                  "Permeability of free space (or 1/(spring constant)).");
+   args.AddOption(&epsilon_, "-eps", "--permittivity",
+                  "Permittivity of free space (or mass constant).");
+   args.AddOption(&sigma_, "-sigma", "--conductivity",
+                  "Conductivity (or damping constant).");
+   args.AddOption(&freq, "-f", "--frequency",
+                  "Frequency (in Hz).");
+#ifdef MFEM_USE_STRUMPACK
+   args.AddOption(&strumpack, "-strumpack", "--strumpack-solver",
+                  "-no-strumpack", "--no-strumpack-solver",
+                  "Use STRUMPACK's double complex linear solver.");
+#endif
+   args.AddOption(&herm_conv, "-herm", "--hermitian", "-no-herm",
+                  "--no-hermitian", "Use convention for Hermitian operators.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   if ( a_coef != 0.0 )
+   {
+      mu_ = 1.0 / a_coef;
+   }
+   if ( freq > 0.0 )
+   {
+      omega_ = 2.0 * M_PI * freq;
+   }
+
+   exact_sol = check_for_inline_mesh(mesh_file);
+   if (myid == 0 && exact_sol)
+   {
+      cout << "Identified an 'inline' mesh" << endl;
+   }
+
+   ComplexOperator::Convention conv =
+      herm_conv ? ComplexOperator::HERMITIAN : ComplexOperator::BLOCK_SYMMETRIC;
+
+   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
+   //    and volume meshes with the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 4. Refine the serial mesh on all processors to increase the resolution.
+   for (int l = 0; l < ser_ref_levels; l++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   for (int l = 0; l < par_ref_levels; l++)
+   {
+      pmesh->UniformRefinement();
+   }
+
+   // 6. Define a parallel finite element space on the parallel
+   //    mesh. Here we use continuous Lagrange, Nedelec, or
+   //    Raviart-Thomas finite elements of the specified order.
+   if (dim == 1 && prob != 0 )
+   {
+      if (myid == 0)
+      {
+         cout << "Switching to problem type 0, H1 basis functions, "
+              << "for 1 dimensional mesh." << endl;
+      }
+      prob = 0;
+   }
+
+   FiniteElementCollection *fec;
+   switch (prob)
+   {
+      case 0:  fec = new H1_FECollection(order, dim);      break;
+      case 1:  fec = new ND_FECollection(order, dim);      break;
+      case 2:  fec = new RT_FECollection(order - 1, dim);  break;
+   }
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   HYPRE_Int size = fespace->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of finite element unknowns: " << size << endl;
+   }
+
+   // 7. Determine the list of true (i.e. parallel conforming) essential
+   //    boundary dofs. In this example, the boundary conditions are defined
+   //    based on the type of mesh and the problem type.
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr;
+   if (pmesh->bdr_attributes.Size())
+   {
+      ess_bdr.SetSize(pmesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      if (exact_sol)
+      {
+         switch (prob)
+         {
+            case 0:   ess_bdr = 0; ess_bdr[0] = 1;  break;
+            default:  ess_bdr = 1; ess_bdr[2] = 0;  break;
+         }
+      }
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   // 8. Set up the parallel linear form b(.) which corresponds to the
+   //    right-hand side of the FEM linear system.
+   ParComplexLinearForm b(fespace, conv);
+   b.Vector::operator=(0.0);
+
+   // 9. Define the solution vector u as a parallel finite element
+   //    grid function corresponding to fespace. Initialize u with
+   //    initial guess of 1+0i or the exact solution if it is known.
+   ParComplexGridFunction u(fespace);
+   ParComplexGridFunction * u_exact = NULL;
+   if (exact_sol) { u_exact = new ParComplexGridFunction(fespace); }
+
+   FunctionCoefficient u0_r(u0_real_exact);
+   FunctionCoefficient u0_i(u0_imag_exact);
+   VectorFunctionCoefficient u1_r(dim, u1_real_exact);
+   VectorFunctionCoefficient u1_i(dim, u1_imag_exact);
+   VectorFunctionCoefficient u2_r(dim, u2_real_exact);
+   VectorFunctionCoefficient u2_i(dim, u2_imag_exact);
+
+   ConstantCoefficient zeroCoef(0.0);
+   ConstantCoefficient oneCoef(1.0);
+
+   Vector zeroVec(dim); zeroVec = 0.0;
+   Vector  oneVec(dim);  oneVec = 0.0; oneVec[(prob==2)?(dim-1):0] = 1.0;
+   VectorConstantCoefficient zeroVecCoef(zeroVec);
+   VectorConstantCoefficient oneVecCoef(oneVec);
+
+   switch (prob)
+   {
+      case 0:
+         u.ProjectBdrCoefficient(oneCoef, zeroCoef, ess_bdr);
+         if (exact_sol) { u_exact->ProjectCoefficient(u0_r, u0_i); }
+         break;
+      case 1:
+         u.ProjectBdrCoefficientTangent(oneVecCoef, zeroVecCoef, ess_bdr);
+         if (exact_sol) { u_exact->ProjectCoefficient(u1_r, u1_i); }
+         break;
+      case 2:
+         u.ProjectBdrCoefficientNormal(oneVecCoef, zeroVecCoef, ess_bdr);
+         if (exact_sol) { u_exact->ProjectCoefficient(u2_r, u2_i); }
+         break;
+   }
+
+   if (visualization && exact_sol)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock_r(vishost, visport);
+      socketstream sol_sock_i(vishost, visport);
+      sol_sock_r << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_i << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_r.precision(8);
+      sol_sock_i.precision(8);
+      sol_sock_r << "solution\n" << *pmesh << u_exact->real()
+                 << "window_title 'Exact Real Part'" << flush;
+      sol_sock_i << "solution\n" << *pmesh << u_exact->imag()
+                 << "window_title 'Exact Imaginary Part'" << flush;
+   }
+
+   // 10. Set up the parallel sesquilinear form a(.,.) on the finite element
+   //     space corresponding to the damped harmonic oscillator operator
+   //     of the appropriate type:
+   //
+   //     0) A scalar H1 field
+   //        -Div(a Grad) - omega^2 b + i omega c
+   //
+   //     1) A vector H(Curl) field
+   //        Curl(a Curl) - omega^2 b + i omega c
+   //
+   //     2) A vector H(Div) field
+   //        -Grad(a Div) - omega^2 b + i omega c
+   //
+   ConstantCoefficient stiffnessCoef(1.0/mu_);
+   ConstantCoefficient massCoef(-omega_ * omega_ * epsilon_);
+   ConstantCoefficient lossCoef(omega_ * sigma_);
+   ConstantCoefficient negMassCoef(omega_ * omega_ * epsilon_);
+
+   ParSesquilinearForm *a = new ParSesquilinearForm(fespace, conv);
+   switch (prob)
+   {
+      case 0:
+         a->AddDomainIntegrator(new DiffusionIntegrator(stiffnessCoef),
+                                NULL);
+         a->AddDomainIntegrator(new MassIntegrator(massCoef),
+                                new MassIntegrator(lossCoef));
+         break;
+      case 1:
+         a->AddDomainIntegrator(new CurlCurlIntegrator(stiffnessCoef),
+                                NULL);
+         a->AddDomainIntegrator(new VectorFEMassIntegrator(massCoef),
+                                new VectorFEMassIntegrator(lossCoef));
+         break;
+      case 2:
+         a->AddDomainIntegrator(new DivDivIntegrator(stiffnessCoef),
+                                NULL);
+         a->AddDomainIntegrator(new VectorFEMassIntegrator(massCoef),
+                                new VectorFEMassIntegrator(lossCoef));
+         break;
+   }
+
+   // 10a. Set up the parallel bilinear form for the preconditioner
+   //      corresponding to the appropriate operator if the STRUMPACK solver
+   //      has not been selected.
+   //
+   //      0) A scalar H1 field
+   //         -Div(a Grad) - omega^2 b + omega c
+   //
+   //      1) A vector H(Curl) field
+   //         Curl(a Curl) + omega^2 b + omega c
+   //
+   //      2) A vector H(Div) field
+   //         -Grad(a Div) - omega^2 b + omega c
+   //
+   ParBilinearForm *pcOp = NULL;
+#ifdef MFEM_USE_STRUMPACK
+   if (!strumpack)
+#endif
+   {
+      pcOp = new ParBilinearForm(fespace);
+      switch (prob)
+      {
+         case 0:
+            pcOp->AddDomainIntegrator(new DiffusionIntegrator(stiffnessCoef));
+            pcOp->AddDomainIntegrator(new MassIntegrator(massCoef));
+            pcOp->AddDomainIntegrator(new MassIntegrator(lossCoef));
+            break;
+         case 1:
+            pcOp->AddDomainIntegrator(new CurlCurlIntegrator(stiffnessCoef));
+            pcOp->AddDomainIntegrator(new VectorFEMassIntegrator(negMassCoef));
+            pcOp->AddDomainIntegrator(new VectorFEMassIntegrator(lossCoef));
+            break;
+         case 2:
+            pcOp->AddDomainIntegrator(new DivDivIntegrator(stiffnessCoef));
+            pcOp->AddDomainIntegrator(new VectorFEMassIntegrator(massCoef));
+            pcOp->AddDomainIntegrator(new VectorFEMassIntegrator(lossCoef));
+            break;
+      }
+   }
+
+   // 11. Assemble the parallel bilinear form and the corresponding linear
+   //     system, applying any necessary transformations such as: parallel
+   //     assembly, eliminating boundary conditions, applying conforming
+   //     constraints for non-conforming AMR, etc.
+   a->Assemble();
+   if (pcOp) { pcOp->Assemble(); }
+
+   OperatorHandle A;
+   Vector B, U;
+
+   a->FormLinearSystem(ess_tdof_list, u, b, A, U, B);
+   u = 0.0;
+   U = 0.0;
+
+   OperatorHandle PCOp;
+   if (pcOp) { pcOp->FormSystemMatrix(ess_tdof_list, PCOp); }
+
+   if (myid == 0)
+   {
+      ComplexHypreParMatrix * Ahyp =
+         dynamic_cast<ComplexHypreParMatrix*>(A.Ptr());
+
+      cout << "Size of linear system: "
+           << 2 * Ahyp->real().GetGlobalNumRows() << endl << endl;
+   }
+
+   // 12. Define and apply a parallel FGMRES solver for AU=B with a
+   //     block diagonal preconditioner based on the appropriate multigrid
+   //     preconditioner from hypre or simply use STRUMPACK.
+#ifdef MFEM_USE_STRUMPACK
+   if (!strumpack)
+#endif
+   {
+      Array<HYPRE_Int> blockTrueOffsets;
+      blockTrueOffsets.SetSize(3);
+      blockTrueOffsets[0] = 0;
+      blockTrueOffsets[1] = PCOp.Ptr()->Height();
+      blockTrueOffsets[2] = PCOp.Ptr()->Height();
+      blockTrueOffsets.PartialSum();
+
+      BlockDiagonalPreconditioner BDP(blockTrueOffsets);
+
+      Operator * pc_r = NULL;
+      Operator * pc_i = NULL;
+
+      switch (prob)
+      {
+         case 0:
+            pc_r =
+               new HypreBoomerAMG(dynamic_cast<HypreParMatrix&>(*PCOp.Ptr()));
+            pc_i = new ScaledOperator(pc_r,
+                                      (conv == ComplexOperator::HERMITIAN) ?
+                                      1.0:-1.0);
+            break;
+         case 1:
+            pc_r = new HypreAMS(dynamic_cast<HypreParMatrix&>(*PCOp.Ptr()),
+                                fespace);
+            pc_i = new ScaledOperator(pc_r,
+                                      (conv == ComplexOperator::HERMITIAN) ?
+                                      1.0:-1.0);
+            break;
+         case 2:
+            if (dim == 2 )
+            {
+               pc_r = new HypreAMS(dynamic_cast<HypreParMatrix&>(*PCOp.Ptr()),
+                                   fespace);
+            }
+            else
+            {
+               pc_r = new HypreADS(dynamic_cast<HypreParMatrix&>(*PCOp.Ptr()),
+                                   fespace);
+            }
+            pc_i = new ScaledOperator(pc_r,
+                                      (conv == ComplexOperator::HERMITIAN) ?
+                                      1.0:-1.0);
+            break;
+      }
+      BDP.SetDiagonalBlock(0, pc_r);
+      BDP.SetDiagonalBlock(1, pc_i);
+      BDP.owns_blocks = 0;
+
+      FGMRESSolver fgmres(MPI_COMM_WORLD);
+      fgmres.SetPreconditioner(BDP);
+      fgmres.SetOperator(*A.Ptr());
+      fgmres.SetRelTol(1e-12);
+      fgmres.SetMaxIter(1000);
+      fgmres.SetPrintLevel(1);
+      fgmres.Mult(B, U);
+   }
+#ifdef MFEM_USE_STRUMPACK
+   else
+   {
+      ComplexHypreParMatrix * Ahyp =
+         dynamic_cast<ComplexHypreParMatrix*>(A.Ptr());
+
+      STRUMPACKRowLocCmplxMatrix A_strmp(Ahyp->real(), Ahyp->imag());
+
+      STRUMPACKCmplxSolver strmp(argc, argv, comm);
+
+      strmp.SetPrintFactorStatistics(true);
+      strmp.SetPrintSolveStatistics(true);
+      // strmp.SetKrylovSolver(strumpack::KrylovSolver::AUTO); // core dump
+      strmp.SetKrylovSolver(strumpack::KrylovSolver::DIRECT); // core dump
+      // strmp.SetKrylovSolver(strumpack::KrylovSolver::REFINE); // core dump
+      // strmp.SetKrylovSolver(strumpack::KrylovSolver::PREC_GMRES); // index out of range asserts from strumpack::DenseMatrix
+      // strmp.SetKrylovSolver(strumpack::KrylovSolver::GMRES); // WORKS
+      strmp.SetReorderingStrategy(strumpack::ReorderingStrategy::METIS);
+      strmp.SetOperator(A_strmp);
+      strmp.SetFromCommandLine();
+      strmp.Mult(B, U);
+   }
+#endif
+
+   // 13. Recover the parallel grid function corresponding to U. This is the
+   //     local finite element solution on each processor.
+   a->RecoverFEMSolution(U, b, u);
+
+   if (exact_sol)
+   {
+      double err_r = -1.0;
+      double err_i = -1.0;
+
+      switch (prob)
+      {
+         case 0:
+            err_r = u.real().ComputeL2Error(u0_r);
+            err_i = u.imag().ComputeL2Error(u0_i);
+            break;
+         case 1:
+            err_r = u.real().ComputeL2Error(u1_r);
+            err_i = u.imag().ComputeL2Error(u1_i);
+            break;
+         case 2:
+            err_r = u.real().ComputeL2Error(u2_r);
+            err_i = u.imag().ComputeL2Error(u2_i);
+            break;
+      }
+
+      if ( myid == 0 )
+      {
+         cout << endl;
+         cout << "|| Re (u_h - u) ||_{L^2} = " << err_r << endl;
+         cout << "|| Im (u_h - u) ||_{L^2} = " << err_i << endl;
+         cout << endl;
+      }
+   }
+
+   // 14. Save the refined mesh and the solution in parallel. This output can be
+   //     viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
+   {
+      ostringstream mesh_name, sol_r_name, sol_i_name;
+      mesh_name << "mesh." << setfill('0') << setw(6) << myid;
+      sol_r_name << "sol_r." << setfill('0') << setw(6) << myid;
+      sol_i_name << "sol_i." << setfill('0') << setw(6) << myid;
+
+      ofstream mesh_ofs(mesh_name.str().c_str());
+      mesh_ofs.precision(8);
+      pmesh->Print(mesh_ofs);
+
+      ofstream sol_r_ofs(sol_r_name.str().c_str());
+      ofstream sol_i_ofs(sol_i_name.str().c_str());
+      sol_r_ofs.precision(8);
+      sol_i_ofs.precision(8);
+      u.real().Save(sol_r_ofs);
+      u.imag().Save(sol_i_ofs);
+   }
+
+   // 15. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock_r(vishost, visport);
+      socketstream sol_sock_i(vishost, visport);
+      sol_sock_r << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_i << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_r.precision(8);
+      sol_sock_i.precision(8);
+      sol_sock_r << "solution\n" << *pmesh << u.real()
+                 << "window_title 'Comp Real Part'" << flush;
+      sol_sock_i << "solution\n" << *pmesh << u.imag()
+                 << "window_title 'Comp Imaginary Part'" << flush;
+   }
+   if (visualization && exact_sol)
+   {
+      *u_exact -= u;
+
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock_r(vishost, visport);
+      socketstream sol_sock_i(vishost, visport);
+      sol_sock_r << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_i << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_r.precision(8);
+      sol_sock_i.precision(8);
+      sol_sock_r << "solution\n" << *pmesh << u_exact->real()
+                 << "window_title 'Exact-Comp Real Part'" << flush;
+      sol_sock_i << "solution\n" << *pmesh << u_exact->imag()
+                 << "window_title 'Exact-Comp Imaginary Part'" << flush;
+   }
+   if (visualization)
+   {
+      ParGridFunction u_t(fespace);
+      u_t = u.real();
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << u_t
+               << "window_title 'Harmonic Solution (t = 0.0 T)'"
+               << "pause\n" << flush;
+      if (myid == 0)
+         cout << "GLVis visualization paused."
+              << " Press space (in the GLVis window) to resume it.\n";
+      int num_frames = 32;
+      int i = 0;
+      while (sol_sock)
+      {
+         double t = (double)(i % num_frames) / num_frames;
+         ostringstream oss;
+         oss << "Harmonic Solution (t = " << t << " T)";
+
+         add(cos( 2.0 * M_PI * t), u.real(),
+             sin(-2.0 * M_PI * t), u.imag(), u_t);
+         sol_sock << "parallel " << num_procs << " " << myid << "\n";
+         sol_sock << "solution\n" << *pmesh << u_t
+                  << "window_title '" << oss.str() << "'" << flush;
+         i++;
+      }
+   }
+
+   // 16. Free the used memory.
+   delete a;
+   delete u_exact;
+   delete pcOp;
+   delete fespace;
+   delete fec;
+   delete pmesh;
+
+   MPI_Finalize();
+
+   return 0;
+}
+
+bool check_for_inline_mesh(const char * mesh_file)
+{
+   string file(mesh_file);
+   size_t p0 = file.find_last_of("/");
+   string s0 = file.substr((p0==string::npos)?0:(p0+1),7);
+   return s0 == "inline-";
+}
+
+complex<double> u0_exact(const Vector &x)
+{
+   int dim = x.Size();
+   complex<double> i(0.0, 1.0);
+   complex<double> alpha = (epsilon_ * omega_ - i * sigma_);
+   complex<double> kappa = std::sqrt(mu_ * omega_* alpha);
+   return std::exp(-i * kappa * x[dim - 1]);
+}
+
+double u0_real_exact(const Vector &x)
+{
+   return u0_exact(x).real();
+}
+
+double u0_imag_exact(const Vector &x)
+{
+   return u0_exact(x).imag();
+}
+
+void u1_real_exact(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+   v.SetSize(dim); v = 0.0; v[0] = u0_real_exact(x);
+}
+
+void u1_imag_exact(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+   v.SetSize(dim); v = 0.0; v[0] = u0_imag_exact(x);
+}
+
+void u2_real_exact(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+   v.SetSize(dim); v = 0.0; v[dim-1] = u0_real_exact(x);
+}
+
+void u2_imag_exact(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+   v.SetSize(dim); v = 0.0; v[dim-1] = u0_imag_exact(x);
+}

examples/ex3p_sp.cpp

@@ -0,0 +1,334 @@
+//                       MFEM Example 3 - Parallel Version
+//
+// Compile with: make ex3p
+//
+// Sample runs:  mpirun -np 4 ex3p -m ../data/star.mesh
+//               mpirun -np 4 ex3p -m ../data/square-disc.mesh -o 2
+//               mpirun -np 4 ex3p -m ../data/beam-tet.mesh
+//               mpirun -np 4 ex3p -m ../data/beam-hex.mesh
+//               mpirun -np 4 ex3p -m ../data/escher.mesh
+//               mpirun -np 4 ex3p -m ../data/escher.mesh -o 2
+//               mpirun -np 4 ex3p -m ../data/fichera.mesh
+//               mpirun -np 4 ex3p -m ../data/fichera-q2.vtk
+//               mpirun -np 4 ex3p -m ../data/fichera-q3.mesh
+//               mpirun -np 4 ex3p -m ../data/square-disc-nurbs.mesh
+//               mpirun -np 4 ex3p -m ../data/beam-hex-nurbs.mesh
+//               mpirun -np 4 ex3p -m ../data/amr-quad.mesh -o 2
+//               mpirun -np 4 ex3p -m ../data/amr-hex.mesh
+//               mpirun -np 4 ex3p -m ../data/star-surf.mesh -o 2
+//               mpirun -np 4 ex3p -m ../data/mobius-strip.mesh -o 2 -f 0.1
+//               mpirun -np 4 ex3p -m ../data/klein-bottle.mesh -o 2 -f 0.1
+//
+// Description:  This example code solves a simple electromagnetic diffusion
+//               problem corresponding to the second order definite Maxwell
+//               equation curl curl E + E = f with boundary condition
+//               E x n = <given tangential field>. Here, we use a given exact
+//               solution E and compute the corresponding r.h.s. f.
+//               We discretize with Nedelec finite elements in 2D or 3D.
+//
+//               The example demonstrates the use of H(curl) finite element
+//               spaces with the curl-curl and the (vector finite element) mass
+//               bilinear form, as well as the computation of discretization
+//               error when the exact solution is known. Static condensation is
+//               also illustrated.
+//
+//               We recommend viewing examples 1-2 before viewing this example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Exact solution, E, and r.h.s., f. See below for implementation.
+void E_exact(const Vector &, Vector &);
+void f_exact(const Vector &, Vector &);
+double freq = 1.0, kappa;
+int dim;
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   const char *mesh_file = "../data/beam-tet.mesh";
+   int order = 1;
+   bool static_cond = false;
+   bool visualization = 1;
+#ifdef MFEM_USE_STRUMPACK
+   bool use_strumpack = false;
+#endif
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree).");
+   args.AddOption(&freq, "-f", "--frequency", "Set the frequency for the exact"
+                  " solution.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+#ifdef MFEM_USE_STRUMPACK
+   args.AddOption(&use_strumpack, "-strumpack", "--strumpack-solver",
+                  "-no-strumpack", "--no-strumpack-solver",
+                  "Use STRUMPACK's double complex linear solver.");
+#endif
+
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+   kappa = freq * M_PI;
+
+   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
+   //    and volume meshes with the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 4. Refine the serial mesh on all processors to increase the resolution. In
+   //    this example we do 'ref_levels' of uniform refinement. We choose
+   //    'ref_levels' to be the largest number that gives a final mesh with no
+   //    more than 1,000 elements.
+   {
+      int ref_levels =
+         (int)floor(log(100000./mesh->GetNE())/log(2.)/dim);
+      for (int l = 0; l < ref_levels; l++)
+      {
+         mesh->UniformRefinement();
+      }
+   }
+
+   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted. Tetrahedral
+   //    meshes need to be reoriented before we can define high-order Nedelec
+   //    spaces on them.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   {
+      int par_ref_levels = 2;
+      for (int l = 0; l < par_ref_levels; l++)
+      {
+         pmesh->UniformRefinement();
+      }
+   }
+   pmesh->ReorientTetMesh();
+
+   // 6. Define a parallel finite element space on the parallel mesh. Here we
+   //    use the Nedelec finite elements of the specified order.
+   FiniteElementCollection *fec = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   HYPRE_Int size = fespace->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of finite element unknowns: " << size << endl;
+   }
+
+   // 7. Determine the list of true (i.e. parallel conforming) essential
+   //    boundary dofs. In this example, the boundary conditions are defined
+   //    by marking all the boundary attributes from the mesh as essential
+   //    (Dirichlet) and converting them to a list of true dofs.
+   Array<int> ess_tdof_list;
+   if (pmesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   // 8. Set up the parallel linear form b(.) which corresponds to the
+   //    right-hand side of the FEM linear system, which in this case is
+   //    (f,phi_i) where f is given by the function f_exact and phi_i are the
+   //    basis functions in the finite element fespace.
+   VectorFunctionCoefficient f(sdim, f_exact);
+   ParLinearForm *b = new ParLinearForm(fespace);
+   b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
+   b->Assemble();
+
+   // 9. Define the solution vector x as a parallel finite element grid function
+   //    corresponding to fespace. Initialize x by projecting the exact
+   //    solution. Note that only values from the boundary edges will be used
+   //    when eliminating the non-homogeneous boundary condition to modify the
+   //    r.h.s. vector b.
+   ParGridFunction x(fespace);
+   VectorFunctionCoefficient E(sdim, E_exact);
+   x.ProjectCoefficient(E);
+
+   // 10. Set up the parallel bilinear form corresponding to the EM diffusion
+   //     operator curl muinv curl + sigma I, by adding the curl-curl and the
+   //     mass domain integrators.
+   Coefficient *muinv = new ConstantCoefficient(1.0);
+   Coefficient *sigma = new ConstantCoefficient(-1.0);
+   ParBilinearForm *a = new ParBilinearForm(fespace);
+   a->AddDomainIntegrator(new CurlCurlIntegrator(*muinv));
+   a->AddDomainIntegrator(new VectorFEMassIntegrator(*sigma));
+
+   // 11. Assemble the parallel bilinear form and the corresponding linear
+   //     system, applying any necessary transformations such as: parallel
+   //     assembly, eliminating boundary conditions, applying conforming
+   //     constraints for non-conforming AMR, static condensation, etc.
+   if (static_cond) { a->EnableStaticCondensation(); }
+   a->Assemble();
+
+   HypreParMatrix A;
+   Vector B, X;
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+
+   if (myid == 0)
+   {
+      cout << "Size of linear system: " << A.GetGlobalNumRows() << endl;
+   }
+
+   StopWatch chrono;
+   chrono.Clear();
+   chrono.Start();
+
+#ifdef MFEM_USE_STRUMPACK
+   if (use_strumpack)
+   {
+      Operator * Arow = new STRUMPACKRowLocMatrix(A);
+
+      STRUMPACKSolver * strumpack = new STRUMPACKSolver(argc, argv, MPI_COMM_WORLD);
+      strumpack->SetPrintFactorStatistics(true);
+      strumpack->SetPrintSolveStatistics(false);
+      strumpack->SetKrylovSolver(strumpack::KrylovSolver::DIRECT);
+      strumpack->SetReorderingStrategy(strumpack::ReorderingStrategy::METIS);
+      // strumpack->SetMC64Job(strumpack::MC64Job::NONE);
+      // strumpack->SetSymmetricPattern(true);
+      strumpack->SetOperator(*Arow);
+      strumpack->SetFromCommandLine();
+      //Solver * precond = strumpack;
+
+      strumpack->Mult(B, X);
+
+      delete strumpack;
+      delete Arow;
+   }
+   else
+#endif
+   {
+      // 12. Define and apply a parallel PCG solver for AX=B with the AMS
+      //     preconditioner from hypre.
+      ParFiniteElementSpace *prec_fespace =
+         (a->StaticCondensationIsEnabled() ? a->SCParFESpace() : fespace);
+      HypreSolver *ams = new HypreAMS(A, prec_fespace);
+      HyprePCG *pcg = new HyprePCG(A);
+      pcg->SetTol(1e-12);
+      pcg->SetMaxIter(500);
+      pcg->SetPrintLevel(2);
+      pcg->SetPreconditioner(*ams);
+      pcg->Mult(B, X);
+
+      delete pcg;
+      delete ams;
+   }
+
+   chrono.Stop();
+   cout << "Solver time " << chrono.RealTime() << endl;
+
+   // 13. Recover the parallel grid function corresponding to X. This is the
+   //     local finite element solution on each processor.
+   a->RecoverFEMSolution(X, *b, x);
+
+   // 14. Compute and print the L^2 norm of the error.
+   {
+      double err = x.ComputeL2Error(E);
+      if (myid == 0)
+      {
+         cout << "\n|| E_h - E ||_{L^2} = " << err << '\n' << endl;
+      }
+   }
+
+   // 15. Save the refined mesh and the solution in parallel. This output can
+   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
+   {
+      ostringstream mesh_name, sol_name;
+      mesh_name << "mesh." << setfill('0') << setw(6) << myid;
+      sol_name << "sol." << setfill('0') << setw(6) << myid;
+
+      ofstream mesh_ofs(mesh_name.str().c_str());
+      mesh_ofs.precision(8);
+      pmesh->Print(mesh_ofs);
+
+      ofstream sol_ofs(sol_name.str().c_str());
+      sol_ofs.precision(8);
+      x.Save(sol_ofs);
+   }
+
+   // 16. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << x << flush;
+   }
+
+   // 17. Free the used memory.
+   delete a;
+   delete sigma;
+   delete muinv;
+   delete b;
+   delete fespace;
+   delete fec;
+   delete pmesh;
+
+   MPI_Finalize();
+
+   return 0;
+}
+
+
+void E_exact(const Vector &x, Vector &E)
+{
+   if (dim == 3)
+   {
+      E(0) = sin(kappa * x(1));
+      E(1) = sin(kappa * x(2));
+      E(2) = sin(kappa * x(0));
+   }
+   else
+   {
+      E(0) = sin(kappa * x(1));
+      E(1) = sin(kappa * x(0));
+      if (x.Size() == 3) { E(2) = 0.0; }
+   }
+}
+
+void f_exact(const Vector &x, Vector &f)
+{
+   if (dim == 3)
+   {
+      f(0) = (1. + kappa * kappa) * sin(kappa * x(1));
+      f(1) = (1. + kappa * kappa) * sin(kappa * x(2));
+      f(2) = (1. + kappa * kappa) * sin(kappa * x(0));
+   }
+   else
+   {
+      f(0) = (1. + kappa * kappa) * sin(kappa * x(1));
+      f(1) = (1. + kappa * kappa) * sin(kappa * x(0));
+      if (x.Size() == 3) { f(2) = 0.0; }
+   }
+}

examples/ex6.cpp

@@ -15,6 +15,11 @@
 //               ex6 -m ../data/square-disc-surf.mesh -o 2
 //               ex6 -m ../data/amr-quad.mesh
 //
+// Device sample runs:
+//             > ex6 -pa -d cuda
+//             > ex6 -pa -d occa-cuda
+//             > ex6 -pa -d raja-omp
+//
 // Description:  This is a version of Example 1 with a simple adaptive mesh
 //               refinement loop. The problem being solved is again the Laplace
 //               equation -Delta u = 1 with homogeneous Dirichlet boundary
@@ -43,13 +48,19 @@ int main(int argc, char *argv[])
    // 1. Parse command-line options.
    const char *mesh_file = "../data/star.mesh";
    int order = 1;
-   bool visualization = 1;
+   bool pa = false;
+   const char *device = "cpu";
+   bool visualization = true;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
                   "Mesh file to use.");
    args.AddOption(&order, "-o", "--order",
                   "Finite element order (polynomial degree).");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -85,10 +96,15 @@ int main(int argc, char *argv[])
    H1_FECollection fec(order, dim);
    FiniteElementSpace fespace(&mesh, &fec);
 
-   // 5. As in Example 1, we set up bilinear and linear forms corresponding to
+   // 5. Set device config parameters from the command line options.
+   Device::Configure(device);
+   Device::Print();
+
+   // 6. As in Example 1, we set up bilinear and linear forms corresponding to
    //    the Laplace problem -\Delta u = 1. We don't assemble the discrete
    //    problem yet, this will be done in the main loop.
    BilinearForm a(&fespace);
+   if (pa) { a.SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    LinearForm b(&fespace);
 
    ConstantCoefficient one(1.0);
@@ -98,18 +114,18 @@ int main(int argc, char *argv[])
    a.AddDomainIntegrator(integ);
    b.AddDomainIntegrator(new DomainLFIntegrator(one));
 
-   // 6. The solution vector x and the associated finite element grid function
+   // 7. The solution vector x and the associated finite element grid function
    //    will be maintained over the AMR iterations. We initialize it to zero.
    GridFunction x(&fespace);
    x = 0.0;
 
-   // 7. All boundary attributes will be used for essential (Dirichlet) BC.
+   // 8. All boundary attributes will be used for essential (Dirichlet) BC.
    MFEM_VERIFY(mesh.bdr_attributes.Size() > 0,
                "Boundary attributes required in the mesh.");
    Array<int> ess_bdr(mesh.bdr_attributes.Max());
    ess_bdr = 1;
 
-   // 8. Connect to GLVis.
+   // 9. Connect to GLVis.
    char vishost[] = "localhost";
    int  visport   = 19916;
    socketstream sol_sock;
@@ -118,23 +134,23 @@ int main(int argc, char *argv[])
       sol_sock.open(vishost, visport);
    }
 
-   // 9. Set up an error estimator. Here we use the Zienkiewicz-Zhu estimator
-   //    that uses the ComputeElementFlux method of the DiffusionIntegrator to
-   //    recover a smoothed flux (gradient) that is subtracted from the element
-   //    flux to get an error indicator. We need to supply the space for the
-   //    smoothed flux: an (H1)^sdim (i.e., vector-valued) space is used here.
+   // 10. Set up an error estimator. Here we use the Zienkiewicz-Zhu estimator
+   //     that uses the ComputeElementFlux method of the DiffusionIntegrator to
+   //     recover a smoothed flux (gradient) that is subtracted from the element
+   //     flux to get an error indicator. We need to supply the space for the
+   //     smoothed flux: an (H1)^sdim (i.e., vector-valued) space is used here.
    FiniteElementSpace flux_fespace(&mesh, &fec, sdim);
    ZienkiewiczZhuEstimator estimator(*integ, x, flux_fespace);
    estimator.SetAnisotropic();
 
-   // 10. A refiner selects and refines elements based on a refinement strategy.
+   // 11. A refiner selects and refines elements based on a refinement strategy.
    //     The strategy here is to refine elements with errors larger than a
    //     fraction of the maximum element error. Other strategies are possible.
    //     The refiner will call the given error estimator.
    ThresholdRefiner refiner(estimator);
    refiner.SetTotalErrorFraction(0.7);
 
-   // 11. The main AMR loop. In each iteration we solve the problem on the
+   // 12. The main AMR loop. In each iteration we solve the problem on the
    //     current mesh, visualize the solution, and refine the mesh.
    const int max_dofs = 50000;
    for (int it = 0; ; it++)
@@ -143,44 +159,55 @@ int main(int argc, char *argv[])
       cout << "\nAMR iteration " << it << endl;
       cout << "Number of unknowns: " << cdofs << endl;
 
-      // 12. Assemble the stiffness matrix and the right-hand side.
-      a.Assemble();
+      // 13. Assemble the right-hand side.
       b.Assemble();
 
-      // 13. Set Dirichlet boundary values in the GridFunction x.
+      // 14. Set Dirichlet boundary values in the GridFunction x.
       //     Determine the list of Dirichlet true DOFs in the linear system.
       Array<int> ess_tdof_list;
       x.ProjectBdrCoefficient(zero, ess_bdr);
       fespace.GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
 
-      // 14. Create the linear system: eliminate boundary conditions, constrain
+      // 15. Switch to the device and assemble the stiffness matrix.
+      Device::Enable();
+      a.Assemble();
+
+      // 16. Create the linear system: eliminate boundary conditions, constrain
       //     hanging nodes and possibly apply other transformations. The system
       //     will be solved for true (unconstrained) DOFs only.
-      SparseMatrix A;
+      OperatorPtr A;
       Vector B, X;
+
       const int copy_interior = 1;
       a.FormLinearSystem(ess_tdof_list, x, b, A, X, B, copy_interior);
 
+      // 17. Solve the linear system A X = B.
+      if (!pa)
+      {
 #ifndef MFEM_USE_SUITESPARSE
-      // 15. Define a simple symmetric Gauss-Seidel preconditioner and use it to
-      //     solve the linear system with PCG.
-      GSSmoother M(A);
-      PCG(A, M, B, X, 3, 200, 1e-12, 0.0);
+         // Use a simple symmetric Gauss-Seidel preconditioner with PCG.
+         GSSmoother M((SparseMatrix&)(*A));
+         PCG(*A, M, B, X, 3, 200, 1e-12, 0.0);
 #else
-      // 15. If MFEM was compiled with SuiteSparse, use UMFPACK to solve the
-      //     the linear system.
-      UMFPackSolver umf_solver;
-      umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
-      umf_solver.SetOperator(A);
-      umf_solver.Mult(B, X);
+         // If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.
+         UMFPackSolver umf_solver;
+         umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+         umf_solver.SetOperator(*A);
+         umf_solver.Mult(B, X);
 #endif
+      }
+      else // No preconditioning for now in partial assembly mode.
+      {
+         CG(*A, B, X, 3, 2000, 1e-12, 0.0);
+      }
 
-      // 16. After solving the linear system, reconstruct the solution as a
+      // 18. After solving the linear system, reconstruct the solution as a
       //     finite element GridFunction. Constrained nodes are interpolated
       //     from true DOFs (it may therefore happen that x.Size() >= X.Size()).
+      Device::Disable();
       a.RecoverFEMSolution(X, b, x);
 
-      // 17. Send solution by socket to the GLVis server.
+      // 19. Send solution by socket to the GLVis server.
       if (visualization && sol_sock.good())
       {
          sol_sock.precision(8);
@@ -193,7 +220,7 @@ int main(int argc, char *argv[])
          break;
       }
 
-      // 18. Call the refiner to modify the mesh. The refiner calls the error
+      // 20. Call the refiner to modify the mesh. The refiner calls the error
       //     estimator to obtain element errors, then it selects elements to be
       //     refined and finally it modifies the mesh. The Stop() method can be
       //     used to determine if a stopping criterion was met.
@@ -204,7 +231,7 @@ int main(int argc, char *argv[])
          break;
       }
 
-      // 19. Update the space to reflect the new state of the mesh. Also,
+      // 21. Update the space to reflect the new state of the mesh. Also,
       //     interpolate the solution x so that it lies in the new space but
       //     represents the same function. This saves solver iterations later
       //     since we'll have a good initial guess of x in the next step.
@@ -213,7 +240,7 @@ int main(int argc, char *argv[])
       fespace.Update();
       x.Update();
 
-      // 20. Inform also the bilinear and linear forms that the space has
+      // 22. Inform also the bilinear and linear forms that the space has
       //     changed.
       a.Update();
       b.Update();

examples/ex6p.cpp

@@ -15,6 +15,11 @@
 //               mpirun -np 4 ex6p -m ../data/square-disc-surf.mesh -o 2
 //               mpirun -np 4 ex6p -m ../data/amr-quad.mesh
 //
+// Device sample runs:
+//             > mpirun -np 4 ex6p -pa -d cuda
+//             > mpirun -np 4 ex6p -pa -d occa-cuda
+//             > mpirun -np 4 ex6p -pa -d raja-omp
+//
 // Description:  This is a version of Example 1 with a simple adaptive mesh
 //               refinement loop. The problem being solved is again the Laplace
 //               equation -Delta u = 1 with homogeneous Dirichlet boundary
@@ -49,6 +54,8 @@ int main(int argc, char *argv[])
    // 2. Parse command-line options.
    const char *mesh_file = "../data/star.mesh";
    int order = 1;
+   bool pa = false;
+   const char *device = "cpu";
    bool visualization = true;
 
    OptionsParser args(argc, argv);
@@ -56,6 +63,10 @@ int main(int argc, char *argv[])
                   "Mesh file to use.");
    args.AddOption(&order, "-o", "--order",
                   "Finite element order (polynomial degree).");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -106,10 +117,15 @@ int main(int argc, char *argv[])
    H1_FECollection fec(order, dim);
    ParFiniteElementSpace fespace(&pmesh, &fec);
 
-   // 7. As in Example 1p, we set up bilinear and linear forms corresponding to
+   // 7. Set device config parameters from the command line options.
+   Device::Configure(device);
+   if (myid == 0) { Device::Print(); }
+
+   // 8. As in Example 1p, we set up bilinear and linear forms corresponding to
    //    the Laplace problem -\Delta u = 1. We don't assemble the discrete
    //    problem yet, this will be done in the main loop.
    ParBilinearForm a(&fespace);
+   if (pa) { a.SetAssemblyLevel(AssemblyLevel::PARTIAL); }
    ParLinearForm b(&fespace);
 
    ConstantCoefficient one(1.0);
@@ -118,12 +134,12 @@ int main(int argc, char *argv[])
    a.AddDomainIntegrator(integ);
    b.AddDomainIntegrator(new DomainLFIntegrator(one));
 
-   // 8. The solution vector x and the associated finite element grid function
+   // 9. The solution vector x and the associated finite element grid function
    //    will be maintained over the AMR iterations. We initialize it to zero.
    ParGridFunction x(&fespace);
    x = 0;
 
-   // 9. Connect to GLVis.
+   // 10. Connect to GLVis.
    char vishost[] = "localhost";
    int  visport   = 19916;
 
@@ -145,7 +161,7 @@ int main(int argc, char *argv[])
       sout.precision(8);
    }
 
-   // 10. Set up an error estimator. Here we use the Zienkiewicz-Zhu estimator
+   // 11. Set up an error estimator. Here we use the Zienkiewicz-Zhu estimator
    //     with L2 projection in the smoothing step to better handle hanging
    //     nodes and parallel partitioning. We need to supply a space for the
    //     discontinuous flux (L2) and a space for the smoothed flux (H(div) is
@@ -159,14 +175,14 @@ int main(int argc, char *argv[])
    // ParFiniteElementSpace smooth_flux_fes(&pmesh, &smooth_flux_fec, dim);
    L2ZienkiewiczZhuEstimator estimator(*integ, x, flux_fes, smooth_flux_fes);
 
-   // 11. A refiner selects and refines elements based on a refinement strategy.
+   // 12. A refiner selects and refines elements based on a refinement strategy.
    //     The strategy here is to refine elements with errors larger than a
    //     fraction of the maximum element error. Other strategies are possible.
    //     The refiner will call the given error estimator.
    ThresholdRefiner refiner(estimator);
    refiner.SetTotalErrorFraction(0.7);
 
-   // 12. The main AMR loop. In each iteration we solve the problem on the
+   // 13. The main AMR loop. In each iteration we solve the problem on the
    //     current mesh, visualize the solution, and refine the mesh.
    const int max_dofs = 100000;
    for (int it = 0; ; it++)
@@ -178,41 +194,48 @@ int main(int argc, char *argv[])
          cout << "Number of unknowns: " << global_dofs << endl;
       }
 
-      // 13. Assemble the stiffness matrix and the right-hand side. Note that
-      //     MFEM doesn't care at this point that the mesh is nonconforming
-      //     and parallel. The FE space is considered 'cut' along hanging
-      //     edges/faces, and also across processor boundaries.
-      a.Assemble();
-      b.Assemble();
-
-      // 14. Create the parallel linear system: eliminate boundary conditions,
-      //     constrain hanging nodes and nodes across processor boundaries.
-      //     The system will be solved for true (unconstrained/unique) DOFs only.
+      // 14. Assemble the right-hand side and determine the list of true
+      //     (i.e. parallel conforming) essential boundary dofs.
       Array<int> ess_tdof_list;
       fespace.GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+      b.Assemble();
 
-      HypreParMatrix A;
+      // 15. Switch to the device and assemble the stiffness matrix. Note that
+      //     MFEM doesn't care at this point that the mesh is nonconforming and
+      //     parallel.  The FE space is considered 'cut' along hanging
+      //     edges/faces, and also across processor boundaries.
+      Device::Enable();
+      a.Assemble();
+
+      // 16. Create the parallel linear system: eliminate boundary conditions.
+      //     The system will be solved for true (unconstrained/unique) DOFs only.
+      OperatorPtr A;
       Vector B, X;
+
       const int copy_interior = 1;
       a.FormLinearSystem(ess_tdof_list, x, b, A, X, B, copy_interior);
 
-      // 15. Define and apply a parallel PCG solver for AX=B with the BoomerAMG
-      //     preconditioner from hypre.
-      HypreBoomerAMG amg;
-      amg.SetPrintLevel(0);
-      CGSolver pcg(A.GetComm());
-      pcg.SetPreconditioner(amg);
-      pcg.SetOperator(A);
-      pcg.SetRelTol(1e-6);
-      pcg.SetMaxIter(200);
-      pcg.SetPrintLevel(3); // print the first and the last iterations only
-      pcg.Mult(B, X);
+      // 17. Solve the linear system A X = B.
+      //     * With full assembly, use the BoomerAMG preconditioner from hypre.
+      //     * With partial assembly, use no preconditioner, for now.
+      HypreBoomerAMG *amg = NULL;
+      if (!pa) { amg = new HypreBoomerAMG; amg->SetPrintLevel(0); }
+      CGSolver cg(MPI_COMM_WORLD);
+      cg.SetRelTol(1e-6);
+      cg.SetMaxIter(2000);
+      cg.SetPrintLevel(3); // print the first and the last iterations only
+      if (amg) { cg.SetPreconditioner(*amg); }
+      cg.SetOperator(*A);
+      cg.Mult(B, X);
+      delete amg;
 
-      // 16. Extract the parallel grid function corresponding to the finite element
-      //     approximation X. This is the local solution on each processor.
+      // 18. Switch back to the host and extract the parallel grid function
+      //     corresponding to the finite element approximation X. This is the
+      //     local solution on each processor.
+      Device::Disable();
       a.RecoverFEMSolution(X, b, x);
 
-      // 17. Send the solution by socket to a GLVis server.
+      // 19. Send the solution by socket to a GLVis server.
       if (visualization)
       {
          sout << "parallel " << num_procs << " " << myid << "\n";
@@ -228,7 +251,7 @@ int main(int argc, char *argv[])
          break;
       }
 
-      // 18. Call the refiner to modify the mesh. The refiner calls the error
+      // 20. Call the refiner to modify the mesh. The refiner calls the error
       //     estimator to obtain element errors, then it selects elements to be
       //     refined and finally it modifies the mesh. The Stop() method can be
       //     used to determine if a stopping criterion was met.
@@ -242,7 +265,7 @@ int main(int argc, char *argv[])
          break;
       }
 
-      // 19. Update the finite element space (recalculate the number of DOFs,
+      // 21. Update the finite element space (recalculate the number of DOFs,
       //     etc.) and create a grid function update matrix. Apply the matrix
       //     to any GridFunctions over the space. In this case, the update
       //     matrix is an interpolation matrix so the updated GridFunction will
@@ -250,7 +273,7 @@ int main(int argc, char *argv[])
       fespace.Update();
       x.Update();
 
-      // 20. Load balance the mesh, and update the space and solution. Currently
+      // 22. Load balance the mesh, and update the space and solution. Currently
       //     available only for nonconforming meshes.
       if (pmesh.Nonconforming())
       {
@@ -262,7 +285,7 @@ int main(int argc, char *argv[])
          x.Update();
       }
 
-      // 21. Inform also the bilinear and linear forms that the space has
+      // 23. Inform also the bilinear and linear forms that the space has
       //     changed.
       a.Update();
       b.Update();

examples/ex9.cpp

@@ -132,8 +132,8 @@ int main(int argc, char *argv[])
 
    // 2. Read the mesh from the given mesh file. We can handle geometrically
    //    periodic meshes in this code.
-   Mesh *mesh = new Mesh(mesh_file, 1, 1);
-   int dim = mesh->Dimension();
+   Mesh mesh(mesh_file, 1, 1);
+   int dim = mesh.Dimension();
 
    // 3. Define the ODE solver used for time integration. Several explicit
    //    Runge-Kutta methods are available.
@@ -147,7 +147,6 @@ int main(int argc, char *argv[])
       case 6: ode_solver = new RK6Solver; break;
       default:
          cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
-         delete mesh;
          return 3;
    }
 
@@ -157,18 +156,18 @@ int main(int argc, char *argv[])
    //    a (piecewise-polynomial) high-order mesh.
    for (int lev = 0; lev < ref_levels; lev++)
    {
-      mesh->UniformRefinement();
+      mesh.UniformRefinement();
    }
-   if (mesh->NURBSext)
+   if (mesh.NURBSext)
    {
-      mesh->SetCurvature(max(order, 1));
+      mesh.SetCurvature(max(order, 1));
    }
-   mesh->GetBoundingBox(bb_min, bb_max, max(order, 1));
+   mesh.GetBoundingBox(bb_min, bb_max, max(order, 1));
 
    // 5. Define the discontinuous DG finite element space of the given
    //    polynomial order on the refined mesh.
    DG_FECollection fec(order, dim);
-   FiniteElementSpace fes(mesh, &fec);
+   FiniteElementSpace fes(&mesh, &fec);
 
    cout << "Number of unknowns: " << fes.GetVSize() << endl;
 
@@ -208,7 +207,7 @@ int main(int argc, char *argv[])
    {
       ofstream omesh("ex9.mesh");
       omesh.precision(precision);
-      mesh->Print(omesh);
+      mesh.Print(omesh);
       ofstream osol("ex9-init.gf");
       osol.precision(precision);
       u.Save(osol);
@@ -222,14 +221,14 @@ int main(int argc, char *argv[])
       if (binary)
       {
 #ifdef MFEM_USE_SIDRE
-         dc = new SidreDataCollection("Example9", mesh);
+         dc = new SidreDataCollection("Example9", &mesh);
 #else
          MFEM_ABORT("Must build with MFEM_USE_SIDRE=YES for binary output.");
 #endif
       }
       else
       {
-         dc = new VisItDataCollection("Example9", mesh);
+         dc = new VisItDataCollection("Example9", &mesh);
          dc->SetPrecision(precision);
       }
       dc->RegisterField("solution", &u);
@@ -254,7 +253,7 @@ int main(int argc, char *argv[])
       else
       {
          sout.precision(precision);
-         sout << "solution\n" << *mesh << u;
+         sout << "solution\n" << mesh << u;
          sout << "pause\n";
          sout << flush;
          cout << "GLVis visualization paused."
@@ -286,7 +285,7 @@ int main(int argc, char *argv[])
 
          if (visualization)
          {
-            sout << "solution\n" << *mesh << u << flush;
+            sout << "solution\n" << mesh << u << flush;
          }
 
          if (visit)

examples/makefile

@@ -22,9 +22,9 @@ MFEM_LIB_FILE = mfem_is_not_built
 -include $(CONFIG_MK)
 
 SEQ_EXAMPLES = ex1 ex2 ex3 ex4 ex5 ex6 ex7 ex8 ex9 ex10 ex14 ex15 ex16 ex17\
-  ex18 ex19 ex20 ex22
+  ex18 ex19 ex20 ex21 ex22
 PAR_EXAMPLES = ex1p ex2p ex3p ex4p ex5p ex6p ex7p ex8p ex9p ex10p ex11p ex12p\
- ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex22p
+ ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex21p ex22p
 
 ifeq ($(MFEM_USE_MPI),NO)
    EXAMPLES = $(SEQ_EXAMPLES)
@@ -96,6 +96,12 @@ ex15-test-seq: ex15
 	@$(call mfem-test,$<,, Serial example,-e 1)
 ex15p-test-par: ex15p
 	@$(call mfem-test,$<, $(RUN_MPI), Parallel example,-e 1)
+# Testing: optional tests
+ifeq ($(MFEM_USE_STRUMPACK),YES)
+ex11p-test-strumpack: ex11p
+	@$(call mfem-test,$<, $(RUN_MPI), STRUMPACK example,--strumpack)
+test-par-YES: ex11p-test-strumpack
+endif
 
 # Testing: "test" target and mfem-test* variables are defined in config/test.mk
 
@@ -112,7 +118,7 @@ clean-build:
 clean-exec:
 	@rm -f refined.mesh displaced.mesh mesh.* ex5.mesh
 	@rm -rf Example5* Example9* Example15* Example16*
-	@rm -f sphere_refined.* sol.* sol_u.* sol_p.*
+	@rm -f sphere_refined.* sol.* sol_u.* sol_p.* sol_r.* sol_i.*
 	@rm -f ex9.mesh ex9-mesh.* ex9-init.* ex9-final.*
 	@rm -f deformed.* velocity.* elastic_energy.* mode_*
 	@rm -f ex16.mesh ex16-mesh.* ex16-init.* ex16-final.*

fem/CMakeLists.txt

@@ -11,7 +11,9 @@
 
 set(SRCS
   bilinearform.cpp
+  bilinearform_ext.cpp
   bilininteg.cpp
+  bilininteg_ext.cpp
   coefficient.cpp
   datacollection.cpp
   eltrans.cpp
@@ -33,7 +35,9 @@ set(SRCS
 
 set(HDRS
   bilinearform.hpp
+  bilinearform_ext.hpp
   bilininteg.hpp
+  bilininteg_ext.hpp
   coefficient.hpp
   datacollection.hpp
   eltrans.hpp

fem/bilinearform.cpp

@@ -12,6 +12,7 @@
 // Implementation of class BilinearForm
 
 #include "fem.hpp"
+#include "../general/device.hpp"
 #include <cmath>
 
 namespace mfem
@@ -54,7 +55,7 @@ void BilinearForm::AllocMat()
 
    int *I = dof_dof.GetI();
    int *J = dof_dof.GetJ();
-   double *data = new double[I[height]];
+   double *data = mfem::New<double>(I[height]);
 
    mat = new SparseMatrix(I, J, data, height, height, true, true, true);
    *mat = 0.0;
@@ -62,7 +63,7 @@ void BilinearForm::AllocMat()
    dof_dof.LoseData();
 }
 
-BilinearForm::BilinearForm (FiniteElementSpace * f)
+BilinearForm::BilinearForm(FiniteElementSpace * f)
    : Matrix (f->GetVSize())
 {
    fes = f;
@@ -74,6 +75,10 @@ BilinearForm::BilinearForm (FiniteElementSpace * f)
    hybridization = NULL;
    precompute_sparsity = 0;
    diag_policy = DIAG_KEEP;
+
+   assembly = AssemblyLevel::FULL;
+   batch = 1;
+   ext = NULL;
 }
 
 BilinearForm::BilinearForm (FiniteElementSpace * f, BilinearForm * bf, int ps)
@@ -89,6 +94,10 @@ BilinearForm::BilinearForm (FiniteElementSpace * f, BilinearForm * bf, int ps)
    precompute_sparsity = ps;
    diag_policy = DIAG_KEEP;
 
+   assembly = AssemblyLevel::FULL;
+   batch = 1;
+   ext = NULL;
+
    // Copy the pointers to the integrators
    dbfi = bf->dbfi;
 
@@ -103,9 +112,48 @@ BilinearForm::BilinearForm (FiniteElementSpace * f, BilinearForm * bf, int ps)
    AllocMat();
 }
 
+void BilinearForm::SetAssemblyLevel(AssemblyLevel assembly_level)
+{
+   if (ext)
+   {
+      MFEM_ABORT("the assembly level has already been set!");
+   }
+   assembly = assembly_level;
+   switch (assembly)
+   {
+      case AssemblyLevel::FULL:
+         if (Device::IsEnabled())
+         {
+            mfem_error("Full assembly not supported yet in device mode!");
+            // ext = new FABilinearFormExtension(this);
+         }
+         // Use the original BilinearForm implementation for now
+         break;
+      case AssemblyLevel::ELEMENT:
+         mfem_error("Element assembly not supported yet... stay tuned!");
+         // ext = new EABilinearFormExtension(this);
+         break;
+      case AssemblyLevel::PARTIAL:
+         ext = new PABilinearFormExtension(this);
+         break;
+      case AssemblyLevel::NONE:
+         mfem_error("Matrix-free action not supported yet... stay tuned!");
+         // ext = new MFBilinearFormExtension(this);
+         break;
+      default:
+         mfem_error("Unknown assembly level");
+   }
+}
+
 void BilinearForm::EnableStaticCondensation()
 {
    delete static_cond;
+   if (assembly != AssemblyLevel::FULL)
+   {
+      static_cond = NULL;
+      MFEM_WARNING("Static condensation not supported for this assembly level");
+      return;
+   }
    static_cond = new StaticCondensation(fes);
    if (static_cond->ReducesTrueVSize())
    {
@@ -125,6 +173,13 @@ void BilinearForm::EnableHybridization(FiniteElementSpace *constr_space,
                                        const Array<int> &ess_tdof_list)
 {
    delete hybridization;
+   if (assembly != AssemblyLevel::FULL)
+   {
+      delete constr_integ;
+      hybridization = NULL;
+      MFEM_WARNING("Hybridization not supported for this assembly level");
+      return;
+   }
    hybridization = new Hybridization(fes, constr_space);
    hybridization->SetConstraintIntegrator(constr_integ);
    hybridization->Init(ess_tdof_list);
@@ -179,9 +234,9 @@ void BilinearForm::Finalize (int skip_zeros)
    if (hybridization) { hybridization->Finalize(); }
 }
 
-void BilinearForm::AddDomainIntegrator (BilinearFormIntegrator * bfi)
+void BilinearForm::AddDomainIntegrator(BilinearFormIntegrator *bfi)
 {
-   dbfi.Append (bfi);
+   dbfi.Append(bfi);
 }
 
 void BilinearForm::AddBoundaryIntegrator (BilinearFormIntegrator * bfi)
@@ -287,20 +342,29 @@ void BilinearForm::AssembleBdrElementMatrix(
    }
 }
 
-void BilinearForm::Assemble (int skip_zeros)
+void BilinearForm::Assemble(int skip_zeros)
 {
+   if (Device::IsEnabled() && (assembly != AssemblyLevel::PARTIAL))
+   {
+      mfem_error("Chosen assembly level not supported yet in device mode!");
+   }
+
+   if (ext)
+   {
+      ext->Assemble();
+      return;
+   }
+
    ElementTransformation *eltrans;
    Mesh *mesh = fes -> GetMesh();
    DenseMatrix elmat, *elmat_p;
 
-   int i;
-
    if (mat == NULL)
    {
       AllocMat();
    }
 
-#ifdef MFEM_USE_OPENMP
+#ifdef MFEM_USE_LEGACY_OPENMP
    int free_element_matrices = 0;
    if (!element_matrices)
    {
@@ -311,7 +375,7 @@ void BilinearForm::Assemble (int skip_zeros)
 
    if (dbfi.Size())
    {
-      for (i = 0; i < fes -> GetNE(); i++)
+      for (int i = 0; i < fes -> GetNE(); i++)
       {
          fes->GetElementVDofs(i, vdofs);
          if (element_matrices)
@@ -368,7 +432,7 @@ void BilinearForm::Assemble (int skip_zeros)
          }
       }
 
-      for (i = 0; i < fes -> GetNBE(); i++)
+      for (int i = 0; i < fes -> GetNBE(); i++)
       {
          const int bdr_attr = mesh->GetBdrAttribute(i);
          if (bdr_attr_marker[bdr_attr-1] == 0) { continue; }
@@ -406,7 +470,7 @@ void BilinearForm::Assemble (int skip_zeros)
       Array<int> vdofs2;
 
       int nfaces = mesh->GetNumFaces();
-      for (i = 0; i < nfaces; i++)
+      for (int i = 0; i < nfaces; i++)
       {
          tr = mesh -> GetInteriorFaceTransformations (i);
          if (tr != NULL)
@@ -451,7 +515,7 @@ void BilinearForm::Assemble (int skip_zeros)
          }
       }
 
-      for (i = 0; i < fes -> GetNBE(); i++)
+      for (int i = 0; i < fes -> GetNBE(); i++)
       {
          const int bdr_attr = mesh->GetBdrAttribute(i);
          if (bdr_attr_marker[bdr_attr-1] == 0) { continue; }
@@ -477,7 +541,7 @@ void BilinearForm::Assemble (int skip_zeros)
       }
    }
 
-#ifdef MFEM_USE_OPENMP
+#ifdef MFEM_USE_LEGACY_OPENMP
    if (free_element_matrices)
    {
       FreeElementMatrices();
@@ -520,11 +584,16 @@ void BilinearForm::ConformingAssemble()
    width = mat->Width();
 }
 
-void BilinearForm::FormLinearSystem(const Array<int> &ess_tdof_list,
-                                    Vector &x, Vector &b,
-                                    SparseMatrix &A, Vector &X, Vector &B,
-                                    int copy_interior)
+void BilinearForm::FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x,
+                                    Vector &b, OperatorHandle &A, Vector &X,
+                                    Vector &B, int copy_interior)
 {
+   if (ext)
+   {
+      ext->FormLinearSystem(ess_tdof_list, x, b, A, X, B, copy_interior);
+      return;
+   }
+
    const SparseMatrix *P = fes->GetConformingProlongation();
 
    FormSystemMatrix(ess_tdof_list, A);
@@ -587,8 +656,14 @@ void BilinearForm::FormLinearSystem(const Array<int> &ess_tdof_list,
 }
 
 void BilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
-                                    SparseMatrix &A)
+                                    OperatorHandle &A)
 {
+   if (ext)
+   {
+      ext->FormSystemMatrix(ess_tdof_list, A);
+      return;
+   }
+
    // Finish the matrix assembly and perform BC elimination, storing the
    // eliminated part of the matrix.
    if (static_cond)
@@ -600,7 +675,7 @@ void BilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
          static_cond->EliminateReducedTrueDofs(diag_policy);
          static_cond->Finalize(); // finalize eliminated part
       }
-      A.MakeRef(static_cond->GetMatrix());
+      A.Reset(&static_cond->GetMatrix(), false);
    }
    else
    {
@@ -614,11 +689,11 @@ void BilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
       }
       if (hybridization)
       {
-         A.MakeRef(hybridization->GetMatrix());
+         A.Reset(&hybridization->GetMatrix(), false);
       }
       else
       {
-         A.MakeRef(*mat);
+         A.Reset(mat, false);
       }
    }
 }
@@ -626,6 +701,12 @@ void BilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
 void BilinearForm::RecoverFEMSolution(const Vector &X,
                                       const Vector &b, Vector &x)
 {
+   if (ext)
+   {
+      ext->RecoverFEMSolution(X, b, x);
+      return;
+   }
+
    const SparseMatrix *P = fes->GetConformingProlongation();
    if (!P) // conforming space
    {
@@ -687,7 +768,7 @@ void BilinearForm::ComputeElementMatrices()
    DenseMatrix tmp;
    IsoparametricTransformation eltrans;
 
-#ifdef MFEM_USE_OPENMP
+#ifdef MFEM_USE_LEGACY_OPENMP
    #pragma omp parallel for private(tmp,eltrans)
 #endif
    for (int i = 0; i < num_elements; i++)
@@ -714,7 +795,8 @@ void BilinearForm::ComputeElementMatrices()
 }
 
 void BilinearForm::EliminateEssentialBC(const Array<int> &bdr_attr_is_ess,
-                                        const Vector &sol, Vector &rhs, DiagonalPolicy dpolicy)
+                                        const Vector &sol, Vector &rhs,
+                                        DiagonalPolicy dpolicy)
 {
    Array<int> ess_dofs, conf_ess_dofs;
    fes->GetEssentialVDofs(bdr_attr_is_ess, ess_dofs);
@@ -888,6 +970,8 @@ void BilinearForm::Update(FiniteElementSpace *nfes)
    }
 
    height = width = fes->GetVSize();
+
+   if (ext) { ext->Update(); }
 }
 
 void BilinearForm::SetDiagonalPolicy(DiagonalPolicy policy)
@@ -911,6 +995,8 @@ BilinearForm::~BilinearForm()
       for (k=0; k < fbfi.Size(); k++) { delete fbfi[k]; }
       for (k=0; k < bfbfi.Size(); k++) { delete bfbfi[k]; }
    }
+
+   delete ext;
 }
 
 

fem/bilinearform.hpp

@@ -18,12 +18,32 @@
 #include "gridfunc.hpp"
 #include "linearform.hpp"
 #include "bilininteg.hpp"
+#include "bilinearform_ext.hpp"
 #include "staticcond.hpp"
 #include "hybridization.hpp"
 
 namespace mfem
 {
 
+/// Enumeration defining the assembly level for bilinear and nonlinear form
+/// classes derived from Operator.
+enum class AssemblyLevel
+{
+   /// Fully assembled form, i.e. a global sparse matrix in MFEM, Hypre or PETSC
+   /// format.
+   FULL,
+   /// Form assembled at element level, which computes and stores dense element
+   /// matrices.
+   ELEMENT,
+   /// Partially-assembled form, which computes and stores data only at
+   /// quadrature points.
+   PARTIAL,
+   /// "Matrix-free" form that computes all of its action on-the-fly without any
+   /// substantial storage.
+   NONE,
+};
+
+
 /** Class for bilinear form - "Matrix" with associated FE space and
     BLFIntegrators. */
 class BilinearForm : public Matrix
@@ -38,6 +58,14 @@ protected:
    /// FE space on which the form lives. Not owned.
    FiniteElementSpace *fes;
 
+   /// The form assembly level (full, partial, etc.)
+   AssemblyLevel assembly;
+   /// Element batch size used in the form action (1, 8, num_elems, etc.)
+   int batch;
+   /** Extension for supporting Full Assembly (FA), Element Assembly (EA),
+       Partial Assembly (PA), or Matrix Free assembly (MF). */
+   BilinearFormExtension *ext;
+
    /// Indicates the Mesh::sequence corresponding to the current state of the
    /// BilinearForm.
    long sequence;
@@ -89,6 +117,9 @@ protected:
       static_cond = NULL; hybridization = NULL;
       precompute_sparsity = 0;
       diag_policy = DIAG_KEEP;
+      assembly = AssemblyLevel::FULL;
+      batch = 1;
+      ext = NULL;
    }
 
 private:
@@ -118,6 +149,10 @@ public:
    /// Get the size of the BilinearForm as a square matrix.
    int Size() const { return height; }
 
+   /// Set the desired assembly level. The default is AssemblyLevel::FULL.
+   /** This method must be called before assembly. */
+   void SetAssemblyLevel(AssemblyLevel assembly_level);
+
    /** Enable the use of static condensation. For details see the description
        for class StaticCondensation in fem/staticcond.hpp This method should be
        called before assembly. If the number of unknowns after static
@@ -291,11 +326,12 @@ public:
    virtual const Operator *GetRestriction() const
    { return fes->GetConformingRestriction(); }
 
-   /// Form a linear system, A X = B.
-   /** Form the linear system A X = B, corresponding to the current bilinear
-       form and b(.), by applying any necessary transformations such as:
-       eliminating boundary conditions; applying conforming constraints for
-       non-conforming AMR; static condensation; hybridization.
+   /** @brief Form the linear system A X = B, corresponding to this bilinear
+       form and the linear form @a b(.). */
+   /** This method applies any necessary transformations to the linear system
+       such as: eliminating boundary conditions; applying conforming constraints
+       for non-conforming AMR; parallel assembly; static condensation;
+       hybridization.
 
        The GridFunction-size vector @a x must contain the essential b.c. The
        BilinearForm and the LinearForm-size vector @a b must be assembled.
@@ -316,12 +352,52 @@ public:
 
        NOTE: If there are no transformations, @a X simply reuses the data of
              @a x. */
+   virtual void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x,
+                                 Vector &b, OperatorHandle &A, Vector &X,
+                                 Vector &B, int copy_interior = 0);
+
+   /** @brief Form the linear system A X = B, corresponding to this bilinear
+       form and the linear form @a b(.). */
+   /** Version of the method FormLinearSystem() where the system matrix is
+       returned in the variable @a A, of type OpType, holding a *reference* to
+       the system matrix (created with the method OpType::MakeRef()). The
+       reference will be invalidated when SetOperatorType(), Update(), or the
+       destructor is called.
+
+       Currently, this method can be used only with AssemblyLevel::FULL. */
+   template <typename OpType>
    void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x, Vector &b,
-                         SparseMatrix &A, Vector &X, Vector &B,
-                         int copy_interior = 0);
+                         OpType &A, Vector &X, Vector &B,
+                         int copy_interior = 0)
+   {
+      OperatorHandle Ah;
+      FormLinearSystem(ess_tdof_list, x, b, Ah, X, B, copy_interior);
+      OpType *A_ptr = Ah.Is<OpType>();
+      MFEM_VERIFY(A_ptr, "invalid OpType used");
+      A.MakeRef(*A_ptr);
+   }
+
+   /// Form the linear system matrix @a A, see FormLinearSystem() for details.
+   virtual void FormSystemMatrix(const Array<int> &ess_tdof_list,
+                                 OperatorHandle &A);
 
    /// Form the linear system matrix A, see FormLinearSystem() for details.
-   void FormSystemMatrix(const Array<int> &ess_tdof_list, SparseMatrix &A);
+   /** Version of the method FormSystemMatrix() where the system matrix is
+       returned in the variable @a A, of type OpType, holding a *reference* to
+       the system matrix (created with the method OpType::MakeRef()). The
+       reference will be invalidated when SetOperatorType(), Update(), or the
+       destructor is called.
+
+       Currently, this method can be used only with AssemblyLevel::FULL. */
+   template <typename OpType>
+   void FormSystemMatrix(const Array<int> &ess_tdof_list, OpType &A)
+   {
+      OperatorHandle Ah;
+      FormSystemMatrix(ess_tdof_list, Ah);
+      OpType *A_ptr = Ah.Is<OpType>();
+      MFEM_VERIFY(A_ptr, "invalid OpType used");
+      A.MakeRef(*A_ptr);
+   }
 
    /// Recover the solution of a linear system formed with FormLinearSystem().
    /** Call this method after solving a linear system constructed using the
@@ -413,6 +489,7 @@ public:
    virtual ~BilinearForm();
 };
 
+
 /**
    Class for assembling of bilinear forms `a(u,v)` defined on different
    trial and test spaces. The assembled matrix `A` is such that

fem/bilinearform_ext.cpp

@@ -0,0 +1,236 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+// Implementations of classes FABilinearFormExtension, EABilinearFormExtension,
+// PABilinearFormExtension and MFBilinearFormExtension.
+
+#include "../general/forall.hpp"
+#include "bilinearform.hpp"
+
+namespace mfem
+{
+
+BilinearFormExtension::BilinearFormExtension(BilinearForm *form)
+   : Operator(form->Size()), a(form)
+{
+   // empty
+}
+
+const Operator *BilinearFormExtension::GetProlongation() const
+{
+   return a->GetProlongation();
+}
+
+const Operator *BilinearFormExtension::GetRestriction() const
+{
+   return a->GetRestriction();
+}
+
+
+// Data and methods for partially-assembled bilinear forms
+PABilinearFormExtension::PABilinearFormExtension(BilinearForm *form) :
+   BilinearFormExtension(form),
+   trialFes(a->FESpace()), testFes(a->FESpace()),
+   localX(trialFes->GetNE() * trialFes->GetFE(0)->GetDof() * trialFes->GetVDim()),
+   localY( testFes->GetNE() * testFes->GetFE(0)->GetDof() * testFes->GetVDim()),
+   elem_restrict(new ElemRestriction(*a->FESpace())) { }
+
+PABilinearFormExtension::~PABilinearFormExtension()
+{
+   delete elem_restrict;
+}
+
+void PABilinearFormExtension::Assemble()
+{
+   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
+   const int integratorCount = integrators.Size();
+   for (int i = 0; i < integratorCount; ++i)
+   {
+      integrators[i]->Assemble(*a->FESpace());
+   }
+}
+
+void PABilinearFormExtension::Update()
+{
+   FiniteElementSpace *fes = a->FESpace();
+   height = width = fes->GetVSize();
+   trialFes = fes;
+   testFes = fes;
+   localX.SetSize(trialFes->GetNE() * trialFes->GetFE(0)->GetDof() *
+                  trialFes->GetVDim());
+   localY.SetSize(testFes->GetNE() * testFes->GetFE(0)->GetDof() *
+                  testFes->GetVDim());
+   delete elem_restrict;
+   elem_restrict = new ElemRestriction(*fes);
+}
+
+void PABilinearFormExtension::FormSystemMatrix(const Array<int> &ess_tdof_list,
+                                               OperatorHandle &A)
+{
+   const Operator* trialP = trialFes->GetProlongationMatrix();
+   const Operator* testP  = testFes->GetProlongationMatrix();
+   Operator *rap = this;
+   if (trialP) { rap = new RAPOperator(*testP, *this, *trialP); }
+   const bool own_A = (rap!=this);
+   A.Reset(new ConstrainedOperator(rap, ess_tdof_list, own_A));
+}
+
+void PABilinearFormExtension::FormLinearSystem(const Array<int> &ess_tdof_list,
+                                               Vector &x, Vector &b,
+                                               OperatorHandle &A,
+                                               Vector &X, Vector &B,
+                                               int copy_interior)
+{
+   Operator *oper;
+   Operator::FormLinearSystem(ess_tdof_list, x, b, oper, X, B, copy_interior);
+   A.Reset(oper); // A will own oper
+}
+
+void PABilinearFormExtension::Mult(const Vector &x, Vector &y) const
+{
+   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
+   elem_restrict->Mult(x, localX);
+   localY = 0.0;
+   const int iSz = integrators.Size();
+   for (int i = 0; i < iSz; ++i)
+   {
+      integrators[i]->MultAssembled(localX, localY);
+   }
+   elem_restrict->MultTranspose(localY, y);
+}
+
+void PABilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
+{
+   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
+   elem_restrict->Mult(x, localX);
+   localY = 0.0;
+   const int iSz = integrators.Size();
+   for (int i = 0; i < iSz; ++i)
+   {
+      integrators[i]->MultAssembledTranspose(localX, localY);
+   }
+   elem_restrict->MultTranspose(localY, y);
+}
+
+
+ElemRestriction::ElemRestriction(const FiniteElementSpace &f)
+   : fes(f),
+     ne(fes.GetNE()),
+     vdim(fes.GetVDim()),
+     byvdim(fes.GetOrdering() == Ordering::byVDIM),
+     ndofs(fes.GetNDofs()),
+     dof(fes.GetFE(0)->GetDof()),
+     nedofs(ne*dof),
+     offsets(ndofs+1),
+     indices(ne*dof)
+{
+   for (int e = 0; e < ne; ++e)
+   {
+      const FiniteElement *fe = fes.GetFE(e);
+      const TensorBasisElement* el =
+         dynamic_cast<const TensorBasisElement*>(fe);
+      if (el) { continue; }
+      mfem_error("Finite element not supported with partial assembly");
+   }
+   const FiniteElement *fe = fes.GetFE(0);
+   const TensorBasisElement* el = dynamic_cast<const TensorBasisElement*>(fe);
+   const Array<int> &dof_map = el->GetDofMap();
+   const bool dof_map_is_identity = (dof_map.Size()==0);
+   const Table& e2dTable = fes.GetElementToDofTable();
+   const int* elementMap = e2dTable.GetJ();
+   // We'll be keeping a count of how many local nodes point to its global dof
+   for (int i = 0; i <= ndofs; ++i)
+   {
+      offsets[i] = 0;
+   }
+   for (int e = 0; e < ne; ++e)
+   {
+      for (int d = 0; d < dof; ++d)
+      {
+         const int gid = elementMap[dof*e + d];
+         ++offsets[gid + 1];
+      }
+   }
+   // Aggregate to find offsets for each global dof
+   for (int i = 1; i <= ndofs; ++i)
+   {
+      offsets[i] += offsets[i - 1];
+   }
+   // For each global dof, fill in all local nodes that point   to it
+   for (int e = 0; e < ne; ++e)
+   {
+      for (int d = 0; d < dof; ++d)
+      {
+         const int did = dof_map_is_identity?d:dof_map[d];
+         const int gid = elementMap[dof*e + did];
+         const int lid = dof*e + d;
+         indices[offsets[gid]++] = lid;
+      }
+   }
+   // We shifted the offsets vector by 1 by using it as a counter
+   // Now we shift it back.
+   for (int i = ndofs; i > 0; --i)
+   {
+      offsets[i] = offsets[i - 1];
+   }
+   offsets[0] = 0;
+}
+
+void ElemRestriction::Mult(const Vector& x, Vector& y) const
+{
+   const int vd = vdim;
+   const bool t = byvdim;
+   const DeviceArray d_offsets(offsets, ndofs+1);
+   const DeviceArray d_indices(indices, nedofs);
+   const DeviceMatrix d_x(x, t?vd:ndofs, t?ndofs:vd);
+   DeviceMatrix d_y(y, t?vd:nedofs, t?nedofs:vd);
+   MFEM_FORALL(i, ndofs,
+   {
+      const int offset = d_offsets[i];
+      const int nextOffset = d_offsets[i+1];
+      for (int c = 0; c < vd; ++c)
+      {
+         const double dofValue = d_x(t?c:i,t?i:c);
+         for (int j = offset; j < nextOffset; ++j)
+         {
+            const int idx_j = d_indices[j];
+            d_y(t?c:idx_j,t?idx_j:c) = dofValue;
+         }
+      }
+   });
+}
+
+void ElemRestriction::MultTranspose(const Vector& x, Vector& y) const
+{
+   const int vd = vdim;
+   const bool t = byvdim;
+   const DeviceArray d_offsets(offsets, ndofs+1);
+   const DeviceArray d_indices(indices, nedofs);
+   const DeviceMatrix d_x(x, t?vd:nedofs, t?nedofs:vd);
+   DeviceMatrix d_y(y, t?vd:ndofs, t?ndofs:vd);
+   MFEM_FORALL(i, ndofs,
+   {
+      const int offset = d_offsets[i];
+      const int nextOffset = d_offsets[i + 1];
+      for (int c = 0; c < vd; ++c)
+      {
+         double dofValue = 0;
+         for (int j = offset; j < nextOffset; ++j)
+         {
+            const int idx_j = d_indices[j];
+            dofValue +=  d_x(t?c:idx_j,t?idx_j:c);
+         }
+         d_y(t?c:i,t?i:c) = dofValue;
+      }
+   });
+}
+
+} // namespace mfem

fem/bilinearform_ext.hpp

@@ -0,0 +1,151 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_BILINEARFORM_EXT
+#define MFEM_BILINEARFORM_EXT
+
+#include "../config/config.hpp"
+#include "fespace.hpp"
+
+namespace mfem
+{
+
+class BilinearForm;
+
+/// Element restriction operator
+class ElemRestriction: public Operator
+{
+public:
+   const FiniteElementSpace &fes;
+   const int ne;
+   const int vdim;
+   const bool byvdim;
+   const int ndofs;
+   const int dof;
+   const int nedofs;
+   Array<int> offsets;
+   Array<int> indices;
+public:
+   ElemRestriction(const FiniteElementSpace&);
+   void Mult(const Vector &x, Vector &y) const;
+   void MultTranspose(const Vector &x, Vector &y) const;
+};
+
+
+class BilinearFormExtension : public Operator
+{
+protected:
+   BilinearForm *a; ///< Not owned
+
+public:
+   BilinearFormExtension(BilinearForm *form);
+
+   /// Get the finite element space prolongation matrix
+   virtual const Operator *GetProlongation() const;
+
+   /// Get the finite element space restriction matrix
+   virtual const Operator *GetRestriction() const;
+
+   virtual void Assemble() = 0;
+   virtual void FormSystemMatrix(const Array<int> &ess_tdof_list,
+                                 OperatorHandle &A) = 0;
+   virtual void FormLinearSystem(const Array<int> &ess_tdof_list,
+                                 Vector &x, Vector &b,
+                                 OperatorHandle &A, Vector &X, Vector &B,
+                                 int copy_interior = 0) = 0;
+   virtual void Update() = 0;
+};
+
+/// Data and methods for fully-assembled bilinear forms
+class FABilinearFormExtension : public BilinearFormExtension
+{
+public:
+   FABilinearFormExtension(BilinearForm *form)
+      : BilinearFormExtension(form) { }
+
+   /// TODO
+   void Assemble() {}
+   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A) {}
+   void FormLinearSystem(const Array<int> &ess_tdof_list,
+                         Vector &x, Vector &b,
+                         OperatorHandle &A, Vector &X, Vector &B,
+                         int copy_interior = 0) {}
+   void Mult(const Vector &x, Vector &y) const {}
+   void MultTranspose(const Vector &x, Vector &y) const {}
+   ~FABilinearFormExtension() {}
+};
+
+/// Data and methods for element-assembled bilinear forms
+class EABilinearFormExtension : public BilinearFormExtension
+{
+public:
+   EABilinearFormExtension(BilinearForm *form)
+      : BilinearFormExtension(form) { }
+
+   /// TODO
+   void Assemble() {}
+   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A) {}
+   void FormLinearSystem(const Array<int> &ess_tdof_list,
+                         Vector &x, Vector &b,
+                         OperatorHandle &A, Vector &X, Vector &B,
+                         int copy_interior = 0) {}
+   void Mult(const Vector &x, Vector &y) const {}
+   void MultTranspose(const Vector &x, Vector &y) const {}
+   ~EABilinearFormExtension() {}
+};
+
+/// Data and methods for partially-assembled bilinear forms
+class PABilinearFormExtension : public BilinearFormExtension
+{
+protected:
+   const FiniteElementSpace *trialFes, *testFes;
+   mutable Vector localX, localY;
+   ElemRestriction *elem_restrict;
+
+public:
+   PABilinearFormExtension(BilinearForm*);
+
+   void Assemble();
+   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A);
+   void FormLinearSystem(const Array<int> &ess_tdof_list,
+                         Vector &x, Vector &b,
+                         OperatorHandle &A, Vector &X, Vector &B,
+                         int copy_interior = 0);
+
+   void Mult(const Vector &x, Vector &y) const;
+   void MultTranspose(const Vector &x, Vector &y) const;
+   void Update();
+
+   ~PABilinearFormExtension();
+};
+
+/// Data and methods for matrix-free bilinear forms
+class MFBilinearFormExtension : public BilinearFormExtension
+{
+public:
+   MFBilinearFormExtension(BilinearForm *form)
+      : BilinearFormExtension(form) { }
+
+   /// TODO
+   void Assemble() {}
+   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A) {}
+   void FormLinearSystem(const Array<int> &ess_tdof_list,
+                         Vector &x, Vector &b,
+                         OperatorHandle &A, Vector &X, Vector &B,
+                         int copy_interior = 0) {}
+   void Mult(const Vector &x, Vector &y) const {}
+   void MultTranspose(const Vector &x, Vector &y) const {}
+   ~MFBilinearFormExtension() {}
+};
+
+}
+
+#endif

fem/bilininteg.cpp

@@ -19,6 +19,23 @@ using namespace std;
 
 namespace mfem
 {
+void BilinearFormIntegrator::Assemble(const FiniteElementSpace&)
+{
+   mfem_error ("BilinearFormIntegrator::Assemble (...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::MultAssembled(Vector&, Vector&)
+{
+   mfem_error ("BilinearFormIntegrator::MultAssembled (...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::MultAssembledTranspose(Vector&, Vector&)
+{
+   mfem_error ("BilinearFormIntegrator::MultAssembledTranspose (...)\n"
+               "   is not implemented for this class.");
+}
 
 void BilinearFormIntegrator::AssembleElementMatrix (
    const FiniteElement &el, ElementTransformation &Trans,

fem/bilininteg.hpp

@@ -14,6 +14,8 @@
 
 #include "../config/config.hpp"
 #include "nonlininteg.hpp"
+#include "fespace.hpp"
+#include "bilininteg_ext.hpp"
 
 namespace mfem
 {
@@ -21,11 +23,20 @@ namespace mfem
 /// Abstract base class BilinearFormIntegrator
 class BilinearFormIntegrator : public NonlinearFormIntegrator
 {
-protected:
+public:
    BilinearFormIntegrator(const IntegrationRule *ir = NULL) :
       NonlinearFormIntegrator(ir) { }
 
 public:
+   /// Method defining partial assembly.
+   virtual void Assemble(const FiniteElementSpace&);
+
+   /// Method for partially assembled action.
+   virtual void MultAssembled(Vector&, Vector&);
+
+   /// Method for partially assembled transposed action.
+   virtual void MultAssembledTranspose(Vector&, Vector&);
+
    /// Given a particular Finite Element computes the element matrix elmat.
    virtual void AssembleElementMatrix(const FiniteElement &el,
                                       ElementTransformation &Trans,
@@ -1634,16 +1645,19 @@ private:
 #endif
    Coefficient *Q;
    MatrixCoefficient *MQ;
-
+   // PA extension
+   DofToQuad *maps;
+   GeometryExtension *geom;
+   int dim, ne, dofs1D, quad1D;
 public:
    /// Construct a diffusion integrator with coefficient Q = 1
-   DiffusionIntegrator() { Q = NULL; MQ = NULL; }
+   DiffusionIntegrator() { Q = NULL; MQ = NULL; maps = NULL; geom = NULL; }
 
    /// Construct a diffusion integrator with a scalar coefficient q
-   DiffusionIntegrator (Coefficient &q) : Q(&q) { MQ = NULL; }
+   DiffusionIntegrator (Coefficient &q) : Q(&q) { MQ = NULL; maps = NULL; geom = NULL; }
 
    /// Construct a diffusion integrator with a matrix coefficient q
-   DiffusionIntegrator (MatrixCoefficient &q) : MQ(&q) { Q = NULL; }
+   DiffusionIntegrator (MatrixCoefficient &q) : MQ(&q) { Q = NULL; maps = NULL; geom = NULL; }
 
    /** Given a particular Finite Element
        computes the element stiffness matrix elmat. */
@@ -1670,6 +1684,12 @@ public:
    virtual double ComputeFluxEnergy(const FiniteElement &fluxelem,
                                     ElementTransformation &Trans,
                                     Vector &flux, Vector *d_energy = NULL);
+
+   /// PA extension
+   virtual void Assemble(const FiniteElementSpace&);
+   virtual void MultAssembled(Vector&, Vector&);
+
+   virtual ~DiffusionIntegrator();
 };
 
 /** Class for local mass matrix assembling a(u,v) := (Q u, v) */
@@ -1680,13 +1700,17 @@ protected:
    Vector shape, te_shape;
 #endif
    Coefficient *Q;
-
+   // PA extension
+   Vector vec;
+   DofToQuad *maps;
+   GeometryExtension *geom;
+   int dim, ne, nq, dofs1D, quad1D;
 public:
    MassIntegrator(const IntegrationRule *ir = NULL)
-      : BilinearFormIntegrator(ir) { Q = NULL; }
+      : BilinearFormIntegrator(ir) { Q = NULL; maps = NULL; geom = NULL; }
    /// Construct a mass integrator with coefficient q
    MassIntegrator(Coefficient &q, const IntegrationRule *ir = NULL)
-      : BilinearFormIntegrator(ir), Q(&q) { }
+      : BilinearFormIntegrator(ir), Q(&q) { maps = NULL; geom = NULL; }
 
    /** Given a particular Finite Element
        computes the element mass matrix elmat. */
@@ -1697,6 +1721,11 @@ public:
                                        const FiniteElement &test_fe,
                                        ElementTransformation &Trans,
                                        DenseMatrix &elmat);
+   /// PA extension
+   virtual void Assemble(const FiniteElementSpace&);
+   virtual void MultAssembled(Vector&, Vector&);
+
+   virtual ~MassIntegrator();
 };
 
 class BoundaryMassIntegrator : public MassIntegrator

fem/bilininteg_ext.hpp

@@ -0,0 +1,80 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_BILININTEG_EXT
+#define MFEM_BILININTEG_EXT
+
+#include "fespace.hpp"
+
+namespace mfem
+{
+
+/// GeometryExtension
+class GeometryExtension
+{
+public:
+   Array<int> eMap;
+   Array<double> nodes;
+   Array<double> X, J, invJ, detJ;
+   static GeometryExtension* Get(const FiniteElementSpace&,
+                                 const IntegrationRule&);
+   static GeometryExtension* Get(const FiniteElementSpace&,
+                                 const IntegrationRule&,
+                                 const Vector&);
+   static void ReorderByVDim(const GridFunction*);
+   static void ReorderByNodes(const GridFunction*);
+};
+
+/// DofToQuad
+class DofToQuad
+{
+private:
+   std::string hash;
+public:
+   ~DofToQuad();
+   void operator=(DofToQuad&);
+   void operator=(DofToQuad const&);
+public:
+   Array<double> W, B, G, Bt, Gt;
+public:
+   static DofToQuad* Get(const FiniteElementSpace&,
+                         const IntegrationRule&,
+                         const bool = false);
+   static DofToQuad* Get(const FiniteElementSpace&,
+                         const FiniteElementSpace&,
+                         const IntegrationRule&,
+                         const bool = false);
+   static DofToQuad* Get(const FiniteElement&,
+                         const FiniteElement&,
+                         const IntegrationRule&,
+                         const bool = false);
+   static DofToQuad* GetTensorMaps(const FiniteElement&,
+                                   const FiniteElement&,
+                                   const IntegrationRule&,
+                                   const bool = false);
+   static DofToQuad* GetD2QTensorMaps(const FiniteElement&,
+                                      const IntegrationRule&,
+                                      const bool = false);
+   static DofToQuad* GetSimplexMaps(const FiniteElement&,
+                                    const IntegrationRule&,
+                                    const bool = false);
+   static DofToQuad* GetSimplexMaps(const FiniteElement&,
+                                    const FiniteElement&,
+                                    const IntegrationRule&,
+                                    const bool = false);
+   static DofToQuad* GetD2QSimplexMaps(const FiniteElement&,
+                                       const IntegrationRule&,
+                                       const bool = false);
+};
+
+}
+
+#endif

fem/coefficient.cpp

@@ -28,6 +28,11 @@ double PWConstCoefficient::Eval(ElementTransformation & T,
    return (constants(att-1));
 }
 
+DeviceFunctionCoefficientPtr FunctionCoefficient::GetDeviceFunction()
+{
+   return DeviceFunction;
+}
+
 double FunctionCoefficient::Eval(ElementTransformation & T,
                                  const IntegrationPoint & ip)
 {
@@ -40,6 +45,10 @@ double FunctionCoefficient::Eval(ElementTransformation & T,
    {
       return ((*Function)(transip));
    }
+   else if (DeviceFunction)
+   {
+      return ((*DeviceFunction)(Vector3(x)));
+   }
    else
    {
       return (*TDFunction)(transip, GetTime());

fem/coefficient.hpp

@@ -96,7 +96,7 @@ public:
    { constants.SetSize(c.Size()); constants=c; }
 
    /// Update constants
-   void UpdateConstants(Vector &c) {constants.SetSize(c.Size()); constants=c;}
+   void UpdateConstants(Vector &c) { constants.SetSize(c.Size()); constants=c; }
 
    /// Member function to access or modify the value of the i-th constant
    double &operator()(int i) { return constants(i-1); }
@@ -112,12 +112,15 @@ public:
                        const IntegrationPoint &ip);
 };
 
+typedef double (*DeviceFunctionCoefficientPtr)(const Vector3&);
+
 /// class for C-function coefficient
 class FunctionCoefficient : public Coefficient
 {
 protected:
    double (*Function)(const Vector &);
    double (*TDFunction)(const Vector &, double);
+   double (*DeviceFunction)(const Vector3&);
 
 public:
    /// Define a time-independent coefficient from a C-function
@@ -125,6 +128,7 @@ public:
    {
       Function = f;
       TDFunction = NULL;
+      DeviceFunction = NULL;
    }
 
    /// Define a time-dependent coefficient from a C-function
@@ -132,6 +136,16 @@ public:
    {
       Function = NULL;
       TDFunction = tdf;
+      DeviceFunction = NULL;
+   }
+
+   /// Define a time-independent coefficient from a C-function using
+   /// Vector3 instead of a Vector.
+   FunctionCoefficient(double (*df)(const Vector3 &))
+   {
+      Function = NULL;
+      TDFunction = NULL;
+      DeviceFunction = df;
    }
 
    /// (DEPRECATED) Define a time-independent coefficient from a C-function
@@ -141,6 +155,7 @@ public:
    {
       Function = reinterpret_cast<double(*)(const Vector&)>(f);
       TDFunction = NULL;
+      DeviceFunction = NULL;
    }
 
    /// (DEPRECATED) Define a time-dependent coefficient from a C-function
@@ -150,11 +165,17 @@ public:
    {
       Function = NULL;
       TDFunction = reinterpret_cast<double(*)(const Vector&,double)>(tdf);
+      DeviceFunction = NULL;
    }
 
    /// Evaluate coefficient
    virtual double Eval(ElementTransformation &T,
                        const IntegrationPoint &ip);
+
+   /// Return the coefficient's C-function that uses Vector3.
+   /// Warning: for now, the returned function can only be used on the
+   /// host inside a MFEM_FORALL.
+   DeviceFunctionCoefficientPtr GetDeviceFunction();
 };
 
 class GridFunction;
@@ -448,6 +469,7 @@ public:
    void SetGridFunction(GridFunction *gf);
    GridFunction * GetGridFunction() const { return GridFunc; }
 
+   using VectorCoefficient::Eval;
    virtual void Eval(Vector &V, ElementTransformation &T,
                      const IntegrationPoint &ip);
 
@@ -790,6 +812,7 @@ public:
    /// Evaluate the coefficient
    virtual void Eval(Vector &V, ElementTransformation &T,
                      const IntegrationPoint &ip);
+   using VectorCoefficient::Eval;
 };
 
 /// Vector coefficient defined as a product of a scalar and a vector
@@ -804,6 +827,7 @@ public:
 
    virtual void Eval(Vector &V, ElementTransformation &T,
                      const IntegrationPoint &ip);
+   using VectorCoefficient::Eval;
 };
 
 /// Vector coefficient defined as a cross product of two vectors
@@ -821,6 +845,7 @@ public:
 
    virtual void Eval(Vector &V, ElementTransformation &T,
                      const IntegrationPoint &ip);
+   using VectorCoefficient::Eval;
 };
 
 /// Vector coefficient defined as a matrix vector product
@@ -838,6 +863,7 @@ public:
 
    virtual void Eval(Vector &V, ElementTransformation &T,
                      const IntegrationPoint &ip);
+   using VectorCoefficient::Eval;
 };
 
 /// Matrix coefficient defined as the identity of dimension d

fem/complex_fem.cpp

@@ -0,0 +1,784 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "complex_fem.hpp"
+
+using namespace std;
+
+namespace mfem
+{
+
+ComplexGridFunction::ComplexGridFunction(FiniteElementSpace *fes)
+   : Vector(2*(fes->GetVSize()))
+{
+   gfr_ = new GridFunction(fes, &data[0]);
+   gfi_ = new GridFunction(fes, &data[fes->GetVSize()]);
+}
+
+void
+ComplexGridFunction::Update()
+{
+   FiniteElementSpace * fes = gfr_->FESpace();
+
+   int vsize = fes->GetVSize();
+
+   const Operator *T = fes->GetUpdateOperator();
+   if (T)
+   {
+      // Update the individual GridFunction objects.  This will allocate
+      // new data arrays for each GridFunction.
+      gfr_->Update();
+      gfi_->Update();
+
+      // Our data array now contains old data as well as being the wrong size
+      // so reallocate it.
+      this->SetSize(2 * vsize);
+
+      // Create temporary vectors which point to the new data array
+      Vector gf_r(&data[0], vsize);
+      Vector gf_i(&data[vsize], vsize);
+
+      // Copy the updated GridFunctions into the new data array
+      gf_r = *gfr_;
+      gf_i = *gfi_;
+
+      // Replace the individual data arrays with pointers into the new data array
+      gfr_->NewDataAndSize(&data[0], vsize);
+      gfi_->NewDataAndSize(&data[vsize], vsize);
+   }
+   else
+   {
+      // The existing data will not be transferred to the new GridFunctions
+      // so delete it a allocate a new array
+      this->SetSize(2 * vsize);
+
+      // Point the individual GridFunctions to the new data array
+      gfr_->NewDataAndSize(&data[0], vsize);
+      gfi_->NewDataAndSize(&data[vsize], vsize);
+
+      // These updates will only set the proper 'sequence' value within
+      // the individual GridFunction objects because their sizes are
+      // already correct
+      gfr_->Update();
+      gfi_->Update();
+   }
+}
+
+void
+ComplexGridFunction::ProjectCoefficient(Coefficient &real_coeff,
+                                        Coefficient &imag_coeff)
+{
+   gfr_->ProjectCoefficient(real_coeff);
+   gfi_->ProjectCoefficient(imag_coeff);
+}
+
+void
+ComplexGridFunction::ProjectCoefficient(VectorCoefficient &real_vcoeff,
+                                        VectorCoefficient &imag_vcoeff)
+{
+   gfr_->ProjectCoefficient(real_vcoeff);
+   gfi_->ProjectCoefficient(imag_vcoeff);
+}
+
+void
+ComplexGridFunction::ProjectBdrCoefficient(Coefficient &real_coeff,
+                                           Coefficient &imag_coeff,
+                                           Array<int> &attr)
+{
+   gfr_->ProjectBdrCoefficient(real_coeff, attr);
+   gfi_->ProjectBdrCoefficient(imag_coeff, attr);
+}
+
+void
+ComplexGridFunction::ProjectBdrCoefficientNormal(VectorCoefficient &real_vcoeff,
+                                                 VectorCoefficient &imag_vcoeff,
+                                                 Array<int> &attr)
+{
+   gfr_->ProjectBdrCoefficientNormal(real_vcoeff, attr);
+   gfi_->ProjectBdrCoefficientNormal(imag_vcoeff, attr);
+}
+
+void
+ComplexGridFunction::ProjectBdrCoefficientTangent(VectorCoefficient
+                                                  &real_vcoeff,
+                                                  VectorCoefficient
+                                                  &imag_vcoeff,
+                                                  Array<int> &attr)
+{
+   gfr_->ProjectBdrCoefficientTangent(real_vcoeff, attr);
+   gfi_->ProjectBdrCoefficientTangent(imag_vcoeff, attr);
+}
+
+
+ComplexLinearForm::ComplexLinearForm(FiniteElementSpace *f,
+                                     ComplexOperator::Convention convention)
+   : Vector(2*(f->GetVSize())),
+     conv_(convention)
+{
+   lfr_ = new LinearForm(f, &data[0]);
+   lfi_ = new LinearForm(f, &data[f->GetVSize()]);
+}
+
+ComplexLinearForm::~ComplexLinearForm()
+{
+   delete lfr_;
+   delete lfi_;
+}
+
+void
+ComplexLinearForm::AddDomainIntegrator(LinearFormIntegrator *lfi_real,
+                                       LinearFormIntegrator *lfi_imag)
+{
+   if ( lfi_real ) { lfr_->AddDomainIntegrator(lfi_real); }
+   if ( lfi_imag ) { lfi_->AddDomainIntegrator(lfi_imag); }
+}
+
+void
+ComplexLinearForm::Update()
+{
+   FiniteElementSpace *fes = lfr_->FESpace();
+
+   this->Update(fes);
+}
+
+void
+ComplexLinearForm::Update(FiniteElementSpace *fes)
+{
+   int vsize = fes->GetVSize();
+   SetSize(2 * vsize);
+
+   Vector lfr(&data[0], vsize);
+   Vector lfi(&data[vsize], vsize);
+
+   lfr_->Update(fes, lfr, 0);
+   lfi_->Update(fes, lfi, 0);
+}
+
+void
+ComplexLinearForm::Assemble()
+{
+   lfr_->Assemble();
+   lfi_->Assemble();
+   if (conv_ == ComplexOperator::BLOCK_SYMMETRIC)
+   {
+      *lfi_ *= -1.0;
+   }
+}
+
+complex<double>
+ComplexLinearForm::operator()(const ComplexGridFunction &gf) const
+{
+   double s = (conv_ == ComplexOperator::HERMITIAN)?1.0:-1.0;
+   return complex<double>((*lfr_)(gf.real()) - s * (*lfi_)(gf.imag()),
+                          (*lfr_)(gf.imag()) + s * (*lfi_)(gf.real()));
+}
+
+
+SesquilinearForm::SesquilinearForm(FiniteElementSpace *f,
+                                   ComplexOperator::Convention convention)
+   : conv_(convention),
+     blfr_(new BilinearForm(f)),
+     blfi_(new BilinearForm(f))
+{}
+
+SesquilinearForm::~SesquilinearForm()
+{
+   delete blfr_;
+   delete blfi_;
+}
+
+void SesquilinearForm::AddDomainIntegrator(BilinearFormIntegrator *bfi_real,
+                                           BilinearFormIntegrator *bfi_imag)
+{
+   if (bfi_real) { blfr_->AddDomainIntegrator(bfi_real); }
+   if (bfi_imag) { blfi_->AddDomainIntegrator(bfi_imag); }
+}
+
+void
+SesquilinearForm::AddBoundaryIntegrator(BilinearFormIntegrator *bfi_real,
+                                        BilinearFormIntegrator *bfi_imag)
+{
+   if (bfi_real) { blfr_->AddBoundaryIntegrator(bfi_real); }
+   if (bfi_imag) { blfi_->AddBoundaryIntegrator(bfi_imag); }
+}
+
+void
+SesquilinearForm::AddBoundaryIntegrator(BilinearFormIntegrator *bfi_real,
+                                        BilinearFormIntegrator *bfi_imag,
+                                        Array<int> & bdr_marker)
+{
+   if (bfi_real) { blfr_->AddBoundaryIntegrator(bfi_real, bdr_marker); }
+   if (bfi_imag) { blfi_->AddBoundaryIntegrator(bfi_imag, bdr_marker); }
+}
+
+void
+SesquilinearForm::Assemble(int skip_zeros)
+{
+   blfr_->Assemble(skip_zeros);
+   blfi_->Assemble(skip_zeros);
+}
+
+void
+SesquilinearForm::Finalize(int skip_zeros)
+{
+   blfr_->Finalize(skip_zeros);
+   blfi_->Finalize(skip_zeros);
+}
+
+ComplexSparseMatrix *
+SesquilinearForm::AssembleCompSpMat()
+{
+   return new ComplexSparseMatrix(&blfr_->SpMat(),
+                                  &blfi_->SpMat(),
+                                  false, false, conv_);
+
+}
+
+void
+SesquilinearForm::FormLinearSystem(const Array<int> &ess_tdof_list,
+                                   Vector &x, Vector &b,
+                                   OperatorHandle &A,
+                                   Vector &X, Vector &B,
+                                   int ci)
+{
+   FiniteElementSpace * fes = blfr_->FESpace();
+
+   int vsize  = fes->GetVSize();
+   // int tvsize = pfes->GetTrueVSize();
+
+   double s = (conv_ == ComplexOperator::HERMITIAN)?1.0:-1.0;
+
+   // Allocate temporary vectors
+   Vector b_0(vsize);  b_0 = 0.0;
+   // Vector B_0(tvsize); B_0 = 0.0;
+
+   // Extract the real and imaginary parts of the input vectors
+   MFEM_ASSERT(x.Size() == 2 * vsize, "Input GridFunction of incorrect size!");
+   Vector x_r(x.GetData(), vsize);
+   Vector x_i(&(x.GetData())[vsize], vsize);
+
+   MFEM_ASSERT(b.Size() == 2 * vsize, "Input LinearForm of incorrect size!");
+   Vector b_r(b.GetData(), vsize);
+   Vector b_i(&(b.GetData())[vsize], vsize);
+   b_i *= s;
+   /*
+   X.SetSize(2 * tvsize);
+   Vector X_r(X.GetData(), tvsize);
+   Vector X_i(&(X.GetData())[tvsize], tvsize);
+
+   B.SetSize(2 * tvsize);
+   Vector B_r(B.GetData(), tvsize);
+   Vector B_i(&(B.GetData())[tvsize], tvsize);
+   */
+   SparseMatrix * A_r = new SparseMatrix;
+   SparseMatrix * A_i = new SparseMatrix;
+   Vector X_0, B_0;
+
+   b_0 = b_r;
+   blfr_->FormLinearSystem(ess_tdof_list, x_r, b_r, *A_r, X_0, B_0, ci);
+
+   int tvsize = B_0.Size();
+   X.SetSize(2 * tvsize);
+   B.SetSize(2 * tvsize);
+   Vector X_r(X.GetData(), tvsize);
+   Vector X_i(&(X.GetData())[tvsize], tvsize);
+   Vector B_r(B.GetData(), tvsize);
+   Vector B_i(&(B.GetData())[tvsize], tvsize);
+   X_r = X_0; B_r = B_0;
+
+   b_0 = 0.0;
+   blfi_->FormLinearSystem(ess_tdof_list, x_i, b_0, *A_i, X_0, B_0, false);
+   B_r -= B_0;
+
+   b_0 = b_i;
+   blfr_->FormLinearSystem(ess_tdof_list, x_i, b_0, *A_r, X_0, B_0, ci);
+   X_i = X_0; B_i = B_0;
+
+   b_0 = 0.0;
+   blfi_->FormLinearSystem(ess_tdof_list, x_r, b_0, *A_i, X_0, B_0, false);
+   B_i += B_0;
+
+   B_i *= s;
+   b_i *= s;
+
+   // A = A_r + i A_i
+   A.Clear();
+   ComplexSparseMatrix * A_sp =
+      new ComplexSparseMatrix(A_r, A_i, true, true, conv_);
+   A.Reset<ComplexSparseMatrix>(A_sp, true);
+}
+
+void
+SesquilinearForm::RecoverFEMSolution(const Vector &X, const Vector &b,
+                                     Vector &x)
+{
+   FiniteElementSpace * fes = blfr_->FESpace();
+
+   const SparseMatrix *P = fes->GetConformingProlongation();
+
+   int vsize  = fes->GetVSize();
+   int tvsize = X.Size() / 2;
+
+   Vector X_r(X.GetData(), tvsize);
+   Vector X_i(&(X.GetData())[tvsize], tvsize);
+
+   Vector x_r(x.GetData(), vsize);
+   Vector x_i(&(x.GetData())[vsize], vsize);
+
+   if (!P)
+   {
+      x = X;
+   }
+   else
+   {
+      // Apply conforming prolongation
+      P->Mult(X_r, x_r);
+      P->Mult(X_i, x_i);
+   }
+}
+
+void
+SesquilinearForm::Update(FiniteElementSpace *nfes)
+{
+   if ( blfr_ ) { blfr_->Update(nfes); }
+   if ( blfi_ ) { blfi_->Update(nfes); }
+}
+
+
+#ifdef MFEM_USE_MPI
+
+ParComplexGridFunction::ParComplexGridFunction(ParFiniteElementSpace *pfes)
+   : Vector(2*(pfes->GetVSize()))
+{
+   pgfr_ = new ParGridFunction(pfes, &data[0]);
+   pgfi_ = new ParGridFunction(pfes, &data[pfes->GetVSize()]);
+}
+
+void
+ParComplexGridFunction::Update()
+{
+   ParFiniteElementSpace * pfes = pgfr_->ParFESpace();
+
+   int vsize = pfes->GetVSize();
+
+   const Operator *T = pfes->GetUpdateOperator();
+   if (T)
+   {
+      // Update the individual GridFunction objects.  This will allocate
+      // new data arrays for each GridFunction.
+      pgfr_->Update();
+      pgfi_->Update();
+
+      // Our data array now contains old data as well as being the wrong size
+      // so reallocate it.
+      this->SetSize(2 * vsize);
+
+      // Create temporary vectors which point to the new data array
+      Vector gf_r(&data[0], vsize);
+      Vector gf_i(&data[vsize], vsize);
+
+      // Copy the updated GridFunctions into the new data array
+      gf_r = *pgfr_;
+      gf_i = *pgfi_;
+
+      // Replace the individual data arrays with pointers into the new data array
+      pgfr_->NewDataAndSize(&data[0], vsize);
+      pgfi_->NewDataAndSize(&data[vsize], vsize);
+   }
+   else
+   {
+      // The existing data will not be transferred to the new GridFunctions
+      // so delete it a allocate a new array
+      this->SetSize(2 * vsize);
+
+      // Point the individual GridFunctions to the new data array
+      pgfr_->NewDataAndSize(&data[0], vsize);
+      pgfi_->NewDataAndSize(&data[vsize], vsize);
+
+      // These updates will only set the proper 'sequence' value within
+      // the individual GridFunction objects because their sizes are
+      // already correct
+      pgfr_->Update();
+      pgfi_->Update();
+   }
+}
+
+void
+ParComplexGridFunction::ProjectCoefficient(Coefficient &real_coeff,
+                                           Coefficient &imag_coeff)
+{
+   pgfr_->ProjectCoefficient(real_coeff);
+   pgfi_->ProjectCoefficient(imag_coeff);
+}
+
+void
+ParComplexGridFunction::ProjectCoefficient(VectorCoefficient &real_vcoeff,
+                                           VectorCoefficient &imag_vcoeff)
+{
+   pgfr_->ProjectCoefficient(real_vcoeff);
+   pgfi_->ProjectCoefficient(imag_vcoeff);
+}
+
+void
+ParComplexGridFunction::ProjectBdrCoefficient(Coefficient &real_coeff,
+                                              Coefficient &imag_coeff,
+                                              Array<int> &attr)
+{
+   pgfr_->ProjectBdrCoefficient(real_coeff, attr);
+   pgfi_->ProjectBdrCoefficient(imag_coeff, attr);
+}
+
+void
+ParComplexGridFunction::ProjectBdrCoefficientNormal(VectorCoefficient
+                                                    &real_vcoeff,
+                                                    VectorCoefficient
+                                                    &imag_vcoeff,
+                                                    Array<int> &attr)
+{
+   pgfr_->ProjectBdrCoefficientNormal(real_vcoeff, attr);
+   pgfi_->ProjectBdrCoefficientNormal(imag_vcoeff, attr);
+}
+
+void
+ParComplexGridFunction::ProjectBdrCoefficientTangent(VectorCoefficient
+                                                     &real_vcoeff,
+                                                     VectorCoefficient
+                                                     &imag_vcoeff,
+                                                     Array<int> &attr)
+{
+   pgfr_->ProjectBdrCoefficientTangent(real_vcoeff, attr);
+   pgfi_->ProjectBdrCoefficientTangent(imag_vcoeff, attr);
+}
+
+void
+ParComplexGridFunction::Distribute(const Vector *tv)
+{
+   ParFiniteElementSpace * pfes = pgfr_->ParFESpace();
+   HYPRE_Int size = pfes->GetTrueVSize();
+
+   double * tvd = tv->GetData();
+   Vector tvr(tvd, size);
+   Vector tvi(&tvd[size], size);
+
+   pgfr_->Distribute(tvr);
+   pgfi_->Distribute(tvi);
+}
+
+void
+ParComplexGridFunction::ParallelProject(Vector &tv) const
+{
+   ParFiniteElementSpace * pfes = pgfr_->ParFESpace();
+   HYPRE_Int size = pfes->GetTrueVSize();
+
+   double * tvd = tv.GetData();
+   Vector tvr(tvd, size);
+   Vector tvi(&tvd[size], size);
+
+   pgfr_->ParallelProject(tvr);
+   pgfi_->ParallelProject(tvi);
+}
+
+
+ParComplexLinearForm::ParComplexLinearForm(ParFiniteElementSpace *pfes,
+                                           ComplexOperator::Convention
+                                           convention)
+   : Vector(2*(pfes->GetVSize())),
+     conv_(convention)
+{
+   plfr_ = new ParLinearForm(pfes, &data[0]);
+   plfi_ = new ParLinearForm(pfes, &data[pfes->GetVSize()]);
+
+   HYPRE_Int * tdof_offsets = pfes->GetTrueDofOffsets();
+
+   int n = (HYPRE_AssumedPartitionCheck()) ? 2 : pfes->GetNRanks();
+   tdof_offsets_ = new HYPRE_Int[n+1];
+
+   for (int i=0; i<=n; i++)
+   {
+      tdof_offsets_[i] = 2 * tdof_offsets[i];
+   }
+}
+
+ParComplexLinearForm::~ParComplexLinearForm()
+{
+   delete plfr_;
+   delete plfi_;
+   delete [] tdof_offsets_;
+}
+
+void
+ParComplexLinearForm::AddDomainIntegrator(LinearFormIntegrator *lfi_real,
+                                          LinearFormIntegrator *lfi_imag)
+{
+   if ( lfi_real ) { plfr_->AddDomainIntegrator(lfi_real); }
+   if ( lfi_imag ) { plfi_->AddDomainIntegrator(lfi_imag); }
+}
+
+void
+ParComplexLinearForm::Update(ParFiniteElementSpace *pf)
+{
+   ParFiniteElementSpace *pfes = (pf!=NULL)?pf:plfr_->ParFESpace();
+   int vsize = pfes->GetVSize();
+   SetSize(2 * vsize);
+
+   Vector plfr(&data[0], vsize);
+   Vector plfi(&data[vsize], vsize);
+
+   plfr_->Update(pfes, plfr, 0);
+   plfi_->Update(pfes, plfi, 0);
+}
+
+void
+ParComplexLinearForm::Assemble()
+{
+   plfr_->Assemble();
+   plfi_->Assemble();
+   if (conv_ == ComplexOperator::BLOCK_SYMMETRIC)
+   {
+      *plfi_ *= -1.0;
+   }
+}
+
+void
+ParComplexLinearForm::ParallelAssemble(Vector &tv)
+{
+   HYPRE_Int size = plfr_->ParFESpace()->GetTrueVSize();
+
+   double * tvd = tv.GetData();
+   Vector tvr(tvd, size);
+   Vector tvi(&tvd[size], size);
+
+   plfr_->ParallelAssemble(tvr);
+   plfi_->ParallelAssemble(tvi);
+}
+
+HypreParVector *
+ParComplexLinearForm::ParallelAssemble()
+{
+   const ParFiniteElementSpace * pfes = plfr_->ParFESpace();
+
+   HypreParVector * tv = new HypreParVector(pfes->GetComm(),
+                                            2*(pfes->GlobalTrueVSize()),
+                                            tdof_offsets_);
+
+   HYPRE_Int size = pfes->GetTrueVSize();
+
+   double * tvd = tv->GetData();
+   Vector tvr(tvd, size);
+   Vector tvi(&tvd[size], size);
+
+   plfr_->ParallelAssemble(tvr);
+   plfi_->ParallelAssemble(tvi);
+
+   return tv;
+}
+
+complex<double>
+ParComplexLinearForm::operator()(const ParComplexGridFunction &gf) const
+{
+   double s = (conv_ == ComplexOperator::HERMITIAN)?1.0:-1.0;
+   return complex<double>((*plfr_)(gf.real()) - s * (*plfi_)(gf.imag()),
+                          (*plfr_)(gf.imag()) + s * (*plfi_)(gf.real()));
+}
+
+
+ParSesquilinearForm::ParSesquilinearForm(ParFiniteElementSpace *pf,
+                                         ComplexOperator::Convention
+                                         convention)
+   : conv_(convention),
+     pblfr_(new ParBilinearForm(pf)),
+     pblfi_(new ParBilinearForm(pf))
+{}
+
+ParSesquilinearForm::~ParSesquilinearForm()
+{
+   delete pblfr_;
+   delete pblfi_;
+}
+
+void ParSesquilinearForm::AddDomainIntegrator(BilinearFormIntegrator *bfi_real,
+                                              BilinearFormIntegrator *bfi_imag)
+{
+   if (bfi_real) { pblfr_->AddDomainIntegrator(bfi_real); }
+   if (bfi_imag) { pblfi_->AddDomainIntegrator(bfi_imag); }
+}
+
+void
+ParSesquilinearForm::AddBoundaryIntegrator(BilinearFormIntegrator *bfi_real,
+                                           BilinearFormIntegrator *bfi_imag)
+{
+   if (bfi_real) { pblfr_->AddBoundaryIntegrator(bfi_real); }
+   if (bfi_imag) { pblfi_->AddBoundaryIntegrator(bfi_imag); }
+}
+
+void
+ParSesquilinearForm::AddBoundaryIntegrator(BilinearFormIntegrator *bfi_real,
+                                           BilinearFormIntegrator *bfi_imag,
+                                           Array<int> & bdr_marker)
+{
+   if (bfi_real) { pblfr_->AddBoundaryIntegrator(bfi_real, bdr_marker); }
+   if (bfi_imag) { pblfi_->AddBoundaryIntegrator(bfi_imag, bdr_marker); }
+}
+
+void
+ParSesquilinearForm::Assemble(int skip_zeros)
+{
+   pblfr_->Assemble(skip_zeros);
+   pblfi_->Assemble(skip_zeros);
+}
+
+void
+ParSesquilinearForm::Finalize(int skip_zeros)
+{
+   pblfr_->Finalize(skip_zeros);
+   pblfi_->Finalize(skip_zeros);
+}
+
+ComplexHypreParMatrix *
+ParSesquilinearForm::ParallelAssemble()
+{
+   return new ComplexHypreParMatrix(pblfr_->ParallelAssemble(),
+                                    pblfi_->ParallelAssemble(),
+                                    true, true, conv_);
+
+}
+
+void
+ParSesquilinearForm::FormLinearSystem(const Array<int> &ess_tdof_list,
+                                      Vector &x, Vector &b,
+                                      OperatorHandle &A,
+                                      Vector &X, Vector &B,
+                                      int ci)
+{
+   ParFiniteElementSpace * pfes = pblfr_->ParFESpace();
+
+   int tvs = pfes->TrueVSize();
+   cout << "TrueVSize returns " << tvs << endl;
+   cout << "GetVSize returns " << pfes->GetVSize() << endl;
+
+   int vsize = x.Size() / 2;
+   // int vsize  = pfes->GetVSize();
+   // int tvsize = pfes->GetTrueVSize();
+
+   cout << "x.Size/2 returns " << vsize << endl;
+
+   double s = (conv_ == ComplexOperator::HERMITIAN)?1.0:-1.0;
+
+   // Allocate temporary vectors
+   Vector b_0(vsize);  b_0 = 0.0;
+   // Vector B_0(tvsize); B_0 = 0.0;
+
+   // Extract the real and imaginary parts of the input vectors
+   // MFEM_ASSERT(x.Size() == 2 * vsize, "Input GridFunction of incorrect size!");
+   Vector x_r(x.GetData(), vsize);
+   Vector x_i(&(x.GetData())[vsize], vsize);
+
+   MFEM_ASSERT(b.Size() == 2 * vsize, "Input LinearForm of incorrect size!");
+   Vector b_r(b.GetData(), vsize);
+   Vector b_i(&(b.GetData())[vsize], vsize);
+   b_i *= s;
+   /*
+   X.SetSize(2 * tvsize);
+   Vector X_r(X.GetData(), tvsize);
+   Vector X_i(&(X.GetData())[tvsize], tvsize);
+
+   B.SetSize(2 * tvsize);
+   Vector B_r(B.GetData(), tvsize);
+   Vector B_i(&(B.GetData())[tvsize], tvsize);
+   */
+   OperatorHandle A_r, A_i;
+   Vector X_0, B_0;
+   cout << "pblfr fls 1" << endl << flush;
+   b_0 = b_r;
+   pblfr_->FormLinearSystem(ess_tdof_list, x_r, b_0, A_r, X_0, B_0, ci);
+
+   int tvsize = B_0.Size();
+   X.SetSize(2 * tvsize);
+   B.SetSize(2 * tvsize);
+   Vector X_r(X.GetData(), tvsize);
+   Vector X_i(&(X.GetData())[tvsize], tvsize);
+   Vector B_r(B.GetData(), tvsize);
+   Vector B_i(&(B.GetData())[tvsize], tvsize);
+   X_r = X_0; B_r = B_0;
+   cout << "pblfi fls 1" << endl << flush;
+   b_0 = 0.0;
+   pblfi_->FormLinearSystem(ess_tdof_list, x_i, b_0, A_i, X_0, B_0, false);
+   B_r -= B_0;
+   cout << "pblfr fls 2" << endl << flush;
+   b_0 = b_i;
+   pblfr_->FormLinearSystem(ess_tdof_list, x_i, b_0, A_r, X_0, B_0, ci);
+   X_i = X_0; B_i = B_0;
+   cout << "pblfi fls 2" << endl << flush;
+   b_0 = 0.0;
+   pblfi_->FormLinearSystem(ess_tdof_list, x_r, b_0, A_i, X_0, B_0, false);
+   B_i += B_0;
+
+   B_i *= s;
+   b_i *= s;
+
+   // A = A_r + i A_i
+   A.Clear();
+   if ( A_r.Type() == Operator::Hypre_ParCSR &&
+        A_i.Type() == Operator::Hypre_ParCSR )
+   {
+      ComplexHypreParMatrix * A_hyp =
+         new ComplexHypreParMatrix(A_r.As<HypreParMatrix>(),
+                                   A_i.As<HypreParMatrix>(),
+                                   A_r.OwnsOperator(),
+                                   A_i.OwnsOperator(),
+                                   conv_);
+      A.Reset<ComplexHypreParMatrix>(A_hyp, true);
+   }
+   else
+   {
+      ComplexOperator * A_op =
+         new ComplexOperator(A_r.As<Operator>(),
+                             A_i.As<Operator>(),
+                             A_r.OwnsOperator(),
+                             A_i.OwnsOperator(),
+                             conv_);
+      A.Reset<ComplexOperator>(A_op, true);
+   }
+}
+
+void
+ParSesquilinearForm::RecoverFEMSolution(const Vector &X, const Vector &b,
+                                        Vector &x)
+{
+   ParFiniteElementSpace * pfes = pblfr_->ParFESpace();
+
+   const Operator &P = *pfes->GetProlongationMatrix();
+
+   int vsize  = pfes->GetVSize();
+   int tvsize = X.Size() / 2;
+
+   Vector X_r(X.GetData(), tvsize);
+   Vector X_i(&(X.GetData())[tvsize], tvsize);
+
+   Vector x_r(x.GetData(), vsize);
+   Vector x_i(&(x.GetData())[vsize], vsize);
+
+   // Apply conforming prolongation
+   P.Mult(X_r, x_r);
+   P.Mult(X_i, x_i);
+}
+
+void
+ParSesquilinearForm::Update(FiniteElementSpace *nfes)
+{
+   if ( pblfr_ ) { pblfr_->Update(nfes); }
+   if ( pblfi_ ) { pblfi_->Update(nfes); }
+}
+
+
+#endif // MFEM_USE_MPI
+
+}

fem/complex_fem.hpp

@@ -0,0 +1,356 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_COMPLEX_FEM
+#define MFEM_COMPLEX_FEM
+
+#include "../linalg/complex_operator.hpp"
+#include "gridfunc.hpp"
+#include "linearform.hpp"
+#include "bilinearform.hpp"
+#ifdef MFEM_USE_MPI
+#include "pgridfunc.hpp"
+#include "plinearform.hpp"
+#include "pbilinearform.hpp"
+#endif
+#include <complex>
+
+namespace mfem
+{
+
+/// Class for complex-valued grid function - Vector with associated FE space.
+class ComplexGridFunction : public Vector
+{
+private:
+
+   GridFunction * gfr_;
+   GridFunction * gfi_;
+
+protected:
+   void Destroy() { delete gfr_; delete gfi_; }
+
+public:
+
+   /* @brief Construct a ComplexGridFunction associated with the
+      FiniteElementSpace @a *f. */
+   ComplexGridFunction(FiniteElementSpace *f);
+
+   void Update();
+
+   /// Assign constant values to the ComplexGridFunction data.
+   ComplexGridFunction &operator=(const std::complex<double> & value)
+   { *gfr_ = value.real(); *gfi_ = value.imag(); return *this; }
+
+   virtual void ProjectCoefficient(Coefficient &real_coeff,
+                                   Coefficient &imag_coeff);
+   virtual void ProjectCoefficient(VectorCoefficient &real_vcoeff,
+                                   VectorCoefficient &imag_vcoeff);
+
+   virtual void ProjectBdrCoefficient(Coefficient &real_coeff,
+                                      Coefficient &imag_coeff,
+                                      Array<int> &attr);
+   virtual void ProjectBdrCoefficientNormal(VectorCoefficient &real_coeff,
+                                            VectorCoefficient &imag_coeff,
+                                            Array<int> &attr);
+   virtual void ProjectBdrCoefficientTangent(VectorCoefficient &real_coeff,
+                                             VectorCoefficient &imag_coeff,
+                                             Array<int> &attr);
+
+   FiniteElementSpace *FESpace() { return gfr_->FESpace(); }
+   const FiniteElementSpace *FESpace() const { return gfr_->FESpace(); }
+
+   GridFunction & real() { return *gfr_; }
+   GridFunction & imag() { return *gfi_; }
+   const GridFunction & real() const { return *gfr_; }
+   const GridFunction & imag() const { return *gfi_; }
+
+   /// Destroys grid function.
+   virtual ~ComplexGridFunction() { Destroy(); }
+
+};
+
+class ComplexLinearForm : public Vector
+{
+private:
+   ComplexOperator::Convention conv_;
+
+protected:
+   LinearForm * lfr_;
+   LinearForm * lfi_;
+
+   // HYPRE_Int * tdof_offsets_;
+
+public:
+
+   ComplexLinearForm(FiniteElementSpace *fes,
+                     ComplexOperator::Convention
+                     convention = ComplexOperator::HERMITIAN);
+
+   virtual ~ComplexLinearForm();
+
+   /// Adds new Domain Integrator.
+   void AddDomainIntegrator(LinearFormIntegrator *lfi_real,
+                            LinearFormIntegrator *lfi_imag);
+
+   FiniteElementSpace *FESpace() const { return lfr_->FESpace(); }
+
+   LinearForm & real() { return *lfr_; }
+   LinearForm & imag() { return *lfi_; }
+   const LinearForm & real() const { return *lfr_; }
+   const LinearForm & imag() const { return *lfi_; }
+
+   void Update();
+   void Update(FiniteElementSpace *f);
+
+   /// Assembles the linear form i.e. sums over all domain/bdr integrators.
+   void Assemble();
+
+   std::complex<double> operator()(const ComplexGridFunction &gf) const;
+
+};
+
+// Class for sesquilinear form
+class SesquilinearForm
+{
+private:
+   ComplexOperator::Convention conv_;
+
+   //protected:
+   BilinearForm *blfr_;
+   BilinearForm *blfi_;
+
+public:
+   SesquilinearForm(FiniteElementSpace *fes,
+                    ComplexOperator::Convention
+                    convention = ComplexOperator::HERMITIAN);
+
+   ComplexOperator::Convention GetConvention() const { return conv_; }
+   void SetConvention(const ComplexOperator::Convention &
+                      convention) { conv_  = convention; }
+
+   BilinearForm & real() { return *blfr_; }
+   BilinearForm & imag() { return *blfi_; }
+   const BilinearForm & real() const { return *blfr_; }
+   const BilinearForm & imag() const { return *blfi_; }
+
+   /// Adds new Domain Integrator.
+   void AddDomainIntegrator(BilinearFormIntegrator *bfi_real,
+                            BilinearFormIntegrator *bfi_imag);
+
+   /// Adds new Boundary Integrator.
+   void AddBoundaryIntegrator(BilinearFormIntegrator *bfi_real,
+                              BilinearFormIntegrator *bfi_imag);
+
+   /// Adds new Boundary Integrator, restricted to specific boundary attributes.
+   void AddBoundaryIntegrator(BilinearFormIntegrator *bfi_real,
+                              BilinearFormIntegrator *bfi_imag,
+                              Array<int> &bdr_marker);
+
+   /// Assemble the local matrix
+   void Assemble(int skip_zeros = 1);
+
+   /// Finalizes the matrix initialization.
+   void Finalize(int skip_zeros = 1);
+
+   /// Returns the matrix assembled on the true dofs, i.e. P^t A P.
+   /** The returned matrix has to be deleted by the caller. */
+   ComplexSparseMatrix *AssembleCompSpMat();
+
+   /// Return the parallel FE space associated with the ParBilinearForm.
+   FiniteElementSpace *FESpace() const { return blfr_->FESpace(); }
+
+   void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x, Vector &b,
+                         OperatorHandle &A, Vector &X, Vector &B,
+                         int copy_interior = 0);
+
+   /** Call this method after solving a linear system constructed using the
+       FormLinearSystem method to recover the solution as a ParGridFunction-size
+       vector in x. Use the same arguments as in the FormLinearSystem call. */
+   virtual void RecoverFEMSolution(const Vector &X, const Vector &b, Vector &x);
+
+   virtual void Update(FiniteElementSpace *nfes = NULL);
+
+   virtual ~SesquilinearForm();
+};
+
+#ifdef MFEM_USE_MPI
+
+/// Class for complex-valued grid function - Vector with associated FE space.
+class ParComplexGridFunction : public Vector
+{
+private:
+
+   ParGridFunction * pgfr_;
+   ParGridFunction * pgfi_;
+
+protected:
+   void Destroy() { delete pgfr_; delete pgfi_; }
+
+public:
+
+   /* @brief Construct a ParComplexGridFunction associated with the
+      ParFiniteElementSpace @a *f. */
+   ParComplexGridFunction(ParFiniteElementSpace *pf);
+
+   void Update();
+
+   /// Assign constant values to the ParComplexGridFunction data.
+   ParComplexGridFunction &operator=(const std::complex<double> & value)
+   { *pgfr_ = value.real(); *pgfi_ = value.imag(); return *this; }
+
+   virtual void ProjectCoefficient(Coefficient &real_coeff,
+                                   Coefficient &imag_coeff);
+   virtual void ProjectCoefficient(VectorCoefficient &real_vcoeff,
+                                   VectorCoefficient &imag_vcoeff);
+
+   virtual void ProjectBdrCoefficient(Coefficient &real_coeff,
+                                      Coefficient &imag_coeff,
+                                      Array<int> &attr);
+   virtual void ProjectBdrCoefficientNormal(VectorCoefficient &real_coeff,
+                                            VectorCoefficient &imag_coeff,
+                                            Array<int> &attr);
+   virtual void ProjectBdrCoefficientTangent(VectorCoefficient &real_coeff,
+                                             VectorCoefficient &imag_coeff,
+                                             Array<int> &attr);
+
+   void Distribute(const Vector *tv);
+   void Distribute(const Vector &tv) { Distribute(&tv); }
+
+   /// Returns the vector restricted to the true dofs.
+   void ParallelProject(Vector &tv) const;
+
+   FiniteElementSpace *FESpace() { return pgfr_->FESpace(); }
+   const FiniteElementSpace *FESpace() const { return pgfr_->FESpace(); }
+
+   ParGridFunction & real() { return *pgfr_; }
+   ParGridFunction & imag() { return *pgfi_; }
+   const ParGridFunction & real() const { return *pgfr_; }
+   const ParGridFunction & imag() const { return *pgfi_; }
+
+   /// Destroys grid function.
+   virtual ~ParComplexGridFunction() { Destroy(); }
+
+};
+
+class ParComplexLinearForm : public Vector
+{
+private:
+   ComplexOperator::Convention conv_;
+
+protected:
+   ParLinearForm * plfr_;
+   ParLinearForm * plfi_;
+
+   HYPRE_Int * tdof_offsets_;
+
+public:
+
+   ParComplexLinearForm(ParFiniteElementSpace *pf,
+                        ComplexOperator::Convention
+                        convention = ComplexOperator::HERMITIAN);
+
+   virtual ~ParComplexLinearForm();
+
+   /// Adds new Domain Integrator.
+   void AddDomainIntegrator(LinearFormIntegrator *lfi_real,
+                            LinearFormIntegrator *lfi_imag);
+
+   ParFiniteElementSpace *ParFESpace() const { return plfr_->ParFESpace(); }
+
+   ParLinearForm & real() { return *plfr_; }
+   ParLinearForm & imag() { return *plfi_; }
+   const ParLinearForm & real() const { return *plfr_; }
+   const ParLinearForm & imag() const { return *plfi_; }
+
+   void Update(ParFiniteElementSpace *pf = NULL);
+
+   /// Assembles the linear form i.e. sums over all domain/bdr integrators.
+   void Assemble();
+
+   /// Assemble the vector on the true dofs, i.e. P^t v.
+   void ParallelAssemble(Vector &tv);
+
+   /// Returns the vector assembled on the true dofs, i.e. P^t v.
+   HypreParVector *ParallelAssemble();
+
+   std::complex<double> operator()(const ParComplexGridFunction &gf) const;
+
+};
+
+// Class for parallel sesquilinear form
+class ParSesquilinearForm
+{
+private:
+   ComplexOperator::Convention conv_;
+
+   //protected:
+   ParBilinearForm *pblfr_;
+   ParBilinearForm *pblfi_;
+
+public:
+   ParSesquilinearForm(ParFiniteElementSpace *pf,
+                       ComplexOperator::Convention
+                       convention = ComplexOperator::HERMITIAN);
+
+   ComplexOperator::Convention GetConvention() const { return conv_; }
+   void SetConvention(const ComplexOperator::Convention &
+                      convention) { conv_  = convention; }
+
+   ParBilinearForm & real() { return *pblfr_; }
+   ParBilinearForm & imag() { return *pblfi_; }
+   const ParBilinearForm & real() const { return *pblfr_; }
+   const ParBilinearForm & imag() const { return *pblfi_; }
+
+   /// Adds new Domain Integrator.
+   void AddDomainIntegrator(BilinearFormIntegrator *bfi_real,
+                            BilinearFormIntegrator *bfi_imag);
+
+   /// Adds new Boundary Integrator.
+   void AddBoundaryIntegrator(BilinearFormIntegrator *bfi_real,
+                              BilinearFormIntegrator *bfi_imag);
+
+   /// Adds new Boundary Integrator, restricted to specific boundary attributes.
+   void AddBoundaryIntegrator(BilinearFormIntegrator *bfi_real,
+                              BilinearFormIntegrator *bfi_imag,
+                              Array<int> &bdr_marker);
+
+   /// Assemble the local matrix
+   void Assemble(int skip_zeros = 1);
+
+   /// Finalizes the matrix initialization.
+   void Finalize(int skip_zeros = 1);
+
+   /// Returns the matrix assembled on the true dofs, i.e. P^t A P.
+   /** The returned matrix has to be deleted by the caller. */
+   ComplexHypreParMatrix *ParallelAssemble();
+
+   /// Return the parallel FE space associated with the ParBilinearForm.
+   ParFiniteElementSpace *ParFESpace() const { return pblfr_->ParFESpace(); }
+
+   void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x, Vector &b,
+                         OperatorHandle &A, Vector &X, Vector &B,
+                         int copy_interior = 0);
+
+   /** Call this method after solving a linear system constructed using the
+       FormLinearSystem method to recover the solution as a ParGridFunction-size
+       vector in x. Use the same arguments as in the FormLinearSystem call. */
+   virtual void RecoverFEMSolution(const Vector &X, const Vector &b, Vector &x);
+
+   virtual void Update(FiniteElementSpace *nfes = NULL);
+
+   virtual ~ParSesquilinearForm();
+};
+
+#endif // MFEM_USE_MPI
+
+}
+
+#endif // MFEM_COMPLEX_FEM

fem/fem.hpp

@@ -18,6 +18,7 @@
 #include "fe_coll.hpp"
 #include "eltrans.hpp"
 #include "coefficient.hpp"
+#include "complex_fem.hpp"
 #include "lininteg.hpp"
 #include "nonlininteg.hpp"
 #include "bilininteg.hpp"

fem/fespace.cpp

@@ -652,9 +652,9 @@ void FiniteElementSpace::BuildConformingInterpolation() const
    // create the conforming restriction matrix cR
    int *cR_J;
    {
-      int *cR_I = new int[n_true_dofs+1];
-      double *cR_A = new double[n_true_dofs];
-      cR_J = new int[n_true_dofs];
+      int *cR_I = mfem::New<int>(n_true_dofs+1);
+      double *cR_A = mfem::New<double>(n_true_dofs);
+      cR_J = mfem::New<int>(n_true_dofs);
       for (int i = 0; i < n_true_dofs; i++)
       {
          cR_I[i] = i;
@@ -975,6 +975,139 @@ void FiniteElementSpace::RefinementOperator
    }
 }
 
+FiniteElementSpace::DerefinementOperator::DerefinementOperator(
+   const FiniteElementSpace *f_fes, const FiniteElementSpace *c_fes,
+   BilinearFormIntegrator *mass_integ)
+   : Operator(c_fes->GetVSize(), f_fes->GetVSize()),
+     fine_fes(f_fes)
+{
+   MFEM_VERIFY(c_fes->GetOrdering() == f_fes->GetOrdering() &&
+               c_fes->GetVDim() == f_fes->GetVDim(),
+               "incompatible coarse and fine FE spaces");
+
+   IsoparametricTransformation emb_tr;
+   Mesh *f_mesh = f_fes->GetMesh();
+   const CoarseFineTransformations &rtrans = f_mesh->GetRefinementTransforms();
+
+   Mesh::GeometryList elem_geoms(*f_mesh);
+   DenseTensor localP[Geometry::NumGeom], localM[Geometry::NumGeom];
+   for (int gi = 0; gi < elem_geoms.Size(); gi++)
+   {
+      const Geometry::Type geom = elem_geoms[gi];
+      DenseTensor &lP = localP[geom], &lM = localM[geom];
+      const FiniteElement *fine_fe =
+         f_fes->fec->FiniteElementForGeometry(geom);
+      const FiniteElement *coarse_fe =
+         c_fes->fec->FiniteElementForGeometry(geom);
+      const DenseTensor &pmats = rtrans.GetPointMatrices(geom);
+
+      lP.SetSize(fine_fe->GetDof(), coarse_fe->GetDof(), pmats.SizeK());
+      lM.SetSize(fine_fe->GetDof(),   fine_fe->GetDof(), pmats.SizeK());
+      emb_tr.SetIdentityTransformation(geom);
+      for (int i = 0; i < pmats.SizeK(); i++)
+      {
+         emb_tr.GetPointMat() = pmats(i);
+         emb_tr.FinalizeTransformation();
+         // Get the local interpolation matrix for this refinement type
+         fine_fe->GetTransferMatrix(*coarse_fe, emb_tr, lP(i));
+         // Get the local mass matrix for this refinement type
+         mass_integ->AssembleElementMatrix(*fine_fe, emb_tr, lM(i));
+      }
+   }
+
+   Table ref_type_to_matrix;
+   rtrans.GetCoarseToFineMap(*f_mesh, coarse_to_fine, coarse_to_ref_type,
+                             ref_type_to_matrix, ref_type_to_geom);
+   MFEM_ASSERT(coarse_to_fine.Size() == c_fes->GetNE(), "");
+
+   const int total_ref_types = ref_type_to_geom.Size();
+   int num_ref_types[Geometry::NumGeom], num_fine_elems[Geometry::NumGeom];
+   Array<int> ref_type_to_coarse_elem_offset(total_ref_types);
+   ref_type_to_fine_elem_offset.SetSize(total_ref_types);
+   std::fill(num_ref_types, num_ref_types+Geometry::NumGeom, 0);
+   std::fill(num_fine_elems, num_fine_elems+Geometry::NumGeom, 0);
+   for (int i = 0; i < total_ref_types; i++)
+   {
+      Geometry::Type g = ref_type_to_geom[i];
+      ref_type_to_coarse_elem_offset[i] = num_ref_types[g];
+      ref_type_to_fine_elem_offset[i] = num_fine_elems[g];
+      num_ref_types[g]++;
+      num_fine_elems[g] += ref_type_to_matrix.RowSize(i);
+   }
+   DenseTensor localPtMP[Geometry::NumGeom];
+   for (int g = 0; g < Geometry::NumGeom; g++)
+   {
+      if (num_ref_types[g] == 0) { continue; }
+      const int fine_dofs = localP[g].SizeI();
+      const int coarse_dofs = localP[g].SizeJ();
+      localPtMP[g].SetSize(coarse_dofs, coarse_dofs, num_ref_types[g]);
+      localR[g].SetSize(coarse_dofs, fine_dofs, num_fine_elems[g]);
+   }
+   for (int i = 0; i < total_ref_types; i++)
+   {
+      Geometry::Type g = ref_type_to_geom[i];
+      DenseMatrix &lPtMP = localPtMP[g](ref_type_to_coarse_elem_offset[i]);
+      int lR_offset = ref_type_to_fine_elem_offset[i]; // offset in localR[g]
+      const int *mi = ref_type_to_matrix.GetRow(i);
+      const int nm = ref_type_to_matrix.RowSize(i);
+      lPtMP = 0.0;
+      for (int s = 0; s < nm; s++)
+      {
+         DenseMatrix &lP = localP[g](mi[s]);
+         DenseMatrix &lM = localM[g](mi[s]);
+         DenseMatrix &lR = localR[g](lR_offset+s);
+         MultAtB(lP, lM, lR); // lR = lP^T lM
+         AddMult(lR, lP, lPtMP); // lPtMP += lP^T lM lP
+      }
+      DenseMatrixInverse lPtMP_inv(lPtMP);
+      for (int s = 0; s < nm; s++)
+      {
+         DenseMatrix &lR = localR[g](lR_offset+s);
+         lPtMP_inv.Mult(lR); // lR <- (P^T M P)^{-1} P^T M
+      }
+   }
+
+   // Make a copy of the coarse element-to-dof Table.
+   coarse_elem_dof = new Table(c_fes->GetElementToDofTable());
+}
+
+FiniteElementSpace::DerefinementOperator::~DerefinementOperator()
+{
+   delete coarse_elem_dof;
+}
+
+void FiniteElementSpace::DerefinementOperator
+::Mult(const Vector &x, Vector &y) const
+{
+   Array<int> c_vdofs, f_vdofs;
+   Vector loc_x, loc_y;
+   DenseMatrix loc_x_mat, loc_y_mat;
+   const int vdim = fine_fes->GetVDim();
+   const int coarse_ndofs = height/vdim;
+   for (int coarse_el = 0; coarse_el < coarse_to_fine.Size(); coarse_el++)
+   {
+      coarse_elem_dof->GetRow(coarse_el, c_vdofs);
+      fine_fes->DofsToVDofs(c_vdofs, coarse_ndofs);
+      loc_y.SetSize(c_vdofs.Size());
+      loc_y = 0.0;
+      loc_y_mat.UseExternalData(loc_y.GetData(), c_vdofs.Size()/vdim, vdim);
+      const int ref_type = coarse_to_ref_type[coarse_el];
+      const Geometry::Type geom = ref_type_to_geom[ref_type];
+      const int *fine_elems = coarse_to_fine.GetRow(coarse_el);
+      const int num_fine_elems = coarse_to_fine.RowSize(coarse_el);
+      const int lR_offset = ref_type_to_fine_elem_offset[ref_type];
+      for (int s = 0; s < num_fine_elems; s++)
+      {
+         const DenseMatrix &lR = localR[geom](lR_offset+s);
+         fine_fes->GetElementVDofs(fine_elems[s], f_vdofs);
+         x.GetSubVector(f_vdofs, loc_x);
+         loc_x_mat.UseExternalData(loc_x.GetData(), f_vdofs.Size()/vdim, vdim);
+         AddMult(lR, loc_x_mat, loc_y_mat);
+      }
+      y.SetSubVector(c_vdofs, loc_y);
+   }
+}
+
 void FiniteElementSpace::GetLocalDerefinementMatrices(Geometry::Type geom,
                                                       DenseTensor &localR) const
 {
@@ -2042,4 +2175,401 @@ void QuadratureSpace::Save(std::ostream &out) const
 }
 
 
+GridTransfer::GridTransfer(FiniteElementSpace &dom_fes_,
+                           FiniteElementSpace &ran_fes_)
+   : dom_fes(dom_fes_), ran_fes(ran_fes_),
+     oper_type(Operator::ANY_TYPE),
+     fw_t_oper(), bw_t_oper()
+{
+#ifdef MFEM_USE_MPI
+   const bool par_dom = dynamic_cast<ParFiniteElementSpace*>(&dom_fes);
+   const bool par_ran = dynamic_cast<ParFiniteElementSpace*>(&ran_fes);
+   MFEM_VERIFY(par_dom == par_ran, "the domain and range FE spaces must both"
+               " be either serial or parallel");
+   parallel = par_dom;
+#endif
+}
+
+const Operator &GridTransfer::MakeTrueOperator(
+   FiniteElementSpace &fes_in, FiniteElementSpace &fes_out,
+   const Operator &oper, OperatorHandle &t_oper)
+{
+   if (t_oper.Ptr())
+   {
+      return *t_oper.Ptr();
+   }
+
+   if (!Parallel())
+   {
+      const SparseMatrix *in_cP = fes_in.GetConformingProlongation();
+      const SparseMatrix *out_cR = fes_out.GetConformingRestriction();
+      if (oper_type == Operator::MFEM_SPARSEMAT)
+      {
+         const SparseMatrix *mat = dynamic_cast<const SparseMatrix *>(&oper);
+         MFEM_VERIFY(mat != NULL, "Operator is not a SparseMatrix");
+         if (!out_cR)
+         {
+            t_oper.Reset(const_cast<SparseMatrix*>(mat), false);
+         }
+         else
+         {
+            t_oper.Reset(mfem::Mult(*out_cR, *mat));
+         }
+         if (in_cP)
+         {
+            t_oper.Reset(mfem::Mult(*t_oper.As<SparseMatrix>(), *in_cP));
+         }
+      }
+      else if (oper_type == Operator::ANY_TYPE)
+      {
+         const int RP_case = bool(out_cR) + 2*bool(in_cP);
+         switch (RP_case)
+         {
+            case 0:
+               t_oper.Reset(const_cast<Operator*>(&oper), false);
+               break;
+            case 1:
+               t_oper.Reset(
+                  new ProductOperator(out_cR, &oper, false, false));
+               break;
+            case 2:
+               t_oper.Reset(
+                  new ProductOperator(&oper, in_cP, false, false));
+               break;
+            case 3:
+               t_oper.Reset(
+                  new TripleProductOperator(
+                     out_cR, &oper, in_cP, false, false, false));
+               break;
+         }
+      }
+      else
+      {
+         MFEM_ABORT("Operator::Type is not supported: " << oper_type);
+      }
+   }
+   else // Parallel() == true
+   {
+#ifdef MFEM_USE_MPI
+      const SparseMatrix *out_R = fes_out.GetRestrictionMatrix();
+      if (oper_type == Operator::Hypre_ParCSR)
+      {
+         const ParFiniteElementSpace *pfes_in =
+            dynamic_cast<const ParFiniteElementSpace *>(&fes_in);
+         const ParFiniteElementSpace *pfes_out =
+            dynamic_cast<const ParFiniteElementSpace *>(&fes_out);
+         const SparseMatrix *sp_mat = dynamic_cast<const SparseMatrix *>(&oper);
+         const HypreParMatrix *hy_mat;
+         if (sp_mat)
+         {
+            SparseMatrix *RA = mfem::Mult(*out_R, *sp_mat);
+            t_oper.Reset(pfes_in->Dof_TrueDof_Matrix()->
+                         LeftDiagMult(*RA, pfes_out->GetTrueDofOffsets()));
+            delete RA;
+         }
+         else if ((hy_mat = dynamic_cast<const HypreParMatrix *>(&oper)))
+         {
+            HypreParMatrix *RA =
+               hy_mat->LeftDiagMult(*out_R, pfes_out->GetTrueDofOffsets());
+            t_oper.Reset(mfem::ParMult(RA, pfes_in->Dof_TrueDof_Matrix()));
+            delete RA;
+         }
+         else
+         {
+            MFEM_ABORT("unknown Operator type");
+         }
+      }
+      else if (oper_type == Operator::ANY_TYPE)
+      {
+         t_oper.Reset(new TripleProductOperator(
+                         out_R, &oper, fes_in.GetProlongationMatrix(),
+                         false, false, false));
+      }
+      else
+      {
+         MFEM_ABORT("Operator::Type is not supported: " << oper_type);
+      }
+#endif
+   }
+
+   return *t_oper.Ptr();
+}
+
+
+InterpolationGridTransfer::~InterpolationGridTransfer()
+{
+   if (own_mass_integ) { delete mass_integ; }
+}
+
+void InterpolationGridTransfer::SetMassIntegrator(
+   BilinearFormIntegrator *mass_integ_, bool own_mass_integ_)
+{
+   if (own_mass_integ) { delete mass_integ; }
+
+   mass_integ = mass_integ_;
+   own_mass_integ = own_mass_integ_;
+}
+
+const Operator &InterpolationGridTransfer::ForwardOperator()
+{
+   if (F.Ptr())
+   {
+      return *F.Ptr();
+   }
+
+   // Costruct F
+   if (oper_type == Operator::ANY_TYPE)
+   {
+      F.Reset(new FiniteElementSpace::RefinementOperator(&ran_fes, &dom_fes));
+   }
+   else if (oper_type == Operator::MFEM_SPARSEMAT)
+   {
+      Mesh::GeometryList elem_geoms(*ran_fes.GetMesh());
+
+      DenseTensor localP[Geometry::NumGeom];
+      for (int i = 0; i < elem_geoms.Size(); i++)
+      {
+         ran_fes.GetLocalRefinementMatrices(dom_fes, elem_geoms[i],
+                                            localP[elem_geoms[i]]);
+      }
+      F.Reset(ran_fes.RefinementMatrix_main(
+                 dom_fes.GetNDofs(), dom_fes.GetElementToDofTable(), localP));
+   }
+   else
+   {
+      MFEM_ABORT("Operator::Type is not supported: " << oper_type);
+   }
+
+   return *F.Ptr();
+}
+
+const Operator &InterpolationGridTransfer::BackwardOperator()
+{
+   if (B.Ptr())
+   {
+      return *B.Ptr();
+   }
+
+   // Construct B
+   // If not set, define a suitable mass_integ
+   if (!mass_integ && ran_fes.GetNE() > 0)
+   {
+      const FiniteElement *f_fe_0 = ran_fes.GetFE(0);
+      const int map_type = f_fe_0->GetMapType();
+      if (map_type == FiniteElement::VALUE ||
+          map_type == FiniteElement::INTEGRAL)
+      {
+         mass_integ = new MassIntegrator;
+      }
+      else if (map_type == FiniteElement::H_DIV ||
+               map_type == FiniteElement::H_CURL)
+      {
+         mass_integ = new VectorFEMassIntegrator;
+      }
+      else
+      {
+         MFEM_ABORT("unknown type of FE space");
+      }
+      own_mass_integ = true;
+   }
+   if (oper_type == Operator::ANY_TYPE)
+   {
+      B.Reset(new FiniteElementSpace::DerefinementOperator(
+                 &ran_fes, &dom_fes, mass_integ));
+   }
+   else
+   {
+      MFEM_ABORT("Operator::Type is not supported: " << oper_type);
+   }
+
+   return *B.Ptr();
+}
+
+
+L2ProjectionGridTransfer::L2Projection::L2Projection(
+   const FiniteElementSpace &fes_ho_, const FiniteElementSpace &fes_lor_)
+   : fes_ho(fes_ho_), fes_lor(fes_lor_)
+{
+   Mesh *mesh_ho = fes_ho.GetMesh();
+   MFEM_VERIFY(mesh_ho->GetNumGeometries(mesh_ho->Dimension()) <= 1,
+               "mixed meshes are not supported");
+
+   // If the local mesh is empty, skip all computations
+   if (mesh_ho->GetNE() == 0) { return; }
+
+   const FiniteElement *fe_lor = fes_lor.GetFE(0);
+   const FiniteElement *fe_ho = fes_ho.GetFE(0);
+   ndof_lor = fe_lor->GetDof();
+   ndof_ho = fe_ho->GetDof();
+
+   const int nel_lor = fes_lor.GetNE();
+   const int nel_ho = fes_ho.GetNE();
+
+   nref = nel_lor/nel_ho;
+
+   // Construct the mapping from HO to LOR
+   // ho2lor.GetRow(iho) will give all the LOR elements contained in iho
+   ho2lor.SetSize(nel_ho, nref);
+   const CoarseFineTransformations &cf_tr =
+      fes_lor.GetMesh()->GetRefinementTransforms();
+   for (int ilor=0; ilor<nel_lor; ++ilor)
+   {
+      int iho = cf_tr.embeddings[ilor].parent;
+      ho2lor.AddConnection(iho, ilor);
+   }
+   ho2lor.ShiftUpI();
+
+   // R will contain the restriction (L^2 projection operator) defined on
+   // each coarse HO element (and corresponding patch of LOR elements)
+   R.SetSize(ndof_lor*nref, ndof_ho, nel_ho);
+   // P will contain the corresponding prolongation operator
+   P.SetSize(ndof_ho, ndof_lor*nref, nel_ho);
+
+   DenseMatrix Minv_lor(ndof_lor*nref, ndof_lor*nref);
+   DenseMatrix M_mixed(ndof_lor*nref, ndof_ho);
+
+   MassIntegrator mi;
+   DenseMatrix M_lor_el(ndof_lor, ndof_lor);
+   DenseMatrixInverse Minv_lor_el(&M_lor_el);
+   DenseMatrix M_lor(ndof_lor*nref, ndof_lor*nref);
+   DenseMatrix M_mixed_el(ndof_lor, ndof_ho);
+
+   Minv_lor = 0.0;
+   M_lor = 0.0;
+
+   DenseMatrix RtMlor(ndof_ho, ndof_lor*nref);
+   DenseMatrix RtMlorR(ndof_ho, ndof_ho);
+   DenseMatrixInverse RtMlorR_inv(&RtMlorR);
+
+   IntegrationPointTransformation ip_tr;
+   IsoparametricTransformation &emb_tr = ip_tr.Transf;
+
+   Vector shape_ho(ndof_ho);
+   Vector shape_lor(ndof_lor);
+
+   const Geometry::Type geom = fe_ho->GetGeomType();
+   const DenseTensor &pmats = cf_tr.GetPointMatrices(geom);
+   emb_tr.SetIdentityTransformation(geom);
+
+   for (int iho=0; iho<nel_ho; ++iho)
+   {
+      for (int iref=0; iref<nref; ++iref)
+      {
+         // Assemble the low-order refined mass matrix and invert locally
+         int ilor = ho2lor.GetRow(iho)[iref];
+         ElementTransformation *el_tr = fes_lor.GetElementTransformation(ilor);
+         mi.AssembleElementMatrix(*fe_lor, *el_tr, M_lor_el);
+         M_lor.CopyMN(M_lor_el, iref*ndof_lor, iref*ndof_lor);
+         Minv_lor_el.Factor();
+         Minv_lor_el.GetInverseMatrix(M_lor_el);
+         // Insert into the diagonal of the patch LOR mass matrix
+         Minv_lor.CopyMN(M_lor_el, iref*ndof_lor, iref*ndof_lor);
+
+         // Now assemble the block-row of the mixed mass matrix associated
+         // with integrating HO functions against LOR functions on the LOR
+         // sub-element.
+
+         // Create the transformation that embeds the fine low-order element
+         // within the coarse high-order element in reference space
+         emb_tr.GetPointMat() = pmats(iref);
+         emb_tr.FinalizeTransformation();
+
+         int order = fe_lor->GetOrder() + fe_ho->GetOrder() + el_tr->OrderW();
+         const IntegrationRule *ir = &IntRules.Get(geom, order);
+         M_mixed_el = 0.0;
+         for (int i = 0; i < ir->GetNPoints(); i++)
+         {
+            const IntegrationPoint &ip_lor = ir->IntPoint(i);
+            IntegrationPoint ip_ho;
+            ip_tr.Transform(ip_lor, ip_ho);
+            fe_lor->CalcShape(ip_lor, shape_lor);
+            fe_ho->CalcShape(ip_ho, shape_ho);
+            el_tr->SetIntPoint(&ip_lor);
+            // For now we use the geometry information from the LOR space
+            // which means we won't be mass conservative if the mesh is curved
+            double w = el_tr->Weight()*ip_lor.weight;
+            shape_lor *= w;
+            AddMultVWt(shape_lor, shape_ho, M_mixed_el);
+         }
+         M_mixed.CopyMN(M_mixed_el, iref*ndof_lor, 0);
+      }
+      mfem::Mult(Minv_lor, M_mixed, R(iho));
+
+      mfem::MultAtB(R(iho), M_lor, RtMlor);
+      mfem::Mult(RtMlor, R(iho), RtMlorR);
+      RtMlorR_inv.Factor();
+      RtMlorR_inv.Mult(RtMlor, P(iho));
+   }
+}
+
+void L2ProjectionGridTransfer::L2Projection::Mult(
+   const Vector &x, Vector &y) const
+{
+   int vdim = fes_ho.GetVDim();
+   Array<int> vdofs;
+   DenseMatrix xel_mat(ndof_ho, vdim);
+   DenseMatrix yel_mat(ndof_lor*nref, vdim);
+   for (int iho=0; iho<fes_ho.GetNE(); ++iho)
+   {
+      fes_ho.GetElementVDofs(iho, vdofs);
+      x.GetSubVector(vdofs, xel_mat.GetData());
+      mfem::Mult(R(iho), xel_mat, yel_mat);
+      // Place result correctly into low-order vector
+      for (int iref=0; iref<nref; ++iref)
+      {
+         int ilor = ho2lor.GetRow(iho)[iref];
+         for (int vd=0; vd<vdim; ++vd)
+         {
+            fes_lor.GetElementDofs(ilor, vdofs);
+            fes_lor.DofsToVDofs(vd, vdofs);
+            y.SetSubVector(vdofs, &yel_mat(iref*ndof_lor,vd));
+         }
+      }
+   }
+}
+
+void L2ProjectionGridTransfer::L2Projection::Prolongate(
+   const Vector &x, Vector &y) const
+{
+   int vdim = fes_ho.GetVDim();
+   Array<int> vdofs;
+   DenseMatrix xel_mat(ndof_lor*nref, vdim);
+   DenseMatrix yel_mat(ndof_ho, vdim);
+   for (int iho=0; iho<fes_ho.GetNE(); ++iho)
+   {
+      // Extract the LOR DOFs
+      for (int iref=0; iref<nref; ++iref)
+      {
+         int ilor = ho2lor.GetRow(iho)[iref];
+         for (int vd=0; vd<vdim; ++vd)
+         {
+            fes_lor.GetElementDofs(ilor, vdofs);
+            fes_lor.DofsToVDofs(vd, vdofs);
+            x.GetSubVector(vdofs, &xel_mat(iref*ndof_lor, vd));
+         }
+      }
+      // Locally prolongate
+      mfem::Mult(P(iho), xel_mat, yel_mat);
+      // Place the result in the HO vector
+      fes_ho.GetElementVDofs(iho, vdofs);
+      y.SetSubVector(vdofs, yel_mat.GetData());
+   }
+}
+
+const Operator &L2ProjectionGridTransfer::ForwardOperator()
+{
+   if (!F) { F = new L2Projection(dom_fes, ran_fes); }
+   return *F;
+}
+
+const Operator &L2ProjectionGridTransfer::BackwardOperator()
+{
+   if (!B)
+   {
+      if (!F) { F = new L2Projection(dom_fes, ran_fes); }
+      B = new L2Prolongation(*F);
+   }
+   return *B;
+}
+
 } // namespace mfem

fem/fespace.hpp

@@ -59,12 +59,17 @@ Ordering::Map<Ordering::byVDIM>(int ndofs, int vdim, int dof, int vd)
 }
 
 
+// Forward declarations
 class NURBSExtension;
+class BilinearFormIntegrator;
+
 
 /** @brief Class FiniteElementSpace - responsible for providing FEM view of the
     mesh, mainly managing the set of degrees of freedom. */
 class FiniteElementSpace
 {
+   friend class InterpolationGridTransfer;
+
 protected:
    /// The mesh that FE space lives on (not owned).
    Mesh *mesh;
@@ -150,6 +155,25 @@ protected:
       virtual ~RefinementOperator();
    };
 
+   // Derefinement operator, used by the friend class InterpolationGridTransfer.
+   class DerefinementOperator : public Operator
+   {
+      const FiniteElementSpace *fine_fes; // Not owned.
+      DenseTensor localR[Geometry::NumGeom];
+      Table *coarse_elem_dof; // Owned.
+      Table coarse_to_fine;
+      Array<int> coarse_to_ref_type;
+      Array<Geometry::Type> ref_type_to_geom;
+      Array<int> ref_type_to_fine_elem_offset;
+
+   public:
+      DerefinementOperator(const FiniteElementSpace *f_fes,
+                           const FiniteElementSpace *c_fes,
+                           BilinearFormIntegrator *mass_integ);
+      virtual void Mult(const Vector &x, Vector &y) const;
+      virtual ~DerefinementOperator();
+   };
+
    // This method makes the same assumptions as the method:
    //    void GetLocalRefinementMatrices(
    //       const FiniteElementSpace &coarse_fes, Geometry::Type geom,
@@ -573,6 +597,215 @@ public:
    void Save(std::ostream &out) const;
 };
 
+
+/** @brief Base class for transfer algorithms that construct transfer Operator%s
+    between two finite element (FE) spaces. */
+/** Generally, the two FE spaces (domain and range) can be defined on different
+    meshes. */
+class GridTransfer
+{
+protected:
+   FiniteElementSpace &dom_fes; ///< Domain FE space
+   FiniteElementSpace &ran_fes; ///< Range FE space
+
+   /** @brief Desired Operator::Type for the construction of all operators
+       defined by the underlying transfer algorithm. It can be ignored by
+       derived classes. */
+   Operator::Type oper_type;
+
+   OperatorHandle fw_t_oper; ///< Forward true-dof operator
+   OperatorHandle bw_t_oper; ///< Backward true-dof operator
+
+#ifdef MFEM_USE_MPI
+   bool parallel;
+#endif
+   bool Parallel() const
+   {
+#ifndef MFEM_USE_MPI
+      return false;
+#else
+      return parallel;
+#endif
+   }
+
+   const Operator &MakeTrueOperator(FiniteElementSpace &fes_in,
+                                    FiniteElementSpace &fes_out,
+                                    const Operator &oper,
+                                    OperatorHandle &t_oper);
+
+public:
+   /** Construct a transfer algorithm between the domain, @a dom_fes_, and
+       range, @a ran_fes_, FE spaces. */
+   GridTransfer(FiniteElementSpace &dom_fes_, FiniteElementSpace &ran_fes_);
+
+   /// Virtual destructor
+   virtual ~GridTransfer() { }
+
+   /** @brief Set the desired Operator::Type for the construction of all
+       operators defined by the underlying transfer algorithm. */
+   /** The default value is Operator::ANY_TYPE which typically corresponds to
+       a matrix-free operator representation. Note that derived classes are not
+       required to support this setting and can ignore it. */
+   void SetOperatorType(Operator::Type type) { oper_type = type; }
+
+   /** @brief Return an Operator that transfers GridFunction%s from the domain
+       FE space to GridFunction%s in the range FE space. */
+   virtual const Operator &ForwardOperator() = 0;
+
+   /** @brief Return an Operator that transfers GridFunction%s from the range
+       FE space back to GridFunction%s in the domain FE space. */
+   virtual const Operator &BackwardOperator() = 0;
+
+   /** @brief Return an Operator that transfers true-dof Vector%s from the
+       domain FE space to true-dof Vector%s in the range FE space. */
+   /** This method is implemented in the base class, based on ForwardOperator(),
+       however, derived classes can overload the construction, if necessary. */
+   virtual const Operator &TrueForwardOperator()
+   {
+      return MakeTrueOperator(dom_fes, ran_fes, ForwardOperator(), fw_t_oper);
+   }
+
+   /** @brief Return an Operator that transfers true-dof Vector%s from the
+       range FE space back to true-dof Vector%s in the domain FE space. */
+   /** This method is implemented in the base class, based on
+       BackwardOperator(), however, derived classes can overload the
+       construction, if necessary. */
+   virtual const Operator &TrueBackwardOperator()
+   {
+      return MakeTrueOperator(ran_fes, dom_fes, BackwardOperator(), bw_t_oper);
+   }
+};
+
+
+/** @brief Transfer data between a coarse mesh and an embedded refined mesh
+    using interpolation. */
+/** The forward, coarse-to-fine, transfer uses nodal interpolation. The
+    backward, fine-to-coarse, transfer is defined locally (on a coarse element)
+    as B = (F^t M_f F)^{-1} F^t M_f, where F is the forward transfer matrix, and
+    M_f is a mass matrix on the union of all fine elements comprising the coarse
+    element. Note that the backward transfer operator, B, is a left inverse of
+    the forward transfer operator, F, i.e. B F = I. Both F and B are defined in
+    reference space and do not depend on the actual physical shape of the mesh
+    elements.
+
+    It is assumed that both the coarse and the fine FiniteElementSpace%s use
+    compatible types of elements, e.g. finite elements with the same map-type
+    (VALUE, INTEGRAL, H_DIV, H_CURL - see class FiniteElement). Generally, the
+    FE spaces can have different orders, however, in order for the backward
+    operator to be well-defined, the (local) number of the fine dofs should not
+    be smaller than the number of coarse dofs. */
+class InterpolationGridTransfer : public GridTransfer
+{
+protected:
+   BilinearFormIntegrator *mass_integ; ///< Ownership depends on #own_mass_integ
+   bool own_mass_integ; ///< Ownership flag for #mass_integ
+
+   OperatorHandle F; ///< Forward, coarse-to-fine, operator
+   OperatorHandle B; ///< Backward, fine-to-coarse, operator
+
+public:
+   InterpolationGridTransfer(FiniteElementSpace &coarse_fes,
+                             FiniteElementSpace &fine_fes)
+      : GridTransfer(coarse_fes, fine_fes),
+        mass_integ(NULL), own_mass_integ(false)
+   { }
+
+   virtual ~InterpolationGridTransfer();
+
+   /** @brief Assign a mass integrator to be used in the construction of the
+       backward, fine-to-coarse, transfer operator. */
+   void SetMassIntegrator(BilinearFormIntegrator *mass_integ_,
+                          bool own_mass_integ_ = true);
+
+   virtual const Operator &ForwardOperator();
+
+   virtual const Operator &BackwardOperator();
+};
+
+
+/** @brief Transfer data between a coarse mesh and an embedded refined mesh
+    using L2 projection. */
+/** The forward, coarse-to-fine, transfer uses L2 projection. The backward,
+    fine-to-coarse, transfer is defined locally (on a coarse element) as
+    B = (F^t M_f F)^{-1} F^t M_f, where F is the forward transfer matrix, and
+    M_f is the mass matrix on the union of all fine elements comprising the
+    coarse element. Note that the backward transfer operator, B, is a left
+    inverse of the forward transfer operator, F, i.e. B F = I. Both F and B are
+    defined in physical space and, generally, vary between different mesh
+    elements.
+
+    This class currently only fully supports L2 finite element spaces and fine
+    meshes that are a uniform refinement of the coarse mesh. Generally, the
+    coarse and fine FE spaces can have different orders, however, in order for
+    the backward operator to be well-defined, the number of the fine dofs (in a
+    coarse element) should not be smaller than the number of coarse dofs.
+
+    If used on H1 finite element spaces, the transfer will be performed locally,
+    and the value of shared (interface) degrees of freedom will be determined by
+    the value of the last transfer to be performed (according to the element
+    numbering in the finite element space). As a consequence, the mass
+    conservation properties for this operator from the L2 case do not carry over
+    to H1 spaces. */
+class L2ProjectionGridTransfer : public GridTransfer
+{
+protected:
+   /** Class representing projection operator between a high-order L2 finite
+       element space on a coarse mesh, and a low-order L2 finite element space
+       on a refined mesh (LOR). We assume that the low-order space, fes_lor,
+       lives on a mesh obtained by refining the mesh of the high-order space,
+       fes_ho. */
+   class L2Projection : public Operator
+   {
+      const FiniteElementSpace &fes_ho;
+      const FiniteElementSpace &fes_lor;
+
+      int ndof_lor, ndof_ho, nref;
+
+      Table ho2lor;
+
+      DenseTensor R, P;
+
+   public:
+      L2Projection(const FiniteElementSpace &fes_ho_,
+                   const FiniteElementSpace &fes_lor_);
+      /// Perform the L2 projection onto the LOR space
+      virtual void Mult(const Vector &x, Vector &y) const;
+      /// Perform the mass conservative left-inverse prolongation operation.
+      /// This functionality is also provided as an Operator by L2Prolongation.
+      void Prolongate(const Vector &x, Vector &y) const;
+      virtual ~L2Projection() { }
+   };
+
+   /** Mass-conservative prolongation operator going in the opposite direction
+       as L2Projection. This operator is a left inverse to the L2Projection. */
+   class L2Prolongation : public Operator
+   {
+      const L2Projection &l2proj;
+
+   public:
+      L2Prolongation(const L2Projection &l2proj_) : l2proj(l2proj_) { }
+      void Mult(const Vector &x, Vector &y) const
+      {
+         l2proj.Prolongate(x, y);
+      }
+      virtual ~L2Prolongation() { }
+   };
+
+   L2Projection   *F; ///< Forward, coarse-to-fine, operator
+   L2Prolongation *B; ///< Backward, fine-to-coarse, operator
+
+public:
+   L2ProjectionGridTransfer(FiniteElementSpace &coarse_fes,
+                            FiniteElementSpace &fine_fes)
+      : GridTransfer(coarse_fes, fine_fes),
+        F(NULL), B(NULL)
+   { }
+
+   virtual const Operator &ForwardOperator();
+
+   virtual const Operator &BackwardOperator();
+};
+
 }
 
 #endif

fem/gridfunc.hpp

@@ -104,7 +104,9 @@ public:
    GridFunction &operator=(const GridFunction &rhs)
    { return operator=((const Vector &)rhs); }
 
-   /// Make the GridFunction the owner of 'fec' and 'fes'
+   /// Make the GridFunction the owner of #fec and #fes.
+   /** If the new FiniteElementCollection, @a _fec, is NULL, ownership of #fec
+       and #fes is taken away. */
    void MakeOwner(FiniteElementCollection *_fec) { fec = _fec; }
 
    FiniteElementCollection *OwnFEC() { return fec; }

fem/intrules.cpp

@@ -886,7 +886,7 @@ const IntegrationRule &IntegrationRules::Get(int GeomType, int Order)
 
    if (!HaveIntRule(*ir_array, Order))
    {
-#ifdef MFEM_USE_OPENMP
+#ifdef MFEM_USE_LEGACY_OPENMP
       #pragma omp critical
 #endif
       {
@@ -1595,20 +1595,20 @@ IntegrationRule *IntegrationRules::TetrahedronIntegrationRule(int Order)
 // Integration rules for reference prism
 IntegrationRule *IntegrationRules::PrismIntegrationRule(int Order)
 {
-   IntegrationRule * irt = GenerateIntegrationRule(Geometry::TRIANGLE, Order);
-   IntegrationRule * irs = GenerateIntegrationRule(Geometry::SEGMENT, Order);
-   int nt = irt->GetNPoints();
-   int ns = irs->GetNPoints();
+   const IntegrationRule & irt = Get(Geometry::TRIANGLE, Order);
+   const IntegrationRule & irs = Get(Geometry::SEGMENT, Order);
+   int nt = irt.GetNPoints();
+   int ns = irs.GetNPoints();
    AllocIntRule(PrismIntRules, Order);
    PrismIntRules[Order] = new IntegrationRule(nt * ns);
 
    for (int ks=0; ks<ns; ks++)
    {
-      const IntegrationPoint & ips = irs->IntPoint(ks);
+      const IntegrationPoint & ips = irs.IntPoint(ks);
       for (int kt=0; kt<nt; kt++)
       {
          int kp = ks * nt + kt;
-         const IntegrationPoint & ipt = irt->IntPoint(kt);
+         const IntegrationPoint & ipt = irt.IntPoint(kt);
          IntegrationPoint & ipp = PrismIntRules[Order]->IntPoint(kp);
          ipp.x = ipt.x;
          ipp.y = ipt.y;

fem/intrules.hpp

@@ -325,6 +325,9 @@ private:
       return Order | 1; // valid for all quad_type's
    }
 
+   /// The following methods allocate new IntegrationRule objects without
+   /// checking if they already exist.  To avoid memory leaks use
+   /// IntegrationRules::Get(int GeomType, int Order) instead.
    IntegrationRule *GenerateIntegrationRule(int GeomType, int Order);
    IntegrationRule *PointIntegrationRule(int Order);
    IntegrationRule *SegmentIntegrationRule(int Order);

fem/linearform.hpp

@@ -81,6 +81,15 @@ public:
        Update(FiniteElementSpace *, Vector &, int). */
    LinearForm() { fes = NULL; extern_lfs = 0; }
 
+   /// Construct a LinearForm using previously allocated array @a data.
+   /** The LinearForm does not assume ownership of @a data which is assumed to
+       be of size at least `f->GetVSize()`. Similar to the Vector constructor
+       for externally allocated array, the pointer @a data can be NULL. The data
+       array can be replaced later using the method SetData().
+    */
+   LinearForm(FiniteElementSpace *f, double *data) : Vector(data, f->GetVSize())
+   { fes = f; }
+
    /// Copy assignment. Only the data of the base class Vector is copied.
    /** It is assumed that this object and @a rhs use FiniteElementSpace%s that
        have the same size.

fem/occa.okl

@@ -0,0 +1,999 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#if Q1D < D1D
+#  define M1D D1D
+#else
+#  define M1D Q1D
+#endif
+#define M2D (M1D*M1D)
+
+#define DQ1D (D1D*Q1D)
+
+#define Q2D (Q1D*Q1D)
+#define Q3D (Q1D*Q1D*Q1D)
+
+#define M2_ELEMENT_BATCH 32
+
+#define QUAD_2D_ID(X, Y) (X + ((Y) * Q1D))
+#define QUAD_3D_ID(X, Y, Z) (X + ((Y) * Q1D) + ((Z) * Q2D))
+
+typedef double* DofToQuad_t @dim(Q1D, D1D);
+typedef double* QuadToDof_t @dim(D1D, Q1D);
+
+typedef double* DLocal2D_t @dim(D1D, D1D, NE);
+typedef double* QLocal2D_t @dim(Q1D, Q1D, NE);
+
+typedef double* DLocal3D_t @dim(D1D, D1D, D1D, NE);
+typedef double* QLocal3D_t @dim(Q1D, Q1D, Q1D, NE);
+
+typedef double* Jacobian2D_t @dim(2, 2, Q2D, NE);
+typedef double* Jacobian3D_t @dim(3, 3, Q3D, NE);
+
+typedef double* SymmOperator2D_t @dim(3, Q2D, NE);
+typedef double* SymmOperator3D_t @dim(6, Q3D, NE);
+
+@kernel void DiffusionSetup2D(const int NE,
+                              @restrict const double *W,
+                              @restrict const Jacobian2D_t J,
+                              const double COEFF,
+                              @restrict SymmOperator2D_t op) {
+  for (int e = 0; e < NE; ++e; @outer) {
+    for (int q = 0; q < Q2D; ++q; @inner) {
+      const double J11 = J(0, 0, q, e), J12 = J(1, 0, q, e);
+      const double J21 = J(0, 1, q, e), J22 = J(1, 1, q, e);
+      const double c_detJ = W[q] * COEFF / ((J11 * J22) - (J21 * J12));
+      op(0, q, e) =  c_detJ * (J21*J21 + J22*J22); // (1,1)
+      op(1, q, e) = -c_detJ * (J21*J11 + J22*J12); // (1,2), (2,1)
+      op(2, q, e) =  c_detJ * (J11*J11 + J12*J12); // (2,2)
+    }
+  }
+}
+
+@kernel void DiffusionSetup3D(const int NE,
+                              @restrict const double *W,
+                              @restrict const Jacobian3D_t J,
+                              const double COEFF,
+                              @restrict SymmOperator3D_t op) {
+  for (int e = 0; e < NE; ++e; @outer) {
+    for (int q = 0; q < Q3D; ++q; @inner) {
+      const double J11 = J(0, 0, q, e), J12 = J(1, 0, q, e), J13 = J(2, 0, q, e);
+      const double J21 = J(0, 1, q, e), J22 = J(1, 1, q, e), J23 = J(2, 1, q, e);
+      const double J31 = J(0, 2, q, e), J32 = J(1, 2, q, e), J33 = J(2, 2, q, e);
+
+      const double detJ = ((J11 * J22 * J33) + (J12 * J23 * J31) + (J13 * J21 * J32) -
+                           (J13 * J22 * J31) - (J12 * J21 * J33) - (J11 * J23 * J32));
+
+      const double c_detJ = W[q] * COEFF / detJ;
+
+      // adj(J)
+      const double A11 = (J22 * J33) - (J23 * J32);
+      const double A12 = (J23 * J31) - (J21 * J33);
+      const double A13 = (J21 * J32) - (J22 * J31);
+
+      const double A21 = (J13 * J32) - (J12 * J33);
+      const double A22 = (J11 * J33) - (J13 * J31);
+      const double A23 = (J12 * J31) - (J11 * J32);
+
+      const double A31 = (J12 * J23) - (J13 * J22);
+      const double A32 = (J13 * J21) - (J11 * J23);
+      const double A33 = (J11 * J22) - (J12 * J21);
+
+      // adj(J)^Tadj(J)
+      op(0, q, e) = c_detJ * (A11*A11 + A21*A21 + A31*A31); // (1,1)
+      op(1, q, e) = c_detJ * (A11*A12 + A21*A22 + A31*A32); // (1,2), (2,1)
+      op(2, q, e) = c_detJ * (A11*A13 + A21*A23 + A31*A33); // (1,3), (3,1)
+      op(3, q, e) = c_detJ * (A12*A12 + A22*A22 + A32*A32); // (2,2)
+      op(4, q, e) = c_detJ * (A12*A13 + A22*A23 + A32*A33); // (2,3), (3,2)
+      op(5, q, e) = c_detJ * (A13*A13 + A23*A23 + A33*A33); // (3,3)
+    }
+  }
+}
+
+@kernel void DiffusionApply2D_CPU(const int NE,
+                                  @restrict const DofToQuad_t B,
+                                  @restrict const DofToQuad_t G,
+                                  @restrict const QuadToDof_t Bt,
+                                  @restrict const QuadToDof_t Gt,
+                                  @restrict const SymmOperator2D_t op,
+                                  @restrict const DLocal2D_t X,
+                                  @restrict DLocal2D_t Y) {
+   // Iterate over elements
+   for (int e = 0; e < NE; ++e; @outer) {
+      for (int dummy = 0; dummy < 1; ++dummy; @inner) {
+         double grad[Q1D][Q1D][2];
+         for (int qy = 0; qy < Q1D; ++qy) {
+            for (int qx = 0; qx < Q1D; ++qx) {
+               grad[qy][qx][0] = 0;
+               grad[qy][qx][1] = 0;
+            }
+         }
+
+         for (int dy = 0; dy < D1D; ++dy) {
+            double gradX[Q1D][2];
+            for (int qx = 0; qx < Q1D; ++qx) {
+               gradX[qx][0] = 0;
+               gradX[qx][1] = 0;
+            }
+
+            for (int dx = 0; dx < D1D; ++dx) {
+               const double s = X(dx, dy, e);
+               for (int qx = 0; qx < Q1D; ++qx) {
+                  gradX[qx][0] += s * B(qx, dx);
+                  gradX[qx][1] += s * G(qx, dx);
+               }
+            }
+
+            for (int qy = 0; qy < Q1D; ++qy) {
+               const double wy  = B(qy, dy);
+               const double wDy = G(qy, dy);
+               for (int qx = 0; qx < Q1D; ++qx) {
+                  grad[qy][qx][0] += gradX[qx][1] * wy;
+                  grad[qy][qx][1] += gradX[qx][0] * wDy;
+               }
+            }
+         }
+
+         // Calculate Dxy, xDy in plane
+         for (int qy = 0; qy < Q1D; ++qy) {
+            for (int qx = 0; qx < Q1D; ++qx) {
+               const int q = QUAD_2D_ID(qx, qy);
+               const double O11 = op(0, q, e);
+               const double O12 = op(1, q, e);
+               const double O22 = op(2, q, e);
+
+               const double gradX = grad[qy][qx][0];
+               const double gradY = grad[qy][qx][1];
+
+               grad[qy][qx][0] = (O11 * gradX) + (O12 * gradY);
+               grad[qy][qx][1] = (O12 * gradX) + (O22 * gradY);
+            }
+         }
+
+         for (int qy = 0; qy < Q1D; ++qy) {
+            double gradX[D1D][2];
+            for (int dx = 0; dx < D1D; ++dx) {
+               gradX[dx][0] = 0;
+               gradX[dx][1] = 0;
+            }
+
+            for (int qx = 0; qx < Q1D; ++qx) {
+               const double gX = grad[qy][qx][0];
+               const double gY = grad[qy][qx][1];
+               for (int dx = 0; dx < D1D; ++dx) {
+                  const double wx  = Bt(dx, qx);
+                  const double wDx = Gt(dx, qx);
+                  gradX[dx][0] += gX * wDx;
+                  gradX[dx][1] += gY * wx;
+               }
+            }
+
+            for (int dy = 0; dy < D1D; ++dy) {
+               const double wy  = Bt(dy, qy);
+               const double wDy = Gt(dy, qy);
+               for (int dx = 0; dx < D1D; ++dx) {
+                  Y(dx, dy, e) += ((gradX[dx][0] * wy) +
+                                   (gradX[dx][1] * wDy));
+               }
+            }
+         }
+      }
+   }
+}
+
+@kernel void DiffusionApply2D_GPU(const int NE,
+                                  @restrict const DofToQuad_t B,
+                                  @restrict const DofToQuad_t G,
+                                  @restrict const QuadToDof_t Bt,
+                                  @restrict const QuadToDof_t Gt,
+                                  @restrict const SymmOperator2D_t op,
+                                  @restrict const DLocal2D_t X,
+                                  @restrict DLocal2D_t Y) {
+   // Iterate over elements
+   for (int eOff = 0; eOff < NE; eOff += M2_ELEMENT_BATCH; @outer) {
+      // Store dof <--> quad mappings
+      @shared double s_B[DQ1D] @dim(Q1D, D1D);
+      @shared double s_G[DQ1D] @dim(Q1D, D1D);
+      @shared double s_Bt[DQ1D] @dim(D1D, Q1D);
+      @shared double s_Gt[DQ1D] @dim(D1D, Q1D);
+
+      // Store xy planes in @shared memory
+      @shared double s_xy[DQ1D] @dim(D1D, Q1D);
+      @shared double s_xDy[DQ1D] @dim(D1D, Q1D);
+      @shared double s_grad[2 * Q2D] @dim(2, Q1D, Q1D);
+
+      @exclusive double r_x[M1D];
+      @exclusive double r_y[Q1D];
+
+      for (int x = 0; x < M1D; ++x; @inner) {
+         for (int id = x; id < DQ1D; id += M1D) {
+            s_B[id]  = B[id];
+            s_G[id] = G[id];
+            s_Bt[id]  = Bt[id];
+            s_Gt[id] = Gt[id];
+         }
+      }
+
+      for (int e = eOff; e < (eOff + M2_ELEMENT_BATCH); ++e) {
+         if (e < NE) {
+            for (int dx = 0; dx < D1D; ++dx; @inner) {
+               if (dx < D1D) {
+                  for (int qy = 0; qy < Q1D; ++qy) {
+                     s_xy(dx, qy) = 0;
+                     s_xDy(dx, qy) = 0;
+                  }
+                  for (int dy = 0; dy < D1D; ++dy) {
+                     r_x[dy] = X(dx, dy, e);
+                  }
+                  for (int qy = 0; qy < Q1D; ++qy) {
+                     double xy = 0;
+                     double xDy = 0;
+                     for (int dy = 0; dy < D1D; ++dy) {
+                        xy  += r_x[dy] * s_B(qy, dy);
+                        xDy += r_x[dy] * s_G(qy, dy);
+                     }
+                     s_xy(dx, qy)  = xy;
+                     s_xDy(dx, qy) = xDy;
+                  }
+               }
+            }
+            for (int qy = 0; qy < M1D; ++qy; @inner) {
+               if (qy < Q1D) {
+                  for (int qx = 0; qx < Q1D; ++qx) {
+                     double gradX = 0, gradY = 0;
+                     for (int dx = 0; dx < D1D; ++dx) {
+                        gradX += s_xy(dx, qy)  * s_G(qx, dx);
+                        gradY += s_xDy(dx, qy) * s_B(qx, dx);
+                     }
+
+                     const int q = QUAD_2D_ID(qx, qy);
+                     const double O11 = op(0, q, e);
+                     const double O12 = op(1, q, e);
+                     const double O22 = op(2, q, e);
+
+                     s_grad(0, qx, qy) = (O11 * gradX) + (O12 * gradY);
+                     s_grad(1, qx, qy) = (O12 * gradX) + (O22 * gradY);
+                  }
+               }
+            }
+
+            for (int qx = 0; qx < Q1D; ++qx; @inner) {
+               if (qx < Q1D) {
+                  for (int dy = 0; dy < D1D; ++dy) {
+                     s_xy(dy, qx) = 0;
+                     s_xDy(dy, qx) = 0;
+                  }
+                  for (int qy = 0; qy < Q1D; ++qy) {
+                     r_x[qy] = s_grad(0, qx, qy);
+                     r_y[qy] = s_grad(1, qx, qy);
+                  }
+                  for (int dy = 0; dy < D1D; ++dy) {
+                     double xy  = 0;
+                     double xDy = 0;
+                     for (int qy = 0; qy < Q1D; ++qy) {
+                        xy  += r_x[qy] * s_Bt(dy, qy);
+                        xDy += r_y[qy] * s_Gt(dy, qy);
+                     }
+                     s_xy(dy, qx) = xy;
+                     s_xDy(dy, qx) = xDy;
+                  }
+               }
+            }
+            for (int dx = 0; dx < D1D; ++dx; @inner) {
+               if (dx < D1D) {
+                  for (int dy = 0; dy < D1D; ++dy) {
+                     double s = 0;
+                     for (int qx = 0; qx < Q1D; ++qx) {
+                        s += ((s_xy(dy, qx) * s_Gt(dx, qx)) +
+                              (s_xDy(dy, qx) * s_Bt(dx, qx)));
+                     }
+                     Y(dx, dy, e) += s;
+                  }
+               }
+            }
+         }
+      }
+   }
+}
+
+@kernel void DiffusionApply3D_CPU(const int NE,
+                                  @restrict const DofToQuad_t B,
+                                  @restrict const DofToQuad_t G,
+                                  @restrict const QuadToDof_t Bt,
+                                  @restrict const QuadToDof_t Gt,
+                                  @restrict const SymmOperator3D_t op,
+                                  @restrict const DLocal3D_t X,
+                                  @restrict DLocal3D_t Y) {
+  // Iterate over elements
+  for (int e = 0; e < NE; ++e; @outer) {
+    for (int dummy = 0; dummy < 1; ++dummy; @inner) {
+      double grad[Q1D][Q1D][Q1D][4];
+      for (int qz = 0; qz < Q1D; ++qz) {
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            grad[qz][qy][qx][0] = 0;
+            grad[qz][qy][qx][1] = 0;
+            grad[qz][qy][qx][2] = 0;
+          }
+        }
+      }
+
+      for (int dz = 0; dz < D1D; ++dz) {
+        double gradXY[Q1D][Q1D][4];
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            gradXY[qy][qx][0] = 0;
+            gradXY[qy][qx][1] = 0;
+            gradXY[qy][qx][2] = 0;
+          }
+        }
+
+        for (int dy = 0; dy < D1D; ++dy) {
+          double gradX[Q1D][2];
+          for (int qx = 0; qx < Q1D; ++qx) {
+            gradX[qx][0] = 0;
+            gradX[qx][1] = 0;
+          }
+
+          for (int dx = 0; dx < D1D; ++dx) {
+            const double s = X(dx, dy, dz, e);
+            for (int qx = 0; qx < Q1D; ++qx) {
+              gradX[qx][0] += s * B(qx, dx);
+              gradX[qx][1] += s * G(qx, dx);
+            }
+          }
+
+          for (int qy = 0; qy < Q1D; ++qy) {
+            const double wy  = B(qy, dy);
+            const double wDy = G(qy, dy);
+            for (int qx = 0; qx < Q1D; ++qx) {
+              const double wx  = gradX[qx][0];
+              const double wDx = gradX[qx][1];
+              gradXY[qy][qx][0] += wDx * wy;
+              gradXY[qy][qx][1] += wx  * wDy;
+              gradXY[qy][qx][2] += wx  * wy;
+            }
+          }
+        }
+
+        for (int qz = 0; qz < Q1D; ++qz) {
+          const double wz  = B(qz, dz);
+          const double wDz = G(qz, dz);
+          for (int qy = 0; qy < Q1D; ++qy) {
+            for (int qx = 0; qx < Q1D; ++qx) {
+              grad[qz][qy][qx][0] += gradXY[qy][qx][0] * wz;
+              grad[qz][qy][qx][1] += gradXY[qy][qx][1] * wz;
+              grad[qz][qy][qx][2] += gradXY[qy][qx][2] * wDz;
+            }
+          }
+        }
+      }
+
+      // Calculate Dxyz, xDyz, xyDz in plane
+      for (int qz = 0; qz < Q1D; ++qz) {
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            const int q = QUAD_3D_ID(qx, qy, qz);
+            const double O11 = op(0, q, e);
+            const double O12 = op(1, q, e);
+            const double O13 = op(2, q, e);
+            const double O22 = op(3, q, e);
+            const double O23 = op(4, q, e);
+            const double O33 = op(5, q, e);
+
+            const double gradX = grad[qz][qy][qx][0];
+            const double gradY = grad[qz][qy][qx][1];
+            const double gradZ = grad[qz][qy][qx][2];
+
+            grad[qz][qy][qx][0] = (O11 * gradX) + (O12 * gradY) + (O13 * gradZ);
+            grad[qz][qy][qx][1] = (O12 * gradX) + (O22 * gradY) + (O23 * gradZ);
+            grad[qz][qy][qx][2] = (O13 * gradX) + (O23 * gradY) + (O33 * gradZ);
+          }
+        }
+      }
+
+      for (int qz = 0; qz < Q1D; ++qz) {
+        double gradXY[D1D][D1D][4];
+        for (int dy = 0; dy < D1D; ++dy) {
+          for (int dx = 0; dx < D1D; ++dx) {
+            gradXY[dy][dx][0] = 0;
+            gradXY[dy][dx][1] = 0;
+            gradXY[dy][dx][2] = 0;
+          }
+        }
+
+        for (int qy = 0; qy < Q1D; ++qy) {
+          double gradX[D1D][4];
+          for (int dx = 0; dx < D1D; ++dx) {
+            gradX[dx][0] = 0;
+            gradX[dx][1] = 0;
+            gradX[dx][2] = 0;
+          }
+
+          for (int qx = 0; qx < Q1D; ++qx) {
+            const double gX = grad[qz][qy][qx][0];
+            const double gY = grad[qz][qy][qx][1];
+            const double gZ = grad[qz][qy][qx][2];
+            for (int dx = 0; dx < D1D; ++dx) {
+              const double wx  = Bt(dx, qx);
+              const double wDx = Gt(dx, qx);
+              gradX[dx][0] += gX * wDx;
+              gradX[dx][1] += gY * wx;
+              gradX[dx][2] += gZ * wx;
+            }
+          }
+
+          for (int dy = 0; dy < D1D; ++dy) {
+            const double wy  = Bt(dy, qy);
+            const double wDy = Gt(dy, qy);
+            for (int dx = 0; dx < D1D; ++dx) {
+              gradXY[dy][dx][0] += gradX[dx][0] * wy;
+              gradXY[dy][dx][1] += gradX[dx][1] * wDy;
+              gradXY[dy][dx][2] += gradX[dx][2] * wy;
+            }
+          }
+        }
+
+        for (int dz = 0; dz < D1D; ++dz) {
+          const double wz  = Bt(dz, qz);
+          const double wDz = Gt(dz, qz);
+          for (int dy = 0; dy < D1D; ++dy) {
+            for (int dx = 0; dx < D1D; ++dx) {
+              Y(dx, dy, dz, e) += ((gradXY[dy][dx][0] * wz) +
+                                   (gradXY[dy][dx][1] * wz) +
+                                   (gradXY[dy][dx][2] * wDz));
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+@kernel void DiffusionApply3D_GPU(const int NE,
+                                  @restrict const DofToQuad_t B,
+                                  @restrict const DofToQuad_t G,
+                                  @restrict const QuadToDof_t Bt,
+                                  @restrict const QuadToDof_t Gt,
+                                  @restrict const SymmOperator3D_t op,
+                                  @restrict const DLocal3D_t X,
+                                  @restrict DLocal3D_t Y) {
+  // Iterate over elements
+  for (int e = 0; e < NE; ++e; @outer) {
+    // Store dof <--> quad mappings
+    @shared double s_B[DQ1D] @dim(Q1D, D1D);
+    @shared double s_G[DQ1D] @dim(Q1D, D1D);
+    @shared double s_Bt[DQ1D] @dim(D1D, Q1D);
+    @shared double s_Gt[DQ1D] @dim(D1D, Q1D);
+
+    // Store xy planes in @shared memory
+    @shared double s_z[M2D] @dim(M1D, M1D);
+    @shared double s_Dz[M2D] @dim(M1D, M1D);
+    @shared double s_xyDz[Q2D] @dim(Q1D, Q1D);
+
+    // Store z axis as registers
+    @exclusive double r_qz[Q1D];
+    @exclusive double r_qDz[Q1D];
+    @exclusive double r_dDxyz[D1D];
+    @exclusive double r_dxDyz[D1D];
+    @exclusive double r_dxyDz[D1D];
+
+    for (int y = 0; y < M1D; ++y; @inner) {
+      for (int x = 0; x < M1D; ++x; @inner) {
+        const int id = (y * M1D) + x;
+        // Fetch Q <--> D maps
+        if (id < DQ1D) {
+          s_B[id]  = B[id];
+          s_G[id] = G[id];
+          s_Bt[id]  = Bt[id];
+          s_Gt[id] = Gt[id];
+        }
+        // Initialize our Z axis
+        for (int qz = 0; qz < Q1D; ++qz) {
+          r_qz[qz] = 0;
+          r_qDz[qz] = 0;
+        }
+        // Initialize our solution updates in the Z axis
+        for (int dz = 0; dz < D1D; ++dz) {
+          r_dDxyz[dz] = 0;
+          r_dxDyz[dz] = 0;
+          r_dxyDz[dz] = 0;
+        }
+      }
+    }
+
+    for (int dy = 0; dy < M1D; ++dy; @inner) {
+      for (int dx = 0; dx < M1D; ++dx; @inner) {
+        if ((dx < D1D) && (dy < D1D)) {
+          for (int dz = 0; dz < D1D; ++dz) {
+            const double s = X(dx, dy, dz, e);
+            // Calculate D -> Q in the Z axis
+            for (int qz = 0; qz < Q1D; ++qz) {
+              r_qz[qz]  += s * s_B(qz, dz);
+              r_qDz[qz] += s * s_G(qz, dz);
+            }
+          }
+        }
+      }
+    }
+    // For each xy plane
+    for (int qz = 0; qz < Q1D; ++qz) {
+      // Fill xy plane at given z position
+      for (int dy = 0; dy < M1D; ++dy; @inner) {
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if ((dx < D1D) && (dy < D1D)) {
+            s_z(dx, dy)  = r_qz[qz];
+            s_Dz(dx, dy) = r_qDz[qz];
+          }
+        }
+      }
+      // Calculate Dxyz, xDyz, xyDz in plane
+      for (int qy = 0; qy < M1D; ++qy; @inner) {
+        for (int qx = 0; qx < M1D; ++qx; @inner) {
+          if ((qx < Q1D) && (qy < Q1D)) {
+            double Dxyz = 0;
+            double xDyz = 0;
+            double xyDz = 0;
+            for (int dy = 0; dy < D1D; ++dy) {
+              const double wy  = s_B(qy, dy);
+              const double wDy = s_G(qy, dy);
+              for (int dx = 0; dx < D1D; ++dx) {
+                const double wx  = s_B(qx, dx);
+                const double wDx = s_G(qx, dx);
+                const double z  = s_z(dx, dy);
+                const double Dz = s_Dz(dx, dy);
+                Dxyz += wDx * wy  * z;
+                xDyz += wx  * wDy * z;
+                xyDz += wx  * wy  * Dz;
+              }
+            }
+
+            const int q = QUAD_3D_ID(qx, qy, qz);
+            const double O11 = op(0, q, e);
+            const double O12 = op(1, q, e);
+            const double O13 = op(2, q, e);
+            const double O22 = op(3, q, e);
+            const double O23 = op(4, q, e);
+            const double O33 = op(5, q, e);
+
+            const double qDxyz = (O11 * Dxyz) + (O12 * xDyz) + (O13 * xyDz);
+            const double qxDyz = (O12 * Dxyz) + (O22 * xDyz) + (O23 * xyDz);
+            const double qxyDz = (O13 * Dxyz) + (O23 * xDyz) + (O33 * xyDz);
+
+            for (int dz = 0; dz < D1D; ++dz) {
+              const double wz  = s_Bt(dz, qz);
+              const double wDz = s_Gt(dz, qz);
+              r_dDxyz[dz] += wz  * qDxyz;
+              r_dxDyz[dz] += wz  * qxDyz;
+              r_dxyDz[dz] += wDz * qxyDz;
+            }
+          }
+        }
+      }
+      @barrier("s_z_s_Dz_sync_1");
+    }
+    // Iterate over xy planes to compute solution
+    for (int dz = 0; dz < D1D; ++dz) {
+      // Place xy plane in @shared memory
+      for (int qy = 0; qy < M1D; ++qy; @inner) {
+        for (int qx = 0; qx < M1D; ++qx; @inner) {
+          if ((qx < Q1D) && (qy < Q1D)) {
+            s_z(qx, qy)    = r_dDxyz[dz];
+            s_Dz(qx, qy)   = r_dxDyz[dz];
+            s_xyDz(qx, qy) = r_dxyDz[dz];
+          }
+        }
+      }
+      // Finalize solution in xy plane
+      for (int dy = 0; dy < M1D; ++dy; @inner) {
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if ((dx < D1D) && (dy < D1D)) {
+            double solZ = 0;
+            for (int qy = 0; qy < Q1D; ++qy) {
+              const double wy  = s_Bt(dy, qy);
+              const double wDy = s_Gt(dy, qy);
+              for (int qx = 0; qx < Q1D; ++qx) {
+                const double wx  = s_Bt(dx, qx);
+                const double wDx = s_Gt(dx, qx);
+                const double Dxyz = s_z(qx, qy);
+                const double xDyz = s_Dz(qx, qy);
+                const double xyDz = s_xyDz(qx, qy);
+                solZ += ((wDx * wy  * Dxyz) +
+                         (wx  * wDy * xDyz) +
+                         (wx  * wy  * xyDz));
+              }
+            }
+            Y(dx, dy, dz, e) += solZ;
+          }
+        }
+      }
+      @barrier("s_z_s_Dz_s_xyDz_sync_1");
+    }
+  }
+}
+
+@kernel void MassApply2D_CPU(const int NE,
+                             @restrict const DofToQuad_t B,
+                             @restrict const QuadToDof_t Bt,
+                             @restrict const QLocal2D_t op,
+                             @restrict const DLocal2D_t X,
+                             @restrict DLocal2D_t Y) {
+  for (int e = 0; e < NE; ++e; @outer) {
+    for (int dummy = 0; dummy < 1; ++dummy; @inner) {
+      double sol_xy[Q1D][Q1D];
+
+      for (int qy = 0; qy < Q1D; ++qy) {
+        for (int qx = 0; qx < Q1D; ++qx) {
+          sol_xy[qy][qx] = 0;
+        }
+      }
+
+      for (int dy = 0; dy < D1D; ++dy) {
+        double sol_x[Q1D];
+        for (int qy = 0; qy < Q1D; ++qy) {
+          sol_x[qy] = 0;
+        }
+
+        for (int dx = 0; dx < D1D; ++dx) {
+          const double s = X(dx, dy, e);
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_x[qx] += B(qx, dx) * s;
+          }
+        }
+
+        for (int qy = 0; qy < Q1D; ++qy) {
+          const double d2q = B(qy, dy);
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_xy[qy][qx] += d2q * sol_x[qx];
+          }
+        }
+      }
+
+      for (int qy = 0; qy < Q1D; ++qy) {
+        for (int qx = 0; qx < Q1D; ++qx) {
+          sol_xy[qy][qx] *= op(qx, qy, e);
+        }
+      }
+
+      for (int qy = 0; qy < Q1D; ++qy) {
+        double sol_x[D1D];
+        for (int dx = 0; dx < D1D; ++dx) {
+          sol_x[dx] = 0;
+        }
+
+        for (int qx = 0; qx < Q1D; ++qx) {
+          const double s = sol_xy[qy][qx];
+          for (int dx = 0; dx < D1D; ++dx) {
+            sol_x[dx] += Bt(dx, qx) * s;
+          }
+        }
+
+        for (int dy = 0; dy < D1D; ++dy) {
+          const double q2d = Bt(dy, qy);
+          for (int dx = 0; dx < D1D; ++dx) {
+            Y(dx, dy, e) += q2d * sol_x[dx];
+          }
+        }
+      }
+    }
+  }
+}
+
+@kernel void MassApply3D_CPU(const int NE,
+                             @restrict const DofToQuad_t B,
+                             @restrict const QuadToDof_t Bt,
+                             @restrict const QLocal3D_t op,
+                             @restrict const DLocal3D_t X,
+                             @restrict DLocal3D_t Y) {
+  // Iterate over elements
+  for (int e = 0; e < NE; ++e; @outer) {
+    for (int dummy = 0; dummy < 1; ++dummy; @inner) {
+      double sol_xyz[Q1D][Q1D][Q1D];
+      for (int qz = 0; qz < Q1D; ++qz) {
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_xyz[qz][qy][qx] = 0;
+          }
+        }
+      }
+
+      for (int dz = 0; dz < D1D; ++dz) {
+        double sol_xy[Q1D][Q1D];
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_xy[qy][qx] = 0;
+          }
+        }
+
+        for (int dy = 0; dy < D1D; ++dy) {
+          double sol_x[Q1D];
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_x[qx] = 0;
+          }
+
+          for (int dx = 0; dx < D1D; ++dx) {
+            const double s = X(dx, dy, dz, e);
+            for (int qx = 0; qx < Q1D; ++qx) {
+              sol_x[qx] += B(qx, dx) * s;
+            }
+          }
+
+          for (int qy = 0; qy < Q1D; ++qy) {
+            const double wy = B(qy, dy);
+            for (int qx = 0; qx < Q1D; ++qx) {
+              sol_xy[qy][qx] += wy * sol_x[qx];
+            }
+          }
+        }
+
+        for (int qz = 0; qz < Q1D; ++qz) {
+          const double wz = B(qz, dz);
+          for (int qy = 0; qy < Q1D; ++qy) {
+            for (int qx = 0; qx < Q1D; ++qx) {
+              sol_xyz[qz][qy][qx] += wz * sol_xy[qy][qx];
+            }
+          }
+        }
+      }
+
+      for (int qz = 0; qz < Q1D; ++qz) {
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_xyz[qz][qy][qx] *= op(qx, qy, qz, e);
+          }
+        }
+      }
+
+      for (int qz = 0; qz < Q1D; ++qz) {
+        double sol_xy[D1D][D1D];
+        for (int dy = 0; dy < D1D; ++dy) {
+          for (int dx = 0; dx < D1D; ++dx) {
+            sol_xy[dy][dx] = 0;
+          }
+        }
+
+        for (int qy = 0; qy < Q1D; ++qy) {
+          double sol_x[D1D];
+          for (int dx = 0; dx < D1D; ++dx) {
+            sol_x[dx] = 0;
+          }
+
+          for (int qx = 0; qx < Q1D; ++qx) {
+            const double s = sol_xyz[qz][qy][qx];
+            for (int dx = 0; dx < D1D; ++dx) {
+              sol_x[dx] += Bt(dx, qx) * s;
+            }
+          }
+
+          for (int dy = 0; dy < D1D; ++dy) {
+            const double wy = Bt(dy, qy);
+            for (int dx = 0; dx < D1D; ++dx) {
+              sol_xy[dy][dx] += wy * sol_x[dx];
+            }
+          }
+        }
+
+        for (int dz = 0; dz < D1D; ++dz) {
+          const double wz = Bt(dz, qz);
+          for (int dy = 0; dy < D1D; ++dy) {
+            for (int dx = 0; dx < D1D; ++dx) {
+              Y(dx, dy, dz, e) += wz * sol_xy[dy][dx];
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+@kernel void MassApply2D_GPU(const int NE,
+                             @restrict const DofToQuad_t B,
+                             @restrict const QuadToDof_t Bt,
+                             @restrict const QLocal2D_t op,
+                             @restrict const DLocal2D_t X,
+                             @restrict DLocal2D_t Y) {
+  // Iterate over elements
+  for (int eOff = 0; eOff < NE; eOff += M2_ELEMENT_BATCH; @outer) {
+    // Store dof <--> quad mappings
+    @shared double s_B[DQ1D] @dim(Q1D, D1D);
+    @shared double s_Bt[DQ1D] @dim(D1D, Q1D);
+
+    // Store xy planes in @shared memory
+    @shared double s_xy[DQ1D] @dim(D1D, Q1D);
+    @shared double s_xy2[Q2D] @dim(Q1D, Q1D);
+
+    @exclusive double r_x[M1D];
+
+    for (int x = 0; x < M1D; ++x; @inner) {
+      for (int id = x; id < DQ1D; id += M1D) {
+        s_B[id]  = B[id];
+        s_Bt[id]  = Bt[id];
+      }
+    }
+
+    for (int e = eOff; e < (eOff + M2_ELEMENT_BATCH); ++e) {
+      if (e < NE) {
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if (dx < D1D) {
+            for (int qy = 0; qy < Q1D; ++qy) {
+              s_xy(dx, qy) = 0;
+            }
+            for (int dy = 0; dy < D1D; ++dy) {
+              r_x[dy] = X(dx, dy, e);
+            }
+            for (int qy = 0; qy < Q1D; ++qy) {
+              double xy = 0;
+              for (int dy = 0; dy < D1D; ++dy) {
+                xy += r_x[dy] * s_B(qy, dy);
+              }
+              s_xy(dx, qy) = xy;
+            }
+          }
+        }
+        for (int qy = 0; qy < M1D; ++qy; @inner) {
+          if (qy < Q1D) {
+            for (int qx = 0; qx < Q1D; ++qx) {
+              double s = 0;
+              for (int dx = 0; dx < D1D; ++dx) {
+                s += s_xy(dx, qy) * s_B(qx, dx);
+              }
+              s_xy2(qx, qy) = s * op(qx, qy, e);
+            }
+          }
+        }
+
+        for (int qx = 0; qx < M1D; ++qx; @inner) {
+          if (qx < Q1D) {
+            for (int dy = 0; dy < D1D; ++dy) {
+              s_xy(dy, qx) = 0;
+            }
+            for (int qy = 0; qy < Q1D; ++qy) {
+              r_x[qy] = s_xy2(qx, qy);
+            }
+            for (int dy = 0; dy < D1D; ++dy) {
+              double s = 0;
+              for (int qy = 0; qy < Q1D; ++qy) {
+                s += r_x[qy] * s_Bt(dy, qy);
+              }
+              s_xy(dy, qx) = s;
+            }
+          }
+        }
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if (dx < D1D) {
+            for (int dy = 0; dy < D1D; ++dy) {
+              double s = 0;
+              for (int qx = 0; qx < Q1D; ++qx) {
+                s += (s_xy(dy, qx) * s_Bt(dx, qx));
+              }
+              Y(dx, dy, e) += s;
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+@kernel void MassApply3D_GPU(const int NE,
+                             @restrict const DofToQuad_t B,
+                             @restrict const QuadToDof_t Bt,
+                             @restrict const QLocal3D_t op,
+                             @restrict const DLocal3D_t X,
+                             @restrict DLocal3D_t Y) {
+  // Iterate over elements
+  for (int e = 0; e < NE; ++e; @outer) {
+    // Store dof <--> quad mappings
+    @shared double s_B[DQ1D] @dim(Q1D, D1D);
+    @shared double s_Bt[DQ1D] @dim(D1D, Q1D);
+
+    // Store xy planes in @shared memory
+    @shared double s_xy[M2D] @dim(M1D, M1D);
+
+    // Store z axis as registers
+    @exclusive double r_z[Q1D];
+    @exclusive double r_z2[D1D];
+
+    for (int y = 0; y < M1D; ++y; @inner) {
+      for (int x = 0; x < M1D; ++x; @inner) {
+        const int id = (y * M1D) + x;
+        // Fetch Q <--> D maps
+        if (id < DQ1D) {
+          s_B[id]  = B[id];
+          s_Bt[id]  = Bt[id];
+        }
+        // Initialize our Z axis
+        for (int qz = 0; qz < Q1D; ++qz) {
+          r_z[qz] = 0;
+        }
+        for (int dz = 0; dz < D1D; ++dz) {
+          r_z2[dz] = 0;
+        }
+      }
+    }
+
+    for (int dy = 0; dy < M1D; ++dy; @inner) {
+      for (int dx = 0; dx < M1D; ++dx; @inner) {
+        if ((dx < D1D) && (dy < D1D)) {
+          for (int dz = 0; dz < D1D; ++dz) {
+            const double s = X(dx, dy, dz, e);
+            // Calculate D -> Q in the Z axis
+            for (int qz = 0; qz < Q1D; ++qz) {
+              r_z[qz] += s * s_B(qz, dz);
+            }
+          }
+        }
+      }
+    }
+    // For each xy plane
+    for (int qz = 0; qz < Q1D; ++qz) {
+      // Fill xy plane at given z position
+      for (int dy = 0; dy < M1D; ++dy; @inner) {
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if ((dx < D1D) && (dy < D1D)) {
+            s_xy(dx, dy) = r_z[qz];
+          }
+        }
+      }
+      // Calculate Dxyz, xDyz, xyDz in plane
+      for (int qy = 0; qy < M1D; ++qy; @inner) {
+        for (int qx = 0; qx < M1D; ++qx; @inner) {
+          if ((qx < Q1D) && (qy < Q1D)) {
+            double s = 0;
+            for (int dy = 0; dy < D1D; ++dy) {
+              const double wy = s_B(qy, dy);
+              for (int dx = 0; dx < D1D; ++dx) {
+                const double wx = s_B(qx, dx);
+                s += wx * wy * s_xy(dx, dy);
+              }
+            }
+
+            s *= op(qx, qy, qz, e);
+
+            for (int dz = 0; dz < D1D; ++dz) {
+              const double wz  = s_Bt(dz, qz);
+              r_z2[dz] += wz * s;
+            }
+          }
+        }
+      }
+      @barrier("s_xy_sync_1");
+    }
+    // Iterate over xy planes to compute solution
+    for (int dz = 0; dz < D1D; ++dz) {
+      // Place xy plane in @shared memory
+      for (int qy = 0; qy < M1D; ++qy; @inner) {
+        for (int qx = 0; qx < M1D; ++qx; @inner) {
+          if ((qx < Q1D) && (qy < Q1D)) {
+            s_xy(qx, qy) = r_z2[dz];
+          }
+        }
+      }
+      // Finalize solution in xy plane
+      for (int dy = 0; dy < M1D; ++dy; @inner) {
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if ((dx < D1D) && (dy < D1D)) {
+            double solZ = 0;
+            for (int qy = 0; qy < Q1D; ++qy) {
+              const double wy = s_Bt(dy, qy);
+              for (int qx = 0; qx < Q1D; ++qx) {
+                const double wx = s_Bt(dx, qx);
+                solZ += wx * wy * s_xy(qx, qy);
+              }
+            }
+            Y(dx, dy, dz, e) += solZ;
+          }
+        }
+      }
+      @barrier("s_xy_sync_2");
+    }
+  }
+}

fem/pbilinearform.cpp

@@ -284,6 +284,12 @@ void ParBilinearForm::FormLinearSystem(
    const Array<int> &ess_tdof_list, Vector &x, Vector &b,
    OperatorHandle &A, Vector &X, Vector &B, int copy_interior)
 {
+   if (ext)
+   {
+      ext->FormLinearSystem(ess_tdof_list, x, b, A, X, B, copy_interior);
+      return;
+   }
+
    // Finish the matrix assembly and perform BC elimination, storing the
    // eliminated part of the matrix.
    FormSystemMatrix(ess_tdof_list, A);
@@ -327,6 +333,12 @@ void ParBilinearForm::FormLinearSystem(
 void ParBilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
                                        OperatorHandle &A)
 {
+   if (ext)
+   {
+      ext->FormSystemMatrix(ess_tdof_list, A);
+      return;
+   }
+
    // Finish the matrix assembly and perform BC elimination, storing the
    // eliminated part of the matrix.
    if (static_cond)
@@ -369,6 +381,12 @@ void ParBilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
 void ParBilinearForm::RecoverFEMSolution(
    const Vector &X, const Vector &b, Vector &x)
 {
+   if (ext)
+   {
+      ext->RecoverFEMSolution(X, b, x);
+      return;
+   }
+
    const Operator &P = *pfes->GetProlongationMatrix();
 
    if (static_cond)
@@ -467,8 +485,8 @@ void ParMixedBilinearForm::TrueAddMult(const Vector &x, Vector &y,
 
 HypreParMatrix* ParDiscreteLinearOperator::ParallelAssemble() const
 {
-   MFEM_ASSERT(mat, "matrix is not assembled");
-   MFEM_ASSERT(mat->Finalized(), "matrix is not finalized");
+   MFEM_ASSERT(mat, "Matrix is not assembled");
+   MFEM_ASSERT(mat->Finalized(), "Matrix is not finalized");
    SparseMatrix* RA = mfem::Mult(*range_fes->GetRestrictionMatrix(), *mat);
    HypreParMatrix* P = domain_fes->Dof_TrueDof_Matrix();
    HypreParMatrix* RAP = P->LeftDiagMult(*RA, range_fes->GetTrueDofOffsets());
@@ -479,7 +497,7 @@ HypreParMatrix* ParDiscreteLinearOperator::ParallelAssemble() const
 void ParDiscreteLinearOperator::GetParBlocks(Array2D<HypreParMatrix *> &blocks)
 const
 {
-   MFEM_VERIFY(mat->Finalized(), "local matrix needs to be finalized for "
+   MFEM_VERIFY(mat->Finalized(), "Local matrix needs to be finalized for "
                "GetParBlocks");
 
    HypreParMatrix* RLP = ParallelAssemble();

fem/pbilinearform.hpp

@@ -75,7 +75,8 @@ public:
        those rows. Must be called before the first Assemble call. */
    void KeepNbrBlock(bool knb = true) { keep_nbr_block = knb; }
 
-   /// Set the operator type id for the parallel matrix/operator.
+   /** @brief Set the operator type id for the parallel matrix/operator when
+       using AssemblyLevel::FULL. */
    /** If using static condensation or hybridization, call this method *after*
        enabling it. */
    void SetOperatorType(Operator::Type tid)
@@ -163,65 +164,15 @@ public:
    virtual const Operator *GetRestriction() const
    { return pfes->GetRestrictionMatrix(); }
 
-   /** Form the linear system A X = B, corresponding to the current bilinear
-       form and b(.), by applying any necessary transformations such as:
-       eliminating boundary conditions; applying conforming constraints for
-       non-conforming AMR; parallel assembly; static condensation;
-       hybridization.
+   using BilinearForm::FormLinearSystem;
+   using BilinearForm::FormSystemMatrix;
 
-       The ParGridFunction-size vector x must contain the essential b.c. The
-       ParBilinearForm and the ParLinearForm-size vector b must be assembled.
+   virtual void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x,
+                                 Vector &b, OperatorHandle &A, Vector &X,
+                                 Vector &B, int copy_interior = 0);
 
-       The vector X is initialized with a suitable initial guess: when using
-       hybridization, the vector X is set to zero; otherwise, the essential
-       entries of X are set to the corresponding b.c. and all other entries are
-       set to zero (copy_interior == 0) or copied from x (copy_interior != 0).
-
-       This method can be called multiple times (with the same ess_tdof_list
-       array) to initialize different right-hand sides and boundary condition
-       values.
-
-       After solving the linear system, the finite element solution x can be
-       recovered by calling RecoverFEMSolution (with the same vectors X, b, and
-       x). */
-   void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x, Vector &b,
-                         OperatorHandle &A, Vector &X, Vector &B,
-                         int copy_interior = 0);
-
-   /** Version of the method FormLinearSystem() where the system matrix is
-       returned in the variable @a A, of type OpType, holding a *reference* to
-       the system matrix (created with the method OpType::MakeRef()). The
-       reference will be invalidated when SetOperatorType(), Update(), or the
-       destructor is called. */
-   template <typename OpType>
-   void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x, Vector &b,
-                         OpType &A, Vector &X, Vector &B,
-                         int copy_interior = 0)
-   {
-      OperatorHandle Ah;
-      FormLinearSystem(ess_tdof_list, x, b, Ah, X, B, copy_interior);
-      OpType *A_ptr = Ah.Is<OpType>();
-      MFEM_VERIFY(A_ptr, "invalid OpType used");
-      A.MakeRef(*A_ptr);
-   }
-
-   /// Form the linear system matrix @a A, see FormLinearSystem() for details.
-   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A);
-
-   /** Version of the method FormSystemMatrix() where the system matrix is
-       returned in the variable @a A, of type OpType, holding a *reference* to
-       the system matrix (created with the method OpType::MakeRef()). The
-       reference will be invalidated when SetOperatorType(), Update(), or the
-       destructor is called. */
-   template <typename OpType>
-   void FormSystemMatrix(const Array<int> &ess_tdof_list, OpType &A)
-   {
-      OperatorHandle Ah;
-      FormSystemMatrix(ess_tdof_list, Ah);
-      OpType *A_ptr = Ah.Is<OpType>();
-      MFEM_VERIFY(A_ptr, "invalid OpType used");
-      A.MakeRef(*A_ptr);
-   }
+   virtual void FormSystemMatrix(const Array<int> &ess_tdof_list,
+                                 OperatorHandle &A);
 
    /** Call this method after solving a linear system constructed using the
        FormLinearSystem method to recover the solution as a ParGridFunction-size

fem/pfespace.cpp

@@ -613,15 +613,15 @@ void ParFiniteElementSpace::Build_Dof_TrueDof_Matrix() const // matrix P
    int ldof  = GetVSize();
    int ltdof = TrueVSize();
 
-   HYPRE_Int *i_diag = new HYPRE_Int[ldof+1];
-   HYPRE_Int *j_diag = new HYPRE_Int[ltdof];
+   HYPRE_Int *i_diag = mfem::New<HYPRE_Int>(ldof+1);
+   HYPRE_Int *j_diag = mfem::New<HYPRE_Int>(ltdof);
    int diag_counter;
 
-   HYPRE_Int *i_offd = new HYPRE_Int[ldof+1];
-   HYPRE_Int *j_offd = new HYPRE_Int[ldof-ltdof];
+   HYPRE_Int *i_offd = mfem::New<HYPRE_Int>(ldof+1);
+   HYPRE_Int *j_offd = mfem::New<HYPRE_Int>(ldof-ltdof);
    int offd_counter;
 
-   HYPRE_Int *cmap   = new HYPRE_Int[ldof-ltdof];
+   HYPRE_Int *cmap   = mfem::New<HYPRE_Int>(ldof-ltdof);
 
    HYPRE_Int *col_starts = GetTrueDofOffsets();
    HYPRE_Int *row_starts = GetDofOffsets();
@@ -2249,7 +2249,7 @@ HypreParMatrix* ParFiniteElementSpace
    }
 
    // create offd column mapping
-   HYPRE_Int *cmap = new HYPRE_Int[col_map.size()];
+   HYPRE_Int *cmap = mfem::New<HYPRE_Int>(col_map.size());
    int offd_col = 0;
    for (std::map<HYPRE_Int, int>::iterator
         it = col_map.begin(); it != col_map.end(); ++it)
@@ -2258,14 +2258,14 @@ HypreParMatrix* ParFiniteElementSpace
       it->second = offd_col++;
    }
 
-   HYPRE_Int *I_diag = new HYPRE_Int[vdim*local_rows + 1];
-   HYPRE_Int *I_offd = new HYPRE_Int[vdim*local_rows + 1];
+   HYPRE_Int *I_diag = mfem::New<HYPRE_Int>(vdim*local_rows + 1);
+   HYPRE_Int *I_offd = mfem::New<HYPRE_Int>(vdim*local_rows + 1);
 
-   HYPRE_Int *J_diag = new HYPRE_Int[nnz_diag];
-   HYPRE_Int *J_offd = new HYPRE_Int[nnz_offd];
+   HYPRE_Int *J_diag = mfem::New<HYPRE_Int>(nnz_diag);
+   HYPRE_Int *J_offd = mfem::New<HYPRE_Int>(nnz_offd);
 
-   double *A_diag = new double[nnz_diag];
-   double *A_offd = new double[nnz_offd];
+   double *A_diag = mfem::New<double>(nnz_diag);
+   double *A_offd = mfem::New<double>(nnz_offd);
 
    int vdim1 = bynodes ? vdim : 1;
    int vdim2 = bynodes ? 1 : vdim;
@@ -2316,7 +2316,7 @@ HypreParMatrix* ParFiniteElementSpace
 
 static HYPRE_Int* make_i_array(int nrows)
 {
-   HYPRE_Int *I = new HYPRE_Int[nrows+1];
+   HYPRE_Int *I = mfem::New<HYPRE_Int>(nrows+1);
    for (int i = 0; i <= nrows; i++) { I[i] = -1; }
    return I;
 }
@@ -2328,7 +2328,7 @@ static HYPRE_Int* make_j_array(HYPRE_Int* I, int nrows)
    {
       if (I[i] >= 0) { nnz++; }
    }
-   HYPRE_Int *J = new HYPRE_Int[nnz];
+   HYPRE_Int *J = mfem::New<HYPRE_Int>(nnz);
 
    I[nrows] = -1;
    for (int i = 0, k = 0; i <= nrows; i++)
@@ -2427,7 +2427,7 @@ ParFiniteElementSpace::RebalanceMatrix(int old_ndofs,
    }
    SortPairs<HYPRE_Int, int>(cmap_offd, offd_cols);
 
-   HYPRE_Int* cmap = new HYPRE_Int[offd_cols];
+   HYPRE_Int* cmap = mfem::New<HYPRE_Int>(offd_cols);
    for (int i = 0; i < offd_cols; i++)
    {
       cmap[i] = cmap_offd[i].one;
@@ -2623,7 +2623,7 @@ ParFiniteElementSpace::ParallelDerefinementMatrix(int old_ndofs,
    offd->SetWidth(col_map.size());
 
    // create offd column mapping for use by hypre
-   HYPRE_Int *cmap = new HYPRE_Int[offd->Width()];
+   HYPRE_Int *cmap = mfem::New<HYPRE_Int>(offd->Width());
    for (std::map<HYPRE_Int, int>::iterator
         it = col_map.begin(); it != col_map.end(); ++it)
    {
@@ -2865,6 +2865,7 @@ void ConformingProlongationOperator::Mult(const Vector &x, Vector &y) const
 
    const double *xdata = x.GetData();
    double *ydata = y.GetData();
+   x.Pull();
    const int m = external_ldofs.Size();
 
    const int in_layout = 2; // 2 - input is ltdofs array
@@ -2881,6 +2882,7 @@ void ConformingProlongationOperator::Mult(const Vector &x, Vector &y) const
 
    const int out_layout = 0; // 0 - output is ldofs array
    gc.BcastEnd(ydata, out_layout);
+   y.Push();
 }
 
 void ConformingProlongationOperator::MultTranspose(
@@ -2891,6 +2893,7 @@ void ConformingProlongationOperator::MultTranspose(
 
    const double *xdata = x.GetData();
    double *ydata = y.GetData();
+   x.Pull();
    const int m = external_ldofs.Size();
 
    gc.ReduceBegin(xdata);
@@ -2906,6 +2909,7 @@ void ConformingProlongationOperator::MultTranspose(
 
    const int out_layout = 2; // 2 - output is an array on all ltdofs
    gc.ReduceEnd<double>(ydata, out_layout, GroupCommunicator::Sum);
+   y.Push();
 }
 
 } // namespace mfem

fem/plinearform.hpp

@@ -45,6 +45,16 @@ public:
    /** The pointer @a pf is not owned by the newly constructed object. */
    ParLinearForm(ParFiniteElementSpace *pf) : LinearForm(pf) { pfes = pf; }
 
+   /// Construct a ParLinearForm using previously allocated array @a data.
+   /** The ParLinearForm does not assume ownership of @a data which is assumed
+       to be of size at least `pf->GetVSize()`. Similar to the LinearForm and
+       Vector constructors for externally allocated array, the pointer @a data
+       can be NULL. The data array can be replaced later using the method
+       SetData().
+    */
+   ParLinearForm(ParFiniteElementSpace *pf, double *data) :
+      LinearForm(pf, data), pfes(pf) { }
+
    /** @brief Create a ParLinearForm on the ParFiniteElementSpace @a *pf, using
        the same integrators as the ParLinearForm @a *plf.
 

general/CMakeLists.txt

@@ -11,10 +11,14 @@
 
 list(APPEND SRCS
   array.cpp
+  cuda.cpp
+  device.cpp
   error.cpp
   globals.cpp
   gzstream.cpp
   isockstream.cpp
+  mem_manager.cpp
+  occa.cpp
   optparser.cpp
   osockstream.cpp
   sets.cpp
@@ -27,12 +31,18 @@ list(APPEND SRCS
 
 list(APPEND HDRS
   array.hpp
+  binaryio.hpp
+  cuda.hpp
+  device.hpp
   error.hpp
   globals.hpp
   gzstream.hpp
   hash.hpp
   isockstream.hpp
   mem_alloc.hpp
+  mem_manager.hpp
+  occa.hpp
+  forall.hpp
   optparser.hpp
   osockstream.hpp
   sets.hpp

general/array.cpp

@@ -13,6 +13,7 @@
 // Abstract array data type
 
 #include "array.hpp"
+#include "../general/forall.hpp"
 #include <fstream>
 
 namespace mfem
@@ -22,7 +23,7 @@ BaseArray::BaseArray(int asize, int ainc, int elementsize)
 {
    if (asize > 0)
    {
-      data = new char[asize * elementsize];
+      data = mfem::New<char>(asize * elementsize);
       size = allocsize = asize;
    }
    else
@@ -37,7 +38,7 @@ BaseArray::~BaseArray()
 {
    if (allocsize > 0)
    {
-      delete [] (char*)data;
+      mfem::Delete((char*)data);
    }
 }
 
@@ -47,14 +48,14 @@ void BaseArray::GrowSize(int minsize, int elementsize)
    int nsize = (inc > 0) ? abs(allocsize) + inc : 2 * abs(allocsize);
    if (nsize < minsize) { nsize = minsize; }
 
-   p = new char[nsize * elementsize];
+   p = mfem::New<char>(nsize * elementsize);
    if (size > 0)
    {
-      memcpy(p, data, size * elementsize);
+      mfem::Memcpy(p, data, size * elementsize);
    }
    if (allocsize > 0)
    {
-      delete [] (char*)data;
+      mfem::Delete((char*)data);
    }
    data = p;
    allocsize = nsize;

general/array.hpp

@@ -13,6 +13,7 @@
 #define MFEM_ARRAY
 
 #include "../config/config.hpp"
+#include "mem_manager.hpp"
 #include "error.hpp"
 #include "globals.hpp"
 
@@ -82,7 +83,7 @@ public:
    /// Copy constructor: deep copy
    Array(const Array<T> &src)
       : BaseArray(src.size, 0, sizeof(T))
-   { std::memcpy(data, src.data, size*sizeof(T)); }
+   { mfem::Memcpy(data, src.data, size*sizeof(T)); }
 
    /// Copy constructor (deep copy) from an Array of convertable type
    template <typename CT>
@@ -190,7 +191,7 @@ public:
    inline void Copy(Array &copy) const
    {
       copy.SetSize(Size());
-      std::memcpy(copy.GetData(), data, Size()*sizeof(T));
+      mfem::Memcpy(copy.GetData(), data, Size()*sizeof(T));
    }
 
    /// Make this Array a reference to a pointer
@@ -713,7 +714,7 @@ inline void Array<T>::DeleteAll()
 {
    if (allocsize > 0)
    {
-      delete [] (char*)data;
+      mfem::Delete((char*)data);
    }
    data = NULL;
    size = allocsize = 0;
@@ -724,7 +725,7 @@ inline void Array<T>::MakeRef(T *p, int s)
 {
    if (allocsize > 0)
    {
-      delete [] (char*)data;
+      mfem::Delete((char*)data);
    }
    data = p;
    size = s;
@@ -736,7 +737,7 @@ inline void Array<T>::MakeRef(const Array &master)
 {
    if (allocsize > 0)
    {
-      delete [] (char*)data;
+      mfem::Delete((char*)data);
    }
    data = master.data;
    size = master.size;

general/cuda.cpp

@@ -0,0 +1,110 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "cuda.hpp"
+
+namespace mfem
+{
+
+void* CuMemAlloc(void** dptr, size_t bytes)
+{
+#ifdef MFEM_USE_CUDA
+   if (CUDA_SUCCESS != ::cuMemAlloc((CUdeviceptr*)dptr, bytes))
+   {
+      mfem_error("Error in CuMemAlloc");
+   }
+#endif
+   return *dptr;
+}
+
+void* CuMemFree(void *dptr)
+{
+#ifdef MFEM_USE_CUDA
+   if (CUDA_SUCCESS != ::cuMemFree((CUdeviceptr)dptr))
+   {
+      mfem_error("Error in CuMemFree");
+   }
+#endif
+   return dptr;
+}
+
+void* CuMemcpyHtoD(void* dst, const void* src, size_t bytes)
+{
+#ifdef MFEM_USE_CUDA
+   if (CUDA_SUCCESS != ::cuMemcpyHtoD((CUdeviceptr)dst, src, bytes))
+   {
+      mfem_error("Error in CuMemcpyHtoD");
+   }
+#endif
+   return dst;
+}
+
+void* CuMemcpyHtoDAsync(void* dst, const void* src, size_t bytes, void *s)
+{
+#ifdef MFEM_USE_CUDA
+   if (CUDA_SUCCESS !=
+       ::cuMemcpyHtoDAsync((CUdeviceptr)dst, src, bytes, (CUstream)s))
+   {
+      mfem_error("Error in CuMemcpyHtoDAsync");
+   }
+#endif
+   return dst;
+}
+
+void* CuMemcpyDtoD(void* dst, void* src, size_t bytes)
+{
+#ifdef MFEM_USE_CUDA
+   if (CUDA_SUCCESS !=
+       ::cuMemcpyDtoD((CUdeviceptr)dst, (CUdeviceptr)src, bytes))
+   {
+      mfem_error("Error in CuMemcpyDtoD");
+   }
+#endif
+   return dst;
+}
+
+void* CuMemcpyDtoDAsync(void* dst, void* src, size_t bytes, void *s)
+{
+#ifdef MFEM_USE_CUDA
+   if (CUDA_SUCCESS !=
+       ::cuMemcpyDtoDAsync((CUdeviceptr)dst, (CUdeviceptr)src,
+                           bytes, (CUstream)s))
+   {
+      mfem_error("Error in CuMemcpyDtoDAsync");
+   }
+#endif
+   return dst;
+}
+
+void* CuMemcpyDtoH(void *dst, void *src, size_t bytes)
+{
+#ifdef MFEM_USE_CUDA
+   if (CUDA_SUCCESS != ::cuMemcpyDtoH(dst, (CUdeviceptr)src, bytes))
+   {
+      mfem_error("Error in CuMemcpyDtoH");
+   }
+#endif
+   return dst;
+}
+
+void* CuMemcpyDtoHAsync(void* dst, void* src, size_t bytes, void *s)
+{
+#ifdef MFEM_USE_CUDA
+   if (CUDA_SUCCESS !=
+       ::cuMemcpyDtoHAsync(dst, (CUdeviceptr)src, bytes, (CUstream)s))
+   {
+      mfem_error("Error in CuMemcpyDtoHAsync");
+   }
+#endif
+   return dst;
+}
+
+} // namespace mfem

general/cuda.hpp

@@ -0,0 +1,140 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_CUDA_HPP
+#define MFEM_CUDA_HPP
+
+#include "../config/config.hpp"
+#include "error.hpp"
+
+#ifdef MFEM_USE_CUDA
+#include <cuda_runtime.h>
+#include <cuda.h>
+#endif
+
+// CUDA block size used by MFEM.
+#define MFEM_CUDA_BLOCKS 256
+
+#ifdef MFEM_USE_CUDA
+#define MFEM_ATTR_DEVICE __device__
+#define MFEM_ATTR_HOST_DEVICE __host__ __device__
+// Define the CUDA debug macros:
+// - MFEM_CUDA_CHECK_DRV(x) where 'x' returns/is type 'CUresult'
+// - MFEM_CUDA_CHECK_RT(x)  where 'x' returns/is type 'cudaError_t'
+#ifdef MFEM_DEBUG
+#define MFEM_CUDA_CHECK_DRV(x) \
+   do \
+   { \
+      CUresult err = (x); \
+      if (err != CUDA_SUCCESS) \
+      { \
+         const char *error_string; \
+         cuGetErrorString(err, &error_string); \
+         _MFEM_MESSAGE("CUDA error: (" << #x \
+                       << ") failed with error:\n --> " \
+                       << error_string, 0); \
+      } \
+   } \
+   while (0)
+#define MFEM_CUDA_CHECK_RT(x) \
+   do \
+   { \
+      cudaError_t err = (x); \
+      if (err != cudaSuccess) \
+      { \
+         _MFEM_MESSAGE("CUDA error: (" << #x \
+                       << ") failed with error:\n --> " \
+                       << cudaGetErrorString(err), 0); \
+      } \
+   } \
+   while (0)
+#else
+#define MFEM_CUDA_CHECK_DRV(x) x
+#define MFEM_CUDA_CHECK_RT(x) x
+#endif
+#else // MFEM_USE_CUDA
+#define MFEM_ATTR_DEVICE
+#define MFEM_ATTR_HOST_DEVICE
+typedef int CUdevice;
+typedef int CUcontext;
+typedef void* CUstream;
+#endif // MFEM_USE_CUDA
+
+
+namespace mfem
+{
+
+// Define 'atomicAdd' function.
+#ifdef __CUDA_ARCH__
+#if __CUDA_ARCH__ < 600
+static __device__ inline double atomicAdd(double* address, double val)
+{
+   unsigned long long int* address_as_ull = (unsigned long long int*)address;
+   unsigned long long int old = *address_as_ull, assumed;
+   do
+   {
+      assumed = old;
+      old =
+         atomicCAS(address_as_ull, assumed,
+                   __double_as_longlong(val +
+                                        __longlong_as_double(assumed)));
+      // Note: uses integer comparison to avoid hang in case of NaN
+      // (since NaN != NaN)
+   }
+   while (assumed != old);
+   return __longlong_as_double(old);
+}
+#endif // __CUDA_ARCH__ < 600
+template<typename T> MFEM_ATTR_DEVICE
+inline T AtomicAdd(T volatile *address, T val)
+{
+   return atomicAdd((T *)address, val);
+}
+#else // __CUDA_ARCH__
+template<typename T> inline T AtomicAdd(T volatile *address, T val)
+{
+#ifdef MFEM_USE_OPENMP
+   #pragma omp atomic
+#endif
+   *address += val;
+   return *address;
+}
+#endif // __CUDA_ARCH__
+
+
+/// Allocates device memory
+void* CuMemAlloc(void **d_ptr, size_t bytes);
+
+/// Frees device memory
+void* CuMemFree(void *d_ptr);
+
+/// Copies memory from Host to Device
+void* CuMemcpyHtoD(void *d_dst, const void *h_src, size_t bytes);
+
+/// Copies memory from Host to Device
+void* CuMemcpyHtoDAsync(void *d_dst, const void *h_src,
+                        size_t bytes, void *stream);
+
+/// Copies memory from Device to Device
+void* CuMemcpyDtoD(void *d_dst, void *d_src, size_t bytes);
+
+/// Copies memory from Device to Device
+void* CuMemcpyDtoDAsync(void *d_dst, void *d_src, size_t bytes, void *stream);
+
+/// Copies memory from Device to Host
+void* CuMemcpyDtoH(void *h_dst, void *d_src, size_t bytes);
+
+/// Copies memory from Device to Host
+void* CuMemcpyDtoHAsync(void *h_dst, void *d_src, size_t bytes, void *stream);
+
+} // namespace mfem
+
+#endif // MFEM_CUDA_HPP

general/device.cpp

@@ -0,0 +1,218 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "forall.hpp"
+#include "cuda.hpp"
+#include "occa.hpp"
+
+#include <string>
+#include <map>
+
+namespace mfem
+{
+
+// Place the following variables in the mfem::internal namespace, so that they
+// will not be included in the doxygen documentation.
+namespace internal
+{
+
+CUstream *cuStream = NULL;
+static CUdevice cuDevice;
+static CUcontext cuContext;
+OccaDevice occaDevice;
+
+// Backends listed by priority, high to low:
+static const Backend::Id backend_list[Backend::NUM_BACKENDS] =
+{
+   Backend::OCCA_CUDA, Backend::RAJA_CUDA, Backend::CUDA,
+   Backend::OCCA_OMP, Backend::RAJA_OMP, Backend::OMP,
+   Backend::OCCA_CPU, Backend::RAJA_CPU, Backend::CPU
+};
+
+// Backend names listed by priority, high to low:
+static const char *backend_name[Backend::NUM_BACKENDS] =
+{
+   "occa-cuda", "raja-cuda", "cuda", "occa-omp", "raja-omp", "omp",
+   "occa-cpu", "raja-cpu", "cpu"
+};
+
+} // namespace mfem::internal
+
+void Device::Configure(const std::string &device, const int dev)
+{
+   std::map<std::string, Backend::Id> bmap;
+   for (int i = 0; i < Backend::NUM_BACKENDS; i++)
+   {
+      bmap[internal::backend_name[i]] = internal::backend_list[i];
+   }
+   std::string::size_type beg = 0, end;
+   while (1)
+   {
+      end = device.find(',', beg);
+      end = (end != std::string::npos) ? end : device.size();
+      const std::string bname = device.substr(beg, end - beg);
+      std::map<std::string, Backend::Id>::iterator it = bmap.find(bname);
+      MFEM_VERIFY(it != bmap.end(), "invalid backend name: '" << bname << '\'');
+      Get().MarkBackend(it->second);
+      if (end == device.size()) { break; }
+      beg = end + 1;
+   }
+
+   // OCCA_CUDA needs CUDA or RAJA_CUDA:
+   Get().allowed_backends = Get().backends;
+   if (Allows(Backend::OCCA_CUDA) && !Allows(Backend::RAJA_CUDA))
+   {
+      Get().MarkBackend(Backend::CUDA);
+   }
+
+   // Activate all backends for Setup().
+   Get().allowed_backends = Get().backends;
+   Get().Setup(dev);
+
+   // Enable only the default host CPU backend.
+   Get().allowed_backends = Backend::CPU;
+}
+
+void Device::Print(std::ostream &out)
+{
+   out << "Device configuration: ";
+   bool add_comma = false;
+   for (int i = 0; i < Backend::NUM_BACKENDS; i++)
+   {
+      if (Get().backends & internal::backend_list[i])
+      {
+         if (add_comma) { out << ','; }
+         add_comma = true;
+         out << internal::backend_name[i];
+      }
+   }
+   out << '\n';
+}
+
+#ifdef MFEM_USE_CUDA
+static void DeviceSetup(const int dev, int &ngpu)
+{
+   cudaGetDeviceCount(&ngpu);
+   MFEM_VERIFY(ngpu>0, "No CUDA device found!");
+   cuInit(0);
+   cuDeviceGet(&internal::cuDevice, dev);
+   cuCtxCreate(&internal::cuContext, CU_CTX_SCHED_AUTO, internal::cuDevice);
+   internal::cuStream = new CUstream;
+   MFEM_VERIFY(internal::cuStream, "CUDA stream could not be created!");
+   cuStreamCreate(internal::cuStream, CU_STREAM_DEFAULT);
+}
+#endif
+
+static void CudaDeviceSetup(const int dev, int &ngpu)
+{
+#ifdef MFEM_USE_CUDA
+   DeviceSetup(dev, ngpu);
+#endif
+}
+
+static void RajaDeviceSetup(const int dev, int &ngpu)
+{
+#ifdef MFEM_USE_CUDA
+   if (ngpu <= 0) { DeviceSetup(dev, ngpu); }
+#endif
+}
+
+static void OccaDeviceSetup(CUdevice cu_dev, CUcontext cu_ctx)
+{
+#ifdef MFEM_USE_OCCA
+   const int cpu  = Device::Allows(Backend::OCCA_CPU);
+   const int omp  = Device::Allows(Backend::OCCA_OMP);
+   const int cuda = Device::Allows(Backend::OCCA_CUDA);
+   if (cpu + omp + cuda > 1)
+   {
+      MFEM_ABORT("Only one OCCA backend can be configured at a time!");
+   }
+   if (cuda)
+   {
+#if OCCA_CUDA_ENABLED
+      internal::occaDevice = occa::cuda::wrapDevice(cu_dev, cu_ctx);
+#else
+      MFEM_ABORT("the OCCA CUDA backend requires OCCA built with CUDA!");
+#endif
+   }
+   else if (omp)
+   {
+#if OCCA_OPENMP_ENABLED
+      internal::occaDevice.setup("mode: 'OpenMP'");
+#else
+      MFEM_ABORT("the OCCA OpenMP backend requires OCCA built with OpenMP!");
+#endif
+   }
+   else
+   {
+      internal::occaDevice.setup("mode: 'Serial'");
+   }
+
+   std::string mfemDir;
+   if (occa::io::exists(MFEM_INSTALL_DIR "/include/mfem/"))
+   {
+      mfemDir = MFEM_INSTALL_DIR "/include/mfem/";
+   }
+   else if (occa::io::exists(MFEM_SOURCE_DIR))
+   {
+      mfemDir = MFEM_SOURCE_DIR;
+   }
+   else
+   {
+      MFEM_ABORT("Cannot find OCCA kernels in MFEM_INSTALL_DIR or MFEM_SOURCE_DIR");
+   }
+
+   occa::io::addLibraryPath("mfem", mfemDir);
+   occa::loadKernels("mfem");
+#else
+   MFEM_ABORT("the OCCA backends require MFEM built with MFEM_USE_OCCA=YES");
+#endif
+}
+
+void Device::Setup(const int device)
+{
+   MFEM_VERIFY(ngpu == -1, "the mfem::Device is already configured!");
+
+   ngpu = 0;
+   dev = device;
+
+#ifndef MFEM_USE_CUDA
+   MFEM_VERIFY(!Allows(Backend::CUDA_MASK),
+               "the CUDA backends require MFEM built with MFEM_USE_CUDA=YES");
+#endif
+#ifndef MFEM_USE_RAJA
+   MFEM_VERIFY(!Allows(Backend::RAJA_MASK),
+               "the RAJA backends require MFEM built with MFEM_USE_RAJA=YES");
+#endif
+#ifndef MFEM_USE_OPENMP
+   MFEM_VERIFY(!Allows(Backend::OMP|Backend::RAJA_OMP),
+               "the OpenMP and RAJA OpenMP backends require MFEM built with"
+               " MFEM_USE_OPENMP=YES");
+#endif
+   // The check for MFEM_USE_OCCA is in the function OccaDeviceSetup().
+
+   // We initialize CUDA and/or RAJA_CUDA first so OccaDeviceSetup() can reuse
+   // the same initialized cuDevice and cuContext objects when OCCA_CUDA is
+   // enabled.
+   if (Allows(Backend::CUDA)) { CudaDeviceSetup(dev, ngpu); }
+   if (Allows(Backend::RAJA_CUDA)) { RajaDeviceSetup(dev, ngpu); }
+   if (Allows(Backend::OCCA_MASK))
+   {
+      OccaDeviceSetup(internal::cuDevice, internal::cuContext);
+   }
+}
+
+Device::~Device()
+{
+   delete internal::cuStream;
+}
+
+} // mfem

general/device.hpp

@@ -0,0 +1,190 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_DEVICE_HPP
+#define MFEM_DEVICE_HPP
+
+#include "globals.hpp"
+
+namespace mfem
+{
+
+/// MFEM backends.
+/** Individual backends will generally implement only a subset of the kernels
+    implemented by the default CPU backend. The goal of the backends is to
+    accelerate data-parallel portions of the code and they can use a device
+    memory space (e.g. GPUs) or share the memory space of the host (OpenMP). */
+struct Backend
+{
+   /** @brief In the documentation below, we use square brackets to indicate the
+       type of the backend: host or device. */
+   enum Id
+   {
+      /// [host] Default CPU backend: sequential execution on each MPI rank.
+      CPU = 1 << 0,
+      /// [host] OpenMP backend. Enabled when MFEM_USE_OPENMP = YES.
+      OMP = 1 << 1,
+      /// [device] CUDA backend. Enabled when MFEM_USE_CUDA = YES.
+      CUDA = 1 << 2,
+      /** @brief [host] RAJA CPU backend: sequential execution on each MPI rank.
+          Enabled when MFEM_USE_RAJA = YES. */
+      RAJA_CPU = 1 << 3,
+      /** @brief [host] RAJA OpenMP backend. Enabled when MFEM_USE_RAJA = YES
+          and MFEM_USE_OPENMP = YES. */
+      RAJA_OMP = 1 << 4,
+      /** @brief [device] RAJA CUDA backend. Enabled when MFEM_USE_RAJA = YES
+          and MFEM_USE_CUDA = YES. */
+      RAJA_CUDA = 1 << 5,
+      /** @brief [host] OCCA CPU backend: sequential execution on each MPI rank.
+          Enabled when MFEM_USE_OCCA = YES. */
+      OCCA_CPU = 1 << 6,
+      /// [host] OCCA OpenMP backend. Enabled when MFEM_USE_OCCA = YES.
+      OCCA_OMP = 1 << 7,
+      /** @brief [device] OCCA CUDA backend. Enabled when MFEM_USE_OCCA = YES
+          and MFEM_USE_CUDA = YES. */
+      OCCA_CUDA = 1 << 8
+   };
+
+   /** @brief Additional useful constants. For example, the *_MASK constants can
+       be used with Device::Allows(). */
+   enum
+   {
+      /// Number of backends: from (1 << 0) to (1 << (NUM_BACKENDS-1)).
+      NUM_BACKENDS = 9,
+      /// Biwise-OR of all CUDA backends
+      CUDA_MASK = CUDA | RAJA_CUDA | OCCA_CUDA,
+      /// Biwise-OR of all RAJA backends
+      RAJA_MASK = RAJA_CPU | RAJA_OMP | RAJA_CUDA,
+      /// Biwise-OR of all OCCA backends
+      OCCA_MASK = OCCA_CPU | OCCA_OMP | OCCA_CUDA,
+      /// Biwise-OR of all OpenMP backends
+      OMP_MASK = OMP | RAJA_OMP | OCCA_OMP,
+      /// Biwise-OR of all device backends
+      DEVICE_MASK = CUDA_MASK
+   };
+};
+
+
+/** @brief The MFEM Device class abstracts hardware devices, such as GPUs, as
+    well as programming models, such as CUDA, OCCA, RAJA and OpenMP. */
+/** This class represents a "virtual device" with the following properties:
+    - There a single object of this class which is controlled by its static
+      methods.
+    - Once configured, the object cannot be re-configured during the program
+      lifetime.
+    - MFEM classes use this object to determine where (host or device) to
+      perform an operation and which backend implementation to use.
+    - Multiple backends can be configured at the same time; currently, a fixed
+      priority order is used to select a specific backend from the list of
+      configured backends. See the Backend class and the Configure() method in
+      this class for details.
+    - The device can be disabled to restrict the backend selection to only the
+      default host CPU backend, see the methods Enable() and Disable(). */
+class Device
+{
+private:
+   enum MODES {SEQUENTIAL, ACCELERATED};
+
+   MODES mode;
+   int dev = 0; ///< Device ID of the configured device.
+   int ngpu = -1; ///< Number of detected devices; -1: not initialized.
+   unsigned long backends; ///< Bitwise-OR of all configured backends.
+   /** Bitwise-OR mask of all allowed backends. All backends are active when the
+       Device is enabled. When the Device is disabled, only the host CPU backend
+       is allowed. */
+   unsigned long allowed_backends;
+
+   Device()
+      : mode(Device::SEQUENTIAL),
+        backends(Backend::CPU),
+        allowed_backends(backends) { }
+   Device(Device const&);
+   void operator=(Device const&);
+   static Device& Get() { static Device singleton; return singleton; }
+
+   /// Setup switcher based on configuration settings
+   void Setup(const int dev = 0);
+
+   void MarkBackend(Backend::Id b) { backends |= b; }
+
+public:
+   /// Configure the Device backends.
+   /** The string parameter @a device must be a comma-separated list of backend
+       string names (see below). The @a dev argument specifies the ID of the
+       actual devices (e.g. GPU) to use.
+       * The available backends are described by the Backend class.
+       * The string name of a backend is the lowercase version of the
+         Backend::Id enumeration constant with '_' replaced by '-', e.g. the
+         string name of 'RAJA_CPU' is 'raja-cpu'.
+       * The 'cpu' backend is always enabled with lowest priority.
+       * The current backend priority from highest to lowest is: 'occa-cuda',
+         'raja-cuda', 'cuda', 'occa-omp', 'raja-omp', 'omp', 'occa-cpu',
+         'raja-cpu', 'cpu'.
+       * Multiple backends can be configured at the same time.
+       * Only one 'occa-*' backend can be configured at a time.
+       * The backend 'occa-cuda' enables the 'cuda' backend unless 'raja-cuda'
+         is already enabled.
+       * After this call, the Device will be disabled. */
+   static void Configure(const std::string &device, const int dev = 0);
+
+   /// Print the configuration of the MFEM virtual device object.
+   static void Print(std::ostream &out = mfem::out);
+
+   /// Return true if Configure() has been called previously.
+   static inline bool IsConfigured() { return Get().ngpu >= 0; }
+
+   /// Return true if an actual device (e.g. GPU) has been configured.
+   static inline bool IsAvailable() { return Get().ngpu > 0; }
+
+   /// Enable the use of the configured device in the code that follows.
+   /** After this call MFEM classes will use the backend kernels whenever
+       possible, transferring data automatically to the device, if necessary.
+
+       If the only configured backend is the default host CPU one, the device
+       will remain disabled. */
+   static inline void Enable()
+   {
+      if (Get().backends & ~Backend::CPU)
+      {
+         Get().mode = Device::ACCELERATED;
+         Get().allowed_backends = Get().backends;
+      }
+   }
+
+   /// Disable the use of the configured device in the code that follows.
+   /** After this call MFEM classes will only use default CPU kernels,
+       transferring data automatically from the device, if necessary. */
+   static inline void Disable()
+   {
+      Get().mode = Device::SEQUENTIAL;
+      Get().allowed_backends = Backend::CPU;
+   }
+
+   /// Return true if the Device is enabled.
+   static inline bool IsEnabled() { return Get().mode == ACCELERATED; }
+
+   /// The opposite of IsEnabled().
+   static inline bool IsDisabled() { return !IsEnabled(); }
+
+   /** @brief Return true if any of the backends in the backend mask, @a b_mask,
+       are allowed. The allowed backends are all configured backends minus the
+       device backends when the Device is disabled. */
+   /** This method can be used with any of the Backend::Id constants, the
+       Backend::*_MASK, or combinations of those. */
+   static inline bool Allows(unsigned long b_mask)
+   { return Get().allowed_backends & b_mask; }
+
+   ~Device();
+};
+
+} // mfem
+
+#endif // MFEM_DEVICE_HPP

general/forall.hpp

@@ -0,0 +1,142 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_FORALL_HPP
+#define MFEM_FORALL_HPP
+
+#include "../config/config.hpp"
+#include "error.hpp"
+#include "cuda.hpp"
+#include "occa.hpp"
+#include "device.hpp"
+#include "mem_manager.hpp"
+#include "../linalg/dtensor.hpp"
+
+#ifdef MFEM_USE_RAJA
+#include "RAJA/RAJA.hpp"
+#endif
+
+namespace mfem
+{
+
+// Implementation of MFEM's "parallel for" (forall) device/host kernel
+// interfaces supporting RAJA, CUDA, OpenMP, and sequential backends.
+
+// The MFEM_FORALL wrapper
+#define MFEM_FORALL(i,N,...)                                     \
+   ForallWrap(N,                                                 \
+              [=] MFEM_ATTR_DEVICE (int i) {__VA_ARGS__},        \
+              [&]                  (int i) {__VA_ARGS__})
+
+
+/// OpenMP backend
+template <typename HBODY>
+void OmpWrap(const int N, HBODY &&h_body)
+{
+#ifdef MFEM_USE_OPENMP
+   #pragma omp parallel for
+   for (int k = 0; k < N; k++)
+   {
+      h_body(k);
+   }
+#else
+   MFEM_ABORT("OpenMP requested for MFEM but OpenMP is not enabled!");
+#endif
+}
+
+
+/// RAJA Cuda backend
+template <int BLOCKS, typename DBODY>
+void RajaCudaWrap(const int N, DBODY &&d_body)
+{
+#if defined(MFEM_USE_RAJA) && defined(RAJA_ENABLE_CUDA)
+   RAJA::forall<RAJA::cuda_exec<BLOCKS>>(RAJA::RangeSegment(0,N),d_body);
+#else
+   MFEM_ABORT("RAJA::Cuda requested but RAJA::Cuda is not enabled!");
+#endif
+}
+
+
+/// RAJA OpenMP backend
+template <typename HBODY>
+void RajaOmpWrap(const int N, HBODY &&h_body)
+{
+#if defined(MFEM_USE_RAJA) && defined(RAJA_ENABLE_OPENMP)
+   RAJA::forall<RAJA::omp_parallel_for_exec>(RAJA::RangeSegment(0,N), h_body);
+#else
+   MFEM_ABORT("RAJA::OpenMP requested but RAJA::OpenMP is not enabled!");
+#endif
+}
+
+
+/// RAJA sequential loop backend
+template <typename HBODY>
+void RajaSeqWrap(const int N, HBODY &&h_body)
+{
+#ifdef MFEM_USE_RAJA
+   RAJA::forall<RAJA::loop_exec>(RAJA::RangeSegment(0,N), h_body);
+#else
+   MFEM_ABORT("RAJA requested but RAJA is not enabled!");
+#endif
+}
+
+
+/// CUDA backend
+#ifdef MFEM_USE_CUDA
+
+template <typename BODY> __global__ static
+void CuKernel(const int N, BODY body)
+{
+   const int k = blockDim.x*blockIdx.x + threadIdx.x;
+   if (k >= N) { return; }
+   body(k);
+}
+
+template <int BLOCKS, typename DBODY>
+void CuWrap(const int N, DBODY &&d_body)
+{
+   if (N==0) { return; }
+   const int GRID = (N+BLOCKS-1)/BLOCKS;
+   CuKernel<<<GRID,BLOCKS>>>(N,d_body);
+   const cudaError_t last = cudaGetLastError();
+   MFEM_VERIFY(last == cudaSuccess, cudaGetErrorString(last));
+}
+
+#else  // MFEM_USE_CUDA
+
+template <int BLOCKS, typename DBODY>
+void CuWrap(const int N, DBODY &&d_body) {}
+
+#endif
+
+
+/// The forall kernel body wrapper
+template <typename DBODY, typename HBODY>
+void ForallWrap(const int N, DBODY &&d_body, HBODY &&h_body)
+{
+   if (Device::Allows(Backend::RAJA_CUDA))
+   { return RajaCudaWrap<MFEM_CUDA_BLOCKS>(N, d_body); }
+
+   if (Device::Allows(Backend::CUDA))
+   { return CuWrap<MFEM_CUDA_BLOCKS>(N, d_body); }
+
+   if (Device::Allows(Backend::RAJA_OMP)) { return RajaOmpWrap(N, h_body); }
+
+   if (Device::Allows(Backend::OMP)) { return OmpWrap(N, h_body); }
+
+   if (Device::Allows(Backend::RAJA_CPU)) { return RajaSeqWrap(N, h_body); }
+
+   for (int k = 0; k < N; k++) { h_body(k); }
+}
+
+} // namespace mfem
+
+#endif // MFEM_FORALL_HPP

general/mem_manager.cpp

@@ -0,0 +1,364 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "../general/forall.hpp"
+
+#include <cstring> // std::memcpy
+
+#include <list>
+#include <unordered_map>
+
+namespace mfem
+{
+
+namespace internal
+{
+
+/// Forward declaration of the Alias structure
+struct Alias;
+
+/// Memory class that holds:
+///   - a boolean telling which memory space is being used
+///   - the size in bytes of this memory region,
+///   - the host and the device pointer,
+///   - a list of all aliases seen using this region (used only to free them).
+struct Memory
+{
+   bool host;
+   const std::size_t bytes;
+   void *const h_ptr;
+   void *d_ptr;
+   std::list<const void*> aliases;
+   Memory(void* const h, const std::size_t size):
+      host(true), bytes(size), h_ptr(h), d_ptr(nullptr), aliases() {}
+};
+
+/// Alias class that holds the base memory region and the offset
+struct Alias
+{
+   Memory *const mem;
+   const long offset;
+};
+
+typedef std::unordered_map<const void*, Memory> MemoryMap;
+typedef std::unordered_map<const void*, const Alias*> AliasMap;
+
+struct Ledger
+{
+   MemoryMap memories;
+   AliasMap aliases;
+};
+
+} // namespace mfem::internal
+
+static internal::Ledger *maps;
+
+MemoryManager::MemoryManager()
+{
+   exists = true;
+   enabled = true;
+   maps = new internal::Ledger();
+}
+
+MemoryManager::~MemoryManager()
+{
+   delete maps;
+   exists = false;
+}
+
+void* MemoryManager::Insert(void *ptr, const std::size_t bytes)
+{
+   if (!UsingMM()) { return ptr; }
+   const bool known = IsKnown(ptr);
+   if (known)
+   {
+      mfem_error("Trying to add an already present address!");
+   }
+   maps->memories.emplace(ptr, internal::Memory(ptr, bytes));
+   return ptr;
+}
+
+void *MemoryManager::Erase(void *ptr)
+{
+   if (!UsingMM()) { return ptr; }
+   if (!ptr) { return ptr; }
+   const bool known = IsKnown(ptr);
+   if (!known)
+   {
+      mfem_error("Trying to erase an unknown pointer!");
+   }
+   internal::Memory &mem = maps->memories.at(ptr);
+   if (mem.d_ptr) { CuMemFree(mem.d_ptr); }
+   for (const void *alias : mem.aliases)
+   {
+      maps->aliases.erase(maps->aliases.find(alias));
+   }
+   mem.aliases.clear();
+   maps->memories.erase(maps->memories.find(ptr));
+   return ptr;
+}
+
+void MemoryManager::SetHostDevicePtr(void *h_ptr, void *d_ptr, const bool host)
+{
+   internal::Memory &base = maps->memories.at(h_ptr);
+   base.d_ptr = d_ptr;
+   base.host = host;
+}
+
+bool MemoryManager::IsKnown(const void *ptr)
+{
+   return maps->memories.find(ptr) != maps->memories.end();
+}
+
+bool MemoryManager::IsOnHost(const void *ptr)
+{
+   return maps->memories.at(ptr).host;
+}
+
+std::size_t MemoryManager::Bytes(const void *ptr)
+{
+   return maps->memories.at(ptr).bytes;
+}
+
+void *MemoryManager::GetDevicePtr(const void *ptr)
+{
+   internal::Memory &base = maps->memories.at(ptr);
+   const size_t bytes = base.bytes;
+   if (!base.d_ptr)
+   {
+      CuMemAlloc(&base.d_ptr, bytes);
+      CuMemcpyHtoD(base.d_ptr, ptr, bytes);
+      base.host = false;
+   }
+   return base.d_ptr;
+}
+
+// Looks if ptr is an alias of one memory
+static const void* AliasBaseMemory(const internal::Ledger *maps,
+                                   const void *ptr)
+{
+   for (internal::MemoryMap::const_iterator mem = maps->memories.begin();
+        mem != maps->memories.end(); mem++)
+   {
+      const void *b_ptr = mem->first;
+      if (b_ptr > ptr) { continue; }
+      const void *end = static_cast<const char*>(b_ptr) + mem->second.bytes;
+      if (ptr < end) { return b_ptr; }
+   }
+   return nullptr;
+}
+
+bool MemoryManager::IsAlias(const void *ptr)
+{
+   const internal::AliasMap::const_iterator found = maps->aliases.find(ptr);
+   if (found != maps->aliases.end()) { return true; }
+   MFEM_ASSERT(!IsKnown(ptr), "Ptr is an already known address!");
+   const void *base = AliasBaseMemory(maps, ptr);
+   if (!base) { return false; }
+   internal::Memory &mem = maps->memories.at(base);
+   const long offset = static_cast<const char*>(ptr) -
+                       static_cast<const char*> (base);
+   const internal::Alias *alias = new internal::Alias{&mem, offset};
+   maps->aliases.emplace(ptr, alias);
+   mem.aliases.push_back(ptr);
+   return true;
+}
+
+static inline bool MmDeviceIniFilter(void)
+{
+   if (!mm.UsingMM()) { return true; }
+   if (!mm.IsEnabled()) { return true; }
+   if (!Device::IsAvailable()) { return true; }
+   if (!Device::IsConfigured()) { return true; }
+   return false;
+}
+
+// Turn a known address into the right host or device address. Alloc, Push, or
+// Pull it if necessary.
+static void *PtrKnown(internal::Ledger *maps, void *ptr)
+{
+   internal::Memory &base = maps->memories.at(ptr);
+   const bool ptr_on_host = base.host;
+   const std::size_t bytes = base.bytes;
+   const bool run_on_device = Device::Allows(Backend::DEVICE_MASK);
+   if (ptr_on_host && !run_on_device) { return ptr; }
+   if (bytes==0) { mfem_error("PtrKnown bytes==0"); }
+   if (!base.d_ptr) { CuMemAlloc(&base.d_ptr, bytes); }
+   if (!base.d_ptr) { mfem_error("PtrKnown !base->d_ptr"); }
+   if (!ptr_on_host && run_on_device) { return base.d_ptr; }
+   if (!ptr) { mfem_error("PtrKnown !ptr"); }
+   if (!ptr_on_host && !run_on_device) // Pull
+   {
+      CuMemcpyDtoH(ptr, base.d_ptr, bytes);
+      base.host = true;
+      return ptr;
+   }
+   // Push
+   if (!(ptr_on_host && run_on_device)) { mfem_error("PtrKnown !(host && gpu)"); }
+   CuMemcpyHtoD(base.d_ptr, ptr, bytes);
+   base.host = false;
+   return base.d_ptr;
+}
+
+// Turn an alias into the right host or device address. Alloc, Push, or Pull it
+// if necessary.
+static void *PtrAlias(internal::Ledger *maps, void *ptr)
+{
+   const bool gpu = Device::Allows(Backend::DEVICE_MASK);
+   const internal::Alias *alias = maps->aliases.at(ptr);
+   const internal::Memory *base = alias->mem;
+   const bool host = base->host;
+   const bool device = !base->host;
+   const std::size_t bytes = base->bytes;
+   if (host && !gpu) { return ptr; }
+   if (bytes==0) { mfem_error("PtrAlias bytes==0"); }
+   if (!base->d_ptr) { CuMemAlloc(&(alias->mem->d_ptr), bytes); }
+   if (!base->d_ptr) { mfem_error("PtrAlias !base->d_ptr"); }
+   void *a_ptr = static_cast<char*>(base->d_ptr) + alias->offset;
+   if (device && gpu) { return a_ptr; }
+   if (!base->h_ptr) { mfem_error("PtrAlias !base->h_ptr"); }
+   if (device && !gpu) // Pull
+   {
+      CuMemcpyDtoH(base->h_ptr, base->d_ptr, bytes);
+      alias->mem->host = true;
+      return ptr;
+   }
+   // Push
+   if (!(host && gpu)) { mfem_error("PtrAlias !(host && gpu)"); }
+   CuMemcpyHtoD(base->d_ptr, base->h_ptr, bytes);
+   alias->mem->host = false;
+   return a_ptr;
+}
+
+void *MemoryManager::Ptr(void *ptr)
+{
+   if (ptr==NULL) { return NULL; };
+   if (MmDeviceIniFilter()) { return ptr; }
+   if (IsKnown(ptr)) { return PtrKnown(maps, ptr); }
+   if (IsAlias(ptr)) { return PtrAlias(maps, ptr); }
+   if (Device::Allows(Backend::DEVICE_MASK))
+   {
+      mfem_error("Trying to use unknown pointer on the DEVICE!");
+   }
+   return ptr;
+}
+
+const void *MemoryManager::Ptr(const void *ptr)
+{
+   return static_cast<const void*>(Ptr(const_cast<void*>(ptr)));
+}
+
+static void PushKnown(internal::Ledger *maps,
+                      const void *ptr, const std::size_t bytes)
+{
+   internal::Memory &base = maps->memories.at(ptr);
+   if (!base.d_ptr) { CuMemAlloc(&base.d_ptr, base.bytes); }
+   CuMemcpyHtoD(base.d_ptr, ptr, bytes == 0 ? base.bytes : bytes);
+}
+
+static void PushAlias(const internal::Ledger *maps,
+                      const void *ptr, const std::size_t bytes)
+{
+   const internal::Alias *alias = maps->aliases.at(ptr);
+   void *dst = static_cast<char*>(alias->mem->d_ptr) + alias->offset;
+   CuMemcpyHtoD(dst, ptr, bytes);
+}
+
+void MemoryManager::Push(const void *ptr, const std::size_t bytes)
+{
+   if (MmDeviceIniFilter()) { return; }
+   if (IsKnown(ptr)) { return PushKnown(maps, ptr, bytes); }
+   if (IsAlias(ptr)) { return PushAlias(maps, ptr, bytes); }
+   if (Device::Allows(Backend::DEVICE_MASK))
+   { mfem_error("Unknown pointer to push to!"); }
+}
+
+static void PullKnown(const internal::Ledger *maps,
+                      const void *ptr, const std::size_t bytes)
+{
+   const internal::Memory &base = maps->memories.at(ptr);
+   const bool host = base.host;
+   if (host) { return; }
+   CuMemcpyDtoH(base.h_ptr, base.d_ptr, bytes == 0 ? base.bytes : bytes);
+}
+
+static void PullAlias(const internal::Ledger *maps,
+                      const void *ptr, const std::size_t bytes)
+{
+   const internal::Alias *alias = maps->aliases.at(ptr);
+   const bool host = alias->mem->host;
+   if (host) { return; }
+   if (!ptr) { mfem_error("PullAlias !ptr"); }
+   if (!alias->mem->d_ptr) { mfem_error("PullAlias !alias->mem->d_ptr"); }
+   CuMemcpyDtoH(const_cast<void*>(ptr),
+                static_cast<char*>(alias->mem->d_ptr) + alias->offset,
+                bytes);
+}
+
+void MemoryManager::Pull(const void *ptr, const std::size_t bytes)
+{
+   if (MmDeviceIniFilter()) { return; }
+   if (IsKnown(ptr)) { return PullKnown(maps, ptr, bytes); }
+   if (IsAlias(ptr)) { return PullAlias(maps, ptr, bytes); }
+   if (Device::Allows(Backend::DEVICE_MASK))
+   { mfem_error("Unknown pointer to pull from!"); }
+}
+
+namespace internal { extern CUstream *cuStream; }
+void* MemoryManager::Memcpy(void *dst, const void *src,
+                            const std::size_t bytes, const bool async)
+{
+   void *d_dst = Ptr(dst);
+   void *d_src = const_cast<void*>(Ptr(src));
+   if (bytes == 0) { return dst; }
+   const bool run_on_host = !Device::Allows(Backend::DEVICE_MASK);
+   if (run_on_host) { return std::memcpy(dst, src, bytes); }
+   if (!async) { return CuMemcpyDtoD(d_dst, d_src, bytes); }
+   return CuMemcpyDtoDAsync(d_dst, d_src, bytes, internal::cuStream);
+}
+
+void MemoryManager::RegisterCheck(void *ptr)
+{
+   if (ptr != NULL && UsingMM())
+   {
+      if (!IsKnown(ptr))
+      {
+         mfem_error("Pointer is not registered!");
+      }
+   }
+}
+
+void MemoryManager::PrintPtrs(void)
+{
+   for (const auto& n : maps->memories)
+   {
+      const internal::Memory &mem = n.second;
+      mfem::out << std::endl
+                << "key " << n.first << ", "
+                << "host " << mem.host << ", "
+                << "h_ptr " << mem.h_ptr << ", "
+                << "d_ptr " << mem.d_ptr;
+   }
+}
+
+void MemoryManager::GetAll(void)
+{
+   for (const auto& n : maps->memories)
+   {
+      const void *ptr = n.first;
+      Ptr(ptr);
+   }
+}
+
+MemoryManager mm;
+bool MemoryManager::exists = false;
+
+} // namespace mfem

general/mem_manager.hpp

@@ -0,0 +1,186 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_MEM_MANAGER_HPP
+#define MFEM_MEM_MANAGER_HPP
+
+#include "globals.hpp"
+
+namespace mfem
+{
+
+// Implementation of MFEM's lightweight device/host memory manager designed to
+// work seamlessly with the OCCA, RAJA, and other kernels supported by MFEM.
+
+/// The memory manager class
+class MemoryManager
+{
+private:
+   /// Allow to enable/disable the Ptr, Pull and Push functionalities
+   /// New and Delete will still continue to register the pointers
+   bool enabled;
+
+   /// Allow to detect if a global memory manager instance exists
+   static bool exists;
+
+public:
+   MemoryManager();
+   ~MemoryManager();
+
+   /// Adds an address in the map
+   void *Insert(void *ptr, const std::size_t bytes);
+
+   /// Remove the address from the map, as well as all its aliases
+   void *Erase(void *ptr);
+
+   /// Return true if the memory manager is used: pointers seen by mfem::New and
+   /// mfem::Delete will be inserted in the ledger and erased from it
+   static inline bool UsingMM()
+   {
+#ifdef MFEM_USE_MM
+      return true;
+#else
+      return false;
+#endif
+   }
+
+   /// Disable the memory manager: Ptr, Push and Pull will be no-op
+   void Disable() { enabled = false; }
+
+   /// Enable the memory manager: Ptr, Push and Pull wont be no-op
+   void Enable() { enabled = true; }
+
+   /// Return true if the memory manager is used and enabled
+   bool IsEnabled() { return UsingMM() && enabled; }
+
+   /// The opposite of IsEnabled().
+   bool IsDisabled() { return !IsEnabled(); }
+
+   /// Return true if a global memory manager instance exists
+   static bool Exists() { return exists; }
+
+   /** @brief Translates ptr to host or device address, depending on what
+       backends are currently allowed by the Device class and on the ptr
+       state. */
+   void *Ptr(void *ptr);
+   const void *Ptr(const void *ptr);
+
+   /// Data will be pushed/pulled before the copy happens on the H or the D
+   void* Memcpy(void *dst, const void *src,
+                std::size_t bytes, const bool async = false);
+
+   /// Return the bytes of the memory region which base address is ptr
+   std::size_t Bytes(const void *ptr);
+
+   /// Return true if the registered pointer is on the host side
+   bool IsOnHost(const void *ptr);
+
+   /// Return true if the pointer has been registered
+   bool IsKnown(const void *ptr);
+
+   /// Return true if the pointer is an alias inside a registered memory region
+   bool IsAlias(const void *ptr);
+
+   /// Push the data to the device
+   void Push(const void *ptr, const std::size_t bytes =0);
+
+   /// Pull the data from the device
+   void Pull(const void *ptr, const std::size_t bytes =0);
+
+   /// Return the corresponding device pointer of ptr, allocating and moving the
+   /// data if needed (used in OccaPtr)
+   void *GetDevicePtr(const void *ptr);
+
+   /// Registers external host pointer in the memory manager which will manage
+   /// the corresponding device pointer, but not the provided host pointer.
+   template<class T>
+   void RegisterHostPtr(T *ptr_host, const std::size_t size)
+   {
+      Insert(ptr_host, size*sizeof(T));
+#ifdef MFEM_DEBUG
+      RegisterCheck(ptr_host);
+#endif
+   }
+
+   /// Registers external host and device pointers in the memory manager.
+   template<class T>
+   void RegisterHostAndDevicePtr(T *ptr_host, T *ptr_device,
+                                 const std::size_t size, const bool host)
+   {
+      RegisterHostPtr(ptr_host, size);
+      SetHostDevicePtr(ptr_host, ptr_device, host);
+   }
+
+   /// Set the host h_ptr, device d_ptr and mode host of the memory region just
+   /// been registered with h_ptr (see RegisterHostAndDevicePtr)
+   void SetHostDevicePtr(void *h_ptr, void *d_ptr, const bool host);
+
+   /// Unregisters the host pointer from the memory manager. To be used with
+   /// memory not allocated by the memory manager.
+   template<class T>
+   void UnregisterHostPtr(T *ptr) { Erase(ptr); }
+
+   /// Check if pointer has been registered in the memory manager
+   void RegisterCheck(void *ptr);
+
+   /// Prints all pointers known by the memory manager
+   void PrintPtrs(void);
+
+   /// Copies all memory to the current memory space
+   void GetAll(void);
+};
+
+/// The (single) global memory manager object
+extern MemoryManager mm;
+
+/// Main memory allocation template function. Allocates n*size bytes and returns
+/// a pointer to the allocated memory.
+template<class T>
+inline T *New(const std::size_t n)
+{
+   T *ptr = new T[n];
+   if (!MemoryManager::Exists()) { return ptr; }
+   return static_cast<T*>(mm.Insert(ptr, n*sizeof(T)));
+}
+
+/// Frees the memory space pointed to by ptr, which must have been returned by a
+/// previous call to mfem::New.
+template<class T>
+inline void Delete(T *ptr)
+{
+   static_assert(!std::is_void<T>::value, "Cannot Delete a void pointer. "
+                 "Explicitly provide the correct type as a template parameter.");
+   if (!ptr) { return; }
+   delete [] ptr;
+   if (!MemoryManager::Exists()) { return; }
+   mm.Erase(ptr);
+}
+
+/// Return a host or device address corresponding to current memory space
+template <class T>
+inline T *Ptr(T *a) { return static_cast<T*>(mm.Ptr(a)); }
+
+/// Data will be pushed/pulled before the copy happens on the host or the device
+inline void* Memcpy(void *dst, const void *src,
+                    std::size_t bytes, const bool async = false)
+{ return mm.Memcpy(dst, src, bytes, async); }
+
+/// Push the data to the device
+inline void Push(const void *ptr, const std::size_t bytes = 0)
+{ return mm.Push(ptr, bytes); }
+
+/// Pull the data from the device
+inline void Pull(const void *ptr, const std::size_t bytes = 0)
+{ return mm.Pull(ptr, bytes); }
+
+} // namespace mfem
+
+#endif // MFEM_MEM_MANAGER_HPP

general/occa.cpp

@@ -0,0 +1,61 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "forall.hpp"
+
+namespace mfem
+{
+
+// This variable is defined in device.cpp:
+namespace internal { extern OccaDevice occaDevice; }
+
+static OccaMemory OccaWrapMemory(const OccaDevice dev, const void *d_adrs,
+                                 const size_t bytes)
+{
+   // This function is called when an OCCA kernel is going to be used.
+#ifdef MFEM_USE_OCCA
+   void *adrs = const_cast<void*>(d_adrs);
+#if defined(MFEM_USE_CUDA) && OCCA_CUDA_ENABLED
+   // If OCCA_CUDA is allowed, it will be used since it has the highest priority
+   if (Device::Allows(Backend::OCCA_CUDA))
+   {
+      return occa::cuda::wrapMemory(dev, adrs, bytes);
+   }
+#endif // MFEM_USE_CUDA && OCCA_CUDA_ENABLED
+   // otherwise, fallback to occa::cpu address space
+   return occa::cpu::wrapMemory(dev, adrs, bytes);
+#else // MFEM_USE_OCCA
+   return (void*)NULL;
+#endif
+}
+
+OccaMemory OccaPtr(const void *ptr)
+{
+   // This function is called when 'ptr' needs to be passed to an OCCA kernel.
+   OccaDevice dev = internal::occaDevice;
+   if (!mm.UsingMM()) { return OccaWrapMemory(dev, ptr, 0); }
+   const bool known = mm.IsKnown(ptr);
+   if (!known) { mfem_error("OccaPtr: Unknown address!"); }
+   const bool ptr_on_host = mm.IsOnHost(ptr);
+   const size_t bytes = mm.Bytes(ptr);
+   const bool run_on_host = !Device::Allows(Backend::DEVICE_MASK);
+   // If the priority of a host OCCA backend is higher than all device OCCA
+   // backends, then we will need to run-on-host even if the Device allows a
+   // device backend.
+   if (ptr_on_host && run_on_host) { return OccaWrapMemory(dev, ptr, bytes); }
+   if (run_on_host) { mfem_error("OccaPtr: !ptr_on_host && run_on_host"); }
+   void *d_ptr = mm.GetDevicePtr(ptr);
+   return OccaWrapMemory(dev, d_ptr, bytes);
+}
+
+OccaDevice OccaDev() { return internal::occaDevice; }
+
+} // namespace mfem

general/occa.hpp

@@ -0,0 +1,44 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_OCCA_HPP
+#define MFEM_OCCA_HPP
+
+#include "../config/config.hpp"
+#include "cuda.hpp" // for CUdevice, CUcontext
+
+#ifdef MFEM_USE_OCCA
+#include <occa.hpp>
+
+#if defined(MFEM_USE_CUDA) && OCCA_CUDA_ENABLED
+#include <occa/modes/cuda/utils.hpp>
+#endif
+
+typedef occa::device OccaDevice;
+typedef occa::memory OccaMemory;
+
+#else // MFEM_USE_OCCA
+
+typedef void* OccaDevice;
+typedef void* OccaMemory;
+
+#endif // MFEM_USE_OCCA
+
+namespace mfem
+{
+
+// Function called when the pointer 'a' needs to be passed to an OCCA kernel.
+OccaMemory OccaPtr(const void *a);
+OccaDevice OccaDev();
+
+} // namespace mfem
+
+#endif // MFEM_OCCA_HPP

general/table.cpp

@@ -15,6 +15,7 @@
 #include "table.hpp"
 #include "error.hpp"
 
+#include "../general/forall.hpp"
 #include <iostream>
 #include <iomanip>
 
@@ -29,8 +30,8 @@ Table::Table(const Table &table)
    if (size >= 0)
    {
       const int nnz = table.I[size];
-      I = new int[size+1];
-      J = new int[nnz];
+      I = mfem::New<int>(size+1);
+      J = mfem::New<int>(nnz);
       memcpy(I, table.I, sizeof(int)*(size+1));
       memcpy(J, table.J, sizeof(int)*nnz);
    }
@@ -55,8 +56,8 @@ Table::Table (int dim, int connections_per_row)
    int i, j, sum = dim * connections_per_row;
 
    size = dim;
-   I = new int[size+1];
-   J = new int[sum];
+   I = mfem::New<int>(size+1);
+   J = mfem::New<int>(sum);
 
    I[0] = 0;
    for (i = 1; i <= size; i++)
@@ -70,8 +71,8 @@ Table::Table (int nrows, int *partitioning)
 {
    size = nrows;
 
-   I = new int[size+1];
-   J = new int[size];
+   I = mfem::New<int>(size+1);
+   J = mfem::New<int>(size);
 
    for (int i = 0; i < size; i++)
    {
@@ -100,7 +101,8 @@ void Table::MakeJ()
       j = I[i], I[i] = k, k += j;
    }
 
-   J = new int[I[size]=k];
+   if (J) { mfem::Delete(J); }
+   J = mfem::New<int>(I[size]=k);
 }
 
 void Table::AddConnections (int r, const int *c, int nc)
@@ -147,14 +149,14 @@ void Table::SetDims(int rows, int nnz)
    if (size != rows)
    {
       size = rows;
-      if (I) { delete [] I; }
-      I = (rows >= 0) ? (new int[rows+1]) : (NULL);
+      if (I) { mfem::Delete(I); }
+      I = (rows >= 0) ? (mfem::New<int>(rows+1)) : (NULL);
    }
 
    if (j != nnz)
    {
-      if (J) { delete [] J; }
-      J = (nnz > 0) ? (new int[nnz]) : (NULL);
+      if (J) { mfem::Delete(J); }
+      J = (nnz > 0) ? (mfem::New<int>(nnz)) : (NULL);
    }
 
    if (size >= 0)
@@ -205,8 +207,8 @@ void Table::SortRows()
 
 void Table::SetIJ(int *newI, int *newJ, int newsize)
 {
-   delete [] I;
-   delete [] J;
+   mfem::Delete(I);
+   mfem::Delete(J);
    I = newI;
    J = newJ;
    if (newsize >= 0)
@@ -248,7 +250,7 @@ void Table::Finalize()
 
    if (sum != I[size])
    {
-      int *NewJ = new int[sum];
+      int *NewJ = mfem::New<int>(sum);
 
       for (i=0; i<size; i++)
       {
@@ -263,7 +265,7 @@ void Table::Finalize()
       }
       I[size] = sum;
 
-      delete [] J;
+      mfem::Delete(J);
 
       J = NewJ;
 
@@ -278,8 +280,8 @@ void Table::MakeFromList(int nrows, const Array<Connection> &list)
    size = nrows;
    int nnz = list.Size();
 
-   I = new int[size+1];
-   J = new int[nnz];
+   I = mfem::New<int>(size+1);
+   J = mfem::New<int>(nnz);
 
    for (int i = 0, k = 0; i <= size; i++)
    {
@@ -353,17 +355,17 @@ void Table::Save(std::ostream &out) const
 
 void Table::Load(std::istream &in)
 {
-   delete [] I;
-   delete [] J;
+   mfem::Delete(I);
+   mfem::Delete(J);
 
    in >> size;
-   I = new int[size+1];
+   I = mfem::New<int>(size+1);
    for (int i = 0; i <= size; i++)
    {
       in >> I[i];
    }
    int nnz = I[size];
-   J = new int[nnz];
+   J =mfem::New<int>(nnz);
    for (int j = 0; j < nnz; j++)
    {
       in >> J[j];
@@ -372,8 +374,8 @@ void Table::Load(std::istream &in)
 
 void Table::Clear()
 {
-   delete [] I;
-   delete [] J;
+   mfem::Delete(I);
+   mfem::Delete(J);
    size = -1;
    I = J = NULL;
 }
@@ -382,8 +384,8 @@ void Table::Copy(Table & copy) const
 {
    if (size >= 0)
    {
-      int * i_copy = new int[size+1];
-      int * j_copy = new int[I[size]];
+      int * i_copy = mfem::New<int>(size+1);
+      int * j_copy = mfem::New<int>(I[size]);
 
       memcpy(i_copy, I, sizeof(int)*(size+1));
       memcpy(j_copy, J, sizeof(int)*I[size]);
@@ -411,8 +413,8 @@ long Table::MemoryUsage() const
 
 Table::~Table ()
 {
-   if (I) { delete [] I; }
-   if (J) { delete [] J; }
+   if (I) { mfem::Delete(I); }
+   if (J) { mfem::Delete(J); }
 }
 
 void Transpose (const Table &A, Table &At, int _ncols_A)

general/version.cpp

@@ -94,8 +94,8 @@ const char *GetConfigStr()
 #ifdef MFEM_THREAD_SAFE
       "MFEM_THREAD_SAFE\n"
 #endif
-#ifdef MFEM_USE_OPENMP
-      "MFEM_USE_OPENMP\n"
+#ifdef MFEM_USE_LEGACY_OPENMP
+      "MFEM_USE_LEGACY_OPENMP\n"
 #endif
 #ifdef MFEM_USE_MEMALLOC
       "MFEM_USE_MEMALLOC\n"
@@ -135,6 +135,18 @@ const char *GetConfigStr()
 #endif
 #ifdef MFEM_USE_SIDRE
       "MFEM_USE_SIDRE\n"
+#endif
+#ifdef MFEM_USE_PUMI
+      "MFEM_USE_PUMI\n"
+#endif
+#ifdef MFEM_USE_CUDA
+      "MFEM_USE_CUDA\n"
+#endif
+#ifdef MFEM_USE_RAJA
+      "MFEM_USE_RAJA\n"
+#endif
+#ifdef MFEM_USE_OCCA
+      "MFEM_USE_OCCA\n"
 #endif
       "MFEM_TIMER_TYPE = " EXPAND_AND_QUOTE(MFEM_TIMER_TYPE)
       ;

linalg/CMakeLists.txt

@@ -31,6 +31,7 @@ list(APPEND HDRS
   blockvector.hpp
   complex_operator.hpp
   densemat.hpp
+  dtensor.hpp
   handle.hpp
   invariants.hpp
   linalg.hpp

linalg/blockmatrix.cpp

@@ -455,9 +455,9 @@ SparseMatrix * BlockMatrix::CreateMonolithic() const
 {
    int nnz = NumNonZeroElems();
 
-   int * i_amono = new int[ row_offsets[nRowBlocks]+2 ];
-   int * j_amono = new int[ nnz ];
-   double * data = new double[ nnz ];
+   int * i_amono = mfem::New<int>(row_offsets[nRowBlocks]+2);
+   int * j_amono = mfem::New<int>(nnz);
+   double * data = mfem::New<double>(nnz);
 
    for (int i = 0; i < row_offsets[nRowBlocks]+2; i++)
    {

linalg/blockvector.hpp

@@ -85,7 +85,7 @@ public:
    //! Get the i-th vector in the block
    void GetBlockView(int i, Vector & blockView);
 
-   int BlockSize(int i) { return blockOffsets[i+1] - blockOffsets[i];}
+   int BlockSize(int i) { return blockOffsets[i+1] - blockOffsets[i]; }
 
    //! Update method
    /**

linalg/complex_operator.cpp

@@ -10,6 +10,8 @@
 // Software Foundation) version 2.1 dated February 1999.
 
 #include "complex_operator.hpp"
+#include <set>
+#include <map>
 
 namespace mfem
 {
@@ -39,6 +41,30 @@ ComplexOperator::~ComplexOperator()
    delete v_;
 }
 
+Operator & ComplexOperator::real()
+{
+   MFEM_ASSERT(Op_Real_, "ComplexOperator has no real part!");
+   return *Op_Real_;
+}
+
+Operator & ComplexOperator::imag()
+{
+   MFEM_ASSERT(Op_Imag_, "ComplexOperator has no imaginary part!");
+   return *Op_Imag_;
+}
+
+const Operator & ComplexOperator::real() const
+{
+   MFEM_ASSERT(Op_Real_, "ComplexOperator has no real part!");
+   return *Op_Real_;
+}
+
+const Operator & ComplexOperator::imag() const
+{
+   MFEM_ASSERT(Op_Imag_, "ComplexOperator has no imaginary part!");
+   return *Op_Imag_;
+}
+
 void ComplexOperator::Mult(const Vector &x, Vector &y) const
 {
    double * x_data = x.GetData();
@@ -120,6 +146,30 @@ void ComplexOperator::MultTranspose(const Vector &x_r, const Vector &x_i,
 }
 
 
+SparseMatrix & ComplexSparseMatrix::real()
+{
+   MFEM_ASSERT(Op_Real_, "ComplexSparseMatrix has no real part!");
+   return dynamic_cast<SparseMatrix &>(*Op_Real_);
+}
+
+SparseMatrix & ComplexSparseMatrix::imag()
+{
+   MFEM_ASSERT(Op_Imag_, "ComplexSparseMatrix has no imaginary part!");
+   return dynamic_cast<SparseMatrix &>(*Op_Imag_);
+}
+
+const SparseMatrix & ComplexSparseMatrix::real() const
+{
+   MFEM_ASSERT(Op_Real_, "ComplexSparseMatrix has no real part!");
+   return dynamic_cast<const SparseMatrix &>(*Op_Real_);
+}
+
+const SparseMatrix & ComplexSparseMatrix::imag() const
+{
+   MFEM_ASSERT(Op_Imag_, "ComplexSparseMatrix has no imaginary part!");
+   return dynamic_cast<const SparseMatrix &>(*Op_Imag_);
+}
+
 SparseMatrix * ComplexSparseMatrix::GetSystemMatrix() const
 {
    SparseMatrix * A_r = dynamic_cast<SparseMatrix*>(Op_Real_);
@@ -184,4 +234,358 @@ SparseMatrix * ComplexSparseMatrix::GetSystemMatrix() const
    return new SparseMatrix(I, J, D, this->Height(), this->Width());
 }
 
+#ifdef MFEM_USE_MPI
+
+ComplexHypreParMatrix::ComplexHypreParMatrix(HypreParMatrix * A_Real,
+                                             HypreParMatrix * A_Imag,
+                                             bool ownReal, bool ownImag,
+                                             Convention convention)
+   : ComplexOperator(A_Real, A_Imag, ownReal, ownImag, convention)
+{
+   comm_ = (A_Real) ? A_Real->GetComm() :
+           ((A_Imag) ? A_Imag->GetComm() : MPI_COMM_WORLD);
+
+   MPI_Comm_rank(comm_, &myid_);
+   MPI_Comm_size(comm_, &nranks_);
+}
+
+HypreParMatrix & ComplexHypreParMatrix::real()
+{
+   MFEM_ASSERT(Op_Real_, "ComplexHypreParMatrix has no real part!");
+   return dynamic_cast<HypreParMatrix &>(*Op_Real_);
+}
+
+HypreParMatrix & ComplexHypreParMatrix::imag()
+{
+   MFEM_ASSERT(Op_Imag_, "ComplexHypreParMatrix has no imaginary part!");
+   return dynamic_cast<HypreParMatrix &>(*Op_Imag_);
+}
+
+const HypreParMatrix & ComplexHypreParMatrix::real() const
+{
+   MFEM_ASSERT(Op_Real_, "ComplexHypreParMatrix has no real part!");
+   return dynamic_cast<const HypreParMatrix &>(*Op_Real_);
+}
+
+const HypreParMatrix & ComplexHypreParMatrix::imag() const
+{
+   MFEM_ASSERT(Op_Imag_, "ComplexHypreParMatrix has no imaginary part!");
+   return dynamic_cast<const HypreParMatrix &>(*Op_Imag_);
+}
+
+HypreParMatrix * ComplexHypreParMatrix::GetSystemMatrix() const
+{
+   HypreParMatrix * A_r = dynamic_cast<HypreParMatrix*>(Op_Real_);
+   HypreParMatrix * A_i = dynamic_cast<HypreParMatrix*>(Op_Imag_);
+
+   if ( A_r == NULL && A_i == NULL ) { return NULL; }
+
+   HYPRE_Int global_num_rows_r = (A_r) ? A_r->GetGlobalNumRows() : 0;
+   HYPRE_Int global_num_rows_i = (A_i) ? A_i->GetGlobalNumRows() : 0;
+   HYPRE_Int global_num_rows = std::max(global_num_rows_r, global_num_rows_i);
+
+   HYPRE_Int global_num_cols_r = (A_r) ? A_r->GetGlobalNumCols() : 0;
+   HYPRE_Int global_num_cols_i = (A_i) ? A_i->GetGlobalNumCols() : 0;
+   HYPRE_Int global_num_cols = std::max(global_num_cols_r, global_num_cols_i);
+
+   int row_starts_size = (HYPRE_AssumedPartitionCheck()) ? 2 : nranks_ + 1;
+   HYPRE_Int * row_starts = hypre_CTAlloc(HYPRE_Int, row_starts_size);
+   HYPRE_Int * col_starts = hypre_CTAlloc(HYPRE_Int, row_starts_size);
+
+   const HYPRE_Int * row_starts_z = (A_r) ? A_r->RowPart() :
+                                    ((A_i) ? A_i->RowPart() : NULL);
+   const HYPRE_Int * col_starts_z = (A_r) ? A_r->ColPart() :
+                                    ((A_i) ? A_i->ColPart() : NULL);
+
+   for (int i = 0; i < row_starts_size; i++)
+   {
+      row_starts[i] = 2 * row_starts_z[i];
+      col_starts[i] = 2 * col_starts_z[i];
+   }
+
+   SparseMatrix diag_r, diag_i, offd_r, offd_i;
+   HYPRE_Int * cmap_r, * cmap_i;
+
+   int nrows_r = 0, nrows_i = 0, ncols_r = 0, ncols_i = 0;
+   int ncols_offd_r = 0, ncols_offd_i = 0;
+   if (A_r)
+   {
+      A_r->GetDiag(diag_r);
+      A_r->GetOffd(offd_r, cmap_r);
+      nrows_r = diag_r.Height();
+      ncols_r = diag_r.Width();
+      ncols_offd_r = offd_r.Width();
+   }
+   if (A_i)
+   {
+      A_i->GetDiag(diag_i);
+      A_i->GetOffd(offd_i, cmap_i);
+      nrows_i = diag_i.Height();
+      ncols_i = diag_i.Width();
+      ncols_offd_i = offd_i.Width();
+   }
+   int nrows = std::max(nrows_r, nrows_i);
+   int ncols = std::max(ncols_r, ncols_i);
+
+   // Determine the unique set of off-diagonal columns global indices
+   std::set<int> cset;
+   for (int i=0; i<ncols_offd_r; i++)
+   {
+      cset.insert(cmap_r[i]);
+   }
+   for (int i=0; i<ncols_offd_i; i++)
+   {
+      cset.insert(cmap_i[i]);
+   }
+   int num_cols_offd = (int)cset.size();
+
+   // Exatract pointers to the various CSR arrays of the diagonal blocks
+   const int * diag_r_I = (A_r) ? diag_r.GetI() : NULL;
+   const int * diag_i_I = (A_i) ? diag_i.GetI() : NULL;
+
+   const int * diag_r_J = (A_r) ? diag_r.GetJ() : NULL;
+   const int * diag_i_J = (A_i) ? diag_i.GetJ() : NULL;
+
+   const double * diag_r_D = (A_r) ? diag_r.GetData() : NULL;
+   const double * diag_i_D = (A_i) ? diag_i.GetData() : NULL;
+
+   int diag_r_nnz = (diag_r_I) ? diag_r_I[nrows] : 0;
+   int diag_i_nnz = (diag_i_I) ? diag_i_I[nrows] : 0;
+   int diag_nnz = 2 * (diag_r_nnz + diag_i_nnz);
+
+   // Exatract pointers to the various CSR arrays of the off-diagonal blocks
+   const int * offd_r_I = (A_r) ? offd_r.GetI() : NULL;
+   const int * offd_i_I = (A_i) ? offd_i.GetI() : NULL;
+
+   const int * offd_r_J = (A_r) ? offd_r.GetJ() : NULL;
+   const int * offd_i_J = (A_i) ? offd_i.GetJ() : NULL;
+
+   const double * offd_r_D = (A_r) ? offd_r.GetData() : NULL;
+   const double * offd_i_D = (A_i) ? offd_i.GetData() : NULL;
+
+   int offd_r_nnz = (offd_r_I) ? offd_r_I[nrows] : 0;
+   int offd_i_nnz = (offd_i_I) ? offd_i_I[nrows] : 0;
+   int offd_nnz = 2 * (offd_r_nnz + offd_i_nnz);
+
+   // Allocate CSR arrays for the combined matrix
+   HYPRE_Int * diag_I = hypre_CTAlloc(HYPRE_Int, 2 * nrows + 1);
+   HYPRE_Int * diag_J = hypre_CTAlloc(HYPRE_Int, diag_nnz);
+   double    * diag_D = hypre_CTAlloc(double, diag_nnz);
+
+   HYPRE_Int * offd_I = hypre_CTAlloc(HYPRE_Int, 2 * nrows + 1);
+   HYPRE_Int * offd_J = hypre_CTAlloc(HYPRE_Int, offd_nnz);
+   double    * offd_D = hypre_CTAlloc(double, offd_nnz);
+   HYPRE_Int * cmap   = hypre_CTAlloc(HYPRE_Int, 2 * num_cols_offd);
+
+   // Fill the CSR arrays for the diagonal portion of the matrix
+   const double factor = (convention_ == HERMITIAN) ? 1.0 : -1.0;
+
+   diag_I[0] = 0;
+   diag_I[nrows] = diag_r_nnz + diag_i_nnz;
+   for (int i=0; i<nrows; i++)
+   {
+      diag_I[i + 1]         = ((diag_r_I)?diag_r_I[i+1]:0) +
+                              ((diag_i_I)?diag_i_I[i+1]:0);
+      diag_I[i + nrows + 1] = diag_I[i+1] + diag_r_nnz + diag_i_nnz;
+
+      if (diag_r_I)
+      {
+         for (int j=0; j<diag_r_I[i+1] - diag_r_I[i]; j++)
+         {
+            diag_J[diag_I[i] + j] = diag_r_J[diag_r_I[i] + j];
+            diag_D[diag_I[i] + j] = diag_r_D[diag_r_I[i] + j];
+
+            diag_J[diag_I[i+nrows] + j] =
+               diag_r_J[diag_r_I[i] + j] + ncols;
+            diag_D[diag_I[i+nrows] + j] =
+               factor * diag_r_D[diag_r_I[i] + j];
+         }
+      }
+      if (diag_i_I)
+      {
+         const int off_r = (diag_r_I)?(diag_r_I[i+1] - diag_r_I[i]):0;
+         for (int j=0; j<diag_i_I[i+1] - diag_i_I[i]; j++)
+         {
+            diag_J[diag_I[i] + off_r + j] =  diag_i_J[diag_i_I[i] + j] + ncols;
+            diag_D[diag_I[i] + off_r + j] = -diag_i_D[diag_i_I[i] + j];
+
+            diag_J[diag_I[i+nrows] + off_r + j] = diag_i_J[diag_i_I[i] + j];
+            diag_D[diag_I[i+nrows] + off_r + j] =
+               factor * diag_i_D[diag_i_I[i] + j];
+         }
+      }
+   }
+
+   // Determine the mappings describing the layout of off-diagonal columns
+   int num_recv_procs = 0;
+   HYPRE_Int * offd_col_start_stop = NULL;
+   this->getColStartStop(A_r, A_i, num_recv_procs, offd_col_start_stop);
+
+   std::set<int>::iterator sit;
+   std::map<int,int> cmapa, cmapb, cinvmap;
+   for (sit=cset.begin(); sit!=cset.end(); sit++)
+   {
+      int col_orig = *sit;
+      int col_2x2  = -1;
+      int col_size = 0;
+      for (int i=0; i<num_recv_procs; i++)
+      {
+         if (offd_col_start_stop[2*i] <= col_orig &&
+             col_orig < offd_col_start_stop[2*i+1])
+         {
+            col_2x2 = offd_col_start_stop[2*i] + col_orig;
+            col_size = offd_col_start_stop[2*i+1] - offd_col_start_stop[2*i];
+            break;
+         }
+      }
+      cmapa[*sit] = col_2x2;
+      cmapb[*sit] = col_2x2 + col_size;
+      cinvmap[col_2x2] = -1;
+      cinvmap[col_2x2 + col_size] = -1;
+   }
+   delete [] offd_col_start_stop;
+
+   std::map<int, int>::iterator mit;
+   int i = 0;
+   for (mit=cinvmap.begin(); mit!=cinvmap.end(); mit++, i++)
+   {
+      mit->second = i;
+      cmap[i] = mit->first;
+   }
+
+   // Fill the CSR arrays for the off-diagonal portion of the matrix
+   offd_I[0] = 0;
+   offd_I[nrows] = offd_r_nnz + offd_i_nnz;
+   for (int i=0; i<nrows; i++)
+   {
+      offd_I[i + 1]         = ((offd_r_I)?offd_r_I[i+1]:0) +
+                              ((offd_i_I)?offd_i_I[i+1]:0);
+      offd_I[i + nrows + 1] = offd_I[i+1] + offd_r_nnz + offd_i_nnz;
+
+      if (offd_r_I)
+      {
+         const int off_i = (offd_i_I)?(offd_i_I[i+1] - offd_i_I[i]):0;
+         for (int j=0; j<offd_r_I[i+1] - offd_r_I[i]; j++)
+         {
+            offd_J[offd_I[i] + j] =
+               cinvmap[cmapa[cmap_r[offd_r_J[offd_r_I[i] + j]]]];
+            offd_D[offd_I[i] + j] = offd_r_D[offd_r_I[i] + j];
+
+            offd_J[offd_I[i+nrows] + off_i + j] =
+               cinvmap[cmapb[cmap_r[offd_r_J[offd_r_I[i] + j]]]];
+            offd_D[offd_I[i+nrows] + off_i + j] =
+               factor * offd_r_D[offd_r_I[i] + j];
+         }
+      }
+      if (offd_i_I)
+      {
+         const int off_r = (offd_r_I)?(offd_r_I[i+1] - offd_r_I[i]):0;
+         for (int j=0; j<offd_i_I[i+1] - offd_i_I[i]; j++)
+         {
+            offd_J[offd_I[i] + off_r + j] =
+               cinvmap[cmapb[cmap_i[offd_i_J[offd_i_I[i] + j]]]];
+            offd_D[offd_I[i] + off_r + j] = -offd_i_D[offd_i_I[i] + j];
+
+            offd_J[offd_I[i+nrows] + j] =
+               cinvmap[cmapa[cmap_i[offd_i_J[offd_i_I[i] + j]]]];
+            offd_D[offd_I[i+nrows] + j] = factor * offd_i_D[offd_i_I[i] + j];
+         }
+      }
+   }
+
+   // Construct the combined matrix
+   HypreParMatrix * A = new HypreParMatrix(comm_,
+                                           2 * global_num_rows,
+                                           2 * global_num_cols,
+                                           row_starts, col_starts,
+                                           diag_I, diag_J, diag_D,
+                                           offd_I, offd_J, offd_D,
+                                           2 * num_cols_offd, cmap);
+
+   // Give the new matrix ownership of its interanl arrays
+   A->SetOwnerFlags(-1,-1,-1);
+   hypre_CSRMatrixSetDataOwner(((hypre_ParCSRMatrix*)(*A))->diag,1);
+   hypre_CSRMatrixSetDataOwner(((hypre_ParCSRMatrix*)(*A))->offd,1);
+   hypre_ParCSRMatrixSetRowStartsOwner((hypre_ParCSRMatrix*)(*A),1);
+   hypre_ParCSRMatrixSetColStartsOwner((hypre_ParCSRMatrix*)(*A),1);
+
+   return A;
+}
+
+void
+ComplexHypreParMatrix::getColStartStop(const HypreParMatrix * A_r,
+                                       const HypreParMatrix * A_i,
+                                       int & num_recv_procs,
+                                       HYPRE_Int *& offd_col_start_stop) const
+{
+   hypre_ParCSRCommPkg * comm_pkg_r =
+      (A_r) ? hypre_ParCSRMatrixCommPkg((hypre_ParCSRMatrix*)(*A_r)) : NULL;
+   hypre_ParCSRCommPkg * comm_pkg_i =
+      (A_i) ? hypre_ParCSRMatrixCommPkg((hypre_ParCSRMatrix*)(*A_i)) : NULL;
+
+   std::set<HYPRE_Int> send_procs, recv_procs;
+   if ( comm_pkg_r )
+   {
+      for (HYPRE_Int i=0; i<comm_pkg_r->num_sends; i++)
+      {
+         send_procs.insert(comm_pkg_r->send_procs[i]);
+      }
+      for (HYPRE_Int i=0; i<comm_pkg_r->num_recvs; i++)
+      {
+         recv_procs.insert(comm_pkg_r->recv_procs[i]);
+      }
+   }
+   if ( comm_pkg_i )
+   {
+      for (HYPRE_Int i=0; i<comm_pkg_i->num_sends; i++)
+      {
+         send_procs.insert(comm_pkg_i->send_procs[i]);
+      }
+      for (HYPRE_Int i=0; i<comm_pkg_i->num_recvs; i++)
+      {
+         recv_procs.insert(comm_pkg_i->recv_procs[i]);
+      }
+   }
+
+   num_recv_procs = (int)recv_procs.size();
+
+   HYPRE_Int loc_start_stop[2];
+   offd_col_start_stop = new HYPRE_Int[2 * num_recv_procs];
+
+   const HYPRE_Int * row_part = (A_r) ? A_r->RowPart() :
+                                ((A_i) ? A_i->RowPart() : NULL);
+
+   int row_part_ind = (HYPRE_AssumedPartitionCheck()) ? 0 : myid_;
+   loc_start_stop[0] = row_part[row_part_ind];
+   loc_start_stop[1] = row_part[row_part_ind+1];
+
+   MPI_Request * req = new MPI_Request[send_procs.size()+recv_procs.size()];
+   MPI_Status * stat = new MPI_Status[send_procs.size()+recv_procs.size()];
+   int send_count = 0;
+   int recv_count = 0;
+   int tag = 0;
+
+   std::set<HYPRE_Int>::iterator sit;
+   for (sit=send_procs.begin(); sit!=send_procs.end(); sit++)
+   {
+      MPI_Isend(loc_start_stop, 2, HYPRE_MPI_INT,
+                *sit, tag, comm_, &req[send_count]);
+      send_count++;
+   }
+   for (sit=recv_procs.begin(); sit!=recv_procs.end(); sit++)
+   {
+      MPI_Irecv(&offd_col_start_stop[2*recv_count], 2, HYPRE_MPI_INT,
+                *sit, tag, comm_, &req[send_count+recv_count]);
+      recv_count++;
+   }
+
+   MPI_Waitall(send_count+recv_count, req, stat);
+
+   delete [] req;
+   delete [] stat;
+}
+
+
+#endif // MFEM_USE_MPI
+
 }

linalg/complex_operator.hpp

@@ -14,6 +14,9 @@
 
 #include "operator.hpp"
 #include "sparsemat.hpp"
+#ifdef MFEM_USE_MPI
+#include "hypre.hpp"
+#endif
 
 namespace mfem
 {
@@ -27,7 +30,8 @@ namespace mfem
     ComplexOperator allows one to choose a convention upon construction, which
     facilitates symmetry.
 
-    Matrix-vector products are then computed as:
+    If we let (y_r + i y_i) = (Op_r + i Op_i)(x_r + i x_i) then Matrix-vector
+    products are then computed as:
 
     1. When Convention::HERMITIAN is used (default)
     / y_r \   / Op_r -Op_i \ / x_r \
@@ -38,6 +42,8 @@ namespace mfem
     / y_r \   / Op_r -Op_i \ / x_r \
     |     | = |            | |     |
     \-y_i /   \-Op_i -Op_r / \ x_i /
+    In other words, Matrix-vector products with Convention::BLOCK_SYMMETRIC
+    compute the complex conjugate of Op*x.
 
     Either convention can be used with a given complex operator,
     however, each of them is best suited for certain classes of
@@ -82,9 +88,30 @@ public:
 
    virtual ~ComplexOperator();
 
+   /** @brief Check for existence of real or imaginary part of the operator
+
+       These methods do not check that the operators are non-zero but
+       only that the operators have been set.
+    */
+   bool hasRealPart() const { return Op_Real_ != NULL; }
+   bool hasImagPart() const { return Op_Imag_ != NULL; }
+
+   /** @brief Real or imaginary part accessor methods
+
+       The following accessor methods should only be called if the
+       requested part of the opertor is known to exist.  This
+       can be checked with hasRealPart() or hasImagPart().
+   */
+   virtual Operator & real();
+   virtual Operator & imag();
+   virtual const Operator & real() const;
+   virtual const Operator & imag() const;
+
    virtual void Mult(const Vector &x, Vector &y) const;
    virtual void MultTranspose(const Vector &x, Vector &y) const;
 
+   virtual Type GetType() const { return Complex_Operator; }
+
 protected:
    // Let this be hidden from the public interface since the implementation
    // depends on internal members
@@ -127,9 +154,74 @@ public:
       : ComplexOperator(A_Real, A_Imag, ownReal, ownImag, convention)
    {}
 
+   virtual SparseMatrix & real();
+   virtual SparseMatrix & imag();
+
+   virtual const SparseMatrix & real() const;
+   virtual const SparseMatrix & imag() const;
+
+   /** Combine the blocks making up this complex operator into a
+       single SparseMatrix.  The resulting matrix can be passed to
+       solvers which require access to the matrix entries themselves,
+       such as sparse direct solvers, rather than simply the action of
+       the opertor.  Note that this combined operator requires roughly
+       twice the memory of the block structured operator. */
    SparseMatrix * GetSystemMatrix() const;
+
+   virtual Type GetType() const { return MFEM_ComplexSparseMat; }
 };
 
+#ifdef MFEM_USE_MPI
+
+/** @brief Specialization of the ComplexOperator built from a pair of
+    HypreParMatrices.
+
+    The purpose of this specialization is to construct a single
+    HypreParMatrix object which is equivalent to the 2x2 block system
+    that the ComplexOperator mimics. The resulting HypreParMatrix can
+    then be passed along to solvers which require access to the CSR
+    matrix data such as SuperLU, STRUMPACK, or similar sparse linear
+    solvers.
+
+    See ComplexOperator documentation in operator.hpp for more information.
+ */
+class ComplexHypreParMatrix : public ComplexOperator
+{
+public:
+   ComplexHypreParMatrix(HypreParMatrix * A_Real, HypreParMatrix * A_Imag,
+                         bool ownReal, bool ownImag,
+                         Convention convention = HERMITIAN);
+
+   virtual HypreParMatrix & real();
+   virtual HypreParMatrix & imag();
+
+   virtual const HypreParMatrix & real() const;
+   virtual const HypreParMatrix & imag() const;
+
+   /** Combine the blocks making up this complex operator into a
+       single HypreParMatrix.  The resulting matrix can be passed to
+       solvers which require access to the matrix entries themselves,
+       such as sparse direct solvers or Hypre preconditioners, rather
+       than simply the action of the opertor.  Note that this combined
+       operator requires roughly twice the memory of the block
+       structured operator. */
+   HypreParMatrix * GetSystemMatrix() const;
+
+   virtual Type GetType() const { return Complex_Hypre_ParCSR; }
+
+private:
+   void getColStartStop(const HypreParMatrix * A_r,
+                        const HypreParMatrix * A_i,
+                        int & num_recv_procs,
+                        HYPRE_Int *& offd_col_start_stop) const;
+
+   MPI_Comm comm_;
+   int myid_;
+   int nranks_;
+};
+
+#endif // MFEM_USE_MPI
+
 }
 
-#endif
+#endif // MFEM_COMPLEX_OPERATOR

linalg/densemat.cpp

@@ -16,6 +16,8 @@
 #include "vector.hpp"
 #include "matrix.hpp"
 #include "densemat.hpp"
+#include "dtensor.hpp"
+#include "../general/forall.hpp"
 #include "../general/table.hpp"
 #include "../general/globals.hpp"
 
@@ -76,9 +78,9 @@ DenseMatrix::DenseMatrix(const DenseMatrix &m) : Matrix(m.height, m.width)
    if (hw > 0)
    {
       MFEM_ASSERT(m.data, "invalid source matrix");
-      data = new double[hw];
+      data = mfem::New<double>(hw);
       capacity = hw;
-      std::memcpy(data, m.data, sizeof(double)*hw);
+      mfem::Memcpy(data, m.data, sizeof(double)*hw);
    }
    else
    {
@@ -87,13 +89,20 @@ DenseMatrix::DenseMatrix(const DenseMatrix &m) : Matrix(m.height, m.width)
    }
 }
 
+static void Set(const double d, const int size, double *data)
+{
+   DeviceVector d_data(data);
+   MFEM_FORALL(i, size, d_data[i] = d;);
+}
+
 DenseMatrix::DenseMatrix(int s) : Matrix(s)
 {
    MFEM_ASSERT(s >= 0, "invalid DenseMatrix size: " << s);
    capacity = s*s;
    if (capacity > 0)
    {
-      data = new double[capacity](); // init with zeroes
+      data = mfem::New<double>(capacity);
+      mfem::Set(0.0, capacity, data);
    }
    else
    {
@@ -108,7 +117,8 @@ DenseMatrix::DenseMatrix(int m, int n) : Matrix(m, n)
    capacity = m*n;
    if (capacity > 0)
    {
-      data = new double[capacity](); // init with zeroes
+      data = mfem::New<double>(capacity);
+      mfem::Set(0.0, capacity, data);
    }
    else
    {
@@ -116,19 +126,28 @@ DenseMatrix::DenseMatrix(int m, int n) : Matrix(m, n)
    }
 }
 
+static void Transpose(const int height, const int width,
+                      double *data, const double *mdata)
+{
+   DeviceVector d_data(data);
+   const DeviceVector d_mdata(mdata);
+   MFEM_FORALL(i, height,
+   {
+      for (int j = 0; j < width; j++)
+      {
+         d_data[i+j*height] = d_mdata[j+i*height];
+      }
+   });
+}
+
 DenseMatrix::DenseMatrix(const DenseMatrix &mat, char ch)
    : Matrix(mat.width, mat.height)
 {
    capacity = height*width;
    if (capacity > 0)
    {
-      data = new double[capacity];
-
-      for (int i = 0; i < height; i++)
-         for (int j = 0; j < width; j++)
-         {
-            (*this)(i,j) = mat(j,i);
-         }
+      data = mfem::New<double>(capacity);
+      mfem::Transpose(height, width, data, mat.Data());
    }
    else
    {
@@ -136,6 +155,13 @@ DenseMatrix::DenseMatrix(const DenseMatrix &mat, char ch)
    }
 }
 
+DenseMatrix::DenseMatrix(double *d, int h, int w)
+   : Matrix(h, w)
+{
+   data = d;
+   capacity = -h*w;
+}
+
 void DenseMatrix::SetSize(int h, int w)
 {
    MFEM_ASSERT(h >= 0 && w >= 0,
@@ -151,10 +177,11 @@ void DenseMatrix::SetSize(int h, int w)
    {
       if (capacity > 0)
       {
-         delete [] data;
+         mfem::Delete(data);
       }
       capacity = hw;
-      data = new double[hw](); // init with zeroes
+      data = mfem::New<double>(capacity);
+      mfem::Set(0.0, capacity, data);
    }
 }
 
@@ -168,32 +195,31 @@ const double &DenseMatrix::Elem(int i, int j) const
    return (*this)(i,j);
 }
 
+static void Mult(const int height, const int width,
+                 const double *data, const double *x, double *y)
+{
+   const DeviceVector d_data(data);
+   const DeviceVector d_x(x);
+   DeviceVector d_y(y);
+   MFEM_FORALL(i, height,
+   {
+      double sum = 0.0;
+      for (int j = 0; j < width; j++)
+      {
+         sum += d_x[j]*d_data[i+j*height];
+      }
+      d_y[i] = sum;
+   });
+}
+
 void DenseMatrix::Mult(const double *x, double *y) const
 {
    if (width == 0)
    {
-      for (int row = 0; row < height; row++)
-      {
-         y[row] = 0.0;
-      }
+      mfem::Set(0.0, height, y);
       return;
    }
-   double *d_col = data;
-   double x_col = x[0];
-   for (int row = 0; row < height; row++)
-   {
-      y[row] = x_col*d_col[row];
-   }
-   d_col += height;
-   for (int col = 1; col < width; col++)
-   {
-      x_col = x[col];
-      for (int row = 0; row < height; row++)
-      {
-         y[row] += x_col*d_col[row];
-      }
-      d_col += height;
-   }
+   mfem::Mult(height, width, data, x, y);
 }
 
 void DenseMatrix::Mult(const Vector &x, Vector &y) const
@@ -564,11 +590,8 @@ void DenseMatrix::Add(const double c, const DenseMatrix &A)
 
 DenseMatrix &DenseMatrix::operator=(double c)
 {
-   int s = Height()*Width();
-   for (int i = 0; i < s; i++)
-   {
-      data[i] = c;
-   }
+   const int s = Height()*Width();
+   mfem::Set(c, s, data);
    return *this;
 }
 
@@ -587,11 +610,9 @@ DenseMatrix &DenseMatrix::operator=(const DenseMatrix &m)
    SetSize(m.height, m.width);
 
    const int hw = height * width;
-   for (int i = 0; i < hw; i++)
-   {
-      data[i] = m.data[i];
-   }
-
+   const DeviceVector d_m(m.GetData());
+   DeviceVector d_data(data);
+   MFEM_FORALL(i, hw, d_data[i] = d_m[i];);
    return *this;
 }
 
@@ -2442,16 +2463,11 @@ void DenseMatrix::GetRowSums(Vector &l) const
 void DenseMatrix::Diag(double c, int n)
 {
    SetSize(n);
+   const int N = n*n;
+   DeviceVector d_data(data);
+   MFEM_FORALL(i, N, d_data[i] = 0.0;);
+   MFEM_FORALL(i, n, d_data[i*(n+1)] = c;);
 
-   int i, N = n*n;
-   for (i = 0; i < N; i++)
-   {
-      data[i] = 0.0;
-   }
-   for (i = 0; i < n; i++)
-   {
-      data[i*(n+1)] = c;
-   }
 }
 
 void DenseMatrix::Diag(double *diag, int n)
@@ -2594,7 +2610,6 @@ void DenseMatrix::GradToCurl(DenseMatrix &curl)
 
 void DenseMatrix::GradToDiv(Vector &div)
 {
-
 #ifdef MFEM_DEBUG
    if (Width()*Height() != div.Size())
    {
@@ -2604,13 +2619,11 @@ void DenseMatrix::GradToDiv(Vector &div)
 
    // div(dof*j+i) <-- (*this)(i,j)
 
-   int n = height * width;
+   const int n = height * width;
    double *ddata = div.GetData();
-
-   for (int i = 0; i < n; i++)
-   {
-      ddata[i] = data[i];
-   }
+   const DeviceVector d_data(data);
+   DeviceVector d_ddata(ddata);
+   MFEM_FORALL(i, n, d_ddata[i] = d_data[i];);
 }
 
 void DenseMatrix::CopyRows(const DenseMatrix &A, int row1, int row2)
@@ -3004,7 +3017,7 @@ DenseMatrix::~DenseMatrix()
 {
    if (capacity > 0)
    {
-      delete [] data;
+      mfem::Delete(data);
    }
 }
 
@@ -3061,20 +3074,20 @@ void Mult(const DenseMatrix &b, const DenseMatrix &c, DenseMatrix &a)
    double *ad = a.Data();
    const double *bd = b.Data();
    const double *cd = c.Data();
-   for (int i = 0; i < ah*aw; i++)
-   {
-      ad[i] = 0.0;
-   }
-   for (int j = 0; j < aw; j++)
+   const DeviceVector d_bd(bd);
+   const DeviceVector d_cd(cd);
+   DeviceVector d_ad(ad);
+   MFEM_FORALL(i, ah*aw, d_ad[i] = 0.0;);
+   MFEM_FORALL(j, aw,
    {
       for (int k = 0; k < bw; k++)
       {
          for (int i = 0; i < ah; i++)
          {
-            ad[i+j*ah] += bd[i+k*ah] * cd[k+j*bw];
+            d_ad[i+j*ah] += d_bd[i+k*ah] * d_cd[k+j*bw];
          }
       }
-   }
+   });
 #endif
 }
 
@@ -3368,16 +3381,22 @@ void CalcOrtho(const DenseMatrix &J, Vector &n)
 
 void MultAAt(const DenseMatrix &a, DenseMatrix &aat)
 {
-   for (int i = 0; i < a.Height(); i++)
-      for (int j = 0; j <= i; j++)
+   const int height = a.Height();
+   const int width = a.Width();
+   const DeviceVector d_a(a.GetData());
+   DeviceVector d_aat(aat.GetData());
+   MFEM_FORALL(i, height,
+   {
+      for (int j=0; j<=i; j++)
       {
-         double temp = 0.;
-         for (int k = 0; k < a.Width(); k++)
+         double temp = 0.0;
+         for (int k=0; k<width; k++)
          {
-            temp += a(i,k) * a(j,k);
+            temp += d_a[i+k*height] * d_a[j+k*height];
          }
-         aat(j,i) = aat(i,j) = temp;
+         d_aat[j+i*height] = d_aat[i+j*height] = temp;
       }
+   });
 }
 
 void AddMultADAt(const DenseMatrix &A, const Vector &D, DenseMatrix &ADAt)
@@ -3903,18 +3922,20 @@ void AddMult_a_VVt(const double a, const Vector &v, DenseMatrix &VVt)
       mfem_error("AddMult_a_VVt(...)");
    }
 #endif
-
-   for (int i = 0; i < n; i++)
+   const int height = VVt.Height();
+   const DeviceVector d_v(v.GetData());
+   DeviceVector d_VVt(VVt.GetData());
+   MFEM_FORALL(i, n,
    {
-      double avi = a * v(i);
+      double avi = a * d_v[i];
       for (int j = 0; j < i; j++)
       {
-         double avivj = avi * v(j);
-         VVt(i, j) += avivj;
-         VVt(j, i) += avivj;
+         double avivj = avi * d_v[j];
+         d_VVt[i+j*height] += avivj;
+         d_VVt[j+i*height] += avivj;
       }
-      VVt(i, i) += avi * v(i);
-   }
+      d_VVt[i+i*height] += avi * d_v[i];
+   });
 }
 
 
@@ -3926,47 +3947,49 @@ void LUFactors::Factor(int m)
    MFEM_VERIFY(!info, "LAPACK: error in DGETRF");
 #else
    // compiling without LAPACK
-   double *data = this->data;
-   for (int i = 0; i < m; i++)
+   DeviceArray d_ipiv(ipiv);
+   DeviceVector d_data(data);
+   MFEM_FORALL(i, m,
    {
       // pivoting
       {
          int piv = i;
-         double a = std::abs(data[piv+i*m]);
+         double a = fabs(d_data[piv+i*m]);
          for (int j = i+1; j < m; j++)
          {
-            const double b = std::abs(data[j+i*m]);
+            const double b = fabs(d_data[j+i*m]);
             if (b > a)
             {
                a = b;
                piv = j;
             }
          }
-         ipiv[i] = piv;
+         d_ipiv[i] = piv;
          if (piv != i)
          {
             // swap rows i and piv in both L and U parts
             for (int j = 0; j < m; j++)
             {
-               Swap<double>(data[i+j*m], data[piv+j*m]);
+               const double tmp = d_data[i+j*m];
+               d_data[i+j*m] = d_data[piv+j*m];
+               d_data[piv+j*m] = tmp;
             }
          }
       }
-      MFEM_ASSERT(data[i+i*m] != 0.0, "division by zero");
-      const double a_ii_inv = 1.0/data[i+i*m];
+      const double a_ii_inv = 1.0/d_data[i+i*m];
       for (int j = i+1; j < m; j++)
       {
-         data[j+i*m] *= a_ii_inv;
+         d_data[j+i*m] *= a_ii_inv;
       }
       for (int k = i+1; k < m; k++)
       {
-         const double a_ik = data[i+k*m];
+         const double a_ik = d_data[i+k*m];
          for (int j = i+1; j < m; j++)
          {
-            data[j+k*m] -= a_ik * data[j+i*m];
+            d_data[j+k*m] -= a_ik * d_data[j+i*m];
          }
       }
-   }
+   });
 #endif
 }
 
@@ -4028,24 +4051,29 @@ void LUFactors::LSolve(int m, int n, double *X) const
    const double *data = this->data;
    const int *ipiv = this->ipiv;
    double *x = X;
-   for (int k = 0; k < n; k++)
+   const DeviceVector d_data(data);
+   const DeviceArray d_ipiv(ipiv);
+   DeviceVector d_x(x);
+   MFEM_FORALL(k, n,
    {
+      double *d_mx = &d_x[k*m];
       // X <- P X
       for (int i = 0; i < m; i++)
       {
-         Swap<double>(x[i], x[ipiv[i]-ipiv_base]);
+         const double tmp = d_mx[i];
+         d_mx[i] = d_mx[d_ipiv[i]];
+         d_mx[d_ipiv[i]] = tmp;
       }
       // X <- L^{-1} X
       for (int j = 0; j < m; j++)
       {
-         const double x_j = x[j];
+         const double d_mx_j = d_mx[j];
          for (int i = j+1; i < m; i++)
          {
-            x[i] -= data[i+j*m] * x_j;
+            d_mx[i] -= d_data[i+j*m] * d_mx_j;
          }
       }
-      x += m;
-   }
+   });
 }
 
 void LUFactors::USolve(int m, int n, double *X) const
@@ -4053,18 +4081,20 @@ void LUFactors::USolve(int m, int n, double *X) const
    const double *data = this->data;
    double *x = X;
    // X <- U^{-1} X
-   for (int k = 0; k < n; k++)
+   const DeviceVector d_data(data);
+   DeviceVector d_x(x);
+   MFEM_FORALL(k, n,
    {
+      double *d_mx = &d_x[k*m];
       for (int j = m-1; j >= 0; j--)
       {
-         const double x_j = ( x[j] /= data[j+j*m] );
+         const double x_j = ( d_mx[j] /= d_data[j+j*m] );
          for (int i = 0; i < j; i++)
          {
-            x[i] -= data[i+j*m] * x_j;
+            d_mx[i] -= d_data[i+j*m] * x_j;
          }
       }
-      x += m;
-   }
+   });
 }
 
 void LUFactors::Solve(int m, int n, double *X) const
@@ -4214,8 +4244,8 @@ DenseMatrixInverse::DenseMatrixInverse(const DenseMatrix &mat)
 {
    MFEM_ASSERT(height == width, "not a square matrix");
    a = &mat;
-   lu.data = new double[width*width];
-   lu.ipiv = new int[width];
+   lu.data = mfem::New<double>(width*width);
+   lu.ipiv = mfem::New<int>(width);
    Factor();
 }
 
@@ -4224,31 +4254,40 @@ DenseMatrixInverse::DenseMatrixInverse(const DenseMatrix *mat)
 {
    MFEM_ASSERT(height == width, "not a square matrix");
    a = mat;
-   lu.data = new double[width*width];
-   lu.ipiv = new int[width];
+   lu.data = mfem::New<double>(width*width);
+   lu.ipiv = mfem::New<int>(width);
 }
 
 void DenseMatrixInverse::Factor()
 {
    MFEM_ASSERT(a, "DenseMatrix is not given");
    const double *adata = a->data;
-   for (int i = 0, s = width*width; i < s; i++)
+   const int N = width*width;
+   const DeviceVector d_data(adata);
+   DeviceVector d_ludata(lu.data);
+   MFEM_FORALL(i, N,
    {
-      lu.data[i] = adata[i];
-   }
+      d_ludata[i] = d_data[i];
+   });
    lu.Factor(width);
 }
 
+void DenseMatrixInverse::GetInverseMatrix(DenseMatrix &Ainv) const
+{
+   Ainv.SetSize(width);
+   lu.GetInverseMatrix(width, Ainv.Data());
+}
+
 void DenseMatrixInverse::Factor(const DenseMatrix &mat)
 {
    MFEM_VERIFY(mat.height == mat.width, "DenseMatrix is not square!");
    if (width != mat.width)
    {
       height = width = mat.width;
-      delete [] lu.data;
-      lu.data = new double[width*width];
-      delete [] lu.ipiv;
-      lu.ipiv = new int[width];
+      mfem::Delete(lu.data);
+      lu.data = mfem::New<double>(width*width);
+      mfem::Delete(lu.ipiv);
+      lu.ipiv = mfem::New<int>(width);
    }
    a = &mat;
    Factor();
@@ -4286,8 +4325,8 @@ void DenseMatrixInverse::TestInversion()
 
 DenseMatrixInverse::~DenseMatrixInverse()
 {
-   delete [] lu.data;
-   delete [] lu.ipiv;
+   mfem::Delete(lu.data);
+   mfem::Delete(lu.ipiv);
 }
 
 

linalg/densemat.hpp

@@ -56,8 +56,7 @@ public:
    /** Construct a DenseMatrix using existing data array. The DenseMatrix does
        not assume ownership of the data array, i.e. it will not delete the
        array. */
-   DenseMatrix(double *d, int h, int w) : Matrix(h, w)
-   { data = d; capacity = -h*w; }
+   DenseMatrix(double *d, int h, int w);
 
    /// Change the data array and the size of the DenseMatrix.
    /** The DenseMatrix does not assume ownership of the data array, i.e. it will
@@ -71,7 +70,7 @@ public:
        not delete the new array @a d. This method will delete the current data
        array, if owned. */
    void Reset(double *d, int h, int w)
-   { if (OwnsData()) { delete [] data; } UseExternalData(d, h, w); }
+   { if (OwnsData()) { mfem::Delete(data); } UseExternalData(d, h, w); }
 
    /** Clear the data array and the dimensions of the DenseMatrix. This method
        should not be used with DenseMatrix that owns its current data array. */
@@ -79,7 +78,7 @@ public:
 
    /// Delete the matrix data array (if owned) and reset the matrix state.
    void Clear()
-   { if (OwnsData()) { delete [] data; } ClearExternalData(); }
+   { if (OwnsData()) { mfem::Delete(data); } ClearExternalData(); }
 
    /// For backward compatibility define Size to be synonym of Width()
    int Size() const { return Width(); }
@@ -580,12 +579,11 @@ public:
    /// Multiply the inverse matrix by another matrix: X = A^{-1} B.
    void Mult(const DenseMatrix &B, DenseMatrix &X) const;
 
+   /// Multiply the inverse matrix by another matrix: X <- A^{-1} X.
+   void Mult(DenseMatrix &X) const { lu.Solve(width, X.Width(), X.Data()); }
+
    /// Compute and return the inverse matrix in Ainv.
-   void GetInverseMatrix(DenseMatrix &Ainv) const
-   {
-      Ainv.SetSize(width);
-      lu.GetInverseMatrix(width, Ainv.Data());
-   }
+   void GetInverseMatrix(DenseMatrix &Ainv) const;
 
    /// Compute the determinant of the original DenseMatrix using the LU factors.
    double Det() const { return lu.Det(width); }
@@ -674,7 +672,7 @@ public:
       : Mk(NULL, i, j)
    {
       nk = k;
-      tdata = new double[i*j*k];
+      tdata = mfem::New<double>(i*j*k);
       own_data = true;
    }
 
@@ -685,8 +683,8 @@ public:
       const int size = Mk.Height()*Mk.Width()*nk;
       if (size > 0)
       {
-         tdata = new double[size];
-         std::memcpy(tdata, other.tdata, sizeof(double) * size);
+         tdata = mfem::New<double>(size);
+         mfem::Memcpy(tdata, other.tdata, sizeof(double) * size);
       }
       else
       {
@@ -700,16 +698,16 @@ public:
 
    void SetSize(int i, int j, int k)
    {
-      if (own_data) { delete [] tdata; }
+      if (own_data) { mfem::Delete(tdata); }
       Mk.UseExternalData(NULL, i, j);
       nk = k;
-      tdata = new double[i*j*k];
+      tdata = mfem::New<double>(i*j*k);
       own_data = true;
    }
 
    void UseExternalData(double *ext_data, int i, int j, int k)
    {
-      if (own_data) { delete [] tdata; }
+      if (own_data) { mfem::Delete(tdata); }
       Mk.UseExternalData(NULL, i, j);
       nk = k;
       tdata = ext_data;
@@ -732,6 +730,8 @@ public:
 
    double *Data() { return tdata; }
 
+   const double *Data() const { return tdata; }
+
    /** Matrix-vector product from unassembled element matrices, assuming both
        'x' and 'y' use the same elem_dof table. */
    void AddMult(const Table &elem_dof, const Vector &x, Vector &y) const;
@@ -743,7 +743,7 @@ public:
 
    ~DenseTensor()
    {
-      if (own_data) { delete [] tdata; }
+      if (own_data) { mfem::Delete(tdata); }
    }
 };
 

linalg/dtensor.hpp

@@ -0,0 +1,155 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_DTENSOR
+#define MFEM_DTENSOR
+
+#include "../general/cuda.hpp"
+#include "../general/mem_manager.hpp"
+
+namespace mfem
+{
+
+/// A Class to compute the real index from the multi-indices of a tensor
+template <int N, int Dim, typename T, typename... Args>
+class TensorInd
+{
+public:
+   MFEM_ATTR_HOST_DEVICE
+   static inline int result(const int* sizes, T first, Args... args)
+   {
+#ifndef MFEM_USE_CUDA
+      MFEM_ASSERT(first<sizes[N-1],"Trying to access out of boundary.");
+#endif
+      return first + sizes[N - 1] * TensorInd < N + 1, Dim, Args... >
+             ::result(sizes, args...);
+   }
+};
+
+// Terminal case
+template <int Dim, typename T, typename... Args>
+class TensorInd<Dim, Dim, T, Args...>
+{
+public:
+   MFEM_ATTR_HOST_DEVICE
+   static inline int result(const int* sizes, T first, Args... args)
+   {
+#ifndef MFEM_USE_CUDA
+      MFEM_ASSERT(first<sizes[Dim-1],"Trying to access out of boundary.");
+#endif
+      return first;
+   }
+};
+
+
+/// A class to initialize the size of a Tensor
+template <int N, int Dim, typename T, typename... Args>
+class Init
+{
+public:
+   static inline int result(int* sizes, T first, Args... args)
+   {
+      sizes[N - 1] = first;
+      return first * Init < N + 1, Dim, Args... >::result(sizes, args...);
+   }
+};
+
+// Terminal case
+template <int Dim, typename T, typename... Args>
+class Init<Dim, Dim, T, Args...>
+{
+public:
+   static inline int result(int* sizes, T first, Args... args)
+   {
+      sizes[Dim - 1] = first;
+      return first;
+   }
+};
+
+
+/// A basic generic Tensor class, appropriate for use on the GPU
+template<int Dim, typename Scalar = double>
+class DeviceTensor
+{
+protected:
+   int capacity;
+   Scalar *data;
+   int sizes[Dim];
+
+public:
+   /// Default constructor
+   DeviceTensor() = delete;
+
+   /// Constructor to initialize a tensor from the Scalar array _data
+   template <typename... Args>
+   DeviceTensor(Scalar* _data, Args... args)
+   {
+      static_assert(sizeof...(args) == Dim, "Wrong number of arguments");
+      // Initialize sizes, and compute the number of values
+      const long int nb = Init<1, Dim, Args...>::result(sizes, args...);
+      capacity = nb;
+      data = (capacity > 0) ? mfem::Ptr(_data) : NULL;
+   }
+
+   /// Constructor to initialize a tensor from the Scalar array _data
+   DeviceTensor(const Scalar* _data)
+   {
+      data = (Scalar*) mfem::Ptr(_data);
+   }
+
+   /// Constructor to initialize a tensor from the Scalar array _data
+   DeviceTensor(Scalar* _data) { data = mfem::Ptr(_data); }
+
+   /// Constructor to initialize a tensor from the const Scalar array _data
+   template <typename... Args>
+   DeviceTensor(const Scalar* _data, Args... args)
+   {
+      static_assert(sizeof...(args) == Dim, "Wrong number of arguments");
+      // Initialize sizes, and compute the number of values
+      const long int nb = Init<1, Dim, Args...>::result(sizes, args...);
+      capacity = nb;
+      data = (capacity > 0) ? (Scalar*)mfem::Ptr(_data) : NULL;
+   }
+
+   /// Copy constructor
+   MFEM_ATTR_HOST_DEVICE DeviceTensor(const DeviceTensor& t)
+   {
+      for (int i = 0; i < Dim; ++i)
+      {
+         sizes[i] = t.sizes[i];
+      }
+      data = t.data;
+   }
+
+   /// Conversion to `Scalar *`.
+   inline operator Scalar *() const { return data; }
+
+   /// Const accessor for the data
+   template <typename... Args> MFEM_ATTR_HOST_DEVICE inline
+   Scalar& operator()(Args... args) const
+   {
+      static_assert(sizeof...(args) == Dim, "Wrong number of arguments");
+      return data[ TensorInd<1, Dim, Args...>::result(sizes, args...) ];
+   }
+
+   MFEM_ATTR_HOST_DEVICE inline Scalar& operator[](int i) const
+   {
+      return data[i];
+   }
+};
+
+typedef DeviceTensor<1,int> DeviceArray;
+typedef DeviceTensor<1,double> DeviceVector;
+typedef DeviceTensor<2,double> DeviceMatrix;
+
+} // mfem namespace
+
+#endif // MFEM_DTENSOR

linalg/handle.cpp

@@ -78,7 +78,8 @@ void OperatorHandle::MakeSquareBlockDiag(MPI_Comm comm, HYPRE_Int glob_size,
       case Operator::PETSC_MATAIJ:
       case Operator::PETSC_MATIS:
          // Assuming that PetscInt is the same size as HYPRE_Int, checked above.
-         oper = new PetscParMatrix(comm, glob_size, row_starts, diag, type_id);
+         oper = new PetscParMatrix(comm, glob_size, (PetscInt*)row_starts, diag,
+                                   type_id);
          break;
 #endif
       default: MFEM_ABORT(not_supported_msg << type_id);
@@ -110,7 +111,7 @@ MakeRectangularBlockDiag(MPI_Comm comm, HYPRE_Int glob_num_rows,
       case Operator::PETSC_MATIS:
          // Assuming that PetscInt is the same size as HYPRE_Int, checked above.
          oper = new PetscParMatrix(comm, glob_num_rows, glob_num_cols,
-                                   row_starts, col_starts, diag, type_id);
+                                   (PetscInt*)row_starts, (PetscInt*)col_starts, diag, type_id);
          break;
 #endif
       default: MFEM_ABORT(not_supported_msg << type_id);

linalg/handle.hpp

@@ -81,6 +81,12 @@ public:
    /// Access the underlying Operator pointer.
    Operator *Ptr() const { return oper; }
 
+   /// Support the use of -> to call methods of the underlying Operator.
+   Operator *operator->() const { return oper; }
+
+   /// Access the underlying Operator.
+   Operator &operator*() { return *oper; }
+
    /// Get the currently set operator type id.
    Operator::Type Type() const { return type_id; }
 
@@ -187,6 +193,10 @@ public:
                     const Vector &X, Vector &B) const;
 };
 
+
+/// Add an alternative name for OperatorHandle -- OperatorPtr.
+typedef OperatorHandle OperatorPtr;
+
 } // namespace mfem
 
 #endif

linalg/hypre.cpp

@@ -79,7 +79,7 @@ template<typename TargetT, typename SourceT>
 static TargetT *DuplicateAs(const SourceT *array, int size,
                             bool cplusplus = true)
 {
-   TargetT *target_array = cplusplus ? new TargetT[size]
+   TargetT *target_array = cplusplus ? mfem::New<TargetT>(size)
                            /*     */ : mfem_hypre_TAlloc(TargetT, size);
    for (int i = 0; i < size; i++)
    {
@@ -641,13 +641,13 @@ HypreParMatrix::HypreParMatrix(MPI_Comm comm, int id, int np,
 
    HYPRE_Int i;
 
-   double *a_diag = new double[diag_nnz];
+   double *a_diag = mfem::New<double>(diag_nnz);
    for (i = 0; i < diag_nnz; i++)
    {
       a_diag[i] = 1.0;
    }
 
-   double *a_offd = new double[offd_nnz];
+   double *a_offd = mfem::New<double>(offd_nnz);
    for (i = 0; i < offd_nnz; i++)
    {
       a_offd[i] = 1.0;
@@ -1010,6 +1010,7 @@ HYPRE_Int HypreParMatrix::Mult(HypreParVector &x, HypreParVector &y,
 
 void HypreParMatrix::Mult(double a, const Vector &x, double b, Vector &y) const
 {
+   x.Pull();
    MFEM_ASSERT(x.Size() == Width(), "invalid x.Size() = " << x.Size()
                << ", expected size = " << Width());
    MFEM_ASSERT(y.Size() == Height(), "invalid y.Size() = " << y.Size()
@@ -1033,11 +1034,13 @@ void HypreParMatrix::Mult(double a, const Vector &x, double b, Vector &y) const
    }
 
    hypre_ParCSRMatrixMatvec(a, A, *X, b, *Y);
+   y.Push();
 }
 
 void HypreParMatrix::MultTranspose(double a, const Vector &x,
                                    double b, Vector &y) const
 {
+   x.Pull();
    MFEM_ASSERT(x.Size() == Height(), "invalid x.Size() = " << x.Size()
                << ", expected size = " << Height());
    MFEM_ASSERT(y.Size() == Width(), "invalid y.Size() = " << y.Size()
@@ -1063,6 +1066,7 @@ void HypreParMatrix::MultTranspose(double a, const Vector &x,
    }
 
    hypre_ParCSRMatrixMatvecT(a, A, *Y, b, *X);
+   y.Push();
 }
 
 HYPRE_Int HypreParMatrix::Mult(HYPRE_ParVector x, HYPRE_ParVector y,
@@ -1477,14 +1481,14 @@ void HypreParMatrix::Destroy()
    {
       if (diagOwner & 1)
       {
-         delete [] hypre_CSRMatrixI(A->diag);
-         delete [] hypre_CSRMatrixJ(A->diag);
+         mfem::Delete(hypre_CSRMatrixI(A->diag));
+         mfem::Delete(hypre_CSRMatrixJ(A->diag));
       }
       hypre_CSRMatrixI(A->diag) = NULL;
       hypre_CSRMatrixJ(A->diag) = NULL;
       if (diagOwner & 2)
       {
-         delete [] hypre_CSRMatrixData(A->diag);
+         mfem::Delete(hypre_CSRMatrixData(A->diag));
       }
       hypre_CSRMatrixData(A->diag) = NULL;
    }
@@ -1492,14 +1496,14 @@ void HypreParMatrix::Destroy()
    {
       if (offdOwner & 1)
       {
-         delete [] hypre_CSRMatrixI(A->offd);
-         delete [] hypre_CSRMatrixJ(A->offd);
+         mfem::Delete(hypre_CSRMatrixI(A->offd));
+         mfem::Delete(hypre_CSRMatrixJ(A->offd));
       }
       hypre_CSRMatrixI(A->offd) = NULL;
       hypre_CSRMatrixJ(A->offd) = NULL;
       if (offdOwner & 2)
       {
-         delete [] hypre_CSRMatrixData(A->offd);
+         mfem::Delete(hypre_CSRMatrixData(A->offd));
       }
       hypre_CSRMatrixData(A->offd) = NULL;
    }
@@ -1507,7 +1511,7 @@ void HypreParMatrix::Destroy()
    {
       if (colMapOwner & 1)
       {
-         delete [] hypre_ParCSRMatrixColMapOffd(A);
+         mfem::Delete(hypre_ParCSRMatrixColMapOffd(A));
       }
       hypre_ParCSRMatrixColMapOffd(A) = NULL;
    }
@@ -3328,7 +3332,7 @@ HypreLOBPCG::SetPreconditioner(Solver & precond)
 void
 HypreLOBPCG::SetOperator(Operator & A)
 {
-   int locSize = A.Width();
+   HYPRE_Int locSize = A.Width();
 
    if (HYPRE_AssumedPartitionCheck())
    {
@@ -3344,7 +3348,7 @@ HypreLOBPCG::SetOperator(Operator & A)
    {
       part = new HYPRE_Int[numProcs+1];
 
-      MPI_Allgather(&locSize, 1, MPI_INT,
+      MPI_Allgather(&locSize, 1, HYPRE_MPI_INT,
                     &part[1], 1, HYPRE_MPI_INT, comm);
 
       part[0] = 0;

linalg/operator.cpp

@@ -10,7 +10,9 @@
 // Software Foundation) version 2.1 dated February 1999.
 
 #include "vector.hpp"
+#include "dtensor.hpp"
 #include "operator.hpp"
+#include "../general/forall.hpp"
 
 #include <iostream>
 #include <iomanip>
@@ -163,25 +165,31 @@ ConstrainedOperator::ConstrainedOperator(Operator *A, const Array<int> &list,
 void ConstrainedOperator::EliminateRHS(const Vector &x, Vector &b) const
 {
    w = 0.0;
-
-   for (int i = 0; i < constraint_list.Size(); i++)
+   const int csz = constraint_list.Size();
+   const DeviceArray idx(constraint_list, csz);
+   const DeviceVector d_x(x, x.Size());
+   DeviceVector d_w(w, w.Size());
+   MFEM_FORALL(i, csz,
    {
-      w(constraint_list[i]) = x(constraint_list[i]);
-   }
+      const int id = idx[i];
+      d_w[id] = d_x[id];
+   });
 
    A->Mult(w, z);
 
    b -= z;
-
-   for (int i = 0; i < constraint_list.Size(); i++)
+   DeviceVector d_b(b, b.Size());
+   MFEM_FORALL(i, csz,
    {
-      b(constraint_list[i]) = x(constraint_list[i]);
-   }
+      const int id = idx[i];
+      d_b[id] = d_x[id];
+   });
 }
 
 void ConstrainedOperator::Mult(const Vector &x, Vector &y) const
 {
-   if (constraint_list.Size() == 0)
+   const int csz = constraint_list.Size();
+   if (csz == 0)
    {
       A->Mult(x, y);
       return;
@@ -189,17 +197,19 @@ void ConstrainedOperator::Mult(const Vector &x, Vector &y) const
 
    z = x;
 
-   for (int i = 0; i < constraint_list.Size(); i++)
-   {
-      z(constraint_list[i]) = 0.0;
-   }
+   const DeviceArray idx(constraint_list, csz);
+   DeviceVector d_z(z, z.Size());
+   MFEM_FORALL(i, csz, d_z[idx[i]] = 0.0;);
 
    A->Mult(z, y);
 
-   for (int i = 0; i < constraint_list.Size(); i++)
+   const DeviceVector d_x(x, x.Size());
+   DeviceVector d_y(y, y.Size());
+   MFEM_FORALL(i, csz,
    {
-      y(constraint_list[i]) = x(constraint_list[i]);
-   }
+      const int id = idx[i];
+      d_y[id] = d_x[id];
+   });
 }
 
 }

linalg/operator.hpp

@@ -124,14 +124,17 @@ public:
    enum Type
    {
       ANY_TYPE,         ///< ID for the base class Operator, i.e. any type.
-      MFEM_SPARSEMAT,   ///< ID for class SparseMatrix
+      MFEM_SPARSEMAT,   ///< ID for class SparseMatrix.
       Hypre_ParCSR,     ///< ID for class HypreParMatrix.
       PETSC_MATAIJ,     ///< ID for class PetscParMatrix, MATAIJ format.
       PETSC_MATIS,      ///< ID for class PetscParMatrix, MATIS format.
       PETSC_MATSHELL,   ///< ID for class PetscParMatrix, MATSHELL format.
       PETSC_MATNEST,    ///< ID for class PetscParMatrix, MATNEST format.
       PETSC_MATHYPRE,   ///< ID for class PetscParMatrix, MATHYPRE format.
-      PETSC_MATGENERIC  ///< ID for class PetscParMatrix, unspecified format.
+      PETSC_MATGENERIC, ///< ID for class PetscParMatrix, unspecified format.
+      Complex_Operator, ///< ID for class ComplexOperator.
+      MFEM_ComplexSparseMat, ///< ID for class ComplexSparseMatrix.
+      Complex_Hypre_ParCSR   ///< ID for class ComplexHypreParMatrix.
    };
 
    /// Return the type ID of the Operator class.
@@ -296,6 +299,27 @@ public:
 
    /// Operator application
    virtual void Mult(const Vector &x, Vector &y) const { y = x; }
+
+   /// Application of the transpose
+   virtual void MultTranspose(const Vector &x, Vector &y) const { y = x; }
+};
+
+
+/// Scaled Operator B: x -> a A(x).
+class ScaledOperator : public Operator
+{
+private:
+   const Operator &A_;
+   double a_;
+
+public:
+   /// Create a scalar product operator related to A.
+   explicit ScaledOperator(const Operator *A, double a)
+      : Operator(A->Width(), A->Height()), A_(*A), a_(a) { }
+
+   /// Operator application
+   virtual void Mult(const Vector &x, Vector &y) const
+   { A_.Mult(x, y); y *= a_; }
 };
 
 

linalg/petsc.cpp

@@ -708,15 +708,21 @@ BlockDiagonalConstructor(MPI_Comm comm,
       // Copy SparseMatrix into PETSc SeqAIJ format
       Mat lA;
       ierr = MatISGetLocalMat(A,&lA); PCHKERRQ(A,ierr);
-      if (sizeof(PetscInt) == sizeof(int))
-      {
-         ierr = MatSeqAIJSetPreallocationCSR(lA,diag->GetI(),diag->GetJ(),
-                                             diag->GetData()); PCHKERRQ(lA,ierr);
-      }
-      else
-      {
-         MFEM_ABORT("64bit indices not yet supported");
-      }
+      int *II = diag->GetI();
+      int *JJ = diag->GetJ();
+#if defined(PETSC_USE_64BIT_INDICES)
+      PetscInt *pII,*pJJ;
+      int m = diag->Height()+1, nnz = II[diag->Height()];
+      ierr = PetscMalloc2(m,&pII,nnz,&pJJ); PCHKERRQ(lA,ierr);
+      for (int i = 0; i < m; i++) { pII[i] = II[i]; }
+      for (int i = 0; i < nnz; i++) { pJJ[i] = JJ[i]; }
+      ierr = MatSeqAIJSetPreallocationCSR(lA,pII,pJJ,
+                                          diag->GetData()); PCHKERRQ(lA,ierr);
+      ierr = PetscFree2(pII,pJJ); PCHKERRQ(lA,ierr);
+#else
+      ierr = MatSeqAIJSetPreallocationCSR(lA,II,JJ,
+                                          diag->GetData()); PCHKERRQ(lA,ierr);
+#endif
    }
    else
    {
@@ -736,13 +742,16 @@ BlockDiagonalConstructor(MPI_Comm comm,
          CCHKERRQ(PETSC_COMM_SELF,ierr);
          ierr = PetscMemcpy(djj,diag->GetJ(),nnz*sizeof(PetscInt));
          CCHKERRQ(PETSC_COMM_SELF,ierr);
-         ierr = PetscMemcpy(da,diag->GetData(),nnz*sizeof(PetscScalar));
-         CCHKERRQ(PETSC_COMM_SELF,ierr);
       }
       else
       {
-         MFEM_ABORT("64bit indices not yet supported");
+         int *iii = diag->GetI();
+         int *jjj = diag->GetJ();
+         for (int i = 0; i < m; i++) { dii[i] = iii[i]; }
+         for (int i = 0; i < nnz; i++) { djj[i] = jjj[i]; }
       }
+      ierr = PetscMemcpy(da,diag->GetData(),nnz*sizeof(PetscScalar));
+      CCHKERRQ(PETSC_COMM_SELF,ierr);
       ierr = PetscCalloc1(m,&oii);
       CCHKERRQ(PETSC_COMM_SELF,ierr);
       if (commsize > 1)
@@ -1051,7 +1060,7 @@ void PetscParMatrix::ConvertOperator(MPI_Comm comm, const Operator &op, Mat* A,
          ierr = MatConvert(*A,MATIS,MAT_INPLACE_MATRIX,A); CCHKERRQ(comm,ierr);
 
          mfem::Array<Mat> *vmatsl2l = new mfem::Array<Mat>(nr);
-         for (PetscInt i=0; i<nr; i++) { (*vmatsl2l)[i] = matsl2l[i]; }
+         for (int i=0; i<(int)nr; i++) { (*vmatsl2l)[i] = matsl2l[i]; }
          ierr = PetscFree(matsl2l); CCHKERRQ(PETSC_COMM_SELF,ierr);
 
          PetscContainer c;
@@ -2150,7 +2159,7 @@ void PetscBCHandler::ApplyBC(const Vector &x, Vector &y)
    y = x;
    if (bctype == ZERO)
    {
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
       {
          y[ess_tdof_list[i]] = 0.0;
       }
@@ -2162,7 +2171,7 @@ void PetscBCHandler::ApplyBC(const Vector &x, Vector &y)
          Eval(eval_t,eval_g);
          eval_t_cached = eval_t;
       }
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
       {
          y[ess_tdof_list[i]] = eval_g[ess_tdof_list[i]];
       }
@@ -2174,7 +2183,7 @@ void PetscBCHandler::ApplyBC(Vector &x)
    (*this).SetUp(x.Size());
    if (bctype == ZERO)
    {
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
       {
          x[ess_tdof_list[i]] = 0.0;
       }
@@ -2186,7 +2195,7 @@ void PetscBCHandler::ApplyBC(Vector &x)
          Eval(eval_t,eval_g);
          eval_t_cached = eval_t;
       }
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
       {
          x[ess_tdof_list[i]] = eval_g[ess_tdof_list[i]];
       }
@@ -2198,14 +2207,14 @@ void PetscBCHandler::FixResidualBC(const Vector& x, Vector& y)
    (*this).SetUp(x.Size());
    if (bctype == ZERO)
    {
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
       {
          y[ess_tdof_list[i]] = x[ess_tdof_list[i]];
       }
    }
    else
    {
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
       {
          y[ess_tdof_list[i]] = x[ess_tdof_list[i]] - eval_g[ess_tdof_list[i]];
       }
@@ -3875,7 +3884,7 @@ static PetscErrorCode __mfem_snes_postcheck(SNESLineSearch ls,Vec X,Vec Y,Vec W,
                                             PetscBool *cy,PetscBool *cw, void* ctx)
 {
    __mfem_snes_ctx* snes_ctx = (__mfem_snes_ctx*)ctx;
-   bool lcy,lcw;
+   bool lcy = false,lcw = false;
 
    PetscFunctionBeginUser;
    mfem::PetscParVector x(X,true);
@@ -4222,7 +4231,21 @@ static PetscErrorCode Convert_Vmarks_IS(MPI_Comm comm,
                          (const PetscInt**)&jj,&done); CHKERRQ(ierr);
       MFEM_VERIFY(done,"Unable to perform MatGetRowIJ on " << i << " l2l matrix");
       ierr = MatGetSize(pl2l[i],NULL,&n); CHKERRQ(ierr);
+#if defined(PETSC_USE_64BIT_INDICES)
+      int  nnz = (int)ii[m];
+      int *mii = new int[m+1];
+      int *mjj = new int[nnz];
+      for (int j = 0; j < m+1; j++) { mii[j] = (int)ii[j]; }
+      for (int j = 0; j < nnz; j++) { mjj[j] = (int)jj[j]; }
+      l2l[i] = new mfem::SparseMatrix(mii,mjj,NULL,m,n,true,true,true);
+#else
       l2l[i] = new mfem::SparseMatrix(ii,jj,NULL,m,n,false,true,true);
+#endif
+      ierr = MatRestoreRowIJ(pl2l[i],0,PETSC_FALSE,PETSC_FALSE,&m,
+                             (const PetscInt**)&ii,
+                             (const PetscInt**)&jj,&done); CHKERRQ(ierr);
+      MFEM_VERIFY(done,"Unable to perform MatRestoreRowIJ on "
+                  << i << " l2l matrix");
    }
    nl = 0;
    for (int i = 0; i < l2l.Size(); i++) { nl += l2l[i]->Width(); }
@@ -4242,14 +4265,6 @@ static PetscErrorCode Convert_Vmarks_IS(MPI_Comm comm,
    ierr = Convert_Array_IS(comm,false,&sub_dof_marker,st,is); CCHKERRQ(comm,ierr);
    for (int i = 0; i < pl2l.Size(); i++)
    {
-      PetscInt  m = l2l[i]->Height();
-      PetscInt  *ii = l2l[i]->GetI(),*jj = l2l[i]->GetJ();
-      PetscBool done;
-      ierr = MatRestoreRowIJ(pl2l[i],0,PETSC_FALSE,PETSC_FALSE,&m,
-                             (const PetscInt**)&ii,
-                             (const PetscInt**)&jj,&done); CHKERRQ(ierr);
-      MFEM_VERIFY(done,"Unable to perform MatRestoreRowIJ on "
-                  << i << " l2l matrix");
       delete l2l[i];
    }
    PetscFunctionReturn(0);

linalg/sparsemat.cpp

@@ -12,6 +12,8 @@
 // Implementation of sparse matrix
 
 #include "linalg.hpp"
+#include "dtensor.hpp"
+#include "../general/forall.hpp"
 #include "../general/table.hpp"
 #include "../general/sort_pairs.hpp"
 
@@ -88,7 +90,7 @@ SparseMatrix::SparseMatrix(int *i, int *j, double *data, int m, int n,
    {
       ownData = true;
       int nnz = I[height];
-      A = new double[ nnz ];
+      A = mfem::New<double>(nnz);
       for (int i=0; i<nnz; ++i)
       {
          A[i] = 0.0;
@@ -108,9 +110,9 @@ SparseMatrix::SparseMatrix(int nrows, int ncols, int rowsize)
 #ifdef MFEM_USE_MEMALLOC
    NodesMem = NULL;
 #endif
-   I = new int[nrows + 1];
-   J = new int[nrows * rowsize];
-   A = new double[nrows * rowsize];
+   I = mfem::New<int>(nrows + 1);
+   J = mfem::New<int>(nrows * rowsize);
+   A = mfem::New<double>(nrows * rowsize);
 
    for (int i = 0; i <= nrows; i++)
    {
@@ -126,8 +128,8 @@ SparseMatrix::SparseMatrix(const SparseMatrix &mat, bool copy_graph)
       const int nnz = mat.I[height];
       if (copy_graph)
       {
-         I = new int[height+1];
-         J = new int[nnz];
+         I = mfem::New<int>(height+1);
+         J = mfem::New<int>(nnz);
          memcpy(I, mat.I, sizeof(int)*(height+1));
          memcpy(J, mat.J, sizeof(int)*nnz);
          ownGraph = true;
@@ -138,7 +140,7 @@ SparseMatrix::SparseMatrix(const SparseMatrix &mat, bool copy_graph)
          J = mat.J;
          ownGraph = false;
       }
-      A = new double[nnz];
+      A = mfem::New<double>(nnz);
       memcpy(A, mat.A, sizeof(double)*nnz);
       ownData = true;
 
@@ -196,9 +198,9 @@ SparseMatrix::SparseMatrix(const Vector &v)
 #ifdef MFEM_USE_MEMALLOC
    NodesMem = NULL;
 #endif
-   I = new int[height + 1];
-   J = new int[height];
-   A = new double[height];
+   I = mfem::New<int>(height + 1);
+   J = mfem::New<int>(height);
+   A = mfem::New<double>(height);
 
    for (int i = 0; i <= height; i++)
    {
@@ -582,16 +584,22 @@ void SparseMatrix::AddMult(const Vector &x, Vector &y, const double a) const
 
    if (a == 1.0)
    {
-#ifndef MFEM_USE_OPENMP
-      for (i = j = 0; i < height; i++)
+#ifndef MFEM_USE_LEGACY_OPENMP
+      const DeviceArray d_I(I);
+      const DeviceArray d_J(J);
+      const DeviceVector d_A(A);
+      const DeviceVector d_x(x, x.Size());
+      DeviceVector d_y(y, y.Size());
+      MFEM_FORALL(i, height,
       {
          double d = 0.0;
-         for (end = Ip[i+1]; j < end; j++)
+         const int end = d_I[i+1];
+         for (int j=d_I[i]; j < end; j++)
          {
-            d += Ap[j] * xp[Jp[j]];
+            d += d_A[j] * d_x[d_J[j]];
          }
-         yp[i] += d;
-      }
+         d_y[i] += d;
+      });
 #else
       #pragma omp parallel for private(j,end)
       for (i = 0; i < height; i++)
@@ -635,13 +643,11 @@ void SparseMatrix::AddMultTranspose(const Vector &x, Vector &y,
                "Output vector size (" << y.Size() << ") must match matrix width (" << width
                << ")");
 
-   int i, j, end;
-   double *yp = y.GetData();
-
    if (A == NULL)
    {
+      double *yp = y.GetData();
       // The matrix is not finalized, but multiplication is still possible
-      for (i = 0; i < height; i++)
+      for (int i = 0; i < height; i++)
       {
          RowNode *row = Rows[i];
          double b = a * x(i);
@@ -652,16 +658,23 @@ void SparseMatrix::AddMultTranspose(const Vector &x, Vector &y,
       }
       return;
    }
-
-   for (i = 0; i < height; i++)
+   // Prepare the lambda capture and get our pointers from the memory manager
+   const int d_height = height;
+   const DeviceArray d_I(I);
+   const DeviceArray d_J(J);
+   const DeviceVector d_A(A);
+   const DeviceVector d_x(x, x.Size());
+   DeviceVector d_y(y, y.Size());
+   MFEM_FORALL(i, d_height,
    {
-      double xi = a * x(i);
-      end = I[i+1];
-      for (j = I[i]; j < end; j++)
+      const double xi = a * d_x[i];
+      const int end = d_I[i+1];
+      for (int j = d_I[i]; j < end; j++)
       {
-         yp[J[j]] += A[j]*xi;
+         const int Jj = d_J[j];
+         AtomicAdd(&d_y[Jj], d_A[j] * xi);
       }
-   }
+   });
 }
 
 void SparseMatrix::PartMult(
@@ -709,18 +722,26 @@ void SparseMatrix::BooleanMult(const Array<int> &x, Array<int> &y) const
    y.SetSize(Height());
    y = 0;
 
-   for (int i = 0; i < Height(); i++)
+   const int height = Height();
+   const int *d_I = mfem::Ptr(I);
+   const int *d_J = mfem::Ptr(J);
+   const DeviceArray d_x(x, x.Size());
+   DeviceArray d_y(y, y.Size());
+   MFEM_FORALL(i, height,
    {
-      int end = I[i+1];
-      for (int j = I[i]; j < end; j++)
+      const int end = d_I[i+1];
+      for (int j = d_I[i]; j < end; j++)
       {
-         if (x[J[j]])
+         if (d_x[d_J[j]])
          {
-            y[i] = x[J[j]];
+            d_y[i] = d_x[d_J[j]];
             break;
          }
       }
-   }
+   });
+   // Sync on host for some verifications in
+   // pfespace GetEssentialTrueDofs and MarkerToList
+   mfem::Pull(y);
 }
 
 void SparseMatrix::BooleanMultTranspose(const Array<int> &x,
@@ -824,7 +845,7 @@ void SparseMatrix::Finalize(int skip_zeros, bool fix_empty_rows)
    delete [] ColPtrNode;
    ColPtrNode = NULL;
 
-   I = new int[height+1];
+   I = mfem::New<int>(height+1);
    I[0] = 0;
    for (i = 1; i <= height; i++)
    {
@@ -839,8 +860,8 @@ void SparseMatrix::Finalize(int skip_zeros, bool fix_empty_rows)
    }
 
    nz = I[height];
-   J = new int[nz];
-   A = new double[nz];
+   J = mfem::New<int>(nz);
+   A = mfem::New<double>(nz);
    // Assume we're sorted until we find out otherwise
    isSorted = true;
    for (j = i = 0; i < height; i++)
@@ -901,7 +922,7 @@ void SparseMatrix::GetBlocks(Array2D<SparseMatrix *> &blocks) const
    {
       for (int i = 0; i < br; i++)
       {
-         int *bI = new int[nr + 1];
+         int *bI = mfem::New<int>(nr + 1);
          for (int k = 0; k <= nr; k++)
          {
             bI[k] = 0;
@@ -945,8 +966,8 @@ void SparseMatrix::GetBlocks(Array2D<SparseMatrix *> &blocks) const
          {
             rs = b.I[k], b.I[k] = nnz, nnz += rs;
          }
-         b.J = new int[nnz];
-         b.A = new double[nnz];
+         b.J = mfem::New<int>(nnz);
+         b.A = mfem::New<double>(nnz);
       }
    }
 
@@ -2708,15 +2729,15 @@ void SparseMatrix::Destroy()
 {
    if (I != NULL && ownGraph)
    {
-      delete [] I;
+      mfem::Delete(I);
    }
    if (J != NULL && ownGraph)
    {
-      delete [] J;
+      mfem::Delete(J);
    }
    if (A != NULL && ownData)
    {
-      delete [] A;
+      mfem::Delete(A);
    }
 
    if (Rows != NULL)
@@ -2806,9 +2827,9 @@ SparseMatrix *Transpose (const SparseMatrix &A)
    A_j    = A.GetJ();
    A_data = A.GetData();
 
-   At_i = new int[n+1];
-   At_j = new int[nnz];
-   At_data = new double[nnz];
+   At_i = mfem::New<int>(n+1);
+   At_j = mfem::New<int>(nnz);
+   At_data = mfem::New<double>(nnz);
 
    for (i = 0; i <= n; i++)
    {
@@ -2877,9 +2898,9 @@ SparseMatrix *TransposeAbstractSparseMatrix (const AbstractSparseMatrix &A,
    }
    nnz = A.NumNonZeroElems();
 
-   At_i = new int[n+1];
-   At_j = new int[nnz];
-   At_data = new double[nnz];
+   At_i = mfem::New<int>(n+1);
+   At_j = mfem::New<int>(nnz);
+   At_data = mfem::New<double>(nnz);
 
    for (i = 0; i <= n; i++)
    {
@@ -2956,7 +2977,7 @@ SparseMatrix *Mult (const SparseMatrix &A, const SparseMatrix &B,
 
    if (OAB == NULL)
    {
-      C_i = new int[nrowsA+1];
+      C_i = mfem::New<int>(nrowsA+1);
 
       C_i[0] = num_nonzeros = 0;
       for (ic = 0; ic < nrowsA; ic++)
@@ -2977,8 +2998,8 @@ SparseMatrix *Mult (const SparseMatrix &A, const SparseMatrix &B,
          C_i[ic+1] = num_nonzeros;
       }
 
-      C_j    = new int[num_nonzeros];
-      C_data = new double[num_nonzeros];
+      C_j    = mfem::New<int>(num_nonzeros);
+      C_data = mfem::New<double>(num_nonzeros);
 
       C = new SparseMatrix (C_i, C_j, C_data, nrowsA, ncolsB);
 
@@ -3081,7 +3102,7 @@ SparseMatrix *MultAbstractSparseMatrix (const AbstractSparseMatrix &A,
       B_marker[ib] = -1;
    }
 
-   C_i = new int[nrowsA+1];
+   C_i = mfem::New<int>(nrowsA+1);
 
    C_i[0] = num_nonzeros = 0;
 
@@ -3107,8 +3128,8 @@ SparseMatrix *MultAbstractSparseMatrix (const AbstractSparseMatrix &A,
       C_i[ic+1] = num_nonzeros;
    }
 
-   C_j    = new int[num_nonzeros];
-   C_data = new double[num_nonzeros];
+   C_j    = mfem::New<int>(num_nonzeros);
+   C_data = mfem::New<double>(num_nonzeros);
 
    C = new SparseMatrix(C_i, C_j, C_data, nrowsA, ncolsB);
 
@@ -3234,7 +3255,7 @@ SparseMatrix * Add(double a, const SparseMatrix & A, double b,
    int nrows = A.Height();
    int ncols = A.Width();
 
-   int * C_i = new int[nrows+1];
+   int * C_i =mfem::New<int>(nrows+1);
    int * C_j;
    double * C_data;
 
@@ -3271,8 +3292,8 @@ SparseMatrix * Add(double a, const SparseMatrix & A, double b,
       C_i[ic+1] = num_nonzeros;
    }
 
-   C_j = new int[num_nonzeros];
-   C_data = new double[num_nonzeros];
+   C_j = mfem::New<int>(num_nonzeros);
+   C_data = mfem::New<double>(num_nonzeros);
 
    for (int ia = 0; ia < ncols; ia++)
    {

linalg/strumpack.cpp

@@ -145,11 +145,34 @@ void STRUMPACKSolver::SetReorderingStrategy( strumpack::ReorderingStrategy
    solver_->options().set_reordering_method( method );
 }
 
-void STRUMPACKSolver::SetMC64Job( strumpack::MC64Job job )
+void STRUMPACKSolver::DisableMatching( )
 {
-   solver_->options().set_mc64job( job );
+#if STRUMPACK_VERSION_MAJOR >= 3
+   solver_->options().set_matching( strumpack::MatchingJob::NONE );
+#else
+   solver_->options().set_mc64job( strumpack::MC64Job::NONE );
+#endif
 }
 
+void STRUMPACKSolver::EnableMatching( )
+{
+#if STRUMPACK_VERSION_MAJOR >= 3
+   solver_->options().set_matching
+   ( strumpack::MatchingJob::MAX_DIAGONAL_PRODUCT_SCALING );
+#else
+   solver_->options().set_mc64job
+   ( strumpack::MC64Job::MAX_DIAGONAL_PRODUCT_SCALING );
+#endif
+}
+
+#if STRUMPACK_VERSION_MAJOR >= 3
+void STRUMPACKSolver::EnableParallelMatching( )
+{
+   solver_->options().set_matching
+   ( strumpack::MatchingJob::COMBBLAS );
+}
+#endif
+
 void STRUMPACKSolver::SetRelTol( double rtol )
 {
    solver_->options().set_rel_tol( rtol );

linalg/strumpack.hpp

@@ -121,21 +121,30 @@ public:
    void SetReorderingStrategy( strumpack::ReorderingStrategy method );
 
    /**
-    * MC64 performs (static) pivoting. Using a matching algorithm, it permutes
-    * the sparse input matrix in order to get nonzero elements on the
-    * diagonal. If the input matrix is already diagonally dominant, this
+    * Disable static pivoting for stability. The static pivoting in strumpack
+    * permutes the sparse input matrix in order to get large (nonzero) elements
+    * on the diagonal. If the input matrix is already diagonally dominant, this
     * reordering can be disabled.
-    * Possible values are:
-    *    NONE:                          Don't do anything
-    *    MAX_CARDINALITY:               Maximum cardinality
-    *    MAX_SMALLEST_DIAGONAL:         Maximize smallest diagonal value
-    *    MAX_SMALLEST_DIAGONAL_2:       Same as MAX_SMALLEST_DIAGONAL, but
-    *                                   different algorithm
-    *    MAX_DIAGONAL_SUM:              Maximize sum of diagonal values
-    *    MAX_DIAGONAL_PRODUCT_SCALING:  Maximize the product of the diagonal
-    *                                   values and perform row & column scaling
     */
-   void SetMC64Job( strumpack::MC64Job job );
+   void DisableMatching();
+
+   /**
+    * Enable static pivoting for stability using the MC64 algorithm with
+    * job=5. Using a matching algorithm, this will permute the sparse input
+    * matrix in order to get nonzero elements (as large as possible) on the
+    * diagonal. And will also scale the rows and columns of the matrix.
+    */
+   void EnableMatching();
+
+#if STRUMPACK_VERSION_MAJOR >= 3
+   /**
+    * Use the AWPM (approximate weight perfect matching) algorithm from the
+    * Combinatorial BLAS library for static pivoting, i.e. getting large
+    * nonzeros on the diagonal. This requires that strumpack was compiled with
+    * support for Combinatorial BLAS.
+    */
+   void EnableParallelMatching();
+#endif
 
 private:
    void Init( int argc, char* argv[] );

linalg/superlu.cpp

@@ -289,7 +289,7 @@ void SuperLUSolver::Init()
    }
 
    npcol_ = (int)(numProcs_ / nprow_);
-   assert(nprow_ * npcol_ == numProcs_);
+   MFEM_ASSERT(nprow_ * npcol_ == numProcs_, "");
 
    PStatInit(stat); // Initialize the statistics variables.
 }
@@ -423,7 +423,7 @@ void SuperLUSolver::SetupGrid()
       }
 
       npcol_ = (int)(numProcs_ / nprow_);
-      assert(nprow_ * npcol_ == numProcs_);
+      MFEM_ASSERT(nprow_ * npcol_ == numProcs_, "");
    }
 
    superlu_gridinit(comm_, nprow_, npcol_, grid);

linalg/vector.cpp

@@ -12,6 +12,8 @@
 // Implementation of data type vector
 
 #include "vector.hpp"
+#include "dtensor.hpp"
+#include "../general/forall.hpp"
 
 #if defined(MFEM_USE_SUNDIALS) && defined(MFEM_USE_MPI)
 #include <nvector/nvector_parallel.h>
@@ -28,6 +30,16 @@
 namespace mfem
 {
 
+void Vector::Push() const
+{
+   mfem::Push(data, size*sizeof(double));
+}
+
+void Vector::Pull() const
+{
+   mfem::Pull(data, size*sizeof(double));
+}
+
 Vector::Vector(const Vector &v)
 {
    int s = v.Size();
@@ -36,8 +48,8 @@ Vector::Vector(const Vector &v)
    {
       MFEM_ASSERT(v.data, "invalid source vector");
       allocsize = size = s;
-      data = new double[s];
-      std::memcpy(data, v.data, sizeof(double)*s);
+      data = mfem::New<double>(s);
+      mfem::Memcpy(data, v.data, sizeof(double)*s);
    }
    else
    {
@@ -88,17 +100,7 @@ const double &Vector::Elem(int i) const
 
 double Vector::operator*(const double *v) const
 {
-   int s = size;
-   const double *d = data;
-   double prod = 0.0;
-#ifdef MFEM_USE_OPENMP
-   #pragma omp parallel for reduction(+:prod)
-#endif
-   for (int i = 0; i < s; i++)
-   {
-      prod += d[i] * v[i];
-   }
-   return prod;
+   return Dot(size, data, v);
 }
 
 double Vector::operator*(const Vector &v) const
@@ -118,7 +120,7 @@ Vector &Vector::operator=(const double *v)
    if (data != v)
    {
       MFEM_ASSERT(data + size <= v || v + size <= data, "Vectors overlap!");
-      std::memcpy(data, v, sizeof(double)*size);
+      mfem::Memcpy(data, v, sizeof(double)*size);
    }
    return *this;
 }
@@ -131,40 +133,30 @@ Vector &Vector::operator=(const Vector &v)
 
 Vector &Vector::operator=(double value)
 {
-   int i, s = size;
-   double *p = data, v = value;
-   for (i = 0; i < s; i++)
-   {
-      *(p++) = v;
-   }
+   DeviceVector y(data, size);
+   MFEM_FORALL(i, size, y[i] = value;);
    return *this;
 }
 
 Vector &Vector::operator*=(double c)
 {
-   for (int i = 0; i < size; i++)
-   {
-      data[i] *= c;
-   }
+   DeviceVector y(data, size);
+   MFEM_FORALL(i, size, y[i] *= c;);
    return *this;
 }
 
 Vector &Vector::operator/=(double c)
 {
-   double m = 1.0/c;
-   for (int i = 0; i < size; i++)
-   {
-      data[i] *= m;
-   }
+   const double m = 1.0/c;
+   DeviceVector y(data, size);
+   MFEM_FORALL(i, size, y[i] *= m;);
    return *this;
 }
 
 Vector &Vector::operator-=(double c)
 {
-   for (int i = 0; i < size; i++)
-   {
-      data[i] -= c;
-   }
+   DeviceVector y(data, size);
+   MFEM_FORALL(i, size, y[i] -= c;);
    return *this;
 }
 
@@ -176,10 +168,10 @@ Vector &Vector::operator-=(const Vector &v)
       mfem_error("Vector::operator-=(const Vector &)");
    }
 #endif
-   for (int i = 0; i < size; i++)
-   {
-      data[i] -= v(i);
-   }
+   const int N = size;
+   DeviceVector y(data, N);
+   const DeviceVector x(v, N);
+   MFEM_FORALL(i, N, y[i] -= x[i];);
    return *this;
 }
 
@@ -191,10 +183,10 @@ Vector &Vector::operator+=(const Vector &v)
       mfem_error("Vector::operator+=(const Vector &)");
    }
 #endif
-   for (int i = 0; i < size; i++)
-   {
-      data[i] += v(i);
-   }
+   const int N = size;
+   DeviceVector y(data, N);
+   const DeviceVector x(v, N);
+   MFEM_FORALL(i, N, y[i] += x[i];);
    return *this;
 }
 
@@ -208,10 +200,10 @@ Vector &Vector::Add(const double a, const Vector &Va)
 #endif
    if (a != 0.0)
    {
-      for (int i = 0; i < size; i++)
-      {
-         data[i] += a * Va(i);
-      }
+      const int N = size;
+      DeviceVector y(data, N);
+      const DeviceVector x(Va, N);
+      MFEM_FORALL(i, N, y[i] += a * x[i];);
    }
    return *this;
 }
@@ -224,10 +216,10 @@ Vector &Vector::Set(const double a, const Vector &Va)
       mfem_error("Vector::Set(const double, const Vector &)");
    }
 #endif
-   for (int i = 0; i < size; i++)
-   {
-      data[i] = a * Va(i);
-   }
+   const int N = size;
+   DeviceVector y(data, N);
+   const DeviceVector x(Va, N);
+   MFEM_FORALL(i, N, y[i] = a * x[i];);
    return *this;
 }
 
@@ -251,10 +243,8 @@ void Vector::SetVector(const Vector &v, int offset)
 
 void Vector::Neg()
 {
-   for (int i = 0; i < size; i++)
-   {
-      data[i] = -data[i];
-   }
+   DeviceVector y(data, size);
+   MFEM_FORALL(i, size, y[i] = -y[i];);
 }
 
 void add(const Vector &v1, const Vector &v2, Vector &v)
@@ -266,13 +256,19 @@ void add(const Vector &v1, const Vector &v2, Vector &v)
    }
 #endif
 
-#ifdef MFEM_USE_OPENMP
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+   const int N = v.size;
+   DeviceVector y(v, N);
+   const DeviceVector x1(v1, N);
+   const DeviceVector x2(v2, N);
+   MFEM_FORALL(i, N, y[i] = x1[i] + x2[i];);
+#else
    #pragma omp parallel for
-#endif
    for (int i = 0; i < v.size; i++)
    {
       v.data[i] = v1.data[i] + v2.data[i];
    }
+#endif
 }
 
 void add(const Vector &v1, double alpha, const Vector &v2, Vector &v)
@@ -295,14 +291,21 @@ void add(const Vector &v1, double alpha, const Vector &v2, Vector &v)
    {
       const double *v1p = v1.data, *v2p = v2.data;
       double *vp = v.data;
-      int s = v.size;
-#ifdef MFEM_USE_OPENMP
+
+      const int s = v.size;
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+      const int N = s;
+      DeviceVector d_z(vp, N);
+      const DeviceVector d_x(v1p, N);
+      const DeviceVector d_y(v2p, N);
+      MFEM_FORALL(i, N, d_z[i] = d_x[i] + alpha * d_y[i];);
+#else
       #pragma omp parallel for
-#endif
       for (int i = 0; i < s; i++)
       {
          vp[i] = v1p[i] + alpha*v2p[i];
       }
+#endif
    }
 }
 
@@ -326,15 +329,19 @@ void add(const double a, const Vector &x, const Vector &y, Vector &z)
       const double *xp = x.data;
       const double *yp = y.data;
       double       *zp = z.data;
-      int            s = x.size;
-
-#ifdef MFEM_USE_OPENMP
+      const int      s = x.size;
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+      DeviceVector z(zp, s);
+      const DeviceVector x(xp, s);
+      const DeviceVector y(yp, s);
+      MFEM_FORALL(i, s, z[i] = a * (x[i] + y[i]););
+#else
       #pragma omp parallel for
-#endif
       for (int i = 0; i < s; i++)
       {
          zp[i] = a * (xp[i] + yp[i]);
       }
+#endif
    }
 }
 
@@ -371,15 +378,20 @@ void add(const double a, const Vector &x,
       const double *xp = x.data;
       const double *yp = y.data;
       double       *zp = z.data;
-      int            s = x.size;
+      const int      s = x.size;
 
-#ifdef MFEM_USE_OPENMP
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+      DeviceVector z(zp, s);
+      const DeviceVector x(xp, s);
+      const DeviceVector y(yp, s);
+      MFEM_FORALL(i, s, z[i] = a * x[i] + b * y[i];);
+#else
       #pragma omp parallel for
-#endif
       for (int i = 0; i < s; i++)
       {
          zp[i] = a * xp[i] + b * yp[i];
       }
+#endif
    }
 }
 
@@ -394,15 +406,20 @@ void subtract(const Vector &x, const Vector &y, Vector &z)
    const double *xp = x.data;
    const double *yp = y.data;
    double       *zp = z.data;
-   int            s = x.size;
+   const int     s = x.size;
 
-#ifdef MFEM_USE_OPENMP
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+   DeviceVector zd(zp, s);
+   const DeviceVector xd(xp, s);
+   const DeviceVector yd(yp, s);
+   MFEM_FORALL(i, s, zd[i] = xd[i] - yd[i];);
+#else
    #pragma omp parallel for
-#endif
    for (int i = 0; i < s; i++)
    {
       zp[i] = xp[i] - yp[i];
    }
+#endif
 }
 
 void subtract(const double a, const Vector &x, const Vector &y, Vector &z)
@@ -426,121 +443,136 @@ void subtract(const double a, const Vector &x, const Vector &y, Vector &z)
       const double *xp = x.data;
       const double *yp = y.data;
       double       *zp = z.data;
-      int            s = x.size;
+      const int      s = x.size;
 
-#ifdef MFEM_USE_OPENMP
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+      DeviceVector zd(zp, s);
+      const DeviceVector xd(xp, s);
+      const DeviceVector yd(yp, s);
+      MFEM_FORALL(i, s, zd[i] = a * (xd[i] - yd[i]););
+#else
       #pragma omp parallel for
-#endif
       for (int i = 0; i < s; i++)
       {
          zp[i] = a * (xp[i] - yp[i]);
       }
+#endif
    }
 }
 
 void Vector::median(const Vector &lo, const Vector &hi)
 {
-   double *v = data;
-
-   for (int i = 0; i < size; i++)
+   const int N = size;
+   DeviceVector v(data, N);
+   const DeviceVector l(lo, N);
+   const DeviceVector h(hi, N);
+   MFEM_FORALL(i, N,
    {
-      if (v[i] < lo[i])
+      if (v[i] < l[i])
       {
-         v[i] = lo[i];
+         v[i] = l[i];
       }
-      else if (v[i] > hi[i])
+      else if (v[i] > h[i])
       {
-         v[i] = hi[i];
+         v[i] = h[i];
       }
-   }
+   });
+}
+
+static void GetSubvector(const int N,
+                         double *y, const double *x, const int* dofs)
+{
+   DeviceVector d_y(y, N);
+   const DeviceVector d_x(x, N);
+   const DeviceArray d_dofs(dofs, N);
+   MFEM_FORALL(i, N,
+   {
+      const int dof_i = d_dofs[i];
+      d_y[i] = dof_i >= 0 ? d_x[dof_i] : -d_x[-dof_i-1];
+   });
 }
 
 void Vector::GetSubVector(const Array<int> &dofs, Vector &elemvect) const
 {
-   int i, j, n = dofs.Size();
-
-   elemvect.SetSize (n);
-
-   for (i = 0; i < n; i++)
-   {
-      if ((j=dofs[i]) >= 0)
-      {
-         elemvect(i) = data[j];
-      }
-      else
-      {
-         elemvect(i) = -data[-1-j];
-      }
-   }
+   const int n = dofs.Size();
+   elemvect.SetSize(n);
+   mfem::GetSubvector(n, elemvect, data, dofs);
 }
 
 void Vector::GetSubVector(const Array<int> &dofs, double *elem_data) const
 {
-   int i, j, n = dofs.Size();
+   mfem::GetSubvector(dofs.Size(), elem_data, data,dofs);
+}
 
-   for (i = 0; i < n; i++)
+static void SetSubvector(const int N, double* y, const double d,
+                         const int* dofs)
+{
+   DeviceVector d_y(y,N);
+   const DeviceArray d_dofs(dofs,N);
+   MFEM_FORALL(i, N,
    {
-      if ((j=dofs[i]) >= 0)
+      const int j = d_dofs[i];
+      if (j >= 0)
       {
-         elem_data[i] = data[j];
+         d_y[j] = d;
       }
       else
       {
-         elem_data[i] = -data[-1-j];
+         d_y[-1-j] = -d;
       }
-   }
+   });
+}
+
+static void SetSubvector(const int N, double *y, const double *x,
+                         const int* dofs)
+{
+   DeviceVector d_y(y,N);
+   const DeviceVector d_x(x,N);
+   const DeviceArray d_dofs(dofs,N);
+   MFEM_FORALL(i, N,
+   {
+      const int dof_i = d_dofs[i];
+      if (dof_i >= 0)
+      {
+         d_y[dof_i] = d_x[i];
+      }
+      else
+      {
+         d_y[-1-dof_i] = -d_x[i];
+      }
+   });
 }
 
 void Vector::SetSubVector(const Array<int> &dofs, const double value)
 {
-   const int n = dofs.Size();
-
-   for (int i = 0; i < n; i++)
-   {
-      const int j = dofs[i];
-      if (j >= 0)
-      {
-         data[j] = value;
-      }
-      else
-      {
-         data[-1-j] = -value;
-      }
-   }
+   mfem::SetSubvector(dofs.Size(), data, value, dofs);
 }
 
 void Vector::SetSubVector(const Array<int> &dofs, const Vector &elemvect)
 {
-   int i, j, n = dofs.Size();
-
-   for (i = 0; i < n; i++)
-   {
-      if ((j=dofs[i]) >= 0)
-      {
-         data[j] = elemvect(i);
-      }
-      else
-      {
-         data[-1-j] = -elemvect(i);
-      }
-   }
+   mfem::SetSubvector(dofs.Size(), data, elemvect, dofs);
 }
 
 void Vector::SetSubVector(const Array<int> &dofs, double *elem_data)
 {
-   int i, j, n = dofs.Size();
+   mfem::SetSubvector(dofs.Size(), data, elem_data, dofs);
+}
 
-   for (i = 0; i < n; i++)
+static void AddElement(const int N, const int *dofs, const double *x, double *y)
+{
+   DeviceVector d_y(y,N);
+   const DeviceVector d_x(x,N);
+   const DeviceArray d_dofs(dofs,N);
+   MFEM_FORALL(i, N,
    {
-      if ((j=dofs[i]) >= 0)
-      {
-         data[j] = elem_data[i];
-      }
+      const int j = d_dofs[i];
+      if (j >= 0)
+         d_y[j] += d_x[i];
       else
       {
-         data[-1-j] = -elem_data[i];
+         d_y[-1-j] -= d_x[i];
       }
-   }
+   });
 }
 
 void Vector::AddElementVector(const Array<int> &dofs, const Vector &elemvect)
@@ -548,55 +580,32 @@ void Vector::AddElementVector(const Array<int> &dofs, const Vector &elemvect)
    MFEM_ASSERT(dofs.Size() == elemvect.Size(), "Size mismatch: "
                "length of dofs is " << dofs.Size() <<
                ", length of elemvect is " << elemvect.Size());
-   int i, j, n = dofs.Size();
-
-   for (i = 0; i < n; i++)
-   {
-      if ((j=dofs[i]) >= 0)
-      {
-         data[j] += elemvect(i);
-      }
-      else
-      {
-         data[-1-j] -= elemvect(i);
-      }
-   }
+   mfem::AddElement(dofs.Size(), dofs, elemvect.GetData(), data);
 }
 
 void Vector::AddElementVector(const Array<int> &dofs, double *elem_data)
 {
-   int i, j, n = dofs.Size();
-
-   for (i = 0; i < n; i++)
-   {
-      if ((j = dofs[i]) >= 0)
-      {
-         data[j] += elem_data[i];
-      }
-      else
-      {
-         data[-1-j] -= elem_data[i];
-      }
-   }
+   mfem::AddElement(dofs.Size(), dofs, elem_data, data);
 }
 
 void Vector::AddElementVector(const Array<int> &dofs, const double a,
                               const Vector &elemvect)
 {
-   MFEM_ASSERT(dofs.Size() == elemvect.Size(), "");
-   int i, j, n = dofs.Size();
-
-   for (i = 0; i < n; i++)
+   const int N = dofs.Size();
+   const double alpha = a;
+   DeviceVector d_y(data, N);
+   const DeviceVector d_x(elemvect, N);
+   const DeviceArray d_dofs(dofs, N);
+   MFEM_FORALL(i, N,
    {
-      if ((j=dofs[i]) >= 0)
-      {
-         data[j] += a * elemvect(i);
-      }
+      const int j = d_dofs[i];
+      if (j >= 0)
+         d_y[j] += alpha * d_x[i];
       else
       {
-         data[-1-j] -= a * elemvect(i);
+         d_y[-1-j] -= alpha * d_x[i];
       }
-   }
+   });
 }
 
 void Vector::SetSubVectorComplement(const Array<int> &dofs, const double val)
@@ -610,7 +619,7 @@ void Vector::SetSubVectorComplement(const Array<int> &dofs, const double val)
 void Vector::Print(std::ostream &out, int width) const
 {
    if (!size) { return; }
-
+   Pull();
    for (int i = 0; 1; )
    {
       out << data[i];
@@ -812,6 +821,146 @@ double Vector::Sum() const
    return sum;
 }
 
+#ifdef MFEM_USE_CUDA
+static __global__ void cuKernelMin(const int N, double *gdsr, const double *x)
+{
+   __shared__ double s_min[MFEM_CUDA_BLOCKS];
+   const int n = blockDim.x*blockIdx.x + threadIdx.x;
+   if (n>=N) { return; }
+   const int bid = blockIdx.x;
+   const int tid = threadIdx.x;
+   const int bbd = bid*blockDim.x;
+   const int rid = bbd+tid;
+   s_min[tid] = x[n];
+   for (int workers=blockDim.x>>1; workers>0; workers>>=1)
+   {
+      __syncthreads();
+      if (tid >= workers) { continue; }
+      if (rid >= N) { continue; }
+      const int dualTid = tid + workers;
+      if (dualTid >= N) { continue; }
+      const int rdd = bbd+dualTid;
+      if (rdd >= N) { continue; }
+      if (dualTid >= blockDim.x) { continue; }
+      s_min[tid] = fmin(s_min[tid], s_min[dualTid]);
+   }
+   if (tid==0) { gdsr[bid] = s_min[0]; }
+}
+
+static double cuVectorMin(const int N, const double *X)
+{
+   const DeviceVector x(X, N);
+   const int tpb = MFEM_CUDA_BLOCKS;
+   const int blockSize = MFEM_CUDA_BLOCKS;
+   const int gridSize = (N+blockSize-1)/blockSize;
+   const int min_sz = (N%tpb)==0? (N/tpb) : (1+N/tpb);
+   const int bytes = min_sz*sizeof(double);
+   static double *h_min = NULL;
+   if (!h_min) { h_min = (double*)calloc(min_sz,sizeof(double)); }
+   static CUdeviceptr gdsr = (CUdeviceptr) NULL;
+   if (!gdsr) { ::cuMemAlloc(&gdsr,bytes); }
+   cuKernelMin<<<gridSize,blockSize>>>(N, (double*)gdsr, x);
+   MFEM_CUDA_CHECK_RT(cudaGetLastError());
+   ::cuMemcpy((CUdeviceptr)h_min,(CUdeviceptr)gdsr,bytes);
+   double min = std::numeric_limits<double>::infinity();
+   for (int i = 0; i < min_sz; i++) { min = fmin(min, h_min[i]); }
+   return min;
+}
+
+static __global__ void cuKernelDot(const int N, double *gdsr,
+                                   const double *x, const double *y)
+{
+   __shared__ double s_dot[MFEM_CUDA_BLOCKS];
+   const int n = blockDim.x*blockIdx.x + threadIdx.x;
+   if (n>=N) { return; }
+   const int bid = blockIdx.x;
+   const int tid = threadIdx.x;
+   const int bbd = bid*blockDim.x;
+   const int rid = bbd+tid;
+   s_dot[tid] = x[n] * y[n];
+   for (int workers=blockDim.x>>1; workers>0; workers>>=1)
+   {
+      __syncthreads();
+      if (tid >= workers) { continue; }
+      if (rid >= N) { continue; }
+      const int dualTid = tid + workers;
+      if (dualTid >= N) { continue; }
+      const int rdd = bbd+dualTid;
+      if (rdd >= N) { continue; }
+      if (dualTid >= blockDim.x) { continue; }
+      s_dot[tid] += s_dot[dualTid];
+   }
+   if (tid==0) { gdsr[bid] = s_dot[0]; }
+}
+
+static double cuVectorDot(const int N, const double *X, const double *Y)
+{
+   const DeviceVector x(X, N);
+   const DeviceVector y(Y, N);
+   static int dot_block_sz = 0;
+   const int tpb = MFEM_CUDA_BLOCKS;
+   const int blockSize = MFEM_CUDA_BLOCKS;
+   const int gridSize = (N+blockSize-1)/blockSize;
+   const int dot_sz = (N%tpb)==0? (N/tpb) : (1+N/tpb);
+   const int bytes = dot_sz*sizeof(double);
+   static double *h_dot = NULL;
+   if (!h_dot or dot_block_sz!=dot_sz)
+   {
+      if (h_dot) { free(h_dot); }
+      h_dot = (double*)calloc(dot_sz,sizeof(double));
+   }
+   static CUdeviceptr gdsr = (CUdeviceptr) NULL;
+   if (!gdsr or dot_block_sz!=dot_sz)
+   {
+      if (gdsr) { MFEM_CUDA_CHECK_DRV(::cuMemFree(gdsr)); }
+      MFEM_CUDA_CHECK_DRV(::cuMemAlloc(&gdsr,bytes));
+   }
+   if (dot_block_sz!=dot_sz)
+   {
+      dot_block_sz = dot_sz;
+   }
+   cuKernelDot<<<gridSize,blockSize>>>(N, (double*)gdsr, x, y);
+   MFEM_CUDA_CHECK_RT(cudaGetLastError());
+   MFEM_CUDA_CHECK_DRV(::cuMemcpy((CUdeviceptr)h_dot,(CUdeviceptr)gdsr,bytes));
+   double dot = 0.0;
+   for (int i = 0; i < dot_sz; i++) { dot += h_dot[i]; }
+   return dot;
+}
+#endif // MFEM_USE_CUDA
+
+double Min(const int N, const double *x)
+{
+   if (Device::Allows(Backend::CUDA_MASK))
+   {
+#ifdef MFEM_USE_CUDA
+      return cuVectorMin(N, x);
+#else
+      mfem_error("Using Min on device w/o support");
+#endif // MFEM_USE_CUDA
+   }
+   double min = std::numeric_limits<double>::infinity();
+   for (int i = 0; i < N; i++) { min = fmin(min, x[i]); }
+   return min;
+}
+
+double Dot(const int N, const double *x, const double *y)
+{
+   if (Device::Allows(Backend::CUDA_MASK))
+   {
+#ifdef MFEM_USE_CUDA
+      return cuVectorDot(N, x, y);
+#else
+      mfem_error("Using Dot on device w/o support");
+#endif // MFEM_USE_CUDA
+   }
+   double dot = 0.0;
+#ifdef MFEM_USE_LEGACY_OPENMP
+   #pragma omp parallel for reduction(+:dot)
+#endif
+   for (int i = 0; i < N; i++) { dot += x[i] * y[i]; }
+   return dot;
+}
+
 #ifdef MFEM_USE_SUNDIALS
 
 #ifndef SUNTRUE

linalg/vector.hpp

@@ -12,10 +12,9 @@
 #ifndef MFEM_VECTOR
 #define MFEM_VECTOR
 
-// Data type vector
-
 #include "../general/array.hpp"
 #include "../general/globals.hpp"
+#include "../general/mem_manager.hpp"
 #ifdef MFEM_USE_SUNDIALS
 #include <nvector/nvector_serial.h>
 #endif
@@ -70,6 +69,12 @@ public:
    Vector (double *_data, int _size)
    { data = _data; size = _size; allocsize = -size; }
 
+   /// Copies data from host to device
+   void Push() const;
+
+   /// Copies data from device to host
+   void Pull() const;
+
    /// Reads a vector from multiple files
    void Load (std::istream ** in, int np, int * dim);
 
@@ -107,7 +112,7 @@ public:
        @sa SetDataAndSize(). */
    void NewDataAndSize(double *d, int s)
    {
-      if (allocsize > 0) { delete [] data; }
+      if (allocsize > 0) { mfem::Delete(data); }
       SetDataAndSize(d, s);
    }
 
@@ -318,7 +323,7 @@ inline Vector::Vector (int s)
    if (s > 0)
    {
       allocsize = size = s;
-      data = new double[s];
+      data = mfem::New<double>(s);
    }
    else
    {
@@ -340,17 +345,17 @@ inline void Vector::SetSize(int s)
    }
    if (allocsize > 0)
    {
-      delete [] data;
+      mfem::Delete(data);
    }
    allocsize = size = s;
-   data = new double[s];
+   data = mfem::New<double>(s);
 }
 
 inline void Vector::Destroy()
 {
    if (allocsize > 0)
    {
-      delete [] data;
+      mfem::Delete(data);
    }
    allocsize = size = 0;
    data = NULL;
@@ -389,7 +394,7 @@ inline Vector::~Vector()
 {
    if (allocsize > 0)
    {
-      delete [] data;
+      mfem::Delete(data);
    }
 }
 
@@ -443,6 +448,28 @@ inline double InnerProduct(MPI_Comm comm, const Vector &x, const Vector &y)
 }
 #endif
 
+/// Kernel returning the minimum value in the array x of size N
+double Min(const int N, const double *x);
+
+/// Kernel of the inner product of arrays x and y of size N
+double Dot(const int N, const double *x, const double *y);
+
+/// Class for a simple Vector of size 3
+class Vector3
+{
+private:
+   double data[3];
+public:
+   Vector3() {}
+   Vector3(const double *x) { data[0]=x[0]; data[1]=x[1]; data[2]=x[2]; }
+   Vector3(const double x0, const double x1 = 0.0, const double x2 = 0.0)
+   { data[0]=x0; data[1]=x1; data[2]=x2; }
+   inline operator double* () { return data; }
+   inline operator const double* () const { return data; }
+   inline double& operator()(const int i) { return data[i]; }
+   inline const double& operator()(const int i) const { return data[i]; }
+};
+
 }
 
 #endif

makefile

@@ -26,6 +26,10 @@ MFEM makefile targets:
    make parallel
    make debug
    make pdebug
+   make cuda
+   make pcuda
+   make cudebug
+   make pcudebug
    make test/check
    make install
    make clean
@@ -55,6 +59,14 @@ make debug
    A shortcut to configure and build the serial debug version of the library.
 make pdebug
    A shortcut to configure and build the parallel debug version of the library.
+make cuda
+   A shortcut to configure and build the serial GPU/CUDA optimized version of the library.
+make pcuda
+   A shortcut to configure and build the parallel GPU/CUDA optimized version of the library.
+make cudebug
+   A shortcut to configure and build the serial GPU/CUDA debug version of the library.
+make pcudebug
+   A shortcut to configure and build the parallel GPU/CUDA debug version of the library.
 make test
    Verify the build by checking the results from running all examples, miniapps,
    and tests.
@@ -147,7 +159,7 @@ $(call mfem-info, BLD       = $(BLD))
 
 # Include $(CONFIG_MK) unless some of the $(SKIP_INCLUDE_TARGETS) are given
 SKIP_INCLUDE_TARGETS = help config clean distclean serial parallel debug pdebug\
- style
+ cuda pcuda cudebug pcudebug style
 HAVE_SKIP_INCLUDE_TARGET = $(filter $(SKIP_INCLUDE_TARGETS),$(MAKECMDGOALS))
 ifeq (,$(HAVE_SKIP_INCLUDE_TARGET))
    $(call mfem-info, Including $(CONFIG_MK))
@@ -182,30 +194,43 @@ CXXFLAGS ?= $(OPTIM_FLAGS)
 
 # MPI configuration
 ifneq ($(MFEM_USE_MPI),YES)
-   MFEM_CXX ?= $(CXX)
+   CXX_OR_MPICXX = $(CXX)
    PKGS_NEED_MPI = SUPERLU STRUMPACK PETSC PUMI
    $(foreach mpidep,$(PKGS_NEED_MPI),$(if $(MFEM_USE_$(mpidep):NO=),\
      $(warning *** [MPI is OFF] setting MFEM_USE_$(mpidep) = NO)\
      $(eval override MFEM_USE_$(mpidep)=NO),))
 else
-   MFEM_CXX ?= $(MPICXX)
+   CXX_OR_MPICXX = $(MPICXX)
    INCFLAGS += $(HYPRE_OPT)
    ALL_LIBS += $(HYPRE_LIB)
 endif
 
+# CUDA configuration
+ifneq ($(MFEM_USE_CUDA),YES)
+   MFEM_CXX ?= $(CXX_OR_MPICXX)
+   XCOMPILER = $(CXX_XCOMPILER)
+   XLINKER   = $(CXX_XLINKER)
+else
+   MFEM_CXX ?= $(CUDA_CXX)
+   CXXFLAGS += $(CUDA_FLAGS) -ccbin $(CXX_OR_MPICXX)
+   XCOMPILER = $(CUDA_XCOMPILER)
+   XLINKER   = $(CUDA_XLINKER)
+   # CUDA_OPT and CUDA_LIB are added below
+endif
+
 DEP_CXX ?= $(MFEM_CXX)
 
 # Check OpenMP configuration
-ifeq ($(MFEM_USE_OPENMP),YES)
+ifeq ($(MFEM_USE_LEGACY_OPENMP),YES)
    MFEM_THREAD_SAFE ?= YES
    ifneq ($(MFEM_THREAD_SAFE),YES)
-      $(error Incompatible config: MFEM_USE_OPENMP requires MFEM_THREAD_SAFE)
+      $(error Incompatible config: MFEM_USE_LEGACY_OPENMP requires MFEM_THREAD_SAFE)
    endif
 endif
 
 # List of MFEM dependencies, that require the *_LIB variable to be non-empty
 MFEM_REQ_LIB_DEPS = SUPERLU METIS CONDUIT SIDRE LAPACK SUNDIALS MESQUITE\
- SUITESPARSE STRUMPACK GECKO GNUTLS NETCDF PETSC MPFR PUMI
+ SUITESPARSE STRUMPACK GECKO GNUTLS NETCDF PETSC MPFR PUMI CUDA OCCA RAJA
 PETSC_ERROR_MSG = $(if $(PETSC_FOUND),,. PETSC config not found: $(PETSC_VARS))
 
 define mfem_check_dependency
@@ -223,6 +248,9 @@ endif
 # List of MFEM dependencies, processed below
 MFEM_DEPENDENCIES = $(MFEM_REQ_LIB_DEPS) LIBUNWIND OPENMP
 
+# List of deprecated MFEM dependencies, processed below
+MFEM_LEGACY_DEPENDENCIES = OPENMP
+
 # Macro for adding dependencies
 define mfem_add_dependency
 ifeq ($(MFEM_USE_$(1)),YES)
@@ -231,8 +259,18 @@ ifeq ($(MFEM_USE_$(1)),YES)
 endif
 endef
 
+# Macro for adding legacy dependencies
+define mfem_add_legacy_dependency
+ifeq ($(MFEM_USE_LEGACY_$(1)),YES)
+   INCFLAGS += $($(1)_OPT)
+   ALL_LIBS += $($(1)_LIB)
+endif
+endef
+
 # Process dependencies
 $(foreach dep,$(MFEM_DEPENDENCIES),$(eval $(call mfem_add_dependency,$(dep))))
+$(foreach dep,$(MFEM_LEGACY_DEPENDENCIES),$(eval $(call \
+   mfem_add_legacy_dependency,$(dep))))
 
 # Timer option
 ifeq ($(MFEM_TIMER_TYPE),2)
@@ -249,10 +287,12 @@ endif
 MFEM_DEFINES = MFEM_VERSION MFEM_VERSION_STRING MFEM_GIT_STRING MFEM_USE_MPI\
  MFEM_USE_METIS MFEM_USE_METIS_5 MFEM_DEBUG MFEM_USE_EXCEPTIONS\
  MFEM_USE_GZSTREAM MFEM_USE_LIBUNWIND MFEM_USE_LAPACK MFEM_THREAD_SAFE\
- MFEM_USE_OPENMP MFEM_USE_MEMALLOC MFEM_TIMER_TYPE MFEM_USE_SUNDIALS\
- MFEM_USE_MESQUITE MFEM_USE_SUITESPARSE MFEM_USE_GECKO MFEM_USE_SUPERLU\
- MFEM_USE_STRUMPACK MFEM_USE_GNUTLS MFEM_USE_NETCDF MFEM_USE_PETSC\
- MFEM_USE_MPFR MFEM_USE_SIDRE MFEM_USE_CONDUIT MFEM_USE_PUMI
+ MFEM_USE_OPENMP MFEM_USE_LEGACY_OPENMP MFEM_USE_MEMALLOC MFEM_TIMER_TYPE\
+ MFEM_USE_SUNDIALS MFEM_USE_MESQUITE MFEM_USE_SUITESPARSE MFEM_USE_GECKO\
+ MFEM_USE_SUPERLU MFEM_USE_STRUMPACK MFEM_USE_GNUTLS MFEM_USE_NETCDF\
+ MFEM_USE_PETSC MFEM_USE_MPFR MFEM_USE_SIDRE MFEM_USE_CONDUIT MFEM_USE_PUMI\
+ MFEM_USE_CUDA MFEM_USE_OCCA MFEM_USE_MM MFEM_USE_RAJA MFEM_SOURCE_DIR\
+ MFEM_INSTALL_DIR
 
 # List of makefile variables that will be written to config.mk:
 MFEM_CONFIG_VARS = MFEM_CXX MFEM_CPPFLAGS MFEM_CXXFLAGS MFEM_INC_DIR\
@@ -279,6 +319,9 @@ MFEM_TEST_MK   ?= @MFEM_DIR@/config/test.mk
 # Use "\n" (interpreted by sed) to add a newline.
 MFEM_CONFIG_EXTRA ?= $(if $(BUILD_DIR_DEF),MFEM_BUILD_DIR ?= @MFEM_DIR@,)
 
+MFEM_SOURCE_DIR  := $(MFEM_REAL_DIR)
+MFEM_INSTALL_DIR := $(BUILD_REAL_DIR)
+
 # If we have 'config' target, export variables used by config/makefile
 ifneq (,$(filter config,$(MAKECMDGOALS)))
    export $(MFEM_DEFINES) MFEM_DEFINES $(MFEM_CONFIG_VARS) MFEM_CONFIG_VARS
@@ -317,7 +360,8 @@ RELSRC_FILES = $(patsubst $(SRC)%,%,$(SOURCE_FILES))
 OBJECT_FILES = $(patsubst $(SRC)%,$(BLD)%,$(SOURCE_FILES:.cpp=.o))
 
 .PHONY: lib all clean distclean install config status info deps serial parallel\
- debug pdebug style check test unittest
+	debug pdebug cuda pcuda cudebug pcudebug style check test unittest\
+	deprecation-warnings
 
 .SUFFIXES:
 .SUFFIXES: .cpp .o
@@ -354,26 +398,38 @@ doc:
 $(BLD)libmfem.a: $(OBJECT_FILES)
 	$(AR) $(ARFLAGS) $(@) $(OBJECT_FILES)
 	$(RANLIB) $(@)
+	@$(MAKE) deprecation-warnings
 
 $(BLD)libmfem.$(SO_EXT): $(BLD)libmfem.$(SO_VER)
 	cd $(@D) && ln -sf $(<F) $(@F)
+	@$(MAKE) deprecation-warnings
 
 # If some of the external libraries are build without -fPIC, linking shared MFEM
 # library may fail. In such cases, one may set EXT_LIBS on the command line.
 EXT_LIBS = $(MFEM_EXT_LIBS)
 $(BLD)libmfem.$(SO_VER): $(OBJECT_FILES)
-	$(MFEM_CXX) $(MFEM_BUILD_FLAGS) $(BUILD_SOFLAGS) $(OBJECT_FILES) \
+	$(MFEM_CXX) $(MFEM_LINK_FLAGS) $(BUILD_SOFLAGS) $(OBJECT_FILES) \
 	   $(EXT_LIBS) -o $(@)
 
-serial debug:    M_MPI=NO
-parallel pdebug: M_MPI=YES
-serial parallel: M_DBG=NO
-debug pdebug:    M_DBG=YES
+# Shortcut targets options
+serial parallel debug pdebug:   M_MM=NO
+serial debug cuda cudebug:      M_MPI=NO
+parallel pdebug pcuda pcudebug: M_MPI=YES
+serial parallel cuda pcuda:     M_DBG=NO
+debug pdebug cudebug pcudebug:  M_DBG=YES
+cuda pcuda cudebug pcudebug:    M_CUDA=YES
+cuda pcuda cudebug pcudebug:    M_MM=YES
+
 serial parallel debug pdebug:
 	$(MAKE) -f $(THIS_MK) config MFEM_USE_MPI=$(M_MPI) MFEM_DEBUG=$(M_DBG) \
 	   $(MAKEOVERRIDES_SAVE)
 	$(MAKE) $(MAKEOVERRIDES_SAVE)
 
+cuda pcuda cudebug pcudebug:
+	$(MAKE) -f $(THIS_MK) config MFEM_USE_MPI=$(M_MPI) MFEM_DEBUG=$(M_DBG) \
+	   MFEM_USE_CUDA=$(M_CUDA) MFEM_USE_MM=$(M_MM) $(MAKEOVERRIDES_SAVE)
+	$(MAKE) $(MAKEOVERRIDES_SAVE)
+
 deps:
 	rm -f $(BLD)deps.mk
 	for i in $(RELSRC_FILES:.cpp=); do \
@@ -442,7 +498,8 @@ install: $(if $(static),$(BLD)libmfem.a) $(if $(shared),$(BLD)libmfem.$(SO_EXT))
 # install remaining includes in each subdirectory
 	for dir in $(DIRS); do \
 	   mkdir -p $(PREFIX_INC)/mfem/$$dir && \
-	   $(INSTALL) -m 640 $(SRC)$$dir/*.hpp $(PREFIX_INC)/mfem/$$dir; done
+	   $(INSTALL) -m 640 $(SRC)$$dir/*.hpp $(SRC)$$dir/*.okl $(PREFIX_INC)/mfem/$$dir; \
+	done
 # install config.mk in $(PREFIX_SHARE)
 	mkdir -p $(PREFIX_SHARE)
 	$(MAKE) -C $(BLD)config config-mk CONFIG_MK=config-install.mk
@@ -490,52 +547,58 @@ help:
 	@true
 
 status info:
-	$(info MFEM_VERSION         = $(MFEM_VERSION) [v$(MFEM_VERSION_STRING)])
-	$(info MFEM_GIT_STRING      = $(MFEM_GIT_STRING))
-	$(info MFEM_USE_MPI         = $(MFEM_USE_MPI))
-	$(info MFEM_USE_METIS       = $(MFEM_USE_METIS))
-	$(info MFEM_USE_METIS_5     = $(MFEM_USE_METIS_5))
-	$(info MFEM_DEBUG           = $(MFEM_DEBUG))
-	$(info MFEM_USE_EXCEPTIONS  = $(MFEM_USE_EXCEPTIONS))
-	$(info MFEM_USE_GZSTREAM    = $(MFEM_USE_GZSTREAM))
-	$(info MFEM_USE_LIBUNWIND   = $(MFEM_USE_LIBUNWIND))
-	$(info MFEM_USE_LAPACK      = $(MFEM_USE_LAPACK))
-	$(info MFEM_THREAD_SAFE     = $(MFEM_THREAD_SAFE))
-	$(info MFEM_USE_OPENMP      = $(MFEM_USE_OPENMP))
-	$(info MFEM_USE_MEMALLOC    = $(MFEM_USE_MEMALLOC))
-	$(info MFEM_TIMER_TYPE      = $(MFEM_TIMER_TYPE))
-	$(info MFEM_USE_SUNDIALS    = $(MFEM_USE_SUNDIALS))
-	$(info MFEM_USE_MESQUITE    = $(MFEM_USE_MESQUITE))
-	$(info MFEM_USE_SUITESPARSE = $(MFEM_USE_SUITESPARSE))
-	$(info MFEM_USE_SUPERLU     = $(MFEM_USE_SUPERLU))
-	$(info MFEM_USE_STRUMPACK   = $(MFEM_USE_STRUMPACK))
-	$(info MFEM_USE_GECKO       = $(MFEM_USE_GECKO))
-	$(info MFEM_USE_GNUTLS      = $(MFEM_USE_GNUTLS))
-	$(info MFEM_USE_NETCDF      = $(MFEM_USE_NETCDF))
-	$(info MFEM_USE_PETSC       = $(MFEM_USE_PETSC))
-	$(info MFEM_USE_MPFR        = $(MFEM_USE_MPFR))
-	$(info MFEM_USE_SIDRE       = $(MFEM_USE_SIDRE))
-	$(info MFEM_USE_CONDUIT     = $(MFEM_USE_CONDUIT))
-	$(info MFEM_USE_PUMI        = $(MFEM_USE_PUMI))
-	$(info MFEM_CXX             = $(value MFEM_CXX))
-	$(info MFEM_CPPFLAGS        = $(value MFEM_CPPFLAGS))
-	$(info MFEM_CXXFLAGS        = $(value MFEM_CXXFLAGS))
-	$(info MFEM_TPLFLAGS        = $(value MFEM_TPLFLAGS))
-	$(info MFEM_INCFLAGS        = $(value MFEM_INCFLAGS))
-	$(info MFEM_FLAGS           = $(value MFEM_FLAGS))
-	$(info MFEM_EXT_LIBS        = $(value MFEM_EXT_LIBS))
-	$(info MFEM_LIBS            = $(value MFEM_LIBS))
-	$(info MFEM_LIB_FILE        = $(value MFEM_LIB_FILE))
-	$(info MFEM_BUILD_TAG       = $(value MFEM_BUILD_TAG))
-	$(info MFEM_PREFIX          = $(value MFEM_PREFIX))
-	$(info MFEM_INC_DIR         = $(value MFEM_INC_DIR))
-	$(info MFEM_LIB_DIR         = $(value MFEM_LIB_DIR))
-	$(info MFEM_STATIC          = $(MFEM_STATIC))
-	$(info MFEM_SHARED          = $(MFEM_SHARED))
-	$(info MFEM_BUILD_DIR       = $(MFEM_BUILD_DIR))
-	$(info MFEM_MPIEXEC         = $(MFEM_MPIEXEC))
-	$(info MFEM_MPIEXEC_NP      = $(MFEM_MPIEXEC_NP))
-	$(info MFEM_MPI_NP          = $(MFEM_MPI_NP))
+	$(info MFEM_VERSION           = $(MFEM_VERSION) [v$(MFEM_VERSION_STRING)])
+	$(info MFEM_GIT_STRING        = $(MFEM_GIT_STRING))
+	$(info MFEM_USE_MPI           = $(MFEM_USE_MPI))
+	$(info MFEM_USE_METIS         = $(MFEM_USE_METIS))
+	$(info MFEM_USE_METIS_5       = $(MFEM_USE_METIS_5))
+	$(info MFEM_DEBUG             = $(MFEM_DEBUG))
+	$(info MFEM_USE_EXCEPTIONS    = $(MFEM_USE_EXCEPTIONS))
+	$(info MFEM_USE_GZSTREAM      = $(MFEM_USE_GZSTREAM))
+	$(info MFEM_USE_LIBUNWIND     = $(MFEM_USE_LIBUNWIND))
+	$(info MFEM_USE_LAPACK        = $(MFEM_USE_LAPACK))
+	$(info MFEM_THREAD_SAFE       = $(MFEM_THREAD_SAFE))
+	$(info MFEM_USE_OPENMP        = $(MFEM_USE_OPENMP))
+	$(info MFEM_USE_LEGACY_OPENMP = $(MFEM_USE_LEGACY_OPENMP))
+	$(info MFEM_USE_MEMALLOC      = $(MFEM_USE_MEMALLOC))
+	$(info MFEM_TIMER_TYPE        = $(MFEM_TIMER_TYPE))
+	$(info MFEM_USE_SUNDIALS      = $(MFEM_USE_SUNDIALS))
+	$(info MFEM_USE_MESQUITE      = $(MFEM_USE_MESQUITE))
+	$(info MFEM_USE_SUITESPARSE   = $(MFEM_USE_SUITESPARSE))
+	$(info MFEM_USE_SUPERLU       = $(MFEM_USE_SUPERLU))
+	$(info MFEM_USE_STRUMPACK     = $(MFEM_USE_STRUMPACK))
+	$(info MFEM_USE_GECKO         = $(MFEM_USE_GECKO))
+	$(info MFEM_USE_GNUTLS        = $(MFEM_USE_GNUTLS))
+	$(info MFEM_USE_NETCDF        = $(MFEM_USE_NETCDF))
+	$(info MFEM_USE_PETSC         = $(MFEM_USE_PETSC))
+	$(info MFEM_USE_MPFR          = $(MFEM_USE_MPFR))
+	$(info MFEM_USE_SIDRE         = $(MFEM_USE_SIDRE))
+	$(info MFEM_USE_CONDUIT       = $(MFEM_USE_CONDUIT))
+	$(info MFEM_USE_PUMI          = $(MFEM_USE_PUMI))
+	$(info MFEM_USE_CUDA          = $(MFEM_USE_CUDA))
+	$(info MFEM_USE_RAJA          = $(MFEM_USE_RAJA))
+	$(info MFEM_USE_OCCA          = $(MFEM_USE_OCCA))
+	$(info MFEM_USE_MM            = $(MFEM_USE_MM))
+	$(info MFEM_CXX               = $(value MFEM_CXX))
+	$(info MFEM_CPPFLAGS          = $(value MFEM_CPPFLAGS))
+	$(info MFEM_CXXFLAGS          = $(value MFEM_CXXFLAGS))
+	$(info MFEM_TPLFLAGS          = $(value MFEM_TPLFLAGS))
+	$(info MFEM_INCFLAGS          = $(value MFEM_INCFLAGS))
+	$(info MFEM_FLAGS             = $(value MFEM_FLAGS))
+	$(info MFEM_LINK_FLAGS        = $(value MFEM_LINK_FLAGS))
+	$(info MFEM_EXT_LIBS          = $(value MFEM_EXT_LIBS))
+	$(info MFEM_LIBS              = $(value MFEM_LIBS))
+	$(info MFEM_LIB_FILE          = $(value MFEM_LIB_FILE))
+	$(info MFEM_BUILD_TAG         = $(value MFEM_BUILD_TAG))
+	$(info MFEM_PREFIX            = $(value MFEM_PREFIX))
+	$(info MFEM_INC_DIR           = $(value MFEM_INC_DIR))
+	$(info MFEM_LIB_DIR           = $(value MFEM_LIB_DIR))
+	$(info MFEM_STATIC            = $(MFEM_STATIC))
+	$(info MFEM_SHARED            = $(MFEM_SHARED))
+	$(info MFEM_BUILD_DIR         = $(MFEM_BUILD_DIR))
+	$(info MFEM_MPIEXEC           = $(MFEM_MPIEXEC))
+	$(info MFEM_MPIEXEC_NP        = $(MFEM_MPIEXEC_NP))
+	$(info MFEM_MPI_NP            = $(MFEM_MPI_NP))
 	@true
 
 ASTYLE = astyle --options=$(SRC)config/mfem.astylerc
@@ -543,6 +606,20 @@ FORMAT_FILES = $(foreach dir,$(DIRS) $(EM_DIRS) config,"$(dir)/*.?pp")
 FORMAT_FILES += "tests/unit/*.cpp"
 FORMAT_FILES += $(foreach dir,$(DIRS),"tests/unit/$(dir)/*.?pp")
 
+DEPRECATION_WARNING := \
+"This feature is planned for removal in the next release."\
+"Please open an issue at github.com/mfem/mfem/issues if you depend on it."
+deprecation-warnings:
+	@if [ -t 1 ]; then\
+	  red="\033[0;31m";\
+	  yellow="\033[0;33m";\
+	  end="\033[0m";\
+	fi;\
+	if [ $(MFEM_USE_LEGACY_OPENMP) = YES ]; then\
+	  printf $$red"[MFEM_USE_LEGACY_OPENMP]"$$end": "$$yellow"%s"$$end"\n"\
+	  $(DEPRECATION_WARNING);\
+	fi
+
 style:
 	@if ! $(ASTYLE) $(FORMAT_FILES) | grep Formatted; then\
 	   echo "No source files were changed.";\

mesh/mesh.cpp

@@ -33,6 +33,7 @@
 
 // METIS 4 prototypes
 #if defined(MFEM_USE_METIS) && !defined(MFEM_USE_METIS_5)
+typedef int idx_t;
 typedef int idxtype;
 extern "C" {
    void METIS_PartGraphRecursive(int*, idxtype*, idxtype*, idxtype*, idxtype*,
@@ -418,27 +419,51 @@ void Mesh::GetBdrElementTransformation(int i, IsoparametricTransformation* ElTr)
 {
    ElTr->Attribute = GetBdrAttribute(i);
    ElTr->ElementNo = i; // boundary element number
+   DenseMatrix &pm = ElTr->GetPointMat();
    if (Nodes == NULL)
    {
-      GetBdrPointMatrix(i, ElTr->GetPointMat());
-      ElTr->SetFE(
-         GetTransformationFEforElementType(GetBdrElementType(i)));
+      GetBdrPointMatrix(i, pm);
+      ElTr->SetFE(GetTransformationFEforElementType(GetBdrElementType(i)));
    }
    else
    {
-      DenseMatrix &pm = ElTr->GetPointMat();
-      Array<int> vdofs;
-      Nodes->FESpace()->GetBdrElementVDofs(i, vdofs);
-      int n = vdofs.Size()/spaceDim;
-      pm.SetSize(spaceDim, n);
-      for (int k = 0; k < spaceDim; k++)
+      const FiniteElement *bdr_el = Nodes->FESpace()->GetBE(i);
+      if (bdr_el)
       {
-         for (int j = 0; j < n; j++)
+         Array<int> vdofs;
+         Nodes->FESpace()->GetBdrElementVDofs(i, vdofs);
+         int n = vdofs.Size()/spaceDim;
+         pm.SetSize(spaceDim, n);
+         for (int k = 0; k < spaceDim; k++)
          {
-            pm(k,j) = (*Nodes)(vdofs[n*k+j]);
+            for (int j = 0; j < n; j++)
+            {
+               pm(k,j) = (*Nodes)(vdofs[n*k+j]);
+            }
          }
+         ElTr->SetFE(bdr_el);
+      }
+      else // L2 Nodes (e.g., periodic mesh)
+      {
+         int elem_id, face_info;
+         GetBdrElementAdjacentElement(i, elem_id, face_info);
+
+         GetLocalFaceTransformation(GetBdrElementType(i),
+                                    GetElementType(elem_id),
+                                    FaceElemTr.Loc1.Transf, face_info);
+         // NOTE: FaceElemTr.Loc1 is overwritten here -- used as a temporary
+
+         const FiniteElement *face_el =
+            Nodes->FESpace()->GetTraceElement(elem_id,
+                                              GetBdrElementBaseGeometry(i));
+
+         IntegrationRule eir(face_el->GetDof());
+         FaceElemTr.Loc1.Transform(face_el->GetNodes(), eir);
+         // 'Transformation' is not used
+         Nodes->GetVectorValues(Transformation, eir, pm);
+
+         ElTr->SetFE(face_el);
       }
-      ElTr->SetFE(Nodes->FESpace()->GetBE(i));
    }
    ElTr->FinalizeTransformation();
 }
@@ -1284,17 +1309,14 @@ void Mesh::FinalizeQuadMesh(int generate_edges, int refine,
 
 
 #ifdef MFEM_USE_GECKO
-void Mesh::GetGeckoElementReordering(Array<int> &ordering)
+void Mesh::GetGeckoElementReordering(Array<int> &ordering,
+                                     int iterations, int window,
+                                     int period, int seed)
 {
    Gecko::Graph graph;
 
-   // We will put some accesors in for these later
    Gecko::Functional *functional =
       new Gecko::FunctionalGeometric(); // ordering functional
-   unsigned int iterations = 1;         // number of V cycles
-   unsigned int window = 2;             // initial window size
-   unsigned int period = 1;             // iterations between window increment
-   unsigned int seed = 0;               // random number seed
 
    // Run through all the elements and insert the nodes in the graph for them
    for (int elemid = 0; elemid < GetNE(); ++elemid)
@@ -2115,7 +2137,7 @@ void Mesh::Finalize(bool refine, bool fix_orientation)
 }
 
 void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
-                  int generate_edges, double sx, double sy, double sz)
+                  double sx, double sy, double sz, bool sfc_ordering)
 {
    int x, y, z;
 
@@ -2158,31 +2180,58 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
 #define VTX(XC, YC, ZC) ((XC)+((YC)+(ZC)*(ny+1))*(nx+1))
 
    // Sets elements and the corresponding indices of vertices
-   for (z = 0; z < nz; z++)
+   if (sfc_ordering && type == Element::HEXAHEDRON)
    {
-      for (y = 0; y < ny; y++)
+      Array<int> sfc;
+      NCMesh::GridSfcOrdering3D(nx, ny, nz, sfc);
+      MFEM_VERIFY(sfc.Size() == 3*nx*ny*nz, "");
+
+      for (int k = 0; k < nx*ny*nz; k++)
       {
-         for (x = 0; x < nx; x++)
+         x = sfc[3*k + 0];
+         y = sfc[3*k + 1];
+         z = sfc[3*k + 2];
+
+         ind[0] = VTX(x  , y  , z  );
+         ind[1] = VTX(x+1, y  , z  );
+         ind[2] = VTX(x+1, y+1, z  );
+         ind[3] = VTX(x  , y+1, z  );
+         ind[4] = VTX(x  , y  , z+1);
+         ind[5] = VTX(x+1, y  , z+1);
+         ind[6] = VTX(x+1, y+1, z+1);
+         ind[7] = VTX(x  , y+1, z+1);
+
+         AddHex(ind, 1);
+      }
+   }
+   else
+   {
+      for (z = 0; z < nz; z++)
+      {
+         for (y = 0; y < ny; y++)
          {
-            ind[0] = VTX(x  , y  , z  );
-            ind[1] = VTX(x+1, y  , z  );
-            ind[2] = VTX(x+1, y+1, z  );
-            ind[3] = VTX(x  , y+1, z  );
-            ind[4] = VTX(x  , y  , z+1);
-            ind[5] = VTX(x+1, y  , z+1);
-            ind[6] = VTX(x+1, y+1, z+1);
-            ind[7] = VTX(x  , y+1, z+1);
-            if (type == Element::TETRAHEDRON)
+            for (x = 0; x < nx; x++)
             {
-               AddHexAsTets(ind, 1);
-            }
-            else if (type == Element::WEDGE)
-            {
-               AddHexAsWedges(ind, 1);
-            }
-            else
-            {
-               AddHex(ind, 1);
+               ind[0] = VTX(x  , y  , z  );
+               ind[1] = VTX(x+1, y  , z  );
+               ind[2] = VTX(x+1, y+1, z  );
+               ind[3] = VTX(x  , y+1, z  );
+               ind[4] = VTX(x  , y  , z+1);
+               ind[5] = VTX(x+1, y  , z+1);
+               ind[6] = VTX(x+1, y+1, z+1);
+               ind[7] = VTX(x  , y+1, z+1);
+               if (type == Element::TETRAHEDRON)
+               {
+                  AddHexAsTets(ind, 1);
+               }
+               else if (type == Element::WEDGE)
+               {
+                  AddHexAsWedges(ind, 1);
+               }
+               else
+               {
+                  AddHex(ind, 1);
+               }
             }
          }
       }
@@ -2191,6 +2240,7 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
    // Sets boundary elements and the corresponding indices of vertices
    // bottom, bdr. attribute 1
    for (y = 0; y < ny; y++)
+   {
       for (x = 0; x < nx; x++)
       {
          ind[0] = VTX(x  , y  , 0);
@@ -2210,8 +2260,10 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
             AddBdrQuad(ind, 1);
          }
       }
+   }
    // top, bdr. attribute 6
    for (y = 0; y < ny; y++)
+   {
       for (x = 0; x < nx; x++)
       {
          ind[0] = VTX(x  , y  , nz);
@@ -2231,8 +2283,10 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
             AddBdrQuad(ind, 6);
          }
       }
+   }
    // left, bdr. attribute 5
    for (z = 0; z < nz; z++)
+   {
       for (y = 0; y < ny; y++)
       {
          ind[0] = VTX(0  , y  , z  );
@@ -2248,8 +2302,10 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
             AddBdrQuad(ind, 5);
          }
       }
+   }
    // right, bdr. attribute 3
    for (z = 0; z < nz; z++)
+   {
       for (y = 0; y < ny; y++)
       {
          ind[0] = VTX(nx, y  , z  );
@@ -2265,8 +2321,10 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
             AddBdrQuad(ind, 3);
          }
       }
+   }
    // front, bdr. attribute 2
    for (x = 0; x < nx; x++)
+   {
       for (z = 0; z < nz; z++)
       {
          ind[0] = VTX(x  , 0, z  );
@@ -2282,8 +2340,10 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
             AddBdrQuad(ind, 2);
          }
       }
+   }
    // back, bdr. attribute 4
    for (x = 0; x < nx; x++)
+   {
       for (z = 0; z < nz; z++)
       {
          ind[0] = VTX(x  , ny, z  );
@@ -2299,6 +2359,9 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
             AddBdrQuad(ind, 4);
          }
       }
+   }
+
+#undef VTX
 
 #if 0
    ofstream test_stream("debug.mesh");
@@ -2311,8 +2374,9 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
    // Finalize(...) can be called after this method, if needed
 }
 
-void Mesh::Make2D(int nx, int ny, Element::Type type, int generate_edges,
-                  double sx, double sy)
+void Mesh::Make2D(int nx, int ny, Element::Type type,
+                  double sx, double sy,
+                  bool generate_edges, bool sfc_ordering)
 {
    int i, j, k;
 
@@ -2349,17 +2413,37 @@ void Mesh::Make2D(int nx, int ny, Element::Type type, int generate_edges,
       }
 
       // Sets elements and the corresponding indices of vertices
-      k = 0;
-      for (j = 0; j < ny; j++)
+      if (sfc_ordering)
       {
-         for (i = 0; i < nx; i++)
+         Array<int> sfc;
+         NCMesh::GridSfcOrdering2D(nx, ny, sfc);
+         MFEM_VERIFY(sfc.Size() == 2*nx*ny, "");
+
+         for (k = 0; k < nx*ny; k++)
          {
+            i = sfc[2*k + 0];
+            j = sfc[2*k + 1];
             ind[0] = i + j*(nx+1);
             ind[1] = i + 1 +j*(nx+1);
             ind[2] = i + 1 + (j+1)*(nx+1);
             ind[3] = i + (j+1)*(nx+1);
             elements[k] = new Quadrilateral(ind);
-            k++;
+         }
+      }
+      else
+      {
+         k = 0;
+         for (j = 0; j < ny; j++)
+         {
+            for (i = 0; i < nx; i++)
+            {
+               ind[0] = i + j*(nx+1);
+               ind[1] = i + 1 +j*(nx+1);
+               ind[2] = i + 1 + (j+1)*(nx+1);
+               ind[3] = i + (j+1)*(nx+1);
+               elements[k] = new Quadrilateral(ind);
+               k++;
+            }
          }
       }
 
@@ -3151,12 +3235,12 @@ Mesh::Mesh(Mesh *mesh_array[], int num_pieces)
 Mesh::Mesh(Mesh *orig_mesh, int ref_factor, int ref_type)
 {
    Dim = orig_mesh->Dimension();
-   MFEM_VERIFY(ref_factor > 1, "the refinement factor must be > 1");
+   MFEM_VERIFY(ref_factor >= 1, "the refinement factor must be >= 1");
    MFEM_VERIFY(ref_type == BasisType::ClosedUniform ||
                ref_type == BasisType::GaussLobatto, "invalid refinement type");
-   MFEM_VERIFY(Dim == 2 || Dim == 3,
-               "only implemented for Hexahedron and Quadrilateral elements in "
-               "2D/3D");
+   MFEM_VERIFY(Dim == 1 || Dim == 2 || Dim == 3,
+               "only implemented for Segment, Quadrilateral and Hexahedron "
+               "elements in 1D/2D/3D");
    MFEM_VERIFY(orig_mesh->GetNumGeometries(Dim) <= 1,
                "meshes with mixed elements are not supported");
 
@@ -3165,7 +3249,7 @@ Mesh::Mesh(Mesh *orig_mesh, int ref_factor, int ref_type)
    H1_FECollection rfec(ref_factor, Dim, ref_type);
    FiniteElementSpace rfes(orig_mesh, &rfec);
 
-   int r_bndr_factor = ref_factor * (Dim == 2 ? 1 : ref_factor);
+   int r_bndr_factor = pow(ref_factor, Dim - 1);
    int r_elem_factor = ref_factor * r_bndr_factor;
 
    int r_num_vert = rfes.GetNDofs();
@@ -3222,18 +3306,34 @@ Mesh::Mesh(Mesh *orig_mesh, int ref_factor, int ref_type)
 
       rfes.GetBdrElementDofs(el, rdofs);
       MFEM_ASSERT(rdofs.Size() == RG.RefPts.Size(), "");
-      const int *c2h_map = rfec.GetDofMap(geom);
-      for (int j = 0; j < RG.RefGeoms.Size()/nvert; j++)
+      if (Dim == 1)
       {
-         Element *elem = NewElement(geom);
-         elem->SetAttribute(attrib);
-         int *v = elem->GetVertices();
-         for (int k = 0; k < nvert; k++)
+         // Dim == 1 is a special case because the boundary elements are
+         // zero-dimensional points, and therefore don't have a DofMap
+         for (int j = 0; j < RG.RefGeoms.Size()/nvert; j++)
          {
-            int cid = RG.RefGeoms[k+nvert*j]; // local Cartesian index
-            v[k] = rdofs[c2h_map[cid]];
+            Element *elem = NewElement(geom);
+            elem->SetAttribute(attrib);
+            int *v = elem->GetVertices();
+            v[0] = rdofs[RG.RefGeoms[nvert*j]];
+            AddBdrElement(elem);
+         }
+      }
+      else
+      {
+         const int *c2h_map = rfec.GetDofMap(geom);
+         for (int j = 0; j < RG.RefGeoms.Size()/nvert; j++)
+         {
+            Element *elem = NewElement(geom);
+            elem->SetAttribute(attrib);
+            int *v = elem->GetVertices();
+            for (int k = 0; k < nvert; k++)
+            {
+               int cid = RG.RefGeoms[k+nvert*j]; // local Cartesian index
+               v[k] = rdofs[c2h_map[cid]];
+            }
+            AddBdrElement(elem);
          }
-         AddBdrElement(elem);
       }
    }
 
@@ -3491,6 +3591,12 @@ void Mesh::SetNodalFESpace(FiniteElementSpace *nfes)
    SetNodalGridFunction(nodes, true);
 }
 
+void Mesh::EnsureNodes()
+{
+   if (Nodes) { return; }
+   SetCurvature(1, false, -1, Ordering::byVDIM);
+}
+
 void Mesh::SetNodalGridFunction(GridFunction *nodes, bool make_owner)
 {
    GetNodes(*nodes);
@@ -4899,21 +5005,42 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
    }
    else
    {
-      int *I, *J, n;
+      idx_t *I, *J, n;
 #ifndef MFEM_USE_METIS_5
-      int wgtflag = 0;
-      int numflag = 0;
-      int options[5];
+      idx_t wgtflag = 0;
+      idx_t numflag = 0;
+      idx_t options[5];
 #else
-      int ncon = 1;
-      int err;
-      int options[40];
+      idx_t ncon = 1;
+      idx_t err;
+      idx_t options[40];
 #endif
-      int edgecut;
+      idx_t edgecut;
+
+      // In case METIS have been compiled with 64bit indices
+      bool freedata = false;
+      idx_t mparts = (idx_t) nparts;
+      idx_t *mpartitioning;
 
       n = NumOfElements;
-      I = el_to_el->GetI();
-      J = el_to_el->GetJ();
+      if (sizeof(idx_t) == sizeof(int))
+      {
+         I = (idx_t*) el_to_el->GetI();
+         J = (idx_t*) el_to_el->GetJ();
+         mpartitioning = (idx_t*) partitioning;
+      }
+      else
+      {
+         int *iI = el_to_el->GetI();
+         int *iJ = el_to_el->GetJ();
+         int m = iI[n];
+         I = new idx_t[n+1];
+         J = new idx_t[m];
+         for (int k = 0; k < n+1; k++) { I[k] = iI[k]; }
+         for (int k = 0; k < m; k++) { J[k] = iJ[k]; }
+         mpartitioning = new idx_t[n];
+         freedata = true;
+      }
 #ifndef MFEM_USE_METIS_5
       options[0] = 0;
 #else
@@ -4930,7 +5057,7 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
             // std::sort(J+I[i], J+I[i+1]);
 
             // Sort in decreasing order, as in previous versions of MFEM.
-            std::sort(J+I[i], J+I[i+1], std::greater<int>());
+            std::sort(J+I[i], J+I[i+1], std::greater<idx_t>());
          }
       }
 
@@ -4940,30 +5067,30 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
       {
 #ifndef MFEM_USE_METIS_5
          METIS_PartGraphRecursive(&n,
-                                  (idxtype *) I,
-                                  (idxtype *) J,
-                                  (idxtype *) NULL,
-                                  (idxtype *) NULL,
+                                  I,
+                                  J,
+                                  NULL,
+                                  NULL,
                                   &wgtflag,
                                   &numflag,
-                                  &nparts,
+                                  &mparts,
                                   options,
                                   &edgecut,
-                                  (idxtype *) partitioning);
+                                  mpartitioning);
 #else
          err = METIS_PartGraphRecursive(&n,
                                         &ncon,
                                         I,
                                         J,
-                                        (idx_t *) NULL,
-                                        (idx_t *) NULL,
-                                        (idx_t *) NULL,
-                                        &nparts,
-                                        (real_t *) NULL,
-                                        (real_t *) NULL,
+                                        NULL,
+                                        NULL,
+                                        NULL,
+                                        &mparts,
+                                        NULL,
+                                        NULL,
                                         options,
                                         &edgecut,
-                                        partitioning);
+                                        mpartitioning);
          if (err != 1)
             mfem_error("Mesh::GeneratePartitioning: "
                        " error in METIS_PartGraphRecursive!");
@@ -4976,30 +5103,30 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
       {
 #ifndef MFEM_USE_METIS_5
          METIS_PartGraphKway(&n,
-                             (idxtype *) I,
-                             (idxtype *) J,
-                             (idxtype *) NULL,
-                             (idxtype *) NULL,
+                             I,
+                             J,
+                             NULL,
+                             NULL,
                              &wgtflag,
                              &numflag,
-                             &nparts,
+                             &mparts,
                              options,
                              &edgecut,
-                             (idxtype *) partitioning);
+                             mpartitioning);
 #else
          err = METIS_PartGraphKway(&n,
                                    &ncon,
                                    I,
                                    J,
-                                   (idx_t *) NULL,
-                                   (idx_t *) NULL,
-                                   (idx_t *) NULL,
-                                   &nparts,
-                                   (real_t *) NULL,
-                                   (real_t *) NULL,
+                                   NULL,
+                                   NULL,
+                                   NULL,
+                                   &mparts,
+                                   NULL,
+                                   NULL,
                                    options,
                                    &edgecut,
-                                   partitioning);
+                                   mpartitioning);
          if (err != 1)
             mfem_error("Mesh::GeneratePartitioning: "
                        " error in METIS_PartGraphKway!");
@@ -5012,31 +5139,31 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
       {
 #ifndef MFEM_USE_METIS_5
          METIS_PartGraphVKway(&n,
-                              (idxtype *) I,
-                              (idxtype *) J,
-                              (idxtype *) NULL,
-                              (idxtype *) NULL,
+                              I,
+                              J,
+                              NULL,
+                              NULL,
                               &wgtflag,
                               &numflag,
-                              &nparts,
+                              &mparts,
                               options,
                               &edgecut,
-                              (idxtype *) partitioning);
+                              mpartitioning);
 #else
          options[METIS_OPTION_OBJTYPE] = METIS_OBJTYPE_VOL;
          err = METIS_PartGraphKway(&n,
                                    &ncon,
                                    I,
                                    J,
-                                   (idx_t *) NULL,
-                                   (idx_t *) NULL,
-                                   (idx_t *) NULL,
-                                   &nparts,
-                                   (real_t *) NULL,
-                                   (real_t *) NULL,
+                                   NULL,
+                                   NULL,
+                                   NULL,
+                                   &mparts,
+                                   NULL,
+                                   NULL,
                                    options,
                                    &edgecut,
-                                   partitioning);
+                                   mpartitioning);
          if (err != 1)
             mfem_error("Mesh::GeneratePartitioning: "
                        " error in METIS_PartGraphKway!");
@@ -5047,6 +5174,17 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
       mfem::out << "Mesh::GeneratePartitioning(...): edgecut = "
                 << edgecut << endl;
 #endif
+      nparts = (int) mparts;
+      if (mpartitioning != (idx_t*)partitioning)
+      {
+         for (int k = 0; k<NumOfElements; k++) { partitioning[k] = mpartitioning[k]; }
+      }
+      if (freedata)
+      {
+         delete[] I;
+         delete[] J;
+         delete[] mpartitioning;
+      }
    }
 
    if (el_to_el)
@@ -5690,6 +5828,7 @@ void Mesh::SetNodes(const Vector &node_coord)
 void Mesh::NewNodes(GridFunction &nodes, bool make_owner)
 {
    if (own_nodes) { delete Nodes; }
+   nodes.Pull();
    Nodes = &nodes;
    spaceDim = Nodes->FESpace()->GetVDim();
    own_nodes = (int)make_owner;

mesh/mesh.hpp

@@ -210,7 +210,7 @@ protected:
    void ReadVTKMesh(std::istream &input, int &curved, int &read_gf,
                     bool &finalize_topo);
    void ReadNURBSMesh(std::istream &input, int &curved, int &read_gf);
-   void ReadInlineMesh(std::istream &input, int generate_edges = 0);
+   void ReadInlineMesh(std::istream &input, bool generate_edges = false);
    void ReadGmshMesh(std::istream &input);
    /* Note NetCDF (optional library) is used for reading cubit files */
 #ifdef MFEM_USE_NETCDF
@@ -408,17 +408,20 @@ protected:
 
    /** Creates mesh for the parallelepiped [0,sx]x[0,sy]x[0,sz], divided into
        nx*ny*nz hexahedra if type=HEXAHEDRON or into 6*nx*ny*nz tetrahedrons if
-       type=TETRAHEDRON. If generate_edges = 0 (default) edges are not
-       generated, if 1 edges are generated. */
-   void Make3D(int nx, int ny, int nz, Element::Type type, int generate_edges,
-               double sx, double sy, double sz);
+       type=TETRAHEDRON. The parameter @a sfc_ordering controls how the elements
+       (when type=HEXAHEDRON) are ordered: true - use space-filling curve
+       ordering, or false - use lexicographic ordering. */
+   void Make3D(int nx, int ny, int nz, Element::Type type,
+               double sx, double sy, double sz, bool sfc_ordering);
 
    /** Creates mesh for the rectangle [0,sx]x[0,sy], divided into nx*ny
        quadrilaterals if type = QUADRILATERAL or into 2*nx*ny triangles if
        type = TRIANGLE. If generate_edges = 0 (default) edges are not generated,
-       if 1 edges are generated. */
-   void Make2D(int nx, int ny, Element::Type type, int generate_edges,
-               double sx, double sy);
+       if 1 edges are generated. The parameter @a sfc_ordering controls how the
+       elements (when type=QUADRILATERAL) are ordered: true - use space-filling
+       curve ordering, or false - use lexicographic ordering. */
+   void Make2D(int nx, int ny, Element::Type type, double sx, double sy,
+               bool generate_edges, bool sfc_ordering);
 
    /// Creates a 1D mesh for the interval [0,sx] divided into n equal intervals.
    void Make1D(int n, double sx = 1.0);
@@ -555,34 +558,46 @@ public:
    /** This is our integration with the Gecko library.  This will call the
        Gecko library to find an element ordering that will increase memory
        coherency by putting elements that are in physical proximity closer in
-       memory. */
-   void GetGeckoElementReordering(Array<int> &ordering);
+       memory. It can also be used to get a space-filling curve ordering for
+       ParNCMesh partitioning.
+       @param[out] ordering Output element ordering.
+       @param[in] iterations Number of V cycles (default 1).
+       @param[in] window Initial window size (default 2).
+       @param[in] period Iterations between window increment (default 1).
+       @param[in] seed Random number seed (default 0). */
+   void GetGeckoElementReordering(Array<int> &ordering,
+                                  int iterations = 1, int window = 2,
+                                  int period = 1, int seed = 0);
 #endif
 
    /** Rebuilds the mesh with a different order of elements.  The ordering
        vector maps the old element number to the new element number.  This also
-       reorders the vertices and nodes edges and faces along with the elements.  */
+       reorders the vertices and nodes edges and faces along with the elements. */
    void ReorderElements(const Array<int> &ordering, bool reorder_vertices = true);
 
    /** Creates mesh for the parallelepiped [0,sx]x[0,sy]x[0,sz], divided into
        nx*ny*nz hexahedra if type=HEXAHEDRON or into 6*nx*ny*nz tetrahedrons if
-       type=TETRAHEDRON. If generate_edges = 0 (default) edges are not
-       generated, if 1 edges are generated. */
-   Mesh(int nx, int ny, int nz, Element::Type type, int generate_edges = 0,
-        double sx = 1.0, double sy = 1.0, double sz = 1.0)
+       type=TETRAHEDRON. If sfc_ordering = true (default), elements are ordered
+       along a space-filling curve, instead of row by row and layer by layer.
+       The parameter @a generate_edges is ignored (for now, it is kept for
+       backward compatibility). */
+   Mesh(int nx, int ny, int nz, Element::Type type, bool generate_edges = false,
+        double sx = 1.0, double sy = 1.0, double sz = 1.0,
+        bool sfc_ordering = true)
    {
-      Make3D(nx, ny, nz, type, generate_edges, sx, sy, sz);
+      Make3D(nx, ny, nz, type, sx, sy, sz, sfc_ordering);
       Finalize(true); // refine = true
    }
 
    /** Creates mesh for the rectangle [0,sx]x[0,sy], divided into nx*ny
        quadrilaterals if type = QUADRILATERAL or into 2*nx*ny triangles if
        type = TRIANGLE. If generate_edges = 0 (default) edges are not generated,
-       if 1 edges are generated. */
-   Mesh(int nx, int ny, Element::Type type, int generate_edges = 0,
-        double sx = 1.0, double sy = 1.0)
+       if 1 edges are generated. If scf_ordering = true (default), elements are
+       ordered along a space-filling curve, instead of row by row. */
+   Mesh(int nx, int ny, Element::Type type, bool generate_edges = false,
+        double sx = 1.0, double sy = 1.0, bool sfc_ordering = true)
    {
-      Make2D(nx, ny, type, generate_edges, sx, sy);
+      Make2D(nx, ny, type, sx, sy, generate_edges, sfc_ordering);
       Finalize(true); // refine = true
    }
 
@@ -1020,6 +1035,10 @@ public:
    /** Return the FiniteElementSpace on which the current mesh nodes are
        defined or NULL if the mesh does not have nodes. */
    const FiniteElementSpace *GetNodalFESpace() const;
+   /** Make sure that the mesh has valid nodes, i.e. its geometry is described
+       by a vector finite element grid function (even if it is a low-order mesh
+       with straight edges). */
+   void EnsureNodes();
 
    /** Set the curvature of the mesh nodes using the given polynomial degree,
        'order', and optionally: discontinuous or continuous FE space, 'discont',

mesh/mesh_readers.cpp

@@ -731,7 +731,7 @@ void Mesh::ReadNURBSMesh(std::istream &input, int &curved, int &read_gf)
    }
 }
 
-void Mesh::ReadInlineMesh(std::istream &input, int generate_edges)
+void Mesh::ReadInlineMesh(std::istream &input, bool generate_edges)
 {
    // Initialize to negative numbers so that we know if they've been set.  We're
    // using Element::POINT as our flag, since we're not going to make a 0D mesh,
@@ -862,7 +862,7 @@ void Mesh::ReadInlineMesh(std::istream &input, int generate_edges)
                   << "   ny = " << ny << "\n"
                   << "   sx = " << sx << "\n"
                   << "   sy = " << sy << "\n");
-      Make2D(nx, ny, type, generate_edges, sx, sy);
+      Make2D(nx, ny, type, sx, sy, generate_edges, true);
    }
    else if (type == Element::TETRAHEDRON || type == Element::WEDGE ||
             type == Element::HEXAHEDRON)
@@ -877,7 +877,8 @@ void Mesh::ReadInlineMesh(std::istream &input, int generate_edges)
                   << "   sx = " << sx << "\n"
                   << "   sy = " << sy << "\n"
                   << "   sz = " << sz << "\n");
-      Make3D(nx, ny, nz, type, generate_edges, sx, sy, sz);
+      Make3D(nx, ny, nz, type, sx, sy, sz, true);
+      // TODO: maybe have an option in the file to control ordering?
    }
    else
    {

mesh/ncmesh.cpp

@@ -11,10 +11,12 @@
 
 #include "mesh_headers.hpp"
 #include "../fem/fem.hpp"
+#include "../general/sort_pairs.hpp"
 
 #include <string>
 #include <cmath>
 #include <climits> // INT_MAX
+#include <map>
 
 namespace mfem
 {
@@ -31,6 +33,118 @@ const DenseTensor &CoarseFineTransformations::GetPointMatrices(
    return pm_it->second;
 }
 
+namespace internal
+{
+
+// Used in CoarseFineTransformations::GetCoarseToFineMap() below.
+struct RefType
+{
+   Geometry::Type geom;
+   int num_children;
+   const Pair<int,int> *children;
+
+   RefType(Geometry::Type g, int n, const Pair<int,int> *c)
+      : geom(g), num_children(n), children(c) { }
+
+   bool operator<(const RefType &other) const
+   {
+      if (geom < other.geom) { return true; }
+      if (geom > other.geom) { return false; }
+      if (num_children < other.num_children) { return true; }
+      if (num_children > other.num_children) { return false; }
+      for (int i = 0; i < num_children; i++)
+      {
+         if (children[i].one < other.children[i].one) { return true; }
+         if (children[i].one > other.children[i].one) { return false; }
+      }
+      return false; // everything is equal
+   }
+};
+
+}
+
+void CoarseFineTransformations::GetCoarseToFineMap(
+   const mfem::Mesh &fine_mesh, Table &coarse_to_fine,
+   Array<int> &coarse_to_ref_type, Table &ref_type_to_matrix,
+   Array<mfem::Geometry::Type> &ref_type_to_geom) const
+{
+   const int fine_ne = embeddings.Size();
+   int coarse_ne = -1;
+   for (int i = 0; i < fine_ne; i++)
+   {
+      coarse_ne = std::max(coarse_ne, embeddings[i].parent);
+   }
+   coarse_ne++;
+
+   coarse_to_ref_type.SetSize(coarse_ne);
+   coarse_to_fine.SetDims(coarse_ne, fine_ne);
+
+   Array<int> cf_i(coarse_to_fine.GetI(), coarse_ne+1);
+   Array<Pair<int,int> > cf_j(fine_ne);
+   cf_i = 0;
+   for (int i = 0; i < fine_ne; i++)
+   {
+      cf_i[embeddings[i].parent+1]++;
+   }
+   cf_i.PartialSum();
+   MFEM_ASSERT(cf_i.Last() == cf_j.Size(), "internal error");
+   for (int i = 0; i < fine_ne; i++)
+   {
+      const Embedding &e = embeddings[i];
+      cf_j[cf_i[e.parent]].one = e.matrix; // used as sort key below
+      cf_j[cf_i[e.parent]].two = i;
+      cf_i[e.parent]++;
+   }
+   std::copy_backward(cf_i.begin(), cf_i.end()-1, cf_i.end());
+   cf_i[0] = 0;
+   for (int i = 0; i < coarse_ne; i++)
+   {
+      std::sort(&cf_j[cf_i[i]], cf_j.GetData() + cf_i[i+1]);
+   }
+   for (int i = 0; i < fine_ne; i++)
+   {
+      coarse_to_fine.GetJ()[i] = cf_j[i].two;
+   }
+
+   using internal::RefType;
+   using std::map;
+   using std::pair;
+
+   map<RefType,int> ref_type_map;
+   for (int i = 0; i < coarse_ne; i++)
+   {
+      const int num_children = cf_i[i+1]-cf_i[i];
+      MFEM_ASSERT(num_children > 0, "");
+      const int fine_el = cf_j[cf_i[i]].two;
+      // Assuming the coarse and the fine elements have the same geometry:
+      const Geometry::Type geom = fine_mesh.GetElementBaseGeometry(fine_el);
+      const RefType ref_type(geom, num_children, &cf_j[cf_i[i]]);
+      pair<map<RefType,int>::iterator,bool> res =
+         ref_type_map.insert(
+            pair<const RefType,int>(ref_type, (int)ref_type_map.size()));
+      coarse_to_ref_type[i] = res.first->second;
+   }
+   ref_type_to_matrix.MakeI((int)ref_type_map.size());
+   ref_type_to_geom.SetSize((int)ref_type_map.size());
+   for (map<RefType,int>::iterator it = ref_type_map.begin();
+        it != ref_type_map.end(); ++it)
+   {
+      ref_type_to_matrix.AddColumnsInRow(it->second, it->first.num_children);
+      ref_type_to_geom[it->second] = it->first.geom;
+   }
+   ref_type_to_matrix.MakeJ();
+   for (map<RefType,int>::iterator it = ref_type_map.begin();
+        it != ref_type_map.end(); ++it)
+   {
+      const RefType &rt = it->first;
+      for (int j = 0; j < rt.num_children; j++)
+      {
+         ref_type_to_matrix.AddConnection(it->second, rt.children[j].one);
+      }
+   }
+   ref_type_to_matrix.ShiftUpI();
+}
+
 NCMesh::GeomInfo NCMesh::GI[Geometry::NumGeom];
 
 NCMesh::GeomInfo& NCMesh::gi_hex  = NCMesh::GI[Geometry::CUBE];
@@ -120,8 +234,7 @@ NCMesh::NCMesh(const Mesh *mesh, std::istream *vertex_parents)
    }
 
    // create the NCMesh::Element struct for each Mesh element
-   root_count = mesh->GetNE();
-   for (int i = 0; i < root_count; i++)
+   for (int i = 0; i < mesh->GetNE(); i++)
    {
       const mfem::Element *elem = mesh->GetElement(i);
 
@@ -180,6 +293,11 @@ NCMesh::NCMesh(const Mesh *mesh, std::istream *vertex_parents)
       }
    }
 
+   if (!vertex_parents) // not loading mesh
+   {
+      InitRootState(mesh->GetNE());
+   }
+
    Update();
 }
 
@@ -190,9 +308,9 @@ NCMesh::NCMesh(const NCMesh &other)
    , nodes(other.nodes)
    , faces(other.faces)
    , elements(other.elements)
-   , root_count(other.root_count)
 {
    other.free_element_ids.Copy(free_element_ids);
+   other.root_state.Copy(root_state);
    other.top_vertex_pos.Copy(top_vertex_pos);
    Update();
 }
@@ -1128,6 +1246,53 @@ void NCMesh::Refine(const Array<Refinement>& refinements)
 
 //// Derefinement //////////////////////////////////////////////////////////////
 
+static int quad_deref_table[3][4 + 4] =
+{
+   { 0, 1, 1, 0, /**/ 1, 1, 0, 0 }, // 1 - X
+   { 0, 0, 1, 1, /**/ 0, 0, 1, 1 }, // 2 - Y
+   { 0, 1, 2, 3, /**/ 1, 1, 3, 3 }  // 3 - iso
+};
+static int hex_deref_table[7][8 + 6] =
+{
+   { 0, 1, 1, 0, 0, 1, 1, 0, /**/ 1, 1, 1, 0, 0, 0 }, // 1 - X
+   { 0, 0, 1, 1, 0, 0, 1, 1, /**/ 0, 0, 0, 1, 1, 1 }, // 2 - Y
+   { 0, 1, 2, 3, 0, 1, 2, 3, /**/ 1, 1, 1, 3, 3, 3 }, // 3 - XY
+   { 0, 0, 0, 0, 1, 1, 1, 1, /**/ 0, 0, 0, 1, 1, 1 }, // 4 - Z
+   { 0, 1, 1, 0, 3, 2, 2, 3, /**/ 1, 1, 1, 3, 3, 3 }, // 5 - XZ
+   { 0, 0, 1, 1, 2, 2, 3, 3, /**/ 0, 0, 0, 3, 3, 3 }, // 6 - YZ
+   { 0, 1, 2, 3, 4, 5, 6, 7, /**/ 1, 1, 1, 7, 7, 7 }  // 7 - iso
+};
+
+
+int NCMesh::RetrieveNode(const Element &el, int index)
+{
+   if (!el.ref_type) { return el.node[index]; }
+
+   // need to retrieve node from a child element (there is always a child
+   // that inherited the parent's corner under the same index)
+   int ch;
+   switch (el.geom)
+   {
+      case Geometry::CUBE:
+         ch = el.child[hex_deref_table[el.ref_type - 1][index]];
+         break;
+
+      case Geometry::SQUARE:
+         ch = el.child[quad_deref_table[el.ref_type - 1][index]];
+         break;
+
+      case Geometry::TRIANGLE:
+         ch = el.child[index];
+         break;
+
+      default:
+         ch = 0; // suppress compiler warning
+         MFEM_ABORT("Unsupported element geometry.");
+   }
+   return RetrieveNode(elements[ch], index);
+}
+
+
 void NCMesh::DerefineElement(int elem)
 {
    Element &el = elements[elem];
@@ -1149,23 +1314,14 @@ void NCMesh::DerefineElement(int elem)
    int fa[6];
    if (el.geom == Geometry::CUBE)
    {
-      const int table[7][8 + 6] =
-      {
-         { 0, 1, 1, 0, 0, 1, 1, 0, /**/ 1, 1, 1, 0, 0, 0 }, // 1 - X
-         { 0, 0, 1, 1, 0, 0, 1, 1, /**/ 0, 0, 0, 1, 1, 1 }, // 2 - Y
-         { 0, 1, 2, 3, 0, 1, 2, 3, /**/ 1, 1, 1, 3, 3, 3 }, // 3 - XY
-         { 0, 0, 0, 0, 1, 1, 1, 1, /**/ 0, 0, 0, 1, 1, 1 }, // 4 - Z
-         { 0, 1, 1, 0, 3, 2, 2, 3, /**/ 1, 1, 1, 3, 3, 3 }, // 5 - XZ
-         { 0, 0, 1, 1, 2, 2, 3, 3, /**/ 0, 0, 0, 3, 3, 3 }, // 6 - YZ
-         { 0, 1, 2, 3, 4, 5, 6, 7, /**/ 1, 1, 1, 7, 7, 7 }  // 7 - iso
-      };
       for (int i = 0; i < 8; i++)
       {
-         el.node[i] = elements[child[table[el.ref_type - 1][i]]].node[i];
+         Element &ch = elements[child[hex_deref_table[el.ref_type - 1][i]]];
+         el.node[i] = ch.node[i];
       }
       for (int i = 0; i < 6; i++)
       {
-         Element &ch = elements[child[table[el.ref_type - 1][i + 8]]];
+         Element &ch = elements[child[hex_deref_table[el.ref_type - 1][i + 8]]];
          const int* fv = gi_hex.faces[i];
          fa[i] = faces.Find(ch.node[fv[0]], ch.node[fv[1]],
                             ch.node[fv[2]], ch.node[fv[3]])->attribute;
@@ -1173,19 +1329,14 @@ void NCMesh::DerefineElement(int elem)
    }
    else if (el.geom == Geometry::SQUARE)
    {
-      const int table[3][4 + 4] =
-      {
-         { 0, 1, 1, 0, /**/ 1, 1, 0, 0 }, // 1 - X
-         { 0, 0, 1, 1, /**/ 0, 0, 1, 1 }, // 2 - Y
-         { 0, 1, 2, 3, /**/ 1, 1, 3, 3 }  // 3 - iso
-      };
       for (int i = 0; i < 4; i++)
       {
-         el.node[i] = elements[child[table[el.ref_type - 1][i]]].node[i];
+         Element &ch = elements[child[quad_deref_table[el.ref_type - 1][i]]];
+         el.node[i] = ch.node[i];
       }
       for (int i = 0; i < 4; i++)
       {
-         Element &ch = elements[child[table[el.ref_type - 1][i + 4]]];
+         Element &ch = elements[child[quad_deref_table[el.ref_type - 1][i + 4]]];
          const int* fv = gi_quad.faces[i];
          fa[i] = faces.Find(ch.node[fv[0]], ch.node[fv[1]],
                             ch.node[fv[2]], ch.node[fv[3]])->attribute;
@@ -1272,7 +1423,7 @@ const Table& NCMesh::GetDerefinementTable()
    Array<Connection> list;
    list.Reserve(leaf_elements.Size());
 
-   for (int i = 0; i < root_count; i++)
+   for (int i = 0; i < root_state.Size(); i++)
    {
       CollectDerefinements(i, list);
    }
@@ -1482,9 +1633,9 @@ void NCMesh::UpdateLeafElements()
 {
    // collect leaf elements from all roots
    leaf_elements.SetSize(0);
-   for (int i = 0; i < root_count; i++)
+   for (int i = 0; i < root_state.Size(); i++)
    {
-      CollectLeafElements(i, 0);
+      CollectLeafElements(i, root_state[i]);
       // TODO: root state should not always be 0, we need a precomputed array
       // with root element states to ensure continuity where possible, also
       // optimized ordering of the root elements themselves (Gecko?)
@@ -1501,6 +1652,69 @@ void NCMesh::AssignLeafIndices()
    }
 }
 
+void NCMesh::InitRootState(int root_count)
+{
+   root_state.SetSize(root_count);
+   root_state = 0;
+
+   char* node_order;
+   int nch;
+
+   switch (GetElementGeometry())
+   {
+      case Geometry::SQUARE:
+         nch = 4;
+         node_order = (char*) quad_hilbert_child_order;
+         break;
+
+      case Geometry::CUBE:
+         nch = 8;
+         node_order = (char*) hex_hilbert_child_order;
+         break;
+
+      default:
+         return; // do nothing, all states stay zero
+   }
+
+   int entry_node = -2;
+
+   // process the root element sequence
+   for (int i = 0; i < root_count; i++)
+   {
+      Element &el = elements[i];
+
+      int v_in = FindNodeExt(el, entry_node, false);
+      if (v_in < 0) { v_in = 0; }
+
+      // determine which nodes are shared with the next element
+      bool shared[8] = { 0, 0, 0, 0, 0, 0, 0, 0 };
+      if (i+1 < root_count)
+      {
+         Element &next = elements[i+1];
+         for (int j = 0; j < nch; j++)
+         {
+            int node = FindNodeExt(el, RetrieveNode(next, j), false);
+            if (node >= 0) { shared[node] = true; }
+         }
+      }
+
+      // select orientation that starts in v_in and exits in shared node
+      int state = Dim*v_in;
+      for (int j = 0; j < Dim; j++)
+      {
+         if (shared[(int) node_order[nch*(state + j) + nch-1]])
+         {
+            state += j;
+            break;
+         }
+      }
+
+      root_state[i] = state;
+
+      entry_node = RetrieveNode(el, node_order[nch*state + nch-1]);
+   }
+}
+
 mfem::Element* NCMesh::NewMeshElement(int geom) const
 {
    switch (geom)
@@ -1703,6 +1917,16 @@ int NCMesh::find_node(const Element &el, int node)
    return -1;
 }
 
+int NCMesh::FindNodeExt(const Element &el, int node, bool abort)
+{
+   for (int i = 0; i < GI[(int) el.geom].nv; i++)
+   {
+      if (RetrieveNode(el, i) == node) { return i; }
+   }
+   if (abort) { MFEM_ABORT("Node not found."); }
+   return -1;
+}
+
 int NCMesh::find_element_edge(const Element &el, int vn0, int vn1)
 {
    MFEM_ASSERT(!el.ref_type, "");
@@ -1939,7 +2163,7 @@ void NCMesh::BuildEdgeList()
    processed_edges = 0;
 
    Array<int> edge_element(nodes.NumIds());
-   Array<char> edge_local(nodes.NumIds());
+   Array<signed char> edge_local(nodes.NumIds());
    edge_local = -1;
 
    // visit edges of leaf elements
@@ -2198,7 +2422,7 @@ void NCMesh::CollectFaceVertices(int v0, int v1, int v2, int v3,
 void NCMesh::BuildElementToVertexTable()
 {
    int nrows = leaf_elements.Size();
-   int* I = new int[nrows + 1];
+   int* I = mfem::New<int>(nrows + 1);
    int** JJ = new int*[nrows];
 
    Array<int> indices;
@@ -2236,8 +2460,8 @@ void NCMesh::BuildElementToVertexTable()
       indices.Unique();
       int size = indices.Size();
       I[i] = size;
-      JJ[i] = new int[size];
-      memcpy(JJ[i], indices.GetData(), size * sizeof(int));
+      JJ[i] = mfem::New<int>(size);
+      std::memcpy(JJ[i], indices.GetData(), size * sizeof(int));
    }
 
    // finalize the I array of the table
@@ -2251,13 +2475,13 @@ void NCMesh::BuildElementToVertexTable()
    I[nrows] = nnz;
 
    // copy the temporarily stored rows into one J array
-   int *J = new int[nnz];
+   int *J = mfem::New<int>(nnz);
    nnz = 0;
    for (int i = 0; i < nrows; i++)
    {
       int cnt = I[i+1] - I[i];
-      memcpy(J+nnz, JJ[i], cnt * sizeof(int));
-      delete [] JJ[i];
+      std::memcpy(J+nnz, JJ[i], cnt * sizeof(int));
+      mfem::Delete(JJ[i]);
       nnz += cnt;
    }
 
@@ -2987,6 +3211,273 @@ void NCMesh::ClearTransforms()
 }
 
 
+//// SFC Ordering //////////////////////////////////////////////////////////////
+
+static int sgn(int x)
+{
+   return (x < 0) ? -1 : (x > 0) ? 1 : 0;
+}
+
+static void HilbertSfc2D(int x, int y, int ax, int ay, int bx, int by,
+                         Array<int> &coords)
+{
+   int w = std::abs(ax + ay);
+   int h = std::abs(bx + by);
+
+   int dax = sgn(ax), day = sgn(ay); // unit major direction ("right")
+   int dbx = sgn(bx), dby = sgn(by); // unit orthogonal direction ("up")
+
+   if (h == 1) // trivial row fill
+   {
+      for (int i = 0; i < w; i++, x += dax, y += day)
+      {
+         coords.Append(x);
+         coords.Append(y);
+      }
+      return;
+   }
+   if (w == 1) // trivial column fill
+   {
+      for (int i = 0; i < h; i++, x += dbx, y += dby)
+      {
+         coords.Append(x);
+         coords.Append(y);
+      }
+      return;
+   }
+
+   int ax2 = ax/2, ay2 = ay/2;
+   int bx2 = bx/2, by2 = by/2;
+
+   int w2 = std::abs(ax2 + ay2);
+   int h2 = std::abs(bx2 + by2);
+
+   if (2*w > 3*h) // long case: split in two parts only
+   {
+      if ((w2 & 0x1) && (w > 2))
+      {
+         ax2 += dax, ay2 += day; // prefer even steps
+      }
+
+      HilbertSfc2D(x, y, ax2, ay2, bx, by, coords);
+      HilbertSfc2D(x+ax2, y+ay2, ax-ax2, ay-ay2, bx, by, coords);
+   }
+   else // standard case: one step up, one long horizontal step, one step down
+   {
+      if ((h2 & 0x1) && (h > 2))
+      {
+         bx2 += dbx, by2 += dby; // prefer even steps
+      }
+
+      HilbertSfc2D(x, y, bx2, by2, ax2, ay2, coords);
+      HilbertSfc2D(x+bx2, y+by2, ax, ay, bx-bx2, by-by2, coords);
+      HilbertSfc2D(x+(ax-dax)+(bx2-dbx), y+(ay-day)+(by2-dby),
+                   -bx2, -by2, -(ax-ax2), -(ay-ay2), coords);
+   }
+}
+
+static void HilbertSfc3D(int x, int y, int z,
+                         int ax, int ay, int az,
+                         int bx, int by, int bz,
+                         int cx, int cy, int cz,
+                         Array<int> &coords)
+{
+   int w = std::abs(ax + ay + az);
+   int h = std::abs(bx + by + bz);
+   int d = std::abs(cx + cy + cz);
+
+   int dax = sgn(ax), day = sgn(ay), daz = sgn(az); // unit major dir ("right")
+   int dbx = sgn(bx), dby = sgn(by), dbz = sgn(bz); // unit ortho dir ("forward")
+   int dcx = sgn(cx), dcy = sgn(cy), dcz = sgn(cz); // unit ortho dir ("up")
+
+   // trivial row/column fills
+   if (h == 1 && d == 1)
+   {
+      for (int i = 0; i < w; i++, x += dax, y += day, z += daz)
+      {
+         coords.Append(x);
+         coords.Append(y);
+         coords.Append(z);
+      }
+      return;
+   }
+   if (w == 1 && d == 1)
+   {
+      for (int i = 0; i < h; i++, x += dbx, y += dby, z += dbz)
+      {
+         coords.Append(x);
+         coords.Append(y);
+         coords.Append(z);
+      }
+      return;
+   }
+   if (w == 1 && h == 1)
+   {
+      for (int i = 0; i < d; i++, x += dcx, y += dcy, z += dcz)
+      {
+         coords.Append(x);
+         coords.Append(y);
+         coords.Append(z);
+      }
+      return;
+   }
+
+   int ax2 = ax/2, ay2 = ay/2, az2 = az/2;
+   int bx2 = bx/2, by2 = by/2, bz2 = bz/2;
+   int cx2 = cx/2, cy2 = cy/2, cz2 = cz/2;
+
+   int w2 = std::abs(ax2 + ay2 + az2);
+   int h2 = std::abs(bx2 + by2 + bz2);
+   int d2 = std::abs(cx2 + cy2 + cz2);
+
+   // prefer even steps
+   if ((w2 & 0x1) && (w > 2))
+   {
+      ax2 += dax, ay2 += day, az2 += daz;
+   }
+   if ((h2 & 0x1) && (h > 2))
+   {
+      bx2 += dbx, by2 += dby, bz2 += dbz;
+   }
+   if ((d2 & 0x1) && (d > 2))
+   {
+      cx2 += dcx, cy2 += dcy, cz2 += dcz;
+   }
+
+   // wide case, split in w only
+   if ((2*w > 3*h) && (2*w > 3*d))
+   {
+      HilbertSfc3D(x, y, z,
+                   ax2, ay2, az2,
+                   bx, by, bz,
+                   cx, cy, cz, coords);
+
+      HilbertSfc3D(x+ax2, y+ay2, z+az2,
+                   ax-ax2, ay-ay2, az-az2,
+                   bx, by, bz,
+                   cx, cy, cz, coords);
+   }
+   // do not split in d
+   else if (3*h > 4*d)
+   {
+      HilbertSfc3D(x, y, z,
+                   bx2, by2, bz2,
+                   cx, cy, cz,
+                   ax2, ay2, az2, coords);
+
+      HilbertSfc3D(x+bx2, y+by2, z+bz2,
+                   ax, ay, az,
+                   bx-bx2, by-by2, bz-bz2,
+                   cx, cy, cz, coords);
+
+      HilbertSfc3D(x+(ax-dax)+(bx2-dbx),
+                   y+(ay-day)+(by2-dby),
+                   z+(az-daz)+(bz2-dbz),
+                   -bx2, -by2, -bz2,
+                   cx, cy, cz,
+                   -(ax-ax2), -(ay-ay2), -(az-az2), coords);
+   }
+   // do not split in h
+   else if (3*d > 4*h)
+   {
+      HilbertSfc3D(x, y, z,
+                   cx2, cy2, cz2,
+                   ax2, ay2, az2,
+                   bx, by, bz, coords);
+
+      HilbertSfc3D(x+cx2, y+cy2, z+cz2,
+                   ax, ay, az,
+                   bx, by, bz,
+                   cx-cx2, cy-cy2, cz-cz2, coords);
+
+      HilbertSfc3D(x+(ax-dax)+(cx2-dcx),
+                   y+(ay-day)+(cy2-dcy),
+                   z+(az-daz)+(cz2-dcz),
+                   -cx2, -cy2, -cz2,
+                   -(ax-ax2), -(ay-ay2), -(az-az2),
+                   bx, by, bz, coords);
+   }
+   // regular case, split in all w/h/d
+   else
+   {
+      HilbertSfc3D(x, y, z,
+                   bx2, by2, bz2,
+                   cx2, cy2, cz2,
+                   ax2, ay2, az2, coords);
+
+      HilbertSfc3D(x+bx2, y+by2, z+bz2,
+                   cx, cy, cz,
+                   ax2, ay2, az2,
+                   bx-bx2, by-by2, bz-bz2, coords);
+
+      HilbertSfc3D(x+(bx2-dbx)+(cx-dcx),
+                   y+(by2-dby)+(cy-dcy),
+                   z+(bz2-dbz)+(cz-dcz),
+                   ax, ay, az,
+                   -bx2, -by2, -bz2,
+                   -(cx-cx2), -(cy-cy2), -(cz-cz2), coords);
+
+      HilbertSfc3D(x+(ax-dax)+bx2+(cx-dcx),
+                   y+(ay-day)+by2+(cy-dcy),
+                   z+(az-daz)+bz2+(cz-dcz),
+                   -cx, -cy, -cz,
+                   -(ax-ax2), -(ay-ay2), -(az-az2),
+                   bx-bx2, by-by2, bz-bz2, coords);
+
+      HilbertSfc3D(x+(ax-dax)+(bx2-dbx),
+                   y+(ay-day)+(by2-dby),
+                   z+(az-daz)+(bz2-dbz),
+                   -bx2, -by2, -bz2,
+                   cx2, cy2, cz2,
+                   -(ax-ax2), -(ay-ay2), -(az-az2), coords);
+   }
+}
+
+void NCMesh::GridSfcOrdering2D(int width, int height, Array<int> &coords)
+{
+   coords.SetSize(0);
+   coords.Reserve(2*width*height);
+
+   if (width >= height)
+   {
+      HilbertSfc2D(0, 0, width, 0, 0, height, coords);
+   }
+   else
+   {
+      HilbertSfc2D(0, 0, 0, height, width, 0, coords);
+   }
+}
+
+void NCMesh::GridSfcOrdering3D(int width, int height, int depth,
+                               Array<int> &coords)
+{
+   coords.SetSize(0);
+   coords.Reserve(3*width*height*depth);
+
+   if (width >= height && width >= depth)
+   {
+      HilbertSfc3D(0, 0, 0,
+                   width, 0, 0,
+                   0, height, 0,
+                   0, 0, depth, coords);
+   }
+   else if (height >= width && height >= depth)
+   {
+      HilbertSfc3D(0, 0, 0,
+                   0, height, 0,
+                   width, 0, 0,
+                   0, 0, depth, coords);
+   }
+   else // depth >= width && depth >= height
+   {
+      HilbertSfc3D(0, 0, 0,
+                   0, 0, depth,
+                   width, 0, 0,
+                   0, height, 0, coords);
+   }
+}
+
+
 //// Utility ///////////////////////////////////////////////////////////////////
 
 void NCMesh::GetEdgeVertices(const MeshId &edge_id, int vert_index[2],
@@ -3420,7 +3911,7 @@ void NCMesh::PrintCoarseElements(std::ostream &out) const
 
    // print the hierarchy recursively
    int coarse_id = leaf_elements.Size();
-   for (int i = 0; i < root_count; i++)
+   for (int i = 0; i < root_state.Size(); i++)
    {
       PrintElements(out, i, coarse_id);
    }
@@ -3500,7 +3991,7 @@ void NCMesh::LoadCoarseElements(std::istream &input)
    index_map = -1;
 
    // copy roots, they need to be at the beginning of 'elements'
-   root_count = 0;
+   int root_count = 0;
    for (elem_iterator el = tmp_elements.begin(); el != tmp_elements.end(); ++el)
    {
       if (el->parent == -1)
@@ -3533,6 +4024,8 @@ void NCMesh::LoadCoarseElements(std::istream &input)
    // set the Iso flag (must be false if there are 3D aniso refinements)
    Iso = iso;
 
+   InitRootState(root_count);
+
    Update();
 }
 
@@ -3582,6 +4075,7 @@ long NCMesh::MemoryUsage() const
           faces.MemoryUsage() +
           elements.MemoryUsage() +
           free_element_ids.MemoryUsage() +
+          root_state.MemoryUsage() +
           top_vertex_pos.MemoryUsage() +
           leaf_elements.MemoryUsage() +
           vertex_nodeId.MemoryUsage() +
@@ -3604,6 +4098,7 @@ int NCMesh::PrintMemoryDetail() const
 
    mfem::out << elements.MemoryUsage() << " elements\n"
              << free_element_ids.MemoryUsage() << " free_element_ids\n"
+             << root_state.MemoryUsage() << " root_state\n"
              << top_vertex_pos.MemoryUsage() << " top_vertex_pos\n"
              << leaf_elements.MemoryUsage() << " leaf_elements\n"
              << vertex_nodeId.MemoryUsage() << " vertex_nodeId\n"
@@ -3646,7 +4141,8 @@ void NCMesh::PrintStats(std::ostream &out) const
            free_element_ids.MemoryUsage())/MiB << " MiB ]\n"
        "      free                     " << std::setw(9)
        << free_element_ids.Size() << "\n"
-       "   number of root elements   : " << std::setw(9) << root_count << "\n"
+       "   number of root elements   : " << std::setw(9)
+       << root_state.Size() << "\n"
        "   number of leaf elements   : " << std::setw(9)
        << leaf_elements.Size() << "\n"
        "   number of vertices        : " << std::setw(9)
@@ -3675,6 +4171,31 @@ void NCMesh::PrintStats(std::ostream &out) const
 }
 
 #ifdef MFEM_DEBUG
+void NCMesh::DebugLeafOrder(std::ostream &out) const
+{
+   tmp_vertex = new TmpVertex[nodes.NumIds()];
+   for (int i = 0; i < leaf_elements.Size(); i++)
+   {
+      const Element* elem = &elements[leaf_elements[i]];
+      for (int j = 0; j < Dim; j++)
+      {
+         double sum = 0.0;
+         int count = 0;
+         for (int k = 0; k < 8; k++)
+         {
+            if (elem->node[k] >= 0)
+            {
+               sum += CalcVertexPos(elem->node[k])[j];
+               count++;
+            }
+         }
+         out << sum / count << " ";
+      }
+      out << "\n";
+   }
+   delete [] tmp_vertex;
+}
+
 void NCMesh::DebugDump(std::ostream &out) const
 {
    // dump nodes

mesh/ncmesh.hpp

@@ -60,6 +60,13 @@ struct CoarseFineTransformations
    Array<Embedding> embeddings;
 
    const DenseTensor &GetPointMatrices(Geometry::Type geom) const;
+
+   void GetCoarseToFineMap(const Mesh &fine_mesh,
+                           Table &coarse_to_fine,
+                           Array<int> &coarse_to_ref_type,
+                           Table &ref_type_to_matrix,
+                           Array<Geometry::Type> &ref_type_to_geom) const;
+
    void Clear() { point_matrices.clear(); embeddings.DeleteAll(); }
    long MemoryUsage() const;
 };
@@ -245,6 +252,24 @@ public:
    void ClearTransforms();
 
 
+   // grid ordering
+
+   /** Return a space filling curve for a rectangular grid of elements.
+       Implemented is a generalized Hilbert curve for arbitrary grid dimensions.
+       If the width is odd, height should be odd too, otherwise one diagonal
+       (vertex-neighbor) step cannot be avoided in the curve. Even dimensions
+       are recommended. */
+   static void GridSfcOrdering2D(int width, int height,
+                                 Array<int> &coords);
+
+   /** Return a space filling curve for a 3D rectangular grid of elements.
+       The Hilbert-curve-like algorithm works well for even dimensions. For odd
+       width/height/depth it tends to produce some diagonal (edge-neighbor)
+       steps. Even dimensions are recommended. */
+   static void GridSfcOrdering3D(int width, int height, int depth,
+                                 Array<int> &coords);
+
+
    // utility
 
    /// Return Mesh vertex indices of an edge identified by 'edge_id'.
@@ -367,7 +392,7 @@ protected: // implementation
       Face() : attribute(-1), index(-1) { elem[0] = elem[1] = -1; }
 
       bool Boundary() const { return attribute >= 0; }
-      bool Unused() const { return elem[0] < 0 && elem[1] < 0;}
+      bool Unused() const { return elem[0] < 0 && elem[1] < 0; }
 
       // add or remove an element from the 'elem[2]' array
       void RegisterElement(int e);
@@ -406,10 +431,12 @@ protected: // implementation
    BlockArray<Element> elements; // storage for all Elements
    Array<int> free_element_ids;  // unused element ids - indices into 'elements'
 
-   // the first 'root_count' entries of 'elements' is the coarse mesh
-   int root_count;
+   /** Initial traversal state (~ element orientation) for each root element
+       NOTE: M = root_state.Size() is the number of root elements.
+       NOTE: the first M items of 'elements' is the coarse mesh. */
+   Array<int> root_state;
 
-   // coordinates of top-level vertices (organized as triples)
+   /// coordinates of top-level vertices (organized as triples)
    Array<double> top_vertex_pos;
 
    typedef HashTable<Node>::iterator node_iterator;
@@ -450,6 +477,12 @@ protected: // implementation
 
    virtual void AssignLeafIndices();
 
+   /** Try to find a space-filling curve friendly orientation of the root
+       elements: set 'root_state' based on the ordering of coarse elements.
+       Note that the coarse mesh itself must be ordered as an SFC by e.g.
+       Mesh::GetGeckoElementReordering. */
+   void InitRootState(int root_count);
+
    virtual bool IsGhost(const Element &el) const { return false; }
    virtual int GetNumGhostElements() const { return 0; }
    virtual int GetNumGhostVertices() const { return 0; }
@@ -528,6 +561,12 @@ protected: // implementation
 
    void CollectDerefinements(int elem, Array<Connection> &list);
 
+   /// Return el.node[index] correctly, even if the element is refined.
+   int RetrieveNode(const Element &el, int index);
+
+   /// Extended version of find_node: works if 'el' is refined; optional abort.
+   int FindNodeExt(const Element &el, int node, bool abort = false);
+
 
    // face/edge lists
 
@@ -737,6 +776,7 @@ protected: // implementation
 
 #ifdef MFEM_DEBUG
 public:
+   void DebugLeafOrder(std::ostream &out) const;
    void DebugDump(std::ostream &out) const;
 #endif
 

mesh/nurbs.cpp

@@ -1668,12 +1668,14 @@ void NURBSExtension::ConnectBoundaries()
 
 void NURBSExtension::ConnectBoundaries2D(int bnd0, int bnd1)
 {
-   int idx0, idx1;
+   int idx0 = -1, idx1 = -1;
    for (int b = 0; b < GetNBP(); b++)
    {
       if (bnd0 == patchTopo->GetBdrAttribute(b)) { idx0 = b; }
       if (bnd1 == patchTopo->GetBdrAttribute(b)) { idx1 = b; }
    }
+   MFEM_VERIFY(idx0 != -1,"Bdr 0 not found");
+   MFEM_VERIFY(idx1 != -1,"Bdr 1 not found");
 
    NURBSPatchMap p2g0(this);
    NURBSPatchMap p2g1(this);
@@ -1742,12 +1744,14 @@ void NURBSExtension::ConnectBoundaries2D(int bnd0, int bnd1)
 
 void NURBSExtension::ConnectBoundaries3D(int bnd0, int bnd1)
 {
-   int idx0, idx1;
+   int idx0 = -1, idx1 = -1;
    for (int b = 0; b < GetNBP(); b++)
    {
       if (bnd0 == patchTopo->GetBdrAttribute(b)) { idx0 = b; }
       if (bnd1 == patchTopo->GetBdrAttribute(b)) { idx1 = b; }
    }
+   MFEM_VERIFY(idx0 != -1,"Bdr 0 not found");
+   MFEM_VERIFY(idx1 != -1,"Bdr 1 not found");
 
    NURBSPatchMap p2g0(this);
    NURBSPatchMap p2g1(this);

mesh/pmesh.cpp

@@ -2988,6 +2988,8 @@ void ParMesh::NonconformingRefinement(const Array<Refinement> &refinements,
                  "serial Mesh)");
    }
 
+   DeleteFaceNbrData();
+
    // NOTE: no check of !refinements.Size(), in parallel we would have to reduce
 
    // do the refinements
@@ -3108,6 +3110,22 @@ void ParMesh::Rebalance()
    last_operation = Mesh::REBALANCE;
    sequence++;
 
+   // Make sure the Nodes use a ParFiniteElementSpace
+   if (Nodes && dynamic_cast<ParFiniteElementSpace*>(Nodes->FESpace()) == NULL)
+   {
+      ParFiniteElementSpace *pfes =
+         new ParFiniteElementSpace(*Nodes->FESpace(), *this);
+      ParGridFunction *new_nodes = new ParGridFunction(pfes);
+      *new_nodes = *Nodes;
+      if (Nodes->OwnFEC())
+      {
+         new_nodes->MakeOwner(Nodes->OwnFEC());
+         Nodes->MakeOwner(NULL); // takes away ownership of 'fec' and 'fes'
+         delete Nodes->FESpace();
+      }
+      delete Nodes;
+      Nodes = new_nodes;
+   }
    UpdateNodes();
 }
 
@@ -3128,16 +3146,16 @@ void ParMesh::RefineGroups(const DSTable &v_to_v, int *middle)
    int *I_group_svert, *J_group_svert;
    int *I_group_sedge, *J_group_sedge;
 
-   I_group_svert = new int[GetNGroups()+1];
-   I_group_sedge = new int[GetNGroups()+1];
+   I_group_svert = mfem::New<int>(GetNGroups()+1);
+   I_group_sedge = mfem::New<int>(GetNGroups()+1);
 
    I_group_svert[0] = I_group_svert[1] = 0;
    I_group_sedge[0] = I_group_sedge[1] = 0;
 
    // overestimate the size of the J arrays
-   J_group_svert = new int[group_svert.Size_of_connections()
-                           + group_sedge.Size_of_connections()];
-   J_group_sedge = new int[2*group_sedge.Size_of_connections()];
+   J_group_svert = mfem::New<int>(group_svert.Size_of_connections()
+                                  + group_sedge.Size_of_connections());
+   J_group_sedge = mfem::New<int>(2*group_sedge.Size_of_connections());
 
    for (int group = 0; group < GetNGroups()-1; group++)
    {
@@ -3375,16 +3393,16 @@ void ParMesh::UniformRefineGroups2D(int old_nv)
    int *I_group_svert, *J_group_svert;
    int *I_group_sedge, *J_group_sedge;
 
-   I_group_svert = new int[GetNGroups()];
-   I_group_sedge = new int[GetNGroups()];
+   I_group_svert = mfem::New<int>(GetNGroups());
+   I_group_sedge = mfem::New<int>(GetNGroups());
 
    I_group_svert[0] = 0;
    I_group_sedge[0] = 0;
 
    // compute the size of the J arrays
-   J_group_svert = new int[group_svert.Size_of_connections()
-                           + group_sedge.Size_of_connections()];
-   J_group_sedge = new int[2*group_sedge.Size_of_connections()];
+   J_group_svert = mfem::New<int>(group_svert.Size_of_connections()
+                                  + group_sedge.Size_of_connections());
+   J_group_sedge = mfem::New<int>(2*group_sedge.Size_of_connections());
 
    for (int group = 0; group < GetNGroups()-1; group++)
    {
@@ -3433,10 +3451,10 @@ void ParMesh::UniformRefineGroups3D(int old_nv, int old_nedges,
    int *I_group_stria, *J_group_stria;
    int *I_group_squad, *J_group_squad;
 
-   I_group_svert = new int[GetNGroups()];
-   I_group_sedge = new int[GetNGroups()];
-   I_group_stria = new int[GetNGroups()];
-   I_group_squad = new int[GetNGroups()];
+   I_group_svert = mfem::New<int>(GetNGroups());
+   I_group_sedge = mfem::New<int>(GetNGroups());
+   I_group_stria = mfem::New<int>(GetNGroups());
+   I_group_squad = mfem::New<int>(GetNGroups());
 
    I_group_svert[0] = 0;
    I_group_sedge[0] = 0;
@@ -3444,14 +3462,14 @@ void ParMesh::UniformRefineGroups3D(int old_nv, int old_nedges,
    I_group_squad[0] = 0;
 
    // compute the size of the J arrays
-   J_group_svert = new int[group_svert.Size_of_connections()
-                           + group_sedge.Size_of_connections()
-                           + group_squad.Size_of_connections()];
-   J_group_sedge = new int[2*group_sedge.Size_of_connections()
-                           + 3*group_stria.Size_of_connections()
-                           + 4*group_squad.Size_of_connections()];
-   J_group_stria = new int[4*group_stria.Size_of_connections()];
-   J_group_squad = new int[4*group_squad.Size_of_connections()];
+   J_group_svert = mfem::New<int>(group_svert.Size_of_connections()
+                                  + group_sedge.Size_of_connections()
+                                  + group_squad.Size_of_connections());
+   J_group_sedge = mfem::New<int>(2*group_sedge.Size_of_connections()
+                                  + 3*group_stria.Size_of_connections()
+                                  + 4*group_squad.Size_of_connections());
+   J_group_stria = mfem::New<int>(4*group_stria.Size_of_connections());
+   J_group_squad = mfem::New<int>(4*group_squad.Size_of_connections());
 
    const int oface = old_nv + old_nedges;
 
@@ -3614,6 +3632,7 @@ void ParMesh::UniformRefinement3D()
    // update the groups
    UniformRefineGroups3D(old_nv, old_nedges, v_to_v, *faces_tbl,
                          f2qf.Size() ? &f2qf : NULL);
+   delete faces_tbl;
 
    UpdateNodes();
 }

mesh/pncmesh.cpp

@@ -921,6 +921,10 @@ void ParNCMesh::GetConformingSharedStructures(ParMesh &pmesh)
    pmesh.group_stria.ShiftUpI();
 
    // create shared_edges
+   for (int i = 0; i < pmesh.shared_edges.Size(); i++)
+   {
+      delete pmesh.shared_edges[i];
+   }
    pmesh.shared_edges.SetSize(pmesh.sedge_ledge.Size());
    for (int i = 0; i < pmesh.shared_edges.Size(); i++)
    {
@@ -1292,7 +1296,7 @@ void ParNCMesh::Prune()
    }
 
    // derefine subtrees whose leaves are all unneeded
-   for (int i = 0; i < root_count; i++)
+   for (int i = 0; i < root_state.Size(); i++)
    {
       if (PruneTree(i)) { DerefineElement(i); }
    }
@@ -2102,7 +2106,7 @@ void ParNCMesh::ElementSet::Encode(const Array<int> &elements)
    // Each refinement tree that contains at least one element from the set
    // is encoded as HEADER + TREE, where HEADER is the root element number and
    // TREE is the output of EncodeTree().
-   for (int i = 0; i < ncmesh->root_count; i++)
+   for (int i = 0; i < ncmesh->root_state.Size(); i++)
    {
       if (ncmesh->elements[i].flag)
       {

mfem.hpp

@@ -15,6 +15,7 @@
 #include "config/config.hpp"
 
 #include "general/error.hpp"
+#include "general/device.hpp"
 #include "general/array.hpp"
 #include "general/sets.hpp"
 #include "general/hash.hpp"

miniapps/common/fem_extras.cpp

@@ -56,9 +56,44 @@ RT_FESpace::~RT_FESpace()
    delete FEC_;
 }
 
+void VisualizeMesh(socketstream &sock, const char *vishost, int visport,
+                   Mesh &mesh, const char *title,
+                   int x, int y, int w, int h, const char * keys, bool vec)
+{
+   bool newly_opened = false;
+   int connection_failed;
+
+   do
+   {
+      if (!sock.is_open() || !sock)
+      {
+         sock.open(vishost, visport);
+         sock.precision(8);
+         newly_opened = true;
+      }
+      sock << "solution\n";
+
+      mesh.Print(sock);
+
+      if (newly_opened)
+      {
+         sock << "window_title '" << title << "'\n"
+              << "window_geometry "
+              << x << " " << y << " " << w << " " << h << "\n";
+         if ( keys ) { sock << "keys " << keys << "\n"; }
+         else { sock << "keys maaAc\n"; }
+         if ( vec ) { sock << "vvv"; }
+         sock << endl;
+      }
+
+      connection_failed = !sock && !newly_opened;
+   }
+   while (connection_failed);
+}
+
 void VisualizeField(socketstream &sock, const char *vishost, int visport,
                     GridFunction &gf, const char *title,
-                    int x, int y, int w, int h, bool vec)
+                    int x, int y, int w, int h, const char * keys, bool vec)
 {
    Mesh &mesh = *gf.FESpace()->GetMesh();
 
@@ -82,8 +117,9 @@ void VisualizeField(socketstream &sock, const char *vishost, int visport,
       {
          sock << "window_title '" << title << "'\n"
               << "window_geometry "
-              << x << " " << y << " " << w << " " << h << "\n"
-              << "keys maaAc";
+              << x << " " << y << " " << w << " " << h << "\n";
+         if ( keys ) { sock << "keys " << keys << "\n"; }
+         else { sock << "keys maaAc\n"; }
          if ( vec ) { sock << "vvv"; }
          sock << endl;
       }

miniapps/common/fem_extras.hpp

@@ -66,13 +66,21 @@ private:
 };
 
 
+/// Visualize the given mesh object, using a GLVis server on the
+/// specified host and port. Set the visualization window title, and optionally,
+/// its geometry.
+void VisualizeMesh(socketstream &sock, const char *vishost, int visport,
+                   Mesh &mesh, const char *title,
+                   int x = 0, int y = 0, int w = 400, int h = 400,
+                   const char *keys = NULL);
+
 /// Visualize the given grid function, using a GLVis server on the
 /// specified host and port. Set the visualization window title, and optionally,
 /// its geometry.
 void VisualizeField(socketstream &sock, const char *vishost, int visport,
                     GridFunction &gf, const char *title,
                     int x = 0, int y = 0, int w = 400, int h = 400,
-                    bool vec = false);
+                    const char *keys = NULL, bool vec = false);
 
 } // namespace miniapps
 

miniapps/common/pfem_extras.cpp

@@ -261,9 +261,57 @@ DivergenceFreeProjector::Update()
    this->IrrotationalProjector::Update();
 }
 
+void VisualizeMesh(socketstream &sock, const char *vishost, int visport,
+                   ParMesh &pmesh, const char *title,
+                   int x, int y, int w, int h, const char *keys, bool vec)
+{
+   MPI_Comm comm = pmesh.GetComm();
+
+   int num_procs, myid;
+   MPI_Comm_size(comm, &num_procs);
+   MPI_Comm_rank(comm, &myid);
+
+   bool newly_opened = false;
+   int connection_failed;
+
+   do
+   {
+      if (myid == 0)
+      {
+         if (!sock.is_open() || !sock)
+         {
+            sock.open(vishost, visport);
+            sock.precision(8);
+            newly_opened = true;
+         }
+         sock << "solution\n";
+      }
+
+      pmesh.PrintAsOne(sock);
+
+      if (myid == 0 && newly_opened)
+      {
+         sock << "window_title '" << title << "'\n"
+              << "window_geometry "
+              << x << " " << y << " " << w << " " << h << "\n";
+         if ( keys ) { sock << "keys " << keys << "\n"; }
+         else { sock << "keys maaAc"; }
+         if ( vec ) { sock << "vvv"; }
+         sock << endl;
+      }
+
+      if (myid == 0)
+      {
+         connection_failed = !sock && !newly_opened;
+      }
+      MPI_Bcast(&connection_failed, 1, MPI_INT, 0, comm);
+   }
+   while (connection_failed);
+}
+
 void VisualizeField(socketstream &sock, const char *vishost, int visport,
                     ParGridFunction &gf, const char *title,
-                    int x, int y, int w, int h, bool vec)
+                    int x, int y, int w, int h, const char *keys, bool vec)
 {
    ParMesh &pmesh = *gf.ParFESpace()->GetParMesh();
    MPI_Comm comm = pmesh.GetComm();
@@ -295,8 +343,9 @@ void VisualizeField(socketstream &sock, const char *vishost, int visport,
       {
          sock << "window_title '" << title << "'\n"
               << "window_geometry "
-              << x << " " << y << " " << w << " " << h << "\n"
-              << "keys maaAc";
+              << x << " " << y << " " << w << " " << h << "\n";
+         if ( keys ) { sock << "keys " << keys << "\n"; }
+         else { sock << "keys maaAc"; }
          if ( vec ) { sock << "vvv"; }
          sock << endl;
       }

miniapps/common/pfem_extras.hpp

@@ -185,13 +185,21 @@ public:
 };
 
 
+/// Visualize the given parallel mesh object, using a GLVis server on the
+/// specified host and port. Set the visualization window title, and optionally,
+/// its geometry.
+void VisualizeMesh(socketstream &sock, const char *vishost, int visport,
+                   ParMesh &pmesh, const char *title,
+                   int x = 0, int y = 0, int w = 400, int h = 400,
+                   const char *keys = NULL);
+
 /// Visualize the given parallel grid function, using a GLVis server on the
 /// specified host and port. Set the visualization window title, and optionally,
 /// its geometry.
 void VisualizeField(socketstream &sock, const char *vishost, int visport,
                     ParGridFunction &gf, const char *title,
                     int x = 0, int y = 0, int w = 400, int h = 400,
-                    bool vec = false);
+                    const char *keys = NULL, bool vec = false);
 
 } // namespace miniapps
 

miniapps/conslaw/.gitignore

@@ -1,4 +0,0 @@
-build
-lib
-*.mesh
-*.gf

miniapps/conslaw/README.md

@@ -1,35 +0,0 @@
-# Discontinuous Galerkin MFEM mini-app using partial assembly
-
-This mini-app demonstrates using partial-assembly to solve hyperbolic
-conservation laws using discontinuous Galerkin methods and explicit
-time integration.
-
-The main object is a `PartialAssembly` object, which provides:
-
-- local interpolation and differentiation operators (including at faces)
-- face access to metric terms at quadrature data
-
-On top of this object, there are several operators that are provided
-(but more are possible). All of these operators allow for a "coefficient"
-to be evaluated at quadrature points, which is referred to as D. These
-operators are:
-
-- `BtDB`, which represents mass or source terms with coefficient `D`
-- `GtDB`, which represents dot product with the gradient of test functions
-- `BtDB_face`, which represents integrating against test functions on faces
-
-Using any of these operators simply requires templating on a class `D` which
-provides an operator to evaluate the coefficient at a quadrature point.
-
-There is a `ConservationLaw` object which is built on these three operators.
-Given a flux function and numerical flux function, it will assemble the 
-corresponding DG residual.
-
-Examples are provided in the `apps` directory for solving the scalar advection
-equation, Burgers' equation and the Euler equations of gas dynamics.
-
-This mini-app is still incomplete. Improvements are needed for:
-
-[ ] Handing of mixed meshes
-[ ] AMR and non-conforming meshes
-[ ] Second-order operators and viscous terms