Updating serial version of example 23

Fixing comment and "make style"
Removing unused data member
2019-04-25 16:21:42 -07:00 · 2019-04-25 16:21:05 -07:00 · 2019-04-25 16:20:32 -07:00 · 2019-04-25 16:20:00 -07:00 · 2019-04-25 16:19:19 -07:00 · 2019-04-25 15:01:13 -07:00
119 changed files with 11039 additions and 1277 deletions
@@ -43,7 +43,9 @@ before_build:
 build_script:
 - cmake --build build_parallel
 - cmake --build build_serial
+- cmake --build build_serial --target exec

 after_build:
 # - cmake --build build_parallel --target check
- cmake --build build_serial --target check
+- cmake --build build_serial --target RUN_TESTS
+
@@ -169,6 +169,7 @@ miniapps/performance/sol.*
 miniapps/tools/display-basis
 miniapps/tools/load-dc
 miniapps/tools/convert-dc
+miniapps/tools/lor-transfer

 miniapps/nurbs/ex1
 miniapps/nurbs/ex1p
@@ -205,6 +205,7 @@ install:
        else
            echo "Reusing cached hypre-2.10.0b/";
        fi;
+        ln -s hypre-2.10.0b hypre;
     else
        echo "Serial build, not using hypre";
     fi
@@ -8,12 +8,68 @@
                               http://mfem.org


-Version 3.4.1 (development)
-===========================
+Version 4.0-RC2, Apr 24, 2019
+=============================
+
+Requirements and Limitations
+----------------------------
+- This is a release candidate for mfem-4.0.
+- Use at your own risk -- not everything will work and the API may change.
+- We are looking for feedback from friendly users.
+- Unlike previous MFEM releases, this version requires a C++11 compiler.
+
+- GPU-related limitations:
+  * Hypre preconditioners are not yet available in GPU mode.
+  * Only constant coefficients are currently supported on GPUs.
+  * Full-assembly (on device), element assembly, and matrix-free bilinear forms
+    are not supported yet. Element batching is currently ignored.
+  * Partial assembly kernels are not implemented yet for simplices.
+
+GPU support
+-----------
+- Added initial support for hardware devices, such as GPUs, and programming
+  models, such as CUDA, OCCA, RAJA and OpenMP.
+
+- The GPU/device support is based on MFEM's new backends and kernels working
+  seamlessly with a new lightweight device/host memory manager. The kernels can
+  be implemented either in OCCA, or as a simple wrapper around for-loops, which
+  can then be dispatched to RAJA and native backends. See the files forall.hpp
+  and mem_manager.hpp in the general/ directory.
+
+- Several of the MFEM example codes (ex1, ex1p, ex6, and ex6p) can now take
+  advantage of GPU acceleration with the backend selectable at runtime. Many of
+  the linear algebra and finite element operations (e.g. partially assembled
+  bilinear forms) have been extended to take advantage of kernel acceleration by
+  simply replacing loops with the MFEM_FORALL() macro.
+
+- In addition to pure CUDA, the library currently supports OCCA, RAJA and OpenMP
+  kernels, which could be mixed and matched in different parts of the same
+  application. We plan on adding support for more programming models and devices
+  in the future, without the need for significant modifications in user code.
+  The list of current backends is: "occa-cuda", "raja-cuda", "cuda", "occa-omp",
+  "raja-omp", "omp", "occa-cpu", "raja-cpu", and "cpu".
+
+Discretization improvements
+---------------------------
+- Added support for a general "low-order refined"-to-"high-order" transfer of
+  GridFunction data from a "low-order refined" (LOR) space defined on a refined
+  mesh to a "high-order" (HO) finite element space defined on a coarse mesh. See
+  the new classes InterpolationGridTransfer and L2ProjectionGridTransfer and the
+  new LOR Transfer miniapp: miniapps/tools/lor-transfer.cpp.
+
+- Added support for derefinement of vector (RT + ND) spaces.
+
+- Added element flux, and flux energy computation in class ElasticityIntegrator,
+  allowing for the use of Zienkiewicz-Zhu type error estimators with the
+  integrator. For an illustration of this addition, see the new Example 22.
+
+- Added a variety of coefficients which are sums or products of existing
+  coefficients as well as grid function coefficients which return the
+  divergence, gradient, or curl of their GridFunctions.

 Support for wedge elements and meshes with mixed element types
 --------------------------------------------------------------
- Added support for wedge shaped mesh elements of arbitrary order (with Geometry
+- Added support for wedge-shaped mesh elements of arbitrary order (with Geometry
  type PRISM) which have two triangular faces and three quadrilateral faces.
  Several examples of such meshes can be found in the data/ directory.

@@ -39,13 +95,18 @@ Other meshing improvements
  follows precisely the paper:

     D. Arnold, A. Mukherjee, and L. Pouly, "Locally Adapted Tetrahedral Meshes
-     Using Bisection", SIAM J. Sci. Comput., 22(2), 431–448.
+     Using Bisection", SIAM J. Sci. Comput. 22 (2000), 431–448.

  This guarantees that the shape regularity of the elements will be preserved
  under refinement.

 - Added support for parallel communication groups on non-conforming meshes.

+- Improved parallel partitioning of non-conforming meshes. If the coarse mesh
+  elements are ordered as a sequence of face-neighbors, the parallel partitions
+  are now guaranteed to be continuous. To that end, inline quadrilateral and
+  hexahedral meshes are now by default ordered along a space-filling curve.
+
 - A boundary in a NURBS mesh can now be connected with another boundary. Such a
  periodic NURBS mesh is a simple way to impose periodic boundary conditions.

@@ -53,24 +114,8 @@ Other meshing improvements
  Cubit meshes.

 - The TMOP mesh optimization algorithms were extended to support user-defined
-  space-dependent limiting terms. Improved the TMOP objective functions by
-  more accurate normalization of the different terms.
-  
-Discretization improvements
---------------------------
- Added support for derefinement of vector (RT + ND) spaces.
-
- Added element flux, and flux energy computation in class ElasticityIntegrator,
-  allowing for the use of Zienkiewicz-Zhu type error estimators with the
-  integrator. For an illustration of this addition, see the new Example 22.
-
- Added a variety of coefficients which are sums or products of existing
-  coefficients as well as grid function coefficients which return the
-  divergence, gradient, or curl of their GridFunctions.
-
-New and improved solvers and preconditioners
--------------------------------------------
- Added support for parallel ILU preconditioning via hypre's Euclid solver.
+  space-dependent limiting terms. Improved the TMOP objective functions by more
+  accurate normalization of the different terms.

 New and updated examples and miniapps
 -------------------------------------
@@ -80,6 +125,9 @@ New and updated examples and miniapps
 - Added a new meshing miniapp, Extruder, that demonstrates the capability to
  produce 3D meshes by extruding 2D meshes.

+- Added a simple miniapp, LOR Transfer, for visualizing the actions of the
+  transfer operators between a high-order and a low-order refined spaces.
+
 - Added a new example, Example 20/20p, that solves a system of 1D ODEs derived
  from a Hamiltonian. The example demonstrates the use of the variable order,
  symplectic integration algorithm implemented in class SIAVSolver.
@@ -87,10 +135,26 @@ New and updated examples and miniapps
 - Added a new example, Example 22/22p, that illustrates the use of AMR to solve
  a linear elasticity problem. This is an extension of Example 2/2p.

+New and improved solvers and preconditioners
+--------------------------------------------
+- Added support for parallel ILU preconditioning via hypre's Euclid solver.
+
+- Added support for STRUMPACK v3 with a small API change in the class
+  STRUMPACKSolver, see "API changes" below.
+
 Miscellaneous
 -------------
+- In SparseMatrix added the option to perform MultTranspose() by matvec with
+  computed and stored transpose matrix. This is required for deterministic
+  results when using devices such as CUDA and OpenMP.
+
 - Added unit tests based on the Catch++ library.

+- Renamed the option MFEM_USE_OPENMP to MFEM_USE_LEGACY_OPENMP. This legacy
+  option is deprecated and planned for removal in a future release. The original
+  option name, MFEM_USE_OPENMP, is now used to enable the new OpenMP backends in
+  the new kernels.
+
 - Altered the way FGMRES counts its iterations so that it matches GMRES.

 - Various other simplifications, extensions, and bugfixes in the code.
@@ -115,6 +179,8 @@ API changes
 - Removed the virtual method Element::GetRefinementFlag, it is only used by the
  derived class Tetrahedron.
 - Added new methods: Array::CopyTo, Tetrahedron::Init.
+- In class STRUMPACKSolver, the method SetMC64Job() was replaced by the new
+  methods: DisableMatching(), EnableMatching(), and EnableParallelMatching().


 Version 3.4, released on May 29, 2018
@@ -13,6 +13,11 @@ cmake_minimum_required(VERSION 2.8.11)
 set(USER_CONFIG "${CMAKE_CURRENT_SOURCE_DIR}/config/user.cmake" CACHE PATH
  "Path to optional user configuration file.")

+# Require C++11 and disable compiler-specific extensions
+set(CMAKE_CXX_STANDARD 11)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(CMAKE_CXX_EXTENSIONS OFF)
+
 # Load user settings before the defaults - this way the defaults will not
 # overwrite the user set options. If the user has not set all options, we still
 # have the defaults.
@@ -81,6 +86,13 @@ include("${CMAKE_CURRENT_SOURCE_DIR}/config/XSDKDefaults.cmake")

 # Enable languages.
 enable_language(CXX)
+if (MFEM_USE_CUDA)
+   # MFEM_USE_CUDA requires CMake 3.8 or newer (for direct CUDA support)
+   cmake_minimum_required(VERSION 3.8 FATAL_ERROR)
+   enable_language(CUDA)
+   message(STATUS "Using CUDA architecture: ${CUDA_ARCH}")
+endif()
+
 if (XSDK_ENABLE_C)
   enable_language(C)
 endif()
@@ -170,12 +182,11 @@ if (MFEM_USE_LAPACK)
 endif()

 # OpenMP
-if (MFEM_USE_OPENMP)
-  if (MFEM_THREAD_SAFE)
-    find_package(OpenMP REQUIRED)
-  else()
-    message(FATAL_ERROR " *** MFEM_USE_OPENMP requires MFEM_THREAD_SAFE=ON.")
+if (MFEM_USE_OPENMP OR MFEM_USE_LEGACY_OPENMP)
+  if (NOT MFEM_THREAD_SAFE AND MFEM_USE_LEGACY_OPENMP)
+    message(FATAL_ERROR " *** MFEM_USE_LEGACY_OPENMP requires MFEM_THREAD_SAFE=ON.")
  endif()
+  find_package(OpenMP REQUIRED)
 endif()

 # SuiteSparse (before SUNDIALS which may depend on KLU)
@@ -262,6 +273,32 @@ if (MFEM_USE_PUMI)
  endif()
 endif()

+# CUDA
+if (MFEM_USE_CUDA)
+   set(CMAKE_CUDA_STANDARD 11)
+   set(CMAKE_CUDA_STANDARD_REQUIRED ON)
+   set(CMAKE_CUDA_EXTENSIONS OFF)
+   set(CMAKE_CUDA_FLAGS "-arch=${CUDA_ARCH} --expt-extended-lambda"
+       CACHE STRING "CUDA flags set for MFEM" FORCE)
+   if (MFEM_USE_MPI)
+      set(CUDA_CCBIN_COMPILER ${MPI_CXX_COMPILER})
+   else()
+      set(CUDA_CCBIN_COMPILER ${CMAKE_CXX_COMPILER})
+   endif()
+   string(APPEND CMAKE_CUDA_FLAGS " -ccbin ${CUDA_CCBIN_COMPILER}")
+   set(MFEM_USE_MM YES CACHE BOOL "Enable MFEM's memory manager" FORCE)
+endif()
+
+# OCCA
+if (MFEM_USE_OCCA)
+   find_package(OCCA REQUIRED)
+endif()
+
+# RAJA
+if (MFEM_USE_RAJA)
+   find_package(RAJA REQUIRED)
+endif()
+
 # MFEM_TIMER_TYPE
 if (NOT DEFINED MFEM_TIMER_TYPE)
  if (APPLE)
@@ -287,7 +324,7 @@ endif()
 #    be before SuiteSparse.
 set(MFEM_TPLS MPI_CXX OPENMP BLAS LAPACK METIS HYPRE SuiteSparse SUNDIALS PETSC
    MESQUITE SuperLUDist STRUMPACK AXOM CONDUIT GECKO GNUTLS NETCDF MPFR PUMI
-    POSIXCLOCKS MFEMBacktrace ZLIB)
+    POSIXCLOCKS MFEMBacktrace ZLIB OCCA RAJA)
 # Add all *_FOUND libraries in the variable TPL_LIBRARIES.
 set(TPL_LIBRARIES "")
 set(TPL_INCLUDE_DIRS "")
@@ -323,6 +360,13 @@ set(MFEM_SOURCE_DIRS general linalg mesh fem)
 foreach(DIR IN LISTS MFEM_SOURCE_DIRS)
  add_subdirectory(${DIR})
 endforeach()
+
+if (MFEM_USE_CUDA)
+   foreach(file IN LISTS SOURCES)
+      set_property(SOURCE ${file} PROPERTY LANGUAGE CUDA)
+   endforeach()
+endif()
+
 add_subdirectory(config)
 set(MASTER_HEADERS
  ${PROJECT_SOURCE_DIR}/mfem.hpp
@@ -333,6 +377,11 @@ set(CMAKE_INSTALL_RPATH_USE_LINK_PATH ON CACHE BOOL "")
 set(CMAKE_INSTALL_RPATH "${_lib_path}" CACHE PATH "")
 set(CMAKE_INSTALL_NAME_DIR "${_lib_path}" CACHE PATH "")

+set(MFEM_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR} CACHE PATH
+    "The MFEM source directory" FORCE)
+set(MFEM_INSTALL_DIR ${CMAKE_INSTALL_PREFIX} CACHE PATH
+    "The MFEM install directory" FORCE)
+
 # Declaring the library
 add_library(mfem ${SOURCES} ${HEADERS} ${MASTER_HEADERS})
 # message(STATUS "TPL_LIBRARIES = ${TPL_LIBRARIES}")
@@ -430,12 +479,12 @@ endif()
 # Add 'check' target - quick test
 if (NOT MFEM_USE_MPI)
  add_custom_target(check
-    ${CMAKE_CTEST_COMMAND} -R '^ex1_ser' -C ${CMAKE_CFG_INTDIR}
+    ${CMAKE_CTEST_COMMAND} -R \"^ex1_ser\" -C ${CMAKE_CFG_INTDIR}
    USES_TERMINAL)
  add_dependencies(check ex1)
 else()
  add_custom_target(check
-    ${CMAKE_CTEST_COMMAND} -R '^ex1p' -C ${CMAKE_CFG_INTDIR}
+    ${CMAKE_CTEST_COMMAND} -R \"^ex1p\" -C ${CMAKE_CFG_INTDIR}
    USES_TERMINAL)
  add_dependencies(check ex1p)
 endif()
@@ -480,6 +529,13 @@ install(DIRECTORY ${MFEM_SOURCE_DIRS}
  DESTINATION ${INSTALL_INCLUDE_DIR}/mfem
  FILES_MATCHING PATTERN "*.hpp")

+# Install the okl files
+if (MFEM_USE_OCCA)
+  install(DIRECTORY ${MFEM_SOURCE_DIRS}
+    DESTINATION ${INSTALL_INCLUDE_DIR}/mfem
+    FILES_MATCHING PATTERN "*.okl")
+endif()
+
 # Install ${HEADERS}
 # ---
 # foreach (HDR ${HEADERS})
@@ -142,6 +142,16 @@ Origin](#developers-certificate-of-origin-11) at the end of this file.*
  + [`HypreParMatrix`](http://mfem.github.io/doxygen/html/classmfem_1_1HypreParMatrix.html) and [`HypreParVector`](http://mfem.github.io/doxygen/html/classmfem_1_1HypreParVector.html)
  + [`HypreSolver`](http://mfem.github.io/doxygen/html/classmfem_1_1HypreSolver.html) and other [hypre classes](http://mfem.github.io/doxygen/html/hypre_8hpp.html)

+- GPU and multi-core CPU support is based on device kernels supporting different
+  backends (CUDA, OCCA, RAJA, OpenMP, etc.) and an internal lightweight
+  device/host memory manager.
+
+- The main device-relevant classes and sources are:
+  + [`Device`](http://mfem.github.io/doxygen/html/device_8hpp.html)
+  + [`MemoryManager`](http://mfem.github.io/doxygen/html/mem_manager_8hpp.html)
+  + the [`MFEM_FORALL`](http://mfem.github.io/doxygen/html/forall_8hpp.html) macro
+  + the [`cuda.hpp`](http://mfem.github.io/doxygen/html/cuda_8hpp.html) and [`occa.hpp`](http://mfem.github.io/doxygen/html/occa_8hpp.html) files
+
 - The `general/` directory contains C++ classes that serve as utilities for
  communication, error handling, arrays, (Boolean) tables, timing, etc.

@@ -13,14 +13,38 @@ of MFEM is a (modern) C++ compiler, such as g++. The parallel version of MFEM
 requires an MPI C++ compiler, as well as the following external libraries:

 - hypre (a library of high-performance preconditioners)
-  http://www.llnl.gov/CASC/hypre
+  https://github.com/hypre-space/hypre

 - METIS (a family of multilevel partitioning algorithms)
  http://glaros.dtc.umn.edu/gkhome/metis/metis/overview

+The hypre dependency can be downloaded as a tarball from GitHub or from the
+project webpage https://www.llnl.gov/casc/hypre. For example, the 2.16.0 release
+of hypre is available at
+
+   https://github.com/hypre-space/hypre/archive/v2.16.0.tar.gz
+
 The METIS dependency can be disabled but that is not generally recommended, see
 the option MFEM_USE_METIS.

+MFEM also includes support for devices such as GPUs, and programming models such
+as CUDA, OCCA, OpenMP and RAJA.
+
+- Starting with version 4.0, MFEM requires a C++11 compiler
+
+- CUDA support requires an NVIDIA GPU and an installation of the CUDA Toolkit
+  https://developer.nvidia.com/cuda-toolkit
+
+- OCCA support requires the OCCA library
+  https://libocca.org
+
+- OpenMP support requires a compiler implementing the OpenMP API
+  https://www.openmp.org
+
+- RAJA support requires installation of the RAJA performance portability layer
+  with (optionally) support for CUDA and OpenMP
+  https://github.com/LLNL/RAJA
+
 The library supports two build systems: one based on GNU make, and a second one
 based on CMake. Both build systems are described below. Some hints for building
 without GNU make or CMake can be found at the end of this file.
@@ -30,7 +54,7 @@ following package managers:

 - Spack, https://github.com/spack/spack
 - OpenHPC, http://openhpc.community
- Homebrew/Science, https://github.com/Homebrew/homebrew-science
+- Homebrew/Science, https://github.com/Homebrew/homebrew-science (deprecated)

 We also recommend downloading and building the MFEM-based GLVis visualization
 tool which can be used to visualize the meshes and solution in MFEM's examples
@@ -42,11 +66,15 @@ Serial build:
   make serial -j 4

 Parallel build:
-   (download hypre 2.10.0b and METIS 4 from above URLs)
+   (download hypre and METIS 4 from above URLs)
   (build METIS 4 in ../metis-4.0 relative to mfem/)
-   (build hypre 2.10.0b in ../hypre-2.10.0b relative to mfem/)
+   (build hypre in ../hypre relative to mfem/)
   make parallel -j 4

+CUDA build:
+   make cuda -j 4
+   (build for a specific compute capability: 'make cuda -j 4 CUDA_ARCH=sm_30')
+
 Example codes (serial/parallel, depending on the build):
   cd examples
   make -j 4
@@ -57,7 +85,6 @@ Build everything (library, examples and miniapps) with current configuration:
 Quick-check the build by running Example 1/1p (optional):
   make check

-
 Quick start with CMake
 ======================
 Serial build:
@@ -66,13 +93,19 @@ Serial build:
   make -j 4   (assuming "UNIX Makefiles" generator)

 Parallel build:
-   (download hypre 2.10.0b and METIS 4 from above URLs)
+   (download hypre and METIS 4 from above URLs)
   (build METIS 4 in ../metis-4.0 relative to mfem/)
-   (build hypre 2.10.0b in ../hypre-2.10.0b relative to mfem/)
+   (build hypre in ../hypre relative to mfem/)
   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
   cmake <mfem-source-dir> -DMFEM_USE_MPI=YES
   make -j 4

+CUDA build:
+   (this build requires CMake 3.8 or newer)
+   mkdir <mfem-build-dir> ; cd <mfem-build-dir>
+   cmake <mfem-source-dir> -DMFEM_USE_CUDA=YES
+   make -j 4
+
 Example codes (serial/parallel, depending on the build):
   make examples -j 4

@@ -132,6 +165,10 @@ are also defined:
   make parallel -> Builds parallel optimized version of the library
   make debug    -> Builds serial debug version of the library
   make pdebug   -> Builds parallel debug version of the library
+   make cuda     -> Builds serial cuda optimized version of the library
+   make pcuda    -> Builds parallel cuda optimized version of the library
+   make cudebug  -> Builds serial cuda debug version of the library
+   make pcudebug -> Builds parallel cuda debug version of the library

 Note that any of the above shortcuts accept configuration options, either at the
 command line or through a user configuration file.
@@ -193,8 +230,9 @@ Configuration options (GNU make)
 See the configuration file config/defaults.mk for the default settings.

 Compilers:
-   CXX    - C++ compiler, serial build
-   MPICXX - MPI C++ compiler, parallel build
+   CXX      - C++ compiler, serial build
+   MPICXX   - MPI C++ compiler, parallel build
+   CUDA_CXX - The CUDA compiler, 'nvcc'

 Compiler options:
   OPTIM_FLAGS - Options for optimized build
@@ -230,7 +268,7 @@ MFEM_DEBUG = YES/NO
   and consistency checks that may simplify bug-hunting.

 MFEM_USE_EXCEPTIONS = YES/NO
-   Enable the use of exceptions. In particular, modifies the default bahavior
+   Enable the use of exceptions. In particular, modifies the default behavior
   when errors are encountered: throw an exception, instead of aborting.

 MFEM_USE_LIBUNWIND = YES/NO
@@ -250,8 +288,12 @@ MFEM_THREAD_SAFE = YES/NO
   Use thread-safe implementation for some classes/methods. This comes at the
   cost of extra memory allocation and de-allocation.

-MFEM_USE_OPENMP = YES/NO
+MFEM_USE_LEGACY_OPENMP = YES/NO
   Enable (basic) experimental OpenMP support. Requires MFEM_THREAD_SAFE.
+   This option is deprecated.
+
+MFEM_USE_OPENMP = YES/NO
+   Enable the OpenMP backend.

 MFEM_USE_MEMALLOC = YES/NO
   Internal MFEM option: enable batch allocation for some small objects.
@@ -362,6 +404,32 @@ MFEM_USE_PUMI = YES/NO
   models and effectively supports automated adaptive analysis. PUMI enables
   support for parallel unstructured mesh modifications in MFEM.

+MFEM_USE_MM = YES/NO
+   Enables support for the MFEM's memory manager (MM), which is required to
+   support devices with different memory spaces. This option is required when
+   CUDA support is enabled, i.e. when MFEM_USE_CUDA=YES.
+
+MFEM_USE_CUDA = YES/NO
+   Enables support for CUDA devices in MFEM. CUDA is a parallel computing
+   platform and programming model for general computing on graphical processing
+   units (GPUs). This option requires MFEM_USE_MM. The variable CUDA_ARCH is
+   used to specify the CUDA compute capability used during compilation (by
+   default, CUDA_ARCH=sm_60). When enabled, this option uses the CUDA_* build
+   options, see below.
+
+MFEM_USE_RAJA = YES/NO
+   Enable support for the RAJA performance portability layer in MFEM. RAJA
+   provides a portable abstraction for loops, supporting different programming
+   model backends. When using RAJA built with CUDA support, CUDA support must be
+   also enabled in MFEM, i.e. MFEM_USE_CUDA=YES must be set.
+
+MFEM_USE_OCCA = YES/NO
+   Enables support for the OCCA library in MFEM. OCCA is an open-source library
+   which aims to make it easy to program different types of devices (e.g. CPU,
+   GPU, FPGA) by providing an unified API for interacting with JIT-compiled
+   backends. In order to use the OCCA CUDA backend, CUDA support must be enabled
+   in MFEM as well, i.e. MFEM_USE_CUDA=YES must be set.
+
 MFEM_BUILD_TAG = (any value)
   An optional tag to characterize the build. Exported to config/config.mk.
   Can be used to identify the MFEM build from other makefiles.
@@ -383,7 +451,7 @@ directory and use the string @MFEM_DIR@, e.g. HYPRE_OPT = -I@MFEM_DIR@/../hypre.
 The specific libraries and their options are:

 - HYPRE, required for the parallel build, i.e. when MFEM_USE_MPI = YES.
-  URL: http://www.llnl.gov/CASC/hypre
+  URL: https://github.com/hypre-space/hypre and https://www.llnl.gov/casc/hypre
  Options: HYPRE_OPT, HYPRE_LIB.

 - METIS, used when MFEM_USE_METIS = YES. If using METIS 5, set
@@ -397,7 +465,8 @@ The specific libraries and their options are:
       http://math-atlas.sourceforge.net (ATLAS)
  Options: LAPACK_OPT (currently not used/needed), LAPACK_LIB.

- OpenMP (optional), usually part of compiler, used when MFEM_USE_OPENMP = YES.
+- OpenMP (optional), usually part of compiler, used when either MFEM_USE_OPENMP
+  or MFEM_USE_LEGACY_OPENMP is set to YES.
  Options: OPENMP_OPT, OPENMP_LIB.

 - High-resolution POSIX clocks: when using MFEM_TIMER_TYPE = 2, it may be
@@ -429,7 +498,8 @@ The specific libraries and their options are:

 - STRUMPACK (optional), used when MFEM_USE_STRUMPACK = YES. Note that STRUMPACK
  requires the PT-Scotch and Scalapack libraries as well as ParMETIS, which
-  includes METIS 5 in its distribution.
+  includes METIS 5 in its distribution. Starting with STRUMPACK v2.2.0, ParMETIS
+  and PT-Scotch are optional dependencies.
  The support for STRUMPACK was added in MFEM v3.3.2 and it requires STRUMPACK
  2.0.0 or later.
  URL: http://portal.nersc.gov/project/sparse/strumpack
@@ -475,6 +545,18 @@ The specific libraries and their options are:
  URL: https://scorec.rpi.edu/pumi
  Options: PUMI_OPT, PUMI_LIB.

+- CUDA, used when MFEM_USE_CUDA = YES.
+  URL: https://developer.nvidia.com/cuda-toolkit
+  Options: CUDA_CXX, CUDA_ARCH, CUDA_OPT, CUDA_LIB.
+
+- OCCA, used when MFEM_USE_OCCA = YES.
+  URL: https://libocca.org
+  Options: OCCA_DIR, OCCA_OPT, OCCA_LIB.
+
+- RAJA, used when MFEM_USE_RAJA = YES.
+  URL: https://github.com/LLNL/RAJA
+  Options: RAJA_DIR, RAJA_OPT, RAJA_LIB.
+
 - MPFR (optional), used when MFEM_USE_MPFR = YES.
  URL: http://mpfr.org, it depends on the GMP library: https://gmplib.org
  Options: MPFR_OPT, MPFR_LIB.
@@ -579,6 +661,8 @@ Configuration variables (CMake)
 ===============================
 See the configuration file config/defaults.cmake for the default settings.

+Note: the option MFEM_USE_CUDA requires CMake version 3.8 or newer!
+
 Non-standard CMake variables for compilers:
   CXX    - If set, overwrite the auto-detected C++ compiler, serial build
   MPICXX - If set, overwrite the auto-detected MPI C++ compiler, parallel build
@@ -596,6 +680,7 @@ MFEM_USE_METIS - Set to ${MFEM_USE_MPI}, can be overwritten.
 MFEM_USE_LIBUNWIND
 MFEM_USE_LAPACK
 MFEM_THREAD_SAFE
+MFEM_USE_LEGACY_OPENMP
 MFEM_USE_OPENMP
 MFEM_USE_MEMALLOC
 MFEM_TIMER_TYPE - Set automatically, can be overwritten.
@@ -608,6 +693,10 @@ MFEM_USE_NETCDF
 MFEM_USE_MPFR
 MFEM_USE_GZSTREAM
 MFEM_USE_PUMI
+MFEM_USE_CUDA
+MFEM_USE_OCCA
+MFEM_USE_RAJA
+MFEM_USE_MM

 The following options are CMake specific:

@@ -654,6 +743,8 @@ The CMake build system adds auto-detection for the following packages/libraries:
 - LIBUNWIND
 - POSIXCLOCKS
 - PUMI
+ - OCCA
+ - RAJA

 The following built-in CMake packages are also used:

@@ -8,9 +8,9 @@
                               http://mfem.org

 MFEM is a modular parallel C++ library for finite element methods. Its goal is
-to enable the research and development of scalable finite element discretization
-and solver algorithms through general finite element abstractions, accurate and
-flexible visualization, and tight integration with the hypre library.
+to enable high-performance scalable finite element discretization research and
+application development on a wide variety of platforms, ranging from laptops to
+supercomputers.

 * For building instructions, see the file INSTALL, or type "make help".

@@ -39,23 +39,24 @@ conforming and non-conforming (AMR) adaptive refinement. Arbitrary element
 transformations, allowing for high-order mesh elements with curved boundaries,
 are also supported.

-MFEM is commonly used as a "finite element to linear algebra translator", since
-it can take a problem described in terms of finite element-type objects, and
-produce the corresponding linear algebra vectors and sparse matrices. In order
-to facilitate this, MFEM uses compressed sparse row (CSR) sparse matrix storage
-and includes simple smoothers and Krylov solvers, such as PCG, MINRES and GMRES,
-as well as support for sequential sparse direct solvers from the SuiteSparse
+When used as a "finite element to linear algebra translator", MFEM can take a
+problem described in terms of finite element-type objects, and produce the
+corresponding linear algebra vectors and fully or partially assembled operators,
+e.g. in the form of global sparse matrices or matrix-free operators. The library
+includes simple smoothers and Krylov solvers, such as PCG, MINRES and GMRES, as
+well as support for sequential sparse direct solvers from the SuiteSparse
 library. Nonlinear solvers (the Newton method), eigensolvers (LOBPCG), and
 several explicit and implicit Runge-Kutta time integrators are also available.

 MFEM supports MPI-based parallelism throughout the library, and can readily be
-used as a scalable unstructured finite element problem generator. MFEM-based
-applications require minimal changes to transition from a serial to a
-high-performing parallel version of the code, where they can take advantage of
-the integrated scalable linear solvers from the hypre library. Comprehensive
-support for other external packages, e.g. PETSc and SUNDIALS is also included,
-giving access to many additional linear and nonlinear solvers, preconditioners,
-time integrators, etc.
+used as a scalable unstructured finite element problem generator. As of version
+4.0, MFEM offers initial support for GPU acceleration, and programming models,
+such as CUDA, OCCA, RAJA and OpenMP. MFEM-based applications require minimal
+changes to switch from a serial to a high-performing MPI-parallel version of the
+code, where they can take advantage of the integrated linear solvers from the
+hypre library.  Comprehensive support for other external packages, e.g. PETSc
+and SUNDIALS is also included, giving access to many additional linear and
+nonlinear solvers, preconditioners, time integrators, etc.

 For examples of using MFEM, see the examples/ and miniapps/ directories, as well
 as the OpenGL visualization tool GLVis which is available at http://glvis.org.
@@ -74,7 +74,7 @@

 IF (NOT COMMAND PRINT_VAR)
  FUNCTION(PRINT_VAR  VAR_NAME)
-    MESSAGE("-- " "${VAR_NAME} = '${${VAR_NAME}}'")
+    MESSAGE(STATUS "${VAR_NAME} = '${${VAR_NAME}}'")
  ENDFUNCTION()
 ENDIF()

@@ -166,14 +166,14 @@ IF (USE_XSDK_DEFAULTS)
    ENDIF()
    XSDK_HANDLE_LANG_DEFAULTS(Fortran  FC  "FFLAGS;FCFLAGS")
  ENDIF()
-  
+
  # Set XSDK defaults for other CMake variables
-  
+
  IF ("${BUILD_SHARED_LIBS}"  STREQUAL  "")
    MESSAGE("-- " "XSDK: Setting default BUILD_SHARED_LIBS=TRUE")
    SET(BUILD_SHARED_LIBS  TRUE  CACHE  BOOL  "Set by default in XSDK mode")
  ENDIF()
-  
+
  IF ("${CMAKE_BUILD_TYPE}"  STREQUAL  "")
    MESSAGE("-- " "XSDK: Setting default CMAKE_BUILD_TYPE=DEBUG")
    SET(CMAKE_BUILD_TYPE  DEBUG  CACHE  STRING  "Set by default in XSDK mode")
@@ -181,6 +181,13 @@ IF (USE_XSDK_DEFAULTS)

 ENDIF()

+
+##################################################################################
+#
+#               MFEM-specific additions: set TPL MFEM_USE_* defaults
+#
+##################################################################################
+
 IF (DEFINED TPL_ENABLE_MPI)
  SET(MFEM_USE_MPI ${TPL_ENABLE_MPI} CACHE BOOL "Enable MPI parallel build" FORCE)
 ENDIF()
@@ -252,3 +259,15 @@ ENDIF()
 IF (DEFINED TPL_ENABLE_PUMI)
  SET(MFEM_USE_PUMI ${TPL_ENABLE_PUMI} CACHE BOOL "Enable PUMI" FORCE)
 ENDIF()
+
+IF (DEFINED TPL_ENABLE_CUDA)
+  SET(MFEM_USE_CUDA ${TPL_ENABLE_CUDA} CACHE BOOL "Enable CUDA" FORCE)
+ENDIF()
+
+IF (DEFINED TPL_ENABLE_OCCA)
+  SET(MFEM_USE_OCCA ${TPL_ENABLE_OCCA} CACHE BOOL "Enable OCCA" FORCE)
+ENDIF()
+
+IF (DEFINED TPL_ENABLE_RAJA)
+  SET(MFEM_USE_RAJA ${TPL_ENABLE_RAJA} CACHE BOOL "Enable RAJA" FORCE)
+ENDIF()
@@ -25,6 +25,7 @@ set(MFEM_USE_LIBUNWIND @MFEM_USE_LIBUNWIND@)
 set(MFEM_USE_LAPACK @MFEM_USE_LAPACK@)
 set(MFEM_THREAD_SAFE @MFEM_THREAD_SAFE@)
 set(MFEM_USE_OPENMP @MFEM_USE_OPENMP@)
+set(MFEM_USE_LEGACY_OPENMP @MFEM_USE_LEGACY_OPENMP@)
 set(MFEM_USE_MEMALLOC @MFEM_USE_MEMALLOC@)
 set(MFEM_TIMER_TYPE @MFEM_TIMER_TYPE@)
 set(MFEM_USE_SUNDIALS @MFEM_USE_SUNDIALS@)
@@ -40,6 +41,10 @@ set(MFEM_USE_MPFR @MFEM_USE_MPFR@)
 set(MFEM_USE_SIDRE @MFEM_USE_SIDRE@)
 set(MFEM_USE_CONDUIT @MFEM_USE_CONDUIT@)
 set(MFEM_USE_PUMI @MFEM_USE_PUMI@)
+set(MFEM_USE_MM @MFEM_USE_MM@)
+set(MFEM_USE_CUDA @MFEM_USE_CUDA@)
+set(MFEM_USE_OCCA @MFEM_USE_OCCA@)
+set(MFEM_USE_RAJA @MFEM_USE_RAJA@)

 set(MFEM_CXX_COMPILER "@CMAKE_CXX_COMPILER@")
 set(MFEM_CXX_FLAGS "@CMAKE_CXX_FLAGS@")
@@ -30,6 +30,12 @@
 #define MFEM_VERSION_MINOR (((MFEM_VERSION)/100)%100)
 #define MFEM_VERSION_PATCH ((MFEM_VERSION)%100)

+// MFEM source directory.
+#define MFEM_SOURCE_DIR "@MFEM_SOURCE_DIR@"
+
+// MFEM install directory.
+#define MFEM_INSTALL_DIR "@MFEM_INSTALL_DIR@"
+
 // Description of the git commit used to build MFEM.
 #cmakedefine MFEM_GIT_STRING "@MFEM_GIT_STRING@"

@@ -62,9 +68,12 @@
 // allocation and de-allocation.
 #cmakedefine MFEM_THREAD_SAFE

-// Enable experimental OpenMP support. Requires MFEM_THREAD_SAFE.
+// Enable the OpenMP backend.
 #cmakedefine MFEM_USE_OPENMP

+// [Deprecated] Enable experimental OpenMP support. Requires MFEM_THREAD_SAFE.
+#cmakedefine MFEM_USE_LEGACY_OPENMP
+
 // Enable MFEM functionality based on the Mesquite library.
 #cmakedefine MFEM_USE_MESQUITE

@@ -101,6 +110,19 @@
 // Enable MFEM functionality based on the PUMI library
 #cmakedefine MFEM_USE_PUMI

+// Build the GPU/CUDA-enabled version of the MFEM library.
+// Requires a CUDA compiler (nvcc).
+#cmakedefine MFEM_USE_CUDA
+
+// Enable MFEM functionality based on the RAJA library
+#cmakedefine MFEM_USE_RAJA
+
+// Enable MFEM functionality based on the OCCA library
+#cmakedefine MFEM_USE_OCCA
+
+// Enable MFEM's internal Memory Manager (needed e.g. for MFEM_USE_CUDA)
+#cmakedefine MFEM_USE_MM
+
 // Which library functions to use in class StopWatch for measuring time.
 // For a list of the available options, see INSTALL.
 // If not defined, an option is selected automatically.
@@ -0,0 +1,19 @@
+# Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at the
+# Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights reserved.
+# See file COPYRIGHT for details.
+#
+# This file is part of the MFEM library. For more information and source code
+# availability see http://mfem.org.
+#
+# MFEM is free software; you can redistribute it and/or modify it under the
+# terms of the GNU Lesser General Public License (as published by the Free
+# Software Foundation) version 2.1 dated February 1999.
+
+# Defines the following variables:
+#   - OCCA_FOUND
+#   - OCCA_LIBRARIES
+#   - OCCA_INCLUDE_DIRS
+
+include(MfemCmakeUtilities)
+mfem_find_package(OCCA OCCA OCCA_DIR "include" "occa.hpp" "lib" "occa"
+  "Paths to headers required by OCCA." "Libraries required by OCCA.")
@@ -0,0 +1,30 @@
+# Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at the
+# Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights reserved.
+# See file COPYRIGHT for details.
+#
+# This file is part of the MFEM library. For more information and source code
+# availability see http://mfem.org.
+#
+# MFEM is free software; you can redistribute it and/or modify it under the
+# terms of the GNU Lesser General Public License (as published by the Free
+# Software Foundation) version 2.1 dated February 1999.
+
+# Defines the following variables:
+#   - RAJA_FOUND
+#   - RAJA_LIBRARIES
+#   - RAJA_INCLUDE_DIRS
+
+include(MfemCmakeUtilities)
+mfem_find_package(RAJA RAJA RAJA_DIR "include" "RAJA/RAJA.hpp" "lib" "RAJA"
+  "Paths to headers required by RAJA." "Libraries required by RAJA.")
+
+if (NOT RAJA_CONFIG_CMAKE)
+   set(RAJA_CONFIG_CMAKE "${RAJA_DIR}/share/raja/cmake/raja-config.cmake")
+endif()
+if (EXISTS "${RAJA_CONFIG_CMAKE}")
+   include("${RAJA_CONFIG_CMAKE}")
+   if (ENABLE_CUDA AND NOT MFEM_USE_CUDA)
+      message(FATAL_ERROR
+         "RAJA is built with CUDA: MFEM_USE_CUDA=YES is required")
+   endif()
+endif()
@@ -232,10 +232,12 @@ function(mfem_find_package Name Prefix DirVar IncSuffixes Header LibSuffixes
  # If we have the TPL_ versions of _INCLUDE_DIRS and _LIBRARIES then set the
  # standard ${Prefix} versions
  if (TPL_${Prefix}_INCLUDE_DIRS)
-    set(${Prefix}_INCLUDE_DIRS ${TPL_${Prefix}_INCLUDE_DIRS} CACHE STRING "TPL_${Prefix}_INCLUDE_DIRS was found." FORCE)
+    set(${Prefix}_INCLUDE_DIRS ${TPL_${Prefix}_INCLUDE_DIRS} CACHE STRING
+      "TPL_${Prefix}_INCLUDE_DIRS was found." FORCE)
  endif()
  if (TPL_${Prefix}_LIBRARIES)
-    set(${Prefix}_LIBRARIES ${TPL_${Prefix}_LIBRARIES} CACHE STRING "TPL_${Prefix}_LIBRARIES was found." FORCE)
+    set(${Prefix}_LIBRARIES ${TPL_${Prefix}_LIBRARIES} CACHE STRING
+      "TPL_${Prefix}_LIBRARIES was found." FORCE)
  endif()

  # Quick return
@@ -714,11 +716,12 @@ function(mfem_export_mk_files)
  # Convert Boolean vars to YES/NO without writting the values to cache
  set(CONFIG_MK_BOOL_VARS MFEM_USE_MPI MFEM_USE_METIS MFEM_USE_METIS_5
      MFEM_DEBUG MFEM_USE_EXCEPTIONS MFEM_USE_GZSTREAM MFEM_USE_LIBUNWIND
-      MFEM_USE_LAPACK MFEM_THREAD_SAFE MFEM_USE_OPENMP MFEM_USE_MEMALLOC
-      MFEM_USE_SUNDIALS MFEM_USE_MESQUITE MFEM_USE_SUITESPARSE MFEM_USE_SUPERLU
-      MFEM_USE_STRUMPACK MFEM_USE_GECKO MFEM_USE_GNUTLS MFEM_USE_NETCDF
-      MFEM_USE_PETSC MFEM_USE_MPFR MFEM_USE_SIDRE MFEM_USE_CONDUIT
-      MFEM_USE_PUMI)
+      MFEM_USE_LAPACK MFEM_THREAD_SAFE MFEM_USE_OPENMP MFEM_USE_LEGACY_OPENMP
+      MFEM_USE_MEMALLOC MFEM_USE_SUNDIALS MFEM_USE_MESQUITE MFEM_USE_SUITESPARSE
+      MFEM_USE_SUPERLU MFEM_USE_STRUMPACK MFEM_USE_GECKO MFEM_USE_GNUTLS
+      MFEM_USE_NETCDF MFEM_USE_PETSC MFEM_USE_MPFR MFEM_USE_SIDRE
+      MFEM_USE_CONDUIT MFEM_USE_PUMI MFEM_USE_MM MFEM_USE_CUDA MFEM_USE_OCCA
+      MFEM_USE_RAJA)
  foreach(var ${CONFIG_MK_BOOL_VARS})
    if (${var})
      set(${var} YES)
@@ -726,6 +729,7 @@ function(mfem_export_mk_files)
      set(${var} NO)
    endif()
  endforeach()
+  # TODO: Add support for MFEM_USE_CUDA=YES
  set(MFEM_CXX ${CMAKE_CXX_COMPILER})
  set(MFEM_CPPFLAGS "")
  string(STRIP "${CMAKE_CXX_FLAGS_${BUILD_TYPE}} ${CMAKE_CXX_FLAGS}"
@@ -15,6 +15,9 @@
 //
 // Otherwise, use the local file: _config.hpp.

+#ifndef MFEM_CONFIG_HPP
+#define MFEM_CONFIG_HPP
+
 #ifdef MFEM_BUILD_DIR
 #define MFEM_QUOTE(a) #a
 #define MFEM_MAKE_PATH(x,y) MFEM_QUOTE(x/y)
@@ -52,3 +55,10 @@
 #error Building with PUMI (MFEM_USE_PUMI=YES) requires MPI (MFEM_USE_MPI=YES)
 #endif
 #endif // MFEM_USE_MPI not defined
+
+// CUDA requires the memory manager
+#if defined(MFEM_USE_CUDA) && !defined(MFEM_USE_MM)
+#error Building with CUDA (MFEM_USE_CUDA=YES) requires MFEM_USE_MM=YES
+#endif
+
+#endif // MFEM_CONFIG_HPP
@@ -30,6 +30,12 @@
 #define MFEM_VERSION_MINOR (((MFEM_VERSION)/100)%100)
 #define MFEM_VERSION_PATCH ((MFEM_VERSION)%100)

+// The absolute path of the MFEM source prefix
+// #define MFEM_SOURCE_DIR "@MFEM_SOURCE_DIR@"
+
+// The absolute path of the MFEM installation prefix
+// #define MFEM_INSTALL_DIR "@MFEM_INSTALL_DIR@"
+
 // Description of the git commit used to build MFEM.
 // #define MFEM_GIT_STRING "@MFEM_GIT_STRING@"

@@ -62,9 +68,12 @@
 // allocation and de-allocation.
 // #define MFEM_THREAD_SAFE

-// Enable experimental OpenMP support. Requires MFEM_THREAD_SAFE.
+// Enable the OpenMP backend.
 // #define MFEM_USE_OPENMP

+// [Deprecated] Enable experimental OpenMP support. Requires MFEM_THREAD_SAFE.
+// #define MFEM_USE_LEGACY_OPENMP
+
 // Internal MFEM option: enable group/batch allocation for some small objects.
 // #define MFEM_USE_MEMALLOC

@@ -112,6 +121,19 @@
 // Enable MFEM functionality based on the PUMI library
 // #define MFEM_USE_PUMI

+// Build the GPU/CUDA-enabled version of the MFEM library.
+// Requires a CUDA compiler (nvcc).
+// #define MFEM_USE_CUDA
+
+// Enable functionality based on the RAJA library.
+// #define MFEM_USE_RAJA
+
+// Enable functionality based on the OCCA library.
+// #define MFEM_USE_OCCA
+
+// Enable MFEM's internal Memory Manager (needed e.g. for MFEM_USE_CUDA)
+// #define MFEM_USE_MM
+
 // Version of HYPRE used for building MFEM.
 // #define MFEM_HYPRE_VERSION @MFEM_HYPRE_VERSION@

@@ -10,34 +10,41 @@
 # Software Foundation) version 2.1 dated February 1999.

 # Variables corresponding to defines in config.hpp (YES, NO, or value)
-MFEM_VERSION         = @MFEM_VERSION@
-MFEM_VERSION_STRING  = @MFEM_VERSION_STRING@
-MFEM_GIT_STRING      = @MFEM_GIT_STRING@
-MFEM_USE_MPI         = @MFEM_USE_MPI@
-MFEM_USE_METIS       = @MFEM_USE_METIS@
-MFEM_USE_METIS_5     = @MFEM_USE_METIS_5@
-MFEM_DEBUG           = @MFEM_DEBUG@
-MFEM_USE_EXCEPTIONS  = @MFEM_USE_EXCEPTIONS@
-MFEM_USE_GZSTREAM    = @MFEM_USE_GZSTREAM@
-MFEM_USE_LIBUNWIND   = @MFEM_USE_LIBUNWIND@
-MFEM_USE_LAPACK      = @MFEM_USE_LAPACK@
-MFEM_THREAD_SAFE     = @MFEM_THREAD_SAFE@
-MFEM_USE_OPENMP      = @MFEM_USE_OPENMP@
-MFEM_USE_MEMALLOC    = @MFEM_USE_MEMALLOC@
-MFEM_TIMER_TYPE      = @MFEM_TIMER_TYPE@
-MFEM_USE_SUNDIALS    = @MFEM_USE_SUNDIALS@
-MFEM_USE_MESQUITE    = @MFEM_USE_MESQUITE@
-MFEM_USE_SUITESPARSE = @MFEM_USE_SUITESPARSE@
-MFEM_USE_SUPERLU     = @MFEM_USE_SUPERLU@
-MFEM_USE_STRUMPACK   = @MFEM_USE_STRUMPACK@
-MFEM_USE_GECKO       = @MFEM_USE_GECKO@
-MFEM_USE_GNUTLS      = @MFEM_USE_GNUTLS@
-MFEM_USE_NETCDF      = @MFEM_USE_NETCDF@
-MFEM_USE_PETSC       = @MFEM_USE_PETSC@
-MFEM_USE_MPFR        = @MFEM_USE_MPFR@
-MFEM_USE_SIDRE       = @MFEM_USE_SIDRE@
-MFEM_USE_CONDUIT     = @MFEM_USE_CONDUIT@
-MFEM_USE_PUMI        = @MFEM_USE_PUMI@
+MFEM_VERSION           = @MFEM_VERSION@
+MFEM_VERSION_STRING    = @MFEM_VERSION_STRING@
+MFEM_SOURCE_DIR        = @MFEM_SOURCE_DIR@
+MFEM_INSTALL_DIR       = @MFEM_INSTALL_DIR@
+MFEM_GIT_STRING        = @MFEM_GIT_STRING@
+MFEM_USE_MPI           = @MFEM_USE_MPI@
+MFEM_USE_METIS         = @MFEM_USE_METIS@
+MFEM_USE_METIS_5       = @MFEM_USE_METIS_5@
+MFEM_DEBUG             = @MFEM_DEBUG@
+MFEM_USE_EXCEPTIONS    = @MFEM_USE_EXCEPTIONS@
+MFEM_USE_GZSTREAM      = @MFEM_USE_GZSTREAM@
+MFEM_USE_LIBUNWIND     = @MFEM_USE_LIBUNWIND@
+MFEM_USE_LAPACK        = @MFEM_USE_LAPACK@
+MFEM_THREAD_SAFE       = @MFEM_THREAD_SAFE@
+MFEM_USE_LEGACY_OPENMP = @MFEM_USE_LEGACY_OPENMP@
+MFEM_USE_OPENMP        = @MFEM_USE_OPENMP@
+MFEM_USE_MEMALLOC      = @MFEM_USE_MEMALLOC@
+MFEM_TIMER_TYPE        = @MFEM_TIMER_TYPE@
+MFEM_USE_SUNDIALS      = @MFEM_USE_SUNDIALS@
+MFEM_USE_MESQUITE      = @MFEM_USE_MESQUITE@
+MFEM_USE_SUITESPARSE   = @MFEM_USE_SUITESPARSE@
+MFEM_USE_SUPERLU       = @MFEM_USE_SUPERLU@
+MFEM_USE_STRUMPACK     = @MFEM_USE_STRUMPACK@
+MFEM_USE_GECKO         = @MFEM_USE_GECKO@
+MFEM_USE_GNUTLS        = @MFEM_USE_GNUTLS@
+MFEM_USE_NETCDF        = @MFEM_USE_NETCDF@
+MFEM_USE_PETSC         = @MFEM_USE_PETSC@
+MFEM_USE_MPFR          = @MFEM_USE_MPFR@
+MFEM_USE_SIDRE         = @MFEM_USE_SIDRE@
+MFEM_USE_CONDUIT       = @MFEM_USE_CONDUIT@
+MFEM_USE_PUMI          = @MFEM_USE_PUMI@
+MFEM_USE_CUDA          = @MFEM_USE_CUDA@
+MFEM_USE_RAJA          = @MFEM_USE_RAJA@
+MFEM_USE_OCCA          = @MFEM_USE_OCCA@
+MFEM_USE_MM            = @MFEM_USE_MM@

 # Compiler, compile options, and link options
 MFEM_CXX       = @MFEM_CXX@
@@ -65,5 +72,8 @@ MFEM_MPIEXEC    = @MFEM_MPIEXEC@
 MFEM_MPIEXEC_NP = @MFEM_MPIEXEC_NP@
 MFEM_MPI_NP     = @MFEM_MPI_NP@

+# The NVCC compiler cannot link with -x=cu
+MFEM_LINK_FLAGS := $(filter-out -x=cu, $(MFEM_FLAGS))
+
 # Optional extra configuration
@MFEM_CONFIG_EXTRA@
@@ -26,7 +26,8 @@ option(MFEM_USE_GZSTREAM "Enable gzstream for compressed data streams." OFF)
 option(MFEM_USE_LIBUNWIND "Enable backtrace for errors." OFF)
 option(MFEM_USE_LAPACK "Enable LAPACK usage" OFF)
 option(MFEM_THREAD_SAFE "Enable thread safety" OFF)
-option(MFEM_USE_OPENMP "Enable OpenMP usage" OFF)
+option(MFEM_USE_OPENMP "Enable the OpenMP backend" OFF)
+option(MFEM_USE_LEGACY_OPENMP "Enable legacy OpenMP usage" OFF)
 option(MFEM_USE_MEMALLOC "Enable the internal MEMALLOC option." ON)
 option(MFEM_USE_SUNDIALS "Enable SUNDIALS usage" OFF)
 option(MFEM_USE_MESQUITE "Enable MESQUITE usage" OFF)
@@ -41,6 +42,10 @@ option(MFEM_USE_MPFR "Enable MPFR usage." OFF)
 option(MFEM_USE_SIDRE "Enable Axom/Sidre usage" OFF)
 option(MFEM_USE_CONDUIT "Enable Conduit usage" OFF)
 option(MFEM_USE_PUMI "Enable PUMI" OFF)
+option(MFEM_USE_MM "Enable MFEM's memory manager" OFF)
+option(MFEM_USE_CUDA "Enable CUDA" OFF)
+option(MFEM_USE_OCCA "Enable OCCA" OFF)
+option(MFEM_USE_RAJA "Enable RAJA" OFF)

 set(MFEM_MPI_NP 4 CACHE STRING "Number of processes used for MPI tests")

@@ -58,13 +63,16 @@ option(MFEM_ENABLE_MINIAPPS "Build all of the miniapps" OFF)
 # set(CXX g++)
 # set(MPICXX mpicxx)

+# Set the target CUDA architecture
+set(CUDA_ARCH "sm_60" CACHE STRING "Target CUDA architecture.")
+
 set(MFEM_DIR ${CMAKE_CURRENT_SOURCE_DIR})

 # The *_DIR paths below will be the first place searched for the corresponding
 # headers and library. If these fail, then standard cmake search is performed.
 # Note: if the variables are already in the cache, they are not overwritten.

-set(HYPRE_DIR "${MFEM_DIR}/../hypre-2.10.0b/src/hypre" CACHE PATH
+set(HYPRE_DIR "${MFEM_DIR}/../hypre/src/hypre" CACHE PATH
    "Path to the hypre library.")
 # If hypre was compiled to depend on BLAS and LAPACK:
 # set(HYPRE_REQUIRED_PACKAGES "BLAS" "LAPACK" CACHE STRING
@@ -102,6 +110,7 @@ set(SuperLUDist_REQUIRED_PACKAGES "MPI" "BLAS" "ParMETIS" CACHE STRING
 set(STRUMPACK_DIR "${MFEM_DIR}/../STRUMPACK-build" CACHE PATH
    "Path to the STRUMPACK library.")
 # STRUMPACK may also depend on "OpenMP", depending on how it was compiled.
+# Starting with v2.2.0 of STRUMPACK, ParMETIS and Scotch are optional.
 set(STRUMPACK_REQUIRED_PACKAGES "MPI" "MPI_Fortran" "ParMETIS" "METIS"
    "ScaLAPACK" "Scotch/ptscotch/ptscotcherr/scotch/scotcherr" CACHE STRING
    "Additional packages required by STRUMPACK.")
@@ -109,7 +118,8 @@ set(STRUMPACK_REQUIRED_PACKAGES "MPI" "MPI_Fortran" "ParMETIS" "METIS"
 # set(STRUMPACK_REQUIRED_LIBRARIES "gfortran" "mpi_mpifh" CACHE STRING
 #     "Additional libraries required by STRUMPACK.")

-# The Scotch library, required by STRUMPACK
+# The Scotch library, required by STRUMPACK <= v2.1.0, optional in STRUMPACK >=
+# v2.2.0.
 set(Scotch_DIR "${MFEM_DIR}/../scotch_6.0.4" CACHE PATH
    "Path to the Scotch and PT-Scotch libraries.")
 set(Scotch_REQUIRED_PACKAGES "Threads" CACHE STRING
@@ -151,6 +161,9 @@ set(Axom_REQUIRED_PACKAGES "Conduit/relay" CACHE STRING
 set(PUMI_DIR "${MFEM_DIR}/../pumi-2.1.0" CACHE STRING
    "Directory where PUMI is installed")

+set(OCCA_DIR "${MFEM_DIR}/../occa" CACHE PATH "Path to OCCA")
+set(RAJA_DIR "${MFEM_DIR}/../raja" CACHE PATH "Path to RAJA")
+
 set(BLAS_INCLUDE_DIRS "" CACHE STRING "Path to BLAS headers.")
 set(BLAS_LIBRARIES "" CACHE STRING "The BLAS library.")
 set(LAPACK_INCLUDE_DIRS "" CACHE STRING "Path to LAPACK headers.")
@@ -21,8 +21,13 @@ NOTMAC := $(subst Darwin,,$(shell uname -s))
 CXX = g++
 MPICXX = mpicxx

-OPTIM_FLAGS = -O3
-DEBUG_FLAGS = -g -Wall
+BASE_FLAGS  = -std=c++11
+OPTIM_FLAGS = -O3 $(BASE_FLAGS)
+DEBUG_FLAGS = -g $(XCOMPILER)-Wall $(BASE_FLAGS)
+
+# Prefixes for passing flags to the compiler and linker when using CXX or MPICXX
+CXX_XCOMPILER =
+CXX_XLINKER   = -Wl,

 # Destination location of make install
 # PREFIX = $(HOME)/mfem
@@ -33,33 +38,41 @@ INSTALL = /usr/bin/install
 STATIC = YES
 SHARED = NO

+# CUDA configuration options
+CUDA_CXX = nvcc
+CUDA_ARCH = sm_60
+CUDA_FLAGS = -x=cu --expt-extended-lambda -arch=$(CUDA_ARCH)
+# Prefixes for passing flags to the host compiler and linker when using CUDA_CXX
+CUDA_XCOMPILER = -Xcompiler=
+CUDA_XLINKER   = -Xlinker=
+
 ifneq ($(NOTMAC),)
   AR      = ar
   ARFLAGS = cruv
   RANLIB  = ranlib
-   PICFLAG = -fPIC
+   PICFLAG = $(XCOMPILER)-fPIC
   SO_EXT  = so
   SO_VER  = so.$(MFEM_VERSION_STRING)
-   BUILD_SOFLAGS = -shared -Wl,-soname,libmfem.$(SO_VER)
-   BUILD_RPATH = -Wl,-rpath,$(BUILD_REAL_DIR)
+   BUILD_SOFLAGS = -shared $(XLINKER)-soname,libmfem.$(SO_VER)
+   BUILD_RPATH = $(XLINKER)-rpath,$(BUILD_REAL_DIR)
   INSTALL_SOFLAGS = $(BUILD_SOFLAGS)
-   INSTALL_RPATH = -Wl,-rpath,@MFEM_LIB_DIR@
+   INSTALL_RPATH = $(XLINKER)-rpath,@MFEM_LIB_DIR@
 else
   # Silence "has no symbols" warnings on Mac OS X
   AR      = ar
   ARFLAGS = Scruv
   RANLIB  = ranlib -no_warning_for_no_symbols
-   PICFLAG = -fPIC
+   PICFLAG = $(XCOMPILER)-fPIC
   SO_EXT  = dylib
   SO_VER  = $(MFEM_VERSION_STRING).dylib
-   MAKE_SOFLAGS = -Wl,-dylib,-install_name,$(1)/libmfem.$(SO_VER),\
+   MAKE_SOFLAGS = $(XLINKER)-dylib,-install_name,$(1)/libmfem.$(SO_VER),\
      -compatibility_version,$(MFEM_VERSION_STRING),\
      -current_version,$(MFEM_VERSION_STRING),\
      -undefined,dynamic_lookup
   BUILD_SOFLAGS = $(subst $1 ,,$(call MAKE_SOFLAGS,$(BUILD_REAL_DIR)))
-   BUILD_RPATH = -Wl,-undefined,dynamic_lookup
+   BUILD_RPATH = $(XLINKER)-undefined,dynamic_lookup
   INSTALL_SOFLAGS = $(subst $1 ,,$(call MAKE_SOFLAGS,$(MFEM_LIB_DIR)))
-   INSTALL_RPATH = -Wl,-undefined,dynamic_lookup
+   INSTALL_RPATH = $(XLINKER)-undefined,dynamic_lookup
 endif

 # Set CXXFLAGS to overwrite the default selection of DEBUG_FLAGS/OPTIM_FLAGS
@@ -82,31 +95,36 @@ MFEM_MPI_NP = 4
 # config.hpp. The values below are the defaults for generating the actual values
 # in config.mk and config.hpp.

-MFEM_USE_MPI         = NO
-MFEM_USE_METIS       = $(MFEM_USE_MPI)
-MFEM_USE_METIS_5     = NO
-MFEM_DEBUG           = NO
-MFEM_USE_EXCEPTIONS  = NO
-MFEM_USE_GZSTREAM    = NO
-MFEM_USE_LIBUNWIND   = NO
-MFEM_USE_LAPACK      = NO
-MFEM_THREAD_SAFE     = NO
-MFEM_USE_OPENMP      = NO
-MFEM_USE_MEMALLOC    = YES
-MFEM_TIMER_TYPE      = $(if $(NOTMAC),2,4)
-MFEM_USE_SUNDIALS    = NO
-MFEM_USE_MESQUITE    = NO
-MFEM_USE_SUITESPARSE = NO
-MFEM_USE_SUPERLU     = NO
-MFEM_USE_STRUMPACK   = NO
-MFEM_USE_GECKO       = NO
-MFEM_USE_GNUTLS      = NO
-MFEM_USE_NETCDF      = NO
-MFEM_USE_PETSC       = NO
-MFEM_USE_MPFR        = NO
-MFEM_USE_SIDRE       = NO
-MFEM_USE_CONDUIT     = NO
-MFEM_USE_PUMI        = NO
+MFEM_USE_MPI           = NO
+MFEM_USE_METIS         = $(MFEM_USE_MPI)
+MFEM_USE_METIS_5       = NO
+MFEM_DEBUG             = NO
+MFEM_USE_EXCEPTIONS    = NO
+MFEM_USE_GZSTREAM      = NO
+MFEM_USE_LIBUNWIND     = NO
+MFEM_USE_LAPACK        = NO
+MFEM_THREAD_SAFE       = NO
+MFEM_USE_OPENMP        = NO
+MFEM_USE_LEGACY_OPENMP = NO
+MFEM_USE_MEMALLOC      = YES
+MFEM_TIMER_TYPE        = $(if $(NOTMAC),2,4)
+MFEM_USE_SUNDIALS      = NO
+MFEM_USE_MESQUITE      = NO
+MFEM_USE_SUITESPARSE   = NO
+MFEM_USE_SUPERLU       = NO
+MFEM_USE_STRUMPACK     = NO
+MFEM_USE_GECKO         = NO
+MFEM_USE_GNUTLS        = NO
+MFEM_USE_NETCDF        = NO
+MFEM_USE_PETSC         = NO
+MFEM_USE_MPFR          = NO
+MFEM_USE_SIDRE         = NO
+MFEM_USE_CONDUIT       = NO
+MFEM_USE_PUMI          = NO
+MFEM_USE_CUDA          = NO
+MFEM_USE_RAJA          = NO
+MFEM_USE_OCCA          = NO
+MFEM_USE_MM            = NO

 # Compile and link options for zlib.
 ZLIB_DIR =
@@ -118,7 +136,7 @@ LIBUNWIND_OPT = -g
 LIBUNWIND_LIB = $(if $(NOTMAC),-lunwind -ldl,)

 # HYPRE library configuration (needed to build the parallel version)
-HYPRE_DIR = @MFEM_DIR@/../hypre-2.10.0b/src/hypre
+HYPRE_DIR = @MFEM_DIR@/../hypre/src/hypre
 HYPRE_OPT = -I$(HYPRE_DIR)/include
 HYPRE_LIB = -L$(HYPRE_DIR)/lib -lHYPRE

@@ -136,6 +154,8 @@ ifeq ($(MFEM_USE_SUPERLU)$(MFEM_USE_STRUMPACK),NONO)
 else
   # ParMETIS: currently needed by SuperLU or STRUMPACK. We assume that METIS 5
   # (included with ParMETIS) is installed in the same location.
+   # Starting with STRUMPACK v2.2.0, ParMETIS is an optional dependency while
+   # METIS is still required.
   METIS_DIR = @MFEM_DIR@/../parmetis-4.0.3
   METIS_OPT = -I$(METIS_DIR)/include
   METIS_LIB = -L$(METIS_DIR)/lib -lparmetis -lmetis
@@ -147,7 +167,7 @@ LAPACK_OPT =
 LAPACK_LIB = $(if $(NOTMAC),-llapack -lblas,-framework Accelerate)

 # OpenMP configuration
-OPENMP_OPT = -fopenmp
+OPENMP_OPT = $(XCOMPILER)-fopenmp
 OPENMP_LIB =

 # Used when MFEM_TIMER_TYPE = 2
@@ -183,7 +203,8 @@ SUPERLU_DIR = @MFEM_DIR@/../SuperLU_DIST_5.1.0
 SUPERLU_OPT = -I$(SUPERLU_DIR)/SRC
 SUPERLU_LIB = -Wl,-rpath,$(SUPERLU_DIR)/SRC -L$(SUPERLU_DIR)/SRC -lsuperlu_dist

-# SCOTCH library configuration (required by STRUMPACK)
+# SCOTCH library configuration (required by STRUMPACK <= v2.1.0, optional in
+# STRUMPACK >= v2.2.0)
 SCOTCH_DIR = @MFEM_DIR@/../scotch_6.0.4
 SCOTCH_OPT = -I$(SCOTCH_DIR)/include
 SCOTCH_LIB = -L$(SCOTCH_DIR)/lib -lptscotch -lptscotcherr -lscotch -lscotcherr\
@@ -279,6 +300,23 @@ PUMI_OPT = -I$(PUMI_DIR)/include
 PUMI_LIB = -L$(PUMI_DIR)/lib -lpumi -lcrv -lma -lmds -lapf -lpcu -lgmi -lparma\
   -llion -lmth -lapf_zoltan -lspr

+# CUDA library configuration (currently not needed)
+CUDA_OPT =
+CUDA_LIB =
+
+# OCCA library configuration
+OCCA_DIR = @MFEM_DIR@/../occa
+OCCA_OPT = -I$(OCCA_DIR)/include
+OCCA_LIB = $(XLINKER)-rpath,$(OCCA_DIR)/lib -L$(OCCA_DIR)/lib -locca
+
+# RAJA library configuration
+RAJA_DIR = @MFEM_DIR@/../raja
+RAJA_OPT = -I$(RAJA_DIR)/include
+ifdef CUB_DIR
+   RAJA_OPT += -I$(CUB_DIR)
+endif
+RAJA_LIB = $(XLINKER)-rpath,$(RAJA_DIR)/lib -L$(RAJA_DIR)/lib -lRAJA
+
 # If YES, enable some informational messages
 VERBOSE = NO

@@ -18,6 +18,8 @@ run_prefix=""
 run_vg="valgrind --leak-check=full --show-reachable=yes --track-origins=yes"
 run_suffix="-no-vis"
 skip_gen_meshes="yes"
+# filter-out device runs ("no") or non-device runs ("yes"):
+device_runs="no"
 cur_dir="${PWD}"
 mfem_dir="$(cd "$(dirname "$0")"/.. && pwd)"
 mfem_build_dir=""
@@ -148,6 +150,11 @@ function extract_sample_runs()
   if [ "$skip_gen_meshes" == "yes" ]; then
      runs=`printf "%s" "$runs" | grep -v ".* -m .*\.gen"`
   fi
+   if [ "$device_runs" == "yes" ]; then
+      runs=`printf "%s" "$runs" | grep ".* -d .*"`
+   else
+      runs=`printf "%s" "$runs" | grep -v ".* -d .*"`
+   fi
   IFS=$'\n'
   runs=(${runs})
   IFS="${old_IFS}"
@@ -169,6 +176,9 @@ function help_message()
      -g <dir> <pattern>
                  Specify explicitly a group (dir + file pattern) to run; This
                  option can be used multiple times to define multiple groups
+      -dev        configure only sample runs using devices.
+                  To test with a parallel build, the parallel (-p|-par) option
+                  should be set first on the command line.
      -v          Enable valgrind
      -o <dir>    [${output_dir:-"<empty>: output goes to stdout"}]
                  If not empty, save output to files inside <dir>
@@ -253,7 +263,7 @@ case "$1" in
   -h|-help)
      opt_help="yes"
      ;;
-   -p|-parallel)
+   -p|-par)
      mfem_config="MFEM_USE_MPI=YES MFEM_DEBUG=NO"
      ;;
   -g)
@@ -264,6 +274,11 @@ case "$1" in
      groups=("${groups[@]}" "${test_group}")
      shift 2
      ;;
+   -dev)
+       device_runs="yes"
+       mfem_config+=" MFEM_USE_CUDA=YES MFEM_USE_MM=YES \
+MFEM_USE_OCCA=YES MFEM_USE_RAJA=YES MFEM_USE_OPENMP=YES"
+      ;;
   -v)
      valgrind="yes"
      ;;
@@ -294,6 +309,10 @@ case "$1" in
   -n)
      run_prefix="echo"
      ;;
+   -*)
+      echo "unknown option: '$1'"
+      exit 1
+      ;;
   *=*)
      eval $1
      ;;
@@ -19,10 +19,10 @@ elements
 1 5 0 1 4 3 9 10 13 12
 1 5 3 4 7 6 12 13 16 15
 1 5 2 3 6 5 11 12 15 14
-1 5 8 9 12 11 17 18 21 20
-1 5 9 10 13 12 18 19 22 21
-1 5 12 13 16 15 21 22 25 24
 1 5 11 12 15 14 20 21 24 23
+1 5 12 13 16 15 21 22 25 24
+1 5 9 10 13 12 18 19 22 21
+1 5 8 9 12 11 17 18 21 20

 boundary
 24
@@ -35,6 +35,10 @@ namespace mfem {
 * - HypreParMatrix and HypreParVector
 * - HypreSolver and other \link hypre.hpp hypre classes\endlink
 *
+ * <H3>Main GPU classes</H3>
+ * - Device
+ * - MemoryManager
+ *
 * <H3>Example codes</H3>
 * - <a class="el" href="examples_2ex1_8cpp_source.html">Example 1</a>: nodal H1 FEM for the Laplace problem
 * - <a class="el" href="examples_2ex1p_8cpp_source.html">Example 1p</a>: parallel nodal H1 FEM for the Laplace problem
@@ -124,6 +128,7 @@ namespace mfem {
 * - <a class="el" href="display-basis_8cpp_source.html">Display Basis</a>: visualize finite element basis functions
 * - <a class="el" href="load-dc_8cpp_source.html">Load DC</a>: visualize fields saved via DataCollection classes
 * - <a class="el" href="convert-dc_8cpp_source.html">Convert DC</a>: convert between diffirent DataCollection formats
+ * - <a class="el" href="lor-transfer_8cpp_source.html">LOR Transfer</a>: map functions between high-order and low-order refined spaces
 * - <a class="el" href="miniapps_2performance_2ex1_8cpp_source.html">HPC Example 1</a>: high-performance nodal H1 FEM for the Laplace problem
 * - <a class="el" href="miniapps_2performance_2ex1p_8cpp_source.html">HPC Example 1p</a>: high-performance parallel nodal H1 FEM for the Laplace problem
 *
@@ -28,6 +28,7 @@ list(APPEND ALL_EXE_SRCS
  ex19.cpp
  ex20.cpp
  ex22.cpp
+  ex23.cpp
  )

 if (MFEM_USE_MPI)
@@ -53,6 +54,7 @@ if (MFEM_USE_MPI)
    ex19p.cpp
    ex20p.cpp
    ex22p.cpp
+    ex23p.cpp
    )
 endif()

@@ -86,6 +88,15 @@ foreach(SRC_FILE ${ALL_EXE_SRCS})
  endif()
 endforeach()

+# If STRUMPACK is enabled, add a test run that uses it.
+if (MFEM_USE_STRUMPACK)
+  add_test(NAME ex11p_strumpack_np=4
+    COMMAND ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} ${MFEM_MPI_NP}
+    ${MPIEXEC_PREFLAGS}
+    $<TARGET_FILE:ex11p> "-no-vis" "--strumpack"
+    ${MPIEXEC_POSTFLAGS})
+endif()
+
 # Include the examples/sundials directory if SUNDIALS is enabled.
 if (MFEM_USE_SUNDIALS)
  add_subdirectory(sundials)
@@ -25,6 +25,14 @@
 //               ex1 -m ../data/mobius-strip.mesh
 //               ex1 -m ../data/mobius-strip.mesh -o -1 -sc
 //
+// Device sample runs:
+//               ex1 -pa -d cuda
+//               ex1 -pa -d raja-cuda
+//               ex1 -pa -d occa-cuda
+//               ex1 -pa -d raja-omp
+//               ex1 -pa -d occa-omp
+//               ex1 -m ../data/beam-hex.mesh -pa -d cuda
+//
 // Description:  This example code demonstrates the use of MFEM to define a
 //               simple finite element discretization of the Laplace problem
 //               -Delta u = 1 with homogeneous Dirichlet boundary conditions.
@@ -53,7 +61,9 @@ int main(int argc, char *argv[])
   const char *mesh_file = "../data/star.mesh";
   int order = 1;
   bool static_cond = false;
-   bool visualization = 1;
+   bool pa = false;
+   const char *device = "cpu";
+   bool visualization = true;

   OptionsParser args(argc, argv);
   args.AddOption(&mesh_file, "-m", "--mesh",
@@ -63,6 +73,10 @@ int main(int argc, char *argv[])
                  " isoparametric space.");
   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                  "--no-visualization",
                  "Enable or disable GLVis visualization.");
@@ -134,48 +148,65 @@ int main(int argc, char *argv[])
   b->AddDomainIntegrator(new DomainLFIntegrator(one));
   b->Assemble();

-   // 7. Define the solution vector x as a finite element grid function
+   // 7. Set device config parameters from the command line options and switch
+   //    to working on the device.
+   Device::Configure(device);
+   Device::Print();
+   Device::Enable();
+
+   // 8. Define the solution vector x as a finite element grid function
   //    corresponding to fespace. Initialize x with initial guess of zero,
   //    which satisfies the boundary conditions.
   GridFunction x(fespace);
   x = 0.0;

-   // 8. Set up the bilinear form a(.,.) on the finite element space
+   // 9. Set up the bilinear form a(.,.) on the finite element space
   //    corresponding to the Laplacian operator -Delta, by adding the Diffusion
   //    domain integrator.
   BilinearForm *a = new BilinearForm(fespace);
+   if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
   a->AddDomainIntegrator(new DiffusionIntegrator(one));

-   // 9. Assemble the bilinear form and the corresponding linear system,
-   //    applying any necessary transformations such as: eliminating boundary
-   //    conditions, applying conforming constraints for non-conforming AMR,
-   //    static condensation, etc.
+   // 10. Assemble the bilinear form and the corresponding linear system,
+   //     applying any necessary transformations such as: eliminating boundary
+   //     conditions, applying conforming constraints for non-conforming AMR,
+   //     static condensation, etc.
   if (static_cond) { a->EnableStaticCondensation(); }
   a->Assemble();

-   SparseMatrix A;
+   OperatorPtr A;
   Vector B, X;
   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);

-   cout << "Size of linear system: " << A.Height() << endl;
+   cout << "Size of linear system: " << A->Height() << endl;

+   // 11. Solve the linear system A X = B.
+   if (!pa)
+   {
 #ifndef MFEM_USE_SUITESPARSE
-   // 10. Define a simple symmetric Gauss-Seidel preconditioner and use it to
-   //     solve the system A X = B with PCG.
-   GSSmoother M(A);
-   PCG(A, M, B, X, 1, 200, 1e-12, 0.0);
+      // Use a simple symmetric Gauss-Seidel preconditioner with PCG.
+      GSSmoother M((SparseMatrix&)(*A));
+      PCG(*A, M, B, X, 1, 200, 1e-12, 0.0);
 #else
-   // 10. If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.
-   UMFPackSolver umf_solver;
-   umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
-   umf_solver.SetOperator(A);
-   umf_solver.Mult(B, X);
+      // If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.
+      UMFPackSolver umf_solver;
+      umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+      umf_solver.SetOperator(*A);
+      umf_solver.Mult(B, X);
 #endif
+   }
+   else // No preconditioning for now in partial assembly mode.
+   {
+      CG(*A, B, X, 1, 2000, 1e-12, 0.0);
+   }

-   // 11. Recover the solution as a finite element grid function.
+   // 12. Recover the solution as a finite element grid function.
   a->RecoverFEMSolution(X, *b, x);

-   // 12. Save the refined mesh and the solution. This output can be viewed later
+   // 13. Switch back to the host.
+   Device::Disable();
+
+   // 14. Save the refined mesh and the solution. This output can be viewed later
   //     using GLVis: "glvis -m refined.mesh -g sol.gf".
   ofstream mesh_ofs("refined.mesh");
   mesh_ofs.precision(8);
@@ -184,7 +215,7 @@ int main(int argc, char *argv[])
   sol_ofs.precision(8);
   x.Save(sol_ofs);

-   // 13. Send the solution by socket to a GLVis server.
+   // 15. Send the solution by socket to a GLVis server.
   if (visualization)
   {
      char vishost[] = "localhost";
@@ -194,7 +225,7 @@ int main(int argc, char *argv[])
      sol_sock << "solution\n" << *mesh << x << flush;
   }

-   // 14. Free the used memory.
+   // 16. Free the used memory.
   delete a;
   delete b;
   delete fespace;
@@ -261,8 +261,7 @@ int main(int argc, char *argv[])
         strumpack->SetPrintSolveStatistics(false);
         strumpack->SetKrylovSolver(strumpack::KrylovSolver::DIRECT);
         strumpack->SetReorderingStrategy(strumpack::ReorderingStrategy::METIS);
-         strumpack->SetMC64Job(strumpack::MC64Job::NONE);
-         // strumpack->SetSymmetricPattern(true);
+         strumpack->DisableMatching();
         strumpack->SetOperator(*Arow);
         strumpack->SetFromCommandLine();
         precond = strumpack;
@@ -25,6 +25,11 @@
 //               mpirun -np 4 ex1p -m ../data/mobius-strip.mesh
 //               mpirun -np 4 ex1p -m ../data/mobius-strip.mesh -o -1 -sc
 //
+// Device sample runs:
+//               mpirun -np 4 ex1p -pa -d cuda
+//               mpirun -np 4 ex1p -pa -d occa-cuda
+//               mpirun -np 4 ex1p -pa -d raja-omp
+//
 // Description:  This example code demonstrates the use of MFEM to define a
 //               simple finite element discretization of the Laplace problem
 //               -Delta u = 1 with homogeneous Dirichlet boundary conditions.
@@ -59,7 +64,9 @@ int main(int argc, char *argv[])
   const char *mesh_file = "../data/star.mesh";
   int order = 1;
   bool static_cond = false;
-   bool visualization = 1;
+   bool pa = false;
+   const char *device = "cpu";
+   bool visualization = true;

   OptionsParser args(argc, argv);
   args.AddOption(&mesh_file, "-m", "--mesh",
@@ -69,6 +76,10 @@ int main(int argc, char *argv[])
                  " isoparametric space.");
   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                  "--no-visualization",
                  "Enable or disable GLVis visualization.");
@@ -166,49 +177,58 @@ int main(int argc, char *argv[])
   b->AddDomainIntegrator(new DomainLFIntegrator(one));
   b->Assemble();

-   // 9. Define the solution vector x as a parallel finite element grid function
-   //    corresponding to fespace. Initialize x with initial guess of zero,
-   //    which satisfies the boundary conditions.
+   // 9. Set device config parameters from the command line options and switch
+   //    to working on the device.
+   Device::Configure(device);
+   if (myid == 0) { Device::Print(); }
+   Device::Enable();
+
+   // 10. Define the solution vector x as a parallel finite element grid function
+   //     corresponding to fespace. Initialize x with initial guess of zero,
+   //     which satisfies the boundary conditions.
   ParGridFunction x(fespace);
   x = 0.0;

-   // 10. Set up the parallel bilinear form a(.,.) on the finite element space
+   // 11. Set up the parallel bilinear form a(.,.) on the finite element space
   //     corresponding to the Laplacian operator -Delta, by adding the Diffusion
   //     domain integrator.
   ParBilinearForm *a = new ParBilinearForm(fespace);
+   if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
   a->AddDomainIntegrator(new DiffusionIntegrator(one));

-   // 11. Assemble the parallel bilinear form and the corresponding linear
+   // 12. Assemble the parallel bilinear form and the corresponding linear
   //     system, applying any necessary transformations such as: parallel
   //     assembly, eliminating boundary conditions, applying conforming
   //     constraints for non-conforming AMR, static condensation, etc.
   if (static_cond) { a->EnableStaticCondensation(); }
   a->Assemble();

-   HypreParMatrix A;
+   OperatorPtr A;
   Vector B, X;
   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);

-   if (myid == 0)
-   {
-      cout << "Size of linear system: " << A.GetGlobalNumRows() << endl;
-   }
+   // 13. Solve the linear system A X = B.
+   //     * With full assembly, use the BoomerAMG preconditioner from hypre.
+   //     * With partial assembly, use no preconditioner, for now.
+   Solver *prec = NULL;
+   if (!pa) { prec = new HypreBoomerAMG; }
+   CGSolver cg(MPI_COMM_WORLD);
+   cg.SetRelTol(1e-12);
+   cg.SetMaxIter(2000);
+   cg.SetPrintLevel(1);
+   if (prec) { cg.SetPreconditioner(*prec); }
+   cg.SetOperator(*A);
+   cg.Mult(B, X);
+   delete prec;

-   // 12. Define and apply a parallel PCG solver for AX=B with the BoomerAMG
-   //     preconditioner from hypre.
-   HypreSolver *amg = new HypreBoomerAMG(A);
-   HyprePCG *pcg = new HyprePCG(A);
-   pcg->SetTol(1e-12);
-   pcg->SetMaxIter(200);
-   pcg->SetPrintLevel(2);
-   pcg->SetPreconditioner(*amg);
-   pcg->Mult(B, X);
-
-   // 13. Recover the parallel grid function corresponding to X. This is the
+   // 14. Recover the parallel grid function corresponding to X. This is the
   //     local finite element solution on each processor.
   a->RecoverFEMSolution(X, *b, x);

-   // 14. Save the refined mesh and the solution in parallel. This output can
+   // 15. Switch back to the host.
+   Device::Disable();
+
+   // 16. Save the refined mesh and the solution in parallel. This output can
   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
   {
      ostringstream mesh_name, sol_name;
@@ -224,7 +244,7 @@ int main(int argc, char *argv[])
      x.Save(sol_ofs);
   }

-   // 15. Send the solution by socket to a GLVis server.
+   // 17. Send the solution by socket to a GLVis server.
   if (visualization)
   {
      char vishost[] = "localhost";
@@ -235,9 +255,7 @@ int main(int argc, char *argv[])
      sol_sock << "solution\n" << *pmesh << x << flush;
   }

-   // 16. Free the used memory.
-   delete pcg;
-   delete amg;
+   // 18. Free the used memory.
   delete a;
   delete b;
   delete fespace;
@@ -0,0 +1,734 @@
+//                               MFEM Example 23
+//
+// Compile with: make ex23
+//
+// Sample runs:
+//    ex23 -m ../data/periodic-segment.mesh -p 0 -s 2 -dt 0.001 -vs 50
+//    ex23 -m ../data/periodic-segment.mesh -p 0 -s 12 -dt 0.01
+//    ex23 -m ../data/periodic-segment.mesh -p 0 -s 22 -dt 0.01
+//    ex23 -m ../data/periodic-segment.mesh -p 0 -s 32 -dt 0.005 -vs 10
+//    ex23 -m ../data/periodic-square.mesh -p 0 -dt 0.01
+//    ex23 -m ../data/periodic-square.mesh -p 0 -s 32 -dt 0.01
+//    ex23 -m ../data/periodic-hexagon.mesh -p 0 -d 0.001 -s 12 -dt 0.02
+//    ex23 -m ../data/periodic-hexagon.mesh -p 0 -d 0.001 -s 32 -dt 0.009 -vs 10
+//    ex23 -m ../data/periodic-square.mesh -p 1 -dt 0.01 -tf 9
+//    ex23 -m ../data/periodic-hexagon.mesh -p 1 -dt 0.01 -tf 9
+//    ex23 -m ../data/amr-quad.mesh -p 1 -dt 0.01 -tf 9 -vs 2
+//    ex23 -m ../data/disc-nurbs.mesh -p 1 -r 3 -dt 0.01 -tf 9
+//    ex23 -m ../data/disc-nurbs.mesh -p 2 -r 3 -dt 0.01 -tf 9
+//    ex23 -m ../data/disc-nurbs.mesh -p 3 -r 3 -dt 0.01 -tf 9 -d 0.02
+//    ex23 -m ../data/periodic-square.mesh -p 3 -r 3 -dt 0.025 -tf 9
+//    ex23 -m ../data/periodic-cube.mesh -p 0 -o 2 -dt 0.025 -tf 8
+//
+// Description:  This example code solves the time-dependent advection-diffusion
+//               equation
+//               du/dt - div(D grad(u)) + v.grad(u) = 0, where
+//               D is a diffusion coefficient,
+//               v is a given fluid velocity, and
+//               u0(x)=u(0,x) is a given initial condition.
+//
+//               The example demonstrates the use of Discontinuous Galerkin (DG)
+//               bilinear forms in MFEM (face integrators), the use of explicit,
+//               implicit, and implicit-explicit ODE time integrators, the
+//               definition of periodic boundary conditions through periodic
+//               meshes, as well as the use of GLVis for persistent
+//               visualization of a time-evolving solution. The saving of
+//               time-dependent data files for external visualization with
+//               VisIt (visit.llnl.gov) is also illustrated.
+//
+//               This example is a merger of examples 9 and 14.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Choice for the problem setup. The fluid velocity, initial condition and
+// boundary condition are chosen based on this parameter.
+int problem;
+
+// Velocity coefficient
+void velocity_function(const Vector &x, Vector &v);
+
+// Initial condition
+double u0_function(const Vector &x);
+
+// Mesh bounding box
+Vector bb_min, bb_max;
+
+
+/** A time-dependent operator for the right-hand side of the ODE for use with
+    explicit ODE solvers. The DG weak form of du/dt = div(D grad(u))-v.grad(u) is
+    M du/dt = - S u + K u + b, where M, S, and K are the mass,
+    stiffness, and advection matrices, and b describes sources and the flow on
+    the boundary.
+    This can be written as a general ODE,
+    du/dt = M^{-1} (-S u + K u + b), and this class is used to compute the RHS
+    and perform the solve for du/dt. */
+class EX_Evolution : public TimeDependentOperator
+{
+private:
+   SparseMatrix &M, &S, &K;
+   const Vector &b;
+
+   DSmoother M_prec;
+   CGSolver M_solver;
+
+   mutable Vector z;
+
+   void initA(double dt);
+
+public:
+   EX_Evolution(SparseMatrix &_M, SparseMatrix &_S, SparseMatrix &_K,
+                const Vector &_b);
+
+   virtual void Mult(const Vector &x, Vector &y) const;
+
+   virtual ~EX_Evolution() {}
+};
+
+/** A time-dependent operator for the right-hand side of the ODE for use with
+    implicit ODE solvers. The DG weak form of du/dt = div(D grad(u))-v.grad(u) is
+    [M + dt (S - K)] du/dt = - S u + K u + b, where M, S, and K are the mass,
+    stiffness, and advection matrices, and b describes sources and the flow on
+    the boundary.
+    This can be written as a general ODE,
+    du/dt = A^{-1} (-S u + K u + b) with A = [M + dt (S - K)], and this class is
+    used to perform the fully implicit solve for du/dt. */
+class IM_Evolution : public TimeDependentOperator
+{
+private:
+   SparseMatrix &M, &S, &K;
+   SparseMatrix *A;
+   const Vector &b;
+
+   DSmoother M_prec;
+   CGSolver M_solver;
+
+   DSmoother *A_prec;
+   GMRESSolver *A_solver;
+   double dt;
+
+   mutable Vector z;
+
+   void initA(double dt);
+
+public:
+   IM_Evolution(SparseMatrix &_M, SparseMatrix &_S, SparseMatrix &_K,
+                const Vector &_b);
+
+   virtual void Mult(const Vector &x, Vector &y) const;
+
+   virtual void ImplicitSolve(const double dt, const Vector &x, Vector &y);
+
+   virtual ~IM_Evolution() { delete A_solver; delete A_prec; delete A; }
+};
+
+/** A time-dependent operator for the right-hand side of the ODE for use with
+    IMEX (Implicit-Explicit) ODE solvers. The DG weak form of
+    du/dt = div(D grad(u))-v.grad(u) is
+    [M + dt S] du/dt = - S u + K u + b, where M, S, and K are the mass,
+    stiffness, and advection matrices, and b describes sources and the flow on
+    the boundary.
+    This can be written as a general ODE,
+    du/dt = A^{-1} (-S u + K u + b) with A = [M + dt (S - K)], and this class is
+    used to perform the implicit or explicit solve for du/dt. */
+class IMEX_Evolution : public TimeDependentOperator
+{
+private:
+   SparseMatrix &M, &S, &K;
+   SparseMatrix *A;
+   const Vector &b;
+
+   DSmoother M_prec;
+   CGSolver M_solver;
+
+   DSmoother *A_prec;
+   CGSolver *A_solver;
+   double dt;
+
+   mutable Vector z;
+
+   void initA(double dt);
+
+public:
+   IMEX_Evolution(SparseMatrix &_M, SparseMatrix &_S, SparseMatrix &_K,
+                  const Vector &_b);
+
+   virtual void ExplicitMult(const Vector &x, Vector &y) const;
+   virtual void Mult(const Vector &x, Vector &y) const;
+
+   virtual void ImplicitSolve(const double dt, const Vector &x, Vector &y);
+
+   virtual ~IMEX_Evolution() { delete A_solver; delete A_prec; delete A; }
+};
+
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options.
+   problem = 0;
+   const char *mesh_file = "../data/periodic-hexagon.mesh";
+   int ref_levels = 2;
+   int order = 3;
+   int ode_solver_type = 12;
+   double t_final = 10.0;
+   double d_coef = 0.01;
+   double dt = 0.01;
+   double sigma = -1.0;
+   double kappa = -1.0;
+   bool visualization = true;
+   bool visit = false;
+   bool binary = false;
+   int vis_steps = 5;
+
+   int precision = 8;
+   cout.precision(precision);
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&problem, "-p", "--problem",
+                  "Problem setup to use. See options in velocity_function().");
+   args.AddOption(&ref_levels, "-r", "--refine",
+                  "Number of times to refine the mesh uniformly.");
+   args.AddOption(&order, "-o", "--order",
+                  "Order (degree) of the finite elements.");
+   args.AddOption(&ode_solver_type, "-s", "--ode-solver",
+                  "ODE solver: 1 - Forward Euler, 2 - RK2, 3 - RK3 SSP,"
+                  " 4 - RK4, 5 - Generalized Alpha,\n\t"
+                  "11 - Backward Euler, 12 - SDIRK2, 13 - SDIRK3,\n\t"
+                  "22 - Implicit Midpoint, 23 SDIRK23, 24 - SDIRK34,\n\t"
+                  "31 - IMEX BE/FE, 32 - IMEX RK2.");
+   args.AddOption(&t_final, "-tf", "--t-final",
+                  "Final time; start time is 0.");
+   args.AddOption(&dt, "-dt", "--time-step",
+                  "Time step.");
+   args.AddOption(&d_coef, "-d", "--diff-coef",
+                  "Diffusion coefficient.");
+   args.AddOption(&sigma, "-s", "--sigma",
+                  "One of the two DG penalty parameters, typically +1/-1."
+                  " See the documentation of class DGDiffusionIntegrator.");
+   args.AddOption(&kappa, "-k", "--kappa",
+                  "One of the two DG penalty parameters, should be positive."
+                  " Negative values are replaced with (order+1)^2.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.AddOption(&visit, "-visit", "--visit-datafiles", "-no-visit",
+                  "--no-visit-datafiles",
+                  "Save data files for VisIt (visit.llnl.gov) visualization.");
+   args.AddOption(&binary, "-binary", "--binary-datafiles", "-ascii",
+                  "--ascii-datafiles",
+                  "Use binary (Sidre) or ascii format for VisIt data files.");
+   args.AddOption(&vis_steps, "-vs", "--visualization-steps",
+                  "Visualize every n-th timestep.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   if (kappa < 0)
+   {
+      kappa = (order+1)*(order+1);
+   }
+   args.PrintOptions(cout);
+
+   // 2. Define the ODE solver used for time integration. Several explicit
+   //    Runge-Kutta methods are available.
+   ODESolver *ode_solver = NULL;
+   switch (ode_solver_type)
+   {
+      // Explicit methods
+      case 1:  ode_solver = new ForwardEulerSolver; break;
+      case 2:  ode_solver = new RK2Solver(0.5); break; // midpoint method
+      case 3:  ode_solver = new RK3SSPSolver; break;
+      case 4:  ode_solver = new RK4Solver; break;
+      case 5:  ode_solver = new GeneralizedAlphaSolver(0.5); break;
+      // Implicit L-stable methods
+      case 11: ode_solver = new BackwardEulerSolver; break;
+      case 12: ode_solver = new SDIRK23Solver(2); break;
+      case 13: ode_solver = new SDIRK33Solver; break;
+      // Implicit A-stable methods (not L-stable)
+      case 22: ode_solver = new ImplicitMidpointSolver; break;
+      case 23: ode_solver = new SDIRK23Solver; break;
+      case 24: ode_solver = new SDIRK34Solver; break;
+      // Implicit-Explicit methods
+      case 31: ode_solver = new IMEX_BE_FE; break;
+      case 32: ode_solver = new IMEXRK2; break;
+      default:
+         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
+         return 3;
+   }
+
+   // 3. Read the serial mesh from the given mesh file on all processors. We can
+   //    handle geometrically periodic meshes in this code.
+   Mesh mesh(mesh_file, 1, 1);
+   int dim = mesh.Dimension();
+
+   // 4. Refine the mesh in serial to increase the resolution. In this example
+   //    we do 'ser_ref_levels' of uniform refinement, where 'ser_ref_levels' is
+   //    a command-line parameter. If the mesh is of NURBS type, we convert it
+   //    to a (piecewise-polynomial) high-order mesh.
+   for (int lev = 0; lev < ref_levels; lev++)
+   {
+      mesh.UniformRefinement();
+   }
+   if (mesh.NURBSext)
+   {
+      mesh.SetCurvature(max(order, 1));
+   }
+   mesh.GetBoundingBox(bb_min, bb_max, max(order, 1));
+
+   // 5. Define the parallel discontinuous DG finite element space on the
+   //    parallel refined mesh of the given polynomial order.
+   DG_FECollection fec(order, dim);
+   FiniteElementSpace fes(&mesh, &fec);
+
+   cout << "Number of unknowns: " << fes.GetVSize() << endl;
+
+   // 6. Set up and assemble the parallel bilinear and linear forms (and the
+   //    parallel hypre matrices) corresponding to the DG discretization. The
+   //    DGTraceIntegrator involves integrals over mesh interior faces.
+   ConstantCoefficient diff_coef(d_coef);
+   VectorFunctionCoefficient velocity(dim, velocity_function);
+   FunctionCoefficient u0(u0_function);
+
+   BilinearForm m(&fes);
+   m.AddDomainIntegrator(new MassIntegrator);
+
+   BilinearForm s(&fes);
+   s.AddDomainIntegrator(new DiffusionIntegrator(diff_coef));
+   s.AddInteriorFaceIntegrator(new DGDiffusionIntegrator(diff_coef, sigma,
+                                                         kappa));
+   s.AddBdrFaceIntegrator(new DGDiffusionIntegrator(diff_coef, sigma, kappa));
+
+   BilinearForm k(&fes);
+   k.AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
+   k.AddInteriorFaceIntegrator(
+      new TransposeIntegrator(new DGTraceIntegrator(velocity, 1.0, -0.5)));
+   k.AddBdrFaceIntegrator(
+      new TransposeIntegrator(new DGTraceIntegrator(velocity, 1.0, -0.5)));
+
+   LinearForm b(&fes);
+   b.AddBdrFaceIntegrator(
+      new DGDirichletLFIntegrator(u0, diff_coef, sigma, kappa));
+
+   int skip_zeros = 0;
+   m.Assemble(skip_zeros);
+   m.Finalize(skip_zeros);
+   s.Assemble(skip_zeros);
+   s.Finalize(skip_zeros);
+   k.Assemble(skip_zeros);
+   k.Finalize(skip_zeros);
+   b.Assemble();
+
+   // 7. Define the initial conditions, save the corresponding grid function to
+   //    a file and (optionally) save data in the VisIt format and initialize
+   //    GLVis visualization.
+   GridFunction u(&fes);
+   u.ProjectCoefficient(u0);
+
+   {
+      ofstream omesh("ex23.mesh");
+      omesh.precision(precision);
+      mesh.Print(omesh);
+      ofstream osol("ex23-init.gf");
+      osol.precision(precision);
+      u.Save(osol);
+   }
+
+   // Create data collection for solution output: either VisItDataCollection for
+   // ascii data files, or SidreDataCollection for binary data files.
+   DataCollection *dc = NULL;
+   if (visit)
+   {
+      if (binary)
+      {
+#ifdef MFEM_USE_SIDRE
+         dc = new SidreDataCollection("Example23", &mesh);
+#else
+         MFEM_ABORT("Must build with MFEM_USE_SIDRE=YES for binary output.");
+#endif
+      }
+      else
+      {
+         dc = new VisItDataCollection("Example23", &mesh);
+         dc->SetPrecision(precision);
+      }
+      dc->RegisterField("solution", &u);
+      dc->SetCycle(0);
+      dc->SetTime(0.0);
+      dc->Save();
+   }
+
+   socketstream sout;
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      sout.open(vishost, visport);
+      if (!sout)
+      {
+         cout << "Unable to connect to GLVis server at "
+              << vishost << ':' << visport << endl;
+         visualization = false;
+         cout << "GLVis visualization disabled.\n";
+      }
+      else
+      {
+         sout.precision(precision);
+         sout << "solution\n" << mesh << u;
+         sout << "pause\n";
+         sout << flush;
+         cout << "GLVis visualization paused."
+              << " Press space (in the GLVis window) to resume it.\n";
+      }
+   }
+
+   // 8. Define the time-dependent evolution operator describing the ODE
+   //    right-hand side, and perform time-integration (looping over the time
+   //    iterations, ti, with a time-step dt).
+
+   TimeDependentOperator *adv = NULL;
+   if (ode_solver_type < 10)
+   {
+      adv = new EX_Evolution(m.SpMat(), s.SpMat(), k.SpMat(), b);
+   }
+   else if (ode_solver_type < 30)
+   {
+      adv = new IM_Evolution(m.SpMat(), s.SpMat(), k.SpMat(), b);
+   }
+   else
+   {
+      adv = new IMEX_Evolution(m.SpMat(), s.SpMat(), k.SpMat(), b);
+   }
+
+   double t = 0.0;
+   adv->SetTime(t);
+   ode_solver->Init(*adv);
+
+   int n_steps = (int)ceil(t_final / dt);
+   double dt_real = t_final / n_steps;
+
+   for (int ti = 0; ti < n_steps; )
+   {
+      ode_solver->Step(u, t, dt_real);
+      ti++;
+
+      if (ti % vis_steps == 0 || ti == n_steps)
+      {
+         cout << "time step: " << ti << ", time: " << t << endl;
+
+         if (visualization)
+         {
+            sout << "solution\n" << mesh << u << flush;
+         }
+
+         if (visit)
+         {
+            dc->SetCycle(ti);
+            dc->SetTime(t);
+            dc->Save();
+         }
+      }
+   }
+
+   // 9. Save the final solution in parallel. This output can be viewed later
+   //    using GLVis: "glvis -np <np> -m ex23-mesh -g ex23-final".
+   {
+      ofstream osol("ex23-final.gf");
+      osol.precision(precision);
+      u.Save(osol);
+   }
+
+   // 10. Free the used memory.
+   delete ode_solver;
+   delete adv;
+   delete dc;
+
+   return 0;
+}
+
+
+// Implementation of class EX_Evolution
+EX_Evolution::EX_Evolution(SparseMatrix &_M, SparseMatrix &_S,
+                           SparseMatrix &_K, const Vector &_b)
+   : TimeDependentOperator(_M.Height()),
+     M(_M), S(_S), K(_K), b(_b), z(_M.Height())
+{
+   M_solver.SetPreconditioner(M_prec);
+   M_solver.SetOperator(M);
+
+   M_solver.iterative_mode = false;
+   M_solver.SetRelTol(1e-9);
+   M_solver.SetAbsTol(0.0);
+   M_solver.SetMaxIter(100);
+   M_solver.SetPrintLevel(0);
+}
+
+void EX_Evolution::Mult(const Vector &x, Vector &y) const
+{
+   // y = M^{-1} (-S x + K x + b)
+   K.Mult(x, z);
+   S.AddMult(x, z, -1.0);
+   z += b;
+   M_solver.Mult(z, y);
+}
+
+// Implementation of class IM_Evolution
+IM_Evolution::IM_Evolution(SparseMatrix &_M, SparseMatrix &_S,
+                           SparseMatrix &_K, const Vector &_b)
+   : TimeDependentOperator(_M.Height()),
+     M(_M), S(_S), K(_K), A(NULL), b(_b),
+     A_prec(NULL), A_solver(NULL), dt(-1.0), z(_M.Height())
+{
+   M_solver.SetPreconditioner(M_prec);
+   M_solver.SetOperator(M);
+
+   M_solver.iterative_mode = false;
+   M_solver.SetRelTol(1e-9);
+   M_solver.SetAbsTol(0.0);
+   M_solver.SetMaxIter(100);
+   M_solver.SetPrintLevel(0);
+}
+
+void IM_Evolution::initA(double _dt)
+{
+   if (fabs(dt - _dt) > 1e-4 * _dt)
+   {
+      delete A_solver;
+      delete A_prec;
+      delete A;
+
+      SparseMatrix * SK = Add(1.0, S, -1.0, K);
+      A = Add(1.0, M, _dt, *SK);
+      delete SK;
+      dt = _dt;
+
+      A_prec = new DSmoother(*A);
+      A_solver = new GMRESSolver;
+      A_solver->SetOperator(*A);
+      A_solver->SetPreconditioner(*A_prec);
+
+      A_solver->iterative_mode = false;
+      A_solver->SetRelTol(1e-9);
+      A_solver->SetAbsTol(0.0);
+      A_solver->SetMaxIter(100);
+      A_solver->SetPrintLevel(0);
+   }
+}
+
+void IM_Evolution::Mult(const Vector &x, Vector &y) const
+{
+   // y = M^{-1} (-S x + K x + b)
+   K.Mult(x, z);
+   S.AddMult(x, z, -1.0);
+   z += b;
+   M_solver.Mult(z, y);
+}
+
+void IM_Evolution::ImplicitSolve(const double _dt, const Vector &x, Vector &y)
+{
+   this->initA(_dt);
+
+   // y = (M + dt S - dt K)^{-1} (-S x + K x + b)
+   K.Mult(x, z);
+   S.AddMult(x, z, -1.0);
+   z += b;
+   A_solver->Mult(z, y);
+}
+
+// Implementation of class IMEX_Evolution
+IMEX_Evolution::IMEX_Evolution(SparseMatrix &_M, SparseMatrix &_S,
+                               SparseMatrix &_K, const Vector &_b)
+   : TimeDependentOperator(_M.Height()),
+     M(_M), S(_S), K(_K), A(NULL), b(_b),
+     A_prec(NULL), A_solver(NULL), dt(-1.0), z(_M.Height())
+{
+   M_solver.SetPreconditioner(M_prec);
+   M_solver.SetOperator(M);
+
+   M_solver.iterative_mode = false;
+   M_solver.SetRelTol(1e-9);
+   M_solver.SetAbsTol(0.0);
+   M_solver.SetMaxIter(100);
+   M_solver.SetPrintLevel(0);
+}
+
+void IMEX_Evolution::initA(double _dt)
+{
+   if (fabs(dt - _dt) > 1e-4 * _dt)
+   {
+      delete A_solver;
+      delete A_prec;
+      delete A;
+
+      A = Add(_dt, S, 1.0, M); // A = M + dt * S
+      dt = _dt;
+
+      A_prec = new DSmoother(*A);
+      A_solver = new CGSolver;
+      A_solver->SetOperator(*A);
+      A_solver->SetPreconditioner(*A_prec);
+
+      A_solver->iterative_mode = false;
+      A_solver->SetRelTol(1e-9);
+      A_solver->SetAbsTol(0.0);
+      A_solver->SetMaxIter(100);
+      A_solver->SetPrintLevel(0);
+   }
+}
+
+void IMEX_Evolution::Mult(const Vector &x, Vector &y) const
+{
+   // y = M^{-1} (-S x + K x + b)
+   K.Mult(x, z);
+   S.AddMult(x, z, -1.0);
+   z += b;
+   M_solver.Mult(z, y);
+}
+
+void IMEX_Evolution::ExplicitMult(const Vector &x, Vector &y) const
+{
+   // y = M^{-1} (K x + b)
+   K.Mult(x, z);
+   z += b;
+   M_solver.Mult(z, y);
+}
+
+void IMEX_Evolution::ImplicitSolve(const double _dt, const Vector &x, Vector &y)
+{
+   this->initA(_dt);
+   // y = (M + dt S)^{-1} (-S x + b)
+   S.Mult(x, z);
+   z *= -1.0;
+   z += b;
+   A_solver->Mult(z, y);
+}
+
+// Velocity coefficient
+void velocity_function(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+
+   // map to the reference [-1,1] domain
+   Vector X(dim);
+   for (int i = 0; i < dim; i++)
+   {
+      double center = (bb_min[i] + bb_max[i]) * 0.5;
+      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
+   }
+
+   switch (problem)
+   {
+      case 0:
+      {
+         // Translations in 1D, 2D, and 3D
+         switch (dim)
+         {
+            case 1: v(0) = 1.0; break;
+            case 2: v(0) = sqrt(2./3.); v(1) = sqrt(1./3.); break;
+            case 3: v(0) = sqrt(3./6.); v(1) = sqrt(2./6.); v(2) = sqrt(1./6.);
+               break;
+         }
+         break;
+      }
+      case 1:
+      case 2:
+      {
+         // Clockwise rotation in 2D around the origin
+         const double w = M_PI/2;
+         switch (dim)
+         {
+            case 1: v(0) = 1.0; break;
+            case 2: v(0) = w*X(1); v(1) = -w*X(0); break;
+            case 3: v(0) = w*X(1); v(1) = -w*X(0); v(2) = 0.0; break;
+         }
+         break;
+      }
+      case 3:
+      {
+         // Clockwise twisting rotation in 2D around the origin
+         const double w = M_PI/2;
+         double d = max((X(0)+1.)*(1.-X(0)),0.) * max((X(1)+1.)*(1.-X(1)),0.);
+         d = d*d;
+         switch (dim)
+         {
+            case 1: v(0) = 1.0; break;
+            case 2: v(0) = d*w*X(1); v(1) = -d*w*X(0); break;
+            case 3: v(0) = d*w*X(1); v(1) = -d*w*X(0); v(2) = 0.0; break;
+         }
+         break;
+      }
+   }
+}
+
+// Initial condition
+double u0_function(const Vector &x)
+{
+   int dim = x.Size();
+
+   // map to the reference [-1,1] domain
+   Vector X(dim);
+   for (int i = 0; i < dim; i++)
+   {
+      double center = (bb_min[i] + bb_max[i]) * 0.5;
+      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
+   }
+
+   switch (problem)
+   {
+      case 0:
+      case 1:
+      {
+         switch (dim)
+         {
+            case 1:
+               return exp(-40.*pow(X(0)-0.5,2));
+            case 2:
+            case 3:
+            {
+               double rx = 0.45, ry = 0.25, cx = 0., cy = -0.2, w = 10.;
+               if (dim == 3)
+               {
+                  const double s = (1. + 0.25*cos(2*M_PI*X(2)));
+                  rx *= s;
+                  ry *= s;
+               }
+               return ( erfc(w*(X(0)-cx-rx))*erfc(-w*(X(0)-cx+rx)) *
+                        erfc(w*(X(1)-cy-ry))*erfc(-w*(X(1)-cy+ry)) )/16;
+            }
+         }
+      }
+      case 2:
+      {
+         double x_ = X(0), y_ = X(1), rho, phi;
+         rho = hypot(x_, y_);
+         phi = atan2(y_, x_);
+         return pow(sin(M_PI*rho),2)*sin(3*phi);
+      }
+      case 3:
+      {
+         const double f = M_PI;
+         return sin(f*X(0))*sin(f*X(1));
+      }
+   }
+   return 0.0;
+}
+
+// Inflow boundary condition (zero for the problems considered in this example)
+double inflow_function(const Vector &x)
+{
+   switch (problem)
+   {
+      case 0:
+      case 1:
+      case 2:
+      case 3: return 0.0;
+   }
+   return 0.0;
+}
@@ -0,0 +1,797 @@
+//                       MFEM Example 23 - Parallel Version
+//
+// Compile with: make ex23p
+//
+// Sample runs:
+//    mpirun -np 4 ex23p -m ../data/periodic-segment.mesh -p 0 -s 2 -dt 0.001 -vs 50
+//    mpirun -np 4 ex23p -m ../data/periodic-segment.mesh -p 0 -s 12 -dt 0.01
+//    mpirun -np 4 ex23p -m ../data/periodic-segment.mesh -p 0 -s 22 -dt 0.01
+//    mpirun -np 4 ex23p -m ../data/periodic-segment.mesh -p 0 -s 32 -dt 0.005 -vs 10
+//    mpirun -np 4 ex23p -m ../data/periodic-square.mesh -p 0 -dt 0.01
+//    mpirun -np 4 ex23p -m ../data/periodic-square.mesh -p 0 -s 32 -dt 0.01
+//    mpirun -np 4 ex23p -m ../data/periodic-hexagon.mesh -p 0 -d 0.001 -s 12 -dt 0.02
+//    mpirun -np 4 ex23p -m ../data/periodic-hexagon.mesh -p 0 -d 0.001 -s 32 -dt 0.009 -vs 10
+//    mpirun -np 4 ex23p -m ../data/periodic-square.mesh -p 1 -dt 0.01 -tf 9
+//    mpirun -np 4 ex23p -m ../data/periodic-hexagon.mesh -p 1 -dt 0.01 -tf 9
+//    mpirun -np 4 ex23p -m ../data/amr-quad.mesh -p 1 -dt 0.01 -tf 9 -vs 2
+//    mpirun -np 4 ex23p -m ../data/disc-nurbs.mesh -p 1 -rp 1 -dt 0.01 -tf 9
+//    mpirun -np 4 ex23p -m ../data/disc-nurbs.mesh -p 2 -rp 1 -dt 0.01 -tf 9
+//    mpirun -np 4 ex23p -m ../data/disc-nurbs.mesh -p 3 -rp 1 -dt 0.01 -tf 9 -d 0.02
+//    mpirun -np 4 ex23p -m ../data/periodic-square.mesh -p 3 -rp 1 -dt 0.025 -tf 9
+//    mpirun -np 4 ex23p -m ../data/periodic-cube.mesh -p 0 -o 2 -dt 0.025 -tf 8
+//
+// Description:  This example code solves the time-dependent advection-diffusion
+//               equation
+//               du/dt - div(D grad(u)) + v.grad(u) = 0, where
+//               D is a diffusion coefficient,
+//               v is a given fluid velocity, and
+//               u0(x)=u(0,x) is a given initial condition.
+//
+//               The example demonstrates the use of Discontinuous Galerkin (DG)
+//               bilinear forms in MFEM (face integrators), the use of explicit,
+//               implicit, and implicit-explicit ODE time integrators, the
+//               definition of periodic boundary conditions through periodic
+//               meshes, as well as the use of GLVis for persistent
+//               visualization of a time-evolving solution. The saving of
+//               time-dependent data files for external visualization with
+//               VisIt (visit.llnl.gov) is also illustrated.
+//
+//               This example is a merger of examples 9 and 14.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Choice for the problem setup. The fluid velocity, initial condition and
+// boundary condition are chosen based on this parameter.
+int problem;
+
+// Velocity coefficient
+void velocity_function(const Vector &x, Vector &v);
+
+// Initial condition
+double u0_function(const Vector &x);
+
+// Mesh bounding box
+Vector bb_min, bb_max;
+
+/** A time-dependent operator for the right-hand side of the ODE for use with
+    explicit ODE solvers. The DG weak form of du/dt = div(D grad(u))-v.grad(u) is
+    M du/dt = - S u + K u + b, where M, S, and K are the mass,
+    stiffness, and advection matrices, and b describes sources and the flow on
+    the boundary.
+    This can be written as a general ODE,
+    du/dt = M^{-1} (-S u + K u + b), and this class is used to compute the RHS
+    and perform the solve for du/dt. */
+class EX_Evolution : public TimeDependentOperator
+{
+private:
+   HypreParMatrix &M, &S, &K;
+   const Vector &b;
+
+   HypreSmoother M_prec;
+   CGSolver M_solver;
+
+   mutable Vector z;
+
+   void initA(double dt);
+
+public:
+   EX_Evolution(HypreParMatrix &_M, HypreParMatrix &_S, HypreParMatrix &_K,
+                const Vector &_b);
+
+   virtual void Mult(const Vector &x, Vector &y) const;
+
+   virtual ~EX_Evolution() {}
+};
+
+/** A time-dependent operator for the right-hand side of the ODE for use with
+    implicit ODE solvers. The DG weak form of du/dt = div(D grad(u))-v.grad(u) is
+    [M + dt (S - K)] du/dt = - S u + K u + b, where M, S, and K are the mass,
+    stiffness, and advection matrices, and b describes sources and the flow on
+    the boundary.
+    This can be written as a general ODE,
+    du/dt = A^{-1} (-S u + K u + b) with A = [M + dt (S - K)], and this class is
+    used to perform the fully implicit solve for du/dt. */
+class IM_Evolution : public TimeDependentOperator
+{
+private:
+   HypreParMatrix &M, &S, &K;
+   HypreParMatrix *A;
+   const Vector &b;
+
+   HypreSmoother M_prec;
+   CGSolver M_solver;
+
+   HypreBoomerAMG *A_prec;
+   GMRESSolver *A_solver;
+   double dt;
+
+   mutable Vector z;
+
+   void initA(double dt);
+
+public:
+   IM_Evolution(HypreParMatrix &_M, HypreParMatrix &_S, HypreParMatrix &_K,
+                const Vector &_b);
+
+   virtual void Mult(const Vector &x, Vector &y) const;
+
+   virtual void ImplicitSolve(const double dt, const Vector &x, Vector &y);
+
+   virtual ~IM_Evolution() { delete A_solver; delete A_prec; delete A; }
+};
+
+/** A time-dependent operator for the right-hand side of the ODE for use with
+    IMEX (Implicit-Explicit) ODE solvers. The DG weak form of
+    du/dt = div(D grad(u))-v.grad(u) is
+    [M + dt S] du/dt = - S u + K u + b, where M, S, and K are the mass,
+    stiffness, and advection matrices, and b describes sources and the flow on
+    the boundary.
+    This can be written as a general ODE,
+    du/dt = A^{-1} (-S u + K u + b) with A = [M + dt (S - K)], and this class is
+    used to perform the implicit or explicit solve for du/dt. */
+class IMEX_Evolution : public TimeDependentOperator
+{
+private:
+   HypreParMatrix &M, &S, &K;
+   HypreParMatrix *A;
+   const Vector &b;
+
+   HypreSmoother M_prec;
+   CGSolver M_solver;
+
+   HypreBoomerAMG *A_prec;
+   CGSolver *A_solver;
+   double dt;
+
+   mutable Vector z;
+
+   void initA(double dt);
+
+public:
+   IMEX_Evolution(HypreParMatrix &_M, HypreParMatrix &_S, HypreParMatrix &_K,
+                  const Vector &_b);
+
+   virtual void ExplicitMult(const Vector &x, Vector &y) const;
+   virtual void Mult(const Vector &x, Vector &y) const;
+
+   virtual void ImplicitSolve(const double dt, const Vector &x, Vector &y);
+
+   virtual ~IMEX_Evolution() { delete A_solver; delete A_prec; delete A; }
+};
+
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   problem = 0;
+   const char *mesh_file = "../data/periodic-hexagon.mesh";
+   int ser_ref_levels = 2;
+   int par_ref_levels = 0;
+   int order = 3;
+   int ode_solver_type = 12;
+   double t_final = 10.0;
+   double d_coef = 0.01;
+   double dt = 0.01;
+   double sigma = -1.0;
+   double kappa = -1.0;
+   bool visualization = true;
+   bool visit = false;
+   bool binary = false;
+   int vis_steps = 5;
+
+   int precision = 8;
+   cout.precision(precision);
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&problem, "-p", "--problem",
+                  "Problem setup to use. See options in velocity_function().");
+   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
+                  "Number of times to refine the mesh uniformly in serial.");
+   args.AddOption(&par_ref_levels, "-rp", "--refine-parallel",
+                  "Number of times to refine the mesh uniformly in parallel.");
+   args.AddOption(&order, "-o", "--order",
+                  "Order (degree) of the finite elements.");
+   args.AddOption(&ode_solver_type, "-s", "--ode-solver",
+                  "ODE solver: 1 - Forward Euler, 2 - RK2, 3 - RK3 SSP,"
+                  " 4 - RK4, 5 - Generalized Alpha,\n\t"
+                  "11 - Backward Euler, 12 - SDIRK2, 13 - SDIRK3,\n\t"
+                  "22 - Implicit Midpoint, 23 SDIRK23, 24 - SDIRK34,\n\t"
+                  "31 - IMEX BE/FE, 32 - IMEX RK2.");
+   args.AddOption(&t_final, "-tf", "--t-final",
+                  "Final time; start time is 0.");
+   args.AddOption(&dt, "-dt", "--time-step",
+                  "Time step.");
+   args.AddOption(&d_coef, "-d", "--diff-coef",
+                  "Diffusion coefficient.");
+   args.AddOption(&sigma, "-s", "--sigma",
+                  "One of the two DG penalty parameters, typically +1/-1."
+                  " See the documentation of class DGDiffusionIntegrator.");
+   args.AddOption(&kappa, "-k", "--kappa",
+                  "One of the two DG penalty parameters, should be positive."
+                  " Negative values are replaced with (order+1)^2.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.AddOption(&visit, "-visit", "--visit-datafiles", "-no-visit",
+                  "--no-visit-datafiles",
+                  "Save data files for VisIt (visit.llnl.gov) visualization.");
+   args.AddOption(&binary, "-binary", "--binary-datafiles", "-ascii",
+                  "--ascii-datafiles",
+                  "Use binary (Sidre) or ascii format for VisIt data files.");
+   args.AddOption(&vis_steps, "-vs", "--visualization-steps",
+                  "Visualize every n-th timestep.");
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (kappa < 0)
+   {
+      kappa = (order+1)*(order+1);
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   // 3. Define the ODE solver used for time integration. Several explicit,
+   //    implicitit, and implicit-explicit Runge-Kutta methods are available.
+   ODESolver *ode_solver = NULL;
+
+   switch (ode_solver_type)
+   {
+      // Explicit methods
+      case 1:  ode_solver = new ForwardEulerSolver; break;
+      case 2:  ode_solver = new RK2Solver(0.5); break; // midpoint method
+      case 3:  ode_solver = new RK3SSPSolver; break;
+      case 4:  ode_solver = new RK4Solver; break;
+      case 5:  ode_solver = new GeneralizedAlphaSolver(0.5); break;
+      // Implicit L-stable methods
+      case 11: ode_solver = new BackwardEulerSolver; break;
+      case 12: ode_solver = new SDIRK23Solver(2); break;
+      case 13: ode_solver = new SDIRK33Solver; break;
+      // Implicit A-stable methods (not L-stable)
+      case 22: ode_solver = new ImplicitMidpointSolver; break;
+      case 23: ode_solver = new SDIRK23Solver; break;
+      case 24: ode_solver = new SDIRK34Solver; break;
+      // Implicit-Explicit methods
+      case 31: ode_solver = new IMEX_BE_FE; break;
+      case 32: ode_solver = new IMEXRK2; break;
+      default:
+         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
+         return 3;
+   }
+
+   // 4. Read the serial mesh from the given mesh file on all processors. We can
+   //    handle geometrically periodic meshes in this code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 5. Refine the mesh in serial to increase the resolution. In this example
+   //    we do 'ser_ref_levels' of uniform refinement, where 'ser_ref_levels' is
+   //    a command-line parameter. If the mesh is of NURBS type, we convert it
+   //    to a (piecewise-polynomial) high-order mesh.
+   for (int lev = 0; lev < ser_ref_levels; lev++)
+   {
+      mesh->UniformRefinement();
+   }
+   if (mesh->NURBSext)
+   {
+      mesh->SetCurvature(max(order, 1));
+   }
+   mesh->GetBoundingBox(bb_min, bb_max, max(order, 1));
+
+   // 6. Define the parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   for (int lev = 0; lev < par_ref_levels; lev++)
+   {
+      pmesh->UniformRefinement();
+   }
+
+   // 7. Define the parallel discontinuous DG finite element space on the
+   //    parallel refined mesh of the given polynomial order.
+   DG_FECollection fec(order, dim);
+   ParFiniteElementSpace *fes = new ParFiniteElementSpace(pmesh, &fec);
+
+   HYPRE_Int global_vSize = fes->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of unknowns: " << global_vSize << endl;
+   }
+
+   // 8. Set up and assemble the parallel bilinear and linear forms (and the
+   //    parallel hypre matrices) corresponding to the DG discretization. The
+   //    DGTraceIntegrator involves integrals over mesh interior faces.
+   ConstantCoefficient diff_coef(d_coef);
+   VectorFunctionCoefficient velocity(dim, velocity_function);
+   FunctionCoefficient u0(u0_function);
+
+   ParBilinearForm *m = new ParBilinearForm(fes);
+   m->AddDomainIntegrator(new MassIntegrator);
+
+   ParBilinearForm *s = new ParBilinearForm(fes);
+   s->AddDomainIntegrator(new DiffusionIntegrator(diff_coef));
+   s->AddInteriorFaceIntegrator(new DGDiffusionIntegrator(diff_coef, sigma,
+                                                          kappa));
+   s->AddBdrFaceIntegrator(new DGDiffusionIntegrator(diff_coef, sigma, kappa));
+
+   ParBilinearForm *k = new ParBilinearForm(fes);
+   k->AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
+   k->AddInteriorFaceIntegrator(
+      new TransposeIntegrator(new DGTraceIntegrator(velocity, 1.0, -0.5)));
+   k->AddBdrFaceIntegrator(
+      new TransposeIntegrator(new DGTraceIntegrator(velocity, 1.0, -0.5)));
+
+   ParLinearForm *b = new ParLinearForm(fes);
+   b->AddBdrFaceIntegrator(
+      new DGDirichletLFIntegrator(u0, diff_coef, sigma, kappa));
+
+   int skip_zeros = 0;
+   m->Assemble(skip_zeros);
+   m->Finalize(skip_zeros);
+   s->Assemble(skip_zeros);
+   s->Finalize(skip_zeros);
+   k->Assemble(skip_zeros);
+   k->Finalize(skip_zeros);
+   b->Assemble();
+
+   HypreParMatrix *M = m->ParallelAssemble();
+   HypreParMatrix *S = s->ParallelAssemble();
+   HypreParMatrix *K = k->ParallelAssemble();
+   HypreParVector *B = b->ParallelAssemble();
+
+   // 9. Define the initial conditions, save the corresponding grid function to
+   //    a file and (optionally) save data in the VisIt format and initialize
+   //    GLVis visualization.
+   ParGridFunction *u = new ParGridFunction(fes);
+   u->ProjectCoefficient(u0);
+   HypreParVector *U = u->GetTrueDofs();
+
+   {
+      ostringstream mesh_name, sol_name;
+      mesh_name << "ex23-mesh." << setfill('0') << setw(6) << myid;
+      sol_name << "ex23-init." << setfill('0') << setw(6) << myid;
+      ofstream omesh(mesh_name.str().c_str());
+      omesh.precision(precision);
+      pmesh->Print(omesh);
+      ofstream osol(sol_name.str().c_str());
+      osol.precision(precision);
+      u->Save(osol);
+   }
+
+   // Create data collection for solution output: either VisItDataCollection for
+   // ascii data files, or SidreDataCollection for binary data files.
+   DataCollection *dc = NULL;
+   if (visit)
+   {
+      if (binary)
+      {
+#ifdef MFEM_USE_SIDRE
+         dc = new SidreDataCollection("Example23-Parallel", pmesh);
+#else
+         MFEM_ABORT("Must build with MFEM_USE_SIDRE=YES for binary output.");
+#endif
+      }
+      else
+      {
+         dc = new VisItDataCollection("Example23-Parallel", pmesh);
+         dc->SetPrecision(precision);
+         // To save the mesh using MFEM's parallel mesh format:
+         // dc->SetFormat(DataCollection::PARALLEL_FORMAT);
+      }
+      dc->RegisterField("solution", u);
+      dc->SetCycle(0);
+      dc->SetTime(0.0);
+      dc->Save();
+   }
+
+   socketstream sout;
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      sout.open(vishost, visport);
+      if (!sout)
+      {
+         if (myid == 0)
+            cout << "Unable to connect to GLVis server at "
+                 << vishost << ':' << visport << endl;
+         visualization = false;
+         if (myid == 0)
+         {
+            cout << "GLVis visualization disabled.\n";
+         }
+      }
+      else
+      {
+         sout << "parallel " << num_procs << " " << myid << "\n";
+         sout.precision(precision);
+         sout << "solution\n" << *pmesh << *u;
+         sout << "pause\n";
+         sout << flush;
+         if (myid == 0)
+            cout << "GLVis visualization paused."
+                 << " Press space (in the GLVis window) to resume it.\n";
+      }
+   }
+
+   // 10. Define the time-dependent evolution operator describing the ODE
+   //     right-hand side, and perform time-integration (looping over the time
+   //     iterations, ti, with a time-step dt).
+
+   TimeDependentOperator *adv = NULL;
+   if (ode_solver_type < 10)
+   {
+      adv = new EX_Evolution(*M, *S, *K, *B);
+   }
+   else if (ode_solver_type < 30)
+   {
+      adv = new IM_Evolution(*M, *S, *K, *B);
+   }
+   else
+   {
+      adv = new IMEX_Evolution(*M, *S, *K, *B);
+   }
+
+   double t = 0.0;
+   adv->SetTime(t);
+   ode_solver->Init(*adv);
+
+   int n_steps = (int)ceil(t_final / dt);
+   double dt_real = t_final / n_steps;
+
+   for (int ti = 0; ti < n_steps; )
+   {
+      ode_solver->Step(*U, t, dt_real);
+      ti++;
+
+      if (ti % vis_steps == 0 || ti == n_steps)
+      {
+         if (myid == 0)
+         {
+            cout << "time step: " << ti << ", time: " << t << endl;
+         }
+
+         // 11. Extract the parallel grid function corresponding to the finite
+         //     element approximation U (the local solution on each processor).
+         *u = *U;
+
+         if (visualization)
+         {
+            sout << "parallel " << num_procs << " " << myid << "\n";
+            sout << "solution\n" << *pmesh << *u << flush;
+         }
+
+         if (visit)
+         {
+            dc->SetCycle(ti);
+            dc->SetTime(t);
+            dc->Save();
+         }
+      }
+   }
+
+   // 12. Save the final solution in parallel. This output can be viewed later
+   //     using GLVis: "glvis -np <np> -m ex23-mesh -g ex23-final".
+   {
+      *u = *U;
+      ostringstream sol_name;
+      sol_name << "ex23-final." << setfill('0') << setw(6) << myid;
+      ofstream osol(sol_name.str().c_str());
+      osol.precision(precision);
+      u->Save(osol);
+   }
+
+   // 13. Free the used memory.
+   delete U;
+   delete u;
+   delete B;
+   delete b;
+   delete K;
+   delete k;
+   delete S;
+   delete s;
+   delete M;
+   delete m;
+   delete fes;
+   delete pmesh;
+   delete ode_solver;
+   delete adv;
+   delete dc;
+
+   MPI_Finalize();
+   return 0;
+}
+
+
+// Implementation of class EX_Evolution
+EX_Evolution::EX_Evolution(HypreParMatrix &_M, HypreParMatrix &_S,
+                           HypreParMatrix &_K, const Vector &_b)
+   : TimeDependentOperator(_M.Height()),
+     M(_M), S(_S), K(_K), b(_b),
+     M_prec(M), M_solver(M.GetComm()), z(M.Height())
+{
+   M_prec.SetType(HypreSmoother::Jacobi);
+   M_solver.SetPreconditioner(M_prec);
+   M_solver.SetOperator(M);
+
+   M_solver.iterative_mode = false;
+   M_solver.SetRelTol(1e-9);
+   M_solver.SetAbsTol(0.0);
+   M_solver.SetMaxIter(100);
+   M_solver.SetPrintLevel(0);
+}
+
+void EX_Evolution::Mult(const Vector &x, Vector &y) const
+{
+   // y = M^{-1} (-S x + K x + b)
+   S.Mult(-1.0, x, 0.0, z);
+   K.Mult(1.0, x, 1.0, z);
+   z += b;
+   M_solver.Mult(z, y);
+}
+
+// Implementation of class IM_Evolution
+IM_Evolution::IM_Evolution(HypreParMatrix &_M, HypreParMatrix &_S,
+                           HypreParMatrix &_K, const Vector &_b)
+   : TimeDependentOperator(_M.Height()),
+     M(_M), S(_S), K(_K), A(NULL), b(_b),
+     M_prec(M), M_solver(M.GetComm()),
+     A_prec(NULL), A_solver(NULL), dt(-1.0), z(M.Height())
+{
+   M_prec.SetType(HypreSmoother::Jacobi);
+   M_solver.SetPreconditioner(M_prec);
+   M_solver.SetOperator(M);
+
+   M_solver.iterative_mode = false;
+   M_solver.SetRelTol(1e-9);
+   M_solver.SetAbsTol(0.0);
+   M_solver.SetMaxIter(100);
+   M_solver.SetPrintLevel(0);
+}
+
+void IM_Evolution::initA(double _dt)
+{
+   if (fabs(dt - _dt) > 1e-4 * _dt)
+   {
+      delete A_solver;
+      delete A_prec;
+      delete A;
+
+      HypreParMatrix * SK = Add(1.0, S, -1.0, K); // SK = S - K
+      A = Add(_dt, *SK, 1.0, M);                  // A = M + dt * (S - K)
+      delete SK;
+      dt = _dt;
+
+      A_prec = new HypreBoomerAMG(*A);
+      A_solver = new GMRESSolver(A->GetComm());
+      A_solver->SetOperator(*A);
+      A_solver->SetPreconditioner(*A_prec);
+
+      A_solver->iterative_mode = false;
+      A_solver->SetRelTol(1e-9);
+      A_solver->SetAbsTol(0.0);
+      A_solver->SetMaxIter(100);
+      A_solver->SetPrintLevel(0);
+   }
+}
+
+void IM_Evolution::Mult(const Vector &x, Vector &y) const
+{
+   // y = M^{-1} (-S x + K x + b)
+   S.Mult(-1.0, x, 0.0, z);
+   K.Mult(1.0, x, 1.0, z);
+   z += b;
+   M_solver.Mult(z, y);
+}
+
+void IM_Evolution::ImplicitSolve(const double _dt, const Vector &x, Vector &y)
+{
+   this->initA(_dt);
+
+   // y = (M + dt S - dt K)^{-1} (-S x + K x + b)
+   S.Mult(-1.0, x, 0.0, z);
+   K.Mult(1.0, x, 1.0, z);
+   z += b;
+   A_solver->Mult(z, y);
+}
+
+// Implementation of class IMEX_Evolution
+IMEX_Evolution::IMEX_Evolution(HypreParMatrix &_M, HypreParMatrix &_S,
+                               HypreParMatrix &_K, const Vector &_b)
+   : TimeDependentOperator(_M.Height()),
+     M(_M), S(_S), K(_K), A(NULL), b(_b),
+     M_prec(M), M_solver(M.GetComm()),
+     A_prec(NULL), A_solver(NULL), dt(-1.0), z(M.Height())
+{
+   M_prec.SetType(HypreSmoother::Jacobi);
+   M_solver.SetPreconditioner(M_prec);
+   M_solver.SetOperator(M);
+
+   M_solver.iterative_mode = false;
+   M_solver.SetRelTol(1e-9);
+   M_solver.SetAbsTol(0.0);
+   M_solver.SetMaxIter(100);
+   M_solver.SetPrintLevel(0);
+}
+
+void IMEX_Evolution::initA(double _dt)
+{
+   if (fabs(dt - _dt) > 1e-4 * _dt)
+   {
+      delete A_solver;
+      delete A_prec;
+      delete A;
+
+      A = Add(_dt, S, 1.0, M); // A = M + dt * S
+      dt = _dt;
+
+      A_prec = new HypreBoomerAMG(*A);
+      A_solver = new CGSolver(A->GetComm());
+      A_solver->SetOperator(*A);
+      A_solver->SetPreconditioner(*A_prec);
+
+      A_solver->iterative_mode = false;
+      A_solver->SetRelTol(1e-9);
+      A_solver->SetAbsTol(0.0);
+      A_solver->SetMaxIter(100);
+      A_solver->SetPrintLevel(0);
+   }
+}
+
+void IMEX_Evolution::Mult(const Vector &x, Vector &y) const
+{
+   // y = M^{-1} (-S x + K x + b)
+   S.Mult(-1.0, x, 0.0, z);
+   K.Mult(1.0, x, 1.0, z);
+   z += b;
+   M_solver.Mult(z, y);
+}
+
+void IMEX_Evolution::ExplicitMult(const Vector &x, Vector &y) const
+{
+   // y = M^{-1} (K x + b)
+   K.Mult(1.0, x, 0.0, z);
+   z += b;
+   M_solver.Mult(z, y);
+}
+
+void IMEX_Evolution::ImplicitSolve(const double _dt, const Vector &x, Vector &y)
+{
+   this->initA(_dt);
+   // y = (M + dt S)^{-1} (-S x + b)
+   S.Mult(-1.0, x, 0.0, z);
+   z += b;
+   A_solver->Mult(z, y);
+}
+
+// Velocity coefficient
+void velocity_function(const Vector &x, Vector &v)
+{
+   int dim = x.Size();
+
+   // map to the reference [-1,1] domain
+   Vector X(dim);
+   for (int i = 0; i < dim; i++)
+   {
+      double center = (bb_min[i] + bb_max[i]) * 0.5;
+      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
+   }
+
+   switch (problem)
+   {
+      case 0:
+      {
+         // Translations in 1D, 2D, and 3D
+         switch (dim)
+         {
+            case 1: v(0) = 1.0; break;
+            case 2: v(0) = sqrt(2./3.); v(1) = sqrt(1./3.); break;
+            case 3: v(0) = sqrt(3./6.); v(1) = sqrt(2./6.); v(2) = sqrt(1./6.);
+               break;
+         }
+         break;
+      }
+      case 1:
+      case 2:
+      {
+         // Clockwise rotation in 2D around the origin
+         const double w = M_PI/2;
+         switch (dim)
+         {
+            case 1: v(0) = 1.0; break;
+            case 2: v(0) = w*X(1); v(1) = -w*X(0); break;
+            case 3: v(0) = w*X(1); v(1) = -w*X(0); v(2) = 0.0; break;
+         }
+         break;
+      }
+      case 3:
+      {
+         // Clockwise twisting rotation in 2D around the origin
+         const double w = M_PI/2;
+         double d = max((X(0)+1.)*(1.-X(0)),0.) * max((X(1)+1.)*(1.-X(1)),0.);
+         d = d*d;
+         switch (dim)
+         {
+            case 1: v(0) = 1.0; break;
+            case 2: v(0) = d*w*X(1); v(1) = -d*w*X(0); break;
+            case 3: v(0) = d*w*X(1); v(1) = -d*w*X(0); v(2) = 0.0; break;
+         }
+         break;
+      }
+   }
+}
+
+// Initial condition
+double u0_function(const Vector &x)
+{
+   int dim = x.Size();
+
+   // map to the reference [-1,1] domain
+   Vector X(dim);
+   for (int i = 0; i < dim; i++)
+   {
+      double center = (bb_min[i] + bb_max[i]) * 0.5;
+      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
+   }
+
+   switch (problem)
+   {
+      case 0:
+      case 1:
+      {
+         switch (dim)
+         {
+            case 1:
+               return exp(-40.*pow(X(0)-0.5,2));
+            case 2:
+            case 3:
+            {
+               double rx = 0.45, ry = 0.25, cx = 0., cy = -0.2, w = 10.;
+               if (dim == 3)
+               {
+                  const double s = (1. + 0.25*cos(2*M_PI*X(2)));
+                  rx *= s;
+                  ry *= s;
+               }
+               return ( erfc(w*(X(0)-cx-rx))*erfc(-w*(X(0)-cx+rx)) *
+                        erfc(w*(X(1)-cy-ry))*erfc(-w*(X(1)-cy+ry)) )/16;
+            }
+         }
+      }
+      case 2:
+      {
+         double x_ = X(0), y_ = X(1), rho, phi;
+         rho = hypot(x_, y_);
+         phi = atan2(y_, x_);
+         return pow(sin(M_PI*rho),2)*sin(3*phi);
+      }
+      case 3:
+      {
+         const double f = M_PI;
+         return sin(f*X(0))*sin(f*X(1));
+      }
+   }
+   return 0.0;
+}
@@ -15,6 +15,11 @@
 //               ex6 -m ../data/square-disc-surf.mesh -o 2
 //               ex6 -m ../data/amr-quad.mesh
 //
+// Device sample runs:
+//               ex6 -pa -d cuda
+//               ex6 -pa -d occa-cuda
+//               ex6 -pa -d raja-omp
+//
 // Description:  This is a version of Example 1 with a simple adaptive mesh
 //               refinement loop. The problem being solved is again the Laplace
 //               equation -Delta u = 1 with homogeneous Dirichlet boundary
@@ -43,13 +48,19 @@ int main(int argc, char *argv[])
   // 1. Parse command-line options.
   const char *mesh_file = "../data/star.mesh";
   int order = 1;
-   bool visualization = 1;
+   bool pa = false;
+   const char *device = "cpu";
+   bool visualization = true;

   OptionsParser args(argc, argv);
   args.AddOption(&mesh_file, "-m", "--mesh",
                  "Mesh file to use.");
   args.AddOption(&order, "-o", "--order",
                  "Finite element order (polynomial degree).");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                  "--no-visualization",
                  "Enable or disable GLVis visualization.");
@@ -85,10 +96,15 @@ int main(int argc, char *argv[])
   H1_FECollection fec(order, dim);
   FiniteElementSpace fespace(&mesh, &fec);

-   // 5. As in Example 1, we set up bilinear and linear forms corresponding to
+   // 5. Set device config parameters from the command line options.
+   Device::Configure(device);
+   Device::Print();
+
+   // 6. As in Example 1, we set up bilinear and linear forms corresponding to
   //    the Laplace problem -\Delta u = 1. We don't assemble the discrete
   //    problem yet, this will be done in the main loop.
   BilinearForm a(&fespace);
+   if (pa) { a.SetAssemblyLevel(AssemblyLevel::PARTIAL); }
   LinearForm b(&fespace);

   ConstantCoefficient one(1.0);
@@ -98,18 +114,18 @@ int main(int argc, char *argv[])
   a.AddDomainIntegrator(integ);
   b.AddDomainIntegrator(new DomainLFIntegrator(one));

-   // 6. The solution vector x and the associated finite element grid function
+   // 7. The solution vector x and the associated finite element grid function
   //    will be maintained over the AMR iterations. We initialize it to zero.
   GridFunction x(&fespace);
   x = 0.0;

-   // 7. All boundary attributes will be used for essential (Dirichlet) BC.
+   // 8. All boundary attributes will be used for essential (Dirichlet) BC.
   MFEM_VERIFY(mesh.bdr_attributes.Size() > 0,
               "Boundary attributes required in the mesh.");
   Array<int> ess_bdr(mesh.bdr_attributes.Max());
   ess_bdr = 1;

-   // 8. Connect to GLVis.
+   // 9. Connect to GLVis.
   char vishost[] = "localhost";
   int  visport   = 19916;
   socketstream sol_sock;
@@ -118,23 +134,23 @@ int main(int argc, char *argv[])
      sol_sock.open(vishost, visport);
   }

-   // 9. Set up an error estimator. Here we use the Zienkiewicz-Zhu estimator
-   //    that uses the ComputeElementFlux method of the DiffusionIntegrator to
-   //    recover a smoothed flux (gradient) that is subtracted from the element
-   //    flux to get an error indicator. We need to supply the space for the
-   //    smoothed flux: an (H1)^sdim (i.e., vector-valued) space is used here.
+   // 10. Set up an error estimator. Here we use the Zienkiewicz-Zhu estimator
+   //     that uses the ComputeElementFlux method of the DiffusionIntegrator to
+   //     recover a smoothed flux (gradient) that is subtracted from the element
+   //     flux to get an error indicator. We need to supply the space for the
+   //     smoothed flux: an (H1)^sdim (i.e., vector-valued) space is used here.
   FiniteElementSpace flux_fespace(&mesh, &fec, sdim);
   ZienkiewiczZhuEstimator estimator(*integ, x, flux_fespace);
   estimator.SetAnisotropic();

-   // 10. A refiner selects and refines elements based on a refinement strategy.
+   // 11. A refiner selects and refines elements based on a refinement strategy.
   //     The strategy here is to refine elements with errors larger than a
   //     fraction of the maximum element error. Other strategies are possible.
   //     The refiner will call the given error estimator.
   ThresholdRefiner refiner(estimator);
   refiner.SetTotalErrorFraction(0.7);

-   // 11. The main AMR loop. In each iteration we solve the problem on the
+   // 12. The main AMR loop. In each iteration we solve the problem on the
   //     current mesh, visualize the solution, and refine the mesh.
   const int max_dofs = 50000;
   for (int it = 0; ; it++)
@@ -143,44 +159,55 @@ int main(int argc, char *argv[])
      cout << "\nAMR iteration " << it << endl;
      cout << "Number of unknowns: " << cdofs << endl;

-      // 12. Assemble the stiffness matrix and the right-hand side.
-      a.Assemble();
+      // 13. Assemble the right-hand side.
      b.Assemble();

-      // 13. Set Dirichlet boundary values in the GridFunction x.
+      // 14. Set Dirichlet boundary values in the GridFunction x.
      //     Determine the list of Dirichlet true DOFs in the linear system.
      Array<int> ess_tdof_list;
      x.ProjectBdrCoefficient(zero, ess_bdr);
      fespace.GetEssentialTrueDofs(ess_bdr, ess_tdof_list);

-      // 14. Create the linear system: eliminate boundary conditions, constrain
+      // 15. Switch to the device and assemble the stiffness matrix.
+      Device::Enable();
+      a.Assemble();
+
+      // 16. Create the linear system: eliminate boundary conditions, constrain
      //     hanging nodes and possibly apply other transformations. The system
      //     will be solved for true (unconstrained) DOFs only.
-      SparseMatrix A;
+      OperatorPtr A;
      Vector B, X;
+
      const int copy_interior = 1;
      a.FormLinearSystem(ess_tdof_list, x, b, A, X, B, copy_interior);

+      // 17. Solve the linear system A X = B.
+      if (!pa)
+      {
 #ifndef MFEM_USE_SUITESPARSE
-      // 15. Define a simple symmetric Gauss-Seidel preconditioner and use it to
-      //     solve the linear system with PCG.
-      GSSmoother M(A);
-      PCG(A, M, B, X, 3, 200, 1e-12, 0.0);
+         // Use a simple symmetric Gauss-Seidel preconditioner with PCG.
+         GSSmoother M((SparseMatrix&)(*A));
+         PCG(*A, M, B, X, 3, 200, 1e-12, 0.0);
 #else
-      // 15. If MFEM was compiled with SuiteSparse, use UMFPACK to solve the
-      //     the linear system.
-      UMFPackSolver umf_solver;
-      umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
-      umf_solver.SetOperator(A);
-      umf_solver.Mult(B, X);
+         // If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.
+         UMFPackSolver umf_solver;
+         umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+         umf_solver.SetOperator(*A);
+         umf_solver.Mult(B, X);
 #endif
+      }
+      else // No preconditioning for now in partial assembly mode.
+      {
+         CG(*A, B, X, 3, 2000, 1e-12, 0.0);
+      }

-      // 16. After solving the linear system, reconstruct the solution as a
+      // 18. After solving the linear system, reconstruct the solution as a
      //     finite element GridFunction. Constrained nodes are interpolated
      //     from true DOFs (it may therefore happen that x.Size() >= X.Size()).
+      Device::Disable();
      a.RecoverFEMSolution(X, b, x);

-      // 17. Send solution by socket to the GLVis server.
+      // 19. Send solution by socket to the GLVis server.
      if (visualization && sol_sock.good())
      {
         sol_sock.precision(8);
@@ -193,7 +220,7 @@ int main(int argc, char *argv[])
         break;
      }

-      // 18. Call the refiner to modify the mesh. The refiner calls the error
+      // 20. Call the refiner to modify the mesh. The refiner calls the error
      //     estimator to obtain element errors, then it selects elements to be
      //     refined and finally it modifies the mesh. The Stop() method can be
      //     used to determine if a stopping criterion was met.
@@ -204,7 +231,7 @@ int main(int argc, char *argv[])
         break;
      }

-      // 19. Update the space to reflect the new state of the mesh. Also,
+      // 21. Update the space to reflect the new state of the mesh. Also,
      //     interpolate the solution x so that it lies in the new space but
      //     represents the same function. This saves solver iterations later
      //     since we'll have a good initial guess of x in the next step.
@@ -213,7 +240,7 @@ int main(int argc, char *argv[])
      fespace.Update();
      x.Update();

-      // 20. Inform also the bilinear and linear forms that the space has
+      // 22. Inform also the bilinear and linear forms that the space has
      //     changed.
      a.Update();
      b.Update();
@@ -15,6 +15,11 @@
 //               mpirun -np 4 ex6p -m ../data/square-disc-surf.mesh -o 2
 //               mpirun -np 4 ex6p -m ../data/amr-quad.mesh
 //
+// Device sample runs:
+//               mpirun -np 4 ex6p -pa -d cuda
+//               mpirun -np 4 ex6p -pa -d occa-cuda
+//               mpirun -np 4 ex6p -pa -d raja-omp
+//
 // Description:  This is a version of Example 1 with a simple adaptive mesh
 //               refinement loop. The problem being solved is again the Laplace
 //               equation -Delta u = 1 with homogeneous Dirichlet boundary
@@ -49,6 +54,8 @@ int main(int argc, char *argv[])
   // 2. Parse command-line options.
   const char *mesh_file = "../data/star.mesh";
   int order = 1;
+   bool pa = false;
+   const char *device = "cpu";
   bool visualization = true;

   OptionsParser args(argc, argv);
@@ -56,6 +63,10 @@ int main(int argc, char *argv[])
                  "Mesh file to use.");
   args.AddOption(&order, "-o", "--order",
                  "Finite element order (polynomial degree).");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                  "--no-visualization",
                  "Enable or disable GLVis visualization.");
@@ -106,10 +117,15 @@ int main(int argc, char *argv[])
   H1_FECollection fec(order, dim);
   ParFiniteElementSpace fespace(&pmesh, &fec);

-   // 7. As in Example 1p, we set up bilinear and linear forms corresponding to
+   // 7. Set device config parameters from the command line options.
+   Device::Configure(device);
+   if (myid == 0) { Device::Print(); }
+
+   // 8. As in Example 1p, we set up bilinear and linear forms corresponding to
   //    the Laplace problem -\Delta u = 1. We don't assemble the discrete
   //    problem yet, this will be done in the main loop.
   ParBilinearForm a(&fespace);
+   if (pa) { a.SetAssemblyLevel(AssemblyLevel::PARTIAL); }
   ParLinearForm b(&fespace);

   ConstantCoefficient one(1.0);
@@ -118,12 +134,12 @@ int main(int argc, char *argv[])
   a.AddDomainIntegrator(integ);
   b.AddDomainIntegrator(new DomainLFIntegrator(one));

-   // 8. The solution vector x and the associated finite element grid function
+   // 9. The solution vector x and the associated finite element grid function
   //    will be maintained over the AMR iterations. We initialize it to zero.
   ParGridFunction x(&fespace);
   x = 0;

-   // 9. Connect to GLVis.
+   // 10. Connect to GLVis.
   char vishost[] = "localhost";
   int  visport   = 19916;

@@ -145,7 +161,7 @@ int main(int argc, char *argv[])
      sout.precision(8);
   }

-   // 10. Set up an error estimator. Here we use the Zienkiewicz-Zhu estimator
+   // 11. Set up an error estimator. Here we use the Zienkiewicz-Zhu estimator
   //     with L2 projection in the smoothing step to better handle hanging
   //     nodes and parallel partitioning. We need to supply a space for the
   //     discontinuous flux (L2) and a space for the smoothed flux (H(div) is
@@ -159,14 +175,14 @@ int main(int argc, char *argv[])
   // ParFiniteElementSpace smooth_flux_fes(&pmesh, &smooth_flux_fec, dim);
   L2ZienkiewiczZhuEstimator estimator(*integ, x, flux_fes, smooth_flux_fes);

-   // 11. A refiner selects and refines elements based on a refinement strategy.
+   // 12. A refiner selects and refines elements based on a refinement strategy.
   //     The strategy here is to refine elements with errors larger than a
   //     fraction of the maximum element error. Other strategies are possible.
   //     The refiner will call the given error estimator.
   ThresholdRefiner refiner(estimator);
   refiner.SetTotalErrorFraction(0.7);

-   // 12. The main AMR loop. In each iteration we solve the problem on the
+   // 13. The main AMR loop. In each iteration we solve the problem on the
   //     current mesh, visualize the solution, and refine the mesh.
   const int max_dofs = 100000;
   for (int it = 0; ; it++)
@@ -178,41 +194,48 @@ int main(int argc, char *argv[])
         cout << "Number of unknowns: " << global_dofs << endl;
      }

-      // 13. Assemble the stiffness matrix and the right-hand side. Note that
-      //     MFEM doesn't care at this point that the mesh is nonconforming
-      //     and parallel. The FE space is considered 'cut' along hanging
-      //     edges/faces, and also across processor boundaries.
-      a.Assemble();
-      b.Assemble();
-
-      // 14. Create the parallel linear system: eliminate boundary conditions,
-      //     constrain hanging nodes and nodes across processor boundaries.
-      //     The system will be solved for true (unconstrained/unique) DOFs only.
+      // 14. Assemble the right-hand side and determine the list of true
+      //     (i.e. parallel conforming) essential boundary dofs.
      Array<int> ess_tdof_list;
      fespace.GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+      b.Assemble();

-      HypreParMatrix A;
+      // 15. Switch to the device and assemble the stiffness matrix. Note that
+      //     MFEM doesn't care at this point that the mesh is nonconforming and
+      //     parallel.  The FE space is considered 'cut' along hanging
+      //     edges/faces, and also across processor boundaries.
+      Device::Enable();
+      a.Assemble();
+
+      // 16. Create the parallel linear system: eliminate boundary conditions.
+      //     The system will be solved for true (unconstrained/unique) DOFs only.
+      OperatorPtr A;
      Vector B, X;
+
      const int copy_interior = 1;
      a.FormLinearSystem(ess_tdof_list, x, b, A, X, B, copy_interior);

-      // 15. Define and apply a parallel PCG solver for AX=B with the BoomerAMG
-      //     preconditioner from hypre.
-      HypreBoomerAMG amg;
-      amg.SetPrintLevel(0);
-      CGSolver pcg(A.GetComm());
-      pcg.SetPreconditioner(amg);
-      pcg.SetOperator(A);
-      pcg.SetRelTol(1e-6);
-      pcg.SetMaxIter(200);
-      pcg.SetPrintLevel(3); // print the first and the last iterations only
-      pcg.Mult(B, X);
+      // 17. Solve the linear system A X = B.
+      //     * With full assembly, use the BoomerAMG preconditioner from hypre.
+      //     * With partial assembly, use no preconditioner, for now.
+      HypreBoomerAMG *amg = NULL;
+      if (!pa) { amg = new HypreBoomerAMG; amg->SetPrintLevel(0); }
+      CGSolver cg(MPI_COMM_WORLD);
+      cg.SetRelTol(1e-6);
+      cg.SetMaxIter(2000);
+      cg.SetPrintLevel(3); // print the first and the last iterations only
+      if (amg) { cg.SetPreconditioner(*amg); }
+      cg.SetOperator(*A);
+      cg.Mult(B, X);
+      delete amg;

-      // 16. Extract the parallel grid function corresponding to the finite element
-      //     approximation X. This is the local solution on each processor.
+      // 18. Switch back to the host and extract the parallel grid function
+      //     corresponding to the finite element approximation X. This is the
+      //     local solution on each processor.
+      Device::Disable();
      a.RecoverFEMSolution(X, b, x);

-      // 17. Send the solution by socket to a GLVis server.
+      // 19. Send the solution by socket to a GLVis server.
      if (visualization)
      {
         sout << "parallel " << num_procs << " " << myid << "\n";
@@ -228,7 +251,7 @@ int main(int argc, char *argv[])
         break;
      }

-      // 18. Call the refiner to modify the mesh. The refiner calls the error
+      // 20. Call the refiner to modify the mesh. The refiner calls the error
      //     estimator to obtain element errors, then it selects elements to be
      //     refined and finally it modifies the mesh. The Stop() method can be
      //     used to determine if a stopping criterion was met.
@@ -242,7 +265,7 @@ int main(int argc, char *argv[])
         break;
      }

-      // 19. Update the finite element space (recalculate the number of DOFs,
+      // 21. Update the finite element space (recalculate the number of DOFs,
      //     etc.) and create a grid function update matrix. Apply the matrix
      //     to any GridFunctions over the space. In this case, the update
      //     matrix is an interpolation matrix so the updated GridFunction will
@@ -250,7 +273,7 @@ int main(int argc, char *argv[])
      fespace.Update();
      x.Update();

-      // 20. Load balance the mesh, and update the space and solution. Currently
+      // 22. Load balance the mesh, and update the space and solution. Currently
      //     available only for nonconforming meshes.
      if (pmesh.Nonconforming())
      {
@@ -262,7 +285,7 @@ int main(int argc, char *argv[])
         x.Update();
      }

-      // 21. Inform also the bilinear and linear forms that the space has
+      // 23. Inform also the bilinear and linear forms that the space has
      //     changed.
      a.Update();
      b.Update();
@@ -132,8 +132,8 @@ int main(int argc, char *argv[])

   // 2. Read the mesh from the given mesh file. We can handle geometrically
   //    periodic meshes in this code.
-   Mesh *mesh = new Mesh(mesh_file, 1, 1);
-   int dim = mesh->Dimension();
+   Mesh mesh(mesh_file, 1, 1);
+   int dim = mesh.Dimension();

   // 3. Define the ODE solver used for time integration. Several explicit
   //    Runge-Kutta methods are available.
@@ -147,7 +147,6 @@ int main(int argc, char *argv[])
      case 6: ode_solver = new RK6Solver; break;
      default:
         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
-         delete mesh;
         return 3;
   }

@@ -157,18 +156,18 @@ int main(int argc, char *argv[])
   //    a (piecewise-polynomial) high-order mesh.
   for (int lev = 0; lev < ref_levels; lev++)
   {
-      mesh->UniformRefinement();
+      mesh.UniformRefinement();
   }
-   if (mesh->NURBSext)
+   if (mesh.NURBSext)
   {
-      mesh->SetCurvature(max(order, 1));
+      mesh.SetCurvature(max(order, 1));
   }
-   mesh->GetBoundingBox(bb_min, bb_max, max(order, 1));
+   mesh.GetBoundingBox(bb_min, bb_max, max(order, 1));

   // 5. Define the discontinuous DG finite element space of the given
   //    polynomial order on the refined mesh.
   DG_FECollection fec(order, dim);
-   FiniteElementSpace fes(mesh, &fec);
+   FiniteElementSpace fes(&mesh, &fec);

   cout << "Number of unknowns: " << fes.GetVSize() << endl;

@@ -208,7 +207,7 @@ int main(int argc, char *argv[])
   {
      ofstream omesh("ex9.mesh");
      omesh.precision(precision);
-      mesh->Print(omesh);
+      mesh.Print(omesh);
      ofstream osol("ex9-init.gf");
      osol.precision(precision);
      u.Save(osol);
@@ -222,14 +221,14 @@ int main(int argc, char *argv[])
      if (binary)
      {
 #ifdef MFEM_USE_SIDRE
-         dc = new SidreDataCollection("Example9", mesh);
+         dc = new SidreDataCollection("Example9", &mesh);
 #else
         MFEM_ABORT("Must build with MFEM_USE_SIDRE=YES for binary output.");
 #endif
      }
      else
      {
-         dc = new VisItDataCollection("Example9", mesh);
+         dc = new VisItDataCollection("Example9", &mesh);
         dc->SetPrecision(precision);
      }
      dc->RegisterField("solution", &u);
@@ -254,7 +253,7 @@ int main(int argc, char *argv[])
      else
      {
         sout.precision(precision);
-         sout << "solution\n" << *mesh << u;
+         sout << "solution\n" << mesh << u;
         sout << "pause\n";
         sout << flush;
         cout << "GLVis visualization paused."
@@ -286,7 +285,7 @@ int main(int argc, char *argv[])

         if (visualization)
         {
-            sout << "solution\n" << *mesh << u << flush;
+            sout << "solution\n" << mesh << u << flush;
         }

         if (visit)
@@ -22,9 +22,9 @@ MFEM_LIB_FILE = mfem_is_not_built
 -include $(CONFIG_MK)

 SEQ_EXAMPLES = ex1 ex2 ex3 ex4 ex5 ex6 ex7 ex8 ex9 ex10 ex14 ex15 ex16 ex17\
-  ex18 ex19 ex20 ex22
+  ex18 ex19 ex20 ex22 ex23
 PAR_EXAMPLES = ex1p ex2p ex3p ex4p ex5p ex6p ex7p ex8p ex9p ex10p ex11p ex12p\
- ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex22p
+ ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex22p ex23p

 ifeq ($(MFEM_USE_MPI),NO)
   EXAMPLES = $(SEQ_EXAMPLES)
@@ -96,6 +96,12 @@ ex15-test-seq: ex15
 	@$(call mfem-test,$<,, Serial example,-e 1)
 ex15p-test-par: ex15p
 	@$(call mfem-test,$<, $(RUN_MPI), Parallel example,-e 1)
+# Testing: optional tests
+ifeq ($(MFEM_USE_STRUMPACK),YES)
+ex11p-test-strumpack: ex11p
+	@$(call mfem-test,$<, $(RUN_MPI), STRUMPACK example,--strumpack)
+test-par-YES: ex11p-test-strumpack
+endif

 # Testing: "test" target and mfem-test* variables are defined in config/test.mk

@@ -111,7 +117,7 @@ clean-build:

 clean-exec:
 	@rm -f refined.mesh displaced.mesh mesh.* ex5.mesh
-	@rm -rf Example5* Example9* Example15* Example16*
+	@rm -rf Example5* Example9* Example15* Example16* Example23*
 	@rm -f sphere_refined.* sol.* sol_u.* sol_p.*
 	@rm -f ex9.mesh ex9-mesh.* ex9-init.* ex9-final.*
 	@rm -f deformed.* velocity.* elastic_energy.* mode_*
@@ -120,3 +126,4 @@ clean-exec:
 	@rm -f deformation.* pressure.*
 	@rm -f ex20.dat ex20p_?????.dat gnuplot_ex20.inp gnuplot_ex20p.inp
 	@rm -f ex22*.mesh ex22*.sol ex22p_*.*
+	@rm -f ex23.mesh ex23-mesh.* ex23-init.* ex23-final.*
@@ -11,7 +11,9 @@

 set(SRCS
  bilinearform.cpp
+  bilinearform_ext.cpp
  bilininteg.cpp
+  bilininteg_ext.cpp
  coefficient.cpp
  datacollection.cpp
  eltrans.cpp
@@ -33,7 +35,9 @@ set(SRCS

 set(HDRS
  bilinearform.hpp
+  bilinearform_ext.hpp
  bilininteg.hpp
+  bilininteg_ext.hpp
  coefficient.hpp
  datacollection.hpp
  eltrans.hpp
@@ -12,6 +12,7 @@
 // Implementation of class BilinearForm

 #include "fem.hpp"
+#include "../general/device.hpp"
 #include <cmath>

 namespace mfem
@@ -54,7 +55,7 @@ void BilinearForm::AllocMat()

   int *I = dof_dof.GetI();
   int *J = dof_dof.GetJ();
-   double *data = new double[I[height]];
+   double *data = mfem::New<double>(I[height]);

   mat = new SparseMatrix(I, J, data, height, height, true, true, true);
   *mat = 0.0;
@@ -62,7 +63,7 @@ void BilinearForm::AllocMat()
   dof_dof.LoseData();
 }

-BilinearForm::BilinearForm (FiniteElementSpace * f)
+BilinearForm::BilinearForm(FiniteElementSpace * f)
   : Matrix (f->GetVSize())
 {
   fes = f;
@@ -74,6 +75,10 @@ BilinearForm::BilinearForm (FiniteElementSpace * f)
   hybridization = NULL;
   precompute_sparsity = 0;
   diag_policy = DIAG_KEEP;
+
+   assembly = AssemblyLevel::FULL;
+   batch = 1;
+   ext = NULL;
 }

 BilinearForm::BilinearForm (FiniteElementSpace * f, BilinearForm * bf, int ps)
@@ -89,6 +94,10 @@ BilinearForm::BilinearForm (FiniteElementSpace * f, BilinearForm * bf, int ps)
   precompute_sparsity = ps;
   diag_policy = DIAG_KEEP;

+   assembly = AssemblyLevel::FULL;
+   batch = 1;
+   ext = NULL;
+
   // Copy the pointers to the integrators
   dbfi = bf->dbfi;

@@ -103,9 +112,48 @@ BilinearForm::BilinearForm (FiniteElementSpace * f, BilinearForm * bf, int ps)
   AllocMat();
 }

+void BilinearForm::SetAssemblyLevel(AssemblyLevel assembly_level)
+{
+   if (ext)
+   {
+      MFEM_ABORT("the assembly level has already been set!");
+   }
+   assembly = assembly_level;
+   switch (assembly)
+   {
+      case AssemblyLevel::FULL:
+         if (Device::IsEnabled())
+         {
+            mfem_error("Full assembly not supported yet in device mode!");
+            // ext = new FABilinearFormExtension(this);
+         }
+         // Use the original BilinearForm implementation for now
+         break;
+      case AssemblyLevel::ELEMENT:
+         mfem_error("Element assembly not supported yet... stay tuned!");
+         // ext = new EABilinearFormExtension(this);
+         break;
+      case AssemblyLevel::PARTIAL:
+         ext = new PABilinearFormExtension(this);
+         break;
+      case AssemblyLevel::NONE:
+         mfem_error("Matrix-free action not supported yet... stay tuned!");
+         // ext = new MFBilinearFormExtension(this);
+         break;
+      default:
+         mfem_error("Unknown assembly level");
+   }
+}
+
 void BilinearForm::EnableStaticCondensation()
 {
   delete static_cond;
+   if (assembly != AssemblyLevel::FULL)
+   {
+      static_cond = NULL;
+      MFEM_WARNING("Static condensation not supported for this assembly level");
+      return;
+   }
   static_cond = new StaticCondensation(fes);
   if (static_cond->ReducesTrueVSize())
   {
@@ -125,6 +173,13 @@ void BilinearForm::EnableHybridization(FiniteElementSpace *constr_space,
                                       const Array<int> &ess_tdof_list)
 {
   delete hybridization;
+   if (assembly != AssemblyLevel::FULL)
+   {
+      delete constr_integ;
+      hybridization = NULL;
+      MFEM_WARNING("Hybridization not supported for this assembly level");
+      return;
+   }
   hybridization = new Hybridization(fes, constr_space);
   hybridization->SetConstraintIntegrator(constr_integ);
   hybridization->Init(ess_tdof_list);
@@ -179,9 +234,9 @@ void BilinearForm::Finalize (int skip_zeros)
   if (hybridization) { hybridization->Finalize(); }
 }

-void BilinearForm::AddDomainIntegrator (BilinearFormIntegrator * bfi)
+void BilinearForm::AddDomainIntegrator(BilinearFormIntegrator *bfi)
 {
-   dbfi.Append (bfi);
+   dbfi.Append(bfi);
 }

 void BilinearForm::AddBoundaryIntegrator (BilinearFormIntegrator * bfi)
@@ -287,20 +342,29 @@ void BilinearForm::AssembleBdrElementMatrix(
   }
 }

-void BilinearForm::Assemble (int skip_zeros)
+void BilinearForm::Assemble(int skip_zeros)
 {
+   if (Device::IsEnabled() && (assembly != AssemblyLevel::PARTIAL))
+   {
+      mfem_error("Chosen assembly level not supported yet in device mode!");
+   }
+
+   if (ext)
+   {
+      ext->Assemble();
+      return;
+   }
+
   ElementTransformation *eltrans;
   Mesh *mesh = fes -> GetMesh();
   DenseMatrix elmat, *elmat_p;

-   int i;
-
   if (mat == NULL)
   {
      AllocMat();
   }

-#ifdef MFEM_USE_OPENMP
+#ifdef MFEM_USE_LEGACY_OPENMP
   int free_element_matrices = 0;
   if (!element_matrices)
   {
@@ -311,7 +375,7 @@ void BilinearForm::Assemble (int skip_zeros)

   if (dbfi.Size())
   {
-      for (i = 0; i < fes -> GetNE(); i++)
+      for (int i = 0; i < fes -> GetNE(); i++)
      {
         fes->GetElementVDofs(i, vdofs);
         if (element_matrices)
@@ -368,7 +432,7 @@ void BilinearForm::Assemble (int skip_zeros)
         }
      }

-      for (i = 0; i < fes -> GetNBE(); i++)
+      for (int i = 0; i < fes -> GetNBE(); i++)
      {
         const int bdr_attr = mesh->GetBdrAttribute(i);
         if (bdr_attr_marker[bdr_attr-1] == 0) { continue; }
@@ -406,7 +470,7 @@ void BilinearForm::Assemble (int skip_zeros)
      Array<int> vdofs2;

      int nfaces = mesh->GetNumFaces();
-      for (i = 0; i < nfaces; i++)
+      for (int i = 0; i < nfaces; i++)
      {
         tr = mesh -> GetInteriorFaceTransformations (i);
         if (tr != NULL)
@@ -451,7 +515,7 @@ void BilinearForm::Assemble (int skip_zeros)
         }
      }

-      for (i = 0; i < fes -> GetNBE(); i++)
+      for (int i = 0; i < fes -> GetNBE(); i++)
      {
         const int bdr_attr = mesh->GetBdrAttribute(i);
         if (bdr_attr_marker[bdr_attr-1] == 0) { continue; }
@@ -477,7 +541,7 @@ void BilinearForm::Assemble (int skip_zeros)
      }
   }

-#ifdef MFEM_USE_OPENMP
+#ifdef MFEM_USE_LEGACY_OPENMP
   if (free_element_matrices)
   {
      FreeElementMatrices();
@@ -520,13 +584,22 @@ void BilinearForm::ConformingAssemble()
   width = mat->Width();
 }

-void BilinearForm::FormLinearSystem(const Array<int> &ess_tdof_list,
-                                    Vector &x, Vector &b,
-                                    SparseMatrix &A, Vector &X, Vector &B,
-                                    int copy_interior)
+void BilinearForm::FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x,
+                                    Vector &b, OperatorHandle &A, Vector &X,
+                                    Vector &B, int copy_interior)
 {
   const SparseMatrix *P = fes->GetConformingProlongation();

+   if (ext)
+   {
+      if (P != NULL && assembly != AssemblyLevel::FULL && Device::IsEnabled())
+      {
+         P->BuildTranspose();
+      }
+      ext->FormLinearSystem(ess_tdof_list, x, b, A, X, B, copy_interior);
+      return;
+   }
+
   FormSystemMatrix(ess_tdof_list, A);

   // Transform the system and perform the elimination in B, based on the
@@ -587,8 +660,14 @@ void BilinearForm::FormLinearSystem(const Array<int> &ess_tdof_list,
 }

 void BilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
-                                    SparseMatrix &A)
+                                    OperatorHandle &A)
 {
+   if (ext)
+   {
+      ext->FormSystemMatrix(ess_tdof_list, A);
+      return;
+   }
+
   // Finish the matrix assembly and perform BC elimination, storing the
   // eliminated part of the matrix.
   if (static_cond)
@@ -600,7 +679,7 @@ void BilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
         static_cond->EliminateReducedTrueDofs(diag_policy);
         static_cond->Finalize(); // finalize eliminated part
      }
-      A.MakeRef(static_cond->GetMatrix());
+      A.Reset(&static_cond->GetMatrix(), false);
   }
   else
   {
@@ -614,11 +693,11 @@ void BilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
      }
      if (hybridization)
      {
-         A.MakeRef(hybridization->GetMatrix());
+         A.Reset(&hybridization->GetMatrix(), false);
      }
      else
      {
-         A.MakeRef(*mat);
+         A.Reset(mat, false);
      }
   }
 }
@@ -626,6 +705,12 @@ void BilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
 void BilinearForm::RecoverFEMSolution(const Vector &X,
                                      const Vector &b, Vector &x)
 {
+   if (ext)
+   {
+      ext->RecoverFEMSolution(X, b, x);
+      return;
+   }
+
   const SparseMatrix *P = fes->GetConformingProlongation();
   if (!P) // conforming space
   {
@@ -687,7 +772,7 @@ void BilinearForm::ComputeElementMatrices()
   DenseMatrix tmp;
   IsoparametricTransformation eltrans;

-#ifdef MFEM_USE_OPENMP
+#ifdef MFEM_USE_LEGACY_OPENMP
   #pragma omp parallel for private(tmp,eltrans)
 #endif
   for (int i = 0; i < num_elements; i++)
@@ -714,7 +799,8 @@ void BilinearForm::ComputeElementMatrices()
 }

 void BilinearForm::EliminateEssentialBC(const Array<int> &bdr_attr_is_ess,
-                                        const Vector &sol, Vector &rhs, DiagonalPolicy dpolicy)
+                                        const Vector &sol, Vector &rhs,
+                                        DiagonalPolicy dpolicy)
 {
   Array<int> ess_dofs, conf_ess_dofs;
   fes->GetEssentialVDofs(bdr_attr_is_ess, ess_dofs);
@@ -888,6 +974,8 @@ void BilinearForm::Update(FiniteElementSpace *nfes)
   }

   height = width = fes->GetVSize();
+
+   if (ext) { ext->Update(); }
 }

 void BilinearForm::SetDiagonalPolicy(DiagonalPolicy policy)
@@ -911,6 +999,8 @@ BilinearForm::~BilinearForm()
      for (k=0; k < fbfi.Size(); k++) { delete fbfi[k]; }
      for (k=0; k < bfbfi.Size(); k++) { delete bfbfi[k]; }
   }
+
+   delete ext;
 }


@@ -18,12 +18,32 @@
 #include "gridfunc.hpp"
 #include "linearform.hpp"
 #include "bilininteg.hpp"
+#include "bilinearform_ext.hpp"
 #include "staticcond.hpp"
 #include "hybridization.hpp"

 namespace mfem
 {

+/// Enumeration defining the assembly level for bilinear and nonlinear form
+/// classes derived from Operator.
+enum class AssemblyLevel
+{
+   /// Fully assembled form, i.e. a global sparse matrix in MFEM, Hypre or PETSC
+   /// format.
+   FULL,
+   /// Form assembled at element level, which computes and stores dense element
+   /// matrices.
+   ELEMENT,
+   /// Partially-assembled form, which computes and stores data only at
+   /// quadrature points.
+   PARTIAL,
+   /// "Matrix-free" form that computes all of its action on-the-fly without any
+   /// substantial storage.
+   NONE,
+};
+
+
 /** Class for bilinear form - "Matrix" with associated FE space and
    BLFIntegrators. */
 class BilinearForm : public Matrix
@@ -38,6 +58,14 @@ protected:
   /// FE space on which the form lives. Not owned.
   FiniteElementSpace *fes;

+   /// The form assembly level (full, partial, etc.)
+   AssemblyLevel assembly;
+   /// Element batch size used in the form action (1, 8, num_elems, etc.)
+   int batch;
+   /** Extension for supporting Full Assembly (FA), Element Assembly (EA),
+       Partial Assembly (PA), or Matrix Free assembly (MF). */
+   BilinearFormExtension *ext;
+
   /// Indicates the Mesh::sequence corresponding to the current state of the
   /// BilinearForm.
   long sequence;
@@ -89,6 +117,9 @@ protected:
      static_cond = NULL; hybridization = NULL;
      precompute_sparsity = 0;
      diag_policy = DIAG_KEEP;
+      assembly = AssemblyLevel::FULL;
+      batch = 1;
+      ext = NULL;
   }

 private:
@@ -118,6 +149,10 @@ public:
   /// Get the size of the BilinearForm as a square matrix.
   int Size() const { return height; }

+   /// Set the desired assembly level. The default is AssemblyLevel::FULL.
+   /** This method must be called before assembly. */
+   void SetAssemblyLevel(AssemblyLevel assembly_level);
+
   /** Enable the use of static condensation. For details see the description
       for class StaticCondensation in fem/staticcond.hpp This method should be
       called before assembly. If the number of unknowns after static
@@ -291,11 +326,12 @@ public:
   virtual const Operator *GetRestriction() const
   { return fes->GetConformingRestriction(); }

-   /// Form a linear system, A X = B.
-   /** Form the linear system A X = B, corresponding to the current bilinear
-       form and b(.), by applying any necessary transformations such as:
-       eliminating boundary conditions; applying conforming constraints for
-       non-conforming AMR; static condensation; hybridization.
+   /** @brief Form the linear system A X = B, corresponding to this bilinear
+       form and the linear form @a b(.). */
+   /** This method applies any necessary transformations to the linear system
+       such as: eliminating boundary conditions; applying conforming constraints
+       for non-conforming AMR; parallel assembly; static condensation;
+       hybridization.

       The GridFunction-size vector @a x must contain the essential b.c. The
       BilinearForm and the LinearForm-size vector @a b must be assembled.
@@ -316,12 +352,52 @@ public:

       NOTE: If there are no transformations, @a X simply reuses the data of
             @a x. */
+   virtual void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x,
+                                 Vector &b, OperatorHandle &A, Vector &X,
+                                 Vector &B, int copy_interior = 0);
+
+   /** @brief Form the linear system A X = B, corresponding to this bilinear
+       form and the linear form @a b(.). */
+   /** Version of the method FormLinearSystem() where the system matrix is
+       returned in the variable @a A, of type OpType, holding a *reference* to
+       the system matrix (created with the method OpType::MakeRef()). The
+       reference will be invalidated when SetOperatorType(), Update(), or the
+       destructor is called.
+
+       Currently, this method can be used only with AssemblyLevel::FULL. */
+   template <typename OpType>
   void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x, Vector &b,
-                         SparseMatrix &A, Vector &X, Vector &B,
-                         int copy_interior = 0);
+                         OpType &A, Vector &X, Vector &B,
+                         int copy_interior = 0)
+   {
+      OperatorHandle Ah;
+      FormLinearSystem(ess_tdof_list, x, b, Ah, X, B, copy_interior);
+      OpType *A_ptr = Ah.Is<OpType>();
+      MFEM_VERIFY(A_ptr, "invalid OpType used");
+      A.MakeRef(*A_ptr);
+   }
+
+   /// Form the linear system matrix @a A, see FormLinearSystem() for details.
+   virtual void FormSystemMatrix(const Array<int> &ess_tdof_list,
+                                 OperatorHandle &A);

   /// Form the linear system matrix A, see FormLinearSystem() for details.
-   void FormSystemMatrix(const Array<int> &ess_tdof_list, SparseMatrix &A);
+   /** Version of the method FormSystemMatrix() where the system matrix is
+       returned in the variable @a A, of type OpType, holding a *reference* to
+       the system matrix (created with the method OpType::MakeRef()). The
+       reference will be invalidated when SetOperatorType(), Update(), or the
+       destructor is called.
+
+       Currently, this method can be used only with AssemblyLevel::FULL. */
+   template <typename OpType>
+   void FormSystemMatrix(const Array<int> &ess_tdof_list, OpType &A)
+   {
+      OperatorHandle Ah;
+      FormSystemMatrix(ess_tdof_list, Ah);
+      OpType *A_ptr = Ah.Is<OpType>();
+      MFEM_VERIFY(A_ptr, "invalid OpType used");
+      A.MakeRef(*A_ptr);
+   }

   /// Recover the solution of a linear system formed with FormLinearSystem().
   /** Call this method after solving a linear system constructed using the
@@ -413,6 +489,7 @@ public:
   virtual ~BilinearForm();
 };

+
 /**
   Class for assembling of bilinear forms `a(u,v)` defined on different
   trial and test spaces. The assembled matrix `A` is such that
@@ -0,0 +1,236 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+// Implementations of classes FABilinearFormExtension, EABilinearFormExtension,
+// PABilinearFormExtension and MFBilinearFormExtension.
+
+#include "../general/forall.hpp"
+#include "bilinearform.hpp"
+
+namespace mfem
+{
+
+BilinearFormExtension::BilinearFormExtension(BilinearForm *form)
+   : Operator(form->Size()), a(form)
+{
+   // empty
+}
+
+const Operator *BilinearFormExtension::GetProlongation() const
+{
+   return a->GetProlongation();
+}
+
+const Operator *BilinearFormExtension::GetRestriction() const
+{
+   return a->GetRestriction();
+}
+
+
+// Data and methods for partially-assembled bilinear forms
+PABilinearFormExtension::PABilinearFormExtension(BilinearForm *form) :
+   BilinearFormExtension(form),
+   trialFes(a->FESpace()), testFes(a->FESpace()),
+   localX(trialFes->GetNE() * trialFes->GetFE(0)->GetDof() * trialFes->GetVDim()),
+   localY( testFes->GetNE() * testFes->GetFE(0)->GetDof() * testFes->GetVDim()),
+   elem_restrict(new ElemRestriction(*a->FESpace())) { }
+
+PABilinearFormExtension::~PABilinearFormExtension()
+{
+   delete elem_restrict;
+}
+
+void PABilinearFormExtension::Assemble()
+{
+   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
+   const int integratorCount = integrators.Size();
+   for (int i = 0; i < integratorCount; ++i)
+   {
+      integrators[i]->Assemble(*a->FESpace());
+   }
+}
+
+void PABilinearFormExtension::Update()
+{
+   FiniteElementSpace *fes = a->FESpace();
+   height = width = fes->GetVSize();
+   trialFes = fes;
+   testFes = fes;
+   localX.SetSize(trialFes->GetNE() * trialFes->GetFE(0)->GetDof() *
+                  trialFes->GetVDim());
+   localY.SetSize(testFes->GetNE() * testFes->GetFE(0)->GetDof() *
+                  testFes->GetVDim());
+   delete elem_restrict;
+   elem_restrict = new ElemRestriction(*fes);
+}
+
+void PABilinearFormExtension::FormSystemMatrix(const Array<int> &ess_tdof_list,
+                                               OperatorHandle &A)
+{
+   const Operator* trialP = trialFes->GetProlongationMatrix();
+   const Operator* testP  = testFes->GetProlongationMatrix();
+   Operator *rap = this;
+   if (trialP) { rap = new RAPOperator(*testP, *this, *trialP); }
+   const bool own_A = (rap!=this);
+   A.Reset(new ConstrainedOperator(rap, ess_tdof_list, own_A));
+}
+
+void PABilinearFormExtension::FormLinearSystem(const Array<int> &ess_tdof_list,
+                                               Vector &x, Vector &b,
+                                               OperatorHandle &A,
+                                               Vector &X, Vector &B,
+                                               int copy_interior)
+{
+   Operator *oper;
+   Operator::FormLinearSystem(ess_tdof_list, x, b, oper, X, B, copy_interior);
+   A.Reset(oper); // A will own oper
+}
+
+void PABilinearFormExtension::Mult(const Vector &x, Vector &y) const
+{
+   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
+   elem_restrict->Mult(x, localX);
+   localY = 0.0;
+   const int iSz = integrators.Size();
+   for (int i = 0; i < iSz; ++i)
+   {
+      integrators[i]->MultAssembled(localX, localY);
+   }
+   elem_restrict->MultTranspose(localY, y);
+}
+
+void PABilinearFormExtension::MultTranspose(const Vector &x, Vector &y) const
+{
+   Array<BilinearFormIntegrator*> &integrators = *a->GetDBFI();
+   elem_restrict->Mult(x, localX);
+   localY = 0.0;
+   const int iSz = integrators.Size();
+   for (int i = 0; i < iSz; ++i)
+   {
+      integrators[i]->MultAssembledTranspose(localX, localY);
+   }
+   elem_restrict->MultTranspose(localY, y);
+}
+
+
+ElemRestriction::ElemRestriction(const FiniteElementSpace &f)
+   : fes(f),
+     ne(fes.GetNE()),
+     vdim(fes.GetVDim()),
+     byvdim(fes.GetOrdering() == Ordering::byVDIM),
+     ndofs(fes.GetNDofs()),
+     dof(fes.GetFE(0)->GetDof()),
+     nedofs(ne*dof),
+     offsets(ndofs+1),
+     indices(ne*dof)
+{
+   for (int e = 0; e < ne; ++e)
+   {
+      const FiniteElement *fe = fes.GetFE(e);
+      const TensorBasisElement* el =
+         dynamic_cast<const TensorBasisElement*>(fe);
+      if (el) { continue; }
+      mfem_error("Finite element not supported with partial assembly");
+   }
+   const FiniteElement *fe = fes.GetFE(0);
+   const TensorBasisElement* el = dynamic_cast<const TensorBasisElement*>(fe);
+   const Array<int> &dof_map = el->GetDofMap();
+   const bool dof_map_is_identity = (dof_map.Size()==0);
+   const Table& e2dTable = fes.GetElementToDofTable();
+   const int* elementMap = e2dTable.GetJ();
+   // We'll be keeping a count of how many local nodes point to its global dof
+   for (int i = 0; i <= ndofs; ++i)
+   {
+      offsets[i] = 0;
+   }
+   for (int e = 0; e < ne; ++e)
+   {
+      for (int d = 0; d < dof; ++d)
+      {
+         const int gid = elementMap[dof*e + d];
+         ++offsets[gid + 1];
+      }
+   }
+   // Aggregate to find offsets for each global dof
+   for (int i = 1; i <= ndofs; ++i)
+   {
+      offsets[i] += offsets[i - 1];
+   }
+   // For each global dof, fill in all local nodes that point   to it
+   for (int e = 0; e < ne; ++e)
+   {
+      for (int d = 0; d < dof; ++d)
+      {
+         const int did = dof_map_is_identity?d:dof_map[d];
+         const int gid = elementMap[dof*e + did];
+         const int lid = dof*e + d;
+         indices[offsets[gid]++] = lid;
+      }
+   }
+   // We shifted the offsets vector by 1 by using it as a counter
+   // Now we shift it back.
+   for (int i = ndofs; i > 0; --i)
+   {
+      offsets[i] = offsets[i - 1];
+   }
+   offsets[0] = 0;
+}
+
+void ElemRestriction::Mult(const Vector& x, Vector& y) const
+{
+   const int vd = vdim;
+   const bool t = byvdim;
+   const DeviceArray d_offsets(offsets, ndofs+1);
+   const DeviceArray d_indices(indices, nedofs);
+   const DeviceMatrix d_x(x, t?vd:ndofs, t?ndofs:vd);
+   DeviceMatrix d_y(y, t?vd:nedofs, t?nedofs:vd);
+   MFEM_FORALL(i, ndofs,
+   {
+      const int offset = d_offsets[i];
+      const int nextOffset = d_offsets[i+1];
+      for (int c = 0; c < vd; ++c)
+      {
+         const double dofValue = d_x(t?c:i,t?i:c);
+         for (int j = offset; j < nextOffset; ++j)
+         {
+            const int idx_j = d_indices[j];
+            d_y(t?c:idx_j,t?idx_j:c) = dofValue;
+         }
+      }
+   });
+}
+
+void ElemRestriction::MultTranspose(const Vector& x, Vector& y) const
+{
+   const int vd = vdim;
+   const bool t = byvdim;
+   const DeviceArray d_offsets(offsets, ndofs+1);
+   const DeviceArray d_indices(indices, nedofs);
+   const DeviceMatrix d_x(x, t?vd:nedofs, t?nedofs:vd);
+   DeviceMatrix d_y(y, t?vd:ndofs, t?ndofs:vd);
+   MFEM_FORALL(i, ndofs,
+   {
+      const int offset = d_offsets[i];
+      const int nextOffset = d_offsets[i + 1];
+      for (int c = 0; c < vd; ++c)
+      {
+         double dofValue = 0;
+         for (int j = offset; j < nextOffset; ++j)
+         {
+            const int idx_j = d_indices[j];
+            dofValue +=  d_x(t?c:idx_j,t?idx_j:c);
+         }
+         d_y(t?c:i,t?i:c) = dofValue;
+      }
+   });
+}
+
+} // namespace mfem
@@ -0,0 +1,151 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_BILINEARFORM_EXT
+#define MFEM_BILINEARFORM_EXT
+
+#include "../config/config.hpp"
+#include "fespace.hpp"
+
+namespace mfem
+{
+
+class BilinearForm;
+
+/// Element restriction operator
+class ElemRestriction: public Operator
+{
+public:
+   const FiniteElementSpace &fes;
+   const int ne;
+   const int vdim;
+   const bool byvdim;
+   const int ndofs;
+   const int dof;
+   const int nedofs;
+   Array<int> offsets;
+   Array<int> indices;
+public:
+   ElemRestriction(const FiniteElementSpace&);
+   void Mult(const Vector &x, Vector &y) const;
+   void MultTranspose(const Vector &x, Vector &y) const;
+};
+
+
+class BilinearFormExtension : public Operator
+{
+protected:
+   BilinearForm *a; ///< Not owned
+
+public:
+   BilinearFormExtension(BilinearForm *form);
+
+   /// Get the finite element space prolongation matrix
+   virtual const Operator *GetProlongation() const;
+
+   /// Get the finite element space restriction matrix
+   virtual const Operator *GetRestriction() const;
+
+   virtual void Assemble() = 0;
+   virtual void FormSystemMatrix(const Array<int> &ess_tdof_list,
+                                 OperatorHandle &A) = 0;
+   virtual void FormLinearSystem(const Array<int> &ess_tdof_list,
+                                 Vector &x, Vector &b,
+                                 OperatorHandle &A, Vector &X, Vector &B,
+                                 int copy_interior = 0) = 0;
+   virtual void Update() = 0;
+};
+
+/// Data and methods for fully-assembled bilinear forms
+class FABilinearFormExtension : public BilinearFormExtension
+{
+public:
+   FABilinearFormExtension(BilinearForm *form)
+      : BilinearFormExtension(form) { }
+
+   /// TODO
+   void Assemble() {}
+   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A) {}
+   void FormLinearSystem(const Array<int> &ess_tdof_list,
+                         Vector &x, Vector &b,
+                         OperatorHandle &A, Vector &X, Vector &B,
+                         int copy_interior = 0) {}
+   void Mult(const Vector &x, Vector &y) const {}
+   void MultTranspose(const Vector &x, Vector &y) const {}
+   ~FABilinearFormExtension() {}
+};
+
+/// Data and methods for element-assembled bilinear forms
+class EABilinearFormExtension : public BilinearFormExtension
+{
+public:
+   EABilinearFormExtension(BilinearForm *form)
+      : BilinearFormExtension(form) { }
+
+   /// TODO
+   void Assemble() {}
+   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A) {}
+   void FormLinearSystem(const Array<int> &ess_tdof_list,
+                         Vector &x, Vector &b,
+                         OperatorHandle &A, Vector &X, Vector &B,
+                         int copy_interior = 0) {}
+   void Mult(const Vector &x, Vector &y) const {}
+   void MultTranspose(const Vector &x, Vector &y) const {}
+   ~EABilinearFormExtension() {}
+};
+
+/// Data and methods for partially-assembled bilinear forms
+class PABilinearFormExtension : public BilinearFormExtension
+{
+protected:
+   const FiniteElementSpace *trialFes, *testFes;
+   mutable Vector localX, localY;
+   ElemRestriction *elem_restrict;
+
+public:
+   PABilinearFormExtension(BilinearForm*);
+
+   void Assemble();
+   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A);
+   void FormLinearSystem(const Array<int> &ess_tdof_list,
+                         Vector &x, Vector &b,
+                         OperatorHandle &A, Vector &X, Vector &B,
+                         int copy_interior = 0);
+
+   void Mult(const Vector &x, Vector &y) const;
+   void MultTranspose(const Vector &x, Vector &y) const;
+   void Update();
+
+   ~PABilinearFormExtension();
+};
+
+/// Data and methods for matrix-free bilinear forms
+class MFBilinearFormExtension : public BilinearFormExtension
+{
+public:
+   MFBilinearFormExtension(BilinearForm *form)
+      : BilinearFormExtension(form) { }
+
+   /// TODO
+   void Assemble() {}
+   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A) {}
+   void FormLinearSystem(const Array<int> &ess_tdof_list,
+                         Vector &x, Vector &b,
+                         OperatorHandle &A, Vector &X, Vector &B,
+                         int copy_interior = 0) {}
+   void Mult(const Vector &x, Vector &y) const {}
+   void MultTranspose(const Vector &x, Vector &y) const {}
+   ~MFBilinearFormExtension() {}
+};
+
+}
+
+#endif
@@ -19,6 +19,23 @@ using namespace std;

 namespace mfem
 {
+void BilinearFormIntegrator::Assemble(const FiniteElementSpace&)
+{
+   mfem_error ("BilinearFormIntegrator::Assemble (...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::MultAssembled(Vector&, Vector&)
+{
+   mfem_error ("BilinearFormIntegrator::MultAssembled (...)\n"
+               "   is not implemented for this class.");
+}
+
+void BilinearFormIntegrator::MultAssembledTranspose(Vector&, Vector&)
+{
+   mfem_error ("BilinearFormIntegrator::MultAssembledTranspose (...)\n"
+               "   is not implemented for this class.");
+}

 void BilinearFormIntegrator::AssembleElementMatrix (
   const FiniteElement &el, ElementTransformation &Trans,
@@ -14,6 +14,8 @@

 #include "../config/config.hpp"
 #include "nonlininteg.hpp"
+#include "fespace.hpp"
+#include "bilininteg_ext.hpp"

 namespace mfem
 {
@@ -21,11 +23,20 @@ namespace mfem
 /// Abstract base class BilinearFormIntegrator
 class BilinearFormIntegrator : public NonlinearFormIntegrator
 {
-protected:
+public:
   BilinearFormIntegrator(const IntegrationRule *ir = NULL) :
      NonlinearFormIntegrator(ir) { }

 public:
+   /// Method defining partial assembly.
+   virtual void Assemble(const FiniteElementSpace&);
+
+   /// Method for partially assembled action.
+   virtual void MultAssembled(Vector&, Vector&);
+
+   /// Method for partially assembled transposed action.
+   virtual void MultAssembledTranspose(Vector&, Vector&);
+
   /// Given a particular Finite Element computes the element matrix elmat.
   virtual void AssembleElementMatrix(const FiniteElement &el,
                                      ElementTransformation &Trans,
@@ -1634,16 +1645,19 @@ private:
 #endif
   Coefficient *Q;
   MatrixCoefficient *MQ;
-
+   // PA extension
+   DofToQuad *maps;
+   GeometryExtension *geom;
+   int dim, ne, dofs1D, quad1D;
 public:
   /// Construct a diffusion integrator with coefficient Q = 1
-   DiffusionIntegrator() { Q = NULL; MQ = NULL; }
+   DiffusionIntegrator() { Q = NULL; MQ = NULL; maps = NULL; geom = NULL; }

   /// Construct a diffusion integrator with a scalar coefficient q
-   DiffusionIntegrator (Coefficient &q) : Q(&q) { MQ = NULL; }
+   DiffusionIntegrator (Coefficient &q) : Q(&q) { MQ = NULL; maps = NULL; geom = NULL; }

   /// Construct a diffusion integrator with a matrix coefficient q
-   DiffusionIntegrator (MatrixCoefficient &q) : MQ(&q) { Q = NULL; }
+   DiffusionIntegrator (MatrixCoefficient &q) : MQ(&q) { Q = NULL; maps = NULL; geom = NULL; }

   /** Given a particular Finite Element
       computes the element stiffness matrix elmat. */
@@ -1670,6 +1684,12 @@ public:
   virtual double ComputeFluxEnergy(const FiniteElement &fluxelem,
                                    ElementTransformation &Trans,
                                    Vector &flux, Vector *d_energy = NULL);
+
+   /// PA extension
+   virtual void Assemble(const FiniteElementSpace&);
+   virtual void MultAssembled(Vector&, Vector&);
+
+   virtual ~DiffusionIntegrator();
 };

 /** Class for local mass matrix assembling a(u,v) := (Q u, v) */
@@ -1680,13 +1700,17 @@ protected:
   Vector shape, te_shape;
 #endif
   Coefficient *Q;
-
+   // PA extension
+   Vector vec;
+   DofToQuad *maps;
+   GeometryExtension *geom;
+   int dim, ne, nq, dofs1D, quad1D;
 public:
   MassIntegrator(const IntegrationRule *ir = NULL)
-      : BilinearFormIntegrator(ir) { Q = NULL; }
+      : BilinearFormIntegrator(ir) { Q = NULL; maps = NULL; geom = NULL; }
   /// Construct a mass integrator with coefficient q
   MassIntegrator(Coefficient &q, const IntegrationRule *ir = NULL)
-      : BilinearFormIntegrator(ir), Q(&q) { }
+      : BilinearFormIntegrator(ir), Q(&q) { maps = NULL; geom = NULL; }

   /** Given a particular Finite Element
       computes the element mass matrix elmat. */
@@ -1697,6 +1721,11 @@ public:
                                       const FiniteElement &test_fe,
                                       ElementTransformation &Trans,
                                       DenseMatrix &elmat);
+   /// PA extension
+   virtual void Assemble(const FiniteElementSpace&);
+   virtual void MultAssembled(Vector&, Vector&);
+
+   virtual ~MassIntegrator();
 };

 class BoundaryMassIntegrator : public MassIntegrator
@@ -0,0 +1,80 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_BILININTEG_EXT
+#define MFEM_BILININTEG_EXT
+
+#include "fespace.hpp"
+
+namespace mfem
+{
+
+/// GeometryExtension
+class GeometryExtension
+{
+public:
+   Array<int> eMap;
+   Array<double> nodes;
+   Array<double> X, J, invJ, detJ;
+   static GeometryExtension* Get(const FiniteElementSpace&,
+                                 const IntegrationRule&);
+   static GeometryExtension* Get(const FiniteElementSpace&,
+                                 const IntegrationRule&,
+                                 const Vector&);
+   static void ReorderByVDim(const GridFunction*);
+   static void ReorderByNodes(const GridFunction*);
+};
+
+/// DofToQuad
+class DofToQuad
+{
+private:
+   std::string hash;
+public:
+   ~DofToQuad();
+   void operator=(DofToQuad&);
+   void operator=(DofToQuad const&);
+public:
+   Array<double> W, B, G, Bt, Gt;
+public:
+   static DofToQuad* Get(const FiniteElementSpace&,
+                         const IntegrationRule&,
+                         const bool = false);
+   static DofToQuad* Get(const FiniteElementSpace&,
+                         const FiniteElementSpace&,
+                         const IntegrationRule&,
+                         const bool = false);
+   static DofToQuad* Get(const FiniteElement&,
+                         const FiniteElement&,
+                         const IntegrationRule&,
+                         const bool = false);
+   static DofToQuad* GetTensorMaps(const FiniteElement&,
+                                   const FiniteElement&,
+                                   const IntegrationRule&,
+                                   const bool = false);
+   static DofToQuad* GetD2QTensorMaps(const FiniteElement&,
+                                      const IntegrationRule&,
+                                      const bool = false);
+   static DofToQuad* GetSimplexMaps(const FiniteElement&,
+                                    const IntegrationRule&,
+                                    const bool = false);
+   static DofToQuad* GetSimplexMaps(const FiniteElement&,
+                                    const FiniteElement&,
+                                    const IntegrationRule&,
+                                    const bool = false);
+   static DofToQuad* GetD2QSimplexMaps(const FiniteElement&,
+                                       const IntegrationRule&,
+                                       const bool = false);
+};
+
+}
+
+#endif
@@ -28,6 +28,11 @@ double PWConstCoefficient::Eval(ElementTransformation & T,
   return (constants(att-1));
 }

+DeviceFunctionCoefficientPtr FunctionCoefficient::GetDeviceFunction()
+{
+   return DeviceFunction;
+}
+
 double FunctionCoefficient::Eval(ElementTransformation & T,
                                 const IntegrationPoint & ip)
 {
@@ -40,6 +45,10 @@ double FunctionCoefficient::Eval(ElementTransformation & T,
   {
      return ((*Function)(transip));
   }
+   else if (DeviceFunction)
+   {
+      return ((*DeviceFunction)(Vector3(x)));
+   }
   else
   {
      return (*TDFunction)(transip, GetTime());
@@ -96,7 +96,7 @@ public:
   { constants.SetSize(c.Size()); constants=c; }

   /// Update constants
-   void UpdateConstants(Vector &c) {constants.SetSize(c.Size()); constants=c;}
+   void UpdateConstants(Vector &c) { constants.SetSize(c.Size()); constants=c; }

   /// Member function to access or modify the value of the i-th constant
   double &operator()(int i) { return constants(i-1); }
@@ -112,12 +112,15 @@ public:
                       const IntegrationPoint &ip);
 };

+typedef double (*DeviceFunctionCoefficientPtr)(const Vector3&);
+
 /// class for C-function coefficient
 class FunctionCoefficient : public Coefficient
 {
 protected:
   double (*Function)(const Vector &);
   double (*TDFunction)(const Vector &, double);
+   double (*DeviceFunction)(const Vector3&);

 public:
   /// Define a time-independent coefficient from a C-function
@@ -125,6 +128,7 @@ public:
   {
      Function = f;
      TDFunction = NULL;
+      DeviceFunction = NULL;
   }

   /// Define a time-dependent coefficient from a C-function
@@ -132,6 +136,16 @@ public:
   {
      Function = NULL;
      TDFunction = tdf;
+      DeviceFunction = NULL;
+   }
+
+   /// Define a time-independent coefficient from a C-function using
+   /// Vector3 instead of a Vector.
+   FunctionCoefficient(double (*df)(const Vector3 &))
+   {
+      Function = NULL;
+      TDFunction = NULL;
+      DeviceFunction = df;
   }

   /// (DEPRECATED) Define a time-independent coefficient from a C-function
@@ -141,6 +155,7 @@ public:
   {
      Function = reinterpret_cast<double(*)(const Vector&)>(f);
      TDFunction = NULL;
+      DeviceFunction = NULL;
   }

   /// (DEPRECATED) Define a time-dependent coefficient from a C-function
@@ -150,11 +165,17 @@ public:
   {
      Function = NULL;
      TDFunction = reinterpret_cast<double(*)(const Vector&,double)>(tdf);
+      DeviceFunction = NULL;
   }

   /// Evaluate coefficient
   virtual double Eval(ElementTransformation &T,
                       const IntegrationPoint &ip);
+
+   /// Return the coefficient's C-function that uses Vector3.
+   /// Warning: for now, the returned function can only be used on the
+   /// host inside a MFEM_FORALL.
+   DeviceFunctionCoefficientPtr GetDeviceFunction();
 };

 class GridFunction;
@@ -448,6 +469,7 @@ public:
   void SetGridFunction(GridFunction *gf);
   GridFunction * GetGridFunction() const { return GridFunc; }

+   using VectorCoefficient::Eval;
   virtual void Eval(Vector &V, ElementTransformation &T,
                     const IntegrationPoint &ip);

@@ -790,6 +812,7 @@ public:
   /// Evaluate the coefficient
   virtual void Eval(Vector &V, ElementTransformation &T,
                     const IntegrationPoint &ip);
+   using VectorCoefficient::Eval;
 };

 /// Vector coefficient defined as a product of a scalar and a vector
@@ -804,6 +827,7 @@ public:

   virtual void Eval(Vector &V, ElementTransformation &T,
                     const IntegrationPoint &ip);
+   using VectorCoefficient::Eval;
 };

 /// Vector coefficient defined as a cross product of two vectors
@@ -821,6 +845,7 @@ public:

   virtual void Eval(Vector &V, ElementTransformation &T,
                     const IntegrationPoint &ip);
+   using VectorCoefficient::Eval;
 };

 /// Vector coefficient defined as a matrix vector product
@@ -838,6 +863,7 @@ public:

   virtual void Eval(Vector &V, ElementTransformation &T,
                     const IntegrationPoint &ip);
+   using VectorCoefficient::Eval;
 };

 /// Matrix coefficient defined as the identity of dimension d
@@ -652,9 +652,9 @@ void FiniteElementSpace::BuildConformingInterpolation() const
   // create the conforming restriction matrix cR
   int *cR_J;
   {
-      int *cR_I = new int[n_true_dofs+1];
-      double *cR_A = new double[n_true_dofs];
-      cR_J = new int[n_true_dofs];
+      int *cR_I = mfem::New<int>(n_true_dofs+1);
+      double *cR_A = mfem::New<double>(n_true_dofs);
+      cR_J = mfem::New<int>(n_true_dofs);
      for (int i = 0; i < n_true_dofs; i++)
      {
         cR_I[i] = i;
@@ -975,6 +975,139 @@ void FiniteElementSpace::RefinementOperator
   }
 }

+FiniteElementSpace::DerefinementOperator::DerefinementOperator(
+   const FiniteElementSpace *f_fes, const FiniteElementSpace *c_fes,
+   BilinearFormIntegrator *mass_integ)
+   : Operator(c_fes->GetVSize(), f_fes->GetVSize()),
+     fine_fes(f_fes)
+{
+   MFEM_VERIFY(c_fes->GetOrdering() == f_fes->GetOrdering() &&
+               c_fes->GetVDim() == f_fes->GetVDim(),
+               "incompatible coarse and fine FE spaces");
+
+   IsoparametricTransformation emb_tr;
+   Mesh *f_mesh = f_fes->GetMesh();
+   const CoarseFineTransformations &rtrans = f_mesh->GetRefinementTransforms();
+
+   Mesh::GeometryList elem_geoms(*f_mesh);
+   DenseTensor localP[Geometry::NumGeom], localM[Geometry::NumGeom];
+   for (int gi = 0; gi < elem_geoms.Size(); gi++)
+   {
+      const Geometry::Type geom = elem_geoms[gi];
+      DenseTensor &lP = localP[geom], &lM = localM[geom];
+      const FiniteElement *fine_fe =
+         f_fes->fec->FiniteElementForGeometry(geom);
+      const FiniteElement *coarse_fe =
+         c_fes->fec->FiniteElementForGeometry(geom);
+      const DenseTensor &pmats = rtrans.GetPointMatrices(geom);
+
+      lP.SetSize(fine_fe->GetDof(), coarse_fe->GetDof(), pmats.SizeK());
+      lM.SetSize(fine_fe->GetDof(),   fine_fe->GetDof(), pmats.SizeK());
+      emb_tr.SetIdentityTransformation(geom);
+      for (int i = 0; i < pmats.SizeK(); i++)
+      {
+         emb_tr.GetPointMat() = pmats(i);
+         emb_tr.FinalizeTransformation();
+         // Get the local interpolation matrix for this refinement type
+         fine_fe->GetTransferMatrix(*coarse_fe, emb_tr, lP(i));
+         // Get the local mass matrix for this refinement type
+         mass_integ->AssembleElementMatrix(*fine_fe, emb_tr, lM(i));
+      }
+   }
+
+   Table ref_type_to_matrix;
+   rtrans.GetCoarseToFineMap(*f_mesh, coarse_to_fine, coarse_to_ref_type,
+                             ref_type_to_matrix, ref_type_to_geom);
+   MFEM_ASSERT(coarse_to_fine.Size() == c_fes->GetNE(), "");
+
+   const int total_ref_types = ref_type_to_geom.Size();
+   int num_ref_types[Geometry::NumGeom], num_fine_elems[Geometry::NumGeom];
+   Array<int> ref_type_to_coarse_elem_offset(total_ref_types);
+   ref_type_to_fine_elem_offset.SetSize(total_ref_types);
+   std::fill(num_ref_types, num_ref_types+Geometry::NumGeom, 0);
+   std::fill(num_fine_elems, num_fine_elems+Geometry::NumGeom, 0);
+   for (int i = 0; i < total_ref_types; i++)
+   {
+      Geometry::Type g = ref_type_to_geom[i];
+      ref_type_to_coarse_elem_offset[i] = num_ref_types[g];
+      ref_type_to_fine_elem_offset[i] = num_fine_elems[g];
+      num_ref_types[g]++;
+      num_fine_elems[g] += ref_type_to_matrix.RowSize(i);
+   }
+   DenseTensor localPtMP[Geometry::NumGeom];
+   for (int g = 0; g < Geometry::NumGeom; g++)
+   {
+      if (num_ref_types[g] == 0) { continue; }
+      const int fine_dofs = localP[g].SizeI();
+      const int coarse_dofs = localP[g].SizeJ();
+      localPtMP[g].SetSize(coarse_dofs, coarse_dofs, num_ref_types[g]);
+      localR[g].SetSize(coarse_dofs, fine_dofs, num_fine_elems[g]);
+   }
+   for (int i = 0; i < total_ref_types; i++)
+   {
+      Geometry::Type g = ref_type_to_geom[i];
+      DenseMatrix &lPtMP = localPtMP[g](ref_type_to_coarse_elem_offset[i]);
+      int lR_offset = ref_type_to_fine_elem_offset[i]; // offset in localR[g]
+      const int *mi = ref_type_to_matrix.GetRow(i);
+      const int nm = ref_type_to_matrix.RowSize(i);
+      lPtMP = 0.0;
+      for (int s = 0; s < nm; s++)
+      {
+         DenseMatrix &lP = localP[g](mi[s]);
+         DenseMatrix &lM = localM[g](mi[s]);
+         DenseMatrix &lR = localR[g](lR_offset+s);
+         MultAtB(lP, lM, lR); // lR = lP^T lM
+         AddMult(lR, lP, lPtMP); // lPtMP += lP^T lM lP
+      }
+      DenseMatrixInverse lPtMP_inv(lPtMP);
+      for (int s = 0; s < nm; s++)
+      {
+         DenseMatrix &lR = localR[g](lR_offset+s);
+         lPtMP_inv.Mult(lR); // lR <- (P^T M P)^{-1} P^T M
+      }
+   }
+
+   // Make a copy of the coarse element-to-dof Table.
+   coarse_elem_dof = new Table(c_fes->GetElementToDofTable());
+}
+
+FiniteElementSpace::DerefinementOperator::~DerefinementOperator()
+{
+   delete coarse_elem_dof;
+}
+
+void FiniteElementSpace::DerefinementOperator
+::Mult(const Vector &x, Vector &y) const
+{
+   Array<int> c_vdofs, f_vdofs;
+   Vector loc_x, loc_y;
+   DenseMatrix loc_x_mat, loc_y_mat;
+   const int vdim = fine_fes->GetVDim();
+   const int coarse_ndofs = height/vdim;
+   for (int coarse_el = 0; coarse_el < coarse_to_fine.Size(); coarse_el++)
+   {
+      coarse_elem_dof->GetRow(coarse_el, c_vdofs);
+      fine_fes->DofsToVDofs(c_vdofs, coarse_ndofs);
+      loc_y.SetSize(c_vdofs.Size());
+      loc_y = 0.0;
+      loc_y_mat.UseExternalData(loc_y.GetData(), c_vdofs.Size()/vdim, vdim);
+      const int ref_type = coarse_to_ref_type[coarse_el];
+      const Geometry::Type geom = ref_type_to_geom[ref_type];
+      const int *fine_elems = coarse_to_fine.GetRow(coarse_el);
+      const int num_fine_elems = coarse_to_fine.RowSize(coarse_el);
+      const int lR_offset = ref_type_to_fine_elem_offset[ref_type];
+      for (int s = 0; s < num_fine_elems; s++)
+      {
+         const DenseMatrix &lR = localR[geom](lR_offset+s);
+         fine_fes->GetElementVDofs(fine_elems[s], f_vdofs);
+         x.GetSubVector(f_vdofs, loc_x);
+         loc_x_mat.UseExternalData(loc_x.GetData(), f_vdofs.Size()/vdim, vdim);
+         AddMult(lR, loc_x_mat, loc_y_mat);
+      }
+      y.SetSubVector(c_vdofs, loc_y);
+   }
+}
+
 void FiniteElementSpace::GetLocalDerefinementMatrices(Geometry::Type geom,
                                                      DenseTensor &localR) const
 {
@@ -2042,4 +2175,401 @@ void QuadratureSpace::Save(std::ostream &out) const
 }


+GridTransfer::GridTransfer(FiniteElementSpace &dom_fes_,
+                           FiniteElementSpace &ran_fes_)
+   : dom_fes(dom_fes_), ran_fes(ran_fes_),
+     oper_type(Operator::ANY_TYPE),
+     fw_t_oper(), bw_t_oper()
+{
+#ifdef MFEM_USE_MPI
+   const bool par_dom = dynamic_cast<ParFiniteElementSpace*>(&dom_fes);
+   const bool par_ran = dynamic_cast<ParFiniteElementSpace*>(&ran_fes);
+   MFEM_VERIFY(par_dom == par_ran, "the domain and range FE spaces must both"
+               " be either serial or parallel");
+   parallel = par_dom;
+#endif
+}
+
+const Operator &GridTransfer::MakeTrueOperator(
+   FiniteElementSpace &fes_in, FiniteElementSpace &fes_out,
+   const Operator &oper, OperatorHandle &t_oper)
+{
+   if (t_oper.Ptr())
+   {
+      return *t_oper.Ptr();
+   }
+
+   if (!Parallel())
+   {
+      const SparseMatrix *in_cP = fes_in.GetConformingProlongation();
+      const SparseMatrix *out_cR = fes_out.GetConformingRestriction();
+      if (oper_type == Operator::MFEM_SPARSEMAT)
+      {
+         const SparseMatrix *mat = dynamic_cast<const SparseMatrix *>(&oper);
+         MFEM_VERIFY(mat != NULL, "Operator is not a SparseMatrix");
+         if (!out_cR)
+         {
+            t_oper.Reset(const_cast<SparseMatrix*>(mat), false);
+         }
+         else
+         {
+            t_oper.Reset(mfem::Mult(*out_cR, *mat));
+         }
+         if (in_cP)
+         {
+            t_oper.Reset(mfem::Mult(*t_oper.As<SparseMatrix>(), *in_cP));
+         }
+      }
+      else if (oper_type == Operator::ANY_TYPE)
+      {
+         const int RP_case = bool(out_cR) + 2*bool(in_cP);
+         switch (RP_case)
+         {
+            case 0:
+               t_oper.Reset(const_cast<Operator*>(&oper), false);
+               break;
+            case 1:
+               t_oper.Reset(
+                  new ProductOperator(out_cR, &oper, false, false));
+               break;
+            case 2:
+               t_oper.Reset(
+                  new ProductOperator(&oper, in_cP, false, false));
+               break;
+            case 3:
+               t_oper.Reset(
+                  new TripleProductOperator(
+                     out_cR, &oper, in_cP, false, false, false));
+               break;
+         }
+      }
+      else
+      {
+         MFEM_ABORT("Operator::Type is not supported: " << oper_type);
+      }
+   }
+   else // Parallel() == true
+   {
+#ifdef MFEM_USE_MPI
+      const SparseMatrix *out_R = fes_out.GetRestrictionMatrix();
+      if (oper_type == Operator::Hypre_ParCSR)
+      {
+         const ParFiniteElementSpace *pfes_in =
+            dynamic_cast<const ParFiniteElementSpace *>(&fes_in);
+         const ParFiniteElementSpace *pfes_out =
+            dynamic_cast<const ParFiniteElementSpace *>(&fes_out);
+         const SparseMatrix *sp_mat = dynamic_cast<const SparseMatrix *>(&oper);
+         const HypreParMatrix *hy_mat;
+         if (sp_mat)
+         {
+            SparseMatrix *RA = mfem::Mult(*out_R, *sp_mat);
+            t_oper.Reset(pfes_in->Dof_TrueDof_Matrix()->
+                         LeftDiagMult(*RA, pfes_out->GetTrueDofOffsets()));
+            delete RA;
+         }
+         else if ((hy_mat = dynamic_cast<const HypreParMatrix *>(&oper)))
+         {
+            HypreParMatrix *RA =
+               hy_mat->LeftDiagMult(*out_R, pfes_out->GetTrueDofOffsets());
+            t_oper.Reset(mfem::ParMult(RA, pfes_in->Dof_TrueDof_Matrix()));
+            delete RA;
+         }
+         else
+         {
+            MFEM_ABORT("unknown Operator type");
+         }
+      }
+      else if (oper_type == Operator::ANY_TYPE)
+      {
+         t_oper.Reset(new TripleProductOperator(
+                         out_R, &oper, fes_in.GetProlongationMatrix(),
+                         false, false, false));
+      }
+      else
+      {
+         MFEM_ABORT("Operator::Type is not supported: " << oper_type);
+      }
+#endif
+   }
+
+   return *t_oper.Ptr();
+}
+
+
+InterpolationGridTransfer::~InterpolationGridTransfer()
+{
+   if (own_mass_integ) { delete mass_integ; }
+}
+
+void InterpolationGridTransfer::SetMassIntegrator(
+   BilinearFormIntegrator *mass_integ_, bool own_mass_integ_)
+{
+   if (own_mass_integ) { delete mass_integ; }
+
+   mass_integ = mass_integ_;
+   own_mass_integ = own_mass_integ_;
+}
+
+const Operator &InterpolationGridTransfer::ForwardOperator()
+{
+   if (F.Ptr())
+   {
+      return *F.Ptr();
+   }
+
+   // Costruct F
+   if (oper_type == Operator::ANY_TYPE)
+   {
+      F.Reset(new FiniteElementSpace::RefinementOperator(&ran_fes, &dom_fes));
+   }
+   else if (oper_type == Operator::MFEM_SPARSEMAT)
+   {
+      Mesh::GeometryList elem_geoms(*ran_fes.GetMesh());
+
+      DenseTensor localP[Geometry::NumGeom];
+      for (int i = 0; i < elem_geoms.Size(); i++)
+      {
+         ran_fes.GetLocalRefinementMatrices(dom_fes, elem_geoms[i],
+                                            localP[elem_geoms[i]]);
+      }
+      F.Reset(ran_fes.RefinementMatrix_main(
+                 dom_fes.GetNDofs(), dom_fes.GetElementToDofTable(), localP));
+   }
+   else
+   {
+      MFEM_ABORT("Operator::Type is not supported: " << oper_type);
+   }
+
+   return *F.Ptr();
+}
+
+const Operator &InterpolationGridTransfer::BackwardOperator()
+{
+   if (B.Ptr())
+   {
+      return *B.Ptr();
+   }
+
+   // Construct B
+   // If not set, define a suitable mass_integ
+   if (!mass_integ && ran_fes.GetNE() > 0)
+   {
+      const FiniteElement *f_fe_0 = ran_fes.GetFE(0);
+      const int map_type = f_fe_0->GetMapType();
+      if (map_type == FiniteElement::VALUE ||
+          map_type == FiniteElement::INTEGRAL)
+      {
+         mass_integ = new MassIntegrator;
+      }
+      else if (map_type == FiniteElement::H_DIV ||
+               map_type == FiniteElement::H_CURL)
+      {
+         mass_integ = new VectorFEMassIntegrator;
+      }
+      else
+      {
+         MFEM_ABORT("unknown type of FE space");
+      }
+      own_mass_integ = true;
+   }
+   if (oper_type == Operator::ANY_TYPE)
+   {
+      B.Reset(new FiniteElementSpace::DerefinementOperator(
+                 &ran_fes, &dom_fes, mass_integ));
+   }
+   else
+   {
+      MFEM_ABORT("Operator::Type is not supported: " << oper_type);
+   }
+
+   return *B.Ptr();
+}
+
+
+L2ProjectionGridTransfer::L2Projection::L2Projection(
+   const FiniteElementSpace &fes_ho_, const FiniteElementSpace &fes_lor_)
+   : fes_ho(fes_ho_), fes_lor(fes_lor_)
+{
+   Mesh *mesh_ho = fes_ho.GetMesh();
+   MFEM_VERIFY(mesh_ho->GetNumGeometries(mesh_ho->Dimension()) <= 1,
+               "mixed meshes are not supported");
+
+   // If the local mesh is empty, skip all computations
+   if (mesh_ho->GetNE() == 0) { return; }
+
+   const FiniteElement *fe_lor = fes_lor.GetFE(0);
+   const FiniteElement *fe_ho = fes_ho.GetFE(0);
+   ndof_lor = fe_lor->GetDof();
+   ndof_ho = fe_ho->GetDof();
+
+   const int nel_lor = fes_lor.GetNE();
+   const int nel_ho = fes_ho.GetNE();
+
+   nref = nel_lor/nel_ho;
+
+   // Construct the mapping from HO to LOR
+   // ho2lor.GetRow(iho) will give all the LOR elements contained in iho
+   ho2lor.SetSize(nel_ho, nref);
+   const CoarseFineTransformations &cf_tr =
+      fes_lor.GetMesh()->GetRefinementTransforms();
+   for (int ilor=0; ilor<nel_lor; ++ilor)
+   {
+      int iho = cf_tr.embeddings[ilor].parent;
+      ho2lor.AddConnection(iho, ilor);
+   }
+   ho2lor.ShiftUpI();
+
+   // R will contain the restriction (L^2 projection operator) defined on
+   // each coarse HO element (and corresponding patch of LOR elements)
+   R.SetSize(ndof_lor*nref, ndof_ho, nel_ho);
+   // P will contain the corresponding prolongation operator
+   P.SetSize(ndof_ho, ndof_lor*nref, nel_ho);
+
+   DenseMatrix Minv_lor(ndof_lor*nref, ndof_lor*nref);
+   DenseMatrix M_mixed(ndof_lor*nref, ndof_ho);
+
+   MassIntegrator mi;
+   DenseMatrix M_lor_el(ndof_lor, ndof_lor);
+   DenseMatrixInverse Minv_lor_el(&M_lor_el);
+   DenseMatrix M_lor(ndof_lor*nref, ndof_lor*nref);
+   DenseMatrix M_mixed_el(ndof_lor, ndof_ho);
+
+   Minv_lor = 0.0;
+   M_lor = 0.0;
+
+   DenseMatrix RtMlor(ndof_ho, ndof_lor*nref);
+   DenseMatrix RtMlorR(ndof_ho, ndof_ho);
+   DenseMatrixInverse RtMlorR_inv(&RtMlorR);
+
+   IntegrationPointTransformation ip_tr;
+   IsoparametricTransformation &emb_tr = ip_tr.Transf;
+
+   Vector shape_ho(ndof_ho);
+   Vector shape_lor(ndof_lor);
+
+   const Geometry::Type geom = fe_ho->GetGeomType();
+   const DenseTensor &pmats = cf_tr.GetPointMatrices(geom);
+   emb_tr.SetIdentityTransformation(geom);
+
+   for (int iho=0; iho<nel_ho; ++iho)
+   {
+      for (int iref=0; iref<nref; ++iref)
+      {
+         // Assemble the low-order refined mass matrix and invert locally
+         int ilor = ho2lor.GetRow(iho)[iref];
+         ElementTransformation *el_tr = fes_lor.GetElementTransformation(ilor);
+         mi.AssembleElementMatrix(*fe_lor, *el_tr, M_lor_el);
+         M_lor.CopyMN(M_lor_el, iref*ndof_lor, iref*ndof_lor);
+         Minv_lor_el.Factor();
+         Minv_lor_el.GetInverseMatrix(M_lor_el);
+         // Insert into the diagonal of the patch LOR mass matrix
+         Minv_lor.CopyMN(M_lor_el, iref*ndof_lor, iref*ndof_lor);
+
+         // Now assemble the block-row of the mixed mass matrix associated
+         // with integrating HO functions against LOR functions on the LOR
+         // sub-element.
+
+         // Create the transformation that embeds the fine low-order element
+         // within the coarse high-order element in reference space
+         emb_tr.GetPointMat() = pmats(iref);
+         emb_tr.FinalizeTransformation();
+
+         int order = fe_lor->GetOrder() + fe_ho->GetOrder() + el_tr->OrderW();
+         const IntegrationRule *ir = &IntRules.Get(geom, order);
+         M_mixed_el = 0.0;
+         for (int i = 0; i < ir->GetNPoints(); i++)
+         {
+            const IntegrationPoint &ip_lor = ir->IntPoint(i);
+            IntegrationPoint ip_ho;
+            ip_tr.Transform(ip_lor, ip_ho);
+            fe_lor->CalcShape(ip_lor, shape_lor);
+            fe_ho->CalcShape(ip_ho, shape_ho);
+            el_tr->SetIntPoint(&ip_lor);
+            // For now we use the geometry information from the LOR space
+            // which means we won't be mass conservative if the mesh is curved
+            double w = el_tr->Weight()*ip_lor.weight;
+            shape_lor *= w;
+            AddMultVWt(shape_lor, shape_ho, M_mixed_el);
+         }
+         M_mixed.CopyMN(M_mixed_el, iref*ndof_lor, 0);
+      }
+      mfem::Mult(Minv_lor, M_mixed, R(iho));
+
+      mfem::MultAtB(R(iho), M_lor, RtMlor);
+      mfem::Mult(RtMlor, R(iho), RtMlorR);
+      RtMlorR_inv.Factor();
+      RtMlorR_inv.Mult(RtMlor, P(iho));
+   }
+}
+
+void L2ProjectionGridTransfer::L2Projection::Mult(
+   const Vector &x, Vector &y) const
+{
+   int vdim = fes_ho.GetVDim();
+   Array<int> vdofs;
+   DenseMatrix xel_mat(ndof_ho, vdim);
+   DenseMatrix yel_mat(ndof_lor*nref, vdim);
+   for (int iho=0; iho<fes_ho.GetNE(); ++iho)
+   {
+      fes_ho.GetElementVDofs(iho, vdofs);
+      x.GetSubVector(vdofs, xel_mat.GetData());
+      mfem::Mult(R(iho), xel_mat, yel_mat);
+      // Place result correctly into low-order vector
+      for (int iref=0; iref<nref; ++iref)
+      {
+         int ilor = ho2lor.GetRow(iho)[iref];
+         for (int vd=0; vd<vdim; ++vd)
+         {
+            fes_lor.GetElementDofs(ilor, vdofs);
+            fes_lor.DofsToVDofs(vd, vdofs);
+            y.SetSubVector(vdofs, &yel_mat(iref*ndof_lor,vd));
+         }
+      }
+   }
+}
+
+void L2ProjectionGridTransfer::L2Projection::Prolongate(
+   const Vector &x, Vector &y) const
+{
+   int vdim = fes_ho.GetVDim();
+   Array<int> vdofs;
+   DenseMatrix xel_mat(ndof_lor*nref, vdim);
+   DenseMatrix yel_mat(ndof_ho, vdim);
+   for (int iho=0; iho<fes_ho.GetNE(); ++iho)
+   {
+      // Extract the LOR DOFs
+      for (int iref=0; iref<nref; ++iref)
+      {
+         int ilor = ho2lor.GetRow(iho)[iref];
+         for (int vd=0; vd<vdim; ++vd)
+         {
+            fes_lor.GetElementDofs(ilor, vdofs);
+            fes_lor.DofsToVDofs(vd, vdofs);
+            x.GetSubVector(vdofs, &xel_mat(iref*ndof_lor, vd));
+         }
+      }
+      // Locally prolongate
+      mfem::Mult(P(iho), xel_mat, yel_mat);
+      // Place the result in the HO vector
+      fes_ho.GetElementVDofs(iho, vdofs);
+      y.SetSubVector(vdofs, yel_mat.GetData());
+   }
+}
+
+const Operator &L2ProjectionGridTransfer::ForwardOperator()
+{
+   if (!F) { F = new L2Projection(dom_fes, ran_fes); }
+   return *F;
+}
+
+const Operator &L2ProjectionGridTransfer::BackwardOperator()
+{
+   if (!B)
+   {
+      if (!F) { F = new L2Projection(dom_fes, ran_fes); }
+      B = new L2Prolongation(*F);
+   }
+   return *B;
+}
+
 } // namespace mfem
@@ -59,12 +59,17 @@ Ordering::Map<Ordering::byVDIM>(int ndofs, int vdim, int dof, int vd)
 }


+// Forward declarations
 class NURBSExtension;
+class BilinearFormIntegrator;
+

 /** @brief Class FiniteElementSpace - responsible for providing FEM view of the
    mesh, mainly managing the set of degrees of freedom. */
 class FiniteElementSpace
 {
+   friend class InterpolationGridTransfer;
+
 protected:
   /// The mesh that FE space lives on (not owned).
   Mesh *mesh;
@@ -150,6 +155,25 @@ protected:
      virtual ~RefinementOperator();
   };

+   // Derefinement operator, used by the friend class InterpolationGridTransfer.
+   class DerefinementOperator : public Operator
+   {
+      const FiniteElementSpace *fine_fes; // Not owned.
+      DenseTensor localR[Geometry::NumGeom];
+      Table *coarse_elem_dof; // Owned.
+      Table coarse_to_fine;
+      Array<int> coarse_to_ref_type;
+      Array<Geometry::Type> ref_type_to_geom;
+      Array<int> ref_type_to_fine_elem_offset;
+
+   public:
+      DerefinementOperator(const FiniteElementSpace *f_fes,
+                           const FiniteElementSpace *c_fes,
+                           BilinearFormIntegrator *mass_integ);
+      virtual void Mult(const Vector &x, Vector &y) const;
+      virtual ~DerefinementOperator();
+   };
+
   // This method makes the same assumptions as the method:
   //    void GetLocalRefinementMatrices(
   //       const FiniteElementSpace &coarse_fes, Geometry::Type geom,
@@ -573,6 +597,215 @@ public:
   void Save(std::ostream &out) const;
 };

+
+/** @brief Base class for transfer algorithms that construct transfer Operator%s
+    between two finite element (FE) spaces. */
+/** Generally, the two FE spaces (domain and range) can be defined on different
+    meshes. */
+class GridTransfer
+{
+protected:
+   FiniteElementSpace &dom_fes; ///< Domain FE space
+   FiniteElementSpace &ran_fes; ///< Range FE space
+
+   /** @brief Desired Operator::Type for the construction of all operators
+       defined by the underlying transfer algorithm. It can be ignored by
+       derived classes. */
+   Operator::Type oper_type;
+
+   OperatorHandle fw_t_oper; ///< Forward true-dof operator
+   OperatorHandle bw_t_oper; ///< Backward true-dof operator
+
+#ifdef MFEM_USE_MPI
+   bool parallel;
+#endif
+   bool Parallel() const
+   {
+#ifndef MFEM_USE_MPI
+      return false;
+#else
+      return parallel;
+#endif
+   }
+
+   const Operator &MakeTrueOperator(FiniteElementSpace &fes_in,
+                                    FiniteElementSpace &fes_out,
+                                    const Operator &oper,
+                                    OperatorHandle &t_oper);
+
+public:
+   /** Construct a transfer algorithm between the domain, @a dom_fes_, and
+       range, @a ran_fes_, FE spaces. */
+   GridTransfer(FiniteElementSpace &dom_fes_, FiniteElementSpace &ran_fes_);
+
+   /// Virtual destructor
+   virtual ~GridTransfer() { }
+
+   /** @brief Set the desired Operator::Type for the construction of all
+       operators defined by the underlying transfer algorithm. */
+   /** The default value is Operator::ANY_TYPE which typically corresponds to
+       a matrix-free operator representation. Note that derived classes are not
+       required to support this setting and can ignore it. */
+   void SetOperatorType(Operator::Type type) { oper_type = type; }
+
+   /** @brief Return an Operator that transfers GridFunction%s from the domain
+       FE space to GridFunction%s in the range FE space. */
+   virtual const Operator &ForwardOperator() = 0;
+
+   /** @brief Return an Operator that transfers GridFunction%s from the range
+       FE space back to GridFunction%s in the domain FE space. */
+   virtual const Operator &BackwardOperator() = 0;
+
+   /** @brief Return an Operator that transfers true-dof Vector%s from the
+       domain FE space to true-dof Vector%s in the range FE space. */
+   /** This method is implemented in the base class, based on ForwardOperator(),
+       however, derived classes can overload the construction, if necessary. */
+   virtual const Operator &TrueForwardOperator()
+   {
+      return MakeTrueOperator(dom_fes, ran_fes, ForwardOperator(), fw_t_oper);
+   }
+
+   /** @brief Return an Operator that transfers true-dof Vector%s from the
+       range FE space back to true-dof Vector%s in the domain FE space. */
+   /** This method is implemented in the base class, based on
+       BackwardOperator(), however, derived classes can overload the
+       construction, if necessary. */
+   virtual const Operator &TrueBackwardOperator()
+   {
+      return MakeTrueOperator(ran_fes, dom_fes, BackwardOperator(), bw_t_oper);
+   }
+};
+
+
+/** @brief Transfer data between a coarse mesh and an embedded refined mesh
+    using interpolation. */
+/** The forward, coarse-to-fine, transfer uses nodal interpolation. The
+    backward, fine-to-coarse, transfer is defined locally (on a coarse element)
+    as B = (F^t M_f F)^{-1} F^t M_f, where F is the forward transfer matrix, and
+    M_f is a mass matrix on the union of all fine elements comprising the coarse
+    element. Note that the backward transfer operator, B, is a left inverse of
+    the forward transfer operator, F, i.e. B F = I. Both F and B are defined in
+    reference space and do not depend on the actual physical shape of the mesh
+    elements.
+
+    It is assumed that both the coarse and the fine FiniteElementSpace%s use
+    compatible types of elements, e.g. finite elements with the same map-type
+    (VALUE, INTEGRAL, H_DIV, H_CURL - see class FiniteElement). Generally, the
+    FE spaces can have different orders, however, in order for the backward
+    operator to be well-defined, the (local) number of the fine dofs should not
+    be smaller than the number of coarse dofs. */
+class InterpolationGridTransfer : public GridTransfer
+{
+protected:
+   BilinearFormIntegrator *mass_integ; ///< Ownership depends on #own_mass_integ
+   bool own_mass_integ; ///< Ownership flag for #mass_integ
+
+   OperatorHandle F; ///< Forward, coarse-to-fine, operator
+   OperatorHandle B; ///< Backward, fine-to-coarse, operator
+
+public:
+   InterpolationGridTransfer(FiniteElementSpace &coarse_fes,
+                             FiniteElementSpace &fine_fes)
+      : GridTransfer(coarse_fes, fine_fes),
+        mass_integ(NULL), own_mass_integ(false)
+   { }
+
+   virtual ~InterpolationGridTransfer();
+
+   /** @brief Assign a mass integrator to be used in the construction of the
+       backward, fine-to-coarse, transfer operator. */
+   void SetMassIntegrator(BilinearFormIntegrator *mass_integ_,
+                          bool own_mass_integ_ = true);
+
+   virtual const Operator &ForwardOperator();
+
+   virtual const Operator &BackwardOperator();
+};
+
+
+/** @brief Transfer data between a coarse mesh and an embedded refined mesh
+    using L2 projection. */
+/** The forward, coarse-to-fine, transfer uses L2 projection. The backward,
+    fine-to-coarse, transfer is defined locally (on a coarse element) as
+    B = (F^t M_f F)^{-1} F^t M_f, where F is the forward transfer matrix, and
+    M_f is the mass matrix on the union of all fine elements comprising the
+    coarse element. Note that the backward transfer operator, B, is a left
+    inverse of the forward transfer operator, F, i.e. B F = I. Both F and B are
+    defined in physical space and, generally, vary between different mesh
+    elements.
+
+    This class currently only fully supports L2 finite element spaces and fine
+    meshes that are a uniform refinement of the coarse mesh. Generally, the
+    coarse and fine FE spaces can have different orders, however, in order for
+    the backward operator to be well-defined, the number of the fine dofs (in a
+    coarse element) should not be smaller than the number of coarse dofs.
+
+    If used on H1 finite element spaces, the transfer will be performed locally,
+    and the value of shared (interface) degrees of freedom will be determined by
+    the value of the last transfer to be performed (according to the element
+    numbering in the finite element space). As a consequence, the mass
+    conservation properties for this operator from the L2 case do not carry over
+    to H1 spaces. */
+class L2ProjectionGridTransfer : public GridTransfer
+{
+protected:
+   /** Class representing projection operator between a high-order L2 finite
+       element space on a coarse mesh, and a low-order L2 finite element space
+       on a refined mesh (LOR). We assume that the low-order space, fes_lor,
+       lives on a mesh obtained by refining the mesh of the high-order space,
+       fes_ho. */
+   class L2Projection : public Operator
+   {
+      const FiniteElementSpace &fes_ho;
+      const FiniteElementSpace &fes_lor;
+
+      int ndof_lor, ndof_ho, nref;
+
+      Table ho2lor;
+
+      DenseTensor R, P;
+
+   public:
+      L2Projection(const FiniteElementSpace &fes_ho_,
+                   const FiniteElementSpace &fes_lor_);
+      /// Perform the L2 projection onto the LOR space
+      virtual void Mult(const Vector &x, Vector &y) const;
+      /// Perform the mass conservative left-inverse prolongation operation.
+      /// This functionality is also provided as an Operator by L2Prolongation.
+      void Prolongate(const Vector &x, Vector &y) const;
+      virtual ~L2Projection() { }
+   };
+
+   /** Mass-conservative prolongation operator going in the opposite direction
+       as L2Projection. This operator is a left inverse to the L2Projection. */
+   class L2Prolongation : public Operator
+   {
+      const L2Projection &l2proj;
+
+   public:
+      L2Prolongation(const L2Projection &l2proj_) : l2proj(l2proj_) { }
+      void Mult(const Vector &x, Vector &y) const
+      {
+         l2proj.Prolongate(x, y);
+      }
+      virtual ~L2Prolongation() { }
+   };
+
+   L2Projection   *F; ///< Forward, coarse-to-fine, operator
+   L2Prolongation *B; ///< Backward, fine-to-coarse, operator
+
+public:
+   L2ProjectionGridTransfer(FiniteElementSpace &coarse_fes,
+                            FiniteElementSpace &fine_fes)
+      : GridTransfer(coarse_fes, fine_fes),
+        F(NULL), B(NULL)
+   { }
+
+   virtual const Operator &ForwardOperator();
+
+   virtual const Operator &BackwardOperator();
+};
+
 }

 #endif
@@ -104,7 +104,9 @@ public:
   GridFunction &operator=(const GridFunction &rhs)
   { return operator=((const Vector &)rhs); }

-   /// Make the GridFunction the owner of 'fec' and 'fes'
+   /// Make the GridFunction the owner of #fec and #fes.
+   /** If the new FiniteElementCollection, @a _fec, is NULL, ownership of #fec
+       and #fes is taken away. */
   void MakeOwner(FiniteElementCollection *_fec) { fec = _fec; }

   FiniteElementCollection *OwnFEC() { return fec; }
@@ -886,7 +886,7 @@ const IntegrationRule &IntegrationRules::Get(int GeomType, int Order)

   if (!HaveIntRule(*ir_array, Order))
   {
-#ifdef MFEM_USE_OPENMP
+#ifdef MFEM_USE_LEGACY_OPENMP
      #pragma omp critical
 #endif
      {
@@ -1595,20 +1595,20 @@ IntegrationRule *IntegrationRules::TetrahedronIntegrationRule(int Order)
 // Integration rules for reference prism
 IntegrationRule *IntegrationRules::PrismIntegrationRule(int Order)
 {
-   IntegrationRule * irt = GenerateIntegrationRule(Geometry::TRIANGLE, Order);
-   IntegrationRule * irs = GenerateIntegrationRule(Geometry::SEGMENT, Order);
-   int nt = irt->GetNPoints();
-   int ns = irs->GetNPoints();
+   const IntegrationRule & irt = Get(Geometry::TRIANGLE, Order);
+   const IntegrationRule & irs = Get(Geometry::SEGMENT, Order);
+   int nt = irt.GetNPoints();
+   int ns = irs.GetNPoints();
   AllocIntRule(PrismIntRules, Order);
   PrismIntRules[Order] = new IntegrationRule(nt * ns);

   for (int ks=0; ks<ns; ks++)
   {
-      const IntegrationPoint & ips = irs->IntPoint(ks);
+      const IntegrationPoint & ips = irs.IntPoint(ks);
      for (int kt=0; kt<nt; kt++)
      {
         int kp = ks * nt + kt;
-         const IntegrationPoint & ipt = irt->IntPoint(kt);
+         const IntegrationPoint & ipt = irt.IntPoint(kt);
         IntegrationPoint & ipp = PrismIntRules[Order]->IntPoint(kp);
         ipp.x = ipt.x;
         ipp.y = ipt.y;
@@ -325,6 +325,9 @@ private:
      return Order | 1; // valid for all quad_type's
   }

+   /// The following methods allocate new IntegrationRule objects without
+   /// checking if they already exist.  To avoid memory leaks use
+   /// IntegrationRules::Get(int GeomType, int Order) instead.
   IntegrationRule *GenerateIntegrationRule(int GeomType, int Order);
   IntegrationRule *PointIntegrationRule(int Order);
   IntegrationRule *SegmentIntegrationRule(int Order);
@@ -0,0 +1,999 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#if Q1D < D1D
+#  define M1D D1D
+#else
+#  define M1D Q1D
+#endif
+#define M2D (M1D*M1D)
+
+#define DQ1D (D1D*Q1D)
+
+#define Q2D (Q1D*Q1D)
+#define Q3D (Q1D*Q1D*Q1D)
+
+#define M2_ELEMENT_BATCH 32
+
+#define QUAD_2D_ID(X, Y) (X + ((Y) * Q1D))
+#define QUAD_3D_ID(X, Y, Z) (X + ((Y) * Q1D) + ((Z) * Q2D))
+
+typedef double* DofToQuad_t @dim(Q1D, D1D);
+typedef double* QuadToDof_t @dim(D1D, Q1D);
+
+typedef double* DLocal2D_t @dim(D1D, D1D, NE);
+typedef double* QLocal2D_t @dim(Q1D, Q1D, NE);
+
+typedef double* DLocal3D_t @dim(D1D, D1D, D1D, NE);
+typedef double* QLocal3D_t @dim(Q1D, Q1D, Q1D, NE);
+
+typedef double* Jacobian2D_t @dim(2, 2, Q2D, NE);
+typedef double* Jacobian3D_t @dim(3, 3, Q3D, NE);
+
+typedef double* SymmOperator2D_t @dim(3, Q2D, NE);
+typedef double* SymmOperator3D_t @dim(6, Q3D, NE);
+
+@kernel void DiffusionSetup2D(const int NE,
+                              @restrict const double *W,
+                              @restrict const Jacobian2D_t J,
+                              const double COEFF,
+                              @restrict SymmOperator2D_t op) {
+  for (int e = 0; e < NE; ++e; @outer) {
+    for (int q = 0; q < Q2D; ++q; @inner) {
+      const double J11 = J(0, 0, q, e), J12 = J(1, 0, q, e);
+      const double J21 = J(0, 1, q, e), J22 = J(1, 1, q, e);
+      const double c_detJ = W[q] * COEFF / ((J11 * J22) - (J21 * J12));
+      op(0, q, e) =  c_detJ * (J21*J21 + J22*J22); // (1,1)
+      op(1, q, e) = -c_detJ * (J21*J11 + J22*J12); // (1,2), (2,1)
+      op(2, q, e) =  c_detJ * (J11*J11 + J12*J12); // (2,2)
+    }
+  }
+}
+
+@kernel void DiffusionSetup3D(const int NE,
+                              @restrict const double *W,
+                              @restrict const Jacobian3D_t J,
+                              const double COEFF,
+                              @restrict SymmOperator3D_t op) {
+  for (int e = 0; e < NE; ++e; @outer) {
+    for (int q = 0; q < Q3D; ++q; @inner) {
+      const double J11 = J(0, 0, q, e), J12 = J(1, 0, q, e), J13 = J(2, 0, q, e);
+      const double J21 = J(0, 1, q, e), J22 = J(1, 1, q, e), J23 = J(2, 1, q, e);
+      const double J31 = J(0, 2, q, e), J32 = J(1, 2, q, e), J33 = J(2, 2, q, e);
+
+      const double detJ = ((J11 * J22 * J33) + (J12 * J23 * J31) + (J13 * J21 * J32) -
+                           (J13 * J22 * J31) - (J12 * J21 * J33) - (J11 * J23 * J32));
+
+      const double c_detJ = W[q] * COEFF / detJ;
+
+      // adj(J)
+      const double A11 = (J22 * J33) - (J23 * J32);
+      const double A12 = (J23 * J31) - (J21 * J33);
+      const double A13 = (J21 * J32) - (J22 * J31);
+
+      const double A21 = (J13 * J32) - (J12 * J33);
+      const double A22 = (J11 * J33) - (J13 * J31);
+      const double A23 = (J12 * J31) - (J11 * J32);
+
+      const double A31 = (J12 * J23) - (J13 * J22);
+      const double A32 = (J13 * J21) - (J11 * J23);
+      const double A33 = (J11 * J22) - (J12 * J21);
+
+      // adj(J)^Tadj(J)
+      op(0, q, e) = c_detJ * (A11*A11 + A21*A21 + A31*A31); // (1,1)
+      op(1, q, e) = c_detJ * (A11*A12 + A21*A22 + A31*A32); // (1,2), (2,1)
+      op(2, q, e) = c_detJ * (A11*A13 + A21*A23 + A31*A33); // (1,3), (3,1)
+      op(3, q, e) = c_detJ * (A12*A12 + A22*A22 + A32*A32); // (2,2)
+      op(4, q, e) = c_detJ * (A12*A13 + A22*A23 + A32*A33); // (2,3), (3,2)
+      op(5, q, e) = c_detJ * (A13*A13 + A23*A23 + A33*A33); // (3,3)
+    }
+  }
+}
+
+@kernel void DiffusionApply2D_CPU(const int NE,
+                                  @restrict const DofToQuad_t B,
+                                  @restrict const DofToQuad_t G,
+                                  @restrict const QuadToDof_t Bt,
+                                  @restrict const QuadToDof_t Gt,
+                                  @restrict const SymmOperator2D_t op,
+                                  @restrict const DLocal2D_t X,
+                                  @restrict DLocal2D_t Y) {
+   // Iterate over elements
+   for (int e = 0; e < NE; ++e; @outer) {
+      for (int dummy = 0; dummy < 1; ++dummy; @inner) {
+         double grad[Q1D][Q1D][2];
+         for (int qy = 0; qy < Q1D; ++qy) {
+            for (int qx = 0; qx < Q1D; ++qx) {
+               grad[qy][qx][0] = 0;
+               grad[qy][qx][1] = 0;
+            }
+         }
+
+         for (int dy = 0; dy < D1D; ++dy) {
+            double gradX[Q1D][2];
+            for (int qx = 0; qx < Q1D; ++qx) {
+               gradX[qx][0] = 0;
+               gradX[qx][1] = 0;
+            }
+
+            for (int dx = 0; dx < D1D; ++dx) {
+               const double s = X(dx, dy, e);
+               for (int qx = 0; qx < Q1D; ++qx) {
+                  gradX[qx][0] += s * B(qx, dx);
+                  gradX[qx][1] += s * G(qx, dx);
+               }
+            }
+
+            for (int qy = 0; qy < Q1D; ++qy) {
+               const double wy  = B(qy, dy);
+               const double wDy = G(qy, dy);
+               for (int qx = 0; qx < Q1D; ++qx) {
+                  grad[qy][qx][0] += gradX[qx][1] * wy;
+                  grad[qy][qx][1] += gradX[qx][0] * wDy;
+               }
+            }
+         }
+
+         // Calculate Dxy, xDy in plane
+         for (int qy = 0; qy < Q1D; ++qy) {
+            for (int qx = 0; qx < Q1D; ++qx) {
+               const int q = QUAD_2D_ID(qx, qy);
+               const double O11 = op(0, q, e);
+               const double O12 = op(1, q, e);
+               const double O22 = op(2, q, e);
+
+               const double gradX = grad[qy][qx][0];
+               const double gradY = grad[qy][qx][1];
+
+               grad[qy][qx][0] = (O11 * gradX) + (O12 * gradY);
+               grad[qy][qx][1] = (O12 * gradX) + (O22 * gradY);
+            }
+         }
+
+         for (int qy = 0; qy < Q1D; ++qy) {
+            double gradX[D1D][2];
+            for (int dx = 0; dx < D1D; ++dx) {
+               gradX[dx][0] = 0;
+               gradX[dx][1] = 0;
+            }
+
+            for (int qx = 0; qx < Q1D; ++qx) {
+               const double gX = grad[qy][qx][0];
+               const double gY = grad[qy][qx][1];
+               for (int dx = 0; dx < D1D; ++dx) {
+                  const double wx  = Bt(dx, qx);
+                  const double wDx = Gt(dx, qx);
+                  gradX[dx][0] += gX * wDx;
+                  gradX[dx][1] += gY * wx;
+               }
+            }
+
+            for (int dy = 0; dy < D1D; ++dy) {
+               const double wy  = Bt(dy, qy);
+               const double wDy = Gt(dy, qy);
+               for (int dx = 0; dx < D1D; ++dx) {
+                  Y(dx, dy, e) += ((gradX[dx][0] * wy) +
+                                   (gradX[dx][1] * wDy));
+               }
+            }
+         }
+      }
+   }
+}
+
+@kernel void DiffusionApply2D_GPU(const int NE,
+                                  @restrict const DofToQuad_t B,
+                                  @restrict const DofToQuad_t G,
+                                  @restrict const QuadToDof_t Bt,
+                                  @restrict const QuadToDof_t Gt,
+                                  @restrict const SymmOperator2D_t op,
+                                  @restrict const DLocal2D_t X,
+                                  @restrict DLocal2D_t Y) {
+   // Iterate over elements
+   for (int eOff = 0; eOff < NE; eOff += M2_ELEMENT_BATCH; @outer) {
+      // Store dof <--> quad mappings
+      @shared double s_B[DQ1D] @dim(Q1D, D1D);
+      @shared double s_G[DQ1D] @dim(Q1D, D1D);
+      @shared double s_Bt[DQ1D] @dim(D1D, Q1D);
+      @shared double s_Gt[DQ1D] @dim(D1D, Q1D);
+
+      // Store xy planes in @shared memory
+      @shared double s_xy[DQ1D] @dim(D1D, Q1D);
+      @shared double s_xDy[DQ1D] @dim(D1D, Q1D);
+      @shared double s_grad[2 * Q2D] @dim(2, Q1D, Q1D);
+
+      @exclusive double r_x[M1D];
+      @exclusive double r_y[Q1D];
+
+      for (int x = 0; x < M1D; ++x; @inner) {
+         for (int id = x; id < DQ1D; id += M1D) {
+            s_B[id]  = B[id];
+            s_G[id] = G[id];
+            s_Bt[id]  = Bt[id];
+            s_Gt[id] = Gt[id];
+         }
+      }
+
+      for (int e = eOff; e < (eOff + M2_ELEMENT_BATCH); ++e) {
+         if (e < NE) {
+            for (int dx = 0; dx < D1D; ++dx; @inner) {
+               if (dx < D1D) {
+                  for (int qy = 0; qy < Q1D; ++qy) {
+                     s_xy(dx, qy) = 0;
+                     s_xDy(dx, qy) = 0;
+                  }
+                  for (int dy = 0; dy < D1D; ++dy) {
+                     r_x[dy] = X(dx, dy, e);
+                  }
+                  for (int qy = 0; qy < Q1D; ++qy) {
+                     double xy = 0;
+                     double xDy = 0;
+                     for (int dy = 0; dy < D1D; ++dy) {
+                        xy  += r_x[dy] * s_B(qy, dy);
+                        xDy += r_x[dy] * s_G(qy, dy);
+                     }
+                     s_xy(dx, qy)  = xy;
+                     s_xDy(dx, qy) = xDy;
+                  }
+               }
+            }
+            for (int qy = 0; qy < M1D; ++qy; @inner) {
+               if (qy < Q1D) {
+                  for (int qx = 0; qx < Q1D; ++qx) {
+                     double gradX = 0, gradY = 0;
+                     for (int dx = 0; dx < D1D; ++dx) {
+                        gradX += s_xy(dx, qy)  * s_G(qx, dx);
+                        gradY += s_xDy(dx, qy) * s_B(qx, dx);
+                     }
+
+                     const int q = QUAD_2D_ID(qx, qy);
+                     const double O11 = op(0, q, e);
+                     const double O12 = op(1, q, e);
+                     const double O22 = op(2, q, e);
+
+                     s_grad(0, qx, qy) = (O11 * gradX) + (O12 * gradY);
+                     s_grad(1, qx, qy) = (O12 * gradX) + (O22 * gradY);
+                  }
+               }
+            }
+
+            for (int qx = 0; qx < Q1D; ++qx; @inner) {
+               if (qx < Q1D) {
+                  for (int dy = 0; dy < D1D; ++dy) {
+                     s_xy(dy, qx) = 0;
+                     s_xDy(dy, qx) = 0;
+                  }
+                  for (int qy = 0; qy < Q1D; ++qy) {
+                     r_x[qy] = s_grad(0, qx, qy);
+                     r_y[qy] = s_grad(1, qx, qy);
+                  }
+                  for (int dy = 0; dy < D1D; ++dy) {
+                     double xy  = 0;
+                     double xDy = 0;
+                     for (int qy = 0; qy < Q1D; ++qy) {
+                        xy  += r_x[qy] * s_Bt(dy, qy);
+                        xDy += r_y[qy] * s_Gt(dy, qy);
+                     }
+                     s_xy(dy, qx) = xy;
+                     s_xDy(dy, qx) = xDy;
+                  }
+               }
+            }
+            for (int dx = 0; dx < D1D; ++dx; @inner) {
+               if (dx < D1D) {
+                  for (int dy = 0; dy < D1D; ++dy) {
+                     double s = 0;
+                     for (int qx = 0; qx < Q1D; ++qx) {
+                        s += ((s_xy(dy, qx) * s_Gt(dx, qx)) +
+                              (s_xDy(dy, qx) * s_Bt(dx, qx)));
+                     }
+                     Y(dx, dy, e) += s;
+                  }
+               }
+            }
+         }
+      }
+   }
+}
+
+@kernel void DiffusionApply3D_CPU(const int NE,
+                                  @restrict const DofToQuad_t B,
+                                  @restrict const DofToQuad_t G,
+                                  @restrict const QuadToDof_t Bt,
+                                  @restrict const QuadToDof_t Gt,
+                                  @restrict const SymmOperator3D_t op,
+                                  @restrict const DLocal3D_t X,
+                                  @restrict DLocal3D_t Y) {
+  // Iterate over elements
+  for (int e = 0; e < NE; ++e; @outer) {
+    for (int dummy = 0; dummy < 1; ++dummy; @inner) {
+      double grad[Q1D][Q1D][Q1D][4];
+      for (int qz = 0; qz < Q1D; ++qz) {
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            grad[qz][qy][qx][0] = 0;
+            grad[qz][qy][qx][1] = 0;
+            grad[qz][qy][qx][2] = 0;
+          }
+        }
+      }
+
+      for (int dz = 0; dz < D1D; ++dz) {
+        double gradXY[Q1D][Q1D][4];
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            gradXY[qy][qx][0] = 0;
+            gradXY[qy][qx][1] = 0;
+            gradXY[qy][qx][2] = 0;
+          }
+        }
+
+        for (int dy = 0; dy < D1D; ++dy) {
+          double gradX[Q1D][2];
+          for (int qx = 0; qx < Q1D; ++qx) {
+            gradX[qx][0] = 0;
+            gradX[qx][1] = 0;
+          }
+
+          for (int dx = 0; dx < D1D; ++dx) {
+            const double s = X(dx, dy, dz, e);
+            for (int qx = 0; qx < Q1D; ++qx) {
+              gradX[qx][0] += s * B(qx, dx);
+              gradX[qx][1] += s * G(qx, dx);
+            }
+          }
+
+          for (int qy = 0; qy < Q1D; ++qy) {
+            const double wy  = B(qy, dy);
+            const double wDy = G(qy, dy);
+            for (int qx = 0; qx < Q1D; ++qx) {
+              const double wx  = gradX[qx][0];
+              const double wDx = gradX[qx][1];
+              gradXY[qy][qx][0] += wDx * wy;
+              gradXY[qy][qx][1] += wx  * wDy;
+              gradXY[qy][qx][2] += wx  * wy;
+            }
+          }
+        }
+
+        for (int qz = 0; qz < Q1D; ++qz) {
+          const double wz  = B(qz, dz);
+          const double wDz = G(qz, dz);
+          for (int qy = 0; qy < Q1D; ++qy) {
+            for (int qx = 0; qx < Q1D; ++qx) {
+              grad[qz][qy][qx][0] += gradXY[qy][qx][0] * wz;
+              grad[qz][qy][qx][1] += gradXY[qy][qx][1] * wz;
+              grad[qz][qy][qx][2] += gradXY[qy][qx][2] * wDz;
+            }
+          }
+        }
+      }
+
+      // Calculate Dxyz, xDyz, xyDz in plane
+      for (int qz = 0; qz < Q1D; ++qz) {
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            const int q = QUAD_3D_ID(qx, qy, qz);
+            const double O11 = op(0, q, e);
+            const double O12 = op(1, q, e);
+            const double O13 = op(2, q, e);
+            const double O22 = op(3, q, e);
+            const double O23 = op(4, q, e);
+            const double O33 = op(5, q, e);
+
+            const double gradX = grad[qz][qy][qx][0];
+            const double gradY = grad[qz][qy][qx][1];
+            const double gradZ = grad[qz][qy][qx][2];
+
+            grad[qz][qy][qx][0] = (O11 * gradX) + (O12 * gradY) + (O13 * gradZ);
+            grad[qz][qy][qx][1] = (O12 * gradX) + (O22 * gradY) + (O23 * gradZ);
+            grad[qz][qy][qx][2] = (O13 * gradX) + (O23 * gradY) + (O33 * gradZ);
+          }
+        }
+      }
+
+      for (int qz = 0; qz < Q1D; ++qz) {
+        double gradXY[D1D][D1D][4];
+        for (int dy = 0; dy < D1D; ++dy) {
+          for (int dx = 0; dx < D1D; ++dx) {
+            gradXY[dy][dx][0] = 0;
+            gradXY[dy][dx][1] = 0;
+            gradXY[dy][dx][2] = 0;
+          }
+        }
+
+        for (int qy = 0; qy < Q1D; ++qy) {
+          double gradX[D1D][4];
+          for (int dx = 0; dx < D1D; ++dx) {
+            gradX[dx][0] = 0;
+            gradX[dx][1] = 0;
+            gradX[dx][2] = 0;
+          }
+
+          for (int qx = 0; qx < Q1D; ++qx) {
+            const double gX = grad[qz][qy][qx][0];
+            const double gY = grad[qz][qy][qx][1];
+            const double gZ = grad[qz][qy][qx][2];
+            for (int dx = 0; dx < D1D; ++dx) {
+              const double wx  = Bt(dx, qx);
+              const double wDx = Gt(dx, qx);
+              gradX[dx][0] += gX * wDx;
+              gradX[dx][1] += gY * wx;
+              gradX[dx][2] += gZ * wx;
+            }
+          }
+
+          for (int dy = 0; dy < D1D; ++dy) {
+            const double wy  = Bt(dy, qy);
+            const double wDy = Gt(dy, qy);
+            for (int dx = 0; dx < D1D; ++dx) {
+              gradXY[dy][dx][0] += gradX[dx][0] * wy;
+              gradXY[dy][dx][1] += gradX[dx][1] * wDy;
+              gradXY[dy][dx][2] += gradX[dx][2] * wy;
+            }
+          }
+        }
+
+        for (int dz = 0; dz < D1D; ++dz) {
+          const double wz  = Bt(dz, qz);
+          const double wDz = Gt(dz, qz);
+          for (int dy = 0; dy < D1D; ++dy) {
+            for (int dx = 0; dx < D1D; ++dx) {
+              Y(dx, dy, dz, e) += ((gradXY[dy][dx][0] * wz) +
+                                   (gradXY[dy][dx][1] * wz) +
+                                   (gradXY[dy][dx][2] * wDz));
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+@kernel void DiffusionApply3D_GPU(const int NE,
+                                  @restrict const DofToQuad_t B,
+                                  @restrict const DofToQuad_t G,
+                                  @restrict const QuadToDof_t Bt,
+                                  @restrict const QuadToDof_t Gt,
+                                  @restrict const SymmOperator3D_t op,
+                                  @restrict const DLocal3D_t X,
+                                  @restrict DLocal3D_t Y) {
+  // Iterate over elements
+  for (int e = 0; e < NE; ++e; @outer) {
+    // Store dof <--> quad mappings
+    @shared double s_B[DQ1D] @dim(Q1D, D1D);
+    @shared double s_G[DQ1D] @dim(Q1D, D1D);
+    @shared double s_Bt[DQ1D] @dim(D1D, Q1D);
+    @shared double s_Gt[DQ1D] @dim(D1D, Q1D);
+
+    // Store xy planes in @shared memory
+    @shared double s_z[M2D] @dim(M1D, M1D);
+    @shared double s_Dz[M2D] @dim(M1D, M1D);
+    @shared double s_xyDz[Q2D] @dim(Q1D, Q1D);
+
+    // Store z axis as registers
+    @exclusive double r_qz[Q1D];
+    @exclusive double r_qDz[Q1D];
+    @exclusive double r_dDxyz[D1D];
+    @exclusive double r_dxDyz[D1D];
+    @exclusive double r_dxyDz[D1D];
+
+    for (int y = 0; y < M1D; ++y; @inner) {
+      for (int x = 0; x < M1D; ++x; @inner) {
+        const int id = (y * M1D) + x;
+        // Fetch Q <--> D maps
+        if (id < DQ1D) {
+          s_B[id]  = B[id];
+          s_G[id] = G[id];
+          s_Bt[id]  = Bt[id];
+          s_Gt[id] = Gt[id];
+        }
+        // Initialize our Z axis
+        for (int qz = 0; qz < Q1D; ++qz) {
+          r_qz[qz] = 0;
+          r_qDz[qz] = 0;
+        }
+        // Initialize our solution updates in the Z axis
+        for (int dz = 0; dz < D1D; ++dz) {
+          r_dDxyz[dz] = 0;
+          r_dxDyz[dz] = 0;
+          r_dxyDz[dz] = 0;
+        }
+      }
+    }
+
+    for (int dy = 0; dy < M1D; ++dy; @inner) {
+      for (int dx = 0; dx < M1D; ++dx; @inner) {
+        if ((dx < D1D) && (dy < D1D)) {
+          for (int dz = 0; dz < D1D; ++dz) {
+            const double s = X(dx, dy, dz, e);
+            // Calculate D -> Q in the Z axis
+            for (int qz = 0; qz < Q1D; ++qz) {
+              r_qz[qz]  += s * s_B(qz, dz);
+              r_qDz[qz] += s * s_G(qz, dz);
+            }
+          }
+        }
+      }
+    }
+    // For each xy plane
+    for (int qz = 0; qz < Q1D; ++qz) {
+      // Fill xy plane at given z position
+      for (int dy = 0; dy < M1D; ++dy; @inner) {
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if ((dx < D1D) && (dy < D1D)) {
+            s_z(dx, dy)  = r_qz[qz];
+            s_Dz(dx, dy) = r_qDz[qz];
+          }
+        }
+      }
+      // Calculate Dxyz, xDyz, xyDz in plane
+      for (int qy = 0; qy < M1D; ++qy; @inner) {
+        for (int qx = 0; qx < M1D; ++qx; @inner) {
+          if ((qx < Q1D) && (qy < Q1D)) {
+            double Dxyz = 0;
+            double xDyz = 0;
+            double xyDz = 0;
+            for (int dy = 0; dy < D1D; ++dy) {
+              const double wy  = s_B(qy, dy);
+              const double wDy = s_G(qy, dy);
+              for (int dx = 0; dx < D1D; ++dx) {
+                const double wx  = s_B(qx, dx);
+                const double wDx = s_G(qx, dx);
+                const double z  = s_z(dx, dy);
+                const double Dz = s_Dz(dx, dy);
+                Dxyz += wDx * wy  * z;
+                xDyz += wx  * wDy * z;
+                xyDz += wx  * wy  * Dz;
+              }
+            }
+
+            const int q = QUAD_3D_ID(qx, qy, qz);
+            const double O11 = op(0, q, e);
+            const double O12 = op(1, q, e);
+            const double O13 = op(2, q, e);
+            const double O22 = op(3, q, e);
+            const double O23 = op(4, q, e);
+            const double O33 = op(5, q, e);
+
+            const double qDxyz = (O11 * Dxyz) + (O12 * xDyz) + (O13 * xyDz);
+            const double qxDyz = (O12 * Dxyz) + (O22 * xDyz) + (O23 * xyDz);
+            const double qxyDz = (O13 * Dxyz) + (O23 * xDyz) + (O33 * xyDz);
+
+            for (int dz = 0; dz < D1D; ++dz) {
+              const double wz  = s_Bt(dz, qz);
+              const double wDz = s_Gt(dz, qz);
+              r_dDxyz[dz] += wz  * qDxyz;
+              r_dxDyz[dz] += wz  * qxDyz;
+              r_dxyDz[dz] += wDz * qxyDz;
+            }
+          }
+        }
+      }
+      @barrier("s_z_s_Dz_sync_1");
+    }
+    // Iterate over xy planes to compute solution
+    for (int dz = 0; dz < D1D; ++dz) {
+      // Place xy plane in @shared memory
+      for (int qy = 0; qy < M1D; ++qy; @inner) {
+        for (int qx = 0; qx < M1D; ++qx; @inner) {
+          if ((qx < Q1D) && (qy < Q1D)) {
+            s_z(qx, qy)    = r_dDxyz[dz];
+            s_Dz(qx, qy)   = r_dxDyz[dz];
+            s_xyDz(qx, qy) = r_dxyDz[dz];
+          }
+        }
+      }
+      // Finalize solution in xy plane
+      for (int dy = 0; dy < M1D; ++dy; @inner) {
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if ((dx < D1D) && (dy < D1D)) {
+            double solZ = 0;
+            for (int qy = 0; qy < Q1D; ++qy) {
+              const double wy  = s_Bt(dy, qy);
+              const double wDy = s_Gt(dy, qy);
+              for (int qx = 0; qx < Q1D; ++qx) {
+                const double wx  = s_Bt(dx, qx);
+                const double wDx = s_Gt(dx, qx);
+                const double Dxyz = s_z(qx, qy);
+                const double xDyz = s_Dz(qx, qy);
+                const double xyDz = s_xyDz(qx, qy);
+                solZ += ((wDx * wy  * Dxyz) +
+                         (wx  * wDy * xDyz) +
+                         (wx  * wy  * xyDz));
+              }
+            }
+            Y(dx, dy, dz, e) += solZ;
+          }
+        }
+      }
+      @barrier("s_z_s_Dz_s_xyDz_sync_1");
+    }
+  }
+}
+
+@kernel void MassApply2D_CPU(const int NE,
+                             @restrict const DofToQuad_t B,
+                             @restrict const QuadToDof_t Bt,
+                             @restrict const QLocal2D_t op,
+                             @restrict const DLocal2D_t X,
+                             @restrict DLocal2D_t Y) {
+  for (int e = 0; e < NE; ++e; @outer) {
+    for (int dummy = 0; dummy < 1; ++dummy; @inner) {
+      double sol_xy[Q1D][Q1D];
+
+      for (int qy = 0; qy < Q1D; ++qy) {
+        for (int qx = 0; qx < Q1D; ++qx) {
+          sol_xy[qy][qx] = 0;
+        }
+      }
+
+      for (int dy = 0; dy < D1D; ++dy) {
+        double sol_x[Q1D];
+        for (int qy = 0; qy < Q1D; ++qy) {
+          sol_x[qy] = 0;
+        }
+
+        for (int dx = 0; dx < D1D; ++dx) {
+          const double s = X(dx, dy, e);
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_x[qx] += B(qx, dx) * s;
+          }
+        }
+
+        for (int qy = 0; qy < Q1D; ++qy) {
+          const double d2q = B(qy, dy);
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_xy[qy][qx] += d2q * sol_x[qx];
+          }
+        }
+      }
+
+      for (int qy = 0; qy < Q1D; ++qy) {
+        for (int qx = 0; qx < Q1D; ++qx) {
+          sol_xy[qy][qx] *= op(qx, qy, e);
+        }
+      }
+
+      for (int qy = 0; qy < Q1D; ++qy) {
+        double sol_x[D1D];
+        for (int dx = 0; dx < D1D; ++dx) {
+          sol_x[dx] = 0;
+        }
+
+        for (int qx = 0; qx < Q1D; ++qx) {
+          const double s = sol_xy[qy][qx];
+          for (int dx = 0; dx < D1D; ++dx) {
+            sol_x[dx] += Bt(dx, qx) * s;
+          }
+        }
+
+        for (int dy = 0; dy < D1D; ++dy) {
+          const double q2d = Bt(dy, qy);
+          for (int dx = 0; dx < D1D; ++dx) {
+            Y(dx, dy, e) += q2d * sol_x[dx];
+          }
+        }
+      }
+    }
+  }
+}
+
+@kernel void MassApply3D_CPU(const int NE,
+                             @restrict const DofToQuad_t B,
+                             @restrict const QuadToDof_t Bt,
+                             @restrict const QLocal3D_t op,
+                             @restrict const DLocal3D_t X,
+                             @restrict DLocal3D_t Y) {
+  // Iterate over elements
+  for (int e = 0; e < NE; ++e; @outer) {
+    for (int dummy = 0; dummy < 1; ++dummy; @inner) {
+      double sol_xyz[Q1D][Q1D][Q1D];
+      for (int qz = 0; qz < Q1D; ++qz) {
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_xyz[qz][qy][qx] = 0;
+          }
+        }
+      }
+
+      for (int dz = 0; dz < D1D; ++dz) {
+        double sol_xy[Q1D][Q1D];
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_xy[qy][qx] = 0;
+          }
+        }
+
+        for (int dy = 0; dy < D1D; ++dy) {
+          double sol_x[Q1D];
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_x[qx] = 0;
+          }
+
+          for (int dx = 0; dx < D1D; ++dx) {
+            const double s = X(dx, dy, dz, e);
+            for (int qx = 0; qx < Q1D; ++qx) {
+              sol_x[qx] += B(qx, dx) * s;
+            }
+          }
+
+          for (int qy = 0; qy < Q1D; ++qy) {
+            const double wy = B(qy, dy);
+            for (int qx = 0; qx < Q1D; ++qx) {
+              sol_xy[qy][qx] += wy * sol_x[qx];
+            }
+          }
+        }
+
+        for (int qz = 0; qz < Q1D; ++qz) {
+          const double wz = B(qz, dz);
+          for (int qy = 0; qy < Q1D; ++qy) {
+            for (int qx = 0; qx < Q1D; ++qx) {
+              sol_xyz[qz][qy][qx] += wz * sol_xy[qy][qx];
+            }
+          }
+        }
+      }
+
+      for (int qz = 0; qz < Q1D; ++qz) {
+        for (int qy = 0; qy < Q1D; ++qy) {
+          for (int qx = 0; qx < Q1D; ++qx) {
+            sol_xyz[qz][qy][qx] *= op(qx, qy, qz, e);
+          }
+        }
+      }
+
+      for (int qz = 0; qz < Q1D; ++qz) {
+        double sol_xy[D1D][D1D];
+        for (int dy = 0; dy < D1D; ++dy) {
+          for (int dx = 0; dx < D1D; ++dx) {
+            sol_xy[dy][dx] = 0;
+          }
+        }
+
+        for (int qy = 0; qy < Q1D; ++qy) {
+          double sol_x[D1D];
+          for (int dx = 0; dx < D1D; ++dx) {
+            sol_x[dx] = 0;
+          }
+
+          for (int qx = 0; qx < Q1D; ++qx) {
+            const double s = sol_xyz[qz][qy][qx];
+            for (int dx = 0; dx < D1D; ++dx) {
+              sol_x[dx] += Bt(dx, qx) * s;
+            }
+          }
+
+          for (int dy = 0; dy < D1D; ++dy) {
+            const double wy = Bt(dy, qy);
+            for (int dx = 0; dx < D1D; ++dx) {
+              sol_xy[dy][dx] += wy * sol_x[dx];
+            }
+          }
+        }
+
+        for (int dz = 0; dz < D1D; ++dz) {
+          const double wz = Bt(dz, qz);
+          for (int dy = 0; dy < D1D; ++dy) {
+            for (int dx = 0; dx < D1D; ++dx) {
+              Y(dx, dy, dz, e) += wz * sol_xy[dy][dx];
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+@kernel void MassApply2D_GPU(const int NE,
+                             @restrict const DofToQuad_t B,
+                             @restrict const QuadToDof_t Bt,
+                             @restrict const QLocal2D_t op,
+                             @restrict const DLocal2D_t X,
+                             @restrict DLocal2D_t Y) {
+  // Iterate over elements
+  for (int eOff = 0; eOff < NE; eOff += M2_ELEMENT_BATCH; @outer) {
+    // Store dof <--> quad mappings
+    @shared double s_B[DQ1D] @dim(Q1D, D1D);
+    @shared double s_Bt[DQ1D] @dim(D1D, Q1D);
+
+    // Store xy planes in @shared memory
+    @shared double s_xy[DQ1D] @dim(D1D, Q1D);
+    @shared double s_xy2[Q2D] @dim(Q1D, Q1D);
+
+    @exclusive double r_x[M1D];
+
+    for (int x = 0; x < M1D; ++x; @inner) {
+      for (int id = x; id < DQ1D; id += M1D) {
+        s_B[id]  = B[id];
+        s_Bt[id]  = Bt[id];
+      }
+    }
+
+    for (int e = eOff; e < (eOff + M2_ELEMENT_BATCH); ++e) {
+      if (e < NE) {
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if (dx < D1D) {
+            for (int qy = 0; qy < Q1D; ++qy) {
+              s_xy(dx, qy) = 0;
+            }
+            for (int dy = 0; dy < D1D; ++dy) {
+              r_x[dy] = X(dx, dy, e);
+            }
+            for (int qy = 0; qy < Q1D; ++qy) {
+              double xy = 0;
+              for (int dy = 0; dy < D1D; ++dy) {
+                xy += r_x[dy] * s_B(qy, dy);
+              }
+              s_xy(dx, qy) = xy;
+            }
+          }
+        }
+        for (int qy = 0; qy < M1D; ++qy; @inner) {
+          if (qy < Q1D) {
+            for (int qx = 0; qx < Q1D; ++qx) {
+              double s = 0;
+              for (int dx = 0; dx < D1D; ++dx) {
+                s += s_xy(dx, qy) * s_B(qx, dx);
+              }
+              s_xy2(qx, qy) = s * op(qx, qy, e);
+            }
+          }
+        }
+
+        for (int qx = 0; qx < M1D; ++qx; @inner) {
+          if (qx < Q1D) {
+            for (int dy = 0; dy < D1D; ++dy) {
+              s_xy(dy, qx) = 0;
+            }
+            for (int qy = 0; qy < Q1D; ++qy) {
+              r_x[qy] = s_xy2(qx, qy);
+            }
+            for (int dy = 0; dy < D1D; ++dy) {
+              double s = 0;
+              for (int qy = 0; qy < Q1D; ++qy) {
+                s += r_x[qy] * s_Bt(dy, qy);
+              }
+              s_xy(dy, qx) = s;
+            }
+          }
+        }
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if (dx < D1D) {
+            for (int dy = 0; dy < D1D; ++dy) {
+              double s = 0;
+              for (int qx = 0; qx < Q1D; ++qx) {
+                s += (s_xy(dy, qx) * s_Bt(dx, qx));
+              }
+              Y(dx, dy, e) += s;
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+@kernel void MassApply3D_GPU(const int NE,
+                             @restrict const DofToQuad_t B,
+                             @restrict const QuadToDof_t Bt,
+                             @restrict const QLocal3D_t op,
+                             @restrict const DLocal3D_t X,
+                             @restrict DLocal3D_t Y) {
+  // Iterate over elements
+  for (int e = 0; e < NE; ++e; @outer) {
+    // Store dof <--> quad mappings
+    @shared double s_B[DQ1D] @dim(Q1D, D1D);
+    @shared double s_Bt[DQ1D] @dim(D1D, Q1D);
+
+    // Store xy planes in @shared memory
+    @shared double s_xy[M2D] @dim(M1D, M1D);
+
+    // Store z axis as registers
+    @exclusive double r_z[Q1D];
+    @exclusive double r_z2[D1D];
+
+    for (int y = 0; y < M1D; ++y; @inner) {
+      for (int x = 0; x < M1D; ++x; @inner) {
+        const int id = (y * M1D) + x;
+        // Fetch Q <--> D maps
+        if (id < DQ1D) {
+          s_B[id]  = B[id];
+          s_Bt[id]  = Bt[id];
+        }
+        // Initialize our Z axis
+        for (int qz = 0; qz < Q1D; ++qz) {
+          r_z[qz] = 0;
+        }
+        for (int dz = 0; dz < D1D; ++dz) {
+          r_z2[dz] = 0;
+        }
+      }
+    }
+
+    for (int dy = 0; dy < M1D; ++dy; @inner) {
+      for (int dx = 0; dx < M1D; ++dx; @inner) {
+        if ((dx < D1D) && (dy < D1D)) {
+          for (int dz = 0; dz < D1D; ++dz) {
+            const double s = X(dx, dy, dz, e);
+            // Calculate D -> Q in the Z axis
+            for (int qz = 0; qz < Q1D; ++qz) {
+              r_z[qz] += s * s_B(qz, dz);
+            }
+          }
+        }
+      }
+    }
+    // For each xy plane
+    for (int qz = 0; qz < Q1D; ++qz) {
+      // Fill xy plane at given z position
+      for (int dy = 0; dy < M1D; ++dy; @inner) {
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if ((dx < D1D) && (dy < D1D)) {
+            s_xy(dx, dy) = r_z[qz];
+          }
+        }
+      }
+      // Calculate Dxyz, xDyz, xyDz in plane
+      for (int qy = 0; qy < M1D; ++qy; @inner) {
+        for (int qx = 0; qx < M1D; ++qx; @inner) {
+          if ((qx < Q1D) && (qy < Q1D)) {
+            double s = 0;
+            for (int dy = 0; dy < D1D; ++dy) {
+              const double wy = s_B(qy, dy);
+              for (int dx = 0; dx < D1D; ++dx) {
+                const double wx = s_B(qx, dx);
+                s += wx * wy * s_xy(dx, dy);
+              }
+            }
+
+            s *= op(qx, qy, qz, e);
+
+            for (int dz = 0; dz < D1D; ++dz) {
+              const double wz  = s_Bt(dz, qz);
+              r_z2[dz] += wz * s;
+            }
+          }
+        }
+      }
+      @barrier("s_xy_sync_1");
+    }
+    // Iterate over xy planes to compute solution
+    for (int dz = 0; dz < D1D; ++dz) {
+      // Place xy plane in @shared memory
+      for (int qy = 0; qy < M1D; ++qy; @inner) {
+        for (int qx = 0; qx < M1D; ++qx; @inner) {
+          if ((qx < Q1D) && (qy < Q1D)) {
+            s_xy(qx, qy) = r_z2[dz];
+          }
+        }
+      }
+      // Finalize solution in xy plane
+      for (int dy = 0; dy < M1D; ++dy; @inner) {
+        for (int dx = 0; dx < M1D; ++dx; @inner) {
+          if ((dx < D1D) && (dy < D1D)) {
+            double solZ = 0;
+            for (int qy = 0; qy < Q1D; ++qy) {
+              const double wy = s_Bt(dy, qy);
+              for (int qx = 0; qx < Q1D; ++qx) {
+                const double wx = s_Bt(dx, qx);
+                solZ += wx * wy * s_xy(qx, qy);
+              }
+            }
+            Y(dx, dy, dz, e) += solZ;
+          }
+        }
+      }
+      @barrier("s_xy_sync_2");
+    }
+  }
+}
@@ -284,6 +284,12 @@ void ParBilinearForm::FormLinearSystem(
   const Array<int> &ess_tdof_list, Vector &x, Vector &b,
   OperatorHandle &A, Vector &X, Vector &B, int copy_interior)
 {
+   if (ext)
+   {
+      ext->FormLinearSystem(ess_tdof_list, x, b, A, X, B, copy_interior);
+      return;
+   }
+
   // Finish the matrix assembly and perform BC elimination, storing the
   // eliminated part of the matrix.
   FormSystemMatrix(ess_tdof_list, A);
@@ -327,6 +333,12 @@ void ParBilinearForm::FormLinearSystem(
 void ParBilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
                                       OperatorHandle &A)
 {
+   if (ext)
+   {
+      ext->FormSystemMatrix(ess_tdof_list, A);
+      return;
+   }
+
   // Finish the matrix assembly and perform BC elimination, storing the
   // eliminated part of the matrix.
   if (static_cond)
@@ -369,6 +381,12 @@ void ParBilinearForm::FormSystemMatrix(const Array<int> &ess_tdof_list,
 void ParBilinearForm::RecoverFEMSolution(
   const Vector &X, const Vector &b, Vector &x)
 {
+   if (ext)
+   {
+      ext->RecoverFEMSolution(X, b, x);
+      return;
+   }
+
   const Operator &P = *pfes->GetProlongationMatrix();

   if (static_cond)
@@ -467,8 +485,8 @@ void ParMixedBilinearForm::TrueAddMult(const Vector &x, Vector &y,

 HypreParMatrix* ParDiscreteLinearOperator::ParallelAssemble() const
 {
-   MFEM_ASSERT(mat, "matrix is not assembled");
-   MFEM_ASSERT(mat->Finalized(), "matrix is not finalized");
+   MFEM_ASSERT(mat, "Matrix is not assembled");
+   MFEM_ASSERT(mat->Finalized(), "Matrix is not finalized");
   SparseMatrix* RA = mfem::Mult(*range_fes->GetRestrictionMatrix(), *mat);
   HypreParMatrix* P = domain_fes->Dof_TrueDof_Matrix();
   HypreParMatrix* RAP = P->LeftDiagMult(*RA, range_fes->GetTrueDofOffsets());
@@ -479,7 +497,7 @@ HypreParMatrix* ParDiscreteLinearOperator::ParallelAssemble() const
 void ParDiscreteLinearOperator::GetParBlocks(Array2D<HypreParMatrix *> &blocks)
 const
 {
-   MFEM_VERIFY(mat->Finalized(), "local matrix needs to be finalized for "
+   MFEM_VERIFY(mat->Finalized(), "Local matrix needs to be finalized for "
               "GetParBlocks");

   HypreParMatrix* RLP = ParallelAssemble();
@@ -75,7 +75,8 @@ public:
       those rows. Must be called before the first Assemble call. */
   void KeepNbrBlock(bool knb = true) { keep_nbr_block = knb; }

-   /// Set the operator type id for the parallel matrix/operator.
+   /** @brief Set the operator type id for the parallel matrix/operator when
+       using AssemblyLevel::FULL. */
   /** If using static condensation or hybridization, call this method *after*
       enabling it. */
   void SetOperatorType(Operator::Type tid)
@@ -163,65 +164,15 @@ public:
   virtual const Operator *GetRestriction() const
   { return pfes->GetRestrictionMatrix(); }

-   /** Form the linear system A X = B, corresponding to the current bilinear
-       form and b(.), by applying any necessary transformations such as:
-       eliminating boundary conditions; applying conforming constraints for
-       non-conforming AMR; parallel assembly; static condensation;
-       hybridization.
+   using BilinearForm::FormLinearSystem;
+   using BilinearForm::FormSystemMatrix;

-       The ParGridFunction-size vector x must contain the essential b.c. The
-       ParBilinearForm and the ParLinearForm-size vector b must be assembled.
+   virtual void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x,
+                                 Vector &b, OperatorHandle &A, Vector &X,
+                                 Vector &B, int copy_interior = 0);

-       The vector X is initialized with a suitable initial guess: when using
-       hybridization, the vector X is set to zero; otherwise, the essential
-       entries of X are set to the corresponding b.c. and all other entries are
-       set to zero (copy_interior == 0) or copied from x (copy_interior != 0).
-
-       This method can be called multiple times (with the same ess_tdof_list
-       array) to initialize different right-hand sides and boundary condition
-       values.
-
-       After solving the linear system, the finite element solution x can be
-       recovered by calling RecoverFEMSolution (with the same vectors X, b, and
-       x). */
-   void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x, Vector &b,
-                         OperatorHandle &A, Vector &X, Vector &B,
-                         int copy_interior = 0);
-
-   /** Version of the method FormLinearSystem() where the system matrix is
-       returned in the variable @a A, of type OpType, holding a *reference* to
-       the system matrix (created with the method OpType::MakeRef()). The
-       reference will be invalidated when SetOperatorType(), Update(), or the
-       destructor is called. */
-   template <typename OpType>
-   void FormLinearSystem(const Array<int> &ess_tdof_list, Vector &x, Vector &b,
-                         OpType &A, Vector &X, Vector &B,
-                         int copy_interior = 0)
-   {
-      OperatorHandle Ah;
-      FormLinearSystem(ess_tdof_list, x, b, Ah, X, B, copy_interior);
-      OpType *A_ptr = Ah.Is<OpType>();
-      MFEM_VERIFY(A_ptr, "invalid OpType used");
-      A.MakeRef(*A_ptr);
-   }
-
-   /// Form the linear system matrix @a A, see FormLinearSystem() for details.
-   void FormSystemMatrix(const Array<int> &ess_tdof_list, OperatorHandle &A);
-
-   /** Version of the method FormSystemMatrix() where the system matrix is
-       returned in the variable @a A, of type OpType, holding a *reference* to
-       the system matrix (created with the method OpType::MakeRef()). The
-       reference will be invalidated when SetOperatorType(), Update(), or the
-       destructor is called. */
-   template <typename OpType>
-   void FormSystemMatrix(const Array<int> &ess_tdof_list, OpType &A)
-   {
-      OperatorHandle Ah;
-      FormSystemMatrix(ess_tdof_list, Ah);
-      OpType *A_ptr = Ah.Is<OpType>();
-      MFEM_VERIFY(A_ptr, "invalid OpType used");
-      A.MakeRef(*A_ptr);
-   }
+   virtual void FormSystemMatrix(const Array<int> &ess_tdof_list,
+                                 OperatorHandle &A);

   /** Call this method after solving a linear system constructed using the
       FormLinearSystem method to recover the solution as a ParGridFunction-size
@@ -613,15 +613,15 @@ void ParFiniteElementSpace::Build_Dof_TrueDof_Matrix() const // matrix P
   int ldof  = GetVSize();
   int ltdof = TrueVSize();

-   HYPRE_Int *i_diag = new HYPRE_Int[ldof+1];
-   HYPRE_Int *j_diag = new HYPRE_Int[ltdof];
+   HYPRE_Int *i_diag = mfem::New<HYPRE_Int>(ldof+1);
+   HYPRE_Int *j_diag = mfem::New<HYPRE_Int>(ltdof);
   int diag_counter;

-   HYPRE_Int *i_offd = new HYPRE_Int[ldof+1];
-   HYPRE_Int *j_offd = new HYPRE_Int[ldof-ltdof];
+   HYPRE_Int *i_offd = mfem::New<HYPRE_Int>(ldof+1);
+   HYPRE_Int *j_offd = mfem::New<HYPRE_Int>(ldof-ltdof);
   int offd_counter;

-   HYPRE_Int *cmap   = new HYPRE_Int[ldof-ltdof];
+   HYPRE_Int *cmap   = mfem::New<HYPRE_Int>(ldof-ltdof);

   HYPRE_Int *col_starts = GetTrueDofOffsets();
   HYPRE_Int *row_starts = GetDofOffsets();
@@ -2249,7 +2249,7 @@ HypreParMatrix* ParFiniteElementSpace
   }

   // create offd column mapping
-   HYPRE_Int *cmap = new HYPRE_Int[col_map.size()];
+   HYPRE_Int *cmap = mfem::New<HYPRE_Int>(col_map.size());
   int offd_col = 0;
   for (std::map<HYPRE_Int, int>::iterator
        it = col_map.begin(); it != col_map.end(); ++it)
@@ -2258,14 +2258,14 @@ HypreParMatrix* ParFiniteElementSpace
      it->second = offd_col++;
   }

-   HYPRE_Int *I_diag = new HYPRE_Int[vdim*local_rows + 1];
-   HYPRE_Int *I_offd = new HYPRE_Int[vdim*local_rows + 1];
+   HYPRE_Int *I_diag = mfem::New<HYPRE_Int>(vdim*local_rows + 1);
+   HYPRE_Int *I_offd = mfem::New<HYPRE_Int>(vdim*local_rows + 1);

-   HYPRE_Int *J_diag = new HYPRE_Int[nnz_diag];
-   HYPRE_Int *J_offd = new HYPRE_Int[nnz_offd];
+   HYPRE_Int *J_diag = mfem::New<HYPRE_Int>(nnz_diag);
+   HYPRE_Int *J_offd = mfem::New<HYPRE_Int>(nnz_offd);

-   double *A_diag = new double[nnz_diag];
-   double *A_offd = new double[nnz_offd];
+   double *A_diag = mfem::New<double>(nnz_diag);
+   double *A_offd = mfem::New<double>(nnz_offd);

   int vdim1 = bynodes ? vdim : 1;
   int vdim2 = bynodes ? 1 : vdim;
@@ -2316,7 +2316,7 @@ HypreParMatrix* ParFiniteElementSpace

 static HYPRE_Int* make_i_array(int nrows)
 {
-   HYPRE_Int *I = new HYPRE_Int[nrows+1];
+   HYPRE_Int *I = mfem::New<HYPRE_Int>(nrows+1);
   for (int i = 0; i <= nrows; i++) { I[i] = -1; }
   return I;
 }
@@ -2328,7 +2328,7 @@ static HYPRE_Int* make_j_array(HYPRE_Int* I, int nrows)
   {
      if (I[i] >= 0) { nnz++; }
   }
-   HYPRE_Int *J = new HYPRE_Int[nnz];
+   HYPRE_Int *J = mfem::New<HYPRE_Int>(nnz);

   I[nrows] = -1;
   for (int i = 0, k = 0; i <= nrows; i++)
@@ -2427,7 +2427,7 @@ ParFiniteElementSpace::RebalanceMatrix(int old_ndofs,
   }
   SortPairs<HYPRE_Int, int>(cmap_offd, offd_cols);

-   HYPRE_Int* cmap = new HYPRE_Int[offd_cols];
+   HYPRE_Int* cmap = mfem::New<HYPRE_Int>(offd_cols);
   for (int i = 0; i < offd_cols; i++)
   {
      cmap[i] = cmap_offd[i].one;
@@ -2623,7 +2623,7 @@ ParFiniteElementSpace::ParallelDerefinementMatrix(int old_ndofs,
   offd->SetWidth(col_map.size());

   // create offd column mapping for use by hypre
-   HYPRE_Int *cmap = new HYPRE_Int[offd->Width()];
+   HYPRE_Int *cmap = mfem::New<HYPRE_Int>(offd->Width());
   for (std::map<HYPRE_Int, int>::iterator
        it = col_map.begin(); it != col_map.end(); ++it)
   {
@@ -2865,6 +2865,7 @@ void ConformingProlongationOperator::Mult(const Vector &x, Vector &y) const

   const double *xdata = x.GetData();
   double *ydata = y.GetData();
+   x.Pull();
   const int m = external_ldofs.Size();

   const int in_layout = 2; // 2 - input is ltdofs array
@@ -2881,6 +2882,7 @@ void ConformingProlongationOperator::Mult(const Vector &x, Vector &y) const

   const int out_layout = 0; // 0 - output is ldofs array
   gc.BcastEnd(ydata, out_layout);
+   y.Push();
 }

 void ConformingProlongationOperator::MultTranspose(
@@ -2891,6 +2893,7 @@ void ConformingProlongationOperator::MultTranspose(

   const double *xdata = x.GetData();
   double *ydata = y.GetData();
+   x.Pull();
   const int m = external_ldofs.Size();

   gc.ReduceBegin(xdata);
@@ -2906,6 +2909,7 @@ void ConformingProlongationOperator::MultTranspose(

   const int out_layout = 2; // 2 - output is an array on all ltdofs
   gc.ReduceEnd<double>(ydata, out_layout, GroupCommunicator::Sum);
+   y.Push();
 }

 } // namespace mfem
@@ -11,10 +11,14 @@

 list(APPEND SRCS
  array.cpp
+  cuda.cpp
+  device.cpp
  error.cpp
  globals.cpp
  gzstream.cpp
  isockstream.cpp
+  mem_manager.cpp
+  occa.cpp
  optparser.cpp
  osockstream.cpp
  sets.cpp
@@ -27,12 +31,18 @@ list(APPEND SRCS

 list(APPEND HDRS
  array.hpp
+  binaryio.hpp
+  cuda.hpp
+  device.hpp
  error.hpp
  globals.hpp
  gzstream.hpp
  hash.hpp
  isockstream.hpp
  mem_alloc.hpp
+  mem_manager.hpp
+  occa.hpp
+  forall.hpp
  optparser.hpp
  osockstream.hpp
  sets.hpp
@@ -13,6 +13,7 @@
 // Abstract array data type

 #include "array.hpp"
+#include "../general/forall.hpp"
 #include <fstream>

 namespace mfem
@@ -22,7 +23,7 @@ BaseArray::BaseArray(int asize, int ainc, int elementsize)
 {
   if (asize > 0)
   {
-      data = new char[asize * elementsize];
+      data = mfem::New<char>(asize * elementsize);
      size = allocsize = asize;
   }
   else
@@ -37,7 +38,7 @@ BaseArray::~BaseArray()
 {
   if (allocsize > 0)
   {
-      delete [] (char*)data;
+      mfem::Delete((char*)data);
   }
 }

@@ -47,14 +48,14 @@ void BaseArray::GrowSize(int minsize, int elementsize)
   int nsize = (inc > 0) ? abs(allocsize) + inc : 2 * abs(allocsize);
   if (nsize < minsize) { nsize = minsize; }

-   p = new char[nsize * elementsize];
+   p = mfem::New<char>(nsize * elementsize);
   if (size > 0)
   {
-      memcpy(p, data, size * elementsize);
+      mfem::Memcpy(p, data, size * elementsize);
   }
   if (allocsize > 0)
   {
-      delete [] (char*)data;
+      mfem::Delete((char*)data);
   }
   data = p;
   allocsize = nsize;
@@ -13,6 +13,7 @@
 #define MFEM_ARRAY

 #include "../config/config.hpp"
+#include "mem_manager.hpp"
 #include "error.hpp"
 #include "globals.hpp"

@@ -82,7 +83,7 @@ public:
   /// Copy constructor: deep copy
   Array(const Array<T> &src)
      : BaseArray(src.size, 0, sizeof(T))
-   { std::memcpy(data, src.data, size*sizeof(T)); }
+   { mfem::Memcpy(data, src.data, size*sizeof(T)); }

   /// Copy constructor (deep copy) from an Array of convertable type
   template <typename CT>
@@ -190,7 +191,7 @@ public:
   inline void Copy(Array &copy) const
   {
      copy.SetSize(Size());
-      std::memcpy(copy.GetData(), data, Size()*sizeof(T));
+      mfem::Memcpy(copy.GetData(), data, Size()*sizeof(T));
   }

   /// Make this Array a reference to a pointer
@@ -713,7 +714,7 @@ inline void Array<T>::DeleteAll()
 {
   if (allocsize > 0)
   {
-      delete [] (char*)data;
+      mfem::Delete((char*)data);
   }
   data = NULL;
   size = allocsize = 0;
@@ -724,7 +725,7 @@ inline void Array<T>::MakeRef(T *p, int s)
 {
   if (allocsize > 0)
   {
-      delete [] (char*)data;
+      mfem::Delete((char*)data);
   }
   data = p;
   size = s;
@@ -736,7 +737,7 @@ inline void Array<T>::MakeRef(const Array &master)
 {
   if (allocsize > 0)
   {
-      delete [] (char*)data;
+      mfem::Delete((char*)data);
   }
   data = master.data;
   size = master.size;
@@ -0,0 +1,94 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "cuda.hpp"
+#include "globals.hpp"
+
+namespace mfem
+{
+
+#ifdef MFEM_USE_CUDA
+void mfem_cuda_error(cudaError_t err, const char *expr, const char *func,
+                     const char *file, int line)
+{
+   mfem::err << "CUDA error: (" << expr << ") failed with error:\n --> "
+             << cudaGetErrorString(err)
+             << "\n ... in function: " << func
+             << "\n ... in file: " << file << ':' << line << '\n';
+   mfem_error();
+}
+#endif
+
+void* CuMemAlloc(void** dptr, size_t bytes)
+{
+#ifdef MFEM_USE_CUDA
+   MFEM_CUDA_CHECK(cudaMalloc(dptr, bytes));
+#endif
+   return *dptr;
+}
+
+void* CuMemFree(void *dptr)
+{
+#ifdef MFEM_USE_CUDA
+   MFEM_CUDA_CHECK(cudaFree(dptr));
+#endif
+   return dptr;
+}
+
+void* CuMemcpyHtoD(void* dst, const void* src, size_t bytes)
+{
+#ifdef MFEM_USE_CUDA
+   MFEM_CUDA_CHECK(cudaMemcpy(dst, src, bytes, cudaMemcpyHostToDevice));
+#endif
+   return dst;
+}
+
+void* CuMemcpyHtoDAsync(void* dst, const void* src, size_t bytes)
+{
+#ifdef MFEM_USE_CUDA
+   MFEM_CUDA_CHECK(cudaMemcpyAsync(dst, src, bytes, cudaMemcpyHostToDevice));
+#endif
+   return dst;
+}
+
+void* CuMemcpyDtoD(void* dst, void* src, size_t bytes)
+{
+#ifdef MFEM_USE_CUDA
+   MFEM_CUDA_CHECK(cudaMemcpy(dst, src, bytes, cudaMemcpyDeviceToDevice));
+#endif
+   return dst;
+}
+
+void* CuMemcpyDtoDAsync(void* dst, void* src, size_t bytes)
+{
+#ifdef MFEM_USE_CUDA
+   MFEM_CUDA_CHECK(cudaMemcpyAsync(dst, src, bytes, cudaMemcpyDeviceToDevice));
+#endif
+   return dst;
+}
+
+void* CuMemcpyDtoH(void *dst, void *src, size_t bytes)
+{
+#ifdef MFEM_USE_CUDA
+   MFEM_CUDA_CHECK(cudaMemcpy(dst, src, bytes, cudaMemcpyDeviceToHost));
+#endif
+   return dst;
+}
+
+void* CuMemcpyDtoHAsync(void* dst, void* src, size_t bytes, void *s)
+{
+#ifdef MFEM_USE_CUDA
+   MFEM_CUDA_CHECK(cudaMemcpyAsync(dst, src, bytes, cudaMemcpyDeviceToHost));
+#endif
+   return dst;
+}
+
+} // namespace mfem
@@ -0,0 +1,83 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_CUDA_HPP
+#define MFEM_CUDA_HPP
+
+#include "../config/config.hpp"
+#include "error.hpp"
+
+#ifdef MFEM_USE_CUDA
+#include <cuda_runtime.h>
+#include <cuda.h>
+#endif
+
+// CUDA block size used by MFEM.
+#define MFEM_CUDA_BLOCKS 256
+
+#ifdef MFEM_USE_CUDA
+#define MFEM_ATTR_DEVICE __device__
+#define MFEM_ATTR_HOST_DEVICE __host__ __device__
+// Define a CUDA error check macro, MFEM_CUDA_CHECK(x), where x returns/is of
+// type 'cudaError_t'. This macro evaluates 'x' and raises an error if the
+// result is not cudaSuccess.
+#define MFEM_CUDA_CHECK(x) \
+   do \
+   { \
+      cudaError_t err = (x); \
+      if (err != cudaSuccess) \
+      { \
+         mfem_cuda_error(err, #x, _MFEM_FUNC_NAME, __FILE__, __LINE__); \
+      } \
+   } \
+   while (0)
+#else // MFEM_USE_CUDA
+#define MFEM_ATTR_DEVICE
+#define MFEM_ATTR_HOST_DEVICE
+#endif // MFEM_USE_CUDA
+
+
+namespace mfem
+{
+
+#ifdef MFEM_USE_CUDA
+// Function used by the macro MFEM_CUDA_CHECK.
+void mfem_cuda_error(cudaError_t err, const char *expr, const char *func,
+                     const char *file, int line);
+#endif
+
+/// Allocates device memory
+void* CuMemAlloc(void **d_ptr, size_t bytes);
+
+/// Frees device memory
+void* CuMemFree(void *d_ptr);
+
+/// Copies memory from Host to Device
+void* CuMemcpyHtoD(void *d_dst, const void *h_src, size_t bytes);
+
+/// Copies memory from Host to Device
+void* CuMemcpyHtoDAsync(void *d_dst, const void *h_src, size_t bytes);
+
+/// Copies memory from Device to Device
+void* CuMemcpyDtoD(void *d_dst, void *d_src, size_t bytes);
+
+/// Copies memory from Device to Device
+void* CuMemcpyDtoDAsync(void *d_dst, void *d_src, size_t bytes);
+
+/// Copies memory from Device to Host
+void* CuMemcpyDtoH(void *h_dst, void *d_src, size_t bytes);
+
+/// Copies memory from Device to Host
+void* CuMemcpyDtoHAsync(void *h_dst, void *d_src, size_t bytes);
+
+} // namespace mfem
+
+#endif // MFEM_CUDA_HPP
@@ -0,0 +1,199 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "forall.hpp"
+#include "cuda.hpp"
+#include "occa.hpp"
+
+#include <string>
+#include <map>
+
+namespace mfem
+{
+
+// Place the following variables in the mfem::internal namespace, so that they
+// will not be included in the doxygen documentation.
+namespace internal
+{
+
+OccaDevice occaDevice;
+
+// Backends listed by priority, high to low:
+static const Backend::Id backend_list[Backend::NUM_BACKENDS] =
+{
+   Backend::OCCA_CUDA, Backend::RAJA_CUDA, Backend::CUDA,
+   Backend::OCCA_OMP, Backend::RAJA_OMP, Backend::OMP,
+   Backend::OCCA_CPU, Backend::RAJA_CPU, Backend::CPU
+};
+
+// Backend names listed by priority, high to low:
+static const char *backend_name[Backend::NUM_BACKENDS] =
+{
+   "occa-cuda", "raja-cuda", "cuda", "occa-omp", "raja-omp", "omp",
+   "occa-cpu", "raja-cpu", "cpu"
+};
+
+} // namespace mfem::internal
+
+void Device::Configure(const std::string &device, const int dev)
+{
+   std::map<std::string, Backend::Id> bmap;
+   for (int i = 0; i < Backend::NUM_BACKENDS; i++)
+   {
+      bmap[internal::backend_name[i]] = internal::backend_list[i];
+   }
+   std::string::size_type beg = 0, end;
+   while (1)
+   {
+      end = device.find(',', beg);
+      end = (end != std::string::npos) ? end : device.size();
+      const std::string bname = device.substr(beg, end - beg);
+      std::map<std::string, Backend::Id>::iterator it = bmap.find(bname);
+      MFEM_VERIFY(it != bmap.end(), "invalid backend name: '" << bname << '\'');
+      Get().MarkBackend(it->second);
+      if (end == device.size()) { break; }
+      beg = end + 1;
+   }
+
+   // OCCA_CUDA needs CUDA or RAJA_CUDA:
+   Get().allowed_backends = Get().backends;
+   if (Allows(Backend::OCCA_CUDA) && !Allows(Backend::RAJA_CUDA))
+   {
+      Get().MarkBackend(Backend::CUDA);
+   }
+
+   // Activate all backends for Setup().
+   Get().allowed_backends = Get().backends;
+   Get().Setup(dev);
+
+   // Enable only the default host CPU backend.
+   Get().allowed_backends = Backend::CPU;
+}
+
+void Device::Print(std::ostream &out)
+{
+   out << "Device configuration: ";
+   bool add_comma = false;
+   for (int i = 0; i < Backend::NUM_BACKENDS; i++)
+   {
+      if (Get().backends & internal::backend_list[i])
+      {
+         if (add_comma) { out << ','; }
+         add_comma = true;
+         out << internal::backend_name[i];
+      }
+   }
+   out << '\n';
+}
+
+#ifdef MFEM_USE_CUDA
+static void DeviceSetup(const int dev, int &ngpu)
+{
+   MFEM_CUDA_CHECK(cudaGetDeviceCount(&ngpu));
+   MFEM_VERIFY(ngpu > 0, "No CUDA device found!");
+   MFEM_CUDA_CHECK(cudaSetDevice(dev));
+}
+#endif
+
+static void CudaDeviceSetup(const int dev, int &ngpu)
+{
+#ifdef MFEM_USE_CUDA
+   DeviceSetup(dev, ngpu);
+#endif
+}
+
+static void RajaDeviceSetup(const int dev, int &ngpu)
+{
+#ifdef MFEM_USE_CUDA
+   if (ngpu <= 0) { DeviceSetup(dev, ngpu); }
+#endif
+}
+
+static void OccaDeviceSetup(const int dev)
+{
+#ifdef MFEM_USE_OCCA
+   const int cpu  = Device::Allows(Backend::OCCA_CPU);
+   const int omp  = Device::Allows(Backend::OCCA_OMP);
+   const int cuda = Device::Allows(Backend::OCCA_CUDA);
+   if (cpu + omp + cuda > 1)
+   {
+      MFEM_ABORT("Only one OCCA backend can be configured at a time!");
+   }
+   if (cuda)
+   {
+#if OCCA_CUDA_ENABLED
+      std::string mode("mode: 'CUDA', device_id : ");
+      internal::occaDevice.setup(mode.append(1,'0'+dev));
+#else
+      MFEM_ABORT("the OCCA CUDA backend requires OCCA built with CUDA!");
+#endif
+   }
+   else if (omp)
+   {
+#if OCCA_OPENMP_ENABLED
+      internal::occaDevice.setup("mode: 'OpenMP'");
+#else
+      MFEM_ABORT("the OCCA OpenMP backend requires OCCA built with OpenMP!");
+#endif
+   }
+   else
+   {
+      internal::occaDevice.setup("mode: 'Serial'");
+   }
+
+   std::string mfemDir;
+   if (occa::io::exists(MFEM_INSTALL_DIR "/include/mfem/"))
+   {
+      mfemDir = MFEM_INSTALL_DIR "/include/mfem/";
+   }
+   else if (occa::io::exists(MFEM_SOURCE_DIR))
+   {
+      mfemDir = MFEM_SOURCE_DIR;
+   }
+   else
+   {
+      MFEM_ABORT("Cannot find OCCA kernels in MFEM_INSTALL_DIR or MFEM_SOURCE_DIR");
+   }
+
+   occa::io::addLibraryPath("mfem", mfemDir);
+   occa::loadKernels("mfem");
+#else
+   MFEM_ABORT("the OCCA backends require MFEM built with MFEM_USE_OCCA=YES");
+#endif
+}
+
+void Device::Setup(const int device)
+{
+   MFEM_VERIFY(ngpu == -1, "the mfem::Device is already configured!");
+
+   ngpu = 0;
+   dev = device;
+
+#ifndef MFEM_USE_CUDA
+   MFEM_VERIFY(!Allows(Backend::CUDA_MASK),
+               "the CUDA backends require MFEM built with MFEM_USE_CUDA=YES");
+#endif
+#ifndef MFEM_USE_RAJA
+   MFEM_VERIFY(!Allows(Backend::RAJA_MASK),
+               "the RAJA backends require MFEM built with MFEM_USE_RAJA=YES");
+#endif
+#ifndef MFEM_USE_OPENMP
+   MFEM_VERIFY(!Allows(Backend::OMP|Backend::RAJA_OMP),
+               "the OpenMP and RAJA OpenMP backends require MFEM built with"
+               " MFEM_USE_OPENMP=YES");
+#endif
+   if (Allows(Backend::CUDA)) { CudaDeviceSetup(dev, ngpu); }
+   if (Allows(Backend::RAJA_CUDA)) { RajaDeviceSetup(dev, ngpu); }
+   // The check for MFEM_USE_OCCA is in the function OccaDeviceSetup().
+   if (Allows(Backend::OCCA_MASK)) { OccaDeviceSetup(dev); }
+}
+
+} // mfem
@@ -0,0 +1,188 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_DEVICE_HPP
+#define MFEM_DEVICE_HPP
+
+#include "globals.hpp"
+
+namespace mfem
+{
+
+/// MFEM backends.
+/** Individual backends will generally implement only a subset of the kernels
+    implemented by the default CPU backend. The goal of the backends is to
+    accelerate data-parallel portions of the code and they can use a device
+    memory space (e.g. GPUs) or share the memory space of the host (OpenMP). */
+struct Backend
+{
+   /** @brief In the documentation below, we use square brackets to indicate the
+       type of the backend: host or device. */
+   enum Id
+   {
+      /// [host] Default CPU backend: sequential execution on each MPI rank.
+      CPU = 1 << 0,
+      /// [host] OpenMP backend. Enabled when MFEM_USE_OPENMP = YES.
+      OMP = 1 << 1,
+      /// [device] CUDA backend. Enabled when MFEM_USE_CUDA = YES.
+      CUDA = 1 << 2,
+      /** @brief [host] RAJA CPU backend: sequential execution on each MPI rank.
+          Enabled when MFEM_USE_RAJA = YES. */
+      RAJA_CPU = 1 << 3,
+      /** @brief [host] RAJA OpenMP backend. Enabled when MFEM_USE_RAJA = YES
+          and MFEM_USE_OPENMP = YES. */
+      RAJA_OMP = 1 << 4,
+      /** @brief [device] RAJA CUDA backend. Enabled when MFEM_USE_RAJA = YES
+          and MFEM_USE_CUDA = YES. */
+      RAJA_CUDA = 1 << 5,
+      /** @brief [host] OCCA CPU backend: sequential execution on each MPI rank.
+          Enabled when MFEM_USE_OCCA = YES. */
+      OCCA_CPU = 1 << 6,
+      /// [host] OCCA OpenMP backend. Enabled when MFEM_USE_OCCA = YES.
+      OCCA_OMP = 1 << 7,
+      /** @brief [device] OCCA CUDA backend. Enabled when MFEM_USE_OCCA = YES
+          and MFEM_USE_CUDA = YES. */
+      OCCA_CUDA = 1 << 8
+   };
+
+   /** @brief Additional useful constants. For example, the *_MASK constants can
+       be used with Device::Allows(). */
+   enum
+   {
+      /// Number of backends: from (1 << 0) to (1 << (NUM_BACKENDS-1)).
+      NUM_BACKENDS = 9,
+      /// Biwise-OR of all CUDA backends
+      CUDA_MASK = CUDA | RAJA_CUDA | OCCA_CUDA,
+      /// Biwise-OR of all RAJA backends
+      RAJA_MASK = RAJA_CPU | RAJA_OMP | RAJA_CUDA,
+      /// Biwise-OR of all OCCA backends
+      OCCA_MASK = OCCA_CPU | OCCA_OMP | OCCA_CUDA,
+      /// Biwise-OR of all OpenMP backends
+      OMP_MASK = OMP | RAJA_OMP | OCCA_OMP,
+      /// Biwise-OR of all device backends
+      DEVICE_MASK = CUDA_MASK
+   };
+};
+
+
+/** @brief The MFEM Device class abstracts hardware devices, such as GPUs, as
+    well as programming models, such as CUDA, OCCA, RAJA and OpenMP. */
+/** This class represents a "virtual device" with the following properties:
+    - There a single object of this class which is controlled by its static
+      methods.
+    - Once configured, the object cannot be re-configured during the program
+      lifetime.
+    - MFEM classes use this object to determine where (host or device) to
+      perform an operation and which backend implementation to use.
+    - Multiple backends can be configured at the same time; currently, a fixed
+      priority order is used to select a specific backend from the list of
+      configured backends. See the Backend class and the Configure() method in
+      this class for details.
+    - The device can be disabled to restrict the backend selection to only the
+      default host CPU backend, see the methods Enable() and Disable(). */
+class Device
+{
+private:
+   enum MODES {SEQUENTIAL, ACCELERATED};
+
+   MODES mode;
+   int dev = 0; ///< Device ID of the configured device.
+   int ngpu = -1; ///< Number of detected devices; -1: not initialized.
+   unsigned long backends; ///< Bitwise-OR of all configured backends.
+   /** Bitwise-OR mask of all allowed backends. All backends are active when the
+       Device is enabled. When the Device is disabled, only the host CPU backend
+       is allowed. */
+   unsigned long allowed_backends;
+
+   Device()
+      : mode(Device::SEQUENTIAL),
+        backends(Backend::CPU),
+        allowed_backends(backends) { }
+   Device(Device const&);
+   void operator=(Device const&);
+   static Device& Get() { static Device singleton; return singleton; }
+
+   /// Setup switcher based on configuration settings
+   void Setup(const int dev = 0);
+
+   void MarkBackend(Backend::Id b) { backends |= b; }
+
+public:
+   /// Configure the Device backends.
+   /** The string parameter @a device must be a comma-separated list of backend
+       string names (see below). The @a dev argument specifies the ID of the
+       actual devices (e.g. GPU) to use.
+       * The available backends are described by the Backend class.
+       * The string name of a backend is the lowercase version of the
+         Backend::Id enumeration constant with '_' replaced by '-', e.g. the
+         string name of 'RAJA_CPU' is 'raja-cpu'.
+       * The 'cpu' backend is always enabled with lowest priority.
+       * The current backend priority from highest to lowest is: 'occa-cuda',
+         'raja-cuda', 'cuda', 'occa-omp', 'raja-omp', 'omp', 'occa-cpu',
+         'raja-cpu', 'cpu'.
+       * Multiple backends can be configured at the same time.
+       * Only one 'occa-*' backend can be configured at a time.
+       * The backend 'occa-cuda' enables the 'cuda' backend unless 'raja-cuda'
+         is already enabled.
+       * After this call, the Device will be disabled. */
+   static void Configure(const std::string &device, const int dev = 0);
+
+   /// Print the configuration of the MFEM virtual device object.
+   static void Print(std::ostream &out = mfem::out);
+
+   /// Return true if Configure() has been called previously.
+   static inline bool IsConfigured() { return Get().ngpu >= 0; }
+
+   /// Return true if an actual device (e.g. GPU) has been configured.
+   static inline bool IsAvailable() { return Get().ngpu > 0; }
+
+   /// Enable the use of the configured device in the code that follows.
+   /** After this call MFEM classes will use the backend kernels whenever
+       possible, transferring data automatically to the device, if necessary.
+
+       If the only configured backend is the default host CPU one, the device
+       will remain disabled. */
+   static inline void Enable()
+   {
+      if (Get().backends & ~Backend::CPU)
+      {
+         Get().mode = Device::ACCELERATED;
+         Get().allowed_backends = Get().backends;
+      }
+   }
+
+   /// Disable the use of the configured device in the code that follows.
+   /** After this call MFEM classes will only use default CPU kernels,
+       transferring data automatically from the device, if necessary. */
+   static inline void Disable()
+   {
+      Get().mode = Device::SEQUENTIAL;
+      Get().allowed_backends = Backend::CPU;
+   }
+
+   /// Return true if the Device is enabled.
+   static inline bool IsEnabled() { return Get().mode == ACCELERATED; }
+
+   /// The opposite of IsEnabled().
+   static inline bool IsDisabled() { return !IsEnabled(); }
+
+   /** @brief Return true if any of the backends in the backend mask, @a b_mask,
+       are allowed. The allowed backends are all configured backends minus the
+       device backends when the Device is disabled. */
+   /** This method can be used with any of the Backend::Id constants, the
+       Backend::*_MASK, or combinations of those. */
+   static inline bool Allows(unsigned long b_mask)
+   { return Get().allowed_backends & b_mask; }
+};
+
+} // mfem
+
+#endif // MFEM_DEVICE_HPP
@@ -0,0 +1,144 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_FORALL_HPP
+#define MFEM_FORALL_HPP
+
+#include "../config/config.hpp"
+#include "error.hpp"
+#include "cuda.hpp"
+#include "occa.hpp"
+#include "device.hpp"
+#include "mem_manager.hpp"
+#include "../linalg/dtensor.hpp"
+
+#ifdef MFEM_USE_RAJA
+#include "RAJA/RAJA.hpp"
+#if defined(RAJA_ENABLE_CUDA) && !defined(MFEM_USE_CUDA)
+#error When RAJA is built with CUDA, MFEM_USE_CUDA=YES is required
+#endif
+#endif
+
+namespace mfem
+{
+
+// Implementation of MFEM's "parallel for" (forall) device/host kernel
+// interfaces supporting RAJA, CUDA, OpenMP, and sequential backends.
+
+// The MFEM_FORALL wrapper
+#define MFEM_FORALL(i,N,...)                                     \
+   ForallWrap(N,                                                 \
+              [=] MFEM_ATTR_DEVICE (int i) {__VA_ARGS__},        \
+              [&]                  (int i) {__VA_ARGS__})
+
+
+/// OpenMP backend
+template <typename HBODY>
+void OmpWrap(const int N, HBODY &&h_body)
+{
+#ifdef MFEM_USE_OPENMP
+   #pragma omp parallel for
+   for (int k = 0; k < N; k++)
+   {
+      h_body(k);
+   }
+#else
+   MFEM_ABORT("OpenMP requested for MFEM but OpenMP is not enabled!");
+#endif
+}
+
+
+/// RAJA Cuda backend
+template <int BLOCKS, typename DBODY>
+void RajaCudaWrap(const int N, DBODY &&d_body)
+{
+#if defined(MFEM_USE_RAJA) && defined(RAJA_ENABLE_CUDA)
+   RAJA::forall<RAJA::cuda_exec<BLOCKS>>(RAJA::RangeSegment(0,N),d_body);
+#else
+   MFEM_ABORT("RAJA::Cuda requested but RAJA::Cuda is not enabled!");
+#endif
+}
+
+
+/// RAJA OpenMP backend
+template <typename HBODY>
+void RajaOmpWrap(const int N, HBODY &&h_body)
+{
+#if defined(MFEM_USE_RAJA) && defined(RAJA_ENABLE_OPENMP)
+   RAJA::forall<RAJA::omp_parallel_for_exec>(RAJA::RangeSegment(0,N), h_body);
+#else
+   MFEM_ABORT("RAJA::OpenMP requested but RAJA::OpenMP is not enabled!");
+#endif
+}
+
+
+/// RAJA sequential loop backend
+template <typename HBODY>
+void RajaSeqWrap(const int N, HBODY &&h_body)
+{
+#ifdef MFEM_USE_RAJA
+   RAJA::forall<RAJA::loop_exec>(RAJA::RangeSegment(0,N), h_body);
+#else
+   MFEM_ABORT("RAJA requested but RAJA is not enabled!");
+#endif
+}
+
+
+/// CUDA backend
+#ifdef MFEM_USE_CUDA
+
+template <typename BODY> __global__ static
+void CuKernel(const int N, BODY body)
+{
+   const int k = blockDim.x*blockIdx.x + threadIdx.x;
+   if (k >= N) { return; }
+   body(k);
+}
+
+template <int BLOCKS, typename DBODY>
+void CuWrap(const int N, DBODY &&d_body)
+{
+   if (N==0) { return; }
+   const int GRID = (N+BLOCKS-1)/BLOCKS;
+   CuKernel<<<GRID,BLOCKS>>>(N,d_body);
+   MFEM_CUDA_CHECK(cudaGetLastError());
+}
+
+#else  // MFEM_USE_CUDA
+
+template <int BLOCKS, typename DBODY>
+void CuWrap(const int N, DBODY &&d_body) {}
+
+#endif
+
+
+/// The forall kernel body wrapper
+template <typename DBODY, typename HBODY>
+void ForallWrap(const int N, DBODY &&d_body, HBODY &&h_body)
+{
+   if (Device::Allows(Backend::RAJA_CUDA))
+   { return RajaCudaWrap<MFEM_CUDA_BLOCKS>(N, d_body); }
+
+   if (Device::Allows(Backend::CUDA))
+   { return CuWrap<MFEM_CUDA_BLOCKS>(N, d_body); }
+
+   if (Device::Allows(Backend::RAJA_OMP)) { return RajaOmpWrap(N, h_body); }
+
+   if (Device::Allows(Backend::OMP)) { return OmpWrap(N, h_body); }
+
+   if (Device::Allows(Backend::RAJA_CPU)) { return RajaSeqWrap(N, h_body); }
+
+   for (int k = 0; k < N; k++) { h_body(k); }
+}
+
+} // namespace mfem
+
+#endif // MFEM_FORALL_HPP
@@ -0,0 +1,363 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "../general/forall.hpp"
+
+#include <cstring> // std::memcpy
+
+#include <list>
+#include <unordered_map>
+
+namespace mfem
+{
+
+namespace internal
+{
+
+/// Forward declaration of the Alias structure
+struct Alias;
+
+/// Memory class that holds:
+///   - a boolean telling which memory space is being used
+///   - the size in bytes of this memory region,
+///   - the host and the device pointer,
+///   - a list of all aliases seen using this region (used only to free them).
+struct Memory
+{
+   bool host;
+   const std::size_t bytes;
+   void *const h_ptr;
+   void *d_ptr;
+   std::list<const void*> aliases;
+   Memory(void* const h, const std::size_t size):
+      host(true), bytes(size), h_ptr(h), d_ptr(nullptr), aliases() {}
+};
+
+/// Alias class that holds the base memory region and the offset
+struct Alias
+{
+   Memory *const mem;
+   const long offset;
+};
+
+typedef std::unordered_map<const void*, Memory> MemoryMap;
+typedef std::unordered_map<const void*, const Alias*> AliasMap;
+
+struct Ledger
+{
+   MemoryMap memories;
+   AliasMap aliases;
+};
+
+} // namespace mfem::internal
+
+static internal::Ledger *maps;
+
+MemoryManager::MemoryManager()
+{
+   exists = true;
+   enabled = true;
+   maps = new internal::Ledger();
+}
+
+MemoryManager::~MemoryManager()
+{
+   delete maps;
+   exists = false;
+}
+
+void* MemoryManager::Insert(void *ptr, const std::size_t bytes)
+{
+   if (!UsingMM()) { return ptr; }
+   const bool known = IsKnown(ptr);
+   if (known)
+   {
+      mfem_error("Trying to add an already present address!");
+   }
+   maps->memories.emplace(ptr, internal::Memory(ptr, bytes));
+   return ptr;
+}
+
+void *MemoryManager::Erase(void *ptr)
+{
+   if (!UsingMM()) { return ptr; }
+   if (!ptr) { return ptr; }
+   const bool known = IsKnown(ptr);
+   if (!known)
+   {
+      mfem_error("Trying to erase an unknown pointer!");
+   }
+   internal::Memory &mem = maps->memories.at(ptr);
+   if (mem.d_ptr) { CuMemFree(mem.d_ptr); }
+   for (const void *alias : mem.aliases)
+   {
+      maps->aliases.erase(maps->aliases.find(alias));
+   }
+   mem.aliases.clear();
+   maps->memories.erase(maps->memories.find(ptr));
+   return ptr;
+}
+
+void MemoryManager::SetHostDevicePtr(void *h_ptr, void *d_ptr, const bool host)
+{
+   internal::Memory &base = maps->memories.at(h_ptr);
+   base.d_ptr = d_ptr;
+   base.host = host;
+}
+
+bool MemoryManager::IsKnown(const void *ptr)
+{
+   return maps->memories.find(ptr) != maps->memories.end();
+}
+
+bool MemoryManager::IsOnHost(const void *ptr)
+{
+   return maps->memories.at(ptr).host;
+}
+
+std::size_t MemoryManager::Bytes(const void *ptr)
+{
+   return maps->memories.at(ptr).bytes;
+}
+
+void *MemoryManager::GetDevicePtr(const void *ptr)
+{
+   internal::Memory &base = maps->memories.at(ptr);
+   const size_t bytes = base.bytes;
+   if (!base.d_ptr)
+   {
+      CuMemAlloc(&base.d_ptr, bytes);
+      CuMemcpyHtoD(base.d_ptr, ptr, bytes);
+      base.host = false;
+   }
+   return base.d_ptr;
+}
+
+// Looks if ptr is an alias of one memory
+static const void* AliasBaseMemory(const internal::Ledger *maps,
+                                   const void *ptr)
+{
+   for (internal::MemoryMap::const_iterator mem = maps->memories.begin();
+        mem != maps->memories.end(); mem++)
+   {
+      const void *b_ptr = mem->first;
+      if (b_ptr > ptr) { continue; }
+      const void *end = static_cast<const char*>(b_ptr) + mem->second.bytes;
+      if (ptr < end) { return b_ptr; }
+   }
+   return nullptr;
+}
+
+bool MemoryManager::IsAlias(const void *ptr)
+{
+   const internal::AliasMap::const_iterator found = maps->aliases.find(ptr);
+   if (found != maps->aliases.end()) { return true; }
+   MFEM_ASSERT(!IsKnown(ptr), "Ptr is an already known address!");
+   const void *base = AliasBaseMemory(maps, ptr);
+   if (!base) { return false; }
+   internal::Memory &mem = maps->memories.at(base);
+   const long offset = static_cast<const char*>(ptr) -
+                       static_cast<const char*> (base);
+   const internal::Alias *alias = new internal::Alias{&mem, offset};
+   maps->aliases.emplace(ptr, alias);
+   mem.aliases.push_back(ptr);
+   return true;
+}
+
+static inline bool MmDeviceIniFilter(void)
+{
+   if (!mm.UsingMM()) { return true; }
+   if (!mm.IsEnabled()) { return true; }
+   if (!Device::IsAvailable()) { return true; }
+   if (!Device::IsConfigured()) { return true; }
+   return false;
+}
+
+// Turn a known address into the right host or device address. Alloc, Push, or
+// Pull it if necessary.
+static void *PtrKnown(internal::Ledger *maps, void *ptr)
+{
+   internal::Memory &base = maps->memories.at(ptr);
+   const bool ptr_on_host = base.host;
+   const std::size_t bytes = base.bytes;
+   const bool run_on_device = Device::Allows(Backend::DEVICE_MASK);
+   if (ptr_on_host && !run_on_device) { return ptr; }
+   if (bytes==0) { mfem_error("PtrKnown bytes==0"); }
+   if (!base.d_ptr) { CuMemAlloc(&base.d_ptr, bytes); }
+   if (!base.d_ptr) { mfem_error("PtrKnown !base->d_ptr"); }
+   if (!ptr_on_host && run_on_device) { return base.d_ptr; }
+   if (!ptr) { mfem_error("PtrKnown !ptr"); }
+   if (!ptr_on_host && !run_on_device) // Pull
+   {
+      CuMemcpyDtoH(ptr, base.d_ptr, bytes);
+      base.host = true;
+      return ptr;
+   }
+   // Push
+   if (!(ptr_on_host && run_on_device)) { mfem_error("PtrKnown !(host && gpu)"); }
+   CuMemcpyHtoD(base.d_ptr, ptr, bytes);
+   base.host = false;
+   return base.d_ptr;
+}
+
+// Turn an alias into the right host or device address. Alloc, Push, or Pull it
+// if necessary.
+static void *PtrAlias(internal::Ledger *maps, void *ptr)
+{
+   const bool gpu = Device::Allows(Backend::DEVICE_MASK);
+   const internal::Alias *alias = maps->aliases.at(ptr);
+   const internal::Memory *base = alias->mem;
+   const bool host = base->host;
+   const bool device = !base->host;
+   const std::size_t bytes = base->bytes;
+   if (host && !gpu) { return ptr; }
+   if (bytes==0) { mfem_error("PtrAlias bytes==0"); }
+   if (!base->d_ptr) { CuMemAlloc(&(alias->mem->d_ptr), bytes); }
+   if (!base->d_ptr) { mfem_error("PtrAlias !base->d_ptr"); }
+   void *a_ptr = static_cast<char*>(base->d_ptr) + alias->offset;
+   if (device && gpu) { return a_ptr; }
+   if (!base->h_ptr) { mfem_error("PtrAlias !base->h_ptr"); }
+   if (device && !gpu) // Pull
+   {
+      CuMemcpyDtoH(base->h_ptr, base->d_ptr, bytes);
+      alias->mem->host = true;
+      return ptr;
+   }
+   // Push
+   if (!(host && gpu)) { mfem_error("PtrAlias !(host && gpu)"); }
+   CuMemcpyHtoD(base->d_ptr, base->h_ptr, bytes);
+   alias->mem->host = false;
+   return a_ptr;
+}
+
+void *MemoryManager::Ptr(void *ptr)
+{
+   if (ptr==NULL) { return NULL; };
+   if (MmDeviceIniFilter()) { return ptr; }
+   if (IsKnown(ptr)) { return PtrKnown(maps, ptr); }
+   if (IsAlias(ptr)) { return PtrAlias(maps, ptr); }
+   if (Device::Allows(Backend::DEVICE_MASK))
+   {
+      mfem_error("Trying to use unknown pointer on the DEVICE!");
+   }
+   return ptr;
+}
+
+const void *MemoryManager::Ptr(const void *ptr)
+{
+   return static_cast<const void*>(Ptr(const_cast<void*>(ptr)));
+}
+
+static void PushKnown(internal::Ledger *maps,
+                      const void *ptr, const std::size_t bytes)
+{
+   internal::Memory &base = maps->memories.at(ptr);
+   if (!base.d_ptr) { CuMemAlloc(&base.d_ptr, base.bytes); }
+   CuMemcpyHtoD(base.d_ptr, ptr, bytes == 0 ? base.bytes : bytes);
+}
+
+static void PushAlias(const internal::Ledger *maps,
+                      const void *ptr, const std::size_t bytes)
+{
+   const internal::Alias *alias = maps->aliases.at(ptr);
+   void *dst = static_cast<char*>(alias->mem->d_ptr) + alias->offset;
+   CuMemcpyHtoD(dst, ptr, bytes);
+}
+
+void MemoryManager::Push(const void *ptr, const std::size_t bytes)
+{
+   if (MmDeviceIniFilter()) { return; }
+   if (IsKnown(ptr)) { return PushKnown(maps, ptr, bytes); }
+   if (IsAlias(ptr)) { return PushAlias(maps, ptr, bytes); }
+   if (Device::Allows(Backend::DEVICE_MASK))
+   { mfem_error("Unknown pointer to push to!"); }
+}
+
+static void PullKnown(const internal::Ledger *maps,
+                      const void *ptr, const std::size_t bytes)
+{
+   const internal::Memory &base = maps->memories.at(ptr);
+   const bool host = base.host;
+   if (host) { return; }
+   CuMemcpyDtoH(base.h_ptr, base.d_ptr, bytes == 0 ? base.bytes : bytes);
+}
+
+static void PullAlias(const internal::Ledger *maps,
+                      const void *ptr, const std::size_t bytes)
+{
+   const internal::Alias *alias = maps->aliases.at(ptr);
+   const bool host = alias->mem->host;
+   if (host) { return; }
+   if (!ptr) { mfem_error("PullAlias !ptr"); }
+   if (!alias->mem->d_ptr) { mfem_error("PullAlias !alias->mem->d_ptr"); }
+   CuMemcpyDtoH(const_cast<void*>(ptr),
+                static_cast<char*>(alias->mem->d_ptr) + alias->offset,
+                bytes);
+}
+
+void MemoryManager::Pull(const void *ptr, const std::size_t bytes)
+{
+   if (MmDeviceIniFilter()) { return; }
+   if (IsKnown(ptr)) { return PullKnown(maps, ptr, bytes); }
+   if (IsAlias(ptr)) { return PullAlias(maps, ptr, bytes); }
+   if (Device::Allows(Backend::DEVICE_MASK))
+   { mfem_error("Unknown pointer to pull from!"); }
+}
+
+void* MemoryManager::Memcpy(void *dst, const void *src,
+                            const std::size_t bytes, const bool async)
+{
+   void *d_dst = Ptr(dst);
+   void *d_src = const_cast<void*>(Ptr(src));
+   if (bytes == 0) { return dst; }
+   const bool run_on_host = !Device::Allows(Backend::DEVICE_MASK);
+   if (run_on_host) { return std::memcpy(dst, src, bytes); }
+   if (!async) { return CuMemcpyDtoD(d_dst, d_src, bytes); }
+   return CuMemcpyDtoDAsync(d_dst, d_src, bytes);
+}
+
+void MemoryManager::RegisterCheck(void *ptr)
+{
+   if (ptr != NULL && UsingMM())
+   {
+      if (!IsKnown(ptr))
+      {
+         mfem_error("Pointer is not registered!");
+      }
+   }
+}
+
+void MemoryManager::PrintPtrs(void)
+{
+   for (const auto& n : maps->memories)
+   {
+      const internal::Memory &mem = n.second;
+      mfem::out << std::endl
+                << "key " << n.first << ", "
+                << "host " << mem.host << ", "
+                << "h_ptr " << mem.h_ptr << ", "
+                << "d_ptr " << mem.d_ptr;
+   }
+}
+
+void MemoryManager::GetAll(void)
+{
+   for (const auto& n : maps->memories)
+   {
+      const void *ptr = n.first;
+      Ptr(ptr);
+   }
+}
+
+MemoryManager mm;
+bool MemoryManager::exists = false;
+
+} // namespace mfem
@@ -0,0 +1,186 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_MEM_MANAGER_HPP
+#define MFEM_MEM_MANAGER_HPP
+
+#include "globals.hpp"
+
+namespace mfem
+{
+
+// Implementation of MFEM's lightweight device/host memory manager designed to
+// work seamlessly with the OCCA, RAJA, and other kernels supported by MFEM.
+
+/// The memory manager class
+class MemoryManager
+{
+private:
+   /// Allow to enable/disable the Ptr, Pull and Push functionalities
+   /// New and Delete will still continue to register the pointers
+   bool enabled;
+
+   /// Allow to detect if a global memory manager instance exists
+   static bool exists;
+
+public:
+   MemoryManager();
+   ~MemoryManager();
+
+   /// Adds an address in the map
+   void *Insert(void *ptr, const std::size_t bytes);
+
+   /// Remove the address from the map, as well as all its aliases
+   void *Erase(void *ptr);
+
+   /// Return true if the memory manager is used: pointers seen by mfem::New and
+   /// mfem::Delete will be inserted in the ledger and erased from it
+   static inline bool UsingMM()
+   {
+#ifdef MFEM_USE_MM
+      return true;
+#else
+      return false;
+#endif
+   }
+
+   /// Disable the memory manager: Ptr, Push and Pull will be no-op
+   void Disable() { enabled = false; }
+
+   /// Enable the memory manager: Ptr, Push and Pull wont be no-op
+   void Enable() { enabled = true; }
+
+   /// Return true if the memory manager is used and enabled
+   bool IsEnabled() { return UsingMM() && enabled; }
+
+   /// The opposite of IsEnabled().
+   bool IsDisabled() { return !IsEnabled(); }
+
+   /// Return true if a global memory manager instance exists
+   static bool Exists() { return exists; }
+
+   /** @brief Translates ptr to host or device address, depending on what
+       backends are currently allowed by the Device class and on the ptr
+       state. */
+   void *Ptr(void *ptr);
+   const void *Ptr(const void *ptr);
+
+   /// Data will be pushed/pulled before the copy happens on the H or the D
+   void* Memcpy(void *dst, const void *src,
+                std::size_t bytes, const bool async = false);
+
+   /// Return the bytes of the memory region which base address is ptr
+   std::size_t Bytes(const void *ptr);
+
+   /// Return true if the registered pointer is on the host side
+   bool IsOnHost(const void *ptr);
+
+   /// Return true if the pointer has been registered
+   bool IsKnown(const void *ptr);
+
+   /// Return true if the pointer is an alias inside a registered memory region
+   bool IsAlias(const void *ptr);
+
+   /// Push the data to the device
+   void Push(const void *ptr, const std::size_t bytes =0);
+
+   /// Pull the data from the device
+   void Pull(const void *ptr, const std::size_t bytes =0);
+
+   /// Return the corresponding device pointer of ptr, allocating and moving the
+   /// data if needed (used in OccaPtr)
+   void *GetDevicePtr(const void *ptr);
+
+   /// Registers external host pointer in the memory manager which will manage
+   /// the corresponding device pointer, but not the provided host pointer.
+   template<class T>
+   void RegisterHostPtr(T *ptr_host, const std::size_t size)
+   {
+      Insert(ptr_host, size*sizeof(T));
+#ifdef MFEM_DEBUG
+      RegisterCheck(ptr_host);
+#endif
+   }
+
+   /// Registers external host and device pointers in the memory manager.
+   template<class T>
+   void RegisterHostAndDevicePtr(T *ptr_host, T *ptr_device,
+                                 const std::size_t size, const bool host)
+   {
+      RegisterHostPtr(ptr_host, size);
+      SetHostDevicePtr(ptr_host, ptr_device, host);
+   }
+
+   /// Set the host h_ptr, device d_ptr and mode host of the memory region just
+   /// been registered with h_ptr (see RegisterHostAndDevicePtr)
+   void SetHostDevicePtr(void *h_ptr, void *d_ptr, const bool host);
+
+   /// Unregisters the host pointer from the memory manager. To be used with
+   /// memory not allocated by the memory manager.
+   template<class T>
+   void UnregisterHostPtr(T *ptr) { Erase(ptr); }
+
+   /// Check if pointer has been registered in the memory manager
+   void RegisterCheck(void *ptr);
+
+   /// Prints all pointers known by the memory manager
+   void PrintPtrs(void);
+
+   /// Copies all memory to the current memory space
+   void GetAll(void);
+};
+
+/// The (single) global memory manager object
+extern MemoryManager mm;
+
+/// Main memory allocation template function. Allocates n*size bytes and returns
+/// a pointer to the allocated memory.
+template<class T>
+inline T *New(const std::size_t n)
+{
+   T *ptr = new T[n];
+   if (!MemoryManager::Exists()) { return ptr; }
+   return static_cast<T*>(mm.Insert(ptr, n*sizeof(T)));
+}
+
+/// Frees the memory space pointed to by ptr, which must have been returned by a
+/// previous call to mfem::New.
+template<class T>
+inline void Delete(T *ptr)
+{
+   static_assert(!std::is_void<T>::value, "Cannot Delete a void pointer. "
+                 "Explicitly provide the correct type as a template parameter.");
+   if (!ptr) { return; }
+   delete [] ptr;
+   if (!MemoryManager::Exists()) { return; }
+   mm.Erase(ptr);
+}
+
+/// Return a host or device address corresponding to current memory space
+template <class T>
+inline T *Ptr(T *a) { return static_cast<T*>(mm.Ptr(a)); }
+
+/// Data will be pushed/pulled before the copy happens on the host or the device
+inline void* Memcpy(void *dst, const void *src,
+                    std::size_t bytes, const bool async = false)
+{ return mm.Memcpy(dst, src, bytes, async); }
+
+/// Push the data to the device
+inline void Push(const void *ptr, const std::size_t bytes = 0)
+{ return mm.Push(ptr, bytes); }
+
+/// Pull the data from the device
+inline void Pull(const void *ptr, const std::size_t bytes = 0)
+{ return mm.Pull(ptr, bytes); }
+
+} // namespace mfem
+
+#endif // MFEM_MEM_MANAGER_HPP
@@ -0,0 +1,61 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "forall.hpp"
+
+namespace mfem
+{
+
+// This variable is defined in device.cpp:
+namespace internal { extern OccaDevice occaDevice; }
+
+static OccaMemory OccaWrapMemory(const OccaDevice dev, const void *d_adrs,
+                                 const size_t bytes)
+{
+   // This function is called when an OCCA kernel is going to be used.
+#ifdef MFEM_USE_OCCA
+   void *adrs = const_cast<void*>(d_adrs);
+#if defined(MFEM_USE_CUDA) && OCCA_CUDA_ENABLED
+   // If OCCA_CUDA is allowed, it will be used since it has the highest priority
+   if (Device::Allows(Backend::OCCA_CUDA))
+   {
+      return occa::cuda::wrapMemory(dev, adrs, bytes);
+   }
+#endif // MFEM_USE_CUDA && OCCA_CUDA_ENABLED
+   // otherwise, fallback to occa::cpu address space
+   return occa::cpu::wrapMemory(dev, adrs, bytes);
+#else // MFEM_USE_OCCA
+   return (void*)NULL;
+#endif
+}
+
+OccaMemory OccaPtr(const void *ptr)
+{
+   // This function is called when 'ptr' needs to be passed to an OCCA kernel.
+   OccaDevice dev = internal::occaDevice;
+   if (!mm.UsingMM()) { return OccaWrapMemory(dev, ptr, 0); }
+   const bool known = mm.IsKnown(ptr);
+   if (!known) { mfem_error("OccaPtr: Unknown address!"); }
+   const bool ptr_on_host = mm.IsOnHost(ptr);
+   const size_t bytes = mm.Bytes(ptr);
+   const bool run_on_host = !Device::Allows(Backend::DEVICE_MASK);
+   // If the priority of a host OCCA backend is higher than all device OCCA
+   // backends, then we will need to run-on-host even if the Device allows a
+   // device backend.
+   if (ptr_on_host && run_on_host) { return OccaWrapMemory(dev, ptr, bytes); }
+   if (run_on_host) { mfem_error("OccaPtr: !ptr_on_host && run_on_host"); }
+   void *d_ptr = mm.GetDevicePtr(ptr);
+   return OccaWrapMemory(dev, d_ptr, bytes);
+}
+
+OccaDevice OccaDev() { return internal::occaDevice; }
+
+} // namespace mfem
@@ -0,0 +1,43 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_OCCA_HPP
+#define MFEM_OCCA_HPP
+
+#include "../config/config.hpp"
+
+#ifdef MFEM_USE_OCCA
+#include <occa.hpp>
+
+#if defined(MFEM_USE_CUDA) && OCCA_CUDA_ENABLED
+#include <occa/modes/cuda/utils.hpp>
+#endif
+
+typedef occa::device OccaDevice;
+typedef occa::memory OccaMemory;
+
+#else // MFEM_USE_OCCA
+
+typedef void* OccaDevice;
+typedef void* OccaMemory;
+
+#endif // MFEM_USE_OCCA
+
+namespace mfem
+{
+
+// Function called when the pointer 'a' needs to be passed to an OCCA kernel.
+OccaMemory OccaPtr(const void *a);
+OccaDevice OccaDev();
+
+} // namespace mfem
+
+#endif // MFEM_OCCA_HPP
@@ -15,6 +15,7 @@
 #include "table.hpp"
 #include "error.hpp"

+#include "../general/forall.hpp"
 #include <iostream>
 #include <iomanip>

@@ -29,8 +30,8 @@ Table::Table(const Table &table)
   if (size >= 0)
   {
      const int nnz = table.I[size];
-      I = new int[size+1];
-      J = new int[nnz];
+      I = mfem::New<int>(size+1);
+      J = mfem::New<int>(nnz);
      memcpy(I, table.I, sizeof(int)*(size+1));
      memcpy(J, table.J, sizeof(int)*nnz);
   }
@@ -55,8 +56,8 @@ Table::Table (int dim, int connections_per_row)
   int i, j, sum = dim * connections_per_row;

   size = dim;
-   I = new int[size+1];
-   J = new int[sum];
+   I = mfem::New<int>(size+1);
+   J = mfem::New<int>(sum);

   I[0] = 0;
   for (i = 1; i <= size; i++)
@@ -70,8 +71,8 @@ Table::Table (int nrows, int *partitioning)
 {
   size = nrows;

-   I = new int[size+1];
-   J = new int[size];
+   I = mfem::New<int>(size+1);
+   J = mfem::New<int>(size);

   for (int i = 0; i < size; i++)
   {
@@ -100,7 +101,8 @@ void Table::MakeJ()
      j = I[i], I[i] = k, k += j;
   }

-   J = new int[I[size]=k];
+   if (J) { mfem::Delete(J); }
+   J = mfem::New<int>(I[size]=k);
 }

 void Table::AddConnections (int r, const int *c, int nc)
@@ -147,14 +149,14 @@ void Table::SetDims(int rows, int nnz)
   if (size != rows)
   {
      size = rows;
-      if (I) { delete [] I; }
-      I = (rows >= 0) ? (new int[rows+1]) : (NULL);
+      if (I) { mfem::Delete(I); }
+      I = (rows >= 0) ? (mfem::New<int>(rows+1)) : (NULL);
   }

   if (j != nnz)
   {
-      if (J) { delete [] J; }
-      J = (nnz > 0) ? (new int[nnz]) : (NULL);
+      if (J) { mfem::Delete(J); }
+      J = (nnz > 0) ? (mfem::New<int>(nnz)) : (NULL);
   }

   if (size >= 0)
@@ -205,8 +207,8 @@ void Table::SortRows()

 void Table::SetIJ(int *newI, int *newJ, int newsize)
 {
-   delete [] I;
-   delete [] J;
+   mfem::Delete(I);
+   mfem::Delete(J);
   I = newI;
   J = newJ;
   if (newsize >= 0)
@@ -248,7 +250,7 @@ void Table::Finalize()

   if (sum != I[size])
   {
-      int *NewJ = new int[sum];
+      int *NewJ = mfem::New<int>(sum);

      for (i=0; i<size; i++)
      {
@@ -263,7 +265,7 @@ void Table::Finalize()
      }
      I[size] = sum;

-      delete [] J;
+      mfem::Delete(J);

      J = NewJ;

@@ -278,8 +280,8 @@ void Table::MakeFromList(int nrows, const Array<Connection> &list)
   size = nrows;
   int nnz = list.Size();

-   I = new int[size+1];
-   J = new int[nnz];
+   I = mfem::New<int>(size+1);
+   J = mfem::New<int>(nnz);

   for (int i = 0, k = 0; i <= size; i++)
   {
@@ -353,17 +355,17 @@ void Table::Save(std::ostream &out) const

 void Table::Load(std::istream &in)
 {
-   delete [] I;
-   delete [] J;
+   mfem::Delete(I);
+   mfem::Delete(J);

   in >> size;
-   I = new int[size+1];
+   I = mfem::New<int>(size+1);
   for (int i = 0; i <= size; i++)
   {
      in >> I[i];
   }
   int nnz = I[size];
-   J = new int[nnz];
+   J =mfem::New<int>(nnz);
   for (int j = 0; j < nnz; j++)
   {
      in >> J[j];
@@ -372,8 +374,8 @@ void Table::Load(std::istream &in)

 void Table::Clear()
 {
-   delete [] I;
-   delete [] J;
+   mfem::Delete(I);
+   mfem::Delete(J);
   size = -1;
   I = J = NULL;
 }
@@ -382,8 +384,8 @@ void Table::Copy(Table & copy) const
 {
   if (size >= 0)
   {
-      int * i_copy = new int[size+1];
-      int * j_copy = new int[I[size]];
+      int * i_copy = mfem::New<int>(size+1);
+      int * j_copy = mfem::New<int>(I[size]);

      memcpy(i_copy, I, sizeof(int)*(size+1));
      memcpy(j_copy, J, sizeof(int)*I[size]);
@@ -411,8 +413,8 @@ long Table::MemoryUsage() const

 Table::~Table ()
 {
-   if (I) { delete [] I; }
-   if (J) { delete [] J; }
+   if (I) { mfem::Delete(I); }
+   if (J) { mfem::Delete(J); }
 }

 void Transpose (const Table &A, Table &At, int _ncols_A)
@@ -94,8 +94,8 @@ const char *GetConfigStr()
 #ifdef MFEM_THREAD_SAFE
      "MFEM_THREAD_SAFE\n"
 #endif
-#ifdef MFEM_USE_OPENMP
-      "MFEM_USE_OPENMP\n"
+#ifdef MFEM_USE_LEGACY_OPENMP
+      "MFEM_USE_LEGACY_OPENMP\n"
 #endif
 #ifdef MFEM_USE_MEMALLOC
      "MFEM_USE_MEMALLOC\n"
@@ -135,6 +135,18 @@ const char *GetConfigStr()
 #endif
 #ifdef MFEM_USE_SIDRE
      "MFEM_USE_SIDRE\n"
+#endif
+#ifdef MFEM_USE_PUMI
+      "MFEM_USE_PUMI\n"
+#endif
+#ifdef MFEM_USE_CUDA
+      "MFEM_USE_CUDA\n"
+#endif
+#ifdef MFEM_USE_RAJA
+      "MFEM_USE_RAJA\n"
+#endif
+#ifdef MFEM_USE_OCCA
+      "MFEM_USE_OCCA\n"
 #endif
      "MFEM_TIMER_TYPE = " EXPAND_AND_QUOTE(MFEM_TIMER_TYPE)
      ;
@@ -31,6 +31,7 @@ list(APPEND HDRS
  blockvector.hpp
  complex_operator.hpp
  densemat.hpp
+  dtensor.hpp
  handle.hpp
  invariants.hpp
  linalg.hpp
@@ -455,9 +455,9 @@ SparseMatrix * BlockMatrix::CreateMonolithic() const
 {
   int nnz = NumNonZeroElems();

-   int * i_amono = new int[ row_offsets[nRowBlocks]+2 ];
-   int * j_amono = new int[ nnz ];
-   double * data = new double[ nnz ];
+   int * i_amono = mfem::New<int>(row_offsets[nRowBlocks]+2);
+   int * j_amono = mfem::New<int>(nnz);
+   double * data = mfem::New<double>(nnz);

   for (int i = 0; i < row_offsets[nRowBlocks]+2; i++)
   {
@@ -85,7 +85,7 @@ public:
   //! Get the i-th vector in the block
   void GetBlockView(int i, Vector & blockView);

-   int BlockSize(int i) { return blockOffsets[i+1] - blockOffsets[i];}
+   int BlockSize(int i) { return blockOffsets[i+1] - blockOffsets[i]; }

   //! Update method
   /**
@@ -16,6 +16,8 @@
 #include "vector.hpp"
 #include "matrix.hpp"
 #include "densemat.hpp"
+#include "dtensor.hpp"
+#include "../general/forall.hpp"
 #include "../general/table.hpp"
 #include "../general/globals.hpp"

@@ -76,9 +78,9 @@ DenseMatrix::DenseMatrix(const DenseMatrix &m) : Matrix(m.height, m.width)
   if (hw > 0)
   {
      MFEM_ASSERT(m.data, "invalid source matrix");
-      data = new double[hw];
+      data = mfem::New<double>(hw);
      capacity = hw;
-      std::memcpy(data, m.data, sizeof(double)*hw);
+      mfem::Memcpy(data, m.data, sizeof(double)*hw);
   }
   else
   {
@@ -87,13 +89,20 @@ DenseMatrix::DenseMatrix(const DenseMatrix &m) : Matrix(m.height, m.width)
   }
 }

+static void Set(const double d, const int size, double *data)
+{
+   DeviceVector d_data(data);
+   MFEM_FORALL(i, size, d_data[i] = d;);
+}
+
 DenseMatrix::DenseMatrix(int s) : Matrix(s)
 {
   MFEM_ASSERT(s >= 0, "invalid DenseMatrix size: " << s);
   capacity = s*s;
   if (capacity > 0)
   {
-      data = new double[capacity](); // init with zeroes
+      data = mfem::New<double>(capacity);
+      mfem::Set(0.0, capacity, data);
   }
   else
   {
@@ -108,7 +117,8 @@ DenseMatrix::DenseMatrix(int m, int n) : Matrix(m, n)
   capacity = m*n;
   if (capacity > 0)
   {
-      data = new double[capacity](); // init with zeroes
+      data = mfem::New<double>(capacity);
+      mfem::Set(0.0, capacity, data);
   }
   else
   {
@@ -116,19 +126,28 @@ DenseMatrix::DenseMatrix(int m, int n) : Matrix(m, n)
   }
 }

+static void Transpose(const int height, const int width,
+                      double *data, const double *mdata)
+{
+   DeviceVector d_data(data);
+   const DeviceVector d_mdata(mdata);
+   MFEM_FORALL(i, height,
+   {
+      for (int j = 0; j < width; j++)
+      {
+         d_data[i+j*height] = d_mdata[j+i*height];
+      }
+   });
+}
+
 DenseMatrix::DenseMatrix(const DenseMatrix &mat, char ch)
   : Matrix(mat.width, mat.height)
 {
   capacity = height*width;
   if (capacity > 0)
   {
-      data = new double[capacity];
-
-      for (int i = 0; i < height; i++)
-         for (int j = 0; j < width; j++)
-         {
-            (*this)(i,j) = mat(j,i);
-         }
+      data = mfem::New<double>(capacity);
+      mfem::Transpose(height, width, data, mat.Data());
   }
   else
   {
@@ -136,6 +155,13 @@ DenseMatrix::DenseMatrix(const DenseMatrix &mat, char ch)
   }
 }

+DenseMatrix::DenseMatrix(double *d, int h, int w)
+   : Matrix(h, w)
+{
+   data = d;
+   capacity = -h*w;
+}
+
 void DenseMatrix::SetSize(int h, int w)
 {
   MFEM_ASSERT(h >= 0 && w >= 0,
@@ -151,10 +177,11 @@ void DenseMatrix::SetSize(int h, int w)
   {
      if (capacity > 0)
      {
-         delete [] data;
+         mfem::Delete(data);
      }
      capacity = hw;
-      data = new double[hw](); // init with zeroes
+      data = mfem::New<double>(capacity);
+      mfem::Set(0.0, capacity, data);
   }
 }

@@ -168,32 +195,31 @@ const double &DenseMatrix::Elem(int i, int j) const
   return (*this)(i,j);
 }

+static void Mult(const int height, const int width,
+                 const double *data, const double *x, double *y)
+{
+   const DeviceVector d_data(data);
+   const DeviceVector d_x(x);
+   DeviceVector d_y(y);
+   MFEM_FORALL(i, height,
+   {
+      double sum = 0.0;
+      for (int j = 0; j < width; j++)
+      {
+         sum += d_x[j]*d_data[i+j*height];
+      }
+      d_y[i] = sum;
+   });
+}
+
 void DenseMatrix::Mult(const double *x, double *y) const
 {
   if (width == 0)
   {
-      for (int row = 0; row < height; row++)
-      {
-         y[row] = 0.0;
-      }
+      mfem::Set(0.0, height, y);
      return;
   }
-   double *d_col = data;
-   double x_col = x[0];
-   for (int row = 0; row < height; row++)
-   {
-      y[row] = x_col*d_col[row];
-   }
-   d_col += height;
-   for (int col = 1; col < width; col++)
-   {
-      x_col = x[col];
-      for (int row = 0; row < height; row++)
-      {
-         y[row] += x_col*d_col[row];
-      }
-      d_col += height;
-   }
+   mfem::Mult(height, width, data, x, y);
 }

 void DenseMatrix::Mult(const Vector &x, Vector &y) const
@@ -564,11 +590,8 @@ void DenseMatrix::Add(const double c, const DenseMatrix &A)

 DenseMatrix &DenseMatrix::operator=(double c)
 {
-   int s = Height()*Width();
-   for (int i = 0; i < s; i++)
-   {
-      data[i] = c;
-   }
+   const int s = Height()*Width();
+   mfem::Set(c, s, data);
   return *this;
 }

@@ -587,11 +610,9 @@ DenseMatrix &DenseMatrix::operator=(const DenseMatrix &m)
   SetSize(m.height, m.width);

   const int hw = height * width;
-   for (int i = 0; i < hw; i++)
-   {
-      data[i] = m.data[i];
-   }
-
+   const DeviceVector d_m(m.GetData());
+   DeviceVector d_data(data);
+   MFEM_FORALL(i, hw, d_data[i] = d_m[i];);
   return *this;
 }

@@ -2442,16 +2463,11 @@ void DenseMatrix::GetRowSums(Vector &l) const
 void DenseMatrix::Diag(double c, int n)
 {
   SetSize(n);
+   const int N = n*n;
+   DeviceVector d_data(data);
+   MFEM_FORALL(i, N, d_data[i] = 0.0;);
+   MFEM_FORALL(i, n, d_data[i*(n+1)] = c;);

-   int i, N = n*n;
-   for (i = 0; i < N; i++)
-   {
-      data[i] = 0.0;
-   }
-   for (i = 0; i < n; i++)
-   {
-      data[i*(n+1)] = c;
-   }
 }

 void DenseMatrix::Diag(double *diag, int n)
@@ -2594,7 +2610,6 @@ void DenseMatrix::GradToCurl(DenseMatrix &curl)

 void DenseMatrix::GradToDiv(Vector &div)
 {
-
 #ifdef MFEM_DEBUG
   if (Width()*Height() != div.Size())
   {
@@ -2604,13 +2619,11 @@ void DenseMatrix::GradToDiv(Vector &div)

   // div(dof*j+i) <-- (*this)(i,j)

-   int n = height * width;
+   const int n = height * width;
   double *ddata = div.GetData();
-
-   for (int i = 0; i < n; i++)
-   {
-      ddata[i] = data[i];
-   }
+   const DeviceVector d_data(data);
+   DeviceVector d_ddata(ddata);
+   MFEM_FORALL(i, n, d_ddata[i] = d_data[i];);
 }

 void DenseMatrix::CopyRows(const DenseMatrix &A, int row1, int row2)
@@ -3004,7 +3017,7 @@ DenseMatrix::~DenseMatrix()
 {
   if (capacity > 0)
   {
-      delete [] data;
+      mfem::Delete(data);
   }
 }

@@ -3061,20 +3074,20 @@ void Mult(const DenseMatrix &b, const DenseMatrix &c, DenseMatrix &a)
   double *ad = a.Data();
   const double *bd = b.Data();
   const double *cd = c.Data();
-   for (int i = 0; i < ah*aw; i++)
-   {
-      ad[i] = 0.0;
-   }
-   for (int j = 0; j < aw; j++)
+   const DeviceVector d_bd(bd);
+   const DeviceVector d_cd(cd);
+   DeviceVector d_ad(ad);
+   MFEM_FORALL(i, ah*aw, d_ad[i] = 0.0;);
+   MFEM_FORALL(j, aw,
   {
      for (int k = 0; k < bw; k++)
      {
         for (int i = 0; i < ah; i++)
         {
-            ad[i+j*ah] += bd[i+k*ah] * cd[k+j*bw];
+            d_ad[i+j*ah] += d_bd[i+k*ah] * d_cd[k+j*bw];
         }
      }
-   }
+   });
 #endif
 }

@@ -3368,16 +3381,22 @@ void CalcOrtho(const DenseMatrix &J, Vector &n)

 void MultAAt(const DenseMatrix &a, DenseMatrix &aat)
 {
-   for (int i = 0; i < a.Height(); i++)
-      for (int j = 0; j <= i; j++)
+   const int height = a.Height();
+   const int width = a.Width();
+   const DeviceVector d_a(a.GetData());
+   DeviceVector d_aat(aat.GetData());
+   MFEM_FORALL(i, height,
+   {
+      for (int j=0; j<=i; j++)
      {
-         double temp = 0.;
-         for (int k = 0; k < a.Width(); k++)
+         double temp = 0.0;
+         for (int k=0; k<width; k++)
         {
-            temp += a(i,k) * a(j,k);
+            temp += d_a[i+k*height] * d_a[j+k*height];
         }
-         aat(j,i) = aat(i,j) = temp;
+         d_aat[j+i*height] = d_aat[i+j*height] = temp;
      }
+   });
 }

 void AddMultADAt(const DenseMatrix &A, const Vector &D, DenseMatrix &ADAt)
@@ -3903,18 +3922,20 @@ void AddMult_a_VVt(const double a, const Vector &v, DenseMatrix &VVt)
      mfem_error("AddMult_a_VVt(...)");
   }
 #endif
-
-   for (int i = 0; i < n; i++)
+   const int height = VVt.Height();
+   const DeviceVector d_v(v.GetData());
+   DeviceVector d_VVt(VVt.GetData());
+   MFEM_FORALL(i, n,
   {
-      double avi = a * v(i);
+      double avi = a * d_v[i];
      for (int j = 0; j < i; j++)
      {
-         double avivj = avi * v(j);
-         VVt(i, j) += avivj;
-         VVt(j, i) += avivj;
+         double avivj = avi * d_v[j];
+         d_VVt[i+j*height] += avivj;
+         d_VVt[j+i*height] += avivj;
      }
-      VVt(i, i) += avi * v(i);
-   }
+      d_VVt[i+i*height] += avi * d_v[i];
+   });
 }


@@ -3926,47 +3947,49 @@ void LUFactors::Factor(int m)
   MFEM_VERIFY(!info, "LAPACK: error in DGETRF");
 #else
   // compiling without LAPACK
-   double *data = this->data;
-   for (int i = 0; i < m; i++)
+   DeviceArray d_ipiv(ipiv);
+   DeviceVector d_data(data);
+   MFEM_FORALL(i, m,
   {
      // pivoting
      {
         int piv = i;
-         double a = std::abs(data[piv+i*m]);
+         double a = fabs(d_data[piv+i*m]);
         for (int j = i+1; j < m; j++)
         {
-            const double b = std::abs(data[j+i*m]);
+            const double b = fabs(d_data[j+i*m]);
            if (b > a)
            {
               a = b;
               piv = j;
            }
         }
-         ipiv[i] = piv;
+         d_ipiv[i] = piv;
         if (piv != i)
         {
            // swap rows i and piv in both L and U parts
            for (int j = 0; j < m; j++)
            {
-               Swap<double>(data[i+j*m], data[piv+j*m]);
+               const double tmp = d_data[i+j*m];
+               d_data[i+j*m] = d_data[piv+j*m];
+               d_data[piv+j*m] = tmp;
            }
         }
      }
-      MFEM_ASSERT(data[i+i*m] != 0.0, "division by zero");
-      const double a_ii_inv = 1.0/data[i+i*m];
+      const double a_ii_inv = 1.0/d_data[i+i*m];
      for (int j = i+1; j < m; j++)
      {
-         data[j+i*m] *= a_ii_inv;
+         d_data[j+i*m] *= a_ii_inv;
      }
      for (int k = i+1; k < m; k++)
      {
-         const double a_ik = data[i+k*m];
+         const double a_ik = d_data[i+k*m];
         for (int j = i+1; j < m; j++)
         {
-            data[j+k*m] -= a_ik * data[j+i*m];
+            d_data[j+k*m] -= a_ik * d_data[j+i*m];
         }
      }
-   }
+   });
 #endif
 }

@@ -4028,24 +4051,29 @@ void LUFactors::LSolve(int m, int n, double *X) const
   const double *data = this->data;
   const int *ipiv = this->ipiv;
   double *x = X;
-   for (int k = 0; k < n; k++)
+   const DeviceVector d_data(data);
+   const DeviceArray d_ipiv(ipiv);
+   DeviceVector d_x(x);
+   MFEM_FORALL(k, n,
   {
+      double *d_mx = &d_x[k*m];
      // X <- P X
      for (int i = 0; i < m; i++)
      {
-         Swap<double>(x[i], x[ipiv[i]-ipiv_base]);
+         const double tmp = d_mx[i];
+         d_mx[i] = d_mx[d_ipiv[i]];
+         d_mx[d_ipiv[i]] = tmp;
      }
      // X <- L^{-1} X
      for (int j = 0; j < m; j++)
      {
-         const double x_j = x[j];
+         const double d_mx_j = d_mx[j];
         for (int i = j+1; i < m; i++)
         {
-            x[i] -= data[i+j*m] * x_j;
+            d_mx[i] -= d_data[i+j*m] * d_mx_j;
         }
      }
-      x += m;
-   }
+   });
 }

 void LUFactors::USolve(int m, int n, double *X) const
@@ -4053,18 +4081,20 @@ void LUFactors::USolve(int m, int n, double *X) const
   const double *data = this->data;
   double *x = X;
   // X <- U^{-1} X
-   for (int k = 0; k < n; k++)
+   const DeviceVector d_data(data);
+   DeviceVector d_x(x);
+   MFEM_FORALL(k, n,
   {
+      double *d_mx = &d_x[k*m];
      for (int j = m-1; j >= 0; j--)
      {
-         const double x_j = ( x[j] /= data[j+j*m] );
+         const double x_j = ( d_mx[j] /= d_data[j+j*m] );
         for (int i = 0; i < j; i++)
         {
-            x[i] -= data[i+j*m] * x_j;
+            d_mx[i] -= d_data[i+j*m] * x_j;
         }
      }
-      x += m;
-   }
+   });
 }

 void LUFactors::Solve(int m, int n, double *X) const
@@ -4214,8 +4244,8 @@ DenseMatrixInverse::DenseMatrixInverse(const DenseMatrix &mat)
 {
   MFEM_ASSERT(height == width, "not a square matrix");
   a = &mat;
-   lu.data = new double[width*width];
-   lu.ipiv = new int[width];
+   lu.data = mfem::New<double>(width*width);
+   lu.ipiv = mfem::New<int>(width);
   Factor();
 }

@@ -4224,31 +4254,40 @@ DenseMatrixInverse::DenseMatrixInverse(const DenseMatrix *mat)
 {
   MFEM_ASSERT(height == width, "not a square matrix");
   a = mat;
-   lu.data = new double[width*width];
-   lu.ipiv = new int[width];
+   lu.data = mfem::New<double>(width*width);
+   lu.ipiv = mfem::New<int>(width);
 }

 void DenseMatrixInverse::Factor()
 {
   MFEM_ASSERT(a, "DenseMatrix is not given");
   const double *adata = a->data;
-   for (int i = 0, s = width*width; i < s; i++)
+   const int N = width*width;
+   const DeviceVector d_data(adata);
+   DeviceVector d_ludata(lu.data);
+   MFEM_FORALL(i, N,
   {
-      lu.data[i] = adata[i];
-   }
+      d_ludata[i] = d_data[i];
+   });
   lu.Factor(width);
 }

+void DenseMatrixInverse::GetInverseMatrix(DenseMatrix &Ainv) const
+{
+   Ainv.SetSize(width);
+   lu.GetInverseMatrix(width, Ainv.Data());
+}
+
 void DenseMatrixInverse::Factor(const DenseMatrix &mat)
 {
   MFEM_VERIFY(mat.height == mat.width, "DenseMatrix is not square!");
   if (width != mat.width)
   {
      height = width = mat.width;
-      delete [] lu.data;
-      lu.data = new double[width*width];
-      delete [] lu.ipiv;
-      lu.ipiv = new int[width];
+      mfem::Delete(lu.data);
+      lu.data = mfem::New<double>(width*width);
+      mfem::Delete(lu.ipiv);
+      lu.ipiv = mfem::New<int>(width);
   }
   a = &mat;
   Factor();
@@ -4286,8 +4325,8 @@ void DenseMatrixInverse::TestInversion()

 DenseMatrixInverse::~DenseMatrixInverse()
 {
-   delete [] lu.data;
-   delete [] lu.ipiv;
+   mfem::Delete(lu.data);
+   mfem::Delete(lu.ipiv);
 }


@@ -56,8 +56,7 @@ public:
   /** Construct a DenseMatrix using existing data array. The DenseMatrix does
       not assume ownership of the data array, i.e. it will not delete the
       array. */
-   DenseMatrix(double *d, int h, int w) : Matrix(h, w)
-   { data = d; capacity = -h*w; }
+   DenseMatrix(double *d, int h, int w);

   /// Change the data array and the size of the DenseMatrix.
   /** The DenseMatrix does not assume ownership of the data array, i.e. it will
@@ -71,7 +70,7 @@ public:
       not delete the new array @a d. This method will delete the current data
       array, if owned. */
   void Reset(double *d, int h, int w)
-   { if (OwnsData()) { delete [] data; } UseExternalData(d, h, w); }
+   { if (OwnsData()) { mfem::Delete(data); } UseExternalData(d, h, w); }

   /** Clear the data array and the dimensions of the DenseMatrix. This method
       should not be used with DenseMatrix that owns its current data array. */
@@ -79,7 +78,7 @@ public:

   /// Delete the matrix data array (if owned) and reset the matrix state.
   void Clear()
-   { if (OwnsData()) { delete [] data; } ClearExternalData(); }
+   { if (OwnsData()) { mfem::Delete(data); } ClearExternalData(); }

   /// For backward compatibility define Size to be synonym of Width()
   int Size() const { return Width(); }
@@ -580,12 +579,11 @@ public:
   /// Multiply the inverse matrix by another matrix: X = A^{-1} B.
   void Mult(const DenseMatrix &B, DenseMatrix &X) const;

+   /// Multiply the inverse matrix by another matrix: X <- A^{-1} X.
+   void Mult(DenseMatrix &X) const { lu.Solve(width, X.Width(), X.Data()); }
+
   /// Compute and return the inverse matrix in Ainv.
-   void GetInverseMatrix(DenseMatrix &Ainv) const
-   {
-      Ainv.SetSize(width);
-      lu.GetInverseMatrix(width, Ainv.Data());
-   }
+   void GetInverseMatrix(DenseMatrix &Ainv) const;

   /// Compute the determinant of the original DenseMatrix using the LU factors.
   double Det() const { return lu.Det(width); }
@@ -674,7 +672,7 @@ public:
      : Mk(NULL, i, j)
   {
      nk = k;
-      tdata = new double[i*j*k];
+      tdata = mfem::New<double>(i*j*k);
      own_data = true;
   }

@@ -685,8 +683,8 @@ public:
      const int size = Mk.Height()*Mk.Width()*nk;
      if (size > 0)
      {
-         tdata = new double[size];
-         std::memcpy(tdata, other.tdata, sizeof(double) * size);
+         tdata = mfem::New<double>(size);
+         mfem::Memcpy(tdata, other.tdata, sizeof(double) * size);
      }
      else
      {
@@ -700,16 +698,16 @@ public:

   void SetSize(int i, int j, int k)
   {
-      if (own_data) { delete [] tdata; }
+      if (own_data) { mfem::Delete(tdata); }
      Mk.UseExternalData(NULL, i, j);
      nk = k;
-      tdata = new double[i*j*k];
+      tdata = mfem::New<double>(i*j*k);
      own_data = true;
   }

   void UseExternalData(double *ext_data, int i, int j, int k)
   {
-      if (own_data) { delete [] tdata; }
+      if (own_data) { mfem::Delete(tdata); }
      Mk.UseExternalData(NULL, i, j);
      nk = k;
      tdata = ext_data;
@@ -732,6 +730,8 @@ public:

   double *Data() { return tdata; }

+   const double *Data() const { return tdata; }
+
   /** Matrix-vector product from unassembled element matrices, assuming both
       'x' and 'y' use the same elem_dof table. */
   void AddMult(const Table &elem_dof, const Vector &x, Vector &y) const;
@@ -743,7 +743,7 @@ public:

   ~DenseTensor()
   {
-      if (own_data) { delete [] tdata; }
+      if (own_data) { mfem::Delete(tdata); }
   }
 };

@@ -0,0 +1,155 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_DTENSOR
+#define MFEM_DTENSOR
+
+#include "../general/cuda.hpp"
+#include "../general/mem_manager.hpp"
+
+namespace mfem
+{
+
+/// A Class to compute the real index from the multi-indices of a tensor
+template <int N, int Dim, typename T, typename... Args>
+class TensorInd
+{
+public:
+   MFEM_ATTR_HOST_DEVICE
+   static inline int result(const int* sizes, T first, Args... args)
+   {
+#ifndef MFEM_USE_CUDA
+      MFEM_ASSERT(first<sizes[N-1],"Trying to access out of boundary.");
+#endif
+      return first + sizes[N - 1] * TensorInd < N + 1, Dim, Args... >
+             ::result(sizes, args...);
+   }
+};
+
+// Terminal case
+template <int Dim, typename T, typename... Args>
+class TensorInd<Dim, Dim, T, Args...>
+{
+public:
+   MFEM_ATTR_HOST_DEVICE
+   static inline int result(const int* sizes, T first, Args... args)
+   {
+#ifndef MFEM_USE_CUDA
+      MFEM_ASSERT(first<sizes[Dim-1],"Trying to access out of boundary.");
+#endif
+      return first;
+   }
+};
+
+
+/// A class to initialize the size of a Tensor
+template <int N, int Dim, typename T, typename... Args>
+class Init
+{
+public:
+   static inline int result(int* sizes, T first, Args... args)
+   {
+      sizes[N - 1] = first;
+      return first * Init < N + 1, Dim, Args... >::result(sizes, args...);
+   }
+};
+
+// Terminal case
+template <int Dim, typename T, typename... Args>
+class Init<Dim, Dim, T, Args...>
+{
+public:
+   static inline int result(int* sizes, T first, Args... args)
+   {
+      sizes[Dim - 1] = first;
+      return first;
+   }
+};
+
+
+/// A basic generic Tensor class, appropriate for use on the GPU
+template<int Dim, typename Scalar = double>
+class DeviceTensor
+{
+protected:
+   int capacity;
+   Scalar *data;
+   int sizes[Dim];
+
+public:
+   /// Default constructor
+   DeviceTensor() = delete;
+
+   /// Constructor to initialize a tensor from the Scalar array _data
+   template <typename... Args>
+   DeviceTensor(Scalar* _data, Args... args)
+   {
+      static_assert(sizeof...(args) == Dim, "Wrong number of arguments");
+      // Initialize sizes, and compute the number of values
+      const long int nb = Init<1, Dim, Args...>::result(sizes, args...);
+      capacity = nb;
+      data = (capacity > 0) ? mfem::Ptr(_data) : NULL;
+   }
+
+   /// Constructor to initialize a tensor from the Scalar array _data
+   DeviceTensor(const Scalar* _data)
+   {
+      data = (Scalar*) mfem::Ptr(_data);
+   }
+
+   /// Constructor to initialize a tensor from the Scalar array _data
+   DeviceTensor(Scalar* _data) { data = mfem::Ptr(_data); }
+
+   /// Constructor to initialize a tensor from the const Scalar array _data
+   template <typename... Args>
+   DeviceTensor(const Scalar* _data, Args... args)
+   {
+      static_assert(sizeof...(args) == Dim, "Wrong number of arguments");
+      // Initialize sizes, and compute the number of values
+      const long int nb = Init<1, Dim, Args...>::result(sizes, args...);
+      capacity = nb;
+      data = (capacity > 0) ? (Scalar*)mfem::Ptr(_data) : NULL;
+   }
+
+   /// Copy constructor
+   MFEM_ATTR_HOST_DEVICE DeviceTensor(const DeviceTensor& t)
+   {
+      for (int i = 0; i < Dim; ++i)
+      {
+         sizes[i] = t.sizes[i];
+      }
+      data = t.data;
+   }
+
+   /// Conversion to `Scalar *`.
+   inline operator Scalar *() const { return data; }
+
+   /// Const accessor for the data
+   template <typename... Args> MFEM_ATTR_HOST_DEVICE inline
+   Scalar& operator()(Args... args) const
+   {
+      static_assert(sizeof...(args) == Dim, "Wrong number of arguments");
+      return data[ TensorInd<1, Dim, Args...>::result(sizes, args...) ];
+   }
+
+   MFEM_ATTR_HOST_DEVICE inline Scalar& operator[](int i) const
+   {
+      return data[i];
+   }
+};
+
+typedef DeviceTensor<1,int> DeviceArray;
+typedef DeviceTensor<1,double> DeviceVector;
+typedef DeviceTensor<2,double> DeviceMatrix;
+
+} // mfem namespace
+
+#endif // MFEM_DTENSOR
@@ -78,7 +78,8 @@ void OperatorHandle::MakeSquareBlockDiag(MPI_Comm comm, HYPRE_Int glob_size,
      case Operator::PETSC_MATAIJ:
      case Operator::PETSC_MATIS:
         // Assuming that PetscInt is the same size as HYPRE_Int, checked above.
-         oper = new PetscParMatrix(comm, glob_size, row_starts, diag, type_id);
+         oper = new PetscParMatrix(comm, glob_size, (PetscInt*)row_starts, diag,
+                                   type_id);
         break;
 #endif
      default: MFEM_ABORT(not_supported_msg << type_id);
@@ -110,7 +111,7 @@ MakeRectangularBlockDiag(MPI_Comm comm, HYPRE_Int glob_num_rows,
      case Operator::PETSC_MATIS:
         // Assuming that PetscInt is the same size as HYPRE_Int, checked above.
         oper = new PetscParMatrix(comm, glob_num_rows, glob_num_cols,
-                                   row_starts, col_starts, diag, type_id);
+                                   (PetscInt*)row_starts, (PetscInt*)col_starts, diag, type_id);
         break;
 #endif
      default: MFEM_ABORT(not_supported_msg << type_id);
@@ -81,6 +81,12 @@ public:
   /// Access the underlying Operator pointer.
   Operator *Ptr() const { return oper; }

+   /// Support the use of -> to call methods of the underlying Operator.
+   Operator *operator->() const { return oper; }
+
+   /// Access the underlying Operator.
+   Operator &operator*() { return *oper; }
+
   /// Get the currently set operator type id.
   Operator::Type Type() const { return type_id; }

@@ -187,6 +193,10 @@ public:
                    const Vector &X, Vector &B) const;
 };

+
+/// Add an alternative name for OperatorHandle -- OperatorPtr.
+typedef OperatorHandle OperatorPtr;
+
 } // namespace mfem

 #endif
@@ -79,7 +79,7 @@ template<typename TargetT, typename SourceT>
 static TargetT *DuplicateAs(const SourceT *array, int size,
                            bool cplusplus = true)
 {
-   TargetT *target_array = cplusplus ? new TargetT[size]
+   TargetT *target_array = cplusplus ? mfem::New<TargetT>(size)
                           /*     */ : mfem_hypre_TAlloc(TargetT, size);
   for (int i = 0; i < size; i++)
   {
@@ -641,13 +641,13 @@ HypreParMatrix::HypreParMatrix(MPI_Comm comm, int id, int np,

   HYPRE_Int i;

-   double *a_diag = new double[diag_nnz];
+   double *a_diag = mfem::New<double>(diag_nnz);
   for (i = 0; i < diag_nnz; i++)
   {
      a_diag[i] = 1.0;
   }

-   double *a_offd = new double[offd_nnz];
+   double *a_offd = mfem::New<double>(offd_nnz);
   for (i = 0; i < offd_nnz; i++)
   {
      a_offd[i] = 1.0;
@@ -1010,6 +1010,7 @@ HYPRE_Int HypreParMatrix::Mult(HypreParVector &x, HypreParVector &y,

 void HypreParMatrix::Mult(double a, const Vector &x, double b, Vector &y) const
 {
+   x.Pull();
   MFEM_ASSERT(x.Size() == Width(), "invalid x.Size() = " << x.Size()
               << ", expected size = " << Width());
   MFEM_ASSERT(y.Size() == Height(), "invalid y.Size() = " << y.Size()
@@ -1033,11 +1034,13 @@ void HypreParMatrix::Mult(double a, const Vector &x, double b, Vector &y) const
   }

   hypre_ParCSRMatrixMatvec(a, A, *X, b, *Y);
+   y.Push();
 }

 void HypreParMatrix::MultTranspose(double a, const Vector &x,
                                   double b, Vector &y) const
 {
+   x.Pull();
   MFEM_ASSERT(x.Size() == Height(), "invalid x.Size() = " << x.Size()
               << ", expected size = " << Height());
   MFEM_ASSERT(y.Size() == Width(), "invalid y.Size() = " << y.Size()
@@ -1063,6 +1066,7 @@ void HypreParMatrix::MultTranspose(double a, const Vector &x,
   }

   hypre_ParCSRMatrixMatvecT(a, A, *Y, b, *X);
+   y.Push();
 }

 HYPRE_Int HypreParMatrix::Mult(HYPRE_ParVector x, HYPRE_ParVector y,
@@ -1477,14 +1481,14 @@ void HypreParMatrix::Destroy()
   {
      if (diagOwner & 1)
      {
-         delete [] hypre_CSRMatrixI(A->diag);
-         delete [] hypre_CSRMatrixJ(A->diag);
+         mfem::Delete(hypre_CSRMatrixI(A->diag));
+         mfem::Delete(hypre_CSRMatrixJ(A->diag));
      }
      hypre_CSRMatrixI(A->diag) = NULL;
      hypre_CSRMatrixJ(A->diag) = NULL;
      if (diagOwner & 2)
      {
-         delete [] hypre_CSRMatrixData(A->diag);
+         mfem::Delete(hypre_CSRMatrixData(A->diag));
      }
      hypre_CSRMatrixData(A->diag) = NULL;
   }
@@ -1492,14 +1496,14 @@ void HypreParMatrix::Destroy()
   {
      if (offdOwner & 1)
      {
-         delete [] hypre_CSRMatrixI(A->offd);
-         delete [] hypre_CSRMatrixJ(A->offd);
+         mfem::Delete(hypre_CSRMatrixI(A->offd));
+         mfem::Delete(hypre_CSRMatrixJ(A->offd));
      }
      hypre_CSRMatrixI(A->offd) = NULL;
      hypre_CSRMatrixJ(A->offd) = NULL;
      if (offdOwner & 2)
      {
-         delete [] hypre_CSRMatrixData(A->offd);
+         mfem::Delete(hypre_CSRMatrixData(A->offd));
      }
      hypre_CSRMatrixData(A->offd) = NULL;
   }
@@ -1507,7 +1511,7 @@ void HypreParMatrix::Destroy()
   {
      if (colMapOwner & 1)
      {
-         delete [] hypre_ParCSRMatrixColMapOffd(A);
+         mfem::Delete(hypre_ParCSRMatrixColMapOffd(A));
      }
      hypre_ParCSRMatrixColMapOffd(A) = NULL;
   }
@@ -3328,7 +3332,7 @@ HypreLOBPCG::SetPreconditioner(Solver & precond)
 void
 HypreLOBPCG::SetOperator(Operator & A)
 {
-   int locSize = A.Width();
+   HYPRE_Int locSize = A.Width();

   if (HYPRE_AssumedPartitionCheck())
   {
@@ -3344,7 +3348,7 @@ HypreLOBPCG::SetOperator(Operator & A)
   {
      part = new HYPRE_Int[numProcs+1];

-      MPI_Allgather(&locSize, 1, MPI_INT,
+      MPI_Allgather(&locSize, 1, HYPRE_MPI_INT,
                    &part[1], 1, HYPRE_MPI_INT, comm);

      part[0] = 0;
@@ -1520,10 +1520,14 @@ hypre_ParCSRMatrixSum(hypre_ParCSRMatrix *A,
   hypre_CSRMatrix *A_offd = hypre_ParCSRMatrixOffd(A);
   hypre_CSRMatrix *B_diag = hypre_ParCSRMatrixDiag(B);
   hypre_CSRMatrix *B_offd = hypre_ParCSRMatrixOffd(B);
+   HYPRE_Int ncols_B_offd  = hypre_CSRMatrixNumCols(B_offd);
   HYPRE_Int error;

   error = hypre_CSRMatrixSum(A_diag, beta, B_diag);
-   error = error ? error : hypre_CSRMatrixSum(A_offd, beta, B_offd);
+   if (ncols_B_offd > 0) /* treat B_offd as zero if it has no columns */
+   {
+      error = error ? error : hypre_CSRMatrixSum(A_offd, beta, B_offd);
+   }

   return error;
 }
@@ -561,6 +561,102 @@ void GeneralizedAlphaSolver::Step(Vector &x, double &t, double &dt)
 }


+void IMEX_BE_FE::Init(TimeDependentOperator &_f)
+{
+   ODESolver::Init(_f);
+   k_imp.SetSize(f->Width());
+   y.SetSize(f->Width());
+   k_exp.SetSize(f->Width());
+}
+
+void IMEX_BE_FE::Step(Vector &x, double &t, double &dt)
+{
+   f->ExplicitMult(x, k_exp);
+   add(x, dt, k_exp, y);
+
+   f->SetTime(t + dt);
+   f->ImplicitSolve(dt, y, k_imp);
+
+   x.Add(dt, k_exp);
+   x.Add(dt, k_imp);
+   t += dt;
+}
+
+
+void IMEXRK2::Init(TimeDependentOperator &_f)
+{
+   ODESolver::Init(_f);
+   f = ODESolver::f;
+   k_imp.SetSize(f->Width());
+   k_exp.SetSize(f->Width());
+   y.SetSize(f->Width());
+   z.SetSize(f->Width());
+}
+
+void IMEXRK2::Step(Vector &x, double &t, double &dt)
+{
+   double gamma = 1 - sqrt(2)/2;
+   double delta = -2*sqrt(2)/3;
+
+   // The method is given by
+   // k1_exp = f(u)
+   // k1_imp = g(u + gamma*dt*k1_exp + gamma*dt*k1_imp)
+   // k2_exp = f(u + gamma*dt*k1_exp + gamma*dt*k1_imp)
+   // k2_imp = g(u + delta*dt*k1_exp + (1-gamma)*dt*k1_imp
+   //              + (1-delta)*dt*k2_exp + gamma*dt*k2_imp)
+   // k3_exp = f(u + delta*dt*k1_exp + (1-gamma)*dt*k1_imp
+   //              + (1-delta)*dt*k2_exp + gamma*dt*k2_imp)
+   // u_new = u + dt*((1-gamma)*k1_imp + (1-gamma)*k2_exp
+   //                 + gamma*k2_imp + gamma*k3_exp)
+
+   // Take first explicit step
+   // k1_exp = f(u)
+   f->ExplicitMult(x, k_exp);
+   // b corresponding to this stage is zero, so don't add to solution
+
+   // Solve first implicit step
+   // y = u + gamma*dt*k1_exp
+   add(x, gamma*dt, k_exp, y);
+   // Solve x1_imp = g(u + gamma*dt*k1_exp + gamma*dt*k1_imp)
+   f->SetTime(t + gamma*dt);
+   f->ImplicitSolve(gamma*dt, y, k_imp);
+   // x = u + (1-gamma)*dt*k1_imp
+   x.Add((1-gamma)*dt, k_imp);
+
+   // Begin setting up rhs for second solve
+   // z = u + (1-gamma)*dt*k_imp + delta*dt*k_exp
+   add(x, delta*dt, k_exp, z);
+
+   // Take second explicit step
+   // y = x + gamma*dt*k1_exp + gamma*dt*k1_imp
+   y.Add(gamma*dt, k_imp);
+   // k2_exp = f(x + gamma*dt*k1_exp + gamma*dt*k1_imp)
+   f->ExplicitMult(y, k_exp);
+   // x = u + (1-gamma)*dt*k1_imp + (1-gamma)*dt*k2_exp
+   x.Add((1-gamma)*dt, k_exp);
+
+   // Finish formoing rhs
+   // z = x + (1-gamma)*dt*k1_imp + delta*dt*k1_exp + (1-delta)*dt*k2_exp
+   z.Add((1-delta)*dt, k_exp);
+
+   // Solve second implicit step for k2_imp
+   f->SetTime(t + dt);
+   f->ImplicitSolve(gamma*dt, z, k_imp);
+   // x = u + (1-gamma)*dt*k1_imp + (1-gamma)*dt*k2_exp + gamma*dt*k2_imp
+   x.Add(gamma*dt, k_imp);
+
+   // Take final explicit step for k3_exp
+   z.Add(gamma*dt, k_imp);
+   f->ExplicitMult(z, k_exp);
+
+   // x = u + (1-gamma)*dt*k1_imp + (1-gamma)*dt*k2_exp + gamma*dt*k2_imp
+   //       + gamma*dt*k3_exp
+   x.Add(gamma*dt, k_exp);
+
+   t += dt;
+}
+
+
 void
 SIASolver::Init(Operator &P, TimeDependentOperator & F)
 {
@@ -305,6 +305,33 @@ public:
 };


+/// IMEX Backward-Forward Euler ODE solver
+class IMEX_BE_FE : public ODESolver
+{
+protected:
+   Vector k_exp, k_imp, y;
+
+public:
+   virtual void Init(TimeDependentOperator &_f);
+
+   virtual void Step(Vector &x, double &t, double &dt);
+};
+
+/** Second-order IMEX (2,3,2) method, from "Implicit-explicit Runge-Kutta
+    methods for time-dependent partial differential equations" by Ascher, Ruuth
+    and Spiteri, Applied Numerical Mathematics (1997). */
+class IMEXRK2 : public ODESolver
+{
+protected:
+   Vector k_exp, k_imp, y, z;
+
+public:
+   virtual void Init(TimeDependentOperator &_f);
+
+   virtual void Step(Vector &x, double &t, double &dt);
+};
+
+
 /// The SIASolver class is based on the Symplectic Integration Algorithm
 /// described in "A Symplectic Integration Algorithm for Separable Hamiltonian
 /// Functions" by J. Candy and W. Rozmus, Journal of Computational Physics,
@@ -10,7 +10,9 @@
 // Software Foundation) version 2.1 dated February 1999.

 #include "vector.hpp"
+#include "dtensor.hpp"
 #include "operator.hpp"
+#include "../general/forall.hpp"

 #include <iostream>
 #include <iomanip>
@@ -163,25 +165,31 @@ ConstrainedOperator::ConstrainedOperator(Operator *A, const Array<int> &list,
 void ConstrainedOperator::EliminateRHS(const Vector &x, Vector &b) const
 {
   w = 0.0;
-
-   for (int i = 0; i < constraint_list.Size(); i++)
+   const int csz = constraint_list.Size();
+   const DeviceArray idx(constraint_list, csz);
+   const DeviceVector d_x(x, x.Size());
+   DeviceVector d_w(w, w.Size());
+   MFEM_FORALL(i, csz,
   {
-      w(constraint_list[i]) = x(constraint_list[i]);
-   }
+      const int id = idx[i];
+      d_w[id] = d_x[id];
+   });

   A->Mult(w, z);

   b -= z;
-
-   for (int i = 0; i < constraint_list.Size(); i++)
+   DeviceVector d_b(b, b.Size());
+   MFEM_FORALL(i, csz,
   {
-      b(constraint_list[i]) = x(constraint_list[i]);
-   }
+      const int id = idx[i];
+      d_b[id] = d_x[id];
+   });
 }

 void ConstrainedOperator::Mult(const Vector &x, Vector &y) const
 {
-   if (constraint_list.Size() == 0)
+   const int csz = constraint_list.Size();
+   if (csz == 0)
   {
      A->Mult(x, y);
      return;
@@ -189,17 +197,19 @@ void ConstrainedOperator::Mult(const Vector &x, Vector &y) const

   z = x;

-   for (int i = 0; i < constraint_list.Size(); i++)
-   {
-      z(constraint_list[i]) = 0.0;
-   }
+   const DeviceArray idx(constraint_list, csz);
+   DeviceVector d_z(z, z.Size());
+   MFEM_FORALL(i, csz, d_z[idx[i]] = 0.0;);

   A->Mult(z, y);

-   for (int i = 0; i < constraint_list.Size(); i++)
+   const DeviceVector d_x(x, x.Size());
+   DeviceVector d_y(y, y.Size());
+   MFEM_FORALL(i, csz,
   {
-      y(constraint_list[i]) = x(constraint_list[i]);
-   }
+      const int id = idx[i];
+      d_y[id] = d_x[id];
+   });
 }

 }
@@ -296,6 +296,9 @@ public:

   /// Operator application
   virtual void Mult(const Vector &x, Vector &y) const { y = x; }
+
+   /// Application of the transpose
+   virtual void MultTranspose(const Vector &x, Vector &y) const { y = x; }
 };


@@ -708,15 +708,21 @@ BlockDiagonalConstructor(MPI_Comm comm,
      // Copy SparseMatrix into PETSc SeqAIJ format
      Mat lA;
      ierr = MatISGetLocalMat(A,&lA); PCHKERRQ(A,ierr);
-      if (sizeof(PetscInt) == sizeof(int))
-      {
-         ierr = MatSeqAIJSetPreallocationCSR(lA,diag->GetI(),diag->GetJ(),
-                                             diag->GetData()); PCHKERRQ(lA,ierr);
-      }
-      else
-      {
-         MFEM_ABORT("64bit indices not yet supported");
-      }
+      int *II = diag->GetI();
+      int *JJ = diag->GetJ();
+#if defined(PETSC_USE_64BIT_INDICES)
+      PetscInt *pII,*pJJ;
+      int m = diag->Height()+1, nnz = II[diag->Height()];
+      ierr = PetscMalloc2(m,&pII,nnz,&pJJ); PCHKERRQ(lA,ierr);
+      for (int i = 0; i < m; i++) { pII[i] = II[i]; }
+      for (int i = 0; i < nnz; i++) { pJJ[i] = JJ[i]; }
+      ierr = MatSeqAIJSetPreallocationCSR(lA,pII,pJJ,
+                                          diag->GetData()); PCHKERRQ(lA,ierr);
+      ierr = PetscFree2(pII,pJJ); PCHKERRQ(lA,ierr);
+#else
+      ierr = MatSeqAIJSetPreallocationCSR(lA,II,JJ,
+                                          diag->GetData()); PCHKERRQ(lA,ierr);
+#endif
   }
   else
   {
@@ -736,13 +742,16 @@ BlockDiagonalConstructor(MPI_Comm comm,
         CCHKERRQ(PETSC_COMM_SELF,ierr);
         ierr = PetscMemcpy(djj,diag->GetJ(),nnz*sizeof(PetscInt));
         CCHKERRQ(PETSC_COMM_SELF,ierr);
-         ierr = PetscMemcpy(da,diag->GetData(),nnz*sizeof(PetscScalar));
-         CCHKERRQ(PETSC_COMM_SELF,ierr);
      }
      else
      {
-         MFEM_ABORT("64bit indices not yet supported");
+         int *iii = diag->GetI();
+         int *jjj = diag->GetJ();
+         for (int i = 0; i < m; i++) { dii[i] = iii[i]; }
+         for (int i = 0; i < nnz; i++) { djj[i] = jjj[i]; }
      }
+      ierr = PetscMemcpy(da,diag->GetData(),nnz*sizeof(PetscScalar));
+      CCHKERRQ(PETSC_COMM_SELF,ierr);
      ierr = PetscCalloc1(m,&oii);
      CCHKERRQ(PETSC_COMM_SELF,ierr);
      if (commsize > 1)
@@ -1051,7 +1060,7 @@ void PetscParMatrix::ConvertOperator(MPI_Comm comm, const Operator &op, Mat* A,
         ierr = MatConvert(*A,MATIS,MAT_INPLACE_MATRIX,A); CCHKERRQ(comm,ierr);

         mfem::Array<Mat> *vmatsl2l = new mfem::Array<Mat>(nr);
-         for (PetscInt i=0; i<nr; i++) { (*vmatsl2l)[i] = matsl2l[i]; }
+         for (int i=0; i<(int)nr; i++) { (*vmatsl2l)[i] = matsl2l[i]; }
         ierr = PetscFree(matsl2l); CCHKERRQ(PETSC_COMM_SELF,ierr);

         PetscContainer c;
@@ -2150,7 +2159,7 @@ void PetscBCHandler::ApplyBC(const Vector &x, Vector &y)
   y = x;
   if (bctype == ZERO)
   {
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
      {
         y[ess_tdof_list[i]] = 0.0;
      }
@@ -2162,7 +2171,7 @@ void PetscBCHandler::ApplyBC(const Vector &x, Vector &y)
         Eval(eval_t,eval_g);
         eval_t_cached = eval_t;
      }
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
      {
         y[ess_tdof_list[i]] = eval_g[ess_tdof_list[i]];
      }
@@ -2174,7 +2183,7 @@ void PetscBCHandler::ApplyBC(Vector &x)
   (*this).SetUp(x.Size());
   if (bctype == ZERO)
   {
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
      {
         x[ess_tdof_list[i]] = 0.0;
      }
@@ -2186,7 +2195,7 @@ void PetscBCHandler::ApplyBC(Vector &x)
         Eval(eval_t,eval_g);
         eval_t_cached = eval_t;
      }
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
      {
         x[ess_tdof_list[i]] = eval_g[ess_tdof_list[i]];
      }
@@ -2198,14 +2207,14 @@ void PetscBCHandler::FixResidualBC(const Vector& x, Vector& y)
   (*this).SetUp(x.Size());
   if (bctype == ZERO)
   {
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
      {
         y[ess_tdof_list[i]] = x[ess_tdof_list[i]];
      }
   }
   else
   {
-      for (PetscInt i = 0; i < ess_tdof_list.Size(); ++i)
+      for (int i = 0; i < ess_tdof_list.Size(); ++i)
      {
         y[ess_tdof_list[i]] = x[ess_tdof_list[i]] - eval_g[ess_tdof_list[i]];
      }
@@ -3875,7 +3884,7 @@ static PetscErrorCode __mfem_snes_postcheck(SNESLineSearch ls,Vec X,Vec Y,Vec W,
                                            PetscBool *cy,PetscBool *cw, void* ctx)
 {
   __mfem_snes_ctx* snes_ctx = (__mfem_snes_ctx*)ctx;
-   bool lcy,lcw;
+   bool lcy = false,lcw = false;

   PetscFunctionBeginUser;
   mfem::PetscParVector x(X,true);
@@ -4222,7 +4231,21 @@ static PetscErrorCode Convert_Vmarks_IS(MPI_Comm comm,
                         (const PetscInt**)&jj,&done); CHKERRQ(ierr);
      MFEM_VERIFY(done,"Unable to perform MatGetRowIJ on " << i << " l2l matrix");
      ierr = MatGetSize(pl2l[i],NULL,&n); CHKERRQ(ierr);
+#if defined(PETSC_USE_64BIT_INDICES)
+      int  nnz = (int)ii[m];
+      int *mii = new int[m+1];
+      int *mjj = new int[nnz];
+      for (int j = 0; j < m+1; j++) { mii[j] = (int)ii[j]; }
+      for (int j = 0; j < nnz; j++) { mjj[j] = (int)jj[j]; }
+      l2l[i] = new mfem::SparseMatrix(mii,mjj,NULL,m,n,true,true,true);
+#else
      l2l[i] = new mfem::SparseMatrix(ii,jj,NULL,m,n,false,true,true);
+#endif
+      ierr = MatRestoreRowIJ(pl2l[i],0,PETSC_FALSE,PETSC_FALSE,&m,
+                             (const PetscInt**)&ii,
+                             (const PetscInt**)&jj,&done); CHKERRQ(ierr);
+      MFEM_VERIFY(done,"Unable to perform MatRestoreRowIJ on "
+                  << i << " l2l matrix");
   }
   nl = 0;
   for (int i = 0; i < l2l.Size(); i++) { nl += l2l[i]->Width(); }
@@ -4242,14 +4265,6 @@ static PetscErrorCode Convert_Vmarks_IS(MPI_Comm comm,
   ierr = Convert_Array_IS(comm,false,&sub_dof_marker,st,is); CCHKERRQ(comm,ierr);
   for (int i = 0; i < pl2l.Size(); i++)
   {
-      PetscInt  m = l2l[i]->Height();
-      PetscInt  *ii = l2l[i]->GetI(),*jj = l2l[i]->GetJ();
-      PetscBool done;
-      ierr = MatRestoreRowIJ(pl2l[i],0,PETSC_FALSE,PETSC_FALSE,&m,
-                             (const PetscInt**)&ii,
-                             (const PetscInt**)&jj,&done); CHKERRQ(ierr);
-      MFEM_VERIFY(done,"Unable to perform MatRestoreRowIJ on "
-                  << i << " l2l matrix");
      delete l2l[i];
   }
   PetscFunctionReturn(0);
@@ -12,6 +12,8 @@
 // Implementation of sparse matrix

 #include "linalg.hpp"
+#include "dtensor.hpp"
+#include "../general/forall.hpp"
 #include "../general/table.hpp"
 #include "../general/sort_pairs.hpp"

@@ -36,6 +38,7 @@ SparseMatrix::SparseMatrix(int nrows, int ncols)
     current_row(-1),
     ColPtrJ(NULL),
     ColPtrNode(NULL),
+     At(NULL),
     ownGraph(true),
     ownData(true),
     isSorted(false)
@@ -58,6 +61,7 @@ SparseMatrix::SparseMatrix(int *i, int *j, double *data, int m, int n)
     Rows(NULL),
     ColPtrJ(NULL),
     ColPtrNode(NULL),
+     At(NULL),
     ownGraph(true),
     ownData(true),
     isSorted(false)
@@ -76,6 +80,7 @@ SparseMatrix::SparseMatrix(int *i, int *j, double *data, int m, int n,
     Rows(NULL),
     ColPtrJ(NULL),
     ColPtrNode(NULL),
+     At(NULL),
     ownGraph(ownij),
     ownData(owna),
     isSorted(issorted)
@@ -88,7 +93,7 @@ SparseMatrix::SparseMatrix(int *i, int *j, double *data, int m, int n,
   {
      ownData = true;
      int nnz = I[height];
-      A = new double[ nnz ];
+      A = mfem::New<double>(nnz);
      for (int i=0; i<nnz; ++i)
      {
         A[i] = 0.0;
@@ -101,6 +106,7 @@ SparseMatrix::SparseMatrix(int nrows, int ncols, int rowsize)
   , Rows(NULL)
   , ColPtrJ(NULL)
   , ColPtrNode(NULL)
+   , At(NULL)
   , ownGraph(true)
   , ownData(true)
   , isSorted(false)
@@ -108,9 +114,9 @@ SparseMatrix::SparseMatrix(int nrows, int ncols, int rowsize)
 #ifdef MFEM_USE_MEMALLOC
   NodesMem = NULL;
 #endif
-   I = new int[nrows + 1];
-   J = new int[nrows * rowsize];
-   A = new double[nrows * rowsize];
+   I = mfem::New<int>(nrows + 1);
+   J = mfem::New<int>(nrows * rowsize);
+   A = mfem::New<double>(nrows * rowsize);

   for (int i = 0; i <= nrows; i++)
   {
@@ -126,8 +132,8 @@ SparseMatrix::SparseMatrix(const SparseMatrix &mat, bool copy_graph)
      const int nnz = mat.I[height];
      if (copy_graph)
      {
-         I = new int[height+1];
-         J = new int[nnz];
+         I = mfem::New<int>(height+1);
+         J = mfem::New<int>(nnz);
         memcpy(I, mat.I, sizeof(int)*(height+1));
         memcpy(J, mat.J, sizeof(int)*nnz);
         ownGraph = true;
@@ -138,7 +144,7 @@ SparseMatrix::SparseMatrix(const SparseMatrix &mat, bool copy_graph)
         J = mat.J;
         ownGraph = false;
      }
-      A = new double[nnz];
+      A = mfem::New<double>(nnz);
      memcpy(A, mat.A, sizeof(double)*nnz);
      ownData = true;

@@ -181,6 +187,7 @@ SparseMatrix::SparseMatrix(const SparseMatrix &mat, bool copy_graph)
   current_row = -1;
   ColPtrJ = NULL;
   ColPtrNode = NULL;
+   At = NULL;
   isSorted = mat.isSorted;
 }

@@ -189,6 +196,7 @@ SparseMatrix::SparseMatrix(const Vector &v)
   , Rows(NULL)
   , ColPtrJ(NULL)
   , ColPtrNode(NULL)
+   , At(NULL)
   , ownGraph(true)
   , ownData(true)
   , isSorted(true)
@@ -196,9 +204,9 @@ SparseMatrix::SparseMatrix(const Vector &v)
 #ifdef MFEM_USE_MEMALLOC
   NodesMem = NULL;
 #endif
-   I = new int[height + 1];
-   J = new int[height];
-   A = new double[height];
+   I = mfem::New<int>(height + 1);
+   J = mfem::New<int>(height);
+   A = mfem::New<double>(height);

   for (int i = 0; i <= height; i++)
   {
@@ -243,6 +251,7 @@ void SparseMatrix::SetEmpty()
   current_row = -1;
   ColPtrJ = NULL;
   ColPtrNode = NULL;
+   At = NULL;
 #ifdef MFEM_USE_MEMALLOC
   NodesMem = NULL;
 #endif
@@ -331,7 +340,7 @@ void SparseMatrix::SetWidth(int newWidth)
      // Nothing to be done here
      return;
   }
-   else if ( newWidth == -1)
+   else if (newWidth == -1)
   {
      // Compute the actual width
      width = ActualWidth();
@@ -550,12 +559,10 @@ void SparseMatrix::Mult(const Vector &x, Vector &y) const

 void SparseMatrix::AddMult(const Vector &x, Vector &y, const double a) const
 {
-   MFEM_ASSERT(width == x.Size(),
-               "Input vector size (" << x.Size() << ") must match matrix width (" << width
-               << ")");
-   MFEM_ASSERT(height == y.Size(),
-               "Output vector size (" << y.Size() << ") must match matrix height (" << height
-               << ")");
+   MFEM_ASSERT(width == x.Size(), "Input vector size (" << x.Size()
+               << ") must match matrix width (" << width << ")");
+   MFEM_ASSERT(height == y.Size(), "Output vector size (" << y.Size()
+               << ") must match matrix height (" << height << ")");

   int i, j, end;
   double *Ap = A, *yp = y.GetData();
@@ -582,16 +589,22 @@ void SparseMatrix::AddMult(const Vector &x, Vector &y, const double a) const

   if (a == 1.0)
   {
-#ifndef MFEM_USE_OPENMP
-      for (i = j = 0; i < height; i++)
+#ifndef MFEM_USE_LEGACY_OPENMP
+      const DeviceArray d_I(I);
+      const DeviceArray d_J(J);
+      const DeviceVector d_A(A);
+      const DeviceVector d_x(x, x.Size());
+      DeviceVector d_y(y, y.Size());
+      MFEM_FORALL(i, height,
      {
         double d = 0.0;
-         for (end = Ip[i+1]; j < end; j++)
+         const int end = d_I[i+1];
+         for (int j=d_I[i]; j < end; j++)
         {
-            d += Ap[j] * xp[Jp[j]];
+            d += d_A[j] * d_x[d_J[j]];
         }
-         yp[i] += d;
-      }
+         d_y[i] += d;
+      });
 #else
      #pragma omp parallel for private(j,end)
      for (i = 0; i < height; i++)
@@ -628,20 +641,16 @@ void SparseMatrix::MultTranspose(const Vector &x, Vector &y) const
 void SparseMatrix::AddMultTranspose(const Vector &x, Vector &y,
                                    const double a) const
 {
-   MFEM_ASSERT(height == x.Size(),
-               "Input vector size (" << x.Size() << ") must match matrix height (" << height
-               << ")");
-   MFEM_ASSERT(width == y.Size(),
-               "Output vector size (" << y.Size() << ") must match matrix width (" << width
-               << ")");
-
-   int i, j, end;
-   double *yp = y.GetData();
+   MFEM_ASSERT(height == x.Size(), "Input vector size (" << x.Size()
+               << ") must match matrix height (" << height << ")");
+   MFEM_ASSERT(width == y.Size(), "Output vector size (" << y.Size()
+               << ") must match matrix width (" << width << ")");

   if (A == NULL)
   {
+      double *yp = y.GetData();
      // The matrix is not finalized, but multiplication is still possible
-      for (i = 0; i < height; i++)
+      for (int i = 0; i < height; i++)
      {
         RowNode *row = Rows[i];
         double b = a * x(i);
@@ -653,17 +662,41 @@ void SparseMatrix::AddMultTranspose(const Vector &x, Vector &y,
      return;
   }

-   for (i = 0; i < height; i++)
+   if (At)
   {
-      double xi = a * x(i);
-      end = I[i+1];
-      for (j = I[i]; j < end; j++)
+      At->AddMult(x, y, a);
+   }
+   else
+   {
+      MFEM_VERIFY(Device::IsDisabled(), "transpose action on device is not "
+                  "enabled; see BuildTranspose() for details.");
+      for (int i = 0; i < height; i++)
      {
-         yp[J[j]] += A[j]*xi;
+         const double xi = a * x[i];
+         const int end = I[i+1];
+         for (int j = I[i]; j < end; j++)
+         {
+            const int Jj = J[j];
+            y[Jj] += A[j] * xi;
+         }
      }
   }
 }

+void SparseMatrix::BuildTranspose() const
+{
+   if (At == NULL)
+   {
+      At = Transpose(*this);
+   }
+}
+
+void SparseMatrix::ResetTranspose() const
+{
+   delete At;
+   At = NULL;
+}
+
 void SparseMatrix::PartMult(
   const Array<int> &rows, const Vector &x, Vector &y) const
 {
@@ -709,18 +742,26 @@ void SparseMatrix::BooleanMult(const Array<int> &x, Array<int> &y) const
   y.SetSize(Height());
   y = 0;

-   for (int i = 0; i < Height(); i++)
+   const int height = Height();
+   const int *d_I = mfem::Ptr(I);
+   const int *d_J = mfem::Ptr(J);
+   const DeviceArray d_x(x, x.Size());
+   DeviceArray d_y(y, y.Size());
+   MFEM_FORALL(i, height,
   {
-      int end = I[i+1];
-      for (int j = I[i]; j < end; j++)
+      const int end = d_I[i+1];
+      for (int j = d_I[i]; j < end; j++)
      {
-         if (x[J[j]])
+         if (d_x[d_J[j]])
         {
-            y[i] = x[J[j]];
+            d_y[i] = d_x[d_J[j]];
            break;
         }
      }
-   }
+   });
+   // Sync on host for some verifications in
+   // pfespace GetEssentialTrueDofs and MarkerToList
+   mfem::Pull(y);
 }

 void SparseMatrix::BooleanMultTranspose(const Array<int> &x,
@@ -824,7 +865,7 @@ void SparseMatrix::Finalize(int skip_zeros, bool fix_empty_rows)
   delete [] ColPtrNode;
   ColPtrNode = NULL;

-   I = new int[height+1];
+   I = mfem::New<int>(height+1);
   I[0] = 0;
   for (i = 1; i <= height; i++)
   {
@@ -839,8 +880,8 @@ void SparseMatrix::Finalize(int skip_zeros, bool fix_empty_rows)
   }

   nz = I[height];
-   J = new int[nz];
-   A = new double[nz];
+   J = mfem::New<int>(nz);
+   A = mfem::New<double>(nz);
   // Assume we're sorted until we find out otherwise
   isSorted = true;
   for (j = i = 0; i < height; i++)
@@ -901,7 +942,7 @@ void SparseMatrix::GetBlocks(Array2D<SparseMatrix *> &blocks) const
   {
      for (int i = 0; i < br; i++)
      {
-         int *bI = new int[nr + 1];
+         int *bI = mfem::New<int>(nr + 1);
         for (int k = 0; k <= nr; k++)
         {
            bI[k] = 0;
@@ -945,8 +986,8 @@ void SparseMatrix::GetBlocks(Array2D<SparseMatrix *> &blocks) const
         {
            rs = b.I[k], b.I[k] = nnz, nnz += rs;
         }
-         b.J = new int[nnz];
-         b.A = new double[nnz];
+         b.J = mfem::New<int>(nnz);
+         b.A = mfem::New<double>(nnz);
      }
   }

@@ -2080,12 +2121,12 @@ void SparseMatrix::Set(const int i, const int j, const double A)
   if ((gi=i) < 0) { gi = -1-gi, s = -1; }
   else { s = 1; }
   MFEM_ASSERT(gi < height,
-               "Trying to insert a row " << gi << " outside the matrix height "
+               "Trying to set a row " << gi << " outside the matrix height "
               << height);
   if ((gj=j) < 0) { gj = -1-gj, t = -s; }
   else { t = s; }
   MFEM_ASSERT(gj < width,
-               "Trying to insert a column " << gj << " outside the matrix width "
+               "Trying to set a column " << gj << " outside the matrix width "
               << width);
   if (t < 0) { a = -a; }
   _Set_(gi, gj, a);
@@ -2121,7 +2162,7 @@ void SparseMatrix::SetSubMatrix(const Array<int> &rows, const Array<int> &cols,
      if ((gi=rows[i]) < 0) { gi = -1-gi, s = -1; }
      else { s = 1; }
      MFEM_ASSERT(gi < height,
-                  "Trying to insert a row " << gi << " outside the matrix height "
+                  "Trying to set a row " << gi << " outside the matrix height "
                  << height);
      SetColPtr(gi);
      for (j = 0; j < cols.Size(); j++)
@@ -2134,7 +2175,7 @@ void SparseMatrix::SetSubMatrix(const Array<int> &rows, const Array<int> &cols,
         if ((gj=cols[j]) < 0) { gj = -1-gj, t = -s; }
         else { t = s; }
         MFEM_ASSERT(gj < width,
-                     "Trying to insert a column " << gj << " outside the matrix width "
+                     "Trying to set a column " << gj << " outside the matrix width "
                     << width);
         if (t < 0) { a = -a; }
         _Set_(gj, a);
@@ -2156,7 +2197,7 @@ void SparseMatrix::SetSubMatrixTranspose(const Array<int> &rows,
      if ((gi=rows[i]) < 0) { gi = -1-gi, s = -1; }
      else { s = 1; }
      MFEM_ASSERT(gi < height,
-                  "Trying to insert a row " << gi << " outside the matrix height "
+                  "Trying to set a row " << gi << " outside the matrix height "
                  << height);
      SetColPtr(gi);
      for (j = 0; j < cols.Size(); j++)
@@ -2169,7 +2210,7 @@ void SparseMatrix::SetSubMatrixTranspose(const Array<int> &rows,
         if ((gj=cols[j]) < 0) { gj = -1-gj, t = -s; }
         else { t = s; }
         MFEM_ASSERT(gj < width,
-                     "Trying to insert a column " << gj << " outside the matrix width "
+                     "Trying to set a column " << gj << " outside the matrix width "
                     << width);
         if (t < 0) { a = -a; }
         _Set_(gj, a);
@@ -2189,7 +2230,7 @@ void SparseMatrix::GetSubMatrix(const Array<int> &rows, const Array<int> &cols,
      if ((gi=rows[i]) < 0) { gi = -1-gi, s = -1; }
      else { s = 1; }
      MFEM_ASSERT(gi < height,
-                  "Trying to insert a row " << gi << " outside the matrix height "
+                  "Trying to read a row " << gi << " outside the matrix height "
                  << height);
      SetColPtr(gi);
      for (j = 0; j < cols.Size(); j++)
@@ -2197,7 +2238,7 @@ void SparseMatrix::GetSubMatrix(const Array<int> &rows, const Array<int> &cols,
         if ((gj=cols[j]) < 0) { gj = -1-gj, t = -s; }
         else { t = s; }
         MFEM_ASSERT(gj < width,
-                     "Trying to insert a column " << gj << " outside the matrix width "
+                     "Trying to read a column " << gj << " outside the matrix width "
                     << width);
         a = _Get_(gj);
         subm(i, j) = (t < 0) ? (-a) : (a);
@@ -2215,7 +2256,7 @@ bool SparseMatrix::RowIsEmpty(const int row) const
      gi = -1-gi;
   }
   MFEM_ASSERT(gi < height,
-               "Trying to insert a row " << gi << " outside the matrix height "
+               "Trying to query a row " << gi << " outside the matrix height "
               << height);
   if (Rows)
   {
@@ -2234,7 +2275,7 @@ int SparseMatrix::GetRow(const int row, Array<int> &cols, Vector &srow) const

   if ((gi=row) < 0) { gi = -1-gi; }
   MFEM_ASSERT(gi < height,
-               "Trying to insert a row " << gi << " outside the matrix height "
+               "Trying to read a row " << gi << " outside the matrix height "
               << height);
   if (Rows)
   {
@@ -2261,7 +2302,7 @@ int SparseMatrix::GetRow(const int row, Array<int> &cols, Vector &srow) const
      j = I[gi];
      cols.MakeRef(J + j, I[gi+1]-j);
      srow.NewDataAndSize(A + j, cols.Size());
-      MFEM_ASSERT(row >= 0, "Row not valid: " << row );
+      MFEM_ASSERT(row >= 0, "Row not valid: " << row << ", height: " << height);
      return 1;
   }
 }
@@ -2275,7 +2316,7 @@ void SparseMatrix::SetRow(const int row, const Array<int> &cols,
   if ((gi=row) < 0) { gi = -1-gi, s = -1; }
   else { s = 1; }
   MFEM_ASSERT(gi < height,
-               "Trying to insert a row " << gi << " outside the matrix height "
+               "Trying to set a row " << gi << " outside the matrix height "
               << height);

   if (!Finalized())
@@ -2286,7 +2327,7 @@ void SparseMatrix::SetRow(const int row, const Array<int> &cols,
         if ((gj=cols[j]) < 0) { gj = -1-gj, t = -s; }
         else { t = s; }
         MFEM_ASSERT(gj < width,
-                     "Trying to insert a column " << gj << " outside the matrix"
+                     "Trying to set a column " << gj << " outside the matrix"
                     " width " << width);
         a = srow(j);
         if (t < 0) { a = -a; }
@@ -2304,7 +2345,7 @@ void SparseMatrix::SetRow(const int row, const Array<int> &cols,
         if ((gj=cols[j]) < 0) { gj = -1-gj, t = -s; }
         else { t = s; }
         MFEM_ASSERT(gj < width,
-                     "Trying to insert a column " << gj << " outside the matrix"
+                     "Trying to set a column " << gj << " outside the matrix"
                     " width " << width);

         J[i] = gj;
@@ -2708,15 +2749,15 @@ void SparseMatrix::Destroy()
 {
   if (I != NULL && ownGraph)
   {
-      delete [] I;
+      mfem::Delete(I);
   }
   if (J != NULL && ownGraph)
   {
-      delete [] J;
+      mfem::Delete(J);
   }
   if (A != NULL && ownData)
   {
-      delete [] A;
+      mfem::Delete(A);
   }

   if (Rows != NULL)
@@ -2750,9 +2791,10 @@ void SparseMatrix::Destroy()
      delete NodesMem;
   }
 #endif
+   delete At;
 }

-int SparseMatrix::ActualWidth()
+int SparseMatrix::ActualWidth() const
 {
   int awidth = 0;
   if (A)
@@ -2796,8 +2838,10 @@ SparseMatrix *Transpose (const SparseMatrix &A)
      "Finalize must be called before Transpose. Use TransposeRowMatrix instead");

   int i, j, end;
-   int m, n, nnz, *A_i, *A_j, *At_i, *At_j;
-   double *A_data, *At_data;
+   const int *A_i, *A_j;
+   int m, n, nnz, *At_i, *At_j;
+   const double *A_data;
+   double *At_data;

   m      = A.Height(); // number of rows of A
   n      = A.Width();  // number of columns of A
@@ -2806,9 +2850,9 @@ SparseMatrix *Transpose (const SparseMatrix &A)
   A_j    = A.GetJ();
   A_data = A.GetData();

-   At_i = new int[n+1];
-   At_j = new int[nnz];
-   At_data = new double[nnz];
+   At_i = mfem::New<int>(n+1);
+   At_j = mfem::New<int>(nnz);
+   At_data = mfem::New<double>(nnz);

   for (i = 0; i <= n; i++)
   {
@@ -2877,9 +2921,9 @@ SparseMatrix *TransposeAbstractSparseMatrix (const AbstractSparseMatrix &A,
   }
   nnz = A.NumNonZeroElems();

-   At_i = new int[n+1];
-   At_j = new int[nnz];
-   At_data = new double[nnz];
+   At_i = mfem::New<int>(n+1);
+   At_j = mfem::New<int>(nnz);
+   At_data = mfem::New<double>(nnz);

   for (i = 0; i <= n; i++)
   {
@@ -2924,8 +2968,10 @@ SparseMatrix *Mult (const SparseMatrix &A, const SparseMatrix &B,
                    SparseMatrix *OAB)
 {
   int nrowsA, ncolsA, nrowsB, ncolsB;
-   int *A_i, *A_j, *B_i, *B_j, *C_i, *C_j, *B_marker;
-   double *A_data, *B_data, *C_data;
+   const int *A_i, *A_j, *B_i, *B_j;
+   int *C_i, *C_j, *B_marker;
+   const double *A_data, *B_data;
+   double *C_data;
   int ia, ib, ic, ja, jb, num_nonzeros;
   int row_start, counter;
   double a_entry, b_entry;
@@ -2956,7 +3002,7 @@ SparseMatrix *Mult (const SparseMatrix &A, const SparseMatrix &B,

   if (OAB == NULL)
   {
-      C_i = new int[nrowsA+1];
+      C_i = mfem::New<int>(nrowsA+1);

      C_i[0] = num_nonzeros = 0;
      for (ic = 0; ic < nrowsA; ic++)
@@ -2977,8 +3023,8 @@ SparseMatrix *Mult (const SparseMatrix &A, const SparseMatrix &B,
         C_i[ic+1] = num_nonzeros;
      }

-      C_j    = new int[num_nonzeros];
-      C_data = new double[num_nonzeros];
+      C_j    = mfem::New<int>(num_nonzeros);
+      C_data = mfem::New<double>(num_nonzeros);

      C = new SparseMatrix (C_i, C_j, C_data, nrowsA, ncolsB);

@@ -3081,7 +3127,7 @@ SparseMatrix *MultAbstractSparseMatrix (const AbstractSparseMatrix &A,
      B_marker[ib] = -1;
   }

-   C_i = new int[nrowsA+1];
+   C_i = mfem::New<int>(nrowsA+1);

   C_i[0] = num_nonzeros = 0;

@@ -3107,8 +3153,8 @@ SparseMatrix *MultAbstractSparseMatrix (const AbstractSparseMatrix &A,
      C_i[ic+1] = num_nonzeros;
   }

-   C_j    = new int[num_nonzeros];
-   C_data = new double[num_nonzeros];
+   C_j    = mfem::New<int>(num_nonzeros);
+   C_data = mfem::New<double>(num_nonzeros);

   C = new SparseMatrix(C_i, C_j, C_data, nrowsA, ncolsB);

@@ -3234,17 +3280,17 @@ SparseMatrix * Add(double a, const SparseMatrix & A, double b,
   int nrows = A.Height();
   int ncols = A.Width();

-   int * C_i = new int[nrows+1];
+   int * C_i =mfem::New<int>(nrows+1);
   int * C_j;
   double * C_data;

-   int * A_i = A.GetI();
-   int * A_j = A.GetJ();
-   double * A_data = A.GetData();
+   const int *A_i = A.GetI();
+   const int *A_j = A.GetJ();
+   const double *A_data = A.GetData();

-   int * B_i = B.GetI();
-   int * B_j = B.GetJ();
-   double * B_data = B.GetData();
+   const int *B_i = B.GetI();
+   const int *B_j = B.GetJ();
+   const double *B_data = B.GetData();

   int * marker = new int[ncols];
   std::fill(marker, marker+ncols, -1);
@@ -3271,8 +3317,8 @@ SparseMatrix * Add(double a, const SparseMatrix & A, double b,
      C_i[ic+1] = num_nonzeros;
   }

-   C_j = new int[num_nonzeros];
-   C_data = new double[num_nonzeros];
+   C_j = mfem::New<int>(num_nonzeros);
+   C_data = mfem::New<double>(num_nonzeros);

   for (int ia = 0; ia < ncols; ia++)
   {
@@ -3472,6 +3518,7 @@ void SparseMatrix::Swap(SparseMatrix &other)
   mfem::Swap(current_row, other.current_row);
   mfem::Swap(ColPtrJ, other.ColPtrJ);
   mfem::Swap(ColPtrNode, other.ColPtrNode);
+   mfem::Swap(At, other.At);

 #ifdef MFEM_USE_MEMALLOC
   mfem::Swap(NodesMem, other.NodesMem);
@@ -64,6 +64,9 @@ protected:
   mutable int* ColPtrJ;
   mutable RowNode ** ColPtrNode;

+   /// Transpose of A. Owned. Used to perform MultTranspose() on devices.
+   mutable SparseMatrix *At;
+
 #ifdef MFEM_USE_MEMALLOC
   typedef MemAlloc <RowNode, 1024> RowNodeAlloc;
   RowNodeAlloc * NodesMem;
@@ -97,6 +100,9 @@ public:

   /** @brief Create a sparse matrix in CSR format. Ownership of @a i, @a j, and
       @a data is optionally transferred to the SparseMatrix. */
+   /** If the parameter @a data is NULL, then the internal #A array is allocated
+       by this constructor (initializing it with zeros and taking ownership,
+       regardless of the parameter @a owna). */
   SparseMatrix(int *i, int *j, double *data, int m, int n, bool ownij,
                bool owna, bool issorted);

@@ -112,7 +118,7 @@ public:
       ownership. */
   SparseMatrix(const SparseMatrix &mat, bool copy_graph = true);

-   /// Create a SparseMatrix with diagonal v, i.e. A = Diag(v)
+   /// Create a SparseMatrix with diagonal @a v, i.e. A = Diag(v)
   SparseMatrix(const Vector & v);


@@ -134,21 +140,35 @@ public:
   /// Check if the SparseMatrix is empty.
   bool Empty() const { return (A == NULL) && (Rows == NULL); }

-   /// Return the array #I
-   inline int *GetI() const { return I; }
-   /// Return the array #J
-   inline int *GetJ() const { return J; }
-   /// Return element data, i.e. array #A
-   inline double *GetData() const { return A; }
-   /// Returns the number of elements in row @a i
+   /// Return the array #I.
+   inline int *GetI() { return I; }
+   /// Return the array #I, const version.
+   inline const int *GetI() const { return I; }
+
+   /// Return the array #J.
+   inline int *GetJ() { return J; }
+   /// Return the array #J, const version.
+   inline const int *GetJ() const { return J; }
+
+   /// Return the element data, i.e. the array #A.
+   inline double *GetData() { return A; }
+   /// Return the element data, i.e. the array #A, const version.
+   inline const double *GetData() const { return A; }
+
+   /// Returns the number of elements in row @a i.
   int RowSize(const int i) const;
-   /// Returns the maximum number of elements among all rows
+
+   /// Returns the maximum number of elements among all rows.
   int MaxRowSize() const;
-   /// Return a pointer to the column indices in a row
+
+   /// Return a pointer to the column indices in a row.
   int *GetRowColumns(const int row);
+   /// Return a pointer to the column indices in a row, const version.
   const int *GetRowColumns(const int row) const;
-   /// Return a pointer to the entries in a row
+
+   /// Return a pointer to the entries in a row.
   double *GetRowEntries(const int row);
+   /// Return a pointer to the entries in a row, const version.
   const double *GetRowEntries(const int row) const;

   /// Change the width of a SparseMatrix.
@@ -163,7 +183,7 @@ public:

   /// Returns the actual Width of the matrix.
   /*! This method can be called for matrices finalized or not. */
-   int ActualWidth();
+   int ActualWidth() const;

   /// Sort the column indices corresponding to each row.
   void SortColumnIndices();
@@ -206,13 +226,45 @@ public:
   void AddMultTranspose(const Vector &x, Vector &y,
                         const double a = 1.0) const;

+   /** @brief Build and store internally the transpose of this matrix which will
+       be used in the methods AddMultTranspose() and MultTranspose(). */
+   /** If this method has been called, the internal transpose matrix will be
+       used to perform the action of the transpose matrix in AddMultTranspose(),
+       and MultTranspose().
+
+       Warning: any changes in this matrix will invalidate the internal
+       transpose. To rebuild the transpose, call ResetTranspose() followed by a
+       call to this method. If the internal transpose is already built, this
+       method has no effect.
+
+       When any non-default backend is enabled, i.e. Device::IsEnabled() is
+       true, the methods AddMultTranspose(), and MultTranspose(), require the
+       internal transpose to be built. If that is not the case (i.e. the
+       internal transpose is not built), these methods will raise an error with
+       an appropriate message pointing to this method. When using the default
+       backend, calling this method is optional.
+
+       This method can only be used when the sparse matrix is finalized. */
+   void BuildTranspose() const;
+
+   /** Reset (destroy) the internal transpose matrix. See BuildTranspose() for
+       more details. */
+   void ResetTranspose() const;
+
   void PartMult(const Array<int> &rows, const Vector &x, Vector &y) const;
   void PartAddMult(const Array<int> &rows, const Vector &x, Vector &y,
                    const double a=1.0) const;

-   /// y = A * x, but treat all elements as booleans (zero=false, nonzero=true).
+   /// y = A * x, treating all entries as booleans (zero=false, nonzero=true).
+   /** The actual values stored in the data array, #A, are not used - this means
+       and that all entries in the sparsity pattern are considered to be true by
+       this method. */
   void BooleanMult(const Array<int> &x, Array<int> &y) const;
-   /// y = At * x, but treat all elements as booleans (zero=false, nonzero=true).
+
+   /// y = At * x, treating all entries as booleans (zero=false, nonzero=true).
+   /** The actual values stored in the data array, #A, are not used - this means
+       and that all entries in the sparsity pattern are considered to be true by
+       this method. */
   void BooleanMultTranspose(const Array<int> &x, Array<int> &y) const;

   /// Compute y^t A x
@@ -452,11 +504,11 @@ SparseMatrix *Transpose(const SparseMatrix &A);
 SparseMatrix *TransposeAbstractSparseMatrix (const AbstractSparseMatrix &A,
                                             int useActualWidth);

-/** Matrix product A.B.
-    If OAB is not NULL, we assume it has the structure
-    of A.B and store the result in OAB.
-    If OAB is NULL, we create a new SparseMatrix to store
+/// Matrix product A.B.
+/** If @a OAB is not NULL, we assume it has the structure of A.B and store the
+    result in @a OAB. If @a OAB is NULL, we create a new SparseMatrix to store
    the result and return a pointer to it.
+
    All matrices must be finalized. */
 SparseMatrix *Mult(const SparseMatrix &A, const SparseMatrix &B,
                   SparseMatrix *OAB = NULL);
@@ -555,16 +607,20 @@ inline void SparseMatrix::SetColPtr(const int row) const
 inline void SparseMatrix::ClearColPtr() const
 {
   if (Rows)
+   {
      for (RowNode *node_p = Rows[current_row]; node_p != NULL;
           node_p = node_p->Prev)
      {
         ColPtrNode[node_p->Column] = NULL;
      }
+   }
   else
+   {
      for (int j = I[current_row], end = I[current_row+1]; j < end; j++)
      {
         ColPtrJ[J[j]] = -1;
      }
+   }
 }

 inline double &SparseMatrix::SearchRow(const int col)
@@ -145,11 +145,34 @@ void STRUMPACKSolver::SetReorderingStrategy( strumpack::ReorderingStrategy
   solver_->options().set_reordering_method( method );
 }

-void STRUMPACKSolver::SetMC64Job( strumpack::MC64Job job )
+void STRUMPACKSolver::DisableMatching( )
 {
-   solver_->options().set_mc64job( job );
+#if STRUMPACK_VERSION_MAJOR >= 3
+   solver_->options().set_matching( strumpack::MatchingJob::NONE );
+#else
+   solver_->options().set_mc64job( strumpack::MC64Job::NONE );
+#endif
 }

+void STRUMPACKSolver::EnableMatching( )
+{
+#if STRUMPACK_VERSION_MAJOR >= 3
+   solver_->options().set_matching
+   ( strumpack::MatchingJob::MAX_DIAGONAL_PRODUCT_SCALING );
+#else
+   solver_->options().set_mc64job
+   ( strumpack::MC64Job::MAX_DIAGONAL_PRODUCT_SCALING );
+#endif
+}
+
+#if STRUMPACK_VERSION_MAJOR >= 3
+void STRUMPACKSolver::EnableParallelMatching( )
+{
+   solver_->options().set_matching
+   ( strumpack::MatchingJob::COMBBLAS );
+}
+#endif
+
 void STRUMPACKSolver::SetRelTol( double rtol )
 {
   solver_->options().set_rel_tol( rtol );
@@ -121,21 +121,30 @@ public:
   void SetReorderingStrategy( strumpack::ReorderingStrategy method );

   /**
-    * MC64 performs (static) pivoting. Using a matching algorithm, it permutes
-    * the sparse input matrix in order to get nonzero elements on the
-    * diagonal. If the input matrix is already diagonally dominant, this
+    * Disable static pivoting for stability. The static pivoting in strumpack
+    * permutes the sparse input matrix in order to get large (nonzero) elements
+    * on the diagonal. If the input matrix is already diagonally dominant, this
    * reordering can be disabled.
-    * Possible values are:
-    *    NONE:                          Don't do anything
-    *    MAX_CARDINALITY:               Maximum cardinality
-    *    MAX_SMALLEST_DIAGONAL:         Maximize smallest diagonal value
-    *    MAX_SMALLEST_DIAGONAL_2:       Same as MAX_SMALLEST_DIAGONAL, but
-    *                                   different algorithm
-    *    MAX_DIAGONAL_SUM:              Maximize sum of diagonal values
-    *    MAX_DIAGONAL_PRODUCT_SCALING:  Maximize the product of the diagonal
-    *                                   values and perform row & column scaling
    */
-   void SetMC64Job( strumpack::MC64Job job );
+   void DisableMatching();
+
+   /**
+    * Enable static pivoting for stability using the MC64 algorithm with
+    * job=5. Using a matching algorithm, this will permute the sparse input
+    * matrix in order to get nonzero elements (as large as possible) on the
+    * diagonal. And will also scale the rows and columns of the matrix.
+    */
+   void EnableMatching();
+
+#if STRUMPACK_VERSION_MAJOR >= 3
+   /**
+    * Use the AWPM (approximate weight perfect matching) algorithm from the
+    * Combinatorial BLAS library for static pivoting, i.e. getting large
+    * nonzeros on the diagonal. This requires that strumpack was compiled with
+    * support for Combinatorial BLAS.
+    */
+   void EnableParallelMatching();
+#endif

 private:
   void Init( int argc, char* argv[] );
@@ -289,7 +289,7 @@ void SuperLUSolver::Init()
   }

   npcol_ = (int)(numProcs_ / nprow_);
-   assert(nprow_ * npcol_ == numProcs_);
+   MFEM_ASSERT(nprow_ * npcol_ == numProcs_, "");

   PStatInit(stat); // Initialize the statistics variables.
 }
@@ -423,7 +423,7 @@ void SuperLUSolver::SetupGrid()
      }

      npcol_ = (int)(numProcs_ / nprow_);
-      assert(nprow_ * npcol_ == numProcs_);
+      MFEM_ASSERT(nprow_ * npcol_ == numProcs_, "");
   }

   superlu_gridinit(comm_, nprow_, npcol_, grid);
@@ -12,6 +12,8 @@
 // Implementation of data type vector

 #include "vector.hpp"
+#include "dtensor.hpp"
+#include "../general/forall.hpp"

 #if defined(MFEM_USE_SUNDIALS) && defined(MFEM_USE_MPI)
 #include <nvector/nvector_parallel.h>
@@ -28,6 +30,16 @@
 namespace mfem
 {

+void Vector::Push() const
+{
+   mfem::Push(data, size*sizeof(double));
+}
+
+void Vector::Pull() const
+{
+   mfem::Pull(data, size*sizeof(double));
+}
+
 Vector::Vector(const Vector &v)
 {
   int s = v.Size();
@@ -36,8 +48,8 @@ Vector::Vector(const Vector &v)
   {
      MFEM_ASSERT(v.data, "invalid source vector");
      allocsize = size = s;
-      data = new double[s];
-      std::memcpy(data, v.data, sizeof(double)*s);
+      data = mfem::New<double>(s);
+      mfem::Memcpy(data, v.data, sizeof(double)*s);
   }
   else
   {
@@ -88,17 +100,7 @@ const double &Vector::Elem(int i) const

 double Vector::operator*(const double *v) const
 {
-   int s = size;
-   const double *d = data;
-   double prod = 0.0;
-#ifdef MFEM_USE_OPENMP
-   #pragma omp parallel for reduction(+:prod)
-#endif
-   for (int i = 0; i < s; i++)
-   {
-      prod += d[i] * v[i];
-   }
-   return prod;
+   return Dot(size, data, v);
 }

 double Vector::operator*(const Vector &v) const
@@ -118,7 +120,7 @@ Vector &Vector::operator=(const double *v)
   if (data != v)
   {
      MFEM_ASSERT(data + size <= v || v + size <= data, "Vectors overlap!");
-      std::memcpy(data, v, sizeof(double)*size);
+      mfem::Memcpy(data, v, sizeof(double)*size);
   }
   return *this;
 }
@@ -131,40 +133,30 @@ Vector &Vector::operator=(const Vector &v)

 Vector &Vector::operator=(double value)
 {
-   int i, s = size;
-   double *p = data, v = value;
-   for (i = 0; i < s; i++)
-   {
-      *(p++) = v;
-   }
+   DeviceVector y(data, size);
+   MFEM_FORALL(i, size, y[i] = value;);
   return *this;
 }

 Vector &Vector::operator*=(double c)
 {
-   for (int i = 0; i < size; i++)
-   {
-      data[i] *= c;
-   }
+   DeviceVector y(data, size);
+   MFEM_FORALL(i, size, y[i] *= c;);
   return *this;
 }

 Vector &Vector::operator/=(double c)
 {
-   double m = 1.0/c;
-   for (int i = 0; i < size; i++)
-   {
-      data[i] *= m;
-   }
+   const double m = 1.0/c;
+   DeviceVector y(data, size);
+   MFEM_FORALL(i, size, y[i] *= m;);
   return *this;
 }

 Vector &Vector::operator-=(double c)
 {
-   for (int i = 0; i < size; i++)
-   {
-      data[i] -= c;
-   }
+   DeviceVector y(data, size);
+   MFEM_FORALL(i, size, y[i] -= c;);
   return *this;
 }

@@ -176,10 +168,10 @@ Vector &Vector::operator-=(const Vector &v)
      mfem_error("Vector::operator-=(const Vector &)");
   }
 #endif
-   for (int i = 0; i < size; i++)
-   {
-      data[i] -= v(i);
-   }
+   const int N = size;
+   DeviceVector y(data, N);
+   const DeviceVector x(v, N);
+   MFEM_FORALL(i, N, y[i] -= x[i];);
   return *this;
 }

@@ -191,10 +183,10 @@ Vector &Vector::operator+=(const Vector &v)
      mfem_error("Vector::operator+=(const Vector &)");
   }
 #endif
-   for (int i = 0; i < size; i++)
-   {
-      data[i] += v(i);
-   }
+   const int N = size;
+   DeviceVector y(data, N);
+   const DeviceVector x(v, N);
+   MFEM_FORALL(i, N, y[i] += x[i];);
   return *this;
 }

@@ -208,10 +200,10 @@ Vector &Vector::Add(const double a, const Vector &Va)
 #endif
   if (a != 0.0)
   {
-      for (int i = 0; i < size; i++)
-      {
-         data[i] += a * Va(i);
-      }
+      const int N = size;
+      DeviceVector y(data, N);
+      const DeviceVector x(Va, N);
+      MFEM_FORALL(i, N, y[i] += a * x[i];);
   }
   return *this;
 }
@@ -224,10 +216,10 @@ Vector &Vector::Set(const double a, const Vector &Va)
      mfem_error("Vector::Set(const double, const Vector &)");
   }
 #endif
-   for (int i = 0; i < size; i++)
-   {
-      data[i] = a * Va(i);
-   }
+   const int N = size;
+   DeviceVector y(data, N);
+   const DeviceVector x(Va, N);
+   MFEM_FORALL(i, N, y[i] = a * x[i];);
   return *this;
 }

@@ -251,10 +243,8 @@ void Vector::SetVector(const Vector &v, int offset)

 void Vector::Neg()
 {
-   for (int i = 0; i < size; i++)
-   {
-      data[i] = -data[i];
-   }
+   DeviceVector y(data, size);
+   MFEM_FORALL(i, size, y[i] = -y[i];);
 }

 void add(const Vector &v1, const Vector &v2, Vector &v)
@@ -266,13 +256,19 @@ void add(const Vector &v1, const Vector &v2, Vector &v)
   }
 #endif

-#ifdef MFEM_USE_OPENMP
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+   const int N = v.size;
+   DeviceVector y(v, N);
+   const DeviceVector x1(v1, N);
+   const DeviceVector x2(v2, N);
+   MFEM_FORALL(i, N, y[i] = x1[i] + x2[i];);
+#else
   #pragma omp parallel for
-#endif
   for (int i = 0; i < v.size; i++)
   {
      v.data[i] = v1.data[i] + v2.data[i];
   }
+#endif
 }

 void add(const Vector &v1, double alpha, const Vector &v2, Vector &v)
@@ -295,14 +291,21 @@ void add(const Vector &v1, double alpha, const Vector &v2, Vector &v)
   {
      const double *v1p = v1.data, *v2p = v2.data;
      double *vp = v.data;
-      int s = v.size;
-#ifdef MFEM_USE_OPENMP
+
+      const int s = v.size;
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+      const int N = s;
+      DeviceVector d_z(vp, N);
+      const DeviceVector d_x(v1p, N);
+      const DeviceVector d_y(v2p, N);
+      MFEM_FORALL(i, N, d_z[i] = d_x[i] + alpha * d_y[i];);
+#else
      #pragma omp parallel for
-#endif
      for (int i = 0; i < s; i++)
      {
         vp[i] = v1p[i] + alpha*v2p[i];
      }
+#endif
   }
 }

@@ -326,15 +329,19 @@ void add(const double a, const Vector &x, const Vector &y, Vector &z)
      const double *xp = x.data;
      const double *yp = y.data;
      double       *zp = z.data;
-      int            s = x.size;
-
-#ifdef MFEM_USE_OPENMP
+      const int      s = x.size;
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+      DeviceVector z(zp, s);
+      const DeviceVector x(xp, s);
+      const DeviceVector y(yp, s);
+      MFEM_FORALL(i, s, z[i] = a * (x[i] + y[i]););
+#else
      #pragma omp parallel for
-#endif
      for (int i = 0; i < s; i++)
      {
         zp[i] = a * (xp[i] + yp[i]);
      }
+#endif
   }
 }

@@ -371,15 +378,20 @@ void add(const double a, const Vector &x,
      const double *xp = x.data;
      const double *yp = y.data;
      double       *zp = z.data;
-      int            s = x.size;
+      const int      s = x.size;

-#ifdef MFEM_USE_OPENMP
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+      DeviceVector z(zp, s);
+      const DeviceVector x(xp, s);
+      const DeviceVector y(yp, s);
+      MFEM_FORALL(i, s, z[i] = a * x[i] + b * y[i];);
+#else
      #pragma omp parallel for
-#endif
      for (int i = 0; i < s; i++)
      {
         zp[i] = a * xp[i] + b * yp[i];
      }
+#endif
   }
 }

@@ -394,15 +406,20 @@ void subtract(const Vector &x, const Vector &y, Vector &z)
   const double *xp = x.data;
   const double *yp = y.data;
   double       *zp = z.data;
-   int            s = x.size;
+   const int     s = x.size;

-#ifdef MFEM_USE_OPENMP
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+   DeviceVector zd(zp, s);
+   const DeviceVector xd(xp, s);
+   const DeviceVector yd(yp, s);
+   MFEM_FORALL(i, s, zd[i] = xd[i] - yd[i];);
+#else
   #pragma omp parallel for
-#endif
   for (int i = 0; i < s; i++)
   {
      zp[i] = xp[i] - yp[i];
   }
+#endif
 }

 void subtract(const double a, const Vector &x, const Vector &y, Vector &z)
@@ -426,121 +443,136 @@ void subtract(const double a, const Vector &x, const Vector &y, Vector &z)
      const double *xp = x.data;
      const double *yp = y.data;
      double       *zp = z.data;
-      int            s = x.size;
+      const int      s = x.size;

-#ifdef MFEM_USE_OPENMP
+#if !defined(MFEM_USE_LEGACY_OPENMP)
+      DeviceVector zd(zp, s);
+      const DeviceVector xd(xp, s);
+      const DeviceVector yd(yp, s);
+      MFEM_FORALL(i, s, zd[i] = a * (xd[i] - yd[i]););
+#else
      #pragma omp parallel for
-#endif
      for (int i = 0; i < s; i++)
      {
         zp[i] = a * (xp[i] - yp[i]);
      }
+#endif
   }
 }

 void Vector::median(const Vector &lo, const Vector &hi)
 {
-   double *v = data;
-
-   for (int i = 0; i < size; i++)
+   const int N = size;
+   DeviceVector v(data, N);
+   const DeviceVector l(lo, N);
+   const DeviceVector h(hi, N);
+   MFEM_FORALL(i, N,
   {
-      if (v[i] < lo[i])
+      if (v[i] < l[i])
      {
-         v[i] = lo[i];
+         v[i] = l[i];
      }
-      else if (v[i] > hi[i])
+      else if (v[i] > h[i])
      {
-         v[i] = hi[i];
+         v[i] = h[i];
      }
-   }
+   });
+}
+
+static void GetSubvector(const int N,
+                         double *y, const double *x, const int* dofs)
+{
+   DeviceVector d_y(y, N);
+   const DeviceVector d_x(x, N);
+   const DeviceArray d_dofs(dofs, N);
+   MFEM_FORALL(i, N,
+   {
+      const int dof_i = d_dofs[i];
+      d_y[i] = dof_i >= 0 ? d_x[dof_i] : -d_x[-dof_i-1];
+   });
 }

 void Vector::GetSubVector(const Array<int> &dofs, Vector &elemvect) const
 {
-   int i, j, n = dofs.Size();
-
-   elemvect.SetSize (n);
-
-   for (i = 0; i < n; i++)
-   {
-      if ((j=dofs[i]) >= 0)
-      {
-         elemvect(i) = data[j];
-      }
-      else
-      {
-         elemvect(i) = -data[-1-j];
-      }
-   }
+   const int n = dofs.Size();
+   elemvect.SetSize(n);
+   mfem::GetSubvector(n, elemvect, data, dofs);
 }

 void Vector::GetSubVector(const Array<int> &dofs, double *elem_data) const
 {
-   int i, j, n = dofs.Size();
+   mfem::GetSubvector(dofs.Size(), elem_data, data,dofs);
+}

-   for (i = 0; i < n; i++)
+static void SetSubvector(const int N, double* y, const double d,
+                         const int* dofs)
+{
+   DeviceVector d_y(y,N);
+   const DeviceArray d_dofs(dofs,N);
+   MFEM_FORALL(i, N,
   {
-      if ((j=dofs[i]) >= 0)
+      const int j = d_dofs[i];
+      if (j >= 0)
      {
-         elem_data[i] = data[j];
+         d_y[j] = d;
      }
      else
      {
-         elem_data[i] = -data[-1-j];
+         d_y[-1-j] = -d;
      }
-   }
+   });
+}
+
+static void SetSubvector(const int N, double *y, const double *x,
+                         const int* dofs)
+{
+   DeviceVector d_y(y,N);
+   const DeviceVector d_x(x,N);
+   const DeviceArray d_dofs(dofs,N);
+   MFEM_FORALL(i, N,
+   {
+      const int dof_i = d_dofs[i];
+      if (dof_i >= 0)
+      {
+         d_y[dof_i] = d_x[i];
+      }
+      else
+      {
+         d_y[-1-dof_i] = -d_x[i];
+      }
+   });
 }

 void Vector::SetSubVector(const Array<int> &dofs, const double value)
 {
-   const int n = dofs.Size();
-
-   for (int i = 0; i < n; i++)
-   {
-      const int j = dofs[i];
-      if (j >= 0)
-      {
-         data[j] = value;
-      }
-      else
-      {
-         data[-1-j] = -value;
-      }
-   }
+   mfem::SetSubvector(dofs.Size(), data, value, dofs);
 }

 void Vector::SetSubVector(const Array<int> &dofs, const Vector &elemvect)
 {
-   int i, j, n = dofs.Size();
-
-   for (i = 0; i < n; i++)
-   {
-      if ((j=dofs[i]) >= 0)
-      {
-         data[j] = elemvect(i);
-      }
-      else
-      {
-         data[-1-j] = -elemvect(i);
-      }
-   }
+   mfem::SetSubvector(dofs.Size(), data, elemvect, dofs);
 }

 void Vector::SetSubVector(const Array<int> &dofs, double *elem_data)
 {
-   int i, j, n = dofs.Size();
+   mfem::SetSubvector(dofs.Size(), data, elem_data, dofs);
+}

-   for (i = 0; i < n; i++)
+static void AddElement(const int N, const int *dofs, const double *x, double *y)
+{
+   DeviceVector d_y(y,N);
+   const DeviceVector d_x(x,N);
+   const DeviceArray d_dofs(dofs,N);
+   MFEM_FORALL(i, N,
   {
-      if ((j=dofs[i]) >= 0)
-      {
-         data[j] = elem_data[i];
-      }
+      const int j = d_dofs[i];
+      if (j >= 0)
+         d_y[j] += d_x[i];
      else
      {
-         data[-1-j] = -elem_data[i];
+         d_y[-1-j] -= d_x[i];
      }
-   }
+   });
 }

 void Vector::AddElementVector(const Array<int> &dofs, const Vector &elemvect)
@@ -548,55 +580,32 @@ void Vector::AddElementVector(const Array<int> &dofs, const Vector &elemvect)
   MFEM_ASSERT(dofs.Size() == elemvect.Size(), "Size mismatch: "
               "length of dofs is " << dofs.Size() <<
               ", length of elemvect is " << elemvect.Size());
-   int i, j, n = dofs.Size();
-
-   for (i = 0; i < n; i++)
-   {
-      if ((j=dofs[i]) >= 0)
-      {
-         data[j] += elemvect(i);
-      }
-      else
-      {
-         data[-1-j] -= elemvect(i);
-      }
-   }
+   mfem::AddElement(dofs.Size(), dofs, elemvect.GetData(), data);
 }

 void Vector::AddElementVector(const Array<int> &dofs, double *elem_data)
 {
-   int i, j, n = dofs.Size();
-
-   for (i = 0; i < n; i++)
-   {
-      if ((j = dofs[i]) >= 0)
-      {
-         data[j] += elem_data[i];
-      }
-      else
-      {
-         data[-1-j] -= elem_data[i];
-      }
-   }
+   mfem::AddElement(dofs.Size(), dofs, elem_data, data);
 }

 void Vector::AddElementVector(const Array<int> &dofs, const double a,
                              const Vector &elemvect)
 {
-   MFEM_ASSERT(dofs.Size() == elemvect.Size(), "");
-   int i, j, n = dofs.Size();
-
-   for (i = 0; i < n; i++)
+   const int N = dofs.Size();
+   const double alpha = a;
+   DeviceVector d_y(data, N);
+   const DeviceVector d_x(elemvect, N);
+   const DeviceArray d_dofs(dofs, N);
+   MFEM_FORALL(i, N,
   {
-      if ((j=dofs[i]) >= 0)
-      {
-         data[j] += a * elemvect(i);
-      }
+      const int j = d_dofs[i];
+      if (j >= 0)
+         d_y[j] += alpha * d_x[i];
      else
      {
-         data[-1-j] -= a * elemvect(i);
+         d_y[-1-j] -= alpha * d_x[i];
      }
-   }
+   });
 }

 void Vector::SetSubVectorComplement(const Array<int> &dofs, const double val)
@@ -610,7 +619,7 @@ void Vector::SetSubVectorComplement(const Array<int> &dofs, const double val)
 void Vector::Print(std::ostream &out, int width) const
 {
   if (!size) { return; }
-
+   Pull();
   for (int i = 0; 1; )
   {
      out << data[i];
@@ -812,6 +821,146 @@ double Vector::Sum() const
   return sum;
 }

+#ifdef MFEM_USE_CUDA
+static __global__ void cuKernelMin(const int N, double *gdsr, const double *x)
+{
+   __shared__ double s_min[MFEM_CUDA_BLOCKS];
+   const int n = blockDim.x*blockIdx.x + threadIdx.x;
+   if (n>=N) { return; }
+   const int bid = blockIdx.x;
+   const int tid = threadIdx.x;
+   const int bbd = bid*blockDim.x;
+   const int rid = bbd+tid;
+   s_min[tid] = x[n];
+   for (int workers=blockDim.x>>1; workers>0; workers>>=1)
+   {
+      __syncthreads();
+      if (tid >= workers) { continue; }
+      if (rid >= N) { continue; }
+      const int dualTid = tid + workers;
+      if (dualTid >= N) { continue; }
+      const int rdd = bbd+dualTid;
+      if (rdd >= N) { continue; }
+      if (dualTid >= blockDim.x) { continue; }
+      s_min[tid] = fmin(s_min[tid], s_min[dualTid]);
+   }
+   if (tid==0) { gdsr[bid] = s_min[0]; }
+}
+
+static double cuVectorMin(const int N, const double *X)
+{
+   const DeviceVector x(X, N);
+   const int tpb = MFEM_CUDA_BLOCKS;
+   const int blockSize = MFEM_CUDA_BLOCKS;
+   const int gridSize = (N+blockSize-1)/blockSize;
+   const int min_sz = (N%tpb)==0? (N/tpb) : (1+N/tpb);
+   const int bytes = min_sz*sizeof(double);
+   static double *h_min = NULL;
+   if (!h_min) { h_min = (double*)calloc(min_sz,sizeof(double)); }
+   static void *gdsr = NULL;
+   if (!gdsr) { MFEM_CUDA_CHECK(cudaMalloc(&gdsr, bytes)); }
+   cuKernelMin<<<gridSize,blockSize>>>(N, (double*)gdsr, x);
+   MFEM_CUDA_CHECK(cudaGetLastError());
+   MFEM_CUDA_CHECK(cudaMemcpy(h_min, gdsr, bytes, cudaMemcpyDeviceToHost));
+   double min = std::numeric_limits<double>::infinity();
+   for (int i = 0; i < min_sz; i++) { min = fmin(min, h_min[i]); }
+   return min;
+}
+
+static __global__ void cuKernelDot(const int N, double *gdsr,
+                                   const double *x, const double *y)
+{
+   __shared__ double s_dot[MFEM_CUDA_BLOCKS];
+   const int n = blockDim.x*blockIdx.x + threadIdx.x;
+   if (n>=N) { return; }
+   const int bid = blockIdx.x;
+   const int tid = threadIdx.x;
+   const int bbd = bid*blockDim.x;
+   const int rid = bbd+tid;
+   s_dot[tid] = x[n] * y[n];
+   for (int workers=blockDim.x>>1; workers>0; workers>>=1)
+   {
+      __syncthreads();
+      if (tid >= workers) { continue; }
+      if (rid >= N) { continue; }
+      const int dualTid = tid + workers;
+      if (dualTid >= N) { continue; }
+      const int rdd = bbd+dualTid;
+      if (rdd >= N) { continue; }
+      if (dualTid >= blockDim.x) { continue; }
+      s_dot[tid] += s_dot[dualTid];
+   }
+   if (tid==0) { gdsr[bid] = s_dot[0]; }
+}
+
+static double cuVectorDot(const int N, const double *X, const double *Y)
+{
+   const DeviceVector x(X, N);
+   const DeviceVector y(Y, N);
+   static int dot_block_sz = 0;
+   const int tpb = MFEM_CUDA_BLOCKS;
+   const int blockSize = MFEM_CUDA_BLOCKS;
+   const int gridSize = (N+blockSize-1)/blockSize;
+   const int dot_sz = (N%tpb)==0? (N/tpb) : (1+N/tpb);
+   const int bytes = dot_sz*sizeof(double);
+   static double *h_dot = NULL;
+   if (!h_dot or dot_block_sz!=dot_sz)
+   {
+      if (h_dot) { free(h_dot); }
+      h_dot = (double*)calloc(dot_sz,sizeof(double));
+   }
+   static void *gdsr = NULL;
+   if (!gdsr or dot_block_sz!=dot_sz)
+   {
+      if (gdsr) { MFEM_CUDA_CHECK(cudaFree(gdsr)); }
+      MFEM_CUDA_CHECK(cudaMalloc(&gdsr,bytes));
+   }
+   if (dot_block_sz!=dot_sz)
+   {
+      dot_block_sz = dot_sz;
+   }
+   cuKernelDot<<<gridSize,blockSize>>>(N, (double*)gdsr, x, y);
+   MFEM_CUDA_CHECK(cudaGetLastError());
+   MFEM_CUDA_CHECK(cudaMemcpy(h_dot, gdsr, bytes, cudaMemcpyDeviceToHost));
+   double dot = 0.0;
+   for (int i = 0; i < dot_sz; i++) { dot += h_dot[i]; }
+   return dot;
+}
+#endif // MFEM_USE_CUDA
+
+double Min(const int N, const double *x)
+{
+   if (Device::Allows(Backend::CUDA_MASK))
+   {
+#ifdef MFEM_USE_CUDA
+      return cuVectorMin(N, x);
+#else
+      mfem_error("Using Min on device w/o support");
+#endif // MFEM_USE_CUDA
+   }
+   double min = std::numeric_limits<double>::infinity();
+   for (int i = 0; i < N; i++) { min = fmin(min, x[i]); }
+   return min;
+}
+
+double Dot(const int N, const double *x, const double *y)
+{
+   if (Device::Allows(Backend::CUDA_MASK))
+   {
+#ifdef MFEM_USE_CUDA
+      return cuVectorDot(N, x, y);
+#else
+      mfem_error("Using Dot on device w/o support");
+#endif // MFEM_USE_CUDA
+   }
+   double dot = 0.0;
+#ifdef MFEM_USE_LEGACY_OPENMP
+   #pragma omp parallel for reduction(+:dot)
+#endif
+   for (int i = 0; i < N; i++) { dot += x[i] * y[i]; }
+   return dot;
+}
+
 #ifdef MFEM_USE_SUNDIALS

 #ifndef SUNTRUE
@@ -12,10 +12,9 @@
 #ifndef MFEM_VECTOR
 #define MFEM_VECTOR

-// Data type vector
-
 #include "../general/array.hpp"
 #include "../general/globals.hpp"
+#include "../general/mem_manager.hpp"
 #ifdef MFEM_USE_SUNDIALS
 #include <nvector/nvector_serial.h>
 #endif
@@ -70,6 +69,12 @@ public:
   Vector (double *_data, int _size)
   { data = _data; size = _size; allocsize = -size; }

+   /// Copies data from host to device
+   void Push() const;
+
+   /// Copies data from device to host
+   void Pull() const;
+
   /// Reads a vector from multiple files
   void Load (std::istream ** in, int np, int * dim);

@@ -107,7 +112,7 @@ public:
       @sa SetDataAndSize(). */
   void NewDataAndSize(double *d, int s)
   {
-      if (allocsize > 0) { delete [] data; }
+      if (allocsize > 0) { mfem::Delete(data); }
      SetDataAndSize(d, s);
   }

@@ -318,7 +323,7 @@ inline Vector::Vector (int s)
   if (s > 0)
   {
      allocsize = size = s;
-      data = new double[s];
+      data = mfem::New<double>(s);
   }
   else
   {
@@ -340,17 +345,17 @@ inline void Vector::SetSize(int s)
   }
   if (allocsize > 0)
   {
-      delete [] data;
+      mfem::Delete(data);
   }
   allocsize = size = s;
-   data = new double[s];
+   data = mfem::New<double>(s);
 }

 inline void Vector::Destroy()
 {
   if (allocsize > 0)
   {
-      delete [] data;
+      mfem::Delete(data);
   }
   allocsize = size = 0;
   data = NULL;
@@ -389,7 +394,7 @@ inline Vector::~Vector()
 {
   if (allocsize > 0)
   {
-      delete [] data;
+      mfem::Delete(data);
   }
 }

@@ -443,6 +448,28 @@ inline double InnerProduct(MPI_Comm comm, const Vector &x, const Vector &y)
 }
 #endif

+/// Kernel returning the minimum value in the array x of size N
+double Min(const int N, const double *x);
+
+/// Kernel of the inner product of arrays x and y of size N
+double Dot(const int N, const double *x, const double *y);
+
+/// Class for a simple Vector of size 3
+class Vector3
+{
+private:
+   double data[3];
+public:
+   Vector3() {}
+   Vector3(const double *x) { data[0]=x[0]; data[1]=x[1]; data[2]=x[2]; }
+   Vector3(const double x0, const double x1 = 0.0, const double x2 = 0.0)
+   { data[0]=x0; data[1]=x1; data[2]=x2; }
+   inline operator double* () { return data; }
+   inline operator const double* () const { return data; }
+   inline double& operator()(const int i) { return data[i]; }
+   inline const double& operator()(const int i) const { return data[i]; }
+};
+
 }

 #endif
@@ -26,6 +26,10 @@ MFEM makefile targets:
   make parallel
   make debug
   make pdebug
+   make cuda
+   make pcuda
+   make cudebug
+   make pcudebug
   make test/check
   make install
   make clean
@@ -55,6 +59,14 @@ make debug
   A shortcut to configure and build the serial debug version of the library.
 make pdebug
   A shortcut to configure and build the parallel debug version of the library.
+make cuda
+   A shortcut to configure and build the serial GPU/CUDA optimized version of the library.
+make pcuda
+   A shortcut to configure and build the parallel GPU/CUDA optimized version of the library.
+make cudebug
+   A shortcut to configure and build the serial GPU/CUDA debug version of the library.
+make pcudebug
+   A shortcut to configure and build the parallel GPU/CUDA debug version of the library.
 make test
   Verify the build by checking the results from running all examples, miniapps,
   and tests.
@@ -147,7 +159,7 @@ $(call mfem-info, BLD       = $(BLD))

 # Include $(CONFIG_MK) unless some of the $(SKIP_INCLUDE_TARGETS) are given
 SKIP_INCLUDE_TARGETS = help config clean distclean serial parallel debug pdebug\
- style
+ cuda pcuda cudebug pcudebug style
 HAVE_SKIP_INCLUDE_TARGET = $(filter $(SKIP_INCLUDE_TARGETS),$(MAKECMDGOALS))
 ifeq (,$(HAVE_SKIP_INCLUDE_TARGET))
   $(call mfem-info, Including $(CONFIG_MK))
@@ -182,30 +194,46 @@ CXXFLAGS ?= $(OPTIM_FLAGS)

 # MPI configuration
 ifneq ($(MFEM_USE_MPI),YES)
-   MFEM_CXX ?= $(CXX)
+   CXX_OR_MPICXX = $(CXX)
   PKGS_NEED_MPI = SUPERLU STRUMPACK PETSC PUMI
   $(foreach mpidep,$(PKGS_NEED_MPI),$(if $(MFEM_USE_$(mpidep):NO=),\
     $(warning *** [MPI is OFF] setting MFEM_USE_$(mpidep) = NO)\
     $(eval override MFEM_USE_$(mpidep)=NO),))
 else
-   MFEM_CXX ?= $(MPICXX)
+   CXX_OR_MPICXX = $(MPICXX)
   INCFLAGS += $(HYPRE_OPT)
   ALL_LIBS += $(HYPRE_LIB)
 endif

+# CUDA configuration
+ifneq ($(MFEM_USE_CUDA),YES)
+   MFEM_CXX ?= $(CXX_OR_MPICXX)
+   XCOMPILER = $(CXX_XCOMPILER)
+   XLINKER   = $(CXX_XLINKER)
+else
+   ifneq ($(MFEM_USE_MM),YES)
+      $(error MFEM_USE_CUDA=YES requires MFEM_USE_MM=YES)
+   endif
+   MFEM_CXX ?= $(CUDA_CXX)
+   CXXFLAGS += $(CUDA_FLAGS) -ccbin $(CXX_OR_MPICXX)
+   XCOMPILER = $(CUDA_XCOMPILER)
+   XLINKER   = $(CUDA_XLINKER)
+   # CUDA_OPT and CUDA_LIB are added below
+endif
+
 DEP_CXX ?= $(MFEM_CXX)

 # Check OpenMP configuration
-ifeq ($(MFEM_USE_OPENMP),YES)
+ifeq ($(MFEM_USE_LEGACY_OPENMP),YES)
   MFEM_THREAD_SAFE ?= YES
   ifneq ($(MFEM_THREAD_SAFE),YES)
-      $(error Incompatible config: MFEM_USE_OPENMP requires MFEM_THREAD_SAFE)
+      $(error Incompatible config: MFEM_USE_LEGACY_OPENMP requires MFEM_THREAD_SAFE)
   endif
 endif

 # List of MFEM dependencies, that require the *_LIB variable to be non-empty
 MFEM_REQ_LIB_DEPS = SUPERLU METIS CONDUIT SIDRE LAPACK SUNDIALS MESQUITE\
- SUITESPARSE STRUMPACK GECKO GNUTLS NETCDF PETSC MPFR PUMI
+ SUITESPARSE STRUMPACK GECKO GNUTLS NETCDF PETSC MPFR PUMI OCCA RAJA
 PETSC_ERROR_MSG = $(if $(PETSC_FOUND),,. PETSC config not found: $(PETSC_VARS))

 define mfem_check_dependency
@@ -221,7 +249,10 @@ ifeq ($(MAKECMDGOALS),config)
 endif

 # List of MFEM dependencies, processed below
-MFEM_DEPENDENCIES = $(MFEM_REQ_LIB_DEPS) LIBUNWIND OPENMP
+MFEM_DEPENDENCIES = $(MFEM_REQ_LIB_DEPS) LIBUNWIND OPENMP CUDA
+
+# List of deprecated MFEM dependencies, processed below
+MFEM_LEGACY_DEPENDENCIES = OPENMP

 # Macro for adding dependencies
 define mfem_add_dependency
@@ -231,8 +262,18 @@ ifeq ($(MFEM_USE_$(1)),YES)
 endif
 endef

+# Macro for adding legacy dependencies
+define mfem_add_legacy_dependency
+ifeq ($(MFEM_USE_LEGACY_$(1)),YES)
+   INCFLAGS += $($(1)_OPT)
+   ALL_LIBS += $($(1)_LIB)
+endif
+endef
+
 # Process dependencies
 $(foreach dep,$(MFEM_DEPENDENCIES),$(eval $(call mfem_add_dependency,$(dep))))
+$(foreach dep,$(MFEM_LEGACY_DEPENDENCIES),$(eval $(call \
+   mfem_add_legacy_dependency,$(dep))))

 # Timer option
 ifeq ($(MFEM_TIMER_TYPE),2)
@@ -249,10 +290,12 @@ endif
 MFEM_DEFINES = MFEM_VERSION MFEM_VERSION_STRING MFEM_GIT_STRING MFEM_USE_MPI\
 MFEM_USE_METIS MFEM_USE_METIS_5 MFEM_DEBUG MFEM_USE_EXCEPTIONS\
 MFEM_USE_GZSTREAM MFEM_USE_LIBUNWIND MFEM_USE_LAPACK MFEM_THREAD_SAFE\
- MFEM_USE_OPENMP MFEM_USE_MEMALLOC MFEM_TIMER_TYPE MFEM_USE_SUNDIALS\
- MFEM_USE_MESQUITE MFEM_USE_SUITESPARSE MFEM_USE_GECKO MFEM_USE_SUPERLU\
- MFEM_USE_STRUMPACK MFEM_USE_GNUTLS MFEM_USE_NETCDF MFEM_USE_PETSC\
- MFEM_USE_MPFR MFEM_USE_SIDRE MFEM_USE_CONDUIT MFEM_USE_PUMI
+ MFEM_USE_OPENMP MFEM_USE_LEGACY_OPENMP MFEM_USE_MEMALLOC MFEM_TIMER_TYPE\
+ MFEM_USE_SUNDIALS MFEM_USE_MESQUITE MFEM_USE_SUITESPARSE MFEM_USE_GECKO\
+ MFEM_USE_SUPERLU MFEM_USE_STRUMPACK MFEM_USE_GNUTLS MFEM_USE_NETCDF\
+ MFEM_USE_PETSC MFEM_USE_MPFR MFEM_USE_SIDRE MFEM_USE_CONDUIT MFEM_USE_PUMI\
+ MFEM_USE_CUDA MFEM_USE_OCCA MFEM_USE_MM MFEM_USE_RAJA MFEM_SOURCE_DIR\
+ MFEM_INSTALL_DIR

 # List of makefile variables that will be written to config.mk:
 MFEM_CONFIG_VARS = MFEM_CXX MFEM_CPPFLAGS MFEM_CXXFLAGS MFEM_INC_DIR\
@@ -279,6 +322,9 @@ MFEM_TEST_MK   ?= @MFEM_DIR@/config/test.mk
 # Use "\n" (interpreted by sed) to add a newline.
 MFEM_CONFIG_EXTRA ?= $(if $(BUILD_DIR_DEF),MFEM_BUILD_DIR ?= @MFEM_DIR@,)

+MFEM_SOURCE_DIR  = $(MFEM_REAL_DIR)
+MFEM_INSTALL_DIR = $(abspath $(MFEM_PREFIX))
+
 # If we have 'config' target, export variables used by config/makefile
 ifneq (,$(filter config,$(MAKECMDGOALS)))
   export $(MFEM_DEFINES) MFEM_DEFINES $(MFEM_CONFIG_VARS) MFEM_CONFIG_VARS
@@ -301,6 +347,11 @@ ifneq (,$(filter install,$(MAKECMDGOALS)))
   MFEM_LIBS     = $(if $(shared),$(INSTALL_RPATH)) -L@MFEM_LIB_DIR@ -lmfem\
      @MFEM_EXT_LIBS@
   MFEM_LIB_FILE = @MFEM_LIB_DIR@/libmfem.$(if $(shared),$(SO_VER),a)
+   ifeq ($(MFEM_USE_OCCA),YES)
+      ifneq ($(MFEM_INSTALL_DIR),$(abspath $(PREFIX))
+         $(error OCCA is enabled: PREFIX must be set during configuration!)
+      endif
+   endif
   MFEM_PREFIX := $(abspath $(PREFIX))
   MFEM_INC_DIR = $(abspath $(PREFIX_INC))
   MFEM_LIB_DIR = $(abspath $(PREFIX_LIB))
@@ -315,9 +366,11 @@ DIRS = general linalg mesh fem
 SOURCE_FILES = $(foreach dir,$(DIRS),$(wildcard $(SRC)$(dir)/*.cpp))
 RELSRC_FILES = $(patsubst $(SRC)%,%,$(SOURCE_FILES))
 OBJECT_FILES = $(patsubst $(SRC)%,$(BLD)%,$(SOURCE_FILES:.cpp=.o))
+OKL_DIRS = fem

 .PHONY: lib all clean distclean install config status info deps serial parallel\
- debug pdebug style check test unittest
+	debug pdebug cuda pcuda cudebug pcudebug style check test unittest\
+	deprecation-warnings

 .SUFFIXES:
 .SUFFIXES: .cpp .o
@@ -354,26 +407,38 @@ doc:
 $(BLD)libmfem.a: $(OBJECT_FILES)
 	$(AR) $(ARFLAGS) $(@) $(OBJECT_FILES)
 	$(RANLIB) $(@)
+	@$(MAKE) deprecation-warnings

 $(BLD)libmfem.$(SO_EXT): $(BLD)libmfem.$(SO_VER)
 	cd $(@D) && ln -sf $(<F) $(@F)
+	@$(MAKE) deprecation-warnings

 # If some of the external libraries are build without -fPIC, linking shared MFEM
 # library may fail. In such cases, one may set EXT_LIBS on the command line.
 EXT_LIBS = $(MFEM_EXT_LIBS)
 $(BLD)libmfem.$(SO_VER): $(OBJECT_FILES)
-	$(MFEM_CXX) $(MFEM_BUILD_FLAGS) $(BUILD_SOFLAGS) $(OBJECT_FILES) \
+	$(MFEM_CXX) $(MFEM_LINK_FLAGS) $(BUILD_SOFLAGS) $(OBJECT_FILES) \
 	   $(EXT_LIBS) -o $(@)

-serial debug:    M_MPI=NO
-parallel pdebug: M_MPI=YES
-serial parallel: M_DBG=NO
-debug pdebug:    M_DBG=YES
+# Shortcut targets options
+serial parallel debug pdebug:   M_MM=NO
+serial debug cuda cudebug:      M_MPI=NO
+parallel pdebug pcuda pcudebug: M_MPI=YES
+serial parallel cuda pcuda:     M_DBG=NO
+debug pdebug cudebug pcudebug:  M_DBG=YES
+cuda pcuda cudebug pcudebug:    M_CUDA=YES
+cuda pcuda cudebug pcudebug:    M_MM=YES
+
 serial parallel debug pdebug:
 	$(MAKE) -f $(THIS_MK) config MFEM_USE_MPI=$(M_MPI) MFEM_DEBUG=$(M_DBG) \
 	   $(MAKEOVERRIDES_SAVE)
 	$(MAKE) $(MAKEOVERRIDES_SAVE)

+cuda pcuda cudebug pcudebug:
+	$(MAKE) -f $(THIS_MK) config MFEM_USE_MPI=$(M_MPI) MFEM_DEBUG=$(M_DBG) \
+	   MFEM_USE_CUDA=$(M_CUDA) MFEM_USE_MM=$(M_MM) $(MAKEOVERRIDES_SAVE)
+	$(MAKE) $(MAKEOVERRIDES_SAVE)
+
 deps:
 	rm -f $(BLD)deps.mk
 	for i in $(RELSRC_FILES:.cpp=); do \
@@ -442,7 +507,13 @@ install: $(if $(static),$(BLD)libmfem.a) $(if $(shared),$(BLD)libmfem.$(SO_EXT))
 # install remaining includes in each subdirectory
 	for dir in $(DIRS); do \
 	   mkdir -p $(PREFIX_INC)/mfem/$$dir && \
-	   $(INSTALL) -m 640 $(SRC)$$dir/*.hpp $(PREFIX_INC)/mfem/$$dir; done
+	   $(INSTALL) -m 640 $(SRC)$$dir/*.hpp $(PREFIX_INC)/mfem/$$dir; \
+	done
+# install *.okl files
+	for dir in $(OKL_DIRS); do \
+	   mkdir -p $(PREFIX_INC)/mfem/$$dir && \
+	   $(INSTALL) -m 640 $(SRC)$$dir/*.okl $(PREFIX_INC)/mfem/$$dir; \
+	done
 # install config.mk in $(PREFIX_SHARE)
 	mkdir -p $(PREFIX_SHARE)
 	$(MAKE) -C $(BLD)config config-mk CONFIG_MK=config-install.mk
@@ -490,52 +561,58 @@ help:
 	@true

 status info:
-	$(info MFEM_VERSION         = $(MFEM_VERSION) [v$(MFEM_VERSION_STRING)])
-	$(info MFEM_GIT_STRING      = $(MFEM_GIT_STRING))
-	$(info MFEM_USE_MPI         = $(MFEM_USE_MPI))
-	$(info MFEM_USE_METIS       = $(MFEM_USE_METIS))
-	$(info MFEM_USE_METIS_5     = $(MFEM_USE_METIS_5))
-	$(info MFEM_DEBUG           = $(MFEM_DEBUG))
-	$(info MFEM_USE_EXCEPTIONS  = $(MFEM_USE_EXCEPTIONS))
-	$(info MFEM_USE_GZSTREAM    = $(MFEM_USE_GZSTREAM))
-	$(info MFEM_USE_LIBUNWIND   = $(MFEM_USE_LIBUNWIND))
-	$(info MFEM_USE_LAPACK      = $(MFEM_USE_LAPACK))
-	$(info MFEM_THREAD_SAFE     = $(MFEM_THREAD_SAFE))
-	$(info MFEM_USE_OPENMP      = $(MFEM_USE_OPENMP))
-	$(info MFEM_USE_MEMALLOC    = $(MFEM_USE_MEMALLOC))
-	$(info MFEM_TIMER_TYPE      = $(MFEM_TIMER_TYPE))
-	$(info MFEM_USE_SUNDIALS    = $(MFEM_USE_SUNDIALS))
-	$(info MFEM_USE_MESQUITE    = $(MFEM_USE_MESQUITE))
-	$(info MFEM_USE_SUITESPARSE = $(MFEM_USE_SUITESPARSE))
-	$(info MFEM_USE_SUPERLU     = $(MFEM_USE_SUPERLU))
-	$(info MFEM_USE_STRUMPACK   = $(MFEM_USE_STRUMPACK))
-	$(info MFEM_USE_GECKO       = $(MFEM_USE_GECKO))
-	$(info MFEM_USE_GNUTLS      = $(MFEM_USE_GNUTLS))
-	$(info MFEM_USE_NETCDF      = $(MFEM_USE_NETCDF))
-	$(info MFEM_USE_PETSC       = $(MFEM_USE_PETSC))
-	$(info MFEM_USE_MPFR        = $(MFEM_USE_MPFR))
-	$(info MFEM_USE_SIDRE       = $(MFEM_USE_SIDRE))
-	$(info MFEM_USE_CONDUIT     = $(MFEM_USE_CONDUIT))
-	$(info MFEM_USE_PUMI        = $(MFEM_USE_PUMI))
-	$(info MFEM_CXX             = $(value MFEM_CXX))
-	$(info MFEM_CPPFLAGS        = $(value MFEM_CPPFLAGS))
-	$(info MFEM_CXXFLAGS        = $(value MFEM_CXXFLAGS))
-	$(info MFEM_TPLFLAGS        = $(value MFEM_TPLFLAGS))
-	$(info MFEM_INCFLAGS        = $(value MFEM_INCFLAGS))
-	$(info MFEM_FLAGS           = $(value MFEM_FLAGS))
-	$(info MFEM_EXT_LIBS        = $(value MFEM_EXT_LIBS))
-	$(info MFEM_LIBS            = $(value MFEM_LIBS))
-	$(info MFEM_LIB_FILE        = $(value MFEM_LIB_FILE))
-	$(info MFEM_BUILD_TAG       = $(value MFEM_BUILD_TAG))
-	$(info MFEM_PREFIX          = $(value MFEM_PREFIX))
-	$(info MFEM_INC_DIR         = $(value MFEM_INC_DIR))
-	$(info MFEM_LIB_DIR         = $(value MFEM_LIB_DIR))
-	$(info MFEM_STATIC          = $(MFEM_STATIC))
-	$(info MFEM_SHARED          = $(MFEM_SHARED))
-	$(info MFEM_BUILD_DIR       = $(MFEM_BUILD_DIR))
-	$(info MFEM_MPIEXEC         = $(MFEM_MPIEXEC))
-	$(info MFEM_MPIEXEC_NP      = $(MFEM_MPIEXEC_NP))
-	$(info MFEM_MPI_NP          = $(MFEM_MPI_NP))
+	$(info MFEM_VERSION           = $(MFEM_VERSION) [v$(MFEM_VERSION_STRING)])
+	$(info MFEM_GIT_STRING        = $(MFEM_GIT_STRING))
+	$(info MFEM_USE_MPI           = $(MFEM_USE_MPI))
+	$(info MFEM_USE_METIS         = $(MFEM_USE_METIS))
+	$(info MFEM_USE_METIS_5       = $(MFEM_USE_METIS_5))
+	$(info MFEM_DEBUG             = $(MFEM_DEBUG))
+	$(info MFEM_USE_EXCEPTIONS    = $(MFEM_USE_EXCEPTIONS))
+	$(info MFEM_USE_GZSTREAM      = $(MFEM_USE_GZSTREAM))
+	$(info MFEM_USE_LIBUNWIND     = $(MFEM_USE_LIBUNWIND))
+	$(info MFEM_USE_LAPACK        = $(MFEM_USE_LAPACK))
+	$(info MFEM_THREAD_SAFE       = $(MFEM_THREAD_SAFE))
+	$(info MFEM_USE_OPENMP        = $(MFEM_USE_OPENMP))
+	$(info MFEM_USE_LEGACY_OPENMP = $(MFEM_USE_LEGACY_OPENMP))
+	$(info MFEM_USE_MEMALLOC      = $(MFEM_USE_MEMALLOC))
+	$(info MFEM_TIMER_TYPE        = $(MFEM_TIMER_TYPE))
+	$(info MFEM_USE_SUNDIALS      = $(MFEM_USE_SUNDIALS))
+	$(info MFEM_USE_MESQUITE      = $(MFEM_USE_MESQUITE))
+	$(info MFEM_USE_SUITESPARSE   = $(MFEM_USE_SUITESPARSE))
+	$(info MFEM_USE_SUPERLU       = $(MFEM_USE_SUPERLU))
+	$(info MFEM_USE_STRUMPACK     = $(MFEM_USE_STRUMPACK))
+	$(info MFEM_USE_GECKO         = $(MFEM_USE_GECKO))
+	$(info MFEM_USE_GNUTLS        = $(MFEM_USE_GNUTLS))
+	$(info MFEM_USE_NETCDF        = $(MFEM_USE_NETCDF))
+	$(info MFEM_USE_PETSC         = $(MFEM_USE_PETSC))
+	$(info MFEM_USE_MPFR          = $(MFEM_USE_MPFR))
+	$(info MFEM_USE_SIDRE         = $(MFEM_USE_SIDRE))
+	$(info MFEM_USE_CONDUIT       = $(MFEM_USE_CONDUIT))
+	$(info MFEM_USE_PUMI          = $(MFEM_USE_PUMI))
+	$(info MFEM_USE_CUDA          = $(MFEM_USE_CUDA))
+	$(info MFEM_USE_RAJA          = $(MFEM_USE_RAJA))
+	$(info MFEM_USE_OCCA          = $(MFEM_USE_OCCA))
+	$(info MFEM_USE_MM            = $(MFEM_USE_MM))
+	$(info MFEM_CXX               = $(value MFEM_CXX))
+	$(info MFEM_CPPFLAGS          = $(value MFEM_CPPFLAGS))
+	$(info MFEM_CXXFLAGS          = $(value MFEM_CXXFLAGS))
+	$(info MFEM_TPLFLAGS          = $(value MFEM_TPLFLAGS))
+	$(info MFEM_INCFLAGS          = $(value MFEM_INCFLAGS))
+	$(info MFEM_FLAGS             = $(value MFEM_FLAGS))
+	$(info MFEM_LINK_FLAGS        = $(value MFEM_LINK_FLAGS))
+	$(info MFEM_EXT_LIBS          = $(value MFEM_EXT_LIBS))
+	$(info MFEM_LIBS              = $(value MFEM_LIBS))
+	$(info MFEM_LIB_FILE          = $(value MFEM_LIB_FILE))
+	$(info MFEM_BUILD_TAG         = $(value MFEM_BUILD_TAG))
+	$(info MFEM_PREFIX            = $(value MFEM_PREFIX))
+	$(info MFEM_INC_DIR           = $(value MFEM_INC_DIR))
+	$(info MFEM_LIB_DIR           = $(value MFEM_LIB_DIR))
+	$(info MFEM_STATIC            = $(MFEM_STATIC))
+	$(info MFEM_SHARED            = $(MFEM_SHARED))
+	$(info MFEM_BUILD_DIR         = $(MFEM_BUILD_DIR))
+	$(info MFEM_MPIEXEC           = $(MFEM_MPIEXEC))
+	$(info MFEM_MPIEXEC_NP        = $(MFEM_MPIEXEC_NP))
+	$(info MFEM_MPI_NP            = $(MFEM_MPI_NP))
 	@true

 ASTYLE = astyle --options=$(SRC)config/mfem.astylerc
@@ -543,6 +620,20 @@ FORMAT_FILES = $(foreach dir,$(DIRS) $(EM_DIRS) config,"$(dir)/*.?pp")
 FORMAT_FILES += "tests/unit/*.cpp"
 FORMAT_FILES += $(foreach dir,$(DIRS),"tests/unit/$(dir)/*.?pp")

+DEPRECATION_WARNING := \
+"This feature is planned for removal in the next release."\
+"Please open an issue at github.com/mfem/mfem/issues if you depend on it."
+deprecation-warnings:
+	@if [ -t 1 ]; then\
+	  red="\033[0;31m";\
+	  yellow="\033[0;33m";\
+	  end="\033[0m";\
+	fi;\
+	if [ $(MFEM_USE_LEGACY_OPENMP) = YES ]; then\
+	  printf $$red"[MFEM_USE_LEGACY_OPENMP]"$$end": "$$yellow"%s"$$end"\n"\
+	  $(DEPRECATION_WARNING);\
+	fi
+
 style:
 	@if ! $(ASTYLE) $(FORMAT_FILES) | grep Formatted; then\
 	   echo "No source files were changed.";\
@@ -33,6 +33,7 @@

 // METIS 4 prototypes
 #if defined(MFEM_USE_METIS) && !defined(MFEM_USE_METIS_5)
+typedef int idx_t;
 typedef int idxtype;
 extern "C" {
   void METIS_PartGraphRecursive(int*, idxtype*, idxtype*, idxtype*, idxtype*,
@@ -418,27 +419,51 @@ void Mesh::GetBdrElementTransformation(int i, IsoparametricTransformation* ElTr)
 {
   ElTr->Attribute = GetBdrAttribute(i);
   ElTr->ElementNo = i; // boundary element number
+   DenseMatrix &pm = ElTr->GetPointMat();
   if (Nodes == NULL)
   {
-      GetBdrPointMatrix(i, ElTr->GetPointMat());
-      ElTr->SetFE(
-         GetTransformationFEforElementType(GetBdrElementType(i)));
+      GetBdrPointMatrix(i, pm);
+      ElTr->SetFE(GetTransformationFEforElementType(GetBdrElementType(i)));
   }
   else
   {
-      DenseMatrix &pm = ElTr->GetPointMat();
-      Array<int> vdofs;
-      Nodes->FESpace()->GetBdrElementVDofs(i, vdofs);
-      int n = vdofs.Size()/spaceDim;
-      pm.SetSize(spaceDim, n);
-      for (int k = 0; k < spaceDim; k++)
+      const FiniteElement *bdr_el = Nodes->FESpace()->GetBE(i);
+      if (bdr_el)
      {
-         for (int j = 0; j < n; j++)
+         Array<int> vdofs;
+         Nodes->FESpace()->GetBdrElementVDofs(i, vdofs);
+         int n = vdofs.Size()/spaceDim;
+         pm.SetSize(spaceDim, n);
+         for (int k = 0; k < spaceDim; k++)
         {
-            pm(k,j) = (*Nodes)(vdofs[n*k+j]);
+            for (int j = 0; j < n; j++)
+            {
+               pm(k,j) = (*Nodes)(vdofs[n*k+j]);
+            }
         }
+         ElTr->SetFE(bdr_el);
+      }
+      else // L2 Nodes (e.g., periodic mesh)
+      {
+         int elem_id, face_info;
+         GetBdrElementAdjacentElement(i, elem_id, face_info);
+
+         GetLocalFaceTransformation(GetBdrElementType(i),
+                                    GetElementType(elem_id),
+                                    FaceElemTr.Loc1.Transf, face_info);
+         // NOTE: FaceElemTr.Loc1 is overwritten here -- used as a temporary
+
+         const FiniteElement *face_el =
+            Nodes->FESpace()->GetTraceElement(elem_id,
+                                              GetBdrElementBaseGeometry(i));
+
+         IntegrationRule eir(face_el->GetDof());
+         FaceElemTr.Loc1.Transform(face_el->GetNodes(), eir);
+         // 'Transformation' is not used
+         Nodes->GetVectorValues(Transformation, eir, pm);
+
+         ElTr->SetFE(face_el);
      }
-      ElTr->SetFE(Nodes->FESpace()->GetBE(i));
   }
   ElTr->FinalizeTransformation();
 }
@@ -1284,17 +1309,14 @@ void Mesh::FinalizeQuadMesh(int generate_edges, int refine,


 #ifdef MFEM_USE_GECKO
-void Mesh::GetGeckoElementReordering(Array<int> &ordering)
+void Mesh::GetGeckoElementReordering(Array<int> &ordering,
+                                     int iterations, int window,
+                                     int period, int seed)
 {
   Gecko::Graph graph;

-   // We will put some accesors in for these later
   Gecko::Functional *functional =
      new Gecko::FunctionalGeometric(); // ordering functional
-   unsigned int iterations = 1;         // number of V cycles
-   unsigned int window = 2;             // initial window size
-   unsigned int period = 1;             // iterations between window increment
-   unsigned int seed = 0;               // random number seed

   // Run through all the elements and insert the nodes in the graph for them
   for (int elemid = 0; elemid < GetNE(); ++elemid)
@@ -2115,7 +2137,7 @@ void Mesh::Finalize(bool refine, bool fix_orientation)
 }

 void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
-                  int generate_edges, double sx, double sy, double sz)
+                  double sx, double sy, double sz, bool sfc_ordering)
 {
   int x, y, z;

@@ -2158,31 +2180,58 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
 #define VTX(XC, YC, ZC) ((XC)+((YC)+(ZC)*(ny+1))*(nx+1))

   // Sets elements and the corresponding indices of vertices
-   for (z = 0; z < nz; z++)
+   if (sfc_ordering && type == Element::HEXAHEDRON)
   {
-      for (y = 0; y < ny; y++)
+      Array<int> sfc;
+      NCMesh::GridSfcOrdering3D(nx, ny, nz, sfc);
+      MFEM_VERIFY(sfc.Size() == 3*nx*ny*nz, "");
+
+      for (int k = 0; k < nx*ny*nz; k++)
      {
-         for (x = 0; x < nx; x++)
+         x = sfc[3*k + 0];
+         y = sfc[3*k + 1];
+         z = sfc[3*k + 2];
+
+         ind[0] = VTX(x  , y  , z  );
+         ind[1] = VTX(x+1, y  , z  );
+         ind[2] = VTX(x+1, y+1, z  );
+         ind[3] = VTX(x  , y+1, z  );
+         ind[4] = VTX(x  , y  , z+1);
+         ind[5] = VTX(x+1, y  , z+1);
+         ind[6] = VTX(x+1, y+1, z+1);
+         ind[7] = VTX(x  , y+1, z+1);
+
+         AddHex(ind, 1);
+      }
+   }
+   else
+   {
+      for (z = 0; z < nz; z++)
+      {
+         for (y = 0; y < ny; y++)
         {
-            ind[0] = VTX(x  , y  , z  );
-            ind[1] = VTX(x+1, y  , z  );
-            ind[2] = VTX(x+1, y+1, z  );
-            ind[3] = VTX(x  , y+1, z  );
-            ind[4] = VTX(x  , y  , z+1);
-            ind[5] = VTX(x+1, y  , z+1);
-            ind[6] = VTX(x+1, y+1, z+1);
-            ind[7] = VTX(x  , y+1, z+1);
-            if (type == Element::TETRAHEDRON)
+            for (x = 0; x < nx; x++)
            {
-               AddHexAsTets(ind, 1);
-            }
-            else if (type == Element::WEDGE)
-            {
-               AddHexAsWedges(ind, 1);
-            }
-            else
-            {
-               AddHex(ind, 1);
+               ind[0] = VTX(x  , y  , z  );
+               ind[1] = VTX(x+1, y  , z  );
+               ind[2] = VTX(x+1, y+1, z  );
+               ind[3] = VTX(x  , y+1, z  );
+               ind[4] = VTX(x  , y  , z+1);
+               ind[5] = VTX(x+1, y  , z+1);
+               ind[6] = VTX(x+1, y+1, z+1);
+               ind[7] = VTX(x  , y+1, z+1);
+               if (type == Element::TETRAHEDRON)
+               {
+                  AddHexAsTets(ind, 1);
+               }
+               else if (type == Element::WEDGE)
+               {
+                  AddHexAsWedges(ind, 1);
+               }
+               else
+               {
+                  AddHex(ind, 1);
+               }
            }
         }
      }
@@ -2191,6 +2240,7 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
   // Sets boundary elements and the corresponding indices of vertices
   // bottom, bdr. attribute 1
   for (y = 0; y < ny; y++)
+   {
      for (x = 0; x < nx; x++)
      {
         ind[0] = VTX(x  , y  , 0);
@@ -2210,8 +2260,10 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
            AddBdrQuad(ind, 1);
         }
      }
+   }
   // top, bdr. attribute 6
   for (y = 0; y < ny; y++)
+   {
      for (x = 0; x < nx; x++)
      {
         ind[0] = VTX(x  , y  , nz);
@@ -2231,8 +2283,10 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
            AddBdrQuad(ind, 6);
         }
      }
+   }
   // left, bdr. attribute 5
   for (z = 0; z < nz; z++)
+   {
      for (y = 0; y < ny; y++)
      {
         ind[0] = VTX(0  , y  , z  );
@@ -2248,8 +2302,10 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
            AddBdrQuad(ind, 5);
         }
      }
+   }
   // right, bdr. attribute 3
   for (z = 0; z < nz; z++)
+   {
      for (y = 0; y < ny; y++)
      {
         ind[0] = VTX(nx, y  , z  );
@@ -2265,8 +2321,10 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
            AddBdrQuad(ind, 3);
         }
      }
+   }
   // front, bdr. attribute 2
   for (x = 0; x < nx; x++)
+   {
      for (z = 0; z < nz; z++)
      {
         ind[0] = VTX(x  , 0, z  );
@@ -2282,8 +2340,10 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
            AddBdrQuad(ind, 2);
         }
      }
+   }
   // back, bdr. attribute 4
   for (x = 0; x < nx; x++)
+   {
      for (z = 0; z < nz; z++)
      {
         ind[0] = VTX(x  , ny, z  );
@@ -2299,6 +2359,9 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
            AddBdrQuad(ind, 4);
         }
      }
+   }
+
+#undef VTX

 #if 0
   ofstream test_stream("debug.mesh");
@@ -2311,8 +2374,9 @@ void Mesh::Make3D(int nx, int ny, int nz, Element::Type type,
   // Finalize(...) can be called after this method, if needed
 }

-void Mesh::Make2D(int nx, int ny, Element::Type type, int generate_edges,
-                  double sx, double sy)
+void Mesh::Make2D(int nx, int ny, Element::Type type,
+                  double sx, double sy,
+                  bool generate_edges, bool sfc_ordering)
 {
   int i, j, k;

@@ -2349,17 +2413,37 @@ void Mesh::Make2D(int nx, int ny, Element::Type type, int generate_edges,
      }

      // Sets elements and the corresponding indices of vertices
-      k = 0;
-      for (j = 0; j < ny; j++)
+      if (sfc_ordering)
      {
-         for (i = 0; i < nx; i++)
+         Array<int> sfc;
+         NCMesh::GridSfcOrdering2D(nx, ny, sfc);
+         MFEM_VERIFY(sfc.Size() == 2*nx*ny, "");
+
+         for (k = 0; k < nx*ny; k++)
         {
+            i = sfc[2*k + 0];
+            j = sfc[2*k + 1];
            ind[0] = i + j*(nx+1);
            ind[1] = i + 1 +j*(nx+1);
            ind[2] = i + 1 + (j+1)*(nx+1);
            ind[3] = i + (j+1)*(nx+1);
            elements[k] = new Quadrilateral(ind);
-            k++;
+         }
+      }
+      else
+      {
+         k = 0;
+         for (j = 0; j < ny; j++)
+         {
+            for (i = 0; i < nx; i++)
+            {
+               ind[0] = i + j*(nx+1);
+               ind[1] = i + 1 +j*(nx+1);
+               ind[2] = i + 1 + (j+1)*(nx+1);
+               ind[3] = i + (j+1)*(nx+1);
+               elements[k] = new Quadrilateral(ind);
+               k++;
+            }
         }
      }

@@ -3151,12 +3235,12 @@ Mesh::Mesh(Mesh *mesh_array[], int num_pieces)
 Mesh::Mesh(Mesh *orig_mesh, int ref_factor, int ref_type)
 {
   Dim = orig_mesh->Dimension();
-   MFEM_VERIFY(ref_factor > 1, "the refinement factor must be > 1");
+   MFEM_VERIFY(ref_factor >= 1, "the refinement factor must be >= 1");
   MFEM_VERIFY(ref_type == BasisType::ClosedUniform ||
               ref_type == BasisType::GaussLobatto, "invalid refinement type");
-   MFEM_VERIFY(Dim == 2 || Dim == 3,
-               "only implemented for Hexahedron and Quadrilateral elements in "
-               "2D/3D");
+   MFEM_VERIFY(Dim == 1 || Dim == 2 || Dim == 3,
+               "only implemented for Segment, Quadrilateral and Hexahedron "
+               "elements in 1D/2D/3D");
   MFEM_VERIFY(orig_mesh->GetNumGeometries(Dim) <= 1,
               "meshes with mixed elements are not supported");

@@ -3165,7 +3249,7 @@ Mesh::Mesh(Mesh *orig_mesh, int ref_factor, int ref_type)
   H1_FECollection rfec(ref_factor, Dim, ref_type);
   FiniteElementSpace rfes(orig_mesh, &rfec);

-   int r_bndr_factor = ref_factor * (Dim == 2 ? 1 : ref_factor);
+   int r_bndr_factor = pow(ref_factor, Dim - 1);
   int r_elem_factor = ref_factor * r_bndr_factor;

   int r_num_vert = rfes.GetNDofs();
@@ -3222,18 +3306,34 @@ Mesh::Mesh(Mesh *orig_mesh, int ref_factor, int ref_type)

      rfes.GetBdrElementDofs(el, rdofs);
      MFEM_ASSERT(rdofs.Size() == RG.RefPts.Size(), "");
-      const int *c2h_map = rfec.GetDofMap(geom);
-      for (int j = 0; j < RG.RefGeoms.Size()/nvert; j++)
+      if (Dim == 1)
      {
-         Element *elem = NewElement(geom);
-         elem->SetAttribute(attrib);
-         int *v = elem->GetVertices();
-         for (int k = 0; k < nvert; k++)
+         // Dim == 1 is a special case because the boundary elements are
+         // zero-dimensional points, and therefore don't have a DofMap
+         for (int j = 0; j < RG.RefGeoms.Size()/nvert; j++)
         {
-            int cid = RG.RefGeoms[k+nvert*j]; // local Cartesian index
-            v[k] = rdofs[c2h_map[cid]];
+            Element *elem = NewElement(geom);
+            elem->SetAttribute(attrib);
+            int *v = elem->GetVertices();
+            v[0] = rdofs[RG.RefGeoms[nvert*j]];
+            AddBdrElement(elem);
+         }
+      }
+      else
+      {
+         const int *c2h_map = rfec.GetDofMap(geom);
+         for (int j = 0; j < RG.RefGeoms.Size()/nvert; j++)
+         {
+            Element *elem = NewElement(geom);
+            elem->SetAttribute(attrib);
+            int *v = elem->GetVertices();
+            for (int k = 0; k < nvert; k++)
+            {
+               int cid = RG.RefGeoms[k+nvert*j]; // local Cartesian index
+               v[k] = rdofs[c2h_map[cid]];
+            }
+            AddBdrElement(elem);
         }
-         AddBdrElement(elem);
      }
   }

@@ -3491,6 +3591,12 @@ void Mesh::SetNodalFESpace(FiniteElementSpace *nfes)
   SetNodalGridFunction(nodes, true);
 }

+void Mesh::EnsureNodes()
+{
+   if (Nodes) { return; }
+   SetCurvature(1, false, -1, Ordering::byVDIM);
+}
+
 void Mesh::SetNodalGridFunction(GridFunction *nodes, bool make_owner)
 {
   GetNodes(*nodes);
@@ -4899,21 +5005,42 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
   }
   else
   {
-      int *I, *J, n;
+      idx_t *I, *J, n;
 #ifndef MFEM_USE_METIS_5
-      int wgtflag = 0;
-      int numflag = 0;
-      int options[5];
+      idx_t wgtflag = 0;
+      idx_t numflag = 0;
+      idx_t options[5];
 #else
-      int ncon = 1;
-      int err;
-      int options[40];
+      idx_t ncon = 1;
+      idx_t err;
+      idx_t options[40];
 #endif
-      int edgecut;
+      idx_t edgecut;
+
+      // In case METIS have been compiled with 64bit indices
+      bool freedata = false;
+      idx_t mparts = (idx_t) nparts;
+      idx_t *mpartitioning;

      n = NumOfElements;
-      I = el_to_el->GetI();
-      J = el_to_el->GetJ();
+      if (sizeof(idx_t) == sizeof(int))
+      {
+         I = (idx_t*) el_to_el->GetI();
+         J = (idx_t*) el_to_el->GetJ();
+         mpartitioning = (idx_t*) partitioning;
+      }
+      else
+      {
+         int *iI = el_to_el->GetI();
+         int *iJ = el_to_el->GetJ();
+         int m = iI[n];
+         I = new idx_t[n+1];
+         J = new idx_t[m];
+         for (int k = 0; k < n+1; k++) { I[k] = iI[k]; }
+         for (int k = 0; k < m; k++) { J[k] = iJ[k]; }
+         mpartitioning = new idx_t[n];
+         freedata = true;
+      }
 #ifndef MFEM_USE_METIS_5
      options[0] = 0;
 #else
@@ -4930,7 +5057,7 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
            // std::sort(J+I[i], J+I[i+1]);

            // Sort in decreasing order, as in previous versions of MFEM.
-            std::sort(J+I[i], J+I[i+1], std::greater<int>());
+            std::sort(J+I[i], J+I[i+1], std::greater<idx_t>());
         }
      }

@@ -4940,30 +5067,30 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
      {
 #ifndef MFEM_USE_METIS_5
         METIS_PartGraphRecursive(&n,
-                                  (idxtype *) I,
-                                  (idxtype *) J,
-                                  (idxtype *) NULL,
-                                  (idxtype *) NULL,
+                                  I,
+                                  J,
+                                  NULL,
+                                  NULL,
                                  &wgtflag,
                                  &numflag,
-                                  &nparts,
+                                  &mparts,
                                  options,
                                  &edgecut,
-                                  (idxtype *) partitioning);
+                                  mpartitioning);
 #else
         err = METIS_PartGraphRecursive(&n,
                                        &ncon,
                                        I,
                                        J,
-                                        (idx_t *) NULL,
-                                        (idx_t *) NULL,
-                                        (idx_t *) NULL,
-                                        &nparts,
-                                        (real_t *) NULL,
-                                        (real_t *) NULL,
+                                        NULL,
+                                        NULL,
+                                        NULL,
+                                        &mparts,
+                                        NULL,
+                                        NULL,
                                        options,
                                        &edgecut,
-                                        partitioning);
+                                        mpartitioning);
         if (err != 1)
            mfem_error("Mesh::GeneratePartitioning: "
                       " error in METIS_PartGraphRecursive!");
@@ -4976,30 +5103,30 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
      {
 #ifndef MFEM_USE_METIS_5
         METIS_PartGraphKway(&n,
-                             (idxtype *) I,
-                             (idxtype *) J,
-                             (idxtype *) NULL,
-                             (idxtype *) NULL,
+                             I,
+                             J,
+                             NULL,
+                             NULL,
                             &wgtflag,
                             &numflag,
-                             &nparts,
+                             &mparts,
                             options,
                             &edgecut,
-                             (idxtype *) partitioning);
+                             mpartitioning);
 #else
         err = METIS_PartGraphKway(&n,
                                   &ncon,
                                   I,
                                   J,
-                                   (idx_t *) NULL,
-                                   (idx_t *) NULL,
-                                   (idx_t *) NULL,
-                                   &nparts,
-                                   (real_t *) NULL,
-                                   (real_t *) NULL,
+                                   NULL,
+                                   NULL,
+                                   NULL,
+                                   &mparts,
+                                   NULL,
+                                   NULL,
                                   options,
                                   &edgecut,
-                                   partitioning);
+                                   mpartitioning);
         if (err != 1)
            mfem_error("Mesh::GeneratePartitioning: "
                       " error in METIS_PartGraphKway!");
@@ -5012,31 +5139,31 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
      {
 #ifndef MFEM_USE_METIS_5
         METIS_PartGraphVKway(&n,
-                              (idxtype *) I,
-                              (idxtype *) J,
-                              (idxtype *) NULL,
-                              (idxtype *) NULL,
+                              I,
+                              J,
+                              NULL,
+                              NULL,
                              &wgtflag,
                              &numflag,
-                              &nparts,
+                              &mparts,
                              options,
                              &edgecut,
-                              (idxtype *) partitioning);
+                              mpartitioning);
 #else
         options[METIS_OPTION_OBJTYPE] = METIS_OBJTYPE_VOL;
         err = METIS_PartGraphKway(&n,
                                   &ncon,
                                   I,
                                   J,
-                                   (idx_t *) NULL,
-                                   (idx_t *) NULL,
-                                   (idx_t *) NULL,
-                                   &nparts,
-                                   (real_t *) NULL,
-                                   (real_t *) NULL,
+                                   NULL,
+                                   NULL,
+                                   NULL,
+                                   &mparts,
+                                   NULL,
+                                   NULL,
                                   options,
                                   &edgecut,
-                                   partitioning);
+                                   mpartitioning);
         if (err != 1)
            mfem_error("Mesh::GeneratePartitioning: "
                       " error in METIS_PartGraphKway!");
@@ -5047,6 +5174,17 @@ int *Mesh::GeneratePartitioning(int nparts, int part_method)
      mfem::out << "Mesh::GeneratePartitioning(...): edgecut = "
                << edgecut << endl;
 #endif
+      nparts = (int) mparts;
+      if (mpartitioning != (idx_t*)partitioning)
+      {
+         for (int k = 0; k<NumOfElements; k++) { partitioning[k] = mpartitioning[k]; }
+      }
+      if (freedata)
+      {
+         delete[] I;
+         delete[] J;
+         delete[] mpartitioning;
+      }
   }

   if (el_to_el)
@@ -5690,6 +5828,7 @@ void Mesh::SetNodes(const Vector &node_coord)
 void Mesh::NewNodes(GridFunction &nodes, bool make_owner)
 {
   if (own_nodes) { delete Nodes; }
+   nodes.Pull();
   Nodes = &nodes;
   spaceDim = Nodes->FESpace()->GetVDim();
   own_nodes = (int)make_owner;
@@ -210,7 +210,7 @@ protected:
   void ReadVTKMesh(std::istream &input, int &curved, int &read_gf,
                    bool &finalize_topo);
   void ReadNURBSMesh(std::istream &input, int &curved, int &read_gf);
-   void ReadInlineMesh(std::istream &input, int generate_edges = 0);
+   void ReadInlineMesh(std::istream &input, bool generate_edges = false);
   void ReadGmshMesh(std::istream &input);
   /* Note NetCDF (optional library) is used for reading cubit files */
 #ifdef MFEM_USE_NETCDF
@@ -408,17 +408,20 @@ protected:

   /** Creates mesh for the parallelepiped [0,sx]x[0,sy]x[0,sz], divided into
       nx*ny*nz hexahedra if type=HEXAHEDRON or into 6*nx*ny*nz tetrahedrons if
-       type=TETRAHEDRON. If generate_edges = 0 (default) edges are not
-       generated, if 1 edges are generated. */
-   void Make3D(int nx, int ny, int nz, Element::Type type, int generate_edges,
-               double sx, double sy, double sz);
+       type=TETRAHEDRON. The parameter @a sfc_ordering controls how the elements
+       (when type=HEXAHEDRON) are ordered: true - use space-filling curve
+       ordering, or false - use lexicographic ordering. */
+   void Make3D(int nx, int ny, int nz, Element::Type type,
+               double sx, double sy, double sz, bool sfc_ordering);

   /** Creates mesh for the rectangle [0,sx]x[0,sy], divided into nx*ny
       quadrilaterals if type = QUADRILATERAL or into 2*nx*ny triangles if
       type = TRIANGLE. If generate_edges = 0 (default) edges are not generated,
-       if 1 edges are generated. */
-   void Make2D(int nx, int ny, Element::Type type, int generate_edges,
-               double sx, double sy);
+       if 1 edges are generated. The parameter @a sfc_ordering controls how the
+       elements (when type=QUADRILATERAL) are ordered: true - use space-filling
+       curve ordering, or false - use lexicographic ordering. */
+   void Make2D(int nx, int ny, Element::Type type, double sx, double sy,
+               bool generate_edges, bool sfc_ordering);

   /// Creates a 1D mesh for the interval [0,sx] divided into n equal intervals.
   void Make1D(int n, double sx = 1.0);
@@ -555,34 +558,46 @@ public:
   /** This is our integration with the Gecko library.  This will call the
       Gecko library to find an element ordering that will increase memory
       coherency by putting elements that are in physical proximity closer in
-       memory. */
-   void GetGeckoElementReordering(Array<int> &ordering);
+       memory. It can also be used to get a space-filling curve ordering for
+       ParNCMesh partitioning.
+       @param[out] ordering Output element ordering.
+       @param[in] iterations Number of V cycles (default 1).
+       @param[in] window Initial window size (default 2).
+       @param[in] period Iterations between window increment (default 1).
+       @param[in] seed Random number seed (default 0). */
+   void GetGeckoElementReordering(Array<int> &ordering,
+                                  int iterations = 1, int window = 2,
+                                  int period = 1, int seed = 0);
 #endif

   /** Rebuilds the mesh with a different order of elements.  The ordering
       vector maps the old element number to the new element number.  This also
-       reorders the vertices and nodes edges and faces along with the elements.  */
+       reorders the vertices and nodes edges and faces along with the elements. */
   void ReorderElements(const Array<int> &ordering, bool reorder_vertices = true);

   /** Creates mesh for the parallelepiped [0,sx]x[0,sy]x[0,sz], divided into
       nx*ny*nz hexahedra if type=HEXAHEDRON or into 6*nx*ny*nz tetrahedrons if
-       type=TETRAHEDRON. If generate_edges = 0 (default) edges are not
-       generated, if 1 edges are generated. */
-   Mesh(int nx, int ny, int nz, Element::Type type, int generate_edges = 0,
-        double sx = 1.0, double sy = 1.0, double sz = 1.0)
+       type=TETRAHEDRON. If sfc_ordering = true (default), elements are ordered
+       along a space-filling curve, instead of row by row and layer by layer.
+       The parameter @a generate_edges is ignored (for now, it is kept for
+       backward compatibility). */
+   Mesh(int nx, int ny, int nz, Element::Type type, bool generate_edges = false,
+        double sx = 1.0, double sy = 1.0, double sz = 1.0,
+        bool sfc_ordering = true)
   {
-      Make3D(nx, ny, nz, type, generate_edges, sx, sy, sz);
+      Make3D(nx, ny, nz, type, sx, sy, sz, sfc_ordering);
      Finalize(true); // refine = true
   }

   /** Creates mesh for the rectangle [0,sx]x[0,sy], divided into nx*ny
       quadrilaterals if type = QUADRILATERAL or into 2*nx*ny triangles if
       type = TRIANGLE. If generate_edges = 0 (default) edges are not generated,
-       if 1 edges are generated. */
-   Mesh(int nx, int ny, Element::Type type, int generate_edges = 0,
-        double sx = 1.0, double sy = 1.0)
+       if 1 edges are generated. If scf_ordering = true (default), elements are
+       ordered along a space-filling curve, instead of row by row. */
+   Mesh(int nx, int ny, Element::Type type, bool generate_edges = false,
+        double sx = 1.0, double sy = 1.0, bool sfc_ordering = true)
   {
-      Make2D(nx, ny, type, generate_edges, sx, sy);
+      Make2D(nx, ny, type, sx, sy, generate_edges, sfc_ordering);
      Finalize(true); // refine = true
   }

@@ -1020,6 +1035,10 @@ public:
   /** Return the FiniteElementSpace on which the current mesh nodes are
       defined or NULL if the mesh does not have nodes. */
   const FiniteElementSpace *GetNodalFESpace() const;
+   /** Make sure that the mesh has valid nodes, i.e. its geometry is described
+       by a vector finite element grid function (even if it is a low-order mesh
+       with straight edges). */
+   void EnsureNodes();

   /** Set the curvature of the mesh nodes using the given polynomial degree,
       'order', and optionally: discontinuous or continuous FE space, 'discont',
@@ -731,7 +731,7 @@ void Mesh::ReadNURBSMesh(std::istream &input, int &curved, int &read_gf)
   }
 }

-void Mesh::ReadInlineMesh(std::istream &input, int generate_edges)
+void Mesh::ReadInlineMesh(std::istream &input, bool generate_edges)
 {
   // Initialize to negative numbers so that we know if they've been set.  We're
   // using Element::POINT as our flag, since we're not going to make a 0D mesh,
@@ -862,7 +862,7 @@ void Mesh::ReadInlineMesh(std::istream &input, int generate_edges)
                  << "   ny = " << ny << "\n"
                  << "   sx = " << sx << "\n"
                  << "   sy = " << sy << "\n");
-      Make2D(nx, ny, type, generate_edges, sx, sy);
+      Make2D(nx, ny, type, sx, sy, generate_edges, true);
   }
   else if (type == Element::TETRAHEDRON || type == Element::WEDGE ||
            type == Element::HEXAHEDRON)
@@ -877,7 +877,8 @@ void Mesh::ReadInlineMesh(std::istream &input, int generate_edges)
                  << "   sx = " << sx << "\n"
                  << "   sy = " << sy << "\n"
                  << "   sz = " << sz << "\n");
-      Make3D(nx, ny, nz, type, generate_edges, sx, sy, sz);
+      Make3D(nx, ny, nz, type, sx, sy, sz, true);
+      // TODO: maybe have an option in the file to control ordering?
   }
   else
   {
@@ -11,10 +11,12 @@

 #include "mesh_headers.hpp"
 #include "../fem/fem.hpp"
+#include "../general/sort_pairs.hpp"

 #include <string>
 #include <cmath>
 #include <climits> // INT_MAX
+#include <map>

 namespace mfem
 {
@@ -31,6 +33,118 @@ const DenseTensor &CoarseFineTransformations::GetPointMatrices(
   return pm_it->second;
 }

+namespace internal
+{
+
+// Used in CoarseFineTransformations::GetCoarseToFineMap() below.
+struct RefType
+{
+   Geometry::Type geom;
+   int num_children;
+   const Pair<int,int> *children;
+
+   RefType(Geometry::Type g, int n, const Pair<int,int> *c)
+      : geom(g), num_children(n), children(c) { }
+
+   bool operator<(const RefType &other) const
+   {
+      if (geom < other.geom) { return true; }
+      if (geom > other.geom) { return false; }
+      if (num_children < other.num_children) { return true; }
+      if (num_children > other.num_children) { return false; }
+      for (int i = 0; i < num_children; i++)
+      {
+         if (children[i].one < other.children[i].one) { return true; }
+         if (children[i].one > other.children[i].one) { return false; }
+      }
+      return false; // everything is equal
+   }
+};
+
+}
+
+void CoarseFineTransformations::GetCoarseToFineMap(
+   const mfem::Mesh &fine_mesh, Table &coarse_to_fine,
+   Array<int> &coarse_to_ref_type, Table &ref_type_to_matrix,
+   Array<mfem::Geometry::Type> &ref_type_to_geom) const
+{
+   const int fine_ne = embeddings.Size();
+   int coarse_ne = -1;
+   for (int i = 0; i < fine_ne; i++)
+   {
+      coarse_ne = std::max(coarse_ne, embeddings[i].parent);
+   }
+   coarse_ne++;
+
+   coarse_to_ref_type.SetSize(coarse_ne);
+   coarse_to_fine.SetDims(coarse_ne, fine_ne);
+
+   Array<int> cf_i(coarse_to_fine.GetI(), coarse_ne+1);
+   Array<Pair<int,int> > cf_j(fine_ne);
+   cf_i = 0;
+   for (int i = 0; i < fine_ne; i++)
+   {
+      cf_i[embeddings[i].parent+1]++;
+   }
+   cf_i.PartialSum();
+   MFEM_ASSERT(cf_i.Last() == cf_j.Size(), "internal error");
+   for (int i = 0; i < fine_ne; i++)
+   {
+      const Embedding &e = embeddings[i];
+      cf_j[cf_i[e.parent]].one = e.matrix; // used as sort key below
+      cf_j[cf_i[e.parent]].two = i;
+      cf_i[e.parent]++;
+   }
+   std::copy_backward(cf_i.begin(), cf_i.end()-1, cf_i.end());
+   cf_i[0] = 0;
+   for (int i = 0; i < coarse_ne; i++)
+   {
+      std::sort(&cf_j[cf_i[i]], cf_j.GetData() + cf_i[i+1]);
+   }
+   for (int i = 0; i < fine_ne; i++)
+   {
+      coarse_to_fine.GetJ()[i] = cf_j[i].two;
+   }
+
+   using internal::RefType;
+   using std::map;
+   using std::pair;
+
+   map<RefType,int> ref_type_map;
+   for (int i = 0; i < coarse_ne; i++)
+   {
+      const int num_children = cf_i[i+1]-cf_i[i];
+      MFEM_ASSERT(num_children > 0, "");
+      const int fine_el = cf_j[cf_i[i]].two;
+      // Assuming the coarse and the fine elements have the same geometry:
+      const Geometry::Type geom = fine_mesh.GetElementBaseGeometry(fine_el);
+      const RefType ref_type(geom, num_children, &cf_j[cf_i[i]]);
+      pair<map<RefType,int>::iterator,bool> res =
+         ref_type_map.insert(
+            pair<const RefType,int>(ref_type, (int)ref_type_map.size()));
+      coarse_to_ref_type[i] = res.first->second;
+   }
+   ref_type_to_matrix.MakeI((int)ref_type_map.size());
+   ref_type_to_geom.SetSize((int)ref_type_map.size());
+   for (map<RefType,int>::iterator it = ref_type_map.begin();
+        it != ref_type_map.end(); ++it)
+   {
+      ref_type_to_matrix.AddColumnsInRow(it->second, it->first.num_children);
+      ref_type_to_geom[it->second] = it->first.geom;
+   }
+   ref_type_to_matrix.MakeJ();
+   for (map<RefType,int>::iterator it = ref_type_map.begin();
+        it != ref_type_map.end(); ++it)
+   {
+      const RefType &rt = it->first;
+      for (int j = 0; j < rt.num_children; j++)
+      {
+         ref_type_to_matrix.AddConnection(it->second, rt.children[j].one);
+      }
+   }
+   ref_type_to_matrix.ShiftUpI();
+}
+
 NCMesh::GeomInfo NCMesh::GI[Geometry::NumGeom];

 NCMesh::GeomInfo& NCMesh::gi_hex  = NCMesh::GI[Geometry::CUBE];
@@ -120,8 +234,7 @@ NCMesh::NCMesh(const Mesh *mesh, std::istream *vertex_parents)
   }

   // create the NCMesh::Element struct for each Mesh element
-   root_count = mesh->GetNE();
-   for (int i = 0; i < root_count; i++)
+   for (int i = 0; i < mesh->GetNE(); i++)
   {
      const mfem::Element *elem = mesh->GetElement(i);

@@ -180,6 +293,11 @@ NCMesh::NCMesh(const Mesh *mesh, std::istream *vertex_parents)
      }
   }

+   if (!vertex_parents) // not loading mesh
+   {
+      InitRootState(mesh->GetNE());
+   }
+
   Update();
 }

@@ -190,9 +308,9 @@ NCMesh::NCMesh(const NCMesh &other)
   , nodes(other.nodes)
   , faces(other.faces)
   , elements(other.elements)
-   , root_count(other.root_count)
 {
   other.free_element_ids.Copy(free_element_ids);
+   other.root_state.Copy(root_state);
   other.top_vertex_pos.Copy(top_vertex_pos);
   Update();
 }
@@ -1128,6 +1246,53 @@ void NCMesh::Refine(const Array<Refinement>& refinements)

 //// Derefinement //////////////////////////////////////////////////////////////

+static int quad_deref_table[3][4 + 4] =
+{
+   { 0, 1, 1, 0, /**/ 1, 1, 0, 0 }, // 1 - X
+   { 0, 0, 1, 1, /**/ 0, 0, 1, 1 }, // 2 - Y
+   { 0, 1, 2, 3, /**/ 1, 1, 3, 3 }  // 3 - iso
+};
+static int hex_deref_table[7][8 + 6] =
+{
+   { 0, 1, 1, 0, 0, 1, 1, 0, /**/ 1, 1, 1, 0, 0, 0 }, // 1 - X
+   { 0, 0, 1, 1, 0, 0, 1, 1, /**/ 0, 0, 0, 1, 1, 1 }, // 2 - Y
+   { 0, 1, 2, 3, 0, 1, 2, 3, /**/ 1, 1, 1, 3, 3, 3 }, // 3 - XY
+   { 0, 0, 0, 0, 1, 1, 1, 1, /**/ 0, 0, 0, 1, 1, 1 }, // 4 - Z
+   { 0, 1, 1, 0, 3, 2, 2, 3, /**/ 1, 1, 1, 3, 3, 3 }, // 5 - XZ
+   { 0, 0, 1, 1, 2, 2, 3, 3, /**/ 0, 0, 0, 3, 3, 3 }, // 6 - YZ
+   { 0, 1, 2, 3, 4, 5, 6, 7, /**/ 1, 1, 1, 7, 7, 7 }  // 7 - iso
+};
+
+
+int NCMesh::RetrieveNode(const Element &el, int index)
+{
+   if (!el.ref_type) { return el.node[index]; }
+
+   // need to retrieve node from a child element (there is always a child
+   // that inherited the parent's corner under the same index)
+   int ch;
+   switch (el.geom)
+   {
+      case Geometry::CUBE:
+         ch = el.child[hex_deref_table[el.ref_type - 1][index]];
+         break;
+
+      case Geometry::SQUARE:
+         ch = el.child[quad_deref_table[el.ref_type - 1][index]];
+         break;
+
+      case Geometry::TRIANGLE:
+         ch = el.child[index];
+         break;
+
+      default:
+         ch = 0; // suppress compiler warning
+         MFEM_ABORT("Unsupported element geometry.");
+   }
+   return RetrieveNode(elements[ch], index);
+}
+
+
 void NCMesh::DerefineElement(int elem)
 {
   Element &el = elements[elem];
@@ -1149,23 +1314,14 @@ void NCMesh::DerefineElement(int elem)
   int fa[6];
   if (el.geom == Geometry::CUBE)
   {
-      const int table[7][8 + 6] =
-      {
-         { 0, 1, 1, 0, 0, 1, 1, 0, /**/ 1, 1, 1, 0, 0, 0 }, // 1 - X
-         { 0, 0, 1, 1, 0, 0, 1, 1, /**/ 0, 0, 0, 1, 1, 1 }, // 2 - Y
-         { 0, 1, 2, 3, 0, 1, 2, 3, /**/ 1, 1, 1, 3, 3, 3 }, // 3 - XY
-         { 0, 0, 0, 0, 1, 1, 1, 1, /**/ 0, 0, 0, 1, 1, 1 }, // 4 - Z
-         { 0, 1, 1, 0, 3, 2, 2, 3, /**/ 1, 1, 1, 3, 3, 3 }, // 5 - XZ
-         { 0, 0, 1, 1, 2, 2, 3, 3, /**/ 0, 0, 0, 3, 3, 3 }, // 6 - YZ
-         { 0, 1, 2, 3, 4, 5, 6, 7, /**/ 1, 1, 1, 7, 7, 7 }  // 7 - iso
-      };
      for (int i = 0; i < 8; i++)
      {
-         el.node[i] = elements[child[table[el.ref_type - 1][i]]].node[i];
+         Element &ch = elements[child[hex_deref_table[el.ref_type - 1][i]]];
+         el.node[i] = ch.node[i];
      }
      for (int i = 0; i < 6; i++)
      {
-         Element &ch = elements[child[table[el.ref_type - 1][i + 8]]];
+         Element &ch = elements[child[hex_deref_table[el.ref_type - 1][i + 8]]];
         const int* fv = gi_hex.faces[i];
         fa[i] = faces.Find(ch.node[fv[0]], ch.node[fv[1]],
                            ch.node[fv[2]], ch.node[fv[3]])->attribute;
@@ -1173,19 +1329,14 @@ void NCMesh::DerefineElement(int elem)
   }
   else if (el.geom == Geometry::SQUARE)
   {
-      const int table[3][4 + 4] =
-      {
-         { 0, 1, 1, 0, /**/ 1, 1, 0, 0 }, // 1 - X
-         { 0, 0, 1, 1, /**/ 0, 0, 1, 1 }, // 2 - Y
-         { 0, 1, 2, 3, /**/ 1, 1, 3, 3 }  // 3 - iso
-      };
      for (int i = 0; i < 4; i++)
      {
-         el.node[i] = elements[child[table[el.ref_type - 1][i]]].node[i];
+         Element &ch = elements[child[quad_deref_table[el.ref_type - 1][i]]];
+         el.node[i] = ch.node[i];
      }
      for (int i = 0; i < 4; i++)
      {
-         Element &ch = elements[child[table[el.ref_type - 1][i + 4]]];
+         Element &ch = elements[child[quad_deref_table[el.ref_type - 1][i + 4]]];
         const int* fv = gi_quad.faces[i];
         fa[i] = faces.Find(ch.node[fv[0]], ch.node[fv[1]],
                            ch.node[fv[2]], ch.node[fv[3]])->attribute;
@@ -1272,7 +1423,7 @@ const Table& NCMesh::GetDerefinementTable()
   Array<Connection> list;
   list.Reserve(leaf_elements.Size());

-   for (int i = 0; i < root_count; i++)
+   for (int i = 0; i < root_state.Size(); i++)
   {
      CollectDerefinements(i, list);
   }
@@ -1482,9 +1633,9 @@ void NCMesh::UpdateLeafElements()
 {
   // collect leaf elements from all roots
   leaf_elements.SetSize(0);
-   for (int i = 0; i < root_count; i++)
+   for (int i = 0; i < root_state.Size(); i++)
   {
-      CollectLeafElements(i, 0);
+      CollectLeafElements(i, root_state[i]);
      // TODO: root state should not always be 0, we need a precomputed array
      // with root element states to ensure continuity where possible, also
      // optimized ordering of the root elements themselves (Gecko?)
@@ -1501,6 +1652,69 @@ void NCMesh::AssignLeafIndices()
   }
 }

+void NCMesh::InitRootState(int root_count)
+{
+   root_state.SetSize(root_count);
+   root_state = 0;
+
+   char* node_order;
+   int nch;
+
+   switch (GetElementGeometry())
+   {
+      case Geometry::SQUARE:
+         nch = 4;
+         node_order = (char*) quad_hilbert_child_order;
+         break;
+
+      case Geometry::CUBE:
+         nch = 8;
+         node_order = (char*) hex_hilbert_child_order;
+         break;
+
+      default:
+         return; // do nothing, all states stay zero
+   }
+
+   int entry_node = -2;
+
+   // process the root element sequence
+   for (int i = 0; i < root_count; i++)
+   {
+      Element &el = elements[i];
+
+      int v_in = FindNodeExt(el, entry_node, false);
+      if (v_in < 0) { v_in = 0; }
+
+      // determine which nodes are shared with the next element
+      bool shared[8] = { 0, 0, 0, 0, 0, 0, 0, 0 };
+      if (i+1 < root_count)
+      {
+         Element &next = elements[i+1];
+         for (int j = 0; j < nch; j++)
+         {
+            int node = FindNodeExt(el, RetrieveNode(next, j), false);
+            if (node >= 0) { shared[node] = true; }
+         }
+      }
+
+      // select orientation that starts in v_in and exits in shared node
+      int state = Dim*v_in;
+      for (int j = 0; j < Dim; j++)
+      {
+         if (shared[(int) node_order[nch*(state + j) + nch-1]])
+         {
+            state += j;
+            break;
+         }
+      }
+
+      root_state[i] = state;
+
+      entry_node = RetrieveNode(el, node_order[nch*state + nch-1]);
+   }
+}
+
 mfem::Element* NCMesh::NewMeshElement(int geom) const
 {
   switch (geom)
@@ -1703,6 +1917,16 @@ int NCMesh::find_node(const Element &el, int node)
   return -1;
 }

+int NCMesh::FindNodeExt(const Element &el, int node, bool abort)
+{
+   for (int i = 0; i < GI[(int) el.geom].nv; i++)
+   {
+      if (RetrieveNode(el, i) == node) { return i; }
+   }
+   if (abort) { MFEM_ABORT("Node not found."); }
+   return -1;
+}
+
 int NCMesh::find_element_edge(const Element &el, int vn0, int vn1)
 {
   MFEM_ASSERT(!el.ref_type, "");
@@ -1939,7 +2163,7 @@ void NCMesh::BuildEdgeList()
   processed_edges = 0;

   Array<int> edge_element(nodes.NumIds());
-   Array<char> edge_local(nodes.NumIds());
+   Array<signed char> edge_local(nodes.NumIds());
   edge_local = -1;

   // visit edges of leaf elements
@@ -2198,7 +2422,7 @@ void NCMesh::CollectFaceVertices(int v0, int v1, int v2, int v3,
 void NCMesh::BuildElementToVertexTable()
 {
   int nrows = leaf_elements.Size();
-   int* I = new int[nrows + 1];
+   int* I = mfem::New<int>(nrows + 1);
   int** JJ = new int*[nrows];

   Array<int> indices;
@@ -2236,8 +2460,8 @@ void NCMesh::BuildElementToVertexTable()
      indices.Unique();
      int size = indices.Size();
      I[i] = size;
-      JJ[i] = new int[size];
-      memcpy(JJ[i], indices.GetData(), size * sizeof(int));
+      JJ[i] = mfem::New<int>(size);
+      std::memcpy(JJ[i], indices.GetData(), size * sizeof(int));
   }

   // finalize the I array of the table
@@ -2251,13 +2475,13 @@ void NCMesh::BuildElementToVertexTable()
   I[nrows] = nnz;

   // copy the temporarily stored rows into one J array
-   int *J = new int[nnz];
+   int *J = mfem::New<int>(nnz);
   nnz = 0;
   for (int i = 0; i < nrows; i++)
   {
      int cnt = I[i+1] - I[i];
-      memcpy(J+nnz, JJ[i], cnt * sizeof(int));
-      delete [] JJ[i];
+      std::memcpy(J+nnz, JJ[i], cnt * sizeof(int));
+      mfem::Delete(JJ[i]);
      nnz += cnt;
   }

@@ -2987,6 +3211,273 @@ void NCMesh::ClearTransforms()
 }


+//// SFC Ordering //////////////////////////////////////////////////////////////
+
+static int sgn(int x)
+{
+   return (x < 0) ? -1 : (x > 0) ? 1 : 0;
+}
+
+static void HilbertSfc2D(int x, int y, int ax, int ay, int bx, int by,
+                         Array<int> &coords)
+{
+   int w = std::abs(ax + ay);
+   int h = std::abs(bx + by);
+
+   int dax = sgn(ax), day = sgn(ay); // unit major direction ("right")
+   int dbx = sgn(bx), dby = sgn(by); // unit orthogonal direction ("up")
+
+   if (h == 1) // trivial row fill
+   {
+      for (int i = 0; i < w; i++, x += dax, y += day)
+      {
+         coords.Append(x);
+         coords.Append(y);
+      }
+      return;
+   }
+   if (w == 1) // trivial column fill
+   {
+      for (int i = 0; i < h; i++, x += dbx, y += dby)
+      {
+         coords.Append(x);
+         coords.Append(y);
+      }
+      return;
+   }
+
+   int ax2 = ax/2, ay2 = ay/2;
+   int bx2 = bx/2, by2 = by/2;
+
+   int w2 = std::abs(ax2 + ay2);
+   int h2 = std::abs(bx2 + by2);
+
+   if (2*w > 3*h) // long case: split in two parts only
+   {
+      if ((w2 & 0x1) && (w > 2))
+      {
+         ax2 += dax, ay2 += day; // prefer even steps
+      }
+
+      HilbertSfc2D(x, y, ax2, ay2, bx, by, coords);
+      HilbertSfc2D(x+ax2, y+ay2, ax-ax2, ay-ay2, bx, by, coords);
+   }
+   else // standard case: one step up, one long horizontal step, one step down
+   {
+      if ((h2 & 0x1) && (h > 2))
+      {
+         bx2 += dbx, by2 += dby; // prefer even steps
+      }
+
+      HilbertSfc2D(x, y, bx2, by2, ax2, ay2, coords);
+      HilbertSfc2D(x+bx2, y+by2, ax, ay, bx-bx2, by-by2, coords);
+      HilbertSfc2D(x+(ax-dax)+(bx2-dbx), y+(ay-day)+(by2-dby),
+                   -bx2, -by2, -(ax-ax2), -(ay-ay2), coords);
+   }
+}
+
+static void HilbertSfc3D(int x, int y, int z,
+                         int ax, int ay, int az,
+                         int bx, int by, int bz,
+                         int cx, int cy, int cz,
+                         Array<int> &coords)
+{
+   int w = std::abs(ax + ay + az);
+   int h = std::abs(bx + by + bz);
+   int d = std::abs(cx + cy + cz);
+
+   int dax = sgn(ax), day = sgn(ay), daz = sgn(az); // unit major dir ("right")
+   int dbx = sgn(bx), dby = sgn(by), dbz = sgn(bz); // unit ortho dir ("forward")
+   int dcx = sgn(cx), dcy = sgn(cy), dcz = sgn(cz); // unit ortho dir ("up")
+
+   // trivial row/column fills
+   if (h == 1 && d == 1)
+   {
+      for (int i = 0; i < w; i++, x += dax, y += day, z += daz)
+      {
+         coords.Append(x);
+         coords.Append(y);
+         coords.Append(z);
+      }
+      return;
+   }
+   if (w == 1 && d == 1)
+   {
+      for (int i = 0; i < h; i++, x += dbx, y += dby, z += dbz)
+      {
+         coords.Append(x);
+         coords.Append(y);
+         coords.Append(z);
+      }
+      return;
+   }
+   if (w == 1 && h == 1)
+   {
+      for (int i = 0; i < d; i++, x += dcx, y += dcy, z += dcz)
+      {
+         coords.Append(x);
+         coords.Append(y);
+         coords.Append(z);
+      }
+      return;
+   }
+
+   int ax2 = ax/2, ay2 = ay/2, az2 = az/2;
+   int bx2 = bx/2, by2 = by/2, bz2 = bz/2;
+   int cx2 = cx/2, cy2 = cy/2, cz2 = cz/2;
+
+   int w2 = std::abs(ax2 + ay2 + az2);
+   int h2 = std::abs(bx2 + by2 + bz2);
+   int d2 = std::abs(cx2 + cy2 + cz2);
+
+   // prefer even steps
+   if ((w2 & 0x1) && (w > 2))
+   {
+      ax2 += dax, ay2 += day, az2 += daz;
+   }
+   if ((h2 & 0x1) && (h > 2))
+   {
+      bx2 += dbx, by2 += dby, bz2 += dbz;
+   }
+   if ((d2 & 0x1) && (d > 2))
+   {
+      cx2 += dcx, cy2 += dcy, cz2 += dcz;
+   }
+
+   // wide case, split in w only
+   if ((2*w > 3*h) && (2*w > 3*d))
+   {
+      HilbertSfc3D(x, y, z,
+                   ax2, ay2, az2,
+                   bx, by, bz,
+                   cx, cy, cz, coords);
+
+      HilbertSfc3D(x+ax2, y+ay2, z+az2,
+                   ax-ax2, ay-ay2, az-az2,
+                   bx, by, bz,
+                   cx, cy, cz, coords);
+   }
+   // do not split in d
+   else if (3*h > 4*d)
+   {
+      HilbertSfc3D(x, y, z,
+                   bx2, by2, bz2,
+                   cx, cy, cz,
+                   ax2, ay2, az2, coords);
+
+      HilbertSfc3D(x+bx2, y+by2, z+bz2,
+                   ax, ay, az,
+                   bx-bx2, by-by2, bz-bz2,
+                   cx, cy, cz, coords);
+
+      HilbertSfc3D(x+(ax-dax)+(bx2-dbx),
+                   y+(ay-day)+(by2-dby),
+                   z+(az-daz)+(bz2-dbz),
+                   -bx2, -by2, -bz2,
+                   cx, cy, cz,
+                   -(ax-ax2), -(ay-ay2), -(az-az2), coords);
+   }
+   // do not split in h
+   else if (3*d > 4*h)
+   {
+      HilbertSfc3D(x, y, z,
+                   cx2, cy2, cz2,
+                   ax2, ay2, az2,
+                   bx, by, bz, coords);
+
+      HilbertSfc3D(x+cx2, y+cy2, z+cz2,
+                   ax, ay, az,
+                   bx, by, bz,
+                   cx-cx2, cy-cy2, cz-cz2, coords);
+
+      HilbertSfc3D(x+(ax-dax)+(cx2-dcx),
+                   y+(ay-day)+(cy2-dcy),
+                   z+(az-daz)+(cz2-dcz),
+                   -cx2, -cy2, -cz2,
+                   -(ax-ax2), -(ay-ay2), -(az-az2),
+                   bx, by, bz, coords);
+   }
+   // regular case, split in all w/h/d
+   else
+   {
+      HilbertSfc3D(x, y, z,
+                   bx2, by2, bz2,
+                   cx2, cy2, cz2,
+                   ax2, ay2, az2, coords);
+
+      HilbertSfc3D(x+bx2, y+by2, z+bz2,
+                   cx, cy, cz,
+                   ax2, ay2, az2,
+                   bx-bx2, by-by2, bz-bz2, coords);
+
+      HilbertSfc3D(x+(bx2-dbx)+(cx-dcx),
+                   y+(by2-dby)+(cy-dcy),
+                   z+(bz2-dbz)+(cz-dcz),
+                   ax, ay, az,
+                   -bx2, -by2, -bz2,
+                   -(cx-cx2), -(cy-cy2), -(cz-cz2), coords);
+
+      HilbertSfc3D(x+(ax-dax)+bx2+(cx-dcx),
+                   y+(ay-day)+by2+(cy-dcy),
+                   z+(az-daz)+bz2+(cz-dcz),
+                   -cx, -cy, -cz,
+                   -(ax-ax2), -(ay-ay2), -(az-az2),
+                   bx-bx2, by-by2, bz-bz2, coords);
+
+      HilbertSfc3D(x+(ax-dax)+(bx2-dbx),
+                   y+(ay-day)+(by2-dby),
+                   z+(az-daz)+(bz2-dbz),
+                   -bx2, -by2, -bz2,
+                   cx2, cy2, cz2,
+                   -(ax-ax2), -(ay-ay2), -(az-az2), coords);
+   }
+}
+
+void NCMesh::GridSfcOrdering2D(int width, int height, Array<int> &coords)
+{
+   coords.SetSize(0);
+   coords.Reserve(2*width*height);
+
+   if (width >= height)
+   {
+      HilbertSfc2D(0, 0, width, 0, 0, height, coords);
+   }
+   else
+   {
+      HilbertSfc2D(0, 0, 0, height, width, 0, coords);
+   }
+}
+
+void NCMesh::GridSfcOrdering3D(int width, int height, int depth,
+                               Array<int> &coords)
+{
+   coords.SetSize(0);
+   coords.Reserve(3*width*height*depth);
+
+   if (width >= height && width >= depth)
+   {
+      HilbertSfc3D(0, 0, 0,
+                   width, 0, 0,
+                   0, height, 0,
+                   0, 0, depth, coords);
+   }
+   else if (height >= width && height >= depth)
+   {
+      HilbertSfc3D(0, 0, 0,
+                   0, height, 0,
+                   width, 0, 0,
+                   0, 0, depth, coords);
+   }
+   else // depth >= width && depth >= height
+   {
+      HilbertSfc3D(0, 0, 0,
+                   0, 0, depth,
+                   width, 0, 0,
+                   0, height, 0, coords);
+   }
+}
+
+
 //// Utility ///////////////////////////////////////////////////////////////////

 void NCMesh::GetEdgeVertices(const MeshId &edge_id, int vert_index[2],
@@ -3420,7 +3911,7 @@ void NCMesh::PrintCoarseElements(std::ostream &out) const

   // print the hierarchy recursively
   int coarse_id = leaf_elements.Size();
-   for (int i = 0; i < root_count; i++)
+   for (int i = 0; i < root_state.Size(); i++)
   {
      PrintElements(out, i, coarse_id);
   }
@@ -3500,7 +3991,7 @@ void NCMesh::LoadCoarseElements(std::istream &input)
   index_map = -1;

   // copy roots, they need to be at the beginning of 'elements'
-   root_count = 0;
+   int root_count = 0;
   for (elem_iterator el = tmp_elements.begin(); el != tmp_elements.end(); ++el)
   {
      if (el->parent == -1)
@@ -3533,6 +4024,8 @@ void NCMesh::LoadCoarseElements(std::istream &input)
   // set the Iso flag (must be false if there are 3D aniso refinements)
   Iso = iso;

+   InitRootState(root_count);
+
   Update();
 }

@@ -3582,6 +4075,7 @@ long NCMesh::MemoryUsage() const
          faces.MemoryUsage() +
          elements.MemoryUsage() +
          free_element_ids.MemoryUsage() +
+          root_state.MemoryUsage() +
          top_vertex_pos.MemoryUsage() +
          leaf_elements.MemoryUsage() +
          vertex_nodeId.MemoryUsage() +
@@ -3604,6 +4098,7 @@ int NCMesh::PrintMemoryDetail() const

   mfem::out << elements.MemoryUsage() << " elements\n"
             << free_element_ids.MemoryUsage() << " free_element_ids\n"
+             << root_state.MemoryUsage() << " root_state\n"
             << top_vertex_pos.MemoryUsage() << " top_vertex_pos\n"
             << leaf_elements.MemoryUsage() << " leaf_elements\n"
             << vertex_nodeId.MemoryUsage() << " vertex_nodeId\n"
@@ -3646,7 +4141,8 @@ void NCMesh::PrintStats(std::ostream &out) const
           free_element_ids.MemoryUsage())/MiB << " MiB ]\n"
       "      free                     " << std::setw(9)
       << free_element_ids.Size() << "\n"
-       "   number of root elements   : " << std::setw(9) << root_count << "\n"
+       "   number of root elements   : " << std::setw(9)
+       << root_state.Size() << "\n"
       "   number of leaf elements   : " << std::setw(9)
       << leaf_elements.Size() << "\n"
       "   number of vertices        : " << std::setw(9)
@@ -3675,6 +4171,31 @@ void NCMesh::PrintStats(std::ostream &out) const
 }

 #ifdef MFEM_DEBUG
+void NCMesh::DebugLeafOrder(std::ostream &out) const
+{
+   tmp_vertex = new TmpVertex[nodes.NumIds()];
+   for (int i = 0; i < leaf_elements.Size(); i++)
+   {
+      const Element* elem = &elements[leaf_elements[i]];
+      for (int j = 0; j < Dim; j++)
+      {
+         double sum = 0.0;
+         int count = 0;
+         for (int k = 0; k < 8; k++)
+         {
+            if (elem->node[k] >= 0)
+            {
+               sum += CalcVertexPos(elem->node[k])[j];
+               count++;
+            }
+         }
+         out << sum / count << " ";
+      }
+      out << "\n";
+   }
+   delete [] tmp_vertex;
+}
+
 void NCMesh::DebugDump(std::ostream &out) const
 {
   // dump nodes
@@ -60,6 +60,13 @@ struct CoarseFineTransformations
   Array<Embedding> embeddings;

   const DenseTensor &GetPointMatrices(Geometry::Type geom) const;
+
+   void GetCoarseToFineMap(const Mesh &fine_mesh,
+                           Table &coarse_to_fine,
+                           Array<int> &coarse_to_ref_type,
+                           Table &ref_type_to_matrix,
+                           Array<Geometry::Type> &ref_type_to_geom) const;
+
   void Clear() { point_matrices.clear(); embeddings.DeleteAll(); }
   long MemoryUsage() const;
 };
@@ -245,6 +252,24 @@ public:
   void ClearTransforms();


+   // grid ordering
+
+   /** Return a space filling curve for a rectangular grid of elements.
+       Implemented is a generalized Hilbert curve for arbitrary grid dimensions.
+       If the width is odd, height should be odd too, otherwise one diagonal
+       (vertex-neighbor) step cannot be avoided in the curve. Even dimensions
+       are recommended. */
+   static void GridSfcOrdering2D(int width, int height,
+                                 Array<int> &coords);
+
+   /** Return a space filling curve for a 3D rectangular grid of elements.
+       The Hilbert-curve-like algorithm works well for even dimensions. For odd
+       width/height/depth it tends to produce some diagonal (edge-neighbor)
+       steps. Even dimensions are recommended. */
+   static void GridSfcOrdering3D(int width, int height, int depth,
+                                 Array<int> &coords);
+
+
   // utility

   /// Return Mesh vertex indices of an edge identified by 'edge_id'.
@@ -367,7 +392,7 @@ protected: // implementation
      Face() : attribute(-1), index(-1) { elem[0] = elem[1] = -1; }

      bool Boundary() const { return attribute >= 0; }
-      bool Unused() const { return elem[0] < 0 && elem[1] < 0;}
+      bool Unused() const { return elem[0] < 0 && elem[1] < 0; }

      // add or remove an element from the 'elem[2]' array
      void RegisterElement(int e);
@@ -406,10 +431,12 @@ protected: // implementation
   BlockArray<Element> elements; // storage for all Elements
   Array<int> free_element_ids;  // unused element ids - indices into 'elements'

-   // the first 'root_count' entries of 'elements' is the coarse mesh
-   int root_count;
+   /** Initial traversal state (~ element orientation) for each root element
+       NOTE: M = root_state.Size() is the number of root elements.
+       NOTE: the first M items of 'elements' is the coarse mesh. */
+   Array<int> root_state;

-   // coordinates of top-level vertices (organized as triples)
+   /// coordinates of top-level vertices (organized as triples)
   Array<double> top_vertex_pos;

   typedef HashTable<Node>::iterator node_iterator;
@@ -450,6 +477,12 @@ protected: // implementation

   virtual void AssignLeafIndices();

+   /** Try to find a space-filling curve friendly orientation of the root
+       elements: set 'root_state' based on the ordering of coarse elements.
+       Note that the coarse mesh itself must be ordered as an SFC by e.g.
+       Mesh::GetGeckoElementReordering. */
+   void InitRootState(int root_count);
+
   virtual bool IsGhost(const Element &el) const { return false; }
   virtual int GetNumGhostElements() const { return 0; }
   virtual int GetNumGhostVertices() const { return 0; }
@@ -528,6 +561,12 @@ protected: // implementation

   void CollectDerefinements(int elem, Array<Connection> &list);

+   /// Return el.node[index] correctly, even if the element is refined.
+   int RetrieveNode(const Element &el, int index);
+
+   /// Extended version of find_node: works if 'el' is refined; optional abort.
+   int FindNodeExt(const Element &el, int node, bool abort = false);
+

   // face/edge lists

@@ -737,6 +776,7 @@ protected: // implementation

 #ifdef MFEM_DEBUG
 public:
+   void DebugLeafOrder(std::ostream &out) const;
   void DebugDump(std::ostream &out) const;
 #endif

@@ -1668,12 +1668,14 @@ void NURBSExtension::ConnectBoundaries()

 void NURBSExtension::ConnectBoundaries2D(int bnd0, int bnd1)
 {
-   int idx0, idx1;
+   int idx0 = -1, idx1 = -1;
   for (int b = 0; b < GetNBP(); b++)
   {
      if (bnd0 == patchTopo->GetBdrAttribute(b)) { idx0 = b; }
      if (bnd1 == patchTopo->GetBdrAttribute(b)) { idx1 = b; }
   }
+   MFEM_VERIFY(idx0 != -1,"Bdr 0 not found");
+   MFEM_VERIFY(idx1 != -1,"Bdr 1 not found");

   NURBSPatchMap p2g0(this);
   NURBSPatchMap p2g1(this);
@@ -1742,12 +1744,14 @@ void NURBSExtension::ConnectBoundaries2D(int bnd0, int bnd1)

 void NURBSExtension::ConnectBoundaries3D(int bnd0, int bnd1)
 {
-   int idx0, idx1;
+   int idx0 = -1, idx1 = -1;
   for (int b = 0; b < GetNBP(); b++)
   {
      if (bnd0 == patchTopo->GetBdrAttribute(b)) { idx0 = b; }
      if (bnd1 == patchTopo->GetBdrAttribute(b)) { idx1 = b; }
   }
+   MFEM_VERIFY(idx0 != -1,"Bdr 0 not found");
+   MFEM_VERIFY(idx1 != -1,"Bdr 1 not found");

   NURBSPatchMap p2g0(this);
   NURBSPatchMap p2g1(this);
@@ -2988,6 +2988,8 @@ void ParMesh::NonconformingRefinement(const Array<Refinement> &refinements,
                 "serial Mesh)");
   }

+   DeleteFaceNbrData();
+
   // NOTE: no check of !refinements.Size(), in parallel we would have to reduce

   // do the refinements
@@ -3088,6 +3090,23 @@ void ParMesh::Rebalance()
                 " meshes.");
   }

+   // Make sure the Nodes use a ParFiniteElementSpace
+   if (Nodes && dynamic_cast<ParFiniteElementSpace*>(Nodes->FESpace()) == NULL)
+   {
+      ParFiniteElementSpace *pfes =
+         new ParFiniteElementSpace(*Nodes->FESpace(), *this);
+      ParGridFunction *new_nodes = new ParGridFunction(pfes);
+      *new_nodes = *Nodes;
+      if (Nodes->OwnFEC())
+      {
+         new_nodes->MakeOwner(Nodes->OwnFEC());
+         Nodes->MakeOwner(NULL); // takes away ownership of 'fec' and 'fes'
+         delete Nodes->FESpace();
+      }
+      delete Nodes;
+      Nodes = new_nodes;
+   }
+
   DeleteFaceNbrData();

   pncmesh->Rebalance();
@@ -3128,16 +3147,16 @@ void ParMesh::RefineGroups(const DSTable &v_to_v, int *middle)
   int *I_group_svert, *J_group_svert;
   int *I_group_sedge, *J_group_sedge;

-   I_group_svert = new int[GetNGroups()+1];
-   I_group_sedge = new int[GetNGroups()+1];
+   I_group_svert = mfem::New<int>(GetNGroups()+1);
+   I_group_sedge = mfem::New<int>(GetNGroups()+1);

   I_group_svert[0] = I_group_svert[1] = 0;
   I_group_sedge[0] = I_group_sedge[1] = 0;

   // overestimate the size of the J arrays
-   J_group_svert = new int[group_svert.Size_of_connections()
-                           + group_sedge.Size_of_connections()];
-   J_group_sedge = new int[2*group_sedge.Size_of_connections()];
+   J_group_svert = mfem::New<int>(group_svert.Size_of_connections()
+                                  + group_sedge.Size_of_connections());
+   J_group_sedge = mfem::New<int>(2*group_sedge.Size_of_connections());

   for (int group = 0; group < GetNGroups()-1; group++)
   {
@@ -3375,16 +3394,16 @@ void ParMesh::UniformRefineGroups2D(int old_nv)
   int *I_group_svert, *J_group_svert;
   int *I_group_sedge, *J_group_sedge;

-   I_group_svert = new int[GetNGroups()];
-   I_group_sedge = new int[GetNGroups()];
+   I_group_svert = mfem::New<int>(GetNGroups());
+   I_group_sedge = mfem::New<int>(GetNGroups());

   I_group_svert[0] = 0;
   I_group_sedge[0] = 0;

   // compute the size of the J arrays
-   J_group_svert = new int[group_svert.Size_of_connections()
-                           + group_sedge.Size_of_connections()];
-   J_group_sedge = new int[2*group_sedge.Size_of_connections()];
+   J_group_svert = mfem::New<int>(group_svert.Size_of_connections()
+                                  + group_sedge.Size_of_connections());
+   J_group_sedge = mfem::New<int>(2*group_sedge.Size_of_connections());

   for (int group = 0; group < GetNGroups()-1; group++)
   {
@@ -3433,10 +3452,10 @@ void ParMesh::UniformRefineGroups3D(int old_nv, int old_nedges,
   int *I_group_stria, *J_group_stria;
   int *I_group_squad, *J_group_squad;

-   I_group_svert = new int[GetNGroups()];
-   I_group_sedge = new int[GetNGroups()];
-   I_group_stria = new int[GetNGroups()];
-   I_group_squad = new int[GetNGroups()];
+   I_group_svert = mfem::New<int>(GetNGroups());
+   I_group_sedge = mfem::New<int>(GetNGroups());
+   I_group_stria = mfem::New<int>(GetNGroups());
+   I_group_squad = mfem::New<int>(GetNGroups());

   I_group_svert[0] = 0;
   I_group_sedge[0] = 0;
@@ -3444,14 +3463,14 @@ void ParMesh::UniformRefineGroups3D(int old_nv, int old_nedges,
   I_group_squad[0] = 0;

   // compute the size of the J arrays
-   J_group_svert = new int[group_svert.Size_of_connections()
-                           + group_sedge.Size_of_connections()
-                           + group_squad.Size_of_connections()];
-   J_group_sedge = new int[2*group_sedge.Size_of_connections()
-                           + 3*group_stria.Size_of_connections()
-                           + 4*group_squad.Size_of_connections()];
-   J_group_stria = new int[4*group_stria.Size_of_connections()];
-   J_group_squad = new int[4*group_squad.Size_of_connections()];
+   J_group_svert = mfem::New<int>(group_svert.Size_of_connections()
+                                  + group_sedge.Size_of_connections()
+                                  + group_squad.Size_of_connections());
+   J_group_sedge = mfem::New<int>(2*group_sedge.Size_of_connections()
+                                  + 3*group_stria.Size_of_connections()
+                                  + 4*group_squad.Size_of_connections());
+   J_group_stria = mfem::New<int>(4*group_stria.Size_of_connections());
+   J_group_squad = mfem::New<int>(4*group_squad.Size_of_connections());

   const int oface = old_nv + old_nedges;

@@ -3614,6 +3633,7 @@ void ParMesh::UniformRefinement3D()
   // update the groups
   UniformRefineGroups3D(old_nv, old_nedges, v_to_v, *faces_tbl,
                         f2qf.Size() ? &f2qf : NULL);
+   delete faces_tbl;

   UpdateNodes();
 }
@@ -921,6 +921,10 @@ void ParNCMesh::GetConformingSharedStructures(ParMesh &pmesh)
   pmesh.group_stria.ShiftUpI();

   // create shared_edges
+   for (int i = 0; i < pmesh.shared_edges.Size(); i++)
+   {
+      delete pmesh.shared_edges[i];
+   }
   pmesh.shared_edges.SetSize(pmesh.sedge_ledge.Size());
   for (int i = 0; i < pmesh.shared_edges.Size(); i++)
   {
@@ -1292,7 +1296,7 @@ void ParNCMesh::Prune()
   }

   // derefine subtrees whose leaves are all unneeded
-   for (int i = 0; i < root_count; i++)
+   for (int i = 0; i < root_state.Size(); i++)
   {
      if (PruneTree(i)) { DerefineElement(i); }
   }
@@ -2102,7 +2106,7 @@ void ParNCMesh::ElementSet::Encode(const Array<int> &elements)
   // Each refinement tree that contains at least one element from the set
   // is encoded as HEADER + TREE, where HEADER is the root element number and
   // TREE is the output of EncodeTree().
-   for (int i = 0; i < ncmesh->root_count; i++)
+   for (int i = 0; i < ncmesh->root_state.Size(); i++)
   {
      if (ncmesh->elements[i].flag)
      {
@@ -15,6 +15,7 @@
 #include "config/config.hpp"

 #include "general/error.hpp"
+#include "general/device.hpp"
 #include "general/array.hpp"
 #include "general/sets.hpp"
 #include "general/hash.hpp"
@@ -56,9 +56,44 @@ RT_FESpace::~RT_FESpace()
   delete FEC_;
 }

+void VisualizeMesh(socketstream &sock, const char *vishost, int visport,
+                   Mesh &mesh, const char *title,
+                   int x, int y, int w, int h, const char * keys, bool vec)
+{
+   bool newly_opened = false;
+   int connection_failed;
+
+   do
+   {
+      if (!sock.is_open() || !sock)
+      {
+         sock.open(vishost, visport);
+         sock.precision(8);
+         newly_opened = true;
+      }
+      sock << "solution\n";
+
+      mesh.Print(sock);
+
+      if (newly_opened)
+      {
+         sock << "window_title '" << title << "'\n"
+              << "window_geometry "
+              << x << " " << y << " " << w << " " << h << "\n";
+         if ( keys ) { sock << "keys " << keys << "\n"; }
+         else { sock << "keys maaAc\n"; }
+         if ( vec ) { sock << "vvv"; }
+         sock << endl;
+      }
+
+      connection_failed = !sock && !newly_opened;
+   }
+   while (connection_failed);
+}
+
 void VisualizeField(socketstream &sock, const char *vishost, int visport,
                    GridFunction &gf, const char *title,
-                    int x, int y, int w, int h, bool vec)
+                    int x, int y, int w, int h, const char * keys, bool vec)
 {
   Mesh &mesh = *gf.FESpace()->GetMesh();

@@ -82,8 +117,9 @@ void VisualizeField(socketstream &sock, const char *vishost, int visport,
      {
         sock << "window_title '" << title << "'\n"
              << "window_geometry "
-              << x << " " << y << " " << w << " " << h << "\n"
-              << "keys maaAc";
+              << x << " " << y << " " << w << " " << h << "\n";
+         if ( keys ) { sock << "keys " << keys << "\n"; }
+         else { sock << "keys maaAc\n"; }
         if ( vec ) { sock << "vvv"; }
         sock << endl;
      }
@@ -66,13 +66,21 @@ private:
 };


+/// Visualize the given mesh object, using a GLVis server on the
+/// specified host and port. Set the visualization window title, and optionally,
+/// its geometry.
+void VisualizeMesh(socketstream &sock, const char *vishost, int visport,
+                   Mesh &mesh, const char *title,
+                   int x = 0, int y = 0, int w = 400, int h = 400,
+                   const char *keys = NULL);
+
 /// Visualize the given grid function, using a GLVis server on the
 /// specified host and port. Set the visualization window title, and optionally,
 /// its geometry.
 void VisualizeField(socketstream &sock, const char *vishost, int visport,
                    GridFunction &gf, const char *title,
                    int x = 0, int y = 0, int w = 400, int h = 400,
-                    bool vec = false);
+                    const char *keys = NULL, bool vec = false);

 } // namespace miniapps

--- a/Show More
+++ b/Show More