fixed euclid issue

Bugfixes for PetscSolvers

.gitignore

@@ -6,8 +6,6 @@
 # Also ignore OS-specific files like .DS_Store on Mac
 # ------------------------------------------------------------------------------
 
-*DS_Store
-
 # Object and library files
 *.o
 /libmfem.*
@@ -26,7 +24,6 @@ CMakeFiles/
 /deps.mk
 config/_config.hpp
 config/config.mk
-config/user.cmake
 config/sample-runs-build.log
 doc/CodeDocumentation.conf
 doc/CodeDocumentation.html

CHANGELOG

@@ -11,12 +11,50 @@
 Version 4.0.1 (development)
 ===========================
 
-GPU support
------------
+Improved GPU support
+--------------------
 - Added initial support for AMD GPUs based on HIP: a C++ runtime API and kernel
-  language that can run on both AMD and NVIDIA hardware. The list of current
-  backends is: "occa-cuda", "raja-cuda", "cuda", "hip", "occa-omp", "raja-omp",
-  "omp", "occa-cpu", "raja-cpu", and "cpu".
+  language that can run on both AMD and NVIDIA hardware. With this change, the
+  list of backends is: "occa-cuda", "raja-cuda", "cuda", "hip", "occa-omp",
+  "raja-omp", "omp", "occa-cpu", "raja-cpu", and "cpu".
+
+- Improved RAJA backend and multi-GPU MPI communications.
+
+Discretization improvements
+---------------------------
+- Added support for non-conforming prism AMR, including coarsening and parallel
+  load balancing. Anisotropic prism refinement is only available in the serial
+  version at the moment.
+
+Meshing improvements
+--------------------
+- The TMOP mesh optimization algorithms were extended to support r-adaptivity.
+  Target matrices can now be constructed either via a given analytical function
+  (e.g. spatial dependence of size, aspect ratio, etc., for each element) or via
+  a (Par)GridFunction specified on the original mesh.
+
+- The TMOP mesh optimization algorithms have been improved to support AMR meshes.
+
+- Added support for creating refined versions of periodic meshes, making use of
+  the new L2ElementRestriction class. This class also allows for computing
+  geometric factors on periodic meshes using partial assembly.
+
+- Improved element numbering after uniform mesh refinement.
+
+New and updated examples and miniapps
+-------------------------------------
+- The mesh-optimizer and pmesh-optimizer miniapps have been updated to
+  demonstrate the new r-adaptivity capabilities of TMOP.
+
+- The (p)mesh-optimizer miniapp has been updated to demonstrate mesh
+  optimization for an AMR mesh.
+
+Miscellaneous
+-------------
+- Upgraded the SUNDIALS interface to utilize SUNDIALS version 5.0.  This
+  necessitated a complete rework of the interface and requires changes at
+  the application level.  Example usage of this new interface can be found
+  in the examples/sundials directory.
 
 
 Version 4.0, released on May 24, 2019
@@ -60,7 +98,6 @@ GPU support
 
 Discretization improvements
 ---------------------------
-
 - Partial assembled finite element operators are now available in the core
   library, based on the new classes PABilinearFormExtension, ElementRestriction,
   DofToQuad and GeometricFactors (associated with the classes BilinearForm,

CMakeLists.txt

@@ -254,7 +254,7 @@ endif()
 
 # Axom/Sidre
 if (MFEM_USE_SIDRE)
-  find_package(Axom REQUIRED Sidre SLIC axom_utils)
+  find_package(Axom REQUIRED Axom)
 endif()
 
 # PUMI

INSTALL

@@ -383,11 +383,11 @@ MFEM_USE_MPFR = YES/NO
    see below.
 
 MFEM_USE_SIDRE = YES/NO
-   Sidre is a component of LLNL's axom project, http://goo.gl/cZyJdn, that
-   provides an HDF5-based file format for visualization or restart capability
-   following the Conduit (https://github.com/LLNL/conduit) mesh blueprint
-   specification. When enabled, this option requires installation of HDF5 (see
-   also MFEM_USE_NETCDF), Conduit and LLNL's axom project.
+   Sidre is a component of LLNL's axom project, https://github.com/LLNL/axom,
+   that provides an HDF5-based file format for visualization or restart
+   capability following the Conduit (https://github.com/LLNL/conduit) mesh
+   blueprint specification. When enabled, this option requires installation of
+   HDF5 (see also MFEM_USE_NETCDF), Conduit and LLNL's axom project.
 
 MFEM_USE_CONDUIT = YES/NO
    Enables support for converting MFEM Mesh and Grid Function objects to and
@@ -488,6 +488,7 @@ The specific libraries and their options are:
 - SUNDIALS (optional), used when MFEM_USE_SUNDIALS = YES.
   Beginning with MFEM v3.3, SUNDIALS v2.7.0 is supported.
   Beginning with MFEM v3.3.2, SUNDIALS v3.0.0 is also supported.
+  Beginning with MFEM v4.1, only SUNDIALS v5.0.0+ is supported.
   If MFEM_USE_MPI is enabled, we expect that SUNDIALS is built with support for
   both MPI and hypre.
   URL: http://computation.llnl.gov/projects/sundials/sundials-software
@@ -542,7 +543,8 @@ The specific libraries and their options are:
   Options: PETSC_OPT, PETSC_LIB.
 
 - Sidre (optional), part of LLNL's axom project, used when MFEM_USE_SIDRE = YES.
-  URL: http://goo.gl/cZyJdn (axom, to be released)
+  Starting with MFEM v4.1, Axom version 0.3.1 or later is required.
+  URL: https://github.com/LLNL/axom
        https://github.com/LLNL/conduit (Conduit)
        https://support.hdfgroup.org/HDF5 (HDF5)
   Options: SIDRE_OPT, SIDRE_LIB.
@@ -570,6 +572,7 @@ The specific libraries and their options are:
   Options: OCCA_DIR, OCCA_OPT, OCCA_LIB.
 
 - RAJA, used when MFEM_USE_RAJA = YES.
+  Beginning with MFEM v4.1, only RAJA v0.10.0+ is supported.
   URL: https://github.com/LLNL/RAJA
   Options: RAJA_DIR, RAJA_OPT, RAJA_LIB.
 
@@ -712,6 +715,7 @@ MFEM_USE_PUMI
 MFEM_USE_CUDA
 MFEM_USE_OCCA
 MFEM_USE_RAJA
+MFEM_USE_SIDRE
 
 The following options are CMake specific:
 
@@ -760,6 +764,7 @@ The CMake build system adds auto-detection for the following packages/libraries:
  - PUMI
  - OCCA
  - RAJA
+ - AXOM - Used when MFEM_USE_SIDRE is enabled
 
 The following built-in CMake packages are also used:
 

config/cmake/modules/FindAxom.cmake

@@ -18,6 +18,4 @@ include(MfemCmakeUtilities)
 # Note: components are enabled based on the find_package() parameters.
 mfem_find_package(Axom AXOM AXOM_DIR "include" "" "lib" ""
   "Paths to headers required by Axom." "Libraries required by Axom."
-  ADD_COMPONENT Sidre "include" sidre/sidre.hpp "lib" sidre
-  ADD_COMPONENT SLIC "include" slic/slic.hpp "lib" slic
-  ADD_COMPONENT axom_utils "include" axom_utils/Utilities.hpp "lib" axom_utils)
+  ADD_COMPONENT Axom "include" axom/config.hpp "lib" axom)

config/defaults.cmake

@@ -81,7 +81,7 @@ set(METIS_DIR "${MFEM_DIR}/../metis-4.0" CACHE PATH "Path to the METIS library."
 
 set(LIBUNWIND_DIR "" CACHE PATH "Path to Libunwind.")
 
-set(SUNDIALS_DIR "${MFEM_DIR}/../sundials-3.0.0" CACHE PATH
+set(SUNDIALS_DIR "${MFEM_DIR}/../sundials-5.0.0/instdir" CACHE PATH
     "Path to the SUNDIALS library.")
 # The following may be necessary, if SUNDIALS was built with KLU:
 # set(SUNDIALS_REQUIRED_PACKAGES "SuiteSparse/KLU/AMD/BTF/COLAMD/config"
@@ -154,7 +154,7 @@ set(CONDUIT_DIR "${MFEM_DIR}/../conduit" CACHE PATH
 
 set(AXOM_DIR "${MFEM_DIR}/../axom" CACHE PATH "Path to the Axom library.")
 # May need to add "Boost" as requirement.
-set(Axom_REQUIRED_PACKAGES "Conduit/relay" CACHE STRING
+set(Axom_REQUIRED_PACKAGES "Conduit/relay/blueprint" CACHE STRING
     "Additional packages required by Axom.")
 
 set(PUMI_DIR "${MFEM_DIR}/../pumi-2.1.0" CACHE STRING

config/defaults.mk

@@ -103,9 +103,9 @@ MFEM_MPI_NP = 4
 # config.hpp. The values below are the defaults for generating the actual values
 # in config.mk and config.hpp.
 
-MFEM_USE_MPI           = YES
+MFEM_USE_MPI           = NO
 MFEM_USE_METIS         = $(MFEM_USE_MPI)
-MFEM_USE_METIS_5       = YES
+MFEM_USE_METIS_5       = NO
 MFEM_DEBUG             = NO
 MFEM_USE_EXCEPTIONS    = NO
 MFEM_USE_GZSTREAM      = NO
@@ -182,9 +182,9 @@ OPENMP_LIB =
 POSIX_CLOCKS_LIB = -lrt
 
 # SUNDIALS library configuration
-SUNDIALS_DIR = @MFEM_DIR@/../sundials-3.0.0
+SUNDIALS_DIR = @MFEM_DIR@/../sundials-5.0.0/instdir
 SUNDIALS_OPT = -I$(SUNDIALS_DIR)/include
-SUNDIALS_LIB = -Wl,-rpath,$(SUNDIALS_DIR)/lib -L$(SUNDIALS_DIR)/lib\
+SUNDIALS_LIB = -Wl,-rpath,$(SUNDIALS_DIR)/lib64 -L$(SUNDIALS_DIR)/lib64\
  -lsundials_arkode -lsundials_cvode -lsundials_nvecserial -lsundials_kinsol
 
 ifeq ($(MFEM_USE_MPI),YES)
@@ -209,7 +209,7 @@ SUITESPARSE_LIB = -Wl,-rpath,$(SUITESPARSE_DIR)/lib -L$(SUITESPARSE_DIR)/lib\
 # SuperLU library configuration
 SUPERLU_DIR = @MFEM_DIR@/../SuperLU_DIST_5.1.0
 SUPERLU_OPT = -I$(SUPERLU_DIR)/SRC
-SUPERLU_LIB = -Wl,-rpath,$(SUPERLU_DIR)/SRC -L$(SUPERLU_DIR)/SRC -lsuperlu_dist
+SUPERLU_LIB = -Wl,-rpath,$(SUPERLU_DIR)/lib -L$(SUPERLU_DIR)/lib -lsuperlu_dist_5.1.0
 
 # SCOTCH library configuration (required by STRUMPACK <= v2.1.0, optional in
 # STRUMPACK >= v2.2.0)
@@ -299,7 +299,7 @@ SIDRE_LIB = \
    -Wl,-rpath,$(SIDRE_DIR)/lib -L$(SIDRE_DIR)/lib \
    -Wl,-rpath,$(CONDUIT_DIR)/lib -L$(CONDUIT_DIR)/lib \
    -Wl,-rpath,$(HDF5_DIR)/lib -L$(HDF5_DIR)/lib \
-   -lsidre -lslic -laxom_utils -lconduit -lconduit_relay -lhdf5 $(ZLIB_LIB) -ldl
+   -laxom -lconduit -lconduit_relay -lconduit_blueprint -lhdf5 $(ZLIB_LIB) -ldl
 
 # PUMI
 # Note that PUMI_DIR is needed -- it is used to check for gmi_sim.h

examples/CMakeLists.txt

@@ -28,7 +28,6 @@ list(APPEND ALL_EXE_SRCS
   ex19.cpp
   ex20.cpp
   ex21.cpp
-  ex23.cpp
   )
 
 if (MFEM_USE_MPI)
@@ -54,7 +53,6 @@ if (MFEM_USE_MPI)
     ex19p.cpp
     ex20p.cpp
     ex21p.cpp
-    ex23p.cpp
     )
 endif()
 

examples/ex12p.cpp

@@ -5,11 +5,11 @@
 // Sample runs:
 //    mpirun -np 4 ex12p -m ../data/beam-tri.mesh
 //    mpirun -np 4 ex12p -m ../data/beam-quad.mesh
-//    mpirun -np 4 ex12p -m ../data/beam-tet.mesh -s 79 -n 10 -o 2 -elast
-//    mpirun -np 4 ex12p -m ../data/beam-hex.mesh -s 3876
-//    mpirun -np 4 ex12p -m ../data/beam-wedge.mesh -s 79
+//    mpirun -np 4 ex12p -m ../data/beam-tet.mesh -s 462 -n 10 -o 2 -elast
+//    mpirun -np 4 ex12p -m ../data/beam-hex.mesh -s 3878
+//    mpirun -np 4 ex12p -m ../data/beam-wedge.mesh -s 81
 //    mpirun -np 4 ex12p -m ../data/beam-tri.mesh -s 3876 -o 2 -sys
-//    mpirun -np 4 ex12p -m ../data/beam-quad.mesh -s 4526 -n 6 -o 3 -elast
+//    mpirun -np 4 ex12p -m ../data/beam-quad.mesh -s 4544 -n 6 -o 3 -elast
 //    mpirun -np 4 ex12p -m ../data/beam-quad-nurbs.mesh
 //    mpirun -np 4 ex12p -m ../data/beam-hex-nurbs.mesh
 //

examples/ex17.cpp

@@ -252,7 +252,7 @@ int main(int argc, char *argv[])
    }
    else
    {
-      GMRES(A, M, B, X, 3, 5000, 50, rtol*rtol, 0.0);
+      GMRES(A, M, B, X, 3, 5000, 100, rtol*rtol, 0.0);
    }
 #else
    // 11. If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.

examples/ex23.cpp

@@ -1,553 +0,0 @@
-//                               MFEM Example 23
-//
-// Compile with: make ex23
-//
-// Sample runs:
-//    ex23 -m ../data/periodic-segment.mesh -p 0 -r 2 -dt 0.005
-//    ex23 -m ../data/periodic-square.mesh -p 0 -r 2 -dt 0.01
-//    ex23 -m ../data/periodic-hexagon.mesh -p 0 -r 2 -dt 0.01
-//    ex23 -m ../data/periodic-square.mesh -p 1 -r 2 -dt 0.005 -tf 9
-//    ex23 -m ../data/periodic-hexagon.mesh -p 1 -r 2 -dt 0.005 -tf 9
-//    ex23 -m ../data/amr-quad.mesh -p 1 -r 2 -dt 0.002 -tf 9
-//    ex23 -m ../data/star-q3.mesh -p 1 -r 2 -dt 0.005 -tf 9
-//    ex23 -m ../data/star-mixed.mesh -p 1 -r 2 -dt 0.005 -tf 9
-//    ex23 -m ../data/disc-nurbs.mesh -p 1 -r 3 -dt 0.005 -tf 9
-//    ex23 -m ../data/disc-nurbs.mesh -p 2 -r 3 -dt 0.005 -tf 9
-//    ex23 -m ../data/disc-nurbs.mesh -p 2 -r 3 -dt 0.005 -tf 9 -d 0.05
-//    ex23 -m ../data/periodic-square.mesh -p 3 -r 4 -dt 0.0025 -tf 9 -vs 20
-//    ex23 -m ../data/periodic-cube.mesh -p 0 -r 2 -o 2 -dt 0.02 -tf 8
-//
-// Description:  This example code solves the time-dependent advection-diffusion
-//               equation
-//               du/dt - div(D grad(u)) + v.grad(u) = 0, where
-//               D is a diffusion coefficient,
-//               v is a given fluid velocity, and
-//               u0(x)=u(0,x) is a given initial condition.
-//
-//               The example demonstrates the use of Discontinuous Galerkin (DG)
-//               bilinear forms in MFEM (face integrators), the use of implicit
-//               ODE time integrators, the definition of periodic boundary
-//               conditions through periodic meshes, as well as the use of GLVis
-//               for persistent visualization of a time-evolving solution. The
-//               saving of time-dependent data files for external visualization
-//               with VisIt (visit.llnl.gov) is also illustrated.
-//
-//               This example is a merger of examples 9 and 14.
-
-#include "mfem.hpp"
-#include <fstream>
-#include <iostream>
-
-using namespace std;
-using namespace mfem;
-
-// Choice for the problem setup. The fluid velocity, initial condition and
-// inflow boundary condition are chosen based on this parameter.
-int problem;
-
-// Velocity coefficient
-void velocity_function(const Vector &x, Vector &v);
-
-// Initial condition
-double u0_function(const Vector &x);
-
-// Inflow boundary condition
-double inflow_function(const Vector &x);
-
-// Mesh bounding box
-Vector bb_min, bb_max;
-
-
-/** A time-dependent operator for the right-hand side of the ODE. The DG weak
-    form of du/dt = div(D grad(u))-v.grad(u) is
-    [M + dt (S - K)] du/dt = - S u + K u + b, where M, S, and K are the mass,
-    stiffness, and advection matrices, and b describes sources and the flow on
-    the boundary.
-    This can be written as a general ODE,
-    du/dt = A^{-1} (-S u + K u + b) with A = [M + dt (S - K)], and this class is
-    used to perform the implicit or explicit solve for du/dt. */
-class FE_Evolution : public TimeDependentOperator
-{
-private:
-   SparseMatrix &M, &S, &K;
-   SparseMatrix *A;
-   const Vector &b;
-
-   DSmoother M_prec;
-   CGSolver M_solver;
-
-   DSmoother *A_prec;
-   GMRESSolver *A_solver;
-   double dt;
-
-   mutable Vector z;
-
-   void initA(double dt);
-
-public:
-   FE_Evolution(SparseMatrix &_M, SparseMatrix &_S, SparseMatrix &_K,
-                const Vector &_b);
-
-   virtual void Mult(const Vector &x, Vector &y) const;
-
-   virtual void ImplicitSolve(const double dt, const Vector &x, Vector &y);
-
-   virtual ~FE_Evolution() { delete A_solver; delete A_prec; delete A; }
-};
-
-
-int main(int argc, char *argv[])
-{
-   // 1. Parse command-line options.
-   problem = 0;
-   const char *mesh_file = "../data/periodic-hexagon.mesh";
-   int ref_levels = 2;
-   int order = 3;
-   int ode_solver_type = 3;
-   double t_final = 10.0;
-   double d_coef = 0.01;
-   double dt = 0.01;
-   double sigma = -1.0;
-   double kappa = -1.0;
-   bool visualization = true;
-   bool visit = false;
-   bool binary = false;
-   int vis_steps = 5;
-
-   int precision = 8;
-   cout.precision(precision);
-
-   OptionsParser args(argc, argv);
-   args.AddOption(&mesh_file, "-m", "--mesh",
-                  "Mesh file to use.");
-   args.AddOption(&problem, "-p", "--problem",
-                  "Problem setup to use. See options in velocity_function().");
-   args.AddOption(&ref_levels, "-r", "--refine",
-                  "Number of times to refine the mesh uniformly.");
-   args.AddOption(&order, "-o", "--order",
-                  "Order (degree) of the finite elements.");
-   args.AddOption(&ode_solver_type, "-s", "--ode-solver",
-                  "ODE solver: 1 - Backward Euler, 2 - SDIRK2, 3 - SDIRK3,\n\t"
-                  "\t   11 - Forward Euler, 12 - RK2, 13 - RK3 SSP, 14 - RK4.");
-   args.AddOption(&t_final, "-tf", "--t-final",
-                  "Final time; start time is 0.");
-   args.AddOption(&dt, "-dt", "--time-step",
-                  "Time step.");
-   args.AddOption(&d_coef, "-d", "--diff-coef",
-                  "Diffusion coefficient.");
-   args.AddOption(&sigma, "-s", "--sigma",
-                  "One of the two DG penalty parameters, typically +1/-1."
-                  " See the documentation of class DGDiffusionIntegrator.");
-   args.AddOption(&kappa, "-k", "--kappa",
-                  "One of the two DG penalty parameters, should be positive."
-                  " Negative values are replaced with (order+1)^2.");
-   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
-                  "--no-visualization",
-                  "Enable or disable GLVis visualization.");
-   args.AddOption(&visit, "-visit", "--visit-datafiles", "-no-visit",
-                  "--no-visit-datafiles",
-                  "Save data files for VisIt (visit.llnl.gov) visualization.");
-   args.AddOption(&binary, "-binary", "--binary-datafiles", "-ascii",
-                  "--ascii-datafiles",
-                  "Use binary (Sidre) or ascii format for VisIt data files.");
-   args.AddOption(&vis_steps, "-vs", "--visualization-steps",
-                  "Visualize every n-th timestep.");
-   args.Parse();
-   if (!args.Good())
-   {
-      args.PrintUsage(cout);
-      return 1;
-   }
-   if (kappa < 0)
-   {
-      kappa = (order+1)*(order+1);
-   }
-   args.PrintOptions(cout);
-
-   // 2. Read the serial mesh from the given mesh file on all processors. We can
-   //    handle geometrically periodic meshes in this code.
-   Mesh mesh(mesh_file, 1, 1);
-   int dim = mesh.Dimension();
-
-   // 3. Define the ODE solver used for time integration. Several explicit
-   //    Runge-Kutta methods are available.
-   ODESolver *ode_solver = NULL;
-   switch (ode_solver_type)
-   {
-      // Implicit L-stable methods
-      case 1:  ode_solver = new BackwardEulerSolver; break;
-      case 2:  ode_solver = new SDIRK23Solver(2); break;
-      case 3:  ode_solver = new SDIRK33Solver; break;
-      // Explicit methods
-      case 11: ode_solver = new ForwardEulerSolver; break;
-      case 12: ode_solver = new RK2Solver(0.5); break; // midpoint method
-      case 13: ode_solver = new RK3SSPSolver; break;
-      case 14: ode_solver = new RK4Solver; break;
-      case 15: ode_solver = new GeneralizedAlphaSolver(0.5); break;
-      // Implicit A-stable methods (not L-stable)
-      case 22: ode_solver = new ImplicitMidpointSolver; break;
-      case 23: ode_solver = new SDIRK23Solver; break;
-      case 24: ode_solver = new SDIRK34Solver; break;
-      default:
-         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
-         return 3;
-   }
-
-   // 4. Refine the mesh in serial to increase the resolution. In this example
-   //    we do 'ser_ref_levels' of uniform refinement, where 'ser_ref_levels' is
-   //    a command-line parameter. If the mesh is of NURBS type, we convert it
-   //    to a (piecewise-polynomial) high-order mesh.
-   for (int lev = 0; lev < ref_levels; lev++)
-   {
-      mesh.UniformRefinement();
-   }
-   if (mesh.NURBSext)
-   {
-      mesh.SetCurvature(max(order, 1));
-   }
-   mesh.GetBoundingBox(bb_min, bb_max, max(order, 1));
-
-   // 5. Define the parallel discontinuous DG finite element space on the
-   //    parallel refined mesh of the given polynomial order.
-   DG_FECollection fec(order, dim);
-   FiniteElementSpace fes(&mesh, &fec);
-
-   cout << "Number of unknowns: " << fes.GetVSize() << endl;
-
-   // 6. Set up and assemble the parallel bilinear and linear forms (and the
-   //    parallel hypre matrices) corresponding to the DG discretization. The
-   //    DGTraceIntegrator involves integrals over mesh interior faces.
-   ConstantCoefficient diff_coef(d_coef);
-   VectorFunctionCoefficient velocity(dim, velocity_function);
-   FunctionCoefficient u0(u0_function);
-
-   BilinearForm m(&fes);
-   m.AddDomainIntegrator(new MassIntegrator);
-
-   BilinearForm s(&fes);
-   s.AddDomainIntegrator(new DiffusionIntegrator(diff_coef));
-   s.AddInteriorFaceIntegrator(new DGDiffusionIntegrator(diff_coef, sigma,
-                                                         kappa));
-   s.AddBdrFaceIntegrator(new DGDiffusionIntegrator(diff_coef, sigma, kappa));
-
-   BilinearForm k(&fes);
-   k.AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
-   k.AddInteriorFaceIntegrator(
-      new TransposeIntegrator(new DGTraceIntegrator(velocity, 1.0, -0.5)));
-   k.AddBdrFaceIntegrator(
-      new TransposeIntegrator(new DGTraceIntegrator(velocity, 1.0, -0.5)));
-
-   LinearForm b(&fes);
-   b.AddBdrFaceIntegrator(
-      new DGDirichletLFIntegrator(u0, diff_coef, sigma, kappa));
-
-   int skip_zeros = 0;
-   m.Assemble(skip_zeros);
-   m.Finalize(skip_zeros);
-   s.Assemble(skip_zeros);
-   s.Finalize(skip_zeros);
-   k.Assemble(skip_zeros);
-   k.Finalize(skip_zeros);
-   b.Assemble();
-
-   // 7. Define the initial conditions, save the corresponding grid function to
-   //    a file and (optionally) save data in the VisIt format and initialize
-   //    GLVis visualization.
-   GridFunction u(&fes);
-   u.ProjectCoefficient(u0);
-
-   {
-      ofstream omesh("ex23.mesh");
-      omesh.precision(precision);
-      mesh.Print(omesh);
-      ofstream osol("ex23-init.gf");
-      osol.precision(precision);
-      u.Save(osol);
-   }
-
-   // Create data collection for solution output: either VisItDataCollection for
-   // ascii data files, or SidreDataCollection for binary data files.
-   DataCollection *dc = NULL;
-   if (visit)
-   {
-      if (binary)
-      {
-#ifdef MFEM_USE_SIDRE
-         dc = new SidreDataCollection("Example23", &mesh);
-#else
-         MFEM_ABORT("Must build with MFEM_USE_SIDRE=YES for binary output.");
-#endif
-      }
-      else
-      {
-         dc = new VisItDataCollection("Example23", &mesh);
-         dc->SetPrecision(precision);
-      }
-      dc->RegisterField("solution", &u);
-      dc->SetCycle(0);
-      dc->SetTime(0.0);
-      dc->Save();
-   }
-
-   socketstream sout;
-   if (visualization)
-   {
-      char vishost[] = "localhost";
-      int  visport   = 19916;
-      sout.open(vishost, visport);
-      if (!sout)
-      {
-         cout << "Unable to connect to GLVis server at "
-              << vishost << ':' << visport << endl;
-         visualization = false;
-         cout << "GLVis visualization disabled.\n";
-      }
-      else
-      {
-         sout.precision(precision);
-         sout << "solution\n" << mesh << u;
-         sout << "pause\n";
-         sout << flush;
-         cout << "GLVis visualization paused."
-              << " Press space (in the GLVis window) to resume it.\n";
-      }
-   }
-
-   // 8. Define the time-dependent evolution operator describing the ODE
-   //    right-hand side, and perform time-integration (looping over the time
-   //    iterations, ti, with a time-step dt).
-   FE_Evolution adv(m.SpMat(), s.SpMat(), k.SpMat(), b);
-
-   double t = 0.0;
-   adv.SetTime(t);
-   ode_solver->Init(adv);
-
-   bool done = false;
-   for (int ti = 0; !done; )
-   {
-      double dt_real = min(dt, t_final - t);
-      ode_solver->Step(u, t, dt_real);
-      ti++;
-
-      done = (t >= t_final - 1e-8*dt);
-
-      if (done || ti % vis_steps == 0)
-      {
-         cout << "time step: " << ti << ", time: " << t << endl;
-
-         if (visualization)
-         {
-            sout << "solution\n" << mesh << u << flush;
-         }
-
-         if (visit)
-         {
-            dc->SetCycle(ti);
-            dc->SetTime(t);
-            dc->Save();
-         }
-      }
-   }
-
-   // 9. Save the final solution in parallel. This output can be viewed later
-   //    using GLVis: "glvis -np <np> -m ex23-mesh -g ex23-final".
-   {
-      ofstream osol("ex23-final.gf");
-      osol.precision(precision);
-      u.Save(osol);
-   }
-
-   // 10. Free the used memory.
-   delete ode_solver;
-   delete dc;
-
-   return 0;
-}
-
-
-// Implementation of class FE_Evolution
-FE_Evolution::FE_Evolution(SparseMatrix &_M, SparseMatrix &_S,
-                           SparseMatrix &_K, const Vector &_b)
-   : TimeDependentOperator(_M.Height()),
-     M(_M), S(_S), K(_K), A(NULL), b(_b),
-     M_prec(M),
-     A_prec(NULL), A_solver(NULL), dt(-1.0), z(M.Height())
-{
-   M_solver.SetPreconditioner(M_prec);
-   M_solver.SetOperator(M);
-
-   M_solver.iterative_mode = false;
-   M_solver.SetRelTol(1e-9);
-   M_solver.SetAbsTol(0.0);
-   M_solver.SetMaxIter(100);
-   M_solver.SetPrintLevel(0);
-}
-
-void FE_Evolution::initA(double _dt)
-{
-   if (fabs(dt - _dt) > 1e-4 * _dt)
-   {
-      delete A_solver;
-      delete A_prec;
-      delete A;
-
-      SparseMatrix * SK = Add(1.0, S, -1.0, K);
-      A = Add(1.0, M, _dt, *SK);
-      delete SK;
-      dt = _dt;
-
-      A_prec = new DSmoother(*A);
-      A_solver = new GMRESSolver;
-      A_solver->SetOperator(*A);
-      A_solver->SetPreconditioner(*A_prec);
-
-      A_solver->iterative_mode = false;
-      A_solver->SetRelTol(1e-9);
-      A_solver->SetAbsTol(0.0);
-      A_solver->SetMaxIter(100);
-      A_solver->SetPrintLevel(0);
-   }
-}
-
-void FE_Evolution::Mult(const Vector &x, Vector &y) const
-{
-   // y = M^{-1} (-S x + K x + b)
-   K.Mult(x, z);
-   S.AddMult(x, z, -1.0);
-   z += b;
-   M_solver.Mult(z, y);
-}
-
-void FE_Evolution::ImplicitSolve(const double _dt, const Vector &x, Vector &y)
-{
-   this->initA(_dt);
-
-   // y = (M + dt S - dt K)^{-1} (-S x + K x + b)
-   K.Mult(x, z);
-   S.AddMult(x, z, -1.0);
-   z += b;
-   A_solver->Mult(z, y);
-}
-
-// Velocity coefficient
-void velocity_function(const Vector &x, Vector &v)
-{
-   int dim = x.Size();
-
-   // map to the reference [-1,1] domain
-   Vector X(dim);
-   for (int i = 0; i < dim; i++)
-   {
-      double center = (bb_min[i] + bb_max[i]) * 0.5;
-      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
-   }
-
-   switch (problem)
-   {
-      case 0:
-      {
-         // Translations in 1D, 2D, and 3D
-         switch (dim)
-         {
-            case 1: v(0) = 1.0; break;
-            case 2: v(0) = sqrt(2./3.); v(1) = sqrt(1./3.); break;
-            case 3: v(0) = sqrt(3./6.); v(1) = sqrt(2./6.); v(2) = sqrt(1./6.);
-               break;
-         }
-         break;
-      }
-      case 1:
-      case 2:
-      {
-         // Clockwise rotation in 2D around the origin
-         const double w = M_PI/2;
-         switch (dim)
-         {
-            case 1: v(0) = 1.0; break;
-            case 2: v(0) = w*X(1); v(1) = -w*X(0); break;
-            case 3: v(0) = w*X(1); v(1) = -w*X(0); v(2) = 0.0; break;
-         }
-         break;
-      }
-      case 3:
-      {
-         // Clockwise twisting rotation in 2D around the origin
-         const double w = M_PI/2;
-         double d = max((X(0)+1.)*(1.-X(0)),0.) * max((X(1)+1.)*(1.-X(1)),0.);
-         d = d*d;
-         switch (dim)
-         {
-            case 1: v(0) = 1.0; break;
-            case 2: v(0) = d*w*X(1); v(1) = -d*w*X(0); break;
-            case 3: v(0) = d*w*X(1); v(1) = -d*w*X(0); v(2) = 0.0; break;
-         }
-         break;
-      }
-   }
-}
-
-// Initial condition
-double u0_function(const Vector &x)
-{
-   int dim = x.Size();
-
-   // map to the reference [-1,1] domain
-   Vector X(dim);
-   for (int i = 0; i < dim; i++)
-   {
-      double center = (bb_min[i] + bb_max[i]) * 0.5;
-      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
-   }
-
-   switch (problem)
-   {
-      case 0:
-      case 1:
-      {
-         switch (dim)
-         {
-            case 1:
-               return exp(-40.*pow(X(0)-0.5,2));
-            case 2:
-            case 3:
-            {
-               double rx = 0.45, ry = 0.25, cx = 0., cy = -0.2, w = 10.;
-               if (dim == 3)
-               {
-                  const double s = (1. + 0.25*cos(2*M_PI*X(2)));
-                  rx *= s;
-                  ry *= s;
-               }
-               return ( erfc(w*(X(0)-cx-rx))*erfc(-w*(X(0)-cx+rx)) *
-                        erfc(w*(X(1)-cy-ry))*erfc(-w*(X(1)-cy+ry)) )/16;
-            }
-         }
-      }
-      case 2:
-      {
-         double x_ = X(0), y_ = X(1), rho, phi;
-         rho = hypot(x_, y_);
-         phi = atan2(y_, x_);
-         return pow(sin(M_PI*rho),2)*sin(3*phi);
-      }
-      case 3:
-      {
-         const double f = M_PI;
-         return sin(f*X(0))*sin(f*X(1));
-      }
-   }
-   return 0.0;
-}
-
-// Inflow boundary condition (zero for the problems considered in this example)
-double inflow_function(const Vector &x)
-{
-   switch (problem)
-   {
-      case 0:
-      case 1:
-      case 2:
-      case 3: return 0.0;
-   }
-   return 0.0;
-}

examples/ex23p.cpp

@@ -1,673 +0,0 @@
-//                       MFEM Example 23 - Parallel Version
-//
-// Compile with: make ex23p
-//
-// Sample runs:
-//    mpirun -np 4 ex23p -m ../data/periodic-segment.mesh -p 0 -dt 0.005
-//    mpirun -np 4 ex23p -m ../data/periodic-square.mesh -p 0 -dt 0.01
-//    mpirun -np 4 ex23p -m ../data/periodic-hexagon.mesh -p 0 -dt 0.01
-//    mpirun -np 4 ex23p -m ../data/periodic-square.mesh -p 1 -dt 0.005 -tf 9
-//    mpirun -np 4 ex23p -m ../data/periodic-hexagon.mesh -p 1 -dt 0.005 -tf 9
-//    mpirun -np 4 ex23p -m ../data/amr-quad.mesh -p 1 -rp 1 -dt 0.002 -tf 9
-//    mpirun -np 4 ex23p -m ../data/star-q3.mesh -p 1 -rp 1 -dt 0.004 -tf 9
-//    mpirun -np 4 ex23p -m ../data/star-mixed.mesh -p 1 -rp 1 -dt 0.004 -tf 9
-//    mpirun -np 4 ex23p -m ../data/disc-nurbs.mesh -p 1 -rp 1 -dt 0.005 -tf 9
-//    mpirun -np 4 ex23p -m ../data/disc-nurbs.mesh -p 2 -rp 1 -dt 0.005 -tf 9
-//    mpirun -np 4 ex23p -m ../data/disc-nurbs.mesh -p 3 -rp 1 -dt 0.005 -tf 9 -d 0.05
-//    mpirun -np 4 ex23p -m ../data/periodic-square.mesh -p 3 -rp 2 -dt 0.0025 -tf 9 -vs 20
-//    mpirun -np 4 ex23p -m ../data/periodic-cube.mesh -p 0 -o 2 -rp 1 -dt 0.01 -tf 8
-//
-// Description:  This example code solves the time-dependent advection-diffusion
-//               equation
-//               du/dt - div(D grad(u)) + v.grad(u) = 0, where
-//               D is a diffusion coefficient,
-//               v is a given fluid velocity, and
-//               u0(x)=u(0,x) is a given initial condition.
-//
-//               The example demonstrates the use of Discontinuous Galerkin (DG)
-//               bilinear forms in MFEM (face integrators), the use of implicit
-//               ODE time integrators, the definition of periodic boundary
-//               conditions through periodic meshes, as well as the use of GLVis
-//               for persistent visualization of a time-evolving solution. The
-//               saving of time-dependent data files for external visualization
-//               with VisIt (visit.llnl.gov) is also illustrated.
-//
-//               This example is a merger of examples 9 and 14.
-
-#include "mfem.hpp"
-#include <fstream>
-#include <iostream>
-
-using namespace std;
-using namespace mfem;
-
-// Choice for the problem setup. The fluid velocity, initial condition and
-// inflow boundary condition are chosen based on this parameter.
-int problem, use_gmres;
-bool use_AIR;
-
-// Velocity coefficient
-void velocity_function(const Vector &x, Vector &v);
-
-// Initial condition
-double u0_function(const Vector &x);
-
-// Inflow boundary condition
-double inflow_function(const Vector &x);
-
-// Mesh bounding box
-Vector bb_min, bb_max;
-
-struct AIR_parameters {
-   double distanceR;
-   std::string prerelax;
-   std::string postrelax;
-   int interp_type;
-   int relax_type;
-   int coarsen_type;
-   double strength_tolC;
-   double strength_tolR;
-   double filter_tolR;
-   double filterA_tol;
-};
-
-/** A time-dependent operator for the right-hand side of the ODE. The DG weak
-    form of du/dt = div(D grad(u))-v.grad(u) is
-    [M + dt (S - K)] du/dt = - S u + K u + b, where M, S, and K are the mass,
-    stiffness, and advection matrices, and b describes sources and the flow on
-    the boundary.
-    This can be written as a general ODE,
-    du/dt = A^{-1} (-S u + K u + b) with A = [M + dt (S - K)], and this class is
-    used to perform the implicit or explicit solve for du/dt. */
-class FE_Evolution : public TimeDependentOperator
-{
-private:
-   HypreParMatrix &M, &S, &K;
-   HypreParMatrix *A;
-   HypreParMatrix A_s;
-   const Vector &b;
-
-   HypreSmoother M_prec;
-   CGSolver M_solver;
-
-   HypreBoomerAMG *AMG_solver;
-   HypreGMRES *GMRES_solver;
-   double dt;
-   int blocksize;
-
-   mutable Vector z;
-
-public:
-   FE_Evolution(HypreParMatrix &_M, HypreParMatrix &_S, HypreParMatrix &_K,
-                const Vector &_b, int order);
-
-   virtual void Mult(const Vector &x, Vector &y) const;
-
-   virtual void ImplicitSolve(const double dt, const Vector &x, Vector &y);
-
-   virtual ~FE_Evolution() { delete GMRES_solver; delete AMG_solver; delete A; }
-};
-
-
-int main(int argc, char *argv[])
-{
-   // 1. Initialize MPI.
-   int num_procs, myid;
-   MPI_Init(&argc, &argv);
-   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
-   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
-
-   // 2. Parse command-line options.
-   problem = 0;
-   use_gmres = true;
-   const char *mesh_file = "../data/periodic-hexagon.mesh";
-   int ser_ref_levels = 2;
-   int par_ref_levels = 0;
-   int order = 3;
-   int ode_solver_type = 3;
-   double t_final = 10.0;
-   double d_coef = 0.01;
-   double dt = 0.01;
-   double sigma = -1.0;
-   double kappa = -1.0;
-   bool visualization = true;
-   bool visit = false;
-   bool binary = false;
-   int vis_steps = 5;
-
-   int precision = 8;
-   cout.precision(precision);
-
-   OptionsParser args(argc, argv);
-   args.AddOption(&mesh_file, "-m", "--mesh",
-                  "Mesh file to use.");
-   args.AddOption(&problem, "-p", "--problem",
-                  "Problem setup to use. See options in velocity_function().");
-   args.AddOption(&ser_ref_levels, "-rs", "--refine-serial",
-                  "Number of times to refine the mesh uniformly in serial.");
-   args.AddOption(&par_ref_levels, "-rp", "--refine-parallel",
-                  "Number of times to refine the mesh uniformly in parallel.");
-   args.AddOption(&order, "-o", "--order",
-                  "Order (degree) of the finite elements.");
-   args.AddOption(&ode_solver_type, "-s", "--ode-solver",
-                  "ODE solver: 1 - Backward Euler, 2 - SDIRK2, 3 - SDIRK3,\n\t"
-                  "\t   11 - Forward Euler, 12 - RK2, 13 - RK3 SSP, 14 - RK4.");
-   args.AddOption(&t_final, "-tf", "--t-final",
-                  "Final time; start time is 0.");
-   args.AddOption(&dt, "-dt", "--time-step",
-                  "Time step.");
-   args.AddOption(&d_coef, "-d", "--diff-coef",
-                  "Diffusion coefficient.");
-   args.AddOption(&sigma, "-s", "--sigma",
-                  "One of the two DG penalty parameters, typically +1/-1."
-                  " See the documentation of class DGDiffusionIntegrator.");
-   args.AddOption(&kappa, "-k", "--kappa",
-                  "One of the two DG penalty parameters, should be positive."
-                  " Negative values are replaced with (order+1)^2.");
-   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
-                  "--no-visualization",
-                  "Enable or disable GLVis visualization.");
-   args.AddOption(&visit, "-visit", "--visit-datafiles", "-no-visit",
-                  "--no-visit-datafiles",
-                  "Save data files for VisIt (visit.llnl.gov) visualization.");
-   args.AddOption(&binary, "-binary", "--binary-datafiles", "-ascii",
-                  "--ascii-datafiles",
-                  "Use binary (Sidre) or ascii format for VisIt data files.");
-   args.AddOption(&vis_steps, "-vs", "--visualization-steps",
-                  "Visualize every n-th timestep.");
-   args.Parse();
-   if (!args.Good())
-   {
-      if (myid == 0)
-      {
-         args.PrintUsage(cout);
-      }
-      MPI_Finalize();
-      return 1;
-   }
-   if (kappa < 0)
-   {
-      kappa = (order+1)*(order+1);
-   }
-   if (myid == 0)
-   {
-      args.PrintOptions(cout);
-   }
-
-   // 3. Read the serial mesh from the given mesh file on all processors. We can
-   //    handle geometrically periodic meshes in this code.
-   Mesh *mesh = new Mesh(mesh_file, 1, 1);
-   int dim = mesh->Dimension();
-
-   // 4. Define the ODE solver used for time integration. Several explicit
-   //    Runge-Kutta methods are available.
-   ODESolver *ode_solver = NULL;
-   switch (ode_solver_type)
-   {
-      // Implicit L-stable methods
-      case 1:  ode_solver = new BackwardEulerSolver; break;
-      case 2:  ode_solver = new SDIRK23Solver(2); break;
-      case 3:  ode_solver = new SDIRK33Solver; break;
-      // Explicit methods
-      case 11: ode_solver = new ForwardEulerSolver; break;
-      case 12: ode_solver = new RK2Solver(0.5); break; // midpoint method
-      case 13: ode_solver = new RK3SSPSolver; break;
-      case 14: ode_solver = new RK4Solver; break;
-      case 15: ode_solver = new GeneralizedAlphaSolver(0.5); break;
-      // Implicit A-stable methods (not L-stable)
-      case 22: ode_solver = new ImplicitMidpointSolver; break;
-      case 23: ode_solver = new SDIRK23Solver; break;
-      case 24: ode_solver = new SDIRK34Solver; break;
-      default:
-         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
-         delete mesh;
-         return 3;
-   }
-
-   // 5. Refine the mesh in serial to increase the resolution. In this example
-   //    we do 'ser_ref_levels' of uniform refinement, where 'ser_ref_levels' is
-   //    a command-line parameter. If the mesh is of NURBS type, we convert it
-   //    to a (piecewise-polynomial) high-order mesh.
-   for (int lev = 0; lev < ser_ref_levels; lev++)
-   {
-      mesh->UniformRefinement();
-   }
-   if (mesh->NURBSext)
-   {
-      mesh->SetCurvature(max(order, 1));
-   }
-   mesh->GetBoundingBox(bb_min, bb_max, max(order, 1));
-
-   // 6. Define the parallel mesh by a partitioning of the serial mesh. Refine
-   //    this mesh further in parallel to increase the resolution. Once the
-   //    parallel mesh is defined, the serial mesh can be deleted.
-   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
-   delete mesh;
-   for (int lev = 0; lev < par_ref_levels; lev++)
-   {
-      pmesh->UniformRefinement();
-   }
-
-   // Get mesh size, compare with time step and diffusion coefficient. Use
-   // classical AMG for diffusion-dominated problems, and nonsymmetric AMG
-   // based on approximate ideal restriction (AIR) for advection dominated.
-   double h_min, h_max, k_min, k_max;
-   pmesh->GetCharacteristics(h_min, h_max, k_min, k_max);
-   if (dt > d_coef*h_max) use_AIR = true;
-   else use_AIR = true;
-   cout << "ratio = " << d_coef*h_max / dt << "\n";
-
-   // 7. Define the parallel discontinuous DG finite element space on the
-   //    parallel refined mesh of the given polynomial order.
-   DG_FECollection fec(order, dim);
-   ParFiniteElementSpace *fes = new ParFiniteElementSpace(pmesh, &fec);
-
-   HYPRE_Int global_vSize = fes->GlobalTrueVSize();
-   if (myid == 0)
-   {
-      cout << "Number of unknowns: " << global_vSize << endl;
-   }
-
-   // 8. Set up and assemble the parallel bilinear and linear forms (and the
-   //    parallel hypre matrices) corresponding to the DG discretization. The
-   //    DGTraceIntegrator involves integrals over mesh interior faces.
-   ConstantCoefficient diff_coef(d_coef);
-   VectorFunctionCoefficient velocity(dim, velocity_function);
-   FunctionCoefficient u0(u0_function);
-
-   ParBilinearForm *m = new ParBilinearForm(fes);
-   m->AddDomainIntegrator(new MassIntegrator);
-
-   ParBilinearForm *s = new ParBilinearForm(fes);
-   s->AddDomainIntegrator(new DiffusionIntegrator(diff_coef));
-   s->AddInteriorFaceIntegrator(new DGDiffusionIntegrator(diff_coef, sigma,
-                                                          kappa));
-   s->AddBdrFaceIntegrator(new DGDiffusionIntegrator(diff_coef, sigma, kappa));
-
-   ParBilinearForm *k = new ParBilinearForm(fes);
-   k->AddDomainIntegrator(new ConvectionIntegrator(velocity, -1.0));
-   k->AddInteriorFaceIntegrator(
-      new TransposeIntegrator(new DGTraceIntegrator(velocity, 1.0, -0.5)));
-   k->AddBdrFaceIntegrator(
-      new TransposeIntegrator(new DGTraceIntegrator(velocity, 1.0, -0.5)));
-
-   ParLinearForm *b = new ParLinearForm(fes);
-   b->AddBdrFaceIntegrator(
-      new DGDirichletLFIntegrator(u0, diff_coef, sigma, kappa));
-
-   int skip_zeros = 0;
-   m->Assemble(skip_zeros);
-   m->Finalize(skip_zeros);
-   s->Assemble(skip_zeros);
-   s->Finalize(skip_zeros);
-   k->Assemble(skip_zeros);
-   k->Finalize(skip_zeros);
-   b->Assemble();
-
-   HypreParMatrix *M = m->ParallelAssemble();
-   HypreParMatrix *S = s->ParallelAssemble();
-   HypreParMatrix *K = k->ParallelAssemble();
-   HypreParVector *B = b->ParallelAssemble();
-
-   // 9. Define the initial conditions, save the corresponding grid function to
-   //    a file and (optionally) save data in the VisIt format and initialize
-   //    GLVis visualization.
-   ParGridFunction *u = new ParGridFunction(fes);
-   u->ProjectCoefficient(u0);
-   HypreParVector *U = u->GetTrueDofs();
-
-   {
-      ostringstream mesh_name, sol_name;
-      mesh_name << "ex23-mesh." << setfill('0') << setw(6) << myid;
-      sol_name << "ex23-init." << setfill('0') << setw(6) << myid;
-      ofstream omesh(mesh_name.str().c_str());
-      omesh.precision(precision);
-      pmesh->Print(omesh);
-      ofstream osol(sol_name.str().c_str());
-      osol.precision(precision);
-      u->Save(osol);
-   }
-
-   // Create data collection for solution output: either VisItDataCollection for
-   // ascii data files, or SidreDataCollection for binary data files.
-   DataCollection *dc = NULL;
-   if (visit)
-   {
-      if (binary)
-      {
-#ifdef MFEM_USE_SIDRE
-         dc = new SidreDataCollection("Example23-Parallel", pmesh);
-#else
-         MFEM_ABORT("Must build with MFEM_USE_SIDRE=YES for binary output.");
-#endif
-      }
-      else
-      {
-         dc = new VisItDataCollection("Example23-Parallel", pmesh);
-         dc->SetPrecision(precision);
-         // To save the mesh using MFEM's parallel mesh format:
-         // dc->SetFormat(DataCollection::PARALLEL_FORMAT);
-      }
-      dc->RegisterField("solution", u);
-      dc->SetCycle(0);
-      dc->SetTime(0.0);
-      dc->Save();
-   }
-
-   socketstream sout;
-   if (visualization)
-   {
-      char vishost[] = "localhost";
-      int  visport   = 19916;
-      sout.open(vishost, visport);
-      if (!sout)
-      {
-         if (myid == 0)
-            cout << "Unable to connect to GLVis server at "
-                 << vishost << ':' << visport << endl;
-         visualization = false;
-         if (myid == 0)
-         {
-            cout << "GLVis visualization disabled.\n";
-         }
-      }
-      else
-      {
-         sout << "parallel " << num_procs << " " << myid << "\n";
-         sout.precision(precision);
-         sout << "solution\n" << *pmesh << *u;
-         sout << "pause\n";
-         sout << flush;
-         if (myid == 0)
-            cout << "GLVis visualization paused."
-                 << " Press space (in the GLVis window) to resume it.\n";
-      }
-   }
-
-   // 10. Define the time-dependent evolution operator describing the ODE
-   //     right-hand side, and perform time-integration (looping over the time
-   //     iterations, ti, with a time-step dt).
-   FE_Evolution adv(*M, *S, *K, *B, order);
-
-   double t = 0.0;
-   adv.SetTime(t);
-   ode_solver->Init(adv);
-
-   bool done = false;
-   for (int ti = 0; !done; )
-   {
-      double dt_real = min(dt, t_final - t);
-      ode_solver->Step(*U, t, dt_real);
-      ti++;
-
-      done = (t >= t_final - 1e-8*dt);
-
-      if (done || ti % vis_steps == 0)
-      {
-         if (myid == 0)
-         {
-            cout << "time step: " << ti << ", time: " << t << endl;
-         }
-
-         // 11. Extract the parallel grid function corresponding to the finite
-         //     element approximation U (the local solution on each processor).
-         *u = *U;
-
-         if (visualization)
-         {
-            sout << "parallel " << num_procs << " " << myid << "\n";
-            sout << "solution\n" << *pmesh << *u << flush;
-         }
-
-         if (visit)
-         {
-            dc->SetCycle(ti);
-            dc->SetTime(t);
-            dc->Save();
-         }
-      }
-   }
-
-   // 12. Save the final solution in parallel. This output can be viewed later
-   //     using GLVis: "glvis -np <np> -m ex23-mesh -g ex23-final".
-   {
-      *u = *U;
-      ostringstream sol_name;
-      sol_name << "ex23-final." << setfill('0') << setw(6) << myid;
-      ofstream osol(sol_name.str().c_str());
-      osol.precision(precision);
-      u->Save(osol);
-   }
-
-   // 13. Free the used memory.
-   delete U;
-   delete u;
-   delete B;
-   delete b;
-   delete K;
-   delete k;
-   delete S;
-   delete s;
-   delete M;
-   delete m;
-   delete fes;
-   delete pmesh;
-   delete ode_solver;
-   delete dc;
-
-   MPI_Finalize();
-   return 0;
-}
-
-
-// Implementation of class FE_Evolution
-FE_Evolution::FE_Evolution(HypreParMatrix &_M, HypreParMatrix &_S,
-                           HypreParMatrix &_K, const Vector &_b, int order)
-   : TimeDependentOperator(_M.Height()),
-     M(_M), S(_S), K(_K), b(_b), GMRES_solver(NULL), AMG_solver(NULL),
-     M_prec(M), M_solver(M.GetComm()), A(NULL),
-     dt(-1.0), z(M.Height())
-{
-   M_prec.SetType(HypreSmoother::Jacobi);
-   M_solver.SetPreconditioner(M_prec);
-   M_solver.SetOperator(M);
-
-   M_solver.iterative_mode = false;
-   M_solver.SetRelTol(1e-9);
-   M_solver.SetAbsTol(0.0);
-   M_solver.SetMaxIter(100);
-   M_solver.SetPrintLevel(0);
-
-   // DG block size given by (FEorder+1)^2 on square meshes.
-   blocksize = (order+1)*(order+1);
-}
-
-
-void FE_Evolution::Mult(const Vector &x, Vector &y) const
-{
-   // y = M^{-1} (-S x + K x + b)
-   S.Mult(-1.0, x, 0.0, z);
-   K.Mult(1.0, x, 1.0, z);
-   z += b;
-   M_solver.Mult(z, y);
-}
-
-
-void FE_Evolution::ImplicitSolve(const double _dt, const Vector &x, Vector &y)
-{
-   if ((fabs(dt - _dt) > 1e-4 * _dt) || !A)
-   {
-      delete GMRES_solver;
-      delete AMG_solver;
-      delete A;
-
-      dt = _dt;
-      HypreParMatrix *SK = Add(1.0, S, -1.0, K);
-      A = Add(1.0, M, dt, *SK);
-      delete SK;
-
-      BlockInvScal(A, &A_s, NULL, NULL, blocksize, 0);
-
-      int print_level = 1;
-      AMG_solver = new HypreBoomerAMG(A_s);
-      AMG_solver->SetMaxLevels(50);      
-      if (use_AIR) {
-         AMG_solver->SetLAIROptions(1.5, "", "FFC", 0.1, 0.01, 0.0,
-                                    100, 3, 0.0, 10, -1, 1);
-                                // 100, 3, 0.0, 6, -1, 1);
-      }
-      else {
-         AMG_solver->SetInterpolation(0);
-         AMG_solver->SetCoarsening(6);
-         AMG_solver->SetAggressiveCoarsening(1);
-      }
-
-      if (use_gmres) {
-         GMRES_solver = new HypreGMRES(A_s);
-         GMRES_solver->SetTol(1e-12);
-         GMRES_solver->SetMaxIter(100);
-         GMRES_solver->SetPrintLevel(print_level);
-         GMRES_solver->SetPreconditioner(*AMG_solver);
-         GMRES_solver->iterative_mode = false;
-      }
-      else {
-         AMG_solver->SetPrintLevel(print_level);
-         AMG_solver->SetTol(1e-12);
-         AMG_solver->SetMaxIter(100);
-      }
-   }
-
-   // y = (M + dt S - dt K)^{-1} (-S x + K x + b)
-   S.Mult(-1.0, x, 0.0, z);
-   K.Mult(1.0, x, 1.0, z);
-   z += b;
-
-   // Scale block inverse to right hand side
-   HypreParVector b_s;
-   BlockInvScal(A, NULL, &z, &b_s, blocksize, 2);
-   GMRES_solver->Mult(b_s, y);
-
-}
-
-// Velocity coefficient
-void velocity_function(const Vector &x, Vector &v)
-{
-   int dim = x.Size();
-
-   // map to the reference [-1,1] domain
-   Vector X(dim);
-   for (int i = 0; i < dim; i++)
-   {
-      double center = (bb_min[i] + bb_max[i]) * 0.5;
-      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
-   }
-
-   switch (problem)
-   {
-      case 0:
-      {
-         // Translations in 1D, 2D, and 3D
-         switch (dim)
-         {
-            case 1: v(0) = 1.0; break;
-            case 2: v(0) = sqrt(2./3.); v(1) = sqrt(1./3.); break;
-            case 3: v(0) = sqrt(3./6.); v(1) = sqrt(2./6.); v(2) = sqrt(1./6.);
-               break;
-         }
-         break;
-      }
-      case 1:
-      case 2:
-      {
-         // Clockwise rotation in 2D around the origin
-         const double w = M_PI/2;
-         switch (dim)
-         {
-            case 1: v(0) = 1.0; break;
-            case 2: v(0) = w*X(1); v(1) = -w*X(0); break;
-            case 3: v(0) = w*X(1); v(1) = -w*X(0); v(2) = 0.0; break;
-         }
-         break;
-      }
-      case 3:
-      {
-         // Clockwise twisting rotation in 2D around the origin
-         const double w = M_PI/2;
-         double d = max((X(0)+1.)*(1.-X(0)),0.) * max((X(1)+1.)*(1.-X(1)),0.);
-         d = d*d;
-         switch (dim)
-         {
-            case 1: v(0) = 1.0; break;
-            case 2: v(0) = d*w*X(1); v(1) = -d*w*X(0); break;
-            case 3: v(0) = d*w*X(1); v(1) = -d*w*X(0); v(2) = 0.0; break;
-         }
-         break;
-      }
-   }
-}
-
-// Initial condition
-double u0_function(const Vector &x)
-{
-   int dim = x.Size();
-
-   // map to the reference [-1,1] domain
-   Vector X(dim);
-   for (int i = 0; i < dim; i++)
-   {
-      double center = (bb_min[i] + bb_max[i]) * 0.5;
-      X(i) = 2 * (x(i) - center) / (bb_max[i] - bb_min[i]);
-   }
-
-   switch (problem)
-   {
-      case 0:
-      case 1:
-      {
-         switch (dim)
-         {
-            case 1:
-               return exp(-40.*pow(X(0)-0.5,2));
-            case 2:
-            case 3:
-            {
-               double rx = 0.45, ry = 0.25, cx = 0., cy = -0.2, w = 10.;
-               if (dim == 3)
-               {
-                  const double s = (1. + 0.25*cos(2*M_PI*X(2)));
-                  rx *= s;
-                  ry *= s;
-               }
-               return ( erfc(w*(X(0)-cx-rx))*erfc(-w*(X(0)-cx+rx)) *
-                        erfc(w*(X(1)-cy-ry))*erfc(-w*(X(1)-cy+ry)) )/16;
-            }
-         }
-      }
-      case 2:
-      {
-         double x_ = X(0), y_ = X(1), rho, phi;
-         rho = hypot(x_, y_);
-         phi = atan2(y_, x_);
-         return pow(sin(M_PI*rho),2)*sin(3*phi);
-      }
-      case 3:
-      {
-         const double f = M_PI;
-         return sin(f*X(0))*sin(f*X(1));
-      }
-   }
-   return 0.0;
-}
-
-// Inflow boundary condition (zero for the problems considered in this example)
-double inflow_function(const Vector &x)
-{
-   switch (problem)
-   {
-      case 0:
-      case 1:
-      case 2:
-      case 3: return 0.0;
-   }
-   return 0.0;
-}

examples/ex3p.cpp

@@ -102,8 +102,7 @@ int main(int argc, char *argv[])
    //    'ref_levels' to be the largest number that gives a final mesh with no
    //    more than 1,000 elements.
    {
-      int ref_levels =
-         (int)floor(log(1000./mesh->GetNE())/log(2.)/dim);
+      int ref_levels = (int)floor(log(1000./mesh->GetNE())/log(2.)/dim);
       for (int l = 0; l < ref_levels; l++)
       {
          mesh->UniformRefinement();

examples/ex9p.cpp

@@ -37,7 +37,7 @@ using namespace mfem;
 
 // Choice for the problem setup. The fluid velocity, initial condition and
 // inflow boundary condition are chosen based on this parameter.
-int problem, trisolve, use_gmres;
+int problem;
 
 // Velocity coefficient
 void velocity_function(const Vector &x, Vector &v);
@@ -51,19 +51,6 @@ double inflow_function(const Vector &x);
 // Mesh bounding box
 Vector bb_min, bb_max;
 
-struct AIR_parameters {
-   double distanceR;
-   std::string prerelax;
-   std::string postrelax;
-   int interp_type;
-   int relax_type;
-   int coarsen_type;
-   double strength_tolC;
-   double strength_tolR;
-   double filter_tolR;
-   double filterA_tol;
-};
-
 
 /** A time-dependent operator for the right-hand side of the ODE. The DG weak
     form of du/dt = -v.grad(u) is M du/dt = K u + b, where M and K are the mass
@@ -73,33 +60,19 @@ struct AIR_parameters {
 class FE_Evolution : public TimeDependentOperator
 {
 private:
-   HypreParMatrix &M, &K, *A, A_s;
+   HypreParMatrix &M, &K;
    const Vector &b;
    HypreSmoother M_prec;
    CGSolver M_solver;
 
-   // Preconditioner/solvers for A
-   HypreBoomerAMG *AMG_solver;
-   HypreGMRES     *GMRES_solver;
-   HypreTriSolve *preconditioner;
-   AIR_parameters &AIR;
-
-   double dt;
-   int blocksize;
-
    mutable Vector z;
 
 public:
-   FE_Evolution(HypreParMatrix &_M, HypreParMatrix &_K, const Vector &_b,
-                int order, AIR_parameters &_AIR);
+   FE_Evolution(HypreParMatrix &_M, HypreParMatrix &_K, const Vector &_b);
 
-   /** Solve the Backward-Euler equation: d = f(x + dt*d, t+dt), where u_t = f(x,t).
-       This is the only requirement for high-order SDIRK implicit integration.*/
-   virtual void ImplicitSolve(const double dt, const Vector &u, Vector &k);
    virtual void Mult(const Vector &x, Vector &y) const;
 
-   virtual ~FE_Evolution();
-
+   virtual ~FE_Evolution() { }
 };
 
 
@@ -113,28 +86,21 @@ int main(int argc, char *argv[])
 
    // 2. Parse command-line options.
    problem = 0;
-   use_gmres = 0;
-   trisolve = 0;
    const char *mesh_file = "../data/periodic-hexagon.mesh";
    int ser_ref_levels = 2;
    int par_ref_levels = 0;
    int order = 3;
-   int ode_solver_type = 3;
+   int ode_solver_type = 4;
    double t_final = 10.0;
    double dt = 0.01;
    bool visualization = true;
    bool visit = false;
    bool binary = false;
    int vis_steps = 5;
-   int basis_type = 1;
 
    int precision = 8;
    cout.precision(precision);
 
-    AIR_parameters AIR = {1, "", "FA", 100, 10, 10, 0.1, 0.01, 0.0, 1e-4};
-    const char* temp_prerelax = "";
-    const char* temp_postrelax = "FA";
-
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
                   "Mesh file to use.");
@@ -153,32 +119,6 @@ int main(int argc, char *argv[])
                   "Final time; start time is 0.");
    args.AddOption(&dt, "-dt", "--time-step",
                   "Time step.");
-   args.AddOption(&basis_type, "-b", "--basis-type",
-                  "DG finite element basis type. 0 for G-Leg, 1 for G-Lob.");
-   args.AddOption(&use_gmres, "-gmres", "--use-gmres",
-                  "Boolean to use GMRES as solver (default with AIR preconditioning).");
-   args.AddOption(&trisolve, "-trisolve", "--precond-trisolve",
-                  "Precondition GMRES with an on-processor triangular solve.");
-   args.AddOption(&(AIR.distanceR), "-Ad", "--AIR-distance",
-                  "Distance restriction neighborhood for AIR.");
-   args.AddOption(&(AIR.interp_type), "-Ai", "--AIR-interpolation",
-                  "Index for hypre interpolation routine.");
-   args.AddOption(&(AIR.coarsen_type), "-Ac", "--AIR-coarsen_type",
-                  "Index for hypre coarsening routine.");
-   args.AddOption(&(AIR.strength_tolC), "-AsC", "--AIR-strengthC",
-                  "Theta value determining strong connections for AIR (coarsen_type).");
-   args.AddOption(&(AIR.strength_tolR), "-AsR", "--AIR-strengthR",
-                  "Theta value determining strong connections for AIR (restriction).");
-   args.AddOption(&(AIR.filter_tolR), "-AfR", "--AIR-filterR",
-                  "Theta value eliminating small entries in restriction (after building).");
-   args.AddOption(&(AIR.filterA_tol), "-Af", "--AIR-filter",
-                  "Theta value to eliminate small connections in AIR hierarchy. Use -1 to specify O(h).");
-   args.AddOption(&(AIR.relax_type), "-Ar", "--AIR-relaxation",
-                  "Index for hypre relaxation routine.");
-   args.AddOption(&temp_prerelax, "-Ar1", "--AIR-prerelax",
-                  "String denoting prerelaxation scheme; e.g., FCC.");
-   args.AddOption(&temp_postrelax, "-Ar2", "--AIR-postrelax",
-                  "String denoting postrelaxation scheme; e.g., FFC.");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
                   "Enable or disable GLVis visualization.");
@@ -191,9 +131,6 @@ int main(int argc, char *argv[])
    args.AddOption(&vis_steps, "-vs", "--visualization-steps",
                   "Visualize every n-th timestep.");
    args.Parse();
-   AIR.prerelax = std::string(temp_prerelax);
-   AIR.postrelax = std::string(temp_postrelax);
-   if (trisolve) use_gmres = 1;
    if (!args.Good())
    {
       if (myid == 0)
@@ -218,21 +155,11 @@ int main(int argc, char *argv[])
    ODESolver *ode_solver = NULL;
    switch (ode_solver_type)
    {
-      // Implicit L-stable methods
-      case 1:  ode_solver = new BackwardEulerSolver; break;
-      case 2:  ode_solver = new SDIRK23Solver(2); break;
-      case 3:  ode_solver = new SDIRK33Solver; break;
-      // Explicit methods
-      case 11: ode_solver = new ForwardEulerSolver; break;
-      case 12: ode_solver = new RK2Solver(1.0); break;
-      case 13: ode_solver = new RK3SSPSolver; break;
-      case 14: ode_solver = new RK4Solver; break;
-      case 15: ode_solver = new GeneralizedAlphaSolver(0.5); break;
-      case 16: ode_solver = new RK6Solver; break;
-      // Implicit A-stable methods (not L-stable)
-      case 22: ode_solver = new ImplicitMidpointSolver; break;
-      case 23: ode_solver = new SDIRK23Solver; break;
-      case 24: ode_solver = new SDIRK34Solver; break;
+      case 1: ode_solver = new ForwardEulerSolver; break;
+      case 2: ode_solver = new RK2Solver(1.0); break;
+      case 3: ode_solver = new RK3SSPSolver; break;
+      case 4: ode_solver = new RK4Solver; break;
+      case 6: ode_solver = new RK6Solver; break;
       default:
          if (myid == 0)
          {
@@ -268,10 +195,8 @@ int main(int argc, char *argv[])
    }
 
    // 7. Define the parallel discontinuous DG finite element space on the
-   //    parallel refined mesh of the given polynomial order. Basis_type=1
-   //    gives Gauss-Lobatto quadrature points, which are preferable for
-   //    nonsymmetric AMG implict solves. 
-   DG_FECollection fec(order, dim, basis_type);
+   //    parallel refined mesh of the given polynomial order.
+   DG_FECollection fec(order, dim);
    ParFiniteElementSpace *fes = new ParFiniteElementSpace(pmesh, &fec);
 
    HYPRE_Int global_vSize = fes->GlobalTrueVSize();
@@ -389,7 +314,7 @@ int main(int argc, char *argv[])
    // 10. Define the time-dependent evolution operator describing the ODE
    //     right-hand side, and perform time-integration (looping over the time
    //     iterations, ti, with a time-step dt).
-   FE_Evolution adv(*M, *K, *B, order, AIR);
+   FE_Evolution adv(*M, *K, *B);
 
    double t = 0.0;
    adv.SetTime(t);
@@ -462,11 +387,9 @@ int main(int argc, char *argv[])
 
 // Implementation of class FE_Evolution
 FE_Evolution::FE_Evolution(HypreParMatrix &_M, HypreParMatrix &_K,
-                           const Vector &_b, int order,
-                           AIR_parameters &_AIR)
-   : TimeDependentOperator(_M.Height()), A(NULL), AMG_solver(NULL),
-     GMRES_solver(NULL), preconditioner(NULL), M(_M), K(_K), b(_b),
-     M_solver(M.GetComm()), z(_M.Height()), AIR(_AIR)
+                           const Vector &_b)
+   : TimeDependentOperator(_M.Height()),
+     M(_M), K(_K), b(_b), M_solver(M.GetComm()), z(_M.Height())
 {
    M_prec.SetType(HypreSmoother::Jacobi);
    M_solver.SetPreconditioner(M_prec);
@@ -477,23 +400,8 @@ FE_Evolution::FE_Evolution(HypreParMatrix &_M, HypreParMatrix &_K,
    M_solver.SetAbsTol(0.0);
    M_solver.SetMaxIter(100);
    M_solver.SetPrintLevel(0);
-
-   // DG block size given by (FEorder+1)^2 on square meshes.
-   blocksize = (order+1)*(order+1);
-   dt = -1;
 }
 
-
-FE_Evolution::~FE_Evolution()
-{
-   BlockInvScal(NULL, NULL, NULL, NULL, 0, -1);
-   if (A) delete A;
-   if (AMG_solver) delete AMG_solver;
-   if (GMRES_solver) delete GMRES_solver;
-   if (preconditioner) delete preconditioner;
-}
-
-
 void FE_Evolution::Mult(const Vector &x, Vector &y) const
 {
    // y = M^{-1} (K x + b)
@@ -503,74 +411,6 @@ void FE_Evolution::Mult(const Vector &x, Vector &y) const
 }
 
 
-// Solve the equation:
-//    u_t = M^{-1}(Ku + b), 
-// by solving associated linear system
-//    (M - dt*K) d = K*u + b
-void FE_Evolution::ImplicitSolve(const double dt_, const Vector &u, Vector &du_dt)
-{
-   // if A is NULL or dt has changed since A was built, rebuild matrix and solver.
-   if ( (fabs(dt - dt_) > 1e-4 * dt) || !A ) {
-      delete GMRES_solver;
-      delete AMG_solver;
-      delete preconditioner;
-      delete A;
-
-      dt = dt_;
-      A = HypreParMatrixAdd(1.0, M, -1.0*dt, K);
-
-      // Scale A by block-diagonal inverse
-      BlockInvScal(A, &A_s, NULL, NULL, blocksize, 0);
-
-      int print_level = 1;
-      if (!trisolve) {
-         AMG_solver = new HypreBoomerAMG(A_s);
-         AMG_solver->SetLAIROptions(AIR.distanceR, AIR.prerelax, AIR.postrelax,
-                                    AIR.strength_tolC, AIR.strength_tolR, AIR.filter_tolR,
-                                    AIR.interp_type, AIR.relax_type, AIR.filterA_tol,
-                                    AIR.coarsen_type, -1, 1);  
-         AMG_solver->SetMaxLevels(50);
-         if (use_gmres) {
-            GMRES_solver = new HypreGMRES(A_s);
-            GMRES_solver->SetTol(1e-12);
-            GMRES_solver->SetMaxIter(100);
-            GMRES_solver->SetPrintLevel(print_level);
-            GMRES_solver->SetPreconditioner(*AMG_solver);
-            GMRES_solver->iterative_mode = false;
-         }
-         else {
-            AMG_solver->SetPrintLevel(print_level);
-            AMG_solver->SetTol(1e-12);
-            AMG_solver->SetMaxIter(100);
-         }
-      }
-      else {
-         preconditioner = new HypreTriSolve();
-         GMRES_solver = new HypreGMRES(A_s);
-         GMRES_solver->SetTol(1e-12);
-         GMRES_solver->SetMaxIter(100);
-         GMRES_solver->SetPrintLevel(print_level);
-         GMRES_solver->SetPreconditioner(*preconditioner);
-         GMRES_solver->SetZeroInintialIterate();
-         GMRES_solver->iterative_mode = false;
-      }
-   }
-
-   K.Mult(u, z);
-   z += b;
-
-   // scale the rhs and solve system 
-   HypreParVector z_s;
-   BlockInvScal(A, NULL, &z, &z_s, blocksize, 2);
-   if (use_gmres){
-      GMRES_solver->Mult(z_s, du_dt);
-   }
-   else {
-      AMG_solver->Mult(z_s, du_dt);
-   }
-}
-
-
 // Velocity coefficient
 void velocity_function(const Vector &x, Vector &v)
 {

examples/makefile

@@ -22,9 +22,9 @@ MFEM_LIB_FILE = mfem_is_not_built
 -include $(CONFIG_MK)
 
 SEQ_EXAMPLES = ex1 ex2 ex3 ex4 ex5 ex6 ex7 ex8 ex9 ex10 ex14 ex15 ex16 ex17\
-  ex18 ex19 ex20 ex21 ex23
+  ex18 ex19 ex20 ex21
 PAR_EXAMPLES = ex1p ex2p ex3p ex4p ex5p ex6p ex7p ex8p ex9p ex10p ex11p ex12p\
- ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex21p MFEM_adv ex23TRp exETRp
+ ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex21p
 
 ifeq ($(MFEM_USE_MPI),NO)
    EXAMPLES = $(SEQ_EXAMPLES)
@@ -117,7 +117,7 @@ clean-build:
 
 clean-exec:
 	@rm -f refined.mesh displaced.mesh mesh.* ex5.mesh
-	@rm -rf Example5* Example9* Example15* Example16* Example23*
+	@rm -rf Example5* Example9* Example15* Example16*
 	@rm -f sphere_refined.* sol.* sol_u.* sol_p.*
 	@rm -f ex9.mesh ex9-mesh.* ex9-init.* ex9-final.*
 	@rm -f deformed.* velocity.* elastic_energy.* mode_*
@@ -125,9 +125,4 @@ clean-exec:
 	@rm -f vortex-mesh.* vortex.mesh vortex-?-init.* vortex-?-final.*
 	@rm -f deformation.* pressure.*
 	@rm -f ex20.dat ex20p_?????.dat gnuplot_ex20.inp gnuplot_ex20p.inp
-<<<<<<< HEAD
-	@rm -f ex22*.mesh ex22*.sol ex22p_*.*
-	@rm -f ex23.mesh ex23-mesh.* ex23-init.* ex23-final.*
-=======
 	@rm -f ex21*.mesh ex21*.sol ex21p_*.*
->>>>>>> f8a3a379d13841c6e63b9d8fbc868aa325b8afc0

examples/sundials/CMakeLists.txt

@@ -42,12 +42,12 @@ add_mfem_examples(SUNDIALS_EXAMPLES_SRCS ${PFX} "" test_sundials)
 #   ctest -R sundials
 
 # Command line options for the tests.
-# Example 9: test explicit CVODE time stepping
-set(EX9_COMMON_OPTS -m ../../data/periodic-hexagon.mesh -p 0 -s 11)
+# Example 9: test CVODE with CV_ADAMS (non-stiff implicit) time stepping
+set(EX9_COMMON_OPTS -m ../../data/periodic-hexagon.mesh -p 0 -s 7)
 set(EX9_TEST_OPTS ${EX9_COMMON_OPTS}  -r  2 -dt 0.0018 -vs 25)
 set(EX9P_TEST_OPTS ${EX9_COMMON_OPTS} -rp 1 -dt 0.0009 -vs 50)
-# Example 10: test implicit CVODE time stepping
-set(EX10_COMMON_OPTS -m ../../data/beam-quad.mesh -o 2 -s 5 -dt 0.15 -vs 10)
+# Example 10: test CVODE with CV_BDF (stiff implicit) time stepping
+set(EX10_COMMON_OPTS -m ../../data/beam-quad.mesh -o 2 -s 5 -dt 0.15 -tf 6 -vs 10)
 set(EX10_TEST_OPTS ${EX10_COMMON_OPTS}  -r  2)
 set(EX10P_TEST_OPTS ${EX10_COMMON_OPTS} -rp 1)
 # Example 16: use the default options

examples/sundials/ex10.cpp

@@ -4,16 +4,16 @@
 // Compile with: make ex10
 //
 // Sample runs:
-//    ex10 -m ../../data/beam-quad.mesh -r 2 -o 2 -s  5 -dt 0.15 -vs 10
-//    ex10 -m ../../data/beam-tri.mesh  -r 2 -o 2 -s  7 -dt 0.3  -vs 5
-//    ex10 -m ../../data/beam-hex.mesh  -r 1 -o 2 -s  5 -dt 0.2  -vs 5
+//    ex10 -m ../../data/beam-quad.mesh -r 2 -o 2 -s 12 -dt 0.15 -vs 10
+//    ex10 -m ../../data/beam-tri.mesh  -r 2 -o 2 -s 16 -dt 0.3  -vs 5
+//    ex10 -m ../../data/beam-hex.mesh  -r 1 -o 2 -s 12 -dt 0.2  -vs 5
 //    ex10 -m ../../data/beam-tri.mesh  -r 2 -o 2 -s  2 -dt 3 -nls kinsol
 //    ex10 -m ../../data/beam-quad.mesh -r 2 -o 2 -s  2 -dt 3 -nls kinsol
 //    ex10 -m ../../data/beam-hex.mesh  -r 1 -o 2 -s  2 -dt 3 -nls kinsol
-//    ex10 -m ../../data/beam-quad.mesh -r 2 -o 2 -s 15 -dt 5e-3 -vs 60
-//    ex10 -m ../../data/beam-tri.mesh  -r 2 -o 2 -s 16 -dt 0.01 -vs 30
-//    ex10 -m ../../data/beam-hex.mesh  -r 1 -o 2 -s 15 -dt 0.01 -vs 30
-//    ex10 -m ../../data/beam-quad-amr.mesh -r 2 -o 2 -s 5 -dt 0.15 -vs 10
+//    ex10 -m ../../data/beam-quad.mesh -r 2 -o 2 -s 14 -dt 0.15 -vs 10
+//    ex10 -m ../../data/beam-tri.mesh  -r 2 -o 2 -s 17 -dt 0.01 -vs 30
+//    ex10 -m ../../data/beam-hex.mesh  -r 1 -o 2 -s 14 -dt 0.15 -vs 10
+//    ex10 -m ../../data/beam-quad-amr.mesh -r 2 -o 2 -s 12 -dt 0.15 -vs 10
 //
 // Description:  This examples solves a time dependent nonlinear elasticity
 //               problem of the form dv/dt = H(x) + S v, dx/dt = v, where H is a
@@ -53,7 +53,6 @@ using namespace std;
 using namespace mfem;
 
 class ReducedSystemOperator;
-class SundialsJacSolver;
 
 /** After spatial discretization, the hyperelastic model can be written as a
  *  system of ODEs:
@@ -92,12 +91,17 @@ protected:
 
    mutable Vector z; // auxiliary vector
 
+   SparseMatrix *grad_H;
+   SparseMatrix *Jacobian;
+
+   double saved_gamma; // saved gamma value from implicit setup
+
 public:
    /// Solver type to use in the ImplicitSolve() method, used by SDIRK methods.
    enum NonlinearSolverType
    {
       NEWTON = 0, ///< Use MFEM's plain NewtonSolver
-      KINSOL = 1  ///< Use SUNDIALS' KINSOL (through MFEM's class KinSolver)
+      KINSOL = 1  ///< Use SUNDIALS' KINSOL (through MFEM's class KINSolver)
    };
 
    HyperelasticOperator(FiniteElementSpace &f, Array<int> &ess_bdr,
@@ -106,15 +110,41 @@ public:
 
    /// Compute the right-hand side of the ODE system.
    virtual void Mult(const Vector &vx, Vector &dvx_dt) const;
+
    /** Solve the Backward-Euler equation: k = f(x + dt*k, t), for the unknown k.
        This is the only requirement for high-order SDIRK implicit integration.*/
    virtual void ImplicitSolve(const double dt, const Vector &x, Vector &k);
 
-   /** Connect the Jacobian linear system solver (SundialsJacSolver) used by
-       SUNDIALS' CVODE and ARKODE time integrators to the internal objects
-       created by HyperelasticOperator. This method is called by the InitSystem
-       method of SundialsJacSolver. */
-   void InitSundialsJacSolver(SundialsJacSolver &sjsolv);
+
+   /// Custom Jacobian system solver for the SUNDIALS time integrators.
+   /** For the ODE system represented by HyperelasticOperator
+
+          M dv/dt = -(H(x) + S*v)
+          dx/dt = v,
+
+       this class facilitates the solution of linear systems of the form
+
+           (M + γS) yv + γJ yx = M bv,   J=(dH/dx)(x)
+                - γ yv +    yx =   bx
+
+       for given bv, bx, x, and γ = GetTimeStep(). */
+
+   /** Linear solve applicable to the SUNDIALS format.
+       Solves (Mass - dt J) y = Mass b, where in our case:
+       Mass = | M  0 |  J = | -S  -grad_H |  y = | v_hat |  b = | b_v |
+              | 0  I |      |  I     0    |      | x_hat |      | b_x |
+       The result replaces the rhs b.
+       We substitute x_hat = b_x + dt v_hat and solve
+       (M + dt S + dt^2 grad_H) v_hat = M b_v - dt grad_H b_x. */
+
+   /** Setup the linear system. This method is used by the implicit
+       SUNDIALS solvers. */
+   virtual int SUNImplicitSetup(const Vector &y, const Vector &fy,
+                                int jok, int *jcur, double gamma);
+
+   /** Solve the linear system. This method is used by the implicit
+       SUNDIALS solvers. */
+   virtual int SUNImplicitSolve(const Vector &b, Vector &x, double tol);
 
    double ElasticEnergy(const Vector &x) const;
    double KineticEnergy(const Vector &v) const;
@@ -152,53 +182,6 @@ public:
    virtual ~ReducedSystemOperator();
 };
 
-/// Custom Jacobian system solver for the SUNDIALS time integrators.
-/** For the ODE system represented by HyperelasticOperator
-
-        M dv/dt = -(H(x) + S*v)
-          dx/dt = v,
-
-    this class facilitates the solution of linear systems of the form
-
-        (M + γS) yv + γJ yx = M bv,   J=(dH/dx)(x)
-             - γ yv +    yx =   bx
-
-    for given bv, bx, x, and γ = GetTimeStep(). */
-class SundialsJacSolver : public SundialsODELinearSolver
-{
-private:
-   BilinearForm *M, *S;
-   NonlinearForm *H;
-   SparseMatrix *grad_H, *Jacobian;
-   Solver *J_solver;
-
-public:
-   SundialsJacSolver()
-      : M(), S(), H(), grad_H(), Jacobian(), J_solver() { }
-
-   /// Connect the solver to the objects created inside HyperelasticOperator.
-   void SetOperators(BilinearForm &M_, BilinearForm &S_,
-                     NonlinearForm &H_, Solver &solver)
-   {
-      M = &M_; S = &S_; H = &H_; J_solver = &solver;
-   }
-
-   /** Linear solve applicable to the SUNDIALS format.
-       Solves (Mass - dt J) y = Mass b, where in our case:
-       Mass = | M  0 |  J = | -S  -grad_H |  y = | v_hat |  b = | b_v |
-              | 0  I |      |  I     0    |      | x_hat |      | b_x |
-       The result replaces the rhs b.
-       We substitute x_hat = b_x + dt v_hat and solve
-       (M + dt S + dt^2 grad_H) v_hat = M b_v - dt grad_H b_x. */
-   int InitSystem(void *sundials_mem);
-   int SetupSystem(void *sundials_mem, int conv_fail,
-                   const Vector &y_pred, const Vector &f_pred, int &jac_cur,
-                   Vector &v_temp1, Vector &v_temp2, Vector &v_temp3);
-   int SolveSystem(void *sundials_mem, Vector &b, const Vector &weight,
-                   const Vector &y_cur, const Vector &f_cur);
-   int FreeSystem(void *sundials_mem);
-};
-
 
 /** Function representing the elastic energy density for the given hyperelastic
     model+deformation. Used in HyperelasticOperator::GetElasticEnergyDensity. */
@@ -243,6 +226,12 @@ int main(int argc, char *argv[])
 
    // Relative and absolute tolerances for CVODE and ARKODE.
    const double reltol = 1e-1, abstol = 1e-1;
+   // Since this example uses the loose tolerances defined above, it is
+   // necessary to lower the linear solver tolerance for CVODE which is relative
+   // to the above tolerances.
+   const double cvode_eps_lin = 1e-4;
+   // Similarly, the nonlinear tolerance for ARKODE needs to be tightened.
+   const double arkode_eps_nonlin = 1e-6;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
@@ -252,15 +241,24 @@ int main(int argc, char *argv[])
    args.AddOption(&order, "-o", "--order",
                   "Order (degree) of the finite elements.");
    args.AddOption(&ode_solver_type, "-s", "--ode-solver",
-                  "ODE solver: 1 - Backward Euler, 2 - SDIRK2, 3 - SDIRK3,\n\t"
-                  "            4 - CVODE implicit, approximate Jacobian,\n\t"
-                  "            5 - CVODE implicit, specified Jacobian,\n\t"
-                  "            6 - ARKODE implicit, approximate Jacobian,\n\t"
-                  "            7 - ARKODE implicit, specified Jacobian,\n\t"
-                  "            11 - Forward Euler, 12 - RK2,\n\t"
-                  "            13 - RK3 SSP, 14 - RK4,\n\t"
-                  "            15 - CVODE (adaptive order) explicit,\n\t"
-                  "            16 - ARKODE default (4th order) explicit.");
+                  "ODE solver:\n\t"
+                  "1  - Backward Euler,\n\t"
+                  "2  - SDIRK2, L-stable\n\t"
+                  "3  - SDIRK3, L-stable\n\t"
+                  "4  - Implicit Midpoint,\n\t"
+                  "5  - SDIRK2, A-stable,\n\t"
+                  "6  - SDIRK3, A-stable,\n\t"
+                  "7  - Forward Euler,\n\t"
+                  "8  - RK2,\n\t"
+                  "9  - RK3 SSP,\n\t"
+                  "10 - RK4,\n\t"
+                  "11 - CVODE implicit BDF, approximate Jacobian,\n\t"
+                  "12 - CVODE implicit BDF, specified Jacobian,\n\t"
+                  "13 - CVODE implicit ADAMS, approximate Jacobian,\n\t"
+                  "14 - CVODE implicit ADAMS, specified Jacobian,\n\t"
+                  "15 - ARKODE implicit, approximate Jacobian,\n\t"
+                  "16 - ARKODE implicit, specified Jacobian,\n\t"
+                  "17 - ARKODE explicit, 4th order.");
    args.AddOption(&nls, "-nls", "--nonlinear-solver",
                   "Nonlinear systems solver: "
                   "\"newton\" (plain Newton) or \"kinsol\" (KINSOL).");
@@ -287,72 +285,19 @@ int main(int argc, char *argv[])
    }
    args.PrintOptions(cout);
 
+   // check for vaild ODE solver option
+   if (ode_solver_type < 1 || ode_solver_type > 17)
+   {
+      cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
+      return 1;
+   }
+
    // 2. Read the mesh from the given mesh file. We can handle triangular,
    //    quadrilateral, tetrahedral and hexahedral meshes with the same code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
 
-   // 3. Define the ODE solver used for time integration. Several implicit
-   //    singly diagonal implicit Runge-Kutta (SDIRK) methods, as well as
-   //    explicit Runge-Kutta methods are available.
-   ODESolver *ode_solver;
-   CVODESolver *cvode = NULL;
-   ARKODESolver *arkode = NULL;
-   SundialsJacSolver *sjsolver = NULL;
-   switch (ode_solver_type)
-   {
-      // Implicit L-stable methods
-      case 1: ode_solver = new BackwardEulerSolver; break;
-      case 2: ode_solver = new SDIRK23Solver(2); break;
-      case 3: ode_solver = new SDIRK33Solver; break;
-      case 4:
-      case 5:
-         cvode = new CVODESolver(CV_BDF, CV_NEWTON);
-         cvode->SetSStolerances(reltol, abstol);
-         cvode->SetMaxStep(dt);
-         if (ode_solver_type == 5)
-         {
-            sjsolver = new SundialsJacSolver;
-            cvode->SetLinearSolver(*sjsolver);
-         }
-         ode_solver = cvode; break;
-      case 6:
-      case 7:
-         arkode = new ARKODESolver(ARKODESolver::IMPLICIT);
-         arkode->SetSStolerances(reltol, abstol);
-         arkode->SetMaxStep(dt);
-         if (ode_solver_type == 7)
-         {
-            // Custom Jacobian inversion.
-            sjsolver = new SundialsJacSolver;
-            arkode->SetLinearSolver(*sjsolver);
-         }
-         ode_solver = arkode; break;
-      // Explicit methods
-      case 11: ode_solver = new ForwardEulerSolver; break;
-      case 12: ode_solver = new RK2Solver(0.5); break; // midpoint method
-      case 13: ode_solver = new RK3SSPSolver; break;
-      case 14: ode_solver = new RK4Solver; break;
-      case 15:
-         cvode = new CVODESolver(CV_ADAMS, CV_FUNCTIONAL);
-         cvode->SetSStolerances(reltol, abstol);
-         cvode->SetMaxStep(dt);
-         ode_solver = cvode; break;
-      case 16:
-         arkode = new ARKODESolver(ARKODESolver::IMPLICIT);
-         arkode->SetSStolerances(reltol, abstol);
-         arkode->SetMaxStep(dt);
-         ode_solver = arkode; break;
-      // Implicit A-stable methods (not L-stable)
-      case 22: ode_solver = new ImplicitMidpointSolver; break;
-      case 23: ode_solver = new SDIRK23Solver; break;
-      case 24: ode_solver = new SDIRK34Solver; break;
-      default:
-         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
-         delete mesh;
-         return 3;
-   }
-
+   // 3. Setup the nonlinear solver
    map<string,HyperelasticOperator::NonlinearSolverType> nls_map;
    nls_map["newton"] = HyperelasticOperator::NEWTON;
    nls_map["kinsol"] = HyperelasticOperator::KINSOL;
@@ -439,11 +384,82 @@ int main(int argc, char *argv[])
    cout << "initial kinetic energy (KE) = " << ke0 << endl;
    cout << "initial   total energy (TE) = " << (ee0 + ke0) << endl;
 
+   // 8. Define the ODE solver used for time integration. Several implicit
+   //    singly diagonal implicit Runge-Kutta (SDIRK) methods, as well as
+   //    explicit Runge-Kutta methods are available.
    double t = 0.0;
    oper.SetTime(t);
-   ode_solver->Init(oper);
 
-   // 8. Perform time-integration (looping over the time iterations, ti, with a
+   ODESolver *ode_solver = NULL;
+   CVODESolver *cvode = NULL;
+   ARKStepSolver *arkode = NULL;
+   switch (ode_solver_type)
+   {
+      // Implicit L-stable methods
+      case 1:  ode_solver = new BackwardEulerSolver; break;
+      case 2:  ode_solver = new SDIRK23Solver(2); break;
+      case 3:  ode_solver = new SDIRK33Solver; break;
+      // Implicit A-stable methods (not L-stable)
+      case 4:  ode_solver = new ImplicitMidpointSolver; break;
+      case 5:  ode_solver = new SDIRK23Solver; break;
+      case 6:  ode_solver = new SDIRK34Solver; break;
+      // Explicit methods
+      case 7:  ode_solver = new ForwardEulerSolver; break;
+      case 8:  ode_solver = new RK2Solver(0.5); break; // midpoint method
+      case 9:  ode_solver = new RK3SSPSolver; break;
+      case 10: ode_solver = new RK4Solver; break;
+      // CVODE BDF
+      case 11:
+      case 12:
+         cvode = new CVODESolver(CV_BDF);
+         cvode->Init(oper);
+         cvode->SetSStolerances(reltol, abstol);
+         CVodeSetEpsLin(cvode->GetMem(), cvode_eps_lin);
+         cvode->SetMaxStep(dt);
+         if (ode_solver_type == 11)
+         {
+            cvode->UseSundialsLinearSolver();
+         }
+         ode_solver = cvode; break;
+      // CVODE Adams
+      case 13:
+      case 14:
+         cvode = new CVODESolver(CV_ADAMS);
+         cvode->Init(oper);
+         cvode->SetSStolerances(reltol, abstol);
+         CVodeSetEpsLin(cvode->GetMem(), cvode_eps_lin);
+         cvode->SetMaxStep(dt);
+         if (ode_solver_type == 13)
+         {
+            cvode->UseSundialsLinearSolver();
+         }
+         ode_solver = cvode; break;
+      // ARKStep Implicit methods
+      case 15:
+      case 16:
+         arkode = new ARKStepSolver(ARKStepSolver::IMPLICIT);
+         arkode->Init(oper);
+         arkode->SetSStolerances(reltol, abstol);
+         ARKStepSetNonlinConvCoef(arkode->GetMem(), arkode_eps_nonlin);
+         arkode->SetMaxStep(dt);
+         if (ode_solver_type == 15)
+         {
+            arkode->UseSundialsLinearSolver();
+         }
+         ode_solver = arkode; break;
+      // ARKStep Explicit methods
+      case 17:
+         arkode = new ARKStepSolver(ARKStepSolver::EXPLICIT);
+         arkode->Init(oper);
+         arkode->SetSStolerances(reltol, abstol);
+         arkode->SetMaxStep(dt);
+         ode_solver = arkode; break;
+   }
+
+   // Initialize MFEM integrators, SUNDIALS integrators are initialized above
+   if (ode_solver_type < 11) { ode_solver->Init(oper); }
+
+   // 9. Perform time-integration (looping over the time iterations, ti, with a
    //    time-step dt).
    bool last_step = false;
    for (int ti = 1; !last_step; ti++)
@@ -478,7 +494,7 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 9. Save the displaced mesh, the velocity and elastic energy.
+   // 10. Save the displaced mesh, the velocity and elastic energy.
    {
       v.SetFromTrueVector(); x.SetFromTrueVector();
       GridFunction *nodes = &x;
@@ -497,9 +513,8 @@ int main(int argc, char *argv[])
       w.Save(ee_ofs);
    }
 
-   // 10. Free the used memory.
+   // 11. Free the used memory.
    delete ode_solver;
-   delete sjsolver;
    delete mesh;
 
    return 0;
@@ -579,81 +594,14 @@ ReducedSystemOperator::~ReducedSystemOperator()
 }
 
 
-int SundialsJacSolver::InitSystem(void *sundials_mem)
-{
-   TimeDependentOperator *td_oper = GetTimeDependentOperator(sundials_mem);
-   HyperelasticOperator *he_oper;
-
-   // During development, we use dynamic_cast<> to ensure the setup is correct:
-   he_oper = dynamic_cast<HyperelasticOperator*>(td_oper);
-   MFEM_VERIFY(he_oper, "operator is not HyperelasticOperator");
-
-   // When the implementation is finalized, we can switch to static_cast<>:
-   // he_oper = static_cast<HyperelasticOperator*>(td_oper);
-
-   he_oper->InitSundialsJacSolver(*this);
-   return 0;
-}
-
-int SundialsJacSolver::SetupSystem(void *sundials_mem, int conv_fail,
-                                   const Vector &y_pred, const Vector &f_pred,
-                                   int &jac_cur, Vector &v_temp1,
-                                   Vector &v_temp2, Vector &v_temp3)
-{
-   int sc = y_pred.Size() / 2;
-   const Vector x(y_pred.GetData() + sc, sc);
-   double dt = GetTimeStep(sundials_mem);
-
-   // J = M + dt*(S + dt*grad(H))
-   delete Jacobian;
-   Jacobian = Add(1.0, M->SpMat(), dt, S->SpMat());
-   grad_H = dynamic_cast<SparseMatrix *>(&H->GetGradient(x));
-   Jacobian->Add(dt * dt, *grad_H);
-
-   J_solver->SetOperator(*Jacobian);
-
-   jac_cur = 1;
-   return 0;
-}
-
-int SundialsJacSolver::SolveSystem(void *sundials_mem, Vector &b,
-                                   const Vector &weight, const Vector &y_cur,
-                                   const Vector &f_cur)
-{
-   int sc = b.Size() / 2;
-   // Vector x(y_cur.GetData() + sc, sc);
-   Vector b_v(b.GetData() +  0, sc);
-   Vector b_x(b.GetData() + sc, sc);
-   Vector rhs(sc);
-   double dt = GetTimeStep(sundials_mem);
-
-   // rhs = M b_v - dt*grad(H) b_x
-   grad_H->Mult(b_x, rhs);
-   rhs *= -dt;
-   M->AddMult(b_v, rhs);
-
-   J_solver->iterative_mode = false;
-   J_solver->Mult(rhs, b_v);
-
-   b_x.Add(dt, b_v);
-
-   return 0;
-}
-
-int SundialsJacSolver::FreeSystem(void *sundials_mem)
-{
-   delete Jacobian;
-   return 0;
-}
-
-
 HyperelasticOperator::HyperelasticOperator(FiniteElementSpace &f,
                                            Array<int> &ess_bdr, double visc,
                                            double mu, double K,
                                            NonlinearSolverType nls_type)
    : TimeDependentOperator(2*f.GetTrueVSize(), 0.0), fespace(f),
      M(&fespace), S(&fespace), H(&fespace),
-     viscosity(visc), z(height/2)
+     viscosity(visc), z(height/2),
+     grad_H(NULL), Jacobian(NULL)
 {
    const double rel_tol = 1e-8;
    const int skip_zero_entries = 0;
@@ -702,23 +650,24 @@ HyperelasticOperator::HyperelasticOperator(FiniteElementSpace &f,
 
    if (nls_type == KINSOL)
    {
-      KinSolver *kinsolver = new KinSolver(KIN_NONE, true);
-      kinsolver->SetMaxSetupCalls(4);
+      KINSolver *kinsolver = new KINSolver(KIN_NONE, true);
       newton_solver = kinsolver;
+      newton_solver->SetOperator(*reduced_oper);
       newton_solver->SetMaxIter(200);
       newton_solver->SetRelTol(rel_tol);
       newton_solver->SetPrintLevel(0);
+      kinsolver->SetMaxSetupCalls(4);
    }
    else
    {
       newton_solver = new NewtonSolver();
+      newton_solver->SetOperator(*reduced_oper);
       newton_solver->SetMaxIter(10);
       newton_solver->SetRelTol(rel_tol);
       newton_solver->SetPrintLevel(-1);
    }
    newton_solver->SetSolver(*J_solver);
    newton_solver->iterative_mode = false;
-   newton_solver->SetOperator(*reduced_oper);
 }
 
 void HyperelasticOperator::Mult(const Vector &vx, Vector &dvx_dt) const
@@ -768,9 +717,53 @@ void HyperelasticOperator::ImplicitSolve(const double dt,
    add(v, dt, dv_dt, dx_dt);
 }
 
-void HyperelasticOperator::InitSundialsJacSolver(SundialsJacSolver &sjsolv)
+int HyperelasticOperator::SUNImplicitSetup(const Vector &y,
+                                           const Vector &fy, int jok, int *jcur,
+                                           double gamma)
 {
-   sjsolv.SetOperators(M, S, H, *J_solver);
+   int sc = y.Size() / 2;
+   const Vector x(y.GetData() + sc, sc);
+
+   // J = M + dt*(S + dt*grad(H))
+   if (Jacobian) { delete Jacobian; }
+   Jacobian = Add(1.0, M.SpMat(), gamma, S.SpMat());
+   grad_H = dynamic_cast<SparseMatrix *>(&H.GetGradient(x));
+   Jacobian->Add(gamma * gamma, *grad_H);
+
+   // Set Jacobian solve operator
+   J_solver->SetOperator(*Jacobian);
+
+   // Indicate that the Jacobian was updated
+   *jcur = 1;
+
+   // Save gamma for use in solve
+   saved_gamma = gamma;
+
+   // Return success
+   return 0;
+}
+
+int HyperelasticOperator::SUNImplicitSolve(const Vector &b, Vector &x,
+                                           double tol)
+{
+   int sc = b.Size() / 2;
+   Vector b_v(b.GetData() +  0, sc);
+   Vector b_x(b.GetData() + sc, sc);
+   Vector x_v(x.GetData() +  0, sc);
+   Vector x_x(x.GetData() + sc, sc);
+   Vector rhs(sc);
+
+   // rhs = M b_v - dt*grad(H) b_x
+   grad_H->Mult(b_x, rhs);
+   rhs *= -saved_gamma;
+   M.AddMult(b_v, rhs);
+
+   J_solver->iterative_mode = false;
+   J_solver->Mult(rhs, x_v);
+
+   add(b_x, saved_gamma, x_v, x_x);
+
+   return 0;
 }
 
 double HyperelasticOperator::ElasticEnergy(const Vector &x) const
@@ -792,6 +785,7 @@ void HyperelasticOperator::GetElasticEnergyDensity(
 
 HyperelasticOperator::~HyperelasticOperator()
 {
+   delete Jacobian;
    delete newton_solver;
    delete J_solver;
    delete J_prec;

examples/sundials/ex10p.cpp

@@ -4,16 +4,16 @@
 // Compile with: make ex10p
 //
 // Sample runs:
-//    mpirun -np 4 ex10p -m ../../data/beam-quad.mesh -rp 1 -o 2 -s  5 -dt 0.15 -vs 10
-//    mpirun -np 4 ex10p -m ../../data/beam-tri.mesh  -rp 1 -o 2 -s  7 -dt 0.25  -vs 10
-//    mpirun -np 4 ex10p -m ../../data/beam-hex.mesh  -rp 0 -o 2 -s  5 -dt 0.15  -vs 10
+//    mpirun -np 4 ex10p -m ../../data/beam-quad.mesh -rp 1 -o 2 -s 12 -dt 0.15 -vs 10
+//    mpirun -np 4 ex10p -m ../../data/beam-tri.mesh  -rp 1 -o 2 -s 16 -dt 0.25 -vs 10
+//    mpirun -np 4 ex10p -m ../../data/beam-hex.mesh  -rp 0 -o 2 -s 12 -dt 0.15 -vs 10
 //    mpirun -np 4 ex10p -m ../../data/beam-tri.mesh  -rp 1 -o 2 -s  2 -dt 3 -nls kinsol
 //    mpirun -np 4 ex10p -m ../../data/beam-quad.mesh -rp 1 -o 2 -s  2 -dt 3 -nls kinsol
 //    mpirun -np 4 ex10p -m ../../data/beam-hex.mesh  -rs 1 -o 2 -s  2 -dt 3 -nls kinsol
-//    mpirun -np 4 ex10p -m ../../data/beam-quad.mesh -rp 1 -o 2 -s 15 -dt 3e-3 -vs 120
-//    mpirun -np 4 ex10p -m ../../data/beam-tri.mesh  -rp 1 -o 2 -s 16 -dt 5e-3 -vs 60
-//    mpirun -np 4 ex10p -m ../../data/beam-hex.mesh  -rp 0 -o 2 -s 15 -dt 5e-3 -vs 60
-//    mpirun -np 4 ex10p -m ../../data/beam-quad-amr.mesh -rp 1 -o 2 -s 5 -dt 0.15 -vs 10
+//    mpirun -np 4 ex10p -m ../../data/beam-quad.mesh -rp 1 -o 2 -s 14 -dt 0.15 -vs 10
+//    mpirun -np 4 ex10p -m ../../data/beam-tri.mesh  -rp 1 -o 2 -s 17 -dt 5e-3 -vs 60
+//    mpirun -np 4 ex10p -m ../../data/beam-hex.mesh  -rp 0 -o 2 -s 14 -dt 0.15 -vs 10
+//    mpirun -np 4 ex10p -m ../../data/beam-quad-amr.mesh -rp 1 -o 2 -s 12 -dt 0.15 -vs 10
 //
 // Description:  This examples solves a time dependent nonlinear elasticity
 //               problem of the form dv/dt = H(x) + S v, dx/dt = v, where H is a
@@ -53,7 +53,6 @@ using namespace std;
 using namespace mfem;
 
 class ReducedSystemOperator;
-class SundialsJacSolver;
 
 /** After spatial discretization, the hyperelastic model can be written as a
  *  system of ODEs:
@@ -94,12 +93,17 @@ protected:
 
    mutable Vector z; // auxiliary vector
 
+   const SparseMatrix *local_grad_H;
+   HypreParMatrix *Jacobian;
+
+   double saved_gamma; // saved gamma value from implicit setup
+
 public:
    /// Solver type to use in the ImplicitSolve() method, used by SDIRK methods.
    enum NonlinearSolverType
    {
       NEWTON = 0, ///< Use MFEM's plain NewtonSolver
-      KINSOL = 1  ///< Use SUNDIALS' KINSOL (through MFEM's class KinSolver)
+      KINSOL = 1  ///< Use SUNDIALS' KINSOL (through MFEM's class KINSolver)
    };
 
    HyperelasticOperator(ParFiniteElementSpace &f, Array<int> &ess_bdr,
@@ -108,15 +112,41 @@ public:
 
    /// Compute the right-hand side of the ODE system.
    virtual void Mult(const Vector &vx, Vector &dvx_dt) const;
+
    /** Solve the Backward-Euler equation: k = f(x + dt*k, t), for the unknown k.
        This is the only requirement for high-order SDIRK implicit integration.*/
    virtual void ImplicitSolve(const double dt, const Vector &x, Vector &k);
 
-   /** Connect the Jacobian linear system solver (SundialsJacSolver) used by
-       SUNDIALS' CVODE and ARKODE time integrators to the internal objects
-       created by HyperelasticOperator. This method is called by the InitSystem
-       method of SundialsJacSolver. */
-   void InitSundialsJacSolver(SundialsJacSolver &sjsolv);
+
+   /// Custom Jacobian system solver for the SUNDIALS time integrators.
+   /** For the ODE system represented by HyperelasticOperator
+
+          M dv/dt = -(H(x) + S*v)
+          dx/dt = v,
+
+       this class facilitates the solution of linear systems of the form
+
+           (M + γS) yv + γJ yx = M bv,   J=(dH/dx)(x)
+                - γ yv +    yx =   bx
+
+       for given bv, bx, x, and γ = GetTimeStep(). */
+
+   /** Linear solve applicable to the SUNDIALS format.
+       Solves (Mass - dt J) y = Mass b, where in our case:
+       Mass = | M  0 |  J = | -S  -grad_H |  y = | v_hat |  b = | b_v |
+              | 0  I |      |  I     0    |      | x_hat |      | b_x |
+       The result replaces the rhs b.
+       We substitute x_hat = b_x + dt v_hat and solve
+       (M + dt S + dt^2 grad_H) v_hat = M b_v - dt grad_H b_x. */
+
+   /** Setup the linear system. This method is used by the implicit
+       SUNDIALS solvers. */
+   virtual int SUNImplicitSetup(const Vector &y, const Vector &fy,
+                                int jok, int *jcur, double gamma);
+
+   /** Solve the linear system. This method is used by the implicit
+       SUNDIALS solvers. */
+   virtual int SUNImplicitSolve(const Vector &b, Vector &x, double tol);
 
    double ElasticEnergy(const ParGridFunction &x) const;
    double KineticEnergy(const ParGridFunction &v) const;
@@ -157,57 +187,6 @@ public:
    virtual ~ReducedSystemOperator();
 };
 
-/// Custom Jacobian system solver for the SUNDIALS time integrators.
-/** For the ODE system represented by HyperelasticOperator
-
-        M dv/dt = -(H(x) + S*v)
-          dx/dt = v,
-
-    this class facilitates the solution of linear systems of the form
-
-        (M + γS) yv + γJ yx = M bv,   J=(dH/dx)(x)
-             - γ yv +    yx =   bx
-
-    for given bv, bx, x, and γ = GetTimeStep(). */
-class SundialsJacSolver : public SundialsODELinearSolver
-{
-private:
-   ParBilinearForm *M, *S;
-   ParNonlinearForm *H;
-   const SparseMatrix *local_grad_H;
-   HypreParMatrix *Jacobian;
-   Solver *J_solver;
-   const Array<int> *ess_tdof_list;
-
-public:
-   SundialsJacSolver()
-      : M(), S(), H(), local_grad_H(), Jacobian(), J_solver() { }
-
-   /// Connect the solver to the objects created inside HyperelasticOperator.
-   void SetOperators(ParBilinearForm &M_, ParBilinearForm &S_,
-                     ParNonlinearForm &H_, Solver &solver,
-                     const Array<int> &ess_tdof_list_)
-   {
-      M = &M_; S = &S_; H = &H_; J_solver = &solver;
-      ess_tdof_list = &ess_tdof_list_;
-   }
-
-   /** Linear solve applicable to the SUNDIALS format.
-       Solves (Mass - dt J) y = Mass b, where in our case:
-       Mass = | M  0 |  J = | -S  -grad_H |  y = | v_hat |  b = | b_v |
-              | 0  I |      |  I     0    |      | x_hat |      | b_x |
-       The result replaces the rhs b.
-       We substitute x_hat = b_x + dt v_hat and solve
-       (M + dt S + dt^2 grad_H) v_hat = M b_v - dt grad_H b_x. */
-   int InitSystem(void *sundials_mem);
-   int SetupSystem(void *sundials_mem, int conv_fail,
-                   const Vector &y_pred, const Vector &f_pred, int &jac_cur,
-                   Vector &v_temp1, Vector &v_temp2, Vector &v_temp3);
-   int SolveSystem(void *sundials_mem, Vector &b, const Vector &weight,
-                   const Vector &y_cur, const Vector &f_cur);
-   int FreeSystem(void *sundials_mem);
-};
-
 
 /** Function representing the elastic energy density for the given hyperelastic
     model+deformation. Used in HyperelasticOperator::GetElasticEnergyDensity. */
@@ -259,6 +238,12 @@ int main(int argc, char *argv[])
 
    // Relative and absolute tolerances for CVODE and ARKODE.
    const double reltol = 1e-1, abstol = 1e-1;
+   // Since this example uses the loose tolerances defined above, it is
+   // necessary to lower the linear solver tolerance for CVODE which is relative
+   // to the above tolerances.
+   const double cvode_eps_lin = 1e-4;
+   // Similarly, the nonlinear tolerance for ARKODE needs to be tightened.
+   const double arkode_eps_nonlin = 1e-6;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
@@ -270,15 +255,24 @@ int main(int argc, char *argv[])
    args.AddOption(&order, "-o", "--order",
                   "Order (degree) of the finite elements.");
    args.AddOption(&ode_solver_type, "-s", "--ode-solver",
-                  "ODE solver: 1 - Backward Euler, 2 - SDIRK2, 3 - SDIRK3,\n\t"
-                  "            4 - CVODE implicit, approximate Jacobian,\n\t"
-                  "            5 - CVODE implicit, specified Jacobian,\n\t"
-                  "            6 - ARKODE implicit, approximate Jacobian,\n\t"
-                  "            7 - ARKODE implicit, specified Jacobian,\n\t"
-                  "            11 - Forward Euler, 12 - RK2,\n\t"
-                  "            13 - RK3 SSP, 14 - RK4,\n\t"
-                  "            15 - CVODE (adaptive order) explicit,\n\t"
-                  "            16 - ARKODE default (4th order) explicit.");
+                  "ODE solver:\n\t"
+                  "1  - Backward Euler,\n\t"
+                  "2  - SDIRK2, L-stable\n\t"
+                  "3  - SDIRK3, L-stable\n\t"
+                  "4  - Implicit Midpoint,\n\t"
+                  "5  - SDIRK2, A-stable,\n\t"
+                  "6  - SDIRK3, A-stable,\n\t"
+                  "7  - Forward Euler,\n\t"
+                  "8  - RK2,\n\t"
+                  "9  - RK3 SSP,\n\t"
+                  "10 - RK4,\n\t"
+                  "11 - CVODE implicit BDF, approximate Jacobian,\n\t"
+                  "12 - CVODE implicit BDF, specified Jacobian,\n\t"
+                  "13 - CVODE implicit ADAMS, approximate Jacobian,\n\t"
+                  "14 - CVODE implicit ADAMS, specified Jacobian,\n\t"
+                  "15 - ARKODE implicit, approximate Jacobian,\n\t"
+                  "16 - ARKODE implicit, specified Jacobian,\n\t"
+                  "17 - ARKODE explicit, 4th order.");
    args.AddOption(&nls, "-nls", "--nonlinear-solver",
                   "Nonlinear systems solver: "
                   "\"newton\" (plain Newton) or \"kinsol\" (KINSOL).");
@@ -312,76 +306,24 @@ int main(int argc, char *argv[])
       args.PrintOptions(cout);
    }
 
+   // check for vaild ODE solver option
+   if (ode_solver_type < 1 || ode_solver_type > 17)
+   {
+      if (myid == 0)
+      {
+         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
+      }
+      MPI_Finalize();
+      return 1;
+   }
+
    // 3. Read the serial mesh from the given mesh file on all processors. We can
    //    handle triangular, quadrilateral, tetrahedral and hexahedral meshes
    //    with the same code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
 
-   // 4. Define the ODE solver used for time integration. Several implicit
-   //    singly diagonal implicit Runge-Kutta (SDIRK) methods, as well as
-   //    explicit Runge-Kutta methods are available.
-   ODESolver *ode_solver;
-   CVODESolver *cvode = NULL;
-   ARKODESolver *arkode = NULL;
-   SundialsJacSolver *sjsolver = NULL;
-   switch (ode_solver_type)
-   {
-      // Implicit L-stable methods
-      case 1: ode_solver = new BackwardEulerSolver; break;
-      case 2: ode_solver = new SDIRK23Solver(2); break;
-      case 3: ode_solver = new SDIRK33Solver; break;
-      case 4:
-      case 5:
-         cvode = new CVODESolver(MPI_COMM_WORLD, CV_BDF, CV_NEWTON);
-         cvode->SetSStolerances(reltol, abstol);
-         cvode->SetMaxStep(dt);
-         if (ode_solver_type == 5)
-         {
-            sjsolver = new SundialsJacSolver;
-            cvode->SetLinearSolver(*sjsolver); // Custom Jacobian inversion.
-         }
-         ode_solver = cvode; break;
-      case 6:
-      case 7:
-         arkode = new ARKODESolver(MPI_COMM_WORLD, ARKODESolver::IMPLICIT);
-         arkode->SetSStolerances(reltol, abstol);
-         arkode->SetMaxStep(dt);
-         if (ode_solver_type == 7)
-         {
-            sjsolver = new SundialsJacSolver;
-            arkode->SetLinearSolver(*sjsolver); // Custom Jacobian inversion.
-         }
-         ode_solver = arkode; break;
-      // Explicit methods
-      case 11: ode_solver = new ForwardEulerSolver; break;
-      case 12: ode_solver = new RK2Solver(0.5); break; // midpoint method
-      case 13: ode_solver = new RK3SSPSolver; break;
-      case 14: ode_solver = new RK4Solver; break;
-      case 15:
-         cvode = new CVODESolver(MPI_COMM_WORLD, CV_ADAMS, CV_FUNCTIONAL);
-         cvode->SetSStolerances(reltol, abstol);
-         cvode->SetMaxStep(dt);
-         ode_solver = cvode; break;
-      case 16:
-         arkode = new ARKODESolver(MPI_COMM_WORLD, ARKODESolver::EXPLICIT);
-         arkode->SetSStolerances(reltol, abstol);
-         arkode->SetMaxStep(dt);
-         ode_solver = arkode; break;
-      // Implicit A-stable methods (not L-stable)
-      case 22: ode_solver = new ImplicitMidpointSolver; break;
-      case 23: ode_solver = new SDIRK23Solver; break;
-      case 24: ode_solver = new SDIRK34Solver; break;
-      default:
-         if (myid == 0)
-         {
-            cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
-         }
-         delete mesh;
-         MPI_Finalize();
-         return 3;
-   }
-
+   // 4. Nonlinear solver
    map<string,HyperelasticOperator::NonlinearSolverType> nls_map;
    nls_map["newton"] = HyperelasticOperator::NEWTON;
    nls_map["kinsol"] = HyperelasticOperator::KINSOL;
@@ -391,7 +333,6 @@ int main(int argc, char *argv[])
       {
          cout << "Unknown type of nonlinear solver: " << nls << endl;
       }
-      delete ode_solver;
       delete mesh;
       MPI_Finalize();
       return 4;
@@ -495,11 +436,82 @@ int main(int argc, char *argv[])
       cout << "initial   total energy (TE) = " << (ee0 + ke0) << endl;
    }
 
+   // 10. Define the ODE solver used for time integration. Several implicit
+   //     singly diagonal implicit Runge-Kutta (SDIRK) methods, as well as
+   //     explicit Runge-Kutta methods are available.
    double t = 0.0;
    oper.SetTime(t);
-   ode_solver->Init(oper);
 
-   // 10. Perform time-integration
+   ODESolver *ode_solver = NULL;
+   CVODESolver *cvode = NULL;
+   ARKStepSolver *arkode = NULL;
+   switch (ode_solver_type)
+   {
+      // Implicit L-stable methods
+      case 1:  ode_solver = new BackwardEulerSolver; break;
+      case 2:  ode_solver = new SDIRK23Solver(2); break;
+      case 3:  ode_solver = new SDIRK33Solver; break;
+      // Implicit A-stable methods (not L-stable)
+      case 4:  ode_solver = new ImplicitMidpointSolver; break;
+      case 5:  ode_solver = new SDIRK23Solver; break;
+      case 6:  ode_solver = new SDIRK34Solver; break;
+      // Explicit methods
+      case 7:  ode_solver = new ForwardEulerSolver; break;
+      case 8:  ode_solver = new RK2Solver(0.5); break; // midpoint method
+      case 9:  ode_solver = new RK3SSPSolver; break;
+      case 10: ode_solver = new RK4Solver; break;
+      // CVODE BDF
+      case 11:
+      case 12:
+         cvode = new CVODESolver(MPI_COMM_WORLD, CV_BDF);
+         cvode->Init(oper);
+         cvode->SetSStolerances(reltol, abstol);
+         CVodeSetEpsLin(cvode->GetMem(), cvode_eps_lin);
+         cvode->SetMaxStep(dt);
+         if (ode_solver_type == 11)
+         {
+            cvode->UseSundialsLinearSolver();
+         }
+         ode_solver = cvode; break;
+      // CVODE Adams
+      case 13:
+      case 14:
+         cvode = new CVODESolver(MPI_COMM_WORLD, CV_ADAMS);
+         cvode->Init(oper);
+         cvode->SetSStolerances(reltol, abstol);
+         CVodeSetEpsLin(cvode->GetMem(), cvode_eps_lin);
+         cvode->SetMaxStep(dt);
+         if (ode_solver_type == 13)
+         {
+            cvode->UseSundialsLinearSolver();
+         }
+         ode_solver = cvode; break;
+      // ARKStep Implicit methods
+      case 15:
+      case 16:
+         arkode = new ARKStepSolver(MPI_COMM_WORLD, ARKStepSolver::IMPLICIT);
+         arkode->Init(oper);
+         arkode->SetSStolerances(reltol, abstol);
+         ARKStepSetNonlinConvCoef(arkode->GetMem(), arkode_eps_nonlin);
+         arkode->SetMaxStep(dt);
+         if (ode_solver_type == 15)
+         {
+            arkode->UseSundialsLinearSolver();
+         }
+         ode_solver = arkode; break;
+      // ARKStep Explicit methods
+      case 17:
+         arkode = new ARKStepSolver(MPI_COMM_WORLD, ARKStepSolver::EXPLICIT);
+         arkode->Init(oper);
+         arkode->SetSStolerances(reltol, abstol);
+         arkode->SetMaxStep(dt);
+         ode_solver = arkode; break;
+   }
+
+   // Initialize MFEM integrators, SUNDIALS integrators are initialized above
+   if (ode_solver_type < 11) { ode_solver->Init(oper); }
+
+   // 11. Perform time-integration
    //     (looping over the time iterations, ti, with a time-step dt).
    bool last_step = false;
    for (int ti = 1; !last_step; ti++)
@@ -538,7 +550,7 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 11. Save the displaced mesh, the velocity and elastic energy.
+   // 12. Save the displaced mesh, the velocity and elastic energy.
    {
       v_gf.SetFromTrueVector(); x_gf.SetFromTrueVector();
       GridFunction *nodes = &x_gf;
@@ -563,9 +575,8 @@ int main(int argc, char *argv[])
       w_gf.Save(ee_ofs);
    }
 
-   // 12. Free the used memory.
+   // 13. Free the used memory.
    delete ode_solver;
-   delete sjsolver;
    delete pmesh;
 
    MPI_Finalize();
@@ -653,92 +664,14 @@ ReducedSystemOperator::~ReducedSystemOperator()
 }
 
 
-int SundialsJacSolver::InitSystem(void *sundials_mem)
-{
-   TimeDependentOperator *td_oper = GetTimeDependentOperator(sundials_mem);
-   HyperelasticOperator *he_oper;
-
-   // During development, we use dynamic_cast<> to ensure the setup is correct:
-   he_oper = dynamic_cast<HyperelasticOperator*>(td_oper);
-   MFEM_VERIFY(he_oper, "operator is not HyperelasticOperator");
-
-   // When the implementation is finalized, we can switch to static_cast<>:
-   // he_oper = static_cast<HyperelasticOperator*>(td_oper);
-
-   he_oper->InitSundialsJacSolver(*this);
-   return 0;
-}
-
-int SundialsJacSolver::SetupSystem(void *sundials_mem, int conv_fail,
-                                   const Vector &y_pred, const Vector &f_pred,
-                                   int &jac_cur, Vector &v_temp1,
-                                   Vector &v_temp2, Vector &v_temp3)
-{
-   int sc = y_pred.Size() / 2;
-   const Vector x(y_pred.GetData() + sc, sc);
-   double dt = GetTimeStep(sundials_mem);
-
-   // J = M + dt*(S + dt*grad(H))
-   delete Jacobian;
-   SparseMatrix *localJ = Add(1.0, M->SpMat(), dt, S->SpMat());
-   local_grad_H = &H->GetLocalGradient(x);
-   localJ->Add(dt*dt, *local_grad_H);
-   Jacobian = M->ParallelAssemble(localJ);
-   delete localJ;
-   HypreParMatrix *Je = Jacobian->EliminateRowsCols(*ess_tdof_list);
-   delete Je;
-
-   J_solver->SetOperator(*Jacobian);
-
-   jac_cur = 1;
-   return 0;
-}
-
-int SundialsJacSolver::SolveSystem(void *sundials_mem, Vector &b,
-                                   const Vector &weight, const Vector &y_cur,
-                                   const Vector &f_cur)
-{
-   int sc = b.Size() / 2;
-   ParFiniteElementSpace *fes = H->ParFESpace();
-   // Vector x(y_cur.GetData() + sc, sc);
-   Vector b_v(b.GetData() +  0, sc);
-   Vector b_x(b.GetData() + sc, sc);
-   Vector rhs(sc);
-   double dt = GetTimeStep(sundials_mem);
-
-   // We can assume that b_v and b_x have zeros at essential tdofs.
-
-   // rhs = M b_v - dt*grad(H) b_x
-   ParGridFunction lb_x(fes), lrhs(fes);
-   lb_x.Distribute(b_x);
-   local_grad_H->Mult(lb_x, lrhs);
-   lrhs.ParallelAssemble(rhs);
-   rhs *= -dt;
-   M->TrueAddMult(b_v, rhs);
-   rhs.SetSubVector(*ess_tdof_list, 0.0);
-
-   J_solver->iterative_mode = false;
-   J_solver->Mult(rhs, b_v);
-
-   b_x.Add(dt, b_v);
-
-   return 0;
-}
-
-int SundialsJacSolver::FreeSystem(void *sundials_mem)
-{
-   delete Jacobian;
-   return 0;
-}
-
-
 HyperelasticOperator::HyperelasticOperator(ParFiniteElementSpace &f,
                                            Array<int> &ess_bdr, double visc,
                                            double mu, double K,
                                            NonlinearSolverType nls_type)
    : TimeDependentOperator(2*f.TrueVSize(), 0.0), fespace(f),
      M(&fespace), S(&fespace), H(&fespace),
-     viscosity(visc), M_solver(f.GetComm()), z(height/2)
+     viscosity(visc), M_solver(f.GetComm()), z(height/2),
+     local_grad_H(NULL), Jacobian(NULL)
 {
    const double rel_tol = 1e-8;
    const int skip_zero_entries = 0;
@@ -788,23 +721,24 @@ HyperelasticOperator::HyperelasticOperator(ParFiniteElementSpace &f,
 
    if (nls_type == KINSOL)
    {
-      KinSolver *kinsolver = new KinSolver(f.GetComm(), KIN_NONE, true);
-      kinsolver->SetMaxSetupCalls(4);
+      KINSolver *kinsolver = new KINSolver(f.GetComm(), KIN_NONE, true);
       newton_solver = kinsolver;
+      newton_solver->SetOperator(*reduced_oper);
       newton_solver->SetMaxIter(200);
       newton_solver->SetRelTol(rel_tol);
       newton_solver->SetPrintLevel(0);
+      kinsolver->SetMaxSetupCalls(4);
    }
    else
    {
       newton_solver = new NewtonSolver(f.GetComm());
+      newton_solver->SetOperator(*reduced_oper);
       newton_solver->SetMaxIter(10);
       newton_solver->SetRelTol(rel_tol);
       newton_solver->SetPrintLevel(-1);
    }
    newton_solver->SetSolver(*J_solver);
    newton_solver->iterative_mode = false;
-   newton_solver->SetOperator(*reduced_oper);
 }
 
 void HyperelasticOperator::Mult(const Vector &vx, Vector &dvx_dt) const
@@ -858,9 +792,64 @@ void HyperelasticOperator::ImplicitSolve(const double dt,
    add(v, dt, dv_dt, dx_dt);
 }
 
-void HyperelasticOperator::InitSundialsJacSolver(SundialsJacSolver &sjsolv)
+int HyperelasticOperator::SUNImplicitSetup(const Vector &y,
+                                           const Vector &fy, int jok, int *jcur,
+                                           double gamma)
 {
-   sjsolv.SetOperators(M, S, H, *J_solver, ess_tdof_list);
+   int sc = y.Size() / 2;
+   const Vector x(y.GetData() + sc, sc);
+
+   // J = M + dt*(S + dt*grad(H))
+   if (Jacobian) { delete Jacobian; }
+   SparseMatrix *localJ = Add(1.0, M.SpMat(), gamma, S.SpMat());
+   local_grad_H = &H.GetLocalGradient(x);
+   localJ->Add(gamma*gamma, *local_grad_H);
+   Jacobian = M.ParallelAssemble(localJ);
+   delete localJ;
+   HypreParMatrix *Je = Jacobian->EliminateRowsCols(ess_tdof_list);
+   delete Je;
+
+   // Set Jacobian solve operator
+   J_solver->SetOperator(*Jacobian);
+
+   // Indicate that the Jacobian was updated
+   *jcur = 1;
+
+   // Save gamma for use in solve
+   saved_gamma = gamma;
+
+   // Return success
+   return 0;
+}
+
+int HyperelasticOperator::SUNImplicitSolve(const Vector &b, Vector &x,
+                                           double tol)
+{
+   int sc = b.Size() / 2;
+   ParFiniteElementSpace *fes = H.ParFESpace();
+   Vector b_v(b.GetData() +  0, sc);
+   Vector b_x(b.GetData() + sc, sc);
+   Vector x_v(x.GetData() +  0, sc);
+   Vector x_x(x.GetData() + sc, sc);
+   Vector rhs(sc);
+
+   // We can assume that b_v and b_x have zeros at essential tdofs.
+
+   // rhs = M b_v - dt*grad(H) b_x
+   ParGridFunction lb_x(fes), lrhs(fes);
+   lb_x.Distribute(b_x);
+   local_grad_H->Mult(lb_x, lrhs);
+   lrhs.ParallelAssemble(rhs);
+   rhs *= -saved_gamma;
+   M.TrueAddMult(b_v, rhs);
+   rhs.SetSubVector(ess_tdof_list, 0.0);
+
+   J_solver->iterative_mode = false;
+   J_solver->Mult(rhs, x_v);
+
+   add(b_x, saved_gamma, x_v, x_x);
+
+   return 0;
 }
 
 double HyperelasticOperator::ElasticEnergy(const ParGridFunction &x) const
@@ -886,6 +875,7 @@ void HyperelasticOperator::GetElasticEnergyDensity(
 
 HyperelasticOperator::~HyperelasticOperator()
 {
+   delete Jacobian;
    delete newton_solver;
    delete J_solver;
    delete J_prec;

examples/sundials/ex16.cpp

@@ -7,9 +7,9 @@
 //               ex16 -m ../../data/inline-tri.mesh
 //               ex16 -m ../../data/disc-nurbs.mesh -tf 2
 //               ex16 -s 12 -a 0.0 -k 1.0
-//               ex16 -s 1 -a 1.0 -k 0.0 -dt 1e-4 -tf 5e-2 -vs 25
-//               ex16 -s 2 -a 0.5 -k 0.5 -o 4 -dt 1e-4 -tf 2e-2 -vs 25
-//               ex16 -s 3 -dt 1.0e-4 -tf 4.0e-2 -vs 40
+//               ex16 -s 8 -a 1.0 -k 0.0 -dt 1e-4 -tf 5e-2 -vs 25
+//               ex16 -s 9 -a 0.5 -k 0.5 -o 4 -dt 1e-4 -tf 2e-2 -vs 25
+//               ex16 -s 10 -dt 1.0e-4 -tf 4.0e-2 -vs 40
 //               ex16 -m ../../data/fichera-q2.mesh
 //               ex16 -m ../../data/escher.mesh
 //               ex16 -m ../../data/beam-tet.mesh -tf 10 -dt 0.1
@@ -58,7 +58,6 @@ protected:
 
    SparseMatrix Mmat, Kmat;
    SparseMatrix *T; // T = M + dt K
-   double current_dt;
 
    CGSolver M_solver; // Krylov solver for inverting the mass matrix M
    DSmoother M_prec;  // Preconditioner for the mass matrix M
@@ -75,13 +74,30 @@ public:
                       const Vector &u);
 
    virtual void Mult(const Vector &u, Vector &du_dt) const;
+
    /** Solve the Backward-Euler equation: k = f(u + dt*k, t), for the unknown k.
        This is the only requirement for high-order SDIRK implicit integration.*/
    virtual void ImplicitSolve(const double dt, const Vector &u, Vector &k);
 
-   /** Solve the system (M + dt K) y = M b. The result y replaces the input b.
-       This method is used by the implicit SUNDIALS solvers. */
-   void SundialsSolve(const double dt, Vector &b);
+   /// Custom Jacobian system solver for the SUNDIALS time integrators.
+   /** For the ODE system represented by ConductionOperator
+
+           M du/dt = -K(u),
+
+       this class facilitates the solution of linear systems of the form
+
+           (M + γK) y = M b,
+
+       for given b, u (not used), and γ = GetTimeStep(). */
+
+   /** Setup the system (M + dt K) x = M b. This method is used by the implicit
+       SUNDIALS solvers. */
+   virtual int SUNImplicitSetup(const Vector &x, const Vector &fx,
+                                int jok, int *jcur, double gamma);
+
+   /** Solve the system (M + dt K) x = M b. This method is used by the implicit
+       SUNDIALS solvers. */
+   virtual int SUNImplicitSolve(const Vector &b, Vector &x, double tol);
 
    /// Update the diffusion BilinearForm K using the given true-dof vector `u`.
    void SetParameters(const Vector &u);
@@ -89,33 +105,6 @@ public:
    virtual ~ConductionOperator();
 };
 
-/// Custom Jacobian system solver for the SUNDIALS time integrators.
-/** For the ODE system represented by ConductionOperator
-
-        M du/dt = -K(u),
-
-    this class facilitates the solution of linear systems of the form
-
-        (M + γK) y = M b,
-
-    for given b, u (not used), and γ = GetTimeStep(). */
-class SundialsJacSolver : public SundialsODELinearSolver
-{
-private:
-   ConductionOperator *oper;
-
-public:
-   SundialsJacSolver() : oper(NULL) { }
-
-   int InitSystem(void *sundials_mem);
-   int SetupSystem(void *sundials_mem, int conv_fail,
-                   const Vector &y_pred, const Vector &f_pred, int &jac_cur,
-                   Vector &v_temp1, Vector &v_temp2, Vector &v_temp3);
-   int SolveSystem(void *sundials_mem, Vector &b, const Vector &weight,
-                   const Vector &y_cur, const Vector &f_cur);
-   int FreeSystem(void *sundials_mem);
-};
-
 double InitialTemperature(const Vector &x);
 
 int main(int argc, char *argv[])
@@ -124,7 +113,7 @@ int main(int argc, char *argv[])
    const char *mesh_file = "../../data/star.mesh";
    int ref_levels = 2;
    int order = 2;
-   int ode_solver_type = 11; // 11 = CVODE implicit
+   int ode_solver_type = 9; // CVODE implicit BDF
    double t_final = 0.5;
    double dt = 1.0e-2;
    double alpha = 1.0e-2;
@@ -147,12 +136,19 @@ int main(int argc, char *argv[])
    args.AddOption(&order, "-o", "--order",
                   "Order (degree) of the finite elements.");
    args.AddOption(&ode_solver_type, "-s", "--ode-solver",
-                  "ODE solver:\n"
-                  "\t 1/11 - CVODE (explicit/implicit),\n"
-                  "\t 2/12 - ARKODE (default explicit/implicit),\n"
-                  "\t 3 - ARKODE (Fehlberg-6-4-5)\n"
-                  "\t 4 - Forward Euler, 5 - RK2, 6 - RK3 SSP, 7 - RK4,\n"
-                  "\t 8 - Backward Euler, 9 - SDIRK23, 10 - SDIRK33.");
+                  "ODE solver:\n\t"
+                  "1  - Forward Euler,\n\t"
+                  "2  - RK2,\n\t"
+                  "3  - RK3 SSP,\n\t"
+                  "4  - RK4,\n\t"
+                  "5  - Backward Euler,\n\t"
+                  "6  - SDIRK 2,\n\t"
+                  "7  - SDIRK 3,\n\t"
+                  "8  - CVODE (implicit Adams),\n\t"
+                  "9  - CVODE (implicit BDF),\n\t"
+                  "10 - ARKODE (default explicit),\n\t"
+                  "11 - ARKODE (explicit Fehlberg-6-4-5),\n\t"
+                  "12 - ARKODE (default impicit).");
    args.AddOption(&t_final, "-tf", "--t-final",
                   "Final time; start time is 0.");
    args.AddOption(&dt, "-dt", "--time-step",
@@ -175,6 +171,11 @@ int main(int argc, char *argv[])
       args.PrintUsage(cout);
       return 1;
    }
+   if (ode_solver_type < 1 || ode_solver_type > 12)
+   {
+      cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
+      return 3;
+   }
    args.PrintOptions(cout);
 
    // 2. Read the mesh from the given mesh file. We can handle triangular,
@@ -182,61 +183,7 @@ int main(int argc, char *argv[])
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
 
-   // 3. Define the ODE solver used for time integration. Several
-   // SUNDIALS solvers are available, as well as included both
-   // explicit and implicit MFEM ODE solvers.
-   ODESolver *ode_solver = NULL;
-   CVODESolver *cvode = NULL;
-   ARKODESolver *arkode = NULL;
-   SundialsJacSolver sun_solver; // Used by the implicit SUNDIALS ode solvers.
-   switch (ode_solver_type)
-   {
-      // SUNDIALS solvers
-      case 1:
-         cvode = new CVODESolver(CV_ADAMS, CV_FUNCTIONAL);
-         cvode->SetSStolerances(reltol, abstol);
-         cvode->SetMaxStep(dt);
-         ode_solver = cvode; break;
-      case 11:
-         cvode = new CVODESolver(CV_BDF, CV_NEWTON);
-         cvode->SetLinearSolver(sun_solver);
-         cvode->SetSStolerances(reltol, abstol);
-         cvode->SetMaxStep(dt);
-         ode_solver = cvode; break;
-      case 2:
-      case 3:
-         arkode = new ARKODESolver(ARKODESolver::EXPLICIT);
-         arkode->SetSStolerances(reltol, abstol);
-         arkode->SetMaxStep(dt);
-         if (ode_solver_type == 3) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
-         ode_solver = arkode; break;
-      case 12:
-         arkode = new ARKODESolver(ARKODESolver::IMPLICIT);
-         arkode->SetLinearSolver(sun_solver);
-         arkode->SetSStolerances(reltol, abstol);
-         arkode->SetMaxStep(dt);
-         ode_solver = arkode; break;
-      // Other MFEM explicit methods
-      case 4: ode_solver = new ForwardEulerSolver; break;
-      case 5: ode_solver = new RK2Solver(0.5); break; // midpoint method
-      case 6: ode_solver = new RK3SSPSolver; break;
-      case 7: ode_solver = new RK4Solver; break;
-      // MFEM implicit L-stable methods
-      case 8:  ode_solver = new BackwardEulerSolver; break;
-      case 9:  ode_solver = new SDIRK23Solver(2); break;
-      case 10: ode_solver = new SDIRK33Solver; break;
-      default:
-         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
-         delete mesh;
-         return 3;
-   }
-
-   // Since we want to update the diffusion coefficient after every time step,
-   // we need to use the "one-step" mode of the SUNDIALS solvers.
-   if (cvode) { cvode->SetStepMode(CV_ONE_STEP); }
-   if (arkode) { arkode->SetStepMode(ARK_ONE_STEP); }
-
-   // 4. Refine the mesh to increase the resolution. In this example we do
+   // 3. Refine the mesh to increase the resolution. In this example we do
    //    'ref_levels' of uniform refinement, where 'ref_levels' is a
    //    command-line parameter.
    for (int lev = 0; lev < ref_levels; lev++)
@@ -244,7 +191,7 @@ int main(int argc, char *argv[])
       mesh->UniformRefinement();
    }
 
-   // 5. Define the vector finite element space representing the current and the
+   // 4. Define the vector finite element space representing the current and the
    //    initial temperature, u_ref.
    H1_FECollection fe_coll(order, dim);
    FiniteElementSpace fespace(mesh, &fe_coll);
@@ -254,14 +201,14 @@ int main(int argc, char *argv[])
 
    GridFunction u_gf(&fespace);
 
-   // 6. Set the initial conditions for u. All boundaries are considered
+   // 5. Set the initial conditions for u. All boundaries are considered
    //    natural.
    FunctionCoefficient u_0(InitialTemperature);
    u_gf.ProjectCoefficient(u_0);
    Vector u;
    u_gf.GetTrueDofs(u);
 
-   // 7. Initialize the conduction operator and the visualization.
+   // 6. Initialize the conduction operator and the visualization.
    ConductionOperator oper(fespace, alpha, kappa, u);
 
    u_gf.SetFromTrueDofs(u);
@@ -307,13 +254,65 @@ int main(int argc, char *argv[])
       }
    }
 
+   // 7. Define the ODE solver used for time integration.
+   double t = 0.0;
+   ODESolver *ode_solver = NULL;
+   CVODESolver *cvode = NULL;
+   ARKStepSolver *arkode = NULL;
+   switch (ode_solver_type)
+   {
+      // MFEM explicit methods
+      case 1: ode_solver = new ForwardEulerSolver; break;
+      case 2: ode_solver = new RK2Solver(0.5); break; // midpoint method
+      case 3: ode_solver = new RK3SSPSolver; break;
+      case 4: ode_solver = new RK4Solver; break;
+      // MFEM implicit L-stable methods
+      case 5: ode_solver = new BackwardEulerSolver; break;
+      case 6: ode_solver = new SDIRK23Solver(2); break;
+      case 7: ode_solver = new SDIRK33Solver; break;
+      // CVODE
+      case 8:
+         cvode = new CVODESolver(CV_ADAMS);
+         cvode->Init(oper);
+         cvode->SetSStolerances(reltol, abstol);
+         cvode->SetMaxStep(dt);
+         ode_solver = cvode; break;
+      case 9:
+         cvode = new CVODESolver(CV_BDF);
+         cvode->Init(oper);
+         cvode->SetSStolerances(reltol, abstol);
+         cvode->SetMaxStep(dt);
+         ode_solver = cvode; break;
+      // ARKODE
+      case 10:
+      case 11:
+         arkode = new ARKStepSolver(ARKStepSolver::EXPLICIT);
+         arkode->Init(oper);
+         arkode->SetSStolerances(reltol, abstol);
+         arkode->SetMaxStep(dt);
+         if (ode_solver_type == 11) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
+         ode_solver = arkode; break;
+      case 12:
+         arkode = new ARKStepSolver(ARKStepSolver::IMPLICIT);
+         arkode->Init(oper);
+         arkode->SetSStolerances(reltol, abstol);
+         arkode->SetMaxStep(dt);
+         ode_solver = arkode; break;
+   }
+
+   // Initialize MFEM integrators, SUNDIALS integrators are initialized above
+   if (ode_solver_type < 8) { ode_solver->Init(oper); }
+
+   // Since we want to update the diffusion coefficient after every time step,
+   // we need to use the "one-step" mode of the SUNDIALS solvers.
+   if (cvode) { cvode->SetStepMode(CV_ONE_STEP); }
+   if (arkode) { arkode->SetStepMode(ARK_ONE_STEP); }
+
    // 8. Perform time-integration (looping over the time iterations, ti, with a
    //    time-step dt).
    cout << "Integrating the ODE ..." << endl;
    tic_toc.Clear();
    tic_toc.Start();
-   ode_solver->Init(oper);
-   double t = 0.0;
 
    bool last_step = false;
    for (int ti = 1; !last_step; ti++)
@@ -371,7 +370,7 @@ int main(int argc, char *argv[])
 ConductionOperator::ConductionOperator(FiniteElementSpace &f, double al,
                                        double kap, const Vector &u)
    : TimeDependentOperator(f.GetTrueVSize(), 0.0), fespace(f), M(NULL), K(NULL),
-     T(NULL), current_dt(0.0), z(height)
+     T(NULL), z(height)
 {
    const double rel_tol = 1e-8;
 
@@ -417,32 +416,14 @@ void ConductionOperator::ImplicitSolve(const double dt,
    // Solve the equation:
    //    du_dt = M^{-1}*[-K(u + dt*du_dt)]
    // for du_dt
-   if (!T)
-   {
-      T = Add(1.0, Mmat, dt, Kmat);
-      current_dt = dt;
-      T_solver.SetOperator(*T);
-   }
-   MFEM_VERIFY(dt == current_dt, ""); // SDIRK methods use the same dt
+   if (T) { delete T; }
+   T = Add(1.0, Mmat, dt, Kmat);
+   T_solver.SetOperator(*T);
    Kmat.Mult(u, z);
    z.Neg();
    T_solver.Mult(z, du_dt);
 }
 
-void ConductionOperator::SundialsSolve(const double dt, Vector &b)
-{
-   // Solve the system (M + dt K) y = M b. The result y replaces the input b.
-   if (!T || dt != current_dt)
-   {
-      delete T;
-      T = Add(1.0, Mmat, dt, Kmat);
-      current_dt = dt;
-      T_solver.SetOperator(*T);
-   }
-   Mmat.Mult(b, z);
-   T_solver.Mult(z, b);
-}
-
 void ConductionOperator::SetParameters(const Vector &u)
 {
    GridFunction u_alpha_gf(&fespace);
@@ -460,8 +441,26 @@ void ConductionOperator::SetParameters(const Vector &u)
    K->AddDomainIntegrator(new DiffusionIntegrator(u_coeff));
    K->Assemble();
    K->FormSystemMatrix(ess_tdof_list, Kmat);
-   delete T;
-   T = NULL; // re-compute T on the next ImplicitSolve or SundialsSolve
+}
+
+int ConductionOperator::SUNImplicitSetup(const Vector &x,
+                                         const Vector &fx, int jok, int *jcur,
+                                         double gamma)
+{
+   // Setup the ODE Jacobian T = M + gamma K.
+   if (T) { delete T; }
+   T = Add(1.0, Mmat, gamma, Kmat);
+   T_solver.SetOperator(*T);
+   *jcur = 1;
+   return (0);
+}
+
+int ConductionOperator::SUNImplicitSolve(const Vector &b, Vector &x, double tol)
+{
+   // Solve the system A x = z => (M - gamma K) x = M b.
+   Mmat.Mult(b, z);
+   T_solver.Mult(z, x);
+   return (0);
 }
 
 ConductionOperator::~ConductionOperator()
@@ -471,46 +470,6 @@ ConductionOperator::~ConductionOperator()
    delete K;
 }
 
-
-int SundialsJacSolver::InitSystem(void *sundials_mem)
-{
-   TimeDependentOperator *td_oper = GetTimeDependentOperator(sundials_mem);
-
-   // During development, we use dynamic_cast<> to ensure the setup is correct:
-   oper = dynamic_cast<ConductionOperator*>(td_oper);
-   MFEM_VERIFY(oper, "operator is not ConductionOperator");
-
-   // When the implementation is finalized, we can switch to static_cast<>:
-   // oper = static_cast<ConductionOperator*>(td_oper);
-
-   return 0;
-}
-
-int SundialsJacSolver::SetupSystem(void *sundials_mem, int conv_fail,
-                                   const Vector &y_pred, const Vector &f_pred,
-                                   int &jac_cur, Vector &v_temp1,
-                                   Vector &v_temp2, Vector &v_temp3)
-{
-   jac_cur = 1;
-
-   return 0;
-}
-
-int SundialsJacSolver::SolveSystem(void *sundials_mem, Vector &b,
-                                   const Vector &weight, const Vector &y_cur,
-                                   const Vector &f_cur)
-{
-   oper->SundialsSolve(GetTimeStep(sundials_mem), b);
-
-   return 0;
-}
-
-int SundialsJacSolver::FreeSystem(void *sundials_mem)
-{
-   return 0;
-}
-
-
 double InitialTemperature(const Vector &x)
 {
    if (x.Norml2() < 0.5)

examples/sundials/ex16p.cpp

@@ -8,9 +8,9 @@
 //     mpirun -np 4 ex16p -m ../../data/inline-tri.mesh
 //     mpirun -np 4 ex16p -m ../../data/disc-nurbs.mesh -tf 2
 //     mpirun -np 4 ex16p -s 12 -a 0.0 -k 1.0
-//     mpirun -np 4 ex16p -s 1 -a 1.0 -k 0.0 -dt 4e-6 -tf 2e-2 -vs 50
-//     mpirun -np 8 ex16p -s 2 -a 0.5 -k 0.5 -o 4 -dt 8e-6 -tf 2e-2 -vs 50
-//     mpirun -np 4 ex16p -s 3 -dt 2.0e-4 -tf 4.0e-2
+//     mpirun -np 4 ex16p -s 8 -a 1.0 -k 0.0 -dt 4e-6 -tf 2e-2 -vs 50
+//     mpirun -np 8 ex16p -s 9 -a 0.5 -k 0.5 -o 4 -dt 8e-6 -tf 2e-2 -vs 50
+//     mpirun -np 4 ex16p -s 10 -dt 2.0e-4 -tf 4.0e-2
 //     mpirun -np 16 ex16p -m ../../data/fichera-q2.mesh
 //     mpirun -np 16 ex16p -m ../../data/escher-p2.mesh
 //     mpirun -np 8 ex16p -m ../../data/beam-tet.mesh -tf 10 -dt 0.1
@@ -77,13 +77,19 @@ public:
                       const Vector &u);
 
    virtual void Mult(const Vector &u, Vector &du_dt) const;
+
    /** Solve the Backward-Euler equation: k = f(u + dt*k, t), for the unknown k.
        This is the only requirement for high-order SDIRK implicit integration.*/
    virtual void ImplicitSolve(const double dt, const Vector &u, Vector &k);
 
-   /** Solve the system (M + dt K) y = M b. The result y replaces the input b.
-       This method is used by the implicit SUNDIALS solvers. */
-   void SundialsSolve(const double dt, Vector &b);
+   /** Setup the system (M + dt K) x = M b. This method is used by the implicit
+       SUNDIALS solvers. */
+   virtual int SUNImplicitSetup(const Vector &x, const Vector &fx,
+                                int jok, int *jcur, double gamma);
+
+   /** Solve the system (M + dt K) x = M b. This method is used by the implicit
+       SUNDIALS solvers. */
+   virtual int SUNImplicitSolve(const Vector &b, Vector &x, double tol);
 
    /// Update the diffusion BilinearForm K using the given true-dof vector `u`.
    void SetParameters(const Vector &u);
@@ -91,33 +97,6 @@ public:
    virtual ~ConductionOperator();
 };
 
-/// Custom Jacobian system solver for the SUNDIALS time integrators.
-/** For the ODE system represented by ConductionOperator
-
-        M du/dt = -K(u),
-
-    this class facilitates the solution of linear systems of the form
-
-        (M + γK) y = M b,
-
-    for given b, u (not used), and γ = GetTimeStep(). */
-class SundialsJacSolver : public SundialsODELinearSolver
-{
-private:
-   ConductionOperator *oper;
-
-public:
-   SundialsJacSolver() : oper(NULL) { }
-
-   int InitSystem(void *sundials_mem);
-   int SetupSystem(void *sundials_mem, int conv_fail,
-                   const Vector &y_pred, const Vector &f_pred, int &jac_cur,
-                   Vector &v_temp1, Vector &v_temp2, Vector &v_temp3);
-   int SolveSystem(void *sundials_mem, Vector &b, const Vector &weight,
-                   const Vector &y_cur, const Vector &f_cur);
-   int FreeSystem(void *sundials_mem);
-};
-
 double InitialTemperature(const Vector &x);
 
 int main(int argc, char *argv[])
@@ -133,7 +112,7 @@ int main(int argc, char *argv[])
    int ser_ref_levels = 2;
    int par_ref_levels = 1;
    int order = 2;
-   int ode_solver_type = 11; // 11 = CVODE implicit
+   int ode_solver_type = 9; // CVODE implicit BDF
    double t_final = 0.5;
    double dt = 1.0e-2;
    double alpha = 1.0e-2;
@@ -158,12 +137,19 @@ int main(int argc, char *argv[])
    args.AddOption(&order, "-o", "--order",
                   "Order (degree) of the finite elements.");
    args.AddOption(&ode_solver_type, "-s", "--ode-solver",
-                  "ODE solver:\n"
-                  "\t 1/11 - CVODE (explicit/implicit),\n"
-                  "\t 2/12 - ARKODE (default explicit/implicit),\n"
-                  "\t 3 - ARKODE (Fehlberg-6-4-5)\n"
-                  "\t 4 - Forward Euler, 5 - RK2, 6 - RK3 SSP, 7 - RK4,\n"
-                  "\t 8 - Backward Euler, 9 - SDIRK23, 10 - SDIRK33.");
+                  "ODE solver:\n\t"
+                  "1  - Forward Euler,\n\t"
+                  "2  - RK2,\n\t"
+                  "3  - RK3 SSP,\n\t"
+                  "4  - RK4,\n\t"
+                  "5  - Backward Euler,\n\t"
+                  "6  - SDIRK 2,\n\t"
+                  "7  - SDIRK 3,\n\t"
+                  "8  - CVODE (implicit Adams),\n\t"
+                  "9  - CVODE (implicit BDF),\n\t"
+                  "10 - ARKODE (default explicit),\n\t"
+                  "11 - ARKODE (explicit Fehlberg-6-4-5),\n\t"
+                  "12 - ARKODE (default impicit).");
    args.AddOption(&t_final, "-tf", "--t-final",
                   "Final time; start time is 0.");
    args.AddOption(&dt, "-dt", "--time-step",
@@ -193,67 +179,24 @@ int main(int argc, char *argv[])
       args.PrintOptions(cout);
    }
 
+   // check for vaild ODE solver option
+   if (ode_solver_type < 1 || ode_solver_type > 12)
+   {
+      if (myid == 0)
+      {
+         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
+      }
+      MPI_Finalize();
+      return 1;
+   }
+
    // 3. Read the serial mesh from the given mesh file on all processors. We can
    //    handle triangular, quadrilateral, tetrahedral and hexahedral meshes
    //    with the same code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
 
-   // 4. Define the ODE solver used for time integration. Several
-   // SUNDIALS solvers are available, as well as included both
-   // explicit and implicit MFEM ODE solvers.
-   ODESolver *ode_solver = NULL;
-   CVODESolver *cvode = NULL;
-   ARKODESolver *arkode = NULL;
-   SundialsJacSolver sun_solver; // Used by the implicit SUNDIALS ode solvers.
-   switch (ode_solver_type)
-   {
-      // SUNDIALS solvers
-      case 1:
-         cvode = new CVODESolver(MPI_COMM_WORLD, CV_ADAMS, CV_FUNCTIONAL);
-         cvode->SetSStolerances(reltol, abstol);
-         cvode->SetMaxStep(dt);
-         ode_solver = cvode; break;
-      case 11:
-         cvode = new CVODESolver(MPI_COMM_WORLD, CV_BDF, CV_NEWTON);
-         cvode->SetLinearSolver(sun_solver);
-         cvode->SetSStolerances(reltol, abstol);
-         cvode->SetMaxStep(dt);
-         ode_solver = cvode; break;
-      case 2:
-      case 3:
-         arkode = new ARKODESolver(MPI_COMM_WORLD, ARKODESolver::EXPLICIT);
-         arkode->SetSStolerances(reltol, abstol);
-         arkode->SetMaxStep(dt);
-         if (ode_solver_type == 3) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
-         ode_solver = arkode; break;
-      case 12:
-         arkode = new ARKODESolver(MPI_COMM_WORLD, ARKODESolver::IMPLICIT);
-         arkode->SetLinearSolver(sun_solver);
-         arkode->SetSStolerances(reltol, abstol);
-         arkode->SetMaxStep(dt);
-         ode_solver = arkode; break;
-      // Other MFEM explicit methods
-      case 4: ode_solver = new ForwardEulerSolver; break;
-      case 5: ode_solver = new RK2Solver(0.5); break; // midpoint method
-      case 6: ode_solver = new RK3SSPSolver; break;
-      case 7: ode_solver = new RK4Solver; break;
-      // MFEM implicit L-stable methods
-      case 8:  ode_solver = new BackwardEulerSolver; break;
-      case 9:  ode_solver = new SDIRK23Solver(2); break;
-      case 10: ode_solver = new SDIRK33Solver; break;
-      default:
-         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
-         delete mesh;
-         return 3;
-   }
-
-   // Since we want to update the diffusion coefficient after every time step,
-   // we need to use the "one-step" mode of the SUNDIALS solvers.
-   if (cvode) { cvode->SetStepMode(CV_ONE_STEP); }
-   if (arkode) { arkode->SetStepMode(ARK_ONE_STEP); }
-
-   // 5. Refine the mesh in serial to increase the resolution. In this example
+   // 4. Refine the mesh in serial to increase the resolution. In this example
    //    we do 'ser_ref_levels' of uniform refinement, where 'ser_ref_levels' is
    //    a command-line parameter.
    for (int lev = 0; lev < ser_ref_levels; lev++)
@@ -261,7 +204,7 @@ int main(int argc, char *argv[])
       mesh->UniformRefinement();
    }
 
-   // 6. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
    //    this mesh further in parallel to increase the resolution. Once the
    //    parallel mesh is defined, the serial mesh can be deleted.
    ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
@@ -271,7 +214,7 @@ int main(int argc, char *argv[])
       pmesh->UniformRefinement();
    }
 
-   // 7. Define the vector finite element space representing the current and the
+   // 6. Define the vector finite element space representing the current and the
    //    initial temperature, u_ref.
    H1_FECollection fe_coll(order, dim);
    ParFiniteElementSpace fespace(pmesh, &fe_coll);
@@ -284,14 +227,14 @@ int main(int argc, char *argv[])
 
    ParGridFunction u_gf(&fespace);
 
-   // 8. Set the initial conditions for u. All boundaries are considered
+   // 7. Set the initial conditions for u. All boundaries are considered
    //    natural.
    FunctionCoefficient u_0(InitialTemperature);
    u_gf.ProjectCoefficient(u_0);
    Vector u;
    u_gf.GetTrueDofs(u);
 
-   // 9. Initialize the conduction operator and the VisIt visualization.
+   // 8. Initialize the conduction operator and the VisIt visualization.
    ConductionOperator oper(fespace, alpha, kappa, u);
 
    u_gf.SetFromTrueDofs(u);
@@ -350,6 +293,60 @@ int main(int argc, char *argv[])
       }
    }
 
+   // 9. Define the ODE solver used for time integration.
+   double t = 0.0;
+   ODESolver *ode_solver = NULL;
+   CVODESolver *cvode = NULL;
+   ARKStepSolver *arkode = NULL;
+   switch (ode_solver_type)
+   {
+      // MFEM explicit methods
+      case 1: ode_solver = new ForwardEulerSolver; break;
+      case 2: ode_solver = new RK2Solver(0.5); break; // midpoint method
+      case 3: ode_solver = new RK3SSPSolver; break;
+      case 4: ode_solver = new RK4Solver; break;
+      // MFEM implicit L-stable methods
+      case 5: ode_solver = new BackwardEulerSolver; break;
+      case 6: ode_solver = new SDIRK23Solver(2); break;
+      case 7: ode_solver = new SDIRK33Solver; break;
+      // CVODE
+      case 8:
+         cvode = new CVODESolver(MPI_COMM_WORLD, CV_ADAMS);
+         cvode->Init(oper);
+         cvode->SetSStolerances(reltol, abstol);
+         cvode->SetMaxStep(dt);
+         ode_solver = cvode; break;
+      case 9:
+         cvode = new CVODESolver(MPI_COMM_WORLD, CV_BDF);
+         cvode->Init(oper);
+         cvode->SetSStolerances(reltol, abstol);
+         cvode->SetMaxStep(dt);
+         ode_solver = cvode; break;
+      // ARKODE
+      case 10:
+      case 11:
+         arkode = new ARKStepSolver(MPI_COMM_WORLD, ARKStepSolver::EXPLICIT);
+         arkode->Init(oper);
+         arkode->SetSStolerances(reltol, abstol);
+         arkode->SetMaxStep(dt);
+         if (ode_solver_type == 11) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
+         ode_solver = arkode; break;
+      case 12:
+         arkode = new ARKStepSolver(MPI_COMM_WORLD, ARKStepSolver::IMPLICIT);
+         arkode->Init(oper);
+         arkode->SetSStolerances(reltol, abstol);
+         arkode->SetMaxStep(dt);
+         ode_solver = arkode; break;
+   }
+
+   // Initialize MFEM integrators, SUNDIALS integrators are initialized above
+   if (ode_solver_type < 8) { ode_solver->Init(oper); }
+
+   // Since we want to update the diffusion coefficient after every time step,
+   // we need to use the "one-step" mode of the SUNDIALS solvers.
+   if (cvode) { cvode->SetStepMode(CV_ONE_STEP); }
+   if (arkode) { arkode->SetStepMode(ARK_ONE_STEP); }
+
    // 10. Perform time-integration (looping over the time iterations, ti, with a
    //     time-step dt).
    if (myid == 0)
@@ -358,8 +355,6 @@ int main(int argc, char *argv[])
    }
    tic_toc.Clear();
    tic_toc.Start();
-   ode_solver->Init(oper);
-   double t = 0.0;
 
    bool last_step = false;
    for (int ti = 1; !last_step; ti++)
@@ -428,7 +423,7 @@ int main(int argc, char *argv[])
 ConductionOperator::ConductionOperator(ParFiniteElementSpace &f, double al,
                                        double kap, const Vector &u)
    : TimeDependentOperator(f.GetTrueVSize(), 0.0), fespace(f), M(NULL), K(NULL),
-     T(NULL), current_dt(0.0),
+     T(NULL),
      M_solver(f.GetComm()), T_solver(f.GetComm()), z(height)
 {
    const double rel_tol = 1e-8;
@@ -476,30 +471,32 @@ void ConductionOperator::ImplicitSolve(const double dt,
    // Solve the equation:
    //    du_dt = M^{-1}*[-K(u + dt*du_dt)]
    // for du_dt
-   if (!T)
-   {
-      T = Add(1.0, Mmat, dt, Kmat);
-      current_dt = dt;
-      T_solver.SetOperator(*T);
-   }
-   MFEM_VERIFY(dt == current_dt, ""); // SDIRK methods use the same dt
+   if (T) { delete T; }
+   T = Add(1.0, Mmat, dt, Kmat);
+   T_solver.SetOperator(*T);
    Kmat.Mult(u, z);
    z.Neg();
    T_solver.Mult(z, du_dt);
 }
 
-void ConductionOperator::SundialsSolve(const double dt, Vector &b)
+int ConductionOperator::SUNImplicitSetup(const Vector &x,
+                                         const Vector &fx, int jok, int *jcur,
+                                         double gamma)
 {
-   // Solve the system (M + dt K) y = M b. The result y replaces the input b.
-   if (!T || dt != current_dt)
-   {
-      delete T;
-      T = Add(1.0, Mmat, dt, Kmat);
-      current_dt = dt;
-      T_solver.SetOperator(*T);
-   }
+   // Setup the ODE Jacobian T = M + gamma K.
+   if (T) { delete T; }
+   T = Add(1.0, Mmat, gamma, Kmat);
+   T_solver.SetOperator(*T);
+   *jcur = 1;
+   return (0);
+}
+
+int ConductionOperator::SUNImplicitSolve(const Vector &b, Vector &x, double tol)
+{
+   // Solve the system A x = z => (M - gamma K) x = M b.
    Mmat.Mult(b, z);
-   T_solver.Mult(z, b);
+   T_solver.Mult(z, x);
+   return (0);
 }
 
 void ConductionOperator::SetParameters(const Vector &u)
@@ -519,8 +516,6 @@ void ConductionOperator::SetParameters(const Vector &u)
    K->AddDomainIntegrator(new DiffusionIntegrator(u_coeff));
    K->Assemble(0); // keep sparsity pattern of M and K the same
    K->FormSystemMatrix(ess_tdof_list, Kmat);
-   delete T;
-   T = NULL; // re-compute T on the next ImplicitSolve or SundialsSolve
 }
 
 ConductionOperator::~ConductionOperator()
@@ -530,46 +525,6 @@ ConductionOperator::~ConductionOperator()
    delete K;
 }
 
-
-int SundialsJacSolver::InitSystem(void *sundials_mem)
-{
-   TimeDependentOperator *td_oper = GetTimeDependentOperator(sundials_mem);
-
-   // During development, we use dynamic_cast<> to ensure the setup is correct:
-   oper = dynamic_cast<ConductionOperator*>(td_oper);
-   MFEM_VERIFY(oper, "operator is not ConductionOperator");
-
-   // When the implementation is finalized, we can switch to static_cast<>:
-   // oper = static_cast<ConductionOperator*>(td_oper);
-
-   return 0;
-}
-
-int SundialsJacSolver::SetupSystem(void *sundials_mem, int conv_fail,
-                                   const Vector &y_pred, const Vector &f_pred,
-                                   int &jac_cur, Vector &v_temp1,
-                                   Vector &v_temp2, Vector &v_temp3)
-{
-   jac_cur = 1;
-
-   return 0;
-}
-
-int SundialsJacSolver::SolveSystem(void *sundials_mem, Vector &b,
-                                   const Vector &weight, const Vector &y_cur,
-                                   const Vector &f_cur)
-{
-   oper->SundialsSolve(GetTimeStep(sundials_mem), b);
-
-   return 0;
-}
-
-int SundialsJacSolver::FreeSystem(void *sundials_mem)
-{
-   return 0;
-}
-
-
 double InitialTemperature(const Vector &x)
 {
    if (x.Norml2() < 0.5)

examples/sundials/ex9.cpp

@@ -4,14 +4,14 @@
 // Compile with: make ex9
 //
 // Sample runs:
-//    ex9 -m ../../data/periodic-segment.mesh -p 0 -r 2 -s 11 -dt 0.005
-//    ex9 -m ../../data/periodic-square.mesh  -p 1 -r 2 -s 12 -dt 0.005 -tf 9
-//    ex9 -m ../../data/periodic-hexagon.mesh -p 0 -r 2 -s 11 -dt 0.0018 -vs 25
-//    ex9 -m ../../data/periodic-hexagon.mesh -p 0 -r 2 -s 13 -dt 0.01 -vs 15
-//    ex9 -m ../../data/amr-quad.mesh         -p 1 -r 2 -s 13 -dt 0.002 -tf 9
-//    ex9 -m ../../data/star-q3.mesh          -p 1 -r 2 -s 13 -dt 0.005 -tf 9
-//    ex9 -m ../../data/disc-nurbs.mesh       -p 1 -r 3 -s 11 -dt 0.005 -tf 9
-//    ex9 -m ../../data/periodic-cube.mesh    -p 0 -r 2 -s 12 -dt 0.02 -tf 8 -o 2
+//    ex9 -m ../../data/periodic-segment.mesh -p 0 -r 2 -s 7 -dt 0.005
+//    ex9 -m ../../data/periodic-square.mesh  -p 1 -r 2 -s 8 -dt 0.005 -tf 9
+//    ex9 -m ../../data/periodic-hexagon.mesh -p 0 -r 2 -s 7 -dt 0.0018 -vs 25
+//    ex9 -m ../../data/periodic-hexagon.mesh -p 0 -r 2 -s 9 -dt 0.01 -vs 15
+//    ex9 -m ../../data/amr-quad.mesh         -p 1 -r 2 -s 9 -dt 0.002 -tf 9
+//    ex9 -m ../../data/star-q3.mesh          -p 1 -r 2 -s 9 -dt 0.005 -tf 9
+//    ex9 -m ../../data/disc-nurbs.mesh       -p 1 -r 3 -s 7 -dt 0.005 -tf 9
+//    ex9 -m ../../data/periodic-cube.mesh    -p 0 -r 2 -s 8 -dt 0.02 -tf 8 -o 2
 //
 // Description:  This example code solves the time-dependent advection equation
 //               du/dt + v.grad(u) = 0, where v is a given fluid velocity, and
@@ -109,11 +109,15 @@ int main(int argc, char *argv[])
    args.AddOption(&order, "-o", "--order",
                   "Order (degree) of the finite elements.");
    args.AddOption(&ode_solver_type, "-s", "--ode-solver",
-                  "ODE solver: 1 - Forward Euler,\n\t"
-                  "            2 - RK2 SSP, 3 - RK3 SSP, 4 - RK4, 6 - RK6,\n\t"
-                  "            11 - CVODE (adaptive order) explicit,\n\t"
-                  "            12 - ARKODE default (4th order) explicit,\n\t"
-                  "            13 - ARKODE RK8.");
+                  "ODE solver:\n\t"
+                  "1 - Forward Euler,\n\t"
+                  "2 - RK2 SSP,\n\t"
+                  "3 - RK3 SSP,\n\t"
+                  "4 - RK4,\n\t"
+                  "6 - RK6,\n\t"
+                  "7 - CVODE (adaptive order implicit Adams),\n\t"
+                  "8 - ARKODE default (4th order) explicit,\n\t"
+                  "9 - ARKODE RK8.");
    args.AddOption(&t_final, "-tf", "--t-final",
                   "Final time; start time is 0.");
    args.AddOption(&dt, "-dt", "--time-step",
@@ -135,65 +139,41 @@ int main(int argc, char *argv[])
       args.PrintUsage(cout);
       return 1;
    }
+   // check for vaild ODE solver option
+   if (ode_solver_type < 1 || ode_solver_type > 9)
+   {
+      cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
+      return 3;
+   }
    args.PrintOptions(cout);
 
    // 2. Read the mesh from the given mesh file. We can handle geometrically
    //    periodic meshes in this code.
-   Mesh *mesh = new Mesh(mesh_file, 1, 1);
-   int dim = mesh->Dimension();
+   Mesh mesh(mesh_file, 1, 1);
+   int dim = mesh.Dimension();
 
-   // 3. Define the ODE solver used for time integration. Several explicit
-   //    Runge-Kutta methods are available.
-   ODESolver *ode_solver = NULL;
-   CVODESolver *cvode = NULL;
-   ARKODESolver *arkode = NULL;
-   switch (ode_solver_type)
-   {
-      case 1: ode_solver = new ForwardEulerSolver; break;
-      case 2: ode_solver = new RK2Solver(1.0); break;
-      case 3: ode_solver = new RK3SSPSolver; break;
-      case 4: ode_solver = new RK4Solver; break;
-      case 6: ode_solver = new RK6Solver; break;
-      case 11:
-         cvode = new CVODESolver(CV_ADAMS, CV_FUNCTIONAL);
-         cvode->SetSStolerances(reltol, abstol);
-         cvode->SetMaxStep(dt);
-         ode_solver = cvode; break;
-      case 12:
-      case 13:
-         arkode = new ARKODESolver(ARKODESolver::EXPLICIT);
-         arkode->SetSStolerances(reltol, abstol);
-         arkode->SetMaxStep(dt);
-         if (ode_solver_type == 13) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
-         ode_solver = arkode; break;
-      default:
-         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
-         delete mesh;
-         return 3;
-   }
-
-   // 4. Refine the mesh to increase the resolution. In this example we do
+   // 3. Refine the mesh to increase the resolution. In this example we do
    //    'ref_levels' of uniform refinement, where 'ref_levels' is a
    //    command-line parameter. If the mesh is of NURBS type, we convert it to
    //    a (piecewise-polynomial) high-order mesh.
    for (int lev = 0; lev < ref_levels; lev++)
    {
-      mesh->UniformRefinement();
+      mesh.UniformRefinement();
    }
-   if (mesh->NURBSext)
+   if (mesh.NURBSext)
    {
-      mesh->SetCurvature(max(order, 1));
+      mesh.SetCurvature(max(order, 1));
    }
-   mesh->GetBoundingBox(bb_min, bb_max, max(order, 1));
+   mesh.GetBoundingBox(bb_min, bb_max, max(order, 1));
 
-   // 5. Define the discontinuous DG finite element space of the given
+   // 4. Define the discontinuous DG finite element space of the given
    //    polynomial order on the refined mesh.
    DG_FECollection fec(order, dim);
-   FiniteElementSpace fes(mesh, &fec);
+   FiniteElementSpace fes(&mesh, &fec);
 
    cout << "Number of unknowns: " << fes.GetVSize() << endl;
 
-   // 6. Set up and assemble the bilinear and linear forms corresponding to the
+   // 5. Set up and assemble the bilinear and linear forms corresponding to the
    //    DG discretization. The DGTraceIntegrator involves integrals over mesh
    //    interior faces.
    VectorFunctionCoefficient velocity(dim, velocity_function);
@@ -220,7 +200,7 @@ int main(int argc, char *argv[])
    k.Finalize(skip_zeros);
    b.Assemble();
 
-   // 7. Define the initial conditions, save the corresponding grid function to
+   // 6. Define the initial conditions, save the corresponding grid function to
    //    a file and (optionally) save data in the VisIt format and initialize
    //    GLVis visualization.
    GridFunction u(&fes);
@@ -229,7 +209,7 @@ int main(int argc, char *argv[])
    {
       ofstream omesh("ex9.mesh");
       omesh.precision(precision);
-      mesh->Print(omesh);
+      mesh.Print(omesh);
       ofstream osol("ex9-init.gf");
       osol.precision(precision);
       u.Save(osol);
@@ -243,14 +223,14 @@ int main(int argc, char *argv[])
       if (binary)
       {
 #ifdef MFEM_USE_SIDRE
-         dc = new SidreDataCollection("Example9", mesh);
+         dc = new SidreDataCollection("Example9", &mesh);
 #else
          MFEM_ABORT("Must build with MFEM_USE_SIDRE=YES for binary output.");
 #endif
       }
       else
       {
-         dc = new VisItDataCollection("Example9", mesh);
+         dc = new VisItDataCollection("Example9", &mesh);
          dc->SetPrecision(precision);
       }
       dc->RegisterField("solution", &u);
@@ -275,7 +255,7 @@ int main(int argc, char *argv[])
       else
       {
          sout.precision(precision);
-         sout << "solution\n" << *mesh << u;
+         sout << "solution\n" << mesh << u;
          sout << "pause\n";
          sout << flush;
          cout << "GLVis visualization paused."
@@ -283,15 +263,46 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 8. Define the time-dependent evolution operator describing the ODE
-   //    right-hand side, and perform time-integration (looping over the time
-   //    iterations, ti, with a time-step dt).
+   // 7. Define the time-dependent evolution operator describing the ODE
+   //    right-hand side, and define the ODE solver used for time integration.
    FE_Evolution adv(m.SpMat(), k.SpMat(), b);
 
    double t = 0.0;
    adv.SetTime(t);
-   ode_solver->Init(adv);
 
+   // Create the time integrator
+   ODESolver *ode_solver = NULL;
+   CVODESolver *cvode = NULL;
+   ARKStepSolver *arkode = NULL;
+   switch (ode_solver_type)
+   {
+      case 1: ode_solver = new ForwardEulerSolver; break;
+      case 2: ode_solver = new RK2Solver(1.0); break;
+      case 3: ode_solver = new RK3SSPSolver; break;
+      case 4: ode_solver = new RK4Solver; break;
+      case 6: ode_solver = new RK6Solver; break;
+      case 7:
+         cvode = new CVODESolver(CV_ADAMS);
+         cvode->Init(adv);
+         cvode->SetSStolerances(reltol, abstol);
+         cvode->SetMaxStep(dt);
+         cvode->UseSundialsLinearSolver();
+         ode_solver = cvode; break;
+      case 8:
+      case 9:
+         arkode = new ARKStepSolver(ARKStepSolver::EXPLICIT);
+         arkode->Init(adv);
+         arkode->SetSStolerances(reltol, abstol);
+         arkode->SetMaxStep(dt);
+         if (ode_solver_type == 9) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
+         ode_solver = arkode; break;
+   }
+
+   // Initialize MFEM integrators, SUNDIALS integrators are initialized above
+   if (ode_solver_type < 7) { ode_solver->Init(adv); }
+
+   // 8. Perform time-integration (looping over the time iterations, ti,
+   //    with a time-step dt).
    bool done = false;
    for (int ti = 0; !done; )
    {
@@ -309,7 +320,7 @@ int main(int argc, char *argv[])
 
          if (visualization)
          {
-            sout << "solution\n" << *mesh << u << flush;
+            sout << "solution\n" << mesh << u << flush;
          }
 
          if (visit)

examples/sundials/ex9p.cpp

@@ -4,14 +4,14 @@
 // Compile with: make ex9p
 //
 // Sample runs:
-//    mpirun -np 4 ex9p -m ../../data/periodic-segment.mesh -p 1 -rp 1 -s 11 -dt 0.0025
-//    mpirun -np 4 ex9p -m ../../data/periodic-square.mesh  -p 1 -rp 1 -s 12 -dt 0.0025 -tf 9
-//    mpirun -np 4 ex9p -m ../../data/periodic-hexagon.mesh -p 0 -rp 1 -s 11 -dt 0.0009 -vs 25
-//    mpirun -np 4 ex9p -m ../../data/periodic-hexagon.mesh -p 0 -rp 1 -s 13 -dt 0.005 -vs 15
-//    mpirun -np 4 ex9p -m ../../data/amr-quad.mesh         -p 1 -rp 1 -s 13 -dt 0.001 -tf 9
-//    mpirun -np 4 ex9p -m ../../data/star-q3.mesh          -p 1 -rp 1 -s 13 -dt 0.0025 -tf 9
-//    mpirun -np 4 ex9p -m ../../data/disc-nurbs.mesh       -p 1 -rp 2 -s 11 -dt 0.0025 -tf 9
-//    mpirun -np 4 ex9p -m ../../data/periodic-cube.mesh    -p 0 -rp 1 -s 12 -dt 0.01 -tf 8 -o 2
+//    mpirun -np 4 ex9p -m ../../data/periodic-segment.mesh -p 1 -rp 1 -s 7 -dt 0.0025
+//    mpirun -np 4 ex9p -m ../../data/periodic-square.mesh  -p 1 -rp 1 -s 8 -dt 0.0025 -tf 9
+//    mpirun -np 4 ex9p -m ../../data/periodic-hexagon.mesh -p 0 -rp 1 -s 7 -dt 0.0009 -vs 25
+//    mpirun -np 4 ex9p -m ../../data/periodic-hexagon.mesh -p 0 -rp 1 -s 9 -dt 0.005 -vs 15
+//    mpirun -np 4 ex9p -m ../../data/amr-quad.mesh         -p 1 -rp 1 -s 9 -dt 0.001 -tf 9
+//    mpirun -np 4 ex9p -m ../../data/star-q3.mesh          -p 1 -rp 1 -s 9 -dt 0.0025 -tf 9
+//    mpirun -np 4 ex9p -m ../../data/disc-nurbs.mesh       -p 1 -rp 2 -s 7 -dt 0.0025 -tf 9
+//    mpirun -np 4 ex9p -m ../../data/periodic-cube.mesh    -p 0 -rp 1 -s 8 -dt 0.01 -tf 8 -o 2
 //
 // Description:  This example code solves the time-dependent advection equation
 //               du/dt + v.grad(u) = 0, where v is a given fluid velocity, and
@@ -117,11 +117,15 @@ int main(int argc, char *argv[])
    args.AddOption(&order, "-o", "--order",
                   "Order (degree) of the finite elements.");
    args.AddOption(&ode_solver_type, "-s", "--ode-solver",
-                  "ODE solver: 1 - Forward Euler,\n\t"
-                  "            2 - RK2 SSP, 3 - RK3 SSP, 4 - RK4, 6 - RK6,\n\t"
-                  "            11 - CVODE (adaptive order) explicit,\n\t"
-                  "            12 - ARKODE default (4th order) explicit,\n\t"
-                  "            13 - ARKODE RK8.");
+                  "ODE solver:\n\t"
+                  "1 - Forward Euler,\n\t"
+                  "2 - RK2 SSP,\n\t"
+                  "3 - RK3 SSP,\n\t"
+                  "4 - RK4,\n\t"
+                  "6 - RK6,\n\t"
+                  "7 - CVODE (adaptive order implicit Adams),\n\t"
+                  "8 - ARKODE default (4th order) explicit,\n\t"
+                  "9 - ARKODE RK8.");
    args.AddOption(&t_final, "-tf", "--t-final",
                   "Final time; start time is 0.");
    args.AddOption(&dt, "-dt", "--time-step",
@@ -151,47 +155,23 @@ int main(int argc, char *argv[])
    {
       args.PrintOptions(cout);
    }
+   // check for vaild ODE solver option
+   if (ode_solver_type < 1 || ode_solver_type > 9)
+   {
+      if (myid == 0)
+      {
+         cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
+      }
+      MPI_Finalize();
+      return 3;
+   }
 
    // 3. Read the serial mesh from the given mesh file on all processors. We can
    //    handle geometrically periodic meshes in this code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
 
-   // 4. Define the ODE solver used for time integration. Several explicit
-   //    Runge-Kutta methods are available.
-   ODESolver *ode_solver = NULL;
-   CVODESolver *cvode = NULL;
-   ARKODESolver *arkode = NULL;
-   switch (ode_solver_type)
-   {
-      case 1: ode_solver = new ForwardEulerSolver; break;
-      case 2: ode_solver = new RK2Solver(1.0); break;
-      case 3: ode_solver = new RK3SSPSolver; break;
-      case 4: ode_solver = new RK4Solver; break;
-      case 6: ode_solver = new RK6Solver; break;
-      case 11:
-         cvode = new CVODESolver(MPI_COMM_WORLD, CV_ADAMS, CV_FUNCTIONAL);
-         cvode->SetSStolerances(reltol, abstol);
-         cvode->SetMaxStep(dt);
-         ode_solver = cvode; break;
-      case 12:
-      case 13:
-         arkode = new ARKODESolver(MPI_COMM_WORLD, ARKODESolver::EXPLICIT);
-         arkode->SetSStolerances(reltol, abstol);
-         arkode->SetMaxStep(dt);
-         if (ode_solver_type == 13) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
-         ode_solver = arkode; break;
-      default:
-         if (myid == 0)
-         {
-            cout << "Unknown ODE solver type: " << ode_solver_type << '\n';
-         }
-         delete mesh;
-         MPI_Finalize();
-         return 3;
-   }
-
-   // 5. Refine the mesh in serial to increase the resolution. In this example
+   // 4. Refine the mesh in serial to increase the resolution. In this example
    //    we do 'ser_ref_levels' of uniform refinement, where 'ser_ref_levels' is
    //    a command-line parameter. If the mesh is of NURBS type, we convert it
    //    to a (piecewise-polynomial) high-order mesh.
@@ -205,7 +185,7 @@ int main(int argc, char *argv[])
    }
    mesh->GetBoundingBox(bb_min, bb_max, max(order, 1));
 
-   // 6. Define the parallel mesh by a partitioning of the serial mesh. Refine
+   // 5. Define the parallel mesh by a partitioning of the serial mesh. Refine
    //    this mesh further in parallel to increase the resolution. Once the
    //    parallel mesh is defined, the serial mesh can be deleted.
    ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
@@ -215,7 +195,7 @@ int main(int argc, char *argv[])
       pmesh->UniformRefinement();
    }
 
-   // 7. Define the parallel discontinuous DG finite element space on the
+   // 6. Define the parallel discontinuous DG finite element space on the
    //    parallel refined mesh of the given polynomial order.
    DG_FECollection fec(order, dim);
    ParFiniteElementSpace *fes = new ParFiniteElementSpace(pmesh, &fec);
@@ -226,7 +206,7 @@ int main(int argc, char *argv[])
       cout << "Number of unknowns: " << global_vSize << endl;
    }
 
-   // 8. Set up and assemble the parallel bilinear and linear forms (and the
+   // 7. Set up and assemble the parallel bilinear and linear forms (and the
    //    parallel hypre matrices) corresponding to the DG discretization. The
    //    DGTraceIntegrator involves integrals over mesh interior faces.
    VectorFunctionCoefficient velocity(dim, velocity_function);
@@ -257,7 +237,7 @@ int main(int argc, char *argv[])
    HypreParMatrix *K = k->ParallelAssemble();
    HypreParVector *B = b->ParallelAssemble();
 
-   // 9. Define the initial conditions, save the corresponding grid function to
+   // 8. Define the initial conditions, save the corresponding grid function to
    //    a file and (optionally) save data in the VisIt format and initialize
    //    GLVis visualization.
    ParGridFunction *u = new ParGridFunction(fes);
@@ -330,15 +310,46 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 10. Define the time-dependent evolution operator describing the ODE
-   //     right-hand side, and perform time-integration (looping over the time
-   //     iterations, ti, with a time-step dt).
+   // 9. Define the time-dependent evolution operator describing the ODE
+   //    right-hand side, and define the ODE solver used for time integration.
    FE_Evolution adv(*M, *K, *B);
 
    double t = 0.0;
    adv.SetTime(t);
-   ode_solver->Init(adv);
 
+   // Create the time integrator
+   ODESolver *ode_solver = NULL;
+   CVODESolver *cvode = NULL;
+   ARKStepSolver *arkode = NULL;
+   switch (ode_solver_type)
+   {
+      case 1: ode_solver = new ForwardEulerSolver; break;
+      case 2: ode_solver = new RK2Solver(1.0); break;
+      case 3: ode_solver = new RK3SSPSolver; break;
+      case 4: ode_solver = new RK4Solver; break;
+      case 6: ode_solver = new RK6Solver; break;
+      case 7:
+         cvode = new CVODESolver(MPI_COMM_WORLD, CV_ADAMS);
+         cvode->Init(adv);
+         cvode->SetSStolerances(reltol, abstol);
+         cvode->SetMaxStep(dt);
+         cvode->UseSundialsLinearSolver();
+         ode_solver = cvode; break;
+      case 8:
+      case 9:
+         arkode = new ARKStepSolver(MPI_COMM_WORLD, ARKStepSolver::EXPLICIT);
+         arkode->Init(adv);
+         arkode->SetSStolerances(reltol, abstol);
+         arkode->SetMaxStep(dt);
+         if (ode_solver_type == 9) { arkode->SetERKTableNum(FEHLBERG_13_7_8); }
+         ode_solver = arkode; break;
+   }
+
+   // Initialize MFEM integrators, SUNDIALS integrators are initialized above
+   if (ode_solver_type < 7) { ode_solver->Init(adv); }
+
+   // 10. Perform time-integration (looping over the time iterations, ti,
+   //     with a time-step dt).
    bool done = false;
    for (int ti = 0; !done; )
    {

examples/sundials/makefile

@@ -60,15 +60,15 @@ PARALLEL_NAME := Parallel SUNDIALS example
 	@$(call mfem-test,$<,, $(SERIAL_NAME))
 
 # Testing: Specific execution options:
-# Example 9: test explicit CVODE time stepping
-EX9_COMMON_ARGS := -m ../../data/periodic-hexagon.mesh -p 0 -s 11
+# Example 9: test CVODE with CV_ADAMS (non-stiff implicit) time stepping
+EX9_COMMON_ARGS := -m ../../data/periodic-hexagon.mesh -p 0 -s 7
 EX9_ARGS  := $(EX9_COMMON_ARGS) -r  2 -dt 0.0018 -vs 25
 EX9P_ARGS := $(EX9_COMMON_ARGS) -rp 1 -dt 0.0009 -vs 50
 ex9-test-seq: ex9
 	@$(call mfem-test,$<,, $(SERIAL_NAME),$(EX9_ARGS))
 ex9p-test-par: ex9p
 	@$(call mfem-test,$<, $(RUN_MPI), $(PARALLEL_NAME),$(EX9P_ARGS))
-# Example 10: test implicit CVODE time stepping
+# Example 10: test CVODE with CV_BDF (stiff implicit) time stepping
 EX10_COMMON_ARGS := -m ../../data/beam-quad.mesh -o 2 -s 5 -dt 0.15 -tf 6 -vs 10
 EX10_ARGS  := $(EX10_COMMON_ARGS) -r  2
 EX10P_ARGS := $(EX10_COMMON_ARGS) -rp 1

fem/CMakeLists.txt

@@ -32,6 +32,7 @@ set(SRCS
   nonlininteg.cpp
   staticcond.cpp
   tmop.cpp
+  tmop_tools.cpp
   )
 
 set(HDRS
@@ -64,6 +65,7 @@ set(HDRS
   tfespace.hpp
   tintrules.hpp
   tmop.hpp
+  tmop_tools.hpp
   )
 
 if (MFEM_USE_SIDRE)

fem/bilininteg_diffusion.cpp

@@ -27,7 +27,7 @@ static void OccaPADiffusionSetup2D(const int D1D,
                                    const int NE,
                                    const Array<double> &W,
                                    const Vector &J,
-                                   const double COEFF,
+                                   const Vector &C,
                                    Vector &op)
 {
    occa::properties props;
@@ -35,7 +35,9 @@ static void OccaPADiffusionSetup2D(const int D1D,
    props["defines/Q1D"] = Q1D;
    const occa::memory o_W = OccaMemoryRead(W.GetMemory(), W.Size());
    const occa::memory o_J = OccaMemoryRead(J.GetMemory(), J.Size());
+   const occa::memory o_C = OccaMemoryRead(C.GetMemory(), C.Size());
    occa::memory o_op = OccaMemoryWrite(op.GetMemory(), op.Size());
+   const bool const_c = C.Size() == 1;
    const occa_id_t id = std::make_pair(D1D,Q1D);
    static occa_kernel_t OccaDiffSetup2D_ker;
    if (OccaDiffSetup2D_ker.find(id) == OccaDiffSetup2D_ker.end())
@@ -45,7 +47,7 @@ static void OccaPADiffusionSetup2D(const int D1D,
                                      "DiffusionSetup2D", props);
       OccaDiffSetup2D_ker.emplace(id, DiffusionSetup2D);
    }
-   OccaDiffSetup2D_ker.at(id)(NE, o_W, o_J, COEFF, o_op);
+   OccaDiffSetup2D_ker.at(id)(NE, o_W, o_J, o_C, o_op, const_c);
 }
 
 static void OccaPADiffusionSetup3D(const int D1D,
@@ -53,7 +55,7 @@ static void OccaPADiffusionSetup3D(const int D1D,
                                    const int NE,
                                    const Array<double> &W,
                                    const Vector &J,
-                                   const double COEFF,
+                                   const Vector &C,
                                    Vector &op)
 {
    occa::properties props;
@@ -61,7 +63,9 @@ static void OccaPADiffusionSetup3D(const int D1D,
    props["defines/Q1D"] = Q1D;
    const occa::memory o_W = OccaMemoryRead(W.GetMemory(), W.Size());
    const occa::memory o_J = OccaMemoryRead(J.GetMemory(), J.Size());
+   const occa::memory o_C = OccaMemoryRead(C.GetMemory(), C.Size());
    occa::memory o_op = OccaMemoryWrite(op.GetMemory(), op.Size());
+   const bool const_c = C.Size() == 1;
    const occa_id_t id = std::make_pair(D1D,Q1D);
    static occa_kernel_t OccaDiffSetup3D_ker;
    if (OccaDiffSetup3D_ker.find(id) == OccaDiffSetup3D_ker.end())
@@ -71,7 +75,7 @@ static void OccaPADiffusionSetup3D(const int D1D,
                                      "DiffusionSetup3D", props);
       OccaDiffSetup3D_ker.emplace(id, DiffusionSetup3D);
    }
-   OccaDiffSetup3D_ker.at(id)(NE, o_W, o_J, COEFF, o_op);
+   OccaDiffSetup3D_ker.at(id)(NE, o_W, o_J, o_C, o_op, const_c);
 }
 #endif // MFEM_USE_OCCA
 
@@ -80,14 +84,16 @@ static void PADiffusionSetup2D(const int Q1D,
                                const int NE,
                                const Array<double> &w,
                                const Vector &j,
-                               const double COEFF,
-                               Vector &op)
+                               const Vector &c,
+                               Vector &d)
 {
    const int NQ = Q1D*Q1D;
+   const bool const_c = c.Size() == 1;
    auto W = w.Read();
 
    auto J = Reshape(j.Read(), NQ, 2, 2, NE);
-   auto y = Reshape(op.Write(), NQ, 3, NE);
+   auto C = const_c ? Reshape(c.Read(), 1, 1) : Reshape(c.Read(), NQ, NE);
+   auto D = Reshape(d.Write(), NQ, 3, NE);
 
    MFEM_FORALL(e, NE,
    {
@@ -97,10 +103,11 @@ static void PADiffusionSetup2D(const int Q1D,
          const double J21 = J(q,1,0,e);
          const double J12 = J(q,0,1,e);
          const double J22 = J(q,1,1,e);
-         const double c_detJ = W[q] * COEFF / ((J11*J22)-(J21*J12));
-         y(q,0,e) =  c_detJ * (J12*J12 + J22*J22); // 1,1
-         y(q,1,e) = -c_detJ * (J12*J11 + J22*J21); // 1,2
-         y(q,2,e) =  c_detJ * (J11*J11 + J21*J21); // 2,2
+         const double coeff = const_c ? C(0,0) : C(q,e);
+         const double c_detJ = W[q] * coeff / ((J11*J22)-(J21*J12));
+         D(q,0,e) =  c_detJ * (J12*J12 + J22*J22); // 1,1
+         D(q,1,e) = -c_detJ * (J12*J11 + J22*J21); // 1,2
+         D(q,2,e) =  c_detJ * (J11*J11 + J21*J21); // 2,2
       }
    });
 }
@@ -110,13 +117,15 @@ static void PADiffusionSetup3D(const int Q1D,
                                const int NE,
                                const Array<double> &w,
                                const Vector &j,
-                               const double COEFF,
-                               Vector &op)
+                               const Vector &c,
+                               Vector &d)
 {
    const int NQ = Q1D*Q1D*Q1D;
+   const bool const_c = c.Size() == 1;
    auto W = w.Read();
    auto J = Reshape(j.Read(), NQ, 3, 3, NE);
-   auto y = Reshape(op.Write(), NQ, 6, NE);
+   auto C = const_c ? Reshape(c.Read(), 1, 1) : Reshape(c.Read(), NQ, NE);
+   auto D = Reshape(d.Write(), NQ, 6, NE);
    MFEM_FORALL(e, NE,
    {
       for (int q = 0; q < NQ; ++q)
@@ -133,7 +142,8 @@ static void PADiffusionSetup3D(const int Q1D,
          const double detJ = J11 * (J22 * J33 - J32 * J23) -
          /* */               J21 * (J12 * J33 - J32 * J13) +
          /* */               J31 * (J12 * J23 - J22 * J13);
-         const double c_detJ = W[q] * COEFF / detJ;
+         const double coeff = const_c ? C(0,0) : C(q,e);
+         const double c_detJ = W[q] * coeff / detJ;
          // adj(J)
          const double A11 = (J22 * J33) - (J23 * J32);
          const double A12 = (J32 * J13) - (J12 * J33);
@@ -145,12 +155,12 @@ static void PADiffusionSetup3D(const int Q1D,
          const double A32 = (J31 * J12) - (J11 * J32);
          const double A33 = (J11 * J22) - (J12 * J21);
          // detJ J^{-1} J^{-T} = (1/detJ) adj(J) adj(J)^T
-         y(q,0,e) = c_detJ * (A11*A11 + A12*A12 + A13*A13); // 1,1
-         y(q,1,e) = c_detJ * (A11*A21 + A12*A22 + A13*A23); // 2,1
-         y(q,2,e) = c_detJ * (A11*A31 + A12*A32 + A13*A33); // 3,1
-         y(q,3,e) = c_detJ * (A21*A21 + A22*A22 + A23*A23); // 2,2
-         y(q,4,e) = c_detJ * (A21*A31 + A22*A32 + A23*A33); // 3,2
-         y(q,5,e) = c_detJ * (A31*A31 + A32*A32 + A33*A33); // 3,3
+         D(q,0,e) = c_detJ * (A11*A11 + A12*A12 + A13*A13); // 1,1
+         D(q,1,e) = c_detJ * (A11*A21 + A12*A22 + A13*A23); // 2,1
+         D(q,2,e) = c_detJ * (A11*A31 + A12*A32 + A13*A33); // 3,1
+         D(q,3,e) = c_detJ * (A21*A21 + A22*A22 + A23*A23); // 2,2
+         D(q,4,e) = c_detJ * (A21*A31 + A22*A32 + A23*A33); // 3,2
+         D(q,5,e) = c_detJ * (A31*A31 + A32*A32 + A33*A33); // 3,3
       }
    });
 }
@@ -161,8 +171,8 @@ static void PADiffusionSetup(const int dim,
                              const int NE,
                              const Array<double> &W,
                              const Vector &J,
-                             const double COEFF,
-                             Vector &op)
+                             const Vector &C,
+                             Vector &D)
 {
    if (dim == 1) { MFEM_ABORT("dim==1 not supported in PADiffusionSetup"); }
    if (dim == 2)
@@ -170,22 +180,22 @@ static void PADiffusionSetup(const int dim,
 #ifdef MFEM_USE_OCCA
       if (DeviceCanUseOcca())
       {
-         OccaPADiffusionSetup2D(D1D, Q1D, NE, W, J, COEFF, op);
+         OccaPADiffusionSetup2D(D1D, Q1D, NE, W, J, C, D);
          return;
       }
 #endif // MFEM_USE_OCCA
-      PADiffusionSetup2D(Q1D, NE, W, J, COEFF, op);
+      PADiffusionSetup2D(Q1D, NE, W, J, C, D);
    }
    if (dim == 3)
    {
 #ifdef MFEM_USE_OCCA
       if (DeviceCanUseOcca())
       {
-         OccaPADiffusionSetup3D(D1D, Q1D, NE, W, J, COEFF, op);
+         OccaPADiffusionSetup3D(D1D, Q1D, NE, W, J, C, D);
          return;
       }
 #endif // MFEM_USE_OCCA
-      PADiffusionSetup3D(Q1D, NE, W, J, COEFF, op);
+      PADiffusionSetup3D(Q1D, NE, W, J, C, D);
    }
 }
 
@@ -205,11 +215,32 @@ void DiffusionIntegrator::AssemblePA(const FiniteElementSpace &fes)
    dofs1D = maps->ndof;
    quad1D = maps->nqpt;
    pa_data.SetSize(symmDims * nq * ne, Device::GetMemoryType());
-   ConstantCoefficient *cQ = dynamic_cast<ConstantCoefficient*>(Q);
-   MFEM_VERIFY(cQ != NULL, "only ConstantCoefficient is supported!");
-   const double coeff = cQ->constant;
-   PADiffusionSetup(dim, dofs1D, quad1D, ne, ir->GetWeights(), geom->J,
-                    coeff, pa_data);
+   Vector coeff;
+   if (Q == nullptr)
+   {
+      coeff.SetSize(1);
+      coeff(0) = 1.0;
+   }
+   else if (ConstantCoefficient* cQ = dynamic_cast<ConstantCoefficient*>(Q))
+   {
+      coeff.SetSize(1);
+      coeff(0) = cQ->constant;
+   }
+   else
+   {
+      coeff.SetSize(nq * ne);
+      auto C = Reshape(coeff.Write(), nq, ne);
+      for (int e = 0; e < ne; ++e)
+      {
+         ElementTransformation& T = *fes.GetElementTransformation(e);
+         for (int q = 0; q < nq; ++q)
+         {
+            C(q,e) = Q->Eval(T, ir->IntPoint(q));
+         }
+      }
+   }
+   PADiffusionSetup(dim, dofs1D, quad1D, ne, ir->GetWeights(), geom->J, coeff,
+                    pa_data);
 }
 
 #ifdef MFEM_USE_OCCA
@@ -1053,40 +1084,7 @@ static void PADiffusionApply(const int dim,
       MFEM_ABORT("OCCA PADiffusionApply unknown kernel!");
    }
 #endif // MFEM_USE_OCCA
-
-   if (Device::Allows(Backend::RAJA_CUDA))
-   {
-      if (dim == 2)
-      {
-         switch ((D1D << 4 ) | Q1D)
-         {
-            case 0x22: return PADiffusionApply2D<2,2>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x33: return PADiffusionApply2D<3,3>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x44: return PADiffusionApply2D<4,4>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x55: return PADiffusionApply2D<5,5>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x66: return PADiffusionApply2D<6,6>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x77: return PADiffusionApply2D<7,7>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x88: return PADiffusionApply2D<8,8>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x99: return PADiffusionApply2D<9,9>(NE,B,G,Bt,Gt,op,x,y);
-            default:   return PADiffusionApply2D(NE,B,G,Bt,Gt,op,x,y,D1D,Q1D);
-         }
-      }
-      if (dim == 3)
-      {
-         switch ((D1D << 4 ) | Q1D)
-         {
-            case 0x23: return PADiffusionApply3D<2,3>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x34: return PADiffusionApply3D<3,4>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x45: return PADiffusionApply3D<4,5>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x56: return PADiffusionApply3D<5,6>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x67: return PADiffusionApply3D<6,7>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x78: return PADiffusionApply3D<7,8>(NE,B,G,Bt,Gt,op,x,y);
-            case 0x89: return PADiffusionApply3D<8,9>(NE,B,G,Bt,Gt,op,x,y);
-            default:   return PADiffusionApply3D(NE,B,G,Bt,Gt,op,x,y,D1D,Q1D);
-         }
-      }
-   }
-   else if (dim == 2)
+   if (dim == 2)
    {
       switch ((D1D << 4 ) | Q1D)
       {

fem/bilininteg_mass.cpp

@@ -38,24 +38,40 @@ void MassIntegrator::AssemblePA(const FiniteElementSpace &fes)
    dofs1D = maps->ndof;
    quad1D = maps->nqpt;
    pa_data.SetSize(ne*nq, Device::GetMemoryType());
-   ConstantCoefficient *const_coeff = dynamic_cast<ConstantCoefficient*>(Q);
-   // TODO: other types of coefficients ...
+   Vector coeff;
+   if (Q == nullptr)
+   {
+      coeff.SetSize(1);
+      coeff(0) = 1.0;
+   }
+   else if (ConstantCoefficient* cQ = dynamic_cast<ConstantCoefficient*>(Q))
+   {
+      coeff.SetSize(1);
+      coeff(0) = cQ->constant;
+   }
+   else
+   {
+      coeff.SetSize(nq * ne);
+      auto C = Reshape(coeff.Write(), nq, ne);
+      for (int e = 0; e < ne; ++e)
+      {
+         ElementTransformation& T = *fes.GetElementTransformation(e);
+         for (int q = 0; q < nq; ++q)
+         {
+            C(q,e) = Q->Eval(T, ir->IntPoint(q));
+         }
+      }
+   }
    if (dim==1) { MFEM_ABORT("Not supported yet... stay tuned!"); }
    if (dim==2)
    {
-      double constant = 0.0;
-      if (const_coeff)
-      {
-         constant = const_coeff->constant;
-      }
-      else
-      {
-         MFEM_ABORT("Coefficient type not supported");
-      }
       const int NE = ne;
       const int NQ = nq;
+      const bool const_c = coeff.Size() == 1;
       auto w = ir->GetWeights().Read();
       auto J = Reshape(geom->J.Read(), NQ,2,2,NE);
+      auto C =
+         const_c ? Reshape(coeff.Read(), 1,1) : Reshape(coeff.Read(), NQ,NE);
       auto v = Reshape(pa_data.Write(), NQ, NE);
       MFEM_FORALL(e, NE,
       {
@@ -66,25 +82,20 @@ void MassIntegrator::AssemblePA(const FiniteElementSpace &fes)
             const double J21 = J(q,0,1,e);
             const double J22 = J(q,1,1,e);
             const double detJ = (J11*J22)-(J21*J12);
-            v(q,e) =  w[q] * constant * detJ;
+            const double coeff = const_c ? C(0,0) : C(q,e);
+            v(q,e) =  w[q] * coeff * detJ;
          }
       });
    }
    if (dim==3)
    {
-      double constant = 0.0;
-      if (const_coeff)
-      {
-         constant = const_coeff->constant;
-      }
-      else
-      {
-         MFEM_ABORT("Coefficient type not supported");
-      }
       const int NE = ne;
       const int NQ = nq;
+      const bool const_c = coeff.Size() == 1;
       auto W = ir->GetWeights().Read();
       auto J = Reshape(geom->J.Read(), NQ,3,3,NE);
+      auto C =
+         const_c ? Reshape(coeff.Read(), 1,1) : Reshape(coeff.Read(), NQ,NE);
       auto v = Reshape(pa_data.Write(), NQ,NE);
       MFEM_FORALL(e, NE,
       {
@@ -96,7 +107,8 @@ void MassIntegrator::AssemblePA(const FiniteElementSpace &fes)
             const double detJ = J11 * (J22 * J33 - J32 * J23) -
             /* */               J21 * (J12 * J33 - J32 * J13) +
             /* */               J31 * (J12 * J23 - J22 * J13);
-            v(q,e) = W[q] * constant * detJ;
+            const double coeff = const_c ? C(0,0) : C(q,e);
+            v(q,e) = W[q] * coeff * detJ;
          }
       });
    }
@@ -749,42 +761,9 @@ static void PAMassApply(const int dim,
       MFEM_ABORT("OCCA PA Mass Apply unknown kernel!");
    }
 #endif // MFEM_USE_OCCA
-
-   if (Device::Allows(Backend::RAJA_CUDA))
+   if (dim == 2)
    {
-      if (dim == 2)
-      {
-         switch ((D1D << 4 ) | Q1D)
-         {
-            case 0x22: return PAMassApply2D<2,2>(NE, B, Bt, op, x, y);
-            case 0x33: return PAMassApply2D<3,3>(NE, B, Bt, op, x, y);
-            case 0x44: return PAMassApply2D<4,4>(NE, B, Bt, op, x, y);
-            case 0x55: return PAMassApply2D<5,5>(NE, B, Bt, op, x, y);
-            case 0x66: return PAMassApply2D<6,6>(NE, B, Bt, op, x, y);
-            case 0x77: return PAMassApply2D<7,7>(NE, B, Bt, op, x, y);
-            case 0x88: return PAMassApply2D<8,8>(NE, B, Bt, op, x, y);
-            case 0x99: return PAMassApply2D<9,9>(NE, B, Bt, op, x, y);
-            default:   return PAMassApply2D(NE, B, Bt, op, x, y, D1D, Q1D);
-         }
-      }
-      if (dim == 3)
-      {
-         switch ((D1D << 4 ) | Q1D)
-         {
-            case 0x23: return PAMassApply3D<2,3>(NE, B, Bt, op, x, y);
-            case 0x34: return PAMassApply3D<3,4>(NE, B, Bt, op, x, y);
-            case 0x45: return PAMassApply3D<4,5>(NE, B, Bt, op, x, y);
-            case 0x56: return PAMassApply3D<5,6>(NE, B, Bt, op, x, y);
-            case 0x67: return PAMassApply3D<6,7>(NE, B, Bt, op, x, y);
-            case 0x78: return PAMassApply3D<7,8>(NE, B, Bt, op, x, y);
-            case 0x89: return PAMassApply3D<8,9>(NE, B, Bt, op, x, y);
-            default:   return PAMassApply3D(NE, B, Bt, op, x, y, D1D, Q1D);
-         }
-      }
-   }
-   else if (dim == 2)
-   {
-      switch ((D1D << 4 ) | Q1D)
+      switch ((D1D << 4) | Q1D)
       {
          case 0x22: return SmemPAMassApply2D<2,2,16>(NE, B, Bt, op, x, y);
          case 0x33: return SmemPAMassApply2D<3,3,16>(NE, B, Bt, op, x, y);
@@ -799,7 +778,7 @@ static void PAMassApply(const int dim,
    }
    else if (dim == 3)
    {
-      switch ((D1D << 4 ) | Q1D)
+      switch ((D1D << 4) | Q1D)
       {
          case 0x23: return SmemPAMassApply3D<2,3>(NE, B, Bt, op, x, y);
          case 0x34: return SmemPAMassApply3D<3,4>(NE, B, Bt, op, x, y);

fem/fe.cpp

@@ -9635,6 +9635,7 @@ void L2_TetrahedronElement::ProjectDelta(int vertex, Vector &dofs) const
             const IntegrationPoint &ip = Nodes.IntPoint(i);
             dofs[i] = pow(ip.y, Order);
          }
+         break;
       case 3:
          for (int i = 0; i < Dof; i++)
          {
@@ -11965,6 +11966,10 @@ Linear3DFiniteElement TetrahedronFE;
 // Object declared in mesh/wedge.hpp.
 // Defined here to ensure it is constructed after 'poly1d' and before
 // 'Geometries'.
+// TODO: define as thread_local to prevent race conditions in GLVis, because
+// there is no "LinearWedgeFiniteElement" and WedgeFE is in turn used from two
+// different threads for different things in GLVis. We also don't want to turn
+// MFEM_THREAD_SAFE on globally. (See PR #731)
 H1_WedgeElement WedgeFE(1);
 
 // Object declared in geom.hpp.

fem/fem.hpp

@@ -31,6 +31,7 @@
 #include "estimators.hpp"
 #include "staticcond.hpp"
 #include "tmop.hpp"
+#include "tmop_tools.hpp"
 
 #ifdef MFEM_USE_MPI
 #include "pfespace.hpp"

fem/fespace.cpp

@@ -567,6 +567,40 @@ bool FiniteElementSpace::DofFinalizable(int dof, const Array<bool>& finalized,
    return true;
 }
 
+void FiniteElementSpace::GetDegenerateFaceDofs(int index,
+                                               Array<int> &dofs) const
+{
+   // In NC meshes with prisms, a special constraint occurs where a prism edge
+   // is slave to a quadrilateral face. Rather than introduce a new edge-face
+   // constraint type, we handle such cases as degenerate face-face constraints,
+   // where the point-matrix rectangle has zero height. This method returns
+   // DOFs for the first edge of the rectangle, duplicated in the orthogonal
+   // direction, to resemble DOFs for a quadrilateral face. The extra DOFs are
+   // ignored by FiniteElementSpace::AddDependencies.
+
+   Array<int> edof;
+   GetEdgeDofs(-1 - index, edof);
+
+   int nv = fec->DofForGeometry(Geometry::POINT);
+   int ne = fec->DofForGeometry(Geometry::SEGMENT);
+   int nn = 2*nv + ne;
+
+   dofs.SetSize(nn*nn);
+   dofs = edof[0];
+
+   // copy first two vertex DOFs
+   for (int i = 0; i < nv; i++)
+   {
+      dofs[i] = edof[i];
+      dofs[nv+i] = edof[nv+i];
+   }
+   // copy first edge DOFs
+   for (int i = 0; i < ne; i++)
+   {
+      dofs[4*nv + i] = edof[2*nv + i];
+   }
+}
+
 void
 FiniteElementSpace::GetEntityDofs(int entity, int index, Array<int> &dofs) const
 {
@@ -574,7 +608,8 @@ FiniteElementSpace::GetEntityDofs(int entity, int index, Array<int> &dofs) const
    {
       case 0: GetVertexDofs(index, dofs); break;
       case 1: GetEdgeDofs(index, dofs); break;
-      case 2: GetFaceDofs(index, dofs); break;
+      case 2: (index >= 0) ? GetFaceDofs(index, dofs)
+         /*             */ : GetDegenerateFaceDofs(index, dofs);
    }
 }
 
@@ -597,28 +632,33 @@ void FiniteElementSpace::BuildConformingInterpolation() const
    // collect local edge/face dependencies
    for (int entity = 1; entity <= 2; entity++)
    {
-      const NCMesh::NCList &list = (entity > 1) ? mesh->ncmesh->GetFaceList()
-                                   /*        */ : mesh->ncmesh->GetEdgeList();
+      const NCMesh::NCList &list = mesh->ncmesh->GetNCList(entity);
       if (!list.masters.size()) { continue; }
 
-      IsoparametricTransformation T;
-      if (entity > 1) { T.SetFE(&QuadrilateralFE); }
-      else { T.SetFE(&SegmentFE); }
-
-      Geometry::Type geom = (entity > 1) ? Geometry::SQUARE : Geometry::SEGMENT;
-      const FiniteElement* fe = fec->FiniteElementForGeometry(geom);
-      if (!fe) { continue; }
-
       Array<int> master_dofs, slave_dofs;
-      DenseMatrix I(fe->GetDof());
+
+      IsoparametricTransformation T;
+      DenseMatrix I;
 
       // loop through all master edges/faces, constrain their slave edges/faces
       for (unsigned mi = 0; mi < list.masters.size(); mi++)
       {
          const NCMesh::Master &master = list.masters[mi];
+
          GetEntityDofs(entity, master.index, master_dofs);
          if (!master_dofs.Size()) { continue; }
 
+         const FiniteElement* fe = fec->FiniteElementForGeometry(master.Geom());
+         if (!fe) { continue; }
+
+         switch (master.geom)
+         {
+            case Geometry::SQUARE:   T.SetFE(&QuadrilateralFE); break;
+            case Geometry::TRIANGLE: T.SetFE(&TriangleFE); break;
+            case Geometry::SEGMENT:  T.SetFE(&SegmentFE); break;
+            default: MFEM_ABORT("unsupported geometry");
+         }
+
          for (int si = master.slaves_begin; si < master.slaves_end; si++)
          {
             const NCMesh::Slave &slave = list.slaves[si];
@@ -791,8 +831,14 @@ const Operator *FiniteElementSpace::GetElementRestriction(
 {
    // Check if we have a discontinuous space using the FE collection:
    const L2_FECollection *dg_space = dynamic_cast<const L2_FECollection*>(fec);
-   if (dg_space) { return NULL; }
-   // TODO: support other DG collections.
+   if (dg_space)
+   {
+      if (L2E_nat.Ptr() == NULL)
+      {
+         L2E_nat.Reset(new L2ElementRestriction(*this));
+      }
+      return L2E_nat.Ptr();
+   }
    if (e_ordering == ElementDofOrdering::LEXICOGRAPHIC)
    {
       if (L2E_lex.Ptr() == NULL)
@@ -905,7 +951,7 @@ void FiniteElementSpace::GetLocalRefinementMatrices(
    const FiniteElement *fe = fec->FiniteElementForGeometry(geom);
 
    const CoarseFineTransformations &rtrans = mesh->GetRefinementTransforms();
-   const DenseTensor &pmats = rtrans.GetPointMatrices(geom);
+   const DenseTensor &pmats = rtrans.point_matrices[geom];
 
    int nmat = pmats.SizeK();
    int ldof = fe->GetDof(); // assuming the same FE everywhere
@@ -944,7 +990,9 @@ FiniteElementSpace::RefinementOperator::RefinementOperator
    : fespace(fespace)
    , old_elem_dof(old_elem_dof)
 {
-   MFEM_VERIFY(fespace->GetNDofs() >= old_ndofs,
+   const Mesh* mesh = fespace->GetMesh();
+   MFEM_VERIFY(mesh->ReduceInt(fespace->GetNDofs()) >=
+               mesh->ReduceInt(old_ndofs),
                "Previous space is not coarser.");
 
    width = old_ndofs * fespace->GetVDim();
@@ -1054,7 +1102,7 @@ FiniteElementSpace::DerefinementOperator::DerefinementOperator(
          f_fes->fec->FiniteElementForGeometry(geom);
       const FiniteElement *coarse_fe =
          c_fes->fec->FiniteElementForGeometry(geom);
-      const DenseTensor &pmats = rtrans.GetPointMatrices(geom);
+      const DenseTensor &pmats = rtrans.point_matrices[geom];
 
       lP.SetSize(fine_fe->GetDof(), coarse_fe->GetDof(), pmats.SizeK());
       lM.SetSize(fine_fe->GetDof(),   fine_fe->GetDof(), pmats.SizeK());
@@ -1170,7 +1218,7 @@ void FiniteElementSpace::GetLocalDerefinementMatrices(Geometry::Type geom,
 
    const CoarseFineTransformations &dtrans =
       mesh->ncmesh->GetDerefinementTransforms();
-   const DenseTensor &pmats = dtrans.GetPointMatrices(geom);
+   const DenseTensor &pmats = dtrans.point_matrices[geom];
 
    const int nmat = pmats.SizeK();
    const int ldof = fe->GetDof();
@@ -1275,7 +1323,7 @@ void FiniteElementSpace::GetLocalRefinementMatrices(
       coarse_fes.fec->FiniteElementForGeometry(geom);
 
    const CoarseFineTransformations &rtrans = mesh->GetRefinementTransforms();
-   const DenseTensor &pmats = rtrans.GetPointMatrices(geom);
+   const DenseTensor &pmats = rtrans.point_matrices[geom];
 
    int nmat = pmats.SizeK();
 
@@ -1368,31 +1416,26 @@ void FiniteElementSpace::UpdateNURBS()
 void FiniteElementSpace::Construct()
 {
    // This method should be used only for non-NURBS spaces.
-   MFEM_ASSERT(!NURBSext, "internal error");
+   MFEM_VERIFY(!NURBSext, "internal error");
 
    elem_dof = NULL;
    bdrElem_dof = NULL;
 
-   nvdofs = mesh->GetNV() * fec->DofForGeometry(Geometry::POINT);
-
-   if ( mesh->Dimension() > 1 )
-   {
-      nedofs = mesh->GetNEdges() * fec->DofForGeometry(Geometry::SEGMENT);
-   }
-   else
-   {
-      nedofs = 0;
-   }
-
    ndofs = 0;
-   nfdofs = 0;
-   nbdofs = 0;
+   nedofs = nfdofs = nbdofs = 0;
    bdofs = NULL;
    fdofs = NULL;
    cP = NULL;
    cR = NULL;
    cP_is_set = false;
-   // Th is initialized/destroyed before this method is called.
+   // 'Th' is initialized/destroyed before this method is called.
+
+   nvdofs = mesh->GetNV() * fec->DofForGeometry(Geometry::POINT);
+
+   if (mesh->Dimension() > 1)
+   {
+      nedofs = mesh->GetNEdges() * fec->DofForGeometry(Geometry::SEGMENT);
+   }
 
    if (mesh->GetNFaces() > 0)
    {
@@ -1424,8 +1467,7 @@ void FiniteElementSpace::Construct()
       bdofs[0] = 0;
       for (int i = 0; i < mesh->GetNE(); i++)
       {
-         Geometry::Type geom = mesh->GetElementBaseGeometry(i);
-         nbdofs += fec->DofForGeometry(geom);
+         nbdofs += fec->DofForGeometry(mesh->GetElementBaseGeometry(i));
          bdofs[i+1] = nbdofs;
       }
    }
@@ -1436,7 +1478,7 @@ void FiniteElementSpace::Construct()
    // later.
 }
 
-void FiniteElementSpace::GetElementDofs (int i, Array<int> &dofs) const
+void FiniteElementSpace::GetElementDofs(int i, Array<int> &dofs) const
 {
    if (elem_dof)
    {
@@ -2513,7 +2555,7 @@ L2ProjectionGridTransfer::L2Projection::L2Projection(
    Vector shape_lor(ndof_lor);
 
    const Geometry::Type geom = fe_ho->GetGeomType();
-   const DenseTensor &pmats = cf_tr.GetPointMatrices(geom);
+   const DenseTensor &pmats = cf_tr.point_matrices[geom];
    emb_tr.SetIdentityTransformation(geom);
 
    for (int iho=0; iho<nel_ho; ++iho)
@@ -2536,7 +2578,7 @@ L2ProjectionGridTransfer::L2Projection::L2Projection(
 
          // Create the transformation that embeds the fine low-order element
          // within the coarse high-order element in reference space
-         emb_tr.GetPointMat() = pmats(iref);
+         emb_tr.GetPointMat() = pmats(cf_tr.embeddings[ilor].matrix);
          emb_tr.FinalizeTransformation();
 
          int order = fe_lor->GetOrder() + fe_ho->GetOrder() + el_tr->OrderW();
@@ -2637,6 +2679,70 @@ const Operator &L2ProjectionGridTransfer::BackwardOperator()
    return *B;
 }
 
+L2ElementRestriction::L2ElementRestriction(const FiniteElementSpace &fes)
+   : ne(fes.GetNE()),
+     vdim(fes.GetVDim()),
+     byvdim(fes.GetOrdering() == Ordering::byVDIM),
+     ndof(ne > 0 ? fes.GetFE(0)->GetDof() : 0)
+{
+   height = vdim*ne*ndof;
+   width = vdim*ne*ndof;
+}
+
+void L2ElementRestriction::Mult(const Vector &x, Vector &y) const
+{
+   for (int iel=0; iel<ne; ++iel)
+   {
+      for (int vd=0; vd<vdim; ++vd)
+      {
+         for (int idof=0; idof<ndof; ++idof)
+         {
+            // E-vector dimensions (dofs, vdim, elements)
+            // L-vector dimensions: byVDIM:  (vdim, dofs, element)
+            //                      byNODES: (dofs, elements, vdim)
+            int yidx = iel*vdim*ndof + vd*ndof + idof;
+            int xidx;
+            if (byvdim)
+            {
+               xidx = iel*ndof*vdim + idof*vdim + vd;
+            }
+            else
+            {
+               xidx = vd*ne*ndof + iel*ndof + idof;
+            }
+            y[yidx] = x[xidx];
+         }
+      }
+   }
+}
+
+void L2ElementRestriction::MultTranspose(const Vector &x, Vector &y) const
+{
+   // Since this restriction is a permutation, the transpose is the inverse
+   for (int iel=0; iel<ne; ++iel)
+   {
+      for (int vd=0; vd<vdim; ++vd)
+      {
+         for (int idof=0; idof<ndof; ++idof)
+         {
+            // E-vector dimensions (dofs, vdim, elements)
+            // L-vector dimensions: byVDIM:  (vdim, dofs, element)
+            //                      byNODES: (dofs, elements, vdim)
+            int xidx = iel*vdim*ndof + vd*ndof + idof;
+            int yidx;
+            if (byvdim)
+            {
+               yidx = iel*ndof*vdim + idof*vdim + vd;
+            }
+            else
+            {
+               yidx = vd*ne*ndof + iel*ndof + idof;
+            }
+            y[yidx] = x[xidx];
+         }
+      }
+   }
+}
 
 ElementRestriction::ElementRestriction(const FiniteElementSpace &f,
                                        ElementDofOrdering e_ordering)

fem/fespace.hpp

@@ -146,6 +146,8 @@ protected:
 
    /// Helper to get vertex, edge or face DOFs (entity=0,1,2 resp.).
    void GetEntityDofs(int entity, int index, Array<int> &dofs) const;
+   // Get degenerate face DOFs: see explanation in method implementation.
+   void GetDegenerateFaceDofs(int index, Array<int> &dofs) const;
 
    /// Calculate the cP and cR matrices for a nonconforming mesh.
    void BuildConformingInterpolation() const;
@@ -156,6 +158,7 @@ protected:
    static bool DofFinalizable(int dof, const Array<bool>& finalized,
                               const SparseMatrix& deps);
 
+   /// Replicate 'mat' in the vector dimension, according to vdim ordering mode.
    void MakeVDimMatrix(SparseMatrix &mat) const;
 
    /// GridFunction interpolation operator applicable after mesh refinement.
@@ -304,8 +307,9 @@ public:
        The parameter @a e_ordering describes how the local DOFs in each element
        should be ordered, see ElementDofOrdering.
 
-       For discontinuous spaces, where the element-restriction is the identity,
-       this method will return NULL.
+       For discontinuous spaces, the element restriction corresponds to a
+       permutation of the degrees of freedom, implemented by the
+       L2ElementRestriction class.
 
        The returned Operator is owned by the FiniteElementSpace. */
    const Operator *GetElementRestriction(ElementDofOrdering e_ordering) const;
@@ -896,6 +900,22 @@ public:
    void MultTranspose(const Vector &x, Vector &y) const;
 };
 
+/// Operator that converts L2 FiniteElementSpace L-vectors to E-vectors.
+/** Objects of this type are typically created and owned by FiniteElementSpace
+    objects, see FiniteElementSpace::GetElementRestriction(). L-vectors
+    corresponding to grid functions in L2 finite element spaces differ from
+    E-vectors only in the ordering of the degrees of freedom. */
+class L2ElementRestriction : public Operator
+{
+   const int ne;
+   const int vdim;
+   const bool byvdim;
+   const int ndof;
+public:
+   L2ElementRestriction(const FiniteElementSpace&);
+   void Mult(const Vector &x, Vector &y) const;
+   void MultTranspose(const Vector &x, Vector &y) const;
+};
 
 /** @brief A class that performs interpolation from an E-vector to quadrature
     point values and/or derivatives (Q-vectors). */

fem/gridfunc.cpp

@@ -1716,6 +1716,7 @@ void GridFunction::ProjectDiscCoefficient(VectorCoefficient &coeff,
    Array<int> vdofs;
    Vector vals;
 
+   HostWrite();
    // maximal element attribute for each dof
    dof_attr.SetSize(fes->GetVSize());
    dof_attr = -1;

fem/gridfunc.hpp

@@ -434,6 +434,8 @@ public:
    /** The GridFunction is resized using the SetSize() method. */
    virtual void SetSpace(FiniteElementSpace *f);
 
+   using Vector::MakeRef;
+
    /** @brief Make the GridFunction reference external data on a new
        FiniteElementSpace. */
    /** This method changes the FiniteElementSpace associated with the

fem/linearform.cpp

@@ -138,7 +138,11 @@ void LinearForm::Assemble()
          eltrans = fes -> GetBdrElementTransformation (i);
          for (int k=0; k < blfi.Size(); k++)
          {
+            if (blfi_marker[k] &&
+                (*blfi_marker[k])[bdr_attr-1] == 0) { continue; }
+
             blfi[k]->AssembleRHSElementVect(*fes->GetBE(i), *eltrans, elemvect);
+
             AddElementVector (vdofs, elemvect);
          }
       }

fem/lininteg.cpp

@@ -350,7 +350,6 @@ void VectorFEDomainLFIntegrator::AssembleDeltaElementVect(
    vshape.Mult(vec, elvect);
 }
 
-
 void VectorBoundaryFluxLFIntegrator::AssembleRHSElementVect(
    const FiniteElement &el, ElementTransformation &Tr, Vector &elvect)
 {
@@ -397,19 +396,26 @@ void VectorFEBoundaryFluxLFIntegrator::AssembleRHSElementVect(
    if (ir == NULL)
    {
       int intorder = 2*el.GetOrder();  // <----------
+      if (F == NULL)
+      {
+         intorder -= el.GetOrder() + 1;
+      }
       ir = &IntRules.Get(el.GetGeomType(), intorder);
    }
 
    for (int i = 0; i < ir->GetNPoints(); i++)
    {
       const IntegrationPoint &ip = ir->IntPoint(i);
-
-      Tr.SetIntPoint (&ip);
-      double val = ip.weight*F.Eval(Tr, ip);
-
       el.CalcShape(ip, shape);
 
-      add(elvect, val, shape, elvect);
+      double val = ip.weight;
+      if (F)
+      {
+         Tr.SetIntPoint (&ip);
+         val *= F->Eval(Tr, ip);
+      }
+
+      elvect.Add(val, shape);
    }
 }
 

fem/lininteg.hpp

@@ -279,11 +279,12 @@ public:
 class VectorFEBoundaryFluxLFIntegrator : public LinearFormIntegrator
 {
 private:
-   Coefficient &F;
+   Coefficient *F;
    Vector shape;
 
 public:
-   VectorFEBoundaryFluxLFIntegrator(Coefficient &f) : F(f) { }
+   VectorFEBoundaryFluxLFIntegrator() : F(NULL) { }
+   VectorFEBoundaryFluxLFIntegrator(Coefficient &f) : F(&f) { }
 
    virtual void AssembleRHSElementVect(const FiniteElement &el,
                                        ElementTransformation &Tr,

fem/nonlinearform.cpp

@@ -65,6 +65,8 @@ double NonlinearForm::GetGridFunctionEnergy(const Vector &x) const
    Vector el_x;
    const FiniteElement *fe;
    ElementTransformation *T;
+   Mesh *mesh = fes->GetMesh();
+
    double energy = 0.0;
 
    if (dnfi.Size())
@@ -84,14 +86,81 @@ double NonlinearForm::GetGridFunctionEnergy(const Vector &x) const
 
    if (fnfi.Size())
    {
-      MFEM_ABORT("TODO: add energy contribution from interior face terms");
+      FaceElementTransformations *tr;
+      const FiniteElement *fe1, *fe2;
+      Array<int> vdofs2;
+
+      for (int i = 0; i < mesh->GetNumFaces(); i++)
+      {
+         tr = mesh->GetInteriorFaceTransformations(i);
+         if (tr != NULL)
+         {
+            fes->GetElementVDofs(tr->Elem1No, vdofs);
+            fes->GetElementVDofs(tr->Elem2No, vdofs2);
+            vdofs.Append (vdofs2);
+            x.GetSubVector(vdofs, el_x);
+            fe1 = fes->GetFE(tr->Elem1No);
+            fe2 = fes->GetFE(tr->Elem2No);
+            for (int k = 0; k < fnfi.Size(); k++)
+            {
+               energy += fnfi[k]->GetFaceEnergy(*fe1, *fe2, *tr, el_x);
+            }
+         }
+      }
    }
 
    if (bfnfi.Size())
    {
-      MFEM_ABORT("TODO: add energy contribution from boundary face terms");
-   }
+      FaceElementTransformations *tr;
+      const FiniteElement *fe1, *fe2;
 
+      // Which boundary attributes need to be processed?
+      Array<int> bdr_attr_marker(mesh->bdr_attributes.Size() ?
+                                 mesh->bdr_attributes.Max() : 0);
+      bdr_attr_marker = 0;
+      for (int k = 0; k < bfnfi.Size(); k++)
+      {
+         if (bfnfi_marker[k] == NULL)
+         {
+            bdr_attr_marker = 1;
+            break;
+         }
+         Array<int> &bdr_marker = *bfnfi_marker[k];
+         MFEM_ASSERT(bdr_marker.Size() == bdr_attr_marker.Size(),
+                     "invalid boundary marker for boundary face integrator #"
+                     << k << ", counting from zero");
+         for (int i = 0; i < bdr_attr_marker.Size(); i++)
+         {
+            bdr_attr_marker[i] |= bdr_marker[i];
+         }
+      }
+
+      for (int i = 0; i < fes -> GetNBE(); i++)
+      {
+         const int bdr_attr = mesh->GetBdrAttribute(i);
+         if (bdr_attr_marker[bdr_attr-1] == 0) { continue; }
+
+         tr = mesh->GetBdrFaceTransformations (i);
+         if (tr != NULL)
+         {
+            fes->GetElementVDofs(tr->Elem1No, vdofs);
+            x.GetSubVector(vdofs, el_x);
+
+            fe1 = fes->GetFE(tr->Elem1No);
+            // The fe2 object is really a dummy and not used on the boundaries,
+            // but we can't dereference a NULL pointer, and we don't want to
+            // actually make a fake element.
+            fe2 = fe1;
+            for (int k = 0; k < bfnfi.Size(); k++)
+            {
+               if (bfnfi_marker[k] &&
+                   (*bfnfi_marker[k])[bdr_attr-1] == 0) { continue; }
+
+               energy += bfnfi[k]->GetFaceEnergy(*fe1, *fe2, *tr, el_x);
+            }
+         }
+      }
+   }
    return energy;
 }
 

fem/nonlinearform.hpp

@@ -111,7 +111,7 @@ public:
        be fes->GetVSize(). */
    double GetGridFunctionEnergy(const Vector &x) const;
 
-   /// Compute the enery corresponding to the state @a x.
+   /// Compute the energy corresponding to the state @a x.
    /** In general, @a x may have non-homogeneous essential boundary values.
 
        The state @a x must be a true-dof vector. */

fem/nonlininteg.cpp

@@ -55,6 +55,14 @@ double NonlinearFormIntegrator::GetElementEnergy(
    return 0.0;
 }
 
+double NonlinearFormIntegrator::GetFaceEnergy(
+   const FiniteElement &el1, const FiniteElement &el2,
+   FaceElementTransformations &Tr, const Vector &elfun)
+{
+   mfem_error("NonlinearFormIntegrator::GetFaceEnergy"
+              " is not overloaded!");
+   return 0.0;
+}
 
 void BlockNonlinearFormIntegrator::AssembleElementVector(
    const Array<const FiniteElement *> &el,

fem/nonlininteg.hpp

@@ -63,11 +63,17 @@ public:
                                  FaceElementTransformations &Tr,
                                  const Vector &elfun, DenseMatrix &elmat);
 
-   /// Compute the local energy
+   /// Compute the local energy/functional
    virtual double GetElementEnergy(const FiniteElement &el,
                                    ElementTransformation &Tr,
                                    const Vector &elfun);
 
+   /// Compute the face(s) contribution to the energy/functional
+   virtual double GetFaceEnergy(const FiniteElement &el1,
+                                const FiniteElement &el2,
+                                FaceElementTransformations &Tr,
+                                const Vector &elfun);
+
    virtual ~NonlinearFormIntegrator() { }
 };
 

fem/occa.okl

@@ -38,19 +38,24 @@ typedef double* QLocal3D_t @dim(Q1D, Q1D, Q1D, NE);
 typedef double* Jacobian2D_t @dim(Q2D, 2, 2, NE);
 typedef double* Jacobian3D_t @dim(Q3D, 3, 3, NE);
 
+typedef double* Coeff2D_t @dim(Q2D, NE);
+typedef double* Coeff3D_t @dim(Q3D, NE);
+
 typedef double* SymmOperator2D_t @dim(Q2D, 3, NE);
 typedef double* SymmOperator3D_t @dim(Q3D, 6, NE);
 
 @kernel void DiffusionSetup2D(const int NE,
                               @restrict const double *W,
                               @restrict const Jacobian2D_t J,
-                              const double COEFF,
-                              @restrict SymmOperator2D_t op) {
+                              @restrict const Coeff2D_t C,
+                              @restrict SymmOperator2D_t op,
+                              const bool const_c) {
   for (int e = 0; e < NE; ++e; @outer) {
     for (int q = 0; q < Q2D; ++q; @inner) {
       const double J11 = J(q, 0, 0, e), J12 = J(q, 1, 0, e);
       const double J21 = J(q, 0, 1, e), J22 = J(q, 1, 1, e);
-      const double c_detJ = W[q] * COEFF / ((J11 * J22) - (J21 * J12));
+      const double coeff = const_c ? C(0,0) : C(q,e);
+      const double c_detJ = W[q] * coeff / ((J11 * J22) - (J21 * J12));
       op(q, 0, e) =  c_detJ * (J21*J21 + J22*J22); // (1,1)
       op(q, 1, e) = -c_detJ * (J21*J11 + J22*J12); // (1,2), (2,1)
       op(q, 2, e) =  c_detJ * (J11*J11 + J12*J12); // (2,2)
@@ -61,8 +66,9 @@ typedef double* SymmOperator3D_t @dim(Q3D, 6, NE);
 @kernel void DiffusionSetup3D(const int NE,
                               @restrict const double *W,
                               @restrict const Jacobian3D_t J,
-                              const double COEFF,
-                              @restrict SymmOperator3D_t op) {
+                              @restrict const Coeff3D_t C,
+                              @restrict SymmOperator3D_t op,
+                              const bool const_c) {
   for (int e = 0; e < NE; ++e; @outer) {
     for (int q = 0; q < Q3D; ++q; @inner) {
       const double J11 = J(q, 0, 0, e), J12 = J(q, 1, 0, e), J13 = J(q, 2, 0, e);
@@ -72,7 +78,8 @@ typedef double* SymmOperator3D_t @dim(Q3D, 6, NE);
       const double detJ = ((J11 * J22 * J33) + (J12 * J23 * J31) + (J13 * J21 * J32) -
                            (J13 * J22 * J31) - (J12 * J21 * J33) - (J11 * J23 * J32));
 
-      const double c_detJ = W[q] * COEFF / detJ;
+      const double coeff = const_c ? C(0,0) : C(q,e);
+      const double c_detJ = W[q] * coeff / detJ;
 
       // adj(J)
       const double A11 = (J22 * J33) - (J23 * J32);

fem/pfespace.cpp

@@ -14,6 +14,7 @@
 #ifdef MFEM_USE_MPI
 
 #include "pfespace.hpp"
+#include "../general/forall.hpp"
 #include "../general/sort_pairs.hpp"
 #include "../mesh/mesh_headers.hpp"
 #include "../general/binaryio.hpp"
@@ -97,6 +98,8 @@ void ParFiniteElementSpace::ParInit(ParMesh *pm)
 
    gcomm = NULL;
 
+   gfdofs = NULL;
+
    P = NULL;
    Pconf = NULL;
    R = NULL;
@@ -147,20 +150,37 @@ void ParFiniteElementSpace::Construct()
       // cut space.
       ConstructTrueDofs();
 
+      ngedofs = ngfdofs = 0;
+      gfdofs = NULL;
+
       // calculate number of ghost DOFs
       ngvdofs = pncmesh->GetNGhostVertices()
                 * fec->DofForGeometry(Geometry::POINT);
 
-      ngedofs = ngfdofs = 0;
       if (pmesh->Dimension() > 1)
       {
          ngedofs = pncmesh->GetNGhostEdges()
                    * fec->DofForGeometry(Geometry::SEGMENT);
       }
+
       if (pmesh->Dimension() > 2)
       {
-         ngfdofs = pncmesh->GetNGhostFaces()
-                   * fec->DofForGeometry(pncmesh->GetGhostFaceGeometry(0));
+         if (fdofs != NULL) // have mixed faces
+         {
+            gfdofs = new int[pncmesh->GetNGhostFaces()+1];
+            gfdofs[0] = 0;
+            for (int i = 0; i < pncmesh->GetNGhostFaces(); i++)
+            {
+               int ghost = pncmesh->GetNFaces() + i;
+               ngfdofs += fec->DofForGeometry(pncmesh->GetFaceGeometry(ghost));
+               gfdofs[i+1] = ngfdofs;
+            }
+         }
+         else
+         {
+            ngfdofs = pncmesh->GetNGhostFaces()
+                      * fec->DofForGeometry(pncmesh->GetFaceGeometry(0));
+         }
       }
 
       // total number of ghost DOFs. Ghost DOFs start at index 'ndofs', i.e.,
@@ -856,7 +876,20 @@ const Operator *ParFiniteElementSpace::GetProlongationMatrix() const
 {
    if (Conforming())
    {
-      if (!Pconf) { Pconf = new ConformingProlongationOperator(*this); }
+      if (!Pconf)
+      {
+         if (!Device::Allows(Backend::DEVICE_MASK))
+         {
+            Pconf = new ConformingProlongationOperator(*this);
+         }
+         else
+         {
+            if (NRanks > 1)
+            {
+               Pconf = new DeviceConformingProlongationOperator(*this);
+            }
+         }
+      }
       return Pconf;
    }
    else
@@ -1310,20 +1343,18 @@ void ParFiniteElementSpace::GetGhostEdgeDofs(const MeshId &edge_id,
 void ParFiniteElementSpace::GetGhostFaceDofs(const MeshId &face_id,
                                              Array<int> &dofs) const
 {
-   const int ghost_face_index = face_id.index - pncmesh->GetNFaces();
-   MFEM_ASSERT(pncmesh->GetGhostFaceGeometry(ghost_face_index)
-               == Geometry::SQUARE, "");
+   int nfv, V[4], E[4], Eo[4];
+   nfv = pmesh->pncmesh->GetFaceVerticesEdges(face_id, V, E, Eo);
 
    int nv = fec->DofForGeometry(Geometry::POINT);
    int ne = fec->DofForGeometry(Geometry::SEGMENT);
-   int nf = fec->DofForGeometry(Geometry::SQUARE);
-   dofs.SetSize(4*nv + 4*ne + nf);
+   int nf = fec->DofForGeometry((nfv == 3) ?
+                                Geometry::TRIANGLE : Geometry::SQUARE);
 
-   int V[4], E[4], Eo[4];
-   pmesh->pncmesh->GetFaceVerticesEdges(face_id, V, E, Eo);
+   dofs.SetSize(nfv*(nv + ne) + nf);
 
    int offset = 0;
-   for (int i = 0; i < 4; i++)
+   for (int i = 0; i < nfv; i++)
    {
       int ghost = pncmesh->GetNVertices();
       int first = (V[i] < ghost) ? V[i]*nv : (ndofs + (V[i] - ghost)*nv);
@@ -1333,7 +1364,7 @@ void ParFiniteElementSpace::GetGhostFaceDofs(const MeshId &face_id,
       }
    }
 
-   for (int i = 0; i < 4; i++)
+   for (int i = 0; i < nfv; i++)
    {
       int ghost = pncmesh->GetNEdges();
       int first = (E[i] < ghost) ? nvdofs + E[i]*ne
@@ -1346,8 +1377,10 @@ void ParFiniteElementSpace::GetGhostFaceDofs(const MeshId &face_id,
       }
    }
 
-   // Assuming all ghost faces have the same number of dofs:
-   int first = ndofs + ngvdofs + ngedofs + ghost_face_index*nf;
+   const int ghost_face_index = face_id.index - pncmesh->GetNFaces();
+   int first = ndofs + ngvdofs + ngedofs;
+   first += gfdofs ? gfdofs[ghost_face_index] : nf*ghost_face_index;
+
    for (int j = 0; j < nf; j++)
    {
       dofs[offset++] = first + j;
@@ -1389,12 +1422,19 @@ void ParFiniteElementSpace::GetBareDofs(int entity, int index,
          break;
 
       default:
-         MFEM_ASSERT(!pmesh->HasGeometry(Geometry::TRIANGLE), "");
-         ned = fec->DofForGeometry(Geometry::SQUARE);
+         ned = fec->DofForGeometry(pncmesh->GetFaceGeometry(index));
          ghost = pncmesh->GetNFaces();
-         first = (index < ghost)
-                 ? nvdofs + nedofs + index*ned // regular face
-                 : ndofs + ngvdofs + ngedofs + (index - ghost)*ned; // ghost
+
+         if (index < ghost) // regular face
+         {
+            first = nvdofs + nedofs + (fdofs ? fdofs[index] : index*ned);
+         }
+         else // ghost face
+         {
+            index -= ghost;
+            first = ndofs + ngvdofs + ngedofs +
+                    (gfdofs ? gfdofs[index] : index*ned);
+         }
          break;
    }
 
@@ -1430,16 +1470,30 @@ int ParFiniteElementSpace::PackDof(int entity, int index, int edof) const
                 : ndofs + ngvdofs + (index - ghost)*ned + edof; // ghost edge
 
       default:
-         MFEM_ASSERT(!pmesh->HasGeometry(Geometry::TRIANGLE), "");
          ghost = pncmesh->GetNFaces();
-         ned = fec->DofForGeometry(Geometry::SQUARE);
+         ned = fec->DofForGeometry(pncmesh->GetFaceGeometry(index));
 
-         return (index < ghost)
-                ? nvdofs + nedofs + index*ned + edof // regular face
-                : ndofs + ngvdofs + ngedofs + (index - ghost)*ned + edof; //ghost
+         if (index < ghost) // regular face
+         {
+            return nvdofs + nedofs + (fdofs ? fdofs[index] : index*ned) + edof;
+         }
+         else // ghost face
+         {
+            index -= ghost;
+            return ndofs + ngvdofs + ngedofs +
+                   (gfdofs ? gfdofs[index] : index*ned) + edof;
+         }
    }
 }
 
+static int bisect(int* array, int size, int value)
+{
+   int* end = array + size;
+   int* pos = std::upper_bound(array, end, value);
+   MFEM_VERIFY(pos != end, "value not found");
+   return pos - array;
+}
+
 /** Dissect a DOF number to obtain the entity type (0=vertex, 1=edge, 2=face),
  *  entity index and the DOF number within the entity.
  */
@@ -1465,9 +1519,17 @@ void ParFiniteElementSpace::UnpackDof(int dof,
       dof -= nedofs;
       if (dof < nfdofs) // regular face
       {
-         MFEM_ASSERT(!pmesh->HasGeometry(Geometry::TRIANGLE), "");
-         int nf = fec->DofForGeometry(Geometry::SQUARE);
-         entity = 2, index = dof / nf, edof = dof % nf;
+         if (fdofs) // have mixed faces
+         {
+            index = bisect(fdofs+1, mesh->GetNFaces(), dof);
+            edof = dof - fdofs[index];
+         }
+         else // uniform faces
+         {
+            int nf = fec->DofForGeometry(pncmesh->GetFaceGeometry(0));
+            index = dof / nf, edof = dof % nf;
+         }
+         entity = 2;
          return;
       }
       MFEM_ABORT("Cannot unpack internal DOF");
@@ -1491,8 +1553,17 @@ void ParFiniteElementSpace::UnpackDof(int dof,
       dof -= ngedofs;
       if (dof < ngfdofs) // ghost face
       {
-         int nf = fec->DofForGeometry(pncmesh->GetGhostFaceGeometry(0));
-         entity = 2, index = pncmesh->GetNFaces() + dof / nf, edof = dof % nf;
+         if (gfdofs) // have mixed faces
+         {
+            index = bisect(gfdofs+1, pncmesh->GetNGhostFaces(), dof);
+            edof = dof - gfdofs[index];
+         }
+         else // uniform faces
+         {
+            int nf = fec->DofForGeometry(pncmesh->GetFaceGeometry(0));
+            index = pncmesh->GetNFaces() + dof / nf, edof = dof % nf;
+         }
+         entity = 2;
          return;
       }
       MFEM_ABORT("Out of range DOF.");
@@ -1675,7 +1746,7 @@ void NeighborRowMessage::Encode(int rank)
          mfem::out << "Rank " << pncmesh->MyRank << " sending to " << rank
                    << ": ent " << ri.entity << ", index " << ri.index
                    << ", edof " << ri.edof << " (id " << id.element << "/"
-                   << id.local << ")" << std::endl;
+                   << int(id.local) << ")" << std::endl;
 #endif
 
          // handle orientation and sign change
@@ -1718,8 +1789,6 @@ void NeighborRowMessage::Decode(int rank)
    rows.clear();
    rows.reserve(nrows);
 
-   Geometry::Type fgeom = pncmesh->GetFaceGeometry();
-
    // read rows
    for (int ent = 0, gi = 0; ent < 3; ent++)
    {
@@ -1738,8 +1807,9 @@ void NeighborRowMessage::Decode(int rank)
          }
          else if (ent == 2)
          {
+            Geometry::Type geom = pncmesh->GetFaceGeometry(id.index);
             int fo = pncmesh->GetFaceOrientation(id.index);
-            ind = fec->DofOrderForOrientation(fgeom, fo);
+            ind = fec->DofOrderForOrientation(geom, fo);
          }
 
          double s = 1.0;
@@ -1828,7 +1898,7 @@ void ParFiniteElementSpace
    for (int i = 0; i < dof_group.Size(); i++)
    {
       os << i << ": ";
-      if (i < (nvdofs + nedofs + nfdofs) || i > ndofs)
+      if (i < (nvdofs + nedofs + nfdofs) || i >= ndofs)
       {
          int ent, idx, edof;
          UnpackDof(i, ent, idx, edof);
@@ -1910,15 +1980,7 @@ int ParFiniteElementSpace
          if (!list.masters.size()) { continue; }
 
          IsoparametricTransformation T;
-         if (entity > 1) { T.SetFE(&QuadrilateralFE); }
-         else { T.SetFE(&SegmentFE); }
-
-         Geometry::Type geom = (entity > 1) ?
-                               Geometry::SQUARE : Geometry::SEGMENT;
-         const FiniteElement* fe = fec->FiniteElementForGeometry(geom);
-         if (!fe) { continue; }
-
-         DenseMatrix I(fe->GetDof());
+         DenseMatrix I;
 
          // process masters that we own or that affect our edges/faces
          for (unsigned mi = 0; mi < list.masters.size(); mi++)
@@ -1932,6 +1994,17 @@ int ParFiniteElementSpace
 
             if (!master_dofs.Size()) { continue; }
 
+            const FiniteElement* fe = fec->FiniteElementForGeometry(mf.Geom());
+            if (!fe) { continue; }
+
+            switch (mf.Geom())
+            {
+               case Geometry::SQUARE:   T.SetFE(&QuadrilateralFE); break;
+               case Geometry::TRIANGLE: T.SetFE(&TriangleFE); break;
+               case Geometry::SEGMENT:  T.SetFE(&SegmentFE); break;
+               default: MFEM_ABORT("unsupported geometry");
+            }
+
             // constrain slaves that exist in our mesh
             for (int si = mf.slaves_begin; si < mf.slaves_end; si++)
             {
@@ -1982,6 +2055,8 @@ int ParFiniteElementSpace
                   (l == 1) ? (const MeshId&) list.masters[i]
                   /*    */ : (const MeshId&) list.slaves[i];
 
+               if (id.index < 0) { continue; }
+
                GroupId owner = pncmesh->GetEntityOwnerId(entity, id.index);
                GroupId group = pncmesh->GetEntityGroupId(entity, id.index);
 
@@ -2478,6 +2553,9 @@ ParFiniteElementSpace::ParallelDerefinementMatrix(int old_ndofs,
    int nrk = HYPRE_AssumedPartitionCheck() ? 2 : NRanks;
 
    MFEM_VERIFY(Nonconforming(), "Not implemented for conforming meshes.");
+   MFEM_VERIFY(pmesh->GetNumGeometries(pmesh->Dimension()) == 1,
+               "Not implemented for mixed meshes.");
+
    MFEM_VERIFY(old_dof_offsets[nrk], "Missing previous (finer) space.");
    MFEM_VERIFY(dof_offsets[nrk] <= old_dof_offsets[nrk],
                "Previous space is not finer.");
@@ -2491,7 +2569,7 @@ ParFiniteElementSpace::ParallelDerefinementMatrix(int old_ndofs,
    Vector row;
 
    ParNCMesh* pncmesh = pmesh->pncmesh;
-   Geometry::Type geom = pncmesh->GetElementGeometry();
+   Geometry::Type geom = pncmesh->GetElementGeometry(0); // TODO mixed meshes
    int ldof = fec->FiniteElementForGeometry(geom)->GetDof();
 
    const CoarseFineTransformations &dtrans = pncmesh->GetDerefinementTransforms();
@@ -2715,6 +2793,8 @@ void ParFiniteElementSpace::Destroy()
    delete Pconf; Pconf = NULL;
    delete R; R = NULL;
 
+   delete [] gfdofs; gfdofs = NULL;
+
    delete gcomm; gcomm = NULL;
 
    num_face_nbr_dofs = -1;
@@ -2932,6 +3012,242 @@ void ConformingProlongationOperator::MultTranspose(
    gc.ReduceEnd<double>(ydata, out_layout, GroupCommunicator::Sum);
 }
 
+DeviceConformingProlongationOperator::DeviceConformingProlongationOperator(
+   const ParFiniteElementSpace &pfes) :
+   ConformingProlongationOperator(pfes),
+   mpi_gpu_aware(Device::GetGPUAwareMPI())
+{
+   MFEM_ASSERT(pfes.Conforming(), "internal error");
+   const SparseMatrix *R = pfes.GetRestrictionMatrix();
+   MFEM_ASSERT(R->Finalized(), "");
+   const int tdofs = R->Height();
+   MFEM_ASSERT(tdofs == pfes.GetTrueVSize(), "");
+   MFEM_ASSERT(tdofs == R->GetI()[tdofs], "");
+   ltdof_ldof = Array<int>(const_cast<int*>(R->GetJ()), tdofs);
+   ltdof_ldof.UseDevice();
+   {
+      Table nbr_ltdof;
+      gc.GetNeighborLTDofTable(nbr_ltdof);
+      const int nb_connections = nbr_ltdof.Size_of_connections();
+      shr_ltdof.SetSize(nb_connections);
+      shr_ltdof.CopyFrom(nbr_ltdof.GetJ());
+      shr_buf.SetSize(nb_connections);
+      shr_buf.UseDevice(true);
+      shr_buf_offsets = nbr_ltdof.GetI();
+      {
+         Array<int> shr_ltdof(nbr_ltdof.GetJ(), nb_connections);
+         Array<int> unique_ltdof(shr_ltdof);
+         unique_ltdof.Sort();
+         unique_ltdof.Unique();
+         // Note: the next loop modifies the J array of nbr_ltdof
+         for (int i = 0; i < shr_ltdof.Size(); i++)
+         {
+            shr_ltdof[i] = unique_ltdof.FindSorted(shr_ltdof[i]);
+            MFEM_ASSERT(shr_ltdof[i] != -1, "internal error");
+         }
+         Table unique_shr;
+         Transpose(shr_ltdof, unique_shr, unique_ltdof.Size());
+         unq_ltdof = Array<int>(unique_ltdof, unique_ltdof.Size());
+         unq_shr_i = Array<int>(unique_shr.GetI(), unique_shr.Size()+1);
+         unq_shr_j = Array<int>(unique_shr.GetJ(), unique_shr.Size_of_connections());
+      }
+      delete [] nbr_ltdof.GetJ();
+      nbr_ltdof.LoseData();
+   }
+   {
+      Table nbr_ldof;
+      gc.GetNeighborLDofTable(nbr_ldof);
+      const int nb_connections = nbr_ldof.Size_of_connections();
+      ext_ldof.SetSize(nb_connections);
+      ext_ldof.CopyFrom(nbr_ldof.GetJ());
+      ext_buf.SetSize(nb_connections);
+      ext_buf.UseDevice(true);
+      ext_buf_offsets = nbr_ldof.GetI();
+      delete [] nbr_ldof.GetJ();
+      nbr_ldof.LoseData();
+   }
+   const GroupTopology &gtopo = gc.GetGroupTopology();
+   int req_counter = 0;
+   for (int nbr = 1; nbr < gtopo.GetNumNeighbors(); nbr++)
+   {
+      const int send_offset = shr_buf_offsets[nbr];
+      const int send_size = shr_buf_offsets[nbr+1] - send_offset;
+      if (send_size > 0) { req_counter++; }
+
+      const int recv_offset = ext_buf_offsets[nbr];
+      const int recv_size = ext_buf_offsets[nbr+1] - recv_offset;
+      if (recv_size > 0) { req_counter++; }
+   }
+   requests = new MPI_Request[req_counter];
+}
+
+static void ExtractSubVector(const int N,
+                             const Array<int> &indices,
+                             const Vector &in, Vector &out)
+{
+   auto y = out.Write();
+   const auto x = in.Read();
+   const auto I = indices.Read();
+   MFEM_FORALL(i, N, y[i] = x[I[i]];); // indices can be repeated
+}
+
+void DeviceConformingProlongationOperator::BcastBeginCopy(
+   const Vector &x) const
+{
+   // shr_buf[i] = src[shr_ltdof[i]]
+   if (shr_ltdof.Size() == 0) { return; }
+   ExtractSubVector(shr_ltdof.Size(), shr_ltdof, x, shr_buf);
+   // If the above kernel is executed asynchronously, we should wait for it to
+   // complete
+   if (mpi_gpu_aware) { Device::Synchronize(); }
+}
+
+static void SetSubVector(const int N,
+                         const Array<int> &indices,
+                         const Vector &in, Vector &out)
+{
+   auto y = out.Write();
+   const auto x = in.Read();
+   const auto I = indices.Read();
+   MFEM_FORALL(i, N, y[I[i]] = x[i];);
+}
+
+void DeviceConformingProlongationOperator::BcastLocalCopy(
+   const Vector &x, Vector &y) const
+{
+   // dst[ltdof_ldof[i]] = src[i]
+   if (ltdof_ldof.Size() == 0) { return; }
+   SetSubVector(ltdof_ldof.Size(), ltdof_ldof, x, y);
+}
+
+void DeviceConformingProlongationOperator::BcastEndCopy(
+   Vector &y) const
+{
+   // dst[ext_ldof[i]] = ext_buf[i]
+   if (ext_ldof.Size() == 0) { return; }
+   SetSubVector(ext_ldof.Size(), ext_ldof, ext_buf, y);
+}
+
+void DeviceConformingProlongationOperator::Mult(const Vector &x,
+                                                Vector &y) const
+{
+   const GroupTopology &gtopo = gc.GetGroupTopology();
+   BcastBeginCopy(x); // copy to 'shr_buf'
+   int req_counter = 0;
+   for (int nbr = 1; nbr < gtopo.GetNumNeighbors(); nbr++)
+   {
+      const int send_offset = shr_buf_offsets[nbr];
+      const int send_size = shr_buf_offsets[nbr+1] - send_offset;
+      if (send_size > 0)
+      {
+         auto send_buf = mpi_gpu_aware ? shr_buf.Read() : shr_buf.HostRead();
+         MPI_Isend(send_buf + send_offset, send_size, MPI_DOUBLE,
+                   gtopo.GetNeighborRank(nbr), 41822,
+                   gtopo.GetComm(), &requests[req_counter++]);
+      }
+      const int recv_offset = ext_buf_offsets[nbr];
+      const int recv_size = ext_buf_offsets[nbr+1] - recv_offset;
+      if (recv_size > 0)
+      {
+         auto recv_buf = mpi_gpu_aware ? ext_buf.Write() : ext_buf.HostWrite();
+         MPI_Irecv(recv_buf + recv_offset, recv_size, MPI_DOUBLE,
+                   gtopo.GetNeighborRank(nbr), 41822,
+                   gtopo.GetComm(), &requests[req_counter++]);
+      }
+   }
+   BcastLocalCopy(x, y);
+   MPI_Waitall(req_counter, requests, MPI_STATUSES_IGNORE);
+   BcastEndCopy(y); // copy from 'ext_buf'
+}
+
+DeviceConformingProlongationOperator::~DeviceConformingProlongationOperator()
+{
+   delete [] requests;
+   delete [] ext_buf_offsets;
+   delete [] shr_buf_offsets;
+}
+
+void DeviceConformingProlongationOperator::ReduceBeginCopy(
+   const Vector &x) const
+{
+   // ext_buf[i] = src[ext_ldof[i]]
+   if (ext_ldof.Size() == 0) { return; }
+   ExtractSubVector(ext_ldof.Size(), ext_ldof, x, ext_buf);
+   // If the above kernel is executed asynchronously, we should wait for it to
+   // complete
+   if (mpi_gpu_aware) { Device::Synchronize(); }
+}
+
+void DeviceConformingProlongationOperator::ReduceLocalCopy(
+   const Vector &x, Vector &y) const
+{
+   // dst[i] = src[ltdof_ldof[i]]
+   if (ltdof_ldof.Size() == 0) { return; }
+   ExtractSubVector(ltdof_ldof.Size(), ltdof_ldof, x, y);
+}
+
+static void AddSubVector(const int num_unique_dst_indices,
+                         const Array<int> &unique_dst_indices,
+                         const Array<int> &unique_to_src_offsets,
+                         const Array<int> &unique_to_src_indices,
+                         const Vector &src,
+                         Vector &dst)
+{
+   auto y = dst.Write();
+   const auto x = src.Read();
+   const auto DST_I = unique_dst_indices.Read();
+   const auto SRC_O = unique_to_src_offsets.Read();
+   const auto SRC_I = unique_to_src_indices.Read();
+   MFEM_FORALL(i, num_unique_dst_indices,
+   {
+      const int dst_idx = DST_I[i];
+      double sum = y[dst_idx];
+      const int end = SRC_O[i+1];
+      for (int j = SRC_O[i]; j != end; ++j) { sum += x[SRC_I[j]]; }
+      y[dst_idx] = sum;
+   });
+}
+
+void DeviceConformingProlongationOperator::ReduceEndAssemble(Vector &y) const
+{
+   // dst[shr_ltdof[i]] += shr_buf[i]
+   const int unq_ltdof_size = unq_ltdof.Size();
+   if (unq_ltdof_size == 0) { return; }
+   AddSubVector(unq_ltdof_size, unq_ltdof, unq_shr_i, unq_shr_j, shr_buf, y);
+}
+
+void DeviceConformingProlongationOperator::MultTranspose(const Vector &x,
+                                                         Vector &y) const
+{
+   const GroupTopology &gtopo = gc.GetGroupTopology();
+   ReduceBeginCopy(x); // copy to 'ext_buf'
+   int req_counter = 0;
+   for (int nbr = 1; nbr < gtopo.GetNumNeighbors(); nbr++)
+   {
+      const int send_offset = ext_buf_offsets[nbr];
+      const int send_size = ext_buf_offsets[nbr+1] - send_offset;
+      if (send_size > 0)
+      {
+         auto send_buf = mpi_gpu_aware ? ext_buf.Read() : ext_buf.HostRead();
+         MPI_Isend(send_buf + send_offset, send_size, MPI_DOUBLE,
+                   gtopo.GetNeighborRank(nbr), 41823,
+                   gtopo.GetComm(), &requests[req_counter++]);
+      }
+      const int recv_offset = shr_buf_offsets[nbr];
+      const int recv_size = shr_buf_offsets[nbr+1] - recv_offset;
+      if (recv_size > 0)
+      {
+         auto recv_buf = mpi_gpu_aware ? shr_buf.Write() : shr_buf.HostWrite();
+         MPI_Irecv(recv_buf + recv_offset, recv_size, MPI_DOUBLE,
+                   gtopo.GetNeighborRank(nbr), 41823,
+                   gtopo.GetComm(), &requests[req_counter++]);
+      }
+   }
+   ReduceLocalCopy(x, y);
+   MPI_Waitall(req_counter, requests, MPI_STATUSES_IGNORE);
+   ReduceEndAssemble(y); // assemble from 'shr_buf'
+}
+
 } // namespace mfem
 
 #endif

fem/pfespace.hpp

@@ -46,6 +46,7 @@ private:
 
    /// Number of vertex/edge/face/total ghost DOFs (nonconforming case).
    int ngvdofs, ngedofs, ngfdofs, ngdofs;
+   int* gfdofs;
 
    /// The group of each local dof.
    Array<int> ldof_group;
@@ -113,7 +114,7 @@ private:
    void GetGhostFaceDofs(const MeshId &face_id, Array<int> &dofs) const;
 
    void GetGhostDofs(int entity, const MeshId &id, Array<int> &dofs) const;
-   // Return the dofs associated with the interior of the given mesh entity.
+   /// Return the dofs associated with the interior of the given mesh entity.
    void GetBareDofs(int entity, int index, Array<int> &dofs) const;
 
    int  PackDof(int entity, int index, int edof) const;
@@ -387,6 +388,52 @@ public:
    virtual void MultTranspose(const Vector &x, Vector &y) const;
 };
 
+/// Auxiliary device class used by ParFiniteElementSpace.
+class DeviceConformingProlongationOperator: public
+   ConformingProlongationOperator
+{
+protected:
+   bool mpi_gpu_aware;
+   Array<int> shr_ltdof, ext_ldof;
+   mutable Vector shr_buf, ext_buf;
+   int *shr_buf_offsets, *ext_buf_offsets;
+   Array<int> ltdof_ldof, unq_ltdof;
+   Array<int> unq_shr_i, unq_shr_j;
+   MPI_Request *requests;
+   // Kernel: copy ltdofs from 'src' to 'shr_buf' - prepare for send.
+   //         shr_buf[i] = src[shr_ltdof[i]]
+   void BcastBeginCopy(const Vector &src) const;
+
+   // Kernel: copy ltdofs from 'src' to ldofs in 'dst'.
+   //         dst[ltdof_ldof[i]] = src[i]
+   void BcastLocalCopy(const Vector &src, Vector &dst) const;
+
+   // Kernel: copy ext. dofs from 'ext_buf' to 'dst' - after recv.
+   //         dst[ext_ldof[i]] = ext_buf[i]
+   void BcastEndCopy(Vector &dst) const;
+
+   // Kernel: copy ext. dofs from 'src' to 'ext_buf' - prepare for send.
+   //         ext_buf[i] = src[ext_ldof[i]]
+   void ReduceBeginCopy(const Vector &src) const;
+
+   // Kernel: copy owned ldofs from 'src' to ltdofs in 'dst'.
+   //         dst[i] = src[ltdof_ldof[i]]
+   void ReduceLocalCopy(const Vector &src, Vector &dst) const;
+
+   // Kernel: assemble dofs from 'shr_buf' into to 'dst' - after recv.
+   //         dst[shr_ltdof[i]] += shr_buf[i]
+   void ReduceEndAssemble(Vector &dst) const;
+
+public:
+   DeviceConformingProlongationOperator(const ParFiniteElementSpace &pfes);
+
+   virtual ~DeviceConformingProlongationOperator();
+
+   virtual void Mult(const Vector &x, Vector &y) const;
+
+   virtual void MultTranspose(const Vector &x, Vector &y) const;
+};
+
 }
 
 #endif // MFEM_USE_MPI

fem/pgridfunc.cpp

@@ -225,11 +225,13 @@ void ParGridFunction::ExchangeFaceNbrData()
    MPI_Request *recv_requests = requests + num_face_nbrs;
    MPI_Status  *statuses = new MPI_Status[num_face_nbrs];
 
+   const double *h_data = this->HostRead();
    for (int i = 0; i < send_data.Size(); i++)
    {
-      send_data[i] = data[send_ldof[i]];
+      send_data[i] = h_data[send_ldof[i]];
    }
 
+   double *h_face_nbr_data = face_nbr_data.HostWrite();
    for (int fn = 0; fn < num_face_nbrs; fn++)
    {
       int nbr_rank = pmesh->GetFaceNbrRank(fn);
@@ -239,7 +241,7 @@ void ParGridFunction::ExchangeFaceNbrData()
                 send_offset[fn+1] - send_offset[fn],
                 MPI_DOUBLE, nbr_rank, tag, MyComm, &send_requests[fn]);
 
-      MPI_Irecv(&face_nbr_data(recv_offset[fn]),
+      MPI_Irecv(&h_face_nbr_data[recv_offset[fn]],
                 recv_offset[fn+1] - recv_offset[fn],
                 MPI_DOUBLE, nbr_rank, tag, MyComm, &recv_requests[fn]);
    }

fem/pgridfunc.hpp

@@ -112,6 +112,8 @@ public:
    /// Associate a new parallel space with the ParGridFunction.
    void SetSpace(ParFiniteElementSpace *f);
 
+   using GridFunction::MakeRef;
+
    /** @brief Make the ParGridFunction reference external data on a new
        FiniteElementSpace. */
    /** This method changes the FiniteElementSpace associated with the

fem/pnonlinearform.cpp

@@ -46,7 +46,7 @@ double ParNonlinearForm::GetParGridFunctionEnergy(const Vector &x) const
 void ParNonlinearForm::Mult(const Vector &x, Vector &y) const
 {
    NonlinearForm::Mult(x, y); // x --(P)--> aux1 --(A_local)--> aux2
-   Y.SetData(aux2.GetData()); // aux2 contains A_local.P.x
+   Y.MakeRef(aux2, 0); // aux2 contains A_local.P.x
 
    if (fnfi.Size())
    {
@@ -58,7 +58,7 @@ void ParNonlinearForm::Mult(const Vector &x, Vector &y) const
       Array<int> vdofs1, vdofs2;
       Vector el_x, el_y;
 
-      X.SetData(aux1.GetData()); // aux1 contains P.x
+      X.MakeRef(aux1, 0); // aux1 contains P.x
       X.ExchangeFaceNbrData();
       const int n_shared_faces = pmesh->GetNSharedFaces();
       for (int i = 0; i < n_shared_faces; i++)

fem/sidredatacollection.cpp

@@ -16,9 +16,7 @@
 
 #include "fem.hpp"
 
-#ifdef MFEM_USE_MPI
-#include <sidre/IOManager.hpp>
-#endif
+#include <axom/sidre.hpp>
 
 #include <string>
 #include <iomanip>      // for setw, setfill
@@ -204,10 +202,10 @@ SidreDataCollection::get_file_path(const std::string &filename) const
 
 axom::sidre::View *
 SidreDataCollection::AllocNamedBuffer(const std::string& buffer_name,
-                                      axom::sidre::SidreLength sz,
+                                      axom::sidre::IndexType sz,
                                       axom::sidre::TypeID type)
 {
-   sz = std::max(sz, sidre::SidreLength(0));
+   sz = std::max(sz, sidre::IndexType(0));
    sidre::Group *f = named_buffers_grp();
    sidre::View  *v = NULL;
 
@@ -825,7 +823,7 @@ void SidreDataCollection::Save(const std::string& filename,
 void SidreDataCollection::
 addScalarBasedGridFunction(const std::string &field_name, GridFunction *gf,
                            const std::string &buffer_name,
-                           axom::sidre::SidreLength offset)
+                           axom::sidre::IndexType offset)
 {
    sidre::Group* grp = m_bp_grp->getGroup("fields/" + field_name);
    MFEM_ASSERT(grp != NULL, "field " << field_name << " does not exist");
@@ -888,7 +886,7 @@ addScalarBasedGridFunction(const std::string &field_name, GridFunction *gf,
 void SidreDataCollection::
 addVectorBasedGridFunction(const std::string& field_name, GridFunction *gf,
                            const std::string &buffer_name,
-                           axom::sidre::SidreLength offset)
+                           axom::sidre::IndexType offset)
 {
    sidre::Group* grp = m_bp_grp->getGroup("fields/" + field_name);
    MFEM_ASSERT(grp != NULL, "field " << field_name << " does not exist");
@@ -1013,7 +1011,7 @@ DeregisterFieldInBPIndex(const std::string& field_name)
 void SidreDataCollection::RegisterField(const std::string &field_name,
                                         GridFunction *gf,
                                         const std::string &buffer_name,
-                                        axom::sidre::SidreLength offset)
+                                        axom::sidre::IndexType offset)
 {
    if ( field_name.empty() || buffer_name.empty() ||
         gf == NULL || gf->FESpace() == NULL )

fem/sidredatacollection.hpp

@@ -25,7 +25,7 @@
 #  pragma GCC diagnostic ignored "-Wpedantic"
 # endif
 #endif
-#include <sidre/sidre.hpp>
+#include <axom/sidre.hpp>
 #ifdef MFEM_HAVE_GCC_PRAGMA_DIAGNOSTIC
 # pragma GCC diagnostic pop
 #endif
@@ -246,7 +246,7 @@ public:
     */
    void RegisterField(const std::string &field_name, GridFunction *gf,
                       const std::string &buffer_name,
-                      axom::sidre::SidreLength offset);
+                      axom::sidre::IndexType offset);
 
    /// Registers an attribute field in the Sidre DataStore
    /** The registration process is similar to that of RegisterField()
@@ -385,7 +385,7 @@ public:
     */
    axom::sidre::View *
    AllocNamedBuffer(const std::string& buffer_name,
-                    axom::sidre::SidreLength sz,
+                    axom::sidre::IndexType sz,
                     axom::sidre::TypeID type =
                        axom::sidre::DOUBLE_ID);
 
@@ -469,7 +469,7 @@ private:
    void addScalarBasedGridFunction(const std::string& field_name,
                                    GridFunction* gf,
                                    const std::string &buffer_name,
-                                   axom::sidre::SidreLength offset);
+                                   axom::sidre::IndexType offset);
 
    /**
     * \brief A private helper function to set up the views associated with the
@@ -483,7 +483,7 @@ private:
    void addVectorBasedGridFunction(const std::string& field_name,
                                    GridFunction* gf,
                                    const std::string &buffer_name,
-                                   axom::sidre::SidreLength offset);
+                                   axom::sidre::IndexType offset);
 
    /** @brief A private helper function to set up the Views associated with
        attribute field named @a field_name */

fem/tmop.cpp

@@ -12,6 +12,7 @@
 #include "tmop.hpp"
 #include "linearform.hpp"
 #include "pgridfunc.hpp"
+#include "tmop_tools.hpp"
 
 namespace mfem
 {
@@ -768,7 +769,7 @@ void TMOP_Metric_352::AssembleH(const DenseMatrix &Jpt,
 void TargetConstructor::ComputeAvgVolume() const
 {
    MFEM_VERIFY(nodes, "Nodes are not given!");
-   MFEM_ASSERT(avg_volume == 0.0, "the average volume is already computed!");
+   MFEM_ASSERT(avg_volume == 0.0, "The average volume is already computed!");
 
    Mesh *mesh = nodes->FESpace()->GetMesh();
    const int NE = mesh->GetNE();
@@ -787,9 +788,13 @@ void TargetConstructor::ComputeAvgVolume() const
          volume += ip.weight * Tr.Weight();
       }
    }
-   if (!Parallel())
+
+   NCMesh *ncmesh = mesh->ncmesh;
+   if (Parallel() == false)
    {
-      avg_volume = volume / NE;
+      avg_volume = (ncmesh == NULL) ?
+                   volume / NE : volume / ncmesh->GetNumRootElements();
+
    }
 #ifdef MFEM_USE_MPI
    else
@@ -797,7 +802,8 @@ void TargetConstructor::ComputeAvgVolume() const
       double area_NE[4];
       area_NE[0] = volume; area_NE[1] = NE;
       MPI_Allreduce(area_NE, area_NE + 2, 2, MPI_DOUBLE, MPI_SUM, comm);
-      avg_volume = area_NE[2] / area_NE[3];
+      avg_volume = (ncmesh == NULL) ?
+                   area_NE[2] / area_NE[3] : area_NE[2] / ncmesh->GetNumRootElements();
    }
 #endif
 }
@@ -805,6 +811,7 @@ void TargetConstructor::ComputeAvgVolume() const
 // virtual method
 void TargetConstructor::ComputeElementTargets(int e_id, const FiniteElement &fe,
                                               const IntegrationRule &ir,
+                                              const Vector &elfun,
                                               DenseTensor &Jtr) const
 {
    MFEM_ASSERT(target_type == IDEAL_SHAPE_UNIT_SIZE || nodes != NULL, "");
@@ -827,7 +834,15 @@ void TargetConstructor::ComputeElementTargets(int e_id, const FiniteElement &fe,
       {
          if (avg_volume == 0.0) { ComputeAvgVolume(); }
          DenseMatrix W(Wideal.Height());
-         W.Set(std::pow(volume_scale * avg_volume / Wideal.Det(),
+
+         NCMesh *ncmesh = nodes->FESpace()->GetMesh()->ncmesh;
+         double el_volume = avg_volume;
+         if (ncmesh)
+         {
+            el_volume = avg_volume / ncmesh->GetElementSizeReduction(e_id);
+         }
+
+         W.Set(std::pow(volume_scale * el_volume / Wideal.Det(),
                         1./W.Height()), Wideal);
          for (int i = 0; i < ir.GetNPoints(); i++) { Jtr(i) = W; }
          break;
@@ -853,7 +868,7 @@ void TargetConstructor::ComputeElementTargets(int e_id, const FiniteElement &fe,
             if (target_type == IDEAL_SHAPE_GIVEN_SIZE)
             {
                const double det = Jtr(i).Det();
-               MFEM_VERIFY(det > 0.0, "Initial mesh is inverted!");
+               MFEM_VERIFY(det > 0.0, "The given mesh is inverted!");
                Jtr(i).Set(std::pow(det / detW, 1./dim), Wideal);
             }
          }
@@ -864,6 +879,162 @@ void TargetConstructor::ComputeElementTargets(int e_id, const FiniteElement &fe,
    }
 }
 
+void AnalyticAdaptTC::SetAnalyticTargetSpec(Coefficient *sspec,
+                                            VectorCoefficient *vspec,
+                                            MatrixCoefficient *mspec)
+{
+   scalar_tspec = sspec;
+   vector_tspec = vspec;
+   matrix_tspec = mspec;
+}
+
+void AnalyticAdaptTC::ComputeElementTargets(int e_id, const FiniteElement &fe,
+                                            const IntegrationRule &ir,
+                                            const Vector &elfun,
+                                            DenseTensor &Jtr) const
+{
+   DenseMatrix point_mat;
+   point_mat.UseExternalData(elfun.GetData(), fe.GetDof(), fe.GetDim());
+
+   switch (target_type)
+   {
+      case GIVEN_FULL:
+      {
+         MFEM_VERIFY(matrix_tspec != NULL,
+                     "Target type GIVEN_FULL requires a MatrixCoefficient.");
+
+         IsoparametricTransformation Tpr;
+         Tpr.SetFE(&fe);
+         Tpr.ElementNo = e_id;
+         Tpr.GetPointMat().Transpose(point_mat);
+
+         for (int i = 0; i < ir.GetNPoints(); i++)
+         {
+            const IntegrationPoint &ip = ir.IntPoint(i);
+            Tpr.SetIntPoint(&ip);
+            matrix_tspec->Eval(Jtr(i), Tpr, ip);
+         }
+         break;
+      }
+      default:
+         MFEM_ABORT("Incompatible target type for analytic adaptation!");
+   }
+}
+
+#ifdef MFEM_USE_MPI
+void DiscreteAdaptTC::SetParDiscreteTargetSpec(ParGridFunction &tspec)
+{
+   target_spec.SetSize(tspec.Size());
+   target_spec = tspec;
+   tspec_fes   = tspec.FESpace();
+
+   // Default evaluator is based on CG advection.
+   if (adapt_eval == NULL) { adapt_eval = new AdvectorCG; }
+
+   adapt_eval->SetParMetaInfo(*tspec.ParFESpace()->GetParMesh(),
+                              *tspec.FESpace()->FEColl(),
+                              tspec.FESpace()->GetVDim());
+
+   adapt_eval->SetInitialField
+   (*tspec.FESpace()->GetMesh()->GetNodes(), target_spec);
+}
+#endif
+
+void DiscreteAdaptTC::SetSerialDiscreteTargetSpec(GridFunction &tspec)
+{
+   target_spec.SetSize(tspec.Size());
+   target_spec = tspec;
+   tspec_fes   = tspec.FESpace();
+
+   // Default evaluator is based on CG advection.
+   if (adapt_eval == NULL) { adapt_eval = new AdvectorCG; }
+
+   adapt_eval->SetSerialMetaInfo(*tspec.FESpace()->GetMesh(),
+                                 *tspec.FESpace()->FEColl(),
+                                 tspec.FESpace()->GetVDim());
+
+   adapt_eval->SetInitialField
+   (*tspec.FESpace()->GetMesh()->GetNodes(), target_spec);
+}
+
+void DiscreteAdaptTC::UpdateTargetSpecification(const Vector &new_x)
+{
+   MFEM_VERIFY(target_spec.Size() > 0, "Target specification is not set!");
+
+   adapt_eval->ComputeAtNewPosition(new_x, target_spec);
+}
+
+void DiscreteAdaptTC::ComputeElementTargets(int e_id, const FiniteElement &fe,
+                                            const IntegrationRule &ir,
+                                            const Vector &elfun,
+                                            DenseTensor &Jtr) const
+{
+   MFEM_VERIFY(tspec_fes, "A call to SetDiscreteTargerSpec() is needed.");
+
+   switch (target_type)
+   {
+      case IDEAL_SHAPE_GIVEN_SIZE:
+      {
+         const DenseMatrix &Wideal =
+            Geometries.GetGeomToPerfGeomJac(fe.GetGeomType());
+         const int dim = Wideal.Height(),
+                   ntspec_dofs = tspec_fes->GetFE(0)->GetDof();
+
+         Vector shape(ntspec_dofs), tspec_vals(ntspec_dofs);
+         Array<int> dofs;
+         tspec_fes->GetElementDofs(e_id, dofs);
+         target_spec.GetSubVector(dofs, tspec_vals);
+
+         const double min_size = tspec_vals.Min();
+         MFEM_ASSERT(min_size > 0.0,
+                     "Non-positive size propagated in the target definition.");
+
+         for (int i = 0; i < ir.GetNPoints(); i++)
+         {
+            const IntegrationPoint &ip = ir.IntPoint(i);
+            tspec_fes->GetFE(e_id)->CalcShape(ip, shape);
+            const double size = std::max(shape * tspec_vals, min_size);
+            Jtr(i).Set(std::pow(size / Wideal.Det(), 1.0/dim), Wideal);
+         }
+         break;
+      }
+      default:
+         MFEM_ABORT("Incompatible target type for analytic adaptation!");
+   }
+}
+
+void AdaptivityEvaluator::SetSerialMetaInfo(const Mesh &m,
+                                            const FiniteElementCollection &fec,
+                                            int num_comp)
+{
+   delete fes;
+   delete mesh;
+   mesh = new Mesh(m, true);
+   fes = new FiniteElementSpace(mesh, &fec, num_comp);
+}
+
+#ifdef MFEM_USE_MPI
+void AdaptivityEvaluator::SetParMetaInfo(const ParMesh &m,
+                                         const FiniteElementCollection &fec,
+                                         int num_comp)
+{
+   delete pfes;
+   delete pmesh;
+   pmesh = new ParMesh(m, true);
+   pfes  = new ParFiniteElementSpace(pmesh, &fec, num_comp);
+}
+#endif
+
+AdaptivityEvaluator::~AdaptivityEvaluator()
+{
+   delete fes;
+   delete mesh;
+#ifdef MFEM_USE_MPI
+   delete pfes;
+   delete pmesh;
+#endif
+}
+
 void TMOP_Integrator::EnableLimiting(const GridFunction &n0,
                                      const GridFunction &dist, Coefficient &w0,
                                      TMOP_LimiterFunction *lfunc)
@@ -921,7 +1092,7 @@ double TMOP_Integrator::GetElementEnergy(const FiniteElement &el,
 
    energy = 0.0;
    DenseTensor Jtr(dim, dim, ir->GetNPoints());
-   targetC->ComputeElementTargets(T.ElementNo, el, *ir, Jtr);
+   targetC->ComputeElementTargets(T.ElementNo, el, *ir, elfun, Jtr);
 
    // Limited case.
    Vector shape, p, p0, d_vals;
@@ -990,6 +1161,7 @@ double TMOP_Integrator::GetElementEnergy(const FiniteElement &el,
       energy += weight * val;
    }
    delete Tpr;
+
    return energy;
 }
 
@@ -1016,7 +1188,7 @@ void TMOP_Integrator::AssembleElementVector(const FiniteElement &el,
 
    elvect = 0.0;
    DenseTensor Jtr(dim, dim, ir->GetNPoints());
-   targetC->ComputeElementTargets(T.ElementNo, el, *ir, Jtr);
+   targetC->ComputeElementTargets(T.ElementNo, el, *ir, elfun, Jtr);
 
    // Limited case.
    DenseMatrix pos0;
@@ -1072,6 +1244,8 @@ void TMOP_Integrator::AssembleElementVector(const FiniteElement &el,
       P *= weight_m;
       AddMultABt(DS, P, PMatO);
 
+      // TODO: derivatives of adaptivity-based targets.
+
       if (coeff0)
       {
          el.CalcShape(ip, shape);
@@ -1107,7 +1281,7 @@ void TMOP_Integrator::AssembleElementGrad(const FiniteElement &el,
 
    elmat = 0.0;
    DenseTensor Jtr(dim, dim, ir->GetNPoints());
-   targetC->ComputeElementTargets(T.ElementNo, el, *ir, Jtr);
+   targetC->ComputeElementTargets(T.ElementNo, el, *ir, elfun, Jtr);
 
    // Limited case.
    DenseMatrix pos0, grad_grad;
@@ -1160,6 +1334,8 @@ void TMOP_Integrator::AssembleElementGrad(const FiniteElement &el,
 
       metric->AssembleH(Jpt, DS, weight_m, elmat);
 
+      // TODO: derivatives of adaptivity-based targets.
+
       if (coeff0)
       {
          el.CalcShape(ip, shape);
@@ -1234,11 +1410,12 @@ void TMOP_Integrator::ComputeNormalizationEnergies(const GridFunction &x,
    for (int i = 0; i < fes->GetNE(); i++)
    {
       fe = fes->GetFE(i);
-      targetC->ComputeElementTargets(i, *fe, *ir, Jtr);
       fes->GetElementVDofs(i, vdofs);
       x.GetSubVector(vdofs, x_vals);
       PMatI.UseExternalData(x_vals.GetData(), dof, dim);
 
+      targetC->ComputeElementTargets(i, *fe, *ir, x_vals, Jtr);
+
       for (int i = 0; i < ir->GetNPoints(); i++)
       {
          const IntegrationPoint &ip = ir->IntPoint(i);
@@ -1274,9 +1451,6 @@ void InterpolateTMOP_QualityMetric(TMOP_QualityMetric &metric,
       const IntegrationRule &ir = metric_gf.FESpace()->GetFE(i)->GetNodes();
       const int nsp = ir.GetNPoints(), dof = fe_pos.GetDof();
 
-      W.SetSize(dim, dim, nsp);
-      tc.ComputeElementTargets(i, fe_pos, ir, W);
-
       dshape.SetSize(dof, dim);
       pos.SetSize(dof, dim);
       posV.SetDataAndSize(pos.Data(), dof * dim);
@@ -1285,6 +1459,9 @@ void InterpolateTMOP_QualityMetric(TMOP_QualityMetric &metric,
       nodes.FESpace()->GetElementVDofs(i, pos_dofs);
       nodes.GetSubVector(pos_dofs, posV);
 
+      W.SetSize(dim, dim, nsp);
+      tc.ComputeElementTargets(i, fe_pos, ir, posV, W);
+
       for (int j = 0; j < nsp; j++)
       {
          const DenseMatrix &Wj = W(j);

fem/tmop.hpp

@@ -12,7 +12,6 @@
 #ifndef MFEM_TMOP_HPP
 #define MFEM_TMOP_HPP
 
-#include "../config/config.hpp"
 #include "../linalg/invariants.hpp"
 #include "nonlininteg.hpp"
 
@@ -514,6 +513,51 @@ public:
    virtual ~TMOP_QuadraticLimiter() { }
 };
 
+class FiniteElementCollection;
+class FiniteElementSpace;
+class ParFiniteElementSpace;
+
+class AdaptivityEvaluator
+{
+protected:
+   // Owned.
+   Mesh *mesh;
+   FiniteElementSpace *fes;
+
+#ifdef MFEM_USE_MPI
+   // Owned.
+   ParMesh *pmesh;
+   ParFiniteElementSpace *pfes;
+#endif
+
+public:
+   AdaptivityEvaluator() : mesh(NULL), fes(NULL)
+   {
+#ifdef MFEM_USE_MPI
+      pmesh = NULL;
+      pfes = NULL;
+#endif
+   }
+   virtual ~AdaptivityEvaluator();
+
+   /** Specifies the Mesh and FiniteElementCollection of the solution that will
+       be evaluated. The given mesh will be copied into the internal object. */
+   void SetSerialMetaInfo(const Mesh &m,
+                          const FiniteElementCollection &fec, int num_comp);
+
+#ifdef MFEM_USE_MPI
+   /// Parallel version of SetSerialMetaInfo.
+   void SetParMetaInfo(const ParMesh &m,
+                       const FiniteElementCollection &fec, int num_comp);
+#endif
+
+   // TODO use GridFunctions to make clear it's on the ldofs?
+   virtual void SetInitialField(const Vector &init_nodes,
+                                const Vector &init_field) = 0;
+
+   virtual void ComputeAtNewPosition(const Vector &new_nodes,
+                                     Vector &new_field) = 0;
+};
 
 /** @brief Base class representing target-matrix construction algorithms for
     mesh optimization via the target-matrix optimization paradigm (TMOP). */
@@ -538,9 +582,11 @@ public:
       IDEAL_SHAPE_GIVEN_SIZE, /**<
          Ideal shape, given size/volume; the given nodes define the target
          volume at all quadrature points. */
-      GIVEN_SHAPE_AND_SIZE /**<
+      GIVEN_SHAPE_AND_SIZE, /**<
          Given shape, given size/volume; the given nodes define the exact target
          Jacobian matrix at all quadrature points. */
+      GIVEN_FULL /**<
+         Full target tensor is specified at every quadrature point. */
    };
 
 protected:
@@ -589,14 +635,89 @@ public:
    void SetVolumeScale(double vol_scale) { volume_scale = vol_scale; }
 
    /** @brief Given an element and quadrature rule, computes ref->target
-       transformation Jacobians for each quadrature point in the element. */
+       transformation Jacobians for each quadrature point in the element.
+       The physical positions of the element's nodes are given by @a elfun. */
    virtual void ComputeElementTargets(int e_id, const FiniteElement &fe,
                                       const IntegrationRule &ir,
+                                      const Vector &elfun,
+                                      DenseTensor &Jtr) const;
+};
+
+class AnalyticAdaptTC : public TargetConstructor
+{
+protected:
+   // Analytic target specification.
+   Coefficient *scalar_tspec;
+   VectorCoefficient *vector_tspec;
+   MatrixCoefficient *matrix_tspec;
+
+public:
+   AnalyticAdaptTC(TargetType ttype)
+      : TargetConstructor(ttype),
+        scalar_tspec(NULL), vector_tspec(NULL), matrix_tspec(NULL) { }
+
+   virtual void SetAnalyticTargetSpec(Coefficient *sspec,
+                                      VectorCoefficient *vspec,
+                                      MatrixCoefficient *mspec);
+
+   /** @brief Given an element and quadrature rule, computes ref->target
+       transformation Jacobians for each quadrature point in the element.
+       The physical positions of the element's nodes are given by @a elfun. */
+   virtual void ComputeElementTargets(int e_id, const FiniteElement &fe,
+                                      const IntegrationRule &ir,
+                                      const Vector &elfun,
                                       DenseTensor &Jtr) const;
 };
 
 class ParGridFunction;
 
+class DiscreteAdaptTC : public TargetConstructor
+{
+protected:
+   // Discrete target specification.
+   // Data is owned, updated by UpdateTargetSpecification.
+   Vector target_spec;
+   // Note: do not use the Nodes of this space as they may not be on the
+   // positions corresponding to the values of tspec.
+   const FiniteElementSpace *tspec_fes;
+
+   // Evaluation of the discrete target specification on different meshes.
+   // Owned.
+   AdaptivityEvaluator *adapt_eval;
+
+public:
+   DiscreteAdaptTC(TargetType ttype)
+      : TargetConstructor(ttype),
+        target_spec(), tspec_fes(NULL), adapt_eval(NULL) { }
+
+   virtual ~DiscreteAdaptTC() { delete adapt_eval; }
+
+   virtual void SetSerialDiscreteTargetSpec(GridFunction &tspec);
+#ifdef MFEM_USE_MPI
+   virtual void SetParDiscreteTargetSpec(ParGridFunction &tspec);
+#endif
+
+   /** Used to update the target specification after the mesh has changed. The
+       new mesh positions are given by new_x. */
+   void UpdateTargetSpecification(const Vector &new_x);
+
+   void SetAdaptivityEvaluator(AdaptivityEvaluator *ae)
+   {
+      if (adapt_eval) { delete adapt_eval; }
+      adapt_eval = ae;
+   }
+
+   /** @brief Given an element and quadrature rule, computes ref->target
+       transformation Jacobians for each quadrature point in the element.
+       The physical positions of the element's nodes are given by @a elfun.
+       Note that this function assumes that UpdateTargetSpecification() has
+       been called with the position vector corresponding to @a elfun. */
+   virtual void ComputeElementTargets(int e_id, const FiniteElement &fe,
+                                      const IntegrationRule &ir,
+                                      const Vector &elfun,
+                                      DenseTensor &Jtr) const;
+};
+
 /** @brief A TMOP integrator class based on any given TMOP_QualityMetric and
     TargetConstructor.
 

fem/tmop_tools.cpp

@@ -0,0 +1,518 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "tmop_tools.hpp"
+#include "nonlinearform.hpp"
+#include "pnonlinearform.hpp"
+#include "../general/osockstream.hpp"
+
+namespace mfem
+{
+
+using namespace mfem;
+
+void AdvectorCG::SetInitialField(const Vector &init_nodes,
+                                 const Vector &init_field)
+{
+   nodes0 = init_nodes;
+   field0 = init_field;
+}
+
+void AdvectorCG::ComputeAtNewPosition(const Vector &new_nodes,
+                                      Vector &new_field)
+{
+   int myid = 0;
+   Mesh *m = mesh;
+
+#ifdef MFEM_USE_MPI
+   if (pfes) { MPI_Comm_rank(pfes->GetComm(), &myid); }
+   if (pmesh) { m = pmesh; }
+#endif
+
+   MFEM_VERIFY(m != NULL, "No mesh has been given to the AdaptivityEvaluator.");
+
+   // This will be used to move the positions.
+   GridFunction *mesh_nodes = m->GetNodes();
+   *mesh_nodes = nodes0;
+   new_field = field0;
+
+   // Velocity of the positions.
+   GridFunction u(mesh_nodes->FESpace());
+   subtract(new_nodes, nodes0, u);
+
+   TimeDependentOperator *oper = NULL;
+   // This must be the fes of the ind, associated with the object's mesh.
+   if (fes)  { oper = new SerialAdvectorCGOper(nodes0, u, *fes); }
+#ifdef MFEM_USE_MPI
+   else if (pfes) { oper = new ParAdvectorCGOper(nodes0, u, *pfes); }
+#endif
+   MFEM_VERIFY(oper != NULL,
+               "No FE space has been given to the AdaptivityEvaluator.");
+   ode_solver.Init(*oper);
+
+   // Compute some time step [mesh_size / speed].
+   double min_h = std::numeric_limits<double>::infinity();
+   for (int i = 0; i < m->GetNE(); i++)
+   {
+      min_h = std::min(min_h, m->GetElementSize(i));
+   }
+   double v_max = 0.0;
+   const int s = u.FESpace()->GetVSize() / 2;
+   for (int i = 0; i < s; i++)
+   {
+      const double vel = u(i) * u(i) + u(i+s) * u(i+s);
+      v_max = std::max(v_max, vel);
+   }
+   if (v_max == 0.0)
+   {
+      // No need to change the field.
+      return;
+   }
+   v_max = std::sqrt(v_max);
+   double dt = 0.5 * min_h / v_max;
+   double glob_dt = dt;
+#ifdef MFEM_USE_MPI
+   if (pfes)
+   {
+      MPI_Allreduce(&dt, &glob_dt, 1, MPI_DOUBLE, MPI_MIN, pfes->GetComm());
+   }
+#endif
+
+   double t = 0.0;
+   bool last_step = false;
+   for (int ti = 1; !last_step; ti++)
+   {
+      if (t + glob_dt >= 1.0)
+      {
+#ifdef MFEM_DEBUG
+         if (myid == 0)
+         {
+            mfem::out << "Remap took " << ti << " steps." << std::endl;
+         }
+#endif
+         glob_dt = 1.0 - t;
+         last_step = true;
+      }
+      ode_solver.Step(new_field, t, glob_dt);
+   }
+
+   // Trim the overshoots and undershoots.
+   const double minv = field0.Min(), maxv = field0.Max();
+   for (int i = 0; i < new_field.Size(); i++)
+   {
+      if (new_field(i) < minv) { new_field(i) = minv; }
+      if (new_field(i) > maxv) { new_field(i) = maxv; }
+   }
+
+   nodes0 = new_nodes;
+   field0 = new_field;
+
+   delete oper;
+}
+
+SerialAdvectorCGOper::SerialAdvectorCGOper(const Vector &x_start,
+                                           GridFunction &vel,
+                                           FiniteElementSpace &fes)
+   : TimeDependentOperator(fes.GetVSize()),
+     x0(x_start), x_now(*fes.GetMesh()->GetNodes()),
+     u(vel), u_coeff(&u), M(&fes), K(&fes)
+{
+   ConvectionIntegrator *Kinteg = new ConvectionIntegrator(u_coeff);
+   K.AddDomainIntegrator(Kinteg);
+   K.Assemble(0);
+   K.Finalize(0);
+
+   MassIntegrator *Minteg = new MassIntegrator;
+   M.AddDomainIntegrator(Minteg);
+   M.Assemble();
+   M.Finalize();
+}
+
+void SerialAdvectorCGOper::Mult(const Vector &ind, Vector &di_dt) const
+{
+   // Move the mesh.
+   const double t = GetTime();
+   add(x0, t, u, x_now);
+
+   // Assemble on the new mesh.
+   K.BilinearForm::operator=(0.0);
+   K.Assemble();
+   Vector rhs(K.Size());
+   K.Mult(ind, rhs);
+   M.BilinearForm::operator=(0.0);
+   M.Assemble();
+
+   di_dt = 0.0;
+   CGSolver lin_solver;
+   DSmoother prec;
+   lin_solver.SetPreconditioner(prec);
+   lin_solver.SetOperator(M.SpMat());
+   lin_solver.SetRelTol(1e-12); lin_solver.SetAbsTol(0.0);
+   lin_solver.SetMaxIter(100);
+   lin_solver.SetPrintLevel(0);
+   lin_solver.Mult(rhs, di_dt);
+}
+
+#ifdef MFEM_USE_MPI
+ParAdvectorCGOper::ParAdvectorCGOper(const Vector &x_start,
+                                     GridFunction &vel,
+                                     ParFiniteElementSpace &pfes)
+   : TimeDependentOperator(pfes.GetVSize()),
+     x0(x_start), x_now(*pfes.GetMesh()->GetNodes()),
+     u(vel), u_coeff(&u), M(&pfes), K(&pfes)
+{
+   ConvectionIntegrator *Kinteg = new ConvectionIntegrator(u_coeff);
+   K.AddDomainIntegrator(Kinteg);
+   K.Assemble(0);
+   K.Finalize(0);
+
+   MassIntegrator *Minteg = new MassIntegrator;
+   M.AddDomainIntegrator(Minteg);
+   M.Assemble();
+   M.Finalize();
+}
+
+void ParAdvectorCGOper::Mult(const Vector &ind, Vector &di_dt) const
+{
+   // Move the mesh.
+   const double t = GetTime();
+   add(x0, t, u, x_now);
+
+   // Assemble on the new mesh.
+   K.BilinearForm::operator=(0.0);
+   K.Assemble();
+   ParGridFunction rhs(K.ParFESpace());
+   K.Mult(ind, rhs);
+   M.BilinearForm::operator=(0.0);
+   M.Assemble();
+
+   HypreParVector *RHS = rhs.ParallelAssemble();
+   HypreParVector X(K.ParFESpace());
+   X = 0.0;
+   HypreParMatrix *Mh  = M.ParallelAssemble();
+
+   CGSolver lin_solver(M.ParFESpace()->GetParMesh()->GetComm());
+   HypreSmoother prec;
+   prec.SetType(HypreSmoother::Jacobi, 1);
+   lin_solver.SetPreconditioner(prec);
+   lin_solver.SetOperator(*Mh);
+   lin_solver.SetRelTol(1e-8);
+   lin_solver.SetAbsTol(0.0);
+   lin_solver.SetMaxIter(100);
+   lin_solver.SetPrintLevel(0);
+   lin_solver.Mult(*RHS, X);
+   K.ParFESpace()->GetProlongationMatrix()->Mult(X, di_dt);
+
+   delete Mh;
+   delete RHS;
+}
+#endif
+
+double TMOPNewtonSolver::ComputeScalingFactor(const Vector &x,
+                                              const Vector &b) const
+{
+   const FiniteElementSpace *fes = NULL;
+   double energy_in = 0.0;
+#ifdef MFEM_USE_MPI
+   const ParNonlinearForm *p_nlf = dynamic_cast<const ParNonlinearForm *>(oper);
+   MFEM_VERIFY(!(parallel && p_nlf == NULL), "Invalid Operator subclass.");
+   if (parallel)
+   {
+      fes = p_nlf->FESpace();
+      energy_in = p_nlf->GetEnergy(x);
+   }
+#endif
+   const bool serial = !parallel;
+   const NonlinearForm *nlf = dynamic_cast<const NonlinearForm *>(oper);
+   MFEM_VERIFY(!(serial && nlf == NULL), "Invalid Operator subclass.");
+   if (serial)
+   {
+      fes = nlf->FESpace();
+      energy_in = nlf->GetEnergy(x);
+   }
+
+   const bool have_b = (b.Size() == Height());
+
+   const int NE = fes->GetMesh()->GetNE(), dim = fes->GetFE(0)->GetDim(),
+             dof = fes->GetFE(0)->GetDof(), nsp = ir.GetNPoints();
+   Array<int> xdofs(dof * dim);
+   DenseMatrix Jpr(dim), dshape(dof, dim), pos(dof, dim);
+   Vector posV(pos.Data(), dof * dim);
+
+   Vector x_out(x.Size()), x_out_loc(fes->GetVSize());
+   bool x_out_ok = false;
+   double scale = 1.0, energy_out;
+   double norm0 = Norm(r);
+
+   // Decreases the scaling of the update until the new mesh is valid.
+   for (int i = 0; i < 12; i++)
+   {
+      add(x, -scale, c, x_out);
+
+      if (serial)
+      {
+         const SparseMatrix *cP = fes->GetConformingProlongation();
+         if (!cP) {x_out_loc.SetData(x_out.GetData());}
+         else {cP->Mult(x_out,x_out_loc);}
+         energy_out = nlf->GetGridFunctionEnergy(x_out_loc);
+      }
+#ifdef MFEM_USE_MPI
+      else
+      {
+         fes->GetProlongationMatrix()->Mult(x_out, x_out_loc);
+         energy_out = p_nlf->GetParGridFunctionEnergy(x_out_loc);
+      }
+#endif
+
+      if (energy_out > 1.2*energy_in || std::isnan(energy_out) != 0)
+      {
+         if (print_level >= 0)
+         { mfem::out << "Scale = " << scale << " Increasing energy.\n"; }
+         scale *= 0.5; continue;
+      }
+
+      int jac_ok = 1;
+      for (int i = 0; i < NE; i++)
+      {
+         fes->GetElementVDofs(i, xdofs);
+         x_out_loc.GetSubVector(xdofs, posV);
+         for (int j = 0; j < nsp; j++)
+         {
+            fes->GetFE(i)->CalcDShape(ir.IntPoint(j), dshape);
+            MultAtB(pos, dshape, Jpr);
+            if (Jpr.Det() <= 0.0) { jac_ok = 0; goto break2; }
+         }
+      }
+   break2:
+      int jac_ok_all = jac_ok;
+#ifdef MFEM_USE_MPI
+      if (parallel)
+      {
+         MPI_Allreduce(&jac_ok, &jac_ok_all, 1, MPI_INT, MPI_LAND,
+                       p_nlf->ParFESpace()->GetComm());
+      }
+#endif
+
+      if (jac_ok_all == 0)
+      {
+         if (print_level >= 0)
+         { mfem::out << "Scale = " << scale << " Neg det(J) found.\n"; }
+         scale *= 0.5; continue;
+      }
+
+      oper->Mult(x_out, r);
+      if (have_b) { r -= b; }
+      double norm = Norm(r);
+
+      if (norm > 1.2*norm0)
+      {
+         if (print_level >= 0)
+         { mfem::out << "Scale = " << scale << " Norm increased.\n"; }
+         scale *= 0.5; continue;
+      }
+      else { x_out_ok = true; break; }
+   }
+
+   if (print_level >= 0)
+   {
+      mfem::out << "Energy decrease: "
+                << (energy_in - energy_out) / energy_in * 100.0
+                << "% with " << scale << " scaling.\n";
+   }
+
+   if (x_out_ok == false) { scale = 0.0; }
+   return scale;
+}
+
+void TMOPNewtonSolver::ProcessNewState(const Vector &x) const
+{
+   if (discr_tc)
+   {
+      if (parallel)
+      {
+#ifdef MFEM_USE_MPI
+         const ParNonlinearForm *nlf =
+            dynamic_cast<const ParNonlinearForm *>(oper);
+         Vector x_loc(nlf->ParFESpace()->GetVSize());
+         nlf->ParFESpace()->GetProlongationMatrix()->Mult(x, x_loc);
+         discr_tc->UpdateTargetSpecification(x_loc);
+#endif
+      }
+      else { discr_tc->UpdateTargetSpecification(x); }
+   }
+}
+
+double TMOPDescentNewtonSolver::ComputeScalingFactor(const Vector &x,
+                                                     const Vector &b) const
+{
+   const FiniteElementSpace *fes = NULL;
+   double energy_in = 0.0;
+#ifdef MFEM_USE_MPI
+   const ParNonlinearForm *p_nlf = dynamic_cast<const ParNonlinearForm *>(oper);
+   MFEM_VERIFY(!(parallel && p_nlf == NULL), "Invalid Operator subclass.");
+   if (parallel)
+   {
+      fes = p_nlf->FESpace();
+      energy_in = p_nlf->GetEnergy(x);
+   }
+#endif
+   const bool serial = !parallel;
+   const NonlinearForm *nlf = dynamic_cast<const NonlinearForm *>(oper);
+   MFEM_VERIFY(!(serial && nlf == NULL), "Invalid Operator subclass.");
+   if (serial)
+   {
+      fes = nlf->FESpace();
+      energy_in = nlf->GetEnergy(x);
+   }
+
+   const int NE = fes->GetMesh()->GetNE(), dim = fes->GetFE(0)->GetDim(),
+             dof = fes->GetFE(0)->GetDof(), nsp = ir.GetNPoints();
+   Array<int> xdofs(dof * dim);
+   DenseMatrix Jpr(dim), dshape(dof, dim), pos(dof, dim);
+   Vector posV(pos.Data(), dof * dim);
+   Vector x_loc(fes->GetVSize());
+
+   double min_detJ = infinity();
+   for (int i = 0; i < NE; i++)
+   {
+      fes->GetElementVDofs(i, xdofs);
+      x_loc.GetSubVector(xdofs, posV);
+
+      for (int j = 0; j < nsp; j++)
+      {
+         fes->GetFE(i)->CalcDShape(ir.IntPoint(j), dshape);
+         MultAtB(pos, dshape, Jpr);
+         min_detJ = std::min(min_detJ, Jpr.Det());
+      }
+   }
+   double min_detJ_all = min_detJ;
+#ifdef MFEM_USE_MPI
+   if (parallel)
+   {
+      MPI_Allreduce(&min_detJ, &min_detJ_all, 1, MPI_DOUBLE, MPI_MIN,
+                    p_nlf->ParFESpace()->GetComm());
+   }
+#endif
+   if (print_level >= 0)
+   {
+      mfem::out << "Minimum det(J) = " << min_detJ_all << '\n';
+   }
+
+   Vector x_out(x.Size());
+   bool x_out_ok = false;
+   double scale = 1.0, energy_out;
+
+   for (int i = 0; i < 7; i++)
+   {
+      add(x, -scale, c, x_out);
+      if (serial)
+      {
+         const SparseMatrix *cP = fes->GetConformingProlongation();
+         if (!cP) {x_loc.SetData(x_out.GetData());}
+         else {cP->Mult(x_out,x_loc);}
+         energy_out = nlf->GetGridFunctionEnergy(x_loc);
+      }
+#ifdef MFEM_USE_MPI
+      else
+      {
+         fes->GetProlongationMatrix()->Mult(x_out, x_loc);
+         energy_out = p_nlf->GetParGridFunctionEnergy(x_loc);
+      }
+#endif
+
+      if (energy_out > energy_in || std::isnan(energy_out) != 0)
+      {
+         scale *= 0.5;
+      }
+      else { x_out_ok = true; break; }
+   }
+
+   if (print_level >= 0)
+   {
+      mfem::out << "Energy decrease: "
+                << (energy_in - energy_out) / energy_in * 100.0
+                << "% with " << scale << " scaling.\n";
+   }
+
+   if (x_out_ok == false) { return 0.0; }
+
+   return scale;
+}
+
+void TMOPDescentNewtonSolver::ProcessNewState(const Vector &x) const
+{
+   if (discr_tc)
+   {
+      if (parallel)
+      {
+#ifdef MFEM_USE_MPI
+         const ParNonlinearForm *nlf =
+            dynamic_cast<const ParNonlinearForm *>(oper);
+         Vector x_loc(nlf->ParFESpace()->GetVSize());
+         nlf->ParFESpace()->GetProlongationMatrix()->Mult(x, x_loc);
+         discr_tc->UpdateTargetSpecification(x_loc);
+#endif
+      }
+      else { discr_tc->UpdateTargetSpecification(x); }
+   }
+}
+
+#ifdef MFEM_USE_MPI
+// Metric values are visualized by creating an L2 finite element functions and
+// computing the metric values at the nodes.
+void vis_tmop_metric_p(int order, TMOP_QualityMetric &qm,
+                       const TargetConstructor &tc, ParMesh &pmesh,
+                       char *title, int position)
+{
+   L2_FECollection fec(order, pmesh.Dimension(), BasisType::GaussLobatto);
+   ParFiniteElementSpace fes(&pmesh, &fec, 1);
+   ParGridFunction metric(&fes);
+   InterpolateTMOP_QualityMetric(qm, tc, pmesh, metric);
+   socketstream sock;
+   if (pmesh.GetMyRank() == 0)
+   {
+      sock.open("localhost", 19916);
+      sock << "solution\n";
+   }
+   pmesh.PrintAsOne(sock);
+   metric.SaveAsOne(sock);
+   if (pmesh.GetMyRank() == 0)
+   {
+      sock << "window_title '"<< title << "'\n"
+           << "window_geometry "
+           << position << " " << 0 << " " << 600 << " " << 600 << "\n"
+           << "keys jRmclA\n";
+   }
+}
+#endif
+
+// Metric values are visualized by creating an L2 finite element functions and
+// computing the metric values at the nodes.
+void vis_tmop_metric_s(int order, TMOP_QualityMetric &qm,
+                       const TargetConstructor &tc, Mesh &mesh,
+                       char *title, int position)
+{
+   L2_FECollection fec(order, mesh.Dimension(), BasisType::GaussLobatto);
+   FiniteElementSpace fes(&mesh, &fec, 1);
+   GridFunction metric(&fes);
+   InterpolateTMOP_QualityMetric(qm, tc, mesh, metric);
+   osockstream sock(19916, "localhost");
+   sock << "solution\n";
+   mesh.Print(sock);
+   metric.Save(sock);
+   sock.send();
+   sock << "window_title '"<< title << "'\n"
+        << "window_geometry "
+        << position << " " << 0 << " " << 600 << " " << 600 << "\n"
+        << "keys jRmclA\n";
+}
+
+}

fem/tmop_tools.hpp

@@ -0,0 +1,140 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#ifndef MFEM_TMOP_TOOLS_HPP
+#define MFEM_TMOP_TOOLS_HPP
+
+#include "bilinearform.hpp"
+#include "pbilinearform.hpp"
+#include "tmop.hpp"
+
+namespace mfem
+{
+
+// Performs the full remap advection loop.
+class AdvectorCG : public AdaptivityEvaluator
+{
+private:
+   RK4Solver ode_solver;
+   Vector nodes0;
+   Vector field0;
+
+public:
+   AdvectorCG() : AdaptivityEvaluator(), ode_solver(), nodes0(), field0() { }
+
+   virtual void SetInitialField(const Vector &init_nodes,
+                                const Vector &init_field);
+
+   virtual void ComputeAtNewPosition(const Vector &new_nodes,
+                                     Vector &new_field);
+};
+
+/// Performs a single remap advection step in serial.
+class SerialAdvectorCGOper : public TimeDependentOperator
+{
+protected:
+   const Vector &x0;
+   Vector &x_now;
+   GridFunction &u;
+   VectorGridFunctionCoefficient u_coeff;
+   mutable BilinearForm M, K;
+
+public:
+   /** Here @a fes is the FESpace of the function that will be moved. Note
+       that Mult() moves the nodes of the mesh corresponding to @a fes. */
+   SerialAdvectorCGOper(const Vector &x_start, GridFunction &vel,
+                        FiniteElementSpace &fes);
+
+   virtual void Mult(const Vector &ind, Vector &di_dt) const;
+};
+
+#ifdef MFEM_USE_MPI
+/// Performs a single remap advection step in parallel.
+class ParAdvectorCGOper : public TimeDependentOperator
+{
+protected:
+   const Vector &x0;
+   Vector &x_now;
+   GridFunction &u;
+   VectorGridFunctionCoefficient u_coeff;
+   mutable ParBilinearForm M, K;
+
+public:
+   /** Here @a pfes is the ParFESpace of the function that will be moved. Note
+       that Mult() moves the nodes of the mesh corresponding to @a pfes. */
+   ParAdvectorCGOper(const Vector &x_start, GridFunction &vel,
+                     ParFiniteElementSpace &pfes);
+
+   virtual void Mult(const Vector &ind, Vector &di_dt) const;
+};
+#endif
+
+class TMOPNewtonSolver : public NewtonSolver
+{
+private:
+   bool parallel;
+
+   // Quadrature points that are checked for negative Jacobians etc.
+   const IntegrationRule &ir;
+
+   mutable DiscreteAdaptTC *discr_tc;
+
+public:
+#ifdef MFEM_USE_MPI
+   TMOPNewtonSolver(MPI_Comm comm, const IntegrationRule &irule)
+      : NewtonSolver(comm), parallel(true), ir(irule), discr_tc(NULL) { }
+#endif
+   TMOPNewtonSolver(const IntegrationRule &irule)
+      : NewtonSolver(), parallel(false), ir(irule), discr_tc(NULL) { }
+
+   void SetDiscreteAdaptTC(DiscreteAdaptTC *tc) { discr_tc = tc; }
+
+   virtual double ComputeScalingFactor(const Vector &x, const Vector &b) const;
+
+   virtual void ProcessNewState(const Vector &x) const;
+};
+
+/// Allows negative Jacobians. Used for untangling.
+class TMOPDescentNewtonSolver : public NewtonSolver
+{
+private:
+   bool parallel;
+
+   // Quadrature points that are checked for negative Jacobians etc.
+   const IntegrationRule &ir;
+
+   mutable DiscreteAdaptTC *discr_tc;
+
+public:
+#ifdef MFEM_USE_MPI
+   TMOPDescentNewtonSolver(MPI_Comm comm, const IntegrationRule &irule)
+      : NewtonSolver(comm), parallel(true), ir(irule), discr_tc(NULL) { }
+#endif
+   TMOPDescentNewtonSolver(const IntegrationRule &irule)
+      : NewtonSolver(), parallel(false), ir(irule), discr_tc(NULL) { }
+
+   virtual double ComputeScalingFactor(const Vector &x, const Vector &b) const;
+
+   virtual void ProcessNewState(const Vector &x) const;
+};
+
+void vis_tmop_metric_s(int order, TMOP_QualityMetric &qm,
+                       const TargetConstructor &tc, Mesh &pmesh,
+                       char *title, int position);
+#ifdef MFEM_USE_MPI
+void vis_tmop_metric_p(int order, TMOP_QualityMetric &qm,
+                       const TargetConstructor &tc, ParMesh &pmesh,
+                       char *title, int position);
+#endif
+
+}
+
+#endif

general/array.hpp

@@ -254,6 +254,10 @@ public:
    template <typename U>
    inline void CopyTo(U *dest) { std::copy(begin(), end(), dest); }
 
+   template <typename U>
+   inline void CopyFrom(const U *src)
+   { std::memcpy(begin(), src, MemoryUsage()); }
+
    // STL-like begin/end
    inline T* begin() { return data; }
    inline T* end() { return data + size; }
@@ -429,7 +433,7 @@ class BlockArray
 public:
    BlockArray(int block_size = 16*1024);
    BlockArray(const BlockArray<T> &other); // deep copy
-   ~BlockArray();
+   ~BlockArray() { Destroy(); }
 
    /// Allocate and construct a new item in the array, return its index.
    int Append();
@@ -459,6 +463,9 @@ public:
    /// Return the current capacity of the BlockArray.
    int Capacity() const { return blocks.Size()*(mask+1); }
 
+   /// Destroy all items, set size to zero.
+   void DeleteAll() { Destroy(); blocks.DeleteAll(); size = 0; }
+
    void Swap(BlockArray<T> &other);
 
    long MemoryUsage() const;
@@ -563,6 +570,8 @@ protected:
       MFEM_ASSERT(index >= 0 && index < size,
                   "Out of bounds access: " << index << ", size = " << size);
    }
+
+   void Destroy();
 };
 
 
@@ -994,7 +1003,7 @@ long BlockArray<T>::MemoryUsage() const
 }
 
 template<typename T>
-BlockArray<T>::~BlockArray()
+void BlockArray<T>::Destroy()
 {
    int bsize = size & mask;
    for (int i = blocks.Size(); i != 0; )

general/communication.cpp

@@ -513,6 +513,78 @@ void GroupCommunicator::SetLTDofTable(const Array<int> &ldof_ltdof)
    group_ltdof.ShiftUpI();
 }
 
+void GroupCommunicator::GetNeighborLTDofTable(Table &nbr_ltdof) const
+{
+   nbr_ltdof.MakeI(nbr_send_groups.Size());
+   for (int nbr = 1; nbr < nbr_send_groups.Size(); nbr++)
+   {
+      const int num_send_groups = nbr_send_groups.RowSize(nbr);
+      if (num_send_groups > 0)
+      {
+         const int *grp_list = nbr_send_groups.GetRow(nbr);
+         for (int i = 0; i < num_send_groups; i++)
+         {
+            const int group = grp_list[i];
+            const int nltdofs = group_ltdof.RowSize(group);
+            nbr_ltdof.AddColumnsInRow(nbr, nltdofs);
+         }
+      }
+   }
+   nbr_ltdof.MakeJ();
+   for (int nbr = 1; nbr < nbr_send_groups.Size(); nbr++)
+   {
+      const int num_send_groups = nbr_send_groups.RowSize(nbr);
+      if (num_send_groups > 0)
+      {
+         const int *grp_list = nbr_send_groups.GetRow(nbr);
+         for (int i = 0; i < num_send_groups; i++)
+         {
+            const int group = grp_list[i];
+            const int nltdofs = group_ltdof.RowSize(group);
+            const int *ltdofs = group_ltdof.GetRow(group);
+            nbr_ltdof.AddConnections(nbr, ltdofs, nltdofs);
+         }
+      }
+   }
+   nbr_ltdof.ShiftUpI();
+}
+
+void GroupCommunicator::GetNeighborLDofTable(Table &nbr_ldof) const
+{
+   nbr_ldof.MakeI(nbr_recv_groups.Size());
+   for (int nbr = 1; nbr < nbr_recv_groups.Size(); nbr++)
+   {
+      const int num_recv_groups = nbr_recv_groups.RowSize(nbr);
+      if (num_recv_groups > 0)
+      {
+         const int *grp_list = nbr_recv_groups.GetRow(nbr);
+         for (int i = 0; i < num_recv_groups; i++)
+         {
+            const int group = grp_list[i];
+            const int nldofs = group_ldof.RowSize(group);
+            nbr_ldof.AddColumnsInRow(nbr, nldofs);
+         }
+      }
+   }
+   nbr_ldof.MakeJ();
+   for (int nbr = 1; nbr < nbr_recv_groups.Size(); nbr++)
+   {
+      const int num_recv_groups = nbr_recv_groups.RowSize(nbr);
+      if (num_recv_groups > 0)
+      {
+         const int *grp_list = nbr_recv_groups.GetRow(nbr);
+         for (int i = 0; i < num_recv_groups; i++)
+         {
+            const int group = grp_list[i];
+            const int nldofs = group_ldof.RowSize(group);
+            const int *ldofs = group_ldof.GetRow(group);
+            nbr_ldof.AddConnections(nbr, ldofs, nldofs);
+         }
+      }
+   }
+   nbr_ldof.ShiftUpI();
+}
+
 template <class T>
 T *GroupCommunicator::CopyGroupToBuffer(const T *ldata, T *buf, int group,
                                         int layout) const

general/communication.hpp

@@ -179,6 +179,12 @@ public:
    /// Get a const reference to the associated GroupTopology object
    const GroupTopology &GetGroupTopology() const { return gtopo; }
 
+   /// Dofs to be sent to communication neighbors
+   void GetNeighborLTDofTable(Table &nbr_ltdof) const;
+
+   /// Dofs to be received from communication neighbors
+   void GetNeighborLDofTable(Table &nbr_ldof) const;
+
    /** @brief Data structure on which we define reduce operations.
 
      The data is associated with (and the operation is performed on) one group
@@ -316,7 +322,9 @@ struct VarMessage
    std::string data;
    MPI_Request send_request;
 
-   /// Non-blocking send to processor 'rank'.
+   /** Non-blocking send to processor 'rank'. Returns immediately. Completion
+       (as tested by MPI_Wait/Test) does not mean the message was received --
+       it may be on its way or just buffered locally. */
    void Isend(int rank, MPI_Comm comm)
    {
       Encode(rank);
@@ -324,12 +332,20 @@ struct VarMessage
                 &send_request);
    }
 
+   /** Non-blocking synchronous send to processor 'rank'. Returns immediately.
+       Completion (MPI_Wait/Test) means that the message was received. */
+   void Issend(int rank, MPI_Comm comm)
+   {
+      Encode(rank);
+      MPI_Issend((void*) data.data(), data.length(), MPI_BYTE, rank, Tag, comm,
+                 &send_request);
+   }
+
    /// Helper to send all messages in a rank-to-message map container.
    template<typename MapT>
    static void IsendAll(MapT& rank_msg, MPI_Comm comm)
    {
-      typename MapT::iterator it;
-      for (it = rank_msg.begin(); it != rank_msg.end(); ++it)
+      for (auto it = rank_msg.begin(); it != rank_msg.end(); ++it)
       {
          it->second.Isend(it->first, comm);
       }
@@ -339,14 +355,32 @@ struct VarMessage
    template<typename MapT>
    static void WaitAllSent(MapT& rank_msg)
    {
-      typename MapT::iterator it;
-      for (it = rank_msg.begin(); it != rank_msg.end(); ++it)
+      for (auto it = rank_msg.begin(); it != rank_msg.end(); ++it)
       {
          MPI_Wait(&it->second.send_request, MPI_STATUS_IGNORE);
          it->second.Clear();
       }
    }
 
+   /** Return true if all messages in the map container were sent, otherwise
+       return false, without waiting. */
+   template<typename MapT>
+   static bool TestAllSent(MapT& rank_msg)
+   {
+      for (auto it = rank_msg.begin(); it != rank_msg.end(); ++it)
+      {
+         VarMessage &msg = it->second;
+         if (msg.send_request != MPI_REQUEST_NULL)
+         {
+            int sent;
+            MPI_Test(&msg.send_request, &sent, MPI_STATUS_IGNORE);
+            if (!sent) { return false; }
+            msg.Clear();
+         }
+      }
+      return true;
+   }
+
    /** Blocking probe for incoming message of this type from any rank.
        Returns the rank and message size. */
    static void Probe(int &rank, int &size, MPI_Comm comm)

general/cuda.cpp

@@ -130,4 +130,13 @@ void* CuMemcpyDtoHAsync(void *dst, const void *src, size_t bytes)
    return dst;
 }
 
+int CuGetDeviceCount()
+{
+   int num_gpus = -1;
+#ifdef MFEM_USE_CUDA
+   MFEM_GPU_CHECK(cudaGetDeviceCount(&num_gpus));
+#endif
+   return num_gpus;
+}
+
 } // namespace mfem

general/cuda.hpp

@@ -39,9 +39,11 @@
       } \
    } \
    while (0)
+#define MFEM_DEVICE_SYNC MFEM_GPU_CHECK(cudaDeviceSynchronize())
 #else
 #define MFEM_DEVICE
 #define MFEM_HOST_DEVICE
+#define MFEM_DEVICE_SYNC
 #endif // MFEM_USE_CUDA
 
 // Define the MFEM inner threading macros
@@ -92,6 +94,9 @@ void* CuMemcpyDtoH(void *h_dst, const void *d_src, size_t bytes);
 /// Copies memory from Device to Host
 void* CuMemcpyDtoHAsync(void *h_dst, const void *d_src, size_t bytes);
 
+/// Get the number of CUDA devices
+int CuGetDeviceCount();
+
 } // namespace mfem
 
 #endif // MFEM_CUDA_HPP

general/device.cpp

@@ -138,7 +138,7 @@ void Device::Enable()
 #ifdef MFEM_USE_CUDA
 static void DeviceSetup(const int dev, int &ngpu)
 {
-   MFEM_GPU_CHECK(cudaGetDeviceCount(&ngpu));
+   ngpu = CuGetDeviceCount();
    MFEM_VERIFY(ngpu > 0, "No CUDA device found!");
    MFEM_GPU_CHECK(cudaSetDevice(dev));
 }

general/device.hpp

@@ -12,6 +12,7 @@
 #ifndef MFEM_DEVICE_HPP
 #define MFEM_DEVICE_HPP
 
+#include "cuda.hpp"
 #include "globals.hpp"
 #include "mem_manager.hpp"
 
@@ -110,6 +111,7 @@ private:
    unsigned long backends; ///< Bitwise-OR of all configured backends.
    /// Set to true during configuration, except in 'device_singleton'.
    bool destroy_mm;
+   bool mpi_gpu_aware;
 
    MemoryType mem_type;    ///< Current Device MemoryType
    MemoryClass mem_class;  ///< Current Device MemoryClass
@@ -147,6 +149,7 @@ public:
       : mode(Device::SEQUENTIAL),
         backends(Backend::CPU),
         destroy_mm(false),
+        mpi_gpu_aware(false),
         mem_type(MemoryType::HOST),
         mem_class(MemoryClass::HOST)
    { }
@@ -161,6 +164,7 @@ public:
       : mode(Device::SEQUENTIAL),
         backends(Backend::CPU),
         destroy_mm(false),
+        mpi_gpu_aware(false),
         mem_type(MemoryType::HOST),
         mem_class(MemoryClass::HOST)
    { Configure(device, dev); }
@@ -216,6 +220,13 @@ public:
    /** @brief Get the current Device MemoryClass. This is the MemoryClass used
        by most MFEM device kernels to access Memory objects. */
    static inline MemoryClass GetMemoryClass() { return Get().mem_class; }
+
+   static void SetGPUAwareMPI(const bool force = true)
+   { Get().mpi_gpu_aware = force; }
+
+   static bool GetGPUAwareMPI() { return Get().mpi_gpu_aware; }
+
+   static void Synchronize() { MFEM_DEVICE_SYNC; }
 };
 
 
@@ -265,7 +276,7 @@ inline T *Write(Memory<T> &mem, int size, bool on_dev = true)
 
 /** @brief Shortcut to Write(const Memory<T> &mem, int size, false) */
 template <typename T>
-inline const T *HostWrite(const Memory<T> &mem, int size)
+inline T *HostWrite(Memory<T> &mem, int size)
 {
    return mfem::Write(mem, size, false);
 }
@@ -287,9 +298,9 @@ inline T *ReadWrite(Memory<T> &mem, int size, bool on_dev = true)
    }
 }
 
-/** @brief Shortcut to ReadWrite(const Memory<T> &mem, int size, false) */
+/** @brief Shortcut to ReadWrite(Memory<T> &mem, int size, false) */
 template <typename T>
-inline const T *HostReadWrite(const Memory<T> &mem, int size)
+inline T *HostReadWrite(Memory<T> &mem, int size)
 {
    return mfem::ReadWrite(mem, size, false);
 }

general/error.hpp

@@ -138,4 +138,10 @@ void mfem_warning(const char *msg = NULL);
 // Generate a warning message - always generated, regardless of MFEM_DEBUG.
 #define MFEM_WARNING(msg) _MFEM_MESSAGE("MFEM Warning: " << msg, 1)
 
+// Macro that checks (in MFEM_DEBUG mode) that i is in the range [imin,imax).
+#define MFEM_ASSERT_INDEX_IN_RANGE(i,imin,imax) \
+   MFEM_ASSERT((imin) <= (i) && (i) < (imax), \
+   "invalid index " #i << " = " << (i) << \
+   ", valid range is [" << (imin) << ',' << (imax) << ')')
+
 #endif

general/forall.hpp

@@ -84,28 +84,76 @@ void OmpWrap(const int N, HBODY &&h_body)
 
 
 /// RAJA Cuda backend
-template <int BLOCKS, typename DBODY>
-void RajaCudaWrap(const int N, DBODY &&d_body)
-{
 #if defined(MFEM_USE_RAJA) && defined(RAJA_ENABLE_CUDA)
-   RAJA::forall<RAJA::cuda_exec<BLOCKS>>(RAJA::RangeSegment(0,N),d_body);
-#else
-   MFEM_ABORT("RAJA::Cuda requested but RAJA::Cuda is not enabled!");
-#endif
+
+using RAJA::statement::Segs;
+
+template <const int BLOCKS = MFEM_CUDA_BLOCKS, typename DBODY>
+void RajaCudaWrap1D(const int N, DBODY &&d_body)
+{
+   //true denotes asynchronous kernel
+   RAJA::forall<RAJA::cuda_exec<BLOCKS,true>>(RAJA::RangeSegment(0,N),d_body);
 }
 
+template <typename DBODY>
+void RajaCudaWrap2D(const int N, DBODY &&d_body,
+                    const int X, const int Y, const int BZ)
+{
+   MFEM_VERIFY(N>0, "");
+   MFEM_VERIFY(BZ>0, "");
+   const int G = (N+BZ-1)/BZ;
+   RAJA::kernel<RAJA::KernelPolicy<
+   RAJA::statement::CudaKernelAsync<
+   RAJA::statement::For<0, RAJA::cuda_block_x_direct,
+        RAJA::statement::For<1, RAJA::cuda_thread_x_direct,
+        RAJA::statement::For<2, RAJA::cuda_thread_y_direct,
+        RAJA::statement::For<3, RAJA::cuda_thread_z_direct,
+        RAJA::statement::Lambda<0, Segs<0>>>>>>>>>
+        (RAJA::make_tuple(RAJA::RangeSegment(0,G), RAJA::RangeSegment(0,X),
+                          RAJA::RangeSegment(0,Y), RAJA::RangeSegment(0,BZ)),
+         [=] RAJA_DEVICE (const int n)
+   {
+      const int k = n*BZ + threadIdx.z;
+      if (k >= N) { return; }
+      d_body(k);
+   });
+   MFEM_GPU_CHECK(cudaGetLastError());
+}
+
+template <typename DBODY>
+void RajaCudaWrap3D(const int N, DBODY &&d_body,
+                    const int X, const int Y, const int Z)
+{
+   MFEM_VERIFY(N>0, "");
+   RAJA::kernel<RAJA::KernelPolicy<
+   RAJA::statement::CudaKernelAsync<
+   RAJA::statement::For<0, RAJA::cuda_block_x_direct,
+        RAJA::statement::For<1, RAJA::cuda_thread_x_direct,
+        RAJA::statement::For<2, RAJA::cuda_thread_y_direct,
+        RAJA::statement::For<3, RAJA::cuda_thread_z_direct,
+        RAJA::statement::Lambda<0, Segs<0>>>>>>>>>
+        (RAJA::make_tuple(RAJA::RangeSegment(0,N), RAJA::RangeSegment(0,X),
+                          RAJA::RangeSegment(0,Y), RAJA::RangeSegment(0,Z)),
+   [=] RAJA_DEVICE (const int k) { d_body(k); });
+   MFEM_GPU_CHECK(cudaGetLastError());
+}
+
+#endif
+
 
 /// RAJA OpenMP backend
+#if defined(MFEM_USE_RAJA) && defined(RAJA_ENABLE_OPENMP)
+
+using RAJA::statement::Segs;
+
 template <typename HBODY>
 void RajaOmpWrap(const int N, HBODY &&h_body)
 {
-#if defined(MFEM_USE_RAJA) && defined(RAJA_ENABLE_OPENMP)
    RAJA::forall<RAJA::omp_parallel_for_exec>(RAJA::RangeSegment(0,N), h_body);
-#else
-   MFEM_ABORT("RAJA::OpenMP requested but RAJA::OpenMP is not enabled!");
-#endif
 }
 
+#endif
+
 
 /// RAJA sequential loop backend
 template <typename HBODY>
@@ -160,6 +208,7 @@ void CuWrap2D(const int N, DBODY &&d_body,
               const int X, const int Y, const int BZ)
 {
    if (N==0) { return; }
+   MFEM_VERIFY(BZ>0, "");
    const int GRID = (N+BZ-1)/BZ;
    const dim3 BLCK(X,Y,BZ);
    CuKernel2D<<<GRID,BLCK>>>(N,d_body,BZ);
@@ -251,8 +300,14 @@ inline void ForallWrap(const bool use_dev, const int N,
 
 #if defined(MFEM_USE_RAJA) && defined(RAJA_ENABLE_CUDA)
    // Handle all allowed CUDA backends except Backend::CUDA
-   if (Device::Allows(Backend::CUDA_MASK & ~Backend::CUDA))
-   { return RajaCudaWrap<MFEM_CUDA_BLOCKS>(N, d_body); }
+   if (DIM == 1 && Device::Allows(Backend::CUDA_MASK & ~Backend::CUDA))
+   { return RajaCudaWrap1D(N, d_body); }
+
+   if (DIM == 2 && Device::Allows(Backend::CUDA_MASK & ~Backend::CUDA))
+   { return RajaCudaWrap2D(N, d_body, X, Y, Z); }
+
+   if (DIM == 3 && Device::Allows(Backend::CUDA_MASK & ~Backend::CUDA))
+   { return RajaCudaWrap3D(N, d_body, X, Y, Z); }
 #endif
 
 #ifdef MFEM_USE_CUDA

general/globals.hpp

@@ -107,4 +107,9 @@ void SetGlobalMPI_Comm(MPI_Comm comm);
 
 } // namespace mfem
 
+
+// Request a global object to be instantiated for each thread in its TLS.
+#define MFEM_THREAD_LOCAL thread_local
+
+
 #endif

general/hash.hpp

@@ -79,22 +79,22 @@ public:
 
    /// Get item whose parents are p1, p2... Create it if it doesn't exist.
    T* Get(int p1, int p2);
-   T* Get(int p1, int p2, int p3, int p4);
+   T* Get(int p1, int p2, int p3, int p4 = -1 /* p4 optional */);
 
    /// Get id of item whose parents are p1, p2... Create it if it doesn't exist.
    int GetId(int p1, int p2);
-   int GetId(int p1, int p2, int p3, int p4);
+   int GetId(int p1, int p2, int p3, int p4 = -1);
 
    /// Find item whose parents are p1, p2... Return NULL if it doesn't exist.
    T* Find(int p1, int p2);
-   T* Find(int p1, int p2, int p3, int p4);
+   T* Find(int p1, int p2, int p3, int p4 = -1);
 
    const T* Find(int p1, int p2) const;
-   const T* Find(int p1, int p2, int p3, int p4) const;
+   const T* Find(int p1, int p2, int p3, int p4 = -1) const;
 
    /// Find id of item whose parents are p1, p2... Return -1 if it doesn't exist.
    int FindId(int p1, int p2) const;
-   int FindId(int p1, int p2, int p3, int p4) const;
+   int FindId(int p1, int p2, int p3, int p4 = -1) const;
 
    /// Return the number of elements currently stored in the HashTable.
    int Size() const { return Base::Size() - unused.Size(); }
@@ -113,9 +113,12 @@ public:
    /** Its id will be reused by newly added items. */
    void Delete(int id);
 
+   /// Remove all items.
+   void DeleteAll();
+
    /// Make an item hashed under different parent IDs.
    void Reparent(int id, int new_p1, int new_p2);
-   void Reparent(int id, int new_p1, int new_p2, int new_p3, int new_p4);
+   void Reparent(int id, int new_p1, int new_p2, int new_p3, int new_p4 = -1);
 
    /// Return total size of allocated memory (tables plus items), in bytes.
    long MemoryUsage() const;
@@ -246,6 +249,18 @@ inline void sort4(int &a, int &b, int &c, int &d)
    sort3(b, c, d);
 }
 
+inline void sort4_ext(int &a, int &b, int &c, int &d)
+{
+   if (d < 0) // support optional last index
+   {
+      sort3(a, b, c);
+   }
+   else
+   {
+      sort4(a, b, c, d);
+   }
+}
+
 } // internal
 
 template<typename T>
@@ -295,7 +310,7 @@ template<typename T>
 int HashTable<T>::GetId(int p1, int p2, int p3, int p4)
 {
    // search for the item in the hashtable
-   internal::sort4(p1, p2, p3, p4);
+   internal::sort4_ext(p1, p2, p3, p4);
    int idx = Hash(p1, p2, p3);
    int id = SearchList(table[idx], p1, p2, p3);
    if (id >= 0) { return id; }
@@ -361,7 +376,7 @@ int HashTable<T>::FindId(int p1, int p2) const
 template<typename T>
 int HashTable<T>::FindId(int p1, int p2, int p3, int p4) const
 {
-   internal::sort4(p1, p2, p3, p4);
+   internal::sort4_ext(p1, p2, p3, p4);
    return SearchList(table[Hash(p1, p2, p3)], p1, p2, p3);
 }
 
@@ -459,6 +474,14 @@ void HashTable<T>::Delete(int id)
    unused.Append(id); // add its id to the unused ids
 }
 
+template<typename T>
+void HashTable<T>::DeleteAll()
+{
+   Base::DeleteAll();
+   for (int i = 0; i <= mask; i++) { table[i] = -1; }
+   unused.DeleteAll();
+}
+
 template<typename T>
 void HashTable<T>::Reparent(int id, int new_p1, int new_p2)
 {
@@ -481,7 +504,7 @@ void HashTable<T>::Reparent(int id,
    T& item = Base::At(id);
    Unlink(Hash(item), id);
 
-   internal::sort4(new_p1, new_p2, new_p3, new_p4);
+   internal::sort4_ext(new_p1, new_p2, new_p3, new_p4);
    item.p1 = new_p1;
    item.p2 = new_p2;
    item.p3 = new_p3;

general/mem_manager.cpp

@@ -11,7 +11,7 @@
 
 #include "../general/forall.hpp"
 
-#include <cstring> // std::memcpy
+#include <cstring> // std::memcpy, std::memcmp
 
 #include <list>
 #include <unordered_map>
@@ -57,6 +57,15 @@ MemoryClass operator*(MemoryClass mc1, MemoryClass mc2)
 }
 
 
+// Instantiate Memory<T>::PrintFlags for T = int and T = double.
+template void Memory<int>::PrintFlags() const;
+template void Memory<double>::PrintFlags() const;
+
+// Instantiate Memory<T>::CompareHostAndDevice for T = int and T = double.
+template int Memory<int>::CompareHostAndDevice(int size) const;
+template int Memory<double>::CompareHostAndDevice(int size) const;
+
+
 namespace internal
 {
 
@@ -695,6 +704,19 @@ void MemoryManager::CopyFromHost_(void *dest_h_ptr, const void *src_h_ptr,
                 ~(dest_on_host ? Mem::VALID_DEVICE : Mem::VALID_HOST);
 }
 
+int MemoryManager::CompareHostAndDevice_(void *h_ptr, size_t size,
+                                         unsigned flags)
+{
+   void *d_ptr = (flags & Mem::ALIAS) ?
+                 mm.GetAliasDevicePtr(h_ptr, size, false) :
+                 mm.GetDevicePtr(h_ptr, size, false);
+   char *h_buf = new char[size];
+   CuMemcpyDtoH(h_buf, d_ptr, size);
+   int res = std::memcmp(h_ptr, h_buf, size);
+   delete [] h_buf;
+   return res;
+}
+
 
 void MemoryPrintFlags(unsigned flags)
 {

general/mem_manager.hpp

@@ -377,6 +377,16 @@ public:
    /// Copy @a size entries from @a *this to the host pointer @a dest.
    /** The given @a size should not exceed the Capacity() of @a *this. */
    inline void CopyToHost(T *dest, int size) const;
+
+   /// Print the internal flags.
+   /** This method can be useful for debugging. It is explicitly instantiated
+       for Memory<T> with T = int and T = double. */
+   inline void PrintFlags() const;
+
+   /// If both the host and the device data are valid, compare their contents.
+   /** This method can be useful for debugging. It is explicitly instantiated
+       for Memory<T> with T = int and T = double. */
+   inline int CompareHostAndDevice(int size) const;
 };
 
 
@@ -447,6 +457,11 @@ private:
    static void CopyFromHost_(void *dest_h_ptr, const void *src_h_ptr,
                              std::size_t size, unsigned &dest_flags);
 
+   // Compare the contents of the host and the device memory - useful for
+   // debugging.
+   static int CompareHostAndDevice_(void *h_ptr, size_t size, unsigned flags);
+
+
    /// Adds an address in the map
    void *Insert(void *ptr, const std::size_t bytes);
 
@@ -727,10 +742,24 @@ inline void Memory<T>::CopyToHost(T *dest, int size) const
 
 
 /** @brief Print the state of a Memory object based on its internal flags.
-    Useful in a debugger. */
+    Useful in a debugger. See also Memory<T>::PrintFlags(). */
 extern void MemoryPrintFlags(unsigned flags);
 
 
+template <typename T>
+inline void Memory<T>::PrintFlags() const
+{
+   MemoryPrintFlags(flags);
+}
+
+template <typename T>
+inline int Memory<T>::CompareHostAndDevice(int size) const
+{
+   if (!(flags & VALID_HOST) || !(flags & VALID_DEVICE)) { return 0; }
+   return MemoryManager::CompareHostAndDevice_(h_ptr, size*sizeof(T), flags);
+}
+
+
 /// The (single) global memory manager object
 extern MemoryManager mm;
 

linalg/densemat.cpp

@@ -4456,7 +4456,8 @@ const
    int n = SizeI(), ne = SizeK();
    const int *I = elem_dof.GetI(), *J = elem_dof.GetJ(), *dofs;
    const double *d_col = tdata;
-   double *yp = y, x_col;
+   double *yp = y.HostReadWrite();
+   double x_col;
    const double *xp = x;
    // the '4' here can be tuned for given platform and compiler
    if (n <= 4)

linalg/densemat.hpp

@@ -721,11 +721,26 @@ public:
    { return const_cast<DenseTensor&>(*this)(k); }
 
    double &operator()(int i, int j, int k)
-   { return tdata[i+SizeI()*(j+SizeJ()*k)]; }
-   const double &operator()(int i, int j, int k) const
-   { return tdata[i+SizeI()*(j+SizeJ()*k)]; }
+   {
+      MFEM_ASSERT_INDEX_IN_RANGE(i, 0, SizeI());
+      MFEM_ASSERT_INDEX_IN_RANGE(j, 0, SizeJ());
+      MFEM_ASSERT_INDEX_IN_RANGE(k, 0, SizeK());
+      return tdata[i+SizeI()*(j+SizeJ()*k)];
+   }
 
-   double *GetData(int k) { return tdata+k*Mk.Height()*Mk.Width(); }
+   const double &operator()(int i, int j, int k) const
+   {
+      MFEM_ASSERT_INDEX_IN_RANGE(i, 0, SizeI());
+      MFEM_ASSERT_INDEX_IN_RANGE(j, 0, SizeJ());
+      MFEM_ASSERT_INDEX_IN_RANGE(k, 0, SizeK());
+      return tdata[i+SizeI()*(j+SizeJ()*k)];
+   }
+
+   double *GetData(int k)
+   {
+      MFEM_ASSERT_INDEX_IN_RANGE(k, 0, SizeK());
+      return tdata+k*Mk.Height()*Mk.Width();
+   }
 
    double *Data() { return tdata; }
 

linalg/hypre.cpp

@@ -172,13 +172,6 @@ HypreParVector::HypreParVector(ParFiniteElementSpace *pfes)
    _SetDataAndSize_();
    own_ParVector = 1;
 }
-  
-void HypreParVector::WrapHypreParVector(hypre_ParVector *y)
-{
-   x = y;
-   _SetDataAndSize_();
-   own_ParVector = 0;
-}
 
 Vector * HypreParVector::GlobalVector() const
 {
@@ -192,7 +185,7 @@ Vector * HypreParVector::GlobalVector() const
 
 HypreParVector& HypreParVector::operator=(double d)
 {
-   hypre_ParVectorSetConstantValues(x,d);
+   Vector::operator=(d);
    return *this;
 }
 
@@ -205,10 +198,7 @@ HypreParVector& HypreParVector::operator=(const HypreParVector &y)
    }
 #endif
 
-   for (int i = 0; i < size; i++)
-   {
-      data[i] = y.data[i];
-   }
+   Vector::operator=(y);
    return *this;
 }
 
@@ -238,10 +228,6 @@ HypreParVector::~HypreParVector()
 
 #ifdef MFEM_USE_SUNDIALS
 
-#ifndef SUNFALSE
-#define SUNFALSE FALSE
-#endif
-
 void HypreParVector::ToNVector(N_Vector &nv)
 {
    MFEM_ASSERT(nv && N_VGetVectorID(nv) == SUNDIALS_NVEC_PARHYP,
@@ -981,26 +967,17 @@ static void MakeWrapper(const hypre_CSRMatrix *mat, SparseMatrix &wrapper)
    wrapper.Swap(tmp);
 }
 
-
 void HypreParMatrix::GetDiag(SparseMatrix &diag) const
 {
    MakeWrapper(A->diag, diag);
 }
 
-
 void HypreParMatrix::GetOffd(SparseMatrix &offd, HYPRE_Int* &cmap) const
 {
    MakeWrapper(A->offd, offd);
    cmap = A->col_map_offd;
 }
 
-
-void HypreParMatrix::GetProcRows(SparseMatrix &colCSRMat)
-{
-   MakeWrapper(hypre_MergeDiagAndOffd(A), colCSRMat);
-}
-
-
 void HypreParMatrix::GetBlocks(Array2D<HypreParMatrix*> &blocks,
                                bool interleaved_rows,
                                bool interleaved_cols) const
@@ -1044,6 +1021,8 @@ HypreParMatrix * HypreParMatrix::Transpose() const
 HYPRE_Int HypreParMatrix::Mult(HypreParVector &x, HypreParVector &y,
                                double a, double b)
 {
+   x.HostRead();
+   (b == 0.0) ? y.HostWrite() : y.HostReadWrite();
    return hypre_ParCSRMatrixMatvec(a, A, x, b, y);
 }
 
@@ -1055,7 +1034,7 @@ void HypreParMatrix::Mult(double a, const Vector &x, double b, Vector &y) const
                << ", expected size = " << Height());
 
    auto x_data = x.HostRead();
-   auto y_data = y.HostWrite();
+   auto y_data = (b == 0.0) ? y.HostWrite() : y.HostReadWrite();
    if (X == NULL)
    {
       X = new HypreParVector(A->comm,
@@ -1087,7 +1066,7 @@ void HypreParMatrix::MultTranspose(double a, const Vector &x,
    // Note: x has the dimensions of Y (height), and
    //       y has the dimensions of X (width)
    auto x_data = x.HostRead();
-   auto y_data = y.HostWrite();
+   auto y_data = (b == 0.0) ? y.HostWrite() : y.HostReadWrite();
    if (X == NULL)
    {
       X = new HypreParVector(A->comm,
@@ -1561,40 +1540,6 @@ void HypreParMatrix::Destroy()
    }
 }
 
-/* job =  0, extract block diagonal of A and scale A into C
- * job =  1, job 0 + scale b into d
- * job =  2, use A to scale b only
- */
-int BlockInvScal(const HypreParMatrix *A, HypreParMatrix *C,
-                 const Vector *b, HypreParVector *d, int block, int job)
-{
-   if (0 == job || 1 == job)
-   {
-      hypre_ParCSRMatrix *C_hypre;
-      hypre_ParcsrBdiagInvScal(*A, block, &C_hypre);
-      /* XXX: FIXME drop in BdiagInvScal */
-      hypre_ParCSRMatrixDropSmallEntries(C_hypre, 1e-15, 1);
-      (*C).WrapHypreParCSRMatrix(C_hypre);
-   }
-
-   if (1 == job || 2 == job)
-   {
-      HypreParVector *b_Hypre = new HypreParVector(A->GetComm(), A->GetGlobalNumRows(), 
-                                                   b->GetData(), A->GetRowStarts());
-      hypre_ParVector *d_hypre;
-      hypre_ParvecBdiagInvScal(*b_Hypre, block, &d_hypre, *A);
-      
-      delete b_Hypre;
-
-      d->WrapHypreParVector(d_hypre);
-      d->SetOwnership(true);
-
-      return 0;
-   }
-
-   return -1;
-}
-
 HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
                     double beta,  const HypreParMatrix &B)
 {
@@ -1612,18 +1557,6 @@ HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
    return C;
 }
 
-HypreParMatrix *HypreParMatrixAdd(double alpha, const HypreParMatrix &A,
-                                  double beta,  const HypreParMatrix &B)
-{
-   hypre_ParCSRMatrix *C_hypre;
-
-   hypre_ParcsrAdd(alpha, A, beta, B, &C_hypre);
-
-   HypreParMatrix *C = new HypreParMatrix(C_hypre);
-
-   return C;
-}
-
 HypreParMatrix * ParMult(const HypreParMatrix *A, const HypreParMatrix *B)
 {
    hypre_ParCSRMatrix * ab;
@@ -2075,10 +2008,12 @@ void HypreSmoother::Mult(const HypreParVector &b, HypreParVector &x) const
       return;
    }
 
+   b.HostRead();
    if (!iterative_mode)
    {
       if (type == 0 && relax_times == 1)
       {
+         x.HostWrite();
          HYPRE_ParCSRDiagScale(NULL, *A, b, x);
          if (relax_weight != 1.0)
          {
@@ -2088,6 +2023,7 @@ void HypreSmoother::Mult(const HypreParVector &b, HypreParVector &x) const
       }
       x = 0.0;
    }
+   x.HostReadWrite();
 
    if (V == NULL)
    {
@@ -2137,21 +2073,25 @@ void HypreSmoother::Mult(const Vector &b, Vector &x) const
       mfem_error("HypreSmoother::Mult (...) : HypreParMatrix A is missing");
       return;
    }
+
+   auto b_data = b.HostRead();
+   auto x_data = iterative_mode ? x.HostReadWrite() : x.HostWrite();
+
    if (B == NULL)
    {
       B = new HypreParVector(A->GetComm(),
                              A -> GetGlobalNumRows(),
-                             b.GetData(),
+                             const_cast<double*>(b_data),
                              A -> GetRowStarts());
       X = new HypreParVector(A->GetComm(),
                              A -> GetGlobalNumCols(),
-                             x.GetData(),
+                             x_data,
                              A -> GetColStarts());
    }
    else
    {
-      B -> SetData(b.GetData());
-      X -> SetData(x.GetData());
+      B -> SetData(const_cast<double*>(b_data));
+      X -> SetData(x_data);
    }
 
    Mult(*B, *X);
@@ -2268,7 +2208,14 @@ HypreSolver::~HypreSolver()
 }
 
 
-HyprePCG::HyprePCG(HypreParMatrix &_A) : HypreSolver(&_A)
+HyprePCG::HyprePCG(MPI_Comm comm) : precond(NULL)
+{
+   iterative_mode = true;
+
+   HYPRE_ParCSRPCGCreate(comm, &pcg_solver);
+}
+
+HyprePCG::HyprePCG(HypreParMatrix &_A) : HypreSolver(&_A), precond(NULL)
 {
    MPI_Comm comm;
 
@@ -2279,6 +2226,26 @@ HyprePCG::HyprePCG(HypreParMatrix &_A) : HypreSolver(&_A)
    HYPRE_ParCSRPCGCreate(comm, &pcg_solver);
 }
 
+void HyprePCG::SetOperator(const Operator &op)
+{
+   const HypreParMatrix *new_A = dynamic_cast<const HypreParMatrix *>(&op);
+   MFEM_VERIFY(new_A, "new Operator must be a HypreParMatrix!");
+
+   // update base classes: Operator, Solver, HypreSolver
+   height = new_A->Height();
+   width  = new_A->Width();
+   A = const_cast<HypreParMatrix *>(new_A);
+   if (precond)
+   {
+      precond->SetOperator(*A);
+      this->SetPreconditioner(*precond);
+   }
+   setup_called = 0;
+   delete X;
+   delete B;
+   B = X = NULL;
+}
+
 void HyprePCG::SetTol(double tol)
 {
    HYPRE_PCGSetTol(pcg_solver, tol);
@@ -2299,12 +2266,14 @@ void HyprePCG::SetPrintLevel(int print_lvl)
    HYPRE_ParCSRPCGSetPrintLevel(pcg_solver, print_lvl);
 }
 
-void HyprePCG::SetPreconditioner(HypreSolver &precond)
+void HyprePCG::SetPreconditioner(HypreSolver &_precond)
 {
+   precond = &_precond;
+
    HYPRE_ParCSRPCGSetPrecond(pcg_solver,
-                             precond.SolveFcn(),
-                             precond.SetupFcn(),
-                             precond);
+                             _precond.SolveFcn(),
+                             _precond.SetupFcn(),
+                             _precond);
 }
 
 void HyprePCG::SetResidualConvergenceOptions(int res_frequency, double rtol)
@@ -2402,35 +2371,62 @@ HyprePCG::~HyprePCG()
 }
 
 
+HypreGMRES::HypreGMRES(MPI_Comm comm) : precond(NULL)
+{
+   iterative_mode = true;
+
+   HYPRE_ParCSRGMRESCreate(comm, &gmres_solver);
+   SetDefaultOptions();
+}
+
 HypreGMRES::HypreGMRES(HypreParMatrix &_A) : HypreSolver(&_A)
 {
    MPI_Comm comm;
 
-   int k_dim    = 50;
-   int max_iter = 100;
-   double tol   = 1e-6;
-
    iterative_mode = true;
 
    HYPRE_ParCSRMatrixGetComm(*A, &comm);
 
    HYPRE_ParCSRGMRESCreate(comm, &gmres_solver);
+   SetDefaultOptions();
+}
+
+void HypreGMRES::SetDefaultOptions()
+{
+   int k_dim    = 50;
+   int max_iter = 100;
+   double tol   = 1e-6;
+
    HYPRE_ParCSRGMRESSetKDim(gmres_solver, k_dim);
    HYPRE_ParCSRGMRESSetMaxIter(gmres_solver, max_iter);
    HYPRE_ParCSRGMRESSetTol(gmres_solver, tol);
 }
 
+void HypreGMRES::SetOperator(const Operator &op)
+{
+   const HypreParMatrix *new_A = dynamic_cast<const HypreParMatrix *>(&op);
+   MFEM_VERIFY(new_A, "new Operator must be a HypreParMatrix!");
+
+   // update base classes: Operator, Solver, HypreSolver
+   height = new_A->Height();
+   width  = new_A->Width();
+   A = const_cast<HypreParMatrix *>(new_A);
+   if (precond)
+   {
+      precond->SetOperator(*A);
+      this->SetPreconditioner(*precond);
+   }
+   setup_called = 0;
+   delete X;
+   delete B;
+   B = X = NULL;
+}
+
 void HypreGMRES::SetTol(double tol)
 {
    HYPRE_GMRESSetTol(gmres_solver, tol);
 }
 
-void HypreGMRES::SetAbsTol(double tol)
-{
-   HYPRE_GMRESSetTol(gmres_solver, 0.0);
-   HYPRE_GMRESSetAbsoluteTol(gmres_solver, tol);
-}
-
 void HypreGMRES::SetMaxIter(int max_iter)
 {
    HYPRE_GMRESSetMaxIter(gmres_solver, max_iter);
@@ -2451,12 +2447,14 @@ void HypreGMRES::SetPrintLevel(int print_lvl)
    HYPRE_GMRESSetPrintLevel(gmres_solver, print_lvl);
 }
 
-void HypreGMRES::SetPreconditioner(HypreSolver &precond)
+void HypreGMRES::SetPreconditioner(HypreSolver &_precond)
 {
+   precond = &_precond;
+
    HYPRE_ParCSRGMRESSetPrecond(gmres_solver,
-                               precond.SolveFcn(),
-                               precond.SetupFcn(),
-                               precond);
+                               _precond.SolveFcn(),
+                               _precond.SetupFcn(),
+                               _precond);
 }
 
 void HypreGMRES::Mult(const HypreParVector &b, HypreParVector &x) const
@@ -2533,10 +2531,40 @@ HypreGMRES::~HypreGMRES()
 }
 
 
+void HypreDiagScale::SetOperator(const Operator &op)
+{
+   const HypreParMatrix *new_A = dynamic_cast<const HypreParMatrix *>(&op);
+   MFEM_VERIFY(new_A, "new Operator must be a HypreParMatrix!");
+
+   // update base classes: Operator, Solver, HypreSolver
+   height = new_A->Height();
+   width  = new_A->Width();
+   A = const_cast<HypreParMatrix *>(new_A);
+   setup_called = 0;
+   delete X;
+   delete B;
+   B = X = NULL;
+}
+
+
+HypreParaSails::HypreParaSails(MPI_Comm comm)
+{
+   HYPRE_ParaSailsCreate(comm, &sai_precond);
+   SetDefaultOptions();
+}
+
 HypreParaSails::HypreParaSails(HypreParMatrix &A) : HypreSolver(&A)
 {
    MPI_Comm comm;
 
+   HYPRE_ParCSRMatrixGetComm(A, &comm);
+
+   HYPRE_ParaSailsCreate(comm, &sai_precond);
+   SetDefaultOptions();
+}
+
+void HypreParaSails::SetDefaultOptions()
+{
    int    sai_max_levels = 1;
    double sai_threshold  = 0.1;
    double sai_filter     = 0.1;
@@ -2545,9 +2573,6 @@ HypreParaSails::HypreParaSails(HypreParMatrix &A) : HypreSolver(&A)
    int    sai_reuse      = 0;
    int    sai_logging    = 1;
 
-   HYPRE_ParCSRMatrixGetComm(A, &comm);
-
-   HYPRE_ParaSailsCreate(comm, &sai_precond);
    HYPRE_ParaSailsSetParams(sai_precond, sai_threshold, sai_max_levels);
    HYPRE_ParaSailsSetFilter(sai_precond, sai_filter);
    HYPRE_ParaSailsSetSym(sai_precond, sai_sym);
@@ -2556,6 +2581,58 @@ HypreParaSails::HypreParaSails(HypreParMatrix &A) : HypreSolver(&A)
    HYPRE_ParaSailsSetLogging(sai_precond, sai_logging);
 }
 
+void HypreParaSails::ResetSAIPrecond(MPI_Comm comm)
+{
+   HYPRE_Int  sai_max_levels;
+   HYPRE_Real sai_threshold;
+   HYPRE_Real sai_filter;
+   HYPRE_Int  sai_sym;
+   HYPRE_Real sai_loadbal;
+   HYPRE_Int  sai_reuse;
+   HYPRE_Int  sai_logging;
+
+   // hypre_ParAMGData *amg_data = (hypre_ParAMGData *)sai_precond;
+   HYPRE_ParaSailsGetNlevels(sai_precond, &sai_max_levels);
+   HYPRE_ParaSailsGetThresh(sai_precond, &sai_threshold);
+   HYPRE_ParaSailsGetFilter(sai_precond, &sai_filter);
+   HYPRE_ParaSailsGetSym(sai_precond, &sai_sym);
+   HYPRE_ParaSailsGetLoadbal(sai_precond, &sai_loadbal);
+   HYPRE_ParaSailsGetReuse(sai_precond, &sai_reuse);
+   HYPRE_ParaSailsGetLogging(sai_precond, &sai_logging);
+
+   HYPRE_ParaSailsDestroy(sai_precond);
+   HYPRE_ParaSailsCreate(comm, &sai_precond);
+
+   HYPRE_ParaSailsSetParams(sai_precond, sai_threshold, sai_max_levels);
+   HYPRE_ParaSailsSetFilter(sai_precond, sai_filter);
+   HYPRE_ParaSailsSetSym(sai_precond, sai_sym);
+   HYPRE_ParaSailsSetLoadbal(sai_precond, sai_loadbal);
+   HYPRE_ParaSailsSetReuse(sai_precond, sai_reuse);
+   HYPRE_ParaSailsSetLogging(sai_precond, sai_logging);
+}
+
+void HypreParaSails::SetOperator(const Operator &op)
+{
+   const HypreParMatrix *new_A = dynamic_cast<const HypreParMatrix *>(&op);
+   MFEM_VERIFY(new_A, "new Operator must be a HypreParMatrix!");
+
+   if (A)
+   {
+      MPI_Comm comm;
+      HYPRE_ParCSRMatrixGetComm(*A, &comm);
+      ResetSAIPrecond(comm);
+   }
+
+   // update base classes: Operator, Solver, HypreSolver
+   height = new_A->Height();
+   width  = new_A->Width();
+   A = const_cast<HypreParMatrix *>(new_A);
+   setup_called = 0;
+   delete X;
+   delete B;
+   B = X = NULL;
+}
+
 void HypreParaSails::SetSymmetry(int sym)
 {
    HYPRE_ParaSailsSetSym(sai_precond, sym);
@@ -2567,19 +2644,30 @@ HypreParaSails::~HypreParaSails()
 }
 
 
+HypreEuclid::HypreEuclid(MPI_Comm comm)
+{
+   HYPRE_EuclidCreate(comm, &euc_precond);
+   SetDefaultOptions();
+}
+
 HypreEuclid::HypreEuclid(HypreParMatrix &A) : HypreSolver(&A)
 {
    MPI_Comm comm;
 
+   HYPRE_ParCSRMatrixGetComm(A, &comm);
+
+   HYPRE_EuclidCreate(comm, &euc_precond);
+   SetDefaultOptions();
+}
+
+void HypreEuclid::SetDefaultOptions()
+{
    int    euc_level = 1; // We use ILU(1)
    int    euc_stats = 0; // No logging
    int    euc_mem   = 0; // No memory logging
    int    euc_bj    = 0; // 1: Use Block Jacobi
    int    euc_ro_sc = 0; // 1: Use Row scaling
 
-   HYPRE_ParCSRMatrixGetComm(A, &comm);
-
-   HYPRE_EuclidCreate(comm, &euc_precond);
    HYPRE_EuclidSetLevel(euc_precond, euc_level);
    HYPRE_EuclidSetStats(euc_precond, euc_stats);
    HYPRE_EuclidSetMem(euc_precond, euc_mem);
@@ -2587,6 +2675,38 @@ HypreEuclid::HypreEuclid(HypreParMatrix &A) : HypreSolver(&A)
    HYPRE_EuclidSetRowScale(euc_precond, euc_ro_sc);
 }
 
+void HypreEuclid::ResetEuclidPrecond(MPI_Comm comm)
+{
+   // Euclid does not seem to offer access to its current configuration, so we
+   // simply reset it to its default options.
+   HYPRE_EuclidDestroy(euc_precond);
+   HYPRE_EuclidCreate(comm, &euc_precond);
+
+   SetDefaultOptions();
+}
+
+void HypreEuclid::SetOperator(const Operator &op)
+{
+   const HypreParMatrix *new_A = dynamic_cast<const HypreParMatrix *>(&op);
+   MFEM_VERIFY(new_A, "new Operator must be a HypreParMatrix!");
+
+   if (A)
+   {
+      MPI_Comm comm;
+      HYPRE_ParCSRMatrixGetComm(*new_A, &comm);
+      ResetEuclidPrecond(comm);
+   }
+
+   // update base classes: Operator, Solver, HypreSolver
+   height = new_A->Height();
+   width  = new_A->Width();
+   A = const_cast<HypreParMatrix *>(new_A);
+   setup_called = 0;
+   delete X;
+   delete B;
+   B = X = NULL;
+}
+
 HypreEuclid::~HypreEuclid()
 {
    HYPRE_EuclidDestroy(euc_precond);
@@ -2605,75 +2725,6 @@ HypreBoomerAMG::HypreBoomerAMG(HypreParMatrix &A) : HypreSolver(&A)
    SetDefaultOptions();
 }
 
-void HypreBoomerAMG::Mult(const HypreParVector &b, HypreParVector &x) const
-{
-   int myid;
-   HYPRE_Int time_index = 0;
-   HYPRE_Int num_iterations;
-   double final_res_norm;
-   MPI_Comm comm;
-   HYPRE_Int print_level;
-
-   HYPRE_BoomerAMGGetPrintLevel(amg_precond, &print_level);
-
-   HYPRE_ParCSRMatrixGetComm(*A, &comm);
-
-   if (!setup_called)
-   {
-      if (print_level > 0)
-      {
-         time_index = hypre_InitializeTiming("BoomerAMG Setup");
-         hypre_BeginTiming(time_index);
-      }
-
-      HYPRE_BoomerAMGSetup(amg_precond, *A, b, x);
-      setup_called = 1;
-
-      if (print_level > 0)
-      {
-         hypre_EndTiming(time_index);
-         hypre_PrintTiming("Setup phase times", comm);
-         hypre_FinalizeTiming(time_index);
-         hypre_ClearTiming();
-      }
-   }
-
-   if (print_level > 0)
-   {
-      time_index = hypre_InitializeTiming("BoomerAMG Solve");
-      hypre_BeginTiming(time_index);
-   }
-
-   if (!iterative_mode)
-   {
-      x = 0.0;
-   }
-
-   HYPRE_BoomerAMGSolve(amg_precond, *A, b, x);
-
-   if (print_level > 0)
-   {
-      hypre_EndTiming(time_index);
-      hypre_PrintTiming("Solve phase times", comm);
-      hypre_FinalizeTiming(time_index);
-      hypre_ClearTiming();
-
-      HYPRE_BoomerAMGGetNumIterations(amg_precond, &num_iterations);
-      HYPRE_BoomerAMGGetFinalRelativeResidualNorm(amg_precond,
-                                                    &final_res_norm);
-
-      MPI_Comm_rank(comm, &myid);
-
-      if (myid == 0)
-      {
-         mfem::out << "BoomerAMG Iterations = " << num_iterations << endl
-                   << "Final Relative Residual Norm = " << final_res_norm
-                   << endl;
-      }
-   }
-}
-
-
 void HypreBoomerAMG::SetDefaultOptions()
 {
    // AMG coarsening options:
@@ -2921,234 +2972,6 @@ void HypreBoomerAMG::SetElasticityOptions(ParFiniteElementSpace *fespace)
    error_mode = IGNORE_HYPRE_ERRORS;
 }
 
-void HypreBoomerAMG::SetCoord(int coord_dim, float *coord)
-{
-   HYPRE_BoomerAMGSetPlotGrids (amg_precond, 1);
-   //HYPRE_BoomerAMGSetPlotFileName (amg_precond, plot_file_name);
-   HYPRE_BoomerAMGSetCoordDim (amg_precond, coord_dim);
-   HYPRE_BoomerAMGSetCoordinates (amg_precond, coord);
-}
-
-
-void HypreBoomerAMG::SetLAIROptions(int distance,
-                                    std::string prerelax,
-                                    std::string postrelax, 
-                                    double strength_tolC,
-                                    double strength_tolR,
-                                    double filter_tolR,
-                                    int interp_type, 
-                                    int relax_type,
-                                    double filterA_tol, 
-                                    int splitting,
-                                    int blksize,
-                                    int Sabs)
-{
-   int ns_down, ns_up, ns_coarse;
-   if (distance > 0)
-   {
-      ns_down = prerelax.length();
-      ns_up = postrelax.length();
-      ns_coarse = 1;
-      std::string F("F");
-      std::string C("C");
-      std::string A("A");
-
-      // Array to store relaxation scheme and pass to Hypre
-      int **grid_relax_points = (int **) malloc(4*sizeof(int *));
-      grid_relax_points[0] = NULL;
-      grid_relax_points[1] = (int *) malloc(sizeof(int)*ns_down);
-      grid_relax_points[2] = (int *) malloc(sizeof(int)*ns_up);
-      grid_relax_points[3] = (int *) malloc(sizeof(int));
-      grid_relax_points[3][0] = 0;
-
-      // set down relax scheme 
-      for(unsigned int i = 0; i<ns_down; i++) {
-         if (prerelax.compare(i,1,F) == 0) {
-            grid_relax_points[1][i] = -1;
-         }
-         else if (prerelax.compare(i,1,C) == 0) {
-            grid_relax_points[1][i] = 1;
-         }
-         else if (prerelax.compare(i,1,A) == 0) {
-            grid_relax_points[1][i] = 0;
-         }
-      }
-
-      // set up relax scheme 
-      for(unsigned int i = 0; i<ns_up; i++) {
-         if (postrelax.compare(i,1,F) == 0) {
-            grid_relax_points[2][i] = -1;
-         }
-         else if (postrelax.compare(i,1,C) == 0) {
-            grid_relax_points[2][i] = 1;
-         }
-         else if (postrelax.compare(i,1,A) == 0) {
-            grid_relax_points[2][i] = 0;
-         }
-      }
-
-      HYPRE_BoomerAMGSetRestriction(amg_precond, distance);
-
-      HYPRE_BoomerAMGSetGridRelaxPoints(amg_precond, grid_relax_points);
-
-      HYPRE_BoomerAMGSetInterpType(amg_precond, interp_type);
-   }
-   
-   //HYPRE_BoomerAMGSetMaxRowSum(amg_precond, 0.8);
-   if (Sabs)
-   {
-      HYPRE_BoomerAMGSetSabs(amg_precond, Sabs);
-   }
-
-   if (blksize > 0)
-   {
-      HYPRE_BoomerAMGSetNumFunctions(amg_precond, blksize);
-      HYPRE_BoomerAMGSetNodal(amg_precond, 1);
-      //HYPRE_BoomerAMGSetNodalLevels(amg_precond, 1);
-   }
-
-   HYPRE_BoomerAMGSetCoarsenType(amg_precond, splitting);
-   
-   /* does not support aggressive coarsening */
-   HYPRE_BoomerAMGSetAggNumLevels(amg_precond, 0);
-   
-   HYPRE_BoomerAMGSetStrongThreshold(amg_precond, strength_tolC);
-   
-   if (distance > 0)
-   {
-      HYPRE_BoomerAMGSetStrongThresholdR(amg_precond, strength_tolR);
-      HYPRE_BoomerAMGSetFilterThresholdR(amg_precond, filter_tolR);
-   }
-
-   if (relax_type > -1)
-   {
-      HYPRE_BoomerAMGSetRelaxType(amg_precond, relax_type);
-   }
-
-   if (distance > 0)
-   {
-      HYPRE_BoomerAMGSetCycleNumSweeps(amg_precond, ns_coarse, 3);
-      HYPRE_BoomerAMGSetCycleNumSweeps(amg_precond, ns_down,   1);
-      HYPRE_BoomerAMGSetCycleNumSweeps(amg_precond, ns_up,     2);
-
-      HYPRE_BoomerAMGSetADropTol(amg_precond, filterA_tol);
-      /* type = -1: drop based on row inf-norm */
-      HYPRE_BoomerAMGSetADropType(amg_precond, -1);
-   }
-
-   //HYPRE_BoomerAMGSetMaxCoarseSize(amg_precond, 1000);
-}
-
-
-void HypreBoomerAMG::SetNAIROptions(int neumann_degree,
-                                    std::string prerelax,
-                                    std::string postrelax, 
-                                    double strength_tolC,
-                                    double strength_tolR,
-                                    double filter_tolR,
-                                    int interp_type, 
-                                    int relax_type,
-                                    double filterA_tol, 
-                                    int splitting,
-                                    int blksize,
-                                    int Sabs)
-{
-   int ns_down, ns_up, ns_coarse;
-   if (neumann_degree > 0)
-   {
-      ns_down = prerelax.length();
-      ns_up = postrelax.length();
-      ns_coarse = 1;
-      std::string F("F");
-      std::string C("C");
-      std::string A("A");
-
-      // Array to store relaxation scheme and pass to Hypre
-      int **grid_relax_points = (int **) malloc(4*sizeof(int *));
-      grid_relax_points[0] = NULL;
-      grid_relax_points[1] = (int *) malloc(sizeof(int)*ns_down);
-      grid_relax_points[2] = (int *) malloc(sizeof(int)*ns_up);
-      grid_relax_points[3] = (int *) malloc(sizeof(int));
-      grid_relax_points[3][0] = 0;
-
-      // set down relax scheme 
-      for(unsigned int i = 0; i<ns_down; i++) {
-         if (prerelax.compare(i,1,F) == 0) {
-            grid_relax_points[1][i] = -1;
-         }
-         else if (prerelax.compare(i,1,C) == 0) {
-            grid_relax_points[1][i] = 1;
-         }
-         else if (prerelax.compare(i,1,A) == 0) {
-            grid_relax_points[1][i] = 0;
-         }
-      }
-
-      // set up relax scheme 
-      for(unsigned int i = 0; i<ns_up; i++) {
-         if (postrelax.compare(i,1,F) == 0) {
-            grid_relax_points[2][i] = -1;
-         }
-         else if (postrelax.compare(i,1,C) == 0) {
-            grid_relax_points[2][i] = 1;
-         }
-         else if (postrelax.compare(i,1,A) == 0) {
-            grid_relax_points[2][i] = 0;
-         }
-      }
-
-      HYPRE_BoomerAMGSetRestriction(amg_precond, 3+neumann_degree);
-
-      HYPRE_BoomerAMGSetGridRelaxPoints(amg_precond, grid_relax_points);
-
-      HYPRE_BoomerAMGSetInterpType(amg_precond, interp_type);
-   }
-   
-   //HYPRE_BoomerAMGSetMaxRowSum(amg_precond, 0.8);
-   if (Sabs)
-   {
-      HYPRE_BoomerAMGSetSabs(amg_precond, Sabs);
-   }
-
-   if (blksize > 0)
-   {
-      HYPRE_BoomerAMGSetNumFunctions(amg_precond, blksize);
-      HYPRE_BoomerAMGSetNodal(amg_precond, 1);
-      //HYPRE_BoomerAMGSetNodalLevels(amg_precond, 1);
-   }
-
-   HYPRE_BoomerAMGSetCoarsenType(amg_precond, splitting);
-   
-   /* does not support aggressive coarsening */
-   HYPRE_BoomerAMGSetAggNumLevels(amg_precond, 0);
-   
-   HYPRE_BoomerAMGSetStrongThreshold(amg_precond, strength_tolC);
-   
-   if (neumann_degree > 0)
-   {
-      HYPRE_BoomerAMGSetStrongThresholdR(amg_precond, strength_tolR);
-   }
-
-   if (relax_type > -1)
-   {
-      HYPRE_BoomerAMGSetRelaxType(amg_precond, relax_type);
-   }
-
-   if (neumann_degree > 0)
-   {
-      HYPRE_BoomerAMGSetCycleNumSweeps(amg_precond, ns_coarse, 3);
-      HYPRE_BoomerAMGSetCycleNumSweeps(amg_precond, ns_down,   1);
-      HYPRE_BoomerAMGSetCycleNumSweeps(amg_precond, ns_up,     2);
-
-      HYPRE_BoomerAMGSetADropTol(amg_precond, filterA_tol);
-      /* type = -1: drop based on row inf-norm */
-      HYPRE_BoomerAMGSetADropType(amg_precond, -1);
-   }
-
-   //HYPRE_BoomerAMGSetMaxCoarseSize(amg_precond, 1000);
-}
-
-
 HypreBoomerAMG::~HypreBoomerAMG()
 {
    for (int i = 0; i < rbms.Size(); i++)
@@ -3159,9 +2982,18 @@ HypreBoomerAMG::~HypreBoomerAMG()
    HYPRE_BoomerAMGDestroy(amg_precond);
 }
 
+HypreAMS::HypreAMS(ParFiniteElementSpace *edge_fespace)
+{
+   Init(edge_fespace);
+}
 
 HypreAMS::HypreAMS(HypreParMatrix &A, ParFiniteElementSpace *edge_fespace)
    : HypreSolver(&A)
+{
+   Init(edge_fespace);
+}
+
+void HypreAMS::Init(ParFiniteElementSpace *edge_fespace)
 {
    int cycle_type       = 13;
    int rlx_type         = 2;
@@ -3345,6 +3177,22 @@ HypreAMS::HypreAMS(HypreParMatrix &A, ParFiniteElementSpace *edge_fespace)
    error_mode = IGNORE_HYPRE_ERRORS;
 }
 
+void HypreAMS::SetOperator(const Operator &op)
+{
+   const HypreParMatrix *new_A = dynamic_cast<const HypreParMatrix *>(&op);
+   MFEM_VERIFY(new_A, "new Operator must be a HypreParMatrix!");
+
+   // update base classes: Operator, Solver, HypreSolver
+   height = new_A->Height();
+   width  = new_A->Width();
+   A = const_cast<HypreParMatrix *>(new_A);
+
+   setup_called = 0;
+   delete X;
+   delete B;
+   B = X = NULL;
+}
+
 HypreAMS::~HypreAMS()
 {
    HYPRE_AMSDestroy(ams);
@@ -3365,8 +3213,18 @@ void HypreAMS::SetPrintLevel(int print_lvl)
    HYPRE_AMSSetPrintLevel(ams, print_lvl);
 }
 
+HypreADS::HypreADS(ParFiniteElementSpace *face_fespace)
+{
+   Init(face_fespace);
+}
+
 HypreADS::HypreADS(HypreParMatrix &A, ParFiniteElementSpace *face_fespace)
    : HypreSolver(&A)
+{
+   Init(face_fespace);
+}
+
+void HypreADS::Init(ParFiniteElementSpace *face_fespace)
 {
    int cycle_type       = 11;
    int rlx_type         = 2;
@@ -3589,6 +3447,22 @@ HypreADS::HypreADS(HypreParMatrix &A, ParFiniteElementSpace *face_fespace)
    error_mode = IGNORE_HYPRE_ERRORS;
 }
 
+void HypreADS::SetOperator(const Operator &op)
+{
+   const HypreParMatrix *new_A = dynamic_cast<const HypreParMatrix *>(&op);
+   MFEM_VERIFY(new_A, "new Operator must be a HypreParMatrix!");
+
+   // update base classes: Operator, Solver, HypreSolver
+   height = new_A->Height();
+   width  = new_A->Width();
+   A = const_cast<HypreParMatrix *>(new_A);
+
+   setup_called = 0;
+   delete X;
+   delete B;
+   B = X = NULL;
+}
+
 HypreADS::~HypreADS()
 {
    HYPRE_ADSDestroy(ads);
@@ -4161,4 +4035,4 @@ HypreAME::StealEigenvectors()
 
 }
 
-#endif
+#endif

linalg/hypre.hpp

@@ -84,9 +84,6 @@ private:
    inline void _SetDataAndSize_();
 
 public:
-
-  HypreParVector() {}
-  
    /** @brief Creates vector with given global size and parallel partitioning of
        the rows/columns given by @a col. */
    /** @anchor hypre_partitioning_descr
@@ -119,9 +116,9 @@ public:
    /// MPI communicator
    MPI_Comm GetComm() { return x->comm; }
 
-   void WrapHypreParVector(hypre_ParVector *y);
-   
-   /// Returns the row partitioning
+   /// Returns the parallel row/column partitioning
+   /** See @ref hypre_partitioning_descr "here" for a description of the
+       partitioning array. */
    inline HYPRE_Int *Partitioning() { return x->partitioning; }
 
    /// Returns the global number of rows
@@ -237,25 +234,22 @@ public:
    /// An empty matrix to be used as a reference to an existing matrix
    HypreParMatrix();
 
-   void WrapHypreParCSRMatrix(hypre_ParCSRMatrix *a, bool owner = true)
+   /// Converts hypre's format to HypreParMatrix
+   /** If @a owner is false, ownership of @a a is not transferred */
+   explicit HypreParMatrix(hypre_ParCSRMatrix *a, bool owner = true)
    {
+      Init();
       A = a;
       if (!owner) { ParCSROwner = 0; }
       height = GetNumRows();
       width = GetNumCols();
    }
 
-   /// Converts hypre's format to HypreParMatrix
-   /** If @a owner is false, ownership of @a a is not transferred */
-   explicit HypreParMatrix(hypre_ParCSRMatrix *a, bool owner = true)
-   {
-      Init();
-      WrapHypreParCSRMatrix(a, owner);
-   }
-
-   /** Creates block-diagonal square parallel matrix. Diagonal is given by diag
-       which must be in CSR format (finalized). The new HypreParMatrix does not
-       take ownership of any of the input arrays.
+   /// Creates block-diagonal square parallel matrix.
+   /** Diagonal is given by @a diag which must be in CSR format (finalized). The
+       new HypreParMatrix does not take ownership of any of the input arrays.
+       See @ref hypre_partitioning_descr "here" for a description of the row
+       partitioning array @a row_starts.
 
        @warning The ordering of the columns in each row in @a *diag may be
        changed by this constructor to ensure that the first entry in each row is
@@ -399,8 +393,6 @@ public:
    void GetDiag(SparseMatrix &diag) const;
    /// Get the local off-diagonal block. NOTE: 'offd' will not own any data.
    void GetOffd(SparseMatrix &offd, HYPRE_Int* &cmap) const;
-   /// Get on-processor rows as CSR matrix.
-   void GetProcRows(SparseMatrix &colCSRMat);
 
    /** Split the matrix into M x N equally sized blocks of parallel matrices.
        The size of 'blocks' must already be set to M x N. */
@@ -549,18 +541,12 @@ public:
    Type GetType() const { return Hypre_ParCSR; }
 };
 
-int BlockInvScal(const HypreParMatrix *A, HypreParMatrix *C,
-                 const Vector *b, HypreParVector *d, int block, int job);
-
 /** @brief Return a new matrix `C = alpha*A + beta*B`, assuming that both `A`
     and `B` use the same row and column partitions and the same `col_map_offd`
     arrays. */
 HypreParMatrix *Add(double alpha, const HypreParMatrix &A,
                     double beta,  const HypreParMatrix &B);
 
-HypreParMatrix *HypreParMatrixAdd(double alpha, const HypreParMatrix &A,
-                                  double beta,  const HypreParMatrix &B);
-
 /// Returns the matrix A * B
 HypreParMatrix * ParMult(const HypreParMatrix *A, const HypreParMatrix *B);
 /// Returns the matrix A + B
@@ -641,7 +627,7 @@ public:
        1001 = Taubin polynomial smoother
        1002 = FIR polynomial smoother. */
    enum Type { Jacobi = 0, l1Jacobi = 1, l1GS = 2, l1GStr = 4, lumpedJacobi = 5,
-               GS = 6, TS = 10, Chebyshev = 16, Taubin = 1001, FIR = 1002
+               GS = 6, Chebyshev = 16, Taubin = 1001, FIR = 1002
              };
 
    HypreSmoother();
@@ -743,34 +729,21 @@ public:
    virtual ~HypreSolver();
 };
 
-
-/// Abstract class for hypre's solvers and preconditioners
-class HypreTriSolve : public HypreSolver
-{
-public:
-   HypreTriSolve() : HypreSolver() { }
-   explicit HypreTriSolve(HypreParMatrix &A) : HypreSolver(&A) { }
-   virtual operator HYPRE_Solver() const { return NULL; }
-
-   virtual HYPRE_PtrToParSolverFcn SetupFcn() const
-   { return (HYPRE_PtrToParSolverFcn) HYPRE_ParCSROnProcTriSetup; }
-   virtual HYPRE_PtrToParSolverFcn SolveFcn() const
-   { return (HYPRE_PtrToParSolverFcn) HYPRE_ParCSROnProcTriSolve; }
-
-   HypreParMatrix* GetData() { return A; }
-   virtual ~HypreTriSolve() { }
-};
-
-
 /// PCG solver in hypre
 class HyprePCG : public HypreSolver
 {
 private:
    HYPRE_Solver pcg_solver;
 
+   HypreSolver * precond;
+
 public:
+   HyprePCG(MPI_Comm comm);
+
    HyprePCG(HypreParMatrix &_A);
 
+   virtual void SetOperator(const Operator &op);
+
    void SetTol(double tol);
    void SetMaxIter(int max_iter);
    void SetLogging(int logging);
@@ -817,11 +790,19 @@ class HypreGMRES : public HypreSolver
 private:
    HYPRE_Solver gmres_solver;
 
+   HypreSolver * precond;
+
+   /// Default, generally robust, GMRES options
+   void SetDefaultOptions();
+
 public:
+   HypreGMRES(MPI_Comm comm);
+
    HypreGMRES(HypreParMatrix &_A);
 
+   virtual void SetOperator(const Operator &op);
+
    void SetTol(double tol);
-   void SetAbsTol(double tol);
    void SetMaxIter(int max_iter);
    void SetKDim(int dim);
    void SetLogging(int logging);
@@ -872,6 +853,8 @@ public:
    explicit HypreDiagScale(HypreParMatrix &A) : HypreSolver(&A) { }
    virtual operator HYPRE_Solver() const { return NULL; }
 
+   virtual void SetOperator(const Operator &op);
+
    virtual HYPRE_PtrToParSolverFcn SetupFcn() const
    { return (HYPRE_PtrToParSolverFcn) HYPRE_ParCSRDiagScaleSetup; }
    virtual HYPRE_PtrToParSolverFcn SolveFcn() const
@@ -887,9 +870,21 @@ class HypreParaSails : public HypreSolver
 private:
    HYPRE_Solver sai_precond;
 
+   /// Default, generally robust, ParaSails options
+   void SetDefaultOptions();
+
+   // If sai_precond is NULL, this method allocates it and sets default options.
+   // Otherwise the method saves the options from sai_precond, destroys it,
+   // allocates a new object, and sets its options to the saved values.
+   void ResetSAIPrecond(MPI_Comm comm);
+
 public:
+   HypreParaSails(MPI_Comm comm);
+
    HypreParaSails(HypreParMatrix &A);
 
+   virtual void SetOperator(const Operator &op);
+
    void SetSymmetry(int sym);
 
    /// The typecast to HYPRE_Solver returns the internal sai_precond
@@ -916,9 +911,21 @@ class HypreEuclid : public HypreSolver
 private:
    HYPRE_Solver euc_precond;
 
+   /// Default, generally robust, Euclid options
+   void SetDefaultOptions();
+
+   // If euc_precond is NULL, this method allocates it and sets default options.
+   // Otherwise the method saves the options from euc_precond, destroys it,
+   // allocates a new object, and sets its options to the saved values.
+   void ResetEuclidPrecond(MPI_Comm comm);
+
 public:
+   HypreEuclid(MPI_Comm comm);
+
    HypreEuclid(HypreParMatrix &A);
 
+   virtual void SetOperator(const Operator &op);
+
    /// The typecast to HYPRE_Solver returns the internal euc_precond
    virtual operator HYPRE_Solver() const { return euc_precond; }
 
@@ -972,79 +979,9 @@ public:
        As with SetSystemsOptions(), this solver assumes Ordering::byVDIM. */
    void SetElasticityOptions(ParFiniteElementSpace *fespace);
 
-   /* distance parameter takes on values {1,2,15} for lAIR, meaning R is built using
-   distance 1 neighbors, distance two neighbors, or distance two on processor and
-   distance 1 off processor (i.e., distance 1.5 --> 15).        */
-   void SetLAIROptions(int distance=15,  std::string prerelax="",
-                      std::string postrelax="FFC", double strength_tol=0.1,
-                      double strength_tolR=0.01, double filter_tolR=0.0,
-                      int interp_type=100, int relax_type=3, double filterA_tol=0.0, 
-                      int splitting=6, int blksize=0, int Sabs=0);
-
-   void SetNAIROptions(int neumann_degree=2,  std::string prerelax="A",
-                      std::string postrelax="F", double strength_tol=0.1,
-                      double strength_tolR=0.01, double filter_tolR=0.0,
-                      int interp_type=100, int relax_type=10, double filterA_tol=0.0, 
-                      int splitting=6, int blksize=0, int Sabs=0);
-
-   void SetCoord(int dim, float *coord);
-   
    void SetPrintLevel(int print_level)
    { HYPRE_BoomerAMGSetPrintLevel(amg_precond, print_level); }
 
-   void SetMaxIter(int max_iter)
-   { HYPRE_BoomerAMGSetMaxIter(amg_precond, max_iter); }
-
-   void SetMaxLevels(int max_levels)
-   { HYPRE_BoomerAMGSetMaxLevels(amg_precond, max_levels); }
-
-   void SetTol(double tol)
-   { HYPRE_BoomerAMGSetTol(amg_precond, tol); }
-
-   void SetStrengthThresh(double strength)
-   { HYPRE_BoomerAMGSetStrongThreshold(amg_precond, strength); }
-
-   void SetStrengthThreshR(double strengthR)
-   { HYPRE_BoomerAMGSetStrongThresholdR(amg_precond, strengthR); }
-
-   void SetFilterThreshR(double filterR)
-   { HYPRE_BoomerAMGSetFilterThresholdR(amg_precond, filterR); }
-
-   void SetInterpolation(int interp_type)
-   { HYPRE_BoomerAMGSetInterpType(amg_precond, interp_type); }
-
-   void SetRestriction(int restrict_type)
-   { HYPRE_BoomerAMGSetRestriction(amg_precond, restrict_type); }
-
-   void SetCoarsening(int coarsen_type)
-   { HYPRE_BoomerAMGSetCoarsenType(amg_precond, coarsen_type); }
-
-   void SetRelaxType(int relax_type)
-   { HYPRE_BoomerAMGSetRelaxType(amg_precond, relax_type); }
-
-   void SetRelaxCycle(int prerelax, int postrelax)
-   { HYPRE_BoomerAMGSetCycleNumSweeps(amg_precond, prerelax,  1);
-     HYPRE_BoomerAMGSetCycleNumSweeps(amg_precond, postrelax, 2); }
-
-   void GetNumIterations(int &num_it)
-   { HYPRE_BoomerAMGGetNumIterations(amg_precond, &num_it); }
-
-   void SetCycleType(int cycle_type)
-   { HYPRE_BoomerAMGSetCycleType(amg_precond, cycle_type); }
-
-   void SetNodal(int blocksize)
-   { HYPRE_BoomerAMGSetNumFunctions(amg_precond, blocksize);
-     HYPRE_BoomerAMGSetNodal(amg_precond, 1); }
-
-   void SetAggressiveCoarsening(int num_levels)
-   { HYPRE_BoomerAMGSetAggNumLevels(amg_precond, num_levels); }
-
-   void SetTriangular()
-   { HYPRE_BoomerAMGSetIsTriangular(amg_precond, 1); }
-
-   void SetGMRESSwitchR(int gmres_switch)
-   { HYPRE_BoomerAMGSetGMRESSwitchR(amg_precond, gmres_switch); }
-
    /// The typecast to HYPRE_Solver returns the internal amg_precond
    virtual operator HYPRE_Solver() const { return amg_precond; }
 
@@ -1053,9 +990,6 @@ public:
    virtual HYPRE_PtrToParSolverFcn SolveFcn() const
    { return (HYPRE_PtrToParSolverFcn) HYPRE_BoomerAMGSolve; }
 
-   virtual void Mult (const HypreParVector &b, HypreParVector &x) const;
-   using HypreSolver::Mult;
-
    virtual ~HypreBoomerAMG();
 };
 
@@ -1070,6 +1004,9 @@ HypreParMatrix* DiscreteCurl(ParFiniteElementSpace *face_fespace,
 class HypreAMS : public HypreSolver
 {
 private:
+   /// Constuct AMS solver from finite element space
+   void Init(ParFiniteElementSpace *edge_space);
+
    HYPRE_Solver ams;
 
    /// Vertex coordinates
@@ -1080,8 +1017,12 @@ private:
    HypreParMatrix *Pi, *Pix, *Piy, *Piz;
 
 public:
+   HypreAMS(ParFiniteElementSpace *edge_fespace);
+
    HypreAMS(HypreParMatrix &A, ParFiniteElementSpace *edge_fespace);
 
+   virtual void SetOperator(const Operator &op);
+
    void SetPrintLevel(int print_lvl);
 
    /// Set this option when solving a curl-curl problem with zero mass term
@@ -1102,6 +1043,9 @@ public:
 class HypreADS : public HypreSolver
 {
 private:
+   /// Constuct ADS solver from finite element space
+   void Init(ParFiniteElementSpace *face_fespace);
+
    HYPRE_Solver ads;
 
    /// Vertex coordinates
@@ -1116,8 +1060,12 @@ private:
    HypreParMatrix *RT_Pi, *RT_Pix, *RT_Piy, *RT_Piz;
 
 public:
+   HypreADS(ParFiniteElementSpace *face_fespace);
+
    HypreADS(HypreParMatrix &A, ParFiniteElementSpace *face_fespace);
 
+   virtual void SetOperator(const Operator &op);
+
    void SetPrintLevel(int print_lvl);
 
    /// The typecast to HYPRE_Solver returns the internal ads object

linalg/operator.cpp

@@ -135,6 +135,76 @@ void Operator::PrintMatlab(std::ostream & out, int n, int m) const
 }
 
 
+void TimeDependentOperator::ExplicitMult(const Vector &, Vector &) const
+{
+   mfem_error("TimeDependentOperator::ExplicitMult() is not overridden!");
+}
+
+void TimeDependentOperator::ImplicitMult(const Vector &, const Vector &,
+                                         Vector &) const
+{
+   mfem_error("TimeDependentOperator::ImplicitMult() is not overridden!");
+}
+
+void TimeDependentOperator::Mult(const Vector &, Vector &) const
+{
+   mfem_error("TimeDependentOperator::Mult() is not overridden!");
+}
+
+void TimeDependentOperator::ImplicitSolve(const double, const Vector &,
+                                          Vector &)
+{
+   mfem_error("TimeDependentOperator::ImplicitSolve() is not overridden!");
+}
+
+Operator &TimeDependentOperator::GetImplicitGradient(
+   const Vector &, const Vector &, double) const
+{
+   mfem_error("TimeDependentOperator::GetImplicitGradient() is "
+              "not overridden!");
+   return const_cast<Operator &>(dynamic_cast<const Operator &>(*this));
+}
+
+Operator &TimeDependentOperator::GetExplicitGradient(const Vector &) const
+{
+   mfem_error("TimeDependentOperator::GetExplicitGradient() is "
+              "not overridden!");
+   return const_cast<Operator &>(dynamic_cast<const Operator &>(*this));
+}
+
+int TimeDependentOperator::SUNImplicitSetup(const Vector &,
+                                            const Vector &,
+                                            int, int *, double)
+{
+   mfem_error("TimeDependentOperator::SUNImplicitSetup() is not overridden!");
+   return (-1);
+}
+
+int TimeDependentOperator::SUNImplicitSolve(const Vector &, Vector &, double)
+{
+   mfem_error("TimeDependentOperator::SUNImplicitSolve() is not overridden!");
+   return (-1);
+}
+
+int TimeDependentOperator::SUNMassSetup()
+{
+   mfem_error("TimeDependentOperator::SUNMassSetup() is not overridden!");
+   return (-1);
+}
+
+int TimeDependentOperator::SUNMassSolve(const Vector &, Vector &, double)
+{
+   mfem_error("TimeDependentOperator::SUNMassSolve() is not overridden!");
+   return (-1);
+}
+
+int TimeDependentOperator::SUNMassMult(const Vector &, Vector &)
+{
+   mfem_error("TimeDependentOperator::SUNMassMult() is not overridden!");
+   return (-1);
+}
+
+
 ProductOperator::ProductOperator(const Operator *A, const Operator *B,
                                  bool ownA, bool ownB)
    : Operator(A->Height(), B->Width()),

linalg/operator.hpp

@@ -188,7 +188,7 @@ public:
 /** Operator of the form: (x,t) -> f(x,t), where k = f(x,t) generally solves the
     algebraic equation F(x,k,t) = G(x,t). The functions F and G represent the
     _implicit_ and _explicit_ parts of the operator, respectively. For explicit
-    operators, F(x,k,t) = k, so f(x,t) = G(x,t).*/
+    operators, F(x,k,t) = k, so f(x,t) = G(x,t). */
 class TimeDependentOperator : public Operator
 {
 public:
@@ -199,21 +199,35 @@ public:
       HOMOGENEOUS ///< This type assumes that G(x,t) = 0.
    };
 
+   /// Evaluation mode. See SetEvalMode() for details.
+   enum EvalMode
+   {
+      /** Normal evaluation. */
+      NORMAL,
+      /** Assuming additive split, f(x,t) = f1(x,t) + f2(x,t), evaluate the
+          first term, f1. */
+      ADDITIVE_TERM_1,
+      /** Assuming additive split, f(x,t) = f1(x,t) + f2(x,t), evaluate the
+          second term, f2. */
+      ADDITIVE_TERM_2
+   };
+
 protected:
    double t;  ///< Current time.
    Type type; ///< Describes the form of the TimeDependentOperator.
+   EvalMode eval_mode; ///< Current evaluation mode.
 
 public:
    /** @brief Construct a "square" TimeDependentOperator y = f(x,t), where x and
        y have the same dimension @a n. */
    explicit TimeDependentOperator(int n = 0, double t_ = 0.0,
                                   Type type_ = EXPLICIT)
-      : Operator(n) { t = t_; type = type_; }
+      : Operator(n) { t = t_; type = type_; eval_mode = NORMAL; }
 
    /** @brief Construct a TimeDependentOperator y = f(x,t), where x and y have
        dimensions @a w and @a h, respectively. */
    TimeDependentOperator(int h, int w, double t_ = 0.0, Type type_ = EXPLICIT)
-      : Operator(h, w) { t = t_; type = type_; }
+      : Operator(h, w) { t = t_; type = type_; eval_mode = NORMAL; }
 
    /// Read the currently set time.
    virtual double GetTime() const { return t; }
@@ -228,33 +242,41 @@ public:
    /// True if #type is #HOMOGENEOUS.
    bool isHomogeneous() const { return (type == HOMOGENEOUS); }
 
+   /// Return the current evaluation mode. See SetEvalMode() for details.
+   EvalMode GetEvalMode() const { return eval_mode; }
+
+   /// Set the evaluation mode of the time-dependent operator.
+   /** The evaluation mode is a switch that allows time-stepping methods to
+       request evaluation of separate components/terms of the time-dependent
+       operator. For example, IMEX methods typically assume additive split of
+       the operator: f(x,t) = f1(x,t) + f2(x,t) and they rely on the ability to
+       evaluate the two terms separately.
+
+       Generally, setting the evaluation mode should affect the behavior of all
+       evaluation-related methods in the class, such as Mult(), ImplicitSolve(),
+       etc. However, the exact list of methods that need to support a specific
+       mode will depend on the used time-stepping method. */
+   virtual void SetEvalMode(const EvalMode new_eval_mode)
+   { eval_mode = new_eval_mode; }
+
    /** @brief Perform the action of the explicit part of the operator, G:
        @a y = G(@a x, t) where t is the current time.
 
        Presently, this method is used by some PETSc ODE solvers, for more
        details, see the PETSc Manual. */
-   virtual void ExplicitMult(const Vector &x, Vector &y) const
-   {
-      mfem_error("TimeDependentOperator::ExplicitMult() is not overridden!");
-   }
+   virtual void ExplicitMult(const Vector &x, Vector &y) const;
 
    /** @brief Perform the action of the implicit part of the operator, F:
        @a y = F(@a x, @a k, t) where t is the current time.
 
        Presently, this method is used by some PETSc ODE solvers, for more
        details, see the PETSc Manual.*/
-   virtual void ImplicitMult(const Vector &x, const Vector &k, Vector &y) const
-   {
-      mfem_error("TimeDependentOperator::ImplicitMult() is not overridden!");
-   }
+   virtual void ImplicitMult(const Vector &x, const Vector &k, Vector &y) const;
 
    /** @brief Perform the action of the operator: @a y = k = f(@a x, t), where
        k solves the algebraic equation F(@a x, k, t) = G(@a x, t) and t is the
        current time. */
-   virtual void Mult(const Vector &x, Vector &y) const
-   {
-      mfem_error("TimeDependentOperator::Mult() is not overridden!");
-   }
+   virtual void Mult(const Vector &x, Vector &y) const;
 
    /** @brief Solve the equation: @a k = f(@a x + @a dt @a k, t), for the
        unknown @a k at the current time t.
@@ -272,10 +294,7 @@ public:
        methods and the backward Euler method in particular.
 
        If not re-implemented, this method simply generates an error. */
-   virtual void ImplicitSolve(const double dt, const Vector &x, Vector &k)
-   {
-      mfem_error("TimeDependentOperator::ImplicitSolve() is not overridden!");
-   }
+   virtual void ImplicitSolve(const double dt, const Vector &x, Vector &k);
 
    /** @brief Return an Operator representing (dF/dk @a shift + dF/dx) at the
        given @a x, @a k, and the currently set time.
@@ -283,24 +302,75 @@ public:
        Presently, this method is used by some PETSc ODE solvers, for more
        details, see the PETSc Manual. */
    virtual Operator& GetImplicitGradient(const Vector &x, const Vector &k,
-                                         double shift) const
-   {
-      mfem_error("TimeDependentOperator::GetImplicitGradient() is "
-                 "not overridden!");
-      return const_cast<Operator &>(dynamic_cast<const Operator &>(*this));
-   }
+                                         double shift) const;
 
    /** @brief Return an Operator representing dG/dx at the given point @a x and
        the currently set time.
 
        Presently, this method is used by some PETSc ODE solvers, for more
        details, see the PETSc Manual. */
-   virtual Operator& GetExplicitGradient(const Vector &x) const
-   {
-      mfem_error("TimeDependentOperator::GetExplicitGradient() is "
-                 "not overridden!");
-      return const_cast<Operator &>(dynamic_cast<const Operator &>(*this));
-   }
+   virtual Operator& GetExplicitGradient(const Vector &x) const;
+
+   /** @brief Setup the ODE linear system \f$ A(x,t) = (I - gamma J) \f$ or
+       \f$ A = (M - gamma J) \f$, where \f$ J(x,t) = \frac{df}{dt(x,t)} \f$.
+
+       @param[in]  x     The state at which \f$A(x,t)\f$ should be evaluated.
+       @param[in]  fx    The current value of the ODE rhs function, \f$f(x,t)\f$.
+       @param[in]  jok   Flag indicating if the Jacobian should be updated.
+       @param[out] jcur  Flag to signal if the Jacobian was updated.
+       @param[in]  gamma The scaled time step value.
+
+       If not re-implemented, this method simply generates an error.
+
+       Presently, this method is used by SUNDIALS ODE solvers, for more
+       details, see the SUNDIALS User Guides. */
+   virtual int SUNImplicitSetup(const Vector &x, const Vector &fx,
+                                int jok, int *jcur, double gamma);
+
+   /** @brief Solve the ODE linear system \f$ A x = b \f$ as setup by
+       the method SUNImplicitSetup().
+
+       @param[in]      b   The linear system right-hand side.
+       @param[in,out]  x   On input, the initial guess. On output, the solution.
+       @param[in]      tol Linear solve tolerance.
+
+       If not re-implemented, this method simply generates an error.
+
+       Presently, this method is used by SUNDIALS ODE solvers, for more
+       details, see the SUNDIALS User Guides. */
+   virtual int SUNImplicitSolve(const Vector &b, Vector &x, double tol);
+
+   /** @brief Setup the mass matrix in the ODE system \f$ M y' = f(y,t) \f$ .
+
+       If not re-implemented, this method simply generates an error.
+
+       Presently, this method is used by SUNDIALS ARKStep integrator, for more
+       details, see the ARKode User Guide. */
+   virtual int SUNMassSetup();
+
+   /** @brief Solve the mass matrix linear system \f$ M x = b \f$
+       as setup by the method SUNMassSetup().
+
+       @param[in]      b   The linear system right-hand side.
+       @param[in,out]  x   On input, the initial guess. On output, the solution.
+       @param[in]      tol Linear solve tolerance.
+
+       If not re-implemented, this method simply generates an error.
+
+       Presently, this method is used by SUNDIALS ARKStep integrator, for more
+       details, see the ARKode User Guide. */
+   virtual int SUNMassSolve(const Vector &b, Vector &x, double tol);
+
+   /** @brief Compute the mass matrix-vector productv \f$ v = M x \f$ .
+
+       @param[in]   x The vector to multiply.
+       @param[out]  v The result of the matrix-vector product.
+
+       If not re-implemented, this method simply generates an error.
+
+       Presently, this method is used by SUNDIALS ARKStep integrator, for more
+       details, see the ARKode User Guide. */
+   virtual int SUNMassMult(const Vector &x, Vector &v);
 
    virtual ~TimeDependentOperator() { }
 };

linalg/petsc.cpp

@@ -422,6 +422,12 @@ PetscParVector& PetscParVector::operator*=(PetscScalar s)
    return *this;
 }
 
+PetscParVector& PetscParVector::operator+=(PetscScalar s)
+{
+   ierr = VecShift(x,s); PCHKERRQ(x,ierr);
+   return *this;
+}
+
 void PetscParVector::PlaceArray(PetscScalar *temp_data)
 {
    ierr = VecPlaceArray(x,temp_data); PCHKERRQ(x,ierr);
@@ -2303,17 +2309,6 @@ void PetscLinearSolver::SetOperator(const Operator &op)
                               (dynamic_cast<const PetscParMatrix *>(&op));
    const Operator       *oA = dynamic_cast<const Operator *>(&op);
 
-   // Preserve Pmat if already set
-   KSP ksp = (KSP)obj;
-   Mat P = NULL;
-   PetscBool pmat;
-   ierr = KSPGetOperatorsSet(ksp,NULL,&pmat); PCHKERRQ(ksp,ierr);
-   if (pmat)
-   {
-      ierr = KSPGetOperators(ksp,NULL,&P); PCHKERRQ(ksp,ierr);
-      ierr = PetscObjectReference((PetscObject)P); PCHKERRQ(ksp,ierr);
-   }
-
    // update base classes: Operator, Solver, PetscLinearSolver
    bool delete_pA = false;
    if (!pA)
@@ -2336,6 +2331,7 @@ void PetscLinearSolver::SetOperator(const Operator &op)
    MFEM_VERIFY(pA, "Unsupported operation!");
 
    // Set operators into PETSc KSP
+   KSP ksp = (KSP)obj;
    Mat A = pA->A;
    if (operatorset)
    {
@@ -2356,15 +2352,7 @@ void PetscLinearSolver::SetOperator(const Operator &op)
          wrap = false;
       }
    }
-   if (P)
-   {
-      ierr = KSPSetOperators(ksp,A,P); PCHKERRQ(ksp,ierr);
-      ierr = MatDestroy(&P); PCHKERRQ(ksp,ierr);
-   }
-   else
-   {
-      ierr = KSPSetOperators(ksp,A,A); PCHKERRQ(ksp,ierr);
-   }
+   ierr = KSPSetOperators(ksp,A,A); PCHKERRQ(ksp,ierr);
 
    // Update PetscSolver
    operatorset = true;
@@ -2500,7 +2488,7 @@ void PetscLinearSolver::SetPreconditioner(Solver &precond)
    }
 }
 
-void PetscLinearSolver::Mult(const Vector &b, Vector &x) const
+void PetscLinearSolver::MultKernel(const Vector &b, Vector &x, bool trans) const
 {
    KSP ksp = (KSP)obj;
 
@@ -2528,11 +2516,28 @@ void PetscLinearSolver::Mult(const Vector &b, Vector &x) const
    PCHKERRQ(ksp, ierr);
 
    // Solve the system.
-   ierr = KSPSolve(ksp, B->x, X->x); PCHKERRQ(ksp,ierr);
+   if (trans)
+   {
+      ierr = KSPSolveTranspose(ksp, B->x, X->x); PCHKERRQ(ksp,ierr);
+   }
+   else
+   {
+      ierr = KSPSolve(ksp, B->x, X->x); PCHKERRQ(ksp,ierr);
+   }
    B->ResetArray();
    X->ResetArray();
 }
 
+void PetscLinearSolver::Mult(const Vector &b, Vector &x) const
+{
+   (*this).MultKernel(b,x,false);
+}
+
+void PetscLinearSolver::MultTranspose(const Vector &b, Vector &x) const
+{
+   (*this).MultKernel(b,x,true);
+}
+
 PetscLinearSolver::~PetscLinearSolver()
 {
    MPI_Comm comm;
@@ -2654,7 +2659,8 @@ void PetscPreconditioner::SetOperator(const Operator &op)
    if (delete_pA) { delete pA; };
 }
 
-void PetscPreconditioner::Mult(const Vector &b, Vector &x) const
+void PetscPreconditioner::MultKernel(const Vector &b, Vector &x,
+                                     bool trans) const
 {
    PC pc = (PC)obj;
 
@@ -2679,11 +2685,28 @@ void PetscPreconditioner::Mult(const Vector &b, Vector &x) const
    Customize();
 
    // Apply the preconditioner.
-   ierr = PCApply(pc, B->x, X->x); PCHKERRQ(pc, ierr);
+   if (trans)
+   {
+      ierr = PCApplyTranspose(pc, B->x, X->x); PCHKERRQ(pc, ierr);
+   }
+   else
+   {
+      ierr = PCApply(pc, B->x, X->x); PCHKERRQ(pc, ierr);
+   }
    B->ResetArray();
    X->ResetArray();
 }
 
+void PetscPreconditioner::Mult(const Vector &b, Vector &x) const
+{
+   (*this).MultKernel(b,x,false);
+}
+
+void PetscPreconditioner::MultTranspose(const Vector &b, Vector &x) const
+{
+   (*this).MultKernel(b,x,true);
+}
+
 PetscPreconditioner::~PetscPreconditioner()
 {
    MPI_Comm comm;
@@ -3188,26 +3211,27 @@ PetscFieldSplitSolver::PetscFieldSplitSolver(MPI_Comm comm, Operator &op,
    : PetscPreconditioner(comm,op,prefix)
 {
    PC pc = (PC)obj;
+   ierr = PCSetType(pc,PCFIELDSPLIT); PCHKERRQ(pc,ierr);
 
    Mat pA;
    ierr = PCGetOperators(pc,&pA,NULL); PCHKERRQ(pc,ierr);
 
    // Check if pA is of type MATNEST
-   // (this requirement can be removed when we can pass fields).
    PetscBool isnest;
    ierr = PetscObjectTypeCompare((PetscObject)pA,MATNEST,&isnest);
-   PCHKERRQ(pA,ierr);
-   MFEM_VERIFY(isnest,
-               "PetscFieldSplitSolver needs the matrix in nested format.");
 
-   PetscInt nr;
-   IS  *isrow;
-   ierr = PCSetType(pc,PCFIELDSPLIT); PCHKERRQ(pc,ierr);
-   ierr = MatNestGetSize(pA,&nr,NULL); PCHKERRQ(pc,ierr);
-   ierr = PetscCalloc1(nr,&isrow); CCHKERRQ(PETSC_COMM_SELF,ierr);
-   ierr = MatNestGetISs(pA,isrow,NULL); PCHKERRQ(pc,ierr);
+   PetscInt nr = 0;
+   IS  *isrow = NULL;
+   if (isnest) // we now the fields
+   {
+      ierr = MatNestGetSize(pA,&nr,NULL); PCHKERRQ(pc,ierr);
+      ierr = PetscCalloc1(nr,&isrow); CCHKERRQ(PETSC_COMM_SELF,ierr);
+      ierr = MatNestGetISs(pA,isrow,NULL); PCHKERRQ(pc,ierr);
+   }
 
    // We need to customize here, before setting the index sets.
+   // This is because PCFieldSplitSetType customizes the function
+   // pointers. SubSolver options will be processed during PCApply
    Customize();
 
    for (PetscInt i=0; i<nr; i++)
@@ -3575,6 +3599,7 @@ void PetscODESolver::Run(Vector &x, double &t, double &dt, double t_final)
 }  // namespace mfem
 
 #include "petsc/private/petscimpl.h"
+#include "petsc/private/matimpl.h"
 
 // auxiliary functions
 static PetscErrorCode __mfem_ts_monitor(TS ts, PetscInt it, PetscReal t, Vec x,
@@ -3730,17 +3755,37 @@ static PetscErrorCode __mfem_ts_ijacobian(TS ts, PetscReal t, Vec x,
       pA->EliminateRowsCols(bchandler->GetTDofs(),dummy,dummy);
    }
 
+   // Get nonzerostate
+   PetscObjectState nonzerostate;
+   ierr = MatGetNonzeroState(P,&nonzerostate); CHKERRQ(ierr);
+
    // Avoid unneeded copy of the matrix by hacking
    Mat B;
    B = pA->ReleaseMat(false);
    ierr = MatHeaderReplace(P,&B); CHKERRQ(ierr);
    if (delete_pA) { delete pA; }
+
+   // Matrix-free case
    if (A && A != P)
    {
       ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
       ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
    }
 
+   // When using MATNEST and PCFIELDSPLIT, the second setup of the
+   // preconditioner fails because MatCreateSubMatrix_Nest does not
+   // actually return a matrix. Instead, for efficiency reasons,
+   // it returns a reference to the submatrix. The second time it
+   // is called, MAT_REUSE_MATRIX is used and MatCreateSubMatrix_Nest
+   // aborts since the two submatrices are actually different.
+   // We circumvent this issue by incrementing the nonzero state
+   // (i.e. PETSc thinks the operator sparsity pattern has changed)
+   // This does not impact performances in the case of MATNEST
+   PetscBool isnest;
+   ierr = PetscObjectTypeCompare((PetscObject)P,MATNEST,&isnest);
+   CHKERRQ(ierr);
+   if (isnest) { P->nonzerostate = nonzerostate + 1; }
+
    // Jacobian reusage
    ierr = PetscObjectStateGet((PetscObject)P,&ts_ctx->cached_ijacstate);
    CHKERRQ(ierr);
@@ -3897,6 +3942,7 @@ static PetscErrorCode __mfem_ts_computesplits(TS ts,PetscReal t,Vec x,Vec xp,
       ierr = MatAXPY(*pJxp,-1.0,*pJx,SAME_NONZERO_PATTERN); PCHKERRQ(ts,ierr);
    }
 
+   // Matrix-free cases
    if (Ax && Ax != Jx)
    {
       ierr = MatAssemblyBegin(Ax,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
@@ -3983,12 +4029,31 @@ static PetscErrorCode __mfem_ts_rhsjacobian(TS ts, PetscReal t, Vec x,
       pA->EliminateRowsCols(bchandler->GetTDofs(),dummy,dummy);
    }
 
+   // Get nonzerostate
+   PetscObjectState nonzerostate;
+   ierr = MatGetNonzeroState(P,&nonzerostate); CHKERRQ(ierr);
+
    // Avoid unneeded copy of the matrix by hacking
    Mat B;
    B = pA->ReleaseMat(false);
    ierr = MatHeaderReplace(P,&B); CHKERRQ(ierr);
    if (delete_pA) { delete pA; }
 
+   // When using MATNEST and PCFIELDSPLIT, the second setup of the
+   // preconditioner fails because MatCreateSubMatrix_Nest does not
+   // actually return a matrix. Instead, for efficiency reasons,
+   // it returns a reference to the submatrix. The second time it
+   // is called, MAT_REUSE_MATRIX is used and MatCreateSubMatrix_Nest
+   // aborts since the two submatrices are actually different.
+   // We circumvent this issue by incrementing the nonzero state
+   // (i.e. PETSc thinks the operator sparsity pattern has changed)
+   // This does not impact performances in the case of MATNEST
+   PetscBool isnest;
+   ierr = PetscObjectTypeCompare((PetscObject)P,MATNEST,&isnest);
+   CHKERRQ(ierr);
+   if (isnest) { P->nonzerostate = nonzerostate + 1; }
+
+   // Matrix-free case
    if (A && A != P)
    {
       ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
@@ -4092,10 +4157,30 @@ static PetscErrorCode __mfem_snes_jacobian(SNES snes, Vec x, Mat A, Mat P,
       pA->EliminateRowsCols(bchandler->GetTDofs(),dummy,dummy);
    }
 
+   // Get nonzerostate
+   PetscObjectState nonzerostate;
+   ierr = MatGetNonzeroState(P,&nonzerostate); CHKERRQ(ierr);
+
    // Avoid unneeded copy of the matrix by hacking
    Mat B = pA->ReleaseMat(false);
    ierr = MatHeaderReplace(P,&B); CHKERRQ(ierr);
    if (delete_pA) { delete pA; }
+
+   // When using MATNEST and PCFIELDSPLIT, the second setup of the
+   // preconditioner fails because MatCreateSubMatrix_Nest does not
+   // actually return a matrix. Instead, for efficiency reasons,
+   // it returns a reference to the submatrix. The second time it
+   // is called, MAT_REUSE_MATRIX is used and MatCreateSubMatrix_Nest
+   // aborts since the two submatrices are actually different.
+   // We circumvent this issue by incrementing the nonzero state
+   // (i.e. PETSc thinks the operator sparsity pattern has changed)
+   // This does not impact performances in the case of MATNEST
+   PetscBool isnest;
+   ierr = PetscObjectTypeCompare((PetscObject)P,MATNEST,&isnest);
+   CHKERRQ(ierr);
+   if (isnest) { P->nonzerostate = nonzerostate + 1; }
+
+   // Matrix-free case
    if (A && A != P)
    {
       ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
@@ -4443,6 +4528,11 @@ PetscErrorCode MakeShellPC(PC pc, mfem::Solver &precond, bool ownsop)
    ctx->factory = NULL;
    ctx->numprec = 0;
 
+   // In case the PC was already of type SHELL, this will destroy any
+   // previous user-defined data structure
+   // We cannot call PCReset as it will wipe out any operator already set
+   ierr = PCSetType(pc,PCNONE); CHKERRQ(ierr);
+
    ierr = PCSetType(pc,PCSHELL); CHKERRQ(ierr);
    ierr = PCShellSetName(pc,"MFEM Solver (unknown Pmat)"); CHKERRQ(ierr);
    ierr = PCShellSetContext(pc,(void *)ctx); CHKERRQ(ierr);
@@ -4467,6 +4557,11 @@ PetscErrorCode MakeShellPCWithFactory(PC pc,
    ctx->factory = factory;
    ctx->numprec = 0;
 
+   // In case the PC was already of type SHELL, this will destroy any
+   // previous user-defined data structure
+   // We cannot call PCReset as it will wipe out any operator already set
+   ierr = PCSetType(pc,PCNONE); CHKERRQ(ierr);
+
    ierr = PCSetType(pc,PCSHELL); CHKERRQ(ierr);
    ierr = PCShellSetName(pc,factory->GetName()); CHKERRQ(ierr);
    ierr = PCShellSetContext(pc,(void *)ctx); CHKERRQ(ierr);

linalg/petsc.hpp

@@ -170,6 +170,7 @@ public:
    PetscParVector& operator+= (const PetscParVector &y);
    PetscParVector& operator-= (const PetscParVector &y);
    PetscParVector& operator*= (PetscScalar d);
+   PetscParVector& operator+= (PetscScalar d);
 
    /** @brief Temporarily replace the data of the PETSc Vec object. To return to
        the original data array, call ResetArray().
@@ -601,6 +602,7 @@ class PetscLinearSolver : public PetscSolver, public Solver
 private:
    /// Internal flag to handle HypreParMatrix conversion or not.
    bool wrap;
+   void MultKernel(const Vector &b, Vector &x, bool trans) const;
 
 public:
    PetscLinearSolver(MPI_Comm comm, const std::string &prefix = std::string(),
@@ -616,6 +618,8 @@ public:
                      const std::string &prefix = std::string());
    virtual ~PetscLinearSolver();
 
+   /// Sets the operator to be used for mat-vec operations and
+   /// for the construction of the preconditioner
    virtual void SetOperator(const Operator &op);
 
    /// Allows to prescribe a different operator (@a pop) to construct
@@ -623,10 +627,12 @@ public:
    void SetOperator(const Operator &op, const Operator &pop);
 
    /// Sets the solver to perform preconditioning
+   /// preserves the linear operator for the mat-vec
    void SetPreconditioner(Solver &precond);
 
    /// Application of the solver.
    virtual void Mult(const Vector &b, Vector &x) const;
+   virtual void MultTranspose(const Vector &b, Vector &x) const;
 
    /// Conversion function to PETSc's KSP type.
    operator KSP() const { return (KSP)obj; }
@@ -646,6 +652,9 @@ public:
 /// Abstract class for PETSc's preconditioners.
 class PetscPreconditioner : public PetscSolver, public Solver
 {
+private:
+   void MultKernel(const Vector &b, Vector &x, bool trans) const;
+
 public:
    PetscPreconditioner(MPI_Comm comm,
                        const std::string &prefix = std::string());
@@ -659,6 +668,7 @@ public:
 
    /// Application of the preconditioner.
    virtual void Mult(const Vector &b, Vector &x) const;
+   virtual void MultTranspose(const Vector &b, Vector &x) const;
 
    /// Conversion function to PETSc's PC type.
    operator PC() const { return (PC)obj; }

linalg/solvers.cpp

@@ -1311,6 +1311,8 @@ void NewtonSolver::Mult(const Vector &b, Vector &x) const
       }
       add(x, -c_scale, c, x);
 
+      ProcessNewState(x);
+
       oper->Mult(x, r);
       if (have_b)
       {

linalg/solvers.hpp

@@ -283,6 +283,10 @@ public:
        value of 0 indicates a failure, interrupting the Newton iteration. */
    virtual double ComputeScalingFactor(const Vector &x, const Vector &b) const
    { return 1.0; }
+
+   /** @brief This method can be overloaded in derived classes to perform
+       computations that need knowledge of the newest Newton state. */
+   virtual void ProcessNewState(const Vector &x) const { }
 };
 
 /** Adaptive restarted GMRES.

linalg/sparsemat.cpp

@@ -2640,9 +2640,11 @@ SparseMatrix &SparseMatrix::operator=(double a)
 {
    if (Rows == NULL)
    {
-      for (int i = 0, nnz = I[height]; i < nnz; i++)
+      const int nnz = J.Capacity();
+      double *h_A = HostWrite(A, nnz);
+      for (int i = 0; i < nnz; i++)
       {
-         A[i] = a;
+         h_A[i] = a;
       }
    }
    else

linalg/sparsemat.hpp

@@ -152,6 +152,54 @@ public:
    /// Return the element data, i.e. the array #A, const version.
    inline const double *GetData() const { return A; }
 
+   // Memory access methods for the #I array.
+   Memory<int> &GetMemoryI() { return I; }
+   const Memory<int> &GetMemoryI() const { return I; }
+   const int *ReadI(bool on_dev = true) const
+   { return mfem::Read(I, Height()+1, on_dev); }
+   int *WriteI(bool on_dev = true)
+   { return mfem::Write(I, Height()+1, on_dev); }
+   int *ReadWriteI(bool on_dev = true)
+   { return mfem::ReadWrite(I, Height()+1, on_dev); }
+   const int *HostReadI() const
+   { return mfem::Read(I, Height()+1, false); }
+   int *HostWriteI()
+   { return mfem::Write(I, Height()+1, false); }
+   int *HostReadWriteI()
+   { return mfem::ReadWrite(I, Height()+1, false); }
+
+   // Memory access methods for the #J array.
+   Memory<int> &GetMemoryJ() { return J; }
+   const Memory<int> &GetMemoryJ() const { return J; }
+   const int *ReadJ(bool on_dev = true) const
+   { return mfem::Read(J, J.Capacity(), on_dev); }
+   int *WriteJ(bool on_dev = true)
+   { return mfem::Write(J, J.Capacity(), on_dev); }
+   int *ReadWriteJ(bool on_dev = true)
+   { return mfem::ReadWrite(J, J.Capacity(), on_dev); }
+   const int *HostReadJ() const
+   { return mfem::Read(J, J.Capacity(), false); }
+   int *HostWriteJ()
+   { return mfem::Write(J, J.Capacity(), false); }
+   int *HostReadWriteJ()
+   { return mfem::ReadWrite(J, J.Capacity(), false); }
+
+   // Memory access methods for the #A array.
+   Memory<double> &GetMemoryData() { return A; }
+   const Memory<double> &GetMemoryData() const { return A; }
+   const double *ReadData(bool on_dev = true) const
+   { return mfem::Read(A, A.Capacity(), on_dev); }
+   double *WriteData(bool on_dev = true)
+   { return mfem::Write(A, A.Capacity(), on_dev); }
+   double *ReadWriteData(bool on_dev = true)
+   { return mfem::ReadWrite(A, A.Capacity(), on_dev); }
+   const double *HostReadData() const
+   { return mfem::Read(A, A.Capacity(), false); }
+   double *HostWriteData()
+   { return mfem::Write(A, A.Capacity(), false); }
+   double *HostReadWriteData()
+   { return mfem::ReadWrite(A, A.Capacity(), false); }
+
    /// Returns the number of elements in row @a i.
    int RowSize(const int i) const;
 

linalg/sundials.hpp

@@ -23,384 +23,458 @@
 #include "ode.hpp"
 #include "solvers.hpp"
 
+#include <sundials/sundials_config.h>
+// Check for appropriate SUNDIALS version
+#if !defined(SUNDIALS_VERSION_MAJOR) || (SUNDIALS_VERSION_MAJOR < 5)
+#error MFEM requires SUNDIALS version 5.0.0 or newer!
+#endif
+#include <sundials/sundials_matrix.h>
+#include <sundials/sundials_linearsolver.h>
 #include <cvode/cvode.h>
-#include <arkode/arkode.h>
+#include <arkode/arkode_arkstep.h>
 #include <kinsol/kinsol.h>
 
-struct KINMemRec;
-
 namespace mfem
 {
 
-/** @brief Abstract base class, wrapping the custom linear solvers interface in
-    SUNDIALS' CVODE and ARKODE solvers. */
-/** For a given ODE system
+// ---------------------------------------------------------------------------
+// Base class for interfacing with SUNDIALS packages
+// ---------------------------------------------------------------------------
 
-        dx/dt = f(x,t)
-
-    the purpose of this class is to facilitate the (approximate) solution of
-    linear systems of the form
-
-        (I - γJ) y = b,   J = J(x,t) = df/dx
-
-    for given b, x, t and γ, where γ = GetTimeStep() is a scaled time step. */
-class SundialsODELinearSolver
-{
-public:
-   enum {CVODE, ARKODE} type; ///< Is CVODE or ARKODE using this object?
-
-protected:
-   SundialsODELinearSolver() { }
-   virtual ~SundialsODELinearSolver() { }
-
-   /// Get the current scaled time step, gamma, from @a sundials_mem.
-   double GetTimeStep(void *sundials_mem);
-   /// Get the TimeDependentOperator associated with @a sundials_mem.
-   TimeDependentOperator *GetTimeDependentOperator(void *sundials_mem);
-
-public:
-   /** @name Linear solver interface methods.
-       These four functions and their parameters are documented in Section 7 of
-       http://computation.llnl.gov/sites/default/files/public/cv_guide.pdf
-       and Section 7.4 of
-       http://computation.llnl.gov/sites/default/files/public/ark_guide.pdf
-
-       The first argument, @a sundials_mem, is one of the pointer types,
-       CVodeMem or ARKodeMem, depending on the value of the data member @a type.
-   */
-   ///@{
-   virtual int InitSystem(void *sundials_mem) = 0;
-   virtual int SetupSystem(void *sundials_mem, int conv_fail,
-                           const Vector &y_pred, const Vector &f_pred,
-                           int &jac_cur, Vector &v_temp1,
-                           Vector &v_temp2, Vector &v_temp3) = 0;
-   virtual int SolveSystem(void *sundials_mem, Vector &b, const Vector &w,
-                           const Vector &y_cur, const Vector &f_cur) = 0;
-   virtual int FreeSystem(void *sundials_mem) = 0;
-   ///@}
-};
-
-
-/// A base class for the MFEM classes wrapping SUNDIALS' solvers.
-/** This class defines some common data and functions used by the SUNDIALS
-    solvers, e.g the common @a #sundials_mem pointer and return @a #flag. */
+/// Base class for interfacing with SUNDIALS packages.
 class SundialsSolver
 {
 protected:
-   void *sundials_mem; ///< Pointer to the SUNDIALS mem object.
-   mutable int flag;   ///< Flag returned by the last call to SUNDIALS.
+   void *sundials_mem;     ///< SUNDIALS mem structure.
+   mutable int flag;       ///< Last flag returned from a call to SUNDIALS.
+   bool reinit;            ///< Flag to signal memory reinitialization is need.
+   long saved_global_size; ///< Global vector length on last initialization.
+
+   N_Vector           y;   ///< State vector.
+   SUNMatrix          A;   ///< Linear system A = I - gamma J, M - gamma J, or J.
+   SUNMatrix          M;   ///< Mass matrix M.
+   SUNLinearSolver    LSA; ///< Linear solver for A.
+   SUNLinearSolver    LSM; ///< Linear solver for M.
+   SUNNonlinearSolver NLS; ///< Nonlinear solver.
 
-   N_Vector y;  ///< Auxiliary N_Vector.
 #ifdef MFEM_USE_MPI
    bool Parallel() const
-   { return (y->ops->nvgetvectorid != N_VGetVectorID_Serial); }
+   { return (N_VGetVectorID(y) != SUNDIALS_NVEC_SERIAL); }
 #else
    bool Parallel() const { return false; }
 #endif
 
-   static const double default_rel_tol;
-   static const double default_abs_tol;
+   /// Default scalar relative tolerance.
+   static constexpr double default_rel_tol = 1e-4;
+   /// Default scalar absolute tolerance.
+   static constexpr double default_abs_tol = 1e-9;
 
-   // Computes the action of a time-dependent operator.
-   /// Callback function used in CVODESolver and ARKODESolver.
-   static int ODEMult(realtype t, const N_Vector y,
-                      N_Vector ydot, void *td_oper);
-
-   /// @name The constructors are protected
-   ///@{
-   SundialsSolver() : sundials_mem(NULL) { }
-   SundialsSolver(void *mem) : sundials_mem(mem) { }
-   ///@}
+   /** @brief Protected constructor: objects of this type should be constructed
+       only as part of a derived class. */
+   SundialsSolver() : sundials_mem(NULL), flag(0), reinit(false),
+      saved_global_size(0), y(NULL), A(NULL), M(NULL),
+      LSA(NULL), LSM(NULL), NLS(NULL) { }
 
 public:
-   /// Access the underlying SUNDIALS object.
-   void *SundialsMem() const { return sundials_mem; }
+   /// Access the SUNDIALS memory structure.
+   void *GetMem() const { return sundials_mem; }
 
-   /// Return the flag returned by the last call to a SUNDIALS function.
+   /// Returns the last flag retured by a call to a SUNDIALS function.
    int GetFlag() const { return flag; }
 };
 
-/// Wrapper for SUNDIALS' CVODE library -- Multi-step time integration.
-/**
-   - http://computation.llnl.gov/projects/sundials
-   - http://computation.llnl.gov/sites/default/files/public/cv_guide.pdf
 
-   @note All methods except Step() can be called before Init().
-   To minimize uncertainty, we advise the user to adhere to the given
-   interface, instead of making similar calls by the CVODE's
-   internal CVodeMem object.
-*/
+// ---------------------------------------------------------------------------
+// Interface to the CVODE library -- linear multi-step methods
+// ---------------------------------------------------------------------------
+
+/// Interface to the CVODE library -- linear multi-step methods.
 class CVODESolver : public ODESolver, public SundialsSolver
 {
+protected:
+   int lmm_type;  ///< Linear multistep method type.
+   int step_mode; ///< CVODE step mode (CV_NORMAL or CV_ONE_STEP).
+
+   /// Wrapper to compute the ODE rhs function.
+   static int RHS(realtype t, const N_Vector y, N_Vector ydot, void *user_data);
+
+   /// Setup the linear system \f$ A x = b \f$.
+   static int LinSysSetup(realtype t, N_Vector y, N_Vector fy, SUNMatrix A,
+                          booleantype jok, booleantype *jcur,
+                          realtype gamma, void *user_data, N_Vector tmp1,
+                          N_Vector tmp2, N_Vector tmp3);
+
+   /// Solve the linear system \f$ A x = b \f$.
+   static int LinSysSolve(SUNLinearSolver LS, SUNMatrix A, N_Vector x,
+                          N_Vector b, realtype tol);
+
 public:
-   /// Construct a serial CVODESolver, a wrapper for SUNDIALS' CVODE solver.
-   /** @param[in] lmm   Specifies the linear multistep method, the options are
-                        CV_ADAMS (explicit methods) or CV_BDF (implicit
-                        methods).
-       @param[in] iter  Specifies type of nonlinear solver iteration, the
-                        options are CV_FUNCTIONAL (usually with CV_ADAMS) or
-                        CV_NEWTON (usually with CV_BDF).
-       For parameter desciption, see the CVodeCreate documentation (cvode.h). */
-   CVODESolver(int lmm, int iter);
+   /// Construct a serial wrapper to SUNDIALS' CVODE integrator.
+   /** @param[in] lmm Specifies the linear multistep method, the options are:
+                      - CV_ADAMS - implicit methods for non-stiff systems,
+                      - CV_BDF   - implicit methods for stiff systems. */
+   CVODESolver(int lmm);
 
 #ifdef MFEM_USE_MPI
-   /// Construct a parallel CVODESolver, a wrapper for SUNDIALS' CVODE solver.
-   /** @param[in] comm  The MPI communicator used to partition the ODE system.
-       @param[in] lmm   Specifies the linear multistep method, the options are
-                        CV_ADAMS (explicit methods) or CV_BDF (implicit
-                        methods).
-       @param[in] iter  Specifies type of nonlinear solver iteration, the
-                        options are CV_FUNCTIONAL (usually with CV_ADAMS) or
-                        CV_NEWTON (usually with CV_BDF).
-       For parameter desciption, see the CVodeCreate documentation (cvode.h). */
-   CVODESolver(MPI_Comm comm, int lmm, int iter);
+   /// Construct a parallel wrapper to SUNDIALS' CVODE integrator.
+   /** @param[in] comm The MPI communicator used to partition the ODE system
+       @param[in] lmm  Specifies the linear multistep method, the options are:
+                       - CV_ADAMS - implicit methods for non-stiff systems,
+                       - CV_BDF   - implicit methods for stiff systems. */
+   CVODESolver(MPI_Comm comm, int lmm);
 #endif
 
+   /** @brief Initialize CVODE: calls CVodeCreate() to create the CVODE
+       memory and set some defaults.
+
+       If the CVODE memory has already been created, it checks if the problem
+       size has changed since the last call to Init(). If the problem is the
+       same then CVodeReInit() will be called in the next call to Step(). If
+       the problem size has changed, the CVODE memory is freed and realloced
+       for the new problem size. */
+   /** @param[in] f_ The TimeDependentOperator that defines the ODE system.
+
+       @note All other methods must be called after Init().
+
+       @note If this method is called a second time with a different problem
+       size, then any non-default user-set options will be lost and will need
+       to be set again. */
+   void Init(TimeDependentOperator &f_);
+
+   /// Integrate the ODE with CVODE using the specified step mode.
+   /** @param[in,out] x  On output, the solution vector at the requested output
+                         time tout = @a t + @a dt.
+       @param[in,out] t  On output, the output time reached.
+       @param[in,out] dt On output, the last time step taken.
+
+       @note On input, the values of @a t and @a dt are used to compute desired
+       output time for the integration, tout = @a t + @a dt.
+   */
+   virtual void Step(Vector &x, double &t, double &dt);
+
+   /** @brief Attach the linear system setup and solve methods from the
+       TimeDependentOperator i.e., SUNImplicitSetup() and SUNImplicitSolve() to
+       CVODE.
+   */
+   void UseMFEMLinearSolver();
+
+   /// Attach SUNDIALS GMRES linear solver to CVODE.
+   void UseSundialsLinearSolver();
+
+   /// Select the CVODE step mode: CV_NORMAL (default) or CV_ONE_STEP.
+   /** @param[in] itask  The desired step mode. */
+   void SetStepMode(int itask);
+
    /// Set the scalar relative and scalar absolute tolerances.
    void SetSStolerances(double reltol, double abstol);
 
-   /// Set a custom Jacobian system solver for the CV_NEWTON option usually used
-   /// with implicit CV_BDF.
-   void SetLinearSolver(SundialsODELinearSolver &ls_spec);
+   /// Set the maximum time step.
+   void SetMaxStep(double dt_max);
 
-   /** @brief CVode supports two modes, specified by itask: CV_NORMAL (default)
-       and CV_ONE_STEP. */
-   /** In the CV_NORMAL mode, the solver steps until it reaches or passes
-       tout = t + dt, where t and dt are specified in Step(), and then
-       interpolates to obtain y(tout). In the CV_ONE_STEP mode, it takes one
-       internal step and returns. */
-   void SetStepMode(int itask);
+   /** @brief Set the maximum method order.
 
-   /// Set the maximum order of the linear multistep method.
-   /** The default is 12 (CV_ADAMS) or 5 (CV_BDF).
        CVODE uses adaptive-order integration, based on the local truncation
-       error. Use this if you know a priori that your system is such that
-       higher order integration formulas are unstable.
+       error. The default values for @a max_order are 12 for CV_ADAMS and
+       5 for CV_BDF. Use this if you know a priori that your system is such
+       that higher order integration formulas are unstable.
+
        @note @a max_order can't be higher than the current maximum order. */
    void SetMaxOrder(int max_order);
 
-   /// Set the maximum time step of the linear multistep method.
-   void SetMaxStep(double dt_max)
-   { flag = CVodeSetMaxStep(sundials_mem, dt_max); }
-
-   /// Set the ODE right-hand-side operator.
-   /** The start time of CVODE is initialized from the current time of @a f_.
-       @note This method calls CVodeInit(). Some CVODE parameters can be set
-       (using the handle returned by SundialsMem()) only after this call. */
-   virtual void Init(TimeDependentOperator &f_);
-
-   /// Use CVODE to integrate over [t, t + dt], with the specified step mode.
-   /** Calls CVode(), which is the main driver of the CVODE package.
-       @param[in,out] x  Solution vector to advance. On input/output x=x(t)
-                         for t corresponding to the input/output value of t,
-                         respectively.
-       @param[in,out] t  Input: the starting time value. Output: the time value
-                         of the solution output, as returned by CVode().
-       @param[in,out] dt Input: desired time step. Output: the last incremental
-                         time step used. */
-   virtual void Step(Vector &x, double &t, double &dt);
-
-   /// Print CVODE statistics.
+   /// Print various CVODE statistics.
    void PrintInfo() const;
 
-   /// Destroy the associated CVODE memory.
+   /// Destroy the associated CVODE memory and SUNDIALS objects.
    virtual ~CVODESolver();
 };
 
-/// Wrapper for SUNDIALS' ARKODE library -- Runge-Kutta time integration.
-/**
-  - http://computation.llnl.gov/projects/sundials
-  - http://computation.llnl.gov/sites/default/files/public/ark_guide.pdf
 
-   @note All methods except Step() can be called before Init().
-   To minimize uncertainty, we advise the user to adhere to the given
-   interface, instead of making similar calls by the ARKODE's
-   internal ARKodeMem object.
-*/
-class ARKODESolver : public ODESolver, public SundialsSolver
+// ---------------------------------------------------------------------------
+// Interface to ARKode's ARKStep module -- Additive Runge-Kutta methods
+// ---------------------------------------------------------------------------
+
+/// Interface to ARKode's ARKStep module -- additive Runge-Kutta methods.
+class ARKStepSolver : public ODESolver, public SundialsSolver
 {
-protected:
-   bool use_implicit;
-   int irk_table, erk_table;
-
 public:
    /// Types of ARKODE solvers.
-   enum Type { EXPLICIT, IMPLICIT };
+   enum Type
+   {
+      EXPLICIT, ///< Explicit RK method
+      IMPLICIT, ///< Implicit RK method
+      IMEX      ///< Implicit-explicit ARK method
+   };
 
-   /// Construct a serial ARKODESolver, a wrapper for SUNDIALS' ARKODE solver.
-   /** @param[in] type  Specifies the #Type of ARKODE solver to construct. */
-   ARKODESolver(Type type = EXPLICIT);
+protected:
+   Type rk_type;      ///< Runge-Kutta type.
+   int step_mode;     ///< ARKStep step mode (ARK_NORMAL or ARK_ONE_STEP).
+   bool use_implicit; ///< True for implicit or imex integration.
+
+   /** @name Wrappers to compute the ODE RHS functions.
+       RHS1 is explicit RHS and RHS2 the implicit RHS for IMEX integration. When
+       purely implicit or explicit only RHS1 is used. */
+   ///@{
+   static int RHS1(realtype t, const N_Vector y, N_Vector ydot, void *user_data);
+   static int RHS2(realtype t, const N_Vector y, N_Vector ydot, void *user_data);
+   ///@}
+
+   /// Setup the linear system \f$ A x = b \f$.
+   static int LinSysSetup(realtype t, N_Vector y, N_Vector fy, SUNMatrix A,
+                          SUNMatrix M, booleantype jok, booleantype *jcur,
+                          realtype gamma, void *user_data, N_Vector tmp1,
+                          N_Vector tmp2, N_Vector tmp3);
+
+   /// Solve the linear system \f$ A x = b \f$.
+   static int LinSysSolve(SUNLinearSolver LS, SUNMatrix A, N_Vector x,
+                          N_Vector b, realtype tol);
+
+   /// Setup the linear system \f$ M x = b \f$.
+   static int MassSysSetup(realtype t, SUNMatrix M, void *user_data,
+                           N_Vector tmp1, N_Vector tmp2, N_Vector tmp3);
+
+   /// Solve the linear system \f$ M x = b \f$.
+   static int MassSysSolve(SUNLinearSolver LS, SUNMatrix M, N_Vector x,
+                           N_Vector b, realtype tol);
+
+   /// Compute the matrix-vector product \f$ v = M x \f$.
+   static int MassMult1(SUNMatrix M, N_Vector x, N_Vector v);
+
+   /// Compute the matrix-vector product \f$v = M_t x \f$ at time t.
+   static int MassMult2(N_Vector x, N_Vector v, realtype t,
+                        void* mtimes_data);
+
+public:
+   /// Construct a serial wrapper to SUNDIALS' ARKode integrator.
+   /** @param[in] type Specifies the RK method type:
+                       - EXPLICIT - explicit RK method (default)
+                       - IMPLICIT - implicit RK method
+                       - IMEX     - implicit-explicit ARK method */
+   ARKStepSolver(Type type = EXPLICIT);
 
 #ifdef MFEM_USE_MPI
-   /// Construct a parallel ARKODESolver, a wrapper for SUNDIALS' ARKODE solver.
-   /** @param[in] comm  The MPI communicator used to partition the ODE system.
-       @param[in] type  Specifies the #Type of ARKODE solver to construct. */
-   ARKODESolver(MPI_Comm comm, Type type = EXPLICIT);
+   /// Construct a parallel wrapper to SUNDIALS' ARKode integrator.
+   /** @param[in] comm The MPI communicator used to partition the ODE system.
+       @param[in] type Specifies the RK method type:
+                       - EXPLICIT - explicit RK method (default)
+                       - IMPLICIT - implicit RK method
+                       - IMEX     - implicit-explicit ARK method */
+   ARKStepSolver(MPI_Comm comm, Type type = EXPLICIT);
 #endif
 
-   /// Specify the scalar relative and scalar absolute tolerances.
-   void SetSStolerances(double reltol, double abstol);
+   /** @brief Initialize ARKode: calls ARKStepCreate() to create the ARKStep
+       memory and set some defaults.
 
-   /// Set a custom Jacobian system solver for implicit methods.
-   void SetLinearSolver(SundialsODELinearSolver &ls_spec);
+       If the ARKStep has already been created, it checks if the problem size
+       has changed since the last call to Init(). If the problem is the same
+       then ARKStepReInit() will be called in the next call to Step(). If the
+       problem size has changed, the ARKStep memory is freed and realloced
+       for the new problem size. */
+   /** @param[in] f_ The TimeDependentOperator that defines the ODE system
 
-   /** @brief ARKode supports two modes, specified by itask: ARK_NORMAL
-       (default) and ARK_ONE_STEP. */
-   /** In the ARK_NORMAL mode, the solver steps until it reaches or passes
-       tout = t + dt, where t and dt are specified in Step(), and then
-       interpolates to obtain y(tout). In the ARK_ONE_STEP mode, it takes one
-       internal step and returns. */
+       @note All other methods must be called after Init().
+
+       @note If this method is called a second time with a different problem
+       size, then any non-default user-set options will be lost and will need
+       to be set again. */
+   void Init(TimeDependentOperator &f_);
+
+   /// Integrate the ODE with ARKode using the specified step mode.
+   /**
+       @param[in,out] x  On output, the solution vector at the requested output
+                         time, tout = @a t + @a dt
+       @param[in,out] t  On output, the output time reached
+       @param[in,out] dt On output, the last time step taken
+
+       @note On input, the values of @a t and @a dt are used to compute desired
+       output time for the integration, tout = @a t + @a dt.
+   */
+   virtual void Step(Vector &x, double &t, double &dt);
+
+   /** @brief Attach the linear system setup and solve methods from the
+       TimeDependentOperator i.e., SUNImplicitSetup() and SUNImplicitSolve() to
+       ARKode.
+   */
+   void UseMFEMLinearSolver();
+
+   /// Attach a SUNDIALS GMRES linear solver to ARKode.
+   void UseSundialsLinearSolver();
+
+   /** @brief Attach mass matrix linear system setup, solve, and matrix-vector
+       product methods from the TimeDependentOperator i.e., SUNMassSetup(),
+       SUNMassSolve(), and SUNMassMult() to ARKode.
+
+       @param[in] tdep    An integer flag indicating if the mass matrix is time
+                          dependent (1) or time independent (0)
+   */
+   void UseMFEMMassLinearSolver(int tdep);
+
+   /** @brief Attach the SUNDIALS GMRES linear solver and the mass matrix
+       matrix-vector product method from the TimeDependentOperator i.e.,
+       SUNMassMult() to ARKode to solve mass matrix systems.
+
+       @param[in] tdep    An integer flag indicating if the mass matrix is time
+                          dependent (1) or time independent (0)
+   */
+   void UseSundialsMassLinearSolver(int tdep);
+
+   /// Select the ARKode step mode: ARK_NORMAL (default) or ARK_ONE_STEP.
+   /** @param[in] itask  The desired step mode */
    void SetStepMode(int itask);
 
+   /// Set the scalar relative and scalar absolute tolerances.
+   void SetSStolerances(double reltol, double abstol);
+
+   /// Set the maximum time step.
+   void SetMaxStep(double dt_max);
+
    /// Chooses integration order for all explicit / implicit / IMEX methods.
-   /** The default is 4, and the allowed ranges are: [2, 8] for explicit; [2, 5]
-       for implicit; [3, 5] for IMEX. */
+   /** The default is 4, and the allowed ranges are: [2, 8] for explicit;
+       [2, 5] for implicit; [3, 5] for IMEX. */
    void SetOrder(int order);
 
-   /// Choose a specific Butcher table for implicit RK method.
-   /** See the documentation for all possible options, stability regions, etc.
-       For example, table_num = ARK548L2SA_DIRK_8_4_5 is 8-stage 5th order. */
-   void SetIRKTableNum(int table_num);
-   /// Choose a specific Butcher table for explicit RK method.
-   /** See the documentation for all possible options, stability regions, etc.*/
+   /// Choose a specific Butcher table for an explicit RK method.
+   /** See ARKODE documentation for all possible options, stability regions, etc.
+       For example, table_num = BOGACKI_SHAMPINE_4_2_3 is 4-stage 3rd order. */
    void SetERKTableNum(int table_num);
 
-   /** @brief Use a fixed time step size, instead of performing any form of
-       temporal adaptivity. */
+   /// Choose a specific Butcher table for a diagonally implicit RK method.
+   /** See ARKODE documentation for all possible options, stability regions, etc.
+       For example, table_num = CASH_5_3_4 is 5-stage 4th order. */
+   void SetIRKTableNum(int table_num);
+
+   /// Choose a specific Butcher table for an IMEX RK method.
+   /** See ARKODE documentation for all possible options, stability regions, etc.
+       For example, etable_num = ARK548L2SA_DIRK_8_4_5 and
+       itable_num = ARK548L2SA_ERK_8_4_5 is 8-stage 5th order. */
+   void SetIMEXTableNum(int etable_num, int itable_num);
+
+   /// Use a fixed time step size (disable temporal adaptivity).
    /** Use of this function is not recommended, since there is no assurance of
        the validity of the computed solutions. It is primarily provided for
        code-to-code verification testing purposes. */
    void SetFixedStep(double dt);
 
-   /// Set the maximum time step of the Runge-Kutta method.
-   void SetMaxStep(double dt_max)
-   { flag = ARKodeSetMaxStep(sundials_mem, dt_max); }
-
-   /// Set the ODE right-hand-side operator.
-   /** The start time of ARKODE is initialized from the current time of @a f_.
-       @note This method calls ARKodeInit(). Some ARKODE parameters can be set
-       (using the handle returned by SundialsMem()) only after this call. */
-   virtual void Init(TimeDependentOperator &f_);
-
-   /// Use ARKODE to integrate over [t, t + dt], with the specified step mode.
-   /** Calls ARKode(), which is the main driver of the ARKODE package.
-       @param[in,out] x  Solution vector to advance. On input/output x=x(t)
-                         for t corresponding to the input/output value of t,
-                         respectively.
-       @param[in,out] t  Input: the starting time value. Output: the time value
-                         of the solution output, as returned by CVode().
-       @param[in,out] dt Input: desired time step. Output: the last incremental
-                         time step used. */
-   virtual void Step(Vector &x, double &t, double &dt);
-
-   /// Print ARKODE statistics.
+   /// Print various ARKStep statistics.
    void PrintInfo() const;
 
-   /// Destroy the associated ARKODE memory.
-   virtual ~ARKODESolver();
+   /// Destroy the associated ARKode memory and SUNDIALS objects.
+   virtual ~ARKStepSolver();
 };
 
-/// Wrapper for SUNDIALS' KINSOL library -- Nonlinear solvers.
-/**
-   - http://computation.llnl.gov/projects/sundials
-   - http://computation.llnl.gov/sites/default/files/public/kin_guide.pdf
 
-   @note To minimize uncertainty, we advise the user to adhere to the given
-   interface, instead of making similar calls by the KINSOL's
-   internal KINMem object.
-*/
-class KinSolver : public NewtonSolver, public SundialsSolver
+// ---------------------------------------------------------------------------
+// Interface to the KINSOL library -- nonlinear solver methods
+// ---------------------------------------------------------------------------
+
+/// Interface to the KINSOL library -- nonlinear solver methods.
+class KINSolver : public NewtonSolver, public SundialsSolver
 {
 protected:
-   bool use_oper_grad;
-   mutable N_Vector y_scale, f_scale;
-   const Operator *jacobian; // stores the result of oper->GetGradient()
+   int global_strategy;               ///< KINSOL solution strategy
+   bool use_oper_grad;                ///< use the Jv prod function
+   mutable N_Vector y_scale, f_scale; ///< scaling vectors
+   const Operator *jacobian;          ///< stores oper->GetGradient()
+   int maa;                           ///< number of acceleration vectors
 
-   /// @name Auxiliary callback functions.
-   ///@{
-   // Computes the non-linear operator action F(u).
-   // The real type of user_data is pointer to KinSolver.
+   /// Wrapper to compute the nonlinear residual \f$ F(u) = 0 \f$.
    static int Mult(const N_Vector u, N_Vector fu, void *user_data);
 
-   // Computes J(u)v. The real type of user_data is pointer to KinSolver.
+   /// Wrapper to compute the Jacobian-vector product \f$ J(u) v = Jv \f$.
    static int GradientMult(N_Vector v, N_Vector Jv, N_Vector u,
                            booleantype *new_u, void *user_data);
 
-   static int LinSysSetup(KINMemRec *kin_mem);
+   /// Setup the linear system \f$ J u = b \f$.
+   static int LinSysSetup(N_Vector u, N_Vector fu, SUNMatrix J,
+                          void *user_data, N_Vector tmp1, N_Vector tmp2);
 
-   static int LinSysSolve(KINMemRec *kin_mem, N_Vector x, N_Vector b,
-                          realtype *sJpnorm, realtype *sFdotJp);
-   ///@}
+   /// Solve the linear system \f$ J u = b \f$.
+   static int LinSysSolve(SUNLinearSolver LS, SUNMatrix J, N_Vector u,
+                          N_Vector b, realtype tol);
 
 public:
-   /// Construct a serial KinSolver, a wrapper for SUNDIALS' KINSOL solver.
+
+   /// Construct a serial wrapper to SUNDIALS' KINSOL nonlinear solver.
    /** @param[in] strategy   Specifies the nonlinear solver strategy:
                              KIN_NONE / KIN_LINESEARCH / KIN_PICARD / KIN_FP.
        @param[in] oper_grad  Specifies whether the solver should use its
                              Operator's GetGradient() method to compute the
                              Jacobian of the system. */
-   KinSolver(int strategy, bool oper_grad = true);
+   KINSolver(int strategy, bool oper_grad = true);
 
 #ifdef MFEM_USE_MPI
-   /// Construct a parallel KinSolver, a wrapper for SUNDIALS' KINSOL solver.
+   /// Construct a parallel wrapper to SUNDIALS' KINSOL nonlinear solver.
    /** @param[in] comm       The MPI communicator used to partition the system.
        @param[in] strategy   Specifies the nonlinear solver strategy:
                              KIN_NONE / KIN_LINESEARCH / KIN_PICARD / KIN_FP.
        @param[in] oper_grad  Specifies whether the solver should use its
                              Operator's GetGradient() method to compute the
                              Jacobian of the system. */
-   KinSolver(MPI_Comm comm, int strategy, bool oper_grad = true);
+   KINSolver(MPI_Comm comm, int strategy, bool oper_grad = true);
 #endif
 
    /// Destroy the associated KINSOL memory.
-   virtual ~KinSolver();
+   virtual ~KINSolver();
 
-   /// Set the nonlinear Operator of the system. This method calls KINInit().
+   /// Set the nonlinear Operator of the system and initialize KINSOL.
+   /** @note If this method is called a second time with a different problem
+       size, then non-default KINSOL-specific options will be lost and will need
+       to be set again. */
    virtual void SetOperator(const Operator &op);
 
    /// Set the linear solver for inverting the Jacobian.
    /** @note This function assumes that Operator::GetGradient(const Vector &)
              is implemented by the Operator specified by
-             SetOperator(const Operator &). */
+             SetOperator(const Operator &).
+
+             This method must be called after SetOperator(). */
    virtual void SetSolver(Solver &solver);
-   /// Equivalent to SetSolver(Solver).
+
+   /// Equivalent to SetSolver(solver).
    virtual void SetPreconditioner(Solver &solver) { SetSolver(solver); }
 
    /// Set KINSOL's scaled step tolerance.
-   /** The default tolerance is U^(2/3), where U = machine unit roundoff. */
+   /** The default tolerance is \f$ U^\frac{2}{3} \f$ , where
+       U = machine unit roundoff.
+       @note This method must be called after SetOperator(). */
    void SetScaledStepTol(double sstol);
-   /// Set KINSOL's functional norm tolerance.
-   /** The default tolerance is U^(1/3), where U = machine unit roundoff.
-        @note This function is equivalent to SetAbsTol(double). */
-   void SetFuncNormTol(double ftol) { abs_tol = ftol; }
 
    /// Set maximum number of nonlinear iterations without a Jacobian update.
-   /** The default is 10. */
+   /** The default is 10.
+       @note This method must be called after SetOperator(). */
    void SetMaxSetupCalls(int max_calls);
 
-   /// Solve the nonlinear system F(x) = 0.
-   /** Calls the other Mult(Vector&, Vector&, Vector&) const method with
-       `x_scale = 1`. The values of 'fx_scale' are determined by comparing
+   /// Set the number of acceleration vectors to use with KIN_FP or KIN_PICARD.
+   /** The default is 0.
+       @ note This method must be called before SetOperator() to set the
+       maximum size of the acceleration space. The value of @a maa can be
+       altered after SetOperator() is called but it can't be higher than initial
+       maximum. */
+   void SetMAA(int maa);
+
+   /// Solve the nonlinear system \f$ F(x) = 0 \f$.
+   /** This method computes the x_scale and fx_scale vectors and calls the
+       other Mult(Vector&, Vector&, Vector&) const method. The x_scale vector
+       is a vector of ones and values of fx_scale are determined by comparing
        the chosen relative and functional norm (i.e. absolute) tolerances.
-       @param[in]     b  Not used, KINSol always assumes zero RHS.
+       @param[in]     b  Not used, KINSOL always assumes zero RHS
        @param[in,out] x  On input, initial guess, if @a #iterative_mode = true,
                          otherwise the initial guess is zero; on output, the
-                         solution. */
+                         solution */
    virtual void Mult(const Vector &b, Vector &x) const;
 
-   /// Solve the nonlinear system F(x) = 0.
+   /// Solve the nonlinear system \f$ F(x) = 0 \f$.
    /** Calls KINSol() to solve the nonlinear system. Before calling KINSol(),
        this functions uses the data members inherited from class IterativeSolver
        to set corresponding KINSOL options.
-       @param[in,out] x        On input, initial guess, if @a #iterative_mode =
-                               true, otherwise the initial guess is zero; on
-                               output, the solution.
-       @param[in]     x_scale  Elements of a diagonal scaling matrix D, s.t.
-                               D*x has all elements roughly the same when
-                               x is close to a solution.
-       @param[in]    fx_scale  Elements of a diagonal scaling matrix E, s.t.
-                               D*F(x) has all elements roughly the same when
-                               x is not too close to a solution. */
+       @param[in,out] x         On input, initial guess, if @a #iterative_mode =
+                                true, otherwise the initial guess is zero; on
+                                output, the solution
+       @param[in]     x_scale   Elements of a diagonal scaling matrix D, s.t.
+                                D*x has all elements roughly the same when
+                                x is close to a solution
+       @param[in]     fx_scale  Elements of a diagonal scaling matrix E, s.t.
+                                D*F(x) has all elements roughly the same when
+                                x is not too close to a solution */
    void Mult(Vector &x, const Vector &x_scale, const Vector &fx_scale) const;
 };
 

linalg/superlu.cpp

@@ -20,6 +20,10 @@
 #include "superlu_defs.h"
 #include "superlu_ddefs.h"
 
+#if XSDK_INDEX_SIZE == 64
+#error "SuperLUDist has been built with 64bit integers. This is not supported"
+#endif
+
 using namespace std;
 
 namespace mfem
@@ -130,6 +134,11 @@ SuperLURowLocMatrix::SuperLURowLocMatrix( const HypreParMatrix & hypParMat )
    // hypre_CSRMatrix.
    hypre_CSRMatrix * csr_op = hypre_MergeDiagAndOffd(parcsr_op);
    hypre_CSRMatrixSetDataOwner(csr_op,0);
+#if MFEM_HYPRE_VERSION >= 21600
+   MFEM_VERIFY(csr_op->num_rows < INT_MAX,"SuperLU: number of local rows "
+               "is too large to store as an integer.");
+   hypre_CSRMatrixBigJtoJ(csr_op);
+#endif
 
    int m         = parcsr_op->global_num_rows;
    int n         = parcsr_op->global_num_cols;

linalg/vector.cpp

@@ -834,9 +834,10 @@ double Vector::Sum() const
 {
    double sum = 0.0;
 
+   const double *h_data = this->HostRead();
    for (int i = 0; i < size; i++)
    {
-      sum += data[i];
+      sum += h_data[i];
    }
 
    return sum;
@@ -1043,13 +1044,6 @@ vector_min_cpu:
 
 #ifdef MFEM_USE_SUNDIALS
 
-#ifndef SUNTRUE
-#define SUNTRUE TRUE
-#endif
-#ifndef SUNFALSE
-#define SUNFALSE FALSE
-#endif
-
 Vector::Vector(N_Vector nv)
 {
    N_Vector_ID nvid = N_VGetVectorID(nv);

linalg/vector.hpp

@@ -140,6 +140,13 @@ public:
        @sa NewDataAndSize(). */
    inline void NewMemoryAndSize(const Memory<double> &mem, int s, bool own_mem);
 
+   /// Reset the Vector to be a reference to a sub-vector of @a base.
+   inline void MakeRef(Vector &base, int offset, int size);
+
+   /** @brief Reset the Vector to be a reference to a sub-vector of @a base
+       without changing its current size. */
+   inline void MakeRef(Vector &base, int offset);
+
    /// Set the Vector data (host pointer) ownership flag.
    void MakeDataOwner() const { data.SetHostPtrOwner(true); }
 
@@ -455,6 +462,19 @@ inline void Vector::NewMemoryAndSize(const Memory<double> &mem, int s,
    if (!own_mem) { data.ClearOwnerFlags(); }
 }
 
+inline void Vector::MakeRef(Vector &base, int offset, int s)
+{
+   data.Delete();
+   size = s;
+   data.MakeAlias(base.GetMemory(), offset, s);
+}
+
+inline void Vector::MakeRef(Vector &base, int offset)
+{
+   data.Delete();
+   data.MakeAlias(base.GetMemory(), offset, size);
+}
+
 inline void Vector::Destroy()
 {
    const bool use_dev = data.UseDevice();

mesh/element.hpp

@@ -74,8 +74,13 @@ public:
 
    virtual const int *GetEdgeVertices(int) const = 0;
 
+   /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
    virtual int GetNFaces(int &nFaceVertices) const = 0;
 
+   virtual int GetNFaces() const = 0;
+
+   virtual int GetNFaceVertices(int fi) const = 0;
+
    virtual const int *GetFaceVertices(int fi) const = 0;
 
    /// Mark the longest edge by assuming/changing the order of the vertices.

mesh/hexahedron.hpp

@@ -51,9 +51,14 @@ public:
    virtual const int *GetEdgeVertices(int ei) const
    { return geom_t::Edges[ei]; }
 
+   /// @deprecated Use GetNFaces(void) and GetNFaceVertices(int) instead.
    virtual int GetNFaces(int &nFaceVertices) const
    { nFaceVertices = 4; return 6; }
 
+   virtual int GetNFaces() const { return 6; }
+
+   virtual int GetNFaceVertices(int) const { return 4; }
+
    virtual const int *GetFaceVertices(int fi) const
    { return geom_t::FaceVert[fi]; }
 

mesh/mesh.cpp

@@ -430,6 +430,8 @@ void Mesh::GetBdrElementTransformation(int i, IsoparametricTransformation* ElTr)
    else
    {
       const FiniteElement *bdr_el = Nodes->FESpace()->GetBE(i);
+      Nodes->HostRead();
+      const GridFunction &nodes = *Nodes;
       if (bdr_el)
       {
          Array<int> vdofs;
@@ -440,7 +442,7 @@ void Mesh::GetBdrElementTransformation(int i, IsoparametricTransformation* ElTr)
          {
             for (int j = 0; j < n; j++)
             {
-               pm(k,j) = (*Nodes)(vdofs[n*k+j]);
+               pm(k,j) = nodes(vdofs[n*k+j]);
             }
          }
          ElTr->SetFE(bdr_el);
@@ -492,6 +494,8 @@ void Mesh::GetFaceTransformation(int FaceNo, IsoparametricTransformation *FTr)
    else // curved mesh
    {
       const FiniteElement *face_el = Nodes->FESpace()->GetFaceElement(FaceNo);
+      Nodes->HostRead();
+      const GridFunction &nodes = *Nodes;
       if (face_el)
       {
          Array<int> vdofs;
@@ -502,7 +506,7 @@ void Mesh::GetFaceTransformation(int FaceNo, IsoparametricTransformation *FTr)
          {
             for (int j = 0; j < n; j++)
             {
-               pm(i, j) = (*Nodes)(vdofs[n*i+j]);
+               pm(i, j) = nodes(vdofs[n*i+j]);
             }
          }
          FTr->SetFE(face_el);
@@ -3298,6 +3302,10 @@ Mesh::Mesh(Mesh *orig_mesh, int ref_factor, int ref_type)
    Array<int> rdofs;
    DenseMatrix phys_pts;
    int max_nv = 0;
+
+   DenseMatrix node_coordinates(spaceDim*pow(2, Dim), r_num_elem);
+   H1_FECollection vertex_fec(1, Dim);
+
    for (int el = 0; el < orig_mesh->GetNE(); el++)
    {
       Geometry::Type geom = orig_mesh->GetElementBaseGeometry(el);
@@ -3312,6 +3320,7 @@ Mesh::Mesh(Mesh *orig_mesh, int ref_factor, int ref_type)
       orig_mesh->GetElementTransformation(el)->Transform(rfe->GetNodes(),
                                                          phys_pts);
       const int *c2h_map = rfec.GetDofMap(geom);
+      const int *vertex_map = vertex_fec.GetDofMap(geom);
       for (int i = 0; i < phys_pts.Width(); i++)
       {
          vertices[rdofs[i]].SetCoords(spaceDim, phys_pts.GetColumn(i));
@@ -3326,9 +3335,24 @@ Mesh::Mesh(Mesh *orig_mesh, int ref_factor, int ref_type)
             int cid = RG.RefGeoms[k+nvert*j]; // local Cartesian index
             v[k] = rdofs[c2h_map[cid]];
          }
+         for (int k = 0; k < nvert; k++)
+         {
+            for (int j = 0; j < spaceDim; ++j)
+            {
+               node_coordinates(k*spaceDim + j, NumOfElements)
+                  = vertices[v[vertex_map[k]]](j);
+            }
+         }
          AddElement(elem);
       }
    }
+
+   SetCurvature(1, true, spaceDim);
+   Vector node_coordinates_vec(
+      node_coordinates.Data(),
+      node_coordinates.Width()*node_coordinates.Height());
+   SetNodes(node_coordinates_vec);
+
    // Add refined boundary elements
    for (int el = 0; el < orig_mesh->GetNBE(); el++)
    {
@@ -3429,6 +3453,28 @@ void Mesh::KnotInsert(Array<KnotVector *> &kv)
    UpdateNURBS();
 }
 
+void Mesh::KnotInsert(Array<Vector *> &kv)
+{
+   if (NURBSext == NULL)
+   {
+      mfem_error("Mesh::KnotInsert : Not a NURBS mesh!");
+   }
+
+   if (kv.Size() != NURBSext->GetNKV())
+   {
+      mfem_error("Mesh::KnotInsert : KnotVector array size mismatch!");
+   }
+
+   NURBSext->ConvertToPatches(*Nodes);
+
+   NURBSext->KnotInsert(kv);
+
+   last_operation = Mesh::NONE; // FiniteElementSpace::Update is not supported
+   sequence++;
+
+   UpdateNURBS();
+}
+
 void Mesh::NURBSUniformRefinement()
 {
    // do not check for NURBSext since this method is protected
@@ -4759,7 +4805,13 @@ void Mesh::GenerateNCFaceInfo()
    for (unsigned i = 0; i < list.slaves.size(); i++)
    {
       const NCMesh::Slave &slave = list.slaves[i];
-      if (slave.index >= nfaces || slave.master >= nfaces) { continue; }
+
+      if (slave.index < 0 || // degenerate slave face
+          slave.index >= nfaces || // ghost slave
+          slave.master >= nfaces) // has ghost master
+      {
+         continue;
+      }
 
       FaceInfo &slave_fi = faces_info[slave.index];
       FaceInfo &master_fi = faces_info[slave.master];
@@ -4912,15 +4964,39 @@ STable3D *Mesh::GetElementToFaceTable(int ret_ftbl)
    return NULL;
 }
 
+// shift cyclically 3 integers so that the smallest is first
+static inline
+void Rotate3(int &a, int &b, int &c)
+{
+   if (a < b)
+   {
+      if (a > c)
+      {
+         ShiftRight(a, b, c);
+      }
+   }
+   else
+   {
+      if (b < c)
+      {
+         ShiftRight(c, b, a);
+      }
+      else
+      {
+         ShiftRight(a, b, c);
+      }
+   }
+}
+
 void Mesh::ReorientTetMesh()
 {
-   int *v;
-
    if (Dim != 3 || !(meshgen & 1))
    {
       return;
    }
 
+   DeleteLazyTables();
+
    DSTable *old_v_to_v = NULL;
    Table *old_elem_vert = NULL;
 
@@ -4933,7 +5009,7 @@ void Mesh::ReorientTetMesh()
    {
       if (GetElementType(i) == Element::TETRAHEDRON)
       {
-         v = elements[i]->GetVertices();
+         int *v = elements[i]->GetVertices();
 
          Rotate3(v[0], v[1], v[2]);
          if (v[0] < v[3])
@@ -4942,7 +5018,7 @@ void Mesh::ReorientTetMesh()
          }
          else
          {
-            ShiftL2R(v[0], v[1], v[3]);
+            ShiftRight(v[0], v[1], v[3]);
          }
       }
    }
@@ -4951,7 +5027,7 @@ void Mesh::ReorientTetMesh()
    {
       if (GetBdrElementType(i) == Element::TRIANGLE)
       {
-         v = boundary[i]->GetVertices();
+         int *v = boundary[i]->GetVertices();
 
          Rotate3(v[0], v[1], v[2]);
       }
@@ -5783,7 +5859,7 @@ void Mesh::SetVertices(const Vector &vert_coord)
       }
 }
 
-void Mesh::GetNode(int i, double *coord)
+void Mesh::GetNode(int i, double *coord) const
 {
    if (Nodes)
    {
@@ -5934,16 +6010,19 @@ void Mesh::UniformRefinement2D()
    const int oedge = NumOfVertices;
    const int oelem = oedge + NumOfEdges;
 
+   Array<Element*> new_elements;
+   Array<Element*> new_boundary;
+
    vertices.SetSize(oelem + quad_counter);
-   elements.SetSize(4 * NumOfElements);
+   new_elements.SetSize(4 * NumOfElements);
    quad_counter = 0;
-   for (int i = 0; i < NumOfElements; i++)
+
+   for (int i = 0, j = 0; i < NumOfElements; i++)
    {
       const Element::Type el_type = elements[i]->GetType();
       const int attr = elements[i]->GetAttribute();
       int *v = elements[i]->GetVertices();
       const int *e = el_to_edge->GetRow(i);
-      const int j = NumOfElements + 3 * i;
       int vv[2];
 
       if (el_type == Element::TRIANGLE)
@@ -5957,12 +6036,14 @@ void Mesh::UniformRefinement2D()
             AverageVertices(vv, 2, oedge+e[ei]);
          }
 
-         elements[j+0] = new Triangle(oedge+e[1], oedge+e[2], oedge+e[0], attr);
-         elements[j+1] = new Triangle(oedge+e[0], v[1], oedge+e[1], attr);
-         elements[j+2] = new Triangle(oedge+e[2], oedge+e[1], v[2], attr);
-
-         v[1] = oedge+e[0];
-         v[2] = oedge+e[2];
+         new_elements[j++] =
+            new Triangle(v[0], oedge+e[0], oedge+e[2], attr);
+         new_elements[j++] =
+            new Triangle(oedge+e[1], oedge+e[2], oedge+e[0], attr);
+         new_elements[j++] =
+            new Triangle(oedge+e[0], v[1], oedge+e[1], attr);
+         new_elements[j++] =
+            new Triangle(oedge+e[2], oedge+e[1], v[2], attr);
       }
       else if (el_type == Element::QUADRILATERAL)
       {
@@ -5979,34 +6060,36 @@ void Mesh::UniformRefinement2D()
             AverageVertices(vv, 2, oedge+e[ei]);
          }
 
-         elements[j+0] = new Quadrilateral(oedge+e[0], v[1], oedge+e[1],
-                                           oelem+qe, attr);
-         elements[j+1] = new Quadrilateral(oelem+qe, oedge+e[1],
-                                           v[2], oedge+e[2], attr);
-         elements[j+2] = new Quadrilateral(oedge+e[3], oelem+qe,
-                                           oedge+e[2], v[3], attr);
-
-         v[1] = oedge+e[0];
-         v[2] = oelem+qe;
-         v[3] = oedge+e[3];
+         new_elements[j++] =
+            new Quadrilateral(v[0], oedge+e[0], oelem+qe, oedge+e[3], attr);
+         new_elements[j++] =
+            new Quadrilateral(oedge+e[0], v[1], oedge+e[1], oelem+qe, attr);
+         new_elements[j++] =
+            new Quadrilateral(oelem+qe, oedge+e[1], v[2], oedge+e[2], attr);
+         new_elements[j++] =
+            new Quadrilateral(oedge+e[3], oelem+qe, oedge+e[2], v[3], attr);
       }
       else
       {
          MFEM_ABORT("unknown element type: " << el_type);
       }
+      FreeElement(elements[i]);
    }
+   mfem::Swap(elements, new_elements);
 
-   boundary.SetSize(2 * NumOfBdrElements);
-   for (int i = 0; i < NumOfBdrElements; i++)
+   // refine boundary elements
+   new_boundary.SetSize(2 * NumOfBdrElements);
+   for (int i = 0, j = 0; i < NumOfBdrElements; i++)
    {
       const int attr = boundary[i]->GetAttribute();
       int *v = boundary[i]->GetVertices();
-      const int j = NumOfBdrElements + i;
 
-      boundary[j] = new Segment(oedge+be_to_edge[i], v[1], attr);
+      new_boundary[j++] = new Segment(v[0], oedge+be_to_edge[i], attr);
+      new_boundary[j++] = new Segment(oedge+be_to_edge[i], v[1], attr);
 
-      v[1] = oedge+be_to_edge[i];
+      FreeElement(boundary[i]);
    }
+   mfem::Swap(boundary, new_boundary);
 
    static const double A = 0.0, B = 0.5, C = 1.0;
    static double tri_children[2*3*4] =
@@ -6024,17 +6107,17 @@ void Mesh::UniformRefinement2D()
       A,B, B,B, B,C, A,C  // upper-left
    };
 
-   CoarseFineTr.point_matrices[Geometry::TRIANGLE].
-   UseExternalData(tri_children, 2, 3, 4);
-   CoarseFineTr.point_matrices[Geometry::SQUARE].
-   UseExternalData(quad_children, 2, 4, 4);
+   CoarseFineTr.point_matrices[Geometry::TRIANGLE]
+   .UseExternalData(tri_children, 2, 3, 4);
+   CoarseFineTr.point_matrices[Geometry::SQUARE]
+   .UseExternalData(quad_children, 2, 4, 4);
    CoarseFineTr.embeddings.SetSize(elements.Size());
 
    for (int i = 0; i < elements.Size(); i++)
    {
       Embedding &emb = CoarseFineTr.embeddings[i];
-      emb.parent = (i < NumOfElements) ? i : (i - NumOfElements) / 3;
-      emb.matrix = (i < NumOfElements) ? 0 : (i - NumOfElements) % 3 + 1;
+      emb.parent = i / 4;
+      emb.matrix = i % 4;
    }
 
    NumOfVertices    = vertices.Size();
@@ -6166,17 +6249,20 @@ void Mesh::UniformRefinement3D_base(Array<int> *f2qf_ptr, DSTable *v_to_v_p)
    const int oface = oedge + NumOfEdges;
    const int oelem = oface + NumOfQuadFaces;
 
+   Array<Element*> new_elements;
+   Array<Element*> new_boundary;
+
    vertices.SetSize(oelem + hex_counter);
-   elements.SetSize(8 * NumOfElements);
-   CoarseFineTr.embeddings.SetSize(elements.Size());
+   new_elements.SetSize(8 * NumOfElements);
+   CoarseFineTr.embeddings.SetSize(new_elements.Size());
+
    hex_counter = 0;
-   for (int i = 0; i < NumOfElements; i++)
+   for (int i = 0, j = 0; i < NumOfElements; i++)
    {
       const Element::Type el_type = elements[i]->GetType();
       const int attr = elements[i]->GetAttribute();
       int *v = elements[i]->GetVertices();
       const int *e = el_to_edge->GetRow(i);
-      const int j = NumOfElements + 7 * i;
       int vv[4], ev[12];
 
       if (e2v.Size())
@@ -6322,51 +6408,54 @@ void Mesh::UniformRefinement3D_base(Array<int> *f2qf_ptr, DSTable *v_to_v_p)
             const int (&mv)[4][4] = mv_all[rt];
 
 #ifndef MFEM_USE_MEMALLOC
-            elements[j+0] = new Tetrahedron(oedge+e[0], v[1],
-                                            oedge+e[3], oedge+e[4], attr);
-            elements[j+1] = new Tetrahedron(oedge+e[1], oedge+e[3],
-                                            v[2], oedge+e[5], attr);
-            elements[j+2] = new Tetrahedron(oedge+e[2], oedge+e[4],
-                                            oedge+e[5], v[3], attr);
+            new_elements[j+0] =
+               new Tetrahedron(v[0], oedge+e[0], oedge+e[1], oedge+e[2], attr);
+            new_elements[j+1] =
+               new Tetrahedron(oedge+e[0], v[1], oedge+e[3], oedge+e[4], attr);
+            new_elements[j+2] =
+               new Tetrahedron(oedge+e[1], oedge+e[3], v[2], oedge+e[5], attr);
+            new_elements[j+3] =
+               new Tetrahedron(oedge+e[2], oedge+e[4], oedge+e[5], v[3], attr);
+
             for (int k = 0; k < 4; k++)
             {
-               elements[j+k+3] =
+               new_elements[j+4+k] =
                   new Tetrahedron(oedge+e[mv[k][0]], oedge+e[mv[k][1]],
                                   oedge+e[mv[k][2]], oedge+e[mv[k][3]], attr);
             }
 #else
             Tetrahedron *tet;
-            elements[j+0] = tet = TetMemory.Alloc();
+            new_elements[j+0] = tet = TetMemory.Alloc();
+            tet->Init(v[0], oedge+e[0], oedge+e[1], oedge+e[2], attr);
+
+            new_elements[j+1] = tet = TetMemory.Alloc();
             tet->Init(oedge+e[0], v[1], oedge+e[3], oedge+e[4], attr);
-            elements[j+1] = tet = TetMemory.Alloc();
+
+            new_elements[j+2] = tet = TetMemory.Alloc();
             tet->Init(oedge+e[1], oedge+e[3], v[2], oedge+e[5], attr);
-            elements[j+2] = tet = TetMemory.Alloc();
+
+            new_elements[j+3] = tet = TetMemory.Alloc();
             tet->Init(oedge+e[2], oedge+e[4], oedge+e[5], v[3], attr);
+
             for (int k = 0; k < 4; k++)
             {
-               elements[j+k+3] = tet = TetMemory.Alloc();
+               new_elements[j+4+k] = tet = TetMemory.Alloc();
                tet->Init(oedge+e[mv[k][0]], oedge+e[mv[k][1]],
                          oedge+e[mv[k][2]], oedge+e[mv[k][3]], attr);
             }
 #endif
-
-            v[1] = oedge+e[0];
-            v[2] = oedge+e[1];
-            v[3] = oedge+e[2];
-            ((Tetrahedron*)elements[i])->SetRefinementFlag(0);
-
-            CoarseFineTr.embeddings[i].parent = i;
-            CoarseFineTr.embeddings[i].matrix = 0;
-            for (int k = 0; k < 3; k++)
+            for (int k = 0; k < 4; k++)
             {
                CoarseFineTr.embeddings[j+k].parent = i;
-               CoarseFineTr.embeddings[j+k].matrix = k+1;
+               CoarseFineTr.embeddings[j+k].matrix = k;
             }
             for (int k = 0; k < 4; k++)
             {
-               CoarseFineTr.embeddings[j+k+3].parent = i;
-               CoarseFineTr.embeddings[j+k+3].matrix = 4*(rt+1)+k;
+               CoarseFineTr.embeddings[j+4+k].parent = i;
+               CoarseFineTr.embeddings[j+4+k].matrix = 4*(rt+1)+k;
             }
+
+            j += 8;
          }
          break;
 
@@ -6396,33 +6485,37 @@ void Mesh::UniformRefinement3D_base(Array<int> *f2qf_ptr, DSTable *v_to_v_p)
             const int qf3 = f2qf[f[3]];
             const int qf4 = f2qf[f[4]];
 
-            elements[j+0] = new Wedge(oedge+e[1], oedge+e[2], oedge+e[0],
-                                      oface+qf3, oface+qf4, oface+qf2,
-                                      attr);
-            elements[j+1] = new Wedge(oedge+e[0], v[1], oedge+e[1],
-                                      oface+qf2, oedge+e[7], oface+qf3,
-                                      attr);
-            elements[j+2] = new Wedge(oedge+e[2], oedge+e[1], v[2],
-                                      oface+qf4, oface+qf3, oedge+e[8],
-                                      attr);
-            elements[j+3] = new Wedge(oedge+e[6], oface+qf2, oface+qf4,
-                                      v[3], oedge+e[3], oedge+e[5],
-                                      attr);
-            elements[j+4] = new Wedge(oface+qf3, oface+qf4, oface+qf2,
-                                      oedge+e[4], oedge+e[5], oedge+e[3],
-                                      attr);
-            elements[j+5] = new Wedge(oface+qf2, oedge+e[7], oface+qf3,
-                                      oedge+e[3], v[4], oedge+e[4],
-                                      attr);
-            elements[j+6] = new Wedge(oface+qf4, oface+qf3, oedge+e[8],
-                                      oedge+e[5], oedge+e[4], v[5],
-                                      attr);
+            new_elements[j++] =
+               new Wedge(v[0], oedge+e[0], oedge+e[2],
+                         oedge+e[6], oface+qf2, oface+qf4, attr);
 
-            v[1] = oedge+e[0];
-            v[2] = oedge+e[2];
-            v[3] = oedge+e[6];
-            v[4] = oface+qf2;
-            v[5] = oface+qf4;
+            new_elements[j++] =
+               new Wedge(oedge+e[1], oedge+e[2], oedge+e[0],
+                         oface+qf3, oface+qf4, oface+qf2, attr);
+
+            new_elements[j++] =
+               new Wedge(oedge+e[0], v[1], oedge+e[1],
+                         oface+qf2, oedge+e[7], oface+qf3, attr);
+
+            new_elements[j++] =
+               new Wedge(oedge+e[2], oedge+e[1], v[2],
+                         oface+qf4, oface+qf3, oedge+e[8], attr);
+
+            new_elements[j++] =
+               new Wedge(oedge+e[6], oface+qf2, oface+qf4,
+                         v[3], oedge+e[3], oedge+e[5], attr);
+
+            new_elements[j++] =
+               new Wedge(oface+qf3, oface+qf4, oface+qf2,
+                         oedge+e[4], oedge+e[5], oedge+e[3], attr);
+
+            new_elements[j++] =
+               new Wedge(oface+qf2, oedge+e[7], oface+qf3,
+                         oedge+e[3], v[4], oedge+e[4], attr);
+
+            new_elements[j++] =
+               new Wedge(oface+qf4, oface+qf3, oedge+e[8],
+                         oedge+e[5], oedge+e[4], v[5], attr);
          }
          break;
 
@@ -6464,35 +6557,38 @@ void Mesh::UniformRefinement3D_base(Array<int> *f2qf_ptr, DSTable *v_to_v_p)
                AverageVertices(vv, 2, oedge+e[ei]);
             }
 
-            elements[j+0] = new Hexahedron(oedge+e[0], v[1], oedge+e[1],
-                                           oface+qf[0], oface+qf[1], oedge+e[9],
-                                           oface+qf[2], oelem+he, attr);
-            elements[j+1] = new Hexahedron(oface+qf[0], oedge+e[1], v[2],
-                                           oedge+e[2], oelem+he, oface+qf[2],
-                                           oedge+e[10], oface+qf[3], attr);
-            elements[j+2] = new Hexahedron(oedge+e[3], oface+qf[0], oedge+e[2],
-                                           v[3], oface+qf[4], oelem+he,
-                                           oface+qf[3], oedge+e[11], attr);
-            elements[j+3] = new Hexahedron(oedge+e[8], oface+qf[1], oelem+he,
-                                           oface+qf[4], v[4], oedge+e[4],
-                                           oface+qf[5], oedge+e[7], attr);
-            elements[j+4] = new Hexahedron(oface+qf[1], oedge+e[9], oface+qf[2],
-                                           oelem+he, oedge+e[4], v[5],
-                                           oedge+e[5], oface+qf[5], attr);
-            elements[j+5] = new Hexahedron(oelem+he, oface+qf[2], oedge+e[10],
-                                           oface+qf[3], oface+qf[5], oedge+e[5],
-                                           v[6], oedge+e[6], attr);
-            elements[j+6] = new Hexahedron(oface+qf[4], oelem+he, oface+qf[3],
-                                           oedge+e[11], oedge+e[7], oface+qf[5],
-                                           oedge+e[6], v[7], attr);
-
-            v[1] = oedge+e[0];
-            v[2] = oface+qf[0];
-            v[3] = oedge+e[3];
-            v[4] = oedge+e[8];
-            v[5] = oface+qf[1];
-            v[6] = oelem+he;
-            v[7] = oface+qf[4];
+            new_elements[j++] =
+               new Hexahedron(v[0], oedge+e[0], oface+qf[0],
+                              oedge+e[3], oedge+e[8], oface+qf[1],
+                              oelem+he, oface+qf[4], attr);
+            new_elements[j++] =
+               new Hexahedron(oedge+e[0], v[1], oedge+e[1],
+                              oface+qf[0], oface+qf[1], oedge+e[9],
+                              oface+qf[2], oelem+he, attr);
+            new_elements[j++] =
+               new Hexahedron(oface+qf[0], oedge+e[1], v[2],
+                              oedge+e[2], oelem+he, oface+qf[2],
+                              oedge+e[10], oface+qf[3], attr);
+            new_elements[j++] =
+               new Hexahedron(oedge+e[3], oface+qf[0], oedge+e[2],
+                              v[3], oface+qf[4], oelem+he,
+                              oface+qf[3], oedge+e[11], attr);
+            new_elements[j++] =
+               new Hexahedron(oedge+e[8], oface+qf[1], oelem+he,
+                              oface+qf[4], v[4], oedge+e[4],
+                              oface+qf[5], oedge+e[7], attr);
+            new_elements[j++] =
+               new Hexahedron(oface+qf[1], oedge+e[9], oface+qf[2],
+                              oelem+he, oedge+e[4], v[5],
+                              oedge+e[5], oface+qf[5], attr);
+            new_elements[j++] =
+               new Hexahedron(oelem+he, oface+qf[2], oedge+e[10],
+                              oface+qf[3], oface+qf[5], oedge+e[5],
+                              v[6], oedge+e[6], attr);
+            new_elements[j++] =
+               new Hexahedron(oface+qf[4], oelem+he, oface+qf[3],
+                              oedge+e[11], oedge+e[7], oface+qf[5],
+                              oedge+e[6], v[7], attr);
          }
          break;
 
@@ -6500,16 +6596,18 @@ void Mesh::UniformRefinement3D_base(Array<int> *f2qf_ptr, DSTable *v_to_v_p)
             MFEM_ABORT("Unknown 3D element type \"" << el_type << "\"");
             break;
       }
+      FreeElement(elements[i]);
    }
+   mfem::Swap(elements, new_elements);
 
-   boundary.SetSize(4 * NumOfBdrElements);
-   for (int i = 0; i < NumOfBdrElements; i++)
+   // refine boundary elements
+   new_boundary.SetSize(4 * NumOfBdrElements);
+   for (int i = 0, j = 0; i < NumOfBdrElements; i++)
    {
       const Element::Type bdr_el_type = boundary[i]->GetType();
       const int attr = boundary[i]->GetAttribute();
       int *v = boundary[i]->GetVertices();
       const int *e = bel_to_edge->GetRow(i);
-      const int j = NumOfBdrElements + 3 * i;
       int ev[4];
 
       if (e2v.Size())
@@ -6521,34 +6619,36 @@ void Mesh::UniformRefinement3D_base(Array<int> *f2qf_ptr, DSTable *v_to_v_p)
 
       if (bdr_el_type == Element::TRIANGLE)
       {
-         boundary[j+0] = new Triangle(oedge+e[1], oedge+e[2], oedge+e[0], attr);
-         boundary[j+1] = new Triangle(oedge+e[0], v[1], oedge+e[1], attr);
-         boundary[j+2] = new Triangle(oedge+e[2], oedge+e[1], v[2], attr);
-
-         v[1] = oedge+e[0];
-         v[2] = oedge+e[2];
+         new_boundary[j++] =
+            new Triangle(v[0], oedge+e[0], oedge+e[2], attr);
+         new_boundary[j++] =
+            new Triangle(oedge+e[1], oedge+e[2], oedge+e[0], attr);
+         new_boundary[j++] =
+            new Triangle(oedge+e[0], v[1], oedge+e[1], attr);
+         new_boundary[j++] =
+            new Triangle(oedge+e[2], oedge+e[1], v[2], attr);
       }
       else if (bdr_el_type == Element::QUADRILATERAL)
       {
          const int qf =
             (f2qf.Size() == 0) ? be_to_face[i] : f2qf[be_to_face[i]];
 
-         boundary[j+0] = new Quadrilateral(oedge+e[0], v[1], oedge+e[1],
-                                           oface+qf, attr);
-         boundary[j+1] = new Quadrilateral(oface+qf, oedge+e[1], v[2],
-                                           oedge+e[2], attr);
-         boundary[j+2] = new Quadrilateral(oedge+e[3], oface+qf,
-                                           oedge+e[2], v[3], attr);
-
-         v[1] = oedge+e[0];
-         v[2] = oface+qf;
-         v[3] = oedge+e[3];
+         new_boundary[j++] =
+            new Quadrilateral(v[0], oedge+e[0], oface+qf, oedge+e[3], attr);
+         new_boundary[j++] =
+            new Quadrilateral(oedge+e[0], v[1], oedge+e[1], oface+qf, attr);
+         new_boundary[j++] =
+            new Quadrilateral(oface+qf, oedge+e[1], v[2], oedge+e[2], attr);
+         new_boundary[j++] =
+            new Quadrilateral(oedge+e[3], oface+qf, oedge+e[2], v[3], attr);
       }
       else
       {
          MFEM_ABORT("boundary Element is not a triangle or a quad!");
       }
+      FreeElement(boundary[i]);
    }
+   mfem::Swap(boundary, new_boundary);
 
    static const double A = 0.0, B = 0.5, C = 1.0;
    static double tet_children[3*4*16] =
@@ -6599,20 +6699,21 @@ void Mesh::UniformRefinement3D_base(Array<int> *f2qf_ptr, DSTable *v_to_v_p)
       A,B,B, B,B,B, B,C,B, A,C,B, A,B,C, B,B,C, B,C,C, A,C,C
    };
 
-   CoarseFineTr.point_matrices[Geometry::TETRAHEDRON].
-   UseExternalData(tet_children, 3, 4, 16);
-   CoarseFineTr.point_matrices[Geometry::PRISM].
-   UseExternalData(pri_children, 3, 6, 8);
-   CoarseFineTr.point_matrices[Geometry::CUBE].
-   UseExternalData(hex_children, 3, 8, 8);
+   CoarseFineTr.point_matrices[Geometry::TETRAHEDRON]
+   .UseExternalData(tet_children, 3, 4, 16);
+   CoarseFineTr.point_matrices[Geometry::PRISM]
+   .UseExternalData(pri_children, 3, 6, 8);
+   CoarseFineTr.point_matrices[Geometry::CUBE]
+   .UseExternalData(hex_children, 3, 8, 8);
 
    for (int i = 0; i < elements.Size(); i++)
    {
-      // Tetrahedron elements are handled above:
+      // tetrahedron elements are handled above:
       if (elements[i]->GetType() == Element::TETRAHEDRON) { continue; }
+
       Embedding &emb = CoarseFineTr.embeddings[i];
-      emb.parent = (i < NumOfElements) ? i : (i - NumOfElements) / 7;
-      emb.matrix = (i < NumOfElements) ? 0 : (i - NumOfElements) % 7 + 1;
+      emb.parent = i / 8;
+      emb.matrix = i % 8;
    }
 
    NumOfVertices    = vertices.Size();
@@ -7196,13 +7297,13 @@ void Mesh::GeneralRefinement(const Array<Refinement> &refinements,
    else if (nonconforming < 0)
    {
       // determine if nonconforming refinement is suitable
-      if (meshgen & 2)
+      if ((meshgen & 2) || (meshgen & 4))
       {
-         nonconforming = 1;
+         nonconforming = 1; // tensor product elements and wedges
       }
       else
       {
-         nonconforming = 0;
+         nonconforming = 0; // simplices
       }
    }
 
@@ -7257,8 +7358,9 @@ void Mesh::EnsureNCMesh(bool triangles_nonconforming)
 
    if (!ncmesh)
    {
-      if ((meshgen & 2) /* quads/hexes */ ||
-          (triangles_nonconforming && Dim == 2 && (meshgen & 1)))
+      if ((meshgen & 0x2) /* quads/hexes */ ||
+          (meshgen & 0x4) /* wedges */ ||
+          (triangles_nonconforming && Dim == 2 && (meshgen & 0x1)))
       {
          MFEM_VERIFY(GetNumGeometries(Dim) <= 1,
                      "mixed meshes are not supported");
@@ -7681,12 +7783,7 @@ void Mesh::UniformRefinement(int i, const DSTable &v_to_v,
 void Mesh::InitRefinementTransforms()
 {
    // initialize CoarseFineTr
-   map<Geometry::Type,DenseTensor> &pms = CoarseFineTr.point_matrices;
-   map<Geometry::Type,DenseTensor>::iterator pms_iter;
-   for (pms_iter = pms.begin(); pms_iter != pms.end(); ++pms_iter)
-   {
-      pms_iter->second.SetSize(0, 0, 0);
-   }
+   CoarseFineTr.Clear();
    CoarseFineTr.embeddings.SetSize(NumOfElements);
    for (int i = 0; i < NumOfElements; i++)
    {
@@ -10016,4 +10113,51 @@ Mesh *Extrude2D(Mesh *mesh, const int nz, const double sz)
    return mesh3d;
 }
 
+#ifdef MFEM_DEBUG
+void Mesh::DebugDump(std::ostream &out) const
+{
+   // dump vertices and edges (NCMesh "nodes")
+   out << NumOfVertices + NumOfEdges << "\n";
+   for (int i = 0; i < NumOfVertices; i++)
+   {
+      const double *v = GetVertex(i);
+      out << i << " " << v[0] << " " << v[1] << " " << v[2]
+          << " 0 0 " << i << " -1 0\n";
+   }
+
+   Array<int> ev;
+   for (int i = 0; i < NumOfEdges; i++)
+   {
+      GetEdgeVertices(i, ev);
+      double mid[3] = {0, 0, 0};
+      for (int j = 0; j < 2; j++)
+      {
+         for (int k = 0; k < spaceDim; k++)
+         {
+            mid[k] += GetVertex(ev[j])[k];
+         }
+      }
+      out << NumOfVertices+i << " "
+          << mid[0]/2 << " " << mid[1]/2 << " " << mid[2]/2 << " "
+          << ev[0] << " " << ev[1] << " -1 " << i << " 0\n";
+   }
+
+   // dump elements
+   out << NumOfElements << "\n";
+   for (int i = 0; i < NumOfElements; i++)
+   {
+      const Element* e = elements[i];
+      out << e->GetNVertices() << " ";
+      for (int j = 0; j < e->GetNVertices(); j++)
+      {
+         out << e->GetVertices()[j] << " ";
+      }
+      out << e->GetAttribute() << " 0 " << i << "\n";
+   }
+
+   // dump faces
+   out << "0\n";
+}
+#endif
+
 }

mesh/mesh.hpp

@@ -386,11 +386,6 @@ protected:
       return FaceIsInterior(FaceNo) || (faces_info[FaceNo].Elem2Inf >= 0);
    }
 
-   // shift cyclically 3 integers left-to-right
-   inline static void ShiftL2R(int &, int &, int &);
-   // shift cyclically 3 integers so that the smallest is first
-   inline static void Rotate3(int &, int &, int &);
-
    void FreeElement(Element *E);
 
    void GenerateFaces();
@@ -1014,7 +1009,7 @@ public:
    // Nodes are only active for higher order meshes, and share locations with
    // the vertices, plus all the higher- order control points within the element
    // and along the edges and on the faces.
-   void GetNode(int i, double *coord);
+   void GetNode(int i, double *coord) const;
    void SetNode(int i, const double *coord);
 
    // Node operations for curved mesh.
@@ -1122,6 +1117,7 @@ public:
 
    ///@{ @name NURBS mesh refinement methods
    void KnotInsert(Array<KnotVector *> &kv);
+   void KnotInsert(Array<Vector *> &kv);
    /* For each knot vector:
          new_degree = max(old_degree, min(old_degree + rel_degree, degree)). */
    void DegreeElevate(int rel_degree, int degree = 16);
@@ -1270,6 +1266,11 @@ public:
 
    /// Destroys Mesh.
    virtual ~Mesh() { DestroyPointers(); }
+
+#ifdef MFEM_DEBUG
+   /// Output an NCMesh-compatible debug dump.
+   void DebugDump(std::ostream &out) const;
+#endif
 };
 
 /** Overload operator<< for std::ostream and Mesh; valid also for the derived
@@ -1358,35 +1359,13 @@ public:
 };
 
 
-// inline functions
-inline void Mesh::ShiftL2R(int &a, int &b, int &c)
+// shift cyclically 3 integers left-to-right
+inline void ShiftRight(int &a, int &b, int &c)
 {
    int t = a;
    a = c;  c = b;  b = t;
 }
 
-inline void Mesh::Rotate3(int &a, int &b, int &c)
-{
-   if (a < b)
-   {
-      if (a > c)
-      {
-         ShiftL2R(a, b, c);
-      }
-   }
-   else
-   {
-      if (b < c)
-      {
-         ShiftL2R(c, b, a);
-      }
-      else
-      {
-         ShiftL2R(a, b, c);
-      }
-   }
-}
-
 }
 
 #endif

mesh/ncmesh.hpp

@@ -15,6 +15,7 @@
 #include "../config/config.hpp"
 #include "../general/hash.hpp"
 #include "../general/globals.hpp"
+#include "../general/sort_pairs.hpp"
 #include "../linalg/densemat.hpp"
 #include "element.hpp"
 #include "vertex.hpp"
@@ -59,24 +60,23 @@ struct Embedding
 struct CoarseFineTransformations
 {
    /// Matrices for IsoparametricTransformation organized by Geometry::Type
-   std::map<Geometry::Type, DenseTensor> point_matrices;
+   DenseTensor point_matrices[Geometry::NumGeom];
    /// Fine element positions in their parents.
    Array<Embedding> embeddings;
 
-   const DenseTensor &GetPointMatrices(Geometry::Type geom) const;
-
    void GetCoarseToFineMap(const Mesh &fine_mesh,
                            Table &coarse_to_fine,
                            Array<int> &coarse_to_ref_type,
                            Table &ref_type_to_matrix,
                            Array<Geometry::Type> &ref_type_to_geom) const;
 
-   void Clear() { point_matrices.clear(); embeddings.DeleteAll(); }
+   void Clear();
+   bool IsInitialized() const;
    long MemoryUsage() const;
 };
 
 
-/** \brief A class for non-conforming AMR on higher-order hexahedral,
+/** \brief A class for non-conforming AMR on higher-order hexahedral, prismatic,
  *  quadrilateral or triangular meshes.
  *
  *  The class is used as follows:
@@ -152,10 +152,13 @@ public:
    {
       int index;   ///< Mesh number
       int element; ///< NCMesh::Element containing this vertex/edge/face
-      int local;   ///< local number within 'element'
+      char local;  ///< local number within 'element'
+      char geom;   ///< Geometry::Type (faces only) (char storage to save RAM)
 
-      MeshId(int index = -1, int element = -1, int local = -1)
-         : index(index), element(element), local(local) {}
+      MeshId(int index = -1, int element = -1, char local = -1, char geom = -1)
+         : index(index), element(element), local(local), geom(geom) {}
+
+      Geometry::Type Geom() const { return Geometry::Type(geom); }
    };
 
    /** Nonconforming edge/face that has more than one neighbor. The neighbors
@@ -164,8 +167,9 @@ public:
    {
       int slaves_begin, slaves_end; ///< slave faces
 
-      Master(int index, int element, int local, int sb, int se)
-         : MeshId(index, element, local), slaves_begin(sb), slaves_end(se) {}
+      Master(int index, int element, char local, char geom, int sb, int se)
+         : MeshId(index, element, local, geom)
+         , slaves_begin(sb), slaves_end(se) {}
    };
 
    /// Nonconforming edge/face within a bigger edge/face.
@@ -175,8 +179,9 @@ public:
       int edge_flags; ///< edge orientation flags
       DenseMatrix point_matrix; ///< position within the master edge/face
 
-      Slave(int index, int element, int local)
-         : MeshId(index, element, local), master(-1), edge_flags(0) {}
+      Slave(int index, int element, char local, char geom)
+         : MeshId(index, element, local, geom)
+         , master(-1), edge_flags(0) {}
 
       /// Return the point matrix oriented according to the master and slave edges
       void OrientedPointMatrix(DenseMatrix &oriented_matrix) const;
@@ -286,10 +291,11 @@ public:
        processor independent. TODO: this seems only partially true? */
    int GetEdgeNCOrientation(const MeshId &edge_id) const;
 
-   /// Return Mesh vertex and edge indices of a face identified by 'face_id'.
-   void GetFaceVerticesEdges(const MeshId &face_id,
-                             int vert_index[4], int edge_index[4],
-                             int edge_orientation[4]) const;
+   /** Return Mesh vertex and edge indices of a face identified by 'face_id'.
+       The return value is the number of face vertices. */
+   int GetFaceVerticesEdges(const MeshId &face_id,
+                            int vert_index[4], int edge_index[4],
+                            int edge_orientation[4]) const;
 
    /** Given an edge (by its vertex indices v1 and v2) return the first
        (geometric) parent edge that exists in the Mesh or -1 if there is no such
@@ -305,14 +311,29 @@ public:
                                    Array<int> &bdr_vertices,
                                    Array<int> &bdr_edges);
 
-   /// Return the type of elements in the mesh.
-   Geometry::Type GetElementGeometry() const { return elements[0].geom; }
+   /// Return element geometry type. @a index is the Mesh element number.
+   Geometry::Type GetElementGeometry(int index) const
+   { return elements[leaf_elements[index]].Geom(); }
 
-   Geometry::Type GetFaceGeometry() const { return Geometry::SQUARE; }
+   /// Return face geometry type. @a index is the Mesh face number.
+   Geometry::Type GetFaceGeometry(int index) const
+   { return Geometry::Type(face_geom[index]); }
+
+   /// Return the number of root elements.
+   int GetNumRootElements() { return root_state.Size(); }
 
    /// Return the distance of leaf 'i' from the root.
    int GetElementDepth(int i) const;
 
+   /** Return the size reduction compared to the root element (ignoring local
+       stretching and curvature). */
+   int GetElementSizeReduction(int i) const;
+
+   /// Return the faces and face attributes of leaf element 'i'.
+   void GetElementFacesAttributes(int i, Array<int> &faces,
+                                  Array<int> &fattr) const;
+
+
    /// I/O: Print the "vertex_parents" section of the mesh file (ver. >= 1.1).
    void PrintVertexParents(std::ostream &out) const;
 
@@ -339,15 +360,17 @@ public:
 
    void PrintStats(std::ostream &out = mfem::out) const;
 
+   typedef int64_t RefCoord;
+
 
 protected: // interface for Mesh to be able to construct itself from NCMesh
 
    friend class Mesh;
 
    /// Return the basic Mesh arrays for the current finest level.
-   void GetMeshComponents(Array<mfem::Vertex>& mvertices,
-                          Array<mfem::Element*>& melements,
-                          Array<mfem::Element*>& mboundary) const;
+   void GetMeshComponents(Array<mfem::Vertex> &mvertices,
+                          Array<mfem::Element*> &melements,
+                          Array<mfem::Element*> &mboundary) const;
 
    /** Get edge and face numbering from 'mesh' (i.e., set all Edge::index and
        Face::index) after a new mesh was created from us. */
@@ -358,6 +381,7 @@ protected: // implementation
 
    int Dim, spaceDim; ///< dimensions of the elements and the vertex coordinates
    bool Iso; ///< true if the mesh only contains isotropic refinements
+   int Geoms; ///< bit mask of element geometries present, see InitGeomFlags()
 
    /** A Node can hold a vertex, an edge, or both. Elements directly point to
        their corner nodes, but edge nodes also exist and can be accessed using
@@ -411,7 +435,7 @@ protected: // implementation
        to its vertex nodes. */
    struct Element
    {
-      Geometry::Type geom; ///< Geometry::Type of the element
+      char geom;     ///< Geometry::Type of the element (char for storage only)
       char ref_type; ///< bit mask of X,Y,Z refinements (bits 0,1,2 respectively)
       char flag;     ///< generic flag/marker, can be used by algorithms
       int index;     ///< element number in the Mesh, -1 if refined
@@ -425,6 +449,8 @@ protected: // implementation
       int parent; ///< parent element, -1 if this is a root element, -2 if free
 
       Element(Geometry::Type geom, int attr);
+
+      Geometry::Type Geom() const { return Geometry::Type(geom); }
    };
 
    // primary data
@@ -470,6 +496,7 @@ protected: // implementation
    NCList vertex_list; ///< lazy-initialized list of vertices, see GetVertexList
 
    Array<int> boundary_faces; ///< subset of all faces, set by BuildFaceList
+   Array<char> face_geom; ///< face geometry by face index, set by OnMeshUpdated
 
    Table element_vertex; ///< leaf-element to vertex table, see FindSetNeighbors
 
@@ -491,10 +518,15 @@ protected: // implementation
    virtual int GetNumGhostElements() const { return 0; }
    virtual int GetNumGhostVertices() const { return 0; }
 
+   void InitGeomFlags();
+   bool HavePrisms() const { return Geoms & (1 << Geometry::PRISM); }
+
 
    // refinement/derefinement
 
    Array<Refinement> ref_stack; ///< stack of scheduled refinements (temporary)
+   HashTable<Node> shadow; ///< temporary storage for reparented nodes
+   Array<Triple<int, int, int> > reparents; ///< scheduled node reparents (tmp)
 
    Table derefinements; ///< possible derefinements, see GetDerefinementTable
 
@@ -519,13 +551,16 @@ protected: // implementation
    }
 
    int NewHexahedron(int n0, int n1, int n2, int n3,
-                     int n4, int n5, int n6, int n7,
-                     int attr,
+                     int n4, int n5, int n6, int n7, int attr,
                      int fattr0, int fattr1, int fattr2,
                      int fattr3, int fattr4, int fattr5);
 
-   int NewQuadrilateral(int n0, int n1, int n2, int n3,
-                        int attr,
+   int NewWedge(int n0, int n1, int n2,
+                int n3, int n4, int n5, int attr,
+                int fattr0, int fattr1,
+                int fattr2, int fattr3, int fattr4);
+
+   int NewQuadrilateral(int n0, int n1, int n2, int n3, int attr,
                         int eattr0, int eattr1, int eattr2, int eattr3);
 
    int NewTriangle(int n0, int n1, int n2,
@@ -533,57 +568,62 @@ protected: // implementation
 
    mfem::Element* NewMeshElement(int geom) const;
 
-   int GetMidEdgeNode(int vn1, int vn2);
-   int GetMidFaceNode(int en1, int en2, int en3, int en4);
+   int QuadFaceSplitType(int v1, int v2, int v3, int v4, int mid[5]
+                         = NULL /*optional output of mid-edge nodes*/) const;
 
-   int FaceSplitType(int v1, int v2, int v3, int v4, int mid[4]
-                     = NULL /*optional output of mid-edge nodes*/) const;
+   bool TriFaceSplit(int v1, int v2, int v3, int mid[3] = NULL) const;
 
    void ForceRefinement(int vn1, int vn2, int vn3, int vn4);
 
+   void FindEdgeElements(int vn1, int vn2, int vn3, int vn4,
+                         Array<MeshId> &prisms) const;
+
+   void CheckAnisoPrism(int vn1, int vn2, int vn3, int vn4,
+                        const Refinement *refs, int nref);
+
    void CheckAnisoFace(int vn1, int vn2, int vn3, int vn4,
                        int mid12, int mid34, int level = 0);
 
    void CheckIsoFace(int vn1, int vn2, int vn3, int vn4,
                      int en1, int en2, int en3, int en4, int midf);
 
-   void RefElement(int elem);
-   void UnrefElement(int elem, Array<int> &elemFaces);
+   void ReparentNode(int node, int new_p1, int new_p2);
+
+   int FindMidEdgeNode(int node1, int node2) const;
+   int GetMidEdgeNode(int node1, int node2);
+
+   int GetMidFaceNode(int en1, int en2, int en3, int en4);
+
+   void ReferenceElement(int elem);
+   void UnreferenceElement(int elem, Array<int> &elemFaces);
 
    Face* GetFace(Element &elem, int face_no);
    void RegisterFaces(int elem, int *fattr = NULL);
    void DeleteUnusedFaces(const Array<int> &elemFaces);
 
-   int FindAltParents(int node1, int node2);
-
-   bool NodeSetX1(int node, int* n);
-   bool NodeSetX2(int node, int* n);
-   bool NodeSetY1(int node, int* n);
-   bool NodeSetY2(int node, int* n);
-   bool NodeSetZ1(int node, int* n);
-   bool NodeSetZ2(int node, int* n);
-
    void CollectDerefinements(int elem, Array<Connection> &list);
 
    /// Return el.node[index] correctly, even if the element is refined.
    int RetrieveNode(const Element &el, int index);
 
-   /// Extended version of find_node: works if 'el' is refined; optional abort.
-   int FindNodeExt(const Element &el, int node, bool abort = false);
+   /// Extended version of find_node: works if 'el' is refined.
+   int FindNodeExt(const Element &el, int node, bool abort = true);
 
 
    // face/edge lists
 
    static int find_node(const Element &el, int node);
-   static int find_element_edge(const Element &el, int vn0, int vn1);
-   static int find_hex_face(int a, int b, int c);
+   static int find_element_edge(const Element &el, int vn0, int vn1,
+                                bool abort = true);
+   static int find_local_face(int geom, int a, int b, int c);
 
    int ReorderFacePointMat(int v0, int v1, int v2, int v3,
                            int elem, DenseMatrix& mat) const;
    struct PointMatrix;
-   void TraverseFace(int vn0, int vn1, int vn2, int vn3,
-                     const PointMatrix& pm, int level);
-
+   void TraverseQuadFace(int vn0, int vn1, int vn2, int vn3,
+                         const PointMatrix& pm, int level, Face* eface[4]);
+   void TraverseTriFace(int vn0, int vn1, int vn2,
+                        const PointMatrix& pm, int level);
    void TraverseEdge(int vn0, int vn1, double t0, double t1, int flags,
                      int level);
 
@@ -626,8 +666,9 @@ protected: // implementation
 
 
    void CollectEdgeVertices(int v0, int v1, Array<int> &indices);
-   void CollectFaceVertices(int v0, int v1, int v2, int v3,
-                            Array<int> &indices);
+   void CollectTriFaceVertices(int v0, int v1, int v2, Array<int> &indices);
+   void CollectQuadFaceVertices(int v0, int v1, int v2, int v3,
+                                Array<int> &indices);
    void BuildElementToVertexTable();
 
    void UpdateElementToVertexTable()
@@ -635,6 +676,15 @@ protected: // implementation
       if (element_vertex.Size() < 0) { BuildElementToVertexTable(); }
    }
 
+   int GetVertexRootCoord(int elem, RefCoord coord[3]) const;
+   void CollectIncidentElements(int elem, const RefCoord coord[3],
+                                Array<int> &list) const;
+
+   /** Return elements neighboring to a local vertex of element 'elem'. Only
+       elements from within the same refinement tree ('cousins') are returned.
+       Complexity is proportional to the depth of elem's refinement tree. */
+   void FindVertexCousins(int elem, int local, Array<int> &cousins) const;
+
 
    // coarse/fine transformations
 
@@ -690,6 +740,13 @@ protected: // implementation
       PointMatrix(const Point& p0, const Point& p1, const Point& p2, const Point& p3)
       { np = 4; points[0] = p0; points[1] = p1; points[2] = p2; points[3] = p3; }
 
+      PointMatrix(const Point& p0, const Point& p1, const Point& p2,
+                  const Point& p3, const Point& p4, const Point& p5)
+      {
+         np = 6;
+         points[0] = p0; points[1] = p1; points[2] = p2;
+         points[3] = p3; points[4] = p4; points[5] = p5;
+      }
       PointMatrix(const Point& p0, const Point& p1, const Point& p2,
                   const Point& p3, const Point& p4, const Point& p5,
                   const Point& p6, const Point& p7)
@@ -707,11 +764,13 @@ protected: // implementation
 
    static PointMatrix pm_tri_identity;
    static PointMatrix pm_quad_identity;
+   static PointMatrix pm_prism_identity;
    static PointMatrix pm_hex_identity;
 
-   static const PointMatrix& GetGeomIdentity(int geom);
+   static const PointMatrix& GetGeomIdentity(Geometry::Type geom);
 
-   void GetPointMatrix(int geom, const char* ref_path, DenseMatrix& matrix);
+   void GetPointMatrix(Geometry::Type geom, const char* ref_path,
+                       DenseMatrix& matrix);
 
    typedef std::map<std::string, int> RefPathMap;
 
@@ -748,9 +807,10 @@ protected: // implementation
 
    void FindFaceNodes(int face, int node[4]);
 
-   int  EdgeSplitLevel(int vn1, int vn2) const;
-   void FaceSplitLevel(int vn1, int vn2, int vn3, int vn4,
-                       int& h_level, int& v_level) const;
+   int EdgeSplitLevel(int vn1, int vn2) const;
+   int TriFaceSplitLevel(int vn1, int vn2, int vn3) const;
+   void QuadFaceSplitLevel(int vn1, int vn2, int vn3, int vn4,
+                           int& h_level, int& v_level) const;
 
    void CountSplits(int elem, int splits[3]) const;
    void GetLimitRefinements(Array<Refinement> &refinements, int max_level);
@@ -765,9 +825,10 @@ protected: // implementation
        (triangles, quads, cubes) */
    struct GeomInfo
    {
-      int nv, ne, nf, nfv; // number of: vertices, edges, faces, face vertices
-      int edges[12][2];    // edge vertices (up to 12 edges)
-      int faces[6][4];     // face vertices (up to 6 faces)
+      int nv, ne, nf;   // number of: vertices, edges, faces
+      int edges[12][2]; // edge vertices (up to 12 edges)
+      int faces[6][4];  // face vertices (up to 6 faces)
+      int nfv[6];       // number of face vertices
 
       bool initialized;
       GeomInfo() : initialized(false) {}
@@ -776,7 +837,7 @@ protected: // implementation
 
    static GeomInfo GI[Geometry::NumGeom];
 
-   static GeomInfo &gi_hex, &gi_quad, &gi_tri;
+   static GeomInfo &gi_hex, &gi_wedge, &gi_quad, &gi_tri;
 
 #ifdef MFEM_DEBUG
 public:

mesh/ncmesh_tables.hpp

@@ -0,0 +1,469 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+namespace mfem
+{
+
+static int ref_type_num_children[8] = { 0, 2, 2, 4, 2, 4, 4, 8 };
+
+
+// derefinement tables
+
+static int quad_deref_table[3][4 + 4] =
+{
+   { 0, 1, 1, 0, /**/ 1, 1, 0, 0 }, // 1 - X
+   { 0, 0, 1, 1, /**/ 0, 0, 1, 1 }, // 2 - Y
+   { 0, 1, 2, 3, /**/ 1, 1, 3, 3 }  // 3 - iso
+};
+
+static int hex_deref_table[7][8 + 6] =
+{
+   { 0, 1, 1, 0, 0, 1, 1, 0, /**/ 1, 1, 1, 0, 0, 0 }, // 1 - X
+   { 0, 0, 1, 1, 0, 0, 1, 1, /**/ 0, 0, 0, 1, 1, 1 }, // 2 - Y
+   { 0, 1, 2, 3, 0, 1, 2, 3, /**/ 1, 1, 1, 3, 3, 3 }, // 3 - XY
+   { 0, 0, 0, 0, 1, 1, 1, 1, /**/ 0, 0, 0, 1, 1, 1 }, // 4 - Z
+   { 0, 1, 1, 0, 3, 2, 2, 3, /**/ 1, 1, 1, 3, 3, 3 }, // 5 - XZ
+   { 0, 0, 1, 1, 2, 2, 3, 3, /**/ 0, 0, 0, 3, 3, 3 }, // 6 - YZ
+   { 0, 1, 2, 3, 4, 5, 6, 7, /**/ 1, 1, 1, 7, 7, 7 }  // 7 - iso
+};
+
+static int prism_deref_table[7][6 + 5] =
+{
+   {-1,-1,-1,-1,-1,-1, /**/ -1,-1,-1,-1,-1 }, // 1
+   {-1,-1,-1,-1,-1,-1, /**/ -1,-1,-1,-1,-1 }, // 2
+   { 0, 1, 2, 0, 1, 2, /**/  0, 0, 0, 1, 0 }, // 3 - XY
+   { 0, 0, 0, 1, 1, 1, /**/  0, 1, 0, 0, 0 }, // 4 - Z
+   {-1,-1,-1,-1,-1,-1, /**/ -1,-1,-1,-1,-1 }, // 5
+   {-1,-1,-1,-1,-1,-1, /**/ -1,-1,-1,-1,-1 }, // 6
+   { 0, 1, 2, 4, 5, 6, /**/  0, 5, 0, 5, 0 }  // 7 - iso
+};
+
+
+// child ordering tables
+
+static char quad_hilbert_child_order[8][4] =
+{
+   {0,1,2,3}, {0,3,2,1}, {1,2,3,0}, {1,0,3,2},
+   {2,3,0,1}, {2,1,0,3}, {3,0,1,2}, {3,2,1,0}
+};
+
+static char quad_hilbert_child_state[8][4] =
+{
+   {1,0,0,5}, {0,1,1,4}, {3,2,2,7}, {2,3,3,6},
+   {5,4,4,1}, {4,5,5,0}, {7,6,6,3}, {6,7,7,2}
+};
+
+static char hex_hilbert_child_order[24][8] =
+{
+   {0,1,2,3,7,6,5,4}, {0,3,7,4,5,6,2,1}, {0,4,5,1,2,6,7,3},
+   {1,0,3,2,6,7,4,5}, {1,2,6,5,4,7,3,0}, {1,5,4,0,3,7,6,2},
+   {2,1,5,6,7,4,0,3}, {2,3,0,1,5,4,7,6}, {2,6,7,3,0,4,5,1},
+   {3,0,4,7,6,5,1,2}, {3,2,1,0,4,5,6,7}, {3,7,6,2,1,5,4,0},
+   {4,0,1,5,6,2,3,7}, {4,5,6,7,3,2,1,0}, {4,7,3,0,1,2,6,5},
+   {5,1,0,4,7,3,2,6}, {5,4,7,6,2,3,0,1}, {5,6,2,1,0,3,7,4},
+   {6,2,3,7,4,0,1,5}, {6,5,1,2,3,0,4,7}, {6,7,4,5,1,0,3,2},
+   {7,3,2,6,5,1,0,4}, {7,4,0,3,2,1,5,6}, {7,6,5,4,0,1,2,3}
+};
+
+static char hex_hilbert_child_state[24][8] =
+{
+   {1,2,2,7,7,21,21,17},     {2,0,0,22,22,16,16,8},    {0,1,1,15,15,6,6,23},
+   {4,5,5,10,10,18,18,14},   {5,3,3,19,19,13,13,11},   {3,4,4,12,12,9,9,20},
+   {8,7,7,17,17,23,23,2},    {6,8,8,0,0,15,15,22},     {7,6,6,21,21,1,1,16},
+   {11,10,10,14,14,20,20,5}, {9,11,11,3,3,12,12,19},   {10,9,9,18,18,4,4,13},
+   {13,14,14,5,5,19,19,10},  {14,12,12,20,20,11,11,4}, {12,13,13,9,9,3,3,18},
+   {16,17,17,2,2,22,22,7},   {17,15,15,23,23,8,8,1},   {15,16,16,6,6,0,0,21},
+   {20,19,19,11,11,14,14,3}, {18,20,20,4,4,10,10,12},  {19,18,18,13,13,5,5,9},
+   {23,22,22,8,8,17,17,0},   {21,23,23,1,1,7,7,15},    {22,21,21,16,16,2,2,6}
+};
+
+
+// child/parent reference domain transforms
+
+typedef NCMesh::RefCoord RefCoord;
+
+// reference domain coordinates as fixed point numbers
+const RefCoord T_HALF = (1ll << 59);
+const RefCoord T_ONE = (1ll << 60);
+const RefCoord T_TWO = (1ll << 61);
+
+// (scaling factors have a different fixed point multiplier)
+const RefCoord S_HALF = 1;
+const RefCoord S_ONE = 2;
+const RefCoord S_TWO = 4;
+
+static RefCoord tri_corners[3][3] =
+{
+   {    0,     0, 0},
+   {T_ONE,     0, 0},
+   {    0, T_ONE, 0}
+};
+
+static RefCoord quad_corners[4][3] =
+{
+   {    0,     0, 0},
+   {T_ONE,     0, 0},
+   {T_ONE, T_ONE, 0},
+   {    0, T_ONE, 0}
+};
+
+static RefCoord hex_corners[8][3] =
+{
+   {    0,     0,     0},
+   {T_ONE,     0,     0},
+   {T_ONE, T_ONE,     0},
+   {    0, T_ONE,     0},
+   {    0,     0, T_ONE},
+   {T_ONE,     0, T_ONE},
+   {T_ONE, T_ONE, T_ONE},
+   {    0, T_ONE, T_ONE}
+};
+
+static RefCoord prism_corners[6][3] =
+{
+   {    0,     0,     0},
+   {T_ONE,     0,     0},
+   {    0, T_ONE,     0},
+   {    0,     0, T_ONE},
+   {T_ONE,     0, T_ONE},
+   {    0, T_ONE, T_ONE}
+};
+
+typedef RefCoord RefPoint[3];
+static RefPoint* geom_corners[7] =
+{
+   NULL, // point
+   NULL, // segment
+   tri_corners,
+   quad_corners,
+   NULL, // tetrahedron
+   hex_corners,
+   prism_corners
+};
+
+// reference domain transform: 3 scales, 3 translations
+struct RefTrf
+{
+   RefCoord s[3], t[3];
+
+   void Apply(const RefCoord src[3], RefCoord dst[3]) const;
+};
+
+static RefTrf quad_parent_rt1[2] =
+{
+   { {S_HALF, S_ONE, 0}, {     0, 0, 0} },
+   { {S_HALF, S_ONE, 0}, {T_HALF, 0, 0} }
+};
+
+static RefTrf quad_child_rt1[2] =
+{
+   { {S_TWO, S_ONE, 0}, {     0, 0, 0} },
+   { {S_TWO, S_ONE, 0}, {-T_ONE, 0, 0} }
+};
+
+static RefTrf quad_parent_rt2[2] =
+{
+   { {S_ONE, S_HALF, 0}, {0,      0, 0} },
+   { {S_ONE, S_HALF, 0}, {0, T_HALF, 0} }
+};
+
+static RefTrf quad_child_rt2[2] =
+{
+   { {S_ONE, S_TWO, 0}, {0,      0, 0} },
+   { {S_ONE, S_TWO, 0}, {0, -T_ONE, 0} }
+};
+
+static RefTrf quad_parent_rt3[4] =
+{
+   { {S_HALF, S_HALF, 0}, {     0,      0, 0} },
+   { {S_HALF, S_HALF, 0}, {T_HALF,      0, 0} },
+   { {S_HALF, S_HALF, 0}, {T_HALF, T_HALF, 0} },
+   { {S_HALF, S_HALF, 0}, {     0, T_HALF, 0} }
+};
+
+static RefTrf quad_child_rt3[4] =
+{
+   { {S_TWO, S_TWO, 0}, {     0,      0, 0} },
+   { {S_TWO, S_TWO, 0}, {-T_ONE,      0, 0} },
+   { {S_TWO, S_TWO, 0}, {-T_ONE, -T_ONE, 0} },
+   { {S_TWO, S_TWO, 0}, {     0, -T_ONE, 0} }
+};
+
+static RefTrf* quad_parent[4] =
+{
+   NULL,
+   quad_parent_rt1,
+   quad_parent_rt2,
+   quad_parent_rt3
+};
+
+static RefTrf* quad_child[4] =
+{
+   NULL,
+   quad_child_rt1,
+   quad_child_rt2,
+   quad_child_rt3
+};
+
+static RefTrf hex_parent_rt1[2] =
+{
+   { {S_HALF, S_ONE, S_ONE}, {     0, 0, 0} },
+   { {S_HALF, S_ONE, S_ONE}, {T_HALF, 0, 0} }
+};
+
+static RefTrf hex_child_rt1[2] =
+{
+   { {S_TWO, S_ONE, S_ONE}, {     0, 0, 0} },
+   { {S_TWO, S_ONE, S_ONE}, {-T_ONE, 0, 0} }
+};
+
+static RefTrf hex_parent_rt2[2] =
+{
+   { {S_ONE, S_HALF, S_ONE}, {0,      0, 0} },
+   { {S_ONE, S_HALF, S_ONE}, {0, T_HALF, 0} }
+};
+
+static RefTrf hex_child_rt2[2] =
+{
+   { {S_ONE, S_TWO, S_ONE}, {0,      0, 0} },
+   { {S_ONE, S_TWO, S_ONE}, {0, -T_ONE, 0} }
+};
+
+static RefTrf hex_parent_rt3[4] =
+{
+   { {S_HALF, S_HALF, S_ONE}, {     0,      0, 0} },
+   { {S_HALF, S_HALF, S_ONE}, {T_HALF,      0, 0} },
+   { {S_HALF, S_HALF, S_ONE}, {T_HALF, T_HALF, 0} },
+   { {S_HALF, S_HALF, S_ONE}, {     0, T_HALF, 0} }
+};
+
+static RefTrf hex_child_rt3[4] =
+{
+   { {S_TWO, S_TWO, S_ONE}, {     0,      0, 0} },
+   { {S_TWO, S_TWO, S_ONE}, {-T_ONE,      0, 0} },
+   { {S_TWO, S_TWO, S_ONE}, {-T_ONE, -T_ONE, 0} },
+   { {S_TWO, S_TWO, S_ONE}, {     0, -T_ONE, 0} }
+};
+
+static RefTrf hex_parent_rt4[2] =
+{
+   { {S_ONE, S_ONE, S_HALF}, {0, 0,      0} },
+   { {S_ONE, S_ONE, S_HALF}, {0, 0, T_HALF} }
+};
+
+static RefTrf hex_child_rt4[2] =
+{
+   { {S_ONE, S_ONE, S_TWO}, {0, 0,      0} },
+   { {S_ONE, S_ONE, S_TWO}, {0, 0, -T_ONE} }
+};
+
+static RefTrf hex_parent_rt5[4] =
+{
+   { {S_HALF, S_ONE, S_HALF}, {     0, 0,      0} },
+   { {S_HALF, S_ONE, S_HALF}, {T_HALF, 0,      0} },
+   { {S_HALF, S_ONE, S_HALF}, {T_HALF, 0, T_HALF} },
+   { {S_HALF, S_ONE, S_HALF}, {     0, 0, T_HALF} }
+};
+
+static RefTrf hex_child_rt5[4] =
+{
+   { {S_TWO, S_ONE, S_TWO}, {     0, 0,      0} },
+   { {S_TWO, S_ONE, S_TWO}, {-T_ONE, 0,      0} },
+   { {S_TWO, S_ONE, S_TWO}, {-T_ONE, 0, -T_ONE} },
+   { {S_TWO, S_ONE, S_TWO}, {     0, 0, -T_ONE} }
+};
+
+static RefTrf hex_parent_rt6[4] =
+{
+   { {S_ONE, S_HALF, S_HALF}, {0,      0,      0} },
+   { {S_ONE, S_HALF, S_HALF}, {0, T_HALF,      0} },
+   { {S_ONE, S_HALF, S_HALF}, {0,      0, T_HALF} },
+   { {S_ONE, S_HALF, S_HALF}, {0, T_HALF, T_HALF} }
+};
+
+static RefTrf hex_child_rt6[4] =
+{
+   { {S_ONE, S_TWO, S_TWO}, {0,      0,      0} },
+   { {S_ONE, S_TWO, S_TWO}, {0, -T_ONE,      0} },
+   { {S_ONE, S_TWO, S_TWO}, {0,      0, -T_ONE} },
+   { {S_ONE, S_TWO, S_TWO}, {0, -T_ONE, -T_ONE} }
+};
+
+static RefTrf hex_parent_rt7[8] =
+{
+   { {S_HALF, S_HALF, S_HALF}, {     0,      0,      0} },
+   { {S_HALF, S_HALF, S_HALF}, {T_HALF,      0,      0} },
+   { {S_HALF, S_HALF, S_HALF}, {T_HALF, T_HALF,      0} },
+   { {S_HALF, S_HALF, S_HALF}, {     0, T_HALF,      0} },
+   { {S_HALF, S_HALF, S_HALF}, {     0,      0, T_HALF} },
+   { {S_HALF, S_HALF, S_HALF}, {T_HALF,      0, T_HALF} },
+   { {S_HALF, S_HALF, S_HALF}, {T_HALF, T_HALF, T_HALF} },
+   { {S_HALF, S_HALF, S_HALF}, {     0, T_HALF, T_HALF} }
+};
+
+static RefTrf hex_child_rt7[8] =
+{
+   { {S_TWO, S_TWO, S_TWO}, {     0,      0,      0} },
+   { {S_TWO, S_TWO, S_TWO}, {-T_ONE,      0,      0} },
+   { {S_TWO, S_TWO, S_TWO}, {-T_ONE, -T_ONE,      0} },
+   { {S_TWO, S_TWO, S_TWO}, {     0, -T_ONE,      0} },
+   { {S_TWO, S_TWO, S_TWO}, {     0,      0, -T_ONE} },
+   { {S_TWO, S_TWO, S_TWO}, {-T_ONE,      0, -T_ONE} },
+   { {S_TWO, S_TWO, S_TWO}, {-T_ONE, -T_ONE, -T_ONE} },
+   { {S_TWO, S_TWO, S_TWO}, {     0, -T_ONE, -T_ONE} }
+};
+
+static RefTrf* hex_parent[8] =
+{
+   NULL,
+   hex_parent_rt1,
+   hex_parent_rt2,
+   hex_parent_rt3,
+   hex_parent_rt4,
+   hex_parent_rt5,
+   hex_parent_rt6,
+   hex_parent_rt7
+};
+
+static RefTrf* hex_child[8] =
+{
+   NULL,
+   hex_child_rt1,
+   hex_child_rt2,
+   hex_child_rt3,
+   hex_child_rt4,
+   hex_child_rt5,
+   hex_child_rt6,
+   hex_child_rt7
+};
+
+static RefTrf tri_parent_rt3[4] =
+{
+   { { S_HALF,  S_HALF, 0}, {     0,      0, 0} },
+   { { S_HALF,  S_HALF, 0}, {T_HALF,      0, 0} },
+   { { S_HALF,  S_HALF, 0}, {     0, T_HALF, 0} },
+   { {-S_HALF, -S_HALF, 0}, {T_HALF, T_HALF, 0} }
+};
+
+static RefTrf tri_child_rt3[4] =
+{
+   { { S_TWO,  S_TWO, 0}, {     0,      0, 0} },
+   { { S_TWO,  S_TWO, 0}, {-T_ONE,      0, 0} },
+   { { S_TWO,  S_TWO, 0}, {     0, -T_ONE, 0} },
+   { {-S_TWO, -S_TWO, 0}, { T_ONE,  T_ONE, 0} }
+};
+
+static RefTrf* tri_parent[4] =
+{
+   NULL, NULL, NULL,
+   tri_parent_rt3
+};
+
+static RefTrf* tri_child[4] =
+{
+   NULL, NULL, NULL,
+   tri_child_rt3
+};
+
+static RefTrf prism_parent_rt3[4] =
+{
+   { { S_HALF,  S_HALF, S_ONE}, {     0,      0, 0} },
+   { { S_HALF,  S_HALF, S_ONE}, {T_HALF,      0, 0} },
+   { { S_HALF,  S_HALF, S_ONE}, {     0, T_HALF, 0} },
+   { {-S_HALF, -S_HALF, S_ONE}, {T_HALF, T_HALF, 0} }
+};
+
+static RefTrf prism_child_rt3[4] =
+{
+   { { S_TWO,  S_TWO, S_ONE}, {     0,      0, 0} },
+   { { S_TWO,  S_TWO, S_ONE}, {-T_ONE,      0, 0} },
+   { { S_TWO,  S_TWO, S_ONE}, {     0, -T_ONE, 0} },
+   { {-S_TWO, -S_TWO, S_ONE}, { T_ONE,  T_ONE, 0} }
+};
+
+static RefTrf prism_parent_rt4[2] =
+{
+   { {S_ONE, S_ONE, S_HALF}, {0, 0,      0} },
+   { {S_ONE, S_ONE, S_HALF}, {0, 0, T_HALF} }
+};
+
+static RefTrf prism_child_rt4[2] =
+{
+   { {S_ONE, S_ONE, S_TWO}, {0, 0,      0} },
+   { {S_ONE, S_ONE, S_TWO}, {0, 0, -T_ONE} }
+};
+
+static RefTrf prism_parent_rt7[8] =
+{
+   { { S_HALF,  S_HALF, S_HALF}, {     0,      0,      0} },
+   { { S_HALF,  S_HALF, S_HALF}, {T_HALF,      0,      0} },
+   { { S_HALF,  S_HALF, S_HALF}, {     0, T_HALF,      0} },
+   { {-S_HALF, -S_HALF, S_HALF}, {T_HALF, T_HALF,      0} },
+   { { S_HALF,  S_HALF, S_HALF}, {     0,      0, T_HALF} },
+   { { S_HALF,  S_HALF, S_HALF}, {T_HALF,      0, T_HALF} },
+   { { S_HALF,  S_HALF, S_HALF}, {     0, T_HALF, T_HALF} },
+   { {-S_HALF, -S_HALF, S_HALF}, {T_HALF, T_HALF, T_HALF} }
+};
+
+static RefTrf prism_child_rt7[8] =
+{
+   { { S_TWO,  S_TWO, S_TWO}, {     0,      0,      0} },
+   { { S_TWO,  S_TWO, S_TWO}, {-T_ONE,      0,      0} },
+   { { S_TWO,  S_TWO, S_TWO}, {     0, -T_ONE,      0} },
+   { {-S_TWO, -S_TWO, S_TWO}, { T_ONE,  T_ONE,      0} },
+   { { S_TWO,  S_TWO, S_TWO}, {     0,      0, -T_ONE} },
+   { { S_TWO,  S_TWO, S_TWO}, {-T_ONE,      0, -T_ONE} },
+   { { S_TWO,  S_TWO, S_TWO}, {     0, -T_ONE, -T_ONE} },
+   { {-S_TWO, -S_TWO, S_TWO}, { T_ONE,  T_ONE, -T_ONE} }
+};
+
+static RefTrf* prism_parent[8] =
+{
+   NULL, NULL, NULL,
+   prism_parent_rt3,
+   prism_parent_rt4,
+   NULL, NULL,
+   prism_parent_rt7
+};
+
+static RefTrf* prism_child[8] =
+{
+   NULL, NULL, NULL,
+   prism_child_rt3,
+   prism_child_rt4,
+   NULL, NULL,
+   prism_child_rt7
+};
+
+static RefTrf** geom_parent[7] =
+{
+   NULL,
+   NULL,
+   tri_parent,
+   quad_parent,
+   NULL,
+   hex_parent,
+   prism_parent
+};
+
+static RefTrf** geom_child[7] =
+{
+   NULL,
+   NULL,
+   tri_child,
+   quad_child,
+   NULL,
+   hex_child,
+   prism_child
+};
+
+
+} // namespace mfem

mesh/nurbs.cpp

@@ -582,9 +582,19 @@ void NURBSPatch::KnotInsert(int dir, const KnotVector &newkv)
    }
 }
 
+void NURBSPatch::KnotInsert(Array<Vector *> &newkv)
+{
+   for (int dir = 0; dir < kv.Size(); dir++)
+   {
+      KnotInsert(dir, *newkv[dir]);
+   }
+}
+
 // Routine from "The NURBS book" - 2nd ed - Piegl and Tiller
 void NURBSPatch::KnotInsert(int dir, const Vector &knot)
 {
+   if (knot.Size() == 0 ) { return; }
+
    if (dir >= kv.Size() || dir < 0)
    {
       mfem_error("NURBSPatch::KnotInsert : Incorrect direction!");
@@ -2982,6 +2992,34 @@ void NURBSExtension::KnotInsert(Array<KnotVector *> &kv)
    }
 }
 
+void NURBSExtension::KnotInsert(Array<Vector *> &kv)
+{
+   Array<int> edges;
+   Array<int> orient;
+
+   Array<Vector *> pkv(Dimension());
+
+   for (int p = 0; p < patches.Size(); p++)
+   {
+      patchTopo->GetElementEdges(p, edges, orient);
+
+      if (Dimension()==2)
+      {
+         pkv[0] = kv[KnotInd(edges[0])];
+         pkv[1] = kv[KnotInd(edges[1])];
+      }
+      else
+      {
+         pkv[0] = kv[KnotInd(edges[0])];
+         pkv[1] = kv[KnotInd(edges[3])];
+         pkv[2] = kv[KnotInd(edges[8])];
+      }
+
+      patches[p]->KnotInsert(pkv);
+   }
+}
+
+
 void NURBSExtension::GetPatchNets(const Vector &coords, int vdim)
 {
    if (Dimension() == 2)