fixed LF integrator

commit messages follow:

gmgsolver first commit

moved GMGsolver class to solvers.hpp and solvers.cpp

simple formatting changes

Modified GMGSolver to accept HypreSmoothers. Added SetSmootherType function to GMGSolver class

(u,curl v) linear integrator added

blk AMG added. GMG example modified to avoid copying the HypreParMatrix

complex operator and pargridfunction added

remove large files

plasma miniapp stix1d & modified ParSesquilinearForm->FormLinearSystem to correctly account for essential bc

.gitignore

@@ -183,3 +183,6 @@ miniapps/nurbs/Example1*
 # Unit test binary and outputs
 tests/unit/output_meshes
 tests/unit/unit_tests
+
+# VPATH builds
+build-*/*

CHANGELOG

@@ -8,22 +8,14 @@
                                http://mfem.org
 
 
-Version 4.0-RC2, Apr 24, 2019
-=============================
+Version 4.0.1 (development)
+===========================
 
-Requirements and Limitations
-----------------------------
-- This is a release candidate for mfem-4.0.
-- Use at your own risk -- not everything will work and the API may change.
-- We are looking for feedback from friendly users.
-- Unlike previous MFEM releases, this version requires a C++11 compiler.
 
-- GPU-related limitations:
-  * Hypre preconditioners are not yet available in GPU mode.
-  * Only constant coefficients are currently supported on GPUs.
-  * Full-assembly (on device), element assembly, and matrix-free bilinear forms
-    are not supported yet. Element batching is currently ignored.
-  * Partial assembly kernels are not implemented yet for simplices.
+Version 4.0, released on May 24, 2019
+=====================================
+
+Unlike previous MFEM releases, this version requires a C++11 compiler.
 
 GPU support
 -----------
@@ -34,7 +26,7 @@ GPU support
   seamlessly with a new lightweight device/host memory manager. The kernels can
   be implemented either in OCCA, or as a simple wrapper around for-loops, which
   can then be dispatched to RAJA and native backends. See the files forall.hpp
-  and mem_manager.hpp in the general/ directory.
+  and mem_manager.hpp in the general/ directory for more details.
 
 - Several of the MFEM example codes (ex1, ex1p, ex6, and ex6p) can now take
   advantage of GPU acceleration with the backend selectable at runtime. Many of
@@ -42,27 +34,45 @@ GPU support
   bilinear forms) have been extended to take advantage of kernel acceleration by
   simply replacing loops with the MFEM_FORALL() macro.
 
-- In addition to pure CUDA, the library currently supports OCCA, RAJA and OpenMP
-  kernels, which could be mixed and matched in different parts of the same
-  application. We plan on adding support for more programming models and devices
-  in the future, without the need for significant modifications in user code.
-  The list of current backends is: "occa-cuda", "raja-cuda", "cuda", "occa-omp",
-  "raja-omp", "omp", "occa-cpu", "raja-cpu", and "cpu".
+- In addition to native CUDA kernels, the library currently supports OCCA, RAJA
+  and OpenMP kernels, which could be mixed and matched in different parts of the
+  same application. We plan on adding support for more programming models and
+  devices in the future, without the need for significant modifications in user
+  code. The list of current backends is: "occa-cuda", "raja-cuda", "cuda",
+  "occa-omp", "raja-omp", "omp", "occa-cpu", "raja-cpu", and "cpu".
+
+- GPU-related limitations:
+  * Hypre preconditioners are not yet available in GPU mode, and in particular
+    hypre must be built in CPU mode.
+  * Only constant coefficients are currently supported on GPUs.
+  * Optimized element assembly, and matrix-free bilinear forms are not
+    implemented yet. Element batching is currently ignored.
+  * In device mode, full assembly is performed on the host (but the matvec
+    action is performed on the device).
+  * Partial assembly kernels are not implemented yet for simplices.
 
 Discretization improvements
 ---------------------------
+
+- Partial assembled finite element operators are now available in the core
+  library, based on the new classes PABilinearFormExtension, ElementRestriction,
+  DofToQuad and GeometricFactors (associated with the classes BilinearForm,
+  FiniteElementSpace, FiniteElement and Mesh, respectively). The kernels for
+  partial assembled Setup/Assembly and Action/Mult are implemented in the
+  BilinearFormIntegrator methods AssemblePA and AddMultPA.
+
 - Added support for a general "low-order refined"-to-"high-order" transfer of
   GridFunction data from a "low-order refined" (LOR) space defined on a refined
   mesh to a "high-order" (HO) finite element space defined on a coarse mesh. See
   the new classes InterpolationGridTransfer and L2ProjectionGridTransfer and the
   new LOR Transfer miniapp: miniapps/tools/lor-transfer.cpp.
 
-- Added support for derefinement of vector (RT + ND) spaces.
-
 - Added element flux, and flux energy computation in class ElasticityIntegrator,
   allowing for the use of Zienkiewicz-Zhu type error estimators with the
   integrator. For an illustration of this addition, see the new Example 21.
 
+- Added support for derefinement of vector (RT + ND) spaces.
+
 - Added a variety of coefficients which are sums or products of existing
   coefficients as well as grid function coefficients which return the
   divergence, gradient, or curl of their GridFunctions.
@@ -73,13 +83,13 @@ Support for wedge elements and meshes with mixed element types
   type PRISM) which have two triangular faces and three quadrilateral faces.
   Several examples of such meshes can be found in the data/ directory.
 
-- Added H1 and L2 finite elements of arbitrary order for Wedge elements.
-
 - Added support for mixed meshes containing triangles and quadrilaterals in 2D
   or tetrahedra, wedges, and hexahedra in 3D. This includes support for uniform
   refinement of such meshes. Several examples of such meshes can be found in the
   data/ directory.
 
+- Added H1 and L2 finite elements of arbitrary order for Wedge elements.
+
 - Added support for reading and writing linear and quadratic meshes containing
   wedge elements in VTK mesh format. Several examples of such meshes can be
   found in the data/ directory.
@@ -100,6 +110,10 @@ Other meshing improvements
   This guarantees that the shape regularity of the elements will be preserved
   under refinement.
 
+- The TMOP mesh optimization algorithms were extended to support user-defined
+  space-dependent limiting terms. Improved the TMOP objective functions by more
+  accurate normalization of the different terms.
+
 - Added support for parallel communication groups on non-conforming meshes.
 
 - Improved parallel partitioning of non-conforming meshes. If the coarse mesh
@@ -113,10 +127,6 @@ Other meshing improvements
 - Added support for reading linear and quadratic 2D quadrilateral and triangular
   Cubit meshes.
 
-- The TMOP mesh optimization algorithms were extended to support user-defined
-  space-dependent limiting terms. Improved the TMOP objective functions by more
-  accurate normalization of the different terms.
-
 New and updated examples and miniapps
 -------------------------------------
 - Added a new meshing miniapp, Toroid, which can produce a variety of torus
@@ -144,17 +154,17 @@ New and improved solvers and preconditioners
 
 Miscellaneous
 -------------
-- In SparseMatrix added the option to perform MultTranspose() by matvec with
-  computed and stored transpose matrix. This is required for deterministic
-  results when using devices such as CUDA and OpenMP.
-
-- Added unit tests based on the Catch++ library.
+- Added unit tests based on the Catch++ library in the test/ directory.
 
 - Renamed the option MFEM_USE_OPENMP to MFEM_USE_LEGACY_OPENMP. This legacy
   option is deprecated and planned for removal in a future release. The original
   option name, MFEM_USE_OPENMP, is now used to enable the new OpenMP backends in
   the new kernels.
 
+- In SparseMatrix added the option to perform MultTranspose() by matvec with
+  computed and stored transpose matrix. This is required for deterministic
+  results when using devices such as CUDA and OpenMP.
+
 - Altered the way FGMRES counts its iterations so that it matches GMRES.
 
 - Various other simplifications, extensions, and bugfixes in the code.

CMakeLists.txt

@@ -50,7 +50,7 @@ project(mfem NONE)
 # Current version of MFEM, see also `makefile`.
 #   mfem_VERSION = (string)
 #   MFEM_VERSION = (int)   [automatically derived from mfem_VERSION]
-set(${PROJECT_NAME}_VERSION 3.4.1)
+set(${PROJECT_NAME}_VERSION 4.0.1)
 
 # Prohibit in-source build
 if (${PROJECT_SOURCE_DIR} STREQUAL ${PROJECT_BINARY_DIR})
@@ -286,7 +286,6 @@ if (MFEM_USE_CUDA)
       set(CUDA_CCBIN_COMPILER ${CMAKE_CXX_COMPILER})
    endif()
    string(APPEND CMAKE_CUDA_FLAGS " -ccbin ${CUDA_CCBIN_COMPILER}")
-   set(MFEM_USE_MM YES CACHE BOOL "Enable MFEM's memory manager" FORCE)
 endif()
 
 # OCCA

INSTALL

@@ -404,18 +404,12 @@ MFEM_USE_PUMI = YES/NO
    models and effectively supports automated adaptive analysis. PUMI enables
    support for parallel unstructured mesh modifications in MFEM.
 
-MFEM_USE_MM = YES/NO
-   Enables support for the MFEM's memory manager (MM), which is required to
-   support devices with different memory spaces. This option is required when
-   CUDA support is enabled, i.e. when MFEM_USE_CUDA=YES.
-
 MFEM_USE_CUDA = YES/NO
    Enables support for CUDA devices in MFEM. CUDA is a parallel computing
    platform and programming model for general computing on graphical processing
-   units (GPUs). This option requires MFEM_USE_MM. The variable CUDA_ARCH is
-   used to specify the CUDA compute capability used during compilation (by
-   default, CUDA_ARCH=sm_60). When enabled, this option uses the CUDA_* build
-   options, see below.
+   units (GPUs). The variable CUDA_ARCH is used to specify the CUDA compute
+   capability used during compilation (by default, CUDA_ARCH=sm_60). When
+   enabled, this option uses the CUDA_* build options, see below.
 
 MFEM_USE_RAJA = YES/NO
    Enable support for the RAJA performance portability layer in MFEM. RAJA
@@ -696,7 +690,6 @@ MFEM_USE_PUMI
 MFEM_USE_CUDA
 MFEM_USE_OCCA
 MFEM_USE_RAJA
-MFEM_USE_MM
 
 The following options are CMake specific:
 

config/cmake/MFEMConfig.cmake.in

@@ -41,7 +41,6 @@ set(MFEM_USE_MPFR @MFEM_USE_MPFR@)
 set(MFEM_USE_SIDRE @MFEM_USE_SIDRE@)
 set(MFEM_USE_CONDUIT @MFEM_USE_CONDUIT@)
 set(MFEM_USE_PUMI @MFEM_USE_PUMI@)
-set(MFEM_USE_MM @MFEM_USE_MM@)
 set(MFEM_USE_CUDA @MFEM_USE_CUDA@)
 set(MFEM_USE_OCCA @MFEM_USE_OCCA@)
 set(MFEM_USE_RAJA @MFEM_USE_RAJA@)

config/cmake/config.hpp.in

@@ -120,9 +120,6 @@
 // Enable MFEM functionality based on the OCCA library
 #cmakedefine MFEM_USE_OCCA
 
-// Enable MFEM's internal Memory Manager (needed e.g. for MFEM_USE_CUDA)
-#cmakedefine MFEM_USE_MM
-
 // Which library functions to use in class StopWatch for measuring time.
 // For a list of the available options, see INSTALL.
 // If not defined, an option is selected automatically.

config/cmake/modules/MfemCmakeUtilities.cmake

@@ -720,8 +720,7 @@ function(mfem_export_mk_files)
       MFEM_USE_MEMALLOC MFEM_USE_SUNDIALS MFEM_USE_MESQUITE MFEM_USE_SUITESPARSE
       MFEM_USE_SUPERLU MFEM_USE_STRUMPACK MFEM_USE_GECKO MFEM_USE_GNUTLS
       MFEM_USE_NETCDF MFEM_USE_PETSC MFEM_USE_MPFR MFEM_USE_SIDRE
-      MFEM_USE_CONDUIT MFEM_USE_PUMI MFEM_USE_MM MFEM_USE_CUDA MFEM_USE_OCCA
-      MFEM_USE_RAJA)
+      MFEM_USE_CONDUIT MFEM_USE_PUMI MFEM_USE_CUDA MFEM_USE_OCCA MFEM_USE_RAJA)
   foreach(var ${CONFIG_MK_BOOL_VARS})
     if (${var})
       set(${var} YES)

config/config.hpp

@@ -56,9 +56,4 @@
 #endif
 #endif // MFEM_USE_MPI not defined
 
-// CUDA requires the memory manager
-#if defined(MFEM_USE_CUDA) && !defined(MFEM_USE_MM)
-#error Building with CUDA (MFEM_USE_CUDA=YES) requires MFEM_USE_MM=YES
-#endif
-
 #endif // MFEM_CONFIG_HPP

config/config.hpp.in

@@ -131,9 +131,6 @@
 // Enable functionality based on the OCCA library.
 // #define MFEM_USE_OCCA
 
-// Enable MFEM's internal Memory Manager (needed e.g. for MFEM_USE_CUDA)
-// #define MFEM_USE_MM
-
 // Version of HYPRE used for building MFEM.
 // #define MFEM_HYPRE_VERSION @MFEM_HYPRE_VERSION@
 

config/config.mk.in

@@ -44,7 +44,6 @@ MFEM_USE_PUMI          = @MFEM_USE_PUMI@
 MFEM_USE_CUDA          = @MFEM_USE_CUDA@
 MFEM_USE_RAJA          = @MFEM_USE_RAJA@
 MFEM_USE_OCCA          = @MFEM_USE_OCCA@
-MFEM_USE_MM            = @MFEM_USE_MM@
 
 # Compiler, compile options, and link options
 MFEM_CXX       = @MFEM_CXX@

config/defaults.cmake

@@ -42,7 +42,6 @@ option(MFEM_USE_MPFR "Enable MPFR usage." OFF)
 option(MFEM_USE_SIDRE "Enable Axom/Sidre usage" OFF)
 option(MFEM_USE_CONDUIT "Enable Conduit usage" OFF)
 option(MFEM_USE_PUMI "Enable PUMI" OFF)
-option(MFEM_USE_MM "Enable MFEM's memory manager" OFF)
 option(MFEM_USE_CUDA "Enable CUDA" OFF)
 option(MFEM_USE_OCCA "Enable OCCA" OFF)
 option(MFEM_USE_RAJA "Enable RAJA" OFF)

config/defaults.mk

@@ -124,7 +124,6 @@ MFEM_USE_PUMI          = NO
 MFEM_USE_CUDA          = NO
 MFEM_USE_RAJA          = NO
 MFEM_USE_OCCA          = NO
-MFEM_USE_MM            = NO
 
 # Compile and link options for zlib.
 ZLIB_DIR =

config/makefile

@@ -36,6 +36,7 @@ CONFIG_MK = config.mk
 all: header config-mk
 
 MPI = $(MFEM_USE_MPI:NO=)
+GHV_CXX ?= $(MFEM_CXX)
 GHV = get_hypre_version
 GHV_FLAGS = $(subst @MFEM_DIR@,$(if $(MFEM_DIR),$(MFEM_DIR),..),$(HYPRE_OPT))
 SMX = $(if $(MFEM_USE_PUMI:NO=),MFEM_USE_SIMMETRIX)
@@ -44,7 +45,7 @@ SMX_FILE = $(subst @MFEM_DIR@,$(if $(MFEM_DIR),$(MFEM_DIR),..),$(SMX_PATH))
 
 $(GHV): $(SRC)$(GHV).cpp
 	$(call mfem-info, Determining HYPRE version ...)
-	$(MFEM_CXX) ${GHV_FLAGS} $(SRC)$(GHV).cpp -o $(GHV)
+	$(GHV_CXX) ${GHV_FLAGS} $(SRC)$(GHV).cpp -o $(GHV)
 $(GHV).out: $(GHV)
 	./$(GHV) > $(GHV).out
 .INTERMEDIATE: $(GHV) $(GHV).out

config/sample-runs.sh

@@ -276,8 +276,7 @@ case "$1" in
       ;;
    -dev)
        device_runs="yes"
-       mfem_config+=" MFEM_USE_CUDA=YES MFEM_USE_MM=YES \
-MFEM_USE_OCCA=YES MFEM_USE_RAJA=YES MFEM_USE_OPENMP=YES"
+       mfem_config+=" MFEM_USE_CUDA=YES MFEM_USE_OCCA=YES MFEM_USE_RAJA=YES MFEM_USE_OPENMP=YES"
       ;;
    -v)
       valgrind="yes"

data/one-hex.mesh

@@ -0,0 +1,42 @@
+MFEM mesh v1.0
+
+#
+# MFEM Geometry Types (see mesh/geom.hpp):
+#
+# POINT       = 0
+# SEGMENT     = 1
+# TRIANGLE    = 2
+# SQUARE      = 3
+# TETRAHEDRON = 4
+# CUBE        = 5
+#
+
+dimension
+3
+
+elements
+1
+1 5 0 1 2 3 4 5 6 7
+
+boundary
+6
+
+1 3 3 2 1 0
+1 3 0 1 5 4
+1 3 2 3 7 6
+1 3 3 0 4 7
+1 3 4 5 6 7
+1 3 1 2 6 5
+
+
+vertices
+8
+3
+0 0 0
+1 0 0
+1 1 0
+0 1 0
+0 0 1
+1 0 1
+1 1 1
+0 1 1

doc/CodeDocumentation.conf.in

@@ -38,7 +38,7 @@ PROJECT_NAME           = "MFEM"
 # could be handy for archiving the generated documentation or if some version
 # control system is used.
 
-PROJECT_NUMBER         = v3.4.1
+PROJECT_NUMBER         = v4.0.1
 
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a

doc/CodeDocumentation.dox

@@ -37,7 +37,9 @@ namespace mfem {
  *
  * <H3>Main GPU classes</H3>
  * - Device
+ * - Memory
  * - MemoryManager
+ * - MFEM_FORALL macro in forall.hpp
  *
  * <H3>Example codes</H3>
  * - <a class="el" href="examples_2ex1_8cpp_source.html">Example 1</a>: nodal H1 FEM for the Laplace problem

examples/AddSchwarz.cpp

@@ -0,0 +1,362 @@
+//                                MFEM Example 1
+//
+// Compile with: make AddScwarz
+//
+// Example run: ./AddSchwarz -sr 4 -o 2 -d 2
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+class AddSchwarzSmoother : public Solver {
+private:
+   /// The linear system matrix
+   SparseMatrix * A;
+   Array<SparseMatrix *> Pid; 
+   Array<SparseMatrix *> A_local;
+   Array<UMFPackSolver *> invA_local;
+   int nrvert;
+   Array<int>vert_dofs;
+
+public:
+   AddSchwarzSmoother(SparseMatrix *A_, FiniteElementSpace *fespace);
+
+   virtual void SetOperator(const Operator &op) {}
+   virtual void Mult(const Vector &r, Vector &z) const;
+   virtual ~AddSchwarzSmoother() {}
+};
+// constructor
+AddSchwarzSmoother::AddSchwarzSmoother(SparseMatrix * A_, FiniteElementSpace *fespace)
+   : Solver(A_->Height(), A_->Width()), A(A_) 
+{
+
+   Array<int> ess_tdof_list;
+   if (fespace->GetMesh()->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(fespace->GetMesh()->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   nrvert = fespace->GetMesh()->GetNV();      
+   int nredge = fespace->GetMesh()->GetNEdges();
+   int nrface = fespace->GetMesh()->GetNFaces();
+   int nrelem = fespace->GetMesh()->GetNE();
+   Array<int>edge_vert;
+   Array<int>edge_int_dofs;
+   Array<int>face_vert;
+   Array<int>face_int_dofs;
+   Array<int>elem_vert;
+   Array<int>elem_int_dofs;
+
+   vert_dofs.SetSize(0);
+   for (int i=0; i<nrvert; i++)
+   {
+      int j = ess_tdof_list.FindSorted(i);
+      if (j == -1 ) 
+         vert_dofs.Append(i);
+   }
+
+   // Build a sparse matrix out of this map to extract the patch submatrix
+   Pid.SetSize(nrvert);
+   Array<int> dofoffset(nrvert);
+   dofoffset = 0;
+   int height = fespace->GetVSize();
+   for (int i=0; i<nrvert; i++)
+   {
+      Pid[i] = new SparseMatrix(height);
+      // skip if its a dirichlet vertex
+      // int m = vert_dofs[i];
+      Pid[i]->Set(i,dofoffset[i],1.0); // Fill in the vertex dof (1 column for each vertex)
+      dofoffset[i]++;
+   }
+
+   // Fill the sparse matrix with the edge dof indices (1 column for each dof)
+   for (int i=0; i< nredge; i++ )
+   {
+      fespace->GetMesh()->GetEdgeVertices(i,edge_vert);
+      int nv = edge_vert.Size();
+      fespace->GetEdgeInteriorDofs(i,edge_int_dofs);
+      int ne = edge_int_dofs.Size();
+      for (int j=0; j<nv ; j++)
+      {
+         int k = edge_vert[j];
+         for (int l=0; l < ne; l++)
+         {
+            int m = edge_int_dofs[l];
+            Pid[k]->Set(m,dofoffset[k],1.0);
+            dofoffset[k]++;
+         }
+      }
+   }
+   // Fill the sparse matrix with the face dof indices (1 column for each dof)
+   for (int i=0; i< nrface; i++ )
+   {
+      fespace->GetMesh()->GetEdgeVertices(i,face_vert);
+      int nv = face_vert.Size();
+      fespace->GetFaceInteriorDofs(i,face_int_dofs);
+      int nf = face_int_dofs.Size();
+      for (int j=0; j<nv ; j++)
+      {
+         int k = face_vert[j];
+         for (int l=0; l < nf; l++)
+         {
+            int m = face_int_dofs[l];
+            Pid[k]->Set(m,dofoffset[k],1.0);
+            dofoffset[k]++;
+         }
+      }
+   }
+   // Fill the sparse matrix with the element (middle) dof indices (1 column for each dof)
+   for (int i=0; i< nrelem; i++ )
+   {
+      fespace->GetMesh()->GetElementVertices(i,elem_vert);
+      int nv = elem_vert.Size();
+      fespace->GetElementInteriorDofs(i,elem_int_dofs);
+      int nel = elem_int_dofs.Size();
+      for (int j=0; j<nv ; j++)
+      {
+         int k = elem_vert[j];
+         for (int l=0; l < nel; l++)
+         {
+            int m = elem_int_dofs[l];
+            Pid[k]->Set(m,dofoffset[k],1.0);
+            dofoffset[k]++;
+         }
+      }
+   }
+
+   A_local.SetSize(nrvert);
+   invA_local.SetSize(nrvert);
+   for (int i=0; i< nrvert; i++ )
+   {
+      Pid[i]->SetWidth(dofoffset[i]);
+      Pid[i]->Finalize();
+
+      // construct the local problems. Factor the patch matrices
+      A_local[i] = RAP(*Pid[i],*A,*Pid[i]);
+      invA_local[i] = new UMFPackSolver;
+      invA_local[i]->Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+      invA_local[i]->SetOperator(*A_local[i]);   
+   }
+}
+
+
+void AddSchwarzSmoother::Mult(const Vector &r, Vector &z) const
+{
+   // Apply the smoother patch on the restriction of the residual
+   Array<Vector> res_local(nrvert);
+   Array<Vector> sol_local(nrvert);
+   Array<Vector> zaux(nrvert);
+   z = 0.0;
+   for (int i=0; i< nrvert; i++)
+   // for (int ii=0; ii< vert_dofs.Size(); ii++)  // loop through only the patches associate with a non-dirichlet vertex
+   {
+      // int i = vert_dofs[ii];
+      res_local[i].SetSize(Pid[i]->NumCols()); 
+      sol_local[i].SetSize(Pid[i]->NumCols()); 
+      Pid[i]->MultTranspose(r,res_local[i]);
+      invA_local[i]->Mult(res_local[i],sol_local[i]);
+      zaux[i].SetSize(r.Size()); zaux[i]=0.0; 
+      Pid[i]->Mult(sol_local[i],zaux[i]); zaux[i] *= 0.5;
+      z += zaux[i];
+   }
+}
+
+void get_solution(const Vector &x, double & u, double & d2u);
+double u_exact(const Vector &x);
+double f_exact(const Vector &x);
+int dim;
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options.
+   const char *mesh_file = "../data/one-hex.mesh";
+   int order = 1;
+   int sdim = 1;
+   bool static_cond = false;
+   bool pa = false;
+   const char *device_config = "cpu";
+   bool visualization = true;
+   int ref_levels = 1;
+
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&sdim, "-d", "--dimension", "Dimension");
+   args.AddOption(&ref_levels, "-sr", "--serial-refinements", "Number of mesh refinements");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   // 2. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   device.Print();
+
+   // 3. Read the mesh from the given mesh file. 
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+
+   Mesh * mesh; 
+   // Define a simple square mesh
+   if (sdim == 2)
+   {
+      mesh = new Mesh(1, 1, Element::QUADRILATERAL, true,1.0, 1.0,false);
+   }
+   else
+   {
+      mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true,1.0, 1.0,1.0, false);
+   }
+   dim = mesh->Dimension();
+
+   // 4. Refine the mesh 
+   {
+      for (int l = 0; l < ref_levels; l++)
+      {
+         mesh->UniformRefinement();
+      }
+   }
+
+   // 5. Define a finite element space on the mesh. 
+   FiniteElementCollection *fec = new H1_FECollection(order, dim);
+   FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
+
+   // 6. Determine the list of true (i.e. conforming) essential boundary dofs.
+   Array<int> ess_tdof_list;
+   if (mesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(mesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   // 7. Set up the linear form b(.) 
+   LinearForm *b = new LinearForm(fespace);
+   ConstantCoefficient one(1.0);
+   FunctionCoefficient f(f_exact);
+   b->AddDomainIntegrator(new DomainLFIntegrator(f));
+   b->Assemble();
+
+   // 8. Define the solution vector x as a finite element grid function
+   GridFunction x(fespace);
+   x = 0.0;
+   FunctionCoefficient u_ex(u_exact);
+   x.ProjectCoefficient(u_ex);
+
+   // 9. Set up the bilinear form a(.,.) 
+   BilinearForm *a = new BilinearForm(fespace);
+   a->AddDomainIntegrator(new DiffusionIntegrator(one));
+   a->Assemble();
+
+   SparseMatrix A;
+   Vector B, X;
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+   cout << "Size of linear system: " << A.Height() << endl;
+
+
+   AddSchwarzSmoother *prec = new AddSchwarzSmoother(&A,fespace);
+   // DSmoother *prec = new DSmoother(A);
+   // GSSmoother *prec = new GSSmoother(A);
+
+   int maxit(100);
+   double rtol(1.e-6);
+   double atol(0.0);
+   CGSolver solver;
+   solver.SetAbsTol(atol);
+   solver.SetRelTol(rtol);
+   solver.SetMaxIter(maxit);
+   solver.SetOperator(A);
+   solver.SetPreconditioner(*prec);
+   solver.SetPrintLevel(1);
+   solver.Mult(B,X);
+
+   // 12. Recover the solution as a finite element grid function.
+   a->RecoverFEMSolution(X, *b, x);
+
+
+   GridFunction ugf(fespace);
+   ugf.ProjectCoefficient(u_ex);
+
+
+   int order_quad = max(2, 2*order+1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i=0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double L2error = x.ComputeL2Error(u_ex);
+      
+   cout << " || u_h - u ||_{L^2} = " << L2error <<  endl;
+
+
+
+   // 14. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      // sol_sock << "mesh\n" << *mesh << flush;
+      sol_sock << "solution\n" << *mesh << x << flush;
+   }
+
+//    // 15. Free the used memory.
+   delete a;
+   delete b;
+   delete fespace;
+   delete mesh;
+   return 0;
+}
+
+void get_solution(const Vector &x, double & u, double & d2u)
+{
+
+   if (dim == 2)
+   {
+      u = x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]);
+      d2u = -2.0* ( x[1]*(1.0 - x[1]) + x[0]*(1.0 - x[0])); 
+   }
+   else
+   {
+      u = x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+      d2u = -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[1]) * x[1] 
+         -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[2]) * x[2] 
+         -2.0*(-1.0 + x[1]) * x[1] * (-1.0 + x[2]) * x[2];
+   }
+}
+
+double u_exact(const Vector &x)
+{
+   double u, d2u;
+   get_solution(x, u, d2u);
+   return u;
+}
+
+double f_exact(const Vector &x)
+{
+   double u, d2u;
+   get_solution(x, u, d2u);
+   return -d2u;
+}

examples/ex1.cpp

@@ -62,7 +62,7 @@ int main(int argc, char *argv[])
    int order = 1;
    bool static_cond = false;
    bool pa = false;
-   const char *device = "cpu";
+   const char *device_config = "cpu";
    bool visualization = true;
 
    OptionsParser args(argc, argv);
@@ -75,7 +75,7 @@ int main(int argc, char *argv[])
                   "--no-static-condensation", "Enable static condensation.");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
                   "--no-partial-assembly", "Enable Partial Assembly.");
-   args.AddOption(&device, "-d", "--device",
+   args.AddOption(&device_config, "-d", "--device",
                   "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
@@ -88,13 +88,18 @@ int main(int argc, char *argv[])
    }
    args.PrintOptions(cout);
 
-   // 2. Read the mesh from the given mesh file. We can handle triangular,
+   // 2. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   device.Print();
+
+   // 3. Read the mesh from the given mesh file. We can handle triangular,
    //    quadrilateral, tetrahedral, hexahedral, surface and volume meshes with
    //    the same code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
 
-   // 3. Refine the mesh to increase the resolution. In this example we do
+   // 4. Refine the mesh to increase the resolution. In this example we do
    //    'ref_levels' of uniform refinement. We choose 'ref_levels' to be the
    //    largest number that gives a final mesh with no more than 50,000
    //    elements.
@@ -107,7 +112,7 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 4. Define a finite element space on the mesh. Here we use continuous
+   // 5. Define a finite element space on the mesh. Here we use continuous
    //    Lagrange finite elements of the specified order. If order < 1, we
    //    instead use an isoparametric/isogeometric space.
    FiniteElementCollection *fec;
@@ -128,7 +133,7 @@ int main(int argc, char *argv[])
    cout << "Number of finite element unknowns: "
         << fespace->GetTrueVSize() << endl;
 
-   // 5. Determine the list of true (i.e. conforming) essential boundary dofs.
+   // 6. Determine the list of true (i.e. conforming) essential boundary dofs.
    //    In this example, the boundary conditions are defined by marking all
    //    the boundary attributes from the mesh as essential (Dirichlet) and
    //    converting them to a list of true dofs.
@@ -140,7 +145,7 @@ int main(int argc, char *argv[])
       fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
    }
 
-   // 6. Set up the linear form b(.) which corresponds to the right-hand side of
+   // 7. Set up the linear form b(.) which corresponds to the right-hand side of
    //    the FEM linear system, which in this case is (1,phi_i) where phi_i are
    //    the basis functions in the finite element fespace.
    LinearForm *b = new LinearForm(fespace);
@@ -148,12 +153,6 @@ int main(int argc, char *argv[])
    b->AddDomainIntegrator(new DomainLFIntegrator(one));
    b->Assemble();
 
-   // 7. Set device config parameters from the command line options and switch
-   //    to working on the device.
-   Device::Configure(device);
-   Device::Print();
-   Device::Enable();
-
    // 8. Define the solution vector x as a finite element grid function
    //    corresponding to fespace. Initialize x with initial guess of zero,
    //    which satisfies the boundary conditions.
@@ -203,10 +202,7 @@ int main(int argc, char *argv[])
    // 12. Recover the solution as a finite element grid function.
    a->RecoverFEMSolution(X, *b, x);
 
-   // 13. Switch back to the host.
-   Device::Disable();
-
-   // 14. Save the refined mesh and the solution. This output can be viewed later
+   // 13. Save the refined mesh and the solution. This output can be viewed later
    //     using GLVis: "glvis -m refined.mesh -g sol.gf".
    ofstream mesh_ofs("refined.mesh");
    mesh_ofs.precision(8);
@@ -215,7 +211,7 @@ int main(int argc, char *argv[])
    sol_ofs.precision(8);
    x.Save(sol_ofs);
 
-   // 15. Send the solution by socket to a GLVis server.
+   // 14. Send the solution by socket to a GLVis server.
    if (visualization)
    {
       char vishost[] = "localhost";
@@ -225,7 +221,7 @@ int main(int argc, char *argv[])
       sol_sock << "solution\n" << *mesh << x << flush;
    }
 
-   // 16. Free the used memory.
+   // 15. Free the used memory.
    delete a;
    delete b;
    delete fespace;

examples/ex19.cpp

@@ -144,7 +144,7 @@ void InitialDeformation(const Vector &x, Vector &y);
 int main(int argc, char *argv[])
 {
    // 1. Parse command-line options
-   const char *mesh_file = "../data/beam-hex.mesh";
+   const char *mesh_file = "../data/beam-tet.mesh";
    int ref_levels = 0;
    int order = 2;
    bool visualization = true;

examples/ex19p.cpp

@@ -150,7 +150,7 @@ int main(int argc, char *argv[])
    MPI_Comm_rank(MPI_COMM_WORLD, &myid);
 
    // 2. Parse command-line options
-   const char *mesh_file = "../data/beam-hex.mesh";
+   const char *mesh_file = "../data/beam-tet.mesh";
    int ser_ref_levels = 0;
    int par_ref_levels = 0;
    int order = 2;

examples/ex1p.cpp

@@ -65,7 +65,7 @@ int main(int argc, char *argv[])
    int order = 1;
    bool static_cond = false;
    bool pa = false;
-   const char *device = "cpu";
+   const char *device_config = "cpu";
    bool visualization = true;
 
    OptionsParser args(argc, argv);
@@ -78,7 +78,7 @@ int main(int argc, char *argv[])
                   "--no-static-condensation", "Enable static condensation.");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
                   "--no-partial-assembly", "Enable Partial Assembly.");
-   args.AddOption(&device, "-d", "--device",
+   args.AddOption(&device_config, "-d", "--device",
                   "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
@@ -98,13 +98,18 @@ int main(int argc, char *argv[])
       args.PrintOptions(cout);
    }
 
-   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   // 3. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   if (myid == 0) { device.Print(); }
+
+   // 4. Read the (serial) mesh from the given mesh file on all processors.  We
    //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
    //    and volume meshes with the same code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
 
-   // 4. Refine the serial mesh on all processors to increase the resolution. In
+   // 5. Refine the serial mesh on all processors to increase the resolution. In
    //    this example we do 'ref_levels' of uniform refinement. We choose
    //    'ref_levels' to be the largest number that gives a final mesh with no
    //    more than 10,000 elements.
@@ -117,7 +122,7 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   // 6. Define a parallel mesh by a partitioning of the serial mesh. Refine
    //    this mesh further in parallel to increase the resolution. Once the
    //    parallel mesh is defined, the serial mesh can be deleted.
    ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
@@ -130,7 +135,7 @@ int main(int argc, char *argv[])
       }
    }
 
-   // 6. Define a parallel finite element space on the parallel mesh. Here we
+   // 7. Define a parallel finite element space on the parallel mesh. Here we
    //    use continuous Lagrange finite elements of the specified order. If
    //    order < 1, we instead use an isoparametric/isogeometric space.
    FiniteElementCollection *fec;
@@ -157,7 +162,7 @@ int main(int argc, char *argv[])
       cout << "Number of finite element unknowns: " << size << endl;
    }
 
-   // 7. Determine the list of true (i.e. parallel conforming) essential
+   // 8. Determine the list of true (i.e. parallel conforming) essential
    //    boundary dofs. In this example, the boundary conditions are defined
    //    by marking all the boundary attributes from the mesh as essential
    //    (Dirichlet) and converting them to a list of true dofs.
@@ -169,7 +174,7 @@ int main(int argc, char *argv[])
       fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
    }
 
-   // 8. Set up the parallel linear form b(.) which corresponds to the
+   // 9. Set up the parallel linear form b(.) which corresponds to the
    //    right-hand side of the FEM linear system, which in this case is
    //    (1,phi_i) where phi_i are the basis functions in fespace.
    ParLinearForm *b = new ParLinearForm(fespace);
@@ -177,12 +182,6 @@ int main(int argc, char *argv[])
    b->AddDomainIntegrator(new DomainLFIntegrator(one));
    b->Assemble();
 
-   // 9. Set device config parameters from the command line options and switch
-   //    to working on the device.
-   Device::Configure(device);
-   if (myid == 0) { Device::Print(); }
-   Device::Enable();
-
    // 10. Define the solution vector x as a parallel finite element grid function
    //     corresponding to fespace. Initialize x with initial guess of zero,
    //     which satisfies the boundary conditions.
@@ -225,10 +224,7 @@ int main(int argc, char *argv[])
    //     local finite element solution on each processor.
    a->RecoverFEMSolution(X, *b, x);
 
-   // 15. Switch back to the host.
-   Device::Disable();
-
-   // 16. Save the refined mesh and the solution in parallel. This output can
+   // 15. Save the refined mesh and the solution in parallel. This output can
    //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
    {
       ostringstream mesh_name, sol_name;
@@ -244,7 +240,7 @@ int main(int argc, char *argv[])
       x.Save(sol_ofs);
    }
 
-   // 17. Send the solution by socket to a GLVis server.
+   // 16. Send the solution by socket to a GLVis server.
    if (visualization)
    {
       char vishost[] = "localhost";
@@ -255,7 +251,7 @@ int main(int argc, char *argv[])
       sol_sock << "solution\n" << *pmesh << x << flush;
    }
 
-   // 18. Free the used memory.
+   // 17. Free the used memory.
    delete a;
    delete b;
    delete fespace;

examples/ex6.cpp

@@ -49,7 +49,7 @@ int main(int argc, char *argv[])
    const char *mesh_file = "../data/star.mesh";
    int order = 1;
    bool pa = false;
-   const char *device = "cpu";
+   const char *device_config = "cpu";
    bool visualization = true;
 
    OptionsParser args(argc, argv);
@@ -59,7 +59,7 @@ int main(int argc, char *argv[])
                   "Finite element order (polynomial degree).");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
                   "--no-partial-assembly", "Enable Partial Assembly.");
-   args.AddOption(&device, "-d", "--device",
+   args.AddOption(&device_config, "-d", "--device",
                   "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
@@ -72,14 +72,19 @@ int main(int argc, char *argv[])
    }
    args.PrintOptions(cout);
 
-   // 2. Read the mesh from the given mesh file. We can handle triangular,
+   // 2. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   device.Print();
+
+   // 3. Read the mesh from the given mesh file. We can handle triangular,
    //    quadrilateral, tetrahedral, hexahedral, surface and volume meshes with
    //    the same code.
    Mesh mesh(mesh_file, 1, 1);
    int dim = mesh.Dimension();
    int sdim = mesh.SpaceDimension();
 
-   // 3. Since a NURBS mesh can currently only be refined uniformly, we need to
+   // 4. Since a NURBS mesh can currently only be refined uniformly, we need to
    //    convert it to a piecewise-polynomial curved mesh. First we refine the
    //    NURBS mesh a bit more and then project the curvature to quadratic Nodes.
    if (mesh.NURBSext)
@@ -91,15 +96,11 @@ int main(int argc, char *argv[])
       mesh.SetCurvature(2);
    }
 
-   // 4. Define a finite element space on the mesh. The polynomial order is
+   // 5. Define a finite element space on the mesh. The polynomial order is
    //    one (linear) by default, but this can be changed on the command line.
    H1_FECollection fec(order, dim);
    FiniteElementSpace fespace(&mesh, &fec);
 
-   // 5. Set device config parameters from the command line options.
-   Device::Configure(device);
-   Device::Print();
-
    // 6. As in Example 1, we set up bilinear and linear forms corresponding to
    //    the Laplace problem -\Delta u = 1. We don't assemble the discrete
    //    problem yet, this will be done in the main loop.
@@ -168,8 +169,7 @@ int main(int argc, char *argv[])
       x.ProjectBdrCoefficient(zero, ess_bdr);
       fespace.GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
 
-      // 15. Switch to the device and assemble the stiffness matrix.
-      Device::Enable();
+      // 15. Assemble the stiffness matrix.
       a.Assemble();
 
       // 16. Create the linear system: eliminate boundary conditions, constrain
@@ -204,7 +204,6 @@ int main(int argc, char *argv[])
       // 18. After solving the linear system, reconstruct the solution as a
       //     finite element GridFunction. Constrained nodes are interpolated
       //     from true DOFs (it may therefore happen that x.Size() >= X.Size()).
-      Device::Disable();
       a.RecoverFEMSolution(X, b, x);
 
       // 19. Send solution by socket to the GLVis server.

examples/ex6p.cpp

@@ -55,7 +55,7 @@ int main(int argc, char *argv[])
    const char *mesh_file = "../data/star.mesh";
    int order = 1;
    bool pa = false;
-   const char *device = "cpu";
+   const char *device_config = "cpu";
    bool visualization = true;
 
    OptionsParser args(argc, argv);
@@ -65,7 +65,7 @@ int main(int argc, char *argv[])
                   "Finite element order (polynomial degree).");
    args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
                   "--no-partial-assembly", "Enable Partial Assembly.");
-   args.AddOption(&device, "-d", "--device",
+   args.AddOption(&device_config, "-d", "--device",
                   "Device configuration string, see Device::Configure().");
    args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
                   "--no-visualization",
@@ -85,14 +85,19 @@ int main(int argc, char *argv[])
       args.PrintOptions(cout);
    }
 
-   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   // 3. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   if (myid == 0) { device.Print(); }
+
+   // 4. Read the (serial) mesh from the given mesh file on all processors.  We
    //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
    //    and volume meshes with the same code.
    Mesh *mesh = new Mesh(mesh_file, 1, 1);
    int dim = mesh->Dimension();
    int sdim = mesh->SpaceDimension();
 
-   // 4. Refine the serial mesh on all processors to increase the resolution.
+   // 5. Refine the serial mesh on all processors to increase the resolution.
    //    Also project a NURBS mesh to a piecewise-quadratic curved mesh. Make
    //    sure that the mesh is non-conforming.
    if (mesh->NURBSext)
@@ -102,7 +107,7 @@ int main(int argc, char *argv[])
    }
    mesh->EnsureNCMesh();
 
-   // 5. Define a parallel mesh by partitioning the serial mesh.
+   // 6. Define a parallel mesh by partitioning the serial mesh.
    //    Once the parallel mesh is defined, the serial mesh can be deleted.
    ParMesh pmesh(MPI_COMM_WORLD, *mesh);
    delete mesh;
@@ -112,15 +117,11 @@ int main(int argc, char *argv[])
    Array<int> ess_bdr(pmesh.bdr_attributes.Max());
    ess_bdr = 1;
 
-   // 6. Define a finite element space on the mesh. The polynomial order is
+   // 7. Define a finite element space on the mesh. The polynomial order is
    //    one (linear) by default, but this can be changed on the command line.
    H1_FECollection fec(order, dim);
    ParFiniteElementSpace fespace(&pmesh, &fec);
 
-   // 7. Set device config parameters from the command line options.
-   Device::Configure(device);
-   if (myid == 0) { Device::Print(); }
-
    // 8. As in Example 1p, we set up bilinear and linear forms corresponding to
    //    the Laplace problem -\Delta u = 1. We don't assemble the discrete
    //    problem yet, this will be done in the main loop.
@@ -200,11 +201,10 @@ int main(int argc, char *argv[])
       fespace.GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
       b.Assemble();
 
-      // 15. Switch to the device and assemble the stiffness matrix. Note that
-      //     MFEM doesn't care at this point that the mesh is nonconforming and
-      //     parallel.  The FE space is considered 'cut' along hanging
-      //     edges/faces, and also across processor boundaries.
-      Device::Enable();
+      // 15. Assemble the stiffness matrix. Note that MFEM doesn't care at this
+      //     point that the mesh is nonconforming and parallel.  The FE space is
+      //     considered 'cut' along hanging edges/faces, and also across
+      //     processor boundaries.
       a.Assemble();
 
       // 16. Create the parallel linear system: eliminate boundary conditions.
@@ -232,7 +232,6 @@ int main(int argc, char *argv[])
       // 18. Switch back to the host and extract the parallel grid function
       //     corresponding to the finite element approximation X. This is the
       //     local solution on each processor.
-      Device::Disable();
       a.RecoverFEMSolution(X, b, x);
 
       // 19. Send the solution by socket to a GLVis server.

examples/makefile

@@ -21,7 +21,7 @@ CONFIG_MK = $(MFEM_BUILD_DIR)/config/config.mk
 MFEM_LIB_FILE = mfem_is_not_built
 -include $(CONFIG_MK)
 
-SEQ_EXAMPLES = ex1 ex2 ex3 ex4 ex5 ex6 ex7 ex8 ex9 ex10 ex14 ex15 ex16 ex17\
+SEQ_EXAMPLES = AddSchwarz ex1 ex2 ex3 ex4 ex5 ex6 ex7 ex8 ex9 ex10 ex14 ex15 ex16 ex17\
   ex18 ex19 ex20 ex21
 PAR_EXAMPLES = ex1p ex2p ex3p ex4p ex5p ex6p ex7p ex8p ex9p ex10p ex11p ex12p\
  ex13p ex14p ex15p ex16p ex17p ex18p ex19p ex20p ex21p

examples/petsc/ex10p.cpp

@@ -27,8 +27,11 @@
 //               method HyperelasticOperator::ImplicitSolve is the only
 //               requirement for high-order implicit (SDIRK) time integration.
 //               If using PETSc to solve the nonlinear problem, use the option
-//               file provided (rc_ex10p) that customizes the
-//               Newton-Krylov method.
+//               files provided (see rc_ex10p, rc_ex10p_mf, rc_ex10p_mfop) that
+//               customize the Newton-Krylov method.
+//               When option --jfnk is used, PETSc will use a Jacobian-free
+//               Newton-Krylov method, using a user-defined preconditioner
+//               constructed with the PetscPreconditionerFactory class.
 //
 //               We recommend viewing examples 2 and 9 before viewing this
 //               example.
@@ -86,12 +89,15 @@ protected:
    Solver *J_solver;
    /// Preconditioner for the Jacobian solve in the Newton method
    Solver *J_prec;
+   /// Preconditioner factory for JFNK
+   PetscPreconditionerFactory *J_factory;
 
    mutable Vector z; // auxiliary vector
 
 public:
    HyperelasticOperator(ParFiniteElementSpace &f, Array<int> &ess_bdr,
-                        double visc, double mu, double K, bool use_petsc);
+                        double visc, double mu, double K,
+                        bool use_petsc, bool petsc_use_jfnk);
 
    /// Compute the right-hand side of the ODE system.
    virtual void Mult(const Vector &vx, Vector &dvx_dt) const;
@@ -136,8 +142,21 @@ public:
    virtual Operator &GetGradient(const Vector &k) const;
 
    virtual ~ReducedSystemOperator();
+
 };
 
+/** Auxiliary class to provide preconditioners for matrix-free methods */
+class PreconditionerFactory : public PetscPreconditionerFactory
+{
+private:
+   // const ReducedSystemOperator& op; // unused for now (generates warning)
+
+public:
+   PreconditionerFactory(const ReducedSystemOperator& op_, const string& name_)
+      : PetscPreconditionerFactory(name_) /* , op(op_) */ {}
+   virtual mfem::Solver* NewPreconditioner(const mfem::OperatorHandle&);
+   virtual ~PreconditionerFactory() {}
+};
 
 /** Function representing the elastic energy density for the given hyperelastic
     model+deformation. Used in HyperelasticOperator::GetElasticEnergyDensity. */
@@ -187,6 +206,7 @@ int main(int argc, char *argv[])
    int vis_steps = 1;
    bool use_petsc = true;
    const char *petscrc_file = "";
+   bool petsc_use_jfnk = false;
 
    OptionsParser args(argc, argv);
    args.AddOption(&mesh_file, "-m", "--mesh",
@@ -221,6 +241,9 @@ int main(int argc, char *argv[])
                   "Use or not PETSc to solve the nonlinear system.");
    args.AddOption(&petscrc_file, "-petscopts", "--petscopts",
                   "PetscOptions file to use.");
+   args.AddOption(&petsc_use_jfnk, "-jfnk", "--jfnk", "-no-jfnk",
+                  "--no-jfnk",
+                  "Use JFNK with user-defined preconditioner factory.");
    args.Parse();
    if (!args.Good())
    {
@@ -344,7 +367,8 @@ int main(int argc, char *argv[])
    // 9. Initialize the hyperelastic operator, the GLVis visualization and print
    //    the initial energies.
    HyperelasticOperator *oper = new HyperelasticOperator(fespace, ess_bdr, visc,
-                                                         mu, K, use_petsc);
+                                                         mu, K, use_petsc,
+                                                         petsc_use_jfnk);
 
    socketstream vis_v, vis_w;
    if (visualization)
@@ -520,7 +544,7 @@ Operator &ReducedSystemOperator::GetGradient(const Vector &k) const
    add(*v, dt, k, w);
    add(*x, dt, w, z);
    localJ->Add(dt*dt, H->GetLocalGradient(z));
-   // if we are using PETSc, the HypreParCSR jacobian will be converted to
+   // if we are using PETSc, the HypreParCSR Jacobian will be converted to
    // PETSc's AIJ on the fly
    Jacobian = M->ParallelAssemble(localJ);
    delete localJ;
@@ -537,7 +561,8 @@ ReducedSystemOperator::~ReducedSystemOperator()
 
 HyperelasticOperator::HyperelasticOperator(ParFiniteElementSpace &f,
                                            Array<int> &ess_bdr, double visc,
-                                           double mu, double K, bool use_petsc)
+                                           double mu, double K, bool use_petsc,
+                                           bool use_petsc_factory)
    : TimeDependentOperator(2*f.TrueVSize(), 0.0), fespace(f),
      M(&fespace), S(&fespace), H(&fespace),
      viscosity(visc), M_solver(f.GetComm()),
@@ -590,6 +615,8 @@ HyperelasticOperator::HyperelasticOperator(ParFiniteElementSpace &f,
       J_minres->SetPreconditioner(*J_prec);
       J_solver = J_minres;
 
+      J_factory = NULL;
+
       newton_solver.iterative_mode = false;
       newton_solver.SetSolver(*J_solver);
       newton_solver.SetOperator(*reduced_oper);
@@ -600,12 +627,20 @@ HyperelasticOperator::HyperelasticOperator(ParFiniteElementSpace &f,
    }
    else
    {
-      // if using PETSc, we create the same solver (NEWTON+MINRES+Jacobi)
+      // if using PETSc, we create the same solver (Newton + MINRES + Jacobi)
       // by command line options (see rc_ex10p)
       J_solver = NULL;
       J_prec = NULL;
+      J_factory = NULL;
       pnewton_solver = new PetscNonlinearSolver(f.GetComm(),
                                                 *reduced_oper);
+
+      // we can setup a factory to construct a "physics-based" preconditioner
+      if (use_petsc_factory)
+      {
+         J_factory = new PreconditionerFactory(*reduced_oper, "JFNK preconditioner");
+         pnewton_solver->SetPreconditionerFactory(J_factory);
+      }
       pnewton_solver->SetPrintLevel(1); // print Newton iterations
       pnewton_solver->SetRelTol(rel_tol);
       pnewton_solver->SetAbsTol(0.0);
@@ -691,12 +726,26 @@ HyperelasticOperator::~HyperelasticOperator()
 {
    delete J_solver;
    delete J_prec;
+   delete J_factory;
    delete reduced_oper;
    delete model;
    delete Mmat;
    delete pnewton_solver;
 }
 
+// This method gets called every time we need a preconditioner "oh"
+// contains the PetscParMatrix that wraps the operator constructed in
+// the GetGradient() method (see also PetscSolver::SetJacobianType()).
+// In this example, we just return a customizable PetscPreconditioner
+// using that matrix. However, the OperatorHandle argument can be
+// ignored, and any "physics-based" solver can be constructed since we
+// have access to the HyperElasticOperator class.
+Solver* PreconditionerFactory::NewPreconditioner(const mfem::OperatorHandle& oh)
+{
+   PetscParMatrix *pP;
+   oh.Get(pP);
+   return new PetscPreconditioner(*pP,"jfnk_");
+}
 
 double ElasticEnergyCoefficient::Eval(ElementTransformation &T,
                                       const IntegrationPoint &ip)
@@ -710,8 +759,8 @@ double ElasticEnergyCoefficient::Eval(ElementTransformation &T,
 
 void InitialDeformation(const Vector &x, Vector &y)
 {
-   // set the initial configuration to be the same as the reference, stress
-   // free, configuration
+   // set the initial configuration to be the same as the reference,
+   // stress free, configuration
    y = x;
 }
 

examples/petsc/makefile

@@ -84,6 +84,10 @@ EX9_E_ARGS       := -m ../../data/periodic-hexagon.mesh --usepetsc --petscopts r
 EX9_ES_ARGS      := -m ../../data/periodic-hexagon.mesh --usepetsc --petscopts rc_ex9p_expl --no-step
 EX9_IS_ARGS      := -m ../../data/periodic-hexagon.mesh --usepetsc --petscopts rc_ex9p_impl --implicit -tf 0.5
 EX10_ARGS        := -m ../../data/beam-quad.mesh --usepetsc --petscopts rc_ex10p -tf 30 -s 3 -rs 2 -dt 3
+EX10_MF_ARGS     := -m ../../data/beam-quad.mesh --usepetsc --petscopts rc_ex10p_mf -tf 6 -s 3 -rs 0 -dt 3
+EX10_MFOP_ARGS   := -m ../../data/beam-quad.mesh --usepetsc --petscopts rc_ex10p_mfop -tf 6 -s 3 -rs 0 -dt 3
+EX10_JFNK_ARGS   := -m ../../data/beam-quad.mesh --usepetsc --petscopts rc_ex10p_jfnk --jfnk -tf 6 -s 3 -rs 0 -dt 3
+
 ex1p-test-par: ex1p
 	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX1_ARGS_W))
 	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX1_ARGS_P))
@@ -107,6 +111,9 @@ ex9p-test-par: ex9p
 	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX9_IS_ARGS))
 ex10p-test-par: ex10p
 	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX10_ARGS))
+	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX10_MF_ARGS))
+	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX10_MFOP_ARGS))
+	@$(call mfem-test,$<, $(RUN_MPI), $(TESTNAME),$(EX10_JFNK_ARGS))
 
 # Testing: "test" target and mfem-test* variables are defined in config/test.mk
 

examples/petsc/rc_ex10p_jfnk

@@ -0,0 +1,5 @@
+# matrix-free Jacobian action, preconditioner constructed using PetscPreconditionerFactory
+-snes_monitor
+-snes_mf_operator
+-ksp_type minres
+-jfnk_pc_type jacobi

examples/petsc/rc_ex10p_mf

@@ -0,0 +1,4 @@
+# matrix free -> no preconditioner
+-snes_monitor
+-snes_mf
+-ksp_type minres

examples/petsc/rc_ex10p_mfop

@@ -0,0 +1,5 @@
+# matrix-free Jacobian action, preconditioner constructed from the matrix obtained by the GetGradient() method
+-snes_monitor
+-snes_mf_operator
+-ksp_type minres
+-pc_type jacobi

examples/solvers-dev/FOSLS2D_maxwell.cpp

@@ -0,0 +1,383 @@
+//                                MFEM Example multigrid-grid Cycle
+//
+// Compile with: make mg_maxwellp
+//
+// Sample runs:  mg_maxwellp -m ../data/one-hex.mesh
+
+#include "mfem.hpp"
+#include "as/schwarz.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Define exact solution
+void E_exact(const Vector &x, Vector &E);
+double H_exact(const Vector &x);
+void scaledf_exact_H(const Vector &x, Vector &f_H);
+void f_exact_H(const Vector &x, Vector &f_H);
+void rotatedf_exact_H(const Vector &x, Vector &f_H);
+void get_maxwell_solution(const Vector &x, double E[], double curlE, double curl2E[]);
+
+int dim;
+double omega;
+int isol = 1;
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Parse command-line options.
+   // geometry file
+   // const char *mesh_file = "../data/star.mesh";
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   int sdim = 3;
+   // static condensation flag
+   bool static_cond = false;
+   // visualization flag
+   bool visualization = 1;
+   int ref_levels = 1;
+   int initref = 1;
+   // number of wavelengths
+   double k = 0.5;
+   double theta = 0.5;
+   double smth_maxit = 1;
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&sdim, "-d", "--dimension", "Dimension");
+   args.AddOption(&ref_levels, "-sr", "--serial-refinements",
+                  "Number of mesh refinements");
+   args.AddOption(&initref, "-iref", "--init-refinements",
+                  "Number of initial mesh refinements");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&smth_maxit, "-sm", "--smoother-maxit",
+                  "Number of smoothing steps.");
+   args.AddOption(&theta, "-th", "--theta",
+                  "Dumping parameter for the smoother.");
+   args.AddOption(&isol, "-sol", "--solution",
+                  "Exact Solution: 0) Polynomial, 1) Sinusoidal.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   // Angular frequency
+   // omega = k;
+   // omega = 2.0 * M_PI * k;
+   omega = 2.0 * M_PI * k;
+
+   // 3. Read the mesh from the given mesh file.
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+
+   Mesh *mesh;
+   // Define a simple square or cubic mesh
+   mesh = new Mesh(1, 1, Element::QUADRILATERAL, true, 1.0, 1.0, false);
+
+   dim = mesh->Dimension();
+   if (dim == 3) {MFEM_ABORT("This is 2D Maxwell")};
+
+   for (int i = 0; i < initref; i++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   // 4. Define a finite element space on the mesh.
+   FiniteElementCollection *NDfec = new ND_FECollection(order, dim);
+   FiniteElementSpace *NDfespace = new FiniteElementSpace(mesh, NDfec);
+
+   FiniteElementSpace *H1fespace;
+   FiniteElementCollection *H1fec = new H1_FECollection(order,dim);
+   H1fespace = new FiniteElementSpace(mesh, H1fec);
+
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr;
+   if (mesh->bdr_attributes.Size())
+   {
+      ess_bdr.SetSize(mesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      // Essential BC on E. Nothing on H
+      NDfespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+   Array<int> block_offsets(3);
+   block_offsets[0] = 0;
+   block_offsets[1] = NDfespace->GetVSize();
+   block_offsets[2] = H1fespace->GetVSize();
+   block_offsets.PartialSum();
+
+   BlockVector x(block_offsets), b(block_offsets);
+   x = 0.0;
+   b = 0.0;
+
+   VectorFunctionCoefficient * Eex = new VectorFunctionCoefficient(sdim, E_exact);
+   GridFunction *E_gf = new GridFunction;
+   E_gf->MakeRef(NDfespace, x.GetBlock(0));
+   E_gf->ProjectCoefficient(*Eex);
+
+   FunctionCoefficient * Hex = new FunctionCoefficient(H_exact);
+   GridFunction *H_gf = new GridFunction;
+   H_gf->MakeRef(H1fespace, x.GetBlock(1));
+   H_gf->ProjectCoefficient(*Hex);
+
+   // 6. Set up the linear form
+   VectorFunctionCoefficient sf_H(sdim, scaledf_exact_H);
+   VectorFunctionCoefficient f_H(sdim, f_exact_H);
+   VectorFunctionCoefficient rotatedf_H(sdim, rotatedf_exact_H);
+
+
+   LinearForm *b_E = new LinearForm;
+   b_E->Update(NDfespace, b.GetBlock(0), 0);
+   b_E->AddDomainIntegrator(new VectorFEDomainLFIntegrator(sf_H));
+   b_E->Assemble();
+
+   LinearForm *b_H = new LinearForm;
+   b_H->Update(H1fespace, b.GetBlock(1), 0);
+   // TO DO
+   // b_H->AddDomainIntegrator(new VectorDomainLFGradIntegrator(rotatedf_H));
+
+   b_H->Assemble();
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ConstantCoefficient one(1.0);
+   ConstantCoefficient sigma(pow(omega, 2));
+   ConstantCoefficient neg(-abs(omega));
+   DenseMatrix mat(2);
+   mat(0,0) = 0.0; mat(0,1) = omega;
+   mat(1,0) = -omega; mat(1,1) = 0.0;
+   MatrixConstantCoefficient rot(mat);
+   DenseMatrix matt(2); 
+   mat(0,0) = 0.0; mat(0,1) = -omega;
+   mat(1,0) = omega; mat(1,1) = 0.0;
+   MatrixConstantCoefficient rott(matt); 
+   IdentityMatrixCoefficient id(2);
+
+   //
+   BilinearForm *a_EE = new BilinearForm(NDfespace);
+   a_EE->AddDomainIntegrator(new CurlCurlIntegrator(one));
+   a_EE->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   a_EE->Assemble();
+   a_EE->EliminateEssentialBC(ess_bdr, x.GetBlock(0), b.GetBlock(0));
+   a_EE->Finalize();
+   SparseMatrix &A_EE = a_EE->SpMat();
+
+   MixedBilinearForm *a_EH = new MixedBilinearForm(NDfespace, H1fespace);
+   a_EH->AddDomainIntegrator(new MixedScalarCurlIntegrator(neg));
+   a_EH->AddDomainIntegrator(new MixedVectorWeakDivergenceIntegrator(rott));
+   a_EH->Assemble();
+   a_EH->EliminateTrialDofs(ess_bdr, x.GetBlock(0), b.GetBlock(1));
+   a_EH->Finalize();
+   SparseMatrix &A_EH = a_EH->SpMat();
+
+   MixedBilinearForm *a_HE = new MixedBilinearForm(H1fespace, NDfespace);
+   a_HE->AddDomainIntegrator(new MixedScalarWeakCurlIntegrator(neg));
+   a_HE->AddDomainIntegrator(new MixedVectorGradientIntegrator(rot));
+   a_HE->Assemble();
+   a_HE->EliminateTestDofs(ess_bdr);
+   a_HE->Finalize();
+   SparseMatrix &A_HE = a_HE->SpMat();
+   // SparseMatrix &A_HE = *Transpose(A_EH);
+
+
+   BilinearForm *a_HH = new BilinearForm(H1fespace);
+   a_HH->AddDomainIntegrator(new DiffusionIntegrator(one)); // one is the coeff
+   a_HH->AddDomainIntegrator(new MassIntegrator(sigma)); // one is the coeff
+   a_HH->Assemble();
+   a_HH->Finalize();
+   SparseMatrix &A_HH = a_HH->SpMat();
+
+   BlockMatrix *LS_Maxwellop = new BlockMatrix(block_offsets);
+   LS_Maxwellop->SetBlock(0, 0, &A_EE);
+   LS_Maxwellop->SetBlock(0, 1, &A_HE);
+   LS_Maxwellop->SetBlock(1, 0, &A_EH);
+   LS_Maxwellop->SetBlock(1, 1, &A_HH);
+
+
+   UMFPackSolver *invE = new UMFPackSolver;
+   invE->Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+   invE->SetOperator(LS_Maxwellop->GetBlock(0,0));
+
+   UMFPackSolver *invH = new UMFPackSolver;
+   invH->Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+   invH->SetOperator(LS_Maxwellop->GetBlock(1,1));
+
+   BlockDiagonalPreconditioner *prec = new BlockDiagonalPreconditioner(block_offsets);
+   prec->SetDiagonalBlock(0, invE);
+   prec->SetDiagonalBlock(1, invH);
+
+   int maxit(5000);
+   double rtol(1.e-8);
+   double atol(0.0);
+   x = 0.0;
+
+   // CGSolver pcg;
+   GMRESSolver pcg;
+   pcg.SetAbsTol(atol);
+   pcg.SetRelTol(rtol);
+   pcg.SetMaxIter(maxit);
+   pcg.SetOperator(*LS_Maxwellop);
+   pcg.SetPreconditioner(*prec);
+   pcg.SetPrintLevel(1);
+   pcg.Mult(b, x);
+
+   E_gf->MakeRef(NDfespace, x.GetBlock(0), 0);
+   H_gf->MakeRef(H1fespace, x.GetBlock(1), 0);
+
+   int order_quad = max(2, 2 * order + 1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i = 0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double Error_E = E_gf->ComputeL2Error(*Eex, irs);
+   double Error_H = H_gf->ComputeL2Error(*Hex, irs);
+   
+   cout << "|| E_h - E || = " << Error_E << "\n";
+   cout << "|| H_h - H || = " << Error_H << "\n";
+   cout << "Total error = " << sqrt(Error_H*Error_H+Error_E*Error_E) << "\n";
+
+   GridFunction *E_exgf = new GridFunction(NDfespace);
+   E_exgf->ProjectCoefficient(*Eex);
+
+   GridFunction *H_exgf = new GridFunction(H1fespace);
+   H_exgf->ProjectCoefficient(*Hex);
+
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int visport = 19916;
+      // socketstream cmesh_sock(vishost, visport);
+      // cmesh_sock.precision(8);
+      // socketstream mesh_sock(vishost, visport);
+      // mesh_sock.precision(8);
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      socketstream ex_sock(vishost, visport);
+      ex_sock.precision(8);
+      socketstream sol_sockH(vishost, visport);
+      sol_sockH.precision(8);
+      socketstream ex_sockH(vishost, visport);
+      ex_sockH.precision(8);
+      if (dim == 2)
+      {
+         sol_sock << "solution\n"
+                  << *mesh << *E_gf << "window_title 'Numerical E'" << "keys rRljc\n"
+                  << flush;
+         ex_sock << "solution\n"
+                 << *mesh << *E_exgf << "window_title 'Exact E'" << "keys rRljc\n"
+                 << flush;
+         sol_sockH << "solution\n"
+                  << *mesh << *H_gf << "window_title 'Numerical H'" << "keys rRljc\n"
+                  << flush;
+         ex_sockH << "solution\n"
+                 << *mesh << *H_exgf << "window_title 'Exact H'" << "keys rRljc\n"
+                 << flush;        
+      }
+      else
+      {
+         sol_sock << "solution\n"
+                  << *mesh << *E_gf << "keys lc\n"
+                  << flush;
+         ex_sock << "solution\n"
+                 << *mesh << *E_exgf << "keys lc\n"
+                 << flush;
+      }
+   }
+   delete a_EE;
+   delete a_EH;
+   delete a_HH;
+   delete b_E;
+   delete b_H;
+   delete NDfec;
+   delete NDfespace;
+   return 0;
+}
+
+//define exact solution
+void E_exact(const Vector &x, Vector &E)
+{
+   double curlE, curl2E[2];
+   get_maxwell_solution(x, E, curlE, curl2E);
+}
+
+double H_exact(const Vector &x)
+{
+   double E[2], curlE, curl2E[2];
+   get_maxwell_solution(x, E, curlE, curl2E);
+   return curlE/omega;  //Scalar
+}
+
+
+void f_exact_H(const Vector &x, Vector &f)
+{
+   double E[2], curlE, curl2E[2];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   f(0) = curl2E[0] / omega - omega * E[0];
+   f(1) = curl2E[1] / omega - omega * E[1];
+}
+
+void rotatedf_exact_H(const Vector &x, Vector &f)
+{
+   double E[2], curlE, curl2E[2];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   f(0) = -(curl2E[1] / omega - omega * E[1]);
+   f(1) = (curl2E[0] / omega - omega * E[0]);
+}
+
+void scaledf_exact_H(const Vector &x, Vector &f)
+{
+   double E[2], curlE, curl2E[2];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   // curl H - omega E = f
+   // = - omega *( curl (curl E / omega) - omega E)
+   f(0) = -omega * (curl2E[0] / omega - omega * E[0]);
+   f(1) = -omega * (curl2E[1] / omega - omega * E[1]);
+}
+
+void get_maxwell_solution(const Vector &X, double E[], double curlE, double curl2E[])
+{
+   double x = X[0];
+   double y = X[1];
+   if (isol == 0) // polynomial
+   {
+      E[0] = x * (1.0 - x) * y * (1.0 - y);
+      E[1] = 0.0;
+      //
+      curlE = x*(1.0-x)*(2.0*y-1.0);
+
+      curl2E[0] = -2.0 * x * (x - 1.0);
+      curl2E[1] = (2.0*x-1.0)*(2.0*y-1.0);
+   }
+   else if (isol == -1) 
+   {
+      E[0] = cos(omega * y);
+      E[1] = 0.0;
+
+      curlE = -omega * sin(omega * y);
+
+      curl2E[0] = omega*omega * cos(omega*y);  
+      curl2E[1] = 0.0;
+   }
+}

examples/solvers-dev/FOSLS_maxwell.cpp

@@ -0,0 +1,435 @@
+//                                MFEM Example multigrid-grid Cycle
+//
+// Compile with: make mg_maxwellp
+//
+// Sample runs:  mg_maxwellp -m ../data/one-hex.mesh
+
+#include "mfem.hpp"
+#include "as/schwarz.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Define exact solution
+void E_exact(const Vector &x, Vector &E);
+void H_exact(const Vector &x, Vector &H);
+void scaledf_exact_H(const Vector &x, Vector &f_H);
+void f_exact_H(const Vector &x, Vector &f_H);
+void get_maxwell_solution(const Vector &x, double E[], double curlE[], double curl2E[]);
+
+int dim;
+double omega;
+int sol = 1;
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Parse command-line options.
+   // geometry file
+   // const char *mesh_file = "../data/star.mesh";
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   int sdim = 3;
+   // static condensation flag
+   bool static_cond = false;
+   // visualization flag
+   bool visualization = 1;
+   int ref_levels = 1;
+   int initref = 1;
+   // number of wavelengths
+   double k = 0.5;
+   double theta = 0.5;
+   double smth_maxit = 1;
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&sdim, "-d", "--dimension", "Dimension");
+   args.AddOption(&ref_levels, "-ref", "--serial-refinements",
+                  "Number of mesh refinements");
+   args.AddOption(&initref, "-initref", "--init-refinements",
+                  "Number of initial mesh refinements");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&smth_maxit, "-sm", "--smoother-maxit",
+                  "Number of smoothing steps.");
+   args.AddOption(&theta, "-th", "--theta",
+                  "Dumping parameter for the smoother.");
+   args.AddOption(&sol, "-sol", "--solution",
+                  "Exact Solution: 0) Polynomial, 1) Sinusoidal.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   // Angular frequency
+   // omega = k;
+   // omega = 2.0 * M_PI * k;
+   omega = 2.0 * M_PI * k;
+
+   // 3. Read the mesh from the given mesh file.
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+
+   Mesh *mesh;
+   // Define a simple square or cubic mesh
+   mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true, 1.0, 1.0, 1.0, false);
+   dim = mesh->Dimension();
+   
+   for (int i = 0; i < initref; i++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   Mesh *cmesh = new Mesh(*mesh);
+
+   for (int l = 0; l < ref_levels; l++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   // 4. Define a finite element space on the mesh.
+   FiniteElementCollection *NDfec = new ND_FECollection(order, dim);
+   FiniteElementSpace *NDfespace = new FiniteElementSpace(mesh, NDfec);
+
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr;
+   if (mesh->bdr_attributes.Size())
+   {
+      ess_bdr.SetSize(mesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      // Essential BC on E. Nothing on H
+      NDfespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+   Array<int> block_offsets(3);
+   block_offsets[0] = 0;
+   block_offsets[1] = NDfespace->GetVSize();
+   block_offsets[2] = NDfespace->GetVSize();
+   block_offsets.PartialSum();
+
+   //    _           _    _  _       _  _
+   //   |             |  |    |     |    |
+   //   |  A00   A01  |  | E  |     |F_E |
+   //   |             |  |    |  =  |    |
+   //   |  A10   A11  |  | H  |     |F_G |
+   //   |_           _|  |_  _|     |_  _|
+   //
+   // A00 = (curl E, curl F) + \omega^2 (E,F)
+   // A01 = - \omega *( (curl E, F) + (E,curl F)
+   // A10 = - \omega *( (curl H, G) + (H,curl G)
+   // A11 = (curl H, curl H) + \omega^2 (H,G)
+
+   BlockVector x(block_offsets), b(block_offsets);
+   x = 0.0;
+   b = 0.0;
+
+
+   VectorFunctionCoefficient * Eex = new VectorFunctionCoefficient(dim, E_exact);
+   GridFunction *E_gf = new GridFunction;
+   E_gf->MakeRef(NDfespace, x.GetBlock(0));
+   E_gf->ProjectCoefficient(*Eex);
+
+   VectorFunctionCoefficient * Hex;
+   Hex = new VectorFunctionCoefficient(sdim, H_exact);
+   
+   GridFunction *H_gf = new GridFunction;
+   H_gf->MakeRef(NDfespace, x.GetBlock(1));
+   H_gf->ProjectCoefficient(*Hex);
+
+   // 6. Set up the linear form
+   VectorFunctionCoefficient sf_H(dim, scaledf_exact_H);
+   VectorFunctionCoefficient f_H(dim, f_exact_H);
+
+
+   LinearForm *b_E = new LinearForm;
+   b_E->Update(NDfespace, b.GetBlock(0), 0);
+   b_E->AddDomainIntegrator(new VectorFEDomainLFIntegrator(sf_H));
+   b_E->Assemble();
+
+   LinearForm *b_H = new LinearForm;
+   b_H->Update(NDfespace, b.GetBlock(1), 0);
+   b_H->AddDomainIntegrator(new VectorFEDomainLFCurlIntegrator(f_H));
+   b_H->Assemble();
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ConstantCoefficient one(1.0);
+   ConstantCoefficient sigma(pow(omega, 2));
+   ConstantCoefficient neg(-abs(omega));
+   ConstantCoefficient pos(abs(omega));
+   //
+   BilinearForm *a_EE = new BilinearForm(NDfespace);
+   a_EE->AddDomainIntegrator(new CurlCurlIntegrator(one));
+   a_EE->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   a_EE->Assemble();
+   a_EE->EliminateEssentialBC(ess_bdr, x.GetBlock(0), b.GetBlock(0));
+   a_EE->Finalize();
+   SparseMatrix &A_EE = a_EE->SpMat();
+
+   MixedBilinearForm *a_EH = new MixedBilinearForm(NDfespace, NDfespace);
+   a_EH->AddDomainIntegrator(new MixedVectorCurlIntegrator(neg));
+   a_EH->AddDomainIntegrator(new MixedVectorWeakCurlIntegrator(neg));
+   a_EH->Assemble();
+   a_EH->EliminateTrialDofs(ess_bdr, x.GetBlock(0), b.GetBlock(1));
+   a_EH->Finalize();
+   SparseMatrix &A_EH = a_EH->SpMat();
+
+   // MixedBilinearForm *a_HE = new MixedBilinearForm(NDfespace, NDfespace);
+   // a_HE->AddDomainIntegrator(new MixedVectorWeakCurlIntegrator(neg));
+   // a_HE->AddDomainIntegrator(new MixedVectorCurlIntegrator(neg));
+   // a_HE->Assemble();
+   // a_HE->EliminateTestDofs(ess_bdr);
+   // a_HE->Finalize();
+   // SparseMatrix &A_HE = a_HE->SpMat();
+   SparseMatrix &A_HE = *Transpose(A_EH);
+
+
+   BilinearForm *a_HH = new BilinearForm(NDfespace);
+   a_HH->AddDomainIntegrator(new CurlCurlIntegrator(one)); // one is the coeff
+   a_HH->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   a_HH->Assemble();
+   a_HH->Finalize();
+   SparseMatrix &A_HH = a_HH->SpMat();
+
+   BlockMatrix *LS_Maxwellop = new BlockMatrix(block_offsets);
+   LS_Maxwellop->SetBlock(0, 0, &A_EE);
+   LS_Maxwellop->SetBlock(0, 1, &A_HE);
+   LS_Maxwellop->SetBlock(1, 0, &A_EH);
+   LS_Maxwellop->SetBlock(1, 1, &A_HH);
+
+   SparseMatrix * S = LS_Maxwellop->CreateMonolithic();
+
+   UMFPackSolver *invE = new UMFPackSolver;
+   invE->Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+   invE->SetOperator(LS_Maxwellop->GetBlock(0,0));
+
+   UMFPackSolver *invH = new UMFPackSolver;
+   invH->Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+   invH->SetOperator(LS_Maxwellop->GetBlock(1,1));
+
+   BlockDiagonalPreconditioner *prec = new BlockDiagonalPreconditioner(block_offsets);
+   prec->SetDiagonalBlock(0, invE);
+   prec->SetDiagonalBlock(1, invH);
+
+   BlkSchwarzSmoother * M = new BlkSchwarzSmoother(cmesh,ref_levels, NDfespace, S);
+   M->SetNumSmoothSteps(smth_maxit);
+   M->SetDumpingParam(theta);
+
+
+   cout << "Size of fine grid system: "
+        << 2.0 * A_EE.NumRows() << " x " << 2.0 * A_EE.NumCols() << endl;
+
+
+   int maxit(100);
+   double rtol(1.e-6);
+   double atol(0.0);
+   x = 0.0;
+
+   CGSolver pcg;
+   // GMRESSolver pcg;
+   pcg.SetAbsTol(atol);
+   pcg.SetRelTol(rtol);
+   pcg.SetMaxIter(maxit);
+   pcg.SetPreconditioner(*M);
+   pcg.SetOperator(*LS_Maxwellop);
+   // pcg.SetOperator(*S);
+   // pcg.SetPreconditioner(*prec);
+   pcg.SetPrintLevel(1);
+   pcg.Mult(b, x);
+
+   E_gf->MakeRef(NDfespace, x.GetBlock(0), 0);
+   H_gf->MakeRef(NDfespace, x.GetBlock(1), 0);
+
+   int order_quad = max(2, 2 * order + 1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i = 0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double Error_E = E_gf->ComputeL2Error(*Eex, irs);
+   double Error_H;
+   Error_H = H_gf->ComputeL2Error(*Hex, irs);
+   cout << "|| E_h - E || = " << Error_E << "\n";
+   cout << "|| H_h - H || = " << Error_H << "\n";
+
+   cout << "Total error = " << sqrt(Error_H*Error_H+Error_E*Error_E) << "\n";
+
+   GridFunction *E_exgf = new GridFunction(NDfespace);
+   E_exgf->ProjectCoefficient(*Eex);
+
+
+   GridFunction *H_exgf = new GridFunction(NDfespace);
+
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int visport = 19916;
+      // socketstream cmesh_sock(vishost, visport);
+      // cmesh_sock.precision(8);
+      // socketstream mesh_sock(vishost, visport);
+      // mesh_sock.precision(8);
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      socketstream ex_sock(vishost, visport);
+      ex_sock.precision(8);
+      socketstream sol_sockH(vishost, visport);
+      sol_sockH.precision(8);
+      socketstream ex_sockH(vishost, visport);
+      ex_sockH.precision(8);
+      if (dim == 2)
+      {
+         sol_sock << "solution\n"
+                  << *mesh << *E_gf << "window_title 'Numerical E'" << "keys rRljc\n"
+                  << flush;
+         ex_sock << "solution\n"
+                 << *mesh << *E_exgf << "window_title 'Exact E'" << "keys rRljc\n"
+                 << flush;
+         sol_sockH << "solution\n"
+                  << *mesh << *H_gf << "window_title 'Numerical H'" << "keys rRljc\n"
+                  << flush;
+         ex_sockH << "solution\n"
+                 << *mesh << *H_exgf << "window_title 'Exact H'" << "keys rRljc\n"
+                 << flush;        
+      }
+      else
+      {
+         sol_sock << "solution\n"
+                  << *mesh << *E_gf << "keys lc\n"
+                  << flush;
+         ex_sock << "solution\n"
+                 << *mesh << *E_exgf << "keys lc\n"
+                 << flush;
+      }
+   }
+   delete a_EE;
+   delete a_EH;
+   delete a_HH;
+   delete b_E;
+   delete b_H;
+   delete NDfec;
+   delete NDfespace;
+   return 0;
+}
+
+//define exact solution
+void E_exact(const Vector &x, Vector &E)
+{
+   double curlE[3], curl2E[3];
+   get_maxwell_solution(x, E, curlE, curl2E);
+}
+
+void H_exact(const Vector &x, Vector &H)
+{
+   double E[3], curlE[3], curl2E[3];
+   get_maxwell_solution(x, E, curlE, curl2E);
+   for (int i = 0; i < dim; i++) H(i) = curlE[i] / omega;
+}
+
+void f_exact_H(const Vector &x, Vector &f)
+{
+   double E[3], curlE[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   // curl H - omega E = f
+   // = curl (curl E / omega) - omega E
+   f(0) = curl2E[0] / omega - omega * E[0];
+   f(1) = curl2E[1] / omega - omega * E[1];
+   f(2) = curl2E[2] / omega - omega * E[2];
+}
+
+
+void scaledf_exact_H(const Vector &x, Vector &f)
+{
+   double E[3], curlE[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   // curl H - omega E = f
+   // = - omega *( curl (curl E / omega) - omega E)
+   f(0) = -omega * (curl2E[0] / omega - omega * E[0]);
+   f(1) = -omega * (curl2E[1] / omega - omega * E[1]);
+   f(2) = -omega * (curl2E[2] / omega - omega * E[2]);
+}
+
+void get_maxwell_solution(const Vector &X, double E[], double curlE[], double curl2E[])
+{
+   double x = X[0];
+   double y = X[1];
+   double z = X[2];
+   if (sol == 0) // polynomial
+   {
+      // Polynomial vanishing on the boundary
+      E[0] = y * z * (1.0 - y) * (1.0 - z);
+      E[1] = (1.0 - x) * x * y * (1.0 - z) * z;
+      E[2] = (1.0 - x) * x * (1.0 - y) * y;
+      //
+      curlE[0] = -(-1.0 + x) * x * (1.0 + y * (-3.0 + 2.0 * z));
+      curlE[1] = -2.0 * (-1.0 + y) * y * (x - z);
+      curlE[2] = (1.0 + (-3.0 + 2.0 * x) * y) * (-1.0 + z) * z;
+
+      curl2E[0] = -2.0 * (-1.0 + y) * y + (-3.0 + 2.0 * x) * (-1.0 + z) * z;
+      curl2E[1] = -2.0 * y * (-x + x * x + (-1.0 + z) * z);
+      curl2E[2] = -2.0 * (-1.0 + y) * y + (-1.0 + x) * x * (-3.0 + 2.0 * z);
+   }
+   else if (sol == 1) // sinusoidal
+   {
+      E[0] = sin(omega * y);
+      E[1] = sin(omega * z);
+      E[2] = sin(omega * x);
+
+      curlE[0] = -omega * cos(omega * z);
+      curlE[1] = -omega * cos(omega * x);
+      curlE[2] = -omega * cos(omega * y);
+
+      curl2E[0] = omega * omega * E[0];
+      curl2E[1] = omega * omega * E[1];
+      curl2E[2] = omega * omega * E[2];
+   }
+   else if (sol == 2) // point source
+   {
+      MFEM_ABORT("Case unfinished");
+   }
+   else if (sol == 3) // plane wave
+   {
+      double coeff = omega / sqrt(3.0);
+      E[0] = cos(coeff * (x + y + z));
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+      curlE[0] = 0.0;
+      curlE[1] = -coeff * sin(coeff * (x + y + z));
+      curlE[2] = coeff * sin(coeff * (x + y + z));
+
+      curl2E[0] = 2.0 * coeff * coeff * E[0];
+      curl2E[1] = -coeff * coeff * E[0];
+      curl2E[2] = -coeff * coeff * E[0];
+   }
+   else if (sol == -1) 
+   {
+      E[0] = cos(omega * y);
+      E[1] = 0.0;
+
+      curlE[0] = 0.0;
+      curlE[1] = 0.0;
+      curlE[2] = -omega * sin(omega * y);
+
+      curl2E[0] = omega*omega * cos(omega*y);  
+      curl2E[1] = 0.0;
+      curl2E[2] = 0.0;
+   }
+}

examples/solvers-dev/FOSLS_maxwellp.cpp

@@ -0,0 +1,618 @@
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include <boost/math/special_functions/airy.hpp>
+#include "mg/multigrid.hpp"
+#include "ams/blkams.hpp"
+#include "petsc.h"
+
+using namespace std;
+using namespace mfem;
+using namespace boost;
+
+// Define exact solution
+void E_exact(const Vector & x, Vector & E);
+void H_exact(const Vector & x, Vector & H);
+void f_exact_H(const Vector & x, Vector & f_H);
+void get_maxwell_solution(const Vector & x, double E[], double curlE[], double curl2E[]);
+void epsilon_func(const Vector &x, DenseMatrix &M);
+void epsilon2_func(const Vector &x, DenseMatrix &M);
+
+int dim;
+double omega;
+int sol = 1;
+
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv); // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid); // Determine process identifier
+   // 1. Parse command-line options.
+   // geometry file
+   // const char *mesh_file = "../data/star.mesh";
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   // visualization flag
+   bool visualization = 1;
+   // number of wavelengths
+   double k = 1.0;
+   // number of mg levels
+   int ref_levels = 1;
+   // number of initial ref
+   int initref = 1;
+
+   const char *petscrc_file = "petscrc_mult_options";
+
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&ref_levels, "-ref", "--ref_levels",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&sol, "-sol", "--exact",
+                  "Exact solution flag - "
+                  " 1:sinusoidal, 2: point source, 3: plane wave");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   // Angular frequency
+   omega = 2.0*k*M_PI;
+   // omega = k;
+
+   // 2. Read the mesh from the given mesh file.
+   Mesh *mesh;
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   double length;
+   length = (sol == 4) ? 0.5: 1.0;
+   mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true, length, length, length, false);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+// 4. Define a finite element space on the mesh.
+   FiniteElementCollection *fec = new ND_FECollection(order, dim);
+   // ParFiniteElementSpace *fespace = new ParFiniteElementSpace(mesh, fec);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   std::vector<ParFiniteElementSpace * > fespaces(ref_levels+1);
+   std::vector<ParMesh * > ParMeshes(ref_levels+1);
+   std::vector<HypreParMatrix*>  P(ref_levels);
+
+   for (int i = 0; i < ref_levels; i++)
+   {
+      ParMeshes[i] =new ParMesh(*pmesh);
+      fespaces[i] = new ParFiniteElementSpace(*fespace, *ParMeshes[i]);
+      pmesh->UniformRefinement();
+      // Update fespace
+      fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      fespace->GetTrueTransferOperator(*fespaces[i], Tr);
+      Tr.SetOperatorOwner(false);
+      Tr.Get(P[i]);
+   }
+   fespaces[ref_levels] = new ParFiniteElementSpace(*fespace);
+
+   Array<int> ess_tdof_listE;
+   Array<int> ess_tdof_listH;
+   Array<int> ess_bdrE(pmesh->bdr_attributes.Max());
+   Array<int> ess_bdrH(pmesh->bdr_attributes.Max());
+   ess_bdrE = 1;
+   ess_bdrH = 0;
+   fespace->GetEssentialTrueDofs(ess_bdrE, ess_tdof_listE);
+   fespace->GetEssentialTrueDofs(ess_bdrH, ess_tdof_listH);
+
+
+   Array<int> block_offsets(3);
+   block_offsets[0] = 0;
+   block_offsets[1] = fespace->GetVSize();
+   block_offsets[2] = fespace->GetVSize();
+   block_offsets.PartialSum();
+
+   Array<int> block_trueOffsets(3);
+   block_trueOffsets[0] = 0;
+   block_trueOffsets[1] = fespace->TrueVSize();
+   block_trueOffsets[2] = fespace->TrueVSize();
+   block_trueOffsets.PartialSum();
+
+   //    _           _    _  _       _  _
+   //   |             |  |    |     |    |
+   //   |  A00   A01  |  | E  |     |F_E |
+   //   |             |  |    |  =  |    |
+   //   |  A10   A11  |  | H  |     |F_G |
+   //   |_           _|  |_  _|     |_  _|
+   //
+   // A00 = (curl E, curl F) + \omega^2 (E,F)
+   // A01 = - \omega *( (curl E, F) + (E,curl F)
+   // A10 = - \omega *( (curl H, G) + (H,curl G)
+   // A11 = (curl H, curl H) + \omega^2 (H,G)
+
+   BlockVector x(block_offsets), rhs(block_offsets);
+   BlockVector trueX(block_trueOffsets), trueRhs(block_trueOffsets);
+
+   x = 0.0;
+   rhs = 0.0;
+   trueX = 0.0;
+   trueRhs = 0.0;
+
+
+   VectorFunctionCoefficient Eex(sdim, E_exact);
+   ParGridFunction * E_gf = new ParGridFunction;
+   ParGridFunction * Exact_gf = new ParGridFunction(fespace);
+   E_gf->MakeRef(fespace, x.GetBlock(0));
+   E_gf->ProjectCoefficient(Eex);
+   Exact_gf->ProjectCoefficient(Eex);
+
+   // VectorFunctionCoefficient Hex(sdim, H_exact);
+   // ParGridFunction * H_gf = new ParGridFunction;   
+   // H_gf->MakeRef(fespace, x.GetBlock(1));
+   // H_gf->ProjectCoefficient(Hex);
+
+
+   // ConstantCoefficient one(1.0);
+   // ConstantCoefficient sigma(pow(omega, 2));
+   // ConstantCoefficient neg(-abs(omega));
+   // ConstantCoefficient pos(abs(omega));
+
+   // // // 6. Set up the linear form
+   // VectorFunctionCoefficient f_H(sdim,f_exact_H);
+   // ScalarVectorProductCoefficient sf_H(neg,f_H);
+
+   // ParLinearForm *b_E = new ParLinearForm;
+   // b_E->Update(fespace, rhs.GetBlock(0), 0);
+   // b_E->AddDomainIntegrator(new VectorFEDomainLFIntegrator(sf_H));
+   // b_E->Assemble();
+
+   // ParLinearForm *b_H = new ParLinearForm;
+   // b_H->Update(fespace, rhs.GetBlock(1), 0);
+   // b_H->AddDomainIntegrator(new VectorFEDomainLFCurlIntegrator(f_H));
+   // b_H->Assemble();
+
+   // MatrixFunctionCoefficient epsilon(dim,epsilon_func);
+   // MatrixFunctionCoefficient epsilon2(dim,epsilon2_func);
+   // ScalarMatrixProductCoefficient coeff(neg,epsilon);
+   // ScalarMatrixProductCoefficient coeff2(sigma,epsilon2);
+
+
+   // // 7. Bilinear form a(.,.) on the finite element space
+   // ParBilinearForm *a_EE = new ParBilinearForm(fespace);
+   // a_EE->AddDomainIntegrator(new CurlCurlIntegrator(one)); 
+   // a_EE->AddDomainIntegrator(new VectorFEMassIntegrator(coeff2));
+   // a_EE->Assemble();
+   // a_EE->Finalize();
+   // HypreParMatrix *A_EE = new HypreParMatrix;
+   // a_EE->FormLinearSystem(ess_tdof_listE, x.GetBlock(0), rhs.GetBlock(0), *A_EE, trueX.GetBlock(0), trueRhs.GetBlock(0));
+
+
+   // ParBilinearForm *a_HH = new ParBilinearForm(fespace);
+   // a_HH->AddDomainIntegrator(new CurlCurlIntegrator(one)); // one is the coeff
+   // a_HH->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   // a_HH->Assemble();
+   // a_HH->Finalize();
+   // HypreParMatrix *A_HH = new HypreParMatrix;
+   // a_HH->FormLinearSystem(ess_tdof_listH, x.GetBlock(1), rhs.GetBlock(1), *A_HH, trueX.GetBlock(1), trueRhs.GetBlock(1));
+
+   // ParMixedBilinearForm *a_HE = new ParMixedBilinearForm(fespace,fespace);
+   // a_HE->AddDomainIntegrator(new MixedVectorCurlIntegrator(neg));
+   // a_HE->AddDomainIntegrator(new MixedVectorWeakCurlIntegrator(coeff)); 
+   // a_HE->Assemble();
+   // a_HE->Finalize();
+   // HypreParMatrix *A_HE = new HypreParMatrix;
+   // a_HE->FormColLinearSystem(ess_tdof_listE,x.GetBlock(0),rhs.GetBlock(1),*A_HE,trueX.GetBlock(0),trueRhs.GetBlock(1));
+
+   // HypreParMatrix *A_EH = A_HE->Transpose();
+
+   // BlockOperator *LS_Maxwellop = new BlockOperator(block_trueOffsets);
+   // LS_Maxwellop->SetBlock(0, 0, A_EE);
+   // LS_Maxwellop->SetBlock(0, 1, A_EH);
+   // LS_Maxwellop->SetBlock(1, 0, A_HE);
+   // LS_Maxwellop->SetBlock(1, 1, A_HH);
+
+
+   // if (myid == 0)
+   // {
+   //    cout << "Size of fine grid system: "
+   //         << 2.0 * A_EE->GetGlobalNumRows() << " x " << 2.0* A_EE->GetGlobalNumCols() << endl;
+   // }
+
+   // MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+   // // Set up the preconditioner
+   // Array2D<HypreParMatrix*> blockA(2,2);
+   // for (int i=0; i<2; ++i)
+   // {
+   //    for (int j=0; j<2; ++j)
+   //    {
+   //       blockA(i,j) = static_cast<HypreParMatrix *>(&LS_Maxwellop->GetBlock(i,j));
+   //    }
+   // }
+
+   // // double nnz = A_HH->NNZ();
+   // // double ndof = A_HH->GetGlobalNumRows();
+   // // double est_mem_b = nnz*12.0 + (ndof+1.0)*4; 
+   // // double gb = est_mem_b*4.0/pow(1024.0,3);
+
+   // // mfem::out << "Estimated memory taken by the global matrix: " <<  gb << endl;
+
+   // int maxit(500);
+   // double rtol(1.e-6);
+   // double atol(1.e-6);
+
+   // // trueX = 0.0;
+   // CGSolver pcg(MPI_COMM_WORLD);
+   // pcg.SetAbsTol(atol);
+   // pcg.SetRelTol(rtol);
+   // pcg.SetMaxIter(maxit);
+   // pcg.SetOperator(*LS_Maxwellop);
+   // pcg.SetPrintLevel(1);
+   // chrono.Clear();
+   // chrono.Start();
+   // BlockMGSolver * precMG = new BlockMGSolver(blockA,P,fespaces);
+   // precMG->SetTheta(1.0/5.0);
+   // // int lv_coarse = min(ref_levels,ref_levels-1);
+   // // int levels = ref_levels - lv_coarse; 
+   // // BlkParSchwarzSmoother * precAS = new BlkParSchwarzSmoother(fespaces[lv_coarse]->GetParMesh(),levels,fespaces[ref_levels],LS_Maxwellop);
+   // chrono.Stop();
+   // if (myid == 0)
+   // {
+   //    cout << "MG Setup time: " << chrono.RealTime() << endl;
+   // }
+   
+   // chrono.Clear();
+   // chrono.Start();
+   // pcg.SetPreconditioner(*precMG);
+   // // pcg.SetPreconditioner(*precAS);
+   // pcg.Mult(trueRhs, trueX);
+   // chrono.Stop();
+   // delete precMG;
+   // // delete precAS;
+   
+   // // trueX = 0.0;
+   // // invA->Mult(trueRhs,trueX);
+   // MFEMFinalizePetsc();
+
+   // if (myid == 0)
+   // {
+   //    cout << "MG Solution time time: " << chrono.RealTime() << endl;
+   // }   
+   // // cin.get();
+   // //  if(myid == 0)
+   // //    cout << "MG prec Solution time: " << chrono.RealTime() << endl;
+
+
+   // // chrono.Clear();
+   // // chrono.Start();
+   // // Block_AMSSolver * precAMS = new Block_AMSSolver(block_trueOffsets,fespaces);
+   // // precAMS->SetSmootherType(Block_AMS::BlkSmootherType::SCHWARZ);
+   // // precAMS->SetSmootherType(Block_AMS::BlkSmootherType::HYPRE);
+   // // precAMS->SetOperator(LS_Maxwellop);
+   // // precAMS->SetTheta(1.0/5.0);
+   // // // 0-Smoother, 1-Grad, 2,3,4-Pix,Piy,Piz
+   // // precAMS->SetCycleType("023414320");
+   // // precAMS->SetNumberofCycles(1);
+   // // chrono.Stop();
+   // // if(myid == 0)
+   // //    cout << "BlkAMS Setup time: " << chrono.RealTime() << endl;
+
+   // // // resolve with block AMS
+   // // trueX = 0; 
+   // // chrono.Clear();
+   // // chrono.Start();
+   // // pcg.SetPreconditioner(*precAMS);
+   // // pcg.Mult(trueRhs, trueX);
+   // // chrono.Stop();
+   // // delete precAMS;
+
+   // // if(myid == 0)
+   // //    cout << "BlockAMS Solution time: " << chrono.RealTime() << endl;
+
+
+   // a_EE->RecoverFEMSolution(trueX.GetBlock(0), *b_E, *E_gf);
+   // a_HH->RecoverFEMSolution(trueX.GetBlock(1), *b_H, *H_gf);
+
+
+   // int order_quad = max(2, 2*order+1);
+   // const IntegrationRule *irs[Geometry::NumGeom];
+   // for (int i=0; i < Geometry::NumGeom; ++i)
+   // {
+   //    irs[i] = &(IntRules.Get(i, order_quad));
+   // }
+
+   // double Error_E = E_gf->ComputeL2Error(Eex, irs);
+   // double norm_E = ComputeGlobalLpNorm(2, Eex, *pmesh, irs);
+
+   // double Error_H = H_gf->ComputeL2Error(Hex, irs);
+   // double norm_H = ComputeGlobalLpNorm(2, Hex , *pmesh, irs);
+
+   // if (myid == 0)
+   // {
+   //    cout << "|| E_h - E || = " << Error_E  << "\n";
+   //    cout << "|| E_h - E ||/||E|| = " << Error_E/norm_E  << "\n";
+   //    cout << "|| H_h - H || = " << Error_H  << "\n";
+   //    cout << "|| H_h - H ||/||H|| = " << Error_H/norm_H  << "\n";
+   //    cout << "Total error = " << setprecision(15) << sqrt(Error_H*Error_H+Error_E*Error_E) << "\n";
+   // }
+
+   // ParGridFunction ExactE(fespace);
+
+   if (visualization)
+   {
+      // 8. Connect to GLVis.
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      // socketstream E_sock(vishost, visport);
+      // E_sock << "parallel " << num_procs << " " << myid << "\n";
+      // E_sock.precision(8);
+      // E_sock << "solution\n" << *pmesh << *E_gf << "window_title 'Electric field'" << endl;
+      socketstream Exact_sock(vishost, visport);
+      Exact_sock << "parallel " << num_procs << " " << myid << "\n";
+      Exact_sock.precision(8);
+      Exact_sock << "solution\n" << *pmesh << *Exact_gf << "window_title 'Electric field'" << endl;
+ 
+    // MPI_Barrier(pmesh->GetComm());
+      // socketstream Eex_sock(vishost, visport);
+      // Eex_sock << "parallel " << num_procs << " " << myid << "\n";
+      // Eex_sock.precision(8);
+      // Eex_sock << "solution\n" << *pmesh << *Exact_gf << "window_title 'Exact Electric field'" << endl;
+   }
+
+   // delete A_EE;
+   // delete A_HE;
+   // delete A_EH;
+   // delete A_HH;
+   // delete LS_Maxwellop;
+   // delete a_EE;
+   // delete a_HE;
+   // delete a_HH;
+   // delete b_E;
+   // delete b_H;
+   delete E_gf;
+   delete Exact_gf;
+   for (auto p: ParMeshes) delete p;
+   for (auto p: fespaces)  delete p;
+   for (auto p: P)  delete p;
+   ParMeshes.clear();
+   fespaces.clear();
+   P.clear();
+   delete fec;
+   delete fespace;
+   delete pmesh;
+   
+   // cout << "Freed memory: " << endl;
+   // cin.get();
+   MPI_Finalize();
+   return 0;
+}
+
+//define exact solution
+void E_exact(const Vector &x, Vector &E)
+{
+   double curlE[3], curl2E[3];
+   get_maxwell_solution(x, E, curlE, curl2E);
+}
+
+void H_exact(const Vector &x, Vector &H)
+{
+   double E[3], curlE[3], curl2E[3];
+   get_maxwell_solution(x, E, curlE, curl2E);
+   for (int i = 0; i<3; i++) H(i) = curlE[i]/omega;
+}
+
+
+void f_exact_H(const Vector &x, Vector &f)
+{
+   // curl H - omega E = f
+   // = curl (curl E / omega) - omega E
+   f = 0.0;
+   if (sol !=4)
+   {
+      double E[3], curlE[3], curl2E[3];
+      get_maxwell_solution(x, E, curlE, curl2E);
+      f(0) = curl2E[0] / omega - omega * E[0];
+      f(1) = curl2E[1] / omega - omega * E[1];
+      f(2) = curl2E[2] / omega - omega * E[2];
+   }
+}
+ 
+   void get_maxwell_solution(const Vector &X, double E[], double curlE[], double curl2E[])
+{
+   double x = X[0];
+   double y = X[1];
+   double z = X[2];
+
+
+
+   if (sol ==-1)
+   {
+      E[0] = y * z * (1.0 - y) * (1.0 - z);
+      E[1] = x * y * z * (1.0 - x) * (1.0 - z);
+      E[2] = x * y * (1.0 - x) * (1.0 - y);
+      
+      curlE[0] = -(x-1.0) * x * (y*(2.0*z-3.0)+1.0);
+      curlE[1] = -2.0*(y-1.0)*y*(x-z);
+      curlE[2] = (z-1)*z*(1.0+y*(2.0*x-3.0));
+      
+      curl2E[0] = 2.0 * y * (1.0 - y) - (2.0 * x - 3.0) * z * (1 - z);
+      curl2E[1] = 2.0 * y * (x * (1.0 - x) + (1.0 - z) * z);
+      curl2E[2] = 2.0 * y * (1.0 - y) + x * (3.0 - 2.0 * z) * (1.0 - x);
+   }
+   else if (sol == 0) // polynomial
+   {
+      // Polynomial vanishing on the boundary
+      E[0] = y * z * (1.0 - y) * (1.0 - z);
+      E[1] = (1.0 - x) * x * y * (1.0 - z) * z;
+      E[2] = (1.0 - x) * x * (1.0 - y) * y;
+      //
+      curlE[0] = -(-1.0 + x) * x * (1.0 + y * (-3.0 + 2.0 * z));
+      curlE[1] = -2.0 * (-1.0 + y) * y * (x - z);
+      curlE[2] = (1.0 + (-3.0 + 2.0 * x) * y) * (-1.0 + z) * z;
+
+      curl2E[0] = -2.0 * (-1.0 + y) * y + (-3.0 + 2.0 * x) * (-1.0 + z) * z;
+      curl2E[1] = -2.0 * y * (-x + x * x + (-1.0 + z) * z);
+      curl2E[2] = -2.0 * (-1.0 + y) * y + (-1.0 + x) * x * (-3.0 + 2.0 * z);
+   }
+   else if (sol == 1) // sinusoidal
+   {
+      E[0] = sin(omega * y);
+      E[1] = sin(omega * z);
+      E[2] = sin(omega * x);
+
+      curlE[0] = -omega * cos(omega * z);
+      curlE[1] = -omega * cos(omega * x);
+      curlE[2] = -omega * cos(omega * y);
+
+      curl2E[0] = omega * omega * E[0];
+      curl2E[1] = omega * omega * E[1];
+      curl2E[2] = omega * omega * E[2];
+   }
+   else if (sol == 2) // point source
+   {
+       // shift to avoid singularity
+      double x0 = x + 0.1;
+      double x1 = y + 0.1;
+      double x2 = z + 0.1;
+      //
+      double r = sqrt(x0 * x0 + x1 * x1 + x2 * x2);
+
+      E[0] = cos(omega * r);
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+      double r_x = x0 / r;
+      double r_y = x1 / r;
+      double r_z = x2 / r;
+      double r_xy = -(r_x / r) * r_y;
+      double r_xz = -(r_x / r) * r_z;
+      double r_yx = r_xy;
+      double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+      double r_zx = r_xz;
+      double r_zz = (1.0 / r) * (1.0 - r_z * r_z);
+
+      curlE[0] = 0.0;
+      curlE[1] = -omega * r_z * sin(omega * r);
+      curlE[2] =  omega * r_y * sin(omega * r);
+
+      curl2E[0] = omega * ((r_yy + r_zz) * sin(omega * r) +
+                           (omega * r_y * r_y + omega * r_z * r_z) * cos(omega * r));
+      curl2E[1] = -omega * (r_yx * sin(omega * r) + omega * r_y * r_x * cos(omega * r));
+      curl2E[2] = -omega * (r_zx * sin(omega * r) + omega * r_z * r_x * cos(omega * r));
+   }
+   else if (sol == 3) // plane wave
+   {
+      double coeff = omega / sqrt(3.0);
+      E[0] = cos(coeff * (x + y + z));
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+      curlE[0] = 0.0;
+      curlE[1] = -coeff * sin(coeff * (x + y + z));
+      curlE[2] = coeff * sin(coeff * (x + y + z));
+
+      curl2E[0] = 2.0 * coeff * coeff * E[0];
+      curl2E[1] = -coeff * coeff * E[0];
+      curl2E[2] = -coeff * coeff * E[0];
+   }
+   else if (sol == -1) 
+   {
+      E[0] = cos(omega * y);
+      E[1] = 0.0;
+
+      curlE[0] = 0.0;
+      curlE[1] = 0.0;
+      curlE[2] = -omega * sin(omega * y);
+
+      curl2E[0] = omega*omega * cos(omega*y);  
+      curl2E[1] = 0.0;
+      curl2E[2] = 0.0;
+   }
+   else if (sol == 4) // Airy function
+   {
+      E[0] = 0;
+      E[1] = 0;
+      // double b = -pow(omega/4.0,2.0/3.0)*(4.0*x(0)-1.0);
+      double b = -pow(omega/4.0,2.0/3.0)*(4.0*x-1.0);
+      E[2] = boost::math::airy_ai(b);
+
+      //  not used
+      curl2E[0] = 0.0;
+      curl2E[1] = 0.0;  
+      curl2E[2] = 0.0;
+   }
+}
+
+
+void epsilon_func(const Vector &x, DenseMatrix &M)
+{
+   M.SetSize(3);
+
+   M = 0.0;
+   M(0,0) = 1.0;
+   M(1,1) = 1.0;
+   if (sol != 4)
+   {
+      M(2,2) = 1.0;
+   }
+   else
+   {
+      M(2,2) = 4.0*x(0)-1.0;
+      // M(2,2) = 2.0;
+   }
+}
+
+void epsilon2_func(const Vector &x, DenseMatrix &M)
+{
+   M.SetSize(3);
+
+   M = 0.0;
+   M(0,0) = 1.0;
+   M(1,1) = 1.0;
+   if (sol != 4)
+   {
+      M(2,2) = 1.0;
+   }
+   else
+   {
+      M(2,2) = pow(4.0*x(0)-1.0,2.0);
+      // M(2,2) = 4.0;
+   }
+}

examples/solvers-dev/ams/blkams.cpp

@@ -0,0 +1,400 @@
+//                                MFEM Example multigrid-grid Cycle
+//
+// Compile with: make mg_maxwellp
+//
+// Sample runs:  mg_maxwellp -m ../data/one-hex.mesh
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include "blkams.hpp"
+
+using namespace std;
+using namespace mfem;
+
+Block_AMSSolver::Block_AMSSolver(Array<int> offsets_, std::vector<ParFiniteElementSpace *> fespaces_)
+      : fespaces(fespaces_), offsets(offsets_), offsetsG(3), offsetsPi(3)
+{
+   nrmeshes = fespaces.size();
+   Grad = GetDiscreteGradientOp(fespaces[nrmeshes-1]);
+   Pi = GetNDInterpolationOp(fespaces[nrmeshes-1]);
+   Pix = Pi(0,0);
+   Piy = Pi(0,1);
+   Piz = Pi(0,2);
+   
+   offsetsG[0]=0;
+   offsetsG[1]=Grad->Width();
+   offsetsG[2]=Grad->Width();
+   offsetsG.PartialSum();
+   offsetsPi[0]=0;
+   offsetsPi[1]=Pix->Width();
+   offsetsPi[2]=Pix->Width();
+   offsetsPi.PartialSum();
+   G      = new BlockOperator(offsets, offsetsG);
+   Px     = new BlockOperator(offsets, offsetsPi);
+   Py     = new BlockOperator(offsets, offsetsPi);
+   Pz     = new BlockOperator(offsets, offsetsPi);
+   GtAG   = new BlockOperator(offsetsG);
+   PxtAPx = new BlockOperator(offsetsPi);
+   PytAPy = new BlockOperator(offsetsPi);
+   PztAPz = new BlockOperator(offsetsPi);
+   A = new BlockOperator(offsets);
+   this->height = 2*Grad->Height();
+   this->width = 2*Grad->Height();
+   blkAMG_G = new BlockDiagonalPreconditioner(offsetsG);
+   blkAMG_Px = new BlockDiagonalPreconditioner(offsetsPi);
+   blkAMG_Py = new BlockDiagonalPreconditioner(offsetsPi);
+   blkAMG_Pz = new BlockDiagonalPreconditioner(offsetsPi);
+}
+
+
+void Block_AMSSolver::SetOperator(BlockOperator * bop)
+{
+   A_array.SetSize(2,2);
+   for (int i=0; i<2; i++)
+   {
+      for (int j=0; j<2; j++)
+      {
+         A_array(i,j) = static_cast<HypreParMatrix *>(&bop->GetBlock(i,j));
+      }
+   }
+   if (sType == Block_AMS::BlkSmootherType::SCHWARZ)
+   {
+      // int lvls = std::min(1,nrmeshes);
+      // int lvls = nrmeshes;
+      int lvls = 1;
+      D = new BlkParSchwarzSmoother(fespaces[nrmeshes-lvls]->GetParMesh(),lvls-1,fespaces[nrmeshes-1],bop);
+      // dynamic_cast<BlkParSchwarzSmoother *>(D)->SetDumpingParam(1.0);
+      // dynamic_cast<BlkParSchwarzSmoother *>(D)->SetNumSmoothSteps(1);
+   }
+   else
+   {
+      l1A00 = new HypreParMatrix(*A_array(0,0));
+      l1A11 = new HypreParMatrix(*A_array(1,1));
+      // DiagAddL1norm();
+      HypreSmoother * D_00 = new HypreSmoother;  
+      // D_00->SetType(HypreSmoother::l1GS);
+      D_00->SetType(HypreSmoother::Jacobi);
+      D_00->SetOperator(*l1A00);
+      HypreSmoother * D_11 = new HypreSmoother;  
+      // D_11->SetType(HypreSmoother::l1GS);
+      D_11->SetType(HypreSmoother::Jacobi);
+      D_11->SetOperator(*l1A11);
+      D = new BlockOperator(offsets);
+      static_cast<BlockOperator *>(D)->SetDiagonalBlock(0, D_00);
+      static_cast<BlockOperator *>(D)->SetDiagonalBlock(1, D_11);
+   }
+   SetOperators();
+}
+
+void Block_AMSSolver::SetOperators()
+{
+   hRAP_G.SetSize(2,2);
+   hRAP_Px.SetSize(2,2);
+   hRAP_Py.SetSize(2,2);
+   hRAP_Pz.SetSize(2,2);
+   for (int i=0; i<2 ; i++)
+   {
+      G->SetBlock(i,i,Grad);
+      Px->SetBlock(i,i,Pix);
+      Py->SetBlock(i,i,Piy);
+      Pz->SetBlock(i,i,Piz);
+      for (int j=0; j<2 ; j++)
+      {
+         A->SetBlock(i,j,A_array(i,j));
+         // hRAP_G(i,j) =  RAP(A_array(i,j),Grad);
+         // hRAP_Px(i,j) =  RAP(A_array(i,j),Pix);
+         // hRAP_Py(i,j) =  RAP(A_array(i,j),Piy);
+         // hRAP_Pz(i,j) =  RAP(A_array(i,j),Piz);
+
+         // GtAG->SetBlock(i,j,hRAP_G(i,j));
+         // PxtAPx->SetBlock(i,j,hRAP_Px(i,j));
+         // PytAPy->SetBlock(i,j,hRAP_Py(i,j));
+         // PztAPz->SetBlock(i,j,hRAP_Pz(i,j));
+         GtAG->SetBlock(i,j,RAP(A_array(i,j),Grad));
+         PxtAPx->SetBlock(i,j,RAP(A_array(i,j),Pix));
+         PytAPy->SetBlock(i,j,RAP(A_array(i,j),Piy));
+         PztAPz->SetBlock(i,j,RAP(A_array(i,j),Piz));
+      }
+   }
+
+   // create hypre matrices from blocks
+   // Array2D<double> coefficients(2,2);
+   // coefficients(0,0) = 1.0;  coefficients(0,1) = 1.0;
+   // coefficients(1,0) = 1.0;  coefficients(1,1) = 1.0;
+   // hGtAG = CreateHypreParMatrixFromBlocks(MPI_COMM_WORLD,offsetsG,hRAP_G,coefficients);
+   // hPxtAPx = CreateHypreParMatrixFromBlocks(MPI_COMM_WORLD,offsetsPi,hRAP_Px,coefficients);
+   // hPytAPy = CreateHypreParMatrixFromBlocks(MPI_COMM_WORLD,offsetsPi,hRAP_Py,coefficients);
+   // hPztAPz = CreateHypreParMatrixFromBlocks(MPI_COMM_WORLD,offsetsPi,hRAP_Pz,coefficients);
+
+
+   // AMG_G = new HypreBoomerAMG(*hGtAG); AMG_G->SetPrintLevel(0);
+   // AMG_Px = new HypreBoomerAMG(*hPxtAPx); AMG_Px->SetPrintLevel(0);
+   // AMG_Py = new HypreBoomerAMG(*hPytAPy); AMG_Py->SetPrintLevel(0);
+   // AMG_Pz = new HypreBoomerAMG(*hPztAPz); AMG_Pz->SetPrintLevel(0);
+
+   for (int i=0; i<2 ; i++)
+   { 
+      HypreBoomerAMG * G_AMG = new HypreBoomerAMG(*RAP(A_array(i,i),Grad));
+      HypreBoomerAMG * Px_AMG = new HypreBoomerAMG(*RAP(A_array(i,i),Pix));
+      HypreBoomerAMG * Py_AMG = new HypreBoomerAMG(*RAP(A_array(i,i),Piy));
+      HypreBoomerAMG * Pz_AMG = new HypreBoomerAMG(*RAP(A_array(i,i),Piz));
+      G_AMG->SetPrintLevel(0);
+      G_AMG->SetErrorMode(HypreSolver::ErrorMode::IGNORE_HYPRE_ERRORS);
+      Px_AMG->SetPrintLevel(0);
+      Px_AMG->SetErrorMode(HypreSolver::ErrorMode::IGNORE_HYPRE_ERRORS);
+      Py_AMG->SetPrintLevel(0);
+      Py_AMG->SetErrorMode(HypreSolver::ErrorMode::IGNORE_HYPRE_ERRORS);
+      Pz_AMG->SetPrintLevel(0);
+      Pz_AMG->SetErrorMode(HypreSolver::ErrorMode::IGNORE_HYPRE_ERRORS);
+      blkAMG_G->SetDiagonalBlock(i,G_AMG);
+      blkAMG_Px->SetDiagonalBlock(i,Px_AMG);
+      blkAMG_Py->SetDiagonalBlock(i,Py_AMG);
+      blkAMG_Pz->SetDiagonalBlock(i,Pz_AMG);
+   }   
+}
+
+void Block_AMSSolver::SetTheta(const double a) {theta = a;}
+void Block_AMSSolver::SetCycleType(const string c_type) {cycle_type = c_type;}
+void Block_AMSSolver::SetNumberofCycles(const int k) {NumberOfCycles = k;}
+
+void Block_AMSSolver::DiagAddL1norm()
+{
+   int n=A_array(1,1)->Height();
+   Vector l1norm0(n);
+   Vector l1norm1(n);
+
+   Getrowl1norm(A_array(0,1), l1norm0);
+   Getrowl1norm(A_array(1,0), l1norm1);
+
+   hypre_ParCSRMatrix * A_00 = (hypre_ParCSRMatrix *)const_cast<HypreParMatrix&>(*l1A00);
+   // Add the L1 norms on the diagonal
+   for (int j = 0; j < n; j++)
+   {
+      A_00->diag->data[A_00->diag->i[j]] += l1norm0(j);
+   }
+
+   hypre_ParCSRMatrix * A_11 = (hypre_ParCSRMatrix *)const_cast<HypreParMatrix&>(*l1A11);
+   // Add the L1 norms on the diagonal
+   for (int j = 0; j < n; j++)
+   {
+      A_11->diag->data[A_11->diag->i[j]] += l1norm1(j);
+   }
+}
+
+void Block_AMSSolver::Getrowl1norm(HypreParMatrix *A , Vector &l1norm)
+{
+   // First cast as hypre_ParCSRMatrix
+   hypre_ParCSRMatrix * Ah = (hypre_ParCSRMatrix *)const_cast<HypreParMatrix&>(*A);
+   HYPRE_Int num_rows = hypre_ParCSRMatrixNumRows(Ah);
+   hypre_CSRMatrix *A_diag = hypre_ParCSRMatrixDiag(Ah);
+   HYPRE_Int *A_diag_I = hypre_CSRMatrixI(A_diag);
+   // HYPRE_Int *A_diag_J = hypre_CSRMatrixJ(A_diag);
+   HYPRE_Real *A_diag_data = hypre_CSRMatrixData(A_diag);
+
+   hypre_CSRMatrix *A_offd = hypre_ParCSRMatrixOffd(Ah);
+   HYPRE_Int *A_offd_I = hypre_CSRMatrixI(A_offd);
+   // HYPRE_Int *A_offd_J = hypre_CSRMatrixJ(A_offd);
+   HYPRE_Real *A_offd_data = hypre_CSRMatrixData(A_offd);
+   HYPRE_Int num_cols_offd = hypre_CSRMatrixNumCols(A_offd);
+
+   //Initialize vector;
+
+   l1norm = 0.0;
+   for (int i = 0; i < num_rows; i++)
+   {
+      /* Add the l1 norm of the diag part of the ith row */
+      for (int j = A_diag_I[i]; j < A_diag_I[i+1]; j++)
+         l1norm(i) += fabs(A_diag_data[j]);
+      /* Add the l1 norm of the offd part of the ith row */
+      if (num_cols_offd)
+      {
+         for (int j = A_offd_I[i]; j < A_offd_I[i+1]; j++)
+            l1norm(i) += fabs(A_offd_data[j]);
+      }
+   }
+}
+
+void Block_AMSSolver::Mult(const Vector &r, Vector &z) const
+{
+   int n = r.Size();
+   int m = A->Height();
+   // int Numit = 0;
+   if (n != m ) {cout << "Size inconsistency" << endl;}
+
+   Vector res(n), raux(n),zaux(n);
+   //initialization
+   res = r; z = 0.0;
+   Array<BlockOperator *> Tr_v(4);
+   Array<BlockOperator *> PtAP_v(4);
+   Array<BlockDiagonalPreconditioner *> blkAMG_v(4);
+
+   // Array<HypreParMatrix *> hPtAP_v(4);
+   // Array<HypreBoomerAMG *> AMG_v(4);
+
+   // hPtAP_v[0]  = hGtAG;    hPtAP_v[1]  = hPxtAPx;   hPtAP_v[2]  = hPytAPy;   hPtAP_v[3]  = hPztAPz;
+   // AMG_v[0]    = AMG_G;    AMG_v[1]    = AMG_Px;    AMG_v[2]    = AMG_Py;    AMG_v[3]    = AMG_Pz;  
+
+
+   Tr_v[0]     = G;        Tr_v[1]     = Px;        Tr_v[2]     = Py;        Tr_v[3]     = Pz;
+   PtAP_v[0]   = GtAG;     PtAP_v[1]   = PxtAPx;    PtAP_v[2]   = PytAPy;    PtAP_v[3]   = PztAPz;
+   blkAMG_v[0] = blkAMG_G; blkAMG_v[1] = blkAMG_Px; blkAMG_v[2] = blkAMG_Py; blkAMG_v[3] = blkAMG_Pz;  
+   //
+   int len = cycle_type.length();
+   Array<int> ii(len);
+   for (int i=0; i<len; i++){ii[i]=cycle_type[i]-'0';}
+   //
+   for (int ic = 0; ic<NumberOfCycles; ic++)
+   {
+      for (int j = 0; j<len ; j++)
+      {
+         int i = ii[j];
+         if (i ==0)
+         {
+            D->Mult(res,zaux); zaux *= theta;
+         }
+         else
+         {
+            GetCorrection(Tr_v[i-1], PtAP_v[i-1], blkAMG_v[i-1], res, zaux);
+            // GetCorrection(Tr_v[i-1], hPtAP_v[i-1], AMG_v[i-1], res, zaux);
+         }
+         z +=zaux; 
+         A->Mult(zaux,raux); res -=raux;
+      }
+   }
+}
+
+void Block_AMSSolver::GetCorrection(BlockOperator* Tr, BlockOperator* op, BlockDiagonalPreconditioner *prec, Vector &r, Vector &z) const
+{
+   int k = Tr->Width();
+   Vector raux(k), zaux(k);
+   // Map trough the Transpose of the Transfer operator
+   Tr->MultTranspose(r,raux);
+   zaux = 0.0;
+
+   int maxit(20);
+   double rtol(0.0);
+   double atol(1e-6);
+   
+   CGSolver cg(MPI_COMM_WORLD);
+   cg.SetAbsTol(atol);
+   cg.SetRelTol(rtol);
+   cg.SetMaxIter(maxit);
+   cg.SetOperator(*op);
+   cg.SetPreconditioner(*prec);
+   cg.SetPrintLevel(-1);
+   cg.Mult(raux, zaux);
+      // prec->Mult(raux,zaux);
+
+   // Map back to the original space through the Tranfer operator
+   Tr->Mult(zaux, z);
+}
+
+
+void Block_AMSSolver::GetCorrection(BlockOperator* Tr, HypreParMatrix* op, HypreBoomerAMG *prec, Vector &r, Vector &z) const
+{
+   int k = Tr->Width();
+   Vector raux(k), zaux(k);
+   // Map trough the Transpose of the Transfer operator
+   Tr->MultTranspose(r,raux);
+   zaux = 0.0;
+
+   int maxit(100);
+   double rtol(0.0);
+   double atol(1e-6);
+
+   CGSolver cg(MPI_COMM_WORLD);
+   cg.SetAbsTol(atol);
+   cg.SetRelTol(rtol);
+   cg.SetMaxIter(maxit);
+   cg.SetOperator(*op);
+   cg.SetPreconditioner(*prec);
+   cg.SetPrintLevel(-1);
+   cg.Mult(raux, zaux);
+      // prec->Mult(raux,zaux);
+
+   // Map back to the original space through the Tranfer operator
+   Tr->Mult(zaux, z);
+}
+
+
+Block_AMSSolver::~Block_AMSSolver()
+{
+   delete Grad;
+   delete Pix;
+   delete Piy;
+   delete Piz;
+   delete G;
+   delete Px;
+   delete Py;
+   delete Pz;
+   delete GtAG;
+   delete PxtAPx;
+   delete PytAPy;
+   delete PztAPz;
+   delete A;
+   delete blkAMG_G; 
+   delete blkAMG_Px;
+   delete blkAMG_Py;
+   delete blkAMG_Pz;
+}
+
+// Discrete gradient matrix
+HypreParMatrix* GetDiscreteGradientOp(ParFiniteElementSpace *fespace)
+{
+   int dim = fespace->GetMesh()->Dimension();
+   // int sdim = fespace->GetMesh()->SpaceDimension();
+   // const FiniteElementCollection *fec = fespace->FEColl();
+   int p = 1;
+   if (fespace->GetNE() > 0)
+   {
+      p = fespace->GetOrder(0);
+   }
+   ParMesh *pmesh = fespace->GetParMesh();
+   FiniteElementCollection *vert_fec;
+   vert_fec = new H1_FECollection(p, dim);
+   ParFiniteElementSpace *vert_fespace = new ParFiniteElementSpace(pmesh,vert_fec);
+   // generate and set the discrete gradient
+   ParDiscreteLinearOperator *grad;
+   grad = new ParDiscreteLinearOperator(vert_fespace, fespace);
+   grad->AddDomainInterpolator(new GradientInterpolator);
+   grad->Assemble();
+   grad->Finalize();
+   HypreParMatrix *G;
+   G = grad->ParallelAssemble();
+   delete vert_fespace;
+   delete grad;
+   return G;
+}
+
+
+// Discrete gradient matrix
+Array2D<HypreParMatrix *> GetNDInterpolationOp(ParFiniteElementSpace *fespace)
+{
+   int dim = fespace->GetMesh()->Dimension();
+   int sdim = fespace->GetMesh()->SpaceDimension();
+   // const FiniteElementCollection *fec = fespace->FEColl();
+   int p = 1;
+   if (fespace->GetNE() > 0)
+   {
+      p = fespace->GetOrder(0);
+   }
+   ParMesh *pmesh = fespace->GetParMesh();
+   FiniteElementCollection *vert_fec;
+   vert_fec = new H1_FECollection(p, dim);
+   
+   Array2D<HypreParMatrix *> Pi_blocks;
+   ParFiniteElementSpace *vert_fespace_d
+         = new ParFiniteElementSpace(pmesh, vert_fec, sdim, Ordering::byVDIM);
+   ParDiscreteLinearOperator *id_ND;
+   id_ND = new ParDiscreteLinearOperator(vert_fespace_d, fespace);
+   id_ND->AddDomainInterpolator(new IdentityInterpolator);
+   id_ND->Assemble();
+   id_ND->Finalize();
+   id_ND->GetParBlocks(Pi_blocks);
+   //
+   delete id_ND;
+   delete vert_fespace_d;
+   delete vert_fec;
+   return Pi_blocks;
+}

examples/solvers-dev/ams/blkams.hpp

@@ -0,0 +1,67 @@
+#pragma once
+
+#include "mfem.hpp"
+#include "../as/blkschwarzp.hpp"
+#include "../mg/util.hpp"
+
+using namespace mfem;
+using namespace std;
+
+
+namespace Block_AMS
+{
+   enum BlkSmootherType{HYPRE, SCHWARZ};
+}
+
+class Block_AMSSolver : public Solver 
+{
+public:
+   Block_AMSSolver(Array<int> offsets_, std::vector<ParFiniteElementSpace *>fespaces_);
+   virtual void SetOperator(const Operator & ) {}
+   virtual void SetOperator(BlockOperator * bop);
+   void SetOperators();
+   void SetTheta(const double a);
+   void SetSmootherType(Block_AMS::BlkSmootherType type){sType = type;}
+   void SetCycleType(const string c_type);
+   void SetNumberofCycles(const int k);
+
+   void Mult(const Vector &r, Vector &z) const;
+   virtual ~Block_AMSSolver();   
+private:
+   int nrmeshes = 1;
+   std::vector<ParFiniteElementSpace *> fespaces;
+   Block_AMS::BlkSmootherType sType=Block_AMS::BlkSmootherType::SCHWARZ; 
+
+   /// The linear system matrix
+   Array2D<HypreParMatrix* > A_array;
+   Array2D<HypreParMatrix* > Pi;
+   HypreParMatrix *Grad, *Pix, *Piy, *Piz;
+   HypreParMatrix *l1A00, *l1A11, *Ah;
+   Array<int> offsets, offsetsG, offsetsPi;
+   BlockOperator *GtAG, *PxtAPx, *PytAPy, *PztAPz;
+   BlockOperator  *A, *G, *Px, *Py, *Pz;
+   Operator * D;
+   HypreBoomerAMG *G00_inv, *Px00_inv, *Py00_inv, *Pz00_inv;
+   HypreBoomerAMG *G11_inv, *Px11_inv, *Py11_inv, *Pz11_inv;;
+   BlockDiagonalPreconditioner *blkAMG_G, *blkAMG_Px, *blkAMG_Py, *blkAMG_Pz;
+   double theta = 1.0;
+   string cycle_type = "023414320"; // 0-Smoother, 1-Grad, 2,3,4-Pix,Piy,Piz
+   HypreSmoother * Dh;
+   int NumberOfCycles=1;
+   
+   HypreParMatrix *hGtAG, *hPxtAPx, *hPytAPy, *hPztAPz;
+   HypreBoomerAMG *AMG_G, *AMG_Px, *AMG_Py, *AMG_Pz;
+   Array2D<HypreParMatrix * > hRAP_G;
+   Array2D<HypreParMatrix * > hRAP_Px;
+   Array2D<HypreParMatrix * > hRAP_Py;
+   Array2D<HypreParMatrix * > hRAP_Pz;
+
+
+   void DiagAddL1norm();
+   void Getrowl1norm(HypreParMatrix *A , Vector &l1norm);
+   void GetCorrection(BlockOperator* Tr, BlockOperator* op, BlockDiagonalPreconditioner *prec, Vector &r, Vector &z) const;
+   void GetCorrection(BlockOperator* Tr, HypreParMatrix* op, HypreBoomerAMG *prec, Vector &r, Vector &z) const;
+};
+
+HypreParMatrix* GetDiscreteGradientOp(ParFiniteElementSpace *fespace);
+Array2D<HypreParMatrix *> GetNDInterpolationOp(ParFiniteElementSpace *fespace);

examples/solvers-dev/as/blkschwarzp.cpp

@@ -0,0 +1,235 @@
+
+#include "mfem.hpp"
+#include "blkschwarzp.hpp"
+#include <iterator>
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+
+BlkParSchwarzSmoother::BlkParSchwarzSmoother(ParMesh * cpmesh_, int ref_levels_,
+ParFiniteElementSpace *fespace_, BlockOperator * bop_)
+: Solver(bop_->Height(),bop_->Width()), bop(bop_)         
+{
+   // construct and invert the patches
+   // This can be modified so only the last part of the assembly is repeated since all the 
+   // matrices have the same structure
+   P.SetSize(2,2);
+   blockA.SetSize(2,2);
+   for (int i=0; i<2; i++)
+   {
+      for (int j=0; j<2; j++)
+      {
+         blockA(i,j) = static_cast<HypreParMatrix *>(&bop->GetBlock(i,j));
+         P(i,j) = new par_patch_assembly(cpmesh_,ref_levels_, fespace_, blockA(i,j));
+      }
+   }
+
+   comm = fespace_->GetComm();
+   nrpatch = P(0,0)->nrpatch;
+   host_rank.SetSize(nrpatch);
+   host_rank = P(0,0)->host_rank;
+   PatchInv.SetSize(nrpatch);
+   PatchInvKLU.SetSize(nrpatch);
+   PatchMat.SetSize(nrpatch);
+   for (int ip = 0; ip < nrpatch; ++ip)
+   {
+      PatchInv[ip]=nullptr;
+      PatchInvKLU[ip] = nullptr;
+      PatchMat[ip]=nullptr;
+      if (P(0,0)->PatchMat[ip])
+      {
+         
+         Array<int>block_offsets(3);
+         block_offsets[0] = 0;
+         block_offsets[1] = P(0,0)->PatchMat[ip]->Height();
+         block_offsets[2] = block_offsets[1];
+         block_offsets.PartialSum();
+         BlockMatrix blockPatchMat(block_offsets);
+         for (int i=0; i<2; ++i)
+         {
+            for (int j=0; j<2; ++j)
+            {
+               blockPatchMat.SetBlock(i,j,P(i,j)->PatchMat[ip]);
+            }
+         }
+         PatchMat[ip] = blockPatchMat.CreateMonolithic();
+         for (int i=0; i<2; ++i)
+         {
+            for (int j=0; j<2; ++j)
+            {
+               delete P(i,j)->PatchMat[ip]; P(i,j)->PatchMat[ip] = nullptr;
+            }
+         }
+         // PatchInv[ip] = new UMFPackSolver;
+         // PatchInv[ip]->Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+         // PatchInv[ip]->SetOperator(*PatchMat[ip]);
+         // PatchInv[ip]->SetPrintLevel(2);
+         PatchInvKLU[ip] = new KLUSolver(*PatchMat[ip]);
+      }
+   }
+   R.SetSize(2,2);
+   R(0,0) = new PatchRestriction(P(0,0));
+   R(0,1) = nullptr;
+   R(1,0) = nullptr;
+   R(1,1) = new PatchRestriction(P(1,1));
+}
+
+
+void BlkParSchwarzSmoother::Mult(const Vector &r, Vector &z) const
+{
+   int num_procs, myid;
+   MPI_Comm_size(comm, &num_procs);
+   MPI_Comm_rank(comm, &myid);
+
+   z = 0.0; // initialize the correction to zero
+
+   Array<int> offsets(3);
+   offsets[0] = 0;
+   offsets[1] = r.Size()/2;
+   offsets[2] = r.Size()/2;
+   offsets.PartialSum();
+
+   double * data = r.GetData();
+   BlockVector rnew_test(data, offsets);
+   BlockVector rnew(rnew_test);
+   BlockVector znew(offsets); 
+   BlockVector raux(offsets);
+   BlockOperator blkA(offsets);
+   blkA.SetBlock(0,0,blockA(0,0));
+   blkA.SetBlock(1,0,blockA(1,0));
+   blkA.SetBlock(0,1,blockA(0,1));
+   blkA.SetBlock(1,1,blockA(1,1));
+
+   for (int iter = 0; iter < maxit; ++iter)
+   {
+      znew = 0.0;
+      Array<BlockVector * > res0;
+      Array<BlockVector * > res1;
+      Array<BlockVector * > res(nrpatch);
+      R(0,0)->Mult(rnew.GetBlock(0),res0);
+      R(1,1)->Mult(rnew.GetBlock(1),res1);
+
+      Array<BlockVector*> sol0(nrpatch);
+      Array<BlockVector*> sol1(nrpatch);
+      Array<BlockVector*> sol(nrpatch);
+
+      for (int ip=0; ip<nrpatch; ++ip)
+      {
+         if(myid == host_rank[ip]) 
+         {
+            Array<int> block_offs(3);
+            block_offs[0] = 0;
+            block_offs[1] = res0[ip]->Size();
+            block_offs[2] = res1[ip]->Size();
+            block_offs.PartialSum();   
+            res[ip] = new BlockVector(block_offs);
+            res[ip]->SetVector(*res0[ip],0);
+            res[ip]->SetVector(*res1[ip],res0[ip]->Size());
+           
+            Array<int> block_offs0(3);
+            block_offs0[0] = 0;
+            block_offs0[1] = res0[ip]->GetBlock(0).Size();
+            block_offs0[2] = res0[ip]->GetBlock(1).Size();
+            block_offs0.PartialSum();
+            Array<int> block_offs1(3);
+            block_offs1[0] = 0;
+            block_offs1[1] = res1[ip]->GetBlock(0).Size();
+            block_offs1[2] = res1[ip]->GetBlock(1).Size();
+            block_offs1.PartialSum();
+
+            sol[ip] = new BlockVector(block_offs); 
+            // PatchInv[ip]->Mult(*res[ip], *sol[ip]);
+            PatchInvKLU[ip]->Mult(*res[ip], *sol[ip]);
+            delete res[ip];
+            sol0[ip] = new BlockVector(sol[ip]->GetBlock(0).GetData(),block_offs0);
+            sol1[ip] = new BlockVector(sol[ip]->GetBlock(1).GetData(),block_offs1);
+         }
+      }
+      R(0,0)->MultTranspose(sol0,znew.GetBlock(0));
+      R(1,1)->MultTranspose(sol1,znew.GetBlock(1));
+      for (int ip=0; ip<nrpatch; ++ip)
+      {
+         if(myid == host_rank[ip]) 
+         {
+            delete res0[ip];
+            delete res1[ip];
+            delete sol0[ip];
+            delete sol1[ip];
+            delete sol[ip];
+         }
+      }
+      znew *= theta;
+      z += znew;
+      blkA.Mult(znew,raux); 
+      rnew -= raux;
+   }
+}
+BlkParSchwarzSmoother::~BlkParSchwarzSmoother()
+{
+   for (int i=0; i<2; i++)
+   {
+      for (int j=0; j<2; j++)
+      {
+         if (P(i,j)) delete P(i,j); 
+         if (R(i,j)) delete R(i,j);
+      }   
+   }
+   P.DeleteAll();
+   R.DeleteAll();
+   // estimate of KLU solver memory
+   // double gb1 = 0.0;
+   // double gb2 = 0.0;
+   // double gb3 = 0.0;
+
+   // double UMFgb1 = 0.0;
+   // double UMFgb2 = 0.0;
+   // double UMFgb3 = 0.0;
+   // mfem::out << "nrpatch = " << nrpatch << endl;
+   // for (int ip=0; ip<nrpatch; ++ip)
+   // {
+   //    if (PatchMat[ip]) 
+   //    {
+   //       int nnz1 = PatchMat[ip]->NumNonZeroElems();
+   //       int n = PatchMat[ip]->Height();
+   //       gb1 += nnz1 * 12.0 + (n+1.0)*4.0;
+   //    }
+   //    if (PatchInv[ip])
+   //    {
+   //       int units = PatchInv[ip]->Info[UMFPACK_SIZE_OF_UNIT];
+   //       int SymGB = PatchInv[ip]->Info[UMFPACK_SYMBOLIC_PEAK_MEMORY];
+   //       int NumGB = PatchInv[ip]->Info[UMFPACK_NUMERIC_SIZE_ESTIMATE];
+   //       int TotGB = PatchInv[ip]->Info[UMFPACK_PEAK_MEMORY_ESTIMATE];
+
+   //       UMFgb1 += SymGB*units;
+   //       UMFgb2 += NumGB*units;
+   //       UMFgb3 += TotGB*units;
+   //       // int nnz2 = pow(PatchInv[ip]->Height(),4.0/3.0);
+   //       // int n = PatchInv[ip]->Height();
+   //       // gb2 += nnz2 * 12.0 + (n+1.0)*4.0;
+   //    } 
+   //    if (PatchInvKLU[ip])
+   //    {
+   //       int nnz3 = pow(PatchInvKLU[ip]->Height(),4.0/3.0);
+   //       int n = PatchInvKLU[ip]->Height();
+   //       gb3 += nnz3 * 12.0 + (n+1.0)*4.0;
+   //    }  
+   // }
+   for (int ip=0; ip<nrpatch; ++ip)
+   {
+       if (PatchMat[ip])  delete PatchMat[ip];
+       if (PatchInv[ip])  delete PatchInv[ip];
+       if (PatchInvKLU[ip])  delete PatchInvKLU[ip];
+   }
+   PatchMat.DeleteAll();
+   PatchInv.DeleteAll();
+   PatchInvKLU.DeleteAll();
+
+   // mfem::out << "Total storage for PatchMat       : " << gb1/pow(1024.0,3) << " GB " << endl;
+   // mfem::out << "Symbolic storage for PatchInvUMF : " << UMFgb1/pow(1024.0,3) << " GB " << endl;
+   // mfem::out << "Numeric storage for PatchInvUMF  : " << UMFgb2/pow(1024.0,3) << " GB " << endl;
+   // mfem::out << "Total storage for PatchInvUMF    : " << UMFgb3/pow(1024.0,3) << " GB " << endl;
+   // mfem::out << "Total storage for PatchInvKLU: " << gb3/pow(1024.0,3) << " GB " << endl;
+}

examples/solvers-dev/as/blkschwarzp.hpp

@@ -0,0 +1,40 @@
+
+#pragma once
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include "schwarzp.hpp"
+using namespace std;
+using namespace mfem;
+
+
+class BlkParSchwarzSmoother : virtual public Solver {
+private:
+   MPI_Comm comm;
+   int nrpatch;
+   int maxit = 1;
+   double theta = 1.0;
+   Array<int> host_rank;
+   // Array<PetscLinearSolver * > PatchInv;
+   Array<UMFPackSolver * > PatchInv;
+   Array<KLUSolver * > PatchInvKLU;
+   Array<SparseMatrix * > PatchMat;
+   /// The linear system matrix
+   Array2D<HypreParMatrix*> blockA;
+   BlockOperator * bop;
+   Array2D<PatchRestriction *> R;
+   Array2D<par_patch_assembly *> P;
+   BlockOperator * blk = nullptr;
+public:
+   // BlkParSchwarzSmoother(ParMesh * cpmesh_, int ref_levels_,ParFiniteElementSpace *fespace_,
+   //                       Array2D<HypreParMatrix*> blockA_);
+   BlkParSchwarzSmoother(ParMesh * cpmesh_, int ref_levels_,ParFiniteElementSpace *fespace_,
+                         BlockOperator * bop_);                      
+   void SetNumSmoothSteps(const int iter) {maxit = iter;}
+   void SetDumpingParam(const double dump_param) {theta = dump_param;}
+  
+   virtual void SetOperator(const Operator &op) {}
+   virtual void Mult(const Vector &r, Vector &z) const;
+   virtual ~BlkParSchwarzSmoother();
+};

examples/solvers-dev/as/schwarz.cpp

@@ -0,0 +1,509 @@
+
+#include "mfem.hpp"
+#include "schwarz.hpp"
+
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+
+void print(std::vector<int> const &input)
+{
+	for (int i = 0; i < (int)input.size(); i++) {
+		std::cout << input.at(i) << ' ';
+	}
+}
+
+// constructor
+patch_nod_info::patch_nod_info(Mesh *mesh_, int ref_levels_)
+    : mesh(mesh_), ref_levels(ref_levels_)
+{
+   /* The patches are defined by all the "active" vertices of the coarse mesh
+   We define a low order H1 fespace and perform refinements so that we can get
+   the H1 prolongation operator recursively. This way we can easily find  
+   all the patches that the fine mesh vertices contribute to. After the vertices 
+   are done the edges, faces and elements can be found easily because they
+   contribute to the same patches as their vertices. */
+
+   // Number of patches
+   nrpatch = mesh->GetNV();
+   int dim = mesh->Dimension();
+   FiniteElementCollection *fec = new H1_FECollection(1, dim);
+   FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
+
+   // First we need to construct a list of non-essential coarse grid vertices
+
+   // SparseMatrix *Pr = nullptr;
+   //initialize Pr with the Identity
+   Vector ones(fespace->GetTrueVSize());
+   ones = 1.0;
+   SparseMatrix * Pr = new SparseMatrix(ones);
+   // 4. Refine the mesh
+   for (int i = 0; i < ref_levels; i++)
+   {
+      const FiniteElementSpace cfespace(*fespace);
+      mesh->UniformRefinement();
+      // Update fespace
+      fespace->Update();
+      OperatorHandle Tr(Operator::MFEM_SPARSEMAT);
+      fespace->GetTransferOperator(cfespace, Tr);
+      Tr.SetOperatorOwner(false);
+      SparseMatrix *P;
+      Tr.Get(P);
+      if (!Pr)
+      {
+         Pr = P;
+      }
+      else
+      {
+         Pr = Mult(*P, *Pr);
+      }
+   }
+   // if there is no refinement the prolongation is the identity
+   Pr->Threshold(0.0);
+   int nvert = mesh->GetNV();
+   vertex_contr.resize(nvert);
+   for (int iv = 0; iv < nvert; iv++)
+   {
+      int nz = Pr->RowSize(iv);
+      vertex_contr[iv].SetSize(nz);
+      int *col = Pr->GetRowColumns(iv);
+      for (int i = 0; i < nz; i++)
+      {
+         vertex_contr[iv][i] = col[i];
+      }
+   }
+
+   delete Pr;
+
+   Array<int> edge_vertices;
+   int nedge = mesh->GetNEdges();
+   edge_contr.resize(nedge);
+   for (int ie = 0; ie < nedge; ie++)
+   {
+      mesh->GetEdgeVertices(ie, edge_vertices);
+      int nv = edge_vertices.Size(); // always 2 but ok
+      // The edge will contribute to the same patches as its vertices
+      for (int iv = 0; iv < nv; iv++)
+      {
+         int ivert = edge_vertices[iv];
+         edge_contr[ie].Append(vertex_contr[ivert]);
+      }
+      edge_contr[ie].Sort();
+      edge_contr[ie].Unique();
+   }
+
+   Array<int> face_vertices;
+   int nface = mesh->GetNFaces();
+   face_contr.resize(nface);
+   for (int ifc = 0; ifc < nface; ifc++)
+   {
+      mesh->GetFaceVertices(ifc, face_vertices);
+      int nv = face_vertices.Size();
+      // The face will contribute to the same patches as its vertices
+      for (int iv = 0; iv < nv; iv++)
+      {
+         int ivert = face_vertices[iv];
+         face_contr[ifc].Append(vertex_contr[ivert]);
+      }
+      face_contr[ifc].Sort();
+      face_contr[ifc].Unique();
+   }
+
+   Array<int> elem_vertices;
+   int nelem = mesh->GetNE();
+   elem_contr.resize(nelem);
+   for (int iel = 0; iel < nelem; iel++)
+   {
+      mesh->GetElementVertices(iel, elem_vertices);
+      int nv = elem_vertices.Size();
+      // The element will contribute to the same patches as its vertices
+      for (int iv = 0; iv < nv; iv++)
+      {
+         int ivert = elem_vertices[iv];
+         elem_contr[iel].Append(vertex_contr[ivert]);
+      }
+      elem_contr[iel].Sort();
+      elem_contr[iel].Unique();
+   }
+   delete fespace;
+   delete fec;
+}
+// Constructor of patch local problems
+patch_assembly::patch_assembly(Mesh *cmesh_, int ref_levels_, FiniteElementSpace *fespace)
+    : cmesh(*cmesh_), ref_levels(ref_levels_)
+{
+   patch_nod_info *patches = new patch_nod_info(&cmesh, ref_levels);
+
+   nrpatch = patches->nrpatch;
+   Pid.SetSize(nrpatch);
+   patch_dof_map.SetSize(nrpatch);
+   // Build a sparse matrix out of this map to extract the patch submatrix
+   Array<int> dofoffset(nrpatch);
+   dofoffset = 0;
+   int height = fespace->GetVSize();
+   // allocation of sparse matrices.
+   for (int i = 0; i < nrpatch; i++)
+   {
+      Pid[i] = new SparseMatrix(height);
+   }
+   // Now the filling of the matrices with vertex,edge,face,interior dofs
+   Mesh *mesh = fespace->GetMesh();
+   int nrvert = mesh->GetNV();
+   int nredge = mesh->GetNEdges();
+   int nrface = mesh->GetNFaces();
+   int nrelem = mesh->GetNE();
+   // First the vertices
+   for (int i = 0; i < nrvert; i++)
+   {
+      int np = patches->vertex_contr[i].Size();
+      Array<int> vertex_dofs;
+      fespace->GetVertexDofs(i, vertex_dofs);
+      int nv = vertex_dofs.Size();
+
+      for (int j = 0; j < np; j++)
+      {
+         int k = patches->vertex_contr[i][j];
+         for (int l = 0; l < nv; l++)
+         {
+            int m = vertex_dofs[l];
+            Pid[k]->Set(m, dofoffset[k], 1.0);
+            dofoffset[k]++;
+         }
+      }
+   }
+   // Edges
+   for (int i = 0; i < nredge; i++)
+   {
+      int np = patches->edge_contr[i].Size();
+      Array<int> edge_dofs;
+      fespace->GetEdgeInteriorDofs(i, edge_dofs);
+      int ne = edge_dofs.Size();
+      for (int j = 0; j < np; j++)
+      {
+         int k = patches->edge_contr[i][j];
+         for (int l = 0; l < ne; l++)
+         {
+            int m = edge_dofs[l];
+            Pid[k]->Set(m, dofoffset[k], 1.0);
+            dofoffset[k]++;
+         }
+      }
+   }
+   // Faces
+   for (int i = 0; i < nrface; i++)
+   {
+      int np = patches->face_contr[i].Size();
+      Array<int> face_dofs;
+      fespace->GetFaceInteriorDofs(i, face_dofs);
+      int nfc = face_dofs.Size();
+      for (int j = 0; j < np; j++)
+      {
+         int k = patches->face_contr[i][j];
+         for (int l = 0; l < nfc; l++)
+         {
+            int m = face_dofs[l];
+            Pid[k]->Set(m, dofoffset[k], 1.0);
+            dofoffset[k]++;
+         }
+      }
+   }
+
+   // The following can be skipped in case of static condensation
+   // Elements
+   for (int i = 0; i < nrelem; i++)
+   {
+      int np = patches->elem_contr[i].Size();
+      Array<int> elem_dofs;
+      fespace->GetElementInteriorDofs(i, elem_dofs);
+      int nel = elem_dofs.Size();
+      for (int j = 0; j < np; j++)
+      {
+         int k = patches->elem_contr[i][j];
+         for (int l = 0; l < nel; l++)
+         {
+            int m = elem_dofs[l];
+            Pid[k]->Set(m, dofoffset[k], 1.0);
+            dofoffset[k]++;
+         }
+      }
+   }
+
+   for (int i = 0; i < nrpatch; i++)
+   {
+      Pid[i]->SetWidth(dofoffset[i]);
+      Pid[i]->Finalize();
+
+      patch_dof_map[i].SetSize(Pid[i]->Width());
+      // copy from sparse matrix to a simple injection map
+      // use the traspose
+      SparseMatrix * temp = Transpose(*Pid[i]);
+      // Extract row by row of the transpose
+      for (int k =0; k<temp->Height(); ++k)
+      {
+         int * col = temp->GetRowColumns(k);
+         patch_dof_map[i][k] = col[0];
+      }
+      delete temp;
+   }
+   delete patches;
+}
+
+patch_assembly:: ~patch_assembly()
+{
+   for (int i=0; i<nrpatch; i++)
+   {
+      delete Pid[i];
+   }
+   Pid.DeleteAll();
+}
+
+// constructor
+SchwarzSmoother::SchwarzSmoother(Mesh *cmesh_, int ref_levels_, FiniteElementSpace *fespace_, SparseMatrix *A_, Array<int> ess_bdr)
+    : Solver(A_->Height(), A_->Width()), A(A_)
+{
+   P = new patch_assembly(cmesh_, ref_levels_, fespace_);
+
+   ess_bdr = 0;
+   GetNonEssentialPatches(cmesh_, ess_bdr, patch_ids);
+
+   // nrpatch = P->nrpatch;
+   nrpatch = patch_ids.size();
+   A_local.SetSize(nrpatch);
+   invA_local.SetSize(nrpatch);
+
+   for (int i = 0; i < nrpatch; i++)
+   {
+      int k = patch_ids[i];
+      SparseMatrix *Pr = P->Pid[k];
+      // construct the local problems. Factor the patch matrices
+      A_local[i] = RAP(*Pr, *A, *Pr);
+      // if (i == 0) A_local[i]->PrintMatlab(cout);
+      invA_local[i] = new KLUSolver;
+      // invA_local[i] = new UMFPackSolver;
+      // invA_local[i]->Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+      invA_local[i]->SetOperator(*A_local[i]);
+   }
+}
+
+void SchwarzSmoother::GetNonEssentialPatches(Mesh *cmesh, const Array<int> &ess_bdr, vector<int> &patch_ids)
+{
+   Array<int> ess_vertices;
+   Array<int> bdr_vertices;
+
+   for (int i = 0; i < cmesh->GetNBE(); i++)
+   {
+      int bdr = cmesh->GetBdrAttribute(i);
+      //check if it's essential;
+      if (ess_bdr[bdr - 1] == 1)
+      {
+         cmesh->GetBdrElementVertices(i, bdr_vertices);
+         ess_vertices.Append(bdr_vertices);
+      }
+   }
+   ess_vertices.Sort();
+   ess_vertices.Unique();
+
+   int nrpatch = cmesh->GetNV() - ess_vertices.Size();
+   patch_ids.resize(nrpatch);
+
+   if (ess_vertices.Size() > 0)
+   {
+      int m = 0;
+      int l = 0;
+      for (int i = 0; i < cmesh->GetNV(); i++)
+      {
+         if (m<ess_vertices.Size() && i == ess_vertices[m])
+         {
+            m++;
+         }
+         else
+         {
+            patch_ids[l] = i;
+            l++;
+         }
+      }
+   }
+   else
+   {
+      for (int i = 0; i < cmesh->GetNV(); i++)
+      {
+         patch_ids[i] = i;
+      }
+   }
+}
+
+void SchwarzSmoother::Mult(const Vector &r, Vector &z) const
+{
+   // Apply the smoother patch on the restriction of the residual
+   z = 0.0;
+   Vector rnew(r);
+   Vector znew(z);
+   Vector raux(znew.Size());
+   Vector res_local, sol_local;
+   switch (sType)
+   {
+   case Schwarz::SmootherType::ADDITIVE:
+   {
+      for (int iter = 0; iter < maxit; iter++)
+      {
+         znew = 0.0;
+         for (int i = 0; i < nrpatch; i++)
+         {
+            int k = patch_ids[i];
+            Array<int> * dof_map = &P->patch_dof_map[k];
+            // SparseMatrix *Pr = P->Pid[k];
+            // res_local.SetSize(Pr->NumCols());
+            // sol_local.SetSize(Pr->NumCols());
+            // Pr->MultTranspose(rnew, res_local[i]);
+            int ndofs = dof_map->Size();
+            res_local.SetSize(ndofs);
+            sol_local.SetSize(ndofs);
+            rnew.GetSubVector(*dof_map, res_local);
+
+            invA_local[i]->Mult(res_local, sol_local);
+            znew.AddElementVector(*dof_map,sol_local);
+            // Pr->Mult(sol_local[i], zaux[i]);
+            // znew += zaux[i];
+         }
+         // Relaxation parameter
+         znew *= theta;
+         z += znew;
+
+         //Update residual
+         if (iter + 1 < maxit)
+         {
+            A->Mult(znew, raux);
+            rnew -= raux;
+         }
+      }
+   }
+   break;
+   case Schwarz::SmootherType::MULTIPLICATIVE:
+   {
+      //   TODO
+   }
+   break;
+   case Schwarz::SmootherType::SYM_MULTIPLICATIVE:
+   {
+      //   TODO
+   }
+   break;
+   }
+}
+
+SchwarzSmoother:: ~SchwarzSmoother() 
+{
+   delete P;
+   for (int ip=0; ip<nrpatch; ++ip)
+   {
+      delete A_local[ip]; 
+      delete invA_local[ip]; 
+   }
+   A_local.DeleteAll();
+   invA_local.DeleteAll();
+}
+
+
+
+BlkSchwarzSmoother::BlkSchwarzSmoother(Mesh *cmesh_, int ref_levels_, FiniteElementSpace* fespace_, SparseMatrix *A_) 
+: Solver(A_->Height(), A_->Width()), A(A_)
+{
+   P = new patch_assembly(cmesh_, ref_levels_, fespace_);
+
+   nrpatch = cmesh_->GetNV();
+   patch_ids.resize(nrpatch);
+   for (int i=0; i<nrpatch; i++) {patch_ids[i]=i;}
+
+   nrpatch = patch_ids.size();
+   A_local.SetSize(nrpatch);
+   invA_local.SetSize(nrpatch);
+
+   for (int i = 0; i < nrpatch; i++)
+   {
+      int k = patch_ids[i];
+      SparseMatrix *Pr  = P->Pid[k];
+      Array<int> offsets_i(3);
+      Array<int> offsets_j(3);
+      offsets_i[0] = 0; 
+      offsets_i[1] = Pr->Height();
+      offsets_i[2] = Pr->Height();
+      offsets_i.PartialSum();
+      offsets_j[0] = 0; 
+      offsets_j[1] = Pr->Width();
+      offsets_j[2] = Pr->Width();
+      offsets_j.PartialSum();
+
+      BlockMatrix * BlockPr = new BlockMatrix(offsets_i,offsets_j);
+      BlockPr->SetBlock(0,0,Pr);
+      BlockPr->SetBlock(1,1,Pr);
+      // Fake blocks
+      SparseMatrix * fakemat = new SparseMatrix(Pr->Height(),Pr->Width()); fakemat->Finalize();
+      BlockPr->SetBlock(0,1,fakemat);
+      BlockPr->SetBlock(1,0,fakemat);
+
+      SparseMatrix *Bpr = BlockPr->CreateMonolithic();
+      A_local[i] = RAP(*Bpr, *A, *Bpr);
+      invA_local[i] = new UMFPackSolver;
+      invA_local[i]->Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+      invA_local[i]->SetOperator(*A_local[i]);
+   }
+}
+
+
+void BlkSchwarzSmoother::Mult(const Vector &r, Vector &z) const
+{
+   // Apply the smoother patch on the restriction of the residual
+   Array<Vector> res_local(nrpatch);
+   Array<Vector> sol_local(nrpatch);
+   Array<Vector> zaux(nrpatch);
+   z = 0.0;
+   Vector rnew(r);
+   Vector znew(z);
+
+   for (int iter = 0; iter < maxit; iter++)
+   {
+      znew = 0.0;
+      for (int i = 0; i < nrpatch; i++)
+      {
+         int k = patch_ids[i];
+         SparseMatrix *Pr = P->Pid[k];
+         Array<int> offsets_i(3);
+         Array<int> offsets_j(3);
+         offsets_i[0] = 0; 
+         offsets_i[1] = Pr->Height();
+         offsets_i[2] = Pr->Height();
+         offsets_i.PartialSum();
+         offsets_j[0] = 0; 
+         offsets_j[1] = Pr->Width();
+         offsets_j[2] = Pr->Width();
+         offsets_j.PartialSum();
+         BlockMatrix * BlockPr = new BlockMatrix(offsets_i,offsets_j);
+         BlockPr->SetBlock(0,0,Pr);
+         BlockPr->SetBlock(1,1,Pr);
+         SparseMatrix * fakemat = new SparseMatrix(Pr->Height(),Pr->Width()); fakemat->Finalize();
+         BlockPr->SetBlock(0,1,fakemat);
+         BlockPr->SetBlock(1,0,fakemat);
+         SparseMatrix *Bpr = BlockPr->CreateMonolithic();
+         res_local[i].SetSize(Bpr->NumCols());
+         sol_local[i].SetSize(Bpr->NumCols());
+         Bpr->MultTranspose(rnew, res_local[i]);
+
+         invA_local[i]->Mult(res_local[i], sol_local[i]);
+         zaux[i].SetSize(r.Size());
+         zaux[i] = 0.0;
+         Bpr->Mult(sol_local[i], zaux[i]);
+         znew += zaux[i];
+      }
+
+      // Relaxation parameter
+      znew *= theta;
+      z += znew;
+      //Update residual
+      Vector raux(znew.Size());
+      A->Mult(znew, raux);
+      rnew -= raux;
+   }
+}

examples/solvers-dev/as/schwarz.hpp

@@ -0,0 +1,92 @@
+#include "mfem.hpp"
+#include "../mg/util.hpp"
+
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+namespace Schwarz
+{
+   enum SmootherType{ADDITIVE, MULTIPLICATIVE, SYM_MULTIPLICATIVE};
+}
+
+struct patch_nod_info 
+{
+   int nrpatch;
+   vector<Array<int>> vertex_contr;
+   vector<Array<int>> edge_contr;
+   vector<Array<int>> face_contr;
+   vector<Array<int>> elem_contr;
+
+   // constructor
+   patch_nod_info(Mesh * mesh_, int ref_levels_);
+private:
+   Mesh *mesh=nullptr;
+   int ref_levels=0;;
+};
+
+struct patch_assembly 
+{
+   int nrpatch;
+   Mesh cmesh;
+   int ref_levels;
+   Array<SparseMatrix *> Pid; 
+   Array<Array<int>> patch_dof_map;
+   // constructor
+   patch_assembly(Mesh * cmesh_, int ref_levels_,FiniteElementSpace *fespace);
+   ~patch_assembly();
+};
+
+class SchwarzSmoother : virtual public Solver {
+private:
+   int nrpatch;
+   /// The linear system matrix
+   SparseMatrix * A;
+   patch_assembly * P;
+   Array<SparseMatrix  *> A_local;
+   // Array<UMFPackSolver *> invA_local;
+   Array<KLUSolver *> invA_local;
+   Array<int>vert_dofs;
+   Schwarz::SmootherType sType=Schwarz::SmootherType::ADDITIVE; 
+   vector<int> patch_ids;
+   int maxit = 1;
+   double theta = 0.5;
+public:
+   SchwarzSmoother(Mesh * cmesh_, int ref_levels_, FiniteElementSpace *fespace,SparseMatrix *A_, Array<int> ess_bdr);
+
+   void SetType(const Schwarz::SmootherType Type) {sType = Type;}
+   void SetNumSmoothSteps(const int iter) {maxit = iter;}
+   void SetDumpingParam(const double dump_param) {theta = dump_param;}
+   virtual void SetOperator(const Operator &op) {}
+   virtual void Mult(const Vector &r, Vector &z) const;
+   void GetNonEssentialPatches(Mesh * cmesh, const Array<int> &ess_bdr, vector <int> & patch_ids);
+   virtual ~SchwarzSmoother();
+};
+
+
+class BlkSchwarzSmoother : public Solver {
+private:
+   int nrpatch;
+   /// The linear system matrix
+   SparseMatrix * A;
+   patch_assembly * P;
+   Array<SparseMatrix  *> A_local;
+   Array<UMFPackSolver *> invA_local;
+   Array<int>vert_dofs;
+   Schwarz::SmootherType sType=Schwarz::SmootherType::ADDITIVE; 
+   vector<int> patch_ids;
+   int maxit = 1;
+   double theta = 0.5;
+public:
+   BlkSchwarzSmoother(Mesh *cmesh_, int ref_levels_, FiniteElementSpace *fespace_, SparseMatrix *A_);
+   void SetType(const Schwarz::SmootherType Type) {sType = Type;}
+   void SetNumSmoothSteps(const int iter) {maxit = iter;}
+   void SetDumpingParam(const double dump_param) {theta = dump_param;}
+   virtual void SetOperator(const Operator &op) {}
+   virtual void Mult(const Vector &r, Vector &z) const;
+   // void GetNonEssentialPatches(Mesh * cmesh, const Array<int> &ess_bdr, vector <int> & patch_ids);
+   virtual ~BlkSchwarzSmoother() {}
+};
+

examples/solvers-dev/as/schwarzp.hpp

@@ -0,0 +1,138 @@
+#pragma once
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+
+class par_patch_nod_info 
+{
+public:
+   int mynrpatch;
+   int nrpatch;
+   vector<Array<int>> vert_contr;
+   vector<Array<int>> edge_contr;
+   vector<Array<int>> face_contr;
+   vector<Array<int>> elem_contr;
+   Array<int> host_rank;
+   Array<int> patch_natural_order_idx;
+   Array<int> patch_global_dofs_ids;
+   // constructor
+   par_patch_nod_info(ParMesh * cpmesh_, int ref_levels_);
+   // Print
+   void Print(int rank_id);
+   ~par_patch_nod_info() {delete aux_fespace;}
+private:
+   ParMesh pmesh;
+   int ref_levels=0;
+   ParFiniteElementSpace *aux_fespace=nullptr;
+};
+
+class par_patch_dof_info 
+{
+public:
+   MPI_Comm comm = MPI_COMM_WORLD;
+   int nrpatch;
+   Array<int> host_rank;
+   vector<Array<int>> patch_tdofs;
+   vector<Array<int>> patch_local_tdofs;
+   // constructor
+   par_patch_dof_info(ParMesh * cpmesh_, int ref_levels_,ParFiniteElementSpace *fespace);
+   void Print();
+   ~par_patch_dof_info(){};
+};
+
+class par_patch_assembly
+{
+public:
+   MPI_Comm comm;
+   int nrpatch;
+   vector<int>tdof_offsets;
+   vector<Array<int>> patch_other_tdofs;
+   vector<Array<int>> patch_owned_other_tdofs;
+   vector<Array<int>> l2gmaps; // patch to global maps for the dofs owned by the processor
+   Array<SparseMatrix* > PatchMat;
+   par_patch_dof_info *patch_tdof_info=nullptr;   
+   Array<int> host_rank; 
+   HypreParMatrix * A = nullptr;
+   int get_rank(int tdof);
+   // constructor
+   par_patch_assembly(ParMesh * cpmesh_, int ref_levels_,ParFiniteElementSpace *fespace_, HypreParMatrix * A_);
+   ~par_patch_assembly();
+private:
+   void compute_trueoffsets();
+   ParFiniteElementSpace *fespace=nullptr;
+};
+
+class PatchRestriction  {
+private:
+   MPI_Comm comm;
+   int num_procs, myid;
+   Array<int> host_rank;
+   par_patch_assembly * P;
+   int nrpatch;
+   Array<int> send_count;
+   Array<int> send_displ;
+   Array<int> recv_count;
+   Array<int> recv_displ;
+   int sbuff_size;
+   int rbuff_size;
+public:
+   PatchRestriction(par_patch_assembly * P_);
+   void Mult(const Vector & r , Array<BlockVector *> & res);
+   void MultTranspose(const Array<BlockVector *> & sol, Vector & z);
+   virtual ~PatchRestriction() {}
+};
+
+class ParSchwarzSmoother : public Solver {
+private:
+   MPI_Comm comm;
+   int nrpatch;
+   int maxit = 1;
+   double theta = 1.0;
+   Array<int> host_rank;
+   Array<UMFPackSolver * > PatchInv;
+   Array<KLUSolver * > PatchInvKLU;
+   /// The linear system matrix
+   HypreParMatrix * A;
+   par_patch_assembly * P;
+   PatchRestriction * R= nullptr;
+public:
+   ParSchwarzSmoother(ParMesh * cpmesh_, int ref_levels_,ParFiniteElementSpace *fespace_, HypreParMatrix * A_);
+   void SetNumSmoothSteps(const int iter) {maxit = iter;}
+   void SetDumpingParam(const double dump_param) {theta = dump_param;}
+   virtual void SetOperator(const Operator &op) {}
+   virtual void Mult(const Vector &r, Vector &z) const;
+   virtual ~ParSchwarzSmoother();
+};
+
+
+
+bool its_a_patch(int iv, Array<int> patch_ids);
+
+bool owned(int tdof, int * offs);
+
+SparseMatrix * GetLocalRestriction(const Array<int> & tdof_i, const int * row_start, 
+                              const int num_rows, const int num_cols);                              
+
+void GetLocal2GlobalMap(const Array<int> & tdof_i, const int * row_start, 
+                              const int num_rows, const int num_cols, Array<int> & l2gmap);
+
+void GetOffdColumnValues(const Array<int> & tdof_i, const Array<int> & tdof_j, SparseMatrix & offd, const int * cmap, 
+                         const int * row_start, SparseMatrix * PatchMat);
+
+void GetArrayIntersection(const Array<int> & A, const Array<int> & B, Array<int>  & C); 
+
+
+
+int GetNumColumns(const int tdof_i, const Array<int> & tdof_j, SparseMatrix & diag,
+SparseMatrix & offd, const int * cmap, const int * row_start);
+
+void GetColumnValues(const int tdof_i,const Array<int> & tdof_j, SparseMatrix & diag ,
+SparseMatrix & offd, const int *cmap, const int * row_start, Array<int> &cols, Array<double> &vals);
+
+
+// SparseMatrix * GetLocalProlongation(const Array<int> & tdof_i, const int * row_start, 
+                              // const int num_rows, const int num_cols);

examples/solvers-dev/ex23p.cpp

@@ -0,0 +1,792 @@
+//                       MFEM Example 22 - Parallel Version
+//
+// Compile with: make ex22p
+//
+// Sample runs:  mpirun -np 4 ex22p -m ../../data/square-disc.mesh -o 2
+//               mpirun -np 4 ex22p -m ../../data/beam-tet.mesh
+//               mpirun -np 4 ex22p -m ../../data/beam-hex.mesh
+//               mpirun -np 4 ex22p -m ../../data/fichera.mesh
+//               mpirun -np 4 ex22p -m ../../data/amr-quad.mesh -o 2
+//               mpirun -np 4 ex22p -m ../../data/amr-hex.mesh
+//               mpirun -np 4 ex22p -m ../../hexa728.mesh
+//               mpirun -np 4 ex22p -m ../../data/rectwhole7_2attr.e
+// Description:  This example code solves a simple electromagnetic wave
+//               propagation problem corresponding to the second order indefinite
+//               Maxwell equation curl curl E - \omega^2 E = f with a PML
+//               We discretize with Nedelec finite elements in 2D or 3D.
+//
+//               The example also demonstrates the use complex valued bilear and linear forms.
+//               We recommend viewing examples 22 before viewing this example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include <boost/math/special_functions/hankel.hpp>
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
+using namespace std;
+using namespace mfem;
+using namespace boost;
+
+#ifndef MFEM_USE_SUPERLU
+#error This example requires that MFEM is built with MFEM_USE_SUPERLU=YES
+#endif
+
+// Exact solution, E, and r.h.s., f. See below for implementation.
+void compute_pml_mesh_data(Mesh * mesh);
+void maxwell_ess_data(const Vector &x, std::vector<std::complex<double>> &Eval);
+void E_bdr_data_Re(const Vector &x, Vector &E);
+void E_bdr_data_Im(const Vector &x, Vector &E);
+double pml_detJ_inv_Re(const Vector &x);
+double pml_detJ_inv_Im(const Vector &x);
+void pml_detJ_JT_J_inv_Re(const Vector &x, DenseMatrix &M);
+void pml_detJ_JT_J_inv_Im(const Vector &x, DenseMatrix &M);
+void pml_detJ_inv_JT_J_Re(const Vector &x, DenseMatrix &M);
+void pml_detJ_inv_JT_J_Im(const Vector &x, DenseMatrix &M);
+void compute_pml_elem_list(ParMesh * pmesh, Array<int> & elem_pml);
+
+
+double omega;
+int dim;
+int src = 2;
+Array2D<double> domain_bdr;
+Array2D<double> pml_lngth;
+Array2D<double> comp_domain_bdr;
+
+enum prob_type 
+{
+   scatter,
+   waveguide,
+};
+prob_type prob = scatter;
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   const char *mesh_file = "../../data/beam-tet.mesh";
+   int order = 1;
+   bool visualization = 1;
+   double freq = 1.0;
+   int ref_levels = 1;
+   int par_ref_levels = 1;
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree).");
+   args.AddOption(&freq, "-f", "--frequency", "Set the frequency for the exact"
+                                              " solution.");
+   args.AddOption(&ref_levels, "-rs", "--refinements-serial", "Number of serial refinements");                                           
+   args.AddOption(&par_ref_levels, "-rp", "--refinements-parallel", "Number of parallel refinements");                                           
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.AddOption(&src, "-src", "--source wave", "Source wave flag -" 
+                  "1: plane wave, 2: Point source, 3: sin in x direction");
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+   omega = 2.0 * M_PI * freq;
+
+   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
+   //    and volume meshes with the same code.
+
+   Mesh *mesh;
+   prob = scatter;
+   // prob = waveguide;
+
+   if (prob == scatter)
+   {
+      // mesh_file = "meshes/rectwhole7_2attr.e";
+      mesh_file = "meshes/hexa728.mesh";
+      mesh = new Mesh(mesh_file,1,1);
+      src = 3;
+   }
+   if (prob == waveguide)
+   {
+      mesh = new Mesh(1, 1, 8, Element::HEXAHEDRON, true, 1, 1, 8, false);
+      src = 1;
+   }
+   
+   dim = mesh->Dimension();
+   compute_pml_mesh_data(mesh);
+
+   int sdim = mesh->SpaceDimension();
+
+   // 4. Refine the serial mesh on all processors to increase the resolution. In
+   //    this example we do 'ref_levels' of uniform refinement. We choose
+   //    'ref_levels' to be the largest number that gives a final mesh with no
+   //    more than 1,000 elements.
+   {
+         //  (int)floor(log(1000. / mesh->GetNE()) / log(2.) / dim);
+      for (int l = 0; l < ref_levels; l++)
+      {
+         mesh->UniformRefinement();
+      }
+   }
+
+   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted. Tetrahedral
+   //    meshes need to be reoriented before we can define high-order Nedelec
+   //    spaces on them.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   {
+      for (int l = 0; l < par_ref_levels; l++)
+      {
+         pmesh->UniformRefinement();
+      }
+   }
+
+   Array<int> elems_pml;
+   compute_pml_elem_list(pmesh, elems_pml);
+   // pmesh->ReorientTetMesh();
+
+   // 6. Define a parallel finite element space on the parallel mesh. Here we
+   //    use the Nedelec finite elements of the specified order.
+   FiniteElementCollection *fec = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   HYPRE_Int size = fespace->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of finite element unknowns: " << size << endl;
+   }
+
+   // 7. Determine the list of true (i.e. parallel conforming) essential
+   //    boundary dofs. In this example, the boundary conditions are defined
+   //    by marking all the boundary attributes from the mesh as essential
+   //    (Dirichlet) and converting them to a list of true dofs.
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr;
+   if (pmesh->bdr_attributes.Size())
+   {
+      ess_bdr.SetSize(pmesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   // 8. Set up the parallel linear form b(.) which corresponds to the
+   //    right-hand side of the FEM linear system, which in this case is
+   //    (f,phi_i) where f is given by the function f_exact and phi_i are the
+   //    basis functions in the finite element fespace.
+   // Right hand side is zero
+   ParComplexLinearForm b(fespace, ComplexOperator::HERMITIAN);
+   b.real().Vector::operator=(0.0);
+   b.imag().Vector::operator=(0.0);
+   b.Assemble();
+
+   // 9. Define the solution vector x as a parallel finite element grid function
+   //    corresponding to fespace. Initialize x by projecting the exact
+   //    solution. Note that only values from the boundary edges will be used
+   //    when eliminating the non-homogeneous boundary condition to modify the
+   //    r.h.s. vector b.
+   ParComplexGridFunction x(fespace);
+   VectorFunctionCoefficient E_Re(sdim, E_bdr_data_Re);
+   VectorFunctionCoefficient E_Im(sdim, E_bdr_data_Im);
+   x.ProjectBdrCoefficientTangent(E_Re, E_Im, ess_bdr);
+
+   // 10. Set up the parallel bilinear form corresponding to the EM diffusion
+   //     operator curl muinv curl + sigma I, by adding the curl-curl and the
+   //     mass domain integrators.
+   ConstantCoefficient muinv(1.0);
+   ConstantCoefficient sigma(-pow(omega, 2));
+   FunctionCoefficient det_inv_Re(pml_detJ_inv_Re);
+   FunctionCoefficient det_inv_Im(pml_detJ_inv_Im);
+   MatrixFunctionCoefficient temp_c1_Re(dim, pml_detJ_inv_JT_J_Re);
+   MatrixFunctionCoefficient temp_c1_Im(dim, pml_detJ_inv_JT_J_Im);
+   MatrixFunctionCoefficient temp_c2_Re(dim, pml_detJ_JT_J_inv_Re);
+   MatrixFunctionCoefficient temp_c2_Im(dim, pml_detJ_JT_J_inv_Im);
+
+   ScalarMatrixProductCoefficient pml_c1_Re(muinv, temp_c1_Re);
+   ScalarMatrixProductCoefficient pml_c1_Im(muinv, temp_c1_Im);
+   ScalarMatrixProductCoefficient pml_c2_Re(sigma, temp_c2_Re);
+   ScalarMatrixProductCoefficient pml_c2_Im(sigma, temp_c2_Im);
+
+
+   ParSesquilinearForm a(fespace, ComplexOperator::HERMITIAN);
+   
+   if (dim == 3)
+   {
+      a.AddDomainIntegrator(new CurlCurlIntegrator(pml_c1_Re),
+                            new CurlCurlIntegrator(pml_c1_Im));
+   }
+   else
+   {
+      a.AddDomainIntegrator(new CurlCurlIntegrator(det_inv_Re),
+                            new CurlCurlIntegrator(det_inv_Im));
+   }
+   a.AddDomainIntegrator(new VectorFEMassIntegrator(pml_c2_Re),
+                         new VectorFEMassIntegrator(pml_c2_Im));
+   a.Assemble();
+
+
+   OperatorHandle Ah;
+   Vector B, X;
+   a.FormLinearSystem(ess_tdof_list, x, b, Ah, X, B);
+
+   // Transform to monolithic HypreParMatrix
+   HypreParMatrix * A = Ah.As<ComplexHypreParMatrix>()->GetSystemMatrix();
+
+   if (myid == 0)
+   {
+      cout << "Size of linear system: " << A->GetGlobalNumRows() << endl;
+   }
+
+
+   // // SuperLU direct solver
+   // SuperLURowLocMatrix *SA = new SuperLURowLocMatrix(*A);
+   // SuperLUSolver *superlu = new SuperLUSolver(MPI_COMM_WORLD);
+   // superlu->SetPrintStatistics(false);
+   // superlu->SetSymmetricPattern(false);
+   // superlu->SetColumnPermutation(superlu::PARMETIS);
+   // superlu->SetOperator(*SA);
+   // superlu->Mult(B, X);
+
+   cout << "Total number of elements: " << elems_pml.Size() << endl;
+   
+   cout << "pml layer elements: " << elems_pml.Size() - elems_pml.Sum(); cout << endl;
+   cout << "computational domain elements: " << elems_pml.Sum(); cout << endl;
+
+
+
+   const char *petscrc_file = "petscrc_mult_options";
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+   PetscLinearSolver * invA = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   PetscParMatrix *PA = new PetscParMatrix(A, Operator::PETSC_MATAIJ);
+   invA->SetOperator(*PA);
+   invA->Mult(B,X);
+   delete PA;
+   MFEMFinalizePetsc();
+
+   // 13. Recover the parallel grid function corresponding to X. This is the
+   //     local finite element solution on each processor.
+   a.RecoverFEMSolution(X, b, x);
+
+
+   ParComplexGridFunction x_gf(fespace);
+   x_gf.ProjectCoefficient(E_Re, E_Im);
+
+
+
+   // Compute error
+   if (prob == scatter && src == 3)
+   {
+      int order_quad = max(2, 2 * order + 1);
+      const IntegrationRule *irs[Geometry::NumGeom];
+      for (int i = 0; i < Geometry::NumGeom; ++i)
+      {
+         irs[i] = &(IntRules.Get(i, order_quad));
+      }
+
+      double L2Error_Re = x.real().ComputeL2Error(E_Re, irs,&elems_pml);
+      double L2Error_Im = x.imag().ComputeL2Error(E_Im, irs,&elems_pml);
+
+      ParComplexGridFunction x_gf0(fespace);
+      x_gf0 = 0.0;
+      double norm_E_Re = x_gf0.real().ComputeL2Error(E_Re, irs,&elems_pml);
+      double norm_E_Im = x_gf0.imag().ComputeL2Error(E_Im, irs,&elems_pml);
+
+      if (myid == 0)
+      {
+         cout << " Real Part: || E_h - E || / ||E|| = " << L2Error_Re / norm_E_Re << '\n' << endl;
+         cout << " Imag Part: || E_h - E || / ||E|| = " << L2Error_Im / norm_E_Im << '\n' << endl;
+
+         cout << " Real Part: || E_h - E || = " << L2Error_Re << '\n' << endl;
+         cout << " Imag Part: || E_h - E || = " << L2Error_Im << '\n' << endl;
+      }
+   }
+
+
+
+
+   // 16. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      string keys;
+      if (dim == 3)
+      {
+         keys = "keys mF\n";
+      }
+      else
+      {
+         keys = "keys arRljcUU\n";
+      }
+      char vishost[] = "localhost";
+      int visport = 19916;
+      socketstream src_sock_re(vishost, visport);
+      src_sock_re << "parallel " << num_procs << " " << myid << "\n";
+      src_sock_re.precision(8);
+      src_sock_re << "solution\n" << *pmesh << x_gf.real() << keys <<"window_title 'Source real part'" << flush;
+
+      MPI_Barrier(MPI_COMM_WORLD);
+      socketstream src_sock_im(vishost, visport);
+      src_sock_im << "parallel " << num_procs << " " << myid << "\n";
+      src_sock_im.precision(8);
+      src_sock_im << "solution\n" << *pmesh << x_gf.imag() << keys <<"window_title 'Source imag part'" << flush;
+
+      MPI_Barrier(MPI_COMM_WORLD);
+      socketstream sol_sock_re(vishost, visport);
+      sol_sock_re << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_re.precision(8);
+      sol_sock_re << "solution\n" << *pmesh << x.real() << keys <<"window_title 'Solution real part'" << flush;
+
+      MPI_Barrier(MPI_COMM_WORLD);
+      socketstream sol_sock_im(vishost, visport);
+      sol_sock_im << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_im.precision(8);
+      sol_sock_im << "solution\n" << *pmesh << x.imag() << keys <<"window_title 'Solution imag part'" << flush;
+   }
+
+
+   // VisItDataCollection visit_dc("Example23", pmesh);
+   // visit_dc.RegisterField("solution", &x.real());
+   // visit_dc.Save();
+   // 17. Free the used memory.
+   // delete superlu;
+   // delete SA;
+   delete fespace;
+   delete fec;
+   delete pmesh;
+
+   MPI_Finalize();
+   return 0;
+}
+
+
+void compute_pml_mesh_data(Mesh * mesh)
+{
+   mesh->EnsureNodes();
+   GridFunction * nodes = mesh->GetNodes();
+   // Assuming square/cubic domain 
+   int ndofs = nodes->FESpace()->GetNDofs();
+   Array2D<double> coords(ndofs,dim);
+   Vector xcoords(ndofs), ycoords(ndofs), zcoords(ndofs);
+
+   for (int comp = 0; comp < nodes->FESpace()->GetVDim(); comp++)
+   {
+      // cout << comp << endl;
+      for (int i = 0; i < ndofs; i++)
+      {
+         if (comp == 0)
+         {
+            xcoords(i) = (*nodes)[nodes->FESpace()->DofToVDof(i, comp)];
+         }
+         else if (comp == 1)
+         {
+            ycoords(i) = (*nodes)[nodes->FESpace()->DofToVDof(i, comp)];
+         }
+         else if (comp == 2)
+         {
+            zcoords(i) = (*nodes)[nodes->FESpace()->DofToVDof(i, comp)];
+         }
+      }
+   }
+
+   domain_bdr.SetSize(3,2);
+   domain_bdr(0,0) = xcoords.Min();
+   domain_bdr(0,1) = xcoords.Max();
+   domain_bdr(1,0) = ycoords.Min();
+   domain_bdr(1,1) = ycoords.Max();
+   domain_bdr(2,0) = zcoords.Min();
+   domain_bdr(2,1) = zcoords.Max();
+
+   pml_lngth.SetSize(dim,2);
+   comp_domain_bdr.SetSize(dim,2);
+   if (prob == scatter)
+   {
+      for (int i=0; i<dim; i++)
+      {
+         for (int j=0; j<2; j++)
+         {
+            pml_lngth(i,j) = 0.125 * (domain_bdr(i,1) - domain_bdr(i,0));
+         }
+         comp_domain_bdr(i,0) = domain_bdr(i,0) + pml_lngth(i,0);
+         comp_domain_bdr(i,1) = domain_bdr(i,1) - pml_lngth(i,1);
+      }
+   }
+   else if (prob == waveguide)
+   {
+      for (int i=0; i<dim; i++)
+      {
+         comp_domain_bdr(i,0) = domain_bdr(i,0);
+         comp_domain_bdr(i,1) = domain_bdr(i,1);
+      }   
+      // pml only in the z direction
+      pml_lngth(2,1) = 0.125 * (domain_bdr(2,1) - domain_bdr(2,0));
+      comp_domain_bdr(2,1) = domain_bdr(2,1) - pml_lngth(2,1);
+   }
+}
+
+
+void compute_pml_elem_list(ParMesh * pmesh, Array<int> & elem_pml)
+{
+   int nrelem = pmesh->GetNE();
+   // initialize list with 1
+   elem_pml.SetSize(nrelem);
+   elem_pml = 1;
+   // loop through the elements and identify which of them are in the pml
+   pmesh->EnsureNodes();
+   GridFunction * nodes = pmesh->GetNodes();
+   // Assuming square/cubic domain 
+   int ndofs = nodes->FESpace()->GetNDofs();
+   Array2D<double> coords(ndofs,dim);
+
+   for (int comp = 0; comp < dim; comp++)
+   {
+      // cout << comp << endl;
+      for (int i = 0; i < ndofs; i++)
+      {
+         coords(i,comp) = (*nodes)[nodes->FESpace()->DofToVDof(i, comp)];
+      }
+   }
+
+   for (int i = 0; i < nrelem; ++i)
+   {
+      Element * el = pmesh->GetElement(i);
+      Array<int> vertices;
+      el->GetVertices(vertices);
+      // get the elent 
+      int nrvert = vertices.Size();
+      // Check if any vertex is in the pml
+      bool in_pml = false;
+      for (int iv=0; iv<nrvert; ++iv)
+      {
+         int vert_idx = vertices[iv];
+         for (int comp = 0; comp<dim; ++ comp)
+         {
+           if (coords(vert_idx,comp) > comp_domain_bdr(comp,1) ||
+               coords(vert_idx,comp) < comp_domain_bdr(comp,0))
+            {  
+               in_pml = true;
+               break;
+            }
+         }
+      }   
+      if (in_pml) elem_pml[i] = 0;
+   }
+
+}
+
+
+
+void maxwell_ess_data(const Vector &x, std::vector<std::complex<double>> &E)
+{
+   // Initialize
+   for (int i = 0; i < dim; ++i)
+      E[i] = complex<double>(0., 0.);
+
+   std::complex<double> zi = std::complex<double>(0., 1.);
+
+   if (prob == waveguide) 
+   {
+      double k10 = sqrt(omega*omega - M_PI * M_PI);
+      E[0] = 0.0;
+      // E[1] = - zi * omega / M_PI * sin(M_PI* x(0)) * exp(-zi * k10 * x(2)); // T_10 mode
+      // if (abs(x(2))<1e-13)
+         E[1] = - zi * 2.0 * sqrt(5.0) * sin(2.0 * M_PI* x(0)) * exp(-zi * k10 * x(2)); // T_20 mode
+         // E[1] = - zi * omega / M_PI * sin(M_PI* x(0)) * exp(-zi * k10 * x(2)); // T_10 mode
+      //    E[1] =  2.0* x(0) * exp(-zi * omega * x(2)); 
+   }
+   else // point source (scattering)
+   {
+      Vector shift(dim);
+      shift = 0.0;
+      for (int i=0; i<dim; ++i) shift(i) = - 0.5 * (domain_bdr(i,0)+domain_bdr(i,1));
+
+      if (dim == 2)
+      {
+         double x0 = x(0)+shift(0);
+         double x1 = x(1)+shift(1);
+         std::complex<double> val, val_x, val_xx, val_xy;
+         double r = sqrt(x0 * x0 + x1 * x1);
+         double beta = omega * r;
+         
+         complex<double> Ho = boost::math::cyl_hankel_1(0,beta);
+         complex<double> Ho_r = - omega * boost::math::cyl_hankel_1(1,beta); 
+         complex<double> Ho_rr = - omega * omega * 
+                                 ( 1.0/beta * boost::math::cyl_hankel_1(1,beta) - 
+                                 boost::math::cyl_hankel_1(2,beta) ); 
+         // derivative with respect to x
+         double r_x = x0 / r; 
+         double r_y = x1 / r; 
+         double r_xy = -(r_x / r) * r_y;
+         double r_xx = (1.0 / r) * (1.0 - r_x * r_x);
+
+         val = 0.25* zi * Ho; // i/4 * H_0^1(omega * r)
+         val_x  = 0.25* zi * r_x * Ho_r;
+         val_xx = 0.25* zi * (r_xx * Ho_r + r_x * r_x * Ho_rr);
+         val_xy = 0.25* zi * (r_xy * Ho_r + r_x * r_y * Ho_rr);
+         E[0] = zi / omega * (omega * omega * val + val_xx);
+         E[1] = zi / omega * val_xy;
+      }
+      else
+      {
+         double x0 = x(0)+shift(0);
+         double x1 = x(1)+shift(1);
+         double x2 = x(2)+shift(2);
+         double r = sqrt(x0 * x0 + x1 * x1 + x2 * x2);
+
+         double r_x = x0/r;
+         double r_y = x1/r;
+         double r_z = x2/r;
+         double r_xx = (1.0 / r) * (1.0 - r_x * r_x);
+         double r_yx = -(r_y / r) * r_x;
+         double r_zx = -(r_z / r) * r_x;
+
+         complex<double> val;
+         complex<double> val_x, val_y, val_z;
+         complex<double> val_xx, val_yx, val_zx;
+         complex<double> val_r, val_rr;
+
+         val = exp(zi*omega*r)/r;
+         val_r = val / r * (zi * omega - 1.0);
+         val_rr = val/(r*r) * (-omega * omega * r * r - 2.*zi * omega * r + 2.);
+         val_x = val_r*r_x;
+         val_y = val_r*r_y;
+         val_z = val_r*r_z;
+
+         val_xx = val_rr * r_x * r_x + val_r * r_xx;
+         val_yx = val_rr * r_x * r_y + val_r * r_yx;
+         val_zx = val_rr * r_x * r_z + val_r * r_zx;
+
+         complex<double> alpha; 
+         alpha = zi*omega / 4.0 / M_PI / omega / omega;
+         // E[0] = alpha * (val + val_xx/pow(omega,2.0));
+         // E[1] = alpha * (val_yx/pow(omega,2.0));
+         // E[2] = alpha * (val_zx/pow(omega,2.0));
+         E[0] = alpha * (omega * omega * val + val_xx);
+         E[1] = alpha * val_yx;
+         E[2] = alpha * val_zx;
+      }
+   }
+}
+
+void E_bdr_data_Re(const Vector &x, Vector &E)
+{
+   // Initialize
+   E = 0.0;
+   bool in_pml = false;
+   if (prob == scatter)
+   {
+      for (int i = 0; i < dim; ++i)
+      {
+         // check if x(i) is in the computational domain or not
+         // if (x(i) < comp_domain_bdr(i,0) || x(i) > comp_domain_bdr(i,1))
+         if (abs(x(i) - domain_bdr(i,0)) < 1e-13 || abs(x(i) - domain_bdr(i,1)) < 1e-13)
+         {
+            in_pml = true;
+            break;
+         }
+      }
+      if (!in_pml)
+      {
+         std::vector<std::complex<double>> Eval(E.Size());
+         maxwell_ess_data(x, Eval);
+         for (int i = 0; i < dim; ++i) E[i] = Eval[i].real();
+      }
+   }
+   else if (prob == waveguide)
+   { // waveguide problem
+      std::vector<std::complex<double>> Eval(E.Size());
+      maxwell_ess_data(x, Eval);
+      for (int i = 0; i < dim; ++i) E[i] = Eval[i].real();
+      if (abs(x(2)-domain_bdr(2,1)) < 1e-13 ) E = 0.0;
+   }
+}
+
+//define bdr_data solution
+void E_bdr_data_Im(const Vector &x, Vector &E)
+{
+  E = 0.0;
+   bool in_pml = false;
+   if (prob == scatter)
+   {
+      for (int i = 0; i < dim; ++i)
+      {
+         // check if x(i) is in the computational domain or not
+         // if (x(i) < comp_domain_bdr(i,0) || x(i) > comp_domain_bdr(i,1))
+         if (abs(x(i) - domain_bdr(i,0)) < 1e-13 || abs(x(i) - domain_bdr(i,1)) < 1e-13)
+         {
+            in_pml = true;
+            break;
+         }
+      }
+      if (!in_pml)
+      {
+         std::vector<std::complex<double>> Eval(E.Size());
+         maxwell_ess_data(x, Eval);
+         for (int i = 0; i < dim; ++i) E[i] = Eval[i].imag();
+      }
+   }
+   else if (prob == waveguide)
+   { // waveguide problem
+      std::vector<std::complex<double>> Eval(E.Size());
+      maxwell_ess_data(x, Eval);
+      for (int i = 0; i < dim; ++i) E[i] = Eval[i].imag();
+      if (abs(x(2)-domain_bdr(2,1)) < 1e-13 ) E = 0.0;
+   }
+}
+
+// PML
+void pml_function(const Vector &x, std::vector<std::complex<double>> &dxs)
+{
+   std::complex<double> zi  = std::complex<double>(0., 1.);
+   std::complex<double> one = std::complex<double>(1., 0.);
+
+   double n = 2.0;
+   double c = 10.0;
+   double coeff;
+
+   // initialize to one
+   for (int i = 0; i < dim; ++i) dxs[i] = one;
+
+   // Stretch in each direction independenly
+   for (int i = 0; i < dim; ++i)
+   {
+      for (int j=0; j<2; ++j)
+      if (x(i) >= comp_domain_bdr(i,1))
+      {
+         coeff = n * c / omega / pow(pml_lngth(i,1), n);
+         dxs[i] = one + zi * coeff * abs(pow(x(i) - comp_domain_bdr(i,1), n - 1.0));
+      }
+      if (x(i) <= comp_domain_bdr(i,0))
+      {
+         coeff = n * c / omega / pow(pml_lngth(i,0), n);
+         dxs[i] = one + zi * coeff * abs(pow(x(i) - comp_domain_bdr(i,0), n - 1.0));
+      }
+   }
+}
+
+double pml_detJ_inv_Re(const Vector &x)
+{
+   std::complex<double> one = std::complex<double>(1., 0.);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0, 0.0);
+   pml_function(x, dxs);
+   for (int i = 0; i < dim; ++i)
+      det *= dxs[i];
+
+   complex<double> det_inv = one / det;
+   return det_inv.real();
+}
+double pml_detJ_inv_Im(const Vector &x)
+{
+   std::complex<double> one = std::complex<double>(1., 0.);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0, 0.0);
+   pml_function(x, dxs);
+   for (int i = 0; i < dim; ++i)
+      det *= dxs[i];
+
+   complex<double> det_inv = one / det;
+   return det_inv.imag();
+}
+void pml_detJ_JT_J_inv_Re(const Vector &x, DenseMatrix &M)
+{
+   std::complex<double> one = std::complex<double>(1., 0.);
+   std::vector<complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0, 0.0);
+   pml_function(x, dxs);
+
+   for (int i = 0; i < dim; ++i)
+   {
+      diag[i] = one / pow(dxs[i], 2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M = 0.0;
+
+   for (int i = 0; i < dim; ++i)
+   {
+      complex<double> temp = det * diag[i];
+      M(i, i) = temp.real();
+   }
+}
+void pml_detJ_JT_J_inv_Im(const Vector &x, DenseMatrix &M)
+{
+   std::complex<double> one = std::complex<double>(1., 0.);
+   std::vector<std::complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det = 1.0;
+   pml_function(x, dxs);
+
+   for (int i = 0; i < dim; ++i)
+   {
+      diag[i] = one / pow(dxs[i], 2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M = 0.0;
+
+   for (int i = 0; i < dim; ++i)
+   {
+      complex<double> temp = det * diag[i];
+      M(i, i) = temp.imag();
+   }
+}
+void pml_detJ_inv_JT_J_Re(const Vector &x, DenseMatrix &M)
+{
+   std::vector<complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0, 0.0);
+   pml_function(x, dxs);
+
+   for (int i = 0; i < dim; ++i)
+   {
+      diag[i] = pow(dxs[i], 2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M = 0.0;
+
+   for (int i = 0; i < dim; ++i)
+   {
+      complex<double> temp = diag[i] / det;
+      M(i, i) = temp.real();
+   }
+}
+void pml_detJ_inv_JT_J_Im(const Vector &x, DenseMatrix &M)
+{
+   std::vector<std::complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det = 1.0;
+   pml_function(x, dxs);
+
+   for (int i = 0; i < dim; ++i)
+   {
+      diag[i] = pow(dxs[i], 2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M = 0.0;
+
+   for (int i = 0; i < dim; ++i)
+   {
+      complex<double> temp = diag[i] / det;
+      M(i, i) = temp.imag();
+   }
+}
+

examples/solvers-dev/helmholtz.cpp

@@ -0,0 +1,340 @@
+//                                MFEM Example 1
+//
+// Compile with: make helmholtz
+//
+
+#include "mfem.hpp"
+#include "as/schwarz.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+void get_solution(const Vector &x, double & u, double & d2u);
+double u_exact(const Vector &x);
+double f_exact(const Vector &x);
+
+int isol=0;
+int dim;
+double omega;
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options.
+   const char *mesh_file = "../data/one-hex.mesh";
+   int order = 1;
+   int sdim = 2;
+   bool static_cond = false;
+   const char *device_config = "cpu";
+   bool visualization = true;
+   int ref_levels = 1;
+   int initref    = 1;
+   // number of wavelengths
+   double k = 0.5;
+   double theta = 0.5;
+   double smth_maxit = 1;
+   StopWatch chrono;
+
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&sdim, "-d", "--dimension", "Dimension");
+   args.AddOption(&ref_levels, "-sr", "--serial-refinements", 
+                  "Number of mesh refinements");
+   args.AddOption(&initref, "-iref", "--init-refinements", 
+                  "Number of initial mesh refinements");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&smth_maxit, "-sm", "--smoother-maxit",
+                  "Number of smoothing steps.");               
+   args.AddOption(&theta, "-th", "--theta",
+                  "Dumping parameter for the smoother.");               
+   args.AddOption(&isol, "-sol", "--solution", 
+                  "Exact Solution: 0) Polynomial, 1) Sinusoidal.");               
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   // Angular frequency
+   omega = 2.0 * M_PI * k;
+
+   // 3. Read the mesh from the given mesh file. 
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+
+   Mesh * mesh; 
+   // Define a simple square or cubic mesh
+   if (sdim == 2)
+   {
+      mesh = new Mesh(1, 1, Element::QUADRILATERAL, true,1.0, 1.0,false);
+      // mesh = new Mesh(1, 1, Element::TRIANGLE, true,1.0, 1.0,false);
+   }
+   else
+   {
+      mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true,1.0, 1.0,1.0, false);
+   }
+   dim = mesh->Dimension();
+   for (int i=0; i<initref; i++) {mesh->UniformRefinement();}
+
+
+   Mesh * cmesh = new Mesh(*mesh);
+
+   for (int l = 0; l < ref_levels; l++)
+   {
+      mesh->UniformRefinement();
+   }
+
+
+   FiniteElementCollection *fec = new H1_FECollection(order, dim);
+   FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
+
+   // 6. Determine the list of true (i.e. conforming) essential boundary dofs.
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr;
+   if (mesh->bdr_attributes.Size())
+   {
+      ess_bdr.SetSize(mesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+
+   // 7. Set up the linear form b(.) 
+   LinearForm *b = new LinearForm(fespace);
+   ConstantCoefficient one(1.0);
+   FunctionCoefficient f(f_exact);
+   b->AddDomainIntegrator(new DomainLFIntegrator(f));
+   // b->AddDomainIntegrator(new DomainLFIntegrator(one));
+   b->Assemble();
+
+   // 8. Define the solution vector x as a finite element grid function
+   GridFunction x(fespace);
+   x = 0.0;
+   FunctionCoefficient u_ex(u_exact);
+   x.ProjectCoefficient(u_ex);
+
+   // 9. Set up the bilinear form a(.,.) 
+   ConstantCoefficient sigma(-pow(omega, 2));
+
+   BilinearForm *a = new BilinearForm(fespace);
+   a->AddDomainIntegrator(new DiffusionIntegrator(one));
+   a->AddDomainIntegrator(new MassIntegrator(sigma));
+   if (static_cond) { a->EnableStaticCondensation(); }
+   a->Assemble();
+
+   SparseMatrix A;
+   Vector B, X;
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+   cout << "Size of linear system: " << A.Height() << endl;
+
+   FiniteElementSpace *prec_fespace = (a->StaticCondensationIsEnabled() ? a->SCFESpace() : fespace);
+   
+   chrono.Clear();
+   chrono.Start();
+   SchwarzSmoother * prec = new SchwarzSmoother(cmesh,ref_levels, prec_fespace, &A, ess_bdr);
+   prec->SetType(Schwarz::SmootherType::ADDITIVE);
+   prec->SetNumSmoothSteps(smth_maxit);
+   prec->SetDumpingParam(theta);
+   chrono.Stop();
+
+   // Need to invasticate the time scalings. TODO
+   cout << "Smoother construction time " << chrono.RealTime() << "s. \n";
+   
+
+   // DSmoother M(A);
+   // GSSmoother M(A);
+
+   int maxit(1000);
+   double rtol(0.0);
+   double atol(1.e-6);
+   // CGSolver solver;
+   GMRESSolver solver;
+   solver.SetAbsTol(atol);
+   solver.SetRelTol(rtol);
+   solver.SetMaxIter(maxit);
+   solver.SetOperator(A);
+   solver.SetPreconditioner(*prec);
+   // solver.SetPreconditioner(M);
+   solver.SetPrintLevel(1);
+   solver.Mult(B,X);
+
+
+
+   a->RecoverFEMSolution(X, *b, x);
+
+
+   GridFunction ugf(fespace);
+   ugf.ProjectCoefficient(u_ex);
+
+
+   int order_quad = max(2, 2*order+1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i=0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double L2error = x.ComputeL2Error(u_ex);
+      
+   cout << " || u_h - u ||_{L^2} = " << L2error <<  endl;
+
+
+
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      socketstream ex_sock(vishost, visport);
+      ex_sock.precision(8);
+      if (dim == 2) 
+      {
+         sol_sock <<  "solution\n" << *mesh << x  << "keys rRljc\n" << flush;
+         ex_sock <<  "solution\n" << *mesh << ugf  << "keys rRljc\n" << flush;
+      }
+      else
+      {
+         sol_sock <<  "solution\n" << *mesh << x  << "keys lc\n" << flush;
+         ex_sock <<  "solution\n" << *mesh << ugf  << "keys lc\n" << flush;
+      }
+   }
+
+
+
+
+   // if (visualization)
+   // {
+   //    char vishost[] = "localhost";
+   //    int  visport   = 19916;
+   //    socketstream sol_sock(vishost, visport);
+   //    sol_sock.precision(8);
+   //    sol_sock << "mesh\n" << *cmesh << flush;
+   // }
+
+   // if (visualization)
+   // {
+   //    char vishost[] = "localhost";
+   //    int  visport   = 19916;
+   //    socketstream sol_sock(vishost, visport);
+   //    sol_sock.precision(8);
+   //    sol_sock << "mesh\n" << *mesh << flush;
+   // }
+
+
+   // 15. Free the used memory.
+   delete a;
+   delete b;
+   delete fec;
+   delete fespace;
+   delete mesh;
+   return 0;
+}
+
+
+void get_solution(const Vector &x, double & u, double & d2u)
+{
+
+   if (dim == 2)
+   {
+      if (isol == 0)
+      {
+         u = x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]);
+         d2u = -2.0* ( x[1]*(1.0 - x[1]) + x[0]*(1.0 - x[0])); 
+      }
+      else if (isol == 1)
+      {  // Point source
+         //shift to avoid singularity
+         double x0 = x(0) + 0.1;
+         double x1 = x(1) + 0.1;
+         //
+         double r = sqrt(x0 * x0 + x1 * x1);
+
+         u = cos(omega * r);
+
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_xx = (1.0 / r) * (1.0 - r_x * r_x);
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+
+         double u_xx = - omega * omega * u * r_x * r_x - omega * sin(omega * r) * r_xx;
+         double u_yy = - omega * omega * u * r_y * r_y - omega * sin(omega * r) * r_yy;
+
+         d2u = u_xx + u_yy;
+      }
+      else
+      {
+         double alpha = omega / sqrt(2.0);
+         u = cos(alpha * (x[0] + x[1]));
+         d2u = -2.0* alpha * alpha * u;
+      }
+   }
+   else
+   {
+      if (isol == 0)
+      {
+         u = x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+         d2u = -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[1]) * x[1] 
+            -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[2]) * x[2] 
+            -2.0*(-1.0 + x[1]) * x[1] * (-1.0 + x[2]) * x[2];
+      }
+      else if (isol == 1)
+      {  // Point source
+         // shift to avoid singularity
+         double x0 = x(0) + 0.1;
+         double x1 = x(1) + 0.1;
+         double x2 = x(2) + 0.1;
+         //
+         double r = sqrt(x0 * x0 + x1 * x1 + x2 * x2);
+
+         u = cos(omega * r);
+
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_z = x2 / r;
+         double r_xx = (1.0 / r) * (1.0 - r_x * r_x);
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+         double r_zz = (1.0 / r) * (1.0 - r_z * r_z);
+
+         double u_xx = - omega * omega * u * r_x * r_x - omega * sin(omega * r) * r_xx;
+         double u_yy = - omega * omega * u * r_y * r_y - omega * sin(omega * r) * r_yy;
+         double u_zz = - omega * omega * u * r_z * r_z - omega * sin(omega * r) * r_zz;
+
+         d2u = u_xx + u_yy + u_zz;
+      }
+      else
+      {
+         double alpha = omega / sqrt(3.0);
+         u = cos(alpha * (x[0] + x[1] + x[2]));
+         d2u = -3.0* alpha * alpha * u;
+      }
+   }
+}
+
+double u_exact(const Vector &x)
+{
+   double u, d2u;
+   get_solution(x, u, d2u);
+   return u;
+}
+
+double f_exact(const Vector &x)
+{
+   double u, d2u;
+   get_solution(x, u, d2u);
+   // return -d2u;
+   return -d2u - omega*omega * u;
+}

examples/solvers-dev/helmholtzPrimalDPG.cpp

@@ -0,0 +1,314 @@
+//                                MFEM Example 8
+//
+// Compile with: make ex8
+//
+// Sample runs:  ex8 -m ../data/square-disc.mesh
+//               ex8 -m ../data/star.mesh
+//               ex8 -m ../data/star-mixed.mesh
+//               ex8 -m ../data/escher.mesh
+//               ex8 -m ../data/fichera.mesh
+//               ex8 -m ../data/fichera-mixed.mesh
+//               ex8 -m ../data/square-disc-p2.vtk
+//               ex8 -m ../data/square-disc-p3.mesh
+//               ex8 -m ../data/star-surf.mesh -o 2
+//               ex8 -m ../data/mobius-strip.mesh
+//
+// Description:  This example code demonstrates the use of the Discontinuous
+//               Petrov-Galerkin (DPG) method in its primal 2x2 block form as a
+//               simple finite element discretization of the Laplace problem
+//               -Delta u = f with homogeneous Dirichlet boundary conditions. We
+//               use high-order continuous trial space, a high-order interfacial
+//               (trace) space, and a high-order discontinuous test space
+//               defining a local dual (H^{-1}) norm.
+//
+//               We use the primal form of DPG, see "A primal DPG method without
+//               a first-order reformulation", Demkowicz and Gopalakrishnan, CAM
+//               2013, DOI:10.1016/j.camwa.2013.06.029.
+//
+//               The example highlights the use of interfacial (trace) finite
+//               elements and spaces, trace face integrators and the definition
+//               of block operators and preconditioners.
+//
+//               We recommend viewing examples 1-5 before viewing this example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include "as/schwarz.hpp"
+using namespace std;
+using namespace mfem;
+
+int isol=0;
+int dim;
+double omega;
+
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options.
+   const char *mesh_file = "../data/one-hex.mesh";
+   int order = 1;
+   int sdim = 2;
+   bool static_cond = false;
+   const char *device_config = "cpu";
+   bool visualization = true;
+   int ref_levels = 1;
+   int initref    = 1;
+   // number of wavelengths
+   double k = 0.5;
+   StopWatch chrono;
+
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&sdim, "-d", "--dimension", "Dimension");
+   args.AddOption(&ref_levels, "-sr", "--serial-refinements", 
+                  "Number of mesh refinements");
+   args.AddOption(&initref, "-iref", "--init-refinements", 
+                  "Number of initial mesh refinements");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&isol, "-sol", "--solution", 
+                  "Exact Solution: 0) Polynomial, 1) Sinusoidal.");               
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   // Angular frequency
+   omega = 2.0 * M_PI * k;
+
+   // 2. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   device.Print();
+
+   // 3. Read the mesh from the given mesh file. 
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+
+   Mesh * mesh; 
+   // Define a simple square or cubic mesh
+   if (sdim == 2)
+   {
+      mesh = new Mesh(1, 1, Element::QUADRILATERAL, true,1.0, 1.0,false);
+      // mesh = new Mesh(1, 1, Element::TRIANGLE, true,1.0, 1.0,false);
+   }
+   else
+   {
+      mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true,1.0, 1.0,1.0, false);
+   }
+   dim = mesh->Dimension();
+   for (int i=0; i<initref; i++) {mesh->UniformRefinement();}
+
+
+   Mesh * cmesh = new Mesh(*mesh);
+
+   for (int l = 0; l < ref_levels; l++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   // 4. Define the trial, interfacial (trace) and test DPG spaces:
+   //    - The trial space, x0_space, contains the non-interfacial unknowns and
+   //      has the essential BC.
+   //    - The interfacial space, xhat_space, contains the interfacial unknowns
+   //      and does not have essential BC.
+   //    - The test space, test_space, is an enriched space where the enrichment
+   //      degree may depend on the spatial dimension of the domain, the type of
+   //      the mesh and the trial space order.
+   unsigned int trial_order = order;
+   unsigned int trace_order = order - 1;
+   unsigned int test_order  = order; /* reduced order, full order is
+                                        (order + dim - 1) */
+   if (dim == 2 && (order%2 == 0 || (mesh->MeshGenerator() & 2 && order > 1)))
+   {
+      test_order++;
+   }
+   if (test_order < trial_order)
+      cerr << "Warning, test space not enriched enough to handle primal"
+           << " trial space\n";
+
+   FiniteElementCollection *x0_fec, *xhat_fec, *test_fec;
+
+   x0_fec   = new H1_FECollection(trial_order, dim);
+   xhat_fec = new RT_Trace_FECollection(trace_order, dim);
+   test_fec = new L2_FECollection(test_order, dim);
+
+   FiniteElementSpace *x0_space   = new FiniteElementSpace(mesh, x0_fec);
+   FiniteElementSpace *xhat_space = new FiniteElementSpace(mesh, xhat_fec);
+   FiniteElementSpace *test_space = new FiniteElementSpace(mesh, test_fec);
+
+   // 5. Define the block structure of the problem, by creating the offset
+   //    variables. Also allocate two BlockVector objects to store the solution
+   //    and rhs.
+   enum {x0_var, xhat_var, NVAR};
+
+   int s0 = x0_space->GetVSize();
+   int s1 = xhat_space->GetVSize();
+   int s_test = test_space->GetVSize();
+
+   Array<int> offsets(NVAR+1);
+   offsets[0] = 0;
+   offsets[1] = s0;
+   offsets[2] = s0+s1;
+
+   Array<int> offsets_test(2);
+   offsets_test[0] = 0;
+   offsets_test[1] = s_test;
+
+   std::cout << "\nNumber of Unknowns:\n"
+             << " Trial space,     X0   : " << s0
+             << " (order " << trial_order << ")\n"
+             << " Interface space, Xhat : " << s1
+             << " (order " << trace_order << ")\n"
+             << " Test space,      Y    : " << s_test
+             << " (order " << test_order << ")\n\n";
+
+   BlockVector x(offsets), b(offsets);
+   x = 0.;
+
+   // 6. Set up the linear form F(.) which corresponds to the right-hand side of
+   //    the FEM linear system, which in this case is (f,phi_i) where f=1.0 and
+   //    phi_i are the basis functions in the test finite element fespace.
+   ConstantCoefficient one(1.0);
+   LinearForm F(test_space);
+   F.AddDomainIntegrator(new DomainLFIntegrator(one));
+   F.Assemble();
+
+   // 7. Set up the mixed bilinear form for the primal trial unknowns, B0,
+   //    the mixed bilinear form for the interfacial unknowns, Bhat,
+   //    the inverse stiffness matrix on the discontinuous test space, Sinv,
+   //    and the stiffness matrix on the continuous trial space, S0.
+   Array<int> ess_bdr(mesh->bdr_attributes.Max());
+   ess_bdr = 1;
+
+   ConstantCoefficient sigma(-pow(omega, 2));
+   MixedBilinearForm *B0 = new MixedBilinearForm(x0_space,test_space);
+   B0->AddDomainIntegrator(new DiffusionIntegrator(one));
+   B0->AddDomainIntegrator(new MassIntegrator(sigma));
+
+   B0->Assemble();
+   B0->EliminateTrialDofs(ess_bdr, x.GetBlock(x0_var), F);
+   B0->Finalize();
+
+   MixedBilinearForm *Bhat = new MixedBilinearForm(xhat_space,test_space);
+   Bhat->AddTraceFaceIntegrator(new TraceJumpIntegrator());
+   Bhat->Assemble();
+   Bhat->Finalize();
+
+   BilinearForm *Sinv = new BilinearForm(test_space);
+   SumIntegrator *Sum = new SumIntegrator;
+   Sum->AddIntegrator(new DiffusionIntegrator(one));
+   Sum->AddIntegrator(new MassIntegrator(one));
+   Sinv->AddDomainIntegrator(new InverseIntegrator(Sum));
+   Sinv->Assemble();
+   Sinv->Finalize();
+
+   BilinearForm *S0 = new BilinearForm(x0_space);
+   S0->AddDomainIntegrator(new DiffusionIntegrator(one));
+   S0->Assemble();
+   S0->EliminateEssentialBC(ess_bdr);
+   S0->Finalize();
+
+   SparseMatrix &matB0   = B0->SpMat();
+   SparseMatrix &matBhat = Bhat->SpMat();
+   SparseMatrix &matSinv = Sinv->SpMat();
+   SparseMatrix &matS0   = S0->SpMat();
+
+   // 8. Set up the 1x2 block Least Squares DPG operator, B = [B0  Bhat],
+   //    the normal equation operator, A = B^t Sinv B, and
+   //    the normal equation right-hand-size, b = B^t Sinv F.
+   BlockOperator B(offsets_test, offsets);
+   B.SetBlock(0,0,&matB0);
+   B.SetBlock(0,1,&matBhat);
+   RAPOperator A(B, matSinv, B);
+   {
+      Vector SinvF(s_test);
+      matSinv.Mult(F,SinvF);
+      B.MultTranspose(SinvF, b);
+   }
+
+   // 9. Set up a block-diagonal preconditioner for the 2x2 normal equation
+   //
+   //        [ S0^{-1}     0     ]
+   //        [   0     Shat^{-1} ]      Shat = (Bhat^T Sinv Bhat)
+   //
+   //    corresponding to the primal (x0) and interfacial (xhat) unknowns.
+   SparseMatrix * Shat = RAP(matBhat, matSinv, matBhat);
+//    SchwarzSmoother * S0inv = new SchwarzSmoother(cmesh,ref_levels, x0_space, &matS0, ess_bdr);
+   Operator *S0inv = new UMFPackSolver(matS0);
+   Operator *Shatinv = new UMFPackSolver(*Shat);
+
+   BlockDiagonalPreconditioner P(offsets);
+   P.SetDiagonalBlock(0, S0inv);
+   P.SetDiagonalBlock(1, Shatinv);
+
+   // 10. Solve the normal equation system using the PCG iterative solver.
+   //     Check the weighted norm of residual for the DPG least square problem.
+   //     Wrap the primal variable in a GridFunction for visualization purposes.
+   PCG(A, P, b, x, 1, 200, 1e-12, 0.0);
+
+   {
+      Vector LSres(s_test);
+      B.Mult(x, LSres);
+      LSres -= F;
+      double res = sqrt(matSinv.InnerProduct(LSres, LSres));
+      cout << "\n|| B0*x0 + Bhat*xhat - F ||_{S^-1} = " << res << endl;
+   }
+
+   GridFunction x0;
+   x0.MakeRef(x0_space, x.GetBlock(x0_var), 0);
+
+   // 11. Save the refined mesh and the solution. This output can be viewed
+   //     later using GLVis: "glvis -m refined.mesh -g sol.gf".
+   {
+      ofstream mesh_ofs("refined.mesh");
+      mesh_ofs.precision(8);
+      mesh->Print(mesh_ofs);
+      ofstream sol_ofs("sol.gf");
+      sol_ofs.precision(8);
+      x0.Save(sol_ofs);
+   }
+
+   // 12. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *mesh << x0 << flush;
+   }
+
+   // 13. Free the used memory.
+   delete S0inv;
+   delete Shatinv;
+   delete Shat;
+   delete Bhat;
+   delete B0;
+   delete S0;
+   delete Sinv;
+   delete test_space;
+   delete test_fec;
+   delete xhat_space;
+   delete xhat_fec;
+   delete x0_space;
+   delete x0_fec;
+   delete mesh;
+
+   return 0;
+}

examples/solvers-dev/helmholtzp.cpp

@@ -0,0 +1,824 @@
+//
+// Compile with: make helmholtz
+//
+// Sample runs:  helmholtz -m ../data/one-hex.mesh
+//               helmholtz -m ../data/fichera.mesh
+//               helmholtz -m ../data/fichera-mixed.mesh
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Helmholtz problem
+//               -Delta p - omega^2 p = 1 with impedance boundary conditiones.
+//
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Exact solution and r.h.s., see below for implementation.
+void get_helmholtz_solution_Re(const Vector &x, double & p, double dp[], double & d2p);
+void get_helmholtz_solution_Im(const Vector &x, double & p, double dp[], double & d2p);
+double p_exact_Re(const Vector &x);
+double p_exact_Im(const Vector &x);
+double f_exact_Re(const Vector &x);
+double f_exact_Im(const Vector &x);
+double g_exact_Re(const Vector &x);
+double g_exact_Im(const Vector &x);
+void grad_exact_Re(const Vector &x, Vector &grad_Re);
+void grad_exact_Im(const Vector &x, Vector &grad_Im);
+
+// pml
+void pml_function(const Vector &x, std::vector<std::complex<double>> & dxs);
+double pml_detJ_Re(const Vector &x);
+double pml_detJ_Im(const Vector &x);
+void pml_detJ_JT_J_inv_Re(const Vector &x, DenseMatrix &M);
+void pml_detJ_JT_J_inv_Im(const Vector &x, DenseMatrix &M);
+
+
+int dim;
+double omega;
+int sol = 1;
+bool pml = false;
+double length = 1.0;
+double pml_length = 0.25;
+bool scatter = false;
+
+#ifndef MFEM_USE_PETSC
+#error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+#endif
+
+int main(int argc, char *argv[])
+{
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);                    // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);      // Determine process identifier
+
+   //-----------------------------------------------------------------------------
+
+   // 2. Parse command-line options.
+   // geometry file
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   bool visualization = 1;
+   // number of wavelengths
+   double k = 0.5;
+   // number of mg levels
+   int ref = 1;
+   // number of initial ref
+   int initref = 1;
+   // dimension
+   int nd = 2;
+
+
+   // PETSC
+   // const char *petscrc_file = "";
+   const char *petscrc_file = "petscrc_mult_options";
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&nd, "-nd", "--dim","Problem space dimension");               
+   args.AddOption(&sol, "-sol", "--exact",
+                  "Exact solution flag - 0:polynomial, 1: plane wave, -1: unknown exact");               
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&pml, "-pml", "--pml", "-no-pml",
+                  "--no-pml", "Enable PML.");         
+   args.AddOption(&pml_length, "-pml_length", "--pml_length",
+                  "Length of the PML region in each direction");        
+   args.AddOption(&length, "-length", "--length",
+                  "length of the domainin in each direction.");                                       
+   args.AddOption(&ref, "-ref", "--ref",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&petscrc_file, "-petscopts", "--petscopts",
+                  "PetscOptions file to use.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&scatter, "-scat", "--scattering-prob", "-no-scat",
+                  "--no-scattering", "Solve a scattering problem");               
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+   // Angular frequency
+   omega = 2.0 * M_PI * k;
+   // omega = k;
+
+   // 2b. Initialize PETSc
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+   //-----------------------------------------------------------------------------
+
+   // if (scatter) pml = true; // for now only scattering problems with pml
+
+   // 3. Read the mesh from the given mesh file.
+   Mesh *mesh;
+   
+   if (nd == 2) 
+   {
+      if (scatter) 
+      {
+         mesh_file = "../../data/rectwhole7_2attr.e";
+         mesh = new Mesh(mesh_file, 1, 1);
+      }
+      else
+      {
+         mesh = new Mesh(1, 1, Element::QUADRILATERAL, true, length, length, false);
+      }
+   }
+   else
+   {
+      if (scatter)
+      {
+         // mesh_file = "../../data/hexwhole7.e";
+         // mesh_file = "../../data/hexwhole.e";
+         // mesh_file = "../../data/hexa728.mesh";
+         // mesh_file = "./hexa728.mesh";
+         mesh_file = "../../data/hexwhole7.e";
+         mesh = new Mesh(mesh_file, 1, 1);
+      }
+      else
+      {
+         mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true, length, length, length, false);
+      }
+   }
+   // normalize mesh
+
+   mesh->EnsureNodes();
+   GridFunction * nodes = mesh->GetNodes();
+   // Assuming square/cubic domain 
+   double min_coord =  nodes->Min();
+   double max_coord =  nodes->Max();
+   double domain_length = abs(max_coord-min_coord); 
+   // shift to zero
+   *nodes -= min_coord;
+   // scale to one
+   *nodes *= 1./domain_length;
+
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   // 5. Define a parallel mesh and delete the serial mesh.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // ----------------------------------------------------------------------------
+
+   // 6. Define a finite element space on the mesh.
+   FiniteElementCollection *fec = new H1_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   std::vector<HypreParMatrix*>  P(ref);
+
+   for (int i = 0; i < ref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*fespace);
+      pmesh->UniformRefinement();
+      fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      fespace->GetTrueTransferOperator(cfespace, Tr);
+      HypreParMatrix * Paux;
+      Tr.Get(Paux);
+      P[i] = new HypreParMatrix(*Paux);
+   }
+
+   // 6. Set up the linear form (Real and Imaginary part)
+   FunctionCoefficient f_Re(f_exact_Re);
+   FunctionCoefficient g_Re(g_exact_Re);
+   VectorFunctionCoefficient grad_Re(sdim, grad_exact_Re);
+   FunctionCoefficient f_Im(f_exact_Im);
+   FunctionCoefficient g_Im(g_exact_Im);
+   VectorFunctionCoefficient grad_Im(sdim, grad_exact_Im);
+
+   // ParLinearForm *b_Re(new ParLinearForm);
+   ParComplexLinearForm b(fespace, ComplexOperator::HERMITIAN);
+   if (!scatter) // if scattering problem the source is zero and is driven by bc
+   {
+      b.AddDomainIntegrator(new DomainLFIntegrator(f_Re),new DomainLFIntegrator(f_Im));
+      if (!pml && sol >=0) // if exact solution exists. Otherwise use homogeneous impedence (gradp . n + i omega p = 0)
+      {
+         b.AddBoundaryIntegrator(new BoundaryNormalLFIntegrator(grad_Re),
+                                 new BoundaryNormalLFIntegrator(grad_Im));
+         b.AddBoundaryIntegrator(new BoundaryLFIntegrator(g_Re),
+                                 new BoundaryLFIntegrator(g_Im));
+      }
+   }
+   b.real().Vector::operator=(0.0);
+   b.imag().Vector::operator=(0.0);                        
+   b.Assemble();
+
+   // 7. Set up the bilinear form (Real and Imaginary part)
+   ConstantCoefficient one(1.0);
+   ConstantCoefficient sigma(-pow(omega, 2));
+
+   ParSesquilinearForm a(fespace,ComplexOperator::HERMITIAN);
+   ConstantCoefficient impedance(omega);
+
+
+   MatrixFunctionCoefficient c1_Re(dim,pml_detJ_JT_J_inv_Re);
+   MatrixFunctionCoefficient c1_Im(dim,pml_detJ_JT_J_inv_Im);
+
+   FunctionCoefficient det_Re(pml_detJ_Re);
+   FunctionCoefficient det_Im(pml_detJ_Im);
+   ProductCoefficient c2_Re(det_Re,sigma);
+   ProductCoefficient c2_Im(det_Im,sigma);
+
+   Array<int> bdr_attr(pmesh->bdr_attributes.Max());
+   bdr_attr = 0;
+   bdr_attr[0] = 1;
+   if (!scatter) bdr_attr = 1;
+   RestrictedCoefficient imp_rest(impedance,bdr_attr);
+
+   a.AddDomainIntegrator(new DiffusionIntegrator(c1_Re),new DiffusionIntegrator(c1_Im));
+   a.AddDomainIntegrator(new MassIntegrator(c2_Re),new MassIntegrator(c2_Im));
+   if (!pml)
+   {
+      a.AddBoundaryIntegrator(NULL,new BoundaryMassIntegrator(imp_rest));
+      // a.AddBoundaryIntegrator(NULL,new BoundaryMassIntegrator(impedance));
+   }
+   a.Assemble();
+   a.Finalize();
+
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+   ess_bdr = 0;
+   if (scatter)
+   {
+      if (pml)
+      {
+         ess_bdr = 1;
+      }
+      else
+      {
+         ess_bdr[1] = 1;
+      }
+   }
+   fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+
+   // Solution grid function
+   ParComplexGridFunction p_gf(fespace);
+   ParComplexGridFunction p_gf_ex(fespace);
+   FunctionCoefficient p_Re(p_exact_Re);
+   FunctionCoefficient p_Im(p_exact_Im);
+   p_gf = 0.0;
+   p_gf_ex.ProjectCoefficient(p_Re,p_Im);
+   if (!pml && sol >= 0 ) 
+   {
+      p_gf.ProjectBdrCoefficient(p_Re,p_Im,ess_bdr);
+   }
+   if (scatter) 
+   {
+      p_gf.ProjectBdrCoefficient(p_Re,p_Im,ess_bdr);
+   }
+
+   OperatorHandle Ah;
+   Vector X, B;
+   a.FormLinearSystem(ess_tdof_list, p_gf, b, Ah, X, B);
+
+   ComplexHypreParMatrix * AZ = Ah.As<ComplexHypreParMatrix>();
+   HypreParMatrix * A = AZ->GetSystemMatrix();
+
+
+   if (myid == 0)
+   {
+      cout << "Size of fine grid system: "
+           << A->GetGlobalNumRows() << " x " << A->GetGlobalNumCols() << endl;
+   }
+
+   PetscLinearSolver * petsc = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   // Convert to PetscParMatrix
+   petsc->SetOperator(PetscParMatrix(A, Operator::PETSC_MATAIJ));
+   petsc->Mult(B,X);
+
+   a.RecoverFEMSolution(X,B,p_gf);
+
+
+   if (!scatter && sol >= 0 )
+   {
+      // int order_quad = max(2, 2*order+1);
+      // const IntegrationRule *irs[Geometry::NumGeom];
+      // for (int i=0; i < Geometry::NumGeom; ++i)
+      // {
+         // irs[i] = &(IntRules.Get(i, order_quad));
+      // }
+
+      const int h1_norm_type = 1;
+      double L2error;
+      double H1error;
+
+      double L2err_Re  = p_gf.real().ComputeL2Error(p_Re);
+      double L2err_Im  = p_gf.imag().ComputeL2Error(p_Im);
+
+      double loc_H1err_Re = p_gf.real().ComputeH1Error(&p_Re, &grad_Re, &one, 1.0, h1_norm_type);
+      double loc_H1err_Im = p_gf.imag().ComputeH1Error(&p_Im, &grad_Im, &one, 1.0, h1_norm_type);
+      double H1err_Re = GlobalLpNorm(2.0, loc_H1err_Re, MPI_COMM_WORLD);
+      double H1err_Im = GlobalLpNorm(2.0, loc_H1err_Im, MPI_COMM_WORLD);
+      // double norm_Re = ComputeGlobalLpNorm(2, p_Re, *pmesh, irs);
+      // double norm_Im = ComputeGlobalLpNorm(2, p_Im, *pmesh, irs);
+         
+      L2error = sqrt(L2err_Re*L2err_Re + L2err_Im*L2err_Im);
+      H1error = sqrt(H1err_Re*H1err_Re + H1err_Im*H1err_Im);
+      // double L2norm = sqrt(norm_Re*norm_Re + norm_Im*norm_Im);
+      if (myid == 0)
+      {
+         cout << " || p_h - p ||_{H^1} = " << H1error <<  endl;
+         cout << " || p_h - p ||_{L^2} = " << L2error <<  endl;
+      }
+   }
+
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      string keys;
+      if(dim ==2 )
+      {
+         keys = "keys mrRljc\n";
+      }
+      else
+      {
+         keys = "keys mc\n";
+      }
+      socketstream sol_sock_re(vishost, visport);
+      sol_sock_re << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_re.precision(8);
+      sol_sock_re << "solution\n" << *pmesh << p_gf.real() << "window_title 'Numerical Pressure (real part)' " 
+               << keys << flush;
+   }
+
+   delete fespace;
+   delete fec;
+   delete pmesh;
+   MFEMFinalizePetsc();
+   MPI_Finalize();
+   return 0;
+}
+
+//define exact solutions
+void get_helmholtz_solution_Re(const Vector &x, double & p, double dp[], double & d2p)
+{
+
+   if (sol == 0) // polynomial
+   {  
+      if (dim == 3)
+      {
+         p = x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+         dp[0] = (1.0 - 2.0 *x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+         dp[1] = (1.0 - 2.0 *x[1]) * x[0]*(1.0 - x[0]) * x[2]*(1.0 - x[2]);
+         dp[2] = (1.0 - 2.0 *x[2]) * x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]);
+         d2p = -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[1]) * x[1] 
+               -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[2]) * x[2] 
+               -2.0*(-1.0 + x[1]) * x[1] * (-1.0 + x[2]) * x[2];   
+      }
+      else
+      {
+         p = x[1] * (1.0 - x[1])* x[0] * (1.0 - x[0]);
+         dp[0] = (1.0 - 2.0 *x[0]) * x[1]*(1.0 - x[1]);
+         dp[1] = (1.0 - 2.0 *x[1]) * x[0]*(1.0 - x[0]);
+         d2p = - 2.0 * x[1] * (1.0 - x[1]) 
+               - 2.0 * x[0] * (1.0 - x[0]);   
+      }
+   }
+
+   else if(sol == 1) // Plane wave
+   {
+      double alpha;
+      if (dim == 2)
+      {
+         alpha = omega/sqrt(2);
+         p = cos(alpha * ( x(0) + x(1) ) );
+         dp[0] = -alpha * sin(alpha * ( x(0) + x(1) ) );
+         dp[1] = dp[0];
+         d2p = -2.0 * alpha * alpha * p;
+      }
+      else 
+      {
+         alpha = omega/sqrt(3);
+         p = cos(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[0] = -alpha * sin(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[1] = dp[0];
+         dp[2] = dp[0];
+         d2p = -3.0 * alpha * alpha * p;
+      }
+   }
+   else if (sol == 2)
+   {
+      if (dim == 2 )
+      {
+         // shift to avoid singularity
+         double shift = 0.1;
+         if (scatter) shift = -0.5;
+         double x0 = x(0) + shift;
+         double x1 = x(1) + shift;
+         //
+         double r = sqrt(x0 * x0 + x1 * x1);
+
+         p = cos(omega * r);
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_xx = (1.0 / r) * (1.0 - r_x * r_x);
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+
+         dp[0] = - omega * sin(omega * r) * r_x; 
+         dp[1] = - omega * sin(omega * r) * r_y; 
+
+         d2p = -omega*omega * cos(omega * r)*r_x * r_x - omega * sin(omega*r) * r_xx
+               -omega*omega * cos(omega * r)*r_y * r_y - omega * sin(omega*r) * r_yy;
+      }
+      else
+      {
+         // shift to avoid singularity
+         double shift = 0.1;
+         if (scatter) shift = -0.5;
+         double x0 = x(0) + shift;
+         double x1 = x(1) + shift;
+         double x2 = x(2) + shift;
+         //
+         double r = sqrt(x0 * x0 + x1 * x1 + x2 * x2);
+
+         p = cos(omega * r);
+
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_z = x2 / r;
+         double r_xx = (1.0 / r) * (1.0 - r_x * r_x);
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+         double r_zz = (1.0 / r) * (1.0 - r_z * r_z);
+
+         dp[0] = - omega * sin(omega * r) * r_x; 
+         dp[1] = - omega * sin(omega * r) * r_y; 
+         dp[2] = - omega * sin(omega * r) * r_z; 
+
+         d2p = -omega*omega * cos(omega * r)*r_x * r_x - omega * sin(omega*r) * r_xx
+               -omega*omega * cos(omega * r)*r_y * r_y - omega * sin(omega*r) * r_yy
+               -omega*omega * cos(omega * r)*r_z * r_z - omega * sin(omega*r) * r_zz;
+      }
+   }
+}
+
+void get_helmholtz_solution_Im(const Vector &x, double & p, double dp[], double & d2p)
+{
+   if (sol == 0) // polynomial
+   {  
+      if (dim == 3)
+      {
+         p = x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+         dp[0] = (1.0 - 2.0 *x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+         dp[1] = (1.0 - 2.0 *x[1]) * x[0]*(1.0 - x[0]) * x[2]*(1.0 - x[2]);
+         dp[2] = (1.0 - 2.0 *x[2]) * x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]);
+         d2p = -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[1]) * x[1] 
+               -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[2]) * x[2] 
+               -2.0*(-1.0 + x[1]) * x[1] * (-1.0 + x[2]) * x[2];   
+      }
+      else
+      {
+         p = x[1] * (1.0 - x[1])* x[0] * (1.0 - x[0]);
+         dp[0] = (1.0 - 2.0 *x[0]) * x[1]*(1.0 - x[1]);
+         dp[1] = (1.0 - 2.0 *x[1]) * x[0]*(1.0 - x[0]);
+         d2p = - 2.0 * x[1] * (1.0 - x[1]) 
+               - 2.0 * x[0] * (1.0 - x[0]);   
+      }
+   }
+   else if (sol == 1)// plane wave
+   {
+      double alpha; 
+      if (dim == 2)
+      {
+         alpha = omega/sqrt(2);
+         p = -sin(alpha * ( x(0) + x(1) ) );
+         dp[0] = -alpha * cos(alpha * ( x(0) + x(1) ) );
+         dp[1] = dp[0];
+         d2p = -2.0 * alpha * alpha * p;
+      }
+      else 
+      {
+         alpha = omega/sqrt(3);
+         p = -sin(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[0] = -alpha * cos(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[1] = dp[0];
+         dp[2] = dp[0];
+         d2p = -3.0 * alpha * alpha * p;
+      }   
+   }
+}
+
+
+double p_exact_Re(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Re(x, p, dp, d2p);
+   if (scatter)
+   {
+      if (dim == 2)
+      {
+         if (abs(x(0)-1.) < 1e-13 || abs(x(1)-1.0) < 1e-13 ||
+             abs(x(0)) < 1e-13    || abs(x(1)) < 1e-13)
+         {
+            p = 0.0;
+         }
+      }
+      else
+      {
+         if (abs(x(0)-1.) < 1e-13 || abs(x(1)-1.0) < 1e-13 || abs(x(2)-1.0) < 1e-13 ||
+             abs(x(0)) < 1e-13    || abs(x(1)) < 1e-13 || abs(x(2)) < 1e-13)
+         {
+            p = 0.0;
+         }
+      }
+   }
+   return p;
+}
+double p_exact_Im(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Im(x, p, dp, d2p);
+   if (scatter)
+   {
+      if (dim == 2)
+      {
+         if (abs(x(0)-1.) < 1e-13 || abs(x(1)-1.0) < 1e-13 ||
+             abs(x(0)) < 1e-13    || abs(x(1)) < 1e-13)
+         {
+            p = 0.0;
+         }
+      }
+      else
+      {
+         if (abs(x(0)-1.) < 1e-13 || abs(x(1)-1.0) < 1e-13 || abs(x(2)-1.0) < 1e-13 ||
+             abs(x(0)) < 1e-13    || abs(x(1)) < 1e-13 || abs(x(2)) < 1e-13)
+         {
+            p = 0.0;
+         }
+      }
+   }
+   // p *= -1.0;
+   return p;
+}
+
+//calculate RHS from exact solution f = - \Delta u
+double f_exact_Re(const Vector &x)
+{
+   double p_re, d2p_re, p_im, d2p_im;
+   double dp_re[3], dp_im[3];
+   double f_re;
+   f_re = 0.0;
+   if (sol < 0)
+   {
+      double x0 = length/2.0;
+      double x1 = length/2.0;
+      double x2 = length/2.0;
+      double alpha,beta;
+      double n = 5.0 * omega/M_PI;
+      double coeff = pow(n,2)/M_PI;
+      beta = pow(x0-x(0),2) + pow(x1-x(1),2);
+      if (dim == 3) beta += pow(x2-x(2),2);
+      alpha = -pow(n,2) * beta;
+      f_re = coeff*exp(alpha);
+   }
+   else
+   {
+      get_helmholtz_solution_Re(x, p_re, dp_re, d2p_re);
+      get_helmholtz_solution_Im(x, p_im, dp_im, d2p_im);
+      f_re = -d2p_re - omega * omega * p_re;
+   }
+   return f_re;
+}
+double f_exact_Im(const Vector &x)
+{
+   double p_re, d2p_re, p_im, d2p_im;
+   double dp_re[3], dp_im[3];
+   double f_im;
+   f_im = 0.0;
+   if (sol < 0)
+   {
+      // double x0 = 0.6;
+      // double x1 = 0.6;
+      // double alpha;
+      // double n = 5.0 * omega/M_PI;
+      // double coeff = pow(n,2)/M_PI;
+      // alpha = -pow(n,2) * sqrt(pow(x0-x(0),2) + pow(x1-x(1),2));
+      // f_im = coeff*exp(alpha);
+   }
+   else
+   {
+      get_helmholtz_solution_Re(x, p_re, dp_re, d2p_re);
+      get_helmholtz_solution_Im(x, p_im, dp_im, d2p_im);
+      f_im = -d2p_im - omega * omega * p_im;
+   }
+   return f_im;   
+}
+
+void grad_exact_Re(const Vector &x, Vector &dp)
+{
+   double p, d2p;
+   get_helmholtz_solution_Re(x, p, dp, d2p);
+}
+void grad_exact_Im(const Vector &x, Vector &dp)
+{
+   double p, d2p;
+   get_helmholtz_solution_Im(x, p, dp, d2p);
+}
+
+//define impedence coefficient: i omega p
+double g_exact_Re(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Im(x, p, dp, d2p);
+
+   return -omega * p;
+}
+double g_exact_Im(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Re(x, p, dp, d2p);
+
+   return omega * p;
+}
+
+
+
+
+
+
+
+void pml_function(const Vector &x, std::vector<std::complex<double>> & dxs)
+{
+   double L = length;
+   double n = 2.0;
+   double lbeg, lend;
+   double c = 50.0;
+   double c1 = pml_length; 
+   double c2 = length-pml_length; 
+   double coeff;
+   // initialize to one
+   for (int i = 0; i<dim; ++i) dxs[i] = complex<double>(1.0,0.0); 
+
+   if (pml)
+   {
+   // Stretch in each direction independenly
+      for (int i = 0; i<dim; ++i)
+      {
+         if (x(i) >= c2)
+         {
+            lbeg = c2;
+            lend = L;
+            coeff = n * c / omega / pow(lend-lbeg,n); 
+            dxs[i] = complex<double>(1.0,0.0) - complex<double>(0.0,coeff * pow(x(i)-lbeg, n-1.0)); 
+         }
+         if (x(i) <= c1)
+         {
+            lbeg = c1;
+            lend = 0.0;
+            coeff = n * c / omega / pow(lend-lbeg,n); 
+            dxs[i] = complex<double>(1.0,0.0) + complex<double>(0.0, coeff * pow(x(i)-lbeg, n-1.0)); 
+         } 
+      }
+   }
+}
+
+double pml_detJ_Re(const Vector &x)
+{
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0,0.0);
+   pml_function(x, dxs);
+   for (int i=0; i<dim; ++i) det *= dxs[i];
+   return det.real();
+}
+
+double pml_detJ_Im(const Vector &x)
+{
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0,0.0);
+   pml_function(x, dxs);
+   for (int i=0; i<dim; ++i) det *= dxs[i];
+   return det.imag();
+}
+
+void pml_detJ_JT_J_inv_Re(const Vector &x, DenseMatrix &M)
+{
+   std::vector<complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0,0.0);
+   pml_function(x, dxs);
+
+   for (int i = 0; i<dim; ++i)
+   {
+      diag[i] = complex<double>(1.0,0.0) / pow(dxs[i],2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M=0.0;
+
+   for (int i = 0; i<dim; ++i)
+   {
+      complex<double> temp = det * diag[i];
+      M(i,i) = temp.real();
+   }
+}
+
+void pml_detJ_JT_J_inv_Im(const Vector &x, DenseMatrix &M)
+{
+   std::vector<std::complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det = 1.0;
+   pml_function(x, dxs);
+
+   for (int i = 0; i<dim; ++i)
+   {
+      diag[i] = complex<double>(1.0,0.0) / pow(dxs[i],2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M=0.0;
+
+   for (int i = 0; i<dim; ++i)
+   {
+      complex<double> temp = det * diag[i];
+      M(i,i) = temp.imag();
+   }
+}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+// int ndofs = nodes->FESpace()->GetNDofs();
+//    Vector xcoords(ndofs), ycoords(ndofs), zcoords(ndofs);
+
+//    for (int comp = 0; comp < nodes->FESpace()->GetVDim(); comp++)
+//    {
+//       for (int i = 0; i < ndofs; i++)
+//       {
+//          if (comp == 0)
+//          {
+//             xcoords(i) = *nodes[nodes->FESpace()->DofToVDof(i, comp)];
+//          }
+//          else if (comp == 1)
+//          {
+//             ycoords(i) = *nodes[nodes->FESpace()->DofToVDof(i, comp)];
+//          }
+//          else if (comp == 2)
+//          {
+//             zcoords(i) = *nodes[nodes->FESpace()->DofToVDof(i, comp)];
+//          }
+//       }
+//    }

examples/solvers-dev/makefile

@@ -0,0 +1,60 @@
+# Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at the
+# Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights reserved.
+# See file COPYRIGHT for details.
+#
+# This file is part of the MFEM library. For more information and source code
+# availability see http://mfem.org.
+#
+# MFEM is free software; you can redistribute it and/or modify it under the
+# terms of the GNU Lesser General Public License (as published by the Free
+# Software Foundation) version 2.1 dated February 1999.
+
+# Use the MFEM build directory
+MFEM_DIR ?= ../..
+MFEM_BUILD_DIR ?= ../..
+SRC = $(if $(MFEM_DIR:../..=),$(MFEM_DIR)/examples/solvers-dev/,)
+CONFIG_MK = $(MFEM_BUILD_DIR)/config/config.mk
+
+MFEM_LIB_FILE = mfem_is_not_built
+-include $(CONFIG_MK)
+
+SEQ_EXAMPLES = helmholtz  maxwell  maxwell_real  FOSLS_maxwell FOSLS2D_maxwell helmholtzPrimalDPG 
+PAR_EXAMPLES = helmholtzp maxwellp maxwellp_real FOSLS_maxwellp
+ifeq ($(MFEM_USE_MPI),NO)
+   EXAMPLES = $(SEQ_EXAMPLES)
+else
+   EXAMPLES = $(PAR_EXAMPLES) $(SEQ_EXAMPLES)
+endif
+
+.SUFFIXES:
+.SUFFIXES: .o .cpp .mk
+.PHONY: all clean
+.PRECIOUS: %.o
+
+COMMON_O= mg/util.o as/blkschwarzp.o ams/blkams.o as/schwarzp.o as/schwarz.o mg/multigrid.o
+
+# Remove built-in rules
+%: %.cpp
+%.o: %.cpp
+
+all: $(EXAMPLES)
+
+# Rules for building the EXAMPLES
+
+%: $(SRC)%.cpp $(COMMON_O) $(MFEM_LIB_FILE) $(CONFIG_MK)
+	$(MFEM_CXX) $(MFEM_FLAGS) $< -o $@ $(COMMON_O) $(MFEM_LIBS)
+
+# Rules for compiling miniapp dependencies
+$(COMMON_O) $($(EXAMPLES)): \
+%.o: $(SRC)%.cpp $(SRC)%.hpp $(CONFIG_MK)
+	$(MFEM_CXX) $(MFEM_FLAGS) -c $(<) -o $(@)
+
+# Generate an error message if the MFEM library is not built and exit
+$(MFEM_LIB_FILE):
+	$(error The MFEM library is not built)
+
+clean:
+	rm -f *.o *~ $(SEQ_EXAMPLES) $(PAR_EXAMPLES)
+	rm -rf *.dSYM *.TVD.*breakpoints
+
+

examples/solvers-dev/maxwell.cpp

@@ -0,0 +1,497 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include <cmath>
+#include <complex>
+
+using namespace std;
+using namespace mfem;
+
+// #define DEFINITE
+
+// #ifndef MFEM_USE_PETSC
+// #error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+// #endif
+
+// Define exact solution
+void E_exact_Re(const Vector & x, Vector & E);
+void f_exact_Re(const Vector & x, Vector & f);
+void get_maxwell_solution_Re(const Vector & x, double E[], double curl2E[]);
+
+void E_exact_Im(const Vector & x, Vector & E);
+void f_exact_Im(const Vector & x, Vector & f);
+void get_maxwell_solution_Im(const Vector & x, double E[], double curl2E[]);
+
+
+// Mesh Size
+int dim;
+double omega;
+double complex_shift;
+int isol = 1;
+
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Initialise MPI
+   MPI_Session mpi(argc, argv);
+   // 1. Parse command-line options.
+   // geometry file
+   const char *mesh_file = "../../data/one-hex.mesh";
+   int order = 1;
+   // number of wavelengths
+   double k = 0.5;
+   //
+   const char *petscrc_file = "petscrc_mult_options";
+   // visualization flag
+   bool visualization = 1;
+   // number of initial ref
+   int initref = 1;
+   // number of mg levels
+   int maxref = 1;
+   // solver
+   int solver = 1;
+   //
+   complex_shift = 0.0;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree)."); 
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths"); 
+   args.AddOption(&complex_shift, "-cs", "--complex_shift",
+                  "Complex shift");                
+   args.AddOption(&isol, "-isol", "--exact",
+                  "Exact solution flag - 0:polynomial, 1: plane wave");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&maxref, "-maxref", "--maxref",
+                  "Number of Refinements.");   
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization."); 
+   args.AddOption(&solver, "-s", "--solver",
+                  "Solver: 1 - GMG-GMRES, 2 - PETSC, 3 - SUPERLU, 4 - STRUMPACK, 5-HSS-GMRES");                       
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if ( mpi.Root() )
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if ( mpi.Root() )
+   {
+      args.PrintOptions(cout);
+   }
+
+   enum SolverType
+   {
+      INVALID_SOL = -1,
+      GMG_GMRES   =  1,
+      PETSC       =  2,
+      SUPERLU     =  3,
+      STRUMPACK   =  4,
+      HSS_GMRES   =  5,
+   };
+
+   // Angular frequency
+   omega = 2.0*k*M_PI;
+
+   // Create serial mesh 
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   // create parallel mesh and delete the serial one
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // Create H(curl) (Nedelec) Finite element space
+   FiniteElementCollection *fec   = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *ND_fespace = new ParFiniteElementSpace(pmesh, fec);
+
+   std::vector<HypreParMatrix*>  P(maxref);
+
+   for (int i = 0; i < maxref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*ND_fespace);
+      pmesh->UniformRefinement();
+      // Update fespace
+      ND_fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      ND_fespace->GetTrueTransferOperator(cfespace, Tr);
+      Tr.SetOperatorOwner(false);
+      Tr.Get(P[i]);
+   }
+
+   // 7. Linear form b(.) (Right hand side)
+   VectorFunctionCoefficient f_Re(dim, f_exact_Re);
+   VectorFunctionCoefficient f_Im(dim, f_exact_Im);
+   ParComplexLinearForm b(ND_fespace,ComplexOperator::HERMITIAN);
+   b.AddDomainIntegrator(new VectorFEDomainLFIntegrator(f_Re), 
+                         new VectorFEDomainLFIntegrator(f_Im));
+   b.real().Vector::operator=(0.0);
+   b.imag().Vector::operator=(0.0);
+   b.Assemble();
+
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ConstantCoefficient muinv(1.0);
+   ConstantCoefficient sigma(-pow(omega, 2));
+   ConstantCoefficient alpha(complex_shift);
+   ParSesquilinearForm a(ND_fespace, ComplexOperator::HERMITIAN);
+   a.AddDomainIntegrator(new CurlCurlIntegrator(muinv),NULL); 
+   a.AddDomainIntegrator(new VectorFEMassIntegrator(sigma),NULL);
+   a.AddDomainIntegrator(NULL,new VectorFEMassIntegrator(alpha));
+   a.Assemble();
+   a.Finalize();
+
+
+   Array<int> ess_tdof_list;
+   if (pmesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      ND_fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   // Solution grid function
+   ParComplexGridFunction E_gf(ND_fespace);
+   VectorFunctionCoefficient E_Re(dim, E_exact_Re);
+   VectorFunctionCoefficient E_Im(dim, E_exact_Im);
+   E_gf.ProjectCoefficient(E_Re,E_Im);
+
+   OperatorHandle Ah;
+   Vector X, B;
+   a.FormLinearSystem(ess_tdof_list, E_gf, b, Ah, X, B);
+
+   ComplexHypreParMatrix * AZ = Ah.As<ComplexHypreParMatrix>();
+   HypreParMatrix * A = AZ->GetSystemMatrix();
+
+   if ( mpi.Root() )
+   {
+      cout << "Size of fine grid system: "
+           << A->GetGlobalNumRows() << " x " << A->GetGlobalNumCols() << endl;
+   }
+
+   chrono.Clear();
+   chrono.Start();
+  
+   switch((SolverType)solver)
+   {
+      case GMG_GMRES: 
+      {
+         if(mpi.Root()) {cout<< "Solver choice: GMG_GMRES" << endl;}
+#ifdef MFEM_USE_PETSC
+      MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+#endif  
+         ComplexGMGSolver M(AZ, P, ComplexGMGSolver::CoarseSolver::PETSC);
+         M.SetTheta(0.5);
+         M.SetSmootherType(HypreSmoother::Jacobi);
+
+         int maxit(5000);
+         double rtol(1.e-6);
+         double atol(0.0);
+         X = 0.0;
+         GMRESSolver gmres(MPI_COMM_WORLD);
+         gmres.SetAbsTol(atol);
+         gmres.SetRelTol(rtol);
+         gmres.SetMaxIter(maxit);
+         gmres.SetOperator(*AZ);
+         gmres.SetPreconditioner(M);
+         gmres.SetPrintLevel(1);
+         gmres.Mult(B,X);
+#ifdef MFEM_USE_PETSC
+         MFEMFinalizePetsc();
+#endif  
+      }   
+         break;
+      case PETSC:
+          {
+#ifndef MFEM_USE_PETSC
+         MFEM_ABORT("Invalid choice of CoarseSolver. MFEM is not linked with STRUMPACK");
+#else             
+         if(mpi.Root()) {cout<< "Solver choice: PETSC" << endl;}
+         MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+         PetscLinearSolver * invA = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+         PetscParMatrix *PA = new PetscParMatrix(A, Operator::PETSC_MATAIJ);
+         invA->SetOperator(*PA);
+         invA->Mult(B,X);
+         delete PA;
+         MFEMFinalizePetsc();
+#endif         
+      }   
+         break;
+      case SUPERLU:
+      {
+#ifndef MFEM_USE_SUPERLU
+         MFEM_ABORT("Invalid choice of CoarseSolver. MFEM is not linked with STRUMPACK");
+#else         
+         if(mpi.Root()) {cout<< "Solver choice: SuperLU" << endl;}
+         SuperLURowLocMatrix *SA = new SuperLURowLocMatrix(*A);
+         SuperLUSolver * superlu = new SuperLUSolver(MPI_COMM_WORLD);
+         // superlu->SetPrintStatistics(true);
+         // superlu->SetSymmetricPattern(false);
+         superlu->SetColumnPermutation(superlu::PARMETIS);
+         superlu->SetOperator(*SA);
+         superlu->Mult(B,X);
+         delete SA;
+         delete superlu;
+#endif         
+      }   
+         break;
+         case STRUMPACK:
+      {   
+#ifndef MFEM_USE_STRUMPACK
+         MFEM_ABORT("Invalid choice of CoarseSolver. MFEM is not linked with STRUMPACK");
+#else         
+         if(mpi.Root()) {cout<< "Solver choice: STRUMPACK" << endl;}
+         STRUMPACKRowLocMatrix *SA = new STRUMPACKRowLocMatrix(*A);
+         STRUMPACKSolver * strumpack = new STRUMPACKSolver(argc, argv, MPI_COMM_WORLD);
+         strumpack->SetPrintFactorStatistics(false);
+         strumpack->SetPrintSolveStatistics(true);
+         strumpack->SetHSS(true);
+         strumpack->SetHssAbsTol(0.0);
+         strumpack->SetHssRelTol(1e-4);
+         strumpack->SetAbsTol(0.0);
+         strumpack->SetRelTol(1e-6);
+         strumpack->SetKrylovSolver(strumpack::KrylovSolver::AUTO);
+         strumpack->SetReorderingStrategy(strumpack::ReorderingStrategy::METIS);
+         strumpack->DisableMatching();
+         strumpack->SetOperator(*SA);
+         strumpack->SetFromCommandLine();
+         strumpack->Mult(B, X);
+         delete SA;
+         delete strumpack;
+#endif         
+      }     
+         break  ;   
+      case HSS_GMRES:
+      {   
+#ifndef MFEM_USE_STRUMPACK
+         MFEM_ABORT("Invalid choice of CoarseSolver. MFEM is not linked with STRUMPACK");
+#else       
+         if(mpi.Root()) {cout<< "Solver choice: STRUMPACK" << endl;}
+         STRUMPACKRowLocMatrix *SA = new STRUMPACKRowLocMatrix(*A);
+         STRUMPACKSolver * prec = new STRUMPACKSolver(argc, argv, MPI_COMM_WORLD);
+         prec->SetPrintFactorStatistics(true);
+         prec->SetPrintSolveStatistics(false);
+         prec->SetHSS(true);
+         prec->SetHssAbsTol(0.0);
+         prec->SetHssRelTol(1e-4);
+         prec->SetKrylovSolver(strumpack::KrylovSolver::DIRECT);
+         prec->SetReorderingStrategy(strumpack::ReorderingStrategy::METIS);
+         prec->DisableMatching();
+         prec->SetOperator(*SA);
+         prec->SetFromCommandLine();
+         
+         int maxit(50);
+         double rtol(1.e-6);
+         double atol(0.0);
+         GMRESSolver gmres(MPI_COMM_WORLD);
+         gmres.SetAbsTol(atol);
+         gmres.SetRelTol(rtol);
+         gmres.SetMaxIter(maxit);
+         gmres.SetOperator(*A);
+         gmres.SetPreconditioner(*prec);
+         gmres.SetPrintLevel(1);
+         gmres.Mult(B,X);
+
+         delete SA;
+         delete prec;
+#endif
+      }     
+         break  ;   
+
+      default: 
+         if(mpi.Root()) {cout<< "Solver choice not valid. Problem not solved" << endl;}
+   }
+
+   chrono.Stop();
+   if (mpi.Root())
+   {
+      cout << "Solver time: " << chrono.RealTime() << endl;
+   }
+   a.RecoverFEMSolution(X,B,E_gf);
+
+
+   // Compute error
+   int order_quad = max(2, 2 * order + 1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i = 0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double L2Error_Re = E_gf.real().ComputeL2Error(E_Re, irs);
+   double norm_E_Re = ComputeGlobalLpNorm(2, E_Re, *pmesh, irs);
+
+   double L2Error_Im = E_gf.imag().ComputeL2Error(E_Im, irs);
+   double norm_E_Im = ComputeGlobalLpNorm(2, E_Im, *pmesh, irs);
+
+
+   if (mpi.Root())
+   {
+      cout << " Real Part: || E_h - E || / ||E|| = " << L2Error_Re / norm_E_Re << '\n' << endl;
+      cout << " Imag Part: || E_h - E || / ||E|| = " << L2Error_Im / norm_E_Im << '\n' << endl;
+
+      cout << " Real Part: || E_h - E || = " << L2Error_Re << '\n' << endl;
+      cout << " Imag Part: || E_h - E || = " << L2Error_Im << '\n' << endl;
+   }
+
+   // visualization   
+   if (visualization)
+   {
+      int num_procs, myid;
+      MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+      MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << E_gf.real() << "window_title 'Real part'" << flush;
+
+      socketstream sol_sock_Im(vishost, visport);
+      sol_sock_Im << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_Im.precision(8);
+      sol_sock_Im << "solution\n" << *pmesh << E_gf.imag() << "window_title 'Imaginary part'" << flush;
+   }
+
+   // // delete invA;
+   delete fec;
+   delete ND_fespace;
+   delete pmesh;
+   return 0;
+}
+//define exact solution
+void E_exact_Re(const Vector &x, Vector &E)
+{
+   double curl2E[3];
+   get_maxwell_solution_Re(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+void f_exact_Re(const Vector &x, Vector &f)
+{
+   double E_Re[3], curl2E_Re[3];
+   double E_Im[3], curl2E_Im[3];
+
+   get_maxwell_solution_Re(x, E_Re, curl2E_Re);
+   get_maxwell_solution_Re(x, E_Im, curl2E_Im);
+
+   // curl ( curl E) - omega^2 E = f
+   double coeff;
+   coeff = -omega * omega;
+   f(0) = curl2E_Re[0] + coeff * E_Re[0];
+   f(1) = curl2E_Re[1] + coeff * E_Re[1];
+   f(2) = curl2E_Re[2] + coeff * E_Re[2];
+
+   // Account for the complex shift
+   f(0) += -complex_shift*E_Im[0];
+   f(1) += -complex_shift*E_Im[1];
+   f(2) += -complex_shift*E_Im[2];
+}
+
+void get_maxwell_solution_Re(const Vector & x, double E[], double curl2E[])
+{
+
+   if (isol == 0) // polynomial
+   {
+      E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+      E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+      E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+      curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+      curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+      curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+   }
+   else
+   {
+      double alpha = omega / sqrt(3);
+      E[0] = cos(alpha*(x(0) + x(1) + x(2)));
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+      curl2E[0] = 2.0 * alpha * alpha * E[0];
+      curl2E[1] = -alpha * alpha * E[0];
+      curl2E[2] = -alpha * alpha * E[0];
+   }
+}
+
+
+
+//define exact solution
+void E_exact_Im(const Vector &x, Vector &E)
+{
+   double curl2E[3];
+   get_maxwell_solution_Re(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+void f_exact_Im(const Vector &x, Vector &f)
+{
+   double E_Re[3], curl2E_Re[3];
+   double E_Im[3], curl2E_Im[3];
+
+   get_maxwell_solution_Re(x, E_Im, curl2E_Im);
+   get_maxwell_solution_Re(x, E_Re, curl2E_Re);
+
+   // curl ( curl E) - omega^2 E = f
+   double coeff;
+   coeff = -omega * omega;
+   f(0) = curl2E_Im[0] + coeff * E_Im[0];
+   f(1) = curl2E_Im[1] + coeff * E_Im[1];
+   f(2) = curl2E_Im[2] + coeff * E_Im[2];
+
+   // Acount for the complex shift
+   f(0) += complex_shift*E_Re[0];
+   f(1) += complex_shift*E_Re[1];
+   f(2) += complex_shift*E_Re[2];
+}
+
+void get_maxwell_solution_Im(const Vector & x, double E[], double curl2E[])
+{
+   if (isol == 0) // polynomial
+   {
+      E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+      E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+      E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+      curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+      curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+      curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+   }
+   else
+   {
+      double alpha = omega / sqrt(3);   
+      E[0] = sin(alpha * (x(0) + x(1) + x(2)));
+      E[1] = 0.0;
+      E[2] = 0.0;
+      curl2E[0] = 2.0 * alpha * alpha * E[0];
+      curl2E[1] = -alpha * alpha * E[0];
+      curl2E[2] = -alpha * alpha * E[0];
+   }   
+}

examples/solvers-dev/maxwell_real.cpp

@@ -0,0 +1,432 @@
+//                                MFEM Example multigrid-grid Cycle
+//
+// Compile with: make mg_maxwellp
+//
+// Sample runs:  mg_maxwellp -m ../data/one-hex.mesh
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include "as/schwarz.hpp"
+
+using namespace std;
+using namespace mfem;
+
+// #define DEFINITE
+
+// Define exact solution
+void E_exact(const Vector & x, Vector & E);
+void f_exact(const Vector & x, Vector & f);
+void get_maxwell_solution(const Vector & x, double E[], double curl2E[]);
+
+int dim;
+double omega;
+int isol = 1;
+
+
+int main(int argc, char *argv[])
+{
+    // 1. Parse command-line options.
+   const char *mesh_file = "../../data/one-hex.mesh";
+   int order = 1;
+   // int sdim = 2;
+   bool static_cond = false;
+   const char *device_config = "cpu";
+   bool visualization = true;
+   int ref_levels = 1;
+   int initref    = 1;
+   // number of wavelengths
+   double k = 0.5;
+   int nd = 3;
+   StopWatch chrono;
+
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&nd, "-nd", "--dimension", "Dimension");
+   args.AddOption(&ref_levels, "-sr", "--serial-refinements", 
+                  "Number of mesh refinements");
+   args.AddOption(&initref, "-iref", "--init-refinements", 
+                  "Number of initial mesh refinements");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&isol, "-sol", "--solution", 
+                  "Exact Solution: 0) Polynomial, 1) Sinusoidal.");               
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   // Angular frequency
+   omega = 2.0 * M_PI * k;
+
+   // 3. Read the mesh from the given mesh file. 
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+
+   Mesh * mesh; 
+   // Define a simple square or cubic mesh
+   if (nd == 2)
+   {
+      mesh = new Mesh(1, 1, Element::QUADRILATERAL, true,1.0, 1.0,false);
+      // mesh = new Mesh(1, 1, Element::TRIANGLE, true,1.0, 1.0,false);
+   }
+   else
+   {
+      mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true,1.0, 1.0,1.0, false);
+      // mesh = new Mesh(mesh_file, 1, 1);
+   }
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+   for (int i=0; i<initref; i++) {mesh->UniformRefinement();}
+
+
+   Mesh * cmesh = new Mesh(*mesh);
+
+   for (int l = 0; l < ref_levels; l++)
+   {
+      mesh->UniformRefinement();
+   }
+
+
+   // 4. Define a finite element space on the mesh.
+   FiniteElementCollection *fec   = new ND_FECollection(order, dim);
+   FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
+
+   Array<int> ess_tdof_list;
+   if (mesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(mesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+
+   ConstantCoefficient muinv(1.0);
+#ifdef DEFINITE
+   ConstantCoefficient sigma(pow(omega, 2));
+#else
+   ConstantCoefficient sigma(-pow(omega, 2));
+#endif
+   // 6. Linear form (i.e RHS b = (f,v) = (1,v))
+   LinearForm *b = new LinearForm(fespace);
+   VectorFunctionCoefficient f(sdim, f_exact);
+   b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
+   b->Assemble();
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   BilinearForm *a = new BilinearForm(fespace);
+   a->AddDomainIntegrator(new CurlCurlIntegrator(muinv)); // one is the coeff
+   a->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   if (static_cond) { a->EnableStaticCondensation(); }
+   a->Assemble();
+
+   GridFunction x(fespace);
+   x = 0.0;
+   VectorFunctionCoefficient E_ex(sdim, E_exact);
+   x.ProjectCoefficient(E_ex);
+
+   SparseMatrix A;
+   Vector B, X;
+
+   a->SetDiagonalPolicy(mfem::Matrix::DIAG_ONE);
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+   cout << "Size of linear system: " << A.Height() << endl;
+
+   FiniteElementSpace *prec_fespace = (a->StaticCondensationIsEnabled() ? a->SCFESpace() : fespace);
+
+
+
+   chrono.Clear();
+   chrono.Start();
+   SchwarzSmoother * prec = new SchwarzSmoother(cmesh,ref_levels, prec_fespace, &A, ess_tdof_list);
+   prec->SetNumSmoothSteps(1);
+   prec->SetDumpingParam(1.0/2.0);
+
+   chrono.Stop();
+   // Need to invastigate the time scalings. TODO
+   cout << "Preconditioner construction time " << chrono.RealTime() << "s. \n";
+   
+   // DSmoother M(A);
+   // GSSmoother M(A);
+   X = 0.0;
+   int maxit(1000);
+   double rtol(0.0);
+   double atol(1.e-12);
+   GMRESSolver solver;
+   solver.SetAbsTol(atol);
+   solver.SetRelTol(rtol);
+   solver.SetMaxIter(maxit);
+   solver.SetPreconditioner(*prec);
+   solver.SetOperator(A);
+   solver.SetPrintLevel(1);
+   
+   chrono.Clear();
+   chrono.Start();
+   solver.Mult(B,X);
+   chrono.Stop();
+   cout << "Solver time: " << chrono.RealTime() << endl;
+
+
+   // UMFPackSolver * invA = new UMFPackSolver;
+   // invA->Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+   // invA->SetOperator(A);
+   // invA->Mult(B,X);
+
+
+   // delete invA;
+
+
+
+   a->RecoverFEMSolution(X, *b, x);
+
+
+
+
+
+
+   GridFunction Egf(fespace);
+   Egf.ProjectCoefficient(E_ex);
+
+
+
+
+
+
+
+   int order_quad = max(2, 2 * order + 1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i = 0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+   double L2Error = x.ComputeL2Error(E_ex, irs);
+
+   double norm_E = ComputeLpNorm(2, E_ex, *mesh, irs);
+
+   cout << "\n || E_h - E || / ||E|| = " << L2Error / norm_E << '\n' << endl;
+
+   // if (visualization)
+   // {
+   //    char vishost[] = "localhost";
+   //    int  visport   = 19916;
+   //    socketstream sol_sock(vishost, visport);
+   //    sol_sock.precision(8);
+   //    sol_sock <<  "mesh\n" << *cmesh  << "keys n\n" << flush;
+   // }
+
+   //  if (visualization)
+   // {
+   //    char vishost[] = "localhost";
+   //    int  visport   = 19916;
+   //    socketstream sol_sock(vishost, visport);
+   //    sol_sock.precision(8);
+   //    sol_sock <<  "mesh\n" << *mesh  << "keys n\n" << flush;
+   // }
+
+
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      if (dim == 2) 
+      {
+         sol_sock <<  "solution\n" << *mesh << x  << "keys rRljc\n" << flush;
+      }
+      else
+      {
+         sol_sock <<  "solution\n" << *mesh << x  << "keys lc\n" << flush;
+      }
+   }
+
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      if (dim == 2) 
+      {
+         sol_sock <<  "solution\n" << *mesh << Egf  << "keys rRljc\n" << flush;
+      }
+      else
+      {
+         sol_sock <<  "solution\n" << *mesh << Egf  << "keys lc\n" << flush;
+      }
+   }
+   delete a;
+   delete b;
+   delete fec;
+   delete fespace;
+   delete mesh;
+   return 0;
+}
+
+
+//define exact solution
+void E_exact(const Vector &x, Vector &E)
+{
+   double curl2E[3];
+   get_maxwell_solution(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+// f = curl (mu curl E ) + omega^2*E
+void f_exact(const Vector &x, Vector &f)
+{
+   double coeff;
+#ifdef DEFINITE
+   coeff = omega * omega;
+#else
+   coeff = -omega * omega;
+#endif
+   double E[3], curl2E[3];
+   get_maxwell_solution(x, E, curl2E);
+   // curl ( curl E) +/- omega^2 E = f
+   f(0) = curl2E[0] + coeff * E[0];
+   f(1) = curl2E[1] + coeff * E[1];
+   if (dim == 2)
+   {
+      if (x.Size() == 3) {f(2)=0.0;}
+   }
+   else
+   {
+      f(2) = curl2E[2] + coeff * E[2];
+   }
+}
+
+void get_maxwell_solution(const Vector & x, double E[], double curl2E[])
+{
+   if (isol == 0) // polynomial
+   {
+      if (dim == 2)
+      {
+         E[0] = x[0] * (1.0 - x[0]) * x[1] * (1.0 - x[1]);
+         E[1] = 0.0;
+         //
+         curl2E[0] =  - 2.0 * x[0] * (x[0] - 1.0);
+         curl2E[1] = (2.0*x[0]-1.0)*(2.0*x[1]-1);
+         curl2E[2] = 0.0;
+      }
+      else
+      {
+         // Polynomial vanishing on the boundary
+         E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+         E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+         E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+         //
+         curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+         curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+         curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+      }
+   }
+
+   else if (isol == 1) // sinusoidal
+   {
+      if (dim == 2)
+      {
+         E[0] = sin(omega * x[1]);
+         E[1] = sin(omega * x[0]);
+         curl2E[0] = omega * omega * E[0];
+         curl2E[1] = omega * omega * E[1];
+         curl2E[2] = 0.0;
+      }
+      else
+      {
+         E[0] = sin(omega * x[1]);
+         E[1] = sin(omega * x[2]);
+         E[2] = sin(omega * x[0]);
+
+         curl2E[0] = omega * omega * E[0];
+         curl2E[1] = omega * omega * E[1];
+         curl2E[2] = omega * omega * E[2];
+      }
+   }
+   else if (isol == 2) //point source
+   {
+      if (dim == 2)
+      {
+         // shift to avoid singularity
+         double x0 = x(0) + 0.1;
+         double x1 = x(1) + 0.1;
+         //
+         double r = sqrt(x0 * x0 + x1 * x1);
+
+         E[0] = cos(omega * r);
+         E[1] = 0.0;
+
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_xy = -(r_x / r) * r_y;
+         double r_yx = r_xy;
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+
+         curl2E[0] = omega * ((r_yy ) * sin(omega * r) + (omega * r_y * r_y) * cos(omega * r));
+         curl2E[1] = -omega * (r_yx * sin(omega * r) + omega * r_y * r_x * cos(omega * r));
+         curl2E[2] = 0.0;
+      }
+      else
+      {
+      // shift to avoid singularity
+         double x0 = x(0) + 0.1;
+         double x1 = x(1) + 0.1;
+         double x2 = x(2) + 0.1;
+         //
+         double r = sqrt(x0 * x0 + x1 * x1 + x2 * x2);
+
+         E[0] = cos(omega * r);
+         E[1] = 0.0;
+         E[2] = 0.0;
+
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_z = x2 / r;
+         double r_xy = -(r_x / r) * r_y;
+         double r_xz = -(r_x / r) * r_z;
+         double r_yx = r_xy;
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+         double r_zx = r_xz;
+         double r_zz = (1.0 / r) * (1.0 - r_z * r_z);
+
+         curl2E[0] = omega * ((r_yy + r_zz) * sin(omega * r) +
+                              (omega * r_y * r_y + omega * r_z * r_z) * cos(omega * r));
+         curl2E[1] = -omega * (r_yx * sin(omega * r) + omega * r_y * r_x * cos(omega * r));
+         curl2E[2] = -omega * (r_zx * sin(omega * r) + omega * r_z * r_x * cos(omega * r));
+      }
+   }
+   else if (isol == 3) // plane wave
+   {
+      if (dim == 2)
+      {
+         E[0] = cos(omega * (x(0) + x(1)) / sqrt(2.0));
+         E[1] = 0.0;
+
+         curl2E[0] = omega * omega * E[0] / 2.0;
+         curl2E[1] = -omega * omega * E[0] / 2.0;
+      }
+      else
+      {
+         E[0] = cos(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+         E[1] = 0.0;
+         E[2] = 0.0;
+
+         curl2E[0] = 2.0 * omega * omega * E[0] / 3.0;
+         curl2E[1] = -omega * omega * E[0] / 3.0;
+         curl2E[2] = -omega * omega * E[0] / 3.0;
+      }
+   }
+
+}

examples/solvers-dev/maxwellp.cpp

@@ -0,0 +1,661 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+
+// #ifndef MFEM_USE_PETSC
+// #error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+// #endif
+
+// Define exact solution
+void E_exact_Re(const Vector & x, Vector & E);
+void f_exact_Re(const Vector & x, Vector & f);
+void get_maxwell_solution_Re(const Vector & x, double E[], double curl2E[]);
+
+void E_exact_Im(const Vector & x, Vector & E);
+void f_exact_Im(const Vector & x, Vector & f);
+
+double pml_detJ_inv_Re(const Vector &x);
+double pml_detJ_inv_Im(const Vector &x);
+void pml_detJ_JT_J_inv_Re(const Vector &x, DenseMatrix &M);
+void pml_detJ_JT_J_inv_Im(const Vector &x, DenseMatrix &M);
+void pml_detJ_inv_JT_J_Re(const Vector &x, DenseMatrix &M);
+void pml_detJ_inv_JT_J_Im(const Vector &x, DenseMatrix &M);
+
+// Mesh Size
+int dim;
+double omega;
+int sol = 1;
+bool scatter = false;
+bool pml = false;
+double length = 1.0;
+double pml_length = 0.25;
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Initialise MPI
+   MPI_Session mpi(argc, argv);
+   // 1. Parse command-line options.
+   // geometry file
+   const char *mesh_file = "../../data/one-hex.mesh";
+   int order = 1;
+   // number of wavelengths
+   double k = 0.5;
+   //
+   const char *petscrc_file = "petscrc_mult_options";
+   // visualization flag
+   bool visualization = 1;
+   // number of initial ref
+   int initref = 1;
+   // number of mg levels
+   int ref = 1;
+   // dimension
+   int nd = 2;
+   //
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree)."); 
+   args.AddOption(&nd, "-nd", "--dim","Problem space dimension");   
+   args.AddOption(&pml, "-pml", "--pml", "-no-pml",
+                  "--no-pml", "Enable PML.");         
+   args.AddOption(&pml_length, "-pml_length", "--pml_length",
+                  "Length of the PML region in each direction");        
+   args.AddOption(&length, "-length", "--length",
+                  "length of the domainin in each direction.");                  
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths"); 
+   args.AddOption(&sol, "-sol", "--exact",
+                  "Exact solution flag - 0:polynomial, 1: plane wave");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&ref, "-ref", "--refinements",
+                  "Number of Refinements.");   
+   args.AddOption(&scatter, "-scat", "--scattering-prob", "-no-scat",
+                  "--no-scattering", "Solve a scattering problem");                  
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization."); 
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if ( mpi.Root() )
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if ( mpi.Root() )
+   {
+      args.PrintOptions(cout);
+   }
+
+   // Angular frequency
+   omega = 2.0*k*M_PI;
+
+   Mesh *mesh;
+
+   // Create serial mesh 
+   double l = 1.0;
+   if (nd == 2) 
+   {
+      if (scatter) 
+      {
+         mesh_file = "../../data/rectwhole7_2attr.e";
+         mesh = new Mesh(mesh_file, 1, 1);
+      }
+      else
+      {
+         mesh = new Mesh(1, 1, Element::QUADRILATERAL, true, l, l, false);
+      }
+   }
+   else
+   {
+      if (scatter) 
+      {
+         mesh_file = "../../data/hexwhole7.e";
+         mesh = new Mesh(mesh_file, 1, 1);
+      }
+      else
+      {
+         mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true, l, l, l, false);
+      }
+   }
+    // normalize mesh
+   mesh->EnsureNodes();
+   GridFunction * nodes = mesh->GetNodes();
+   // Assuming square/cubic domain 
+   double min_coord =  nodes->Min();
+   double max_coord =  nodes->Max();
+   double domain_length = abs(max_coord-min_coord); 
+   // shift to zero
+   *nodes -= min_coord;
+   // scale to one
+   *nodes *= 1./domain_length;
+
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   // create parallel mesh and delete the serial one
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // Create H(curl) (Nedelec) Finite element space
+   FiniteElementCollection *fec   = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *ND_fespace = new ParFiniteElementSpace(pmesh, fec);
+
+   for (int i = 0; i < ref; i++)
+   {
+      pmesh->UniformRefinement();
+      // Update fespace
+      ND_fespace->Update();
+   }
+
+   // 7. Linear form b(.) (Right hand side)
+   VectorFunctionCoefficient f_Re(dim, f_exact_Re);
+   VectorFunctionCoefficient f_Im(dim, f_exact_Im);
+   ParComplexLinearForm b(ND_fespace,ComplexOperator::HERMITIAN);
+   if (! scatter)
+   {
+      b.AddDomainIntegrator(new VectorFEDomainLFIntegrator(f_Re), 
+                            new VectorFEDomainLFIntegrator(f_Im));
+   }
+   b.real().Vector::operator=(0.0);
+   b.imag().Vector::operator=(0.0);
+   b.Assemble();
+
+   // setup coefficients
+   ConstantCoefficient muinv(1.0);
+   ConstantCoefficient sigma(-pow(omega, 2));
+   // pml coefficients
+   FunctionCoefficient det_inv_Re(pml_detJ_inv_Re);
+   FunctionCoefficient det_inv_Im(pml_detJ_inv_Im);
+
+
+   MatrixFunctionCoefficient c1_Re(dim,pml_detJ_inv_JT_J_Re);
+   MatrixFunctionCoefficient c1_Im(dim,pml_detJ_inv_JT_J_Im);
+
+   MatrixFunctionCoefficient temp_c2_Re(dim,pml_detJ_JT_J_inv_Re);
+   MatrixFunctionCoefficient temp_c2_Im(dim,pml_detJ_JT_J_inv_Im);
+
+   ScalarMatrixProductCoefficient c2_Re(sigma,temp_c2_Re);
+   ScalarMatrixProductCoefficient c2_Im(sigma,temp_c2_Im);
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ParSesquilinearForm a(ND_fespace, ComplexOperator::HERMITIAN);
+
+   // a.AddDomainIntegrator(new CurlCurlIntegrator(muinv),NULL); 
+   // a.AddDomainIntegrator(new VectorFEMassIntegrator(sigma),NULL);
+   if (dim == 3)
+   {
+      a.AddDomainIntegrator(new CurlCurlIntegrator(c1_Re),
+                            new CurlCurlIntegrator(c1_Im)); 
+   }
+   else
+   {
+      a.AddDomainIntegrator(new CurlCurlIntegrator(det_inv_Re),
+                            new CurlCurlIntegrator(det_inv_Im)); 
+   }
+   a.AddDomainIntegrator(new VectorFEMassIntegrator(c2_Re),
+                         new VectorFEMassIntegrator(c2_Im));
+
+   
+   a.Assemble();
+   a.Finalize();
+
+
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+   ess_bdr = 1;
+   ND_fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+
+   // Solution grid function
+   ParComplexGridFunction E_gf(ND_fespace);
+   E_gf = 0.0;
+   VectorFunctionCoefficient E_Re(dim, E_exact_Re);
+   VectorFunctionCoefficient E_Im(dim, E_exact_Im);
+   // if (sol >=0)  E_gf.ProjectBdrCoefficientTangent(E_Re,E_Im,ess_bdr);
+   if (sol >=0)  E_gf.ProjectBdrCoefficientTangent(E_Re,E_Re,ess_bdr);
+   // E_gf.ProjectCoefficient(E_Re,E_Im);
+
+   OperatorHandle Ah;
+   Vector X, B;
+   a.FormLinearSystem(ess_tdof_list, E_gf, b, Ah, X, B);
+
+   ComplexHypreParMatrix * AZ = Ah.As<ComplexHypreParMatrix>();
+   HypreParMatrix * A = AZ->GetSystemMatrix();
+
+   if ( mpi.Root() )
+   {
+      cout << "Size of fine grid system: "
+           << A->GetGlobalNumRows() << " x " << A->GetGlobalNumCols() << endl;
+   }
+
+   chrono.Clear();
+   chrono.Start();
+  
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+   PetscLinearSolver * invA = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   PetscParMatrix *PA = new PetscParMatrix(A, Operator::PETSC_MATAIJ);
+   invA->SetOperator(*PA);
+   invA->Mult(B,X);
+   delete PA;
+   MFEMFinalizePetsc();
+   a.RecoverFEMSolution(X,B,E_gf);
+
+
+   // Compute error
+   if (sol >= 0 && !pml)
+   {
+      int order_quad = max(2, 2 * order + 1);
+      const IntegrationRule *irs[Geometry::NumGeom];
+      for (int i = 0; i < Geometry::NumGeom; ++i)
+      {
+         irs[i] = &(IntRules.Get(i, order_quad));
+      }
+
+      double L2Error_Re = E_gf.real().ComputeL2Error(E_Re, irs);
+      double norm_E_Re = ComputeGlobalLpNorm(2, E_Re, *pmesh, irs);
+
+      double L2Error_Im = E_gf.imag().ComputeL2Error(E_Im, irs);
+      double norm_E_Im = ComputeGlobalLpNorm(2, E_Im, *pmesh, irs);
+
+      if (mpi.Root())
+      {
+         cout << " Real Part: || E_h - E || / ||E|| = " << L2Error_Re / norm_E_Re << '\n' << endl;
+         cout << " Imag Part: || E_h - E || / ||E|| = " << L2Error_Im / norm_E_Im << '\n' << endl;
+
+         cout << " Real Part: || E_h - E || = " << L2Error_Re << '\n' << endl;
+         cout << " Imag Part: || E_h - E || = " << L2Error_Im << '\n' << endl;
+      }
+   }
+
+   // visualization   
+   if (visualization)
+   {
+      int num_procs, myid;
+      MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+      MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      string keys;
+      if (dim ==3) 
+      {
+         keys = "keys mF\n";
+      }
+      else
+      {
+         keys = "keys mrRljcUUuu\n";
+      }
+      sol_sock << "solution\n" << *pmesh << E_gf.real() << "window_title 'Real part'" 
+      << keys << flush;
+   }
+   delete fec;
+   delete ND_fespace;
+   delete pmesh;
+   return 0;
+}
+//define exact solution
+void E_exact_Re(const Vector &x, Vector &E)
+{
+   double curl2E[3];
+   get_maxwell_solution_Re(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+void f_exact_Re(const Vector &x, Vector &f)
+{
+   double E_Re[3], curl2E_Re[3];
+   double E_Im[3], curl2E_Im[3];
+
+   get_maxwell_solution_Re(x, E_Re, curl2E_Re);
+   get_maxwell_solution_Re(x, E_Im, curl2E_Im);
+
+   // curl ( curl E) - omega^2 E = f
+   double coeff;
+   coeff = -omega * omega;
+   f(0) = curl2E_Re[0] + coeff * E_Re[0];
+   f(1) = curl2E_Re[1] + coeff * E_Re[1];
+   if (dim == 3)
+   {
+      f(2) = curl2E_Re[2] + coeff * E_Re[2];
+   }
+
+   if (sol < 0)
+   {
+      double x0 = length/2.0;
+      double x1 = length/2.0;
+      double x2 = length/2.0;
+      double alpha,beta;
+      double n = 5.0 * omega/M_PI;
+      double coeff = pow(n,2)/M_PI;
+      beta = pow(x0-x(0),2) + pow(x1-x(1),2);
+      if (dim == 3) beta += pow(x2-x(2),2);
+      alpha = -pow(n,2) * beta;
+      f = 0.0;
+      f(0) = coeff*exp(alpha);
+      f(1) = coeff*exp(alpha);
+      if (dim == 3) f(2) = coeff*exp(alpha);
+   }
+
+}
+
+void get_maxwell_solution_Re(const Vector & x, double E[], double curl2E[])
+{
+   if (sol == 0) // polynomial
+   {
+      if (dim == 2)
+      {
+         E[0] = x[1] * (1.0 - x[1])* x[0] * (1.0 - x[0]);
+         E[1] = x[1] * (1.0 - x[1])* x[0] * (1.0 - x[0]);
+         E[2] = 0.0;
+         curl2E[0] = -2.0 * x[0] * x[0] + 4.0*x[0]*x[1] - 2.0*x[1] + 1.0;
+         curl2E[1] = x[0] * (4.0 * x[1] - 2.0) - 2.0 * x[1] * x[1] + 1.0;
+         curl2E[2] = 0.0;
+      }
+      else
+      {
+         E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+         E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+         E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+         curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+         curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+         curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+      }
+   }
+   else if (sol == 1)
+   {
+      if (dim == 2)
+      {
+         double alpha = omega / sqrt(2);
+         E[0] = cos(alpha*(x(0) + x(1)));
+         E[1] = 0.0;
+         E[2] = 0.0;
+         curl2E[0] = alpha * alpha * E[0];
+         curl2E[1] = -alpha * alpha * E[0];
+         curl2E[2] = 0.0;
+      }
+      else
+      {
+         double alpha = omega / sqrt(3);
+         E[0] = cos(alpha*(x(0) + x(1) + x(2)));
+         E[1] = 0.0;
+         E[2] = 0.0;
+
+         curl2E[0] = 2.0 * alpha * alpha * E[0];
+         curl2E[1] = -alpha * alpha * E[0];
+         curl2E[2] = -alpha * alpha * E[0];
+      }
+   }
+   else if (sol == 2)
+   {
+      if (dim == 2)
+      {
+         double shift = 0.1;
+         if (scatter) shift = -0.5;
+         double x0 = x(0) + shift;
+         double x1 = x(1) + shift;
+         double r = sqrt(x0 * x0 + x1 * x1);
+         E[0] = cos(omega*r);
+         E[1] = 0.0;
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_xy = -(r_x / r) * r_y;
+         double r_yx = r_xy;
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+         curl2E[0] =  omega*(r_yy * sin(omega * r) + omega * r_y * r_y * cos(omega * r));
+         curl2E[1] = -omega*(r_yx * sin(omega * r) + omega * r_y * r_x * cos(omega * r));
+      }
+      else
+      {
+         double shift = 0.1;
+         if (scatter) shift = -0.5;
+         double x0 = x(0) + shift;
+         double x1 = x(1) + shift;
+         double x2 = x(2) + shift;
+         double r = sqrt(x0 * x0 + x1 * x1 + x2 * x2);
+         E[0] = cos(omega * r);
+         E[1] = 0.0;
+         E[2] = 0.0;
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_z = x2 / r;
+         double r_xy = -(r_x / r) * r_y;
+         double r_xz = -(r_x / r) * r_z;
+         double r_yx = r_xy;
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+         double r_zx = r_xz;
+         double r_zz = (1.0 / r) * (1.0 - r_z * r_z);
+
+         curl2E[0] = omega * ((r_yy + r_zz) * sin(omega * r) +
+                              (omega * r_y * r_y + omega * r_z * r_z) * cos(omega * r));
+         curl2E[1] = -omega * (r_yx * sin(omega * r) + omega * r_y * r_x * cos(omega * r));
+         curl2E[2] = -omega * (r_zx * sin(omega * r) + omega * r_z * r_x * cos(omega * r));
+      }
+   }
+   if (pml)
+   {
+      if (abs(x(0)-1.) < 1e-13 || abs(x(1)-1.0) < 1e-13 ||
+          abs(x(0)) < 1e-13    || abs(x(1)) < 1e-13)
+      {
+         E[0] = 0.0;
+         E[1] = 0.0;
+         curl2E[0] = 0.0;
+         curl2E[1] = 0.0;
+      }
+      if (dim == 3)
+      {
+         if (abs(x(2)-1.) < 1e-13 || abs(x(2)) < 1e-13) 
+         {
+            E[0] = 0.0;
+            E[1] = 0.0;
+            E[2]=0.0;
+            curl2E[0] = 0.0;
+            curl2E[1] = 0.0;
+            curl2E[2] = 0.0;
+         }
+      }
+   }
+}
+
+//define exact solution
+void E_exact_Im(const Vector &x, Vector &E)
+{
+   double curl2E[3];
+   get_maxwell_solution_Re(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+void f_exact_Im(const Vector &x, Vector &f)
+{
+   double E_Re[3], curl2E_Re[3];
+   double E_Im[3], curl2E_Im[3];
+
+   get_maxwell_solution_Re(x, E_Im, curl2E_Im);
+   get_maxwell_solution_Re(x, E_Re, curl2E_Re);
+
+   // curl ( curl E) - omega^2 E = f
+   double coeff;
+   coeff = -omega * omega;
+   f(0) = curl2E_Im[0] + coeff * E_Im[0];
+   f(1) = curl2E_Im[1] + coeff * E_Im[1];
+   if (dim == 3)
+   {
+      f(2) = curl2E_Im[2] + coeff * E_Im[2];
+   }
+}
+
+// PML
+void pml_function(const Vector &x, std::vector<std::complex<double>> & dxs)
+{
+   double L = length;
+   double n = 2.0;
+   double lbeg, lend;
+   double c = 50.0;
+   double c1 = pml_length; 
+   double c2 = length - pml_length; 
+   double coeff;
+   // initialize to one
+   for (int i = 0; i<dim; ++i) dxs[i] = complex<double>(1.0,0.0); 
+
+   if (pml)
+   {
+   // Stretch in each direction independenly
+      for (int i = 0; i<dim; ++i)
+      {
+         if (x(i) >= c2)
+         {
+            lbeg = c2;
+            lend = L;
+            coeff = n * c / omega / pow(lend-lbeg,n); 
+            dxs[i] = complex<double>(1.0,0.0) + complex<double>(0.0,coeff * pow(x(i)-lbeg, n-1.0)); 
+         }
+         if (x(i) <= c1)
+         {
+            lbeg = c1;
+            lend = 0.0;
+            coeff = n * c / omega / pow(lend-lbeg,n); 
+            dxs[i] = complex<double>(1.0,0.0) - complex<double>(0.0, coeff * pow(x(i)-lbeg, n-1.0)); 
+         } 
+      }
+   }
+}
+
+
+double pml_detJ_inv_Re(const Vector &x)
+{
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0,0.0);
+   pml_function(x, dxs);
+   for (int i=0; i<dim; ++i) det *= dxs[i];
+   
+   complex<double> det_inv = complex<double>(1.0,0.0)/det;
+   return det_inv.real();
+}
+double pml_detJ_inv_Im(const Vector &x)
+{
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0,0.0);
+   pml_function(x, dxs);
+   for (int i=0; i<dim; ++i) det *= dxs[i];
+
+   complex<double> det_inv = complex<double>(1.0,0.0)/det;
+   return det_inv.imag();
+}
+void pml_detJ_JT_J_inv_Re(const Vector &x, DenseMatrix &M)
+{
+   std::vector<complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0,0.0);
+   pml_function(x, dxs);
+
+   for (int i = 0; i<dim; ++i)
+   {
+      diag[i] = complex<double>(1.0,0.0) / pow(dxs[i],2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M=0.0;
+
+   for (int i = 0; i<dim; ++i)
+   {
+      complex<double> temp = det * diag[i];
+      M(i,i) = temp.real();
+   }
+}
+void pml_detJ_JT_J_inv_Im(const Vector &x, DenseMatrix &M)
+{
+   std::vector<std::complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det = 1.0;
+   pml_function(x, dxs);
+
+   for (int i = 0; i<dim; ++i)
+   {
+      diag[i] = complex<double>(1.0,0.0) / pow(dxs[i],2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M=0.0;
+
+   for (int i = 0; i<dim; ++i)
+   {
+      complex<double> temp = det * diag[i];
+      M(i,i) = temp.imag();
+   }
+}
+void pml_detJ_inv_JT_J_Re(const Vector &x, DenseMatrix &M)
+{
+   std::vector<complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0,0.0);
+   pml_function(x, dxs);
+
+   for (int i = 0; i<dim; ++i)
+   {
+      diag[i] = pow(dxs[i],2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M=0.0;
+
+   for (int i = 0; i<dim; ++i)
+   {
+      complex<double> temp = diag[i]/det;
+      M(i,i) = temp.real();
+   }
+}
+void pml_detJ_inv_JT_J_Im(const Vector &x, DenseMatrix &M)
+{
+   std::vector<std::complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det = 1.0;
+   pml_function(x, dxs);
+
+   for (int i = 0; i<dim; ++i)
+   {
+      diag[i] = pow(dxs[i],2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M=0.0;
+
+   for (int i = 0; i<dim; ++i)
+   {
+      complex<double> temp = diag[i]/det;
+      M(i,i) = temp.imag();
+   }
+}

examples/solvers-dev/maxwellp_real.cpp

@@ -0,0 +1,507 @@
+//                                MFEM Example multigrid-grid Cycle
+//
+// Compile with: make mg_maxwellp
+//
+// Sample runs:  mg_maxwellp -m ../data/one-hex.mesh
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include <boost/math/special_functions/airy.hpp>
+// #include "Schwarzp.hpp"
+#include "mg/multigrid.hpp"
+using namespace std;
+using namespace mfem;
+using namespace boost;
+
+// #define DEFINITE
+
+// #ifndef MFEM_USE_PETSC
+// #error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+// #endif
+
+// Define exact solution
+void E_exact(const Vector & x, Vector & E);
+void f_exact(const Vector & x, Vector & f);
+void get_maxwell_solution(const Vector & x, double E[], double curl2E[]);
+void epsilon_func(const Vector &x, DenseMatrix &M);
+
+int dim;
+double omega;
+int sol = 1;
+
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv); // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid); // Determine process identifier
+   // 1. Parse command-line options.
+   // geometry file
+   // const char *mesh_file = "../data/star.mesh";
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   // visualization flag
+   bool visualization = true;
+   // number of wavelengths
+   double k = 0.5;
+   // number of mg levels
+   int ref_levels = 1;
+   // number of initial ref
+   int initref = 1;
+   // solver
+   int solver = 1;
+   // Space Dimension
+   int nd = 3;
+   // PETSC
+   // const char *petscrc_file = "petscrc_direct";
+   const char *petscrc_file = "petscrc_mult_options";
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&ref_levels, "-ref", "--ref_levels",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&sol, "-sol", "--exact",
+                  "Exact solution flag - "
+                  " 1:sinusoidal, 2: point source, 3: plane wave");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.AddOption(&solver, "-s", "--solver",
+                  "Solver: 0 - SCHWARZ, 1 - GMG-GMRES, 2 - PETSC, 3 - SUPERLU, 4 - STRUMPACK, 5-HSS-GMRES");       
+   args.AddOption(&nd, "-nd", "--dimension", "Dimension: 2D or 3D");                                      
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   enum SolverType
+   {
+      INVALID_SOL = -1,
+      SCHWARZ     =  0,
+      GMG_GMRES   =  1,
+      PETSC       =  2,
+      SUPERLU     =  3,
+      STRUMPACK   =  4,
+      HSS_GMRES   =  5,
+   };
+
+
+   // Angular frequency
+   omega = 2.0*k*M_PI;
+   // omega = k;
+
+   // 2. Read the mesh from the given mesh file.
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   Mesh *mesh;
+   if(nd == 3)
+   {
+      double length;
+      length = (sol == 4) ? 0.5: 1.0;
+      mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true, length, length, length, false);
+   }
+   else
+   {
+      mesh = new Mesh(1, 1, Element::QUADRILATERAL, true, 1.0, 1.0,false);
+   }
+   
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   // 4. Define a finite element space on the mesh.
+   FiniteElementCollection *fec   = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   std::vector<ParFiniteElementSpace * > fespaces(ref_levels+1);
+   std::vector<ParMesh * > ParMeshes(ref_levels+1);
+   std::vector<HypreParMatrix*>  P(ref_levels);
+   for (int i = 0; i < ref_levels; i++)
+   {
+      ParMeshes[i] =new ParMesh(*pmesh);
+      fespaces[i] = new ParFiniteElementSpace(*fespace, *ParMeshes[i]);
+      pmesh->UniformRefinement();
+      // Update fespace
+      fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      fespace->GetTrueTransferOperator(*fespaces[i], Tr);
+      Tr.SetOperatorOwner(false);
+      Tr.Get(P[i]);
+   }
+   fespaces[ref_levels] = new ParFiniteElementSpace(*fespace);
+   ConstantCoefficient muinv(1.0);
+#ifdef DEFINITE
+   ConstantCoefficient sigma(pow(omega, 2));
+#else
+   ConstantCoefficient sigma(-pow(omega, 2));
+#endif
+   // 6. Linear form (i.e RHS b = (f,v) = (1,v))
+   ParLinearForm *b = new ParLinearForm(fespace);
+   VectorFunctionCoefficient f(sdim, f_exact);
+   b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
+   b->Assemble();
+
+   MatrixFunctionCoefficient epsilon(dim,epsilon_func);
+   ScalarMatrixProductCoefficient coeff(sigma,epsilon);
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ParBilinearForm *a = new ParBilinearForm(fespace);
+   a->AddDomainIntegrator(new CurlCurlIntegrator(muinv)); // one is the coeff
+   a->AddDomainIntegrator(new VectorFEMassIntegrator(coeff));
+   a->Assemble();
+
+   Array<int> ess_tdof_list;
+   // if (pmesh->bdr_attributes.Size())
+   // {
+      Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   // }
+
+   ParGridFunction x(fespace);
+   x = 0.0;
+   VectorFunctionCoefficient E(sdim, E_exact);
+   // x.ProjectCoefficient(E);
+   x.ProjectBdrCoefficientTangent(E,ess_bdr);
+   ParGridFunction Eex(fespace);
+   Eex.ProjectCoefficient(E);
+
+   HypreParMatrix * A = new HypreParMatrix;
+   Vector B, X;
+
+
+   a->FormLinearSystem(ess_tdof_list, x, *b, *A, X, B);
+
+   if (myid == 0)
+   {
+      cout << "Size of fine grid system: "
+           << A->GetGlobalNumRows() << " x " << A->GetGlobalNumCols() << endl;
+   }
+
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+   chrono.Clear();
+   chrono.Start();
+   // GMGSolver * M1 = new GMGSolver(A, P, GMGSolver::CoarseSolver::PETSC);
+   MGSolver * M1 = new MGSolver(A, P,fespaces);
+   M1->SetTheta(1.0/5.0);
+   chrono.Stop();
+   
+   // if (myid == 0)
+   // {
+   cout << "MG preconditioner construction time: " << chrono.RealTime() << endl;
+   // }
+
+   int maxit(2000);
+   double rtol(1.e-6);
+   double atol(1.e-6);
+   X = 0.0;
+   GMRESSolver gmres(MPI_COMM_WORLD);
+   gmres.SetAbsTol(atol);
+   gmres.SetRelTol(rtol);
+   gmres.SetMaxIter(maxit);
+   gmres.SetPreconditioner(*M1);
+   gmres.SetOperator(*A);
+   gmres.SetPrintLevel(1);
+   chrono.Clear();
+   chrono.Start();
+   gmres.Mult(B,X);
+   chrono.Stop();
+
+   // if (myid == 0)
+   // {
+   cout << "MG-Preconditioned Solver time: " << chrono.RealTime() << endl;
+   // }
+   delete M1;
+
+   MFEMFinalizePetsc();
+
+   a->RecoverFEMSolution(X, *b, x);
+
+   int order_quad = max(2, 2 * order + 1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i = 0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double L2Error = x.ComputeL2Error(E, irs);
+   double norm_E = ComputeGlobalLpNorm(2, E, *pmesh, irs);
+
+   if (myid == 0)
+   {
+      cout << "\n || E_h - E || / ||E|| = " << L2Error / norm_E << '\n' << endl;
+   }
+
+   // int precision = 8;
+   // VisItDataCollection *dc = NULL;
+   // dc = new VisItDataCollection("Maxwellp_real", pmesh);
+   // dc->SetPrefixPath("output");
+   // dc->SetPrecision(precision);
+   // dc->RegisterField("solution",&x);
+   // dc->Save();
+
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << x  << "window_title 'Numerical E'" << "keys rRljc\n" << flush;
+      
+      socketstream exact_sock(vishost, visport);
+      exact_sock << "parallel " << num_procs << " " << myid << "\n";
+      exact_sock.precision(8);
+      exact_sock << "solution\n" << *pmesh << Eex << "window_title 'Exact E'" << "keys rRljc\n" << flush;
+   }
+   // ---------------------------------------------------------------------
+
+   delete A;
+   delete a;
+   delete b;
+   for (auto p: ParMeshes) delete p;
+   for (auto p: fespaces)  delete p;
+   for (auto p: P)  delete p;
+   ParMeshes.clear();
+   fespaces.clear();
+   P.clear();
+   delete fec;
+   delete fespace;
+   delete pmesh;
+   MPI_Finalize();
+   return 0;
+}
+
+
+//define exact solution
+void E_exact(const Vector &x, Vector &E)
+{
+   double curl2E[3];
+   get_maxwell_solution(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+// f = curl (mu curl E ) + coeff*E
+void f_exact(const Vector &x, Vector &f)
+{
+   double coeff;
+#ifdef DEFINITE
+   coeff = omega * omega;
+#else
+   coeff = -omega * omega;
+#endif
+   f = 0.0;
+   if (sol != 4)
+   {
+      double E[3], curl2E[3];
+      get_maxwell_solution(x, E, curl2E);
+      // curl ( curl E) +/- omega^2 E = f
+      f(0) = curl2E[0] + coeff * E[0];
+      f(1) = curl2E[1] + coeff * E[1];
+      if (dim == 2)
+      {
+         if (x.Size() == 3) {f(2)=0.0;}
+      }
+      else
+      {
+         f(2) = curl2E[2] + coeff * E[2];
+      }
+   }
+}
+
+void get_maxwell_solution(const Vector & x, double E[], double curl2E[])
+{
+   
+   if (sol ==-1)
+   {
+      E[0] = x[1] * x[2] * (1.0 - x[1]) * (1.0 - x[2]);
+      E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+      E[2] = x[0] * x[1] * (1.0 - x[0]) * (1.0 - x[1]);
+      
+      curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+      curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+      curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+   }
+   else if (sol == 0) // polynomial
+   {
+      if (dim == 2)
+      {
+         E[0] = x[0] * (1.0 - x[0]) * x[1] * (1.0 - x[1]);
+         E[1] = 0.0;
+         //
+         curl2E[0] =  - 2.0 * x[0] * (x[0] - 1.0);
+         curl2E[1] = (2.0*x[0]-1.0)*(2.0*x[1]-1);
+         curl2E[2] = 0.0;
+      }
+      else
+      {
+         // Polynomial vanishing on the boundary
+         E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+         E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+         E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+         //
+         curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+         curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+         curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+      }
+   }
+
+   else if (sol == 1) // sinusoidal
+   {
+      if (dim == 2)
+      {
+         E[0] = sin(omega * x[1]);
+         E[1] = sin(omega * x[0]);
+         curl2E[0] = omega * omega * E[0];
+         curl2E[1] = omega * omega * E[1];
+         curl2E[2] = 0.0;
+      }
+      else
+      {
+         E[0] = sin(omega * x[1]);
+         E[1] = sin(omega * x[2]);
+         E[2] = sin(omega * x[0]);
+
+         curl2E[0] = omega * omega * E[0];
+         curl2E[1] = omega * omega * E[1];
+         curl2E[2] = omega * omega * E[2];
+      }
+   }
+   else if (sol == 2) //point source
+   {
+      if (dim == 2)
+      {
+         // shift to avoid singularity
+         double x0 = x(0) + 0.1;
+         double x1 = x(1) + 0.1;
+         //
+         double r = sqrt(x0 * x0 + x1 * x1);
+
+         E[0] = cos(omega * r);
+         E[1] = 0.0;
+
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_xy = -(r_x / r) * r_y;
+         double r_yx = r_xy;
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+
+         curl2E[0] = omega * ((r_yy ) * sin(omega * r) + (omega * r_y * r_y) * cos(omega * r));
+         curl2E[1] = -omega * (r_yx * sin(omega * r) + omega * r_y * r_x * cos(omega * r));
+         curl2E[2] = 0.0;
+      }
+      else
+      {
+      // shift to avoid singularity
+         double x0 = x(0) + 0.1;
+         double x1 = x(1) + 0.1;
+         double x2 = x(2) + 0.1;
+         //
+         double r = sqrt(x0 * x0 + x1 * x1 + x2 * x2);
+
+         E[0] = cos(omega * r);
+         E[1] = 0.0;
+         E[2] = 0.0;
+
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_z = x2 / r;
+         double r_xy = -(r_x / r) * r_y;
+         double r_xz = -(r_x / r) * r_z;
+         double r_yx = r_xy;
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+         double r_zx = r_xz;
+         double r_zz = (1.0 / r) * (1.0 - r_z * r_z);
+
+         curl2E[0] = omega * ((r_yy + r_zz) * sin(omega * r) +
+                              (omega * r_y * r_y + omega * r_z * r_z) * cos(omega * r));
+         curl2E[1] = -omega * (r_yx * sin(omega * r) + omega * r_y * r_x * cos(omega * r));
+         curl2E[2] = -omega * (r_zx * sin(omega * r) + omega * r_z * r_x * cos(omega * r));
+      }
+   }
+   else if (sol == 3) // plane wave
+   {
+      if (dim == 2)
+      {
+         E[0] = cos(omega * (x(0) + x(1)) / sqrt(2.0));
+         E[1] = 0.0;
+
+         curl2E[0] = omega * omega * E[0] / 2.0;
+         curl2E[1] = -omega * omega * E[0] / 2.0;
+      }
+      else
+      {
+         E[0] = cos(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+         E[1] = 0.0;
+         E[2] = 0.0;
+
+         curl2E[0] = 2.0 * omega * omega * E[0] / 3.0;
+         curl2E[1] = -omega * omega * E[0] / 3.0;
+         curl2E[2] = -omega * omega * E[0] / 3.0;
+      }
+   }
+   else if (sol == 4) // Airy function
+   {
+      E[0] = 0;
+      E[1] = 0;
+      // double b = -pow(omega/4.0,2.0/3.0)*(4.0*x(0)-1.0);
+      double b = -pow(omega/4.0,2.0/3.0)*(4.0*x(0)-1.0);
+      E[2] = boost::math::airy_ai(b);
+   }
+}
+
+void epsilon_func(const Vector &x, DenseMatrix &M)
+{
+   M.SetSize(dim);
+
+   M = 0.0;
+   M(0,0) = 1.0;
+   M(1,1) = 1.0;
+   if (dim == 3)
+   {
+      if (sol != 4)
+      {
+         M(2,2) = 1.0;
+      }
+      else
+      {
+         M(2,2) = 4.0*x(0)-1.0;
+      }
+   }
+   
+}

examples/solvers-dev/meshes/hexa26.mesh

@@ -0,0 +1,176 @@
+MFEM mesh v1.0
+
+#
+# MFEM Geometry Types (see mesh/geom.hpp):
+#
+# POINT       = 0
+# SEGMENT     = 1
+# TRIANGLE    = 2
+# SQUARE      = 3
+# TETRAHEDRON = 4
+# CUBE        = 5
+# PRISM       = 6
+#
+
+dimension
+3
+
+elements
+26
+30 5 0 1 2 3 4 5 6 7
+31 5 7 3 8 9 6 2 10 11
+6 5 12 13 14 15 16 17 18 19
+7 5 20 12 15 21 22 16 19 23
+8 5 24 20 21 25 26 22 23 27
+9 5 28 29 30 31 19 15 14 18
+10 5 32 33 34 35 28 29 36 37
+11 5 25 21 33 38 27 23 32 39
+12 5 33 34 40 38 32 35 41 39
+13 5 42 36 29 30 43 37 28 31
+14 5 3 8 44 45 7 9 46 47
+15 5 28 32 33 29 19 23 21 15
+16 5 39 27 23 32 48 49 46 47
+17 5 18 19 16 17 11 9 50 51
+18 5 5 43 31 6 4 37 28 7
+19 5 31 28 19 18 6 7 9 11
+20 5 50 16 22 52 9 19 23 46
+21 5 23 27 26 22 46 49 53 52
+22 5 4 37 28 7 54 35 32 47
+23 5 55 41 35 54 48 39 32 47
+24 5 49 56 57 53 46 44 58 52
+25 5 49 56 44 46 48 59 45 47
+26 5 59 60 61 45 48 55 54 47
+27 5 61 0 3 45 54 4 7 47
+28 5 8 62 58 44 9 50 52 46
+29 5 9 8 62 50 11 10 63 51
+
+boundary
+60
+1 3 25 38 33 21
+1 3 39 38 25 27
+1 3 53 26 22 52
+1 3 27 26 53 49
+1 3 12 13 17 16
+1 3 12 15 14 13
+1 3 13 14 18 17
+1 3 42 30 29 36
+1 3 42 36 37 43
+1 3 31 30 42 43
+1 3 3 45 44 8
+1 3 29 30 14 15
+1 3 14 30 31 18
+1 3 63 62 50 51
+1 3 63 51 11 10
+1 3 8 62 63 10
+1 3 62 58 52 50
+1 3 8 44 58 62
+1 3 21 33 29 15
+1 3 20 12 16 22
+1 3 20 21 15 12
+1 3 39 27 49 48
+1 3 24 20 22 26
+1 3 24 25 21 20
+1 3 27 25 24 26
+1 3 33 34 36 29
+1 3 36 34 35 37
+1 3 55 41 39 48
+1 3 55 54 35 41
+1 3 10 11 6 2
+1 3 3 8 10 2
+1 3 4 37 35 54
+1 3 50 52 22 16
+1 3 33 38 40 34
+1 3 41 40 38 39
+1 3 34 40 41 35
+1 3 50 16 17 51
+1 3 51 17 18 11
+1 3 5 43 37 4
+1 3 5 6 31 43
+1 3 11 18 31 6
+1 3 61 0 4 54
+1 3 61 45 3 0
+1 3 59 60 55 48
+1 3 60 61 54 55
+1 3 59 45 61 60
+1 3 0 1 5 4
+1 3 1 2 6 5
+1 3 0 3 2 1
+1 3 58 57 53 52
+1 3 49 53 57 56
+1 3 56 57 58 44
+1 3 49 56 59 48
+1 3 56 44 45 59
+2 3 28 32 23 19
+2 3 46 47 7 9
+2 3 23 46 9 19
+2 3 32 28 7 47
+2 3 46 23 32 47
+2 3 28 19 9 7
+
+vertices
+64
+3
+-1.5 1.5 0.5
+-1.5 1.5 1.5
+-1.5 0.5 1.5
+-1.5 0.5 0.5
+-0.5 1.5 0.5
+-0.5 1.5 1.5
+-0.5 0.5 1.5
+-0.5 0.5 0.5
+-1.5 -0.5 0.5
+-0.5 -0.5 0.5
+-1.5 -0.5 1.5
+-0.5 -0.5 1.5
+1.5 -1.5 0.5
+1.5 -1.5 1.5
+1.5 -0.5 1.5
+1.5 -0.5 0.5
+0.5 -1.5 0.5
+0.5 -1.5 1.5
+0.5 -0.5 1.5
+0.5 -0.5 0.5
+1.5 -1.5 -0.5
+1.5 -0.5 -0.5
+0.5 -1.5 -0.5
+0.5 -0.5 -0.5
+1.5 -1.5 -1.5
+1.5 -0.5 -1.5
+0.5 -1.5 -1.5
+0.5 -0.5 -1.5
+0.5 0.5 0.5
+1.5 0.5 0.5
+1.5 0.5 1.5
+0.5 0.5 1.5
+0.5 0.5 -0.5
+1.5 0.5 -0.5
+1.5 1.5 -0.5
+0.5 1.5 -0.5
+1.5 1.5 0.5
+0.5 1.5 0.5
+1.5 0.5 -1.5
+0.5 0.5 -1.5
+1.5 1.5 -1.5
+0.5 1.5 -1.5
+1.5 1.5 1.5
+0.5 1.5 1.5
+-1.5 -0.5 -0.5
+-1.5 0.5 -0.5
+-0.5 -0.5 -0.5
+-0.5 0.5 -0.5
+-0.5 0.5 -1.5
+-0.5 -0.5 -1.5
+-0.5 -1.5 0.5
+-0.5 -1.5 1.5
+-0.5 -1.5 -0.5
+-0.5 -1.5 -1.5
+-0.5 1.5 -0.5
+-0.5 1.5 -1.5
+-1.5 -0.5 -1.5
+-1.5 -1.5 -1.5
+-1.5 -1.5 -0.5
+-1.5 0.5 -1.5
+-1.5 1.5 -1.5
+-1.5 1.5 -0.5
+-1.5 -1.5 0.5
+-1.5 -1.5 1.5

examples/solvers-dev/meshes/l-shape.mesh

@@ -0,0 +1,61 @@
+MFEM mesh v1.0
+
+#
+# MFEM Geometry Types (see mesh/geom.hpp):
+#
+# POINT       = 0
+# SEGMENT     = 1
+# TRIANGLE    = 2
+# SQUARE      = 3
+# TETRAHEDRON = 4
+# CUBE        = 5
+# PRISM       = 6
+#
+
+dimension
+2
+
+elements
+8
+1 3 0 1 5 4
+1 3 1 2 6 5
+1 3 3 4 8 7
+1 3 4 5 9 8
+1 3 5 6 10 9
+1 3 7 8 12 11
+1 3 8 9 13 12
+1 3 9 10 14 13
+
+boundary
+12
+1 1 0 1
+1 1 4 0
+1 1 1 2
+1 1 2 6
+1 1 3 4
+1 1 7 3
+1 1 6 10
+1 1 12 11
+1 1 11 7
+1 1 13 12
+1 1 10 14
+1 1 14 13
+
+vertices
+15
+2
+1 0
+2 0
+3 0
+0 1
+1 1
+2 1
+3 1
+0 2
+1 2
+2 2
+3 2
+0 3
+1 3
+2 3
+3 3

examples/solvers-dev/mg/multigrid.cpp

@@ -0,0 +1,266 @@
+#include "mfem.hpp"
+#include "multigrid.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+
+MGSolver::MGSolver(HypreParMatrix * Af_, std::vector<HypreParMatrix *> P_,std::vector<ParFiniteElementSpace * > fespaces)
+   : Solver(Af_->Height(), Af_->Width()), Af(Af_), P(P_) {
+
+   StopWatch chrono;
+   NumGrids = P.size();
+   S.resize(NumGrids);
+   A.resize(NumGrids + 1);
+
+   // chrono.Clear();
+   // chrono.Start();
+   A[NumGrids] = Af;
+   for (int i = NumGrids ; i > 0; i--)
+   {
+      A[i - 1] = RAP(A[i], P[i - 1]);
+   }
+
+   // chrono.Stop();
+   // cout << "Setting up matrices A: " << chrono.RealTime() << endl;
+
+   // Set up coarse solve operator
+   // chrono.Clear();
+   // chrono.Start();
+   
+   petsc = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   // Convert to PetscParMatrix
+   petsc->SetOperator(PetscParMatrix(A[0], Operator::PETSC_MATAIJ));
+   invAc = petsc;
+   // chrono.Stop();
+   // cout << "Setting up coarse grid LU: " << chrono.RealTime() << endl;
+
+
+   // GMGSolver * M1 = new GMGSolver(A, P, GMGSolver::CoarseSolver::PETSC);
+   
+   // chrono.Clear();
+   // chrono.Start();
+   for (int i = NumGrids - 1; i >= 0 ; i--)
+   {
+      S[i] = new ParSchwarzSmoother(fespaces[i]->GetParMesh(),1,fespaces[i+1],A[i+1]);
+      // S[i]->SetDumpingParam(1.0/5.0);
+      // S[i]->SetType(HypreSmoother::Jacobi);
+      // S[i]->SetOperator(*A[i+1]);
+   }
+   // chrono.Stop();
+   // cout << "Setting up Smoothers: " << chrono.RealTime() << endl;
+
+}
+
+void MGSolver::Mult(const Vector &r, Vector &z) const
+{
+   // Residual vectors
+   std::vector<Vector> rv(NumGrids + 1);
+   // correction vectors
+   std::vector<Vector> zv(NumGrids + 1);
+   // allocation
+   for (int i = 0; i <= NumGrids ; i++)
+   {
+      int n = A[i]->Width();
+      rv[i].SetSize(n);
+      zv[i].SetSize(n);
+   }
+   // Initial residual
+   rv[NumGrids] = r;
+   // smooth and update residuals down to the coarsest level
+   for (int i = NumGrids; i > 0 ; i--)
+   {
+      // Pre smooth
+      S[i - 1]->Mult(rv[i], zv[i]); zv[i] *= theta;
+      // compute residual
+      Vector w(A[i]->Height());
+      A[i]->Mult(zv[i], w);
+      rv[i] -= w;
+      // Restrict
+      P[i - 1]->MultTranspose(rv[i], rv[i - 1]);
+   }
+
+   // Coarse grid Solve
+   invAc->Mult(rv[0], zv[0]);
+   //
+   for (int i = 1; i <= NumGrids ; i++)
+   {
+      // Prolong correction
+      Vector u(P[i - 1]->Height());
+      P[i - 1]->Mult(zv[i - 1], u);
+      // Update correction
+      zv[i] += u;
+      // Update residual
+      Vector v(A[i]->Height());
+      A[i]->Mult(u, v); rv[i] -= v;
+      // Post smooth
+      S[i - 1]->Mult(rv[i], v); v *= theta;
+      // Update correction
+      zv[i] += v;
+   }
+   z = zv[NumGrids];
+}
+
+MGSolver::~MGSolver() 
+{
+   int n = S.size();
+   for (int i = n - 1; i >= 0 ; i--)
+   {
+      delete S[i];
+      delete A[i];
+   }
+   S.clear();
+   A.clear();
+   delete invAc;
+}
+
+BlockMGSolver::BlockMGSolver(Array2D<HypreParMatrix *> Af_, std::vector<HypreParMatrix *> P_,std::vector<ParFiniteElementSpace * > fespaces)
+   : Solver(Af_(0,0)->Height()+Af_(1,0)->Height(), Af_(0,0)->Width()+Af_(1,0)->Width()), Af(Af_), P(P_) {
+
+   NumGrids = P.size();
+   BlkP.resize(NumGrids);
+   BlkA.resize(NumGrids+1);
+   S.resize(NumGrids);
+   A.resize(NumGrids + 1);
+   A[NumGrids] = Af;
+   Aoffsets.resize(NumGrids+1);
+   Poffsets_i.resize(NumGrids);
+   Poffsets_j.resize(NumGrids);
+   // Construct Bilinear form Matrices on each level
+   for (int k = NumGrids ; k > 0; k--)
+   {
+      A[k - 1].SetSize(2,2);
+      Aoffsets[k].SetSize(3); Aoffsets[k][0] = 0;
+      Aoffsets[k][1] = A[k](0,0)->Height();
+      Aoffsets[k][2] = A[k](1,1)->Height();
+      Aoffsets[k].PartialSum();
+      BlkA[k] = new BlockOperator(Aoffsets[k]);
+      for (int i=0; i<2; i++)
+      {
+         for (int j=0; j<2; j++)
+         {
+            A[k - 1](i,j) = RAP(A[k](i,j), P[k - 1]);
+            BlkA[k]->SetBlock(i,j,A[k](i,j));
+         }
+      }
+
+      Poffsets_i[k-1].SetSize(3); Poffsets_i[k-1][0] = 0;
+      Poffsets_j[k-1].SetSize(3); Poffsets_j[k-1][0] = 0;
+      Poffsets_i[k-1][1] = P[k-1]->Height(); Poffsets_j[k-1][1] = P[k-1]->Width();
+      Poffsets_i[k-1][2] = P[k-1]->Height(); Poffsets_j[k-1][2] = P[k-1]->Width();
+      Poffsets_i[k-1].PartialSum();
+      Poffsets_j[k-1].PartialSum();
+
+      BlkP[k-1] = new BlockOperator(Poffsets_i[k-1],Poffsets_j[k-1]);
+      BlkP[k-1]->SetBlock(0,0,P[k-1]);
+      BlkP[k-1]->SetBlock(1,1,P[k-1]);
+   }
+   // Set up coarse solve operator
+   // Convert the coarse grid blockmatrix to a HypreParMatrix
+   Array<int> offsets(3);
+   offsets[0]=0;
+   offsets[1]=A[0](0,0)->Height();
+   offsets[2]=A[0](1,1)->Height();
+   offsets.PartialSum();
+
+   BlkA[0] = new BlockOperator(offsets);
+   for (int i=0; i<2; i++)
+   {
+      for (int j=0; j<2; j++)
+      {
+         BlkA[0]->SetBlock(i,j,A[0](i,j));
+      }
+   }
+
+   // Convert to PetscParMatrix
+   HypreParMatrix * Ac;
+   // Ac = CreateHypreParMatrixFromBlocks(MPI_COMM_WORLD, offsets, A[0]);
+   Ac = CreateHypreParMatrixFromBlocks(offsets, BlkA[0]);
+   // Convert to PetscParMatrix
+   PetscParMatrix * petsc = new PetscParMatrix(Ac, Operator::PETSC_MATAIJ);
+   delete Ac;
+   invAc = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   invAc->SetOperator(*petsc);
+   delete petsc;
+
+   // Smoother
+   for (int i = NumGrids - 1; i >= 0 ; i--)
+   {
+      S[i] = new BlkParSchwarzSmoother(fespaces[i]->GetParMesh(),1,fespaces[i+1],BlkA[i+1]);
+      // S[i]->SetDumpingParam(1.0/5.0);
+   }
+}
+
+ void BlockMGSolver::Mult(const Vector &r, Vector &z) const
+{
+   // Residual vectors
+   std::vector<Vector> rv(NumGrids + 1);
+   // correction vectors
+   std::vector<Vector> zv(NumGrids + 1);
+   // allocation
+   for (int i = 0; i <= NumGrids ; i++)
+   {
+      int n = (i==0) ? invAc->Height(): BlkA[i]->Width();
+      rv[i].SetSize(n);
+      zv[i].SetSize(n);
+   }
+   // Initial residual
+   rv[NumGrids] = r;
+
+   // smooth and update residuals down to the coarsest level
+   for (int i = NumGrids; i > 0 ; i--)
+   {
+      // Pre smooth
+      S[i - 1]->Mult(rv[i], zv[i]); zv[i] *= theta;
+      // compute residual
+      int n = BlkA[i]->Width();
+      Vector w(n);
+      BlkA[i]->Mult(zv[i], w);
+      rv[i] -= w;
+      // Restrict
+      BlkP[i - 1]->MultTranspose(rv[i], rv[i - 1]);
+   }
+
+   // Coarse grid Solve
+   invAc->Mult(rv[0], zv[0]);
+   //
+   for (int i = 1; i <= NumGrids ; i++)
+   {
+      // Prolong correction
+      Vector u(BlkP[i - 1]->Height());
+      BlkP[i - 1]->Mult(zv[i - 1], u);
+      // Update correction
+      zv[i] += u;
+      // Update residual
+      Vector v(BlkA[i]->Height());
+      BlkA[i]->Mult(u, v); rv[i] -= v;
+      // Post smooth
+      S[i - 1]->Mult(rv[i], v); v *= theta;
+      // Update correction
+      zv[i] += v;
+   }
+   z = zv[NumGrids];
+}
+
+BlockMGSolver::~BlockMGSolver() 
+{
+   for (int i = NumGrids - 1; i >= 0 ; i--)
+   {
+      delete S[i];
+      delete BlkP[i];
+      delete BlkA[i];
+      for (int j=0; j<2; j++)
+      {
+         for (int k=0; k<2; k++)
+         {
+            delete A[i](j,k);
+         }
+      }
+      A[i].DeleteAll();
+   }
+   delete BlkA[NumGrids];
+   delete invAc;
+   A.clear();
+}

examples/solvers-dev/mg/multigrid.hpp

@@ -0,0 +1,63 @@
+#pragma once
+
+#include "mfem.hpp"
+#include "util.hpp"
+#include "../as/blkschwarzp.hpp"
+#include "petsc.h"
+
+namespace mfem {
+
+class MGSolver : public Solver
+{
+private:
+   /// The linear system matrix
+   HypreParMatrix *Af;
+   std::vector<HypreParMatrix *> A;
+   std::vector<HypreParMatrix *> P;
+   std::vector<ParSchwarzSmoother *> S;
+   int NumGrids;
+   PetscLinearSolver *petsc = nullptr;
+   Solver *invAc = nullptr;
+   double theta = 1.0;
+
+public:
+   MGSolver(HypreParMatrix *Af_, std::vector<HypreParMatrix *> P_, std::vector<ParFiniteElementSpace *> fespaces);
+
+   virtual void SetOperator(const Operator &op) {}
+
+   virtual void SetTheta(const double a) { theta = a; }
+
+   virtual void Mult(const Vector &r, Vector &z) const;
+   virtual ~MGSolver();
+};
+
+class BlockMGSolver : public Solver
+{
+private:
+   /// The linear system matrix
+   Array2D<HypreParMatrix *> Af;
+   vector<Array<int>>Aoffsets;
+   vector<Array<int>>Poffsets_i;
+   vector<Array<int>>Poffsets_j;
+   std::vector<Array2D<HypreParMatrix *>> A;
+   std::vector<HypreParMatrix *> P;
+   std::vector<BlockOperator *> BlkP;
+   std::vector<BlockOperator *> BlkA;
+   std::vector<BlkParSchwarzSmoother *> S;
+   HypreParMatrix * Ac;
+   int NumGrids;
+   PetscLinearSolver *invAc = nullptr;
+   double theta = 1.0/5.0;
+
+public:
+   BlockMGSolver(Array2D<HypreParMatrix *> Af_, std::vector<HypreParMatrix *> P_, std::vector<ParFiniteElementSpace *> fespaces);
+
+   virtual void SetOperator(const Operator &op) {}
+
+   virtual void SetTheta(const double a) { theta = a; }
+
+   virtual void Mult(const Vector &r, Vector &z) const;
+   virtual ~BlockMGSolver();
+};
+
+} // namespace mfem

examples/solvers-dev/mg/util.cpp

@@ -0,0 +1,294 @@
+
+#include "util.hpp"
+
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// This is very costly
+SparseMatrix *GetSparseMatrixFromOperator(Operator *op)
+{
+   const int n = op->Height();
+   MFEM_VERIFY(n == op->Width(), "");
+
+   SparseMatrix *S = new SparseMatrix(n);
+   Vector x(n);
+   Vector y(n);
+
+   for (int j = 0; j < n; ++j)
+   {
+      x = 0.0;
+      x[j] = 1.0;
+      op->Mult(x, y);
+      for (int i = 0; i < n; ++i)
+      {
+         //if (y[i] != 0.0)
+         if (fabs(y[i]) > 1.0e-15)
+         {
+            S->Set(i, j, y[i]);
+         }
+      }
+   }
+   S->Finalize();
+   return S;
+}
+
+SparseMatrix *GetSparseMatrixFromBlockMatrix(BlockMatrix *blk_mat)
+{
+   int n = blk_mat->Height();
+   int m = blk_mat->Width();
+   SparseMatrix *S = new SparseMatrix(n, m);
+
+   int numRowblocks = blk_mat->NumRowBlocks();
+   int numColblocks = blk_mat->NumColBlocks();
+
+   Array<int> offsets_i(numRowblocks);
+   offsets_i[0] = 0;
+   Array<int> offsets_j(numColblocks);
+   offsets_j[0] = 0;
+
+   for (int i = 0; i < numRowblocks - 1; i++)
+   {
+      offsets_i[i + 1] = blk_mat->GetBlock(i, 0).Height();
+   }
+   for (int j = 0; j < numColblocks - 1; j++)
+   {
+      offsets_j[j + 1] = blk_mat->GetBlock(0, j).Width();
+   }
+   offsets_i.PartialSum();
+   offsets_j.PartialSum();
+
+   for (int i = 0; i < numRowblocks; i++)
+   {
+      for (int j = 0; j < numColblocks; j++)
+      {
+         SparseMatrix *block = &blk_mat->GetBlock(i, j);
+         int nrows = block->NumRows();
+         for (int k = 0; k < nrows; k++)
+         {
+            int *col = block->GetRowColumns(k);
+            int ncols = block->RowSize(k);
+            double *data = block->GetRowEntries(k);
+            for (int l = 0; l < ncols; l++)
+            {
+               S->Set(offsets_i[i] + k, offsets_j[j] + col[l], data[l]);
+            }
+         }
+      }
+   }
+   S->Finalize();
+   return S;
+}
+
+//---------------------------------------------------------------------
+// From Dylan
+//---------------------------------------------------------------------
+hypre_CSRMatrix *GetHypreParMatrixData(const HypreParMatrix &hypParMat)
+{
+   // First cast the parameter to a hypre_ParCSRMatrix
+   hypre_ParCSRMatrix *parcsr_op =
+       (hypre_ParCSRMatrix *)const_cast<HypreParMatrix &>(hypParMat);
+
+   MFEM_ASSERT(parcsr_op != NULL, "STRUMPACK: const_cast failed in SetOperator");
+
+   // Create the CSRMatrixMPI A_ by borrowing the internal data from a hypre_CSRMatrix.
+   return hypre_MergeDiagAndOffd(parcsr_op);
+}
+
+HypreParMatrix *CreateHypreParMatrixFromBlocks(Array<int> &offsets, BlockOperator *blockA)
+{
+   const int numBlocks = offsets.Size() - 1;
+   Array2D<HypreParMatrix *> blocks(numBlocks, numBlocks);
+   for (int i = 0; i < 2; ++i)
+   {
+      for (int j = 0; j < 2; ++j)
+      {
+         blocks(i, j) = static_cast<HypreParMatrix *>(&blockA->GetBlock(i, j));
+      }
+   }
+
+   MPI_Comm comm = blocks(0,0)->GetComm();
+   const int num_loc_rows = offsets[numBlocks];
+
+   int nprocs, rank;
+   MPI_Comm_rank(comm, &rank);
+   MPI_Comm_size(comm, &nprocs);
+
+   std::vector<int> all_num_loc_rows(nprocs);
+   std::vector<int> procOffsets(nprocs);
+   std::vector<std::vector<int>> all_block_num_loc_rows(numBlocks);
+   std::vector<std::vector<int>> blockProcOffsets(numBlocks);
+   std::vector<std::vector<int>> procBlockOffsets(nprocs);
+
+   MPI_Allgather(&num_loc_rows, 1, MPI_INT, all_num_loc_rows.data(), 1, MPI_INT, comm);
+
+   for (int j = 0; j < numBlocks; ++j)
+   {
+      all_block_num_loc_rows[j].resize(nprocs);
+      blockProcOffsets[j].resize(nprocs);
+
+      const int blockNumRows = offsets[j + 1] - offsets[j];
+      MPI_Allgather(&blockNumRows, 1, MPI_INT, all_block_num_loc_rows[j].data(), 1, MPI_INT, comm);
+      blockProcOffsets[j][0] = 0;
+      for (int i = 0; i < nprocs - 1; ++i)
+      {
+         blockProcOffsets[j][i + 1] = blockProcOffsets[j][i] + all_block_num_loc_rows[j][i];
+      }
+   }
+
+   int first_loc_row = 0;
+   int glob_nrows = 0;
+   procOffsets[0] = 0;
+   for (int i = 0; i < nprocs; ++i)
+   {
+      glob_nrows += all_num_loc_rows[i];
+      if (i < rank)
+         first_loc_row += all_num_loc_rows[i];
+
+      if (i < nprocs - 1)
+         procOffsets[i + 1] = procOffsets[i] + all_num_loc_rows[i];
+
+      procBlockOffsets[i].resize(numBlocks);
+      procBlockOffsets[i][0] = 0;
+      for (int j = 1; j < numBlocks; ++j)
+         procBlockOffsets[i][j] = procBlockOffsets[i][j - 1] + all_block_num_loc_rows[j - 1][i];
+   }
+
+   const int glob_ncols = glob_nrows;
+
+   std::vector<int> opI(num_loc_rows + 1);
+   std::vector<int> cnt(num_loc_rows);
+
+   for (int i = 0; i < num_loc_rows; ++i)
+   {
+      opI[i] = 0;
+      cnt[i] = 0;
+   }
+
+   opI[num_loc_rows] = 0;
+
+   Array2D<hypre_CSRMatrix *> csr_blocks(numBlocks, numBlocks);
+
+   // Loop over all blocks, to determine nnz for each row.
+   for (int i = 0; i < numBlocks; ++i)
+   {
+      for (int j = 0; j < numBlocks; ++j)
+      {
+         if (blocks(i, j) == NULL)
+         {
+            csr_blocks(i, j) = NULL;
+         }
+         else
+         {
+            csr_blocks(i, j) = GetHypreParMatrixData(*(blocks(i, j)));
+
+            const int nrows = csr_blocks(i, j)->num_rows;
+
+            for (int k = 0; k < nrows; ++k)
+            {
+               const int rowg = offsets[i] + k;
+               //(*(leftInjection(i, j)))[k]
+               opI[rowg + 1] += csr_blocks(i, j)->i[k + 1] - csr_blocks(i, j)->i[k];
+            }
+         }
+      }
+   }
+
+   // Now opI[i] is nnz for row i-1. Do a partial sum to get offsets.
+   for (int i = 0; i < num_loc_rows; ++i)
+      opI[i + 1] += opI[i];
+
+   const int nnz = opI[num_loc_rows];
+
+   std::vector<HYPRE_Int> opJ(nnz);
+   std::vector<double> data(nnz);
+
+   // Loop over all blocks, to set matrix data.
+   for (int i = 0; i < numBlocks; ++i)
+   {
+      for (int j = 0; j < numBlocks; ++j)
+      {
+         if (csr_blocks(i, j) != NULL)
+         {
+            const int nrows = csr_blocks(i, j)->num_rows;
+
+            for (int k = 0; k < nrows; ++k)
+            {
+               const int rowg = offsets[i] + k; // process-local row
+               const int nnz_k = csr_blocks(i, j)->i[k + 1] - csr_blocks(i, j)->i[k];
+               const int osk = csr_blocks(i, j)->i[k];
+
+               for (int l = 0; l < nnz_k; ++l)
+               {
+                  // Find the column process offset for the block.
+                  const int bcol = csr_blocks(i, j)->j[osk + l];
+                  int bcolproc = 0;
+
+                  for (int p = 1; p < nprocs; ++p)
+                  {
+                     if (blockProcOffsets[j][p] > bcol)
+                     {
+                        bcolproc = p - 1;
+                        break;
+                     }
+                  }
+                  if (blockProcOffsets[j][nprocs - 1] <= bcol)
+                     bcolproc = nprocs - 1;
+
+                  const int colg = procOffsets[bcolproc] + procBlockOffsets[bcolproc][j] + (bcol - blockProcOffsets[j][bcolproc]);
+
+                  if (colg < 0)
+                     cout << "BUG, negative global column index" << endl;
+
+                  opJ[opI[rowg] + cnt[rowg]] = colg;
+                  data[opI[rowg] + cnt[rowg]] = csr_blocks(i, j)->data[osk + l];
+                  cnt[rowg]++;
+               }
+            }
+         }
+      }
+   }
+
+   bool cntCheck = true;
+   for (int i = 0; i < num_loc_rows; ++i)
+   {
+      if (cnt[i] != opI[i + 1] - opI[i])
+         cntCheck = false;
+   }
+
+   MFEM_VERIFY(cntCheck, "");
+
+   for (int i = 0; i < numBlocks; ++i)
+   {
+      for (int j = 0; j < numBlocks; ++j)
+      {
+         if (csr_blocks(i, j) != NULL)
+         {
+            hypre_CSRMatrixDestroy(csr_blocks(i, j));
+         }
+      }
+   }
+
+   std::vector<HYPRE_Int> rowStarts2(2);
+   rowStarts2[0] = first_loc_row;
+   rowStarts2[1] = first_loc_row + all_num_loc_rows[rank];
+
+   HYPRE_Int minJ = opJ[0];
+   HYPRE_Int maxJ = opJ[0];
+   for (int i = 0; i < nnz; ++i)
+   {
+      minJ = std::min(minJ, opJ[i]);
+      maxJ = std::max(maxJ, opJ[i]);
+
+      if (opJ[i] >= glob_ncols)
+         cout << "Column indices out of range" << endl;
+   }
+
+   HypreParMatrix *hmat;
+   hmat = new HypreParMatrix(comm, num_loc_rows, glob_nrows, glob_ncols, (int *)opI.data(), (HYPRE_Int *)opJ.data(), (double *)data.data(),
+                             (HYPRE_Int *)rowStarts2.data(), (HYPRE_Int *)rowStarts2.data());
+
+   return hmat;
+}

examples/solvers-dev/mg/util.hpp

@@ -0,0 +1,12 @@
+
+#pragma once
+
+#include "mfem.hpp"
+
+mfem::SparseMatrix* GetSparseMatrixFromOperator(mfem::Operator *op);
+mfem::SparseMatrix* GetSparseMatrixFromBlockMatrix(mfem::BlockMatrix * blk_mat);
+hypre_CSRMatrix* GetHypreParMatrixData(const mfem::HypreParMatrix & hypParMat);
+// mfem::HypreParMatrix* CreateHypreParMatrixFromBlocks(MPI_Comm comm, mfem::Array<int> & offsets, 
+//          mfem::Array2D<mfem::HypreParMatrix*> & blocks, mfem::Array2D<double> & coefficient);
+mfem::HypreParMatrix* CreateHypreParMatrixFromBlocks(mfem::Array<int> & offsets, mfem::BlockOperator * blockA);         
+   

examples/solvers-dev/output/memory_mesurements.txt

@@ -0,0 +1,41 @@
+
+
+p=2
+Memory in GB - Serial
+h\H     |    1/2   |  1/4   |  1/8   |   1/16 |  ndof  |Est A Memory |   memory factor
+-------------------------------------------------------------
+1/4     |   0.110  |   -    |   -    |   -    |   3888 | 0.00746407  |    14.73
+1/8     |   0.834  |  1.05  |   -    |   -    |  27744 | 0.0585069   |    14.25
+1/16    |   7.422  | 7.7362 | 10.376 |   -    | 209088 | 0.463266    |    16.02
+1/32    |  58.718  | 59.536 |   *    |   *    |1622400 | 3.68702     |    15.92
+
+
+
+p=2
+Memory in GB - Parallel 4 cores
+h\H     |    1/2   |  1/4   |  1/8   |   1/16 |  ndof  |Est A Memory | memory factor
+-------------------------------------------------------------
+1/4     |   0.177  |   -    |   -    |   -    |   3888 | 0.00746407  |   
+1/8     |   0.961  |  1.280 |   -    |   -    |  27744 | 0.0585069   |   
+1/16    |   7.559  |  8.026 | 11.686 |   -    | 209088 | 0.463266    |   
+1/32    |   61.50  |  61.43 |   *    |   *    |1622400 |  3.68702    |  
+
+
+p=2
+Memory in GB - Parallel 4 cores (only fine grid smoother)
+h\H     |    GB    |    ndof  |Est A Memory | 
+------------------------------------
+1/4     |   0.132  |     3888 | 0.00746407  |   
+1/8     |   0.727  |    27744 | 0.0585069   |   
+1/16    |   6.741  |   209088 | 0.463266    |   
+1/32    |   56.22  |  1622400 |  3.68702    |  
+
+
+
+
+
+total: 5.659763336
+PatchMat in GB: 1.730564
+Total    PatchInv in GB: 13.25743  (at most)
+Numeric  PatchInv in GB:  8.54946  (at most)
+Symbolic PatchInv in GB:  2.07769

examples/solvers-dev/petscrc_mult_options

@@ -0,0 +1,16 @@
+-prefix_push direct
+#-ksp_view
+#-ksp_monitor
+-ksp_type preonly
+#-ksp_type gmres
+-pc_type lu
+-pc_factor_mat_solver_type mumps
+#-pc_factor_mat_solver_type strumpack
+#-pc_factor_mat_solver_type superlu_dist 
+#-ksp_view
+#-prefix_pop
+#
+#-prefix_push smoother
+#-ksp_type preonly
+#-pc_type jacobi
+#-prefix_pop

examples/solvers-dev/pml_precond/additive_schwarz.cpp

@@ -0,0 +1,279 @@
+#include "additive_schwarz.hpp"
+
+// constructor
+mesh_partition::mesh_partition(Mesh *mesh_) : mesh(mesh_)
+{
+   nrpatch = mesh->GetNV();
+   int dim = mesh->Dimension();
+   element_map.resize(nrpatch);
+   //every element will contribute to the the patches of its vertices
+   // loop through the elements
+   int nrelems = mesh->GetNE();
+   for (int iel=0; iel<nrelems; ++iel)
+   {
+      // get element vertex index
+      Array<int> vertices;
+      mesh->GetElementVertices(iel,vertices);
+      int nrvert = vertices.Size();
+      // fill in the element contribution lists
+      for (int iv = 0; iv< nrvert; ++iv)
+      {
+         int ip = vertices[iv];
+         element_map[ip].Append(iel);
+      }
+   }
+
+   // Compute and store vertices coordinates of the global mesh
+   // mesh->EnsureNodes();
+   // GridFunction * nodes = mesh->GetNodes();
+   // int ndofs = nodes->FESpace()->GetNDofs();
+   // Array2D<double> coords(ndofs,dim);
+   // for (int comp = 0; comp < dim; comp++)
+   // {
+   //    // cout << comp << endl;
+   //    for (int i = 0; i < ndofs; i++)
+   //    {
+   //       coords(i,comp) = (*nodes)[nodes->FESpace()->DofToVDof(i, comp)];
+   //    }
+   // }
+
+   patch_mesh.SetSize(nrpatch);
+   for (int ip = 0; ip<nrpatch; ++ip)
+   {
+      int patch_nrelems = element_map[ip].Size();
+      element_map[ip].SetSize(patch_nrelems);
+      // need to ensure that a vertex is not added more than once
+      // and that the ordering of vertices is known for when the element is added
+      // create a list of for this patch including possible repetitions
+      // loop through elements in the patch 
+      Array<int> patch_vertices;
+      for (int iel=0; iel<patch_nrelems; ++iel)
+      {
+         // get the vertices list for the element
+         Array<int> elem_vertices;
+         int iel_idx = element_map[ip][iel];
+         mesh->GetElementVertices(iel_idx,elem_vertices);
+         patch_vertices.Append(elem_vertices);
+      }
+      patch_vertices.Sort();   
+      patch_vertices.Unique();   
+      int patch_nrvertices = patch_vertices.Size();
+
+      // create the mesh
+      patch_mesh[ip] = new Mesh(dim,patch_nrvertices,patch_nrelems);
+      // Add the vertices
+      for (int iv = 0; iv<patch_nrvertices; ++iv)
+      {
+         int vert_idx = patch_vertices[iv];
+         patch_mesh[ip]->AddVertex(mesh->GetVertex(vert_idx));
+      }
+
+      // Add the elements (for now search through all the vertices in the patch is need)
+      for (int iel=0; iel<patch_nrelems; ++iel)
+      {
+         // get the vertices list for the element
+         Array<int> elem_vertices;
+         int iel_idx = element_map[ip][iel];
+         mesh->GetElementVertices(iel_idx,elem_vertices);
+         int nrvert = elem_vertices.Size();
+         int ind[nrvert];
+         for (int iv = 0; iv<nrvert; ++iv)
+         {
+            ind[iv] = patch_vertices.FindSorted(elem_vertices[iv]);   
+         }
+         mfem::Element::Type elem_type = mesh->GetElementType(element_map[ip][iel]);
+         
+         AddElementToMesh(patch_mesh[ip],elem_type,ind);
+         
+      }
+      patch_mesh[ip]->FinalizeTopology();
+   }
+}
+
+void mesh_partition::AddElementToMesh(Mesh * mesh,mfem::Element::Type elem_type,int * ind)
+{
+   switch (elem_type)
+   {
+   case Element::QUADRILATERAL:
+      mesh->AddQuad(ind);
+      break;
+   case Element::TRIANGLE :
+      mesh->AddTri(ind);
+      break;
+   case Element::HEXAHEDRON :
+      mesh->AddHex(ind);
+      break;
+   case Element::TETRAHEDRON :
+      mesh->AddTet(ind);
+      break;
+   default:
+      MFEM_ABORT("Unknown element type");
+      break;
+   }
+}
+
+void mesh_partition::print_element_map()
+{
+   mfem::out << "Element map" << endl;
+   for (int ip = 0; ip<nrpatch; ++ip)
+   {
+      mfem::out << "Patch No: " << ip;
+      mfem::out << ", element map: " ; element_map[ip].Print(cout,element_map[ip].Size()); 
+   }
+}
+
+void mesh_partition::save_mesh_partition()
+{
+   for (int ip = 0; ip<nrpatch; ++ip)
+   {
+      ostringstream mesh_name;
+      mesh_name << "output/mesh." << setfill('0') << setw(6) << ip;
+      ofstream mesh_ofs(mesh_name.str().c_str());
+      mesh_ofs.precision(8);
+      patch_mesh[ip]->Print(mesh_ofs);
+   }
+}
+mesh_partition::~mesh_partition()
+{
+   for (int ip = 0; ip<nrpatch; ++ip)
+   {
+      delete patch_mesh[ip]; patch_mesh[ip] = nullptr;
+   }
+   patch_mesh.DeleteAll();
+}
+
+// constructor
+PatchAssembly::PatchAssembly(BilinearForm *bf_) : bf(bf_)
+{
+   fespace = bf->FESpace();
+   Mesh * mesh = fespace->GetMesh();
+   const FiniteElementCollection *fec = fespace->FEColl();
+
+   mesh_partition * p = new mesh_partition(mesh); 
+   nrpatch = p->nrpatch;
+   patch_fespaces.SetSize(nrpatch);
+   patch_dof_map.resize(nrpatch);
+   patch_mat.SetSize(nrpatch);
+   patch_mat_inv.SetSize(nrpatch);
+   ess_tdof_list.resize(nrpatch);
+   for (int ip=0; ip<nrpatch; ++ip)
+   {
+      // create finite element spaces for each patch
+      patch_fespaces[ip] = new FiniteElementSpace(p->patch_mesh[ip],fec);
+      // construct the patch tdof to global tdof map
+      int nrdof = patch_fespaces[ip]->GetTrueVSize();
+      patch_dof_map[ip].SetSize(nrdof);
+      // loop through the elements in the patch
+      for (int iel = 0; iel<p->element_map[ip].Size(); ++iel)
+      {
+         // index in the global mesh
+         int iel_idx = p->element_map[ip][iel];
+         // get the dofs of this element
+         Array<int> patch_elem_dofs;
+         Array<int> global_elem_dofs;
+         patch_fespaces[ip]->GetElementDofs(iel,patch_elem_dofs);
+         fespace->GetElementDofs(iel_idx,global_elem_dofs);
+         // the sizes have to match
+         MFEM_VERIFY(patch_elem_dofs.Size() == global_elem_dofs.Size(), "Size inconsistency");
+         // loop through the dofs and take into account the signs;
+         int ndof = patch_elem_dofs.Size();
+         for (int i = 0; i<ndof; ++i)
+         {
+            int pdof_ = patch_elem_dofs[i];
+            int gdof_ = global_elem_dofs[i];
+            int pdof = (pdof_ >= 0) ? pdof_ : abs(pdof_) - 1;
+            int gdof = (gdof_ >= 0) ? gdof_ : abs(gdof_) - 1;
+            patch_dof_map[ip][pdof] = gdof;
+         }
+      }
+      // Define the patch bilinear form and apply boundary conditions (only the LHS)
+      if (p->patch_mesh[ip]->bdr_attributes.Size())
+      {
+         Array<int> ess_bdr(p->patch_mesh[ip]->bdr_attributes.Max());
+         ess_bdr = 1;
+         patch_fespaces[ip]->GetEssentialTrueDofs(ess_bdr, ess_tdof_list[ip]);
+      }
+      BilinearForm a(patch_fespaces[ip], bf);
+      a.Assemble(); 
+      OperatorPtr Alocal;
+      a.FormSystemMatrix(ess_tdof_list[ip],Alocal);
+      delete patch_fespaces[ip];
+      patch_mat[ip] = new SparseMatrix((SparseMatrix&)(*Alocal));
+      patch_mat[ip]->Threshold(0.0);
+      // Save the inverse
+      patch_mat_inv[ip] = new KLUSolver;
+      patch_mat_inv[ip]->SetOperator(*patch_mat[ip]);
+   }
+   delete p;
+}
+
+void PatchAssembly::print_patch_dof_map()
+{
+   mfem::out << "Patch dof map" << endl;
+   for (int ip = 0; ip<nrpatch; ++ip)
+   {
+      mfem::out << "Patch No: " << ip;
+      mfem::out << ", dof map: " ; patch_dof_map[ip].Print(cout,patch_dof_map[ip].Size()); 
+   }
+}
+
+PatchAssembly::~PatchAssembly()
+{
+   for (int ip=0; ip<nrpatch; ++ip)
+   {
+      // delete patch_fespaces[ip]; patch_fespaces[ip]=nullptr;
+      delete patch_mat_inv[ip]; patch_mat_inv[ip]=nullptr;
+      delete patch_mat[ip]; patch_mat[ip]=nullptr;
+   }
+   patch_fespaces.DeleteAll();
+   patch_mat.DeleteAll();
+   patch_mat_inv.DeleteAll();
+}
+
+AddSchwarz::AddSchwarz(BilinearForm * bf_) 
+: Solver(bf_->FESpace()->GetTrueVSize(), bf_->FESpace()->GetTrueVSize())
+{
+   p = new PatchAssembly(bf_);
+   nrpatch = p->nrpatch;
+}
+
+void AddSchwarz::Mult(const Vector &r, Vector &z) const
+{
+   z = 0.0;
+   Vector rnew(r);
+   Vector znew(z);
+   Vector raux(znew.Size());
+   Vector res_local, sol_local;
+   for (int iter = 0; iter < maxit; iter++)
+   {
+      znew = 0.0;
+      for (int ip = 0; ip < nrpatch; ip++)
+      {
+         Array<int> * dof_map = &p->patch_dof_map[ip];
+         int ndofs = dof_map->Size();
+         res_local.SetSize(ndofs);
+         sol_local.SetSize(ndofs);
+
+         rnew.GetSubVector(*dof_map, res_local);
+         Array<int> ess_bdr_indices = p->ess_tdof_list[ip];
+         // need to zero out the entries corresponding to the ess_bdr
+         p->patch_mat_inv[ip]->Mult(res_local, sol_local);
+         sol_local.SetSubVector(ess_bdr_indices,0.0);
+         znew.AddElementVector(*dof_map,sol_local);
+      }
+      // Relaxation parameter
+      znew *= theta;
+      z += znew;
+      // Update residual
+      if (iter + 1 < maxit)
+      {
+         A->Mult(znew, raux);
+         rnew -= raux;
+      }
+   }
+}
+
+AddSchwarz::~AddSchwarz()
+{
+   delete p;
+}

examples/solvers-dev/pml_precond/additive_schwarz.hpp

@@ -0,0 +1,63 @@
+#pragma once
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+using namespace std;
+using namespace mfem;
+
+
+class mesh_partition // for now every vertex defines a patch 
+{
+private:
+   Mesh *mesh=nullptr;
+   void AddElementToMesh(Mesh * mesh,mfem::Element::Type elem_type,int * ind);
+   void print_element_map();
+   void save_mesh_partition();
+public:
+   int nrpatch;
+   std::vector<Array<int>> element_map; // map local (patch) element to global (original mesh) element
+   Array<Mesh *> patch_mesh;
+   // constructor
+   mesh_partition(Mesh * mesh_);
+   ~mesh_partition();
+};
+
+
+class PatchAssembly // for now every vertex defines a patch 
+{
+   FiniteElementSpace *fespace=nullptr;
+   BilinearForm *bf=nullptr;
+   void print_patch_dof_map();
+public:
+   int nrpatch;
+   Array<FiniteElementSpace *> patch_fespaces;
+   std::vector<Array<int>> patch_dof_map;
+   Array<SparseMatrix *> patch_mat;
+   Array<KLUSolver * > patch_mat_inv;
+   std::vector<Array<int>> ess_tdof_list;
+
+   // constructor
+   PatchAssembly(BilinearForm * bf_);
+   ~PatchAssembly();
+};
+
+class AddSchwarz : public Solver// 
+{
+private:
+   int nrpatch;
+   int maxit = 1;
+   double theta = 0.5;
+   FiniteElementSpace *fespace=nullptr;
+   PatchAssembly * p;
+   const Operator * A;
+   BilinearForm * bf;
+public:
+   AddSchwarz(BilinearForm * bf_);
+   void SetNumSmoothSteps(const int iter) {maxit = iter;}
+   void SetDumpingParam(const double dump_param) {theta = dump_param;}
+   virtual void SetOperator(const Operator &op) {A = &op;}
+   virtual void Mult(const Vector &r, Vector &z) const;
+   virtual ~AddSchwarz();
+};
+
+

examples/solvers-dev/pml_precond/additive_schwarzp.hpp

@@ -0,0 +1,182 @@
+#pragma once
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include <unordered_map>
+
+using namespace std;
+using namespace mfem;
+
+
+struct UniqueIndexGenerator
+{
+   int counter = 0;
+   std::unordered_map<int,int> idx;
+   int Get(int i)
+   {
+      std::unordered_map<int,int>::iterator f = idx.find(i);
+      if (f == idx.end())
+      {
+         idx[i] = counter;
+         return counter++;
+      }
+      else
+      {
+         return (*f).second;
+      }
+   }
+   void Reset()
+   {
+      counter = 0;
+      idx.clear();
+   }
+};
+
+
+class CartesianMeshPartition // for now every vertex defines a patch 
+{
+private:
+   ParMesh *pmesh=nullptr;
+public:
+   int nrpatch;
+   Array<int> patch_rank; 
+   std::vector<Array<int>> local_element_map;
+   // constructor
+   CartesianMeshPartition(ParMesh * pmesh_);
+   ~CartesianMeshPartition(){};
+};
+
+
+
+class VertexMeshPartition 
+{
+private:
+   ParMesh *pmesh=nullptr;
+public:
+   int nrpatch;
+   Array<int> patch_rank; 
+   std::vector<Array<int>> local_element_map;
+   // constructor
+   VertexMeshPartition(ParMesh * pmesh_);
+   ~VertexMeshPartition(){};
+};
+
+
+
+
+class ParMeshPartition 
+{
+private:
+   MPI_Comm comm;
+   ParMesh *pmesh=nullptr;
+   void AddElementToMesh(Mesh * mesh,mfem::Element::Type elem_type,int * ind);
+   void GetNumVertices(int type, mfem::Element::Type & elem_type, int & nrvert);
+   void save_mesh_partition();
+public:
+   int nrpatch;
+   int myelem_offset = 0;
+   Array<int> patch_rank; 
+   std::vector<Array<int>> element_map; 
+   std::vector<Array<int>> local_element_map; 
+   Array<Mesh *> patch_mesh;
+   // constructor
+   ParMeshPartition(ParMesh * pmesh_);
+   ~ParMeshPartition();
+};
+
+
+class ParPatchDofInfo 
+{
+public:
+   MPI_Comm comm = MPI_COMM_WORLD;
+   int nrpatch;
+   Array<int> patch_rank;
+   vector<Array<int>> PatchGlobalTrueDofs; // list of all the true dofs in a patch
+   vector<Array<int>> PatchTrueDofs; // list of only
+   Array<FiniteElementSpace *> patch_fespaces;
+   std::vector<Array<int>> patch_dof_map;
+   ParMeshPartition * p;
+   // constructor
+   ParPatchDofInfo(ParFiniteElementSpace *fespace);
+   // void Print();
+   ~ParPatchDofInfo();
+};
+
+
+
+
+
+class ParPatchAssembly // for now every vertex defines a patch 
+{
+   std::vector<int> tdof_offsets;
+   ParBilinearForm *bf=nullptr;
+   void compute_trueoffsets();
+   void AssemblePatchMatrices(ParPatchDofInfo * p);
+   void print_patch_dof_map(){};
+public:
+   MPI_Comm comm;
+   int nrpatch;
+   ParFiniteElementSpace *fespace=nullptr;
+   Array<int> patch_rank;
+   std::vector<Array<int>> patch_true_dofs;
+   std::vector<Array<int>> patch_local_dofs;
+
+   Array<SparseMatrix *> patch_mat;
+   Array<BilinearForm * > patch_bilinear_forms;
+   Array<KLUSolver * > patch_mat_inv;
+   std::vector<Array<int>> ess_tdof_list;
+
+   // constructor
+   ParPatchAssembly(ParBilinearForm * bf_);
+   int get_rank(int tdof);
+   ~ParPatchAssembly();
+};
+
+
+class ParPatchRestriction  {
+private:
+   MPI_Comm comm;
+   int num_procs, myid;
+   Array<int> patch_rank;
+   ParPatchAssembly * P;
+   int nrpatch;
+   Array<int> send_count;
+   Array<int> send_displ;
+   Array<int> recv_count;
+   Array<int> recv_displ;
+   int sbuff_size, rbuff_size;
+public:
+   ParPatchRestriction(ParPatchAssembly * P_);
+   // void Mult(const Vector & r , Array<BlockVector *> & res);
+   void Mult(const Vector & r , std::vector<Vector  > & res);
+   // void MultTranspose(const Array<BlockVector*> & sol, Vector & z);
+   void MultTranspose(const std::vector<Vector  > & sol, Vector & z);
+   virtual ~ParPatchRestriction(){}
+};
+
+
+
+
+
+class ParAddSchwarz : public Solver// 
+{
+private:
+   MPI_Comm comm;
+   int nrpatch;
+   int maxit = 1;
+   double theta = 0.5;
+   FiniteElementSpace *fespace=nullptr;
+   ParPatchAssembly * p;
+   const Operator * A;
+   BilinearForm * bf;
+   ParPatchRestriction * R;
+public:
+   ParAddSchwarz(ParBilinearForm * bf_);
+   void SetNumSmoothSteps(const int iter) {maxit = iter;}
+   void SetDumpingParam(const double dump_param) {theta = dump_param;}
+   virtual void SetOperator(const Operator &op) {A = &op;}
+   virtual void Mult(const Vector &r, Vector &z) const;
+   virtual ~ParAddSchwarz();
+};
+
+

examples/solvers-dev/pml_precond/complex_additive_schwarzp.cpp

@@ -0,0 +1,83 @@
+#include "complex_additive_schwarzp.hpp"
+
+// constructor
+ComplexParPatchAssembly::ComplexParPatchAssembly(ParSesquilinearForm * bf_) : bf(bf_)
+{
+
+
+
+}
+
+
+void ComplexParPatchAssembly::AssemblePatchMatrices(ParPatchDofInfo * p)
+{
+   patch_mat.SetSize(nrpatch);
+   patch_bilinear_forms.SetSize(nrpatch);
+   patch_mat_inv.SetSize(nrpatch);
+   ess_tdof_list.resize(nrpatch);
+   for (int ip=0; ip<nrpatch; ++ip)
+   {
+      patch_bilinear_forms[ip] = nullptr;
+      patch_mat_inv[ip] = nullptr;
+      patch_mat[ip] = nullptr;
+      if (p->p->patch_mesh[ip])
+      {
+         // Define the patch bilinear form and apply boundary conditions (only the LHS)
+         FiniteElementSpace * patch_fespace = p->patch_fespaces[ip];
+         Mesh * patch_mesh = p->p->patch_mesh[ip];
+         if (patch_mesh->bdr_attributes.Size())
+         {
+            Array<int> ess_bdr(patch_mesh->bdr_attributes.Max());
+            ess_bdr = 1;
+            patch_fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list[ip]);
+         }
+         patch_bilinear_forms[ip] = new SesquilinearForm(patch_fespace, bf);
+         patch_bilinear_forms[ip]->Assemble(); 
+         OperatorPtr Alocal;
+         // need to add the method FormSystemMatrix to complex_fem
+         // patch_bilinear_forms[ip]->FormSystemMatrix(ess_tdof_list[ip],Alocal);
+         patch_mat[ip] = &(SparseMatrix&)(*Alocal);
+         patch_mat[ip]->Threshold(0.0);
+         // Save the inverse
+         patch_mat_inv[ip] = new KLUSolver;
+         patch_mat_inv[ip]->SetOperator(*patch_mat[ip]);
+      }   
+   }
+}
+
+
+ComplexParPatchAssembly::~ComplexParPatchAssembly(){};
+
+
+ComplexParPatchRestriction::ComplexParPatchRestriction(ComplexParPatchAssembly * P_)
+{}
+
+void ComplexParPatchRestriction::Mult(const Vector & r , std::vector<Vector > & res)
+{}
+
+
+void ComplexParPatchRestriction::MultTranspose(const std::vector<Vector > & sol, Vector & z)
+{}
+
+
+ComplexParAddSchwarz::ComplexParAddSchwarz(ParSesquilinearForm * pbf_)
+: Solver(2*pbf_->ParFESpace()->GetTrueVSize(), 2*pbf_->ParFESpace()->GetTrueVSize())
+{
+   cout << "Testing ComplexParAddSchwarz" << endl;
+   comm = pbf_->ParFESpace()->GetComm();
+   p = new ComplexParPatchAssembly(pbf_);
+   nrpatch = p->nrpatch;
+   R = new ComplexParPatchRestriction(p);
+}
+
+void ComplexParAddSchwarz::Mult(const Vector &r, Vector &z) const
+{}
+
+ComplexParAddSchwarz::~ComplexParAddSchwarz()
+{
+   delete p;
+   delete R;
+}
+
+
+

examples/solvers-dev/pml_precond/complex_additive_schwarzp.hpp

@@ -0,0 +1,77 @@
+#pragma once
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include "additive_schwarzp.hpp"
+using namespace std;
+using namespace mfem;
+
+
+
+class ComplexParPatchAssembly 
+{
+   std::vector<int> tdof_offsets;
+   ParSesquilinearForm * bf=nullptr;
+   void compute_trueoffsets();
+   void AssemblePatchMatrices(ParPatchDofInfo * p);
+public:
+   MPI_Comm comm;
+   int nrpatch;
+   ParFiniteElementSpace *fespace=nullptr;
+   Array<int> patch_rank;
+   std::vector<Array<int>> patch_true_dofs;
+   std::vector<Array<int>> patch_local_dofs;
+   Array<SparseMatrix *> patch_mat;
+   Array<SesquilinearForm * > patch_bilinear_forms;
+   Array<KLUSolver * > patch_mat_inv;
+   std::vector<Array<int>> ess_tdof_list;
+
+   // constructor
+   ComplexParPatchAssembly(ParSesquilinearForm * bf_);
+   int get_rank(int tdof);
+   ~ComplexParPatchAssembly();
+};
+
+
+class ComplexParPatchRestriction  {
+private:
+   MPI_Comm comm;
+   int num_procs, myid;
+   Array<int> patch_rank;
+   ParPatchAssembly * P;
+   int nrpatch;
+   Array<int> send_count;
+   Array<int> send_displ;
+   Array<int> recv_count;
+   Array<int> recv_displ;
+   int sbuff_size, rbuff_size;
+public:
+   ComplexParPatchRestriction(ComplexParPatchAssembly * P_);
+   void Mult(const Vector & r , std::vector<Vector > & res);
+   void MultTranspose(const std::vector<Vector > & sol, Vector & z);
+   virtual ~ComplexParPatchRestriction(){}
+};
+
+
+class ComplexParAddSchwarz : public Solver// 
+{
+private:
+   MPI_Comm comm;
+   int nrpatch;
+   int maxit = 1;
+   double theta = 0.5;
+   FiniteElementSpace *fespace=nullptr;
+   ComplexParPatchAssembly * p;
+   const Operator * A;
+   ParSesquilinearForm * pbf;
+   ComplexParPatchRestriction * R;
+public:
+   ComplexParAddSchwarz(ParSesquilinearForm * pbf_);
+   void SetNumSmoothSteps(const int iter) {maxit = iter;}
+   void SetDumpingParam(const double dump_param) {theta = dump_param;}
+   virtual void SetOperator(const Operator &op) {A = &op;}
+   virtual void Mult(const Vector &r, Vector &z) const;
+   virtual ~ComplexParAddSchwarz();
+};
+
+

examples/solvers-dev/pml_precond/example1.cpp

@@ -0,0 +1,174 @@
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include "additive_schwarz.hpp"
+#include "../as/schwarz.hpp"
+
+using namespace std;
+using namespace mfem;
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options.
+   const char *mesh_file = "../../../data/star.mesh";
+   // const char *mesh_file = "../../../data/beam-quad.mesh";
+   int order = 1;
+   int ref_levels = 1;
+   bool visualization = true;
+   StopWatch chrono;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&ref_levels, "-ref", "--ref_levels",
+                  "Number of uniform h-refinements");               
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+
+   Mesh *mesh;
+   // mesh = new Mesh(mesh_file, 1, 1);
+   mesh = new Mesh(1, 1, Element::QUADRILATERAL, true, 1, 1, false);
+   int dim = mesh->Dimension();
+
+   for (int l = 0; l < ref_levels; l++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   FiniteElementCollection *fec = new H1_FECollection(order, dim);
+   // FiniteElementCollection *fec = new ND_FECollection(order, dim);
+   FiniteElementSpace * fespace = new FiniteElementSpace(mesh, fec);
+   Array<int> ess_tdof_list;
+   if (mesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(mesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   // 7. Set up the linear form b(.) which corresponds to the right-hand side of
+   //    the FEM linear system, which in this case is (1,phi_i) where phi_i are
+   //    the basis functions in the finite element fespace.
+   LinearForm *b = new LinearForm(fespace);
+   ConstantCoefficient one(1.0);
+   b->AddDomainIntegrator(new DomainLFIntegrator(one));
+   b->Assemble();
+
+   // 8. Define the solution vector x as a finite element grid function
+   //    corresponding to fespace. Initialize x with initial guess of zero,
+   //    which satisfies the boundary conditions.
+   GridFunction x(fespace);
+   x = 0.0;
+
+   // 9. Set up the bilinear form a(.,.) on the finite element space
+   //    corresponding to the Laplacian operator -Delta, by adding the Diffusion
+   //    domain integrator.
+   BilinearForm *a = new BilinearForm(fespace);
+   a->SetDiagonalPolicy(mfem::Matrix::DIAG_ONE);
+   a->AddDomainIntegrator(new DiffusionIntegrator(one));
+
+   // 10. Assemble the bilinear form and the corresponding linear system,
+   //     applying any necessary transformations such as: eliminating boundary
+   //     conditions, applying conforming constraints for non-conforming AMR,
+   //     static condensation, etc.
+   a->Assemble();
+
+   OperatorPtr A;
+   Vector B, X;
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+
+   cout << "Size of linear system: " << A->Height() << endl;
+
+   // Use a simple symmetric Gauss-Seidel preconditioner with PCG.
+   // GSSmoother M((SparseMatrix&)(*A));
+   // Array<double> times;
+   // GSSmoother M((SparseMatrix&)(*A));
+   // chrono.Clear();
+   // chrono.Start();
+   AddSchwarz * S1 = new AddSchwarz(a);
+   S1->SetOperator((SparseMatrix&)(*A));
+   // B = 1.0;
+   // B.Randomize(1);
+   // S1->Mult(B,X);
+   // S1->SetDumpingParam(2.0/3.0);
+   S1->SetNumSmoothSteps(1);
+   // chrono.Stop();
+   // times.Append(chrono.RealTime());
+
+   // chrono.Clear();
+   // chrono.Start();
+   // SchwarzSmoother * S2 = new SchwarzSmoother(mesh, 0, fespace,&(SparseMatrix&)(*A), ess_tdof_list);
+   // S2->SetDumpingParam(2.0/3.0);
+   // S2->SetNumSmoothSteps(3);
+   // chrono.Stop();
+   // times.Append(chrono.RealTime());
+   int maxit = 2000;
+   double rtol = 1e-8;
+   double atol = 1e-8;
+   CGSolver pcg;
+   pcg.SetPrintLevel(1);
+   pcg.SetMaxIter(maxit);
+   pcg.SetRelTol(rtol);
+   pcg.SetAbsTol(atol);
+   pcg.SetOperator((SparseMatrix&)(*A));
+   X = 0.0;
+   // pcg.SetPreconditioner(M);
+   // // pcg.Mult(B, X);
+   // // X = 0.0;
+   // chrono.Clear();
+   // chrono.Start();
+   pcg.SetPreconditioner(*S1);
+   pcg.Mult(B, X);
+   // chrono.Stop();
+   // times.Append(chrono.RealTime());
+   // X = 0.0;
+
+   // chrono.Clear();
+   // chrono.Start();
+   // pcg.SetPreconditioner(*S2);
+   // pcg.Mult(B, X);
+   // chrono.Stop();
+   // times.Append(chrono.RealTime());
+
+   // cout << "S1 Times: " << times[0] << ", " << times[2] << endl;
+   // cout << "S2 Times: " << times[1] << ", " << times[3] << endl;
+
+
+
+   // 12. Recover the solution as a finite element grid function.
+   a->RecoverFEMSolution(X, *b, x);
+
+   // 14. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      sol_sock << "mesh\n" << *mesh << flush;
+   }
+
+   // 15. Free the used memory.
+   delete S1;
+   // delete S2;
+   delete a;
+   delete b;
+   delete fespace;
+   delete fec;
+   delete mesh;
+
+   return 0;
+}

examples/solvers-dev/pml_precond/example1p.cpp

@@ -0,0 +1,196 @@
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include "additive_schwarzp.hpp"
+#include "../as/schwarzp.hpp"
+
+using namespace std;
+using namespace mfem;
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   const char *mesh_file = "../../../data/star.mesh";
+   int order = 1;
+   int ref_levels = 1;
+   bool visualization = true;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&ref_levels, "-ref", "--ref_levels",
+                  "Number of uniform h-refinements");                              
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   Mesh * mesh = new Mesh(1, 1, Element::QUADRILATERAL, true, 1, 1, false);
+   // Mesh * mesh = new Mesh(1, 1,1, Element::HEXAHEDRON, true, 1, 1, 1, false);
+   int dim = mesh->Dimension();
+
+   for (int l = 0; l < ref_levels; l++)
+   {
+      mesh->UniformRefinement();
+   }
+
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+
+   FiniteElementCollection *fec = new H1_FECollection(order, dim);
+   // FiniteElementCollection *fec = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   HYPRE_Int size = fespace->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "Number of finite element unknowns: " << size << endl;
+   }
+
+   Array<int> ess_tdof_list;
+   if (pmesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   ParLinearForm *b = new ParLinearForm(fespace);
+   ConstantCoefficient one(1.0);
+   b->AddDomainIntegrator(new DomainLFIntegrator(one));
+   b->Assemble();
+
+   ParGridFunction x(fespace);
+   x = 0.0;
+
+   ParBilinearForm *a = new ParBilinearForm(fespace);
+   a->AddDomainIntegrator(new DiffusionIntegrator(one));
+
+   chrono.Clear();
+   chrono.Start();
+   a->Assemble();
+
+   HypreParMatrix A;
+   Vector B, X;
+   a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+   chrono.Stop();
+   if (myid == 0)  cout << "Form Linear System time: " << chrono.RealTime() << endl;
+
+   // Array<int>elem_vertices;
+   // for (int iel = 0; iel<pmesh->GetNE(); iel++)
+   // {
+   //    pmesh->GetElementVertices(iel,elem_vertices);
+   //    cout << "myid, iel: " << myid <<", " << iel << ", " ; elem_vertices.Print(cout,10);
+   // }
+
+   Array<double> times(4);
+
+   int maxit = 200;
+   double rtol = 1e-8;
+   double atol = 1e-8;
+   CGSolver pcg(MPI_COMM_WORLD);
+   pcg.SetPrintLevel(1);
+   pcg.SetMaxIter(maxit);
+   pcg.SetRelTol(rtol);
+   pcg.SetAbsTol(atol);
+   pcg.SetOperator(A);
+
+
+   chrono.Clear();
+   chrono.Start();
+   ParSchwarzSmoother * prec1 = new ParSchwarzSmoother(pmesh,0,fespace, &A);
+   prec1->SetNumSmoothSteps(1);
+   prec1->SetDumpingParam(0.5);
+   chrono.Stop();
+   times[0] = chrono.RealTime();
+
+   pcg.SetPreconditioner(*prec1); 
+   X = 0.0;
+   chrono.Clear();
+   chrono.Start();
+   pcg.Mult(B, X);
+   chrono.Stop();
+   times[1] = chrono.RealTime();
+
+   delete prec1;
+
+   chrono.Clear();
+   chrono.Start();
+   ParAddSchwarz *prec2 = new ParAddSchwarz(a);
+   prec2->SetOperator(A);
+   prec2->SetNumSmoothSteps(1);
+   prec2->SetDumpingParam(0.5);
+   chrono.Stop();
+   times[2] = chrono.RealTime();
+
+
+   pcg.SetPreconditioner(*prec2); 
+   X = 0.0;
+   chrono.Clear();
+   chrono.Start();
+   pcg.Mult(B, X);
+   chrono.Stop();
+   times[3] = chrono.RealTime();
+   delete prec2;
+
+   if (myid == 0)
+   {
+      cout << "prec 1 construction time: " << times[0] << endl;
+      // cout << "prec 1 solution time:     " << times[1] << endl;
+      cout << "prec 2 construction time: " << times[2] << endl;
+      // cout << "prec 2 solution time:     " << times[3] << endl;
+   }
+   
+   a->RecoverFEMSolution(X, *b, x);
+
+   // 16. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      // socketstream mesh_sock(vishost, visport);
+      // mesh_sock << "parallel " << num_procs << " " << myid << "\n";
+      // mesh_sock.precision(8);
+      // mesh_sock << "mesh\n" << *pmesh  << "keys n/n" << flush;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh  << x <<"keys " << flush;
+   }
+
+   // // 17. Free the used memory.
+   delete a;
+   delete b;
+   delete fespace;
+   if (order > 0) { delete fec; }
+   delete pmesh;
+
+   MPI_Finalize();
+
+   return 0;
+}

examples/solvers-dev/pml_precond/helmholtzp.cpp

@@ -0,0 +1,834 @@
+//
+// Compile with: make helmholtz
+//
+// Sample runs:  helmholtz -m ../data/one-hex.mesh
+//               helmholtz -m ../data/fichera.mesh
+//               helmholtz -m ../data/fichera-mixed.mesh
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Helmholtz problem
+//               -Delta p - omega^2 p = 1 with impedance boundary condition.
+//
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include "complex_additive_schwarzp.hpp"
+
+using namespace std;
+using namespace mfem;
+
+// Exact solution and r.h.s., see below for implementation.
+void get_helmholtz_solution_Re(const Vector &x, double & p, double dp[], double & d2p);
+void get_helmholtz_solution_Im(const Vector &x, double & p, double dp[], double & d2p);
+double p_exact_Re(const Vector &x);
+double p_exact_Im(const Vector &x);
+double f_exact_Re(const Vector &x);
+double f_exact_Im(const Vector &x);
+double g_exact_Re(const Vector &x);
+double g_exact_Im(const Vector &x);
+void grad_exact_Re(const Vector &x, Vector &grad_Re);
+void grad_exact_Im(const Vector &x, Vector &grad_Im);
+
+// pml
+void pml_function(const Vector &x, std::vector<std::complex<double>> & dxs);
+double pml_detJ_Re(const Vector &x);
+double pml_detJ_Im(const Vector &x);
+void pml_detJ_JT_J_inv_Re(const Vector &x, DenseMatrix &M);
+void pml_detJ_JT_J_inv_Im(const Vector &x, DenseMatrix &M);
+
+
+int dim;
+double omega;
+int sol = 1;
+bool pml = false;
+double length = 1.0;
+double pml_length = 0.25;
+bool scatter = false;
+
+#ifndef MFEM_USE_PETSC
+#error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+#endif
+
+int main(int argc, char *argv[])
+{
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);                    // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);      // Determine process identifier
+
+   //-----------------------------------------------------------------------------
+
+   // 2. Parse command-line options.
+   // geometry file
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   bool visualization = 1;
+   // number of wavelengths
+   double k = 0.5;
+   // number of mg levels
+   int ref = 1;
+   // number of initial ref
+   int initref = 1;
+   // dimension
+   int nd = 2;
+
+
+   // PETSC
+   // const char *petscrc_file = "";
+   const char *petscrc_file = "petscrc_mult_options";
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&nd, "-nd", "--dim","Problem space dimension");               
+   args.AddOption(&sol, "-sol", "--exact",
+                  "Exact solution flag - 0:polynomial, 1: plane wave, -1: unknown exact");               
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&pml, "-pml", "--pml", "-no-pml",
+                  "--no-pml", "Enable PML.");         
+   args.AddOption(&pml_length, "-pml_length", "--pml_length",
+                  "Length of the PML region in each direction");        
+   args.AddOption(&length, "-length", "--length",
+                  "length of the domainin in each direction.");                                       
+   args.AddOption(&ref, "-ref", "--ref",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&petscrc_file, "-petscopts", "--petscopts",
+                  "PetscOptions file to use.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&scatter, "-scat", "--scattering-prob", "-no-scat",
+                  "--no-scattering", "Solve a scattering problem");               
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+   // Angular frequency
+   omega = 2.0 * M_PI * k;
+   // omega = k;
+
+   // 2b. Initialize PETSc
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+   //-----------------------------------------------------------------------------
+
+   // if (scatter) pml = true; // for now only scattering problems with pml
+
+   // 3. Read the mesh from the given mesh file.
+   Mesh *mesh;
+   
+   if (nd == 2) 
+   {
+      if (scatter) 
+      {
+         mesh_file = "../../data/rectwhole7_2attr.e";
+         mesh = new Mesh(mesh_file, 1, 1);
+      }
+      else
+      {
+         mesh = new Mesh(1, 1, Element::QUADRILATERAL, true, length, length, false);
+      }
+   }
+   else
+   {
+      if (scatter)
+      {
+         // mesh_file = "../../data/hexwhole7.e";
+         // mesh_file = "../../data/hexwhole.e";
+         // mesh_file = "../../data/hexa728.mesh";
+         // mesh_file = "./hexa728.mesh";
+         mesh_file = "../../data/hexwhole7.e";
+         mesh = new Mesh(mesh_file, 1, 1);
+      }
+      else
+      {
+         mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true, length, length, length, false);
+      }
+   }
+   // normalize mesh
+
+   mesh->EnsureNodes();
+   GridFunction * nodes = mesh->GetNodes();
+   // Assuming square/cubic domain 
+   double min_coord =  nodes->Min();
+   double max_coord =  nodes->Max();
+   double domain_length = abs(max_coord-min_coord); 
+   // shift to zero
+   *nodes -= min_coord;
+   // scale to one
+   *nodes *= 1./domain_length;
+
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   // 5. Define a parallel mesh and delete the serial mesh.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   // delete mesh;
+
+   // ----------------------------------------------------------------------------
+
+   // 6. Define a finite element space on the mesh.
+   FiniteElementCollection *fec = new H1_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+   std::vector<HypreParMatrix*>  P(ref);
+
+   for (int i = 0; i < ref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*fespace);
+      pmesh->UniformRefinement();
+      fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      fespace->GetTrueTransferOperator(cfespace, Tr);
+      HypreParMatrix * Paux;
+      Tr.Get(Paux);
+      P[i] = new HypreParMatrix(*Paux);
+   }
+
+   // 6. Set up the linear form (Real and Imaginary part)
+   FunctionCoefficient f_Re(f_exact_Re);
+   FunctionCoefficient g_Re(g_exact_Re);
+   VectorFunctionCoefficient grad_Re(sdim, grad_exact_Re);
+   FunctionCoefficient f_Im(f_exact_Im);
+   FunctionCoefficient g_Im(g_exact_Im);
+   VectorFunctionCoefficient grad_Im(sdim, grad_exact_Im);
+
+   // ParLinearForm *b_Re(new ParLinearForm);
+   ParComplexLinearForm b(fespace, ComplexOperator::HERMITIAN);
+   if (!scatter) // if scattering problem the source is zero and is driven by bc
+   {
+      b.AddDomainIntegrator(new DomainLFIntegrator(f_Re),new DomainLFIntegrator(f_Im));
+      if (!pml && sol >=0) // if exact solution exists. Otherwise use homogeneous impedence (gradp . n + i omega p = 0)
+      {
+         b.AddBoundaryIntegrator(new BoundaryNormalLFIntegrator(grad_Re),
+                                 new BoundaryNormalLFIntegrator(grad_Im));
+         b.AddBoundaryIntegrator(new BoundaryLFIntegrator(g_Re),
+                                 new BoundaryLFIntegrator(g_Im));
+      }
+   }
+   b.real().Vector::operator=(0.0);
+   b.imag().Vector::operator=(0.0);                        
+   b.Assemble();
+
+   // 7. Set up the bilinear form (Real and Imaginary part)
+   ConstantCoefficient one(1.0);
+   ConstantCoefficient sigma(-pow(omega, 2));
+
+   ParSesquilinearForm * a = new ParSesquilinearForm(fespace,ComplexOperator::HERMITIAN);
+   ConstantCoefficient impedance(omega);
+
+
+   MatrixFunctionCoefficient c1_Re(dim,pml_detJ_JT_J_inv_Re);
+   MatrixFunctionCoefficient c1_Im(dim,pml_detJ_JT_J_inv_Im);
+
+   FunctionCoefficient det_Re(pml_detJ_Re);
+   FunctionCoefficient det_Im(pml_detJ_Im);
+   ProductCoefficient c2_Re(det_Re,sigma);
+   ProductCoefficient c2_Im(det_Im,sigma);
+
+   Array<int> bdr_attr(pmesh->bdr_attributes.Max());
+   bdr_attr = 0;
+   bdr_attr[0] = 1;
+   if (!scatter) bdr_attr = 1;
+   RestrictedCoefficient imp_rest(impedance,bdr_attr);
+
+   a->AddDomainIntegrator(new DiffusionIntegrator(c1_Re),new DiffusionIntegrator(c1_Im));
+   a->AddDomainIntegrator(new MassIntegrator(c2_Re),new MassIntegrator(c2_Im));
+   if (!pml)
+   {
+      a->AddBoundaryIntegrator(NULL,new BoundaryMassIntegrator(imp_rest));
+      // a.AddBoundaryIntegrator(NULL,new BoundaryMassIntegrator(impedance));
+   }
+   a->Assemble();
+   a->Finalize();
+
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+   ess_bdr = 0;
+   if (scatter)
+   {
+      if (pml)
+      {
+         ess_bdr = 1;
+      }
+      else
+      {
+         ess_bdr[1] = 1;
+      }
+   }
+   fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+
+   // Solution grid function
+   ParComplexGridFunction p_gf(fespace);
+   ParComplexGridFunction p_gf_ex(fespace);
+   FunctionCoefficient p_Re(p_exact_Re);
+   FunctionCoefficient p_Im(p_exact_Im);
+   p_gf = 0.0;
+   p_gf_ex.ProjectCoefficient(p_Re,p_Im);
+   if (!pml && sol >= 0 ) 
+   {
+      p_gf.ProjectBdrCoefficient(p_Re,p_Im,ess_bdr);
+   }
+   if (scatter) 
+   {
+      p_gf.ProjectBdrCoefficient(p_Re,p_Im,ess_bdr);
+   }
+
+   OperatorHandle Ah;
+   Vector X, B;
+
+   a->FormLinearSystem(ess_tdof_list, p_gf, b, Ah, X, B);
+
+   ComplexHypreParMatrix * AZ = Ah.As<ComplexHypreParMatrix>();
+   HypreParMatrix * A = AZ->GetSystemMatrix();
+
+
+   if (myid == 0)
+   {
+      cout << "Size of fine grid system: "
+           << A->GetGlobalNumRows() << " x " << A->GetGlobalNumCols() << endl;
+   }
+
+   PetscLinearSolver * petsc = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   // Convert to PetscParMatrix
+   petsc->SetOperator(PetscParMatrix(A, Operator::PETSC_MATAIJ));
+   petsc->Mult(B,X);
+
+   a->RecoverFEMSolution(X,B,p_gf);
+
+   
+
+   ComplexParAddSchwarz * test = new ComplexParAddSchwarz(a);
+
+   delete test;
+
+
+
+   if (!scatter && sol >= 0 )
+   {
+      // int order_quad = max(2, 2*order+1);
+      // const IntegrationRule *irs[Geometry::NumGeom];
+      // for (int i=0; i < Geometry::NumGeom; ++i)
+      // {
+         // irs[i] = &(IntRules.Get(i, order_quad));
+      // }
+
+      const int h1_norm_type = 1;
+      double L2error;
+      double H1error;
+
+      double L2err_Re  = p_gf.real().ComputeL2Error(p_Re);
+      double L2err_Im  = p_gf.imag().ComputeL2Error(p_Im);
+
+      double loc_H1err_Re = p_gf.real().ComputeH1Error(&p_Re, &grad_Re, &one, 1.0, h1_norm_type);
+      double loc_H1err_Im = p_gf.imag().ComputeH1Error(&p_Im, &grad_Im, &one, 1.0, h1_norm_type);
+      double H1err_Re = GlobalLpNorm(2.0, loc_H1err_Re, MPI_COMM_WORLD);
+      double H1err_Im = GlobalLpNorm(2.0, loc_H1err_Im, MPI_COMM_WORLD);
+      // double norm_Re = ComputeGlobalLpNorm(2, p_Re, *pmesh, irs);
+      // double norm_Im = ComputeGlobalLpNorm(2, p_Im, *pmesh, irs);
+         
+      L2error = sqrt(L2err_Re*L2err_Re + L2err_Im*L2err_Im);
+      H1error = sqrt(H1err_Re*H1err_Re + H1err_Im*H1err_Im);
+      // double L2norm = sqrt(norm_Re*norm_Re + norm_Im*norm_Im);
+      if (myid == 0)
+      {
+         cout << " || p_h - p ||_{H^1} = " << H1error <<  endl;
+         cout << " || p_h - p ||_{L^2} = " << L2error <<  endl;
+      }
+   }
+
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      string keys;
+      if(dim ==2 )
+      {
+         keys = "keys mrRljc\n";
+      }
+      else
+      {
+         keys = "keys mc\n";
+      }
+      socketstream sol_sock_re(vishost, visport);
+      sol_sock_re << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_re.precision(8);
+      sol_sock_re << "solution\n" << *pmesh << p_gf.real() << "window_title 'Numerical Pressure (real part)' " 
+               << keys << flush;
+   }
+
+   delete a;
+   delete fespace;
+   delete fec;
+   delete pmesh;
+   MFEMFinalizePetsc();
+   MPI_Finalize();
+   return 0;
+}
+
+//define exact solutions
+void get_helmholtz_solution_Re(const Vector &x, double & p, double dp[], double & d2p)
+{
+
+   if (sol == 0) // polynomial
+   {  
+      if (dim == 3)
+      {
+         p = x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+         dp[0] = (1.0 - 2.0 *x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+         dp[1] = (1.0 - 2.0 *x[1]) * x[0]*(1.0 - x[0]) * x[2]*(1.0 - x[2]);
+         dp[2] = (1.0 - 2.0 *x[2]) * x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]);
+         d2p = -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[1]) * x[1] 
+               -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[2]) * x[2] 
+               -2.0*(-1.0 + x[1]) * x[1] * (-1.0 + x[2]) * x[2];   
+      }
+      else
+      {
+         p = x[1] * (1.0 - x[1])* x[0] * (1.0 - x[0]);
+         dp[0] = (1.0 - 2.0 *x[0]) * x[1]*(1.0 - x[1]);
+         dp[1] = (1.0 - 2.0 *x[1]) * x[0]*(1.0 - x[0]);
+         d2p = - 2.0 * x[1] * (1.0 - x[1]) 
+               - 2.0 * x[0] * (1.0 - x[0]);   
+      }
+   }
+
+   else if(sol == 1) // Plane wave
+   {
+      double alpha;
+      if (dim == 2)
+      {
+         alpha = omega/sqrt(2);
+         p = cos(alpha * ( x(0) + x(1) ) );
+         dp[0] = -alpha * sin(alpha * ( x(0) + x(1) ) );
+         dp[1] = dp[0];
+         d2p = -2.0 * alpha * alpha * p;
+      }
+      else 
+      {
+         alpha = omega/sqrt(3);
+         p = cos(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[0] = -alpha * sin(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[1] = dp[0];
+         dp[2] = dp[0];
+         d2p = -3.0 * alpha * alpha * p;
+      }
+   }
+   else if (sol == 2)
+   {
+      if (dim == 2 )
+      {
+         // shift to avoid singularity
+         double shift = 0.1;
+         if (scatter) shift = -0.5;
+         double x0 = x(0) + shift;
+         double x1 = x(1) + shift;
+         //
+         double r = sqrt(x0 * x0 + x1 * x1);
+
+         p = cos(omega * r);
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_xx = (1.0 / r) * (1.0 - r_x * r_x);
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+
+         dp[0] = - omega * sin(omega * r) * r_x; 
+         dp[1] = - omega * sin(omega * r) * r_y; 
+
+         d2p = -omega*omega * cos(omega * r)*r_x * r_x - omega * sin(omega*r) * r_xx
+               -omega*omega * cos(omega * r)*r_y * r_y - omega * sin(omega*r) * r_yy;
+      }
+      else
+      {
+         // shift to avoid singularity
+         double shift = 0.1;
+         if (scatter) shift = -0.5;
+         double x0 = x(0) + shift;
+         double x1 = x(1) + shift;
+         double x2 = x(2) + shift;
+         //
+         double r = sqrt(x0 * x0 + x1 * x1 + x2 * x2);
+
+         p = cos(omega * r);
+
+         double r_x = x0 / r;
+         double r_y = x1 / r;
+         double r_z = x2 / r;
+         double r_xx = (1.0 / r) * (1.0 - r_x * r_x);
+         double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+         double r_zz = (1.0 / r) * (1.0 - r_z * r_z);
+
+         dp[0] = - omega * sin(omega * r) * r_x; 
+         dp[1] = - omega * sin(omega * r) * r_y; 
+         dp[2] = - omega * sin(omega * r) * r_z; 
+
+         d2p = -omega*omega * cos(omega * r)*r_x * r_x - omega * sin(omega*r) * r_xx
+               -omega*omega * cos(omega * r)*r_y * r_y - omega * sin(omega*r) * r_yy
+               -omega*omega * cos(omega * r)*r_z * r_z - omega * sin(omega*r) * r_zz;
+      }
+   }
+}
+
+void get_helmholtz_solution_Im(const Vector &x, double & p, double dp[], double & d2p)
+{
+   if (sol == 0) // polynomial
+   {  
+      if (dim == 3)
+      {
+         p = x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+         dp[0] = (1.0 - 2.0 *x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+         dp[1] = (1.0 - 2.0 *x[1]) * x[0]*(1.0 - x[0]) * x[2]*(1.0 - x[2]);
+         dp[2] = (1.0 - 2.0 *x[2]) * x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]);
+         d2p = -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[1]) * x[1] 
+               -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[2]) * x[2] 
+               -2.0*(-1.0 + x[1]) * x[1] * (-1.0 + x[2]) * x[2];   
+      }
+      else
+      {
+         p = x[1] * (1.0 - x[1])* x[0] * (1.0 - x[0]);
+         dp[0] = (1.0 - 2.0 *x[0]) * x[1]*(1.0 - x[1]);
+         dp[1] = (1.0 - 2.0 *x[1]) * x[0]*(1.0 - x[0]);
+         d2p = - 2.0 * x[1] * (1.0 - x[1]) 
+               - 2.0 * x[0] * (1.0 - x[0]);   
+      }
+   }
+   else if (sol == 1)// plane wave
+   {
+      double alpha; 
+      if (dim == 2)
+      {
+         alpha = omega/sqrt(2);
+         p = -sin(alpha * ( x(0) + x(1) ) );
+         dp[0] = -alpha * cos(alpha * ( x(0) + x(1) ) );
+         dp[1] = dp[0];
+         d2p = -2.0 * alpha * alpha * p;
+      }
+      else 
+      {
+         alpha = omega/sqrt(3);
+         p = -sin(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[0] = -alpha * cos(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[1] = dp[0];
+         dp[2] = dp[0];
+         d2p = -3.0 * alpha * alpha * p;
+      }   
+   }
+}
+
+
+double p_exact_Re(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Re(x, p, dp, d2p);
+   if (scatter)
+   {
+      if (dim == 2)
+      {
+         if (abs(x(0)-1.) < 1e-13 || abs(x(1)-1.0) < 1e-13 ||
+             abs(x(0)) < 1e-13    || abs(x(1)) < 1e-13)
+         {
+            p = 0.0;
+         }
+      }
+      else
+      {
+         if (abs(x(0)-1.) < 1e-13 || abs(x(1)-1.0) < 1e-13 || abs(x(2)-1.0) < 1e-13 ||
+             abs(x(0)) < 1e-13    || abs(x(1)) < 1e-13 || abs(x(2)) < 1e-13)
+         {
+            p = 0.0;
+         }
+      }
+   }
+   return p;
+}
+double p_exact_Im(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Im(x, p, dp, d2p);
+   if (scatter)
+   {
+      if (dim == 2)
+      {
+         if (abs(x(0)-1.) < 1e-13 || abs(x(1)-1.0) < 1e-13 ||
+             abs(x(0)) < 1e-13    || abs(x(1)) < 1e-13)
+         {
+            p = 0.0;
+         }
+      }
+      else
+      {
+         if (abs(x(0)-1.) < 1e-13 || abs(x(1)-1.0) < 1e-13 || abs(x(2)-1.0) < 1e-13 ||
+             abs(x(0)) < 1e-13    || abs(x(1)) < 1e-13 || abs(x(2)) < 1e-13)
+         {
+            p = 0.0;
+         }
+      }
+   }
+   // p *= -1.0;
+   return p;
+}
+
+//calculate RHS from exact solution f = - \Delta u
+double f_exact_Re(const Vector &x)
+{
+   double p_re, d2p_re, p_im, d2p_im;
+   double dp_re[3], dp_im[3];
+   double f_re;
+   f_re = 0.0;
+   if (sol < 0)
+   {
+      double x0 = length/2.0;
+      double x1 = length/2.0;
+      double x2 = length/2.0;
+      double alpha,beta;
+      double n = 5.0 * omega/M_PI;
+      double coeff = pow(n,2)/M_PI;
+      beta = pow(x0-x(0),2) + pow(x1-x(1),2);
+      if (dim == 3) beta += pow(x2-x(2),2);
+      alpha = -pow(n,2) * beta;
+      f_re = coeff*exp(alpha);
+   }
+   else
+   {
+      get_helmholtz_solution_Re(x, p_re, dp_re, d2p_re);
+      get_helmholtz_solution_Im(x, p_im, dp_im, d2p_im);
+      f_re = -d2p_re - omega * omega * p_re;
+   }
+   return f_re;
+}
+double f_exact_Im(const Vector &x)
+{
+   double p_re, d2p_re, p_im, d2p_im;
+   double dp_re[3], dp_im[3];
+   double f_im;
+   f_im = 0.0;
+   if (sol < 0)
+   {
+      // double x0 = 0.6;
+      // double x1 = 0.6;
+      // double alpha;
+      // double n = 5.0 * omega/M_PI;
+      // double coeff = pow(n,2)/M_PI;
+      // alpha = -pow(n,2) * sqrt(pow(x0-x(0),2) + pow(x1-x(1),2));
+      // f_im = coeff*exp(alpha);
+   }
+   else
+   {
+      get_helmholtz_solution_Re(x, p_re, dp_re, d2p_re);
+      get_helmholtz_solution_Im(x, p_im, dp_im, d2p_im);
+      f_im = -d2p_im - omega * omega * p_im;
+   }
+   return f_im;   
+}
+
+void grad_exact_Re(const Vector &x, Vector &dp)
+{
+   double p, d2p;
+   get_helmholtz_solution_Re(x, p, dp, d2p);
+}
+void grad_exact_Im(const Vector &x, Vector &dp)
+{
+   double p, d2p;
+   get_helmholtz_solution_Im(x, p, dp, d2p);
+}
+
+//define impedence coefficient: i omega p
+double g_exact_Re(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Im(x, p, dp, d2p);
+
+   return -omega * p;
+}
+double g_exact_Im(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Re(x, p, dp, d2p);
+
+   return omega * p;
+}
+
+
+
+
+
+
+
+void pml_function(const Vector &x, std::vector<std::complex<double>> & dxs)
+{
+   double L = length;
+   double n = 2.0;
+   double lbeg, lend;
+   double c = 50.0;
+   double c1 = pml_length; 
+   double c2 = length-pml_length; 
+   double coeff;
+   // initialize to one
+   for (int i = 0; i<dim; ++i) dxs[i] = complex<double>(1.0,0.0); 
+
+   if (pml)
+   {
+   // Stretch in each direction independenly
+      for (int i = 0; i<dim; ++i)
+      {
+         if (x(i) >= c2)
+         {
+            lbeg = c2;
+            lend = L;
+            coeff = n * c / omega / pow(lend-lbeg,n); 
+            dxs[i] = complex<double>(1.0,0.0) - complex<double>(0.0,coeff * pow(x(i)-lbeg, n-1.0)); 
+         }
+         if (x(i) <= c1)
+         {
+            lbeg = c1;
+            lend = 0.0;
+            coeff = n * c / omega / pow(lend-lbeg,n); 
+            dxs[i] = complex<double>(1.0,0.0) + complex<double>(0.0, coeff * pow(x(i)-lbeg, n-1.0)); 
+         } 
+      }
+   }
+}
+
+double pml_detJ_Re(const Vector &x)
+{
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0,0.0);
+   pml_function(x, dxs);
+   for (int i=0; i<dim; ++i) det *= dxs[i];
+   return det.real();
+}
+
+double pml_detJ_Im(const Vector &x)
+{
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0,0.0);
+   pml_function(x, dxs);
+   for (int i=0; i<dim; ++i) det *= dxs[i];
+   return det.imag();
+}
+
+void pml_detJ_JT_J_inv_Re(const Vector &x, DenseMatrix &M)
+{
+   std::vector<complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det(1.0,0.0);
+   pml_function(x, dxs);
+
+   for (int i = 0; i<dim; ++i)
+   {
+      diag[i] = complex<double>(1.0,0.0) / pow(dxs[i],2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M=0.0;
+
+   for (int i = 0; i<dim; ++i)
+   {
+      complex<double> temp = det * diag[i];
+      M(i,i) = temp.real();
+   }
+}
+
+void pml_detJ_JT_J_inv_Im(const Vector &x, DenseMatrix &M)
+{
+   std::vector<std::complex<double>> diag(dim);
+   std::vector<std::complex<double>> dxs(dim);
+   complex<double> det = 1.0;
+   pml_function(x, dxs);
+
+   for (int i = 0; i<dim; ++i)
+   {
+      diag[i] = complex<double>(1.0,0.0) / pow(dxs[i],2);
+      det *= dxs[i];
+   }
+
+   M.SetSize(dim);
+   M=0.0;
+
+   for (int i = 0; i<dim; ++i)
+   {
+      complex<double> temp = det * diag[i];
+      M(i,i) = temp.imag();
+   }
+}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+// int ndofs = nodes->FESpace()->GetNDofs();
+//    Vector xcoords(ndofs), ycoords(ndofs), zcoords(ndofs);
+
+//    for (int comp = 0; comp < nodes->FESpace()->GetVDim(); comp++)
+//    {
+//       for (int i = 0; i < ndofs; i++)
+//       {
+//          if (comp == 0)
+//          {
+//             xcoords(i) = *nodes[nodes->FESpace()->DofToVDof(i, comp)];
+//          }
+//          else if (comp == 1)
+//          {
+//             ycoords(i) = *nodes[nodes->FESpace()->DofToVDof(i, comp)];
+//          }
+//          else if (comp == 2)
+//          {
+//             zcoords(i) = *nodes[nodes->FESpace()->DofToVDof(i, comp)];
+//          }
+//       }
+//    }

examples/solvers-dev/pml_precond/makefile

@@ -0,0 +1,60 @@
+# Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at the
+# Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights reserved.
+# See file COPYRIGHT for details.
+#
+# This file is part of the MFEM library. For more information and source code
+# availability see http://mfem.org.
+#
+# MFEM is free software; you can redistribute it and/or modify it under the
+# terms of the GNU Lesser General Public License (as published by the Free
+# Software Foundation) version 2.1 dated February 1999.
+
+# Use the MFEM build directory
+MFEM_DIR ?= ../../..
+MFEM_BUILD_DIR ?= ../../..
+SRC = $(if $(MFEM_DIR:../../..=),$(MFEM_DIR)/examples/solvers-dev/pml_precond,)
+CONFIG_MK = $(MFEM_BUILD_DIR)/config/config.mk
+
+MFEM_LIB_FILE = mfem_is_not_built
+-include $(CONFIG_MK)
+
+SEQ_EXAMPLES = example1
+PAR_EXAMPLES = example1p
+ifeq ($(MFEM_USE_MPI),NO)
+   EXAMPLES = $(SEQ_EXAMPLES)
+else
+   EXAMPLES = $(PAR_EXAMPLES) $(SEQ_EXAMPLES)
+endif
+
+.SUFFIXES:
+.SUFFIXES: .o .cpp .mk
+.PHONY: all clean
+.PRECIOUS: %.o
+
+COMMON_O= complex_additive_schwarzp.o additive_schwarz.o additive_schwarzp.o ../as/schwarz.o ../as/schwarzp.o
+
+# Remove built-in rules
+%: %.cpp
+%.o: %.cpp
+
+all: $(EXAMPLES)
+
+# Rules for building the EXAMPLES
+
+%: $(SRC)%.cpp $(COMMON_O) $(MFEM_LIB_FILE) $(CONFIG_MK)
+	$(MFEM_CXX) $(MFEM_FLAGS) $< -o $@ $(COMMON_O) $(MFEM_LIBS)
+
+# Rules for compiling miniapp dependencies
+$(COMMON_O) $($(EXAMPLES)): \
+%.o: $(SRC)%.cpp $(SRC)%.hpp $(CONFIG_MK)
+	$(MFEM_CXX) $(MFEM_FLAGS) -c $(<) -o $(@)
+
+# Generate an error message if the MFEM library is not built and exit
+$(MFEM_LIB_FILE):
+	$(error The MFEM library is not built)
+
+clean:
+	rm -f *.o *~ $(SEQ_EXAMPLES) $(PAR_EXAMPLES)
+	rm -rf *.dSYM *.TVD.*breakpoints
+
+

examples/solvers-dev/run.py

@@ -0,0 +1,28 @@
+import subprocess
+import os
+import sys 
+
+
+print ("Python Script Running\n")
+
+args = []
+
+args.append(' -o 2') 
+args.append(' -k 1.59154943092') 
+args.append(' -initref 2') 
+args.append(' -ref 2') 
+args.append(' -sol 3') 
+run = 'mpirun'
+nprocs = ' -np 6'
+ex = ' FOSLS_maxwellp'
+
+cmd = run + nprocs + ex
+
+for i in args:
+	cmd += i
+
+
+print(cmd)
+os.system(cmd) 
+ 
+#end thats all

examples/waves/AMS_LS.hpp

@@ -0,0 +1,66 @@
+#include "mfem.hpp"
+
+using namespace mfem;
+using namespace std;
+
+
+// Discrete gradient matrix
+HypreParMatrix* GetDiscreteGradientOp(ParFiniteElementSpace *fespace)
+{
+   int dim = fespace->GetMesh()->Dimension();
+   // int sdim = fespace->GetMesh()->SpaceDimension();
+   // const FiniteElementCollection *fec = fespace->FEColl();
+   int p = 1;
+   if (fespace->GetNE() > 0)
+   {
+      p = fespace->GetOrder(0);
+   }
+   ParMesh *pmesh = fespace->GetParMesh();
+   FiniteElementCollection *vert_fec;
+   vert_fec = new H1_FECollection(p, dim);
+   ParFiniteElementSpace *vert_fespace = new ParFiniteElementSpace(pmesh,vert_fec);
+   // generate and set the discrete gradient
+   ParDiscreteLinearOperator *grad;
+   grad = new ParDiscreteLinearOperator(vert_fespace, fespace);
+   grad->AddDomainInterpolator(new GradientInterpolator);
+   grad->Assemble();
+   grad->Finalize();
+   HypreParMatrix *G;
+   G = grad->ParallelAssemble();
+   delete vert_fespace;
+   delete grad;
+   return G;
+}
+
+
+
+// Discrete gradient matrix
+Array2D<HypreParMatrix *> GetNDInterpolationOp(ParFiniteElementSpace *fespace)
+{
+   int dim = fespace->GetMesh()->Dimension();
+   int sdim = fespace->GetMesh()->SpaceDimension();
+   // const FiniteElementCollection *fec = fespace->FEColl();
+   int p = 1;
+   if (fespace->GetNE() > 0)
+   {
+      p = fespace->GetOrder(0);
+   }
+   ParMesh *pmesh = fespace->GetParMesh();
+   FiniteElementCollection *vert_fec;
+   vert_fec = new H1_FECollection(p, dim);
+   
+   Array2D<HypreParMatrix *> Pi_blocks;
+   ParFiniteElementSpace *vert_fespace_d
+         = new ParFiniteElementSpace(pmesh, vert_fec, sdim, Ordering::byVDIM);
+   ParDiscreteLinearOperator *id_ND;
+   id_ND = new ParDiscreteLinearOperator(vert_fespace_d, fespace);
+   id_ND->AddDomainInterpolator(new IdentityInterpolator);
+   id_ND->Assemble();
+   id_ND->Finalize();
+   id_ND->GetParBlocks(Pi_blocks);
+   //
+   delete id_ND;
+   delete vert_fespace_d;
+   delete vert_fec;
+   return Pi_blocks;
+}

examples/waves/DPG_prim.cpp

@@ -0,0 +1,395 @@
+//                       MFEM Example 8 - Parallel Version
+//
+// Compile with: make ex8p
+//
+// Sample runs:  mpirun -np 4 ex8p -m ../data/square-disc.mesh
+//               mpirun -np 4 ex8p -m ../data/star.mesh
+//               mpirun -np 4 ex8p -m ../data/star-mixed.mesh
+//               mpirun -np 4 ex8p -m ../data/escher.mesh
+//               mpirun -np 4 ex8p -m ../data/fichera.mesh
+//               mpirun -np 4 ex8p -m ../data/fichera-mixed.mesh
+//               mpirun -np 4 ex8p -m ../data/square-disc-p2.vtk
+//               mpirun -np 4 ex8p -m ../data/square-disc-p3.mesh
+//               mpirun -np 4 ex8p -m ../data/star-surf.mesh -o 2
+//
+// Description:  This example code demonstrates the use of the Discontinuous
+//               Petrov-Galerkin (DPG) method in its primal 2x2 block form as a
+//               simple finite element discretization of the Laplace problem
+//               -Delta u = f with homogeneous Dirichlet boundary conditions. We
+//               use high-order continuous trial space, a high-order interfacial
+//               (trace) space, and a high-order discontinuous test space
+//               defining a local dual (H^{-1}) norm.
+//
+//               We use the primal form of DPG, see "A primal DPG method without
+//               a first-order reformulation", Demkowicz and Gopalakrishnan, CAM
+//               2013, DOI:10.1016/j.camwa.2013.06.029.
+//
+//               The example highlights the use of interfacial (trace) finite
+//               elements and spaces, trace face integrators and the definition
+//               of block operators and preconditioners. The use of the ADS
+//               preconditioner from hypre for interfacially-reduced H(div)
+//               problems is also illustrated.
+//
+//               We recommend viewing examples 1-5 before viewing this example.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+int main(int argc, char *argv[])
+{
+   // 1. Initialize MPI.
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+
+   // 2. Parse command-line options.
+   // const char *mesh_file = "../../data/star.mesh";
+   const char *mesh_file = "../../data/one-hex.mesh";
+   int order = 1;
+   // PETSC
+   // const char *petscrc_file = "petscrc_direct";
+   const char *petscrc_file = "petscrc_mult_options";
+   bool visualization = 1;
+   // weighted H1 norm weight
+   double k = 1.0; 
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree).");
+   args.AddOption(&k, "-k", "--weight", 
+                  "weight for L2 term of H1 norm");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+
+   // 3. Read the (serial) mesh from the given mesh file on all processors.  We
+   //    can handle triangular, quadrilateral, tetrahedral, hexahedral, surface
+   //    and volume meshes with the same code.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   int dim = mesh->Dimension();
+
+   // 4. Refine the serial mesh on all processors to increase the resolution. In
+   //    this example we do 'ref_levels' of uniform refinement. We choose
+   //    'ref_levels' to be the largest number that gives a final mesh with no
+   //    more than 10,000 elements.
+   {
+      // int ref_levels =
+         // (int)floor(log(10000./mesh->GetNE())/log(2.)/dim);
+      int ref_levels = 2;
+      for (int l = 0; l < ref_levels; l++)
+      {
+         mesh->UniformRefinement();
+      }
+   }
+
+   // 5. Define a parallel mesh by a partitioning of the serial mesh. Refine
+   //    this mesh further in parallel to increase the resolution. Once the
+   //    parallel mesh is defined, the serial mesh can be deleted.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+   {
+      int par_ref_levels = 0;
+      for (int l = 0; l < par_ref_levels; l++)
+      {
+         pmesh->UniformRefinement();
+      }
+   }
+   pmesh->ReorientTetMesh();
+
+   // 6. Define the trial, interfacial (trace) and test DPG spaces:
+   //    - The trial space, x0_space, contains the non-interfacial unknowns and
+   //      has the essential BC.
+   //    - The interfacial space, xhat_space, contains the interfacial unknowns
+   //      and does not have essential BC.
+   //    - The test space, test_space, is an enriched space where the enrichment
+   //      degree may depend on the spatial dimension of the domain, the type of
+   //      the mesh and the trial space order.
+   unsigned int trial_order = order;
+   unsigned int trace_order = order-1;
+   unsigned int test_order  = order; /* reduced order, full order is
+                                        (order + dim - 1) */
+   if (dim == 2 && (order%2 == 0 || (pmesh->MeshGenerator() & 2 && order > 1)))
+   {
+      test_order++;
+   }
+   if (test_order < trial_order)
+   {
+      if (myid == 0)
+      {
+         cerr << "Warning, test space not enriched enough to handle primal"
+              << " trial space\n";
+      }
+   }
+
+   FiniteElementCollection *x0_fec, *xhat_fec, *test_fec;
+
+   x0_fec   = new H1_FECollection(trial_order, dim);
+   xhat_fec = new RT_Trace_FECollection(trace_order, dim);
+   test_fec = new L2_FECollection(test_order, dim);
+   
+   ParFiniteElementSpace *x0_space, *xhat_space, *test_space;
+
+
+   x0_space   = new ParFiniteElementSpace(pmesh, x0_fec);
+
+   int maxref = 3;
+   std::vector<HypreParMatrix*>  Pv(maxref);
+
+   for (int i = 0; i < maxref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*x0_space);
+      pmesh->UniformRefinement();
+      // Update x0_space
+      x0_space->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      x0_space->GetTrueTransferOperator(cfespace, Tr);
+      HypreParMatrix * Paux;
+      Tr.Get(Paux);
+      Pv[i] = new HypreParMatrix(*Paux);
+   }
+
+
+
+   xhat_space = new ParFiniteElementSpace(pmesh, xhat_fec);
+   test_space = new ParFiniteElementSpace(pmesh, test_fec);
+
+   HYPRE_Int glob_true_s0     =   x0_space->GlobalTrueVSize();
+   HYPRE_Int glob_true_s1     = xhat_space->GlobalTrueVSize();
+   HYPRE_Int glob_true_s_test = test_space->GlobalTrueVSize();
+   if (myid == 0)
+   {
+      cout << "\nNumber of Unknowns:\n"
+           << " Trial space,     X0   : " << glob_true_s0
+           << " (order " << trial_order << ")\n"
+           << " Interface space, Xhat : " << glob_true_s1
+           << " (order " << trace_order << ")\n"
+           << " Test space,      Y    : " << glob_true_s_test
+           << " (order " << test_order << ")\n\n";
+   }
+
+   // 7. Set up the linear form F(.) which corresponds to the right-hand side of
+   //    the FEM linear system, which in this case is (f,phi_i) where f=1.0 and
+   //    phi_i are the basis functions in the test finite element fespace.
+   ConstantCoefficient one(1.0);
+   ParLinearForm * F = new ParLinearForm(test_space);
+   F->AddDomainIntegrator(new DomainLFIntegrator(one));
+   F->Assemble();
+
+   ParGridFunction * x0 = new ParGridFunction(x0_space);
+   *x0 = 0.;
+
+   // 8. Set up the mixed bilinear form for the primal trial unknowns, B0,
+   //    the mixed bilinear form for the interfacial unknowns, Bhat,
+   //    the inverse stiffness matrix on the discontinuous test space, Sinv,
+   //    and the stiffness matrix on the continuous trial space, S0.
+   Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+   ess_bdr = 1;
+   Array<int> ess_dof;
+   x0_space->GetEssentialVDofs(ess_bdr, ess_dof);
+
+   ConstantCoefficient w(k);
+   ParMixedBilinearForm *B0 = new ParMixedBilinearForm(x0_space,test_space);
+   B0->AddDomainIntegrator(new DiffusionIntegrator(one));
+   B0->AddDomainIntegrator(new MassIntegrator(w));
+   B0->Assemble();
+   B0->EliminateEssentialBCFromTrialDofs(ess_dof, *x0, *F);
+   B0->Finalize();
+
+   ParMixedBilinearForm *Bhat = new ParMixedBilinearForm(xhat_space,test_space);
+   Bhat->AddTraceFaceIntegrator(new TraceJumpIntegrator());
+   Bhat->Assemble();
+   Bhat->Finalize();
+
+   ParBilinearForm *Sinv = new ParBilinearForm(test_space);
+   SumIntegrator *Sum = new SumIntegrator;
+   Sum->AddIntegrator(new DiffusionIntegrator(one));
+   // ConstantCoefficient pos(1.0);
+   ConstantCoefficient pos(abs(k));
+   Sum->AddIntegrator(new MassIntegrator(pos));
+   Sinv->AddDomainIntegrator(new InverseIntegrator(Sum));
+   Sinv->Assemble();
+   Sinv->Finalize();
+
+   ParBilinearForm *S0 = new ParBilinearForm(x0_space);
+   S0->AddDomainIntegrator(new DiffusionIntegrator(one));
+   S0->AddDomainIntegrator(new MassIntegrator(pos));
+   S0->Assemble();
+   S0->EliminateEssentialBC(ess_bdr);
+   S0->Finalize();
+
+   HypreParMatrix * matB0   = B0->ParallelAssemble();    delete B0;
+   HypreParMatrix * matBhat = Bhat->ParallelAssemble();  delete Bhat;
+   HypreParMatrix * matSinv = Sinv->ParallelAssemble();  delete Sinv;
+   HypreParMatrix * matS0   = S0->ParallelAssemble();    delete S0;
+
+   // 9. Define the block structure of the problem, by creating the offset
+   //    variables. Also allocate two BlockVector objects to store the solution
+   //    and rhs.
+   enum {x0_var, xhat_var, NVAR};
+
+   int true_s0     = x0_space->TrueVSize();
+   int true_s1     = xhat_space->TrueVSize();
+   int true_s_test = test_space->TrueVSize();
+
+   Array<int> true_offsets(NVAR+1);
+   true_offsets[0] = 0;
+   true_offsets[1] = true_s0;
+   true_offsets[2] = true_s0+true_s1;
+
+   Array<int> true_offsets_test(2);
+   true_offsets_test[0] = 0;
+   true_offsets_test[1] = true_s_test;
+
+   BlockVector x(true_offsets), b(true_offsets);
+   x = 0.0;
+   b = 0.0;
+
+   // 10. Set up the 1x2 block Least Squares DPG operator, B = [B0 Bhat],
+   //     the normal equation operator, A = B^t Sinv B, and
+   //     the normal equation right-hand-size, b = B^t Sinv F.
+   BlockOperator B(true_offsets_test, true_offsets);
+   B.SetBlock(0, 0, matB0);
+   B.SetBlock(0, 1, matBhat);
+
+   RAPOperator A(B, *matSinv, B); //B^T Sinv B
+
+   HypreParVector *trueF = F->ParallelAssemble();
+   {
+      HypreParVector SinvF(test_space);
+      matSinv->Mult(*trueF, SinvF);
+      B.MultTranspose(SinvF, b);
+   }
+
+   // 11. Set up a block-diagonal preconditioner for the 2x2 normal equation
+   //
+   //        [ S0^{-1}     0     ]
+   //        [   0     Shat^{-1} ]      Shat = (Bhat^T Sinv Bhat)
+   //
+   //     corresponding to the primal (x0) and interfacial (xhat) unknowns.
+   //     Since the Shat operator is equivalent to an H(div) matrix reduced to
+   //     the interfacial skeleton, we approximate its inverse with one V-cycle
+   //     of the ADS preconditioner from the hypre library (in 2D we use AMS for
+   //     the rotated H(curl) problem).
+
+
+   // HypreBoomerAMG *S0inv = new HypreBoomerAMG(*matS0);
+   // S0inv->SetPrintLevel(0);
+
+   GMGSolver *S0inv = new GMGSolver(matS0, Pv,GMGSolver::CoarseSolver::PETSC);
+   S0inv->SetTheta(0.5);
+
+
+   HypreParMatrix *Shat = RAP(matSinv, matBhat);
+
+   HypreSolver *Shatinv;
+   if (dim == 2) { Shatinv = new HypreAMS(*Shat, xhat_space); }
+   else          { Shatinv = new HypreADS(*Shat, xhat_space); }
+
+
+   BlockDiagonalPreconditioner P(true_offsets);
+   P.SetDiagonalBlock(0, S0inv);
+   P.SetDiagonalBlock(1, Shatinv);
+
+   // 12. Solve the normal equation system using the PCG iterative solver.
+   //     Check the weighted norm of residual for the DPG least square problem.
+   //     Wrap the primal variable in a GridFunction for visualization purposes.
+   CGSolver pcg(MPI_COMM_WORLD);
+   pcg.SetOperator(A);
+   pcg.SetPreconditioner(P);
+   pcg.SetRelTol(1e-6);
+   pcg.SetMaxIter(100);
+   pcg.SetPrintLevel(1);
+   pcg.Mult(b, x);
+
+   {
+      HypreParVector LSres(test_space), tmp(test_space);
+      B.Mult(x, LSres);
+      LSres -= *trueF;
+      matSinv->Mult(LSres, tmp);
+      double res = sqrt(InnerProduct(LSres, tmp));
+      if (myid == 0)
+      {
+         cout << "\n|| B0*x0 + Bhat*xhat - F ||_{S^-1} = " << res << endl;
+      }
+   }
+
+   x0->Distribute(x.GetBlock(x0_var));
+
+   // 13. Save the refined mesh and the solution in parallel. This output can
+   //     be viewed later using GLVis: "glvis -np <np> -m mesh -g sol".
+   {
+      ostringstream mesh_name, sol_name;
+      mesh_name << "mesh." << setfill('0') << setw(6) << myid;
+      sol_name << "sol." << setfill('0') << setw(6) << myid;
+
+      ofstream mesh_ofs(mesh_name.str().c_str());
+      mesh_ofs.precision(8);
+      pmesh->Print(mesh_ofs);
+
+      ofstream sol_ofs(sol_name.str().c_str());
+      sol_ofs.precision(8);
+      x0->Save(sol_ofs);
+   }
+
+   // 14. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << *x0 << flush;
+   }
+
+   for (int i = 0 ; i < maxref; i++)
+   {
+      delete Pv[i];
+   }
+   // 15. Free the used memory.
+   delete trueF;
+   delete Shatinv;
+   delete S0inv;
+   delete Shat;
+   delete matB0;
+   delete matBhat;
+   delete matSinv;
+   delete matS0;
+   delete x0;
+   delete F;
+   delete test_space;
+   delete xhat_space;
+   delete x0_space;
+   delete test_fec;
+   delete xhat_fec;
+   delete x0_fec;
+   delete pmesh;
+
+   MFEMFinalizePetsc();
+   MPI_Finalize();
+
+   return 0;
+}

examples/waves/LS_Maxwellp.cpp

@@ -0,0 +1,562 @@
+//                                MFEM Example multigrid-grid Cycle
+//
+// Compile with: make mg_maxwellp
+//
+// Sample runs:  mg_maxwellp -m ../data/one-hex.mesh
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+// #include "ddoper.hpp"
+// #include "AMS_LS.hpp"
+
+using namespace std;
+using namespace mfem;
+
+#ifndef MFEM_USE_PETSC
+#error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+#endif
+
+// Define exact solution
+void E_exact(const Vector & x, Vector & E);
+void H_exact(const Vector & x, Vector & H);
+void scaledf_exact_E(const Vector & x, Vector & f_E);
+void scaledf_exact_H(const Vector & x, Vector & f_H);
+void f_exact_E(const Vector & x, Vector & f_E);
+void f_exact_H(const Vector & x, Vector & f_H);
+void get_maxwell_solution(const Vector & x, double E[], double curlE[], double curl2E[]);
+
+int dim;
+double omega;
+int isol = 1;
+
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv); // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid); // Determine process identifier
+   // 1. Parse command-line options.
+   // geometry file
+   // const char *mesh_file = "../data/star.mesh";
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   // visualization flag
+   bool visualization = 1;
+   // number of wavelengths
+   double k = 1.0;
+   // number of mg levels
+   int maxref = 1;
+   // number of initial ref
+   int initref = 1;
+   // PETSC
+   // const char *petscrc_file = "petscrc_direct";
+   const char *petscrc_file = "petscrc_mult_options";
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&maxref, "-ref", "--maxref",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&isol, "-isol", "--exact",
+                  "Exact solution flag - "
+                  " 1:sinusoidal, 2: point source, 3: plane wave");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   // Angular frequency
+   // omega = 2.0*k*M_PI;
+   omega = k;
+
+   // 2b. We initialize PETSc
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+
+   // 2. Read the mesh from the given mesh file.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // 4. Define a finite element space on the mesh.
+   FiniteElementCollection *fec = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+
+   std::vector<HypreParMatrix*>  P(maxref);
+
+   for (int i = 0; i < maxref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*fespace);
+      pmesh->UniformRefinement();
+      fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      fespace->GetTrueTransferOperator(cfespace, Tr);
+      HypreParMatrix * Paux;
+      Tr.Get(Paux);
+      P[i] = new HypreParMatrix(*Paux);
+   }
+
+
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+   ess_bdr = 1;
+   fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+
+   Array<int> block_offsets(3);
+   block_offsets[0] = 0;
+   block_offsets[1] = fespace->GetVSize();
+   block_offsets[2] = fespace->GetVSize();
+   block_offsets.PartialSum();
+
+   Array<int> block_trueOffsets(3);
+   block_trueOffsets[0] = 0;
+   block_trueOffsets[1] = fespace->TrueVSize();
+   block_trueOffsets[2] = fespace->TrueVSize();
+   block_trueOffsets.PartialSum();
+
+   //    _           _    _  _       _  _
+   //   |             |  |    |     |    |
+   //   |  A00   A01  |  | E  |     |F_E |
+   //   |             |  |    |  =  |    |
+   //   |  A10   A11  |  | H  |     |F_G |
+   //   |_           _|  |_  _|     |_  _|
+   //
+   // A00 = (curl E, curl F) + \omega^2 (E,F)
+   // A01 = - \omega *( (curl E, F) + (E,curl F)
+   // A10 = - \omega *( (curl H, G) + (H,curl G)
+   // A11 = (curl H, curl H) + \omega^2 (H,G)
+
+   BlockVector x(block_offsets), rhs(block_offsets);
+   BlockVector trueX(block_trueOffsets), trueRhs(block_trueOffsets);
+
+   x = 0.0;
+   rhs = 0.0;
+   trueX = 0.0;
+   trueRhs = 0.0;
+
+   VectorFunctionCoefficient Eex(sdim, E_exact);
+   ParGridFunction * E_gf = new ParGridFunction;
+   E_gf->MakeRef(fespace, x.GetBlock(0));
+   E_gf->ProjectCoefficient(Eex);
+
+   VectorFunctionCoefficient Hex(sdim, H_exact);
+   ParGridFunction * H_gf = new ParGridFunction;   
+   H_gf->MakeRef(fespace, x.GetBlock(1));
+   H_gf->ProjectCoefficient(Hex);
+
+   // // 6. Set up the linear form
+   VectorFunctionCoefficient sf_E(sdim,scaledf_exact_E);
+   VectorFunctionCoefficient sf_H(sdim,scaledf_exact_H);
+   VectorFunctionCoefficient f_E(sdim,f_exact_E);
+   VectorFunctionCoefficient f_H(sdim,f_exact_H);
+
+   ParLinearForm *b_E = new ParLinearForm;
+   b_E->Update(fespace, rhs.GetBlock(0), 0);
+   b_E->AddDomainIntegrator(new VectorFEDomainLFIntegrator(sf_H));
+   b_E->AddDomainIntegrator(new VectorFEDomainLFCurlIntegrator(f_E));
+   b_E->Assemble();
+
+
+   ParLinearForm *b_H = new ParLinearForm;
+   b_H->Update(fespace, rhs.GetBlock(1), 0);
+   b_H->AddDomainIntegrator(new VectorFEDomainLFIntegrator(sf_E));
+   b_H->AddDomainIntegrator(new VectorFEDomainLFCurlIntegrator(f_H));
+   b_H->Assemble();
+
+
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ConstantCoefficient one(1.0);
+   ConstantCoefficient sigma(pow(omega, 2));
+   ConstantCoefficient neg(-abs(omega));
+   ConstantCoefficient pos(abs(omega));
+   //
+   ParBilinearForm *a_EE = new ParBilinearForm(fespace);
+   a_EE->AddDomainIntegrator(new CurlCurlIntegrator(one)); 
+   a_EE->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   a_EE->Assemble();
+   a_EE->EliminateEssentialBC(ess_bdr,x.GetBlock(0), rhs.GetBlock(0));
+   a_EE->Finalize();
+   HypreParMatrix *A_EE = a_EE->ParallelAssemble();
+
+   ParMixedBilinearForm *a_EH = new ParMixedBilinearForm(fespace,fespace);
+   a_EH->AddDomainIntegrator(new MixedVectorCurlIntegrator(neg));
+   a_EH->AddDomainIntegrator(new MixedVectorWeakCurlIntegrator(neg)); 
+   a_EH->Assemble();
+   a_EH->EliminateTrialDofs(ess_bdr, x.GetBlock(0), rhs.GetBlock(1));
+   a_EH->Finalize();
+   HypreParMatrix *A_EH = a_EH->ParallelAssemble();
+
+   HypreParMatrix *A_HE = A_EH->Transpose();
+
+   ParBilinearForm *a_HH = new ParBilinearForm(fespace);
+   a_HH->AddDomainIntegrator(new CurlCurlIntegrator(one)); // one is the coeff
+   a_HH->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   a_HH->Assemble();
+   a_HH->Finalize();
+   HypreParMatrix *A_HH = a_HH->ParallelAssemble();
+
+   BlockOperator *LS_Maxwellop = new BlockOperator(block_trueOffsets);
+   LS_Maxwellop->SetBlock(0, 0, A_EE);
+   LS_Maxwellop->SetBlock(0, 1, A_HE);
+   LS_Maxwellop->SetBlock(1, 0, A_EH);
+   LS_Maxwellop->SetBlock(1, 1, A_HH);
+   
+
+   HypreSolver *amsE = new HypreAMS(*A_EE,fespace);
+   HypreSolver *amsH = new HypreAMS(*A_HH,fespace);
+
+   GMGSolver * gmgE = new GMGSolver(A_EE, P, GMGSolver::CoarseSolver::PETSC);
+   gmgE->SetTheta(0.5);
+   gmgE->SetSmootherType(HypreSmoother::Jacobi);
+
+   GMGSolver * gmgH = new GMGSolver(A_HH, P,GMGSolver::CoarseSolver::PETSC);
+   gmgH->SetTheta(0.5);
+   gmgH->SetSmootherType(HypreSmoother::Jacobi);
+
+
+   PetscLinearSolver * invE = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   invE->SetOperator(PetscParMatrix(A_EE, Operator::PETSC_MATAIJ));
+
+   PetscLinearSolver * invH = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   invH->SetOperator(PetscParMatrix(A_HH, Operator::PETSC_MATAIJ));
+
+   BlockDiagonalPreconditioner *LSprec1 = new BlockDiagonalPreconditioner(block_trueOffsets);
+   BlockDiagonalPreconditioner *LSprec2 = new BlockDiagonalPreconditioner(block_trueOffsets);
+   BlockDiagonalPreconditioner *LSprec3 = new BlockDiagonalPreconditioner(block_trueOffsets);
+   LSprec1->SetDiagonalBlock(0, amsE);
+   LSprec1->SetDiagonalBlock(1, amsH);
+
+   LSprec2->SetDiagonalBlock(0, gmgE);
+   LSprec2->SetDiagonalBlock(1, gmgH);
+
+   LSprec3->SetDiagonalBlock(0, invE);
+   LSprec3->SetDiagonalBlock(1, invH);
+
+   fespace->GetRestrictionMatrix()->Mult(x.GetBlock(0), trueX.GetBlock(0));
+   fespace->GetProlongationMatrix()->MultTranspose(rhs.GetBlock(0),trueRhs.GetBlock(0));
+
+   fespace->GetRestrictionMatrix()->Mult(x.GetBlock(1), trueX.GetBlock(1));
+   fespace->GetProlongationMatrix()->MultTranspose(rhs.GetBlock(1),trueRhs.GetBlock(1));
+
+   if (myid == 0)
+   {
+      cout << "Size of fine grid system: "
+           << 2.0 * A_EE->GetGlobalNumRows() << " x " << 2.0* A_EE->GetGlobalNumCols() << endl;
+   }
+   int maxit(5000);
+   double rtol(1.e-6);
+   double atol(0.0);
+   trueX = 0.0;
+
+   CGSolver pcg(MPI_COMM_WORLD);
+   pcg.SetAbsTol(atol);
+   pcg.SetRelTol(rtol);
+   pcg.SetMaxIter(maxit);
+   pcg.SetOperator(*LS_Maxwellop);
+   pcg.SetPreconditioner(*LSprec1);
+   pcg.SetPrintLevel(1);
+   pcg.Mult(trueRhs, trueX);
+
+   if (myid == 0)
+   {
+      cout << "PCG with AMS finished" << endl;
+   }
+   trueX = 0.0;
+
+   pcg.SetPreconditioner(*LSprec2);
+   pcg.Mult(trueRhs, trueX);
+
+   if (myid == 0)
+   {
+      cout << "PCG with GMG finished" << endl;
+   }
+
+   trueX = 0.0;
+
+   pcg.SetPreconditioner(*LSprec3);
+   pcg.Mult(trueRhs, trueX);
+
+   if (myid == 0)
+   {
+      cout << "PCG with Petsc exact inverse finished" << endl;
+   }
+
+
+   *E_gf = 0.0;
+   *H_gf = 0.0;
+
+   E_gf->Distribute(&(trueX.GetBlock(0)));
+   H_gf->Distribute(&(trueX.GetBlock(1)));
+
+   int order_quad = max(2, 2*order+1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i=0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double Error_E = E_gf->ComputeL2Error(Eex, irs);
+   double norm_E = ComputeGlobalLpNorm(2, Eex, *pmesh, irs);
+
+   double Error_H = H_gf->ComputeL2Error(Hex, irs);
+   double norm_H = ComputeGlobalLpNorm(2, Hex , *pmesh, irs);
+
+   if (myid == 0)
+   {
+      // cout << "|| E_h - E || / || E || = " << Error_E / norm_E << "\n";
+      // cout << "|| H_h - H || / || H || = " << Error_H / norm_H << "\n";
+      cout << "|| E_h - E || = " << Error_E  << "\n";
+      cout << "|| H_h - H || = " << Error_H  << "\n";
+
+      cout << "Total Relative error = " <<  Error_E / norm_E + Error_H / norm_H  << "\n";
+
+      // cout << "|| E || = " << norm_E  << "\n";
+      // cout << "|| H || = " << norm_H  << "\n";
+   }
+
+   if (visualization)
+   {
+      // ParGridFunction * Eex_gf = new ParGridFunction;   
+      // Eex_gf->MakeRef(fespace, x.GetBlock(0));
+      // Eex_gf->ProjectCoefficient(Eex);
+      // ParGridFunction * Hex_gf = new ParGridFunction;   
+      // Hex_gf->MakeRef(fespace, x.GetBlock(1));
+      // Hex_gf->ProjectCoefficient(Hex);
+
+      // 8. Connect to GLVis.
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream E_sock(vishost, visport);
+      E_sock << "parallel " << num_procs << " " << myid << "\n";
+      E_sock.precision(8);
+      E_sock << "solution\n" << *pmesh << *E_gf << "window_title 'Electric field'" << endl;
+      // MPI_Barrier(pmesh->GetComm());
+      // socketstream Eex_sock(vishost, visport);
+      // Eex_sock << "parallel " << num_procs << " " << myid << "\n";
+      // Eex_sock.precision(8);
+      // Eex_sock << "solution\n" << *pmesh << *Eex_gf << "window_title 'Exact Electric Field'" << endl;
+      MPI_Barrier(pmesh->GetComm());
+      socketstream H_sock(vishost, visport);
+      H_sock << "parallel " << num_procs << " " << myid << "\n";
+      H_sock.precision(8);
+      H_sock << "solution\n" << *pmesh << *H_gf << "window_title 'Magnetic field'" << endl;
+      // MPI_Barrier(pmesh->GetComm());
+      // socketstream Hex_sock(vishost, visport);
+      // Hex_sock << "parallel " << num_procs << " " << myid << "\n";
+      // Hex_sock.precision(8);
+      // Hex_sock << "solution\n" << *pmesh << *Hex_gf << "window_title 'Exact Magnetic field'" << endl;
+   }
+
+
+   delete a_EE;
+   delete a_EH;
+   delete a_HH;
+   delete b_E;
+   delete b_H;
+   delete fec;
+   delete fespace;
+   delete pmesh;
+   MFEMFinalizePetsc();
+   MPI_Finalize();
+   return 0;
+}
+
+
+//define exact solution
+void E_exact(const Vector &x, Vector &E)
+{
+   double curlE[3], curl2E[3];
+   get_maxwell_solution(x, E, curlE, curl2E);
+}
+
+void H_exact(const Vector &x, Vector &H)
+{
+   double E[3], curlE[3], curl2E[3];
+   get_maxwell_solution(x, E, curlE, curl2E);
+   for (int i = 0; i<3; i++) {H(i) = curlE[i]/omega;}
+}
+
+//calculate RHS from exact solution
+void f_exact_E(const Vector &x, Vector &f)
+{
+   double E[3], curlE[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   // curl E - omega H = 0 
+   f(0) = curlE[0] - omega * (curlE[0]/omega); // = 0
+   f(1) = curlE[1] - omega * (curlE[1]/omega); // = 0
+   f(2) = curlE[2] - omega * (curlE[2]/omega); // = 0
+}
+
+void f_exact_H(const Vector &x, Vector &f)
+{
+   if (dim != 3)
+   {
+      cout << "2D not set up yet: " << endl;
+      exit(0);
+   }
+   double E[3], curlE[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   // curl H - omega E = f 
+   // = curl (curl E / omega) - omega E 
+   f(0) = curl2E[0]/omega - omega * E[0];
+   f(1) = curl2E[1]/omega - omega * E[1];
+   f(2) = curl2E[2]/omega - omega * E[2];
+}
+
+void scaledf_exact_E(const Vector &x, Vector &f)
+{
+   double E[3], curlE[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   //  - omega *( curl E - omega H) = 0 
+   f(0) =-omega * (curlE[0] - omega * (curlE[0]/omega)); // = 0
+   f(1) =-omega * (curlE[1] - omega * (curlE[1]/omega)); // = 0
+   f(2) =-omega * (curlE[2] - omega * (curlE[2]/omega)); // = 0
+}
+
+void scaledf_exact_H(const Vector &x, Vector &f)
+{
+   double E[3], curlE[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   // curl H - omega E = f 
+   // = - omega *( curl (curl E / omega) - omega E) 
+   
+   f(0) = -omega * (curl2E[0]/omega - omega * E[0]);
+   f(1) = -omega * (curl2E[1]/omega - omega * E[1]);
+   f(2) = -omega * (curl2E[2]/omega - omega * E[2]);
+}
+
+void get_maxwell_solution(const Vector & X, double E[], double curlE[], double curl2E[])
+{
+   const double x = X[0];
+   const double y = X[1];
+   const double z = X[2];
+
+   if (isol == 0) // polynomial
+   {
+      // Polynomial vanishing on the boundary
+      E[0] = y * z * (1.0 - y) * (1.0 - z);
+      E[1] = (1.0 - x) * x * y * (1.0 - z) * z;
+      E[2] = (1.0 - x) * x * (1.0 - y) * y;
+      //
+
+      curlE[0] = -(-1.0 + x) * x * (1.0 + y * (-3.0 + 2.0 * z));
+      curlE[1] = -2.0 * (-1.0 + y) * y * (x - z);
+      curlE[2] = (1.0 + (-3.0 + 2.0 * x) * y) * (-1.0 + z) * z; 
+
+      curl2E[0] = -2.0 * (-1.0 + y) * y + (-3.0 + 2.0 * x) * (-1.0 + z) * z;
+      curl2E[1] = -2.0 * y * (-x + x*x + (-1.0 + z) * z);
+      curl2E[2] = -2.0 * (-1.0 + y) * y + (-1.0 + x) * x * (-3.0 + 2.0 * z);
+
+   }
+
+   else if (isol == 1) // sinusoidal
+   {
+      E[0] = sin(omega * y);
+      E[1] = sin(omega * z);
+      E[2] = sin(omega * x);
+
+      curlE[0] = -omega * cos(omega * z);
+      curlE[1] = -omega * cos(omega * x);;
+      curlE[2] = -omega * cos(omega * y);; 
+
+      curl2E[0] = omega * omega * E[0];
+      curl2E[1] = omega * omega * E[1];
+      curl2E[2] = omega * omega * E[2];
+
+   }
+   else if (isol == 2) //simple polynomial
+   {
+      E[0] = y;
+      E[1] = z;
+      E[2] = x;
+
+      curlE[0] = -1.0;
+      curlE[1] = -1.0;
+      curlE[2] = -1.0; 
+
+      curl2E[0] =0.0;
+      curl2E[1] =0.0;
+      curl2E[2] =0.0;
+   }
+      else if (isol == 4) //constant
+   {
+      E[0] = 1.0;
+      E[1] = 1.0;
+      E[2] = 1.0;
+
+      curlE[0] = 0.0;
+      curlE[1] = 0.0;
+      curlE[2] = 0.0; 
+
+      curl2E[0] =0.0;
+      curl2E[1] =0.0;
+      curl2E[2] =0.0;
+   }
+   else if (isol == 3) // plane wave
+   {
+      double coeff = omega / sqrt(3.0);
+      E[0] = cos(coeff * (x + y + z));
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+
+      curlE[0] = 0.0;
+      curlE[1] = -coeff * sin(coeff * (x+y+z));
+      curlE[2] = coeff * sin(coeff * (x+y+z)); 
+
+      curl2E[0] = 2.0 * coeff * coeff * E[0];
+      curl2E[1] = -coeff * coeff * E[0];
+      curl2E[2] = -coeff * coeff * E[0];
+   }
+
+}
+

examples/waves/LS_blkAMS.cpp

@@ -0,0 +1,781 @@
+//                                MFEM Example multigrid-grid Cycle
+//
+// Compile with: make mg_maxwellp
+//
+// Sample runs:  mg_maxwellp -m ../data/one-hex.mesh
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include "AMS_LS.hpp"
+
+using namespace std;
+using namespace mfem;
+
+class Block_AMSSolver : public Solver {
+private:
+   /// The linear system matrix
+   Array2D<HypreParMatrix* > A_array;
+   Array2D<HypreParMatrix* > Pi;
+   HypreParMatrix *Grad, *Pix, *Piy, *Piz;
+   HypreParMatrix *l1A00, *l1A11;
+   BlockOperator* GtAG;
+   BlockOperator* PxtAPx;
+   BlockOperator* PytAPy;
+   BlockOperator* PztAPz;
+   Array<int> offsets;
+   Array<int> offsetsG;
+   Array<int> offsetsPi;
+   BlockOperator * D;
+   BlockOperator * A;
+   BlockOperator * G;
+   BlockOperator * Px;
+   BlockOperator * Py;
+   BlockOperator * Pz;
+   HypreBoomerAMG *G00_inv, *Px00_inv, *Py00_inv, *Pz00_inv;
+   HypreBoomerAMG *G11_inv, *Px11_inv, *Py11_inv, *Pz11_inv;;
+   BlockDiagonalPreconditioner * blkAMG_G;
+   BlockDiagonalPreconditioner * blkAMG_Px;
+   BlockDiagonalPreconditioner * blkAMG_Py;
+   BlockDiagonalPreconditioner * blkAMG_Pz;
+   double theta = 1.0;
+   string cycle_type = "023414320"; // 0-Smoother, 1-Grad, 2,3,4-Pix,Piy,Piz
+   HypreSmoother * Dh;
+   HypreParMatrix* Ah;
+   int NumberOfCycles=1;
+public:
+   Block_AMSSolver(Array<int> offsets_, ParFiniteElementSpace *fespace)
+      :  offsets(offsets_), offsetsG(3), offsetsPi(3)
+   {
+      Grad = new HypreParMatrix(*GetDiscreteGradientOp(fespace));
+      Pi = GetNDInterpolationOp(fespace);
+      Pix = new HypreParMatrix(*Pi(0,0));
+      Piy = new HypreParMatrix(*Pi(0,1));
+      Piz = new HypreParMatrix(*Pi(0,2));
+      offsetsG[0]=0;
+      offsetsG[1]=Grad->Width();
+      offsetsG[2]=Grad->Width();
+      offsetsG.PartialSum();
+      offsetsPi[0]=0;
+      offsetsPi[1]=Pix->Width();
+      offsetsPi[2]=Pix->Width();
+      offsetsPi.PartialSum();
+      G      = new BlockOperator(offsets, offsetsG);
+      Px     = new BlockOperator(offsets, offsetsPi);
+      Py     = new BlockOperator(offsets, offsetsPi);
+      Pz     = new BlockOperator(offsets, offsetsPi);
+      GtAG   = new BlockOperator(offsetsG);
+      PxtAPx = new BlockOperator(offsetsPi);
+      PytAPy = new BlockOperator(offsetsPi);
+      PztAPz = new BlockOperator(offsetsPi);
+      A = new BlockOperator(offsets);
+      this->height = 2*Grad->Height();
+      this->width = 2*Grad->Height();
+      blkAMG_G = new BlockDiagonalPreconditioner(offsetsG);
+      blkAMG_Px = new BlockDiagonalPreconditioner(offsetsPi);
+      blkAMG_Py = new BlockDiagonalPreconditioner(offsetsPi);
+      blkAMG_Pz = new BlockDiagonalPreconditioner(offsetsPi);
+   }
+   virtual void SetOperator(const Operator & ) {}
+   virtual void SetOperator(Array2D<HypreParMatrix*> Op) {
+      A_array = Op;
+      l1A00 = new HypreParMatrix(*A_array(0,0));
+      l1A11 = new HypreParMatrix(*A_array(1,1));
+      // DiagAddL1norm();
+      HypreSmoother * D_00 = new HypreSmoother;  
+      D_00->SetType(HypreSmoother::l1GS);
+      // D_00->SetType(HypreSmoother::Jacobi);
+      D_00->SetOperator(*l1A00);
+      HypreSmoother * D_11 = new HypreSmoother;  
+      D_11->SetType(HypreSmoother::l1GS);
+      // D_11->SetType(HypreSmoother::Jacobi);
+      D_11->SetOperator(*l1A11);
+      
+      D = new BlockOperator(offsets);
+      D->SetDiagonalBlock(0, D_00);
+      D->SetDiagonalBlock(1, D_11);
+      SetOperators();
+   }
+
+   virtual void SetOperators() {
+      int i,j;
+      for (i=0; i<2 ; i++)
+      {
+         A->SetBlock(i,i,A_array(i,i));
+         G->SetBlock(i,i,Grad);
+         Px->SetBlock(i,i,Pix);
+         Py->SetBlock(i,i,Piy);
+         Pz->SetBlock(i,i,Piz);
+         for (j=0; j<2 ; j++)
+         {
+            A->SetBlock(i,j,A_array(i,j));
+            GtAG->SetBlock(i,j,RAP(A_array(i,j),Grad));
+            PxtAPx->SetBlock(i,j,RAP(A_array(i,j),Pix));
+            PytAPy->SetBlock(i,j,RAP(A_array(i,j),Piy));
+            PztAPz->SetBlock(i,j,RAP(A_array(i,j),Piz));
+         }
+      }
+
+      for (i=0; i<2 ; i++)
+      { 
+         HypreBoomerAMG * G_AMG = new HypreBoomerAMG(*RAP(A_array(i,i),Grad));
+         HypreBoomerAMG * Px_AMG = new HypreBoomerAMG(*RAP(A_array(i,i),Pix));
+         HypreBoomerAMG * Py_AMG = new HypreBoomerAMG(*RAP(A_array(i,i),Piy));
+         HypreBoomerAMG * Pz_AMG = new HypreBoomerAMG(*RAP(A_array(i,i),Piz));
+         G_AMG->SetPrintLevel(0);
+         G_AMG->SetErrorMode(HypreSolver::ErrorMode::IGNORE_HYPRE_ERRORS);
+         Px_AMG->SetPrintLevel(0);
+         Px_AMG->SetErrorMode(HypreSolver::ErrorMode::IGNORE_HYPRE_ERRORS);
+         Py_AMG->SetPrintLevel(0);
+         Py_AMG->SetErrorMode(HypreSolver::ErrorMode::IGNORE_HYPRE_ERRORS);
+         Pz_AMG->SetPrintLevel(0);
+         Pz_AMG->SetErrorMode(HypreSolver::ErrorMode::IGNORE_HYPRE_ERRORS);
+         blkAMG_G->SetDiagonalBlock(i,G_AMG);
+         blkAMG_Px->SetDiagonalBlock(i,Px_AMG);
+         blkAMG_Py->SetDiagonalBlock(i,Py_AMG);
+         blkAMG_Pz->SetDiagonalBlock(i,Pz_AMG);
+      }   
+   }
+   virtual void SetTheta(const double a) {theta = a;}
+   virtual void SetCycleType(const string c_type) {cycle_type = c_type;}
+   virtual void SetNumberofCycles(const int k) {NumberOfCycles = k;}
+
+
+   virtual void DiagAddL1norm()
+   {
+      
+      int n=A_array(1,1)->Height();
+      Vector l1norm0(n);
+      Vector l1norm1(n);
+
+      Getrowl1norm(A_array(0,1), l1norm0);
+      Getrowl1norm(A_array(1,0), l1norm1);
+
+      hypre_ParCSRMatrix * A_00 = (hypre_ParCSRMatrix *)const_cast<HypreParMatrix&>(*l1A00);
+      // Add the L1 norms on the diagonal
+      for (int j = 0; j < n; j++)
+      {
+         A_00->diag->data[A_00->diag->i[j]] += l1norm0(j);
+      }
+
+      hypre_ParCSRMatrix * A_11 = (hypre_ParCSRMatrix *)const_cast<HypreParMatrix&>(*l1A11);
+      // Add the L1 norms on the diagonal
+      for (int j = 0; j < n; j++)
+      {
+         A_11->diag->data[A_11->diag->i[j]] += l1norm1(j);
+      }
+   }
+
+   virtual void Getrowl1norm(HypreParMatrix *A , Vector &l1norm)
+   {
+      // First cast as hypre_ParCSRMatrix
+      hypre_ParCSRMatrix * Ah = (hypre_ParCSRMatrix *)const_cast<HypreParMatrix&>(*A);
+      HYPRE_Int num_rows = hypre_ParCSRMatrixNumRows(Ah);
+      hypre_CSRMatrix *A_diag = hypre_ParCSRMatrixDiag(Ah);
+      HYPRE_Int *A_diag_I = hypre_CSRMatrixI(A_diag);
+      HYPRE_Int *A_diag_J = hypre_CSRMatrixJ(A_diag);
+      HYPRE_Real *A_diag_data = hypre_CSRMatrixData(A_diag);
+
+      hypre_CSRMatrix *A_offd = hypre_ParCSRMatrixOffd(Ah);
+      HYPRE_Int *A_offd_I = hypre_CSRMatrixI(A_offd);
+      HYPRE_Int *A_offd_J = hypre_CSRMatrixJ(A_offd);
+      HYPRE_Real *A_offd_data = hypre_CSRMatrixData(A_offd);
+      HYPRE_Int num_cols_offd = hypre_CSRMatrixNumCols(A_offd);
+
+      //Initialize vector;
+
+      l1norm = 0.0;
+
+      for (int i = 0; i < num_rows; i++)
+      {
+         /* Add the l1 norm of the diag part of the ith row */
+         for (int j = A_diag_I[i]; j < A_diag_I[i+1]; j++)
+            l1norm(i) += fabs(A_diag_data[j]);
+         /* Add the l1 norm of the offd part of the ith row */
+         if (num_cols_offd)
+         {
+            for (int j = A_offd_I[i]; j < A_offd_I[i+1]; j++)
+               l1norm(i) += fabs(A_offd_data[j]);
+         }
+      }
+
+   }
+
+   virtual void Mult(const Vector &r, Vector &z) const
+   {
+      int n = r.Size();
+      int m = A->Height();
+      int Numit = 0;
+      // int k = G->Width();
+      if (n != m ) {cout << "Size inconsistency" << endl;}
+
+      Vector res(n), raux(n),zaux(n);
+      //initialization
+      res = r; z = 0.0;
+      //
+      Array<BlockOperator *> Tr_v(4);
+      Array<BlockOperator *> PtAP_v(4);
+      Array<BlockDiagonalPreconditioner *> blkAMG_v(4);
+      Tr_v[0]     = G;        Tr_v[1]     = Px;        Tr_v[2]     = Py;        Tr_v[3]     = Pz;
+      PtAP_v[0]   = GtAG;     PtAP_v[1]   = PxtAPx;    PtAP_v[2]   = PytAPy;    PtAP_v[3]   = PztAPz;
+      blkAMG_v[0] = blkAMG_G; blkAMG_v[1] = blkAMG_Px; blkAMG_v[2] = blkAMG_Py; blkAMG_v[3] = blkAMG_Pz;  
+      //
+      int len = cycle_type.length();
+      Array<int> ii(len);
+      for (int i=0; i<len; i++){ii[i]=cycle_type[i]-'0';}
+      //
+      for (int ic = 0; ic<NumberOfCycles; ic++)
+      {
+         for (int j = 0; j<len ; j++)
+         {
+            int i = ii[j];
+            if (i ==0)
+            {
+               D->Mult(res,zaux); zaux *= theta;
+            }
+            else
+            {
+               GetCorrection(Tr_v[i-1], PtAP_v[i-1], blkAMG_v[i-1], res, zaux);
+            }
+            z +=zaux; 
+            A->Mult(zaux,raux); res -=raux;
+         }
+         // Numit++;
+         // // double beta = Norm(res);
+         // double beta = sqrt(InnerProduct(MPI_COMM_WORLD, res, res));
+         // if(beta < 1e-6)
+         // {
+         //    int myid;
+         //    MPI_Comm_rank(MPI_COMM_WORLD, &myid); // Determine process identifier
+         //    if (myid == 0){
+         //       mfem::out << "Convergend in " << Numit << " iterations. " << 
+         //          "||r||_L2 = " << beta << "\n";
+         //    }      
+         // break;
+         // } 
+      }
+   }
+
+   void GetCorrection(BlockOperator* Tr, BlockOperator* op, BlockDiagonalPreconditioner *prec, Vector &r, Vector &z) const
+   {
+      int k = Tr->Width();
+      Vector raux(k), zaux(k);
+      // Map trough the Transpose of the Transfer operator
+      Tr->MultTranspose(r,raux);
+      zaux = 0.0;
+
+      int maxit(3000);
+      double rtol(0.0);
+      double atol(1e-8);
+      
+      // CGSolver cg(MPI_COMM_WORLD);
+      // cg.SetAbsTol(atol);
+      // cg.SetRelTol(rtol);
+      // cg.SetMaxIter(maxit);
+      // cg.SetOperator(*op);
+      // cg.SetPreconditioner(*prec);
+      // cg.SetPrintLevel(0);
+      // cg.Mult(raux, zaux);
+         prec->Mult(raux,zaux);
+
+      // Map back to the original space through the Tranfer operator
+      Tr->Mult(zaux, z);
+   }
+   virtual ~Block_AMSSolver(){}
+};
+// Define exact solution
+void E_exact(const Vector & x, Vector & E);
+void H_exact(const Vector & x, Vector & H);
+void scaledf_exact_E(const Vector & x, Vector & f_E);
+void scaledf_exact_H(const Vector & x, Vector & f_H);
+void f_exact_E(const Vector & x, Vector & f_E);
+void f_exact_H(const Vector & x, Vector & f_H);
+void get_maxwell_solution(const Vector & x, double E[], double curlE[], double curl2E[]);
+
+int dim;
+double omega;
+int isol = 1;
+
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv); // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid); // Determine process identifier
+   // 1. Parse command-line options.
+   // geometry file
+   // const char *mesh_file = "../data/star.mesh";
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   // visualization flag
+   bool visualization = 1;
+   // number of wavelengths
+   double k = 1.0;
+   // number of mg levels
+   int maxref = 1;
+   // number of initial ref
+   int initref = 1;
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&maxref, "-ref", "--maxref",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&isol, "-isol", "--exact",
+                  "Exact solution flag - "
+                  " 1:sinusoidal, 2: point source, 3: plane wave");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   // Angular frequency
+   omega = 2.0*k*M_PI;
+   // omega = k;
+
+   // 2. Read the mesh from the given mesh file.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // 4. Define a finite element space on the mesh.
+   FiniteElementCollection *fec = new ND_FECollection(order, dim);
+   // ParFiniteElementSpace *fespace = new ParFiniteElementSpace(mesh, fec);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+   ess_bdr = 1;
+   fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+
+   Array<int> block_offsets(3);
+   block_offsets[0] = 0;
+   block_offsets[1] = fespace->GetVSize();
+   block_offsets[2] = fespace->GetVSize();
+   block_offsets.PartialSum();
+
+   Array<int> block_trueOffsets(3);
+   block_trueOffsets[0] = 0;
+   block_trueOffsets[1] = fespace->TrueVSize();
+   block_trueOffsets[2] = fespace->TrueVSize();
+   block_trueOffsets.PartialSum();
+
+   //    _           _    _  _       _  _
+   //   |             |  |    |     |    |
+   //   |  A00   A01  |  | E  |     |F_E |
+   //   |             |  |    |  =  |    |
+   //   |  A10   A11  |  | H  |     |F_G |
+   //   |_           _|  |_  _|     |_  _|
+   //
+   // A00 = (curl E, curl F) + \omega^2 (E,F)
+   // A01 = - \omega *( (curl E, F) + (E,curl F)
+   // A10 = - \omega *( (curl H, G) + (H,curl G)
+   // A11 = (curl H, curl H) + \omega^2 (H,G)
+
+   BlockVector x(block_offsets), rhs(block_offsets);
+   BlockVector trueX(block_trueOffsets), trueRhs(block_trueOffsets);
+
+   x = 0.0;
+   rhs = 0.0;
+   trueX = 0.0;
+   trueRhs = 0.0;
+
+   VectorFunctionCoefficient Eex(sdim, E_exact);
+   ParGridFunction * E_gf = new ParGridFunction;
+   E_gf->MakeRef(fespace, x.GetBlock(0));
+   E_gf->ProjectCoefficient(Eex);
+
+   VectorFunctionCoefficient Hex(sdim, H_exact);
+   ParGridFunction * H_gf = new ParGridFunction;   
+   H_gf->MakeRef(fespace, x.GetBlock(1));
+   H_gf->ProjectCoefficient(Hex);
+
+   // // 6. Set up the linear form
+   VectorFunctionCoefficient sf_E(sdim,scaledf_exact_E);
+   VectorFunctionCoefficient sf_H(sdim,scaledf_exact_H);
+   VectorFunctionCoefficient f_E(sdim,f_exact_E);
+   VectorFunctionCoefficient f_H(sdim,f_exact_H);
+
+   ParLinearForm *b_E = new ParLinearForm;
+   b_E->Update(fespace, rhs.GetBlock(0), 0);
+   b_E->AddDomainIntegrator(new VectorFEDomainLFIntegrator(sf_H));
+   b_E->AddDomainIntegrator(new VectorFEDomainLFCurlIntegrator(f_E));
+   b_E->Assemble();
+
+
+   ParLinearForm *b_H = new ParLinearForm;
+   b_H->Update(fespace, rhs.GetBlock(1), 0);
+   b_H->AddDomainIntegrator(new VectorFEDomainLFIntegrator(sf_E));
+   b_H->AddDomainIntegrator(new VectorFEDomainLFCurlIntegrator(f_H));
+   b_H->Assemble();
+
+
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ConstantCoefficient one(1.0);
+   ConstantCoefficient sigma(pow(omega, 2));
+   ConstantCoefficient neg(-abs(omega));
+   ConstantCoefficient pos(abs(omega));
+   //
+   ParBilinearForm *a_EE = new ParBilinearForm(fespace);
+   a_EE->AddDomainIntegrator(new CurlCurlIntegrator(one)); 
+   a_EE->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   a_EE->Assemble();
+   a_EE->EliminateEssentialBC(ess_bdr,x.GetBlock(0), rhs.GetBlock(0));
+   a_EE->Finalize();
+   HypreParMatrix *A_EE = a_EE->ParallelAssemble();
+
+   ParMixedBilinearForm *a_HE = new ParMixedBilinearForm(fespace,fespace);
+   a_HE->AddDomainIntegrator(new MixedVectorCurlIntegrator(neg));
+   a_HE->AddDomainIntegrator(new MixedVectorWeakCurlIntegrator(neg)); 
+   a_HE->Assemble();
+   a_HE->EliminateTrialDofs(ess_bdr, x.GetBlock(0), rhs.GetBlock(1));
+   a_HE->Finalize();
+
+   HypreParMatrix *A_HE = a_HE->ParallelAssemble();
+
+   HypreParMatrix *A_EH = A_HE->Transpose();
+
+   ParBilinearForm *a_HH = new ParBilinearForm(fespace);
+   a_HH->AddDomainIntegrator(new CurlCurlIntegrator(one)); // one is the coeff
+   a_HH->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   a_HH->Assemble();
+   a_HH->Finalize();
+   HypreParMatrix *A_HH = a_HH->ParallelAssemble();
+
+   BlockOperator *LS_Maxwellop = new BlockOperator(block_trueOffsets);
+   LS_Maxwellop->SetBlock(0, 0, A_EE);
+   LS_Maxwellop->SetBlock(0, 1, A_EH);
+   LS_Maxwellop->SetBlock(1, 0, A_HE);
+   LS_Maxwellop->SetBlock(1, 1, A_HH);
+   
+
+   fespace->GetRestrictionMatrix()->Mult(x.GetBlock(0), trueX.GetBlock(0));
+   fespace->GetProlongationMatrix()->MultTranspose(rhs.GetBlock(0),trueRhs.GetBlock(0));
+
+   fespace->GetRestrictionMatrix()->Mult(x.GetBlock(1), trueX.GetBlock(1));
+   fespace->GetProlongationMatrix()->MultTranspose(rhs.GetBlock(1),trueRhs.GetBlock(1));
+
+   if (myid == 0)
+   {
+      cout << "Size of fine grid system: "
+           << 2.0 * A_EE->GetGlobalNumRows() << " x " << 2.0* A_EE->GetGlobalNumCols() << endl;
+   }
+
+   // Set up the preconditioner
+   Array2D<HypreParMatrix*> blockA(2,2);
+   blockA(0,0) = A_EE;
+   blockA(0,1) = A_EH;
+   blockA(1,0) = A_HE;
+   blockA(1,1) = A_HH;
+
+   Block_AMSSolver * blkAMS;
+   blkAMS = new Block_AMSSolver(block_trueOffsets, fespace);
+   blkAMS->SetOperator(blockA);
+   blkAMS->SetTheta(1.0);
+   //0-Smoother, 1-Grad, 2,3,4-Pix,Piy,Piz
+   blkAMS->SetCycleType("023414320");
+   // blkAMS->SetCycleType("000000000023414320000000000");
+   blkAMS->SetNumberofCycles(1);
+   // blkAMS->SetCycleType("012343210");
+
+
+   int maxit(500);
+   double rtol(1.e-6);
+   double atol(0.0);
+   trueX = 0.0;
+
+   CGSolver pcg(MPI_COMM_WORLD);
+   pcg.SetAbsTol(atol);
+   pcg.SetRelTol(rtol);
+   pcg.SetMaxIter(maxit);
+   pcg.SetPreconditioner(*blkAMS);
+   pcg.SetOperator(*LS_Maxwellop);
+   pcg.SetPrintLevel(1);
+   pcg.Mult(trueRhs, trueX);
+
+
+   if (myid == 0)
+   {
+      cout << "PCG with Block AMS finished" << endl;
+   }
+
+   *E_gf = 0.0;
+   *H_gf = 0.0;
+
+   E_gf->Distribute(&(trueX.GetBlock(0)));
+   H_gf->Distribute(&(trueX.GetBlock(1)));
+
+   int order_quad = max(2, 2*order+1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i=0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double Error_E = E_gf->ComputeL2Error(Eex, irs);
+   double norm_E = ComputeGlobalLpNorm(2, Eex, *pmesh, irs);
+
+   double Error_H = H_gf->ComputeL2Error(Hex, irs);
+   double norm_H = ComputeGlobalLpNorm(2, Hex , *pmesh, irs);
+
+   if (myid == 0)
+   {
+      // cout << "|| E_h - E || / || E || = " << Error_E / norm_E << "\n";
+      // cout << "|| H_h - H || / || H || = " << Error_H / norm_H << "\n";
+      cout << "|| E_h - E || = " << Error_E  << "\n";
+      cout << "|| H_h - H || = " << Error_H  << "\n";
+
+      cout << "Total error = " << sqrt(Error_H*Error_H+Error_E*Error_E) << "\n";
+
+      // cout << "Total Relative error = " <<  Error_E / norm_E + Error_H / norm_H  << "\n";
+      // cout << "E Relative error = " <<  Error_E / norm_E  << "\n";
+      // cout << "H Relative error = " <<  Error_H / norm_H  << "\n";
+
+      // cout << "|| E || = " << norm_E  << "\n";
+      // cout << "|| H || = " << norm_H  << "\n";
+   }
+
+   if (visualization)
+   {
+      // ParGridFunction * Eex_gf = new ParGridFunction;   
+      // Eex_gf->MakeRef(fespace, x.GetBlock(0));
+      // Eex_gf->ProjectCoefficient(Eex);
+      // ParGridFunction * Hex_gf = new ParGridFunction;   
+      // Hex_gf->MakeRef(fespace, x.GetBlock(1));
+      // Hex_gf->ProjectCoefficient(Hex);
+
+      // 8. Connect to GLVis.
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream E_sock(vishost, visport);
+      E_sock << "parallel " << num_procs << " " << myid << "\n";
+      E_sock.precision(8);
+      E_sock << "solution\n" << *pmesh << *E_gf << "window_title 'Electric field'" << endl;
+      // MPI_Barrier(pmesh->GetComm());
+      // socketstream Eex_sock(vishost, visport);
+      // Eex_sock << "parallel " << num_procs << " " << myid << "\n";
+      // Eex_sock.precision(8);
+      // Eex_sock << "solution\n" << *pmesh << *Eex_gf << "window_title 'Exact Electric Field'" << endl;
+      MPI_Barrier(pmesh->GetComm());
+      socketstream H_sock(vishost, visport);
+      H_sock << "parallel " << num_procs << " " << myid << "\n";
+      H_sock.precision(8);
+      H_sock << "solution\n" << *pmesh << *H_gf << "window_title 'Magnetic field'" << endl;
+      // MPI_Barrier(pmesh->GetComm());
+      // socketstream Hex_sock(vishost, visport);
+      // Hex_sock << "parallel " << num_procs << " " << myid << "\n";
+      // Hex_sock.precision(8);
+      // Hex_sock << "solution\n" << *pmesh << *Hex_gf << "window_title 'Exact Magnetic field'" << endl;
+   }
+
+
+   delete a_EE;
+   delete a_HE;
+   delete a_HH;
+   delete b_E;
+   delete b_H;
+   delete fec;
+   delete fespace;
+   delete pmesh;
+   MPI_Finalize();
+   return 0;
+}
+
+
+//define exact solution
+void E_exact(const Vector &x, Vector &E)
+{
+   double curlE[3], curl2E[3];
+   get_maxwell_solution(x, E, curlE, curl2E);
+}
+
+void H_exact(const Vector &x, Vector &H)
+{
+   double E[3], curlE[3], curl2E[3];
+   get_maxwell_solution(x, E, curlE, curl2E);
+   for (int i = 0; i<3; i++) {H(i) = curlE[i]/omega;}
+}
+
+//calculate RHS from exact solution
+void f_exact_E(const Vector &x, Vector &f)
+{
+   double E[3], curlE[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   // curl E - omega H = 0 
+   f(0) = curlE[0] - omega * (curlE[0]/omega); // = 0
+   f(1) = curlE[1] - omega * (curlE[1]/omega); // = 0
+   f(2) = curlE[2] - omega * (curlE[2]/omega); // = 0
+}
+
+void f_exact_H(const Vector &x, Vector &f)
+{
+   if (dim != 3)
+   {
+      cout << "2D not set up yet: " << endl;
+      exit(0);
+   }
+   double E[3], curlE[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   // curl H - omega E = f 
+   // = curl (curl E / omega) - omega E 
+   f(0) = curl2E[0]/omega - omega * E[0];
+   f(1) = curl2E[1]/omega - omega * E[1];
+   f(2) = curl2E[2]/omega - omega * E[2];
+}
+
+void scaledf_exact_E(const Vector &x, Vector &f)
+{
+   double E[3], curlE[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   //  - omega *( curl E - omega H) = 0 
+   f(0) =-omega * (curlE[0] - omega * (curlE[0]/omega)); // = 0
+   f(1) =-omega * (curlE[1] - omega * (curlE[1]/omega)); // = 0
+   f(2) =-omega * (curlE[2] - omega * (curlE[2]/omega)); // = 0
+}
+
+void scaledf_exact_H(const Vector &x, Vector &f)
+{
+   double E[3], curlE[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curlE, curl2E);
+
+   // curl H - omega E = f 
+   // = - omega *( curl (curl E / omega) - omega E) 
+   
+   f(0) = -omega * (curl2E[0]/omega - omega * E[0]);
+   f(1) = -omega * (curl2E[1]/omega - omega * E[1]);
+   f(2) = -omega * (curl2E[2]/omega - omega * E[2]);
+}
+
+void get_maxwell_solution(const Vector & X, double E[], double curlE[], double curl2E[])
+{
+   const double x = X[0];
+   const double y = X[1];
+   const double z = X[2];
+
+   if (isol == 0) // polynomial
+   {
+      // Polynomial vanishing on the boundary
+      E[0] = y * z * (1.0 - y) * (1.0 - z);
+      E[1] = (1.0 - x) * x * y * (1.0 - z) * z;
+      E[2] = (1.0 - x) * x * (1.0 - y) * y;
+      //
+
+      curlE[0] = -(-1.0 + x) * x * (1.0 + y * (-3.0 + 2.0 * z));
+      curlE[1] = -2.0 * (-1.0 + y) * y * (x - z);
+      curlE[2] = (1.0 + (-3.0 + 2.0 * x) * y) * (-1.0 + z) * z; 
+
+      curl2E[0] = -2.0 * (-1.0 + y) * y + (-3.0 + 2.0 * x) * (-1.0 + z) * z;
+      curl2E[1] = -2.0 * y * (-x + x*x + (-1.0 + z) * z);
+      curl2E[2] = -2.0 * (-1.0 + y) * y + (-1.0 + x) * x * (-3.0 + 2.0 * z);
+
+   }
+
+   else if (isol == 1) // sinusoidal
+   {
+      E[0] = sin(omega * y);
+      E[1] = sin(omega * z);
+      E[2] = sin(omega * x);
+
+      curlE[0] = -omega * cos(omega * z);
+      curlE[1] = -omega * cos(omega * x);;
+      curlE[2] = -omega * cos(omega * y);; 
+
+      curl2E[0] = omega * omega * E[0];
+      curl2E[1] = omega * omega * E[1];
+      curl2E[2] = omega * omega * E[2];
+
+   }
+   else if (isol == 2) //simple polynomial
+   {
+      E[0] = y;
+      E[1] = z;
+      E[2] = x;
+
+      curlE[0] = -1.0;
+      curlE[1] = -1.0;
+      curlE[2] = -1.0; 
+
+      curl2E[0] =0.0;
+      curl2E[1] =0.0;
+      curl2E[2] =0.0;
+   }
+      else if (isol == 4) //constant
+   {
+      E[0] = 1.0;
+      E[1] = 1.0;
+      E[2] = 1.0;
+
+      curlE[0] = 0.0;
+      curlE[1] = 0.0;
+      curlE[2] = 0.0; 
+
+      curl2E[0] =0.0;
+      curl2E[1] =0.0;
+      curl2E[2] =0.0;
+   }
+   else if (isol == 3) // plane wave
+   {
+      double coeff = omega / sqrt(3.0);
+      E[0] = cos(coeff * (x + y + z));
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+
+      curlE[0] = 0.0;
+      curlE[1] = -coeff * sin(coeff * (x+y+z));
+      curlE[2] = coeff * sin(coeff * (x+y+z)); 
+
+      curl2E[0] = 2.0 * coeff * coeff * E[0];
+      curl2E[1] = -coeff * coeff * E[0];
+      curl2E[2] = -coeff * coeff * E[0];
+   }
+
+}
+

examples/waves/back/AddSchwarz.cpp

@@ -0,0 +1,446 @@
+//                                MFEM Example 1
+//
+// Compile with: make AddScwarz
+//
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+int main(int argc, char *argv[])
+{
+   // 1. Parse command-line options.
+   const char *mesh_file = "../data/one-hex.mesh";
+   int order = 1;
+   int sdim = 1;
+   bool static_cond = false;
+   bool pa = false;
+   const char *device_config = "cpu";
+   bool visualization = true;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&sdim, "-d", "--dimension", "Dimension");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&pa, "-pa", "--partial-assembly", "-no-pa",
+                  "--no-partial-assembly", "Enable Partial Assembly.");
+   args.AddOption(&device_config, "-d", "--device",
+                  "Device configuration string, see Device::Configure().");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   if (!args.Good())
+   {
+      args.PrintUsage(cout);
+      return 1;
+   }
+   args.PrintOptions(cout);
+
+   // 2. Enable hardware devices such as GPUs, and programming models such as
+   //    CUDA, OCCA, RAJA and OpenMP based on command line options.
+   Device device(device_config);
+   device.Print();
+
+   // 3. Read the mesh from the given mesh file. We can handle triangular,
+   //    quadrilateral, tetrahedral, hexahedral, surface and volume meshes with
+   //    the same code.
+   // Mesh *mesh = new Mesh(mesh_file, 1, 1);
+
+   Mesh * mesh; 
+   // Define a simple square mesh
+   if (sdim == 2)
+   {
+      mesh = new Mesh(1, 1, Element::QUADRILATERAL, true,1.0, 1.0,false);
+   }
+   else
+   {
+      mesh = new Mesh(1, 1, 1, Element::HEXAHEDRON, true,1.0, 1.0,1.0, false);
+   }
+
+
+   int dim = mesh->Dimension();
+
+   // 4. Refine the mesh to increase the resolution. In this example we do
+   //    'ref_levels' of uniform refinement. We choose 'ref_levels' to be the
+   //    largest number that gives a final mesh with no more than 50,000
+   //    elements.
+   {
+      // int ref_levels =
+      //    (int)floor(log(50000./mesh->GetNE())/log(2.)/dim);
+       int ref_levels = 1;
+      for (int l = 0; l < ref_levels; l++)
+      {
+         mesh->UniformRefinement();
+      }
+   }
+
+   // mesh->PrintInfo(cout);
+
+   int nrelem = mesh->GetNE();
+   int nrvert = mesh->GetNV();
+   int nredge = mesh->GetNEdges();
+   int nrface = mesh->GetNFaces();
+
+   // 5. Define a finite element space on the mesh. 
+   FiniteElementCollection *fec = new H1_FECollection(order, dim);
+   FiniteElementSpace *fespace = new FiniteElementSpace(mesh, fec);
+
+
+   // cout << "Element to dof table " << endl;
+   // fespace->GetElementToDofTable().Print();
+
+   // Array<int>edge_dofs;
+   // Array<int>edge_vert;
+   // for (int i=0; i< nredge; i++ )
+   // {
+   //    // fespace->GetEdgeDofs(i,edge_dofs);
+   //    mesh->GetEdgeVertices(i,edge_vert);
+   //    cout << "edge no " << i << " vertices :" ; edge_vert.Print();
+   // }
+   // Array<int>face_dofs;
+   // Array<int>face_vert;
+   // for (int i=0; i< nrface; i++ )
+   // {
+   //    // fespace->GetFaceDofs(i,face_dofs);
+   //    mesh->GetFaceVertices(i,face_vert);
+   //    // cout << " face no " << i << " dofs :" ; face_dofs.Print();
+   //    cout << "face no " << i << " vertices :" ; face_vert.Print(); 
+   // }
+   // Array<int>elem_vert;
+   // for (int i=0; i< nrelem; i++ )
+   // {
+   //    // fespace->GetFaceDofs(i,face_dofs);
+   //    mesh->GetElementVertices(i,elem_vert);
+   //    // cout << " face no " << i << " dofs :" ; face_dofs.Print();
+   //    cout << "elem no " << i << " vertices :" ; elem_vert.Print();
+   // }
+
+   // construct a list of indices for each patch/vertex (that is not essential)
+   // // Get essential
+   // Array<int> ess_tdof_list;
+   // if (mesh->bdr_attributes.Size())
+   // {
+   //    Array<int> ess_bdr(mesh->bdr_attributes.Max());
+   //    ess_bdr = 1;
+   //    fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   // }
+   // cout<< "essential boundary dofs: " ; ess_tdof_list.Print(); 
+
+
+
+   Array<Array<int>> patch(nrvert);
+   // Initialize each patch by an array consisting of the vertex its self
+   // Numbering of vertices starts from 0
+   for (int i=0; i<nrvert; i++)
+   {
+      Array<int> vert(1);
+      vert=i;
+      patch[i] = vert;
+   }
+   // Loop through all the edges and find the the vertices they contribute to
+   Array<int>edge_vert;
+   Array<int>edge_int_dofs;
+   for (int i=0; i< nredge; i++ )
+   {
+      mesh->GetEdgeVertices(i,edge_vert);
+      int nv = edge_vert.Size();
+      fespace->GetEdgeInteriorDofs(i,edge_int_dofs);
+      for (int j=0; j<nv ; j++)
+      {
+         int k = edge_vert[j];
+         patch[k].Append(edge_int_dofs);
+      }
+   }
+// Loop through all the faces and find the the vertices they contribute to
+   Array<int>face_vert;
+   Array<int>face_int_dofs;
+   for (int i=0; i< nrface; i++ )
+   {
+      mesh->GetFaceVertices(i,face_vert); 
+      int nv = face_vert.Size();
+      fespace->GetFaceInteriorDofs(i,face_int_dofs);
+      for (int j=0; j<nv ; j++)
+      {
+         int k = face_vert[j];
+         patch[k].Append(face_int_dofs);
+      }
+   }
+// Loop through all the elements and find the the vertices they contribute to
+   Array<int>elem_vert;
+   Array<int>elem_int_dofs;
+   for (int i=0; i< nrelem; i++ )
+   {
+      mesh->GetElementVertices(i,elem_vert); 
+      int nv = elem_vert.Size();
+      fespace->GetElementInteriorDofs(i,elem_int_dofs);
+      for (int j=0; j<nv ; j++)
+      {
+         int k = elem_vert[j];
+         patch[k].Append(elem_int_dofs);
+      }
+   }
+   for (int i=0; i<nrvert; i++)
+   {
+      cout << "Patch no: " << i << " dofs " ;  
+      patch[i].Print();   
+   }
+
+   // Build a sparse matrix out of this map to extract the patch submatrix
+   Array<SparseMatrix *> Pid(nrvert); 
+   Array<int> dofoffset(nrvert);
+   dofoffset = 0;
+   for (int i=0; i<nrvert; i++)
+   {
+      int height = fespace->GetVSize();
+      int width  = patch[i].Size();
+      Pid[i] = new SparseMatrix(height,width);
+      Pid[i]->Set(i,dofoffset[i],1.0); // Fill in the vertex dof (1 column for each vertex)
+      dofoffset[i]++;
+   }
+
+   // Fill the sparse matrix with the edge dof indices (1 column for each dof)
+   for (int i=0; i< nredge; i++ )
+   {
+      mesh->GetEdgeVertices(i,edge_vert);
+      int nv = edge_vert.Size();
+      fespace->GetEdgeInteriorDofs(i,edge_int_dofs);
+      int ne = edge_int_dofs.Size();
+      for (int j=0; j<nv ; j++)
+      {
+         int k = edge_vert[j];
+         for (int l=0; l < ne; l++)
+         {
+            int m = edge_int_dofs[l];
+            Pid[k]->Set(m,dofoffset[k],1.0);
+            dofoffset[k]++;
+         }
+      }
+   }
+   // Fill the sparse matrix with the face dof indices (1 column for each dof)
+   for (int i=0; i< nrface; i++ )
+   {
+      mesh->GetEdgeVertices(i,face_vert);
+      int nv = face_vert.Size();
+      fespace->GetFaceInteriorDofs(i,face_int_dofs);
+      int nf = face_int_dofs.Size();
+      for (int j=0; j<nv ; j++)
+      {
+         int k = face_vert[j];
+         for (int l=0; l < nf; l++)
+         {
+            int m = face_int_dofs[l];
+            Pid[k]->Set(m,dofoffset[k],1.0);
+            dofoffset[k]++;
+         }
+      }
+   }
+
+   // Fill the sparse matrix with the element (middle) dof indices (1 column for each dof)
+   for (int i=0; i< nrelem; i++ )
+   {
+      mesh->GetElementVertices(i,elem_vert);
+      int nv = elem_vert.Size();
+      fespace->GetElementInteriorDofs(i,elem_int_dofs);
+      int nel = elem_int_dofs.Size();
+      for (int j=0; j<nv ; j++)
+      {
+         int k = elem_vert[j];
+         for (int l=0; l < nel; l++)
+         {
+            int m = elem_int_dofs[l];
+            Pid[k]->Set(m,dofoffset[k],1.0);
+            dofoffset[k]++;
+         }
+      }
+   }
+
+   Pid[0]->Finalize();
+   Pid[0]->PrintMatlab(cout);
+
+
+   // SparseMatrix * S = new SparseMatrix(5,5);
+   // // S->PrintMatlab(std::cout);
+   // S->Set(1,1,1.0);
+   // S->Finalize();
+   // // S->Print(cout);
+   // S->PrintMatlab(cout);
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+   // cout << "Edges to vertex table " << endl;
+   // mesh->GetEdgeVertexTable()->Print();
+
+
+   // if (sdim == 3)
+   // {
+   //    cout << "Faces to edges table " << endl;
+   //    mesh->GetFaceEdgeTable()->Print(); // is this really Face to Vertex table?
+   // }
+
+
+   // Custom vertex patch partitioning partitioning
+   // Array<int> vertex_dofs;
+   // for (int i=0; i< nrelems; i++ )
+   // {
+   //    fespace->GetElementVertices(i,vertex_dofs);
+   //    std::cout << "Element "<< i+1 << " Vertex dofs: " ; vertex_dofs.Print();
+   // }
+
+   // This is local numbering of nodes
+   // for (int i=0; i< nrelems; i++ )
+   // {
+   //    cout << "Element " << i+1 << " Number of vertices: " << 
+   //                          mesh->GetElement(i)->GetNVertices() << endl;
+
+   //    const int ne =  mesh->GetElement(i)->GetNEdges();
+   //    for (int j=0; j< ne; j++ )
+   //    {
+   //       const int *ev = mesh->GetElement(i)->GetEdgeVertices(j);
+   //       cout << "Edge " << j << " vertices " <<  ev[0] << ", "<< ev[1] << endl;
+   //    }
+   // }
+
+   // Array<int> vertex_dofs;
+   // fespace->GetElementToDofTable().Print();
+   // for (int i=0; i< nrelems; i++ )
+   // {
+   //    std::cout << "Vertex dofs " << endl;
+   //    fespace->GetElementVertices(i,vertex_dofs);
+   //    vertex_dofs.Print();
+   // }
+
+ 
+
+   // Array<int> vertex_dofs;
+   // Array<int> interior_dofs;
+   // for (int i=0; i< nrelems; i++ )
+   // {
+   //    std::cout << "Element " << i+1 << endl;
+   //    fespace->GetElementVertices(i,vertex_dofs);
+   //    std::cout << "Vertex dofs " << endl;
+   //    vertex_dofs.Print();
+   //    fespace->GetElementInteriorDofs(i,interior_dofs);
+   //    std::cout << "Interior dofs " << endl;
+   //    interior_dofs.Print();
+
+
+   // }
+
+   // std::cout << "Number of global unknowns: " << fespace->GetVSize() << endl;
+   // std::cout << "Number of vertex dofs " << fespace->GetNVDofs() << endl;
+   // std::cout << "Number of edge   dofs " << fespace->GetNEDofs() << endl;
+   // std::cout << "Number of face   dofs " << fespace->GetNFDofs() << endl;
+   // std::cout << "Number of total  dofs " << fespace->GetNDofs() << endl;
+
+
+//    cout << "Number of finite element unknowns: "
+//         << fespace->GetTrueVSize() << endl;
+
+//    // 6. Determine the list of true (i.e. conforming) essential boundary dofs.
+//    //    In this example, the boundary conditions are defined by marking all
+//    //    the boundary attributes from the mesh as essential (Dirichlet) and
+//    //    converting them to a list of true dofs.
+   // Array<int> ess_tdof_list;
+   // if (mesh->bdr_attributes.Size())
+   // {
+   //    Array<int> ess_bdr(mesh->bdr_attributes.Max());
+   //    ess_bdr = 1;
+   //    fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   // }
+
+   // 7. Set up the linear form b(.) which corresponds to the right-hand side of
+   //    the FEM linear system, which in this case is (1,phi_i) where phi_i are
+   //    the basis functions in the finite element fespace.
+   // LinearForm *b = new LinearForm(fespace);
+   // ConstantCoefficient one(1.0);
+   // b->AddDomainIntegrator(new DomainLFIntegrator(one));
+   // b->Assemble();
+
+   // 8. Define the solution vector x as a finite element grid function
+   //    corresponding to fespace. Initialize x with initial guess of zero,
+   //    which satisfies the boundary conditions.
+   // GridFunction x(fespace);
+   // x = 0.0;
+
+   // 9. Set up the bilinear form a(.,.) on the finite element space
+   //    corresponding to the Laplacian operator -Delta, by adding the Diffusion
+   //    domain integrator.
+   // BilinearForm *a = new BilinearForm(fespace);
+   // if (pa) { a->SetAssemblyLevel(AssemblyLevel::PARTIAL); }
+   // a->AddDomainIntegrator(new DiffusionIntegrator(one));
+   // a->Assemble();
+
+   // OperatorPtr A;
+   // Vector B, X;
+   // a->FormLinearSystem(ess_tdof_list, x, *b, A, X, B);
+
+   // cout << "Size of linear system: " << A->Height() << endl;
+
+
+   // A->PrintMatlab(cout);
+// #ifndef MFEM_USE_SUITESPARSE
+//       // Use a simple symmetric Gauss-Seidel preconditioner with PCG.
+//       GSSmoother M((SparseMatrix&)(*A));
+//       PCG(*A, M, B, X, 1, 200, 1e-12, 0.0);
+// #else
+//       // If MFEM was compiled with SuiteSparse, use UMFPACK to solve the system.
+//       UMFPackSolver umf_solver;
+//       umf_solver.Control[UMFPACK_ORDERING] = UMFPACK_ORDERING_METIS;
+//       umf_solver.SetOperator(*A);
+//       umf_solver.Mult(B, X);
+// #endif
+
+//    // 12. Recover the solution as a finite element grid function.
+//    a->RecoverFEMSolution(X, *b, x);
+
+   // 14. Send the solution by socket to a GLVis server.
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock.precision(8);
+      sol_sock << "mesh\n" << *mesh << flush;
+   }
+
+//    // 15. Free the used memory.
+//    delete a;
+//    delete b;
+//    delete fespace;
+//    if (order > 0) { delete fec; }
+   delete mesh;
+
+   return 0;
+}

examples/waves/back/blk_complex_maxwellp.cpp

@@ -0,0 +1,420 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include <cmath>
+#include <complex>
+
+using namespace std;
+using namespace mfem;
+
+// #define DEFINITE
+
+#ifndef MFEM_USE_PETSC
+#error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+#endif
+
+// Define exact solution
+void E_exact_Re(const Vector & x, Vector & E);
+void f_exact_Re(const Vector & x, Vector & f);
+void get_maxwell_solution_Re(const Vector & x, double E[], double curl2E[]);
+
+void E_exact_Im(const Vector & x, Vector & E);
+void f_exact_Im(const Vector & x, Vector & f);
+void get_maxwell_solution_Im(const Vector & x, double E[], double curl2E[]);
+
+
+// Mesh Size
+Vector mesh_dim_(0); // x, y, z dimensions of mesh
+int dim;
+double omega;
+double complex_shift;
+int isol = 1;
+
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv); // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid); // Determine process identifier
+
+   const char *mesh_file = "../../data/one-hex.mesh";
+   int order = 1;
+   // number of wavelengths
+   double k = 0.5;
+   //
+   const char *petscrc_file = "petscrc_mult_options";
+   // visualization flag
+   bool visualization = 1;
+   // number of initial ref
+   int initref = 1;
+   // number of mg levels
+   int maxref = 1;
+   //
+   complex_shift = 0.0;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree)."); 
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths"); 
+   args.AddOption(&complex_shift, "-cs", "--complex_shift",
+                  "Complex shift");                
+   args.AddOption(&isol, "-isol", "--exact",
+                  "Exact solution flag - 0:polynomial, 1: plane wave");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&maxref, "-maxref", "--maxref",
+                  "Number of Refinements.");   
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");                           
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   // Angular frequency
+   omega = 2.0*k*M_PI;
+
+// 2b. We initialize PETSc
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+   // Create serial mesh 
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   // create parallel mesh and delete the serial one
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // Create H(curl) (Nedelec) Finite element space
+   FiniteElementCollection *fec   = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *ND_fespace = new ParFiniteElementSpace(pmesh, fec);
+
+   std::vector<HypreParMatrix*>  P(maxref);
+
+   for (int i = 0; i < maxref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*ND_fespace);
+      pmesh->UniformRefinement();
+      // Update fespace
+      ND_fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      ND_fespace->GetTrueTransferOperator(cfespace, Tr);
+      Tr.SetOperatorOwner(false);
+      Tr.Get(P[i]);
+   }
+
+   Array<int> block_offsets(3);
+   block_offsets[0] = 0;
+   block_offsets[1] = ND_fespace->GetVSize();
+   block_offsets[2] = ND_fespace->GetVSize();
+   block_offsets.PartialSum();
+
+   Array<int> block_trueOffsets(3);
+   block_trueOffsets[0] = 0;
+   block_trueOffsets[1] = ND_fespace->TrueVSize();
+   block_trueOffsets[2] = ND_fespace->TrueVSize();
+   block_trueOffsets.PartialSum();
+
+   BlockVector x(block_offsets), rhs(block_offsets);
+   BlockVector trueX(block_trueOffsets), trueRhs(block_trueOffsets);
+
+   x = 0.0;
+   rhs = 0.0;
+   trueX = 0.0;
+   trueRhs = 0.0;
+
+   Array<int> ess_tdof_list;
+   Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+   ess_bdr = 1;
+   ND_fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+
+
+   VectorFunctionCoefficient E_Re(sdim, E_exact_Re);
+   VectorFunctionCoefficient E_Im(sdim, E_exact_Im);
+
+   ParComplexGridFunction * E_gf = new ParComplexGridFunction(ND_fespace);
+   E_gf->real().MakeRef(ND_fespace, x.GetBlock(0));
+   E_gf->imag().MakeRef(ND_fespace, x.GetBlock(1));
+   E_gf->ProjectCoefficient(E_Re,E_Im);
+
+   // 7. Linear form b(.) (Right hand side)
+   VectorFunctionCoefficient f_Re(dim, f_exact_Re);
+   VectorFunctionCoefficient f_Im(dim, f_exact_Im);
+   ParLinearForm b_Re(ND_fespace);
+   b_Re.AddDomainIntegrator(new VectorFEDomainLFIntegrator(f_Re));
+   b_Re.Assemble();
+   ParLinearForm b_Im(ND_fespace);
+   b_Im.AddDomainIntegrator(new VectorFEDomainLFIntegrator(f_Im));
+   b_Im.Assemble();
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ConstantCoefficient muinv(1.0);
+   ConstantCoefficient sigma(-pow(omega, 2));
+   ConstantCoefficient alpha(complex_shift);
+   ParBilinearForm a_Re(ND_fespace);
+   a_Re.AddDomainIntegrator(new CurlCurlIntegrator(muinv)); 
+   a_Re.AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   a_Re.Assemble();
+   a_Re.Finalize();
+
+
+   ParBilinearForm a_Im(ND_fespace);
+   a_Im.AddDomainIntegrator(new VectorFEMassIntegrator(alpha));
+   a_Im.Assemble();
+   a_Im.Finalize();
+   HypreParMatrix * A_Re = new HypreParMatrix;
+   HypreParMatrix * A_Im = new HypreParMatrix;
+
+   Vector b_aux(b_Re.Size()), B_aux, X_aux;
+   a_Re.FormLinearSystem(ess_tdof_list, x.GetBlock(0), b_Re, *A_Re, trueX.GetBlock(0), trueRhs.GetBlock(0));
+   a_Re.FormLinearSystem(ess_tdof_list, x.GetBlock(1), b_Im, *A_Re, trueX.GetBlock(1), trueRhs.GetBlock(1));
+
+   b_aux = 0.0;
+   a_Im.FormLinearSystem(ess_tdof_list, x.GetBlock(0), b_aux, *A_Im, X_aux, B_aux);
+   trueRhs.GetBlock(1) += B_aux;
+
+   b_aux = 0.0;
+   a_Im.FormLinearSystem(ess_tdof_list, x.GetBlock(1), b_aux, *A_Im, X_aux, B_aux);
+   trueRhs.GetBlock(0) -= B_aux;
+
+   // // Modify RHS and diagonal entries of A_Im
+   int n = ess_tdof_list.Size();
+   int j;
+
+   hypre_ParCSRMatrix * Ah = (hypre_ParCSRMatrix *)const_cast<HypreParMatrix&>(*A_Im);
+   for (int k=0; k<n; k++)
+   {
+      j=ess_tdof_list[k];
+      trueRhs.GetBlock(0)(j) = trueX.GetBlock(0)(j);
+      trueRhs.GetBlock(1)(j) = trueX.GetBlock(1)(j);
+      Ah->diag->data[Ah->diag->i[j]] = 0.0;
+   }
+
+   ComplexHypreParMatrix * AZ = new ComplexHypreParMatrix(A_Re, A_Im, false, false, ComplexOperator::HERMITIAN);
+   HypreParMatrix * A = AZ->GetSystemMatrix();
+
+   if (myid == 0)
+   {
+      cout << "Size of fine grid system: "
+           << A->GetGlobalNumRows() << " x " << A->GetGlobalNumCols() << endl;
+   }
+
+   ComplexGMGSolver M(AZ, P);
+   M.SetTheta(0.5);
+   M.SetSmootherType(HypreSmoother::Jacobi);
+
+   int maxit(5000);
+   double rtol(1.e-12);
+   double atol(0.0);
+
+   trueX = 0.0;
+   GMRESSolver gmres(MPI_COMM_WORLD);
+   gmres.SetAbsTol(atol);
+   gmres.SetRelTol(rtol);
+   gmres.SetMaxIter(maxit);
+   gmres.SetOperator(*A);
+   gmres.SetPreconditioner(M);
+   gmres.SetPrintLevel(1);
+   gmres.Mult(trueRhs, trueX);
+
+   // PetscLinearSolver * invA = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   // invA->SetOperator(PetscParMatrix(A, Operator::PETSC_MATAIJ));
+   // invA->Mult(trueRhs,trueX);
+
+   E_gf->real().Distribute(&(trueX.GetBlock(0)));
+   E_gf->imag().Distribute(&(trueX.GetBlock(1)));
+
+   // Compute error
+   int order_quad = max(2, 2 * order + 1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i = 0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double L2Error_Re = E_gf->real().ComputeL2Error(E_Re, irs);
+   double norm_E_Re = ComputeGlobalLpNorm(2, E_Re, *pmesh, irs);
+
+   double L2Error_Im = E_gf->imag().ComputeL2Error(E_Im, irs);
+   double norm_E_Im = ComputeGlobalLpNorm(2, E_Im, *pmesh, irs);
+
+
+   if (myid == 0)
+   {
+      cout << " Real Part: || E_h - E || / ||E|| = " << L2Error_Re / norm_E_Re << '\n' << endl;
+      cout << " Imag Part: || E_h - E || / ||E|| = " << L2Error_Im / norm_E_Im << '\n' << endl;
+
+      cout << " Real Part: || E_h - E || = " << L2Error_Re << '\n' << endl;
+      cout << " Imag Part: || E_h - E || = " << L2Error_Im << '\n' << endl;
+   }
+
+
+   // visualization   
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << E_gf->real() << "window_title 'Real part'" << flush;
+
+      socketstream sol_sock_Im(vishost, visport);
+      sol_sock_Im << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_Im.precision(8);
+      sol_sock_Im << "solution\n" << *pmesh << E_gf->imag() << "window_title 'Imaginary part'" << flush;
+   }
+
+   // delete invA;
+   delete fec;
+   delete ND_fespace;
+   delete pmesh;
+
+   MFEMFinalizePetsc();
+   MPI_Finalize();
+
+}
+//define exact solution
+void E_exact_Re(const Vector &x, Vector &E)
+{
+   double curl2E[3];
+   get_maxwell_solution_Re(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+void f_exact_Re(const Vector &x, Vector &f)
+{
+   double E_Re[3], curl2E_Re[3];
+   double E_Im[3], curl2E_Im[3];
+
+   get_maxwell_solution_Re(x, E_Re, curl2E_Re);
+   get_maxwell_solution_Re(x, E_Im, curl2E_Im);
+
+   // curl ( curl E) - omega^2 E = f
+   double coeff;
+   coeff = -omega * omega;
+   f(0) = curl2E_Re[0] + coeff * E_Re[0];
+   f(1) = curl2E_Re[1] + coeff * E_Re[1];
+   f(2) = curl2E_Re[2] + coeff * E_Re[2];
+
+   // Acount for the complex shift
+   f(0) += -complex_shift*E_Im[0];
+   f(1) += -complex_shift*E_Im[1];
+   f(2) += -complex_shift*E_Im[2];
+}
+
+void get_maxwell_solution_Re(const Vector & x, double E[], double curl2E[])
+{
+
+   if (isol == 0) // polynomial
+   {
+      E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+      E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+      E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+      curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+      curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+      curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+   }
+   else
+   {
+      double alpha = omega / sqrt(3);
+      E[0] = cos(alpha*(x(0) + x(1) + x(2)));
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+      curl2E[0] = 2.0 * alpha * alpha * E[0];
+      curl2E[1] = -alpha * alpha * E[0];
+      curl2E[2] = -alpha * alpha * E[0];
+   }
+}
+
+
+
+//define exact solution
+void E_exact_Im(const Vector &x, Vector &E)
+{
+   double curl2E[3];
+   get_maxwell_solution_Re(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+void f_exact_Im(const Vector &x, Vector &f)
+{
+   double E_Re[3], curl2E_Re[3];
+   double E_Im[3], curl2E_Im[3];
+
+   get_maxwell_solution_Re(x, E_Im, curl2E_Im);
+   get_maxwell_solution_Re(x, E_Re, curl2E_Re);
+
+   // curl ( curl E) - omega^2 E = f
+   double coeff;
+   coeff = -omega * omega;
+   f(0) = curl2E_Im[0] + coeff * E_Im[0];
+   f(1) = curl2E_Im[1] + coeff * E_Im[1];
+   f(2) = curl2E_Im[2] + coeff * E_Im[2];
+
+   // Acount for the complex shift
+   f(0) += complex_shift*E_Re[0];
+   f(1) += complex_shift*E_Re[1];
+   f(2) += complex_shift*E_Re[2];
+}
+
+void get_maxwell_solution_Im(const Vector & x, double E[], double curl2E[])
+{
+   if (isol == 0) // polynomial
+   {
+      E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+      E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+      E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+      curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+      curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+      curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+   }
+   else
+   {
+      double alpha = omega / sqrt(3);   
+      E[0] = sin(alpha * (x(0) + x(1) + x(2)));
+      E[1] = 0.0;
+      E[2] = 0.0;
+      curl2E[0] = 2.0 * alpha * alpha * E[0];
+      curl2E[1] = -alpha * alpha * E[0];
+      curl2E[2] = -alpha * alpha * E[0];
+   }   
+}

examples/waves/back/helmholtzp.cpp

@@ -0,0 +1,413 @@
+//
+// Compile with: make helmholtz
+//
+// Sample runs:  helmholtz -m ../data/one-hex.mesh
+//               helmholtz -m ../data/fichera.mesh
+//               helmholtz -m ../data/fichera-mixed.mesh
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Helmholtz problem
+//               -Delta p - omega^2 p = 1 with impedance boundary conditiones.
+//
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Exact solution and r.h.s., see below for implementation.
+double p_exact_Re(const Vector &x);
+double p_exact_Im(const Vector &x);
+double f_exact_Re(const Vector &x);
+double f_exact_Im(const Vector &x);
+double g_exact_Re(const Vector &x);
+double g_exact_Im(const Vector &x);
+void grad_exact_Re(const Vector &x, Vector &grad_Re);
+void grad_exact_Im(const Vector &x, Vector &grad_Im);
+
+int dim;
+double omega;
+
+// flag for definite or indefinite
+// #define DEFINITE
+
+#ifndef MFEM_USE_PETSC
+#error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+#endif
+
+int main(int argc, char *argv[])
+{
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);                    // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);      // Determine process identifier
+
+   //-----------------------------------------------------------------------------
+
+   // 2. Parse command-line options.
+   // geometry file
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   bool visualization = 1;
+   // number of wavelengths
+   double k = 0.5;
+   // number of mg levels
+   int maxref = 1;
+   // number of initial ref
+   int initref = 1;
+   // PETSC
+   // const char *petscrc_file = "";
+   const char *petscrc_file = "petscrc_mult_options";
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&maxref, "-ref", "--maxref",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&petscrc_file, "-petscopts", "--petscopts",
+                  "PetscOptions file to use.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+   // Angular frequency
+   // omega = 2.0 * M_PI * k;
+   omega = k;
+
+   //-----------------------------------------------------------------------------
+
+   // 3. Read the mesh from the given mesh file.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // ----------------------------------------------------------------------------
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   // ----------------------------------------------------------------------------
+
+   // 5. Define a parallel mesh and delete the serial mesh.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // ----------------------------------------------------------------------------
+
+   // 6. Define a finite element space on the mesh.
+   FiniteElementCollection *fec = new H1_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+
+   std::vector<HypreParMatrix*>  P(maxref);
+
+   for (int i = 0; i < maxref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*fespace);
+      pmesh->UniformRefinement();
+      fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      fespace->GetTrueTransferOperator(cfespace, Tr);
+      HypreParMatrix * Paux;
+      Tr.Get(Paux);
+      P[i] = new HypreParMatrix(*Paux);
+   }
+
+   // 2b. We initialize PETSc
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+   Array<int> block_offsets(3);
+   block_offsets[0] = 0;
+   block_offsets[1] = fespace->GetVSize();
+   block_offsets[2] = fespace->GetVSize();
+   block_offsets.PartialSum();
+
+   Array<int> block_trueOffsets(3);
+   block_trueOffsets[0] = 0;
+   block_trueOffsets[1] = fespace->TrueVSize();
+   block_trueOffsets[2] = fespace->TrueVSize();
+   block_trueOffsets.PartialSum();
+
+   //    _           _    _  _       _  _
+   //   |             |  |    |     |    |
+   //   |  A      B   |  |x_Re|     |b_Re|
+   //   |             |  |    |  =  |    |
+   //   |  C      D   |  |x_Im|     |b_Im|
+   //   |_           _|  |_  _|     |_  _|
+
+   FunctionCoefficient p_Re(p_exact_Re);
+   FunctionCoefficient p_Im(p_exact_Im);
+   BlockVector x(block_offsets), rhs(block_offsets);
+   BlockVector trueX(block_trueOffsets), trueRhs(block_trueOffsets);
+
+   // 6. Set up the linear form (Real and Imaginary part)
+   FunctionCoefficient f_Re(f_exact_Re);
+   FunctionCoefficient g_Re(g_exact_Re);
+   VectorFunctionCoefficient grad_Re(sdim, grad_exact_Re);
+   ParLinearForm *b_Re(new ParLinearForm);
+   b_Re->Update(fespace, rhs.GetBlock(0), 0);
+   b_Re->AddDomainIntegrator(new DomainLFIntegrator(f_Re));
+   b_Re->AddBoundaryIntegrator(new BoundaryLFIntegrator(g_Re));
+   b_Re->AddBoundaryIntegrator(new BoundaryNormalLFIntegrator(grad_Re));
+   b_Re->Assemble();
+   b_Re->ParallelAssemble(trueRhs.GetBlock(0));
+
+   FunctionCoefficient f_Im(f_exact_Im);
+   FunctionCoefficient g_Im(g_exact_Im);
+   VectorFunctionCoefficient grad_Im(sdim, grad_exact_Im);
+   ParLinearForm *b_Im(new ParLinearForm);
+   b_Im->Update(fespace, rhs.GetBlock(1), 0);
+   b_Im->AddDomainIntegrator(new DomainLFIntegrator(f_Im));
+   b_Im->AddBoundaryIntegrator(new BoundaryLFIntegrator(g_Im));
+   b_Im->AddBoundaryIntegrator(new BoundaryNormalLFIntegrator(grad_Im));
+   b_Im->Assemble();
+   b_Im->ParallelAssemble(trueRhs.GetBlock(1));
+
+   // 7. Set up the bilinear form (Real and Imaginary part)
+   ConstantCoefficient one(1.0);
+   ConstantCoefficient sigma(-pow(omega, 2));
+
+   ParBilinearForm *a_rr(new ParBilinearForm(fespace));
+   a_rr->AddDomainIntegrator(new DiffusionIntegrator(one));
+   a_rr->AddDomainIntegrator(new MassIntegrator(sigma));
+   a_rr->Assemble();
+   a_rr->Finalize();
+   HypreParMatrix *A_rr = a_rr->ParallelAssemble();
+
+   ParBilinearForm *a_ri = new ParBilinearForm(fespace);
+   ConstantCoefficient negimpedance(-omega);
+   a_ri->AddBoundaryIntegrator(new BoundaryMassIntegrator(negimpedance));
+   a_ri->Assemble();
+   a_ri->Finalize();
+   HypreParMatrix *A_ri = a_ri->ParallelAssemble();
+
+   ParBilinearForm *a_ir = new ParBilinearForm(fespace);
+   ConstantCoefficient impedance(omega);
+   a_ir->AddBoundaryIntegrator(new BoundaryMassIntegrator(impedance));
+   a_ir->Assemble();
+   a_ir->Finalize();
+   HypreParMatrix *A_ir = a_ir->ParallelAssemble();
+
+   ParBilinearForm *a_ii = new ParBilinearForm(fespace);
+   a_ii->AddDomainIntegrator(new DiffusionIntegrator(one));
+   a_ii->AddDomainIntegrator(new MassIntegrator(sigma));
+   a_ii->Assemble();
+   a_ii->Finalize();
+   HypreParMatrix *A_ii = a_ii->ParallelAssemble();
+
+
+
+
+
+
+
+
+   BlockOperator *HelmholtzOp = new BlockOperator(block_trueOffsets);
+   HelmholtzOp->SetBlock(0, 0, A_rr);
+   HelmholtzOp->SetBlock(0, 1, A_ri);
+   HelmholtzOp->SetBlock(1, 0, A_ir);
+   HelmholtzOp->SetBlock(1, 1, A_ii);
+
+   PetscLinearSolver * invRe = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   invRe->SetOperator(PetscParMatrix(A_rr, Operator::PETSC_MATAIJ));
+   PetscLinearSolver * invIm = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   invIm->SetOperator(PetscParMatrix(A_ii, Operator::PETSC_MATAIJ));
+   BlockDiagonalPreconditioner *precInv = new BlockDiagonalPreconditioner(block_trueOffsets);
+   precInv->SetDiagonalBlock(0, invRe);
+   precInv->SetDiagonalBlock(1, invIm);
+
+
+
+   GMGSolver * gmgRe = new GMGSolver(A_rr, P);
+   gmgRe->SetTheta(0.5);
+   gmgRe->SetSmootherType(HypreSmoother::Jacobi);
+   GMGSolver * gmgIm = new GMGSolver(A_ii, P);
+   gmgIm->SetTheta(0.5);
+   gmgIm->SetSmootherType(HypreSmoother::Jacobi);
+   BlockDiagonalPreconditioner *precGMG = new BlockDiagonalPreconditioner(block_trueOffsets);
+   precGMG->SetDiagonalBlock(0, gmgRe);
+   precGMG->SetDiagonalBlock(1, gmgIm);
+
+
+   fespace->GetRestrictionMatrix()->Mult(x.GetBlock(0), trueX.GetBlock(0));
+   fespace->GetProlongationMatrix()->MultTranspose(rhs.GetBlock(0),trueRhs.GetBlock(0));
+
+   fespace->GetRestrictionMatrix()->Mult(x.GetBlock(1), trueX.GetBlock(1));
+   fespace->GetProlongationMatrix()->MultTranspose(rhs.GetBlock(1),trueRhs.GetBlock(1));
+
+   if (myid == 0)
+   {
+      cout << "Size of fine grid system: "
+           << 2.0 * A_rr->GetGlobalNumRows() << " x " << 2.0* A_rr->GetGlobalNumCols() << endl;
+   }
+   int maxit(5000);
+   double rtol(1.e-6);
+   double atol(0.0);
+   trueX = 0.0;
+   GMRESSolver gmres(MPI_COMM_WORLD);
+   gmres.SetAbsTol(atol);
+   gmres.SetRelTol(rtol);
+   gmres.SetMaxIter(maxit);
+   gmres.SetOperator(*HelmholtzOp);
+   // gmres.SetOperator(*cSysMat);
+   gmres.SetPreconditioner(*precGMG);
+   gmres.SetPrintLevel(1);
+   gmres.Mult(trueRhs, trueX);
+
+   if (myid == 0)
+   {
+      cout << "GMRES with GMG finished" << endl;
+   }
+   trueX = 0.0;
+   gmres.SetPreconditioner(*precInv);
+   gmres.Mult(trueRhs, trueX);
+
+   if (myid == 0)
+   {
+      cout << "GMRES with exact Inv finished" << endl;
+   }
+
+   // Direct solve
+   ComplexHypreParMatrix chpm(A_rr, A_ir, false, false);
+   HypreParMatrix *CA = chpm.GetSystemMatrix();
+   PetscLinearSolver * invCA = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   invCA->SetOperator(PetscParMatrix(CA, Operator::PETSC_MATAIJ));
+   trueX = 0.0;
+   invCA->Mult(trueRhs,trueX);
+
+   if (myid == 0)
+   {
+      cout << "Direct solver finished" << endl;
+   }
+
+
+   ParGridFunction *x_Re(new ParGridFunction);
+   ParGridFunction *x_Im(new ParGridFunction);
+
+   x_Re->MakeRef(fespace, x.GetBlock(0), 0);
+   x_Im->MakeRef(fespace, x.GetBlock(1), 0);
+
+   x_Re->Distribute(&(trueX.GetBlock(0)));
+   x_Im->Distribute(&(trueX.GetBlock(1)));
+
+   int order_quad = max(2, 2*order+1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i=0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   const int h1_norm_type = 1;
+   double L2error;
+   double H1error;
+
+   double L2err_Re  = x_Re->ComputeL2Error(p_Re);
+   double L2err_Im  = x_Im->ComputeL2Error(p_Im);
+
+   double loc_H1err_Re = x_Re->ComputeH1Error(&p_Re, &grad_Re, &one, 1.0, h1_norm_type);
+   double loc_H1err_Im = x_Im->ComputeH1Error(&p_Im, &grad_Im, &one, 1.0, h1_norm_type);
+   double H1err_Re = GlobalLpNorm(2.0, loc_H1err_Re, MPI_COMM_WORLD);
+   double H1err_Im = GlobalLpNorm(2.0, loc_H1err_Im, MPI_COMM_WORLD);
+   // double norm_Re = ComputeGlobalLpNorm(2, p_Re, *pmesh, irs);
+   // double norm_Im = ComputeGlobalLpNorm(2, p_Im, *pmesh, irs);
+      
+   L2error = sqrt(L2err_Re*L2err_Re + L2err_Im*L2err_Im);
+   H1error = sqrt(H1err_Re*H1err_Re + H1err_Im*H1err_Im);
+   // double L2norm = sqrt(norm_Re*norm_Re + norm_Im*norm_Im);
+   if (myid == 0)
+   {
+      cout << " || p_h - p ||_{H^1} = " << H1error <<  endl;
+      cout << " || p_h - p ||_{L^2} = " << L2error <<  endl;
+      // cout << " || p_h - p ||_{L^2}/||p||_{L^2} = " << L2error/L2norm <<  endl;
+   }
+
+   // release memory
+   delete b_Re;
+   delete b_Im;
+   delete a_rr;
+   delete a_ri;
+   delete a_ir;
+   delete a_ii;
+
+   delete fespace;
+   delete fec;
+   delete pmesh;
+   MFEMFinalizePetsc();
+   MPI_Finalize();
+   return 0;
+}
+
+//define exact solution plane wave
+double p_exact_Re(const Vector &x)
+{
+   return cos(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+}
+double p_exact_Im(const Vector &x)
+{
+   return -sin(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+}
+
+//calculate RHS from exact solution f = - \Delta u
+double f_exact_Re(const Vector &x)
+{
+   return 0.0;
+}
+double f_exact_Im(const Vector &x)
+{
+   return 0.0;
+}
+
+void grad_exact_Re(const Vector &x, Vector &grad_Re)
+{
+   grad_Re[0] = -omega / sqrt(3.0) * sin(omega / sqrt(3.0) * (x(0) + x(1) + x(2)));
+   grad_Re[1] = grad_Re[0];
+   grad_Re[2] = grad_Re[1];
+}
+void grad_exact_Im(const Vector &x, Vector &grad_Im)
+{
+
+   grad_Im[0] = -omega / sqrt(3.0) * cos(omega / sqrt(3.0) * (x(0) + x(1) + x(2)));
+   grad_Im[1] = grad_Im[0];
+   grad_Im[2] = grad_Im[1];
+}
+
+//define impedence coefficient: i omega p
+double g_exact_Re(const Vector &x)
+{
+   return omega * sin(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+}
+double g_exact_Im(const Vector &x)
+{
+   return omega * cos(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+}

examples/waves/back/helmholtzp_2.cpp

@@ -0,0 +1,307 @@
+//
+// Compile with: make helmholtz
+//
+// Sample runs:  helmholtz -m ../data/one-hex.mesh
+//               helmholtz -m ../data/fichera.mesh
+//               helmholtz -m ../data/fichera-mixed.mesh
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Helmholtz problem
+//               -Delta p - omega^2 p = 1 with impedance boundary conditiones.
+//
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Exact solution and r.h.s., see below for implementation.
+double p_exact_Re(const Vector &x);
+double p_exact_Im(const Vector &x);
+double f_exact_Re(const Vector &x);
+double f_exact_Im(const Vector &x);
+double g_exact_Re(const Vector &x);
+double g_exact_Im(const Vector &x);
+void grad_exact_Re(const Vector &x, Vector &grad_Re);
+void grad_exact_Im(const Vector &x, Vector &grad_Im);
+
+int dim;
+double omega;
+
+// flag for definite or indefinite
+// #define DEFINITE
+
+#ifndef MFEM_USE_PETSC
+#error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+#endif
+
+int main(int argc, char *argv[])
+{
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);                    // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);      // Determine process identifier
+
+   //-----------------------------------------------------------------------------
+
+   // 2. Parse command-line options.
+   // geometry file
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   bool visualization = 1;
+   // number of wavelengths
+   double k = 0.5;
+   // number of mg levels
+   int maxref = 1;
+   // number of initial ref
+   int initref = 1;
+   // PETSC
+   // const char *petscrc_file = "";
+   const char *petscrc_file = "petscrc_mult_options";
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&maxref, "-ref", "--maxref",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&petscrc_file, "-petscopts", "--petscopts",
+                  "PetscOptions file to use.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+   // Angular frequency
+   // omega = 2.0 * M_PI * k;
+   omega = k;
+
+   //-----------------------------------------------------------------------------
+
+   // 3. Read the mesh from the given mesh file.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // ----------------------------------------------------------------------------
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   // ----------------------------------------------------------------------------
+
+   // 5. Define a parallel mesh and delete the serial mesh.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // ----------------------------------------------------------------------------
+
+   // 6. Define a finite element space on the mesh.
+   FiniteElementCollection *fec = new H1_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+
+   std::vector<HypreParMatrix*>  P(maxref);
+
+   for (int i = 0; i < maxref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*fespace);
+      pmesh->UniformRefinement();
+      fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      fespace->GetTrueTransferOperator(cfespace, Tr);
+      HypreParMatrix * Paux;
+      Tr.Get(Paux);
+      P[i] = new HypreParMatrix(*Paux);
+   }
+
+   // 2b. We initialize PETSc
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+   Array<int> block_offsets(3);
+   block_offsets[0] = 0;
+   block_offsets[1] = fespace->GetVSize();
+   block_offsets[2] = fespace->GetVSize();
+   block_offsets.PartialSum();
+
+   Array<int> block_trueOffsets(3);
+   block_trueOffsets[0] = 0;
+   block_trueOffsets[1] = fespace->TrueVSize();
+   block_trueOffsets[2] = fespace->TrueVSize();
+   block_trueOffsets.PartialSum();
+
+   FunctionCoefficient p_Re(p_exact_Re);
+   FunctionCoefficient p_Im(p_exact_Im);
+   BlockVector x(block_offsets), rhs(block_offsets);
+   BlockVector trueX(block_trueOffsets), trueRhs(block_trueOffsets);
+
+   // 6. Set up the linear form (Real and Imaginary part)
+   FunctionCoefficient f_Re(f_exact_Re);
+   FunctionCoefficient g_Re(g_exact_Re);
+   VectorFunctionCoefficient grad_Re(sdim, grad_exact_Re);
+   ParLinearForm *b_Re(new ParLinearForm);
+   b_Re->Update(fespace, rhs.GetBlock(0), 0);
+   b_Re->AddDomainIntegrator(new DomainLFIntegrator(f_Re));
+   b_Re->AddBoundaryIntegrator(new BoundaryLFIntegrator(g_Re));
+   b_Re->AddBoundaryIntegrator(new BoundaryNormalLFIntegrator(grad_Re));
+   b_Re->Assemble();
+   b_Re->ParallelAssemble(trueRhs.GetBlock(0));
+
+   FunctionCoefficient f_Im(f_exact_Im);
+   FunctionCoefficient g_Im(g_exact_Im);
+   VectorFunctionCoefficient grad_Im(sdim, grad_exact_Im);
+   ParLinearForm *b_Im(new ParLinearForm);
+   b_Im->Update(fespace, rhs.GetBlock(1), 0);
+   b_Im->AddDomainIntegrator(new DomainLFIntegrator(f_Im));
+   b_Im->AddBoundaryIntegrator(new BoundaryLFIntegrator(g_Im));
+   b_Im->AddBoundaryIntegrator(new BoundaryNormalLFIntegrator(grad_Im));
+   b_Im->Assemble();
+   b_Im->ParallelAssemble(trueRhs.GetBlock(1));
+
+   // 7. Set up the bilinear form (Real and Imaginary part)
+   ConstantCoefficient one(1.0);
+   ConstantCoefficient sigma(-pow(omega, 2));
+
+   ParBilinearForm *a_rr(new ParBilinearForm(fespace));
+   a_rr->AddDomainIntegrator(new DiffusionIntegrator(one));
+   a_rr->AddDomainIntegrator(new MassIntegrator(sigma));
+   a_rr->Assemble();
+   a_rr->Finalize();
+   HypreParMatrix *A_rr = a_rr->ParallelAssemble();
+
+   ParBilinearForm *a_ir = new ParBilinearForm(fespace);
+   ConstantCoefficient impedance(omega);
+   a_ir->AddBoundaryIntegrator(new BoundaryMassIntegrator(impedance));
+   a_ir->Assemble();
+   a_ir->Finalize();
+   HypreParMatrix *A_ir = a_ir->ParallelAssemble();
+
+   // Direct solve
+   ComplexHypreParMatrix chpm(A_rr, A_ir, false, false);
+   HypreParMatrix *CA = chpm.GetSystemMatrix();
+   PetscLinearSolver * invCA = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   invCA->SetOperator(PetscParMatrix(CA, Operator::PETSC_MATAIJ));
+   trueX = 0.0;
+   invCA->Mult(trueRhs,trueX);
+
+   if (myid == 0)
+   {
+      cout << "Direct solver finished" << endl;
+   }
+
+
+   ParGridFunction *x_Re(new ParGridFunction);
+   ParGridFunction *x_Im(new ParGridFunction);
+
+   x_Re->MakeRef(fespace, x.GetBlock(0), 0);
+   x_Im->MakeRef(fespace, x.GetBlock(1), 0);
+
+   x_Re->Distribute(&(trueX.GetBlock(0)));
+   x_Im->Distribute(&(trueX.GetBlock(1)));
+
+   const int h1_norm_type = 1;
+   double L2error;
+   double H1error;
+
+   double L2err_Re  = x_Re->ComputeL2Error(p_Re);
+   double L2err_Im  = x_Im->ComputeL2Error(p_Im);
+
+   double loc_H1err_Re = x_Re->ComputeH1Error(&p_Re, &grad_Re, &one, 1.0, h1_norm_type);
+   double loc_H1err_Im = x_Im->ComputeH1Error(&p_Im, &grad_Im, &one, 1.0, h1_norm_type);
+   double H1err_Re = GlobalLpNorm(2.0, loc_H1err_Re, MPI_COMM_WORLD);
+   double H1err_Im = GlobalLpNorm(2.0, loc_H1err_Im, MPI_COMM_WORLD);
+      
+   L2error = sqrt(L2err_Re*L2err_Re + L2err_Im*L2err_Im);
+   H1error = sqrt(H1err_Re*H1err_Re + H1err_Im*H1err_Im);
+   if (myid == 0)
+   {
+      cout << " || p_h - p ||_{H^1} = " << H1error <<  endl;
+      cout << " || p_h - p ||_{L^2} = " << L2error <<  endl;
+   }
+
+   // release memory
+   delete b_Re;
+   delete b_Im;
+   delete a_rr;
+   delete a_ir;
+
+   delete fespace;
+   delete fec;
+   delete pmesh;
+   MFEMFinalizePetsc();
+   MPI_Finalize();
+   return 0;
+}
+
+//define exact solution plane wave
+double p_exact_Re(const Vector &x)
+{
+   return cos(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+}
+double p_exact_Im(const Vector &x)
+{
+   return -sin(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+}
+
+//calculate RHS from exact solution f = - \Delta u
+double f_exact_Re(const Vector &x)
+{
+   return 0.0;
+}
+double f_exact_Im(const Vector &x)
+{
+   return 0.0;
+}
+
+void grad_exact_Re(const Vector &x, Vector &grad_Re)
+{
+   grad_Re[0] = -omega / sqrt(3.0) * sin(omega / sqrt(3.0) * (x(0) + x(1) + x(2)));
+   grad_Re[1] = grad_Re[0];
+   grad_Re[2] = grad_Re[1];
+}
+void grad_exact_Im(const Vector &x, Vector &grad_Im)
+{
+
+   grad_Im[0] = -omega / sqrt(3.0) * cos(omega / sqrt(3.0) * (x(0) + x(1) + x(2)));
+   grad_Im[1] = grad_Im[0];
+   grad_Im[2] = grad_Im[1];
+}
+
+//define impedence coefficient: i omega p
+double g_exact_Re(const Vector &x)
+{
+   return omega * sin(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+}
+double g_exact_Im(const Vector &x)
+{
+   return omega * cos(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+}

examples/waves/back/maxwell_real.cpp

@@ -0,0 +1,399 @@
+//                                MFEM Example multigrid-grid Cycle
+//
+// Compile with: make mg_maxwellp
+//
+// Sample runs:  mg_maxwellp -m ../data/one-hex.mesh
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// #define DEFINITE
+
+#ifndef MFEM_USE_PETSC
+#error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+#endif
+
+// Define exact solution
+void E_exact(const Vector & x, Vector & E);
+void f_exact(const Vector & x, Vector & f);
+void get_maxwell_solution(const Vector & x, double E[], double curl2E[]);
+
+int dim;
+double omega;
+int isol = 1;
+
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv); // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid); // Determine process identifier
+   // 1. Parse command-line options.
+   // geometry file
+   // const char *mesh_file = "../data/star.mesh";
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   // visualization flag
+   bool visualization = 1;
+   // number of wavelengths
+   double k = 0.5;
+   // number of mg levels
+   int maxref = 1;
+   // number of initial ref
+   int initref = 1;
+   // PETSC
+   // const char *petscrc_file = "petscrc_direct";
+   const char *petscrc_file = "petscrc_mult_options";
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&maxref, "-maxref", "--maxref",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&isol, "-isol", "--exact",
+                  "Exact solution flag - "
+                  " 1:sinusoidal, 2: point source, 3: plane wave");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+
+   // Angular frequency
+   omega = 2.0*k*M_PI;
+   // omega = k;
+
+   // 2b. We initialize PETSc
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+
+   // 2. Read the mesh from the given mesh file.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // 4. Define a finite element space on the mesh.
+   FiniteElementCollection *fec   = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+
+
+   std::vector<HypreParMatrix*>  P(maxref);
+
+   for (int i = 0; i < maxref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*fespace);
+      pmesh->UniformRefinement();
+      // Update fespace
+      fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      fespace->GetTrueTransferOperator(cfespace, Tr);
+      Tr.SetOperatorOwner(false);
+      Tr.Get(P[i]);
+   }
+
+   ConstantCoefficient muinv(1.0);
+#ifdef DEFINITE
+   ConstantCoefficient sigma(pow(omega, 2));
+#else
+   ConstantCoefficient sigma(-pow(omega, 2));
+#endif
+   // 6. Linear form (i.e RHS b = (f,v) = (1,v))
+   ParLinearForm *b = new ParLinearForm(fespace);
+   VectorFunctionCoefficient f(sdim, f_exact);
+   b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
+   b->Assemble();
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ParBilinearForm *a = new ParBilinearForm(fespace);
+   a->AddDomainIntegrator(new CurlCurlIntegrator(muinv)); // one is the coeff
+   a->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   a->Assemble();
+
+   Array<int> ess_tdof_list;
+   if (pmesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   ParGridFunction x(fespace);
+   VectorFunctionCoefficient E(sdim, E_exact);
+   x.ProjectCoefficient(E);
+
+   HypreParMatrix * A = new HypreParMatrix;
+   Vector B, X;
+
+   a->FormLinearSystem(ess_tdof_list, x, *b, *A, X, B);
+
+   if (myid == 0)
+   {
+      cout << "Size of fine grid system: "
+           << A->GetGlobalNumRows() << " x " << A->GetGlobalNumCols() << endl;
+   }
+
+   chrono.Clear();
+   chrono.Start();
+   GMGSolver M(A, P);
+   M.SetTheta(0.5);
+   M.SetSmootherType(HypreSmoother::Jacobi);
+   chrono.Stop();
+
+   if (myid == 0)
+   {
+      cout << "Construction of MG precond time: " << chrono.RealTime() << endl;
+   }
+
+   int maxit(1000);
+   double rtol(1.e-6);
+   double atol(0.0);
+
+   chrono.Clear();
+   chrono.Start();
+
+   X = 0.0;
+   GMRESSolver gmres(MPI_COMM_WORLD);
+   gmres.SetAbsTol(atol);
+   gmres.SetRelTol(rtol);
+   gmres.SetMaxIter(maxit);
+   gmres.SetOperator(*A);
+   gmres.SetPreconditioner(M);
+   gmres.SetPrintLevel(1);
+   gmres.Mult(B, X);
+
+   chrono.Stop();
+   if (myid == 0)
+   {
+      cout << " GMRES time: " << chrono.RealTime() << endl;
+   }
+
+
+   // chrono.Clear();
+   // chrono.Start();
+
+   // MINRESSolver minres(MPI_COMM_WORLD);
+   // minres.SetAbsTol(atol);
+   // minres.SetRelTol(rtol);
+   // minres.SetMaxIter(maxit);
+   // minres.SetOperator(*A);
+   // minres.SetPreconditioner(M);
+   // minres.SetPrintLevel(1);
+   // minres.Mult(B, X);
+
+   // chrono.Stop();
+   // if (myid == 0)
+   // {
+   //    cout << "MINRES time: " << chrono.RealTime() << endl;
+   // }
+   a->RecoverFEMSolution(X, *b, x);
+
+   int order_quad = max(2, 2 * order + 1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i = 0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double L2Error = x.ComputeL2Error(E, irs);
+   double norm_E = ComputeGlobalLpNorm(2, E, *pmesh, irs);
+
+   if (myid == 0)
+   {
+      cout << "\n GMRES:  || E_h - E || / ||E|| = " << L2Error / norm_E << '\n' << endl;
+   }
+
+
+
+   // Compare with direct solve
+   // X = 0.0;
+
+   // PetscLinearSolver *invA;
+   // invA = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+   // invA->SetOperator(PetscParMatrix(A, Operator::PETSC_MATAIJ));
+   // invA->Mult(B, X);
+
+   // a->RecoverFEMSolution(X, *b, x);
+   // double L2Error_d = x.ComputeL2Error(E);
+   // double norm_E_d = ComputeGlobalLpNorm(2, E, *pmesh, irs);
+
+   // if (myid == 0)
+   // {
+   //    cout << "\n Direct: || E_h - E || / ||E|| = " << L2Error_d / norm_E_d << '\n' << endl;
+   // }
+
+
+   if (visualization)
+   {
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << x << flush;
+   }
+   // ---------------------------------------------------------------------
+
+   for (int i = 0 ; i < maxref; i++)
+   {
+      delete P[i];
+   }
+   delete A;
+   delete a;
+   delete b;
+   delete fec;
+   delete fespace;
+   delete pmesh;
+   MFEMFinalizePetsc();
+   MPI_Finalize();
+   return 0;
+}
+
+
+//define exact solution
+void E_exact(const Vector &x, Vector &E)
+{
+   if (dim != 3)
+   {
+      cout << "2D not set up yet: " << endl;
+      exit(0);
+   }
+   double curl2E[3];
+   get_maxwell_solution(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+// f = curl (mu curl E ) + omega^2*E
+void f_exact(const Vector &x, Vector &f)
+{
+   if (dim != 3)
+   {
+      cout << "2D not set up yet: " << endl;
+      exit(0);
+   }
+   double E[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curl2E);
+
+   // curl ( curl E) + omega^2 E = f
+   double coeff;
+#ifdef DEFINITE
+   coeff = omega * omega;
+#else
+   coeff = -omega * omega;
+#endif
+
+   f(0) = curl2E[0] + coeff * E[0];
+   f(1) = curl2E[1] + coeff * E[1];
+   f(2) = curl2E[2] + coeff * E[2];
+}
+
+void get_maxwell_solution(const Vector & x, double E[], double curl2E[])
+{
+
+   if (isol == 0) // polynomial
+   {
+      // Polynomial vanishing on the boundary
+      E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+      E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+      E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+      //
+      curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+      curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+      curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+
+   }
+
+   else if (isol == 1) // sinusoidal
+   {
+      E[0] = sin(omega * x[1]);
+      E[1] = sin(omega * x[2]);
+      E[2] = sin(omega * x[0]);
+
+      curl2E[0] = omega * omega * E[0];
+      curl2E[1] = omega * omega * E[1];
+      curl2E[2] = omega * omega * E[2];
+
+   }
+   else if (isol == 2) //point source
+   {
+      // shift to avoid singularity
+      double x0 = x(0) + 0.1;
+      double x1 = x(1) + 0.1;
+      double x2 = x(2) + 0.1;
+      //
+      double r = sqrt(x0 * x0 + x1 * x1 + x2 * x2);
+
+      E[0] = cos(omega * r);
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+      double r_x = x0 / r;
+      double r_y = x1 / r;
+      double r_z = x2 / r;
+      double r_xy = -(r_x / r) * r_y;
+      double r_xz = -(r_x / r) * r_z;
+      double r_yx = r_xy;
+      double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+      double r_zx = r_xz;
+      double r_zz = (1.0 / r) * (1.0 - r_z * r_z);
+
+      curl2E[0] = omega * ((r_yy + r_zz) * sin(omega * r) +
+                           (omega * r_y * r_y + omega * r_z * r_z) * cos(omega * r));
+      curl2E[1] = -omega * (r_yx * sin(omega * r) + omega * r_y * r_x * cos(omega * r));
+      curl2E[2] = -omega * (r_zx * sin(omega * r) + omega * r_z * r_x * cos(omega * r));
+
+   }
+   else if (isol == 3) // plane wave
+   {
+      E[0] = cos(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+      curl2E[0] = 2.0 * omega * omega * E[0] / 3.0;
+      curl2E[1] = -omega * omega * E[0] / 3.0;
+      curl2E[2] = -omega * omega * E[0] / 3.0;
+   }
+
+}

examples/waves/ddoper.hpp

@@ -0,0 +1,204 @@
+
+#include "mfem.hpp"
+
+using namespace mfem;
+using namespace std;
+
+
+hypre_CSRMatrix* GetHypreParMatrixData(const HypreParMatrix & hypParMat)
+{
+  // First cast the parameter to a hypre_ParCSRMatrix
+  hypre_ParCSRMatrix * parcsr_op =
+    (hypre_ParCSRMatrix *)const_cast<HypreParMatrix&>(hypParMat);
+
+  MFEM_ASSERT(parcsr_op != NULL,"STRUMPACK: const_cast failed in SetOperator");
+
+  // Create the CSRMatrixMPI A_ by borrowing the internal data from a hypre_CSRMatrix.
+  return hypre_MergeDiagAndOffd(parcsr_op);
+}
+
+
+// Row and column offsets are assumed to be the same.
+// Array offsets stores process-local offsets with respect to the blocks. Process offsets are not included.
+HypreParMatrix* CreateHypreParMatrixFromBlocks(MPI_Comm comm, Array<int> const& offsets, Array2D<HypreParMatrix*> const& blocks,
+                      Array2D<double> const& coefficient)
+{
+  const int numBlocks = offsets.Size() - 1;
+  
+  const int num_loc_rows = offsets[numBlocks];
+
+  int nprocs, rank;
+  MPI_Comm_rank(comm, &rank);
+  MPI_Comm_size(comm, &nprocs);
+
+  std::vector<int> all_num_loc_rows(nprocs);
+  std::vector<int> procOffsets(nprocs);
+  std::vector<std::vector<int> > all_block_num_loc_rows(numBlocks);
+  std::vector<std::vector<int> > blockProcOffsets(numBlocks);
+  
+  MPI_Allgather(&num_loc_rows, 1, MPI_INT, all_num_loc_rows.data(), 1, MPI_INT, comm);
+
+  for (int j=0; j<numBlocks; ++j)
+    {
+      all_block_num_loc_rows[j].resize(nprocs);
+      blockProcOffsets[j].resize(nprocs);
+      
+      const int blockNumRows = offsets[j+1] - offsets[j];
+      MPI_Allgather(&blockNumRows, 1, MPI_INT, all_block_num_loc_rows[j].data(), 1, MPI_INT, comm);
+
+      blockProcOffsets[j][0] = 0;
+      for (int i=0; i<nprocs-1; ++i)
+   blockProcOffsets[j][i+1] = blockProcOffsets[j][i] + all_block_num_loc_rows[j][i];
+    }
+  
+  int first_loc_row = 0;
+  int glob_nrows = 0;
+  procOffsets[0] = 0;
+  for (int i=0; i<nprocs; ++i)
+    {
+      glob_nrows += all_num_loc_rows[i];
+      if (i < rank)
+   first_loc_row += all_num_loc_rows[i];
+
+      if (i < nprocs-1)
+   procOffsets[i+1] = procOffsets[i] + all_num_loc_rows[i];
+    }
+    
+  const int glob_ncols = glob_nrows;
+
+  std::vector<int> opI(num_loc_rows+1);
+  std::vector<int> cnt(num_loc_rows);
+
+  for (int i=0; i<num_loc_rows; ++i)
+    {
+      opI[i] = 0;
+      cnt[i] = 0;
+    }
+
+  opI[num_loc_rows] = 0;
+  
+  Array2D<hypre_CSRMatrix*> csr_blocks(numBlocks, numBlocks);
+  
+  // Loop over all blocks, to determine nnz for each row.
+  for (int i=0; i<numBlocks; ++i)
+    {
+      for (int j=0; j<numBlocks; ++j)
+   {
+     if (blocks(i, j) == NULL)
+       {
+         csr_blocks(i, j) = NULL;
+       }
+     else
+       {
+         csr_blocks(i, j) = GetHypreParMatrixData(*(blocks(i, j)));
+
+         const int nrows = csr_blocks(i, j)->num_rows;
+
+         for (int k=0; k<nrows; ++k)
+      {
+        const int rowg = offsets[i] + k;
+        //(*(leftInjection(i, j)))[k]
+        opI[rowg + 1] += csr_blocks(i, j)->i[k+1] - csr_blocks(i, j)->i[k];
+      }
+       }
+   }
+    }
+
+  // Now opI[i] is nnz for row i-1. Do a partial sum to get offsets.
+  for (int i=0; i<num_loc_rows; ++i)
+    opI[i+1] += opI[i];
+
+  const int nnz = opI[num_loc_rows];
+
+  std::vector<HYPRE_Int> opJ(nnz);
+  std::vector<double> data(nnz);
+
+  // Loop over all blocks, to set matrix data.
+  for (int i=0; i<numBlocks; ++i)
+    {
+      for (int j=0; j<numBlocks; ++j)
+   {
+     if (csr_blocks(i, j) != NULL)
+       {
+         const int nrows = csr_blocks(i, j)->num_rows;
+         const double coef = coefficient(i, j);
+
+        //  const bool failure = (nrows != offsets[i+1] - offsets[i]);
+         
+         MFEM_VERIFY(nrows == offsets[i+1] - offsets[i], "");
+         
+         for (int k=0; k<nrows; ++k)
+      {
+        const int rowg = offsets[i] + k;  // process-local row
+        const int nnz_k = csr_blocks(i, j)->i[k+1] - csr_blocks(i, j)->i[k];
+        const int osk = csr_blocks(i, j)->i[k];
+        
+        for (int l=0; l<nnz_k; ++l)
+          {
+            // Find the column process offset for the block.
+            const int bcol = csr_blocks(i, j)->j[osk + l];
+            int bcolproc = 0;
+
+            for (int p=1; p<nprocs; ++p)
+         {
+           if (blockProcOffsets[j][p] > bcol)
+             {
+               bcolproc = p-1;
+               break;
+             }
+         }
+
+            if (blockProcOffsets[j][nprocs - 1] <= bcol)
+         bcolproc = nprocs - 1;
+
+            const int colg = procOffsets[bcolproc] + offsets[j] + (bcol - blockProcOffsets[j][bcolproc]);
+
+            if (colg < 0)
+         cout << "BUG, negative global column index" << endl;
+            
+            opJ[opI[rowg] + cnt[rowg]] = colg;
+            data[opI[rowg] + cnt[rowg]] = coef * csr_blocks(i, j)->data[osk + l];
+            cnt[rowg]++;
+          }
+      }
+       }
+   }
+    }
+
+  bool cntCheck = true;
+  for (int i=0; i<num_loc_rows; ++i)
+    {
+      if (cnt[i] != opI[i+1] - opI[i])
+   cntCheck = false;
+    }
+
+  MFEM_VERIFY(cntCheck, "");
+
+  for (int i=0; i<numBlocks; ++i)
+    {
+      for (int j=0; j<numBlocks; ++j)
+   {
+     if (csr_blocks(i, j) != NULL)
+       {
+         hypre_CSRMatrixDestroy(csr_blocks(i, j));
+       }
+   }
+    }
+  
+  std::vector<HYPRE_Int> rowStarts2(2);
+  rowStarts2[0] = first_loc_row;
+  rowStarts2[1] = first_loc_row + all_num_loc_rows[rank];
+
+  HYPRE_Int minJ = opJ[0];
+  HYPRE_Int maxJ = opJ[0];
+  for (int i=0; i<nnz; ++i)
+    {
+      minJ = std::min(minJ, opJ[i]);
+      maxJ = std::max(maxJ, opJ[i]);
+    }
+  
+  HypreParMatrix *hmat = new HypreParMatrix(comm, num_loc_rows, glob_nrows, glob_ncols, (int*) opI.data(), (HYPRE_Int*) opJ.data(), (double*) data.data(),
+                   (HYPRE_Int*) rowStarts2.data(), (HYPRE_Int*) rowStarts2.data());
+  
+  return hmat;
+}

examples/waves/helmholtzp.cpp

@@ -0,0 +1,416 @@
+//
+// Compile with: make helmholtz
+//
+// Sample runs:  helmholtz -m ../data/one-hex.mesh
+//               helmholtz -m ../data/fichera.mesh
+//               helmholtz -m ../data/fichera-mixed.mesh
+//
+// Description:  This example code demonstrates the use of MFEM to define a
+//               simple finite element discretization of the Helmholtz problem
+//               -Delta p - omega^2 p = 1 with impedance boundary conditiones.
+//
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// Exact solution and r.h.s., see below for implementation.
+void get_helmholtz_solution_Re(const Vector &x, double & p, double dp[], double & d2p);
+void get_helmholtz_solution_Im(const Vector &x, double & p, double dp[], double & d2p);
+double p_exact_Re(const Vector &x);
+double p_exact_Im(const Vector &x);
+double f_exact_Re(const Vector &x);
+double f_exact_Im(const Vector &x);
+double g_exact_Re(const Vector &x);
+double g_exact_Im(const Vector &x);
+void grad_exact_Re(const Vector &x, Vector &grad_Re);
+void grad_exact_Im(const Vector &x, Vector &grad_Im);
+
+int dim;
+double omega;
+double complex_shift;
+int isol = 1;
+
+#ifndef MFEM_USE_PETSC
+#error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+#endif
+
+int main(int argc, char *argv[])
+{
+
+   // 1. Initialise MPI
+   int num_procs, myid;
+   MPI_Init(&argc, &argv);                    // Initialise MPI
+   MPI_Comm_size(MPI_COMM_WORLD, &num_procs); //total number of processors available
+   MPI_Comm_rank(MPI_COMM_WORLD, &myid);      // Determine process identifier
+
+   //-----------------------------------------------------------------------------
+
+   // 2. Parse command-line options.
+   // geometry file
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   bool visualization = 1;
+   // number of wavelengths
+   double k = 0.5;
+   // number of mg levels
+   int maxref = 1;
+   // number of initial ref
+   int initref = 1;
+   // complex shift
+   complex_shift = 0.0;
+
+   // PETSC
+   // const char *petscrc_file = "";
+   const char *petscrc_file = "petscrc_mult_options";
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&isol, "-isol", "--exact",
+                  "Exact solution flag - 0:polynomial, 1: plane wave");               
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&complex_shift, "-cs", "--complex_shift",
+                  "Complex shift");                    
+   args.AddOption(&maxref, "-ref", "--maxref",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&petscrc_file, "-petscopts", "--petscopts",
+                  "PetscOptions file to use.");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if (myid == 0)
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if (myid == 0)
+   {
+      args.PrintOptions(cout);
+   }
+   // Angular frequency
+   omega = 2.0 * M_PI * k;
+
+   // 2b. Initialize PETSc
+   MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+   //-----------------------------------------------------------------------------
+
+   // 3. Read the mesh from the given mesh file.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   // Mesh *mesh = new Mesh(1, 1, Element::QUADRILATERAL, true, 1.0, 1.0, false);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // ----------------------------------------------------------------------------
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   // ----------------------------------------------------------------------------
+
+   // 5. Define a parallel mesh and delete the serial mesh.
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // ----------------------------------------------------------------------------
+
+   // 6. Define a finite element space on the mesh.
+   FiniteElementCollection *fec = new H1_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+
+   std::vector<HypreParMatrix*>  P(maxref);
+
+   for (int i = 0; i < maxref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*fespace);
+      pmesh->UniformRefinement();
+      fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      fespace->GetTrueTransferOperator(cfespace, Tr);
+      HypreParMatrix * Paux;
+      Tr.Get(Paux);
+      P[i] = new HypreParMatrix(*Paux);
+   }
+
+   // 6. Set up the linear form (Real and Imaginary part)
+   FunctionCoefficient f_Re(f_exact_Re);
+   FunctionCoefficient g_Re(g_exact_Re);
+   VectorFunctionCoefficient grad_Re(sdim, grad_exact_Re);
+   FunctionCoefficient f_Im(f_exact_Im);
+   FunctionCoefficient g_Im(g_exact_Im);
+   VectorFunctionCoefficient grad_Im(sdim, grad_exact_Im);
+
+   // ParLinearForm *b_Re(new ParLinearForm);
+   ParComplexLinearForm b(fespace, ComplexOperator::HERMITIAN);
+   b.AddDomainIntegrator(new DomainLFIntegrator(f_Re),new DomainLFIntegrator(f_Im));
+   b.AddBoundaryIntegrator(new BoundaryNormalLFIntegrator(grad_Re),
+                           new BoundaryNormalLFIntegrator(grad_Im));
+   b.AddBoundaryIntegrator(new BoundaryLFIntegrator(g_Re),new BoundaryLFIntegrator(g_Im));
+   b.real().Vector::operator=(0.0);
+   b.imag().Vector::operator=(0.0);                        
+   b.Assemble();
+
+   // 7. Set up the bilinear form (Real and Imaginary part)
+   ConstantCoefficient one(1.0);
+   ConstantCoefficient sigma(-pow(omega, 2));
+   ConstantCoefficient alpha(complex_shift);
+
+   ParSesquilinearForm a(fespace,ComplexOperator::HERMITIAN);
+   ConstantCoefficient impedance(omega);
+   a.AddDomainIntegrator(new DiffusionIntegrator(one),NULL);
+   a.AddDomainIntegrator(new MassIntegrator(sigma),new MassIntegrator(alpha));
+   a.AddBoundaryIntegrator(NULL,new BoundaryMassIntegrator(impedance));
+   a.Assemble();
+   a.Finalize();
+
+   Array<int> ess_tdof_list;
+   if (pmesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+      ess_bdr = 0;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   // Solution grid function
+   ParComplexGridFunction p_gf(fespace);
+   FunctionCoefficient p_Re(p_exact_Re);
+   FunctionCoefficient p_Im(p_exact_Im);
+   p_gf.ProjectCoefficient(p_Re,p_Im);
+
+   OperatorHandle Ah;
+   Vector X, B;
+   a.FormLinearSystem(ess_tdof_list, p_gf, b, Ah, X, B);
+
+   ComplexHypreParMatrix * AZ = Ah.As<ComplexHypreParMatrix>();
+   HypreParMatrix * A = AZ->GetSystemMatrix();
+
+   if (myid == 0)
+   {
+      cout << "Size of fine grid system: "
+           << A->GetGlobalNumRows() << " x " << A->GetGlobalNumCols() << endl;
+   }
+
+
+   ComplexGMGSolver M(AZ, P,ComplexGMGSolver::CoarseSolver::PETSC);
+   M.SetTheta(0.5);
+   M.SetSmootherType(HypreSmoother::Jacobi);
+
+   int maxit(5000);
+   double rtol(1.e-12);
+   double atol(0.0);
+
+   X = 0.0;
+   GMRESSolver gmres(MPI_COMM_WORLD);
+   gmres.SetAbsTol(atol);
+   gmres.SetRelTol(rtol);
+   gmres.SetMaxIter(maxit);
+   gmres.SetOperator(*AZ);
+   gmres.SetPreconditioner(M);
+   gmres.SetPrintLevel(1);
+   gmres.Mult(B, X);
+
+   a.RecoverFEMSolution(X,B,p_gf);
+
+   int order_quad = max(2, 2*order+1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i=0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   const int h1_norm_type = 1;
+   double L2error;
+   double H1error;
+
+   double L2err_Re  = p_gf.real().ComputeL2Error(p_Re);
+   double L2err_Im  = p_gf.imag().ComputeL2Error(p_Im);
+
+   double loc_H1err_Re = p_gf.real().ComputeH1Error(&p_Re, &grad_Re, &one, 1.0, h1_norm_type);
+   double loc_H1err_Im = p_gf.imag().ComputeH1Error(&p_Im, &grad_Im, &one, 1.0, h1_norm_type);
+   double H1err_Re = GlobalLpNorm(2.0, loc_H1err_Re, MPI_COMM_WORLD);
+   double H1err_Im = GlobalLpNorm(2.0, loc_H1err_Im, MPI_COMM_WORLD);
+   // double norm_Re = ComputeGlobalLpNorm(2, p_Re, *pmesh, irs);
+   // double norm_Im = ComputeGlobalLpNorm(2, p_Im, *pmesh, irs);
+      
+   L2error = sqrt(L2err_Re*L2err_Re + L2err_Im*L2err_Im);
+   H1error = sqrt(H1err_Re*H1err_Re + H1err_Im*H1err_Im);
+   // double L2norm = sqrt(norm_Re*norm_Re + norm_Im*norm_Im);
+   if (myid == 0)
+   {
+      cout << " || p_h - p ||_{H^1} = " << H1error <<  endl;
+      cout << " || p_h - p ||_{L^2} = " << L2error <<  endl;
+      // cout << " || p_h - p ||_{L^2}/||p||_{L^2} = " << L2error/L2norm <<  endl;
+   }
+
+   delete fespace;
+   delete fec;
+   delete pmesh;
+   MFEMFinalizePetsc();
+   MPI_Finalize();
+   return 0;
+}
+
+//define exact solution plane wave
+
+void get_helmholtz_solution_Re(const Vector &x, double & p, double dp[], double & d2p)
+{
+
+   if (isol == 0) // polynomial
+   {  
+      p = x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+      dp[0] = (1.0 - 2.0 *x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+      dp[1] = (1.0 - 2.0 *x[1]) * x[0]*(1.0 - x[0]) * x[2]*(1.0 - x[2]);
+      dp[2] = (1.0 - 2.0 *x[2]) * x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]);
+      d2p = -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[1]) * x[1] 
+            -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[2]) * x[2] 
+            -2.0*(-1.0 + x[1]) * x[1] * (-1.0 + x[2]) * x[2];
+   }
+   else
+   {
+      double alpha;
+      if (dim == 2)
+      {
+         alpha = omega/sqrt(2);
+         p = cos(alpha * ( x(0) + x(1) ) );
+         dp[0] = -alpha * sin(alpha * ( x(0) + x(1) ) );
+         dp[1] = dp[0];
+         d2p = -2.0 * alpha * alpha * p;
+      }
+      else 
+      {
+         alpha = omega/sqrt(3);
+         p = cos(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[0] = -alpha * sin(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[1] = dp[0];
+         dp[2] = dp[0];
+         d2p = -3.0 * alpha * alpha * p;
+      }
+   }
+   
+}
+
+void get_helmholtz_solution_Im(const Vector &x, double & p, double dp[], double & d2p)
+{
+
+   if (isol == 0) // polynomial
+   {
+      p = x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+      dp[0] = (1.0 - 2.0 *x[0]) * x[1]*(1.0 - x[1]) * x[2]*(1.0 - x[2]);
+      dp[1] = (1.0 - 2.0 *x[1]) * x[0]*(1.0 - x[0]) * x[2]*(1.0 - x[2]);
+      dp[2] = (1.0 - 2.0 *x[2]) * x[0]*(1.0 - x[0]) * x[1]*(1.0 - x[1]);
+      d2p = -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[1]) * x[1] 
+            -2.0*(-1.0 + x[0]) * x[0] * (-1.0 + x[2]) * x[2] 
+            -2.0*(-1.0 + x[1]) * x[1] * (-1.0 + x[2]) * x[2];
+   }
+   else
+   {
+      double alpha;
+      if (dim == 2)
+      {
+         alpha = omega/sqrt(2);
+         p = -sin(alpha * ( x(0) + x(1) ) );
+         dp[0] = -alpha * cos(alpha * ( x(0) + x(1) ) );
+         dp[1] = dp[0];
+         d2p = -2.0 * alpha * alpha * p;
+      }
+      else 
+      {
+         alpha = omega/sqrt(3);
+         p = -sin(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[0] = -alpha * cos(alpha * ( x(0) + x(1) + x(2) ) );
+         dp[1] = dp[0];
+         dp[2] = dp[0];
+         d2p = -3.0 * alpha * alpha * p;
+      }   
+   }
+}
+
+
+double p_exact_Re(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Re(x, p, dp, d2p);
+   return p;
+}
+double p_exact_Im(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Im(x, p, dp, d2p);
+   return p;
+}
+
+//calculate RHS from exact solution f = - \Delta u
+double f_exact_Re(const Vector &x)
+{
+   double p_re, d2p_re, p_im, d2p_im;
+   double dp_re[3], dp_im[3];
+
+   get_helmholtz_solution_Re(x, p_re, dp_re, d2p_re);
+   get_helmholtz_solution_Im(x, p_im, dp_im, d2p_im);
+   return -d2p_re - omega * omega * p_re - complex_shift*p_im ;
+}
+double f_exact_Im(const Vector &x)
+{
+   double p_re, d2p_re, p_im, d2p_im;
+   double dp_re[3], dp_im[3];
+
+   get_helmholtz_solution_Re(x, p_re, dp_re, d2p_re);
+   get_helmholtz_solution_Im(x, p_im, dp_im, d2p_im);
+   return -d2p_im - omega * omega * p_im + complex_shift*p_re;
+}
+
+void grad_exact_Re(const Vector &x, Vector &dp)
+{
+
+   double alpha = omega/sqrt(3);
+   double p, d2p;
+   get_helmholtz_solution_Re(x, p, dp, d2p);
+}
+void grad_exact_Im(const Vector &x, Vector &dp)
+{
+   double alpha = omega/sqrt(3);
+   double p, d2p;
+   get_helmholtz_solution_Im(x, p, dp, d2p);
+}
+
+//define impedence coefficient: i omega p
+double g_exact_Re(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Im(x, p, dp, d2p);
+
+   return -omega * p;
+}
+double g_exact_Im(const Vector &x)
+{
+   double p, d2p;
+   double dp[3];
+   get_helmholtz_solution_Re(x, p, dp, d2p);
+
+   return omega * p;
+}

examples/waves/makefile

@@ -0,0 +1,69 @@
+# Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at the
+# Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights reserved.
+# See file COPYRIGHT for details.
+#
+# This file is part of the MFEM library. For more information and source code
+# availability see http://mfem.org.
+#
+# MFEM is free software; you can redistribute it and/or modify it under the
+# terms of the GNU Lesser General Public License (as published by the Free
+# Software Foundation) version 2.1 dated February 1999.
+
+# Use the MFEM build directory
+MFEM_DIR ?= ../..
+MFEM_BUILD_DIR ?= ../..
+SRC = $(if $(MFEM_DIR:../..=),$(MFEM_DIR)/examples/waves/,)
+CONFIG_MK = $(MFEM_BUILD_DIR)/config/config.mk
+# Use the MFEM install directory
+# MFEM_INSTALL_DIR = ../../mfem
+# CONFIG_MK = $(MFEM_INSTALL_DIR)/share/mfem/config.mk
+
+MFEM_LIB_FILE = mfem_is_not_built
+-include $(CONFIG_MK)
+
+SEQ_EXAMPLES =
+PAR_EXAMPLES = helmholtzp maxwellp DPG_prim LS_Maxwellp maxwell_real
+ifeq ($(MFEM_USE_MPI),NO)
+   EXAMPLES = $(SEQ_EXAMPLES)
+else
+   EXAMPLES = $(PAR_EXAMPLES)
+endif
+RC_FILES = $(patsubst $(SRC)%,%,$(wildcard $(SRC)rc_*))
+
+.SUFFIXES:
+.SUFFIXES: .o .cpp .mk
+.PHONY: all clean clean-build clean-exec
+
+# Remove built-in rule
+%: %.cpp
+
+# Replace the default implicit rule for *.cpp files
+%: $(SRC)%.cpp $(MFEM_LIB_FILE) $(CONFIG_MK)
+	$(MFEM_CXX) $(MFEM_FLAGS) $< -o $@ $(MFEM_LIBS)
+
+all: $(EXAMPLES)
+
+# Examples depend on their corresponding rc_* files:
+make-rc-rule = $(1): | $(filter rc_$(1)%,$(RC_FILES))
+$(foreach ex,$(EXAMPLES),$(eval $(call make-rc-rule,$(ex))))
+
+# Rules to copy the rc_* files when building out-of-source:
+ifneq ($(SRC),)
+$(RC_FILES): %: $(SRC)%
+	cp -pf $(<) .
+endif
+
+# Generate an error message if the MFEM library is not built and exit
+$(MFEM_LIB_FILE):
+	$(error The MFEM library is not built)
+
+clean: clean-build clean-exec
+
+clean-build:
+	rm -f *.o *~ $(SEQ_EXAMPLES) $(PAR_EXAMPLES)
+	rm -rf *.dSYM *.TVD.*breakpoints
+
+clean-exec:
+	@rm -rf mesh.* sol.* sol_p.* sol_u.* Example5*
+	@rm -f ex9-mesh.* ex9-init.* ex9-final.* Example9*
+	@rm -f deformed.* velocity.* elastic_energy.*

examples/waves/maxwell_real.cpp

@@ -0,0 +1,480 @@
+//                                MFEM Example multigrid-grid Cycle
+//
+// Compile with: make mg_maxwellp
+//
+// Sample runs:  mg_maxwellp -m ../data/one-hex.mesh
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+
+using namespace std;
+using namespace mfem;
+
+// #define DEFINITE
+
+// #ifndef MFEM_USE_PETSC
+// #error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+// #endif
+
+// Define exact solution
+void E_exact(const Vector & x, Vector & E);
+void f_exact(const Vector & x, Vector & f);
+void get_maxwell_solution(const Vector & x, double E[], double curl2E[]);
+
+int dim;
+double omega;
+int isol = 1;
+
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Initialise MPI
+   MPI_Session mpi(argc, argv);
+   // 1. Parse command-line options.
+   // geometry file
+   // const char *mesh_file = "../data/star.mesh";
+   const char *mesh_file = "../../data/one-hex.mesh";
+   // finite element order of approximation
+   int order = 1;
+   // static condensation flag
+   bool static_cond = false;
+   // visualization flag
+   bool visualization = 1;
+   // number of wavelengths
+   double k = 0.5;
+   // number of mg levels
+   int maxref = 1;
+   // number of initial ref
+   int initref = 1;
+   // solver
+   int solver = 1;
+   // PETSC
+   // const char *petscrc_file = "petscrc_direct";
+   const char *petscrc_file = "petscrc_mult_options";
+   // optional command line inputs
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree) or -1 for"
+                  " isoparametric space.");
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths.");
+   args.AddOption(&maxref, "-maxref", "--maxref",
+                  "Number of Refinements.");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&isol, "-isol", "--exact",
+                  "Exact solution flag - "
+                  " 1:sinusoidal, 2: point source, 3: plane wave");
+   args.AddOption(&static_cond, "-sc", "--static-condensation", "-no-sc",
+                  "--no-static-condensation", "Enable static condensation.");
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization.");
+   args.AddOption(&solver, "-s", "--solver",
+                  "Solver: 1 - GMG-GMRES, 2 - PETSC, 3 - SUPERLU, 4 - STRUMPACK, 5-HSS-GMRES");                       
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if ( mpi.Root() )
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if ( mpi.Root() )
+   {
+      args.PrintOptions(cout);
+   }
+
+   enum SolverType
+   {
+      INVALID_SOL = -1,
+      GMG_GMRES   =  1,
+      PETSC       =  2,
+      SUPERLU     =  3,
+      STRUMPACK   =  4,
+      HSS_GMRES   =  5,
+   };
+
+
+   // Angular frequency
+   omega = 2.0*k*M_PI;
+   // omega = k;
+
+   // 2. Read the mesh from the given mesh file.
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+   int sdim = mesh->SpaceDimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // 4. Define a finite element space on the mesh.
+   FiniteElementCollection *fec   = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *fespace = new ParFiniteElementSpace(pmesh, fec);
+
+
+   std::vector<HypreParMatrix*>  P(maxref);
+
+   for (int i = 0; i < maxref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*fespace);
+      pmesh->UniformRefinement();
+      // Update fespace
+      fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      fespace->GetTrueTransferOperator(cfespace, Tr);
+      Tr.SetOperatorOwner(false);
+      Tr.Get(P[i]);
+   }
+
+   ConstantCoefficient muinv(1.0);
+#ifdef DEFINITE
+   ConstantCoefficient sigma(pow(omega, 2));
+#else
+   ConstantCoefficient sigma(-pow(omega, 2));
+#endif
+   // 6. Linear form (i.e RHS b = (f,v) = (1,v))
+   ParLinearForm *b = new ParLinearForm(fespace);
+   VectorFunctionCoefficient f(sdim, f_exact);
+   b->AddDomainIntegrator(new VectorFEDomainLFIntegrator(f));
+   b->Assemble();
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ParBilinearForm *a = new ParBilinearForm(fespace);
+   a->AddDomainIntegrator(new CurlCurlIntegrator(muinv)); // one is the coeff
+   a->AddDomainIntegrator(new VectorFEMassIntegrator(sigma));
+   a->Assemble();
+
+   Array<int> ess_tdof_list;
+   if (pmesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   ParGridFunction x(fespace);
+   VectorFunctionCoefficient E(sdim, E_exact);
+   x.ProjectCoefficient(E);
+
+   HypreParMatrix * A = new HypreParMatrix;
+   Vector B, X;
+
+   a->FormLinearSystem(ess_tdof_list, x, *b, *A, X, B);
+
+   if ( mpi.Root() )
+   {
+      cout << "Size of fine grid system: "
+           << A->GetGlobalNumRows() << " x " << A->GetGlobalNumCols() << endl;
+   }
+
+   chrono.Clear();
+   chrono.Start();
+   switch((SolverType)solver)
+   {
+      case GMG_GMRES: 
+      {
+         if(mpi.Root()) {cout<< "Solver choice: GMG_GMRES" << endl;}
+#ifdef MFEM_USE_PETSC
+      // MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+#endif            
+         GMGSolver M(A, P, GMGSolver::CoarseSolver::PETSC);
+         // GMGSolver M(A, P, GMGSolver::CoarseSolver::SUPERLU);
+         M.SetTheta(0.5);
+         M.SetSmootherType(HypreSmoother::Jacobi);
+
+         int maxit(5000);
+         double rtol(1.e-6);
+         double atol(0.0);
+         X = 0.0;
+         GMRESSolver gmres(MPI_COMM_WORLD);
+         gmres.SetAbsTol(atol);
+         gmres.SetRelTol(rtol);
+         gmres.SetMaxIter(maxit);
+         gmres.SetOperator(*A);
+         gmres.SetPreconditioner(M);
+         gmres.SetPrintLevel(1);
+         gmres.Mult(B,X);
+#ifdef MFEM_USE_PETSC
+         // MFEMFinalizePetsc();
+#endif         
+      }   
+         break;
+      case PETSC:
+          {
+#ifndef MFEM_USE_PETSC
+         MFEM_ABORT("Invalid choice of Solver. MFEM is not linked with PETSC");
+#else     
+         if(mpi.Root()) {cout<< "Solver choice: PETSC" << endl;}
+         MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+         PetscLinearSolver * invA = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+         PetscParMatrix *PA = new PetscParMatrix(A, Operator::PETSC_MATAIJ);
+         invA->SetOperator(*PA);
+         invA->Mult(B,X);
+         delete PA;
+         MFEMFinalizePetsc();
+#endif         
+      }   
+         break;
+      case SUPERLU:
+      {
+#ifndef MFEM_USE_SUPERLU
+         MFEM_ABORT("Invalid choice of Solver. MFEM is not linked with STRUMPACK");
+#else  
+         if(mpi.Root()) {cout<< "Solver choice: SuperLU" << endl;}
+         SuperLURowLocMatrix *SA = new SuperLURowLocMatrix(*A);
+         SuperLUSolver * superlu = new SuperLUSolver(MPI_COMM_WORLD);
+         // superlu->SetPrintStatistics(true);
+         // superlu->SetSymmetricPattern(false);
+         superlu->SetColumnPermutation(superlu::PARMETIS);
+         superlu->SetOperator(*SA);
+         superlu->Mult(B,X);
+         delete SA;
+         delete superlu;
+#endif         
+      }   
+         break;
+         case STRUMPACK:
+      {   
+#ifndef MFEM_USE_STRUMPACK
+         MFEM_ABORT("Invalid choice of Solver. MFEM is not linked with STRUMPACK");
+#else     
+         if(mpi.Root()) {cout<< "Solver choice: STRUMPACK" << endl;}
+         STRUMPACKRowLocMatrix *SA = new STRUMPACKRowLocMatrix(*A);
+         STRUMPACKSolver * strumpack = new STRUMPACKSolver(argc, argv, MPI_COMM_WORLD);
+         strumpack->SetPrintFactorStatistics(false);
+         strumpack->SetPrintSolveStatistics(true);
+         // strumpack->SetHSS(true);
+         // strumpack->SetHssAbsTol(0.0);
+         // strumpack->SetHssRelTol(1e-4);
+         // strumpack->SetAbsTol(0.0);
+         // strumpack->SetRelTol(1e-6);
+         strumpack->SetKrylovSolver(strumpack::KrylovSolver::DIRECT);
+         // strumpack->SetKrylovSolver(strumpack::KrylovSolver::AUTO);
+         strumpack->SetReorderingStrategy(strumpack::ReorderingStrategy::METIS);
+         strumpack->DisableMatching();
+         strumpack->SetOperator(*SA);
+         strumpack->SetFromCommandLine();
+         strumpack->Mult(B, X);
+         delete SA;
+         delete strumpack;
+#endif         
+      }     
+         break  ;   
+      case HSS_GMRES:
+      {   
+#ifndef MFEM_USE_STRUMPACK
+         MFEM_ABORT("Invalid choice of Solver. MFEM is not linked with PETSC");
+#else     
+         if(mpi.Root()) {cout<< "Solver choice: STRUMPACK" << endl;}
+         STRUMPACKRowLocMatrix *SA = new STRUMPACKRowLocMatrix(*A);
+         STRUMPACKSolver * prec = new STRUMPACKSolver(argc, argv, MPI_COMM_WORLD);
+         prec->SetPrintFactorStatistics(true);
+         prec->SetPrintSolveStatistics(false);
+         prec->SetHSS(false);
+         prec->SetHssAbsTol(0.0);
+         prec->SetHssRelTol(0.0);
+         // prec->SetBlr(true);
+         // prec->SetBlrAbsTol(0.1);
+         // prec->SetBlrRelTol(0.1);
+         prec->SetKrylovSolver(strumpack::KrylovSolver::DIRECT);
+         prec->SetReorderingStrategy(strumpack::ReorderingStrategy::METIS);
+         prec->DisableMatching();
+         prec->SetOperator(*SA);
+         prec->SetFromCommandLine();
+         
+         int maxit(50);
+         double rtol(1.e-6);
+         double atol(0.0);
+         GMRESSolver gmres(MPI_COMM_WORLD);
+         gmres.SetAbsTol(atol);
+         gmres.SetRelTol(rtol);
+         gmres.SetMaxIter(maxit);
+         gmres.SetOperator(*A);
+         gmres.SetPreconditioner(*prec);
+         gmres.SetPrintLevel(1);
+         gmres.Mult(B,X);
+
+         delete SA;
+         delete prec;
+#endif
+      }     
+         break  ;   
+
+      default: 
+         if(mpi.Root()) {cout<< "Solver choice not valid. Problem not solved" << endl;}
+   }
+
+   chrono.Stop();
+   if (mpi.Root())
+   {
+      cout << "Solver time: " << chrono.RealTime() << endl;
+   }
+   a->RecoverFEMSolution(X, *b, x);
+
+   int order_quad = max(2, 2 * order + 1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i = 0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double L2Error = x.ComputeL2Error(E, irs);
+   double norm_E = ComputeGlobalLpNorm(2, E, *pmesh, irs);
+
+   if (mpi.Root())
+   {
+      cout << "\n || E_h - E || / ||E|| = " << L2Error / norm_E << '\n' << endl;
+   }
+
+
+
+   if (visualization)
+   {
+      int num_procs, myid;
+      MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+      MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << x << flush;
+   }
+   // ---------------------------------------------------------------------
+
+   for (int i = 0 ; i < maxref; i++)
+   {
+      delete P[i];
+   }
+   delete A;
+   delete a;
+   delete b;
+   delete fec;
+   delete fespace;
+   delete pmesh;
+   return 0;
+}
+
+
+//define exact solution
+void E_exact(const Vector &x, Vector &E)
+{
+   if (dim != 3)
+   {
+      cout << "2D not set up yet: " << endl;
+      exit(0);
+   }
+   double curl2E[3];
+   get_maxwell_solution(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+// f = curl (mu curl E ) + omega^2*E
+void f_exact(const Vector &x, Vector &f)
+{
+   if (dim != 3)
+   {
+      cout << "2D not set up yet: " << endl;
+      exit(0);
+   }
+   double E[3], curl2E[3];
+
+   get_maxwell_solution(x, E, curl2E);
+
+   // curl ( curl E) + omega^2 E = f
+   double coeff;
+#ifdef DEFINITE
+   coeff = omega * omega;
+#else
+   coeff = -omega * omega;
+#endif
+
+   f(0) = curl2E[0] + coeff * E[0];
+   f(1) = curl2E[1] + coeff * E[1];
+   f(2) = curl2E[2] + coeff * E[2];
+}
+
+void get_maxwell_solution(const Vector & x, double E[], double curl2E[])
+{
+
+   if (isol == 0) // polynomial
+   {
+      // Polynomial vanishing on the boundary
+      E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+      E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+      E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+      //
+      curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+      curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+      curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+
+   }
+
+   else if (isol == 1) // sinusoidal
+   {
+      E[0] = sin(omega * x[1]);
+      E[1] = sin(omega * x[2]);
+      E[2] = sin(omega * x[0]);
+
+      curl2E[0] = omega * omega * E[0];
+      curl2E[1] = omega * omega * E[1];
+      curl2E[2] = omega * omega * E[2];
+
+   }
+   else if (isol == 2) //point source
+   {
+      // shift to avoid singularity
+      double x0 = x(0) + 0.1;
+      double x1 = x(1) + 0.1;
+      double x2 = x(2) + 0.1;
+      //
+      double r = sqrt(x0 * x0 + x1 * x1 + x2 * x2);
+
+      E[0] = cos(omega * r);
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+      double r_x = x0 / r;
+      double r_y = x1 / r;
+      double r_z = x2 / r;
+      double r_xy = -(r_x / r) * r_y;
+      double r_xz = -(r_x / r) * r_z;
+      double r_yx = r_xy;
+      double r_yy = (1.0 / r) * (1.0 - r_y * r_y);
+      double r_zx = r_xz;
+      double r_zz = (1.0 / r) * (1.0 - r_z * r_z);
+
+      curl2E[0] = omega * ((r_yy + r_zz) * sin(omega * r) +
+                           (omega * r_y * r_y + omega * r_z * r_z) * cos(omega * r));
+      curl2E[1] = -omega * (r_yx * sin(omega * r) + omega * r_y * r_x * cos(omega * r));
+      curl2E[2] = -omega * (r_zx * sin(omega * r) + omega * r_z * r_x * cos(omega * r));
+
+   }
+   else if (isol == 3) // plane wave
+   {
+      E[0] = cos(omega * (x(0) + x(1) + x(2)) / sqrt(3.0));
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+      curl2E[0] = 2.0 * omega * omega * E[0] / 3.0;
+      curl2E[1] = -omega * omega * E[0] / 3.0;
+      curl2E[2] = -omega * omega * E[0] / 3.0;
+   }
+
+}

examples/waves/maxwellp.cpp

@@ -0,0 +1,497 @@
+// Copyright (c) 2010, Lawrence Livermore National Security, LLC. Produced at
+// the Lawrence Livermore National Laboratory. LLNL-CODE-443211. All Rights
+// reserved. See file COPYRIGHT for details.
+//
+// This file is part of the MFEM library. For more information and source code
+// availability see http://mfem.org.
+//
+// MFEM is free software; you can redistribute it and/or modify it under the
+// terms of the GNU Lesser General Public License (as published by the Free
+// Software Foundation) version 2.1 dated February 1999.
+
+#include "mfem.hpp"
+#include <fstream>
+#include <iostream>
+#include <cmath>
+#include <complex>
+
+using namespace std;
+using namespace mfem;
+
+// #define DEFINITE
+
+// #ifndef MFEM_USE_PETSC
+// #error This example requires that MFEM is built with MFEM_USE_PETSC=YES
+// #endif
+
+// Define exact solution
+void E_exact_Re(const Vector & x, Vector & E);
+void f_exact_Re(const Vector & x, Vector & f);
+void get_maxwell_solution_Re(const Vector & x, double E[], double curl2E[]);
+
+void E_exact_Im(const Vector & x, Vector & E);
+void f_exact_Im(const Vector & x, Vector & f);
+void get_maxwell_solution_Im(const Vector & x, double E[], double curl2E[]);
+
+
+// Mesh Size
+int dim;
+double omega;
+double complex_shift;
+int isol = 1;
+
+
+int main(int argc, char *argv[])
+{
+   StopWatch chrono;
+
+   // 1. Initialise MPI
+   MPI_Session mpi(argc, argv);
+   // 1. Parse command-line options.
+   // geometry file
+   const char *mesh_file = "../../data/one-hex.mesh";
+   int order = 1;
+   // number of wavelengths
+   double k = 0.5;
+   //
+   const char *petscrc_file = "petscrc_mult_options";
+   // visualization flag
+   bool visualization = 1;
+   // number of initial ref
+   int initref = 1;
+   // number of mg levels
+   int maxref = 1;
+   // solver
+   int solver = 1;
+   //
+   complex_shift = 0.0;
+
+   OptionsParser args(argc, argv);
+   args.AddOption(&mesh_file, "-m", "--mesh",
+                  "Mesh file to use.");
+   args.AddOption(&order, "-o", "--order",
+                  "Finite element order (polynomial degree)."); 
+   args.AddOption(&k, "-k", "--wavelengths",
+                  "Number of wavelengths"); 
+   args.AddOption(&complex_shift, "-cs", "--complex_shift",
+                  "Complex shift");                
+   args.AddOption(&isol, "-isol", "--exact",
+                  "Exact solution flag - 0:polynomial, 1: plane wave");
+   args.AddOption(&initref, "-initref", "--initref",
+                  "Number of initial refinements.");
+   args.AddOption(&maxref, "-maxref", "--maxref",
+                  "Number of Refinements.");   
+   args.AddOption(&visualization, "-vis", "--visualization", "-no-vis",
+                  "--no-visualization",
+                  "Enable or disable GLVis visualization."); 
+   args.AddOption(&solver, "-s", "--solver",
+                  "Solver: 1 - GMG-GMRES, 2 - PETSC, 3 - SUPERLU, 4 - STRUMPACK, 5-HSS-GMRES");                       
+   args.Parse();
+   // check if the inputs are correct
+   if (!args.Good())
+   {
+      if ( mpi.Root() )
+      {
+         args.PrintUsage(cout);
+      }
+      MPI_Finalize();
+      return 1;
+   }
+   if ( mpi.Root() )
+   {
+      args.PrintOptions(cout);
+   }
+
+   enum SolverType
+   {
+      INVALID_SOL = -1,
+      GMG_GMRES   =  1,
+      PETSC       =  2,
+      SUPERLU     =  3,
+      STRUMPACK   =  4,
+      HSS_GMRES   =  5,
+   };
+
+   // Angular frequency
+   omega = 2.0*k*M_PI;
+
+   // Create serial mesh 
+   Mesh *mesh = new Mesh(mesh_file, 1, 1);
+   dim = mesh->Dimension();
+
+   // 3. Executing uniform h-refinement
+   for (int i = 0; i < initref; i++ )
+   {
+      mesh->UniformRefinement();
+   }
+
+   // create parallel mesh and delete the serial one
+   ParMesh *pmesh = new ParMesh(MPI_COMM_WORLD, *mesh);
+   delete mesh;
+
+   // Create H(curl) (Nedelec) Finite element space
+   FiniteElementCollection *fec   = new ND_FECollection(order, dim);
+   ParFiniteElementSpace *ND_fespace = new ParFiniteElementSpace(pmesh, fec);
+
+   std::vector<HypreParMatrix*>  P(maxref);
+
+   for (int i = 0; i < maxref; i++)
+   {
+      const ParFiniteElementSpace cfespace(*ND_fespace);
+      pmesh->UniformRefinement();
+      // Update fespace
+      ND_fespace->Update();
+      OperatorHandle Tr(Operator::Hypre_ParCSR);
+      ND_fespace->GetTrueTransferOperator(cfespace, Tr);
+      Tr.SetOperatorOwner(false);
+      Tr.Get(P[i]);
+   }
+
+   // 7. Linear form b(.) (Right hand side)
+   VectorFunctionCoefficient f_Re(dim, f_exact_Re);
+   VectorFunctionCoefficient f_Im(dim, f_exact_Im);
+   ParComplexLinearForm b(ND_fespace,ComplexOperator::HERMITIAN);
+   b.AddDomainIntegrator(new VectorFEDomainLFIntegrator(f_Re), 
+                         new VectorFEDomainLFIntegrator(f_Im));
+   b.real().Vector::operator=(0.0);
+   b.imag().Vector::operator=(0.0);
+   b.Assemble();
+
+
+   // 7. Bilinear form a(.,.) on the finite element space
+   ConstantCoefficient muinv(1.0);
+   ConstantCoefficient sigma(-pow(omega, 2));
+   ConstantCoefficient alpha(complex_shift);
+   ParSesquilinearForm a(ND_fespace, ComplexOperator::HERMITIAN);
+   a.AddDomainIntegrator(new CurlCurlIntegrator(muinv),NULL); 
+   a.AddDomainIntegrator(new VectorFEMassIntegrator(sigma),NULL);
+   a.AddDomainIntegrator(NULL,new VectorFEMassIntegrator(alpha));
+   a.Assemble();
+   a.Finalize();
+
+
+   Array<int> ess_tdof_list;
+   if (pmesh->bdr_attributes.Size())
+   {
+      Array<int> ess_bdr(pmesh->bdr_attributes.Max());
+      ess_bdr = 1;
+      ND_fespace->GetEssentialTrueDofs(ess_bdr, ess_tdof_list);
+   }
+
+   // Solution grid function
+   ParComplexGridFunction E_gf(ND_fespace);
+   VectorFunctionCoefficient E_Re(dim, E_exact_Re);
+   VectorFunctionCoefficient E_Im(dim, E_exact_Im);
+   E_gf.ProjectCoefficient(E_Re,E_Im);
+
+   OperatorHandle Ah;
+   Vector X, B;
+   a.FormLinearSystem(ess_tdof_list, E_gf, b, Ah, X, B);
+
+   ComplexHypreParMatrix * AZ = Ah.As<ComplexHypreParMatrix>();
+   HypreParMatrix * A = AZ->GetSystemMatrix();
+
+   if ( mpi.Root() )
+   {
+      cout << "Size of fine grid system: "
+           << A->GetGlobalNumRows() << " x " << A->GetGlobalNumCols() << endl;
+   }
+
+   chrono.Clear();
+   chrono.Start();
+  
+   switch((SolverType)solver)
+   {
+      case GMG_GMRES: 
+      {
+         if(mpi.Root()) {cout<< "Solver choice: GMG_GMRES" << endl;}
+#ifdef MFEM_USE_PETSC
+      MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+#endif  
+         ComplexGMGSolver M(AZ, P, ComplexGMGSolver::CoarseSolver::PETSC);
+         M.SetTheta(0.5);
+         M.SetSmootherType(HypreSmoother::Jacobi);
+
+         int maxit(5000);
+         double rtol(1.e-6);
+         double atol(0.0);
+         X = 0.0;
+         GMRESSolver gmres(MPI_COMM_WORLD);
+         gmres.SetAbsTol(atol);
+         gmres.SetRelTol(rtol);
+         gmres.SetMaxIter(maxit);
+         gmres.SetOperator(*AZ);
+         gmres.SetPreconditioner(M);
+         gmres.SetPrintLevel(1);
+         gmres.Mult(B,X);
+#ifdef MFEM_USE_PETSC
+         MFEMFinalizePetsc();
+#endif  
+      }   
+         break;
+      case PETSC:
+          {
+#ifndef MFEM_USE_PETSC
+         MFEM_ABORT("Invalid choice of CoarseSolver. MFEM is not linked with STRUMPACK");
+#else             
+         if(mpi.Root()) {cout<< "Solver choice: PETSC" << endl;}
+         MFEMInitializePetsc(NULL, NULL, petscrc_file, NULL);
+
+         PetscLinearSolver * invA = new PetscLinearSolver(MPI_COMM_WORLD, "direct");
+         PetscParMatrix *PA = new PetscParMatrix(A, Operator::PETSC_MATAIJ);
+         invA->SetOperator(*PA);
+         invA->Mult(B,X);
+         delete PA;
+         MFEMFinalizePetsc();
+#endif         
+      }   
+         break;
+      case SUPERLU:
+      {
+#ifndef MFEM_USE_SUPERLU
+         MFEM_ABORT("Invalid choice of CoarseSolver. MFEM is not linked with STRUMPACK");
+#else         
+         if(mpi.Root()) {cout<< "Solver choice: SuperLU" << endl;}
+         SuperLURowLocMatrix *SA = new SuperLURowLocMatrix(*A);
+         SuperLUSolver * superlu = new SuperLUSolver(MPI_COMM_WORLD);
+         // superlu->SetPrintStatistics(true);
+         // superlu->SetSymmetricPattern(false);
+         superlu->SetColumnPermutation(superlu::PARMETIS);
+         superlu->SetOperator(*SA);
+         superlu->Mult(B,X);
+         delete SA;
+         delete superlu;
+#endif         
+      }   
+         break;
+         case STRUMPACK:
+      {   
+#ifndef MFEM_USE_STRUMPACK
+         MFEM_ABORT("Invalid choice of CoarseSolver. MFEM is not linked with STRUMPACK");
+#else         
+         if(mpi.Root()) {cout<< "Solver choice: STRUMPACK" << endl;}
+         STRUMPACKRowLocMatrix *SA = new STRUMPACKRowLocMatrix(*A);
+         STRUMPACKSolver * strumpack = new STRUMPACKSolver(argc, argv, MPI_COMM_WORLD);
+         strumpack->SetPrintFactorStatistics(false);
+         strumpack->SetPrintSolveStatistics(true);
+         strumpack->SetHSS(true);
+         strumpack->SetHssAbsTol(0.0);
+         strumpack->SetHssRelTol(1e-4);
+         strumpack->SetAbsTol(0.0);
+         strumpack->SetRelTol(1e-6);
+         strumpack->SetKrylovSolver(strumpack::KrylovSolver::AUTO);
+         strumpack->SetReorderingStrategy(strumpack::ReorderingStrategy::METIS);
+         strumpack->DisableMatching();
+         strumpack->SetOperator(*SA);
+         strumpack->SetFromCommandLine();
+         strumpack->Mult(B, X);
+         delete SA;
+         delete strumpack;
+#endif         
+      }     
+         break  ;   
+      case HSS_GMRES:
+      {   
+#ifndef MFEM_USE_STRUMPACK
+         MFEM_ABORT("Invalid choice of CoarseSolver. MFEM is not linked with STRUMPACK");
+#else       
+         if(mpi.Root()) {cout<< "Solver choice: STRUMPACK" << endl;}
+         STRUMPACKRowLocMatrix *SA = new STRUMPACKRowLocMatrix(*A);
+         STRUMPACKSolver * prec = new STRUMPACKSolver(argc, argv, MPI_COMM_WORLD);
+         prec->SetPrintFactorStatistics(true);
+         prec->SetPrintSolveStatistics(false);
+         prec->SetHSS(true);
+         prec->SetHssAbsTol(0.0);
+         prec->SetHssRelTol(1e-4);
+         prec->SetKrylovSolver(strumpack::KrylovSolver::DIRECT);
+         prec->SetReorderingStrategy(strumpack::ReorderingStrategy::METIS);
+         prec->DisableMatching();
+         prec->SetOperator(*SA);
+         prec->SetFromCommandLine();
+         
+         int maxit(50);
+         double rtol(1.e-6);
+         double atol(0.0);
+         GMRESSolver gmres(MPI_COMM_WORLD);
+         gmres.SetAbsTol(atol);
+         gmres.SetRelTol(rtol);
+         gmres.SetMaxIter(maxit);
+         gmres.SetOperator(*A);
+         gmres.SetPreconditioner(*prec);
+         gmres.SetPrintLevel(1);
+         gmres.Mult(B,X);
+
+         delete SA;
+         delete prec;
+#endif
+      }     
+         break  ;   
+
+      default: 
+         if(mpi.Root()) {cout<< "Solver choice not valid. Problem not solved" << endl;}
+   }
+
+   chrono.Stop();
+   if (mpi.Root())
+   {
+      cout << "Solver time: " << chrono.RealTime() << endl;
+   }
+   a.RecoverFEMSolution(X,B,E_gf);
+
+
+   // Compute error
+   int order_quad = max(2, 2 * order + 1);
+   const IntegrationRule *irs[Geometry::NumGeom];
+   for (int i = 0; i < Geometry::NumGeom; ++i)
+   {
+      irs[i] = &(IntRules.Get(i, order_quad));
+   }
+
+   double L2Error_Re = E_gf.real().ComputeL2Error(E_Re, irs);
+   double norm_E_Re = ComputeGlobalLpNorm(2, E_Re, *pmesh, irs);
+
+   double L2Error_Im = E_gf.imag().ComputeL2Error(E_Im, irs);
+   double norm_E_Im = ComputeGlobalLpNorm(2, E_Im, *pmesh, irs);
+
+
+   if (mpi.Root())
+   {
+      cout << " Real Part: || E_h - E || / ||E|| = " << L2Error_Re / norm_E_Re << '\n' << endl;
+      cout << " Imag Part: || E_h - E || / ||E|| = " << L2Error_Im / norm_E_Im << '\n' << endl;
+
+      cout << " Real Part: || E_h - E || = " << L2Error_Re << '\n' << endl;
+      cout << " Imag Part: || E_h - E || = " << L2Error_Im << '\n' << endl;
+   }
+
+   // visualization   
+   if (visualization)
+   {
+      int num_procs, myid;
+      MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
+      MPI_Comm_rank(MPI_COMM_WORLD, &myid);
+      char vishost[] = "localhost";
+      int  visport   = 19916;
+      socketstream sol_sock(vishost, visport);
+      sol_sock << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock.precision(8);
+      sol_sock << "solution\n" << *pmesh << E_gf.real() << "window_title 'Real part'" << flush;
+
+      socketstream sol_sock_Im(vishost, visport);
+      sol_sock_Im << "parallel " << num_procs << " " << myid << "\n";
+      sol_sock_Im.precision(8);
+      sol_sock_Im << "solution\n" << *pmesh << E_gf.imag() << "window_title 'Imaginary part'" << flush;
+   }
+
+   // // delete invA;
+   delete fec;
+   delete ND_fespace;
+   delete pmesh;
+   return 0;
+}
+//define exact solution
+void E_exact_Re(const Vector &x, Vector &E)
+{
+   double curl2E[3];
+   get_maxwell_solution_Re(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+void f_exact_Re(const Vector &x, Vector &f)
+{
+   double E_Re[3], curl2E_Re[3];
+   double E_Im[3], curl2E_Im[3];
+
+   get_maxwell_solution_Re(x, E_Re, curl2E_Re);
+   get_maxwell_solution_Re(x, E_Im, curl2E_Im);
+
+   // curl ( curl E) - omega^2 E = f
+   double coeff;
+   coeff = -omega * omega;
+   f(0) = curl2E_Re[0] + coeff * E_Re[0];
+   f(1) = curl2E_Re[1] + coeff * E_Re[1];
+   f(2) = curl2E_Re[2] + coeff * E_Re[2];
+
+   // Account for the complex shift
+   f(0) += -complex_shift*E_Im[0];
+   f(1) += -complex_shift*E_Im[1];
+   f(2) += -complex_shift*E_Im[2];
+}
+
+void get_maxwell_solution_Re(const Vector & x, double E[], double curl2E[])
+{
+
+   if (isol == 0) // polynomial
+   {
+      E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+      E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+      E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+      curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+      curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+      curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+   }
+   else
+   {
+      double alpha = omega / sqrt(3);
+      E[0] = cos(alpha*(x(0) + x(1) + x(2)));
+      E[1] = 0.0;
+      E[2] = 0.0;
+
+      curl2E[0] = 2.0 * alpha * alpha * E[0];
+      curl2E[1] = -alpha * alpha * E[0];
+      curl2E[2] = -alpha * alpha * E[0];
+   }
+}
+
+
+
+//define exact solution
+void E_exact_Im(const Vector &x, Vector &E)
+{
+   double curl2E[3];
+   get_maxwell_solution_Re(x, E, curl2E);
+}
+
+//calculate RHS from exact solution
+void f_exact_Im(const Vector &x, Vector &f)
+{
+   double E_Re[3], curl2E_Re[3];
+   double E_Im[3], curl2E_Im[3];
+
+   get_maxwell_solution_Re(x, E_Im, curl2E_Im);
+   get_maxwell_solution_Re(x, E_Re, curl2E_Re);
+
+   // curl ( curl E) - omega^2 E = f
+   double coeff;
+   coeff = -omega * omega;
+   f(0) = curl2E_Im[0] + coeff * E_Im[0];
+   f(1) = curl2E_Im[1] + coeff * E_Im[1];
+   f(2) = curl2E_Im[2] + coeff * E_Im[2];
+
+   // Acount for the complex shift
+   f(0) += complex_shift*E_Re[0];
+   f(1) += complex_shift*E_Re[1];
+   f(2) += complex_shift*E_Re[2];
+}
+
+void get_maxwell_solution_Im(const Vector & x, double E[], double curl2E[])
+{
+   if (isol == 0) // polynomial
+   {
+      E[0] = x[1] * x[2]      * (1.0 - x[1]) * (1.0 - x[2]);
+      E[1] = x[0] * x[1] * x[2] * (1.0 - x[0]) * (1.0 - x[2]);
+      E[2] = x[0] * x[1]      * (1.0 - x[0]) * (1.0 - x[1]);
+      curl2E[0] = 2.0 * x[1] * (1.0 - x[1]) - (2.0 * x[0] - 3.0) * x[2] * (1 - x[2]);
+      curl2E[1] = 2.0 * x[1] * (x[0] * (1.0 - x[0]) + (1.0 - x[2]) * x[2]);
+      curl2E[2] = 2.0 * x[1] * (1.0 - x[1]) + x[0] * (3.0 - 2.0 * x[2]) * (1.0 - x[0]);
+   }
+   else
+   {
+      double alpha = omega / sqrt(3);   
+      E[0] = sin(alpha * (x(0) + x(1) + x(2)));
+      E[1] = 0.0;
+      E[2] = 0.0;
+      curl2E[0] = 2.0 * alpha * alpha * E[0];
+      curl2E[1] = -alpha * alpha * E[0];
+      curl2E[2] = -alpha * alpha * E[0];
+   }   
+}

examples/waves/petscrc_direct

@@ -0,0 +1,6 @@
+#-ksp_view
+#-mat_type mpiaij
+-ksp_type preonly
+-pc_type lu
+#-pc_factor_mat_solver_type mumps
+-pc_factor_mat_solver_type superlu_dist 

examples/waves/petscrc_mult_options

@@ -0,0 +1,13 @@
+-prefix_push direct
+-ksp_type preonly
+-pc_type lu
+-pc_factor_mat_solver_type mumps
+#-pc_factor_mat_solver_type strumpack
+#-pc_factor_mat_solver_type superlu_dist 
+#-ksp_view
+#-prefix_pop
+#
+#-prefix_push smoother
+#-ksp_type preonly
+#-pc_type jacobi
+#-prefix_pop