Update and release version 2.17.0.

Repair MOEAD

.travis.yml

@@ -1,5 +1,5 @@
 os: linux
-dist: trusty
+dist: focal
 language: cpp
 
 env:

CONTRIBUTING.md

@@ -22,7 +22,7 @@ merge, to ensure that:
 
 Please do make sure that if you contribute a new optimizer or other new
 functionality, that you've added some tests in the `tests/` directory.  And if
-you are fixing a bug, it's always nice to include a test case if possible to so
+you are fixing a bug, it's always nice to include a test case if possible so
 that the bug won't happen again.
 
 ## Build/test process
@@ -100,7 +100,7 @@ vcpkg install ensmallen:x64-windows
 
 This section describes how to build the **ensmallen** tests from source. **ensmallen** uses CMake as its build system and [Catch2](https://github.com/catchorg/Catch2) as the unit test framework.
 
-First, clone the source code from Github and change into the cloned directory. Or alternatively, you can download the latest relese from the [website](http://ensmallen.org) and extract it.
+First, clone the source code from Github and change into the cloned directory. Alternatively, you can download the latest release from the [website](http://ensmallen.org) and extract it.
 
 ```bash
 $ git clone https://github.com/mlpack/ensmallen
@@ -108,8 +108,8 @@ $ cd ensmallen
 
 # - or -
 
-$ wget http://ensmallen.org/files/ensmallen-2.14.0.tar.gz
-$ tar -xvzpf ensmallen-2.14.0.tar.gz
+$ wget http://ensmallen.org/files/ensmallen-2.17.0.tar.gz
+$ tar -xvzpf ensmallen-2.17.0.tar.gz
 $ cd ensmallen-latest
 ```
 

HISTORY.md

@@ -1,3 +1,111 @@
+### ensmallen 2.17.0: "Pachis Din Me Pesa Double"
+###### 2021-07-06
+ * CheckArbitraryFunctionTypeAPI extended for MOO support
+   ([#283](https://github.com/mlpack/ensmallen/pull/283)).
+
+ * Refactor NSGA2
+   ([#263](https://github.com/mlpack/ensmallen/pull/263),
+   [#304](https://github.com/mlpack/ensmallen/pull/304)).
+
+ * Add Indicators for Multiobjective optimizers
+   ([#285](https://github.com/mlpack/ensmallen/pull/285)).
+
+ * Make Callback flexible for MultiObjective Optimizers
+   ([#289](https://github.com/mlpack/ensmallen/pull/289)).
+
+ * Add ZDT Test Suite
+   ([#273](https://github.com/mlpack/ensmallen/pull/273)).
+
+ * Add MOEA-D/DE Optimizer
+   ([#269](https://github.com/mlpack/ensmallen/pull/269)).
+
+ * Introduce Policy Methods for MOEA/D-DE
+   ([#293](https://github.com/mlpack/ensmallen/pull/293)).
+
+ * Add Das-Dennis weight initialization method
+   ([#295](https://github.com/mlpack/ensmallen/pull/295)).
+
+ * Add Dirichlet Weight Initialization
+   ([#296](https://github.com/mlpack/ensmallen/pull/296)).
+
+ * Improved installation and compilation instructions
+   ([#300](https://github.com/mlpack/ensmallen/pull/300)).
+
+ * Disable building the tests by default for faster installation
+   ([#303](https://github.com/mlpack/ensmallen/pull/303)).
+
+ * Modify matrix initialisation to take into account
+   default element zeroing in Armadillo 10.5
+   ([#305](https://github.com/mlpack/ensmallen/pull/305)).
+
+### ensmallen 2.16.2: "Severely Dented Can Of Polyurethane"
+###### 2021-03-24
+ * Fix CNE test trials
+   ([#267](https://github.com/mlpack/ensmallen/pull/267)).
+
+ * Update Catch2 to 2.13.4
+   ([#268](https://github.com/mlpack/ensmallen/pull/268)).
+
+ * Fix typos in documentation
+   ([#270](https://github.com/mlpack/ensmallen/pull/270),
+    [#271](https://github.com/mlpack/ensmallen/pull/271)).
+
+ * Add clarifying comments in problems/ implementations
+   ([#276](https://github.com/mlpack/ensmallen/pull/276)).
+
+### ensmallen 2.16.1: "Severely Dented Can Of Polyurethane"
+###### 2021-03-02
+ * Fix test compilation issue when `ENS_USE_OPENMP` is set
+   ([#255](https://github.com/mlpack/ensmallen/pull/255)).
+
+ * Fix CNE initial population generation to use normal distribution
+   ([#258](https://github.com/mlpack/ensmallen/pull/258)).
+
+ * Fix compilation warnings
+   ([#259](https://github.com/mlpack/ensmallen/pull/259)).
+
+ * Remove `AdamSchafferFunctionN2Test` test from Adam test suite to prevent
+   spurious issue on some aarch64 ([#265](https://github.com/mlpack/ensmallen/pull/259)).
+
+### ensmallen 2.16.0: "Severely Dented Can Of Polyurethane"
+###### 2021-02-11
+ * Expand README with example installation and add simple example program
+   showing usage of the L-BFGS optimizer
+   ([#248](https://github.com/mlpack/ensmallen/pull/248)).
+
+ * Refactor tests to increase stability and reduce random errors
+   ([#249](https://github.com/mlpack/ensmallen/pull/249)).
+
+### ensmallen 2.15.1: "Why Can't I Manage To Grow Any Plants?"
+###### 2020-11-05
+ * Fix include order to ensure traits is loaded before reports
+   ([#239](https://github.com/mlpack/ensmallen/pull/239)).
+
+### ensmallen 2.15.0: "Why Can't I Manage To Grow Any Plants?"
+###### 2020-11-01
+ * Make a few tests more robust
+   ([#228](https://github.com/mlpack/ensmallen/pull/228)).
+
+ * Add release date to version information. ([#226](https://github.com/mlpack/ensmallen/pull/226))
+
+ * Fix typo in release script
+   ([#236](https://github.com/mlpack/ensmallen/pull/236)).
+
+### ensmallen 2.14.2: "No Direction Home"
+###### 2020-08-31
+ * Fix implementation of fonesca fleming problem function f1 and f2
+   type usage and negative signs. ([#223](https://github.com/mlpack/ensmallen/pull/223))
+
+### ensmallen 2.14.1: "No Direction Home"
+###### 2020-08-19
+ * Fix release script (remove hardcoded information, trim leading whitespaces
+   introduced by `wc -l` in MacOS)
+    ([#216](https://github.com/mlpack/ensmallen/pull/216),
+     [#220](https://github.com/mlpack/ensmallen/pull/220)).
+
+ * Adjust tolerance for AugLagrangian convergence based on element type
+   ([#217](https://github.com/mlpack/ensmallen/pull/217)).
+
 ### ensmallen 2.14.0: "No Direction Home"
 ###### 2020-08-10
  * Add NSGA2 optimizer for multi-objective functions
@@ -12,6 +120,9 @@
  * Fix L-BFGS convergence when starting from a minimum
    ([#201](https://github.com/mlpack/ensmallen/pull/201)).
 
+* Add optimizer summary report callback
+   ([#213](https://github.com/mlpack/ensmallen/pull/213)).
+
 ### ensmallen 2.13.0: "Automatically Automated Automation"
 ###### 2020-07-15
  * Fix CMake package export

README.md

@@ -22,11 +22,73 @@ gradient-free optimizers, and constrained optimization.
 
 ### Installation
 
-ensmallen can be installed with CMake 3.3 or later.
-If CMake is not already available on your system, it can be obtained from https://cmake.org
+ensmallen can be installed in several ways: either manually or via cmake, 
+with or without root access.
 
-If you are using an older system such as RHEL 7 or CentOS 7,
-an updated version of CMake is also available via the EPEL repository via the `cmake3` package.
+The cmake based installation will check the requirements 
+and optionally build the tests. If cmake 3.3 (or a later version) 
+is not already available on your system, it can be obtained 
+from [cmake.org](https://cmake.org). If you are using an older 
+system such as RHEL 7 or CentOS 7, an updated version of cmake 
+is also available via the EPEL repository (see the `cmake3` package).
+
+Example cmake based installation with root access:
+
+```
+mkdir build
+cd build
+cmake ..
+sudo make install
+```
+
+Example cmake based installation without root access, 
+installing into `/home/blah/` (adapt as required):
+
+```
+mkdir build
+cd build
+cmake .. -DCMAKE_INSTALL_PREFIX:PATH=/home/blah/
+make install
+```
+
+The above will create a directory named `/home/blah/include/` 
+and place all ensmallen headers there.
+
+To optionally build and run the tests
+(after running cmake as above),
+use the following additional commands:
+
+```
+make ensmallen_tests
+./ensmallen_tests --durations yes
+```
+
+Manual installation involves simply copying the `include/ensmallen.hpp` header 
+***and*** the associated `include/ensmallen_bits` directory to a location 
+such as `/usr/include/` which is searched by your C++ compiler.
+If you can't use `sudo` or don't have write access to `/usr/include/`, 
+use a directory within your own home directory (eg. `/home/blah/include/`).
+
+
+### Example Compilation
+
+If you have installed ensmallen in a standard location such as `/usr/include/`:
+
+    g++ prog.cpp -o prog -O2 -larmadillo
+    
+If you have installed ensmallen in a non-standard location, 
+such as `/home/blah/include/`, you will need to make sure 
+that your C++ compiler searches `/home/blah/include/` 
+by explicitly specifying the directory as an argument/option. 
+For example, using the `-I` switch in gcc and clang:
+
+    g++ prog.cpp -o prog -O2 -I /home/blah/include/ -larmadillo
+
+
+### Example Optimization
+
+See [`example.cpp`](example.cpp) for example usage of the L-BFGS optimizer 
+in a linear regression setting.
 
 
 ### License
@@ -34,7 +96,8 @@ an updated version of CMake is also available via the EPEL repository via the `c
 Unless stated otherwise, the source code for **ensmallen** is licensed under the
 3-clause BSD license (the "License").  A copy of the License is included in the
 "LICENSE.txt" file.  You may also obtain a copy of the License at
-http://opensource.org/licenses/BSD-3-Clause .
+http://opensource.org/licenses/BSD-3-Clause
+
 
 ### Citation
 
@@ -42,8 +105,8 @@ Please cite the following paper if you use ensmallen in your research and/or
 software. Citations are useful for the continued development and maintenance of
 the library.
 
-* S. Bhardwaj, R. Curtin, M. Edel, Y. Mentekidis, C. Sanderson.
-  [ensmallen: a flexible C++ library for efficient function optimization](http://www.ensmallen.org/files/ensmallen_2018.pdf).
+* S. Bhardwaj, R. Curtin, M. Edel, Y. Mentekidis, C. Sanderson.  
+  [ensmallen: a flexible C++ library for efficient function optimization](http://www.ensmallen.org/files/ensmallen_2018.pdf).  
   Workshop on Systems for ML and Open Source Software at NIPS 2018.
 
 ```

doc/callbacks.md

@@ -208,6 +208,95 @@ optimizer.Optimize(f, coordinates, ProgressBar());
 
 </details>
 
+### Report 
+
+Callback that prints a optimizer report to stdout or a specified output stream.
+
+#### Constructors
+
+ * `Report()`
+ * `Report(`_`iterationsPercentage`_`)`
+ * `Report(`_`iterationsPercentage, output`_`)`
+ * `Report(`_`iterationsPercentage, output, outputMatrixSize`_`)`
+
+#### Attributes
+
+| **type** | **name** | **description** | **default** |
+|----------|----------|-----------------|-------------|
+| `double` | **`iterationsPercentage`** | The number of iterations to report in percent, between [0, 1]. | `0.1` |
+| `std::ostream` | **`output`** | Ostream which receives output from this object. | `stdout` |
+| `size_t` | **`outputMatrixSize`** | The number of values to output for the function coordinates. | `4` |
+
+#### Examples:
+
+<details open>
+<summary>Click to collapse/expand example code.
+</summary>
+
+```c++
+AdaDelta optimizer(1.0, 1, 0.99, 1e-8, 1000, 1e-9, true);
+
+RosenbrockFunction f;
+arma::mat coordinates = f.GetInitialPoint();
+optimizer.Optimize(f, coordinates, Report(0.1));
+```
+
+<details open>
+<summary>Click to collapse/expand example output.
+</summary>
+
+```
+Optimization Report
+--------------------------------------------------------------------------------
+
+Initial Coordinates:
+  -1.2000   1.0000
+
+Final coordinates:
+  -1.0490   1.1070
+
+iter          loss          loss change   |gradient|    step size     total time
+0             24.2          0             233           1             4.27e-05
+100           8.6           15.6          104           1             0.000215
+200           5.26          3.35          48.7          1             0.000373
+300           4.49          0.767         23.4          1             0.000533
+400           4.31          0.181         11.3          1             0.000689
+500           4.27          0.0431        5.4           1             0.000846
+600           4.26          0.012         2.86          1             0.00101
+700           4.25          0.00734       2.09          1             0.00117
+800           4.24          0.00971       1.95          1             0.00132
+900           4.22          0.0146        1.91          1             0.00148
+
+--------------------------------------------------------------------------------
+
+Version:
+ensmallen:                    2.13.0 (Automatically Automated Automation)
+armadillo:                    9.900.1 (Nocturnal Misbehaviour)
+
+Function:
+Number of functions:          1
+Coordinates rows:             2
+Coordinates columns:          1
+
+Loss:
+Initial                       24.2
+Final                         4.2
+Change                        20
+
+Optimizer:
+Maximum iterations:           1000
+Reached maximum iterations:   true
+Batchsize:                    1
+Iterations:                   1000
+Number of epochs:             1001
+Initial step size:            1
+Final step size:              1
+Coordinates max. norm:        233
+Evaluate calls:               1000
+Gradient calls:               1000
+Time (in seconds):            0.00163
+```
+
 ### StoreBestCoordinates
 
 Callback that stores the model parameter after every epoch if the objective
@@ -242,7 +331,7 @@ StoreBestCoordinates<arma::mat> cb;
 optimizer.Optimize(f, coordinates, cb);
 
 std::cout << "The optimized model found by AdaDelta has the "
-      << "parameters " << cb.BestCoordinatest();
+      << "parameters " << cb.BestCoordinates();
 ```
 
 </details>
@@ -383,6 +472,23 @@ an estimate depending on `exactObjective` value.
 | `size_t` | **`epoch`** | The index of the current epoch. |
 | `double` | **`objective`** | Objective value of the current point. |
 
+### GenerationalStepTaken
+
+Called after the evolution of a single generation. Intended specifically for
+MultiObjective Optimizers.
+
+ * `GenerationalStepTaken(`_`optimizer, function, coordinates, objectives, frontIndices`_`)`
+
+#### Attributes
+
+| **type** | **name** | **description** |
+|----------|----------|-----------------|
+| `OptimizerType` | **`optimizer`** | The optimizer used to update the function. |
+| `FunctionType` | **`function`** | The function to be optimized. |
+| `MatType` | **`coordinates`** | The current function parameter. |
+| `ObjectivesVecType` | **`objectives`** | The set of calculated objectives so far. |
+| `IndicesType` | **`frontIndices`** | The indices of the members belonging to Pareto Front. |
+
 ## Custom Callbacks
 
 ### Learning rate scheduling

doc/function_types.md

@@ -876,18 +876,19 @@ NSGA2 nsga;
 double bestFrontSum = nsga.Optimize(objectives, coordinates);
 
 // Set `bestFront` to contain all of the coordinates on the best front.
-std::vector<arma::mat> bestFront = optimizer.Front();
+arma::cube bestFront = optimizer.ParetoFront();
 }
 ```
 
 </details>
 
 *Note*: all multi-objective function optimizers have both the function `Optimize()` to find the
-best front, and also the function `Front()` to return all sets of coordinates that are on the
+best front, and also the function `ParetoFront()` to return all sets of solutions that are on the
 front.
 
 The following optimizers can be used with multi-objective functions:
 - [NSGA2](#nsga2)
+- [MOEA/D-DE](#moead)
 
 ## Constrained functions
 

doc/optimizers.md

@@ -431,12 +431,12 @@ optimizer uses [L-BFGS](#l-bfgs).
 | **type** | **name** | **description** | **default** |
 |----------|----------|-----------------|-------------|
 | `size_t` | **`maxIterations`** | Maximum number of iterations allowed (0 means no limit). | `1000` |
-| `double` | **`penaltyThresholdFactor`** | When penalty threshold is updated, set it to this multiplied by the penalty. | `10.0` |
-| `double` | **`sigmaUpdateFactor`** | When sigma is updated, multiply it by this. | `0.25` |
+| `double` | **`penaltyThresholdFactor`** | When penalty threshold is updated, set it to this multiplied by the penalty. | `0.25` |
+| `double` | **`sigmaUpdateFactor`** | When sigma is updated, multiply it by this. | `10.0` |
 | `L_BFGS&` | **`lbfgs`** | Internal l-bfgs optimizer. | `L_BFGS()` |
 
 The attributes of the optimizer may also be modified via the member methods
-`MaxIterations()`, `PenaltyThresholdFactor()`, `SigmaUpdateFactor()` and `L_BFGS()`.
+`MaxIterations()`, `PenaltyThresholdFactor()`, `SigmaUpdateFactor()` and `LBFGS()`.
 
 <details open>
 <summary>Click to collapse/expand example code.
@@ -1563,6 +1563,96 @@ optimizer.Optimize(f, coordinates);
  * [SGD in Wikipedia](https://en.wikipedia.org/wiki/Stochastic_gradient_descent)
  * [Differentiable separable functions](#differentiable-separable-functions)
 
+## MOEA/D-DE
+*An optimizer for arbitrary multi-objective functions.*
+MOEA/D-DE (Multi Objective Evolutionary Algorithm based on Decomposition - Differential Evolution) is a multi
+objective optimization algorithm. It works by decomposing the problem into a number of scalar optimization
+subproblems which are solved simultaneously per generation. MOEA/D in itself is a framework, this particular
+algorithm uses Differential Crossover followed by Polynomial Mutation to create offsprings which are then
+decomposed to form a Single Objective Problem. A diversity preserving mechanism is also employed which encourages
+a varied set of solution.
+
+#### Constructors
+* `MOEAD<`_`InitPolicyType, DecompPolicyType`_`>()`
+* `MOEAD<`_`InitPolicyType, DecompPolicyType`_`>(`_`populationSize, maxGenerations, crossoverProb,  neighborProb, neighborSize, distributionIndex, differentialWeight, maxReplace, epsilon, lowerBound, upperBound`_`)`
+
+The _`InitPolicyType`_ template parameter refers to the strategy used to
+initialize the reference directions.
+
+The following types are available:
+
+ * **`Uniform`**
+ * **`BayesianBootstrap`**
+ * **`Dirichlet`**
+
+The _`DecompPolicyType`_ template parameter refers to the strategy used to
+decompose the weight vectors to form a scalar objective function.
+
+The following types are available:
+
+ * **`Tchebycheff`**
+ * **`WeightedAverage`**
+ * **`PenaltyBoundaryIntersection`**
+
+For convenience the following types can be used:
+
+ * **`DefaultMOEAD`** (equivalent to `MOEAD<Uniform, Tchebycheff>`): utilizes Uniform method for weight initialization
+ and Tchebycheff for weight decomposition.
+ 
+ * **`BBSMOEAD`** (equivalent to `MOEAD<BayesianBootstrap, Tchebycheff>`): utilizes Bayesian Bootstrap method for weight initialization and Tchebycheff for weight decomposition.
+ 
+ * **`DirichletMOEAD`** (equivalent to `MOEAD<Dirichlet, Tchebycheff>`): utilizes Dirichlet sampling for weight init
+ and Tchebycheff for weight decomposition.
+
+#### Attributes
+
+| **type** | **name** | **description** | **default** |
+|----------|----------|-----------------|-------------|
+| `size_t` | **`populationSize`** | The number of candidates in the population. | `150` |
+| `size_t` | **`maxGenerations`** | The maximum number of generations allowed. | `300` |
+| `double` | **`crossoverProb`** | Probability that a crossover will occur. | `1.0` |
+| `double` | **`neighborProb`** | The probability of sampling from neighbor. | `0.9` |
+| `size_t` | **`neighborSize`** | The number of nearest-neighbours to consider per weight vector.  | `20` |
+| `double` | **`distributionIndex`** | The crowding degree of the mutation. | `20` |
+| `double` | **`differentialWeight`** | Amplification factor of the differentiation. | `0.5` |
+| `size_t` | **`maxReplace`** | The limit of solutions allowed to be replaced by a child. | `2`|
+| `double` | **`epsilon`** | Handles numerical stability after weight initialization. | `1E-10`|
+| `double`, `arma::vec` | **`lowerBound`** | Lower bound of the coordinates on the coordinates of the whole population during the search process. | `0` |
+| `double`, `arma::vec` | **`upperBound`** | Lower bound of the coordinates on the coordinates of the whole population during the search process. | `1` |
+| `InitPolicyType` | **`initPolicy`** | Instantiated init policy used to initialize weights. | `InitPolicyType()` |
+| `DecompPolicyType` | **`decompPolicy`** | Instantiated decomposition policy used to create scalar objective problem. | `DecompPolicyType()` |
+
+Attributes of the optimizer may also be changed via the member methods
+`PopulationSize()`, `MaxGenerations()`, `CrossoverRate()`, `NeighborProb()`, `NeighborSize()`, `DistributionIndex()`,
+`DifferentialWeight()`, `MaxReplace()`, `Epsilon()`, `LowerBound()`, `UpperBound()`, `InitPolicy()` and `DecompPolicy()`.
+
+#### Examples:
+
+<details open>
+<summary>Click to collapse/expand example code.
+</summary>
+
+```c++
+SchafferFunctionN1<arma::mat> SCH;
+arma::vec lowerBound("-10 -10");
+arma::vec upperBound("10 10");
+DefaultMOEAD opt(300, 300, 1.0, 0.9, 20, 20, 0.5, 2, 1E-10, lowerBound, upperBound);
+typedef decltype(SCH.objectiveA) ObjectiveTypeA;
+typedef decltype(SCH.objectiveB) ObjectiveTypeB;
+arma::mat coords = SCH.GetInitialPoint();
+std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
+// obj will contain the minimum sum of objectiveA and objectiveB found on the best front.
+double obj = opt.Optimize(objectives, coords);
+// Now obtain the best front.
+arma::cube bestFront = opt.ParetoFront();
+```
+</details>
+
+#### See also
+* [MOEA/D-DE Algorithm](https://ieeexplore.ieee.org/document/4633340)
+* [Multi-objective Functions in Wikipedia](https://en.wikipedia.org/wiki/Test_functions_for_optimization#Test_functions_for_multi-objective_optimization)
+* [Multi-objective functions](#multi-objective-functions)
+
 ## NSGA2
 
 *An optimizer for arbitrary multi-objective functions.*
@@ -1619,7 +1709,7 @@ std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
 // obj will contain the minimum sum of objectiveA and objectiveB found on the best front.
 double obj = opt.Optimize(objectives, coords);
 // Now obtain the best front.
-std::vector<arma::mat> bestFront = opt.Front();
+arma::cube bestFront = opt.Front();
 ```
 
 </details>

example.cpp

@@ -0,0 +1,70 @@
+// Example implementation of an objective function class for linear regression
+// and usage of the L-BFGS optimizer.
+// 
+// Compilation:
+// g++ example.cpp -o example -O3 -larmadillo
+
+
+#include <iostream>
+#include <armadillo>
+#include <ensmallen.hpp>
+
+
+class LinearRegressionFunction
+{
+  public:
+  
+  LinearRegressionFunction(arma::mat& X, arma::vec& y) : X(X), y(y) { }
+  
+  double EvaluateWithGradient(const arma::mat& theta, arma::mat& gradient)
+  {
+    const arma::vec tmp = X.t() * theta - y;
+    gradient = 2 * X * tmp;
+    return arma::dot(tmp,tmp);
+  }
+  
+  private:
+  
+  const arma::mat& X;
+  const arma::vec& y;
+};
+
+
+int main(int argc, char** argv)
+{
+  if (argc < 3)
+  {
+    std::cout << "usage: " << argv[0] << " n_dims n_points" << std::endl;
+    return -1;
+  }
+  
+  int n_dims   = atoi(argv[1]);
+  int n_points = atoi(argv[2]);
+  
+  // generate noisy dataset with a slight linear pattern
+  arma::mat X(n_dims, n_points, arma::fill::randu);
+  arma::vec y(        n_points, arma::fill::randu);
+  
+  for (size_t i = 0; i < n_points; ++i)
+  {
+    double a = arma::randu();
+    X(1, i) += a;
+    y(i)    += a;
+  }
+  
+  LinearRegressionFunction lrf(X, y);
+  
+  // create a Limited-memory BFGS optimizer object with default parameters
+  ens::L_BFGS opt;
+  opt.MaxIterations() = 10;
+  
+  // initial point (uniform random)
+  arma::vec theta(n_dims, arma::fill::randu);
+  
+  opt.Optimize(lrf, theta);
+  
+  // theta now contains the optimized parameters
+  theta.print("theta:");
+  
+  return 0;
+}

include/ensmallen.hpp

@@ -65,12 +65,19 @@
 
 #include "ensmallen_bits/utility/any.hpp"
 #include "ensmallen_bits/utility/arma_traits.hpp"
+#include "ensmallen_bits/utility/indicators/epsilon.hpp"
+#include "ensmallen_bits/utility/indicators/igd_plus.hpp"
+
+// Contains traits, must be placed before report callback.
+#include "ensmallen_bits/function.hpp" // TODO: should move to function/
 
 // Callbacks.
 #include "ensmallen_bits/callbacks/callbacks.hpp"
 #include "ensmallen_bits/callbacks/early_stop_at_min_loss.hpp"
 #include "ensmallen_bits/callbacks/print_loss.hpp"
 #include "ensmallen_bits/callbacks/progress_bar.hpp"
+#include "ensmallen_bits/callbacks/query_front.hpp"
+#include "ensmallen_bits/callbacks/report.hpp"
 #include "ensmallen_bits/callbacks/store_best_coordinates.hpp"
 #include "ensmallen_bits/callbacks/timer_stop.hpp"
 
@@ -89,8 +96,6 @@
 #include "ensmallen_bits/eve/eve.hpp"
 #include "ensmallen_bits/ftml/ftml.hpp"
 
-#include "ensmallen_bits/function.hpp" // TODO: should move to function/
-
 #include "ensmallen_bits/fw/frank_wolfe.hpp"
 #include "ensmallen_bits/gradient_descent/gradient_descent.hpp"
 #include "ensmallen_bits/grid_search/grid_search.hpp"
@@ -98,6 +103,7 @@
 #include "ensmallen_bits/katyusha/katyusha.hpp"
 #include "ensmallen_bits/lbfgs/lbfgs.hpp"
 #include "ensmallen_bits/lookahead/lookahead.hpp"
+#include "ensmallen_bits/moead/moead.hpp"
 #include "ensmallen_bits/nsga2/nsga2.hpp"
 #include "ensmallen_bits/padam/padam.hpp"
 #include "ensmallen_bits/parallel_sgd/parallel_sgd.hpp"

include/ensmallen_bits/aug_lagrangian/aug_lagrangian_impl.hpp

@@ -108,6 +108,10 @@ AugLagrangian::Optimize(
   // Track the last objective to compare for convergence.
   ElemType lastObjective = function.Evaluate(coordinates);
 
+  // Convergence tolerance---depends on the epsilon of the type we are using for
+  // optimization.
+  ElemType tolerance = 1e3 * std::numeric_limits<ElemType>::epsilon();
+
   // Then, calculate the current penalty.
   ElemType penalty = 0;
   for (size_t i = 0; i < function.NumConstraints(); i++)
@@ -144,7 +148,7 @@ AugLagrangian::Optimize(
 
     // Check if we are done with the entire optimization (the threshold we are
     // comparing with is arbitrary).
-    if (std::abs(lastObjective - objective) < 1e-10 &&
+    if (std::abs(lastObjective - objective) < tolerance &&
         augfunc.Sigma() > 500000)
     {
       lambda = std::move(augfunc.Lambda());
@@ -198,6 +202,13 @@ AugLagrangian::Optimize(
       // We multiply sigma by a constant value.
       augfunc.Sigma() *= sigmaUpdateFactor;
       Info << "Updated sigma to " << augfunc.Sigma() << "." << std::endl;
+      if (augfunc.Sigma() >= std::numeric_limits<ElemType>::max() / 2.0)
+      {
+        Warn << "AugLagrangian::Optimize(): sigma too large for element type; "
+            << "terminating." << std::endl;
+        Callback::EndOptimization(*this, function, coordinates, callbacks...);
+        return false;
+      }
     }
 
     terminate |= Callback::StepTaken(*this, function, coordinates,

include/ensmallen_bits/callbacks/callbacks.hpp

@@ -744,6 +744,79 @@ class Callback
                     MatType& /* coordinates */)
   { return false; }
 
+ /**
+  * Invoke the GenerationalStepTaken() callback if it exists.
+  * Specialization for MultiObjective case.
+  *
+  * @param callback The callback to call.
+  * @param optimizer The optimizer used to update the function.
+  * @param function Function to optimize.
+  * @param coordinates Starting point.
+  * @param objectives The set of calculated objectives so far.
+  * @param frontIndices The indices of the members belonging to Pareto Front.
+  */
+  template<typename CallbackType,
+           typename OptimizerType,
+           typename FunctionType,
+           typename MatType,
+           typename ObjectivesVecType,
+           typename IndicesType>
+  static typename std::enable_if<
+      callbacks::traits::HasGenerationalStepTakenSignature<
+      CallbackType, OptimizerType, FunctionType, MatType, ObjectivesVecType,
+      IndicesType>::hasBool, bool>::type
+  GenerationalStepTakenFunction(CallbackType& callback,
+                                OptimizerType& optimizer,
+                                FunctionType& function,
+                                MatType& coordinates,
+                                ObjectivesVecType& objectives,
+                                IndicesType& frontIndices)
+  {
+    return const_cast<CallbackType&>(callback).GenerationalStepTaken(
+        optimizer, function, coordinates, objectives, frontIndices);
+  }
+
+  template<typename CallbackType,
+           typename OptimizerType,
+           typename FunctionType,
+           typename MatType,
+           typename ObjectivesVecType,
+           typename IndicesType>
+  static typename std::enable_if<
+      callbacks::traits::HasGenerationalStepTakenSignature<
+      CallbackType, OptimizerType, FunctionType, MatType, ObjectivesVecType,
+      IndicesType>::hasVoid, bool>::type
+  GenerationalStepTakenFunction(CallbackType& callback,
+                                OptimizerType& optimizer,
+                                FunctionType& function,
+                                MatType& coordinates,
+                                ObjectivesVecType& objectives,
+                                IndicesType& frontIndices)
+  {
+    const_cast<CallbackType&>(callback).GenerationalStepTaken(
+        optimizer, function, coordinates, objectives, frontIndices);
+
+    return false;
+  }
+
+  template<typename CallbackType,
+           typename OptimizerType,
+           typename FunctionType,
+           typename MatType,
+           typename ObjectivesVecType,
+           typename IndicesType>
+  static typename std::enable_if<
+      callbacks::traits::HasGenerationalStepTakenSignature<
+      CallbackType, OptimizerType, FunctionType, MatType, ObjectivesVecType,
+      IndicesType>::hasNone, bool>::type
+  GenerationalStepTakenFunction(CallbackType& /* callback */,
+                                OptimizerType& /* optimizer */,
+                                FunctionType& /* function */,
+                                MatType& /* coordinates */,
+                                ObjectivesVecType& /* objectives */,
+                                IndicesType& /* frontIndices */)
+  { return false; }
+
   /**
    * Iterate over the callbacks and invoke the StepTaken() callback if it
    * exists.
@@ -769,8 +842,41 @@ class Callback
             function, coordinates)... };
      return result;
   }
-};
 
+/**
+ * Iterate over the callbacks and invoke the GenerationalStepTaken() callback if it
+ * exists.
+ *
+ * Specialization for MultiObjective case.
+ *
+ * @param optimizer The optimizer used to update the function.
+ * @param function Function to optimize.
+ * @param coordinates Starting point.
+ * @param objectives The set of calculated objectives so far.
+ * @param frontIndices The indices of the members belonging to Pareto Front.
+ * @param callbacks The callbacks container.
+ */
+template<typename OptimizerType,
+         typename FunctionType,
+         typename ObjectivesVecType,
+         typename IndicesType,
+         typename MatType,
+         typename ...CallbackTypes>
+  static bool GenerationalStepTaken(OptimizerType& optimizer,
+                                    FunctionType& functions,
+                                    MatType& coordinates,
+                                    ObjectivesVecType& objectives,
+                                    IndicesType& frontIndices,
+                                    CallbackTypes&... callbacks)
+{
+  // This will return immediately once a callback returns true.
+  bool result = false;
+  (void)std::initializer_list<bool>{ result =
+    result || Callback::GenerationalStepTakenFunction(callbacks, optimizer,
+      functions, coordinates, objectives, frontIndices)... };
+  return result;
+}
+};
 } // namespace ens
 
 #endif

include/ensmallen_bits/callbacks/query_front.hpp

@@ -0,0 +1,85 @@
+/**
+ * @file query_front.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * Implementation of the query front callback function.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#ifndef ENSMALLEN_CALLBACKS_QUERY_FRONT_HPP
+#define ENSMALLEN_CALLBACKS_QUERY_FRONT_HPP
+
+namespace ens {
+
+/**
+ * Query the current Pareto Front after every GenerationalStepTaken callback function.
+ */
+class QueryFront
+{
+ public:
+  /**
+   * Set up the query front callback class with the specified inputs.
+   *
+   * @param queryRate The frequency at which the Pareto Front is queried.
+   * @param paretoFrontArray A reference to a vector of cube to store the queried fronts.
+   */
+    QueryFront(const size_t queryRate, std::vector<arma::cube>& paretoFrontArray) :
+        queryRate(queryRate),
+        paretoFrontArray(paretoFrontArray),
+        genCounter(0)
+    { /* Nothing to do here */ }
+
+   /**
+    * Callback function called at the end of a single generational run.
+    *
+    * @param optimizer The optimizer used to update the function.
+    * @param function Function to optimize.
+    * @param coordinates Starting point.
+    * @param objectives The set of calculated objectives so far.
+    * @param frontIndices The indices of the members belonging to Pareto Front.
+    */
+    template<typename OptimizerType,
+             typename FunctionType,
+             typename MatType,
+             typename ObjectivesVecType,
+             typename IndicesType>
+    void GenerationalStepTaken(OptimizerType& opt,
+                               FunctionType& /* function */,
+                               const MatType& /* coordinates */,
+                               const ObjectivesVecType& objectives,
+                               const IndicesType& frontIndices)
+    {
+      arma::cube currentParetoFront{};
+
+      if (genCounter % queryRate == 0)
+      {
+        currentParetoFront.resize(objectives[0].n_rows, objectives[0].n_cols,
+            frontIndices[0].size());
+        for (size_t solutionIdx = 0; solutionIdx < frontIndices[0].size(); ++solutionIdx)
+        {
+          currentParetoFront.slice(solutionIdx) =
+              arma::conv_to<arma::mat>::from(objectives[frontIndices[0][solutionIdx]]);
+        }
+
+        paretoFrontArray.emplace_back(std::move(currentParetoFront));
+      }
+
+      ++genCounter;
+    }
+
+
+ private:
+  //! The rate of query.
+  size_t queryRate;
+  //! A reference to the array of pareto fronts.
+  std::vector<arma::cube>& paretoFrontArray;
+  //! A counter for the current generation.
+  size_t genCounter;
+};
+
+} // namespace ens
+
+#endif

include/ensmallen_bits/callbacks/report.hpp

@@ -0,0 +1,613 @@
+/**
+ * @file report.hpp
+ * @author Marcus Edel
+ *
+ * Implementation of a simple report callback function.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#ifndef ENSMALLEN_CALLBACKS_REPORT_HPP
+#define ENSMALLEN_CALLBACKS_REPORT_HPP
+
+#include <ensmallen_bits/function.hpp>
+#include <iomanip>
+
+namespace ens {
+
+/**
+ * A simple optimization report.
+ */
+class Report
+{
+ public:
+  /**
+   * Set up the report callback class with the given output stream.
+   *
+   * @param iterationsPercentageIn The number of iterations to report in
+   *     percent, between [0, 1]).
+   * @param outputIn Ostream which receives output from this object.
+   * @param outputMatrixSizeIn The number of values to output for the function
+   *     coordinates.
+   */
+  Report(const double iterationsPercentageIn = 0.1,
+         std::ostream& outputIn = arma::get_cout_stream(),
+         const size_t outputMatrixSizeIn = 4) :
+      iterationsPercentage(iterationsPercentageIn),
+      output(outputIn),
+      outputMatrixSize(outputMatrixSizeIn),
+      objective(0),
+      gradientNorm(0),
+      hasGradient(false),
+      hasEndEpoch(false),
+      gradientCalls(0),
+      evaluateCalls(0),
+      epochCalls(0)
+  { /* Nothing to do here. */ }
+
+  /**
+   * Callback function called at the begin of the optimization process.
+   *
+   * @param optimizer The optimizer used to update the function.
+   * @param function Function to optimize.
+   * @param coordinates Starting point.
+   */
+  template<typename OptimizerType, typename FunctionType, typename MatType>
+  void BeginOptimization(OptimizerType& /* optimizer */,
+                         FunctionType& /* function */,
+                         MatType& coordinates)
+  {
+    initialCoordinates = coordinates;
+    optimizationTimer.tic();
+  }
+
+  /**
+   * Callback function called at the begin of the optimization process.
+   *
+   * @param optimizer The optimizer used to update the function.
+   * @param function Function to optimize.
+   * @param coordinates Starting point.
+   */
+  template<typename OptimizerType, typename FunctionType, typename MatType>
+  void EndOptimization(OptimizerType& optimizer,
+                       FunctionType& function,
+                       MatType& coordinates)
+  {
+    output << "Optimization Report" << std::endl;
+    output << std::string(80, '-') << std::endl << std::endl;
+
+    std::streamsize streamPrecision = output.precision(4);
+
+    if (coordinates.n_rows > outputMatrixSize ||
+        coordinates.n_cols > outputMatrixSize)
+    {
+      output << "Initial coordinates: " << std::endl;
+      TruncatePrint(initialCoordinates, outputMatrixSize);
+      output << std::endl << "Final coordinates: " << std::endl;
+      TruncatePrint(coordinates, outputMatrixSize);
+    }
+    else
+    {
+      output << "Initial Coordinates:" << std::endl << initialCoordinates.t();
+      output << std::endl << "Final coordinates:" << std::endl
+          << coordinates.t() << std::endl;
+    }
+
+    PrettyPrintElement("iter");
+    PrettyPrintElement("loss");
+    PrettyPrintElement("loss change");
+
+    if (hasGradient)
+      PrettyPrintElement("|gradient|");
+
+    if (!stepsizes.empty())
+      PrettyPrintElement("step size");
+
+    PrettyPrintElement("total time");
+    output << std::endl;
+
+    size_t iterationStep = objectives.size() / (iterationsPercentage * 100);
+    if (iterationStep <= 0)
+      iterationStep = 1;
+
+    for (size_t i = 0; i < objectives.size(); i += iterationStep)
+    {
+      PrettyPrintElement(i);
+      PrettyPrintElement(objectives[i]);
+      PrettyPrintElement(
+          i > 0 ? objectives[i - iterationStep] - objectives[i] : 0);
+
+      if (hasGradient)
+        PrettyPrintElement(gradientsNorm[i]);
+
+      if (!stepsizes.empty())
+        PrettyPrintElement(stepsizes[i]);
+
+      PrettyPrintElement(timings[i]);
+      output << std::endl;
+    }
+
+    output << std::endl << std::string(80, '-') << std::endl << std::endl;
+    output << "Version:" << std::endl;
+    PrettyPrintElement("ensmallen:", 30);
+    output << ens::version::as_string() << std::endl;
+    PrettyPrintElement("armadillo:", 30);
+    output << arma::arma_version::as_string() << std::endl << std::endl;
+
+    output << "Function:" << std::endl;
+    std::stringstream functionStream;
+
+    PrintNumFunctions(function, functionStream);
+    if (functionStream.rdbuf()->in_avail() > 0)
+      output << functionStream.str();
+
+    PrettyPrintElement("Coordinates rows:", 30);
+    output << coordinates.n_rows << std::endl;
+    PrettyPrintElement("Coordinates columns:", 30);
+    output << coordinates.n_cols << std::endl;
+    output << std::endl;
+
+    // If we did not take any steps, at least fill what the initial objective
+    // was.
+    const bool tookStep = (objectives.size() > 0);
+    if (objectives.size() == 0 && evaluateCalls > 0)
+    {
+      objectives.push_back(objective);
+      timings.push_back(optimizationTimer.toc());
+    }
+    else if (evaluateCalls == 0)
+    {
+      // It's not entirely clear how to compute the objective (since the
+      // function could implement many different ways of evaluating the
+      // objective), so issue an error and return.
+      output << "Objective never computed.  Did the optimization fail?"
+          << std::endl;
+      PrettyPrintElement("Time (in seconds):", 30);
+      output << optimizationTimer.toc() << std::endl;
+      return;
+    }
+
+    output << "Loss:" << std::endl;
+    PrettyPrintElement("Initial", 30);
+    output << objectives[0] << std::endl;
+    PrettyPrintElement("Final", 30);
+    output << objectives[objectives.size() - 1] << std::endl;
+    PrettyPrintElement("Change", 30);
+    output << objectives[0] - objectives[objectives.size() - 1] << std::endl;
+
+    output << std::endl << "Optimizer:" << std::endl;
+    std::stringstream optimizerStream;
+
+    PrintMaxIterations(optimizer, optimizerStream);
+    PrintBatchSize(optimizer, optimizerStream);
+    if (functionStream.rdbuf()->in_avail() > 0)
+      output << optimizerStream.str();
+
+    PrettyPrintElement("Iterations:", 30);
+    if (tookStep)
+      output << objectives.size() << std::endl;
+    else
+      output << "0 (No steps taken! Did the optimization fail?)" << std::endl;
+
+    if (epochCalls > 0)
+    {
+      PrettyPrintElement("Number of epochs:", 30);
+      output << epochCalls << std::endl;
+    }
+
+    if (!stepsizes.empty())
+    {
+      PrettyPrintElement("Initial step size:", 30);
+      output << stepsizes.front() << std::endl;
+
+      PrettyPrintElement("Final step size:", 30);
+      output << stepsizes.back() << std::endl;
+    }
+
+    if (hasGradient && gradientsNorm.size() > 0)
+    {
+      PrettyPrintElement("Coordinates max. norm:", 30);
+      output << *std::max_element(std::begin(gradientsNorm),
+          std::end(gradientsNorm)) << std::endl;
+    }
+
+    PrettyPrintElement("Evaluate calls:", 30);
+    output << evaluateCalls << std::endl;
+
+    if (hasGradient)
+    {
+      PrettyPrintElement("Gradient calls:", 30);
+      output << gradientCalls << std::endl;
+    }
+
+    PrettyPrintElement("Time (in seconds):", 30);
+    output << timings[timings.size() - 1] << std::endl;
+
+    // Restore precision.
+    output.precision(streamPrecision);
+  }
+
+  /**
+   * Callback function called at the beginning of a pass over the data.
+   *
+   * @param optimizer The optimizer used to update the function.
+   * @param function Function to optimize.
+   * @param coordinates Starting point.
+   * @param epoch The index of the current epoch.
+   * @param objective Objective value of the current point.
+   */
+  template<typename OptimizerType, typename FunctionType, typename MatType>
+  void BeginEpoch(OptimizerType& /* optimizer */,
+                  FunctionType& /* function */,
+                  const MatType& /* coordinates */,
+                  const size_t /* epoch */,
+                  const double /* objective */)
+  {
+    epochCalls++;
+  }
+
+  /**
+   * Callback function called at the end of a pass over the data.
+   *
+   * @param optimizer The optimizer used to update the function.
+   * @param function Function to optimize.
+   * @param coordinates Starting point.
+   * @param epoch The index of the current epoch.
+   * @param objective Objective value of the current point.
+   */
+  template<typename OptimizerType, typename FunctionType, typename MatType>
+  void EndEpoch(OptimizerType& optimizer,
+                FunctionType& /* function */,
+                const MatType& /* coordinates */,
+                const size_t /* epoch */,
+                const double objective)
+  {
+    // In case StepTaken() has been called first we clear the existing data.
+    if (!hasEndEpoch)
+    {
+      hasEndEpoch = true;
+
+      objectives.clear();
+      timings.clear();
+      gradientsNorm.clear();
+      stepsizes.clear();
+    }
+
+    objectives.push_back(objective);
+    timings.push_back(optimizationTimer.toc());
+
+    if (hasGradient)
+      gradientsNorm.push_back(gradientNorm);
+
+    SaveStepSize(optimizer);
+  }
+
+  /**
+   * Callback function called once a step is taken.
+   *
+   * @param optimizer The optimizer used to update the function.
+   * @param function Function to optimize.
+   * @param coordinates Starting point.
+   * @param objective Objective value of the current point.
+   */
+  template<typename OptimizerType, typename FunctionType, typename MatType>
+  void StepTaken(OptimizerType& optimizer,
+                 FunctionType& /* function */,
+                 const MatType& /* coordinates */)
+  {
+    if (!hasEndEpoch)
+    {
+      objectives.push_back(objective);
+      timings.push_back(optimizationTimer.toc());
+
+      if (hasGradient)
+        gradientsNorm.push_back(gradientNorm);
+
+      SaveStepSize(optimizer);
+    }
+  }
+
+  /**
+   * Callback function called at any call to Evaluate().
+   *
+   * @param optimizer The optimizer used to update the function.
+   * @param function Function to optimize.
+   * @param coordinates Starting point.
+   * @param objectiveIn Objective value of the current point.
+   */
+  template<typename OptimizerType, typename FunctionType, typename MatType>
+  void Evaluate(OptimizerType& /* optimizer */,
+                FunctionType& /* function */,
+                const MatType& /* coordinates */,
+                const double objectiveIn)
+  {
+    objective = objectiveIn;
+    evaluateCalls++;
+  }
+
+  /**
+    * Callback function called at any call to EvaluateConstraint().
+    *
+    * @param optimizer The optimizer used to update the function.
+    * @param function Function to optimize.
+    * @param coordinates Starting point.
+    * @param constraint The index of the constraint;
+    * @param objectiveIn Objective value of the current point.
+    */
+  template<typename OptimizerType, typename FunctionType, typename MatType>
+  void EvaluateConstraint(OptimizerType& /* optimizer */,
+                          FunctionType& /* function */,
+                          const MatType& /* coordinates */,
+                          const size_t /* constraint */,
+                          const double objectiveIn)
+  {
+    objective += objectiveIn;
+    evaluateCalls++;
+  }
+
+  /**
+   * Callback function called at any call to Gradient().
+   *
+   * @param optimizer The optimizer used to update the function.
+   * @param function Function to optimize.
+   * @param coordinates Starting point.
+   * @param gradientIn Matrix that holds the gradient.
+   */
+  template<typename OptimizerType, typename FunctionType, typename MatType>
+  void Gradient(OptimizerType& /* optimizer */,
+                FunctionType& /* function */,
+                const MatType& /* coordinates */,
+                const MatType& gradientIn)
+  {
+    hasGradient = true;
+    gradientNorm = arma::norm(gradientIn);
+    gradientCalls++;
+  }
+
+  /**
+   * Callback function called at any call to GradientConstraint().
+   *
+   * @param optimizer The optimizer used to update the function.
+   * @param function Function to optimize.
+   * @param coordinates Starting point.
+   * @param constraint The index of the constraint;
+   * @param gradient Matrix that holds the gradient;
+   */
+  template<typename OptimizerType, typename FunctionType, typename MatType>
+  void GradientConstraint(OptimizerType& optimizer,
+                          FunctionType& function,
+                          const MatType& coordinates,
+                          const size_t /* constraint */,
+                          const MatType& gradient)
+  {
+    Gradient(optimizer, function, coordinates, gradient);
+  }
+
+ private:
+  /**
+   * Helper function to print the number of function to the specified output
+   * stream.
+   *
+   * @param function The instantiated function that implements NumFunctions().
+   * @param stream The output stream.
+   */
+  template<typename FunctionType>
+  typename std::enable_if<
+      traits::HasNumFunctionsSignature<FunctionType>::value, void>::type
+  PrintNumFunctions(const FunctionType& function, std::stringstream& stream)
+  {
+    PrettyPrintElement(stream, "Number of functions:", 30);
+    stream << function.NumFunctions() << std::endl;
+  }
+
+  template<typename FunctionType>
+  typename std::enable_if<
+      !traits::HasNumFunctionsSignature<FunctionType>::value, void>::type
+  PrintNumFunctions(const FunctionType& /* function */,
+                    std::stringstream& /* stream */) { }
+
+  /**
+   * Helper function to output the max-iterations to the specified output
+   * stream.
+   *
+   * @param optimizer The instantiated optimizer that implements
+   *     MaxIterations().
+   * @param stream The output stream.
+   */
+  template<typename OptimizerType>
+  typename std::enable_if<
+      traits::HasMaxIterationsSignature<OptimizerType>::value, void>::type
+  PrintMaxIterations(const OptimizerType& optimizer, std::stringstream& stream)
+  {
+    PrettyPrintElement(stream, "Maximum iterations:", 30);
+    stream << optimizer.MaxIterations() << std::endl;
+
+    PrettyPrintElement(stream, "Reached maximum iterations:", 30);
+    stream << std::string(optimizer.MaxIterations() == objectives.size() ?
+        "true" : "false") << std::endl;
+  }
+
+  template<typename OptimizerType>
+  typename std::enable_if<
+      !traits::HasMaxIterationsSignature<OptimizerType>::value, void>::type
+  PrintMaxIterations(const OptimizerType& /* optimizer */,
+                     std::stringstream& /* stream */) { }
+
+  /**
+   * Helper function to output the batch-size to the specified output stream.
+   *
+   * @param optimizer The instantiated optimizer that implements BatchSize().
+   * @param stream The output stream.
+   */
+  template<typename OptimizerType>
+  typename std::enable_if<traits::HasBatchSizeSignature<OptimizerType>::value,
+      void>::type
+  PrintBatchSize(const OptimizerType& optimizer, std::stringstream& stream)
+  {
+    PrettyPrintElement(stream, "Batch size:", 30);
+    stream << optimizer.BatchSize() << std::endl;
+  }
+
+  template<typename OptimizerType>
+  typename std::enable_if<!traits::HasBatchSizeSignature<OptimizerType>::value,
+      void>::type
+  PrintBatchSize(const OptimizerType& /* optimizer */,
+                 std::stringstream& /* stream */) { }
+
+  /**
+   * Output formatted data.
+   *
+   * @param out Output stream.
+   * @param data The data to print on the given stream.
+   * @param width The width of the the formatted output data.
+   */
+  template<typename T>
+  void PrettyPrintElement(std::ostream& out,
+                          const T& data,
+                          const size_t width = 14)
+  {
+    out << std::left << std::setw(width) << std::setfill(' ')
+        << std::setprecision(3) << data;
+  }
+
+  /**
+   * Output formatted data.
+   *
+   * @param data The data to print on the given stream.
+   * @param width The width of the the formatted output data.
+   */
+  template<typename T>
+  void PrettyPrintElement(const T& data, const size_t width = 14)
+  {
+    PrettyPrintElement(output, data, width);
+  }
+
+  /**
+   * Outputs the given matrix in a truncated format. For example, the matrix:
+   *
+   * 1 2 3 4 5
+   * 6 7 8 9 10
+   * 11 12 13 14
+   * 15 16 17 18
+   *
+   * will be truncated to:
+   *
+   * 1 2 ... 5
+   * 6 7 ... 10
+   * ...
+   * 15 16 ... 18
+   *
+   * @param data The data to print on the given stream in a truncated format.
+   * @param size The number of elements per column/row.
+   */
+  template<typename T>
+  void TruncatePrint(const T& data, const size_t size)
+  {
+    // We can't directly output the result of submat or use .print, because
+    // both introduce a new line at the end, so we iterate over the elements.
+    for (size_t c = 0, n = 0; c < data.n_cols; ++c)
+    {
+      // Skip to the last column.
+      if (c >= (size - 1))
+      {
+        output << "..." << std::endl;
+        n = (data.n_cols - 2) * data.n_rows - 1;
+      }
+
+      for (size_t r = 0; r < data.n_rows; ++r)
+      {
+        // Check if need to skip to the last row.
+        if (r < (size - 1))
+        {
+          output << std::fixed;
+
+          // Add space for positive value, to align with negative values.
+          if (data(n) >= 0)
+            output << " ";
+
+          output << data(n++) << " ";
+        }
+        else
+        {
+          n = (c + 1) * data.n_rows - 1;
+          output << " ... " << data(n) << std::endl;
+          break;
+        }
+      }
+
+      if (c >= (size - 1))
+        break;
+    }
+  }
+
+  /**
+   * Helper function to store the step-size.
+   *
+   * @param optimizer The instantiated optimzer that implements StepSize().
+   */
+  template<typename OptimizerType>
+  typename std::enable_if<traits::HasStepSizeSignature<OptimizerType>::value,
+      void>::type
+  SaveStepSize(const OptimizerType& optimizer)
+  {
+    stepsizes.push_back(optimizer.StepSize());
+  }
+
+  template<typename OptimizerType>
+  typename std::enable_if<!traits::HasStepSizeSignature<OptimizerType>::value,
+      void>::type
+  SaveStepSize(const OptimizerType& /* optimizer */) { }
+
+  //! The number of iterations to print in percent.
+  double iterationsPercentage;
+
+  //! The output stream that all data is to be sent to; example: std::cout.
+  std::ostream& output;
+
+  //! The number of values to print for the function coordinates.
+  size_t outputMatrixSize;
+  //! The initial coordinates.
+  arma::mat initialCoordinates;
+
+  //! Gradient norm storage.
+  std::vector<double> gradientsNorm;
+
+  //! Objective storage.
+  std::vector<double> objectives;
+
+  //! Timing storage.
+  std::vector<double> timings;
+
+  //! Step-size storage.
+  std::vector<double> stepsizes;
+
+  //! Objective over the current epoch.
+  double objective;
+
+  //! Locally-stored gradient norm for a single step.
+  double gradientNorm;
+
+  //! Whether Gradient() was called.
+  bool hasGradient;
+
+  //! Whether EndEpoch() was called.
+  bool hasEndEpoch;
+
+  //! The number of Gradient() calls.
+  size_t gradientCalls;
+
+  //! The number of Evaluate() calls.
+  size_t evaluateCalls;
+
+  //! The number of BeginEpoch() calls.
+  size_t epochCalls;
+
+  //! Locally-stored optimization step timer object.
+  arma::wall_clock optimizationTimer;
+};
+
+} // namespace ens
+
+#endif

include/ensmallen_bits/callbacks/store_best_coordinates.hpp

@@ -55,7 +55,7 @@ class StoreBestCoordinates
   //! Get the best coordinates.
   ModelMatType const& BestCoordinates() const { return bestCoordinates; }
   //! Modify the best coordinates.
-  ModelMatType& BestCoordinatesl() { return bestCoordinates; }
+  ModelMatType& BestCoordinates() { return bestCoordinates; }
 
   //! Get the best objective.
   double const& BestObjective() const { return bestObjective; }

include/ensmallen_bits/callbacks/traits.hpp

@@ -37,6 +37,8 @@ ENS_HAS_EXACT_METHOD_FORM(BeginEpoch, HasBeginEpoch)
 ENS_HAS_EXACT_METHOD_FORM(EndEpoch, HasEndEpoch)
 //! Detect an StepTaken() method.
 ENS_HAS_EXACT_METHOD_FORM(StepTaken, HasStepTaken)
+//! Detect an GenerationalStepTaken() method.
+ENS_HAS_EXACT_METHOD_FORM(GenerationalStepTaken, HasGenerationalStepTaken)
 
 template<typename OptimizerType,
          typename FunctionType,
@@ -365,6 +367,69 @@ struct HasStepTakenSignature
          FunctionType, MatType>::template StepTakenVoidForm>::value;
 };
 
+//! A utility struct for Typed Forms required in
+//! callbacks for MultiObjective Optimizers.
+template<typename OptimizerType,
+         typename FunctionType,
+         typename MatType,
+         typename ObjectivesVecType,
+         typename IndicesType,
+         typename GradType = MatType>
+struct MOOTypedForms
+{
+  //! This is the form of a bool GenerationalStepTaken() for MOO callback method.
+  template<typename CallbackType>
+  using GenerationalStepTakenBoolForm =
+      bool(CallbackType::*)(OptimizerType&,
+                            FunctionType&,
+                            const MatType&,
+                            const ObjectivesVecType&,
+                            const IndicesType&);
+
+  //! This is the form of a void StepTaken() for MOO callback method.
+  template<typename CallbackType>
+  using GenerationalStepTakenVoidForm =
+      void(CallbackType::*)(OptimizerType&,
+                            FunctionType&,
+                            const MatType&,
+                            const ObjectivesVecType&,
+                            const IndicesType&);
+};
+
+//! Utility struct, check if either void StepTaken() or bool StepTaken() exists.
+//! Specialization for Multiobjective case.
+template<typename CallbackType,
+         typename OptimizerType,
+         typename FunctionType,
+         typename ObjectivesVecType,
+         typename IndicesType,
+         typename MatType>
+ struct HasGenerationalStepTakenSignature
+{
+  const static bool hasBool =
+    HasGenerationalStepTaken<CallbackType, MOOTypedForms<OptimizerType,
+        FunctionType, MatType, ObjectivesVecType, IndicesType>::
+        template GenerationalStepTakenBoolForm>::value &&
+    !HasGenerationalStepTaken<CallbackType, MOOTypedForms<OptimizerType,
+        FunctionType, MatType, ObjectivesVecType, IndicesType>::
+        template GenerationalStepTakenVoidForm>::value;
+
+  const static bool hasVoid =
+    !HasGenerationalStepTaken<CallbackType, MOOTypedForms<OptimizerType,
+        FunctionType, MatType, ObjectivesVecType, IndicesType>::
+        template GenerationalStepTakenBoolForm>::value &&
+    HasGenerationalStepTaken<CallbackType, MOOTypedForms<OptimizerType,
+        FunctionType, MatType, ObjectivesVecType, IndicesType>::
+        template GenerationalStepTakenVoidForm>::value;
+
+  const static bool hasNone =
+    !HasGenerationalStepTaken<CallbackType, MOOTypedForms<OptimizerType,
+        FunctionType, MatType, ObjectivesVecType, IndicesType>::
+        template GenerationalStepTakenBoolForm>::value &&
+    !HasGenerationalStepTaken<CallbackType, MOOTypedForms<OptimizerType,
+        FunctionType, MatType, ObjectivesVecType, IndicesType>::
+        template GenerationalStepTakenVoidForm>::value;
+};
 } // namespace traits
 } // namespace callbacks
 } // namespace ens

include/ensmallen_bits/cmaes/cmaes.hpp

@@ -97,9 +97,9 @@ class CMAES
                                        MatType& iterate,
                                        CallbackTypes&&... callbacks);
 
-  //! Get the step size.
+  //! Get the population size.
   size_t PopulationSize() const { return lambda; }
-  //! Modify the step size.
+  //! Modify the population size.
   size_t& PopulationSize() { return lambda; }
 
   //! Get the lower bound of decision variables.

include/ensmallen_bits/cne/cne_impl.hpp

@@ -93,7 +93,7 @@ typename MatType::elem_type CNE::Optimize(ArbitraryFunctionType& function,
   std::vector<BaseMatType> population;
   for (size_t i = 0 ; i < populationSize; ++i)
   {
-    population.push_back(arma::randu<BaseMatType>(iterate.n_rows,
+    population.push_back(arma::randn<BaseMatType>(iterate.n_rows,
         iterate.n_cols) + iterate);
   }
 

include/ensmallen_bits/ens_version.hpp

@@ -15,13 +15,17 @@
 #define ENS_VERSION_MAJOR 2
 // The minor version is two digits so regular numerical comparisons of versions
 // work right.  The first minor version of a release is always 10.
-#define ENS_VERSION_MINOR 14
+#define ENS_VERSION_MINOR 17
 #define ENS_VERSION_PATCH 0
 // If this is a release candidate, it will be reflected in the version name
 // (i.e. the version name will be "RC1", "RC2", etc.).  Otherwise the version
 // name will typically be a seemingly arbitrary set of words that does not
 // contain the capitalized string "RC".
-#define ENS_VERSION_NAME "No Direction Home"
+#define ENS_VERSION_NAME "Pachis Din Me Pesa Double"
+// Incorporate the date the version was released.
+#define ENS_VERSION_YEAR "2021"
+#define ENS_VERSION_MONTH "07"
+#define ENS_VERSION_DAY "06"
 
 namespace ens {
 
@@ -41,6 +45,14 @@ struct version
 
     return ss.str();
   }
+    
+  static inline std::string date()
+  {
+    std::stringstream ss;
+    ss << ENS_VERSION_YEAR << '-' << ENS_VERSION_MONTH << '-' << ENS_VERSION_DAY;
+
+    return ss.str();
+  }
 };
 
 } // namespace ens

include/ensmallen_bits/function/static_checks.hpp

@@ -387,6 +387,25 @@ inline void CheckArbitraryFunctionTypeAPI()
 #endif
 }
 
+template<typename FunctionType, typename... RemainingTypes>
+typename std::enable_if<(sizeof...(RemainingTypes) > 1), void>::type
+CheckArbitraryFunctionTypeAPI()
+{
+#ifndef ENS_DISABLE_TYPE_CHECKS
+  constexpr size_t size = sizeof...(RemainingTypes);
+  using TupleType = typename std::tuple<RemainingTypes...>;
+  using MatType = typename std::tuple_element<size - 1, TupleType>::type;
+
+  static_assert(CheckEvaluate<FunctionType, MatType, MatType>::value,
+      "One of the provided FunctionType does not have a correct definition of Evaluate(). "
+      "Please check that the corresponding FunctionType fully satisfies the requirements "
+      "of the ArbitraryFunctionType API; see the optimizer tutorial for "
+      "more details.");
+
+  CheckArbitraryFunctionTypeAPI<RemainingTypes...>();
+#endif
+}
+
 /**
  * Perform checks for the ResolvableFunctionType API.
  */

include/ensmallen_bits/function/traits.hpp

@@ -45,6 +45,8 @@ ENS_HAS_EXACT_METHOD_FORM(MaxIterations, HasMaxIterations)
 ENS_HAS_EXACT_METHOD_FORM(ResetPolicy, HasResetPolicy)
 //! Detect an BatchSize() method.
 ENS_HAS_EXACT_METHOD_FORM(BatchSize, HasBatchSize)
+//! Detect an StepSize() method.
+ENS_HAS_EXACT_METHOD_FORM(StepSize, HasStepSize)
 
 template<typename MatType, typename GradType>
 struct TypedForms
@@ -391,6 +393,22 @@ struct HasBatchSizeSignature
       HasBatchSize<OptimizerType, BatchSizeConstForm>::value;
 };
 
+//! Utility struct, check if size_t StepSize() const or size_t StepSize()
+//! exists.
+template<typename OptimizerType>
+struct HasStepSizeSignature
+{
+  template<typename C>
+  using StepSizeConstForm = double(C::*)(void) const;
+
+  template<typename C>
+  using StepSizeForm = double(C::*)(void);
+
+  const static bool value =
+      HasStepSize<OptimizerType, StepSizeForm>::value ||
+      HasStepSize<OptimizerType, StepSizeConstForm>::value;
+};
+
 //! Utility struct, check if size_t MaxIterations() const exists.
 template<typename OptimizerType>
 struct HasMaxIterationsSignature

include/ensmallen_bits/fw/proximal/proximal_impl.hpp

@@ -45,7 +45,7 @@ inline void Proximal::ProjectToL1Ball(MatType& v, double tau)
   MatType simplexSum = arma::cumsum(simplexSol);
 
   double nu = 0;
-  size_t rho = 0;
+  size_t rho = simplexSol.n_rows - 1;
   for (size_t j = 1; j <= simplexSol.n_rows; j++)
   {
     rho = simplexSol.n_rows - j;

include/ensmallen_bits/moead/decomposition_policies/pbi_decomposition.hpp

@@ -0,0 +1,81 @@
+/**
+ * @file pbi_decomposition.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * The Penalty Based Boundary Intersection (PBI) decomposition policy.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#ifndef ENSMALLEN_MOEAD_PBI_HPP
+#define ENSMALLEN_MOEAD_PBI_HPP
+
+namespace ens {
+
+/**
+ * Penalty Based Boundary Intersection (PBI) method is a weight decomposition method,
+ * it tries to find the intersection between bottom-most boundary of the attainable
+ * objective set with the reference directions.
+ *
+ * The goal is to minimize the distance between objective vectors with the ideal point
+ * along the reference direction. To handle equality constraints, a penalty parameter
+ * theta is used.
+ *
+ * For more information, see the following:
+ * @code
+ * article{zhang2007moea,
+ *   title={MOEA/D: A multiobjective evolutionary algorithm based on decomposition},
+ *   author={Zhang, Qingfu and Li, Hui},
+ *   journal={IEEE Transactions on evolutionary computation},
+ *   pages={712--731},
+ *   year={2007}
+ * @endcode
+ */
+class PenaltyBoundaryIntersection
+{
+ public:
+  /**
+   * Constructor for Penalty Based Boundary Intersection decomposition
+   * policy.
+   *
+   * @param theta The penalty value.
+   */
+  PenaltyBoundaryIntersection(const double theta = 5) :
+      theta(theta)
+  {
+    /* Nothing to do. */
+  }
+
+  /**
+   * Decompose the weight vectors.
+   *
+   * @tparam VecType The type of the vector used in the decommposition.
+   * @param weight The weight vector corresponding to a subproblem.
+   * @param idealPoint The reference point in the objective space.
+   * @param candidateFitness The objective vector of the candidate.
+   */
+  template<typename VecType>
+  typename VecType::elem_type Apply(const VecType& weight,
+                                    const VecType& idealPoint,
+                                    const VecType& candidateFitness)
+  {
+    typedef typename VecType::elem_type ElemType;
+    //! A unit vector in the same direction as the provided weight vector.
+    const VecType referenceDirection = weight / arma::norm(weight);
+    //! Distance of F(x) from the idealPoint along the reference direction.
+    const ElemType d1 = arma::dot(candidateFitness - idealPoint, referenceDirection);
+    //! The perpendicular distance of F(x) from reference direction.
+    const ElemType d2 = arma::norm(candidateFitness - (idealPoint + d1 * referenceDirection));
+
+    return d1 + static_cast<ElemType>(theta) * d2;
+  }
+
+  private:
+    double theta;
+};
+
+} // namespace ens
+
+#endif

include/ensmallen_bits/moead/decomposition_policies/tchebycheff_decomposition.hpp

@@ -0,0 +1,66 @@
+/**
+ * @file tchebycheff_decomposition.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * The Tchebycheff Weight decomposition policy.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#ifndef ENSMALLEN_MOEAD_TCHEBYCHEFF_HPP
+#define ENSMALLEN_MOEAD_TCHEBYCHEFF_HPP
+
+namespace ens {
+
+/**
+ * The Tchebycheff method works by taking the maximum of element-wise product
+ * between reference direction and the line connecting objective vector and
+ * ideal point.
+ *
+ * Under mild conditions, for each Pareto Optimal point there exists a reference
+ * direction such that the given point is also the optimal solution
+ * to this scalar objective.
+ *
+ * For more information, see the following:
+ * @code
+ * article{zhang2007moea,
+ *   title={MOEA/D: A multiobjective evolutionary algorithm based on decomposition},
+ *   author={Zhang, Qingfu and Li, Hui},
+ *   journal={IEEE Transactions on evolutionary computation},
+ *   pages={712--731},
+ *   year={2007}
+ * @endcode
+ */
+class Tchebycheff
+{
+ public:
+  /**
+   * Constructor for Tchebycheff decomposition policy.
+   */
+  Tchebycheff()
+  {
+    /* Nothing to do. */
+  }
+
+  /**
+   * Decompose the weight vectors.
+   *
+   * @tparam VecType The type of the vector used in the decommposition.
+   * @param weight The weight vector corresponding to a subproblem.
+   * @param idealPoint The reference point in the objective space.
+   * @param candidateFitness The objective vector of the candidate.
+   */
+  template<typename VecType>
+  typename VecType::elem_type Apply(const VecType& weight,
+                                    const VecType& idealPoint,
+                                    const VecType& candidateFitness)
+  {
+    return arma::max(weight % arma::abs(candidateFitness - idealPoint));
+  }
+};
+
+} // namespace ens
+
+#endif

include/ensmallen_bits/moead/decomposition_policies/weighted_decomposition.hpp

@@ -0,0 +1,62 @@
+/**
+ * @file weighted_decomposition.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * The Weighted Average decomposition policy.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#ifndef ENSMALLEN_MOEAD_WEIGHTED_HPP
+#define ENSMALLEN_MOEAD_WEIGHTED_HPP
+
+namespace ens {
+
+/**
+ * The Weighted average method of decomposition. The working principle is to
+ * minimize the dot product between reference direction and the line connecting
+ * objective vector and ideal point.
+ *
+ * For more information, see the following:
+ * @code
+ * article{zhang2007moea,
+ *   title={MOEA/D: A multiobjective evolutionary algorithm based on decomposition},
+ *   author={Zhang, Qingfu and Li, Hui},
+ *   journal={IEEE Transactions on evolutionary computation},
+ *   pages={712--731},
+ *   year={2007}
+ * @endcode
+ */
+class WeightedAverage
+{
+ public:
+  /**
+   * Constructor for Weighted Average decomposition policy.
+   */
+  WeightedAverage()
+  {
+    /* Nothing to do. */
+  }
+
+  /**
+   * Decompose the weight vectors.
+   *
+   * @tparam VecType The type of the vector used in the decommposition.
+   * @param weight The weight vector corresponding to a subproblem.
+   * @param idealPoint The reference point in the objective space.
+   * @param candidateFitness The objective vector of the candidate.
+   */
+  template<typename VecType>
+  typename VecType::elem_type Apply(const VecType& weight,
+                                    const VecType& /* idealPoint */,
+                                    const VecType& candidateFitness)
+  {
+    return arma::dot(weight, candidateFitness);
+  }
+};
+
+} // namespace ens
+
+#endif

include/ensmallen_bits/moead/moead.hpp

@@ -0,0 +1,338 @@
+/**
+ * @file moead.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * MOEA/D-DE is a multi objective optimization algorithm. MOEA/D-DE
+ * uses genetic algorithms along with a set of reference directions
+ * to drive the population towards the Optimal Front.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#ifndef ENSMALLEN_MOEAD_MOEAD_HPP
+#define ENSMALLEN_MOEAD_MOEAD_HPP
+
+//! Decomposition policies.
+#include "decomposition_policies/tchebycheff_decomposition.hpp"
+#include "decomposition_policies/weighted_decomposition.hpp"
+#include "decomposition_policies/pbi_decomposition.hpp"
+
+//! Weight initialization policies.
+#include "weight_init_policies/uniform_init.hpp"
+#include "weight_init_policies/bbs_init.hpp"
+#include "weight_init_policies/dirichlet_init.hpp"
+
+namespace ens {
+
+/**
+ * MOEA/D-DE (Multi Objective Evolutionary Algorithm based on Decompositon - 
+ * Differential Variant) is a multiobjective optimization algorithm. This class 
+ * implements the said optimizer. 
+ *
+ * The algorithm works by generating a candidate population from a fixed starting point. 
+ * Reference directions are generated to guide the optimization process towards the Pareto Front. 
+ * Further, a decomposition function is defined to decompose the problem to a scalar optimization 
+ * objective. Utilizing genetic operators, offsprings are generated with better decomposition values 
+ * to replace the neighboring parent solutions. 
+ *
+ * For more information, see the following:
+ * @code
+ * @article{li2008multiobjective,
+ *   title={Multiobjective optimization problems with complicated Pareto sets, MOEA/D and NSGA-II},
+ *   author={Li, Hui and Zhang, Qingfu},
+ *   journal={IEEE transactions on evolutionary computation},
+ *   pages={284--302},
+ *   year={2008},
+ * @endcode
+ */
+template<typename InitPolicyType = Uniform,
+         typename DecompPolicyType = Tchebycheff>
+class MOEAD {
+ public:
+  /**
+   * Constructor for the MOEA/D optimizer.
+   *
+   * The default values provided here are not necessarily suitable for a
+   * given function. Therefore, it is highly recommended to adjust the
+   * parameters according to the problem.
+   *
+   * @param populationSize The number of elements in the population.
+   * @param maxGenerations The maximum number of generations allowed.
+   * @param crossoverProb The probability that a crossover will occur.
+   * @param neighborProb The probability of sampling from neighbor.
+   * @param neighborSize The number of nearest neighbours of weights
+   *    to find.
+   * @param distributionIndex The crowding degree of the mutation.
+   * @param differentialWeight A parameter used in the mutation of candidate
+   *     solutions controls amplification factor of the differentiation.
+   * @param maxReplace The limit of solutions allowed to be replaced by a child.
+   * @param epsilon Handle numerical stability after weight initialization.
+   * @param lowerBound The lower bound on each variable of a member
+   *    of the variable space.
+   * @param upperBound The upper bound on each variable of a member
+   *    of the variable space.
+   */
+  MOEAD(const size_t populationSize = 300,
+        const size_t maxGenerations = 500,
+        const double crossoverProb = 1.0,
+        const double neighborProb = 0.9,
+        const size_t neighborSize = 20,
+        const double distributionIndex = 20,
+        const double differentialWeight = 0.5,
+        const size_t maxReplace = 2,
+        const double epsilon = 1E-10,
+        const arma::vec& lowerBound = arma::zeros(1, 1),
+        const arma::vec& upperBound = arma::ones(1, 1),
+        const InitPolicyType initPolicy = InitPolicyType(),
+        const DecompPolicyType decompPolicy = DecompPolicyType());
+
+  /**
+   * Constructor for the MOEA/D optimizer. This constructor is provides an
+   * overload to use lowerBound and upperBound as doubles, in case all the
+   * variables in the problem have the same limits.
+   *
+   * The default values provided here are not necessarily suitable for a
+   * given function. Therefore, it is highly recommended to adjust the
+   * parameters according to the problem.
+   *
+   * @param populationSize The number of elements in the population.
+   * @param maxGenerations The maximum number of generations allowed.
+   * @param crossoverProb The probability that a crossover will occur.
+   * @param neighborProb The probability of sampling from neighbor.
+   * @param neighborSize The number of nearest neighbours of weights
+   *    to find.
+   * @param distributionIndex The crowding degree of the mutation.
+   * @param differentialWeight A parameter used in the mutation of candidate
+   *     solutions controls amplification factor of the differentiation.
+   * @param maxReplace The limit of solutions allowed to be replaced by a child.
+   * @param epsilon Handle numerical stability after weight initialization.
+   * @param lowerBound The lower bound on each variable of a member
+   *    of the variable space.
+   * @param upperBound The upper bound on each variable of a member
+   *    of the variable space.
+   */
+    MOEAD(const size_t populationSize = 300,
+          const size_t maxGenerations = 500,
+          const double crossoverProb = 1.0,
+          const double neighborProb = 0.9,
+          const size_t neighborSize = 20,
+          const double distributionIndex = 20,
+          const double differentialWeight = 0.5,
+          const size_t maxReplace = 2,
+          const double epsilon = 1E-10,
+          const double lowerBound = 0,
+          const double upperBound = 1,
+          const InitPolicyType initPolicy = InitPolicyType(),
+          const DecompPolicyType decompPolicy = DecompPolicyType());
+
+  /**
+   * Optimize a set of objectives. The initial population is generated
+   * using the initial point. The output is the best generated front.
+   *
+   * @tparam MatType The type of matrix used to store coordinates.
+   * @tparam ArbitraryFunctionType The type of objective function.
+   * @tparam CallbackTypes Types of callback function.
+   * @param objectives std::tuple of the objective functions.
+   * @param iterate The initial reference point for generating population.
+   * @param callbacks The callback functions.
+   */
+  template<typename MatType,
+           typename... ArbitraryFunctionType,
+           typename... CallbackTypes>
+  typename MatType::elem_type Optimize(std::tuple<ArbitraryFunctionType...>& objectives,
+                                       MatType& iterate,
+                                       CallbackTypes&&... callbacks);
+
+  //! Retrieve population size.
+  size_t PopulationSize() const { return populationSize; }
+  //! Modify the population size.
+  size_t& PopulationSize() { return populationSize; }
+
+  //! Retrieve number of generations.
+  size_t MaxGenerations() const { return maxGenerations; }
+  //! Modify the number of generations.
+  size_t& MaxGenerations() { return maxGenerations; }
+
+  //! Retrieve crossover rate.
+  double CrossoverRate() const { return crossoverProb; }
+  //! Modify the crossover rate.
+  double& CrossoverRate() { return crossoverProb; }
+
+  //! Retrieve size of the weight neighbor.
+  size_t NeighborSize() const { return neighborSize; }
+  //! Modify the size of the weight neighbor.
+  size_t& NeighborSize() { return neighborSize; }
+
+  //! Retrieve value of the distribution index.
+  double DistributionIndex() const { return distributionIndex; }
+  //! Modify the value of the distribution index.
+  double& DistributionIndex() { return distributionIndex; }
+
+  //! Retrieve value of neighbor probability.
+  double NeighborProb() const { return neighborProb; }
+  //! Modify the value of neigbourhood probability.
+  double& NeighborProb() { return neighborProb; }
+
+  //! Retrieve value of scaling factor.
+  double DifferentialWeight() const { return differentialWeight; }
+  //! Modify the value of scaling factor.
+  double& DifferentialWeight() { return differentialWeight; }
+
+  //! Retrieve value of maxReplace.
+  size_t MaxReplace() const { return maxReplace; }
+  //! Modify value of maxReplace.
+  size_t& MaxReplace() { return maxReplace; }
+
+   //! Retrieve value of epsilon.
+  double Epsilon() const { return epsilon; }
+  //! Modify value of maxReplace.
+  double& Epsilon() { return epsilon; }
+
+  //! Retrieve value of lowerBound.
+  const arma::vec& LowerBound() const { return lowerBound; }
+  //! Modify value of lowerBound.
+  arma::vec& LowerBound() { return lowerBound; }
+
+  //! Retrieve value of upperBound.
+  const arma::vec& UpperBound() const { return upperBound; }
+  //! Modify value of upperBound.
+  arma::vec& UpperBound() { return upperBound; }
+
+  //! Retrieve the Pareto optimal points in variable space. This returns an empty cube
+  //! until `Optimize()` has been called.
+  const arma::cube& ParetoSet() const { return paretoSet; }
+
+  //! Retrieve the best front (the Pareto frontier). This returns an empty cube until
+  //! `Optimize()` has been called.
+  const arma::cube& ParetoFront() const { return paretoFront; }
+
+  //! Get the weight initialization policy.
+  const InitPolicyType& InitPolicy() const { return initPolicy; }
+  //! Modify the weight initialization policy.
+  InitPolicyType& InitPolicy() { return initPolicy; }
+
+  //! Get the weight decomposition policy.
+  const DecompPolicyType& DecompPolicy() const { return decompPolicy; }
+  //! Modify the weight decomposition policy.
+  DecompPolicyType& DecompPolicy() { return decompPolicy; }
+
+ private:
+  /**
+   * @brief Randomly selects two members from the population.
+   *
+   * @param subProblemIdx Index of the current subproblem.
+   * @param neighborSize A matrix containing indices of the neighbors.
+   * @return std::tuple<size_t, size_t> The chosen pair of indices.
+   */
+  std::tuple<size_t, size_t> Mating(size_t subProblemIdx,
+                                    const arma::umat& neighborSize,
+                                    bool sampleNeighbor);
+
+  /**
+   * Mutate the child formed by the crossover of two random members of the
+   * population. Uses polynomial mutation.
+   *
+   * @tparam MatType The type of matrix used to store coordinates.
+   * @param child The candidate to be mutated.
+   * @param mutationRate The probability of mutation.
+   * @param lowerBound The lower bound on each variable in the matrix.
+   * @param upperBound The upper bound on each variable in the matrix.
+   * @return The mutated child.
+   */
+  template<typename MatType>
+  void Mutate(MatType& child,
+              double mutationRate,
+              const MatType& lowerBound,
+              const MatType& upperBound);
+
+  /**
+   * Evaluate objectives for the elite population.
+   *
+   * @tparam ArbitraryFunctionType std::tuple of multiple function types.
+   * @tparam MatType Type of matrix to optimize.
+   * @param population The elite population.
+   * @param objectives The set of objectives.
+   * @param calculatedObjectives Vector to store calculated objectives.
+   */
+  template<std::size_t I = 0,
+           typename MatType,
+           typename ...ArbitraryFunctionType>
+  typename std::enable_if<I == sizeof...(ArbitraryFunctionType), void>::type
+  EvaluateObjectives(
+                     std::vector<MatType>&,
+                     std::tuple<ArbitraryFunctionType...>&,
+                     std::vector<arma::Col<typename MatType::elem_type> >&);
+
+  template<std::size_t I = 0,
+           typename MatType,
+           typename ...ArbitraryFunctionType>
+  typename std::enable_if<I < sizeof...(ArbitraryFunctionType), void>::type
+  EvaluateObjectives(
+                     std::vector<MatType>& population,
+                     std::tuple<ArbitraryFunctionType...>& objectives,
+                     std::vector<arma::Col<typename MatType::elem_type> >&
+                     calculatedObjectives);
+
+  //! Size of the population.
+  size_t populationSize;
+
+  //! Maximum number of generations before termination criteria is met.
+  size_t maxGenerations;
+
+  //! Probability of crossover between two members.
+  double crossoverProb;
+
+  //! The probability that two elements will be chosen from the neighbor.
+  double neighborProb;
+
+  //! Number of nearest neighbours of weights to consider.
+  size_t neighborSize;
+
+  //! The crowding degree of the mutation. Higher value produces a mutant
+  //! resembling its parent.
+  double distributionIndex;
+
+  //! Amplification factor for differentiation.
+  double differentialWeight;
+
+  //! Maximum number of childs which can replace the parent. Higher value
+  //! leads to a loss of diversity.
+  size_t maxReplace;
+
+  //! A small numeric value to be added to the weights after initialization.
+  //! Prevents zero value inside inited weights.
+  double epsilon;
+
+  //! Lower bound on each variable in the variable space.
+  arma::vec lowerBound;
+
+  //! Upper bound on each variable in the variable space.
+  arma::vec upperBound;
+
+  //! The set of all the Pareto optimal points.
+  //! Stored after Optimize() is called.
+  arma::cube paretoSet;
+
+  //! The set of all the Pareto optimal objective vectors.
+  //! Stored after Optimize() is called.
+  arma::cube paretoFront;
+
+  //! Policy to initialize the reference directions (weights) matrix.
+  InitPolicyType initPolicy;
+
+  //! Policy to decompose the weights.
+  DecompPolicyType decompPolicy;
+};
+
+using DefaultMOEAD = MOEAD<Uniform, Tchebycheff>;
+using BBSMOEAD = MOEAD<BayesianBootstrap, Tchebycheff>;
+using DirichletMOEAD = MOEAD<Dirichlet, Tchebycheff>;
+} // namespace ens
+
+// Include implementation.
+#include "moead_impl.hpp"
+
+#endif

include/ensmallen_bits/moead/moead_impl.hpp

@@ -0,0 +1,441 @@
+/**
+ * @file moead_impl.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * Implementation of the MOEA/D-DE algorithm. Used for multi-objective
+ * optimization problems on arbitrary functions.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more Information.
+ */
+
+#ifndef ENSMALLEN_MOEAD_MOEAD_IMPL_HPP
+#define ENSMALLEN_MOEAD_MOEAD_IMPL_HPP
+
+#include "moead.hpp"
+#include <assert.h>
+
+namespace ens {
+template <typename InitPolicyType, typename DecompPolicyType>
+inline MOEAD<InitPolicyType, DecompPolicyType>::
+MOEAD(const size_t populationSize,
+      const size_t maxGenerations,
+      const double crossoverProb,
+      const double neighborProb,
+      const size_t neighborSize,
+      const double distributionIndex,
+      const double differentialWeight,
+      const size_t maxReplace,
+      const double epsilon,
+      const arma::vec& lowerBound,
+      const arma::vec& upperBound,
+      const InitPolicyType initPolicy,
+      const DecompPolicyType decompPolicy) :
+    populationSize(populationSize),
+    maxGenerations(maxGenerations),
+    crossoverProb(crossoverProb),
+    neighborProb(neighborProb),
+    neighborSize(neighborSize),
+    distributionIndex(distributionIndex),
+    differentialWeight(differentialWeight),
+    maxReplace(maxReplace),
+    epsilon(epsilon),
+    lowerBound(lowerBound),
+    upperBound(upperBound),
+    initPolicy(initPolicy),
+    decompPolicy(decompPolicy)
+  { /* Nothing to do here. */ }
+
+template <typename InitPolicyType, typename DecompPolicyType>
+inline MOEAD<InitPolicyType, DecompPolicyType>::
+MOEAD(const size_t populationSize,
+      const size_t maxGenerations,
+      const double crossoverProb,
+      const double neighborProb,
+      const size_t neighborSize,
+      const double distributionIndex,
+      const double differentialWeight,
+      const size_t maxReplace,
+      const double epsilon,
+      const double lowerBound,
+      const double upperBound,
+      const InitPolicyType initPolicy,
+      const DecompPolicyType decompPolicy) :
+    populationSize(populationSize),
+    maxGenerations(maxGenerations),
+    crossoverProb(crossoverProb),
+    neighborProb(neighborProb),
+    neighborSize(neighborSize),
+    distributionIndex(distributionIndex),
+    differentialWeight(differentialWeight),
+    maxReplace(maxReplace),
+    epsilon(epsilon),
+    lowerBound(lowerBound * arma::ones(1, 1)),
+    upperBound(upperBound * arma::ones(1, 1)),
+    initPolicy(initPolicy),
+    decompPolicy(decompPolicy)
+  { /* Nothing to do here. */ }
+
+//! Optimize the function.
+template <typename InitPolicyType, typename DecompPolicyType>
+template<typename MatType,
+         typename... ArbitraryFunctionType,
+         typename... CallbackTypes>
+typename MatType::elem_type MOEAD<InitPolicyType, DecompPolicyType>::
+Optimize(std::tuple<ArbitraryFunctionType...>& objectives,
+         MatType& iterateIn,
+         CallbackTypes&&... callbacks)
+{
+  // Population Size must be at least 3 for MOEA/D-DE to work.
+  if (populationSize < 3)
+  {
+    throw std::logic_error("MOEA/D-DE::Optimize(): population size should be at least"
+        " 3!");
+  }
+
+  // Convenience typedefs.
+  typedef typename MatType::elem_type ElemType;
+  typedef typename MatTypeTraits<MatType>::BaseMatType BaseMatType;
+
+  BaseMatType& iterate = (BaseMatType&) iterateIn;
+
+  // Make sure that we have the methods that we need.  Long name...
+  traits::CheckArbitraryFunctionTypeAPI<ArbitraryFunctionType...,
+      BaseMatType>();
+  RequireDenseFloatingPointType<BaseMatType>();
+
+  if (neighborSize < 2)
+  {
+    throw std::invalid_argument(
+        "neighborSize should be atleast 2, however "
+        + std::to_string(neighborSize) + " was detected."
+    );
+  }
+
+  if (neighborSize > populationSize - 1u)
+  {
+    std::ostringstream oss;
+    oss << "MOEAD::Optimize(): " << "neighborSize is " << neighborSize
+        << " but populationSize is " << populationSize << "(should be"
+        << " atleast " << (neighborSize + 1u) << ")" << std::endl;
+    throw std::logic_error(oss.str());
+  }
+
+  // Check if lower bound is a vector of a single dimension.
+  if (lowerBound.n_rows == 1)
+    lowerBound = lowerBound(0, 0) * arma::ones(iterate.n_rows, iterate.n_cols);
+
+  // Check if upper bound is a vector of a single dimension.
+  if (upperBound.n_rows == 1)
+    upperBound = upperBound(0, 0) * arma::ones(iterate.n_rows, iterate.n_cols);
+
+  // Check the dimensions of lowerBound and upperBound.
+  assert(lowerBound.n_rows == iterate.n_rows && "The dimensions of "
+      "lowerBound are not the same as the dimensions of iterate.");
+  assert(upperBound.n_rows == iterate.n_rows && "The dimensions of "
+      "upperBound are not the same as the dimensions of iterate.");
+
+  const size_t numObjectives = sizeof...(ArbitraryFunctionType);
+  const size_t numVariables = iterate.n_rows;
+
+  //! Useful temporaries for float-like comparisons.
+  const BaseMatType castedLowerBound = arma::conv_to<BaseMatType>::from(lowerBound);
+  const BaseMatType castedUpperBound = arma::conv_to<BaseMatType>::from(upperBound);
+
+  // Controls early termination of the optimization process.
+  bool terminate = false;
+
+  // The weight matrix. Each vector represents a decomposition subproblem (M X N).
+  const BaseMatType weights = initPolicy.template Generate<BaseMatType>(
+      numObjectives, populationSize, epsilon);
+
+  // 1.1 Storing the indices of nearest neighbors of each weight vector.
+  arma::umat neighborIndices(neighborSize, populationSize);
+  for (size_t i = 0; i < populationSize; ++i)
+  {
+    // Cache the distance between weights[i] and other weights.
+    const arma::Row<ElemType> distances =
+        arma::sqrt(arma::sum(arma::square(weights.col(i) - weights.each_col())));
+    arma::uvec sortedIndices = arma::stable_sort_index(distances);
+    // Ignore distance from self.
+    neighborIndices.col(i) = sortedIndices(arma::span(1, neighborSize));
+  }
+
+  // 1.2 Random generation of the initial population.
+  std::vector<BaseMatType> population(populationSize);
+  for (BaseMatType& individual : population)
+  {
+    individual = arma::randu<BaseMatType>(
+        iterate.n_rows, iterate.n_cols) - 0.5 + iterate;
+
+    // Constrain all genes to be within bounds.
+    individual = arma::min(arma::max(individual, castedLowerBound), castedUpperBound);
+  }
+
+  Info << "MOEA/D-DE initialized successfully. Optimization started." << std::endl;
+
+  std::vector<arma::Col<ElemType>> populationFitness(populationSize);
+  std::fill(populationFitness.begin(), populationFitness.end(),
+      arma::Col<ElemType>(numObjectives, arma::fill::zeros));
+  EvaluateObjectives(population, objectives, populationFitness);
+
+  // 1.3 Initialize the ideal point z.
+  arma::Col<ElemType> idealPoint(numObjectives);
+  idealPoint.fill(std::numeric_limits<ElemType>::max());
+
+  for (const arma::Col<ElemType>& individualFitness : populationFitness)
+    idealPoint = arma::min(idealPoint, individualFitness);
+
+  terminate |= Callback::BeginOptimization(*this, objectives, iterate, callbacks...);
+
+  // 2 The main loop.
+  for (size_t generation = 1; generation <= maxGenerations && !terminate; ++generation)
+  {
+    // Shuffle indexes of subproblems.
+    const arma::uvec shuffle = arma::shuffle(
+        arma::linspace<arma::uvec>(0, populationSize - 1, populationSize));
+    for (size_t subProblemIdx : shuffle)
+    {
+      // 2.1 Randomly select two indices in neighborIndices[subProblemIdx] and use them
+      // to make a child.
+      size_t r1, r2, r3;
+      r1 = subProblemIdx;
+      // Randomly choose to sample from the population or the neighbors.
+      const bool sampleNeighbor = arma::randu() < neighborProb;
+      std::tie(r2, r3) =
+          Mating(subProblemIdx, neighborIndices, sampleNeighbor);
+
+      // 2.2 - 2.3 Reproduction and Repair: Differential Operator followed by
+      // Polynomial Mutation.
+      BaseMatType candidate(iterate.n_rows, iterate.n_cols);
+
+      for (size_t geneIdx = 0; geneIdx < numVariables; ++geneIdx)
+      {
+        if (arma::randu() < crossoverProb)
+        {
+          candidate(geneIdx) = population[r1](geneIdx) +
+              differentialWeight * (population[r2](geneIdx) -
+                  population[r3](geneIdx));
+
+          // Boundary conditions.
+          if (candidate(geneIdx) < castedLowerBound(geneIdx))
+          {
+            candidate(geneIdx) = castedLowerBound(geneIdx) +
+                arma::randu() * (population[r1](geneIdx) - castedLowerBound(geneIdx));
+          }
+          if (candidate(geneIdx) > castedUpperBound(geneIdx))
+          {
+            candidate(geneIdx) = castedUpperBound(geneIdx) -
+                arma::randu() * (castedUpperBound(geneIdx) - population[r1](geneIdx));
+          }
+        }
+        else
+          candidate(geneIdx) = population[r1](geneIdx);
+      }
+
+      Mutate(candidate, 1.0 / static_cast<double>(numVariables),
+          castedLowerBound, castedUpperBound);
+
+      arma::Col<ElemType> candidateFitness(numObjectives);
+      //! Creating temp vectors to pass to EvaluateObjectives.
+      std::vector<BaseMatType> candidateContainer { candidate };
+      std::vector<arma::Col<ElemType>> fitnessContainer { candidateFitness };
+      EvaluateObjectives(candidateContainer, objectives, fitnessContainer);
+      candidateFitness = std::move(fitnessContainer[0]);
+      //! Flush out the dummy containers.
+      fitnessContainer.clear();
+      candidateContainer.clear();
+
+      // 2.4 Update of ideal point.
+      idealPoint = arma::min(idealPoint, candidateFitness);
+
+      // 2.5 Update of the population.
+      size_t replaceCounter = 0;
+      const size_t sampleSize = sampleNeighbor ? neighborSize : populationSize;
+
+      const arma::uvec idxShuffle = arma::shuffle(
+          arma::linspace<arma::uvec>(0, sampleSize - 1, sampleSize));
+
+      for (size_t idx : idxShuffle)
+      {
+        // Preserve diversity by controlling replacement of neighbors
+        // by child solution.
+        if (replaceCounter >= maxReplace)
+          break;
+
+        const size_t pick = sampleNeighbor ?
+            neighborIndices(idx, subProblemIdx) : idx;
+
+        const ElemType candidateDecomposition = decompPolicy.template
+            Apply<arma::Col<ElemType>>(weights.col(pick), idealPoint, candidateFitness);
+        const ElemType parentDecomposition =  decompPolicy.template
+            Apply<arma::Col<ElemType>>(weights.col(pick), idealPoint, populationFitness[pick]);
+
+        if (candidateDecomposition < parentDecomposition)
+        {
+          population[pick] = candidate;
+          populationFitness[pick] = candidateFitness;
+          ++replaceCounter;
+        }
+      }
+    } // End of pass over all subproblems.
+
+    //  The final population itself is the best front.
+    const std::vector<arma::uvec> frontIndices { arma::shuffle(
+        arma::linspace<arma::uvec>(0, populationSize - 1, populationSize)) };
+
+    terminate |= Callback::GenerationalStepTaken(*this, objectives, iterate,
+        populationFitness, frontIndices, callbacks...);
+  } // End of pass over all the generations.
+
+  // Set the candidates from the Pareto Set as the output.
+  paretoSet.set_size(population[0].n_rows, population[0].n_cols, population.size());
+
+  // The Pareto Front is stored, can be obtained via ParetoSet() getter.
+  for (size_t solutionIdx = 0; solutionIdx < population.size(); ++solutionIdx)
+  {
+    paretoSet.slice(solutionIdx) =
+        arma::conv_to<arma::mat>::from(population[solutionIdx]);
+  }
+
+  // Set the candidates from the Pareto Front as the output.
+  paretoFront.set_size(populationFitness[0].n_rows, populationFitness[0].n_cols,
+      populationFitness.size());
+
+  // The Pareto Front is stored, can be obtained via ParetoFront() getter.
+  for (size_t solutionIdx = 0; solutionIdx < populationFitness.size(); ++solutionIdx)
+  {
+    paretoFront.slice(solutionIdx) =
+        arma::conv_to<arma::mat>::from(populationFitness[solutionIdx]);
+  }
+
+  // Assign iterate to first element of the Pareto Set.
+  iterate = population[0];
+
+  Callback::EndOptimization(*this, objectives, iterate, callbacks...);
+
+  ElemType performance = std::numeric_limits<ElemType>::max();
+
+  for (size_t geneIdx = 0; geneIdx < numObjectives; ++geneIdx)
+  {
+    if (arma::accu(populationFitness[geneIdx]) < performance)
+      performance = arma::accu(populationFitness[geneIdx]);
+  }
+
+  return performance;
+}
+
+//! Randomly chooses to select from parents or neighbors.
+template <typename InitPolicyType, typename DecompPolicyType>
+inline std::tuple<size_t, size_t>
+MOEAD<InitPolicyType, DecompPolicyType>::
+Mating(size_t subProblemIdx,
+       const arma::umat& neighborIndices,
+       bool sampleNeighbor)
+{
+  //! Indexes of two points from the sample space.
+  size_t pointA = sampleNeighbor
+      ? neighborIndices(
+            arma::randi(arma::distr_param(0, neighborSize - 1u)), subProblemIdx)
+      : arma::randi(arma::distr_param(0, populationSize - 1u));
+
+  size_t pointB = sampleNeighbor
+      ? neighborIndices(
+            arma::randi(arma::distr_param(0, neighborSize - 1u)), subProblemIdx)
+      : arma::randi(arma::distr_param(0, populationSize - 1u));
+
+  //! If the sampled points are equal, then modify one of them
+  //! within reasonable bounds.
+  if (pointA == pointB)
+  {
+    if (pointA == populationSize - 1u)
+      --pointA;
+    else
+      ++pointA;
+  }
+
+  return std::make_tuple(pointA, pointB);
+}
+
+//! Perform Polynomial mutation of the candidate.
+template <typename InitPolicyType, typename DecompPolicyType>
+template<typename MatType>
+inline void MOEAD<InitPolicyType, DecompPolicyType>::
+Mutate(MatType& candidate,
+       double mutationRate,
+       const MatType& lowerBound,
+       const MatType& upperBound)
+{
+    const size_t numVariables = candidate.n_rows;
+    for (size_t geneIdx = 0; geneIdx < numVariables; ++geneIdx)
+    {
+      // Should this gene be mutated?
+      if (arma::randu() > mutationRate)
+        continue;
+
+      const double geneRange = upperBound(geneIdx) - lowerBound(geneIdx);
+      // Normalised distance from the bounds.
+      const double lowerDelta = (candidate(geneIdx) - lowerBound(geneIdx)) / geneRange;
+      const double upperDelta = (upperBound(geneIdx) - candidate(geneIdx)) / geneRange;
+      const double mutationPower = 1. / (distributionIndex + 1.0);
+      const double rand = arma::randu();
+      double value, perturbationFactor;
+      if (rand < 0.5)
+      {
+        value = 2.0 * rand + (1.0 - 2.0 * rand) *
+            std::pow(upperDelta, distributionIndex + 1.0);
+        perturbationFactor = std::pow(value, mutationPower) - 1.0;
+      }
+      else
+      {
+        value = 2.0 * (1.0 - rand) + 2.0 *(rand - 0.5) *
+            std::pow(lowerDelta, distributionIndex + 1.0);
+        perturbationFactor = 1.0 - std::pow(value, mutationPower);
+      }
+
+      candidate(geneIdx) += perturbationFactor * geneRange;
+    }
+    //! Enforce bounds.
+    candidate = arma::min(arma::max(candidate, lowerBound), upperBound);
+}
+
+//! No objectives to evaluate.
+template <typename InitPolicyType, typename DecompPolicyType>
+template<std::size_t I,
+         typename MatType,
+         typename ...ArbitraryFunctionType>
+typename std::enable_if<I == sizeof...(ArbitraryFunctionType), void>::type
+MOEAD<InitPolicyType, DecompPolicyType>::
+EvaluateObjectives(
+    std::vector<MatType>&,
+    std::tuple<ArbitraryFunctionType...>&,
+    std::vector<arma::Col<typename MatType::elem_type> >&)
+{
+ // Nothing to do here.
+}
+
+//! Evaluate the objectives for the entire population.
+template <typename InitPolicyType, typename DecompPolicyType>
+template<std::size_t I,
+         typename MatType,
+         typename ...ArbitraryFunctionType>
+typename std::enable_if<I < sizeof...(ArbitraryFunctionType), void>::type
+MOEAD<InitPolicyType, DecompPolicyType>::
+EvaluateObjectives(
+    std::vector<MatType>& population,
+    std::tuple<ArbitraryFunctionType...>& objectives,
+    std::vector<arma::Col<typename MatType::elem_type> >& calculatedObjectives)
+{
+  for (size_t i = 0; i < population.size(); i++)
+  {
+    calculatedObjectives[i](I) = std::get<I>(objectives).Evaluate(population[i]);
+    EvaluateObjectives<I+1, MatType, ArbitraryFunctionType...>(population, objectives,
+                                                               calculatedObjectives);
+  }
+}
+
+}  // namespace ens
+
+#endif

include/ensmallen_bits/moead/weight_init_policies/bbs_init.hpp

@@ -0,0 +1,76 @@
+/**
+ * @file bbs_init.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * The Bayesian Bootstrap (BBS) method of Weight Initialization.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#ifndef ENSMALLEN_MOEAD_BBS_HPP
+#define ENSMALLEN_MOEAD_BBS_HPP
+
+namespace ens {
+
+/**
+ * The Bayesian Bootstrap method for initializing weights. Samples are randomly picked from uniform 
+ * distribution followed by sorting and finding adjacent difference. This gives you a list of 
+ * numbers which is guaranteed to sum up to 1.
+ *
+ * @code
+ * @article{rubin1981bayesian,
+ *   title={The bayesian bootstrap},
+ *   author={Rubin, Donald B},
+ *   journal={The annals of statistics},
+ *   pages={130--134},
+ *   year={1981},
+ * @endcode
+ *
+ */
+class BayesianBootstrap
+{
+ public:
+  /**
+   * Constructor for Bayesian Bootstrap policy.
+   */
+  BayesianBootstrap()
+  {
+    /* Nothing to do. */
+  }
+
+  /**
+   * Generate the reference direction matrix.
+   *
+   * @tparam MatType The type of the matrix used for constructing weights.
+   * @param numObjectives The dimensionality of objective space.
+   * @param numPoints The number of reference directions requested.
+   * @param epsilon Handle numerical stability after weight initialization.
+   */
+  template<typename MatType>
+  MatType Generate(const size_t numObjectives,
+                   const size_t numPoints,
+                   const double epsilon)
+  {
+      typedef typename MatType::elem_type ElemType;
+      typedef typename arma::Col<ElemType> VecType;
+
+      MatType weights(numObjectives, numPoints);
+      for (size_t pointIdx = 0; pointIdx < numPoints; ++pointIdx)
+      {
+        VecType referenceDirection(numObjectives + 1, arma::fill::randu);
+        referenceDirection(0) = 0;
+        referenceDirection(numObjectives) = 1;
+        referenceDirection = arma::sort(referenceDirection);
+        referenceDirection = arma::diff(referenceDirection);
+        weights.col(pointIdx) = std::move(referenceDirection) + epsilon;
+      }
+
+      return weights;
+  }
+};
+
+} // namespace ens
+
+#endif

include/ensmallen_bits/moead/weight_init_policies/dirichlet_init.hpp

@@ -0,0 +1,55 @@
+/**
+ * @file dirichlet_init.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * The Dirichlet method of Weight Initialization.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#ifndef ENSMALLEN_MOEAD_DIRICHLET_HPP
+#define ENSMALLEN_MOEAD_DIRICHLET_HPP
+
+namespace ens {
+
+/**
+ * The Dirichlet method for initializing weights. Sampling a 
+ * Dirichlet distribution with parameters set to one returns 
+ * point lying on unit simplex with uniform distribution.
+ */
+class Dirichlet
+{
+ public:
+  /**
+   * Constructor for Dirichlet policy.
+   */
+  Dirichlet()
+  {
+    /* Nothing to do. */
+  }
+
+  /**
+   * Generate the reference direction matrix.
+   *
+   * @tparam MatType The type of the matrix used for constructing weights.
+   * @param numObjectives The dimensionality of objective space.
+   * @param numPoints The number of reference directions requested.
+   * @param epsilon Handle numerical stability after weight initialization.
+   */
+  template<typename MatType>
+  MatType Generate(const size_t numObjectives,
+                   const size_t numPoints,
+                   const double epsilon)
+  {
+    MatType weights = arma::randg<MatType>(numObjectives, numPoints,
+        arma::distr_param(1.0, 1.0)) + epsilon;
+    // Normalize each column.
+    return arma::normalise(weights, 1, 0);
+  }
+};
+
+} // namespace ens
+
+#endif

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp

@@ -0,0 +1,208 @@
+/**
+ * @file uniform_init.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * The Uniform (Das Dennis) methodology of Weight Initialization.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#ifndef ENSMALLEN_MOEAD_UNIFORM_HPP
+#define ENSMALLEN_MOEAD_UNIFORM_HPP
+
+namespace ens {
+
+/**
+ * The Uniform (Das Dennis) method for initializing weights. This algorithm guarantees
+ * that the distance between adjacent points would be uniform.
+ *
+ * For more information, see the following:
+ *
+ * @code
+ * article{zhang2007moea,
+ *   title={MOEA/D: A multiobjective evolutionary algorithm based on decomposition},
+ *   author={Zhang, Qingfu and Li, Hui},
+ *   journal={IEEE Transactions on evolutionary computation},
+ *   pages={712--731},
+ *   year={2007}
+ * @endcode
+ */
+class Uniform
+{
+ public:
+  /**
+   * Constructor for Uniform Weight Initializatoin Policy.
+   */
+  Uniform()
+  {
+    /* Nothing to do. */
+  }
+
+  /**
+   * Generate the reference direction matrix.
+   *
+   * @tparam MatType The type of the matrix used for constructing weights.
+   * @param numObjectives The dimensionality of objective space.
+   * @param numPoints The number of reference directions requested.
+   * @param epsilon Handle numerical stability after weight initialization.
+   */
+  template<typename MatType>
+  MatType Generate(size_t numObjectives,
+                   size_t numPoints,
+                   double epsilon)
+  {
+    size_t numPartitions  = FindNumParitions(numObjectives, numPoints);
+    size_t validNumPoints = FindNumUniformPoints(numObjectives, numPartitions);
+
+    //! The requested number of points is not matching any partition number.
+    if (numPoints != validNumPoints)
+    {
+      size_t nextValidNumPoints = FindNumUniformPoints(numObjectives, numPartitions + 1);
+      std::ostringstream oss;
+      oss << "DasDennis::Generate(): " << "The requested numPoints " << numPoints
+          << " cannot be generated uniformly.\n " << "Either choose numPoints as "
+          << validNumPoints << " (numPartition = " << numPartitions << ") or "
+          << "numPoints as " << nextValidNumPoints << " (numPartition = "
+          << numPartitions + 1 << ").";
+      throw std::logic_error(oss.str());
+    }
+
+    return DasDennis<MatType>(numObjectives, numPoints,
+        numPartitions, epsilon);
+  }
+
+ private:
+  /**
+   * Finds the number of points which can be sampled uniformly from a 
+   * unit simplex given the number of partitions.
+   */
+  size_t FindNumUniformPoints(const size_t numObjectives,
+                              const size_t numPartitions)
+  {
+    //! O(N) algorithm to calculate binomial coefficient.
+    //! Source: https://www.geeksforgeeks.org/space-and-time-efficient-binomial-coefficient/
+    auto BinomialCoefficient =
+        [](size_t n, size_t k) -> size_t
+        {
+          size_t retval = 1;
+          // Since, C(n, k) = C(n, n - k).
+          if (k > n - k)
+            k = n - k;
+
+          // [n * (n - 1) * .... * (n - k + 1)] / [k * (k - 1) * .... * 1].
+          for (size_t i = 0; i < k; ++i)
+          {
+            retval *= (n - i);
+            retval /= (i + 1);
+          }
+
+          return retval;
+        };
+    return BinomialCoefficient(numObjectives + numPartitions - 1, numPartitions);
+  }
+
+  /**
+   *  Calculates the appropriate number of partitions such that, the binomial
+   *  coefficient value is closest to the number of points requested.
+   */
+  size_t FindNumParitions(size_t numObjectives, size_t numPoints)
+  {
+    if (numObjectives == 1) return 0;
+    // Iteratively increase numPartitions so that the binomial coefficient
+    // comes near to numPoints;
+    size_t numPartitions {1};
+    size_t sampledNumPoints = FindNumUniformPoints(numPartitions,
+        numObjectives);
+    while (sampledNumPoints <= numPoints)
+    {
+       ++numPartitions;
+       sampledNumPoints = FindNumUniformPoints(numObjectives,
+          numPartitions);
+    }
+
+    return numPartitions - 1;
+  }
+
+  /**
+   * A helper function for DasDennis
+   */
+  template<typename AuxInfoStackType,
+           typename MatType>
+  void DasDennisHelper(AuxInfoStackType& progressStack,
+                       MatType& weights,
+                       const size_t numObjectives,
+                       const size_t numPoints,
+                       const size_t numPartitions,
+                       const double epsilon)
+  {
+    typedef typename MatType::elem_type ElemType;
+    typedef typename arma::Row<ElemType> RowType;
+
+    size_t counter = 0;
+    const ElemType delta = 1.0 / (ElemType)numPartitions;
+
+    while ((counter < numPoints) && !progressStack.empty())
+    {
+      MatType point{};
+      size_t beta{};
+      std::tie(point, beta) = progressStack.back();
+      progressStack.pop_back();
+
+      if (point.size() + 1 == numObjectives)
+      {
+        point.insert_rows(point.n_rows, RowType(1).fill(
+            delta * static_cast<ElemType>(beta)));
+        weights.col(counter) = point + epsilon;
+        ++counter;
+      }
+
+      else
+      {
+        for (size_t i = 0; i <= beta; ++i)
+        {
+          MatType pointClone(point);
+          pointClone.insert_rows(pointClone.n_rows, RowType(1).fill(
+              delta * static_cast<ElemType>(i)));
+          progressStack.push_back({pointClone, beta - i});
+        }
+      }
+    }
+  }
+
+  /**
+   * Generates the weight matrix after verifying the
+   * validity of the parameters.
+   */
+  template <typename MatType>
+  MatType DasDennis(const size_t numObjectives,
+                    const size_t numPoints,
+                    const size_t numPartitions,
+                    const double epsilon)
+  {
+    //! Holds auxillary information required for the helper function. 
+    //! Holds the current point and beta value.
+    using AuxContainer = std::pair<MatType, size_t>;
+
+    std::vector<AuxContainer> progressStack{};
+    //! Init the progress stack.
+    progressStack.push_back({{}, numPartitions});
+    MatType weights(numObjectives, numPoints);
+    weights.fill(arma::datum::nan);
+    DasDennisHelper<decltype(progressStack), MatType>(
+        progressStack,
+        weights,
+        numObjectives,
+        numPoints,
+        numPartitions,
+        epsilon);
+
+    return weights;
+  }
+
+};
+
+} // namespace ens
+
+#endif

include/ensmallen_bits/nsga2/nsga2.hpp

@@ -1,6 +1,7 @@
 /**
  * @file nsga2.hpp
  * @author Sayan Goswami
+ * @author Nanubala Gnana Sai
  *
  * NSGA-II is a multi-objective optimization algorithm, widely used in
  * many real-world applications. NSGA-II generates offsprings using
@@ -50,7 +51,8 @@ namespace ens {
  * see the documentation on function types included with this distribution or
  * on the ensmallen website.
  */
-class NSGA2 {
+class NSGA2
+{
  public:
   /**
    * Constructor for the NSGA2 optimizer.
@@ -168,9 +170,33 @@ class NSGA2 {
   //! Modify value of upperBound.
   arma::vec& UpperBound() { return upperBound; }
 
-  //! Retrieve the best front (the Pareto frontier).  This returns an empty vector until `Optimize()`
-  //! has been called.
-  const std::vector<arma::mat>& Front() const { return bestFront; }
+  //! Retrieve the Pareto optimal points in variable space. This returns an empty cube
+  //! until `Optimize()` has been called.
+  const arma::cube& ParetoSet() const { return paretoSet; }
+
+  //! Retrieve the best front (the Pareto frontier). This returns an empty cube until
+  //! `Optimize()` has been called.
+  const arma::cube& ParetoFront() const { return paretoFront; }
+
+  /**
+   * Retrieve the best front (the Pareto frontier).  This returns an empty
+   * vector until `Optimize()` has been called.  Note that this function is
+   * deprecated and will be removed in ensmallen 3.x!  Use `ParetoFront()`
+   * instead.
+   */
+  ens_deprecated const std::vector<arma::mat>& Front()
+  {
+    if (rcFront.size() == 0)
+    {
+      // Match the old return format.
+      for (size_t i = 0; i < paretoFront.n_slices; ++i)
+      {
+        rcFront.push_back(arma::mat(paretoFront.slice(i)));
+      }
+    }
+
+    return rcFront;
+  }
 
  private:
   /**
@@ -188,7 +214,7 @@ class NSGA2 {
   typename std::enable_if<I == sizeof...(ArbitraryFunctionType), void>::type
   EvaluateObjectives(std::vector<MatType>&,
                      std::tuple<ArbitraryFunctionType...>&,
-                     std::vector<arma::Col<double> >&);
+                     std::vector<arma::Col<typename MatType::elem_type> >&);
 
   template<std::size_t I = 0,
            typename MatType,
@@ -196,7 +222,8 @@ class NSGA2 {
   typename std::enable_if<I < sizeof...(ArbitraryFunctionType), void>::type
   EvaluateObjectives(std::vector<MatType>& population,
                      std::tuple<ArbitraryFunctionType...>& objectives,
-                     std::vector<arma::Col<double> >& calculatedObjectives);
+                     std::vector<arma::Col<typename MatType::elem_type> >&
+                     calculatedObjectives);
 
   /**
    * Reproduce candidates from the elite population to generate a new
@@ -210,8 +237,8 @@ class NSGA2 {
    */
   template<typename MatType>
   void BinaryTournamentSelection(std::vector<MatType>& population,
-                                 const arma::vec& lowerBound,
-                                 const arma::vec& upperBound);
+                                 const MatType& lowerBound,
+                                 const MatType& upperBound);
 
   /**
    * Crossover two parents to create a pair of new children.
@@ -239,8 +266,8 @@ class NSGA2 {
    */
   template<typename MatType>
   void Mutate(MatType& child,
-              const arma::vec& lowerBound,
-              const arma::vec& upperBound);
+              const MatType& lowerBound,
+              const MatType& upperBound);
 
   /**
    * Sort the candidate population using their domination count and the set of
@@ -281,11 +308,15 @@ class NSGA2 {
    * Assigns crowding distance metric for sorting.
    *
    * @param front The previously generated Pareto fronts.
-   * @param objectives The set of objectives.
-   * @param crowdingDistance The previously calculated objectives.
+   * @param calculatedObjectives The previously calculated objectives.
+   * @param crowdingDistance The crowding distance for each individual in
+   *    the population.
    */
-  void CrowdingDistanceAssignment(const std::vector<size_t>& front,
-                                  std::vector<double>& crowdingDistance);
+  template <typename MatType>
+  void CrowdingDistanceAssignment(
+      const std::vector<size_t>& front,
+      std::vector<arma::Col<typename MatType::elem_type>>& calculatedObjectives,
+      std::vector<typename MatType::elem_type>& crowdingDistance);
 
   /**
    * The operator used in the crowding distance based sorting.
@@ -299,13 +330,15 @@ class NSGA2 {
    * @param idxQ The index of the second cadidate from the elite population
    *     being sorted.
    * @param ranks The previously calculated ranks.
-   * @param crowdingDistance The previously calculated objectives.
+   * @param crowdingDistance The crowding distance for each individual in
+   *    the population.
    * @return true if the first candidate is preferred, otherwise, false.
    */
+  template<typename MatType>
   bool CrowdingOperator(size_t idxP,
                         size_t idxQ,
                         const std::vector<size_t>& ranks,
-                        const std::vector<double>& crowdingDistance);
+                        const std::vector<typename MatType::elem_type>& crowdingDistance);
 
   //! The number of objectives being optimised for.
   size_t numObjectives;
@@ -337,8 +370,18 @@ class NSGA2 {
   //! Upper bound of the initial swarm.
   arma::vec upperBound;
 
-  //! Best front, stored after Optimize() is called.
-  std::vector<arma::mat> bestFront;
+  //! The set of all the Pareto optimal points.
+  //! Stored after Optimize() is called.
+  arma::cube paretoSet;
+
+  //! The set of all the Pareto optimal objective vectors.
+  //! Stored after Optimize() is called.
+  arma::cube paretoFront;
+
+  //! A different representation of the Pareto front, for reverse compatibility
+  //! purposes.  This can be removed when ensmallen 3.x is released!  (Along
+  //! with `Front()`.)  This is only populated when `Front()` is called.
+  std::vector<arma::mat> rcFront;
 };
 
 } // namespace ens

include/ensmallen_bits/nsga2/nsga2_impl.hpp

@@ -1,6 +1,7 @@
 /**
  * @file nsga2_impl.hpp
  * @author Sayan Goswami
+ * @author Nanubala Gnana Sai
  *
  * Implementation of the NSGA-II algorithm. Used for multi-objective
  * optimization problems on arbitrary functions.
@@ -27,6 +28,8 @@ inline NSGA2::NSGA2(const size_t populationSize,
                     const double epsilon,
                     const arma::vec& lowerBound,
                     const arma::vec& upperBound) :
+    numObjectives(0),
+    numVariables(0),
     populationSize(populationSize),
     maxGenerations(maxGenerations),
     crossoverProb(crossoverProb),
@@ -45,6 +48,8 @@ inline NSGA2::NSGA2(const size_t populationSize,
                     const double epsilon,
                     const double lowerBound,
                     const double upperBound) :
+    numObjectives(0),
+    numVariables(0),
     populationSize(populationSize),
     maxGenerations(maxGenerations),
     crossoverProb(crossoverProb),
@@ -61,7 +66,7 @@ template<typename MatType,
          typename... CallbackTypes>
 typename MatType::elem_type NSGA2::Optimize(
     std::tuple<ArbitraryFunctionType...>& objectives,
-    MatType& iterate,
+    MatType& iterateIn,
     CallbackTypes&&... callbacks)
 {
   // Make sure for evolution to work at least four candidates are present.
@@ -71,11 +76,22 @@ typename MatType::elem_type NSGA2::Optimize(
         " least 4, and, a multiple of 4!");
   }
 
+  // Convenience typedefs.
+  typedef typename MatType::elem_type ElemType;
+  typedef typename MatTypeTraits<MatType>::BaseMatType BaseMatType;
+
+  BaseMatType& iterate = (BaseMatType&) iterateIn;
+
+  // Make sure that we have the methods that we need.  Long name...
+  traits::CheckArbitraryFunctionTypeAPI<ArbitraryFunctionType...,
+      BaseMatType>();
+  RequireDenseFloatingPointType<BaseMatType>();
+
   // Check if lower bound is a vector of a single dimension.
   if (lowerBound.n_rows == 1)
     lowerBound = lowerBound(0, 0) * arma::ones(iterate.n_rows, iterate.n_cols);
 
-  // Check if lower bound is a vector of a single dimension.
+  // Check if upper bound is a vector of a single dimension.
   if (upperBound.n_rows == 1)
     upperBound = upperBound(0, 0) * arma::ones(iterate.n_rows, iterate.n_cols);
 
@@ -85,29 +101,29 @@ typename MatType::elem_type NSGA2::Optimize(
   assert(upperBound.n_rows == iterate.n_rows && "The dimensions of "
       "upperBound are not the same as the dimensions of iterate.");
 
-  // Convenience typedefs.
-  typedef typename MatType::elem_type ElemType;
-
   numObjectives = sizeof...(ArbitraryFunctionType);
   numVariables = iterate.n_rows;
 
   // Cache calculated objectives.
-  std::vector<arma::Col<ElemType> > calculatedObjectives;
-  // Pre-allocate space for the calculated objectives.
-  calculatedObjectives.resize(populationSize);
+  std::vector<arma::Col<ElemType> > calculatedObjectives(populationSize);
 
   // Population size reserved to 2 * populationSize + 1 to accommodate
   // for the size of intermediate candidate population.
-  std::vector<MatType> population;
+  std::vector<BaseMatType> population;
   population.reserve(2 * populationSize + 1);
 
   // Pareto fronts, initialized during non-dominated sorting.
+  // Stores indices of population belonging to a certain front.
   std::vector<std::vector<size_t> > fronts;
   // Initialised in CrowdingDistanceAssignment.
-  std::vector<double> crowdingDistance;
+  std::vector<ElemType> crowdingDistance;
   // Initialised during non-dominated sorting.
   std::vector<size_t> ranks;
 
+  //! Useful temporaries for float-like comparisons.
+  const BaseMatType castedLowerBound = arma::conv_to<BaseMatType>::from(lowerBound);
+  const BaseMatType castedUpperBound = arma::conv_to<BaseMatType>::from(upperBound);
+
   // Controls early termination of the optimization process.
   bool terminate = false;
 
@@ -115,87 +131,103 @@ typename MatType::elem_type NSGA2::Optimize(
   // starting point.
   for (size_t i = 0; i < populationSize; i++)
   {
-    population.push_back(arma::randu<MatType>(iterate.n_rows,
+    population.push_back(arma::randu<BaseMatType>(iterate.n_rows,
         iterate.n_cols) - 0.5 + iterate);
+
+    // Constrain all genes to be within bounds.
+    population[i] = arma::min(arma::max(population[i], castedLowerBound), castedUpperBound);
   }
 
   Info << "NSGA2 initialized successfully. Optimization started." << std::endl;
 
-  // Evaluate the fitness before optimization.
-  for (size_t i = 0; i < population.size(); i++)
-    calculatedObjectives[i] = arma::Col<ElemType>(numObjectives, arma::fill::zeros);
-  EvaluateObjectives(population, objectives, calculatedObjectives);
-
   // Iterate until maximum number of generations is obtained.
   terminate |= Callback::BeginOptimization(*this, objectives, iterate, callbacks...);
 
   for (size_t generation = 1; generation <= maxGenerations && !terminate; generation++)
   {
     Info << "NSGA2: iteration " << generation << "." << std::endl;
-    terminate |= Callback::StepTaken(*this, objectives, iterate, callbacks...);
 
     // Create new population of candidate from the present elite population.
     // Have P_t, generate G_t using P_t.
-    BinaryTournamentSelection(population, lowerBound, upperBound);
+    BinaryTournamentSelection(population, castedLowerBound, castedUpperBound);
 
     // Evaluate the objectives for the new population.
     calculatedObjectives.resize(population.size());
-    for (size_t i = 0; i < population.size(); i++)
-      calculatedObjectives[i] = arma::Col<ElemType>(numObjectives, arma::fill::zeros);
+    std::fill(calculatedObjectives.begin(), calculatedObjectives.end(),
+        arma::Col<ElemType>(numObjectives, arma::fill::zeros));
     EvaluateObjectives(population, objectives, calculatedObjectives);
 
     // Perform fast non dominated sort on P_t ∪ G_t.
     ranks.resize(population.size());
-    FastNonDominatedSort<MatType>(fronts, ranks, calculatedObjectives);
+    FastNonDominatedSort<BaseMatType>(fronts, ranks, calculatedObjectives);
 
     // Perform crowding distance assignment.
     crowdingDistance.resize(population.size());
-
+    std::fill(crowdingDistance.begin(), crowdingDistance.end(), 0.);
     for (size_t fNum = 0; fNum < fronts.size(); fNum++)
     {
-      CrowdingDistanceAssignment(fronts[fNum], crowdingDistance);
+      CrowdingDistanceAssignment<BaseMatType>(
+          fronts[fNum], calculatedObjectives, crowdingDistance);
     }
 
     // Sort based on crowding distance.
     std::sort(population.begin(), population.end(),
-              [this, ranks, crowdingDistance, population](MatType candidateP,
-                                                          MatType candidateQ)
-              {
-                size_t idxP, idxQ;
-                for (size_t i = 0; i < population.size(); i++)
-                {
-                  if (arma::approx_equal(population[i], candidateP, "absdiff", epsilon))
-                    idxP = i;
+      [this, ranks, crowdingDistance, population]
+        (BaseMatType candidateP, BaseMatType candidateQ)
+          {
+            size_t idxP{}, idxQ{};
+            for (size_t i = 0; i < population.size(); i++)
+            {
+              if (arma::approx_equal(population[i], candidateP, "absdiff", epsilon))
+                idxP = i;
 
-                  if (arma::approx_equal(population[i], candidateQ, "absdiff", epsilon))
-                    idxQ = i;
-                }
+              if (arma::approx_equal(population[i], candidateQ, "absdiff", epsilon))
+                idxQ = i;
+            }
 
-                return CrowdingOperator(idxP, idxQ, ranks, crowdingDistance);
-              }
+            return CrowdingOperator<BaseMatType>(idxP, idxQ, ranks, crowdingDistance);
+          }
     );
 
     // Yield a new population P_{t+1} of size populationSize.
+    // Discards unfit population from the R_{t} to yield P_{t+1}.
     population.resize(populationSize);
+
+    terminate |= Callback::GenerationalStepTaken(*this, objectives, iterate,
+        calculatedObjectives, fronts, callbacks...);
   }
 
-  // Set the candidates from the best front as the output.
-  std::vector<MatType> front;
+  // Set the candidates from the Pareto Set as the output.
+  paretoSet.set_size(population[0].n_rows, population[0].n_cols, fronts[0].size());
+  // The Pareto Set is stored, can be obtained via ParetoSet() getter.
+  for (size_t solutionIdx = 0; solutionIdx < fronts[0].size(); ++solutionIdx)
+  {
+    paretoSet.slice(solutionIdx) =
+      arma::conv_to<arma::mat>::from(population[fronts[0][solutionIdx]]);
+  }
 
-  for (size_t f: fronts[0])
-    front.push_back(population[f]);
+  // Set the candidates from the Pareto Front as the output.
+  paretoFront.set_size(calculatedObjectives[0].n_rows, calculatedObjectives[0].n_cols,
+      fronts[0].size());
+  // The Pareto Front is stored, can be obtained via ParetoFront() getter.
+  for (size_t solutionIdx = 0; solutionIdx < fronts[0].size(); ++solutionIdx)
+  {
+    paretoFront.slice(solutionIdx) =
+      arma::conv_to<arma::mat>::from(calculatedObjectives[fronts[0][solutionIdx]]);
+  }
 
-  // bestFront is stored, can be obtained by the Front() getter.
-  bestFront = front;
+  // Clear rcFront, in case it is later requested by the user for reverse
+  // compatibility reasons.
+  rcFront.clear();
 
-  // Assign iterate to first element of the best front.
-  iterate = bestFront[0];
+  // Assign iterate to first element of the Pareto Set.
+  iterate = population[fronts[0][0]];
 
   Callback::EndOptimization(*this, objectives, iterate, callbacks...);
 
   ElemType performance = std::numeric_limits<ElemType>::max();
 
-  for(arma::Col<ElemType> objective: calculatedObjectives)
+  for (const arma::Col<ElemType>& objective: calculatedObjectives)
     if (arma::accu(objective) < performance)
       performance = arma::accu(objective);
 
@@ -210,7 +242,7 @@ typename std::enable_if<I == sizeof...(ArbitraryFunctionType), void>::type
 NSGA2::EvaluateObjectives(
     std::vector<MatType>&,
     std::tuple<ArbitraryFunctionType...>&,
-    std::vector<arma::Col<double> >&)
+    std::vector<arma::Col<typename MatType::elem_type> >&)
 {
   // Nothing to do here.
 }
@@ -223,7 +255,7 @@ typename std::enable_if<I < sizeof...(ArbitraryFunctionType), void>::type
 NSGA2::EvaluateObjectives(
     std::vector<MatType>& population,
     std::tuple<ArbitraryFunctionType...>& objectives,
-    std::vector<arma::Col<double> >& calculatedObjectives)
+    std::vector<arma::Col<typename MatType::elem_type> >& calculatedObjectives)
 {
   for (size_t i = 0; i < populationSize; i++)
   {
@@ -236,8 +268,8 @@ NSGA2::EvaluateObjectives(
 //! Reproduce and generate new candidates.
 template<typename MatType>
 inline void NSGA2::BinaryTournamentSelection(std::vector<MatType>& population,
-                                             const arma::vec& lowerBound,
-                                             const arma::vec& upperBound)
+                                             const MatType& lowerBound,
+                                             const MatType& upperBound)
 {
   std::vector<MatType> children;
 
@@ -292,20 +324,14 @@ inline void NSGA2::Crossover(MatType& childA,
 //! Perform mutation of the candidates weights with some noise.
 template<typename MatType>
 inline void NSGA2::Mutate(MatType& child,
-                          const arma::vec& lowerBound,
-                          const arma::vec& upperBound)
+                          const MatType& lowerBound,
+                          const MatType& upperBound)
 {
   child += (arma::randu<MatType>(child.n_rows, child.n_cols) < mutationProb) %
       (mutationStrength * arma::randn<MatType>(child.n_rows, child.n_cols));
 
   // Constrain all genes to be between bounds.
-  for (size_t idx = 0; idx < numVariables; idx++)
-  {
-    if (child[idx] < lowerBound(idx))
-      child[idx] = lowerBound(idx);
-    else if (child[idx] > upperBound(idx))
-      child[idx] = upperBound(idx);
-  }
+  child = arma::min(arma::max(child, lowerBound), upperBound);
 }
 
 //! Sort population into Pareto fronts.
@@ -344,7 +370,7 @@ inline void NSGA2::FastNonDominatedSort(
 
   size_t i = 0;
 
-  while (fronts[i].size() > 0)
+  while (!fronts[i].empty())
   {
     std::vector<size_t> nextFront;
 
@@ -365,6 +391,8 @@ inline void NSGA2::FastNonDominatedSort(
     i++;
     fronts.push_back(nextFront);
   }
+  // Remove the empty final set.
+  fronts.pop_back();
 }
 
 //! Check if a candidate Pareto dominates another candidate.
@@ -393,37 +421,59 @@ inline bool NSGA2::Dominates(
 }
 
 //! Assign crowding distance to the population.
-inline void NSGA2::CrowdingDistanceAssignment(const std::vector<size_t>& front,
-                                              std::vector<double>& crowdingDistance)
+template <typename MatType>
+inline void NSGA2::CrowdingDistanceAssignment(
+	const std::vector<size_t>& front,
+	std::vector<arma::Col<typename MatType::elem_type>>& calculatedObjectives,
+	std::vector<typename MatType::elem_type>& crowdingDistance)
 {
-  if (front.size() > 0)
+  // Convenience typedefs.
+  typedef typename MatType::elem_type ElemType;
+
+  size_t fSize = front.size();
+  // Stores the sorted indices of the fronts.
+  arma::uvec sortedIdx  = arma::regspace<arma::uvec>(0, 1, fSize - 1);
+
+  for (size_t m = 0; m < numObjectives; m++)
   {
-    for (size_t elem: front)
-      crowdingDistance[elem] = 0;
+    // Cache fValues of individuals for current objective.
+    arma::Col<ElemType> fValues(fSize);
+    std::transform(front.begin(), front.end(), fValues.begin(),
+      [&](const size_t& individual)
+        {
+          return calculatedObjectives[individual](m);
+        });
 
-    size_t fSize = front.size();
+    // Sort front indices by ascending fValues for current objective.
+    std::sort(sortedIdx.begin(), sortedIdx.end(),
+      [&](const size_t& frontIdxA, const size_t& frontIdxB)
+        {
+          return (fValues(frontIdxA) < fValues(frontIdxB));
+        });
 
-    for (size_t m = 0; m < numObjectives; m++)
+    crowdingDistance[front[sortedIdx(0)]] =
+        std::numeric_limits<ElemType>::max();
+    crowdingDistance[front[sortedIdx(fSize - 1)]] =
+        std::numeric_limits<ElemType>::max();
+    ElemType minFval = fValues(sortedIdx(0));
+    ElemType maxFval = fValues(sortedIdx(fSize - 1));
+    ElemType scale =
+        std::abs(maxFval - minFval) == 0. ? 1. : std::abs(maxFval - minFval);
+
+    for (size_t i = 1; i < fSize - 1; i++)
     {
-      crowdingDistance[front[0]] = std::numeric_limits<double>::max();
-      crowdingDistance[front[fSize - 1]] = std::numeric_limits<double>::max();
-
-      for (size_t i = 1; i < fSize - 1 ; i++)
-      {
-        crowdingDistance[front[i]] += (crowdingDistance[front[i - 1]] -
-            crowdingDistance[front[i + 1]]) /
-            (std::numeric_limits<double>::max() -
-            std::numeric_limits<double>::min());
-      }
+      crowdingDistance[front[sortedIdx(i)]] +=
+          (fValues(sortedIdx(i + 1)) - fValues(sortedIdx(i - 1))) / scale;
     }
   }
 }
 
 //! Comparator for crowding distance based sorting.
+template<typename MatType>
 inline bool NSGA2::CrowdingOperator(size_t idxP,
                                     size_t idxQ,
                                     const std::vector<size_t>& ranks,
-                                    const std::vector<double>& crowdingDistance)
+                                    const std::vector<typename MatType::elem_type>& crowdingDistance)
 {
   if (ranks[idxP] < ranks[idxQ])
     return true;

include/ensmallen_bits/problems/ackley_function.hpp

@@ -59,10 +59,6 @@ class AckleyFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-5.0; 5.0"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -116,6 +112,22 @@ class AckleyFunction
   //! Modify the value used for numerical stability.
   double& Epsilon() { return epsilon; }
 
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("0.02; 0.02"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("0.0; 0.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
+
  private:
   //! The value of the multiplicative constant.
   double c;

include/ensmallen_bits/problems/beale_function.hpp

@@ -52,10 +52,6 @@ class BealeFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-4.5; 4.5"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -98,6 +94,22 @@ class BealeFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("2.8; 0.35"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("3.0; 0.5"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/booth_function.hpp

@@ -52,10 +52,6 @@ class BoothFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-9; -9"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -98,6 +94,22 @@ class BoothFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("-9; -9"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("1.0; 3.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/bukin_function.hpp

@@ -57,10 +57,6 @@ class BukinFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-10; -2.0"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -109,6 +105,22 @@ class BukinFunction
   //! Modify the value used for numerical stability.
   double& Epsilon() { return epsilon; }
 
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("-10; -2.0"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("-10.0; 1.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
+
  private:
   //! The value used for numerical stability.
   double epsilon;

include/ensmallen_bits/problems/colville_function.hpp

@@ -53,10 +53,6 @@ class ColvilleFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-5; 3; 1; -9"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -99,6 +95,22 @@ class ColvilleFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient) const;
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("-5; 3; 1; -9"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("1; 1; 1; 1"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/cross_in_tray_function.hpp

@@ -57,10 +57,6 @@ class CrossInTrayFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("0; 0"); }
-
   /*
    * Evaluate a function for a particular batch-size.
    *
@@ -80,6 +76,14 @@ class CrossInTrayFunction
    */
   template<typename MatType>
   typename MatType::elem_type Evaluate(const MatType& coordinates) const;
+
+  // Note: GetInitialPoint() is not required for using ensmallen to optimize
+  // this function!  It is specifically used as a convenience just for
+  // ensmallen's testing infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("0; 0"); }
 };
 
 } // namespace test

include/ensmallen_bits/problems/drop_wave_function.hpp

@@ -52,10 +52,6 @@ class DropWaveFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("0.5; 0.5"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -98,6 +94,22 @@ class DropWaveFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("0.5; 0.5"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("0.0; 0.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/easom_function.hpp

@@ -52,10 +52,6 @@ class EasomFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-90.0; 90.0"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -98,6 +94,22 @@ class EasomFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("2.9; 2.9"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("3.14; 3.14"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return -1.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/eggholder_function.hpp

@@ -53,10 +53,6 @@ class EggholderFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-333; -333"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -99,6 +95,22 @@ class EggholderFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("-333; -333"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("512; 404.2319"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return -959.6407; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/fonseca_fleming_function.hpp

@@ -22,8 +22,8 @@ namespace test {
  * The Fonseca Fleming function N.1 is defined by
  *
  * \f[
- * f_1(x) = 1 - \exp(\sum_1^3{(x_i - \frac{1}{\sqrt3})^2})
- * f_2(x) = 1 - \exp(\sum_1^3{(x_i + \frac{1}{\sqrt3})^2})
+ * f_{1}\left(\boldsymbol{x}\right) = 1 - \exp \left[-\sum_{i=1}^{3} \left(x_{i} - \frac{1}{\sqrt{n}} \right)^{2} \right] \\
+ * f_{2}\left(\boldsymbol{x}\right) = 1 - \exp \left[-\sum_{i=1}^{3} \left(x_{i} + \frac{1}{\sqrt{n}} \right)^{2} \right] \\
  * \f]
  *
  * The optimal solutions to this multi-objective function lie in the
@@ -64,16 +64,21 @@ class FonsecaFlemingFunction
   //! Get the starting point.
   MatType GetInitialPoint()
   {
-    return arma::vec(numVariables, 1, arma::fill::zeros);
+    // Convenience typedef.
+    typedef typename MatType::elem_type ElemType;
+
+    return arma::Col<ElemType>(numVariables, 1, arma::fill::zeros);
   }
 
   struct ObjectiveA
   {
     typename MatType::elem_type Evaluate(const MatType& coords)
     {
-        return 1.0f - exp(-pow(coords[0] - 1.0f / sqrt(3), 2) -
-            - pow(coords[1] - 1.0f / sqrt(3), 2)
-            - pow(coords[2] - 1.0f / sqrt(3), 2));
+        return 1.0 - exp(
+             -pow(static_cast<double>(coords[0]) - 1.0 / sqrt(3.0), 2.0)
+             -pow(static_cast<double>(coords[1]) - 1.0 / sqrt(3.0), 2.0)
+             -pow(static_cast<double>(coords[2]) - 1.0 / sqrt(3.0), 2.0)
+        );
     }
   } objectiveA;
 
@@ -81,9 +86,11 @@ class FonsecaFlemingFunction
   {
     typename MatType::elem_type Evaluate(const MatType& coords)
     {
-        return 1.0f - exp(-pow(coords[0] + 1.0f / sqrt(3), 2) -
-            - pow(coords[1] + 1.0f / sqrt(3), 2)
-            - pow(coords[2] + 1.0f / sqrt(3), 2));
+        return 1.0 - exp(
+            -pow(static_cast<double>(coords[0]) + 1.0 / sqrt(3.0), 2.0)
+            -pow(static_cast<double>(coords[1]) + 1.0 / sqrt(3.0), 2.0)
+            -pow(static_cast<double>(coords[2]) + 1.0 / sqrt(3.0), 2.0)
+        );
     }
   } objectiveB;
 

include/ensmallen_bits/problems/generalized_rosenbrock_function.hpp

@@ -61,13 +61,6 @@ class GeneralizedRosenbrockFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return n - 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  const MatType GetInitialPoint() const
-  {
-    return arma::conv_to<MatType>::from(initialPoint);
-  }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -111,6 +104,28 @@ class GeneralizedRosenbrockFunction
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient) const;
 
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  const MatType GetInitialPoint() const
+  {
+    return arma::conv_to<MatType>::from(initialPoint);
+  }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  const MatType GetFinalPoint() const
+  {
+    return arma::ones<MatType>(initialPoint.n_rows, initialPoint.n_cols);
+  }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
+
  private:
   //! Locally-stored Initial point.
   arma::mat initialPoint;

include/ensmallen_bits/problems/goldstein_price_function.hpp

@@ -61,10 +61,6 @@ class GoldsteinPriceFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-2; 2"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -107,6 +103,22 @@ class GoldsteinPriceFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("0.2; -0.5"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("0.0; -1.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 3.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/gradient_descent_test_function.hpp

@@ -25,10 +25,6 @@ class GDTestFunction
   //! Nothing to do for the constructor.
   GDTestFunction() { }
 
-  //! Get the starting point.
-  template<typename MatType>
-  MatType GetInitialPoint() const { return MatType("1; 3; 2"); }
-
   //! Evaluate a function.
   template<typename MatType>
   typename MatType::elem_type Evaluate(const MatType& coordinates) const;
@@ -36,6 +32,22 @@ class GDTestFunction
   //! Evaluate the gradient of a function.
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient) const;
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType>
+  MatType GetInitialPoint() const { return MatType("1; 3; 2"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("0; 0; 0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/levy_function_n13.hpp

@@ -48,10 +48,6 @@ class LevyFunctionN13
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-10; 10"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -94,6 +90,22 @@ class LevyFunctionN13
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("0.9; 1.1"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("1.0; 1.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/matyas_function.hpp

@@ -52,10 +52,6 @@ class MatyasFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-3; 3"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -98,6 +94,22 @@ class MatyasFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("-3; 3"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("0.0; 0.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/mc_cormick_function.hpp

@@ -52,10 +52,6 @@ class McCormickFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-2; 4"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -98,6 +94,22 @@ class McCormickFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("-2; 4"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("-0.54719; -1.54719"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return -1.9133; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/problems.hpp

@@ -42,5 +42,10 @@
 #include "styblinski_tang_function.hpp"
 #include "three_hump_camel_function.hpp"
 #include "wood_function.hpp"
+#include "zdt/zdt1_function.hpp"
+#include "zdt/zdt2_function.hpp"
+#include "zdt/zdt3_function.hpp"
+#include "zdt/zdt4_function.hpp"
+#include "zdt/zdt6_function.hpp"
 
 #endif

include/ensmallen_bits/problems/rastrigin_function.hpp

@@ -44,24 +44,17 @@ class RastriginFunction
    *
    * @param n Number of dimensions for the function.
    */
-  RastriginFunction(const size_t n);
+  RastriginFunction(const size_t n = 2);
 
   /**
    * Shuffle the order of function visitation. This may be called by the
    * optimizer.
    */
- void Shuffle();
+  void Shuffle();
 
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return n; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const
-  {
-    return arma::conv_to<MatType>::from(initialPoint);
-  }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -104,6 +97,29 @@ class RastriginFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const
+  {
+    return arma::conv_to<MatType>::from(initialPoint);
+  }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const
+  {
+    return arma::zeros<MatType>(initialPoint.n_rows, initialPoint.n_cols);
+  }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
+
  private:
   //! Number of dimensions for the function.
   size_t n;

include/ensmallen_bits/problems/rosenbrock_function.hpp

@@ -55,16 +55,6 @@ class RosenbrockFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const
-  {
-    MatType m(2, 1);
-    m[0] = -1.2;
-    m[1] = 1.0;
-    return m;
-  }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -114,6 +104,22 @@ class RosenbrockFunction
   template<typename MatType, typename GradType>
   typename MatType::elem_type EvaluateWithGradient(const MatType& coordinates,
                                                    GradType& gradient) const;
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("-1.2; 1.0"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("1.0; 1.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/rosenbrock_wood_function.hpp

@@ -39,13 +39,6 @@ class RosenbrockWoodFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  const MatType GetInitialPoint() const
-  {
-    return arma::conv_to<MatType>::from(initialPoint);
-  }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -89,6 +82,28 @@ class RosenbrockWoodFunction
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient) const;
 
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  const MatType GetInitialPoint() const
+  {
+    return arma::conv_to<MatType>::from(initialPoint);
+  }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const
+  {
+    return arma::ones<MatType>(initialPoint.n_rows, initialPoint.n_cols);
+  }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
+
  private:
   //! Locally-stored initial point.
   arma::mat initialPoint;

include/ensmallen_bits/problems/schaffer_function_n1.hpp

@@ -63,7 +63,10 @@ class SchafferFunctionN1
   //! Get the starting point.
   MatType GetInitialPoint()
   {
-    return arma::vec(numVariables, 1, arma::fill::zeros);
+    // Convenience typedef.
+    typedef typename MatType::elem_type ElemType;
+
+    return arma::Col<ElemType>(numVariables, 1, arma::fill::zeros);
   }
 
   struct ObjectiveA

include/ensmallen_bits/problems/schaffer_function_n2.hpp

@@ -53,10 +53,6 @@ class SchafferFunctionN2
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-100; 100"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -99,6 +95,22 @@ class SchafferFunctionN2
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("-100; 100"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("0.0; 0.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/schwefel_function.hpp

@@ -55,13 +55,6 @@ class SchwefelFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return n; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const
-  {
-    return arma::conv_to<MatType>::from(initialPoint);
-  }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -104,6 +97,31 @@ class SchwefelFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const
+  {
+    return arma::conv_to<MatType>::from(initialPoint);
+  }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const
+  {
+    MatType result(initialPoint.n_rows, initialPoint.n_cols, arma::fill::none);
+    result.fill(420.9687);
+    return result;
+  }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
+
  private:
   //! Number of dimensions for the function.
   size_t n;

include/ensmallen_bits/problems/sgd_test_function.hpp

@@ -37,10 +37,6 @@ class SGDTestFunction
   //! Return 3 (the number of functions).
   size_t NumFunctions() const { return 3; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("6; -45.6; 6.2"); }
-
   //! Evaluate a function for a particular batch-size.
   template<typename MatType>
   typename MatType::elem_type Evaluate(const MatType& coordinates,
@@ -53,6 +49,22 @@ class SGDTestFunction
                 const size_t begin,
                 GradType& gradient,
                 const size_t batchSize) const;
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("6; -45.6; 6.2"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("0.0; 0.0; 0.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return -1.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/sparse_test_function.hpp

@@ -32,10 +32,6 @@ class SparseTestFunction
   //! Return 4 (the number of features).
   size_t NumFeatures() const { return 4; }
 
-  //! Get the starting point.
-  template<typename MatType>
-  MatType GetInitialPoint() const { return MatType("0 0 0 0;"); }
-
   //! Evaluate a function.
   template<typename MatType>
   typename MatType::elem_type Evaluate(const MatType& coordinates,
@@ -61,6 +57,22 @@ class SparseTestFunction
                        const size_t j,
                        GradType& gradient) const;
 
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType>
+  MatType GetInitialPoint() const { return MatType("0 0 0 0;"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("2.0 1.0 1.5 4.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 123.75; }
+
  private:
   // Each quadratic polynomial is monic. The intercept and coefficient of the
   // first order term is stored.

include/ensmallen_bits/problems/sphere_function.hpp

@@ -45,7 +45,7 @@ class SphereFunction
    *
    * @param n Number of dimensions for the function.
    */
-  SphereFunction(const size_t n);
+  SphereFunction(const size_t n = 2);
 
   /**
    * Shuffle the order of function visitation. This may be called by the
@@ -56,13 +56,6 @@ class SphereFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return n; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const
-  {
-    return arma::conv_to<MatType>::from(initialPoint);
-  }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -106,6 +99,28 @@ class SphereFunction
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
 
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const
+  {
+    return arma::conv_to<MatType>::from(initialPoint);
+  }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const
+  {
+    return arma::zeros<MatType>(initialPoint.n_rows, initialPoint.n_cols);
+  }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
+
  private:
   //! Number of dimensions for the function.
   size_t n;

include/ensmallen_bits/problems/styblinski_tang_function.hpp

@@ -46,7 +46,7 @@ class StyblinskiTangFunction
    *
    * @param n Number of dimensions for the function.
    */
-  StyblinskiTangFunction(const size_t n);
+  StyblinskiTangFunction(const size_t n = 2);
 
   /**
    * Shuffle the order of function visitation. This may be called by the
@@ -57,13 +57,6 @@ class StyblinskiTangFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return n; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const
-  {
-    return arma::conv_to<MatType>::from(initialPoint);
-  }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -107,6 +100,31 @@ class StyblinskiTangFunction
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
 
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const
+  {
+    return arma::conv_to<MatType>::from(initialPoint);
+  }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const
+  {
+    MatType result(initialPoint.n_rows, initialPoint.n_cols);
+    for (size_t i = 0; i < result.n_elem; ++i)
+      result[i] = -2.903534;
+    return result;
+  }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return -39.16599 * n; }
+
  private:
   //! Number of dimensions for the function.
   size_t n;

include/ensmallen_bits/problems/three_hump_camel_function.hpp

@@ -58,10 +58,6 @@ class ThreeHumpCamelFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-4.5; 4.5"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -104,6 +100,22 @@ class ThreeHumpCamelFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient);
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("1; 1"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("0.0; 0.0"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/wood_function.hpp

@@ -59,10 +59,6 @@ class WoodFunction
   //! Return 1 (the number of functions).
   size_t NumFunctions() const { return 1; }
 
-  //! Get the starting point.
-  template<typename MatType = arma::mat>
-  MatType GetInitialPoint() const { return MatType("-3; -1; -3; -1"); }
-
   /**
    * Evaluate a function for a particular batch-size.
    *
@@ -105,6 +101,22 @@ class WoodFunction
    */
   template<typename MatType, typename GradType>
   void Gradient(const MatType& coordinates, GradType& gradient) const;
+
+  // Note: GetInitialPoint(), GetFinalPoint(), and GetFinalObjective() are not
+  // required for using ensmallen to optimize this function!  They are
+  // specifically used as a convenience just for ensmallen's testing
+  // infrastructure.
+
+  //! Get the starting point.
+  template<typename MatType = arma::mat>
+  MatType GetInitialPoint() const { return MatType("-3; -1; -3; -1"); }
+
+  //! Get the final point.
+  template<typename MatType = arma::mat>
+  MatType GetFinalPoint() const { return MatType("1; 1; 1; 1"); }
+
+  //! Get the final objective.
+  double GetFinalObjective() const { return 0.0; }
 };
 
 } // namespace test

include/ensmallen_bits/problems/zdt/zdt1_function.hpp

@@ -0,0 +1,157 @@
+/**
+ * @file zdt1_function.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * Implementation of the first ZDT(Zitzler, Deb and Thiele) test.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#ifndef ENSMALLEN_PROBLEMS_ZDT_ONE_FUNCTION_HPP
+#define ENSMALLEN_PROBLEMS_ZDT_ONE_FUNCTION_HPP
+
+namespace ens {
+namespace test {
+
+/**
+ * The ZDT1 function, defined by:
+ * \f[
+ * g(x) = 1 + 9(\sum_{i=2}^{n} x_i )/(n-1)
+ * f_1(x) = x_1
+ * h(f_1, g) = g(x)[1-\sqrt{f_1/g}\ ]
+ * f_2(x) = g(x) * h(f_1, g)
+ * \f]
+ *
+ * This is a 30-variable problem (n = 30) with a convex optimal set.
+ *
+ * Bounds of the variable space is:
+ * 0 <= x_i <= 1 for i = 1,...,n.
+ *
+ * This should be optimized to g(x) = 1.0, at:
+ * x_1* in [0, 1] ; x_i* = 0 for i = 2,...,n
+ *
+ * For more information, please refer to:
+ *
+ * @code
+ * @article{Zitzler2000,
+ *   title   = {Comparison of multiobjective evolutionary algorithms:
+ *              Empirical results},
+ *   author  = {Zitzler, Eckart and Deb, Kalyanmoy and Thiele, Lothar},
+ *   journal = {Evolutionary computation},
+ *   year    = {2000},
+ *   doi     = {10.1162/106365600568202}
+ * }
+ * @endcode
+ *
+ * @tparam MatType Type of matrix to optimize.
+ */
+  template<typename MatType = arma::mat>
+  class ZDT1
+  {
+   private:
+    size_t numParetoPoints {100};
+    size_t numObjectives {2};
+    size_t numVariables {30};
+
+   public:
+     //! Initialize the ZDT1
+    ZDT1(size_t numParetoPoints = 100) :
+        numParetoPoints(numParetoPoints),
+        objectiveF1(*this),
+        objectiveF2(*this)
+    {/* Nothing to do here. */}
+
+    /**
+     * Evaluate the objectives with the given coordinate.
+     *
+     * @param coords The function coordinates.
+     * @return arma::Col<typename MatType::elem_type>
+     */
+    arma::Col<typename MatType::elem_type> Evaluate(const MatType& coords)
+    {
+      // Convenience typedef.
+      typedef typename MatType::elem_type ElemType;
+
+      arma::Col<ElemType> objectives(numObjectives);
+      objectives(0) = coords[0];
+      ElemType sum = arma::accu(coords(arma::span(1, numVariables - 1), 0));
+      ElemType g = 1. + 9. * sum / (static_cast<ElemType>(numVariables) - 1.);
+      ElemType objectiveRatio = objectives(0) / g;
+      objectives(1) = g * (1. - std::sqrt(objectiveRatio));
+
+      return objectives;
+    }
+
+    //! Get the starting point.
+    MatType GetInitialPoint()
+    {
+      // Convenience typedef.
+      typedef typename MatType::elem_type ElemType;
+
+      return arma::Col<ElemType>(numVariables, 1, arma::fill::zeros);
+    }
+
+    struct ObjectiveF1
+    {
+      ObjectiveF1(ZDT1& zdtClass) : zdtClass(zdtClass)
+      {/*Nothing to do here */}
+
+      typename MatType::elem_type Evaluate(const MatType& coords)
+      {
+        return coords[0];
+      }
+
+      ZDT1& zdtClass;
+    };
+
+    struct ObjectiveF2
+    {
+      ObjectiveF2(ZDT1& zdtClass) : zdtClass(zdtClass)
+      {/*Nothing to do here */}
+
+      typename MatType::elem_type Evaluate(const MatType& coords)
+      {
+        // Convenience typedef.
+        typedef typename MatType::elem_type ElemType;
+
+        size_t numVariables = zdtClass.numVariables;
+        ElemType sum = arma::accu(coords(arma::span(1, numVariables - 1), 0));
+        ElemType g = 1. + 9. * sum / (static_cast<ElemType>(numVariables - 1));
+        ElemType objectiveRatio = zdtClass.objectiveF1.Evaluate(coords) / g;
+
+        return g * (1. - std::sqrt(objectiveRatio));
+      }
+
+      ZDT1& zdtClass;
+    };
+
+    //! Get objective functions.
+    std::tuple<ObjectiveF1, ObjectiveF2> GetObjectives()
+    {
+      return std::make_tuple(objectiveF1, objectiveF2);
+    }
+
+    //! Get the Reference Front.
+    //! Refer PR #273 Ipynb notebook to see the plot of Reference
+    //! Front. The implementation has been taken from pymoo.
+    arma::cube GetReferenceFront()
+    {
+      arma::cube front(2, 1, numParetoPoints);
+      arma::vec x = arma::linspace(0, 1, numParetoPoints);
+      arma::vec y = 1 - arma::sqrt(x);
+      for (size_t idx = 0; idx < numParetoPoints; ++idx)
+        front.slice(idx) = arma::vec{ x(idx), y(idx) };
+
+      return front;
+    }
+
+    ObjectiveF1 objectiveF1;
+    ObjectiveF2 objectiveF2;
+  };
+  } //namespace test
+  } //namespace ens
+
+#endif

include/ensmallen_bits/problems/zdt/zdt2_function.hpp

@@ -0,0 +1,158 @@
+/**
+ * @file zdt2_function.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * Implementation of the second ZDT(Zitzler, Deb and Thiele) test.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#ifndef ENSMALLEN_PROBLEMS_ZDT_TWO_FUNCTION_HPP
+#define ENSMALLEN_PROBLEMS_ZDT_TWO_FUNCTION_HPP
+
+namespace ens {
+namespace test {
+
+/**
+ * The ZDT2 function, defined by:
+ * \f[
+ * g(x) = 1 + 9(\sum_{i=2}^{n} x_i )/(n-1)
+ * f_1(x) = x_1
+ * h(f_1, g) = 1 - (f_1/g)^2
+ * f_2(x) = g(x) * h(f_1, g)
+ * \f]
+ *
+ * This is a 30-variable problem(n = 30) with a
+ * non convex optimal set.
+ *
+ * Bounds of the variable space is:
+ * 0 <= x_i <= 1 for i = 1,..,n.
+ *
+ * This should be optimized to g(x) = 1.0, at:
+ * x_1* in [0, 1] ; x_i* = 0 for i = 2,...,n
+ *
+ * For more information, please refer to:
+ *
+ * @code
+ * @article{Zitzler2000,
+ *   title   = {Comparison of multiobjective evolutionary algorithms:
+ *              Empirical results},
+ *   author  = {Zitzler, Eckart and Deb, Kalyanmoy and Thiele, Lothar},
+ *   journal = {Evolutionary computation},
+ *   year    = {2000},
+ *   doi     = {10.1162/106365600568202}
+ * }
+ * @endcode
+ *
+ * @tparam MatType Type of matrix to optimize.
+ */
+  template<typename MatType = arma::mat>
+  class ZDT2
+  {
+   private:
+    size_t numParetoPoints {100};
+    size_t numObjectives {2};
+    size_t numVariables {30};
+
+   public:
+     //! Initialize the ZDT2
+    ZDT2(size_t numParetoPoints = 100) :
+        numParetoPoints(numParetoPoints),
+        objectiveF1(*this),
+        objectiveF2(*this)
+    {/* Nothing to do here. */}
+
+    /**
+     * Evaluate the objectives with the given coordinate.
+     *
+     * @param coords The function coordinates.
+     * @return arma::Col<typename MatType::elem_type>
+     */
+    arma::Col<typename MatType::elem_type> Evaluate(const MatType& coords)
+    {
+      // Convenience typedef.
+      typedef typename MatType::elem_type ElemType;
+
+      arma::Col<ElemType> objectives(numObjectives);
+      objectives(0) = coords[0];
+      ElemType sum = arma::accu(coords(arma::span(1, numVariables - 1), 0));
+      ElemType g = 1. + 9. * sum / (static_cast<ElemType>(numVariables) - 1.);
+      ElemType objectiveRatio = objectives(0) / g;
+      objectives(1) = g * (1. - std::pow(objectiveRatio, 2));
+
+      return objectives;
+    }
+
+    //! Get the starting point.
+    MatType GetInitialPoint()
+    {
+      // Convenience typedef.
+      typedef typename MatType::elem_type ElemType;
+
+      return arma::Col<ElemType>(numVariables, 1, arma::fill::zeros);
+    }
+
+    struct ObjectiveF1
+    {
+      ObjectiveF1(ZDT2& zdtClass) : zdtClass(zdtClass)
+      {/*Nothing to do here */}
+
+      typename MatType::elem_type Evaluate(const MatType& coords)
+      {
+        return coords[0];
+      }
+
+      ZDT2& zdtClass;
+    };
+
+    struct ObjectiveF2
+    {
+      ObjectiveF2(ZDT2& zdtClass) : zdtClass(zdtClass)
+      {/*Nothing to do here */}
+
+      typename MatType::elem_type Evaluate(const MatType& coords)
+      {
+        // Convenience typedef.
+        typedef typename MatType::elem_type ElemType;
+
+        size_t numVariables = zdtClass.numVariables;
+        ElemType sum = arma::accu(coords(arma::span(1, numVariables - 1), 0));
+        ElemType g = 1. + 9. * sum / (static_cast<ElemType>(numVariables - 1));
+        ElemType objectiveRatio = zdtClass.objectiveF1.Evaluate(coords) / g;
+
+        return g * (1. - std::pow(objectiveRatio, 2));
+      }
+
+      ZDT2& zdtClass;
+    };
+
+    //! Get objective functions.
+    std::tuple<ObjectiveF1, ObjectiveF2> GetObjectives()
+    {
+      return std::make_tuple(objectiveF1, objectiveF2);
+    }
+
+    //! Get the Reference Front.
+    //! Refer PR #273 Ipynb notebook to see the plot of Reference
+    //! Front. The implementation has been taken from pymoo.
+    arma::cube GetReferenceFront()
+    {
+      arma::cube front(2, 1, numParetoPoints);
+      arma::vec x = arma::linspace(0, 1, numParetoPoints);
+      arma::vec y = 1 - arma::square(x);
+      for (size_t idx = 0; idx < numParetoPoints; ++idx)
+        front.slice(idx) = arma::vec{ x(idx), y(idx) };
+
+      return front;
+    }
+
+    ObjectiveF1 objectiveF1;
+    ObjectiveF2 objectiveF2;
+  };
+  } //namespace test
+  } //namespace ens
+
+#endif

include/ensmallen_bits/problems/zdt/zdt3_function.hpp

@@ -0,0 +1,188 @@
+/**
+ * @file zdt3_function.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * Implementation of the third ZDT(Zitzler, Deb and Thiele) test.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#ifndef ENSMALLEN_PROBLEMS_ZDT_THREE_FUNCTION_HPP
+#define ENSMALLEN_PROBLEMS_ZDT_THREE_FUNCTION_HPP
+
+namespace ens {
+namespace test {
+
+/**
+ * The ZDT3 function, defined by:
+ * \f[
+ * g(x) = 1 + 9(\sum_{i=2}^{n} x_i )/(n-1)
+ * f_1(x) = x_1
+ * h(f_1,g) = 1 - \sqrt{f_1/g}  - (f_1/g)sin(10\pi f_1)
+ * f_2(x) = g(x) * h(f_1, g)
+ * \f]
+ *
+ * This is a 30-variable problem(n = 30) with a number
+ * of disconnected optimal fronts.
+ *
+ * Bounds of the variable space is:
+ * 0 <= x_i <= 1 for i = 1,...,n.
+ *
+ * This should be optimized to g(x) = 1.0, at:
+ *
+ * x_1* in [0.0000, 0.0830] OR
+ * x_1* in [0.1822, 0.2577] OR
+ * x_1* in [0.4093, 0.4538] OR
+ * x_1* in [0.6183, 0.6525] OR
+ * x_1* in [0.8233, 0.8518].
+ *
+ * x_i* = 0 for i = 2,...,n.
+ *
+ * For more information, please refer to:
+ *
+ * @code
+ * @article{Zitzler2000,
+ *   title   = {Comparison of multiobjective evolutionary algorithms:
+ *              Empirical results},
+ *   author  = {Zitzler, Eckart and Deb, Kalyanmoy and Thiele, Lothar},
+ *   journal = {Evolutionary computation},
+ *   year    = {2000},
+ *   doi     = {10.1162/106365600568202}
+ * }
+ * @endcode
+ *
+ * @tparam MatType Type of matrix to optimize.
+ */
+  template<typename MatType = arma::mat>
+  class ZDT3
+  {
+   private:
+    size_t numParetoPoints {100};
+    size_t numObjectives {2};
+    size_t numVariables {30};
+
+   public:
+     //! Initialize the ZDT3
+    ZDT3(size_t numParetoPoints = 100) :
+        numParetoPoints(numParetoPoints),
+        objectiveF1(*this),
+        objectiveF2(*this)
+    {/* Nothing to do here. */}
+
+    /**
+     * Evaluate the objectives with the given coordinate.
+     *
+     * @param coords The function coordinates.
+     * @return arma::Col<typename MatType::elem_type>
+     */
+    arma::Col<typename MatType::elem_type> Evaluate(const MatType& coords)
+    {
+      typedef typename MatType::elem_type ElemType;
+
+      arma::Col<ElemType> objectives(numObjectives);
+      objectives(0) = coords[0];
+      ElemType sum = arma::accu(coords(arma::span(1, numVariables - 1), 0));
+      ElemType g = 1. + 9. * sum / (static_cast<ElemType>(numVariables) - 1.);
+      ElemType objectiveRatio = objectives(0) / g;
+      objectives(1) = g * (1. - std::sqrt(objectiveRatio) -
+          (objectiveRatio) * std::sin(10. * arma::datum::pi * coords[0]));
+
+      return objectives;
+    }
+
+    //! Get the starting point.
+    MatType GetInitialPoint()
+    {
+      // Convenience typedef.
+      typedef typename MatType::elem_type ElemType;
+
+      return arma::Col<ElemType>(numVariables, 1, arma::fill::zeros);
+    }
+
+    struct ObjectiveF1
+    {
+      ObjectiveF1(ZDT3& zdtClass) : zdtClass(zdtClass)
+      {/*Nothing to do here */}
+
+      typename MatType::elem_type Evaluate(const MatType& coords)
+      {
+        return coords[0];
+      }
+
+      ZDT3& zdtClass;
+    };
+
+    struct ObjectiveF2
+    {
+      ObjectiveF2(ZDT3& zdtClass) : zdtClass(zdtClass)
+      {/*Nothing to do here */}
+
+      typename MatType::elem_type Evaluate(const MatType& coords)
+      {
+        typedef typename MatType::elem_type ElemType;
+
+        size_t numVariables = zdtClass.numVariables;
+        ElemType sum = arma::accu(coords(arma::span(1, numVariables - 1), 0));
+        ElemType g = 1. + 9. * sum / (static_cast<ElemType>(numVariables - 1));
+        ElemType objectiveRatio = zdtClass.objectiveF1.Evaluate(coords) / g;
+
+        return g * (1. - std::sqrt(objectiveRatio) -
+            (objectiveRatio) * std::sin(10. * arma::datum::pi * coords[0]));
+      }
+
+      ZDT3& zdtClass;
+    };
+
+    //! Get objective functions.
+    std::tuple<ObjectiveF1, ObjectiveF2> GetObjectives()
+    {
+      return std::make_tuple(objectiveF1, objectiveF2);
+    }
+
+    //! Get the Reference Front.
+    //! Refer PR #273 Ipynb notebook to see the plot of Reference
+    //! Front. The implementation has been taken from pymoo.
+    arma::cube GetReferenceFront()
+    {
+      size_t numRegions = 5;
+      size_t regionDensity = std::floor(numParetoPoints / numRegions);
+      size_t apparentParetoPoints = numRegions * regionDensity;
+      arma::cube front(2, 1, apparentParetoPoints);
+
+      arma::mat regions{
+        {0.0, 0.182228780, 0.4093136748,
+         0.6183967944, 0.8233317983},
+        {0.0830015349, 0.2577623634, 0.4538821041,
+         0.6525117038, 0.8518328654}
+      };
+
+      for (size_t regionIdx = 0; regionIdx < numRegions; ++regionIdx)
+      {
+        arma::vec region = regions.col(regionIdx);
+        //! Generate x and y coordinates for the region.
+        arma::vec x = arma::linspace(
+            region(0), region(1), regionDensity);
+        arma::vec y = 1 - arma::sqrt(x) - x
+            % arma::sin(10 * arma::datum::pi * x);
+
+        //! Fill the front with the generated points.
+        for (size_t pointIdx = 0; pointIdx < regionDensity; ++pointIdx)
+        {
+          size_t sliceIdx = regionIdx * regionDensity + pointIdx;
+          front.slice(sliceIdx) = arma::vec{ x(pointIdx), y(pointIdx) };
+        }
+      }
+
+      return front;
+    }
+
+    ObjectiveF1 objectiveF1;
+    ObjectiveF2 objectiveF2;
+  };
+  } //namespace test
+  } //namespace ens
+
+#endif

include/ensmallen_bits/problems/zdt/zdt4_function.hpp

@@ -0,0 +1,160 @@
+/**
+ * @file zdt4_function.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * Implementation of the fourth ZDT(Zitzler, Deb and Thiele) test.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#ifndef ENSMALLEN_PROBLEMS_ZDT_FOUR_FUNCTION_HPP
+#define ENSMALLEN_PROBLEMS_ZDT_FOUR_FUNCTION_HPP
+
+namespace ens {
+namespace test {
+/**
+ * The ZDT4 function, defined by:
+ * \f[
+ * g(x) = 1 + 10(n-1) + \sum_{i=2}^{n}(x_i^2 - 10cos(4\pi x_i))
+ * f_1(x) = x_i
+ * h(f_1,g) = 1 - \sqrt{f_i/g}
+ * f_2(x) = g(x) * h(f_1, g)
+ * \f]
+ *
+ * This is a 10-variable problem(n = 10) with a convex
+ * optimal front. This problem contains several local
+ * optimum, making it difficult to reach the global optimum.
+ *
+ * Bounds of the variable space is:
+ *  0 <= x_1 <= 1;
+ * -10 <= x_i <= 10 for i = 2,...,n.
+ *
+ * This should be optimized to g(x) = 1.0, at:
+ * x_1* in [0, 1] ; x_i* = 0 for i = 2,...,n
+ *
+ * For more information, please refer to:
+ *
+ * @code
+ * @article{Zitzler2000,
+ *   title   = {Comparison of multiobjective evolutionary algorithms:
+ *              Empirical results},
+ *   author  = {Zitzler, Eckart and Deb, Kalyanmoy and Thiele, Lothar},
+ *   journal = {Evolutionary computation},
+ *   year    = {2000},
+ *   doi     = {10.1162/106365600568202}
+ * }
+ * @endcode
+ *
+ * @tparam MatType Type of matrix to optimize.
+ */
+  template<typename MatType = arma::mat>
+  class ZDT4
+  {
+   private:
+    size_t numParetoPoints {100};
+    size_t numObjectives {2};
+    size_t numVariables {10};
+
+   public:
+     //! Initialize the ZDT4
+    ZDT4(size_t numParetoPoints = 100) :
+        numParetoPoints(numParetoPoints),
+        objectiveF1(*this),
+        objectiveF2(*this)
+    {/* Nothing to do here. */}
+
+    /**
+     * Evaluate the objectives with the given coordinate.
+     *
+     * @param coords The function coordinates.
+     * @return arma::Col<typename MatType::elem_type>
+     */
+    arma::Col<typename MatType::elem_type> Evaluate(const MatType& coords)
+    {
+      typedef typename MatType::elem_type ElemType;
+
+      arma::Col<ElemType> objectives(numObjectives);
+      objectives(0) = coords[0];
+      MatType truncatedCoords = coords(arma::span(1, numVariables - 1), 0);
+      ElemType sum = arma::accu(arma::square(truncatedCoords) -
+          10. * arma::cos(4 * arma::datum::pi * truncatedCoords));
+      ElemType g = 1. + 10. * static_cast<ElemType>(numVariables - 1) + sum;
+      ElemType objectiveRatio = objectives(0) / g;
+	    objectives(1) = g * (1. - std::sqrt(objectiveRatio));
+
+      return objectives;
+    }
+
+    //! Get the starting point.
+    MatType GetInitialPoint()
+    {
+      // Convenience typedef.
+      typedef typename MatType::elem_type ElemType;
+
+      return arma::Col<ElemType>(numVariables, 1, arma::fill::zeros);
+    }
+
+    struct ObjectiveF1
+    {
+      ObjectiveF1(ZDT4& zdtClass) : zdtClass(zdtClass)
+      {/*Nothing to do here */}
+
+      typename MatType::elem_type Evaluate(const MatType& coords)
+      {
+        return coords[0];
+      }
+
+      ZDT4& zdtClass;
+    };
+
+    struct ObjectiveF2
+    {
+      ObjectiveF2(ZDT4& zdtClass) : zdtClass(zdtClass)
+      {/*Nothing to do here */}
+
+      typename MatType::elem_type Evaluate(const MatType& coords)
+      {
+        typedef typename MatType::elem_type ElemType;
+
+        size_t numVariables = zdtClass.numVariables;
+        MatType truncatedCoords = coords(arma::span(1, numVariables - 1), 0);
+        ElemType sum = arma::accu(arma::square(truncatedCoords) -
+            10. * arma::cos(4 * arma::datum::pi * truncatedCoords));
+        ElemType g = 1. + 10 * static_cast<ElemType>(numVariables - 1) + sum;
+        ElemType objectiveRatio = zdtClass.objectiveF1.Evaluate(coords) / g;
+
+        return  g * (1. - std::sqrt(objectiveRatio));
+      }
+
+      ZDT4& zdtClass;
+    };
+
+    //! Get objective functions.
+    std::tuple<ObjectiveF1, ObjectiveF2> GetObjectives()
+    {
+      return std::make_tuple(objectiveF1, objectiveF2);
+    }
+
+    //! Get the Reference Front.
+    //! Refer PR #273 Ipynb notebook to see the plot of Reference
+    //! Front. The implementation has been taken from pymoo.
+    arma::cube GetReferenceFront()
+    {
+      arma::cube front(2, 1, numParetoPoints);
+      arma::vec x = arma::linspace(0, 1, numParetoPoints);
+      arma::vec y = 1 - arma::sqrt(x);
+      for (size_t idx = 0; idx < numParetoPoints; ++idx)
+        front.slice(idx) = arma::vec{ x(idx), y(idx) };
+
+      return front;
+    }
+
+    ObjectiveF1 objectiveF1;
+    ObjectiveF2 objectiveF2;
+  };
+  } //namespace test
+  } //namespace ens
+#endif

include/ensmallen_bits/problems/zdt/zdt6_function.hpp

@@ -0,0 +1,162 @@
+/**
+ * @file zdt6_function.hpp
+ * @author Nanubala Gnana Sai
+ *
+ * Implementation of the sixth ZDT(Zitzler, Deb and Thiele) test.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#ifndef ENSMALLEN_PROBLEMS_ZDT_SIX_FUNCTION_HPP
+#define ENSMALLEN_PROBLEMS_ZDT_SIX_FUNCTION_HPP
+
+namespace ens {
+namespace test {
+/**
+ * The ZDT6 function, defined by:
+ * \f[
+ * g(x) = 1 + 9[ \sum_{i=2}^{n}(x_i^2)/9]^{0.25}
+ * f_1(x) = 1 - e^{-4x_1}sin^{6}(6\pi x_1)
+ * h(f1, g) = 1 - (f_1/g)^{2}
+ * f_2(x) = g(x) * h(f_1, g)
+ * \f]
+ *
+ * This is a 10-variable problem(n = 10) with a
+ * non-convex optimal front. The density of the
+ * solutions across optimal region is non-uniform.
+ *
+ * Bounds of the variable space is:
+ * 0 <= x_i <= 1 for i = 1,...,n
+ *
+ * This should be optimized to g(x) = 1.0, at:
+ * x_1* in [0, 1] ; x_i* = 0 for i = 2,...,n
+ *
+ *
+ * For more information, please refer to:
+ *
+ * @code
+ * @article{Zitzler2000,
+ *   title   = {Comparison of multiobjective evolutionary algorithms:
+ *              Empirical results},
+ *   author  = {Zitzler, Eckart and Deb, Kalyanmoy and Thiele, Lothar},
+ *   journal = {Evolutionary computation},
+ *   year    = {2000},
+ *   doi     = {10.1162/106365600568202}
+ * }
+ * @endcode
+ *
+ * @tparam MatType Type of matrix to optimize.
+ */
+  template<typename MatType = arma::mat>
+  class ZDT6
+  {
+   private:
+    size_t numParetoPoints {100};
+    size_t numObjectives {2};
+    size_t numVariables {10};
+
+   public:
+     //! Initialize the ZDT6
+    ZDT6(size_t numParetoPoints = 100) :
+        numParetoPoints(numParetoPoints),
+        objectiveF1(*this),
+        objectiveF2(*this)
+    {/* Nothing to do here. */}
+
+    /**
+     * Evaluate the objectives with the given coordinate.
+     *
+     * @param coords The function coordinates.
+     * @return arma::Col<typename MatType::elem_type>
+     */
+    arma::Col<typename MatType::elem_type> Evaluate(const MatType& coords)
+    {
+      // Convenience typedef.
+      typedef typename MatType::elem_type ElemType;
+
+      arma::Col<ElemType> objectives(numObjectives);
+      objectives(0) = 1. - std::exp(-4 * coords[0]) *
+          std::pow(std::sin(6 * arma::datum::pi * coords[0]), 6);
+      ElemType sum = std::pow(
+          arma::accu(coords(arma::span(1, numVariables - 1), 0)) / 9, 0.25);
+      ElemType g = 1. + 9. * sum;
+      ElemType objectiveRatio = objectives(0) / g;
+      objectives(1) = g * (1. - std::pow(objectiveRatio, 2));
+
+      return objectives;
+    }
+
+    //! Get the starting point.
+    MatType GetInitialPoint()
+    {
+      // Convenience typedef.
+      typedef typename MatType::elem_type ElemType;
+
+      return arma::Col<ElemType>(numVariables, 1, arma::fill::zeros);
+    }
+
+    struct ObjectiveF1
+    {
+      ObjectiveF1(ZDT6& zdtClass) : zdtClass(zdtClass)
+      {/*Nothing to do here */}
+
+      typename MatType::elem_type Evaluate(const MatType& coords)
+      {
+        return 1. - std::exp(-4. * coords[0]) *
+            std::pow(std::sin(6. * arma::datum::pi * coords[0]), 6.);
+      }
+
+      ZDT6& zdtClass;
+    };
+
+    struct ObjectiveF2
+    {
+      ObjectiveF2(ZDT6& zdtClass) : zdtClass(zdtClass)
+      {/*Nothing to do here */}
+
+      typename MatType::elem_type Evaluate(const MatType& coords)
+      {
+        typedef typename MatType::elem_type ElemType;
+
+        size_t numVariables = zdtClass.numVariables;
+
+        ElemType sum = std::pow(
+            arma::accu(coords(arma::span(1, numVariables - 1), 0)) / 9, 0.25);
+        ElemType g = 1. + 9. * sum;
+        ElemType objectiveRatio = zdtClass.objectiveF1.Evaluate(coords) / g;
+
+        return  g * (1. - std::pow(objectiveRatio, 2));
+    }
+
+      ZDT6& zdtClass;
+    };
+
+    //! Get objective functions.
+    std::tuple<ObjectiveF1, ObjectiveF2> GetObjectives()
+    {
+      return std::make_tuple(objectiveF1, objectiveF2);
+    }
+
+    //! Get the Reference Front.
+    //! Refer PR #273 Ipynb notebook to see the plot of Reference
+    //! Front. The implementation has been taken from pymoo.
+    arma::cube GetReferenceFront()
+    {
+      arma::cube front(2, 1, numParetoPoints);
+      arma::vec x = arma::linspace(0.2807753191, 1, numParetoPoints);
+      arma::vec y = 1 - arma::square(x);
+      for (size_t idx = 0; idx < numParetoPoints; ++idx)
+        front.slice(idx) = arma::vec{ x(idx), y(idx) };
+
+      return front;
+    }
+
+    ObjectiveF1 objectiveF1;
+    ObjectiveF2 objectiveF2;
+  };
+  } //namespace test
+  } //namespace ens
+#endif

include/ensmallen_bits/pso/update_policies/lbest_update.hpp

@@ -73,7 +73,8 @@ class LBestUpdate
      *
      * @param parent Instantiated parent class.
      */
-     Policy(const LBestUpdate& /* parent */) { /* Do nothing. */ }
+     Policy(const LBestUpdate& /* parent */) : n(0)
+     { /* Do nothing. */ }
 
      /**
       * The Initialize method is called by PSO Optimizer method before the

include/ensmallen_bits/sa/sa_impl.hpp

@@ -76,7 +76,7 @@ typename MatType::elem_type SA<CoolingScheduleType>::Optimize(
   size_t sweepCounter = 0;
 
   BaseMatType accept(rows, cols, arma::fill::zeros);
-  BaseMatType moveSize(rows, cols);
+  BaseMatType moveSize(rows, cols, arma::fill::none);
   moveSize.fill(initMoveCoef);
 
   terminate |= Callback::BeginOptimization(*this, function, iterate,

include/ensmallen_bits/sarah/sarah_impl.hpp

@@ -89,11 +89,6 @@ SARAHType<UpdatePolicyType>::Optimize(
   BaseGradType gradient0(iterate.n_rows, iterate.n_cols);
   BaseMatType iterate0;
 
-  // Find the number of batches.
-  size_t numBatches = numFunctions / batchSize;
-  if (numFunctions % batchSize != 0)
-    ++numBatches; // Capture last few.
-
   const size_t actualMaxIterations = (maxIterations == 0) ?
       std::numeric_limits<size_t>::max() : maxIterations;
   terminate |= Callback::BeginOptimization(*this, function, iterate,

include/ensmallen_bits/sdp/primal_dual_impl.hpp

@@ -162,8 +162,8 @@ SolveKKTSystem(const SparseConstraintType& aSparse,
         "solve KKT system.");
   }
 
-  MatType subTerm(aSparse.n_cols, 1);
-  subTerm.zeros();
+  MatType subTerm(aSparse.n_cols, 1, arma::fill::zeros);
+  
   if (aSparse.n_rows)
   {
     dySparse = dy(arma::span(0, aSparse.n_rows - 1), 0);
@@ -303,7 +303,7 @@ typename MatType::elem_type PrimalDualSolver::Optimize(
 
   eInvFaSparseT.set_size(n2bar, sdp.NumSparseConstraints());
   eInvFaDenseT.set_size(n2bar, sdp.NumDenseConstraints());
-  m.set_size(sdp.NumConstraints(), sdp.NumConstraints());
+  m.zeros(sdp.NumConstraints(), sdp.NumConstraints());
 
   // Controls early termination of the optimization process.
   bool terminate = false;

include/ensmallen_bits/sgdr/snapshot_ensembles.hpp

@@ -64,6 +64,7 @@ class SnapshotEnsembles
     constStepSize(stepSize),
     nextRestart(epochRestart),
     batchRestart(0),
+    epochBatches(0),
     epoch(0)
   {
     snapshotEpochs = 0;

include/ensmallen_bits/spalera_sgd/spalera_stepsize.hpp

@@ -52,7 +52,8 @@ class SPALeRAStepsize
                   const double adaptRate = 3.10e-8) :
       alpha(alpha),
       epsilon(epsilon),
-      adaptRate(adaptRate)
+      adaptRate(adaptRate),
+      lambda(0)
   {
     /* Nothing to do here. */
   }

include/ensmallen_bits/utility/indicators/epsilon.hpp

@@ -0,0 +1,92 @@
+/**
+ * @file epsilon.hpp
+ * @author Rahul Ganesh Prabhu
+ * @author Nanubala Gnana Sai
+ *
+ * Epsilon indicator
+ * A binary quality indicator that is capable of detecting whether one
+ * approximation set is better than another.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#ifndef ENSMALLEN_INDICATORS_EPSILON_HPP
+#define ENSMALLEN_INDICATORS_EPSILON_HPP
+
+namespace ens {
+
+/**
+ * The epsilon indicator is one of the binary quality indicators that was proposed by
+ * Zitzler et. al.. The indicator originally calculates a weak dominance relation 
+ * between two approximation sets. It returns "epsilon" which is the factor by which 
+ * the given approximation set is worse than the reference front with respect to 
+ * all the objectives.
+ * 
+ * \f[ I_{\epsilon}(A,B) = \max_{z^2 \in B} \
+ *                \min_{z^1 \in A} \
+ *                \max_{1 \leq i \leq n} \ \frac{z^1_i}{z^2_i}\
+ *                \f]
+ *
+ * For more information, please see:
+ *
+ * @code
+ * @article{1197687,
+ *    author   = {E. Zitzler and L. Thiele and M. Laumanns and C. M. Fonseca and
+ *                V. G. da Fonseca},
+ *    title    = {Performance assessment of multiobjective optimizers: an
+ *                analysis and review},
+ *    journal  = {IEEE Transactions on Evolutionary Computation},
+ *    year     = {2003},
+ * }
+ * @endcode
+ */
+  class Epsilon
+  {
+   public:
+    /**
+     * Default constructor does nothing, but is required to satisfy the Indicator
+     * policy.
+     */
+    Epsilon() { }
+
+    /**
+     * Find the epsilon value of the front with respect to the given reference
+     * front.
+     *
+     * @tparam CubeType The cube data type of front.
+     * @param front The given approximation front.
+     * @param referenceFront The given reference front.
+     * @return The epsilon value of the front.
+     */
+    template<typename CubeType>
+    static typename CubeType::elem_type Evaluate(const CubeType& front,
+                                                 const CubeType& referenceFront)
+    {
+      // Convenience typedefs.
+      typedef typename CubeType::elem_type ElemType;
+      ElemType eps = 0;
+      for (size_t i = 0; i < referenceFront.n_slices; i++)
+      {
+        ElemType epsjMin = std::numeric_limits<ElemType>::max();
+        for (size_t j = 0; j < front.n_slices; j++)
+        {
+          arma::Mat<ElemType> frontRatio = front.slice(j) / referenceFront.slice(i);
+          frontRatio.replace(arma::datum::inf, -1.); // Handle zero division case.
+          ElemType epsj = frontRatio.max();
+          if (epsj < epsjMin)
+            epsjMin = epsj;
+        }
+        if (epsjMin > eps)
+          eps = epsjMin;
+      }
+
+      return eps;
+    }
+  };
+
+} // namespace ens
+
+#endif

include/ensmallen_bits/utility/indicators/igd_plus.hpp

@@ -0,0 +1,94 @@
+/**
+ * @file igd_plus.hpp
+ * @author Rahul Ganesh Prabhu
+ * @author Nanubala Gnana Sai
+ *
+ * Inverse Generational Distance Plus (IGD+) indicator.
+ * The average distance from each reference point to its nearest solution.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#ifndef ENSMALLEN_INDICATORS_IGD_PLUS_HPP
+#define ENSMALLEN_INDICATORS_IGD_PLUS_HPP
+
+namespace ens {
+
+/**
+ * The IGD indicator returns the average distance from each point in the reference
+ * front to the nearest point to it's solution. IGD+ is an improvement upon
+ * the IGD indicator, which fixes misleading results given by IGD in certain
+ * cases via a different distance metric:
+ *
+ * \f[ d^{+}(z,a) = \sqrt{\sum_{i = 1}^{n}( \max\{a_i - z_i, 0\})^2 \ } \
+ *    \f]
+ *
+ * For more information see:
+ *
+ * @code
+ * @article{10.1007/978-3-319-15892-1_8,
+ *    author   = {Ishibuchi, Hisao and Masuda, Hiroyuki and Tanigaki, Yuki
+ *                and Nojima, Yusuke},
+ *    title    = {Modified Distance Calculation in Generational Distance
+ *                and Inverted Generational Distance},
+ *    book     = {Evolutionary Multi-Criterion Optimization}
+ *    year     = {2015}
+ * }
+ * @endcode
+ */
+  class IGDPlus
+  {
+   public:
+    /**
+     * Default constructor does nothing, but is required to satisfy the Indicator
+     * policy.
+     */
+    IGDPlus() { }
+
+    /**
+     * Find the IGD+ value of the front with respect to the given reference
+     * front.
+     *
+     * @tparam CubeType The cube data type of front.
+     * @param front The given approximation front.
+     * @param referenceFront The given reference front.
+     * @return The IGD value of the front.
+     */
+    template<typename CubeType>
+    static typename CubeType::elem_type Evaluate(const CubeType& front,
+                                                 const CubeType& referenceFront)
+    {
+      // Convenience typedefs.
+      typedef typename CubeType::elem_type ElemType;
+      ElemType igd = 0;
+      for (size_t i = 0; i < referenceFront.n_slices; i++)
+      {
+        ElemType min = std::numeric_limits<ElemType>::max();
+        for (size_t j = 0; j < front.n_slices; j++)
+        {
+          ElemType dist = 0;
+          for (size_t k = 0; k < front.slice(j).n_rows; k++)
+          {
+            ElemType z = referenceFront(k, 0, i);
+            ElemType a = front(k, 0, j);
+            // Assuming minimization of all objectives.
+            dist += std::pow(std::max<ElemType>(a - z, 0), 2);
+          }
+          dist = std::sqrt(dist);
+          if (dist < min)
+            min = dist;
+        }
+        igd += min;
+      }
+      igd /= referenceFront.n_slices;
+
+      return igd;
+    }
+  };
+
+} // namespace ens
+
+#endif

scripts/ensmallen-release.sh

@@ -23,7 +23,8 @@ if [ "$#" -gt 5 ]; then
 fi
 
 # Make sure that the branch is clean.
-lines=`git diff | wc -l`;
+# Truncate leading whitespaces since wc -l on MacOS adds an extra \t.
+lines=`git diff | wc -l | sed -e 's/^\s*//g'`;
 if [ "$lines" != "0" ]; then
   echo "git diff returned a nonzero result!";
   echo "";
@@ -123,6 +124,14 @@ new_line="ensmallen $MAJOR.$MINOR.$PATCH: \"$version_name\"";
 sed -i "s/### ensmallen ?.??.?: \"???\"/### $new_line/" HISTORY.md;
 sed -i "s/###### ????-??-??/###### $year-$month-$day/" HISTORY.md;
 
+# Update date in ens_version.hpp
+sed -i 's/ENS_VERSION_YEAR[ ]*\".*\"$/ENS_VERSION_YEAR \"'"$year"'\"/' \
+    include/ensmallen_bits/ens_version.hpp;
+sed -i 's/ENS_VERSION_MONTH[ ]*\".*\"$/ENS_VERSION_MONTH \"'"$month"'\"/' \
+    include/ensmallen_bits/ens_version.hpp;
+sed -i 's/ENS_VERSION_DAY[ ]*\".*\"$/ENS_VERSION_DAY \"'"$day"'\"/' \
+    include/ensmallen_bits/ens_version.hpp;
+
 # Now, we'll do all this on a new release branch.
 git checkout -b release-$MAJOR.$MINOR.$PATCH;
 
@@ -132,7 +141,7 @@ git add HISTORY.md;
 git commit -m "Update and release version $MAJOR.$MINOR.$PATCH.";
 
 changelog_str=`cat HISTORY.md |\
-    awk '/^### /{f=0} /^### ensmallen 2.13.0: "Automatically Automated Automation"/{f=1} f{print}' |\
+    awk '/^### /{f=0} /^### ensmallen '"$MAJOR"'.'"$MINOR"'.'"$PATCH"': "'"$version_name"'"/{f=1} f{print}' |\
     grep -v '^#' |\
     tr '\n' '!' |\
     sed -e 's/!  [ ]*/ /g' |\
@@ -168,6 +177,6 @@ hub pull-request \
 
 echo "";
 echo "Switching back to 'master' branch.";
-echo "If you want to access the release branch again, use \`git checkout" \
+echo "If you want to access the release branch again, use \`git checkout " \
     "release-$MAJOR.$MINOR.$PATCH\`.";
 exit 0;

tests/CMakeLists.txt

@@ -23,6 +23,7 @@ set(ENSMALLEN_TESTS_SOURCES
     line_search_test.cpp
     lookahead_test.cpp
     lrsdp_test.cpp
+    moead_test.cpp
     momentum_sgd_test.cpp
     nesterov_momentum_sgd_test.cpp
     nsga2_test.cpp
@@ -47,7 +48,7 @@ set(ENSMALLEN_TESTS_SOURCES
 )
 
 set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})
-add_executable(ensmallen_tests ${ENSMALLEN_TESTS_SOURCES})
+add_executable(ensmallen_tests EXCLUDE_FROM_ALL ${ENSMALLEN_TESTS_SOURCES})
 target_link_libraries(ensmallen_tests PRIVATE ensmallen)
 
 # Copy test data into place.
@@ -59,4 +60,4 @@ add_custom_command(TARGET ensmallen_tests
 
 enable_testing()
 add_test(NAME ensmallen_tests COMMAND ensmallen_tests
-WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
+WORKING_DIRECTORY ${CMAKE_BINARY_DIR})

tests/ada_bound_test.cpp

@@ -20,31 +20,19 @@ using namespace ens::test;
  */
 TEST_CASE("AdaBoundSphereFunctionTest", "[AdaBoundTest]")
 {
-  SphereFunction f(2);
   AdaBound optimizer(0.001, 2, 0.1, 1e-3, 0.9, 0.999, 1e-8, 500000,
       1e-3, false);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction>(optimizer, 0.5, 0.1);
 }
 
 /**
  * Test the AdaBound optimizer on the Sphere function with arma::fmat.
  */
-TEST_CASE("AdaBoundphereFunctionTestFMat", "[AdaBoundTest]")
+TEST_CASE("AdaBoundSphereFunctionTestFMat", "[AdaBoundTest]")
 {
-  SphereFunction f(2);
   AdaBound optimizer(0.001, 2, 0.1, 1e-3, 0.9, 0.999, 1e-8, 500000,
       1e-3, false);
-
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction, arma::fmat>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -52,15 +40,9 @@ TEST_CASE("AdaBoundphereFunctionTestFMat", "[AdaBoundTest]")
  */
 TEST_CASE("AMSBoundSphereFunctionTest", "[AdaBoundTest]")
 {
-  SphereFunction f(2);
   AMSBound optimizer(0.001, 2, 0.1, 1e-3, 0.9, 0.999, 1e-8, 500000,
       1e-3, false);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction, arma::mat>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -68,15 +50,9 @@ TEST_CASE("AMSBoundSphereFunctionTest", "[AdaBoundTest]")
  */
 TEST_CASE("AMSBoundphereFunctionTestFMat", "[AdaBoundTest]")
 {
-  SphereFunction f(2);
   AMSBound optimizer(0.001, 2, 0.1, 1e-3, 0.9, 0.999, 1e-8, 500000,
       1e-3, false);
-
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction, arma::fmat>(optimizer, 0.5, 0.1);
 }
 
 #if ARMA_VERSION_MAJOR > 9 ||\
@@ -87,15 +63,9 @@ TEST_CASE("AMSBoundphereFunctionTestFMat", "[AdaBoundTest]")
  */
 TEST_CASE("AdaBoundSphereFunctionTestSpMat", "[AdaBoundTest]")
 {
-  SphereFunction f(2);
   AdaBound optimizer(0.001, 2, 0.1, 1e-3, 0.9, 0.999, 1e-8, 500000,
       1e-3, false);
-
-  arma::sp_mat coordinates = f.GetInitialPoint<arma::sp_mat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction, arma::sp_mat>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -120,15 +90,9 @@ TEST_CASE("AdaBoundSphereFunctionTestSpMatDenseGradient", "[AdaBoundTest]")
  */
 TEST_CASE("AMSBoundSphereFunctionTestSpMat", "[AdaBoundTest]")
 {
-  SphereFunction f(2);
   AMSBound optimizer(0.001, 2, 0.1, 1e-3, 0.9, 0.999, 1e-8, 500000,
       1e-3, false);
-
-  arma::sp_mat coordinates = f.GetInitialPoint<arma::sp_mat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction, arma::sp_mat>(optimizer, 0.5, 0.1);
 }
 
 /**

tests/ada_delta_test.cpp

@@ -18,62 +18,13 @@
 using namespace ens;
 using namespace ens::test;
 
-/**
- * Tests the Adadelta optimizer using a simple test function.
- */
-TEST_CASE("SimpleAdaDeltaTestFunction", "[AdaDeltaTest]")
-{
-  SGDTestFunction f;
-  AdaDelta optimizer(1.0, 1, 0.05, 1e-6, 5000000, 1e-15, true, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.003));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.003));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(0.003));
-}
-
 /**
  * Run AdaDelta on logistic regression and make sure the results are acceptable.
  */
 TEST_CASE("AdaDeltaLogisticRegressionTest", "[AdaDeltaTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   AdaDelta adaDelta;
-  arma::mat coordinates = lr.GetInitialPoint();
-  adaDelta.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
-}
-
-/**
- * Tests the Adadelta optimizer using a simple test function with arma::fmat as
- * the type.
- */
-TEST_CASE("SimpleAdaDeltaTestFunctionFMat", "[AdaDeltaTest]")
-{
-  SGDTestFunction f;
-  AdaDelta optimizer(1.0, 1, 0.05, 1e-6, 5000000, 1e-15, true, true);
-
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0f).margin(0.01));
-  REQUIRE(coordinates(1) == Approx(0.0f).margin(0.01));
-  REQUIRE(coordinates(2) == Approx(0.0f).margin(0.01));
+  LogisticRegressionFunctionTest(adaDelta, 0.003, 0.006, 1);
 }
 
 /**
@@ -82,22 +33,6 @@ TEST_CASE("SimpleAdaDeltaTestFunctionFMat", "[AdaDeltaTest]")
  */
 TEST_CASE("AdaDeltaLogisticRegressionTestFMat", "[AdaDeltaTest]")
 {
-  arma::fmat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
   AdaDelta adaDelta;
-  arma::fmat coordinates = lr.GetInitialPoint();
-  adaDelta.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest(adaDelta, 0.003, 0.006, 1);
 }

tests/ada_grad_test.cpp

@@ -17,61 +17,13 @@
 using namespace ens;
 using namespace ens::test;
 
-/**
- * Tests the Adagrad optimizer using a simple test function.
- */
-TEST_CASE("SimpleAdaGradTestFunction", "[AdaGradTest]")
-{
-  SGDTestFunction f;
-  AdaGrad optimizer(0.99, 1, 1e-8, 5000000, 1e-9, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.003));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.003));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(0.003));
-}
-
 /**
  * Run AdaGrad on logistic regression and make sure the results are acceptable.
  */
 TEST_CASE("AdaGradLogisticRegressionTest", "[AdaGradTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   AdaGrad adagrad(0.99, 32, 1e-8, 5000000, 1e-9, true);
-  arma::mat coordinates = lr.GetInitialPoint();
-  adagrad.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
-}
-
-/**
- * Tests the Adagrad optimizer using a simple test function with arma::fmat.
- */
-TEST_CASE("SimpleAdaGradTestFunctionFMat", "[AdaGradTest]")
-{
-  SGDTestFunction f;
-  AdaGrad optimizer(0.99, 1, 1e-8, 5000000, 1e-9, true);
-
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0f).margin(0.01));
-  REQUIRE(coordinates(1) == Approx(0.0f).margin(0.01));
-  REQUIRE(coordinates(2) == Approx(0.0f).margin(0.01));
+  LogisticRegressionFunctionTest(adagrad, 0.003, 0.006);
 }
 
 /**
@@ -79,22 +31,6 @@ TEST_CASE("SimpleAdaGradTestFunctionFMat", "[AdaGradTest]")
  */
 TEST_CASE("AdaGradLogisticRegressionTestFMat", "[AdaGradTest]")
 {
-  arma::fmat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
   AdaGrad adagrad(0.99, 32, 1e-8, 5000000, 1e-9, true);
-  arma::fmat coordinates = lr.GetInitialPoint();
-  adagrad.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest<arma::fmat>(adagrad, 0.003, 0.006);
 }

tests/adam_test.cpp

@@ -25,14 +25,8 @@ using namespace ens::test;
  */
 TEST_CASE("AdamSphereFunctionTest", "[AdamTest]")
 {
-  SphereFunction f(2);
   Adam optimizer(0.5, 2, 0.7, 0.999, 1e-8, 500000, 1e-3, false);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction>(optimizer, 0.5, 0.2);
 }
 
 /**
@@ -40,14 +34,8 @@ TEST_CASE("AdamSphereFunctionTest", "[AdamTest]")
  */
 TEST_CASE("AdamSphereFunctionTestFMat", "[AdamTest]")
 {
-  SphereFunction f(2);
   Adam optimizer(0.5, 2, 0.7, 0.999, 1e-8, 500000, 1e-3, false);
-
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction, arma::fmat>(optimizer, 0.5, 0.2);
 }
 
 #if ARMA_VERSION_MAJOR > 9 ||\
@@ -58,14 +46,8 @@ TEST_CASE("AdamSphereFunctionTestFMat", "[AdamTest]")
  */
 TEST_CASE("AdamSphereFunctionTestSpMat", "[AdamTest]")
 {
-  SphereFunction f(2);
   Adam optimizer(0.5, 2, 0.7, 0.999, 1e-8, 500000, 1e-3, false);
-
-  arma::sp_mat coordinates = f.GetInitialPoint<arma::sp_mat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction, arma::sp_mat>(optimizer, 0.5, 0.2);
 }
 
 /**
@@ -91,14 +73,8 @@ TEST_CASE("AdamSphereFunctionTestSpMatDenseGradient", "[AdamTest]")
  */
 TEST_CASE("AdamStyblinskiTangFunctionTest", "[AdamTest]")
 {
-  StyblinskiTangFunction f(2);
   Adam optimizer(0.5, 2, 0.7, 0.999, 1e-8, 500000, 1e-3, false);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(-2.9).epsilon(0.01)); // 1% error tolerance.
-  REQUIRE(coordinates(1) == Approx(-2.9).epsilon(0.01)); // 1% error tolerance.
+  FunctionTest<StyblinskiTangFunction>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -106,15 +82,8 @@ TEST_CASE("AdamStyblinskiTangFunctionTest", "[AdamTest]")
  */
 TEST_CASE("AdamMcCormickFunctionTest", "[AdamTest]")
 {
-  McCormickFunction f;
   Adam optimizer(0.5, 1, 0.7, 0.999, 1e-8, 500000, 1e-5, false);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  // 3% error tolerance.
-  REQUIRE(coordinates(0) == Approx(-0.547).epsilon(0.03));
-  REQUIRE(coordinates(1) == Approx(-1.547).epsilon(0.03));
+  FunctionTest<McCormickFunction>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -122,17 +91,8 @@ TEST_CASE("AdamMcCormickFunctionTest", "[AdamTest]")
  */
 TEST_CASE("AdamMatyasFunctionTest", "[AdamTest]")
 {
-  MatyasFunction f;
   Adam optimizer(0.5, 1, 0.7, 0.999, 1e-8, 500000, 1e-5, false);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  // 3% error tolerance.
-  REQUIRE((std::trunc(100.0 * coordinates(0)) / 100.0) ==
-      Approx(0.0).epsilon(0.003));
-  REQUIRE((std::trunc(100.0 * coordinates(1)) / 100.0) ==
-      Approx(0.0).epsilon(0.003));
+  FunctionTest<MatyasFunction>(optimizer, 0.1, 0.01);
 }
 
 /**
@@ -140,17 +100,8 @@ TEST_CASE("AdamMatyasFunctionTest", "[AdamTest]")
  */
 TEST_CASE("AdamEasomFunctionTest", "[AdamTest]")
 {
-  EasomFunction f;
   Adam optimizer(0.2, 1, 0.7, 0.999, 1e-8, 500000, 1e-5, false);
-
-  arma::mat coordinates = arma::mat("2.9; 2.9");
-  optimizer.Optimize(f, coordinates);
-
-  // 5% error tolerance.
-  REQUIRE((std::trunc(100.0 * coordinates(0)) / 100.0) ==
-      Approx(3.14).epsilon(0.005));
-  REQUIRE((std::trunc(100.0 * coordinates(1)) / 100.0) ==
-      Approx(3.14).epsilon(0.005));
+  FunctionTest<EasomFunction>(optimizer, 1.5, 0.01);
 }
 
 /**
@@ -158,77 +109,18 @@ TEST_CASE("AdamEasomFunctionTest", "[AdamTest]")
  */
 TEST_CASE("AdamBoothFunctionTest", "[AdamTest]")
 {
-  BoothFunction f;
   Adam optimizer(1e-1, 1, 0.7, 0.999, 1e-8, 500000, 1e-9, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(1.0).epsilon(0.002));
-  REQUIRE(coordinates(1) == Approx(3.0).epsilon(0.002));
+  FunctionTest<BoothFunction>(optimizer);
 }
 
-/**
- * Tests the Adam optimizer using a simple test function.
- */
-TEST_CASE("SimpleAdamTestFunction", "[AdamTest]")
-{
-  SGDTestFunction f;
-  Adam optimizer(1e-3, 1, 0.9, 0.999, 1e-8, 500000, 1e-9, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.3));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.3));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(0.3));
-}
-
-/**
- * Tests the AdaMax optimizer using a simple test function.
- */
-TEST_CASE("SimpleAdaMaxTestFunction", "[AdamTest]")
-{
-  SGDTestFunction f;
-  AdaMax optimizer(2e-3, 1, 0.9, 0.999, 1e-8, 500000, 1e-9, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.3));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.3));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(0.3));
-}
-
-/**
- * Tests the AMSGrad optimizer using a simple test function.
- */
-TEST_CASE("SimpleAMSGradTestFunction", "[AdamTest]")
-{
-  SGDTestFunction f;
-  AMSGrad optimizer(1e-3, 1, 0.9, 0.999, 1e-8, 500000, 1e-11, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.3));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.3));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(0.3));
-}
 
 /**
  * Test the AMSGrad optimizer on the Sphere function with arma::fmat.
  */
 TEST_CASE("AMSGradSphereFunctionTestFMat", "[AdamTest]")
 {
-  SphereFunction f(2);
   AMSGrad optimizer(1e-3, 1, 0.9, 0.999, 1e-8, 500000, 1e-11, true);
-
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction, arma::fmat>(optimizer, 0.5, 0.1);
 }
 
 #if ARMA_VERSION_MAJOR > 9 || \
@@ -239,14 +131,8 @@ TEST_CASE("AMSGradSphereFunctionTestFMat", "[AdamTest]")
  */
 TEST_CASE("AMSGradSphereFunctionTestSpMat", "[AdamTest]")
 {
-  SphereFunction f(2);
   AMSGrad optimizer(1e-3, 1, 0.9, 0.999, 1e-8, 500000, 1e-11, true);
-
-  arma::sp_mat coordinates = f.GetInitialPoint<arma::sp_mat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction, arma::sp_mat>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -272,24 +158,8 @@ TEST_CASE("AMSGradSphereFunctionTestSpMatDenseGradient", "[AdamTest]")
  */
 TEST_CASE("AdamLogisticRegressionTest", "[AdamTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   Adam adam;
-  arma::mat coordinates = lr.GetInitialPoint();
-  adam.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest(adam, 0.003, 0.006);
 }
 
 /**
@@ -297,24 +167,8 @@ TEST_CASE("AdamLogisticRegressionTest", "[AdamTest]")
  */
 TEST_CASE("AdaMaxLogisticRegressionTest", "[AdamTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   AdaMax adamax(1e-3, 1, 0.9, 0.999, 1e-8, 5000000, 1e-9, true);
-  arma::mat coordinates = lr.GetInitialPoint();
-  adamax.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest(adamax, 0.003, 0.006);
 }
 
 /**
@@ -322,40 +176,8 @@ TEST_CASE("AdaMaxLogisticRegressionTest", "[AdamTest]")
  */
 TEST_CASE("AMSGradLogisticRegressionTest", "[AdamTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   AMSGrad amsgrad(1e-3, 1, 0.9, 0.999, 1e-8, 500000, 1e-11, true);
-  arma::mat coordinates = lr.GetInitialPoint();
-  amsgrad.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
-}
-
-/**
- * Tests the Nadam optimizer using a simple test function.
- */
-TEST_CASE("SimpleNadamTestFunction", "[AdamTest]")
-{
-  SGDTestFunction f;
-  Nadam optimizer(1e-3, 1, 0.9, 0.99, 1e-8, 500000, 1e-9, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.3));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.3));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(0.3));
+  LogisticRegressionFunctionTest(amsgrad, 0.003, 0.006);
 }
 
 /**
@@ -363,40 +185,8 @@ TEST_CASE("SimpleNadamTestFunction", "[AdamTest]")
  */
 TEST_CASE("NadamLogisticRegressionTest", "[AdamTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   Nadam nadam;
-  arma::mat coordinates = lr.GetInitialPoint();
-  nadam.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
-}
-
-/**
- * Tests the NadaMax optimizer using a simple test function.
- */
-TEST_CASE("SimpleNadaMaxTestFunction", "[AdamTest]")
-{
-  SGDTestFunction f;
-  NadaMax optimizer(1e-3, 1, 0.9, 0.99, 1e-8, 500000, 1e-9, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.3));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.3));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(0.3));
+  LogisticRegressionFunctionTest(nadam, 0.003, 0.006);
 }
 
 /**
@@ -404,101 +194,18 @@ TEST_CASE("SimpleNadaMaxTestFunction", "[AdamTest]")
  */
 TEST_CASE("NadaMaxLogisticRegressionTest", "[AdamTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   NadaMax nadamax(1e-3, 1, 0.9, 0.999, 1e-8, 5000000, 1e-9, true);
-  arma::mat coordinates = lr.GetInitialPoint();
-  nadamax.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest(nadamax, 0.003, 0.006);
 }
 
 /**
- * Tests the OptimisticAdam optimizer using a simple test function.
- */
-TEST_CASE("SimpleOptimisticAdamTestFunction", "[AdamTest]")
-{
-  // Sometimes this test can fail randomly, so we allow it to run up to three
-  // times.
-  bool success = false;
-  for (size_t trial = 0; trial < 3; ++trial)
-  {
-    SGDTestFunction f;
-    OptimisticAdam optimizer(1e-2, 1, 0.9, 0.99, 1e-8);
-
-    arma::mat coordinates = f.GetInitialPoint();
-    optimizer.Optimize(f, coordinates);
-
-    success = (coordinates(0) == Approx(0.0).margin(0.3)) &&
-              (coordinates(1) == Approx(0.0).margin(0.3)) &&
-              (coordinates(2) == Approx(0.0).margin(0.3));
-    if (success)
-      break;
-  }
-
-  REQUIRE(success == true);
-}
-
-/**
- * Run OptimisticAdam on logistic regression and make sure the results are acceptable.
+ * Run OptimisticAdam on logistic regression and make sure the results are
+ * acceptable.
  */
 TEST_CASE("OptimisticAdamLogisticRegressionTest", "[AdamTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   OptimisticAdam optimisticAdam;
-  arma::mat coordinates = lr.GetInitialPoint();
-  optimisticAdam.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
-}
-
-/**
- * Tests the Padam optimizer using a simple test function.
- */
-TEST_CASE("SimplePadamTestFunction", "[AdamTest]")
-{
-  // Sometimes this test can fail randomly, so we allow it to run up to three
-  // times.
-  bool success = false;
-  for (size_t trial = 0; trial < 3; ++trial)
-  {
-    SGDTestFunction f;
-    Padam optimizer(1e-2, 1, 0.9, 0.99, 0.25, 1e-8);
-
-    arma::mat coordinates = f.GetInitialPoint();
-    optimizer.Optimize(f, coordinates);
-
-    success = (coordinates(0) == Approx(0.0).margin(0.3)) &&
-              (coordinates(1) == Approx(0.0).margin(0.3)) &&
-              (coordinates(2) == Approx(0.0).margin(0.3));
-    if (success)
-      break;
-  }
-
-  REQUIRE(success == true);
+  LogisticRegressionFunctionTest(optimisticAdam, 0.003, 0.006);
 }
 
 /**
@@ -506,41 +213,8 @@ TEST_CASE("SimplePadamTestFunction", "[AdamTest]")
  */
 TEST_CASE("PadamLogisticRegressionTest", "[AdamTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   Padam optimizer;
-  arma::mat coordinates = lr.GetInitialPoint();
-  optimizer.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
-}
-
-/**
- * Tests the QHadam optimizer using a simple test function.
- */
-TEST_CASE("SimpleQHAdamTestFunction", "[AdamTest]")
-{
-  SGDTestFunction f;
-  QHAdam optimizer(0.02, 32, 0.6, 0.9, 0.9, 0.999, 1e-8, 200000, 1e-7, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  bool success = (coordinates(0) == Approx(0.0).margin(0.3)) &&
-                 (coordinates(1) == Approx(0.0).margin(0.3)) &&
-                 (coordinates(2) == Approx(0.0).margin(0.3));
-  REQUIRE(success == true);
+  LogisticRegressionFunctionTest(optimizer, 0.003, 0.006);
 }
 
 /**
@@ -548,24 +222,8 @@ TEST_CASE("SimpleQHAdamTestFunction", "[AdamTest]")
  */
 TEST_CASE("QHAdamLogisticRegressionTest", "[AdamTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   QHAdam optimizer;
-  arma::mat coordinates = lr.GetInitialPoint();
-  optimizer.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest(optimizer, 0.003, 0.006);
 }
 
 /**
@@ -574,24 +232,8 @@ TEST_CASE("QHAdamLogisticRegressionTest", "[AdamTest]")
  */
 TEST_CASE("QHAdamLogisticRegressionFMatTest", "[AdamTest]")
 {
-  arma::fmat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
   QHAdam optimizer;
-  arma::fmat coordinates = lr.GetInitialPoint();
-  optimizer.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const float acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.03)); // 3% error tolerance.
-
-  const float testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.06)); // 6% error tolerance.
+  LogisticRegressionFunctionTest<arma::fmat>(optimizer, 0.03, 0.06);
 }
 
 #if ARMA_VERSION_MAJOR > 9 ||\
@@ -603,24 +245,8 @@ TEST_CASE("QHAdamLogisticRegressionFMatTest", "[AdamTest]")
  */
 TEST_CASE("QHAdamLogisticRegressionSpMatTest", "[AdamTest]")
 {
-  arma::sp_mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<arma::sp_mat> lr(shuffledData, shuffledResponses, 0.5);
-
   QHAdam optimizer;
-  arma::sp_mat coordinates = lr.GetInitialPoint();
-  optimizer.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest<arma::sp_mat>(optimizer, 0.003, 0.006);
 }
 
 #endif
@@ -632,14 +258,8 @@ TEST_CASE("QHAdamLogisticRegressionSpMatTest", "[AdamTest]")
  */
 TEST_CASE("AdamAckleyFunctionTest", "[AdamTest]")
 {
-  AckleyFunction f;
   Adam optimizer(0.001, 2, 0.7, 0.999, 1e-8, 500000, 1e-7, false);
-
-  arma::mat coordinates = arma::mat("0.02; 0.02");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.001));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.001));
+  FunctionTest<AckleyFunction>(optimizer);
 }
 
 /**
@@ -649,14 +269,8 @@ TEST_CASE("AdamAckleyFunctionTest", "[AdamTest]")
  */
 TEST_CASE("AdamBealeFunctionTest", "[AdamTest]")
 {
-  BealeFunction f;
   Adam optimizer(0.001, 2, 0.7, 0.999, 1e-8, 500000, 1e-7, false);
-
-  arma::mat coordinates = arma::mat("2.8; 0.35");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(3.0).margin(0.01));
-  REQUIRE(coordinates(1) == Approx(0.5).margin(0.01));
+  FunctionTest<BealeFunction>(optimizer, 0.1, 0.01);
 }
 
 /**
@@ -666,14 +280,8 @@ TEST_CASE("AdamBealeFunctionTest", "[AdamTest]")
  */
 TEST_CASE("AdamGoldsteinPriceFunctionTest", "[AdamTest]")
 {
-  GoldsteinPriceFunction f;
   Adam optimizer(0.0001, 2, 0.7, 0.999, 1e-8, 500000, 1e-9, false);
-
-  arma::mat coordinates = arma::mat("0.2; -0.5");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0).margin(0.01));
-  REQUIRE(coordinates(1) == Approx(-1).margin(0.01));
+  FunctionTest<GoldsteinPriceFunction>(optimizer, 0.1, 0.01);
 }
 
 /**
@@ -683,14 +291,8 @@ TEST_CASE("AdamGoldsteinPriceFunctionTest", "[AdamTest]")
  */
 TEST_CASE("AdamLevyFunctionTest", "[AdamTest]")
 {
-  LevyFunctionN13 f;
   Adam optimizer(0.001, 2, 0.7, 0.999, 1e-8, 500000, 1e-9, false);
-
-  arma::mat coordinates = arma::mat("0.9; 1.1");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(1).margin(0.01));
-  REQUIRE(coordinates(1) == Approx(1).margin(0.01));
+  FunctionTest<LevyFunctionN13>(optimizer, 0.1, 0.01);
 }
 
 /**
@@ -717,29 +319,6 @@ TEST_CASE("AdamHimmelblauFunctionTest", "[AdamTest]")
  */
 TEST_CASE("AdamThreeHumpCamelFunctionTest", "[AdamTest]")
 {
-  ThreeHumpCamelFunction f;
   Adam optimizer(0.001, 2, 0.7, 0.999, 1e-8, 500000, 1e-9, false);
-
-  arma::mat coordinates = arma::mat("1; 1");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.01));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.01));
-}
-
-/**
- * Test the Adam optimizer on Schaffer function N.2.
- * This is to test Schaffer function N.2 and not Adam.
- * This test will be removed later.
- */
-TEST_CASE("AdamSchafferFunctionN2Test", "[AdamTest]")
-{
-  SchafferFunctionN2 f;
-  Adam optimizer(0.001, 2, 0.7, 0.999, 1e-8, 500000, 1e-9, false);
-
-  arma::mat coordinates = arma::mat("1; 1");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.01));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.01));
+  FunctionTest<ThreeHumpCamelFunction>(optimizer, 0.1, 0.01);
 }

tests/bigbatch_sgd_test.cpp

@@ -22,28 +22,11 @@ using namespace ens::test;
  */
 TEST_CASE("BBSBBLogisticRegressionTest", "[BigBatchSGDTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
-  for (size_t batchSize = 40; batchSize < 50; batchSize += 5)
+  // Run big-batch SGD with a couple of batch sizes.
+  for (size_t batchSize = 350; batchSize < 360; batchSize += 5)
   {
-    BBS_Armijo bbsgd(batchSize, 0.005, 0.1, 10000, 1e-6, true, true);
-
-    LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-    arma::mat coordinates = lr.GetInitialPoint();
-    bbsgd.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+    BBS_BB bbsgd(batchSize, 0.001, 0.1, 10000, 1e-8, true, true);
+    LogisticRegressionFunctionTest(bbsgd, 0.003, 0.006, 3);
   }
 }
 
@@ -53,28 +36,11 @@ TEST_CASE("BBSBBLogisticRegressionTest", "[BigBatchSGDTest]")
  */
 TEST_CASE("BBSArmijoLogisticRegressionTest", "[BigBatchSGDTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run big-batch SGD with a couple of batch sizes.
   for (size_t batchSize = 40; batchSize < 50; batchSize += 1)
   {
     BBS_Armijo bbsgd(batchSize, 0.005, 0.1, 10000, 1e-6, true, true);
-
-    LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-    arma::mat coordinates = lr.GetInitialPoint();
-    bbsgd.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+    LogisticRegressionFunctionTest(bbsgd, 0.003, 0.006, 3);
   }
 }
 
@@ -84,28 +50,11 @@ TEST_CASE("BBSArmijoLogisticRegressionTest", "[BigBatchSGDTest]")
  */
 TEST_CASE("BBSBBLogisticRegressionFMatTest", "[BigBatchSGDTest]")
 {
-  arma::fmat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run big-batch SGD with a couple of batch sizes.
   for (size_t batchSize = 350; batchSize < 360; batchSize += 5)
   {
     BBS_BB bbsgd(batchSize, 0.001, 0.1, 10000, 1e-8, true, true);
-
-    LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-    arma::fmat coordinates = lr.GetInitialPoint();
-    bbsgd.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+    LogisticRegressionFunctionTest<arma::fmat>(bbsgd, 0.003, 0.006, 3);
   }
 }
 
@@ -115,28 +64,11 @@ TEST_CASE("BBSBBLogisticRegressionFMatTest", "[BigBatchSGDTest]")
  */
 TEST_CASE("BBSArmijoLogisticRegressionFMatTest", "[BigBatchSGDTest]")
 {
-  arma::fmat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run big-batch SGD with a couple of batch sizes.
   for (size_t batchSize = 40; batchSize < 50; batchSize += 1)
   {
     BBS_Armijo bbsgd(batchSize, 0.01, 0.1, 10000, 1e-6, true, true);
-
-    LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-    arma::fmat coordinates = lr.GetInitialPoint();
-    bbsgd.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+    LogisticRegressionFunctionTest<arma::fmat>(bbsgd, 0.003, 0.006, 5);
   }
 }
 
@@ -149,28 +81,11 @@ TEST_CASE("BBSArmijoLogisticRegressionFMatTest", "[BigBatchSGDTest]")
  */
 TEST_CASE("BBSBBLogisticRegressionSpMatTest", "[BigBatchSGDTest]")
 {
-  arma::sp_mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run big-batch SGD with a couple of batch sizes.
   for (size_t batchSize = 350; batchSize < 360; batchSize += 5)
   {
     BBS_BB bbsgd(batchSize, 0.005, 0.5, 10000, 1e-8, true, true);
-
-    LogisticRegression<arma::sp_mat> lr(shuffledData, shuffledResponses, 0.5);
-    arma::sp_mat coordinates = lr.GetInitialPoint();
-    bbsgd.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+    LogisticRegressionFunctionTest<arma::sp_mat>(bbsgd, 0.003, 0.006, 3);
   }
 }
 
@@ -180,28 +95,11 @@ TEST_CASE("BBSBBLogisticRegressionSpMatTest", "[BigBatchSGDTest]")
  */
 TEST_CASE("BBSArmijoLogisticRegressionSpMatTest", "[BigBatchSGDTest]")
 {
-  arma::sp_mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run big-batch SGD with a couple of batch sizes.
   for (size_t batchSize = 40; batchSize < 50; batchSize += 1)
   {
     BBS_Armijo bbsgd(batchSize, 0.01, 0.001, 10000, 1e-6, true, true);
-
-    LogisticRegression<arma::sp_mat> lr(shuffledData, shuffledResponses, 0.5);
-    arma::sp_mat coordinates = lr.GetInitialPoint();
-    bbsgd.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+    LogisticRegressionFunctionTest<arma::sp_mat>(bbsgd, 0.003, 0.006, 3);
   }
 }
 

tests/callbacks_test.cpp

@@ -33,7 +33,8 @@ class CompleteCallbackTestFunction
       calledEndOptimization(false),
       calledEvaluateConstraint(false),
       calledGradientConstraint(false),
-      calledStepTaken(false)
+      calledStepTaken(false),
+      calledGenerationalStepTaken(false)
   { }
 
   template<typename OptimizerType, typename FunctionType, typename MatType>
@@ -106,6 +107,18 @@ class CompleteCallbackTestFunction
                  MatType& /* coordinates */)
   { calledStepTaken = true; }
 
+    template<typename OptimizerType,
+             typename FunctionType,
+             typename MatType,
+             typename ObjectivesVecType,
+             typename IndicesType>
+    void GenerationalStepTaken(OptimizerType& /* optimizer */,
+                               FunctionType& /* function */,
+                               MatType& /* coordinates */,
+                               ObjectivesVecType& /* objectives */,
+                               IndicesType& /* frontIndices */)
+    { calledGenerationalStepTaken = true; }
+
   bool calledEvaluate;
   bool calledGradient;
   bool calledBeginEpoch;
@@ -115,6 +128,7 @@ class CompleteCallbackTestFunction
   bool calledEvaluateConstraint;
   bool calledGradientConstraint;
   bool calledStepTaken;
+  bool calledGenerationalStepTaken;
 };
 
 template<typename OptimizerType>
@@ -162,7 +176,8 @@ void CallbacksFullMultiobjectiveFunctionTest(OptimizerType& optimizer,
                                              bool calledEndOptimization,
                                              bool calledEvaluateConstraint,
                                              bool calledGradientConstraint,
-                                             bool calledStepTaken)
+                                             bool calledStepTaken,
+                                             bool calledGenerationalStepTaken)
 {
   SchafferFunctionN1<arma::mat> SCH;
 
@@ -185,6 +200,7 @@ void CallbacksFullMultiobjectiveFunctionTest(OptimizerType& optimizer,
   REQUIRE(cb.calledEvaluateConstraint == calledEvaluateConstraint);
   REQUIRE(cb.calledGradientConstraint == calledGradientConstraint);
   REQUIRE(cb.calledStepTaken == calledStepTaken);
+  REQUIRE(cb.calledGenerationalStepTaken == calledGenerationalStepTaken);
 }
 
 template<typename OptimizerType>
@@ -380,7 +396,19 @@ TEST_CASE("NSGA2CallbacksFullFunctionTest", "[CallbackTest]")
   arma::vec upperBound = {1000};
   NSGA2 optimizer(20, 5000, 0.5, 0.5, 1e-3, 1e-6, lowerBound, upperBound);
   CallbacksFullMultiobjectiveFunctionTest(optimizer, false, false, false, false,
-      true, true, false, false, true);
+      true, true, false, false, false, true);
+}
+
+/**
+ * Make sure we invoke all callbacks (MOEA/D-DE).
+ */
+TEST_CASE("MOEADCallbacksFullFunctionTest", "[CallbackTest]")
+{
+  arma::vec lowerBound = {-1000};
+  arma::vec upperBound = {1000};
+  DefaultMOEAD optimizer(150, 300, 1.0, 0.9, 20, 20, 0.5, 2, 1E-10, lowerBound, upperBound);
+  CallbacksFullMultiobjectiveFunctionTest(optimizer, false, false, false, false,
+      true, true, false, false, false, true);
 }
 
 /**
@@ -713,3 +741,45 @@ TEST_CASE("ProgressBarCallbackEpochTest", "[CallbacksTest]")
   s.Optimize(f, coordinates, ProgressBar(10, stream));
   REQUIRE(stream.str().find("Epoch 1/1") != std::string::npos);
 }
+
+/**
+ * Make sure the Report callback will show the report on the specified
+ * output stream.
+ */
+TEST_CASE("ReportCallbackTest", "[CallbacksTest]")
+{
+  std::stringstream stream;
+
+  SGDTestFunction f0;
+  StandardSGD s(0.0003, 1, 10000, 1e-9, true);
+
+  arma::mat coordinates = f0.GetInitialPoint();
+  s.Optimize(f0, coordinates, Report(0.1, stream));
+  REQUIRE(stream.str().length() > 0);
+
+  stream.str("");
+  RosenbrockWoodFunction f1;
+  L_BFGS lbfgs;
+  lbfgs.MaxIterations() = 100;
+
+  coordinates = f1.GetInitialPoint();
+  lbfgs.Optimize(f1, coordinates, Report(0.1, stream));
+  REQUIRE(stream.str().length() > 0);
+
+  stream.str("");
+  SchafferFunctionN2 f2;
+  CNE cne;
+  cne.MaxGenerations() = 100;
+
+  coordinates = f2.GetInitialPoint();
+  cne.Optimize(f2, coordinates, Report(0.1, stream));
+  REQUIRE(stream.str().length() > 0);
+
+  stream.str("");
+  AugLagrangianTestFunction f3;
+  AugLagrangian aug;
+
+  coordinates = f3.GetInitialPoint();
+  aug.Optimize(f3, coordinates, Report(0.1, stream));
+  REQUIRE(stream.str().length() > 0);
+}

tests/cmaes_test.cpp

@@ -17,55 +17,14 @@
 using namespace ens;
 using namespace ens::test;
 
-/**
- * Tests the CMA-ES optimizer using a simple test function.
- */
-TEST_CASE("SimpleTestFunction", "[CMAESTest]")
-{
-  SGDTestFunction f;
-  CMAES<> optimizer(0, -1, 1, 32, 200, -1);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.003));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.003));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(0.003));
-}
-
 /**
  * Run CMA-ES with the full selection policy on logistic regression and
  * make sure the results are acceptable.
  */
 TEST_CASE("CMAESLogisticRegressionTest", "[CMAESTest]")
 {
-  const size_t trials = 3;
-  bool success = false;
-  for (size_t trial = 0; trial < trials; ++trial)
-  {
-    arma::mat data, testData, shuffledData;
-    arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-    LogisticRegressionTestData(data, testData, shuffledData,
-        responses, testResponses, shuffledResponses);
-    LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
-    CMAES<> cmaes(0, -1, 1, 32, 200, 1e-3);
-    arma::mat coordinates = lr.GetInitialPoint();
-    cmaes.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    if (acc >= 99.7 && testAcc >= 99.4)
-    {
-      success = true;
-      break;
-    }
-  }
-
-  REQUIRE(success == true);
+  CMAES<> cmaes(0, -1, 1, 32, 200, 1e-3);
+  LogisticRegressionFunctionTest(cmaes, 0.003, 0.006, 5);
 }
 
 /**
@@ -74,33 +33,8 @@ TEST_CASE("CMAESLogisticRegressionTest", "[CMAESTest]")
  */
 TEST_CASE("ApproxCMAESLogisticRegressionTest", "[CMAESTest]")
 {
-  const size_t trials = 3;
-  bool success = false;
-  for (size_t trial = 0; trial < trials; ++trial)
-  {
-    arma::mat data, testData, shuffledData;
-    arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-    LogisticRegressionTestData(data, testData, shuffledData,
-        responses, testResponses, shuffledResponses);
-    LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
-    ApproxCMAES<> cmaes(0, -1, 1, 32, 200, 1e-3);
-    arma::mat coordinates = lr.GetInitialPoint();
-    cmaes.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    if (acc >= 99.7 && testAcc >= 99.4)
-    {
-      success = true;
-      break;
-    }
-  }
-
-  REQUIRE(success == true);
+  ApproxCMAES<> cmaes(0, -1, 1, 32, 200, 1e-3);
+  LogisticRegressionFunctionTest(cmaes, 0.003, 0.006, 5);
 }
 
 /**
@@ -109,33 +43,8 @@ TEST_CASE("ApproxCMAESLogisticRegressionTest", "[CMAESTest]")
  */
 TEST_CASE("CMAESLogisticRegressionFMatTest", "[CMAESTest]")
 {
-  const size_t trials = 3;
-  bool success = false;
-  for (size_t trial = 0; trial < trials; ++trial)
-  {
-    arma::fmat data, testData, shuffledData;
-    arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-    LogisticRegressionTestData(data, testData, shuffledData,
-        responses, testResponses, shuffledResponses);
-    LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
-    CMAES<> cmaes(0, -1, 1, 32, 200, 1e-3);
-    arma::fmat coordinates = lr.GetInitialPoint();
-    cmaes.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    if (acc >= 99.0 && testAcc >= 98.0)
-    {
-      success = true;
-      break;
-    }
-  }
-
-  REQUIRE(success == true);
+  CMAES<> cmaes(0, -1, 1, 32, 200, 1e-3);
+  LogisticRegressionFunctionTest<arma::fmat>(cmaes, 0.01, 0.02, 5);
 }
 
 /**
@@ -144,31 +53,6 @@ TEST_CASE("CMAESLogisticRegressionFMatTest", "[CMAESTest]")
  */
 TEST_CASE("ApproxCMAESLogisticRegressionFMatTest", "[CMAESTest]")
 {
-  const size_t trials = 3;
-  bool success = false;
-  for (size_t trial = 0; trial < trials; ++trial)
-  {
-    arma::fmat data, testData, shuffledData;
-    arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-    LogisticRegressionTestData(data, testData, shuffledData,
-        responses, testResponses, shuffledResponses);
-    LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
-    ApproxCMAES<> cmaes(0, -1, 1, 32, 200, 1e-3);
-    arma::fmat coordinates = lr.GetInitialPoint();
-    cmaes.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    if (acc >= 99.0 && testAcc >= 98.0)
-    {
-      success = true;
-      break;
-    }
-  }
-
-  REQUIRE(success == true);
+  ApproxCMAES<> cmaes(0, -1, 1, 32, 200, 1e-3);
+  LogisticRegressionFunctionTest<arma::fmat>(cmaes, 0.01, 0.02, 5);
 }

tests/cne_test.cpp

@@ -10,7 +10,6 @@
  * the 3-clause BSD license along with ensmallen.  If not, see
  * http://www.opensource.org/licenses/BSD-3-Clause for more information.
  */
-
 #include <ensmallen.hpp>
 #include "catch.hpp"
 #include "test_function_tools.hpp"
@@ -24,24 +23,8 @@ using namespace std;
  */
 TEST_CASE("CNELogisticRegressionTest", "[CNETest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   CNE opt(300, 150, 0.2, 0.2, 0.2, -1);
-  arma::mat coordinates = lr.GetInitialPoint();
-  opt.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest(opt, 0.003, 0.006);
 }
 
 /**
@@ -50,24 +33,8 @@ TEST_CASE("CNELogisticRegressionTest", "[CNETest]")
  */
 TEST_CASE("CNELogisticRegressionFMatTest", "[CNETest]")
 {
-  arma::fmat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
   CNE opt(300, 150, 0.2, 0.2, 0.2, -1);
-  arma::fmat coordinates = lr.GetInitialPoint();
-  opt.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest<arma::fmat>(opt, 0.003, 0.006);
 }
 
 /**
@@ -90,14 +57,8 @@ TEST_CASE("CNECrossInTrayFunctionTest", "[CNETest]")
  */
 TEST_CASE("CNEAckleyFunctionTest", "[CNETest]")
 {
-  AckleyFunction f;
   CNE optimizer(450, 1500, 0.3, 0.3, 0.3, -1);
-
-  arma::mat coordinates = arma::mat("3; 3");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0).margin(0.1));
+  FunctionTest<AckleyFunction>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -105,14 +66,8 @@ TEST_CASE("CNEAckleyFunctionTest", "[CNETest]")
  */
 TEST_CASE("CNEBealeFunctionTest", "[CNETest]")
 {
-  BealeFunction f;
   CNE optimizer(450, 1500, 0.3, 0.3, 0.3, -1);
-
-  arma::mat coordinates = arma::mat("3; 3");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(3).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.5).margin(0.1));
+  FunctionTest<BealeFunction>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -120,14 +75,8 @@ TEST_CASE("CNEBealeFunctionTest", "[CNETest]")
  */
 TEST_CASE("CNEGoldsteinPriceFunctionTest", "[CNETest]")
 {
-  GoldsteinPriceFunction f;
   CNE optimizer(450, 1500, 0.3, 0.3, 0.1, -1);
-
-  arma::mat coordinates = arma::mat("0.5; -0.5");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(-1).margin(0.1));
+  FunctionTest<GoldsteinPriceFunction>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -135,14 +84,8 @@ TEST_CASE("CNEGoldsteinPriceFunctionTest", "[CNETest]")
  */
 TEST_CASE("CNELevyFunctionN13Test", "[CNETest]")
 {
-  LevyFunctionN13 f;
   CNE optimizer(450, 1500, 0.3, 0.3, 0.02, -1);
-
-  arma::mat coordinates = arma::mat("1.5; 0.5");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(1).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(1).margin(0.1));
+  FunctionTest<LevyFunctionN13>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -165,14 +108,8 @@ TEST_CASE("CNEHimmelblauFunctionTest", "[CNETest]")
  */
 TEST_CASE("CNEThreeHumpCamelFunctionTest", "[CNETest]")
 {
-  ThreeHumpCamelFunction f;
   CNE optimizer(450, 1500, 0.3, 0.3, 0.3, -1);
-
-  arma::mat coordinates = arma::mat("1; 1");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0).margin(0.1));
+  FunctionTest<ThreeHumpCamelFunction>(optimizer, 0.5, 0.1);
 }
 
 // TODO: The CNE optimizer with the given parameter occasionally fails to find a
@@ -186,11 +123,26 @@ TEST_CASE("CNESchafferFunctionN4Test", "[CNETest]")
   SchafferFunctionN4 f;
   CNE optimizer(500, 1600, 0.3, 0.3, 0.3, -1);
 
-  arma::mat coordinates = arma::mat("0.5; 2");
-  optimizer.Optimize(f, coordinates);
+  // We allow a few trials.
+  for (size_t trial = 0; trial < 5; ++trial)
+  {
+    arma::mat coordinates = arma::mat("0.5; 2");
+    optimizer.Optimize(f, coordinates);
 
-  REQUIRE(coordinates(0) == Approx(0).margin(0.1));
-  REQUIRE(abs(coordinates(1)) == Approx(1.25313).margin(0.1));
+    if (trial != 4)
+    {
+      if (coordinates(0) != Approx(0).margin(0.1))
+        continue;
+      if (abs(coordinates(1)) != Approx(1.25313).margin(0.1))
+        continue;
+    }
+
+    REQUIRE(coordinates(0) == Approx(0).margin(0.1));
+    REQUIRE(abs(coordinates(1)) == Approx(1.25313).margin(0.1));
+
+    // The test was successfull or reached the maximum number of trials.
+    break;
+  }
 }
 
 /**
@@ -198,12 +150,7 @@ TEST_CASE("CNESchafferFunctionN4Test", "[CNETest]")
  */
 TEST_CASE("CNESchafferFunctionN2Test", "[CNETest]")
 {
-  SchafferFunctionN2 f;
+  // We allow a few trials in case convergence is not achieved.
   CNE optimizer(500, 1600, 0.3, 0.3, 0.3, -1);
-
-  arma::mat coordinates = arma::mat("0.5; -0.5");
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0).margin(0.1));
+  FunctionTest<SchafferFunctionN2>(optimizer, 0.5, 0.1, 7);
 }

tests/de_test.cpp

@@ -20,24 +20,8 @@ using namespace ens::test;
  */
 TEST_CASE("DELogisticRegressionTest", "[DETest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   DE opt(200, 1000, 0.6, 0.8, 1e-5);
-  arma::mat coordinates = lr.GetInitialPoint();
-  opt.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest(opt, 0.01, 0.02, 3);
 }
 
 /**
@@ -46,22 +30,6 @@ TEST_CASE("DELogisticRegressionTest", "[DETest]")
  */
 TEST_CASE("DELogisticRegressionFMatTest", "[DETest]")
 {
-  arma::fmat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData, responses,
-      testResponses, shuffledResponses);
-  LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
   DE opt(200, 1000, 0.6, 0.8, 1e-5);
-  arma::fmat coordinates = lr.GetInitialPoint();
-  opt.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const float acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.03)); // 3% error tolerance.
-
-  const float testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.06)); // 6% error tolerance.
+  LogisticRegressionFunctionTest<arma::fmat>(opt, 0.03, 0.06, 3);
 }

tests/eve_test.cpp

@@ -17,45 +17,13 @@
 using namespace ens;
 using namespace ens::test;
 
-/**
- * Test the Eve optimizer on the simple SGD function.
- */
-TEST_CASE("EveSGDFunction","[EveTest]")
-{
-  SGDTestFunction f;
-  Eve optimizer(1e-3, 1, 0.9, 0.999, 0.999, 1e-8, 10, 400000, 1e-9, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(0.1));
-}
-
 /**
  * Run Eve on logistic regression and make sure the results are acceptable.
  */
 TEST_CASE("EveLogisticRegressionTest","[EveTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   Eve optimizer(1e-3, 1, 0.9, 0.999, 0.999, 1e-8, 10000, 500000, 1e-9, true);
-  arma::mat coordinates = lr.GetInitialPoint();
-  optimizer.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest(optimizer, 0.003, 0.006);
 }
 
 /**
@@ -63,14 +31,8 @@ TEST_CASE("EveLogisticRegressionTest","[EveTest]")
  */
 TEST_CASE("EveSphereFunctionTest","[EveTest]")
 {
-  SphereFunction f(2);
   Eve optimizer(1e-3, 2, 0.9, 0.999, 0.999, 1e-8, 10000, 500000, 1e-9, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -78,14 +40,8 @@ TEST_CASE("EveSphereFunctionTest","[EveTest]")
  */
 TEST_CASE("EveStyblinskiTangFunctionTest","[EveTest]")
 {
-  StyblinskiTangFunction f(2);
   Eve optimizer(1e-3, 2, 0.9, 0.999, 0.999, 1e-8, 10000, 500000, 1e-9, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(-2.9).epsilon(0.01));
-  REQUIRE(coordinates(1) == Approx(-2.9).epsilon(0.01));
+  FunctionTest<StyblinskiTangFunction>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -94,14 +50,8 @@ TEST_CASE("EveStyblinskiTangFunctionTest","[EveTest]")
  */
 TEST_CASE("EveStyblinskiTangFunctionFMatTest","[EveTest]")
 {
-  StyblinskiTangFunction f(2);
   Eve optimizer(1e-3, 2, 0.9, 0.999, 0.999, 1e-8, 10000, 500000, 1e-9, true);
-
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(-2.9).epsilon(0.01));
-  REQUIRE(coordinates(1) == Approx(-2.9).epsilon(0.01));
+  FunctionTest<StyblinskiTangFunction, arma::fmat>(optimizer, 0.5, 0.1);
 }
 
 #if ARMA_VERSION_MAJOR > 9 ||\
@@ -113,14 +63,8 @@ TEST_CASE("EveStyblinskiTangFunctionFMatTest","[EveTest]")
  */
 TEST_CASE("EveStyblinskiTangFunctionSpMatTest","[EveTest]")
 {
-  StyblinskiTangFunction f(2);
   Eve optimizer(1e-3, 2, 0.9, 0.999, 0.999, 1e-8, 10000, 500000, 1e-9, true);
-
-  arma::sp_mat coordinates = f.GetInitialPoint<arma::sp_mat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(-2.9).epsilon(0.01));
-  REQUIRE(coordinates(1) == Approx(-2.9).epsilon(0.01));
+  FunctionTest<StyblinskiTangFunction, arma::sp_mat>(optimizer, 0.5, 0.1);
 }
 
 #endif

tests/ftml_test.cpp

@@ -17,45 +17,13 @@
 using namespace ens;
 using namespace ens::test;
 
-/**
- * Test the FTML optimizer on the simple SGD function.
- */
-TEST_CASE("FTMLSGDFunction", "[FTMLTest]")
-{
-  SGDTestFunction f;
-  FTML optimizer(0.005, 1, 0.9, 0.999, 1e-8, 1000000, 1e-9, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(0.1));
-}
-
 /**
  * Run FTML on logistic regression and make sure the results are acceptable.
  */
 TEST_CASE("FTMLLogisticRegressionTest", "[FTMLTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   FTML optimizer(0.001, 1, 0.9, 0.999, 1e-8, 100000, 1e-5, true);
-  arma::mat coordinates = lr.GetInitialPoint();
-  optimizer.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest(optimizer, 0.003, 0.006);
 }
 
 /**
@@ -63,13 +31,8 @@ TEST_CASE("FTMLLogisticRegressionTest", "[FTMLTest]")
  */
 TEST_CASE("FTMLSphereFunctionTest", "[FTMLTest]")
 {
-  SphereFunction f(2);
   FTML optimizer(0.001, 2, 0.9, 0.999, 1e-8, 500000, 1e-9, true);
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -77,14 +40,8 @@ TEST_CASE("FTMLSphereFunctionTest", "[FTMLTest]")
  */
 TEST_CASE("FTMLStyblinskiTangFunctionTest", "[FTMLTest]")
 {
-  StyblinskiTangFunction f(2);
   FTML optimizer(0.001, 2, 0.9, 0.999, 1e-8, 100000, 1e-5, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(-2.9).epsilon(0.01));
-  REQUIRE(coordinates(1) == Approx(-2.9).epsilon(0.01));
+  FunctionTest<StyblinskiTangFunction>(optimizer, 0.5, 0.1);
 }
 
 /**
@@ -93,14 +50,8 @@ TEST_CASE("FTMLStyblinskiTangFunctionTest", "[FTMLTest]")
  */
 TEST_CASE("FTMLStyblinskiTangFunctionFMatTest", "[FTMLTest]")
 {
-  StyblinskiTangFunction f(2);
   FTML optimizer(0.001, 2, 0.9, 0.999, 1e-8, 100000, 1e-5, true);
-
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(-2.9).epsilon(0.01));
-  REQUIRE(coordinates(1) == Approx(-2.9).epsilon(0.01));
+  FunctionTest<StyblinskiTangFunction, arma::fmat>(optimizer, 0.5, 0.1);
 }
 
 // A test with sp_mat is not done, because FTML uses some parts internally that

tests/gradient_descent_test.cpp

@@ -12,50 +12,25 @@
 
 #include <ensmallen.hpp>
 #include "catch.hpp"
+#include "test_function_tools.hpp"
 
 using namespace ens;
 using namespace ens::test;
 
 TEST_CASE("SimpleGDTestFunction", "[GradientDescentTest]")
 {
-  GDTestFunction f;
   GradientDescent s(0.01, 5000000, 1e-9);
-
-  arma::mat coordinates = f.GetInitialPoint<arma::mat>();
-  double result = s.Optimize(f, coordinates);
-
-  REQUIRE(result == Approx(0.0).margin(1e-4));
-  REQUIRE(coordinates(0) == Approx(0.0).margin(1e-2));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(1e-2));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(1e-2));
+  FunctionTest<GDTestFunction>(s, 0.1, 0.01);
 }
 
 TEST_CASE("GDRosenbrockTest", "[GradientDescentTest]")
 {
-  // Create the Rosenbrock function.
-  RosenbrockFunction f;
-
   GradientDescent s(0.001, 0, 1e-15);
-
-  arma::mat coordinates = f.GetInitialPoint<arma::mat>();
-  double result = s.Optimize(f, coordinates);
-
-  REQUIRE(result == Approx(0.0).margin(1e-10));
-  for (size_t j = 0; j < 2; ++j)
-    REQUIRE(coordinates(j) == Approx(1.0).epsilon(1e-5));
+  FunctionTest<RosenbrockFunction>(s, 0.01, 0.001);
 }
 
 TEST_CASE("GDRosenbrockFMatTest", "[GradientDescentTest]")
 {
-  // Create the Rosenbrock function.
-  RosenbrockFunction f;
-
   GradientDescent s(0.001, 0, 1e-15);
-
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
-  float result = s.Optimize(f, coordinates);
-
-  REQUIRE(result == Approx(0.0).margin(1e-5));
-  for (size_t j = 0; j < 2; ++j)
-    REQUIRE(coordinates(j) == Approx(1.0).epsilon(1e-3));
+  FunctionTest<RosenbrockFunction, arma::fmat>(s, 0.1, 0.01);
 }

tests/indicators_test.cpp

@@ -0,0 +1,97 @@
+/**
+ * @file indicators_test.cpp
+ * @author Nanubala Gnana Sai
+ *
+ * Test file for all the indicators: Epsilon, IGD+.
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#include <ensmallen.hpp>
+#include "catch.hpp"
+
+using namespace ens;
+using namespace ens::test;
+
+/**
+ * Calculates the Epsilon metric for the pair of fronts.
+ * Tests for data of type double.
+ * The reference numerical results have been taken from hand calculated values.
+ * Refer the IPynb notebook in https://github.com/mlpack/ensmallen/pull/285
+ * for more.
+ */
+TEST_CASE("EpsilonDoubleTest", "[IndicatorsTest]")
+{
+  arma::cube referenceFront(2, 1, 3);
+  double tol = 1e-10;
+  referenceFront.slice(0) = arma::vec{0.01010101, 0.89949622};
+  referenceFront.slice(1) = arma::vec{0.02020202, 0.85786619};
+  referenceFront.slice(2) = arma::vec{0.03030303, 0.82592234};
+  arma::cube front = referenceFront * 1.1;
+  double eps =  Epsilon::Evaluate(front, referenceFront);
+
+  REQUIRE(eps == Approx(1.1).margin(tol));
+}
+
+/**
+ * Calculates the Epsilon metric for the pair of fronts.
+ * Tests for data of type float.
+ * The reference numerical results have been taken from hand calculated values.
+ * Refer the IPynb notebook in https://github.com/mlpack/ensmallen/pull/285
+ * for more.
+ */
+TEST_CASE("EpsilonFloatTest", "[IndicatorsTest]")
+{
+  arma::fcube referenceFront(2, 1, 3);
+  float tol = 1e-10;
+  referenceFront.slice(0) = arma::fvec{0.01010101f, 0.89949622f};
+  referenceFront.slice(1) = arma::fvec{0.02020202f, 0.85786619f};
+  referenceFront.slice(2) = arma::fvec{0.03030303f, 0.82592234f};
+  arma::fcube front = referenceFront * 1.1;
+  double eps =  Epsilon::Evaluate(front, referenceFront);
+
+  REQUIRE(eps == Approx(1.1).margin(tol));
+}
+
+/**
+ * Calculates the IGD+ metric for the pair of fronts.
+ * Tests for data of type double.
+ * The reference numerical results have been taken from hand calculated values.
+ * Refer the IPynb notebook in https://github.com/mlpack/ensmallen/pull/285
+ * for more.
+ */
+TEST_CASE("IGDPlusDoubleTest", "[IndicatorsTest]")
+{
+  arma::cube referenceFront(2, 1, 3);
+  double tol = 1e-10;
+  referenceFront.slice(0) = arma::vec{0.01010101, 0.89949622};
+  referenceFront.slice(1) = arma::vec{0.02020202, 0.85786619};
+  referenceFront.slice(2) = arma::vec{0.03030303, 0.82592234};
+  arma::cube front = referenceFront * 1.1;
+  double igdPlus = IGDPlus::Evaluate(front, referenceFront);
+
+  REQUIRE(igdPlus == Approx(0.05329735411078149).margin(tol));
+}
+
+/**
+ * Calculates the IGD+ metric for the pair of fronts.
+ * Tests for data of type float.
+ * The reference numerical results have been taken from hand calculated values.
+ * Refer the IPynb notebook in https://github.com/mlpack/ensmallen/pull/285
+ * for more.
+ */
+TEST_CASE("IGDPlusFloatTest", "[IndicatorsTest]")
+{
+  arma::fcube referenceFront(2, 1, 3);
+  float tol = 1e-10;
+  referenceFront.slice(0) = arma::fvec{0.01010101f, 0.89949622f};
+  referenceFront.slice(1) = arma::fvec{0.02020202f, 0.85786619f};
+  referenceFront.slice(2) = arma::fvec{0.03030303f, 0.82592234f};
+  arma::fcube front = referenceFront * 1.1;
+  float igdPlus = IGDPlus::Evaluate(front, referenceFront);
+
+  REQUIRE(igdPlus == Approx(0.05329735411078149).margin(tol));
+}

tests/iqn_test.cpp

@@ -22,29 +22,11 @@ using namespace ens::test;
  */
 TEST_CASE("IQNLogisticRegressionTest", "[IQNTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
-  // Now run SGDR with snapshot ensembles on a couple of batch sizes.
+  // Run on a couple of batch sizes.
   for (size_t batchSize = 1; batchSize < 9; batchSize += 4)
   {
     IQN iqn(0.01, batchSize, 5000, 0.01);
-    LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
-    arma::mat coordinates = lr.GetInitialPoint();
-    iqn.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.013)); // 1.3% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.016)); // 1.6% error tolerance.
+    LogisticRegressionFunctionTest(iqn, 0.013, 0.016);
   }
 }
 
@@ -54,28 +36,10 @@ TEST_CASE("IQNLogisticRegressionTest", "[IQNTest]")
  */
 TEST_CASE("IQNLogisticRegressionFMatTest", "[IQNTest]")
 {
-  arma::fmat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
-  // Now run SGDR with snapshot ensembles on a couple of batch sizes.
+  // Run on a couple of batch sizes.
   for (size_t batchSize = 1; batchSize < 9; batchSize += 4)
   {
     IQN iqn(0.001, batchSize, 5000, 0.01);
-    LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
-    arma::fmat coordinates = lr.GetInitialPoint();
-    iqn.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.013)); // 1.3% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.016)); // 1.6% error tolerance.
+    LogisticRegressionFunctionTest<arma::fmat>(iqn, 0.013, 0.016);
   }
 }

tests/katyusha_test.cpp

@@ -21,28 +21,11 @@ using namespace ens::test;
  */
 TEST_CASE("KatyushaLogisticRegressionTest", "[KatyushaTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run with a couple of batch sizes.
   for (size_t batchSize = 30; batchSize < 45; batchSize += 5)
   {
     Katyusha optimizer(1.0, 10.0, batchSize, 100, 0, 1e-10, true);
-    LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
-    arma::mat coordinates = lr.GetInitialPoint();
-    optimizer.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
+    LogisticRegressionFunctionTest(optimizer, 0.015, 0.015);
   }
 }
 
@@ -52,28 +35,11 @@ TEST_CASE("KatyushaLogisticRegressionTest", "[KatyushaTest]")
  */
 TEST_CASE("KatyushaProximalLogisticRegressionTest", "[KatyushaTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run with a couple of batch sizes.
   for (size_t batchSize = 30; batchSize < 45; batchSize += 5)
   {
     KatyushaProximal optimizer(1.0, 10.0, batchSize, 100, 0, 1e-10, true);
-    LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
-    arma::mat coordinates = lr.GetInitialPoint();
-    optimizer.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
+    LogisticRegressionFunctionTest(optimizer, 0.015, 0.015);
   }
 }
 
@@ -83,28 +49,11 @@ TEST_CASE("KatyushaProximalLogisticRegressionTest", "[KatyushaTest]")
  */
 TEST_CASE("KatyushaLogisticRegressionFMatTest", "[KatyushaTest]")
 {
-  arma::fmat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run with a couple of batch sizes.
   for (size_t batchSize = 30; batchSize < 45; batchSize += 5)
   {
     Katyusha optimizer(1.0, 10.0, batchSize, 100, 0, 1e-10, true);
-    LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
-    arma::fmat coordinates = lr.GetInitialPoint();
-    optimizer.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
+    LogisticRegressionFunctionTest<arma::fmat>(optimizer, 0.015, 0.015);
   }
 }
 
@@ -114,28 +63,11 @@ TEST_CASE("KatyushaLogisticRegressionFMatTest", "[KatyushaTest]")
  */
 TEST_CASE("KatyushaProximalLogisticRegressionFMatTest", "[KatyushaTest]")
 {
-  arma::fmat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run with a couple of batch sizes.
   for (size_t batchSize = 30; batchSize < 45; batchSize += 5)
   {
     KatyushaProximal optimizer(1.0, 10.0, batchSize, 100, 0, 1e-10, true);
-    LogisticRegression<arma::fmat> lr(shuffledData, shuffledResponses, 0.5);
-
-    arma::fmat coordinates = lr.GetInitialPoint();
-    optimizer.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
+    LogisticRegressionFunctionTest<arma::fmat>(optimizer, 0.015, 0.015);
   }
 }
 
@@ -148,28 +80,11 @@ TEST_CASE("KatyushaProximalLogisticRegressionFMatTest", "[KatyushaTest]")
  */
 TEST_CASE("KatyushaLogisticRegressionSpMatTest", "[KatyushaTest]")
 {
-  arma::sp_mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run with a couple of batch sizes.
   for (size_t batchSize = 30; batchSize < 45; batchSize += 5)
   {
     Katyusha optimizer(1.0, 10.0, batchSize, 100, 0, 1e-10, true);
-    LogisticRegression<arma::sp_mat> lr(shuffledData, shuffledResponses, 0.5);
-
-    arma::sp_mat coordinates = lr.GetInitialPoint();
-    optimizer.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
+    LogisticRegressionFunctionTest<arma::sp_mat>(optimizer, 0.015, 0.015);
   }
 }
 
@@ -179,28 +94,11 @@ TEST_CASE("KatyushaLogisticRegressionSpMatTest", "[KatyushaTest]")
  */
 TEST_CASE("KatyushaProximalLogisticRegressionSpMatTest", "[KatyushaTest]")
 {
-  arma::sp_mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-
-  // Now run big-batch SGD with a couple of batch sizes.
+  // Run with a couple of batch sizes.
   for (size_t batchSize = 30; batchSize < 45; batchSize += 5)
   {
     KatyushaProximal optimizer(1.0, 10.0, batchSize, 100, 0, 1e-10, true);
-    LogisticRegression<arma::sp_mat> lr(shuffledData, shuffledResponses, 0.5);
-
-    arma::sp_mat coordinates = lr.GetInitialPoint();
-    optimizer.Optimize(lr, coordinates);
-
-    // Ensure that the error is close to zero.
-    const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-    REQUIRE(acc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
-
-    const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-        coordinates);
-    REQUIRE(testAcc == Approx(100.0).epsilon(0.015)); // 1.5% error tolerance.
+    LogisticRegressionFunctionTest<arma::sp_mat>(optimizer, 0.015, 0.015);
   }
 }
 

tests/lbfgs_test.cpp

@@ -12,6 +12,7 @@
 
 #include <ensmallen.hpp>
 #include "catch.hpp"
+#include "test_function_tools.hpp"
 
 using namespace ens;
 using namespace ens::test;
@@ -21,18 +22,9 @@ using namespace ens::test;
  */
 TEST_CASE("RosenbrockFunctionTest", "[LBFGSTest]")
 {
-  RosenbrockFunction f;
   L_BFGS lbfgs;
   lbfgs.MaxIterations() = 10000;
-
-  arma::mat coords = f.GetInitialPoint();
-  lbfgs.Optimize(f, coords);
-
-  double finalValue = f.Evaluate(coords);
-
-  REQUIRE(finalValue == Approx(0.0).margin(1e-5));
-  REQUIRE(coords(0) == Approx(1.0).epsilon(1e-7));
-  REQUIRE(coords(1) == Approx(1.0).epsilon(1e-7));
+  FunctionTest<RosenbrockFunction>(lbfgs, 0.01, 0.001);
 }
 
 /**
@@ -40,18 +32,9 @@ TEST_CASE("RosenbrockFunctionTest", "[LBFGSTest]")
  */
 TEST_CASE("RosenbrockFunctionFloatTest", "[LBFGSTest]")
 {
-  RosenbrockFunction f;
   L_BFGS lbfgs;
   lbfgs.MaxIterations() = 10000;
-
-  arma::fmat coords = f.GetInitialPoint<arma::fvec>();
-  lbfgs.Optimize(f, coords);
-
-  float finalValue = f.Evaluate(coords);
-
-  REQUIRE(finalValue == Approx(0.0f).margin(1e-3));
-  REQUIRE(coords(0) == Approx(1.0f).epsilon(1e-4));
-  REQUIRE(coords(1) == Approx(1.0f).epsilon(1e-4));
+  FunctionTest<RosenbrockFunction, arma::fmat>(lbfgs, 0.1, 0.01);
 }
 
 /**
@@ -79,18 +62,9 @@ TEST_CASE("RosenbrockFunctionSpGradTest", "[LBFGSTest]")
  */
 TEST_CASE("RosenbrockFunctionSpMatTest", "[LBFGSTest]")
 {
-  RosenbrockFunction f;
   L_BFGS lbfgs;
   lbfgs.MaxIterations() = 10000;
-
-  arma::sp_mat coords = f.GetInitialPoint<arma::sp_vec>();
-  lbfgs.Optimize(f, coords);
-
-  double finalValue = f.Evaluate(coords);
-
-  REQUIRE(finalValue == Approx(0.0).margin(1e-5));
-  REQUIRE(coords(0) == Approx(1.0).epsilon(1e-7));
-  REQUIRE(coords(1) == Approx(1.0).epsilon(1e-7));
+  FunctionTest<RosenbrockFunction, arma::sp_mat>(lbfgs, 0.01, 0.001);
 }
 
 /**
@@ -98,15 +72,9 @@ TEST_CASE("RosenbrockFunctionSpMatTest", "[LBFGSTest]")
  */
 TEST_CASE("ColvilleFunctionTest", "[LBFGSTest]")
 {
-  ColvilleFunction f;
   L_BFGS lbfgs;
   lbfgs.MaxIterations() = 10000;
-
-  arma::vec coords = f.GetInitialPoint();
-  lbfgs.Optimize(f, coords);
-
-  REQUIRE(coords(0) == Approx(1.0).epsilon(1e-7));
-  REQUIRE(coords(1) == Approx(1.0).epsilon(1e-7));
+  FunctionTest<ColvilleFunction>(lbfgs, 0.01, 0.001);
 }
 
 /**
@@ -114,20 +82,9 @@ TEST_CASE("ColvilleFunctionTest", "[LBFGSTest]")
  */
 TEST_CASE("WoodFunctionTest", "[LBFGSTest]")
 {
-  WoodFunction f;
   L_BFGS lbfgs;
   lbfgs.MaxIterations() = 10000;
-
-  arma::vec coords = f.GetInitialPoint();
-  lbfgs.Optimize(f, coords);
-
-  double finalValue = f.Evaluate(coords);
-
-  REQUIRE(finalValue == Approx(0.0).margin(1e-5));
-  REQUIRE(coords(0) == Approx(1.0).epsilon(1e-7));
-  REQUIRE(coords(1) == Approx(1.0).epsilon(1e-7));
-  REQUIRE(coords(2) == Approx(1.0).epsilon(1e-7));
-  REQUIRE(coords(3) == Approx(1.0).epsilon(1e-7));
+  FunctionTest<WoodFunction>(lbfgs, 0.01, 0.001);
 }
 
 /**
@@ -164,19 +121,7 @@ TEST_CASE("GeneralizedRosenbrockFunctionTest", "[LBFGSTest]")
  */
 TEST_CASE("RosenbrockWoodFunctionTest", "[LBFGSTest]")
 {
-  RosenbrockWoodFunction f;
   L_BFGS lbfgs;
   lbfgs.MaxIterations() = 10000;
-
-  arma::mat coords = f.GetInitialPoint();
-  lbfgs.Optimize(f, coords);
-
-  double finalValue = f.Evaluate(coords);
-
-  REQUIRE(finalValue == Approx(0.0).margin(1e-5));
-  for (int row = 0; row < 4; row++)
-  {
-    REQUIRE((coords(row, 0)) == Approx(1.0).epsilon(1e-7));
-    REQUIRE((coords(row, 1)) == Approx(1.0).epsilon(1e-7));
-  }
+  FunctionTest<RosenbrockWoodFunction>(lbfgs, 0.01, 0.001);
 }

tests/lookahead_test.cpp

@@ -20,38 +20,13 @@ using namespace ens::test;
  */
 TEST_CASE("LookaheadAdamSphereFunctionTest", "[LookaheadTest]")
 {
-  SphereFunction f(2);
-
   Lookahead<> optimizer(0.5, 5, 100000, 1e-5, NoDecay(), false, true);
   optimizer.BaseOptimizer().StepSize() = 0.1;
   optimizer.BaseOptimizer().BatchSize() = 2;
   optimizer.BaseOptimizer().Beta1() = 0.7;
   optimizer.BaseOptimizer().Tolerance() = 1e-15;
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
-}
-
-/**
- * Test the Lookahead - Adam optimizer on the SGDTest function.
- */
-TEST_CASE("LookaheadAdamSimpleSGDTestFunction", "[LookaheadTest]")
-{
-  SGDTestFunction f;
-
-  Adam adam(0.001, 1, 0.9, 0.999, 1e-8, 5, 1e-19, false, true);
-  Lookahead<Adam> optimizer(adam, 0.5, 5, 100000, 1e-15, NoDecay(),
-      false, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(1e-3));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(1e-3));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(1e-3));
+  // We allow a few trials.
+  FunctionTest<SphereFunction>(optimizer, 0.5, 0.2, 3);
 }
 
 /**
@@ -59,17 +34,10 @@ TEST_CASE("LookaheadAdamSimpleSGDTestFunction", "[LookaheadTest]")
  */
 TEST_CASE("LookaheadAdaGradSphereFunction", "[LookaheadTest]")
 {
-  SphereFunction f(2);
-
   AdaGrad adagrad(0.99, 1, 1e-8, 5, 1e-15, true);
   Lookahead<AdaGrad> optimizer(adagrad, 0.5, 5, 5000000, 1e-15, NoDecay(),
       false, true);
-
-  arma::mat coordinates = f.GetInitialPoint();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(0.1));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(0.1));
+  FunctionTest<SphereFunction>(optimizer, 0.5, 0.2, 3);
 }
 
 /**
@@ -78,44 +46,19 @@ TEST_CASE("LookaheadAdaGradSphereFunction", "[LookaheadTest]")
  */
 TEST_CASE("LookaheadAdamLogisticRegressionTest","[LookaheadTest]")
 {
-  arma::mat data, testData, shuffledData;
-  arma::Row<size_t> responses, testResponses, shuffledResponses;
-
-  LogisticRegressionTestData(data, testData, shuffledData,
-      responses, testResponses, shuffledResponses);
-  LogisticRegression<> lr(shuffledData, shuffledResponses, 0.5);
-
   Adam adam(0.001, 32, 0.9, 0.999, 1e-8, 5, 1e-19);
   Lookahead<Adam> optimizer(adam, 0.5, 20, 100000, 1e-15, NoDecay(),
       false, true);
-
-  arma::mat coordinates = lr.GetInitialPoint();
-  optimizer.Optimize(lr, coordinates);
-
-  // Ensure that the error is close to zero.
-  const double acc = lr.ComputeAccuracy(data, responses, coordinates);
-  REQUIRE(acc == Approx(100.0).epsilon(0.003)); // 0.3% error tolerance.
-
-  const double testAcc = lr.ComputeAccuracy(testData, testResponses,
-      coordinates);
-  REQUIRE(testAcc == Approx(100.0).epsilon(0.006)); // 0.6% error tolerance.
+  LogisticRegressionFunctionTest(optimizer, 0.003, 0.006);
 }
 
 /**
- * Test the Lookahead - Adam optimizer on the SGDTest function (float).
+ * Test the Lookahead - Adam optimizer on the Sphere function (float).
  */
-TEST_CASE("LookaheadAdamSimpleSGDTestFunctionFloat", "[LookaheadTest]")
+TEST_CASE("LookaheadAdamSimpleSphereFunctionFloat", "[LookaheadTest]")
 {
-  SGDTestFunction f;
-
   Adam adam(0.001, 1, 0.9, 0.999, 1e-8, 5, 1e-19, false, true);
   Lookahead<Adam> optimizer(adam, 0.5, 5, 100000, 1e-15, NoDecay(),
       false, true);
-
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
-  optimizer.Optimize(f, coordinates);
-
-  REQUIRE(coordinates(0) == Approx(0.0).margin(1e-3));
-  REQUIRE(coordinates(1) == Approx(0.0).margin(1e-3));
-  REQUIRE(coordinates(2) == Approx(0.0).margin(1e-3));
+  FunctionTest<SphereFunction, arma::fmat>(optimizer, 0.5, 0.2, 3);
 }

tests/main.cpp

@@ -17,18 +17,23 @@
 
 int main(int argc, char** argv)
 {
-  /**
-   * Uncomment these three lines if you want to test with different random seeds
-   * each run.  This is good for ensuring that a test's tolerance is sufficient
-   * across many different runs.
-   */
-  //size_t seed = std::time(NULL);
-  //srand((unsigned int) seed);
-  //arma::arma_rng::set_seed(seed);
+  Catch::Session session;
+  const int returnCode = session.applyCommandLine(argc, argv);
+  // Check for a command line error.
+  if (returnCode != 0)
+    return returnCode;
 
   std::cout << "ensmallen version: " << ens::version::as_string() << std::endl;
-
   std::cout << "armadillo version: " << arma::arma_version::as_string() << std::endl;
 
-  return Catch::Session().run(argc, argv);
+  // Use Catch2 command-line to set the random seed.
+  // -rng-seed <'time'|number>
+  // If a number is provided this is used directly as the seed. Alternatively
+  // if the keyword 'time' is provided then the result of calling std::time(0)
+  // is used.
+  const size_t seed = session.config().rngSeed();
+  srand((unsigned int) seed);
+  arma::arma_rng::set_seed(seed);
+
+  return session.run();
 }

tests/moead_test.cpp

@@ -0,0 +1,618 @@
+/**
+ * @file moead_test.cpp
+ * @author Nanubala Gnana Sai
+ *
+ * ensmallen is free software; you may redistribute it and/or modify it under
+ * the terms of the 3-clause BSD license.  You should have received a copy of
+ * the 3-clause BSD license along with ensmallen.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#include <ensmallen.hpp>
+#include "catch.hpp"
+#include "test_function_tools.hpp"
+
+using namespace ens;
+using namespace ens::test;
+using namespace std;
+
+/**
+ * Checks if low <= value <= high. Used by MOEADFonsecaFlemingTest.
+ *
+ * @param value The value being checked.
+ * @param low The lower bound.
+ * @param high The upper bound.
+ * @param roundoff To round off precision.
+ * @tparam The type of elements in the population set.
+ * @return true if value lies in the range [low, high].
+ * @return false if value does not lie in the range [low, high].
+ */
+template<typename ElemType>
+bool IsInBounds(const ElemType& value,
+                const ElemType& low,
+                const ElemType& high,
+                const ElemType& roundoff)
+{
+  return !(value < (low - roundoff)) && !((high + roundoff) < value);
+}
+
+/**
+ * Optimize for the Schaffer N.1 function using NSGA-II optimizer.
+ * Tests for data of type double.
+ */
+TEST_CASE("MOEADSchafferN1DoubleTest", "[MOEADTest]")
+{
+  SchafferFunctionN1<arma::mat> SCH;
+  const double lowerBound = -1000;
+  const double upperBound = 1000;
+  const double expectedLowerBound = 0.0;
+  const double expectedUpperBound = 2.0;
+
+  DefaultMOEAD opt(
+        300, // Population size.
+        300,  // Max generations.
+        1.0,  // Crossover probability.
+        0.9, // Probability of sampling from neighbor.
+        20, // Neighborhood size.
+        20, // Perturbation index.
+        0.5, // Differential weight.
+        2, // Max childrens to replace parents.
+        1E-10, // epsilon.
+        lowerBound, // Lower bound.
+        upperBound // Upper bound.
+      );
+
+  typedef decltype(SCH.objectiveA) ObjectiveTypeA;
+  typedef decltype(SCH.objectiveB) ObjectiveTypeB;
+
+  // We allow a few trials in case of poor convergence.
+  bool success = false;
+  for (size_t trial = 0; trial < 3; ++trial)
+  {
+    arma::mat coords = SCH.GetInitialPoint();
+    std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
+
+    opt.Optimize(objectives, coords);
+    arma::cube paretoSet= opt.ParetoSet();
+
+    bool allInRange = true;
+
+    for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+    {
+      double val = arma::as_scalar(paretoSet.slice(solutionIdx));
+      if (!IsInBounds<double>(val, expectedLowerBound, expectedUpperBound, 0.1))
+      {
+        allInRange = false;
+        break;
+      }
+    }
+
+    if (allInRange)
+    {
+      success = true;
+      break;
+    }
+  }
+
+  REQUIRE(success == true);
+}
+
+/**
+ * Optimize for the Schaffer N.1 function using NSGA-II optimizer.
+ * Tests for data of type double.
+ */
+TEST_CASE("MOEADSchafferN1TestVectorDoubleBounds", "[MOEADTest]")
+{
+  // This test can be a little flaky, so we try it a few times.
+  SchafferFunctionN1<arma::mat> SCH;
+  const arma::vec lowerBound = {-1000};
+  const arma::vec upperBound = {1000};
+  const double expectedLowerBound = 0.0;
+  const double expectedUpperBound = 2.0;
+
+  DefaultMOEAD opt(
+        300, // Population size.
+        300,  // Max generations.
+        1.0,  // Crossover probability.
+        0.9, // Probability of sampling from neighbor.
+        20, // Neighborhood size.
+        20, // Perturbation index.
+        0.5, // Differential weight.
+        2, // Max childrens to replace parents.
+        1E-10, // epsilon.
+        lowerBound, // Lower bound.
+        upperBound // Upper bound.
+      );
+
+  typedef decltype(SCH.objectiveA) ObjectiveTypeA;
+  typedef decltype(SCH.objectiveB) ObjectiveTypeB;
+
+  bool success = false;
+  for (size_t trial = 0; trial < 3; ++trial)
+  {
+    arma::mat coords = SCH.GetInitialPoint();
+    std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
+
+    opt.Optimize(objectives, coords);
+    arma::cube paretoSet = opt.ParetoSet();
+
+    bool allInRange = true;
+
+    for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+    {
+      double val = arma::as_scalar(paretoSet.slice(solutionIdx));
+      if (!IsInBounds<double>(val, expectedLowerBound, expectedUpperBound, 0.1))
+      {
+        allInRange = false;
+        break;
+      }
+    }
+
+    if (allInRange)
+    {
+      success = true;
+      break;
+    }
+  }
+
+  REQUIRE(success == true);
+}
+
+/**
+ * Optimize for the Fonseca Fleming function using NSGA-II optimizer.
+ * Tests for data of type double.
+ */
+TEST_CASE("MOEADFonsecaFlemingDoubleTest", "[MOEADTest]")
+{
+  FonsecaFlemingFunction<arma::mat> FON;
+  const double lowerBound = -4;
+  const double upperBound = 4;
+  const double expectedLowerBound = -1.0 / sqrt(3);
+  const double expectedUpperBound = 1.0 / sqrt(3);
+
+  DefaultMOEAD opt(
+        300,  // Max generations.
+        300,  // Max generations.
+        1.0,  // Crossover probability.
+        0.9, // Probability of sampling from neighbor.
+        20, // Neighborhood size.
+        20, // Perturbation index.
+        0.5, // Differential weight.
+        2, // Max childrens to replace parents.
+        1E-10, // epsilon.
+        lowerBound, // Lower bound.
+        upperBound // Upper bound.
+      );
+  typedef decltype(FON.objectiveA) ObjectiveTypeA;
+  typedef decltype(FON.objectiveB) ObjectiveTypeB;
+
+  arma::mat coords = FON.GetInitialPoint();
+  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = FON.GetObjectives();
+
+  opt.Optimize(objectives, coords);
+  arma::cube paretoSet = opt.ParetoSet();
+
+  bool allInRange = true;
+
+  for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+  {
+    const arma::mat solution = paretoSet.slice(solutionIdx);
+    double valX = arma::as_scalar(solution(0));
+    double valY = arma::as_scalar(solution(1));
+    double valZ = arma::as_scalar(solution(2));
+
+    if (!IsInBounds<double>(valX, expectedLowerBound, expectedUpperBound, 0.1) ||
+        !IsInBounds<double>(valY, expectedLowerBound, expectedUpperBound, 0.1) ||
+        !IsInBounds<double>(valZ, expectedLowerBound, expectedUpperBound, 0.1))
+    {
+      allInRange = false;
+      break;
+    }
+  }
+
+  REQUIRE(allInRange);
+}
+
+/**
+ * Optimize for the Fonseca Fleming function using NSGA-II optimizer.
+ * Tests for data of type double.
+ */
+TEST_CASE("MOEADFonsecaFlemingTestVectorDoubleBounds", "[MOEADTest]")
+{
+  FonsecaFlemingFunction<arma::mat> FON;
+  const arma::vec lowerBound = {-4, -4, -4};
+  const arma::vec upperBound = {4, 4, 4};
+  const double expectedLowerBound = -1.0 / sqrt(3);
+  const double expectedUpperBound = 1.0 / sqrt(3);
+
+  DefaultMOEAD opt(
+        300,  // Max generations.
+        300,  // Max generations.
+        1.0,  // Crossover probability.
+        0.9, // Probability of sampling from neighbor.
+        20, // Neighborhood size.
+        20, // Perturbation index.
+        0.5, // Differential weight.
+        2, // Max childrens to replace parents.
+        1E-10, // epsilon.
+        lowerBound, // Lower bound.
+        upperBound // Upper bound.
+      );
+  typedef decltype(FON.objectiveA) ObjectiveTypeA;
+  typedef decltype(FON.objectiveB) ObjectiveTypeB;
+
+  arma::mat coords = FON.GetInitialPoint();
+  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = FON.GetObjectives();
+
+  opt.Optimize(objectives, coords);
+  arma::cube paretoSet = opt.ParetoSet();
+
+  bool allInRange = true;
+
+  for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+  {
+    const arma::mat solution = paretoSet.slice(solutionIdx);
+    double valX = arma::as_scalar(solution(0));
+    double valY = arma::as_scalar(solution(1));
+    double valZ = arma::as_scalar(solution(2));
+
+    if (!IsInBounds<double>(valX, expectedLowerBound, expectedUpperBound, 0.1) ||
+        !IsInBounds<double>(valY, expectedLowerBound, expectedUpperBound, 0.1) ||
+        !IsInBounds<double>(valZ, expectedLowerBound, expectedUpperBound, 0.1))
+    {
+      allInRange = false;
+      break;
+    }
+  }
+
+  REQUIRE(allInRange);
+}
+
+/**
+ * Optimize for the Schaffer N.1 function using NSGA-II optimizer.
+ * Tests for data of type float.
+ */
+TEST_CASE("MOEADSchafferN1FloatTest", "[MOEADTest]")
+{
+  SchafferFunctionN1<arma::fmat> SCH;
+  const double lowerBound = -1000;
+  const double upperBound = 1000;
+  const double expectedLowerBound = 0.0;
+  const double expectedUpperBound = 2.0;
+
+  DefaultMOEAD opt(
+        300, // Population size.
+        300,  // Max generations.
+        1.0,  // Crossover probability.
+        0.9, // Probability of sampling from neighbor.
+        20, // Neighborhood size.
+        20, // Perturbation index.
+        0.5, // Differential weight.
+        2, // Max childrens to replace parents.
+        1E-10, // epsilon.
+        lowerBound, // Lower bound.
+        upperBound // Upper bound.
+      );
+
+  typedef decltype(SCH.objectiveA) ObjectiveTypeA;
+  typedef decltype(SCH.objectiveB) ObjectiveTypeB;
+
+  // We allow a few trials in case of poor convergence.
+  bool success = false;
+  for (size_t trial = 0; trial < 3; ++trial)
+  {
+    arma::fmat coords = SCH.GetInitialPoint();
+    std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
+
+    opt.Optimize(objectives, coords);
+    arma::fcube paretoSet = arma::conv_to<arma::fcube>::from(opt.ParetoSet());
+
+    bool allInRange = true;
+
+    for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+    {
+      float val = arma::as_scalar(paretoSet.slice(solutionIdx));
+      if (!IsInBounds<float>(val, expectedLowerBound, expectedUpperBound, 0.1))
+      {
+        allInRange = false;
+        break;
+      }
+    }
+
+    if (allInRange)
+    {
+      success = true;
+      break;
+    }
+  }
+
+  REQUIRE(success == true);
+}
+
+/**
+ * Optimize for the Schaffer N.1 function using NSGA-II optimizer.
+ * Tests for data of type float.
+ */
+TEST_CASE("MOEADSchafferN1TestVectorFloatBounds", "[MOEADTest]")
+{
+  // This test can be a little flaky, so we try it a few times.
+  SchafferFunctionN1<arma::fmat> SCH;
+  const arma::vec lowerBound = {-1000};
+  const arma::vec upperBound = {1000};
+  const double expectedLowerBound = 0.0;
+  const double expectedUpperBound = 2.0;
+
+  DefaultMOEAD opt(
+        300, // Population size.
+        300,  // Max generations.
+        1.0,  // Crossover probability.
+        0.9, // Probability of sampling from neighbor.
+        20, // Neighborhood size.
+        20, // Perturbation index.
+        0.5, // Differential weight.
+        2, // Max childrens to replace parents.
+        1E-10, // epsilon.
+        lowerBound, // Lower bound.
+        upperBound // Upper bound.
+      );
+
+  typedef decltype(SCH.objectiveA) ObjectiveTypeA;
+  typedef decltype(SCH.objectiveB) ObjectiveTypeB;
+
+  bool success = false;
+  for (size_t trial = 0; trial < 3; ++trial)
+  {
+    arma::fmat coords = SCH.GetInitialPoint();
+    std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
+
+    opt.Optimize(objectives, coords);
+    arma::fcube paretoSet = arma::conv_to<arma::fcube>::from(opt.ParetoSet());
+
+    bool allInRange = true;
+
+    for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+    {
+      float val = arma::as_scalar(paretoSet.slice(solutionIdx));
+      if (!IsInBounds<float>(val, expectedLowerBound, expectedUpperBound, 0.1))
+      {
+        allInRange = false;
+        break;
+      }
+    }
+
+    if (allInRange)
+    {
+      success = true;
+      break;
+    }
+  }
+
+  REQUIRE(success == true);
+}
+
+/**
+ * Optimize for the Fonseca Fleming function using NSGA-II optimizer.
+ * Tests for data of type float.
+ */
+TEST_CASE("MOEADFonsecaFlemingFloatTest", "[MOEADTest]")
+{
+  FonsecaFlemingFunction<arma::fmat> FON;
+  const double lowerBound = -4;
+  const double upperBound = 4;
+  const float expectedLowerBound = -1.0 / sqrt(3);
+  const float expectedUpperBound = 1.0 / sqrt(3);
+
+  DefaultMOEAD opt(
+        300,  // Max generations.
+        300,  // Max generations.
+        1.0,  // Crossover probability.
+        0.9, // Probability of sampling from neighbor.
+        20, // Neighborhood size.
+        20, // Perturbation index.
+        0.5, // Differential weight.
+        2, // Max childrens to replace parents.
+        1E-10, // epsilon.
+        lowerBound, // Lower bound.
+        upperBound // Upper bound.
+      );
+  typedef decltype(FON.objectiveA) ObjectiveTypeA;
+  typedef decltype(FON.objectiveB) ObjectiveTypeB;
+
+  arma::fmat coords = FON.GetInitialPoint();
+  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = FON.GetObjectives();
+
+  opt.Optimize(objectives, coords);
+  arma::fcube paretoSet = arma::conv_to<arma::fcube>::from(opt.ParetoSet());
+
+  bool allInRange = true;
+
+  for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+  {
+    const arma::fmat solution = paretoSet.slice(solutionIdx);
+    float valX = arma::as_scalar(solution(0));
+    float valY = arma::as_scalar(solution(1));
+    float valZ = arma::as_scalar(solution(2));
+
+    if (!IsInBounds<float>(valX, expectedLowerBound, expectedUpperBound, 0.1) ||
+        !IsInBounds<float>(valY, expectedLowerBound, expectedUpperBound, 0.1) ||
+        !IsInBounds<float>(valZ, expectedLowerBound, expectedUpperBound, 0.1))
+    {
+      allInRange = false;
+      break;
+    }
+  }
+
+  REQUIRE(allInRange);
+}
+
+/**
+ * Optimize for the Fonseca Fleming function using NSGA-II optimizer.
+ * Tests for data of type float.
+ */
+TEST_CASE("MOEADFonsecaFlemingTestVectorFloatBounds", "[MOEADTest]")
+{
+  FonsecaFlemingFunction<arma::fmat> FON;
+  const arma::vec lowerBound = {-4, -4, -4};
+  const arma::vec upperBound = {4, 4, 4};
+  const float expectedLowerBound = -1.0 / sqrt(3);
+  const float expectedUpperBound = 1.0 / sqrt(3);
+
+  DefaultMOEAD opt(
+        300,  // Max generations.
+        300,  // Max generations.
+        1.0,  // Crossover probability.
+        0.9, // Probability of sampling from neighbor.
+        20, // Neighborhood size.
+        20, // Perturbation index.
+        0.5, // Differential weight.
+        2, // Max childrens to replace parents.
+        1E-10, // epsilon.
+        lowerBound, // Lower bound.
+        upperBound // Upper bound.
+      );
+  typedef decltype(FON.objectiveA) ObjectiveTypeA;
+  typedef decltype(FON.objectiveB) ObjectiveTypeB;
+
+  arma::fmat coords = FON.GetInitialPoint();
+  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = FON.GetObjectives();
+
+  opt.Optimize(objectives, coords);
+  arma::fcube paretoSet = arma::conv_to<arma::fcube>::from(opt.ParetoSet());
+
+  bool allInRange = true;
+
+  for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+  {
+    const arma::fmat solution = paretoSet.slice(solutionIdx);
+    float valX = arma::as_scalar(solution(0));
+    float valY = arma::as_scalar(solution(1));
+    float valZ = arma::as_scalar(solution(2));
+
+    if (!IsInBounds<float>(valX, expectedLowerBound, expectedUpperBound, 0.1) ||
+        !IsInBounds<float>(valY, expectedLowerBound, expectedUpperBound, 0.1) ||
+        !IsInBounds<float>(valZ, expectedLowerBound, expectedUpperBound, 0.1))
+    {
+      allInRange = false;
+      break;
+    }
+  }
+
+  REQUIRE(allInRange);
+}
+
+/**
+ * Test against the first problem of ZDT Test Suite.  ZDT-1 is a 30 
+ * variable-2 objective problem with a convex Pareto Front. 
+ *
+ * NOTE: For the sake of runtime, only ZDT-1 is tested against the
+ * algorithm. Others have been tested separately.
+ */
+TEST_CASE("MOEADZDTONETest", "[MOEADTest]")
+{
+  //! Parameters taken from original ZDT Paper.
+  ZDT1<> ZDT_ONE(100);
+  const double lowerBound = 0;
+  const double upperBound = 1;
+
+  DefaultMOEAD opt(
+      300, // Population size.
+      150,  // Max generations.
+      1.0,  // Crossover probability.
+      0.9, // Probability of sampling from neighbor.
+      20, // Neighborhood size.
+      20, // Perturbation index.
+      0.5, // Differential weight.
+      2, // Max childrens to replace parents.
+      1E-10, // epsilon.
+      lowerBound, // Lower bound.
+      upperBound // Upper bound.
+    );
+
+  typedef decltype(ZDT_ONE.objectiveF1) ObjectiveTypeA;
+  typedef decltype(ZDT_ONE.objectiveF2) ObjectiveTypeB;
+
+  arma::mat coords = ZDT_ONE.GetInitialPoint();
+  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = ZDT_ONE.GetObjectives();
+
+  opt.Optimize(objectives, coords);
+
+  //! Refer the ZDT_ONE implementation for g objective implementation.
+  //! The optimal g value is taken from the docs of ZDT_ONE.
+  size_t numVariables = coords.size();
+  double sum = arma::accu(coords(arma::span(1, numVariables - 1), 0));
+  double g = 1. + 9. * sum / (static_cast<double>(numVariables - 1));
+  REQUIRE(g == Approx(1.0).margin(0.99));
+}
+
+/**
+ * Check if the final population lies in the optimal region in variable space.
+ *
+ * @param paretoSet The final population in variable space.
+ */
+bool VariableBoundsCheck(const arma::cube& paretoSet)
+{
+  bool inBounds = true;
+  const arma::mat regions{
+    {0.0, 0.182228780, 0.4093136748,
+      0.6183967944, 0.8233317983},
+    {0.0830015349, 0.2577623634, 0.4538821041,
+      0.6525117038, 0.8518328654}
+  };
+
+  for (size_t pointIdx = 0; pointIdx < paretoSet.n_slices; ++pointIdx)
+  {
+    const arma::mat& point = paretoSet.slice(pointIdx);
+    const double firstVariable = point(0, 0);
+
+    const bool notInRegion0 = !IsInBounds<double>(firstVariable, regions(0, 0), regions(1, 0), 1e-2);
+    const bool notInRegion1 = !IsInBounds<double>(firstVariable, regions(0, 1), regions(1, 1), 1e-2);
+    const bool notInRegion2 = !IsInBounds<double>(firstVariable, regions(0, 2), regions(1, 2), 1e-2);
+    const bool notInRegion3 = !IsInBounds<double>(firstVariable, regions(0, 3), regions(1, 3), 1e-2);
+    const bool notInRegion4 = !IsInBounds<double>(firstVariable, regions(0, 4), regions(1, 4), 1e-2);
+
+    if (notInRegion0 && notInRegion1 && notInRegion2 && notInRegion3 && notInRegion4)
+    {
+      inBounds = false;
+      break;
+    }
+  }
+
+  return inBounds;
+}
+
+/**
+ * Test DirichletMOEAD against the third problem of ZDT Test Suite. ZDT-3 is a 30 
+ * variable-2 objective problem with disconnected Pareto Fronts. 
+ */
+TEST_CASE("MOEADDIRICHLETZDT3Test", "[MOEADTest]")
+{
+  //! Parameters taken from original ZDT Paper.
+  ZDT3<> ZDT_THREE(300);
+  const double lowerBound = 0;
+  const double upperBound = 1;
+
+  DirichletMOEAD opt(
+      300, // Population size.
+      300,  // Max generations.
+      1.0,  // Crossover probability.
+      0.9, // Probability of sampling from neighbor.
+      20, // Neighborhood size.
+      20, // Perturbation index.
+      0.5, // Differential weight.
+      2, // Max childrens to replace parents.
+      1E-10, // epsilon.
+      lowerBound, // Lower bound.
+      upperBound // Upper bound.
+    );
+
+  typedef decltype(ZDT_THREE.objectiveF1) ObjectiveTypeA;
+  typedef decltype(ZDT_THREE.objectiveF2) ObjectiveTypeB;
+
+  arma::mat coords = ZDT_THREE.GetInitialPoint();
+  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = ZDT_THREE.GetObjectives();
+
+  opt.Optimize(objectives, coords);
+
+  const arma::cube& finalPopulation = opt.ParetoSet();
+  REQUIRE(VariableBoundsCheck(finalPopulation));
+}

tests/nesterov_momentum_sgd_test.cpp

@@ -29,7 +29,7 @@ TEST_CASE("NesterovMomentumSGDSpeedUpTestFunction", "[NesterovMomentumSGDTest]")
   arma::mat coordinates = f.GetInitialPoint();
   double result = s.Optimize(f, coordinates);
 
-  REQUIRE(result == Approx(-1.0).epsilon(0.0025));
+  REQUIRE(result == Approx(-1.0).margin(0.01));
   REQUIRE(coordinates(0) == Approx(0.0).margin(3e-3));
   REQUIRE(coordinates(1) == Approx(0.0).margin(1e-6));
   REQUIRE(coordinates(2) == Approx(0.0).margin(1e-6));
@@ -54,7 +54,7 @@ TEST_CASE("NesterovMomentumSGDGeneralizedRosenbrockTest", "[NesterovMomentumSGDT
 
     REQUIRE(result == Approx(0.0).margin(1e-4));
     for (size_t j = 0; j < i; ++j)
-      REQUIRE(coordinates(j) == Approx(1.0).epsilon(1e-5));
+      REQUIRE(coordinates(j) == Approx(1.0).epsilon(0.003));
   }
 }
 
@@ -76,15 +76,15 @@ TEST_CASE("NesterovMomentumSGDGeneralizedRosenbrockFMatTest",
     size_t trial = 0;
     float result = std::numeric_limits<float>::max();
     arma::fmat coordinates;
-    while (trial++ < 5 && result > 0.1)
+    while (trial++ < 8 && result > 0.1)
     {
       coordinates = f.GetInitialPoint<arma::fmat>();
       result = s.Optimize(f, coordinates);
     }
 
-    REQUIRE(result == Approx(0.0).margin(1e-2));
+    REQUIRE(result == Approx(0.0).margin(0.02));
     for (size_t j = 0; j < i; ++j)
-      REQUIRE(coordinates(j) == Approx(1.0).epsilon(1e-2));
+      REQUIRE(coordinates(j) == Approx(1.0).margin(0.05));
   }
 }
 
@@ -108,6 +108,6 @@ TEST_CASE("NesterovMomentumSGDGeneralizedRosenbrockSpMatTest",
 
     REQUIRE(result == Approx(0.0).margin(1e-4));
     for (size_t j = 0; j < i; ++j)
-      REQUIRE(coordinates(j) == Approx(1.0).epsilon(1e-5));
+      REQUIRE(coordinates(j) == Approx(1.0).epsilon(0.003));
   }
 }

tests/nsga2_test.cpp

@@ -1,6 +1,7 @@
 /**
  * @file nsga2_test.cpp
  * @author Sayan Goswami
+ * @author Nanubala Gnana Sai
  *
  * ensmallen is free software; you may redistribute it and/or modify it under
  * the terms of the 3-clause BSD license.  You should have received a copy of
@@ -22,88 +23,120 @@ using namespace std;
  * @param value The value being checked.
  * @param low The lower bound.
  * @param high The upper bound.
+ * @tparam The type of elements in the population set.
  * @return true if value lies in the range [low, high].
  * @return false if value does not lie in the range [low, high].
  */
-bool IsInBounds(const double& value, const double& low, const double& high)
+template<typename ElemType>
+bool IsInBounds(const ElemType& value, const ElemType& low, const ElemType& high)
 {
-  return !(value < low) && !(high < value);
+  ElemType roundoff = 0.1;
+  return !(value < (low - roundoff)) && !((high + roundoff) < value);
 }
 
 /**
  * Optimize for the Schaffer N.1 function using NSGA-II optimizer.
+ * Tests for data of type double.
  */
-TEST_CASE("NSGA2SchafferN1Test", "[NSGA2Test]")
+TEST_CASE("NSGA2SchafferN1DoubleTest", "[NSGA2Test]")
 {
   SchafferFunctionN1<arma::mat> SCH;
   const double lowerBound = -1000;
   const double upperBound = 1000;
+  const double expectedLowerBound = 0.0;
+  const double expectedUpperBound = 2.0;
 
-  NSGA2 opt(20, 5000, 0.5, 0.5, 1e-3, 1e-6, lowerBound, upperBound);
+  NSGA2 opt(20, 300, 0.5, 0.5, 1e-3, 1e-6, lowerBound, upperBound);
 
   typedef decltype(SCH.objectiveA) ObjectiveTypeA;
   typedef decltype(SCH.objectiveB) ObjectiveTypeB;
 
-  arma::mat coords = SCH.GetInitialPoint();
-  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
-
-  opt.Optimize(objectives, coords);
-  std::vector<arma::mat> bestFront = opt.Front();
-
-  bool allInRange = true;
-
-  for (arma::mat solution: bestFront)
+  // We allow a few trials in case of poor convergence.
+  bool success = false;
+  for (size_t trial = 0; trial < 3; ++trial)
   {
-    double val = arma::as_scalar(solution);
+    arma::mat coords = SCH.GetInitialPoint();
+    std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
 
-    if (val < 0.0 || val > 2.0)
+    opt.Optimize(objectives, coords);
+    arma::cube paretoSet = opt.ParetoSet();
+
+    bool allInRange = true;
+
+    for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
     {
-      allInRange = false;
+      double val = arma::as_scalar(paretoSet.slice(solutionIdx));
+      if (!IsInBounds<double>(val, expectedLowerBound, expectedUpperBound))
+      {
+        allInRange = false;
+        break;
+      }
+    }
+
+    if (allInRange)
+    {
+      success = true;
       break;
     }
   }
-  REQUIRE(allInRange);
+
+  REQUIRE(success == true);
 }
 
 /**
  * Optimize for the Schaffer N.1 function using NSGA-II optimizer.
+ * Tests for data of type double.
  */
-TEST_CASE("NSGA2SchafferN1TestVectorBounds", "[NSGA2Test]")
+TEST_CASE("NSGA2SchafferN1TestVectorDoubleBounds", "[NSGA2Test]")
 {
+  // This test can be a little flaky, so we try it a few times.
   SchafferFunctionN1<arma::mat> SCH;
   const arma::vec lowerBound = {-1000};
   const arma::vec upperBound = {1000};
+  const double expectedLowerBound = 0.0;
+  const double expectedUpperBound = 2.0;
 
-  NSGA2 opt(20, 5000, 0.5, 0.5, 1e-3, 1e-6, lowerBound, upperBound);
+  NSGA2 opt(20, 300, 0.5, 0.5, 1e-3, 1e-6, lowerBound, upperBound);
 
   typedef decltype(SCH.objectiveA) ObjectiveTypeA;
   typedef decltype(SCH.objectiveB) ObjectiveTypeB;
 
-  arma::mat coords = SCH.GetInitialPoint();
-  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
-
-  opt.Optimize(objectives, coords);
-  std::vector<arma::mat> bestFront = opt.Front();
-
-  bool allInRange = true;
-
-  for (arma::mat solution: bestFront)
+  bool success = false;
+  for (size_t trial = 0; trial < 3; ++trial)
   {
-    double val = arma::as_scalar(solution);
+    arma::mat coords = SCH.GetInitialPoint();
+    std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
 
-    if (val < 0.0 || val > 2.0)
+    opt.Optimize(objectives, coords);
+    arma::cube paretoSet = opt.ParetoSet();
+
+    bool allInRange = true;
+
+    for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
     {
-      allInRange = false;
+      double val = arma::as_scalar(paretoSet.slice(solutionIdx));
+      if (!IsInBounds<double>(val, expectedLowerBound, expectedUpperBound))
+      {
+        allInRange = false;
+        break;
+      }
+    }
+
+    if (allInRange)
+    {
+      success = true;
       break;
     }
   }
-  REQUIRE(allInRange);
+
+  REQUIRE(success == true);
 }
 
 /**
  * Optimize for the Fonseca Fleming function using NSGA-II optimizer.
+ * Tests for data of type double.
  */
-TEST_CASE("NSGA2FonsecaFlemingTest", "[NSGA2Test]")
+TEST_CASE("NSGA2FonsecaFlemingDoubleTest", "[NSGA2Test]")
 {
   FonsecaFlemingFunction<arma::mat> FON;
   const double lowerBound = -4;
@@ -113,7 +146,7 @@ TEST_CASE("NSGA2FonsecaFlemingTest", "[NSGA2Test]")
   const double expectedLowerBound = -1.0 / sqrt(3);
   const double expectedUpperBound = 1.0 / sqrt(3);
 
-  NSGA2 opt(20, 4000, 0.6, 0.3, strength, tolerance, lowerBound, upperBound);
+  NSGA2 opt(20, 300, 0.6, 0.3, strength, tolerance, lowerBound, upperBound);
 
   typedef decltype(FON.objectiveA) ObjectiveTypeA;
   typedef decltype(FON.objectiveB) ObjectiveTypeB;
@@ -122,32 +155,34 @@ TEST_CASE("NSGA2FonsecaFlemingTest", "[NSGA2Test]")
   std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = FON.GetObjectives();
 
   opt.Optimize(objectives, coords);
-  std::vector<arma::mat> bestFront = opt.Front();
+  arma::cube paretoSet = opt.ParetoSet();
 
   bool allInRange = true;
 
-  for (size_t i = 0; i < bestFront.size(); i++)
+  for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
   {
-    const arma::mat solution = bestFront[i];
+    const arma::mat solution = paretoSet.slice(solutionIdx);
     double valX = arma::as_scalar(solution(0));
     double valY = arma::as_scalar(solution(1));
     double valZ = arma::as_scalar(solution(2));
 
-    if (!IsInBounds(valX, expectedLowerBound, expectedUpperBound) ||
-        !IsInBounds(valY, expectedLowerBound, expectedUpperBound) ||
-        !IsInBounds(valZ, expectedLowerBound, expectedUpperBound))
+    if (!IsInBounds<double>(valX, expectedLowerBound, expectedUpperBound) ||
+        !IsInBounds<double>(valY, expectedLowerBound, expectedUpperBound) ||
+        !IsInBounds<double>(valZ, expectedLowerBound, expectedUpperBound))
     {
       allInRange = false;
       break;
     }
   }
+
   REQUIRE(allInRange);
 }
 
 /**
  * Optimize for the Fonseca Fleming function using NSGA-II optimizer.
+ * Tests for data of type double.
  */
-TEST_CASE("NSGA2FonsecaFlemingTestVectorBounds", "[NSGA2Test]")
+TEST_CASE("NSGA2FonsecaFlemingTestVectorDoubleBounds", "[NSGA2Test]")
 {
   FonsecaFlemingFunction<arma::mat> FON;
   const arma::vec lowerBound = {-4, -4, -4};
@@ -157,7 +192,7 @@ TEST_CASE("NSGA2FonsecaFlemingTestVectorBounds", "[NSGA2Test]")
   const double expectedLowerBound = -1.0 / sqrt(3);
   const double expectedUpperBound = 1.0 / sqrt(3);
 
-  NSGA2 opt(20, 4000, 0.6, 0.3, strength, tolerance, lowerBound, upperBound);
+  NSGA2 opt(20, 300, 0.6, 0.3, strength, tolerance, lowerBound, upperBound);
 
   typedef decltype(FON.objectiveA) ObjectiveTypeA;
   typedef decltype(FON.objectiveB) ObjectiveTypeB;
@@ -166,24 +201,285 @@ TEST_CASE("NSGA2FonsecaFlemingTestVectorBounds", "[NSGA2Test]")
   std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = FON.GetObjectives();
 
   opt.Optimize(objectives, coords);
-  std::vector<arma::mat> bestFront = opt.Front();
+  arma::cube paretoSet = opt.ParetoSet();
 
   bool allInRange = true;
 
-  for (size_t i = 0; i < bestFront.size(); i++)
+  for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
   {
-    const arma::mat solution = bestFront[i];
+    const arma::mat solution = paretoSet.slice(solutionIdx);
     double valX = arma::as_scalar(solution(0));
     double valY = arma::as_scalar(solution(1));
     double valZ = arma::as_scalar(solution(2));
 
-    if (!IsInBounds(valX, expectedLowerBound, expectedUpperBound) ||
-        !IsInBounds(valY, expectedLowerBound, expectedUpperBound) ||
-        !IsInBounds(valZ, expectedLowerBound, expectedUpperBound))
+    if (!IsInBounds<double>(valX, expectedLowerBound, expectedUpperBound) ||
+        !IsInBounds<double>(valY, expectedLowerBound, expectedUpperBound) ||
+        !IsInBounds<double>(valZ, expectedLowerBound, expectedUpperBound))
     {
       allInRange = false;
       break;
     }
   }
+
   REQUIRE(allInRange);
 }
+
+/**
+ * Optimize for the Schaffer N.1 function using NSGA-II optimizer.
+ * Tests for data of type float.
+ */
+TEST_CASE("NSGA2SchafferN1FloatTest", "[NSGA2Test]")
+{
+  SchafferFunctionN1<arma::fmat> SCH;
+  const double lowerBound = -1000;
+  const double upperBound = 1000;
+  const double expectedLowerBound = 0.0;
+  const double expectedUpperBound = 2.0;
+
+  NSGA2 opt(20, 300, 0.5, 0.5, 1e-3, 1e-6, lowerBound, upperBound);
+
+  typedef decltype(SCH.objectiveA) ObjectiveTypeA;
+  typedef decltype(SCH.objectiveB) ObjectiveTypeB;
+
+  // We allow a few trials in case of poor convergence.
+  bool success = false;
+  for (size_t trial = 0; trial < 3; ++trial)
+  {
+    arma::fmat coords = SCH.GetInitialPoint();
+    std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
+
+    opt.Optimize(objectives, coords);
+    arma::fcube paretoSet = arma::conv_to<arma::fcube>::from(opt.ParetoSet());
+
+    bool allInRange = true;
+
+    for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+    {
+      float val = arma::as_scalar(paretoSet.slice(solutionIdx));
+      if (!IsInBounds<float>(val, expectedLowerBound, expectedUpperBound))
+      {
+        allInRange = false;
+        break;
+      }
+    }
+
+    if (allInRange)
+    {
+      success = true;
+      break;
+    }
+  }
+
+  REQUIRE(success == true);
+}
+
+/**
+ * Optimize for the Schaffer N.1 function using NSGA-II optimizer.
+ * Tests for data of type float.
+ */
+TEST_CASE("NSGA2SchafferN1TestVectorFloatBounds", "[NSGA2Test]")
+{
+  // This test can be a little flaky, so we try it a few times.
+  SchafferFunctionN1<arma::fmat> SCH;
+  const arma::vec lowerBound = {-1000};
+  const arma::vec upperBound = {1000};
+  const double expectedLowerBound = 0.0;
+  const double expectedUpperBound = 2.0;
+
+  NSGA2 opt(20, 300, 0.5, 0.5, 1e-3, 1e-6, lowerBound, upperBound);
+
+  typedef decltype(SCH.objectiveA) ObjectiveTypeA;
+  typedef decltype(SCH.objectiveB) ObjectiveTypeB;
+
+  bool success = false;
+  for (size_t trial = 0; trial < 3; ++trial)
+  {
+    arma::fmat coords = SCH.GetInitialPoint();
+    std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
+
+    opt.Optimize(objectives, coords);
+    arma::fcube paretoSet = arma::conv_to<arma::fcube>::from(opt.ParetoSet());
+
+    bool allInRange = true;
+
+    for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+    {
+      float val = arma::as_scalar(paretoSet.slice(solutionIdx));
+      if (!IsInBounds<float>(val, expectedLowerBound, expectedUpperBound))
+      {
+        allInRange = false;
+        break;
+      }
+    }
+
+    if (allInRange)
+    {
+      success = true;
+      break;
+    }
+  }
+
+  REQUIRE(success == true);
+}
+
+/**
+ * Optimize for the Fonseca Fleming function using NSGA-II optimizer.
+ * Tests for data of type float.
+ */
+TEST_CASE("NSGA2FonsecaFlemingFloatTest", "[NSGA2Test]")
+{
+  FonsecaFlemingFunction<arma::fmat> FON;
+  const double lowerBound = -4;
+  const double upperBound = 4;
+  const double tolerance = 1e-6;
+  const double strength = 1e-4;
+  const float expectedLowerBound = -1.0 / sqrt(3);
+  const float expectedUpperBound = 1.0 / sqrt(3);
+
+  NSGA2 opt(20, 300, 0.6, 0.3, strength, tolerance, lowerBound, upperBound);
+
+  typedef decltype(FON.objectiveA) ObjectiveTypeA;
+  typedef decltype(FON.objectiveB) ObjectiveTypeB;
+
+  arma::fmat coords = FON.GetInitialPoint();
+  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = FON.GetObjectives();
+
+  opt.Optimize(objectives, coords);
+  arma::fcube paretoSet = arma::conv_to<arma::fcube>::from(opt.ParetoSet());
+
+  bool allInRange = true;
+
+  for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+  {
+    const arma::fmat solution = paretoSet.slice(solutionIdx);
+    float valX = arma::as_scalar(solution(0));
+    float valY = arma::as_scalar(solution(1));
+    float valZ = arma::as_scalar(solution(2));
+
+    if (!IsInBounds<float>(valX, expectedLowerBound, expectedUpperBound) ||
+        !IsInBounds<float>(valY, expectedLowerBound, expectedUpperBound) ||
+        !IsInBounds<float>(valZ, expectedLowerBound, expectedUpperBound))
+    {
+      allInRange = false;
+      break;
+    }
+  }
+
+  REQUIRE(allInRange);
+}
+
+/**
+ * Optimize for the Fonseca Fleming function using NSGA-II optimizer.
+ * Tests for data of type float.
+ */
+TEST_CASE("NSGA2FonsecaFlemingTestVectorFloatBounds", "[NSGA2Test]")
+{
+  FonsecaFlemingFunction<arma::fmat> FON;
+  const arma::vec lowerBound = {-4, -4, -4};
+  const arma::vec upperBound = {4, 4, 4};
+  const double tolerance = 1e-6;
+  const double strength = 1e-4;
+  const float expectedLowerBound = -1.0 / sqrt(3);
+  const float expectedUpperBound = 1.0 / sqrt(3);
+
+  NSGA2 opt(20, 300, 0.6, 0.3, strength, tolerance, lowerBound, upperBound);
+
+  typedef decltype(FON.objectiveA) ObjectiveTypeA;
+  typedef decltype(FON.objectiveB) ObjectiveTypeB;
+
+  arma::fmat coords = FON.GetInitialPoint();
+  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = FON.GetObjectives();
+
+  opt.Optimize(objectives, coords);
+  arma::fcube paretoSet = arma::conv_to<arma::fcube>::from(opt.ParetoSet());
+
+  bool allInRange = true;
+
+  for (size_t solutionIdx = 0; solutionIdx < paretoSet.n_slices; ++solutionIdx)
+  {
+    const arma::fmat solution = paretoSet.slice(solutionIdx);
+    float valX = arma::as_scalar(solution(0));
+    float valY = arma::as_scalar(solution(1));
+    float valZ = arma::as_scalar(solution(2));
+
+    if (!IsInBounds<float>(valX, expectedLowerBound, expectedUpperBound) ||
+        !IsInBounds<float>(valY, expectedLowerBound, expectedUpperBound) ||
+        !IsInBounds<float>(valZ, expectedLowerBound, expectedUpperBound))
+    {
+      allInRange = false;
+      break;
+    }
+  }
+
+  REQUIRE(allInRange);
+}
+
+/**
+ * Test against the first problem of ZDT Test Suite.  ZDT-1 is a 30 
+ * variable-2 objective problem with a convex Pareto Front.
+ * 
+ * NOTE: For the sake of runtime, only ZDT-1 is tested against the
+ * algorithm. Others have been tested separately.
+ */
+TEST_CASE("NSGA2ZDTONETest", "[NSGA2Test]")
+{
+  //! Parameters taken from original ZDT Paper.
+  ZDT1<> ZDT_ONE(100);
+  const double lowerBound = 0;
+  const double upperBound = 1;
+  const double tolerance = 1e-6;
+  const double mutationRate = 1e-2;
+  const double crossoverRate = 0.8;
+  const double strength = 1e-4;
+
+  NSGA2 opt(100, 250, crossoverRate, mutationRate, strength,
+    tolerance, lowerBound, upperBound);
+
+  typedef decltype(ZDT_ONE.objectiveF1) ObjectiveTypeA;
+  typedef decltype(ZDT_ONE.objectiveF2) ObjectiveTypeB;
+
+  arma::mat coords = ZDT_ONE.GetInitialPoint();
+  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = ZDT_ONE.GetObjectives();
+
+  opt.Optimize(objectives, coords);
+
+  //! Refer the ZDT_ONE implementation for g objective implementation.
+  //! The optimal g value is taken from the docs of ZDT_ONE.
+  size_t numVariables = coords.size();
+  double sum = arma::accu(coords(arma::span(1, numVariables - 1), 0));
+  double g = 1. + 9. * sum / (static_cast<double>(numVariables - 1));
+
+  REQUIRE(g == Approx(1.0).margin(0.99));
+}
+
+/**
+ * Ensure that the reverse-compatible Front() function works.
+ *
+ * This test can be removed when Front() is removed, in ensmallen 3.x.
+ */
+TEST_CASE("NSGA2FrontTest", "[NSGA2Test]")
+{
+  SchafferFunctionN1<arma::mat> SCH;
+  const double lowerBound = -1000;
+  const double upperBound = 1000;
+
+  NSGA2 opt(20, 300, 0.5, 0.5, 1e-3, 1e-6, lowerBound, upperBound);
+
+  typedef decltype(SCH.objectiveA) ObjectiveTypeA;
+  typedef decltype(SCH.objectiveB) ObjectiveTypeB;
+
+  arma::mat coords = SCH.GetInitialPoint();
+  std::tuple<ObjectiveTypeA, ObjectiveTypeB> objectives = SCH.GetObjectives();
+
+  opt.Optimize(objectives, coords);
+  arma::cube paretoFront = opt.ParetoFront();
+
+  std::vector<arma::mat> rcFront = opt.Front();
+
+  REQUIRE(paretoFront.n_slices == rcFront.size());
+  for (size_t i = 0; i < paretoFront.n_slices; ++i)
+  {
+    arma::mat paretoM = paretoFront.slice(i);
+    CheckMatrices(paretoM, rcFront[i]);
+  }
+}

tests/parallel_sgd_test.cpp

@@ -30,8 +30,6 @@ using namespace ens::test;
  */
 TEST_CASE("SimpleParallelSGDTest", "[ParallelSGDTest]")
 {
-  SparseTestFunction f;
-
   ConstantStep decayPolicy(0.4);
 
   // The batch size for this test should be chosen according to the threads
@@ -40,6 +38,7 @@ TEST_CASE("SimpleParallelSGDTest", "[ParallelSGDTest]")
 
   size_t threadsAvailable = omp_get_max_threads();
 
+  SparseTestFunction f;
   for (size_t i = threadsAvailable; i > 0; --i)
   {
     omp_set_num_threads(i);
@@ -47,19 +46,7 @@ TEST_CASE("SimpleParallelSGDTest", "[ParallelSGDTest]")
     size_t batchSize = std::ceil((float) f.NumFunctions() / i);
 
     ParallelSGD<ConstantStep> s(10000, batchSize, 1e-5, true, decayPolicy);
-
-    arma::mat coordinates = f.GetInitialPoint<arma::mat>();
-    double result = s.Optimize(f, coordinates);
-
-    // The final value of the objective function should be close to the optimal
-    // value, that is the sum of values at the vertices of the parabolas.
-    REQUIRE(result == Approx(123.75).epsilon(0.0001));
-
-    // The co-ordinates should be the vertices of the parabolas.
-    REQUIRE(coordinates(0) == Approx(2.0).epsilon(0.0002));
-    REQUIRE(coordinates(1) == Approx(1.0).epsilon(0.0002));
-    REQUIRE(coordinates(2) == Approx(1.5).epsilon(0.0002));
-    REQUIRE(coordinates(3) == Approx(4.0).epsilon(0.0002));
+    FunctionTest<SparseTestFunction>(s, 0.01, 0.001);
   }
 }
 

tests/pso_test.cpp

@@ -66,14 +66,28 @@ TEST_CASE("LBestPSORosenbrockTest","[PSOTest]")
   lowerBound.fill(50);
   upperBound.fill(60);
 
-  LBestPSO s(250, lowerBound, upperBound, 3000, 600, 1e-30, 2.05, 2.05);
-  arma::vec coordinates = f.GetInitialPoint();
+  // We allow a few trials.
+  for (size_t trial = 0; trial < 3; ++trial)
+  {
+    LBestPSO s(250, lowerBound, upperBound, 3000, 600, 1e-30, 2.05, 2.05);
+    arma::vec coordinates = f.GetInitialPoint();
 
-  const double result = s.Optimize(f, coordinates);
+    const double result = s.Optimize(f, coordinates);
 
-  REQUIRE(result == Approx(0.0).margin(1e-3));
-  REQUIRE(coordinates(0) == Approx(1.0).epsilon(1e-2));
-  REQUIRE(coordinates(1) == Approx(1.0).epsilon(1e-2));
+    if (trial != 2)
+    {
+      if (result != Approx(0.0).margin(0.03))
+        continue;
+      if (coordinates(0) != Approx(1.0).margin(0.02))
+        continue;
+      if (coordinates(1) != Approx(1.0).margin(0.02))
+        continue;
+    }
+
+    REQUIRE(result == Approx(0.0).margin(0.03));
+    REQUIRE(coordinates(0) == Approx(1.0).margin(0.02));
+    REQUIRE(coordinates(1) == Approx(1.0).margin(0.02));
+  }
 }
 
 /**
@@ -89,14 +103,28 @@ TEST_CASE("LBestPSORosenbrockFMatTest","[PSOTest]")
   lowerBound.fill(50);
   upperBound.fill(60);
 
-  LBestPSO s(250, lowerBound, upperBound, 5000, 600, 1e-30, 2.05, 2.05);
-  arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
+  // We allow a few trials.
+  for (size_t trial = 0; trial < 5; ++trial)
+  {
+    LBestPSO s(250, lowerBound, upperBound, 5000, 600, 1e-30, 2.05, 2.05);
+    arma::fmat coordinates = f.GetInitialPoint<arma::fmat>();
 
-  const double result = s.Optimize(f, coordinates);
+    const double result = s.Optimize(f, coordinates);
 
-  REQUIRE(result == Approx(0.0).margin(1e-3));
-  REQUIRE(coordinates(0) == Approx(1.0).epsilon(1e-2));
-  REQUIRE(coordinates(1) == Approx(1.0).epsilon(1e-2));
+    if (trial != 4)
+    {
+      if (result != Approx(0.0).margin(0.03))
+        continue;
+      if (coordinates(0) != Approx(1.0).margin(0.03))
+        continue;
+      if (coordinates(1) != Approx(1.0).margin(0.03))
+        continue;
+    }
+
+    REQUIRE(result == Approx(0.0).margin(0.03));
+    REQUIRE(coordinates(0) == Approx(1.0).margin(0.03));
+    REQUIRE(coordinates(1) == Approx(1.0).margin(0.03));
+  }
 }
 
 /**
@@ -111,14 +139,28 @@ TEST_CASE("LBestPSORosenbrockDoubleTest","[PSOTest]")
   double lowerBound = 50;
   double upperBound = 60;
 
-  LBestPSO s(250, lowerBound, upperBound, 5000, 400, 1e-30, 2.05, 2.05);
-  arma::vec coordinates = f.GetInitialPoint();
+  // We allow a few trials.
+  for (size_t trial = 0; trial < 3; ++trial)
+  {
+    LBestPSO s(250, lowerBound, upperBound, 5000, 400, 1e-30, 2.05, 2.05);
+    arma::vec coordinates = f.GetInitialPoint();
 
-  const double result = s.Optimize(f, coordinates);
+    const double result = s.Optimize(f, coordinates);
 
-  REQUIRE(result == Approx(0.0).margin(1e-3));
-  REQUIRE(coordinates(0) == Approx(1.0).epsilon(1e-3));
-  REQUIRE(coordinates(1) == Approx(1.0).epsilon(1e-3));
+    if (trial != 2)
+    {
+      if (result != Approx(0.0).margin(1e-3))
+        continue;
+      if (coordinates(0) != Approx(1.0).epsilon(1e-2))
+        continue;
+      if (coordinates(1) != Approx(1.0).epsilon(1e-2))
+        continue;
+    }
+
+    REQUIRE(result == Approx(0.0).margin(0.005));
+    REQUIRE(coordinates(0) == Approx(1.0).margin(0.005));
+    REQUIRE(coordinates(1) == Approx(1.0).margin(0.005));
+  }
 }
 
 /**
@@ -134,13 +176,29 @@ TEST_CASE("LBestPSOCrossInTrayFunctionTest", "[PSOTest]")
   lowerBound.fill(-1);
   upperBound.fill(1);
 
-  LBestPSO s(500, lowerBound, upperBound, 6000, 400, 1e-30, 2.05, 2.05);
-  arma::mat coordinates = arma::mat("10; 10");
-  const double result = s.Optimize(f, coordinates);
+  // We allow many trials---sometimes this can have trouble converging.
+  for (size_t trial = 0; trial < 15; ++trial)
+  {
+    LBestPSO s(500, lowerBound, upperBound, 6000, 400, 1e-30, 2.05, 2.05);
+    arma::mat coordinates = arma::mat("10; 10");
+    const double result = s.Optimize(f, coordinates);
 
-  REQUIRE(result == Approx(-2.06261).margin(0.01));
-  REQUIRE(abs(coordinates(0)) == Approx(1.34941).margin(0.01));
-  REQUIRE(abs(coordinates(1)) == Approx(1.34941).margin(0.01));
+    if (trial != 14)
+    {
+      if (std::isinf(result) || std::isnan(result))
+        continue;
+      if (result != Approx(-2.06261).margin(0.01))
+        continue;
+      if (abs(coordinates(0)) != Approx(1.34941).margin(0.01))
+        continue;
+      if (abs(coordinates(1)) != Approx(1.34941).margin(0.01))
+        continue;
+    }
+
+    REQUIRE(result == Approx(-2.06261).margin(0.01));
+    REQUIRE(abs(coordinates(0)) == Approx(1.34941).margin(0.01));
+    REQUIRE(abs(coordinates(1)) == Approx(1.34941).margin(0.01));
+  }
 }
 
 /**
@@ -201,13 +259,25 @@ TEST_CASE("LBestPSOGoldsteinPriceFunctionTest", "[PSOTest]")
   lowerBound.fill(1.6);
   upperBound.fill(2);
 
-  LBestPSO s(64, lowerBound, upperBound);
+  // Allow a few trials in case of failure.
+  for (size_t trial = 0; trial < 10; ++trial)
+  {
+    LBestPSO s(64, lowerBound, upperBound);
 
-  arma::mat coordinates = arma::mat("1; 0");
-  s.Optimize(f, coordinates);
+    arma::mat coordinates = arma::mat("1; 0");
+    s.Optimize(f, coordinates);
 
-  REQUIRE(coordinates(0) == Approx(0).margin(0.01));
-  REQUIRE(coordinates(1) == Approx(-1).margin(0.01));
+    if (trial != 9)
+    {
+      if (coordinates(0) != Approx(0).margin(0.01))
+        continue;
+      if (coordinates(1) != Approx(-1).margin(0.01))
+        continue;
+    }
+
+    REQUIRE(coordinates(0) == Approx(0).margin(0.01));
+    REQUIRE(coordinates(1) == Approx(-1).margin(0.01));
+  }
 }
 
 /**