Merge branch 'master' into gan-callbacks

COPYRIGHT.txt

@@ -117,6 +117,7 @@ Copyright:
   Copyright 2019, Yashwant Singh Parihar <yashwantsingh.sngh@gmail.com>
   Copyright 2019, Heet Sankesara <heetsankesara3@gmail.com>
   Copyright 2019, Jeffin Sam <sam.jeffin@gmail.com>
+  Copyright 2019, Vikas S Shetty <shettyvikas209@gmail.com>
 
 License: BSD-3-clause
   All rights reserved.

HISTORY.md

@@ -1,5 +1,15 @@
 ### mlpack ?.?.?
 ###### ????-??-??
+  * Add Model() to the FFN class to access individual layers (#2043).
+
+  * Update documentation for pip and conda installation packages (#2044).
+
+  * Add bindings for linear SVM (#1935); `mlpack_linear_svm` from the
+    command-line, `linear_svm()` from Python.
+
+  * Add support to return the layer name as `std::string` (#1987).
+
+  * Speed and memory improvements for the Transposed Convolution layer (#1493).
 
 ### mlpack 3.2.1
 ###### 2019-10-01

README.md

@@ -23,7 +23,7 @@ src="https://cdn.rawgit.com/mlpack/mlpack.org/e7d36ed8/mlpack-black.svg" style="
 <p align="center">
   <em>
     Download:
-    <a href="https://www.mlpack.org/files/mlpack-3.2.0.tar.gz">current stable version (3.2.0)</a>
+    <a href="https://www.mlpack.org/files/mlpack-3.2.1.tar.gz">current stable version (3 2.1)</a>
   </em>
 </p>
 
@@ -109,7 +109,7 @@ If you are compiling Armadillo by hand, ensure that LAPACK and BLAS are enabled.
 ### 4. Building mlpack from source
 
 This section discusses how to build mlpack from source.  However, mlpack is in
-the repositories of many Linux distributions and so it may be easier to use the
+the repositories of many Linux distributions, so it may be easier to use the
 package manager for your system.  For example, on Ubuntu, you can install mlpack
 with the following command:
 
@@ -120,13 +120,13 @@ available---for instance, at the time of this writing, Ubuntu 16.04 only has
 mlpack 2.0.1 available.  Options include upgrading your Ubuntu version, finding
 a PPA or other non-official sources, or installing with a manual build.
 
-There are some other useful pages to consult in addition to this section:
+There are some useful pages to consult in addition to this section:
 
   - [Building mlpack From Source](https://www.mlpack.org/doc/mlpack-git/doxygen/build.html)
   - [Building mlpack From Source on Windows](https://www.mlpack.org/doc/mlpack-git/doxygen/build_windows.html)
 
 mlpack uses CMake as a build system and allows several flexible build
-configuration options. One can consult any of numerous CMake tutorials for
+configuration options. You can consult any of the CMake tutorials for
 further documentation, but this tutorial should be enough to get mlpack built
 and installed.
 
@@ -136,21 +136,20 @@ use mlpack-x.y.z where x.y.z is the version.
     $ tar -xzf mlpack-x.y.z.tar.gz
     $ cd mlpack-x.y.z
 
-Then, make a build directory.  The directory can have any name, not just
-'build', but 'build' is sufficient.
+Then, make a build directory.  The directory can have any name, but 'build' is
+sufficient.
 
     $ mkdir build
     $ cd build
 
 The next step is to run CMake to configure the project.  Running CMake is the
 equivalent to running `./configure` with autotools. If you run CMake with no
-options, it will configure the project to build with no debugging symbols and no
-profiling information:
+options, it will configure the project to build with no debugging symbols and 
+no profiling information:
 
     $ cmake ../
 
-You can specify options to compile with debugging information and profiling
-information:
+Options can be specified to compile with debugging information and profiling information:
 
     $ cmake -D DEBUG=ON -D PROFILE=ON ../
 
@@ -187,22 +186,21 @@ Once CMake is configured, building the library is as simple as typing 'make'.
 This will build all library components as well as 'mlpack_test'.
 
     $ make
-
-You can specify individual components which you want to build, if you do not
-want to build everything in the library:
+If you do not want to build everything in the library, individual components 
+of the build can be specified:
 
     $ make mlpack_pca mlpack_knn mlpack_kfn
 
 If the build fails and you cannot figure out why, register an account on Github
-and submit an issue; the mlpack developers will quickly help you figure it out:
+and submit an issue. The mlpack developers will quickly help you figure it out:
 
 [mlpack on Github](https://www.github.com/mlpack/mlpack/)
 
 Alternately, mlpack help can be found in IRC at `#mlpack` on irc.freenode.net.
 
-If you wish to install mlpack to `/usr/local/include/mlpack/` and `/usr/local/lib/`
-and `/usr/local/bin/`, once it has built, make sure you have root privileges (or
-write permissions to those three directories), and simply type
+If you wish to install mlpack to `/usr/local/include/mlpack/`, `/usr/local/lib/`,
+and `/usr/local/bin/`, make sure you have root privileges (or write permissions 
+to those three directories), and simply type
 
     $ make install
 
@@ -210,7 +208,7 @@ You can now run the executables by name; you can link against mlpack with
     `-lmlpack`
 and the mlpack headers are found in
     `/usr/local/include/mlpack/`
-and if Python bindings were built, they will be accessible with the `mlpack`
+and if Python bindings were built, you can access them with the `mlpack`
 package in Python.
 
 If running the programs (i.e. `$ mlpack_knn -h`) gives an error of the form

doc/guide/python_quickstart.hpp

@@ -17,11 +17,11 @@ Installing the mlpack bindings for Python is straightforward.  It's easy to use
 conda or pip to do this:
 
 @code{.sh}
-pip install mlpack3
+pip install mlpack
 @endcode
 
 @code{.sh}
-conda install -c mlpack mlpack
+conda install -c conda-forge mlpack
 @endcode
 
 Otherwise, we can build the Python bindings from scratch, as follows.  First we

src/mlpack/core/dists/gaussian_distribution.hpp

@@ -168,6 +168,12 @@ class GaussianDistribution
 
   void Covariance(arma::mat&& covariance);
 
+  //! Return the invCov.
+  const arma::mat& InvCov() const { return invCov; }
+
+  //! Return the logDetCov.
+  double LogDetCov() const { return logDetCov; }
+
   /**
    * Serialize the distribution.
    */

src/mlpack/core/tree/cover_tree/cover_tree.hpp

@@ -238,6 +238,20 @@ class CoverTree
    */
   CoverTree(CoverTree&& other);
 
+  /**
+   * Copy the given Cover Tree.
+   *
+   * @param other The tree to be copied.
+   */
+  CoverTree& operator=(const CoverTree& other);
+
+  /**
+   * Take ownership of the given Cover Tree.
+   *
+   * @param other The tree to take ownership of.
+   */
+  CoverTree& operator=(CoverTree&& other);
+
   /**
    * Create a cover tree from a boost::serialization archive.
    */

src/mlpack/core/tree/cover_tree/cover_tree_impl.hpp

@@ -497,6 +497,7 @@ CoverTree<MetricType, StatisticType, MatType, RootPointPolicy>::CoverTree(
     this->metric = new MetricType();
 }
 
+// Copy Constructor.
 template<
     typename MetricType,
     typename StatisticType,
@@ -515,9 +516,9 @@ CoverTree<MetricType, StatisticType, MatType, RootPointPolicy>::CoverTree(
     parent(other.parent),
     parentDistance(other.parentDistance),
     furthestDescendantDistance(other.furthestDescendantDistance),
-    localMetric(false),
+    localMetric(other.localMetric),
     localDataset(other.parent == NULL && other.localDataset),
-    metric(other.metric),
+    metric((other.localMetric ? new MetricType() : other.metric)),
     distanceComps(0)
 {
   // Copy each child by hand.
@@ -547,6 +548,76 @@ CoverTree<MetricType, StatisticType, MatType, RootPointPolicy>::CoverTree(
   }
 }
 
+// Copy Assignment.
+template<
+    typename MetricType,
+    typename StatisticType,
+    typename MatType,
+    typename RootPointPolicy
+>
+CoverTree<MetricType, StatisticType, MatType, RootPointPolicy>&
+CoverTree<MetricType, StatisticType, MatType, RootPointPolicy>::
+operator=(const CoverTree& other)
+{
+  if (this == &other)
+    return *this;
+
+  // Freeing memory that will not be used anymore.
+  if (localDataset)
+    delete dataset;
+
+  if (localMetric)
+    delete metric;
+
+  for (size_t i = 0; i < children.size(); ++i)
+    delete children[i];
+  children.clear();
+
+  dataset = ((other.parent == NULL && other.localDataset) ?
+      new MatType(*other.dataset) : other.dataset);
+  point = other.point;
+  scale = other.scale;
+  base = other.base;
+  stat = other.stat;
+  numDescendants = other.numDescendants;
+  parent = other.parent;
+  parentDistance = other.parentDistance;
+  furthestDescendantDistance = other.furthestDescendantDistance;
+  localMetric = other.localMetric;
+  localDataset = (other.parent == NULL && other.localDataset);
+  metric = (other.localMetric ? new MetricType() : other.metric);
+  distanceComps = 0;
+
+  // Copy each child by hand.
+  for (size_t i = 0; i < other.NumChildren(); ++i)
+  {
+    children.push_back(new CoverTree(other.Child(i)));
+    children[i]->Parent() = this;
+  }
+
+  // Propagate matrix, but only if we are the root.
+  if (parent == NULL && localDataset)
+  {
+    std::queue<CoverTree*> queue;
+
+    for (size_t i = 0; i < NumChildren(); ++i)
+      queue.push(children[i]);
+
+    while (!queue.empty())
+    {
+      CoverTree* node = queue.front();
+      queue.pop();
+
+      node->dataset = dataset;
+      for (size_t i = 0; i < node->NumChildren(); ++i)
+        queue.push(node->children[i]);
+    }
+  }
+
+  return *this;
+}
+
+// Move Constructor.
 template<
     typename MetricType,
     typename StatisticType,
@@ -587,6 +658,64 @@ CoverTree<MetricType, StatisticType, MatType, RootPointPolicy>::CoverTree(
   other.metric = NULL;
 }
 
+// Move Assignment.
+template<
+    typename MetricType,
+    typename StatisticType,
+    typename MatType,
+    typename RootPointPolicy
+>
+CoverTree<MetricType, StatisticType, MatType, RootPointPolicy>&
+CoverTree<MetricType, StatisticType, MatType, RootPointPolicy>::
+operator=(CoverTree&& other)
+{
+  if (this == &other)
+    return *this;
+
+  // Freeing memory that will not be used anymore.
+  if (localDataset)
+    delete dataset;
+
+  if (localMetric)
+    delete metric;
+
+  for (size_t i = 0; i < children.size(); ++i)
+    delete children[i];
+
+  dataset = other.dataset;
+  point = other.point;
+  children = std::move(other.children);
+  scale = other.scale;
+  base = other.base;
+  stat = std::move(other.stat);
+  numDescendants = other.numDescendants;
+  parent = other.parent;
+  parentDistance = other.parentDistance;
+  furthestDescendantDistance = other.furthestDescendantDistance;
+  localMetric = other.localMetric;
+  localDataset = other.localDataset;
+  metric = other.metric;
+  distanceComps = other.distanceComps;
+
+  // Set proper parent pointer.
+  for (size_t i = 0; i < children.size(); ++i)
+    children[i]->Parent() = this;
+
+  other.dataset = NULL;
+  other.point = 0;
+  other.scale = INT_MIN;
+  other.base = 0;
+  other.numDescendants = 0;
+  other.parent = NULL;
+  other.parentDistance = 0;
+  other.furthestDescendantDistance = 0;
+  other.localMetric = false;
+  other.localDataset = false;
+  other.metric = NULL;
+
+  return *this;
+}
+
 // Construct from a boost::serialization archive.
 template<
     typename MetricType,

src/mlpack/core/tree/spill_tree/spill_tree.hpp

@@ -209,6 +209,20 @@ class SpillTree
    */
   SpillTree(SpillTree&& other);
 
+   /**
+   * Copy the given Spill Tree.
+   *
+   * @param other The tree to be copied.
+   */
+  SpillTree& operator=(const SpillTree& other);
+
+  /**
+   * Take ownership of the given Spill Tree.
+   *
+   * @param other The tree to take ownership of.
+   */
+  SpillTree& operator=(SpillTree&& other);
+
   /**
    * Initialize the tree from a boost::serialization archive.
    *

src/mlpack/core/tree/spill_tree/spill_tree_impl.hpp

@@ -192,6 +192,88 @@ SpillTree(const SpillTree& other) :
   }
 }
 
+/**
+ * Copy Assignment.
+ */
+template<typename MetricType,
+         typename StatisticType,
+         typename MatType,
+         template<typename HyperplaneMetricType> class HyperplaneType,
+         template<typename SplitMetricType, typename SplitMatType>
+             class SplitType>
+SpillTree<MetricType, StatisticType, MatType, HyperplaneType, SplitType>&
+SpillTree<MetricType, StatisticType, MatType, HyperplaneType, SplitType>::
+operator=(const SpillTree& other)
+{
+  if (this == &other)
+    return *this;
+
+  // Freeing memory that will not be used anymore.
+  if (localDataset)
+    delete dataset;
+
+  delete pointsIndex;
+  delete left;
+  delete right;
+
+  left = NULL;
+  right = NULL;
+  parent = other.parent;
+  count = other.count;
+  pointsIndex = NULL;
+  overlappingNode = other.overlappingNode;
+  hyperplane = other.hyperplane;
+  bound = other.bound;
+  stat = other.stat;
+  parentDistance = other.parentDistance;
+  furthestDescendantDistance = other.furthestDescendantDistance;
+
+  // Copy matrix, but only if we are the root and the other tree has its own
+  // copy of the dataset.
+  dataset = (other.parent == NULL && other.localDataset) ?
+      new MatType(*other.dataset) : other.dataset;
+  localDataset = other.parent == NULL && other.localDataset;
+
+  // Create left and right children (if any).
+  if (other.Left())
+  {
+    left = new SpillTree(*other.Left());
+    left->Parent() = this; // Set parent to this, not other tree.
+  }
+
+  if (other.Right())
+  {
+    right = new SpillTree(*other.Right());
+    right->Parent() = this; // Set parent to this, not other tree.
+  }
+
+  // If vector of indexes, copy it.
+  if (other.pointsIndex)
+    pointsIndex = new arma::Col<size_t>(*other.pointsIndex);
+
+  // Propagate matrix, but only if we are the root.
+  if (parent == NULL && localDataset)
+  {
+    std::queue<SpillTree*> queue;
+    if (left)
+      queue.push(left);
+    if (right)
+      queue.push(right);
+    while (!queue.empty())
+    {
+      SpillTree* node = queue.front();
+      queue.pop();
+
+      node->dataset = dataset;
+      if (node->left)
+        queue.push(node->left);
+      if (node->right)
+        queue.push(node->right);
+    }
+  }
+  return *this;
+}
+
 /**
  * Move constructor.
  */
@@ -237,6 +319,66 @@ SpillTree(SpillTree&& other) :
     right->parent = this;
 }
 
+/**
+ * Move Assignment.
+ */
+template<typename MetricType,
+         typename StatisticType,
+         typename MatType,
+         template<typename HyperplaneMetricType> class HyperplaneType,
+         template<typename SplitMetricType, typename SplitMatType>
+             class SplitType>
+SpillTree<MetricType, StatisticType, MatType, HyperplaneType, SplitType>&
+SpillTree<MetricType, StatisticType, MatType, HyperplaneType, SplitType>::
+operator=(SpillTree&& other)
+{
+  if (this == &other)
+    return *this;
+
+  // Freeing memory that will not be used anymore.
+  if (localDataset)
+    delete dataset;
+
+  delete pointsIndex;
+  delete left;
+  delete right;
+
+  left = other.left;
+  right = other.right;
+  parent = other.parent;
+  count = other.count;
+  pointsIndex = other.pointsIndex;
+  overlappingNode = other.overlappingNode;
+  hyperplane = other.hyperplane;
+  bound = std::move(other.bound);
+  stat = std::move(other.stat);
+  parentDistance = other.parentDistance;
+  furthestDescendantDistance = other.furthestDescendantDistance;
+  minimumBoundDistance = other.minimumBoundDistance;
+  dataset = other.dataset;
+  localDataset = other.localDataset;
+
+  // Now we are a clone of the other tree.  But we must also clear the other
+  // tree's contents, so it doesn't delete anything when it is destructed.
+  other.left = NULL;
+  other.right = NULL;
+  other.count = 0;
+  other.pointsIndex = NULL;
+  other.parentDistance = 0.0;
+  other.furthestDescendantDistance = 0.0;
+  other.minimumBoundDistance = 0.0;
+  other.dataset = NULL;
+  other.localDataset = false;
+
+  // Set new parent.
+  if (left)
+    left->parent = this;
+  if (right)
+    right->parent = this;
+
+  return *this;
+}
+
 /**
  * Initialize the tree from an archive.
  */

src/mlpack/methods/ann/CMakeLists.txt

@@ -7,6 +7,7 @@ set(SOURCES
   rnn_impl.hpp
   brnn.hpp
   brnn_impl.hpp
+  layer_names.hpp
 )
 
 add_subdirectory(visitor)

src/mlpack/methods/ann/ffn.hpp

@@ -94,15 +94,18 @@ class FFN
    * object, be sure to use std::move to avoid unnecessary copy.
    *
    * @tparam OptimizerType Type of optimizer to use to train the model.
+   * @tparam CallbackTypes Types of Callback Functions.
    * @param predictors Input training variables.
    * @param responses Outputs results from input training variables.
    * @param optimizer Instantiated optimizer used to train the model.
+   * @param callbacks Callback Functions.
    * @return The final objective of the trained model (NaN or Inf on error).
    */
-  template<typename OptimizerType>
+  template<typename OptimizerType, typename... CallbackTypes>
   double Train(arma::mat predictors,
                arma::mat responses,
-               OptimizerType& optimizer);
+               OptimizerType& optimizer,
+               CallbackTypes&&... callbacks);
 
   /**
    * Train the feedforward network on the given input data. By default, the
@@ -118,11 +121,15 @@ class FFN
    *
    * @tparam OptimizerType Type of optimizer to use to train the model.
    * @param predictors Input training variables.
+   * @tparam CallbackTypes Types of Callback Functions.
    * @param responses Outputs results from input training variables.
+   * @param callbacks Callback Functions.
    * @return The final objective of the trained model (NaN or Inf on error).
    */
-  template<typename OptimizerType = ens::RMSProp>
-  double Train(arma::mat predictors, arma::mat responses);
+  template<typename OptimizerType = ens::RMSProp, typename... CallbackTypes>
+  double Train(arma::mat predictors,
+               arma::mat responses,
+               CallbackTypes&&... callbacks);
 
   /**
    * Predict the responses to a given set of predictors. The responses will
@@ -257,6 +264,14 @@ class FFN
    */
   void Add(LayerTypes<CustomLayers...> layer) { network.push_back(layer); }
 
+  //! Get the network model.
+  const std::vector<LayerTypes<CustomLayers...> >& Model() const
+  {
+    return network;
+  }
+  //! Modify the network model.
+  std::vector<LayerTypes<CustomLayers...> >& Model() { return network; }
+
   //! Return the number of separable functions (the number of predictor points).
   size_t NumFunctions() const { return numFunctions; }
 

src/mlpack/methods/ann/ffn_impl.hpp

@@ -69,17 +69,18 @@ void FFN<OutputLayerType, InitializationRuleType, CustomLayers...>::ResetData(
 
 template<typename OutputLayerType, typename InitializationRuleType,
          typename... CustomLayers>
-template<typename OptimizerType>
+template<typename OptimizerType, typename... CallbackTypes>
 double FFN<OutputLayerType, InitializationRuleType, CustomLayers...>::Train(
       arma::mat predictors,
       arma::mat responses,
-      OptimizerType& optimizer)
+      OptimizerType& optimizer,
+      CallbackTypes&&... callbacks)
 {
   ResetData(std::move(predictors), std::move(responses));
 
   // Train the model.
   Timer::Start("ffn_optimization");
-  const double out = optimizer.Optimize(*this, parameter);
+  const double out = optimizer.Optimize(*this, parameter, callbacks...);
   Timer::Stop("ffn_optimization");
 
   Log::Info << "FFN::FFN(): final objective of trained model is " << out
@@ -89,9 +90,11 @@ double FFN<OutputLayerType, InitializationRuleType, CustomLayers...>::Train(
 
 template<typename OutputLayerType, typename InitializationRuleType,
          typename... CustomLayers>
-template<typename OptimizerType>
+template<typename OptimizerType, typename... CallbackTypes>
 double FFN<OutputLayerType, InitializationRuleType, CustomLayers...>::Train(
-    arma::mat predictors, arma::mat responses)
+    arma::mat predictors,
+    arma::mat responses,
+    CallbackTypes&&... callbacks)
 {
   ResetData(std::move(predictors), std::move(responses));
 
@@ -99,7 +102,7 @@ double FFN<OutputLayerType, InitializationRuleType, CustomLayers...>::Train(
 
   // Train the model.
   Timer::Start("ffn_optimization");
-  const double out = optimizer.Optimize(*this, parameter);
+  const double out = optimizer.Optimize(*this, parameter, callbacks...);
   Timer::Stop("ffn_optimization");
 
   Log::Info << "FFN::FFN(): final objective of trained model is " << out

src/mlpack/methods/ann/gan/gan.hpp

@@ -66,9 +66,12 @@ class GAN
   /**
    * Constructor for GAN class.
    *
-   * @param trainData The real data.
    * @param generator Generator network.
    * @param discriminator Discriminator network.
+   * @param initializeRule Initialization rule to use for initializing
+   *                       parameters.
+   * @param noiseFunction Function to be used for generating noise.
+   * @param noiseDim Dimension of noise vector to be created.
    * @param batchSize Batch size to be used for training.
    * @param generatorUpdateStep Number of steps to train Discriminator
    *                            before updating Generator.
@@ -77,8 +80,7 @@ class GAN
    * @param clippingParameter Weight range for enforcing Lipschitz constraint.
    * @param lambda Parameter for setting the gradient penalty.
    */
-  GAN(arma::mat& trainData,
-      Model generator,
+  GAN(Model generator,
       Model discriminator,
       InitializationRuleType& initializeRule,
       Noise& noiseFunction,
@@ -96,6 +98,14 @@ class GAN
   //! Move constructor.
   GAN(GAN&&);
 
+  /**
+   * Initialize the generator, discriminator and weights of the model for
+   * training. This function won't actually trigger training process.
+   *
+   * @param trainData The data points of real distribution.
+   */
+  void ResetData(arma::mat trainData);
+
   // Reset function.
   void Reset();
 
@@ -110,7 +120,6 @@ class GAN
    */
   template<typename OptimizerType, typename... CallbackTypes>
   double Train(OptimizerType& Optimizer, CallbackTypes&&... callbacks);
-
   /**
    * Evaluate function for the Standard GAN and DCGAN.
    * This function gives the performance of the Standard GAN or DCGAN on the
@@ -280,11 +289,10 @@ class GAN
   /**
    * This function predicts the output of the network on the given input.
    *
-   * @param input The input the Discriminator network.
+   * @param input The input of the Generator network.
    * @param output Result of the Discriminator network.
    */
-  void Predict(arma::mat&& input,
-               arma::mat& output);
+  void Predict(arma::mat input, arma::mat& output);
 
   //! Return the parameters of the network.
   const arma::mat& Parameters() const { return parameter; }
@@ -318,6 +326,12 @@ class GAN
   void serialize(Archive& ar, const unsigned int /* version */);
 
  private:
+  /**
+  * Reset the module status by setting the current deterministic parameter
+  * for the discriminator and generator networks and their respective layers.
+  */
+  void ResetDeterministic();
+
   //! Locally stored parameter for training data + noise data.
   arma::mat predictors;
   //! Locally stored parameters of the network.
@@ -336,8 +350,6 @@ class GAN
   size_t numFunctions;
   //! Locally stored batch size parameter.
   size_t batchSize;
-  //! Locally stored number of iterations that have been completed.
-  size_t counter;
   //! Locally stored batch number which is being processed.
   size_t currentBatch;
   //! Locally stored number of training step before Generator is trained.
@@ -378,8 +390,12 @@ class GAN
   arma::mat noise;
   //! Locally stored gradient for Generator.
   arma::mat gradientGenerator;
-  //! Locally stored output of the Generator network.
-  arma::mat ganOutput;
+  //! The current evaluation mode (training or testing).
+  bool deterministic;
+  //! To keep track of number of generator weights in total weights.
+  size_t genWeights;
+  //! To keep track of number of discriminator weights in total weights.
+  size_t discWeights;
 };
 
 } // namespace ann

src/mlpack/methods/ann/gan/gan_impl.hpp

@@ -30,7 +30,6 @@ template<
   typename PolicyType
 >
 GAN<Model, InitializationRuleType, Noise, PolicyType>::GAN(
-    arma::mat& predictors,
     Model generator,
     Model discriminator,
     InitializationRuleType& initializeRule,
@@ -47,41 +46,23 @@ GAN<Model, InitializationRuleType, Noise, PolicyType>::GAN(
     initializeRule(initializeRule),
     noiseFunction(noiseFunction),
     noiseDim(noiseDim),
+    numFunctions(0),
     batchSize(batchSize),
+    currentBatch(0),
     generatorUpdateStep(generatorUpdateStep),
     preTrainSize(preTrainSize),
     multiplier(multiplier),
     clippingParameter(clippingParameter),
     lambda(lambda),
-    reset(false)
+    reset(false),
+    deterministic(false),
+    genWeights(0),
+    discWeights(0)
 {
   // Insert IdentityLayer for joining the Generator and Discriminator.
   this->discriminator.network.insert(
       this->discriminator.network.begin(),
       new IdentityLayer<>());
-
-  counter = 0;
-  currentBatch = 0;
-
-  this->discriminator.deterministic = this->generator.deterministic = true;
-
-  this->predictors.set_size(predictors.n_rows, predictors.n_cols + batchSize);
-  this->predictors.cols(0, predictors.n_cols - 1) = predictors;
-  this->discriminator.predictors = arma::mat(this->predictors.memptr(),
-      this->predictors.n_rows, this->predictors.n_cols, false, false);
-
-  responses.ones(1, predictors.n_cols + batchSize);
-  responses.cols(predictors.n_cols,
-      predictors.n_cols + batchSize - 1) = arma::zeros(1, batchSize);
-  this->discriminator.responses = arma::mat(this->responses.memptr(),
-      this->responses.n_rows, this->responses.n_cols, false, false);
-
-  numFunctions = predictors.n_cols;
-
-  noise.set_size(noiseDim, batchSize);
-
-  this->generator.predictors.set_size(noiseDim, batchSize);
-  this->generator.responses.set_size(predictors.n_rows, batchSize);
 }
 
 template<
@@ -106,11 +87,13 @@ GAN<Model, InitializationRuleType, Noise, PolicyType>::GAN(
     clippingParameter(network.clippingParameter),
     lambda(network.lambda),
     reset(network.reset),
-    counter(network.counter),
     currentBatch(network.currentBatch),
     parameter(network.parameter),
     numFunctions(network.numFunctions),
-    noise(network.noise)
+    noise(network.noise),
+    deterministic(network.deterministic),
+    genWeights(network.genWeights),
+    discWeights(network.discWeights)
 {
   /* Nothing to do here */
 }
@@ -137,15 +120,59 @@ GAN<Model, InitializationRuleType, Noise, PolicyType>::GAN(
     clippingParameter(network.clippingParameter),
     lambda(network.lambda),
     reset(network.reset),
-    counter(network.counter),
     currentBatch(network.currentBatch),
     parameter(std::move(network.parameter)),
     numFunctions(network.numFunctions),
-    noise(std::move(network.noise))
+    noise(std::move(network.noise)),
+    deterministic(network.deterministic),
+    genWeights(network.genWeights),
+    discWeights(network.discWeights)
 {
   /* Nothing to do here */
 }
 
+template<
+  typename Model,
+  typename InitializationRuleType,
+  typename Noise,
+  typename PolicyType
+>
+void GAN<Model, InitializationRuleType, Noise, PolicyType>::ResetData(
+    arma::mat trainData)
+{
+  currentBatch = 0;
+
+  numFunctions = trainData.n_cols;
+  noise.set_size(noiseDim, batchSize);
+
+  deterministic = true;
+  ResetDeterministic();
+
+  /**
+   * These predictors are shared by the discriminator network. The additional
+   * batch size predictors are taken from the generator network while training.
+   * For more details please look in EvaluateWithGradient() function.
+   */
+  this->predictors.set_size(trainData.n_rows, numFunctions + batchSize);
+  this->predictors.cols(0, numFunctions - 1) = std::move(trainData);
+  this->discriminator.predictors = arma::mat(this->predictors.memptr(),
+      this->predictors.n_rows, this->predictors.n_cols, false, false);
+
+  responses.ones(1, numFunctions + batchSize);
+  responses.cols(numFunctions, numFunctions + batchSize - 1) =
+      arma::zeros(1, batchSize);
+  this->discriminator.responses = arma::mat(this->responses.memptr(),
+      this->responses.n_rows, this->responses.n_cols, false, false);
+
+  this->generator.predictors.set_size(noiseDim, batchSize);
+  this->generator.responses.set_size(predictors.n_rows, batchSize);
+
+  if (!reset)
+  {
+    Reset();
+  }
+}
+
 template<
   typename Model,
   typename InitializationRuleType,
@@ -154,8 +181,8 @@ template<
 >
 void GAN<Model, InitializationRuleType, Noise, PolicyType>::Reset()
 {
-  size_t genWeights = 0;
-  size_t discWeights = 0;
+  genWeights = 0;
+  discWeights = 0;
 
   NetworkInitialization<InitializationRuleType> networkInit(initializeRule);
 
@@ -214,8 +241,16 @@ GAN<Model, InitializationRuleType, Noise, PolicyType>::Evaluate(
     const size_t i,
     const size_t /* batchSize */)
 {
-  if (!reset)
+  if (parameter.is_empty())
+  {
     Reset();
+  }
+
+  if (!deterministic)
+  {
+    deterministic = true;
+    ResetDeterministic();
+  }
 
   currentInput = arma::mat(predictors.memptr() + (i * predictors.n_rows),
       predictors.n_rows, batchSize, false, false);
@@ -263,8 +298,10 @@ EvaluateWithGradient(const arma::mat& /* parameters */,
                      GradType& gradient,
                      const size_t /* batchSize */)
 {
-  if (!reset)
+  if (parameter.is_empty())
+  {
     Reset();
+  }
 
   if (gradient.is_empty())
   {
@@ -275,6 +312,12 @@ EvaluateWithGradient(const arma::mat& /* parameters */,
   else
     gradient.zeros();
 
+  if (this->deterministic)
+  {
+    this->deterministic = false;
+    ResetDeterministic();
+  }
+
   if (noiseGradientDiscriminator.is_empty())
   {
     noiseGradientDiscriminator = arma::zeros<arma::mat>(
@@ -326,14 +369,8 @@ EvaluateWithGradient(const arma::mat& /* parameters */,
     gradientGenerator *= multiplier;
   }
 
-  counter++;
   currentBatch++;
 
-  // Revert the counter to zero, if the total dataset get's covered.
-  if (counter * batchSize >= numFunctions)
-  {
-    counter = 0;
-  }
 
   if (preTrainSize > 0)
   {
@@ -383,12 +420,13 @@ template<
 void GAN<Model, InitializationRuleType, Noise, PolicyType>::Forward(
     arma::mat&& input)
 {
-  if (!reset)
+  if (parameter.is_empty())
+  {
     Reset();
+  }
 
   generator.Forward(std::move(input));
-  ganOutput = boost::apply_visitor(
-      outputParameterVisitor,
+  arma::mat ganOutput = boost::apply_visitor(outputParameterVisitor,
       generator.network.back());
 
   discriminator.Forward(std::move(ganOutput));
@@ -401,10 +439,18 @@ template<
   typename PolicyType
 >
 void GAN<Model, InitializationRuleType, Noise, PolicyType>::
-Predict(arma::mat&& input, arma::mat& output)
+Predict(arma::mat input, arma::mat& output)
 {
-  if (!reset)
+  if (parameter.is_empty())
+  {
     Reset();
+  }
+
+  if (!deterministic)
+  {
+    deterministic = true;
+    ResetDeterministic();
+  }
 
   Forward(std::move(input));
 
@@ -412,6 +458,21 @@ Predict(arma::mat&& input, arma::mat& output)
       discriminator.network.back());
 }
 
+template<
+  typename Model,
+  typename InitializationRuleType,
+  typename Noise,
+  typename PolicyType
+>
+void GAN<Model, InitializationRuleType, Noise, PolicyType>::
+ResetDeterministic()
+{
+  this->discriminator.deterministic = deterministic;
+  this->generator.deterministic = deterministic;
+  this->discriminator.ResetDeterministic();
+  this->generator.ResetDeterministic();
+}
+
 template<
   typename Model,
   typename InitializationRuleType,
@@ -425,7 +486,39 @@ serialize(Archive& ar, const unsigned int /* version */)
   ar & BOOST_SERIALIZATION_NVP(parameter);
   ar & BOOST_SERIALIZATION_NVP(generator);
   ar & BOOST_SERIALIZATION_NVP(discriminator);
-  ar & BOOST_SERIALIZATION_NVP(noiseFunction);
+  ar & BOOST_SERIALIZATION_NVP(reset);
+  ar & BOOST_SERIALIZATION_NVP(genWeights);
+  ar & BOOST_SERIALIZATION_NVP(discWeights);
+
+  if (Archive::is_loading::value)
+  {
+    // Share the parameters between the network.
+    generator.Parameters() = arma::mat(parameter.memptr(), genWeights, 1, false,
+        false);
+    discriminator.Parameters() = arma::mat(parameter.memptr() + genWeights,
+        discWeights, 1, false, false);
+
+    size_t offset = 0;
+    for (size_t i = 0; i < generator.network.size(); ++i)
+    {
+      offset += boost::apply_visitor(WeightSetVisitor(std::move(
+          generator.parameter), offset), generator.network[i]);
+
+      boost::apply_visitor(resetVisitor, generator.network[i]);
+    }
+
+    offset = 0;
+    for (size_t i = 0; i < discriminator.network.size(); ++i)
+    {
+      offset += boost::apply_visitor(WeightSetVisitor(std::move(
+          discriminator.parameter), offset), discriminator.network[i]);
+
+      boost::apply_visitor(resetVisitor, discriminator.network[i]);
+    }
+
+    deterministic = true;
+    ResetDeterministic();
+  }
 }
 
 } // namespace ann

src/mlpack/methods/ann/gan/wgan_impl.hpp

@@ -34,8 +34,16 @@ GAN<Model, InitializationRuleType, Noise, PolicyType>::Evaluate(
     const size_t i,
     const size_t /* batchSize */)
 {
-  if (!reset)
+  if (parameter.is_empty())
+  {
     Reset();
+  }
+
+  if (!deterministic)
+  {
+    deterministic = true;
+    ResetDeterministic();
+  }
 
   currentInput = arma::mat(predictors.memptr() + (i * predictors.n_rows),
       predictors.n_rows, batchSize, false, false);
@@ -82,8 +90,10 @@ EvaluateWithGradient(const arma::mat& /* parameters */,
                      GradType& gradient,
                      const size_t /* batchSize */)
 {
-  if (!reset)
+  if (parameter.is_empty())
+  {
     Reset();
+  }
 
   if (gradient.is_empty())
   {
@@ -94,6 +104,12 @@ EvaluateWithGradient(const arma::mat& /* parameters */,
   else
     gradient.zeros();
 
+  if (this->deterministic)
+  {
+    this->deterministic = false;
+    ResetDeterministic();
+  }
+
   if (noiseGradientDiscriminator.is_empty())
   {
     noiseGradientDiscriminator = arma::zeros<arma::mat>(
@@ -147,15 +163,8 @@ EvaluateWithGradient(const arma::mat& /* parameters */,
     gradientGenerator *= multiplier;
   }
 
-  counter++;
   currentBatch++;
 
-  // Revert the counter to zero, if the total dataset get's covered.
-  if (counter * batchSize >= numFunctions)
-  {
-    counter = 0;
-  }
-
   if (preTrainSize > 0)
   {
     preTrainSize--;

src/mlpack/methods/ann/gan/wgangp_impl.hpp

@@ -35,8 +35,16 @@ GAN<Model, InitializationRuleType, Noise, PolicyType>::Evaluate(
     const size_t i,
     const size_t /* batchSize */)
 {
-  if (!reset)
+  if ((parameter.is_empty()))
+  {
     Reset();
+  }
+
+  if (!deterministic)
+  {
+    deterministic = true;
+    ResetDeterministic();
+  }
 
   currentInput = arma::mat(predictors.memptr() + (i * predictors.n_rows),
       predictors.n_rows, batchSize, false, false);
@@ -96,8 +104,10 @@ EvaluateWithGradient(const arma::mat& /* parameters */,
                      GradType& gradient,
                      const size_t /* batchSize */)
 {
-  if (!reset)
+  if (parameter.is_empty())
+  {
     Reset();
+  }
 
   if (gradient.is_empty())
   {
@@ -108,6 +118,12 @@ EvaluateWithGradient(const arma::mat& /* parameters */,
   else
     gradient.zeros();
 
+  if (this->deterministic)
+  {
+    this->deterministic = false;
+    ResetDeterministic();
+  }
+
   if (noiseGradientDiscriminator.is_empty())
   {
     noiseGradientDiscriminator = arma::zeros<arma::mat>(
@@ -171,15 +187,8 @@ EvaluateWithGradient(const arma::mat& /* parameters */,
     gradientGenerator *= multiplier;
   }
 
-  counter++;
   currentBatch++;
 
-  // Revert the counter to zero, if the total dataset get's covered.
-  if (counter * batchSize >= numFunctions)
-  {
-    counter = 0;
-  }
-
   if (preTrainSize > 0)
   {
     preTrainSize--;

src/mlpack/methods/ann/layer/atrous_convolution.hpp

@@ -309,6 +309,28 @@ class AtrousConvolution
 } // namespace ann
 } // namespace mlpack
 
+//! Set the serialization version of the AtrousConvolution class.
+namespace boost {
+namespace serialization {
+
+template<
+    typename ForwardConvolutionRule,
+    typename BackwardConvolutionRule,
+    typename GradientConvolutionRule,
+    typename InputDataType,
+    typename OutputDataType
+>
+struct version<
+    mlpack::ann::AtrousConvolution<ForwardConvolutionRule,
+        BackwardConvolutionRule, GradientConvolutionRule, InputDataType,
+        OutputDataType> >
+{
+  BOOST_STATIC_CONSTANT(int, value = 1);
+};
+
+} // namespace serialization
+} // namespace boost
+
 // Include implementation
 #include "atrous_convolution_impl.hpp"
 

src/mlpack/methods/ann/layer/atrous_convolution_impl.hpp

@@ -339,8 +339,7 @@ void AtrousConvolution<
     GradientConvolutionRule,
     InputDataType,
     OutputDataType
->::serialize(
-    Archive& ar, const unsigned int /* version */)
+>::serialize(Archive& ar, const unsigned int version)
 {
   ar & BOOST_SERIALIZATION_NVP(inSize);
   ar & BOOST_SERIALIZATION_NVP(outSize);
@@ -358,6 +357,9 @@ void AtrousConvolution<
   ar & BOOST_SERIALIZATION_NVP(dilationW);
   ar & BOOST_SERIALIZATION_NVP(dilationH);
 
+  if (version > 0)
+    ar & BOOST_SERIALIZATION_NVP(padding);
+
   if (Archive::is_loading::value)
     weights.set_size((outSize * inSize * kW * kH) + outSize, 1);
 }

src/mlpack/methods/ann/layer/convolution.hpp

@@ -300,6 +300,27 @@ class Convolution
 } // namespace ann
 } // namespace mlpack
 
+//! Set the serialization version of the Convolution class.
+namespace boost {
+namespace serialization {
+
+template<
+    typename ForwardConvolutionRule,
+    typename BackwardConvolutionRule,
+    typename GradientConvolutionRule,
+    typename InputDataType,
+    typename OutputDataType
+>
+struct version<
+    mlpack::ann::Convolution<ForwardConvolutionRule, BackwardConvolutionRule,
+        GradientConvolutionRule, InputDataType, OutputDataType> >
+{
+  BOOST_STATIC_CONSTANT(int, value = 1);
+};
+
+} // namespace serialization
+} // namespace boost
+
 // Include implementation.
 #include "convolution_impl.hpp"
 

src/mlpack/methods/ann/layer/convolution_impl.hpp

@@ -319,8 +319,7 @@ void Convolution<
     GradientConvolutionRule,
     InputDataType,
     OutputDataType
->::serialize(
-    Archive& ar, const unsigned int /* version */)
+>::serialize(Archive& ar, const unsigned int version)
 {
   ar & BOOST_SERIALIZATION_NVP(inSize);
   ar & BOOST_SERIALIZATION_NVP(outSize);
@@ -336,6 +335,9 @@ void Convolution<
   ar & BOOST_SERIALIZATION_NVP(outputWidth);
   ar & BOOST_SERIALIZATION_NVP(outputHeight);
 
+  if (version > 0)
+    ar & BOOST_SERIALIZATION_NVP(padding);
+
   if (Archive::is_loading::value)
     weights.set_size((outSize * inSize * kW * kH) + outSize, 1);
 }

src/mlpack/methods/ann/layer/layer_types.hpp

@@ -213,7 +213,7 @@ using LayerTypes = boost::variant<
                 NaiveConvolution<FullConvolution>,
                 NaiveConvolution<ValidConvolution>, arma::mat, arma::mat>*,
     TransposedConvolution<NaiveConvolution<ValidConvolution>,
-            NaiveConvolution<FullConvolution>,
+            NaiveConvolution<ValidConvolution>,
             NaiveConvolution<ValidConvolution>, arma::mat, arma::mat>*,
     DropConnect<arma::mat, arma::mat>*,
     Dropout<arma::mat, arma::mat>*,

src/mlpack/methods/ann/layer/transposed_convolution.hpp

@@ -40,7 +40,7 @@ namespace ann /** Artificial Neural Network. */ {
  */
 template <
     typename ForwardConvolutionRule = NaiveConvolution<ValidConvolution>,
-    typename BackwardConvolutionRule = NaiveConvolution<FullConvolution>,
+    typename BackwardConvolutionRule = NaiveConvolution<ValidConvolution>,
     typename GradientConvolutionRule = NaiveConvolution<ValidConvolution>,
     typename InputDataType = arma::mat,
     typename OutputDataType = arma::mat
@@ -52,8 +52,14 @@ class TransposedConvolution
   TransposedConvolution();
 
   /**
-   * Create the Transposed Convolution object using the specified number of input maps,
-   * output maps, filter size, stride and padding parameter.
+   * Create the Transposed Convolution object using the specified number of
+   * input maps, output maps, filter size, stride and padding parameter.
+   *
+   * Note: The equivalent stride of a transposed convolution operation is always
+   * equal to 1. In this implementation, stride of filter represents the stride
+   * of the associated convolution operation.
+   * Note: Padding of input represents padding of associated convolution
+   * operation.
    *
    * @param inSize The number of input maps.
    * @param outSize The number of output maps.
@@ -65,6 +71,8 @@ class TransposedConvolution
    * @param padH Padding height of the input.
    * @param inputWidth The width of the input data.
    * @param inputHeight The height of the input data.
+   * @param outputWidth The width of the output data.
+   * @param outputHeight The height of the output data.
    */
   TransposedConvolution(const size_t inSize,
                         const size_t outSize,
@@ -75,7 +83,9 @@ class TransposedConvolution
                         const size_t padW = 0,
                         const size_t padH = 0,
                         const size_t inputWidth = 0,
-                        const size_t inputHeight = 0);
+                        const size_t inputHeight = 0,
+                        const size_t outputWidth = 0,
+                        const size_t outputHeight = 0);
 
   /*
    * Set the weight and bias term.
@@ -173,24 +183,6 @@ class TransposedConvolution
   void serialize(Archive& ar, const unsigned int /* version */);
 
  private:
-  /*
-   * Return the transposed convolution output size.
-   *
-   * @param size The size of the input (row or column).
-   * @param k The size of the filter (width or height).
-   * @param s The stride size (x or y direction).
-   * @param p The size of the padding (width or height).
-   * @return The transposed convolution output size.
-   */
-  size_t TransposedConvOutSize(const size_t size,
-                               const size_t k,
-                               const size_t s,
-                               const size_t p)
-  {
-    size_t out = std::floor(size - k + 2 * p) / s;
-    return out * s + 2 * (k - p) - 1 + ((((size + 2 * p - k) % s) + s) % s);
-  }
-
   /*
    * Rotates a 3rd-order tensor counterclockwise by 180 degrees.
    *
@@ -220,6 +212,119 @@ class TransposedConvolution
     output = arma::fliplr(arma::flipud(input));
   }
 
+  /*
+   * Pad the given input data.
+   *
+   * @param input The input to be padded.
+   * @param wPad Padding width of the input.
+   * @param hPad Padding height of the input.
+   * @param wExtra The number of extra zeros to the right.
+   * @param hExtra The number of extra zeros to the bottom.
+   * @param output The padded output data.
+   */
+  template<typename eT>
+  void Pad(const arma::Mat<eT>& input,
+           const size_t wPad,
+           const size_t hPad,
+           const size_t wExtra,
+           const size_t hExtra,
+           arma::Mat<eT>& output)
+  {
+    if (output.n_rows != input.n_rows + wPad * 2 + wExtra ||
+        output.n_cols != input.n_cols + hPad * 2 + hExtra)
+    {
+      output = arma::zeros(input.n_rows + wPad * 2 + wExtra,
+          input.n_cols + hPad * 2 + hExtra);
+    }
+
+    output.submat(wPad, hPad, wPad + input.n_rows - 1,
+        hPad + input.n_cols - 1) = input;
+  }
+
+  /*
+   * Pad the given input data.
+   *
+   * @param input The input to be padded.
+   * @param wPad Padding width of the input.
+   * @param hPad Padding height of the input.
+   * @param wExtra The number of extra zeros to the right.
+   * @param hExtra The number of extra zeros to the bottom.
+   * @param output The padded output data.
+   */
+  template<typename eT>
+  void Pad(const arma::Cube<eT>& input,
+           const size_t wPad,
+           const size_t hPad,
+           const size_t wExtra,
+           const size_t hExtra,
+           arma::Cube<eT>& output)
+  {
+    output = arma::zeros(input.n_rows + wPad * 2 + wExtra,
+        input.n_cols + hPad * 2 + hExtra, input.n_slices);
+
+    for (size_t i = 0; i < input.n_slices; ++i)
+    {
+      Pad<eT>(input.slice(i), wPad, hPad, wExtra, hExtra, output.slice(i));
+    }
+  }
+
+  /*
+   * Insert zeros between the units of the given input data.
+   * Note: This function should be used before the Pad() function.
+   *
+   * @param input The input to be padded.
+   * @param dW Stride of filter application in the x direction.
+   * @param dH Stride of filter application in the y direction.
+   * @param output The padded output data.
+   */
+  template<typename eT>
+  void InsertZeros(const arma::Mat<eT>& input,
+                   const size_t dW,
+                   const size_t dH,
+                   arma::Mat<eT>& output)
+  {
+    if (output.n_rows != input.n_rows * dW - dW + 1 ||
+        output.n_cols != input.n_cols * dH - dH + 1)
+    {
+      output = arma::zeros(input.n_rows * dW - dW + 1,
+          input.n_cols * dH - dH + 1);
+    }
+
+    for (size_t i = 0; i < output.n_rows; i += dH)
+    {
+      for (size_t j = 0; j < output.n_cols; j += dW)
+      {
+        // TODO: Use [] instead of () for speedup after this is completely
+        // debugged and approved.
+        output(i, j) = input(i / dH, j / dW);
+      }
+    }
+  }
+
+  /*
+   * Insert zeros between the units of the given input data.
+   * Note: This function should be used before the Pad() function.
+   *
+   * @param input The input to be padded.
+   * @param dW Stride of filter application in the x direction.
+   * @param dH Stride of filter application in the y direction.
+   * @param output The padded output data.
+   */
+  template<typename eT>
+  void InsertZeros(const arma::Cube<eT>& input,
+                   const size_t dW,
+                   const size_t dH,
+                   arma::Cube<eT>& output)
+  {
+    output = arma::zeros(input.n_rows * dW - dW + 1,
+        input.n_cols * dH - dH + 1, input.n_slices);
+
+    for (size_t i = 0; i < input.n_slices; ++i)
+    {
+      InsertZeros<eT>(input.slice(i), dW, dH, output.slice(i));
+    }
+  }
+
   //! Locally-stored number of input channels.
   size_t inSize;
 
@@ -247,6 +352,12 @@ class TransposedConvolution
   //! Locally-stored padding height.
   size_t padH;
 
+  //! Locally-stored number of zeros added to the right of input.
+  size_t aW;
+
+  //! Locally-stored number of zeros added to the top of input.
+  size_t aH;
+
   //! Locally-stored weight object.
   OutputDataType weights;
 
@@ -277,6 +388,9 @@ class TransposedConvolution
   //! Locally-stored transformed padded input parameter.
   arma::cube inputPaddedTemp;
 
+  //! Locally-stored transformed expanded input parameter.
+  arma::cube inputExpandedTemp;
+
   //! Locally-stored transformed error parameter.
   arma::cube gTemp;
 

src/mlpack/methods/ann/layer/transposed_convolution_impl.hpp

@@ -60,22 +60,37 @@ TransposedConvolution<
     const size_t padW,
     const size_t padH,
     const size_t inputWidth,
-    const size_t inputHeight) :
+    const size_t inputHeight,
+    const size_t outputWidth,
+    const size_t outputHeight) :
     inSize(inSize),
     outSize(outSize),
     kW(kW),
     kH(kH),
     dW(dW),
     dH(dH),
-    padW(padW),
-    padH(padH),
+    padW(kW - padW - 1),
+    padH(kH - padH - 1),
     inputWidth(inputWidth),
     inputHeight(inputHeight),
-    outputWidth(0),
-    outputHeight(0)
+    outputWidth(outputWidth),
+    outputHeight(outputHeight)
 {
   weights.set_size((outSize * inSize * kW * kH) + outSize, 1);
-  padding = new Padding<>(padW, padW, padH, padH);
+  // TODO: Use the Padding layer.
+  // padding = new Padding<>(this->padW, this->padW, this->padH, this->padH);
+
+  aW = (outputWidth + kW - 2 * this->padW - 2) % dW;
+  aH = (outputHeight + kH - 2 * this->padH - 2) % dH;
+
+  // Check if the output height and width are possible given the other
+  // parameters of the layer.
+  if (outputWidth != dW * (inputWidth - 1) + aW + 2 * this->padW + 2 - kW ||
+      outputHeight != dH * (inputHeight - 1) + aH + 2 * this->padH + 2 - kH)
+  {
+    Log::Fatal << "The output width / output height is not possible given "
+        << "the other parameters of the layer." << std::endl;
+  }
 }
 
 template<
@@ -119,8 +134,23 @@ void TransposedConvolution<
   inputTemp = arma::cube(const_cast<arma::Mat<eT>&&>(input).memptr(),
       inputWidth, inputHeight, inSize * batchSize, false, false);
 
-  outputWidth = TransposedConvOutSize(inputWidth, kW, dW, padW);
-  outputHeight = TransposedConvOutSize(inputHeight, kH, dH, padH);
+  if (dW > 1 || dH > 1)
+  {
+    InsertZeros(inputTemp, dW, dH, inputExpandedTemp);
+
+    if (padW != 0 || padH != 0 || aW != 0 || aH != 0)
+      Pad(inputExpandedTemp, padW, padH, aW, aH, inputPaddedTemp);
+    else
+    {
+      inputPaddedTemp = arma::Cube<eT>(inputExpandedTemp.memptr(),
+          inputExpandedTemp.n_rows, inputExpandedTemp.n_cols,
+          inputExpandedTemp.n_slices, false, false);;
+    }
+  }
+  else if (padW != 0 || padH != 0 || aW != 0 || aH != 0)
+  {
+    Pad(inputTemp, padW, padH, aW, aH, inputPaddedTemp);
+  }
 
   output.set_size(outputWidth * outputHeight * outSize, batchSize);
   outputTemp = arma::Cube<eT>(output.memptr(), outputWidth, outputHeight,
@@ -141,8 +171,16 @@ void TransposedConvolution<
       arma::Mat<eT> convOutput, rotatedFilter;
       Rotate180(weight.slice(outMapIdx), rotatedFilter);
 
-      BackwardConvolutionRule::Convolution(inputTemp.slice(inMap +
-          batchCount * inSize), rotatedFilter, convOutput, 1, 1);
+      if (dW > 1 || dH > 1 || padW != 0 || padH != 0 || aW != 0 || aH != 0)
+      {
+        ForwardConvolutionRule::Convolution(inputPaddedTemp.slice(inMap +
+            batchCount * inSize), rotatedFilter, convOutput, 1, 1);
+      }
+      else
+      {
+        ForwardConvolutionRule::Convolution(inputTemp.slice(inMap +
+            batchCount * inSize), rotatedFilter, convOutput, 1, 1);
+      }
 
       outputTemp.slice(outMap) += convOutput;
     }
@@ -168,8 +206,13 @@ void TransposedConvolution<
 >::Backward(
     const arma::Mat<eT>&& /* input */, arma::Mat<eT>&& gy, arma::Mat<eT>&& g)
 {
-  arma::cube mappedError(gy.memptr(), outputWidth, outputHeight,
+  arma::Cube<eT> mappedError(gy.memptr(), outputWidth, outputHeight,
       outSize * batchSize, false, false);
+
+  arma::Cube<eT> mappedErrorPadded;
+  if ((int)(kW - padW - 1) > 0 || (int)(kH - padH - 1) > 0)
+    Pad(mappedError, kW - padW - 1, kH - padH - 1, 0, 0, mappedErrorPadded);
+
   g.set_size(inputTemp.n_rows * inputTemp.n_cols * inSize, batchSize);
   gTemp = arma::Cube<eT>(g.memptr(), inputTemp.n_rows,
       inputTemp.n_cols, inputTemp.n_slices, false, false);
@@ -189,8 +232,16 @@ void TransposedConvolution<
     {
       arma::Mat<eT> output;
 
-      ForwardConvolutionRule::Convolution(mappedError.slice(outMap),
-          weight.slice(outMapIdx), output, 1, 1);
+      if ((int)(kW - padW - 1) > 0 || (int)(kH - padH - 1) > 0)
+      {
+        BackwardConvolutionRule::Convolution(mappedErrorPadded.slice(outMap),
+            weight.slice(outMapIdx), output, dW, dH);
+      }
+      else
+      {
+        BackwardConvolutionRule::Convolution(mappedError.slice(outMap),
+            weight.slice(outMapIdx), output, dW, dH);
+      }
 
       gTemp.slice(inMap + batchCount * inSize) += output;
     }
@@ -216,7 +267,7 @@ void TransposedConvolution<
     arma::Mat<eT>&& error,
     arma::Mat<eT>&& gradient)
 {
-  arma::cube mappedError(error.memptr(), outputWidth,
+  arma::Cube<eT> mappedError(error.memptr(), outputWidth,
       outputHeight, outSize * batchSize, false, false);
 
   gradient.set_size(weights.n_elem, 1);
@@ -224,6 +275,8 @@ void TransposedConvolution<
       weight.n_cols, weight.n_slices, false, false);
   gradientTemp.zeros();
 
+  arma::Mat<eT> inputSlice, output, deltaSlice, rotatedOutput;
+
   for (size_t outMap = 0, outMapIdx = 0, batchCount = 0; outMap <
       outSize * batchSize; outMap++)
   {
@@ -233,16 +286,23 @@ void TransposedConvolution<
       outMapIdx = 0;
     }
 
+    deltaSlice = mappedError.slice(outMap);
+
     for (size_t inMap = 0; inMap < inSize; inMap++, outMapIdx++)
     {
-      arma::Mat<eT> inputSlice, output;
-      inputSlice = inputTemp.slice(inMap + batchCount * inSize);
-      arma::Mat<eT> deltaSlice = mappedError.slice(outMap);
+      if (dW > 1 || dH > 1 || padW != 0 || padH != 0 || aW != 0 || aH != 0)
+      {
+        inputSlice = inputPaddedTemp.slice(inMap + batchCount * inSize);
+      }
+      else
+      {
+        inputSlice = inputTemp.slice(inMap + batchCount * inSize);
+      }
 
-      GradientConvolutionRule::Convolution(deltaSlice, inputSlice,
+      GradientConvolutionRule::Convolution(inputSlice, deltaSlice,
           output, 1, 1);
-
-      gradientTemp.slice(outMapIdx) += output;
+      Rotate180(output, rotatedOutput);
+      gradientTemp.slice(outMapIdx) += rotatedOutput;
     }
 
     gradient.submat(weight.n_elem + (outMap % outSize), 0, weight.n_elem +
@@ -282,7 +342,12 @@ void TransposedConvolution<
   ar & BOOST_SERIALIZATION_NVP(outputHeight);
 
   if (Archive::is_loading::value)
+  {
     weights.set_size((outSize * inSize * kW * kH) + outSize, 1);
+
+    aW = (outputWidth + kW - 2 * padW - 2) % dW;
+    aH = (outputHeight + kH - 2 * padH - 2) % dH;
+  }
 }
 
 } // namespace ann

src/mlpack/methods/ann/layer_names.hpp

@@ -0,0 +1,322 @@
+/**
+ * @file layer_names.hpp
+ * @author Sreenik Seal
+ *
+ * Implementation of a class that converts a given ann layer to string format.
+ *
+ * mlpack is free software; you may redistribute it and/or modify it under the
+ * terms of the 3-clause BSD license.  You should have received a copy of the
+ * 3-clause BSD license along with mlpack.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+
+#include <mlpack/core.hpp>
+#include <mlpack/methods/ann/layer/layer.hpp>
+#include <mlpack/methods/ann/layer/layer_types.hpp>
+#include <boost/variant/static_visitor.hpp>
+#include <string>
+
+using namespace mlpack::ann;
+
+/**
+ * Implementation of a class that returns the string representation of the
+ * name of the given layer.
+ */
+class LayerNameVisitor : public boost::static_visitor<std::string>
+{
+ public:
+  //! Create the LayerNameVisitor object.
+  LayerNameVisitor()
+  {
+  }
+
+  /*
+   * Return the name of the given layer of type AtrousConvolution as a string.
+   * 
+   * @param Given layer of type AtrousConvolution.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(AtrousConvolution<>* /*layer*/) const
+  {
+    return "atrousconvolution";
+  }
+
+  /*
+   * Return the name of the given layer of type AlphaDropout as a string.
+   * 
+   * @param Given layer of type AlphaDropout.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(AlphaDropout<>* /*layer*/) const
+  {
+    return "alphadropout";
+  }
+
+  /*
+   * Return the name of the given layer of type BatchNorm as a string.
+   * 
+   * @param Given layer of type BatchNorm.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(BatchNorm<>* /*layer*/) const
+  {
+    return "batchnorm";
+  }
+
+  /*
+   * Return the name of the given layer of type Constant as a string.
+   * 
+   * @param Given layer of type Constant.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(Constant<>* /*layer*/) const
+  {
+    return "constant";
+  }
+
+  /*
+   * Return the name of the given layer of type Convolution as a string.
+   * 
+   * @param Given layer of type Convolution.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(Convolution<>* /*layer*/) const
+  {
+    return "convolution";
+  }
+
+  /*
+   * Return the name of the given layer of type DropConnect as a string.
+   * 
+   * @param Given layer of type DropConnect.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(DropConnect<>* /*layer*/) const
+  {
+    return "dropconnect";
+  }
+
+  /*
+   * Return the name of the given layer of type Dropout as a string.
+   * 
+   * @param Given layer of type Dropout.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(Dropout<>* /*layer*/) const
+  {
+    return "dropout";
+  }
+
+  /*
+   * Return the name of the given layer of type FlexibleReLU as a string.
+   * 
+   * @param Given layer of type FlexibleReLU.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(FlexibleReLU<>* /*layer*/) const
+  {
+    return "flexiblerelu";
+  }
+
+  /*
+   * Return the name of the given layer of type LayerNorm as a string.
+   * 
+   * @param Given layer of type LayerNorm.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(LayerNorm<>* /*layer*/) const
+  {
+    return "layernorm";
+  }
+
+  /*
+   * Return the name of the given layer of type Linear as a string.
+   * 
+   * @param Given layer of type Linear.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(Linear<>* /*layer*/) const
+  {
+    return "linear";
+  }
+
+  /*
+   * Return the name of the given layer of type LinearNoBias as a string.
+   * 
+   * @param Given layer of type LinearNoBias.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(LinearNoBias<>* /*layer*/) const
+  {
+    return "linearnobias";
+  }
+
+  /*
+   * Return the name of the given layer of type MaxPooling as a string.
+   * 
+   * @param Given layer of type MaxPooling.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(MaxPooling<>* /*layer*/) const
+  {
+    return "maxpooling";
+  }
+
+  /*
+   * Return the name of the given layer of type MeanPooling as a string.
+   * 
+   * @param Given layer of type MeanPooling.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(MeanPooling<>* /*layer*/) const
+  {
+    return "meanpooling";
+  }
+
+  /*
+   * Return the name of the given layer of type MultiplyConstant as a string.
+   * 
+   * @param Given layer of type MultiplyConstant.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(MultiplyConstant<>* /*layer*/) const
+  {
+    return "multiplyconstant";
+  }
+
+  /*
+   * Return the name of the given layer of type ReLULayer as a string.
+   * 
+   * @param Given layer of type ReLULayer.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(ReLULayer<>* /*layer*/) const
+  {
+    return "relu";
+  }
+
+  /*
+   * Return the name of the given layer of type TransposedConvolution as a
+   * string.
+   * 
+   * @param Given layer of type TransposedConvolution.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(TransposedConvolution<>* /*layer*/) const
+  {
+    return "transposedconvolution";
+  }
+
+  /*
+   * Return the name of the given layer of type IdentityLayer as a string.
+   * 
+   * @param Given layer of type IdentityLayer.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(IdentityLayer<>* /*layer*/) const
+  {
+    return "identity";
+  }
+
+  /*
+   * Return the name of the given layer of type TanHLayer as a string.
+   * 
+   * @param Given layer of type TanHLayer.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(TanHLayer<>* /*layer*/) const
+  {
+    return "tanh";
+  }
+
+  /*
+   * Return the name of the given layer of type ELU as a string.
+   * 
+   * @param Given layer of type ELU.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(ELU<>* /*layer*/) const
+  {
+    return "elu";
+  }
+
+  /*
+   * Return the name of the given layer of type HardTanH as a string.
+   * 
+   * @param Given layer of type HardTanH.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(HardTanH<>* /*layer*/) const
+  {
+    return "hardtanh";
+  }
+
+  /*
+   * Return the name of the given layer of type LeakyReLU as a string.
+   * 
+   * @param Given layer of type LeakyReLU.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(LeakyReLU<>* /*layer*/) const
+  {
+    return "leakyrelu";
+  }
+
+  /*
+   * Return the name of the given layer of type PReLU as a string.
+   * 
+   * @param Given layer of type PReLU.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(PReLU<>* /*layer*/) const
+  {
+    return "prelu";
+  }
+
+  /*
+   * Return the name of the given layer of type SigmoidLayer as a string.
+   * 
+   * @param Given layer of type SigmoidLayer.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(SigmoidLayer<>* /*layer*/) const
+  {
+    return "sigmoid";
+  }
+
+  /*
+   * Return the name of the given layer of type LogSoftMax as a string.
+   * 
+   * @param Given layer of type LogSoftMax.
+   * @return The string representation of the layer.
+   */
+  std::string LayerString(LogSoftMax<>* /*layer*/) const
+  {
+    return "logsoftmax";
+  }
+
+  /*
+   * Return the name of the layer of specified type as a string.
+   * 
+   * @param Given layer of any type.
+   * @return A string declaring that the layer is unsupported.
+   */
+  template<typename T>
+  std::string LayerString(T* /*layer*/) const
+  {
+    return "unsupported";
+  }
+
+  //! Overload function call.
+  std::string operator()(MoreTypes layer) const
+  {
+    return layer.apply_visitor(*this);
+  }
+
+  //! Overload function call.
+  template<typename LayerType>
+  std::string operator()(LayerType* layer) const
+  {
+    return LayerString(layer);
+  }
+};

src/mlpack/methods/ann/rnn.hpp

@@ -91,15 +91,18 @@ class RNN
    * at time slice k.
    *
    * @tparam OptimizerType Type of optimizer to use to train the model.
+   * @tparam CallbackTypes Types of Callback Functions.
    * @param predictors Input training variables.
    * @param responses Outputs results from input training variables.
    * @param optimizer Instantiated optimizer used to train the model.
+   * @param callbacks Callback functions.
    * @return The final objective of the trained model (NaN or Inf on error).
    */
-  template<typename OptimizerType>
+  template<typename OptimizerType, typename... CallbackTypes>
   double Train(arma::cube predictors,
                arma::cube responses,
-               OptimizerType& optimizer);
+               OptimizerType& optimizer,
+               CallbackTypes&&... callbacks);
 
   /**
    * Train the recurrent neural network on the given input data. By default, the
@@ -121,12 +124,16 @@ class RNN
    * at time slice k.
    *
    * @tparam OptimizerType Type of optimizer to use to train the model.
+   * @tparam CallbackTypes Types of Callback Functions.
    * @param predictors Input training variables.
    * @param responses Outputs results from input training variables.
+   * @param callbacks Callback functions.
    * @return The final objective of the trained model (NaN or Inf on error).
    */
-  template<typename OptimizerType = ens::StandardSGD>
-  double Train(arma::cube predictors, arma::cube responses);
+  template<typename OptimizerType = ens::StandardSGD, typename... CallbackTypes>
+  double Train(arma::cube predictors,
+               arma::cube responses,
+               CallbackTypes&&... callbacks);
 
   /**
    * Predict the responses to a given set of predictors. The responses will

src/mlpack/methods/ann/rnn_impl.hpp

@@ -63,11 +63,12 @@ RNN<OutputLayerType, InitializationRuleType, CustomLayers...>::~RNN()
 
 template<typename OutputLayerType, typename InitializationRuleType,
          typename... CustomLayers>
-template<typename OptimizerType>
+template<typename OptimizerType, typename... CallbackTypes>
 double RNN<OutputLayerType, InitializationRuleType, CustomLayers...>::Train(
     arma::cube predictors,
     arma::cube responses,
-    OptimizerType& optimizer)
+    OptimizerType& optimizer,
+    CallbackTypes&&... callbacks)
 {
   numFunctions = responses.n_cols;
 
@@ -84,7 +85,7 @@ double RNN<OutputLayerType, InitializationRuleType, CustomLayers...>::Train(
 
   // Train the model.
   Timer::Start("rnn_optimization");
-  const double out = optimizer.Optimize(*this, parameter);
+  const double out = optimizer.Optimize(*this, parameter, callbacks...);
   Timer::Stop("rnn_optimization");
 
   Log::Info << "RNN::RNN(): final objective of trained model is " << out
@@ -105,10 +106,11 @@ void RNN<OutputLayerType, InitializationRuleType,
 
 template<typename OutputLayerType, typename InitializationRuleType,
          typename... CustomLayers>
-template<typename OptimizerType>
+template<typename OptimizerType, typename... CallbackTypes>
 double RNN<OutputLayerType, InitializationRuleType, CustomLayers...>::Train(
     arma::cube predictors,
-    arma::cube responses)
+    arma::cube responses,
+    CallbackTypes&&... callbacks)
 {
   numFunctions = responses.n_cols;
 
@@ -127,7 +129,7 @@ double RNN<OutputLayerType, InitializationRuleType, CustomLayers...>::Train(
 
   // Train the model.
   Timer::Start("rnn_optimization");
-  const double out = optimizer.Optimize(*this, parameter);
+  const double out = optimizer.Optimize(*this, parameter, callbacks...);
   Timer::Stop("rnn_optimization");
 
   Log::Info << "RNN::RNN(): final objective of trained model is " << out

src/mlpack/methods/linear_svm/CMakeLists.txt

@@ -16,3 +16,7 @@ endforeach()
 # append sources (with directory name) to list of all mlpack sources (used at
 # the parent scope)
 set(MLPACK_SRCS ${MLPACK_SRCS} ${DIR_SRCS} PARENT_SCOPE)
+
+add_cli_executable(linear_svm)
+add_python_binding(linear_svm)
+add_markdown_docs(linear_svm "cli;python" "classification")

src/mlpack/methods/linear_svm/linear_svm.hpp

@@ -119,6 +119,20 @@ class LinearSVM
             const double lambda = 0.0001,
             const double delta = 1.0,
             const bool fitIntercept = false);
+  /**
+   * Initialize the Linear SVM without performing training.  Default
+   * value of lambda is 0.0001.  Be sure to use Train() before calling
+   * Classify() or ComputeAccuracy(), otherwise the results may be meaningless.
+   *
+   * @param numClasses Number of classes for classification.
+   * @param lambda L2-regularization constant.
+   * @paran delta Margin of difference between correct class and other classes.
+   * @param fitIntercept add intercept term or not.
+   */
+  LinearSVM(const size_t numClasses = 0,
+            const double lambda = 0.0001,
+            const double delta = 1.0,
+            const bool fitIntercept = false);
 
   /**
    * Classify the given points, returning the predicted labels for each point.
@@ -207,6 +221,14 @@ class LinearSVM
   //! Gets the regularization parameter.
   double Lambda() const { return lambda; }
 
+  //! Sets the margin between the correct class and all other classes.
+  double& Delta() { return delta; }
+  //! Gets the margin between the correct class and all other classes.
+  double Delta() const { return delta; }
+
+  //! Sets the intercept term flag.
+  bool& FitIntercept() { return fitIntercept; }
+
   //! Set the model parameters.
   arma::mat& Parameters() { return parameters; }
   //! Get the model parameters.

src/mlpack/methods/linear_svm/linear_svm_function_impl.hpp

@@ -320,7 +320,7 @@ void LinearSVMFunction<MatType>::Gradient(
   {
     scores = parameters.rows(0, dataset.n_rows - 1).t()
         * dataset.cols(firstId, lastId)
-        + arma::repmat(parameters.row(dataset.n_rows).t(), 1, dataset.n_cols);
+        + arma::repmat(parameters.row(dataset.n_rows).t(), 1, batchSize);
   }
 
   arma::mat margin = scores - (arma::repmat(arma::ones(numClasses).t()

src/mlpack/methods/linear_svm/linear_svm_impl.hpp

@@ -52,6 +52,20 @@ LinearSVM<MatType>::LinearSVM(
       numClasses, fitIntercept);
 }
 
+template <typename MatType>
+LinearSVM<MatType>::LinearSVM(
+    const size_t numClasses,
+    const double lambda,
+    const double delta,
+    const bool fitIntercept) :
+    numClasses(numClasses),
+    lambda(lambda),
+    delta(delta),
+    fitIntercept(fitIntercept)
+{
+  // No training to do here.
+}
+
 template <typename MatType>
 template <typename OptimizerType>
 double LinearSVM<MatType>::Train(
@@ -60,6 +74,11 @@ double LinearSVM<MatType>::Train(
     const size_t numClasses,
     OptimizerType optimizer)
 {
+  if (numClasses <= 1)
+  {
+    throw std::invalid_argument("LinearSVM dataset has 0 number of classes!");
+  }
+
   LinearSVMFunction<MatType> svm(data, labels, numClasses, lambda, delta,
       fitIntercept);
   if (parameters.is_empty())

src/mlpack/methods/linear_svm/linear_svm_main.cpp

@@ -0,0 +1,449 @@
+/**
+ * @file linear_svm_main.cpp
+ * @author Yashwant Singh Parihar
+ *
+ * Main executable for linear svm.
+ *
+ * mlpack is free software; you may redistribute it and/or modify it under the
+ * terms of the 3-clause BSD license.  You should have received a copy of the
+ * 3-clause BSD license along with mlpack.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#include <mlpack/prereqs.hpp>
+#include <mlpack/core/util/cli.hpp>
+#include <mlpack/core.hpp>
+#include <mlpack/core/util/mlpack_main.hpp>
+
+#include "linear_svm.hpp"
+
+#include <ensmallen.hpp>
+
+using namespace std;
+using namespace mlpack;
+using namespace mlpack::svm;
+using namespace mlpack::util;
+
+PROGRAM_INFO("Linear SVM is an L2-regularized support vector machine.",
+    // Short description.
+    "An implementation of linear SVM for multiclass classification. "
+    "Given labeled data, a model can be trained and saved for "
+    "future use; or, a pre-trained model can be used to classify new points.",
+    // Long description.
+    "An implementation of linear SVMs that uses either L-BFGS or parallel SGD"
+    " (stochastic gradient descent) to train the model."
+    "\n\n"
+    "This program allows loading a linear SVM model (via the " +
+    PRINT_PARAM_STRING("input_model") + " parameter) "
+    "or training a linear SVM model given training data (specified "
+    "with the " + PRINT_PARAM_STRING("training") + " parameter), or both "
+    "those things at once.  In addition, this program allows classification on "
+    "a test dataset (specified with the " + PRINT_PARAM_STRING("test") + " "
+    "parameter) and the classification results may be saved with the " +
+    PRINT_PARAM_STRING("predictions") + " output parameter."
+    " The trained linear SVM model may be saved using the " +
+    PRINT_PARAM_STRING("output_model") + " output parameter."
+    "\n\n"
+    "The training data, if specified, may have class labels as its last "
+    "dimension.  Alternately, the " + PRINT_PARAM_STRING("labels") + " "
+    "parameter may be used to specify a separate vector of labels."
+    "\n\n"
+    "When a model is being trained, there are many options.  L2 regularization "
+    "(to prevent overfitting) can be specified with the " +
+    PRINT_PARAM_STRING("lambda") + " option, and the number of classes can be "
+    "manually specified with the " + PRINT_PARAM_STRING("num_classes") +
+    "and if an intercept term is not desired in the model, the " +
+    PRINT_PARAM_STRING("no_intercept") + " parameter can be specified."
+    "Margin of difference between correct class and other classes can "
+    "be specified with the " + PRINT_PARAM_STRING("delta") + " option."
+    "The optimizer used to train the model can be specified with the " +
+    PRINT_PARAM_STRING("optimizer") + " parameter.  Available options are "
+    "'psgd' (parallel stochastic gradient descent) and 'lbfgs' (the L-BFGS"
+    " optimizer).  There are also various parameters for the optimizer; the " +
+    PRINT_PARAM_STRING("max_iterations") + " parameter specifies the maximum "
+    "number of allowed iterations, and the " +
+    PRINT_PARAM_STRING("tolerance") + " parameter specifies the tolerance for "
+    "convergence.  For the parallel SGD optimizer, the " +
+    PRINT_PARAM_STRING("step_size") + " parameter controls the step size taken "
+    "at each iteration by the optimizer and the maximum number of epochs "
+    "(specified with " + PRINT_PARAM_STRING("epochs") + "). If the "
+    "objective function for your data is oscillating between Inf and 0, the "
+    "step size is probably too large.  There are more parameters for the "
+    "optimizers, but the C++ interface must be used to access these."
+    "\n\n"
+    "Optionally, the model can be used to predict the labels for another "
+    "matrix of data points, if " + PRINT_PARAM_STRING("test") + " is "
+    "specified.  The " + PRINT_PARAM_STRING("test") + " parameter can be "
+    "specified without the " + PRINT_PARAM_STRING("training") + " parameter, "
+    "so long as an existing linear SVM model is given with the " +
+    PRINT_PARAM_STRING("input_model") + " parameter.  The output predictions "
+    "from the linear SVM model may be saved with the " +
+    PRINT_PARAM_STRING("predictions") + " parameter." +
+    "\n\n"
+    "As an example, to train a LinaerSVM on the data '" +
+    PRINT_DATASET("data") + "' with labels '" + PRINT_DATASET("labels") + "' "
+    "with L2 regularization of 0.1, saving the model to '" +
+    PRINT_MODEL("lsvm_model") + "', the following command may be used:"
+    "\n\n" +
+    PRINT_CALL("linear_svm", "training", "data", "labels", "labels",
+        "lambda", 0.1, "delta", 1.0, "num_classes", 0,
+        "output_model", "lsvm_model") +
+    "\n\n"
+    "Then, to use that model to predict classes for the dataset '" +
+    PRINT_DATASET("test") + "', storing the output predictions in '" +
+    PRINT_DATASET("predictions") + "', the following command may be used: "
+    "\n\n" +
+    PRINT_CALL("linear_svm", "input_model", "lsvm_model", "test", "test",
+        "predictions", "predictions"),
+    SEE_ALSO("@random_forest", "#random_forest"),
+    SEE_ALSO("@logistic_regression", "#logistic_regression"),
+    SEE_ALSO("LinearSVM on Wikipedia",
+        "https://en.wikipedia.org/wiki/Support-vector_machine"),
+    SEE_ALSO("mlpack::svm::LinearSVM C++ class documentation",
+        "@doxygen/classmlpack_1_1svm_1_1LinearSVM.html"));
+
+// Training parameters.
+PARAM_MATRIX_IN("training", "A matrix containing the training set (the matrix "
+    "of predictors, X).", "t");
+PARAM_UROW_IN("labels", "A matrix containing labels (0 or 1) for the points "
+    "in the training set (y).", "l");
+
+// Optimizer parameters.
+PARAM_DOUBLE_IN("lambda", "L2-regularization parameter for training.", "r",
+    0.0001);
+PARAM_DOUBLE_IN("delta", "Margin of difference between correct class and other "
+    "classes.", "d", 1.0);
+PARAM_INT_IN("num_classes", "Number of classes for classification; if "
+    "unspecified (or 0), the number of classes found in the labels will be "
+    "used.", "c", 0);
+PARAM_FLAG("no_intercept", "Do not add the intercept term to the model.", "N");
+PARAM_STRING_IN("optimizer", "Optimizer to use for training ('lbfgs' or "
+    "'psgd').", "O", "lbfgs");
+PARAM_DOUBLE_IN("tolerance", "Convergence tolerance for optimizer.", "e",
+    1e-10);
+PARAM_INT_IN("max_iterations", "Maximum iterations for optimizer (0 indicates "
+    "no limit).", "n", 10000);
+PARAM_DOUBLE_IN("step_size", "Step size for parallel SGD optimizer.",
+    "a", 0.01);
+PARAM_FLAG("shuffle", "Don't shuffle the order in which data points are "
+    "visited for parallel SGD.", "S");
+PARAM_INT_IN("epochs", "Maximum number of full epochs over dataset for "
+    "psgd", "E", 50);
+PARAM_INT_IN("seed", "Random seed.  If 0, 'std::time(NULL)' is used.", "s", 0);
+
+class LinearSVMModel
+{
+ public:
+  arma::Col<size_t> mappings;
+  LinearSVM<> svm;
+
+  template<typename Archive>
+  void serialize(Archive& ar, const unsigned int /* version */)
+  {
+    ar & BOOST_SERIALIZATION_NVP(mappings);
+    ar & BOOST_SERIALIZATION_NVP(svm);
+  }
+};
+
+
+// Model loading/saving.
+PARAM_MODEL_IN(LinearSVMModel, "input_model", "Existing model "
+    "(parameters).", "m");
+PARAM_MODEL_OUT(LinearSVMModel, "output_model", "Output for trained "
+    "linear svm model.", "M");
+
+// Testing.
+PARAM_MATRIX_IN("test", "Matrix containing test dataset.", "T");
+PARAM_UROW_IN("test_labels", "Matrix containing test labels.", "L");
+PARAM_UROW_OUT("predictions", "If test data is specified, this matrix is where "
+    "the predictions for the test set will be saved.", "P");
+PARAM_MATRIX_OUT("probabilities", "If test data is specified, this "
+    "matrix is where the class probabilities for the test set will be saved.",
+    "p");
+
+static void mlpackMain()
+{
+  if (CLI::GetParam<int>("seed") != 0)
+    math::RandomSeed((size_t) CLI::GetParam<int>("seed"));
+  else
+    math::RandomSeed((size_t) std::time(NULL));
+
+  // Collect command-line options.
+  const double lambda = CLI::GetParam<double>("lambda");
+  const double delta = CLI::GetParam<double>("delta");
+  const string optimizerType = CLI::GetParam<string>("optimizer");
+  const double tolerance = CLI::GetParam<double>("tolerance");
+  const bool intercept = CLI::HasParam("no_intercept");
+  const size_t epochs = (size_t) CLI::GetParam<int>("epochs");
+  const size_t maxIterations = (size_t) CLI::GetParam<int>("max_iterations");
+
+  // One of training and input_model must be specified.
+  RequireAtLeastOnePassed({ "training", "input_model" }, true);
+
+  // If no output file is given, the user should know that the model will not be
+  // saved, but only if a model is being trained.
+  RequireAtLeastOnePassed({ "output_model", "predictions", "probabilities"},
+      false, "no output will be saved");
+
+  ReportIgnoredParam({{ "test", false }}, "predictions");
+  ReportIgnoredParam({{ "test", false }}, "probabilities");
+  ReportIgnoredParam({{ "test", false }}, "test_labels");
+
+  // Max Iterations needs to be positive.
+  RequireParamValue<int>("max_iterations", [](int x) { return x >= 0; },
+      true, "max_iterations must be non-negative");
+
+  // Tolerance needs to be positive.
+  RequireParamValue<double>("tolerance", [](double x) { return x >= 0.0; },
+      true, "tolerance must be non-negative");
+
+  // Optimizer has to be L-BFGS or parallel SGD.
+  RequireParamInSet<string>("optimizer", { "lbfgs", "psgd" },
+      true, "unknown optimizer");
+
+  // Epochs needs to be non-negative.
+  RequireParamValue<int>("epochs", [](int x) { return x >= 0; }, true,
+      "maximum number of epochs must be non-negative");
+
+  if (optimizerType != "psgd")
+  {
+    if (CLI::HasParam("step_size"))
+    {
+      Log::Warn << PRINT_PARAM_STRING("step_size") << " ignored because "
+          << "optimizer type is not 'psgd'." << std::endl;
+    }
+    if (CLI::HasParam("shuffle"))
+    {
+      Log::Warn << PRINT_PARAM_STRING("shuffle") << " ignored because "
+          << "optimizer type is not 'psgd'." << std::endl;
+    }
+    if (CLI::HasParam("epochs"))
+    {
+      Log::Warn << PRINT_PARAM_STRING("epochs") << " ignored because "
+          << "optimizer type is not 'psgd'." << std::endl;
+    }
+  }
+
+  if (optimizerType != "lbfgs")
+  {
+    if (CLI::HasParam("max_iterations"))
+    {
+      Log::Warn << PRINT_PARAM_STRING("max_iterations") << " ignored because "
+          << "optimizer type is not 'lbfgs'." << std::endl;
+    }
+  }
+
+  // Step Size must be positive.
+  RequireParamValue<double>("step_size", [](double x) { return x > 0.0; },
+      true, "step size must be positive");
+
+  // Lambda must be positive.
+  RequireParamValue<double>("lambda", [](double x) { return x >= 0.0; },
+      true, "lambda must be non-negative");
+
+  // Number of Classes must be Non-Negative
+  RequireParamValue<int>("num_classes", [](int x) { return x >= 0; },
+                         true, "number of classes must be greater than or "
+                         "equal to 0 (equal to 0 in case of unspecified.)");
+
+  // Delta must be positive.
+  RequireParamValue<double>("delta", [](double x) { return x >= 0.0; }, true,
+      "delta must be non-negative");
+
+  // Delta must be positive.
+  RequireParamValue<int>("epochs", [](int x) { return x > 0; }, true,
+      "epochs must be non-negative");
+
+  // These are the matrices we might use.
+  arma::mat trainingSet;
+  arma::Row<size_t> labels;
+  arma::Row<size_t> rawLabels;
+  arma::mat testSet;
+  arma::Row<size_t> predictedLabels;
+  size_t numClasses;
+
+  // Load data matrix.
+  if (CLI::HasParam("training"))
+    trainingSet = std::move(CLI::GetParam<arma::mat>("training"));
+
+  // Check if the labels are in a separate file.
+  if (CLI::HasParam("training") && CLI::HasParam("labels"))
+  {
+    rawLabels = std::move(CLI::GetParam<arma::Row<size_t>>("labels"));
+    if (trainingSet.n_cols != rawLabels.n_cols)
+    {
+      Log::Fatal << "The labels must have the same number of points as the "
+          << "training dataset." << endl;
+    }
+  }
+  else if (CLI::HasParam("training"))
+  {
+    // Checking the size of training data if no labels are passed.
+    if (trainingSet.n_rows < 2)
+    {
+      Log::Fatal << "Can't get labels from training data since it has less "
+          << "than 2 rows." << endl;
+    }
+
+    // The initial predictors for y, Nx1.
+    rawLabels = arma::conv_to<arma::Row<size_t>>::from(
+        trainingSet.row(trainingSet.n_rows - 1));
+    trainingSet.shed_row(trainingSet.n_rows - 1);
+  }
+
+  // Load the model, if necessary.
+  LinearSVMModel* model;
+  if (CLI::HasParam("input_model"))
+  {
+    model = CLI::GetParam<LinearSVMModel*>("input_model");
+  }
+  else
+  {
+    model = new LinearSVMModel();
+  }
+
+  // Now, do the training.
+  if (CLI::HasParam("training"))
+  {
+    data::NormalizeLabels(rawLabels, labels, model->mappings);
+    numClasses = CLI::GetParam<int>("num_classes") == 0 ?
+        model->mappings.n_elem : CLI::GetParam<int>("num_classes");
+    model->svm.Lambda() = lambda;
+    model->svm.Delta() = delta;
+    model->svm.NumClasses() = numClasses;
+    model->svm.FitIntercept() = intercept;
+
+    if (optimizerType == "lbfgs")
+    {
+      ens::L_BFGS lbfgsOpt;
+      lbfgsOpt.MaxIterations() = maxIterations;
+      lbfgsOpt.MinGradientNorm() = tolerance;
+
+      Log::Info << "Training model with L-BFGS optimizer." << endl;
+
+      // This will train the model.
+      model->svm.Train(trainingSet, labels, numClasses, lbfgsOpt);
+    }
+    else if (optimizerType == "psgd")
+    {
+      const double stepSize = CLI::GetParam<double>("step_size");
+      const bool shuffle = !CLI::HasParam("shuffle");
+      const size_t maxIt = epochs * trainingSet.n_cols;
+
+      ens::ConstantStep decayPolicy(stepSize);
+
+      #ifdef HAS_OPENMP
+      size_t threads = omp_get_max_threads();
+      #else
+      size_t threads = 1;
+      Log::Warn << "Using parallel SGD, but OpenMP support is "
+                << "not available!" << endl;
+      #endif
+
+      ens::ParallelSGD<ens::ConstantStep> psgdOpt(maxIt, std::ceil(
+        (float) trainingSet.n_cols / threads), tolerance, shuffle,
+        decayPolicy);
+
+      Log::Info << "Training model with ParallelSGD optimizer." << endl;
+
+      // This will train the model.
+      model->svm.Train(trainingSet, labels, numClasses, psgdOpt);
+    }
+  }
+  if (CLI::HasParam("test"))
+  {
+    if (!CLI::HasParam("training"))
+    {
+      numClasses = model->svm.NumClasses();
+    }
+    // Get the test dataset, and get predictions.
+    testSet = std::move(CLI::GetParam<arma::mat>("test"));
+    arma::Row<size_t> predictions;
+    size_t trainingDimensionality;
+
+    // Set the dimensionality according to fitintercept.
+    if (intercept)
+      trainingDimensionality = model->svm.Parameters().n_rows - 1;
+    else
+      trainingDimensionality = model->svm.Parameters().n_rows;
+
+    // Checking the dimensionality of the test data.
+    if (testSet.n_rows != trainingDimensionality)
+    {
+      Log::Fatal << "Test data dimensionality (" << testSet.n_rows << ") must "
+          << "be the same as the dimensionality of the training data ("
+          << trainingDimensionality << ")!" << endl;
+    }
+
+    // Save class probabilities, if desired.
+    if (CLI::HasParam("probabilities"))
+    {
+      Log::Info << "Calculating class probabilities of points in "
+          << CLI::GetPrintableParam<arma::mat>("test") << "." << endl;
+      arma::mat probabilities;
+      model->svm.Classify(testSet, probabilities);
+      CLI::GetParam<arma::mat>("probabilities") = std::move(probabilities);
+    }
+
+    model->svm.Classify(testSet, predictedLabels);
+    data::RevertLabels(predictedLabels, model->mappings, predictions);
+
+    // Calculate accuracy, if desired.
+    if (CLI::HasParam("test_labels"))
+    {
+      arma::Row<size_t> testLabels;
+      arma::Row<size_t> testRawLabels =
+          std::move(CLI::GetParam<arma::Row<size_t>>("test_labels"));
+
+      data::NormalizeLabels(testRawLabels, testLabels, model->mappings);
+
+      if (testSet.n_cols != testLabels.n_elem)
+      {
+        Log::Fatal << "Test data given with " << PRINT_PARAM_STRING("test")
+            << " has " << testSet.n_cols << " points, but labels in "
+            << PRINT_PARAM_STRING("test_labels") << " have "
+            << testLabels.n_elem << " labels!" << endl;
+      }
+
+      numClasses = CLI::GetParam<int>("num_classes") == 0 ?
+          model->mappings.n_elem : CLI::GetParam<int>("num_classes");
+      arma::Col<size_t> correctClassCounts;
+      arma::Col<size_t> labelSize;
+      correctClassCounts.zeros(numClasses);
+      labelSize.zeros(numClasses);
+
+      for (arma::uword i = 0; i != predictions.n_elem; ++i)
+      {
+        if (predictions(i) == testLabels(i))
+        {
+          ++correctClassCounts[testLabels(i)];
+        }
+        ++labelSize[testLabels(i)];
+      }
+
+      size_t totalCorrectClass = 0;
+      for (size_t i = 0; i != correctClassCounts.size(); ++i)
+      {
+        Log::Info << "Accuracy for points with label " << i << " is "
+            << (correctClassCounts[i] / static_cast<double>(labelSize[i]))
+            << " (" << correctClassCounts[i] << " of " << labelSize[i] << ")."
+            << endl;
+        totalCorrectClass += correctClassCounts[i];
+      }
+
+      Log::Info << "Total accuracy for all points is "
+          << (totalCorrectClass) / static_cast<double>(predictions.n_elem)
+          << " (" << totalCorrectClass << " of " << predictions.n_elem << ")."
+          << endl;
+    }
+
+    // Save predictions, if desired.
+    if (CLI::HasParam("predictions"))
+    {
+      Log::Info << "Predicting classes of points in '"
+          << CLI::GetPrintableParam<arma::mat>("test") << "'." << endl;
+      CLI::GetParam<arma::Row<size_t>>("predictions") = std::move(predictions);
+    }
+  }
+
+  CLI::GetParam<LinearSVMModel*>("output_model") = model;
+}

src/mlpack/methods/logistic_regression/logistic_regression.hpp

@@ -123,13 +123,16 @@ class LogisticRegression
    * parameters vector directly with Parameters() and modify it as desired.
    *
    * @tparam OptimizerType Type of optimizer to use to train the model.
+   * @tparam CallbackTypes Types of Callback Functions.
    * @param predictors Input training variables.
    * @param responses Outputs results from input training variables.
+   * @param callbacks Callback Functions.
    * @return The final objective of the trained model (NaN or Inf on error)
    */
-  template<typename OptimizerType = ens::L_BFGS>
+  template<typename OptimizerType = ens::L_BFGS, typename... CallbackTypes>
   double Train(const MatType& predictors,
-               const arma::Row<size_t>& responses);
+               const arma::Row<size_t>& responses,
+               CallbackTypes&&... callbacks);
 
   /**
    * Train the LogisticRegression model with the given instantiated optimizer.
@@ -143,15 +146,19 @@ class LogisticRegression
    * optimizer.Function().GetInitialPoint() to the current parameters vector,
    * accessible via Parameters().
    *
+   * @tparam OptimizerType Type of optimizer to use to train the model.
+   * @tparam CallbackTypes Types of Callback Functions.
    * @param predictors Input training variables.
    * @param responses Outputs results from input training variables.
    * @param optimizer Instantiated optimizer with instantiated error function.
+   * @param callbacks Callback Functions.
    * @return The final objective of the trained model (NaN or Inf on error)
    */
-  template<typename OptimizerType>
+  template<typename OptimizerType, typename... CallbackTypes>
   double Train(const MatType& predictors,
                const arma::Row<size_t>& responses,
-               OptimizerType& optimizer);
+               OptimizerType& optimizer,
+               CallbackTypes&&... callbacks);
 
   //! Return the parameters (the b vector).
   const arma::rowvec& Parameters() const { return parameters; }

src/mlpack/methods/logistic_regression/logistic_regression_impl.hpp

@@ -67,20 +67,23 @@ LogisticRegression<MatType>::LogisticRegression(
 }
 
 template<typename MatType>
-template<typename OptimizerType>
-double LogisticRegression<MatType>::Train(const MatType& predictors,
-                                        const arma::Row<size_t>& responses)
+template<typename OptimizerType, typename... CallbackTypes>
+double LogisticRegression<MatType>::Train(
+        const MatType& predictors,
+        const arma::Row<size_t>& responses,
+        CallbackTypes&&... callbacks)
 {
   OptimizerType optimizer;
-  return Train(predictors, responses, optimizer);
+  return Train(predictors, responses, optimizer, callbacks...);
 }
 
 template<typename MatType>
-template<typename OptimizerType>
+template<typename OptimizerType, typename... CallbackTypes>
 double LogisticRegression<MatType>::Train(
     const MatType& predictors,
     const arma::Row<size_t>& responses,
-    OptimizerType& optimizer)
+    OptimizerType& optimizer,
+    CallbackTypes&&... callbacks)
 {
   LogisticRegressionFunction<MatType> errorFunction(predictors,
                                                     responses,
@@ -88,7 +91,8 @@ double LogisticRegression<MatType>::Train(
   errorFunction.InitialPoint() = parameters;
 
   Timer::Start("logistic_regression_optimization");
-  const double out = optimizer.Optimize(errorFunction, parameters);
+  const double out = optimizer.Optimize(errorFunction, parameters,
+      callbacks...);
   Timer::Stop("logistic_regression_optimization");
 
   Log::Info << "LogisticRegression::LogisticRegression(): final objective of "

src/mlpack/tests/CMakeLists.txt

@@ -16,6 +16,7 @@ add_executable(mlpack_test
   bias_svd_test.cpp
   binarize_test.cpp
   block_krylov_svd_test.cpp
+  callback_test.cpp
   cf_test.cpp
   cli_binding_test.cpp
   cli_test.cpp
@@ -52,6 +53,7 @@ add_executable(mlpack_test
   krann_search_test.cpp
   ksinit_test.cpp
   lars_test.cpp
+  layer_names_test.cpp
   lin_alg_test.cpp
   linear_regression_test.cpp
   linear_svm_test.cpp
@@ -134,6 +136,7 @@ add_executable(mlpack_test
   main_tests/kfn_test.cpp
   main_tests/knn_test.cpp
   main_tests/linear_regression_test.cpp
+  main_tests/linear_svm_test.cpp
   main_tests/logistic_regression_test.cpp
   main_tests/local_coordinate_coding_test.cpp
   main_tests/lmnn_test.cpp

src/mlpack/tests/ann_layer_test.cpp

@@ -1726,8 +1726,8 @@ BOOST_AUTO_TEST_CASE(SimpleTransposedConvolutionLayerTest)
 {
   arma::mat output, input, delta;
 
-  TransposedConvolution<> module1(1, 1, 3, 3, 1, 1, 0, 0, 4, 4);
-  // Test the Forward function.
+  TransposedConvolution<> module1(1, 1, 3, 3, 1, 1, 0, 0, 4, 4, 6, 6);
+  // Test the forward function.
   input = arma::linspace<arma::colvec>(0, 15, 16);
   module1.Parameters() = arma::mat(9 + 1, 1, arma::fill::zeros);
   module1.Parameters()(0) = 1.0;
@@ -1737,11 +1737,12 @@ BOOST_AUTO_TEST_CASE(SimpleTransposedConvolutionLayerTest)
   // Value calculated using tensorflow.nn.conv2d_transpose()
   BOOST_REQUIRE_EQUAL(arma::accu(output), 360.0);
 
-  // Test the Backward function.
+  // Test the backward function.
   module1.Backward(std::move(input), std::move(output), std::move(delta));
-  BOOST_REQUIRE_EQUAL(arma::accu(delta), 720);
+  // Value calculated using tensorflow.nn.conv2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(delta), 720.0);
 
-  TransposedConvolution<> module2(1, 1, 4, 4, 1, 1, 2, 2, 5, 5);
+  TransposedConvolution<> module2(1, 1, 4, 4, 1, 1, 1, 1, 5, 5, 6, 6);
   // Test the forward function.
   input = arma::linspace<arma::colvec>(0, 24, 25);
   module2.Parameters() = arma::mat(16 + 1, 1, arma::fill::zeros);
@@ -1753,14 +1754,15 @@ BOOST_AUTO_TEST_CASE(SimpleTransposedConvolutionLayerTest)
   module2.Parameters()(15) = 2.0;
   module2.Reset();
   module2.Forward(std::move(input), std::move(output));
-  // Value calculated using tensorflow.nn.conv2d_transpose()
-  BOOST_REQUIRE_EQUAL(arma::accu(output), 2100.0);
+  // Value calculated using torch.nn.functional.conv_transpose2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(output), 1512.0);
 
   // Test the backward function.
   module2.Backward(std::move(input), std::move(output), std::move(delta));
-  BOOST_REQUIRE_EQUAL(arma::accu(delta), 7740);
+  // Value calculated using torch.nn.functional.conv2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(delta), 6504.0);
 
-  TransposedConvolution<> module3(1, 1, 3, 3, 1, 1, 1, 1, 5, 5);
+  TransposedConvolution<> module3(1, 1, 3, 3, 1, 1, 1, 1, 5, 5, 5, 5);
   // Test the forward function.
   input = arma::linspace<arma::colvec>(0, 24, 25);
   module3.Parameters() = arma::mat(9 + 1, 1, arma::fill::zeros);
@@ -1770,14 +1772,15 @@ BOOST_AUTO_TEST_CASE(SimpleTransposedConvolutionLayerTest)
   module3.Parameters()(8) = 1.0;
   module3.Reset();
   module3.Forward(std::move(input), std::move(output));
-  // Value calculated using tensorflow.nn.conv2d_transpose()
-  BOOST_REQUIRE_EQUAL(arma::accu(output), 3000.0);
+  // Value calculated using torch.nn.functional.conv_transpose2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(output), 2370.0);
 
   // Test the backward function.
   module3.Backward(std::move(input), std::move(output), std::move(delta));
-  BOOST_REQUIRE_EQUAL(arma::accu(delta), 21480);
+  // Value calculated using torch.nn.functional.conv2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(delta), 19154.0);
 
-  TransposedConvolution<> module4(1, 1, 3, 3, 1, 1, 2, 2, 5, 5);
+  TransposedConvolution<> module4(1, 1, 3, 3, 1, 1, 0, 0, 5, 5, 7, 7);
   // Test the forward function.
   input = arma::linspace<arma::colvec>(0, 24, 25);
   module4.Parameters() = arma::mat(9 + 1, 1, arma::fill::zeros);
@@ -1787,33 +1790,35 @@ BOOST_AUTO_TEST_CASE(SimpleTransposedConvolutionLayerTest)
   module4.Parameters()(8) = 8.0;
   module4.Reset();
   module4.Forward(std::move(input), std::move(output));
-  // Value calculated using tensorflow.nn.conv2d_transpose()
+  // Value calculated using torch.nn.functional.conv_transpose2d()
   BOOST_REQUIRE_EQUAL(arma::accu(output), 6000.0);
 
   // Test the backward function.
   module4.Backward(std::move(input), std::move(output), std::move(delta));
-  BOOST_REQUIRE_EQUAL(arma::accu(delta), 86208);
+  // Value calculated using torch.nn.functional.conv2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(delta), 86208.0);
 
-  TransposedConvolution<> module5(1, 1, 3, 3, 2, 2, 0, 0, 5, 5);
+  TransposedConvolution<> module5(1, 1, 3, 3, 2, 2, 0, 0, 2, 2, 5, 5);
   // Test the forward function.
-  input = arma::linspace<arma::colvec>(0, 24, 25);
-  module5.Parameters() = arma::mat(9 + 1, 1, arma::fill::zeros);
+  input = arma::linspace<arma::colvec>(0, 3, 4);
+  module5.Parameters() = arma::mat(25 + 1, 1, arma::fill::zeros);
   module5.Parameters()(2) = 8.0;
   module5.Parameters()(4) = 6.0;
   module5.Parameters()(6) = 4.0;
   module5.Parameters()(8) = 2.0;
   module5.Reset();
   module5.Forward(std::move(input), std::move(output));
-  // Value calculated using tensorflow.nn.conv2d_transpose()
-  BOOST_REQUIRE_EQUAL(arma::accu(output), 6000.0);
+  // Value calculated using torch.nn.functional.conv_transpose2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(output), 120.0);
 
   // Test the backward function.
   module5.Backward(std::move(input), std::move(output), std::move(delta));
-  BOOST_REQUIRE_EQUAL(arma::accu(delta), 83808);
+  // Value calculated using torch.nn.functional.conv2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(delta), 960.0);
 
-  TransposedConvolution<> module6(1, 1, 3, 3, 2, 2, 1, 1, 5, 5);
+  TransposedConvolution<> module6(1, 1, 3, 3, 2, 2, 1, 1, 3, 3, 5, 5);
   // Test the forward function.
-  input = arma::linspace<arma::colvec>(0, 24, 25);
+  input = arma::linspace<arma::colvec>(0, 8, 9);
   module6.Parameters() = arma::mat(9 + 1, 1, arma::fill::zeros);
   module6.Parameters()(0) = 8.0;
   module6.Parameters()(3) = 6.0;
@@ -1821,16 +1826,17 @@ BOOST_AUTO_TEST_CASE(SimpleTransposedConvolutionLayerTest)
   module6.Parameters()(8) = 4.0;
   module6.Reset();
   module6.Forward(std::move(input), std::move(output));
-  // Value calculated using tensorflow.nn.conv2d_transpose()
-  BOOST_REQUIRE_EQUAL(arma::accu(output), 6000.0);
+  // Value calculated using torch.nn.functional.conv_transpose2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(output), 410.0);
 
   // Test the backward function.
   module6.Backward(std::move(input), std::move(output), std::move(delta));
-  BOOST_REQUIRE_EQUAL(arma::accu(delta), 87264);
+  // Value calculated using torch.nn.functional.conv2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(delta), 4444.0);
 
-  TransposedConvolution<> module7(1, 1, 3, 3, 2, 2, 1, 1, 6, 6);
+  TransposedConvolution<> module7(1, 1, 3, 3, 2, 2, 1, 1, 3, 3, 6, 6);
   // Test the forward function.
-  input = arma::linspace<arma::colvec>(0, 35, 36);
+  input = arma::linspace<arma::colvec>(0, 8, 9);
   module7.Parameters() = arma::mat(9 + 1, 1, arma::fill::zeros);
   module7.Parameters()(0) = 8.0;
   module7.Parameters()(2) = 6.0;
@@ -1838,12 +1844,12 @@ BOOST_AUTO_TEST_CASE(SimpleTransposedConvolutionLayerTest)
   module7.Parameters()(8) = 4.0;
   module7.Reset();
   module7.Forward(std::move(input), std::move(output));
-  // Value calculated using tensorflow.nn.conv2d_transpose()
-  BOOST_REQUIRE_EQUAL(arma::accu(output), 12600.0);
+  // Value calculated using torch.nn.functional.conv_transpose2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(output), 606.0);
 
-  // Test the backward function.
   module7.Backward(std::move(input), std::move(output), std::move(delta));
-  BOOST_REQUIRE_EQUAL(arma::accu(delta), 185500);
+  // Value calculated using torch.nn.functional.conv2d()
+  BOOST_REQUIRE_EQUAL(arma::accu(delta), 7732.0);
 }
 
 /**
@@ -1866,8 +1872,8 @@ BOOST_AUTO_TEST_CASE(GradientTransposedConvolutionLayerTest)
         model = new FFN<NegativeLogLikelihood<>, RandomInitialization>();
         model->Predictors() = input;
         model->Responses() = target;
-        model->Add<Linear<> >(36, 36);
-        model->Add<TransposedConvolution<> >(1, 1, 3, 3, 2, 2, 1, 1, 6, 6);
+        model->Add<TransposedConvolution<> >
+            (1, 1, 3, 3, 2, 2, 1, 1, 6, 6, 12, 12);
         model->Add<LogSoftMax<> >();
       }
 
@@ -2691,7 +2697,7 @@ void ANNLayerSerializationTest(LayerType& layer)
   model.Train(input, output, opt);
 
   arma::mat originalOutput;
-  model.Predict(input.col(0), originalOutput);
+  model.Predict(input, originalOutput);
 
   // Now serialize the model.
   FFN<NegativeLogLikelihood<>, ann::RandomInitialization> xmlModel, textModel,
@@ -2700,10 +2706,10 @@ void ANNLayerSerializationTest(LayerType& layer)
 
   // Ensure that predictions are the same.
   arma::mat modelOutput, xmlOutput, textOutput, binaryOutput;
-  model.Predict(input.col(0), modelOutput);
-  xmlModel.Predict(input.col(0), xmlOutput);
-  textModel.Predict(input.col(0), textOutput);
-  binaryModel.Predict(input.col(0), binaryOutput);
+  model.Predict(input, modelOutput);
+  xmlModel.Predict(input, xmlOutput);
+  textModel.Predict(input, textOutput);
+  binaryModel.Predict(input, binaryOutput);
 
   CheckMatrices(originalOutput, modelOutput, 1e-5);
   CheckMatrices(originalOutput, xmlOutput, 1e-5);

src/mlpack/tests/callback_test.cpp

@@ -0,0 +1,138 @@
+
+#include <ensmallen.hpp>
+#include <ensmallen_bits/callbacks/callbacks.hpp>
+#include <mlpack/core.hpp>
+#include <mlpack/methods/ann/ffn.hpp>
+#include <mlpack/methods/ann/rnn.hpp>
+#include <mlpack/methods/ann/loss_functions/mean_squared_error.hpp>
+#include <mlpack/methods/logistic_regression/logistic_regression.hpp>
+
+#include <boost/test/unit_test.hpp>
+
+using namespace mlpack;
+using namespace mlpack::ann;
+using namespace mlpack::regression;
+
+BOOST_AUTO_TEST_SUITE(CallbackTest);
+
+/**
+ * Test a FFN model with PrintLoss callback.
+ */
+BOOST_AUTO_TEST_CASE(FFNCallbackTest)
+{
+  arma::mat data;
+  arma::mat labels;
+
+  data::Load("lab1.csv", data, true);
+  data::Load("lab3.csv", labels, true);
+
+  FFN<MeanSquaredError<>, RandomInitialization> model;
+
+  model.Add<Linear<>>(1, 2);
+  model.Add<SigmoidLayer<>>();
+  model.Add<Linear<>>(2, 1);
+  model.Add<SigmoidLayer<>>();
+
+  std::stringstream stream;
+  model.Train(data, labels, ens::PrintLoss(stream));
+
+  BOOST_REQUIRE_GT(stream.str().length(), 0);
+}
+
+/**
+ * Test a FFN model with PrintLoss callback and optimizer parameter.
+ */
+BOOST_AUTO_TEST_CASE(FFNWithOptimizerCallbackTest)
+{
+  arma::mat data;
+  arma::mat labels;
+
+  data::Load("lab1.csv", data, true);
+  data::Load("lab3.csv", labels, true);
+
+  FFN<MeanSquaredError<>, RandomInitialization> model;
+
+  model.Add<Linear<>>(1, 2);
+  model.Add<SigmoidLayer<>>();
+  model.Add<Linear<>>(2, 1);
+  model.Add<SigmoidLayer<>>();
+
+  std::stringstream stream;
+  ens::StandardSGD opt(0.1, 1, 5);
+  model.Train(data, labels, opt, ens::PrintLoss(stream));
+
+  BOOST_REQUIRE_GT(stream.str().length(), 0);
+}
+
+/**
+ * Test a RNN model with PrintLoss callback.
+ */
+BOOST_AUTO_TEST_CASE(RNNCallbackTest)
+{
+  const size_t rho = 5;
+  arma::cube input = arma::randu(1, 1, 5);
+  arma::cube target = arma::ones(1, 1, 5);
+  RandomInitialization init(0.5, 0.5);
+
+  // Create model with user defined rho parameter.
+  RNN<NegativeLogLikelihood<>, RandomInitialization> model(
+      rho, false, NegativeLogLikelihood<>(), init);
+  model.Add<IdentityLayer<> >();
+  model.Add<Linear<> >(1, 10);
+
+  // Use LSTM layer with rho.
+  model.Add<LSTM<> >(10, 3, rho);
+  model.Add<LogSoftMax<> >();
+
+  std::stringstream stream;
+  model.Train(input, target, ens::PrintLoss(stream));
+
+  BOOST_REQUIRE_GT(stream.str().length(), 0);
+}
+
+/**
+ * Test a RNN model with PrintLoss callback and optimizer parameter.
+ */
+BOOST_AUTO_TEST_CASE(RNNWithOptimizerCallbackTest)
+{
+  const size_t rho = 5;
+  arma::cube input = arma::randu(1, 1, 5);
+  arma::cube target = arma::ones(1, 1, 5);
+  RandomInitialization init(0.5, 0.5);
+
+  // Create model with user defined rho parameter.
+  RNN<NegativeLogLikelihood<>, RandomInitialization> model(
+      rho, false, NegativeLogLikelihood<>(), init);
+  model.Add<IdentityLayer<> >();
+  model.Add<Linear<> >(1, 10);
+
+  // Use LSTM layer with rho.
+  model.Add<LSTM<> >(10, 3, rho);
+  model.Add<LogSoftMax<> >();
+
+  std::stringstream stream;
+  ens::StandardSGD opt(0.1, 1, 5);
+  model.Train(input, target, opt, ens::PrintLoss(stream));
+
+  BOOST_REQUIRE_GT(stream.str().length(), 0);
+}
+
+/**
+ * Test Logistic regression implementation with PrintLoss callback.
+ */
+BOOST_AUTO_TEST_CASE(LRWithOptimizerCallback)
+{
+    arma::mat data("1 2 3;"
+                   "1 2 3");
+    arma::Row<size_t> responses("1 1 0");
+
+    ens::StandardSGD sgd(0.1, 1, 5);
+    LogisticRegression<> logisticRegression(data, responses, sgd, 0.001);
+    std::stringstream stream;
+    logisticRegression.Train<ens::StandardSGD>(data, responses, sgd,
+        ens::PrintLoss(stream));
+
+    BOOST_REQUIRE_GT(stream.str().length(), 0);
+}
+
+BOOST_AUTO_TEST_SUITE_END();

src/mlpack/tests/cf_test.cpp

@@ -159,7 +159,7 @@ void GetRecommendationsQueriedUser()
  */
 template<typename DecompositionPolicy,
          typename NormalizationType = NoNormalization>
-void RecommendationAccuracy()
+void RecommendationAccuracy(const size_t allowedFailures = 17)
 {
   DecompositionPolicy decomposition;
 
@@ -214,7 +214,7 @@ void RecommendationAccuracy()
   }
 
   // Make sure the right item showed up in at least 2/3 of the recommendations.
-  BOOST_REQUIRE_LT(failures, 17);
+  BOOST_REQUIRE_LT(failures, allowedFailures);
 }
 
 // Make sure that Predict() is returning reasonable results.
@@ -712,7 +712,9 @@ BOOST_AUTO_TEST_CASE(RecommendationAccuracySVDIncompleteTest)
  */
 BOOST_AUTO_TEST_CASE(RecommendationAccuracyBiasSVDTest)
 {
-  RecommendationAccuracy<BiasSVDPolicy>();
+  // This algorithm seems to be far less effective than others.
+  // We therefore allow failures on 44% of the runs.
+  RecommendationAccuracy<BiasSVDPolicy>(22);
 }
 
 /**

src/mlpack/tests/dcgan_test.cpp

@@ -22,6 +22,7 @@
 
 #include <boost/test/unit_test.hpp>
 #include "test_tools.hpp"
+#include "serialization.hpp"
 
 using namespace mlpack;
 using namespace mlpack::ann;
@@ -75,7 +76,7 @@ BOOST_AUTO_TEST_CASE(DCGANMNISTTest)
             << trainData.n_cols << ")" << std::endl;
   Log::Info << trainData.n_rows << "--------" << trainData.n_cols << std::endl;
 
-  // Create the Discriminator network
+  // Create the Discriminator network.
   FFN<SigmoidCrossEntropyError<> > discriminator;
   discriminator.Add<Convolution<> >(1, dNumKernels, 4, 4, 2, 2, 1, 1, 28, 28);
   discriminator.Add<LeakyReLU<> >(0.2);
@@ -91,48 +92,48 @@ BOOST_AUTO_TEST_CASE(DCGANMNISTTest)
   discriminator.Add<Convolution<> >(8 * dNumKernels, 1, 4, 4, 1, 1,
       1, 1, 2, 2);
 
-  // Create the Generator network
+  // Create the Generator network.
   FFN<SigmoidCrossEntropyError<> > generator;
   generator.Add<TransposedConvolution<> >(noiseDim, 8 * dNumKernels, 2, 2,
-      1, 1, 1, 1, 1, 1);
+      1, 1, 0, 0, 1, 1, 2, 2);
   generator.Add<BatchNorm<> >(1024);
   generator.Add<ReLULayer<> >();
   generator.Add<TransposedConvolution<> >(8 * dNumKernels, 4 * dNumKernels,
-      2, 2, 1, 1, 0, 0, 2, 2);
+      2, 2, 1, 1, 0, 0, 2, 2, 3, 3);
   generator.Add<BatchNorm<> >(1152);
   generator.Add<ReLULayer<> >();
   generator.Add<TransposedConvolution<> >(4 * dNumKernels, 2 * dNumKernels,
-      5, 5, 2, 2, 1, 1, 3, 3);
+      5, 5, 2, 2, 1, 1, 3, 3, 7, 7);
   generator.Add<BatchNorm<> >(3136);
   generator.Add<ReLULayer<> >();
-  generator.Add<TransposedConvolution<> >(2 * dNumKernels, dNumKernels, 8, 8,
-      1, 1, 1, 1, 7, 7);
+  generator.Add<TransposedConvolution<> >(2 * dNumKernels, dNumKernels, 4, 4,
+      2, 2, 1, 1, 7, 7, 14, 14);
   generator.Add<BatchNorm<> >(6272);
   generator.Add<ReLULayer<> >();
-  generator.Add<TransposedConvolution<> >(dNumKernels, 1, 15, 15, 1, 1, 1, 1,
-      14, 14);
+  generator.Add<TransposedConvolution<> >(dNumKernels, 1, 4, 4, 2, 2, 1, 1,
+      14, 14, 28, 28);
   generator.Add<TanHLayer<> >();
 
-  // Create DCGAN
+  // Create DCGAN.
   GaussianInitialization gaussian(0, 1);
   ens::Adam optimizer(stepSize, batchSize, 0.9, 0.999, eps, numIterations,
       tolerance, shuffle);
   std::function<double()> noiseFunction = [] () {
       return math::RandNormal(0, 1);};
   GAN<FFN<SigmoidCrossEntropyError<> >, GaussianInitialization,
-      std::function<double()>, DCGAN> dcgan(trainData, generator, discriminator,
-      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      std::function<double()>, DCGAN> dcgan(generator, discriminator, gaussian,
+      noiseFunction, noiseDim, batchSize, generatorUpdateStep,
       discriminatorPreTrain, multiplier);
 
   Log::Info << "Training..." << std::endl;
-  double objVal = dcgan.Train(optimizer);
+  double objVal = dcgan.Train(trainData, optimizer);
 
   // Test that objective value returned by GAN::Train() is finite.
   BOOST_REQUIRE_EQUAL(std::isfinite(objVal), true);
 
-  // Generate samples
+  // Generate samples.
   Log::Info << "Sampling..." << std::endl;
-  arma::mat noise(noiseDim, 1);
+  arma::mat noise(noiseDim, batchSize);
   size_t dim = std::sqrt(trainData.n_rows);
   arma::mat generatedData(2 * dim, dim * numSamples);
 
@@ -156,6 +157,38 @@ BOOST_AUTO_TEST_CASE(DCGANMNISTTest)
   }
 
   Log::Info << "Output generated!" << std::endl;
+
+  // Check that Serialization is working correctly.
+  arma::mat orgPredictions;
+  dcgan.Predict(noise, orgPredictions);
+
+  GAN<FFN<SigmoidCrossEntropyError<> >, GaussianInitialization,
+      std::function<double()>, DCGAN> dcganText(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  GAN<FFN<SigmoidCrossEntropyError<> >, GaussianInitialization,
+      std::function<double()>, DCGAN> dcganXml(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  GAN<FFN<SigmoidCrossEntropyError<> >, GaussianInitialization,
+      std::function<double()>, DCGAN> dcganBinary(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  SerializeObjectAll(dcgan, dcganXml, dcganText, dcganBinary);
+
+  arma::mat predictions, xmlPredictions, textPredictions, binaryPredictions;
+  dcgan.Predict(noise, predictions);
+  dcganXml.Predict(noise, xmlPredictions);
+  dcganText.Predict(noise, textPredictions);
+  dcganBinary.Predict(noise, binaryPredictions);
+
+  CheckMatrices(orgPredictions, predictions);
+  CheckMatrices(orgPredictions, xmlPredictions);
+  CheckMatrices(orgPredictions, textPredictions);
+  CheckMatrices(orgPredictions, binaryPredictions);
 }
 
 
@@ -357,7 +390,7 @@ BOOST_AUTO_TEST_CASE(DCGANCelebATest)
             << trainData.n_cols << ")" << std::endl;
   Log::Info << trainData.n_rows << "--------" << trainData.n_cols << std::endl;
 
-  // Create the Discriminator network
+  // Create the Discriminator network.
   FFN<SigmoidCrossEntropyError<> > discriminator;
   discriminator.Add<Convolution<> >(3, dNumKernels, 4, 4, 2, 2, 1, 1, 64, 64);
   discriminator.Add<LeakyReLU<> >(0.2);
@@ -373,7 +406,7 @@ BOOST_AUTO_TEST_CASE(DCGANCelebATest)
   discriminator.Add<Convolution<> >(8 * dNumKernels, 1, 4, 4, 1, 1,
       0, 0, 4, 4);
 
-  // Create the Generator network
+  // Create the Generator network.
   FFN<SigmoidCrossEntropyError<> > generator;
   generator.Add<TransposedConvolution<> >(noiseDim, 8 * dNumKernels, 4, 4,
       1, 1, 2, 2, 1, 1);
@@ -395,7 +428,7 @@ BOOST_AUTO_TEST_CASE(DCGANCelebATest)
       32, 32);
   generator.Add<TanHLayer<> >();
 
-  // Create DCGAN
+  // Create DCGAN.
   GaussianInitialization gaussian(0, 1);
   ens::Adam optimizer(stepSize, batchSize, 0.9, 0.999, eps, numIterations,
       tolerance, shuffle);
@@ -409,7 +442,7 @@ BOOST_AUTO_TEST_CASE(DCGANCelebATest)
   Log::Info << "Training..." << std::endl;
   dcgan.Train(optimizer);
 
-  // Generate samples
+  // Generate samples.
   Log::Info << "Sampling..." << std::endl;
   arma::mat noise(noiseDim, 1);
   size_t dim = std::sqrt(trainData.n_rows);

src/mlpack/tests/feedforward_network_test.cpp

@@ -575,4 +575,41 @@ BOOST_AUTO_TEST_CASE(FFNTrainReturnObjective)
 
   BOOST_REQUIRE_EQUAL(std::isfinite(objVal), true);
 }
+
+/**
+ * Test that FFN::Model() allows us to access the instantiated network.
+ */
+BOOST_AUTO_TEST_CASE(FFNReturnModel)
+{
+  // Create dummy network.
+  FFN<NegativeLogLikelihood<> > model;
+  Linear<>* linearA = new Linear<>(3, 3);
+  model.Add(linearA);
+  Linear<>* linearB = new Linear<>(3, 4);
+  model.Add(linearB);
+
+  // Initialize network parameter.
+  model.ResetParameters();
+
+  // Set all network parameter to one.
+  model.Parameters().ones();
+
+  // Zero the second layer parameter.
+  linearB->Parameters().zeros();
+
+  // Get the layer parameter from layer A and layer B and store them in
+  // parameterA and parameterB.
+  arma::mat parameterA, parameterB;
+  boost::apply_visitor(ParametersVisitor(std::move(parameterA)),
+      model.Model()[0]);
+  boost::apply_visitor(ParametersVisitor(std::move(parameterB)),
+      model.Model()[1]);
+
+  CheckMatrices(parameterA, arma::ones(3 * 3 + 3, 1));
+  CheckMatrices(parameterB, arma::zeros(3 * 4 + 4, 1));
+
+  CheckMatrices(linearA->Parameters(), arma::ones(3 * 3 + 3, 1));
+  CheckMatrices(linearB->Parameters(), arma::zeros(3 * 4 + 4, 1));
+}
+
 BOOST_AUTO_TEST_SUITE_END();

src/mlpack/tests/gan_test.cpp

@@ -22,6 +22,7 @@
 
 #include <boost/test/unit_test.hpp>
 #include "test_tools.hpp"
+#include "serialization.hpp"
 
 using namespace mlpack;
 using namespace mlpack::ann;
@@ -53,7 +54,7 @@ BOOST_AUTO_TEST_CASE(GANTest)
   trainData.imbue( [&]() { return arma::as_scalar(RandNormal(4, 0.5));});
   trainData = arma::sort(trainData);
 
-  // Create the Discriminator network
+  // Create the Discriminator network.
   FFN<SigmoidCrossEntropyError<> > discriminator;
   discriminator.Add<Linear<> > (
       generatorOutputSize, discriminatorHiddenLayerSize * 2);
@@ -67,30 +68,29 @@ BOOST_AUTO_TEST_CASE(GANTest)
   discriminator.Add<Linear<> > (
       discriminatorHiddenLayerSize * 2, discriminatorOutputSize);
 
-  // Create the Generator network
+  // Create the Generator network.
   FFN<SigmoidCrossEntropyError<> > generator;
   generator.Add<Linear<> >(noiseDim, generatorHiddenLayerSize);
   generator.Add<SoftPlusLayer<> >();
   generator.Add<Linear<> >(generatorHiddenLayerSize, generatorOutputSize);
 
-  // Create GAN
+  // Create GAN.
   GaussianInitialization gaussian(0, 0.1);
   std::function<double ()> noiseFunction = [](){ return math::Random(-8, 8) +
       math::RandNormal(0, 1) * 0.01;};
   GAN<FFN<SigmoidCrossEntropyError<> >,
       GaussianInitialization,
       std::function<double()> >
-  gan(trainData, generator, discriminator, gaussian, noiseFunction,
-      noiseDim, batchSize, generatorUpdateStep, discriminatorPreTrain,
-      multiplier);
-  gan.Reset();
+  gan(generator, discriminator, gaussian, noiseFunction, noiseDim, batchSize,
+      generatorUpdateStep, discriminatorPreTrain, multiplier);
+  gan.ResetData(trainData);
 
   Log::Info << "Loading Parameters" << std::endl;
   arma::mat parameters, generatorParameters;
   parameters.load("preTrainedGAN.arm");
   gan.Parameters() = parameters;
 
-  // Generate samples
+  // Generate samples.
   Log::Info << "Sampling..." << std::endl;
   arma::mat noise(noiseDim, batchSize);
 
@@ -173,7 +173,7 @@ BOOST_AUTO_TEST_CASE(GANMNISTTest)
             << trainData.n_cols << ")" << std::endl;
   Log::Info << trainData.n_rows << "--------" << trainData.n_cols << std::endl;
 
-  // Create the Discriminator network
+  // Create the Discriminator network.
   FFN<SigmoidCrossEntropyError<> > discriminator;
   discriminator.Add<Convolution<> >(1, dNumKernels, 5, 5, 1, 1, 2, 2, 28, 28);
   discriminator.Add<ReLULayer<> >();
@@ -186,7 +186,7 @@ BOOST_AUTO_TEST_CASE(GANMNISTTest)
   discriminator.Add<ReLULayer<> >();
   discriminator.Add<Linear<> >(1024, 1);
 
-  // Create the Generator network
+  // Create the Generator network.
   FFN<SigmoidCrossEntropyError<> > generator;
   generator.Add<Linear<> >(noiseDim, 3136);
   generator.Add<BatchNorm<> >(3136);
@@ -203,14 +203,14 @@ BOOST_AUTO_TEST_CASE(GANMNISTTest)
   generator.Add<Convolution<> >(noiseDim / 4, 1, 3, 3, 2, 2, 1, 1, 56, 56);
   generator.Add<TanHLayer<> >();
 
-  // Create GAN
+  // Create GAN.
   GaussianInitialization gaussian(0, 1);
   ens::Adam optimizer(stepSize, batchSize, 0.9, 0.999, eps, numIterations,
       tolerance, shuffle);
   std::function<double()> noiseFunction = [] () {
       return math::RandNormal(0, 1);};
   GAN<FFN<SigmoidCrossEntropyError<> >, GaussianInitialization,
-      std::function<double()> > gan(trainData, generator, discriminator,
+      std::function<double()> > gan(generator, discriminator,
       gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
       discriminatorPreTrain, multiplier);
 
@@ -246,6 +246,38 @@ BOOST_AUTO_TEST_CASE(GANMNISTTest)
   }
 
   Log::Info << "Output generated!" << std::endl;
+
+  // Check that Serialization is working correctly.
+  arma::mat orgPredictions;
+  gan.Predict(noise, orgPredictions);
+
+  GAN<FFN<SigmoidCrossEntropyError<> >, GaussianInitialization,
+      std::function<double()> > ganText(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  GAN<FFN<SigmoidCrossEntropyError<> >, GaussianInitialization,
+      std::function<double()> > ganXml(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  GAN<FFN<SigmoidCrossEntropyError<> >, GaussianInitialization,
+      std::function<double()> > ganBinary(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  SerializeObjectAll(gan, ganXml, ganText, ganBinary);
+
+  arma::mat predictions, xmlPredictions, textPredictions, binaryPredictions;
+  gan.Predict(noise, predictions);
+  ganXml.Predict(noise, xmlPredictions);
+  ganText.Predict(noise, textPredictions);
+  ganBinary.Predict(noise, binaryPredictions);
+
+  CheckMatrices(orgPredictions, predictions);
+  CheckMatrices(orgPredictions, xmlPredictions);
+  CheckMatrices(orgPredictions, textPredictions);
+  CheckMatrices(orgPredictions, binaryPredictions);
 }
 
 /*
@@ -263,12 +295,17 @@ BOOST_AUTO_TEST_CASE(GANMemorySharingTest)
   size_t noiseDim = 1;
   size_t generatorUpdateStep = 1;
   double multiplier = 1;
+  double eps = 1e-8;
+  double stepSize = 0.0003;
+  size_t numIterations = 8;
+  double tolerance = 1e-5;
+  bool shuffle = true;
 
   arma::mat trainData(1, 10000);
   trainData.imbue( [&]() { return arma::as_scalar(RandNormal(4, 0.5));});
   trainData = arma::sort(trainData);
 
-  // Create the Discriminator network
+  // Create the Discriminator network.
   FFN<SigmoidCrossEntropyError<> > discriminator;
   discriminator.Add<Linear<> > (
       generatorOutputSize, discriminatorHiddenLayerSize * 2);
@@ -282,23 +319,27 @@ BOOST_AUTO_TEST_CASE(GANMemorySharingTest)
   discriminator.Add<Linear<> > (
       discriminatorHiddenLayerSize * 2, discriminatorOutputSize);
 
-  // Create the Generator network
+  // Create the Generator network.
   FFN<SigmoidCrossEntropyError<> > generator;
   generator.Add<Linear<> >(noiseDim, generatorHiddenLayerSize);
   generator.Add<SoftPlusLayer<> >();
   generator.Add<Linear<> >(generatorHiddenLayerSize, generatorOutputSize);
 
-  // Create GAN
+  // Create GAN.
   GaussianInitialization gaussian(0, 0.1);
+  ens::Adam optimizer(stepSize, batchSize, 0.9, 0.999, eps, numIterations,
+      tolerance, shuffle);
   std::function<double ()> noiseFunction = [](){ return math::Random(-8, 8) +
       math::RandNormal(0, 1) * 0.01;};
   GAN<FFN<SigmoidCrossEntropyError<> >,
       GaussianInitialization,
       std::function<double()> >
-  gan(trainData, generator, discriminator, gaussian, noiseFunction,
+  gan(generator, discriminator, gaussian, noiseFunction,
       noiseDim, batchSize, generatorUpdateStep, discriminatorPreTrain,
       multiplier);
 
+  gan.Train(trainData, optimizer);
+
   CheckMatrices(gan.Predictors().head_cols(trainData.n_cols), trainData);
   CheckMatrices(gan.Predictors(), gan.Discriminator().Predictors());
   gan.Shuffle();

src/mlpack/tests/knn_test.cpp

@@ -1329,6 +1329,62 @@ BOOST_AUTO_TEST_CASE(CopyConstructorAndOperatorRTreeTest)
   CheckMatrices(distances, distances3);
 }
 
+/**
+ * Test the copy constructor and copy operator using the Cover Tree.
+ */
+BOOST_AUTO_TEST_CASE(CopyConstructorAndOperatorCoverTreeTest)
+{
+  arma::mat dataset = arma::randu<arma::mat>(5, 500);
+  typedef NeighborSearch<NearestNeighborSort, EuclideanDistance, arma::mat,
+      StandardCoverTree> NeighborSearchType;
+  NeighborSearchType knn(std::move(dataset));
+
+  // Copy constructor and operator.
+  NeighborSearchType knn2(knn);
+  NeighborSearchType knn3 = knn;
+
+  // Get results.
+  arma::mat distances, distances2, distances3;
+  arma::Mat<size_t> neighbors, neighbors2, neighbors3;
+
+  knn.Search(3, neighbors, distances);
+  knn2.Search(3, neighbors2, distances2);
+  knn3.Search(3, neighbors3, distances3);
+
+  CheckMatrices(neighbors, neighbors2);
+  CheckMatrices(neighbors, neighbors3);
+  CheckMatrices(distances, distances2);
+  CheckMatrices(distances, distances3);
+}
+
+/**
+ * Test the copy constructor and copy operator using the Spill Tree.
+ */
+BOOST_AUTO_TEST_CASE(CopyConstructorAndOperatorSpillTreeTest)
+{
+  arma::mat dataset = arma::randu<arma::mat>(5, 500);
+  typedef NeighborSearch<NearestNeighborSort, EuclideanDistance, arma::mat,
+      SPTree> NeighborSearchType;
+  NeighborSearchType knn(std::move(dataset));
+
+  // Copy constructor and operator.
+  NeighborSearchType knn2(knn);
+  NeighborSearchType knn3 = knn;
+
+  // Get results.
+  arma::mat distances, distances2, distances3;
+  arma::Mat<size_t> neighbors, neighbors2, neighbors3;
+
+  knn.Search(3, neighbors, distances);
+  knn2.Search(3, neighbors2, distances2);
+  knn3.Search(3, neighbors3, distances3);
+
+  CheckMatrices(neighbors, neighbors2);
+  CheckMatrices(neighbors, neighbors3);
+  CheckMatrices(distances, distances2);
+  CheckMatrices(distances, distances3);
+}
+
 /**
  * Test the move constructor.
  */
@@ -1382,6 +1438,72 @@ BOOST_AUTO_TEST_CASE(MoveConstructorRTreeTest)
 }
 
 
+/**
+ * Test the move constructor & move assignment using Cover Tree.
+ */
+BOOST_AUTO_TEST_CASE(MoveConstructorCoverTreeTest)
+{
+  arma::mat dataset = arma::randu<arma::mat>(5, 500);
+  typedef NeighborSearch<NearestNeighborSort, EuclideanDistance, arma::mat,
+      StandardCoverTree> NeighborSearchType;
+  NeighborSearchType* knn = new NeighborSearchType(std::move(dataset));
+
+  // Get predictions.
+  arma::mat distances, distances2, distances3;
+  arma::Mat<size_t> neighbors, neighbors2, neighbors3;
+
+  knn->Search(3, neighbors, distances);
+
+  // Use move constructor.
+  NeighborSearchType knn2(std::move(*knn));
+
+  delete knn;
+
+  knn2.Search(3, neighbors2, distances2);
+
+  // Use move assignment.
+  NeighborSearchType knn3 = std::move(knn2);
+  knn3.Search(3, neighbors3, distances3);
+
+  CheckMatrices(neighbors, neighbors2);
+  CheckMatrices(neighbors, neighbors3);
+  CheckMatrices(distances, distances2);
+  CheckMatrices(distances, distances3);
+}
+
+/**
+ * Test the move constructor & move assignment using Spill Tree.
+ */
+BOOST_AUTO_TEST_CASE(MoveConstructorSpillTreeTest)
+{
+  arma::mat dataset = arma::randu<arma::mat>(5, 500);
+  typedef NeighborSearch<NearestNeighborSort, EuclideanDistance, arma::mat,
+      SPTree> NeighborSearchType;
+  NeighborSearchType* knn = new NeighborSearchType(std::move(dataset));
+
+  // Get predictions.
+  arma::mat distances, distances2, distances3;
+  arma::Mat<size_t> neighbors, neighbors2, neighbors3;
+
+  knn->Search(3, neighbors, distances);
+
+  // Use move constructor.
+  NeighborSearchType knn2(std::move(*knn));
+
+  delete knn;
+
+  knn2.Search(3, neighbors2, distances2);
+
+  // Use move assignment.
+  NeighborSearchType knn3 = std::move(knn2);
+  knn3.Search(3, neighbors3, distances3);
+
+  CheckMatrices(neighbors, neighbors2);
+  CheckMatrices(neighbors, neighbors3);
+  CheckMatrices(distances, distances2);
+  CheckMatrices(distances, distances3);
+}
+
 /**
  * Test the move operator.
  */

src/mlpack/tests/layer_names_test.cpp

@@ -0,0 +1,136 @@
+/**
+ * @file layer_names_test.cpp
+ * @author Sreenik Seal
+ *
+ * Tests for testing the string representation of
+ * layers in mlpack's ANN module.
+ *
+ * mlpack is free software; you may redistribute it and/or modify it under the
+ * terms of the 3-clause BSD license.  You should have received a copy of the
+ * 3-clause BSD license along with mlpack.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#include <mlpack/core.hpp>
+#include <mlpack/methods/ann/layer/layer.hpp>
+#include <mlpack/methods/ann/layer/layer_types.hpp>
+#include <mlpack/methods/ann/layer_names.hpp>
+
+#include <boost/test/unit_test.hpp>
+#include "test_tools.hpp"
+
+using namespace mlpack;
+using namespace ann;
+
+BOOST_AUTO_TEST_SUITE(LayerNamesTest);
+
+/**
+ * Test if the LayerNameVisitor works properly.
+ */
+BOOST_AUTO_TEST_CASE(LayerNameVisitorTest)
+{
+  LayerTypes<> atrousConvolution = new AtrousConvolution<>();
+  LayerTypes<> alphaDropout = new AlphaDropout<>();
+  LayerTypes<> batchNorm = new BatchNorm<>();
+  LayerTypes<> constant = new Constant<>();
+  LayerTypes<> convolution = new Convolution<>();
+  LayerTypes<> dropConnect = new DropConnect<>();
+  LayerTypes<> dropout = new Dropout<>();
+  LayerTypes<> flexibleReLU = new FlexibleReLU<>();
+  LayerTypes<> layerNorm = new LayerNorm<>();
+  LayerTypes<> linear = new Linear<>();
+  LayerTypes<> linearNoBias = new LinearNoBias<>();
+  LayerTypes<> maxPooling = new MaxPooling<>();
+  LayerTypes<> meanPooling = new MeanPooling<>();
+  LayerTypes<> multiplyConstant = new MultiplyConstant<>();
+  LayerTypes<> reLULayer = new ReLULayer<>();
+  LayerTypes<> transposedConvolution = new TransposedConvolution<>();
+  LayerTypes<> identityLayer = new IdentityLayer<>();
+  LayerTypes<> tanHLayer = new TanHLayer<>();
+  LayerTypes<> eLU = new ELU<>();
+  LayerTypes<> hardTanH = new HardTanH<>();
+  LayerTypes<> leakyReLU = new LeakyReLU<>();
+  LayerTypes<> pReLU = new PReLU<>();
+  LayerTypes<> sigmoidLayer = new SigmoidLayer<>();
+  LayerTypes<> logSoftMax = new LogSoftMax<>();
+  // Bilinear interpolation is not yet supported by the string converter.
+  LayerTypes<> unsupportedLayer = new BilinearInterpolation<>();
+
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      atrousConvolution) == "atrousconvolution");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      alphaDropout) == "alphadropout");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      batchNorm) == "batchnorm");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      constant) == "constant");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      convolution) == "convolution");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      dropConnect) == "dropconnect");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      dropout) == "dropout");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      flexibleReLU) == "flexiblerelu");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      layerNorm) == "layernorm");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      linear) == "linear");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      linearNoBias) == "linearnobias");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      maxPooling) == "maxpooling");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      meanPooling) == "meanpooling");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      multiplyConstant) == "multiplyconstant");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      reLULayer) == "relu");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      transposedConvolution) == "transposedconvolution");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      identityLayer) == "identity");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      tanHLayer) == "tanh");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      eLU) == "elu");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      hardTanH) == "hardtanh");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      leakyReLU) == "leakyrelu");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      pReLU) == "prelu");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      sigmoidLayer) == "sigmoid");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      logSoftMax) == "logsoftmax");
+  BOOST_REQUIRE(boost::apply_visitor(LayerNameVisitor(),
+                      unsupportedLayer) == "unsupported");
+  // Delete all instances.
+  boost::apply_visitor(DeleteVisitor(), atrousConvolution);
+  boost::apply_visitor(DeleteVisitor(), alphaDropout);
+  boost::apply_visitor(DeleteVisitor(), batchNorm);
+  boost::apply_visitor(DeleteVisitor(), constant);
+  boost::apply_visitor(DeleteVisitor(), convolution);
+  boost::apply_visitor(DeleteVisitor(), dropConnect);
+  boost::apply_visitor(DeleteVisitor(), dropout);
+  boost::apply_visitor(DeleteVisitor(), flexibleReLU);
+  boost::apply_visitor(DeleteVisitor(), layerNorm);
+  boost::apply_visitor(DeleteVisitor(), linear);
+  boost::apply_visitor(DeleteVisitor(), linearNoBias);
+  boost::apply_visitor(DeleteVisitor(), maxPooling);
+  boost::apply_visitor(DeleteVisitor(), meanPooling);
+  boost::apply_visitor(DeleteVisitor(), multiplyConstant);
+  boost::apply_visitor(DeleteVisitor(), reLULayer);
+  boost::apply_visitor(DeleteVisitor(), transposedConvolution);
+  boost::apply_visitor(DeleteVisitor(), identityLayer);
+  boost::apply_visitor(DeleteVisitor(), tanHLayer);
+  boost::apply_visitor(DeleteVisitor(), eLU);
+  boost::apply_visitor(DeleteVisitor(), hardTanH);
+  boost::apply_visitor(DeleteVisitor(), leakyReLU);
+  boost::apply_visitor(DeleteVisitor(), pReLU);
+  boost::apply_visitor(DeleteVisitor(), sigmoidLayer);
+  boost::apply_visitor(DeleteVisitor(), logSoftMax);
+  boost::apply_visitor(DeleteVisitor(), unsupportedLayer);
+}
+
+BOOST_AUTO_TEST_SUITE_END();

src/mlpack/tests/main_tests/linear_svm_test.cpp

@@ -0,0 +1,947 @@
+/**
+ * @file linear_svm_test.cpp
+ * @author Yashwant Singh Parihar
+ *
+ * Test mlpackMain() of logistic_regression_main.cpp
+ *
+ * mlpack is free software; you may redistribute it and/or modify it under the
+ * terms of the 3-clause BSD license.  You should have received a copy of the
+ * 3-clause BSD license along with mlpack.  If not, see
+ * http://www.opensource.org/licenses/BSD-3-Clause for more information.
+ */
+#include <string>
+
+#define BINDING_TYPE BINDING_TYPE_TEST
+
+static const std::string testName = "LinearSVM";
+
+#include <mlpack/core.hpp>
+#include <mlpack/methods/linear_svm/linear_svm_main.cpp>
+#include <mlpack/core/util/mlpack_main.hpp>
+#include "test_helper.hpp"
+
+#include <boost/test/unit_test.hpp>
+#include "../test_tools.hpp"
+
+using namespace mlpack;
+
+struct LinearSVMTestFixture
+{
+ public:
+  LinearSVMTestFixture()
+  {
+    // Cache in the options for this program.
+    CLI::RestoreSettings(testName);
+  }
+
+  ~LinearSVMTestFixture()
+  {
+    // Clear the settings.
+    bindings::tests::CleanMemory();
+    CLI::ClearSettings();
+  }
+};
+
+BOOST_FIXTURE_TEST_SUITE(LinearSVMMainTest,
+                         LinearSVMTestFixture);
+
+/**
+ * Ensure that trainingSet are necessarily passed when training.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMNoTrainingData)
+{
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("labels", std::move(trainLabels));
+
+  // Training data is not provided. Should throw a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Checking that that size and dimensionality of prediction is correct.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMOutputDimensionTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  arma::mat testData;
+  if (!data::Load("iris_test.csv", testData))
+    BOOST_FAIL("Cannot load test dataset iris_test.csv!");
+
+  size_t testSize = testData.n_cols;
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("test", std::move(testData));
+
+  // Training the model.
+  mlpackMain();
+
+  // Get the output predictions of the test data.
+  const arma::Row<size_t>& testLabels =
+      CLI::GetParam<arma::Row<size_t>>("predictions");
+
+  // Output predictions size must match the test data set size.
+  BOOST_REQUIRE_EQUAL(testLabels.n_rows, 1);
+  BOOST_REQUIRE_EQUAL(testLabels.n_cols, testSize);
+}
+
+/**
+ * Ensuring that the labels size is checked.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMCheckLabelsSizeTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("vc2_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset vc2_labels.txt!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+
+  // Labels with incorrect size. It should throw a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Checking two options of specifying labels (extra row in train matrix and
+ * extra parameter) and ensuring that predictions are the same.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMLabelsRepresentationTest)
+{
+  arma::mat trainData1({{1.0, 2.0, 3.0}, {1.0, 4.0, 9.0}, {0, 1, 1}});
+  arma::mat testData({{4.0, 5.0}, {1.0, 6.0}});
+
+  SetInputParam("training", std::move(trainData1));
+  SetInputParam("test", testData);
+
+  // The first solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the output.
+  const arma::Row<size_t> testLabels1 =
+      std::move(CLI::GetParam<arma::Row<size_t>>("predictions"));
+
+  // Reset the settings.
+  bindings::tests::CleanMemory();
+  CLI::ClearSettings();
+  CLI::RestoreSettings(testName);
+
+  // Now train by providing labels as extra parameter.
+  arma::mat trainData2({{1.0, 2.0, 3.0}, {1.0, 4.0, 9.0}});
+  arma::Row<size_t> trainLabels({0, 1, 1});
+
+  SetInputParam("training", std::move(trainData2));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("test", std::move(testData));
+
+  // The second solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // get the output
+  const arma::Row<size_t>& testLabels2 =
+      CLI::GetParam<arma::Row<size_t>>("predictions");
+
+  // Both solutions should be equal.
+  CheckMatrices(testLabels1, testLabels2);
+}
+
+/**
+ * Ensure that saved model can be used again.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMModelReuseTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  arma::mat testData;
+  if (!data::Load("iris_test.csv", testData))
+    BOOST_FAIL("Cannot load test dataset iris_test.csv!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("test", testData);
+
+  // First solution
+  mlpackMain();
+
+  // Get the output model obtained from training.
+  LinearSVMModel* model =
+      CLI::GetParam<LinearSVMModel*>("output_model");
+  // Get the output.
+  const arma::Row<size_t>& testLabels1 =
+      std::move(CLI::GetParam<arma::Row<size_t>>("predictions"));
+
+  // Reset the data passed.
+  CLI::GetSingleton().Parameters()["training"].wasPassed = false;
+  CLI::GetSingleton().Parameters()["labels"].wasPassed = false;
+  CLI::GetSingleton().Parameters()["test"].wasPassed = false;
+
+  SetInputParam("input_model", model);
+  SetInputParam("test", std::move(testData));
+
+  // Second solution.
+  mlpackMain();
+
+  // Get the output.
+  const arma::Row<size_t>& testLabels2 =
+      CLI::GetParam<arma::Row<size_t>>("predictions");
+
+  // Both solutions should be equal.
+  CheckMatrices(testLabels1, testLabels2);
+}
+
+/**
+ * Checking for dimensionality of the test data set.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMCheckDimOfTestData)
+{
+  // Dimensionality of trainingSet is trainData.n_rows - 1 because labels are
+  // not provided.
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::mat testData;
+  if (!data::Load("iris_test.csv", testData))
+    BOOST_FAIL("Cannot load test dataset iris_test.csv!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("test", std::move(testData));
+
+  // Dimensionality of test data is wrong. It should throw a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring that test data dimensionality is checked when model is loaded.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMCheckDimOfTestData2)
+{
+  // Dimensionality of trainingSet is trainData.n_rows - 1 because labels are
+  // not provided.
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::mat testData;
+  if (!data::Load("iris_test.csv", testData))
+    BOOST_FAIL("Cannot load test dataset iris_test.csv!");
+
+  SetInputParam("training", std::move(trainData));
+
+  // Training the model.
+  mlpackMain();
+
+  // Get the output model obtained from training.
+  LinearSVMModel* model =
+      CLI::GetParam<LinearSVMModel*>("output_model");
+
+  // Reset the data passed.
+  CLI::GetSingleton().Parameters()["training"].wasPassed = false;
+  CLI::GetSingleton().Parameters()["labels"].wasPassed = false;
+
+  SetInputParam("input_model", model);
+  SetInputParam("test", std::move(testData));
+
+  // Test data dimensionality is wrong. It should throw a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring that max iteration for optimizers is non negative.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMNonNegativeMaxIterationTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("max_iterations", int(-1));
+
+  // Maximum iterations is negative. It should a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring that lambda for optimizers is non negative.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMNonNegativeLambdaTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("lambda", double(-0.01));
+
+  // Lambda is negative. It should a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring that number of classes for optimizers is non negative.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMNonNegativeNumberOfClassesTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("num_classes", int(-1));
+
+  // Number of classes is negative. It should a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring that tolerance is non negative.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMNonNegativeToleranceTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("tolerance", double(-0.01));
+
+  // Tolerance is negative. It should throw a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring that delta is non negative.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMNonNegativeDeltaTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("delta", double(-0.01));
+
+  // Delta is negative. It should throw a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring that epochs is non negative.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMNonNegativeEpochsTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("epochs", int(-1));
+
+  // Epochs is negative. It should throw a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring that number classes must not be zero.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMZeroNumberOfClassesTest)
+{
+  arma::mat trainData = "2 0 0;"
+                        "0 0 0;"
+                        "0 2 1;"
+                        "1 0 2;"
+                        "0 1 0";
+
+  arma::Row<size_t> trainLabels = "0 0 0";
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+
+  // Number of classes for optimizer is zero.
+  // It should throw a invalid_argument error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::invalid_argument);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring that Optimizer must be correct.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMOptimizerTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("optimizer", std::string("hello"));
+
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring changing Maximum number of iterations changes the output model.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMDiffMaxIterationsTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", trainData);
+  SetInputParam("labels", trainLabels);
+  SetInputParam("max_iterations", int(1));
+
+  // First solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after first training.
+  const arma::mat parameters1 = std::move(
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters());
+
+  // Reset the settings.
+  bindings::tests::CleanMemory();
+  CLI::ClearSettings();
+  CLI::RestoreSettings(testName);
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("max_iterations", int(100));
+
+  // Second solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after second training.
+  const arma::mat& parameters2 =
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters();
+
+  // Both solutions should be not equal.
+  CheckMatricesNotEqual(parameters1, parameters2);
+}
+
+/**
+ * Ensuring that lambda has some effects on the output.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMDiffLambdaTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", trainData);
+  SetInputParam("labels", trainLabels);
+  SetInputParam("lambda", double(0.001));
+
+  // First solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after first training.
+  const arma::mat parameters1 = std::move(
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters());
+
+  // Reset the settings.
+  bindings::tests::CleanMemory();
+  CLI::ClearSettings();
+  CLI::RestoreSettings(testName);
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("lambda", double(1000));
+
+  // Second solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after second training.
+  const arma::mat& parameters2 =
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters();
+
+  // Both solutions should be not equal.
+  CheckMatricesNotEqual(parameters1, parameters2);
+}
+
+/**
+ * Ensuring that delta has some effects on the output.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMDiffDeltaTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", trainData);
+  SetInputParam("labels", trainLabels);
+  SetInputParam("delta", double(1.0));
+
+  // First solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after first training.
+  const arma::mat parameters1 = std::move(
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters());
+
+  // Reset the settings.
+  bindings::tests::CleanMemory();
+  CLI::ClearSettings();
+  CLI::RestoreSettings(testName);
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("delta", double(1000));
+
+  // Second solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after second training.
+  const arma::mat& parameters2 =
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters();
+
+  // Both solutions should be not equal.
+  CheckMatricesNotEqual(parameters1, parameters2);
+}
+
+/**
+ * Ensuring that no_intercept has some effects on the output.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMDiffInterceptTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", trainData);
+  SetInputParam("labels", trainLabels);
+
+  // First solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after first training.
+  const arma::mat parameters1 = std::move(
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters());
+
+  // Reset the settings.
+  bindings::tests::CleanMemory();
+  CLI::ClearSettings();
+  CLI::RestoreSettings(testName);
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("no_intercept", bool(true));
+
+  // Second solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after second training.
+  const arma::mat& parameters2 =
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters();
+
+  // Both solutions should be not equal.
+  CheckMatricesNotEqual(parameters1, parameters2);
+}
+
+/**
+ * Ensuring that no_intercept has some effects on the output
+ * when the optimizer is 'psgd'.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMDiffInterceptTestWithPsgd)
+{
+  arma::mat trainData = "2 0 0;"
+                        "0 0 0;"
+                        "0 2 1;"
+                        "1 0 2;"
+                        "0 1 0";
+
+  arma::Row<size_t> trainLabels = "1 0 1";
+
+  SetInputParam("training", trainData);
+  SetInputParam("labels", trainLabels);
+  SetInputParam("optimizer", std::string("psgd"));
+
+  // First solution.
+  #ifdef HAS_OPENMP
+  omp_set_num_threads(1);
+  #endif
+
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after first training.
+  const arma::mat parameters1 = std::move(
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters());
+
+  // Reset the settings.
+  bindings::tests::CleanMemory();
+  CLI::ClearSettings();
+  CLI::RestoreSettings(testName);
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("optimizer", std::string("psgd"));
+  SetInputParam("no_intercept", bool(true));
+
+  // Second solution.
+  #ifdef HAS_OPENMP
+  omp_set_num_threads(1);
+  #endif
+
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after second training.
+  const arma::mat& parameters2 =
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters();
+
+  // Both solutions should be not equal.
+  CheckMatricesNotEqual(parameters1, parameters2);
+}
+
+/**
+ * Ensuring that step size for optimizer is non negative.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMNonNegativeStepSizeTest)
+{
+  arma::mat trainData;
+  if (!data::Load("iris.csv", trainData))
+    BOOST_FAIL("Cannot load test dataset iris.csv!");
+
+  arma::Row<size_t> trainLabels;
+  if (!data::Load("iris_labels.txt", trainLabels))
+    BOOST_FAIL("Cannot load test dataset iris_labels.txt!");
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("optimizer", std::string("psgd"));
+  SetInputParam("step_size", double(-0.01));
+
+  // Step size for optimizer is negative. It should throw a runtime error.
+  Log::Fatal.ignoreInput = true;
+  BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
+  Log::Fatal.ignoreInput = false;
+}
+
+/**
+ * Ensuring that epochs has some effects on the output.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMDiffEpochsTest)
+{
+  arma::mat trainData = "2 0 0;"
+                        "0 0 0;"
+                        "0 2 1;"
+                        "1 0 2;"
+                        "0 1 0";
+
+  arma::Row<size_t> trainLabels = "1 0 1";
+
+  SetInputParam("training", trainData);
+  SetInputParam("labels", trainLabels);
+  SetInputParam("optimizer", std::string("psgd"));
+  SetInputParam("epochs", int(5));
+
+  // First solution.
+  #ifdef HAS_OPENMP
+  omp_set_num_threads(1);
+  #endif
+
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after first training.
+  const arma::mat parameters1 = std::move(
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters());
+
+  // Reset the settings.
+  bindings::tests::CleanMemory();
+  CLI::ClearSettings();
+  CLI::RestoreSettings(testName);
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("optimizer", std::string("psgd"));
+  SetInputParam("epochs", int(10));
+
+  // Second solution.
+  #ifdef HAS_OPENMP
+  omp_set_num_threads(1);
+  #endif
+
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after second training.
+  const arma::mat& parameters2 =
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters();
+
+  // Both solutions should be not equal.
+  CheckMatricesNotEqual(parameters1, parameters2);
+}
+
+/**
+ * Ensuring that Step size has some effects on the output.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMDiffStepSizeTest)
+{
+  arma::mat trainData = "2 0 0;"
+                        "0 0 0;"
+                        "0 2 1;"
+                        "1 0 2;"
+                        "0 1 0";
+
+  arma::Row<size_t> trainLabels = "1 0 1";
+
+  SetInputParam("training", trainData);
+  SetInputParam("labels", trainLabels);
+  SetInputParam("optimizer", std::string("psgd"));
+  SetInputParam("num_classes", int(2));
+  SetInputParam("step_size", double(0.02));
+
+  // First solution.
+  #ifdef HAS_OPENMP
+  omp_set_num_threads(1);
+  #endif
+
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after first training.
+  const arma::mat parameters1 = std::move(
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters());
+
+  // Reset the settings.
+  bindings::tests::CleanMemory();
+  CLI::ClearSettings();
+  CLI::RestoreSettings(testName);
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("optimizer", std::string("psgd"));
+  SetInputParam("num_classes", int(2));
+  SetInputParam("step_size", double(1.02));
+
+  // Second solution.
+  #ifdef HAS_OPENMP
+  omp_set_num_threads(1);
+  #endif
+
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after second training.
+  const arma::mat& parameters2 =
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters();
+
+  // Both solutions should be not equal.
+  CheckMatricesNotEqual(parameters1, parameters2);
+}
+
+/**
+ * Ensuring that tolerance has some effects on the output.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMDiffToleranceTest)
+{
+  arma::mat trainData = "2 0 0;"
+                        "0 0 0;"
+                        "0 2 1;"
+                        "1 0 2;"
+                        "0 1 0";
+
+  arma::Row<size_t> trainLabels = "1 0 1";
+
+  SetInputParam("training", trainData);
+  SetInputParam("labels", trainLabels);
+  SetInputParam("optimizer", std::string("psgd"));
+  SetInputParam("num_classes", int(2));
+  SetInputParam("tolerance", double(1e-1));
+
+  // First solution.
+  #ifdef HAS_OPENMP
+  omp_set_num_threads(1);
+  #endif
+
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after first training.
+  const arma::mat parameters1 = std::move(
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters());
+
+  // Reset the settings.
+  bindings::tests::CleanMemory();
+  CLI::ClearSettings();
+  CLI::RestoreSettings(testName);
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("optimizer", std::string("psgd"));
+  SetInputParam("num_classes", int(2));
+  SetInputParam("tolerance", double(1e-10));
+
+  // Second solution.
+  #ifdef HAS_OPENMP
+  omp_set_num_threads(1);
+  #endif
+
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after second training.
+  const arma::mat& parameters2 =
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters();
+
+  // Both solutions should be not equal.
+  CheckMatricesNotEqual(parameters1, parameters2);
+}
+
+/**
+ * Ensuring that lbfgs optimizer converges to a different result than psgd.
+ */
+BOOST_AUTO_TEST_CASE(LinearSVMDiffOptimizerTest)
+{
+  arma::mat trainData = "2 0 0;"
+                        "0 0 0;"
+                        "0 2 1;"
+                        "1 0 2;"
+                        "0 1 0";
+
+  arma::Row<size_t> trainLabels = "1 0 1";
+
+  SetInputParam("training", trainData);
+  SetInputParam("labels", trainLabels);
+  SetInputParam("optimizer", std::string("lbfgs"));
+
+  // First solution.
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after first training.
+  const arma::mat parameters1 = std::move(
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters());
+
+  // Reset the settings.
+  bindings::tests::CleanMemory();
+  CLI::ClearSettings();
+  CLI::RestoreSettings(testName);
+
+  SetInputParam("training", std::move(trainData));
+  SetInputParam("labels", std::move(trainLabels));
+  SetInputParam("optimizer", std::string("psgd"));
+
+  // Second solution.
+  #ifdef HAS_OPENMP
+  omp_set_num_threads(1);
+  #endif
+
+  mlpack::math::FixedRandomSeed();
+  mlpackMain();
+
+  // Get the parameters of the output model obtained after second training.
+  const arma::mat& parameters2 =
+      CLI::GetParam<LinearSVMModel*>("output_model")->svm.Parameters();
+
+  // Both solutions should be not equal.
+  CheckMatricesNotEqual(parameters1, parameters2);
+}
+
+BOOST_AUTO_TEST_SUITE_END();
+

src/mlpack/tests/wgan_test.cpp

@@ -22,6 +22,7 @@
 
 #include <boost/test/unit_test.hpp>
 #include "test_tools.hpp"
+#include "serialization.hpp"
 
 using namespace mlpack;
 using namespace mlpack::ann;
@@ -76,7 +77,7 @@ BOOST_AUTO_TEST_CASE(WGANMNISTTest)
             << trainData.n_cols << ")" << std::endl;
   Log::Info << trainData.n_rows << "--------" << trainData.n_cols << std::endl;
 
-  // Create the Discriminator network
+  // Create the Discriminator network.
   FFN<EarthMoverDistance<> > discriminator;
   discriminator.Add<Convolution<> >(1, dNumKernels, 4, 4, 2, 2, 1, 1, 28, 28);
   discriminator.Add<LeakyReLU<> >(0.2);
@@ -93,46 +94,46 @@ BOOST_AUTO_TEST_CASE(WGANMNISTTest)
       1, 1, 2, 2);
   discriminator.Add<SigmoidLayer<> >();
 
-  // Create the Generator network
+  // Create the Generator network.
   FFN<EarthMoverDistance<> > generator;
   generator.Add<TransposedConvolution<> >(noiseDim, 8 * dNumKernels, 2, 2,
-      1, 1, 1, 1, 1, 1);
+      1, 1, 0, 0, 1, 1, 2, 2);
   generator.Add<BatchNorm<> >(1024);
   generator.Add<ReLULayer<> >();
   generator.Add<TransposedConvolution<> >(8 * dNumKernels, 4 * dNumKernels,
-      2, 2, 1, 1, 0, 0, 2, 2);
+      2, 2, 1, 1, 0, 0, 2, 2, 3, 3);
   generator.Add<BatchNorm<> >(1152);
   generator.Add<ReLULayer<> >();
   generator.Add<TransposedConvolution<> >(4 * dNumKernels, 2 * dNumKernels,
-      5, 5, 2, 2, 1, 1, 3, 3);
+      5, 5, 2, 2, 1, 1, 3, 3, 7, 7);
   generator.Add<BatchNorm<> >(3136);
   generator.Add<ReLULayer<> >();
-  generator.Add<TransposedConvolution<> >(2 * dNumKernels, dNumKernels, 8, 8,
-      1, 1, 1, 1, 7, 7);
+  generator.Add<TransposedConvolution<> >(2 * dNumKernels, dNumKernels, 4, 4,
+      2, 2, 1, 1, 7, 7, 14, 14);
   generator.Add<BatchNorm<> >(6272);
   generator.Add<ReLULayer<> >();
-  generator.Add<TransposedConvolution<> >(dNumKernels, 1, 15, 15, 1, 1, 1, 1,
-      14, 14);
+  generator.Add<TransposedConvolution<> >(dNumKernels, 1, 4, 4, 2, 2, 1, 1,
+      14, 14, 28, 28);
   generator.Add<TanHLayer<> >();
 
-  // Create WGAN
+  // Create WGAN.
   GaussianInitialization gaussian(0, 1);
   ens::Adam optimizer(stepSize, batchSize, 0.9, 0.999, eps, numIterations,
       tolerance, shuffle);
   std::function<double()> noiseFunction = [] () {
       return math::RandNormal(0, 1);};
   GAN<FFN<EarthMoverDistance<> >, GaussianInitialization,
-      std::function<double()>, WGAN> wgan(trainData, generator, discriminator,
-      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      std::function<double()>, WGAN> wgan(generator, discriminator, gaussian,
+      noiseFunction, noiseDim, batchSize, generatorUpdateStep,
       discriminatorPreTrain, multiplier, clippingParameter);
 
   Log::Info << "Training..." << std::endl;
-  double objVal = wgan.Train(optimizer);
+  double objVal = wgan.Train(trainData, optimizer);
 
   // Test that objective value returned by GAN::Train() is finite.
   BOOST_REQUIRE_EQUAL(std::isfinite(objVal), true);
 
-  // Generate samples
+  // Generate samples.
   Log::Info << "Sampling..." << std::endl;
   arma::mat noise(noiseDim, batchSize);
   size_t dim = std::sqrt(trainData.n_rows);
@@ -158,6 +159,38 @@ BOOST_AUTO_TEST_CASE(WGANMNISTTest)
   }
 
   Log::Info << "Output generated!" << std::endl;
+
+  // Check that Serialization is working correctly.
+  arma::mat orgPredictions;
+  wgan.Predict(noise, orgPredictions);
+
+  GAN<FFN<EarthMoverDistance<> >, GaussianInitialization,
+      std::function<double()>, WGAN> wganText(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  GAN<FFN<EarthMoverDistance<> >, GaussianInitialization,
+      std::function<double()>, WGAN> wganXml(generator, discriminator, gaussian,
+      noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  GAN<FFN<EarthMoverDistance<> >, GaussianInitialization,
+      std::function<double()>, WGAN> wganBinary(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  SerializeObjectAll(wgan, wganXml, wganText, wganBinary);
+
+  arma::mat predictions, xmlPredictions, textPredictions, binaryPredictions;
+  wgan.Predict(noise, predictions);
+  wganXml.Predict(noise, xmlPredictions);
+  wganText.Predict(noise, textPredictions);
+  wganBinary.Predict(noise, binaryPredictions);
+
+  CheckMatrices(orgPredictions, predictions);
+  CheckMatrices(orgPredictions, xmlPredictions);
+  CheckMatrices(orgPredictions, textPredictions);
+  CheckMatrices(orgPredictions, binaryPredictions);
 }
 
 /*
@@ -206,7 +239,7 @@ BOOST_AUTO_TEST_CASE(WGANGPMNISTTest)
             << trainData.n_cols << ")" << std::endl;
   Log::Info << trainData.n_rows << "--------" << trainData.n_cols << std::endl;
 
-  // Create the Discriminator network
+  // Create the Discriminator network.
   FFN<EarthMoverDistance<> > discriminator;
   discriminator.Add<Convolution<> >(1, dNumKernels, 4, 4, 2, 2, 1, 1, 28, 28);
   discriminator.Add<LeakyReLU<> >(0.2);
@@ -223,47 +256,46 @@ BOOST_AUTO_TEST_CASE(WGANGPMNISTTest)
       1, 1, 2, 2);
   discriminator.Add<SigmoidLayer<> >();
 
-  // Create the Generator network
+  // Create the Generator network.
   FFN<EarthMoverDistance<> > generator;
   generator.Add<TransposedConvolution<> >(noiseDim, 8 * dNumKernels, 2, 2,
-      1, 1, 1, 1, 1, 1);
+      1, 1, 0, 0, 1, 1, 2, 2);
   generator.Add<BatchNorm<> >(1024);
   generator.Add<ReLULayer<> >();
   generator.Add<TransposedConvolution<> >(8 * dNumKernels, 4 * dNumKernels,
-      2, 2, 1, 1, 0, 0, 2, 2);
+      2, 2, 1, 1, 0, 0, 2, 2, 3, 3);
   generator.Add<BatchNorm<> >(1152);
   generator.Add<ReLULayer<> >();
   generator.Add<TransposedConvolution<> >(4 * dNumKernels, 2 * dNumKernels,
-      5, 5, 2, 2, 1, 1, 3, 3);
+      5, 5, 2, 2, 1, 1, 3, 3, 7, 7);
   generator.Add<BatchNorm<> >(3136);
   generator.Add<ReLULayer<> >();
-  generator.Add<TransposedConvolution<> >(2 * dNumKernels, dNumKernels, 8, 8,
-      1, 1, 1, 1, 7, 7);
+  generator.Add<TransposedConvolution<> >(2 * dNumKernels, dNumKernels, 4, 4,
+      2, 2, 1, 1, 7, 7, 14, 14);
   generator.Add<BatchNorm<> >(6272);
   generator.Add<ReLULayer<> >();
-  generator.Add<TransposedConvolution<> >(dNumKernels, 1, 15, 15, 1, 1, 1, 1,
-      14, 14);
+  generator.Add<TransposedConvolution<> >(dNumKernels, 1, 4, 4, 2, 2, 1, 1,
+      14, 14, 28, 28);
   generator.Add<TanHLayer<> >();
 
-  // Create WGANGP
+  // Create WGANGP.
   GaussianInitialization gaussian(0, 1);
   ens::Adam optimizer(stepSize, batchSize, 0.9, 0.999, eps, numIterations,
       tolerance, shuffle);
   std::function<double()> noiseFunction = [] () {
       return math::RandNormal(0, 1);};
   GAN<FFN<EarthMoverDistance<> >, GaussianInitialization,
-      std::function<double()>, WGANGP > wganGP(trainData, generator,
-      discriminator, gaussian, noiseFunction, noiseDim, batchSize,
-      generatorUpdateStep, discriminatorPreTrain, multiplier, clippingParameter,
-      lambda);
+      std::function<double()>, WGANGP> wganGP(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier, clippingParameter, lambda);
 
   Log::Info << "Training..." << std::endl;
-  double objVal = wganGP.Train(optimizer);
+  double objVal = wganGP.Train(trainData, optimizer);
 
   // Test that objective value returned by GAN::Train() is finite.
   BOOST_REQUIRE_EQUAL(std::isfinite(objVal), true);
 
-  // Generate samples
+  // Generate samples.
   Log::Info << "Sampling..." << std::endl;
   arma::mat noise(noiseDim, batchSize);
   size_t dim = std::sqrt(trainData.n_rows);
@@ -289,6 +321,38 @@ BOOST_AUTO_TEST_CASE(WGANGPMNISTTest)
   }
 
   Log::Info << "Output generated!" << std::endl;
+
+  // Check that Serialization is working correctly.
+  arma::mat orgPredictions;
+  wganGP.Predict(noise, orgPredictions);
+
+  GAN<FFN<EarthMoverDistance<> >, GaussianInitialization,
+      std::function<double()>, WGANGP> wganGPText(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  GAN<FFN<EarthMoverDistance<> >, GaussianInitialization,
+      std::function<double()>, WGANGP> wganGPXml(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  GAN<FFN<EarthMoverDistance<> >, GaussianInitialization,
+      std::function<double()>, WGANGP> wganGPBinary(generator, discriminator,
+      gaussian, noiseFunction, noiseDim, batchSize, generatorUpdateStep,
+      discriminatorPreTrain, multiplier);
+
+  SerializeObjectAll(wganGP, wganGPXml, wganGPText, wganGPBinary);
+
+  arma::mat predictions, xmlPredictions, textPredictions, binaryPredictions;
+  wganGP.Predict(noise, predictions);
+  wganGPXml.Predict(noise, xmlPredictions);
+  wganGPText.Predict(noise, textPredictions);
+  wganGPBinary.Predict(noise, binaryPredictions);
+
+  CheckMatrices(orgPredictions, predictions);
+  CheckMatrices(orgPredictions, xmlPredictions);
+  CheckMatrices(orgPredictions, textPredictions);
+  CheckMatrices(orgPredictions, binaryPredictions);
 }
 
 BOOST_AUTO_TEST_SUITE_END();