diff --git a/COPYRIGHT.txt b/COPYRIGHT.txt index 609b873fb0..e02c52d769 100644 --- a/COPYRIGHT.txt +++ b/COPYRIGHT.txt @@ -96,6 +96,8 @@ Copyright: Copyright 2018, Roberto Hueso Copyright 2018, Prabhat Sharma Copyright 2018, Tan Jun An + Copyright 2018, Moksh Jain + Copyright 2018, Manthan-R-Sheth License: BSD-3-clause All rights reserved. diff --git a/doc/tutorials/README.md b/doc/tutorials/README.md index 0e68acd6a8..500f1f31ee 100644 --- a/doc/tutorials/README.md +++ b/doc/tutorials/README.md @@ -1,7 +1,7 @@ ## Tutorials -Tutorials for mlpack can be found [here : mlpack tutorials](http://www.mlpack.org/tutorials.html). +Tutorials for mlpack can be found [here : mlpack tutorials](https://www.mlpack.org/docs/mlpack-git/doxygen/tutorials.html). ### General mlpack tutorials diff --git a/doc/tutorials/ann/ann.txt b/doc/tutorials/ann/ann.txt new file mode 100644 index 0000000000..546e39de11 --- /dev/null +++ b/doc/tutorials/ann/ann.txt @@ -0,0 +1,674 @@ +/*! +@file ann.txt +@author Marcus Edel (kurg.org) +@brief Tutorial for how to use the neural network code in mlpack. + +@page anntutorial Neural Network tutorial + +@section intro_anntut Introduction + +There is vast literature on neural networks and their uses, as well as +strategies for choosing initial points effectively, keeping the algorithm from +converging in local minima, choosing the best model structure, choosing the best +optimizers, and so forth. mlpack implements many of these building blocks, +making it very easy to create different neural networks in a modular way. + +mlpack currently implements two easy-to-use forms of neural networks: \c Feed- +Forward \c Networks (this includes convolutional neural networks) and \c +Recurrent \c Neural \c Networks. + +@section toc_anntut Table of Contents + +This tutorial is split into the following sections: + + - \ref intro_anntut + - \ref toc_anntut + - \ref model_api_anntut + - \ref layer_api_anntut + - \ref model_setup_training_anntut + - \ref model_saving_loading_anntut + - \ref extracting_parameters_anntut + - \ref further_anntut + +@section model_api_anntut Model API + +There are two main neural network classes that are meant to be used as container +for neural network layers that \b mlpack implements; each class is suited to a +different setting: + +- \c FFN: the Feed Forward Network model provides a means to plug layers + together in a feed-forward fully connected manner. This is the 'standard' + type of deep learning model, and includes convolutional neural networks + (CNNs). + +- \c RNN: the Recurrent Neural Network model provides a means to consider + successive calls to forward as different time-steps in a sequence. This is + often used for time sequence modeling tasks, such as predicting the next + character in a sequence. + +Below is some basic guidance on what should be used. Note that the question of +"which algorithm should be used" is a very difficult question to answer, so the +guidance below is just that---guidance---and may not be right for a particular +problem. + + - \c Feed-forward Networks allow signals or inputs to travel one way only. + There is no feedback within the network; for instance, the output of any + layer does only affect the upcoming layer. That makes Feed-Forward Networks + straightforward and very effective. They are extensively used in pattern + recognition and are ideally suitable for modeling relationships between a + set of input and one or more output variables. + + + - \c Recurrent Networks allow signals or inputs to travel in both directions by + introducing loops in the network. Computations derived from earlier inputs are + fed back into the network, which gives the recurrent network some kind of + memory. RNNs are currently being used for all kinds of sequential tasks; for + instance, time series prediction, sequence labeling, and + sequence classification. + +In order to facilitate consistent implementations, the \c FFN and \c RNN classes +have a number of methods in common: + + - \c Train(): trains the initialized model on the given input data. Optionally + an optimizer object can be passed to control the optimization process. + + - \c Predict(): predicts the responses to a given set of predictors. Note the + responses will reflect the output of the specified output layer. + + - \c Add(): this method can be used to add a layer to the model. + +@note +To be able to optimize the network, both classes implement the OptimizerFunction +API; see \ref optimizertutorial "Optimizer API" for more information. In short, +the \c FNN and \c RNN class implement two methods: \c Evaluate() and \c +Gradient(). This enables the optimization given some learner and some +performance measure. + +Similar to the existing layer infrastructure, the \c FFN and \c RNN classes are +very extensible, having the following template arguments; which can be modified +to change the behavior of the network: + + - \c OutputLayerType: this type defines the output layer used to evaluate the + network; by default, \c NegativeLogLikelihood is used. + + - \c InitializationRuleType: this type defines the method by which initial + parameters are set; by default, \c RandomInitialization is used. + +@code +template< + typename OutputLayerType = NegativeLogLikelihood<>, + typename InitializationRuleType = RandomInitialization +> +class FNN; +@endcode + +Internally, the \c FFN and \c RNN class keeps an instantiated \c OutputLayerType +class (which can be given in the constructor). This is useful for using +different loss functions like the Negative-Log-Likelihood function or the \c +VRClassReward function, which takes an optional score parameter. Therefore, you +can write a non-static OutputLayerType class and use it seamlessly in +combination with the \c FNN and \c RNN class. The same applies to the \c +InitializationRuleType template parameter. + +By choosing different components for each of these template classes in +conjunction with the \c Add() method, a very arbitrary network object can be +constructed. + +Below are several examples of how the \c FNN and \c RNN classes might be used. +The first examples focus on the \c FNN class, and the last shows how the \c +RNN class can be used. + +The simplest way to use the FNN<> class is to pass in a dataset with the +corresponding labels, and receive the classification in return. Note that the +dataset must be column-major – that is, one column corresponds to one point. See +the \ref matrices "matrices guide" for more information. + +The code below builds a simple feed-forward network with the default options, +then queries for the assignments for every point in the \c queries matrix. + +\dot +digraph G { + fontname = "Hilda 10" + rankdir=LR + splines=line + nodesep=.08; + ranksep=1; + edge [color=black, arrowsize=.5]; + node [fixedsize=true,label="",style=filled,color=none,fillcolor=gray,shape=circle] + + subgraph cluster_0 { + color=none; + node [style=filled, color=white, penwidth=15,fillcolor=black shape=circle]; + l10 l11 l12 l13 l14 l15 ; + label = Input; + } + + subgraph cluster_1 { + color=none; + node [style=filled, color=white, penwidth=15,fillcolor=gray shape=circle]; + l20 l21 l22 l23 l24 l25 l26 l27 ; + label = Linear; + } + + subgraph cluster_2 { + color=none; + node [style=filled, color=white, penwidth=15,fillcolor=gray shape=circle]; + l30 l31 l32 l33 l34 l35 l36 l37 ; + label = Linear; + } + + subgraph cluster_3 { + color=none; + node [style=filled, color=white, penwidth=15,fillcolor=black shape=circle]; + l40 l41 l42 ; + label = LogSoftMax; + } + + l10 -> l20 l10 -> l21 l10 -> l22 l10 -> l23 l10 -> l24 l10 -> l25 + l10 -> l26 l10 -> l27 l11 -> l20 l11 -> l21 l11 -> l22 l11 -> l23 + l11 -> l24 l11 -> l25 l11 -> l26 l11 -> l27 l12 -> l20 l12 -> l21 + l12 -> l22 l12 -> l23 l12 -> l24 l12 -> l25 l12 -> l26 l12 -> l27 + l13 -> l20 l13 -> l21 l13 -> l22 l13 -> l23 l13 -> l24 l13 -> l25 + l13 -> l26 l13 -> l27 l14 -> l20 l14 -> l21 l14 -> l22 l14 -> l23 + l14 -> l24 l14 -> l25 l14 -> l26 l14 -> l27 l15 -> l20 l15 -> l21 + l15 -> l22 l15 -> l23 l15 -> l24 l15 -> l25 l15 -> l26 l15 -> l27 + l20 -> l30 l20 -> l31 l20 -> l32 l20 -> l33 l20 -> l34 l20 -> l35 + l20 -> l36 l20 -> l37 l21 -> l30 l21 -> l31 l21 -> l32 l21 -> l33 + l21 -> l34 l21 -> l35 l21 -> l36 l21 -> l37 l22 -> l30 l22 -> l31 + l22 -> l32 l22 -> l33 l22 -> l34 l22 -> l35 l22 -> l36 l22 -> l37 + l23 -> l30 l23 -> l31 l23 -> l32 l23 -> l33 l23 -> l34 l23 -> l35 + l23 -> l36 l23 -> l37 l24 -> l30 l24 -> l31 l24 -> l32 l24 -> l33 + l24 -> l34 l24 -> l35 l24 -> l36 l24 -> l37 l25 -> l30 l25 -> l31 + l25 -> l32 l25 -> l33 l25 -> l34 l25 -> l35 l25 -> l36 l25 -> l37 + l26 -> l30 l26 -> l31 l26 -> l32 l26 -> l33 l26 -> l34 l26 -> l35 + l26 -> l36 l26 -> l37 l27 -> l30 l27 -> l31 l27 -> l32 l27 -> l33 + l27 -> l34 l27 -> l35 l27 -> l36 l27 -> l37 l30 -> l40 l30 -> l41 + l30 -> l42 l31 -> l40 l31 -> l41 l31 -> l42 l32 -> l40 l32 -> l41 + l32 -> l42 l33 -> l40 l33 -> l41 l33 -> l42 l34 -> l40 l34 -> l41 + l34 -> l42 l35 -> l40 l35 -> l41 l35 -> l42 l36 -> l40 l36 -> l41 + l36 -> l42 l37 -> l40 l37 -> l41 l37 -> l42 +} +\enddot +@note +The number of inputs in the above graph doesn't match with the real +number of features in the thyroid dataset and are just used as an abstract +representation. + +@code +// Load the training set. +arma::mat dataset; +data::Load("thyroid_train.csv", dataset, true); + +// Split the labels from the training set. +arma::mat trainData = dataset.submat(0, 0, dataset.n_rows - 4, + dataset.n_cols - 1); + +// Split the data from the training set. +arma::mat trainLabelsTemp = dataset.submat(dataset.n_rows - 3, 0, + dataset.n_rows - 1, dataset.n_cols - 1); + +// Initialize the network. +FFN<> model; +model.Add >(trainData.n_rows, 8); +model.Add >(); +model.Add >(8, 3); +model.Add >(); + +// Train the model. +model.Train(trainData, trainLabels); + +// Use the Predict method to get the assignments. +arma::mat assignments; +model.Predict(trainData, assignments); +@endcode + +Now, the matrix assignments holds the classification of each point in the +dataset. + +In the next example, we create simple noisy sine sequences, which are trained +later on, using the RNN class. + +@code +void GenerateNoisySines(arma::mat& data, + arma::mat& labels, + const size_t points, + const size_t sequences, + const double noise = 0.3) +{ + arma::colvec x = arma::linspace>(0, + points - 1, points) / points * 20.0; + arma::colvec y1 = arma::sin(x + arma::as_scalar(arma::randu(1)) * 3.0); + arma::colvec y2 = arma::sin(x / 2.0 + arma::as_scalar(arma::randu(1)) * 3.0); + + data = arma::zeros(points, sequences * 2); + labels = arma::zeros(2, sequences * 2); + + for (size_t seq = 0; seq < sequences; seq++) + { + data.col(seq) = arma::randu(points) * noise + y1 + + arma::as_scalar(arma::randu(1) - 0.5) * noise; + labels(0, seq) = 1; + + data.col(sequences + seq) = arma::randu(points) * noise + y2 + + arma::as_scalar(arma::randu(1) - 0.5) * noise; + labels(1, sequences + seq) = 1; + } + + const size_t rho = 10; + + // Generate 12 (2 * 6) noisy sines. A single sine contains rho + // points/features. + arma::mat input, labelsTemp; + GenerateNoisySines(input, labelsTemp, rho, 6); + + arma::mat labels = arma::zeros(rho, labelsTemp.n_cols); + for (size_t i = 0; i < labelsTemp.n_cols; ++i) + { + const int value = arma::as_scalar(arma::find( + arma::max(labelsTemp.col(i)) == labelsTemp.col(i), 1)) + 1; + labels.col(i).fill(value); + } + + /** + * Construct a network with 1 input unit, 4 hidden units and 10 output + * units. The hidden layer is connected to itself. The network structure + * looks like: + * + * Input Hidden Output + * Layer(1) Layer(4) Layer(10) + * +-----+ +-----+ +-----+ + * | | | | | | + * | +------>| +------>| | + * | | ..>| | | | + * +-----+ . +--+--+ +-----+ + * . . + * . . + * ....... + */ + Add<> add(4); + Linear<> lookup(1, 4); + SigmoidLayer<> sigmoidLayer; + Linear<> linear(4, 4); + Recurrent<> recurrent(add, lookup, linear, sigmoidLayer, rho); + + RNN<> model(rho); + model.Add >(); + model.Add(recurrent); + model.Add >(4, 10); + model.Add >(); + + StandardSGD opt(0.1, 1, input.n_cols /* 1 epoch */, -100); + model.Train(input, labels, opt); +} +@endcode + +For further examples on the usage of the ann classes, see [mlpack +models](https://github.com/mlpack/models). + +@section layer_api_anntut Layer API + +In order to facilitate consistent implementations, we have defined a LayerType +API that describes all the methods that a \c layer may implement. mlpack offers +a few variations of this API, each designed to cover some of the model +characteristics mentioned in the previous section. Any \c layer requires the +implementation of a \c Forward() method. The interface looks like: + +@code +template +void Forward(const arma::Mat&& input, arma::Mat&& output); +@endcode + +The method should calculate the output of the layer given the input matrix and +store the result in the given output matrix. Next, any \c layer must implement +the Backward() method, which uses certain computations obtained during the +forward pass and should calculate the function f(x) by propagating x backward +through f: + +@code +template +void Backward(const arma::Mat&& input, + arma::Mat&& gy, + arma::Mat&& g); +@endcode + +Finally, if the layer is differentiable, the layer must also implement +a Gradient() method: + +@code +template +void Gradient(const arma::Mat&& input, + arma::Mat&& error, + arma::Mat&& gradient); +@endcode + +The Gradient function should calculate the gradient with respect to the input +activations \c input and calculated errors \c error and place the results into +the gradient matrix object \c gradient that is passed as an argument. + +@note +Note that each method accepts a template parameter InputType, OutputType +or GradientType, which may be arma::mat (dense Armadillo matrix) or arma::sp_mat +(sparse Armadillo matrix). This allows support for both sparse-supporting and +non-sparse-supporting \c layer without explicitly passing the type. + +In addition, each layer must implement the Parameters(), InputParameter(), +OutputParameter(), Delta() methods, differentiable layer should also provide +access to the gradient by implementing the Gradient(), Parameters() member +function. Note each function is a single line that looks like: + +@code +OutputDataType const& Parameters() const { return weights; } +@endcode + +Below is an example that shows each function with some additional boilerplate +code. + +@note +Note this is not an actual layer but instead an example that exists to show and +document all the functions that mlpack layer must implement. For a better +overview of the various layers, see \ref mlpack::ann. Also be aware that the +implementations of each of the methods in this example are entirely fake and do +not work; this example exists for its API, not its implementation. + +Note that layer sometimes have different properties. These properties are +known at compile-time through the mlpack::ann::LayerTraits class, and some +properties may imply the existence (or non-existence) of certain functions. +Refer to the LayerTraits @ref LayerTraits for more documentation on that. + +The two template parameters below must be template parameters to the layer, in +the order given below. More template parameters are fine, but they must come +after the first two. + + - \c InputDataType: this defines the internally used input type for example to + store the parameter matrix. Note, a layer could be built on a dense matrix or + a sparse matrix. All mlpack trees should be able to support any Armadillo- + compatible matrix type. When the layer is written it should be assumed that + MatType has the same functionality as arma::mat. Note that + + - \c OutputDataType: this defines the internally used input type for example to + store the parameter matrix. Note, a layer could be built on a dense matrix or + a sparse matrix. All mlpack trees should be able to support any Armadillo- + compatible matrix type. When the layer is written it should be assumed that + MatType has the same functionality as arma::mat. + +@code +template +class ExampleLayer +{ + public: + ExampleLayer(const size_t inSize, const size_t outSize) : + inputSize(inSize), outputSize(outSize) + { + /* Nothing to do here */ + } +} +@endcode + +The constructor for \c ExampleLayer will build the layer given the input and +output size. Note that, if the input or output size information isn't used +internally it's not necessary to provide a specific constructor. Also, one could +add additional or other information that are necessary for the layer +construction. One example could be: + +@code +ExampleLayer(const double ratio = 0.5) : ratio(ratio) {/* Nothing to do here*/} +@endcode + +When this constructor is finished, the entire layer will be built and is ready +to be used. Next, as pointed out above, each layer has to follow the LayerType +API, so we must implement some additional functions. + +@code +template +void Forward(const InputType&& input, OutputType&& output) +{ + output = arma::ones(input.n_rows, input.n_cols); +} + +template +void Backward(const InputType&& input, ErrorType&& gy, GradientType&& g) +{ + g = arma::zeros(gy.n_rows, gy.n_cols) + gy; +} + +template +void Gradient(const InputType&& input, + ErrorType&& error, + GradientType&& gradient) +{ + gradient = arma::zeros(input.n_rows, input.n_cols) * error; +} +@endcode + +The three functions \c Forward(), \c Backward() and \c Gradient() (which is +needed for a differentiable layer) contain the main logic of the layer. The +following functions are just to access and manipulate the different layer +parameters. + +@code +OutputDataType& Parameters() { return weights; } +InputDataType& InputParameter() { return inputParameter; } +OutputDataType& OutputParameter() { return outputParameter; } +OutputDataType& Delta() { return delta; } +OutputDataType& Gradient() { return gradient; } +@endcode + +Since some of this methods return internal class members we have to define them. + +@code +private: + size_t inSize, outSize; + OutputDataType weights, delta, gradient, outputParameter; + InputDataType inputParameter; +@endcode + +Note some members are just here so \c ExampleLayer compiles without warning. +For instance, \c inputSize is not required to be a member of every type of +layer. + +There is one last method that is especially interesting for a layer that shares +parameter. Since the layer weights are set once the complete model is defined, +it's not possible to split the weights during the construction time. To solve +this issue, a layer can implement the \c Reset() method which is called once the +layer parameter is set. + +@section model_setup_training_anntut Model Setup & Training + +Once the base container is selected (\c FNN or \c RNN), the \c Add method can be +used to add layers to the model. The code below adds two linear layers to the +model---the first takes 512 units as input and gives 256 output units, and +the second takes 256 units as input and gives 128 output units. + +@code +FFN<> model; +model.Add >(512, 256); +model.Add >(256, 128); +@endcode + +The model is trained on Armadillo matrices. For training a model, you will +typically use the \c Train() function: + +@code +arma::mat trainingSet, trainingLabels; +model.Train(trainingSet, trainingLabels); +@endcode + +You can use mlpack's \c Load() function to load a dataset like this: + +@code +arma::mat trainingSet; +data::Load("dataset.csv", dataset, true); +@endcode + +@code +$ cat dataset.csv +0, 1, 4 +1, 0, 5 +1, 1, 1 +2, 0, 2 +@endcode + +The type does not necessarily need to be a CSV; it can be any supported storage +format, assuming that it is a coordinate-format file in the format specified +above. For more information on mlpack file formats, see the documentation for +mlpack::data::Load(). + +@note +It’s often a good idea to normalize or standardize your data, for example using: + +@code +for (size_t i = 0; i < dataset.n_cols; ++i) + dataset.col(i) /= norm(dataset.col(i), 2); +@endcode + +Also, it is possible to retrain a model with new parameters or with +a new reference set. This is functionally equivalent to creating a new model. + +@section model_saving_loading_anntut Saving & Loading + +Using \c boost::serialization (for more information about the internals see +[Serialization - Boost C++ Libraries](www.boost.org/libs/serialization/doc/)), +mlpack is able to load and save machine learning models with ease. To save a +trained neural network to disk. The example below builds a model on the \c +thyroid dataset and then saves the model to the file \c model.xml for later use. + +@code +// Load the training set. +arma::mat dataset; +data::Load("thyroid_train.csv", dataset, true); + +// Split the labels from the training set. +arma::mat trainData = dataset.submat(0, 0, dataset.n_rows - 4, + dataset.n_cols - 1); + +// Split the data from the training set. +arma::mat trainLabelsTemp = dataset.submat(dataset.n_rows - 3, 0, + dataset.n_rows - 1, dataset.n_cols - 1); + +// Initialize the network. +FFN<> model; +model.Add >(trainData.n_rows, 3); +model.Add >(); +model.Add >(); + +// Train the model. +model.Train(trainData, trainLabels); + +// Use the Predict method to get the assignments. +arma::mat assignments; +model.Predict(trainData, assignments); + +data::Save("model.xml", "model", model, false); +@endcode + +After this, the file model.xml will be available in the current working +directory. + +Now, we can look at the output model file, \c model.xml: + +@code +$ cat model.xml + + + + + + 66 + 1 + 66 + 0 + -7.55971528334903642e+00 + -9.95435955058058930e+00 + 9.31133928948225353e+00 + -5.36784434861701953e+00 + ... + + 0 + 0 + + 0 + 0 + 0 + 0 + + + 3 + 0 + + 18 + + 21 + 3 + + + + 2 + + + + 20 + + + + + +@endcode + +As you can see, the \c section of \c model.xml contains the trained +network weights. We can see that this section also contains the network input +size, which is 66 rows and 1 column. Note that in this example, we used three +different layers, as can be seen by looking at the \c section. Each +node has a unique id that is used to reconstruct the model when loading. + +The models can also be saved as \c .bin or \c .txt; the \c .xml format provides +a human-inspectable format (though the models tend to be quite complex and may +be difficult to read). These models can then be re-used to be used for +classification or other tasks. + +So, instead of saving or training a network, mlpack can also load a pre-trained +model. For instance, the example below will load the model from \c model.xml and +then generate the class predictions for the \c thyroid test dataset. + +@code +data::Load("thyroid_test.csv", dataset, true); + +arma::mat testData = dataset.submat(0, 0, dataset.n_rows - 4, + dataset.n_cols - 1); + +data::Load("model.xml", "model", model); + +arma::mat predictions; +model.Predict(testData, predictions); +@endcode + +This enables the possibility to distribute a model without having to train it +first or simply to save a model for later use. Note that loading will also work +on different machines. + +@section extracting_parameters_anntut Extracting Parameters + +To access the weights from the neural network layers, you can call the following +function on any initialized network: + +@code +model.Parameters(); +@endcode + +which will return the complete model parameters as an armadillo matrix object; +however often it is useful to not only have the parameters for the complete +network, but the parameters of a specific layer. Another method, \c Model(), +makes this easily possible: + +@code +model.Model()[1].Parameters(); +@endcode + +In the example above, we get the weights of the second layer. + +@section further_anntut Further documentation + +For further documentation on the ann classes, consult the \ref mlpack::ann +"complete API documentation". + +*/ diff --git a/doc/tutorials/tutorials.txt b/doc/tutorials/tutorials.txt index 169902e54d..a8df927130 100644 --- a/doc/tutorials/tutorials.txt +++ b/doc/tutorials/tutorials.txt @@ -39,6 +39,7 @@ progress to complex, extensible uses. - \ref cftutorial - \ref akfntutorial - \ref cnetutorial + - \ref anntutorial @section policy_tut Policy Class Documentation diff --git a/src/mlpack/core.hpp b/src/mlpack/core.hpp index 845ccc723a..6a0d6b7e03 100644 --- a/src/mlpack/core.hpp +++ b/src/mlpack/core.hpp @@ -239,6 +239,8 @@ * - Roberto Hueso * - Prabhat Sharma * - Tan Jun An + * - Moksh Jain + * - Manthan-R-Sheth */ // First, include all of the prerequisites. diff --git a/src/mlpack/core/math/random.hpp b/src/mlpack/core/math/random.hpp index 6d615f4a92..4e1425466b 100644 --- a/src/mlpack/core/math/random.hpp +++ b/src/mlpack/core/math/random.hpp @@ -48,6 +48,23 @@ inline void RandomSeed(const size_t seed) #endif } +/** + * Set the random seed to a fixed number. + * This function is used in binding tests to set a fixed random seed before + * calling mlpack(). In this way we can test whether a certain parameter makes + * a difference to execution of CLI binding. + * Refer to pull request #1306 for discussion on this function. + */ +#if (BINDING_TYPE == BINDING_TYPE_TEST) +inline void FixedRandomSeed() +{ + const static size_t seed = rand(); + randGen.seed((uint32_t) seed); + srand((unsigned int) seed); + arma::arma_rng::set_seed(seed); +} +#endif + /** * Generates a uniform random number between 0 and 1. */ diff --git a/src/mlpack/core/optimizers/adam/CMakeLists.txt b/src/mlpack/core/optimizers/adam/CMakeLists.txt index 6340ba68ba..fd5788618b 100644 --- a/src/mlpack/core/optimizers/adam/CMakeLists.txt +++ b/src/mlpack/core/optimizers/adam/CMakeLists.txt @@ -6,6 +6,7 @@ set(SOURCES amsgrad_update.hpp nadam_update.hpp nadamax_update.hpp + optimisticadam_update.hpp ) set(DIR_SRCS) diff --git a/src/mlpack/core/optimizers/adam/adam.hpp b/src/mlpack/core/optimizers/adam/adam.hpp index 0b3b2a5a14..e1fa7d6df8 100644 --- a/src/mlpack/core/optimizers/adam/adam.hpp +++ b/src/mlpack/core/optimizers/adam/adam.hpp @@ -30,6 +30,7 @@ #include "amsgrad_update.hpp" #include "nadam_update.hpp" #include "nadamax_update.hpp" +#include "optimisticadam_update.hpp" namespace mlpack { namespace optimization { @@ -186,6 +187,8 @@ using Nadam = AdamType; using NadaMax = AdamType; +using OptimisticAdam = AdamType; + } // namespace optimization } // namespace mlpack diff --git a/src/mlpack/core/optimizers/adam/optimisticadam_update.hpp b/src/mlpack/core/optimizers/adam/optimisticadam_update.hpp new file mode 100644 index 0000000000..06b45d9274 --- /dev/null +++ b/src/mlpack/core/optimizers/adam/optimisticadam_update.hpp @@ -0,0 +1,148 @@ +/** + * @file optimisticadam_update.hpp + * @author Moksh Jain + * + * OptmisticAdam optimizer. Implements Optimistic Adam, an algorithm which + * uses Optimistic Mirror Descent with the Adam optimizer. + * + * mlpack is free software; you may redistribute it and/or modify it under the + * terms of the 3-clause BSD license. You should have received a copy of the + * 3-clause BSD license along with mlpack. If not, see + * http://www.opensource.org/licenses/BSD-3-Clause for more information. + */ +#ifndef MLPACK_CORE_OPTIMIZERS_ADAM_OPTIMISTICADAM_UPDATE_HPP +#define MLPACK_CORE_OPTIMIZERS_ADAM_OPTIMISTICADAM_UPDATE_HPP + +#include + +namespace mlpack { +namespace optimization { + +/** + * OptimisticAdam is an optimizer which implements the Optimistic Adam + * algorithm which uses Optmistic Mirror Descent with the Adam Optimizer. + * It addresses the problem of limit cycling while training GANs. It uses + * OMD to achieve faster regret rates in solving the zero sum game of + * training a GAN. It consistently achieves a smaller KL divergnce with + * respect to the true underlying data distribution. + * + * For more information, see the following. + * + * @code + * @article{ + * author = {Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, + * Haoyang Zeng}, + * title = {Training GANs with Optimism}, + * year = {2017}, + * url = {https://arxiv.org/abs/1711.00141} + * } + * @endcode + */ +class OptimisticAdamUpdate +{ + public: + /** + * Construct the OptimisticAdam update policy with the given parameters. + * + * @param epsilon The epsilon value used to initialize the squared gradient + * parameter. + * @param beta1 The smoothing parameter. + * @param beta2 The second moment coefficient. + */ + OptimisticAdamUpdate(const double epsilon = 1e-8, + const double beta1 = 0.9, + const double beta2 = 0.999) : + epsilon(epsilon), + beta1(beta1), + beta2(beta2), + iteration(0) + { + // Nothing to do. + } + + /** + * The Initialize method is called by SGD Optimizer method before the start of + * the iteration update process. + * + * @param rows Number of rows in the gradient matrix. + * @param cols Number of columns in the gradient matrix. + */ + void Initialize(const size_t rows, const size_t cols) + { + m = arma::zeros(rows, cols); + v = arma::zeros(rows, cols); + g = arma::zeros(rows, cols); + } + + /** + * Update step for OptimisticAdam. + * + * @param iterate Parameters that minimize the function. + * @param stepSize Step size to be used for the given iteration. + * @param gradient The gradient matrix. + */ + void Update(arma::mat& iterate, + const double stepSize, + const arma::mat& gradient) + { + // Increment the iteration counter variable. + ++iteration; + + // And update the iterate. + m *= beta1; + m += (1 - beta1) * gradient; + + v *= beta2; + v += (1 - beta2) * arma::square(gradient); + + arma::mat mCorrected = m / (1.0 - std::pow(beta1, iteration)); + arma::mat vCorrected = v / (1.0 - std::pow(beta2, iteration)); + + arma::mat update = mCorrected / (arma::sqrt(vCorrected) + epsilon); + + iterate -= (2 * stepSize * update - stepSize * g); + + g = std::move(update); + } + + //! Get the value used to initialize the squared gradient parameter. + double Epsilon() const { return epsilon; } + //! Modify the value used to initialize the squared gradient parameter. + double& Epsilon() { return epsilon; } + + //! Get the smoothing parameter. + double Beta1() const { return beta1; } + //! Modify the smoothing parameter. + double& Beta1() { return beta1; } + + //! Get the second moment coefficient. + double Beta2() const { return beta2; } + //! Modify the second moment coefficient. + double& Beta2() { return beta2; } + + private: + // The epsilon value used to initialize the squared gradient parameter. + double epsilon; + + // The smoothing parameter. + double beta1; + + // The second moment coefficient. + double beta2; + + // The exponential moving average of gradient values. + arma::mat m; + + // The exponential moving average of squared gradient values. + arma::mat v; + // The previous update. + arma::mat g; + + // The number of iterations. + double iteration; +}; + +} // namespace optimization +} // namespace mlpack + +#endif diff --git a/src/mlpack/core/optimizers/sdp/lrsdp_function_impl.hpp b/src/mlpack/core/optimizers/sdp/lrsdp_function_impl.hpp index 978c7da33f..f27c3d458c 100644 --- a/src/mlpack/core/optimizers/sdp/lrsdp_function_impl.hpp +++ b/src/mlpack/core/optimizers/sdp/lrsdp_function_impl.hpp @@ -51,7 +51,8 @@ LRSDPFunction::LRSDPFunction(const size_t numSparseConstraints, } template -double LRSDPFunction::Evaluate(const arma::mat& coordinates) const +double LRSDPFunction::Evaluate(const arma::mat& /* coordinates */) + const { // Note: We don't require to update the R*R^T matrix here as the current // function is only used by AugLagrangian, which do not update the coordinates diff --git a/src/mlpack/core/optimizers/sgd/CMakeLists.txt b/src/mlpack/core/optimizers/sgd/CMakeLists.txt index 343d024d90..b4c266316c 100644 --- a/src/mlpack/core/optimizers/sgd/CMakeLists.txt +++ b/src/mlpack/core/optimizers/sgd/CMakeLists.txt @@ -2,6 +2,7 @@ set(SOURCES decay_policies/no_decay.hpp update_policies/gradient_clipping.hpp update_policies/momentum_update.hpp + update_policies/nesterov_momentum_update.hpp update_policies/vanilla_update.hpp sgd.hpp sgd_impl.hpp diff --git a/src/mlpack/core/optimizers/sgd/sgd.hpp b/src/mlpack/core/optimizers/sgd/sgd.hpp index 023ee63950..523539ec73 100644 --- a/src/mlpack/core/optimizers/sgd/sgd.hpp +++ b/src/mlpack/core/optimizers/sgd/sgd.hpp @@ -3,6 +3,7 @@ * @author Ryan Curtin * @author Arun Reddy * @author Abhinav Moudgil + * @author Sourabh Varshney * * Stochastic Gradient Descent (SGD). * @@ -17,6 +18,7 @@ #include #include "update_policies/vanilla_update.hpp" #include "update_policies/momentum_update.hpp" +#include "update_policies/nesterov_momentum_update.hpp" #include "decay_policies/no_decay.hpp" namespace mlpack { @@ -202,6 +204,8 @@ using StandardSGD = SGD; using MomentumSGD = SGD; +using NesterovMomentumSGD = SGD; + } // namespace optimization } // namespace mlpack diff --git a/src/mlpack/core/optimizers/sgd/update_policies/nesterov_momentum_update.hpp b/src/mlpack/core/optimizers/sgd/update_policies/nesterov_momentum_update.hpp new file mode 100644 index 0000000000..befb974913 --- /dev/null +++ b/src/mlpack/core/optimizers/sgd/update_policies/nesterov_momentum_update.hpp @@ -0,0 +1,100 @@ +/** + * @file nesterov_momentum_update.hpp + * @author Sourabh Varshney + * + * Nesterov Momentum Update for Stochastic Gradient Descent. + * + * mlpack is free software; you may redistribute it and/or modify it under the + * terms of the 3-clause BSD license. You should have received a copy of the + * 3-clause BSD license along with mlpack. If not, see + * http://www.opensource.org/licenses/BSD-3-Clause for more information. + */ +#ifndef MLPACK_CORE_OPTIMIZERS_SGD_NESTEROV_MOMENTUM_UPDATE_HPP +#define MLPACK_CORE_OPTIMIZERS_SGD_NESTEROV_MOMENTUM_UPDATE_HPP + +#include + +namespace mlpack { +namespace optimization { + +/** + * Nesterov Momentum update policy for Stochastic Gradient Descent (SGD). + * + * Learning with SGD can be slow. Applying Standard momentum can accelerate + * the rate of convergence. Nesterov Momentum application can accelerate the + * rate of convergence to O(1/k^2). + * + * @code + * @techreport{Nesterov1983, + * title = {A Method Of Solving A Convex Programming Problem With + * Convergence Rate O(1/K^2)}, + * author = {Yuri Nesterov}, + * institution = {Soviet Math. Dokl.}, + * volume = {27}, + * year = {1983}, + * } + * @endcode + */ +class NesterovMomentumUpdate +{ + public: + /** + * Construct the Nesterov Momentum update policy with the given parameters. + * + */ + NesterovMomentumUpdate(const double momentum = 0.5) : + momentum(momentum) + { + // Nothing to do. + } + + /** + * The Initialize method is called by SGD Optimizer method before the start of + * the iteration update process. In the momentum update policy the velocity + * matrix is initialized to the zeros matrix with the same size as the + * gradient matrix (see mlpack::optimization::SGD::Optimizer ) + * + * @param rows Number of rows in the gradient matrix. + * @param cols Number of columns in the gradient matrix. + */ + void Initialize(const size_t rows, const size_t cols) + { + // Initialize an empty velocity matrix. + velocity = arma::zeros(rows, cols); + } + + /** + * Update step for SGD. The momentum term makes the convergence faster on the + * way as momentum term increases for dimensions pointing in the same direction + * and reduces updates for dimensions whose gradients change directions. + * + * @param iterate Parameters that minimize the function. + * @param stepSize Step size to be used for the given iteration. + * @param gradient The gradient matrix. + */ + void Update(arma::mat& iterate, + const double stepSize, + const arma::mat& gradient) + { + velocity = momentum * velocity - stepSize * gradient; + + iterate += momentum * velocity - stepSize * gradient; + } + + //! Get the value used to initialize the momentum coefficient. + double Momentum() const { return momentum; } + //! Modify the value used to initialize the momentum coefficient. + double& Momentum() { return momentum; } + + private: + // The velocity matrix. + arma::mat velocity; + + // The Momentum coefficient. + double momentum; +}; + +} // namespace optimization +} // namespace mlpack + +#endif diff --git a/src/mlpack/methods/ann/convolution_rules/fft_convolution.hpp b/src/mlpack/methods/ann/convolution_rules/fft_convolution.hpp index f573ce9ca2..b362d3ecce 100644 --- a/src/mlpack/methods/ann/convolution_rules/fft_convolution.hpp +++ b/src/mlpack/methods/ann/convolution_rules/fft_convolution.hpp @@ -21,8 +21,8 @@ namespace ann /** Artificial Neural Network. */ { /** * Computes the two-dimensional convolution through fft. This class allows - * specification of the type of the border type. The convolution can be compute - * with the valid border type of the full border type (default). + * specification of the type of the border type. The convolution can be + * computed with the valid border type of the full border type (default). * * FullConvolution: returns the full two-dimensional convolution. * ValidConvolution: returns only those parts of the convolution that are @@ -40,12 +40,12 @@ class FFTConvolution /* * Perform a convolution through fft (valid mode). This method only supports * input which is even on the last dimension. In case of an odd input width, a - * user can manually pad the imput or specify the padLastDim parameter which + * user can manually pad the input or specify the padLastDim parameter which * takes care of the padding. The filter instead can have any size. When using - * the valid mode the filters has to be smaller than the input. + * the valid mode the filter has to be smaller than the input. * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output Output data that contains the results of the convolution. */ template @@ -64,23 +64,23 @@ class FFTConvolution // Pad filter and input to the output shape. filterPadded.resize(inputPadded.n_rows, inputPadded.n_cols); - output = arma::real(ifft2(arma::fft2(inputPadded) % arma::fft2( + arma::Mat temp = arma::real(ifft2(arma::fft2(inputPadded) % arma::fft2( filterPadded))); // Extract the region of interest. We don't need to handle the padLastDim in // a special way we just cut it out from the output matrix. - output = output.submat(filter.n_rows - 1, filter.n_cols - 1, + output = temp.submat(filter.n_rows - 1, filter.n_cols - 1, input.n_rows - 1, input.n_cols - 1); } /* * Perform a convolution through fft (full mode). This method only supports * input which is even on the last dimension. In case of an odd input width, a - * user can manually pad the imput or specify the padLastDim parameter which + * user can manually pad the input or specify the padLastDim parameter which * takes care of the padding. The filter instead can have any size. * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output Output data that contains the results of the convolution. */ template @@ -110,12 +110,12 @@ class FFTConvolution filterPadded.resize(outputRows, outputCols); // Perform FFT and IFFT - output = arma::real(ifft2(arma::fft2(inputPadded) % arma::fft2( + arma::Mat temp = arma::real(ifft2(arma::fft2(inputPadded) % arma::fft2( filterPadded))); // Extract the region of interest. We don't need to handle the padLastDim // parameter in a special way we just cut it out from the output matrix. - output = output.submat(filter.n_rows - 1, filter.n_cols - 1, + output = temp.submat(filter.n_rows - 1, filter.n_cols - 1, 2 * (filter.n_rows - 1) + input.n_rows - 1, 2 * (filter.n_cols - 1) + input.n_cols - 1); } @@ -123,12 +123,12 @@ class FFTConvolution /* * Perform a convolution through fft using 3rd order tensors. This method only * supports input which is even on the last dimension. In case of an odd input - * width, a user can manually pad the imput or specify the padLastDim + * width, a user can manually pad the input or specify the padLastDim * parameter which takes care of the padding. The filter instead can have any * size. * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output Output data that contains the results of the convolution. */ template @@ -147,8 +147,7 @@ class FFTConvolution for (size_t i = 1; i < input.n_slices; i++) { FFTConvolution::Convolution(input.slice(i), filter.slice(i), - convOutput); - output.slice(i) = convOutput; + output.slice(i)); } } @@ -156,11 +155,11 @@ class FFTConvolution * Perform a convolution through fft using dense matrix as input and a 3rd * order tensors as filter and output. This method only supports input which * is even on the last dimension. In case of an odd input width, a user can - * manually pad the imput or specify the padLastDim parameter which takes care + * manually pad the input or specify the padLastDim parameter which takes care * of the padding. The filter instead can have any size. * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output Output data that contains the results of the convolution. */ template @@ -179,8 +178,7 @@ class FFTConvolution for (size_t i = 1; i < filter.n_slices; i++) { FFTConvolution::Convolution(input, filter.slice(i), - convOutput); - output.slice(i) = convOutput; + output.slice(i)); } } @@ -189,7 +187,7 @@ class FFTConvolution * dense matrix as filter. * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output Output data that contains the results of the convolution. */ template @@ -208,8 +206,7 @@ class FFTConvolution for (size_t i = 1; i < input.n_slices; i++) { FFTConvolution::Convolution(input.slice(i), filter, - convOutput); - output.slice(i) = convOutput; + output.slice(i)); } } }; // class FFTConvolution diff --git a/src/mlpack/methods/ann/convolution_rules/naive_convolution.hpp b/src/mlpack/methods/ann/convolution_rules/naive_convolution.hpp index c27225f127..90882c3b7d 100644 --- a/src/mlpack/methods/ann/convolution_rules/naive_convolution.hpp +++ b/src/mlpack/methods/ann/convolution_rules/naive_convolution.hpp @@ -39,7 +39,7 @@ class NaiveConvolution * Perform a convolution (valid mode). * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output Output data that contains the results of the convolution. * @param dW Stride of filter application in the x direction. * @param dH Stride of filter application in the y direction. @@ -79,7 +79,7 @@ class NaiveConvolution * Perform a convolution (full mode). * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output Output data that contains the results of the convolution. * @param dW Stride of filter application in the x direction. * @param dH Stride of filter application in the y direction. @@ -111,7 +111,7 @@ class NaiveConvolution * Perform a convolution using 3rd order tensors. * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output Output data that contains the results of the convolution. * @param dW Stride of filter application in the x direction. * @param dH Stride of filter application in the y direction. @@ -143,7 +143,7 @@ class NaiveConvolution * as filter and output. * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output Output data that contains the results of the convolution. * @param dW Stride of filter application in the x direction. * @param dH Stride of filter application in the y direction. @@ -175,7 +175,7 @@ class NaiveConvolution * dense matrix as filter. * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output Output data that contains the results of the convolution. * @param dW Stride of filter application in the x direction. * @param dH Stride of filter application in the y direction. diff --git a/src/mlpack/methods/ann/convolution_rules/svd_convolution.hpp b/src/mlpack/methods/ann/convolution_rules/svd_convolution.hpp index 7cd50470ab..8ed0a9c415 100644 --- a/src/mlpack/methods/ann/convolution_rules/svd_convolution.hpp +++ b/src/mlpack/methods/ann/convolution_rules/svd_convolution.hpp @@ -3,7 +3,7 @@ * @author Marcus Edel * * Implementation of the convolution using the singular value decomposition to - * speeded up the computation. + * speed up the computation. * * mlpack is free software; you may redistribute it and/or modify it under the * terms of the 3-clause BSD license. You should have received a copy of the @@ -24,7 +24,7 @@ namespace ann /** Artificial Neural Network. */ { /** * Computes the two-dimensional convolution using singular value decomposition. * This class allows specification of the type of the border type. The - * convolution can be compute with the valid border type of the full border + * convolution can be computed with the valid border type of the full border * type (default). * * FullConvolution: returns the full two-dimensional convolution. @@ -87,13 +87,13 @@ class SVDConvolution NaiveConvolution::Convolution(subOutput, U.unsafe_col(0), output); + arma::Mat temp; for (size_t r = 1; r < rank; r++) { subFilter = V.unsafe_col(r) * s(r); NaiveConvolution::Convolution(input, subFilter, subOutput); - arma::Mat temp; subOutput = subOutput.t(); NaiveConvolution::Convolution(subOutput, U.unsafe_col(r), temp); @@ -134,8 +134,7 @@ class SVDConvolution for (size_t i = 1; i < input.n_slices; i++) { SVDConvolution::Convolution(input.slice(i), filter.slice(i), - convOutput); - output.slice(i) = convOutput; + output.slice(i)); } } @@ -164,8 +163,7 @@ class SVDConvolution for (size_t i = 1; i < filter.n_slices; i++) { SVDConvolution::Convolution(input, filter.slice(i), - convOutput); - output.slice(i) = convOutput; + output.slice(i)); } } @@ -194,8 +192,7 @@ class SVDConvolution for (size_t i = 1; i < input.n_slices; i++) { SVDConvolution::Convolution(input.slice(i), filter, - convOutput); - output.slice(i) = convOutput; + output.slice(i)); } } }; // class SVDConvolution diff --git a/src/mlpack/methods/ann/layer/CMakeLists.txt b/src/mlpack/methods/ann/layer/CMakeLists.txt index 57be256193..38dfa4377b 100644 --- a/src/mlpack/methods/ann/layer/CMakeLists.txt +++ b/src/mlpack/methods/ann/layer/CMakeLists.txt @@ -8,6 +8,8 @@ set(SOURCES base_layer.hpp bilinear_interpolation.hpp bilinear_interpolation_impl.hpp + batch_norm.hpp + batch_norm_impl.hpp concat.hpp concat_impl.hpp concat_performance.hpp diff --git a/src/mlpack/methods/ann/layer/batch_norm.hpp b/src/mlpack/methods/ann/layer/batch_norm.hpp new file mode 100644 index 0000000000..1bd4e1afb0 --- /dev/null +++ b/src/mlpack/methods/ann/layer/batch_norm.hpp @@ -0,0 +1,202 @@ +/** + * @file batch_norm.hpp + * @author Praveen Ch + * @author Manthan-R-Sheth + * + * Definition of the Batch Normalisation layer class + * + * mlpack is free software; you may redistribute it and/or modify it under the + * terms of the 3-clause BSD license. You should have received a copy of the + * 3-clause BSD license along with mlpack. If not, see + * http://www.opensource.org/licenses/BSD-3-Clause for more information. + */ + +#ifndef MLPACK_METHODS_ANN_LAYER_BATCHNORM_HPP +#define MLPACK_METHODS_ANN_LAYER_BATCHNORM_HPP + +#include + +namespace mlpack { +namespace ann /** Artificial Neural Network. */ { + +/** + * Declaration of the Batch Normalisation layer class. The layer tranforms + * the input data into zero mean and unit variance and then scales and shifts + * the data by parameters, gamma and beta respectively. These parameters are + * learnt by the network. + * + * If deterministic is false (training), the mean and variance over the batch is + * calculated and the data is normalized. If it is set to true (testing) then + * the mean and variance accrued over the training set is used. + * + * For more information, refer to the following paper, + * + * @code + * @article{DBLP:journals/corr/IoffeS15, + * author = {Sergey Ioffe and + * Christian Szegedy}, + * title = {Batch Normalization: Accelerating Deep Network Training by + * Reducing Internal Covariate Shift}, + * journal = {CoRR}, + * volume = {abs/1502.03167} + * } + * + * @endcode + * + * @tparam InputDataType Type of the input data (arma::colvec, arma::mat, + * arma::sp_mat or arma::cube). + * @tparam OutputDataType Type of the output data (arma::colvec, arma::mat, + * arma::sp_mat or arma::cube). + */ + +template < + typename InputDataType = arma::mat, + typename OutputDataType = arma::mat +> +class BatchNorm +{ + public: + //! Create the BatchNorm object. + BatchNorm(); + + /** + * Create the BatchNorm layer object for a specified number of input units. + * + * @param size The number of input units. + * @param eps The epsilon added to variance to ensure numerical stability. + */ + BatchNorm(const size_t size, const double eps = 0.001); + + /** + * Reset the layer parameters + */ + void Reset(); + + /** + * Forward pass of the Batch Normalization layer. Transforms the input data + * into zero mean and unit variance, scales the data by a factor gamma and + * shifts it by beta. + * + * @param input Input data for the layer + * @param output Resulting output activations. + */ + template + void Forward(const arma::Mat&& input, arma::Mat&& output); + + /** + * Backward pass through the layer. + * + * @param input The input activations + * @param gy The backpropagated error. + * @param g The calculated gradient. + */ + template + void Backward(const arma::Mat&& input, + arma::Mat&& gy, + arma::Mat&& g); + + /** + * Calculate the gradient using the output delta and the input activations. + * + * @param input The input activations + * @param error The calculated error + * @param gradient The calculated gradient. + */ + template + void Gradient(const arma::Mat&& input, + arma::Mat&& error, + arma::Mat&& gradient); + + //! Get the parameters. + OutputDataType const& Parameters() const { return weights; } + //! Modify the parameters. + OutputDataType& Parameters() { return weights; } + + //! Get the input parameter. + InputDataType const& InputParameter() const { return inputParameter; } + //! Modify the input parameter. + InputDataType& InputParameter() { return inputParameter; } + + //! Get the output parameter. + OutputDataType const& OutputParameter() const { return outputParameter; } + //! Modify the output parameter. + OutputDataType& OutputParameter() { return outputParameter; } + + //! Get the delta. + OutputDataType const& Delta() const { return delta; } + //! Modify the delta. + OutputDataType& Delta() { return delta; } + + //! Get the gradient. + OutputDataType const& Gradient() const { return gradient; } + //! Modify the gradient. + OutputDataType& Gradient() { return gradient; } + + //! Get the value of deterministic parameter. + bool Deterministic() const { return deterministic; } + //! Modify the value of deterministic parameter. + bool& Deterministic() { return deterministic; } + + //! Get the mean over the training data. + OutputDataType TrainingMean() { return stats.mean(); } + + //! Get the variance over the training data. + OutputDataType TrainingVariance() { return stats.var(1); } + + /** + * Serialize the layer + */ + template + void serialize(Archive& ar, const unsigned int /* version */); + + private: + //! Locally-stored number of input units. + size_t size; + + //! Locally-stored epsilon value. + double eps; + + //! Locally-stored scale parameter. + OutputDataType gamma; + + //! Locally-stored shift parameter. + OutputDataType beta; + + //! Locally-stored parameters. + OutputDataType weights; + + /** + * If true then mean and variance over the training set will be considered + * instead of being calculated over the batch. + */ + bool deterministic; + + //! Locally-stored mean object. + OutputDataType mean; + + //! Locally-stored variance object. + OutputDataType variance; + + //! Locally-stored running statistics object. + arma::running_stat_vec stats; + + //! Locally-stored gradient object. + OutputDataType gradient; + + //! Locally-stored delta object. + OutputDataType delta; + + //! Locally-stored input parameter object. + InputDataType inputParameter; + + //! Locally-stored output parameter object. + OutputDataType outputParameter; +}; // class BatchNorm + +} // namespace ann +} // namespace mlpack + +// Include the implementation. +#include "batch_norm_impl.hpp" + +#endif diff --git a/src/mlpack/methods/ann/layer/batch_norm_impl.hpp b/src/mlpack/methods/ann/layer/batch_norm_impl.hpp new file mode 100644 index 0000000000..9d5dd4bfd6 --- /dev/null +++ b/src/mlpack/methods/ann/layer/batch_norm_impl.hpp @@ -0,0 +1,130 @@ +/** + * @file batch_norm_impl.hpp + * @author Praveen Ch + * @author Manthan-R-Sheth + * + * Implementation of the Batch Normalization Layer. + * + * mlpack is free software; you may redistribute it and/or modify it under the + * terms of the 3-clause BSD license. You should have received a copy of the + * 3-clause BSD license along with mlpack. If not, see + * http://www.opensource.org/licenses/BSD-3-Clause for more information. + */ + +#ifndef MLPACK_METHODS_ANN_LAYER_BATCHNORM_IMPL_HPP +#define MLPACK_METHODS_ANN_LAYER_BATCHNORM_IMPL_HPP + +// In case it is not included. +#include "batch_norm.hpp" + +namespace mlpack { +namespace ann { /** Artificial Neural Network. */ + +template +BatchNorm::BatchNorm() : + size(10), + eps(1e-7), + deterministic(false) +{ + // Nothing to do here. +} + +template +BatchNorm::BatchNorm( + const size_t size, const double eps) : + size(size), + eps(eps), + deterministic(false) +{ + weights.set_size(size + size, 1); +} + +template +void BatchNorm::Reset() +{ + gamma = arma::mat(weights.memptr(), size, 1, false, false); + beta = arma::mat(weights.memptr() + gamma.n_elem, size, 1, false, false); + deterministic = false; + gamma.fill(1.0); + beta.fill(0.0); + stats.reset(); +} + +template +template +void BatchNorm::Forward( + const arma::Mat&& input, arma::Mat&& output) +{ + output.reshape(input.n_rows, input.n_cols); + + // Mean and variance over the entire training set will be used to compute + // the forward pass when deterministic is set to true. + if (deterministic) + { + mean = stats.mean(); + variance = stats.var(1); + } + else + { + mean = arma::mean(input, 1); + variance = arma::var(input, 1, 1); + + for (size_t i = 0; i < output.n_cols; i++) + { + stats(input.col(i)); + } + } + + output = input.each_col() - mean; + output.each_col() %= gamma / arma::sqrt(variance + eps); + output.each_col() += beta; +} + +template +template +void BatchNorm::Backward( + const arma::Mat&& input, arma::Mat&& gy, arma::Mat&& g) +{ + mean = arma::mean(input, 1); + variance = arma::var(input, 1, 1); + + arma::mat m = arma::sum(gy % (input.each_col() - mean), 1); + g = (mean - input.each_col()); + g.each_col() %= m; + g.each_col() %= 1.0/(variance + eps); + g += (gy.each_col() - arma::sum(gy, 1)); + g += (input.n_cols - 1) * gy; + g.each_col() %= ((1.0 / input.n_cols) * gamma); + g.each_col() %= (1.0 / arma::sqrt(variance + eps)); +} + +template +template +void BatchNorm::Gradient( + const arma::Mat&& input, + arma::Mat&& error, + arma::Mat&& gradient) +{ + gradient.set_size(size + size, 1); + + arma::mat normalized = input.each_col() - arma::mean(input, 1) + / arma::sqrt(arma::var(input, 1, 1) + eps); + + gradient.submat(0, 0, gamma.n_elem - 1, 0) = arma::sum(normalized % error, 1); + gradient.submat(gamma.n_elem, 0, gradient.n_elem - 1, 0) = + arma::sum(error, 1); +} + +template +template +void BatchNorm::serialize( + Archive& ar, const unsigned int /* version */) +{ + ar & BOOST_SERIALIZATION_NVP(gamma); + ar & BOOST_SERIALIZATION_NVP(beta); +} + +} // namespace ann +} // namespace mlpack + +#endif diff --git a/src/mlpack/methods/ann/layer/convolution_impl.hpp b/src/mlpack/methods/ann/layer/convolution_impl.hpp index 3b01406d1b..48da43af38 100644 --- a/src/mlpack/methods/ann/layer/convolution_impl.hpp +++ b/src/mlpack/methods/ann/layer/convolution_impl.hpp @@ -171,8 +171,8 @@ void Convolution< >::Backward( const arma::Mat&& /* input */, arma::Mat&& gy, arma::Mat&& g) { - arma::cube mappedError = arma::cube(gy.memptr(), - outputWidth, outputHeight, outSize); + arma::cube mappedError(gy.memptr(), outputWidth, outputHeight, outSize, + false, false); gTemp = arma::zeros >(inputTemp.n_rows, inputTemp.n_cols, inputTemp.n_slices); @@ -265,12 +265,10 @@ void Convolution< { for (size_t i = 0; i < output.n_slices; i++) { - arma::mat subOutput = output.slice(i); - - gradientTemp.slice(s) += subOutput.submat(subOutput.n_rows / 2, - subOutput.n_cols / 2, - subOutput.n_rows / 2 + gradientTemp.n_rows - 1, - subOutput.n_cols / 2 + gradientTemp.n_cols - 1); + gradientTemp.slice(s) += output.slice(i).submat(output.n_rows / 2, + output.n_cols / 2, + output.n_rows / 2 + gradientTemp.n_rows - 1, + output.n_cols / 2 + gradientTemp.n_cols - 1); } } else diff --git a/src/mlpack/methods/ann/layer/layer.hpp b/src/mlpack/methods/ann/layer/layer.hpp index f47eadfd2f..f50b988be6 100644 --- a/src/mlpack/methods/ann/layer/layer.hpp +++ b/src/mlpack/methods/ann/layer/layer.hpp @@ -13,6 +13,7 @@ #define MLPACK_METHODS_ANN_LAYER_LAYER_HPP #include "add_merge.hpp" +#include "batch_norm.hpp" #include "concat_performance.hpp" #include "convolution.hpp" #include "dropconnect.hpp" diff --git a/src/mlpack/methods/ann/layer/layer_types.hpp b/src/mlpack/methods/ann/layer/layer_types.hpp index 867c665156..9d609effdd 100644 --- a/src/mlpack/methods/ann/layer/layer_types.hpp +++ b/src/mlpack/methods/ann/layer/layer_types.hpp @@ -17,6 +17,7 @@ // Layer modules. #include #include +#include #include #include #include @@ -45,6 +46,8 @@ namespace mlpack { namespace ann { + +template class BatchNorm; template class DropConnect; template class Glimpse; template class Linear; @@ -108,6 +111,7 @@ using LayerTypes = boost::variant< BaseLayer*, BaseLayer*, BaseLayer*, + BatchNorm*, BilinearInterpolation*, Concat*, ConcatPerformance, diff --git a/src/mlpack/methods/decision_tree/all_categorical_split.hpp b/src/mlpack/methods/decision_tree/all_categorical_split.hpp index b92702ebc9..2823fa9bb5 100644 --- a/src/mlpack/methods/decision_tree/all_categorical_split.hpp +++ b/src/mlpack/methods/decision_tree/all_categorical_split.hpp @@ -61,6 +61,7 @@ class AllCategoricalSplit const size_t numClasses, const WeightVecType& weights, const size_t minimumLeafSize, + const double minimumGainSplit, arma::Col& classProbabilities, AuxiliarySplitInfo& aux); diff --git a/src/mlpack/methods/decision_tree/all_categorical_split_impl.hpp b/src/mlpack/methods/decision_tree/all_categorical_split_impl.hpp index b15c7381f4..115a9afb7e 100644 --- a/src/mlpack/methods/decision_tree/all_categorical_split_impl.hpp +++ b/src/mlpack/methods/decision_tree/all_categorical_split_impl.hpp @@ -25,6 +25,7 @@ double AllCategoricalSplit::SplitIfBetter( const size_t numClasses, const WeightVecType& weights, const size_t minimumLeafSize, + const double minimumGainSplit, arma::Col& classProbabilities, AuxiliarySplitInfo& /* aux */) { @@ -96,7 +97,7 @@ double AllCategoricalSplit::SplitIfBetter( overallGain += childPct * childGain; } - if (overallGain > bestGain + epsilon) + if (overallGain > bestGain + minimumGainSplit + epsilon) { // This is better, so set up the class probabilities vector and return. classProbabilities.set_size(1); diff --git a/src/mlpack/methods/decision_tree/best_binary_numeric_split.hpp b/src/mlpack/methods/decision_tree/best_binary_numeric_split.hpp index 4b0f039fad..c187e5f08d 100644 --- a/src/mlpack/methods/decision_tree/best_binary_numeric_split.hpp +++ b/src/mlpack/methods/decision_tree/best_binary_numeric_split.hpp @@ -58,6 +58,7 @@ class BestBinaryNumericSplit const size_t numClasses, const WeightVecType& weights, const size_t minimumLeafSize, + const double minimumGainSplit, arma::Col& classProbabilities, AuxiliarySplitInfo& aux); diff --git a/src/mlpack/methods/decision_tree/best_binary_numeric_split_impl.hpp b/src/mlpack/methods/decision_tree/best_binary_numeric_split_impl.hpp index 82413e796f..8641f1bf01 100644 --- a/src/mlpack/methods/decision_tree/best_binary_numeric_split_impl.hpp +++ b/src/mlpack/methods/decision_tree/best_binary_numeric_split_impl.hpp @@ -24,6 +24,7 @@ double BestBinaryNumericSplit::SplitIfBetter( const size_t numClasses, const WeightVecType& weights, const size_t minimumLeafSize, + const double minimumGainSplit, arma::Col& classProbabilities, AuxiliarySplitInfo& /* aux */) { @@ -104,7 +105,7 @@ double BestBinaryNumericSplit::SplitIfBetter( data[sortedIndices[index]]) / 2.0; return gain; } - else if (gain > bestFoundGain) + else if (gain > bestFoundGain + minimumGainSplit) { // We still have a better split. bestFoundGain = gain; diff --git a/src/mlpack/methods/decision_tree/decision_tree.hpp b/src/mlpack/methods/decision_tree/decision_tree.hpp index c52ee9fe2e..a8108cdd94 100644 --- a/src/mlpack/methods/decision_tree/decision_tree.hpp +++ b/src/mlpack/methods/decision_tree/decision_tree.hpp @@ -52,45 +52,49 @@ class DecisionTree : /** * Construct the decision tree on the given data and labels, where the data - * can be both numeric and categorical. Setting minimumLeafSize too small may - * cause the tree to overfit, but setting it too large may cause it to - * underfit. + * can be both numeric and categorical. Setting minimumLeafSize and + * minimumGainSplit too small may cause the tree to overfit, but setting them + * too large may cause it to underfit. * * @param data Dataset to train on. * @param datasetInfo Type information for each dimension of the dataset. * @param labels Labels for each training point. * @param numClasses Number of classes in the dataset. * @param minimumLeafSize Minimum number of points in each leaf node. + * @param minimumGainSplit Minimum gain for the node to split. */ template DecisionTree(MatType&& data, const data::DatasetInfo& datasetInfo, LabelsType&& labels, const size_t numClasses, - const size_t minimumLeafSize = 10); + const size_t minimumLeafSize = 10, + const double minimumGainSplit = 1e-7); /** * Construct the decision tree on the given data and labels, assuming that the - * data is all of the numeric type. Setting minimumLeafSize too small may - * cause the tree to overfit, but setting it too large may cause it to - * underfit. + * data is all of the numeric type. Setting minimumLeafSize and + * minimumGainSplit too small may cause the tree to overfit, but setting them + * too large may cause it to underfit. * * @param data Dataset to train on. * @param labels Labels for each training point. * @param numClasses Number of classes in the dataset. * @param minimumLeafSize Minimum number of points in each leaf node. + * @param minimumGainSplit Minimum gain for the node to split. */ template DecisionTree(MatType&& data, LabelsType&& labels, const size_t numClasses, - const size_t minimumLeafSize = 10); + const size_t minimumLeafSize = 10, + const double minimumGainSplit = 1e-7); /** * Construct the decision tree on the given data and labels with weights, - * where the data can be both numeric and categorical. Setting - * minimumLeafSize too small may cause the tree to overfit, but setting it too - * large may cause it to underfit. + * where the data can be both numeric and categorical. Setting minimumLeafSize + * and minimumGainSplit too small may cause the tree to overfit, but setting + * them too large may cause it to underfit. * * @param data Dataset to train on. * @param datasetInfo Type information for each dimension of the dataset. @@ -98,6 +102,7 @@ class DecisionTree : * @param numClasses Number of classes in the dataset. * @param weights The weight list of given label. * @param minimumLeafSize Minimum number of points in each leaf node. + * @param minimumGainSplit Minimum gain for the node to split. */ template DecisionTree(MatType&& data, @@ -106,21 +111,23 @@ class DecisionTree : const size_t numClasses, WeightsType&& weights, const size_t minimumLeafSize = 10, + const double minimumGainSplit = 1e-7, const std::enable_if_t::type>::value>* = 0); /** * Construct the decision tree on the given data and labels with weights, - * assuming that the data is all of the numeric type. Setting minimumLeafSize - * too small may cause the tree to overfit, but setting it too large may cause - * it to underfit. + * assuming that the data is all of the numeric type. Setting minimumLeafSize + * and minimumGainSplit too small may cause the tree to overfit, but setting + * them too large may cause it to underfit. * * @param data Dataset to train on. * @param labels Labels for each training point. * @param numClasses Number of classes in the dataset. * @param weights The Weight list of given labels. * @param minimumLeafSize Minimum number of points in each leaf node. + * @param minimumGainSplit Minimum gain for the node to split. */ template DecisionTree(MatType&& data, @@ -128,6 +135,7 @@ class DecisionTree : const size_t numClasses, WeightsType&& weights, const size_t minimumLeafSize = 10, + const double minimumGainSplit = 1e-7, const std::enable_if_t::type>::value>* = 0); @@ -179,8 +187,9 @@ class DecisionTree : /** * Train the decision tree on the given data. This will overwrite the * existing model. The data may have numeric and categorical types, specified - * by the datasetInfo parameter. Setting minimumLeafSize too small may cause - * the tree to overfit, but setting it too large may cause it to underfit. + * by the datasetInfo parameter. Setting minimumLeafSize and + * minimumGainSplit too small may cause the tree to overfit, but setting them + * too large may cause it to underfit. * * @param data Dataset to train on. * @param datasetInfo Type information for each dimension. @@ -188,38 +197,42 @@ class DecisionTree : * @param numClasses Number of classes in the dataset. * @param weights Weights of all the labels * @param minimumLeafSize Minimum number of points in each leaf node. + * @param minimumGainSplit Minimum gain for the node to split. */ template void Train(MatType&& data, const data::DatasetInfo& datasetInfo, LabelsType&& labels, const size_t numClasses, - const size_t minimumLeafSize = 10); + const size_t minimumLeafSize = 10, + const double minimumGainSplit = 1e-7); /** * Train the decision tree on the given data, assuming that all dimensions are - * numeric. This will overwrite the given model. Setting minimumLeafSize too - * small may cause the tree to overfit, but setting it too large may cause it - * to underfit. + * numeric. This will overwrite the given model. Setting minimumLeafSize and + * minimumGainSplit too small may cause the tree to overfit, but setting them + * too large may cause it to underfit. * * @param data Dataset to train on. * @param labels Labels for each training point. * @param numClasses Number of classes in the dataset. * @param weights Weights of all the labels * @param minimumLeafSize Minimum number of points in each leaf node. + * @param minimumGainSplit Minimum gain for the node to split. */ template void Train(MatType&& data, LabelsType&& labels, const size_t numClasses, - const size_t minimumLeafSize = 10); + const size_t minimumLeafSize = 10, + const double minimumGainSplit = 1e-7); /** * Train the decision tree on the given weighted data. This will overwrite * the existing model. The data may have numeric and categorical types, - * specified by the datasetInfo parameter. Setting minimumLeafSize too small - * may cause the tree to overfit, but setting it too large may cause it to - * underfit. + * specified by the datasetInfo parameter. Setting minimumLeafSize and + * minimumGainSplit too small may cause the tree to overfit, but setting them + * too large may cause it to underfit. * * @param data Dataset to train on. * @param datasetInfo Type information for each dimension. @@ -227,6 +240,7 @@ class DecisionTree : * @param numClasses Number of classes in the dataset. * @param weights Weights of all the labels * @param minimumLeafSize Minimum number of points in each leaf node. + * @param minimumGainSplit Minimum gain for the node to split. */ template void Train(MatType&& data, @@ -235,20 +249,22 @@ class DecisionTree : const size_t numClasses, WeightsType&& weights, const size_t minimumLeafSize = 10, + const double minimumGainSplit = 1e-7, const std::enable_if_t::type>::value>* = 0); /** * Train the decision tree on the given weighted data, assuming that all - * dimensions are numeric. This will overwrite the given model. Setting - * minimumLeafSize too small may cause the tree to overfit, but setting it too - * large may cause it to underfit. + * dimensions are numeric. This will overwrite the given model. Setting + * minimumLeafSize and minimumGainSplit too small may cause the tree to + * overfit, but setting them too large may cause it to underfit. * * @param data Dataset to train on. * @param labels Labels for each training point. * @param numClasses Number of classes in the dataset. * @param weights Weights of all the labels * @param minimumLeafSize Minimum number of points in each leaf node. + * @param minimumGainSplit Minimum gain for the node to split. */ template void Train(MatType&& data, @@ -256,6 +272,7 @@ class DecisionTree : const size_t numClasses, WeightsType&& weights, const size_t minimumLeafSize = 10, + const double minimumGainSplit = 1e-7, const std::enable_if_t::type>::value>* = 0); @@ -383,6 +400,7 @@ class DecisionTree : * @param labels Labels for each training point. * @param numClasses Number of classes in the dataset. * @param minimumLeafSize Minimum number of points in each leaf node. + * @param minimumGainSplit Minimum gain for the node to split. */ template void Train(MatType& data, @@ -392,7 +410,8 @@ class DecisionTree : arma::Row& labels, const size_t numClasses, arma::rowvec& weights, - const size_t minimumLeafSize = 10); + const size_t minimumLeafSize = 10, + const double minimumGainSplit = 1e-7); /** * Corresponding to the public Train() method, this method is designed for @@ -406,6 +425,7 @@ class DecisionTree : * @param labels Labels for each training point. * @param numClasses Number of classes in the dataset. * @param minimumLeafSize Minimum number of points in each leaf node. + * @param minimumGainSplit Minimum gain for the node to split. */ template void Train(MatType& data, @@ -414,7 +434,8 @@ class DecisionTree : arma::Row& labels, const size_t numClasses, arma::rowvec& weights, - const size_t minimumLeafSize = 10); + const size_t minimumLeafSize = 10, + const double minimumGainSplit = 1e-7); }; /** diff --git a/src/mlpack/methods/decision_tree/decision_tree_impl.hpp b/src/mlpack/methods/decision_tree/decision_tree_impl.hpp index a855e02b34..91a0827982 100644 --- a/src/mlpack/methods/decision_tree/decision_tree_impl.hpp +++ b/src/mlpack/methods/decision_tree/decision_tree_impl.hpp @@ -32,7 +32,8 @@ DecisionTree::type; using TrueLabelsType = typename std::decay::type; @@ -44,7 +45,7 @@ DecisionTree(tmpData, 0, tmpData.n_cols, datasetInfo, tmpLabels, numClasses, - weights, minimumLeafSize); + weights, minimumLeafSize, minimumGainSplit); } //! Construct and train. @@ -63,7 +64,8 @@ DecisionTree::DecisionTree(MatType&& data, LabelsType&& labels, const size_t numClasses, - const size_t minimumLeafSize) + const size_t minimumLeafSize, + const double minimumGainSplit) { using TrueMatType = typename std::decay::type; using TrueLabelsType = typename std::decay::type; @@ -75,7 +77,7 @@ DecisionTree(tmpData, 0, tmpData.n_cols, tmpLabels, numClasses, weights, - minimumLeafSize); + minimumLeafSize, minimumGainSplit); } //! Construct and train with weights. @@ -97,6 +99,7 @@ DecisionTree(tmpData, 0, tmpData.n_cols, datasetInfo, tmpLabels, numClasses, - tmpWeights, minimumLeafSize); + tmpWeights, minimumLeafSize, minimumGainSplit); } //! Construct and train with weights. @@ -134,6 +137,7 @@ DecisionTree(tmpData, 0, tmpData.n_cols, tmpLabels, numClasses, tmpWeights, - minimumLeafSize); + minimumLeafSize, minimumGainSplit); } //! Construct, don't train. @@ -345,7 +349,8 @@ void DecisionTree(tmpData, 0, tmpData.n_cols, datasetInfo, tmpLabels, numClasses, - weights, minimumLeafSize); + weights, minimumLeafSize, minimumGainSplit); } //! Train on the given data, assuming all dimensions are numeric. @@ -386,7 +391,8 @@ void DecisionTree::Train(MatType&& data, LabelsType&& labels, const size_t numClasses, - const size_t minimumLeafSize) + const size_t minimumLeafSize, + const double minimumGainSplit) { // Sanity check on data. if (data.n_cols != labels.n_elem) @@ -408,7 +414,7 @@ void DecisionTree(tmpData, 0, tmpData.n_cols, tmpLabels, numClasses, weights, - minimumLeafSize); + minimumLeafSize, minimumGainSplit); } //! Train on the given weighted data. @@ -430,6 +436,7 @@ void DecisionTree::type>::value>*) @@ -455,7 +462,7 @@ void DecisionTree(tmpData, 0, tmpData.n_cols, datasetInfo, tmpLabels, numClasses, - tmpWeights, minimumLeafSize); + tmpWeights, minimumLeafSize, minimumGainSplit); } //! Train on the given weighted data. @@ -476,6 +483,7 @@ void DecisionTree::type>::value>*) @@ -501,7 +509,7 @@ void DecisionTree(tmpData, 0, tmpData.n_cols, tmpLabels, numClasses, tmpWeights, - minimumLeafSize); + minimumLeafSize, minimumGainSplit); } //! Train on the given data. @@ -524,7 +532,8 @@ void DecisionTree& labels, const size_t numClasses, arma::rowvec& weights, - const size_t minimumLeafSize) + const size_t minimumLeafSize, + const double minimumGainSplit) { // Clear children if needed. for (size_t i = 0; i < children.size(); ++i) @@ -533,7 +542,7 @@ void DecisionTree( @@ -555,6 +564,7 @@ void DecisionTreeTrain(data, currentChildBegin, currentCol - currentChildBegin, datasetInfo, labels, numClasses, - weights, currentCol - currentChildBegin); + weights, currentCol - currentChildBegin, minimumGainSplit); } else { child->Train(data, currentChildBegin, currentCol - currentChildBegin, datasetInfo, labels, numClasses, - weights, minimumLeafSize); + weights, minimumLeafSize, minimumGainSplit); } children.push_back(child); } @@ -685,7 +696,8 @@ void DecisionTree& labels, const size_t numClasses, arma::rowvec& weights, - const size_t minimumLeafSize) + const size_t minimumLeafSize, + const double minimumGainSplit) { // Clear children if needed. for (size_t i = 0; i < children.size(); ++i) @@ -716,6 +728,7 @@ void DecisionTreeTrain(data, currentChildBegin, currentCol - currentChildBegin, labels, numClasses, weights, - currentCol - currentChildBegin); + currentCol - currentChildBegin, minimumGainSplit); } else { child->Train(data, currentChildBegin, currentCol - currentChildBegin, labels, numClasses, weights, - minimumLeafSize); + minimumLeafSize, minimumGainSplit); } children.push_back(child); } diff --git a/src/mlpack/methods/decision_tree/decision_tree_main.cpp b/src/mlpack/methods/decision_tree/decision_tree_main.cpp index e4ee8fcedc..258689fb6f 100644 --- a/src/mlpack/methods/decision_tree/decision_tree_main.cpp +++ b/src/mlpack/methods/decision_tree/decision_tree_main.cpp @@ -39,7 +39,9 @@ PROGRAM_INFO("Decision tree", "may not be specified when the " + PRINT_PARAM_STRING("training") + " " "parameter is specified. The " + PRINT_PARAM_STRING("minimum_leaf_size") + " parameter specifies the minimum number of training points that must fall" - " into each leaf for it to be split. If " + + " into each leaf for it to be split. The " + + PRINT_PARAM_STRING("minimum_gain_split") + " parameter specifies " + "the minimum gain that is needed for the node to split. If " + PRINT_PARAM_STRING("print_training_error") + " is specified, the training " "error will be printed." "\n\n" @@ -58,8 +60,8 @@ PROGRAM_INFO("Decision tree", "call" "\n\n" + PRINT_CALL("decision_tree", "training", "data", "labels", "labels", - "output_model", "tree", "minimum_leaf_size", 20, - "print_training_error", true) + + "output_model", "tree", "minimum_leaf_size", 20, "minimum_gain_split", + 1e-3, "print_training_error", true) + "\n\n" "Then, to use that model to classify points in " + PRINT_DATASET("test_set") + " and print the test error given the " @@ -82,6 +84,8 @@ PARAM_UMATRIX_IN("test_labels", "Test point labels, if accuracy calculation " // Training parameters. PARAM_INT_IN("minimum_leaf_size", "Minimum number of points in a leaf.", "n", 20); +PARAM_DOUBLE_IN("minimum_gain_split", "Minimum gain for node splitting.", "g", + 1e-7); PARAM_FLAG("print_training_error", "Print the training error.", "e"); // Output parameters. @@ -136,6 +140,10 @@ static void mlpackMain() RequireParamValue("minimum_leaf_size", [](int x) { return x > 0; }, true, "leaf size must be positive"); + RequireParamValue("minimum_gain_split", [](double x) + { return (x > 0.0 && x < 1.0); }, true, + "gain split must be a fraction in range [0,1]"); + // Load the model or build the tree. DecisionTreeModel* model; arma::mat trainingSet; @@ -164,6 +172,8 @@ static void mlpackMain() // Now build the tree. const size_t minLeafSize = (size_t) CLI::GetParam("minimum_leaf_size"); + const double minimumGainSplit = + (double) CLI::GetParam("minimum_gain_split"); // Create decision tree with weighted labels. if (CLI::HasParam("weights")) @@ -171,12 +181,12 @@ static void mlpackMain() arma::Row weights = std::move(CLI::GetParam>("weights")); model->tree = DecisionTree<>(trainingSet, model->info, labels, - numClasses, weights, minLeafSize); + numClasses, weights, minLeafSize, minimumGainSplit); } else { model->tree = DecisionTree<>(trainingSet, model->info, labels, - numClasses, minLeafSize); + numClasses, minLeafSize, minimumGainSplit); } // Do we need to print training error? diff --git a/src/mlpack/prereqs.hpp b/src/mlpack/prereqs.hpp index d373378cf9..a8e0a70809 100644 --- a/src/mlpack/prereqs.hpp +++ b/src/mlpack/prereqs.hpp @@ -65,7 +65,7 @@ using enable_if_t = typename enable_if::type; #undef BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS #undef BOOST_MPL_LIMIT_LIST_SIZE #define BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS -#define BOOST_MPL_LIMIT_LIST_SIZE 40 +#define BOOST_MPL_LIMIT_LIST_SIZE 50 // We'll need the necessary boost::serialization features, as well as what we // use with mlpack. In Boost 1.59 and newer, the BOOST_PFTO code is no longer diff --git a/src/mlpack/tests/CMakeLists.txt b/src/mlpack/tests/CMakeLists.txt index 10b82a5c57..3296fcda94 100644 --- a/src/mlpack/tests/CMakeLists.txt +++ b/src/mlpack/tests/CMakeLists.txt @@ -75,6 +75,7 @@ add_executable(mlpack_test momentum_sgd_test.cpp nbc_test.cpp nca_test.cpp + nesterov_momentum_sgd_test.cpp nmf_test.cpp nystroem_method_test.cpp octree_test.cpp diff --git a/src/mlpack/tests/adam_test.cpp b/src/mlpack/tests/adam_test.cpp index 0d581cffdd..dff4844db3 100644 --- a/src/mlpack/tests/adam_test.cpp +++ b/src/mlpack/tests/adam_test.cpp @@ -12,6 +12,7 @@ * 3-clause BSD license along with mlpack. If not, see * http://www.opensource.org/licenses/BSD-3-Clause for more information. */ + #include #include @@ -507,4 +508,80 @@ BOOST_AUTO_TEST_CASE(NadaMaxLogisticRegressionTest) BOOST_REQUIRE_CLOSE(testAcc, 100.0, 0.6); // 0.6% error tolerance. } +/** + * Tests the OptimisticAdam optimizer using a simple test function. + */ +BOOST_AUTO_TEST_CASE(SimpleOptimisticAdamTestFunction) +{ + SGDTestFunction f; + OptimisticAdam optimizer(1e-2, 1, 0.9, 0.99, 1e-8); + + arma::mat coordinates = f.GetInitialPoint(); + optimizer.Optimize(f, coordinates); + + BOOST_REQUIRE_SMALL(coordinates[0], 0.1); + BOOST_REQUIRE_SMALL(coordinates[1], 0.1); + BOOST_REQUIRE_SMALL(coordinates[2], 0.1); +} + +/** + * Run OptimisticAdam on logistic regression and make sure the results are acceptable. + */ +BOOST_AUTO_TEST_CASE(OptimisticAdamLogisticRegressionTest) +{ + // Generate a two-Gaussian dataset. + GaussianDistribution g1(arma::vec("1.0 1.0 1.0"), + arma::eye(3, 3)); + GaussianDistribution g2(arma::vec("9.0 9.0 9.0"), + arma::eye(3, 3)); + + arma::mat data(3, 1000); + arma::Row responses(1000); + for (size_t i = 0; i < 500; ++i) + { + data.col(i) = g1.Random(); + responses[i] = 0; + } + for (size_t i = 500; i < 1000; ++i) + { + data.col(i) = g2.Random(); + responses[i] = 1; + } + + // Shuffle the dataset. + arma::uvec indices = arma::shuffle(arma::linspace(0, + data.n_cols - 1, data.n_cols)); + arma::mat shuffledData(3, 1000); + arma::Row shuffledResponses(1000); + for (size_t i = 0; i < data.n_cols; ++i) + { + shuffledData.col(i) = data.col(indices[i]); + shuffledResponses[i] = responses[indices[i]]; + } + + // Create a test set. + arma::mat testData(3, 1000); + arma::Row testResponses(1000); + for (size_t i = 0; i < 500; ++i) + { + testData.col(i) = g1.Random(); + testResponses[i] = 0; + } + for (size_t i = 500; i < 1000; ++i) + { + testData.col(i) = g2.Random(); + testResponses[i] = 1; + } + + OptimisticAdam optimisticAdam; + LogisticRegression<> lr(shuffledData, shuffledResponses, optimisticAdam, 0.5); + + // Ensure that the error is close to zero. + const double acc = lr.ComputeAccuracy(data, responses); + BOOST_REQUIRE_CLOSE(acc, 100.0, 0.3); // 0.3% error tolerance. + + const double testAcc = lr.ComputeAccuracy(testData, testResponses); + BOOST_REQUIRE_CLOSE(testAcc, 100.0, 0.6); // 0.6% error tolerance. +} + BOOST_AUTO_TEST_SUITE_END(); diff --git a/src/mlpack/tests/ann_layer_test.cpp b/src/mlpack/tests/ann_layer_test.cpp index caaa88fe64..04552fe775 100644 --- a/src/mlpack/tests/ann_layer_test.cpp +++ b/src/mlpack/tests/ann_layer_test.cpp @@ -1,6 +1,7 @@ /** * @file ann_layer_test.cpp * @author Marcus Edel + * @author Praveen Ch * * Tests the ann layer modules. * @@ -1317,4 +1318,127 @@ BOOST_AUTO_TEST_CASE(SimpleBilinearInterpolationLayerTest) arma::zeros(input.n_rows), 1e-12); } +/** + * Tests the BatchNorm Layer, compares the layers parameters with + * the values from another implementation. + * Link to the implementation - http://cthorey.github.io./backpropagation/ + */ +BOOST_AUTO_TEST_CASE(BatchNormTest) +{ + arma::mat input, output; + input << 5.1 << 3.5 << 1.4 << arma::endr + << 4.9 << 3.0 << 1.4 << arma::endr + << 4.7 << 3.2 << 1.3 << arma::endr; + + BatchNorm<> model(input.n_rows); + model.Reset(); + + // Non-Deteministic Forward Pass Test. + model.Deterministic() = false; + model.Forward(std::move(input), std::move(output)); + arma::mat result; + result << 1.1658 << 0.1100 << -1.2758 << arma::endr + << 1.2579 << -0.0699 << -1.1880 << arma::endr + << 1.1737 << 0.0958 << -1.2695 << arma::endr; + + CheckMatrices(output, result, 1e-1); + result.clear(); + + // Backward Pass Test. + arma::mat gy; + gy << 0.8402 << 0.9116 << 0.2778 << arma::endr + << 0.3944 << 0.1976 << 0.5540 << arma::endr + << 0.7831 << 0.3352 << 0.4774 << arma::endr; + + model.Backward(std::move(input), std::move(gy), std::move(output)); + result << -0.0780 << 0.1376 << -0.0596 << arma::endr + << 0.0602 << -0.1317 << 0.0715 << arma::endr + << 0.0835 << -0.1493 << 0.0658 << arma::endr; + + CheckMatrices(output, result, 1e-1); + result.clear(); + + // Gradient Test. + model.Gradient(std::move(input), std::move(gy), std::move(output)); + result << 3.4003 << arma::endr + << 0.8183 << arma::endr + << 1.8574 << arma::endr + << 2.0296 << arma::endr + << 1.1460 << arma::endr + << 1.5957 << arma::endr; + + CheckMatrices(output, result, 1e-1); + result.clear(); + + // Deterministic Forward Pass test. + output = model.TrainingMean(); + result << 3.33333333 << arma::endr + << 3.1 << arma::endr + << 3.06666666 << arma::endr; + + CheckMatrices(output, result, 1e-1); + result.clear(); + + output = model.TrainingVariance(); + result << 2.2956 << arma::endr + << 2.0467 << arma::endr + << 1.9356 << arma::endr; + + CheckMatrices(output, result, 1e-1); + result.clear(); + + model.Deterministic() = true; + model.Forward(std::move(input), std::move(output)); + + result << 1.1658 << 0.1100 << -1.2757 << arma::endr + << 1.2579 << -0.0699 << -1.1880 << arma::endr + << 1.1737 << 0.0958 << -1.2695 << arma::endr; + + CheckMatrices(output, result, 1e-1); +} + +/** + * BatchNorm layer numerically gradient test. + */ +BOOST_AUTO_TEST_CASE(GradientBatchNormLayerTest) +{ + // Add function gradient instantiation. + struct GradientFunction + { + GradientFunction() + { + input = arma::randn(10, 256); + arma::mat target; + target.ones(1, 256); + + model = new FFN, NguyenWidrowInitialization>( + input, target); + model->Add >(); + model->Add >(10); + model->Add >(10, 2); + model->Add >(); + } + + ~GradientFunction() + { + delete model; + } + + double Gradient(arma::mat& gradient) const + { + arma::mat output; + double error = model->Evaluate(model->Parameters(), 0, 256, false); + model->Gradient(model->Parameters(), 0, gradient, 256); + return error; + } + + arma::mat& Parameters() { return model->Parameters(); } + + FFN, NguyenWidrowInitialization>* model; + arma::mat input, target; + } function; + + BOOST_REQUIRE_LE(CheckGradient(function), 1e-3); +} + BOOST_AUTO_TEST_SUITE_END(); diff --git a/src/mlpack/tests/convolution_test.cpp b/src/mlpack/tests/convolution_test.cpp index a277b9cb41..739a7cae40 100644 --- a/src/mlpack/tests/convolution_test.cpp +++ b/src/mlpack/tests/convolution_test.cpp @@ -29,7 +29,7 @@ BOOST_AUTO_TEST_SUITE(ConvolutionTest); * Implementation of the convolution function test. * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output The reference output data that contains the results of the * convolution. * @@ -43,7 +43,7 @@ void Convolution2DMethodTest(const arma::mat input, arma::mat convOutput; ConvolutionFunction::Convolution(input, filter, convOutput); - // Check the outut dimension. + // Check the output dimension. bool b = (convOutput.n_rows == output.n_rows) && (convOutput.n_cols == output.n_cols); BOOST_REQUIRE_EQUAL(b, 1); @@ -59,7 +59,7 @@ void Convolution2DMethodTest(const arma::mat input, * Implementation of the convolution function test using 3rd order tensors. * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output The reference output data that contains the results of the * convolution. * @@ -91,7 +91,7 @@ void Convolution3DMethodTest(const arma::cube input, * and a 3rd order tensors as filter and output (batch modus). * * @param input Input used to perform the convolution. - * @param filter Filter used to perform the conolution. + * @param filter Filter used to perform the convolution. * @param output The reference output data that contains the results of the * convolution. * @@ -146,7 +146,7 @@ BOOST_AUTO_TEST_CASE(ValidConvolution2DTest) output); // Perform the convolution using singular value decomposition to - // speeded up the computation. + // speed up the computation. Convolution2DMethodTest >(input, filter, output); } @@ -183,7 +183,7 @@ BOOST_AUTO_TEST_CASE(FullConvolution2DTest) output); // Perform the convolution using singular value decomposition to - // speeded up the computation. + // speed up the computation. Convolution2DMethodTest >(input, filter, output); } @@ -228,7 +228,7 @@ BOOST_AUTO_TEST_CASE(ValidConvolution3DTest) filterCube, outputCube); // Perform the convolution using using the singular value decomposition to - // speeded up the computation. + // speed up the computation. Convolution3DMethodTest >(inputCube, filterCube, outputCube); } @@ -277,7 +277,7 @@ BOOST_AUTO_TEST_CASE(FullConvolution3DTest) filterCube, outputCube); // Perform the convolution using using the singular value decomposition to - // speeded up the computation. + // speed up the computation. Convolution3DMethodTest >(inputCube, filterCube, outputCube); } @@ -319,7 +319,7 @@ BOOST_AUTO_TEST_CASE(ValidConvolutionBatchTest) filterCube, outputCube); // Perform the convolution using using the singular value decomposition to - // speeded up the computation. + // speed up the computation. ConvolutionMethodBatchTest >(input, filterCube, outputCube); } @@ -365,7 +365,7 @@ BOOST_AUTO_TEST_CASE(FullConvolutionBatchTest) filterCube, outputCube); // Perform the convolution using using the singular value decomposition to - // speeded up the computation. + // speed up the computation. ConvolutionMethodBatchTest >(input, filterCube, outputCube); } diff --git a/src/mlpack/tests/decision_tree_test.cpp b/src/mlpack/tests/decision_tree_test.cpp index 63f42f6324..b2b45abd0f 100644 --- a/src/mlpack/tests/decision_tree_test.cpp +++ b/src/mlpack/tests/decision_tree_test.cpp @@ -288,10 +288,10 @@ BOOST_AUTO_TEST_CASE(BestBinaryNumericSplitSimpleSplitTest) // Call the method to do the splitting. const double bestGain = GiniGain::Evaluate(labels, 2, weights); const double gain = BestBinaryNumericSplit::SplitIfBetter( - bestGain, values, labels, 2, weights, 3, classProbabilities, aux); + bestGain, values, labels, 2, weights, 3, 1e-7, classProbabilities, aux); const double weightedGain = BestBinaryNumericSplit::SplitIfBetter(bestGain, values, - labels, 2, weights, 3, classProbabilities, aux); + labels, 2, weights, 3, 1e-7, classProbabilities, aux); // Make sure that a split was made. BOOST_REQUIRE_GT(gain, bestGain); @@ -325,11 +325,11 @@ BOOST_AUTO_TEST_CASE(BestBinaryNumericSplitMinSamplesTest) // Call the method to do the splitting. const double bestGain = GiniGain::Evaluate(labels, 2, weights); const double gain = BestBinaryNumericSplit::SplitIfBetter( - bestGain, values, labels, 2, weights, 8, classProbabilities, aux); + bestGain, values, labels, 2, weights, 8, 1e-7, classProbabilities, aux); // This should make no difference because it won't split at all. const double weightedGain = BestBinaryNumericSplit::SplitIfBetter(bestGain, values, - labels, 2, weights, 8, classProbabilities, aux); + labels, 2, weights, 8, 1e-7, classProbabilities, aux); // Make sure that no split was made. BOOST_REQUIRE_EQUAL(gain, bestGain); @@ -360,7 +360,7 @@ BOOST_AUTO_TEST_CASE(BestBinaryNumericSplitNoGainTest) // Call the method to do the splitting. const double bestGain = GiniGain::Evaluate(labels, 2, weights); const double gain = BestBinaryNumericSplit::SplitIfBetter( - bestGain, values, labels, 2, weights, 10, classProbabilities, aux); + bestGain, values, labels, 2, weights, 10, 1e-7, classProbabilities, aux); // Make sure there was no split. BOOST_REQUIRE_EQUAL(gain, bestGain); @@ -384,10 +384,11 @@ BOOST_AUTO_TEST_CASE(AllCategoricalSplitSimpleSplitTest) // Call the method to do the splitting. const double bestGain = GiniGain::Evaluate(labels, 3, weights); const double gain = AllCategoricalSplit::SplitIfBetter( - bestGain, values, 4, labels, 3, weights, 3, classProbabilities, aux); + bestGain, values, 4, labels, 3, weights, 3, 1e-7, classProbabilities, + aux); const double weightedGain = AllCategoricalSplit::SplitIfBetter(bestGain, values, 4, - labels, 3, weights, 3, classProbabilities, aux); + labels, 3, weights, 3, 1e-7, classProbabilities, aux); // Make sure that a split was made. BOOST_REQUIRE_GT(gain, bestGain); @@ -419,7 +420,8 @@ BOOST_AUTO_TEST_CASE(AllCategoricalSplitMinSamplesTest) // Call the method to do the splitting. const double bestGain = GiniGain::Evaluate(labels, 3, weights); const double gain = AllCategoricalSplit::SplitIfBetter( - bestGain, values, 4, labels, 3, weights, 4, classProbabilities, aux); + bestGain, values, 4, labels, 3, weights, 4, 1e-7, classProbabilities, + aux); // Make sure it's not split. BOOST_REQUIRE_EQUAL(gain, bestGain); @@ -451,10 +453,11 @@ BOOST_AUTO_TEST_CASE(AllCategoricalSplitNoGainTest) // Call the method to do the splitting. const double bestGain = GiniGain::Evaluate(labels, 3, weights); const double gain = AllCategoricalSplit::SplitIfBetter( - bestGain, values, 10, labels, 3, weights, 10, classProbabilities, aux); + bestGain, values, 10, labels, 3, weights, 10, 1e-7, classProbabilities, + aux); const double weightedGain = AllCategoricalSplit::SplitIfBetter(bestGain, values, 10, - labels, 3, weights, 10, classProbabilities, aux); + labels, 3, weights, 10, 1e-7, classProbabilities, aux); // Make sure that there was no split. BOOST_REQUIRE_EQUAL(gain, bestGain); @@ -539,7 +542,7 @@ BOOST_AUTO_TEST_CASE(PerfectTrainingSet) } /** - * onstruct the decision tree with weighted labels + * Construct the decision tree with weighted labels */ BOOST_AUTO_TEST_CASE(PerfectTrainingSetWithWeight) { @@ -1082,4 +1085,45 @@ BOOST_AUTO_TEST_CASE(ConstDataTest) constWeights); } +/** + * Construct the decision tree with splitting only if gain is more than + * threshold. + */ +BOOST_AUTO_TEST_CASE(RegularisedDecisionTree) +{ + // Completely random dataset with no structure. + arma::mat dataset(10, 1000, arma::fill::randu); + arma::Row labels(1000); + for (size_t i = 0; i < 1000; ++i) + labels[i] = i % 3; // 3 classes. + arma::rowvec weights(labels.n_elem); + weights.ones(); + + // Minimum leaf size of 1. + DecisionTree<> d(dataset, labels, 3, weights, 1, 1e-7); + + // Minimum leaf size of 1 and Minimum gain split of 0.01. + DecisionTree<> dRegularised(dataset, labels, 3, weights, 1, 0.01); + + size_t count = 0; + // This part of code is dupliacte with no weighted one. + for (size_t i = 0; i < 1000; ++i) + { + size_t prediction, predictionsregularised; + arma::vec probabilities, probabilitiesRegularised; + + d.Classify(dataset.col(i), prediction, probabilities); + dRegularised.Classify(dataset.col(i), predictionsregularised, + probabilitiesRegularised); + + if (prediction != predictionsregularised) + count++; + + BOOST_REQUIRE_EQUAL(probabilities.n_elem, 3); + BOOST_REQUIRE_EQUAL(probabilitiesRegularised.n_elem, 3); + } + + BOOST_REQUIRE_GT(count, 0); +} + BOOST_AUTO_TEST_SUITE_END(); diff --git a/src/mlpack/tests/main_tests/decision_tree_test.cpp b/src/mlpack/tests/main_tests/decision_tree_test.cpp index 1d47b4c127..56ae2c88de 100644 --- a/src/mlpack/tests/main_tests/decision_tree_test.cpp +++ b/src/mlpack/tests/main_tests/decision_tree_test.cpp @@ -167,6 +167,89 @@ BOOST_AUTO_TEST_CASE(DecisionTreeMinimumLeafSizeTest) Log::Fatal.ignoreInput = false; } +/** + * Make sure minimum gain split is always a fraction in range [0,1]. + */ +BOOST_AUTO_TEST_CASE(DecisionMinimumGainSplitTest) +{ + arma::mat inputData; + DatasetInfo info; + if (!data::Load("braziltourism.arff", inputData, info)) + BOOST_FAIL("Cannot load train dataset braziltourism.arff!"); + + arma::Row labels; + if (!data::Load("braziltourism_labels.txt", labels)) + BOOST_FAIL("Cannot load labels for braziltourism_labels.txt"); + + // Initialize an all-ones weight matrix. + arma::mat weights(1, labels.n_cols, arma::fill::ones); + + // Input training data. + SetInputParam("training", std::move(std::make_tuple(info, inputData))); + SetInputParam("labels", std::move(labels)); + SetInputParam("weights", std::move(weights)); + + SetInputParam("minimum_gain_split", 1.5); // Invalid. + + Log::Fatal.ignoreInput = true; + BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error); + Log::Fatal.ignoreInput = false; +} + +/** + * Make sure minimum gain split produces regularised tree. + */ +BOOST_AUTO_TEST_CASE(DecisionRegularisationTest) +{ + arma::mat inputData; + DatasetInfo info; + if (!data::Load("braziltourism.arff", inputData, info)) + BOOST_FAIL("Cannot load train dataset braziltourism.arff!"); + + arma::Row labels; + if (!data::Load("braziltourism_labels.txt", labels)) + BOOST_FAIL("Cannot load labels for braziltourism_labels.txt"); + + // Initialize an all-ones weight matrix. + arma::mat weights(1, labels.n_cols, arma::fill::ones); + + // Input training data. + SetInputParam("training", std::make_tuple(info, inputData)); + SetInputParam("labels", labels); + SetInputParam("weights", weights); + + SetInputParam("minimum_gain_split", 1e-7); + + // Input test data. + SetInputParam("test", std::make_tuple(info, inputData)); + arma::Row pred; + mlpackMain(); + pred = std::move(CLI::GetParam>("predictions")); + + // Input training data. + SetInputParam("training", std::make_tuple(info, inputData)); + SetInputParam("labels", std::move(labels)); + SetInputParam("weights", std::move(weights)); + + SetInputParam("minimum_gain_split", 0.01); + + // Input test data. + SetInputParam("test", std::move(std::make_tuple(info, inputData))); + arma::Row predRegularised; + mlpackMain(); + predRegularised = std::move(CLI::GetParam>("predictions")); + + size_t count = 0; + // This part of code is dupliacte with no weighted one. + for (size_t i = 0; i < 1000; ++i) + { + if (pred[i] != predRegularised[i]) + count++; + } + + BOOST_REQUIRE_GT(count, 0); +} + /** * Ensure that saved model can be used again. */ diff --git a/src/mlpack/tests/nesterov_momentum_sgd_test.cpp b/src/mlpack/tests/nesterov_momentum_sgd_test.cpp new file mode 100644 index 0000000000..265a112965 --- /dev/null +++ b/src/mlpack/tests/nesterov_momentum_sgd_test.cpp @@ -0,0 +1,72 @@ +/** + * @file nesterov_momentum_sgd_test.cpp + * @author Sourabh Varshney + * + * Test file for NesterovMomentumSGD (Stochastic gradient descent with + * nesterov momentum updates). + * + * mlpack is free software; you may redistribute it and/or modify it under the + * terms of the 3-clause BSD license. You should have received a copy of the + * 3-clause BSD license along with mlpack. If not, see + * http://www.opensource.org/licenses/BSD-3-Clause for more information. + */ +#include +#include +#include +#include +#include +#include + +#include +#include "test_tools.hpp" + +using namespace std; +using namespace arma; +using namespace mlpack; +using namespace mlpack::optimization; +using namespace mlpack::optimization::test; + +BOOST_AUTO_TEST_SUITE(NesterovMomentumSGDTest); + +/* +* Tests the Nesterov Momentum SGD update policy. +*/ +BOOST_AUTO_TEST_CASE(NesterovMomentumSGDSpeedUpTestFunction) +{ + SGDTestFunction f; + NesterovMomentumUpdate nesterovMomentumUpdate(0.9); + NesterovMomentumSGD s(0.0003, 1, 2500000, 1e-9, true, + nesterovMomentumUpdate); + + arma::mat coordinates = f.GetInitialPoint(); + double result = s.Optimize(f, coordinates); + + BOOST_REQUIRE_CLOSE(result, -1.0, 0.15); + BOOST_REQUIRE_SMALL(coordinates[0], 1e-3); + BOOST_REQUIRE_SMALL(coordinates[1], 1e-7); + BOOST_REQUIRE_SMALL(coordinates[2], 1e-7); +} + +/* +* Tests the Nesterov Momentum SGD with Generalized Rosenbrock Test. +*/ +BOOST_AUTO_TEST_CASE(GeneralizedRosenbrockTest) +{ + // Loop over several variants. + for (size_t i = 10; i < 50; i += 5) + { + // Create the generalized Rosenbrock function. + GeneralizedRosenbrockFunction f(i); + NesterovMomentumUpdate nesterovMomentumUpdate(0.9); + NesterovMomentumSGD s(0.0001, 1, 0, 1e-15, true, nesterovMomentumUpdate); + + arma::mat coordinates = f.GetInitialPoint(); + double result = s.Optimize(f, coordinates); + + BOOST_REQUIRE_SMALL(result, 1e-4); + for (size_t j = 0; j < i; ++j) + BOOST_REQUIRE_CLOSE(coordinates[j], (double) 1.0, 1e-3); + } +} + +BOOST_AUTO_TEST_SUITE_END();