apaam/mlpack
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' into misc-test-fixes

Browse Source
This commit is contained in:
Ryan Curtin
2018-03-22 16:48:17 -04:00
parent 1bf681a284 793fe4abe4
commit f4e364413b
37 changed files with 1857 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files
COPYRIGHT.txt
+2
View File
@@ -96,6 +96,8 @@ Copyright:
Copyright 2018, Roberto Hueso <robertohueso96@gmail.com>
Copyright 2018, Prabhat Sharma <prabhatsharma7298@gmail.com>
Copyright 2018, Tan Jun An <yamidarkxxx@gmail.com>
Copyright 2018, Moksh Jain <mokshjn00@gmail.com>
Copyright 2018, Manthan-R-Sheth <manthanrsheth96@gmail.com>
License: BSD-3-clause
All rights reserved.
doc/tutorials/README.md
+1 -1
View File
@@ -1,7 +1,7 @@
## Tutorials
Tutorials for mlpack can be found [here : mlpack tutorials](http://www.mlpack.org/tutorials.html).
Tutorials for mlpack can be found [here : mlpack tutorials](https://www.mlpack.org/docs/mlpack-git/doxygen/tutorials.html).
### General mlpack tutorials
doc/tutorials/ann/ann.txt
+674
View File
@@ -0,0 +1,674 @@
/*!
@file ann.txt
@author Marcus Edel (kurg.org)
@brief Tutorial for how to use the neural network code in mlpack.
@page anntutorial Neural Network tutorial
@section intro_anntut Introduction
There is vast literature on neural networks and their uses, as well as
strategies for choosing initial points effectively, keeping the algorithm from
converging in local minima, choosing the best model structure, choosing the best
optimizers, and so forth. mlpack implements many of these building blocks,
making it very easy to create different neural networks in a modular way.
mlpack currently implements two easy-to-use forms of neural networks: \c Feed-
Forward \c Networks (this includes convolutional neural networks) and \c
Recurrent \c Neural \c Networks.
@section toc_anntut Table of Contents
This tutorial is split into the following sections:
- \ref intro_anntut
- \ref toc_anntut
- \ref model_api_anntut
- \ref layer_api_anntut
- \ref model_setup_training_anntut
- \ref model_saving_loading_anntut
- \ref extracting_parameters_anntut
- \ref further_anntut
@section model_api_anntut Model API
There are two main neural network classes that are meant to be used as container
for neural network layers that \b mlpack implements; each class is suited to a
different setting:
- \c FFN: the Feed Forward Network model provides a means to plug layers
together in a feed-forward fully connected manner. This is the 'standard'
type of deep learning model, and includes convolutional neural networks
(CNNs).
- \c RNN: the Recurrent Neural Network model provides a means to consider
successive calls to forward as different time-steps in a sequence. This is
often used for time sequence modeling tasks, such as predicting the next
character in a sequence.
Below is some basic guidance on what should be used. Note that the question of
"which algorithm should be used" is a very difficult question to answer, so the
guidance below is just that---guidance---and may not be right for a particular
problem.
- \c Feed-forward Networks allow signals or inputs to travel one way only.
There is no feedback within the network; for instance, the output of any
layer does only affect the upcoming layer. That makes Feed-Forward Networks
straightforward and very effective. They are extensively used in pattern
recognition and are ideally suitable for modeling relationships between a
set of input and one or more output variables.
- \c Recurrent Networks allow signals or inputs to travel in both directions by
introducing loops in the network. Computations derived from earlier inputs are
fed back into the network, which gives the recurrent network some kind of
memory. RNNs are currently being used for all kinds of sequential tasks; for
instance, time series prediction, sequence labeling, and
sequence classification.
In order to facilitate consistent implementations, the \c FFN and \c RNN classes
have a number of methods in common:
- \c Train(): trains the initialized model on the given input data. Optionally
an optimizer object can be passed to control the optimization process.
- \c Predict(): predicts the responses to a given set of predictors. Note the
responses will reflect the output of the specified output layer.
- \c Add(): this method can be used to add a layer to the model.
@note
To be able to optimize the network, both classes implement the OptimizerFunction
API; see \ref optimizertutorial "Optimizer API" for more information. In short,
the \c FNN and \c RNN class implement two methods: \c Evaluate() and \c
Gradient(). This enables the optimization given some learner and some
performance measure.
Similar to the existing layer infrastructure, the \c FFN and \c RNN classes are
very extensible, having the following template arguments; which can be modified
to change the behavior of the network:
- \c OutputLayerType: this type defines the output layer used to evaluate the
network; by default, \c NegativeLogLikelihood is used.
- \c InitializationRuleType: this type defines the method by which initial
parameters are set; by default, \c RandomInitialization is used.
@code
template<
typename OutputLayerType = NegativeLogLikelihood<>,
typename InitializationRuleType = RandomInitialization
>
class FNN;
@endcode
Internally, the \c FFN and \c RNN class keeps an instantiated \c OutputLayerType
class (which can be given in the constructor). This is useful for using
different loss functions like the Negative-Log-Likelihood function or the \c
VRClassReward function, which takes an optional score parameter. Therefore, you
can write a non-static OutputLayerType class and use it seamlessly in
combination with the \c FNN and \c RNN class. The same applies to the \c
InitializationRuleType template parameter.
By choosing different components for each of these template classes in
conjunction with the \c Add() method, a very arbitrary network object can be
constructed.
Below are several examples of how the \c FNN and \c RNN classes might be used.
The first examples focus on the \c FNN class, and the last shows how the \c
RNN class can be used.
The simplest way to use the FNN<> class is to pass in a dataset with the
corresponding labels, and receive the classification in return. Note that the
dataset must be column-major – that is, one column corresponds to one point. See
the \ref matrices "matrices guide" for more information.
The code below builds a simple feed-forward network with the default options,
then queries for the assignments for every point in the \c queries matrix.
\dot
digraph G {
fontname = "Hilda 10"
rankdir=LR
splines=line
nodesep=.08;
ranksep=1;
edge [color=black, arrowsize=.5];
node [fixedsize=true,label="",style=filled,color=none,fillcolor=gray,shape=circle]
subgraph cluster_0 {
color=none;
node [style=filled, color=white, penwidth=15,fillcolor=black shape=circle];
l10 l11 l12 l13 l14 l15 ;
label = Input;
}
subgraph cluster_1 {
color=none;
node [style=filled, color=white, penwidth=15,fillcolor=gray shape=circle];
l20 l21 l22 l23 l24 l25 l26 l27 ;
label = Linear;
}
subgraph cluster_2 {
color=none;
node [style=filled, color=white, penwidth=15,fillcolor=gray shape=circle];
l30 l31 l32 l33 l34 l35 l36 l37 ;
label = Linear;
}
subgraph cluster_3 {
color=none;
node [style=filled, color=white, penwidth=15,fillcolor=black shape=circle];
l40 l41 l42 ;
label = LogSoftMax;
}
l10 -> l20 l10 -> l21 l10 -> l22 l10 -> l23 l10 -> l24 l10 -> l25
l10 -> l26 l10 -> l27 l11 -> l20 l11 -> l21 l11 -> l22 l11 -> l23
l11 -> l24 l11 -> l25 l11 -> l26 l11 -> l27 l12 -> l20 l12 -> l21
l12 -> l22 l12 -> l23 l12 -> l24 l12 -> l25 l12 -> l26 l12 -> l27
l13 -> l20 l13 -> l21 l13 -> l22 l13 -> l23 l13 -> l24 l13 -> l25
l13 -> l26 l13 -> l27 l14 -> l20 l14 -> l21 l14 -> l22 l14 -> l23
l14 -> l24 l14 -> l25 l14 -> l26 l14 -> l27 l15 -> l20 l15 -> l21
l15 -> l22 l15 -> l23 l15 -> l24 l15 -> l25 l15 -> l26 l15 -> l27
l20 -> l30 l20 -> l31 l20 -> l32 l20 -> l33 l20 -> l34 l20 -> l35
l20 -> l36 l20 -> l37 l21 -> l30 l21 -> l31 l21 -> l32 l21 -> l33
l21 -> l34 l21 -> l35 l21 -> l36 l21 -> l37 l22 -> l30 l22 -> l31
l22 -> l32 l22 -> l33 l22 -> l34 l22 -> l35 l22 -> l36 l22 -> l37
l23 -> l30 l23 -> l31 l23 -> l32 l23 -> l33 l23 -> l34 l23 -> l35
l23 -> l36 l23 -> l37 l24 -> l30 l24 -> l31 l24 -> l32 l24 -> l33
l24 -> l34 l24 -> l35 l24 -> l36 l24 -> l37 l25 -> l30 l25 -> l31
l25 -> l32 l25 -> l33 l25 -> l34 l25 -> l35 l25 -> l36 l25 -> l37
l26 -> l30 l26 -> l31 l26 -> l32 l26 -> l33 l26 -> l34 l26 -> l35
l26 -> l36 l26 -> l37 l27 -> l30 l27 -> l31 l27 -> l32 l27 -> l33
l27 -> l34 l27 -> l35 l27 -> l36 l27 -> l37 l30 -> l40 l30 -> l41
l30 -> l42 l31 -> l40 l31 -> l41 l31 -> l42 l32 -> l40 l32 -> l41
l32 -> l42 l33 -> l40 l33 -> l41 l33 -> l42 l34 -> l40 l34 -> l41
l34 -> l42 l35 -> l40 l35 -> l41 l35 -> l42 l36 -> l40 l36 -> l41
l36 -> l42 l37 -> l40 l37 -> l41 l37 -> l42
}
\enddot
@note
The number of inputs in the above graph doesn't match with the real
number of features in the thyroid dataset and are just used as an abstract
representation.
@code
// Load the training set.
arma::mat dataset;
data::Load("thyroid_train.csv", dataset, true);
// Split the labels from the training set.
arma::mat trainData = dataset.submat(0, 0, dataset.n_rows - 4,
dataset.n_cols - 1);
// Split the data from the training set.
arma::mat trainLabelsTemp = dataset.submat(dataset.n_rows - 3, 0,
dataset.n_rows - 1, dataset.n_cols - 1);
// Initialize the network.
FFN<> model;
model.Add<Linear<> >(trainData.n_rows, 8);
model.Add<SigmoidLayer<> >();
model.Add<Linear<> >(8, 3);
model.Add<LogSoftMax<> >();
// Train the model.
model.Train(trainData, trainLabels);
// Use the Predict method to get the assignments.
arma::mat assignments;
model.Predict(trainData, assignments);
@endcode
Now, the matrix assignments holds the classification of each point in the
dataset.
In the next example, we create simple noisy sine sequences, which are trained
later on, using the RNN class.
@code
void GenerateNoisySines(arma::mat& data,
arma::mat& labels,
const size_t points,
const size_t sequences,
const double noise = 0.3)
{
arma::colvec x = arma::linspace<arma::Col<double>>(0,
points - 1, points) / points * 20.0;
arma::colvec y1 = arma::sin(x + arma::as_scalar(arma::randu(1)) * 3.0);
arma::colvec y2 = arma::sin(x / 2.0 + arma::as_scalar(arma::randu(1)) * 3.0);
data = arma::zeros(points, sequences * 2);
labels = arma::zeros(2, sequences * 2);
for (size_t seq = 0; seq < sequences; seq++)
{
data.col(seq) = arma::randu(points) * noise + y1 +
arma::as_scalar(arma::randu(1) - 0.5) * noise;
labels(0, seq) = 1;
data.col(sequences + seq) = arma::randu(points) * noise + y2 +
arma::as_scalar(arma::randu(1) - 0.5) * noise;
labels(1, sequences + seq) = 1;
}
const size_t rho = 10;
// Generate 12 (2 * 6) noisy sines. A single sine contains rho
// points/features.
arma::mat input, labelsTemp;
GenerateNoisySines(input, labelsTemp, rho, 6);
arma::mat labels = arma::zeros<arma::mat>(rho, labelsTemp.n_cols);
for (size_t i = 0; i < labelsTemp.n_cols; ++i)
{
const int value = arma::as_scalar(arma::find(
arma::max(labelsTemp.col(i)) == labelsTemp.col(i), 1)) + 1;
labels.col(i).fill(value);
}
/**
* Construct a network with 1 input unit, 4 hidden units and 10 output
* units. The hidden layer is connected to itself. The network structure
* looks like:
*
* Input Hidden Output
* Layer(1) Layer(4) Layer(10)
* +-----+ +-----+ +-----+
* | | | | | |
* | +------>| +------>| |
* | | ..>| | | |
* +-----+ . +--+--+ +-----+
* . .
* . .
* .......
*/
Add<> add(4);
Linear<> lookup(1, 4);
SigmoidLayer<> sigmoidLayer;
Linear<> linear(4, 4);
Recurrent<> recurrent(add, lookup, linear, sigmoidLayer, rho);
RNN<> model(rho);
model.Add<IdentityLayer<> >();
model.Add(recurrent);
model.Add<Linear<> >(4, 10);
model.Add<LogSoftMax<> >();
StandardSGD opt(0.1, 1, input.n_cols /* 1 epoch */, -100);
model.Train(input, labels, opt);
}
@endcode
For further examples on the usage of the ann classes, see [mlpack
models](https://github.com/mlpack/models).
@section layer_api_anntut Layer API
In order to facilitate consistent implementations, we have defined a LayerType
API that describes all the methods that a \c layer may implement. mlpack offers
a few variations of this API, each designed to cover some of the model
characteristics mentioned in the previous section. Any \c layer requires the
implementation of a \c Forward() method. The interface looks like:
@code
template<typename eT>
void Forward(const arma::Mat<eT>&& input, arma::Mat<eT>&& output);
@endcode
The method should calculate the output of the layer given the input matrix and
store the result in the given output matrix. Next, any \c layer must implement
the Backward() method, which uses certain computations obtained during the
forward pass and should calculate the function f(x) by propagating x backward
through f:
@code
template<typename eT>
void Backward(const arma::Mat<eT>&& input,
arma::Mat<eT>&& gy,
arma::Mat<eT>&& g);
@endcode
Finally, if the layer is differentiable, the layer must also implement
a Gradient() method:
@code
template<typename eT>
void Gradient(const arma::Mat<eT>&& input,
arma::Mat<eT>&& error,
arma::Mat<eT>&& gradient);
@endcode
The Gradient function should calculate the gradient with respect to the input
activations \c input and calculated errors \c error and place the results into
the gradient matrix object \c gradient that is passed as an argument.
@note
Note that each method accepts a template parameter InputType, OutputType
or GradientType, which may be arma::mat (dense Armadillo matrix) or arma::sp_mat
(sparse Armadillo matrix). This allows support for both sparse-supporting and
non-sparse-supporting \c layer without explicitly passing the type.
In addition, each layer must implement the Parameters(), InputParameter(),
OutputParameter(), Delta() methods, differentiable layer should also provide
access to the gradient by implementing the Gradient(), Parameters() member
function. Note each function is a single line that looks like:
@code
OutputDataType const& Parameters() const { return weights; }
@endcode
Below is an example that shows each function with some additional boilerplate
code.
@note
Note this is not an actual layer but instead an example that exists to show and
document all the functions that mlpack layer must implement. For a better
overview of the various layers, see \ref mlpack::ann. Also be aware that the
implementations of each of the methods in this example are entirely fake and do
not work; this example exists for its API, not its implementation.
Note that layer sometimes have different properties. These properties are
known at compile-time through the mlpack::ann::LayerTraits class, and some
properties may imply the existence (or non-existence) of certain functions.
Refer to the LayerTraits @ref LayerTraits for more documentation on that.
The two template parameters below must be template parameters to the layer, in
the order given below. More template parameters are fine, but they must come
after the first two.
- \c InputDataType: this defines the internally used input type for example to
store the parameter matrix. Note, a layer could be built on a dense matrix or
a sparse matrix. All mlpack trees should be able to support any Armadillo-
compatible matrix type. When the layer is written it should be assumed that
MatType has the same functionality as arma::mat. Note that
- \c OutputDataType: this defines the internally used input type for example to
store the parameter matrix. Note, a layer could be built on a dense matrix or
a sparse matrix. All mlpack trees should be able to support any Armadillo-
compatible matrix type. When the layer is written it should be assumed that
MatType has the same functionality as arma::mat.
@code
template<typename InputDataType = arma::mat,
typename OutputDataType = arma::mat>
class ExampleLayer
{
public:
ExampleLayer(const size_t inSize, const size_t outSize) :
inputSize(inSize), outputSize(outSize)
{
/* Nothing to do here */
}
}
@endcode
The constructor for \c ExampleLayer will build the layer given the input and
output size. Note that, if the input or output size information isn't used
internally it's not necessary to provide a specific constructor. Also, one could
add additional or other information that are necessary for the layer
construction. One example could be:
@code
ExampleLayer(const double ratio = 0.5) : ratio(ratio) {/* Nothing to do here*/}
@endcode
When this constructor is finished, the entire layer will be built and is ready
to be used. Next, as pointed out above, each layer has to follow the LayerType
API, so we must implement some additional functions.
@code
template<typename InputType, typename OutputType>
void Forward(const InputType&& input, OutputType&& output)
{
output = arma::ones(input.n_rows, input.n_cols);
}
template<typename InputType, typename ErrorType, typename GradientType>
void Backward(const InputType&& input, ErrorType&& gy, GradientType&& g)
{
g = arma::zeros(gy.n_rows, gy.n_cols) + gy;
}
template<typename InputType, typename ErrorType, typename GradientType>
void Gradient(const InputType&& input,
ErrorType&& error,
GradientType&& gradient)
{
gradient = arma::zeros(input.n_rows, input.n_cols) * error;
}
@endcode
The three functions \c Forward(), \c Backward() and \c Gradient() (which is
needed for a differentiable layer) contain the main logic of the layer. The
following functions are just to access and manipulate the different layer
parameters.
@code
OutputDataType& Parameters() { return weights; }
InputDataType& InputParameter() { return inputParameter; }
OutputDataType& OutputParameter() { return outputParameter; }
OutputDataType& Delta() { return delta; }
OutputDataType& Gradient() { return gradient; }
@endcode
Since some of this methods return internal class members we have to define them.
@code
private:
size_t inSize, outSize;
OutputDataType weights, delta, gradient, outputParameter;
InputDataType inputParameter;
@endcode
Note some members are just here so \c ExampleLayer compiles without warning.
For instance, \c inputSize is not required to be a member of every type of
layer.
There is one last method that is especially interesting for a layer that shares
parameter. Since the layer weights are set once the complete model is defined,
it's not possible to split the weights during the construction time. To solve
this issue, a layer can implement the \c Reset() method which is called once the
layer parameter is set.
@section model_setup_training_anntut Model Setup & Training
Once the base container is selected (\c FNN or \c RNN), the \c Add method can be
used to add layers to the model. The code below adds two linear layers to the
model---the first takes 512 units as input and gives 256 output units, and
the second takes 256 units as input and gives 128 output units.
@code
FFN<> model;
model.Add<Linear<> >(512, 256);
model.Add<Linear<> >(256, 128);
@endcode
The model is trained on Armadillo matrices. For training a model, you will
typically use the \c Train() function:
@code
arma::mat trainingSet, trainingLabels;
model.Train(trainingSet, trainingLabels);
@endcode
You can use mlpack's \c Load() function to load a dataset like this:
@code
arma::mat trainingSet;
data::Load("dataset.csv", dataset, true);
@endcode
@code
$ cat dataset.csv
0, 1, 4
1, 0, 5
1, 1, 1
2, 0, 2
@endcode
The type does not necessarily need to be a CSV; it can be any supported storage
format, assuming that it is a coordinate-format file in the format specified
above. For more information on mlpack file formats, see the documentation for
mlpack::data::Load().
@note
It’s often a good idea to normalize or standardize your data, for example using:
@code
for (size_t i = 0; i < dataset.n_cols; ++i)
dataset.col(i) /= norm(dataset.col(i), 2);
@endcode
Also, it is possible to retrain a model with new parameters or with
a new reference set. This is functionally equivalent to creating a new model.
@section model_saving_loading_anntut Saving & Loading
Using \c boost::serialization (for more information about the internals see
[Serialization - Boost C++ Libraries](www.boost.org/libs/serialization/doc/)),
mlpack is able to load and save machine learning models with ease. To save a
trained neural network to disk. The example below builds a model on the \c
thyroid dataset and then saves the model to the file \c model.xml for later use.
@code
// Load the training set.
arma::mat dataset;
data::Load("thyroid_train.csv", dataset, true);
// Split the labels from the training set.
arma::mat trainData = dataset.submat(0, 0, dataset.n_rows - 4,
dataset.n_cols - 1);
// Split the data from the training set.
arma::mat trainLabelsTemp = dataset.submat(dataset.n_rows - 3, 0,
dataset.n_rows - 1, dataset.n_cols - 1);
// Initialize the network.
FFN<> model;
model.Add<Linear<> >(trainData.n_rows, 3);
model.Add<SigmoidLayer<> >();
model.Add<LogSoftMax<> >();
// Train the model.
model.Train(trainData, trainLabels);
// Use the Predict method to get the assignments.
arma::mat assignments;
model.Predict(trainData, assignments);
data::Save("model.xml", "model", model, false);
@endcode
After this, the file model.xml will be available in the current working
directory.
Now, we can look at the output model file, \c model.xml:
@code
$ cat model.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE boost_serialization>
<boost_serialization signature="serialization::archive" version="15">
<model class_id="0" tracking_level="0" version="0">
<parameter class_id="1" tracking_level="1" version="0" object_id="_0">
<n_rows>66</n_rows>
<n_cols>1</n_cols>
<n_elem>66</n_elem>
<vec_state>0</vec_state>
<item>-7.55971528334903642e+00</item>
<item>-9.95435955058058930e+00</item>
<item>9.31133928948225353e+00</item>
<item>-5.36784434861701953e+00</item>
...
</parameter>
<width>0</width>
<height>0</height>
<currentInput object_id="_1">
<n_rows>0</n_rows>
<n_cols>0</n_cols>
<n_elem>0</n_elem>
<vec_state>0</vec_state>
</currentInput>
<network class_id="2" tracking_level="0" version="0">
<count>3</count>
<item_version>0</item_version>
<item class_id="3" tracking_level="0" version="0">
<which>18</which>
<value class_id="4" tracking_level="1" version="0" object_id="_2">
<inSize>21</inSize>
<outSize>3</outSize>
</value>
</item>
<item>
<which>2</which>
<value class_id="5" tracking_level="1" version="0" object_id="_3"></value>
</item>
<item>
<which>20</which>
<value class_id="6" tracking_level="1" version="0" object_id="_4"></value>
</item>
</network>
</model>
</boost_serialization>
@endcode
As you can see, the \c <parameter> section of \c model.xml contains the trained
network weights. We can see that this section also contains the network input
size, which is 66 rows and 1 column. Note that in this example, we used three
different layers, as can be seen by looking at the \c <network> section. Each
node has a unique id that is used to reconstruct the model when loading.
The models can also be saved as \c .bin or \c .txt; the \c .xml format provides
a human-inspectable format (though the models tend to be quite complex and may
be difficult to read). These models can then be re-used to be used for
classification or other tasks.
So, instead of saving or training a network, mlpack can also load a pre-trained
model. For instance, the example below will load the model from \c model.xml and
then generate the class predictions for the \c thyroid test dataset.
@code
data::Load("thyroid_test.csv", dataset, true);
arma::mat testData = dataset.submat(0, 0, dataset.n_rows - 4,
dataset.n_cols - 1);
data::Load("model.xml", "model", model);
arma::mat predictions;
model.Predict(testData, predictions);
@endcode
This enables the possibility to distribute a model without having to train it
first or simply to save a model for later use. Note that loading will also work
on different machines.
@section extracting_parameters_anntut Extracting Parameters
To access the weights from the neural network layers, you can call the following
function on any initialized network:
@code
model.Parameters();
@endcode
which will return the complete model parameters as an armadillo matrix object;
however often it is useful to not only have the parameters for the complete
network, but the parameters of a specific layer. Another method, \c Model(),
makes this easily possible:
@code
model.Model()[1].Parameters();
@endcode
In the example above, we get the weights of the second layer.
@section further_anntut Further documentation
For further documentation on the ann classes, consult the \ref mlpack::ann
"complete API documentation".
*/
doc/tutorials/tutorials.txt
+1
View File
@@ -39,6 +39,7 @@ progress to complex, extensible uses.
- \ref cftutorial
- \ref akfntutorial
- \ref cnetutorial
- \ref anntutorial
@section policy_tut Policy Class Documentation
src/mlpack/core.hpp
+2
View File
@@ -239,6 +239,8 @@
* - Roberto Hueso <robertohueso96@gmail.com>
* - Prabhat Sharma <prabhatsharma7298@gmail.com>
* - Tan Jun An <yamidarkxxx@gmail.com>
* - Moksh Jain <mokshjn00@gmail.com>
* - Manthan-R-Sheth <manthanrsheth96@gmail.com>
*/
// First, include all of the prerequisites.
src/mlpack/core/math/random.hpp
+17
View File
@@ -48,6 +48,23 @@ inline void RandomSeed(const size_t seed)
#endif
}
/**
* Set the random seed to a fixed number.
* This function is used in binding tests to set a fixed random seed before
* calling mlpack(). In this way we can test whether a certain parameter makes
* a difference to execution of CLI binding.
* Refer to pull request #1306 for discussion on this function.
*/
#if (BINDING_TYPE == BINDING_TYPE_TEST)
inline void FixedRandomSeed()
{
const static size_t seed = rand();
randGen.seed((uint32_t) seed);
srand((unsigned int) seed);
arma::arma_rng::set_seed(seed);
}
#endif
/**
* Generates a uniform random number between 0 and 1.
*/
src/mlpack/core/optimizers/adam/CMakeLists.txt
+1
View File
@@ -6,6 +6,7 @@ set(SOURCES
amsgrad_update.hpp
nadam_update.hpp
nadamax_update.hpp
optimisticadam_update.hpp
)
set(DIR_SRCS)
src/mlpack/core/optimizers/adam/adam.hpp
+3
View File
@@ -30,6 +30,7 @@
#include "amsgrad_update.hpp"
#include "nadam_update.hpp"
#include "nadamax_update.hpp"
#include "optimisticadam_update.hpp"
namespace mlpack {
namespace optimization {
@@ -186,6 +187,8 @@ using Nadam = AdamType<NadamUpdate>;
using NadaMax = AdamType<NadaMaxUpdate>;
using OptimisticAdam = AdamType<OptimisticAdamUpdate>;
} // namespace optimization
} // namespace mlpack
src/mlpack/core/optimizers/adam/optimisticadam_update.hpp
+148
View File
@@ -0,0 +1,148 @@
/**
* @file optimisticadam_update.hpp
* @author Moksh Jain
*
* OptmisticAdam optimizer. Implements Optimistic Adam, an algorithm which
* uses Optimistic Mirror Descent with the Adam optimizer.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_CORE_OPTIMIZERS_ADAM_OPTIMISTICADAM_UPDATE_HPP
#define MLPACK_CORE_OPTIMIZERS_ADAM_OPTIMISTICADAM_UPDATE_HPP
#include <mlpack/prereqs.hpp>
namespace mlpack {
namespace optimization {
/**
* OptimisticAdam is an optimizer which implements the Optimistic Adam
* algorithm which uses Optmistic Mirror Descent with the Adam Optimizer.
* It addresses the problem of limit cycling while training GANs. It uses
* OMD to achieve faster regret rates in solving the zero sum game of
* training a GAN. It consistently achieves a smaller KL divergnce with
* respect to the true underlying data distribution.
*
* For more information, see the following.
*
* @code
* @article{
* author = {Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis,
* Haoyang Zeng},
* title = {Training GANs with Optimism},
* year = {2017},
* url = {https://arxiv.org/abs/1711.00141}
* }
* @endcode
*/
class OptimisticAdamUpdate
{
public:
/**
* Construct the OptimisticAdam update policy with the given parameters.
*
* @param epsilon The epsilon value used to initialize the squared gradient
* parameter.
* @param beta1 The smoothing parameter.
* @param beta2 The second moment coefficient.
*/
OptimisticAdamUpdate(const double epsilon = 1e-8,
const double beta1 = 0.9,
const double beta2 = 0.999) :
epsilon(epsilon),
beta1(beta1),
beta2(beta2),
iteration(0)
{
// Nothing to do.
}
/**
* The Initialize method is called by SGD Optimizer method before the start of
* the iteration update process.
*
* @param rows Number of rows in the gradient matrix.
* @param cols Number of columns in the gradient matrix.
*/
void Initialize(const size_t rows, const size_t cols)
{
m = arma::zeros<arma::mat>(rows, cols);
v = arma::zeros<arma::mat>(rows, cols);
g = arma::zeros<arma::mat>(rows, cols);
}
/**
* Update step for OptimisticAdam.
*
* @param iterate Parameters that minimize the function.
* @param stepSize Step size to be used for the given iteration.
* @param gradient The gradient matrix.
*/
void Update(arma::mat& iterate,
const double stepSize,
const arma::mat& gradient)
{
// Increment the iteration counter variable.
++iteration;
// And update the iterate.
m *= beta1;
m += (1 - beta1) * gradient;
v *= beta2;
v += (1 - beta2) * arma::square(gradient);
arma::mat mCorrected = m / (1.0 - std::pow(beta1, iteration));
arma::mat vCorrected = v / (1.0 - std::pow(beta2, iteration));
arma::mat update = mCorrected / (arma::sqrt(vCorrected) + epsilon);
iterate -= (2 * stepSize * update - stepSize * g);
g = std::move(update);
}
//! Get the value used to initialize the squared gradient parameter.
double Epsilon() const { return epsilon; }
//! Modify the value used to initialize the squared gradient parameter.
double& Epsilon() { return epsilon; }
//! Get the smoothing parameter.
double Beta1() const { return beta1; }
//! Modify the smoothing parameter.
double& Beta1() { return beta1; }
//! Get the second moment coefficient.
double Beta2() const { return beta2; }
//! Modify the second moment coefficient.
double& Beta2() { return beta2; }
private:
// The epsilon value used to initialize the squared gradient parameter.
double epsilon;
// The smoothing parameter.
double beta1;
// The second moment coefficient.
double beta2;
// The exponential moving average of gradient values.
arma::mat m;
// The exponential moving average of squared gradient values.
arma::mat v;
// The previous update.
arma::mat g;
// The number of iterations.
double iteration;
};
} // namespace optimization
} // namespace mlpack
#endif
src/mlpack/core/optimizers/sdp/lrsdp_function_impl.hpp
+2 -1
View File
@@ -51,7 +51,8 @@ LRSDPFunction<SDPType>::LRSDPFunction(const size_t numSparseConstraints,
}
template <typename SDPType>
double LRSDPFunction<SDPType>::Evaluate(const arma::mat& coordinates) const
double LRSDPFunction<SDPType>::Evaluate(const arma::mat& /* coordinates */)
const
{
// Note: We don't require to update the R*R^T matrix here as the current
// function is only used by AugLagrangian, which do not update the coordinates
src/mlpack/core/optimizers/sgd/CMakeLists.txt
+1
View File
@@ -2,6 +2,7 @@ set(SOURCES
decay_policies/no_decay.hpp
update_policies/gradient_clipping.hpp
update_policies/momentum_update.hpp
update_policies/nesterov_momentum_update.hpp
update_policies/vanilla_update.hpp
sgd.hpp
sgd_impl.hpp
src/mlpack/core/optimizers/sgd/sgd.hpp
+4
View File
@@ -3,6 +3,7 @@
* @author Ryan Curtin
* @author Arun Reddy
* @author Abhinav Moudgil
* @author Sourabh Varshney
*
* Stochastic Gradient Descent (SGD).
*
@@ -17,6 +18,7 @@
#include <mlpack/prereqs.hpp>
#include "update_policies/vanilla_update.hpp"
#include "update_policies/momentum_update.hpp"
#include "update_policies/nesterov_momentum_update.hpp"
#include "decay_policies/no_decay.hpp"
namespace mlpack {
@@ -202,6 +204,8 @@ using StandardSGD = SGD<VanillaUpdate>;
using MomentumSGD = SGD<MomentumUpdate>;
using NesterovMomentumSGD = SGD<NesterovMomentumUpdate>;
} // namespace optimization
} // namespace mlpack
src/mlpack/core/optimizers/sgd/update_policies/nesterov_momentum_update.hpp
+100
View File
@@ -0,0 +1,100 @@
/**
* @file nesterov_momentum_update.hpp
* @author Sourabh Varshney
*
* Nesterov Momentum Update for Stochastic Gradient Descent.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_CORE_OPTIMIZERS_SGD_NESTEROV_MOMENTUM_UPDATE_HPP
#define MLPACK_CORE_OPTIMIZERS_SGD_NESTEROV_MOMENTUM_UPDATE_HPP
#include <mlpack/prereqs.hpp>
namespace mlpack {
namespace optimization {
/**
* Nesterov Momentum update policy for Stochastic Gradient Descent (SGD).
*
* Learning with SGD can be slow. Applying Standard momentum can accelerate
* the rate of convergence. Nesterov Momentum application can accelerate the
* rate of convergence to O(1/k^2).
*
* @code
* @techreport{Nesterov1983,
* title = {A Method Of Solving A Convex Programming Problem With
* Convergence Rate O(1/K^2)},
* author = {Yuri Nesterov},
* institution = {Soviet Math. Dokl.},
* volume = {27},
* year = {1983},
* }
* @endcode
*/
class NesterovMomentumUpdate
{
public:
/**
* Construct the Nesterov Momentum update policy with the given parameters.
*
*/
NesterovMomentumUpdate(const double momentum = 0.5) :
momentum(momentum)
{
// Nothing to do.
}
/**
* The Initialize method is called by SGD Optimizer method before the start of
* the iteration update process. In the momentum update policy the velocity
* matrix is initialized to the zeros matrix with the same size as the
* gradient matrix (see mlpack::optimization::SGD::Optimizer )
*
* @param rows Number of rows in the gradient matrix.
* @param cols Number of columns in the gradient matrix.
*/
void Initialize(const size_t rows, const size_t cols)
{
// Initialize an empty velocity matrix.
velocity = arma::zeros<arma::mat>(rows, cols);
}
/**
* Update step for SGD. The momentum term makes the convergence faster on the
* way as momentum term increases for dimensions pointing in the same direction
* and reduces updates for dimensions whose gradients change directions.
*
* @param iterate Parameters that minimize the function.
* @param stepSize Step size to be used for the given iteration.
* @param gradient The gradient matrix.
*/
void Update(arma::mat& iterate,
const double stepSize,
const arma::mat& gradient)
{
velocity = momentum * velocity - stepSize * gradient;
iterate += momentum * velocity - stepSize * gradient;
}
//! Get the value used to initialize the momentum coefficient.
double Momentum() const { return momentum; }
//! Modify the value used to initialize the momentum coefficient.
double& Momentum() { return momentum; }
private:
// The velocity matrix.
arma::mat velocity;
// The Momentum coefficient.
double momentum;
};
} // namespace optimization
} // namespace mlpack
#endif
src/mlpack/methods/ann/convolution_rules/fft_convolution.hpp
+19 -22
View File
@@ -21,8 +21,8 @@ namespace ann /** Artificial Neural Network. */ {
/**
* Computes the two-dimensional convolution through fft. This class allows
* specification of the type of the border type. The convolution can be compute
* with the valid border type of the full border type (default).
* specification of the type of the border type. The convolution can be
* computed with the valid border type of the full border type (default).
*
* FullConvolution: returns the full two-dimensional convolution.
* ValidConvolution: returns only those parts of the convolution that are
@@ -40,12 +40,12 @@ class FFTConvolution
/*
* Perform a convolution through fft (valid mode). This method only supports
* input which is even on the last dimension. In case of an odd input width, a
* user can manually pad the imput or specify the padLastDim parameter which
* user can manually pad the input or specify the padLastDim parameter which
* takes care of the padding. The filter instead can have any size. When using
* the valid mode the filters has to be smaller than the input.
* the valid mode the filter has to be smaller than the input.
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output Output data that contains the results of the convolution.
*/
template<typename eT, typename Border = BorderMode>
@@ -64,23 +64,23 @@ class FFTConvolution
// Pad filter and input to the output shape.
filterPadded.resize(inputPadded.n_rows, inputPadded.n_cols);
output = arma::real(ifft2(arma::fft2(inputPadded) % arma::fft2(
arma::Mat<eT> temp = arma::real(ifft2(arma::fft2(inputPadded) % arma::fft2(
filterPadded)));
// Extract the region of interest. We don't need to handle the padLastDim in
// a special way we just cut it out from the output matrix.
output = output.submat(filter.n_rows - 1, filter.n_cols - 1,
output = temp.submat(filter.n_rows - 1, filter.n_cols - 1,
input.n_rows - 1, input.n_cols - 1);
}
/*
* Perform a convolution through fft (full mode). This method only supports
* input which is even on the last dimension. In case of an odd input width, a
* user can manually pad the imput or specify the padLastDim parameter which
* user can manually pad the input or specify the padLastDim parameter which
* takes care of the padding. The filter instead can have any size.
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output Output data that contains the results of the convolution.
*/
template<typename eT, typename Border = BorderMode>
@@ -110,12 +110,12 @@ class FFTConvolution
filterPadded.resize(outputRows, outputCols);
// Perform FFT and IFFT
output = arma::real(ifft2(arma::fft2(inputPadded) % arma::fft2(
arma::Mat<eT> temp = arma::real(ifft2(arma::fft2(inputPadded) % arma::fft2(
filterPadded)));
// Extract the region of interest. We don't need to handle the padLastDim
// parameter in a special way we just cut it out from the output matrix.
output = output.submat(filter.n_rows - 1, filter.n_cols - 1,
output = temp.submat(filter.n_rows - 1, filter.n_cols - 1,
2 * (filter.n_rows - 1) + input.n_rows - 1,
2 * (filter.n_cols - 1) + input.n_cols - 1);
}
@@ -123,12 +123,12 @@ class FFTConvolution
/*
* Perform a convolution through fft using 3rd order tensors. This method only
* supports input which is even on the last dimension. In case of an odd input
* width, a user can manually pad the imput or specify the padLastDim
* width, a user can manually pad the input or specify the padLastDim
* parameter which takes care of the padding. The filter instead can have any
* size.
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output Output data that contains the results of the convolution.
*/
template<typename eT>
@@ -147,8 +147,7 @@ class FFTConvolution
for (size_t i = 1; i < input.n_slices; i++)
{
FFTConvolution<BorderMode>::Convolution(input.slice(i), filter.slice(i),
convOutput);
output.slice(i) = convOutput;
output.slice(i));
}
}
@@ -156,11 +155,11 @@ class FFTConvolution
* Perform a convolution through fft using dense matrix as input and a 3rd
* order tensors as filter and output. This method only supports input which
* is even on the last dimension. In case of an odd input width, a user can
* manually pad the imput or specify the padLastDim parameter which takes care
* manually pad the input or specify the padLastDim parameter which takes care
* of the padding. The filter instead can have any size.
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output Output data that contains the results of the convolution.
*/
template<typename eT>
@@ -179,8 +178,7 @@ class FFTConvolution
for (size_t i = 1; i < filter.n_slices; i++)
{
FFTConvolution<BorderMode>::Convolution(input, filter.slice(i),
convOutput);
output.slice(i) = convOutput;
output.slice(i));
}
}
@@ -189,7 +187,7 @@ class FFTConvolution
* dense matrix as filter.
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output Output data that contains the results of the convolution.
*/
template<typename eT>
@@ -208,8 +206,7 @@ class FFTConvolution
for (size_t i = 1; i < input.n_slices; i++)
{
FFTConvolution<BorderMode>::Convolution(input.slice(i), filter,
convOutput);
output.slice(i) = convOutput;
output.slice(i));
}
}
}; // class FFTConvolution
src/mlpack/methods/ann/convolution_rules/naive_convolution.hpp
+5 -5
View File
@@ -39,7 +39,7 @@ class NaiveConvolution
* Perform a convolution (valid mode).
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output Output data that contains the results of the convolution.
* @param dW Stride of filter application in the x direction.
* @param dH Stride of filter application in the y direction.
@@ -79,7 +79,7 @@ class NaiveConvolution
* Perform a convolution (full mode).
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output Output data that contains the results of the convolution.
* @param dW Stride of filter application in the x direction.
* @param dH Stride of filter application in the y direction.
@@ -111,7 +111,7 @@ class NaiveConvolution
* Perform a convolution using 3rd order tensors.
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output Output data that contains the results of the convolution.
* @param dW Stride of filter application in the x direction.
* @param dH Stride of filter application in the y direction.
@@ -143,7 +143,7 @@ class NaiveConvolution
* as filter and output.
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output Output data that contains the results of the convolution.
* @param dW Stride of filter application in the x direction.
* @param dH Stride of filter application in the y direction.
@@ -175,7 +175,7 @@ class NaiveConvolution
* dense matrix as filter.
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output Output data that contains the results of the convolution.
* @param dW Stride of filter application in the x direction.
* @param dH Stride of filter application in the y direction.
src/mlpack/methods/ann/convolution_rules/svd_convolution.hpp
+6 -9
View File
@@ -3,7 +3,7 @@
* @author Marcus Edel
*
* Implementation of the convolution using the singular value decomposition to
* speeded up the computation.
* speed up the computation.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
@@ -24,7 +24,7 @@ namespace ann /** Artificial Neural Network. */ {
/**
* Computes the two-dimensional convolution using singular value decomposition.
* This class allows specification of the type of the border type. The
* convolution can be compute with the valid border type of the full border
* convolution can be computed with the valid border type of the full border
* type (default).
*
* FullConvolution: returns the full two-dimensional convolution.
@@ -87,13 +87,13 @@ class SVDConvolution
NaiveConvolution<BorderMode>::Convolution(subOutput, U.unsafe_col(0),
output);
arma::Mat<eT> temp;
for (size_t r = 1; r < rank; r++)
{
subFilter = V.unsafe_col(r) * s(r);
NaiveConvolution<BorderMode>::Convolution(input, subFilter,
subOutput);
arma::Mat<eT> temp;
subOutput = subOutput.t();
NaiveConvolution<BorderMode>::Convolution(subOutput, U.unsafe_col(r),
temp);
@@ -134,8 +134,7 @@ class SVDConvolution
for (size_t i = 1; i < input.n_slices; i++)
{
SVDConvolution<BorderMode>::Convolution(input.slice(i), filter.slice(i),
convOutput);
output.slice(i) = convOutput;
output.slice(i));
}
}
@@ -164,8 +163,7 @@ class SVDConvolution
for (size_t i = 1; i < filter.n_slices; i++)
{
SVDConvolution<BorderMode>::Convolution(input, filter.slice(i),
convOutput);
output.slice(i) = convOutput;
output.slice(i));
}
}
@@ -194,8 +192,7 @@ class SVDConvolution
for (size_t i = 1; i < input.n_slices; i++)
{
SVDConvolution<BorderMode>::Convolution(input.slice(i), filter,
convOutput);
output.slice(i) = convOutput;
output.slice(i));
}
}
}; // class SVDConvolution
src/mlpack/methods/ann/layer/CMakeLists.txt
+2
View File
@@ -8,6 +8,8 @@ set(SOURCES
base_layer.hpp
bilinear_interpolation.hpp
bilinear_interpolation_impl.hpp
batch_norm.hpp
batch_norm_impl.hpp
concat.hpp
concat_impl.hpp
concat_performance.hpp
src/mlpack/methods/ann/layer/batch_norm.hpp
+202
View File
@@ -0,0 +1,202 @@
/**
* @file batch_norm.hpp
* @author Praveen Ch
* @author Manthan-R-Sheth
*
* Definition of the Batch Normalisation layer class
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_BATCHNORM_HPP
#define MLPACK_METHODS_ANN_LAYER_BATCHNORM_HPP
#include <mlpack/prereqs.hpp>
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
/**
* Declaration of the Batch Normalisation layer class. The layer tranforms
* the input data into zero mean and unit variance and then scales and shifts
* the data by parameters, gamma and beta respectively. These parameters are
* learnt by the network.
*
* If deterministic is false (training), the mean and variance over the batch is
* calculated and the data is normalized. If it is set to true (testing) then
* the mean and variance accrued over the training set is used.
*
* For more information, refer to the following paper,
*
* @code
* @article{DBLP:journals/corr/IoffeS15,
* author = {Sergey Ioffe and
* Christian Szegedy},
* title = {Batch Normalization: Accelerating Deep Network Training by
* Reducing Internal Covariate Shift},
* journal = {CoRR},
* volume = {abs/1502.03167}
* }
*
* @endcode
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class BatchNorm
{
public:
//! Create the BatchNorm object.
BatchNorm();
/**
* Create the BatchNorm layer object for a specified number of input units.
*
* @param size The number of input units.
* @param eps The epsilon added to variance to ensure numerical stability.
*/
BatchNorm(const size_t size, const double eps = 0.001);
/**
* Reset the layer parameters
*/
void Reset();
/**
* Forward pass of the Batch Normalization layer. Transforms the input data
* into zero mean and unit variance, scales the data by a factor gamma and
* shifts it by beta.
*
* @param input Input data for the layer
* @param output Resulting output activations.
*/
template<typename eT>
void Forward(const arma::Mat<eT>&& input, arma::Mat<eT>&& output);
/**
* Backward pass through the layer.
*
* @param input The input activations
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>&& input,
arma::Mat<eT>&& gy,
arma::Mat<eT>&& g);
/**
* Calculate the gradient using the output delta and the input activations.
*
* @param input The input activations
* @param error The calculated error
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>&& input,
arma::Mat<eT>&& error,
arma::Mat<eT>&& gradient);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the input parameter.
InputDataType const& InputParameter() const { return inputParameter; }
//! Modify the input parameter.
InputDataType& InputParameter() { return inputParameter; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
//! Get the value of deterministic parameter.
bool Deterministic() const { return deterministic; }
//! Modify the value of deterministic parameter.
bool& Deterministic() { return deterministic; }
//! Get the mean over the training data.
OutputDataType TrainingMean() { return stats.mean(); }
//! Get the variance over the training data.
OutputDataType TrainingVariance() { return stats.var(1); }
/**
* Serialize the layer
*/
template<typename Archive>
void serialize(Archive& ar, const unsigned int /* version */);
private:
//! Locally-stored number of input units.
size_t size;
//! Locally-stored epsilon value.
double eps;
//! Locally-stored scale parameter.
OutputDataType gamma;
//! Locally-stored shift parameter.
OutputDataType beta;
//! Locally-stored parameters.
OutputDataType weights;
/**
* If true then mean and variance over the training set will be considered
* instead of being calculated over the batch.
*/
bool deterministic;
//! Locally-stored mean object.
OutputDataType mean;
//! Locally-stored variance object.
OutputDataType variance;
//! Locally-stored running statistics object.
arma::running_stat_vec<arma::colvec> stats;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored input parameter object.
InputDataType inputParameter;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
}; // class BatchNorm
} // namespace ann
} // namespace mlpack
// Include the implementation.
#include "batch_norm_impl.hpp"
#endif
src/mlpack/methods/ann/layer/batch_norm_impl.hpp
+130
View File
@@ -0,0 +1,130 @@
/**
* @file batch_norm_impl.hpp
* @author Praveen Ch
* @author Manthan-R-Sheth
*
* Implementation of the Batch Normalization Layer.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_BATCHNORM_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_BATCHNORM_IMPL_HPP
// In case it is not included.
#include "batch_norm.hpp"
namespace mlpack {
namespace ann { /** Artificial Neural Network. */
template<typename InputDataType, typename OutputDataType>
BatchNorm<InputDataType, OutputDataType>::BatchNorm() :
size(10),
eps(1e-7),
deterministic(false)
{
// Nothing to do here.
}
template <typename InputDataType, typename OutputDataType>
BatchNorm<InputDataType, OutputDataType>::BatchNorm(
const size_t size, const double eps) :
size(size),
eps(eps),
deterministic(false)
{
weights.set_size(size + size, 1);
}
template<typename InputDataType, typename OutputDataType>
void BatchNorm<InputDataType, OutputDataType>::Reset()
{
gamma = arma::mat(weights.memptr(), size, 1, false, false);
beta = arma::mat(weights.memptr() + gamma.n_elem, size, 1, false, false);
deterministic = false;
gamma.fill(1.0);
beta.fill(0.0);
stats.reset();
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void BatchNorm<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>&& input, arma::Mat<eT>&& output)
{
output.reshape(input.n_rows, input.n_cols);
// Mean and variance over the entire training set will be used to compute
// the forward pass when deterministic is set to true.
if (deterministic)
{
mean = stats.mean();
variance = stats.var(1);
}
else
{
mean = arma::mean(input, 1);
variance = arma::var(input, 1, 1);
for (size_t i = 0; i < output.n_cols; i++)
{
stats(input.col(i));
}
}
output = input.each_col() - mean;
output.each_col() %= gamma / arma::sqrt(variance + eps);
output.each_col() += beta;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void BatchNorm<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>&& input, arma::Mat<eT>&& gy, arma::Mat<eT>&& g)
{
mean = arma::mean(input, 1);
variance = arma::var(input, 1, 1);
arma::mat m = arma::sum(gy % (input.each_col() - mean), 1);
g = (mean - input.each_col());
g.each_col() %= m;
g.each_col() %= 1.0/(variance + eps);
g += (gy.each_col() - arma::sum(gy, 1));
g += (input.n_cols - 1) * gy;
g.each_col() %= ((1.0 / input.n_cols) * gamma);
g.each_col() %= (1.0 / arma::sqrt(variance + eps));
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void BatchNorm<InputDataType, OutputDataType>::Gradient(
const arma::Mat<eT>&& input,
arma::Mat<eT>&& error,
arma::Mat<eT>&& gradient)
{
gradient.set_size(size + size, 1);
arma::mat normalized = input.each_col() - arma::mean(input, 1)
/ arma::sqrt(arma::var(input, 1, 1) + eps);
gradient.submat(0, 0, gamma.n_elem - 1, 0) = arma::sum(normalized % error, 1);
gradient.submat(gamma.n_elem, 0, gradient.n_elem - 1, 0) =
arma::sum(error, 1);
}
template<typename InputDataType, typename OutputDataType>
template<typename Archive>
void BatchNorm<InputDataType, OutputDataType>::serialize(
Archive& ar, const unsigned int /* version */)
{
ar & BOOST_SERIALIZATION_NVP(gamma);
ar & BOOST_SERIALIZATION_NVP(beta);
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/convolution_impl.hpp
+6 -8
View File
@@ -171,8 +171,8 @@ void Convolution<
>::Backward(
const arma::Mat<eT>&& /* input */, arma::Mat<eT>&& gy, arma::Mat<eT>&& g)
{
arma::cube mappedError = arma::cube(gy.memptr(),
outputWidth, outputHeight, outSize);
arma::cube mappedError(gy.memptr(), outputWidth, outputHeight, outSize,
false, false);
gTemp = arma::zeros<arma::Cube<eT> >(inputTemp.n_rows,
inputTemp.n_cols, inputTemp.n_slices);
@@ -265,12 +265,10 @@ void Convolution<
{
for (size_t i = 0; i < output.n_slices; i++)
{
arma::mat subOutput = output.slice(i);
gradientTemp.slice(s) += subOutput.submat(subOutput.n_rows / 2,
subOutput.n_cols / 2,
subOutput.n_rows / 2 + gradientTemp.n_rows - 1,
subOutput.n_cols / 2 + gradientTemp.n_cols - 1);
gradientTemp.slice(s) += output.slice(i).submat(output.n_rows / 2,
output.n_cols / 2,
output.n_rows / 2 + gradientTemp.n_rows - 1,
output.n_cols / 2 + gradientTemp.n_cols - 1);
}
}
else
src/mlpack/methods/ann/layer/layer.hpp
+1
View File
@@ -13,6 +13,7 @@
#define MLPACK_METHODS_ANN_LAYER_LAYER_HPP
#include "add_merge.hpp"
#include "batch_norm.hpp"
#include "concat_performance.hpp"
#include "convolution.hpp"
#include "dropconnect.hpp"
src/mlpack/methods/ann/layer/layer_types.hpp
+4
View File
@@ -17,6 +17,7 @@
// Layer modules.
#include <mlpack/methods/ann/layer/add.hpp>
#include <mlpack/methods/ann/layer/base_layer.hpp>
#include <mlpack/methods/ann/layer/batch_norm.hpp>
#include <mlpack/methods/ann/layer/bilinear_interpolation.hpp>
#include <mlpack/methods/ann/layer/constant.hpp>
#include <mlpack/methods/ann/layer/cross_entropy_error.hpp>
@@ -45,6 +46,8 @@
namespace mlpack {
namespace ann {
template<typename InputDataType, typename OutputDataType> class BatchNorm;
template<typename InputDataType, typename OutputDataType> class DropConnect;
template<typename InputDataType, typename OutputDataType> class Glimpse;
template<typename InputDataType, typename OutputDataType> class Linear;
@@ -108,6 +111,7 @@ using LayerTypes = boost::variant<
BaseLayer<IdentityFunction, arma::mat, arma::mat>*,
BaseLayer<TanhFunction, arma::mat, arma::mat>*,
BaseLayer<RectifierFunction, arma::mat, arma::mat>*,
BatchNorm<arma::mat, arma::mat>*,
BilinearInterpolation<arma::mat, arma::mat>*,
Concat<arma::mat, arma::mat>*,
ConcatPerformance<NegativeLogLikelihood<arma::mat, arma::mat>,
src/mlpack/methods/decision_tree/all_categorical_split.hpp
+1
View File
@@ -61,6 +61,7 @@ class AllCategoricalSplit
const size_t numClasses,
const WeightVecType& weights,
const size_t minimumLeafSize,
const double minimumGainSplit,
arma::Col<typename VecType::elem_type>& classProbabilities,
AuxiliarySplitInfo<typename VecType::elem_type>& aux);
src/mlpack/methods/decision_tree/all_categorical_split_impl.hpp
+2 -1
View File
@@ -25,6 +25,7 @@ double AllCategoricalSplit<FitnessFunction>::SplitIfBetter(
const size_t numClasses,
const WeightVecType& weights,
const size_t minimumLeafSize,
const double minimumGainSplit,
arma::Col<typename VecType::elem_type>& classProbabilities,
AuxiliarySplitInfo<typename VecType::elem_type>& /* aux */)
{
@@ -96,7 +97,7 @@ double AllCategoricalSplit<FitnessFunction>::SplitIfBetter(
overallGain += childPct * childGain;
}
if (overallGain > bestGain + epsilon)
if (overallGain > bestGain + minimumGainSplit + epsilon)
{
// This is better, so set up the class probabilities vector and return.
classProbabilities.set_size(1);
src/mlpack/methods/decision_tree/best_binary_numeric_split.hpp
+1
View File
@@ -58,6 +58,7 @@ class BestBinaryNumericSplit
const size_t numClasses,
const WeightVecType& weights,
const size_t minimumLeafSize,
const double minimumGainSplit,
arma::Col<typename VecType::elem_type>& classProbabilities,
AuxiliarySplitInfo<typename VecType::elem_type>& aux);
src/mlpack/methods/decision_tree/best_binary_numeric_split_impl.hpp
+2 -1
View File
@@ -24,6 +24,7 @@ double BestBinaryNumericSplit<FitnessFunction>::SplitIfBetter(
const size_t numClasses,
const WeightVecType& weights,
const size_t minimumLeafSize,
const double minimumGainSplit,
arma::Col<typename VecType::elem_type>& classProbabilities,
AuxiliarySplitInfo<typename VecType::elem_type>& /* aux */)
{
@@ -104,7 +105,7 @@ double BestBinaryNumericSplit<FitnessFunction>::SplitIfBetter(
data[sortedIndices[index]]) / 2.0;
return gain;
}
else if (gain > bestFoundGain)
else if (gain > bestFoundGain + minimumGainSplit)
{
// We still have a better split.
bestFoundGain = gain;
src/mlpack/methods/decision_tree/decision_tree.hpp
+50 -29
View File
@@ -52,45 +52,49 @@ class DecisionTree :
/**
* Construct the decision tree on the given data and labels, where the data
* can be both numeric and categorical. Setting minimumLeafSize too small may
* cause the tree to overfit, but setting it too large may cause it to
* underfit.
* can be both numeric and categorical. Setting minimumLeafSize and
* minimumGainSplit too small may cause the tree to overfit, but setting them
* too large may cause it to underfit.
*
* @param data Dataset to train on.
* @param datasetInfo Type information for each dimension of the dataset.
* @param labels Labels for each training point.
* @param numClasses Number of classes in the dataset.
* @param minimumLeafSize Minimum number of points in each leaf node.
* @param minimumGainSplit Minimum gain for the node to split.
*/
template<typename MatType, typename LabelsType>
DecisionTree(MatType&& data,
const data::DatasetInfo& datasetInfo,
LabelsType&& labels,
const size_t numClasses,
const size_t minimumLeafSize = 10);
const size_t minimumLeafSize = 10,
const double minimumGainSplit = 1e-7);
/**
* Construct the decision tree on the given data and labels, assuming that the
* data is all of the numeric type. Setting minimumLeafSize too small may
* cause the tree to overfit, but setting it too large may cause it to
* underfit.
* data is all of the numeric type. Setting minimumLeafSize and
* minimumGainSplit too small may cause the tree to overfit, but setting them
* too large may cause it to underfit.
*
* @param data Dataset to train on.
* @param labels Labels for each training point.
* @param numClasses Number of classes in the dataset.
* @param minimumLeafSize Minimum number of points in each leaf node.
* @param minimumGainSplit Minimum gain for the node to split.
*/
template<typename MatType, typename LabelsType>
DecisionTree(MatType&& data,
LabelsType&& labels,
const size_t numClasses,
const size_t minimumLeafSize = 10);
const size_t minimumLeafSize = 10,
const double minimumGainSplit = 1e-7);
/**
* Construct the decision tree on the given data and labels with weights,
* where the data can be both numeric and categorical. Setting
* minimumLeafSize too small may cause the tree to overfit, but setting it too
* large may cause it to underfit.
* where the data can be both numeric and categorical. Setting minimumLeafSize
* and minimumGainSplit too small may cause the tree to overfit, but setting
* them too large may cause it to underfit.
*
* @param data Dataset to train on.
* @param datasetInfo Type information for each dimension of the dataset.
@@ -98,6 +102,7 @@ class DecisionTree :
* @param numClasses Number of classes in the dataset.
* @param weights The weight list of given label.
* @param minimumLeafSize Minimum number of points in each leaf node.
* @param minimumGainSplit Minimum gain for the node to split.
*/
template<typename MatType, typename LabelsType, typename WeightsType>
DecisionTree(MatType&& data,
@@ -106,21 +111,23 @@ class DecisionTree :
const size_t numClasses,
WeightsType&& weights,
const size_t minimumLeafSize = 10,
const double minimumGainSplit = 1e-7,
const std::enable_if_t<arma::is_arma_type<
typename std::remove_reference<WeightsType>::type>::value>*
= 0);
/**
* Construct the decision tree on the given data and labels with weights,
* assuming that the data is all of the numeric type. Setting minimumLeafSize
* too small may cause the tree to overfit, but setting it too large may cause
* it to underfit.
* assuming that the data is all of the numeric type. Setting minimumLeafSize
* and minimumGainSplit too small may cause the tree to overfit, but setting
* them too large may cause it to underfit.
*
* @param data Dataset to train on.
* @param labels Labels for each training point.
* @param numClasses Number of classes in the dataset.
* @param weights The Weight list of given labels.
* @param minimumLeafSize Minimum number of points in each leaf node.
* @param minimumGainSplit Minimum gain for the node to split.
*/
template<typename MatType, typename LabelsType, typename WeightsType>
DecisionTree(MatType&& data,
@@ -128,6 +135,7 @@ class DecisionTree :
const size_t numClasses,
WeightsType&& weights,
const size_t minimumLeafSize = 10,
const double minimumGainSplit = 1e-7,
const std::enable_if_t<arma::is_arma_type<
typename std::remove_reference<WeightsType>::type>::value>*
= 0);
@@ -179,8 +187,9 @@ class DecisionTree :
/**
* Train the decision tree on the given data. This will overwrite the
* existing model. The data may have numeric and categorical types, specified
* by the datasetInfo parameter. Setting minimumLeafSize too small may cause
* the tree to overfit, but setting it too large may cause it to underfit.
* by the datasetInfo parameter. Setting minimumLeafSize and
* minimumGainSplit too small may cause the tree to overfit, but setting them
* too large may cause it to underfit.
*
* @param data Dataset to train on.
* @param datasetInfo Type information for each dimension.
@@ -188,38 +197,42 @@ class DecisionTree :
* @param numClasses Number of classes in the dataset.
* @param weights Weights of all the labels
* @param minimumLeafSize Minimum number of points in each leaf node.
* @param minimumGainSplit Minimum gain for the node to split.
*/
template<typename MatType, typename LabelsType>
void Train(MatType&& data,
const data::DatasetInfo& datasetInfo,
LabelsType&& labels,
const size_t numClasses,
const size_t minimumLeafSize = 10);
const size_t minimumLeafSize = 10,
const double minimumGainSplit = 1e-7);
/**
* Train the decision tree on the given data, assuming that all dimensions are
* numeric. This will overwrite the given model. Setting minimumLeafSize too
* small may cause the tree to overfit, but setting it too large may cause it
* to underfit.
* numeric. This will overwrite the given model. Setting minimumLeafSize and
* minimumGainSplit too small may cause the tree to overfit, but setting them
* too large may cause it to underfit.
*
* @param data Dataset to train on.
* @param labels Labels for each training point.
* @param numClasses Number of classes in the dataset.
* @param weights Weights of all the labels
* @param minimumLeafSize Minimum number of points in each leaf node.
* @param minimumGainSplit Minimum gain for the node to split.
*/
template<typename MatType, typename LabelsType>
void Train(MatType&& data,
LabelsType&& labels,
const size_t numClasses,
const size_t minimumLeafSize = 10);
const size_t minimumLeafSize = 10,
const double minimumGainSplit = 1e-7);
/**
* Train the decision tree on the given weighted data. This will overwrite
* the existing model. The data may have numeric and categorical types,
* specified by the datasetInfo parameter. Setting minimumLeafSize too small
* may cause the tree to overfit, but setting it too large may cause it to
* underfit.
* specified by the datasetInfo parameter. Setting minimumLeafSize and
* minimumGainSplit too small may cause the tree to overfit, but setting them
* too large may cause it to underfit.
*
* @param data Dataset to train on.
* @param datasetInfo Type information for each dimension.
@@ -227,6 +240,7 @@ class DecisionTree :
* @param numClasses Number of classes in the dataset.
* @param weights Weights of all the labels
* @param minimumLeafSize Minimum number of points in each leaf node.
* @param minimumGainSplit Minimum gain for the node to split.
*/
template<typename MatType, typename LabelsType, typename WeightsType>
void Train(MatType&& data,
@@ -235,20 +249,22 @@ class DecisionTree :
const size_t numClasses,
WeightsType&& weights,
const size_t minimumLeafSize = 10,
const double minimumGainSplit = 1e-7,
const std::enable_if_t<arma::is_arma_type<typename
std::remove_reference<WeightsType>::type>::value>* = 0);
/**
* Train the decision tree on the given weighted data, assuming that all
* dimensions are numeric. This will overwrite the given model. Setting
* minimumLeafSize too small may cause the tree to overfit, but setting it too
* large may cause it to underfit.
* dimensions are numeric. This will overwrite the given model. Setting
* minimumLeafSize and minimumGainSplit too small may cause the tree to
* overfit, but setting them too large may cause it to underfit.
*
* @param data Dataset to train on.
* @param labels Labels for each training point.
* @param numClasses Number of classes in the dataset.
* @param weights Weights of all the labels
* @param minimumLeafSize Minimum number of points in each leaf node.
* @param minimumGainSplit Minimum gain for the node to split.
*/
template<typename MatType, typename LabelsType, typename WeightsType>
void Train(MatType&& data,
@@ -256,6 +272,7 @@ class DecisionTree :
const size_t numClasses,
WeightsType&& weights,
const size_t minimumLeafSize = 10,
const double minimumGainSplit = 1e-7,
const std::enable_if_t<arma::is_arma_type<typename
std::remove_reference<WeightsType>::type>::value>* = 0);
@@ -383,6 +400,7 @@ class DecisionTree :
* @param labels Labels for each training point.
* @param numClasses Number of classes in the dataset.
* @param minimumLeafSize Minimum number of points in each leaf node.
* @param minimumGainSplit Minimum gain for the node to split.
*/
template<bool UseWeights, typename MatType>
void Train(MatType& data,
@@ -392,7 +410,8 @@ class DecisionTree :
arma::Row<size_t>& labels,
const size_t numClasses,
arma::rowvec& weights,
const size_t minimumLeafSize = 10);
const size_t minimumLeafSize = 10,
const double minimumGainSplit = 1e-7);
/**
* Corresponding to the public Train() method, this method is designed for
@@ -406,6 +425,7 @@ class DecisionTree :
* @param labels Labels for each training point.
* @param numClasses Number of classes in the dataset.
* @param minimumLeafSize Minimum number of points in each leaf node.
* @param minimumGainSplit Minimum gain for the node to split.
*/
template<bool UseWeights, typename MatType>
void Train(MatType& data,
@@ -414,7 +434,8 @@ class DecisionTree :
arma::Row<size_t>& labels,
const size_t numClasses,
arma::rowvec& weights,
const size_t minimumLeafSize = 10);
const size_t minimumLeafSize = 10,
const double minimumGainSplit = 1e-7);
};
/**
src/mlpack/methods/decision_tree/decision_tree_impl.hpp
+32 -19
View File
@@ -32,7 +32,8 @@ DecisionTree<FitnessFunction,
const data::DatasetInfo& datasetInfo,
LabelsType&& labels,
const size_t numClasses,
const size_t minimumLeafSize)
const size_t minimumLeafSize,
const double minimumGainSplit)
{
using TrueMatType = typename std::decay<MatType>::type;
using TrueLabelsType = typename std::decay<LabelsType>::type;
@@ -44,7 +45,7 @@ DecisionTree<FitnessFunction,
// Pass off work to the Train() method.
arma::rowvec weights; // Fake weights, not used.
Train<false>(tmpData, 0, tmpData.n_cols, datasetInfo, tmpLabels, numClasses,
weights, minimumLeafSize);
weights, minimumLeafSize, minimumGainSplit);
}
//! Construct and train.
@@ -63,7 +64,8 @@ DecisionTree<FitnessFunction,
NoRecursion>::DecisionTree(MatType&& data,
LabelsType&& labels,
const size_t numClasses,
const size_t minimumLeafSize)
const size_t minimumLeafSize,
const double minimumGainSplit)
{
using TrueMatType = typename std::decay<MatType>::type;
using TrueLabelsType = typename std::decay<LabelsType>::type;
@@ -75,7 +77,7 @@ DecisionTree<FitnessFunction,
// Pass off work to the Train() method.
arma::rowvec weights; // Fake weights, not used.
Train<false>(tmpData, 0, tmpData.n_cols, tmpLabels, numClasses, weights,
minimumLeafSize);
minimumLeafSize, minimumGainSplit);
}
//! Construct and train with weights.
@@ -97,6 +99,7 @@ DecisionTree<FitnessFunction,
const size_t numClasses,
WeightsType&& weights,
const size_t minimumLeafSize,
const double minimumGainSplit,
const std::enable_if_t<
arma::is_arma_type<
typename std::remove_reference<
@@ -113,7 +116,7 @@ DecisionTree<FitnessFunction,
// Pass off work to the weighted Train() method.
Train<true>(tmpData, 0, tmpData.n_cols, datasetInfo, tmpLabels, numClasses,
tmpWeights, minimumLeafSize);
tmpWeights, minimumLeafSize, minimumGainSplit);
}
//! Construct and train with weights.
@@ -134,6 +137,7 @@ DecisionTree<FitnessFunction,
const size_t numClasses,
WeightsType&& weights,
const size_t minimumLeafSize,
const double minimumGainSplit,
const std::enable_if_t<
arma::is_arma_type<
typename std::remove_reference<
@@ -150,7 +154,7 @@ DecisionTree<FitnessFunction,
// Pass off work to the weighted Train() method.
Train<true>(tmpData, 0, tmpData.n_cols, tmpLabels, numClasses, tmpWeights,
minimumLeafSize);
minimumLeafSize, minimumGainSplit);
}
//! Construct, don't train.
@@ -345,7 +349,8 @@ void DecisionTree<FitnessFunction,
const data::DatasetInfo& datasetInfo,
LabelsType&& labels,
const size_t numClasses,
const size_t minimumLeafSize)
const size_t minimumLeafSize,
const double minimumGainSplit)
{
// Sanity check on data.
if (data.n_cols != labels.n_elem)
@@ -367,7 +372,7 @@ void DecisionTree<FitnessFunction,
// Pass off work to the Train() method.
arma::rowvec weights; // Fake weights, not used.
Train<false>(tmpData, 0, tmpData.n_cols, datasetInfo, tmpLabels, numClasses,
weights, minimumLeafSize);
weights, minimumLeafSize, minimumGainSplit);
}
//! Train on the given data, assuming all dimensions are numeric.
@@ -386,7 +391,8 @@ void DecisionTree<FitnessFunction,
NoRecursion>::Train(MatType&& data,
LabelsType&& labels,
const size_t numClasses,
const size_t minimumLeafSize)
const size_t minimumLeafSize,
const double minimumGainSplit)
{
// Sanity check on data.
if (data.n_cols != labels.n_elem)
@@ -408,7 +414,7 @@ void DecisionTree<FitnessFunction,
// Pass off work to the Train() method.
arma::rowvec weights; // Fake weights, not used.
Train<false>(tmpData, 0, tmpData.n_cols, tmpLabels, numClasses, weights,
minimumLeafSize);
minimumLeafSize, minimumGainSplit);
}
//! Train on the given weighted data.
@@ -430,6 +436,7 @@ void DecisionTree<FitnessFunction,
const size_t numClasses,
WeightsType&& weights,
const size_t minimumLeafSize,
const double minimumGainSplit,
const std::enable_if_t<arma::is_arma_type<
typename std::remove_reference<
WeightsType>::type>::value>*)
@@ -455,7 +462,7 @@ void DecisionTree<FitnessFunction,
// Pass off work to the Train() method.
Train<true>(tmpData, 0, tmpData.n_cols, datasetInfo, tmpLabels, numClasses,
tmpWeights, minimumLeafSize);
tmpWeights, minimumLeafSize, minimumGainSplit);
}
//! Train on the given weighted data.
@@ -476,6 +483,7 @@ void DecisionTree<FitnessFunction,
const size_t numClasses,
WeightsType&& weights,
const size_t minimumLeafSize,
const double minimumGainSplit,
const std::enable_if_t<arma::is_arma_type<
typename std::remove_reference<
WeightsType>::type>::value>*)
@@ -501,7 +509,7 @@ void DecisionTree<FitnessFunction,
// Pass off work to the Train() method.
Train<true>(tmpData, 0, tmpData.n_cols, tmpLabels, numClasses, tmpWeights,
minimumLeafSize);
minimumLeafSize, minimumGainSplit);
}
//! Train on the given data.
@@ -524,7 +532,8 @@ void DecisionTree<FitnessFunction,
arma::Row<size_t>& labels,
const size_t numClasses,
arma::rowvec& weights,
const size_t minimumLeafSize)
const size_t minimumLeafSize,
const double minimumGainSplit)
{
// Clear children if needed.
for (size_t i = 0; i < children.size(); ++i)
@@ -533,7 +542,7 @@ void DecisionTree<FitnessFunction,
// Look through the list of dimensions and obtain the gain of the best split.
// We'll cache the best numeric and categorical split auxiliary information in
// numericAux and categoricalAux (and clear them later if we make not split),
// numericAux and categoricalAux (and clear them later if we make no split),
// and use classProbabilities as auxiliary information. Later we'll overwrite
// classProbabilities to the empirical class probabilities if we do not split.
double bestGain = FitnessFunction::template Evaluate<UseWeights>(
@@ -555,6 +564,7 @@ void DecisionTree<FitnessFunction,
numClasses,
UseWeights ? weights.subvec(begin, begin + count - 1) : weights,
minimumLeafSize,
minimumGainSplit,
classProbabilities,
*this);
}
@@ -566,6 +576,7 @@ void DecisionTree<FitnessFunction,
numClasses,
UseWeights ? weights.subvec(begin, begin + count - 1) : weights,
minimumLeafSize,
minimumGainSplit,
classProbabilities,
*this);
}
@@ -641,13 +652,13 @@ void DecisionTree<FitnessFunction,
{
child->Train<UseWeights>(data, currentChildBegin,
currentCol - currentChildBegin, datasetInfo, labels, numClasses,
weights, currentCol - currentChildBegin);
weights, currentCol - currentChildBegin, minimumGainSplit);
}
else
{
child->Train<UseWeights>(data, currentChildBegin,
currentCol - currentChildBegin, datasetInfo, labels, numClasses,
weights, minimumLeafSize);
weights, minimumLeafSize, minimumGainSplit);
}
children.push_back(child);
}
@@ -685,7 +696,8 @@ void DecisionTree<FitnessFunction,
arma::Row<size_t>& labels,
const size_t numClasses,
arma::rowvec& weights,
const size_t minimumLeafSize)
const size_t minimumLeafSize,
const double minimumGainSplit)
{
// Clear children if needed.
for (size_t i = 0; i < children.size(); ++i)
@@ -716,6 +728,7 @@ void DecisionTree<FitnessFunction,
weights.cols(begin, begin + count - 1) :
weights,
minimumLeafSize,
minimumGainSplit,
classProbabilities,
*this);
@@ -776,13 +789,13 @@ void DecisionTree<FitnessFunction,
{
child->Train<UseWeights>(data, currentChildBegin,
currentCol - currentChildBegin, labels, numClasses, weights,
currentCol - currentChildBegin);
currentCol - currentChildBegin, minimumGainSplit);
}
else
{
child->Train<UseWeights>(data, currentChildBegin,
currentCol - currentChildBegin, labels, numClasses, weights,
minimumLeafSize);
minimumLeafSize, minimumGainSplit);
}
children.push_back(child);
}
src/mlpack/methods/decision_tree/decision_tree_main.cpp
+15 -5
View File
@@ -39,7 +39,9 @@ PROGRAM_INFO("Decision tree",
"may not be specified when the " + PRINT_PARAM_STRING("training") + " "
"parameter is specified. The " + PRINT_PARAM_STRING("minimum_leaf_size") +
" parameter specifies the minimum number of training points that must fall"
" into each leaf for it to be split. If " +
" into each leaf for it to be split. The " +
PRINT_PARAM_STRING("minimum_gain_split") + " parameter specifies "
"the minimum gain that is needed for the node to split. If " +
PRINT_PARAM_STRING("print_training_error") + " is specified, the training "
"error will be printed."
"\n\n"
@@ -58,8 +60,8 @@ PROGRAM_INFO("Decision tree",
"call"
"\n\n" +
PRINT_CALL("decision_tree", "training", "data", "labels", "labels",
"output_model", "tree", "minimum_leaf_size", 20,
"print_training_error", true) +
"output_model", "tree", "minimum_leaf_size", 20, "minimum_gain_split",
1e-3, "print_training_error", true) +
"\n\n"
"Then, to use that model to classify points in " +
PRINT_DATASET("test_set") + " and print the test error given the "
@@ -82,6 +84,8 @@ PARAM_UMATRIX_IN("test_labels", "Test point labels, if accuracy calculation "
// Training parameters.
PARAM_INT_IN("minimum_leaf_size", "Minimum number of points in a leaf.", "n",
20);
PARAM_DOUBLE_IN("minimum_gain_split", "Minimum gain for node splitting.", "g",
1e-7);
PARAM_FLAG("print_training_error", "Print the training error.", "e");
// Output parameters.
@@ -136,6 +140,10 @@ static void mlpackMain()
RequireParamValue<int>("minimum_leaf_size", [](int x) { return x > 0; }, true,
"leaf size must be positive");
RequireParamValue<double>("minimum_gain_split", [](double x)
{ return (x > 0.0 && x < 1.0); }, true,
"gain split must be a fraction in range [0,1]");
// Load the model or build the tree.
DecisionTreeModel* model;
arma::mat trainingSet;
@@ -164,6 +172,8 @@ static void mlpackMain()
// Now build the tree.
const size_t minLeafSize = (size_t) CLI::GetParam<int>("minimum_leaf_size");
const double minimumGainSplit =
(double) CLI::GetParam<double>("minimum_gain_split");
// Create decision tree with weighted labels.
if (CLI::HasParam("weights"))
@@ -171,12 +181,12 @@ static void mlpackMain()
arma::Row<double> weights =
std::move(CLI::GetParam<arma::Mat<double>>("weights"));
model->tree = DecisionTree<>(trainingSet, model->info, labels,
numClasses, weights, minLeafSize);
numClasses, weights, minLeafSize, minimumGainSplit);
}
else
{
model->tree = DecisionTree<>(trainingSet, model->info, labels,
numClasses, minLeafSize);
numClasses, minLeafSize, minimumGainSplit);
}
// Do we need to print training error?
src/mlpack/prereqs.hpp
+1 -1
View File
@@ -65,7 +65,7 @@ using enable_if_t = typename enable_if<B, T>::type;
#undef BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS
#undef BOOST_MPL_LIMIT_LIST_SIZE
#define BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS
#define BOOST_MPL_LIMIT_LIST_SIZE 40
#define BOOST_MPL_LIMIT_LIST_SIZE 50
// We'll need the necessary boost::serialization features, as well as what we
// use with mlpack. In Boost 1.59 and newer, the BOOST_PFTO code is no longer
src/mlpack/tests/CMakeLists.txt
+1
View File
@@ -75,6 +75,7 @@ add_executable(mlpack_test
momentum_sgd_test.cpp
nbc_test.cpp
nca_test.cpp
nesterov_momentum_sgd_test.cpp
nmf_test.cpp
nystroem_method_test.cpp
octree_test.cpp
src/mlpack/tests/adam_test.cpp
+77
View File
@@ -12,6 +12,7 @@
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#include <mlpack/core.hpp>
#include <mlpack/core/optimizers/adam/adam.hpp>
@@ -507,4 +508,80 @@ BOOST_AUTO_TEST_CASE(NadaMaxLogisticRegressionTest)
BOOST_REQUIRE_CLOSE(testAcc, 100.0, 0.6); // 0.6% error tolerance.
}
/**
* Tests the OptimisticAdam optimizer using a simple test function.
*/
BOOST_AUTO_TEST_CASE(SimpleOptimisticAdamTestFunction)
{
SGDTestFunction f;
OptimisticAdam optimizer(1e-2, 1, 0.9, 0.99, 1e-8);
arma::mat coordinates = f.GetInitialPoint();
optimizer.Optimize(f, coordinates);
BOOST_REQUIRE_SMALL(coordinates[0], 0.1);
BOOST_REQUIRE_SMALL(coordinates[1], 0.1);
BOOST_REQUIRE_SMALL(coordinates[2], 0.1);
}
/**
* Run OptimisticAdam on logistic regression and make sure the results are acceptable.
*/
BOOST_AUTO_TEST_CASE(OptimisticAdamLogisticRegressionTest)
{
// Generate a two-Gaussian dataset.
GaussianDistribution g1(arma::vec("1.0 1.0 1.0"),
arma::eye<arma::mat>(3, 3));
GaussianDistribution g2(arma::vec("9.0 9.0 9.0"),
arma::eye<arma::mat>(3, 3));
arma::mat data(3, 1000);
arma::Row<size_t> responses(1000);
for (size_t i = 0; i < 500; ++i)
{
data.col(i) = g1.Random();
responses[i] = 0;
}
for (size_t i = 500; i < 1000; ++i)
{
data.col(i) = g2.Random();
responses[i] = 1;
}
// Shuffle the dataset.
arma::uvec indices = arma::shuffle(arma::linspace<arma::uvec>(0,
data.n_cols - 1, data.n_cols));
arma::mat shuffledData(3, 1000);
arma::Row<size_t> shuffledResponses(1000);
for (size_t i = 0; i < data.n_cols; ++i)
{
shuffledData.col(i) = data.col(indices[i]);
shuffledResponses[i] = responses[indices[i]];
}
// Create a test set.
arma::mat testData(3, 1000);
arma::Row<size_t> testResponses(1000);
for (size_t i = 0; i < 500; ++i)
{
testData.col(i) = g1.Random();
testResponses[i] = 0;
}
for (size_t i = 500; i < 1000; ++i)
{
testData.col(i) = g2.Random();
testResponses[i] = 1;
}
OptimisticAdam optimisticAdam;
LogisticRegression<> lr(shuffledData, shuffledResponses, optimisticAdam, 0.5);
// Ensure that the error is close to zero.
const double acc = lr.ComputeAccuracy(data, responses);
BOOST_REQUIRE_CLOSE(acc, 100.0, 0.3); // 0.3% error tolerance.
const double testAcc = lr.ComputeAccuracy(testData, testResponses);
BOOST_REQUIRE_CLOSE(testAcc, 100.0, 0.6); // 0.6% error tolerance.
}
BOOST_AUTO_TEST_SUITE_END();
src/mlpack/tests/ann_layer_test.cpp
+124
View File
@@ -1,6 +1,7 @@
/**
* @file ann_layer_test.cpp
* @author Marcus Edel
* @author Praveen Ch
*
* Tests the ann layer modules.
*
@@ -1317,4 +1318,127 @@ BOOST_AUTO_TEST_CASE(SimpleBilinearInterpolationLayerTest)
arma::zeros(input.n_rows), 1e-12);
}
/**
* Tests the BatchNorm Layer, compares the layers parameters with
* the values from another implementation.
* Link to the implementation - http://cthorey.github.io./backpropagation/
*/
BOOST_AUTO_TEST_CASE(BatchNormTest)
{
arma::mat input, output;
input << 5.1 << 3.5 << 1.4 << arma::endr
<< 4.9 << 3.0 << 1.4 << arma::endr
<< 4.7 << 3.2 << 1.3 << arma::endr;
BatchNorm<> model(input.n_rows);
model.Reset();
// Non-Deteministic Forward Pass Test.
model.Deterministic() = false;
model.Forward(std::move(input), std::move(output));
arma::mat result;
result << 1.1658 << 0.1100 << -1.2758 << arma::endr
<< 1.2579 << -0.0699 << -1.1880 << arma::endr
<< 1.1737 << 0.0958 << -1.2695 << arma::endr;
CheckMatrices(output, result, 1e-1);
result.clear();
// Backward Pass Test.
arma::mat gy;
gy << 0.8402 << 0.9116 << 0.2778 << arma::endr
<< 0.3944 << 0.1976 << 0.5540 << arma::endr
<< 0.7831 << 0.3352 << 0.4774 << arma::endr;
model.Backward(std::move(input), std::move(gy), std::move(output));
result << -0.0780 << 0.1376 << -0.0596 << arma::endr
<< 0.0602 << -0.1317 << 0.0715 << arma::endr
<< 0.0835 << -0.1493 << 0.0658 << arma::endr;
CheckMatrices(output, result, 1e-1);
result.clear();
// Gradient Test.
model.Gradient(std::move(input), std::move(gy), std::move(output));
result << 3.4003 << arma::endr
<< 0.8183 << arma::endr
<< 1.8574 << arma::endr
<< 2.0296 << arma::endr
<< 1.1460 << arma::endr
<< 1.5957 << arma::endr;
CheckMatrices(output, result, 1e-1);
result.clear();
// Deterministic Forward Pass test.
output = model.TrainingMean();
result << 3.33333333 << arma::endr
<< 3.1 << arma::endr
<< 3.06666666 << arma::endr;
CheckMatrices(output, result, 1e-1);
result.clear();
output = model.TrainingVariance();
result << 2.2956 << arma::endr
<< 2.0467 << arma::endr
<< 1.9356 << arma::endr;
CheckMatrices(output, result, 1e-1);
result.clear();
model.Deterministic() = true;
model.Forward(std::move(input), std::move(output));
result << 1.1658 << 0.1100 << -1.2757 << arma::endr
<< 1.2579 << -0.0699 << -1.1880 << arma::endr
<< 1.1737 << 0.0958 << -1.2695 << arma::endr;
CheckMatrices(output, result, 1e-1);
}
/**
* BatchNorm layer numerically gradient test.
*/
BOOST_AUTO_TEST_CASE(GradientBatchNormLayerTest)
{
// Add function gradient instantiation.
struct GradientFunction
{
GradientFunction()
{
input = arma::randn(10, 256);
arma::mat target;
target.ones(1, 256);
model = new FFN<NegativeLogLikelihood<>, NguyenWidrowInitialization>(
input, target);
model->Add<IdentityLayer<> >();
model->Add<BatchNorm<> >(10);
model->Add<Linear<> >(10, 2);
model->Add<LogSoftMax<> >();
}
~GradientFunction()
{
delete model;
}
double Gradient(arma::mat& gradient) const
{
arma::mat output;
double error = model->Evaluate(model->Parameters(), 0, 256, false);
model->Gradient(model->Parameters(), 0, gradient, 256);
return error;
}
arma::mat& Parameters() { return model->Parameters(); }
FFN<NegativeLogLikelihood<>, NguyenWidrowInitialization>* model;
arma::mat input, target;
} function;
BOOST_REQUIRE_LE(CheckGradient(function), 1e-3);
}
BOOST_AUTO_TEST_SUITE_END();
src/mlpack/tests/convolution_test.cpp
+10 -10
View File
@@ -29,7 +29,7 @@ BOOST_AUTO_TEST_SUITE(ConvolutionTest);
* Implementation of the convolution function test.
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output The reference output data that contains the results of the
* convolution.
*
@@ -43,7 +43,7 @@ void Convolution2DMethodTest(const arma::mat input,
arma::mat convOutput;
ConvolutionFunction::Convolution(input, filter, convOutput);
// Check the outut dimension.
// Check the output dimension.
bool b = (convOutput.n_rows == output.n_rows) &&
(convOutput.n_cols == output.n_cols);
BOOST_REQUIRE_EQUAL(b, 1);
@@ -59,7 +59,7 @@ void Convolution2DMethodTest(const arma::mat input,
* Implementation of the convolution function test using 3rd order tensors.
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output The reference output data that contains the results of the
* convolution.
*
@@ -91,7 +91,7 @@ void Convolution3DMethodTest(const arma::cube input,
* and a 3rd order tensors as filter and output (batch modus).
*
* @param input Input used to perform the convolution.
* @param filter Filter used to perform the conolution.
* @param filter Filter used to perform the convolution.
* @param output The reference output data that contains the results of the
* convolution.
*
@@ -146,7 +146,7 @@ BOOST_AUTO_TEST_CASE(ValidConvolution2DTest)
output);
// Perform the convolution using singular value decomposition to
// speeded up the computation.
// speed up the computation.
Convolution2DMethodTest<SVDConvolution<ValidConvolution> >(input, filter,
output);
}
@@ -183,7 +183,7 @@ BOOST_AUTO_TEST_CASE(FullConvolution2DTest)
output);
// Perform the convolution using singular value decomposition to
// speeded up the computation.
// speed up the computation.
Convolution2DMethodTest<SVDConvolution<FullConvolution> >(input, filter,
output);
}
@@ -228,7 +228,7 @@ BOOST_AUTO_TEST_CASE(ValidConvolution3DTest)
filterCube, outputCube);
// Perform the convolution using using the singular value decomposition to
// speeded up the computation.
// speed up the computation.
Convolution3DMethodTest<SVDConvolution<ValidConvolution> >(inputCube,
filterCube, outputCube);
}
@@ -277,7 +277,7 @@ BOOST_AUTO_TEST_CASE(FullConvolution3DTest)
filterCube, outputCube);
// Perform the convolution using using the singular value decomposition to
// speeded up the computation.
// speed up the computation.
Convolution3DMethodTest<SVDConvolution<FullConvolution> >(inputCube,
filterCube, outputCube);
}
@@ -319,7 +319,7 @@ BOOST_AUTO_TEST_CASE(ValidConvolutionBatchTest)
filterCube, outputCube);
// Perform the convolution using using the singular value decomposition to
// speeded up the computation.
// speed up the computation.
ConvolutionMethodBatchTest<SVDConvolution<ValidConvolution> >(input,
filterCube, outputCube);
}
@@ -365,7 +365,7 @@ BOOST_AUTO_TEST_CASE(FullConvolutionBatchTest)
filterCube, outputCube);
// Perform the convolution using using the singular value decomposition to
// speeded up the computation.
// speed up the computation.
ConvolutionMethodBatchTest<SVDConvolution<FullConvolution> >(input,
filterCube, outputCube);
}
src/mlpack/tests/decision_tree_test.cpp
+55 -11
View File
@@ -288,10 +288,10 @@ BOOST_AUTO_TEST_CASE(BestBinaryNumericSplitSimpleSplitTest)
// Call the method to do the splitting.
const double bestGain = GiniGain::Evaluate<false>(labels, 2, weights);
const double gain = BestBinaryNumericSplit<GiniGain>::SplitIfBetter<false>(
bestGain, values, labels, 2, weights, 3, classProbabilities, aux);
bestGain, values, labels, 2, weights, 3, 1e-7, classProbabilities, aux);
const double weightedGain =
BestBinaryNumericSplit<GiniGain>::SplitIfBetter<true>(bestGain, values,
labels, 2, weights, 3, classProbabilities, aux);
labels, 2, weights, 3, 1e-7, classProbabilities, aux);
// Make sure that a split was made.
BOOST_REQUIRE_GT(gain, bestGain);
@@ -325,11 +325,11 @@ BOOST_AUTO_TEST_CASE(BestBinaryNumericSplitMinSamplesTest)
// Call the method to do the splitting.
const double bestGain = GiniGain::Evaluate<false>(labels, 2, weights);
const double gain = BestBinaryNumericSplit<GiniGain>::SplitIfBetter<false>(
bestGain, values, labels, 2, weights, 8, classProbabilities, aux);
bestGain, values, labels, 2, weights, 8, 1e-7, classProbabilities, aux);
// This should make no difference because it won't split at all.
const double weightedGain =
BestBinaryNumericSplit<GiniGain>::SplitIfBetter<true>(bestGain, values,
labels, 2, weights, 8, classProbabilities, aux);
labels, 2, weights, 8, 1e-7, classProbabilities, aux);
// Make sure that no split was made.
BOOST_REQUIRE_EQUAL(gain, bestGain);
@@ -360,7 +360,7 @@ BOOST_AUTO_TEST_CASE(BestBinaryNumericSplitNoGainTest)
// Call the method to do the splitting.
const double bestGain = GiniGain::Evaluate<false>(labels, 2, weights);
const double gain = BestBinaryNumericSplit<GiniGain>::SplitIfBetter<false>(
bestGain, values, labels, 2, weights, 10, classProbabilities, aux);
bestGain, values, labels, 2, weights, 10, 1e-7, classProbabilities, aux);
// Make sure there was no split.
BOOST_REQUIRE_EQUAL(gain, bestGain);
@@ -384,10 +384,11 @@ BOOST_AUTO_TEST_CASE(AllCategoricalSplitSimpleSplitTest)
// Call the method to do the splitting.
const double bestGain = GiniGain::Evaluate<false>(labels, 3, weights);
const double gain = AllCategoricalSplit<GiniGain>::SplitIfBetter<false>(
bestGain, values, 4, labels, 3, weights, 3, classProbabilities, aux);
bestGain, values, 4, labels, 3, weights, 3, 1e-7, classProbabilities,
aux);
const double weightedGain =
AllCategoricalSplit<GiniGain>::SplitIfBetter<true>(bestGain, values, 4,
labels, 3, weights, 3, classProbabilities, aux);
labels, 3, weights, 3, 1e-7, classProbabilities, aux);
// Make sure that a split was made.
BOOST_REQUIRE_GT(gain, bestGain);
@@ -419,7 +420,8 @@ BOOST_AUTO_TEST_CASE(AllCategoricalSplitMinSamplesTest)
// Call the method to do the splitting.
const double bestGain = GiniGain::Evaluate<false>(labels, 3, weights);
const double gain = AllCategoricalSplit<GiniGain>::SplitIfBetter<false>(
bestGain, values, 4, labels, 3, weights, 4, classProbabilities, aux);
bestGain, values, 4, labels, 3, weights, 4, 1e-7, classProbabilities,
aux);
// Make sure it's not split.
BOOST_REQUIRE_EQUAL(gain, bestGain);
@@ -451,10 +453,11 @@ BOOST_AUTO_TEST_CASE(AllCategoricalSplitNoGainTest)
// Call the method to do the splitting.
const double bestGain = GiniGain::Evaluate<false>(labels, 3, weights);
const double gain = AllCategoricalSplit<GiniGain>::SplitIfBetter<false>(
bestGain, values, 10, labels, 3, weights, 10, classProbabilities, aux);
bestGain, values, 10, labels, 3, weights, 10, 1e-7, classProbabilities,
aux);
const double weightedGain =
AllCategoricalSplit<GiniGain>::SplitIfBetter<true>(bestGain, values, 10,
labels, 3, weights, 10, classProbabilities, aux);
labels, 3, weights, 10, 1e-7, classProbabilities, aux);
// Make sure that there was no split.
BOOST_REQUIRE_EQUAL(gain, bestGain);
@@ -539,7 +542,7 @@ BOOST_AUTO_TEST_CASE(PerfectTrainingSet)
}
/**
* onstruct the decision tree with weighted labels
* Construct the decision tree with weighted labels
*/
BOOST_AUTO_TEST_CASE(PerfectTrainingSetWithWeight)
{
@@ -1082,4 +1085,45 @@ BOOST_AUTO_TEST_CASE(ConstDataTest)
constWeights);
}
/**
* Construct the decision tree with splitting only if gain is more than
* threshold.
*/
BOOST_AUTO_TEST_CASE(RegularisedDecisionTree)
{
// Completely random dataset with no structure.
arma::mat dataset(10, 1000, arma::fill::randu);
arma::Row<size_t> labels(1000);
for (size_t i = 0; i < 1000; ++i)
labels[i] = i % 3; // 3 classes.
arma::rowvec weights(labels.n_elem);
weights.ones();
// Minimum leaf size of 1.
DecisionTree<> d(dataset, labels, 3, weights, 1, 1e-7);
// Minimum leaf size of 1 and Minimum gain split of 0.01.
DecisionTree<> dRegularised(dataset, labels, 3, weights, 1, 0.01);
size_t count = 0;
// This part of code is dupliacte with no weighted one.
for (size_t i = 0; i < 1000; ++i)
{
size_t prediction, predictionsregularised;
arma::vec probabilities, probabilitiesRegularised;
d.Classify(dataset.col(i), prediction, probabilities);
dRegularised.Classify(dataset.col(i), predictionsregularised,
probabilitiesRegularised);
if (prediction != predictionsregularised)
count++;
BOOST_REQUIRE_EQUAL(probabilities.n_elem, 3);
BOOST_REQUIRE_EQUAL(probabilitiesRegularised.n_elem, 3);
}
BOOST_REQUIRE_GT(count, 0);
}
BOOST_AUTO_TEST_SUITE_END();
src/mlpack/tests/main_tests/decision_tree_test.cpp
+83
View File
@@ -167,6 +167,89 @@ BOOST_AUTO_TEST_CASE(DecisionTreeMinimumLeafSizeTest)
Log::Fatal.ignoreInput = false;
}
/**
* Make sure minimum gain split is always a fraction in range [0,1].
*/
BOOST_AUTO_TEST_CASE(DecisionMinimumGainSplitTest)
{
arma::mat inputData;
DatasetInfo info;
if (!data::Load("braziltourism.arff", inputData, info))
BOOST_FAIL("Cannot load train dataset braziltourism.arff!");
arma::Row<size_t> labels;
if (!data::Load("braziltourism_labels.txt", labels))
BOOST_FAIL("Cannot load labels for braziltourism_labels.txt");
// Initialize an all-ones weight matrix.
arma::mat weights(1, labels.n_cols, arma::fill::ones);
// Input training data.
SetInputParam("training", std::move(std::make_tuple(info, inputData)));
SetInputParam("labels", std::move(labels));
SetInputParam("weights", std::move(weights));
SetInputParam("minimum_gain_split", 1.5); // Invalid.
Log::Fatal.ignoreInput = true;
BOOST_REQUIRE_THROW(mlpackMain(), std::runtime_error);
Log::Fatal.ignoreInput = false;
}
/**
* Make sure minimum gain split produces regularised tree.
*/
BOOST_AUTO_TEST_CASE(DecisionRegularisationTest)
{
arma::mat inputData;
DatasetInfo info;
if (!data::Load("braziltourism.arff", inputData, info))
BOOST_FAIL("Cannot load train dataset braziltourism.arff!");
arma::Row<size_t> labels;
if (!data::Load("braziltourism_labels.txt", labels))
BOOST_FAIL("Cannot load labels for braziltourism_labels.txt");
// Initialize an all-ones weight matrix.
arma::mat weights(1, labels.n_cols, arma::fill::ones);
// Input training data.
SetInputParam("training", std::make_tuple(info, inputData));
SetInputParam("labels", labels);
SetInputParam("weights", weights);
SetInputParam("minimum_gain_split", 1e-7);
// Input test data.
SetInputParam("test", std::make_tuple(info, inputData));
arma::Row<size_t> pred;
mlpackMain();
pred = std::move(CLI::GetParam<arma::Row<size_t>>("predictions"));
// Input training data.
SetInputParam("training", std::make_tuple(info, inputData));
SetInputParam("labels", std::move(labels));
SetInputParam("weights", std::move(weights));
SetInputParam("minimum_gain_split", 0.01);
// Input test data.
SetInputParam("test", std::move(std::make_tuple(info, inputData)));
arma::Row<size_t> predRegularised;
mlpackMain();
predRegularised = std::move(CLI::GetParam<arma::Row<size_t>>("predictions"));
size_t count = 0;
// This part of code is dupliacte with no weighted one.
for (size_t i = 0; i < 1000; ++i)
{
if (pred[i] != predRegularised[i])
count++;
}
BOOST_REQUIRE_GT(count, 0);
}
/**
* Ensure that saved model can be used again.
*/
src/mlpack/tests/nesterov_momentum_sgd_test.cpp
+72
View File
@@ -0,0 +1,72 @@
/**
* @file nesterov_momentum_sgd_test.cpp
* @author Sourabh Varshney
*
* Test file for NesterovMomentumSGD (Stochastic gradient descent with
* nesterov momentum updates).
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#include <mlpack/core.hpp>
#include <mlpack/core/optimizers/sgd/sgd.hpp>
#include <mlpack/core/optimizers/sgd/update_policies/gradient_clipping.hpp>
#include <mlpack/core/optimizers/sgd/update_policies/nesterov_momentum_update.hpp>
#include <mlpack/core/optimizers/problems/generalized_rosenbrock_function.hpp>
#include <mlpack/core/optimizers/problems/sgd_test_function.hpp>
#include <boost/test/unit_test.hpp>
#include "test_tools.hpp"
using namespace std;
using namespace arma;
using namespace mlpack;
using namespace mlpack::optimization;
using namespace mlpack::optimization::test;
BOOST_AUTO_TEST_SUITE(NesterovMomentumSGDTest);
/*
* Tests the Nesterov Momentum SGD update policy.
*/
BOOST_AUTO_TEST_CASE(NesterovMomentumSGDSpeedUpTestFunction)
{
SGDTestFunction f;
NesterovMomentumUpdate nesterovMomentumUpdate(0.9);
NesterovMomentumSGD s(0.0003, 1, 2500000, 1e-9, true,
nesterovMomentumUpdate);
arma::mat coordinates = f.GetInitialPoint();
double result = s.Optimize(f, coordinates);
BOOST_REQUIRE_CLOSE(result, -1.0, 0.15);
BOOST_REQUIRE_SMALL(coordinates[0], 1e-3);
BOOST_REQUIRE_SMALL(coordinates[1], 1e-7);
BOOST_REQUIRE_SMALL(coordinates[2], 1e-7);
}
/*
* Tests the Nesterov Momentum SGD with Generalized Rosenbrock Test.
*/
BOOST_AUTO_TEST_CASE(GeneralizedRosenbrockTest)
{
// Loop over several variants.
for (size_t i = 10; i < 50; i += 5)
{
// Create the generalized Rosenbrock function.
GeneralizedRosenbrockFunction f(i);
NesterovMomentumUpdate nesterovMomentumUpdate(0.9);
NesterovMomentumSGD s(0.0001, 1, 0, 1e-15, true, nesterovMomentumUpdate);
arma::mat coordinates = f.GetInitialPoint();
double result = s.Optimize(f, coordinates);
BOOST_REQUIRE_SMALL(result, 1e-4);
for (size_t j = 0; j < i; ++j)
BOOST_REQUIRE_CLOSE(coordinates[j], (double) 1.0, 1e-3);
}
}
BOOST_AUTO_TEST_SUITE_END();