diff --git a/doc/user/methods/lmnn.md b/doc/user/methods/lmnn.md new file mode 100644 index 0000000000..322a5f683e --- /dev/null +++ b/doc/user/methods/lmnn.md @@ -0,0 +1,454 @@ +## LMNN + +The `LMNN` class implements large margin nearest neighbor, which can be used +as both a linear dimensionality reduction technique and a distance learning +technique (also called metric learning). LMNN finds a linear transformation of +the dataset that improves `k`-nearest-neighbor classification performance. + +#### Simple usage example: + +```c++ +// Learn a distance metric that improves kNN classification performance. + +// All data and labels are uniform random; 10 dimensional data, 5 classes. +// Replace with a data::Load() call or similar for a real application. +arma::mat dataset(10, 1000, arma::fill::randu); // 1000 points. +arma::Row labels = + arma::randi>(1000, arma::distr_param(0, 4)); + +mlpack::LMNN lmnn(3 /* neighbors to consider */); // Step 1: create object. +arma::mat distance; +lmnn.LearnDistance(dataset, labels, distance); // Step 2: learn distance. + +// `distance` can now be used as a transformation matrix for the data. +arma::mat transformedData = distance * dataset; +// Or, you can create a MahalanobisDistance to evaluate points in the +// transformed dataset space. +arma::mat q = distance.t() * distance; +mlpack::MahalanobisDistance d(std::move(q)); + +std::cout << "Distance between points 0 and 1:" << std::endl; +std::cout << " - Before LMNN: " + << mlpack::EuclideanDistance::Evaluate(dataset.col(0), dataset.col(1)) + << "." << std::endl; +std::cout << " - After LMNN: " + << d.Evaluate(dataset.col(0), dataset.col(1)) << "." << std::endl; +``` +

More examples...

+ +#### Quick links: + + * [Constructors](#constructors): create `LMNN` objects. + * [`LearnDistance()`](#learning-distances): learn distance metrics. + * [Other functionality](#other-functionality) for loading and saving. + * [Examples](#simple-examples) of simple usage and integration with other + techniques. + +#### See also: + + + + * [mlpack distance metrics](../core.md#distances) + * [`NCA`](nca.md) + * [Metric learning on Wikipedia](https://en.wikipedia.org/wiki/Similarity_learning#Metric_learning) + * [Large margin nearest neighbor on Wikipedia](https://en.wikipedia.org/wiki/Large_margin_nearest_neighbor) + * [Distance metric learning for Large Margin Nearest Neighbor Classification (pdf)](https://proceedings.neurips.cc/paper_files/paper/2005/file/a7f592cef8b130a6967a90617db5681b-Paper.pdf) + +### Constructors + + * `lmnn = LMNN(k, regularization=0.5, updateInterval=1)` + - Create an `LMNN` object considering the specified number `k` of neighbors. + - Optionally, specify the regularization to be applied to the LMNN cost + function (a `double`), and the number of iterations between recomputation + of neighbors (`updateInterval`, a `size_t`). + +--- + + * `lmnn = LMNN(k, regularization=0.5, updateInterval=1)` + * `lmnn = LMNN(k, regularization, updateInterval, distance)` + - Create an `LMNN` object using a custom + [`DistanceType`](../core.md#distances). + - `k` specifies the number of neighbors to consider. + - `regularization` specifies the regularization penalty to be applied to the + LMNN cost function (a `double`). + - `updateInterval` specifies the number of iterations between recomputation + of neighbors (a `size_t`). + - An instantiated `DistanceType` can optionally be passed with the `distance` + parameter. + - Using a custom `DistanceType` means that `LearnDistance()` will learn a + linear transformation for the data *in the metric space of the custom + `DistanceType`*. + * This means any learned distance may not necessarily improve + classification performance with the + [Euclidean distance](../core.md#lmetric). + * Instead, classification performance will be improved when the learned + distance is used with the given `DistanceType` only. + - Any mlpack `DistanceType` can be used as a drop-in replacement, or a + [custom `DistanceType`](../../developer/distances.md). + * A list of mlpack's provided distance metrics can be found + [here](../core.md#distances). + - ***Note: be sure that you understand the implications of a custom + `DistanceType` before using this version.*** + +--- + +***Notes***: + + - A larger `k` will cause `LearnDistance()` to take longer to compute, but will + give more accurate results. It is generally suggested to keep `k` in roughly + the `3` to `5` range, depending on the dataset. Using `k = 1` can provide + fast convergence, but the learned distance metric may be of lower quality. + + - `regularization` controls the balance between encouraging small distances for + points of the same class and penalizing small distances for points of + different classes. When `regularization` is increased, small distances for + points of different classes are further penalized. + + - Setting `updateInterval` greater than `1` will allow the LMNN algorithm to + take multiple steps without the expensive recomputation of neighbors, but + this means that subsequent optimization steps may not be using the true + nearest neighbors. + * If using an SGD-like algorithm (i.e. an optimizer for a + [differentiable separable function](https://www.ensmallen.org/docs.html#differentiable-separable-functions)), + this can often be set to a relatively high value (100 is not unreasonable). + * If using an optimizer like L-BFGS (i.e. a full-batch optimizer for + [differentiable functions](https://www.ensmallen.org/docs.html#differentiable-functions)), + this should be kept relatively low (going above 10 is not advised). + * It is worth cross-validating different values of the parameter to see what + works for your dataset. + +--- + +### Learning Distances + +Once an `LMNN` object has been created, the `LearnDistance()` method can be used +to learn a distance. + + * `lmnn.LearnDistance(data, labels, distance, [callbacks...])` + * `lmnn.LearnDistance(data, labels, distance, optimizer, [callbacks...])` + - Learn a distance metric on the given `data` and `labels`, filling + `distance` with a transformation matrix that can be used to map the data + into the space of the learned distance. + - Optionally, pass an instantiated + [ensmallen optimizer](https://www.ensmallen.org) and/or + [ensmallen callbacks](https://www.ensmallen.org/docs.html#callback-documentation) + to be used for the learning process. + - If no optimizer is passed, + [`ens::AMSGrad`](https://www.ensmallen.org/docs.html#amsgrad) is used. + - If `distance` already has size `r` x `data.n_rows` for some `r` less than + or equal to `data.n_rows`, it will be used as the starting point for + optimization. Otherwise, the identity matrix with size `data.n_rows` x + `data.n_rows` will be used. + - When optimization is complete, `distance` will have size `r` x + `data.n_rows`, where `r` is less than or equal to `data.n_rows`. + * *Note*: If `r < data.n_rows`, then LMNN has learned a distance metric + that also reduces the dimensionality of the data. See the + [last example](#simple-examples). + +To use `distance`, either: + + * Compute a new transformed dataset as `distance * data`, or + * Use an instantiated [`MahalanobisDistance`](../core.md#mahalanobisdistance) + with `distance.t() * distance` as the `Q` matrix. + +See the [examples section](#simple-examples) for more details. + +#### `LearnDistance()` Parameters: + +| **name** | **type** | **description** | +|----------|----------|-----------------| +| `data` | [`arma::mat`](../matrices.md) | [Column-major](../matrices.md#representing-data-in-mlpack) training matrix. | +| `labels` | [`arma::Row`](../matrices.md) | Training labels, [between `0` and `numClasses - 1`](../load_save.md#normalizing-labels) (inclusive). Should have length `data.n_cols`. | +| `distance` | [`arma::mat`](../matrices.md) | Output matrix to store transformation matrix representing learned distance. | +| `optimizer` | [any ensmallen optimizer](https://www.ensmallen.org) | Instantiated ensmallen optimizer for [differentiable functions](https://www.ensmallen.org/docs.html#differentiable-functions) or [differentiable separable functions](https://www.ensmallen.org/docs.html#differentiable-separable-functions). | `ens::AMSGrad()` | +| `callbacks...` | [any set of ensmallen callbacks](https://www.ensmallen.org/docs.html#callback-documentation) | Optional callbacks for the ensmallen optimizer, such as e.g. `ens::ProgressBar()`, `ens::Report()`, or others. | _(N/A)_ | + +***Note***: any matrix type can be used for `data` and `distance`, so long as +that type implements the Armadillo API. So, e.g., `arma::fmat` can be used. + +### Other Functionality + + * An `LMNN` object can be serialized with + [`data::Save()` and `data::Load()`](../load_save.md#mlpack-objects). + Note that this is only meaningful if a custom `DistanceType` is being used, + and that custom `DistanceType` has state to be saved. + + * `lmnn.K()` returns the number of neighbors used by LMNN, and `lmnn.K() = k` + will set the number of neighbors to use to `k`. + + * `lmnn.Regularization()` returns the current regularization value of the LMNN + object (as a `double`), and `lmnn.Regularization() = r` can be used to set + the regularization value to `r`. + + * `lmnn.UpdateInterval()` returns the current number of iterations between + neighbor recomputation (as a `size_t`), and `lmnn.UpdateInterval() = i` sets + the number of iterations between neighbor recomputation to `i`. + + * `lmnn.Distance()` will return the `DistanceType` being used for learning. + Unless a custom `DistanceType` was specified in the constructor, + this simply returns a [`SquaredEuclideanDistance`](../core.md#lmetric) + object. + +### Simple Examples + +Learn a distance metric to improve classification performance on the iris +dataset, and show improved performance when using +[`NaiveBayesClassifier`](naive_bayes_classifier.md). + +```c++ +// See https://datasets.mlpack.org/satellite.test.csv. +// (We are using the test set here just because it is a little smaller and +// we want this example to run quickly.) +arma::mat dataset; +mlpack::data::Load("satellite.test.csv", dataset, true); +// See https://datasets.mlpack.org/satellite.test.labels.csv. +arma::Row labels; +mlpack::data::Load("satellite.test.labels.csv", labels, true); + +// Create an LMNN object using 5 nearest neighbors and learn a distance. +arma::mat distance; +mlpack::LMNN lmnn(5); +lmnn.LearnDistance(dataset, labels, distance); + +// The distance matrix has size equal to the dimensionality of the data. +std::cout << "Learned distance size: " << distance.n_rows << " x " + << distance.n_cols << "." << std::endl; + +// Learn a NaiveBayesClassifier model on the data and print the performance. +mlpack::NaiveBayesClassifier nbc1(dataset, labels, 2); +arma::Row predictions; +nbc1.Classify(dataset, predictions); +std::cout << "Naive Bayes Classifier without LMNN: " + << arma::accu(labels == predictions) << " of " << labels.n_elem + << " correct." << std::endl; + +// Now transform the data and learn another NaiveBayesClassifier. +arma::mat transformedDataset = distance * dataset; +mlpack::NaiveBayesClassifier nbc2(transformedDataset, labels, 2); +nbc2.Classify(transformedDataset, predictions); +std::cout << "Naive Bayes Classifier with LMNN: " + << arma::accu(labels == predictions) << " of " << labels.n_elem + << " correct." << std::endl; +``` + +--- + +Learn a distance metric on the vehicle dataset, using 32-bit floating point to +represent the data and metric. + +```c++ +// See https://datasets.mlpack.org/vehicle.csv. +arma::fmat dataset; +mlpack::data::Load("vehicle.csv", dataset, true); + +// The labels are contained as the last row of the dataset. +arma::Row labels = + arma::conv_to>::from(dataset.row(dataset.n_rows - 1)); +dataset.shed_row(dataset.n_rows - 1); + +// Create an LMNN object with k=1 and learn distance on float32 data. +// Set updateInterval to a large value (100) because we are using the default +// AMSGrad optimizer (which will take very many small steps). +arma::fmat distance; +mlpack::LMNN lmnn(1, 0.5, 100); + +lmnn.LearnDistance(dataset, labels, distance, ens::ProgressBar()); + +// We want to compute six quantities: +// +// - Average distance to points of the same class before LMNN. +// - Average distance to points of the same class after LMNN, using +// MahalanobisDistance. +// - Average distance to points of the same class after LMNN, using the +// transformed dataset. +// +// - The same three quantities above, but for points of the other class. +// +// LMNN should reduce the average distance to points in the same class, while +// increasing the average distance to points in other classes. +float distSums[6] = { 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f }; +size_t sameCount = 0; +arma::fmat q = distance.t() * distance; +mlpack::MahalanobisDistance md(std::move(q)); +arma::fmat transformedDataset = distance * dataset; +for (size_t i = 1; i < dataset.n_cols; ++i) +{ + const double d1 = mlpack::EuclideanDistance::Evaluate( + dataset.col(0), dataset.col(i)); + const double d2 = md.Evaluate(dataset.col(0), dataset.col(i)); + const double d3 = mlpack::EuclideanDistance::Evaluate( + transformedDataset.col(0), transformedDataset.col(i)); + + // Determine whether the point has the same label as point 0. + if (labels[i] == labels[0]) + { + distSums[0] += d1; + distSums[1] += d2; + distSums[2] += d3; + ++sameCount; + } + else + { + distSums[3] += d1; + distSums[4] += d2; + distSums[5] += d3; + } +} + +// Turn the results into average distances across the class. +distSums[0] /= sameCount; +distSums[1] /= sameCount; +distSums[2] /= sameCount; +distSums[3] /= (dataset.n_cols - sameCount); +distSums[4] /= (dataset.n_cols - sameCount); +distSums[5] /= (dataset.n_cols - sameCount); + +// Print the results. +std::cout << "Average distance between point 0 and other points of the same " + << "class:" << std::endl; +std::cout << " - Before LMNN: " << distSums[0] << "." + << std::endl; +std::cout << " - After LMNN (with MahalanobisDistance): " << distSums[1] << "." + << std::endl; +std::cout << " - After LMNN (with transformed dataset): " << distSums[2] << "." + << std::endl; +std::cout << std::endl; + +std::cout << "Average distance between point 0 and points of other classes: " + << std::endl; +std::cout << " - Before LMNN: " << distSums[3] << "." + << std::endl; +std::cout << " - After LMNN (with MahalanobisDistance): " << distSums[4] << "." + << std::endl; +std::cout << " - After LMNN (with transformed dataset): " << distSums[5] << "." + << std::endl; +std::cout << std::endl; + +std::cout << "Ratio of other-class to same-class distances:" << std::endl; +std::cout << "(We expect this to go up.)" << std::endl; +std::cout << " - Before LMNN: " << (distSums[3] / distSums[0]) << "." + << std::endl; +std::cout << " - After LMNN: " << (distSums[5] / distSums[2]) << "." + << std::endl; +``` + +--- + +Learn a distance metric on the iris dataset, using the L-BFGS optimizer with +callbacks. + +```c++ +// See https://datasets.mlpack.org/iris.csv. +arma::mat dataset; +mlpack::data::Load("iris.csv", dataset, true); +// See https://datasets.mlpack.org/iris.labels.csv. +arma::Row labels; +mlpack::data::Load("iris.labels.csv", labels, true); + +// Learn a distance with ensmallen's L-BFGS optimizer. +ens::L_BFGS lbfgs; +lbfgs.NumBasis() = 5; +lbfgs.MaxIterations() = 1000; + +// Use 5 neighbors for LMNN, and leave updateInterval at the default of 1, +// because we are using L-BFGS (a full-back optimizer). +mlpack::LMNN lmnn(5); + +// Use a callback that prints a final optimization report. +arma::mat distance; +lmnn.LearnDistance(dataset, labels, distance, lbfgs, ens::Report()); +``` + +--- + +Learn a distance metric on the vehicle dataset, but instead of using the +Euclidean distance as the underlying metric, use the Manhattan distance. This +means that LMNN is optimizing k-NN performance under the Manhattan distance, not +under the Euclidean distance. + +```c++ +// See https://datasets.mlpack.org/vehicle.csv. +arma::mat dataset; +mlpack::data::Load("vehicle.csv", dataset, true); + +// The labels are contained as the last row of the dataset. +arma::Row labels = + arma::conv_to>::from(dataset.row(dataset.n_rows - 1)); +dataset.shed_row(dataset.n_rows - 1); + +// Create the LMNN object and optimize. Use k=3 and Nesterov momentum SGD, +// printing a progress bar during optimization. Because Nesterov momentum SGD +// is an ensmallen optimizer for differentiable separable functions, we increase +// updateInterval to reduce the number of neighbor recomputations. We also set +// the regularization parameter to 1.0 to increase the penalty for nearby +// neighbors of a different class. +mlpack::LMNN lmnn(3, 1.0, 100); +arma::mat distance; +ens::NesterovMomentumSGD opt(0.000001 /* step size */, + 32 /* batch size */, + 20 * dataset.n_cols /* 20 epochs */); +lmnn.LearnDistance(dataset, labels, distance, opt, ens::ProgressBar()); + +// Now inspect distances between points with the Euclidean distance and with the +// inner product distance. +arma::mat transformedDataset = distance * dataset; + +// Points 0 and 1 have the same label (0). See their original distance---with +// both the Euclidean and Manhattan distances---and their transformed distances. +// We expect these points to get closer together, in the Manhattan distance. +const double d1 = mlpack::ManhattanDistance::Evaluate( + dataset.col(0), dataset.col(1)); +const double d2 = mlpack::ManhattanDistance::Evaluate( + transformedDataset.col(0), transformedDataset.col(1)); + +std::cout << "Distance between points 0 and 1 (same class):" << std::endl; +std::cout << " - Manhattan distance:" << std::endl; +std::cout << " * Before LMNN: " << d1 << std::endl; +std::cout << " * After LMNN: " << d2 << std::endl; +std::cout << std::endl; + +// Point 3 has a different label. We therefore expect this point to get further +// from point 0 with the Manhattan distance, but not necessarily with the +// Euclidean distance. +const double d3 = mlpack::ManhattanDistance::Evaluate( + dataset.col(0), dataset.col(3)); +const double d4 = mlpack::ManhattanDistance::Evaluate( + transformedDataset.col(0), transformedDataset.col(3)); + +std::cout << "Distance between points 0 and 3 (different class):" << std::endl; +std::cout << " - Manhattan distance:" << std::endl; +std::cout << " * Before LMNN: " << d3 << std::endl; +std::cout << " * After LMNN: " << d4 << std::endl; + +// Note that point 3 has been moved further away from point 0 than point 1. +``` + +--- + +Learn a distance metric while also performing dimensionality reduction, reducing +the dimensionality of the satellite dataset by 3 dimensions. + +```c++ +// See https://datasets.mlpack.org/satellite.train.csv. +arma::mat dataset; +mlpack::data::Load("satellite.train.csv", dataset, true); +// See https://datasets.mlpack.org/satellite.labels.csv. +arma::Row labels; +mlpack::data::Load("satellite.train.labels.csv", labels, true); + +// Use a random initialization for the distance transformation, with the +// specified output dimensionality. +arma::mat distance(dataset.n_rows - 3, dataset.n_rows, arma::fill::randu); +mlpack::LMNN lmnn(3); +ens::L_BFGS opt; +opt.MaxIterations() = 10; // You may want more in a real application. +lmnn.LearnDistance(dataset, labels, distance, opt, ens::Report()); + +// Now transform the dataset. +arma::mat transformedData = distance * dataset; + +std::cout << "Original data has size " << dataset.n_rows << " x " + << dataset.n_cols << "." << std::endl; +std::cout << "Transformed data has size " << transformedData.n_rows << " x " + << transformedData.n_cols << "." << std::endl; +``` diff --git a/src/mlpack/methods/lmnn/constraints.hpp b/src/mlpack/methods/lmnn/constraints.hpp index 082b8961ed..7704d239fe 100644 --- a/src/mlpack/methods/lmnn/constraints.hpp +++ b/src/mlpack/methods/lmnn/constraints.hpp @@ -27,12 +27,25 @@ namespace mlpack { * data point) and Triplets() (Generates sets of {dataset, target neighbors, * impostors} tripltets.) */ -template +template, + typename DistanceType = SquaredEuclideanDistance> class Constraints { public: //! Convenience typedef. - typedef NeighborSearch KNN; + typedef NeighborSearch KNN; + + // Convenience typedef for element type of data. + typedef typename MatType::elem_type ElemType; + // Convenience typedef for column vector of data. + typedef typename GetColType::type VecType; + // Convenience typedef for cube of data. + typedef typename GetCubeType::type CubeType; + // Convenience typedef for dense matrix of indices. + typedef typename GetUDenseMatType::type UMatType; + // Convenience typedef for dense vector of indices. + typedef typename GetColType::type UVecType; /** * Constructor for creating a Constraints instance. @@ -41,8 +54,8 @@ class Constraints * @param labels Input dataset labels. * @param k Number of target neighbors, impostors & triplets. */ - Constraints(const arma::mat& dataset, - const arma::Row& labels, + Constraints(const MatType& dataset, + const LabelsType& labels, const size_t k); /** @@ -54,10 +67,10 @@ class Constraints * @param labels Input dataset labels. * @param norms Input dataset norms. */ - void TargetNeighbors(arma::Mat& outputMatrix, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms); + void TargetNeighbors(UMatType& outputMatrix, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms); /** * Calculates k similar labeled nearest neighbors for a batch of dataset and @@ -70,10 +83,10 @@ class Constraints * @param begin Index of the initial point of dataset. * @param batchSize Number of data points to use. */ - void TargetNeighbors(arma::Mat& outputMatrix, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms, + void TargetNeighbors(UMatType& outputMatrix, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms, const size_t begin, const size_t batchSize); @@ -86,10 +99,10 @@ class Constraints * @param labels Input dataset labels. * @param norms Input dataset norms. */ - void Impostors(arma::Mat& outputMatrix, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms); + void Impostors(UMatType& outputMatrix, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms); /** * Calculates k differently labeled nearest neighbors & distances to @@ -101,11 +114,11 @@ class Constraints * @param labels Input dataset labels. * @param norms Input dataset norms. */ - void Impostors(arma::Mat& outputNeighbors, - arma::mat& outputDistance, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms); + void Impostors(UMatType& outputNeighbors, + MatType& outputDistance, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms); /** * Calculates k differently labeled nearest neighbors for a batch of dataset @@ -118,10 +131,10 @@ class Constraints * @param begin Index of the initial point of dataset. * @param batchSize Number of data points to use. */ - void Impostors(arma::Mat& outputMatrix, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms, + void Impostors(UMatType& outputMatrix, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms, const size_t begin, const size_t batchSize); @@ -137,11 +150,11 @@ class Constraints * @param begin Index of the initial point of dataset. * @param batchSize Number of data points to use. */ - void Impostors(arma::Mat& outputNeighbors, - arma::mat& outputDistance, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms, + void Impostors(UMatType& outputNeighbors, + MatType& outputDistance, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms, const size_t begin, const size_t batchSize); @@ -158,12 +171,12 @@ class Constraints * @param points Indices of data points to calculate impostors on. * @param numPoints Number of points to actually calculate impostors on. */ - void Impostors(arma::Mat& outputNeighbors, - arma::mat& outputDistance, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms, - const arma::uvec& points, + void Impostors(UMatType& outputNeighbors, + MatType& outputDistance, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms, + const UVecType& points, const size_t numPoints); /** @@ -175,10 +188,10 @@ class Constraints * @param labels Input dataset labels. * @param norms Input dataset norms. */ - void Triplets(arma::Mat& outputMatrix, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms); + void Triplets(UMatType& outputMatrix, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms); //! Get the number of target neighbors (k). const size_t& K() const { return k; } @@ -195,13 +208,13 @@ class Constraints size_t k; //! Store unique labels. - arma::Row uniqueLabels; + LabelsType uniqueLabels; //! Store indices of data points having similar label. - std::vector indexSame; + std::vector indexSame; //! Store indices of data points having different label. - std::vector indexDiff; + std::vector indexDiff; //! False if nothing has ever been precalculated. bool precalculated; @@ -210,15 +223,15 @@ class Constraints * Precalculate the unique labels, and indices of similar * and different datapoints on the basis of labels. */ - inline void Precalculate(const arma::Row& labels); + inline void Precalculate(const LabelsType& labels); /** * Re-order neighbors on the basis of increasing norm in case * of ties among distances. */ - inline void ReorderResults(const arma::mat& distances, - arma::Mat& neighbors, - const arma::vec& norms); + inline void ReorderResults(const MatType& distances, + UMatType& neighbors, + const VecType& norms); }; } // namespace mlpack diff --git a/src/mlpack/methods/lmnn/constraints_impl.hpp b/src/mlpack/methods/lmnn/constraints_impl.hpp index 6f227fd970..27ed28417a 100644 --- a/src/mlpack/methods/lmnn/constraints_impl.hpp +++ b/src/mlpack/methods/lmnn/constraints_impl.hpp @@ -17,10 +17,10 @@ namespace mlpack { -template -Constraints::Constraints( - const arma::mat& /* dataset */, - const arma::Row& labels, +template +Constraints::Constraints( + const MatType& /* dataset */, + const LabelsType& labels, const size_t k) : k(k), precalculated(false) @@ -36,11 +36,11 @@ Constraints::Constraints( } } -template -inline void Constraints::ReorderResults( - const arma::mat& distances, - arma::Mat& neighbors, - const arma::vec& norms) +template +inline void Constraints::ReorderResults( + const MatType& distances, + UMatType& neighbors, + const VecType& norms) { // Shortcut... if (neighbors.n_rows == 1) @@ -64,24 +64,21 @@ inline void Constraints::ReorderResults( if (start != end) { // We must sort these elements by norm. - arma::Col newNeighbors = - neighbors.col(i).subvec(start, end - 1); - arma::uvec indices = ConvTo::From(newNeighbors); - - arma::uvec order = arma::sort_index(norms.elem(indices)); - neighbors.col(i).subvec(start, end - 1) = - newNeighbors.elem(order); + UVecType indices = neighbors.col(i).subvec(start, end - 1); + UVecType order = arma::sort_index(norms.elem(indices)); + neighbors.col(i).subvec(start, end - 1) = indices.elem(order); } } } } // Calculates k similar labeled nearest neighbors. -template -void Constraints::TargetNeighbors(arma::Mat& outputMatrix, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms) +template +void Constraints::TargetNeighbors( + UMatType& outputMatrix, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms) { // Perform pre-calculation. If neccesary. Precalculate(labels); @@ -89,8 +86,8 @@ void Constraints::TargetNeighbors(arma::Mat& outputMatrix, // KNN instance. KNN knn; - arma::Mat neighbors; - arma::mat distances; + UMatType neighbors; + MatType distances; for (size_t i = 0; i < uniqueLabels.n_cols; ++i) { @@ -114,28 +111,29 @@ void Constraints::TargetNeighbors(arma::Mat& outputMatrix, // Calculates k similar labeled nearest neighbors on a // batch of data points. -template -void Constraints::TargetNeighbors(arma::Mat& outputMatrix, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms, - const size_t begin, - const size_t batchSize) +template +void Constraints::TargetNeighbors( + UMatType& outputMatrix, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms, + const size_t begin, + const size_t batchSize) { // Perform pre-calculation. If neccesary. Precalculate(labels); - arma::mat subDataset = dataset.cols(begin, begin + batchSize - 1); - arma::Row sublabels = labels.cols(begin, begin + batchSize - 1); + MatType subDataset = dataset.cols(begin, begin + batchSize - 1); + LabelsType sublabels = labels.cols(begin, begin + batchSize - 1); // KNN instance. KNN knn; - arma::Mat neighbors; - arma::mat distances; + UMatType neighbors; + MatType distances; // Vectors to store indices. - arma::uvec subIndexSame; + UVecType subIndexSame; for (size_t i = 0; i < uniqueLabels.n_cols; ++i) { @@ -161,11 +159,12 @@ void Constraints::TargetNeighbors(arma::Mat& outputMatrix, } // Calculates k differently labeled nearest neighbors. -template -void Constraints::Impostors(arma::Mat& outputMatrix, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms) +template +void Constraints::Impostors( + UMatType& outputMatrix, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms) { // Perform pre-calculation. If neccesary. Precalculate(labels); @@ -173,8 +172,8 @@ void Constraints::Impostors(arma::Mat& outputMatrix, // KNN instance. KNN knn; - arma::Mat neighbors; - arma::mat distances; + UMatType neighbors; + MatType distances; for (size_t i = 0; i < uniqueLabels.n_cols; ++i) { @@ -198,12 +197,13 @@ void Constraints::Impostors(arma::Mat& outputMatrix, // Calculates k differently labeled nearest neighbors. The function // writes back calculated neighbors & distances to passed matrices. -template -void Constraints::Impostors(arma::Mat& outputNeighbors, - arma::mat& outputDistance, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms) +template +void Constraints::Impostors( + UMatType& outputNeighbors, + MatType& outputDistance, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms) { // Perform pre-calculation. If neccesary. Precalculate(labels); @@ -211,8 +211,8 @@ void Constraints::Impostors(arma::Mat& outputNeighbors, // KNN instance. KNN knn; - arma::Mat neighbors; - arma::mat distances; + UMatType neighbors; + MatType distances; for (size_t i = 0; i < uniqueLabels.n_cols; ++i) { @@ -237,28 +237,29 @@ void Constraints::Impostors(arma::Mat& outputNeighbors, // Calculates k differently labeled nearest neighbors on a // batch of data points. -template -void Constraints::Impostors(arma::Mat& outputMatrix, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms, - const size_t begin, - const size_t batchSize) +template +void Constraints::Impostors( + UMatType& outputMatrix, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms, + const size_t begin, + const size_t batchSize) { // Perform pre-calculation. If neccesary. Precalculate(labels); - arma::mat subDataset = dataset.cols(begin, begin + batchSize - 1); - arma::Row sublabels = labels.cols(begin, begin + batchSize - 1); + MatType subDataset = dataset.cols(begin, begin + batchSize - 1); + LabelsType sublabels = labels.cols(begin, begin + batchSize - 1); // KNN instance. KNN knn; - arma::Mat neighbors; - arma::mat distances; + UMatType neighbors; + MatType distances; // Vectors to store indices. - arma::uvec subIndexSame; + UVecType subIndexSame; for (size_t i = 0; i < uniqueLabels.n_cols; ++i) { @@ -285,29 +286,30 @@ void Constraints::Impostors(arma::Mat& outputMatrix, // Calculates k differently labeled nearest neighbors & distances on a // batch of data points. -template -void Constraints::Impostors(arma::Mat& outputNeighbors, - arma::mat& outputDistance, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms, - const size_t begin, - const size_t batchSize) +template +void Constraints::Impostors( + UMatType& outputNeighbors, + MatType& outputDistance, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms, + const size_t begin, + const size_t batchSize) { // Perform pre-calculation. If neccesary. Precalculate(labels); - arma::mat subDataset = dataset.cols(begin, begin + batchSize - 1); - arma::Row sublabels = labels.cols(begin, begin + batchSize - 1); + MatType subDataset = dataset.cols(begin, begin + batchSize - 1); + LabelsType sublabels = labels.cols(begin, begin + batchSize - 1); // KNN instance. KNN knn; - arma::Mat neighbors; - arma::mat distances; + UMatType neighbors; + MatType distances; // Vectors to store indices. - arma::uvec subIndexSame; + UVecType subIndexSame; for (size_t i = 0; i < uniqueLabels.n_cols; ++i) { @@ -335,14 +337,15 @@ void Constraints::Impostors(arma::Mat& outputNeighbors, // Calculates k differently labeled nearest neighbors & distances over some // data points. -template -void Constraints::Impostors(arma::Mat& outputNeighbors, - arma::mat& outputDistance, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms, - const arma::uvec& points, - const size_t numPoints) +template +void Constraints::Impostors( + UMatType& outputNeighbors, + MatType& outputDistance, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms, + const UVecType& points, + const size_t numPoints) { // Perform pre-calculation. If neccesary. Precalculate(labels); @@ -350,11 +353,11 @@ void Constraints::Impostors(arma::Mat& outputNeighbors, // KNN instance. KNN knn; - arma::Mat neighbors; - arma::mat distances; + UMatType neighbors; + MatType distances; // Vectors to store indices. - arma::uvec subIndexSame; + UVecType subIndexSame; for (size_t i = 0; i < uniqueLabels.n_cols; ++i) { @@ -384,31 +387,35 @@ void Constraints::Impostors(arma::Mat& outputNeighbors, // Generates {data point, target neighbors, impostors} triplets using // TargetNeighbors() and Impostors(). -template -void Constraints::Triplets(arma::Mat& outputMatrix, - const arma::mat& dataset, - const arma::Row& labels, - const arma::vec& norms) +template +void Constraints::Triplets( + UMatType& outputMatrix, + const MatType& dataset, + const LabelsType& labels, + const VecType& norms) { // Perform pre-calculation. If neccesary. Precalculate(labels); size_t N = dataset.n_cols; - arma::Mat impostors(k, dataset.n_cols); + UMatType impostors(k, dataset.n_cols); Impostors(impostors, dataset, labels, norms); - arma::Mat targetNeighbors(k, dataset.n_cols);; + UMatType targetNeighbors(k, dataset.n_cols);; TargetNeighbors(targetNeighbors, dataset, labels, norms); - outputMatrix = arma::Mat(3, k * k * N , arma::fill::zeros); + outputMatrix = UMatType(3, k * k * N , arma::fill::zeros); - for (size_t i = 0, r = 0; i < N; ++i) + #pragma omp parallel for collapse(3) + for (size_t i = 0; i < N; ++i) { for (size_t j = 0; j < k; ++j) { - for (size_t l = 0; l < k; l++, r++) + for (size_t l = 0; l < k; l++) { + const size_t r = i * (k * k) + j * k + l; + // Generate triplets. outputMatrix(0, r) = i; outputMatrix(1, r) = targetNeighbors(j, i); @@ -418,9 +425,9 @@ void Constraints::Triplets(arma::Mat& outputMatrix, } } -template -inline void Constraints::Precalculate( - const arma::Row& labels) +template +inline void Constraints::Precalculate( + const LabelsType& labels) { // Make sure the calculation is necessary. if (precalculated) @@ -431,6 +438,7 @@ inline void Constraints::Precalculate( indexSame.resize(uniqueLabels.n_elem); indexDiff.resize(uniqueLabels.n_elem); + #pragma omp parallel for for (size_t i = 0; i < uniqueLabels.n_elem; ++i) { // Store same and diff indices. diff --git a/src/mlpack/methods/lmnn/lmnn.hpp b/src/mlpack/methods/lmnn/lmnn.hpp index 92646df11a..786cbcf924 100644 --- a/src/mlpack/methods/lmnn/lmnn.hpp +++ b/src/mlpack/methods/lmnn/lmnn.hpp @@ -14,6 +14,7 @@ #include +#include "../nca/first_element_is_arma.hpp" #include "constraints.hpp" #include "lmnn_function.hpp" @@ -49,7 +50,7 @@ namespace mlpack { * @tparam OptimizerType Optimizer to use for developing distance. */ template + typename DeprecatedOptimizerType = ens::AMSGrad> class LMNN { public: @@ -63,11 +64,27 @@ class LMNN * @param k Number of targets to consider. * @param distance Type of distance metric used for computation. */ + [[deprecated("Will be removed in mlpack 5.0.0. Pass the dataset directly to " + "LearnDistance() instead.")]] LMNN(const arma::mat& dataset, const arma::Row& labels, const size_t k, const DistanceType distance = DistanceType()); + /** + * Construct the LMNN object, optionally with an instantiated distance metric. + * + * @param k Number of target neighbors to consider. + * @param regularization Penalty to apply to objective function. + * @param updateInterval Number of iterations between each recomputation of + * true neighbors and impostors. + * @param distance Instantiated distance metric for computation. + */ + LMNN(const size_t k, + const double regularization = 0.5, + const size_t updateInterval = 1, + DistanceType distance = DistanceType()); + /** * Perform Large Margin Nearest Neighbors metric learning. The output @@ -80,25 +97,99 @@ class LMNN * @param callbacks Callback function for ensmallen optimizer `OptimizerType`. * See https://www.ensmallen.org/docs.html#callback-documentation. */ - template + template::value>::type, + typename = typename std::enable_if< + !FirstElementIsArma::value + >::type> + [[deprecated("Will be removed in mlpack 5.0.0. Use the version that takes a " + "dataset as a parameter.")]] void LearnDistance(arma::mat& outputMatrix, CallbackTypes&&... callbacks); + /** + * Perform Large Margin Nearest Neighbors metric learning. The output + * distance matrix is written into the passed reference. If the + * LearnDistance() is called with an outputMatrix with correct dimensions, + * then that matrix will be used as the starting point for optimization. + * + * @param dataset Dataset to learn distance metric on. + * @param labels Labels for dataset. + * @param outputMatrix Covariance matrix of Mahalanobis distance. + * @param callbacks Callback function for ensmallen optimizer `OptimizerType`. + * See https://www.ensmallen.org/docs.html#callback-documentation. + */ + template::type, + LMNNFunction, + MatType + >::value>::type, + typename = typename std::enable_if::value>::type> + void LearnDistance(const MatType& dataset, + const LabelsType& labels, + MatType& outputMatrix, + CallbackTypes&&... callbacks) const; + + /** + * Perform Large Margin Nearest Neighbors metric learning. The output + * distance matrix is written into the passed reference. If the + * LearnDistance() is called with an outputMatrix with correct dimensions, + * then that matrix will be used as the starting point for optimization. + * + * @param dataset Dataset to learn distance metric on. + * @param labels Labels for dataset. + * @param optimizer Instantiated ensmallen optimizer to use for LMNN. + * @param outputMatrix Covariance matrix of Mahalanobis distance. + * @param callbacks Callback function for ensmallen optimizer `OptimizerType`. + * See https://www.ensmallen.org/docs.html#callback-documentation. + */ + template, + MatType + >::value>::type> + void LearnDistance(const MatType& dataset, + const LabelsType& labels, + MatType& outputMatrix, + OptimizerType& optimizer, + CallbackTypes&&... callbacks) const; //! Get the dataset reference. - const arma::mat& Dataset() const { return dataset; } + [[deprecated("Will be removed in mlpack 5.0.0. Use the LearnDistance() " + "version that takes the optimizer as a parameter instead.")]] + const arma::mat& Dataset() const { return *dataset; } //! Get the labels reference. - const arma::Row& Labels() const { return labels; } + [[deprecated("Will be removed in mlpack 5.0.0. Use the LearnDistance() " + "version that takes the optimizer as a parameter instead.")]] + const arma::Row& Labels() const { return *labels; } //! Access the regularization value. const double& Regularization() const { return regularization; } //! Modify the regularization value. double& Regularization() { return regularization; } - //! Access the range value. - const size_t& Range() const { return range; } - //! Modify the range value. - size_t& Range() { return range; } + //! Access the iteration update interval value. + const size_t& UpdateInterval() const { return updateInterval; } + //! Modify the iteration update interval value. + size_t& UpdateInterval() { return updateInterval; } + + [[deprecated("Will be removed in mlpack 5.0.0. Use UpdateInterval() " + "instead.")]] + const size_t& Range() const { return updateInterval; } + [[deprecated("Will be removed in mlpack 5.0.0. Use UpdateInterval() " + "instead.")]] + size_t& Range() { return updateInterval; } //! Access the value of k. const size_t& K() const { return k; } @@ -106,15 +197,23 @@ class LMNN size_t K() { return k; } //! Get the optimizer. - const OptimizerType& Optimizer() const { return optimizer; } - OptimizerType& Optimizer() { return optimizer; } + [[deprecated("Will be removed in mlpack 5.0.0. Use the LearnDistance() " + "version that takes the optimizer as a parameter instead.")]] + const DeprecatedOptimizerType& Optimizer() const { return optimizer; } + //! Modify the optimizer. + [[deprecated("Will be removed in mlpack 5.0.0. Use the LearnDistance() " + "version that takes the optimizer as a parameter instead.")]] + DeprecatedOptimizerType& Optimizer() { return optimizer; } + + // Serialize the LMNN object. + template + void serialize(Archive& ar, const unsigned int /* version */); private: - //! Dataset reference. - const arma::mat& dataset; - - //! Labels reference. - const arma::Row& labels; + //! Dataset pointer (will be removed in mlpack 5.0.0). + const arma::mat* dataset; + //! Labels pointer (will be removed in mlpack 5.0.0). + const arma::Row* labels; //! Number of target points. size_t k; @@ -122,14 +221,14 @@ class LMNN //! Regularization value. double regularization; - //! Range after which impostors need to be recalculated. - size_t range; + //! Number of iterations after which impostors need to be recalculated. + size_t updateInterval; //! Distance to be used. DistanceType distance; - //! The optimizer to use. - OptimizerType optimizer; + //! The optimizer to use (will be removed in mlpack 5.0.0). + DeprecatedOptimizerType optimizer; }; // class LMNN } // namespace mlpack diff --git a/src/mlpack/methods/lmnn/lmnn_function.hpp b/src/mlpack/methods/lmnn/lmnn_function.hpp index f35fcf8cd0..e64b8ae0d0 100644 --- a/src/mlpack/methods/lmnn/lmnn_function.hpp +++ b/src/mlpack/methods/lmnn/lmnn_function.hpp @@ -41,9 +41,22 @@ namespace mlpack { * operate on one point in the dataset. This is useful for optimizers like * stochastic gradient descent (see ens::SGD). */ -template +template, + typename DistanceType = SquaredEuclideanDistance> class LMNNFunction { + // Convenience typedef for element type of data. + typedef typename MatType::elem_type ElemType; + // Convenience typedef for column vector of data. + typedef typename GetColType::type VecType; + // Convenience typedef for cube of data. + typedef typename GetCubeType::type CubeType; + // Convenience typedef for dense matrix of indices. + typedef typename GetUDenseMatType::type UMatType; + // Convenience typedef for dense vector of indices. + typedef typename GetColType::type UVecType; + public: /** * Constructor for LMNNFunction class. @@ -52,14 +65,14 @@ class LMNNFunction * @param labels Input dataset labels. * @param k Number of target neighbors to be used. * @param regularization Regularization value. - * @param range Range after which impostors need to be recalculated. + * @param updateInterval Number of iterations before impostors are recomputed. * @param distance Type of distance metric used for computation. */ - LMNNFunction(const arma::mat& dataset, - const arma::Row& labels, + LMNNFunction(const MatType& dataset, + const LabelsType& labels, size_t k, double regularization, - size_t range, + size_t updateInterval, DistanceType distance = DistanceType()); @@ -69,13 +82,13 @@ class LMNNFunction void Shuffle(); /** - * Evaluate the LMNN function for the given transformation matrix. This is the - * non-separable implementation, where the objective function is not + * Evaluate the LMNN function for the given transformation matrix. This is + * the non-separable implementation, where the objective function is not * decomposed into the sum of several objective functions. * * @param transformation Transformation matrix of Mahalanobis distance. */ - double Evaluate(const arma::mat& transformation); + ElemType Evaluate(const MatType& transformation); /** * Evaluate the LMNN objective function for the given transformation matrix on @@ -89,9 +102,9 @@ class LMNNFunction * @param begin Index of the initial point to use for objective function. * @param batchSize Number of points to use for objective function. */ - double Evaluate(const arma::mat& transformation, - const size_t begin, - const size_t batchSize = 1); + ElemType Evaluate(const MatType& transformation, + const size_t begin, + const size_t batchSize = 1); /** * Evaluate the gradient of the LMNN function for the given transformation @@ -103,7 +116,7 @@ class LMNNFunction * @param gradient Matrix to store the calculated gradient in. */ template - void Gradient(const arma::mat& transformation, GradType& gradient); + void Gradient(const MatType& transformation, GradType& gradient); /** * Evaluate the gradient of the LMNN function for the given transformation @@ -121,7 +134,7 @@ class LMNNFunction * @param batchSize Number of points to use for objective function. */ template - void Gradient(const arma::mat& transformation, + void Gradient(const MatType& transformation, const size_t begin, GradType& gradient, const size_t batchSize = 1); @@ -137,8 +150,8 @@ class LMNNFunction * @param gradient Matrix to store the calculated gradient in. */ template - double EvaluateWithGradient(const arma::mat& transformation, - GradType& gradient); + ElemType EvaluateWithGradient(const MatType& transformation, + GradType& gradient); /** * Evaluate the LMNN objective function together with gradient for the given @@ -156,13 +169,13 @@ class LMNNFunction * @param batchSize Number of points to use for objective function. */ template - double EvaluateWithGradient(const arma::mat& transformation, - const size_t begin, - GradType& gradient, - const size_t batchSize = 1); + ElemType EvaluateWithGradient(const MatType& transformation, + const size_t begin, + GradType& gradient, + const size_t batchSize = 1); //! Return the initial point for the optimization. - const arma::mat& GetInitialPoint() const { return initialPoint; } + const MatType& GetInitialPoint() const { return initialPoint; } /** * Get the number of functions the objective function can be decomposed into. @@ -171,7 +184,7 @@ class LMNNFunction size_t NumFunctions() const { return dataset.n_cols; } //! Return the dataset passed into the constructor. - const arma::mat& Dataset() const { return dataset; } + const MatType& Dataset() const { return dataset; } //! Access the regularization value. const double& Regularization() const { return regularization; } @@ -183,26 +196,26 @@ class LMNNFunction //! Modify the value of k. size_t& K() { return k; } - //! Access the value of range. - const size_t& Range() const { return range; } - //! Modify the value of k. - size_t& Range() { return range; } + //! Access the number of iterations between impostor recomputation. + const size_t& UpdateInterval() const { return updateInterval; } + //! Modify the number of iterations between impostor recomputation.. + size_t& UpdateInterval() { return updateInterval; } private: //! data. This will be an alias until Shuffle() is called. - arma::mat dataset; + MatType dataset; //! labels. This will be an alias until Shuffle() is called. - arma::Row labels; + LabelsType labels; //! Initial parameter point. - arma::mat initialPoint; + MatType initialPoint; //! Store transformed dataset. - arma::mat transformedDataset; + MatType transformedDataset; //! Store target neighbors of data points. - arma::Mat targetNeighbors; + UMatType targetNeighbors; //! Initial impostors. - arma::Mat impostors; + UMatType impostors; //! Cache distance. Used to avoid repetive calculation. - arma::mat distanceMat; + MatType distanceMat; //! Number of target neighbors. size_t k; //! The instantiated distance metric. @@ -211,28 +224,28 @@ class LMNNFunction double regularization; //! Keep iterations count. size_t iteration; - //! Range after which impostors need to be recalculated. - size_t range; + //! Number of iterations before impostors need to be recalculated. + size_t updateInterval; //! Constraints Object. - Constraints constraint; + Constraints constraint; //! Holds pre-calculated cij. - arma::mat pCij; + MatType pCij; //! Holds the norm of each data point. - arma::vec norm; + VecType norm; //! Hold previous eval values for each datapoint. - arma::cube evalOld; + CubeType evalOld; //! Hold previous maximum norm of impostor. - arma::mat maxImpNorm; + MatType maxImpNorm; //! Holds previous transformation matrix. Used for L-BFGS like optimizer. - arma::mat transformationOld; + MatType transformationOld; //! Holds previous transformation matrices. - std::vector oldTransformationMatrices; + std::vector oldTransformationMatrices; //! Holds number of points which are using each transformation matrix. std::vector oldTransformationCounts; //! Holds points to transformation matrix mapping. - arma::vec lastTransformationIndices; + VecType lastTransformationIndices; //! Used for storing points to re-calculate impostors for. - arma::uvec points; + UVecType points; //! Flag for controlling use of bounds over impostors. bool impBounds; /** @@ -242,12 +255,12 @@ class LMNNFunction */ inline void Precalculate(); //! Update cache transformation matrices. - inline void UpdateCache(const arma::mat& transformation, + inline void UpdateCache(const MatType& transformation, const size_t begin, const size_t batchSize); //! Calculate norm of change in transformation. - inline void TransDiff(std::map& transformationDiffs, - const arma::mat& transformation, + inline void TransDiff(std::unordered_map& transDiffs, + const MatType& transformation, const size_t begin, const size_t batchSize); }; diff --git a/src/mlpack/methods/lmnn/lmnn_function_impl.hpp b/src/mlpack/methods/lmnn/lmnn_function_impl.hpp index 01aa77afea..58dc54011c 100644 --- a/src/mlpack/methods/lmnn/lmnn_function_impl.hpp +++ b/src/mlpack/methods/lmnn/lmnn_function_impl.hpp @@ -18,18 +18,19 @@ namespace mlpack { -template -LMNNFunction::LMNNFunction(const arma::mat& datasetIn, - const arma::Row& labelsIn, - size_t k, - double regularization, - size_t range, - DistanceType distance) : +template +LMNNFunction::LMNNFunction( + const MatType& datasetIn, + const LabelsType& labelsIn, + size_t k, + double regularization, + size_t updateInterval, + DistanceType distance) : k(k), distance(distance), regularization(regularization), iteration(0), - range(range), + updateInterval(updateInterval), constraint(datasetIn, labelsIn, k), points(datasetIn.n_cols), impBounds(false) @@ -60,7 +61,7 @@ LMNNFunction::LMNNFunction(const arma::mat& datasetIn, lastTransformationIndices.zeros(); // Reserve the first element of cache. - arma::mat emptyMat; + MatType emptyMat; oldTransformationMatrices.push_back(emptyMat); oldTransformationCounts.push_back(dataset.n_cols); @@ -92,18 +93,18 @@ LMNNFunction::LMNNFunction(const arma::mat& datasetIn, } //! Shuffle the dataset. -template -void LMNNFunction::Shuffle() +template +void LMNNFunction::Shuffle() { - arma::mat newDataset = dataset; - arma::Mat newLabels = labels; - arma::cube newEvalOld = evalOld; - arma::vec newlastTransformationIndices = lastTransformationIndices; - arma::mat newMaxImpNorm = maxImpNorm; - arma::vec newNorm = norm; + MatType newDataset = dataset; + LabelsType newLabels = labels; + CubeType newEvalOld = evalOld; + VecType newlastTransformationIndices = lastTransformationIndices; + MatType newMaxImpNorm = maxImpNorm; + VecType newNorm = norm; // Generate ordering. - arma::uvec ordering = arma::shuffle(arma::linspace(0, + UVecType ordering = arma::shuffle(arma::linspace(0, dataset.n_cols - 1, dataset.n_cols)); ClearAlias(dataset); @@ -126,9 +127,9 @@ void LMNNFunction::Shuffle() } // Update cache transformation matrices. -template -inline void LMNNFunction::UpdateCache( - const arma::mat& transformation, +template +inline void LMNNFunction::UpdateCache( + const MatType& transformation, const size_t begin, const size_t batchSize) { @@ -162,31 +163,13 @@ inline void LMNNFunction::UpdateCache( } oldTransformationCounts[index] += batchSize; - - #ifdef DEBUG - size_t total = 0; - for (size_t i = 1; i < oldTransformationCounts.size(); ++i) - { - std::ostringstream oss; - oss << "transformation counts for matrix " << i - << " invalid (" << oldTransformationCounts[i] << ")!"; - Log::Assert(oldTransformationCounts[i] <= dataset.n_cols, oss.str()); - total += oldTransformationCounts[i]; - } - - std::ostringstream oss; - oss << "total count for transformation matrices invalid (" << total - << ", " << "should be " << dataset.n_cols << "!"; - if (begin + batchSize == dataset.n_cols) - Log::Assert(total == dataset.n_cols, oss.str()); - #endif } // Calculate norm of change in transformation. -template -inline void LMNNFunction::TransDiff( - std::map& transformationDiffs, - const arma::mat& transformation, +template +inline void LMNNFunction::TransDiff( + std::unordered_map& transformationDiffs, + const MatType& transformation, const size_t begin, const size_t batchSize) { @@ -209,22 +192,24 @@ inline void LMNNFunction::TransDiff( } //! Evaluate cost over whole dataset. -template -double LMNNFunction::Evaluate(const arma::mat& transformation) +template +typename MatType::elem_type +LMNNFunction::Evaluate( + const MatType& transformation) { - double cost = 0; + ElemType cost = 0; // Apply distance metric over dataset. transformedDataset = transformation * dataset; - double transformationDiff = 0; + ElemType transformationDiff = 0; if (!transformationOld.is_empty()) { // Calculate norm of change in transformation. transformationDiff = arma::norm(transformation - transformationOld); } - if (!transformationOld.is_empty() && iteration++ % range == 0) + if (!transformationOld.is_empty() && iteration++ % updateInterval == 0) { if (impBounds) { @@ -251,7 +236,7 @@ double LMNNFunction::Evaluate(const arma::mat& transformation) norm); } } - else if (iteration++ % range == 0) + else if (iteration++ % updateInterval == 0) { // Re-calculate impostors on transformed dataset. constraint.Impostors(impostors, distanceMat, transformedDataset, labels, @@ -263,7 +248,7 @@ double LMNNFunction::Evaluate(const arma::mat& transformation) for (size_t j = 0; j < k ; ++j) { // Calculate cost due to distance between target neighbors & data point. - double eval = distance.Evaluate(transformedDataset.col(i), + ElemType eval = distance.Evaluate(transformedDataset.col(i), transformedDataset.col(targetNeighbors(j, i))); cost += (1 - regularization) * eval; } @@ -276,7 +261,7 @@ double LMNNFunction::Evaluate(const arma::mat& transformation) { // Calculate cost due to {data point, target neighbors, impostors} // triplets. - double eval = 0; + ElemType eval = 0; // Bounds for eval. if (!transformationOld.is_empty() && evalOld(l, j, i) < -1) @@ -292,7 +277,7 @@ double LMNNFunction::Evaluate(const arma::mat& transformation) // Calculate exact eval value. if (eval > -1) { - if (iteration - 1 % range == 0) + if (iteration - 1 % updateInterval == 0) { eval = distance.Evaluate(transformedDataset.col(i), transformedDataset.col(targetNeighbors(j, i))) - @@ -338,21 +323,23 @@ double LMNNFunction::Evaluate(const arma::mat& transformation) } //! Calculate cost over batches. -template -double LMNNFunction::Evaluate(const arma::mat& transformation, - const size_t begin, - const size_t batchSize) +template +typename MatType::elem_type +LMNNFunction::Evaluate( + const MatType& transformation, + const size_t begin, + const size_t batchSize) { - double cost = 0; + ElemType cost = 0; // Calculate norm of change in transformation. - std::map transformationDiffs; + std::unordered_map transformationDiffs; TransDiff(transformationDiffs, transformation, begin, batchSize); // Apply distance metric over dataset. transformedDataset = transformation * dataset; - if (impBounds && iteration++ % range == 0) + if (impBounds && iteration++ % updateInterval == 0) { // Track number of data points to use for impostors calculatiom. size_t numPoints = 0; @@ -378,7 +365,7 @@ double LMNNFunction::Evaluate(const arma::mat& transformation, constraint.Impostors(impostors, distanceMat, transformedDataset, labels, norm, points, numPoints); } - else if (iteration++ % range == 0) + else if (iteration++ % updateInterval == 0) { // Re-calculate impostors on transformed dataset. constraint.Impostors(impostors, distanceMat, transformedDataset, labels, @@ -390,7 +377,7 @@ double LMNNFunction::Evaluate(const arma::mat& transformation, for (size_t j = 0; j < k ; ++j) { // Calculate cost due to distance between target neighbors & data point. - double eval = distance.Evaluate(transformedDataset.col(i), + ElemType eval = distance.Evaluate(transformedDataset.col(i), transformedDataset.col(targetNeighbors(j, i))); cost += (1 - regularization) * eval; } @@ -403,7 +390,7 @@ double LMNNFunction::Evaluate(const arma::mat& transformation, { // Calculate cost due to {data point, target neighbors, impostors} // triplets. - double eval = 0; + ElemType eval = 0; // Bounds for eval. if (lastTransformationIndices(i) && evalOld(l, j, i) < -1) @@ -419,7 +406,7 @@ double LMNNFunction::Evaluate(const arma::mat& transformation, // Calculate exact eval value. if (eval > -1) { - if (iteration - 1 % range == 0) + if (iteration - 1 % updateInterval == 0) { eval = distance.Evaluate(transformedDataset.col(i), transformedDataset.col(targetNeighbors(j, i))) - @@ -467,16 +454,16 @@ double LMNNFunction::Evaluate(const arma::mat& transformation, } //! Compute gradient over whole dataset. -template +template template -void LMNNFunction::Gradient(const arma::mat& transformation, - GradType& gradient) +void LMNNFunction::Gradient( + const MatType& transformation, GradType& gradient) { // Apply distance metric over dataset. transformedDataset = transformation * dataset; - double transformationDiff = 0; - if (!transformationOld.is_empty() && iteration++ % range == 0) + ElemType transformationDiff = 0; + if (!transformationOld.is_empty() && iteration++ % updateInterval == 0) { // Calculate norm of change in transformation. transformationDiff = arma::norm(transformation - transformationOld); @@ -506,7 +493,7 @@ void LMNNFunction::Gradient(const arma::mat& transformation, norm); } } - else if (iteration++ % range == 0) + else if (iteration++ % updateInterval == 0) { // Re-calculate impostors on transformed dataset. constraint.Impostors(impostors, distanceMat, transformedDataset, labels, @@ -516,10 +503,10 @@ void LMNNFunction::Gradient(const arma::mat& transformation, gradient.zeros(transformation.n_rows, transformation.n_cols); // Calculate gradient due to target neighbors. - arma::mat cij = pCij; + MatType cij = pCij; // Calculate gradient due to impostors. - arma::mat cil = zeros(dataset.n_rows, dataset.n_rows); + MatType cil = zeros(dataset.n_rows, dataset.n_rows); for (size_t i = 0; i < dataset.n_cols; ++i) { @@ -530,7 +517,7 @@ void LMNNFunction::Gradient(const arma::mat& transformation, { // Calculate cost due to {data point, target neighbors, impostors} // triplets. - double eval = 0; + ElemType eval = 0; // Bounds for eval. if (!transformationOld.is_empty() && evalOld(l, j, i) < -1) @@ -546,7 +533,7 @@ void LMNNFunction::Gradient(const arma::mat& transformation, // Calculate exact eval value. if (eval > -1) { - if (iteration - 1 % range == 0) + if (iteration - 1 % updateInterval == 0) { eval = distance.Evaluate(transformedDataset.col(i), transformedDataset.col(targetNeighbors(j, i))) - @@ -581,7 +568,7 @@ void LMNNFunction::Gradient(const arma::mat& transformation, } // Caculate gradient due to impostors. - arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); + VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); cil += diff * trans(diff); diff = dataset.col(i) - dataset.col(impostors(l, i)); @@ -598,21 +585,22 @@ void LMNNFunction::Gradient(const arma::mat& transformation, } //! Compute gradient over a batch of data points. -template +template template -void LMNNFunction::Gradient(const arma::mat& transformation, - const size_t begin, - GradType& gradient, - const size_t batchSize) +void LMNNFunction::Gradient( + const MatType& transformation, + const size_t begin, + GradType& gradient, + const size_t batchSize) { // Apply distance metric over dataset. transformedDataset = transformation * dataset; // Calculate norm of change in transformation. - std::map transformationDiffs; + std::unordered_map transformationDiffs; TransDiff(transformationDiffs, transformation, begin, batchSize); - if (impBounds && iteration++ % range == 0) + if (impBounds && iteration++ % updateInterval == 0) { // Track number of data points to use for impostors calculatiom. size_t numPoints = 0; @@ -638,7 +626,7 @@ void LMNNFunction::Gradient(const arma::mat& transformation, constraint.Impostors(impostors, distanceMat, transformedDataset, labels, norm, points, numPoints); } - else if (iteration++ % range == 0) + else if (iteration++ % updateInterval == 0) { // Re-calculate impostors on transformed dataset. constraint.Impostors(impostors, distanceMat, transformedDataset, labels, @@ -647,15 +635,15 @@ void LMNNFunction::Gradient(const arma::mat& transformation, gradient.zeros(transformation.n_rows, transformation.n_cols); - arma::mat cij = zeros(dataset.n_rows, dataset.n_rows); - arma::mat cil = zeros(dataset.n_rows, dataset.n_rows); + MatType cij = zeros(dataset.n_rows, dataset.n_rows); + MatType cil = zeros(dataset.n_rows, dataset.n_rows); for (size_t i = begin; i < begin + batchSize; ++i) { for (size_t j = 0; j < k ; ++j) { // Calculate gradient due to target neighbors. - arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); + VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); cij += diff * trans(diff); } @@ -666,7 +654,7 @@ void LMNNFunction::Gradient(const arma::mat& transformation, { // Calculate cost due to {data point, target neighbors, impostors} // triplets. - double eval = 0; + ElemType eval = 0; // Bounds for eval. if (lastTransformationIndices(i) && evalOld(l, j, i) < -1) @@ -682,7 +670,7 @@ void LMNNFunction::Gradient(const arma::mat& transformation, // Calculate exact eval value. if (eval > -1) { - if (iteration - 1 % range == 0) + if (iteration - 1 % updateInterval == 0) { eval = distance.Evaluate(transformedDataset.col(i), transformedDataset.col(targetNeighbors(j, i))) - @@ -719,7 +707,7 @@ void LMNNFunction::Gradient(const arma::mat& transformation, } // Caculate gradient due to impostors. - arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); + VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); cil += diff * trans(diff); diff = dataset.col(i) - dataset.col(impostors(l, i)); @@ -736,25 +724,26 @@ void LMNNFunction::Gradient(const arma::mat& transformation, } //! Compute cost & gradient over whole dataset. -template +template template -double LMNNFunction::EvaluateWithGradient( - const arma::mat& transformation, +typename MatType::elem_type +LMNNFunction::EvaluateWithGradient( + const MatType& transformation, GradType& gradient) { - double cost = 0; + ElemType cost = 0; // Apply distance metric over dataset. transformedDataset = transformation * dataset; - double transformationDiff = 0; + ElemType transformationDiff = 0; if (!transformationOld.is_empty()) { // Calculate norm of change in transformation. transformationDiff = arma::norm(transformation - transformationOld); } - if (!transformationOld.is_empty() && iteration++ % range == 0) + if (!transformationOld.is_empty() && iteration++ % updateInterval == 0) { if (impBounds) { @@ -781,7 +770,7 @@ double LMNNFunction::EvaluateWithGradient( norm); } } - else if (iteration++ % range == 0) + else if (iteration++ % updateInterval == 0) { // Re-calculate impostors on transformed dataset. constraint.Impostors(impostors, distanceMat, transformedDataset, labels, @@ -791,17 +780,17 @@ double LMNNFunction::EvaluateWithGradient( gradient.zeros(transformation.n_rows, transformation.n_cols); // Calculate gradient due to target neighbors. - arma::mat cij = pCij; + MatType cij = pCij; // Calculate gradient due to impostors. - arma::mat cil = zeros(dataset.n_rows, dataset.n_rows); + MatType cil = zeros(dataset.n_rows, dataset.n_rows); for (size_t i = 0; i < dataset.n_cols; ++i) { for (size_t j = 0; j < k ; ++j) { // Calculate cost due to distance between target neighbors & data point. - double eval = distance.Evaluate(transformedDataset.col(i), + ElemType eval = distance.Evaluate(transformedDataset.col(i), transformedDataset.col(targetNeighbors(j, i))); cost += (1 - regularization) * eval; } @@ -813,7 +802,7 @@ double LMNNFunction::EvaluateWithGradient( { // Calculate cost due to {data point, target neighbors, impostors} // triplets. - double eval = 0; + ElemType eval = 0; // Bounds for eval. if (!transformationOld.is_empty() && evalOld(l, j, i) < -1) @@ -829,7 +818,7 @@ double LMNNFunction::EvaluateWithGradient( // Calculate exact eval value. if (eval > -1) { - if (iteration - 1 % range == 0) + if (iteration - 1 % updateInterval == 0) { eval = distance.Evaluate(transformedDataset.col(i), transformedDataset.col(targetNeighbors(j, i))) - @@ -858,7 +847,7 @@ double LMNNFunction::EvaluateWithGradient( cost += regularization * (1 + eval); // Caculate gradient due to impostors. - arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); + VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); cil += diff * trans(diff); diff = dataset.col(i) - dataset.col(impostors(l, i)); @@ -877,24 +866,25 @@ double LMNNFunction::EvaluateWithGradient( } //! Compute cost & gradient over a batch of data points. -template +template template -double LMNNFunction::EvaluateWithGradient( - const arma::mat& transformation, +typename MatType::elem_type +LMNNFunction::EvaluateWithGradient( + const MatType& transformation, const size_t begin, GradType& gradient, const size_t batchSize) { - double cost = 0; + ElemType cost = 0; // Calculate norm of change in transformation. - std::map transformationDiffs; + std::unordered_map transformationDiffs; TransDiff(transformationDiffs, transformation, begin, batchSize); // Apply distance metric over dataset. transformedDataset = transformation * dataset; - if (impBounds && iteration++ % range == 0) + if (impBounds && iteration++ % updateInterval == 0) { // Track number of data points to use for impostors calculatiom. size_t numPoints = 0; @@ -920,7 +910,7 @@ double LMNNFunction::EvaluateWithGradient( constraint.Impostors(impostors, distanceMat, transformedDataset, labels, norm, points, numPoints); } - else if (iteration++ % range == 0) + else if (iteration++ % updateInterval == 0) { // Re-calculate impostors on transformed dataset. constraint.Impostors(impostors, distanceMat, transformedDataset, labels, @@ -929,20 +919,20 @@ double LMNNFunction::EvaluateWithGradient( gradient.zeros(transformation.n_rows, transformation.n_cols); - arma::mat cij = zeros(dataset.n_rows, dataset.n_rows); - arma::mat cil = zeros(dataset.n_rows, dataset.n_rows); + MatType cij = zeros(dataset.n_rows, dataset.n_rows); + MatType cil = zeros(dataset.n_rows, dataset.n_rows); for (size_t i = begin; i < begin + batchSize; ++i) { for (size_t j = 0; j < k ; ++j) { // Calculate cost due to distance between target neighbors & data point. - double eval = distance.Evaluate(transformedDataset.col(i), + ElemType eval = distance.Evaluate(transformedDataset.col(i), transformedDataset.col(targetNeighbors(j, i))); cost += (1 - regularization) * eval; // Calculate gradient due to target neighbors. - arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); + VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); cij += diff * trans(diff); } @@ -953,7 +943,7 @@ double LMNNFunction::EvaluateWithGradient( { // Calculate cost due to {data point, target neighbors, impostors} // triplets. - double eval = 0; + ElemType eval = 0; // Bounds for eval. if (lastTransformationIndices(i) && evalOld(l, j, i) < -1) @@ -969,7 +959,7 @@ double LMNNFunction::EvaluateWithGradient( // Calculate exact eval value. if (eval > -1) { - if (iteration - 1 % range == 0) + if (iteration - 1 % updateInterval == 0) { eval = distance.Evaluate(transformedDataset.col(i), transformedDataset.col(targetNeighbors(j, i))) - @@ -998,7 +988,7 @@ double LMNNFunction::EvaluateWithGradient( cost += regularization * (1 + eval); // Caculate gradient due to impostors. - arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); + VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); cil += diff * trans(diff); diff = dataset.col(i) - dataset.col(impostors(l, i)); @@ -1016,8 +1006,8 @@ double LMNNFunction::EvaluateWithGradient( return cost; } -template -inline void LMNNFunction::Precalculate() +template +inline void LMNNFunction::Precalculate() { pCij.zeros(dataset.n_rows, dataset.n_rows); @@ -1026,7 +1016,7 @@ inline void LMNNFunction::Precalculate() for (size_t j = 0; j < k ; ++j) { // Calculate gradient due to target neighbors. - arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); + VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i)); pCij += diff * trans(diff); } } diff --git a/src/mlpack/methods/lmnn/lmnn_impl.hpp b/src/mlpack/methods/lmnn/lmnn_impl.hpp index 740a3a6c0c..5dfc87bb78 100644 --- a/src/mlpack/methods/lmnn/lmnn_impl.hpp +++ b/src/mlpack/methods/lmnn/lmnn_impl.hpp @@ -21,27 +21,83 @@ namespace mlpack { * Takes in a reference to the dataset. Copies the data, initializes * all of the member variables and constraint object and generate constraints. */ -template -LMNN::LMNN(const arma::mat& dataset, - const arma::Row& labels, - const size_t k, - const DistanceType distance) : - dataset(dataset), - labels(labels), +template +LMNN::LMNN( + const arma::mat& dataset, + const arma::Row& labels, + const size_t k, + const DistanceType distance) : + dataset(&dataset), + labels(&labels), k(k), regularization(0.5), - range(1), + updateInterval(1), distance(distance) { /* nothing to do */ } -template -template -void LMNN::LearnDistance(arma::mat& outputMatrix, +template +LMNN::LMNN( + const size_t k, + const double regularization, + const size_t updateInterval, + const DistanceType distance) : + k(k), + regularization(regularization), + updateInterval(updateInterval), + distance(distance) +{ /* nothing to do */ } + +template +template +void LMNN::LearnDistance( + arma::mat& outputMatrix, CallbackTypes&&... callbacks) +{ + if (!dataset || !labels) + { + throw std::runtime_error("LMNN::LearnDistance(): cannot call without a " + "dataset!"); + } + + LearnDistance(*dataset, *labels, outputMatrix, optimizer, + std::forward(callbacks)...); +} + +template +template +void LMNN::LearnDistance( + const MatType& dataset, + const LabelsType& labels, + MatType& outputMatrix, + CallbackTypes&&... callbacks) const +{ + // This should be replaced with ens::StandardSGD when the deprecated members + // are removed for mlpack 5.0.0. + DeprecatedOptimizerType opt; + LearnDistance(dataset, labels, outputMatrix, opt, + std::forward(callbacks)...); +} + +template +template +void LMNN::LearnDistance( + const MatType& dataset, + const LabelsType& labels, + MatType& outputMatrix, + OptimizerType& opt, + CallbackTypes&&... callbacks) const { // LMNN objective function. - LMNNFunction objFunction(dataset, labels, k, - regularization, range); + LMNNFunction objFunction(dataset, labels, + k, regularization, updateInterval); // See if we were passed an initialized matrix. outputMatrix (L) must be // having r x d dimensionality. @@ -49,15 +105,23 @@ void LMNN::LearnDistance(arma::mat& outputMatrix, (outputMatrix.n_rows > dataset.n_rows) || !(arma::is_finite(outputMatrix))) { - Log::Info << "Initial learning point have invalid dimensionality. " - "Identity matrix will be used as initial learning point for " - "optimization." << std::endl; outputMatrix.eye(dataset.n_rows, dataset.n_rows); } - optimizer.Optimize(objFunction, outputMatrix, callbacks...); + opt.Optimize(objFunction, outputMatrix, callbacks...); } +// Serialize the LMNN object. +template +template +void LMNN::serialize( + Archive& ar, const unsigned int /* version */) +{ + ar(CEREAL_NVP(k)); + ar(CEREAL_NVP(regularization)); + ar(CEREAL_NVP(updateInterval)); + ar(CEREAL_NVP(distance)); +} } // namespace mlpack diff --git a/src/mlpack/methods/lmnn/lmnn_main.cpp b/src/mlpack/methods/lmnn/lmnn_main.cpp index 821c4bd62d..0c270624e6 100644 --- a/src/mlpack/methods/lmnn/lmnn_main.cpp +++ b/src/mlpack/methods/lmnn/lmnn_main.cpp @@ -57,7 +57,7 @@ BINDING_LONG_DESC( PRINT_PARAM_STRING("regularization") + "), In addition, this " "implementation of LMNN includes a parameter to decide the interval " "after which impostors must be re-calculated (specified with " + - PRINT_PARAM_STRING("range") + ")." + PRINT_PARAM_STRING("update_interval") + ")." "\n\n" "Output can either be the learned distance matrix (specified with " + PRINT_PARAM_STRING("output") +"), or the transformed dataset " @@ -124,11 +124,11 @@ BINDING_EXAMPLE( PRINT_CALL("lmnn", "input", "iris", "labels", "iris_labels", "k", 3, "optimizer", "bbsgd", "output", "output") + "\n\n" - "An another program call making use of range & regularization parameter " - "with dataset having labels as last column can be made as: " + "Another program call making use of update interval & regularization " + "parameter with dataset having labels as last column can be made as: " "\n\n" + PRINT_CALL("lmnn", "input", "letter_recognition", "k", 5, - "range", 10, "regularization", 0.4, "output", "output")); + "update_interval", 10, "regularization", 0.4, "output", "output")); // See also... BINDING_SEE_ALSO("@nca", "#nca"); @@ -174,8 +174,8 @@ PARAM_DOUBLE_IN("step_size", "Step size for AMSGrad, BB_SGD and SGD (alpha).", PARAM_FLAG("linear_scan", "Don't shuffle the order in which data points are " "visited for SGD or mini-batch SGD.", "L"); PARAM_INT_IN("batch_size", "Batch size for mini-batch SGD.", "b", 50); -PARAM_INT_IN("range", "Number of iterations after which impostors needs to be " - "recalculated", "R", 1); +PARAM_INT_IN("update_interval", "Number of iterations after which impostors " + "need to be recalculated.", "R", 1); PARAM_INT_IN("seed", "Random seed. If 0, 'std::time(NULL)' is used.", "s", 0); using namespace mlpack; @@ -264,8 +264,8 @@ void BINDING_FUNCTION(util::Params& params, util::Timers& timers) RequireParamValue(params, "k", [](int x) { return x > 0; }, true, "number of targets must be positive"); - RequireParamValue(params, "range", [](int x) { return x > 0; }, true, - "range must be positive"); + RequireParamValue(params, "update_interval", [](int x) { return x > 0; }, + true, "update interval must be positive"); RequireParamValue(params, "batch_size", [](int x) { return x > 0; }, true, "batch size must be positive"); RequireParamValue(params, "regularization", [](double x) @@ -294,7 +294,7 @@ void BINDING_FUNCTION(util::Params& params, util::Timers& timers) const bool printAccuracy = params.Has("print_accuracy"); const bool shuffle = !params.Has("linear_scan"); const size_t batchSize = (size_t) params.Get("batch_size"); - const size_t range = (size_t) params.Get("range"); + const size_t updateInterval = (size_t) params.Get("update_interval"); const size_t rank = (size_t) params.Get("rank"); // Load data. @@ -359,56 +359,49 @@ void BINDING_FUNCTION(util::Params& params, util::Timers& timers) // Now create the LMNN object and run the optimization. timers.Start("lmnn_optimization"); + LMNN lmnn(k, regularization, updateInterval); if (optimizerType == "amsgrad") { - LMNN> lmnn(data, labels, k); - lmnn.Regularization() = regularization; - lmnn.Range() = range; - lmnn.Optimizer().StepSize() = stepSize; - lmnn.Optimizer().MaxIterations() = passes * data.n_cols; - lmnn.Optimizer().Tolerance() = tolerance; - lmnn.Optimizer().Shuffle() = shuffle; - lmnn.Optimizer().BatchSize() = batchSize; + ens::AMSGrad opt; + opt.StepSize() = stepSize; + opt.MaxIterations() = passes * data.n_cols; + opt.Tolerance() = tolerance; + opt.Shuffle() = shuffle; + opt.BatchSize() = batchSize; - lmnn.LearnDistance(distance); + lmnn.LearnDistance(data, labels, distance, opt); } else if (optimizerType == "bbsgd") { - LMNN, ens::BBS_BB> lmnn(data, labels, k); - lmnn.Regularization() = regularization; - lmnn.Range() = range; - lmnn.Optimizer().StepSize() = stepSize; - lmnn.Optimizer().MaxIterations() = passes * data.n_cols; - lmnn.Optimizer().Tolerance() = tolerance; - lmnn.Optimizer().Shuffle() = shuffle; - lmnn.Optimizer().BatchSize() = batchSize; + ens::BBS_BB opt; + opt.StepSize() = stepSize; + opt.MaxIterations() = passes * data.n_cols; + opt.Tolerance() = tolerance; + opt.Shuffle() = shuffle; + opt.BatchSize() = batchSize; - lmnn.LearnDistance(distance); + lmnn.LearnDistance(data, labels, distance, opt); } else if (optimizerType == "sgd") { // Using SGD is not recommended as the learning matrix can // diverge to inf causing serious memory problems. - LMNN, ens::StandardSGD> lmnn(data, labels, k); - lmnn.Regularization() = regularization; - lmnn.Range() = range; - lmnn.Optimizer().StepSize() = stepSize; - lmnn.Optimizer().MaxIterations() = passes * data.n_cols; - lmnn.Optimizer().Tolerance() = tolerance; - lmnn.Optimizer().Shuffle() = shuffle; - lmnn.Optimizer().BatchSize() = batchSize; + ens::StandardSGD opt; + opt.StepSize() = stepSize; + opt.MaxIterations() = passes * data.n_cols; + opt.Tolerance() = tolerance; + opt.Shuffle() = shuffle; + opt.BatchSize() = batchSize; - lmnn.LearnDistance(distance); + lmnn.LearnDistance(data, labels, distance, opt); } else if (optimizerType == "lbfgs") { - LMNN, ens::L_BFGS> lmnn(data, labels, k); - lmnn.Regularization() = regularization; - lmnn.Range() = range; - lmnn.Optimizer().MaxIterations() = maxIterations; - lmnn.Optimizer().MinGradientNorm() = tolerance; + ens::L_BFGS opt; + opt.MaxIterations() = maxIterations; + opt.MinGradientNorm() = tolerance; - lmnn.LearnDistance(distance); + lmnn.LearnDistance(data, labels, distance, opt); } timers.Stop("lmnn_optimization"); diff --git a/src/mlpack/methods/neighbor_search/neighbor_search.hpp b/src/mlpack/methods/neighbor_search/neighbor_search.hpp index b8f61c0155..d2470b1b19 100644 --- a/src/mlpack/methods/neighbor_search/neighbor_search.hpp +++ b/src/mlpack/methods/neighbor_search/neighbor_search.hpp @@ -222,9 +222,11 @@ class NeighborSearch * @param distances Matrix storing distances of neighbors for each query * point. */ + // TODO: templatize further to remove Armadillo type requirement + template void Search(const MatType& querySet, const size_t k, - arma::Mat& neighbors, + arma::Mat& neighbors, arma::Mat& distances); /** @@ -247,9 +249,11 @@ class NeighborSearch * @param sameSet Denotes whether or not the reference and query sets are the * same. */ + // TODO: templatize further to remove Armadillo type requirement + template void Search(Tree& queryTree, const size_t k, - arma::Mat& neighbors, + arma::Mat& neighbors, arma::Mat& distances, bool sameSet = false); @@ -267,8 +271,10 @@ class NeighborSearch * @param distances Matrix storing distances of neighbors for each query * point. */ + // TODO: templatize further to remove Armadillo type requirement + template void Search(const size_t k, - arma::Mat& neighbors, + arma::Mat& neighbors, arma::Mat& distances); /** @@ -300,8 +306,10 @@ class NeighborSearch * query point. * @return Recall. */ - static double Recall(arma::Mat& foundNeighbors, - arma::Mat& realNeighbors); + // TODO: templatize further to remove Armadillo type requirement + template + static double Recall(arma::Mat& foundNeighbors, + arma::Mat& realNeighbors); //! Return the total number of base case evaluations performed during the last //! search. diff --git a/src/mlpack/methods/neighbor_search/neighbor_search_impl.hpp b/src/mlpack/methods/neighbor_search/neighbor_search_impl.hpp index c73b7855f0..0aa0a7de0d 100644 --- a/src/mlpack/methods/neighbor_search/neighbor_search_impl.hpp +++ b/src/mlpack/methods/neighbor_search/neighbor_search_impl.hpp @@ -360,11 +360,12 @@ template class TreeType, template class DualTreeTraversalType, template class SingleTreeTraversalType> +template void NeighborSearch::Search( const MatType& querySet, const size_t k, - arma::Mat& neighbors, + arma::Mat& neighbors, arma::Mat& distances) { if (k > referenceSet->n_cols) @@ -385,7 +386,7 @@ DualTreeTraversalType, SingleTreeTraversalType>::Search( // indices back to their original indices when this computation is finished. // To avoid an extra copy, we will store the neighbors and distances in a // separate matrix. - arma::Mat* neighborPtr = &neighbors; + arma::Mat* neighborPtr = &neighbors; arma::Mat* distancePtr = &distances; // Mapping is only necessary if the tree rearranges points. @@ -394,10 +395,10 @@ DualTreeTraversalType, SingleTreeTraversalType>::Search( if (searchMode == DUAL_TREE_MODE) { distancePtr = new arma::Mat; // Query indices need to be mapped. - neighborPtr = new arma::Mat; + neighborPtr = new arma::Mat; } else if (!oldFromNewReferences.empty()) - neighborPtr = new arma::Mat; // Reference indices need mapping. + neighborPtr = new arma::Mat; // Reference indices need mapping. } // Set the size of the neighbor and distance matrices. @@ -565,11 +566,12 @@ template class TreeType, template class DualTreeTraversalType, template class SingleTreeTraversalType> +template void NeighborSearch::Search( Tree& queryTree, const size_t k, - arma::Mat& neighbors, + arma::Mat& neighbors, arma::Mat& distances, bool sameSet) { @@ -593,10 +595,10 @@ DualTreeTraversalType, SingleTreeTraversalType>::Search( const MatType& querySet = queryTree.Dataset(); // We won't need to map query indices, but will we need to map distances? - arma::Mat* neighborPtr = &neighbors; + arma::Mat* neighborPtr = &neighbors; if (!oldFromNewReferences.empty() && TreeTraits::RearrangesDataset) - neighborPtr = new arma::Mat; + neighborPtr = new arma::Mat; neighborPtr->set_size(k, querySet.n_cols); distances.set_size(k, querySet.n_cols); @@ -644,10 +646,11 @@ template class TreeType, template class DualTreeTraversalType, template class SingleTreeTraversalType> +template void NeighborSearch::Search( const size_t k, - arma::Mat& neighbors, + arma::Mat& neighbors, arma::Mat& distances) { if (k > referenceSet->n_cols) @@ -669,14 +672,14 @@ DualTreeTraversalType, SingleTreeTraversalType>::Search( baseCases = 0; scores = 0; - arma::Mat* neighborPtr = &neighbors; + arma::Mat* neighborPtr = &neighbors; arma::Mat* distancePtr = &distances; if (!oldFromNewReferences.empty() && TreeTraits::RearrangesDataset) { // We will always need to rearrange in this case. distancePtr = new MatType; - neighborPtr = new arma::Mat; + neighborPtr = new arma::Mat; } // Initialize results. @@ -861,10 +864,11 @@ template class TreeType, template class DualTreeTraversalType, template class SingleTreeTraversalType> +template double NeighborSearch::Recall( - arma::Mat& foundNeighbors, - arma::Mat& realNeighbors) + arma::Mat& foundNeighbors, + arma::Mat& realNeighbors) { if (foundNeighbors.n_rows != realNeighbors.n_rows || foundNeighbors.n_cols != realNeighbors.n_cols) diff --git a/src/mlpack/methods/neighbor_search/neighbor_search_rules.hpp b/src/mlpack/methods/neighbor_search/neighbor_search_rules.hpp index ef886663a7..057515ca7a 100644 --- a/src/mlpack/methods/neighbor_search/neighbor_search_rules.hpp +++ b/src/mlpack/methods/neighbor_search/neighbor_search_rules.hpp @@ -63,7 +63,10 @@ class NeighborSearchRules * @param distances Matrix storing distances of neighbors for each query * point. */ - void GetResults(arma::Mat& neighbors, arma::Mat& distances); + // TODO: templatize fully to remove requirement of Armadillo matrix + template + void GetResults(arma::Mat& neighbors, + arma::Mat& distances); /** * Get the distance from the query point to the reference point. diff --git a/src/mlpack/methods/neighbor_search/neighbor_search_rules_impl.hpp b/src/mlpack/methods/neighbor_search/neighbor_search_rules_impl.hpp index 24fa48c6d5..cfbd350090 100644 --- a/src/mlpack/methods/neighbor_search/neighbor_search_rules_impl.hpp +++ b/src/mlpack/methods/neighbor_search/neighbor_search_rules_impl.hpp @@ -59,8 +59,9 @@ NeighborSearchRules::NeighborSearchRules( } template +template void NeighborSearchRules::GetResults( - arma::Mat& neighbors, + arma::Mat& neighbors, arma::Mat& distances) { neighbors.set_size(k, querySet.n_cols); @@ -71,7 +72,7 @@ void NeighborSearchRules::GetResults( CandidateList& pqueue = candidates[i]; for (size_t j = 1; j <= k; ++j) { - neighbors(k - j, i) = pqueue.top().second; + neighbors(k - j, i) = (IndexType) pqueue.top().second; distances(k - j, i) = pqueue.top().first; pqueue.pop(); } diff --git a/src/mlpack/tests/callback_test.cpp b/src/mlpack/tests/callback_test.cpp index 942a56ddc4..ae9b912c9d 100644 --- a/src/mlpack/tests/callback_test.cpp +++ b/src/mlpack/tests/callback_test.cpp @@ -151,12 +151,13 @@ TEST_CASE("LMNNWithOptimizerCallback", "[CallbackTest]") " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; arma::Row labels = " 0 0 0 1 1 1 "; - LMNN<> lmnn(dataset, labels, 1); + LMNN<> lmnn(1); arma::mat outputMatrix; std::stringstream stream; - lmnn.LearnDistance(outputMatrix, ens::ProgressBar(70, stream)); + lmnn.LearnDistance(dataset, labels, outputMatrix, + ens::ProgressBar(70, stream)); REQUIRE(stream.str().length() > 0); } diff --git a/src/mlpack/tests/lmnn_test.cpp b/src/mlpack/tests/lmnn_test.cpp index 44eb0b0d66..6b0c0d254a 100644 --- a/src/mlpack/tests/lmnn_test.cpp +++ b/src/mlpack/tests/lmnn_test.cpp @@ -30,25 +30,27 @@ using namespace ens; * The target neighbors function should be correct. * point. */ -TEST_CASE("LMNNTargetNeighborsTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNTargetNeighborsTest", "[LMNNTest]", float, double) { - // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + typedef TestType ElemType; - Constraints<> constraint(dataset, labels, 1); + // Useful but simple dataset with six points and two classes. + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; + + Constraints, arma::Row> constraint(dataset, + labels, 1); // Calculate norm of datapoints. - arma::vec norm(dataset.n_cols); + arma::Col norm(dataset.n_cols); for (size_t i = 0; i < dataset.n_cols; ++i) { norm(i) = arma::norm(dataset.col(i)); } //! Store target neighbors of data points. - arma::Mat targetNeighbors = - arma::Mat(1, dataset.n_cols, arma::fill::zeros); + arma::umat targetNeighbors(1, dataset.n_cols, arma::fill::zeros); constraint.TargetNeighbors(targetNeighbors, dataset, labels, norm); @@ -63,25 +65,27 @@ TEST_CASE("LMNNTargetNeighborsTest", "[LMNNTest]") /** * The impostors function should be correct. */ -TEST_CASE("LMNNImpostorsTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNImpostorsTest", "[LMNNTest]", float, double) { - // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + typedef TestType ElemType; - Constraints<> constraint(dataset, labels, 1); + // Useful but simple dataset with six points and two classes. + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; + + Constraints, arma::Row> constraint(dataset, + labels, 1); // Calculate norm of datapoints. - arma::vec norm(dataset.n_cols); + arma::Col norm(dataset.n_cols); for (size_t i = 0; i < dataset.n_cols; ++i) { norm(i) = arma::norm(dataset.col(i)); } //! Store impostors of data points. - arma::Mat impostors = - arma::Mat(1, dataset.n_cols, arma::fill::zeros); + arma::umat impostors(1, dataset.n_cols, arma::fill::zeros); constraint.Impostors(impostors, dataset, labels, norm); @@ -101,300 +105,339 @@ TEST_CASE("LMNNImpostorsTest", "[LMNNTest]") * The LMNN function should return the identity matrix as its initial * point. */ -TEST_CASE("LMNNInitialPointTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNInitialPointTest", "[LMNNTest]", float, double) { + typedef TestType ElemType; + // Cheap fake dataset. - arma::mat dataset = arma::randu(5, 5); + arma::Mat dataset = arma::randu>(5, 5); arma::Row labels = "0 1 1 0 0"; - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.5, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.5, 1); // Verify the initial point is the identity matrix. - arma::mat initialPoint = lmnnfn.GetInitialPoint(); + const double eps = std::is_same::value ? 1e-4 : 1e-7; + const double margin = std::is_same::value ? 1e-4 : 1e-5; + arma::Mat initialPoint = lmnnfn.GetInitialPoint(); for (int row = 0; row < 5; row++) { for (int col = 0; col < 5; col++) { if (row == col) - REQUIRE(initialPoint(row, col) == Approx(1.0).epsilon(1e-7)); + REQUIRE(initialPoint(row, col) == Approx(1.0).epsilon(eps)); else - REQUIRE(initialPoint(row, col) == Approx(0.0).margin(1e-5)); + REQUIRE(initialPoint(row, col) == Approx(0.0).margin(margin)); } } } /*** - * Ensure non-seprable objective function is right. + * Ensure non-separable objective function is right. */ -TEST_CASE("LMNNInitialEvaluationTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNInitialEvaluationTest", "[LMNNTest]", float, double) { + typedef TestType ElemType; + // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); - double objective = lmnnfn.Evaluate(arma::eye(2, 2)); + ElemType objective = lmnnfn.Evaluate(arma::eye>(2, 2)); // Result calculated by hand. - REQUIRE(objective == Approx(9.456).epsilon(1e-7)); + const double eps = std::is_same::value ? 1e-4 : 1e-7; + REQUIRE(objective == Approx(9.456).epsilon(eps)); } /** - * Ensure non-seprable gradient function is right. + * Ensure non-separable gradient function is right. */ -TEST_CASE("LMNNInitialGradientTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNInitialGradientTest", "[LMNNTest]", float, double) { + typedef TestType ElemType; + // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); - arma::mat gradient; - arma::mat coordinates = arma::eye(2, 2); + arma::Mat gradient; + arma::Mat coordinates = arma::eye>(2, 2); lmnnfn.Gradient(coordinates, gradient); // Result calculated by hand. - REQUIRE(gradient(0, 0) == Approx(-0.288).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).margin(1e-5)); - REQUIRE(gradient(0, 1) == Approx(0.0).margin(1e-5)); - REQUIRE(gradient(1, 1) == Approx(12.0).epsilon(1e-7)); + const double eps = std::is_same::value ? 1e-4 : 1e-7; + const double margin = std::is_same::value ? 1e-4 : 1e-5; + REQUIRE(gradient(0, 0) == Approx(-0.288).epsilon(eps)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(12.0).epsilon(eps)); } /*** - * Ensure non-seprable EvaluateWithGradient function is right. + * Ensure non-separable EvaluateWithGradient function is right. */ -TEST_CASE("LMNNInitialEvaluateWithGradientTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNInitialEvaluateWithGradientTest", "[LMNNTest]", float, + double) { + typedef TestType ElemType; + // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); - arma::mat gradient; - arma::mat coordinates = arma::eye(2, 2); - double objective = lmnnfn.EvaluateWithGradient(coordinates, gradient); + arma::Mat gradient; + arma::Mat coordinates = arma::eye>(2, 2); + ElemType objective = lmnnfn.EvaluateWithGradient(coordinates, gradient); + + const double eps = std::is_same::value ? 1e-4 : 1e-7; + const double margin = std::is_same::value ? 1e-4 : 1e-5; // Result calculated by hand. - REQUIRE(objective == Approx(9.456).epsilon(1e-7)); + REQUIRE(objective == Approx(9.456).epsilon(eps)); // Check Gradient - REQUIRE(gradient(0, 0) == Approx(-0.288).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).margin(1e-5)); - REQUIRE(gradient(0, 1) == Approx(0.0).margin(1e-5)); - REQUIRE(gradient(1, 1) == Approx(12.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.288).epsilon(eps)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(12.0).epsilon(eps)); } /** * Ensure the separable objective function is right. */ -TEST_CASE("LMNNSeparableObjectiveTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNSeparableObjectiveTest", "[LMNNTest]", float, double) { - // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + typedef TestType ElemType; - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + // Useful but simple dataset with six points and two classes. + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; + + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); // Result calculated by hand. - arma::mat coordinates = arma::eye(2, 2); - REQUIRE(lmnnfn.Evaluate(coordinates, 0, 1) == Approx(1.576).epsilon(1e-7)); - REQUIRE(lmnnfn.Evaluate(coordinates, 1, 1) == Approx(1.576).epsilon(1e-7)); - REQUIRE(lmnnfn.Evaluate(coordinates, 2, 1) == Approx(1.576).epsilon(1e-7)); - REQUIRE(lmnnfn.Evaluate(coordinates, 3, 1) == Approx(1.576).epsilon(1e-7)); - REQUIRE(lmnnfn.Evaluate(coordinates, 4, 1) == Approx(1.576).epsilon(1e-7)); - REQUIRE(lmnnfn.Evaluate(coordinates, 5, 1) == Approx(1.576).epsilon(1e-7)); + const double eps = std::is_same::value ? 1e-4 : 1e-7; + arma::Mat coordinates = arma::eye>(2, 2); + REQUIRE(lmnnfn.Evaluate(coordinates, 0, 1) == Approx(1.576).epsilon(eps)); + REQUIRE(lmnnfn.Evaluate(coordinates, 1, 1) == Approx(1.576).epsilon(eps)); + REQUIRE(lmnnfn.Evaluate(coordinates, 2, 1) == Approx(1.576).epsilon(eps)); + REQUIRE(lmnnfn.Evaluate(coordinates, 3, 1) == Approx(1.576).epsilon(eps)); + REQUIRE(lmnnfn.Evaluate(coordinates, 4, 1) == Approx(1.576).epsilon(eps)); + REQUIRE(lmnnfn.Evaluate(coordinates, 5, 1) == Approx(1.576).epsilon(eps)); } /** * Ensure the separable gradient is right. */ -TEST_CASE("LMNNSeparableGradientTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNSeparableGradientTest", "[LMNNTest]", float, double) { + typedef TestType ElemType; + // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); - arma::mat coordinates = arma::eye(2, 2); - arma::mat gradient(2, 2); + arma::Mat coordinates = arma::eye>(2, 2); + arma::Mat gradient(2, 2); lmnnfn.Gradient(coordinates, 0, gradient, 1); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + const double eps = std::is_same::value ? 1e-4 : 1e-7; + const double margin = std::is_same::value ? 1e-4 : 1e-5; + + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); lmnnfn.Gradient(coordinates, 1, gradient, 1); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); lmnnfn.Gradient(coordinates, 2, gradient, 1); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); lmnnfn.Gradient(coordinates, 3, gradient, 1); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); lmnnfn.Gradient(coordinates, 4, gradient, 1); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); lmnnfn.Gradient(coordinates, 5, gradient, 1); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); } /** * Ensure the separable EvaluateWithGradient function is right. */ -TEST_CASE("LMNNSeparableEvaluateWithGradientTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNSeparableEvaluateWithGradientTest", "[LMNNTest]", float, + double) { + typedef TestType ElemType; + // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); - arma::mat coordinates = arma::eye(2, 2); - arma::mat gradient(2, 2); + arma::Mat coordinates = arma::eye>(2, 2); + arma::Mat gradient(2, 2); - double objective = lmnnfn.EvaluateWithGradient(coordinates, 0, gradient, 1); + ElemType objective = lmnnfn.EvaluateWithGradient(coordinates, 0, gradient, 1); - REQUIRE(objective == Approx(1.576).epsilon(1e-7)); + const double eps = std::is_same::value ? 1e-4 : 1e-7; + const double margin = std::is_same::value ? 1e-4 : 1e-5; - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(objective == Approx(1.576).epsilon(eps)); + + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); objective = lmnnfn.EvaluateWithGradient(coordinates, 1, gradient, 1); - REQUIRE(objective == Approx(1.576).epsilon(1e-7)); + REQUIRE(objective == Approx(1.576).epsilon(eps)); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); objective = lmnnfn.EvaluateWithGradient(coordinates, 2, gradient, 1); - REQUIRE(objective == Approx(1.576).epsilon(1e-7)); + REQUIRE(objective == Approx(1.576).epsilon(eps)); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); objective = lmnnfn.EvaluateWithGradient(coordinates, 3, gradient, 1); - REQUIRE(objective == Approx(1.576).epsilon(1e-7)); + REQUIRE(objective == Approx(1.576).epsilon(eps)); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); objective = lmnnfn.EvaluateWithGradient(coordinates, 4, gradient, 1); - REQUIRE(objective == Approx(1.576).epsilon(1e-7)); + REQUIRE(objective == Approx(1.576).epsilon(eps)); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); objective = lmnnfn.EvaluateWithGradient(coordinates, 5, gradient, 1); - REQUIRE(objective == Approx(1.576).epsilon(1e-7)); + REQUIRE(objective == Approx(1.576).epsilon(eps)); - REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7)); - REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7)); - REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7)); + REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps)); + REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin)); + REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps)); } // Check that final objective value using SGD optimizer is optimal. -TEST_CASE("LMNNSGDSimpleDatasetTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNSGDSimpleDatasetTest", "[LMNNTest]", float, double) { + typedef TestType ElemType; + // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; - LMNN<> lmnn(dataset, labels, 1); + LMNN<> lmnn(1); - arma::mat outputMatrix; - lmnn.LearnDistance(outputMatrix); + arma::Mat outputMatrix; + lmnn.LearnDistance(dataset, labels, outputMatrix); // Ensure that the objective function is better now. - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); - double initObj = lmnnfn.Evaluate(arma::eye(2, 2)); - double finalObj = lmnnfn.Evaluate(outputMatrix); + ElemType initObj = lmnnfn.Evaluate(arma::eye>(2, 2)); + ElemType finalObj = lmnnfn.Evaluate(outputMatrix); // finalObj must be less than initObj. REQUIRE(finalObj < initObj); } // Check that final objective value using L-BFGS optimizer is optimal. -TEST_CASE("LMNNLBFGSSimpleDatasetTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNLBFGSSimpleDatasetTest", "[LMNNTest]", float, double) { + typedef TestType ElemType; + // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; - LMNN lmnn(dataset, labels, 1); + LMNN lmnn(1); - arma::mat outputMatrix; - lmnn.LearnDistance(outputMatrix); + arma::Mat outputMatrix; + ens::L_BFGS lbfgs; + lmnn.LearnDistance(dataset, labels, outputMatrix, lbfgs); // Ensure that the objective function is better now. - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); - double initObj = lmnnfn.Evaluate(arma::eye(2, 2)); - double finalObj = lmnnfn.Evaluate(outputMatrix); + ElemType initObj = lmnnfn.Evaluate(arma::eye>(2, 2)); + ElemType finalObj = lmnnfn.Evaluate(outputMatrix); // finalObj must be less than initObj. REQUIRE(finalObj < initObj); } -double KnnAccuracy(const arma::mat& dataset, - const arma::Row& labels, +template +double KnnAccuracy(const MatType& dataset, + const LabelsType& labels, const size_t k) { - arma::Row uniqueLabels = arma::unique(labels); + typedef typename MatType::elem_type ElemType; + + LabelsType uniqueLabels = arma::unique(labels); arma::Mat neighbors; - arma::mat distances; + arma::Mat distances; - KNN knn; + NeighborSearch knn; knn.Train(dataset); knn.Search(k, neighbors, distances); @@ -404,43 +447,44 @@ double KnnAccuracy(const arma::mat& dataset, for (size_t i = 0; i < dataset.n_cols; ++i) { - arma::vec Map; - Map.zeros(uniqueLabels.n_cols); + arma::Col m; + m.zeros(uniqueLabels.n_cols); for (size_t j = 0; j < k; ++j) - Map(labels(neighbors(j, i))) += - 1 / std::pow(distances(j, i) + 1, 2); + m(labels(neighbors(j, i))) += 1 / std::pow(distances(j, i) + 1, 2); - size_t index = ConvTo::From(arma::find(Map - == arma::max(Map))); + size_t index = ConvTo::From(arma::find(m == arma::max(m))); // Increase count if labels match. if (index == labels(i)) count++; } - // return accuracy. + // Return accuracy. return ((double) count / dataset.n_cols) * 100; } // Check that final accuracy is greater than initial accuracy on // simple dataset. -TEST_CASE("LMNNAccuracyTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNAccuracyTest", "[LMNNTest]", float, double) { + typedef TestType ElemType; + // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; // Taking k = 3 as the case of k = 1 can be easily observed. double initAccuracy = KnnAccuracy(dataset, labels, 3); - LMNN<> lmnn(dataset, labels, 2); + LMNN<> lmnn(2); - arma::mat outputMatrix; - lmnn.LearnDistance(outputMatrix); + arma::Mat outputMatrix; + lmnn.LearnDistance(dataset, labels, outputMatrix); - double finalAccuracy = KnnAccuracy(outputMatrix * dataset, labels, 3); + arma::Mat transformedData = outputMatrix * dataset; + double finalAccuracy = KnnAccuracy(transformedData, labels, 3); // finalObj must be less than initObj. REQUIRE(initAccuracy < finalAccuracy); @@ -452,18 +496,20 @@ TEST_CASE("LMNNAccuracyTest", "[LMNNTest]") // Check that accuracy while learning square distance matrix is the same as when // we are learning low rank matrix. I'm ok if this passes only once out of // three tries. -TEST_CASE("LMNNLowRankAccuracyLBFGSTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNLowRankAccuracyLBFGSTest", "[LMNNTest]", float, double) { + typedef TestType ElemType; + bool success = false; for (size_t trial = 0; trial < 3; ++trial) { - arma::mat dataPart1; + arma::Mat dataPart1; dataPart1.randn(5, 50); arma::Row labelsPart1(50); labelsPart1.fill(0); - arma::mat dataPart2; + arma::Mat dataPart2; dataPart2.randn(5, 50); arma::Row labelsPart2(50); @@ -473,26 +519,29 @@ TEST_CASE("LMNNLowRankAccuracyLBFGSTest", "[LMNNTest]") arma::uvec ordering = arma::shuffle(arma::linspace(0, 99, 100)); // Generate datasets. - arma::mat dataset = join_rows(dataPart1, dataPart2); + arma::Mat dataset = join_rows(dataPart1, dataPart2); dataset = dataset.cols(ordering); // Generate labels. arma::Row labels = join_rows(labelsPart1, labelsPart2); labels = labels.cols(ordering); - LMNN lmnn(dataset, labels, 1); + LMNN lmnn(1); // Learn a square matrix. - arma::mat outputMatrix; - lmnn.LearnDistance(outputMatrix); + arma::Mat outputMatrix; + L_BFGS lbfgs; + lmnn.LearnDistance(dataset, labels, outputMatrix, lbfgs); - double acc1 = KnnAccuracy(outputMatrix * dataset, labels, 1); + arma::Mat transformedData = outputMatrix * dataset; + double acc1 = KnnAccuracy(transformedData, labels, 1); // Learn a low rank matrix. - outputMatrix = arma::randu(4, 5); - lmnn.LearnDistance(outputMatrix); + outputMatrix = arma::randu>(4, 5); + lmnn.LearnDistance(dataset, labels, outputMatrix, lbfgs); - double acc2 = KnnAccuracy(outputMatrix * dataset, labels, 1); + transformedData = outputMatrix * dataset; + double acc2 = KnnAccuracy(transformedData, labels, 1); // We keep the tolerance very high. We need to ensure the accuracy drop // isn't any more than 10%. @@ -507,18 +556,20 @@ TEST_CASE("LMNNLowRankAccuracyLBFGSTest", "[LMNNTest]") // Check that accuracy while learning square distance matrix is the same as when // we are learning low rank matrix. I'm ok if this passes only once out of // three tries. -TEST_CASE("LMNNLowRankAccuracyTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNLowRankAccuracyTest", "[LMNNTest]", float, double) { + typedef TestType ElemType; + bool success = false; for (size_t trial = 0; trial < 3; ++trial) { - arma::mat dataPart1; + arma::Mat dataPart1; dataPart1.randn(5, 50); arma::Row labelsPart1(50); labelsPart1.fill(0); - arma::mat dataPart2; + arma::Mat dataPart2; dataPart2.randn(5, 50); arma::Row labelsPart2(50); @@ -528,26 +579,28 @@ TEST_CASE("LMNNLowRankAccuracyTest", "[LMNNTest]") arma::uvec ordering = arma::shuffle(arma::linspace(0, 99, 100)); // Generate datasets. - arma::mat dataset = join_rows(dataPart1, dataPart2); + arma::Mat dataset = join_rows(dataPart1, dataPart2); dataset = dataset.cols(ordering); // Generate labels. arma::Row labels = join_rows(labelsPart1, labelsPart2); labels = labels.cols(ordering); - LMNN<> lmnn(dataset, labels, 1); + LMNN<> lmnn(1); // Learn a square matrix. - arma::mat outputMatrix; - lmnn.LearnDistance(outputMatrix); + arma::Mat outputMatrix; + lmnn.LearnDistance(dataset, labels, outputMatrix); - double acc1 = KnnAccuracy(outputMatrix * dataset, labels, 1); + arma::Mat transformedData = outputMatrix * dataset; + double acc1 = KnnAccuracy(transformedData, labels, 1); // Learn a low rank matrix. - outputMatrix = arma::randu(4, 5); - lmnn.LearnDistance(outputMatrix); + outputMatrix = arma::randu>(4, 5); + lmnn.LearnDistance(dataset, labels, outputMatrix); - double acc2 = KnnAccuracy(outputMatrix * dataset, labels, 1); + transformedData = outputMatrix * dataset; + double acc2 = KnnAccuracy(transformedData, labels, 1); // We keep the tolerance very high. We need to ensure the accuracy drop // isn't any more than 10%. @@ -621,29 +674,31 @@ TEST_CASE("LMNNLowRankAccuracyBBSGDTest", "[LMNNTest]") // Comprehensive gradient tests by Marcus Edel & Ryan Curtin. // Simple numerical gradient checker. -template +template double CheckGradient(FunctionType& function, - arma::mat& coordinates, - const double eps = 1e-7) + MatType& coordinates, + const typename MatType::elem_type eps = 1e-7) { + typedef typename MatType::elem_type ElemType; + // Get gradients for the current parameters. - arma::mat orgGradient, gradient, estGradient; + MatType orgGradient, gradient, estGradient; function.Gradient(coordinates, orgGradient); - estGradient = arma::zeros(orgGradient.n_rows, orgGradient.n_cols); + estGradient = arma::zeros(orgGradient.n_rows, orgGradient.n_cols); // Compute numeric approximations to gradient. for (size_t i = 0; i < orgGradient.n_elem; ++i) { - double tmp = coordinates(i); + ElemType tmp = coordinates(i); // Perturb parameter with a positive constant and get costs. coordinates(i) += eps; - double costPlus = function.Evaluate(coordinates); + ElemType costPlus = function.Evaluate(coordinates); // Perturb parameter with a negative constant and get costs. coordinates(i) -= (2 * eps); - double costMinus = function.Evaluate(coordinates); + ElemType costMinus = function.Evaluate(coordinates); // Restore the parameter value. coordinates(i) = tmp; @@ -657,74 +712,84 @@ double CheckGradient(FunctionType& function, arma::norm(orgGradient + estGradient); } -TEST_CASE("LMNNFunctionGradientTest", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNFunctionGradientTest", "[LMNNTest]", float, double) { + typedef TestType ElemType; + // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; arma::Row labels = " 0 0 0 1 1 1 "; - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); // 10 trials with random positions. for (size_t i = 0; i < 10; ++i) { - arma::mat coordinates(2, 2, arma::fill::randn); + arma::Mat coordinates(2, 2, arma::fill::randn); CheckGradient(lmnnfn, coordinates); } } -TEST_CASE("LMNNFunctionGradientTest2", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNFunctionGradientTest2", "[LMNNTest]", float, double) { - // Useful but simple dataset with six points and two classes. - arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" - " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; - arma::Row labels = " 0 0 0 1 1 1 "; + typedef TestType ElemType; - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + // Useful but simple dataset with six points and two classes. + arma::Mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;" + " 1.0 0.0 -1.0 1.0 0.0 -1.0 "; + arma::Row labels = " 0 0 0 1 1 1 "; + + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); // 10 trials with random positions. for (size_t i = 0; i < 10; ++i) { - arma::mat coordinates(2, 2, arma::fill::randu); + arma::Mat coordinates(2, 2, arma::fill::randu); CheckGradient(lmnnfn, coordinates); } } -TEST_CASE("LMNNFunctionGradientTest3", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNFunctionGradientTest3", "[LMNNTest]", float, double) { - arma::mat dataset; + typedef TestType ElemType; + + arma::Mat dataset; arma::Row labels; if (!data::Load("iris.csv", dataset)) FAIL("Cannot load dataset iris.csv"); if (!data::Load("iris_labels.txt", labels)) FAIL("Cannot load dataset iris_labels.txt"); - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); // 10 trials with random positions. for (size_t i = 0; i < 10; ++i) { - arma::mat coordinates(dataset.n_rows, dataset.n_rows, arma::fill::randn); + arma::Mat coordinates(dataset.n_rows, dataset.n_rows, + arma::fill::randn); CheckGradient(lmnnfn, coordinates); } } -TEST_CASE("LMNNFunctionGradientTest4", "[LMNNTest]") +TEMPLATE_TEST_CASE("LMNNFunctionGradientTest4", "[LMNNTest]", float, double) { - arma::mat dataset; + typedef TestType ElemType; + + arma::Mat dataset; arma::Row labels; if (!data::Load("iris.csv", dataset)) FAIL("Cannot load dataset iris.csv"); if (!data::Load("iris_labels.txt", labels)) FAIL("Cannot load dataset iris_labels.txt"); - LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1); + LMNNFunction> lmnnfn(dataset, labels, 1, 0.6, 1); // 10 trials with random positions. for (size_t i = 0; i < 10; ++i) { - arma::mat coordinates(dataset.n_rows, dataset.n_rows, arma::fill::randu); + arma::Mat coordinates(dataset.n_rows, dataset.n_rows, + arma::fill::randu); CheckGradient(lmnnfn, coordinates); } } diff --git a/src/mlpack/tests/main_tests/lmnn_test.cpp b/src/mlpack/tests/main_tests/lmnn_test.cpp index 13b68b3d47..901033856a 100644 --- a/src/mlpack/tests/main_tests/lmnn_test.cpp +++ b/src/mlpack/tests/main_tests/lmnn_test.cpp @@ -542,7 +542,7 @@ TEST_CASE_METHOD(LMNNTestFixture, "LMNNDiffRegularizationTest", } /** - * Ensure that different value of range results in a + * Ensure that different value of update interval results in a * different output matrix. */ TEST_CASE_METHOD(LMNNTestFixture, "LMNNDiffRangeTest", @@ -573,7 +573,7 @@ TEST_CASE_METHOD(LMNNTestFixture, "LMNNDiffRangeTest", SetInputParam("input", std::move(inputData)); SetInputParam("labels", std::move(labels)); SetInputParam("linear_scan", (bool) true); - SetInputParam("range", 100); + SetInputParam("update_interval", 100); RUN_BINDING(); @@ -674,9 +674,9 @@ TEST_CASE_METHOD(LMNNTestFixture, "LMNNDiffPassesTest", } /** - * Ensure that number of targets, range, batch size must be always positive - * and regularization, step size, max iterations, rank, passes & tolerance are - * always non-negative + * Ensure that number of targets, update interval, batch size must be always + * positive and regularization, step size, max iterations, rank, passes & + * tolerance are always non-negative. */ TEST_CASE_METHOD(LMNNTestFixture, "LMNNBoundsTest", "[LMNNMainTest][BindingTests]") @@ -701,12 +701,12 @@ TEST_CASE_METHOD(LMNNTestFixture, "LMNNBoundsTest", // Reset settings. ResetSettings(); - // Test for range value. + // Test for update interval value. // Input training data. SetInputParam("input", inputData); SetInputParam("labels", labels); - SetInputParam("range", (int) 0); + SetInputParam("update_interval", (int) 0); REQUIRE_THROWS_AS(RUN_BINDING(), std::runtime_error);