apaam/mlpack
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Templatize LMNN and add documentation.

Browse Source
This commit is contained in:
Ryan Curtin
2024-07-06 12:40:55 -04:00
parent 83af6c5adb
commit d351a378ab
15 changed files with 1363 additions and 647 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/user/methods/lmnn.md
+454
View File
@@ -0,0 +1,454 @@
## LMNN
The `LMNN` class implements large margin nearest neighbor, which can be used
as both a linear dimensionality reduction technique and a distance learning
technique (also called metric learning). LMNN finds a linear transformation of
the dataset that improves `k`-nearest-neighbor classification performance.
#### Simple usage example:
```c++
// Learn a distance metric that improves kNN classification performance.
// All data and labels are uniform random; 10 dimensional data, 5 classes.
// Replace with a data::Load() call or similar for a real application.
arma::mat dataset(10, 1000, arma::fill::randu); // 1000 points.
arma::Row<size_t> labels =
arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 4));
mlpack::LMNN lmnn(3 /* neighbors to consider */); // Step 1: create object.
arma::mat distance;
lmnn.LearnDistance(dataset, labels, distance); // Step 2: learn distance.
// `distance` can now be used as a transformation matrix for the data.
arma::mat transformedData = distance * dataset;
// Or, you can create a MahalanobisDistance to evaluate points in the
// transformed dataset space.
arma::mat q = distance.t() * distance;
mlpack::MahalanobisDistance d(std::move(q));
std::cout << "Distance between points 0 and 1:" << std::endl;
std::cout << " - Before LMNN: "
<< mlpack::EuclideanDistance::Evaluate(dataset.col(0), dataset.col(1))
<< "." << std::endl;
std::cout << " - After LMNN: "
<< d.Evaluate(dataset.col(0), dataset.col(1)) << "." << std::endl;
```
<p style="text-align: center; font-size: 85%"><a href="#simple-examples">More examples...</a></p>
#### Quick links:
* [Constructors](#constructors): create `LMNN` objects.
* [`LearnDistance()`](#learning-distances): learn distance metrics.
* [Other functionality](#other-functionality) for loading and saving.
* [Examples](#simple-examples) of simple usage and integration with other
techniques.
#### See also:
<!-- TODO: link to kNN -->
* [mlpack distance metrics](../core.md#distances)
* [`NCA`](nca.md)
* [Metric learning on Wikipedia](https://en.wikipedia.org/wiki/Similarity_learning#Metric_learning)
* [Large margin nearest neighbor on Wikipedia](https://en.wikipedia.org/wiki/Large_margin_nearest_neighbor)
* [Distance metric learning for Large Margin Nearest Neighbor Classification (pdf)](https://proceedings.neurips.cc/paper_files/paper/2005/file/a7f592cef8b130a6967a90617db5681b-Paper.pdf)
### Constructors
* `lmnn = LMNN(k, regularization=0.5, updateInterval=1)`
- Create an `LMNN` object considering the specified number `k` of neighbors.
- Optionally, specify the regularization to be applied to the LMNN cost
function (a `double`), and the number of iterations between recomputation
of neighbors (`updateInterval`, a `size_t`).
---
* `lmnn = LMNN<DistanceType>(k, regularization=0.5, updateInterval=1)`
* `lmnn = LMNN<DistanceType>(k, regularization, updateInterval, distance)`
- Create an `LMNN` object using a custom
[`DistanceType`](../core.md#distances).
- `k` specifies the number of neighbors to consider.
- `regularization` specifies the regularization penalty to be applied to the
LMNN cost function (a `double`).
- `updateInterval` specifies the number of iterations between recomputation
of neighbors (a `size_t`).
- An instantiated `DistanceType` can optionally be passed with the `distance`
parameter.
- Using a custom `DistanceType` means that `LearnDistance()` will learn a
linear transformation for the data *in the metric space of the custom
`DistanceType`*.
* This means any learned distance may not necessarily improve
classification performance with the
[Euclidean distance](../core.md#lmetric).
* Instead, classification performance will be improved when the learned
distance is used with the given `DistanceType` only.
- Any mlpack `DistanceType` can be used as a drop-in replacement, or a
[custom `DistanceType`](../../developer/distances.md).
* A list of mlpack's provided distance metrics can be found
[here](../core.md#distances).
- ***Note: be sure that you understand the implications of a custom
`DistanceType` before using this version.***
---
***Notes***:
- A larger `k` will cause `LearnDistance()` to take longer to compute, but will
give more accurate results. It is generally suggested to keep `k` in roughly
the `3` to `5` range, depending on the dataset. Using `k = 1` can provide
fast convergence, but the learned distance metric may be of lower quality.
- `regularization` controls the balance between encouraging small distances for
points of the same class and penalizing small distances for points of
different classes. When `regularization` is increased, small distances for
points of different classes are further penalized.
- Setting `updateInterval` greater than `1` will allow the LMNN algorithm to
take multiple steps without the expensive recomputation of neighbors, but
this means that subsequent optimization steps may not be using the true
nearest neighbors.
* If using an SGD-like algorithm (i.e. an optimizer for a
[differentiable separable function](https://www.ensmallen.org/docs.html#differentiable-separable-functions)),
this can often be set to a relatively high value (100 is not unreasonable).
* If using an optimizer like L-BFGS (i.e. a full-batch optimizer for
[differentiable functions](https://www.ensmallen.org/docs.html#differentiable-functions)),
this should be kept relatively low (going above 10 is not advised).
* It is worth cross-validating different values of the parameter to see what
works for your dataset.
---
### Learning Distances
Once an `LMNN` object has been created, the `LearnDistance()` method can be used
to learn a distance.
* `lmnn.LearnDistance(data, labels, distance, [callbacks...])`
* `lmnn.LearnDistance(data, labels, distance, optimizer, [callbacks...])`
- Learn a distance metric on the given `data` and `labels`, filling
`distance` with a transformation matrix that can be used to map the data
into the space of the learned distance.
- Optionally, pass an instantiated
[ensmallen optimizer](https://www.ensmallen.org) and/or
[ensmallen callbacks](https://www.ensmallen.org/docs.html#callback-documentation)
to be used for the learning process.
- If no optimizer is passed,
[`ens::AMSGrad`](https://www.ensmallen.org/docs.html#amsgrad) is used.
- If `distance` already has size `r` x `data.n_rows` for some `r` less than
or equal to `data.n_rows`, it will be used as the starting point for
optimization. Otherwise, the identity matrix with size `data.n_rows` x
`data.n_rows` will be used.
- When optimization is complete, `distance` will have size `r` x
`data.n_rows`, where `r` is less than or equal to `data.n_rows`.
* *Note*: If `r < data.n_rows`, then LMNN has learned a distance metric
that also reduces the dimensionality of the data. See the
[last example](#simple-examples).
To use `distance`, either:
* Compute a new transformed dataset as `distance * data`, or
* Use an instantiated [`MahalanobisDistance`](../core.md#mahalanobisdistance)
with `distance.t() * distance` as the `Q` matrix.
See the [examples section](#simple-examples) for more details.
#### `LearnDistance()` Parameters:
| **name** | **type** | **description** |
|----------|----------|-----------------|
| `data` | [`arma::mat`](../matrices.md) | [Column-major](../matrices.md#representing-data-in-mlpack) training matrix. |
| `labels` | [`arma::Row<size_t>`](../matrices.md) | Training labels, [between `0` and `numClasses - 1`](../load_save.md#normalizing-labels) (inclusive). Should have length `data.n_cols`. |
| `distance` | [`arma::mat`](../matrices.md) | Output matrix to store transformation matrix representing learned distance. |
| `optimizer` | [any ensmallen optimizer](https://www.ensmallen.org) | Instantiated ensmallen optimizer for [differentiable functions](https://www.ensmallen.org/docs.html#differentiable-functions) or [differentiable separable functions](https://www.ensmallen.org/docs.html#differentiable-separable-functions). | `ens::AMSGrad()` |
| `callbacks...` | [any set of ensmallen callbacks](https://www.ensmallen.org/docs.html#callback-documentation) | Optional callbacks for the ensmallen optimizer, such as e.g. `ens::ProgressBar()`, `ens::Report()`, or others. | _(N/A)_ |
***Note***: any matrix type can be used for `data` and `distance`, so long as
that type implements the Armadillo API. So, e.g., `arma::fmat` can be used.
### Other Functionality
* An `LMNN` object can be serialized with
[`data::Save()` and `data::Load()`](../load_save.md#mlpack-objects).
Note that this is only meaningful if a custom `DistanceType` is being used,
and that custom `DistanceType` has state to be saved.
* `lmnn.K()` returns the number of neighbors used by LMNN, and `lmnn.K() = k`
will set the number of neighbors to use to `k`.
* `lmnn.Regularization()` returns the current regularization value of the LMNN
object (as a `double`), and `lmnn.Regularization() = r` can be used to set
the regularization value to `r`.
* `lmnn.UpdateInterval()` returns the current number of iterations between
neighbor recomputation (as a `size_t`), and `lmnn.UpdateInterval() = i` sets
the number of iterations between neighbor recomputation to `i`.
* `lmnn.Distance()` will return the `DistanceType` being used for learning.
Unless a custom `DistanceType` was specified in the constructor,
this simply returns a [`SquaredEuclideanDistance`](../core.md#lmetric)
object.
### Simple Examples
Learn a distance metric to improve classification performance on the iris
dataset, and show improved performance when using
[`NaiveBayesClassifier`](naive_bayes_classifier.md).
```c++
// See https://datasets.mlpack.org/satellite.test.csv.
// (We are using the test set here just because it is a little smaller and
// we want this example to run quickly.)
arma::mat dataset;
mlpack::data::Load("satellite.test.csv", dataset, true);
// See https://datasets.mlpack.org/satellite.test.labels.csv.
arma::Row<size_t> labels;
mlpack::data::Load("satellite.test.labels.csv", labels, true);
// Create an LMNN object using 5 nearest neighbors and learn a distance.
arma::mat distance;
mlpack::LMNN lmnn(5);
lmnn.LearnDistance(dataset, labels, distance);
// The distance matrix has size equal to the dimensionality of the data.
std::cout << "Learned distance size: " << distance.n_rows << " x "
<< distance.n_cols << "." << std::endl;
// Learn a NaiveBayesClassifier model on the data and print the performance.
mlpack::NaiveBayesClassifier nbc1(dataset, labels, 2);
arma::Row<size_t> predictions;
nbc1.Classify(dataset, predictions);
std::cout << "Naive Bayes Classifier without LMNN: "
<< arma::accu(labels == predictions) << " of " << labels.n_elem
<< " correct." << std::endl;
// Now transform the data and learn another NaiveBayesClassifier.
arma::mat transformedDataset = distance * dataset;
mlpack::NaiveBayesClassifier nbc2(transformedDataset, labels, 2);
nbc2.Classify(transformedDataset, predictions);
std::cout << "Naive Bayes Classifier with LMNN: "
<< arma::accu(labels == predictions) << " of " << labels.n_elem
<< " correct." << std::endl;
```
---
Learn a distance metric on the vehicle dataset, using 32-bit floating point to
represent the data and metric.
```c++
// See https://datasets.mlpack.org/vehicle.csv.
arma::fmat dataset;
mlpack::data::Load("vehicle.csv", dataset, true);
// The labels are contained as the last row of the dataset.
arma::Row<size_t> labels =
arma::conv_to<arma::Row<size_t>>::from(dataset.row(dataset.n_rows - 1));
dataset.shed_row(dataset.n_rows - 1);
// Create an LMNN object with k=1 and learn distance on float32 data.
// Set updateInterval to a large value (100) because we are using the default
// AMSGrad optimizer (which will take very many small steps).
arma::fmat distance;
mlpack::LMNN lmnn(1, 0.5, 100);
lmnn.LearnDistance(dataset, labels, distance, ens::ProgressBar());
// We want to compute six quantities:
//
// - Average distance to points of the same class before LMNN.
// - Average distance to points of the same class after LMNN, using
// MahalanobisDistance.
// - Average distance to points of the same class after LMNN, using the
// transformed dataset.
//
// - The same three quantities above, but for points of the other class.
//
// LMNN should reduce the average distance to points in the same class, while
// increasing the average distance to points in other classes.
float distSums[6] = { 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f };
size_t sameCount = 0;
arma::fmat q = distance.t() * distance;
mlpack::MahalanobisDistance md(std::move(q));
arma::fmat transformedDataset = distance * dataset;
for (size_t i = 1; i < dataset.n_cols; ++i)
{
const double d1 = mlpack::EuclideanDistance::Evaluate(
dataset.col(0), dataset.col(i));
const double d2 = md.Evaluate(dataset.col(0), dataset.col(i));
const double d3 = mlpack::EuclideanDistance::Evaluate(
transformedDataset.col(0), transformedDataset.col(i));
// Determine whether the point has the same label as point 0.
if (labels[i] == labels[0])
{
distSums[0] += d1;
distSums[1] += d2;
distSums[2] += d3;
++sameCount;
}
else
{
distSums[3] += d1;
distSums[4] += d2;
distSums[5] += d3;
}
}
// Turn the results into average distances across the class.
distSums[0] /= sameCount;
distSums[1] /= sameCount;
distSums[2] /= sameCount;
distSums[3] /= (dataset.n_cols - sameCount);
distSums[4] /= (dataset.n_cols - sameCount);
distSums[5] /= (dataset.n_cols - sameCount);
// Print the results.
std::cout << "Average distance between point 0 and other points of the same "
<< "class:" << std::endl;
std::cout << " - Before LMNN: " << distSums[0] << "."
<< std::endl;
std::cout << " - After LMNN (with MahalanobisDistance): " << distSums[1] << "."
<< std::endl;
std::cout << " - After LMNN (with transformed dataset): " << distSums[2] << "."
<< std::endl;
std::cout << std::endl;
std::cout << "Average distance between point 0 and points of other classes: "
<< std::endl;
std::cout << " - Before LMNN: " << distSums[3] << "."
<< std::endl;
std::cout << " - After LMNN (with MahalanobisDistance): " << distSums[4] << "."
<< std::endl;
std::cout << " - After LMNN (with transformed dataset): " << distSums[5] << "."
<< std::endl;
std::cout << std::endl;
std::cout << "Ratio of other-class to same-class distances:" << std::endl;
std::cout << "(We expect this to go up.)" << std::endl;
std::cout << " - Before LMNN: " << (distSums[3] / distSums[0]) << "."
<< std::endl;
std::cout << " - After LMNN: " << (distSums[5] / distSums[2]) << "."
<< std::endl;
```
---
Learn a distance metric on the iris dataset, using the L-BFGS optimizer with
callbacks.
```c++
// See https://datasets.mlpack.org/iris.csv.
arma::mat dataset;
mlpack::data::Load("iris.csv", dataset, true);
// See https://datasets.mlpack.org/iris.labels.csv.
arma::Row<size_t> labels;
mlpack::data::Load("iris.labels.csv", labels, true);
// Learn a distance with ensmallen's L-BFGS optimizer.
ens::L_BFGS lbfgs;
lbfgs.NumBasis() = 5;
lbfgs.MaxIterations() = 1000;
// Use 5 neighbors for LMNN, and leave updateInterval at the default of 1,
// because we are using L-BFGS (a full-back optimizer).
mlpack::LMNN lmnn(5);
// Use a callback that prints a final optimization report.
arma::mat distance;
lmnn.LearnDistance(dataset, labels, distance, lbfgs, ens::Report());
```
---
Learn a distance metric on the vehicle dataset, but instead of using the
Euclidean distance as the underlying metric, use the Manhattan distance. This
means that LMNN is optimizing k-NN performance under the Manhattan distance, not
under the Euclidean distance.
```c++
// See https://datasets.mlpack.org/vehicle.csv.
arma::mat dataset;
mlpack::data::Load("vehicle.csv", dataset, true);
// The labels are contained as the last row of the dataset.
arma::Row<size_t> labels =
arma::conv_to<arma::Row<size_t>>::from(dataset.row(dataset.n_rows - 1));
dataset.shed_row(dataset.n_rows - 1);
// Create the LMNN object and optimize. Use k=3 and Nesterov momentum SGD,
// printing a progress bar during optimization. Because Nesterov momentum SGD
// is an ensmallen optimizer for differentiable separable functions, we increase
// updateInterval to reduce the number of neighbor recomputations. We also set
// the regularization parameter to 1.0 to increase the penalty for nearby
// neighbors of a different class.
mlpack::LMNN<mlpack::ManhattanDistance> lmnn(3, 1.0, 100);
arma::mat distance;
ens::NesterovMomentumSGD opt(0.000001 /* step size */,
32 /* batch size */,
20 * dataset.n_cols /* 20 epochs */);
lmnn.LearnDistance(dataset, labels, distance, opt, ens::ProgressBar());
// Now inspect distances between points with the Euclidean distance and with the
// inner product distance.
arma::mat transformedDataset = distance * dataset;
// Points 0 and 1 have the same label (0). See their original distance---with
// both the Euclidean and Manhattan distances---and their transformed distances.
// We expect these points to get closer together, in the Manhattan distance.
const double d1 = mlpack::ManhattanDistance::Evaluate(
dataset.col(0), dataset.col(1));
const double d2 = mlpack::ManhattanDistance::Evaluate(
transformedDataset.col(0), transformedDataset.col(1));
std::cout << "Distance between points 0 and 1 (same class):" << std::endl;
std::cout << " - Manhattan distance:" << std::endl;
std::cout << " * Before LMNN: " << d1 << std::endl;
std::cout << " * After LMNN: " << d2 << std::endl;
std::cout << std::endl;
// Point 3 has a different label. We therefore expect this point to get further
// from point 0 with the Manhattan distance, but not necessarily with the
// Euclidean distance.
const double d3 = mlpack::ManhattanDistance::Evaluate(
dataset.col(0), dataset.col(3));
const double d4 = mlpack::ManhattanDistance::Evaluate(
transformedDataset.col(0), transformedDataset.col(3));
std::cout << "Distance between points 0 and 3 (different class):" << std::endl;
std::cout << " - Manhattan distance:" << std::endl;
std::cout << " * Before LMNN: " << d3 << std::endl;
std::cout << " * After LMNN: " << d4 << std::endl;
// Note that point 3 has been moved further away from point 0 than point 1.
```
---
Learn a distance metric while also performing dimensionality reduction, reducing
the dimensionality of the satellite dataset by 3 dimensions.
```c++
// See https://datasets.mlpack.org/satellite.train.csv.
arma::mat dataset;
mlpack::data::Load("satellite.train.csv", dataset, true);
// See https://datasets.mlpack.org/satellite.labels.csv.
arma::Row<size_t> labels;
mlpack::data::Load("satellite.train.labels.csv", labels, true);
// Use a random initialization for the distance transformation, with the
// specified output dimensionality.
arma::mat distance(dataset.n_rows - 3, dataset.n_rows, arma::fill::randu);
mlpack::LMNN lmnn(3);
ens::L_BFGS opt;
opt.MaxIterations() = 10; // You may want more in a real application.
lmnn.LearnDistance(dataset, labels, distance, opt, ens::Report());
// Now transform the dataset.
arma::mat transformedData = distance * dataset;
std::cout << "Original data has size " << dataset.n_rows << " x "
<< dataset.n_cols << "." << std::endl;
std::cout << "Transformed data has size " << transformedData.n_rows << " x "
<< transformedData.n_cols << "." << std::endl;
```
src/mlpack/methods/lmnn/constraints.hpp
+60 -47
View File
@@ -27,12 +27,25 @@ namespace mlpack {
* data point) and Triplets() (Generates sets of {dataset, target neighbors,
* impostors} tripltets.)
*/
template<typename DistanceType = SquaredEuclideanDistance>
template<typename MatType = arma::mat,
typename LabelsType = arma::Row<size_t>,
typename DistanceType = SquaredEuclideanDistance>
class Constraints
{
public:
//! Convenience typedef.
typedef NeighborSearch<NearestNeighborSort, DistanceType> KNN;
typedef NeighborSearch<NearestNeighborSort, DistanceType, MatType> KNN;
// Convenience typedef for element type of data.
typedef typename MatType::elem_type ElemType;
// Convenience typedef for column vector of data.
typedef typename GetColType<MatType>::type VecType;
// Convenience typedef for cube of data.
typedef typename GetCubeType<MatType>::type CubeType;
// Convenience typedef for dense matrix of indices.
typedef typename GetUDenseMatType<MatType>::type UMatType;
// Convenience typedef for dense vector of indices.
typedef typename GetColType<UMatType>::type UVecType;
/**
* Constructor for creating a Constraints instance.
@@ -41,8 +54,8 @@ class Constraints
* @param labels Input dataset labels.
* @param k Number of target neighbors, impostors & triplets.
*/
Constraints(const arma::mat& dataset,
const arma::Row<size_t>& labels,
Constraints(const MatType& dataset,
const LabelsType& labels,
const size_t k);
/**
@@ -54,10 +67,10 @@ class Constraints
* @param labels Input dataset labels.
* @param norms Input dataset norms.
*/
void TargetNeighbors(arma::Mat<size_t>& outputMatrix,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms);
void TargetNeighbors(UMatType& outputMatrix,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms);
/**
* Calculates k similar labeled nearest neighbors for a batch of dataset and
@@ -70,10 +83,10 @@ class Constraints
* @param begin Index of the initial point of dataset.
* @param batchSize Number of data points to use.
*/
void TargetNeighbors(arma::Mat<size_t>& outputMatrix,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms,
void TargetNeighbors(UMatType& outputMatrix,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms,
const size_t begin,
const size_t batchSize);
@@ -86,10 +99,10 @@ class Constraints
* @param labels Input dataset labels.
* @param norms Input dataset norms.
*/
void Impostors(arma::Mat<size_t>& outputMatrix,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms);
void Impostors(UMatType& outputMatrix,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms);
/**
* Calculates k differently labeled nearest neighbors & distances to
@@ -101,11 +114,11 @@ class Constraints
* @param labels Input dataset labels.
* @param norms Input dataset norms.
*/
void Impostors(arma::Mat<size_t>& outputNeighbors,
arma::mat& outputDistance,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms);
void Impostors(UMatType& outputNeighbors,
MatType& outputDistance,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms);
/**
* Calculates k differently labeled nearest neighbors for a batch of dataset
@@ -118,10 +131,10 @@ class Constraints
* @param begin Index of the initial point of dataset.
* @param batchSize Number of data points to use.
*/
void Impostors(arma::Mat<size_t>& outputMatrix,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms,
void Impostors(UMatType& outputMatrix,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms,
const size_t begin,
const size_t batchSize);
@@ -137,11 +150,11 @@ class Constraints
* @param begin Index of the initial point of dataset.
* @param batchSize Number of data points to use.
*/
void Impostors(arma::Mat<size_t>& outputNeighbors,
arma::mat& outputDistance,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms,
void Impostors(UMatType& outputNeighbors,
MatType& outputDistance,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms,
const size_t begin,
const size_t batchSize);
@@ -158,12 +171,12 @@ class Constraints
* @param points Indices of data points to calculate impostors on.
* @param numPoints Number of points to actually calculate impostors on.
*/
void Impostors(arma::Mat<size_t>& outputNeighbors,
arma::mat& outputDistance,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms,
const arma::uvec& points,
void Impostors(UMatType& outputNeighbors,
MatType& outputDistance,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms,
const UVecType& points,
const size_t numPoints);
/**
@@ -175,10 +188,10 @@ class Constraints
* @param labels Input dataset labels.
* @param norms Input dataset norms.
*/
void Triplets(arma::Mat<size_t>& outputMatrix,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms);
void Triplets(UMatType& outputMatrix,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms);
//! Get the number of target neighbors (k).
const size_t& K() const { return k; }
@@ -195,13 +208,13 @@ class Constraints
size_t k;
//! Store unique labels.
arma::Row<size_t> uniqueLabels;
LabelsType uniqueLabels;
//! Store indices of data points having similar label.
std::vector<arma::uvec> indexSame;
std::vector<UVecType> indexSame;
//! Store indices of data points having different label.
std::vector<arma::uvec> indexDiff;
std::vector<UVecType> indexDiff;
//! False if nothing has ever been precalculated.
bool precalculated;
@@ -210,15 +223,15 @@ class Constraints
* Precalculate the unique labels, and indices of similar
* and different datapoints on the basis of labels.
*/
inline void Precalculate(const arma::Row<size_t>& labels);
inline void Precalculate(const LabelsType& labels);
/**
* Re-order neighbors on the basis of increasing norm in case
* of ties among distances.
*/
inline void ReorderResults(const arma::mat& distances,
arma::Mat<size_t>& neighbors,
const arma::vec& norms);
inline void ReorderResults(const MatType& distances,
UMatType& neighbors,
const VecType& norms);
};
} // namespace mlpack
src/mlpack/methods/lmnn/constraints_impl.hpp
+107 -99
View File
@@ -17,10 +17,10 @@
namespace mlpack {
template<typename DistanceType>
Constraints<DistanceType>::Constraints(
const arma::mat& /* dataset */,
const arma::Row<size_t>& labels,
template<typename MatType, typename LabelsType, typename DistanceType>
Constraints<MatType, LabelsType, DistanceType>::Constraints(
const MatType& /* dataset */,
const LabelsType& labels,
const size_t k) :
k(k),
precalculated(false)
@@ -36,11 +36,11 @@ Constraints<DistanceType>::Constraints(
}
}
template<typename DistanceType>
inline void Constraints<DistanceType>::ReorderResults(
const arma::mat& distances,
arma::Mat<size_t>& neighbors,
const arma::vec& norms)
template<typename MatType, typename LabelsType, typename DistanceType>
inline void Constraints<MatType, LabelsType, DistanceType>::ReorderResults(
const MatType& distances,
UMatType& neighbors,
const VecType& norms)
{
// Shortcut...
if (neighbors.n_rows == 1)
@@ -64,24 +64,21 @@ inline void Constraints<DistanceType>::ReorderResults(
if (start != end)
{
// We must sort these elements by norm.
arma::Col<size_t> newNeighbors =
neighbors.col(i).subvec(start, end - 1);
arma::uvec indices = ConvTo<arma::uvec>::From(newNeighbors);
arma::uvec order = arma::sort_index(norms.elem(indices));
neighbors.col(i).subvec(start, end - 1) =
newNeighbors.elem(order);
UVecType indices = neighbors.col(i).subvec(start, end - 1);
UVecType order = arma::sort_index(norms.elem(indices));
neighbors.col(i).subvec(start, end - 1) = indices.elem(order);
}
}
}
}
// Calculates k similar labeled nearest neighbors.
template<typename DistanceType>
void Constraints<DistanceType>::TargetNeighbors(arma::Mat<size_t>& outputMatrix,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms)
template<typename MatType, typename LabelsType, typename DistanceType>
void Constraints<MatType, LabelsType, DistanceType>::TargetNeighbors(
UMatType& outputMatrix,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms)
{
// Perform pre-calculation. If neccesary.
Precalculate(labels);
@@ -89,8 +86,8 @@ void Constraints<DistanceType>::TargetNeighbors(arma::Mat<size_t>& outputMatrix,
// KNN instance.
KNN knn;
arma::Mat<size_t> neighbors;
arma::mat distances;
UMatType neighbors;
MatType distances;
for (size_t i = 0; i < uniqueLabels.n_cols; ++i)
{
@@ -114,28 +111,29 @@ void Constraints<DistanceType>::TargetNeighbors(arma::Mat<size_t>& outputMatrix,
// Calculates k similar labeled nearest neighbors on a
// batch of data points.
template<typename DistanceType>
void Constraints<DistanceType>::TargetNeighbors(arma::Mat<size_t>& outputMatrix,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms,
const size_t begin,
const size_t batchSize)
template<typename MatType, typename LabelsType, typename DistanceType>
void Constraints<MatType, LabelsType, DistanceType>::TargetNeighbors(
UMatType& outputMatrix,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms,
const size_t begin,
const size_t batchSize)
{
// Perform pre-calculation. If neccesary.
Precalculate(labels);
arma::mat subDataset = dataset.cols(begin, begin + batchSize - 1);
arma::Row<size_t> sublabels = labels.cols(begin, begin + batchSize - 1);
MatType subDataset = dataset.cols(begin, begin + batchSize - 1);
LabelsType sublabels = labels.cols(begin, begin + batchSize - 1);
// KNN instance.
KNN knn;
arma::Mat<size_t> neighbors;
arma::mat distances;
UMatType neighbors;
MatType distances;
// Vectors to store indices.
arma::uvec subIndexSame;
UVecType subIndexSame;
for (size_t i = 0; i < uniqueLabels.n_cols; ++i)
{
@@ -161,11 +159,12 @@ void Constraints<DistanceType>::TargetNeighbors(arma::Mat<size_t>& outputMatrix,
}
// Calculates k differently labeled nearest neighbors.
template<typename DistanceType>
void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputMatrix,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms)
template<typename MatType, typename LabelsType, typename DistanceType>
void Constraints<MatType, LabelsType, DistanceType>::Impostors(
UMatType& outputMatrix,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms)
{
// Perform pre-calculation. If neccesary.
Precalculate(labels);
@@ -173,8 +172,8 @@ void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputMatrix,
// KNN instance.
KNN knn;
arma::Mat<size_t> neighbors;
arma::mat distances;
UMatType neighbors;
MatType distances;
for (size_t i = 0; i < uniqueLabels.n_cols; ++i)
{
@@ -198,12 +197,13 @@ void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputMatrix,
// Calculates k differently labeled nearest neighbors. The function
// writes back calculated neighbors & distances to passed matrices.
template<typename DistanceType>
void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputNeighbors,
arma::mat& outputDistance,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms)
template<typename MatType, typename LabelsType, typename DistanceType>
void Constraints<MatType, LabelsType, DistanceType>::Impostors(
UMatType& outputNeighbors,
MatType& outputDistance,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms)
{
// Perform pre-calculation. If neccesary.
Precalculate(labels);
@@ -211,8 +211,8 @@ void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputNeighbors,
// KNN instance.
KNN knn;
arma::Mat<size_t> neighbors;
arma::mat distances;
UMatType neighbors;
MatType distances;
for (size_t i = 0; i < uniqueLabels.n_cols; ++i)
{
@@ -237,28 +237,29 @@ void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputNeighbors,
// Calculates k differently labeled nearest neighbors on a
// batch of data points.
template<typename DistanceType>
void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputMatrix,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms,
const size_t begin,
const size_t batchSize)
template<typename MatType, typename LabelsType, typename DistanceType>
void Constraints<MatType, LabelsType, DistanceType>::Impostors(
UMatType& outputMatrix,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms,
const size_t begin,
const size_t batchSize)
{
// Perform pre-calculation. If neccesary.
Precalculate(labels);
arma::mat subDataset = dataset.cols(begin, begin + batchSize - 1);
arma::Row<size_t> sublabels = labels.cols(begin, begin + batchSize - 1);
MatType subDataset = dataset.cols(begin, begin + batchSize - 1);
LabelsType sublabels = labels.cols(begin, begin + batchSize - 1);
// KNN instance.
KNN knn;
arma::Mat<size_t> neighbors;
arma::mat distances;
UMatType neighbors;
MatType distances;
// Vectors to store indices.
arma::uvec subIndexSame;
UVecType subIndexSame;
for (size_t i = 0; i < uniqueLabels.n_cols; ++i)
{
@@ -285,29 +286,30 @@ void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputMatrix,
// Calculates k differently labeled nearest neighbors & distances on a
// batch of data points.
template<typename DistanceType>
void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputNeighbors,
arma::mat& outputDistance,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms,
const size_t begin,
const size_t batchSize)
template<typename MatType, typename LabelsType, typename DistanceType>
void Constraints<MatType, LabelsType, DistanceType>::Impostors(
UMatType& outputNeighbors,
MatType& outputDistance,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms,
const size_t begin,
const size_t batchSize)
{
// Perform pre-calculation. If neccesary.
Precalculate(labels);
arma::mat subDataset = dataset.cols(begin, begin + batchSize - 1);
arma::Row<size_t> sublabels = labels.cols(begin, begin + batchSize - 1);
MatType subDataset = dataset.cols(begin, begin + batchSize - 1);
LabelsType sublabels = labels.cols(begin, begin + batchSize - 1);
// KNN instance.
KNN knn;
arma::Mat<size_t> neighbors;
arma::mat distances;
UMatType neighbors;
MatType distances;
// Vectors to store indices.
arma::uvec subIndexSame;
UVecType subIndexSame;
for (size_t i = 0; i < uniqueLabels.n_cols; ++i)
{
@@ -335,14 +337,15 @@ void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputNeighbors,
// Calculates k differently labeled nearest neighbors & distances over some
// data points.
template<typename DistanceType>
void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputNeighbors,
arma::mat& outputDistance,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms,
const arma::uvec& points,
const size_t numPoints)
template<typename MatType, typename LabelsType, typename DistanceType>
void Constraints<MatType, LabelsType, DistanceType>::Impostors(
UMatType& outputNeighbors,
MatType& outputDistance,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms,
const UVecType& points,
const size_t numPoints)
{
// Perform pre-calculation. If neccesary.
Precalculate(labels);
@@ -350,11 +353,11 @@ void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputNeighbors,
// KNN instance.
KNN knn;
arma::Mat<size_t> neighbors;
arma::mat distances;
UMatType neighbors;
MatType distances;
// Vectors to store indices.
arma::uvec subIndexSame;
UVecType subIndexSame;
for (size_t i = 0; i < uniqueLabels.n_cols; ++i)
{
@@ -384,31 +387,35 @@ void Constraints<DistanceType>::Impostors(arma::Mat<size_t>& outputNeighbors,
// Generates {data point, target neighbors, impostors} triplets using
// TargetNeighbors() and Impostors().
template<typename DistanceType>
void Constraints<DistanceType>::Triplets(arma::Mat<size_t>& outputMatrix,
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const arma::vec& norms)
template<typename MatType, typename LabelsType, typename DistanceType>
void Constraints<MatType, LabelsType, DistanceType>::Triplets(
UMatType& outputMatrix,
const MatType& dataset,
const LabelsType& labels,
const VecType& norms)
{
// Perform pre-calculation. If neccesary.
Precalculate(labels);
size_t N = dataset.n_cols;
arma::Mat<size_t> impostors(k, dataset.n_cols);
UMatType impostors(k, dataset.n_cols);
Impostors(impostors, dataset, labels, norms);
arma::Mat<size_t> targetNeighbors(k, dataset.n_cols);;
UMatType targetNeighbors(k, dataset.n_cols);;
TargetNeighbors(targetNeighbors, dataset, labels, norms);
outputMatrix = arma::Mat<size_t>(3, k * k * N , arma::fill::zeros);
outputMatrix = UMatType(3, k * k * N , arma::fill::zeros);
for (size_t i = 0, r = 0; i < N; ++i)
#pragma omp parallel for collapse(3)
for (size_t i = 0; i < N; ++i)
{
for (size_t j = 0; j < k; ++j)
{
for (size_t l = 0; l < k; l++, r++)
for (size_t l = 0; l < k; l++)
{
const size_t r = i * (k * k) + j * k + l;
// Generate triplets.
outputMatrix(0, r) = i;
outputMatrix(1, r) = targetNeighbors(j, i);
@@ -418,9 +425,9 @@ void Constraints<DistanceType>::Triplets(arma::Mat<size_t>& outputMatrix,
}
}
template<typename DistanceType>
inline void Constraints<DistanceType>::Precalculate(
const arma::Row<size_t>& labels)
template<typename MatType, typename LabelsType, typename DistanceType>
inline void Constraints<MatType, LabelsType, DistanceType>::Precalculate(
const LabelsType& labels)
{
// Make sure the calculation is necessary.
if (precalculated)
@@ -431,6 +438,7 @@ inline void Constraints<DistanceType>::Precalculate(
indexSame.resize(uniqueLabels.n_elem);
indexDiff.resize(uniqueLabels.n_elem);
#pragma omp parallel for
for (size_t i = 0; i < uniqueLabels.n_elem; ++i)
{
// Store same and diff indices.
src/mlpack/methods/lmnn/lmnn.hpp
+118 -19
View File
@@ -14,6 +14,7 @@
#include <mlpack/core.hpp>
#include "../nca/first_element_is_arma.hpp"
#include "constraints.hpp"
#include "lmnn_function.hpp"
@@ -49,7 +50,7 @@ namespace mlpack {
* @tparam OptimizerType Optimizer to use for developing distance.
*/
template<typename DistanceType = SquaredEuclideanDistance,
typename OptimizerType = ens::AMSGrad>
typename DeprecatedOptimizerType = ens::AMSGrad>
class LMNN
{
public:
@@ -63,11 +64,27 @@ class LMNN
* @param k Number of targets to consider.
* @param distance Type of distance metric used for computation.
*/
[[deprecated("Will be removed in mlpack 5.0.0. Pass the dataset directly to "
"LearnDistance() instead.")]]
LMNN(const arma::mat& dataset,
const arma::Row<size_t>& labels,
const size_t k,
const DistanceType distance = DistanceType());
/**
* Construct the LMNN object, optionally with an instantiated distance metric.
*
* @param k Number of target neighbors to consider.
* @param regularization Penalty to apply to objective function.
* @param updateInterval Number of iterations between each recomputation of
* true neighbors and impostors.
* @param distance Instantiated distance metric for computation.
*/
LMNN(const size_t k,
const double regularization = 0.5,
const size_t updateInterval = 1,
DistanceType distance = DistanceType());
/**
* Perform Large Margin Nearest Neighbors metric learning. The output
@@ -80,25 +97,99 @@ class LMNN
* @param callbacks Callback function for ensmallen optimizer `OptimizerType`.
* See https://www.ensmallen.org/docs.html#callback-documentation.
*/
template<typename... CallbackTypes>
template<typename... CallbackTypes,
typename = typename std::enable_if<IsEnsCallbackTypes<
CallbackTypes...
>::value>::type,
typename = typename std::enable_if<
!FirstElementIsArma<CallbackTypes...>::value
>::type>
[[deprecated("Will be removed in mlpack 5.0.0. Use the version that takes a "
"dataset as a parameter.")]]
void LearnDistance(arma::mat& outputMatrix, CallbackTypes&&... callbacks);
/**
* Perform Large Margin Nearest Neighbors metric learning. The output
* distance matrix is written into the passed reference. If the
* LearnDistance() is called with an outputMatrix with correct dimensions,
* then that matrix will be used as the starting point for optimization.
*
* @param dataset Dataset to learn distance metric on.
* @param labels Labels for dataset.
* @param outputMatrix Covariance matrix of Mahalanobis distance.
* @param callbacks Callback function for ensmallen optimizer `OptimizerType`.
* See https://www.ensmallen.org/docs.html#callback-documentation.
*/
template<typename MatType,
typename LabelsType,
typename... CallbackTypes,
typename = typename std::enable_if<!IsEnsOptimizer<
typename First<CallbackTypes...>::type,
LMNNFunction<MatType, LabelsType, DistanceType>,
MatType
>::value>::type,
typename = typename std::enable_if<IsEnsCallbackTypes<
CallbackTypes...
>::value>::type>
void LearnDistance(const MatType& dataset,
const LabelsType& labels,
MatType& outputMatrix,
CallbackTypes&&... callbacks) const;
/**
* Perform Large Margin Nearest Neighbors metric learning. The output
* distance matrix is written into the passed reference. If the
* LearnDistance() is called with an outputMatrix with correct dimensions,
* then that matrix will be used as the starting point for optimization.
*
* @param dataset Dataset to learn distance metric on.
* @param labels Labels for dataset.
* @param optimizer Instantiated ensmallen optimizer to use for LMNN.
* @param outputMatrix Covariance matrix of Mahalanobis distance.
* @param callbacks Callback function for ensmallen optimizer `OptimizerType`.
* See https://www.ensmallen.org/docs.html#callback-documentation.
*/
template<typename MatType,
typename LabelsType,
typename OptimizerType,
typename... CallbackTypes,
typename = typename std::enable_if<IsEnsOptimizer<
OptimizerType,
LMNNFunction<MatType, LabelsType, DistanceType>,
MatType
>::value>::type>
void LearnDistance(const MatType& dataset,
const LabelsType& labels,
MatType& outputMatrix,
OptimizerType& optimizer,
CallbackTypes&&... callbacks) const;
//! Get the dataset reference.
const arma::mat& Dataset() const { return dataset; }
[[deprecated("Will be removed in mlpack 5.0.0. Use the LearnDistance() "
"version that takes the optimizer as a parameter instead.")]]
const arma::mat& Dataset() const { return *dataset; }
//! Get the labels reference.
const arma::Row<size_t>& Labels() const { return labels; }
[[deprecated("Will be removed in mlpack 5.0.0. Use the LearnDistance() "
"version that takes the optimizer as a parameter instead.")]]
const arma::Row<size_t>& Labels() const { return *labels; }
//! Access the regularization value.
const double& Regularization() const { return regularization; }
//! Modify the regularization value.
double& Regularization() { return regularization; }
//! Access the range value.
const size_t& Range() const { return range; }
//! Modify the range value.
size_t& Range() { return range; }
//! Access the iteration update interval value.
const size_t& UpdateInterval() const { return updateInterval; }
//! Modify the iteration update interval value.
size_t& UpdateInterval() { return updateInterval; }
[[deprecated("Will be removed in mlpack 5.0.0. Use UpdateInterval() "
"instead.")]]
const size_t& Range() const { return updateInterval; }
[[deprecated("Will be removed in mlpack 5.0.0. Use UpdateInterval() "
"instead.")]]
size_t& Range() { return updateInterval; }
//! Access the value of k.
const size_t& K() const { return k; }
@@ -106,15 +197,23 @@ class LMNN
size_t K() { return k; }
//! Get the optimizer.
const OptimizerType& Optimizer() const { return optimizer; }
OptimizerType& Optimizer() { return optimizer; }
[[deprecated("Will be removed in mlpack 5.0.0. Use the LearnDistance() "
"version that takes the optimizer as a parameter instead.")]]
const DeprecatedOptimizerType& Optimizer() const { return optimizer; }
//! Modify the optimizer.
[[deprecated("Will be removed in mlpack 5.0.0. Use the LearnDistance() "
"version that takes the optimizer as a parameter instead.")]]
DeprecatedOptimizerType& Optimizer() { return optimizer; }
// Serialize the LMNN object.
template<typename Archive>
void serialize(Archive& ar, const unsigned int /* version */);
private:
//! Dataset reference.
const arma::mat& dataset;
//! Labels reference.
const arma::Row<size_t>& labels;
//! Dataset pointer (will be removed in mlpack 5.0.0).
const arma::mat* dataset;
//! Labels pointer (will be removed in mlpack 5.0.0).
const arma::Row<size_t>* labels;
//! Number of target points.
size_t k;
@@ -122,14 +221,14 @@ class LMNN
//! Regularization value.
double regularization;
//! Range after which impostors need to be recalculated.
size_t range;
//! Number of iterations after which impostors need to be recalculated.
size_t updateInterval;
//! Distance to be used.
DistanceType distance;
//! The optimizer to use.
OptimizerType optimizer;
//! The optimizer to use (will be removed in mlpack 5.0.0).
DeprecatedOptimizerType optimizer;
}; // class LMNN
} // namespace mlpack
src/mlpack/methods/lmnn/lmnn_function.hpp
+59 -46
View File
@@ -41,9 +41,22 @@ namespace mlpack {
* operate on one point in the dataset. This is useful for optimizers like
* stochastic gradient descent (see ens::SGD).
*/
template<typename DistanceType = SquaredEuclideanDistance>
template<typename MatType = arma::mat,
typename LabelsType = arma::Row<size_t>,
typename DistanceType = SquaredEuclideanDistance>
class LMNNFunction
{
// Convenience typedef for element type of data.
typedef typename MatType::elem_type ElemType;
// Convenience typedef for column vector of data.
typedef typename GetColType<MatType>::type VecType;
// Convenience typedef for cube of data.
typedef typename GetCubeType<MatType>::type CubeType;
// Convenience typedef for dense matrix of indices.
typedef typename GetUDenseMatType<MatType>::type UMatType;
// Convenience typedef for dense vector of indices.
typedef typename GetColType<UMatType>::type UVecType;
public:
/**
* Constructor for LMNNFunction class.
@@ -52,14 +65,14 @@ class LMNNFunction
* @param labels Input dataset labels.
* @param k Number of target neighbors to be used.
* @param regularization Regularization value.
* @param range Range after which impostors need to be recalculated.
* @param updateInterval Number of iterations before impostors are recomputed.
* @param distance Type of distance metric used for computation.
*/
LMNNFunction(const arma::mat& dataset,
const arma::Row<size_t>& labels,
LMNNFunction(const MatType& dataset,
const LabelsType& labels,
size_t k,
double regularization,
size_t range,
size_t updateInterval,
DistanceType distance = DistanceType());
@@ -69,13 +82,13 @@ class LMNNFunction
void Shuffle();
/**
* Evaluate the LMNN function for the given transformation matrix. This is the
* non-separable implementation, where the objective function is not
* Evaluate the LMNN function for the given transformation matrix. This is
* the non-separable implementation, where the objective function is not
* decomposed into the sum of several objective functions.
*
* @param transformation Transformation matrix of Mahalanobis distance.
*/
double Evaluate(const arma::mat& transformation);
ElemType Evaluate(const MatType& transformation);
/**
* Evaluate the LMNN objective function for the given transformation matrix on
@@ -89,9 +102,9 @@ class LMNNFunction
* @param begin Index of the initial point to use for objective function.
* @param batchSize Number of points to use for objective function.
*/
double Evaluate(const arma::mat& transformation,
const size_t begin,
const size_t batchSize = 1);
ElemType Evaluate(const MatType& transformation,
const size_t begin,
const size_t batchSize = 1);
/**
* Evaluate the gradient of the LMNN function for the given transformation
@@ -103,7 +116,7 @@ class LMNNFunction
* @param gradient Matrix to store the calculated gradient in.
*/
template<typename GradType>
void Gradient(const arma::mat& transformation, GradType& gradient);
void Gradient(const MatType& transformation, GradType& gradient);
/**
* Evaluate the gradient of the LMNN function for the given transformation
@@ -121,7 +134,7 @@ class LMNNFunction
* @param batchSize Number of points to use for objective function.
*/
template<typename GradType>
void Gradient(const arma::mat& transformation,
void Gradient(const MatType& transformation,
const size_t begin,
GradType& gradient,
const size_t batchSize = 1);
@@ -137,8 +150,8 @@ class LMNNFunction
* @param gradient Matrix to store the calculated gradient in.
*/
template<typename GradType>
double EvaluateWithGradient(const arma::mat& transformation,
GradType& gradient);
ElemType EvaluateWithGradient(const MatType& transformation,
GradType& gradient);
/**
* Evaluate the LMNN objective function together with gradient for the given
@@ -156,13 +169,13 @@ class LMNNFunction
* @param batchSize Number of points to use for objective function.
*/
template<typename GradType>
double EvaluateWithGradient(const arma::mat& transformation,
const size_t begin,
GradType& gradient,
const size_t batchSize = 1);
ElemType EvaluateWithGradient(const MatType& transformation,
const size_t begin,
GradType& gradient,
const size_t batchSize = 1);
//! Return the initial point for the optimization.
const arma::mat& GetInitialPoint() const { return initialPoint; }
const MatType& GetInitialPoint() const { return initialPoint; }
/**
* Get the number of functions the objective function can be decomposed into.
@@ -171,7 +184,7 @@ class LMNNFunction
size_t NumFunctions() const { return dataset.n_cols; }
//! Return the dataset passed into the constructor.
const arma::mat& Dataset() const { return dataset; }
const MatType& Dataset() const { return dataset; }
//! Access the regularization value.
const double& Regularization() const { return regularization; }
@@ -183,26 +196,26 @@ class LMNNFunction
//! Modify the value of k.
size_t& K() { return k; }
//! Access the value of range.
const size_t& Range() const { return range; }
//! Modify the value of k.
size_t& Range() { return range; }
//! Access the number of iterations between impostor recomputation.
const size_t& UpdateInterval() const { return updateInterval; }
//! Modify the number of iterations between impostor recomputation..
size_t& UpdateInterval() { return updateInterval; }
private:
//! data. This will be an alias until Shuffle() is called.
arma::mat dataset;
MatType dataset;
//! labels. This will be an alias until Shuffle() is called.
arma::Row<size_t> labels;
LabelsType labels;
//! Initial parameter point.
arma::mat initialPoint;
MatType initialPoint;
//! Store transformed dataset.
arma::mat transformedDataset;
MatType transformedDataset;
//! Store target neighbors of data points.
arma::Mat<size_t> targetNeighbors;
UMatType targetNeighbors;
//! Initial impostors.
arma::Mat<size_t> impostors;
UMatType impostors;
//! Cache distance. Used to avoid repetive calculation.
arma::mat distanceMat;
MatType distanceMat;
//! Number of target neighbors.
size_t k;
//! The instantiated distance metric.
@@ -211,28 +224,28 @@ class LMNNFunction
double regularization;
//! Keep iterations count.
size_t iteration;
//! Range after which impostors need to be recalculated.
size_t range;
//! Number of iterations before impostors need to be recalculated.
size_t updateInterval;
//! Constraints Object.
Constraints<DistanceType> constraint;
Constraints<MatType, LabelsType, DistanceType> constraint;
//! Holds pre-calculated cij.
arma::mat pCij;
MatType pCij;
//! Holds the norm of each data point.
arma::vec norm;
VecType norm;
//! Hold previous eval values for each datapoint.
arma::cube evalOld;
CubeType evalOld;
//! Hold previous maximum norm of impostor.
arma::mat maxImpNorm;
MatType maxImpNorm;
//! Holds previous transformation matrix. Used for L-BFGS like optimizer.
arma::mat transformationOld;
MatType transformationOld;
//! Holds previous transformation matrices.
std::vector<arma::mat> oldTransformationMatrices;
std::vector<MatType> oldTransformationMatrices;
//! Holds number of points which are using each transformation matrix.
std::vector<size_t> oldTransformationCounts;
//! Holds points to transformation matrix mapping.
arma::vec lastTransformationIndices;
VecType lastTransformationIndices;
//! Used for storing points to re-calculate impostors for.
arma::uvec points;
UVecType points;
//! Flag for controlling use of bounds over impostors.
bool impBounds;
/**
@@ -242,12 +255,12 @@ class LMNNFunction
*/
inline void Precalculate();
//! Update cache transformation matrices.
inline void UpdateCache(const arma::mat& transformation,
inline void UpdateCache(const MatType& transformation,
const size_t begin,
const size_t batchSize);
//! Calculate norm of change in transformation.
inline void TransDiff(std::map<size_t, double>& transformationDiffs,
const arma::mat& transformation,
inline void TransDiff(std::unordered_map<size_t, ElemType>& transDiffs,
const MatType& transformation,
const size_t begin,
const size_t batchSize);
};
src/mlpack/methods/lmnn/lmnn_function_impl.hpp
+108 -118
View File
@@ -18,18 +18,19 @@
namespace mlpack {
template<typename DistanceType>
LMNNFunction<DistanceType>::LMNNFunction(const arma::mat& datasetIn,
const arma::Row<size_t>& labelsIn,
size_t k,
double regularization,
size_t range,
DistanceType distance) :
template<typename MatType, typename LabelsType, typename DistanceType>
LMNNFunction<MatType, LabelsType, DistanceType>::LMNNFunction(
const MatType& datasetIn,
const LabelsType& labelsIn,
size_t k,
double regularization,
size_t updateInterval,
DistanceType distance) :
k(k),
distance(distance),
regularization(regularization),
iteration(0),
range(range),
updateInterval(updateInterval),
constraint(datasetIn, labelsIn, k),
points(datasetIn.n_cols),
impBounds(false)
@@ -60,7 +61,7 @@ LMNNFunction<DistanceType>::LMNNFunction(const arma::mat& datasetIn,
lastTransformationIndices.zeros();
// Reserve the first element of cache.
arma::mat emptyMat;
MatType emptyMat;
oldTransformationMatrices.push_back(emptyMat);
oldTransformationCounts.push_back(dataset.n_cols);
@@ -92,18 +93,18 @@ LMNNFunction<DistanceType>::LMNNFunction(const arma::mat& datasetIn,
}
//! Shuffle the dataset.
template<typename DistanceType>
void LMNNFunction<DistanceType>::Shuffle()
template<typename MatType, typename LabelsType, typename DistanceType>
void LMNNFunction<MatType, LabelsType, DistanceType>::Shuffle()
{
arma::mat newDataset = dataset;
arma::Mat<size_t> newLabels = labels;
arma::cube newEvalOld = evalOld;
arma::vec newlastTransformationIndices = lastTransformationIndices;
arma::mat newMaxImpNorm = maxImpNorm;
arma::vec newNorm = norm;
MatType newDataset = dataset;
LabelsType newLabels = labels;
CubeType newEvalOld = evalOld;
VecType newlastTransformationIndices = lastTransformationIndices;
MatType newMaxImpNorm = maxImpNorm;
VecType newNorm = norm;
// Generate ordering.
arma::uvec ordering = arma::shuffle(arma::linspace<arma::uvec>(0,
UVecType ordering = arma::shuffle(arma::linspace<UVecType>(0,
dataset.n_cols - 1, dataset.n_cols));
ClearAlias(dataset);
@@ -126,9 +127,9 @@ void LMNNFunction<DistanceType>::Shuffle()
}
// Update cache transformation matrices.
template<typename DistanceType>
inline void LMNNFunction<DistanceType>::UpdateCache(
const arma::mat& transformation,
template<typename MatType, typename LabelsType, typename DistanceType>
inline void LMNNFunction<MatType, LabelsType, DistanceType>::UpdateCache(
const MatType& transformation,
const size_t begin,
const size_t batchSize)
{
@@ -162,31 +163,13 @@ inline void LMNNFunction<DistanceType>::UpdateCache(
}
oldTransformationCounts[index] += batchSize;
#ifdef DEBUG
size_t total = 0;
for (size_t i = 1; i < oldTransformationCounts.size(); ++i)
{
std::ostringstream oss;
oss << "transformation counts for matrix " << i
<< " invalid (" << oldTransformationCounts[i] << ")!";
Log::Assert(oldTransformationCounts[i] <= dataset.n_cols, oss.str());
total += oldTransformationCounts[i];
}
std::ostringstream oss;
oss << "total count for transformation matrices invalid (" << total
<< ", " << "should be " << dataset.n_cols << "!";
if (begin + batchSize == dataset.n_cols)
Log::Assert(total == dataset.n_cols, oss.str());
#endif
}
// Calculate norm of change in transformation.
template<typename DistanceType>
inline void LMNNFunction<DistanceType>::TransDiff(
std::map<size_t, double>& transformationDiffs,
const arma::mat& transformation,
template<typename MatType, typename LabelsType, typename DistanceType>
inline void LMNNFunction<MatType, LabelsType, DistanceType>::TransDiff(
std::unordered_map<size_t, ElemType>& transformationDiffs,
const MatType& transformation,
const size_t begin,
const size_t batchSize)
{
@@ -209,22 +192,24 @@ inline void LMNNFunction<DistanceType>::TransDiff(
}
//! Evaluate cost over whole dataset.
template<typename DistanceType>
double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation)
template<typename MatType, typename LabelsType, typename DistanceType>
typename MatType::elem_type
LMNNFunction<MatType, LabelsType, DistanceType>::Evaluate(
const MatType& transformation)
{
double cost = 0;
ElemType cost = 0;
// Apply distance metric over dataset.
transformedDataset = transformation * dataset;
double transformationDiff = 0;
ElemType transformationDiff = 0;
if (!transformationOld.is_empty())
{
// Calculate norm of change in transformation.
transformationDiff = arma::norm(transformation - transformationOld);
}
if (!transformationOld.is_empty() && iteration++ % range == 0)
if (!transformationOld.is_empty() && iteration++ % updateInterval == 0)
{
if (impBounds)
{
@@ -251,7 +236,7 @@ double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation)
norm);
}
}
else if (iteration++ % range == 0)
else if (iteration++ % updateInterval == 0)
{
// Re-calculate impostors on transformed dataset.
constraint.Impostors(impostors, distanceMat, transformedDataset, labels,
@@ -263,7 +248,7 @@ double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation)
for (size_t j = 0; j < k ; ++j)
{
// Calculate cost due to distance between target neighbors & data point.
double eval = distance.Evaluate(transformedDataset.col(i),
ElemType eval = distance.Evaluate(transformedDataset.col(i),
transformedDataset.col(targetNeighbors(j, i)));
cost += (1 - regularization) * eval;
}
@@ -276,7 +261,7 @@ double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation)
{
// Calculate cost due to {data point, target neighbors, impostors}
// triplets.
double eval = 0;
ElemType eval = 0;
// Bounds for eval.
if (!transformationOld.is_empty() && evalOld(l, j, i) < -1)
@@ -292,7 +277,7 @@ double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation)
// Calculate exact eval value.
if (eval > -1)
{
if (iteration - 1 % range == 0)
if (iteration - 1 % updateInterval == 0)
{
eval = distance.Evaluate(transformedDataset.col(i),
transformedDataset.col(targetNeighbors(j, i))) -
@@ -338,21 +323,23 @@ double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation)
}
//! Calculate cost over batches.
template<typename DistanceType>
double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation,
const size_t begin,
const size_t batchSize)
template<typename MatType, typename LabelsType, typename DistanceType>
typename MatType::elem_type
LMNNFunction<MatType, LabelsType, DistanceType>::Evaluate(
const MatType& transformation,
const size_t begin,
const size_t batchSize)
{
double cost = 0;
ElemType cost = 0;
// Calculate norm of change in transformation.
std::map<size_t, double> transformationDiffs;
std::unordered_map<size_t, ElemType> transformationDiffs;
TransDiff(transformationDiffs, transformation, begin, batchSize);
// Apply distance metric over dataset.
transformedDataset = transformation * dataset;
if (impBounds && iteration++ % range == 0)
if (impBounds && iteration++ % updateInterval == 0)
{
// Track number of data points to use for impostors calculatiom.
size_t numPoints = 0;
@@ -378,7 +365,7 @@ double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation,
constraint.Impostors(impostors, distanceMat,
transformedDataset, labels, norm, points, numPoints);
}
else if (iteration++ % range == 0)
else if (iteration++ % updateInterval == 0)
{
// Re-calculate impostors on transformed dataset.
constraint.Impostors(impostors, distanceMat, transformedDataset, labels,
@@ -390,7 +377,7 @@ double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation,
for (size_t j = 0; j < k ; ++j)
{
// Calculate cost due to distance between target neighbors & data point.
double eval = distance.Evaluate(transformedDataset.col(i),
ElemType eval = distance.Evaluate(transformedDataset.col(i),
transformedDataset.col(targetNeighbors(j, i)));
cost += (1 - regularization) * eval;
}
@@ -403,7 +390,7 @@ double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation,
{
// Calculate cost due to {data point, target neighbors, impostors}
// triplets.
double eval = 0;
ElemType eval = 0;
// Bounds for eval.
if (lastTransformationIndices(i) && evalOld(l, j, i) < -1)
@@ -419,7 +406,7 @@ double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation,
// Calculate exact eval value.
if (eval > -1)
{
if (iteration - 1 % range == 0)
if (iteration - 1 % updateInterval == 0)
{
eval = distance.Evaluate(transformedDataset.col(i),
transformedDataset.col(targetNeighbors(j, i))) -
@@ -467,16 +454,16 @@ double LMNNFunction<DistanceType>::Evaluate(const arma::mat& transformation,
}
//! Compute gradient over whole dataset.
template<typename DistanceType>
template<typename MatType, typename LabelsType, typename DistanceType>
template<typename GradType>
void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
GradType& gradient)
void LMNNFunction<MatType, LabelsType, DistanceType>::Gradient(
const MatType& transformation, GradType& gradient)
{
// Apply distance metric over dataset.
transformedDataset = transformation * dataset;
double transformationDiff = 0;
if (!transformationOld.is_empty() && iteration++ % range == 0)
ElemType transformationDiff = 0;
if (!transformationOld.is_empty() && iteration++ % updateInterval == 0)
{
// Calculate norm of change in transformation.
transformationDiff = arma::norm(transformation - transformationOld);
@@ -506,7 +493,7 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
norm);
}
}
else if (iteration++ % range == 0)
else if (iteration++ % updateInterval == 0)
{
// Re-calculate impostors on transformed dataset.
constraint.Impostors(impostors, distanceMat, transformedDataset, labels,
@@ -516,10 +503,10 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
gradient.zeros(transformation.n_rows, transformation.n_cols);
// Calculate gradient due to target neighbors.
arma::mat cij = pCij;
MatType cij = pCij;
// Calculate gradient due to impostors.
arma::mat cil = zeros(dataset.n_rows, dataset.n_rows);
MatType cil = zeros<MatType>(dataset.n_rows, dataset.n_rows);
for (size_t i = 0; i < dataset.n_cols; ++i)
{
@@ -530,7 +517,7 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
{
// Calculate cost due to {data point, target neighbors, impostors}
// triplets.
double eval = 0;
ElemType eval = 0;
// Bounds for eval.
if (!transformationOld.is_empty() && evalOld(l, j, i) < -1)
@@ -546,7 +533,7 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
// Calculate exact eval value.
if (eval > -1)
{
if (iteration - 1 % range == 0)
if (iteration - 1 % updateInterval == 0)
{
eval = distance.Evaluate(transformedDataset.col(i),
transformedDataset.col(targetNeighbors(j, i))) -
@@ -581,7 +568,7 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
}
// Caculate gradient due to impostors.
arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
cil += diff * trans(diff);
diff = dataset.col(i) - dataset.col(impostors(l, i));
@@ -598,21 +585,22 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
}
//! Compute gradient over a batch of data points.
template<typename DistanceType>
template<typename MatType, typename LabelsType, typename DistanceType>
template<typename GradType>
void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
const size_t begin,
GradType& gradient,
const size_t batchSize)
void LMNNFunction<MatType, LabelsType, DistanceType>::Gradient(
const MatType& transformation,
const size_t begin,
GradType& gradient,
const size_t batchSize)
{
// Apply distance metric over dataset.
transformedDataset = transformation * dataset;
// Calculate norm of change in transformation.
std::map<size_t, double> transformationDiffs;
std::unordered_map<size_t, ElemType> transformationDiffs;
TransDiff(transformationDiffs, transformation, begin, batchSize);
if (impBounds && iteration++ % range == 0)
if (impBounds && iteration++ % updateInterval == 0)
{
// Track number of data points to use for impostors calculatiom.
size_t numPoints = 0;
@@ -638,7 +626,7 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
constraint.Impostors(impostors, distanceMat,
transformedDataset, labels, norm, points, numPoints);
}
else if (iteration++ % range == 0)
else if (iteration++ % updateInterval == 0)
{
// Re-calculate impostors on transformed dataset.
constraint.Impostors(impostors, distanceMat, transformedDataset, labels,
@@ -647,15 +635,15 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
gradient.zeros(transformation.n_rows, transformation.n_cols);
arma::mat cij = zeros(dataset.n_rows, dataset.n_rows);
arma::mat cil = zeros(dataset.n_rows, dataset.n_rows);
MatType cij = zeros<MatType>(dataset.n_rows, dataset.n_rows);
MatType cil = zeros<MatType>(dataset.n_rows, dataset.n_rows);
for (size_t i = begin; i < begin + batchSize; ++i)
{
for (size_t j = 0; j < k ; ++j)
{
// Calculate gradient due to target neighbors.
arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
cij += diff * trans(diff);
}
@@ -666,7 +654,7 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
{
// Calculate cost due to {data point, target neighbors, impostors}
// triplets.
double eval = 0;
ElemType eval = 0;
// Bounds for eval.
if (lastTransformationIndices(i) && evalOld(l, j, i) < -1)
@@ -682,7 +670,7 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
// Calculate exact eval value.
if (eval > -1)
{
if (iteration - 1 % range == 0)
if (iteration - 1 % updateInterval == 0)
{
eval = distance.Evaluate(transformedDataset.col(i),
transformedDataset.col(targetNeighbors(j, i))) -
@@ -719,7 +707,7 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
}
// Caculate gradient due to impostors.
arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
cil += diff * trans(diff);
diff = dataset.col(i) - dataset.col(impostors(l, i));
@@ -736,25 +724,26 @@ void LMNNFunction<DistanceType>::Gradient(const arma::mat& transformation,
}
//! Compute cost & gradient over whole dataset.
template<typename DistanceType>
template<typename MatType, typename LabelsType, typename DistanceType>
template<typename GradType>
double LMNNFunction<DistanceType>::EvaluateWithGradient(
const arma::mat& transformation,
typename MatType::elem_type
LMNNFunction<MatType, LabelsType, DistanceType>::EvaluateWithGradient(
const MatType& transformation,
GradType& gradient)
{
double cost = 0;
ElemType cost = 0;
// Apply distance metric over dataset.
transformedDataset = transformation * dataset;
double transformationDiff = 0;
ElemType transformationDiff = 0;
if (!transformationOld.is_empty())
{
// Calculate norm of change in transformation.
transformationDiff = arma::norm(transformation - transformationOld);
}
if (!transformationOld.is_empty() && iteration++ % range == 0)
if (!transformationOld.is_empty() && iteration++ % updateInterval == 0)
{
if (impBounds)
{
@@ -781,7 +770,7 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
norm);
}
}
else if (iteration++ % range == 0)
else if (iteration++ % updateInterval == 0)
{
// Re-calculate impostors on transformed dataset.
constraint.Impostors(impostors, distanceMat, transformedDataset, labels,
@@ -791,17 +780,17 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
gradient.zeros(transformation.n_rows, transformation.n_cols);
// Calculate gradient due to target neighbors.
arma::mat cij = pCij;
MatType cij = pCij;
// Calculate gradient due to impostors.
arma::mat cil = zeros(dataset.n_rows, dataset.n_rows);
MatType cil = zeros<MatType>(dataset.n_rows, dataset.n_rows);
for (size_t i = 0; i < dataset.n_cols; ++i)
{
for (size_t j = 0; j < k ; ++j)
{
// Calculate cost due to distance between target neighbors & data point.
double eval = distance.Evaluate(transformedDataset.col(i),
ElemType eval = distance.Evaluate(transformedDataset.col(i),
transformedDataset.col(targetNeighbors(j, i)));
cost += (1 - regularization) * eval;
}
@@ -813,7 +802,7 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
{
// Calculate cost due to {data point, target neighbors, impostors}
// triplets.
double eval = 0;
ElemType eval = 0;
// Bounds for eval.
if (!transformationOld.is_empty() && evalOld(l, j, i) < -1)
@@ -829,7 +818,7 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
// Calculate exact eval value.
if (eval > -1)
{
if (iteration - 1 % range == 0)
if (iteration - 1 % updateInterval == 0)
{
eval = distance.Evaluate(transformedDataset.col(i),
transformedDataset.col(targetNeighbors(j, i))) -
@@ -858,7 +847,7 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
cost += regularization * (1 + eval);
// Caculate gradient due to impostors.
arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
cil += diff * trans(diff);
diff = dataset.col(i) - dataset.col(impostors(l, i));
@@ -877,24 +866,25 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
}
//! Compute cost & gradient over a batch of data points.
template<typename DistanceType>
template<typename MatType, typename LabelsType, typename DistanceType>
template<typename GradType>
double LMNNFunction<DistanceType>::EvaluateWithGradient(
const arma::mat& transformation,
typename MatType::elem_type
LMNNFunction<MatType, LabelsType, DistanceType>::EvaluateWithGradient(
const MatType& transformation,
const size_t begin,
GradType& gradient,
const size_t batchSize)
{
double cost = 0;
ElemType cost = 0;
// Calculate norm of change in transformation.
std::map<size_t, double> transformationDiffs;
std::unordered_map<size_t, ElemType> transformationDiffs;
TransDiff(transformationDiffs, transformation, begin, batchSize);
// Apply distance metric over dataset.
transformedDataset = transformation * dataset;
if (impBounds && iteration++ % range == 0)
if (impBounds && iteration++ % updateInterval == 0)
{
// Track number of data points to use for impostors calculatiom.
size_t numPoints = 0;
@@ -920,7 +910,7 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
constraint.Impostors(impostors, distanceMat,
transformedDataset, labels, norm, points, numPoints);
}
else if (iteration++ % range == 0)
else if (iteration++ % updateInterval == 0)
{
// Re-calculate impostors on transformed dataset.
constraint.Impostors(impostors, distanceMat, transformedDataset, labels,
@@ -929,20 +919,20 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
gradient.zeros(transformation.n_rows, transformation.n_cols);
arma::mat cij = zeros(dataset.n_rows, dataset.n_rows);
arma::mat cil = zeros(dataset.n_rows, dataset.n_rows);
MatType cij = zeros<MatType>(dataset.n_rows, dataset.n_rows);
MatType cil = zeros<MatType>(dataset.n_rows, dataset.n_rows);
for (size_t i = begin; i < begin + batchSize; ++i)
{
for (size_t j = 0; j < k ; ++j)
{
// Calculate cost due to distance between target neighbors & data point.
double eval = distance.Evaluate(transformedDataset.col(i),
ElemType eval = distance.Evaluate(transformedDataset.col(i),
transformedDataset.col(targetNeighbors(j, i)));
cost += (1 - regularization) * eval;
// Calculate gradient due to target neighbors.
arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
cij += diff * trans(diff);
}
@@ -953,7 +943,7 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
{
// Calculate cost due to {data point, target neighbors, impostors}
// triplets.
double eval = 0;
ElemType eval = 0;
// Bounds for eval.
if (lastTransformationIndices(i) && evalOld(l, j, i) < -1)
@@ -969,7 +959,7 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
// Calculate exact eval value.
if (eval > -1)
{
if (iteration - 1 % range == 0)
if (iteration - 1 % updateInterval == 0)
{
eval = distance.Evaluate(transformedDataset.col(i),
transformedDataset.col(targetNeighbors(j, i))) -
@@ -998,7 +988,7 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
cost += regularization * (1 + eval);
// Caculate gradient due to impostors.
arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
cil += diff * trans(diff);
diff = dataset.col(i) - dataset.col(impostors(l, i));
@@ -1016,8 +1006,8 @@ double LMNNFunction<DistanceType>::EvaluateWithGradient(
return cost;
}
template<typename DistanceType>
inline void LMNNFunction<DistanceType>::Precalculate()
template<typename MatType, typename LabelsType, typename DistanceType>
inline void LMNNFunction<MatType, LabelsType, DistanceType>::Precalculate()
{
pCij.zeros(dataset.n_rows, dataset.n_rows);
@@ -1026,7 +1016,7 @@ inline void LMNNFunction<DistanceType>::Precalculate()
for (size_t j = 0; j < k ; ++j)
{
// Calculate gradient due to target neighbors.
arma::vec diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
VecType diff = dataset.col(i) - dataset.col(targetNeighbors(j, i));
pCij += diff * trans(diff);
}
}
src/mlpack/methods/lmnn/lmnn_impl.hpp
+81 -17
View File
@@ -21,27 +21,83 @@ namespace mlpack {
* Takes in a reference to the dataset. Copies the data, initializes
* all of the member variables and constraint object and generate constraints.
*/
template<typename DistanceType, typename OptimizerType>
LMNN<DistanceType, OptimizerType>::LMNN(const arma::mat& dataset,
const arma::Row<size_t>& labels,
const size_t k,
const DistanceType distance) :
dataset(dataset),
labels(labels),
template<typename DistanceType, typename DeprecatedOptimizerType>
LMNN<DistanceType, DeprecatedOptimizerType>::LMNN(
const arma::mat& dataset,
const arma::Row<size_t>& labels,
const size_t k,
const DistanceType distance) :
dataset(&dataset),
labels(&labels),
k(k),
regularization(0.5),
range(1),
updateInterval(1),
distance(distance)
{ /* nothing to do */ }
template<typename DistanceType, typename OptimizerType>
template<typename... CallbackTypes>
void LMNN<DistanceType, OptimizerType>::LearnDistance(arma::mat& outputMatrix,
template<typename DistanceType, typename DeprecatedOptimizerType>
LMNN<DistanceType, DeprecatedOptimizerType>::LMNN(
const size_t k,
const double regularization,
const size_t updateInterval,
const DistanceType distance) :
k(k),
regularization(regularization),
updateInterval(updateInterval),
distance(distance)
{ /* nothing to do */ }
template<typename DistanceType, typename DeprecatedOptimizerType>
template<typename... CallbackTypes, typename, typename>
void LMNN<DistanceType, DeprecatedOptimizerType>::LearnDistance(
arma::mat& outputMatrix,
CallbackTypes&&... callbacks)
{
if (!dataset || !labels)
{
throw std::runtime_error("LMNN::LearnDistance(): cannot call without a "
"dataset!");
}
LearnDistance(*dataset, *labels, outputMatrix, optimizer,
std::forward<CallbackTypes>(callbacks)...);
}
template<typename DistanceType, typename DeprecatedOptimizerType>
template<typename MatType,
typename LabelsType,
typename... CallbackTypes,
typename /* SFINAE check that first callback is not an optimizer */,
typename /* callback SFINAE check */>
void LMNN<DistanceType, DeprecatedOptimizerType>::LearnDistance(
const MatType& dataset,
const LabelsType& labels,
MatType& outputMatrix,
CallbackTypes&&... callbacks) const
{
// This should be replaced with ens::StandardSGD when the deprecated members
// are removed for mlpack 5.0.0.
DeprecatedOptimizerType opt;
LearnDistance(dataset, labels, outputMatrix, opt,
std::forward<CallbackTypes>(callbacks)...);
}
template<typename DistanceType, typename DeprecatedOptimizerType>
template<typename MatType,
typename LabelsType,
typename OptimizerType,
typename... CallbackTypes,
typename /* SFINAE check that opt is an ensmallen optimizer */>
void LMNN<DistanceType, DeprecatedOptimizerType>::LearnDistance(
const MatType& dataset,
const LabelsType& labels,
MatType& outputMatrix,
OptimizerType& opt,
CallbackTypes&&... callbacks) const
{
// LMNN objective function.
LMNNFunction<DistanceType> objFunction(dataset, labels, k,
regularization, range);
LMNNFunction<MatType, LabelsType, DistanceType> objFunction(dataset, labels,
k, regularization, updateInterval);
// See if we were passed an initialized matrix. outputMatrix (L) must be
// having r x d dimensionality.
@@ -49,15 +105,23 @@ void LMNN<DistanceType, OptimizerType>::LearnDistance(arma::mat& outputMatrix,
(outputMatrix.n_rows > dataset.n_rows) ||
!(arma::is_finite(outputMatrix)))
{
Log::Info << "Initial learning point have invalid dimensionality. "
"Identity matrix will be used as initial learning point for "
"optimization." << std::endl;
outputMatrix.eye(dataset.n_rows, dataset.n_rows);
}
optimizer.Optimize(objFunction, outputMatrix, callbacks...);
opt.Optimize(objFunction, outputMatrix, callbacks...);
}
// Serialize the LMNN object.
template<typename DistanceType, typename DeprecatedOptimizerType>
template<typename Archive>
void LMNN<DistanceType, DeprecatedOptimizerType>::serialize(
Archive& ar, const unsigned int /* version */)
{
ar(CEREAL_NVP(k));
ar(CEREAL_NVP(regularization));
ar(CEREAL_NVP(updateInterval));
ar(CEREAL_NVP(distance));
}
} // namespace mlpack
src/mlpack/methods/lmnn/lmnn_main.cpp
+35 -42
View File
@@ -57,7 +57,7 @@ BINDING_LONG_DESC(
PRINT_PARAM_STRING("regularization") + "), In addition, this "
"implementation of LMNN includes a parameter to decide the interval "
"after which impostors must be re-calculated (specified with " +
PRINT_PARAM_STRING("range") + ")."
PRINT_PARAM_STRING("update_interval") + ")."
"\n\n"
"Output can either be the learned distance matrix (specified with " +
PRINT_PARAM_STRING("output") +"), or the transformed dataset "
@@ -124,11 +124,11 @@ BINDING_EXAMPLE(
PRINT_CALL("lmnn", "input", "iris", "labels", "iris_labels", "k", 3,
"optimizer", "bbsgd", "output", "output") +
"\n\n"
"An another program call making use of range & regularization parameter "
"with dataset having labels as last column can be made as: "
"Another program call making use of update interval & regularization "
"parameter with dataset having labels as last column can be made as: "
"\n\n" +
PRINT_CALL("lmnn", "input", "letter_recognition", "k", 5,
"range", 10, "regularization", 0.4, "output", "output"));
"update_interval", 10, "regularization", 0.4, "output", "output"));
// See also...
BINDING_SEE_ALSO("@nca", "#nca");
@@ -174,8 +174,8 @@ PARAM_DOUBLE_IN("step_size", "Step size for AMSGrad, BB_SGD and SGD (alpha).",
PARAM_FLAG("linear_scan", "Don't shuffle the order in which data points are "
"visited for SGD or mini-batch SGD.", "L");
PARAM_INT_IN("batch_size", "Batch size for mini-batch SGD.", "b", 50);
PARAM_INT_IN("range", "Number of iterations after which impostors needs to be "
"recalculated", "R", 1);
PARAM_INT_IN("update_interval", "Number of iterations after which impostors "
"need to be recalculated.", "R", 1);
PARAM_INT_IN("seed", "Random seed. If 0, 'std::time(NULL)' is used.", "s", 0);
using namespace mlpack;
@@ -264,8 +264,8 @@ void BINDING_FUNCTION(util::Params& params, util::Timers& timers)
RequireParamValue<int>(params, "k", [](int x) { return x > 0; }, true,
"number of targets must be positive");
RequireParamValue<int>(params, "range", [](int x) { return x > 0; }, true,
"range must be positive");
RequireParamValue<int>(params, "update_interval", [](int x) { return x > 0; },
true, "update interval must be positive");
RequireParamValue<int>(params, "batch_size", [](int x) { return x > 0; }, true,
"batch size must be positive");
RequireParamValue<double>(params, "regularization", [](double x)
@@ -294,7 +294,7 @@ void BINDING_FUNCTION(util::Params& params, util::Timers& timers)
const bool printAccuracy = params.Has("print_accuracy");
const bool shuffle = !params.Has("linear_scan");
const size_t batchSize = (size_t) params.Get<int>("batch_size");
const size_t range = (size_t) params.Get<int>("range");
const size_t updateInterval = (size_t) params.Get<int>("update_interval");
const size_t rank = (size_t) params.Get<int>("rank");
// Load data.
@@ -359,56 +359,49 @@ void BINDING_FUNCTION(util::Params& params, util::Timers& timers)
// Now create the LMNN object and run the optimization.
timers.Start("lmnn_optimization");
LMNN lmnn(k, regularization, updateInterval);
if (optimizerType == "amsgrad")
{
LMNN<LMetric<2>> lmnn(data, labels, k);
lmnn.Regularization() = regularization;
lmnn.Range() = range;
lmnn.Optimizer().StepSize() = stepSize;
lmnn.Optimizer().MaxIterations() = passes * data.n_cols;
lmnn.Optimizer().Tolerance() = tolerance;
lmnn.Optimizer().Shuffle() = shuffle;
lmnn.Optimizer().BatchSize() = batchSize;
ens::AMSGrad opt;
opt.StepSize() = stepSize;
opt.MaxIterations() = passes * data.n_cols;
opt.Tolerance() = tolerance;
opt.Shuffle() = shuffle;
opt.BatchSize() = batchSize;
lmnn.LearnDistance(distance);
lmnn.LearnDistance(data, labels, distance, opt);
}
else if (optimizerType == "bbsgd")
{
LMNN<LMetric<2>, ens::BBS_BB> lmnn(data, labels, k);
lmnn.Regularization() = regularization;
lmnn.Range() = range;
lmnn.Optimizer().StepSize() = stepSize;
lmnn.Optimizer().MaxIterations() = passes * data.n_cols;
lmnn.Optimizer().Tolerance() = tolerance;
lmnn.Optimizer().Shuffle() = shuffle;
lmnn.Optimizer().BatchSize() = batchSize;
ens::BBS_BB opt;
opt.StepSize() = stepSize;
opt.MaxIterations() = passes * data.n_cols;
opt.Tolerance() = tolerance;
opt.Shuffle() = shuffle;
opt.BatchSize() = batchSize;
lmnn.LearnDistance(distance);
lmnn.LearnDistance(data, labels, distance, opt);
}
else if (optimizerType == "sgd")
{
// Using SGD is not recommended as the learning matrix can
// diverge to inf causing serious memory problems.
LMNN<LMetric<2>, ens::StandardSGD> lmnn(data, labels, k);
lmnn.Regularization() = regularization;
lmnn.Range() = range;
lmnn.Optimizer().StepSize() = stepSize;
lmnn.Optimizer().MaxIterations() = passes * data.n_cols;
lmnn.Optimizer().Tolerance() = tolerance;
lmnn.Optimizer().Shuffle() = shuffle;
lmnn.Optimizer().BatchSize() = batchSize;
ens::StandardSGD opt;
opt.StepSize() = stepSize;
opt.MaxIterations() = passes * data.n_cols;
opt.Tolerance() = tolerance;
opt.Shuffle() = shuffle;
opt.BatchSize() = batchSize;
lmnn.LearnDistance(distance);
lmnn.LearnDistance(data, labels, distance, opt);
}
else if (optimizerType == "lbfgs")
{
LMNN<LMetric<2>, ens::L_BFGS> lmnn(data, labels, k);
lmnn.Regularization() = regularization;
lmnn.Range() = range;
lmnn.Optimizer().MaxIterations() = maxIterations;
lmnn.Optimizer().MinGradientNorm() = tolerance;
ens::L_BFGS opt;
opt.MaxIterations() = maxIterations;
opt.MinGradientNorm() = tolerance;
lmnn.LearnDistance(distance);
lmnn.LearnDistance(data, labels, distance, opt);
}
timers.Stop("lmnn_optimization");
src/mlpack/methods/neighbor_search/neighbor_search.hpp
+13 -5
View File
@@ -222,9 +222,11 @@ class NeighborSearch
* @param distances Matrix storing distances of neighbors for each query
* point.
*/
// TODO: templatize further to remove Armadillo type requirement
template<typename IndexType = size_t>
void Search(const MatType& querySet,
const size_t k,
arma::Mat<size_t>& neighbors,
arma::Mat<IndexType>& neighbors,
arma::Mat<ElemType>& distances);
/**
@@ -247,9 +249,11 @@ class NeighborSearch
* @param sameSet Denotes whether or not the reference and query sets are the
* same.
*/
// TODO: templatize further to remove Armadillo type requirement
template<typename IndexType = size_t>
void Search(Tree& queryTree,
const size_t k,
arma::Mat<size_t>& neighbors,
arma::Mat<IndexType>& neighbors,
arma::Mat<ElemType>& distances,
bool sameSet = false);
@@ -267,8 +271,10 @@ class NeighborSearch
* @param distances Matrix storing distances of neighbors for each query
* point.
*/
// TODO: templatize further to remove Armadillo type requirement
template<typename IndexType = size_t>
void Search(const size_t k,
arma::Mat<size_t>& neighbors,
arma::Mat<IndexType>& neighbors,
arma::Mat<ElemType>& distances);
/**
@@ -300,8 +306,10 @@ class NeighborSearch
* query point.
* @return Recall.
*/
static double Recall(arma::Mat<size_t>& foundNeighbors,
arma::Mat<size_t>& realNeighbors);
// TODO: templatize further to remove Armadillo type requirement
template<typename IndexType = size_t>
static double Recall(arma::Mat<IndexType>& foundNeighbors,
arma::Mat<IndexType>& realNeighbors);
//! Return the total number of base case evaluations performed during the last
//! search.
src/mlpack/methods/neighbor_search/neighbor_search_impl.hpp
+16 -12
View File
@@ -360,11 +360,12 @@ template<typename SortPolicy,
typename TreeMatType> class TreeType,
template<typename> class DualTreeTraversalType,
template<typename> class SingleTreeTraversalType>
template<typename IndexType>
void NeighborSearch<SortPolicy, DistanceType, MatType, TreeType,
DualTreeTraversalType, SingleTreeTraversalType>::Search(
const MatType& querySet,
const size_t k,
arma::Mat<size_t>& neighbors,
arma::Mat<IndexType>& neighbors,
arma::Mat<ElemType>& distances)
{
if (k > referenceSet->n_cols)
@@ -385,7 +386,7 @@ DualTreeTraversalType, SingleTreeTraversalType>::Search(
// indices back to their original indices when this computation is finished.
// To avoid an extra copy, we will store the neighbors and distances in a
// separate matrix.
arma::Mat<size_t>* neighborPtr = &neighbors;
arma::Mat<IndexType>* neighborPtr = &neighbors;
arma::Mat<ElemType>* distancePtr = &distances;
// Mapping is only necessary if the tree rearranges points.
@@ -394,10 +395,10 @@ DualTreeTraversalType, SingleTreeTraversalType>::Search(
if (searchMode == DUAL_TREE_MODE)
{
distancePtr = new arma::Mat<ElemType>; // Query indices need to be mapped.
neighborPtr = new arma::Mat<size_t>;
neighborPtr = new arma::Mat<IndexType>;
}
else if (!oldFromNewReferences.empty())
neighborPtr = new arma::Mat<size_t>; // Reference indices need mapping.
neighborPtr = new arma::Mat<IndexType>; // Reference indices need mapping.
}
// Set the size of the neighbor and distance matrices.
@@ -565,11 +566,12 @@ template<typename SortPolicy,
typename TreeMatType> class TreeType,
template<typename> class DualTreeTraversalType,
template<typename> class SingleTreeTraversalType>
template<typename IndexType>
void NeighborSearch<SortPolicy, DistanceType, MatType, TreeType,
DualTreeTraversalType, SingleTreeTraversalType>::Search(
Tree& queryTree,
const size_t k,
arma::Mat<size_t>& neighbors,
arma::Mat<IndexType>& neighbors,
arma::Mat<ElemType>& distances,
bool sameSet)
{
@@ -593,10 +595,10 @@ DualTreeTraversalType, SingleTreeTraversalType>::Search(
const MatType& querySet = queryTree.Dataset();
// We won't need to map query indices, but will we need to map distances?
arma::Mat<size_t>* neighborPtr = &neighbors;
arma::Mat<IndexType>* neighborPtr = &neighbors;
if (!oldFromNewReferences.empty() && TreeTraits<Tree>::RearrangesDataset)
neighborPtr = new arma::Mat<size_t>;
neighborPtr = new arma::Mat<IndexType>;
neighborPtr->set_size(k, querySet.n_cols);
distances.set_size(k, querySet.n_cols);
@@ -644,10 +646,11 @@ template<typename SortPolicy,
typename TreeMatType> class TreeType,
template<typename> class DualTreeTraversalType,
template<typename> class SingleTreeTraversalType>
template<typename IndexType>
void NeighborSearch<SortPolicy, DistanceType, MatType, TreeType,
DualTreeTraversalType, SingleTreeTraversalType>::Search(
const size_t k,
arma::Mat<size_t>& neighbors,
arma::Mat<IndexType>& neighbors,
arma::Mat<ElemType>& distances)
{
if (k > referenceSet->n_cols)
@@ -669,14 +672,14 @@ DualTreeTraversalType, SingleTreeTraversalType>::Search(
baseCases = 0;
scores = 0;
arma::Mat<size_t>* neighborPtr = &neighbors;
arma::Mat<IndexType>* neighborPtr = &neighbors;
arma::Mat<ElemType>* distancePtr = &distances;
if (!oldFromNewReferences.empty() && TreeTraits<Tree>::RearrangesDataset)
{
// We will always need to rearrange in this case.
distancePtr = new MatType;
neighborPtr = new arma::Mat<size_t>;
neighborPtr = new arma::Mat<IndexType>;
}
// Initialize results.
@@ -861,10 +864,11 @@ template<typename SortPolicy,
typename TreeMatType> class TreeType,
template<typename> class DualTreeTraversalType,
template<typename> class SingleTreeTraversalType>
template<typename IndexType>
double NeighborSearch<SortPolicy, DistanceType, MatType, TreeType,
DualTreeTraversalType, SingleTreeTraversalType>::Recall(
arma::Mat<size_t>& foundNeighbors,
arma::Mat<size_t>& realNeighbors)
arma::Mat<IndexType>& foundNeighbors,
arma::Mat<IndexType>& realNeighbors)
{
if (foundNeighbors.n_rows != realNeighbors.n_rows ||
foundNeighbors.n_cols != realNeighbors.n_cols)
src/mlpack/methods/neighbor_search/neighbor_search_rules.hpp
+4 -1
View File
@@ -63,7 +63,10 @@ class NeighborSearchRules
* @param distances Matrix storing distances of neighbors for each query
* point.
*/
void GetResults(arma::Mat<size_t>& neighbors, arma::Mat<ElemType>& distances);
// TODO: templatize fully to remove requirement of Armadillo matrix
template<typename IndexType = size_t>
void GetResults(arma::Mat<IndexType>& neighbors,
arma::Mat<ElemType>& distances);
/**
* Get the distance from the query point to the reference point.
src/mlpack/methods/neighbor_search/neighbor_search_rules_impl.hpp
+3 -2
View File
@@ -59,8 +59,9 @@ NeighborSearchRules<SortPolicy, DistanceType, TreeType>::NeighborSearchRules(
}
template<typename SortPolicy, typename DistanceType, typename TreeType>
template<typename IndexType>
void NeighborSearchRules<SortPolicy, DistanceType, TreeType>::GetResults(
arma::Mat<size_t>& neighbors,
arma::Mat<IndexType>& neighbors,
arma::Mat<ElemType>& distances)
{
neighbors.set_size(k, querySet.n_cols);
@@ -71,7 +72,7 @@ void NeighborSearchRules<SortPolicy, DistanceType, TreeType>::GetResults(
CandidateList& pqueue = candidates[i];
for (size_t j = 1; j <= k; ++j)
{
neighbors(k - j, i) = pqueue.top().second;
neighbors(k - j, i) = (IndexType) pqueue.top().second;
distances(k - j, i) = pqueue.top().first;
pqueue.pop();
}
src/mlpack/tests/callback_test.cpp
+3 -2
View File
@@ -151,12 +151,13 @@ TEST_CASE("LMNNWithOptimizerCallback", "[CallbackTest]")
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNN<> lmnn(dataset, labels, 1);
LMNN<> lmnn(1);
arma::mat outputMatrix;
std::stringstream stream;
lmnn.LearnDistance(outputMatrix, ens::ProgressBar(70, stream));
lmnn.LearnDistance(dataset, labels, outputMatrix,
ens::ProgressBar(70, stream));
REQUIRE(stream.str().length() > 0);
}
src/mlpack/tests/lmnn_test.cpp
+295 -230
View File
@@ -30,25 +30,27 @@ using namespace ens;
* The target neighbors function should be correct.
* point.
*/
TEST_CASE("LMNNTargetNeighborsTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNTargetNeighborsTest", "[LMNNTest]", float, double)
{
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
typedef TestType ElemType;
Constraints<> constraint(dataset, labels, 1);
// Useful but simple dataset with six points and two classes.
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
Constraints<arma::Mat<ElemType>, arma::Row<size_t>> constraint(dataset,
labels, 1);
// Calculate norm of datapoints.
arma::vec norm(dataset.n_cols);
arma::Col<ElemType> norm(dataset.n_cols);
for (size_t i = 0; i < dataset.n_cols; ++i)
{
norm(i) = arma::norm(dataset.col(i));
}
//! Store target neighbors of data points.
arma::Mat<size_t> targetNeighbors =
arma::Mat<size_t>(1, dataset.n_cols, arma::fill::zeros);
arma::umat targetNeighbors(1, dataset.n_cols, arma::fill::zeros);
constraint.TargetNeighbors(targetNeighbors, dataset, labels, norm);
@@ -63,25 +65,27 @@ TEST_CASE("LMNNTargetNeighborsTest", "[LMNNTest]")
/**
* The impostors function should be correct.
*/
TEST_CASE("LMNNImpostorsTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNImpostorsTest", "[LMNNTest]", float, double)
{
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
typedef TestType ElemType;
Constraints<> constraint(dataset, labels, 1);
// Useful but simple dataset with six points and two classes.
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
Constraints<arma::Mat<ElemType>, arma::Row<size_t>> constraint(dataset,
labels, 1);
// Calculate norm of datapoints.
arma::vec norm(dataset.n_cols);
arma::Col<ElemType> norm(dataset.n_cols);
for (size_t i = 0; i < dataset.n_cols; ++i)
{
norm(i) = arma::norm(dataset.col(i));
}
//! Store impostors of data points.
arma::Mat<size_t> impostors =
arma::Mat<size_t>(1, dataset.n_cols, arma::fill::zeros);
arma::umat impostors(1, dataset.n_cols, arma::fill::zeros);
constraint.Impostors(impostors, dataset, labels, norm);
@@ -101,300 +105,339 @@ TEST_CASE("LMNNImpostorsTest", "[LMNNTest]")
* The LMNN function should return the identity matrix as its initial
* point.
*/
TEST_CASE("LMNNInitialPointTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNInitialPointTest", "[LMNNTest]", float, double)
{
typedef TestType ElemType;
// Cheap fake dataset.
arma::mat dataset = arma::randu(5, 5);
arma::Mat<ElemType> dataset = arma::randu<arma::Mat<ElemType>>(5, 5);
arma::Row<size_t> labels = "0 1 1 0 0";
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.5, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.5, 1);
// Verify the initial point is the identity matrix.
arma::mat initialPoint = lmnnfn.GetInitialPoint();
const double eps = std::is_same<ElemType, float>::value ? 1e-4 : 1e-7;
const double margin = std::is_same<ElemType, float>::value ? 1e-4 : 1e-5;
arma::Mat<ElemType> initialPoint = lmnnfn.GetInitialPoint();
for (int row = 0; row < 5; row++)
{
for (int col = 0; col < 5; col++)
{
if (row == col)
REQUIRE(initialPoint(row, col) == Approx(1.0).epsilon(1e-7));
REQUIRE(initialPoint(row, col) == Approx(1.0).epsilon(eps));
else
REQUIRE(initialPoint(row, col) == Approx(0.0).margin(1e-5));
REQUIRE(initialPoint(row, col) == Approx(0.0).margin(margin));
}
}
}
/***
* Ensure non-seprable objective function is right.
* Ensure non-separable objective function is right.
*/
TEST_CASE("LMNNInitialEvaluationTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNInitialEvaluationTest", "[LMNNTest]", float, double)
{
typedef TestType ElemType;
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
double objective = lmnnfn.Evaluate(arma::eye<arma::mat>(2, 2));
ElemType objective = lmnnfn.Evaluate(arma::eye<arma::Mat<ElemType>>(2, 2));
// Result calculated by hand.
REQUIRE(objective == Approx(9.456).epsilon(1e-7));
const double eps = std::is_same<ElemType, float>::value ? 1e-4 : 1e-7;
REQUIRE(objective == Approx(9.456).epsilon(eps));
}
/**
* Ensure non-seprable gradient function is right.
* Ensure non-separable gradient function is right.
*/
TEST_CASE("LMNNInitialGradientTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNInitialGradientTest", "[LMNNTest]", float, double)
{
typedef TestType ElemType;
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
arma::mat gradient;
arma::mat coordinates = arma::eye<arma::mat>(2, 2);
arma::Mat<ElemType> gradient;
arma::Mat<ElemType> coordinates = arma::eye<arma::Mat<ElemType>>(2, 2);
lmnnfn.Gradient(coordinates, gradient);
// Result calculated by hand.
REQUIRE(gradient(0, 0) == Approx(-0.288).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(1e-5));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(1e-5));
REQUIRE(gradient(1, 1) == Approx(12.0).epsilon(1e-7));
const double eps = std::is_same<ElemType, float>::value ? 1e-4 : 1e-7;
const double margin = std::is_same<ElemType, float>::value ? 1e-4 : 1e-5;
REQUIRE(gradient(0, 0) == Approx(-0.288).epsilon(eps));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(12.0).epsilon(eps));
}
/***
* Ensure non-seprable EvaluateWithGradient function is right.
* Ensure non-separable EvaluateWithGradient function is right.
*/
TEST_CASE("LMNNInitialEvaluateWithGradientTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNInitialEvaluateWithGradientTest", "[LMNNTest]", float,
double)
{
typedef TestType ElemType;
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
arma::mat gradient;
arma::mat coordinates = arma::eye<arma::mat>(2, 2);
double objective = lmnnfn.EvaluateWithGradient(coordinates, gradient);
arma::Mat<ElemType> gradient;
arma::Mat<ElemType> coordinates = arma::eye<arma::Mat<ElemType>>(2, 2);
ElemType objective = lmnnfn.EvaluateWithGradient(coordinates, gradient);
const double eps = std::is_same<ElemType, float>::value ? 1e-4 : 1e-7;
const double margin = std::is_same<ElemType, float>::value ? 1e-4 : 1e-5;
// Result calculated by hand.
REQUIRE(objective == Approx(9.456).epsilon(1e-7));
REQUIRE(objective == Approx(9.456).epsilon(eps));
// Check Gradient
REQUIRE(gradient(0, 0) == Approx(-0.288).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(1e-5));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(1e-5));
REQUIRE(gradient(1, 1) == Approx(12.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.288).epsilon(eps));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(12.0).epsilon(eps));
}
/**
* Ensure the separable objective function is right.
*/
TEST_CASE("LMNNSeparableObjectiveTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNSeparableObjectiveTest", "[LMNNTest]", float, double)
{
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
typedef TestType ElemType;
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
// Useful but simple dataset with six points and two classes.
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
// Result calculated by hand.
arma::mat coordinates = arma::eye<arma::mat>(2, 2);
REQUIRE(lmnnfn.Evaluate(coordinates, 0, 1) == Approx(1.576).epsilon(1e-7));
REQUIRE(lmnnfn.Evaluate(coordinates, 1, 1) == Approx(1.576).epsilon(1e-7));
REQUIRE(lmnnfn.Evaluate(coordinates, 2, 1) == Approx(1.576).epsilon(1e-7));
REQUIRE(lmnnfn.Evaluate(coordinates, 3, 1) == Approx(1.576).epsilon(1e-7));
REQUIRE(lmnnfn.Evaluate(coordinates, 4, 1) == Approx(1.576).epsilon(1e-7));
REQUIRE(lmnnfn.Evaluate(coordinates, 5, 1) == Approx(1.576).epsilon(1e-7));
const double eps = std::is_same<ElemType, float>::value ? 1e-4 : 1e-7;
arma::Mat<ElemType> coordinates = arma::eye<arma::Mat<ElemType>>(2, 2);
REQUIRE(lmnnfn.Evaluate(coordinates, 0, 1) == Approx(1.576).epsilon(eps));
REQUIRE(lmnnfn.Evaluate(coordinates, 1, 1) == Approx(1.576).epsilon(eps));
REQUIRE(lmnnfn.Evaluate(coordinates, 2, 1) == Approx(1.576).epsilon(eps));
REQUIRE(lmnnfn.Evaluate(coordinates, 3, 1) == Approx(1.576).epsilon(eps));
REQUIRE(lmnnfn.Evaluate(coordinates, 4, 1) == Approx(1.576).epsilon(eps));
REQUIRE(lmnnfn.Evaluate(coordinates, 5, 1) == Approx(1.576).epsilon(eps));
}
/**
* Ensure the separable gradient is right.
*/
TEST_CASE("LMNNSeparableGradientTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNSeparableGradientTest", "[LMNNTest]", float, double)
{
typedef TestType ElemType;
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
arma::mat coordinates = arma::eye<arma::mat>(2, 2);
arma::mat gradient(2, 2);
arma::Mat<ElemType> coordinates = arma::eye<arma::Mat<ElemType>>(2, 2);
arma::Mat<ElemType> gradient(2, 2);
lmnnfn.Gradient(coordinates, 0, gradient, 1);
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
const double eps = std::is_same<ElemType, float>::value ? 1e-4 : 1e-7;
const double margin = std::is_same<ElemType, float>::value ? 1e-4 : 1e-5;
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
lmnnfn.Gradient(coordinates, 1, gradient, 1);
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
lmnnfn.Gradient(coordinates, 2, gradient, 1);
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
lmnnfn.Gradient(coordinates, 3, gradient, 1);
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
lmnnfn.Gradient(coordinates, 4, gradient, 1);
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
lmnnfn.Gradient(coordinates, 5, gradient, 1);
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
}
/**
* Ensure the separable EvaluateWithGradient function is right.
*/
TEST_CASE("LMNNSeparableEvaluateWithGradientTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNSeparableEvaluateWithGradientTest", "[LMNNTest]", float,
double)
{
typedef TestType ElemType;
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
arma::mat coordinates = arma::eye<arma::mat>(2, 2);
arma::mat gradient(2, 2);
arma::Mat<ElemType> coordinates = arma::eye<arma::Mat<ElemType>>(2, 2);
arma::Mat<ElemType> gradient(2, 2);
double objective = lmnnfn.EvaluateWithGradient(coordinates, 0, gradient, 1);
ElemType objective = lmnnfn.EvaluateWithGradient(coordinates, 0, gradient, 1);
REQUIRE(objective == Approx(1.576).epsilon(1e-7));
const double eps = std::is_same<ElemType, float>::value ? 1e-4 : 1e-7;
const double margin = std::is_same<ElemType, float>::value ? 1e-4 : 1e-5;
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(objective == Approx(1.576).epsilon(eps));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
objective = lmnnfn.EvaluateWithGradient(coordinates, 1, gradient, 1);
REQUIRE(objective == Approx(1.576).epsilon(1e-7));
REQUIRE(objective == Approx(1.576).epsilon(eps));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
objective = lmnnfn.EvaluateWithGradient(coordinates, 2, gradient, 1);
REQUIRE(objective == Approx(1.576).epsilon(1e-7));
REQUIRE(objective == Approx(1.576).epsilon(eps));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
objective = lmnnfn.EvaluateWithGradient(coordinates, 3, gradient, 1);
REQUIRE(objective == Approx(1.576).epsilon(1e-7));
REQUIRE(objective == Approx(1.576).epsilon(eps));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
objective = lmnnfn.EvaluateWithGradient(coordinates, 4, gradient, 1);
REQUIRE(objective == Approx(1.576).epsilon(1e-7));
REQUIRE(objective == Approx(1.576).epsilon(eps));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
objective = lmnnfn.EvaluateWithGradient(coordinates, 5, gradient, 1);
REQUIRE(objective == Approx(1.576).epsilon(1e-7));
REQUIRE(objective == Approx(1.576).epsilon(eps));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(1e-7));
REQUIRE(gradient(0, 1) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 0) == Approx(0.0).epsilon(1e-7));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(1e-7));
REQUIRE(gradient(0, 0) == Approx(-0.048).epsilon(eps));
REQUIRE(gradient(0, 1) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 0) == Approx(0.0).margin(margin));
REQUIRE(gradient(1, 1) == Approx(2.0).epsilon(eps));
}
// Check that final objective value using SGD optimizer is optimal.
TEST_CASE("LMNNSGDSimpleDatasetTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNSGDSimpleDatasetTest", "[LMNNTest]", float, double)
{
typedef TestType ElemType;
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNN<> lmnn(dataset, labels, 1);
LMNN<> lmnn(1);
arma::mat outputMatrix;
lmnn.LearnDistance(outputMatrix);
arma::Mat<ElemType> outputMatrix;
lmnn.LearnDistance(dataset, labels, outputMatrix);
// Ensure that the objective function is better now.
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
double initObj = lmnnfn.Evaluate(arma::eye<arma::mat>(2, 2));
double finalObj = lmnnfn.Evaluate(outputMatrix);
ElemType initObj = lmnnfn.Evaluate(arma::eye<arma::Mat<ElemType>>(2, 2));
ElemType finalObj = lmnnfn.Evaluate(outputMatrix);
// finalObj must be less than initObj.
REQUIRE(finalObj < initObj);
}
// Check that final objective value using L-BFGS optimizer is optimal.
TEST_CASE("LMNNLBFGSSimpleDatasetTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNLBFGSSimpleDatasetTest", "[LMNNTest]", float, double)
{
typedef TestType ElemType;
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNN<SquaredEuclideanDistance, L_BFGS> lmnn(dataset, labels, 1);
LMNN lmnn(1);
arma::mat outputMatrix;
lmnn.LearnDistance(outputMatrix);
arma::Mat<ElemType> outputMatrix;
ens::L_BFGS lbfgs;
lmnn.LearnDistance(dataset, labels, outputMatrix, lbfgs);
// Ensure that the objective function is better now.
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
double initObj = lmnnfn.Evaluate(arma::eye<arma::mat>(2, 2));
double finalObj = lmnnfn.Evaluate(outputMatrix);
ElemType initObj = lmnnfn.Evaluate(arma::eye<arma::Mat<ElemType>>(2, 2));
ElemType finalObj = lmnnfn.Evaluate(outputMatrix);
// finalObj must be less than initObj.
REQUIRE(finalObj < initObj);
}
double KnnAccuracy(const arma::mat& dataset,
const arma::Row<size_t>& labels,
template<typename MatType, typename LabelsType>
double KnnAccuracy(const MatType& dataset,
const LabelsType& labels,
const size_t k)
{
arma::Row<size_t> uniqueLabels = arma::unique(labels);
typedef typename MatType::elem_type ElemType;
LabelsType uniqueLabels = arma::unique(labels);
arma::Mat<size_t> neighbors;
arma::mat distances;
arma::Mat<ElemType> distances;
KNN knn;
NeighborSearch<NearestNeighborSort, EuclideanDistance, MatType> knn;
knn.Train(dataset);
knn.Search(k, neighbors, distances);
@@ -404,43 +447,44 @@ double KnnAccuracy(const arma::mat& dataset,
for (size_t i = 0; i < dataset.n_cols; ++i)
{
arma::vec Map;
Map.zeros(uniqueLabels.n_cols);
arma::Col<ElemType> m;
m.zeros(uniqueLabels.n_cols);
for (size_t j = 0; j < k; ++j)
Map(labels(neighbors(j, i))) +=
1 / std::pow(distances(j, i) + 1, 2);
m(labels(neighbors(j, i))) += 1 / std::pow(distances(j, i) + 1, 2);
size_t index = ConvTo<size_t>::From(arma::find(Map
== arma::max(Map)));
size_t index = ConvTo<size_t>::From(arma::find(m == arma::max(m)));
// Increase count if labels match.
if (index == labels(i))
count++;
}
// return accuracy.
// Return accuracy.
return ((double) count / dataset.n_cols) * 100;
}
// Check that final accuracy is greater than initial accuracy on
// simple dataset.
TEST_CASE("LMNNAccuracyTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNAccuracyTest", "[LMNNTest]", float, double)
{
typedef TestType ElemType;
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
// Taking k = 3 as the case of k = 1 can be easily observed.
double initAccuracy = KnnAccuracy(dataset, labels, 3);
LMNN<> lmnn(dataset, labels, 2);
LMNN<> lmnn(2);
arma::mat outputMatrix;
lmnn.LearnDistance(outputMatrix);
arma::Mat<ElemType> outputMatrix;
lmnn.LearnDistance(dataset, labels, outputMatrix);
double finalAccuracy = KnnAccuracy(outputMatrix * dataset, labels, 3);
arma::Mat<ElemType> transformedData = outputMatrix * dataset;
double finalAccuracy = KnnAccuracy(transformedData, labels, 3);
// finalObj must be less than initObj.
REQUIRE(initAccuracy < finalAccuracy);
@@ -452,18 +496,20 @@ TEST_CASE("LMNNAccuracyTest", "[LMNNTest]")
// Check that accuracy while learning square distance matrix is the same as when
// we are learning low rank matrix. I'm ok if this passes only once out of
// three tries.
TEST_CASE("LMNNLowRankAccuracyLBFGSTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNLowRankAccuracyLBFGSTest", "[LMNNTest]", float, double)
{
typedef TestType ElemType;
bool success = false;
for (size_t trial = 0; trial < 3; ++trial)
{
arma::mat dataPart1;
arma::Mat<ElemType> dataPart1;
dataPart1.randn(5, 50);
arma::Row<size_t> labelsPart1(50);
labelsPart1.fill(0);
arma::mat dataPart2;
arma::Mat<ElemType> dataPart2;
dataPart2.randn(5, 50);
arma::Row<size_t> labelsPart2(50);
@@ -473,26 +519,29 @@ TEST_CASE("LMNNLowRankAccuracyLBFGSTest", "[LMNNTest]")
arma::uvec ordering = arma::shuffle(arma::linspace<arma::uvec>(0, 99, 100));
// Generate datasets.
arma::mat dataset = join_rows(dataPart1, dataPart2);
arma::Mat<ElemType> dataset = join_rows(dataPart1, dataPart2);
dataset = dataset.cols(ordering);
// Generate labels.
arma::Row<size_t> labels = join_rows(labelsPart1, labelsPart2);
labels = labels.cols(ordering);
LMNN<SquaredEuclideanDistance, L_BFGS> lmnn(dataset, labels, 1);
LMNN<SquaredEuclideanDistance> lmnn(1);
// Learn a square matrix.
arma::mat outputMatrix;
lmnn.LearnDistance(outputMatrix);
arma::Mat<ElemType> outputMatrix;
L_BFGS lbfgs;
lmnn.LearnDistance(dataset, labels, outputMatrix, lbfgs);
double acc1 = KnnAccuracy(outputMatrix * dataset, labels, 1);
arma::Mat<ElemType> transformedData = outputMatrix * dataset;
double acc1 = KnnAccuracy(transformedData, labels, 1);
// Learn a low rank matrix.
outputMatrix = arma::randu(4, 5);
lmnn.LearnDistance(outputMatrix);
outputMatrix = arma::randu<arma::Mat<ElemType>>(4, 5);
lmnn.LearnDistance(dataset, labels, outputMatrix, lbfgs);
double acc2 = KnnAccuracy(outputMatrix * dataset, labels, 1);
transformedData = outputMatrix * dataset;
double acc2 = KnnAccuracy(transformedData, labels, 1);
// We keep the tolerance very high. We need to ensure the accuracy drop
// isn't any more than 10%.
@@ -507,18 +556,20 @@ TEST_CASE("LMNNLowRankAccuracyLBFGSTest", "[LMNNTest]")
// Check that accuracy while learning square distance matrix is the same as when
// we are learning low rank matrix. I'm ok if this passes only once out of
// three tries.
TEST_CASE("LMNNLowRankAccuracyTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNLowRankAccuracyTest", "[LMNNTest]", float, double)
{
typedef TestType ElemType;
bool success = false;
for (size_t trial = 0; trial < 3; ++trial)
{
arma::mat dataPart1;
arma::Mat<ElemType> dataPart1;
dataPart1.randn(5, 50);
arma::Row<size_t> labelsPart1(50);
labelsPart1.fill(0);
arma::mat dataPart2;
arma::Mat<ElemType> dataPart2;
dataPart2.randn(5, 50);
arma::Row<size_t> labelsPart2(50);
@@ -528,26 +579,28 @@ TEST_CASE("LMNNLowRankAccuracyTest", "[LMNNTest]")
arma::uvec ordering = arma::shuffle(arma::linspace<arma::uvec>(0, 99, 100));
// Generate datasets.
arma::mat dataset = join_rows(dataPart1, dataPart2);
arma::Mat<ElemType> dataset = join_rows(dataPart1, dataPart2);
dataset = dataset.cols(ordering);
// Generate labels.
arma::Row<size_t> labels = join_rows(labelsPart1, labelsPart2);
labels = labels.cols(ordering);
LMNN<> lmnn(dataset, labels, 1);
LMNN<> lmnn(1);
// Learn a square matrix.
arma::mat outputMatrix;
lmnn.LearnDistance(outputMatrix);
arma::Mat<ElemType> outputMatrix;
lmnn.LearnDistance(dataset, labels, outputMatrix);
double acc1 = KnnAccuracy(outputMatrix * dataset, labels, 1);
arma::Mat<ElemType> transformedData = outputMatrix * dataset;
double acc1 = KnnAccuracy(transformedData, labels, 1);
// Learn a low rank matrix.
outputMatrix = arma::randu(4, 5);
lmnn.LearnDistance(outputMatrix);
outputMatrix = arma::randu<arma::Mat<ElemType>>(4, 5);
lmnn.LearnDistance(dataset, labels, outputMatrix);
double acc2 = KnnAccuracy(outputMatrix * dataset, labels, 1);
transformedData = outputMatrix * dataset;
double acc2 = KnnAccuracy(transformedData, labels, 1);
// We keep the tolerance very high. We need to ensure the accuracy drop
// isn't any more than 10%.
@@ -621,29 +674,31 @@ TEST_CASE("LMNNLowRankAccuracyBBSGDTest", "[LMNNTest]")
// Comprehensive gradient tests by Marcus Edel & Ryan Curtin.
// Simple numerical gradient checker.
template<class FunctionType>
template<typename FunctionType, typename MatType>
double CheckGradient(FunctionType& function,
arma::mat& coordinates,
const double eps = 1e-7)
MatType& coordinates,
const typename MatType::elem_type eps = 1e-7)
{
typedef typename MatType::elem_type ElemType;
// Get gradients for the current parameters.
arma::mat orgGradient, gradient, estGradient;
MatType orgGradient, gradient, estGradient;
function.Gradient(coordinates, orgGradient);
estGradient = arma::zeros(orgGradient.n_rows, orgGradient.n_cols);
estGradient = arma::zeros<MatType>(orgGradient.n_rows, orgGradient.n_cols);
// Compute numeric approximations to gradient.
for (size_t i = 0; i < orgGradient.n_elem; ++i)
{
double tmp = coordinates(i);
ElemType tmp = coordinates(i);
// Perturb parameter with a positive constant and get costs.
coordinates(i) += eps;
double costPlus = function.Evaluate(coordinates);
ElemType costPlus = function.Evaluate(coordinates);
// Perturb parameter with a negative constant and get costs.
coordinates(i) -= (2 * eps);
double costMinus = function.Evaluate(coordinates);
ElemType costMinus = function.Evaluate(coordinates);
// Restore the parameter value.
coordinates(i) = tmp;
@@ -657,74 +712,84 @@ double CheckGradient(FunctionType& function,
arma::norm(orgGradient + estGradient);
}
TEST_CASE("LMNNFunctionGradientTest", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNFunctionGradientTest", "[LMNNTest]", float, double)
{
typedef TestType ElemType;
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
// 10 trials with random positions.
for (size_t i = 0; i < 10; ++i)
{
arma::mat coordinates(2, 2, arma::fill::randn);
arma::Mat<ElemType> coordinates(2, 2, arma::fill::randn);
CheckGradient(lmnnfn, coordinates);
}
}
TEST_CASE("LMNNFunctionGradientTest2", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNFunctionGradientTest2", "[LMNNTest]", float, double)
{
// Useful but simple dataset with six points and two classes.
arma::mat dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
typedef TestType ElemType;
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
// Useful but simple dataset with six points and two classes.
arma::Mat<ElemType> dataset = "-0.1 -0.1 -0.1 0.1 0.1 0.1;"
" 1.0 0.0 -1.0 1.0 0.0 -1.0 ";
arma::Row<size_t> labels = " 0 0 0 1 1 1 ";
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
// 10 trials with random positions.
for (size_t i = 0; i < 10; ++i)
{
arma::mat coordinates(2, 2, arma::fill::randu);
arma::Mat<ElemType> coordinates(2, 2, arma::fill::randu);
CheckGradient(lmnnfn, coordinates);
}
}
TEST_CASE("LMNNFunctionGradientTest3", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNFunctionGradientTest3", "[LMNNTest]", float, double)
{
arma::mat dataset;
typedef TestType ElemType;
arma::Mat<ElemType> dataset;
arma::Row<size_t> labels;
if (!data::Load("iris.csv", dataset))
FAIL("Cannot load dataset iris.csv");
if (!data::Load("iris_labels.txt", labels))
FAIL("Cannot load dataset iris_labels.txt");
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
// 10 trials with random positions.
for (size_t i = 0; i < 10; ++i)
{
arma::mat coordinates(dataset.n_rows, dataset.n_rows, arma::fill::randn);
arma::Mat<ElemType> coordinates(dataset.n_rows, dataset.n_rows,
arma::fill::randn);
CheckGradient(lmnnfn, coordinates);
}
}
TEST_CASE("LMNNFunctionGradientTest4", "[LMNNTest]")
TEMPLATE_TEST_CASE("LMNNFunctionGradientTest4", "[LMNNTest]", float, double)
{
arma::mat dataset;
typedef TestType ElemType;
arma::Mat<ElemType> dataset;
arma::Row<size_t> labels;
if (!data::Load("iris.csv", dataset))
FAIL("Cannot load dataset iris.csv");
if (!data::Load("iris_labels.txt", labels))
FAIL("Cannot load dataset iris_labels.txt");
LMNNFunction<> lmnnfn(dataset, labels, 1, 0.6, 1);
LMNNFunction<arma::Mat<ElemType>> lmnnfn(dataset, labels, 1, 0.6, 1);
// 10 trials with random positions.
for (size_t i = 0; i < 10; ++i)
{
arma::mat coordinates(dataset.n_rows, dataset.n_rows, arma::fill::randu);
arma::Mat<ElemType> coordinates(dataset.n_rows, dataset.n_rows,
arma::fill::randu);
CheckGradient(lmnnfn, coordinates);
}
}
src/mlpack/tests/main_tests/lmnn_test.cpp
+7 -7
View File
@@ -542,7 +542,7 @@ TEST_CASE_METHOD(LMNNTestFixture, "LMNNDiffRegularizationTest",
}
/**
* Ensure that different value of range results in a
* Ensure that different value of update interval results in a
* different output matrix.
*/
TEST_CASE_METHOD(LMNNTestFixture, "LMNNDiffRangeTest",
@@ -573,7 +573,7 @@ TEST_CASE_METHOD(LMNNTestFixture, "LMNNDiffRangeTest",
SetInputParam("input", std::move(inputData));
SetInputParam("labels", std::move(labels));
SetInputParam("linear_scan", (bool) true);
SetInputParam("range", 100);
SetInputParam("update_interval", 100);
RUN_BINDING();
@@ -674,9 +674,9 @@ TEST_CASE_METHOD(LMNNTestFixture, "LMNNDiffPassesTest",
}
/**
* Ensure that number of targets, range, batch size must be always positive
* and regularization, step size, max iterations, rank, passes & tolerance are
* always non-negative
* Ensure that number of targets, update interval, batch size must be always
* positive and regularization, step size, max iterations, rank, passes &
* tolerance are always non-negative.
*/
TEST_CASE_METHOD(LMNNTestFixture, "LMNNBoundsTest",
"[LMNNMainTest][BindingTests]")
@@ -701,12 +701,12 @@ TEST_CASE_METHOD(LMNNTestFixture, "LMNNBoundsTest",
// Reset settings.
ResetSettings();
// Test for range value.
// Test for update interval value.
// Input training data.
SetInputParam("input", inputData);
SetInputParam("labels", labels);
SetInputParam("range", (int) 0);
SetInputParam("update_interval", (int) 0);
REQUIRE_THROWS_AS(RUN_BINDING(), std::runtime_error);