432 lines
14 KiB
Markdown
432 lines
14 KiB
Markdown
# NeighborSearch tutorial (k-nearest-neighbors)
|
|
|
|
Nearest-neighbors search is a common machine learning task. In this setting, we
|
|
have a *query* and a *reference* dataset. For each point in the *query*
|
|
dataset, we wish to know the `k` points in the *reference* dataset which are
|
|
closest to the given query point.
|
|
|
|
Alternately, if the query and reference datasets are the same, the problem can
|
|
be stated more simply: for each point in the dataset, we wish to know the `k`
|
|
nearest points to that point.
|
|
|
|
mlpack provides:
|
|
|
|
- a simple command-line executable to run nearest-neighbors search (and
|
|
furthest-neighbors search)
|
|
- a simple C++ interface to perform nearest-neighbors search (and
|
|
furthest-neighbors search)
|
|
- a generic, extensible, and powerful C++ class (`NeighborSearch`) for complex
|
|
usage
|
|
|
|
## Command-line `mlpack_knn`
|
|
|
|
The simplest way to perform nearest-neighbors search in mlpack is to use the
|
|
`mlpack_knn` executable. *(Note that mlpack also provides bindings to other
|
|
languages, so, e.g., the `knn()` function is available in Python and Julia and
|
|
has the same options. So, any example here can be readily adapted to another
|
|
language that mlpack provides bindings for.)*
|
|
|
|
The `mlpack_knn` program will perform nearest-neighbors search and place the
|
|
resultant neighbors into one file and the resultant distances into another. The
|
|
output files are organized such that the first row corresponds to the nearest
|
|
neighbors of the first query point, with the first column corresponding to the
|
|
nearest neighbor, and so forth.
|
|
|
|
Below are several examples of simple usage (and the resultant output). The `-v`
|
|
option is used so that output is given. Further documentation on each
|
|
individual option can be found by typing
|
|
|
|
```sh
|
|
$ mlpack_knn --help
|
|
```
|
|
|
|
### One dataset, 5 nearest neighbors
|
|
|
|
```sh
|
|
$ mlpack_knn -r dataset.csv -n neighbors_out.csv -d distances_out.csv -k 5 -v
|
|
[INFO ] Loading 'dataset.csv' as CSV data. Size is 3 x 1000.
|
|
[INFO ] Loaded reference data from 'dataset.csv' (3 x 1000).
|
|
[INFO ] Building reference tree...
|
|
[INFO ] Tree built.
|
|
[INFO ] Searching for 5 nearest neighbors with dual-tree kd-tree search...
|
|
[INFO ] 18412 node combinations were scored.
|
|
[INFO ] 54543 base cases were calculated.
|
|
[INFO ] Search complete.
|
|
[INFO ] Saving CSV data to 'neighbors_out.csv'.
|
|
[INFO ] Saving CSV data to 'distances_out.csv'.
|
|
[INFO ]
|
|
[INFO ] Execution parameters:
|
|
[INFO ] distances_file: distances_out.csv
|
|
[INFO ] help: false
|
|
[INFO ] info: ""
|
|
[INFO ] input_model_file: ""
|
|
[INFO ] k: 5
|
|
[INFO ] leaf_size: 20
|
|
[INFO ] naive: false
|
|
[INFO ] neighbors_file: neighbors_out.csv
|
|
[INFO ] output_model_file: ""
|
|
[INFO ] query_file: ""
|
|
[INFO ] random_basis: false
|
|
[INFO ] reference_file: dataset.csv
|
|
[INFO ] seed: 0
|
|
[INFO ] single_mode: false
|
|
[INFO ] tree_type: kd
|
|
[INFO ] verbose: true
|
|
[INFO ] version: false
|
|
[INFO ]
|
|
[INFO ] Program timers:
|
|
[INFO ] computing_neighbors: 0.108968s
|
|
[INFO ] loading_data: 0.006495s
|
|
[INFO ] saving_data: 0.003843s
|
|
[INFO ] total_time: 0.126036s
|
|
[INFO ] tree_building: 0.003442s
|
|
```
|
|
|
|
Convenient program timers are given for different parts of the calculation at
|
|
the bottom of the output, as well as the parameters the simulation was run with.
|
|
Now, if we look at the output files:
|
|
|
|
```sh
|
|
$ head neighbors_out.csv
|
|
862,344,224,43,885
|
|
703,499,805,639,450
|
|
867,472,972,380,601
|
|
397,319,277,443,323
|
|
840,827,865,38,438
|
|
732,876,751,492,616
|
|
563,222,569,985,940
|
|
361,97,928,437,79
|
|
547,695,419,961,716
|
|
982,113,689,843,634
|
|
|
|
$ head distances_out.csv
|
|
5.986076164057e-02,7.664920518084e-02,1.116050961847e-01,1.155595474371e-01,1.169810085522e-01
|
|
7.532635022982e-02,1.012564715841e-01,1.127846944644e-01,1.209584396720e-01,1.216543647014e-01
|
|
7.659571546879e-02,1.014588981948e-01,1.025114621511e-01,1.128082429187e-01,1.131659758673e-01
|
|
2.079405647909e-02,4.710724516732e-02,7.597622408419e-02,9.171977778898e-02,1.037033340864e-01
|
|
7.082206779700e-02,9.002355499742e-02,1.044181406406e-01,1.093149568834e-01,1.139700558608e-01
|
|
5.688056488896e-02,9.478072514474e-02,1.085637706630e-01,1.114177921451e-01,1.139370265105e-01
|
|
7.882260880455e-02,9.454474078041e-02,9.724494179950e-02,1.023829575445e-01,1.066927013814e-01
|
|
7.005321598247e-02,9.131417221561e-02,9.498248889074e-02,9.897964162308e-02,1.121202216165e-01
|
|
5.295654132754e-02,5.509877761894e-02,8.108227366619e-02,9.785461174861e-02,1.043968140367e-01
|
|
3.992859920333e-02,4.471418646159e-02,7.346053904990e-02,9.181982339584e-02,9.843075910782e-02
|
|
```
|
|
|
|
So, the nearest neighbor to point 0 is point 862, with a distance of
|
|
`5.986076164057e-02`. The second nearest neighbor to point 0 is point 344, with
|
|
a distance of `7.664920518084e-02`. The third nearest neighbor to point 5 is
|
|
point 751, with a distance of `1.085637706630e-01`.
|
|
|
|
### Query and reference dataset, 10 nearest neighbors
|
|
|
|
```sh
|
|
$ mlpack_knn -q query_dataset.csv -r reference_dataset.csv \
|
|
> -n neighbors_out.csv -d distances_out.csv -k 10 -v
|
|
[INFO ] Loading 'reference_dataset.csv' as CSV data. Size is 3 x 1000.
|
|
[INFO ] Loaded reference data from 'reference_dataset.csv' (3 x 1000).
|
|
[INFO ] Building reference tree...
|
|
[INFO ] Tree built.
|
|
[INFO ] Loading 'query_dataset.csv' as CSV data. Size is 3 x 50.
|
|
[INFO ] Loaded query data from 'query_dataset.csv' (3x50).
|
|
[INFO ] Searching for 10 nearest neighbors with dual-tree kd-tree search...
|
|
[INFO ] Building query tree...
|
|
[INFO ] Tree built.
|
|
[INFO ] Search complete.
|
|
[INFO ] Saving CSV data to 'neighbors_out.csv'.
|
|
[INFO ] Saving CSV data to 'distances_out.csv'.
|
|
[INFO ]
|
|
[INFO ] Execution parameters:
|
|
[INFO ] distances_file: distances_out.csv
|
|
[INFO ] help: false
|
|
[INFO ] info: ""
|
|
[INFO ] input_model_file: ""
|
|
[INFO ] k: 10
|
|
[INFO ] leaf_size: 20
|
|
[INFO ] naive: false
|
|
[INFO ] neighbors_file: neighbors_out.csv
|
|
[INFO ] output_model_file: ""
|
|
[INFO ] query_file: query_dataset.csv
|
|
[INFO ] random_basis: false
|
|
[INFO ] reference_file: reference_dataset.csv
|
|
[INFO ] seed: 0
|
|
[INFO ] single_mode: false
|
|
[INFO ] tree_type: kd
|
|
[INFO ] verbose: true
|
|
[INFO ] version: false
|
|
[INFO ]
|
|
[INFO ] Program timers:
|
|
[INFO ] computing_neighbors: 0.022589s
|
|
[INFO ] loading_data: 0.003572s
|
|
[INFO ] saving_data: 0.000755s
|
|
[INFO ] total_time: 0.032197s
|
|
[INFO ] tree_building: 0.002590s
|
|
```
|
|
|
|
### One dataset, 3 nearest neighbors, leaf size of 15 points
|
|
|
|
```sh
|
|
$ mlpack_knn -r dataset.csv -n neighbors_out.csv -d distances_out.csv -k 3 -l 15 -v
|
|
[INFO ] Loading 'dataset.csv' as CSV data. Size is 3 x 1000.
|
|
[INFO ] Loaded reference data from 'dataset.csv' (3 x 1000).
|
|
[INFO ] Building reference tree...
|
|
[INFO ] Tree built.
|
|
[INFO ] Searching for 3 nearest neighbors with dual-tree kd-tree search...
|
|
[INFO ] 19692 node combinations were scored.
|
|
[INFO ] 36263 base cases were calculated.
|
|
[INFO ] Search complete.
|
|
[INFO ] Saving CSV data to 'neighbors_out.csv'.
|
|
[INFO ] Saving CSV data to 'distances_out.csv'.
|
|
[INFO ]
|
|
[INFO ] Execution parameters:
|
|
[INFO ] distances_file: distances_out.csv
|
|
[INFO ] help: false
|
|
[INFO ] info: ""
|
|
[INFO ] input_model_file: ""
|
|
[INFO ] k: 3
|
|
[INFO ] leaf_size: 15
|
|
[INFO ] naive: false
|
|
[INFO ] neighbors_file: neighbors_out.csv
|
|
[INFO ] output_model_file: ""
|
|
[INFO ] query_file: ""
|
|
[INFO ] random_basis: false
|
|
[INFO ] reference_file: dataset.csv
|
|
[INFO ] seed: 0
|
|
[INFO ] single_mode: false
|
|
[INFO ] tree_type: kd
|
|
[INFO ] verbose: true
|
|
[INFO ] version: false
|
|
[INFO ]
|
|
[INFO ] Program timers:
|
|
[INFO ] computing_neighbors: 0.059020s
|
|
[INFO ] loading_data: 0.002791s
|
|
[INFO ] saving_data: 0.002369s
|
|
[INFO ] total_time: 0.069277s
|
|
[INFO ] tree_building: 0.002713s
|
|
```
|
|
|
|
Further documentation on options should be found by using the `--help` option.
|
|
|
|
## The `KNN` class
|
|
|
|
The `KNN` class is, specifically, a typedef of the more extensible
|
|
`NeighborSearch` class, querying for nearest neighbors using the Euclidean
|
|
distance.
|
|
|
|
```c++
|
|
using KNN = NeighborSearch<NearestNeighborSort, EuclideanDistance>;
|
|
```
|
|
|
|
Using the `KNN` class is particularly simple; first, the object must be
|
|
constructed and given a dataset. Then, the method is run, and two matrices are
|
|
returned: one which holds the indices of the nearest neighbors, and one which
|
|
holds the distances of the nearest neighbors. These are of the same structure
|
|
as the output `--neighbors_file` and `--distances_file` for the command-line
|
|
program (see above). A handful of examples of simple usage of the KNN class are
|
|
given below.
|
|
|
|
### 5 nearest neighbors on a single dataset
|
|
|
|
```c++
|
|
#include <mlpack.hpp>
|
|
|
|
using namespace mlpack;
|
|
|
|
// Our dataset matrix, which is column-major.
|
|
extern arma::mat data;
|
|
|
|
KNN a(data);
|
|
|
|
// The matrices we will store output in.
|
|
arma::Mat<size_t> resultingNeighbors;
|
|
arma::mat resultingDistances;
|
|
|
|
a.Search(5, resultingNeighbors, resultingDistances);
|
|
```
|
|
|
|
The output of the search is stored in `resultingNeighbors` and
|
|
`resultingDistances`.
|
|
|
|
### 10 nearest neighbors on a query and reference dataset
|
|
|
|
```c++
|
|
#include <mlpack.hpp>
|
|
|
|
using namespace mlpack;
|
|
|
|
// Our dataset matrices, which are column-major.
|
|
extern arma::mat queryData, referenceData;
|
|
|
|
KNN a(referenceData);
|
|
|
|
// The matrices we will store output in.
|
|
arma::Mat<size_t> resultingNeighbors;
|
|
arma::mat resultingDistances;
|
|
|
|
a.Search(queryData, 10, resultingNeighbors, resultingDistances);
|
|
```
|
|
|
|
### Naive (exhaustive) search for 6 nearest neighbors on one dataset
|
|
|
|
This example uses the `O(n^2)` naive search (not the tree-based search).
|
|
|
|
```c++
|
|
#include <mlpack.hpp>
|
|
|
|
using namespace mlpack;
|
|
|
|
// Our dataset matrix, which is column-major.
|
|
extern arma::mat dataset;
|
|
|
|
KNN a(dataset, true);
|
|
|
|
// The matrices we will store output in.
|
|
arma::Mat<size_t> resultingNeighbors;
|
|
arma::mat resultingDistances;
|
|
|
|
a.Search(6, resultingNeighbors, resultingDistances);
|
|
```
|
|
|
|
Needless to say, naive search can be very slow...
|
|
|
|
## The extensible `NeighborSearch` class
|
|
|
|
The `NeighborSearch` class is very extensible, having the following template
|
|
arguments:
|
|
|
|
```c++
|
|
template<
|
|
typename SortPolicy = NearestNeighborSort,
|
|
typename DistanceType = EuclideanDistance,
|
|
typename MatType = arma::mat,
|
|
template<typename TreeDistanceType,
|
|
typename TreeStatType,
|
|
typename TreeMatType> class TreeType = KDTree,
|
|
template<typename RuleType> class TraversalType =
|
|
TreeType<DistanceType, NeighborSearchStat<SortPolicy>,
|
|
MatType>::template DualTreeTraverser>
|
|
>
|
|
class NeighborSearch;
|
|
```
|
|
|
|
By choosing different components for each of these template classes, a very
|
|
arbitrary neighbor searching object can be constructed. Note that each of these
|
|
template parameters have defaults, so it is not necessary to specify each one.
|
|
|
|
### `SortPolicy` policy class
|
|
|
|
The `SortPolicy` template parameter allows specification of how the
|
|
NeighborSearch object will decide which points are to be searched for. The
|
|
`NearestNeighborSort` class is a well-documented example. A custom `SortPolicy`
|
|
class must implement the same methods which `NearestNeighborSort` does:
|
|
|
|
```c++
|
|
static size_t SortDistance(const arma::vec& list, double newDistance);
|
|
|
|
static bool IsBetter(const double value, const double ref);
|
|
|
|
template<typename TreeType>
|
|
static double BestNodeToNodeDistance(const TreeType* queryNode,
|
|
const TreeType* referenceNode);
|
|
|
|
template<typename TreeType>
|
|
static double BestPointToNodeDistance(const arma::vec& queryPoint,
|
|
const TreeType* referenceNode);
|
|
|
|
static const double WorstDistance();
|
|
|
|
static const double BestDistance();
|
|
```
|
|
|
|
The `FurthestNeighborSort` class is another implementation, which is used to
|
|
create the `KFN` typedef class, which finds the furthest neighbors, as opposed
|
|
to the nearest neighbors.
|
|
|
|
## `DistanceType` policy class
|
|
|
|
The `DistanceType` policy class allows the neighbor search to take place in any
|
|
arbitrary metric space. The `LMetric` class is a good example implementation.
|
|
A `DistanceType` class must provide the following functions:
|
|
|
|
```c++
|
|
// Empty constructor is required.
|
|
DistanceType();
|
|
|
|
// Compute the distance between two points.
|
|
template<typename VecType>
|
|
double Evaluate(const VecType& a, const VecType& b);
|
|
```
|
|
|
|
Internally, the `NeighborSearch` class keeps an instantiated `DistanceType` class
|
|
(which can be given in the constructor). This is useful for a distance metric
|
|
like the Mahalanobis distance (`MahalanobisDistance`), which must store state
|
|
(the covariance matrix). Therefore, you can write a non-static DistanceType
|
|
class and use it seamlessly with `NeighborSearch`.
|
|
|
|
For more information on the `DistanceType` policy, see the [documentation for
|
|
`DistanceType`](../developer/distances.md).
|
|
|
|
### `MatType` policy class
|
|
|
|
The `MatType` template parameter specifies the type of data matrix used. This
|
|
type must implement the same operations as an Armadillo matrix, and so standard
|
|
choices are `arma::mat` and `arma::sp_mat`.
|
|
|
|
### `TreeType` policy class
|
|
|
|
The NeighborSearch class allows great extensibility in the selection of the type
|
|
of tree used for search. This type must follow the typical mlpack TreeType
|
|
policy, documented [here](../developer/trees.md).
|
|
|
|
Typical choices might include `KDTree`, `BallTree`, `StandardCoverTree`,
|
|
`RTree`, or `RStarTree`. It is easily possible to make your own tree type for
|
|
use with NeighborSearch; consult the [TreeType
|
|
documentation](../developer/trees.md) for more details.
|
|
|
|
An example of using the `NeighborSearch` class with a ball tree is given below.
|
|
|
|
```c++
|
|
// Construct a NeighborSearch object with ball bounds.
|
|
NeighborSearch<
|
|
NearestNeighborSort,
|
|
EuclideanDistance,
|
|
arma::mat,
|
|
BallTree
|
|
> neighborSearch(dataset);
|
|
```
|
|
|
|
### `TraverserType` policy class
|
|
|
|
The last template parameter the `NeighborSearch` class offers is the
|
|
`TraverserType` class. The `TraverserType` class holds the strategy used to
|
|
traverse the trees in either single-tree or dual-tree search mode. By default,
|
|
it is set to use the default traverser of the given `TreeType` (which is the
|
|
member `TreeType::DualTreeTraverser`).
|
|
|
|
This class must implement the following two methods:
|
|
|
|
```c++
|
|
// Instantiate with a given RuleType.
|
|
TraverserType(RuleType& rule);
|
|
|
|
// Traverse with two trees.
|
|
void Traverse(TreeType& queryNode, TreeType& referenceNode);
|
|
```
|
|
|
|
The `RuleType` class provides the following functions for use in the traverser:
|
|
|
|
```c++
|
|
// Evaluate the base case between two points.
|
|
double BaseCase(const size_t queryIndex, const size_t referenceIndex);
|
|
|
|
// Score the two nodes to see if they can be pruned, returning DBL_MAX if so.
|
|
double Score(TreeType& queryNode, TreeType& referenceNode);
|
|
```
|
|
|
|
Note also that any traverser given must satisfy the definition of a pruning
|
|
dual-tree traversal given in the paper "Tree-independent dual-tree algorithms".
|
|
|
|
## Further documentation
|
|
|
|
For further documentation on the NeighborSearch class, consult the comments in
|
|
the source code, found in `mlpack/methods/neighbor_search/`.
|