diff --git a/doc/user/methods/decision_tree_regressor.md b/doc/user/methods/decision_tree_regressor.md new file mode 100644 index 0000000000..b5ed0edafb --- /dev/null +++ b/doc/user/methods/decision_tree_regressor.md @@ -0,0 +1,620 @@ +## `DecisionTreeRegressor` + +The `DecisionTreeRegressor` class implements a decision tree regressor that +supports numerical and categorical features, by default using MSE (minimum +squared error) to choose which feature to split on. The class offers several +template parameters and runtime options that can be used to control the behavior +of the tree. + +The `DecisionTreeRegressor` class is useful for regressions; i.e., predicting +_continuous values_ (`0.3`, `1.2`, etc.). For predicting _discrete labels_ +(classification), see [`DecisionTree`](#decision_tree). + +#### Basic usage example excerpt: + +```c++ +DecisionTreeRegressor tree; // Step 1: construct object. +tree.Train(data, responses, 3); // Step 2: train model. +tree.Predict(test_data, test_predictions); // Step 3: predict values with model. +``` + +#### Quick links: + + * [Constructors](#constructors): create `DecisionTreeRegressor` objects. + * [`Train()`](#training): train model. + * [`Predict()`](#prediction): predict values with a trained model. + * [Other functionality](#other-functionality) for loading, saving, and + inspecting. + * [Examples](#simple-examples) of simple usage and links to detailed example + projects. + * [Template parameters](#advanced-functionality-template-parameters) for custom + behavior. + +#### See also: + + * [`DecisionTree`](#decision_tree) + * [Random forests](#random_forests) + * [mlpack regression techniques](#mlpack_regression_techniques) + * [Decision tree on Wikipedia](https://en.wikipedia.org/wiki/Decision_tree) + * [Decision tree learning on Wikipedia](https://en.wikipedia.org/wiki/Decision_tree_learning) + +### Constructors + +Construct a `DecisionTreeRegressor` object using one of the constructors below. +Defaults and types are detailed in the [Constructor +Parameters](#constructor-parameters) section below. + +#### Forms: + + * `DecisionTreeRegressor()` + - Initialize tree without training. + - You will need to call [`Train()`](#training) later to train the tree before + calling [`Predict()`](#prediction). + +--- + + * `DecisionTreeRegressor(data, responses)` + * `DecisionTreeRegressor(data, responses, weights)` + * `DecisionTreeRegressor(data, responses, minimumLeafSize, minimumGainSplit, maximumDepth)` + * `DecisionTreeRegressor(data, responses, weights, minimumLeafSize, minimumGainSplit, maximumDepth)` + - Train on numerical-only data (optionally with instance weights). + - If hyperparameters are not specified, default values are used. + - `responses` should be a vector of length `data.n_cols`, containing + continuous real values corresponding to the response for each data point. + - If specified, `weights` should be a vector of length `data.n_cols`, + containing instance weights for each point in `data`. + +--- + + * `DecisionTreeRegressor(data, datasetInfo, responses)` + * `DecisionTreeRegressor(data, datasetInfo, responses, weights)` + * `DecisionTreeRegressor(data, datasetInfo, responses, minimumLeafSize, minimumGainSplit, maximumDepth)` + * `DecisionTreeRegressor(data, datasetInfo, responses, weights, minimumLeafSize, minimumGainSplit, maximumDepth)` + - Train on mixed categorical data (optionally with instance weights). + - If hyperparameters are not specified, default values are used. + - `responses` should be a vector of length `data.n_cols`, containing + continuous real values corresponding to the response for each data point. + - If specified, `weights` should be a vector of length `data.n_cols`, + containing instance weights for each point in `data`. + +--- + +#### Constructor parameters: + + + +| **name** | **type** | **description** | **default** | +|----------|----------|-----------------|-------------| +| `data` | [`arma::mat`](../matrices.md) | [Column-major](../matrices.md) training matrix. | _(N/A)_ | +| `datasetInfo` | [`data::DatasetInfo`](../../tutorials/datasetmapper.md) | Dataset information, specifying type information for each dimension. | _(N/A)_ | +| `labels` | [`arma::Row`]('../matrices.md') | Training labels, between `0` and `numClasses - 1` (inclusive). Should have length `data.n_cols`. | _(N/A)_ | +| `weights` | [`arma::rowvec`]('../matrices.md') | Weights for each training point. Should have length `data.n_cols`. | _(N/A)_ | +| `numClasses` | `size_t` | Number of classes in the dataset. | _(N/A)_ | +| `minimumLeafSize` | `size_t` | Minimum number of points in each leaf node. | `10` | +| `minimumGainSplit` | `double` | Minimum gain for a node to split. | `1e-7` | +| `maximumDepth` | `size_t` | Maximum depth for the tree. (0 means no limit.) | `0` | + + * Setting `minimumLeafSize` too small (e.g. `1`) may cause the tree to overfit + to its training data, and may create a very large tree. However, setting it + too large may cause the tree to be very small and underfit. + * `minimumGainSplit` has similar behavior: if it is too small, the tree may + overfit; if too large, it may underfit. + +***Note:*** different types can be used for `data`, `responses`, and `weights` +(e.g., `arma::fmat`, `arma::sp_mat`). However, the element type of `data`, +`responses`, and `weights` all must match; for example, if `data` has type +`arma::fmat`, then `responses` and `weights` must have type `arma::frowvec`. + +### Training + +If training is not done as a part of the constructor call, it can be done with +one of the versions of the `Train()` member function. For an instance of +`DecisionTree` named `tree`, the following functions for training are available: + + * `tree.Train(data, responses)` + * `tree.Train(data, responses, weights)` + * `tree.Train(data, responses, minimumLeafSize, minimumGainSplit, maximumDepth)` + * `tree.Train(data, responses, weights, minimumLeafSize, minimumGainSplit, maximumDepth)` + - Train on numerical-only data (optionally with instance weights). + - If hyperparameters are not specified, default values are used. + - `responses` should be a vector of length `data.n_cols`, containing + continuous real values corresponding to the response for each data point. + - If specified, `weights` should be a vector of length `data.n_cols`, + containing instance weights for each point in `data`. + - Returns a `double` with the final gain of the tree (the Gini gain, unless a + different [`FitnessFunction` template parameter](#fully-custom-behavior) is + specified. + +--- + + * `tree.Train(data, datasetInfo, responses)` + * `tree.Train(data, datasetInfo, responses, weights)` + * `tree.Train(data, datasetInfo, responses, minimumLeafSize, minimumGainSplit, maximumDepth)` + * `tree.Train(data, datasetInfo, responses, weights, minimumLeafSize, minimumGainSplit, maximumDepth)` + - Train on mixed categorical data (optionally with instance weights). + - If hyperparameters are not specified, default values are used. + - `responses` should be a vector of length `data.n_cols`, containing + continuous real values corresponding to the response for each data point. + - If specified, `weights` should be a vector of length `data.n_cols`, + containing instance weights for each point in `data`. + - Returns a `double` with the final gain of the tree (the Gini gain, unless a + different [`FitnessFunction` template parameter](#fully-custom-behavior) is + specified. + +--- + +Types of each argument are the same as in the table for constructors +[above](#constructor-parameters). + +***Note***: training is not incremental. A second call to `Train()` will +retrain the decision tree from scratch. + +### Prediction + +Once a `DecisionTreeRegressor` is trained, the `Predict()` member function can +be used to make class predictions for new data. Defaults and types are detailed +in the [Prediction Parameters](#prediction-parameters) section below. + +#### Forms: + + * `double predictedValue = tree.Predict(point)` + - ***(Single-point)*** + - Predict and return the value for a single point. + +--- + + * `tree.Predict(data, predictions)` + - ***(Multi-point)*** + - Predict and return values for every point in the given matrix `data`. + - The predictions for each point are stored in `predictions`, which is set to + length `data.n_cols`. + - The prediction for data point `i` can be accessed with `predictions[i]`. + +--- + +#### Prediction Parameters: + +| **usage** | **name** | **type** | **description** | +|-----------|----------|----------|-----------------| +| _single-point_ | `point` | [`arma::vec`](../matrices.md) | Single point for prediction. | +|||| +| _multi-point_ | `data` | [`arma::mat`](../matrices.md) | Set of [column-major](../matrices.md) points for prediction. | +| _multi-point_ | `predictions` | [`arma::rowvec&`](../matrices.md) | Vector to store predictions into. | + +***Note:*** different types can be used for `data` and `point` (e.g. +`arma::fmat`, `arma::sp_mat`, `arma::sp_vec`, etc.). However, the element type +that is used should be the same type that was used for training. + +### Other functionality + + + + * A `DecisionTreeRegressor` can be serialized with + [`data::Save()`](../formats.md) and [`data::Load()`](../formats.md). + + * `tree.NumChildren()` will return a `size_t` indicating the number of children + in the node `tree`. + + * `tree.NumLeaves()` will return the total number of leaf nodes that are + descendants of the node `tree`. + + * `tree.Child(i)` will return a `DecisionTreeRegressor` object representing the + `i`th child of the node `tree`. + + * `tree.SplitDimension()` returns a `size_t` indicating which dimension the + node `tree` splits on. + +For complete functionality, the [source +code](/src/mlpack/methods/decision_tree/decision_tree_regressor.hpp) can be +consulted. Each method is fully documented. + +### Simple examples + +Train a decision tree regressor on random numeric data and make predictions on a +test set: + +```c++ +// 1000 random points in 10 dimensions. +arma::mat dataset(10, 1000); +// Random responses, normally distributed, for each point. +arma::rowvec responses = arma::randn(1000); + +// Train in the constructor. +DecisionTreeRegressor<> tree(data, responses); + +// Create test data (500 points). +arma::mat testDataset(10, 500); +arma::rowvec predictions; +tree.Predict(testDataset, predictions); +// Now `predictions` holds predictions for the test dataset. + +// Print some information about the test predictions. +std::cout << arma::accu(predictions > 3) << " test points predicted to have " + << "responses greater than 3." << std::endl; +std::cout << arma::accu(predictions < 0) << " test points predicted to have " + << "negative responses." << std::endl; +``` + +--- + +Train a decision tree regressor on random mixed categorical data. + +```c++ +// Load a categorical dataset. +arma::mat data; +data::DatasetInfo info; +// See https://datasets.mlpack.org/telecom_churn.arff. +data::Load("telecom_churn.arff", data, info, true); + +arma::rowvec responses; +// See https://datasets.mlpack.org/telecom_churn.responses.csv. +data::Load("telecom_churn.responses.csv", responses, true); + +// Split data into training set (80%) and test set (20%). +arma::mat trainData, testData; +arma::rowvec trainResponses, testResponses; +data::Split(data, responses, trainData, testData, trainResponses, testResponses, + 0.2); + +// Create the tree. +DecisionTreeRegressor<> tree; +// Train on the given dataset, specifying a minimum gain of 1e-6 and keeping the +// default minimum leaf size. +const double mse = tree.Train(trainData, info, trainResponses, + 10 /* minimum leaf size */, 1e-6 /* minimum gain */); +// Print the MSE of the trained tree. +std::cout << "MSE of trained tree is " << mse << "." << std::endl; + +// Compute prediction on the first test point. +const double firstPrediction = tree.Predict(testData.col(0)); +std::cout << "Predicted value for first test point is " << firstPrediction + << "." << std::endl; + +// Compute predictions on test data. +arma::rowvec testPredictions; +tree.Predict(testData, testPredictions); + +// Compute the average error on the test set. +const double testAverageError = arma::mean(testResponses - testPredictions); +std::cout << "Average error on test set: " << testAverageError << "." + << std::endl; +``` + +--- + +Load a tree and print some information about it. + +```c++ +DecisionTreeRegression<> tree; +// This call assumes a tree called "tree" has already been saved to `tree.bin` +// with `data::Save()`. +data::Load("tree.bin", "tree", tree, true); + +std::cout << "Information about the DecisionTreeRegressor in `tree.bin`:" + << std::endl; +std::cout << " * The root node has " << tree.NumChildren() << " children." + << std::endl; +std::cout << " * The tree has " << tree.NumLeaves() << " leaves." << std::endl; +if (tree.NumChildren() > 0) +{ + for (size_t i = 0; i < tree.NumChildren(); ++i) + { + std::cout << " * Child " << i << " of the root has " + << tree.Child(i).NumLeaves() << " leaves in its subtree." << std::endl; + } +} +``` + +### Advanced Functionality: Template Parameters + +#### Using different element types. + +`DecisionTreeRegressor`'s constructors, `Train()`, and `Predict()` functions +support any data type, so long as it supports the Armadillo matrix API. So, for +instance, learning can be done on single-precision floating-point data: + +```c++ +// 1000 random points in 10 dimensions. +arma::fmat dataset(10, 1000); +// Random responses for each point, with a normal distribution. +arma::frowvec responses = arma::randn(1000); + +// Train in the constructor. +DecisionTreeRegressor<> tree(data, responses, 5); + +// Create test data (500 points). +arma::fmat testDataset(10, 500); +arma::frowvec predictions; +tree.Classify(testDataset, predictions); +// Now `predictions` holds predictions for the test dataset. + +// Print some information about the test predictions. +std::cout << arma::accu(predictions > 1) << " test points predicted to have " + << "value greater than 1." << std::endl; +``` + +--- + +#### Fully custom behavior. + +The `DecisionTreeRegressor<>` class also supports several template parameters, +which can be used for custom behavior during learning. The full signature of +the class is as follows: + +```c++ +DecisionTreeRegressor +``` + + * `FitnessFunction`: the measure of goodness to use when deciding on tree + splits + * `NumericSplitType`: the strategy used for finding splits on numeric data + dimensions + * `CategoricalSplitType`: the strategy used for finding splits on categorical + data dimensions + * `DimensionSelectionType`: the strategy used for proposing dimensions to + attempt to split on + * `NoRecursion`: a boolean indicating whether or not to build a tree or a stump + (one level tree) + +Below, details are given for the requirements of each of these template types. + +--- + +#### `FitnessFunction` + + * Specifies the fitness function to use when learning a decision tree. + * The `MSEGain` _(default)_ and `MADGain` classes are available for drop-in + usage. + * A custom class must implement three functions: + +```c++ +// You can use this as a starting point for implementation. +class CustomFitnessFunction +{ + // Compute the gain for the given vector of values, where `values[i]` has an + // associated instance weight `weights[i]`. + // + // `RowType` and `WeightVecType` will be vector types following the Armadillo + // API. If `UseWeights` is `false`, then the `weights` vector should be + // ignored (e.g. the responses are not weighted). + // + // In the version with `begin` and `end` parameters, only the subset between + // `labels[begin]` and `labels[end]` (inclusive) should be considered. + template + double Evaluate(const RowType& labels, + const WeightVecType& weights); + + template + double Evaluate(const RowType& labels, + const WeightVecType& weights, + const size_t begin, + const size_t end); + + // Return the output value for prediction for a leaf node whose training + // values are made up of the values in the vector `responses` (optionally with + // associated instance weights `weights`). + // + // `ResponsesType` and `WeightsType` will be vector types following the + // Armadillo API. If `UseWeights` is `false`, then the `weights` vector + // should be ignored (e.g. the responses are not weighted). + template + double OutputLeafValue(const ResponsesType& responses, + const WeightsType& weights); +}; +``` + +***Note:*** this API differs from the `FitnessFunction` API required for +[`DecisionTree`](#decision_tree)! + +--- + + * `NumericSplitType` + - Specifies the strategy to be used during training when splitting a numeric + feature. + - The `BestBinaryNumericSplit` _(default)_ class is available for drop-in + usage and finds the best binary (two-way) split among all possible binary + splits. + - The `RandomBinaryNumericSplit` class is available for drop-in usage and + will select a split randomly between the minimum and maximum values of a + dimension. It is very efficient but does not yield splits that maximize + the gain. (Used by the `ExtraTrees` variant of + [`RandomForest`](#random_forest).) + - A custom class must implement three functions and have an internal + structure `AuxiliarySplitInfo` that is used at classification time. + +```c++ +class CustomNumericSplit +{ + public: + // If a split with better resulting gain than `bestGain` is found, then + // information about the new, better split should be stored in `splitInfo` and + // `aux`. Specifically, a split is better than `bestGain` if the sum of the + // gains that the children will have (call this `sumChildrenGains`) is + // sufficiently better than the gain of the unsplit node (call this + // `unsplitGain`): + // + // split if `sumChildrenGains - unsplitGain > bestGain`, and + // `sumChildrenGains - unsplitGain > minimumGainSplit`, and + // each child will have at least `minimumLeafSize` points + // + // The new best split value should be returned (or anything greater than or + // equal to `bestGain` if no better split is found). + // + // If a new best split is found, then `splitInfo` and `aux` should be + // populated with the information that will be needed for + // `CalculateDirection()` to successfully choose the child for a given point. + // `splitInfo` should be set to a vector of length 1. The format of `aux` is + // arbitrary and is detailed more below. + // + // If `UseWeights` is false, the vector `weights` should be ignored. + // Otherwise, they are instance weighs for each value in `data` (one dimension + // of the input data). + template + double SplitIfBetter(const double bestGain, + const VecType& data, + const ResponsesType& responses, + const WeightVecType& weights, + const size_t minimumLeafSize, + const double minimumGainSplit, + arma::vec& splitInfo, + AuxiliarySplitInfo& aux, + FitnessFunction& function); + + // Return the number of children for a given split (stored as the single + // element from `splitInfo` and auxiliary data `aux` in `SplitIfBetter()`). + size_t NumChildren(const double& splitInfo, + const AuxiliarySplitInfo& aux); + + // Given a point with value `point`, and split information `splitInfo` and + // `aux`, return the index of the child that corresponds to the point. So, + // e.g., if the split type was a binary split on the value `splitInfo`, you + // might return `0` if `point < splitInfo`, and `1` otherwise. + template + static size_t CalculateDirection( + const ElemType& point, + const double& splitInfo, + const AuxiliarySplitInfo& /* aux */); + + // This class can hold any extra data that is necessary to encode a split. It + // should only be non-empty if a single `double` value cannot be used to hold + // the information corresponding to a split. + class AuxiliarySplitInfo { }; +}; +``` + +***Note:*** this API differs from the `NumericSplitType` API required for +[`DecisionTree`](#decision_tree)! + +--- + +```c++ +class CustomCategoricalSplit +{ + public: + // If a split with better resulting gain than `bestGain` is found, then + // information about the new, better split should be stored in `splitInfo` and + // `aux`. Specifically, a split is better than `bestGain` if the sum of the + // gains that the children will have (call this `sumChildrenGains`) is + // sufficiently better than the gain of the unsplit node (call this + // `unsplitGain`): + // + // split if `sumChildrenGains - unsplitGain > bestGain`, and + // `sumChildrenGains - unsplitGain > minimumGainSplit`, and + // each child will have at least `minimumLeafSize` points + // + // The new best split value should be returned (or anything greater than or + // equal to `bestGain` if no better split is found). + // + // If a new best split is found, then `splitInfo` and `aux` should be + // populated with the information that will be needed for + // `CalculateDirection()` to successfully choose the child for a given point. + // `splitInfo` should be set to a vector of length 1. The format of `aux` is + // arbitrary and is detailed more below. + // + // If `UseWeights` is false, the vector `weights` should be ignored. + // Otherwise, they are instance weighs for each value in `data` (one + // categorical dimension of the input data, which takes values between `0` and + // `numCategories - 1`). + template + static double SplitIfBetter( + const double bestGain, + const VecType& data, + const size_t numCategories, + const ResponsesType& labels, + const WeightVecType& weights, + const size_t minimumLeafSize, + const double minimumGainSplit, + arma::vec& splitInfo, + AuxiliarySplitInfo& aux, + FitnessFunction& fitnessFunction); + + // Return the number of children for a given split (stored as the single + // element from `splitInfo` and auxiliary data `aux` in `SplitIfBetter()`). + size_t NumChildren(const double& splitInfo, + const AuxiliarySplitInfo& aux); + + // Given a point with (categorical) value `point`, and split information + // `splitInfo` and `aux`, return the index of the child that corresponds to + // the point. So, e.g., for `AllCategoricalSplit`, which splits a categorical + // dimension into one child for each category, this simply returns `point`. + template + static size_t CalculateDirection( + const ElemType& point, + const double& splitInfo, + const AuxiliarySplitInfo& /* aux */); + + // This class can hold any extra data that is necessary to encode a split. It + // should only be non-empty if a single `double` value cannot be used to hold + // the information corresponding to a split. + class AuxiliarySplitInfo { }; +}; +``` + +***Note:*** this API differs from the `CategoricalSplitType` API required for +[`DecisionTree`](#decision_tree)! + +--- + +#### `DimensionSelectionType` + + * When splitting a decision tree, `DimensionSelectionType` proposes possible + dimensions to try splitting on. + * `AllDimensionSplit` _(default)_ is available for drop-in usage and proposes + all dimensions for splits. + * `MultipleRandomDimensionSelect`, constructed as + `MultipleRandomDimensionSplit(n)`, selects `n` different random dimensions as + candidates at each decision tree node. + * Each `DecisionTreeRegressor` [constructor](#constructors) optionally accepts + an instantiated `DimensionSelectionType` object as a last parameter (after + `maximumDepth`), in case some internal state in the dimension selection + mechanism is required. + * A custom class must implement three simple functions: + +```c++ +class CustomDimensionSelect +{ + public: + // Get the first dimension to try. + // This should return a value between `0` and `data.n_rows`. + size_t Begin(); + + // Get the next dimension to try. Note that internal state can be used to + // track which candidate dimension is currently being looked at. + // This should return a value between `0` and `data.n_rows`. + size_t Next(); + + // Get a value indicating that all dimensions have been tried. + size_t End() const; + + // The usage pattern of `DimensionSelectionType` by `DecisionTree` is as + // follows, assuming that `dim` is an instantiated `DimensionSelectionType` + // object: + // + // for (size_t dim = dim.Begin(); dim != dim.End(); dim = dim.Next()) + // { + // // ... try to split on dimension `dim` ... + // } +}; +``` + +--- + +#### `NoRecursion` + + * A `bool` value that indicates whether a decision tree should be + constructed recursively. + * If `true` _(default)_, a full decision tree will be built. + * If `false`, only the root node will be split (producing a decision + stump). diff --git a/src/mlpack/methods/decision_tree/best_binary_numeric_split.hpp b/src/mlpack/methods/decision_tree/best_binary_numeric_split.hpp index 8ba08b1d43..34d397c829 100644 --- a/src/mlpack/methods/decision_tree/best_binary_numeric_split.hpp +++ b/src/mlpack/methods/decision_tree/best_binary_numeric_split.hpp @@ -103,7 +103,7 @@ class BestBinaryNumericSplit * @param splitInfo Stores split information on a successful split. * @param aux Auxiliary split information, which may be modified on a * successful split. - * @param fitnessFunction The FitnessFunction object instance. It it used to + * @param fitnessFunction The FitnessFunction object instance. It is used to * evaluate the gain for the split. */ template void Predict(const MatType& data, - arma::Row& predictions) const; + arma::Row& predictions) const; /** * Serialize the tree.