450 lines
13 KiB
Markdown
450 lines
13 KiB
Markdown
# Linear/ridge regression tutorial
|
|
|
|
Linear regression and ridge regression are simple machine learning techniques
|
|
that aim to estimate the parameters of a linear model. Assuming we have `n`
|
|
*predictor* points `x_i`, of dimensionality `d`, and `n` responses `y_i`, we are
|
|
trying to estimate the best fit for `b_i` with `0 <= i <= d` in the linear model
|
|
|
|
```
|
|
y_i = b_0 + sum_j (b_j x_ij)
|
|
```
|
|
|
|
for each predictor `x_i` and response `y_i`. If we take each predictor `x_i` as
|
|
a row in the matrix `X` and each response `y_i` as an entry of the vector `y`,
|
|
we can represent the model in vector form:
|
|
|
|
```
|
|
y = Xb + b_0
|
|
```
|
|
|
|
The result of this method is the vector `b`, including the offset term (or
|
|
intercept term) `b_0`.
|
|
|
|
## Command-line `mlpack_linear_regression`
|
|
|
|
The simplest way to perform linear regression or ridge regression in mlpack
|
|
is to use the `mlpack_linear_regression` program. This program will perform
|
|
linear regression and place the resultant coefficients into one file. Note that
|
|
this guide details the `mlpack_linear_regression` command-line program, but
|
|
because mlpack also has bindings to other languages, functions like
|
|
`linear_regression()` exist in Python and Julia, and each example below can be
|
|
easily adapted to those languages.
|
|
|
|
The output file holds a vector of coefficients in increasing order of dimension;
|
|
that is, the offset term (`b_0`), the coefficient for dimension 1 (`b_1`, then
|
|
dimension 2 (`b_2`) and so forth, as well as the intercept. This executable can
|
|
also predict the `y` values of a second dataset based on the computed
|
|
coefficients.
|
|
|
|
Below are several examples of simple usage (and the resultant output). The `-v`
|
|
option is used so that verbose output is given. Further documentation on each
|
|
individual option can be found by typing
|
|
|
|
```sh
|
|
$ mlpack_linear_regression --help
|
|
```
|
|
|
|
### One file, generating the function coefficients
|
|
|
|
```sh
|
|
$ mlpack_linear_regression --training_file dataset.csv -v -M lr.xml
|
|
[INFO ] Loading 'dataset.csv' as CSV data. Size is 2 x 5.
|
|
[INFO ]
|
|
[INFO ] Execution parameters:
|
|
[INFO ] help: false
|
|
[INFO ] info: ""
|
|
[INFO ] input_model_file: ""
|
|
[INFO ] lambda: 0
|
|
[INFO ] output_model_file: lr.xml
|
|
[INFO ] output_predictions: predictions.csv
|
|
[INFO ] test_file: ""
|
|
[INFO ] training_file: dataset.csv
|
|
[INFO ] training_responses: ""
|
|
[INFO ] verbose: true
|
|
[INFO ] version: false
|
|
[INFO ]
|
|
[INFO ] Program timers:
|
|
[INFO ] load_regressors: 0.000263s
|
|
[INFO ] loading_data: 0.000220s
|
|
[INFO ] regression: 0.000392s
|
|
[INFO ] total_time: 0.001920s
|
|
```
|
|
|
|
Convenient program timers are given for different parts of the calculation at
|
|
the bottom of the output, as well as the parameters the simulation was run with.
|
|
Now, if we look at the output model file, which is `lr.xml`,
|
|
|
|
```sh
|
|
$ cat dataset.csv
|
|
0,0
|
|
1,1
|
|
2,2
|
|
3,3
|
|
4,4
|
|
|
|
$ cat lr.xml
|
|
<?xml version="1.0" encoding="utf-8"?>
|
|
<cereal>
|
|
<model>
|
|
<cereal_class_version>0</cereal_class_version>
|
|
<parameters>
|
|
<n_rows>2</n_rows>
|
|
<n_cols>1</n_cols>
|
|
<vec_state>1</vec_state>
|
|
<elem>0</elem>
|
|
<elem>1</elem>
|
|
</parameters>
|
|
<lambda>0</lambda>
|
|
<intercept>true</intercept>
|
|
</model>
|
|
</cereal>
|
|
```
|
|
|
|
As you can see, the function for this input is `f(y) = 0 + 1 x_1`. We can see
|
|
that the model we have trained catches this; in the `<parameters>` section of
|
|
`lr.xml`, we can see that there are two elements, which are (approximately) 0
|
|
and 1. The first element corresponds to the intercept 0, and the second column
|
|
corresponds to the coefficient 1 for the variable `x_1`. Note that in this
|
|
example, the regressors for the dataset are the second column. That is, the
|
|
dataset is one dimensional, and the last column has the `y` values, or
|
|
responses, for each row. You can specify these responses in a separate file if
|
|
you want, using the `--input_responses`, or `-r`, option.
|
|
|
|
### Train a multivariate linear regression model
|
|
|
|
Multivariate linear regression means that the response variable is predicted by
|
|
more than just one input variable. In this example we will try to fit a
|
|
multivariate linear regression model to data that contains four variables,
|
|
stored in `dataset_2.csv`.
|
|
|
|
```sh
|
|
$ cat dataset_2.csv
|
|
0,0,0,0,14
|
|
1,1,1,1,24
|
|
2,1,0,2,27
|
|
1,2,2,2,32
|
|
-1,-3,0,2,17
|
|
```
|
|
|
|
Now let's run `mlpack_linear_regression` as usual:
|
|
|
|
```sh
|
|
$ mlpack_linear_regression --training_file dataset_2.csv -v -M lr.xml
|
|
[INFO ] Loading 'dataset_2.csv' as CSV data. Size is 5 x 5.
|
|
[INFO ]
|
|
[INFO ] Execution parameters:
|
|
[INFO ] help: 0
|
|
[INFO ] info:
|
|
[INFO ] input_model_file:
|
|
[INFO ] lambda: 0
|
|
[INFO ] output_model_file: lr.xml
|
|
[INFO ] output_predictions_file:
|
|
[INFO ] test_file:
|
|
[INFO ] training_file: dataset_2.csv
|
|
[INFO ] training_responses_file:
|
|
[INFO ] verbose: 1
|
|
[INFO ] version: 0
|
|
[INFO ] Program timers:
|
|
[INFO ] load_regressors: 0.000060s
|
|
[INFO ] loading_data: 0.000050s
|
|
[INFO ] regression: 0.000049s
|
|
[INFO ] total_time: 0.000118s
|
|
|
|
$ cat lr.xml
|
|
<?xml version="1.0" encoding="utf-8"?>
|
|
<cereal>
|
|
<model>
|
|
<cereal_class_version>0</cereal_class_version>
|
|
<parameters>
|
|
<n_rows>5</n_rows>
|
|
<n_cols>1</n_cols>
|
|
<vec_state>1</vec_state>
|
|
<elem>14.00000000000002</elem>
|
|
<elem>1.9999999999999447</elem>
|
|
<elem>1.0000000000000431</elem>
|
|
<elem>2.9999999999999516</elem>
|
|
<elem>4.0000000000000249</elem>
|
|
</parameters>
|
|
<lambda>0</lambda>
|
|
<intercept>true</intercept>
|
|
</model>
|
|
</cereal>
|
|
```
|
|
|
|
If we take a look at the `lr.xml` output we can see the `<parameters>` part has
|
|
five elements which the first corresponds to `b_0`, the second corresponds to
|
|
`b_1` , and so on. This is equivalent to `f(y) = b_0 + b_1 x_1 + b_2 x_2 + b_3
|
|
x_3 + b_4 x_4`, or `f(y) = 14 + 2 x_1 + 1 x_2 + 3 x_3 + 4 x_4`.
|
|
|
|
### Compute model and predict at the same time
|
|
|
|
```sh
|
|
$ mlpack_linear_regression --training_file dataset.csv --test_file predict.csv --output_predictions_file predictions.csv \
|
|
> -v
|
|
[WARN ] '--output_predictions_file (-o)' ignored because '--test_file (-T)' is specified!
|
|
[INFO ] Loading 'dataset.csv' as CSV data. Size is 2 x 5.
|
|
[INFO ] Loading 'predict.csv' as raw ASCII formatted data. Size is 1 x 3.
|
|
[INFO ] Saving CSV data to 'predictions.csv'.
|
|
[INFO ]
|
|
[INFO ] Execution parameters:
|
|
[INFO ] help: 0
|
|
[INFO ] info:
|
|
[INFO ] input_model_file:
|
|
[INFO ] lambda: 0
|
|
[INFO ] output_model_file:
|
|
[INFO ] output_predictions_file: 'predictions.csv' (1x3 matrix)
|
|
[INFO ] test_file: 'predict.csv' (0x0 matrix)
|
|
[INFO ] training_file: 'dataset.csv' (0x0 matrix)
|
|
[INFO ] training_responses_file: ''
|
|
[INFO ] verbose: 1
|
|
[INFO ] version: 0
|
|
[INFO ] Program timers:
|
|
[INFO ] load_regressors: 0.000069s
|
|
[INFO ] load_test_points: 0.000031s
|
|
[INFO ] loading_data: 0.000079s
|
|
[INFO ] prediction: 0.000001s
|
|
[INFO ] regression: 0.000054s
|
|
[INFO ] saving_data: 0.000055s
|
|
[INFO ] total_time: 0.000203s
|
|
|
|
|
|
$ cat dataset.csv
|
|
0,0
|
|
1,1
|
|
2,2
|
|
3,3
|
|
4,4
|
|
|
|
$ cat predict.csv
|
|
2
|
|
3
|
|
4
|
|
|
|
$ cat predictions.csv
|
|
2.0000000000e+00
|
|
3.0000000000e+00
|
|
4.0000000000e+00
|
|
```
|
|
|
|
We used the same dataset, so we got the same parameters. The key thing to note
|
|
about the `predict.csv` dataset is that it has the same dimensionality as the
|
|
dataset used to create the model, one. If the model generating dataset has `d`
|
|
dimensions, so must the dataset we want to predict for.
|
|
|
|
### Prediction using a precomputed model
|
|
|
|
```sh
|
|
$ mlpack_linear_regression --input_model_file lr.xml --test_file predict.csv --output_predictions_file predictions.csv -v
|
|
[WARN ] '--output_predictions_file (-o)' ignored because '--test_file (-T)' is specified!
|
|
[INFO ] Loading 'predict.csv' as raw ASCII formatted data. Size is 1 x 3.
|
|
[INFO ] Saving CSV data to 'predictions.csv'.
|
|
[INFO ]
|
|
[INFO ] Execution parameters:
|
|
[INFO ] help: 0
|
|
[INFO ] info:
|
|
[INFO ] input_model_file: lr.xml
|
|
[INFO ] lambda: 0
|
|
[INFO ] output_model_file:
|
|
[INFO ] output_predictions_file: 'predictions.csv' (1x3 matrix)
|
|
[INFO ] test_file: 'predict.csv' (0x0 matrix)
|
|
[INFO ] training_file: ''
|
|
[INFO ] training_responses_file: ''
|
|
[INFO ] verbose: 1
|
|
[INFO ] version: 0
|
|
[INFO ] Program timers:
|
|
[INFO ] load_model: 0.000051s
|
|
[INFO ] load_test_points: 0.000052s
|
|
[INFO ] loading_data: 0.000044s
|
|
[INFO ] prediction: 0.000010s
|
|
[INFO ] saving_data: 0.000079s
|
|
[INFO ] total_time: 0.000160s
|
|
|
|
|
|
$ cat lr.xml
|
|
<?xml version="1.0" encoding="utf-8"?>
|
|
<cereal>
|
|
<model>
|
|
<cereal_class_version>0</cereal_class_version>
|
|
<parameters>
|
|
<n_rows>2</n_rows>
|
|
<n_cols>1</n_cols>
|
|
<vec_state>1</vec_state>
|
|
<elem>0</elem>
|
|
<elem>1</elem>
|
|
</parameters>
|
|
<lambda>0</lambda>
|
|
<intercept>true</intercept>
|
|
</model>
|
|
</cereal>
|
|
|
|
|
|
$ cat predict.csv
|
|
2
|
|
3
|
|
4
|
|
|
|
$ cat predictions.csv
|
|
2.0000000000e+00
|
|
3.0000000000e+00
|
|
4.0000000000e+00
|
|
```
|
|
|
|
### Using ridge regression
|
|
|
|
Sometimes, the input matrix of predictors has a covariance matrix that is not
|
|
invertible, or the system is overdetermined. In this case, ridge regression is
|
|
useful: it adds a normalization term to the covariance matrix to make it
|
|
invertible. Ridge regression is a standard technique and documentation for the
|
|
mathematics behind it can be found anywhere on the Internet. In short, the
|
|
covariance matrix `X' X` is replaced with `X' X + l I` where `I` is the identity
|
|
matrix. So, an `l` parameter greater than zero should be specified to perform
|
|
ridge regression, using the `--lambda` (or `-l`) option. An example is given
|
|
below.
|
|
|
|
```sh
|
|
$ mlpack_linear_regression --training_file dataset.csv -v --lambda 0.5 -M lr.xml
|
|
[INFO ] Loading 'dataset.csv' as CSV data. Size is 2 x 5.
|
|
[INFO ]
|
|
[INFO ] Execution parameters:
|
|
[INFO ] help: false
|
|
[INFO ] info: ""
|
|
[INFO ] input_model_file: ""
|
|
[INFO ] lambda: 0.5
|
|
[INFO ] output_model_file: lr.xml
|
|
[INFO ] output_predictions: predictions.csv
|
|
[INFO ] test_file: ""
|
|
[INFO ] training_file: dataset.csv
|
|
[INFO ] training_responses: ""
|
|
[INFO ] verbose: true
|
|
[INFO ] version: false
|
|
[INFO ]
|
|
[INFO ] Program timers:
|
|
[INFO ] load_regressors: 0.000210s
|
|
[INFO ] loading_data: 0.000170s
|
|
[INFO ] regression: 0.000332s
|
|
[INFO ] total_time: 0.001835s
|
|
```
|
|
|
|
Further documentation on options should be found by using the `--help` option.
|
|
|
|
## The `LinearRegression` class
|
|
|
|
The `LinearRegression` class is a simple implementation of linear regression.
|
|
|
|
Using the `LinearRegression` class is very simple. It has two available
|
|
constructors; one for generating a model from a matrix of predictors and a
|
|
vector of responses, and one for loading an already computed model from a given
|
|
file.
|
|
|
|
The class provides one method that performs computation:
|
|
|
|
```c++
|
|
void Predict(const arma::mat& points, arma::rowvec& predictions);
|
|
```
|
|
|
|
Once you have generated or loaded a model, you can call this method and pass it
|
|
a matrix of data points to predict values for using the model. The second
|
|
parameter, predictions, will be modified to contain the predicted values
|
|
corresponding to each row of the points matrix.
|
|
|
|
### Generating a model
|
|
|
|
```c++
|
|
#include <mlpack.hpp>
|
|
|
|
using namespace mlpack;
|
|
|
|
arma::mat data; // The dataset itself.
|
|
arma::rowvec responses; // The responses, one row for each row in data.
|
|
|
|
// Regress.
|
|
LinearRegression lr(data, responses);
|
|
|
|
// Get the parameters, or coefficients.
|
|
arma::vec parameters = lr.Parameters();
|
|
```
|
|
|
|
### Setting a model
|
|
|
|
Assuming you already have a model and do not need to create one, this is how
|
|
you would set the parameters for a `LinearRegression` instance.
|
|
|
|
```c++
|
|
arma::vec parameters; // Your model.
|
|
|
|
LinearRegression lr; // Create a new LinearRegression instance or reuse one.
|
|
lr.Parameters() = parameters; // Set the model.
|
|
```
|
|
|
|
### Load a model from file
|
|
|
|
If you have a generated model in a file somewhere you would like to load and
|
|
use, you can use `data::Load()` to load it.
|
|
|
|
```c++
|
|
std::string filename; // The path and name of your file.
|
|
|
|
LinearRegression lr;
|
|
data::Load(filename, "lr_model", lr);
|
|
```
|
|
|
|
Here, `"lr_model"` is the name given to the model. The
|
|
`mlpack_linear_regression` command-line program uses the default name `"model"`,
|
|
so if you want to load a model in C++ that was created by the command-line
|
|
program, be sure to use the name `"model"` when loading.
|
|
|
|
### Prediction
|
|
|
|
Once you have generated or loaded a model using one of the aforementioned
|
|
methods, you can predict values for a dataset.
|
|
|
|
```c++
|
|
LinearRegression lr();
|
|
// Load or generate your model.
|
|
|
|
// The dataset we want to predict on; each row is a data point.
|
|
arma::mat points;
|
|
// This will store the predictions; one row for each point.
|
|
arma::rowvec predictions;
|
|
|
|
lr.Predict(points, predictions); // Predict.
|
|
|
|
// Now, the row vector 'predictions' will contain the predicted values.
|
|
```
|
|
|
|
### Setting lambda for ridge regression
|
|
|
|
As discussed in an earlier example, ridge regression is useful when the
|
|
covariance of the predictors is not invertible. The standard constructor can be
|
|
used to set a value of lambda:
|
|
|
|
```c++
|
|
#include <mlpack.hpp>
|
|
|
|
using namespace mlpack;
|
|
|
|
arma::mat data; // The dataset itself.
|
|
arma::rowvec responses; // The responses, one row for each row in data.
|
|
|
|
// Regress, with a lambda of 0.5.
|
|
LinearRegression lr(data, responses, 0.5);
|
|
|
|
// Get the parameters, or coefficients.
|
|
arma::vec parameters = lr.Parameters();
|
|
```
|
|
|
|
In addition, the `Lambda()` function can be used to get or modify the lambda
|
|
value:
|
|
|
|
```c++
|
|
LinearRegression lr;
|
|
lr.Lambda() = 0.5;
|
|
Log::Info << "Lambda is " << lr.Lambda() << "." << std::endl;
|
|
```
|
|
|
|
## Further documentation
|
|
|
|
For further documentation on the LinearRegression class, consult the comments in
|
|
the source code of the `LinearRegression` class, found in
|
|
`mlpack/methods/linear_regression/`.
|