Shikhar Bhardwaj
1d42aa9d47
Use arma::dot instead of arma::norm in LogisiticRegressionFunction::PartialGradient
115 lines
4.0 KiB
C++
115 lines
4.0 KiB
C++
/*! @page function The FunctionType policy in mlpack
|
|
|
|
@section Overview
|
|
|
|
To represent the various types of loss functions encountered in machine
|
|
learning problems, mlpack provides the \c FunctionType template parameter in
|
|
the optimizer interface. The various optimizers available in the core library
|
|
rely on this policy to gain the necessary information required by the optimizing
|
|
algorithm.
|
|
|
|
The \c FunctionType template parameter required by the Optimizer class can have
|
|
additional requirements imposed on it, depending on the type of optimizer used.
|
|
|
|
@section requirements Interface requirements
|
|
|
|
The most basic requirements for the \c FunctionType parameter are the
|
|
implementations of two public member functions, with the following interface
|
|
and semantics
|
|
|
|
@code
|
|
// Evaluate the loss function at the given coordinates.
|
|
double Evaluate(const arma::mat& coordinates);
|
|
@endcode
|
|
|
|
|
|
@code
|
|
// Evaluate the gradient at the given coordinates, where 'gradient' is an
|
|
// output parameter for the required gradient.
|
|
void Gradient(const arma::mat& coordinates, arma::mat& gradient);
|
|
@endcode
|
|
|
|
|
|
Optimizers like SGD and RMSProp require a \c DecomposableFunctionType having the
|
|
following requirements
|
|
|
|
@code
|
|
// Return the number of functions. In a data-dependent function, this would
|
|
// return the number of points in the dataset.
|
|
size_t NumFunctions();
|
|
@endcode
|
|
|
|
|
|
@code
|
|
// Evaluate the 'i' th loss function. For example, for a data-dependent
|
|
// function, Evaluate(coordinates, 0) should evaluate the loss function at the
|
|
// first point in the dataset.
|
|
double Evaluate(const arma::mat& coordinates, const size_t i);
|
|
@endcode
|
|
|
|
@code
|
|
// Evaluate the gradient of the 'i' th loss function at the given coordinates,
|
|
// where 'gradient' is an output parameter for the required gradient.
|
|
void Gradient(const arma::mat& coordinates, const size_t i, arma::mat& gradient);
|
|
@endcode
|
|
|
|
|
|
|
|
\c ParallelSGD optimizer requires a \c SparseFunctionType interface.
|
|
\c SparseFunctionType requires the gradient to be in a sparse matrix (\c
|
|
arma::sp_mat), as ParallelSGD, implemented with the HOGWILD! scheme of
|
|
unsynchronised updates, is expected to be relevant only in situations where the
|
|
individual gradients are sparse. So, the interface requires function with the
|
|
following signatures
|
|
|
|
@code
|
|
// Return the number of functions. In a data-dependent function, this would
|
|
// return the number of points in the dataset.
|
|
size_t NumFunctions();
|
|
@endcode
|
|
|
|
|
|
@code
|
|
// Evaluate the loss function at the given coordinates.
|
|
double Evaluate(const arma::mat& coordinates);
|
|
@endcode
|
|
|
|
|
|
@code
|
|
// Evaluate the (sparse) gradient of the 'i' th loss function at the given
|
|
// coordinates, where 'gradient' is an output parameter for the required
|
|
// gradient.
|
|
void Gradient(const arma::mat& coordinates, const size_t i, arma::sp_mat& gradient);
|
|
@endcode
|
|
|
|
|
|
The \c SCD optimizer requires a \c ResolvableFunctionType interface, to
|
|
calculate partial gradients with respect to individual features. The optimizer
|
|
requires the decision variable to be arranged in a particular fashion to allow
|
|
for disjoint updates. The features should be arranged columnwise in the decision
|
|
variable. For example, in \c SoftmaxRegressionFunction the decision variable has
|
|
size \c numClasses x \c featureSize (+ 1 if an intercept also needs to be fit).
|
|
Similarly, for \c LogisticRegression, the decision variable is a row vector,
|
|
with the number of columns determined by the dimensionality of the dataset.
|
|
|
|
The interface expects the following member functions from the function class
|
|
|
|
@code
|
|
// Return the number of features in the decision variable.
|
|
size_t NumFeatures();
|
|
@endcode
|
|
|
|
@code
|
|
// Evaluate the loss function at the given coordinates.
|
|
double Evaluate(const arma::mat& coordinates);
|
|
@endcode
|
|
|
|
@code
|
|
// Evaluate the partial gradient of the loss function with respect to the 'j' th
|
|
// coordinate at the given coordinates, where 'gradient' is an output parameter
|
|
// for the required gradient. The 'gradient' matrix is supposed to be non-zero
|
|
// in the jth column, which contains the relevant partial gradient.
|
|
void PartialGradient(const arma::mat& coordinates, const size_t j, arma::sp_mat& gradient);
|
|
@endcode
|
|
*/
|