Stochastic Average Gradient Method for training of linear models,. More...
#include <shark/Algorithms/Trainers/LinearSAGTrainer.h>
Public Types | |
typedef Base::ModelType | ModelType |
typedef Base::WeightedDatasetType | WeightedDatasetType |
typedef detail::LinearSAGTrainerBase< InputType, LabelType >::LossType | LossType |
Public Types inherited from shark::AbstractWeightedTrainer< Model, LabelTypeT > | |
typedef base_type::ModelType | ModelType |
typedef base_type::InputType | InputType |
typedef base_type::LabelType | LabelType |
typedef base_type::DatasetType | DatasetType |
typedef WeightedLabeledData< InputType, LabelType > | WeightedDatasetType |
Public Types inherited from shark::AbstractTrainer< Model, LabelTypeT > | |
typedef Model | ModelType |
typedef ModelType::InputType | InputType |
typedef LabelTypeT | LabelType |
typedef LabeledData< InputType, LabelType > | DatasetType |
Public Types inherited from shark::IParameterizable<> | |
typedef RealVector | ParameterVectorType |
Public Member Functions | |
LinearSAGTrainer (LossType const *loss, double lambda=0, bool offset=true) | |
Constructor. More... | |
std::string | name () const |
From INameable: return the class name. More... | |
void | train (ModelType &model, WeightedDatasetType const &dataset) |
Executes the algorithm and trains a model on the given weighted data. More... | |
std::size_t | epochs () const |
void | setEpochs (std::size_t value) |
Set the number of training epochs. A value of 0 indicates that the default of max(10, dimensionOfData) should be used. More... | |
double | lambda () const |
Return the value of the regularization parameter lambda. More... | |
void | setLambda (double lambda) |
Set the value of the regularization parameter lambda. More... | |
bool | trainOffset () const |
Check whether the model to be trained should include an offset term. More... | |
void | setTrainOffset (bool offset) |
Sets whether the model to be trained should include an offset term. More... | |
RealVector | parameterVector () const |
Returns the vector of hyper-parameters(same as lambda) More... | |
void | setParameterVector (RealVector const &newParameters) |
Sets the vector of hyper-parameters(same as lambda) More... | |
size_t | numberOfParameters () const |
Returns the number of hyper-parameters. More... | |
Public Member Functions inherited from shark::AbstractWeightedTrainer< Model, LabelTypeT > | |
virtual void | train (ModelType &model, DatasetType const &dataset) |
Executes the algorithm and trains a model on the given unweighted data. More... | |
Public Member Functions inherited from shark::INameable | |
virtual | ~INameable () |
Public Member Functions inherited from shark::ISerializable | |
virtual | ~ISerializable () |
Virtual d'tor. More... | |
virtual void | read (InArchive &archive) |
Read the component from the supplied archive. More... | |
virtual void | write (OutArchive &archive) const |
Write the component to the supplied archive. More... | |
void | load (InArchive &archive, unsigned int version) |
Versioned loading of components, calls read(...). More... | |
void | save (OutArchive &archive, unsigned int version) const |
Versioned storing of components, calls write(...). More... | |
BOOST_SERIALIZATION_SPLIT_MEMBER () | |
Public Member Functions inherited from shark::IParameterizable<> | |
virtual | ~IParameterizable () |
Stochastic Average Gradient Method for training of linear models,.
Given a differentiable loss function L(f, y) and a model f_j(x)= w_j^Tx+b this trainer solves the regularized risk minimization problem
\[ \min \frac{1}{2} \sum_j \frac{\lambda}{2}\|w_j\|^2 + \frac 1 {\ell} \sum_i L(y_i, f(x_i)), \]
where i runs over training data, j over the model outputs, and lambda > 0 is the regularization parameter.
The algorithm uses averaging of the algorithm to obtain a good estimate of the gradient. Averaging is performed by summing over the last gradient value obtained for each data point. At the beginning this estimate is far off as old gradient values are outdated, but as the algorithm converges, this gives linear convergence on strictly convex functions and O(1/T) convergence on not-strictly convex functions.
The algorithm supports classification and regresseion, dense and sparse inputs and weighted and unweighted datasets Reference: Schmidt, Mark, Nicolas Le Roux, and Francis Bach. "Minimizing finite sums with the stochastic average gradient." arXiv preprint arXiv:1309.2388 (2013).
Definition at line 90 of file LinearSAGTrainer.h.
typedef detail::LinearSAGTrainerBase<InputType,LabelType>::LossType shark::LinearSAGTrainer< InputType, LabelType >::LossType |
Definition at line 97 of file LinearSAGTrainer.h.
typedef Base::ModelType shark::LinearSAGTrainer< InputType, LabelType >::ModelType |
Definition at line 95 of file LinearSAGTrainer.h.
typedef Base::WeightedDatasetType shark::LinearSAGTrainer< InputType, LabelType >::WeightedDatasetType |
Definition at line 96 of file LinearSAGTrainer.h.
|
inline |
Constructor.
loss | (sub-)differentiable loss function |
lambda | regularization parameter fort wo-norm regularization, 0 by default |
offset | whether to train with offset/bias parameter or not, default is true |
Definition at line 105 of file LinearSAGTrainer.h.
|
inline |
the number of training epochs. A value of 0 indicates that the default of max(10, dimensionOfData) should be used.
Definition at line 124 of file LinearSAGTrainer.h.
|
inline |
Return the value of the regularization parameter lambda.
Definition at line 134 of file LinearSAGTrainer.h.
|
inlinevirtual |
From INameable: return the class name.
Reimplemented from shark::INameable.
Definition at line 113 of file LinearSAGTrainer.h.
|
inlinevirtual |
Returns the number of hyper-parameters.
Reimplemented from shark::IParameterizable<>.
Definition at line 163 of file LinearSAGTrainer.h.
References shark::Classifier< LinearModel< VectorType > >::decisionFunction(), shark::inputDimension(), shark::LinearModel< InputType, ActivationFunction >::inputShape(), shark::labelDimension(), shark::numberOfClasses(), shark::LinearModel< InputType, ActivationFunction >::outputShape(), shark::LinearModel< InputType, ActivationFunction >::setStructure(), and shark::DataView< DatasetType >::size().
|
inlinevirtual |
Returns the vector of hyper-parameters(same as lambda)
Reimplemented from shark::IParameterizable<>.
Definition at line 150 of file LinearSAGTrainer.h.
|
inline |
Set the number of training epochs. A value of 0 indicates that the default of max(10, dimensionOfData) should be used.
Definition at line 129 of file LinearSAGTrainer.h.
Referenced by run().
|
inline |
Set the value of the regularization parameter lambda.
Definition at line 138 of file LinearSAGTrainer.h.
References lambda.
|
inlinevirtual |
Sets the vector of hyper-parameters(same as lambda)
Reimplemented from shark::IParameterizable<>.
Definition at line 156 of file LinearSAGTrainer.h.
References SIZE_CHECK.
|
inline |
Sets whether the model to be trained should include an offset term.
Definition at line 146 of file LinearSAGTrainer.h.
|
inlinevirtual |
Executes the algorithm and trains a model on the given weighted data.
Implements shark::AbstractWeightedTrainer< Model, LabelTypeT >.
Definition at line 117 of file LinearSAGTrainer.h.
References shark::random::globalRng.
Referenced by run().
|
inline |
Check whether the model to be trained should include an offset term.
Definition at line 142 of file LinearSAGTrainer.h.