Generic stochastic gradient descent training for kernel-based models. More...
#include <shark/Algorithms/Trainers/KernelSGDTrainer.h>
Public Types | |
typedef AbstractKernelFunction< InputType > | KernelType |
typedef KernelClassifier< InputType > | ClassifierType |
typedef KernelExpansion< InputType > | ModelType |
typedef AbstractLoss< unsigned int, RealVector > | LossType |
typedef ConstProxyReference< typename Batch< InputType >::type const >::type | ConstBatchInputReference |
typedef CacheType | QpFloatType |
typedef KernelMatrix< InputType, QpFloatType > | KernelMatrixType |
typedef PartlyPrecomputedMatrix< KernelMatrixType > | PartlyPrecomputedMatrixType |
Public Types inherited from shark::AbstractTrainer< KernelClassifier< InputType > > | |
typedef KernelClassifier< InputType > | ModelType |
typedef ModelType::InputType | InputType |
typedef typename KernelClassifier< InputType > ::OutputType | LabelType |
typedef LabeledData< InputType, LabelType > | DatasetType |
Public Types inherited from shark::IParameterizable<> | |
typedef RealVector | ParameterVectorType |
Public Member Functions | |
KernelSGDTrainer (KernelType *kernel, const LossType *loss, double C, bool offset, bool unconstrained=false, size_t cacheSize=0x4000000) | |
Constructor. More... | |
double | cacheSize () const |
return current cachesize More... | |
void | setCacheSize (std::size_t size) |
std::string | name () const |
From INameable: return the class name. More... | |
void | train (ClassifierType &classifier, const LabeledData< InputType, unsigned int > &dataset) |
std::size_t | epochs () const |
void | setEpochs (std::size_t value) |
KernelType * | kernel () |
get the kernel function More... | |
const KernelType * | kernel () const |
get the kernel function More... | |
void | setKernel (KernelType *kernel) |
set the kernel function More... | |
bool | isUnconstrained () const |
double | C () const |
return the value of the regularization parameter More... | |
void | setC (double value) |
set the value of the regularization parameter (must be positive) More... | |
bool | trainOffset () const |
check whether the model to be trained should include an offset term More... | |
RealVector | parameterVector () const |
Returns the vector of hyper-parameters. More... | |
void | setParameterVector (RealVector const &newParameters) |
Sets the vector of hyper-parameters. More... | |
size_t | numberOfParameters () const |
Returns the number of hyper-parameters. More... | |
Public Member Functions inherited from shark::AbstractTrainer< KernelClassifier< InputType > > | |
virtual void | train (ModelType &model, DatasetType const &dataset)=0 |
Core of the Trainer interface. More... | |
Public Member Functions inherited from shark::INameable | |
virtual | ~INameable () |
Public Member Functions inherited from shark::ISerializable | |
virtual | ~ISerializable () |
Virtual d'tor. More... | |
virtual void | read (InArchive &archive) |
Read the component from the supplied archive. More... | |
virtual void | write (OutArchive &archive) const |
Write the component to the supplied archive. More... | |
void | load (InArchive &archive, unsigned int version) |
Versioned loading of components, calls read(...). More... | |
void | save (OutArchive &archive, unsigned int version) const |
Versioned storing of components, calls write(...). More... | |
BOOST_SERIALIZATION_SPLIT_MEMBER () | |
Public Member Functions inherited from shark::IParameterizable<> | |
virtual | ~IParameterizable () |
Protected Attributes | |
KernelType * | m_kernel |
pointer to kernel function More... | |
const LossType * | m_loss |
pointer to loss function More... | |
double | m_C |
regularization parameter More... | |
bool | m_offset |
should the resulting model have an offset term? More... | |
bool | m_unconstrained |
should C be stored as log(C) as a parameter? More... | |
std::size_t | m_epochs |
number of training epochs (sweeps over the data), or 0 for default = max(10, C) More... | |
std::size_t | m_cacheSize |
Generic stochastic gradient descent training for kernel-based models.
Given a differentiable loss function L(f, y) for classification this trainer solves the regularized risk minimization problem
\[ \min \frac{1}{2} \sum_j \|w_j\|^2 + C \sum_i L(y_i, f(x_i)), \]
where i runs over training data, j over classes, and C > 0 is the regularization parameter.
\[ w_j \leftarrow (1 - 1/t) w_j + \frac{C}{t} \frac{\partial L(y_i, f(x_i))}{\partial w_j} \]
for random index i. The only notable trick borrowed from that paper is the representation of the weight vectors in the form\[ w_j = s \cdot \sum_i \alpha_{i,j} k(x_i, \cdot) \]
with a scalar factor s (called alphaScale in the code). This enables scaling with factor (1 - 1/t) in constant time.Definition at line 90 of file KernelSGDTrainer.h.
typedef KernelClassifier<InputType> shark::KernelSGDTrainer< InputType, CacheType >::ClassifierType |
Definition at line 95 of file KernelSGDTrainer.h.
typedef ConstProxyReference<typename Batch<InputType>::type const>::type shark::KernelSGDTrainer< InputType, CacheType >::ConstBatchInputReference |
Definition at line 98 of file KernelSGDTrainer.h.
typedef KernelMatrix<InputType, QpFloatType> shark::KernelSGDTrainer< InputType, CacheType >::KernelMatrixType |
Definition at line 101 of file KernelSGDTrainer.h.
typedef AbstractKernelFunction<InputType> shark::KernelSGDTrainer< InputType, CacheType >::KernelType |
Definition at line 94 of file KernelSGDTrainer.h.
typedef AbstractLoss<unsigned int, RealVector> shark::KernelSGDTrainer< InputType, CacheType >::LossType |
Definition at line 97 of file KernelSGDTrainer.h.
typedef KernelExpansion<InputType> shark::KernelSGDTrainer< InputType, CacheType >::ModelType |
Definition at line 96 of file KernelSGDTrainer.h.
typedef PartlyPrecomputedMatrix< KernelMatrixType > shark::KernelSGDTrainer< InputType, CacheType >::PartlyPrecomputedMatrixType |
Definition at line 102 of file KernelSGDTrainer.h.
typedef CacheType shark::KernelSGDTrainer< InputType, CacheType >::QpFloatType |
Definition at line 99 of file KernelSGDTrainer.h.
|
inline |
Constructor.
kernel | kernel function to use for training and prediction |
loss | (sub-)differentiable loss function |
C | regularization parameter - always the 'true' value of C, even when unconstrained is set |
offset | whether to train with offset/bias parameter or not |
unconstrained | when a C-value is given via setParameter, should it be piped through the exp-function before using it in the solver? |
cacheSize | size of the cache |
Definition at line 113 of file KernelSGDTrainer.h.
|
inline |
return the value of the regularization parameter
Definition at line 224 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_C.
|
inline |
return current cachesize
Definition at line 125 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_cacheSize.
|
inline |
Return the number of training epochs. A value of 0 indicates that the default of max(10, C) should be used.
Definition at line 199 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_epochs.
|
inline |
check whether the parameter C is represented as log(C), thus, in a form suitable for unconstrained optimization, in the parameter vector
Definition at line 220 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_unconstrained.
|
inline |
get the kernel function
Definition at line 208 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_kernel.
Referenced by shark::KernelSGDTrainer< InputType, CacheType >::setKernel().
|
inline |
get the kernel function
Definition at line 211 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_kernel.
|
inlinevirtual |
From INameable: return the class name.
Reimplemented from shark::INameable.
Definition at line 137 of file KernelSGDTrainer.h.
|
inlinevirtual |
Returns the number of hyper-parameters.
Reimplemented from shark::IParameterizable<>.
Definition at line 254 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_kernel, and shark::IParameterizable< VectorType >::numberOfParameters().
|
inlinevirtual |
Returns the vector of hyper-parameters.
Reimplemented from shark::IParameterizable<>.
Definition at line 239 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_C, shark::KernelSGDTrainer< InputType, CacheType >::m_kernel, shark::KernelSGDTrainer< InputType, CacheType >::m_unconstrained, and shark::IParameterizable< VectorType >::parameterVector().
|
inline |
set the value of the regularization parameter (must be positive)
Definition at line 228 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_C, and RANGE_CHECK.
|
inline |
Definition at line 131 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_cacheSize.
|
inline |
Set the number of training epochs. A value of 0 indicates that the default of max(10, C) should be used.
Definition at line 204 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_epochs.
|
inline |
set the kernel function
Definition at line 214 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::kernel(), and shark::KernelSGDTrainer< InputType, CacheType >::m_kernel.
|
inlinevirtual |
Sets the vector of hyper-parameters.
Reimplemented from shark::IParameterizable<>.
Definition at line 245 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_kernel, shark::IParameterizable< VectorType >::numberOfParameters(), shark::IParameterizable< VectorType >::setParameterVector(), and SHARK_ASSERT.
|
inline |
Definition at line 140 of file KernelSGDTrainer.h.
References shark::KernelExpansion< InputType >::alpha(), shark::createBatch(), shark::Classifier< Model >::decisionFunction(), shark::Data< Type >::elements(), shark::LabeledData< InputT, LabelT >::inputs(), shark::LabeledData< InputT, LabelT >::labels(), lambda, shark::KernelSGDTrainer< InputType, CacheType >::m_C, shark::KernelSGDTrainer< InputType, CacheType >::m_cacheSize, shark::KernelSGDTrainer< InputType, CacheType >::m_epochs, shark::KernelSGDTrainer< InputType, CacheType >::m_kernel, shark::KernelSGDTrainer< InputType, CacheType >::m_offset, shark::numberOfClasses(), shark::LabeledData< InputT, LabelT >::numberOfElements(), and shark::KernelExpansion< InputType >::setStructure().
Referenced by main().
|
inline |
check whether the model to be trained should include an offset term
Definition at line 235 of file KernelSGDTrainer.h.
References shark::KernelSGDTrainer< InputType, CacheType >::m_offset.
|
protected |
regularization parameter
Definition at line 261 of file KernelSGDTrainer.h.
Referenced by shark::KernelSGDTrainer< InputType, CacheType >::C(), shark::KernelSGDTrainer< InputType, CacheType >::parameterVector(), shark::KernelSGDTrainer< InputType, CacheType >::setC(), and shark::KernelSGDTrainer< InputType, CacheType >::train().
|
protected |
Definition at line 267 of file KernelSGDTrainer.h.
Referenced by shark::KernelSGDTrainer< InputType, CacheType >::cacheSize(), shark::KernelSGDTrainer< InputType, CacheType >::setCacheSize(), and shark::KernelSGDTrainer< InputType, CacheType >::train().
|
protected |
number of training epochs (sweeps over the data), or 0 for default = max(10, C)
Definition at line 264 of file KernelSGDTrainer.h.
Referenced by shark::KernelSGDTrainer< InputType, CacheType >::epochs(), shark::KernelSGDTrainer< InputType, CacheType >::setEpochs(), and shark::KernelSGDTrainer< InputType, CacheType >::train().
|
protected |
pointer to kernel function
Definition at line 259 of file KernelSGDTrainer.h.
Referenced by shark::KernelSGDTrainer< InputType, CacheType >::kernel(), shark::KernelSGDTrainer< InputType, CacheType >::numberOfParameters(), shark::KernelSGDTrainer< InputType, CacheType >::parameterVector(), shark::KernelSGDTrainer< InputType, CacheType >::setKernel(), shark::KernelSGDTrainer< InputType, CacheType >::setParameterVector(), and shark::KernelSGDTrainer< InputType, CacheType >::train().
|
protected |
pointer to loss function
Definition at line 260 of file KernelSGDTrainer.h.
|
protected |
should the resulting model have an offset term?
Definition at line 262 of file KernelSGDTrainer.h.
Referenced by shark::KernelSGDTrainer< InputType, CacheType >::train(), and shark::KernelSGDTrainer< InputType, CacheType >::trainOffset().
|
protected |
should C be stored as log(C) as a parameter?
Definition at line 263 of file KernelSGDTrainer.h.
Referenced by shark::KernelSGDTrainer< InputType, CacheType >::isUnconstrained(), and shark::KernelSGDTrainer< InputType, CacheType >::parameterVector().