shark::KernelSGDTrainer< InputType, CacheType > Class Template Reference

Generic stochastic gradient descent training for kernel-based models. More...

#include <shark/Algorithms/Trainers/KernelSGDTrainer.h>

+ Inheritance diagram for shark::KernelSGDTrainer< InputType, CacheType >:

Public Types

typedef AbstractKernelFunction< InputTypeKernelType
 
typedef KernelClassifier< InputTypeClassifierType
 
typedef KernelExpansion< InputTypeModelType
 
typedef AbstractLoss< unsigned int, RealVector > LossType
 
typedef ConstProxyReference< typename Batch< InputType >::type const >::type ConstBatchInputReference
 
typedef CacheType QpFloatType
 
typedef KernelMatrix< InputType, QpFloatTypeKernelMatrixType
 
typedef PartlyPrecomputedMatrix< KernelMatrixTypePartlyPrecomputedMatrixType
 
- Public Types inherited from shark::AbstractTrainer< KernelClassifier< InputType > >
typedef KernelClassifier< InputTypeModelType
 
typedef ModelType::InputType InputType
 
typedef typename KernelClassifier< InputType > ::OutputType LabelType
 
typedef LabeledData< InputType, LabelTypeDatasetType
 
- Public Types inherited from shark::IParameterizable<>
typedef RealVector ParameterVectorType
 

Public Member Functions

 KernelSGDTrainer (KernelType *kernel, const LossType *loss, double C, bool offset, bool unconstrained=false, size_t cacheSize=0x4000000)
 Constructor. More...
 
double cacheSize () const
 return current cachesize More...
 
void setCacheSize (std::size_t size)
 
std::string name () const
 From INameable: return the class name. More...
 
void train (ClassifierType &classifier, const LabeledData< InputType, unsigned int > &dataset)
 
std::size_t epochs () const
 
void setEpochs (std::size_t value)
 
KernelTypekernel ()
 get the kernel function More...
 
const KernelTypekernel () const
 get the kernel function More...
 
void setKernel (KernelType *kernel)
 set the kernel function More...
 
bool isUnconstrained () const
 
double C () const
 return the value of the regularization parameter More...
 
void setC (double value)
 set the value of the regularization parameter (must be positive) More...
 
bool trainOffset () const
 check whether the model to be trained should include an offset term More...
 
RealVector parameterVector () const
 Returns the vector of hyper-parameters. More...
 
void setParameterVector (RealVector const &newParameters)
 Sets the vector of hyper-parameters. More...
 
size_t numberOfParameters () const
 Returns the number of hyper-parameters. More...
 
- Public Member Functions inherited from shark::AbstractTrainer< KernelClassifier< InputType > >
virtual void train (ModelType &model, DatasetType const &dataset)=0
 Core of the Trainer interface. More...
 
- Public Member Functions inherited from shark::INameable
virtual ~INameable ()
 
- Public Member Functions inherited from shark::ISerializable
virtual ~ISerializable ()
 Virtual d'tor. More...
 
virtual void read (InArchive &archive)
 Read the component from the supplied archive. More...
 
virtual void write (OutArchive &archive) const
 Write the component to the supplied archive. More...
 
void load (InArchive &archive, unsigned int version)
 Versioned loading of components, calls read(...). More...
 
void save (OutArchive &archive, unsigned int version) const
 Versioned storing of components, calls write(...). More...
 
 BOOST_SERIALIZATION_SPLIT_MEMBER ()
 
- Public Member Functions inherited from shark::IParameterizable<>
virtual ~IParameterizable ()
 

Protected Attributes

KernelTypem_kernel
 pointer to kernel function More...
 
const LossTypem_loss
 pointer to loss function More...
 
double m_C
 regularization parameter More...
 
bool m_offset
 should the resulting model have an offset term? More...
 
bool m_unconstrained
 should C be stored as log(C) as a parameter? More...
 
std::size_t m_epochs
 number of training epochs (sweeps over the data), or 0 for default = max(10, C) More...
 
std::size_t m_cacheSize
 

Detailed Description

template<class InputType, class CacheType = float>
class shark::KernelSGDTrainer< InputType, CacheType >

Generic stochastic gradient descent training for kernel-based models.

Given a differentiable loss function L(f, y) for classification this trainer solves the regularized risk minimization problem

\[ \min \frac{1}{2} \sum_j \|w_j\|^2 + C \sum_i L(y_i, f(x_i)), \]

where i runs over training data, j over classes, and C > 0 is the regularization parameter.

This implementation is an adaptation of the PEGASOS algorithm, see the paper Shalev-Shwartz et al. "Pegasos: Primal estimated sub-gradient solver for SVM." Mathematical Programming 127.1 (2011): 3-30.

However, the (non-essential) projection step is dropped, and the algorithm is applied to a kernelized model. The resulting optimization scheme amounts to plain standard stochastic gradient descent (SGD) with update steps of the form

\[ w_j \leftarrow (1 - 1/t) w_j + \frac{C}{t} \frac{\partial L(y_i, f(x_i))}{\partial w_j} \]

for random index i. The only notable trick borrowed from that paper is the representation of the weight vectors in the form

\[ w_j = s \cdot \sum_i \alpha_{i,j} k(x_i, \cdot) \]

with a scalar factor s (called alphaScale in the code). This enables scaling with factor (1 - 1/t) in constant time.
NOTE: Being an SGD-based solver, this algorithm is relatively fast for differentiable loss functions such as the logistic loss (class CrossEntropy). It suffers from significantly slower convergence for non-differentiable losses, e.g., the hinge loss for SVM training.

Definition at line 90 of file KernelSGDTrainer.h.

Member Typedef Documentation

◆ ClassifierType

template<class InputType, class CacheType = float>
typedef KernelClassifier<InputType> shark::KernelSGDTrainer< InputType, CacheType >::ClassifierType

Definition at line 95 of file KernelSGDTrainer.h.

◆ ConstBatchInputReference

template<class InputType, class CacheType = float>
typedef ConstProxyReference<typename Batch<InputType>::type const>::type shark::KernelSGDTrainer< InputType, CacheType >::ConstBatchInputReference

Definition at line 98 of file KernelSGDTrainer.h.

◆ KernelMatrixType

template<class InputType, class CacheType = float>
typedef KernelMatrix<InputType, QpFloatType> shark::KernelSGDTrainer< InputType, CacheType >::KernelMatrixType

Definition at line 101 of file KernelSGDTrainer.h.

◆ KernelType

template<class InputType, class CacheType = float>
typedef AbstractKernelFunction<InputType> shark::KernelSGDTrainer< InputType, CacheType >::KernelType

Definition at line 94 of file KernelSGDTrainer.h.

◆ LossType

template<class InputType, class CacheType = float>
typedef AbstractLoss<unsigned int, RealVector> shark::KernelSGDTrainer< InputType, CacheType >::LossType

Definition at line 97 of file KernelSGDTrainer.h.

◆ ModelType

template<class InputType, class CacheType = float>
typedef KernelExpansion<InputType> shark::KernelSGDTrainer< InputType, CacheType >::ModelType

Definition at line 96 of file KernelSGDTrainer.h.

◆ PartlyPrecomputedMatrixType

template<class InputType, class CacheType = float>
typedef PartlyPrecomputedMatrix< KernelMatrixType > shark::KernelSGDTrainer< InputType, CacheType >::PartlyPrecomputedMatrixType

Definition at line 102 of file KernelSGDTrainer.h.

◆ QpFloatType

template<class InputType, class CacheType = float>
typedef CacheType shark::KernelSGDTrainer< InputType, CacheType >::QpFloatType

Definition at line 99 of file KernelSGDTrainer.h.

Constructor & Destructor Documentation

◆ KernelSGDTrainer()

template<class InputType, class CacheType = float>
shark::KernelSGDTrainer< InputType, CacheType >::KernelSGDTrainer ( KernelType kernel,
const LossType loss,
double  C,
bool  offset,
bool  unconstrained = false,
size_t  cacheSize = 0x4000000 
)
inline

Constructor.

Parameters
kernelkernel function to use for training and prediction
loss(sub-)differentiable loss function
Cregularization parameter - always the 'true' value of C, even when unconstrained is set
offsetwhether to train with offset/bias parameter or not
unconstrainedwhen a C-value is given via setParameter, should it be piped through the exp-function before using it in the solver?
cacheSizesize of the cache

Definition at line 113 of file KernelSGDTrainer.h.

Member Function Documentation

◆ C()

template<class InputType, class CacheType = float>
double shark::KernelSGDTrainer< InputType, CacheType >::C ( ) const
inline

return the value of the regularization parameter

Definition at line 224 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_C.

◆ cacheSize()

template<class InputType, class CacheType = float>
double shark::KernelSGDTrainer< InputType, CacheType >::cacheSize ( ) const
inline

return current cachesize

Definition at line 125 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_cacheSize.

◆ epochs()

template<class InputType, class CacheType = float>
std::size_t shark::KernelSGDTrainer< InputType, CacheType >::epochs ( ) const
inline

Return the number of training epochs. A value of 0 indicates that the default of max(10, C) should be used.

Definition at line 199 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_epochs.

◆ isUnconstrained()

template<class InputType, class CacheType = float>
bool shark::KernelSGDTrainer< InputType, CacheType >::isUnconstrained ( ) const
inline

check whether the parameter C is represented as log(C), thus, in a form suitable for unconstrained optimization, in the parameter vector

Definition at line 220 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_unconstrained.

◆ kernel() [1/2]

template<class InputType, class CacheType = float>
KernelType* shark::KernelSGDTrainer< InputType, CacheType >::kernel ( )
inline

◆ kernel() [2/2]

template<class InputType, class CacheType = float>
const KernelType* shark::KernelSGDTrainer< InputType, CacheType >::kernel ( ) const
inline

get the kernel function

Definition at line 211 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_kernel.

◆ name()

template<class InputType, class CacheType = float>
std::string shark::KernelSGDTrainer< InputType, CacheType >::name ( ) const
inlinevirtual

From INameable: return the class name.

Reimplemented from shark::INameable.

Definition at line 137 of file KernelSGDTrainer.h.

◆ numberOfParameters()

template<class InputType, class CacheType = float>
size_t shark::KernelSGDTrainer< InputType, CacheType >::numberOfParameters ( ) const
inlinevirtual

Returns the number of hyper-parameters.

Reimplemented from shark::IParameterizable<>.

Definition at line 254 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_kernel, and shark::IParameterizable< VectorType >::numberOfParameters().

◆ parameterVector()

template<class InputType, class CacheType = float>
RealVector shark::KernelSGDTrainer< InputType, CacheType >::parameterVector ( ) const
inlinevirtual

◆ setC()

template<class InputType, class CacheType = float>
void shark::KernelSGDTrainer< InputType, CacheType >::setC ( double  value)
inline

set the value of the regularization parameter (must be positive)

Definition at line 228 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_C, and RANGE_CHECK.

◆ setCacheSize()

template<class InputType, class CacheType = float>
void shark::KernelSGDTrainer< InputType, CacheType >::setCacheSize ( std::size_t  size)
inline

◆ setEpochs()

template<class InputType, class CacheType = float>
void shark::KernelSGDTrainer< InputType, CacheType >::setEpochs ( std::size_t  value)
inline

Set the number of training epochs. A value of 0 indicates that the default of max(10, C) should be used.

Definition at line 204 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_epochs.

◆ setKernel()

template<class InputType, class CacheType = float>
void shark::KernelSGDTrainer< InputType, CacheType >::setKernel ( KernelType kernel)
inline

◆ setParameterVector()

template<class InputType, class CacheType = float>
void shark::KernelSGDTrainer< InputType, CacheType >::setParameterVector ( RealVector const &  newParameters)
inlinevirtual

◆ train()

◆ trainOffset()

template<class InputType, class CacheType = float>
bool shark::KernelSGDTrainer< InputType, CacheType >::trainOffset ( ) const
inline

check whether the model to be trained should include an offset term

Definition at line 235 of file KernelSGDTrainer.h.

References shark::KernelSGDTrainer< InputType, CacheType >::m_offset.

Member Data Documentation

◆ m_C

◆ m_cacheSize

template<class InputType, class CacheType = float>
std::size_t shark::KernelSGDTrainer< InputType, CacheType >::m_cacheSize
protected

◆ m_epochs

template<class InputType, class CacheType = float>
std::size_t shark::KernelSGDTrainer< InputType, CacheType >::m_epochs
protected

number of training epochs (sweeps over the data), or 0 for default = max(10, C)

Definition at line 264 of file KernelSGDTrainer.h.

Referenced by shark::KernelSGDTrainer< InputType, CacheType >::epochs(), shark::KernelSGDTrainer< InputType, CacheType >::setEpochs(), and shark::KernelSGDTrainer< InputType, CacheType >::train().

◆ m_kernel

◆ m_loss

template<class InputType, class CacheType = float>
const LossType* shark::KernelSGDTrainer< InputType, CacheType >::m_loss
protected

pointer to loss function

Definition at line 260 of file KernelSGDTrainer.h.

◆ m_offset

template<class InputType, class CacheType = float>
bool shark::KernelSGDTrainer< InputType, CacheType >::m_offset
protected

should the resulting model have an offset term?

Definition at line 262 of file KernelSGDTrainer.h.

Referenced by shark::KernelSGDTrainer< InputType, CacheType >::train(), and shark::KernelSGDTrainer< InputType, CacheType >::trainOffset().

◆ m_unconstrained

template<class InputType, class CacheType = float>
bool shark::KernelSGDTrainer< InputType, CacheType >::m_unconstrained
protected

The documentation for this class was generated from the following file: