shark::Adam Class Reference

Adaptive Moment Estimation Algorithm (ADAM) More...

#include <shark/Algorithms/GradientDescent/Adam.h>

+ Inheritance diagram for shark::Adam:

Public Member Functions

 Adam ()
 
std::string name () const
 From INameable: return the class name. More...
 
void init (ObjectiveFunctionType const &objectiveFunction, SearchPointType const &startingPoint)
 initializes the optimizer using a predefined starting point More...
 
double eta () const
 get learning rate eta More...
 
void setEta (double eta)
 set learning rate eta More...
 
double beta1 () const
 get gradient averaging parameter beta1 More...
 
void setBeta1 (double beta1)
 set gradient averaging parameter beta1 More...
 
double beta2 () const
 get gradient averaging parameter beta2 More...
 
void setBeta2 (double beta2)
 set gradient averaging parameter beta2 More...
 
double epsilon () const
 get minimum noise estimate epsilon More...
 
void setEpsilon (double epsilon)
 set minimum noise estimate epsilon More...
 
void step (ObjectiveFunctionType const &objectiveFunction)
 Performs a step of the optimization. More...
 
virtual void read (InArchive &archive)
 Read the component from the supplied archive. More...
 
virtual void write (OutArchive &archive) const
 Write the component to the supplied archive. More...
 
- Public Member Functions inherited from shark::AbstractSingleObjectiveOptimizer< RealVector >
std::size_t numInitPoints () const
 By default most single objective optimizers only require a single point. More...
 
virtual void init (ObjectiveFunctionType const &function, std::vector< SearchPointType > const &initPoints)
 Initialize the optimizer for the supplied objective function using a set of initialisation points. More...
 
virtual const SolutionTypesolution () const
 returns the current solution of the optimizer More...
 
- Public Member Functions inherited from shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > >
const Featuresfeatures () const
 
virtual void updateFeatures ()
 
bool requiresValue () const
 
bool requiresFirstDerivative () const
 
bool requiresSecondDerivative () const
 
bool canSolveConstrained () const
 
bool requiresClosestFeasible () const
 
virtual ~AbstractOptimizer ()
 
virtual void init (ObjectiveFunctionType const &function)
 Initialize the optimizer for the supplied objective function. More...
 
- Public Member Functions inherited from shark::INameable
virtual ~INameable ()
 
- Public Member Functions inherited from shark::ISerializable
virtual ~ISerializable ()
 Virtual d'tor. More...
 
void load (InArchive &archive, unsigned int version)
 Versioned loading of components, calls read(...). More...
 
void save (OutArchive &archive, unsigned int version) const
 Versioned storing of components, calls write(...). More...
 
 BOOST_SERIALIZATION_SPLIT_MEMBER ()
 

Additional Inherited Members

- Public Types inherited from shark::AbstractSingleObjectiveOptimizer< RealVector >
typedef base_type::SearchPointType SearchPointType
 
typedef base_type::SolutionType SolutionType
 
typedef base_type::ResultType ResultType
 
typedef base_type::ObjectiveFunctionType ObjectiveFunctionType
 
- Public Types inherited from shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > >
enum  Feature
 Models features that the optimizer requires from the objective function. More...
 
typedef RealVector SearchPointType
 
typedef double ResultType
 
typedef SingleObjectiveResultSet< RealVector > SolutionType
 
typedef AbstractObjectiveFunction< RealVector, ResultTypeObjectiveFunctionType
 
typedef TypedFlags< FeatureFeatures
 
typedef TypedFeatureNotAvailableException< FeatureFeatureNotAvailableException
 
- Protected Member Functions inherited from shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > >
void checkFeatures (ObjectiveFunctionType const &objectiveFunction)
 Convenience function that checks whether the features of the supplied objective function match with the required features of the optimizer. More...
 
- Protected Attributes inherited from shark::AbstractSingleObjectiveOptimizer< RealVector >
SolutionType m_best
 Current solution of the optimizer. More...
 
- Protected Attributes inherited from shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > >
Features m_features
 

Detailed Description

Adaptive Moment Estimation Algorithm (ADAM)

Performs SGD by using a long term average of the gradient as well as its second moment to adapt a step size for each coordinate.

Definition at line 45 of file Adam.h.

Constructor & Destructor Documentation

◆ Adam()

Member Function Documentation

◆ beta1()

double shark::Adam::beta1 ( ) const
inline

get gradient averaging parameter beta1

Definition at line 88 of file Adam.h.

Referenced by setBeta1().

◆ beta2()

double shark::Adam::beta2 ( ) const
inline

get gradient averaging parameter beta2

Definition at line 99 of file Adam.h.

Referenced by setBeta2().

◆ epsilon()

double shark::Adam::epsilon ( ) const
inline

get minimum noise estimate epsilon

Definition at line 110 of file Adam.h.

Referenced by setEpsilon().

◆ eta()

double shark::Adam::eta ( ) const
inline

get learning rate eta

Definition at line 77 of file Adam.h.

Referenced by setEta().

◆ init()

void shark::Adam::init ( ObjectiveFunctionType const &  function,
SearchPointType const &  startingPoint 
)
inlinevirtual

◆ name()

std::string shark::Adam::name ( ) const
inlinevirtual

From INameable: return the class name.

Reimplemented from shark::INameable.

Definition at line 58 of file Adam.h.

◆ read()

virtual void shark::Adam::read ( InArchive archive)
inlinevirtual

Read the component from the supplied archive.

Parameters
[in,out]archiveThe archive to read from.

Reimplemented from shark::ISerializable.

Definition at line 142 of file Adam.h.

References shark::AbstractSingleObjectiveOptimizer< RealVector >::m_best.

◆ setBeta1()

void shark::Adam::setBeta1 ( double  beta1)
inline

set gradient averaging parameter beta1

Definition at line 93 of file Adam.h.

References beta1(), and SHARK_RUNTIME_CHECK.

◆ setBeta2()

void shark::Adam::setBeta2 ( double  beta2)
inline

set gradient averaging parameter beta2

Definition at line 104 of file Adam.h.

References beta2(), and SHARK_RUNTIME_CHECK.

◆ setEpsilon()

void shark::Adam::setEpsilon ( double  epsilon)
inline

set minimum noise estimate epsilon

Definition at line 115 of file Adam.h.

References epsilon(), and SHARK_RUNTIME_CHECK.

◆ setEta()

void shark::Adam::setEta ( double  eta)
inline

set learning rate eta

Definition at line 82 of file Adam.h.

References eta(), and SHARK_RUNTIME_CHECK.

◆ step()

void shark::Adam::step ( ObjectiveFunctionType const &  objectiveFunction)
inlinevirtual

Performs a step of the optimization.

First the current guess for gradient and its second moment are updated using

\[ g_t = \beta_1 g_{t-1} + (1-\beta1) \frac{\partial}{\partial x} f(x_{t-1})\]

\[ v_t = \beta_2 v_{t-1} + (1-\beta2) (\frac{\partial}{\partial x} f(x_{t-1}))^2\]

The step is then performed as

\[ x_{t} = x_{t-1} - \eta * g_t *(sqrt(v_t) + \epsilon)^{-1} \]

where a slight step correction is used to remove the bias in the first few iterations where the means are close to 0.

Implements shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > >.

Definition at line 128 of file Adam.h.

Referenced by main().

◆ write()

virtual void shark::Adam::write ( OutArchive archive) const
inlinevirtual

Write the component to the supplied archive.

Parameters
[in,out]archiveThe archive to write to.

Reimplemented from shark::ISerializable.

Definition at line 156 of file Adam.h.

References shark::AbstractSingleObjectiveOptimizer< RealVector >::m_best.


The documentation for this class was generated from the following file: