Adaptive Moment Estimation Algorithm (ADAM) More...
#include <shark/Algorithms/GradientDescent/Adam.h>
Public Member Functions | |
Adam () | |
std::string | name () const |
From INameable: return the class name. More... | |
void | init (ObjectiveFunctionType const &objectiveFunction, SearchPointType const &startingPoint) |
initializes the optimizer using a predefined starting point More... | |
double | eta () const |
get learning rate eta More... | |
void | setEta (double eta) |
set learning rate eta More... | |
double | beta1 () const |
get gradient averaging parameter beta1 More... | |
void | setBeta1 (double beta1) |
set gradient averaging parameter beta1 More... | |
double | beta2 () const |
get gradient averaging parameter beta2 More... | |
void | setBeta2 (double beta2) |
set gradient averaging parameter beta2 More... | |
double | epsilon () const |
get minimum noise estimate epsilon More... | |
void | setEpsilon (double epsilon) |
set minimum noise estimate epsilon More... | |
void | step (ObjectiveFunctionType const &objectiveFunction) |
Performs a step of the optimization. More... | |
virtual void | read (InArchive &archive) |
Read the component from the supplied archive. More... | |
virtual void | write (OutArchive &archive) const |
Write the component to the supplied archive. More... | |
Public Member Functions inherited from shark::AbstractSingleObjectiveOptimizer< RealVector > | |
std::size_t | numInitPoints () const |
By default most single objective optimizers only require a single point. More... | |
virtual void | init (ObjectiveFunctionType const &function, std::vector< SearchPointType > const &initPoints) |
Initialize the optimizer for the supplied objective function using a set of initialisation points. More... | |
virtual const SolutionType & | solution () const |
returns the current solution of the optimizer More... | |
Public Member Functions inherited from shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > > | |
const Features & | features () const |
virtual void | updateFeatures () |
bool | requiresValue () const |
bool | requiresFirstDerivative () const |
bool | requiresSecondDerivative () const |
bool | canSolveConstrained () const |
bool | requiresClosestFeasible () const |
virtual | ~AbstractOptimizer () |
virtual void | init (ObjectiveFunctionType const &function) |
Initialize the optimizer for the supplied objective function. More... | |
Public Member Functions inherited from shark::INameable | |
virtual | ~INameable () |
Public Member Functions inherited from shark::ISerializable | |
virtual | ~ISerializable () |
Virtual d'tor. More... | |
void | load (InArchive &archive, unsigned int version) |
Versioned loading of components, calls read(...). More... | |
void | save (OutArchive &archive, unsigned int version) const |
Versioned storing of components, calls write(...). More... | |
BOOST_SERIALIZATION_SPLIT_MEMBER () | |
Additional Inherited Members | |
Public Types inherited from shark::AbstractSingleObjectiveOptimizer< RealVector > | |
typedef base_type::SearchPointType | SearchPointType |
typedef base_type::SolutionType | SolutionType |
typedef base_type::ResultType | ResultType |
typedef base_type::ObjectiveFunctionType | ObjectiveFunctionType |
Public Types inherited from shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > > | |
enum | Feature |
Models features that the optimizer requires from the objective function. More... | |
typedef RealVector | SearchPointType |
typedef double | ResultType |
typedef SingleObjectiveResultSet< RealVector > | SolutionType |
typedef AbstractObjectiveFunction< RealVector, ResultType > | ObjectiveFunctionType |
typedef TypedFlags< Feature > | Features |
typedef TypedFeatureNotAvailableException< Feature > | FeatureNotAvailableException |
Protected Member Functions inherited from shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > > | |
void | checkFeatures (ObjectiveFunctionType const &objectiveFunction) |
Convenience function that checks whether the features of the supplied objective function match with the required features of the optimizer. More... | |
Protected Attributes inherited from shark::AbstractSingleObjectiveOptimizer< RealVector > | |
SolutionType | m_best |
Current solution of the optimizer. More... | |
Protected Attributes inherited from shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > > | |
Features | m_features |
Adaptive Moment Estimation Algorithm (ADAM)
Performs SGD by using a long term average of the gradient as well as its second moment to adapt a step size for each coordinate.
|
inline |
|
inline |
get gradient averaging parameter beta1
Definition at line 88 of file Adam.h.
Referenced by setBeta1().
|
inline |
get gradient averaging parameter beta2
Definition at line 99 of file Adam.h.
Referenced by setBeta2().
|
inline |
get minimum noise estimate epsilon
Definition at line 110 of file Adam.h.
Referenced by setEpsilon().
|
inline |
|
inlinevirtual |
initializes the optimizer using a predefined starting point
Implements shark::AbstractSingleObjectiveOptimizer< RealVector >.
Definition at line 61 of file Adam.h.
References shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > >::checkFeatures(), shark::AbstractObjectiveFunction< PointType, ResultT >::numberOfVariables(), and SHARK_RUNTIME_CHECK.
Referenced by main().
|
inlinevirtual |
From INameable: return the class name.
Reimplemented from shark::INameable.
|
inlinevirtual |
Read the component from the supplied archive.
[in,out] | archive | The archive to read from. |
Reimplemented from shark::ISerializable.
Definition at line 142 of file Adam.h.
References shark::AbstractSingleObjectiveOptimizer< RealVector >::m_best.
|
inline |
set gradient averaging parameter beta1
Definition at line 93 of file Adam.h.
References beta1(), and SHARK_RUNTIME_CHECK.
|
inline |
set gradient averaging parameter beta2
Definition at line 104 of file Adam.h.
References beta2(), and SHARK_RUNTIME_CHECK.
|
inline |
set minimum noise estimate epsilon
Definition at line 115 of file Adam.h.
References epsilon(), and SHARK_RUNTIME_CHECK.
|
inline |
set learning rate eta
Definition at line 82 of file Adam.h.
References eta(), and SHARK_RUNTIME_CHECK.
|
inlinevirtual |
Performs a step of the optimization.
First the current guess for gradient and its second moment are updated using
\[ g_t = \beta_1 g_{t-1} + (1-\beta1) \frac{\partial}{\partial x} f(x_{t-1})\]
\[ v_t = \beta_2 v_{t-1} + (1-\beta2) (\frac{\partial}{\partial x} f(x_{t-1}))^2\]
The step is then performed as
\[ x_{t} = x_{t-1} - \eta * g_t *(sqrt(v_t) + \epsilon)^{-1} \]
where a slight step correction is used to remove the bias in the first few iterations where the means are close to 0.
Implements shark::AbstractOptimizer< RealVector, double, SingleObjectiveResultSet< RealVector > >.
Definition at line 128 of file Adam.h.
Referenced by main().
|
inlinevirtual |
Write the component to the supplied archive.
[in,out] | archive | The archive to write to. |
Reimplemented from shark::ISerializable.
Definition at line 156 of file Adam.h.
References shark::AbstractSingleObjectiveOptimizer< RealVector >::m_best.