shark::RNNet Class Reference

A recurrent neural network regression model that learns with Back Propagation Through Time. More...

#include <shark/Models/RNNet.h>

+ Inheritance diagram for shark::RNNet:

Public Member Functions

 RNNet (RecurrentStructure *structure)
 
std::string name () const
 From INameable: return the class name. More...
 
void setWarmUpSequence (Sequence const &warmUpSequence=Sequence())
 Sets the warm up sequence. More...
 
boost::shared_ptr< StatecreateState () const
 Creates an internal state of the model. More...
 
SHARK_EXPORT_SYMBOL void eval (BatchInputType const &pattern, BatchOutputType &output, State &state) const
 Feed a data series to the model. The output (i.e., the time series of activations of the output neurons) it copied into the output buffer. More...
 
std::size_t inputSize () const
 obtain the input dimension More...
 
std::size_t outputSize () const
 obtain the output dimension More...
 
SHARK_EXPORT_SYMBOL void weightedParameterDerivative (BatchInputType const &patterns, BatchInputType const &coefficients, State const &state, RealVector &gradient) const
 calculates the weighted sum of gradients w.r.t the parameters More...
 
RealVector parameterVector () const
 get internal parameters of the model More...
 
void setParameterVector (RealVector const &newParameters)
 
std::size_t numberOfParameters () const
 number of parameters of the network More...
 
- Public Member Functions inherited from shark::AbstractModel< Sequence, Sequence >
 AbstractModel ()
 
virtual ~AbstractModel ()
 
const Featuresfeatures () const
 
virtual void updateFeatures ()
 
bool hasFirstParameterDerivative () const
 Returns true when the first parameter derivative is implemented. More...
 
bool hasSecondParameterDerivative () const
 Returns true when the second parameter derivative is implemented. More...
 
bool hasFirstInputDerivative () const
 Returns true when the first input derivative is implemented. More...
 
bool hasSecondInputDerivative () const
 Returns true when the second parameter derivative is implemented. More...
 
bool isSequential () const
 
virtual void read (InArchive &archive)
 From ISerializable, reads a model from an archive. More...
 
virtual void write (OutArchive &archive) const
 writes a model to an archive More...
 
virtual void eval (BatchInputType const &patterns, BatchOutputType &outputs) const
 Standard interface for evaluating the response of the model to a batch of patterns. More...
 
virtual void eval (InputType const &pattern, OutputType &output) const
 Standard interface for evaluating the response of the model to a single pattern. More...
 
Data< OutputTypeoperator() (Data< InputType > const &patterns) const
 Model evaluation as an operator for a whole dataset. This is a convenience function. More...
 
OutputType operator() (InputType const &pattern) const
 Model evaluation as an operator for a single pattern. This is a convenience function. More...
 
BatchOutputType operator() (BatchInputType const &patterns) const
 Model evaluation as an operator for a single pattern. This is a convenience function. More...
 
virtual void weightedParameterDerivative (BatchInputType const &pattern, BatchOutputType const &coefficients, State const &state, RealVector &derivative) const
 calculates the weighted sum of derivatives w.r.t the parameters. More...
 
virtual void weightedParameterDerivative (BatchInputType const &pattern, BatchOutputType const &coefficients, Batch< RealMatrix >::type const &errorHessian, State const &state, RealVector &derivative, RealMatrix &hessian) const
 calculates the weighted sum of derivatives w.r.t the parameters More...
 
virtual void weightedInputDerivative (BatchInputType const &pattern, BatchOutputType const &coefficients, State const &state, BatchInputType &derivative) const
 calculates the weighted sum of derivatives w.r.t the inputs More...
 
virtual void weightedInputDerivative (BatchInputType const &pattern, BatchOutputType const &coefficients, typename Batch< RealMatrix >::type const &errorHessian, State const &state, RealMatrix &derivative, Batch< RealMatrix >::type &hessian) const
 calculates the weighted sum of derivatives w.r.t the inputs More...
 
virtual void weightedDerivatives (BatchInputType const &patterns, BatchOutputType const &coefficients, State const &state, RealVector &parameterDerivative, BatchInputType &inputDerivative) const
 calculates weighted input and parameter derivative at the same time More...
 
- Public Member Functions inherited from shark::IParameterizable
virtual ~IParameterizable ()
 
- Public Member Functions inherited from shark::INameable
virtual ~INameable ()
 
- Public Member Functions inherited from shark::ISerializable
virtual ~ISerializable ()
 Virtual d'tor. More...
 
void load (InArchive &archive, unsigned int version)
 Versioned loading of components, calls read(...). More...
 
void save (OutArchive &archive, unsigned int version) const
 Versioned storing of components, calls write(...). More...
 
 BOOST_SERIALIZATION_SPLIT_MEMBER ()
 

Protected Attributes

Sequence m_warmUpSequence
 the warm up sequence of the network More...
 
RecurrentStructurempe_structure
 the topology of the network. More...
 
RealMatrix m_errorDerivative
 
- Protected Attributes inherited from shark::AbstractModel< Sequence, Sequence >
Features m_features
 

Additional Inherited Members

- Public Types inherited from shark::AbstractModel< Sequence, Sequence >
enum  Feature
 
typedef Sequence InputType
 Defines the input type of the model. More...
 
typedef Sequence OutputType
 Defines the output type of the model. More...
 
typedef Batch< InputType >::type BatchInputType
 defines the batch type of the input type. More...
 
typedef Batch< OutputType >::type BatchOutputType
 defines the batch type of the output type More...
 
typedef TypedFlags< FeatureFeatures
 
typedef TypedFeatureNotAvailableException< FeatureFeatureNotAvailableException
 

Detailed Description

A recurrent neural network regression model that learns with Back Propagation Through Time.

This class defines a recurrent neural network regression model. Its inputs and output types are Matrices which represet sequences of inputs. The gradient is calculated via BackPropagationTroughTime (BPTT).

The inputs of this Network are not sigmoidal, but the hidden and output neurons are.

This class is optimized for batch learning. See OnlineRNNet for an online version.

Definition at line 56 of file RNNet.h.

Constructor & Destructor Documentation

◆ RNNet()

shark::RNNet::RNNet ( RecurrentStructure structure)
inline

creates a neural network with a potentially shared structure

Parameters
structurethe structure of this neural network. It can be shared between multiple instances or with then online version of this net.

Definition at line 72 of file RNNet.h.

References shark::AbstractModel< Sequence, Sequence >::HAS_FIRST_PARAMETER_DERIVATIVE, shark::AbstractModel< Sequence, Sequence >::m_features, mpe_structure, and SHARK_RUNTIME_CHECK.

Member Function Documentation

◆ createState()

boost::shared_ptr<State> shark::RNNet::createState ( ) const
inlinevirtual

Creates an internal state of the model.

The state is needed when the derivatives are to be calculated. Eval can store a state which is then reused to speed up the calculations of the derivatives. This also allows eval to be evaluated in parallel!

Reimplemented from shark::AbstractModel< Sequence, Sequence >.

Definition at line 104 of file RNNet.h.

References eval(), and SHARK_EXPORT_SYMBOL.

◆ eval()

SHARK_EXPORT_SYMBOL void shark::RNNet::eval ( BatchInputType const &  pattern,
BatchOutputType output,
State state 
) const
virtual

Feed a data series to the model. The output (i.e., the time series of activations of the output neurons) it copied into the output buffer.

Parameters
patternbatch of timeseries for the network.
outputUsed to store the outputs of the network.
statestores additional information which can be reused for the computation of the derivative

Implements shark::AbstractModel< Sequence, Sequence >.

Referenced by createState().

◆ inputSize()

std::size_t shark::RNNet::inputSize ( ) const
inline

obtain the input dimension

Definition at line 119 of file RNNet.h.

References shark::RecurrentStructure::inputs(), and mpe_structure.

◆ name()

std::string shark::RNNet::name ( ) const
inlinevirtual

From INameable: return the class name.

Reimplemented from shark::INameable.

Definition at line 78 of file RNNet.h.

◆ numberOfParameters()

std::size_t shark::RNNet::numberOfParameters ( ) const
inlinevirtual

number of parameters of the network

Reimplemented from shark::IParameterizable.

Definition at line 165 of file RNNet.h.

References mpe_structure, and shark::RecurrentStructure::parameters().

◆ outputSize()

std::size_t shark::RNNet::outputSize ( ) const
inline

obtain the output dimension

Definition at line 124 of file RNNet.h.

References mpe_structure, shark::RecurrentStructure::outputs(), SHARK_EXPORT_SYMBOL, and weightedParameterDerivative().

◆ parameterVector()

RealVector shark::RNNet::parameterVector ( ) const
inlinevirtual

get internal parameters of the model

Reimplemented from shark::IParameterizable.

Definition at line 154 of file RNNet.h.

References mpe_structure, and shark::RecurrentStructure::parameterVector().

◆ setParameterVector()

void shark::RNNet::setParameterVector ( RealVector const &  newParameters)
inlinevirtual

set internal parameters of the model

Parameters
newParametersthe new parameters of the model. this changes the internal referenced RecurrentStructure

Reimplemented from shark::IParameterizable.

Definition at line 160 of file RNNet.h.

References mpe_structure, and shark::RecurrentStructure::setParameterVector().

◆ setWarmUpSequence()

void shark::RNNet::setWarmUpSequence ( Sequence const &  warmUpSequence = Sequence())
inline

Sets the warm up sequence.

Usually, when processing a new data series all the `states' of the network are reset to zero. By `states' I mean the buffered activations to which time-delayed synapses refer to. Effectively, this means one assumes a zero activation history.

The advantage of this is, that it makes the model behavior well defined. The disadvantage is that you can't predict a time series well with a zero history. Thus, one should use a data series to initialize the network, i.e., to let it converge into a `normal' dynamic state from which prediction of new data is possible. This phase is called the warmup phase.

With this method, the warm up sequence can be set, which is then used during the warm up phase.

Parameters
warmUpSequencethe warm up sequence used before each batch of data. The default is an empty sequence

Definition at line 100 of file RNNet.h.

References m_warmUpSequence.

◆ weightedParameterDerivative()

SHARK_EXPORT_SYMBOL void shark::RNNet::weightedParameterDerivative ( BatchInputType const &  patterns,
BatchInputType const &  coefficients,
State const &  state,
RealVector &  gradient 
) const

calculates the weighted sum of gradients w.r.t the parameters

The RNNet uses internally BPTT to calculate the gradient. Stores the BPTT error values for the calculation of the gradient.

Given the gradient of the loss function \( \frac{\delta L(t)}{\delta y_i(t)}\), the BPTT error is calculated as

\[ \frac{\delta E}{\delta y_i(t)}= \mu_i \frac{\delta L(t)}{\delta y_i(t)} +\sum_{j=1}^N \frac{\delta E}{\delta y_i(t+1)} y_i'(t+1) w^R_{ij} \]

Where \( L \) is the loss, \( y_i \) the ith neuron and \( w^R_ij\) is the recurrent weight of the connection from neuron i to j. The factor \( \mu_i \) is one of the neuron is an output neuron, else zero.

Todo:
expand documentation
Parameters
patternsthe batch of patterns to evaluate
coefficientsthe coefficients which are used to calculate the weighted sum
statethe last state stord during eval
gradientthe calculated gradient

Referenced by outputSize().

Member Data Documentation

◆ m_errorDerivative

RealMatrix shark::RNNet::m_errorDerivative
protected

Definition at line 175 of file RNNet.h.

◆ m_warmUpSequence

Sequence shark::RNNet::m_warmUpSequence
protected

the warm up sequence of the network

Definition at line 170 of file RNNet.h.

Referenced by setWarmUpSequence().

◆ mpe_structure

RecurrentStructure* shark::RNNet::mpe_structure
protected

the topology of the network.

Definition at line 173 of file RNNet.h.

Referenced by inputSize(), numberOfParameters(), outputSize(), parameterVector(), RNNet(), and setParameterVector().


The documentation for this class was generated from the following file: