A recurrent neural network regression model that learns with Back Propagation Through Time. More...

#include <shark/Models/RNNet.h>

Inheritance diagram for shark::RNNet:

Public Member Functions
	RNNet (RecurrentStructure *structure)

std::string	name () const
	From INameable: return the class name. More...

void	setWarmUpSequence (Sequence const &warmUpSequence=Sequence())
	Sets the warm up sequence. More...

boost::shared_ptr< State >	createState () const
	Creates an internal state of the model. More...

SHARK_EXPORT_SYMBOL void	eval (BatchInputType const &pattern, BatchOutputType &output, State &state) const
	Feed a data series to the model. The output (i.e., the time series of activations of the output neurons) it copied into the output buffer. More...

std::size_t	inputSize () const
	obtain the input dimension More...

std::size_t	outputSize () const
	obtain the output dimension More...

SHARK_EXPORT_SYMBOL void	weightedParameterDerivative (BatchInputType const &patterns, BatchInputType const &coefficients, State const &state, RealVector &gradient) const
	calculates the weighted sum of gradients w.r.t the parameters More...

RealVector	parameterVector () const
	get internal parameters of the model More...

void	setParameterVector (RealVector const &newParameters)

std::size_t	numberOfParameters () const
	number of parameters of the network More...

Public Member Functions inherited from shark::AbstractModel< Sequence, Sequence >
	AbstractModel ()

virtual	~AbstractModel ()

const Features &	features () const

virtual void	updateFeatures ()

bool	hasFirstParameterDerivative () const
	Returns true when the first parameter derivative is implemented. More...

bool	hasSecondParameterDerivative () const
	Returns true when the second parameter derivative is implemented. More...

bool	hasFirstInputDerivative () const
	Returns true when the first input derivative is implemented. More...

bool	hasSecondInputDerivative () const
	Returns true when the second parameter derivative is implemented. More...

bool	isSequential () const

virtual void	read (InArchive &archive)
	From ISerializable, reads a model from an archive. More...

virtual void	write (OutArchive &archive) const
	writes a model to an archive More...

virtual void	eval (BatchInputType const &patterns, BatchOutputType &outputs) const
	Standard interface for evaluating the response of the model to a batch of patterns. More...

virtual void	eval (InputType const &pattern, OutputType &output) const
	Standard interface for evaluating the response of the model to a single pattern. More...

Data< OutputType >	operator() (Data< InputType > const &patterns) const
	Model evaluation as an operator for a whole dataset. This is a convenience function. More...

OutputType	operator() (InputType const &pattern) const
	Model evaluation as an operator for a single pattern. This is a convenience function. More...

BatchOutputType	operator() (BatchInputType const &patterns) const
	Model evaluation as an operator for a single pattern. This is a convenience function. More...

virtual void	weightedParameterDerivative (BatchInputType const &pattern, BatchOutputType const &coefficients, State const &state, RealVector &derivative) const
	calculates the weighted sum of derivatives w.r.t the parameters. More...

virtual void	weightedParameterDerivative (BatchInputType const &pattern, BatchOutputType const &coefficients, Batch< RealMatrix >::type const &errorHessian, State const &state, RealVector &derivative, RealMatrix &hessian) const
	calculates the weighted sum of derivatives w.r.t the parameters More...

virtual void	weightedInputDerivative (BatchInputType const &pattern, BatchOutputType const &coefficients, State const &state, BatchInputType &derivative) const
	calculates the weighted sum of derivatives w.r.t the inputs More...

virtual void	weightedInputDerivative (BatchInputType const &pattern, BatchOutputType const &coefficients, typename Batch< RealMatrix >::type const &errorHessian, State const &state, RealMatrix &derivative, Batch< RealMatrix >::type &hessian) const
	calculates the weighted sum of derivatives w.r.t the inputs More...

virtual void	weightedDerivatives (BatchInputType const &patterns, BatchOutputType const &coefficients, State const &state, RealVector &parameterDerivative, BatchInputType &inputDerivative) const
	calculates weighted input and parameter derivative at the same time More...

Public Member Functions inherited from shark::IParameterizable
virtual	~IParameterizable ()

Public Member Functions inherited from shark::INameable
virtual	~INameable ()

Public Member Functions inherited from shark::ISerializable
virtual	~ISerializable ()
	Virtual d'tor. More...

void	load (InArchive &archive, unsigned int version)
	Versioned loading of components, calls read(...). More...

void	save (OutArchive &archive, unsigned int version) const
	Versioned storing of components, calls write(...). More...

	BOOST_SERIALIZATION_SPLIT_MEMBER ()

Protected Attributes
Sequence	m_warmUpSequence
	the warm up sequence of the network More...

RecurrentStructure *	mpe_structure
	the topology of the network. More...

RealMatrix	m_errorDerivative

Protected Attributes inherited from shark::AbstractModel< Sequence, Sequence >
Features	m_features

Additional Inherited Members
Public Types inherited from shark::AbstractModel< Sequence, Sequence >
enum	Feature

typedef Sequence	InputType
	Defines the input type of the model. More...

typedef Sequence	OutputType
	Defines the output type of the model. More...

typedef Batch< InputType >::type	BatchInputType
	defines the batch type of the input type. More...

typedef Batch< OutputType >::type	BatchOutputType
	defines the batch type of the output type More...

typedef TypedFlags< Feature >	Features

typedef TypedFeatureNotAvailableException< Feature >	FeatureNotAvailableException

Detailed Description

A recurrent neural network regression model that learns with Back Propagation Through Time.

This class defines a recurrent neural network regression model. Its inputs and output types are Matrices which represet sequences of inputs. The gradient is calculated via BackPropagationTroughTime (BPTT).

The inputs of this Network are not sigmoidal, but the hidden and output neurons are.

This class is optimized for batch learning. See OnlineRNNet for an online version.

Definition at line 56 of file RNNet.h.

Constructor & Destructor Documentation

◆ RNNet()

shark::RNNet::RNNet ( RecurrentStructure * structure )

inline

creates a neural network with a potentially shared structure

Parameters

structure the structure of this neural network. It can be shared between multiple instances or with then online version of this net.

Definition at line 72 of file RNNet.h.

References shark::AbstractModel< Sequence, Sequence >::HAS_FIRST_PARAMETER_DERIVATIVE, shark::AbstractModel< Sequence, Sequence >::m_features, mpe_structure, and SHARK_RUNTIME_CHECK.

Member Function Documentation

◆ createState()

boost::shared_ptr<State> shark::RNNet::createState ( ) const

inlinevirtual

Creates an internal state of the model.

The state is needed when the derivatives are to be calculated. Eval can store a state which is then reused to speed up the calculations of the derivatives. This also allows eval to be evaluated in parallel!

Reimplemented from shark::AbstractModel< Sequence, Sequence >.

Definition at line 104 of file RNNet.h.

References eval(), and SHARK_EXPORT_SYMBOL.

◆ eval()

SHARK_EXPORT_SYMBOL void shark::RNNet::eval	(	BatchInputType const &	pattern,
		BatchOutputType &	output,
		State &	state
	)		const

virtual

Feed a data series to the model. The output (i.e., the time series of activations of the output neurons) it copied into the output buffer.

Parameters

pattern	batch of timeseries for the network.
output	Used to store the outputs of the network.
state	stores additional information which can be reused for the computation of the derivative

Implements shark::AbstractModel< Sequence, Sequence >.

Referenced by createState().

◆ inputSize()

std::size_t shark::RNNet::inputSize ( ) const

inline

obtain the input dimension

Definition at line 119 of file RNNet.h.

References shark::RecurrentStructure::inputs(), and mpe_structure.

◆ name()

std::string shark::RNNet::name ( ) const

inlinevirtual

From INameable: return the class name.

Reimplemented from shark::INameable.

Definition at line 78 of file RNNet.h.

◆ numberOfParameters()

std::size_t shark::RNNet::numberOfParameters ( ) const

inlinevirtual

number of parameters of the network

Reimplemented from shark::IParameterizable.

Definition at line 165 of file RNNet.h.

References mpe_structure, and shark::RecurrentStructure::parameters().

◆ outputSize()

std::size_t shark::RNNet::outputSize ( ) const

inline

obtain the output dimension

Definition at line 124 of file RNNet.h.

References mpe_structure, shark::RecurrentStructure::outputs(), SHARK_EXPORT_SYMBOL, and weightedParameterDerivative().

◆ parameterVector()

RealVector shark::RNNet::parameterVector ( ) const

inlinevirtual

get internal parameters of the model

Reimplemented from shark::IParameterizable.

Definition at line 154 of file RNNet.h.

References mpe_structure, and shark::RecurrentStructure::parameterVector().

◆ setParameterVector()

void shark::RNNet::setParameterVector ( RealVector const & newParameters )

inlinevirtual

set internal parameters of the model

Parameters

newParameters the new parameters of the model. this changes the internal referenced RecurrentStructure

Reimplemented from shark::IParameterizable.

Definition at line 160 of file RNNet.h.

References mpe_structure, and shark::RecurrentStructure::setParameterVector().

◆ setWarmUpSequence()

void shark::RNNet::setWarmUpSequence ( Sequence const & warmUpSequence = Sequence() )

inline

Sets the warm up sequence.

Usually, when processing a new data series all the `states' of the network are reset to zero. By `states' I mean the buffered activations to which time-delayed synapses refer to. Effectively, this means one assumes a zero activation history.

The advantage of this is, that it makes the model behavior well defined. The disadvantage is that you can't predict a time series well with a zero history. Thus, one should use a data series to initialize the network, i.e., to let it converge into a `normal' dynamic state from which prediction of new data is possible. This phase is called the warmup phase.

With this method, the warm up sequence can be set, which is then used during the warm up phase.

Parameters

warmUpSequence the warm up sequence used before each batch of data. The default is an empty sequence

Definition at line 100 of file RNNet.h.

References m_warmUpSequence.

◆ weightedParameterDerivative()

SHARK_EXPORT_SYMBOL void shark::RNNet::weightedParameterDerivative	(	BatchInputType const &	patterns,
		BatchInputType const &	coefficients,
		State const &	state,
		RealVector &	gradient
	)		const

calculates the weighted sum of gradients w.r.t the parameters

The RNNet uses internally BPTT to calculate the gradient. Stores the BPTT error values for the calculation of the gradient.

Given the gradient of the loss function \( \frac{\delta L(t)}{\delta y_i(t)}\), the BPTT error is calculated as

\[ \frac{\delta E}{\delta y_i(t)}= \mu_i \frac{\delta L(t)}{\delta y_i(t)} +\sum_{j=1}^N \frac{\delta E}{\delta y_i(t+1)} y_i'(t+1) w^R_{ij} \]

Where \( L \) is the loss, \( y_i \) the ith neuron and \( w^R_ij\) is the recurrent weight of the connection from neuron i to j. The factor \( \mu_i \) is one of the neuron is an output neuron, else zero.

Todo:: expand documentation

Parameters

patterns	the batch of patterns to evaluate
coefficients	the coefficients which are used to calculate the weighted sum
state	the last state stord during eval
gradient	the calculated gradient

Referenced by outputSize().

Member Data Documentation

◆ m_errorDerivative

RealMatrix shark::RNNet::m_errorDerivative

protected

Definition at line 175 of file RNNet.h.

◆ m_warmUpSequence

Sequence shark::RNNet::m_warmUpSequence

protected

the warm up sequence of the network

Definition at line 170 of file RNNet.h.

Referenced by setWarmUpSequence().

◆ mpe_structure

RecurrentStructure* shark::RNNet::mpe_structure

protected

the topology of the network.

Definition at line 173 of file RNNet.h.

Referenced by inputSize(), numberOfParameters(), outputSize(), parameterVector(), RNNet(), and setParameterVector().

The documentation for this class was generated from the following file:

include/shark/Models/RNNet.h

Public Member Functions

Protected Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ RNNet()

Member Function Documentation

◆ createState()

◆ eval()

◆ inputSize()

◆ name()

◆ numberOfParameters()

◆ outputSize()

◆ parameterVector()

◆ setParameterVector()

◆ setWarmUpSequence()

◆ weightedParameterDerivative()

Member Data Documentation

◆ m_errorDerivative

◆ m_warmUpSequence

◆ mpe_structure