Extreme Learning Machine

During the last few years, the Extreme Learning Machine (ELM) got a lot of attention. Shark does not have a class tailored to ELMs but supports all building blocks to create one.

In theory an ELM is not more than a simple Feed Forward neural network where the hidden weights are not trained. For the ELM, we need the following include files:

#include <shark/Models/LinearModel.h>
#include <shark/Models/ConcatenatedModel.h>
#include <shark/Models/FFNet.h>
#include <shark/Algorithms/Trainers/NormalizeComponentsUnitVariance.h>
#include <shark/Algorithms/Trainers/LinearRegression.h>

To prevent that single inputs with high variance dominate the behaviour of the ELM, the inputs are required to have zero mean and unit variance. So we need a model which applies this transformation to the data. This is a linear transformation which can be learned by an affine linear model using the proper trainer:

Normalizer<> normalizer;
NormalizeComponentsUnitVariance<> normalizingTrainer(true);
normalizingTrainer.train(normalizer,data.inputs());

Now we can create the ELM. We first create an FFNet with a single hidden layer and initialize all its weights randomly. We will keep the random weights of the hidden units but overwrite the output weights later:

FFNet<LogisticNeuron,LinearNeuron> elmNetwork;
elmNetwork.setStructure(inputDim,hiddenNeurons,labelDimension(data));
initRandomNormal(elmNetwork,1);

Since the output weights of the ELM are linear, we can now use linear regression to train them. The best way to do is, is to propagate the training data forward through the normalization and hidden layer and use the outputs as input data for the linear regression. After the linear regression is done, we set the weights of the second layer of the FFNet to the linear weights:

RegressionDataset transformedData = transformInputs(data,normalizer);
transformedData.inputs() = elmNetwork.evalLayer(0,transformedData.inputs());
LinearModel<> elmOutput;
LinearRegression trainer;
trainer.train(elmOutput,transformedData);

//we need to set the learned weights of the hidden layer of the elm
elmNetwork.setLayer(1,elmOutput.matrix(),elmOutput.offset());

After training is done, the last thing we have to do is concatenate the normalizer with the network and present our shiny new elm to the world!

ConcatenatedModel<RealVector,RealVector> elm = normalizer >> elmNetwork;

Full example program

The full example program is elmTutorial.cpp.