# Other Activation Functions¶

Activation functions are simple functions used to transform the outputs of another model. They appear often in the design of neural networks which are a cascade of linear or convolutional models, followed by a nonlinear activation function.

There are currently three models for which an activation function can be specified.

• LinearModel creates a model of the form

$f(x) = g(Ax+b)$

where g is the nonlinear activation function.

• Conv2DModel behaves like a linear model but uses a convolution to model the linear part

• NeuronLayer simply computes:

$f(x) = g(x)$

The activation function is chosen as a template argument, e.g. LinearModel<RealVector, LogisticNeuron> chooses g as the logistic function. Most activation functions are applied elementwise, but also interactions between several outputs can be modeled. The following neurons are implemented in shark:

• LinearNeuron: not a good choice for hidden neurons, but for output neurons when the output is not bounded. This activation function $$f(x)=x$$ is the typical choice for regression tasks and when training with cross-entropy. It is also the default activation for LinearModel and Conv2DModel.
• LogisticNeuron: is a sigmoid (S-shaped) function with outputs in the range [0,1] and the following definition
$f(x) = \frac 1 {1+e^{-x}}.$
• TanhNeuron: the hyperbolic tangens, can be viewed as a rescaled version of the Logistic function with outputs ranging from [-1,1]. It has the formula
$f(x) = \tanh(x) = \frac 2 {1+e^{-2x}}-1.$
• FastSigmoidNeuron: a sigmoidal function which is faster to evaluate than the previous two activation functions. It has also “bigger tails” (i.e., the gradient does not vanish as quickly).
$f(x) = \frac x {1+|x|}.$
• RectifierNeuron: Also known as ReLu-Unit. The neuron’s activation is kept at 0 for negative activation levels and is linear for all positive values. A network with these neurons is effectively a piecewise linear function.
\begin{align}\begin{aligned}f(x) = \max(0,x).\\* :doxy:NormalizerNeuron: Normalizes each output by the sum of activations. Assumes that activations are positive and thus can be used to tansform positive numbers (e.g. counts) to probabilities\end{aligned}\end{align}
$f_i(x) = \frac {x_i}{\sum_j x_j}.$
• SoftmaxNeuron: Computes the softmax activation of the inputs. This model is the correct function to transform the outputs of a model trained with CrossEntropy to a proper class probability.

$f_i(x) = \frac {\exp(x_i=}{\sum_j exp(x_j)}.$