Activiation Functions

Why Activation Functions

In basic network like perceptron, all of the relationships are linear which can’t include all the situations. Whatever the hidden layer is, the connections are still linear. Therefore, activation functions are needed. After calculating every result with the activation, it is possible to get nonlinear functions.

What is activation functions

Activation is a simple function that can transform the result of linear calculation into nonlinear result.

Several common activation functions

• Relu
  Relu( The Rectified Linear Unit) is a often used activation function in hidden layers.
  Function:
  Figure:
  

• Sigmoid
  Also known as logistic function. It is usually seen in the study of herd population. In computer science, it can mapping the output into variables between 0 and 1.
  Function:
  Figure:
  

• Tanh
  Tanh (also known as Hyperbolic tangent) is generated by sinh and cosh
  Function:
  Figure:

• Softmax
  Sometimes, when doing project like recognizing the handwritten numbers, we need the model to output a single result. Therefore, the last dense layer is typically connected to a Softmax function that turns the output values into probabilities.
  Function:
  Figure: