An activation function is the function or layer which enables neural network to learn complex(non-linear) relationships by transforming the output of the previous layer. *Without activation functions, neural network can only learn linear relationships.
There are popular activation functions as shown below:
(1) Step Function:
- can convert input values to 0 or 1. *If
input
< 0, then 0 while ifinput
>= 0, then 1. - is also called Binary Step Function, Unit Step Function, Binary Threshold Function, Threshold Function, Heaviside Step Function or Heaviside Function.
- is heaviside() in PyTorch.
(2) ReLU(Rectified Linear Unit) Function:
- can convert input values to the output values between 0 and
input
. *Ifinput
<= 0, then 0 while ifinput
> 0, theninput
. - is ReLU() in PyTorch.
(3) Leaky ReLU(Leaky Rectified Linear Unit) Function:
- can convert input values to the output values between
input
*slope
andinput
. *Memos:- If
input
<= 0, theninput
*slope
while ifinput
> 0, theninput
. -
slope
is 0.01 basically.
- If
- is the improved version of ReLU Function.
- is LeakyReLU() in PyTorch.
(4) ELU(Exponential Linear Unit) Function:
- can convert input values to the output values between
a
* (einput
- 1) andinput
. *Memos:- If
input
<= 0, thena
* (einput
- 1) while ifinput
> 0, theninput
. -
a
is 1.0 basically.
- If
- is the improved version of ReLU Function.
- is ELU() in PyTorch.
(5) Sigmoid Function:
- can convert input values to the output values between 0 and 1.
- 's formula is y = 1 / (1 + e-x).
- is Sigmoid() in PyTorch.
(6) Tanh Function:
- can convert input values to the output values between -1 and 1.
- 's formula is y = (ex - e-x) / (ex + e-x).
- is also called Hyperbolic Tangent Function.
- is Tanh() in PyTorch.
(7) Softmax Function:
- can convert input values to the output values between 0 and 1 each and whose sum is 1(100%). *If input values are [5, 4, -1], then the output values are [0.730, 0.268, 0.002] which is 0.730(73%) + 0.268(26.8%) + 0.002(0.2%) = 1(100%).
- 's formula is:
- is Softmax() in PyTorch.