Popular Activation Functions In Neural Networks
Popular Activation Functions In Neural Networks
In the neural network introduction article, we have discussed the basics of neural networks. This article focus is on different types of activation functions using in building neural networks.
In the deep learning literate or in neural network online courses, these activation functions are popularly called transfer functions.
The main focus of this article is to give you a complete overview of various activation functions and their properties. We’ll also see how to implement them in python.
Popular activation functions in neural networks.
Before we drive further, Let’s see the topic you are going to learn in this article.
So let’s begin with understanding what is the activation function. If you remember decision tree, at each node the decision tree algorithm needs to take decision to split the further data, we can related this to understand about activation functions.
What is Activation Function?
The name activation is self explainable. As the name suggests, the activation function is to alert or fire the neurons/node in neural networks.
If we treat these functions as a black box, like we treat many classifiction algorihtms , these functions will take input and return output, which helps the neural network to pass the value to the next nodes of the network.
Activation functions are vital components in the neural networks, which helps the network to learn the intricate patterns in train data, which helps in predicting the future.
In mathematical words, activation functions are used in neural networks to compute the weighted sum of input and biases, which is used to decide if a neuron can be fired or not.
We can related the computing the weighted sum with the linear regression concept.
It manipulates the presented data through some gradient processing, usually gradient descent, and afterward produces an output for the neural network that contains the parameters in the data.
Activation functions are often referred to as a transfer function in deep learning research papers literature. These activation functions are having a set of properties to follow.
Let’s discuss this.
Properties of activation functions
Properties of activation functions
These functions having the set of properties to follow,
- Computational Inexpensive
- Differentiable
- Zero Centered
Computational Inexpensive
The activation function computation has to be very minimal, as this impacts the neural network training period/time.
For the complicated neural network architectures such as the Convolutional neural network (CNN), Recurrent Neural Network (RNN) needs many parameters to optimize.
This optimization needs to compute the activation functions at each layer. If the activation functions are computational high, it will take a hell lot of time for getting the optimized weights at each layer in the network.
So the key properties activation function should follow computational inexpensiveness.
Differentiable
The second fundamental property is differentiable.
Activation functions have to be differentiable, even though we are having linear functions which are non-differentiable, to learn the complex patterns in the training data, the activation functions need to be differentiable.
Now raised the other questions
why the activation functions need to differentiable?
If you remember in the neural networks introduction article we explained the concept call backpropagation. Using the backpropagation the networks calculate the errors it’ made previously and using this information, it updates the weights accordingly to reduce the overall network error.
To perform this the network uses the gradient descent approach which needs the differential of the activation functions.
Zero Centered
The output of the activation functions needs to be zero centered, So this will help in the calculated gradients to be in the same direction and shifting across.
We discussed the key properties of the activation functions, now let’s discuss various categories of these functions.
Activation Function Categories
Activation Function Categories
At a high level, the activation function categorized into 3 types.
- Binary step functions
- Linear activation functions
- Nonlinear activation functions
Binary step functions
The simpler activation function is a step function. The output value depends on the threshold value we are considering. If the input value is greater than the threshold value the output will be 1, else the output will be 0.
This means if the input value is more than the threshold value which means the node has to fire, else no.
This is similar to the way how logistic regression predicts the binary target class.
In the above graph, we are considering the threshold value as zero. If the graph is not visible, scroll the code, you will find the graph.
This activation function can be used in binary classifications as the name says, however, it can not be used in a situation where you have multiple classes to deal with.
Why is it used?
Some cases call for a function which applies a hard threshold: either the output is precisely a single value, or not.
The other functions we have looked at have an intrinsic probabilistic output to them i.e. a higher output in decimal format implying a greater probability of being 1 (or high output).
The step function does away with this opting for a definite high or low output depending on some threshold on the input T.
However, the step-function is discontinuous and therefore non-differentiable. Therefore the use of this function in practice is not done with back-propagation.
Linear activation functions
The linear activation function is the simplest form of activation. If you use a linear activation function the wrong way, your whole neural network ends up being a regression.
Not sure about that,
Just think
How the network will be if we simple use the linear activation functions?
In the end, we need to add all the nodes activation functions, if we are using the linear activation function we will be adding all the linear functions. So the sum of all the linear functions is a linear function.
This makes the network a regression equation.
Linear activations are only needed when you’re considering a regression problem, as the last layer.
Why is it used?
If there’s a situation where we want a node to give its output without applying any thresholds, then the identity or linear function is the way to go.
The linear function is not used in the hidden layers. We must use non-linear transfer functions in the hidden layer nodes or else output will only ever end up being a linearly separable function.
Pros
Cons
Both the binary step function and the linear activation functions are not so famous in terms of deep learning complex and modern architectures. The nonlinear activation functions are mostly used.
So let’s discuss various nonlinear activation functions.
Nonlinear activation functions
We are having numerous non-linear activation functions, in this article we are mainly focussing on the below functions.
- Sigmoid Function
- Tanh Function
- Gaussian
- Relu
- Leaky Relu
Let’s start with the sigmoid function.
Sigmoid function
The sigmoid activation function is sometimes referred to as the logistic function or squashing function in some literature.
Why it is used?
This function maps the input to a value between 0 and 1 (but not equal to 0 or 1). This means the output from the node will be a high signal (if the input is positive) or a low one (if the input is negative).
The simplicity of its derivative allows us to efficiently perform backpropagation without using any fancy packages or approximations. The fact that this function is smooth, continuous, monotonic, and bounded means that backpropagation will work well.
The sigmoid’s natural threshold is 0.5, meaning that any input that maps to a value above 0.5 will be considered high (or 1) in binary terms.
Similary to this we have the softwax function which can used for multi classification problems. You can have a look at the key difference by reading the softmax Vs sigmod article.
Pros
Cons
Hyperbolic Tangent function
The hyperbolic tangent function known as the tanh function is a smoother zero-entered function whose range lies between -1 to 1.
Why is it used?
This is a very similar function to the previous sigmoid function and has much of the same properties, even its derivative is straight forward to compute. However, this function allows us to map the input to any value between -1 and 1 (but not inclusive of those).
In effect, this allows us to apply a penalty to the node (negative) rather than just have the node not fire at all.
This function has a natural threshold of 0, meaning that any input value greater than 0 is considered high (or 1) in binary terms.
Again, the fact that this function is smooth, continuous, monotonic, and bounded means that backpropagation will work well.
The subsequent functions don’t have all these properties which makes them more difficult to use in backpropagation.
Pros
Cons
Now the question is
what is the difference between sigmoid and hyperbolic tangent?
They both achieve a similar mapping, both are continuous, smooth, monotonic, and differentiable, but give out different values.
For a sigmoid function, a larger negative input generates an almost zero output. This lack of output will affect all subsequent weights in the network which may not be desirable - effectively stopping the next nodes from learning.
In contrast, the tanh function supplies -1 for negative values, maintaining the output of the node, and allowing subsequent nodes to learn from it.
Gaussian Function
Why is it used?
The Gaussian function is an even function, thus it gives the same output for equally positive and negative values of input. It gives its maximal output when there is no input and has decreasing output with increasing distance from zero.
We can perhaps imagine this function is used in a node where the input feature is less likely to contribute to the final result.
Rectified Linear Unit (ReLU)
This Relu is widely used in Convolutional Neural networks. As the function is just the max of input and zero, It so is easy to compute and does not saturate and does not cause the Vanishing Gradient Problem.
You may came accross these activation function in extracting the text from the hand written images.
Why is it used?
The ReLU represents a nearly linear function and therefore preserves the properties of linear models that made them easy to optimize, with gradient-descent methods.
This function rectifies the values of the inputs less than zero thereby forcing them to zero and eliminating the vanishing gradient problem observed in the earlier types of the activation function.
Pros
Cons
LeakyReLU
The LeakyRelu is a variant of ReLU. Instead of being 0 when 𝑧<0 z<0, a leaky ReLU allows a small, non-zero, constant gradient.
Why is it used?
As said before leakyRelu is a variant of Relu. Here alpha is a hyperparameter generally set to 0.01. Leaky ReLU solves the “dying ReLU” problem to some extent.
If you observe if we set α as 1 then Leaky ReLU will become a linear function f(x) = x and will be of no use. Hence, the value of alpha is never set close to 1. If we set alpha as a hyperparameter for each neuron separately, we get parametric ReLU or PReLU.
The activations functions are not limited to these but we have discussed widely used activation functions in the industry.
The below figure shows the different types of activation functions.
Different activation functions (Source: wikipedia)
Conclusion
To conclude, we provided a comprehensive summary of the activation functions used in deep learning.
The activation functions have the capability to improve the learning of the patterns in data there by automating the process of feature detection and justifying their use in the hidden layers of the neural networks.
Recommended Deep Learning Courses
Deep Learning A to Z Course
Rating: 4.5/5
Learn Deep Learning With Tensorflow
Rating: 4/5
Python Deep Learning Specialization
Rating: 4.5/5
