Activation functions

Activation functions are an integral part of any deep learning model. An activation function is a mathematical function that squashes the input values into a certain range. Suppose you feed in a neural network with real number inputs and initialize the weight matrix with random numbers and wish to use the output to classify; that is, you need the output value to be in between zero and one, but your neuron can output any value like -2.2453 or 17854.763. So, there is a need for scaling the output to a specific range. This is what an activation function does:

There are a lot of activation functions depending on the requirements. We will discuss some of the activation functions that are used quite often in deep learning.