Activation functions

One of the important components in a neural network is the activation function. The activation function can transform the input of a neural network node into non-linear output. Such activation functions enable the neural network to learn arbitrary non-linear mappings or patterns from data. The activation can be thought of as an event of firing a node. As explained previously, in the case of a unit step function, the perceptron is either fired or not fired, corresponding to a value of 1 or 0, respectively. There are other kinds of activation functions, such as sigmoid, hyperbolic target, and rectified linear unit (ReLU), which will be discussed next.