官术网_书友最值得收藏!

Rectified linear unit (ReLU)

The output of a ReLU is linear when the total input to the neuron is greater than zero, and the output is zero when the total input to the neuron is negative. This simple activation function provides nonlinearity to a neural network, and, at the same time, it provides a constant gradient of one with respect to the total input. This constant gradient helps to keep the neural network from developing saturating or vanishing gradient problems, as seen in activation functions, such as sigmoid and tanh activation units. The ReLU function output (as shown in Figure 1.8) can be expressed as follows:

The ReLU activation function can be plotted as follows:

Figure 1.8: ReLU activation function

One of the constraints for ReLU is its zero gradients for negative values of input. This may slow down the training, especially at the initial phase. Leaky ReLU activation functions (as shown in Figure 1.9) can be useful in this scenario, where the output and gradients are nonzero, even for negative values of the input. A leaky ReLU output function can be expressed as follows:

       

The  parameter is to be provided for leaky ReLU activation functions, whereas for a parametric ReLU,  is a parameter that the neural network will learn through training. The following graph shows the output of the leaky ReLU activation function:

Figure 1.9: Leaky ReLU activation function
主站蜘蛛池模板: 韩城市| 莲花县| 治多县| 清苑县| 兴义市| 阿城市| 西丰县| 安化县| 政和县| 名山县| 辽源市| 苗栗县| 枞阳县| 彭山县| 彰化县| 鄢陵县| 阿克| 北宁市| 商丘市| 师宗县| 综艺| 蒙城县| 灵川县| 邛崃市| 江津市| 佛坪县| 马边| 阿图什市| 泽州县| 五原县| 宁波市| 临颍县| 天长市| 霍山县| 宝山区| 绥滨县| 介休市| 宁南县| 邻水| 木里| 美姑县|