官术网_书友最值得收藏!

Rectified linear unit (ReLU)

The output of a ReLU is linear when the total input to the neuron is greater than zero, and the output is zero when the total input to the neuron is negative. This simple activation function provides nonlinearity to a neural network, and, at the same time, it provides a constant gradient of one with respect to the total input. This constant gradient helps to keep the neural network from developing saturating or vanishing gradient problems, as seen in activation functions, such as sigmoid and tanh activation units. The ReLU function output (as shown in Figure 1.8) can be expressed as follows:

The ReLU activation function can be plotted as follows:

Figure 1.8: ReLU activation function

One of the constraints for ReLU is its zero gradients for negative values of input. This may slow down the training, especially at the initial phase. Leaky ReLU activation functions (as shown in Figure 1.9) can be useful in this scenario, where the output and gradients are nonzero, even for negative values of the input. A leaky ReLU output function can be expressed as follows:

       

The  parameter is to be provided for leaky ReLU activation functions, whereas for a parametric ReLU,  is a parameter that the neural network will learn through training. The following graph shows the output of the leaky ReLU activation function:

Figure 1.9: Leaky ReLU activation function
主站蜘蛛池模板: 屯留县| 慈利县| 抚顺市| 门头沟区| 安西县| 新龙县| 徐汇区| 西乌珠穆沁旗| 汤阴县| 遂昌县| 漾濞| 华坪县| 河西区| 常山县| 班玛县| 信丰县| 漳浦县| 玉门市| 镶黄旗| 登封市| 阿坝| 灌南县| 莱阳市| 扶沟县| 白河县| 南宫市| 嵊泗县| 太谷县| 额济纳旗| 诏安县| 海宁市| 清涧县| 平武县| 雷波县| 德阳市| 唐河县| 上饶市| 和平县| 旌德县| 宝应县| 上杭县|