- Deep Learning with PyTorch
- Vishnu Subramanian
- 182字
- 2021-06-24 19:16:28
ReLU
ReLU has become more popular in the recent years; we can find either its usage or one of its variants' usages in almost any modern architecture. It has a simple mathematical formulation:
f(x)=max(0,x)
In simple words, ReLU squashes any input that is negative to zero and leaves positive numbers as they are. We can visualize the ReLU function as follows:

Image source: http://datareview.info/article/eto-nuzhno-znat-klyuchevyie-rekomendatsii-po-glubokomu-obucheniyu-chast-2/
Some of the pros and cons of using ReLU are as follows:
- It helps the optimizer in finding the right set of weights sooner. More technically it makes the convergence of stochastic gradient descent faster.
- It is computationally inexpensive, as we are just thresholding and not calculating anything like we did for the sigmoid and tangent functions.
- ReLU has one disadvantage; when a large gradient passes through it during the backward propagation, they often become non-responsive; these are called dead neutrons, which can be controlled by carefully choosing the learning rate. We will discuss how to choose learning rates when we discuss the different ways to adjust the learning rate in Chapter 4, Fundamentals of Machine Learning.
推薦閱讀
- 用“芯”探核:龍芯派開發(fā)實(shí)戰(zhàn)
- 精選單片機(jī)設(shè)計(jì)與制作30例(第2版)
- 平衡掌控者:游戲數(shù)值經(jīng)濟(jì)設(shè)計(jì)
- AMD FPGA設(shè)計(jì)優(yōu)化寶典:面向Vivado/SystemVerilog
- Camtasia Studio 8:Advanced Editing and Publishing Techniques
- Learning Stencyl 3.x Game Development Beginner's Guide
- Learning Game Physics with Bullet Physics and OpenGL
- 分布式微服務(wù)架構(gòu):原理與實(shí)戰(zhàn)
- Visual Media Processing Using Matlab Beginner's Guide
- 筆記本電腦使用、維護(hù)與故障排除從入門到精通(第5版)
- OpenGL Game Development By Example
- 單片機(jī)開發(fā)與典型工程項(xiàng)目實(shí)例詳解
- Neural Network Programming with Java(Second Edition)
- Hands-On One-shot Learning with Python
- 計(jì)算機(jī)組裝與維護(hù)立體化教程(微課版)