官术网_书友最值得收藏!

Detecting outliers

Given a variable, outliers are values that are very distant from other values of that variable. Outliers are quite common, and often caused by human or measurement errors. Outliers can strongly derail a model.

To demonstrate, let's look at two simple datasets and see how their mean is influenced by the presence of an outlier.

Consider the two datasets with few samples each: A = [1,2,3,4] and B = [1,2,3,4, 100]. The 5th value in the B dataset,  100, is obviously an outlier: mean(A) = 2.5, while mean(B) = 22. An outlier can have a large impact on a metric. Since most machine learning algorithms are based on distance or variance measurements, outliers can have a high impact on the performance of a model.

Multiple linear regression is sensitive to outlier effects, as shown in the following graph where adding a single outlier point derails the solid regression line into the dashed one:

Removing the samples associated with the outliers is the simplest solution.

Another solution can be to apply quantile binning to the predictor by splitting the values into N ordered intervals or bins, each approximately containing an equal number of samples. This will transform a numeric (continuous) predictor into a categorical one. For example, [1,2,3,4,5,6,7,8,9,10,11,100] split into three equally sized bins becomes [1,1,1,1,2,2,2,2,3,3,3,3]; the outlier value 100 has been included in the third bin and hidden.

The downside of quantile binning is that some granularity of information is lost in the process, which may degrade the performance of the model.

Quantile binning is available as a data transformation process in Amazon ML and is also used to quantify non-linearities in the original dataset.

In fact, Quantile Binning (QB) is applied by default by Amazon ML to all continuous variables that do not exhibit a straightforward linear relation to the outcome. In all our trials, and contrary to our prior assumptions, we have found that QB is a very efficient data transformation in the Amazon ML context.

主站蜘蛛池模板: 马关县| 阳谷县| 大庆市| 喀什市| 昂仁县| 望城县| 庄河市| 崇明县| 张北县| 离岛区| 嘉善县| 新宁县| 综艺| 肥城市| 时尚| 卢氏县| 特克斯县| 十堰市| 永吉县| 左贡县| 乌审旗| 太康县| 三原县| 乐东| 沈阳市| 阿城市| 永康市| 宁津县| 内江市| 湘潭县| 巢湖市| 江门市| 大田县| 麻城市| 四会市| 辉县市| 庐江县| 昌平区| 高碑店市| 绥江县| 江门市|