官术网_书友最值得收藏!

Detecting outliers

Given a variable, outliers are values that are very distant from other values of that variable. Outliers are quite common, and often caused by human or measurement errors. Outliers can strongly derail a model.

To demonstrate, let's look at two simple datasets and see how their mean is influenced by the presence of an outlier.

Consider the two datasets with few samples each: A = [1,2,3,4] and B = [1,2,3,4, 100]. The 5th value in the B dataset,  100, is obviously an outlier: mean(A) = 2.5, while mean(B) = 22. An outlier can have a large impact on a metric. Since most machine learning algorithms are based on distance or variance measurements, outliers can have a high impact on the performance of a model.

Multiple linear regression is sensitive to outlier effects, as shown in the following graph where adding a single outlier point derails the solid regression line into the dashed one:

Removing the samples associated with the outliers is the simplest solution.

Another solution can be to apply quantile binning to the predictor by splitting the values into N ordered intervals or bins, each approximately containing an equal number of samples. This will transform a numeric (continuous) predictor into a categorical one. For example, [1,2,3,4,5,6,7,8,9,10,11,100] split into three equally sized bins becomes [1,1,1,1,2,2,2,2,3,3,3,3]; the outlier value 100 has been included in the third bin and hidden.

The downside of quantile binning is that some granularity of information is lost in the process, which may degrade the performance of the model.

Quantile binning is available as a data transformation process in Amazon ML and is also used to quantify non-linearities in the original dataset.

In fact, Quantile Binning (QB) is applied by default by Amazon ML to all continuous variables that do not exhibit a straightforward linear relation to the outcome. In all our trials, and contrary to our prior assumptions, we have found that QB is a very efficient data transformation in the Amazon ML context.

主站蜘蛛池模板: 义乌市| 奈曼旗| 靖州| 仙居县| 朝阳市| 慈溪市| 桐城市| 大同市| 聂荣县| 大同市| 桃源县| 宁晋县| 茶陵县| 沂南县| 平阳县| 迭部县| 铁岭市| 鸡西市| 兴化市| 黄石市| 垦利县| 双峰县| 翁源县| 自贡市| 浦北县| 武威市| 松潘县| 永济市| 吴桥县| 清新县| 资讯 | 永和县| 原平市| 繁峙县| 新蔡县| 修武县| 宁陕县| 龙海市| 普安县| 苍山县| 华阴市|