官术网_书友最值得收藏!

Creating an Outlier report to give to SMEs>

It is quite common that the data miner has to rely on others to either provide data or interpret data, or both. Even when the data miner is working with data from their own organization there will be input variables that they don't have direct access to, or that are outside their day-to-day experience.

Are zero values normal? What about negative values? Null values? Are 1500 balance inquiries in a month even possible? How could a wallet cost $19,500? The concept of outliers is something that all analysts are familiar with. Even novice users of Modeler could easily find a dozen ways of identifying some. This recipe is about identifying outliers systematically and quickly so that you can produce a report designed to inspire curiosity.

There is no presumption that the data is in error, or that they should be removed. It is simply an attempt to put the information in the hands of Subject Matter Experts, so quirky values can be discussed in the earliest phases of the projects. It is important to provide whichever primary keys are necessary for the SMEs to look up the records. On one of the author's recent projects, the team started calling these reports quirk reports.

Getting ready

We will start with the Outlier Report.str stream that uses the TELE_CHURN_preprep data set.

How to do it...

To create an Outlier report:

  1. Open the stream Outlier Report.str.
  2. Add a Data Audit node and examine the results.
  3. Adjust the stream options to allow for 25 rows to be shown in a data preview. We will be using the preview feature later in the recipe.
  4. Add a Statistics node. Choose Mean, Min, Max, and Median for the variables DATA_gb, PEAK_mins, and TEXT_count. These three have either unusually high maximums or surprising negative values as shown in the Data Audit node.
  5. Consider taking a screenshot of the Statistics node for later use.
  6. Add a Sort node. Starting with the first variable, DATA_gb, sort in ascending order.
  7. Add a Filter node downstream of the Sort node dropping CHURN, DROPPED_CALLS, and LATE_PAYMENTS. It is important to work with your SME to know which variables put quirky values into context.
  8. Preview the Filter node. Consider the following screenshot:
  9. Reverse the sort, now choosing descending order, and preview the Filter node. Consider the following screenshot for later use:
  10. Sort in descending order on the next variable, PEAK_mins. Preview the Filter node.
  11. Finally sort the variable, TEXT_count, in descending order and preview the Filter node.
  12. Examine Outliers.docx to see an example of what this might look like in Word.

How it works...

There is no deep theoretical foundation to this recipe; it is as straightforward as it seems. It is simply a way of quickly getting information to an SME. They will not be frequent Modeler users. Also summary statistics only give them a part of the story. Providing the min, max, mean and median alone will not allow an SME to give you the information that you need. If there is a usual min such as a negative value, you need to know how many negatives there are, and need at least a handful of actual examples with IDs. An SME might look up to values in their own resources and the net result could be the addition of more variables to the analysis. Alternatively, negative values might be turned into nulls or zeros. Negative values might be deemed out of scope and removed from the analysis. There is no way to know until you assess why they are negative. Sometimes values that are exactly zero are of interest. High values, NULL values, and rare categories are all of potential interest. The most important thing is to be curious (and pleasantly persistent) and to inspire collaborators to be curious as well.

See also

  • The Selecting variables using the CHAID Modeling node recipe in Chapter 2, Data Preparation – Select
  • The Removing redundant variables using correlation matrices recipe in Chapter 2, Data Preparation – Select
主站蜘蛛池模板: 隆化县| 金溪县| 府谷县| 无棣县| 花莲市| 大兴区| 共和县| 清镇市| 永福县| 武邑县| 高雄县| 乌苏市| 拉孜县| 清远市| 利津县| 平度市| 南和县| 米脂县| 长乐市| 板桥市| 虞城县| 调兵山市| 元氏县| 习水县| 开江县| 汪清县| 东安县| 齐齐哈尔市| 延长县| 全南县| 泗水县| 蒲城县| 宜兰县| 肃宁县| 友谊县| 滁州市| 内乡县| 中牟县| 夏河县| 阳东县| 应城市|