- Practical Data Analysis Cookbook
- Tomasz Drabas
- 498字
- 2021-07-16 11:13:54
Binning the observations
Binning the observations comes in handy when we want to check the shape of the distribution visually or we want to transform the data into an ordinal form.
Getting ready
To execute this recipe, you will need the pandas
and NumPy
modules.
No other prerequisites are required.
How to do it…
To bin your observations (as in a histogram), you can use the following code (data_binning.py file):
# create bins for the price that are based on the # linearly spaced range of the price values bins = np.linspace( csv_read['price_mean'].min(), csv_read['price_mean'].max(), 6 ) # and apply the bins to the data csv_read['b_price'] = np.digitize( csv_read['price_mean'], bins )
How it works…
First, we create bins. For our price (with the mean imputed in place of missing observations), we create six bins, evenly spread between the minimum and maximum values for the price. The .linspace(...)
method does exactly this: it creates a NumPy
array with six elements, each greater than the preceding one by the same value. For example, a .linspace(0,6,6)
command would generate an array, [0., 1.2, 2.4, 3.6, 4.8, 6.]
.
Note
NumPy
is a powerful numerical library for linear algebra. It can easily handle large arrays and matrices and offers a plethora of supplemental functions to operate on such data. For more information, visit http://www.numpy.org.
The .digitize(...)
method returns, for each value in the specified column, the index of the bin that the value belongs to. The first parameter is the column to be binned and the second one is the array with bins.
To count the records within each bin, we use the .value_counts()
method of DataFrame, counts_b = csv_read['b_price'].value_counts()
.
There's more…
Sometimes, instead of having evenly-spaced values, we would like to have equal counts in each bucket. To attain such a goal, we can use quantiles.
Tip
Quantiles are closely related to percentiles. The difference is percentiles return values at a given sample percentage, while quantiles return values at the sample fraction. For more information, visit https://www.stat.auckland.ac.nz/~ihaka/787/lectures-quantiles-handouts.pdf.
What we want to achieve is splitting our column into deciles, that is, 10 bins of (more or less) equal size. To do this, we can use the following code (you can easily spot the similarities with the previous approach):
# create bins based on deciles decile = csv_read['price_mean'].quantile(np.linspace(0,1,11)) # and apply the decile bins to the data csv_read['p_price'] = np.digitize( csv_read['price_mean'], decile )
The .quantile(...)
method can either take one number (between 0
and 1
) indicating the percentile to return (for example, 0.5
being the median and 0.25
and 0.75
being lower and upper quartiles). However, it can also return an array of values corresponding to the percentiles passed as a list to the method. The .linspace(0,1,11)
command will produce the following array:
[ 0., 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]
So, the .quantile(...)
method will return a list starting with a minimum and followed by all the deciles up to the maximum for the price_mean
column.
- Unreal Engine Physics Essentials
- 數據庫系統教程(第2版)
- Learn Type:Driven Development
- Windows系統管理與服務配置
- Learning RabbitMQ
- 認識編程:以Python語言講透編程的本質
- NativeScript for Angular Mobile Development
- 深入理解Java7:核心技術與最佳實踐
- Securing WebLogic Server 12c
- Python完全自學教程
- Haxe Game Development Essentials
- C程序設計實踐教程
- 0 bug:C/C++商用工程之道
- Spring Boot+MVC實戰指南
- Learning Image Processing with OpenCV