官术网_书友最值得收藏!

Dealing with missing data

First, let's look at the missing codes for different languages:

Table 3.7: Missing codes for R, Python, Julia, and Octave

For R, the missing code is NA. Here are several functions we could use to remove those missing observations, shown in an example:

> head(na_example,20) 
[1]  2  1  3  2  1  3  1  4  3  2  2 NA  2  2  1  4 NA  1  1  2 
> length(na_example) 
[1] 1000 
> x<-na.exclude(na_example) 
> length(x) 
[1] 855 
> head(x,20) 
[1] 2 1 3 2 1 3 1 4 3 2 2 2 2 1 4 1 1 2 1 2 

In the previous example, we removed 145 missing values by using the R function called na.exclude(). We could also use the apropos() function to find more functions dealing with missing code in R, as shown here:

 > apropos("^na.") 
 [1] "na.action"              "na.contiguous"          
 [3] "na.exclude"             "na.fail"                
 [5] "na.omit"                "na.pass"                
 [7] "na_example"             "names"                  
 [9] "names.POSIXlt"          "names<-"                
[11] "names<-.POSIXlt"        "namespaceExport"        
[13] "namespaceImport"        "namespaceImportClasses" 
[15] "namespaceImportFrom"    "namespaceImportMethods" 
[17] "napredict"              "naprint"                
[19] "naresid"                "nargs" 
 

For Python, we have the following example, First, let’s generate a dataset called z.csv, see the R code given next. For the program, we generate 100 zeros as our missing values:

set.seed(123)
n=500
x<-rnorm(n)
x2<-x
m=100
y<-as.integer(runif(m)*n)
x[y]<-0
z<-matrix(x,n/5,5)
outFile<-"c:/temp/z.csv"
write.table(z,file=outFile,quote=F,row.names=F,col.names=F,sep=',')

The following Python program checks missing values for 5 columns, replace them with NaN or with the averages of each columns:

import scipy as sp
import pandas as pd
path="https://canisius.edu/~yany/data/"
dataSet="z.csv"
infile=path+dataset
#infile=”c:/temp/z.csv”
x=pd.read_csv(infile,header=None)
print(x.head())
print((x[[1,1,2,3,4,5]] ==0).sum())

The related output is shown here:

At this stage, we just know that for the first five columns, zero represents a missing value. The code of print((x[[1,2,3,4,5]] == 0).sum()) shows the number of zeros for five columns. For instance, there are five zeros for the first column. We could use scipy.NaN to replace those zeros, as shown here:

x2=x
x2[[1,2,3,4,5]] = x2[[1,2,3,4,5]].replace(0, sp.NaN)
print(x2.head())

The output with zeros is replaced with sp.NaN, as shown here:

If we plan to use the mean to replace those NaNs, we have the following code:

x3=x2
x3.fillna(x3.mean(), inplace=True)
print(x3.head())

The output is shown here:

主站蜘蛛池模板: 策勒县| 鄂托克前旗| 阆中市| 石棉县| 仪陇县| 榆树市| 阿图什市| 错那县| 兰考县| 定边县| 积石山| 仙桃市| 北安市| 北海市| 阿城市| 宜城市| 盐池县| 通化市| 阿拉尔市| 灌云县| 梁平县| 泾阳县| 建德市| 天台县| 利川市| 日喀则市| 应用必备| 观塘区| 高尔夫| 宕昌县| 九龙城区| 尉犁县| 大方县| 常宁市| 肥城市| 济阳县| 莱州市| 晋城| 交口县| 邯郸县| 上虞市|