通常，我们拿到一份数据不能马上分析，要先搞清楚这份数据的特点是什么。探索性数据分析就是为了解决这个问题而设计的，下面我们通过一个案例进行展示。本案例数据可在这里获得：https://github.com/DataScienceUB/introduction-datascience-python-book

Python

数据导入

#数据读入

file = open('G:/Py/introduction-datascience-python-book-master/files/ch03/adult.data', 'r')

def chr_int(a):
    if a.isdigit():
        return int(a)
    else:
        return 0

data=[]
for line in file:
     data1=line.split(', ')
     if len(data1)==15:
        data.append([chr_int(data1[0]),data1[1],chr_int(data1[2]),data1[3],chr_int(data1[4]),data1[5],data1[6],\
            data1[7],data1[8],data1[9],chr_int(data1[10]),chr_int(data1[11]),chr_int(data1[12]),data1[13],\
            data1[14]])

print (data[1:2])
[[50, 'Self-emp-not-inc', 83311, 'Bachelors', 13, 'Married-civ-spouse', 'Exec-managerial', 'Husband', 'White', 'Male', 0, 0, 13, 'United-States', '<=50K\n']]

数据转化与审视

#转化为数据框格式

%matplotlib inline
import pandas as pd

df = pd.DataFrame(data) #  Two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes 

df.columns = ['age', 'type_employer', 'fnlwgt', 'education', 
                "education_num","marital", "occupation", "relationship", "race","sex",
                "capital_gain", "capital_loss", "hr_per_week","country","income"]

#观察前几行
df.head()

age type_employer fnlwgt education education_num marital occupation relationship race sex capital_gain capital_loss hr_per_week country income 0 39 State-gov 77516 Bachelors 13 Never-married Adm-clerical Not-in-family White Male 2174 0 40 United-States <=50K\n 1 50 Self-emp-not-inc 83311 Bachelors 13 Married-civ-spouse Exec-managerial Husband White Male 0 0 13 United-States <=50K\n 2 38 Private 215646 HS-grad 9 Divorced Handlers-cleaners Not-in-family White Male 0 0 40 United-States <=50K\n 3 53 Private 234721 11th 7 Married-civ-spouse Handlers-cleaners Husband Black Male 0 0 40 United-States <=50K\n 4 28 Private 338409 Bachelors 13 Married-civ-spouse Prof-specialty Wife Black Female 0 0 40 Cuba <=50K\n

#观察后几行
df.tail()

age type_employer fnlwgt education education_num marital occupation relationship race sex capital_gain capital_loss hr_per_week country income 32556 27 Private 257302 Assoc-acdm 12 Married-civ-spouse Tech-support Wife White Female 0 0 38 United-States <=50K\n 32557 40 Private 154374 HS-grad 9 Married-civ-spouse Machine-op-inspct Husband White Male 0 0 40 United-States >50K\n 32558 58 Private 151910 HS-grad 9 Widowed Adm-clerical Unmarried White Female 0 0 40 United-States <=50K\n 32559 22 Private 201490 HS-grad 9 Never-married Adm-clerical Own-child White Male 0 0 20 United-States <=50K\n 32560 52 Self-emp-inc 287927 HS-grad 9 Married-civ-spouse Exec-managerial Wife White Female 15024 0 40 United-States >50K\n

#查询行列数量
df.shape
(32561, 15)
#对国家进行汇总统计

counts = df.groupby('country').size()

print (counts)
country
?                               583
Cambodia                         19
Canada                          121
China                            75
Columbia                         59
Cuba                             95
Dominican-Republic               70
Ecuador                          28
El-Salvador                     106
England                          90
France                           29
Germany                         137
Greece                           29
Guatemala                        64
Haiti                            44
Holand-Netherlands                1
Honduras                         13
Hong                             20
Hungary                          13
India                           100
Iran                             43
Ireland                          24
Italy                            73
Jamaica                          81
Japan                            62
Laos                             18
Mexico                          643
Nicaragua                        34
Outlying-US(Guam-USVI-etc)       14
Peru                             31
Philippines                     198
Poland                           60
Portugal                         37
Puerto-Rico                     114
Scotland                         12
South                            80
Taiwan                           51
Thailand                         18
Trinadad&Tobago                  19
United-States                 29170
Vietnam                          67
Yugoslavia                       16
dtype: int64

这里大家可以自行尝试对其他变量进行汇总统计

数据分割

#根据性别、收入水平对数据取子集

ml = df[(df.sex == 'Male')]
ml1 = df[(df.sex == 'Male')&(df.income=='>50K\n')]
fm =df[(df.sex == 'Female')]
fm1 =df[(df.sex == 'Female')&(df.income=='>50K\n')]
df1=df[(df.income=='>50K\n')]

这里大家可以尝试看看每个子集的行列数量，并可以试着计算高收入群体的的比例、男/女性群体中高收入群体的占比。

基本统计量展示

#求均值、方差、标准差

ml_mu = ml['age'].mean()
fm_mu = fm['age'].mean()
ml_var = ml['age'].var()
fm_var = fm['age'].var()
ml_std = ml['age'].std()
fm_std = fm['age'].std()
print ('Statistics of age for men: mu:', ml_mu, 'var:', ml_var, 'std:', ml_std)
print ('Statistics of age for women: mu:', fm_mu, 'var:', fm_var, 'std:', fm_std)

#求中位数
ml_median= ml['age'].median()
fm_median= fm['age'].median()
print ("Median age per men and women: ", ml_median, fm_median)

ml_median_age= ml1['age'].median()
fm_median_age= fm1['age'].median()
print ("Median age per men and women with high-income: ", ml_median_age, fm_median_age)
Statistics of age for men: mu: 39.43354749885268 var: 178.77375174529985 std: 13.370630192526448
Statistics of age for women: mu: 36.85823043357163 var: 196.3837063948063 std: 14.013697099438332
Median age per men and women:  38.0 35.0
Median age per men and women with high-income:  44.0 41.0

清除离群点

首先明确什么是离群点，离群点就是离中位数或均值非常遥远的数据点，它们可能是由于观察、操作错误引起的，也有可能是随机事件，也就是极端的情况。如果我们希望发现总体情况的时候，应该清楚这些数据点，否则对我们的结果造成非常大的影响，误导我们的认识。在这个案例中，我们清除掉比中位数大或小3倍标准差的点。

df.shape
(32561, 15)
df2 = df.drop(df.index[(df['hr_per_week'] > (df['hr_per_week'].median() + 3*df['hr_per_week'].std())) &
                       (df['hr_per_week'] < (df['hr_per_week'].median() - 3*df['hr_per_week'].std()))])
df2.shape
(32561, 15)

上面的函数，本质意义是找到中位数大于上界或小于下界的点，然后清楚掉，剩下的就是非离群点。没有清除掉任何离群点。

R

下面来看在R中如何实现相同步骤。

数据导入与审视

#读取

library(tidyverse)

read_csv("G:/Py/introduction-datascience-python-book-master/files/ch03/adult.data",col_names = F) -> df

#设置列名称
names(df) = c('age', 'type_employer', 'fnlwgt', 'education', 
                "education_num","marital", "occupation", "relationship", "race","sex",
                "capital_gain", "capital_loss", "hr_per_week","country","income")

#观察前几行
df %>% head
-- Attaching packages --------------------------------------- tidyverse 1.2.1 --
√ ggplot2 3.0.0     √ purrr   0.2.5
√ tibble  1.4.2     √ dplyr   0.7.6
√ tidyr   0.8.1     √ stringr 1.3.1
√ readr   1.1.1     √ forcats 0.3.0
-- Conflicts ------------------------------------------ tidyverse_conflicts() --
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()
Parsed with column specification:
cols(
  X1 = col_integer(),
  X2 = col_character(),
  X3 = col_integer(),
  X4 = col_character(),
  X5 = col_integer(),
  X6 = col_character(),
  X7 = col_character(),
  X8 = col_character(),
  X9 = col_character(),
  X10 = col_character(),
  X11 = col_integer(),
  X12 = col_integer(),
  X13 = col_integer(),
  X14 = col_character(),
  X15 = col_character()
)

agetype_employerfnlwgteducationeducation_nummaritaloccupationrelationshipracesexcapital_gaincapital_losshr_per_weekcountryincome39 State-gov 77516 Bachelors 13 Never-married Adm-clerical Not-in-family White Male 2174 0 40 United-States <=50K 50 Self-emp-not-inc 83311 Bachelors 13 Married-civ-spouse Exec-managerial Husband White Male 00 13 United-States <=50K 38 Private 215646 HS-grad 9 Divorced Handlers-cleaners Not-in-family White Male 00 40 United-States <=50K 53 Private 234721 11th 7 Married-civ-spouse Handlers-cleaners Husband Black Male 00 40 United-States <=50K 28 Private 338409 Bachelors 13 Married-civ-spouse Prof-specialty Wife Black Female 00 40 Cuba <=50K 37 Private 284582 Masters 14 Married-civ-spouse Exec-managerial Wife White Female 00 40 United-States <=50K

#观察后几行
df %>% tail

agetype_employerfnlwgteducationeducation_nummaritaloccupationrelationshipracesexcapital_gaincapital_losshr_per_weekcountryincome22 Private 310152 Some-college 10 Never-married Protective-serv Not-in-family White Male 00 40 United-States <=50K 27 Private 257302 Assoc-acdm 12 Married-civ-spouse Tech-support Wife White Female 00 38 United-States <=50K 40 Private 154374 HS-grad 9 Married-civ-spouse Machine-op-inspct Husband White Male 00 40 United-States >50K 58 Private 151910 HS-grad 9 Widowed Adm-clerical Unmarried White Female 00 40 United-States <=50K 22 Private 201490 HS-grad 9 Never-married Adm-clerical Own-child White Male 00 20 United-States <=50K 52 Self-emp-inc 287927 HS-grad 9 Married-civ-spouse Exec-managerial Wife White Female 15024 0 40 United-States >50K

df %>% dim

32561
15

df %>% count(country)

countryn? 583 Cambodia 19 Canada 121 China 75 Columbia 59 Cuba 95 Dominican-Republic 70 Ecuador 28 El-Salvador 106 England 90 France 29 Germany 137 Greece 29 Guatemala 64 Haiti 44 Holand-Netherlands 1 Honduras 13 Hong 20 Hungary 13 India 100 Iran 43 Ireland 24 Italy 73 Jamaica 81 Japan 62 Laos 18 Mexico 643 Nicaragua 34 Outlying-US(Guam-USVI-etc) 14 Peru 31 Philippines 198 Poland 60 Portugal 37 Puerto-Rico 114 Scotland 12 South 80 Taiwan 51 Thailand 18 Trinadad&Tobago 19United-States 29170 Vietnam 67 Yugoslavia 16

数据分割

df %>% filter(sex == 'Male') -> ml
df %>% filter(sex == 'Male',income == '>50K\n') -> ml1
df %>% filter(sex == 'Female') -> fml
df %>% filter(sex == 'Female',income == '>50K\n') -> fml1
df %>% filter(income == '>50K\n') -> df1

基本统计量展示

ml %>% pull(age) %>% mean -> ml_mu
ml %>% pull(age) %>% var -> ml_var
ml %>% pull(age) %>% sd -> ml_std
ml %>% pull(age) %>% median -> ml_median

print(c(ml_mu,ml_var,ml_std,ml_median))
[1]  39.43355 178.77375  13.37063  38.00000

清除离群点

df %>%dim

32561
15

df %>% pull(hr_per_week) %>% sd -> hr_sd
df %>% pull(hr_per_week) %>% median -> hr_median

df %>%
 filter(hr_per_week <= (hr_median + 3*hr_median) | 
        hr_per_week >= (hr_median - 3*hr_median))  -> df2

dim(df2)

32561
15

上面的操作本质是筛选在上下界之间的数据点。

分析

在清楚离群点的时候，两个语言用的函数和思路不一样，需要体会一下。值得注意的是，使用dplyr管道的时候要随时注意数据集的改变，因此对中位数和标准差的计算需要在外部进行，然后在最后的函数中直接以常数的形式被利用。

ITPub博客

R与Python手牵手：数据探索性分析案例展示

Python

数据导入

数据转化与审视

数据分割

基本统计量展示

清除离群点

R

数据导入与审视

数据分割

基本统计量展示

清除离群点

分析

最新文章