利用机器学习测试你的心动指数

利用机器学习测试你的心动指数

数据集下载

[1]:

 

!wget https://pai-public-data.oss-cn-beijing.aliyuncs.com/speed_dating/Speed%20Dating%20Data%20Key.doc
--2020-08-12 11:06:06--  https://pai-public-data.oss-cn-beijing.aliyuncs.com/speed_dating/Speed%20Dating%20Data%20Key.doc
Resolving pai-public-data.oss-cn-beijing.aliyuncs.com (pai-public-data.oss-cn-beijing.aliyuncs.com)... 47.95.85.22
Connecting to pai-public-data.oss-cn-beijing.aliyuncs.com (pai-public-data.oss-cn-beijing.aliyuncs.com)|47.95.85.22|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 161792 (158K) [application/msword]
Saving to: 'Speed Dating Data Key.doc'

100%[======================================>] 161,792     --.-K/s   in 0.07s   

2020-08-12 11:06:06 (2.08 MB/s) - 'Speed Dating Data Key.doc' saved [161792/161792]

[2]:

 

!wget https://pai-public-data.oss-cn-beijing.aliyuncs.com/speed_dating/Speed%20Dating%20Data.csv
--2020-08-12 11:06:07--  https://pai-public-data.oss-cn-beijing.aliyuncs.com/speed_dating/Speed%20Dating%20Data.csv
Resolving pai-public-data.oss-cn-beijing.aliyuncs.com (pai-public-data.oss-cn-beijing.aliyuncs.com)... 47.95.85.22
Connecting to pai-public-data.oss-cn-beijing.aliyuncs.com (pai-public-data.oss-cn-beijing.aliyuncs.com)|47.95.85.22|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5192296 (5.0M) [text/csv]
Saving to: 'Speed Dating Data.csv'

100%[======================================>] 5,192,296   18.7MB/s   in 0.3s   

2020-08-12 11:06:08 (18.7 MB/s) - 'Speed Dating Data.csv' saved [5192296/5192296]

安装以及引入调用的包

[3]:

 

!pip install palettable --user
Collecting palettable
  Downloading https://mirrors.aliyun.com/pypi/packages/ca/46/5198aa24e61bb7eef28d06cb69e56bfa1942f4b6807d95a0b5ce361fe09b/palettable-3.3.0-py2.py3-none-any.whl (111kB)
    100% |################################| 112kB 1.1MB/s 
Installing collected packages: palettable
Successfully installed palettable-3.3.0
You are using pip version 9.0.1, however version 20.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[4]:

 

!pip install imbalanced-learn --user
Collecting imbalanced-learn
  Downloading https://mirrors.aliyun.com/pypi/packages/c8/81/8db4d87b03b998fda7c6f835d807c9ae4e3b141f978597b8d7f31600be15/imbalanced_learn-0.7.0-py3-none-any.whl (167kB)
    100% |################################| 174kB 11.0MB/s 
Requirement already satisfied: scikit-learn>=0.23 in /home/admin/.local/lib/python3.6/site-packages (from imbalanced-learn)
Requirement already satisfied: scipy>=0.19.1 in /opt/conda/lib/python3.6/site-packages (from imbalanced-learn)
Requirement already satisfied: joblib>=0.11 in /home/admin/.local/lib/python3.6/site-packages (from imbalanced-learn)
Requirement already satisfied: numpy>=1.13.3 in /home/admin/.local/lib/python3.6/site-packages (from imbalanced-learn)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/admin/.local/lib/python3.6/site-packages (from scikit-learn>=0.23->imbalanced-learn)
Installing collected packages: imbalanced-learn
Successfully installed imbalanced-learn-0.7.0
You are using pip version 9.0.1, however version 20.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[6]:

 

!pip install seaborn --user
Collecting seaborn
  Downloading https://mirrors.aliyun.com/pypi/packages/c7/e6/54aaaafd0b87f51dfba92ba73da94151aa3bc179e5fe88fc5dfb3038e860/seaborn-0.10.1-py3-none-any.whl (215kB)
    100% |################################| 225kB 13.2MB/s 
Requirement already satisfied: matplotlib>=2.1.2 in /opt/conda/lib/python3.6/site-packages (from seaborn)
Requirement already satisfied: numpy>=1.13.3 in /home/admin/.local/lib/python3.6/site-packages (from seaborn)
Requirement already satisfied: pandas>=0.22.0 in /opt/conda/lib/python3.6/site-packages (from seaborn)
Requirement already satisfied: scipy>=1.0.1 in /opt/conda/lib/python3.6/site-packages (from seaborn)
Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/lib/python3.6/site-packages (from matplotlib>=2.1.2->seaborn)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.6/site-packages (from matplotlib>=2.1.2->seaborn)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.6/site-packages (from matplotlib>=2.1.2->seaborn)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/conda/lib/python3.6/site-packages (from matplotlib>=2.1.2->seaborn)
Requirement already satisfied: pytz>=2011k in /opt/conda/lib/python3.6/site-packages (from pandas>=0.22.0->seaborn)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.6/site-packages (from python-dateutil>=2.1->matplotlib>=2.1.2->seaborn)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from kiwisolver>=1.0.1->matplotlib>=2.1.2->seaborn)
Installing collected packages: seaborn
Successfully installed seaborn-0.10.1
You are using pip version 9.0.1, however version 20.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[8]:

 

# importing packages
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn import metrics
​
import imblearn
from palettable.colorbrewer.qualitative import Pastel1_3
​

EDA 探索性数据分析

[9]:

 

df = pd.read_csv('Speed Dating Data.csv', encoding='gbk')
df.head()

[9]:

  iid id gender idg condtn wave round position positin1 order ... attr3_3 sinc3_3 intel3_3 fun3_3 amb3_3 attr5_3 sinc5_3 intel5_3 fun5_3 amb5_3
0 1 1.0 0 1 1 1 10 7 NaN 4 ... 5.0 7.0 7.0 7.0 7.0 NaN NaN NaN NaN NaN
1 1 1.0 0 1 1 1 10 7 NaN 3 ... 5.0 7.0 7.0 7.0 7.0 NaN NaN NaN NaN NaN
2 1 1.0 0 1 1 1 10 7 NaN 10 ... 5.0 7.0 7.0 7.0 7.0 NaN NaN NaN NaN NaN
3 1 1.0 0 1 1 1 10 7 NaN 5 ... 5.0 7.0 7.0 7.0 7.0 NaN NaN NaN NaN NaN
4 1 1.0 0 1 1 1 10 7 NaN 7 ... 5.0 7.0 7.0 7.0 7.0 NaN NaN NaN NaN NaN

5 rows × 195 columns

[10]:

 

print(df.shape)
(8378, 195)

[11]:

 

# 计算出每个特征有多少百分比是缺失的
percent_missing = df.isnull().sum() * 100 / len(df)
missing_value_df = pd.DataFrame({
    'column_name': df.columns,
    'percent_missing': percent_missing
})

[12]:

 

missing_value_df.sort_values(by='percent_missing', ascending=False).head(10)

[12]:

  column_name percent_missing
num_in_3 num_in_3 92.026737
numdat_3 numdat_3 82.143710
expnum expnum 78.515159
sinc7_2 sinc7_2 76.665075
amb7_2 amb7_2 76.665075
shar7_2 shar7_2 76.438291
attr7_2 attr7_2 76.318931
intel7_2 intel7_2 76.318931
fun7_2 fun7_2 76.318931
amb5_3 amb5_3 75.936978

多少人通过Speed Dating找到了对象

[13]:

 

# 多少人通过Speed Dating找到了对象
plt.subplots(figsize=(3,3), dpi=110,)
# 构造数据
size_of_groups=df.match.value_counts().values
​
single_percentage = round(size_of_groups[0]/sum(size_of_groups) * 100,2) 
matched_percentage = round(size_of_groups[1]/sum(size_of_groups)* 100,2) 
names = [
    'Single:' + str(single_percentage) + '%',
    'Matched' + str(matched_percentage) + '%']
 
# 创建饼图
plt.pie(
    size_of_groups, 
    labels=names, 
    labeldistance=1.2, 
    colors=Pastel1_3.hex_colors
)
plt.show()
​

利用机器学习测试你的心动指数

[ ]:

 

​

[14]:

 

# 多少女生通过Speed Dating找到了对象
plt.subplots(figsize=(3,3), dpi=110,)
# 构造数据
size_of_groups=df[df.gender == 0].match.value_counts().values
​
single_percentage = round(size_of_groups[0]/sum(size_of_groups) * 100,2) 
matched_percentage = round(size_of_groups[1]/sum(size_of_groups)* 100,2) 
names = [
    'Single:' + str(single_percentage) + '%',
    'Matched' + str(matched_percentage) + '%']
 
# 创建饼图
plt.pie(
    size_of_groups, 
    labels=names, 
    labeldistance=1.2, 
    colors=Pastel1_3.hex_colors
)
plt.show()
​

利用机器学习测试你的心动指数

[15]:

 

# 多少男生通过Speed Dating找到了对象
plt.subplots(figsize=(3,3), dpi=110,)
# 构造数据
size_of_groups=df[df.gender == 1].match.value_counts().values
​
single_percentage = round(size_of_groups[0]/sum(size_of_groups) * 100,2) 
matched_percentage = round(size_of_groups[1]/sum(size_of_groups)* 100,2) 
names = [
    'Single:' + str(single_percentage) + '%',
    'Matched' + str(matched_percentage) + '%']
 
# 创建饼图
plt.pie(
    size_of_groups, 
    labels=names, 
    labeldistance=1.2, 
    colors=Pastel1_3.hex_colors
)
plt.show()
​

利用机器学习测试你的心动指数

年龄分布

[16]:

 

# 年龄分布
age = df[np.isfinite(df['age'])]['age']
plt.hist(age,bins=35)
plt.xlabel('Age')
plt.ylabel('Frequency')

[16]:

Text(0, 0.5, 'Frequency')

利用机器学习测试你的心动指数

[17]:

 

date_df = df[[
    'iid', 'gender', 'pid', 'match', 'int_corr', 'samerace', 'age_o',
       'race_o', 'pf_o_att', 'pf_o_sin', 'pf_o_int', 'pf_o_fun', 'pf_o_amb',
       'pf_o_sha', 'dec_o', 'attr_o', 'sinc_o', 'intel_o', 'fun_o', 'like_o',
       'prob_o', 'met_o', 'age', 'race', 'imprace', 'imprelig', 'goal', 'date',
       'go_out', 'career_c', 'sports', 'tvsports', 'exercise', 'dining',
       'museums', 'art', 'hiking', 'gaming', 'clubbing', 'reading', 'tv',
       'theater', 'movies', 'concerts', 'music', 'shopping', 'yoga', 'attr1_1',
       'sinc1_1', 'intel1_1', 'fun1_1', 'amb1_1', 'attr3_1', 'sinc3_1',
       'fun3_1', 'intel3_1', 'dec', 'attr', 'sinc', 'intel', 'fun', 'like',
       'prob', 'met'
]]

[18]:

 

# heatmap
plt.subplots(figsize=(20,15))
ax = plt.axes()
ax.set_title("Correlation Heatmap")
corr = date_df.corr()
sns.heatmap(corr, 
            xticklabels=corr.columns.values,
            yticklabels=corr.columns.values)

[18]:

<matplotlib.axes._subplots.AxesSubplot at 0x7f9662c23f98>

利用机器学习测试你的心动指数

模型构建

数据准备

[19]:

 

# preparing the data
clean_df = df[['attr_o','sinc_o','intel_o','fun_o','amb_o','shar_o','match']]
clean_df.dropna(inplace=True)
X=clean_df[['attr_o','sinc_o','intel_o','fun_o','amb_o','shar_o',]]
y=clean_df['match']
​
/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until

[20]:

 

oversample = imblearn.over_sampling.SVMSMOTE()
X, y = oversample.fit_resample(X, y)

[21]:

 

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)

模型训练

[22]:

 

 
# logistic regression classification model
model = LogisticRegression(C=1, random_state=0)
lrc = model.fit(X_train, y_train)
predict_train_lrc = lrc.predict(X_train)
predict_test_lrc = lrc.predict(X_test)
print('Training Accuracy:', metrics.accuracy_score(y_train, predict_train_lrc))
print('Validation Accuracy:', metrics.accuracy_score(y_test, predict_test_lrc))
Training Accuracy: 0.765040825096691
Validation Accuracy: 0.7555841924398625

测试结果

[24]:

 

lrc.predict_proba([[8.0,6.0,7.0,7.0,6.0,8.0,]])

[24]:

array([[0.29710471, 0.70289529]])

[ ]:

 

​

相关文章：

  • 2021-04-23
  • 2021-10-17
  • 2021-04-24
  • 2021-07-26
  • 2021-04-14
  • 2022-01-02
  • 2021-06-04
  • 2022-12-23
猜你喜欢
  • 2021-04-15
  • 2021-08-19
  • 2021-04-05
  • 2022-12-23
  • 2021-05-12
  • 2021-08-09
相关资源
相似解决方案
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode