使用 numpy/pandas、矩阵计算从头开始计算卡方统计量答案

【问题标题】：Computing a chi square statistic from scratch using numpy/pandas, matrix computations使用 numpy/pandas、矩阵计算从头开始计算卡方统计量
【发布时间】：2020-09-19 22:43:52
【问题描述】：

我只是在查看 https://en.wikipedia.org/wiki/Chi-squared_test 并想重新创建示例“Example chi-squared test for categorical data”。

我觉得我所采取的方法可能还有改进的余地，所以想知道如何做到这一点。

代码如下：

csv = """\
,A,B,C,D
White collar,90,60,104,95
Blue collar,30,50,51,20
No collar,30,40,45,35
"""
observed_workers = pd.read_csv(io.StringIO(csv), index_col=0)

col_sums = dt.apply(sum)
row_sums = dt.apply(sum, axis=1)

l = list(x[1] * (x[0] / col_sums.sum()) for x in itertools.product(row_sums, col_sums))

expected_workers = pd.DataFrame(
    np.array(l).reshape((3, 4)),
    columns=observed_workers.columns,
    index=observed_workers.index,
)

chi_squared_stat = (
    ((observed_workers - expected_workers) ** 2).div(expected_workers).sum().sum()
)

这会返回正确的值，但可能不知道使用某些特定 numpy / pandas 方法的更好方法。

【问题讨论】：

标签： python pandas numpy linear-algebra statistical-test

【解决方案1】：

使用 numpy/scipy：

csv = """\
,A,B,C,D
White collar,90,60,104,95
Blue collar,30,50,51,20
No collar,30,40,45,35
"""

import io
from numpy import genfromtxt, outer
from scipy.stats.contingency import margins

observed = genfromtxt(io.StringIO(csv), delimiter=',', skip_header=True, usecols=range(1, 5))
row_sums, col_sums = margins(observed)
expected = outer(row_sums, col_sums) / observed.sum()
chi_squared_stat = ((observed - expected)**2 / expected).sum()

print(chi_squared_stat)

使用熊猫：

import io
import pandas as pd

csv = """\
work_group,A,B,C,D
White collar,90,60,104,95
Blue collar,30,50,51,20
No collar,30,40,45,35
"""
df = pd.read_csv(io.StringIO(csv))

df_melt = df.melt(id_vars ='work_group', var_name='group', value_name='observed')
df_melt['col_sum'] = df_melt.groupby('group')['observed'].transform(np.sum)
df_melt['row_sum'] = df_melt.groupby('work_group')['observed'].transform(np.sum)
total = df_melt['observed'].sum()

df_melt['expected'] = df_melt.apply(lambda row: row['col_sum']*row['row_sum']/total, axis=1)
chi_squared_stat = df_melt.apply(lambda row: ((row['observed'] - row['expected'])**2) / row['expected'], axis=1).sum()

print(chi_squared_stat)

【讨论】：