跨列对 pandas 数据框应用多项质量检查

Question

我有一个如下所示的数据框

df = pd.DataFrame({'subject_id': [101,102,103,201,202],
                  'test_id':['A1:123,A25668','B1:TEST,B2456,B3#123','B3:456','B3:678,C1:345','C2:367,C3']})

目前，我想验证数据框中的选定列是否符合质量标准

a) 列不应包含重复项

b) 列不应包含缺失值

c) 列应仅包含数字/整数值。应该没有字符串/浮点值

所以，我尝试了以下质量检查

q_1 = np.where(df['subject_id'].isna(),"No","Yes")
q_2 = np.where(df['subject_id'].duplicated(keep=False),"No","Yes")
q_3 = np.where(df['subject_id'].str.isdigit(),"Yes", "No") #but this throws error due int64.
q_4 = np.where(df['test_id'].isna(),"No","Yes")
q_5 = np.where(df['test_id'].duplicated(keep=False),"No","Yes")
q_6 = np.where(df['test_id'].str.isdigit(),"Yes", "No") #but this throws error due int64.

如何在数据框中的不同列中优雅高效地执行此操作？你可以看到我为不同的列多次重复同一行代码

如何验证 subject_id 是否仅包含整数而不包含字符串/浮点数？我想逐行检查。

Answer 1

我们可以遍历给定列表中的列名，然后为每一列检查给定条件并创建相应的标志列

cols = ['subject_id', 'test_id']

for c in cols:
    df[c + '_missing'] = df[c].isna()
    df[c + '_duplicated'] = df[c].duplicated(keep=False)
    df[c + '_numeric'] = pd.to_numeric(df[c], errors='coerce') % 1 == 0

---

print(df)

   subject_id               test_id  subject_id_missing  subject_id_duplicated  subject_id_numeric  test_id_missing  test_id_duplicated  test_id_numeric
0         101         A1:123,A25668               False                  False                True            False               False             True
1         102  B1:TEST,B2456,B3#123               False                  False                True            False               False             True
2         103                B3:456               False                  False                True            False               False             True
3         201         B3:678,C1:345               False                  False                True            False               False             True
4         202             C2:367,C3               False                  False                True            False               False             True