Pandas 数据框根据条件更改列中的值答案

【问题标题】：Pandas dataframe change values in a column based on conditionsPandas 数据框根据条件更改列中的值
【发布时间】：2020-12-01 06:26:33
【问题描述】：

我在下面有一个大数据框：

此处用作示例的数据“education_val.csv”可在此处找到https://github.com/ENLK/Py-Projects-/blob/master/education_val.csv

import pandas as pd 

edu = pd.read_csv('education_val.csv')
del edu['Unnamed: 0']
edu.head(10)

ID  Year    Education
22445   1991    higher education
29925   1991    No qualifications
76165   1991    No qualifications
223725  1991    Other
280165  1991    intermediate qualifications
333205  1991    No qualifications
387605  1991    higher education
541285  1991    No qualifications
541965  1991    No qualifications
599765  1991    No qualifications

Education 列中的值为：

edu.Education.value_counts()

intermediate qualifications 153705
higher education    67020
No qualifications   55842
Other   36915

我想通过以下方式替换 Education 列中的值：

如果ID 在Education 列中的某一年中具有值higher education，则该ID 的所有未来年份在Education 列中也将具有higher education。
如果 ID 在一年中具有值 intermediate qualifications，则该 ID 的所有未来年份将在相应的 Education 列中具有 intermediate qualifications。但是，如果值 higher education 出现在此 ID 的任何后续年份，则 higher education 在后续年份替换 intermediate qualifications，无论是 Other 还是 No qualifications occur。

例如在下面的DataFrame中，ID22445在1991的年份有higher education的值，22445的所有后续值Education在以后的年份应该替换为higher education，到今年2017.

edu.loc[edu['ID'] == 22445]

ID  Year    Education
22445   1991    higher education
22445   1992    higher education
22445   1993    higher education
22445   1994    higher education
22445   1995    higher education
22445   1996    intermediate qualifications
22445   1997    intermediate qualifications
22445   1998    Other
22445   1999    No qualifications
22445   2000    intermediate qualifications
22445   2001    intermediate qualifications
22445   2002    intermediate qualifications
22445   2003    intermediate qualifications
22445   2004    intermediate qualifications
22445   2005    intermediate qualifications
22445   2006    intermediate qualifications
22445   2007    intermediate qualifications
22445   2008    intermediate qualifications
22445   2010    intermediate qualifications
22445   2011    intermediate qualifications
22445   2012    intermediate qualifications
22445   2013    intermediate qualifications
22445   2014    intermediate qualifications
22445   2015    intermediate qualifications
22445   2016    intermediate qualifications
22445   2017    intermediate qualifications

同样，下面数据框中的ID 1587125 在1991 年的值intermediate qualifications，并在1993 中更改为higher education。 1587125 在未来几年（从 1993 年起）Education 列中的所有后续值都应为 higher education。

edu.loc[edu['ID'] == 1587125]

ID  Year    Education
1587125 1991    intermediate qualifications
1587125 1992    intermediate qualifications
1587125 1993    higher education
1587125 1994    higher education
1587125 1995    higher education
1587125 1996    higher education
1587125 1997    higher education
1587125 1998    higher education
1587125 1999    higher education
1587125 2000    higher education
1587125 2001    higher education
1587125 2002    higher education
1587125 2003    higher education
1587125 2004    Other
1587125 2005    No qualifications
1587125 2006    intermediate qualifications
1587125 2007    intermediate qualifications
1587125 2008    intermediate qualifications
1587125 2010    intermediate qualifications
1587125 2011    higher education
1587125 2012    higher education
1587125 2013    higher education
1587125 2014    higher education
1587125 2015    higher education
1587125 2016    higher education
1587125 2017    higher education

数据中有 12,057 个唯一的ID，Year 列从 1991 年到 2017 年。如何根据上述条件更改所有 12、057 的 Education 的值？我不确定如何以统一的方式为所有独特的IDs 执行此操作。此处用作示例的示例数据附在上面的 Github 链接中。非常感谢。

【问题讨论】：

标签： python pandas panel-data

【解决方案1】：

您可以像这样使用categorical data 来做到这一点：

df = pd.read_csv('https://raw.githubusercontent.com/ENLK/Py-Projects-/master/education_val.csv')

eddtype = pd.CategoricalDtype(['No qualifications', 
                               'Other',
                               'intermediate qualifications',
                               'higher education'], 
                               ordered=True)
df['EducationCat'] = df['Education'].str.strip().astype(eddtype)

df['EduMax'] = df.sort_values('Year').groupby('ID')['EducationCat']\
                 .transform(lambda x: eddtype.categories[x.cat.codes.cummax()] )

它被明确地分解，所以你可以看到我正在使用的数据操作。

创建教育categorical dtype with order
接下来，更改 Education 列的 dtype 以使用该分类 dtype (EducationCat)
使用分类代码进行 cummax 计算
通过索引返回由 cummax 计算 (EduMax) 定义的类别

输出：

df[df['ID'] == 1587125]

            ID  Year                    Education                 EducationCat                       EduMax
18      1587125  1991  intermediate qualifications  intermediate qualifications  intermediate qualifications
12075   1587125  1992  intermediate qualifications  intermediate qualifications  intermediate qualifications
24132   1587125  1993             higher education             higher education             higher education
36189   1587125  1994             higher education             higher education             higher education
48246   1587125  1995             higher education             higher education             higher education
60303   1587125  1996             higher education             higher education             higher education
72360   1587125  1997             higher education             higher education             higher education
84417   1587125  1998             higher education             higher education             higher education
96474   1587125  1999             higher education             higher education             higher education
108531  1587125  2000             higher education             higher education             higher education
120588  1587125  2001             higher education             higher education             higher education
132645  1587125  2002             higher education             higher education             higher education
144702  1587125  2003             higher education             higher education             higher education
156759  1587125  2004                        Other                        Other             higher education
168816  1587125  2005            No qualifications            No qualifications             higher education
180873  1587125  2006  intermediate qualifications  intermediate qualifications             higher education
192930  1587125  2007  intermediate qualifications  intermediate qualifications             higher education
204987  1587125  2008  intermediate qualifications  intermediate qualifications             higher education
217044  1587125  2010  intermediate qualifications  intermediate qualifications             higher education
229101  1587125  2011             higher education             higher education             higher education
241158  1587125  2012             higher education             higher education             higher education
253215  1587125  2013             higher education             higher education             higher education
265272  1587125  2014             higher education             higher education             higher education
277329  1587125  2015             higher education             higher education             higher education
289386  1587125  2016             higher education             higher education             higher education
301443  1587125  2017             higher education             higher education             higher education

【讨论】：

【解决方案2】：

显然，教育水平是有顺序的。您的问题可以重述为“滚动最大值”问题：一个人在某一年的最高教育水平是多少？

试试这个：

# A dictionary mapping each label to a rank
mappings = {e: i for i, e in enumerate(['No qualifications', 'Other', 'intermediate qualifications', 'higher education'])}

# Convert the label to its rank
edu['Education'] = edu['Education'].map(mappings)

# The gist of the solution: an expanding max level of education per person
tmp = edu.sort_values('Year').groupby('ID')['Education'].expanding().max()

# The first index level in tmp is the ID, the second level is the original index
# We only need the original index, hence the droplevel
# We also convert the rank back to the label (swapping keys and values in the mappings dictionary)
tmp = tmp.droplevel(0).map({v: k for k, v in mappings.items()})

edu['Education'] = tmp

测试：

edu[edu['ID'] == 1587125]

    ID  Year                    Education
1587125  1991  intermediate qualifications
1587125  1992  intermediate qualifications
1587125  1993             higher education
1587125  1994             higher education
1587125  1995             higher education
1587125  1996             higher education
1587125  1997             higher education
1587125  1998             higher education
1587125  1999             higher education
1587125  2000             higher education
1587125  2001             higher education
1587125  2002             higher education
1587125  2003             higher education
1587125  2004             higher education
1587125  2005             higher education
1587125  2006             higher education
1587125  2007             higher education
1587125  2008             higher education
1587125  2010             higher education
1587125  2011             higher education
1587125  2012             higher education
1587125  2013             higher education
1587125  2014             higher education
1587125  2015             higher education
1587125  2016             higher education
1587125  2017             higher education

【讨论】：

我认为你的可能更快。可能需要运行一些时间才能看到。
快跑吧，它实际上比你的慢一点。 lambda 内部转换非常酷。没想到
感谢您运行这些时间，感谢您的评论。只是为了满足我充分利用 pandas 库的优势。有时不利于使用 numpy 或本机 python 方法。 :)

【解决方案3】：

您可以遍历 ID，然后遍历这些年。 DataFrame 是按时间顺序排列的，因此如果一个人在单元格中具有“高等教育”或“中级资格”，您可以保存这些知识并将其应用于后续单元格：

edu = edu.set_index('ID')
ids = edu.index.unique()

for id in ids:
    # booleans to keep track of education statuses we've seen
    higher_ed = False
    inter_qual = False

    rows = edu.loc[id]
    for _, row in rows:
        # check for intermediate qualifications
        if inter_qual:
            row['Education'] = 'intermediate qualifications'
        elif row['Education'] = 'intermediate qualifications':
            inter_qual = True

        # check for higher education
        if higher_ed:
            row['Education'] = 'higher education'
        elif row['Education'] = 'higher education':
            higher_ed = True

我们可能不止一次地覆盖每个状态并不重要 - 如果一个人同时具有“中级资格”和“高等教育”，我们只需确保最后设置“高等教育”即可。

我通常不建议使用 for 循环来处理 DataFrame - 但每个单元格值可能依赖于它上面的值，并且 Dataframe 不会大到不可行。

【讨论】：