制作包含缺失值的熊猫数据框中唯一行的频率表

Question

我想为包含缺失值的 pandas 数据帧创建一个频率表。这是一个带有缺失值的示例数据框来说明我的问题：

import pandas as pd
import numpy as np
car_names = pd.DataFrame({'name' : ['Batmobile','Toyota Corolla','Bike',
                                     'Bike','Batmobile'],
      'hp': [1000,120,np.nan,np.nan,900]})
car_attr = pd.DataFrame({"name": ["Bike","Toyota Corolla"],
                         "color": ["blue","red"]})
cars = car_names.merge(car_attr,how='left',on='name')

    name            hp      color
0   Batmobile       1000.0  NaN
1   Toyota Corolla  120.0   red
2   Bike            NaN     blue
3   Bike            NaN     blue
4   Batmobile       900.0   NaN

我想要的输出，显示一个值组合出现两次的频率表：

    name            hp      color    count
0   Bike            NaN     blue     2
1   Batmobile       1000.0  NaN      1
2   Toyota Corolla  120.0   red      1
3   Batmobile       900.0   NaN      1

我曾尝试使用 groupby().size() 但此方法排除了缺失值的行（即我的数据框中的每一行，除了第二行）：

cars.groupby(['name','hp','color']).size()

name            hp     color
Toyota Corolla  120.0  red      1

我尝试过的另一种方法是将 pandas 数据框转换为列表列表（其中每一行都是一个列表）并使用 list.index() 函数来计算唯一行的出现次数，但我遇到了这个奇怪的错误：

my_rows = cars.values.tolist()
my_rows_dedup = cars.drop_duplicates().values.tolist()

for x in my_rows:
    print(x)
    print('Row index: ', my_rows.index(x),
    ' Unique Index: ', my_rows_dedup.index(x))

['Batmobile', 1000.0, nan]
Row index:  0  Unique Index:  0
['Toyota Corolla', 120.0, 'red']
Row index:  1  Unique Index:  1
['Bike', nan, 'blue']

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-100-f17351883e95> in <module>
      4 for x in my_rows:
      5     print(x)
----> 6     print('Row index: ', my_rows.index(x), ' Unique Index: ', my_rows_dedup.index(x))

ValueError: ['Bike', nan, 'blue'] is not in list

这个错误对我来说没有意义，因为 ['Bike', nan, 'blue'] 是 my_rows_dedup 中的一个值。

Answer 1

这对某些人来说可能有点矫枉过正，但这是一个符合 DataNovice 答案的功能。我生成一个随机字符串来填充 NA 值并确保随机字符串不在我的数据集中。我还创建了一个百分比列并进行了一些格式化以在 jupyter 笔记本中显示（隐藏索引、格式化逗号和百分比）。

import random
import string
import pandas as pd
import numpy as np
from IPython.display import display

# column names (optional) defined as string or list of strings
# Returns frequency table that includes missing values for df
# n limits the output to the first n rows
def tidy_count(df,columns=None,count_col='n',return_df=False,n=None):
    # subset df by columns if specified
    if columns != None:
        if type(columns) != list:
            columns = [columns]
        df = df[columns]

    col_vals = df.columns.values # all column values

    # Find all distinct values in the dataset for both data values
    # and column names, store in the list allvals
    pds_allvals = pd.Series(col_vals) 
    for col in col_vals:
        pds_allvals = pd.concat([pds_allvals,df[str(col)]]).drop_duplicates()
    allvals = pds_allvals.tolist()
    #print(allvals)

    filler_val=None
    # generate a random string for filler_val until it is 
    # not something that occurs in our data
    loop_iters=1
    while filler_val == None or filler_val in allvals:
        filler_val = ''.join(random.choices(string.digits + string.ascii_letters, k=np.random.randint(4,12)))  
        loop_iters += 1
        if loop_iters > 8:
            print('Error: Cannot find unique string to fill NA values.')
            break
    #print(filler_val)

    # reset count column name until it is unique among column names
    loop_iters=1
    while count_col in col_vals:
        if count_col == 'n':
            count_col = count_col + 'n'
        else:
            count_col = count_col + '_'

        if loop_iters > 8:
            print('Error: Cannot find unique string for counter column.')
            break    
        loop_iters += 1

    freq_table = df.fillna(filler_val).groupby([str(x) for x in col_vals]).size().\
            reset_index(name=count_col).\
         replace(filler_val,np.NaN).sort_values(count_col,ascending=False)

    freq_table['percent'] = freq_table[count_col] / sum(freq_table[count_col])

    # limit frequency table to first n rows if specified
    if n != None:
        freq_table = freq_table.head(n)

    # Formatted pandas display for jupyter notebooks
    df_display = display(freq_table.style.hide_index().\
                format({'n': '{:,}','percent': '{0:.1%}'}))

    if return_df == False:
        df_display
    else:
        df_display
        return(freq_table)

在 seaborn 包中的数据集上使用示例：

## Import a dataset from seaborn and add missing values at random
import seaborn as sns
import numpy as np
planets = sns.load_dataset("planets")
import numpy as np
planets_method = planets.sample(frac=.40,random_state=42)
planets_year = planets.sample(frac=.20,random_state=84)
planets_method['method'] = np.nan
planets_year['year'] = np.nan

planets_combi = pd.concat([planets.\
                    drop(planets_method.index.union(planets_year.index)),
                          planets_method,planets_year]).sort_index()

## Use the function above to create a frequency table
# Note that it is cut off at row 7 (n=7) for display purposes
tidy_count(planets_combi,['method','year'],n=7)

method               year   n   percent
Radial Velocity      nan    104 9.2%
Transit              nan    82  7.3%
nan                  2011   70  6.2%
Transit              2012   56  5.0%
Radial Velocity      2011   52  4.6%
nan                  2013   48  4.3%
nan                  2010   47  4.2%

Answer 2

好吧，如果你想处理空值，你可以先填充它们，然后如果你想将它们转换回 NaN 值，则稍后替换

x = 'g8h.|$hTdo+jC9^@'    
(cars.fillna(x).groupby(['name','hp','color']).size().reset_index()
               .rename(columns={0 : 'count'}).replace(x,np.NaN))
            name         hp     color   count
    0   Batmobile       900.0   NaN     1
    1   Batmobile       1000.0  NaN     1
    2   Bike            NaN     blue    2
    3   Toyota Corolla  120.0   red     1

Answer 3

您可以通过在 groupby 时替换它们来解决 NaN 问题，然后再恢复它们。可能是这样的：

# Fillna, groupby columns and count duplicates
counts = cars.fillna(-1).groupby(cars.columns.tolist()).agg(len)
# Reshape the dataframe
cars_no_dups = pd.DataFrame(counts.reset_index().to_records()).set_index("index")
# Restore duplicates
cars_no_dups[cars_no_dups==-1] = np.nan