Python - 在使用 FOR 循环遍历 pandas DataFrame 时使用 IF 语句在字符串中查找子字符串

Question

我有一个看起来像这样的 DataFrame...

                                     Variable
0                         Religion - Buddhism
1                            Source: Clickerz
2                            Religion - Islam
3                            Source: SRZ FREE
4   Ethnicity - Mixed - White & Black African

我想操纵variable列来创建一个看起来像这样的new column...

                                        Variable           New Column
    0                         Religion - Buddhism           Buddhism
    1                            Source: Clickerz           Clickerz
    2                            Religion - Islam            Islam
    3                            Source: SRZ FREE            SRZ FREE
    4   Ethnicity - Mixed - White & Black African         Mixed - White and Black African

这样我最终可以拥有一个看起来像这样的 DataFrame...

                            Variable                      New Column
    0                       Religion                        Buddhism
    1                         Source                        Clickerz
    2                       Religion                           Islam
    3                         Source                        SRZ FREE
    4                      Ethnicity         Mixed - White and Black African

我想遍历Variable 列并操作数据以创建New Column。我计划使用多个 if 语句来查找特定单词，例如 'Ethnicity' 或 'Religion'，然后应用操作。

例如...

For row in df['Variable']:

      if 'Religion' in row:

              df['New Column'] = ...
      
      elif 'Ethnicity' in row:

              df['New Column'] = ...

      elif: 'Source' in row:

              df['New Column'] = ...

      else:

              df['New Column'] = 'Not Applicable'

尽管type(row) 返回'str' 表示它属于类字符串，但此代码仍将新列全部返回为“不适用”，这意味着它未检测到数据中任何行中的任何字符串即使我可以看到它们在那里，也可以框架。

我确信有一种简单的方法可以做到这一点...请帮助！

我也尝试了以下...

For row in df['Variable']:

  if row.find('Religion') != -1:

          df['New Column'] = ...

  elif row.find('Ethnicity') != -1:

          df['New Column'] = ...

  elif: row.find('Source') != -1:

          df['New Column'] = ...

  else:

          df['New Column'] = 'Not Applicable'

我继续将新列的所有条目都设为“不适用”。再次在现有列中找不到字符串。

是数据类型的问题还是什么？

Answer 1

您可以使用嵌套的for 循环：

# For each row in the dataframe
for row in df['column_variable']:
    # Set boolean to indicate if a substring was found
    substr_found = False

    # For each substring
    for sub_str in ["substring1", "substring2"]:
        # If the substring is in the row
        if sub_str in row:
            # Execute code...
            df['new_column'] = ...

            # Substring was found!
            substr_found = True

    # If substring was not found
    if not substr_found:
        # Set invalid code...
        df['new column'] = 'Not Applicable'

Answer 2

在操作DataFrame 时，应尽可能避免遍历行。这个article 解释了哪些是更有效的替代方案。

您基本上是在尝试根据某些固定映射来翻译字符串。自然会想到dict：

substring_map = {
    "at": "pseudo-cat",
    "dog": "true dog",
    "bre": "something else",    
    "na": "not applicable"
}

在处理大量子字符串的场景中，可以从文件（例如 JSON 文件）中读取此映射。

现在可以将子字符串匹配逻辑与映射定义解耦：

def translate_substring(x):
  for substring, new_string in substring_map.items():
    if substring in x:
      return new_string
  return "not applicable"

使用apply 和'mapping'函数来生成你的目标列：

df = pd.DataFrame({"name":
  ["cat", "dogg", "breeze", "bred", "hat", "misty"]})

df["new_column"] = df["name"].apply(translate_substring)

# df:
#      name      new_column
# 0     cat      pseudo-cat
# 1    dogg        true dog
# 2  breeze  something else
# 3    bred  something else
# 4     hat      pseudo-cat
# 5   misty  not applicable

此代码应用于pd.concat([df] * 10000)（60,000 行），在 Colab 笔记本中运行时间为 42 毫秒。相比之下，使用iterrows 在 3.67 秒内完成——加速了 87 倍。

Answer 3

您可以创建一个空列表，在其中添加新值，然后在最后一步创建新列：

all_data = []
for row in df["column_variable"]:
    if "substring1" in row:
        all_data.append("Found 1")
    elif "substring2" in row:
        all_data.append("Found 2")
    elif "substring3" in row:
        all_data.append("Found 3")
    else:
        all_data.append("Not Applicable")

df["new column"] = all_data

print(df)

打印：

      column_variable new column
0  this is substring1    Found 1
1  this is substring2    Found 2
2  this is substring1    Found 1
3  this is substring3    Found 3

Answer 4

也许是我能想到的最短路径：

#Dummy DataFrame
df = pd.DataFrame([[1,"substr1"],[3,"bla"],[5,"bla"]],columns=["abc","col_to_check"])

substrings = ["substr1","substr2", "substr3"]
content = df["col_to_check"].unique().tolist() # Unique content of column

for subs in substrings: # Go through all your substrings
    if subs in content: # Check if substring is in column
        df[subs] = 0 # Fill your new column with whatever you want

Answer 5

已更新以匹配您的数据框！

import pandas as pd

您的数据框

lst = []

for i in ['Religion - Buddhism','Source: Clickerz','Religion - Islam','Source: SRZ FREE','Ethnicity - Mixed - White & Black African']:
    item = [i]
    lst.append(item)

df = pd.DataFrame.from_records(lst)
df.columns = ['variable']
print(df)

                                    variable
0                        Religion - Buddhism
1                           Source: Clickerz
2                           Religion - Islam
3                           Source: SRZ FREE
4  Ethnicity - Mixed - White & Black African

结合使用 For 循环和部分字符串匹配 .loc 来设置新值

for x,y in df['variable'].iteritems():
    if 'religion' in y.lower():
        z = y.split('-')
        df.loc[x, 'variable'] = z[0].strip()
        df.loc[x, 'value'] = ''.join(z[1:]).strip()
    if 'source' in y.lower():
        z = y.split(':')
        df.loc[x, 'variable'] = z[0].strip()
        df.loc[x, 'value'] = ''.join(z[1:]).strip()
    if 'ethnicity' in y.lower():
        z = y.split('-')
        df.loc[x, 'variable'] = z[0].strip()
        df.loc[x, 'value'] = ''.join(z[1:]).strip()

print(df)

    variable                         value
0   Religion                      Buddhism
1     Source                      Clickerz
2   Religion                         Islam
3     Source                      SRZ FREE
4  Ethnicity  Mixed  White & Black African

Answer 6

我做了一个函数'string_splitter'并将它应用到一个lambda函数中，这解决了这个问题。

我创建了以下函数，用于根据单元格中包含的不同子字符串以不同方式拆分字符串。

def string_splitter(cell):

word_list1 = ['Age', 'Disability', 'Religion', 'Gender']
word_list2 = ['Number shortlisted', 'Number Hired', 'Number Interviewed']

if any([word in cell for word in word_list1]):
    
    result = cell.split("-")[1]
    result = result.strip()
    
elif 'Source' in cell:
    
    result = cell.split(":")[1]
    result = result.strip()
    
elif 'Ethnicity' in cell:
    
    result_list = cell.split("-")[1:3]
    result = "-".join(result_list)
    result = result.strip()

elif any([word in cell for word in word_list2]):
    
    result = cell.split(" ")[1]
    result = result.strip()

elif 'Number of Applicants' in cell:
    
    result = cell


return result

然后我在使用 lambda 操作时调用了string_splitter。当代码遍历数据框中指定列的每一行时，这会将函数单独应用于每个单元格。如下图：

df['Answer'] = df['Visual Type'].apply(lambda x: string_splitter(x))

string_splitter 允许我创建 New column。

然后，我创建了另一个函数 column_formatter 来在创建 New Column 后操作 Variable 列。第二个函数如下图：

def column_formatter(cell):

word_list1 = ['Age', 'Gender', 'Ethnicity', 'Religion']
word_list2 = ['Number of Applicants', 'Number Hired', 'Number shortlisted', 'Number Interviewed']

if any([word in cell for word in word_list1]):
    
    result = cell.split("-")[0]
    result = result.strip()

elif 'Source' in cell:
    
    result = cell.split(":")[0]
    result = result.strip()

elif 'Disability' in cell:
    
    result = cell.split(" ")[0]
    result = result.strip()

elif any([word in cell for word in word_list2]):
    
    result = 'Number of Applicants'
    
else:
    
    result = 'Something wrong here'


return result

然后以同样的方式调用函数，如下：

df['Visual Type'] = df['Visual Type'].apply(lambda x: column_formatter(x))