带有正则表达式解析的新 Pandas 列

Question

我正在尝试根据另一列字段中的某些标签和值解析 Pandas DataFrame 中的文本数据，并将它们存储在自己的列中。例如，如果我创建了这个数据框，df:

df = pd.DataFrame([[1,2],['A: this is a value B: this is the b val C: and here is c.','A: and heres another a. C: and another c']])
df = df.T
df.columns = ['col1','col2']


df['tags'] = df['col2'].apply(lambda x: re.findall('(?:\s|)(\w*)(?::)',x))
all_tags = []

for val in df['tags']:
    all_tags = all_tags + val
all_tags = list(set(all_tags))
for val in all_tags:
    df[val] = ''

df:
  col1                                               col2       tags A C B
0    1  A: this is a value B: this is the b val C: and...  [A, B, C]      
1    2           A: and heres another a. C: and another c     [A, C]

如何使用 col2 中的值填充每个新的“标签”列，以便得到这个 df：

col1                                               col2           tags  \
0    1  A: this is a value B: this is the b val C: and...  [A, B, C]   
1    2           A: and heres another a. C: and another c     [A, C]   

                  A               C                  B  
0       this is a value  and here is c.  this is the b val  
1  and heres another a.   and another c 

Answer 1

另一个使用 str.extractall 和 regex 的选项 (?P<key>\w+):(?P<val>[^:]*)(?=\w+:|$):

regex 将分号前的键 (?P<key>\w+) 和分号后的值 (?P<val>[^:]*) 捕获为两个单独的列 key 和 val，val 将匹配非: 个字符，直到它到达受前瞻语法限制的下一个键值对 (?=\w+:|$);这假设密钥始终是一个单词，否则会模棱两可：

import re
pat = re.compile("(?P<key>\w+):(?P<val>[^:]*)(?=\w+:|$)")

pd.concat([
    df,
    (
        df.col2.str.extractall(pat)
          .reset_index('match', drop=True)
          .set_index('key', append=True)
          .val.unstack('key')
    )
], axis=1).fillna('')

---

str.extractall 给出的位置：

df.col2.str.extractall(pat)

然后您旋转结果并与原始数据框连接。

Answer 2

这是一种方法

In [683]: (df.col2.str.findall('[\S]+(?:\s(?!\S+:)\S+)+')
             .apply(lambda x: pd.Series(dict([v.split(':', 1) for v in x])))
          )
Out[683]:
                       A                   B                C
0        this is a value   this is the b val   and here is c.
1   and heres another a.                 NaN    and another c

您可以使用join 追加结果

In [690]: df.join(df.col2.str.findall('[\S]+(?:\s(?!\S+:)\S+)+')
                    .apply(lambda x: pd.Series(dict([v.split(':', 1) for v in x]))))
Out[690]:
  col1                                               col2       tags  \
0    1  A: this is a value B: this is the b val C: and...  [A, B, C]
1    2           A: and heres another a. C: and another c     [A, C]

                       A                   B                C
0        this is a value   this is the b val   and here is c.
1   and heres another a.                 NaN    and another c

事实上，你可以使用字符串方法得到df['tags']

In [688]: df.col2.str.findall('(?:\s|)(\w*)(?::)')
Out[688]:
0    [A, B, C]
1       [A, C]
Name: col2, dtype: object

详情：

将组拆分为列表

In [684]: df.col2.str.findall('[\S]+(?:\s(?!\S+:)\S+)+')
Out[684]:
0    [A: this is a value, B: this is the b val, C: ...
1          [A: and heres another a., C: and another c]
Name: col2, dtype: object

现在，到列表的键值对。

In [685]: (df.col2.str.findall('[\S]+(?:\s(?!\S+:)\S+)+')
             .apply(lambda x: [v.split(':', 1) for v in x]))
Out[685]:
0    [[A,  this is a value], [B,  this is the b val...
1    [[A,  and heres another a.], [C,  and another c]]
Name: col2, dtype: object