字符串到单词的元组

Question

我将单词定义为一系列字符（从 a 到 Z），其中可能还包含撇号。我希望将一个句子拆分成单词，去掉单词中的撇号。

我目前正在执行以下操作以从一段文本中获取单词。

import re
text = "Don't ' thread \r\n on \nme ''\n "
words_iter = re.finditer(r'(\w|\')+', text)
words = (word.group(0).lower() for word in words_iter)
for i in words:
    print(i)

这给了我：

don't
'
thread
on
me
''

但我不想要的是：

dont
thread
on
me

如何更改我的代码以实现此目的？

请注意，我的输出中没有'。

我也希望words 成为生成器。

Answer 1

这看起来像是正则表达式的工作。

import re

text = "Don't ' thread \r\n on \nme ''\n "

# Define a function so as to make a generator
def get_words(text):

    # Find each block, separated by spaces
    for section in re.finditer("[^\s]+", text):

        # Get the text from the selection, lowercase it
        # (`.lower()` for Python 2 or if you hate people who use Unicode)
        section = section.group().casefold()

        # Filter so only letters are kept and yield
        section = "".join(char for char in section if char.isalpha())
        if section:
            yield section

list(get_words(text))
#>>> ['dont', 'thread', 'on', 'me']

正则表达式的解释：

[^    # An "inverse set" of characters, matches anything that isn't in the set
\s    # Any whitespace character
]+    # One or more times

所以这匹配任何非空白字符块。

Answer 2

words = (x.replace("'", '') for x in text.split())
result = tuple(x for x in words if x)

...只对拆分数据进行一次迭代。

如果数据集很大，使用re.finditer而不是str.split()，避免将整个数据集读入内存：

words = (x.replace("'", '') for x in re.finditer(r'[^\s]+', text))
result = tuple(x for x in words if x)

...虽然，tuple()-ing 数据无论如何都会读取内存中的所有内容。

Answer 3

import string
tuple(str(filter(lambda x: x if x in string.letters + string.whitespace else '', "strings don't have '")).split())

Answer 4

使用str.translate 和re.finditer：

>>> text = "Don't ' thread \r\n on \nme ''\n "
>>> import re
>>> from string import punctuation
>>> tab = dict.fromkeys(map(ord, punctuation))
def solve(text):
    for m in re.finditer(r'\b(\S+)\b', text):
        x = m.group(1).translate(tab).lower()
        if x : yield x
>>> list(solve(text))
['dont', 'thread', 'on', 'me']

时序对比：

>>> strs = text * 1000
>>> %timeit list(solve(strs))
10 loops, best of 3: 11.1 ms per loop
>>> %timeit list(get_words(strs))
10 loops, best of 3: 36.7 ms per loop
>>> strs = text * 10000
>>> %timeit list(solve(strs))
1 loops, best of 3: 146 ms per loop
>>> %timeit list(get_words(strs))
1 loops, best of 3: 411 ms per loop