python - 如何根据python中的空格将文本文件拆分为多个列表？

Question

嗨，我是 python 编程的新手，请帮助我创建一个函数，该函数将文本文件作为参数并创建一个单词列表，从而删除所有标点符号和列表“拆分”双空格。我的意思是该列表应该在文本文件中的每个双空格出现时创建存在。

这是我的功能：

def tokenize(document):
    file = open("document.txt","r+").read()
    print re.findall(r'\w+', file)

输入文本文件有如下字符串：

What's did the little boy tell the game warden?     His dad was in the kitchen poaching eggs!

注意：守望者后有双倍间距？在他之前

我的函数给了我这样的输出

['what','s','did','the','little','boy','tell','the','game','warden','His','dad','was','in','the','kitchen','poaching','eggs']

期望的输出：

[['what','s','did','the','little','boy','tell','the','game','warden'],
['His','dad','was','in','the','kitchen','poaching','eggs']]

Answer 1

首先split将整个文本放在双空格上，然后将每个项目传递给regex：

>>> file = "What's did the little boy tell the game warden?  His dad was in the kitchen poaching eggs!"
>>> file = text.split('  ')
>>> file
["What's did the little boy tell the game warden?", 'His dad was in the kitchen poaching eggs!']
>>> res = []
>>> for sen in file:
...    res.append(re.findall(r'\w+', sen))
... 
>>> res
[['What', 's', 'did', 'the', 'little', 'boy', 'tell', 'the', 'game', 'warden'], ['His', 'dad', 'was', 'in', 'the', 'kitchen', 'poaching', 'eggs']]

Answer 2

这是一个合理的全 RE 方法：

def tokenize(document):
    with open("document.txt") as f:
        text = f.read()
    blocks = re.split(r'\s\s+', text)
    return [re.findall(r'\w+', b) for b in blocks]

Answer 3

内置拆分功能允许拆分多个空格。

这个：

a = "hello world.  How are you"
b = a.split('  ')
c = [ x.split(' ') for x in b ]

产量：

c = [['hello', 'world.'], ['how', 'are', 'you?']]

如果您也想删除标点符号，请将正则表达式应用于“b”中的元素或第三条语句中的“x”。

Answer 4

首先用标点符号分割文件，然后在第二遍用空格分割结果字符串。

def splitByPunct(s):
    return (x.group(0) for x in  re.finditer(r'[^\.\,\?\!]+', s) if x and x.group(0))

[x.split() for x in splitByPunct("some string, another   string! The phrase")]

这会产生

[['some', 'string'], ['another', 'string'], ['The', 'phrase']]