在 Python 中使用正则表达式将文本拆分为句子 [重复]

Question

我正在尝试将一段示例文本拆分成一个句子列表，每个句子的末尾没有分隔符，也没有空格。

示例文本：

您第一次看到《第二次文艺复兴》时可能会觉得很无聊。至少看两次，一定要看第 2 部分。它会改变你对矩阵的看法。人类是战争的始作俑者吗？人工智能是坏事吗？

进入这个（期望的输出）：

['The first time you see The Second Renaissance it may look boring', 'Look at it at least twice and definitely watch part 2', 'It will change your view of the matrix', 'Are the human people the ones who started the war', 'Is AI a bad thing']

我的代码目前是：

def sent_tokenize(text):
    sentences = re.split(r"[.!?]", text)
    sentences = [sent.strip(" ") for sent in sentences]
    return sentences

但是这个输出（当前输出）：

['The first time you see The Second Renaissance it may look boring', 'Look at it at least twice and definitely watch part 2', 'It will change your view of the matrix', 'Are the human people the ones who started the war', 'Is AI a bad thing', '']

注意末尾多余的 ''。

关于如何删除当前输出末尾多余的 '' 有什么想法吗？

Answer 1

nltk's sent_tokenize

如果您从事 NLP 业务，我强烈推荐 sent_tokenize 来自 nltk 包。

>>> from nltk.tokenize import sent_tokenize
>>> sent_tokenize(text)
[
    'The first time you see The Second Renaissance it may look boring.',
    'Look at it at least twice and definitely watch part 2.',
    'It will change your view of the matrix.',
    'Are the human people the ones who started the war?',
    'Is AI a bad thing?'
] 

它比正则表达式强大得多，并且提供了很多选项来完成工作。更多信息请访问official documentation。

如果您对尾随分隔符很挑剔，您可以使用 nltk.tokenize.RegexpTokenizer 与稍有不同的模式：

>>> from nltk.tokenize import RegexpTokenizer
>>> tokenizer = RegexpTokenizer(r'[^.?!]+')
>>> list(map(str.strip, tokenizer.tokenize(text)))    
[
    'The first time you see The Second Renaissance it may look boring',
    'Look at it at least twice and definitely watch part 2',
    'It will change your view of the matrix',
    'Are the human people the ones who started the war',
    'Is AI a bad thing'
]

---

基于正则表达式的re.split

如果您必须使用regex，那么您需要通过添加否定前瞻来修改您的模式 -

>>> list(map(str.strip, re.split(r"[.!?](?!$)", text)))
[
    'The first time you see The Second Renaissance it may look boring',
    'Look at it at least twice and definitely watch part 2',
    'It will change your view of the matrix',
    'Are the human people the ones who started the war',
    'Is AI a bad thing?'
]

添加的(?!$) 指定仅在尚未到达行尾时才拆分。不幸的是，我不确定是否可以在不执行result[-1] = result[-1][:-1] 之类的操作的情况下合理地删除最后一句的尾随分隔符。

Answer 2

您可以使用过滤器来删除空元素

例如：

import re
text = """The first time you see The Second Renaissance it may look boring. Look at it at least twice and definitely watch part 2. It will change your view of the matrix. Are the human people the ones who started the war? Is AI a bad thing?"""
def sent_tokenize(text):
    sentences = re.split(r"[.!?]", text)
    sentences = [sent.strip(" ") for sent in sentences]
    return filter(None, sentences)

print sent_tokenize(text)

Answer 3

关于如何删除当前结尾处多余的 '' 的任何想法 输出？

你可以这样做删除它：

sentences[:-1]

或更快（由ᴄᴏʟᴅsᴘᴇᴇᴅ）

del result[-1]

输出：

['The first time you see The Second Renaissance it may look boring', 'Look at it at least twice and definitely watch part 2', 'It will change your view of the matrix', 'Are the human people the ones who started the war', 'Is AI a bad thing']

Answer 4

您可以先strip您的段落，然后再拆分它，或者过滤掉结果中的空字符串。