【问题标题】:How to split text into sentences by including corner cases如何通过包含极端情况将文本拆分为句子
【发布时间】:2021-08-06 05:32:29
【问题描述】:

我正在使用此链接将文本拆分为句子:

How can I split a text into sentences?.

代码如下:

%%time
import re
alphabets= "([A-Za-z])"
prefixes = "(Mr|St|Mrs|Ms|Dr)[.]"
suffixes = "(Inc|Ltd|Jr|Sr|Co)"
starters = "(Mr|Mrs|Ms|Dr|He\s|She\s|It\s|They\s|Their\s|Our\s|We\s|But\s|However\s|That\s|This\s|Wherever)"
acronyms = "([A-Z][.][A-Z][.](?:[A-Z][.])?)"
websites = "[.](com|net|org|io|gov|ly)"
digits = "([0-9])"


def split_into_sentences(text):
    text = " " + text + "  "
    text = text.replace("\n"," ")
    text = re.sub(prefixes,"\\1<prd>",text)
    text = re.sub(websites,"<prd>\\1",text)
    text = re.sub(digits + "[.]" + digits,"\\1<prd>\\2",text)
    
    if "Ph.D" in text: text = text.replace("Ph.D.","Ph<prd>D<prd>")
        
    text = re.sub("\s" + alphabets + "[.] "," \\1<prd> ",text)
    text = re.sub(acronyms+" "+starters,"\\1<stop> \\2",text)
    text = re.sub(alphabets + "[.]" + alphabets + "[.]" + alphabets + "[.]","\\1<prd>\\2<prd>\\3<prd>",text)
    text = re.sub(alphabets + "[.]" + alphabets + "[.]","\\1<prd>\\2<prd>",text)
    text = re.sub(" "+suffixes+"[.] "+starters," \\1<stop> \\2",text)
    text = re.sub(" "+suffixes+"[.]"," \\1<prd>",text)
    text = re.sub(" " + alphabets + "[.]"," \\1<prd>",text)
    if "”" in text: text = text.replace(".”","”.")
    if "\"" in text: text = text.replace(".\"","\".")
    if "!" in text: text = text.replace("!\"","\"!")
    if "?" in text: text = text.replace("?\"","\"?")
    text = text.replace(".",".<stop>")
    text = text.replace("?","?<stop>")
    text = text.replace("!","!<stop>")
    text = text.replace("<prd>",".")
    sentences = text.split("<stop>")
    sentences = sentences[:-1]
    sentences = [s.strip() for s in sentences]
    
    if len(sentences)==0:
        k=[]
        return [text]
    else:
        return sentences

虽然上面的代码适用于大多数极端情况。但如果在少数情况下失败,例如:

text="Thank you for contacting back. Request you to please help us with the transaction ID for $&lt;***&gt;.92 ? - Charlie."

它将$&lt;***&gt;.92 分解为$&lt;***&gt;.92。我怎样才能在上面的代码中包含这个?

【问题讨论】:

  • 使用 NLTK 库或类似的东西。

标签: python python-3.x text nlp python-re


【解决方案1】:

如果你想扩展你的代码,你可以在浮点值解析中添加美元符号($):

text = re.sub("$" + digits + "[.]" + digits,"\\1<prd>\\2",text)

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 2022-11-04
    • 2013-04-28
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2014-02-11
    相关资源
    最近更新 更多