【问题标题】:Reduce redundant block of text减少多余的文本块
【发布时间】:2013-05-21 05:13:24
【问题描述】:

如何减少多余的文本?例如,我有 2 个输入,我需要将它们减少到以下输出

输入 1: 丰田是红色的。本田是红色的。宝马是红色的。这 梅赛德斯是绿色的。

输出 1: 丰田、本田和宝马是红色的,但 梅赛德斯是绿色的。

输入 2: 丰田是红色的。本田是红色的。宝马是红色的。这 梅赛德斯是红色的。

输出 2:所有汽车都是红色的。

我假设这是一个 NLP 问题。理想情况下,我希望在 Python 中执行此操作(但任何其他语言也不错,首选开发简单)

【问题讨论】:

  • 我认为首先应该定义某种语法。例如:简单赋值:The <variable> is <value>. 有了这个之后,你可以简单地继续,将它们连接在一起..
  • 它是“grammar!”这些问题的准确性很重要:)
  • 抱歉,我要解决这个问题——英语不是我的母语——我猜你刚刚意识到 :)
  • 天哪,这是->舌头:P:P:P

标签: python nlp nltk


【解决方案1】:

正如我对您的问题所评论的那样:我认为您首先应该定义某种语法。例如一个简单的赋值语句:The <variable> is <value>.。我为你的第一行创建了一个小例子,我想你会明白这个想法,所以你也可以为下一行做:

import re

def reducer(text):  # Catch statements and add them to a dictionary
    catched = dict()
    for v, k in re.findall(r'The\s(?P<variable>\w+)\sis\s(?P<value>\w+)', text):
        try:
            catched[k].append(v)
        except KeyError:
            catched[k] = [v]
    return catched

def comma_and(ls):  # Create human-like enumeration with words
    output = [ls[0]]
    for i in xrange(1, len(ls)):
        output.append('%s%s' % (' and ' if i == len(ls) - 1 else ', ', ls[i]))
    return ''.join(output)

def rephrase(text):  # Rephrase separated statements into one sentence
    stmnts = reducer(text)
    part1 = str()
    part2 = str()
    for key in stmnts:
        if len(stmnts[key]) <= 1:
            part2 = 'but the {variable} is {value}.'.format(
                variable=stmnts[key][0], value=key)
        else:
            part1 = 'The {variables} are {value}'.format(
                variables=comma_and(stmnts[key]), value=key)
    print part1 + ' ' + part2

演示:

rephraser('The Toyota is red. The Honda is red. The BMW is red. The Mercedes is green.')

输出是:

# The Toyota, Honda and BMW are red but the Mercedes is green.

你的下一行是这样的:检查字典 catched 是否只有一个键,如果该键中的值都是汽车,请使用 All &lt;type&gt; are &lt;value&gt;.

【讨论】:

    【解决方案2】:

    有趣的案例! 我做了类似以下的事情

    import re, sys, random
    def main():
        sentence = get_sentence()
        print("Simplifying sentence:")
        print(""" \" %s \" """ % sentence )
        mapped_elements = map_colors(sentence)
        # 3 possible casses
        number_of_parts = len(mapped_elements)
        if number_of_parts == 1:
            color = mapped_elements[0][0]
            elements = mapped_elements[0][1]
            if elements == 1:
                simplified = "The %s is %s." % (elements, color)
            else:
                simplified = "All cars are %s." % (color)
    
        elif number_of_parts == 2:
            part1 = mapped_elements[0]
            part2 = mapped_elements[1]
    
            part1_sentence = "The %s %s %s" % (gather_elements(part1[1]) ,pluralize(len(part1[1])),part1[0])
            part2_sentence = "the %s %s %s" % (gather_elements(part2[1]), pluralize(len(part2[1])),part2[0])
            simplified = "%s but %s." % (part1_sentence, part2_sentence)
        else:
            all_parts = []
            for color, nouns in mapped_elements:
                part_sentence = "The %s %s %s. " % (gather_elements(nouns) , pluralize(len(nouns)), color)
                all_parts.append(part_sentence)
            simplified = ''.join(all_parts)
    
        print("Result: %s" % simplified)
        return simplified
    
    def pluralize(n):
        if n == 1: return "is"
        return "are"
    
    def get_sentence():
        if len(sys.argv) > 1:
            sentence = sys.argv[1]
        elif test_mode:
            sentence = random.choice(sample_test)
        else:
            sentence = raw_input("Please enter a sentence to simplify: ")
        return sentence
    
    
    
    def gather_elements(elements):
        if len(elements) == 1: 
            return elements[0]
        partial = ', '.join(elements[:-1])
        partial += " and %s " % (elements[-1])
        return partial
    
    def map_colors(sentence):
        colors = {}
        r = re.compile(r'The (\w+) is (\w+)\.')
        pairs = re.findall( r, sentence)
        for noun, color in pairs:
            if color in colors:
                colors[color].append(noun)
            else:
                colors[color] = [noun]
        return colors.items()
    

    【讨论】:

    • 抱歉,我只能选择 1 个答案,但你的答案也一样好 - 谢谢 mariano - 我也对这些行有一个大致的了解
    • 没问题! :-) 感谢您提出这个有趣的问题。
    猜你喜欢
    • 1970-01-01
    • 2019-06-06
    • 1970-01-01
    • 2014-08-28
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2019-06-20
    • 1970-01-01
    相关资源
    最近更新 更多