忽略词性模式NLTK中的填充词

Question

我编写了基于规则的文本匹配程序，该程序根据使用特定 POS 模式创建的规则进行操作。例如，一条规则是：

pattern = [('PRP', "i'll"), ('VB', ('jump', 'play', 'bite', 'destroy'))]

在这种情况下，当分析我的输入文本时，这只会返回一个在语法上符合此特定模式的字符串，因此：

I'll jump
I'll play
I'll bite
I'll destroy

我的问题涉及当人们使用相同的文本但添加最高级或任何不会改变上下文的单词时提取文本的相同含义，现在它只进行 exact 匹配，但不会捕捉到此示例中的第一个字符串之类的短语：

I'll 'freaking' jump 
'Dammit' I'll play
I'll play 'dammit'

单词不必特别具体，只需确保程序仍然可以通过添加非上下文最高级或具有相同目的的任何其他类型的单词来识别相同的模式。这是我写的标记器，我给出了一个示例字符串：

string_list = [('Its', 'PRP$'), ('annoying', 'NN'), ('when', 'WRB'), ('a', 'DT'), ('kid', 'NN'), ('keeps', 'VBZ'), ('asking', 'VBG'), ('you', 'PRP'), ('to', 'TO'), ('play', 'VB'), ('but', 'CC'), ("I'll", 'NNP'), ('bloody', 'VBP'), ('play', 'VBP'), ('so', 'RB'), ('it', 'PRP'), ('doesnt', 'VBZ'), ('cry', 'NN')]

def find_match_pattern(string_list, pattern_dict):

    from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

    analyzer = SentimentIntensityAnalyzer() # does a sentiment analysis on the output string

    filt_ = ['Filter phrases'] # not the patterns just phrases I know I dont want

    filt_tup = [x.lower() for x in filt_]

    for rule, pattern in pattern_dict.items(): # pattern dict is an Ordered Diction courtesy of collections

        num_matched = 0

        for idx, tuple in enumerate(string_list): # string_list is the input string that has been POS tagged
            matched = False

            if tuple[1] == list(pattern.keys())[num_matched]:

                if tuple[0] in pattern[tuple[1]]:
                    num_matched += 1
                else:
                    num_matched = 0
            else:
                num_matched = 0

            if num_matched == len(pattern): # if the number of matching words equals the length of the pattern do this

                matched_string = ' '.join([i[0] for i in string_list]) # Joined for the sentiment analysis score
                vs = analyzer.polarity_scores(matched_string)
                sentiment = vs['compound']

                if matched_string in filt_tup:
                    break

                elif (matched_string not in filt_tup) or (sentiment < -0.8):
                    matched = True
                    print(matched, '\n', matched_string, '\n', sentiment)

                return (matched, sentiment, matched_string, rule)

我知道这是一个非常抽象的问题（或者说非常抽象），所以这可能是一个讨论，但如果有人有这方面的经验，那么看到你推荐的东西会很棒。

Answer 1

您的问题可以使用 Spacy 的依赖标记器来回答。 Spacy 提供了一个带有许多可选和可切换选项的匹配器。

在下面的例子中，重点不是基于特定的单词或词性，而是关注某些sintatic功能，例如名词主语和助动词。

这是一个简单的例子：

import spacy
from spacy.matcher import Matcher
nlp = spacy.load('en')
matcher = Matcher(nlp.vocab, validate=True)
pattern = [{'DEP': 'nsubj', 'OP': '+'}, # OP + means it has to be at least one nominal subject - usually a pronoun
           {'DEP': 'aux', 'OP': '?'}, # OP ? means it can have one or zero auxiliary verbs
           {'POS': 'ADV', 'OP': '?'}, # Now it looks for an adverb. Also, it is not needed (OP?)
           {'POS': 'VERB'}] # Finally, I've generallized it with a verb, but you can make one pattern for each verb or write a loop to do it.
matcher.add("NVAV", None, pattern)
phrases = ["I\'ll really jump.",
           "Okay, I\'ll play.",
           "Dammit I\'ll play",
           "I\'ll play dammit",
          "He constantly plays it",
          "She usually works there"]

for phrase in phrases:
    doc = nlp(phrase)

    matches = matcher(doc)
    for match_id, start, end in matches:
        span = doc[start:end]
        print('Matched:',span.text)

匹配：我真的会跳

匹配：我会玩

匹配：我会玩

匹配：我会玩

匹配：他经常玩

匹配：她通常工作

您始终可以在实时示例中测试您的模式：Spacy Live Example

您可以随意扩展它。在这里阅读更多：https://spacy.io/usage/rule-based-matching