Python：检查句子是否包含列表中的任何单词（模糊匹配）答案

【问题标题】：Python: Check if the sentence contains any word from List (with fuzzy match)Python：检查句子是否包含列表中的任何单词（模糊匹配）
【发布时间】：2019-02-21 13:01:35
【问题描述】：

我想从给定 list_of_keywords 的句子中提取关键字。

我设法提取了确切的单词

[word for word in Sentence if word in set(list_of_keywords)]

是否可以提取与给定list_of_keywords具有良好相似性的单词，即两个单词之间的余弦相似度> 0.8

例如，给定列表中的关键字是“过敏”，现在句子写成

“对她吃的那顿饭中的坚果产生了严重的过敏反应。”

“过敏”和“过敏”之间的余弦距离可以计算如下

cosdis(word2vec('allergy'), word2vec('allergic'))
Out[861]: 0.8432740427115677

如何根据余弦相似度从句子中提取“过敏”？

【问题讨论】：

既然您知道您要与集合比较哪个单词，为什么不对cosdis 的 returned 值进行阈值检查，如果返回值是否大于阈值？
发布您的cosdis() 方法
stackoverflow.com/a/29485365/10447069
确实如此！谢谢

标签： python text-mining fuzzy-search

【解决方案1】：

def word2vec(word):
    from collections import Counter
    from math import sqrt

    # count the characters in word
    cw = Counter(word)
    # precomputes a set of the different characters
    sw = set(cw)
    # precomputes the "length" of the word vector
    lw = sqrt(sum(c*c for c in cw.values()))

    # return a tuple
    return cw, sw, lw

def cosdis(v1, v2):
    # which characters are common to the two words?
    common = v1[1].intersection(v2[1])
    # by definition of cosine distance we have
    return sum(v1[0][ch]*v2[0][ch] for ch in common)/v1[2]/v2[2]


list_of_keywords = ['allergy', 'something']
Sentence = 'a severe allergic reaction to nuts in the meal she had consumed.'

threshold = 0.80
for key in list_of_keywords:
    for word in Sentence.split():
        try:
            # print(key)
            # print(word)
            res = cosdis(word2vec(word), word2vec(key))
            # print(res)
            if res > threshold:
                print("Found a word with cosine distance > 80 : {} with original word: {}".format(word, key))
        except IndexError:
            pass

输出：

Found a word with cosine distance > 80 : allergic with original word: allergy

编辑：

单线杀手：

print([x for x in Sentence.split() for y in list_of_keywords if cosdis(word2vec(x), word2vec(y)) > 0.8])

输出：

['allergic']

【讨论】：

正是我的意思:)
感觉是相互的 :) 顺便说一句，虽然你的函数 word2vec 做了所需要的，OP 可能正在使用 this word2vec
@RickM。不是真的，OP 在 cmets 中提到了相同的方法！我刚刚猜对了！ :p
非常感谢代码和想法 (@RickM.)。它有很大帮助：D

【解决方案2】：

必须根据所有关键字检查单词的距离，并且仅当任何关键字达到阈值时才会包括在内。我在原始列表推导中添加了一个额外的条件，其中嵌套的列表推导正是这样做的。

def distance(words):
    return cosdis(word2vec(words[0]), word2vec(words[1]))

threshold = 0.8
keywords = set(list_of_keywords)
matches = [word for word in Sentence if word in keywords and 
           any([distance(word, keyword) > threshold for keyword in keywords])]

【讨论】：

感谢您的回答

【解决方案3】：

senectence = 'a severe allergic reaction to nuts in the meal she had consumed.'
list_of_keywords = ['allergy','reaction']
word_list = []
for keyword in list_of_keywords:
    for word in senectence.split():
        if(cosdis(word2vec(keyword), word2vec(word)) > 0.8):
            word_list.append(word)

或者如果您只想根据关键字“过敏”提取单词

[word for word in Sentence if cosdis(word2vec('allergy'), word2vec(word)) > 0.8]

【讨论】：