语义搜索微调答案

【问题标题】：Semantic Search fine-tune语义搜索微调
【发布时间】：2021-08-31 14:51:17
【问题描述】：

例如。句子余弦相似度的预训练 BERT 结果

======================

Query: milk with chocolate flavor

Top 10 most similar sentences in corpus:
Milka milk chocolate 100 g (Score: 0.8672)
Alpro, Chocolate soy drink 1 ltr (Score: 0.6821)
Danone, HiPRO 25g Protein chocolate flavor 330 ml (Score: 0.6692)

在上面的示例中，我正在搜索牛奶，结果应该首先与牛奶相关，但在这里它首先返回巧克力。如何微调结果的相似性？

我用谷歌搜索了它，但没有找到任何合适的解决方案，请帮助我。

代码：

import scipy
import numpy as np
from sentence_transformers import models, SentenceTransformer
model = SentenceTransformer('distilbert-base-multilingual-cased')

corpus = [
          "Alpro, Chocolate soy drink 1 ltr",
          "Milka milk chocolate 100 g",
          "Danone, HiPRO 25g Protein chocolate flavor 330 ml"
         ]
corpus_embeddings = model.encode(corpus)

queries = [
            'milk with chocolate flavor',
          ]
query_embeddings = model.encode(queries)

# Calculate Cosine similarity of query against each sentence i
closest_n = 10
for query, query_embedding in zip(queries, query_embeddings):
    distances = scipy.spatial.distance.cdist([query_embedding], corpus_embeddings, "cosine")[0]

    results = zip(range(len(distances)), distances)
    results = sorted(results, key=lambda x: x[1])

    print("\n======================\n")
    print("Query:", query)
    print("\nTop 10 most similar sentences in corpus:")

    for idx, distance in results[0:closest_n]:
        print(corpus[idx].strip(), "(Score: %.4f)" % (1-distance))

【问题讨论】：

你能为你的语料库提供任何类型的标签吗？例如sim(sample1, sample2)=score
标签在上面的例子中有什么帮助？是的，我可以给它添加标签。
我认为您的标签问题可以转化为某种Natural Language Inference 问题。

标签： python nlp bert-language-model fine-tune

【解决方案1】：

尝试距离阈值

import scipy
import numpy as np
from sentence_transformers import models, SentenceTransformer
model = SentenceTransformer('distilbert-base-multilingual-cased')

corpus = [
          "Alpro, Chocolate soy drink 1 ltr",
          "Milka milk chocolate 100 g",
          "Danone, HiPRO 25g Protein chocolate flavor 330 ml"
         ]
corpus_embeddings = model.encode(corpus)

queries = [
            'milk with chocolate flavor',
          ]
query_embeddings = model.encode(queries)

# Calculate Cosine similarity of query against each sentence i
closest_n = 10
for query, query_embedding in zip(queries, query_embeddings):
    distances = scipy.spatial.distance.cdist([query_embedding], corpus_embeddings, "cosine")[0]

    results = zip(range(len(distances)), distances)
    results = sorted(results, key=lambda x: x[1])

    print("\n======================\n")
    print("Query:", query)
    print("\nTop 10 most similar sentences in corpus:")

    for idx, distance in results[0:closest_n]:
        if 1-distance>0.7:
            print(corpus[idx].strip(), "(Score: %.4f)" % (1-distance))

【讨论】：

这有什么关系？它会返回 Milka 牛奶巧克力 100 g（分数：0.8672），这是错误的。我认为你没有理解问题陈述。