两个不同DataFrame的列之间的余弦相似度

Question

我想计算两个 DataFrame 之间的余弦相似度（对于不同的大小）并将结果存储在新数据中。使用 BERT 嵌入计算相似度

 df1
title
Lorem ipsum dolor sit amet
Lorem ipsum dolor sit amet
Lorem ipsum dolor sit amet

df2
claim
fact checked claims one
fact checked claims tweet

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('bert-base-nli-mean-tokens')
df_1['title_embeddings']=df_1['title'].apply(lambda x: model.encode(x))
df_2['claim_embeddings']=df_2['claim'].apply(lambda x: model.encode(x))

sim_score=[]
text =[]
for i in range(len(df['claim_embeddings'])):
   t=df['title_embeddings'].apply(lambda x: cosine_similarity(x, df['claim_embeddings'][i]))
   sim_score.append(t)
   text.append(claim_embeddings'][i])

当前错误

ValueError: Expected 2D array, got 1D array instead:

预期输出

df
title                       claims                  sim score
Lorem ipsum dolor sit amet fact checked claims one    0
Lorem ipsum dolor sit amet fact checked claims one    0
Lorem ipsum dolor sit amet fact checked claims one    0
Lorem ipsum dolor sit amet fact checked claims tweet   0  
Lorem ipsum dolor sit amet fact checked claims tweet   0
Lorem ipsum dolor sit amet fact checked claims tweet   0

我试过Calculate cosine similarity for vectors between two pandas columns?，但没有解决问题。

Answer 1

这里必须使用array.reshape(1, -1)，因为您只比较单个样本。例如cosine_similarity(x.reshape(1,-1),y.reshape(1,-1))

from sklearn.metrics.pairwise import cosine_similarity
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('bert-base-nli-mean-tokens')
title=df['title'].tolist()
claim=df['claim'].tolist()
title=model.encode(title)
claim=model.encode(claim)
from sklearn.metrics.pairwise import cosine_similarity
cosine_similarity(title[0].reshape(1,-1),claim[0].reshape(1,-1))