word2vec 使用 tf-idf 和可视化

Question

我想在亚马逊评论数据集上使用 word2vec 和 tfidf，但我不知道该怎么做....我已经对包含 5k 条正面评分和 5k 条负面评分的评论的数据进行了抽样...... ...有一个对应于 +ve 或 -ve 的列分数。我试过但我有以下错误......

#TF-IDF


final_tf_idf = tf_idf_vect.fit_transform(final_data['CleanedText'].values)
tfidf_feat = tf_idf_vect.get_feature_names()
 # tfidf words/col-names
# final_tf_idf is the sparse matrix with row= sentence, col=word and cell_val = tfidf

tfidf_sent_vectors = []; # the tfidf-w2v for each sentence/review is stored in this list
row=0;
for sent in list_of_sent: # for each review/sentence
    sent_vec = np.zeros(50) # as word vectors are of zero length
    weight_sum =0; # num of words with a valid vector in the sentence/review
    for word in sent: # for each word in a review/sentence
        try:
            vec = w2v_model.wv[word]
            # obtain the tf_idfidf of a word in a sentence/review
            tfidf = X[row, tfidf_feat.index(word)]
            sent_vec += (vec * tfidf)
            weight_sum += tfidf
        except:
            pass
    sent_vec /= weight_sum
    print(np.isnan(np.sum(sent_vec)))

    tfidf_sent_vectors.append(sent_vec)
    row += 1

错误：在 true_divide 中遇到无效值 看起来我在 send_vec 的每一行中都有 NaN 值

任何帮助将不胜感激...!!

Answer 1

下面这行代码有问题

tfidf = X[row, tfidf_feat.index(word)]

这里的 'X' 不是你的向量，而是用 final_tf_idf 替换它

tfidf = final_tf_idf [row, tfidf_feat.index(word)]

试试看，然后告诉我。