将 LDA 训练模型应用于给定语料库时出现意外输出答案

【问题标题】：Unexpected output when applying LDA trained model to given corpus将 LDA 训练模型应用于给定语料库时出现意外输出
【发布时间】：2020-07-08 18:12:52
【问题描述】：

我使用以下参数训练了一个 LDA 模型：

>> model = gensim.models.ldamodel.LdaModel(corpus=corpus,
 id2word=id2word,
 num_topics=25,
 passes=10,
 minimum_probability=0)

然后，我将此模型应用于给定的语料库：

>> lda_corpus = model[corpus]

我期望 lda_corpus 是列表或二维矩阵的列表，其中行数是文档数，列数是主题数，每个元素矩阵，形式为 (topic_index,可能性）。但是我得到了这个奇怪的结果，其中一些元素又是一个列表：


>> print(lda_model_1[corpus[0]])


>> ([(0, 0.012841966), (3, 0.073988825), (4, 0.05184835), (8, 0.38537887), (10, 0.022958927), (11, 0.24562633), (13, 0.05168812), (17, 0.06522224), (21, 0.024792604)], [(0, [11]), (1, [8, 3, 17, 13]), (2, [3, 17, 8, 13]), (3, [8, 3]), (4, [11]), (5, [8, 17, 3]), (6, [4]), (7, [4, 8]), (8, [8, 13, 3]), (9, [11]), (10, [8, 0]), (11, [8, 13, 0]), (12, [21]), (13, [11]), (14, [11]), (15, [8]), (16, [8, 11, 13, 0]), (17, [11]), (18, [11, 17]), (19, [8, 13, 17, 3]), (20, [17, 13, 8]), (21, [17, 11, 8]), (22, [11]), (23, [8]), (24, [8, 13]), (25, [8, 3, 13])], [(0, [(11, 1.0)]), (1, [(3, 0.15384258), (8, 0.71774876), (13, 0.011975089), (17, 0.11643356)]), (2, [(3, 0.45133045), (8, 0.21692151), (13, 0.09479065), (17, 0.23232804)]), (3, [(3, 0.24423833), (8, 0.75576156)]), (4, [(11, 1.0)]), (5, [(3, 0.02001735), (8, 1.6895359), (17, 0.2904468)]), (6, [(4, 1.0)]), (7, [(4, 1.2565874), (8, 0.7367453)]), (8, [(3, 0.05150538), (8, 0.8553984), (13, 0.07775658)]), (9, [(11, 2.0)]), (10, [(0, 0.13937186), (8, 0.8588695)]), (11, [(0, 0.023420962), (8, 0.7131521), (13, 0.263427)]), (12, [(21, 1.0)]), (13, [(11, 0.99124163)]), (14, [(11, 2.0)]), (15, [(8, 1.0)]), (16, [(0, 0.011193657), (8, 1.7189965), (11, 0.23104382), (13, 0.029387457)]), (17, [(11, 1.9989293)]), (18, [(11, 0.9135094), (17, 0.08400644)]), (19, [(3, 0.07146881), (8, 2.1837764), (13, 0.38799366), (17, 0.352704)]), (20, [(8, 0.22638415), (13, 0.24114841), (17, 0.52740365)]), (21, [(8, 0.02224951), (11, 0.24574266), (17, 0.7231928)]), (22, [(11, 1.0)]), (23, [(8, 1.0)]), (24, [(8, 0.972818), (13, 0.027181994)]), (25, [(3, 0.16742931), (8, 0.7671518), (13, 0.05224549)])])

我将不胜感激。

【问题讨论】：

您的第二行与第三行不同。你能检查你的第二行的输出： lda_corpus = model[corpus]
您可以尝试以下方法：lda_corpus = [dict(model[x]) for x in corpus]
嗨@Nils_Denter，我设法解决了。请检查我的答案

标签： python gensim lda topic-modeling

【解决方案1】：

问题与模型参数有关。我正在使用以下配置：

lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,
                                               id2word=id2word,
                                               num_topics=ntopics,
                                               random_state=100,
                                               update_every=1,
                                               chunksize=100,
                                               passes=10,
                                               alpha='auto',
                                               per_word_topics=True)

但是，其中有一些是不必要的，并且造成了麻烦。我现在使用的配置如下：

lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus, id2word=id2word, num_topics=ntopics, \
                           update_every=1, chunksize=10000, passes=1)

【讨论】：