KeyError：“词汇‘限制’不在词汇表中”，同时为文本生成词嵌入向量，从文本文件中读取

Question

当我读取文本文件以生成词嵌入向量时，我收到此错误：“KeyError：单词'restriction'不在词汇表中”，而单词'restrictions'在文本文件中。我想知道我读取文本文件（一个简单的段落）的代码是否错误？

我的代码写在下面：

from gensim.models import Word2Vec
# define training data
with open('D:\\test.txt', 'r') as file:
sentences = ""
#read from textfile
for line in file:
    for word in line.split(' '):
        sentences += word + ' '
# train model
model = Word2Vec(sentences, min_count=1)
# summarize the loaded model
print(model)
# summarize vocabulary
words = list(model.wv.vocab)
# save model
model.save('model.bin')
# load model
new_model = Word2Vec.load('model.bin')
print(new_model)
print(str(model['restriction']))

当我在代码中使用预先写好的句子时，不会出现此错误，如下所示：

from gensim.models import Word2Vec
# define training data
sentences = [['this', 'is', 'the', 'first', 'sentence', 'for', 'word2vec'],  
                ['this', 'is', 'the', 'second', 'sentence'],  
                ['yet', 'another', 'sentence'],  
                ['one', 'more', 'sentence', 'with', 'restriction'],
                ['and', 'the', 'final', 'sentence']]
# train model
model = Word2Vec(sentences, min_count=1)
# summarize the loaded model
print(model)
# summarize vocabulary
words = list(model.wv.vocab)
print(words)
# access vector for one word
print(model['sentence'])
# save model
model.save('model.bin')
# load model
new_model = Word2Vec.load('model.bin')
print(new_model)
print('the model prints: ')
print(model['restriction'])

Answer 1

在您的代码中显示问题，在构建它之后，请仔细检查sentences 987654321，看看它是否是您期望的格式（或者在工作情况的sentences 987654322的格式）。我怀疑不是。

还，看看令人失望的模型学习的单词列表 - 你创建的words 5变量应该足够了。它也可能看起来不像你期望的话。

具体地，您的代码...

sentences = ""
for line in file:
    for word in line.split(' '):
        sentences += word + ' '

... make sentences一串长串，有很多空间分开的单词。如果您在工作代码中的sentences @ 987654326，您将不再有列表，其中每个项目是令牌列表。 （这是Word2Vec。）的良好输入格式，而是有一个巨大的run-on string：

sentences = 'this is the first sentence for word2vec this is the second sentence yet another sentence one more sentence with restriction and the final sentence'

尝试：

sentences = []  # empty list
# OOPS, DON'T DO: sentences = ""
for line in file:
    sentences.append(line.split(' '))

...那么你的@ 987654330将是一个字符串列表（如工作情况），而不是只是一个字符串（如破坏的情况）。