训练好的 word2vec 模型词汇表中缺少的单词

Question

我目前正在使用 python，我使用我提供的句子训练 Word2Vec 模型。然后，我保存并加载模型以获取用于训练模型的句子中每个单词的词嵌入。但是，我收到以下错误。

KeyError：“单词 'n1985_chicago_bears' 不在词汇表中”

然而，训练期间提供的句子之一如下。

sportsteam n1985_chicago_bears teamplaysincity city chicago

因此，我想知道为什么词汇表中缺少一些单词，尽管我接受了该句子语料库中的这些单词的训练。

在自己的语料库上训练 word2vec 模型

import nltk
import numpy as np
from termcolor import colored
from gensim.models import Word2Vec
from gensim.models import KeyedVectors
from sklearn.decomposition import PCA


#PREPARING DATA

fname = '../data/sentences.txt'

with open(fname) as f:
    content = f.readlines()

# remove whitespace characters like `\n` at the end of each line
content = [x.strip() for x in content]


#TOKENIZING SENTENCES

sentences = []

for x in content:
    nltk_tokens = nltk.word_tokenize(x)
    sentences.append(nltk_tokens)

#TRAINING THE WORD2VEC MODEL

model = Word2Vec(sentences)

words = list(model.wv.vocab)
model.wv.save_word2vec_format('model.bin')

sentence.txt 中的示例句子

sportsteam hawks teamplaysincity city atlanta
stadiumoreventvenue honda_center stadiumlocatedincity city anaheim
sportsteam ducks teamplaysincity city anaheim
sportsteam n1985_chicago_bears teamplaysincity city chicago
stadiumoreventvenue philips_arena stadiumlocatedincity city atlanta
stadiumoreventvenue united_center stadiumlocatedincity city chicago
...

sentences.txt 文件中有 1860 行这样的行，每行正好包含 5 个单词且没有停用词。

保存模型后，我尝试从与保存的model.bin 相同的目录中的不同python文件加载它，如下所示。

加载保存的model.bin

import nltk
import numpy as np
from gensim import models

w = models.KeyedVectors.load_word2vec_format('model.bin', binary=True)
print(w['n1985_chicago_bears'])

但是，我最终得到以下错误

KeyError: "word 'n1985_chicago_bears' not in vocabulary"

有没有办法使用相同的方法为训练好的句子语料库中的每个单词获取单词嵌入？

在这方面的任何建议将不胜感激。

Answer 1

用于 gensim 的 Word2Vec 实现的默认 min_count=5 看起来像您正在寻找的令牌 n1985_chicago_bears 在您的语料库中出现少于 5 次。适当更改您的最小计数。

Method signature:

class gensim.models.word2vec.Word2Vec(sentences=None, corpus_file=None, size=100, alpha=0.025, window=5, min_count=5, max_vocab_size=无，样本=0.001，种子=1，工人=3， min_alpha=0.0001, sg=0, hs=0, 负数=5, ns_exponent=0.75, cbow_mean=1, hashfxn=, iter=5, null_word=0, trim_rule=None，sorted_vocab=1，batch_words=10000，compute_loss=False， callbacks=(), max_final_vocab=None)

content = [
    "sportsteam hawks teamplaysincity city atlanta",
    "stadiumoreventvenue honda_center stadiumlocatedincity city anaheim",
    "sportsteam ducks teamplaysincity city anaheim",
    "sportsteam n1985_chicago_bears teamplaysincity city chicago",
    "stadiumoreventvenue philips_arena stadiumlocatedincity city atlanta",
    "stadiumoreventvenue united_center stadiumlocatedincity city chicago"
]

sentences = []

for x in content:
    nltk_tokens = nltk.word_tokenize(x)
    sentences.append(nltk_tokens)

model = Word2Vec(sentences, min_count=1)
print (model['n1985_chicago_bears'])