在 python 中加载预训练的手套向量

Question

我从网上下载了预训练手套矢量文件。它是一个 .txt 文件。我无法加载和访问它。使用 gensim 很容易加载和访问词向量二进制文件，但是当它是文本文件格式时我不知道该怎么做。

提前致谢

Answer 1

手套模型文件采用词向量格式。您可以打开文本文件来验证这一点。以下是一小段可用于加载预训练手套文件的 sn-p 代码：

import numpy as np

def load_glove_model(File):
    print("Loading Glove Model")
    glove_model = {}
    with open(File,'r') as f:
        for line in f:
            split_line = line.split()
            word = split_line[0]
            embedding = np.array(split_line[1:], dtype=np.float64)
            glove_model[word] = embedding
    print(f"{len(glove_model)} words loaded!")
    return glove_model

然后，您只需使用 gloveModel 变量即可访问词向量。

print(gloveModel['hello'])

Answer 2

你可以用 pandas 更快地做到这一点：

import pandas as pd
import csv

words = pd.read_table(glove_data_file, sep=" ", index_col=0, header=None, quoting=csv.QUOTE_NONE)

然后得到一个单词的向量：

def vec(w):
  return words.loc[w].as_matrix()

并找到最接近向量的词：

words_matrix = words.as_matrix()

def find_closest_word(v):
  diff = words_matrix - v
  delta = np.sum(diff * diff, axis=1)
  i = np.argmin(delta)
  return words.iloc[i].name

Answer 3

我建议使用gensim 来做所有事情。您可以阅读该文件，还可以从这个出色的包上已经实现的许多方法中受益。

假设您使用 C++ 程序生成了GloVe 向量，并且您的“-save-file”参数是“向量”。 Glove 可执行文件将为您生成两个文件，“vectors.bin”和“vectors.txt”。

使用glove2word2vec将文本格式的GloVe向量转换为word2vec文本格式：

from gensim.scripts.glove2word2vec import glove2word2vec
glove2word2vec(glove_input_file="vectors.txt", word2vec_output_file="gensim_glove_vectors.txt")

最后，使用KeyedVectors将word2vec txt读入gensim模型：

from gensim.models.keyedvectors import KeyedVectors
glove_model = KeyedVectors.load_word2vec_format("gensim_glove_vectors.txt", binary=False)

现在您可以随意使用 gensim word2vec 方法（例如相似度）。

Answer 4

我发现这种方法更快。

import pandas as pd

df = pd.read_csv('glove.840B.300d.txt', sep=" ", quoting=3, header=None, index_col=0)
glove = {key: val.values for key, val in df.T.items()}

保存字典：

import pickle
with open('glove.840B.300d.pkl', 'wb') as fp:
    pickle.dump(glove, fp)

Answer 5

如果你想要的只是嵌入矩阵，这里有一个衬线

np.loadtxt(path, usecols=range(1, dim+1), comments=None)

path 是您下载的 GloVe 文件的路径，dim 是词嵌入的维度。

如果你想要单词和对应的向量，你可以这样做

glove = np.loadtxt(path, dtype='str', comments=None)

并将单词和向量分开如下

words = glove[:, 0]
vectors = glove[:, 1:].astype('float')

Answer 6

从文本文件加载词嵌入（在我的情况下是 glove.42B.300d 嵌入）需要一点时间（在我的机器上147.2s）。

首先将文本文件转换为两个新文件有帮助：一个仅包含单词的文本文件（例如embeddings.vocab）和一个二进制文件文件，其中包含作为 numpy 结构的嵌入向量（例如 embeddings.npy）。

转换后，我只需 4.96 秒 即可将相同的嵌入加载到内存中。这种方法最终得到与从文本文件中加载完全相同的字典。它在访问时间上同样高效，不需要任何额外的框架，但加载时间要快得多。

使用此代码，您可以将嵌入文本文件转换为两个新文件：

def convert_to_binary(embedding_path):
    f = codecs.open(embedding_path + ".txt", 'r', encoding='utf-8')
    wv = []

    with codecs.open(embedding_path + ".vocab", "w", encoding='utf-8') as vocab_write:
        count = 0
        for line in f:
            splitlines = line.split()
            vocab_write.write(splitlines[0].strip())
            vocab_write.write("\n")
            wv.append([float(val) for val in splitlines[1:]])
        count += 1

    np.save(embedding_path + ".npy", np.array(wv))

通过这种方法，您可以有效地将其加载到您的内存中：

def load_word_emb_binary(embedding_file_name_w_o_suffix):
    print("Loading binary word embedding from {0}.vocab and {0}.npy".format(embedding_file_name_w_o_suffix))

    with codecs.open(embedding_file_name_w_o_suffix + '.vocab', 'r', 'utf-8') as f_in:
        index2word = [line.strip() for line in f_in]

    wv = np.load(embedding_file_name_w_o_suffix + '.npy')
    word_embedding_map = {}
    for i, w in enumerate(index2word):
        word_embedding_map[w] = wv[i]

    return word_embedding_map

免责声明：此代码无耻地从https://blog.ekbana.com/loading-glove-pre-trained-word-embedding-model-from-text-file-faster-5d3e8f2b8455 窃取。但它可能对这个线程有所帮助。

Answer 7

Python3 版本也可以处理二元组和三元组：

import numpy as np


def load_glove_model(glove_file):
    print("Loading Glove Model")
    f = open(glove_file, 'r')
    model = {}
    vector_size = 300
    for line in f:
        split_line = line.split()
        word = " ".join(split_line[0:len(split_line) - vector_size])
        embedding = np.array([float(val) for val in split_line[-vector_size:]])
        model[word] = embedding
    print("Done.\n" + str(len(model)) + " words loaded!")
    return model

Answer 8

import os
import numpy as np

# store all the pre-trained word vectors
print('Loading word vectors...')
word2vec = {}
with open(os.path.join('glove/glove.6B.%sd.txt' % EMBEDDING_DIM)) as f: #enter the path where you unzipped the glove file
  # is just a space-separated text file in the format:
  # word vec[0] vec[1] vec[2] ...
    for line in f:
        values = line.split()
        word = values[0]
        vec = np.asarray(values[1:], dtype='float32')
        word2vec[word] = vec
print('Found %s word vectors.' % len(word2vec))

Answer 9

此代码需要一些时间将手套嵌入存储在货架上，但与其他方法相比，加载它的速度要快得多。

import os
import numpy as np
from contextlib import closing
import shelve

def store_glove_to_shelf(glove_file_path,shelf):
    print('Loading Glove')
    with open(os.path.join(glove_file_path)) as f:
        for line in f:
            values = line.split()
            word = values[0]
            vec = np.asarray(values[1:], dtype='float32')
            shelf[word] = vec

shelf_file_name = "glove_embeddings"
glove_file_path = "glove/glove.840B.300d.txt"

# Storing glove embeddings to shelf for faster load
with closing(shelve.open(shelf_file_name + '.shelf', 'c')) as shelf:
    store_glove_to_shelf(glove_file_path,shelf)
    print("Stored glove embeddings from {} to {}".format(glove_file_path,shelf_file_name+'.shelf'))

# To reuse the glove embeddings stored in shelf
with closing(shelve.open(shelf_file_name + '.shelf')) as embeddings_index:
    # USE embeddings_index here , which is a dictionary
    print("Loaded glove embeddings from {}".format(shelf_file_name+'.shelf'))
    print("Found glove embeddings with {} words".format(len(embeddings_index)))

Answer 10

每个语料库需要以包含词汇大小和向量大小的行开头。

打开手套模型的.txt文件，在第一行输入向量的维度，先回车：

例如，对于glove.6B.50d.txt，只需在第一行添加400000 50。

然后使用 gensim 将原始 .txt 矢量文件转换为 gensim 矢量格式：

import gensim

word_vectors = gensim.models.KeyedVectors.load_word2vec_format('path/glove.6B.50d.txt', binary=False)
word_vectors.save('path/glove_gensim.txt')

Answer 11

这里的其他一些方法需要更多存储空间（例如拆分文件），或者在我的个人笔记本电脑上运行速度很慢。我尝试了货架数据库，但它似乎在存储大小上爆炸了。这是一种“就地”方法，具有一次性文件读取时间成本和非常低的额外存储成本。我们将原始文本文件视为数据库，只存储每个单词的位置。例如，当您研究词向量的属性时，这非常有效。

# First create a map from words to position in the file
def get_db_mapping(fname):
    char_ct = 0    # cumulative position in file
    pos_map = dict()

    with open(fname + ".txt", 'r', encoding='utf-8') as f:
        for line in tqdm(f):
            new_len = len(line)     # len of line

            # get the word
            splitlines = line.split()
            word = splitlines[0].strip()

            # store and increment counter
            pos_map[word] = char_ct
            char_ct += new_len

    # write dict
    with open(fname + '.db', 'wb') as handle:
        pickle.dump(pos_map, handle)


class Embedding:
"""Small wrapper so that we can use [] notation to fetch word vectors.
It would be better to just have the file pointer and the pos_map as part
of this class, but that's not how I wrote it initially."""
    def __init__(self, emb_fn):
        self.emb_fn = emb_fn

    def __getitem__(self, item):
        return self.emb_fn(item)


def load_db_mapping(fname, cache_size=1000) -> Embedding:
    """Creates a function closure that wraps access to the db mapping
    and the text file that functions as db. Returns them as an
    Embedding object"""
    # get the two state objects: mapping and file pointer
    with open(fname + '.db', 'rb') as handle:
        pos_map = pickle.load(handle)
    f = open(fname + ".txt", 'r', encoding='utf-8')

    @lru_cache(maxsize=cache_size)
    def get_vector(word: str):
        pos = pos_map[word]
        f.seek(pos, 0)

        # special logic needed because of small count errors
        fail_ct = 0
        read_word = ""
        while fail_ct < 5 and read_word != word:
            fail_ct += 1
            l = f.readline()
            try:
                splitlines = l.split()
                read_word = splitlines[0].strip()
            except:
                continue
        if read_word != word:
            raise ValueError('word not found')

        # actually return
        return np.array([float(val) for val in splitlines[1:]])

    return Embedding(get_vector)

# to run
k_glove_vector_name = 'glove.42B.300d'   # omit .txt
get_db_mapping(k_glove_vector_name)      # run only once; creates .db
word_embedding = load_db_mapping(k_glove_vector_name)
word_embedding['hello']

Answer 12

EMBEDDING_LIFE = 'path/to/your/glove.txt'

def get_coefs(word,*arr): 
      return word, np.asarray(arr, dtype='float32')

embeddings_index = dict(get_coefs(*o.strip().split()) for o in open(EMBEDDING_FILE))

all_embs = np.stack(embeddings_index.values())
emb_mean,emb_std = all_embs.mean(), all_embs.std()
word_index = tokenizer.word_index
nb_words = min(max_features, len(word_index))

embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))

for word, i in word_index.items():
if i >= max_features: continue
embedding_vector = embeddings_index.get(word)
if embedding_vector is not None: embedding_matrix[i] = embedding_vector