【发布时间】:2021-08-03 16:14:03
【问题描述】:
我正在尝试使用 Fasttext 提取僧伽罗语中的变形/相似词。 但是 FastText 需要 1 秒才能写 2.64 个单词。如何在不改变模型大小的情况下提高速度?
我的代码如下所示:
import fasttext
fasttext.util.download_model('si', if_exists='ignore') # Sinhala
ft = fasttext.load_model('cc.si.300.bin')
words_file = open(r'/Datasets/si_words_filtered.txt')
words = words_file.readlines()
words = words[0:300]
synon_dict = dict()
from tqdm import tqdm_notebook
for i in tqdm_notebook(range(len(words))):
word = words[i].strip()
synon = ft.get_nearest_neighbors(word)[0][1] ### takes a lot of time
if is_strictly_sinhala_word(synon):
synon_dict[word] = synon
import json
with open("out.json", "w", encoding='utf8') as f:
json.dump(synon_dict, f, ensure_ascii=False)
【问题讨论】:
标签: python machine-learning nlp gensim fasttext