变形金刚：WordLevel 分词器产生奇怪的词汇答案

【问题标题】：Transformers: WordLevel tokenizer produces strange vocabulary变形金刚：WordLevel 分词器产生奇怪的词汇
【发布时间】：2021-10-25 21:14:23
【问题描述】：

训练 WordLevel 分词器我收到了奇怪的词汇。下面是我的代码：

data = [
    "Beautiful is better than ugly."
    "Explicit is better than implicit."
    "Simple is better than complex."
    "Complex is better than complicated."
    "Flat is better than nested."
    "Sparse is better than dense."
    "Readability counts."
]

from tokenizers.models import WordLevel
from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, decoders, trainers

tokenizer = Tokenizer(models.WordLevel())

trainer = trainers.WordLevelTrainer(
    vocab_size=100000,
)

tokenizer.train_from_iterator(data, trainer=trainer)

tokenizer.get_vocab()

输出如下：

{'Beautiful is better than ugly.Explicit is better than implicit.Simple is better than complex.Complex is better than complicated.Flat is better than nested.Sparse is better than dense.Readability counts.': 0}

请解释我做错了什么......

【问题讨论】：

标签： python huggingface-transformers huggingface-tokenizers

【解决方案1】：

您的数据定义不正确，您的数据的 len() 为 1。它需要逗号，如下所示：

data = [
    "Beautiful is better than ugly.",
    "Explicit is better than implicit.",
    "Simple is better than complex.",
    "Complex is better than complicated.",
    "Flat is better than nested.",
    "Sparse is better than dense.",
    "Readability counts.",
]

另外，如果你想传入一个序列序列，你可以使用 map() 函数来应用 split()，如下所示：

tokenizer.train_from_iterator(map(lambda x: x.split(), data), trainer=trainer)

【讨论】：