huggingface 的 ReformerForMaskedLM 配置问题答案

【问题标题】：huggingface's ReformerForMaskedLM configuration issuehuggingface 的 ReformerForMaskedLM 配置问题
【发布时间】：2021-06-12 00:12:07
【问题描述】：

我正在尝试将所有拥抱脸的 ...ForMaskedLM 传递给 FitBert 模型以执行填空任务，并查看哪个预训练在我准备的数据上产生最佳结果。但是在改革者模块中，我有这个错误说我需要做'config.is_decoder = False'但我真的不明白这意味着什么（这是我第一次使用拥抱脸）。我试图将 ReformerConfig(is_decoder=False) 传递给模型，但仍然得到相同的错误。我该如何解决这个问题？

我的代码：

pretrained_weights = ['google/reformer-crime-and-punishment', 
                      'google/reformer-enwik8']
configurations = ReformerConfig(is_decoder=False)
for weight in pretrained_weights:
    print(weight)
    model = ReformerForMaskedLM(configurations).from_pretrained(weight)
    tokenizer = ReformerTokenizer.from_pretrained(weight)
    fb = FitBert(model=model, tokenizer=tokenizer)
    predicts = []
    for _, row in df.iterrows():
        predicts.append(fb.rank(row['question'], options=[row['1'], row['2'], row['3'], row['4']])[0])
    print(weight,':', np.sum(df.anwser==predicts) / df.shape[0])

错误：

AssertionError                            Traceback (most recent call last)
<ipython-input-5-a6016e0015ba> in <module>()
      4 for weight in pretrained_weights:
      5     print(weight)
----> 6     model = ReformerForMaskedLM(configurations).from_pretrained(weight)
      7     tokenizer = ReformerTokenizer.from_pretrained(weight)
      8     fb = FitBert(model=model, tokenizer=tokenizer)
/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1032 
   1033         # Instantiate model.
-> 1034         model = cls(config, *model_args, **model_kwargs)
   1035 
   1036         if state_dict is None and not from_tf:

/usr/local/lib/python3.7/dist-packages/transformers/models/reformer/modeling_reformer.py in __init__(self, config)
   2304         assert (
   2305             not config.is_decoder
-> 2306         ), "If you want to use `ReformerForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention."
   2307         self.reformer = ReformerModel(config)
   2308         self.lm_head = ReformerOnlyLMHead(config)

AssertionError: If you want to use `ReformerForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.

【问题讨论】：

标签： nlp bert-language-model huggingface-transformers

【解决方案1】：

您可以通过单独加载模型配置并将其作为from_pretrained() 方法的参数提供来覆盖某些模型配置。这将确保您使用正确的模型配置和所做的更改：

from transformers import ReformerConfig, ReformerForMaskedLM

config = ReformerConfig.from_pretrained('google/reformer-crime-and-punishment')
print(config.is_decoder)
config.is_decoder=False
print(config.is_decoder)
model = ReformerForMaskedLM.from_pretrained('google/reformer-crime-and-punishment', config=config)

输出：

True
False

【讨论】：