Pytorch BERT：形状错误的输入答案

【问题标题】：Pytorch BERT: Misshaped inputsPytorch BERT：形状错误的输入
【发布时间】：2021-01-10 14:43:48
【问题描述】：

我遇到了在大型输入序列上评估拥抱脸的 BERT 模型（“bert-base-uncased”）的问题。

model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
token_ids = [101, 1014, 1016, ...] # len(token_ids) == 33286
token_tensors = torch.tensor([token_ids]) # shape == [1, 33286]
segment_tensors = torch.tensor([[1] * len(token_ids)]) # shape == [1, 33286]
model(token_tensors, segment_tensors)

Traceback
self.model(token_tensors, segment_tensors)
  File "/home/.../python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/.../python3.8/site-packages/transformers/modeling_bert.py", line 824, in forward
    embedding_output = self.embeddings(
  File "/home/.../python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/.../python3.8/site-packages/transformers/modeling_bert.py", line 211, in forward
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: The size of tensor a (33286) must match the size of tensor b (512) at non-singleton dimension 1

我注意到model.embeddings.positional_embeddings.weight.shape == (512, 768)。 IE。当我将输入大小限制为model(token_tensors[:, :10], segment_tensors[:, :10]) 时，它可以工作。我误解了 token_tensors 和 segment_tensors 应该如何塑造。我认为它们的大小应该是(batch_size, sequence_length)

感谢您的帮助

【问题讨论】：

标签： python pytorch huggingface-transformers

【解决方案1】：

我刚刚发现来自 huggingface 的预训练 BERT 模型的最大输入长度为 512 (https://github.com/huggingface/transformers/issues/225)

【讨论】：