RuntimeError：CUDA 错误：设备端断言已触发 - 第二次调用模型时答案

【问题标题】：RuntimeError: CUDA error: device-side assert triggered - When calling a model for the second timeRuntimeError：CUDA 错误：设备端断言已触发 - 第二次调用模型时
【发布时间】：2022-09-24 00:16:27
【问题描述】：

使用 PyTorch 模型时出现以下错误：

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2197         # remove once script supports set_grad_enabled
   2198         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2199     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   2200 
   2201 

RuntimeError: CUDA error: device-side assert triggered

该错误似乎仅在我第二次调用模型时发生我的代码：

epochs =  500
losses = []
model.to(device)

for e in range(epochs):
  running_loss = 0
  current_batch = 1

  for x1, x2, y in data_loader:    
    print(\"x1 to device\")
    x3 = x1.to(device)
    print(\"--- Computing embedding1 ---\")
    embedding1 = model(x3, pooling_method=pooling_method)
    print(embedding1.size())

    print(\"x2 to device\")
    x4 = x2.to(device)
    print(\"--- Computing embedding2 ---\")
    embedding2 = model(x4, pooling_method=pooling_method)
    print(embedding2.size())

输出：

x1 to device
--- Computing embedding1 ---
torch.Size([64, 768])
x2 to device
--- Computing embedding2 ---
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-29-6b36cff704b2> in <module>
     21     x4 = x2.to(device)
     22     print(\"--- Computing embedding2 ---\")
---> 23     embedding2 = model(x4, pooling_method=pooling_method)
     24     print(embedding2.size())
     25 

8 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2197         # remove once script supports set_grad_enabled
   2198         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2199     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   2200 
   2201 

RuntimeError: CUDA error: device-side assert triggered

输入具有相同的形状，因此问题不在于形状。该错误似乎发生在模型计算输出时，但只是第二次。

该设备是：

device(type=\'cuda\', index=0)

如有必要，模型为：

class BERT(nn.Module):
    \"\"\"
    Torch model based on CamemBERT, in order to make sentence embeddings
    \"\"\"
    def __init__(self, tokenizer, model_name=model_name, output_size=100):
        super().__init__()

        self.bert = CamembertModel.from_pretrained(model_name)
        self.bert.resize_token_embeddings(len(tokenizer))

        
    def forward(self, x, pooling_method=\'cls\'):
        hidden_states = self.bert(x).last_hidden_state
        embedding = pooling(hidden_states, pooling_method=pooling_method)

        return embedding

有谁知道如何解决这个问题？

我试过了，但没有解决问题： import os os.environ[\'CUDA_LAUNCH_BLOCKING\'] = \"1\"
但它是否给您提供了更多信息的错误消息？

标签： python pytorch runtime-error

【解决方案1】：

以下两个原因会导致出现 CUDA 错误：

标签/类别数与输出单位。
损失函数的输入可能不正确。
在此处查看解决方案 --> https://builtin.com/software-engineering-perspectives/cuda-error-device-side-assert-triggered

【讨论】：