【发布时间】：2021-12-28 13:57:53
【问题描述】：
我正在使用 Pytorch GRU 在 文本分类 任务（输出维度为 5）上训练模型。我的网络是按照下面的代码实现的。
class GRU(nn.Module):
def __init__(self, model_param: ModelParam):
super(GRU, self).__init__()
self.embedding = nn.Embedding(model_param.vocab_size, model_param.embed_dim)
# Build with pre-trained embedding vectors, if given.
if model_param.vocab_embedding is not None:
self.embedding.weight.data.copy_(model_param.vocab_embedding)
self.embedding.weight.requires_grad = False
self.rnn = nn.GRU(model_param.embed_dim,
model_param.hidden_dim,
num_layers=2,
bias=True,
batch_first=True,
dropout=0.5,
bidirectional=False)
self.dropout = nn.Dropout(0.5)
self.fc = nn.Sequential(
nn.Linear(in_features=model_param.hidden_dim, out_features=128),
nn.Linear(in_features=128, out_features=model_param.output_dim)
)
def forward(self, x, labels=None):
'''
:param x: torch.tensor, of shape [batch_size, max_seq_len].
:param labels: torch.tensor, of shape [batch_size]. Not used in this model.
:return outputs: torch.tensor, of shape [batch_size, output_dim].
'''
# [batch_size, max_seq_len, embed_dim].
features = self.dropout(self.embedding(x))
# [batch_size, max_seq_len, hidden_dim].
outputs, _ = self.rnn(features)
# [batch_size, hidden_dim].
outputs = outputs[:, -1, :]
return self.fc(self.dropout(outputs))
我将 nn.CrossEntropyLoss() 用于损失函数，将 optim.SGD 用于优化器。损失函数和优化器的定义如下。
# Loss function and optimizer.
loss_func = nn.CrossEntropyLoss()
optimizer = SGD(model.parameters(), lr=learning_rate, weight_decay=0.9)
而我的训练过程大致如下所示。
for batch in train_iter:
optimizer.zero_grad()
# The prediction of model, and its corresponding loss.
prediction = model(batch.text.type(torch.LongTensor).to(device), batch.label.to(device))
loss = loss_func(prediction, batch.label.to(device))
loss.backward()
optimizer.step()
# Record total loss.
epoch_losses.append(loss.item() / batch_size)
当我训练这个模型时，验证准确度和损失是这样报告的。
Epoch 1/300 valid acc: [0.839] (16668 in 19873), time spent 631.497 sec. Validate loss 1.506138. Best validate epoch is 1.
Epoch 2/300 valid acc: [0.839] (16668 in 19873), time spent 627.631 sec. Validate loss 1.577007. Best validate epoch is 2.
Epoch 3/300 valid acc: [0.839] (16668 in 19873), time spent 631.427 sec. Validate loss 1.580756. Best validate epoch is 3.
Epoch 4/300 valid acc: [0.839] (16668 in 19873), time spent 605.352 sec. Validate loss 1.581306. Best validate epoch is 4.
Epoch 5/300 valid acc: [0.839] (16668 in 19873), time spent 388.487 sec. Validate loss 1.581431. Best validate epoch is 5.
Epoch 6/300 valid acc: [0.839] (16668 in 19873), time spent 360.344 sec. Validate loss 1.581464. Best validate epoch is 6.
Epoch 7/300 valid acc: [0.839] (16668 in 19873), time spent 624.345 sec. Validate loss 1.581473. Best validate epoch is 7.
Epoch 8/300 valid acc: [0.839] (16668 in 19873), time spent 622.059 sec. Validate loss 1.581477. Best validate epoch is 8.
Epoch 9/300 valid acc: [0.839] (16668 in 19873), time spent 651.425 sec. Validate loss 1.581478. Best validate epoch is 9.
Epoch 10/300 valid acc: [0.839] (16668 in 19873), time spent 697.475 sec. Validate loss 1.581478. Best validate epoch is 10.
...
这表明验证损失在 epoch 9 之后没有减少，并且验证准确度自第一个 epoch 以来保持不变（请注意，在我的数据集中，其中一个标签占 83%，由此可以推断我的模型倾向于将所有序列预测为相同的标签，但是当我在另一个相对不平衡的数据集上训练时也会发生这种情况）。有没有人遇到过这种情况B4？我想知道我在设计模型或训练程序时是否犯了错误。谢谢你的帮助XD。
11 月 19 日更新，我添加了一个图表，显示了损失在训练时的表现。从这个图中可以看出，在第 5 个 epoch 之后，训练损失和验证损失都变成了常数。 training and validating loss in 20 epochs
【问题讨论】：
-
您的训练损失表现如何？你能在训练步骤上展示它的情节吗？
-
@Ivan 抱歉我的回复晚了。上面显示了训练损失的行为方式。如图所示，在第 5 个 epoch 之后，训练损失和验证损失都保持不变。
标签： python machine-learning neural-network pytorch recurrent-neural-network