我的 DDQN 网络是否正确实施？

Question

这是我的重播/训练函数实现。我制作了 DDQN，使model 在回放/训练期间落后于model2 1 个批量大小。通过设置self.ddqn = False，它变成了一个普通的DQN。这是否正确实施？我用这篇论文作为参考：

http://papers.nips.cc/paper/3964-double-q-learning.pdf

DDQN 代码

    def replay(self, batch_size):
        if self.ddqn:
            self.model2.load_state_dict(self.model.state_dict()) # copies model weights to model2
        minibatch = random.sample(self.memory, batch_size)
        for state, action, reward, next_state, done in minibatch:
            state = torch.Tensor(state)
            next_state = torch.Tensor(next_state)
            if self.cuda:
                state = torch.Tensor(state).cuda()
                next_state = torch.Tensor(next_state).cuda()
            Q_current = self.model(state)
            Q_target = Q_current.clone() # TODO: test copy.deepcopy() and Tensor.copy_()
            Q_next = (1-done)*self.model(next_state).cpu().detach().numpy()
            next_action = np.argmax(Q_next)
            if self.ddqn:
                Q_next = (1-done)*self.model2(next_state).cpu().detach().numpy()
            Q_target[action] = Q_current[action] + self.alpha*(reward + self.gamma*Q_next[next_action] - Q_current[action])

            self.optim.zero_grad()
            loss = self.loss(Q_current, Q_target)
            loss.backward()
            self.optim.step()

        if self.epsilon > self.epsilon_min:
            self.epsilon = max(self.epsilon*self.epsilon_decay, self.epsilon_min)

Answer 1

我建议将next_action 移到下面并使用 if-else:

if self.ddqn:
    Q_next = (1-done)*self.model2(next_state).cpu().detach().numpy()
else:
    Q_next = (1-done)*self.model(next_state).cpu().detach().numpy()
next_action = np.argmax(Q_next)

其余的看起来还可以。