为什么我的值字典需要浅拷贝才能正确更新？

Question

我正在编写 Python 2.7.11 中的 Agent 类，它使用 Markov Decision Process (MDP) 在 GridWorld 中搜索最优策略 π。我正在使用以下贝尔曼方程为所有GridWorld 状态的 100 次迭代实现基本值迭代：

• T(s,a,s')是从当前状态ss'的概率函数/strong> 采取行动a。
• R(s,a,s')是从s过渡到s'的奖励。李>
• γ (gamma) 是折扣因子，其中 0 ≤ γ≤ 1.
• V_k(s') 是一个递归调用，一旦达到 s' 就重复计算。
• V_k+1(s) 表示在足够的 k 次迭代发生后， V_k 迭代值会收敛，变得等价于V_k+1

这个方程是从一个 Q 值函数的最大值推导出来的，这是我在我的程序中使用的：

在构造我的Agent时，传递了一个MDP，这是一个包含以下方法的抽象类：

# Returns all states in the GridWorld
def getStates()

# Returns all legal actions the agent can take given the current state
def getPossibleActions(state)

# Returns all possible successor states to transition to from the current state 
# given an action, and the probability of reaching each with that action
def getTransitionStatesAndProbs(state, action)

# Returns the reward of going from the current state to the successor state
def getReward(state, action, nextState)

我的Agent 也传递了折扣因子，并进行了多次迭代。我还使用dictionary 来跟踪我的价值观。这是我的代码：

class IterationAgent:

    def __init__(self, mdp, discount = 0.9, iterations = 100):
        self.mdp = mdp
        self.discount = discount
        self.iterations = iterations
        self.values = util.Counter() # A Counter is a dictionary with default 0

        for transition in range(0, self.iterations, 1):
            states = self.mdp.getStates()
            valuesCopy = self.values.copy()
            for state in states:
                legalMoves = self.mdp.getPossibleActions(state)
                convergedValue = 0
                for move in legalMoves:
                    value = self.computeQValueFromValues(state, move)
                    if convergedValue <= value or convergedValue == 0:
                        convergedValue = value

                valuesCopy.update({state: convergedValue})

            self.values = valuesCopy

    def computeQValueFromValues(self, state, action):
        successors = self.mdp.getTransitionStatesAndProbs(state, action)
        reward = self.mdp.getReward(state, action, successors)
        qValue = 0
        for successor, probability in successors:
            # The Q value equation: Q*(a,s) = T(s,a,s')[R(s,a,s') + gamma(V*(s'))]
            qValue += probability * (reward + (self.discount * self.values[successor]))
        return qValue

这个实现是正确的，虽然我不确定为什么我需要valuesCopy 来成功更新我的self.values 字典。我尝试了以下方法来省略复制，但它不起作用，因为它返回的值稍微不正确：

for i in range(0, self.iterations, 1):
    states = self.mdp.getStates()
    for state in states:
        legalMoves = self.mdp.getPossibleActions(state)
        convergedValue = 0
        for move in legalMoves:
            value = self.computeQValueFromValues(state, move)
            if convergedValue <= value or convergedValue == 0:
                convergedValue = value

        self.values.update({state: convergedValue})

我的问题是，当valuesCopy = self.values.copy() 每次迭代都复制字典时，为什么要包含我的self.values 字典的副本才能正确更新我的值？不应该在同一更新中更新原始结果中的值吗？

Answer 1

拥有或不拥有副本在算法上有所不同：

# You update your copy here, so the original will be used unchanged, which is not the 
# case if you don't have the copy
valuesCopy.update({state: convergedValue})

# If you have the copy, you'll be using the old value stored in self.value here, 
# not the updated one
qValue += probability * (reward + (self.discount * self.values[successor]))