强化学习：用动态规划解决 MoutainCar 问题

Question

我目前正在学习强化学习，在阅读 Richard S. Sutton 的书时，我问自己如何将所有这些算法应用于问题。我已经为 GYM 的 MoutainCar 问题编写了 Q-learning 算法。因此，我目前正在尝试使用动态规划来解决同样的问题。实际上，我正在努力处理政策评估部分。

这是我的部分实现：

import gym
import numpy as np
import matplotlib.pyplot as plt
import time



gamma = 0.5

env = gym.make("MountainCar-v0")

discrete_obs_space_size = [20] *len(env.observation_space.high) # [20,20] -> 20 separations for each observations
discrete_obs_range_step = (env.observation_space.high-env.observation_space.low)/discrete_obs_space_size

def moutainCar(policy):
    
    discrete_state = get_discrete_state(env.reset())
    done = False
    
    while not done : 
    
        action = policy[discrete_state[0], discrete_state[1]]
        new_state, reward, done, _ = env.step(action) # observation, reward, terminated
        new_discrete_state = get_discrete_state(new_state)
        env.render(mode='rgb_array')
            
        if new_state[0] >= env.goal_position :
            print(f"Task is achieved on episode {episode} !")
            break
            
        discrete_state = new_discrete_state 

def get_discrete_state(state) :
    discrete_state = (state-env.observation_space.low)/discrete_obs_range_step
    return tuple(discrete_state.astype(np.int32))

def get_continuous_state(discrete_state) :
    state = env.observation_space.low + discrete_state*discrete_obs_range_step
    return tuple(state.astype(np.float64))

def transition_dynamics(action, xt, vt):
    force = 0.001
    gravity = 0.0025
    vt1 = max(min(vt + (action-1)*force - np.cos(3*xt)*gravity, env.observation_space.high[1]), env.observation_space.low[1])
    xt1 = max(min(xt+vt, env.observation_space.high[0]), env.observation_space.low[0])
    
    return (xt1, vt1)

def transition_probabilities():
    
    states_to_states_prime = {}
    for i in range(discrete_obs_space_size[0]):
        for j in range(discrete_obs_space_size[1]): # For Loops : (i,j) = state_ij
            for k in range(env.action_space.n):
                xt, vt = get_continuous_state((i,j))
                new_state = get_discrete_state(transition_dynamics(k, xt, vt))
                states_to_states_prime[(i,j,new_state[0], new_state[1], k)] = 1.
                #1/(discrete_obs_space_size[0]**2)
                
    return states_to_states_prime

我构建了几个不同的函数：两个从离散观察空间到连续观察空间（我不知道 int 操作的舍入是否会导致一些问题，因为我丢失了其中的信息）。我提供了动态规划所必需的过渡动力学。我尝试了不同的转移概率，你可以看到我将值 1 设置为从状态 s 到状态 s'，但我想我应该除以观察空间的大小，以便所有概率总和为 1。

这是我的政策评估代码：

def policy_evaluation(policy, theta = 0.01):
    V = np.zeros((discrete_obs_space_size[0], discrete_obs_space_size[1]), dtype=int)
    delta = 0
    
    # Transitions Probabilities
    tr_prob = transition_probabilities()
    
    while True :
        for i in range(discrete_obs_space_size[0]):
            for j in range(discrete_obs_space_size[1]): # For Loops on state
                v = V[i,j]
                tmp = 0.
                for i_prime in range(discrete_obs_space_size[0]):
                    for j_prime in range(discrete_obs_space_size[1]): # For Loop on state prime
                        try :
                            tr = tr_prob[(i, j, i_prime, j_prime, policy[i,j])]
                        except :
                            tr = 0.
                            
                        if (i_prime == 19) and (j_prime == 19):
                            reward = 0
                        else:
                            reward = -1
                            
                        tmp +=  tr*(reward+gamma*V[i_prime,j_prime])
                        
                V[i,j] = tmp
                
                delta = max(delta, np.abs(v - V[i,j]))
                print(delta)
        if delta < theta : 
            break
                
                
    return V

我尝试了不同的方式，但这是我感到受阻的地方。问题是增量（给定状态 s 中 V 的新旧值之差）趋于 1 并且不再更新，我猜这意味着 V 也不再更新。我不明白我做错了什么，有人可以帮助我吗？

先感谢您 ！

Answer 1

https://www.amazon.com/dp/B0BS51F7Q5

这本书对强化学习有很大帮助！