带有 MultiDiscrete ActionSpace AttributeError 的健身房 RL：\'MultiDiscrete\' 对象没有属性 \'spaces\'答案

【问题标题】：gym RL with MultiDiscrete ActionSpace AttributeError: 'MultiDiscrete' object has no attribute 'spaces'带有 MultiDiscrete ActionSpace AttributeError 的健身房 RL：\'MultiDiscrete\' 对象没有属性 \'spaces\'
【发布时间】：2022-10-30 06:38:57
【问题描述】：

我正在尝试构建一个强化学习算法，它可以播放MasterMind Game。我正在使用多离散动作和观察空间。动作空间占用 4 个插槽，每个插槽 6 种颜色，观察空间为 2x4。我创建了一个自定义环境来连接我的编程游戏。由于发生错误，环境尚未准备好。也许有人可以帮我解决这个问题。

import gym as gym
from gym import Env
from gym.spaces import Discrete, Box, MultiDiscrete, Dict
from stable_baselines3.common.policies import MultiInputActorCriticPolicy

action_space = MultiDiscrete(np.array([6,6,6,6]), dtype=int)
observation_space = MultiDiscrete(np.array([4,4]), dtype=int)

...

class MasterMindEnv(Env):
    def __init__(self) -> None:
        super(MasterMindEnv, self).__init__()
        self.action_space = action_space
        self.observation_space = observation_space

    def step(self, action:np.ndarray):
        pass_action(action)
        output = get_output()
        print(output)

        reward = output[0] + output[1]
        print(reward)
        
        done = False
        info = {}

        return observation_space.sample(), 1, done, info

    def reset(self):
        return self.observation_space.sample()
        
...

model = A2C(MultiInputActorCriticPolicy, env)
model.learn(total_timesteps=1000)

错误是：

AttributeError                            Traceback (most recent call last)
c:\...\model.ipynb Zelle 10 in <module>
----> 1 model = A2C(MultiInputActorCriticPolicy, env)
      2 model.learn(total_timesteps=1000)


File c:\...\Python310\lib\site-packages\stable_baselines3\a2c\a2c.py:126, in A2C.__init__(self, policy, env, learning_rate, n_steps, gamma, gae_lambda, ent_coef, vf_coef, max_grad_norm, rms_prop_eps, use_rms_prop, use_sde, sde_sample_freq, normalize_advantage, tensorboard_log, create_eval_env, policy_kwargs, verbose, seed, device, _init_setup_model)
    123     self.policy_kwargs["optimizer_kwargs"] = dict(alpha=0.99, eps=rms_prop_eps, weight_decay=0)
    125 if _init_setup_model:
--> 126     self._setup_model()

File c:\...\Python310\lib\site-packages\stable_baselines3\common\on_policy_algorithm.py:123, in OnPolicyAlgorithm._setup_model(self)
    112 buffer_cls = DictRolloutBuffer if isinstance(self.observation_space, gym.spaces.Dict) else RolloutBuffer
    114 self.rollout_buffer = buffer_cls(
    115     self.n_steps,
    116     self.observation_space,
   (...)
    121     n_envs=self.n_envs,
    122 )
--> 123 self.policy = self.policy_class(  # pytype:disable=not-instantiable
...
--> 258 for key, subspace in observation_space.spaces.items():
    259     if is_image_space(subspace):
    260         extractors[key] = NatureCNN(subspace, features_dim=cnn_output_dim)

AttributeError: 'MultiDiscrete' object has no attribute 'spaces'

【问题讨论】：

标签： python artificial-intelligence reinforcement-learning openai-gym stable-baselines

【解决方案1】：

observation_space = MultiDiscrete(np.array([4,4]), dtype=int)
...
model = A2C(MultiInputActorCriticPolicy, env)
...
for key, subspace in observation_space.spaces.items():

MultiDiscrete 空间不需要 MultiInput。它仍然只是一个观察空间，而在提供多个观察空间时需要 MultiInput。要么不使用 MultiInput 策略，要么换行。

Stable Baselines3 supports handling of multiple inputs by using Dict Gym space. 
This can be done using MultiInputPolicy, which by default uses the 
CombinedExtractor feature extractor to turn multiple inputs into a single 
vector, handled by the net_arch network.

【讨论】：