RLlib: RecursionError when using MultiBinary space

[cool-RR]

What happened + What you expected to happen

The bug

I'm getting this recursion error (truncated because it's very large and repetitive:

(RolloutWorker pid=2429685)   File "/home/ramrachum/.venvs/ray_env/lib/python3.10/site-packages/ray/rllib/models/catalog.py", line 713, in get_model_v2
(RolloutWorker pid=2429685)     return wrapper(
(RolloutWorker pid=2429685)   File "/home/ramrachum/.venvs/ray_env/lib/python3.10/site-packages/ray/rllib/models/tf/complex_input_net.py", line 68, in __init__
(RolloutWorker pid=2429685)     self.cnns[i] = ModelCatalog.get_model_v2(
(RolloutWorker pid=2429685)   File "/home/ramrachum/.venvs/ray_env/lib/python3.10/site-packages/ray/rllib/models/catalog.py", line 713, in get_model_v2
(RolloutWorker pid=2429685)     return wrapper(
(RolloutWorker pid=2429685)   File "/home/ramrachum/.venvs/ray_env/lib/python3.10/site-packages/ray/rllib/models/tf/complex_input_net.py", line 50, in __init__
(RolloutWorker pid=2429685)     self.flattened_input_space = flatten_space(self.original_space)
(RolloutWorker pid=2429685)   File "/home/ramrachum/.venvs/ray_env/lib/python3.10/site-packages/ray/rllib/utils/spaces/space_utils.py", line 37, in flatten_space
(RolloutWorker pid=2429685)     _helper_flatten(space, ret)
(RolloutWorker pid=2429685)   File "/home/ramrachum/.venvs/ray_env/lib/python3.10/site-packages/ray/rllib/utils/spaces/space_utils.py", line 27, in _helper_flatten
(RolloutWorker pid=2429685)     if isinstance(space_, Tuple):
(RolloutWorker pid=2429685)   File "/usr/lib/python3.10/abc.py", line 119, in __instancecheck__
(RolloutWorker pid=2429685)     return _abc_instancecheck(cls, instance)
(RolloutWorker pid=2429685) RecursionError: maximum recursion depth exceeded in comparison

Expected behavior

I expected the code to run without error, and to train PPO on my environment.

Versions / Dependencies

Python 3.10.4

absl-py==1.2.0
aiosignal==1.2.0
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asttokens==2.0.8
astunparse==1.6.3
attrs==22.1.0
backcall==0.2.0
beautifulsoup4==4.11.1
bleach==5.0.1
cachetools==5.2.0
certifi==2022.9.14
cffi==1.15.1
charset-normalizer==2.1.1
click==8.0.4
cloudpickle==2.2.0
contourpy==1.0.5
cycler==0.11.0
debugpy==1.6.3
decorator==5.1.1
defusedxml==0.7.1
distlib==0.3.6
dm-tree==0.1.7
entrypoints==0.4
executing==1.0.0
fastjsonschema==2.16.1
filelock==3.8.0
flatbuffers==2.0.7
fonttools==4.37.2
frozenlist==1.3.1
gast==0.4.0
google-auth==2.11.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.43.0
gym==0.23.1
gym-notices==0.0.8
h5py==3.7.0
idna==3.4
imageio==2.21.3
ipykernel==6.15.3
ipython==8.5.0
ipython-genutils==0.2.0
ipywidgets==8.0.2
jedi==0.18.1
Jinja2==3.1.2
jsonschema==4.16.0
jupyter==1.0.0
jupyter-console==6.4.4
jupyter-core==4.11.1
jupyter_client==7.3.5
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.3
keras==2.10.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.4
libclang==14.0.6
lxml==4.9.1
lz4==4.0.2
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib==3.6.0
matplotlib-inline==0.1.6
mistune==2.0.4
msgpack==1.0.4
nbclient==0.6.8
nbconvert==7.0.0
nbformat==5.5.0
nest-asyncio==1.5.5
networkx==2.8.6
notebook==6.4.12
numpy==1.23.3
oauthlib==3.2.1
opt-einsum==3.3.0
packaging==21.3
pandas==1.4.4
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.2.0
platformdirs==2.5.2
prometheus-client==0.14.1
prompt-toolkit==3.0.31
protobuf==3.19.5
psutil==5.9.2
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
Pygments==2.13.0
pyparsing==3.0.9
pyrsistent==0.18.1
python-dateutil==2.8.2
pytz==2022.2.1
PyWavelets==1.4.0
PyYAML==6.0
pyzmq==24.0.0
qtconsole==5.3.2
QtPy==2.2.0
ray==2.0.0
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.9
scikit-image==0.19.3
scipy==1.9.1
Send2Trash==1.8.0
six==1.16.0
soupsieve==2.3.2.post1
stack-data==0.5.0
tabulate==0.8.10
tensorboard==2.10.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorboardX==2.5.1
tensorflow==2.10.0
tensorflow-estimator==2.10.0
tensorflow-io-gcs-filesystem==0.27.0
termcolor==2.0.1
terminado==0.15.0
tifffile==2022.8.12
tinycss2==1.1.1
tornado==6.2
traitlets==5.4.0
typing_extensions==4.3.0
urllib3==1.26.12
virtualenv==20.16.5
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==2.2.2
widgetsnbextension==4.0.3
wrapt==1.14.1

Reproduction script

from ray.rllib.agents.ppo import PPOTrainer
from ray.rllib.env.multi_agent_env import MultiAgentEnv
from gym.spaces import Discrete, MultiBinary
import numpy as np


class ProblemEnv(MultiAgentEnv):
    def __init__(self, config=None):
        self.timestep_limit = 100
        self.observation_space = MultiBinary([3, 10, 10])
        self.action_space = Discrete(4)
        self.reset()

    def reset(self):
        return np.zeros((3, 10, 10))

    def step(self, action: dict):
        # I removed this code to make the example as short as possible. 
        raise NotImplementedError


env = ProblemEnv()

ppo_trainer = PPOTrainer(
    config={
    'env': ProblemEnv,
    'create_env_on_driver': True,
    'multiagent': {
        'policies': {
            'policy1': (None, env.observation_space, env.action_space, {}),
            'policy2': (None, env.observation_space, env.action_space, {}),
        },
        'policy_mapping_fn': lambda agent_id: 'policy1' if agent_id == 'agent1' else 'policy2',
    },
}
)
results = ppo_trainer.train()

Issue Severity

High: It blocks me from completing my task.

[PaulFenton]

I'm able to reproduce this with the provided script on ray 3.0.0.dev0, python 3.8.10.