[Air] Missing pyarrow module causes import issues

[pclay614]

What happened + What you expected to happen

Importing tune from ray should work and not fail.

This is due to the inclusion of this line, which assumes pyarrow imported successfully.

Versions / Dependencies

python: 3.8.10
os: Ubuntu 20.04.5
ray: 2.6.0

Reproduction script

1. Create a virtualenv
2. pip install ray==2.6.0
3. Start a python3 interactive interpreter
4. import ray (works)
5. from ray import tune (fails)

Python 3.8.10 (default, May 26 2023, 14:05:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
>>> from ray import tune
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/me/.virtualenvs/test_ray/lib/python3.8/site-packages/ray/tune/__init__.py", line 2, in <module>
    from ray.tune.tune import run_experiments, run
  File "/home/me/.virtualenvs/test_ray/lib/python3.8/site-packages/ray/tune/tune.py", line 27, in <module>
    from ray.air import CheckpointConfig
  File "/home/me/.virtualenvs/test_ray/lib/python3.8/site-packages/ray/air/__init__.py", line 1, in <module>
    from ray.air.checkpoint import Checkpoint
  File "/home/me/.virtualenvs/test_ray/lib/python3.8/site-packages/ray/air/checkpoint.py", line 22, in <module>
    from ray.air._internal.remote_storage import (
  File "/home/me/.virtualenvs/test_ray/lib/python3.8/site-packages/ray/air/_internal/remote_storage.py", line 142, in <module>
    _cached_fs: Dict[tuple, Tuple[float, pyarrow.fs.FileSystem]] = {}
AttributeError: 'NoneType' object has no attribute 'fs'

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[pclay614]

Suggestion. It seems that the import is being allowed to fail, so it should be allowed to fail gracefully.

try:
    import pyarrow
    import pyarrow.fs

except (ImportError, ModuleNotFoundError):
    pyarrow_fs = type("", (), {"FileSystem": None})
    pyarrow = type("", (), {"fs": pyarrow_fs})

[richardliaw]

cc @matthewdeng @pcmoritz @justinvyu we probably want to include a warning message that informs user that they should rerun pip install ray[tune] or ray[air].

[matthewdeng]

Thanks for reporting, I've put up a fix for this here: #37670.