Ray dependency conflict (typing) with Python 3.7

[jasonnance]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): OS X Mojave / Linux Debian Stretch in Docker
• Ray installed from (source or binary): Binary
• Ray version: 0.7.1
• Python version: 3.7.1
• Exact command to reproduce:

import ray

@ray.remote
class T:
    def test(self):
        print('test')

ray.init()
t = T.remote()
t.test.remote()

Describe the problem

Ray version 0.7.1 depends on the third-party typing module, which conflicts with the first-party typing module in Python 3.7. See: python/typing#573 Manually uninstalling the third-party typing module solves the problem.

I see you've removed the dependency in the master branch -- if it's possible to apply the fix in python/typing#573 (comment) to the stable version, that would be much appreciated.

Source code / logs

The above code generates this traceback:

2019-06-24 13:44:45,876	ERROR worker.py:1668 -- Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 920, in _process_task
    self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 831, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 383, in put_object
    self.store_and_register(object_id, value)
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 317, in store_and_register
    self.task_driver_id))
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 246, in get_serialization_context
    _initialize_serialization(driver_id)
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 1141, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/usr/local/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 350, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/usr/local/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 345, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/usr/local/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 148, in _register_custom_pandas_handlers
    import pandas as pd
  File "/usr/local/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/usr/local/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/usr/local/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/usr/local/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in <module>
    ray.worker.global_worker.main_loop()
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 1042, in main_loop
    self._wait_for_and_process_task(task)
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 993, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 924, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 933, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 831, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 383, in put_object
    self.store_and_register(object_id, value)
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 317, in store_and_register
    self.task_driver_id))
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 246, in get_serialization_context
    _initialize_serialization(driver_id)
  File "/usr/local/lib/python3.7/site-packages/ray/worker.py", line 1141, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/usr/local/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 350, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/usr/local/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 345, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/usr/local/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 148, in _register_custom_pandas_handlers
    import pandas as pd
  File "/usr/local/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/usr/local/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/usr/local/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/usr/local/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".
  
2019-06-24 13:44:45,881	ERROR worker.py:1668 -- A worker died or was killed while executing task 14a3d74ee6146bfc1668169743eff6ba.

[jasonnance]

Looks like this is fixed in 0.7.2. Thanks!