cloudpickle v1.5.0 breaks TensorFlow Probability

[matthewfeickert]

Hi. I'm just reporting that cloudpickle v1.5.0 currently breaks TensorFlow Probability v0.10.0 entirely. This is more of a heads up as you'll probably be getting some other reports.

This is currenlty detailed in the following Issues:

• pyhf Issue 913
• TensorFlow Probability Issue 991

A minimal failing example is found in pyhf Issue 913 but also reproduced here:

$ python --version
Python 3.7.5
$ python -m venv ${HOME}/debug-this
$ source ${HOME}/debug-this/bin/activate
(debug-this) $ python -m pip install --upgrade pip setuptools wheel
(debug-this) $ python -m pip install tensorflow tensorflow-probability
(debug-this) $ python -c "import tensorflow_probability"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/feickert/debug-this/lib/python3.7/site-packages/tensorflow_probability/__init__.py", line 76, in <module>
    from tensorflow_probability.python import *  # pylint: disable=wildcard-import
  File "/home/feickert/debug-this/lib/python3.7/site-packages/tensorflow_probability/python/__init__.py", line 23, in <module>
    from tensorflow_probability.python import distributions
  File "/home/feickert/debug-this/lib/python3.7/site-packages/tensorflow_probability/python/distributions/__init__.py", line 88, in <module>
    from tensorflow_probability.python.distributions.pixel_cnn import PixelCNN
  File "/home/feickert/debug-this/lib/python3.7/site-packages/tensorflow_probability/python/distributions/pixel_cnn.py", line 37, in <module>
    from tensorflow_probability.python.layers import weight_norm
  File "/home/feickert/debug-this/lib/python3.7/site-packages/tensorflow_probability/python/layers/__init__.py", line 31, in <module>
    from tensorflow_probability.python.layers.distribution_layer import CategoricalMixtureOfOneHotCategorical
  File "/home/feickert/debug-this/lib/python3.7/site-packages/tensorflow_probability/python/layers/distribution_layer.py", line 28, in <module>
    from cloudpickle.cloudpickle import CloudPickler
ImportError: cannot import name 'CloudPickler' from 'cloudpickle.cloudpickle' (/home/feickert/debug-this/lib/python3.7/site-packages/cloudpickle/cloudpickle.py)
(debug-this) $ pip list | grep cloudpickle
cloudpickle            1.5.0
(debug-this) $ pip list | grep tensor
tensorboard            2.2.2
tensorboard-plugin-wit 1.7.0
tensorflow             2.2.0
tensorflow-estimator   2.2.0
tensorflow-probability 0.10.0

This is obviously already known to the cloudpickle devs, but for anyone else, as can be seen above this fails because

from cloudpickle.cloudpickle import CloudPickler

is invalid and should be from cloudpickle.cloudpickle_fast import CloudPickler

(edit)

from cloudpickle import CloudPickler

[ogrisel]

CloudPickler should be imported from the top level package:

from cloudpickle import CloudPickler

[matthewfeickert]

Thank you for the correction!

[ogrisel]

@pierreglaser we might need to implement the hack you suggested to implement a Python 3.6 compatible __getattr__...

[pierreglaser]

Yes most probably.