cwltool fails to install in python3.7

[ThomasHickman]

$ docker run -it python:3.7 /bin/bash
root@0b95bcccdca4:/# pip install cwltool
Collecting cwltool
  Downloading https://files.pythonhosted.org/packages/a4/4c/2e4034c9dc7f32bd0a4bf86cbf0dd9eeff4e10c1611e4d92aad73b28d5c5/cwltool-1.0.20180711112827-py2.py3-none-any.whl (666kB)
    100% |████████████████████████████████| 675kB 11.3MB/s
Collecting bagit>=1.6.4 (from cwltool)
  Downloading https://files.pythonhosted.org/packages/bd/d6/7f505929284541705c6acf83726eb5c74b092e7eaed3a166bc428215bb89/bagit-1.7.0-py2.py3-none-any.whl
Collecting six>=1.8.0 (from cwltool)
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting typing>=3.5.3 (from cwltool)
  Downloading https://files.pythonhosted.org/packages/05/2b/2b05bf1d5a9dd450447c9a5df3e118a465e5d3cb12b73b7220a5064a403f/typing-3.6.4-py3-none-any.whl
Collecting schema-salad<3,>=2.6.20170927145003 (from cwltool)
  Downloading https://files.pythonhosted.org/packages/37/eb/9ed3748d0cc19acb4ac4d17893a647d70103cb37668aa5182fe2ee3f54ba/schema_salad-2.7.20180611133406-py2.py3-none-any.whl (380kB)
    100% |████████████████████████████████| 389kB 13.4MB/s
Collecting subprocess32>=3.5.0; os_name == "posix" (from cwltool)
  Downloading https://files.pythonhosted.org/packages/c3/5f/7117737fc7114061837a4f51670d863dd7f7f9c762a6546fa8a0dcfe61c8/subprocess32-3.5.2.tar.gz (96kB)
    100% |████████████████████████████████| 102kB 6.0MB/s
Collecting shellescape<3.5,>=3.4.1 (from cwltool)
  Downloading https://files.pythonhosted.org/packages/51/b6/986c99a10040beaaefca1ad6c93bd7738cb8e4f52f6caed13d3ed1caa7e4/shellescape-3.4.1-py2.py3-none-any.whl
Collecting requests>=2.4.3 (from cwltool)
  Downloading https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 9.1MB/s
Collecting ruamel.yaml<0.15,>=0.12.4 (from cwltool)
  Downloading https://files.pythonhosted.org/packages/cf/86/7ef6c24317e20e6cef0eab407520d49f3d3836e454d5377bdcbcf03d385c/ruamel.yaml-0.14.12.tar.gz (242kB)
    100% |████████████████████████████████| 245kB 12.3MB/s
    Complete output from command python setup.py egg_info:
    Warning: 'keywords' should be a list, got type 'NoneType'
    sys.argv ['-c', 'egg_info', '--egg-base', 'pip-egg-info']
    test compiling test_ruamel_yaml
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-j620vg4m/ruamel.yaml/setup.py", line 858, in <module>
        main()
      File "/tmp/pip-install-j620vg4m/ruamel.yaml/setup.py", line 847, in main
        setup(**kw)
      File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/usr/local/lib/python3.7/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/local/lib/python3.7/site-packages/setuptools/dist.py", line 370, in __init__
        k: v for k, v in attrs.items()
      File "/usr/local/lib/python3.7/distutils/dist.py", line 267, in __init__
        getattr(self.metadata, "set_" + key)(val)
      File "/usr/local/lib/python3.7/distutils/dist.py", line 1203, in set_keywords
        self.keywords = _ensure_list(value, 'keywords')
      File "/usr/local/lib/python3.7/distutils/dist.py", line 40, in _ensure_list
        value = list(value)
    TypeError: 'NoneType' object is not iterable

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-j620vg4m/ruamel.yaml/

[tetron]

It is a ruamel.yaml problem, which isn't a complete surprise because it has an eccentric setup.py script.

I suspect a newer ruamel.yaml would work, but it has a different API. So supporting multiple ruamel.yaml versions is a bit of extra work:

https://yaml.readthedocs.io/en/latest/api.html#transparent-usage-of-new-and-old-api

[ThomasHickman]

What is the reasoning behind using an older version of ruamel.yaml?

[tetron]

Just as I said. It has a new API and we haven't adapted to it. I think as a secondary issue, it has come up in packaging (eg bioconda) that a certain version of ruamel.yaml is available, and we want to use the already-packaged version and not force it to use a newer one.

[mr-c]

From http://yaml.readthedocs.io/en/latest/basicuse.html

This is the new (0.15+) interface for ruamel.yaml, it is still in the process of being fleshed out. Please pin your dependency to ruamel.yaml<0.15 for production software.

[tetron]

There are something like 43 releases in the ruamel.yaml 0.15 series so it is probably getting to be reasonably stable, but someone still needs to do the work for schema-salad and cwltool to support both pre-0.15 and 0.15+ APIs.

[tetron]

(another thought, if the round trip loader metadata changed, i.e. the way it records line numbers and columns for document fragments, that would also be a huge challenge)

[ThomasHickman]

It looks like the newer versions of ruamel.yaml contains the old api:

$ python
Python 3.7.0 (default, Jun 29 2018, 20:14:27)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ruamel.yaml as yaml
>>> yaml.__version__
'0.15.44'
>>> yaml.safe_dump
<function safe_dump at 0x1055e16a8>
>>>

I've tried out all the conformance tests using ruamel.yaml versions 15.0 and 0.15.44 and in both instances all the conformance tests pass, so it should be safe to change the dependency to ruamel.yaml >= 0.12.4 (and maybe ':python_version >= "3.7"': ['ruamel.yaml >= 0.15.37'] as well)