Tox 3.1 regression, ConfigError: substitution key not found

[peterjc]

Thanks for submitting an issue!

If submitting a BUG please provide:

• Minimal reproducible example or detailed description, assign "bug"
• OS and pip list output

Reproducible, testing on Linux CentOS with Python via conda:

$ which python
/mnt/shared/scratch/xxx/apps/conda/bin/python
$ python --version
Python 2.7.15
$ python -m pip list
Package                       Version    
----------------------------- -----------
absl-py                       0.1.10     
argh                          0.26.1     
asn1crypto                    0.24.0     
astor                         0.6.2      
avro                          1.8.0      
backports.functools-lru-cache 1.5        
backports.weakref             1.0rc1     
biopython                     1.68       
bleach                        1.5.0      
certifi                       2018.4.16  
cffi                          1.11.5     
chardet                       3.0.4      
codecov                       2.0.15     
conda                         4.5.5      
ConsensusCore                 1.0.2      
coverage                      4.5.1      
cryptography                  2.2.1      
cycler                        0.10.0     
Cython                        0.28.2     
decorator                     4.3.0      
enum34                        1.1.6      
funcsigs                      1.0.2      
functools32                   3.2.3.post2
future                        0.16.0     
futures                       3.2.0      
gast                          0.2.0      
GenomicConsensus              1.1.0      
grpcio                        1.11.0     
h5py                          2.7.0      
html5lib                      0.9999999  
idna                          2.6        
ipaddress                     1.0.22     
iso8601                       0.1.12     
jsonschema                    2.6.0      
Keras                         2.1.5      
kiwisolver                    1.0.1      
Mako                          1.0.7      
Markdown                      2.6.11     
MarkupSafe                    1.0        
matplotlib                    2.2.2      
mkl-fft                       1.0.2      
mkl-random                    1.0.1      
mmtf-python                   1.0.2      
mock                          2.0.0      
msgpack                       0.5.6      
msgpack-python                0.5.6      
ncbi-genome-download          0.2.6      
networkx                      1.11       
numpy                         1.14.3     
olefile                       0.45.1     
packaging                     17.1       
pandas                        0.22.0     
pathtools                     0.1.2      
pbcommand                     0.3.29     
pbcore                        1.2.10     
pbr                           4.0.2      
Pillow                        5.0.0      
pip                           10.0.1     
pluggy                        0.6.0      
protobuf                      3.5.2      
py                            1.5.4      
pycosat                       0.6.3      
pycparser                     2.18       
pygpu                         0.7.5      
pyOpenSSL                     17.5.0     
pyparsing                     2.2.0      
pypeflow                      2.0.2      
pysam                         0.14.1     
PySocks                       1.6.8      
python-dateutil               2.3        
pytz                          2017.3     
PyYAML                        3.12       
reportlab                     3.4.0      
requests                      2.13.0     
ruamel-yaml                   0.15.35    
scipy                         1.1.0      
setuptools                    39.1.0     
six                           1.11.0     
subprocess32                  3.5.1      
tensorboard                   1.6.0      
tensorflow                    1.6.0      
termcolor                     1.1.0      
Theano                        1.0.1      
tox                           3.0.0      
urllib3                       1.12       
virtualenv                    16.0.0     
watchdog                      0.8.3      
webencodings                  0.5        
Werkzeug                      0.14.1     
wheel                         0.31.0     
xmlbuilder                    1.0 

Creating test case:

$ cd /tmp
$ mkdir tox-regression
$ cd tox-regression
$ curl -L -o tox.ini https://github.com/biopython/biopython/raw/biopython-172/.travis-tox.ini

Using tox v3.0.0 works,

$ python -m pip install tox==3.0.0
...
$ python -m tox --version
3.0.0 imported from /mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/__init__.pyc
$ rm -rf .tox
$ python -m tox -e py27-nocov --notest
py27-nocov create: /tmp/tox-regression/.tox/py27-nocov
py27-nocov installdeps: unittest2, mysql-python, mmtf-python, reportlab, psycopg2-binary, mysql-connector-python-rf, rdflib, numpy, networkx
py27-nocov installed: aenum==1.4.5,args==0.1.0,avro==1.8.1,awscli==1.11.41,backports.lzma==0.0.1,beautifulsoup4==4.5.1,bioblend==0.10.0,biopython==1.71,boto==2.38.0,botocore==1.5.4,CacheControl==0.11.7,click==4.0,clint==0.5.1,colorama==0.3.7,configparser==3.5.0,cwltool==1.0.20170828135420,decorator==4.3.0,docutils==0.12,enum34==1.1.6,ephemeris==0.4.0,epydoc==3.0.1,falcon-kit==1.0+git.f20989451b7e6826bcedf8729ad4b56237bc8b83,flake8==3.2.1,flake8-import-order==0.11,funcsigs==0.4,futures==3.0.5,galaxy-lib==17.9.10,GenomeTools==0.1,gff3toembl==1.1.2,glob2==0.4.1,gxformat2==0.1.1,html5lib==1.0b3,isodate==0.5.4,Jinja2==2.7.3,jmespath==0.9.0,keepalive==0.5,khmer==0.4,linecache2==1.0.0,lockfile==0.12.2,lxml==3.6.4,MarkupSafe==0.23,matplotlib==1.1.1,mccabe==0.5.2,mistune==0.7.3,mmtf-python==1.0.5,mock==1.3.0,msgpack-python==0.4.8,mysql-connector-python==2.0.4,mysql-connector-python-rf==2.2.2,MySQL-python==1.2.5,networkx==2.1,nose==1.2.1,numpy==1.14.2,olefile==0.44,pandas==0.22.0,pbr==1.8.1,pep8==1.7.0,Pillow==4.1.0,pkginfo==1.3.2,planemo==0.48.0.dev0,pluggy==0.3.1,poster==0.8.1,psycopg2-binary==2.7.5,py==1.4.30,pyaml==16.11.4,pyani==0.1.3.2,pyasn1==0.1.9,pycodestyle==2.2.0,pydablooms==0.8.2,pyflakes==1.3.0,PyGithub==1.25.2,pyparsing==2.1.5,pytest==3.0.5,python-dateutil==2.5.1,pytz==2016.3,PyYAML==3.11,rdflib==4.2.2,rdflib-jsonld==0.4.0,reportlab==3.4.0,requests==2.10.0,requests-toolbelt==0.7.0,rpy2==2.7.0,rsa==3.4.2,ruamel.ordereddict==0.4.9,ruamel.yaml==0.14.8,s3transfer==0.1.10,schema-salad==2.6.20171201034858,scipy==0.17.1,screed==0.7,seaborn==0.7.0,shellescape==3.4.1,singledispatch==3.4.0.3,six==1.10.0,SPARQLWrapper==1.7.6,tox==2.1.1,traceback2==1.4.0,twine==1.8.1,typing==3.6.4,unittest2==1.1.0,virtualenv==13.1.2
__________________________________________________________________ summary __________________________________________________________________
  py27-nocov: skipped tests
  congratulations :)

Now trying with the latest tox,

$ python -m pip install tox==3.1.1
...
$ python -m tox --version
3.1.1 imported from /mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/__init__.pyc
$ rm -rf .tox
$ python -m tox -e py27-nocov --notest
Traceback (most recent call last):
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/__main__.py", line 4, in <module>
    tox.cmdline()
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/session.py", line 39, in cmdline
    main(args)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/session.py", line 44, in main
    config = prepare(args)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/session.py", line 26, in prepare
    config = parseconfig(args)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 246, in parseconfig
    parseini(config, inipath)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 975, in __init__
    config.envconfigs[name] = self.make_envconfig(name, section, reader._subs, config)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1012, in make_envconfig
    res = reader.getlist(env_attr.name, sep="\n")
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1156, in getlist
    s = self.getstring(name, None)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1218, in getstring
    x = self._replace_if_needed(x, name, replace, crossonly)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1261, in _replace_if_needed
    x = self._replace(x, name=name, crossonly=crossonly)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1248, in _replace
    replaced = Replacer(self, crossonly=crossonly).do_replace(value)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1289, in do_replace
    expanded = substitute_once(value)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1287, in substitute_once
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1330, in _replace_match
    return self._replace_substitution(match)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1365, in _replace_substitution
    val = self._substitute_from_other_section(sub_key)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1359, in _substitute_from_other_section
    raise tox.exception.ConfigError("substitution key {!r} not found".format(key))
tox.exception.ConfigError: ConfigError: substitution key 'py27' not found

Interestingly trying to use the other Python versions, e.g. py36, also gives this warning about py27,

$ python -m tox -e py36-nocov --notest
Traceback (most recent call last):
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
...
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1359, in _substitute_from_other_section
    raise tox.exception.ConfigError("substitution key {!r} not found".format(key))
tox.exception.ConfigError: ConfigError: substitution key 'py27' not found

Note there is no explicit usage of baseversion in this configuration file, which starts:

[tox]
minversion = 2.0
skipsdist = True
envlist =
    style
    sdist
    bdist_wheel
    {py27,py34,py35,py36,pypy,pypy3}-cover
    {py27,py34,py35,py36,pypy,pypy3}-nocov

[testenv]
# TODO: Try tox default sdist based install instead:
skip_install = True
sitepackages = True
passenv =
    CI
    TRAVIS
    TRAVIS_*
    TOXENV
    CODECOV_*
whitelist_externals =
    bash
    echo
# Want to avoid overhead of compiling numpy or scipy:
# (But must compile numpy for PyPy right now)
install_command = pip install --only-binary=scipy {opts} {packages}
deps =
    #Lines startings xxx: are filtered by the environment.
    #Leaving py34 without any soft dependencies (just numpy)
    cover: coverage
    cover: codecov
    {py27}: unittest2
    {py27}: mysql-python
    {py27,py36}: mmtf-python
    {py27,py35}: reportlab
    {py27,py34,py35,py36}: psycopg2-binary
    {py27,py34,py35,py35}: mysql-connector-python-rf
    {py27,py35,pypy}: rdflib
    {pypy,pypy3}: numpy==1.12.1
    {py27,py34,py36}: numpy
    {py36}: scipy
    {py27}: networkx
    {py36}: matplotlib
commands =
    #The bash call is a work around for special characters
    #The /dev/null is to hide the verbose output but leave warnings
    bash -c \'python setup.py install > /dev/null\'
    #The bash call is a work around for the cd command
    nocov: bash -c \'cd Tests && python run_tests.py --offline\'
    #See https://codecov.io/ and https://github.com/codecov/example-python
    cover: bash -c \'rm -rf Tests/coverage.xml\'
    cover: bash -c \'cd Tests && coverage run run_tests.py --offline && coverage xml\'
    cover: codecov --file Tests/coverage.xml -X pycov -X gcov

...

I suspect it is linked to how we have environments py27-nocov and py27-cover setup (and likewise for the other versions) to define testing with and without coverage tracking, both using py27 for their dependencies.

[gaborbernat]

hello @peterjc thanks for the report will look into it 👍

[gaborbernat]

it seems the factor expansion is broken, is threated as substitution instead, mvp:

[tox]
envlist = {py27, py37}-cover
[testenv]
deps = {py27}: lib
       {py36}: lib

[gaborbernat]

Seems like a side effect of the feature #772. The problem is that {py27} in that context can signal both substitution and a factor. Note:

[tox]
envlist = {py27, py37}-cover
[testenv]
deps = py27: lib
       py36: lib

would work.

[peterjc]

Thank you :)

Removing the curly brackets in the deps = section does seem to work with a limited test locally - I have started a full test on TravisCI.

Is the space after the comma in the envlist = {py27, py37}-cover important or just stylistic?

[peterjc]

Your suggestion works for me - I'll leave this issue open as I am unclear if you consider this a tox bug or not. Thank you.

[gaborbernat]

The space in between those is important. It signifies a factor expression; e.g. you want py27-cover,py37-cover but you don't want to repeat the cover that many time so you did {py27, py37}-cover. You could also it even shorter py-{27,37}-cover. In this case you would use 27.

[gaborbernat]

For now, I would say, it's a won't fix as in our documentation and examples don't encourage this. Let's think this through and see if anyone reports this.

[ssbarnea]

This is a critical regression in tox as I am starting to get lots of build failures due to it. Can you please hide the new release from pypi until we find a way to avoid it?

When it comes to the scale of this issue, I believe that this is likely to break a 5 digit number of python packages (glad not all are building master daily).

I didn't had time to do a wild grep but I think that at least ⅓ python packages usign tox.ini are likely to encounter this.

Apparently the workaround is to add spaces after commas inside tox.ini expandable list of environments.

# broken:
    {py27,venv,cover,docs,pep8}: python2
# working:
    {py27, venv, cover, docs, pep8}: python2

[gaborbernat]

@ssbarnea if will not fail if you remove the single {py35}. Technically you could drop all {} from the per environment specification.

[gaborbernat]

Also trying to figure this out but where did you read you need the {}? In our documentation I can see without it and only in factor expressions I see it, see http://tox.readthedocs.io/en/latest/example/basic.html#compressing-dependency-matrix

Basic usage — tox 3.1.2.dev6 documentation

[obestwalter]

I feel obliged to channel the original author @hpk42 here and I am pretty sure that he would say that this is a regression that breaks backwards compatibility without warning the user, so should be fixed. The reason that I tend to be on the very conservative side regarding changes in tox is exactly this one. When we break something, we break thousands of CI builds and cause a lot of friction for a lot of people. Most people using tox are not madly interested in new features - they just want their builds to work. If we now start discussing with users whether we should fix a regression we unknowinlgy introduced, I could imagine that this decreases trust in the project and its maintainers.

I am open to discussing our approach to these things in the future, but right now I think this bug should be fixed - especially because the thrown error is not really helpful for the user.

[gaborbernat]

As true may be this would mean reverting 772. I don't see an easy fix other than revert here.

[obestwalter]

I think reverting this and make a quick bugfix release will be the best idea for now then.

[gaborbernat]

Feel free to go ahead with it then, sadly I'm on my holiday and without computer until Tuesday.

[obestwalter]

I will prepare a bugfix release within the next hours.

[obestwalter]

Tested with 3.1.2 just now. Fixed.