Having interpreters that aren't installed in envlist causes tox to refuse to run

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/280
• Originally reported by: @ceridwen
• Originally created at: 2015-10-27T19:43:45.686

I have a tox.ini that looks like this.

[tox]
# envlist = py27, py33, py34, py35, pypy, jython, pylint
envlist = py27, py33, py34
skip_missing_interpreters = true

[testenv]
deps =
  py27,pypy,jython: funcsigs
  lazy-object-proxy
  py27,py33,pypy,jython: singledispatch
  six
  wrapt
  pylint: hg+https://bitbucket.org/logilab/pylint
commands = python -m unittest discover -s {envsitepackagesdir}/astroid/tests -p "unittest*.py"

As I understand it, this should run with any missing interpreters skipped. Instead, running tox with CPython 2.7 and 3.4 installed, but not 3.3, gives:

ERROR: InterpreterNotFound: python3.3
ERROR: unknown environment 'py27'

Different combinations of missing interpreters can give different InterpreterNotFound and unknown environment errors, and it's not deterministic which will be reported.

Commenting out the commands = line causes it to run with:

  py27: commands succeeded
SKIPPED:  py33: InterpreterNotFound: python3.3
  py34: commands succeeded
  congratulations :)

As I'd expect, but of course this then doesn't run the tests. tox -e using py27 or py34 will run, tox -e py33 gives the same error as tox.

This is on Ubuntu Linux. I've tried both version 1.8.0, the version packaged by Ubuntu, and version 2.1.1 installed with pip, and they exhibit the same behavior.

[pytoxbot]

Original comment by @hpk42

fix issue280: fix case where --skip-missing-interpreter would not work if {envsitepackagesdir} substitution was used.

→ <>

[pytoxbot]

Original comment by @ceridwen

Note: setting skip_missing_interpreters to true in the config file or on the command line doesn't affect tox's behavior at all, it will still refuse to run without all of the interpreters available, when it should output something like:

ERROR:   py27: commands failed
ERROR:   py33: InterpreterNotFound: python3.3
ERROR:   py34: commands failed

Here is a smaller tox.ini:

[tox]
envlist = py27, py33

[testenv]
commands = python -m unittest discover -s {envsitepackagesdir}/astroid/tests -p "unittest*.py"

With all of the dependencies removed, I'd expect InvocationErrors, but instead it gives the same InterpreterNotFound and unknown environment errors and terminates. As commenting out that commands line allows it to run, it's something specific about that commands line.

[pytoxbot]

Original comment by @hpk42

Looks like there are code paths that don't honor skip_missing_interpreters. Could you take a look yourself? Or, failing that, provide a minimal tox.ini that exhibits the behaviour?