venv/bin/python does not set VIRTUAL_ENV (and PATH) environment

[muelli]

I was surprised to find out that there is a difference between sourcing the bin/activate file (which seems to be only compatible to Bash and Fish) and running bin/python directly.

I expected the latter to work equally well. It is unfortunately much easier to call a specific python interpreter than to set up the environment when dealing with multiple virtual environments.

The way I read the current documentation, it is also the expected behaviour that it "just works" when calling scripts inside the virtual env directly. (citing https://virtualenv.pypa.io/en/latest/userguide/, emphasis mine)

This will change your $PATH so its first entry is the virtualenv’s bin/ directory. (You have to use source because it changes your shell environment in-place.) This is all it does; it’s purely a convenience. If you directly run a script or the python interpreter from the virtualenv’s bin/ directory (e.g. path/to/ENV/bin/pip or /path/to/ENV/bin/python-script.py) there’s no need for activation.

This, however, is not completely true. If you subprocess.call(pip) from an activated python, you get different results from calling it from venv/bin/python. This is probably what lead to fiduswriter/fiduswriter#195. The difference is at least in the VIRTUAL_ENV and PATH variables. But other people also have this problem, it seems:
http://stackoverflow.com/questions/11963019/running-python-script-from-inside-virtualenv-bin-is-not-working
http://stackoverflow.com/questions/8052926/running-subprocess-within-different-virtualenv-with-python
http://stackoverflow.com/questions/3287038/cron-and-virtualenv

This issue seems to have cropped up here (#247) albeit in a different flavour.

It'd be nice if this could be addressed and a "one-stop" solution be provided (rather than the two-step source and call).

[Ivoz]

Yes, the subprocess is creating a new environment for the execution of the call, one in which your modified PATH (as set by an activate script) is no longer present.

When running directly from venv\bin\python I believe you are coming across a coincidence that since venv\bin is the current working directory, the first pip found is the one find inside there.

For instance, one example of solving this would be passing the current PATH through the env= dictionary as an argument one of the subprocess module functions, so that the so-called 'convenience' of the activate script flows through to the new subprocess.

I do not believe there is a general or universal use case for a virtualenv python to be modifying all subprocess calls by default. A new environment might be wanted by the coder, whether they are using a virtualenv'ed python or not.

[muelli]

I don't follow your statement.

Yes, the subprocess is creating a new environment for the execution of the call, one in which your modified PATH (as set by an activate script) is no longer present.

As environment variables are inherited, I doubt that this is true.
The problem begins earlier in that the virtualenv's bin/python interpreter doesn't even set the PATH. That behaviour is different from bin/activate and, according to the way I read the documentation, unexpected and thus a bug.

[pfmoore]

python doesn't set any environment variables, it just inherits the parent environment. So if you don't set $PATH in your shell, it's not set in python. Personally, I don't see that as being in contradiction to the documentation you quoted - but if you want to suggest an alternative wording as a PR, feel free to do so.

If you want to call pip from Python and be sure you get the pip associated with the Python interpreter you're using, a much better way of doing so is

subprocess.call([sys.executable, '-m', 'pip', "pip arguments here"])

That ensures you get the version of pip that you want, regardless of PATH or any other environment variables..

[muelli]

I don't see that as being in contradiction to the documentation you quoted

hm. I can indeed imagine a way to read it in a non-contradicting way. But it still suggests that both ways of using the venv are equivalent. And it seems that many people assume equivalent behaviour as I've shown with the links in the initial post. I've just done a quick google search for "virtualenv cronjob". Out of the 12 results on the first page only one got it right by suggesting to write a simple three-line wrapper to activate the venv. Others, including an official looking mailinglist, also suggest to write such a wrapper but then conclude that running a program in the venv directly is as good (which it apparently isn't, because some env variables will not be set).

Is the different behaviour intentional?
Does it have a use case to set PATH and VIRTUAL_ENV in one case but not the other?

Otherwise I suggest to make it behave uniformly. Either don't set PATH and make it an alias or something or make the venv python set these variables just as the shell scripts do. Or provide such a run_in_venv.sh for people like me who want a simple hit-and-run style execution. Heck, you could accept arguments to the sourced script and then execute the argument.

[th3goose]

Seems like I'm having the same issue with my path not being modified when activating my virtuenv. I checked my path and it still references the system python path rather then my local virtualenv environment. Is this a bug in documentation or a bug in virtualenv not modifying the path?

[muelli]

I have the feeling that it depends on who you ask ;-)
I consider it a bug in both the documentation and virtualenv.

[pfmoore]

@adamcooley If the activate script doesn't set PATH, then that's a bug - that's the whole pint of the script. However, it's unlikely that this is actually the case, as loads of people use the script successfully - so it's potentially the case that you're using it wrongly. You need to "source" the script, not just run it, in order for it to affect your current shell's environment - precisely how you do this depends on your OS and shell, and you don't say what you're using, so it's hard to be more specific.

But note that this is different from the issue originally raised here, which is that the Python executable doesn't set the PATH or VIRTUAL_ENV environment variables. That's by design, and is not a bug.

@muelli - just to be 100% clear, the current behaviour is not a bug, it's by design. It's not even a behaviour of virtualenv, it's how the Python interpreter itself works. The only thing you might be able to get support for is adding some comments to the documentation to clarify that you shouldn't be writing scripts that assume sys.executable is on $PATH, or that VIRTUAL_ENV is set - but I'd argue that that is just as true for general Python scripts, and is unrelated to whether you are running in a virtualenv.

[muelli]

``You need to "source" the script, not just run it, in order for it to affect your current shell's environment'' and this is the decision that people like me are objecting. Not the first half, but the second half. I'm not necessarily interested in making it affect my currently running shell session. Most of the time, if not always, I'm interested in making it spawn a modified shell session s.t. I can run it from, say, a cronjob. So far it doesn't seem to have been acknowledged that this is an issue. However, I have pointed out that this is a real issue affecting people. People have created workarounds. Mostly bad ones. The good ones are the three line wrappers.

A fix has been pointed out, too. Probably the most trivial solution is to make the sourced script take arguments and simply execute the arguments. E.g. put smth like "shift $@; exec $@" at the end of the activate script. That'd be a one line modification allowing a cron job to execute /path/to/venv/bin/activate /path/to/venv/bin/myprogram.

I think it'd be nicer if /path/to/venv/bin/python would behave the very same way of /path/to/venv/bin/activate, though. Which, btw, you don't seem to have acknowledged yet. One could make /path/to/venv/bin/python that three line wrapper that everybody is writing and install the real python as _python or so.

[th3goose]

Yeah I am sourcing it. Source bin/activate I'll get home and update with some specifics on the environment. It's running on google compute engine.

[pfmoore]

Most of the time, if not always, I'm interested in making it spawn a modified shell session

There are tools that can do that (vex is, I believe, one of them). Or you can write your own. At the end of the day, all that the activate script is is a convenience wrapper to modify PATH and set VIRTUAL_ENV. The extra details (setting prompt, changing the current shell) are just that - details of one specific convenience wrapper., Feel free to ignore it and write your own.

We could change the behaviour of the activate scripts, or add alternative convenience wrappers. But changing the current ones is a backward compatibility problem, and adding more just increases the difficulty in explaining why we have all these options. Personally, I'd rather remove the activate scripts and let everyone write their own personal versions. But that's not likely to happen, precisely because of backward compatibility (which you seem happy to ignore).

I think it'd be nicer if /path/to/venv/bin/python would behave the very same way of /path/to/venv/bin/activate, though. Which, btw, you don't seem to have acknowledged yet.

I've acknowledged that you think that repeatedly. I just don't agree with you. I don;'t want python in a virtualenv to work differently than the system python.

One could make /path/to/venv/bin/python that three line wrapper that everybody is writing and install the real python as _python or so.

That's way harder than you're making out. For example, on Windows the wrapper needs to be an executable. If you want to do this for yourself you can write a virtualenv bootstrap (see https://virtualenv.pypa.io/en/stable/reference/#extending-virtualenv). But again, doing this for everyone is a much less straightforward exercise in making sure you don't break someone's workflow.

[pfmoore]

It's running on google compute engine.

Hmm, maybe compute engine is odd. I've no experience with that, I'm afraid.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.