How can I get a predictable virtualenv dirname?

[wonderbeyond]

My environment

1. Ubuntu 16.04.3 LTS
2. Python version: Python 2.7.12
3. Pipenv version: pipenv, version 8.3.2

What confused me?

$ echo $WORKON_HOME
/home/wonder/PyEnvs

$ pipenv --where
/home/wonder/workspace/myproj

$ pipenv --venv
/home/wonder/PyEnvs/myproj-BKbQCeJj

Why is BKbQCeJj in my virtualenv dirname? Can it be predictable? And can I custom it to be simply myproj?

Why it is a problem?

The Autocomplete-Python plugin of Atom needs to know the Python executable paths to work properly with every individual project.

I've been using virutalenvwrapper, which gives me predictable path names, So I can pass /home/wonder/PyEnvs/$PROJECT_NAME/bin/python to Autocomplete-Python, let it find the right python path for a project.

Now with pipenv's postfixed paths, Autocomplete-Python stopped working. So any helps?

[jacebrowning]

You can set the PIPENV_VENV_IN_PROJECT environment variable on every machine you work on: https://docs.pipenv.org/advanced.html#configuration-with-environment-variables

IMHO, this should be the default because it's less likely to cause confusion than the current behavior. Grouping virtual environments in a semi-hidden directory with obscure names seems like an advanced feature that should be opt-in. cc @nateprewitt

[techalchemy]

Oh, @wonderbeyond, it is predictable! We determine the full path here: https://github.com/kennethreitz/pipenv/blob/a4aca53b5d02a4168214a487714d9836c9415a0c/pipenv/project.py#L140 but the name itself is generated here: https://github.com/kennethreitz/pipenv/blob/a4aca53b5d02a4168214a487714d9836c9415a0c/pipenv/project.py#L116

@property
def virtualenv_name(self):
    # Replace dangerous characters into '_'. The length of the sanitized
    # project name is limited as 42 because of the limit of linux kernel
    #
    # 42 = 127 - len('/home//.local/share/virtualenvs//bin/python2') - 32 - len('-HASHHASH')
    #
    #      127 : BINPRM_BUF_SIZE - 1
    #       32 : Maximum length of username
    #
    # References:
    #   https://www.gnu.org/software/bash/manual/html_node/Double-Quotes.html
    #   http://www.tldp.org/LDP/abs/html/special-chars.html#FIELDREF
    #   https://github.com/torvalds/linux/blob/2bfe01ef/include/uapi/linux/binfmts.h#L18
    sanitized = re.sub(r'[ $`!*@"\\\r\n\t]', '_', self.name)[0:42]

    # Hash the full path of the pipfile
    hash = hashlib.sha256(self.pipfile_location.encode()).digest()[:6]
    encoded_hash = base64.urlsafe_b64encode(hash).decode()

    # If the pipfile was located at '/home/user/MY_PROJECT/Pipfile',
    # the name of its virtualenv will be 'my-project-wyUfYPqE'
    return sanitized + '-' + encoded_hash

[wonderbeyond]

@techalchemy Can we use the project name directly without postfix as venv name?

@jacebrowning I am used to store all venvs in the same directory while I was using virtualenvwrapper, and I think it's ok.

[techalchemy]

@wonderbeyond you can't make pipenv create a virtualenv other than a postfixed one (there are a few reasons for this but I won't get into that now) or one at .venv in your project root. However, you can create and activate a virtualenv yourself, install pipenv, and it will recognize that it is in a virtualenv when you do things like pipenv install etc. This is what I normally do when I am developing.

[wonderbeyond]

@techalchemy You mean I still need virtualenvwrapper to manage my venvs...
When can I use a single pipenv?

[joshfriend]

I wholeheartedly agree that PIPENV_VENV_IN_PROJECT should be default! The name and location are always predictable, its just <project root>/.venv.

[techalchemy]

@wonderbeyond I don't mean that you need virtualenvwrapper to manage your virtualenvs, you can use the solution that @joshfriend and @jacebrowning mentioned which is to set the environment variable in your shell config which will use .venv as the default path in your project directory

Otherwise pipenv will use WORKON_HOME, whether you personally use virtualenvwrapper or not. I don't think this behavior is very likely to change so I'm going to close this out for now

[cdaringe]

1. the URI above has been moved to: https://docs.pipenv.org/advanced/#custom-virtual-environment-location
2. i know this issue is closed, but @techalchemy, users here have said "we want this" and youve said "no", but what's missing from the conversation is a compelling "why we want this" case. i want this because i collaborate with a team using a common set of tools and configs. we often share our development configuration, and we often inspect our site-packages. a unique location of the env on each system prevents sharing of our config. having the venv local to your project makes collaboration w/ teamsters one step easier (no ENV config), and, adds stupid-simple clarity to exactly which py you're using and the state of the env. the python community seems to be somewhat hostile in my experience to the node community (🤕) but npm, yarn, etc do this at least w/ dependencies, and it's a really excellent model. our teams use a smattering of tech stacks, and python's bootstrapping & packaging ergonomics have been a real headache. pipenv seems like a huge improvement, but little issues like this my peers and i feel add complexity when PIPENV_VENV_IN_PROJECT=1 seems like a really good dev UX default. thanks for your time. keep up the good work

[techalchemy]

@cdaringe to clarify, my position isn't that I don't understand why people are asking for it, and my position also is not that I don't see any value in having it. My position is only that I don't think it's very likely that this will be bundled in pipenv, because generally speaking having dozens of extra knobs only tends to overcomplicate things for the standard use case.

It has always been my position, and I'm sure others would share this, that such additional functionality should be built out in additional packages. For your specific case you may want to check out the new project by @kennethreitz called pipenvlib which provides an API for interacting with pipenv managed environments.

[joshfriend]

@cdaringe I'll try to add another concrete example. I use PIPENV_VENV_IN_PROJECT=1 because (as far as i know) theres no way to tell pipenv what you want the venv to be called. This is a problem for me because without the ability to know the name of the environment, it is difficult to get other tools to integrate with the created venv. For example, in VSCode, with PIPENV_VENV_IN_PROJECT=1 set, I know that I can configure the python plugin to use "python.pythonPath": "${workspaceRoot}/.venv/bin/python" as the path to the venv.

Without that option set, other tools have no idea how to find the venv. I suppose the default name of .venv could be a problem if you need multiple venvs and i'm not sure how to work around that. Perhaps the name could be configurable via the Pipfile, though I don't think that PEP508 gives a way to do that.

I realize that this is something that could be added to the vscode python tools via pipenvlib, but until that happens, its somewhat of a pain point.

[cdaringe]

@joshfriend, my problem is your problem, almost verbatim. i must have not conveyed that clearly :/

@techalchemy, thx for the response. to be clear, i wasn't advocating for additional functionality--rather, i was (apparently poorly 😄 ) advocating a +1 for @joshfriend's claim

"PIPENV_VENV_IN_PROJECT should be default!"

[nateprewitt]

@cdaringe @joshfriend, PIPENV_VENV_IN_PROJECT was actually the default for the first few months of pipenv’s existence. We later moved to pew because there was demand from people using things like VirtualenvWrapper that wanted their environments to exist in a centralized place. It also prevents users from worrying about including their venv in their git repository. Each method has trade offs, and we’re never going to make everyone happy here. I’m also in your camp of preferring having the environment with the project, but realize we need a STRONG reason to break the existing functionality.

Currently, for most users it’s a binary decision. Either you want all of your environments with the project or you want them centrally managed. The flag allows people to opt into that and not have to change anything after, which I think is reasonable. I’m happy to hear out more discussion, but realize that if we change the default, another mirror issue will rise up with the opposite complaint and the cycle will continue. Defaults are hard.

[jacebrowning]

@nateprewitt Could you point me to some of those discussions? I must have missed them.

For reference, these are all examples of people being surprised by the current default behavior and I expect that issues like this will continue to crop up:

• use .venv #341
• consistent virtualenv location? #589
• How can I get a predictable virtualenv dirname? #1049
• Name of the virtualenv #1184
• How to set the full name of the virtualenv created #1226
• Provide argument which specifies the subname of the virtual environment folder instead of automatic hashname #1292
• allow virtualenv location and name to be configured #1307
• PIPENV_VENV_IN_PROJECT=true should be default behavior #1382
• Control where virtualenv is created #1385
• custom prompt names upon virtualenv creation #1509
• Can I specify PIPENV_VENV_IN_PROJECT in Pipfile? #2197

Isn't one of the major selling points of pipenv that users no longer need to manage virtual environments for a project because pipenv essentially takes care of activation? What exactly is the workflow that involves both pipenv and virtualenvwrapper?

IMO, virtualenvwrapper is more or less a workaround for a problem that pipenv solves and the default behavior shouldn't cater to the less common workflow.

---

I have thought about this quite a bit, so here's my proposal for how a change could be rolled out:

• First minor release:
  • Add PIPENV_VENV_HOME, which defaults to ~/.local/share/virtualenvs (or equivalent)
  • Use PIPENV_VENV_HOME as the root directory for virtual environments unless PIPENV_VENV_IN_PROJECT is set
• Second minor release:
  • Add a depreciation warning about PIPENV_VENV_IN_PROJECT being removed in the next major release
  • Add a depreciation warning about needing to set a value for PIPENV_VENV_HOME in the next major release to keep the current behavior
• Next major release:
  • Remove PIPENV_VENV_IN_PROJECT
  • No longer set a default value for PIPENV_VENV_HOME
  • Create virtual environments in the project root unless PIPENV_VENV_HOME is set

[nateprewitt]

@jacebrowning #10 and #173 were two of the earlier ones. There was also a fair amount of discussion around this in IRC and email. Kenneth decided it was the better default and that’s when we made the change. Like I said, I personally feel that .venv is the better option, but I’m not convinced we aren’t a vocal minority here, or that we won’t see a backlash changing the default.

It’s not that people want virtualenvwrapper and pipenv, it’s that they want it to work like virtualenvwrapper. There were requests for workon support which requires the centralized environments and that has been the default for most of the previous virtualenv related tooling.

I don’t think there’s anything wrong with your proposal, in fact it seems like a good roadmap. I think this is relatively low priority though in the scope of changes for pipenv. It’s functionality that works, and requires fairly minimal overhead to change, and as I said I’m not convinced it helps everyone. It likely just shifts a burden from one group to another.

[cjw296]

@nateprewitt - having just filed #1307 for very similar reasons, please can I request that this issue is re-opened, even if only to prevent more dupes being opened?

#1292 can likely be closed as a dupe of this issue...

Also, if motivated people such as @jacebrowning or myself wanted to make progress on the roadmap proposed, would there be any objections or pushback?

[wonderbeyond]

Hi everyone, I'm the author of this issue.

Till now, I don't care the virtualenv locations generated by Pipenv, Because I have been using pyenv-virtualenv to manage virtualenvs, and then let Pipenv work in a virtualenv created by pyenv-virtualenv.

I think:

• pyenv is a nice tool for managing multiple python versions
• pyenv-virtualenv is a nice tool for isolate python environments for different projects
• Pipenv is a nice tool for managing dependencies for a environment

So why not use them in combination?

# Install Python 2.7
$ pyenv install 2.7

# Create a virtualenv of Python 2.7, named "my-awesome-project"
$ pyenv virtualenv 2.7 my-awesome-project

# Whenever we get into "my-awesome-project" directory, automatically activate the virtualenv
$ echo "my-awesome-project" > ~/workspace/my-awesome-project/.python-version

$ deactivate
$ cd ~
$ cd ~/workspace/my-awesome-project
# Now the virtualenv "my-awesome-project" is active, let's do pipenv things...

@techalchemy I don't think Pipenv should "automatically creates and manages a virtualenv for your projects". What do you think about?

[cdaringe]

ya, but isn't an easy and great developer experience important? "Go download, install, and learn 3 tools (4 if you rightfully including publishing) to write a python library" seems like a tall order. it seems like the runtime and the package manager alone should suffice. python tends to be keys included in code only, not in UX. It seems like pipenv is the perfect place to rectify that

[wonderbeyond]

@cdaringe How about writing a script to install them with only one command?

$ curl -L https://raw.githubusercontent.com/pypa/pipenv/master/scripts/pip-mates-installer | bash

[kennethreitz]

We're not changing our approach here.

[cjw296]

@kennethreitz - what is the approach you're referring to here?

[jacebrowning]

#1382 (by @olegantonyan) was an exactly duplicate of this. Given that the compelling reasons for this change are already documented here, can we reopen this issue to prevent further duplicate issues from opening?

[uranusjr]

@jacebrowning I highly doubt it is likely to help. People don’t really look for duplicates before opening issues (anecdotal experience). The issue template already tells the reporter clearly to search for existing issues, open or closed. Every maintainer uses the issue tracker differently; no point changing the issue’s status after triage if the resolution is not changing.

[phoikoi]

@wonderbeyond I agree with your combination solution, but instead of the last stanza with the deactivate, cd ~, etc., I use pyenv local:

pyenv install 3.6.5
mkdir foo; cd foo
pyenv virtualenv 3.6.5 foo
pyenv local foo # This makes the .python-version file and activates the venv

After this, pipenv will recognize the activated venv and chug happily along doing its thing, assuming it's been installed globally (by following these instructions or something similar).

As a side note, I have a quick and dirty script I use which very much simplifies installation of pyenv et. al. on Debian-flavor Linux environments. It is basically some instructions from the author of pyenv, cobbled together into script form:

function pyenv_prereqs () {
    # Install packages for pyenv
    # sudo is used here to enable user install of pyenv
    sudo apt-get install \
        git build-essential libssl-dev zlib1g-dev \
        libbz2-dev libreadline-dev libsqlite3-dev \
        wget curl llvm libncurses5-dev
}

function install_pyenv() {
    pyenv_prereqs
    if [ -d ~/rc ]; then
        PYENV_INIT=~/rc/pyenv.sh
    else
        PYENV_INIT=~/.bash_profile
    fi
    cat >>${PYENV_INIT} <<EOD
if [ -d ~/.pyenv ]; then
    export PATH="\$HOME/.pyenv/bin:\$PATH"
    eval "\$(pyenv init -)"
    eval "\$(pyenv virtualenv-init -)"
fi
EOD
    source ~/.bash_profile
    curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash
}

install_pyenv

[airtonix]

Those using docker just need to do this:

# ./docker/backend/Dockerfile

...

FROM base as install

COPY backend/Pipfile \
     backend/Pipfile.lock \
     ./
RUN PIPENV_VENV_IN_PROJECT=1 \
    pipenv install --deploy


FROM install as prod

COPY backend ./
COPY --from=install /opt/app/.venv ./

...

FROM prod as test

RUN PIPENV_VENV_IN_PROJECT=1 \
    pipenv install --dev --deploy

...

Then when using vscode-remote you'll be able to configure with .devcontainer.json or .vscode/settings.json where your python.pythonPath is.