Isolate .local (i.e. pip) installations for each environment

[kcpevey]

Feature description

Currently, if I pip install a package into a conda env from the command line, the packages get added to my private .local space. However, if I have a second environment which does the same thing, THOSE get added to the .local space as well. This creates a situation where the pip packages from BOTH environments are now available in BOTH environments.

The real kicker is that if the second environment required different versions of the pip packages, it goes ahead and updates the .local, leaving the first environment in a broken state.

This happens very much behind the scenes and users have no idea what happened to their first environment since they "didn't touch it".

We should isolate the pip installed packages so that there is no cross-contamination between environments.

Value and/or benefit

Clean, isolated environments that pip can't break and happy users :)

Anything else?

No response

[costrouc]

A hack that could be added into a notebook to load a given venv on top of conda-store

python .... create venv

import sys
sys.path.append(venv python path)
import os
os.environ['PATH'] = venv bin path + os.environ['PATH']

Also other things that we mentioned is completely disabling pip usage within the jupyterlab notebook. With the pip environment variable.

We should not allow installing things inside of pip .local. The only use case I can imagine of needing to pip install ... is for development of a package. Otherwise conda-store should be used.

Alternative to prevent adding a new virtual environment. This will work pretty well for most use cases.

import sys
sys.path.append('path to library')
python -m ...

[pavithraes]

Short-term -- disallow pip installation in JupyterLab/terminal, point folks to using pip through conda-store + docs
long-term - implement/document the hack Chris mentioned above for development use-cases

Edit: See the next comment

[dharhas]

correction.

short-term -

disallow pip installation in JupyterLab/terminal, point folks to using pip through conda-store + docs
test and document the hack Chris mentioned above for development use-cases

long-term -

Think through how to make pip package installation easier in conda-store ui, currently this an advanced feature requiring using the yaml editor and knowing how env.yaml are structured.

[viniciusdc]

I discussed this last week with @costrouc; I will push a PR this afternoon. There is two env variables that will be included in the jupyterlab (user) pod, that will block usage of pip in an environment -- based on the docs so far this seems to work just fine, but I need to test it first

[dharhas]

I discussed this last week with @costrouc; I will push a PR this afternoon. There is two env variables that will be included in the jupyterlab (user) pod, that will block usage of pip in an environment

I only see one env var in the PR?

[kcpevey]

Documentation for developing packages on nebari now that the pip option is disabled: nebari-dev/nebari-docs#411