support for venv --system-site-packages

[TwinT]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Adding support for --system-site-packages, as described in venv documentation, makes possible to have a set of base common packages available for a number of virtual environments, thus saving disk space and reducing environment creation time. I believe that this option gives some useful extra flexibility on package and environment management.

[kam1sh]

As I see, virtual environments creates via venv.EnvBuilder and virtualenv.create_environment:
https://github.com/sdispater/poetry/blob/df98fd289b9b629d2171900c1b1bddcee3696cff/poetry/utils/env.py#L301-L319
...so it shouldn'be hard.

I see this as a setting in poetry config with something like settings.virtualenvs.include-system-packages. Will (try to) implement tomorrow.

[rnc]

This would be very useful as its a significant space saving - e.g. on a single small repository the difference was 150MB of dependencies - multiply that out and its soon worthwhile.

[danielsig727]

someone please revive #1404 :'(
Or I'm willing to help if there're more to do to complete the feature.
I need that feature since I have needs some packages to be globally available in order to run setup.py of my private package installed from git.

[frankier]

I agree this functionality would be useful in Poetry, but for the meantime here are some workarounds (cross-posting across related issues):

Use case 1) I just want to save space use by my virtualenvs:

1. Install https://rdfind.pauldreik.se/
2. cd ~/.cache/pypoetry/virtualenvs/ && rdfind -makehardlinks true .

Use case 2) I need to use a globally installed package from my poetry venv.

1. Open ~/.cache/pypoetry/virtualenvs/myvenv/pyvenv.cfg
2. Set include-system-site-packages = true

[stackErr-NameNotResolved]

This issue seems to be merged and closed but actually never released? Using Poetry 1.1.13:

poetry config virtualenvs.options.system-site-packages true

ValueError

Setting virtualenvs.options.system-site-packages does not exist

[ghost]

It looks like it is included in the release notes for 1.2.0a1

[couling]

What's the behaviour of this? Will it default to using the system site package versions when doing poetry update or poetry install without a lock file?

[awoimbee]

I wanted to use this in a docker container so I could avoid virtualenvs.create false which breaks some system deps. But poetry install wants to "update" everything (eg Updating certifi (2022.6.15 /opt/conda/lib/python3.7/site-packages -> 2022.6.15)). So it's useless for me 😣.

EDIT:
I'm using poetry 1.2.0b3
Also I guess this answers you question @couling
This behavior is problematic when the goal is to share dependencies for eg a container

[rpdelaney]

@awoimbee Can't you just install whatever dependencies at build time? I would think that dockerizing the environment would obviate a --system-site-packages option, but maybe I don't understand your use case?

[awoimbee]

@rpdelaney
As is often the case here, my issue is mostly with pytorch. It's >1GB, which is an issue when building/pushing/pulling the same container again and again (my poetry install layer is way too big).
To reuse image layers (faster image build/push/pull), I want a base image with cuda, torch, ... (think https://hub.docker.com/r/pytorch/pytorch but custom) and then I want to build my application container on top of this base image.
The resulting application container shall be used in prod and as a dev container (with minor differences between the two).

Possible solutions for the application image to inherit packages from the base image:

• virtualenvs.create false: breaks some system dependencies, is not suitable for a dev env
• conda environments: not the best (I won't go into details)
• settings.virtualenvs.include-system-packages: seems like the perfect solution
  But no ! poetry with settings.virtualenvs.include-system-packages will re-install torch (and other packages), leading to an even slower build time and an even larger container.

[rpdelaney]

OK, thanks for the explanation. My docker-fu is not strong enough to suggest solutions in that direction...

[doomroep5]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Adding support for --system-site-packages, as described in venv documentation, makes possible to have a set of base common packages available for a number of virtual environments, thus saving disk space and reducing environment creation time. I believe that this option gives some useful extra flexibility on package and environment management.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.