namespace packages take priority over setuptools PEP 660 editables in lenient mode

[jrbourbeau]

I have an editable version of distributed installed locally with pip install -e . from the root of the repo. When I drop into ipython I see

In [1]: import distributed
distr
In [2]: distributed.__file__
Out[2]: '/Users/james/projects/dask/distributed/distributed/__init__.py'

In [3]: distributed.__version__
Out[3]: '2023.4.0+1.g5c0a412c'

This looks like I expect it to. When I then navigate to another directory, for example, over to the root of dask/dask on my laptop, I no longer see the same distributed.__version__ and distributed.__file__:

In [1]: import distributed

In [2]: distributed.__file__

In [3]: distributed.__version__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 distributed.__version__

AttributeError: module 'distributed' has no attribute '__version__'

I've not run git bisect, but my guess is #7629 is related

cc @graingert for visibility

[graingert]

@jrbourbeau I can't reproduce this, what are the values of repr(distributed.__spec__), repr(distributed) and sys.path?

[jrbourbeau]

Hmm I don't seem to be able to reproduce myself...I'll go ahead and close for now and re-open if this shows up again. Apologies for the noise.

[graingert]

I can now reproduce this issue and have narrowed it down to the "lenient" setuptools PEP 660 mode

[graingert]

pip install -e .

Editable installs allow you to install your project without copying any files. Instead, the files in the development directory are added to Python’s import path. See https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs

There are two different implementations of editable installs in pip, "legacy" setup.py develop and the current PEP 660 editable wheels.

legacy setup.py develop editables

without a pyproject.toml file pip directly calls setup.py develop which appends the current project directory to site-packages/easy-install.pth eg:

$ cat /home/graingert/anaconda3/envs/dask-distributed/lib/python3.11/site-packages/easy-install.pth
/home/graingert/projects/distributed

this causes the project directory to be added to the sys.path eg:

>>> import sys
>>> sys.path
['', '/home/graingert/anaconda3/envs/dask-distributed/lib/python311.zip', '/home/graingert/anaconda3/envs/dask-distributed/lib/python3.11', '/home/graingert/anaconda3/envs/dask-distributed/lib/python3.11/lib-dynload', '/home/graingert/anaconda3/envs/dask-distributed/lib/python3.11/site-packages', '/home/graingert/projects/distributed']

this has the advantage of being very simple, however it also means that all the files and directories in the project dir become importable, not just the "distributed" package: eg

>>> import build
>>> build
<module 'build' (<_frozen_importlib_external.NamespaceLoader object at 0x7f3b94241110>)>
>>> import continuous_integration
>>> continuous_integration
<module 'continuous_integration' (<_frozen_importlib_external.NamespaceLoader object at 0x7f3b94133cd0>)>
>>> import docs
>>> docs
<module 'docs' (<_frozen_importlib_external.NamespaceLoader object at 0x7f3b94133e90>)>
>>> import icons
>>> icons
<module 'icons' (<_frozen_importlib_external.NamespaceLoader object at 0x7f3b94133c50>)>
>>> 

in addition this is legacy behaviour and the pip maintainers want to remove any direct calls to setup.py from pip in the future, this is part of the original motivation for moving to pyproject.toml in dask and distributed

PEP 660

Editable installs for pyproject.toml based builds defined by https://peps.python.org/pep-0660/ instead of calling setup.py directly
you define a build backend module with a def build_editable that creates a .whl file that uses some strategy to add the project files to python import system.

a number of strategies are suggested in the PEP: https://peps.python.org/pep-0660/#what-to-put-in-the-wheel and setuptools implements three different approaches https://setuptools.pypa.io/en/latest/userguide/development_mode.html

setuptools lenient mode

"lenient" mode is the default behaviour. setuptools will use a .pth file to add a MetaPathFinder to sys.meta_path:

cat /home/graingert/anaconda3/envs/dask-distributed/lib/python3.11/site-packages/__editable__.distributed-2023.4.0+8.g76bbfaf9f.pth
import __editable___distributed_2023_4_0_8_g76bbfaf9f_finder; __editable___distributed_2023_4_0_8_g76bbfaf9f_finder.install()

>>> import sys
>>> sys.meta_path
[<_distutils_hack.DistutilsMetaFinder object at 0x7fcf8c0175d0>, <class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>, <class '__editable___distributed_2023_4_0_8_g76bbfaf9f_finder._EditableFinder'>]

however this results in namespace packages on the current directory taking precedence over the installed editable package because _frozen_importlib_external.PathFinder finds namespace packages on sys.path before ..._EditableFinder causing the problem in this github issue.

setuptools compat mode

"compat" mode is an optional mode enabled by running pip install -e . --config-settings editable_mode=compat this works by adding a .pth file that appends the project directory to the sys.path eg:

cat /home/graingert/anaconda3/envs/dask-distributed/lib/python3.11/site-packages/__editable__.distributed-2023.4.0+8.g76bbfaf9f.pth
/home/graingert/projects/distributed

>>> import sys
>>> sys.path
['', '/home/graingert/anaconda3/envs/dask-distributed/lib/python311.zip', '/home/graingert/anaconda3/envs/dask-distributed/lib/python3.11', '/home/graingert/anaconda3/envs/dask-distributed/lib/python3.11/lib-dynload', '/home/graingert/anaconda3/envs/dask-distributed/lib/python3.11/site-packages', '/home/graingert/projects/distributed']

this mode behaves almost the same as the legacy setup.py develop approach and because the project directory is on the sys.path regular packages take precedence over namespace packages, avoiding the problem in this github issue.

unfortunately this mode is considered deprecated and was scheduled to be removed December 2022

the official documentation for this mode is https://setuptools.pypa.io/en/latest/userguide/development_mode.html#legacy-behavior

setuptools strict mode

"strict" mode is an optional mode enabled by running pip install -e . --config-settings editable_mode=strict
setuptools will create a tree of file links in the build directory and add it to sys.path via a .pth files. Because the link tree directory is on the sys.path regular packages take precedence over namespace packages, avoiding the problem in this github issue

the official documentation for this mode is https://setuptools.pypa.io/en/latest/userguide/development_mode.html#strict-editable-installs

[graingert]

There is no way to configure setuptools to always use a particular mode and so we could only update our developer documentation to recommend pip install -e . --config-settings editable_mode=compat or pip install -e . --config-settings editable_mode=strict

[graingert]

alternatively the build backend hatchling implements editable installs using the same approach as setuptools "compat" mode however we'd have to wait for setuptools_scm v8 to include the distributed.__git_revision__ attribute.