Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid version detection when using setuptools_scm + pyproject #154

Closed
cyraxjoe opened this issue Aug 19, 2020 · 8 comments
Closed

Invalid version detection when using setuptools_scm + pyproject #154

cyraxjoe opened this issue Aug 19, 2020 · 8 comments

Comments

@cyraxjoe
Copy link
Contributor

Apparently poetry2nix is unable to detect the proper version of the package when the package distributes a source package with pyproject.toml (included by default in the sdist packages from setuptools) and the package is using setuptools_scm with the toml extra, which relies on having the pyproject file in the source distribution.

An example of a package that will cause problems when used as a depedency in poetry2nix is jaraco.functools, which has this pyproject.toml, building a wheel from the sdist of jaraco.functools either with python setup.py wheel or pip wheel without the pyproject.toml will end up with a wheel with version 0.0.0, messing up the following build, because the dependency at the expected version is not met.

The removal happens specifically at this hook script https://github.com/nix-community/poetry2nix/blob/master/hooks/pip-build-hook.sh#L8, I'm not sure what are the implications of not removing the pyproject.toml but given the new default of including the pyproject in the source distribution, I think that more and more packages will start to relay in the presence of that file, the simple removal of the file seems that will cause additional surprises in the future.

The issue can be replicated with:

Empty fooapp.py file.

pyproject.toml:

[tool.poetry]
name = "fooapp"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.8"
"jaraco.functools" = "^3.0.1"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

poetry.lock:

[[package]]
category = "main"
description = "Functools like those found in stdlib"
name = "jaraco.functools"
optional = false
python-versions = ">=3.6"
version = "3.0.1"

[package.dependencies]
more-itertools = "*"

[package.extras]
docs = ["sphinx", "jaraco.packaging (>=3.2)", "rst.linker (>=1.9)"]
testing = ["pytest (>=3.5,<3.7.3 || >3.7.3)", "pytest-checkdocs (>=1.2.3)", "pytest-flake8", "pytest-black (>=0.3.7)", "pytest-cov", "jaraco.classes"]

[[package]]
category = "main"
description = "More routines for operating on iterables, beyond itertools"
name = "more-itertools"
optional = false
python-versions = ">=3.5"
version = "8.4.0"

[metadata]
content-hash = "2f9d7655f613372204310a0159ec45ddc65d28894ab1076f766d09730d588d34"
python-versions = "^3.8"

[metadata.files]
"jaraco.functools" = [
    {file = "jaraco.functools-3.0.1-py3-none-any.whl", hash = "sha256:d3dc9f6c1a1d45d7f59682a3bf77aceb685c1a60891606c7e4161e72ecc399ad"},
    {file = "jaraco.functools-3.0.1.tar.gz", hash = "sha256:9fedc4be3117512ca3e03e1b2ffa7a6a6ffa589bfb7d02bfb324e55d493b94f4"},
]
more-itertools = [
    {file = "more-itertools-8.4.0.tar.gz", hash = "sha256:68c70cc7167bdf5c7c9d8f6954a7837089c6a36bf565383919bb595efb8a17e5"},
    {file = "more_itertools-8.4.0-py3-none-any.whl", hash = "sha256:b78134b2063dd214000685165d81c154522c3ee0a1c0d4d113c80361c234c5a2"},
]

default.nix:

{ pkgs ? import <nixpkgs> {} }:
pkgs.poetry2nix.mkPoetryApplication {
  projectDir = ./.;
}

The build will result in something similar to:

these derivations will be built:
  /nix/store/pjvxqcp9av8jsl62sjn09rqnpann1nmz-python3.8-jaraco-functools-3.0.1.drv
  /nix/store/2pf8bav7qjkg178wf6a5ylbqz21v79ay-python3.8-fooapp-0.1.0.drv
building '/nix/store/pjvxqcp9av8jsl62sjn09rqnpann1nmz-python3.8-jaraco-functools-3.0.1.drv'...
Sourcing python-recompile-bytecode-hook.sh
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing pip-build-hook
Using pipBuildPhase
Using pipShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
unpacking sources
unpacking source archive /nix/store/iszp86g5fccjrvlk1xf7waai34m2i9a6-jaraco.functools-3.0.1.tar.gz
source root is jaraco.functools-3.0.1
setting SOURCE_DATE_EPOCH to timestamp 1587850974 of file jaraco.functools-3.0.1/setup.cfg
patching sources
configuring
no configure script, doing nothing
building
Executing pipBuildPhase
Removing pyproject.toml...
Creating a wheel...
WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /tmp/nix-build-python3.8-jaraco-functools-3.0.1.drv-0/jaraco.functools-3.0.1
Building wheels for collected packages: jaraco.functools
  Building wheel for jaraco.functools (setup.py) ... done
  Created wheel for jaraco.functools: filename=jaraco.functools-0.0.0-py3-none-any.whl size=6713 sha256=6ee9aa396b670157852642ba20ce1994e4363839a46ebd6f69c7e95033aad40b
  Stored in directory: /tmp/nix-build-python3.8-jaraco-functools-3.0.1.drv-0/pip-ephem-wheel-cache-lrkoy6lw/wheels/a0/dc/41/0243086d14e048dd360a92810a81b12ba2cb0964c740af6bbd
Successfully built jaraco.functools
Finished creating a wheel...
Finished executing pipBuildPhase
installing
Executing pipInstallPhase
/tmp/nix-build-python3.8-jaraco-functools-3.0.1.drv-0/jaraco.functools-3.0.1/dist /tmp/nix-build-python3.8-jaraco-functools-3.0.1.drv-0/jaraco.functools-3.0.1
Processing ./jaraco.functools-0.0.0-py3-none-any.whl
Requirement already satisfied: more-itertools in /nix/store/dvgb29m492ph7cyrl5r01fz6xampgzd4-python3.8-more-itertools-8.4.0/lib/python3.8/site-packages (from jaraco.functools==0.0.0) (8.4.0)
Installing collected packages: jaraco.functools
Successfully installed jaraco.functools-0.0.0
/tmp/nix-build-python3.8-jaraco-functools-3.0.1.drv-0/jaraco.functools-3.0.1
Finished executing pipInstallPhase
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/mzf6h8dnwxw5qxbjglqggbm1c1qykyp9-python3.8-jaraco-functools-3.0.1
strip is /nix/store/1ar9gcmvikdk1wij0cw4fnwvl5i07ag1-binutils-2.31.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/mzf6h8dnwxw5qxbjglqggbm1c1qykyp9-python3.8-jaraco-functools-3.0.1/lib
patching script interpreter paths in /nix/store/mzf6h8dnwxw5qxbjglqggbm1c1qykyp9-python3.8-jaraco-functools-3.0.1
checking for references to /tmp/nix-build-python3.8-jaraco-functools-3.0.1.drv-0/ in /nix/store/mzf6h8dnwxw5qxbjglqggbm1c1qykyp9-python3.8-jaraco-functools-3.0.1...
Executing pythonRemoveTestsDir
Finished executing pythonRemoveTestsDir
pythonCatchConflictsPhase
pythonRemoveBinBytecodePhase
pythonImportsCheckPhase
Executing pythonImportsCheckPhase
building '/nix/store/2pf8bav7qjkg178wf6a5ylbqz21v79ay-python3.8-fooapp-0.1.0.drv'...
Sourcing python-recompile-bytecode-hook.sh
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing pip-build-hook
Using pipBuildPhase
Using pipShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
unpacking sources
unpacking source archive /nix/store/yvd084abvfhw9a1z3dz18fmkdj72gfmg-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/pyproject.toml
patching sources
configuring
no configure script, doing nothing
building
Executing pipBuildPhase
Creating a wheel...
WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /tmp/nix-build-python3.8-fooapp-0.1.0.drv-0/source
    Preparing wheel metadata ... done
Building wheels for collected packages: fooapp
  Building wheel for fooapp (PEP 517) ... done
  Created wheel for fooapp: filename=fooapp-0.1.0-py3-none-any.whl size=985 sha256=10a24e3620aa86513f5d48e59c37a0b543d5cd49ec529df5cb20e66ed8a5e100
  Stored in directory: /tmp/nix-build-python3.8-fooapp-0.1.0.drv-0/pip-ephem-wheel-cache-9uzghopt/wheels/32/62/b1/ecb1251499c3844bc82b02af6ca5bd8a1ad8ca3edda2a878f4
Successfully built fooapp
Finished creating a wheel...
Finished executing pipBuildPhase
installing
Executing pipInstallPhase
/tmp/nix-build-python3.8-fooapp-0.1.0.drv-0/source/dist /tmp/nix-build-python3.8-fooapp-0.1.0.drv-0/source
Processing ./fooapp-0.1.0-py3-none-any.whl
ERROR: Could not find a version that satisfies the requirement jaraco.functools<4.0.0,>=3.0.1 (from fooapp==0.1.0) (from versions: none)
ERROR: No matching distribution found for jaraco.functools<4.0.0,>=3.0.1 (from fooapp==0.1.0)
builder for '/nix/store/2pf8bav7qjkg178wf6a5ylbqz21v79ay-python3.8-fooapp-0.1.0.drv' failed with exit code 1
error: build of '/nix/store/2pf8bav7qjkg178wf6a5ylbqz21v79ay-python3.8-fooapp-0.1.0.drv' failed
@cyraxjoe
Copy link
Contributor Author

I end up using the variable that was in place for the dependencies that required pyproject.toml.

Not sure if the alternative is to keep adding those. If so... then maybe I could create a PR with cyraxjoe@f6de689

@cyraxjoe
Copy link
Contributor Author

After realizing that those can be easily overridden with something like:

  poetry2nix.overrides.withDefaults (
    self: super: {
      jaraco-functools = super.jaraco-functools.overridePythonAttrs (
        old: {
          dontPreferSetupPy = true;
        }
      );

      tempora = super.tempora.overridePythonAttrs (
        old: {
          dontPreferSetupPy = true;
        }
      );

      portend = super.portend.overridePythonAttrs (
        old: {
          dontPreferSetupPy = true;
        }
      );
    }
  )

I'm thinking that maybe is more of a documentation thing, something like mentioning the potential need of explicitly preserving the pyproject.toml, alternatively a code solution could be to inspect pyproject.toml for any setuptools_scm[toml].

@adisbladis
Copy link
Member

adisbladis commented Aug 20, 2020

I'm not sure what are the implications of not removing the pyproject.toml

This setup.py removal thing is really a hack to reduce the number of dependencies not managed by Poetry.
Poetry doesn't currently manage build time dependencies, so if you have something like flit defined as your build-system the build will fail. That's why we default to removing it to try to avoid adding overrides for things that have both a pyproject.toml & a setup.py.

The true proper fix to all of this is to fix Poetry to manage dependencies used when building from source, though that's obviously a lot more work than hacking around the issue.

@adisbladis
Copy link
Member

Not sure if the alternative is to keep adding those. If so... then maybe I could create a PR with

For now that would be great!

@adisbladis
Copy link
Member

I'm thinking that maybe is more of a documentation thing, something like mentioning the potential need of explicitly preserving the pyproject.toml, alternatively a code solution could be to inspect pyproject.toml for any setuptools_scm[toml].

Re the "code solution" we cannot do that as that would require fetching the dependency at eval time and introspect the toml file, resulting in IFD (import from derivation) which has some pretty nasty implications.

Re documentation I agree, I would love to have a few tutorial style docs where we could explain these quite complicated things.

@mickours
Copy link

FYI there is a draft PR to support build-system requirements in Poetry: python-poetry/poetry#2794

@mickours
Copy link

Also related to #218 and #154

@adisbladis
Copy link
Member

Closed in #513

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants