Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data files are missing from wheel when building from git archive #49

Closed
mgorny opened this issue Jun 25, 2022 · 10 comments · Fixed by #53
Closed

Data files are missing from wheel when building from git archive #49

mgorny opened this issue Jun 25, 2022 · 10 comments · Fixed by #53

Comments

@mgorny
Copy link

mgorny commented Jun 25, 2022

I suspect something goes wrong around setuptools_scm.

To reproduce:

wget https://github.com/abravalheri/validate-pyproject/archive/refs/tags/v0.9.tar.gz
tar -xf v0.9.tar.gz
cd validate-pyproject-0.9
export SETUPTOOLS_SCM_PRETEND_VERSION=0.9
python -m build
unzip -l dist/validate_pyproject-0.9-py3-none-any.whl

At least the JSON schemas are missing.

@abravalheri
Copy link
Owner

Yes, setuptools-scm (at least out of the box) requires the project to be cloned instead of retrieved from the tar ball.

Have you ever experienced a similar issue with other projects using setuptools-scm (e.g. https://github.com/jaraco/zipp, https://github.com/jaraco/keyring, https://github.com/python/importlib_metadata etc..)? In those cases what is the alternative used?

@mgorny
Copy link
Author

mgorny commented Jun 25, 2022

Actually, no, I don't recall seeing a similar issue over the last few months, at least. Many projects require SETUPTOOLS_SCM_PRETEND_VERSION but they get around installing data files somehow. To be honest, I was hoping you'd have a better idea how to solve it ;-).

I mean, I've just switched to the pypi sdist since it works in this case (though I don't feel exactly good about relying on the redirector service for old URLs which sounds kinda temporary by definition) but I still think it's worth fixing.

The simplest idea I have is adding a minimal MANIFEST.in:

recursive-include src *.json *.template *.rst LICENSE

Ofc, you'd need to add more than if you wanted to have sdist working as well.

@abravalheri
Copy link
Owner

Since you are building from a tar.gz what is the biggest advantage in using GitHub's one vs the actual sdist?

I suspect that a better to get around this problem would be defining package_data, but that is a shame... setuptools-scm is supposed to facilitate this scenario.

@mgorny
Copy link
Author

mgorny commented Jun 25, 2022

Since you are building from a tar.gz what is the biggest advantage in using GitHub's one vs the actual sdist?

Well, two I can think of:

  1. We pretty much default to using GitHub archives everywhere because sdist tarballs usually tend to be missing tests, docs etc. What's even worse is that in some cases only single files are missing, and it's hard to notice that something is broken because of that. So as a policy, we go for "use GitHub archive whenever possible since it's less likely to miss important files".
  2. The modern PyPI URLs are unusable to packagers, as the hash part is unpredictable (and we really, really don't want to have to hardwire it for every release), so we rely on the "old URL" redirector service. While it has worked all these years, it's not exactly something I'd like to rely on staying alive and working forever.

@mgorny
Copy link
Author

mgorny commented Jun 25, 2022

I suspect that a better to get around this problem would be defining package_data, but that is a shame... setuptools-scm is supposed to facilitate this scenario.

Actually, I thought nowadays include_package_data was supposed to grab everything from package directories anyway.

@abravalheri
Copy link
Owner

Actually, I thought nowadays include_package_data was supposed to grab everything from package directories anyway.

Unfortunately not yet. include_package_data will grab everything from the sdist, but the sdist needs either a MANIFEST.in or a VCS plugin to be built.

I will have a look on this and probably add package_data, but that is low in my priority list :P (so it might take a while).


Side note, it seems that we need some sort of software that downloads files from PyPI by navigating the JSON API and getting the correct hashes... I noticed that before in other projects, so I may decide some day to write one.

The modern PyPI URLs are unusable to packagers, as the hash part is unpredictable (and we really, really don't want to have to hardwire it for every release), so we rely on the "old URL" redirector service. While it has worked all these years, it's not exactly something I'd like to rely on staying alive and working forever.

@wwuck
Copy link
Contributor

wwuck commented Jun 27, 2022

https://github.com/pypa/setuptools_scm/blob/main/CHANGELOG.rst

setuptools_scm v7 recent release mentions support for git archives. Are you using the latest version?

Otherwise there is also https://pypi.org/project/setuptools-scm-git-archive/ which might help here?

@abravalheri
Copy link
Owner

abravalheri commented Jul 14, 2022

setuptools_scm v7 recent release mentions support for git archives. Are you using the latest version?

@wwuck I suspect that the support for git archives in setuptools_scm only fixes the unknown version problem, not the file listing problems.

(I just did some experiments and that was my conclusion, so it seems that the only reliable way of building packages from source is indeed the sdist).

@abravalheri
Copy link
Owner

I have opened pypa/setuptools-scm#752. Depending on how it goes, maybe we just need to add the git archive setup.

@cclauss
Copy link
Contributor

cclauss commented Sep 20, 2022

The approach in the original post seems overly complicated. Why not just use https://pypa.github.io/pipx and do... ?

pipx install --force https://github.com/abravalheri/validate-pyproject.git@v0.9
or
pipx install --force 'https://github.com/abravalheri/validate-pyproject.git@v0.9[all]'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants