Add workflow to publish to pypi

[steven-murray]

No description provided.

[codecov]

Codecov Report

Merging #62 into master will increase coverage by 0.11%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #62      +/-   ##
==========================================
+ Coverage   65.41%   65.52%   +0.11%     
==========================================
  Files          22       23       +1     
  Lines        2495     2509      +14     
==========================================
+ Hits         1632     1644      +12     
- Misses        863      865       +2     

Impacted Files	Coverage Δ
src/hmf/__init__.py	85.71% <ø> (ø)
src/hmf/_internals/_cache.py	85.56% <ø> (ø)
src/hmf/_internals/_framework.py	89.41% <ø> (ø)
src/hmf/_internals/_utils.py	36.36% <ø> (ø)
src/hmf/alternatives/wdm.py	92.66% <ø> (ø)
src/hmf/cosmology/__init__.py	100.00% <ø> (ø)
src/hmf/cosmology/growth_factor.py	78.04% <ø> (ø)
src/hmf/density_field/__init__.py	100.00% <ø> (ø)
src/hmf/density_field/filters.py	68.05% <ø> (ø)
src/hmf/density_field/halofit.py	82.85% <ø> (ø)
... and 14 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a218f2a...c0a796e. Read the comment docs.

[webknjaz]

I checked out this branch and found out that adding use_scm_version = True to the setuptools.setup() invocation helps: pypa/setuptools-scm#414 (comment).

P.S. Why don't you use pep517package as shown in my tutorial? This would probably work w/o changes to setup.py.
P.P.S. You have code in setup.py, it's generally discouraged, you can migrate the metadata to setup.cfg and keep only import setuptools; setuptools.setup() (yes, just that!) in setup.py. Remember, it's not a Makefile. It's common to use tox/invoke/nox/make for automations like building, testing and publishing it's then easier to map that to CIs.

[steven-murray]

@webknjaz Thanks for helping out here. I probably should move to setup.cfg -- I have never really gotten the benefit of it over just putting the values into setup.py, but if it helps, I will!

I did actually start by trying pep517.build but it was giving me errors (something about missing parameters) and I couldn't find the relevant documentation to sort out the errors. That would be nicer though!

[webknjaz]

Ah, I see. Looking at the log, it's noticeable that it complains about the unspecified PEP 517 build backend: https://github.com/steven-murray/hmf/runs/694053891#step:6:19.

When using setuptools, just add build-backend = "setuptools.build_meta" under [build-system] in pyproject.toml: https://github.com/sanitizers/octomachinery/blob/master/pyproject.toml#L8.

This basically tells the build frontend tool (pip/pep517/flit/poetry/etc.) what it should import and call in order to build the project: https://www.python.org/dev/peps/pep-0517/#source-trees

[webknjaz]

Suggested change

	requires = ["setuptools>=42", "wheel", "setuptools_scm[toml]>=3.4"]
	requires = ["setuptools>=42", "wheel", "setuptools_scm[toml]>=3.4"]
	build-backend = "setuptools.build_meta"

[steven-murray]

Yup. I found that 🤦

Thanks!

[webknjaz]

The thing is that some frontends (like pip) implement a fallback and set it to setuptools.build_meta.__legacy__ and others don't.

[steven-murray]

@webknjaz Two things:

First -- thanks for the very easy to follow documentation here.

Second, if you have a moment to answer me -- in those instructions, the actual PyPI upload is only done when a tag is commited. I wonder how this works -- if I push a tag to a branch which is on a PR, will that commit also be uploaded to PyPI, even before I know if it's passed tests etc and merged?

[webknjaz]

You should add such files to gitignore instead of comitting.

[webknjaz]

FYI this action is already v2.

[webknjaz]

could also be python -m setuptools_scm ls

[webknjaz]

FYI you may also like this hook that Anthony has: https://github.com/asottile/setup-cfg-fmt#as-a-pre-commit-hook

[webknjaz]

This looks wrong...

[webknjaz]

This attr doesn't have a kebab-case alias like some of the others: ee4b1d7#r39336456 / https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata.

[webknjaz]

It is recommended to use console_scripts entrypoint instead of standalone in-repo scripts like you used to.

[webknjaz]

This completely defeats the purpose of having src-layout in the repo. Here, read some explanation: https://blog.ganssle.io/articles/2019/08/test-as-installed.html / https://hynek.me/articles/testing-packaging/.

[webknjaz]

@steven-murray I've added a few additional review comments on how to fix some of the broken things.

Second, if you have a moment to answer me -- in those instructions, the actual PyPI upload is only done when a tag is commited. I wonder how this works -- if I push a tag to a branch which is on a PR, will that commit also be uploaded to PyPI, even before I know if it's passed tests etc and merged?

GitHub has a system of internal events that get published to all the listeners (pre-repo webhook integrations, GitHub Apps, GitHub Actions that are actually built on top of GitHub Apps, and so on). By adding a GHA workflow to your repo you subscribe to some events that you specify there. In your case, it's push.

Now, this even happens whenever somebody does git push and GHA (among others) will receive an event and trigger a workflow(s). It's important to understand that this event will hit your workflow if it happens inside of your repo. Meaning that this git push should be done from the inside. It doesn't matter if it's related to a branch or a tag, both would equally produce this push event.

There's also a pull_request event that happens when somebody interacts with PRs. That's why you can often see a double amount of jobs in your PRs when the PR branch belongs to your repo — two events will happen on the same commit: push and pull_request. And this will trigger two separate workflows (or more, depending on your setup, of course).

When somebody forks your repo and pushes things there, your workflows won't receive those events. But if they open a PR against your repo, you'll get pull_request events triggering your workflows but no push events. This is because GHA is scoped to only access your things.

If you have multiple workflows, they will be triggered simultaneously. There's no way to say that one workflow depends on another. This is handy when you want to have something like linters run in parallel but not preventing your tests from running on linter violations.

OTOH inside a workflow, you may have multiple jobs and you can set up dependencies between them, using the needs setting.

Finally, within a job, each step depends on the previous one by default, unless changed with things like if: failure().

What does this leave us with? In order to ensure that your project is tested before publishing, you should put it in the same workflow as your tests and declare a needs relation. You can customize the publish conditions, using if setting on the step of job level.

You could also make sure to wait for the tests on your target commit manually and only then push a tag, this will generate a separate workflow run anyway. Some people switch from git push origin v1.0.0 to using a different event like release (it's triggered by creating a release on the GitHub UI).
Besides, there's also a possibility to have your very own custom event using repository_dispatch but triggering it requires hitting GitHub's API yourself so it may seem slightly harder.

I myself, like the idea of building the dists, then testing those dists (not the source in the project dir!), and only then publishing them to PyPI (those same dists from the step 1, instead of building them again w/o any guarantee that they will work the same way as the source you tested).
I have a state-of-the-art implementation of this idea @ https://github.com/ansible/pylibssh/blob/devel/.github/workflows/build-test-n-publish.yml.

Have fun :)

[steven-murray]

@webknjaz Thanks so much for that detailed response. I'm still getting used to what Github really means by "push" and "pull_request" etc (as in, specifically what events trigger what things), and that helped a lot.

While playing around I kind of arrived at this solution: test on every push/PR and also publish to Test PyPI on every push, then on pushes to master (i.e. merged PRs, since I have protections on master) I run an automatic semantic version tagger, then since that makes a push, I have another workflow which runs only on pushes that create a tag, which publishes to actual PyPI.

Coming to think of it, your example in the tutorial probably already does this for free with its if: statement (though I didn't understand the function that it uses there, so I was hesitant).

I'll also have a closer look through your state of the art implementation and try stealing some ideas :-)

Thanks again for all your help!

[webknjaz]

run an automatic semantic version tagger, then since that makes a push, I have another workflow which runs only on pushes that create a tag, which publishes to actual PyPI.

That only works if you don't use the built-in secrets.GITHUB_TOKEN because this one doesn't produce events.

I'll also have a closer look through your state of the art implementation and try stealing some ideas :-)

Have fun :) Just beware that it's probably more complicated than you need because it has to build like 20 OS-specific wheels which makes the matrix big.