Store project metadata in pyproject.toml

[gerritholl]

Feature Request

Is your feature request related to a problem? Please describe.

Satpy package metadata is currently spread and partly duplicated between pyproject.toml, setup.cfg, setup.py, conf.py, MANIFEST.IN, and various tool-specific configuration files.

PEP 621 defines how project metadata should be stored in pyproject.toml. This PEP was declared final on 2021-03-03. The community is migrating to pyproject.toml, which will deprecate setup.py, setup.cfg, and — eventually — project specific configuration files.

Describe the solution you'd like

I would like that Satpy stores as much as possible in pyproject.toml. I would like nothing else will be needed anymore.

Describe any changes to existing user workflow

Most users will not notice much, but there will be some changes to the build process.

Additional context

Status quo works. Tooling is not entirely ready yet. Setuptools does not yet support pyproject.toml, but ppsetuptools does and probably setuptools will sooner or later. Can be read by pytest already. Hopefully other tools like sphinx and flake8 will follow in migrating to pyproject.toml. Perhaps this issue should wait a bit still.

This issue applies equally to almost all other Pytroll projects.

[djhoese]

Most users will not notice much, but there will be some changes to the build process.

This would require users to have a modern version of setuptools installed. Also, what specifically would have to change in the build process? Shouldn't the actual commands to do the building be the same?

I've noticed that xarray puts all their stuff in setup.cfg:

https://github.com/pydata/xarray/blob/main/setup.cfg

How can/should this be done in pyproject.toml? Do you have an example?

[gerritholl]

Actually, setuptools doesn't even support it yet, but build dependencies can be defined in the [build-system] section of pyproject.toml, so pip install will pick those up if needed before doing anything. I don't know how it would work with conda, though. Maybe it's too early, and this issue should be seen as being still blocked.

I was playing around with pyproject.toml yesterday. With this content for pyproject.toml:

[project]
name = "trollstorm"
project_name = "trollstorm"
description = "Satellite-based storm research, detection, and nowcasting"
requires-python = ">=3.9"
classifiers = [
    "Programming Language :: Python",
    "Development Status :: 2 - Pre-Alpha",
    "Natural Language :: English",
    "Intended Audience :: Science/Research",
    "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
    "Operating System :: OS Independent",
    "Topic :: Scientific/Engineering :: Atmospheric Science",
    "Topic :: Scientific/Engineering :: Physics",
]
dynamic = ["version"]
readme = "README.rst"
license = "LICENSE"
keywords = ["nowcasting"]
packages = ["trollstorm", "trollstorm.tests"]  # replace when setuptools/ppsetuptools supports find

[[project.authors]]
name = "Gerrit Holl"
email = "gerrit.holl@dwd.de"

[project.urls]
repo = "https://github.com/gerritholl/trollstorm"

[project.optional-dependencies]
test = ["pytest", "pytest-cov"]
doc = ["sphinx-pyproject"]

[build-system]
requires = ["ppsetuptools", "setuptools>=42", "toml >= 0.10.1", "setuptools_scm[toml]>=3.4"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]
write_to = "trollstorm/version.py"

[tool.pytest.ini_options]
addopts = "-v --cov trollstorm --cov-report term-missing"
testpaths = "tests"

and this content for setup.py (I don't understand why it's necessary, but it appears to be):

from ppsetuptools import setup

setup()

With this, the package it's installable without pre-installing anything else. Pytest and setuptools_scm already read their configuration from pyproject.toml, but sphinx and flake8 don't (there exist bridge packages here too, but I found them too obscure). Installing this in a clean conda environment (conda create -n test python=3.9) with pip install -v results in pip first picking up the build dependencies and then running setup.py (apparently; I thought it would use the build-backend, but not really).

pip install .

Using pip 21.1.3 from /data/gholl/miniconda3/envs/py39test/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-8fqek20x
Created temporary directory: /tmp/pip-req-tracker-z0to7oja
Initialized build tracking at /tmp/pip-req-tracker-z0to7oja
Created build tracker: /tmp/pip-req-tracker-z0to7oja
Entered build tracker: /tmp/pip-req-tracker-z0to7oja
Created temporary directory: /tmp/pip-install-gc0k6jjw
Processing /home/gholl/checkouts/trollstorm
  Created temporary directory: /tmp/pip-req-build-foyhnfz2
  Added file:///home/gholl/checkouts/trollstorm to build tracker '/tmp/pip-req-tracker-z0to7oja'
  Created temporary directory: /tmp/pip-build-env-r15gyxqk
  Created temporary directory: /tmp/pip-standalone-pip-s_w43p2r
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Created temporary directory: /tmp/pip-standalone-pip-_t_qmx24
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
    Created temporary directory: /tmp/pip-modern-metadata-hd7fl7s5
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Source in /tmp/pip-req-build-foyhnfz2 has version 0.1.dev7+g862624e, which satisfies requirement trollstorm==0.1.dev7+g862624e from file:///home/gholl/checkouts/trollstorm
  Removed trollstorm==0.1.dev7+g862624e from file:///home/gholl/checkouts/trollstorm from build tracker '/tmp/pip-req-tracker-z0to7oja'
Created temporary directory: /tmp/pip-unpack-kbxv4yqv
Building wheels for collected packages: trollstorm
  Created temporary directory: /tmp/pip-wheel-kir93gre
  Destination directory: /tmp/pip-wheel-kir93gre
  Building wheel for trollstorm (PEP 517): started
  Building wheel for trollstorm (PEP 517): finished with status 'done'
  Created wheel for trollstorm: filename=trollstorm-0.1.dev7+g862624e-py3-none-any.whl size=15336 sha256=51a6cd14cd6d7f2507fc527a77f56d1100e50eef175f968f721ea31bc24c1b11
  Stored in directory: /tmp/pip-ephem-wheel-cache-8fqek20x/wheels/d2/f4/2c/e0f2d8a6426cfa3041438029b617336e1dca9885faf4dee438
Successfully built trollstorm
Installing collected packages: trollstorm

Successfully installed trollstorm-0.1.dev7+g862624e
Removed build tracker: '/tmp/pip-req-tracker-z0to7oja'

The verbose install log including stderr is too long to include inline (1844 lines), but includes all the details including the installation of ppsetuptools, which would be needed until setuptools natively supports pyproject.toml, which is probably soon. In either case, the user installing with pip needs to do nothing; I don't know what happens with conda, though. I don't know if I could use the test-dependencies from a GitHub runner directly.

pip-install-log.txt.gz

[djhoese]

This is really cool! Thanks for the example.

I don't know what happens with conda, though. I don't know if I could use the test-dependencies from a GitHub runner directly.

Do you mean for the conda-forge package? That would still have to be a separate dependency list and that is done manually. The only thing that matters as far as the setuptools/setup.py dependency list is whether or not the conda-forge bot can read it since it makes suggestions based on changes it detects.

For the github runner, are you saying get rid of our conda yaml that we use in the CI? That would be cool, but since conda packages don't necessarily have to match the PyPI package name I'm not sure we can guarantee this will work forever.

[gerritholl]

No, I didn't get rid of the conda yaml file. But I hope that as tools move to supporting pyproject.toml, we may eventually avoid such duplication of configuration. PEP 621 has only been final for four months, so I suppose projects still need some time to catch up.

[mraspaud]

That's really nice @gerritholl ! Some time ago I also looked a poetry which looks awesome and supporting pyproject.toml would allow us to use it. And if we do, then there is a tool to convert pyproject.toml to conda format: https://pypi.org/project/poetry2conda/

[gerritholl]

Closed by #2763 .