feature: add support for ABI3 wheels #1091

mayeut · 2022-04-18T15:05:20Z

Closes #78

While #569 errors out on duplicate abi3 wheel with an updated error message, this PR skips compatible abi3 builds and do proper testing with all python versions as specified by the build selector.

real-world usage can be seen there: https://github.com/mayeut/psutil/actions/runs/2184123889

henryiii

This is much easier to review after checking "ignore whitespace" in GH. :) The extra indentation is a minor downside - I think we should see if we can pull out each step (build, test, etc) into functions, then call them - that will keep the whitespace from growing here, and keep the logic easier to read (if this, build this, if that, test that) instead of having a massive inline function, too. Not for this PR though.

cibuildwheel/util.py

joerick

Great stuff @mayeut. Thank you for this, it's been on my todo list for a while!

I enjoy how pragmatic it is, simpler than I'd have thought to do it.

The normal operation of cibuildwheel is to build from oldest Python to newest. Is that what we want in this case? I.e. the abi wheel will be built on the oldest Python that supports limited api.
Documentation - I suppose we could add a FAQ entry about limited API, and include how to do that within setup.py, since we don't allow passing the --py_limited_api option.

joerick · 2022-04-20T08:15:06Z

test/test_limited_api.py

+    setup_py_add=textwrap.dedent(
+        r"""
+        cmdclass = {}
+        extension_kwargs = {}
+        if sys.version_info[:2] >= (3, 8):
+            from wheel.bdist_wheel import bdist_wheel as _bdist_wheel
+
+            class bdist_wheel_abi3(_bdist_wheel):
+                def finalize_options(self):
+                    _bdist_wheel.finalize_options(self)
+                    self.root_is_pure = False
+
+                def get_tag(self):
+                    python, abi, plat = _bdist_wheel.get_tag(self)
+                    return python, "abi3", plat
+
+            cmdclass["bdist_wheel"] = bdist_wheel_abi3
+            extension_kwargs["define_macros"] = [("Py_LIMITED_API", "0x03080000")]
+            extension_kwargs["py_limited_api"] = True
+    """
+    ),
+    setup_py_extension_args_add="**extension_kwargs",
+    setup_py_setup_args_add="cmdclass=cmdclass",
+)


I suppose this is the way to get abi3 wheel from pip wheel/python -m build? Because we can't pass --py-limited-api? I wonder if there's a cleaner way to do it. I suppose we'd want to document this too.

A follow-up on this question - should a reference version of this specify in the tag that it's a "3.8-ABI" wheel, rather than using whatever it's being built by?

Most packages will hard code this into setup.py. A package either produces abi3 wheels or regular wheels, usually they don't select between the two, I think. So you'd hardcode in the define in the source code, and py_limited_api=True is passed to setup().

I tested something a bit more complicated than what most packages will do. As @henryiii mentioned, and I think so to, "a package either produces abi3 wheels or regular wheels, usually they don't select between the two" and it'll be hardcoded in the source code.
Even for some packages that still support cp27 or some that depends on an API being made stable in a version more recent than the minimum supported, the logic will probably hardcoded.
I really don't think it's cibuildwheel's place to document how to build an abi3 wheel. Only if workarounds were to arise, specific to cibuildwheel, those should be documented then.

I tested something a bit more complicated than what most packages will do.

Understood.

I really don't think it's cibuildwheel's place to document how to build an abi3 wheel. Only if workarounds were to arise, specific to cibuildwheel, those should be documented then.

The only thing that might be considered specific is the inability to pass --py-limited-api to bdist_wheel - most docs online still reference this. But since that calling method is deprecated, it does indeed make sense to document a more modern way across the ecosystem. Ideally, indeed, in packaging.python.org. Then we can link it from our docs.

cibuildwheel/util.py

cibuildwheel/linux.py

mayeut · 2022-04-23T09:16:50Z

The normal operation of cibuildwheel is to build from oldest Python to newest. Is that what we want in this case? I.e. the abi wheel will be built on the oldest Python that supports limited api.

I think that's what we want in this case. This way, when a new version of CPython is available, it'll be tested but it won't change the build so I'd say it allows more reproducibility in the long run.

Documentation - I suppose we could add a FAQ entry about limited API, and include how to do that within setup.py, since we don't allow passing the --py_limited_api option.

I really don't think it belongs in cibuildwheel. IMHO, it belongs in https://packaging.python.org/en/latest/ and cibuildwheel should reference an entry there but it does not exist yet.

cibuildwheel/linux.py

cibuildwheel/macos.py

cibuildwheel/windows.py

joerick · 2022-04-26T08:11:11Z

test/test_limited_api.py

+    setup_py_add=textwrap.dedent(
+        r"""
+        cmdclass = {}
+        extension_kwargs = {}
+        if sys.version_info[:2] >= (3, 8):
+            from wheel.bdist_wheel import bdist_wheel as _bdist_wheel
+
+            class bdist_wheel_abi3(_bdist_wheel):
+                def finalize_options(self):
+                    _bdist_wheel.finalize_options(self)
+                    self.root_is_pure = False
+
+                def get_tag(self):
+                    python, abi, plat = _bdist_wheel.get_tag(self)
+                    return python, "abi3", plat
+
+            cmdclass["bdist_wheel"] = bdist_wheel_abi3
+            extension_kwargs["define_macros"] = [("Py_LIMITED_API", "0x03080000")]
+            extension_kwargs["py_limited_api"] = True
+    """
+    ),
+    setup_py_extension_args_add="**extension_kwargs",
+    setup_py_setup_args_add="cmdclass=cmdclass",
+)


I tested something a bit more complicated than what most packages will do.

Understood.

I really don't think it's cibuildwheel's place to document how to build an abi3 wheel. Only if workarounds were to arise, specific to cibuildwheel, those should be documented then.

The only thing that might be considered specific is the inability to pass --py-limited-api to bdist_wheel - most docs online still reference this. But since that calling method is deprecated, it does indeed make sense to document a more modern way across the ecosystem. Ideally, indeed, in packaging.python.org. Then we can link it from our docs.

henryiii · 2022-04-27T13:56:24Z

LGTM

joerick · 2022-04-29T11:02:30Z

I'm currently thinking how best to document this...

This extends the mechanism introduced in pypa#1091 for `abi3` wheels. Most of the mentions to `abi3` have been removed and replaced by a more generic `compatible_wheel`. This allows to build a wheel `foo-0.1-py3-none-win_amd64.whl` only once and still test with every configured python.

* feature: add support for `py3-none-{platform}` wheels This extends the mechanism introduced in #1091 for `abi3` wheels. Most of the mentions to `abi3` have been removed and replaced by a more generic `compatible_wheel`. This allows to build a wheel `foo-0.1-py3-none-win_amd64.whl` only once and still test with every configured python. * Add integration test for py3-none abi wheels * Fix expected_wheels for py3-none abi * Limit test to three pythons and check for certain log messages * chore: add some comments to explain filter process Co-authored-by: Joe Rickerby <joerick@mac.com> Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>

henryiii approved these changes Apr 18, 2022

View reviewed changes

henryiii reviewed Apr 18, 2022

View reviewed changes

cibuildwheel/util.py Outdated Show resolved Hide resolved

joerick reviewed Apr 20, 2022

View reviewed changes

mayeut force-pushed the abi3 branch 2 times, most recently from 7a4a5d2 to 8cc44fc Compare April 23, 2022 09:54

joerick reviewed Apr 26, 2022

View reviewed changes

feature: add support for ABI3 wheels

e3ff027

mayeut force-pushed the abi3 branch from 8cc44fc to e3ff027 Compare April 26, 2022 20:38

Merge remote-tracking branch 'origin/main' into abi3

acc0796

joerick added 2 commits April 29, 2022 13:32

Space out the log statements a little

46702f5

Add tips+tricks entry for ABI3 building

7599841

joerick mentioned this pull request Apr 29, 2022

Error on creation of wheel with a name that already exists, and add Py_LIMITED_API to the docs' FAQ #569

Closed

3 tasks

joerick merged commit 61bcac8 into pypa:main Apr 29, 2022

mayeut deleted the abi3 branch April 29, 2022 21:12

mayeut mentioned this pull request Jun 19, 2022

feature: add support for py3-none-{platform} wheels #1151

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: add support for ABI3 wheels #1091

feature: add support for ABI3 wheels #1091

mayeut commented Apr 18, 2022

henryiii left a comment

joerick left a comment

joerick Apr 20, 2022

joerick Apr 20, 2022 •

edited

Loading

henryiii Apr 21, 2022

mayeut Apr 23, 2022

joerick Apr 26, 2022

mayeut commented Apr 23, 2022

joerick Apr 26, 2022

henryiii commented Apr 27, 2022

joerick commented Apr 29, 2022

feature: add support for ABI3 wheels #1091

feature: add support for ABI3 wheels #1091

Conversation

mayeut commented Apr 18, 2022

henryiii left a comment

Choose a reason for hiding this comment

joerick left a comment

Choose a reason for hiding this comment

joerick Apr 20, 2022

Choose a reason for hiding this comment

joerick Apr 20, 2022 • edited Loading

Choose a reason for hiding this comment

henryiii Apr 21, 2022

Choose a reason for hiding this comment

mayeut Apr 23, 2022

Choose a reason for hiding this comment

joerick Apr 26, 2022

Choose a reason for hiding this comment

mayeut commented Apr 23, 2022

joerick Apr 26, 2022

Choose a reason for hiding this comment

henryiii commented Apr 27, 2022

joerick commented Apr 29, 2022

joerick Apr 20, 2022 •

edited

Loading