From f7ccb485745a6e3eac07f5e60849c91948634a6e Mon Sep 17 00:00:00 2001 From: Hugo van Kemenade Date: Sun, 12 Nov 2023 13:48:34 +0200 Subject: [PATCH 01/16] Infra: Add `make htmllive` to rebuild and reload HTML files in your browser (#3521) Add 'make htmllive' to rebuild and reload HTML files in your browser --- Makefile | 6 ++++++ requirements.txt | 2 ++ 2 files changed, 8 insertions(+) diff --git a/Makefile b/Makefile index 43f72f8a988..42844084f76 100644 --- a/Makefile +++ b/Makefile @@ -26,6 +26,12 @@ html: venv htmlview: html $(PYTHON) -c "import os, webbrowser; webbrowser.open('file://' + os.path.realpath('build/index.html'))" +## htmllive to rebuild and reload HTML files in your browser +.PHONY: htmllive +htmllive: SPHINXBUILD = $(VENVDIR)/bin/sphinx-autobuild +htmllive: SPHINXERRORHANDLING = --re-ignore="/\.idea/|/venv/|/pep-0000.rst|/topic/" +htmllive: html + ## dirhtml to render PEPs to "index.html" files within "pep-NNNN" directories .PHONY: dirhtml dirhtml: BUILDER = dirhtml diff --git a/requirements.txt b/requirements.txt index bf7a8470ede..1c716bc5d27 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,6 +5,8 @@ Pygments >= 2.9.0 Sphinx >= 5.1.1, != 6.1.0, != 6.1.1 docutils >= 0.19.0 +sphinx-autobuild + # For tests pytest pytest-cov From eb71e6a42274ec579265538b56d52618e8020b4e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cristi=C3=A1n=20Maureira-Fredes?= Date: Tue, 14 Nov 2023 12:00:51 +0100 Subject: [PATCH 02/16] PEP 733: Adjust binding generator for Qt (GH-3533) PySide was listed as a binding generator for Qt, but PySide is the final project. The name of the used binding generator that PySide uses, is Shiboken. --- peps/pep-0733.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0733.rst b/peps/pep-0733.rst index 409cdef5eba..333021c890a 100644 --- a/peps/pep-0733.rst +++ b/peps/pep-0733.rst @@ -208,7 +208,8 @@ Examples are `pybind11 `__ and `nanobind `__ for C++, `PyO3 `__ for Rust, -`PySide `__ for Qt, +`Shiboken `__ used by +PySide for Qt, `PyGObject `__ for GTK, `Pygolo `__ for Go, `JPype `__ for Java, From bf0b381704265f2d013255744a92671895287332 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Tue, 14 Nov 2023 07:46:11 -0800 Subject: [PATCH 03/16] PEP 387: Use the @deprecated decorator if possible (#3525) PEP 702 says that the deprecation policy should be updated to encourage use of @deprecated for deprecations. --- peps/pep-0387.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/peps/pep-0387.rst b/peps/pep-0387.rst index 1915dc648c9..068684b5d27 100644 --- a/peps/pep-0387.rst +++ b/peps/pep-0387.rst @@ -160,6 +160,10 @@ several releases: coexist for many releases [#warnings]_. The warning message should include the release the incompatibility is expected to become the default and a link to an issue that users can post feedback to. + When feasible, also change `typeshed `_ + to add the ``@deprecated`` decorator (see :pep:`702`) to the deprecated API, + so that users of static type checkers have another way to learn + about the deprecation. For C API, a compiler warning generated by the ``Py_DEPRECATED`` macro is also acceptable. From 6643c37d31ceee1476a270eb55654519b306b1ca Mon Sep 17 00:00:00 2001 From: Sebastian Rittau Date: Tue, 14 Nov 2023 19:41:48 +0100 Subject: [PATCH 04/16] Infra: Improve print style (#3486) --- pep_sphinx_extensions/pep_theme/static/mq.css | 62 ++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-) diff --git a/pep_sphinx_extensions/pep_theme/static/mq.css b/pep_sphinx_extensions/pep_theme/static/mq.css index 7977585e61f..2609a959ebd 100644 --- a/pep_sphinx_extensions/pep_theme/static/mq.css +++ b/pep_sphinx_extensions/pep_theme/static/mq.css @@ -88,8 +88,12 @@ img { max-width: 100% !important; } + /* Page margins according to DIN 5008, leaves space for punched holes. */ @page { - margin: 0.5cm; + margin-top: 2cm; + margin-bottom: 2cm; + margin-left: 2.5cm; + margin-right: 2.5cm; } p, h2, @@ -102,4 +106,60 @@ h3 { page-break-after: avoid; } + h1 { + font-size: 18pt; + font-weight: bold; + text-align: center; + } + h2, details > summary { + font-size: 15pt; + font-weight: normal; + } + h3 { + font-size: 13pt; + font-weight: normal; + } + h4 { + font-size: 10pt; + font-weight: 600; + } + a, abbr { + text-decoration: none; + } + + details { + display: none; + } + details[open] { + display: block; + } + + h1.page-title:first-child { + margin-top: 0; + } + + section#pep-page-section { + display: flex; + justify-content: center; + padding: 0; + margin: 0 auto; + } + + section#pep-page-section > header, + nav#pep-sidebar { + display: none; + } + + section#pep-page-section > article { + float: none; + max-width: 17.5cm; + width: auto; + margin: 0; + padding: 0; + } + + /* This blocks a small portion on each page. */ + readthedocs-flyout { + display: none; + } } From f7b84aab4ea74cd59c7a0ae1e0d9a7801a4c1fb1 Mon Sep 17 00:00:00 2001 From: Josh Cannon Date: Tue, 14 Nov 2023 13:22:34 -0600 Subject: [PATCH 05/16] PEP 712: Update with suggestions/clarifications from discussion (#3496) --- peps/pep-0712.rst | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/peps/pep-0712.rst b/peps/pep-0712.rst index cbf9f51b5c5..e9a6e9673e7 100644 --- a/peps/pep-0712.rst +++ b/peps/pep-0712.rst @@ -93,6 +93,9 @@ Example .. code-block:: python + def str_or_none(x: Any) -> str | None: + return str(x) if x is not None else None + @dataclasses.dataclass class InventoryItem: # `converter` as a type (including a GenericAlias). @@ -209,11 +212,42 @@ users of converters are likely to encounter. Such pitfalls include: * Forgetting to convert values in the bodies of user-defined ``__setattr__`` in non-frozen dataclasses. + +Additionally, potentially confusing pattern matching semantics should be covered: + +.. code-block:: python + + @dataclass + class Point: + x: int = field(converter=int) + y: int + + match Point(x="0", y=0): + case Point(x="0", y=0): # Won't be matched + ... + case Point(): # Will be matched + ... + case _: + ... + +However it's worth noting this behavior is true of any type that does conversion +in its initializer, and type-checkers should be able to catch this pitfall: + +.. code-block:: python + + match int("0"): + case int("0"): # Won't be matched + ... + case _: # Will be matched + ... + Reference Implementation ======================== The attrs library `already includes `__ a ``converter`` -parameter containing converter semantics. +parameter exhibiting the same converter semantics (converting in the +initializer and on attribute setting) when using the ``@define`` class +decorator. CPython support is implemented on `a branch in the author's fork `__. From c0870debbcf01eacd798fdbeacef98bde613285e Mon Sep 17 00:00:00 2001 From: Russell Keith-Magee Date: Thu, 16 Nov 2023 09:13:49 +0800 Subject: [PATCH 06/16] PEP 730: More revisions and clarifications (#3529) --- peps/pep-0730.rst | 156 +++++++++++++++++++++++++++++++++------------- 1 file changed, 113 insertions(+), 43 deletions(-) diff --git a/peps/pep-0730.rst b/peps/pep-0730.rst index 50ebc93694f..1ddb37317fe 100644 --- a/peps/pep-0730.rst +++ b/peps/pep-0730.rst @@ -63,6 +63,14 @@ significant platform differences between iOS and macOS. iOS code is compiled for compatibility against a minimum iOS version. +Apple frequently refers to "iPadOS" in their marketing material. However, from a +development perspective, there is no discernable difference between iPadOS and +iOS. A binary that has been compiled for the ``iphoneos`` or ``iphonesimulator`` +ABIs can be deployed on iPad. + +Other Apple platforms, such as tvOS, watchOS, and visionOS, use different ABIs, +and are not covered by this PEP. + POSIX compliance ---------------- @@ -111,26 +119,16 @@ a Framework. This also means that the common assumption that a Python module can construct the location of a binary module by using the ``__file__`` attribute of the Python module no longer holds. -As with macOS, compiling a dynamic library requires the use of the ``--undefined -dynamic_lookup`` option to avoid linking libPython into every binary module. -This option currently raises a deprecation warning when it is used. This warning -*was* previously raised on macOS builds as well; however, responses from Apple -staff suggests this was unintentional, and they `did not intend to break the -CPython ecosystem by removing this option -`__. It is difficult to judge -whether iOS support would fall under the same umbrella. - -Distribution ------------- - -Adding iOS as a Tier 3 platform only requires adding support for compiling an -iOS-compatible build from an unpatched CPython code checkout. It does not -require production of officially distributed iOS artefacts for use by end-users. - -If/when iOS is updated to Tier 2 or 1 support, there should be a process for -producing iOS distribution artefacts. This could be in the form of an "embedded -distribution" analogous to the Windows embedded distribution, or as a Cocoapod -or Swift Package that could be added to an Xcode project. +As with macOS, compiling a binary module that is accessible from a +statically-linked build of Python requires the use of the ``--undefined +dynamic_lookup`` option to avoid linking ``libpython3.x`` into every binary +module. However, on iOS, this compiler flag raises a deprecation warning when it +is used. A warning from this flag has been observed on macOS as well - however, +responses from Apple staff suggest that they `do not intend to break the CPython +ecosystem by removing this option +`__. As +Python does not currently have a notable presence on iOS, it is difficult to +judge whether iOS usage of this flag would fall under the same umbrella. Console and interactive usage ----------------------------- @@ -141,8 +139,7 @@ not be considered a goal of this work. Mobile devices (including iOS) do not provide a TTY-style console. They do not provide ``stdin``, ``stdout`` or ``stderr``. iOS provides a system log, and it is possible to install a redirection so that all ``stdout`` and ``stderr`` -content is redirected to the system log; but there is no analog for -``stdin``. +content is redirected to the system log; but there is no analog for ``stdin``. In addition, iOS places restrictions on downloading additional code at runtime (as this behavior would be functionally indistinguishable from trying to work @@ -179,7 +176,7 @@ devices. ``platform`` '''''''''''' -``platform`` will be modified to support returning iOS-specific details. Most of +``platform`` will be modified to support returning iOS-specific details. Most of the values returned by the ``platform`` module will match those returned by ``os.uname()``, with the exception of: @@ -193,9 +190,10 @@ In addition, a ``platform.ios_ver()`` method will be added. This mirrors ``ios_ver()`` will return a namedtuple that contains the following: * ``release`` - the iOS version, as a string (e.g., ``"16.6.1"``). -* ``min_release`` - the minimum supported iOS version, as a string (e.g., ``"12.0"``) -* ``model`` - the model identifier of the device, as a string (e.g., ``"iPhone13,2"``). - On simulators, this will return ``"iPhoneSimulator"``. +* ``min_release`` - the minimum supported iOS version, as a string (e.g., + ``"12.0"``) +* ``model`` - the model identifier of the device, as a string (e.g., + ``"iPhone13,2"``). On simulators, this will return ``"iPhoneSimulator"``. * ``is_simulator`` - a boolean indicating if the device is a simulator. ``os`` @@ -215,8 +213,8 @@ This approach treats the ``os`` module as a "raw" interface to system APIs, and ''''''''''''' The ``sysconfig`` module will use the minimum iOS version as part of -``sysconfig.get_platform()`` (e.g., ``"ios-12.0-iphoneos-arm64"``). -The ``sysconfigdata_name`` and Config makefile will follow the same patterns as +``sysconfig.get_platform()`` (e.g., ``"ios-12.0-iphoneos-arm64"``). The +``sysconfigdata_name`` and Config makefile will follow the same patterns as existing platforms (using ``sys.platform``, ``sys.implementation._multiarch`` etc.) to construct identifiers. @@ -241,6 +239,46 @@ unique Framework name by using the full module name as the framework name (i.e., ``foo.bar._whiz.framework``). A framework is a directory; the finder will look for ``_whiz.dylib`` in that directory. +Compilation +----------- + +The only binary format that will be supported is a dynamically-linkable +``libpython3.x.dylib``, packaged in an iOS-compatible framework format. While +the ``--undefined dynamic_lookup`` compiler option currently works, the +long-term viability of the option cannot be guaranteed. Rather than rely on a +compiler flag with an uncertain future, binary modules on iOS will be linked +with ``libpython3.x.dylib``. This means iOS binary modules will not be loadable +by an executable that has been statically linked against ``libpython3.x.a``. +Therefore, a static ``libpython3.x.a`` iOS library will not be supported. This +is the same pattern used by CPython on Windows. + +Building CPython for iOS requires the use of the cross-platform tooling in +CPython's ``configure`` build system. A single ``configure``/``make``/``make +install`` pass will produce a ``Python.framework`` artefact that can be used on +a single ABI and architecture. + +Additional tooling will be required to merge the ``Python.framework`` builds for +multiple architectures into a single "fat" library. Tooling will also be +required to merge multiple ABIs into the ``XCframework`` format that Apple uses +to distribute multiple frameworks for different ABIs in a single bundle. + +An Xcode project will be provided for the purpose of running the CPython test +suite. Tooling will be provided to automate the process of compiling the test +suite binary, start the simulator, install the test suite, and execute it. + +Distribution +------------ + +Adding iOS as a Tier 3 platform only requires adding support for compiling an +iOS-compatible build from an unpatched CPython code checkout. It does not +require production of officially distributed iOS artefacts for use by end-users. + +If/when iOS is updated to Tier 2 or 1 support, the tooling used to generate an +``XCframework`` package could be used to produce an iOS distribution artefact. +This could then be distributed as an "embedded distribution" analogous to the +Windows embedded distribution, or as a CocoaPod or Swift package that could be +added to an Xcode project. + CI resources ------------ @@ -268,8 +306,8 @@ iOS wheels will use tags: * ``ios_12_0_iphonesimulator_x86_64`` In these tags, "12.0" is the minimum supported iOS version. As with macOS, the -tag will incorporate the minimum iOS version that is selected when the wheel -is compiled; a wheel compiled with a minimum iOS version of 15.0 would use the +tag will incorporate the minimum iOS version that is selected when the wheel is +compiled; a wheel compiled with a minimum iOS version of 15.0 would use the ``ios_15_0_iphone*`` tags. At time of writing, iOS 12.0 exposes most significant iOS features, while reaching near 100% of devices; this will be used as a floor for iOS version matching. @@ -389,8 +427,9 @@ The decision was made to use ``arm64-apple-ios`` and 1. The ``autoconf`` toolchain already contains support for ``ios`` as a platform in ``config.sub``; it's only the simulator that doesn't have a representation. 2. The third part of the host triple is used as ``sys.platform``. -3. When Apple's own tools reference CPU architecture, they use ``arm64``, and the - GNU tooling usage of the architecture isn't visible outside the build process. +3. When Apple's own tools reference CPU architecture, they use ``arm64``, and + the GNU tooling usage of the architecture isn't visible outside the build + process. 4. When Apple's own tools reference simulator status independent of the OS (e.g., in the naming of Swift submodules), they use a ``-simulator`` suffix. 5. While *some* iOS packages will use Rust, *all* iOS packages will use Apple's @@ -407,10 +446,10 @@ It would be conceptually possible to offer an analogous "universal" iOS wheel format. However, this PEP does not use this approach, for 2 reasons. Firstly, the experience on macOS, especially in the numerical Python ecosystem, -has been that universal wheels can be exceedingly difficult to accommodate. While -native macOS libraries maintain strong multi-platform support, and Python itself -has been updated, the vast majority of upstream non-Python libraries do not -provide multi-architecture build support. As a result, compiling universal +has been that universal wheels can be exceedingly difficult to accommodate. +While native macOS libraries maintain strong multi-platform support, and Python +itself has been updated, the vast majority of upstream non-Python libraries do +not provide multi-architecture build support. As a result, compiling universal wheels inevitably requires multiple compilation passes, and complex decisions over how to distribute header files for different architectures. As a result of this complexity, many popular projects (including NumPy and Pillow) do not @@ -420,10 +459,10 @@ wheels. Secondly, historical experience is that iOS would require a much more fluid "universal" definition. In the last 10 years, there have been *at least* 5 different possible interpretations of "universal" that would apply to iOS, -including various combinations of armv6, armv7, armv7s, arm64, x86 and -x86_64 architectures, on device and simulator. If defined right now, -"universal-iOS" would likely include x86_64 and arm64 on simulator, and arm64 on -device; however, the pending deprecation of x86_64 hardware would add another +including various combinations of armv6, armv7, armv7s, arm64, x86 and x86_64 +architectures, on device and simulator. If defined right now, "universal-iOS" +would likely include x86_64 and arm64 on simulator, and arm64 on device; +however, the pending deprecation of x86_64 hardware would add another interpretation; and there may be a need to add arm64e as a new device architecture in the future. Specifying iOS wheels as single-platform-only means the Python core team can avoid an ongoing standardization discussion about the @@ -441,6 +480,37 @@ At present, no binary merging is required, as there is only one on-device architecture, and simulator binaries are not considered to be distributable artefacts, so only one architecture is needed to build an app for a simulator. +Supporting static builds +------------------------ + +While the long-term viability of the ``--undefined dynamic_lookup`` option +cannot be guaranteed, the option does exist, and it works. One option would be +to ignore the deprecation warning, and hope that Apple either reverses the +deprecation decision, or never finalizes the deprecation. + +Given that Apple's decision-making process is entirely opaque, this would be, at +best, a risky option. When combined with the fact that the broader iOS +development ecosystem encourages the use of frameworks, there are no legacy uses +of a static library to consider, and the only benefit to a statically-linked iOS +``libpython3.x.a`` is a very slightly reduced app startup time, omitting support +for static builds of ``libpython3.x`` seems a reasonable compromise. + +It is worth noting that there has been some discussion on `an alternate approach +to linking on macOS `__ that +would remove the need for the ``--undefined dynamic_lookup`` option, although +discussion on this approach appears to have stalled due to complications in +implementation. If those complications were to be overcome, it is highly likely +that the same approach *could* be used on iOS, which *would* make a statically +linked ``libpython3.x.a`` plausible. + +The decision to link binary modules against ``libpython3.x.dylib`` would +complicate the introduction of static ``libpython3.x.a`` builds in the future, +as the process of moving to a different binary module linking approach would +require a clear way to differentate "dynamically-linked" iOS binary modules from +"static-compatible" iOS binary modules. However, given the lack of tangible +benefits of a static ``libpython3.x.a``, it seems unlikely that there will be +any requirement to make this change. + Interactive/REPL mode --------------------- @@ -471,9 +541,9 @@ pose a significant impediment to adoption or long term maintenance. On-device testing ----------------- -CI testing on simulators can be accommodated reasonably easily. -On-device testing is much harder, as availability of device farms that could be -configured to provide Buildbots or Github Actions runners is limited. +CI testing on simulators can be accommodated reasonably easily. On-device +testing is much harder, as availability of device farms that could be configured +to provide Buildbots or Github Actions runners is limited. However, on device testing may not be necessary. As a data point - Apple's Xcode Cloud solution doesn't provide on-device testing. They rely on the fact that the From ca28a17523239255128ccdc20a22e158b512b116 Mon Sep 17 00:00:00 2001 From: "Erlend E. Aasland" Date: Thu, 16 Nov 2023 10:37:18 +0100 Subject: [PATCH 07/16] PEP 7: Add C pre-processor macro style recommendations (#3516) --- peps/pep-0007.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/peps/pep-0007.rst b/peps/pep-0007.rst index d88373f29b3..fc86f045067 100644 --- a/peps/pep-0007.rst +++ b/peps/pep-0007.rst @@ -135,6 +135,25 @@ Code lay-out return 0; /* "Forgive" adding a __dict__ only */ } +* Vertically align line continuation characters in multi-line macros. + +* Macros intended to be used as a statement should use the + ``do { ... } while (0)`` macro idiom, + without a final semicolon. + Example:: + + #define ADD_INT_MACRO(MOD, INT) \ + do { \ + if (PyModule_AddIntConstant((MOD), (#INT), (INT)) < 0) { \ + goto error; \ + } \ + } while (0) + + // To be used like a statement with a semicolon: + ADD_INT_MACRO(m, SOME_CONSTANT); + +* ``#undef`` file local macros after use. + * Put blank lines around functions, structure definitions, and major sections inside functions. @@ -170,6 +189,9 @@ Naming conventions * Macros should have a MixedCase prefix and then use upper case, for example: ``PyString_AS_STRING``, ``Py_PRINT_RAW``. +* Macro parameters should use ``ALL_CAPS`` style, + so they are easily distinguishable from C variables and struct members. + Documentation Strings ===================== From ae7f8fb9309406fe451e6a19523a7fab5b450d90 Mon Sep 17 00:00:00 2001 From: Guido van Rossum Date: Tue, 21 Nov 2023 11:52:19 -0800 Subject: [PATCH 08/16] PEP 731: List members explicitly and mark Active (#3540) Also drop the sentence from the abstract implying the WG members are the PEP authors (the correspondence is now a mere coincidence). --- peps/pep-0731.rst | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/peps/pep-0731.rst b/peps/pep-0731.rst index a53911362fa..090edb10237 100644 --- a/peps/pep-0731.rst +++ b/peps/pep-0731.rst @@ -6,11 +6,12 @@ Author: Guido van Rossum , Steve Dower , Irit Katriel Discussions-To: https://discuss.python.org/t/pep-731-c-api-working-group-charter/36117 -Status: Draft +Status: Active Type: Process Topic: Governance Created: 11-Oct-2023 Post-History: `13-Oct-2023 `__ +Resolution: https://discuss.python.org/t/pep-731-c-api-working-group-charter/36117/12 Abstract ======== @@ -32,7 +33,6 @@ whether in the context of CPython, using an alternate Python implementation, or using a binding framework for other programming languages (such as C++ and Rust). The working group serves at the pleasure of the Python Steering Council. -The working group's members are the listed authors of this PEP. This document serves as the working group's charter. Epigraph @@ -92,6 +92,18 @@ and publishing guidelines that the core developers can refer to. The "operations and process" section below describes how the working group operates and how it is governed. +Members +------- + +The members of the working group are: + +- Guido van Rossum +- Petr Viktorin +- Victor Stinner +- Steve Dower +- Irit Katriel + + Mandate ------- From bf89a7c5920b2e35748d4b26bd16ecf9a6fe7662 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Wed, 22 Nov 2023 17:28:29 -0800 Subject: [PATCH 09/16] PEP 729: Mark as Active (#3542) Co-authored-by: Hugo van Kemenade --- peps/pep-0729.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0729.rst b/peps/pep-0729.rst index 1c4adf262ce..344d965bb57 100644 --- a/peps/pep-0729.rst +++ b/peps/pep-0729.rst @@ -2,12 +2,13 @@ PEP: 729 Title: Typing governance process Author: Jelle Zijlstra , Shantanu Jain Discussions-To: https://discuss.python.org/t/pep-729-typing-governance-process/35362 -Status: Draft +Status: Active Type: Process Topic: Governance, Typing Created: 19-Sep-2023 Post-History: `04-Oct-2023 `__, `20-Sep-2023 `__ +Resolution: https://discuss.python.org/t/pep-729-typing-governance-process/35362/12 Abstract ======== From 76cf190e5e34bd0c7d443b17f49a262abc64477b Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Thu, 23 Nov 2023 11:45:52 -0500 Subject: [PATCH 10/16] PEP-8105: update with nominations and voter roll (#3543) --- peps/pep-8105.rst | 103 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 98 insertions(+), 5 deletions(-) diff --git a/peps/pep-8105.rst b/peps/pep-8105.rst index 254c720f4df..d5a7352452b 100644 --- a/peps/pep-8105.rst +++ b/peps/pep-8105.rst @@ -29,10 +29,10 @@ Director of Infrastructure, Ee Durbin, to administer the election. Schedule ======== -There will be a two-week nomination period, followed by a two-week +There was a two-week nomination period, followed by a two-week vote. -The nomination period will be: November 9, 2023 through `November 22, 2023 AoE +The nomination period was: November 9, 2023 through `November 22, 2023 AoE `_ [#note-aoe]_. The voting period will be: November 27, 2023 through `December 11, 2023 AoE @@ -45,9 +45,16 @@ Candidates Candidates must be nominated by a core team member. If the candidate is a core team member, they may nominate themselves. -Nominees (in alphabetical order): +Nominees (in alphabetical order by first name): -- TBD +- `Barry Warsaw `_ +- `Donghee Na `_ +- `Emily Morehouse `_ +- `Gregory P. Smith `_ +- `Mariatta `_ +- `Pablo Galindo Salgado `_ +- `Thomas Wouters `_ +- `Victor Stinner `_ Withdrawn nominations: @@ -157,7 +164,93 @@ Active Python core developers .. code-block:: text - TBD + Adam Turner + Alex Gaynor + Alex Waygood + Alexander Belopolsky + Alyssa Coghlan + Ammar Askar + Andrew Svetlov + Antoine Pitrou + Barney Gale + Barry Warsaw + Batuhan Taskaya + Benjamin Peterson + Berker Peksağ + Brandt Bucher + Brett Cannon + Brian Curtin + C.A.M. Gerlach + Carl Friedrich Bolz-Tereick + Carl Meyer + Carol Willing + Cheryl Sabella + Chris Withers + Christian Heimes + Dennis Sweeney + Dino Viehland + Donghee Na + Emily Morehouse + Éric Araujo + Eric Snow + Eric V. Smith + Erlend Egeberg Aasland + Ethan Furman + Ezio Melotti + Facundo Batista + Filipe Laíns + Georg Brandl + Giampaolo Rodolà + Gregory P. Smith + Guido van Rossum + Hugo van Kemenade + Hynek Schlawack + Inada Naoki + Irit Katriel + Ivan Levkivskyi + Jason R. Coombs + Jelle Zijlstra + Jeremy Kloth + Joannah Nanjekye + Julien Palard + Karthikeyan Singaravelan + Ken Jin + Kumar Aditya + Kurt B. Kaiser + Kushal Das + Larry Hastings + Łukasz Langa + Lysandros Nikolaou + Marc-André Lemburg + Mariatta + Mark Dickinson + Mark Shannon + Nathaniel J. Smith + Ned Deily + Neil Schemenauer + Pablo Galindo + Paul Ganssle + Paul Moore + Petr Viktorin + Pradyun Gedam + R. David Murray + Raymond Hettinger + Ronald Oussoren + Senthil Kumaran + Serhiy Storchaka + Shantanu Jain + Stefan Behnel + Stéphane Wirtel + Steve Dower + Tal Einat + Terry Jan Reedy + Thomas Wouters + Tim Golden + Tim Peters + Victor Stinner + Vinay Sajip + Yury Selivanov + Zachary Ware .. [#note-voters] This repository is private and accessible only to Python Core Developers, administrators, and Python Software Foundation Staff as it From 3db0f12cb164f4a10760e30920a0509f393f019e Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Fri, 24 Nov 2023 15:06:11 -0800 Subject: [PATCH 11/16] PEP 729: Reword confusing "end users" wording (#3539) --- peps/pep-0729.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/peps/pep-0729.rst b/peps/pep-0729.rst index 344d965bb57..6d7ed19b50e 100644 --- a/peps/pep-0729.rst +++ b/peps/pep-0729.rst @@ -321,8 +321,9 @@ The specification has a few audiences: * For people who want to propose changes to the type system, it provides a foundation for any new proposals. -Notably, the specification is not aimed at end users of typing, who typically do not -need to worry about compatibility across type checkers. Such users +Notably, the specification is not aimed at application developers who use +typing. Such users typically do not +need to worry about compatibility across type checkers. They are better served by a more informal user-facing reference, which is discussed in the next section. From 78fdac86891a717eb16eb76fdd1175cb71d89534 Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Fri, 10 Nov 2023 20:19:04 +0100 Subject: [PATCH 12/16] Minor copy-edits and improvements to Examples The added Pillow example and changes to Cryptography and Scipy code from testing builds in https://github.com/rgommers/external-deps-build. --- peps/pep-0725.rst | 43 ++++++++++++++++++++++++++++--------------- 1 file changed, 28 insertions(+), 15 deletions(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 172afbd86bc..4ed05eb410b 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -36,13 +36,13 @@ this PEP are to: information. Packaging ecosystems like Linux distros, Conda, Homebrew, Spack, and Nix need -full sets of dependencies for Python packages, and have tools like pyp2rpm_ +full sets of dependencies for Python packages, and have tools like pyp2spec_ (Fedora), Grayskull_ (Conda), and dh_python_ (Debian) which attempt to -automatically generate dependency metadata from the metadata in +automatically generate dependency metadata for their own package managers from the metadata in upstream Python packages. External dependencies are currently handled manually, because there is no metadata for this in ``pyproject.toml`` or any other standard location. Enabling automating this conversion is a key benefit of -this PEP, making packaging Python easier and more reliable. In addition, the +this PEP, making packaging Python packages for distros easier and more reliable. In addition, the authors envision other types of tools making use of this information, e.g., dependency analysis tools like Repology_, Dependabot_ and libraries.io_. Software bill of materials (SBOM) generation tools may also be able to use this @@ -100,7 +100,7 @@ Cross compilation Cross compilation is not yet (as of August 2023) well-supported by stdlib modules and ``pyproject.toml`` metadata. It is however important when translating external dependencies to those of other packaging systems (with -tools like ``pyp2rpm``). Introducing support for cross compilation immediately +tools like ``pyp2spec``). Introducing support for cross compilation immediately in this PEP is much easier than extending ``[external]`` in the future, hence the authors choose to include this now. @@ -204,9 +204,9 @@ Virtual package specification There is no ready-made support for virtual packages in PURL or another standard. There are a relatively limited number of such dependencies though, -and adoption a scheme similar to PURL but with the ``virtual:`` rather than +and adopting a scheme similar to PURL but with the ``virtual:`` rather than ``pkg:`` scheme seems like it will be understandable and map well to Linux -distros with virtual packages and the likes of Conda and Spack. +distros with virtual packages and to the likes of Conda and Spack. The two known virtual package types are ``compiler`` and ``interface``. @@ -347,10 +347,13 @@ cryptography 39.0: [external] build-requires = [ + "virtual:compiler/c", "virtual:compiler/rust", + "pkg:generic/pkg-config", ] host-requires = [ "pkg:generic/openssl", + "pkg:generic/libffi", ] SciPy 1.10: @@ -363,19 +366,14 @@ SciPy 1.10: "virtual:compiler/cpp", "virtual:compiler/fortran", "pkg:generic/ninja", + "pkg:generic/pkg-config", ] host-requires = [ "virtual:interface/blas", "virtual:interface/lapack", # >=3.7.1 (can't express version ranges with PURL yet) ] - [external.optional-host-requires] - dependency_detection = [ - "pkg:generic/pkg-config", - "pkg:generic/cmake", - ] - -pygraphviz 1.10: +Pillow 10.1.0: .. code:: toml @@ -384,9 +382,24 @@ pygraphviz 1.10: "virtual:compiler/c", ] host-requires = [ - "pkg:generic/graphviz", + "pkg:generic/libjpeg", + "pkg:generic/zlib", ] + [external.optional-host-requires] + extra = [ + "pkg:generic/lcms2", + "pkg:generic/freetype", + "pkg:generic/libimagequant", + "pkg:generic/libraqm", + "pkg:generic/libtiff", + "pkg:generic/libxcb", + "pkg:generic/libwebp", + "pkg:generic/openjpeg", # add >=2.0 once we have version specifiers + "pkg:generic/tk", + ] + + NAVis 1.4.0: .. code:: toml @@ -574,7 +587,7 @@ CC0-1.0-Universal license, whichever is more permissive. .. _PURL: https://github.com/package-url/purl-spec/ .. _vers: https://github.com/package-url/purl-spec/blob/version-range-spec/VERSION-RANGE-SPEC.rst .. _vers implementation for PURL: https://github.com/package-url/purl-spec/pull/139 -.. _pyp2rpm: https://github.com/fedora-python/pyp2rpm +.. _pyp2spec: https://github.com/befeleme/pyp2spec .. _Grayskull: https://github.com/conda/grayskull .. _dh_python: https://www.debian.org/doc/packaging-manuals/python-policy/index.html#dh-python .. _Repology: https://repology.org/ From 0c132e38edcb3775e2173fcdcb9832320e187857 Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Fri, 10 Nov 2023 21:58:04 +0100 Subject: [PATCH 13/16] Improve abstract to state briefly what the PEP adds to pyproject.toml --- peps/pep-0725.rst | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 4ed05eb410b..58d4871429f 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -18,6 +18,18 @@ This PEP specifies how to write a project's external, or non-PyPI, build and runtime dependencies in a ``pyproject.toml`` file for packaging-related tools to consume. +This PEP proposes to add an ``[external]`` table to ``pyproject.toml`` with +three keys: "build-requires", "host-requires" and "dependencies". These +are for specifying three types of dependencies: + +1. ``build-requires``, build tools to run on the build machine +2. ``host-requires``, build dependencies needed for host machine but also needed at build time. +3. ``dependencies``, needed at runtime on the host machine but not needed at build time. + +Cross compilation is taken into account by distinguishing build and host dependencies. +Optional build-time and runtime dependencies are supported too, in a manner analogies +to how that is supported in the ``[project]`` table. + Motivation ========== @@ -324,7 +336,7 @@ strings of the arrays MUST be valid PURL_ strings. with values of arrays of PURL_ strings (``optional-dependencies``) - `Core metadata`_: ``Requires-External``, N/A -The (optional) dependencies of the project. +The (optional) runtime dependencies of the project. For ``dependencies``, it is a key whose value is an array of strings. Each string represents a dependency of the project and MUST be formatted as either a From 2776369b196fff67c0aa08fb030c2d046d606402 Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Fri, 10 Nov 2023 22:45:04 +0100 Subject: [PATCH 14/16] Add section on split `pkg / `pkg-dev` packages Closes https://github.com/rgommers/peps/issues/10 --- peps/pep-0725.rst | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 58d4871429f..5af02acbc38 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -274,6 +274,39 @@ allow a version of a dependency for a wheel that isn't allowed for an sdist, nor contain new dependencies that are not listed in the sdist's metadata at all. +Canonical names of dependencies and ``-dev(el)`` split packages +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +It is fairly common for distros to split a package into two or more packages. +In particular, runtime components are often separately installable from +development components (headers, pkg-config and CMake files, etc.). The latter +then typically has a name with ``-dev`` or ``-devel`` appended to the +project/library name. This split is the responsibility of each distro to +maintain, and should not be reflected in the ``[external]`` table. It is not +possible to specify this in a reasonable way that works across distros, hence +only the canonical name should be used in ``[external]``. + +The intended meaning of using a PURL or virtual dependency is "the full package +with the name specified". It will depend on the context in which the metadata +is used whether the split is relevant. For example, if ``libffi`` is a host +dependency and a tool wants to prepare an environment for building a wheel, +then if a distro has split off the headers for ``libffi`` into a +``libffi-devel`` package then the tool has to install both ``libffi`` and +``libffi-devel``. + +Python development headers +'''''''''''''''''''''''''' + +Python headers and other build support files may also be split. This is the +same situation as in the section above (because Python is simply a regular +package in distros). *However*, a ``python-dev|devel`` dependency is special because +in ``pyproject.toml`` Python itself is an implicit rather than an explicit +dependency. Hence a choice needs to be made here - add ``python-dev`` implicitly, +or make each package author add it explicitly under ``[external]``. For +consistency between Python dependencies and external dependencies, we choose to +add it implicitly. Python development headers must be assumed to be necessary +when an ``[external]`` table contains one or more compiler packages. + Specification ============= From 46fbdc879d3e78f40bcd313a622b92cee0765086 Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Fri, 10 Nov 2023 23:11:03 +0100 Subject: [PATCH 15/16] Improve the "reference implementation" section Closes https://github.com/rgommers/peps/issues/11 --- peps/pep-0725.rst | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 5af02acbc38..80c99254b07 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -538,7 +538,22 @@ information about that in its documentation, as will tools like ``auditwheel``. Reference Implementation ======================== -There is no reference implementation at this time. +This PEP contains a metadata specification, rather that a code feature - hence +there will not be code implementing the metadata spec as a whole. However, +there are parts that do have a reference implementation: + +1. The ``[external]`` table has to be valid TOML and therefore can be loaded + with ``tomllib``. +2. The PURL specification, as a key part of this spec, has a Python package + with a reference implementation for constructing and parsing PURLs: + `packageurl-python`_. + +There are multiple possible consumers and use cases of this metadata, once +that metadata gets added to Python packages. Tested metadata for all of the +top 150 most-downloaded packages from PyPI with published platform-specific +wheels can be found in `rgommers/external-deps-build`_. This metadata has +been validated by using it to build wheels from sdists patched with that +metadata in clean Docker containers. Rejected Ideas @@ -630,6 +645,7 @@ CC0-1.0-Universal license, whichever is more permissive. .. _setuptools metadata: https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata .. _SPDX: https://spdx.dev/ .. _PURL: https://github.com/package-url/purl-spec/ +.. _packageurl-python: https://pypi.org/project/packageurl-python/ .. _vers: https://github.com/package-url/purl-spec/blob/version-range-spec/VERSION-RANGE-SPEC.rst .. _vers implementation for PURL: https://github.com/package-url/purl-spec/pull/139 .. _pyp2spec: https://github.com/befeleme/pyp2spec @@ -643,3 +659,4 @@ CC0-1.0-Universal license, whichever is more permissive. .. _auditwheel: https://github.com/pypa/auditwheel .. _delocate: https://github.com/matthew-brett/delocate .. _delvewheel: https://github.com/adang1345/delvewheel +.. _rgommers/external-deps-build: https://github.com/rgommers/external-deps-build From 3d840a9f5dad472b8643c00ab1319b00d09d7f96 Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Fri, 17 Nov 2023 15:24:48 +0100 Subject: [PATCH 16/16] Versioning and canonical names of virtual dependencies --- peps/pep-0725.rst | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 80c99254b07..13fc866d6f7 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -589,6 +589,43 @@ Support in PURL for version expressions and ranges is still pending. The pull request at `vers implementation for PURL`_ seems close to being merged, at which point this PEP could adopt it. +Versioning of virtual dependencies +---------------------------------- + +Once PURL supports version expressions, virtual dependencies can be versioned +with the same syntax. It must be better specified however what the version +scheme is, because this is not as clear for virtual dependencies as it is for +PURLs (e.g., there can be multiple implementations, and abstract interfaces may +not be unambiguously versioned). E.g.: + +- OpenMP: has regular ``MAJOR.MINOR`` versions of its standard, so would look + like ``>=4.5``. +- BLAS/LAPACK: should use the versioning used by `Reference LAPACK`_, which + defines what the standard APIs are. Uses ``MAJOR.MINOR.MICRO``, so would look + like ``>=3.10.0``. +- Compilers: these implement language standards. For C, C++ and Fortran these + are versioned by year. In order for versions to sort correctly, we choose to + use the full year (four digits). So "at least C99" would be ``>=1999``, and + selecting C++14 or Fortran 77 would be ``==2014`` or ``==1977`` respectively. + Other languages may use different versioning schemes. These should be + described somewhere before they are used in ``pyproject.toml``. + +A logistical challenge is where to describe the versioning - given that this +will evolve over time, this PEP itself is not the right location for it. +Instead, this PEP should point at that (to be created) location. + +Who defines canonical names and canonical package structure? +------------------------------------------------------------ + +Similarly to the logistics around versioning is the question about what names +are allowed and where they are described. And then who is in control of that +description and responsible for maintaining it. Our tentative answer is: there +should be a central list for virtual dependencies and ``pkg:generic`` PURLs, +maintained as a PyPA project. See +https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml/31888/62. +TODO: once that list/project is prototyped, include it in the PEP and close +this open issue. + Syntax for virtual dependencies ------------------------------- @@ -660,3 +697,4 @@ CC0-1.0-Universal license, whichever is more permissive. .. _delocate: https://github.com/matthew-brett/delocate .. _delvewheel: https://github.com/adang1345/delvewheel .. _rgommers/external-deps-build: https://github.com/rgommers/external-deps-build +.. _Reference LAPACK: https://github.com/Reference-LAPACK/lapack