Unvendor libmpdec sources

[zware]

To facilitate cleaner updates of the externally-maintained libmpdec required by the _decimal module, we should migrate away from the bundled copy in Modules/_decimal/libmpdec and towards an "external" in cpython-source-deps for Windows and --with-system-libmpdec by default elsewhere.

My tentative plan is as follows:

• In 3.13:
  • Add mpdecimal-2.5.1 and mpdecimal-4.0.0 to cpython-source-deps:
    • Sort list of downloads, and add link to mpdecimal sources cpython-source-deps#32
    • https://github.com/python/cpython-source-deps/tree/mpdecimal
  • Switch Windows build to mpdecimal-2.5.1 external (gh-115119: Switch Windows build to mpdecimal external #115182)
  • gh-115119: Detect _decimal dependencies using pkg-config #115406
  • Update deps and build instructions wrt. libmpdecimal devguide#1313
  • gh-115119: Update macOS installer to build with libmpdec 4.0.0 #118382
  • gh-115119: Update Windows installer to mpdecimal 4.0.0 #118506
  • Update macOS build to mpdecimal-4.0.0
  • gh-115119: Default to --with-system-libmpdec=yes #118539
  • gh-115119: Bump CI to use Ubuntu 22.04 #118631
  • Post #118539: amend Ubuntu CI to use system installed mpdecimal
  • Update the docs; see Unvendor libmpdec sources #115119 (comment)
  • gh-115119: Fall back to bundled libmpdec if system libmpdec is not found #119196
• In 3.14:
  • Remove the implicit fallback to the bundled copy
  • Keep support for explicit --without-system-libmpdec (with warning)
• In 3.15:
  • Remove --with[out]-system-libmpdec configure options and Modules/_decimal/libmpdec/

Linked PRs

• gh-115119: Switch Windows build to mpdecimal external #115182
• gh-115119: Detect _decimal dependencies using pkg-config #115406
• gh-115119: Update macOS installer to build with libmpdec 4.0.0 #118382
• gh-115119: Update Windows installer to mpdecimal 4.0.0 #118506
• gh-115119: Default to --with-system-libmpdec=yes #118539
• gh-115119: Bump CI to use Ubuntu 22.04 #118631
• [3.12] gh-115119: Bump CI to use Ubuntu 22.04 (#118631) #118634
• gh-115119: Fall back to bundled libmpdec if system libmpdec is not found #119196
• [3.13] gh-115119: Fall back to bundled libmpdec if system libmpdec is not found (GH-119196) #119217

[erlend-aasland]

[...] with a fallback to the bundled copy (maybe with a warning?)

+1 for an autoconf warning.

[ned-deily]

Adding an external libmpdec to the macOS installer build shouldn't be a big issue. Assign it to me if you proceed with this.

[erlend-aasland]

FTR: the source deps repo is now updated.

[erlend-aasland]

@ned-deily: for build-installer.py, is there more to do than adding a configuration for mpdecimal? I've used the following configure options: ['--disable-cxx', 'MACHINE=universal']

[ned-deily]

Likely yes, I'll take care of it. Thanks.

[ned-deily]

I took a quick look at this and be aware that this change proposal will likely impact everyone building Python 3.13 on macOS. macOS does not ship a copy of libmpdec so, if the bundled version is removed, anyone building Python on macOS will now need to provide a local copy, either building from source or using a version from one of the widely-used third-party package managers, like Homebrew or MacPorts. Both currently have mpdecimal packages at the 4.0.0 level. (Recall that, unlike Windows builds, we do not currently provide non-vendored third-party library binaries for macOS users, other than those built for and linked into the Pythons provided by python.org installers for macOS.) In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected. And, in any case, the devguide instructions for installing dependencies will need to be updated to cover libmpdec. We should add all this and get this working before moving on to changing the installer.

[erlend-aasland]

In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected.

Last time I checked, libmpdec did not provide pkgconfig files:

$ ls $(brew --prefix libmpdec)/lib/pkgconfig
ls: /opt/homebrew/opt/mpdecimal/lib/pkgconfig: No such file or directory
$ cd mpdecimal-2.5.1; find . | grep pkg
$

[ned-deily]

With the current 4.0.0 version, it appears both Homebrew and MacPorts ship them:

$ pkg-config libmpdec --cflags
-I/opt/homebrew/Cellar/mpdecimal/4.0.0/include
$ pkg-config libmpdec --libs
-L/opt/homebrew/Cellar/mpdecimal/4.0.0/lib -lmpdec -lm

...

$ pkg-config libmpdec --libs
-L/opt/macports/lib -lmpdec -lm

[erlend-aasland]

Yes, the 4.0.0 version does indeed provide pkg-config configuration files. I'll work on an Autoconf patch.

[erlend-aasland]

@zware are we still aiming for 3.13 for the first batch of items?

[erlend-aasland]

In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected.

Resolved by:

• gh-115119: Detect _decimal dependencies using pkg-config #115406

[skirpichev]

It seems, that due to https://bugs.debian.org/1056785 - more projects now use _pydecimal implementation, that gets battle-tested. Here is an example of a failure: https://github.com/aleaxit/gmpy/actions/runs/9153743194/job/25163188162 due to difference in the tp_name: "decimal.Decimal" vs "Decimal".

[we-are-all-individuals]

Reaching out to the Debian CPython maintainer @doko42. It would be very nice to make mpdecimal available again in Debian. According to https://www.bytereef.org/mpdecimal/changelog.html the Sphinx issues no longer exist in version 4.0.0.

[vfazio]

Is the plan to still unvendor this if Debian doesn't reinstate the package?

[skirpichev]

Why Debian should be a special snowflake for CPython? Referenced bug thread ignores any feedback, including comments from CPython core dev. I think it's a Debian problem, not ours.

[vfazio]

Well, Debian is a major distro and the root of other distros like Ubuntu. But i guess they'll figure it out once Python fails to build

[skirpichev]

Well, Debian is a major distro and the root of other distros like Ubuntu.

Say also thanks to people, who aren't part of the Debian maintainers now.

But i guess they'll figure it out once Python fails to build

It will not fail. IIUC, with --with-system-libmpdec=no & without bundled sources it will use the pure-Python decimal module, see above: #115119 (comment).

Apparently they don't notice this.

[vfazio]

I missed the decimal -> _decimal or _pydecimal so thanks for pointing that out. I'll have to update my build scripts so i don't suffer the performance penalty

[skirpichev]

I don't think you can do here something. Once the CPython will not bundle libmpdec - we will have no choice but fallback to the _pydecimal if --with-system-libmpdec=no.

Someone must do real job and package back the libmpdec for Debian.

[vfazio]

Hmm..

I'll have to look into it more. I'll was expecting to have to --with-system-libmpdec=yes and specify LIBMPDEC_CFLAGS and LIBMPDEC_LIBS to work around the calls to pkg-config. Worst case I compile and deploy the files to a custom directory and set PKG_CONFIG_LIBDIR and let the build system figure it out without the overrides.

Obviously this is not ideal, but until us Debian users get some traction on that issue, that's probably the best I can hope for.

When I get some spare time I'll test out a build with the vendored code nuked and see if the overrides work like I expect.

[skirpichev]

I'll was expecting to have to --with-system-libmpdec=yes and specify LIBMPDEC_CFLAGS and LIBMPDEC_LIBS to work around the calls to pkg-config.

Ah, yes - that will work.

[tanzim]

What would be the guidance for folks building embedded distributions this case? Use relevant CFLAGS to relatively link to a copy of libmpdec shipped with the embedded distribution?