bpo-43112: detect musl as a separate SOABI #24502

ncopa · 2021-02-10T17:52:29Z

musl libc and gnu libc are not ABI compatible so we need set different
SOABI for musl and not simply assume that all linux is linux-gnu.

Replace linux-gnu with the detected os for the build from config.guess
for linux-musl*.

https://bugs.python.org/issue43112

Automerge-Triggered-By: GH:brettcannon

musl libc and gnu libc are not ABI compatible so we need set different SOABI for musl and not simply assume that all linux is linux-gnu. Replace linux-gnu with the detected os for the build from config.guess for linux-musl*.

tiran

You also have to re-generate configure script with autoreconf and commit the new configure file.

bedevere-bot · 2021-02-10T18:44:32Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

ncopa · 2021-02-10T19:05:31Z

I have made the requested changes; please review again

bedevere-bot · 2021-02-10T19:05:34Z

Thanks for making the requested changes!

@tiran: please review the changes made to this pull request.

tiran · 2021-02-11T09:57:07Z

Looks ok to me.

Let's wait for input on https://discuss.python.org/t/pre-pep-platform-tag-for-linux-distributions-using-musl/7165

uranusjr · 2021-02-11T17:47:42Z

Unfortunately I don’t have the required autoconf knowledge to understand what’s going on here :(

stratakis · 2021-02-11T22:05:11Z

I'd remove the runstatedir changes, as those are generated due to either a later version of autoconf, or from one where that change was backported from upstream. However when someone makes another change to the configure script it can be removed again depending on the autoconf capabilities of the developer's machine.

See also: https://bugs.python.org/issue40216

tiran · 2021-02-12T11:16:12Z

Unfortunately I don’t have the required autoconf knowledge to understand what’s going on here :(

The magic happens here:

  case "$build_os" in
  linux-musl*)
    PLATFORM_TRIPLET=`echo "$PLATFORM_TRIPLET" | sed 's/linux-gnu/linux-musl/'`
    ;;
  esac

The shell code roughly translates to

if build_os.startswith("linux-musl"):
    PLATFORM_TRIPLET = PLATFORM_TRIPLET.replace("linux-gnu", "linux-musl")

This changes the SOABI from cpython-39-x86_64-linux-gnu to cpython-39-x86_64-linux-musl on all platforms that identify as Linux with musl libc.

uranusjr · 2021-02-12T17:27:46Z

Thanks for the explaination. Where does $build_os come from?

Am I correct to assume this needs to be done in the post-processing instead of conftest.c because musl provides no compile time detection on its usage (instead of glibc)?

tiran · 2021-02-12T17:40:50Z

build_os is provided by autoconf, https://www.gnu.org/software/autoconf/manual/autoconf-2.68/autoconf.html#index-build_005fos-1848

We could modify conftest.c, but it would take much more effort. We would have add a dozen ifdef checks for musl. GCC uses a similar approach like this patch in https://github.com/gcc-mirror/gcc/blob/master/gcc/config/t-musl . $(subst ...) is a Makefile command to substitute text.

uranusjr · 2021-02-12T19:20:16Z

Thanks. I still can’t say I really know how the code works, but kind of understand now what’s going on, and the approach makes sense to me.

ncopa · 2021-03-02T10:31:09Z

I think we may also need something like this:

diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py
index e279957e26..83ce74aa6f 100644
--- a/Lib/test/test_sysconfig.py
+++ b/Lib/test/test_sysconfig.py
@@ -387,10 +387,13 @@ def test_triplet_in_ext_suffix(self):
         if re.match('(i[3-6]86|x86_64)$', machine):
             if ctypes.sizeof(ctypes.c_char_p()) == 4:
                 self.assertTrue(suffix.endswith('i386-linux-gnu.so') or
-                                suffix.endswith('x86_64-linux-gnux32.so'),
+                                suffix.endswith('x86_64-linux-gnux32.so') or
+                                suffix.endswith('i386-linux-musl.so'),
                                 suffix)
             else: # 8 byte pointer size
-                self.assertTrue(suffix.endswith('x86_64-linux-gnu.so'), suffix)
+                self.assertTrue(suffix.endswith('x86_64-linux-gnu.so') or
+                                suffix.endswith('x86_64-linux-musl.so'),
+                                suffix)
 
     @unittest.skipUnless(sys.platform == 'darwin', 'OS X-specific test')
     def test_osx_ext_suffix(self):

github-actions · 2021-04-02T00:16:58Z

This PR is stale because it has been open for 30 days with no activity.

alex · 2021-11-01T21:08:48Z

@ncopa I believe you are correct that those tests need to be updated

bedevere-bot · 2022-01-29T00:23:46Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE RHEL8 LTO + PGO 3.x has failed when building commit 1f036ed.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/458/builds/1303) and take a look at the build logs.
Check if the failure is related to this commit (1f036ed) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/458/builds/1303

Failed tests:

test_sysconfig

Failed subtests:

test_triplet_in_ext_suffix - test.test_sysconfig.TestSysConfig

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

418 tests OK.

10 slowest tests:

test_concurrent_futures: 3 min 3 sec
test_multiprocessing_spawn: 1 min 47 sec
test_multiprocessing_forkserver: 1 min 14 sec
test_tokenize: 1 min 13 sec
test_multiprocessing_fork: 1 min 8 sec
test_asyncio: 57.7 sec
test_unparse: 52.9 sec
test_capi: 49.0 sec
test_signal: 47.9 sec
test_io: 40.7 sec

1 test failed:
test_sysconfig

13 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
test_winreg test_winsound test_zipfile64

1 re-run test:
test_sysconfig

Total duration: 3 min 58 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_baa5c5d3'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_987e13b0'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/test/test_sysconfig.py", line 439, in test_triplet_in_ext_suffix
    self.assertTrue(suffix.endswith(expected_suffixes),
                                    ^^^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'expected_suffixes' where it is not associated with a value


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto-pgo/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_c047c888'

follow-up to GH-24502

merwok · 2022-01-29T00:29:43Z

Thanks tianon, created #31001

bedevere-bot · 2022-01-29T00:38:42Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE RHEL8 LTO 3.x has failed when building commit 1f036ed.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/361/builds/1306) and take a look at the build logs.
Check if the failure is related to this commit (1f036ed) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/361/builds/1306

Failed tests:

test_sysconfig

Failed subtests:

test_triplet_in_ext_suffix - test.test_sysconfig.TestSysConfig

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

419 tests OK.

10 slowest tests:

test_peg_generator: 7 min 33 sec
test_gdb: 4 min 22 sec
test_concurrent_futures: 3 min 12 sec
test_multiprocessing_spawn: 1 min 58 sec
test_multiprocessing_forkserver: 1 min 18 sec
test_tokenize: 1 min 15 sec
test_multiprocessing_fork: 1 min 8 sec
test_asyncio: 1 min 1 sec
test_unparse: 58.7 sec
test_io: 57.5 sec

1 test failed:
test_sysconfig

12 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_startfile
test_tix test_tk test_ttk_guionly test_winconsoleio test_winreg
test_winsound test_zipfile64

1 re-run test:
test_sysconfig

Total duration: 7 min 56 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/test/test_sysconfig.py", line 439, in test_triplet_in_ext_suffix
    self.assertTrue(suffix.endswith(expected_suffixes),
                                    ^^^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'expected_suffixes' where it is not associated with a value


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_bce9806a'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_9eb00698'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le.lto/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_104a5c42'

Accidentally introduced by GH-24502

autobakterie · 2022-08-04T17:40:25Z

configure

@@ -5351,6 +5351,11 @@ EOF

 if $CPP $CPPFLAGS conftest.c >conftest.out 2>/dev/null; then
  PLATFORM_TRIPLET=`grep -v '^#' conftest.out | grep -v '^ *$' | tr -d ' 	'`
+  case "$build_os" in


If I'm not mistaken, PLATFORM_TRIPLET should refer to the target platform on which cpython will run. If that is the case, the musl libc vs. glibc decision should be based on $host_os rather than $build_os as the former is based on autoconf's AC_CANONICAL_HOST macro¹ which refers to the target platform that might differ from build platform in case of cross-compilation.

For example, the decision based on $build_os leads to
cpython 3.9.13 build failing² when cross-compiling on AMD64 Linux with glibc for mpc8548 Linux (OpenWrt) with musl. (build runs correctly if $host_os is used)

Footnotes

https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Canonicalizing.html ↩

"internal configure error for the platform triplet, please file a bug report" ↩

Yes, the code is wrong. In fact the entire block should be revisited. We really should check for __GLIBC__ and use a heuristic to detect musl. Could you please open a new issue for the problem?

No problem, but there's #87278 already. Is yet another issue needed?

Created #95855

Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz>

Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz> (cherry picked from commit 992fcd1)

Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz>

Patch 030: Backported from Python main branch[^1] for Python to distinguish between glibc and musl libc SOABI. Patch 131: Changes PLATFORM_TRIPLET -gnu/-musl suffix detection (performed by the backported patch) to be based on the target OS instead of the building OS. See included patches for more detailed descriptions. Specifically this fixes cross-compilation for mpc8548 CPUs with SPE instructions[^2] enabled. [^1]: merged to python:main as python/cpython#24502 'bpo-43112: detect musl as a separate SOABI' [^2]: https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf Co-authored-by: Pali Rohár <pali@kernel.org> Signed-off-by: Šimon Bořek <simon.borek@nic.cz> (cherry picked from commit 992fcd1)

bpo-43112: detect musl as a separate SOABI

82bf05d

musl libc and gnu libc are not ABI compatible so we need set different SOABI for musl and not simply assume that all linux is linux-gnu. Replace linux-gnu with the detected os for the build from config.guess for linux-musl*.

the-knights-who-say-ni added the CLA signed label Feb 10, 2021

bedevere-bot added the awaiting review label Feb 10, 2021

📜🤖 Added by blurb_it.

a2f26c7

tiran previously requested changes Feb 10, 2021

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Feb 10, 2021

regenerate configure

e45d9f5

bedevere-bot added awaiting change review and removed awaiting changes labels Feb 10, 2021

bedevere-bot requested a review from tiran February 10, 2021 19:05

Revert the runstate changes in the generated configure

85e0602

github-actions bot added the stale Stale PR or inactive for long period of time. label Apr 2, 2021

tiran removed their request for review April 17, 2021 21:16

lithammer mentioned this pull request Nov 18, 2021

Build musllinux wheels psycopg/psycopg2#1392

Merged

repsejnworb mentioned this pull request Nov 18, 2021

Add image for musllinux_1_2 policy pypa/manylinux#1225

Closed

dvarrazzo mentioned this pull request Nov 23, 2021

Some musllinux binary package not correctly working psycopg/psycopg#161

Closed

merwok added a commit that referenced this pull request Jan 29, 2022

bpo-43112: fix tests

a331ff9

follow-up to GH-24502

merwok mentioned this pull request Jan 29, 2022

bpo-43112: fix new sysconfig test #31001

Closed

pablogsal pushed a commit that referenced this pull request Jan 29, 2022

Fix a UnboundLocalError bug in test_sysconfig (GH-31000)

8b1b27f

Accidentally introduced by GH-24502

ncopa deleted the bpo-43112 branch March 16, 2022 10:59

autobakterie reviewed Aug 4, 2022

View reviewed changes

This was referenced Aug 4, 2022

python3: backport and fix target musl libc detection openwrt/packages#19110

Merged

Target libc based suffix (-musl/-gnu) in PLATFORM_TRIPLET is chosen based on build machine configuration instead of configuration of the target #95855

Closed

tlaurion mentioned this pull request Sep 26, 2022

CPython integration help needed linuxboot/heads#689

Open

ncopa mannequin mentioned this pull request Jan 19, 2023

SOABI on Linux does not distinguish between GNU libc and musl libc #87278

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-43112: detect musl as a separate SOABI #24502

bpo-43112: detect musl as a separate SOABI #24502

ncopa commented Feb 10, 2021 •

edited by miss-islington

Loading

tiran left a comment

bedevere-bot commented Feb 10, 2021

ncopa commented Feb 10, 2021

bedevere-bot commented Feb 10, 2021

tiran commented Feb 11, 2021

uranusjr commented Feb 11, 2021

stratakis commented Feb 11, 2021

tiran commented Feb 12, 2021

uranusjr commented Feb 12, 2021 •

edited

Loading

tiran commented Feb 12, 2021

uranusjr commented Feb 12, 2021

ncopa commented Mar 2, 2021

github-actions bot commented Apr 2, 2021

alex commented Nov 1, 2021

bedevere-bot commented Jan 29, 2022

merwok commented Jan 29, 2022

bedevere-bot commented Jan 29, 2022

autobakterie Aug 4, 2022

tiran Aug 4, 2022

autobakterie Aug 5, 2022

autobakterie Aug 10, 2022

bpo-43112: detect musl as a separate SOABI #24502

bpo-43112: detect musl as a separate SOABI #24502

Conversation

ncopa commented Feb 10, 2021 • edited by miss-islington Loading

tiran left a comment

Choose a reason for hiding this comment

bedevere-bot commented Feb 10, 2021

ncopa commented Feb 10, 2021

bedevere-bot commented Feb 10, 2021

tiran commented Feb 11, 2021

uranusjr commented Feb 11, 2021

stratakis commented Feb 11, 2021

tiran commented Feb 12, 2021

uranusjr commented Feb 12, 2021 • edited Loading

tiran commented Feb 12, 2021

uranusjr commented Feb 12, 2021

ncopa commented Mar 2, 2021

github-actions bot commented Apr 2, 2021

alex commented Nov 1, 2021

bedevere-bot commented Jan 29, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

merwok commented Jan 29, 2022

bedevere-bot commented Jan 29, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

autobakterie Aug 4, 2022

Choose a reason for hiding this comment

Footnotes

tiran Aug 4, 2022

Choose a reason for hiding this comment

autobakterie Aug 5, 2022

Choose a reason for hiding this comment

autobakterie Aug 10, 2022

Choose a reason for hiding this comment

ncopa commented Feb 10, 2021 •

edited by miss-islington

Loading

uranusjr commented Feb 12, 2021 •

edited

Loading