bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 #23556

fxcoudert · 2020-11-29T15:28:29Z

This pull request provides a fix for https://bugs.python.org/issue42504

https://bugs.python.org/issue42504

the-knights-who-say-ni · 2020-11-29T15:28:32Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

CLA Missing

Our records indicate the following people have not signed the CLA:

@fxcoudert

For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

fxcoudert · 2020-11-29T15:31:35Z

I have now signed the CLA

lewis262626 · 2020-11-29T16:19:23Z

Looks good to me 👍

ned-deily

Thanks for the PR. There is more work to be done, though. With:

MACOSX_DEPLOYMENT_TARGET=11 ./configure

making this change in setup.py gets past the first error only to fail later in distutils/spawn.py:

  File "source/Lib/distutils/spawn.py", line 66, in spawn
    if _cfg_target_split > [int(x) for x in cur_target.split('.')]:
TypeError: '>' not supported between instances of 'NoneType' and 'list'

Are there other users of MACOSX_DEPLOYMENT_TARGET in the source that need to be changed as well?

bedevere-bot · 2020-11-30T04:08:25Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

ronaldoussoren · 2020-11-30T08:26:10Z

A nicer fix is to ensure that sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') evaluates to a string. I haven't checked how easy it would be to do this.

fxcoudert · 2020-11-30T08:46:31Z

@ronaldoussoren the code at https://github.com/python/cpython/blob/3.9/Lib/sysconfig.py#L461 shows it's very deliberate that int-type values are returned as int

fxcoudert · 2020-11-30T09:34:15Z

Two places I am not 100% sure what to do:

https://github.com/python/cpython/blob/master/configure.ac#L1970 this assumes that cur_target="${cur_target_major}.${cur_target_minor}", so it may set MACOSX_DEPLOYMENT_TARGET to 11.0. I think that's acceptable.
https://github.com/python/cpython/blob/master/Mac/BuildScript/build-installer.py has lots of use cases, but I think they're all string-based, and therefore fine

fxcoudert · 2020-11-30T09:47:06Z

I have made the requested changes; please review again.
I have confirmed that this makes the formula build on 11.0 when setting MACOSX_DEPLOYMENT_TARGET=11 (https://github.com/Homebrew/homebrew-core/pull/65933/checks?check_run_id=1473199510)

Travis is telling me:

Fixing Python file whitespace ... 1 file:
  Lib/test/test_posix.py

but I haven't changed whitespace in this file, and it is not telling me what the issue actually is.

bedevere-bot · 2020-11-30T09:47:10Z

Thanks for making the requested changes!

@ned-deily: please review the changes made to this pull request.

ned-deily

Thanks for the changes. Running test_distutils:
./python(.exe) -m test -w test_distutils

gives a number of failing test cases, basically due to one of two problems.

The missing str() at spawn.py:66
Failures in test_build_ext.py:501:

File "source/Lib/distutils/tests/test_build_ext.py", line 501, in _try_compile_deployment_target
    target = '%02d%02d00' % target
TypeError: not enough arguments for format string

I don't think we need to make any changes to configure.ac or build_installer.py.

ned-deily · 2020-12-01T02:11:42Z

Lib/distutils/spawn.py

@@ -57,7 +57,7 @@ def spawn(cmd, search_path=1, verbose=0, dry_run=0):
            _cfg_target = sysconfig.get_config_var(
                                  'MACOSX_DEPLOYMENT_TARGET') or ''
            if _cfg_target:
-                _cfg_target_split = [int(x) for x in _cfg_target.split('.')]
+                _cfg_target_split = [int(x) for x in str(_cfg_target).split('.')]


A similar change is needed below at line 66.

bedevere-bot · 2020-12-01T02:22:11Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

fxcoudert · 2020-12-02T08:31:08Z

@nilleb wait I see the problem you're referring to, we're passing that int as an environment variable. That's where it should be fixed, actually. If it's defined, _cfg_target should always be a string to begin with. Then things go smoothly.

fxcoudert · 2020-12-02T08:58:51Z

This was tested with a full build of Python on macOS Big Sur (ARM), as well as building several python-based packages. Without the latest version of the patch, some failed in subprocess.py. Now all pass.

nilleb · 2020-12-02T09:10:23Z

hey @fxcoudert the problem comes from line https://github.com/python/cpython/pull/23556/files#diff-a06590fdcc4068035ba08a86c9c816cb70a78d80cb9ff0babfbb8dae85dd41d0L75
where we pass env=env to the subprocess call (this has nothing to do with _cfg_target)
Since the variable MACOSX_DEPLOYMENT_TARGET has a 11 (int) value
3 stack levels beneath, it will raise an exception.

you can reproduce it by

git clone https://github.com/nilleb/magpie
cd magpie
bin/setup

edit: the interesting part of the stack

      File "/opt/homebrew/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/spawn.py", line 75, in spawn
        proc = subprocess.Popen(cmd, env=env)
      File "/opt/homebrew/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 947, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/opt/homebrew/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1740, in _execute_child
        val = os.fsencode(v)
      File "/opt/homebrew/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 810, in fsencode
        filename = fspath(filename)  # Does type-checking of `filename`.
    TypeError: expected str, bytes or os.PathLike object, not int

fxcoudert · 2020-12-02T09:38:05Z

@nilleb this is fixed in the latest version of this patch. The problem is we pass cur_target as MACOSX_DEPLOYMENT_TARGET, and cur_target defaults to _cfg_target in some cases, and _cfg_target could be an int. Now it's always a str, and there is no problem.

guillochon · 2020-12-02T15:41:38Z

@fxcoudert can you add the blurb using the blurb-it tool so that check will pass? https://blurb-it.herokuapp.com/

guillochon · 2020-12-02T18:37:39Z

@ned-deily mind taking a gander at this one again? It looks ready to go

ned-deily

LGTM now, thanks! No doubt we will find other implications of moving from 10.x to 1x release version but this should be a good start.

miss-islington · 2020-12-03T03:20:44Z

Thanks @fxcoudert for the PR, and @ned-deily for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

macOS releases numbering has changed as of macOS 11 Big Sur. Previously, major releases were of the form 10.x, 10.x+1, 10.x+2, etc; as of Big Sur, they are now x, x+1, etc, so, for example, 10.15, 10.15.1, ..., 10.15.7, 11, 11.0.1, 11.1, ..., 12, 12.1, etc. Allow Python to build with single-digit deployment target values. Patch provided by FX Coudert. (cherry picked from commit 5291639) Co-authored-by: FX Coudert <fxcoudert@gmail.com>

bedevere-bot · 2020-12-03T03:20:57Z

GH-23622 is a backport of this pull request to the 3.9 branch.

macOS releases numbering has changed as of macOS 11 Big Sur. Previously, major releases were of the form 10.x, 10.x+1, 10.x+2, etc; as of Big Sur, they are now x, x+1, etc, so, for example, 10.15, 10.15.1, ..., 10.15.7, 11, 11.0.1, 11.1, ..., 12, 12.1, etc. Allow Python to build with single-digit deployment target values. Patch provided by FX Coudert. (cherry picked from commit 5291639) Co-authored-by: FX Coudert <fxcoudert@gmail.com>

fxcoudert · 2020-12-03T12:59:14Z

Related PR, with wheel: pypa/wheel#385

macOS releases numbering has changed as of macOS 11 Big Sur. Previously, major releases were of the form 10.x, 10.x+1, 10.x+2, etc; as of Big Sur, they are now x, x+1, etc, so, for example, 10.15, 10.15.1, ..., 10.15.7, 11, 11.0.1, 11.1, ..., 12, 12.1, etc. Allow Python to build with single-digit deployment target values. Patch provided by FX Coudert.

macOS releases numbering has changed as of macOS 11 Big Sur. Previously, major releases were of the form 10.x, 10.x+1, 10.x+2, etc; as of Big Sur, they are now x, x+1, etc, so, for example, 10.15, 10.15.1, ..., 10.15.7, 11, 11.0.1, 11.1, ..., 12, 12.1, etc. Allow Python to build with single-digit deployment target values. Patch provided by FX Coudert. (cherry picked from commit 5291639) Co-authored-by: FX Coudert <fxcoudert@gmail.com>

) * bpo-41100: Support macOS 11 and Apple Silicon on Python 3.8 This is a partial backport of bpo-41100 changes `e8b1c038b14b5fc8120aab62c9bf5fb840274cb6` and `96d906b144e6e6aa96c5ffebecbcc5d38034bbda` for Python 3.8. We introduce the ability to build Python from source for `arm64` on macOS, but we do not make a promise of support. This allows us to omit support for Universal2 binaries as well as weak-linking of symbols from the macOS SDK based on the deployment target, which are larger changes much more difficult to merge. This also includes a backport of subsequent bpo-42688 change `7e729978fa08a360cbf936dc215ba7dd25a06a08` to fix build errors with external `libffi`. * bpo-41116: Ensure system supplied libraries are found on macOS 11 (GH-23301) (GH-23455) On macOS system provided libraries are in a shared library cache and not at their usual location. This PR teaches distutils to search in the SDK, even if there was no "-sysroot" argument in the compiler flags. (cherry picked from commit 404a719) * bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 (GH-23556) macOS releases numbering has changed as of macOS 11 Big Sur. Previously, major releases were of the form 10.x, 10.x+1, 10.x+2, etc; as of Big Sur, they are now x, x+1, etc, so, for example, 10.15, 10.15.1, ..., 10.15.7, 11, 11.0.1, 11.1, ..., 12, 12.1, etc. Allow Python to build with single-digit deployment target values. Patch provided by FX Coudert. (cherry picked from commit 5291639) * bpo-42504: Ensure that get_config_var('MACOSX_DEPLOYMENT_TARGET') is a string (GH-24341) (GH-24410) * bpo-42504: Ensure that get_config_var('MACOSX_DEPLOYMENT_TARGET') is a string (cherry picked from commit 49926cf) Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com> Co-authored-by: FX Coudert <fxcoudert@gmail.com> Co-authored-by: Max Bélanger <aeromax@gmail.com>

the-knights-who-say-ni added the CLA not signed label Nov 29, 2020

bedevere-bot added the awaiting review label Nov 29, 2020

fxcoudert mentioned this pull request Nov 29, 2020

Python@3.9 fails to install on ARM Big Sur Homebrew/homebrew-core#65866

Closed

5 tasks

guillochon approved these changes Nov 29, 2020

View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels Nov 29, 2020

ned-deily requested changes Nov 30, 2020

View reviewed changes

bedevere-bot removed the awaiting core review label Nov 30, 2020

bedevere-bot added the awaiting changes label Nov 30, 2020

fxcoudert mentioned this pull request Nov 30, 2020

Python 3.9 Big Sur testing Homebrew/homebrew-core#65933

Closed

bedevere-bot added awaiting change review and removed awaiting changes labels Nov 30, 2020

bedevere-bot requested a review from ned-deily November 30, 2020 09:47

ned-deily requested changes Dec 1, 2020

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting change review labels Dec 1, 2020

fxcoudert mentioned this pull request Dec 1, 2020

Python 3.9.1 bigsur Homebrew/homebrew-core#65793

Closed

2 tasks

the-knights-who-say-ni added CLA signed and removed CLA not signed labels Dec 1, 2020

fxcoudert requested a review from ned-deily December 1, 2020 07:58

setup.py: fix for MACOSX_DEPLOYMENT_TARGET=11

6511bf5

📜🤖 Added by blurb_it.

90ce1b1

mitchhentges mentioned this pull request Dec 2, 2020

"'int' object has no attribute 'split'" when pip installing zstandard on Big Sur Homebrew/homebrew-core#66096

Closed

4 tasks

ned-deily approved these changes Dec 3, 2020

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels Dec 3, 2020

ned-deily merged commit 5291639 into python:master Dec 3, 2020

bedevere-bot removed the awaiting merge label Dec 3, 2020

ned-deily added the needs backport to 3.9 only security fixes label Dec 3, 2020

bedevere-bot removed the needs backport to 3.9 only security fixes label Dec 3, 2020

fxcoudert deleted the patch-1 branch December 3, 2020 07:27

This was referenced Dec 3, 2020

Update python@3.9.rb Homebrew/homebrew-core#66105

Closed

wheel 0.36.0 fails on macOS 11 (Big Sur) when MACOSX_DEPLOYMENT_TARGET=11 pypa/wheel#385

Closed

This was referenced Dec 3, 2020

BUG: Fix a MacOS build failure numpy/numpy#17906

Merged

python@3.9 3.9.1 Homebrew/homebrew-core#66489

Closed

loganjhennessy mentioned this pull request Jan 12, 2021

Unable To Install Tensorflow on macOS Big Sur python-poetry/poetry#3566

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 #23556

bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 #23556

fxcoudert commented Nov 29, 2020 •

edited by bedevere-bot

Loading

the-knights-who-say-ni commented Nov 29, 2020

fxcoudert commented Nov 29, 2020

lewis262626 commented Nov 29, 2020

ned-deily left a comment

bedevere-bot commented Nov 30, 2020

ronaldoussoren commented Nov 30, 2020

fxcoudert commented Nov 30, 2020

fxcoudert commented Nov 30, 2020

fxcoudert commented Nov 30, 2020 •

edited

Loading

bedevere-bot commented Nov 30, 2020

ned-deily left a comment

ned-deily Dec 1, 2020 •

edited

Loading

bedevere-bot commented Dec 1, 2020

fxcoudert commented Dec 2, 2020

fxcoudert commented Dec 2, 2020

nilleb commented Dec 2, 2020 •

edited

Loading

fxcoudert commented Dec 2, 2020

guillochon commented Dec 2, 2020

guillochon commented Dec 2, 2020

ned-deily left a comment

miss-islington commented Dec 3, 2020

bedevere-bot commented Dec 3, 2020

fxcoudert commented Dec 3, 2020

bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 #23556

bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 #23556

Conversation

fxcoudert commented Nov 29, 2020 • edited by bedevere-bot Loading

the-knights-who-say-ni commented Nov 29, 2020

CLA Missing

fxcoudert commented Nov 29, 2020

lewis262626 commented Nov 29, 2020

ned-deily left a comment

Choose a reason for hiding this comment

bedevere-bot commented Nov 30, 2020

ronaldoussoren commented Nov 30, 2020

fxcoudert commented Nov 30, 2020

fxcoudert commented Nov 30, 2020

fxcoudert commented Nov 30, 2020 • edited Loading

bedevere-bot commented Nov 30, 2020

ned-deily left a comment

Choose a reason for hiding this comment

ned-deily Dec 1, 2020 • edited Loading

Choose a reason for hiding this comment

bedevere-bot commented Dec 1, 2020

fxcoudert commented Dec 2, 2020

fxcoudert commented Dec 2, 2020

nilleb commented Dec 2, 2020 • edited Loading

fxcoudert commented Dec 2, 2020

guillochon commented Dec 2, 2020

guillochon commented Dec 2, 2020

ned-deily left a comment

Choose a reason for hiding this comment

miss-islington commented Dec 3, 2020

bedevere-bot commented Dec 3, 2020

fxcoudert commented Dec 3, 2020

fxcoudert commented Nov 29, 2020 •

edited by bedevere-bot

Loading

fxcoudert commented Nov 30, 2020 •

edited

Loading

ned-deily Dec 1, 2020 •

edited

Loading

nilleb commented Dec 2, 2020 •

edited

Loading