Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use explicit setup-python versions over implicit expose- (Cherry-pick of #21568, #21582) #21668

Merged
merged 5 commits into from
Nov 21, 2024

Conversation

huonw
Copy link
Contributor

@huonw huonw commented Nov 19, 2024

The project is currently using a mix of the GitHub supplied setup-python action and pantsbuild/actions/expose-pythons. On GitHub managed runners they behave similarly:

  • setup-python will install a version of Python if missing, or add it to the PATH if already present at the expected location (managed runners historically have multiple Pythons baked into the image)
  • expose-pythons will add all Python's at the expected GitHub location to the PATH. (Which ones is up to GitHub.)

So today the invocation of setup-python followed by expose-pythons is redundant.

Consolidating on setup-python let's us be explicit about expected versions (more like the ARM image) and reduces the number of custom actions the project manages while still making multiple Python versions available.

NOTE: The awkward double newlines in the final yaml are a pre-existing issue
https://stackoverflow.com/questions/45004464/yaml-dump-adding-unwanted-newlines-in-multiline-strings

See #21552 for some history regarding the setup- vs expose- actions.

cburroughs and others added 2 commits November 19, 2024 12:52
The project is currently using a mix of the GitHub supplied
`setup-python` action and `pantsbuild/actions/expose-pythons`. On GitHub
managed runners they behave similarly:
* `setup-python` will install a version of Python if missing, or add it
to the PATH if already present at the expected location (managed runners
historically have multiple Pythons baked into the image)
* `expose-pythons` will add all Python's at the expected GitHub location
to the PATH. (Which ones is up to GitHub.)

So today the invocation of `setup-python` followed by `expose-pythons`
is redundant.

Consolidating on `setup-python` let's us be explicit about expected
versions (more like the ARM image) and reduces the number of custom
actions the project manages while still making multiple Python versions
available.

NOTE: The awkward double newlines in the final yaml are a pre-existing
issue

https://stackoverflow.com/questions/45004464/yaml-dump-adding-unwanted-newlines-in-multiline-strings

See #21552 for some history regarding the `setup-` vs `expose-` actions.
This fixes the "Build wheels (macOS12-x86_64)" CI job, by installing
Python. It was previously failing with errors like:

```
> Run ./pants run src/python/pants_release/release.py -- build-wheels
pants: Failed to find a Python 3.9 interpreter
```

I think this was a small oversight in #21568 (in particular
https://github.com/pantsbuild/pants/pull/21568/files#diff-b980287839311482e79f8adac1fdd7768274cf1e677a1103d9444185e4aabbd1L890),
which wasn't caught by CI because the build-wheels jobs were skipped.
@huonw huonw added this to the 2.23.x milestone Nov 19, 2024
@huonw huonw added the category:internal CI, fixes for not-yet-released features, etc. label Nov 19, 2024
@huonw huonw requested a review from cburroughs November 19, 2024 01:55
@huonw huonw changed the title use explicit setup-python versions over implicit expose- (Cherry-pick of #21568) use explicit setup-python versions over implicit expose- (Cherry-pick of #21568, #21582) Nov 19, 2024
@huonw
Copy link
Contributor Author

huonw commented Nov 19, 2024

I haven't had time to debug the errors here. Recorded for posterity/convenience:

02:26:58.14 [ERROR] 1 Exception encountered:

Engine traceback:
  in root
    ..
  in pants.core.goals.lint.lint
    `lint` goal

Traceback (most recent call last):
  File "/home/runner/work/pants/pants/src/python/pants/core/goals/lint.py", line 469, in lint
    all_batch_results = await MultiGet(
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 376, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 174, in __await__
    result = yield self.gets
  File "/home/runner/work/pants/pants/src/python/pants/backend/build_files/fmt/black/register.py", line 21, in black_fmt
    return await _run_black(request, black, black_ics)
  File "/home/runner/work/pants/pants/src/python/pants/backend/python/lint/black/rules.py", line 41, in _run_black
    black_pex, config_files = await MultiGet(black_pex_get, config_files_get)
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 406, in MultiGet
    return await _MultiGet((__arg0, __arg1))
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 174, in __await__
    result = yield self.gets
  File "/home/runner/work/pants/pants/src/python/pants/backend/build_files/fmt/black/register.py", line 21, in black_fmt
    return await _run_black(request, black, black_ics)
  File "/home/runner/work/pants/pants/src/python/pants/backend/python/lint/black/rules.py", line 41, in _run_black
    black_pex, config_files = await MultiGet(black_pex_get, config_files_get)
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 406, in MultiGet
    return await _MultiGet((__arg0, __arg1))
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 174, in __await__
    result = yield self.gets
  File "/home/runner/work/pants/pants/src/python/pants/backend/build_files/fmt/black/register.py", line 21, in black_fmt
    return await _run_black(request, black, black_ics)
  File "/home/runner/work/pants/pants/src/python/pants/backend/python/lint/black/rules.py", line 41, in _run_black
    black_pex, config_files = await MultiGet(black_pex_get, config_files_get)
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 406, in MultiGet
    return await _MultiGet((__arg0, __arg1))
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 174, in __await__
    result = yield self.gets
  File "/home/runner/work/pants/pants/src/python/pants/backend/python/util_rules/pex.py", line 1085, in create_venv_pex
    venv_pex_result = await Get(BuildPexResult, PexRequest, seeded_venv_request)
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 124, in __await__
    result = yield self
  File "/home/runner/work/pants/pants/src/python/pants/backend/python/util_rules/pex.py", line 698, in build_pex
    pex_python_setup, requirements_setup, sources_digest_as_subdir = await concurrently(
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 421, in MultiGet
    return await _MultiGet((__arg0, __arg1, __arg2))
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 174, in __await__
    result = yield self.gets
  File "/home/runner/work/pants/pants/src/python/pants/engine/rules.py", line 74, in wrapper
    return await call
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 85, in __await__
    result = yield self
  File "/home/runner/work/pants/pants/src/python/pants/backend/python/util_rules/pex.py", line 451, in _determine_pex_python_and_platforms
    python = await Get(
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 124, in __await__
    result = yield self
  File "/home/runner/work/pants/pants/src/python/pants/backend/python/util_rules/pex.py", line 365, in find_interpreter
    result = await Get(
  File "/home/runner/work/pants/pants/src/python/pants/engine/internals/selectors.py", line 124, in __await__
    result = yield self
  File "/home/runner/work/pants/pants/src/python/pants/engine/process.py", line 325, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython<4,>=3.7' failed with exit code 1.
stdout:

stderr:
No supported version of Pip is compatible with the given targets:
cp310-cp310-manylinux_2_31_x86_64 interpreter at /opt/hostedtoolcache/Python/3.10.15/x64/bin/python3.10
cp311-cp311-manylinux_2_31_x86_64 interpreter at /opt/hostedtoolcache/Python/3.11.10/x64/bin/python3.11
cp312-cp312-manylinux_2_31_x86_64 interpreter at /opt/hostedtoolcache/Python/3.12.7/x64/bin/python3.12
cp313-cp313-manylinux_2_31_x86_64 interpreter at /opt/hostedtoolcache/Python/3.13.0/x64/bin/python3.13
cp37-cp37m-manylinux_2_31_x86_64 interpreter at /opt/hostedtoolcache/Python/3.7.17/x64/bin/python3.7
cp38-cp38-manylinux_2_31_x86_64 interpreter at /opt/hostedtoolcache/Python/3.8.18/x64/bin/python3.8
cp38-cp38-manylinux_2_31_x86_64 interpreter at /usr/bin/python3.8
cp39-cp39-manylinux_2_31_x86_64 interpreter at /home/runner/.cache/pants/pants_dev_deps/5c49e358f3370dc4ffcf9529336c30e328019799.venv/bin/python3.9
cp39-cp39-manylinux_2_31_x86_64 interpreter at /opt/hostedtoolcache/Python/3.9.20/x64/bin/python3.9

@cburroughs
Copy link
Contributor

I think the compatible with recent Python pip version is only in Pants 2.24, correct?

@huonw
Copy link
Contributor Author

huonw commented Nov 20, 2024

Ah yes, not installing Python 3.13 works better.

@huonw huonw marked this pull request as draft November 20, 2024 01:40
@huonw
Copy link
Contributor Author

huonw commented Nov 20, 2024

(Marking this as draft, it doesn't need to be merged before the 2.23.0 release and so better to wait.)

@huonw huonw marked this pull request as ready for review November 21, 2024 05:34
@huonw huonw merged commit 3413de5 into 2.23.x Nov 21, 2024
25 checks passed
@huonw huonw deleted the cherry-pick-21568-to-2.23.x branch November 21, 2024 05:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:internal CI, fixes for not-yet-released features, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants