Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make MODULE.bazel.lock file have the same contents no matter the platform bazel is executed on #1643

Closed
aignas opened this issue Dec 21, 2023 · 0 comments · Fixed by #1937 or #1946
Closed

Make MODULE.bazel.lock file have the same contents no matter the platform bazel is executed on #1643

aignas opened this issue Dec 21, 2023 · 0 comments · Fixed by #1937 or #1946
Assignees
@aignas
Labels
type: bzlmod bzlmod work type: feature request type: pip pip/pypi integration

Comments

@aignas
Copy link
Collaborator

aignas commented Dec 21, 2023 •
edited
Loading

This is a ticket to tidy up and do feature work to ensure that the pip bzlmod extension can generate a lock file that is os,arch independent. The rough list of steps that we need to take are:

  • Create a host toolchain that can be passed to the python_interpreter_target. We need to ensure that the label that is passed to the whl_library repo rule is the same no matter the platform.
  • Introduce a way to handle multiple requirements files and generate the wheel repositories similar to how it is done in exp: multi-arch whl_library in bzlmod #1625.
  • Remove the requirement file selection based on the host platform and instead generate the repos for all available requirement files. Because the platform specific requirements files may have different versions of packages, we have to go all the way through and go with the approach outlined in exp: multi-arch whl_library in bzlmod #1625 in the general case to be fully correct.

To make the pip extension more maintainable we might also need to have:

  • config_setting values for (pyXY, os, cpu) triples so that we can more easily create the required aliases in the hub repository.
  • Making the dependency resolution aware of the python abi.

The above would make the number of layers of indirection fewer and hopefully make it easier to extend the code in general.

This general work may be related fix some aspects of #310 for bzlmod.

Original issue that spurred a feature on bazel itself: bazelbuild/bazel#19154. The motivation for restarting the effort is the fact that maintaining a lock file that has os,arch specific sections is hard and requires updating the lock file by running bazel on a number of platforms and this just does not scale in the long term.
aignas added a commit to aignas/rules_python that referenced this issue Dec 21, 2023
@aignas
feat(toolchain, pip.parse): introduce a new '_host' toolchain repo
a82d3d7 
This is for passing it in repository_rules and relies on the canonical
label representation introduced in bazel 6.0 and symlink support (needs
to be present on Windows) to work.

This allows the users to not need to `load` the interpreter label from
a `.bzl` file but instead specify the label in the form of
`@<mytoolchainrepo>_host//:python`.

Work towards bazelbuild#1643.
This was referenced Dec 21, 2023
Merged
Open
aignas added a commit to aignas/rules_python that referenced this issue Dec 21, 2023
@aignas
feat(toolchain, pip.parse): introduce a new '_host' toolchain repo
3d6e8bd 
This is for passing it in repository_rules and relies on the canonical
label representation introduced in bazel 6.0 and symlink support (needs
to be present on Windows) to work.

This allows the users to not need to `load` the interpreter label from
a `.bzl` file but instead specify the label in the form of
`@<mytoolchainrepo>_host//:python`.

Work towards bazelbuild#1643.
aignas added a commit to aignas/rules_python that referenced this issue Dec 21, 2023
@aignas
feat(toolchain, pip.parse): introduce a new '_host' toolchain repo
6e8f8c9 
This is for passing it in repository_rules and relies on the canonical
label representation introduced in bazel 6.0 and symlink support (needs
to be present on Windows) to work.

This allows the users to not need to `load` the interpreter label from
a `.bzl` file but instead specify the label in the form of
`@<mytoolchainrepo>_host//:python`.

Work towards bazelbuild#1643.
aignas added a commit to aignas/rules_python that referenced this issue Dec 22, 2023
@aignas
feat(toolchain, pip.parse): introduce a new '_host' toolchain repo
9305185 
This is for passing it in repository_rules and relies on the canonical
label representation introduced in bazel 6.0 and symlink support (needs
to be present on Windows) to work.

This allows the users to not need to `load` the interpreter label from
a `.bzl` file but instead specify the label in the form of
`@<mytoolchainrepo>_host//:python`.

Work towards bazelbuild#1643.
@dtolnay dtolnay mentioned this issue Dec 28, 2023
Merged
@leungster leungster mentioned this issue Jan 5, 2024
Closed
aignas added a commit to aignas/rules_python that referenced this issue Jan 9, 2024
@aignas
feat(toolchain, pip.parse): introduce a new '_host' toolchain repo
991d4c0 
This is for passing it in repository_rules and relies on the canonical
label representation introduced in bazel 6.0 and symlink support (needs
to be present on Windows) to work.

This allows the users to not need to `load` the interpreter label from
a `.bzl` file but instead specify the label in the form of
`@<mytoolchainrepo>_host//:python`.

Work towards bazelbuild#1643.
github-merge-queue bot pushed a commit that referenced this issue Jan 9, 2024
@aignas
feat(toolchain, pip.parse): introduce a new '_host' toolchain repo (#…
4fe0db3 
…1644)

This is for passing it in repository_rules and relies on the canonical
label representation introduced in bazel 6.0 and symlink support (needs
to be present on Windows) to work.

This allows the users to not need to `load` the interpreter label from
a `.bzl` file but instead specify the label in the form of
`@<mytoolchainrepo>_host//:python`.

In order to make it work robustly on Windows, we do
`repository_ctx.path(python_interpreter_label).realpath` to get the
actual
path of the Python interpreter on both, bzlmod and non-bzlmod setups on
Windows within the `whl_library` repository rule. 

Work towards #1643.
This was referenced Jan 14, 2024
Merged
Closed
github-merge-queue bot pushed a commit that referenced this issue Jan 14, 2024
@aignas
refactor(internal): simplify and cleanup dependency resolution (#1691)
e58d2a2 
Summary:
- refactor: add an OS.interpreter constructor
- refactor: add an Arch.interpreter constructor
- test: add a specializations tests
- refactor: simplify sorting implementation
- refactor: introduce `__str__` to enums

Minor cleanup that will be useful to make this codebase
work for multiple Python versions.

Work towards #1643.
@aignas aignas mentioned this issue Jan 14, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Jan 25, 2024
@aignas
feat: support parsing whl METADATA on any python version (#1693)
0aa5ea4 
With this PR we can deterministically parse the METADATA and generate a
`BUILD.bazel` file using the config settings introduced in #1555. Let's
imagine we had a `requirements.txt` file that used only wheels, we could
use the host interpreter to parse the wheel metadata for all the target
platforms and use the version aware toolchain at runtime. This
potentially
unlocks more clever layouts of the `bzlmod` hub repos explored in #1625
where we could have a single `whl_library` instance for all versions
within
a single hub repo.

Work towards #1643.
@aignas aignas added type: feature request type: bzlmod bzlmod work type: pip pip/pypi integration labels Feb 13, 2024
@aignas aignas mentioned this issue Mar 10, 2024
Closed
2 tasks
@aignas aignas self-assigned this Apr 17, 2024
@aignas aignas mentioned this issue May 17, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue May 19, 2024
@aignas
feat(pip): support specifying requirements per (os, arch) (#1885)
a6cb620 
This PR implements a better way of specifying the requirements files for
different (os, cpu) tuples. It allows for more granular specification
than what
is available today and allows for future extension to have all of the
sources
in the select statements in the hub repository.

This is replacing the previous selection of the requirements and there
are a
few differences in behaviour that should not be visible to the external
user.
Instead of selecting the right file which we should then use to create
`whl_library` instances we parse all of the provided requirements files
and
merge them based on the contents. The merging is done based on the
blocks
within the requirement file and this allows the starlark code to
understand if
we are working with different versions of the same package on different
target
platforms.

Fixes #1868
Work towards #1643, #735
@aignas aignas mentioned this issue Jun 1, 2024
Merged
aignas added a commit to aignas/rules_python that referenced this issue Jun 2, 2024
@aignas
fix(pip.parse): make the pip extension reproducible if PyPI is not ca…
9d35b35 
…lled

With this PR we can finally have fewer lock file entries in setups which
do not use the `experimental_index_url` feature yet. This is fully
backwards compatible change as it relies on `bazel` doing the right
thing and regenerating the lock file.

Fixes bazelbuild#1643.
@aignas aignas mentioned this issue Jun 2, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Jun 6, 2024
@aignas
fix(pip.parse): make the pip extension reproducible if PyPI is not ca…
e42f8f4 
…lled (#1937)

With this PR we can finally have fewer lock file entries in setups which
do not use the `experimental_index_url` feature yet. This is fully
backwards compatible change as it relies on `bazel` doing the right
thing and regenerating the lock file.

Fixes #1643.
aignas added a commit to aignas/rules_python that referenced this issue Jun 10, 2024
@aignas
fix: make the pythons_hub platform independent
3e0c432 
With this change we can finally have platform independent bzlmod lock
files in rules_python. This cleans up code paths that were not being
used in the `pip` extension for a while and the replacement is noted in
the CHANGELOG.

Fixes bazelbuild#1643
@aignas aignas mentioned this issue Jun 10, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Jun 10, 2024
@aignas
fix: make the pythons_hub platform independent (#1946)
49cdf7d 
With this change we can finally have platform independent bzlmod lock
files in rules_python. This cleans up code paths that were not being
used in the `pip` extension for a while and the replacement is noted in
the CHANGELOG.

Fixes #1643
@willjschmitt willjschmitt mentioned this issue Aug 7, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aignas aignas

Labels
type: bzlmod bzlmod work type: feature request type: pip pip/pypi integration
Projects
None yet
Milestone
No milestone
1 participant
@aignas