Add a uv build backend #3957
Comments
uv should provide a build backend
I personally believe that this is pretty much against the whole idea of modern approach and splitting the backend vs. frontend responsibilities. The idea (philosophically) behind the backend/frontend split is that the maintainers of the project (via pyproject.toml) choose the backend that should be used to build their tool, while the user installing Python project is free to choose whatever fronted they prefer. This is IMHO a gamechanger in the python packaging and we should rather see a stronger push in that direction than weakening it. And I think it's not going to go back - because more and more projects will revert to use pyproject.toml and backend specification of build environment. In case of Airflow for example - as of December there is no way to install airlfow "natively" without actually installing hatchling and build environment. But we do not give the option to the frontend. You MUST use hatchling in specified version and few other dependencies specified in specific versions to build airflow. Full stop. UV won't be able to make their own choices (It will be able to choose a way how to create such an environment but not to choose what should be used to build airlfow). But also maybe my understanding of it is wrong and maybe you are proposing something different than I understand. BTW. I do not think access to PyPI is needed to build a project with build backend. The frontend my still choose any mechanism (including private repos if needed) to install build environment, no PyPI is needed for it, the only requirement is that pyproject.toml specifies the environment. |
|
I agree and think the backend and frontend specifications should be separate, I am merely suggesting that You do need access to PyPI or some repo hosting the build backend to build a project. Python does not include IMO, Without the capability to build from source out of the box, Python is hamstringed and can only run the most rudimentary scripts, leaving users with multi-module projects to resort to ugly |
But this is where the whole packaging for Python is heading. The PEPs of packaging precisely specify this is the direction and if project maintainers choose so, unless you manually (like conda) maintain your build recipes for all the packages our there, you won't be able to build project "natively". Just to give you example of Airflow. Without The only way to find out what dependencies Airflow needs for editable build, or to build a wheel package is to get the right version of hatchling and get the frontend execute the build_hook - the build hook returrns such dependencies dynamically. You can see it yourself here https://github.com/apache/airflow/blob/main/pyproject.toml -> there is no way to build airflow from sources in current main without actually installing those packages: And letting hatchling invoke And I think - personally (though I was not part of it) - that the decisions made by the packaging team were pretty sound and smart, and they deliberately left the decision for maintainers of a project to choose the right backend packages needed (and set of 3rd-party tools) and all frontends have no choice but to follow it. I understand you might have different opinion, but here - the process of Python Software Foundation and Packaging team is not an opinion - they have authoritative power to decide it by voting and PEP approval. And the only way to change it is to get another PEP approved. Here is the list of those PEPs (and I am actually quite happy Airflow after 10 years finally migrated out of setuptools and setup.py by following those standards as finally the tooling - including the modern backends you mentioned support it for sufficently long time):
|
|
And BTW. if uv provides a build backend, you will still be able to choose it when you maintain your project - but it will also be a 3rd-party dependency :) Someone installing your project with any frontend will have to download and install according to your specification in pyproject.toml. Similarly as hatch/hatchling pair that are separate. Even if you use hatch to install airlfow, it has to anyhow download hatchling in the version specified by the maintainer of the project you are installing and use it to build the package. |
|
It's interesting that One of the nice things about Astral tools like If you use I think [build-system]
requires = ["uv"]
build-backend = "uv.api" |
Well. There are always trade-offs and what you see from your position might be important for you, might not be important for others and the other way. For example - If (like we did with airflow being popular package) you had gone through some of the pains where new releases of setuptools suddently started breaking packages being built - because deliberately or accidentally breaking compatibilities and suddenly being flooded by 100s of your users having problem with installing your package without you doing anything you'd understand that bundling the way how things are run with specific version of frontend is a good idea when you have even moderately big package. That's what I love that we as maintainers can choose and "lock" the backend to the tools and version of our choice - rather than relying that the version of the tool that our users choose will behave consistently over the years. That was a very smart choice of packaging team based on actual learnings from their millions of users over many years, and while I often criticized their choices in the past, I came to understanding that it's my vision that is short-sighted and limited - I learned a bit of empathy. I'd strongly recommend a bit more reading and understanding what they were (and still do) cooking there. Python actually deliberately removeed setuptools in order to drive more the adoption of what's being developed as packaging standards (and that's really smart plan that was laid out years ago and is meticulously and consistently, step-by-step put in motion. And I admire the packaging team for that to be honest. What you really think about is not following and bringing back old "setuptools" behaviour is something else that packaging team has already accepted and a number of tools are implementing https://peps.python.org/pep-0723/ - which allows you to define a small subset of pyproject.toml metadata (specifically dependencies) in the single-file scripts. And this is really where yes - any front-end implementing PEP-723 should indeed prepare a venv, install dependencies and run the script that specifies such dependencies. Anything more complex that really has a bit more complex packaging need - putting more files together, should really define their backend, in order to maintain "build consistency", otherwise you start to be at mercy of tool developers who might change their behaviours at any time and suddenly not only you but anyone else who want to build your package will suddenly have problems with it. But yes. If BTW. Piece of advise - for that very reason, as a maintainer you should do: [build-system]
requires = ["uv==x.y.z"]
build-backend = "uv.build"Otherwise you never know which version of uv people have and whether they have a version that is capable of building your package at all (i.e. old version that has no backend). |
|
Also - again if you have proposal how to improve packaging, there are discourse threads there and PEP could be written, so I also recommend you, if you are strongly convinced that you can come up with a complete and better solution - please start discussion there about new PEP, and propose it, lead to approval and likely help to implement in a number of tools - this is the way how standard in packaging are being developed :) |
FYI, I beleive this is because pip maintainers are very conservative when it comes to breaking changes, not because it is the intended future of Python packaging. For example, the old resolver, which can easily install a broken environment, is still available to use even though the new resolver has been available for over 5 years and turned on by default for over 4 years. |
|
A uv-aware build backend would enable private git packages to depend on other private git packages, and still be installed with |
|
Just some more thoughts, notes, and ramblings on this. Full stack uvThe build backend capability could be rolled into the This would give developers the option to utilize uv as either a frontend, a backend, or both, while still maintaining the single static dependency. Developers electing to use uv as both frontend and backend could have access to some optimizations that might not be possible individually. We could call this use case “full stack uv” since that’s usually what we call frontend + backend, right? 🤪 Full stack uv without a specified versionrequires = ["uv"]
build-backend = "uv"Backend Or,
Full stack uv with pinned or maximum versionrequires = ["uv==0.5"]
build-backend = "uv"PEP 517 build isolation can guarantee that frontend
However, a decision could be made to maintain backward compatibility for the backend, such that newer versions of uv could satisfy whichever version is declared. On PEP 517 complianceThe backend feature could be PEP 517 compliant so that other frontends (like poetry, pdm, pip, hatch, etc.) can use uv as a backend, but this could be distinct from full stack uv.
uv would need to expose some hooks in the build environment via Python API. The mandatory and optional hooks at the time of this writing are: # Mandatory hooks
def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
...
def build_sdist(sdist_directory, config_settings=None):
...
# Optional hooks
def get_requires_for_build_wheel(config_settings=None):
...
def prepare_metadata_for_build_wheel(metadata_directory, config_settings=None):
...
def get_requires_for_build_sdist(config_settings=None):
...Perhaps a highly optimized, importable module or package named
|
|
If it was up to me, and assuming the build backend isn't too complicated, I'd consider doing both:
|
|
Hatch does 2. with Hatchling. |
|
Yes, thanks @nathanscain, I guess I'm just a bit surprised 🙂 From what I understood reading the standard, that was not the intended use-case. Unless any other build frontend can also send these build options to the flit backend, then these are not build options, it's just backend logic hardcoded in the frontend.
Of course, but that should be specified as backend options, not front-end ones 😄 |
|
I checked flit's backend source code, and it doesn't use the received |
|
Oh wow... that is unfortunate... |
sdists do not matter for building the project at all. In fact, when using the github archives generated tarball of a project instead of the sdist, you can build with any frontend plus flit_core and get the same files in the wheel and installed to your python environment. This is already the case today since many projects using hatchling or poetry don't include important files needed to run testsuites in their sdists, since they explicitly Again, it doesn't matter whether you use flit's backend hook for producing an sdist produces an sdist that is useless for a number of purposes that people use sdists for, and can only be used for building a wheel. That's fine if what you want is to build a wheel. Note also that flit doesn't support dynamic versions from a build stage such as VCS info. For that you need https://pypi.org/project/flit-scm/ (a separate build backend that first runs setuptools-scm to write VCS info to a file and then reimports
The standard only allows sending user configuration to the backend, in the form of pyproject-build's
Because flit doesn't support build options. User configuration in the form of build options is a direct analogue to GNU Autotools / meson / cmake support for things like Flit doesn't support dynamism or compiled C extensions so config_settings is irrelevant to it. |
I don't really understand the purpose of this FUD, sorry. Can you please point to me where you think it's philosophically possible to get different results from porting flit_core into rust code, "because it calls the python standard library which yields differences"? I'm not sure what you are getting at. Are you, I dunno, afraid that python's filesystem tree-walking algorithm for iterating over all files in src/ is going to yield a different set of installable files compared to a rust-based filesystem tree-walking algorithm for iterating over all files in src/?
That is not the meaning of the word "algorithm", and that is not the meaning of the word "standard", so it seems we are at an impasse but not the impasse that I expected. |
The point was that instead of letting users pass something like EDIT: well, |
Trying to identically copy a tool is bad. It leads to ad hoc implied standards, bugs, and descrepencies. There are a myriad reasons, but I'm firmlymof the opinion the Python packaging system wasn't able to evolve until we were able to move past "do whatever setuptools and pip does". I would be sad for us to move to "do whatever flit does".
I'm talking about a standard in the context of the Python packaging ecosystem which means an accepted PEP, it is all willing and interested parties coming together to agree how something should be defined for implementation. As we're talking in the context of Python packaging I would suggest we should all be using this term this way, anything else is just going to add confusion. By algorithm I mean something that you could write out confidentially in psudo code. I think this would be non trivial for flit, least of which because of it's giant regexes which depend on the Python regex engine, as well as, yes, it's various IO interfaces which depend on Python implementations. I agree there are other and more broad usages and definitions of the word. |
Genuinely baffled. The reason why the Python packaging system wasn't "able to evolve" except by moving past setuptools is because setuptools does not document what it does or how it works and it supports vast numbers of moving parts. But primarily because it exposes an extensive (and again, effectively undocumented) python API which people code against. This is a design problem but it's not one that a generic axiom can be derived from. There is no logical reason why another build backend has to leave the majority of its own features undocumented. (Regarding pip, lots of people moved past "do whatever pip does", way back in 2000 and in some cases earlier. Conda, a latecomer to the party, moved past it in 2012. It doesn't actually matter either way -- pip has never constrained the packaging system, it's only constrained workflow UX which I'm not all that interested in and isn't the topic of this issue really.)
I object to your definition of the word standard in the context of the Python packaging ecosystem! A standard, in the context of the Python packaging ecosystem, is when multiple ways to do the same generic concept exist and a document (referred to by the term "standard") says how tools should expose an interoperability interface for other tools to use. Fortuitously, that's not specific to the Python packaging ecosystem -- that's also how the term "standard" is used in other fields of endeavor -- the Python packaging ecosystem part is that standards are proposed via the PEP process. Standards are unrelated to whether a specific program documents its own behavior with enough fidelity to port it to another programming language as necessary. That's not an interoperability topic.
I'm only aware of two places in which flit_core meaningfully utilizes regular expressions and both of them are PEPs that have been transcribed from pseudocode into python's re module so there is no requirement to depend on the python regex engine. The tool.flit section doesn't support passing user-defined regular expressions as far as I can tell, but I'd love to hear more if you're aware of something I missed. Regarding IO interfaces which depend on python implementations, I simply don't believe you at all, and if what you claimed were true then it would be impossible to write software in python at all1 because you could never be sure if it behaved according to https://docs.python.org and you would encounter impossible filesystem IO errors that defy your operating system kernel's documentation. If that's not what you mean then please do explain what you mean when you say that it is not practical for an individual to compare behavior of a python program's usage of IO apis and a rust / C++ program's usage of IO apis and write the same program in multiple programming languages. Footnotes
|
Yeah, it seems we don't have enough shared language to communicate on this one, so I'll leave this as my final post to this particular thread, if you're still baffled I don't think me replying any further will help sorry.
Okay, but the way that was "fixed" is by definining standards as PEPs, and getting pip and setuptools to follow them (or close enough), that other tools could successfully interoperate with them.
I think your timeline is off, maybe you're thinking of easy install? Pip was first released in 2008. Conda is an excellent example of why packaging standards were needed here for both frontend and backend. Conda is not part of the Python packaging ecosystem, it is a meta packager that has to keep its own metadata separate from the Python packaging data. In a huge number of cases the build step of a conda package that packages a Python package is to call pip. If uv had to do what conda did we wouldn't be talking about front ends or back ends or interoperability. I see this as a great success of the Python packaging standards (PEPs and their implementations).
I'm a bit confused why because you go on to describe the PEP process:
This is the PEP process and the standards documents that a PEP contributes to is at: https://packaging.python.org/en/latest/specifications/
That's not what I said or what I meant, but I genuinely don't think it's worth hashing over between the two of us. It only matters if Astral think that 1) creating a flit fast path meets their design goals? And 2) if they agree with you that it's doable to copy flits behavior so exactly that it will not cause issues (at least for them)? Though I suspect the answer to 1) though is no, making hashing over 2) doubly pointless! But we'll see. |
|
If anything - the length of this thread indicates how important and how complex the problem is. Which should - I hope - give |
|
And since holidays are coming - I hope we can all stay with our families over the next few days and get a bit of a pause on all github-related stuff. Happy Holidays everyone ! |
My timeline (2000 for the date when people moved past the need for tools first released in 2008) was highly intentional and could be seen as a wry comment about pip itself. Remember that pip is merely a workflow tool and many people never adopted it in the first place. It's simply a bad comparison to build backends.
None of that metadata is defined by pip-centered standards, and conda also distributes the metadata from build backends, just like pip does. The additional metadata that conda tracks but pip doesn't track is metadata that requires packaging and distributing non-python software, which naturally doesn't participate in localized python packaging ecosystem standards and never will (conda installs pure python wheels the same way pip does, and pip doesn't install GCC or boost or OpenBLAS at all but conda does). The proposed fixes for this discrepancy between conda and pip have been about enhancing metadata for build backends, not build frontends, because it's not a frontend problem! But this highlights something that bothers me about the python community as a whole, and it is: people discover that software is hard and ISVs actually do serve a useful role, and respond in fear by refusing to talk about or consider making software a better place except where mandated by a PEP standard. Case in point: "no one should reimplement flit_core in rust unless flit_core is described in a PEP standard. Having flit_core documentation is not good enough because it's not official unless it's a PEP, you can't know how to implement flit_core unless it's a PEP". Which is a pretty fearful way of looking at the world, I have to say.
My apologies for the confusion. In the later two paragraphs I implied that I felt I needed to describe the PEP process because you were not describing the PEP process.
Okay, magic smoke and pixie dust it is. You originally said "platforms have quirks, flit calls the Python standard library which uv can't, and there are different implementations of the standard library on certain platforms" as a reason for why my suggestion to port a program from python to rust is impractical (not create a competing implementation, program using the same configuration, port the original program). This analysis worries me because I have a use case (not related to uv) for implementing flit_core according to its documentation (but not in rust specifically) and it sounds like you would call the results buggy due to an intrinsic unreliability problem of the python programming language (but I don't see the bug). |
I would be happy to discuss on the relevant repo in detail, I believe enough detail has been discussed here given no one for Astral has participated in a while, especially on their design goals. It could work quite well for your needs, which I think are likely different from uv's ("just works" for millions of users and configurations and platforms and impacts a significant % of the packaging ecosystem). Bluntly said though, I don't appreciate the magic smoke, FUD, and other subjective negative meta comments, I won't engage with you in any context if they continue. I appreciate you're frustrated I'm not willing to be dragged into details and sticking to high level examples and points, but I don't think it's worth writing technical papers here, Astral know better their technical experience implementing Python packaging tooling in Rust than I do. Maybe I'm 100% wrong. We're all here in good faith trying to help the Python packaging ecosystem with the best experience we can, and I think we've all got relevant experience here. |
|
I appreciate all the discussion and we will of course take these concerns into account as we figure out our exact plans here. For big questions and projects like this, we tend to take our time in working through our answers, since we're a small team (with some folks are out for the holidays in this case) and want to able to think deeply (over reacting quickly) on anything with a long-term impact and complex design considerations. So, please bear with us. In the meantime, I'd just ask that we curtail any more conversation around the pros and cons. My impression is that everyone has had a chance to speak their minds (and should feel heard -- we'll read it all), and I'd prefer to end with productive disagreement rather than any other negative sentiment. Happy Holidays! |
|
To make it both sides of this thread happy I want to point to @hauntsaninja's comment above. In addition, Definitely, that would have some maintainability costs:
Current vs Future implementationI think that current path of implementation is correct (which is still in preview) that it provides an option for build backend (which is correctly not set as default) to force every build frontend to use Building projects with
|
|
Interesting. So, like what Flit is doing actually, without the mistake (IMO) of extending the backend features in the frontend. Also what Hatch is doing based on a comment from @ofek earlier. I wouldn't recommend having two build-backends though, but only one that is pure-python. Then uv, as a frontend, can detect it and hijack it with a Rust implementation. Other frontends would work as usual with the pure Python backend. |
|
The Astral team requested that we not keep going back and forth with pros/cons etc until they have time to review the current discussion and talk it out internally As Charlie said, everyone seems to have said their piece and has been heard. Have a Merry Christmas everyone 🎄 |
|
@charliermarsh Are there any news on this? Is it discussed only internally or is there some other location where the discussion takes place? |
|
We're discussing this internally. We'll chime in here eventually. |
|
We've been thinking about all the concerns noted here and have some thoughts to share. As a TL;DR:
I’ll talk through our decision, starting with some background on our motivations. When we first launched uv's project management capabilities, we tried using a third-party backend by default. The feedback from users about this experience was very negative: there was confusion that errors were coming from another tool, that they needed to read its documentation to understand what was happening, and that they often had to add configuration for it to work with their project. In response to this feedback, we decided not to use a build system in We cannot deliver the user experience we want with existing build backends — we need better error messages, better defaults, and more robust handling of edge cases. Due to the intentionally isolated nature of a build backend, there is context about the user's intent that cannot be captured without integration with the build frontend. For example, an integrated backend lets us understand why the frontend is building the package or why the backend failed so we can provide error messages that guide people to a successful build. We have devoted a lot of resources to improving build error messages already by parsing the output of a build backend to provide hints to users, but that’s brittle. While it’s a major benefit that we can special case our own backend (as done in Poetry and Hatch), the build backend will be usable with a different frontend than uv and all of the other build backends will continue to be supported by uv’s frontend. We’ve always focused heavily on standards compliance and will continue to do so. Shifting gears, there are a few alternative suggestions in the discussion:
I'll respond to these next. With uv, we've invested heavily in a solid foundation for Python packaging in Rust. We have a lot of code and abstractions, with a focus on correctness and performance. It's not feasible for us to maintain duplicate implementations in Python, as in (1). We think this is infeasible both due to time constraints and, as mentioned by some, foundational differences in behavior between the Rust and Python standard libraries — subtly different behavior here would be a disaster. We want to keep our rate of development high and deliver features to users — we can't do that while maintaining multiple versions of our core abstractions. For similar reasons, we think that (2) is infeasible. Even if we accept the premise that it is feasible, it's against our product principles to shadow another tool implicitly and re-use their configuration. Additionally, our goal is not just to "make a more performant build backend". As some mentioned, the overhead of a build backend is generally minimal (though noticeable when using uv). We want to innovate on the experience of building Python packages because it is a core part of working on Python projects. The feedback from our users (in the form of questions and bug reports in our issue tracker) has made it clear that improving integration with a build backend will be very high impact. At this point, we’re at “build a backend in Rust” or “don’t do it at all” (3) — there are a few recurring concerns that support the latter conclusion. We’re taking these concerns seriously, but want to balance them with the value we see for users. I'll do my best to address them here, but I'll admit that packaging is a very hard problem and I do not have good answers to all of the concerns raised.
With that, I want to take a second to thank everyone for their feedback here. It’s helped us prioritize some changes (like a separate package) that we may not have otherwise. Going forward, I'd like to focus this issue on the behavior of the build backend within this framing instead of debating if it should be done or not. |
(2) should help a lot, I think. In particular it seems reasonable to assume that the MSRV of a dedicated build-backend crate can be a lot lower than the MSRV of uv as a whole. In particular, if you can avoid entirely the use of problem crates such as ring. That being said, the arguments with regard to (1) aren't as compelling as they could be. There are CPU architectures where Rust isn't available at all, and therefore cryptography isn't available either. I would say pydantic isn't available either, except that really, pydantic can be commonly downloaded all it wants but it is hardly core infrastructure. Cryptography isn't universally needed, there's absolutely loads of stuff Gentoo users commonly install that are written in Python, without ever ending up hitting a dependency tree that leads back to cryptography... but there are definitely some packages that are uninstallable for Gentoo users on less common architectures, solely because of cryptography. Build backends are a much bigger influence here than something like pydantic or cryptography, because they could potentially influence "core infrastructure" packages that are written in pure python. For something verging on a worst-case example, what if "trove-classifiers" ported to the uv build backend because they heard it was so popular nowadays, and also it's basically a data package so what's the big deal and who really needs it except for other package developers. Except, setuptools and flit don't care but hatchling depends on it so now a massive number of packages can't be built on non-rust platforms. There are other packages that are common runtime packages needed by other software which could be a huge problem as well, e.g. attrs, certifi, appdirs, xdg... so we end up in a situation where people are told about a new integrated backend that Having a pure python version would at least prevent this invisible line across which package maintainers have to magically know that it's socially bad to use a specific backend.
I'm not sure it's conceptually solvable. Some projects install their testsuite inside the wheel, that's about all you can do. But most projects consider it bad to install tests as part of the wheel, as it penalizes pip install users. The handful that install their testsuite tend to argue that they are the exceptional case and that end users creating a virtualenv should then go ahead and import and run the testsuite to make sure everything actually works... because they're pretty confident that there are cases where it doesn't work. It's a common concern in the science ecosystem, for example (delicate interactions between various moving parts can do that for you 🤷). We may also have to first solve the problem where large subsets of the python packaging ecosystem object to including tests even in the sdist, as they regard sdists as purely for pip to build wheels and see tests as dead weight. That includes going around to other projects and advocating to not include tests in the sdist! :( One possibility is to download both the wheel and the sdist and then install the wheel and test it using the sdist copy of the test suite. But it's not immediately clear what the advantages are over just building the sdist using a build backend. It would add great complexity to the process, doesn't solve the occasional need to backport a patch that fixes crashes, can't be used for anything that has compiled extensions, doubles or triples the download size... It's something I've thought about before, but with the association "worst case scenario, we might be able to do this to get out of a very sticky situation". |
I should take this back. Sigstore appears to rely on it, which means it is extremely problematic to validate cryptographic signatures for cpython starting with 3.14. Oh well, we can just assume it was verified on an amd64 developer machine and disable signature verification -- and the Gentoo PGP signature upon the checksum manifest for cpython will prove that that was checked by someone else. Not great, but not the end of the world. Still, I do admit I was wrong. Pydantic is used for more than I thought... |
uv is a fantastic tool that is ahead of its time. In the same vein as ruff, it is bundling many capabilities that Python developers need into a single tool. It currently provides the capabilities of
pip,pip-tools, andvirtualenvin one convenient binary.Python can't build out of the box
As of Python 3.12, which has removed
setuptoolsandwheelfrom standard Python installations, a user is unable to performpip install -e .orpip install .in a localpyproject.tomlproject without pulling in external third-party package dependencies.This means that in an offline environment without access to PyPI, a developer is dead in the water and cannot even install their own project from source. This is a glaring flaw with Python, which is supposed to be "batteries included."
uv can fix this
I propose that the
uvbinary expand its capabilities to also function as a build backend.If uv could natively build projects from source, it would be a game changer!
The text was updated successfully, but these errors were encountered: