uv lock specifier instability? #6316
Comments
|
Is this happening with a lockfile initialized with 0.3.0, i.e. does 0.3.0 ever write out |
|
For context, i have confirmed that hatchling flips the order of the specifier ( |
|
Yes, it happens when starting with uv 0.3.0. This whimsical sequence of commands ends up with a diff like in the issue report diff --git uv.lock uv.lock
index 1f890d7..f606d83 100644
--- uv.lock
+++ uv.lock
@@ -346,7 +346,7 @@ dependencies = [
[package.metadata]
requires-dist = [
- { name = "numpy", specifier = "<2,>=1.25.2" },
+ { name = "numpy", specifier = ">=1.25.2,<2" },
{ name = "papermill", specifier = ">=2.4.0" },
]I forgot to pin python in this command sequence. So it reproduced with python 3.12.4 for me there. |
konstin
added a commit
that referenced
this issue
Aug 21, 2024
…lly from `pyproject.toml` or dynamically from the build backend. Python's `packaging` [sorts](https://github.com/pypa/packaging/blob/cc938f984bbbe43c5734b9656c9837ab3a28191f/src/packaging/specifiers.py#L777) specifiers before emitting them, so all build backends built on top of it - such as hatchling - will change the specifier order compared to pyproject.toml. The core metadata spec does say "If a field is not marked as Dynamic, then the value of the field in any wheel built from the sdist MUST match the value in the sdist", but it doesn't specify if "match" means string equivalent or semantically equivalent, so it's arguable if that spec conformant. This change means that the specifiers have a different ordering when coming from the build backend than when read statically from pyproject.toml. Previously, we tried to read path dep metadata in order: * From the (built wheel) cache (`packaging` order) * From pyproject.toml (verbatim specifier) * From a fresh build (`packaging` order) This behaviour is unstable: On the first run, we cache is cold, so we read the verbatim specifier from `pyproject.toml`, then we build and store the metadata in the cache. On the second run, we read the `packaging` sorted specifier from the cache. Reproducer: ```shell rm -rf newproj uv init -q --no-config newproj cd newproj/ uv add -q "anyio>=4,<5" cat uv.lock | grep "requires-dist" uv sync -q cat uv.lock | grep "requires-dist" cd .. ``` ``` requires-dist = [{ name = "anyio", specifier = ">=4,<5" }] requires-dist = [{ name = "anyio", specifier = "<5,>=4" }] ``` A project either has static metadata, so we can read from pyproject.toml, or it doesn't, and we always read from the build through `packaging`. We can use this to stabilize the behavior by slightly switching the order. * From pyproject.toml (verbatim specifier) * From the (built wheel) cache (`packaging` order) * From a fresh build (`packaging` order) Potentially, we still want to sort the specifiers we get anyway, after all, the is no guarantee that the specifiers from a build backend are deterministic. But our metadata reading behavior should be independent of the cache state, hence changing the order in the PR. Fixes #6316
konstin
added a commit
that referenced
this issue
Aug 21, 2024
For a path dep such as the root project, uv can read metadata statically from `pyproject.toml` or dynamically from the build backend. Python's `packaging` [sorts](https://github.com/pypa/packaging/blob/cc938f984bbbe43c5734b9656c9837ab3a28191f/src/packaging/specifiers.py#L777) specifiers before emitting them, so all build backends built on top of it - such as hatchling - will change the specifier order compared to pyproject.toml. The core metadata spec does say "If a field is not marked as Dynamic, then the value of the field in any wheel built from the sdist MUST match the value in the sdist", but it doesn't specify if "match" means string equivalent or semantically equivalent, so it's arguable if that spec conformant. This change means that the specifiers have a different ordering when coming from the build backend than when read statically from pyproject.toml. Previously, we tried to read path dep metadata in order: * From the (built wheel) cache (`packaging` order) * From pyproject.toml (verbatim specifier) * From a fresh build (`packaging` order) This behaviour is unstable: On the first run, we cache is cold, so we read the verbatim specifier from `pyproject.toml`, then we build and store the metadata in the cache. On the second run, we read the `packaging` sorted specifier from the cache. Reproducer: ```shell rm -rf newproj uv init -q --no-config newproj cd newproj/ uv add -q "anyio>=4,<5" cat uv.lock | grep "requires-dist" uv sync -q cat uv.lock | grep "requires-dist" cd .. ``` ``` requires-dist = [{ name = "anyio", specifier = ">=4,<5" }] requires-dist = [{ name = "anyio", specifier = "<5,>=4" }] ``` A project either has static metadata, so we can read from pyproject.toml, or it doesn't, and we always read from the build through `packaging`. We can use this to stabilize the behavior by slightly switching the order. * From pyproject.toml (verbatim specifier) * From the (built wheel) cache (`packaging` order) * From a fresh build (`packaging` order) Potentially, we still want to sort the specifiers we get anyway, after all, the is no guarantee that the specifiers from a build backend are deterministic. But our metadata reading behavior should be independent of the cache state, hence changing the order in the PR. Fixes #6316
konstin
added a commit
that referenced
this issue
Aug 21, 2024
For a path dep such as the root project, uv can read metadata statically from `pyproject.toml` or dynamically from the build backend. Python's `packaging` [sorts](https://github.com/pypa/packaging/blob/cc938f984bbbe43c5734b9656c9837ab3a28191f/src/packaging/specifiers.py#L777) specifiers before emitting them, so all build backends built on top of it - such as hatchling - will change the specifier order compared to pyproject.toml. The core metadata spec does say "If a field is not marked as Dynamic, then the value of the field in any wheel built from the sdist MUST match the value in the sdist", but it doesn't specify if "match" means string equivalent or semantically equivalent, so it's arguable if that spec conformant. This change means that the specifiers have a different ordering when coming from the build backend than when read statically from pyproject.toml. Previously, we tried to read path dep metadata in order: * From the (built wheel) cache (`packaging` order) * From pyproject.toml (verbatim specifier) * From a fresh build (`packaging` order) This behaviour is unstable: On the first run, we cache is cold, so we read the verbatim specifier from `pyproject.toml`, then we build and store the metadata in the cache. On the second run, we read the `packaging` sorted specifier from the cache. Reproducer: ```shell rm -rf newproj uv init -q --no-config newproj cd newproj/ uv add -q "anyio>=4,<5" cat uv.lock | grep "requires-dist" uv sync -q cat uv.lock | grep "requires-dist" cd .. ``` ``` requires-dist = [{ name = "anyio", specifier = ">=4,<5" }] requires-dist = [{ name = "anyio", specifier = "<5,>=4" }] ``` A project either has static metadata, so we can read from pyproject.toml, or it doesn't, and we always read from the build through `packaging`. We can use this to stabilize the behavior by slightly switching the order. * From pyproject.toml (verbatim specifier) * From the (built wheel) cache (`packaging` order) * From a fresh build (`packaging` order) Potentially, we still want to sort the specifiers we get anyway, after all, the is no guarantee that the specifiers from a build backend are deterministic. But our metadata reading behavior should be independent of the cache state, hence changing the order in the PR. Fixes #6316
|
Thank you for the great reproducer! |
konstin
added a commit
that referenced
this issue
Aug 21, 2024
For a path dep such as the root project, uv can read metadata statically from `pyproject.toml` or dynamically from the build backend. Python's `packaging` [sorts](https://github.com/pypa/packaging/blob/cc938f984bbbe43c5734b9656c9837ab3a28191f/src/packaging/specifiers.py#L777) specifiers before emitting them, so all build backends built on top of it - such as hatchling - will change the specifier order compared to pyproject.toml. The core metadata spec does say "If a field is not marked as Dynamic, then the value of the field in any wheel built from the sdist MUST match the value in the sdist", but it doesn't specify if "match" means string equivalent or semantically equivalent, so it's arguable if that spec conformant. This change means that the specifiers have a different ordering when coming from the build backend than when read statically from pyproject.toml. Previously, we tried to read path dep metadata in order: * From the (built wheel) cache (`packaging` order) * From pyproject.toml (verbatim specifier) * From a fresh build (`packaging` order) This behaviour is unstable: On the first run, we cache is cold, so we read the verbatim specifier from `pyproject.toml`, then we build and store the metadata in the cache. On the second run, we read the `packaging` sorted specifier from the cache. Reproducer: ```shell rm -rf newproj uv init -q --no-config newproj cd newproj/ uv add -q "anyio>=4,<5" cat uv.lock | grep "requires-dist" uv sync -q cat uv.lock | grep "requires-dist" cd .. ``` ``` requires-dist = [{ name = "anyio", specifier = ">=4,<5" }] requires-dist = [{ name = "anyio", specifier = "<5,>=4" }] ``` A project either has static metadata, so we can read from pyproject.toml, or it doesn't, and we always read from the build through `packaging`. We can use this to stabilize the behavior by slightly switching the order. * From pyproject.toml (verbatim specifier) * From the (built wheel) cache (`packaging` order) * From a fresh build (`packaging` order) Potentially, we still want to sort the specifiers we get anyway, after all, the is no guarantee that the specifiers from a build backend are deterministic. But our metadata reading behavior should be independent of the cache state, hence changing the order in the PR. Fixes #6316
|
heh, thank you, your reduced sequence was better |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In some cases I see the following diff in my project's lock file. The order of the specifiers is flipping back and forth. (Both sides of the diff are using uv 0.3.0).
The source is
pyproject.toml's dependencies = ["numpy>=1.25.2,<2"] (with more dependencies specified).Is it possible there is some leeway in the ordering for these? Using uv 0.3.0.
Seems to reproduce with the full dependency list, after a few uv sync and uv locks.
Using
uv python pin 3.11. Defacto python iscpython-3.11.7-linux-x86_64-gnu.(Edited here: reduced problem to only two dependencies)
Seems to in fact correlate with adding/removing aiohttp too, which happens without change to pyproject.toml.
The text was updated successfully, but these errors were encountered: