Venvs from uv and from native python works a little different on nix #4450
Comments
|
Is this roughly the same as #1795 -- that if we resolved the symlink on the interpreter, we would match the behavior you're expecting? |
I'm not sure.
Sounds like this more about creating venv from existing venv. |
|
The fact that python from nix store isn't treated as a system interpreter isn't really an issue imo, as we can easily set the UV_PYTHON env dynamically in a derivation. The real issue is the fact that uv wants to create a And with that said, I think we would need to at least be able to set where the lock file is created... unless that defeats the purpose of a lock file, but I don't know if that's a standard for python-related software to be honest... |
|
We could make the |
|
Hmm, I think the problem could be elsewhere. I've managed to make So for my setup which uses nix flake, the first fix was to make Since this is a nix flake and not NixOS distro which has the entire /nix/store directory mounted as read-only, I just made that path Then I've realized that after I unset the variable, the lock gets created, as I suspect correctly, inside |
I think, I quite disagree there. .lock problem is kinda a symptom, while wrong pythons selection / strange venvs are disease.
That’s interesting. I did not use environment variables at all, only cli args. I’ll recheck this issue using env vars. |
|
Well, there is an issue with symlink resolution for sure, but I agree 100% with what you are saying. After reproducing this issue on a different machine, I realized that I didn't manage to make it work... What worked was actually a different tool (hatch), as since it can be configured to use uv, I mistakenly assumed that it also uses it for venv creation. I believe that hatch uses |
|
And now I'm really confused. Creating venv via python moduleuv venvAnd fun fact, with venv module, |
Yeah -) This is literally the spirit of this issue) uv’s venv have python pointing to python wrapper in nix’s python environment, while the native python venv have py pointing to original python binary (and derivation, ofc) |
|
I just need to find time to setup Nix and play around with the behaviors. It’s a little hard for me to keep track of what’s going wrong from here. |
|
Please correct me if I'm wrong, @Rubikoid @kreha1 , but this is an issue if you're using uv and trying to point to a globally managed nix python installation, right? I think this is the case because I currently use a devshell for managing python/uv, and i haven't had a single installation issue. Isn't using a devshell the solution to this? I know @kreha1 you said you only got it working through hatch, but I don't think you need hatch, pretty sure you can use uv to install things alongside the python from nixpkgs. Here's the devshell {
description = "Python development environment with uv";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
flake-utils.url = "github:numtide/flake-utils";
};
outputs = { self, nixpkgs, flake-utils }:
flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs { inherit system; };
in
{
devShells.default = pkgs.mkShell {
buildInputs = with pkgs; [
python312
uv
];
# this runs when we do `nix develop .`
shellHook = ''
# Create a virtual environment if it doesn't exist
if [ ! -d ".venv" ]; then
uv venv .venv
fi
source .venv/bin/activate
echo "uv pip env ready"
'';
};
}
);
}direnv bonus copy paste: Have installed torch w/cuda support, jupyter, all sorts of things that are usually a massive pain on nixos with a modified version of this flake. Could you let me know if this solves or helps the issue, or if I'm just misunderstanding things? |
I started a couple months ago! It's a LOT. Fastest way to hit the ground running and have a uv + py nix-managed environment is:
|
|
@darinkishore This is exactly what I'm doing, but not on all systems this works. {
description = "A Nix-flake-based Python development environment";
inputs.nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
outputs = { self, nixpkgs }:
let
supportedSystems = [ "x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ];
forEachSupportedSystem = f: nixpkgs.lib.genAttrs supportedSystems (system: f rec {
pkgs = import nixpkgs { inherit system; };
supportedPython = pkgs.python310;
});
in
{
devShells = forEachSupportedSystem ({ pkgs, supportedPython }: {
default = pkgs.mkShell {
venvDir = ".venv";
packages = [
(supportedPython.withPackages(ps: [ps.uv]))
];
shellHook = ''
rm -rf .venv && echo " * Deleted previous .venv"
uv venv -p ${supportedPython} -v
. .venv/bin/activate
echo " * Active python $(which python3) resolves to $(realpath $(which python3))"
uv pip install numpy -v
echo " * All python versions:"
which -a python3 | while read path; do echo "$path is $($path -V)"; done
echo " * System info:"
lsb_release -a
'';
};
});
};
}Ubuntu 20.04 (has only system python 3.8.10 installed)Ubuntu 22.04 (system python 3.10.12 and pyenv managed versions)So maybe it because of pyenv? @Rubikoid do you also have pyenv managed python on your system? But still, without providing explicitly python version, on both system the non-nix version gets selected (pyenv > system version) I might try to containerize this flake, as to eliminate any noise once I have more time. |
Partially. I have globally managed nix python installation with some additional python packages, builded using Due to how nix work, when you use So my setup is more like this: {
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
outputs = { self, nixpkgs }:
let
supportedSystems = [ "x86_64-linux" "aarch64-darwin" "x86_64-darwin" "aarch64-linux" ];
forEachSystem = f: nixpkgs.lib.genAttrs supportedSystems (system: f {
pkgs = import nixpkgs { inherit system; };
});
in
{
devShells = forEachSystem ({ pkgs }: {
default = pkgs.mkShell (let
python = pkgs.python312;
pythonWithEnv = (python.withPackages (ps: [ ps.pydantic ]));
in {
venvDir = ".venv";
packages = [
python
pythonWithEnv
pkgs.uv
];
shellHook = ''
rm -rf .venv && echo " * Deleted previous .venv"
uv venv -p ${pythonWithEnv} -v
. .venv/bin/activate
echo " * Active python $(which python3) resolves to $(realpath $(which python3))"
uv pip install numpy -v
echo " * All python versions:"
which -a python3 | while read path; do echo "$path is $($path -V)"; done
'';
});
});
};
}With result of: So here: Also, if I replace @kreha1, no, i don't use |
|
That makes sense, thanks for explaining, as I don't have much experience with nix :). But that is still odd, that this same flake results in different behaviour on two systems - on 20.04 uv venv chooses clean python as provided, but on 22.04 it will choose dirty python, resulting in writing to read-only directory. shellHook = ''
echo "Clean python is ${supportedPython}"
which -a python3 | while read path; do echo "$path is $($path -V)"; done
rm -rf .venv && echo " * Deleted previous .venv"
uv venv -p ${supportedPython} -v
'';ubuntu 20.04ubuntu 22.04 |
|
I have this problem with all virtualenvs on NixOS, whether created with $ source .venv/bin/activate
$ pip install -U pip
error: failed to create file `/nix/store/pnavhjx4pdya95nx2apl2yxz6x46snh2-python3-3.12.4-env/.lock`
Caused by: Read-only file system (os error 30)I'm working around it by making sure I always
source .venv/bin/activate
export UV_PYTHON=$VIRTUAL_ENV/bin/pythonSee #7395 for more details. |
|
So far, I've come to the following bizzare solution for creating working envs: uv venv -p `python3 -c 'import sys; from pathlib import Path; print(str(Path(sys.path[1]).parent.parent))'`This works on |
|
Instead of setting all the environment variables in bash, it seems much better to set them in your nix flake or whatever you're using for your environment. It's more robust and precise. UV_PYTHON_PREFERENCE = "only-system";
UV_PYTHON = "${pkgs.python310}";Unfortunately, the uv-installed numpy version isn't nixos compatible ( |
Yeah, great idea, I did't mind.
I guess, this can be fixed if (or when) the "using packages from other venv in venv" thing is implemented, which is, i think, core issue here. |
|
Just to add to the plathora of solutions I usually just have this which haven't caused any issues so far EDIT: tried in Ubuntu, NixOS, and PiOS. {
inputs = { utils.url = "github:numtide/flake-utils"; };
outputs = { self, nixpkgs, utils }:
utils.lib.eachDefaultSystem (system:
let pkgs = nixpkgs.legacyPackages.${system};
in {
devShell = pkgs.mkShell {
buildInputs = with pkgs; [ uv python312 ];
shellHook = ''
export UV_PYTHON=${pkgs.python312}
uv venv
'';
};
});
} |
I would like to point out that this is normal in nix and not something that is solvable by UV. |
About issue
uv platform: MacOS Sonoma 14.3.1
uv version:
uv 0.2.13sys_prefixon uv-created venvs points to the wrong place on nix-managed python.I'm using Nix for python installation, so I have kinda two variations of python:
/nix/store/blahA-python3-3.12.3/with pure python without any packages/nix/store/blahB-python3-3.12.3-env/with partially symlinks to/nix/store/blahA-python3-3.12.3/and have packages installed.From the user view, all symlinks to python goes to
*-envthing.When creating venvs with uv they have wrong
sys_prefixset, so it is impossible to install any package in read-only python site-packages.Reproducing
uv thinks that my
*-envpython is not "system" (which it quite right) so when I specify just-p $versionto uv, it founds nothingBecause of it i have to specify direct path to python.
Creating venvs with vu
So for the first i create venv with
*-envpython:And then venv from "original" python:
Creating venvs with python venv module
And then the same thing, but with the native venv module:
Checking results
Everything run at
uv/crates/uv-toolchain:uv
uv venv created from
*-envpython:uv venv created from fresh python:
Native python
Native python venv, created from
*-envpython:Native python venv, created from fresh python:
Result
sys_prefixfor the first created venv different from others.Impossible to install package:
Maybe this is more a nix than uv issue, if so - feel free to say it and close this.
I don't find anything about patching venvs on nix, only
sitecustomisationThe text was updated successfully, but these errors were encountered: