Be more purposeful in the Python version in pyvenv.cfg

[brettcannon]

In pyvenv.cfg, there's a version field that records ".".join(sys.version_info[:3]). The interesting thing about this is it isn't loose enough to reflect the version accurately if the Python interpreter is updated in-place (e.g. going from Python 3.11.1 to 3.11.2). But it's also not accurate enough to reflect the exact Python version (e.g. Python 3.12.0a6 is recorded as 3.12.0).

Because of this I'm not sure what the number is meant to represent. If it's meant to record what the version was when the virtual environment was created then we should either change the field to record the release level or add another field that records that. But if it's just meant to record what the version might be, then it's a bit too specific.

[brettcannon]

/cc @vsajip

[vsajip]

Perhaps just add the release level in another field, to avoid breakage in case someone's code is parsing the current version field and expecting it to always have 3 components.

[brettcannon]

One interesting wrinkle is the version number is incomplete at creation time, but it can also become inaccurate even today if you upgrade your Python version in-place since the symlinks are only major.minor specific (e.g. 3.11.2 to 3.11.3). As such, this really acts as a recording of what was used when the virtual environment was created.

Regardless, making the release level a separate key/field is fine with me!

[FFY00]

Regardless, making the release level a separate key/field is fine with me!

Do you mean adding a new key that reflects the version of the current interpreter?

If so, do you have an use-case for it?

[vsajip]

Do you mean adding a new key that reflects the version of the current interpreter?

That's how I read it. For it to be useful, it needs IMO to reflect the current interpreter - i.e. as well as setting it up on creation, it should also be updated if the Python in the venv is upgraded.

[FFY00]

I think the issue with that is that we would have to update it then, which I think can only ever happen when the interpreter is run, meaning there's always the possibility the field is out of date.

That's why I asked for an use-case. I think that's the key thing we should be looking at here.

[vsajip]

meaning there's always the possibility the field is out of date

There's no reason why the code that does the --upgrade of the venv can't invoke the updated interpreter to get the release field and then update pyvenv.cfg with it. So the field needn't be out of date.

That's why I asked for an use-case. I think that's the key thing we should be looking at here.

Agreed.

[FFY00]

There's no reason why the code that does the --upgrade of the venv can't invoke the updated interpreter to get the release field and then update pyvenv.cfg with it. So the field needn't be out of date.

But this already happens.

$ python3.9 -m venv test
$ cat test/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.9.9
$ python3.10 -m venv --upgrade test
$ cat test/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.10.9

I think the issue here is when the interpreter gets upgraded, but you don't run venv --upgrade.

side-note: We probably should be raising a warning when we are in a virtual environment created for a different interpreter version, right?

[carljm]

You can safely "upgrade" a virtualenv without using venv --upgrade, by just changing the underlying Python in-place, as long as it's only a minor version upgrade (e.g. 3.11.2 to 3.11.3), since all the directory names will still be the same. I think that's why Brett was initially suggesting that if we want the version information to always be correct, it needs to be less precise, and instead only specify the range of compatible versions for this venv (i.e. record only 3.11). If we record anything more than that, it will always be possibly out of date.

[vsajip]

But this already happens.

Yes, but not for any new "release" field that would be added.

I think the issue here is when the interpreter gets upgraded, but you don't run venv --upgrade.

You can safely "upgrade" a virtualenv without using venv --upgrade, by just changing the underlying Python in-place, as long as it's only a minor version upgrade (e.g. 3.11.2 to 3.11.3), since all the symlink target directories will still be the same.

Why wouldn't you use venv --upgrade to upgrade the interpreter? That's the purpose for which it was provided, after all. Is the alternative invocation somehow simpler? Does it do something differently to venv --upgrade?

[carljm]

Why wouldn't you use venv --upgrade to upgrade the interpreter? That's the purpose for which it was provided, after all. Is the alternative invocation somehow simpler? Does it do something differently to venv --upgrade?

I'm not sure what you mean by the "alternative invocation." On Linux, by default, the interpreter is symlinked. So a minor version "upgrade" of a venv can happen silently and automatically just by virtue of the underlying Python installation being upgraded in-place (which is generally considered safe, and distros do it). There need not be any invocation of anything specific to the venv at all.

[brettcannon]

Do you mean adding a new key that reflects the version of the current interpreter?

Either that or another key that only records the release level (e.g. "final", "a6", etc.).

If so, do you have an use-case for it?

If it was completely accurate then I have a use-case for the Python Launcher for Unix (being able to know what the full version of Python is without having to actually run the interpreter). But otherwise the value is a bit misleading, so I figured maybe acting as a registration of what version was used to create the environment would help make it more useful.

I think the issue here is when the interpreter gets upgraded, but you don't run venv --upgrade.

Yep, at least when you're not on Windows.

You can safely "upgrade" a virtualenv without using venv --upgrade, by just changing the underlying Python in-place, as long as it's only a minor version upgrade (e.g. 3.11.2 to 3.11.3), since all the directory names will still be the same. I think that's why Brett was initially suggesting that if we want the version information to always be correct, it needs to be less precise, and instead only specify the range of compatible versions for this venv (i.e. record only 3.11).

Which is what also helped me realize the current version is not accurate either for when the virtual environment was created or what is currently running.

For it to be useful, it needs IMO to reflect the current interpreter - i.e. as well as setting it up on creation, it should also be updated if the Python in the venv is upgraded.

Updating the file as appropriate is not necessarily the worst thing since we already have to read pyvenv.cfg to get the system-site-packages value anyway.

[vsajip]

On Linux, by default, the interpreter is symlinked. So a minor version "upgrade" of a venv can happen silently and automatically just by virtue of the underlying Python installation being upgraded in-place (which is generally considered safe, and distros do it)

Ah, right, I didn't think of that.

[FFY00]

If it was completely accurate then I have a use-case for the Python Launcher for Unix (being able to know what the full version of Python is without having to actually run the interpreter). But otherwise the value is a bit misleading, so I figured maybe acting as a registration of what version was used to create the environment would help make it more useful.

Gotcha. And running the interpreter is not possible here?

I think the key issue is that we simply cannot guarantee the accuracy of the version logged in the file, as the underlying interpreter is updated independently (on Unix at least).

A possible solution would be installing a file that logs the interpreter version as part of the interpreter installation, that way it will always be up-to-date. The Python Launcher for Unix would then be able to simply read that file, allowing it to get the Python version without having to run the interpreter.

Updating the file as appropriate is not necessarily the worst thing since we already have to read pyvenv.cfg to get the system-site-packages value anyway.

But between the interpreter being updated and it being run in the virtual environment for the first time afterwards, won't the value be outdated?

[FFY00]

The Python Launcher for Unix would then be able to simply read that file, allowing it to get the Python version without having to run the interpreter.

To clarify here, the proposal would be for it to read pyvenv.cfg, build the version file path based on base-prefix, and then read that file.

[brettcannon]

running the interpreter is not possible here?

No, it's entirely possible, just way more expensive than simply reading a file.

A possible solution would be installing a file that logs the interpreter version as part of the interpreter installation

I've considered that, but I'm not ready to propose that and go through the discussion.

But between the interpreter being updated and it being run in the virtual environment for the first time afterwards, won't the value be outdated?

True, so there's always the risk of it being outdated.

[brettcannon]

So, it seems the version in that file will always run the risk of being inaccurate, even if it's only for the length of time from new install to first run.

And I know @vsajip didn't want to change it to avoid breaking anyone assuming there's always a micro number. So maybe I should just close this and consider the field a quirky thing that somewhat records the version at environment creation time?

[FFY00]

I've considered that, but I'm not ready to propose that and go through the discussion.

Well, good news then. I am planning to propose something like that, in an effort to bring first-party cross compilation support to Python. The file would not only contain the version, but rather all information required to describe the binary details of other interpreters, including cross compilation. Then the sysconfig API could either load the data from the local interpreter, or an a description file from another one, making cross compilation much easier. I also got some time from $dayjob to dedicate to this, so I am very excited to finally be able to tackle all these issues, which have historically been a PITA 😊

I think in the meantime, looking at patchlevel.h should be an acceptable solution, no? Especially since it will work for older version too.

[brettcannon]

The file would not only contain the version, but rather all information required to describe the binary details of other interpreters, including cross compilation.

Interesting! I'm specifically interested in all of this for brettcannon/python-launcher#168 to make getting interpreter details as cheaply as possible while still being accurate.