Drop Python 3.8

[jakirkham]

Recently we added the Python 3.12 migration ( #4914 ). Typically when we have added a new Python version we have bumped the minimum Python version supported as well. Currently Python 3.8 is the minimum. So am proposing we drop Python 3.8 moving our minimum Python version to 3.9. Thoughts?

cc @conda-forge/core

[xhochy]

We had a bit of discussion on this topic in some place (maybe the private core chat?). The gist of it (and if we never find a reference, happy to take this simply as the expression of solely my opinion) was to keep Python 3.8 as part of the CI matrix as long as we don't see issues with the load.

In the end, this comes down to the decision on whether we, as conda-forge, want to follow CPython's schedule for support or NumPy's NEP29 which is a bit stricter in dropping old versions.

[h-vetinari]

This was discussed in one of the recent core calls, and should be in the minutes. While we're free to adapt our choice as necessary, there was a clear desire to match CPython's support window, as long as it doesn't blow out our resources.

[jakirkham]

Well offline the other day there was also a discussion about the CI queue getting larger. So think it is worth raising what resources we want to commit to this

Put another way every decision where we commit resources to one item means another item may not get them. Not to say it is exactly zero sum (we can ask for more resources from those who make asks of us), but there is some trade off that we need to be considering and the impact it has on what we are able to do

[h-vetinari]

Yes, we're all aware of the resource constraints, but not nearly all of the growth in our consumption is due to another python version (e.g. we dropped pypy3.8, so the full matrix is still 6 builds); there's also an overall growth of conda-forge in terms of packages, and some stacks just have 2-3+h builds across multiple feedstocks, so if those get touched by a migration it's a big hit (and the combination of py312 + boost + protobuf is quite the whammy migration-wise).

@jezdez mentioned that we should be able to get the pool from azure pipelines to be increased, which would obviate that discussion, but in any case, dropping 3.8 was considered having a broad impact on the wider ecosystem, so we decided to try to keep supporting it (with a caveat that it's best effort and we can change our minds of course), and see how things develop.

[h-vetinari]

I updated #5790 to build against 2.0.0rc1 directly; it looks like it still needs a smithy fix, see conda-forge/conda-smithy#1911

Sorry, wrong thread 🙈

[hmaarrfk]

In the end, this comes down to the decision on whether we, as conda-forge, want to follow CPython's schedule for support or NumPy's NEP29 which is a bit stricter in dropping old versions.

One way that I'm reconciling with this is:

1. CPython's schedule is for the builds should be followed for the builds of the "CPython" package itself. It shouldn't add burden to other downstream maintainers. CPython may choose to release security patches when it finds it necessary to do so. The maintainers of the python package at conda-forge should be able to make the best decision there.
2. NEP29 (or SPEC0) is specifically about how maintainers should feel about building new packages against old python versions. As such, it would encourage dropping python 3.8 from the build matrix.

One way to reconcile this is to:

1. Drop python 3.8 from the default build matrix. Note that users of Python 3.8 will be able to continue to use any packages that are currently built against Python 3.8.
2. Write a "python migrator" but "freeze it immediately".
3. Those that wish to still build against 3.8 would be able to pull in the migrator file to their feedstock, and build for python 3.8 as an "opt in".

[xhochy]

I feel like the hurdles we experience in #5790 support dropping Python 3.8.

Personally, I feel like we would help ourselves a lot by adopting a certain policy here. This would make it easier for all consumers what is expected to be supported. Given that a fair share of people involved are interested in scientific Python packages, I have the preference to adopt SPEC0.

[isuruf]

I don't think we should adopt SPEC0 as conda-forge has grown out of a being a scientific python community to a general python community. Dropping a python version because of a need to release an ABI breaking downstream package (albeit a very important package) as a pre-release seems like the wrong thing to do. We should have this discussion after numpy 2.0 is released to not have that over our heads so that we decide on what's best for the community.

[hmaarrfk]

I don't think we should adopt SPEC0 as conda-forge has grown out of a being a scientific python community to a general python community.

Whether it is SPEC0 or not I would prefer to move to a strict "time bound" policy for dropping versions.

Its very "tiring" to have these discussions about backward compatibility. The reason I support NEP29/SPEC0 is that somebody took the time to address the concerns, and draw a line somewhere. (FWIW, I'm asking Numpy to adhere to their own suggestions and drop python 3.9 https://mail.python.org/archives/list/numpy-discussion@python.org/thread/AHTATJKGUEOILBNUI5IGGZPXJ5FXIRAU/ )

Dropping a python version because of a need to release an ABI breaking downstream package (albeit a very important package) as a pre-release seems like the wrong thing to do.

Preparing for Numpy 2.0 has been chosen by many of the core developers as a project that is of importance to the community.
We must weigh the maintenance time of the "3.8" ecosystem, to that of the "future numpy ecosystem".

I don't think there is ever "a right time" to have these dicussions, the pending numpy 2.0 release is just a good forcing function for this discussion.

[hmaarrfk]

Is there really still appetite to build new packages for python 3.8 or 3.9?

My feeling is that:

1. Continue to release patch release of python 3.8 and 3.9 as they come in.
2. Stop rerendering recipes as python 3.8.
3. Encourage maintainers to write python >=3.10 in their noarch packages.

This way we:

1. Continue to release security releases for python 3.8, 3.9 for as long as python supports it.
2. Continue to allow people to use python 3.8 or 3.9 should they want to in new environments with the same old packages they always had.
3. Free up our own time and mental space.

Again, we wouldn't stop buliding new patch versions of the python package itself, just stop reduce our downstream matrix to:

• Today: 3.10, 3.11, 3.12
• And in accordance with spec0, 3.11, 3.12, 3.13 later in the year.

"3 versions" of python seems plenty for me.

[isuruf]

Yes, let's keep building downstreams for 3.9. As long as python 3.9 is supported, some packages will keep supporting 3.9 and conda-forge packages are used in their CI configurations and having new packages is important for development. (For eg: pytorch)

[dopplershift]

Encourage maintainers to write python >=3.10 in their noarch packages.

Strong -1 on this. Make the recipe reflect the metadata that's encoded by upstream. I can see an argument for the unclear "Python>3", but otherwise we're just breaking configurations that would otherwise work and create confusion against configurations that upstream intends to support. And in the case of noarch, it comes with no cost to us.

[jakirkham]

🤷‍♂️ At this point we are 6 weeks from Python 3.8 EOL, might as well push to the end. Figure at this point we would drop it whenever we add Python 3.13, which is likely not far away either

Should add I'm saying this as someone who has advocated for dropping Python 3.8 a lot earlier. It just seems like this discussion will resolve itself very soon

For other Python versions and policy discussions, they were not the focus of this issue. So don't think we should be having them here. We should start new issue(s) scoped for those topics. These conversations are too important to not have in places where they are easily searchable and discoverable (discussing them under a Python 3.8 issue makes them a bit harder to find and creates avoidable confusion)

[h-vetinari]

Encourage maintainers to write python >=3.10 in their noarch packages.

Strong -1 on this. Make the recipe reflect the metadata that's encoded by upstream. I can see an argument for the unclear "Python>3", but otherwise we're just breaking configurations that would otherwise work and create confusion against configurations that upstream intends to support.

The flipside of this is that we have a lot of python >=3.6 and 3.7 lower bounds in noarch recipes, which are broken already in many cases because upstream started using newer python features and the feedstocks never change the bound because CI remains green.

I think it would make sense to set the lower bound in noarch recipes to our oldest supported python version. That's realistically the only thing we can test and support. Upstream may support more, but conda-forge doesn't have to - when we drop python 3.8, that IMO should include noarch packages.

Perhaps not as an obligation, but at least as a linting rule.

Figure at this point we would drop it whenever we add Python 3.13, which is likely not far away either

👍 That's the plan AFAIU it

[jakirkham]

The flipside of this is that we have a lot of python >=3.6 and 3.7 lower bounds in noarch recipes, which are broken already in many cases because upstream started using newer python features and the feedstocks never change the bound because CI remains green.

I think it would make sense to set the lower bound in noarch recipes to our oldest supported python version. That's realistically the only thing we can test and support. Upstream may support more, but conda-forge doesn't have to - when we drop python 3.8, that IMO should include noarch packages.

Perhaps not as an obligation, but at least as a linting rule.

Could we please move this discussion to issue ( conda-forge/conda-forge.github.io#2210 )?

Think this is a good conversation to have and there's already some thoughts in that issue

[h-vetinari]

@jezdez mentioned that we should be able to get the pool from azure pipelines to be increased, which would obviate that discussion, [...]

Reading this old thread again, I wonder if we can prioritize the interaction with MSFT / Azure Pipeline again. Conda-forge has grown a lot, but our limit of 200(?) agents hasn't changed in a long time.

[isuruf]

Opened a PR to drop in #6344

[hmaarrfk]

Did i miss a conversation? I thought we were on track to keeping support until Oct 2024.

[isuruf]

Yes, but it's only 5 weeks away and I don't think we would lose anything by dropping building packages a month early. Python 3.13 migration is about to start and it will be a waste of resources to keep building new packages in the migration (because the migration will have a build number bump) just to drop it a month later. Python 3.13 is orthogonal to 3.8 discussion, but in terms of practicality, I think it's okay to drop it now instead of waiting another month. What do you think?

[h-vetinari]

I agree with Isuru here. My understanding of the discussion w.r.t. PEP 602 (one example from the core call minutes is here) was that we'd do a best effort to keep building 3.8 towards its EOL, even if that means one more entry in the CI matrix everywhere.

But we've ~always used new python migrations as a reason to get rid of a version that's (close to) EOL, and I think we shouldn't change that, because it makes sense on a lot of levels (c.f. Isuru's comment above). Let's keep the CI matrix from blowing up, the migration will be enough work as-is IMO.

[hmaarrfk]

I understand if we want to say "until the appropriate RC is released"

[isuruf]

That's a good idea. We can say

python 3.X is supported until EOL or until python 3.(X+5).rc1 is released whichever comes first.

[jaimergp]

python 3.X is supported until EOL or until python 3.(X+5).rc1 is released whichever comes first.

Should we encode this as a CFEP?

[jakirkham]

We could. Though I think it is important to remember that this is a volunteer effort with donated resources. IOW we should not tie ourselves to an anchor that makes it hard for us to adapt to changing constraints

[jaimergp]

At the very least document it so we have a reference we can use next time this conversation comes up.

[jakirkham]

Yes this is why I was suggesting we open a dedicated issue on this topic to discuss as opposed to having it in the annual Python bringup/drop issues & PRs

Simply having the conversation in one easy to find place would be quite helpful for our future selves