Skip backport packages for newer python #36776

orlitzky · 2023-11-26T04:57:24Z

Three of our SPKGs are backports that provide python-3.11 features:

importlib_metadata
importlib_resources
typing_extensions

As a result, they are not needed when sage's python is v3.11 or newer. We un-require them in spkg-configure.m4

orlitzky · 2023-11-26T04:58:53Z

I ran ./configure with python-3.10 and python-3.11 to ensure that the checks work but I'm entrusting the build itself to the CI.

mkoeppe · 2023-11-26T05:01:20Z

Need to change install-requires.txt for these packages then to conditionalize on Python version.

orlitzky · 2023-11-26T05:09:33Z

Hopefully that's the right syntax, I based it on https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#platform-specific-dependencies

dimpase · 2023-11-28T12:10:55Z

you need to check that you're not using Python from the system (currently needing this package). Just wrap everything in
SAGE_SPKG_DEPCHECK([python3],...)

orlitzky · 2023-11-28T12:24:03Z

The new version checks $PYTHON_MINOR, which should be set regardless of where python comes from, instead of executing python code. I also added an spkg-configure.m4 for the new exceptiongroup package we'll be getting.

tornaria · 2023-12-09T22:51:59Z

https://docs.python.org/3/library/importlib.resources.html "New in version 3.7."
https://docs.python.org/3/library/importlib.metadata.html "New in version 3.8."

tornaria · 2023-12-09T23:53:25Z

Also: nothing in sagelib uses either package, and afaiu python modules are pip-installed. Seems highly redundant to document dependencies of dependencies.

It seems to me only dependencies of packages with custom install should be kept. E.g. typing_extensions is only a dependency of setuptools_scm, so it's not really a dependency of sagemath at all, and it shouldn't be listed in requirements of sagemath-standard, etc. BTW, there are tons of packages where spkg-install.in is nothing more than sdh_pip_install ., can those be removed?

BTW six seems to be completely unnecessary (as in: not even dependencies need it anymore).

mkoeppe · 2023-12-09T23:55:36Z

Clearer dependencies will be possible with #36740, please review

mkoeppe · 2023-12-10T00:12:02Z

https://docs.python.org/3/library/importlib.resources.html "New in version 3.7."

https://docs.python.org/3/library/importlib.metadata.html "New in version 3.8."

Yes that's when the libraries were added, but newer Python versions provide new functions and, important for us, better support for PEP 420 namespace packages.

mkoeppe · 2023-12-10T00:15:08Z

afaiu python modules are pip-installed.

Normal packages (with checksums.ini) do not have access to PyPI. So we are installing all dependencies; and when installed from source, also build dependencies.

tornaria · 2023-12-10T00:16:16Z

Clearer dependencies will be possible with #36740, please review

I'm not interested in build/* at all. In fact, I want to be able to build sagelib without it. There's nothing in there that is really necessary at all but it's all tangled up.

mkoeppe · 2023-12-10T00:17:39Z

OK why talk about spkg-* anything then?

mkoeppe · 2023-12-10T00:19:51Z

src/pyproject.toml.m4 and src/setup.cfg.m4 is all that will matter to you then.

mkoeppe · 2023-12-10T00:23:09Z

It's true that currently these backport packages are not used in sagelib.
We have declared them as dependencies so that sagelib developers can rely on the new library features even though we still support Python 3.9. See https://doc.sagemath.org/html/en/developer/coding_in_python.html#python-language-standard

tornaria · 2023-12-10T00:29:54Z

OK why talk about spkg-* anything then?

I just came by this PR since src/setup.cfg lists as required some packages that I don't think belong there. The most obvious ones being six, importlib_metadata, importlib_resources, my search led here, etc.

I made a comment in good faith. My mistake, sorry. I'll retract back to my corner of the world.

mkoeppe · 2023-12-10T00:33:44Z

You are right that six should be removed from src/setup.cfg.m4. PR please?

mkoeppe · 2023-12-10T00:35:06Z

BTW, there are tons of packages where spkg-install.in is nothing more than sdh_pip_install ., can those be removed?

Yes, I think that could be a nice simplification.

tornaria · 2023-12-10T00:46:46Z

You are right that six should be removed from src/setup.cfg.m4. PR please?

Sure. What about importlib_resources, importlib_metadata, typing_extensions ? They don't seem to be required by sagemath-standard (eventually as a transitive dependency).

Also: sphinx, not really necessary for running sage, afaict.

Another thing is sage_conf: I do fine with a two-line sage_conf.py (only one line after #36792) although I'd rather have this as a configuration file inside sage package.

mkoeppe · 2023-12-10T00:47:44Z

Sure. What about importlib_resources, importlib_metadata, typing_extensions ? They don't seem to be required by sagemath-standard

As I said in #36776 (comment), these are there deliberately

tornaria · 2023-12-10T15:55:05Z

we'd put python/cpython#112215 into a doctest.

If you were to put that in a doctest, you force me to either patch it out, or stop shipping sagemath (which, for me, it would mean stop using it). I think we should move in the direction of removing these kind of tests, instead of adding them. Sage is completely usable even if you think that is a bug.

python x.y.0 are always weird a bit. So Void, unlike any other distro I know, only carries 1 version of Python? How'd people port their Python code if needed?

It's never that hard. The low level stuff is the more complicated (e.g. they seem to enjoy playing with integers lately) but it's mostly concentrated in a few packages that abstract the changes away. Other stuff usually has deprecation warnings and packages that are well maintained don't take too long to adjust.

It's ok if you prefer to move a little bit slower; I never proposed that sagemath moves to python 3.12, only that requirements.txt stops claiming that it won't work with python 3.12, because it does work. Ultimately, I don't really care about that either; except I tried to pip install sage_conf to see how it works and I was forbidden to do it (and I don't think sage_conf is going to be affected by the recursion issue, is it?).

dimpase · 2023-12-10T16:39:36Z

my point is that major Python3 upgrades introduce incompatibilities.

A distro that's serious about its users ought to keep an older version of Python around, just in case.
(e.g. if you wrote serious deeply recursive code then 3.11 -> 3.12.0 would spell big trouble for you)

mkoeppe · 2023-12-10T16:53:35Z

I never proposed that sagemath moves to python 3.12, only that requirements.txt stops claiming that it won't work with python 3.12, because it does work.

@tornaria Given your report of this, I would support making this change. PR please?

mkoeppe · 2023-12-10T17:03:59Z

Also: sphinx, not really necessary for running sage, afaict.

This one comes in through sage.misc.sagedoc / sage.misc.sphinxify

Nice to have: maybe. Required: nope.

@tornaria I know. I made that work in #32759.

But as you observed elsewhere, it provides the formatting of the docstrings in the interactive help.

That's a standard functionality, so it's a dependency of sagemath-standard.

orlitzky · 2023-12-11T12:36:47Z

System gap is usable. The system data packages I place in /usr/share/sagemath and they are usable (although SAGE_SHARE needs to be set to that, which in turns means I have to set GAP_SHARE_DIR and JMOL_DIR). sagelib (i.e. sagemath-standard) can be built using setuptools but one has to do some acrobacies. It would be nice if one could just cd sagemath-standard && python -m build --no-isolation --wheel and be done with it.

Yeah that's exactly what I mean. You shouldn't have to run your own distro to be able to build sagelib. Sage should be able to detect GAP and find those databases on its own rather than requiring you to place them exactly where it wants them and/or hand-edit conf.py.

mkoeppe · 2023-12-12T20:47:23Z

I never proposed that sagemath moves to python 3.12, only that requirements.txt stops claiming that it won't work with python 3.12, because it does work.

@tornaria Given your report of this, I would support making this change. PR please?

That's now #36869

mkoeppe · 2023-12-23T20:22:16Z

build/pkgs/importlib_metadata/spkg-configure.m4

+  # These packages are therefore not needed with >=python-3.11. Here
+  # we test for a python minor version component greater than or equal
+  # to 11, and mark this package as "not required" if we succeed.
+  AC_MSG_CHECKING([for >=python-3.11])


Why does the Python version need a separate check in spkg-configure.m4?
Does the conditionalization in install-requires.txt not do the job already? Checking whether importlib_metadata >=4.13; python_version<'3.11' is available will be trivially true for Python >= 3.11

Oh. I wrote that check before you pointed out that we needed to change install-requires.txt. It's probably redundant now but I have to get flint-3.x packaged before I can run ./configure now.

I had to change the quoting in the macro but it basically works, if you don't care that it says "will use the system foo" rather than "foo is not required" when it skips the package.

I think that's fine

Some version specifiers in install-requires.txt will contain single quotes; for example, python_version<'3.11'. (This is according to the setuptools "Platform Specific Dependencies" documentation.) To support that, we double-quote the spec string rather than single quote it.

… 3.11

…= 3.11

…3.11

mkoeppe · 2023-12-25T23:20:38Z

m4/sage_python_package_check.m4

@@ -75,10 +75,13 @@ AC_DEFUN([SAGE_PYTHON_PACKAGE_CHECK], [
        PYTHONUSERBASE="${HOME}/.sage/local"
      ])

+      dnl double-quote SAGE_PKG_VERSPEC because platform-specific
+      dnl dependencies like python_version<'3.11' will have single
+      dnl quotes in them.


Actually, you can use double quotes in install-requires.txt.

And in fact double quotes are better because then m4/sage_spkg_versions_toml.m4 will be generated correctly!

(relevant for #36951)

Ok, double quotes it is. I changed the package-check macro to normalize them (back to single quotes) so that both are supported, at least in that one macro.

For the moment, this is an easy fix for the ./bootstrap generation of sage_spkg_versions_toml.m4, which single-quotes the contents of install-requires.

Platform version constraints in install-requires can be either single- or double-quoted. To keep things simple, we'd like to assume in the macro that they are single-quoted, following the examples in the documentation. We now use sed to normalize them.

github-actions · 2023-12-26T15:32:22Z

Documentation preview for this PR (built with commit 0b08052; changes) is ready! 🎉

mkoeppe

LGTM, thanks.

Three of our SPKGs are backports that provide python-3.11 features: * importlib_metadata * importlib_resources * typing_extensions As a result, they are not needed when sage's python is v3.11 or newer. We un-require them in `spkg-configure.m4` URL: sagemath#36776 Reported by: Michael Orlitzky Reviewer(s): Matthias Köppe, Michael Orlitzky

After sagemath#36129 and sagemath#36776 we are still left with a bunch of external (non- Sage) Python packages which need `spkg-configure.m4` and distros info. Here we add these. ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - sagemath#36129 - sagemath#36776  URL: sagemath#36777 Reported by: Dima Pasechnik Reviewer(s):

…kages       - sagemath#35203 added these packages as unconditional dependencies - prompted by a sporadic use of typing_extensions in the Sage library (see sagemath#34831) - motivated further by the demand to immediately drop support for Python 3.8 so that newer typing features can be used in the Sage library (sagemath#35404) - sagemath#36776 reduced the packages to conditional dependencies Here we improve the documentation in the section "Language standard" of the developer's guide so that it aligns with how the conditional dependencies are declared. Based on changes split out from sagemath#37399. ### 📝 Checklist  - [ ] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies    URL: sagemath#37654 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

@culler

…gemath#36776       As reported in https://groups.google.com/g/sage-devel/c/8ZsBcHGmn6s @culler ### 📝 Checklist  - [x] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies    URL: sagemath#37959 Reported by: Matthias Köppe Reviewer(s):

orlitzky requested review from dimpase and mkoeppe November 26, 2023 04:57

dimpase mentioned this pull request Nov 26, 2023

spkg-configure.m4 for most external Python pkgs #36777

Merged

5 tasks

dimpase added c: build: configure s: needs review labels Nov 27, 2023

orlitzky mentioned this pull request Nov 27, 2023

Notebook 7, ipykernel 6.27, ipython 8.17 #36129

Merged

5 tasks

dimpase added s: needs work and removed s: needs review labels Nov 28, 2023

orlitzky force-pushed the skip-backport-packages branch from 82c596c to d271eb0 Compare November 28, 2023 12:21

orlitzky added s: needs review and removed s: needs work labels Nov 28, 2023

orlitzky mentioned this pull request Dec 11, 2023

check ipython for sphinxext, extend python_package_check macro #36848

Open

mkoeppe reviewed Dec 23, 2023

View reviewed changes

orlitzky added 4 commits December 24, 2023 20:40

build/pkgs/importlib_metadata/install-requires.txt: skip if python >=…

6181dec

… 3.11

build/pkgs/importlib_resources/install-requires.txt: skip if python >…

9e5df3c

…= 3.11

build/pkgs/typing_extensions/install-requires.txt: skip if python >= …

c192fb5

…3.11

orlitzky force-pushed the skip-backport-packages branch from d271eb0 to c192fb5 Compare December 25, 2023 01:40

mkoeppe reviewed Dec 25, 2023

View reviewed changes

orlitzky added 2 commits December 26, 2023 06:55

build/pkgs: double-quote platform dependencies in install-requires.txt

4c3f7ac

For the moment, this is an easy fix for the ./bootstrap generation of sage_spkg_versions_toml.m4, which single-quotes the contents of install-requires.

mkoeppe approved these changes Dec 26, 2023

View reviewed changes

mkoeppe added s: positive review and removed s: needs review labels Dec 26, 2023

vbraun merged commit f7ae5bf into sagemath:develop Jan 14, 2024
27 checks passed

mkoeppe added this to the sage-10.3 milestone Jan 14, 2024

github-actions bot removed the s: positive review label Jan 14, 2024

orlitzky deleted the skip-backport-packages branch January 24, 2024 14:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip backport packages for newer python #36776

Skip backport packages for newer python #36776

orlitzky commented Nov 26, 2023

orlitzky commented Nov 26, 2023

mkoeppe commented Nov 26, 2023

orlitzky commented Nov 26, 2023

dimpase commented Nov 28, 2023

orlitzky commented Nov 28, 2023

tornaria commented Dec 9, 2023

tornaria commented Dec 9, 2023

mkoeppe commented Dec 9, 2023

mkoeppe commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

tornaria commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

tornaria commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

tornaria commented Dec 10, 2023

mkoeppe commented Dec 10, 2023 •

edited

Loading

tornaria commented Dec 10, 2023

dimpase commented Dec 10, 2023 •

edited

Loading

mkoeppe commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

orlitzky commented Dec 11, 2023

mkoeppe commented Dec 12, 2023

mkoeppe Dec 23, 2023

orlitzky Dec 24, 2023

orlitzky Dec 25, 2023

mkoeppe Dec 25, 2023

mkoeppe Dec 25, 2023

mkoeppe Dec 25, 2023

orlitzky Dec 26, 2023

github-actions bot commented Dec 26, 2023

mkoeppe left a comment

Skip backport packages for newer python #36776

Skip backport packages for newer python #36776

Conversation

orlitzky commented Nov 26, 2023

orlitzky commented Nov 26, 2023

mkoeppe commented Nov 26, 2023

orlitzky commented Nov 26, 2023

dimpase commented Nov 28, 2023

orlitzky commented Nov 28, 2023

tornaria commented Dec 9, 2023

tornaria commented Dec 9, 2023

mkoeppe commented Dec 9, 2023

mkoeppe commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

tornaria commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

tornaria commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

tornaria commented Dec 10, 2023

mkoeppe commented Dec 10, 2023 • edited Loading

tornaria commented Dec 10, 2023

dimpase commented Dec 10, 2023 • edited Loading

mkoeppe commented Dec 10, 2023

mkoeppe commented Dec 10, 2023

orlitzky commented Dec 11, 2023

mkoeppe commented Dec 12, 2023

mkoeppe Dec 23, 2023

Choose a reason for hiding this comment

orlitzky Dec 24, 2023

Choose a reason for hiding this comment

orlitzky Dec 25, 2023

Choose a reason for hiding this comment

mkoeppe Dec 25, 2023

Choose a reason for hiding this comment

mkoeppe Dec 25, 2023

Choose a reason for hiding this comment

mkoeppe Dec 25, 2023

Choose a reason for hiding this comment

orlitzky Dec 26, 2023

Choose a reason for hiding this comment

github-actions bot commented Dec 26, 2023

mkoeppe left a comment

Choose a reason for hiding this comment

mkoeppe commented Dec 10, 2023 •

edited

Loading

dimpase commented Dec 10, 2023 •

edited

Loading