make sagelib use setuptools instead of distutils

[ohanar]

Currently the sage library uses distutils. Switching to setuptools will give us the setup.py develop command, which bypasses the need to rebuild the sage library when modifying python files.

See also:
#29845 - PEP 517 buildapi for sage_setup

---

References:

https://stackoverflow.com/questions/3779915/why-does-python-setup-py-sdist-create-unwanted-project-egg-info-in-project-r

Depends on #29702
Depends on #29803

CC: @kini @jdemeyer @robertwb @kcrisman @mwhansen @jhpalmieri @embray @dimpase @kiwifb

Component: build

Branch/Commit: u/mkoeppe/make_sagelib_use_setuptools_instead_of_distutils @ 93bbbfe

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/13190

[ohanar]

Dependencies: #13031

[mwhansen]

comment:3

I would imagine that you get problems similar to #12659 . Have you tried this out?

[ohanar]

comment:4

Replying to @mwhansen:

I would imagine that you get problems similar to #12659 . Have you tried this out?

Do you mean the sdist issues? No, I have not tried them yet, but I wouldn't doubt that there may be issues with them. Also, as noted in deps, there is an issue with parallel instances of setuptools, so either deps has to be hacked to force sage into the serial build, or we depend on #12994 (my vote is the later, since it is cleaner).

[ohanar]

comment:5

I've decided to make #12994 a dependency -- makes it much easier to add setuptool packages to the build system.

[ohanar]

Changed dependencies from #13031 to #12994, #13031

[ohanar]

Changed dependencies from #12994, #13031 to #12994, #13031, #13197

[ohanar]

comment:7

There are some issues with using relative paths with setuptools (which we want for relocation purposes). #13197 fixes these issues.

[ohanar]

Changed dependencies from #12994, #13031, #13197 to #11874, #12994, #13031, #13197

[jdemeyer]

comment:9

I am a bit lost on the inter-relations between the tickets #12994, #13190, #13201, #13197 (and perhaps #12659). Could you please clarify the dependencies?

Also, I personally think it would be better to split this in two tickets:
A. Use setuptools
B. Use setup.py develop

[ohanar]

comment:10

Replying to @jdemeyer:

I am a bit lost on the inter-relations between the tickets #12994, #13190, #13201, #13197 (and perhaps #12659). Could you please clarify the dependencies?

Also, I personally think it would be better to split this in two tickets:
A. Use setuptools

This should only require #13031.

B. Use setup.py develop

This is more or less #12659 after switching to setuptools, and is where the other dependencies creep up (they aren't strictly necessary, but the alternative is to hack around setuptools, which is less preferable than patching it IMO).

I'll rework the patch to only do A once we have a 5.4 beta (mainly because #13031 more or less needs to be rebased with every development release).

[ohanar]

apply to root repo

[ohanar]

Attachment: trac13190_root.patch.gz

Attachment: trac13190_scripts.patch.gz

apply to scripts repo

[ohanar]

Changed dependencies from #11874, #12994, #13031, #13197 to #11874, #12994, #13031

[ohanar]

comment:11

ok, rebased and only does the switch to setuptools (doesn't switch to setup.py develop anymore). Dependencies are now accurate.

[ohanar]

Attachment: trac13190_library.patch.gz

apply to sage library

[roed314]

comment:13

Are we trying to make this happen before the git transition or afterward?

[ohanar]

comment:14

Certainly after, although I'm not currently convinced that we should proceed in this direction.

[jdemeyer]

comment:24

Replying to @jdemeyer:

There is at least one problem: setuptools does not support setup(data_files=...), see setuptools #130. It's a good reason to avoid setuptools.

I recently learned about the (almost undocumented) setuptools options --old-and-unmanageable. It forces a non-egg build which is exactly what we want. So that would make this ticket possible again.

[jdemeyer]

Changed dependencies from #20108 to none

[jdemeyer]

comment:26

Also, it seems that Sage is no longer using either data_files or package_data. So that discussion has also disappeared.

[embray]

comment:27

I forget, what is the difference between --old-and-unmanageable and --single-version-externally-managed, where I thought the latter should be sufficient...

Also I do want to use package_data for Sage (see e.g. #21732) but I don't see what the problem is.

[jdemeyer]

comment:28

Replying to @embray:

Also I do want to use package_data for Sage (see e.g. #21732) but I don't see what the problem is.

There is no problem with package_data if you want to use it in the way that it was intended to be used (which may not align with actual use cases).

There is a problem with data_files: setuptools still doesn't really support that. One of the reasons that this ticket stalled was because Sage was using data_files at the time to install Cython-related files. You suggested that it would be easy to use package_data instead (#20108). That didn't actually work, but now we are using custom code (neither data_files not package_data, see #21600) to install those files.

[jdemeyer]

comment:29

Replying to @embray:

I forget, what is the difference between --old-and-unmanageable and --single-version-externally-managed

Mainly that the latter requires a --record argument also.

[mkoeppe]

Changed author from R. Andrew Ohana to none

[mkoeppe]

comment:30

Time to revive this ancient ticket.

[mkoeppe]

Dependencies: #29702

[mkoeppe]

Branch: u/mkoeppe/make_sagelib_use_setuptools_instead_of_distutils

[mkoeppe]

Last 10 new commits:

0a61795	Trac #29345: don't force SHELL=bash any longer.
5db5318	Trac #29345: remove "break" statements from AC_SEARCH_LIBS.
e810ad1	Trac #29345: don't use sage's config.status for the lrcalc build.
93c9921	Trac #29345: replace the function that populates the CVXOPT_* variables.
0e66a0a	Trac #29345: add Dima's SPKG patches for ksh compatibility.
df3f05e	build/make/Makefile.in [SCRIPT_PACKAGE_templ]: cd into the SPKG directory; adjust spkg-install scripts
5372065	Merge branch 't/29793/script_packages_should_cd_into_the_spkg_directory' into t/29411/make_sagelib_a_script_package
c166b97	Merge branch 't/29411/make_sagelib_a_script_package' into t/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
5f33436	src/setup.py: from setuptools import setup
93bbbfe	Add src/setup.cfg

[mkoeppe]

Commit: 93bbbfe

[mkoeppe]

Changed dependencies from #29702 to #29702, #29803

[mkoeppe]

comment:37

This ticket is superseded by #29950 and can be closed.

[dimpase]

Reviewer: Dima Pasechnik