Enable VPATH builds (several independent build trees connected to one source tree) #21469

mkoeppe · 2016-09-11T03:59:17Z

Executive summary Currently, --prefix selects the install tree (SAGE_LOCAL) of sage-the-distribution. But often we want several builds out of (almost) the same source tree.

The vpath mechanism proposed here selects the build tree (SAGE_ROOT) of sage-the-distribution. This includes the configuration and build artefacts such as src/build/....

VPATH builds are a powerful feature of autotools (and other) build systems. This feature allows the developer to build a package in a separate (initially empty) build directory tree. The source directory tree is only read from (and could even be mounted read-only) and is therefore always clean.

This is a powerful feature for the developer because from the same source tree many different configurations can be built and tested, without having to go through "make distclean" and reconfiguration. The source tree can also be shared, for example using a networked file system between different hosts, running different architectures. Another modern use case involves VMs. For example, Docker allows to mount the source directory from the host in the VM (see #21474).

For Sage, "different configurations" could mean different architectures (via VMs), different sets of installed packages, Py2 vs Py3, etc.

This is how it is used.

 cd SAGE_ROOT
 autoreconf

and then

 mkdir BUILDDIR
 cd BUILDDIR
 SRCDIR/configure --srcdir=SRCDIR
 make

The present patch implements this. BUILDDIR from the above example is reflected in the old environment variable $SAGE_ROOT; this is now to be distinguished from SRCDIR, which is reflected in the new environment variable $SAGE_VPATH. $SAGE_LOCAL defaults to BUILDDIR/local, but can of course be changed using configure --prefix.

The present patch changes the various places where $SAGE_ROOT is used.
When "./configure" detects a VPATH build, it installs patched copies of sage, Makefile and build/make/install in BUILDDIR.

With the current set of patches, the compilation goes through and gives a working sage, except for the docbuild.

CC: @sagetrac-felixs @jdemeyer @kiwifb @embray @nexttime @vbraun @dimpase @fchapoton @jhpalmieri

Component: build: configure

Work Issues: merge conflicts

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/vpath_build @ 05f3a91

Issue created by migration from https://trac.sagemath.org/ticket/21469

The text was updated successfully, but these errors were encountered:

mkoeppe · 2016-09-11T04:23:20Z

Branch: u/mkoeppe/vpath_build

mkoeppe · 2016-09-11T18:12:37Z

Commit: ac27cdc

mkoeppe · 2016-09-11T18:12:37Z

New commits:

`ac27cdc`	`configure.ac: VPATH fixes`

sagetrac-git · 2016-09-11T21:36:12Z

Branch pushed to git repo; I updated commit sha1. New commits:

`5698f8e`	`Distinguish SAGE_ROOT and SAGE_SRC_ROOT for VPATH build`
`a98cfa2`	`Use AC_CONFIG_COMMANDS`
`e9b1c0f`	`Symlink build/pkgs (temporary solution)`

sagetrac-git · 2016-09-11T21:36:12Z

Changed commit from ac27cdc to e9b1c0f

sagetrac-git · 2016-09-11T22:55:07Z

Branch pushed to git repo; I updated commit sha1. New commits:

`e15ada6`	`Define SAGE_SRC_ROOT for VPATH build`

sagetrac-git · 2016-09-11T22:55:07Z

Changed commit from e9b1c0f to e15ada6

sagetrac-git · 2016-09-12T00:04:23Z

Changed commit from e15ada6 to 9323c01

sagetrac-git · 2016-09-12T00:04:23Z

Branch pushed to git repo; I updated commit sha1. New commits:

`9323c01`	`Use SAGE_SRC_ROOT in package scripts`

sagetrac-git · 2016-09-12T00:41:35Z

Branch pushed to git repo; I updated commit sha1. New commits:

`7a3913a`	`More SAGE_SRC_ROOT`

sagetrac-git · 2016-09-12T00:41:35Z

Changed commit from 9323c01 to 7a3913a

sagetrac-git · 2016-09-12T05:17:29Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4cc3954`	`More SAGE_SRC_ROOT`

sagetrac-git · 2016-09-12T05:17:29Z

Changed commit from 7a3913a to 4cc3954

mkoeppe · 2016-09-12T05:32:23Z

comment:9

The current patch makes some high-level preparations for a VPATH build, in particular the package system.

"make" runs through until it gets to sagelib, which needs some more VPATH work (the current patch tries to get away by just making a symlink from $SAGE_SRC_ROOT/src to $SAGE_ROOT/src, which of course fails).

I've set this to 'needs review' in the hope of receiving some feedback regarding the approach of this patch.

kiwifb · 2016-09-12T05:49:26Z

comment:10

I am not sure what the objective of this ticket is. Considering the nature of sage's build system I am not sure what VPATH brings me.

This is most evident in the latest commit. As a distro maintainer of sage there are already build time things in env.py that are not needed at runtime and in a way are superfluous. You have just added one. Here the problem that I am forced to consider, is that going by the separation rules that I try to follow, there should be stuff in sage/env.py and some in a new file sage_setup/build_env.py but that's marginal to what you are doing.

Other people may have a different opinion but there is no value in this for me so far. Possibly more work in fact.

mkoeppe · 2016-09-12T05:52:50Z

comment:11

Replying to @kiwifb:

I am not sure what the objective of this ticket is. Considering the nature of sage's build system I am not sure what VPATH brings me.

A VPATH build makes it easy to build from the same source in many different configurations. Concretely, I've started to run some VMs (with Linux 64 bit, 32 bit) via Docker, which mount the Sage source directory from the host.

sagetrac-git · 2016-09-12T06:17:00Z

Branch pushed to git repo; I updated commit sha1. New commits:

`7bb8718`	`Make sage script in build dir executable`

sagetrac-git · 2016-09-12T06:17:00Z

Changed commit from 4cc3954 to 7bb8718

mkoeppe · 2016-09-12T06:21:15Z

comment:13

Replying to @kiwifb:

As a distro maintainer of sage there are already build time things in env.py that are not needed at runtime and in a way are superfluous. You have just added one. Here the problem that I am forced to consider, is that going by the separation rules that I try to follow, there should be stuff in sage/env.py and some in a new file sage_setup/build_env.py but that's marginal to what you are doing.

Yes, this ticket does not address any of the issues that would arise in distribution packaging.

Instead it adds a standard feature expected from autotools build system, for those who use sage as a distribution.

kiwifb · 2016-09-12T07:43:46Z

comment:14

I should apologize for being so negative. Currently configure is run by the initial call to make, it would be nice to get to the stage where we run configure first :)

Seriously, yes VPATH as lots of advantages, since most people run sage from where they build it, being able to do several builds from the same source without make distclean or git clone is a good thing.

I guess, it mostly started my brain on the fact that the separation build and runtime is not clean in vanilla sage. Your VPATH work may help to clarify it in some areas. Let's try to minimise where it would mix things further.

mkoeppe · 2016-09-12T07:51:50Z

comment:15

Replying to @kiwifb:

I should apologize for being so negative.

No worries.

Currently configure is run by the initial call to make, it would be nice to get to the stage where we run configure first :)

That already works with this patch.

Seriously, yes VPATH as lots of advantages, since most people run sage from where they build it, being able to do several builds from the same source without make distclean or git clone is a good thing.

I guess, it mostly started my brain on the fact that the separation build and runtime is not clean in vanilla sage. Your VPATH work may help to clarify it in some areas. Let's try to minimise where it would mix things further.

Sounds good. Yes, I'm hoping too that it helps clarify some things as a side-effect.

jdemeyer · 2016-09-12T11:57:41Z

comment:16

Replying to @mkoeppe:

Instead it adds a standard feature expected from autotools build system, for those who use sage as a distribution.

I sort of agree with François.

Part of the problem is that Sage is a distribution (a collection of packages) as well as a Python library. The fact this this is currently mixed up in the sources (the sources of the distribution-part and the library-part are in one repo) makes it more difficult to understand what this ticket is about.

For the distribution part, there really is no such thing as a build directory. There is a a temporary directory in local/var/tmp/sage/build but I would not consider that a build directory in the usual sense.

For the library part, a VPATH build is meaningful. Although, I think that VPATH builds are unusual in the Python world. Luckily, the default build directories are platform-dependendent. Anyway, it seems like this ticket isn't about that.

jdemeyer · 2016-09-12T12:00:22Z

comment:17

I think the ticket description should have a lot more information about

which problem that this ticket is trying to solve (the why).
what the branch actually does (the what).

Note that this are 2 independent things (which are unfortunately often confused).

mkoeppe · 2017-09-08T19:33:38Z

comment:101

Replying to @dimpase:

I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).

It would be nice if the Sage version number that it displays at startup would show the git commit from which it is built.
But this is entirely unrelated to the present ticket.

mkoeppe · 2017-09-08T19:36:00Z

comment:102

Replying to @embray:

When making a VPATH build you typically specify the path to the build tree yourself, so you could do something like VPATH=builds/$(git symbolic-ref --short HEAD).

Just a quick note. When one says "VPATH", it refers to the source directory, not to the build directory. https://www.gnu.org/software/make/manual/html_node/General-Search.html

mkoeppe · 2017-09-08T19:37:06Z

comment:103

Replying to @dimpase:

Would the old way to build still work unamended, or BUILDDIR will need to be created separately?

Yes, with this patch one can still do non-VPATH builds, without any change.

dimpase · 2017-09-08T20:12:29Z

comment:104

Replying to @mkoeppe:

Replying to @dimpase:

I wonder how this can be syncronised with git branches. Namely, it would be nice if a build is tagged by the source commit, so that it is known at which source state the build was done (I gather there could even be a git feature for this...).

It would be nice if the Sage version number that it displays at startup would show the git commit from which it is built.
But this is entirely unrelated to the present ticket.

I suppose I just don't understand what happens to a VPATH build after the source tree it is built from is changed. Will it become broken, or not?

jhpalmieri · 2017-09-28T18:47:12Z

comment:105

Are there any immediate issues left here beside comment:93?

saraedum · 2018-01-14T01:57:36Z

Work Issues: merge conflicts

saraedum · 2020-02-28T00:04:39Z

comment:107

I actually could really use this feature right now :)

mkoeppe, should I try to resolve conflicts here? Or do you have time to do so, so I can review afterwards and maybe fix comment:93?

mkoeppe · 2020-02-28T00:14:32Z

comment:108

I hope to get back to it within the next month, but not right now. It will likely interact with some of my build-related tickets that are waiting for review...

If you need this earlier, please feel free to work on it.

I don't think comment 93 should hold up this ticket. Improvements could always be done in a follow-up ticket.

mkoeppe · 2021-02-13T20:51:01Z

comment:111

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

mkoeppe · 2021-07-19T00:44:56Z

comment:112

Setting a new milestone for this ticket based on a cursory review.

mkoeppe added this to the sage-7.4 milestone Sep 11, 2016

mkoeppe added c: build: configure labels Sep 11, 2016

mkoeppe added the s: needs review label Sep 11, 2016

jdemeyer added s: needs info and removed s: needs review labels Sep 12, 2016

saraedum added s: needs work and removed s: needs review labels Jan 14, 2018

mkoeppe modified the milestones: sage-8.2, sage-9.2 Apr 9, 2020

mkoeppe modified the milestones: sage-9.2, sage-9.3 Aug 13, 2020

mkoeppe modified the milestones: sage-9.3, sage-9.4 Feb 13, 2021

mkoeppe modified the milestones: sage-9.4, sage-9.5 Jul 19, 2021

mkoeppe modified the milestones: sage-9.5, sage-9.6 Dec 14, 2021

mkoeppe modified the milestones: sage-9.6, sage-9.7 Mar 5, 2022

mkoeppe modified the milestones: sage-9.7, sage-9.8 Aug 31, 2022

mkoeppe modified the milestones: sage-9.8, sage-9.9 Jan 7, 2023

jhpalmieri mentioned this issue Dec 8, 2017

sage.env: _add_variable_or_fallback depends on the order of a dict #23758

Closed

mkoeppe removed this from the sage-10.0 milestone Mar 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable VPATH builds (several independent build trees connected to one source tree) #21469

Enable VPATH builds (several independent build trees connected to one source tree) #21469

mkoeppe commented Sep 11, 2016

mkoeppe commented Sep 11, 2016

mkoeppe commented Sep 11, 2016

mkoeppe commented Sep 11, 2016

sagetrac-git mannequin commented Sep 11, 2016

sagetrac-git mannequin commented Sep 11, 2016

sagetrac-git mannequin commented Sep 11, 2016

sagetrac-git mannequin commented Sep 11, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

mkoeppe commented Sep 12, 2016

kiwifb commented Sep 12, 2016

mkoeppe commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

mkoeppe commented Sep 12, 2016

kiwifb commented Sep 12, 2016

mkoeppe commented Sep 12, 2016

jdemeyer commented Sep 12, 2016

jdemeyer commented Sep 12, 2016

mkoeppe commented Sep 8, 2017

mkoeppe commented Sep 8, 2017

mkoeppe commented Sep 8, 2017

dimpase commented Sep 8, 2017

jhpalmieri commented Sep 28, 2017

saraedum commented Jan 14, 2018

saraedum commented Feb 28, 2020

mkoeppe commented Feb 28, 2020

mkoeppe commented Feb 13, 2021

mkoeppe commented Jul 19, 2021

Enable VPATH builds (several independent build trees connected to one source tree) #21469

Enable VPATH builds (several independent build trees connected to one source tree) #21469

Comments

mkoeppe commented Sep 11, 2016

mkoeppe commented Sep 11, 2016

mkoeppe commented Sep 11, 2016

mkoeppe commented Sep 11, 2016

sagetrac-git mannequin commented Sep 11, 2016

sagetrac-git mannequin commented Sep 11, 2016

sagetrac-git mannequin commented Sep 11, 2016

sagetrac-git mannequin commented Sep 11, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

mkoeppe commented Sep 12, 2016

kiwifb commented Sep 12, 2016

mkoeppe commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

sagetrac-git mannequin commented Sep 12, 2016

mkoeppe commented Sep 12, 2016

kiwifb commented Sep 12, 2016

mkoeppe commented Sep 12, 2016

jdemeyer commented Sep 12, 2016

jdemeyer commented Sep 12, 2016

mkoeppe commented Sep 8, 2017

mkoeppe commented Sep 8, 2017

mkoeppe commented Sep 8, 2017

dimpase commented Sep 8, 2017

jhpalmieri commented Sep 28, 2017

saraedum commented Jan 14, 2018

saraedum commented Feb 28, 2020

mkoeppe commented Feb 28, 2020

mkoeppe commented Feb 13, 2021

mkoeppe commented Jul 19, 2021