Fix "warning: -jN forced in submake: disabling jobserver mode" warnings

[mkoeppe]

We should try to fix these warnings:

warning: -jN forced in submake: disabling jobserver mode.

There are 3 different issues:

1. The warnings appears several times at the top-level (not inside a package build) and during the build of the Sage library. This can probably be fixed by moving this snippet from sage-env higher up:

# If MAKEFLAGS exists, assume it got set by make.
# Therefore, remove all flags from $MAKE
if [ "${MAKEFLAGS-__unset__}" != "__unset__" ]; then
    MAKE=`echo "$MAKE" | sed 's/ .*//'`
fi
export MAKE

2. For several packages, the warning correctly appears because we explicitly force -j1 (presumably to fix parallel build issues):

• brial
• gap
• pari
• pkgconf
• python2
• singular
• zeromq

3. Other packages where this warnings appears:

• openblas: no idea where it comes from, probably the openblas build system itself.

CC: @embray @jdemeyer @vbraun @jhpalmieri @dimpase @orlitzky

Component: build

Issue created by migration from https://trac.sagemath.org/ticket/21610

[dimpase]

comment:1

I always wondered about the reason for the current recommendation; it would be good to get to the bottom of it.

[jdemeyer]

comment:2

The make -jNUM command can only work properly if you use make for everything. In Sage, we don't use make for all Python-related builds, in particular the build of the Sage library and the documentation. For this, we actually look at the environment variable MAKE. So this proposal just doesn't work.

I am adjusting the ticket issue.

[jdemeyer]

comment:3

In other words, the MAKE="make -jNUM thing is a hack, but it works.

[dimpase]

comment:4

Replying to @jdemeyer:

The make -jNUM command can only work properly if you use make for everything. In Sage, we don't use make for all Python-related builds, in particular the build of the Sage library and the documentation. For this, we actually look at the environment variable MAKE. So this proposal just doesn't work.

OK, would this mean that it's just an unfortunate choice of the name for this variable?
So if we renamed it to something it would also fix these warnings, etc?

[jdemeyer]

comment:5

Replying to @dimpase:

OK, would this mean that it's just an unfortunate choice of the name for this variable?

No, because it also serves as MAKE variable.

So if we renamed it to something it would also fix these warnings, etc?

I think that we can fix the warnings without changing the variable name.

[jdemeyer]

comment:6

I am currently doing a build from scratch of Sage, I'll check when those warnings appear.

[jdemeyer]

comment:7

In all cases where the warning occurs inside a package build, it is because of problems with the upstream package and it has nothing to do with the MAKE variable. Sometimes the upstream Makefile does not properly support a parallel build and we need to force -j1.

[dimpase]

comment:12

Still, I don't understand why the toplevel makefile can't set MAKE (or appropriately renamed) variable as it sees fit (from the value of -j parameter). Why do we do this highly non-standard way of starting the build, with setting MAKE variable manually?

[embray]

comment:13

I've had this same thought before--really we should not be passing down MAKE=make -jN to sub-makes.

The problem is that for Sage's build it's being used for cross-purposes. For packages that use make we want them to be able to take advantage of parallel builds (but jobserver mode is disabled that means the number of jobs started by make can actually well overtake N). But at the same time we want to take advantage of it for Sage to build multiple packages simultaneously.

I experimented with this a bit in the past and found that it could actually speed things up quite a bit, but only in limited cases. To really make it work properly we need further reworking of the spkg structure so that the builds for packages that use make are actually launched as proper sub-makes.

[jdemeyer]

comment:14

Replying to @dimpase:

Still, I don't understand why the toplevel makefile can't set MAKE (or appropriately renamed) variable as it sees fit (from the value of -j parameter).

Because there is no simple way to figure out the value of -j from a running instance of make.

[jdemeyer]

comment:15

Replying to @embray:

jobserver mode is disabled that means the number of jobs started by make can actually well overtake N).

This statement is wrong. In Sage, parallel make works for packages using make. The problem is the parts of the build which do not use make, in particular the build of the Sage library and documentation.

[dimpase]

comment:17

Replying to @jdemeyer:

Replying to @dimpase:

Still, I don't understand why the toplevel makefile can't set MAKE (or appropriately renamed) variable as it sees fit (from the value of -j parameter).

Because there is no simple way to figure out the value of -j from a running instance of make.

How about this:

$ cat Makefile 
all: opts

opts:
	@echo $(MAKEFLAGS)

Now:

$ make -j10
-j10 --jobserver-auth=3,4

That is, you can parse top-level MAKEFLAGS and get the value you need.

[jdemeyer]

comment:18

What does make --version say? I would swear that I have tested this at some point and then it didn't work.

[jdemeyer]

comment:19

Well, your trick does not work with

GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

The output is -j --jobserver-fds=3,4

[dimpase]

comment:20

Replying to @jdemeyer:

Well, your trick does not work with

GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

The output is -j --jobserver-fds=3,4

mine is 4.2.1. Well, a bug in make...
In 3.81 it is --jobserver-fds=3,4 -j So they tried to fix it ;-)

[jdemeyer]

comment:21

So there is hope that we might be able to implement this in the future...

[dimpase]

comment:22

Replying to @jdemeyer:

So there is hope that we might be able to implement this in the future...

or we can add make 4.2.1 package and have it now ;-)

[jdemeyer]

comment:23

Replying to @dimpase:

or we can add make 4.2.1 package and have it now ;-)

Not really because we are talking about the user running the system make.

What we could do is try to support both mechanisms: for users running a recent enough version of make, we could support make -j10.

[dimpase]

comment:24

just for the sake of it, added this piece of make info as an answer at http://stackoverflow.com/questions/10898528/how-can-i-tell-what-j-option-was-provided-to-make

[mkoeppe]

comment:27

See #30369 for another approach -- actually using the jobserver protocol.

[mkoeppe]

comment:29

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.