Ability to require more than 1 thread for an action

[ndmitchell]

With unsafeExtraThread you can acquire additional threads while running. It might be useful to have a withThreads equivalent that allows more threads to be used by a single rule, e.g. when running make -j as a subaction. See snowleopard/hadrian#289 for an example.

[waldheinz]

We're using shake to build bootable Linux images, not unlike what buildroot does. There are enough packages to build so that building individual packages with -j1 and letting shake take care of doing things in parallel usually gives pretty good CPU utilization.

One case where it doesn't work so well is for bigger packages like Mesa: When we change Mesa's configuration, the build system will rebuild Mesa and then the packages that depend on Mesa. That rebuild of Mesa will take quite a while, because it's using only one core...

So I wonder what a good API for withThreads would look like? I think something that allows to say "I'd like to use at least m and up to n threads" would be nice, and withThreads providing the actual number of cores acquired. This would allow that

• when building big packages in parallel to smaller ones (like when doing an initial build), Mesa gets a reasonable share of the cores and
• when Mesa is the only thing being built, it gets all the cores.

Obviously m and n would be capped to the -j parameter given to shake itself.

Another thing that sprung to my mind is that it might make sense to delay building bigger packages until one or more smaller packages are finished, to give the bigger ones some extra CPU horsepower. But maybe that's too tricky to get right... :-)

[ndmitchell]

Normally when executing an Action you have a single thread. I can imagine two possible API's:

-- run an action with N threads
withThreads :: Int -> Action () -> Action ()

-- run an action with (N..M) threads, saying how many you got
withThreadsRange :: (Int, Int) -> (Int -> Action ()) -> Action ()

I imagine the semantics would be to come out of the thread pool, and add a new entry that requires N threads. Initially I'd probably require the inner Action () to be non-blocking (no calls to need), but that's potentially solveable. For the range, trying to reschedule things to optimise those targets would be infeasible. Having a range at all is possible, but complicates things quite a bit. Is a range really that useful? Why not just say Mesa always takes 4 CPU and have a table of "expensive" builds?

[mpickering]

I had a look at implementing this and ran into #823

Relevant GHC issue is https://gitlab.haskell.org/ghc/ghc/-/issues/21136