build-tool-depends executables don't appear in shellFor

[ocharles]

I want to use tasty-discover, which I have added to my .cabal file as:

  build-tool-depends: tasty-discover:tasty-discover

As described in input-output-hk/nix-tools#95, this adds a build-tools entry in the cabal-to-nix expression, which is used in setup-builder.nix to form executableToolDepends. However, it doesn't propagate correctly into the shell produced with shellFor. For build-tool-depends to work, we need a tasty-discover entry in the ghc-pkg database that also has the tasty-discover executable. As things currently stand, there is no entry at all. We can use additional = pkgs: [ pkgs.tasty-discover ], but that will only add a tasty-discover to the package database as a library - no executables will get included.

[michaelpj]

Related/dupe: #760 #231

[ocharles]

Ah, thanks!

[rvl]

I think I know how to fix this now, thanks @ocharles.

[ocharles]

@rvl Oh, fantastic! Do you need any help with anything? Happy to test anything, if that helps!

[JBetz]

A similar issue seems to happen if your shell dependencies have build tool dependencies.

E.g., cardano-transactions has a test suite with a build tool dependency on cardano-tx, and when I try to create a shell on a cabal project that uses it, it errors with:

error: Neither the Haskell package set or the Nixpkgs package set contain the package: cardano-tx (build tool dependency).
If this is a system dependency:
You may need to augment the system package mapping in haskell.nix so that it can be found.
If this is a Haskell dependency:
If you are using Stackage, make sure that you are using a snapshot that contains the package. Otherwise you may need to update the Hackage snapshot you are using, usually by updating haskell.nix.

Not sure if the same thing happens on stack projects, but I would presume not considering that cardano-transactions uses stack.

[michaelpj]

I spent a little bit of time looking at this.

Firstly, I noticed that we don't pass build-tools into the fake component that we use to create the wrapped GHC for shellFor. Easy enough to fix.

But then it still doesn't appear in the package db. It turns out that makeConfigFiles doesn't use build-tools at all. So... how does anything work? My guess is that the Setup.hs interface to cabal doesn't insist on build-tool-depends being in the package db, and so we get away with just providing the executable. But probably we should register them, I'm just not sure how right now.

[Ericson2314]

Yeah I don't think they should be registered in the same package db, especially if package DBs are to be per machine for cross. When you need a library, you usually need lots of information beyond the machine code (unfoldings for inlining, etc.) but for an executable dependency, one just needs to be able to run the thing, and being agnostic to how the sausage is made is probably better for sake of determinism / caching.

[michaelpj]

Yeah I don't think they should be registered in the same package db, especially if package DBs are to be per machine for cross.

Well, not having it in the package db confuses cabal, so I don't see how we can avoid it (if we want this usecase to work!).

As far as cross goes, obviously the build-tools should be in the native package db in the glorious future once we've got a separate native package db. Then I think there's no real problem with them being there.

[matthewbauer]

Note that this really isn't specific shellFor, but specific to v2 cabal. You can always use v1-* cabal commands to get the old behavior (see SolveExecutables in cabal code). I think a reason this is coming up recently is that cabal 3.4.0.0 made v2 the default.

The reason this is really difficult is that both Nixpkgs & haskell.nix use the Setup.hs method of configure / install, which doesn't install a .conf file for executable components. I'm not sure if it makes more sense to patch Cabal so Setup.hs method installs executables correctly, or try to switch to the new install method.

See Cabal comment for reference:

build-tools dependencies
NB: Only include these dependencies if SolveExecutables
is True. It might be false in the legacy solver
codepath, in which case there won't be any record of
an executable we need.

https://github.com/haskell/cabal/blob/master/cabal-install-solver/src/Distribution/Solver/Modular/IndexConversion.hs#L348-L357

[Ericson2314]

Thanks for digging into that!

[fgaz]

As far as cross goes, obviously the build-tools should be in the native package db in the glorious future once we've got a separate native package db.

v2 cabal-install recently got a --package-db flag that an be used to specify additional package dbs, though I'm not sure if it can actually help with this issue. Here's the pr for refernece: haskell/cabal#7676

[tscholak]

For the record, this is also an issue for llvm-hs, which needs the hsc2hs executable: https://github.com/llvm-hs/llvm-hs/blob/eda85a2bbe362a0b89df5adce0cb65e4e755eac5/llvm-hs/llvm-hs.cabal#L74.

[codygman]

The latest state of this issue seems to be: #1367 (comment)

[michaelpj]

No, I think that's unrelated. This isn't anything to do with source-repository-packages. This is specifically about how build-tool-depends works.

[L-as]

It seems like this is a duplicate of #231?

[michaelpj]

This one is more general, yes the other is a dupe.

[parsonsmatt]

Is there any work that can be done to help solve this problem? We're trying to migrate to haskell.nix but this is a problem for us. We'd be happy to contribute fixes or consulting time to the issue.

[ParetoOptimalDev]

Is there any work that can be done to help solve this problem? We're trying to migrate to haskell.nix but this is a problem for us. We'd be happy to contribute fixes or consulting time to the issue.

I haven't been able to look into this yet, but I really hope you are able to push this front forward.

My primary selling point for haskell.nix has been "You can avoid knowing nix and just update your cabal.project/package.cabal file but get cross compilation, reproducibility, docker containers, etc". @parsonsmatt I see in your linked issue your team is wanting the same: Use cabal.project as a source of truth.

Partial cabal.project support breaks this sellling point. I hope to deep dive into this issue and figure out haskell.nix well enough to solve some of these problems myself in the future, but I have other high priority things blocking it sadly 😢

[andreabedini]

After commenting on the linked cabal issue, I came here to say this is a short-coming of shellFor and should be fixed in haskell.nix, but then I noticed I do get tasty-discover in my devshell!

❯ nix develop --command which tasty-discover
/nix/store/6wmdh78kw1qzb4kgl6nchpsyhlxsds25-tasty-discover-exe-tasty-discover-5.0.0/bin/tasty-discover

Did the issue solve itself? Or maybe the issue is that, inside the shell, cabal will recompile tasty-discover anyway? (which is related to #1662).

[JackKelly-Bellroy]

I don't think we're all the way there. I too am seeing tasty-discover show up in my shell now, but I tried deleting my ~/.cabal/store and rebuilt a tasty-discover-using project, and cabal still wanted to build tasty-discover.

[andreabedini]

@JackKelly-Bellroy thanks for double checking! Yes, unfortunately I would expect cabal to rebuild it (issue I have been discussing in the cabal repo). At least they should be exactly the same version; if not, it's a bug in haskell.nix we need to fix.

[ScottFreeCode]

FWIW this is an underlying nix-shell + cabal v2 problem and not just a haskell.nix thing – it's also possible in general to use cabal as the source of truth with nix's callCabal2nix and save haskell.nix for when you truly need its more complex features.

I have a straightforward reproduction without iohk's/iog's haskell.nix, here: https://gist.github.com/ScottFreeCode/c1adb5681c5b61c61373615f63a4a7d2

I've collected GitHub issues relating to this problem here: https://gist.github.com/ScottFreeCode/ef9f254e2dd91544bba4a068852fc81f

@andreabedini Here's another one you checked in on recently that appears to be this same issue.

[toonn]

I'm observing different behavior from that described here.
tasty-discover does show up on my PATH and cabal-install doesn't seem to rebuild it but instead it rejects the installed tasty-discover.

> type -p tasty-discover
/nix/store/z4bhg9m76k78gkln10smq7bv9bph0ym7-tasty-discover-exe-tasty-discover-5.0.0/bin/tasty-discover
> cabal test --enable-tests
Warning: No remote package servers have been specified. Usually you would have
one specified in the config file.
Resolving dependencies...
Error: [Cabal-7107]
Could not resolve dependencies:
[__0] trying: taskell-1.11.4 (user goal)
[__1] trying: taskell:*test
[__2] next goal: taskell:tasty-discover:exe.tasty-discover (dependency of taskell *test)
[__2] rejecting: taskell:tasty-discover:exe.tasty-discover-5.0.0/installed-68WXL8qsKSKFFqrGegwKHL (does not contain executable 'tasty-discover', which is required by taskell *test)
[__2] fail (backjumping, conflict set: taskell, taskell:tasty-discover:exe.tasty-discover, taskell:test)
After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: taskell, taskell:test, taskell:tasty-discover:exe.tasty-discover, tasty-discover
Try running with --minimize-conflict-set to improve the error message.

Haven't been able to come up with a workaround so far.

EDIT: Workaround proferred by sclv is to provide tasty-discover on the PATH in another way and remove the build-tool-depends from the cabal file. This does break the single-source-of-truth that we're all after though.