-
Notifications
You must be signed in to change notification settings - Fork 704
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build-tool-depends
executables don't have a .conf
file
#8434
Comments
I'm unfortunately very ignorant of Nix. What is your situation? Are you committed to Nix or does the "new install method to use" include something without Nix? Alternatively, I seem to remember Nix people that use the Nix shell to just normally call cabal, presumably including v2- commands. Is that an option for you? Is |
We currently are running In the In the I'd frankly prefer we not use
The value-prop of As far as I can understand it, the real question is:
|
So let's focus on this one. Please correct me as I spell this out in my own words. I see the files in my project:
These have been generated by So the idea is that I suppose, in the best case, it's just the matter of adding a flag to Setup.hs that calls the already existing code from v2- to create the .conf file. In the worst case, the Setup.hs code is deep within the Cabal library and would need to call some code deep within the cabal-install package and would lack tools to create the arguments the cabal-install code needs. Does anybody know which it is? Could |
Right, the main issue seems to be using
I don't think that's true. Setup.hs will generate them with https://cabal.readthedocs.io/en/3.4/setup-commands.html#runhaskell-setup-hs-register I think the issue originates from these lines: cabal/Cabal/src/Distribution/Simple.hs Line 790 in 76f6d3c
cabal/Cabal/src/Distribution/Simple.hs Line 805 in 76f6d3c
Since with v2- we do want to register even if there are no library components.
That's possible. Might be worth asking haskell.nix if that would work. I actually didn't realize there was a v2-install until today. I wonder if any other package managers are using v2-install to package haskell. |
Oh, right. If Setup.hs thinks exes don't need to be registered, that could be fixed and there should be no harm in doing so. I wonder if Setup.hs knows how to register exes (is that only about
That's quite possible for package managers (e.g., the HomeBrew one) that don't package Haskell libraries but just build exes from Hackage and package those, which is a very sane thing to do. Otherwise, |
Because if I have a haskell.nix project using a build tool like hspec-discover I would rather nix do the building. I'd rather nix do the building because it will likely skip building entirely and download the prebuilt cached hspec-discover binary from cache.nixos.org or my cachix binary cache. The problem is cabal v2 won't trust the nix built hspec-discover as a build tool. From my perspective it seems either:
I hope it's not the second, because then cabal.project can't be a source of truth for Haskell.nix. |
Something that may clear up confusion:
This issue is about that last point being true for buildTools. |
I see, so it's nix that installs packages globally, and to v2 they look like just a bigger set of packages bundled with GHC, so it knows how to handle that. Except that the bundled packages (libraries) are never used as @gbaz: you have an intimate knowledge of these things; how hard would it be to teach v2 to use any random executables in path for |
@Mikolaj Is there a deprecation plan for The current best work-around for using Nix correctly in the workflow where Nix provides packages and build-tools is to use
I suppose I could send a patch to I feel uncomfortable basing my workflows on I saw an interesting proposal in #4646 about getting Nix and Cabal to cache dependencies in a way where each is compatible with the other:
That sounds quite ideal, maybe that's worth opening an issue on for wider discussion? Edit: I also see a proposal for Build Drivers that could help this issues, but I don't fully understand everything yet: #7906 In any case, thanks for your help 🙂 |
The only plan I know of is shutting down the lights in a major version release. Which is why we are not keen on doing that until somehow the world sorts itself out (people's workflows update, v2 supports more of v1, etc.). Semi-maintaining v1 costs something, but so far not so much as justify harsh solutions IMHO.
Is that a real problem? Afterwards cabal knows about the binary and will not (often) install again.
Yes, that looks fine and the other tickets you linked, as well. Unfortunately, lately we have quite many valuable public discussions and comparably few actual implementations. If there was a party committed to implementing a particular feature then the relevant discussions would be a much better use of our scarce resources, I'm very much open to imperfect designs and implementations that scratch a real itch and are maintained by users, but not so much to public debates of universal solutions. We just don't have the resources. |
Arguably no, but I'll list a few issues:
I very much agree with about imperfect designs and implementations being more valuable than public debates of universal solutions. I want to pull the trigger and try either directly patching cabal-v2 to allow Nix to provide it My fear, and perhaps the fears of others, is putting work into a patch and then the contribution not being found acceptable. Is there a best practice for avoiding this? Should I create some issue like "RFC: Allow cabal-v2 to trust build-tools from Nix" and state my intentions to work on that if no one disagrees? My other looming fear I haven't touched on is that it feels like there is some resistance to integrating better with Nix. Challenging that feeling though:
So perhaps my fears are unfounded and I would even be okay with just trying to hack on Cabal to allow it to trust build-tools? In the spirit of respecting scarce resources I'll say that I would have made this response shorter if I had the time 🙂 |
Fair enough. At the very least, for reproducibility and curbing chaos, it should be clearly delineated which parts of the "haskell toolchain" cabal takes care of and which Nix does.
Go for it!
Some, though not all, features can be maintained as separate tools (see https://github.com/phadej/cabal-extras or cabal-plan). This is actually a very good way of starting the development process, regardless if and when they get merged into cabal proper or advertised in cabal manual or release notes (we should do that more often).
Yes, please do, even if that's going to be an external tool (no idea if that makes sense).
I think the fight has gone out of us. ;D Expect no fierce resistance. Seriously, though, there is certainly some duplication of functionality there, with the accompanying maintenance effort whenever the functionality in questions changes in GHC and/or cabal. So the extent to which we duplicate needs a pragmatic justification (not just noble causes, like "every binary should be cached" or Debian's "every library should be shared"; we see where this leads) and a plan for future maintenance. I'm sure we never had a problem with Nix, only with the natural scenario of abandoned volatile features we don't understand, but we eventually need to (un)maintain. I'm sure we can sort it out in a ticket. |
This is absolutely a concern of ours at Mercury, as well. We're happy to put effort/cash towards fixing this issue, but we don't to waste those resources on a design that wouldn't be accepted. |
That's understandable. Let's fail fast, already at initial design phase or at planing phase, e.g., if we see that's going to be too expensive for any of the parties. However, if an external tool that uses some of the four cabal libraries (that's why we split them off) is an acceptable fit, perhaps with some extra effort required to make things less monolithic, then the risk of rejection diminishes. The tool could also use some of the already existing external tools, e.g., for parsing cabal files. In general, it would be ideal if we had a thriving ecosystem of external cabal tools/plugins, with only as much changes as needed done within cabal proper. Only the tools that "everybody uses" and/or that require minimal maintenance would be merged with cabal at some point, if the author is fine with the drawn out release cadence. |
My understanding from talking to past cabal developers and maintainers is that this is something we absolutely should seek to fix. For libraries, cabal builds and leverages a proper packagedb. For executables, the executabledb was built in a somewhat ad-hoc way and never fully specified for external interaction. We should pin down the spec of the executabledb so that nix or any other package management system can properly register packages with it. If the mercury team (or anyone!) were to come up with a proposal for a proper executabledb that nix and others could leverage, I think that would be a good step consistent with the design of v2-build and in accord with what the intent always was. Of course I'd need to see the details of such a proposal before stating I support it :-) I do very much think that the solution provided should not be specific to nix, but rather something more general, as I believe the discussion in this thread has already pointed to. In particular, if Setup.hs installing executables with a conf file really suffices, then this is a very small change which could potentially have a very nice result. (However, I think we would want to carefully consider what knock-on impact, if any, this may have -- in particular, what directory would the conf files go in, how would cabal-the-executable know to look in that directory, etc). |
FYI I'm at least getting my feet wet with the Cabal codebase and starting to understand the control flow now that I've gotten it building locally. I know where checks in a executable database would go in the repo at least, so progress 😄 Today I'll try to understand the library package database code so I can start imagining what an executable database might look like. |
My attempt of making progress on this issue via modeling the library package database behavior won't work because from a Nix perspective even Given cabal.project:
myproj.cabal:
And having cborg in the package database whether from Nix or otherwise so that this it's listed by $ nix develop -c ghc-pkg list | grep cbor
cborg-0.2.8.0 Cabal will still go to clone the cborg library. I guess that makes sense becausethere isn't a good way for cabal to tell that the library in the package database fulfills what's asked for by the This is just the tension of both Which means that hybrid Nix and cabal project can't exist without hacky solutions like a separate Another recent workaround for this recommended as a best practice here is building your own internal hackage with foliage and eschewing Maintaining compatibility with the typical cabal.project workflow is paramount though as someone introducing new teams to Nix on a trial basis though. "Replace source-package-repository with your own internal hackage" would likely be laughed away and make the entire endeavor a non-starter. The closest issue I can find to this is #5444. So I'm diverting my attention from this issue and to #5444, specifically starting with reflecting on this comment where a cabal maintainer states:
|
build-tool-depends doesn't play nice with Setup.hs installs, and tries to rebuild cabal-install/doctest locally. This has recently caused some compilation errors so there is now a 'ci-depends' flag to toggle between build-tool-depends (for CI systems) and build-depends (for users with the needed tools installed to their system). This reverts commit 51bc8c1. See: gentoo-haskell/gentoo-haskell#1074 See: haskell/cabal#8434 Signed-off-by: hololeap <hololeap@users.noreply.github.com>
build-tool-depends doesn't play nice with Setup.hs installs, and tries to rebuild cabal-install/doctest locally. This has recently caused some compilation errors when it tries to build cabal-install, so we'll just remove it from build-tool-depends on the assumption that whoever is running doctest-v2 probably has /usr/bin/cabal on their system. This reverts commit 51bc8c1. See: gentoo-haskell/gentoo-haskell#1074 See: haskell/cabal#8434 Signed-off-by: hololeap <hololeap@users.noreply.github.com>
build-tool-depends doesn't play nice with Setup.hs installs, and tries to rebuild cabal-install/doctest locally. This has recently caused some compilation errors when it tries to build cabal-install, so we'll just remove it from build-tool-depends on the assumption that whoever is running doctest-v2 probably has /usr/bin/cabal on their system. Stop using all but Cabal-3.8 on GitHub tests. The others fail during dependency resolution complaining that they cannot build ghc. This reverts commit 51bc8c1. See: gentoo-haskell/gentoo-haskell#1074 See: haskell/cabal#8434 Signed-off-by: hololeap <hololeap@users.noreply.github.com>
build-tool-depends doesn't play nice with Setup.hs installs, and tries to rebuild cabal-install/doctest locally. This has recently caused some compilation errors when it tries to build cabal-install, so we'll just remove it from build-tool-depends on the assumption that whoever is running doctest-v2 probably has /usr/bin/cabal on their system. All Cabal versions except 3.8 fail on GitHub tests complaining that they cannot build ghc: ``` cabal: Failed to build ghc-8.10.7 (which is required by exe:doctest from doctest-0.21.1) ``` This reverts commit 51bc8c1. See: gentoo-haskell/gentoo-haskell#1074 See: haskell/cabal#8434 Signed-off-by: hololeap <hololeap@users.noreply.github.com>
build-tool-depends doesn't play nice with Setup.hs installs, and tries to rebuild cabal-install/doctest locally. This has recently caused some compilation errors when it tries to build cabal-install, so we'll just remove it from build-tool-depends on the assumption that whoever is running doctest-v2 probably has /usr/bin/cabal on their system. Switch to `cabal-version: 'latest'` instead of a matrix of cabal versions for github action. Use template from haskell/actions/setup homepage where possible: https://github.com/haskell/actions/tree/main/setup#model-cabal-workflow-with-caching This reverts commit 51bc8c1. See: gentoo-haskell/gentoo-haskell#1074 See: haskell/cabal#8434 Signed-off-by: hololeap <hololeap@users.noreply.github.com>
build-tool-depends doesn't play nice with Setup.hs installs, and tries to rebuild cabal-install/doctest locally. This has recently caused some compilation errors when it tries to build cabal-install, so we'll just remove it from build-tool-depends on the assumption that whoever is running doctest-v2 probably has /usr/bin/cabal on their system. Switch to `cabal-version: 'latest'` instead of a matrix of cabal versions for github action. Use template from haskell/actions/setup homepage where possible: https://github.com/haskell/actions/tree/main/setup#model-cabal-workflow-with-caching This reverts commit 51bc8c1. See: gentoo-haskell/gentoo-haskell#1074 See: haskell/cabal#8434 Signed-off-by: hololeap <hololeap@users.noreply.github.com>
build-tool-depends doesn't play nice with Setup.hs installs, and tries to rebuild cabal-install/doctest locally. This has recently caused some compilation errors when it tries to build cabal-install, so we'll just remove it from build-tool-depends on the assumption that whoever is running doctest-v2 probably has /usr/bin/cabal on their system. Switch to `cabal-version: 'latest'` instead of a matrix of cabal versions for github action. Use template from haskell/actions/setup homepage where possible: https://github.com/haskell/actions/tree/main/setup#model-cabal-workflow-with-caching This reverts commit 51bc8c1. See: gentoo-haskell/gentoo-haskell#1074 See: haskell/cabal#8434 Signed-off-by: hololeap <hololeap@users.noreply.github.com>
build-tool-depends doesn't play nice with Setup.hs installs, and tries to rebuild cabal-install/doctest locally. This has recently caused some compilation errors when it tries to build cabal-install, so we'll just remove it from build-tool-depends on the assumption that whoever is running doctest-v2 probably has /usr/bin/cabal on their system. Switch to `cabal-version: 'latest'` instead of a matrix of cabal versions for github action. Use template from haskell/actions/setup homepage where possible: https://github.com/haskell/actions/tree/main/setup#model-cabal-workflow-with-caching This reverts commit 51bc8c1. See: gentoo-haskell/gentoo-haskell#1074 See: haskell/cabal#8434 Signed-off-by: hololeap <hololeap@users.noreply.github.com>
Re-reading this after a long time (and maybe missing some detail, it's a bit of a long thread across different repos). From a cabal POV, it's not clear what value would bring to put exe components in a packagedb. We have packagedb to tell GHC where things are and GHC is never going to load an executable component (AFAIK). In terms of allowing other tools to understand what
When this level of detail is not enough, one can always pick inside cabal-install's brain using From a haskell.nix POV, I would say this is a short-coming of how |
Another aspect is cabal rebuilding build-tool-depends, even if they are already "available" somewhere. I believe that problem is complex and might not even be entirely solvable (I guess this balances the optimism in the previous comment :P). cabal-install does use packagedb to pick pre-installed libraries when appropriate and maybe this is where the idea of .conf files for executables come from. Unfortunately, this mechanism has already plenty of issues (e.g. #6039, #8702). Taking a step back, if the problem is that cabal will recompile tools already available, maybe a simple |
@parsonsmatt @ParetoOptimalDev You actually don't need haskell.nix to have Nix treat Cabal as the source of truth. Just an underdocumented function in Nix called (EDITED to add: This still runs into the same issue about v2 not working inside nix shells.) |
I have a straightforward reproduction without iohk's/iog's haskell.nix, here: https://gist.github.com/ScottFreeCode/c1adb5681c5b61c61373615f63a4a7d2 I've collected GitHub issues relating to this problem here: https://gist.github.com/ScottFreeCode/ef9f254e2dd91544bba4a068852fc81f |
@ScottFreeCode Thanks. I won't promise I will finish reading all this anytime soon but I appreciate your notes! They are valuable. |
@ScottFreeCode Treating cabal as the source of truth requires also parsing There is a project that experiments with parsing the Right now in a Nix project that uses callCabal2Nix any One workaround that exists for making sure Nix packages are used is to remove all To clarify my use case is "Nix provides ghc, cabal, and all haskell packages" and then cabal is just used as a build tool without network access. |
@andreabedini I don't understand your comments. You seem to agree why a conf file would be useful -- so that cabal could use an existing executable that's been registered rather than compiling its own and duplicating work, just as it can already use an existing globally-registered library rather than compiling its own. However, you seem to go out of your way to propose a lot of other solutions that may address certain cases. What objection, if any, do you have to just having the executable db -- for use of cabal -- as this ticket proposes? |
@ParetoOptimalDev I'm not as familiar with That said/asked, I'm assuming there may still be relatively more complex cases that haskell.nix makes easier; but I know I only tried it because I couldn't find an explanation of how to make Nix piggyback on the package.cabal definition in general – which turned out to be possible (easier to set up and more efficient than haskell.nix, even!) but just rare to find examples or documentation to work off of. So I wanted to put it out there, not that there's never a need for haskell.nix, but that it's still possible to use that Cabal-first approach with "vanilla" Nix to start and switch to haskell.nix only when some additional requirement cannot be easily met with |
(Assuming you refer to #8434 (comment))
My only (soft) objection is that there are already many things that packagedb + solver cannot do at this moment. I agree with you (in #8434 (comment)) that this needs to be solved; but I am still not sure of the design. (To be better discussed elsewhere but my question I get stuck trying to answer is: who owns packagedb and what is that for? if GHC owns it, then why does cabal-install writes cabal-install-specific build information in it? If Cabal owns it, would that be any different? (Cabal /= cabal-install, stack uses Cabal). Note GHC already throws away most of the content of packagedb entries and uses its own internal database (Which it calls "cache")) |
I don't think this would add any complexity to the solver. We just check if something suitable is already registered, and if not, solve for it. The solving for it part we do already -- so this comes before the solver, and eases its burden if anything. Nor would this necessarily be the same packagedb already in use -- just something conceptually similar to it. So I understand your reservations about putting more load on those two items, but this proposal doesn't seem to do either. That said, I would say on the packagedb that "ghc owns the packagedb jointly with Cabal-the-library" just as a matter of the engineering fact that I think they need to jointly agree on any changes to the datastructure. |
This is significantly different from what I had immagined i.e. having pre-installed exe units/components that the solver could choose to use rather than their corresponding source package. In this case the difference with what I was proposing in #8434 (comment) is just a matter of input format: we can either register build-tool-depends in a file somewhere or list the available ones on the command line. I am still not sure how this (either version) is going to work though. We already have some logic to skip solving for them (as @matthewbauer observed a while ago), so we can easily surface that flag to project config and/or command line and skip solving for exe dependencies (or, with a little change, only some of them). This would roughly reproduce the bahaviour of v1 builds and installs w.r.t exe depends. But how do we maintain purity in the cabal store? Could we use the exe binary hash instead of its installed-unit-it? The store is only used to avoid recompilation, it does not influence solving, so the externally provided tools won't have to match those hashes. |
Thanks for the pointer & reminder. The issue I had: trying to install a library, needed for a compiler bug repro case, with GHC HEAD compiler built in only the No Nix, no flake, no cabal-project, no nothing — just direct usage of cabal-install & ghc-pkg. Bug & WTF counters overflow. To elaborate... briefly.
Another bug?.. Does Without Trying to add ... meanwhile: |
alex and happy aren't available for cabal v2-build in shellFor due to haskell/cabal#8434 NixOS/nixpkgs#130556 NixOS/nixpkgs#176887
alex and happy aren't available for cabal v2-build in shellFor due to haskell/cabal#8434 NixOS/nixpkgs#130556 NixOS/nixpkgs#176887 for some reason still needs `cabal update`, but doesn't download anything from there
alex and happy aren't available for cabal v2-build in shellFor due to haskell/cabal#8434 NixOS/nixpkgs#130556 NixOS/nixpkgs#176887 for some reason still needs `cabal update`, but doesn't download anything from there
alex and happy aren't available for cabal v2-build in shellFor due to haskell/cabal#8434 NixOS/nixpkgs#130556 NixOS/nixpkgs#176887 for some reason still needs `cabal update`, but doesn't download anything from there
alex and happy aren't available for cabal v2-build in shellFor due to haskell/cabal#8434 NixOS/nixpkgs#130556 NixOS/nixpkgs#176887 for some reason still needs `cabal update`, but doesn't download anything from there
alex and happy aren't available for cabal v2-build in shellFor due to haskell/cabal#8434 NixOS/nixpkgs#130556 NixOS/nixpkgs#176887 for some reason still needs `cabal update`, but doesn't download anything from there
This is a problem that manifests when trying to use
haskell.nix
. If it sees thebuild-tool-depends
entries, it does try to build them, but thencabal build
is unable to see or execute them.Cf input-output-hk/haskell.nix#839 (comment)
I'm unclear whether or not this is a feasible or acceptable approach, and would appreciate guidance on this. @matthewbauer says that we can either
The text was updated successfully, but these errors were encountered: