source-repository-package seems to be treated as local

[quasicomputational]

This means that --enable-tests will build the tests of those dependencies, --enable-benchmarks will build the benchmarks, new-test will (presumably) run their tests, and so on.

This was quite surprising to me, since my primary use case for source-repository-package is to swap in a broken version of a dependency with a fixed pre-release version from the VCS. But doing that means that, if the dependency's test suite is not in great condition (e.g., it doesn't even compile because I'm trying to port things to a new GHC version), I'm suddenly going to be affected by that!

The discussion in #2189 suggests that this is for implementation convenience (which is entirely reasonable), rather than a deliberate decision. Are there use cases where packages being pulled from a VCS should be local for principled reasons? I can't think of any off the top of my head, but there's a whole universe of potential weirdness out there.

Edit: Also, I suspect we could decouple the relevant aspects of 'localness': a source-repository-package could be treated as local for the purposes of recompilation checking and not going into the global store, but non-local for the purposes of 'should this package's tests/benchmarks be available for building/running?'.

[23Skidoo]

/cc @dcoutts @typedrat

[hvr]

Fwiw, I consider having source-repository-package cached in the global store a killer feature; and I feel like in the majority of use-cases you'd want to err on caching it globally rather than not. That being said, I welcome the ability to opt-out and force it to be treated as local if the user wishes so.

Something I think I briefly discussed with Duncan in the context of the source-repo-package UI was to have a generic uniform "package" syntax where you could explicitly force/control whether a package in the build-plan is demoted to a local package or not.

[typedrat]

How do packages that are in an index that is referenced in a specific cabal.project but is not globally available handled? Because honestly I think internally treating the source-repository-package as a source of packages rather than adding a new local package for it makes more sense.

[RyanGlScott]

What is the status of this ticket after #5708 was merged into cabal-install-2.4? I was originally redirected to this ticket after seeing #5612 (which I was directly affected by in haskell-CI/haskell-ci#186), and after upgrading to cabal-install-2.4, I can confirm that I am no longer affected by #5612. Since #5612 was closed as a duplicate of this ticket, does that mean this ticket is also fixed now?

[hvr]

@RyanGlScott this ticket is still relevant insofar as the source-repository-packages are still treated as if they were local packages whereas it's desirable instead to have the ability to treat them as non-local ones.

[harendra-kumar]

In a CI build --enables-tests starts building tests of a source-repository-package and sometimes the dependencies of tests even fail to build (for ghcjs in my current use case). Therefore, this issue is a must fix to enable the CI use case.

[hvr]

@harendra-kumar is this workaroundable by adding a package stanza matching the source-repository-package disabling the tests again? i.e. for a package foo pulled in via git,

package foo
  tests: False
  benchmarks: False

?

(if yes, maybe we can have this "hack" wired into cabal automatically, as a short-term workaround/quickfix )

[harendra-kumar]

@hvr this will work for me. I guess it makes sense to do it only if this is significantly easier/faster to do compared to the right long term fix.

[harendra-kumar]

@hvr thanks for that workaround, works great!

[michaelpj]

I just hit something that I suspect is another effect of this: packages that are included as a source-repository-package will never be taken from the package-db.

Why would you want this? If you are using Nix and you are in a shell which has been prepared with your dependencies (including the source-repository-pacakges), then you want to actually use those, and not recompile them! (Why list them as source-repository-packages then? Because the cabal files are still the source of truth for the nix tooling.)

But I think this is just a consequence of source-repository-packages being treated more like local packages than "normal" dependencies.

[symbiont-sam-halliday]

Another consequence of this: if we have a freeze file (e.g. from stackage) and then a source dependency where the source version has move the version forward, then the constraint solver fails to find a solution. I think the constraint solver should ignore the constraint on the package in the case that there is an explicit source dependency (this is what stack does).

[michaelpj]

I just hit precisely the issue Sam mentions today.

[deepfire]

I just hit something that I suspect is another effect of this: packages that are included as a source-repository-package will never be taken from the package-db.

So, from discussion with @dcoutts, it has been further confirmed, that this shadowing is by design -- the source-repository-package is intended to designate a mandatory request to use a particular source for the particular package.

If we follow this design, then the proper solution for package DB un-shadowing would be conditionalisation of the clauses -- as per #6249

[deepfire]

Just as a matter of recording here the information:

Duncan says that to change "what is considered local for the purpose of applying project-wide flags/settings", the place to modify is
https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectPlanning.hs#L1926-L1936

..and so we want to drop the repositories there -- and further, that:

In fact we probably want to only include LocalUnpackedPackage

[jchia]

With the issue closed, is a source-repository-package now supposed to be cached in the global store? I just ran cabal freeze on my project with 3.4.0.0-rc1 and saw cabal still trying to clone a github source-repository-package into dist-newstyle/src in my project directory, not somewhere in the global store. Is this expected? If so, is it correct to say that every time I build my project from scratch, cabal will still do the cloning of this github source-repository-package?

[phadej]

The repository is cloned, but should be built only once. We decide whether it's the same via the contents, i.e. if you clone it from different fork, or get the sources somehow otherwise, it's still the same.

E.g. if the very same sdist is uploaded to the Hackage, which we get from git clone (which should be more likely if uploaders use cabal-install-3.4) then all those will be considered "the same version".

[jchia]

I suppose for a Hackage package you decide whether the package is the same by the package name and version? For a source-repository-package, at least for a git package, can we decide whether the package is the same via the location and tag (commit hash)? This way, the package (source) can also stored in the global store instead of the project's dist-newstyle/src. dist-newstyle is cleared every time "cabal clean" is run and then "cabal build" will have to download the source-repository-package all over again.

[phadej]

I suppose for a Hackage package you decide whether...

No, the contents of tarball also matter. The hash of contents (and .cabal file) are however available in 01-index, which we download from Hackage, so we don't need eagerly download sources of packages from Hackage.

[jchia]

Is it a design choice to cache the source-repository-package locally instead of in the global cache or is that just the thing that happens to be easy to implement?

[fgaz]

afaik it's a design choice. there should be a pr/issue somewhere with relevant discussion