cabal v2-install does not re-register a package unregistered by ghc-pkg

[demokritos]

Describe the bug
If a package is unregistered by ghc-pkg, cabal v2-install does not register the package again.
v2-install seems to check the package directory in the store, but not package db.

To Reproduce
Steps to reproduce the behavior:

1. In package source directory, cabal v2-install
2. ghc-pkg unregister --package-db=$HOME/.cabal/store/ghc-8.8.1/package.db
3. In package source directory, cabal v2-install ==> just print "up to date"
4. ghc-pkg list --package-db=$HOME/.cabal/store/ghc-8.8.1/package.db ==> no that package

$ ghc-pkg list --package-db=/home/esrevinu/.cabal/store/ghc-8.8.1/package.db
/home/esrevinu/.cabal/store/ghc-8.8.1/package.db
    hakyll-4.13.0.1
    lrucache-1.2.0.1
    time-locale-compat-0.1.1.5
$ ghc-pkg unregister hakyll --package-db=/home/esrevinu/.cabal/store/ghc-8.8.1/package.db
$ ghc-pkg list --package-db=/home/esrevinu/.cabal/store/ghc-8.8.1/package.db
/home/esrevinu/.cabal/store/ghc-8.8.1/package.db
    lrucache-1.2.0.1
    time-locale-compat-0.1.1.5
$ cabal v2-install --disable-documentation --ghc-option=-dynamic --allow-newer
Wrote tarball sdist to
/home/esrevinu/local/hakyll/dist-newstyle/sdist/hakyll-4.13.0.1.tar.gz
Resolving dependencies...
Up to date
Symlinking 'hakyll-init'
cabal v2-install --disable-documentation --ghc-option=-dynamic --allow-newer  4.92s user 0.51s system 99% cpu 5.459 total
$ ghc-pkg list --package-db=/home/esrevinu/.cabal/store/ghc-8.8.1/package.db
/home/esrevinu/.cabal/store/ghc-8.8.1/package.db
    lrucache-1.2.0.1
    time-locale-compat-0.1.1.5

Expected behavior
The package should be re-added in the package db.

System information

• Operating system : Arch Linux
• cabal, ghc versions : cabal-install version 3.0.0.0
  compiled using version 3.0.0.0 of the Cabal library

[Tritlo]

I'm having the same issue on 3.4.0.0. I fixed it by deleting the corresponding .lock file in $HOME/.cabal/store/ghc-8.10.4/incoming/

[Mikolaj]

Oh, interesting. Does it look like ghc-pkg bug (doesn't remove .lock file) or cabal bug (should ignore .lock or at least look also elsewhere)?

[phadej]

I don't think this is a bug. Why OP is poking cabal store manually with ghc-pkg in the first place? Do they try to remove a package from the store? Using ghc-pkg manually is a wrong way. There is no right way though, as that shouldn't be needed.

[phadej]

IIRC cabal-install checks whether a package-id exists in the store just by checking whether the directory exists, i.e. it doesn't use ghc-pkg (checking the file-system is a lot faster when you need to check a lot of stuff). This is an implementation detail. So by using ghc-pkg manually and not doing other stuff cabal-install does user breaks the invariants cabal-install assumes, and this is an user error.

---

There is somewhat related issue, #6060, with an idea that cabal-install should have commands to check the status of the store, and possibly repair it (cabal-store-check in my -extras is one idea).

cabal-store-check does the opposite check, if the directory is removed but db cache not updated, it will report that.
I guess other way around should also be checked (if it doesn't, I don't remember).

I think this issue can be closed as invalid.

[Mikolaj]

@phadej: thank you very much for your input. I thought ghc-pkg and cabal are intended to work together without any restrictions, but from what you say, cabal treats ghc-pkg as its internal command that should be not invoked by users. Good to know.

@demokritos, @Tritlo: how can we help you with your workflows that involve manual invocation of ghc-pkg? E.g., would cabal-env from the repo @phadej mentioned (https://github.com/phadej/cabal-extras/tree/master/cabal-env) let you work without using ghc-pkg directly? Does any other workflow similar to those from comments in #6481 suffice? Or are your needs completely different?

[Tritlo]

Ideally: I'm installing a local version of my own packag that I want to have in global scope, and cabal install --lib works great. But when I want to update said package, there's no way for me to remove it from the database. So my options are to 1) bump the version (slow to bump for every tiny change while prototyping), 2) manually unregister and delete the lock file, or 3) nuke the whole store and start from scratch. I'd very much like to avoid tthe nuking if possible, but still be able to bump a package (that I know has no downstream dependencies!)

[jneira]

if the use case is "reinstall" a package from the cabal store (and it seems so to me from the initial description) my way to go is:

• determine what version of the package is being used in the project including the hash (cause the store usually has several entries for the same package and version number but different hashes, which are used to suffix the package directory entry inside the store dir)
  • not at my laptop right now but I think the package+hash being used is in the GHC package db inside dist-newstyle
• delete the directory inside the cabal store (if I don't care about all pkg+hash I delete all of them skipping the previous step)
• run cabal-store-check -w ghc-${version} --repair to fix the package db used by cabal
• install the package again

It is not completely satisfactory but it works for me.
I guess the reason to had not automating this is that if everything goes well you should not reinstall nothing in the store. cabal should reuse or install the package+version as many times as it is needed using different hashes, as the store is immutable (like nix)
But not always everything works as intended.

So I woul like ask reporters why did they have to reinstall the package (to tackle the root issue) and offer the workaround described above

[gbaz]

Just for reference, as I understand it, ghc-pkg doesn't work well with the v2 store because the v2 store basically uses a bunch of semi-documented/proprietary extensions to the package database that ghc-pkg hasn't been taught about. v1 was built over an existing ghc-pkg, but v2 was built in turn on top of ghc-pkg, and it was never in the design for the store to interact with ghc-pkg. Further, the ghc team isn't particularly interested in extensions ghc-pkg to do much with the v2 store. So any further work on extensions to v2-store functionality needs to be built into cabal-install itself.

[Tritlo]

@jneira that's a good workaround, but requires me to manually find a directory in the cabal store and delete it (including figuring out the hash), and an addtional tool. I'm personally not versed well enough in cabal to be comfortable with deleting store directories, so it would be better if there were an "offical" way to reinstall a package (or at least to clear out a package) from the store.

[gbaz]

A simple solution would be to provide an --overwrite flag for installing a particular individual package into the store. Even with cabal env this would be necessary, right? Because installing into an env still requires putting the package into the store, and if the source has changed that doesn't suffice on its own to change the hash.

Note that this has sort of the same issue as deleting a package from the store though, in that what happens if other packages in the store depended on this package and now its deleted or changed? Now they're broken in a way cabal can't know/see.

Another stray thought --

A different approach might be to say that when installing a local package into the store it should be hashed not only by the usual version/flags/etc parameters but also by an actual pick-your-hash of its actual tarball or source-dir. That seems the safest and most sound thing to do overall, assuming the hashing-in-store mechanism is flexible enough to allow it. But then of course if someone else depends on the package, how do they know what hash to pick... :-)

[jneira]

@Tritlo

I'm personally not versed well enough in cabal to be comfortable with deleting store directories, so it would be better if there were an "offical" way to reinstall a package (or at least to clear out a package) from the store.

Yeah, it involves some manual steps. However, as commented above, reinstall a package should not be needed if everything works so, what is the cause you need reintall it? Maybe we could attack the root issue and resolve it better.

Thanks to @gbaz i've remember another awful step in the process of manually reinstall a package:

• You have to reinstall the entire tree of dependent packages of the original one, repeating the process for all of them. So if it is, we say, the text package, practically you have to delete the entire store (wipe the entire store is the other, but worse, workaround, as you have to rebuild everything)

But it reveal the insightful @gbaz considerations (i will rephrase them):

• Introduce delete and update in a append only store increases complexity dramatically (and bugs, unintented effects, etc etc)
• Change the way you compute the ids (hashes) of the store would have a large impact in the code using it. Now cabal has an automating way to compute it and it can regenerate the hash wherever is neccesary, as it depends on pure package and build data (without taking in account the sources cause it would multiply exponentially the number of versions in the store and its size). Introduce a user final way to alter that would suppose lot of changes in the codebase, i guess.

So we should consider carefully add those changes. As i mentioned above, if we can tacke the root issues causing the need of remove or reinstall packages we will get two improvements for free.
For the cases where it still will be needed we could improve cabal-store-check (o better another tool using it) to include optionally all the steps:

• identify the package hash of the actual project build data
• delete the directory in the store (?)
• fix the store ghc package database
• repeat for entire dependency tree

[Tritlo]

@jneira That would be great! But in the mean time, at least give a more informative message. Right now it says "up to date" when you try to reinstall the unregistered package, but then the package cannot be resolved. Maybe check for the lock file and the actual directory itself on installation, and print a warning about a broken store?

[dcoutts]

I don't think this is a bug. Why OP is poking cabal store manually with ghc-pkg in the first place? Do they try to remove a package from the store?

What phadej said above is entirely correct. It's not a bug. Corrupting the store (by unregistering packages from the ghc pkg db) will result in sadness. Don't do that.

Just for reference, as I understand it, ghc-pkg doesn't work well with the v2 store because the v2 store basically uses a bunch of semi-documented/proprietary extensions to the package database that ghc-pkg hasn't been taught about. v1 was built over an existing ghc-pkg, but v2 was built in turn on top of ghc-pkg, and it was never in the design for the store to interact with ghc-pkg. Further, the ghc team isn't particularly interested in extensions ghc-pkg to do much with the v2 store. So any further work on extensions to v2-store functionality needs to be built into cabal-install itself.

I don't think it uses any semi-documented/proprietary extensions. The ghc package db within the store is a perfectly ordinary ghc package db. Cabal uses ghc-pkg to update and read the package db.

The point is that the overall cabal store is the combination of a lot of files (with a specific layout) and also a ghc package db. There are consistency rules, and cabal doesn't expect other tools to go and modify the store (and certainly not without respecting those rules). To delete a package from the store the process would be: delete the installed package directory, delete the corresponding file within the package.db/ dir and run ghc-pkg recache on that package db. Simply deleting the package registration leaves things inconsistent.

[phadej]

To delete a package from the store the process would be

... and all the transitive reverse dependencies. i.e. so ghc-pkg check doesn't complain. Forgetting to do this won't make cabal-install break right away, but it definitely may start behave weirdly.

Don't poke with store manually. Only cabal-intstalls should touch it. If you wish to (selectively) remove packages from the store, open a feature request. IMHO, cabal-install should provide "plumbing" tools to work with store.