Packaging: support for patching the nix package

[roberth]

Is your feature request related to a problem?

This is about how Nix itself is packaged with Nix.

Since the packaging now consists of multiple derivations that share a single source (Nixpkgs) or portions of it (flake), it is now next to impossible to override the source or provide a patch.

Proposed solution

Add patching logic to everything.nix. Upstream it into Nixpkgs too.

Alternative solutions

Only do it in Nixpkgs. It is easier there because it has a single src.
I find this unsatisfactory because these packages should be drop-in replacements. Otherwise users will run into unnecessary issues when switching between Nixpkgs and the flake.

Additional context

• I'd really want package-local non-derivation variables for this.

  • Single package fixpoint step 2: introduce mkPackage nixpkgs#296769

• The components.nix / dependencies.nix scope(s) and the everything.nix package should be declared in the same fixpoint to avoid fixpoint hell.

---

Add 👍 to issues you find important.

[Ericson2314]

TBH if there is an makeScope scope withe source attribute, this appears pretty easy?

I do like #6507 of course, but I also think we should expose the individual non-test libraries at least as separate packages because IMO they really are separate packages.

(OTOH, main lib vs building tests vs running tests is definitely classic #6507 :))

[roberth]

I also think we should expose the individual non-test libraries at least as separate packages

They already are separate packages, but they reside in a "passthru".

Whether a package set has a type and an outPath, i.e. a default package, doesn't matter much conceptually. It's just a matter of presentation and UX. How the package set is presented should not have to affect its internal structure, but I don't know for sure how mature Nixpkgs utilities are in this regard until I've tried this particular thing.

Maybe we should do the simple thing first and have a pkgs.nixPackages.nix_2_26 package set first and smack an overrideSrc function on there to produce another such set. Funny name I guess. nixComponents.nix_2_26?
It takes a bit more fiddling for users, compared to the monolithic situation or the "default package" setup, but it'd at least be possible.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/overriding-the-nix-2-26-componentized-build/60428/2

[Mic92]

I keep wondering if all the complexity for nixpkgs is actually worth it for being able to re-use the same nix expression across the nix flake and nixpkgs.

[roberth]

Componentized builds in the repo are very beneficial for performance reasons.
I personally care about nix packages being maximally interchangeable, because I tend to use both a lot in other projects.

The time investment into these expressions is in my opinion worth it, because of related efforts, such as

• Single package fixpoint step 2: introduce mkPackage nixpkgs#296769
• [RFC 0092] Computed derivations rfcs#92 (comment)
• Redefine outputs in terms of language-level "package", not necessarily store-level derivation (RFC-92, and multi-drv packages, docs) #6507

Those are all about organizing packages and derivations, and those efforts cross-pollinate.

[roberth]

Takeaways from #12503:

• this package set has a default package
• the default package is not the root of the entire package set
• re-exposing package-set-level overrides on the default package is easy when it's ok not to add those to the overrideAttrs result
• we don't have a Nixpkgs mechanism that lets us cleanly implement overrideAttrs for the default package
  • mkDerivation has its own fixpoint
  • makeScope has its own fixpoint
  • because we can only handle one fixpoint well, either one has to go. mkDerivation is a natural candidate for removing its fixpoint. Declaring a whole package set in the default package's mkDerivation call would be an odd thing to do, especially if we care about exposing the other components more directly (agreeing with @Ericson2314)
• adding the package set as a "passthru" on the default package seems like a useful convenience, and can be implemented correctly
  • certainly for a single "passthru"; possibly also for re-exposing through multiple packages (although I believe this has little value)

[Ericson2314]

@Mic92 IMO it's good for us to dogfood a bit on the larger question of

How should upstream developers use Nix and support downstream packaging in Nixpkgs?

When we were really niche, this was a non-issue. But as we get more popular, it will come up more and more. I doubt we've settled on a final design --- it does feel quite complex today. But over time we'll work on the division of labor and abstractions between us and Nixpkgs, and I think we'll get better.