Enhancing lib.makeScope

[nspin]

I'm interested in enhancing lib.makeScope or adding similar functions with more features.
I have three features in mind:

• Splicing for scopes that contain both build and host derivations
• Member access control (public, private, etc.)
• Ability to override ancestor scopes

I seek your thoughts on the features themselves and whether they belong in Nixpkgs. Nixpkgs does not use scopes heavily, so these features may be overkill.

For reference, here is lib.makeScope:

nixpkgs/lib/customisation.nix

Lines 185 to 204 in 83686eb

	/* Make a set of packages with a common scope. All packages called
	with the provided `callPackage' will be evaluated with the same
	arguments. Any package in the set may depend on any other. The
	`overrideScope'` function allows subsequent modification of the package
	set in a consistent way, i.e. all packages in the set will be
	called with the overridden packages. The package sets may be
	hierarchical: the packages in the set are called with the scope
	provided by `newScope' and the set provides a `newScope' attribute
	which can form the parent scope for later package sets. */
	makeScope = newScope: f:
	let self = f self // {
	newScope = scope: newScope (self // scope);
	callPackage = self.newScope {};
	overrideScope = g: lib.warn
	"`overrideScope` (from `lib.makeScope`) is deprecated. Do `overrideScope' (self: super: { … })` instead of `overrideScope (super: self: { … })`. All other overrides have the parameters in that order, including other definitions of `overrideScope`. This was the only definition violating the pattern."
	(makeScope newScope (lib.fixedPoints.extends (lib.flip g) f));
	overrideScope' = g: makeScope newScope (lib.fixedPoints.extends g f);
	packages = f;
	};
	in self;

Splicing

Currently, scopes created with lib.makeScope are not spliced. If one member of a scope depends on another member of that same scope as a native build input, it cannot resolve that dependency directly through callPackage. As a workaround, I've been using buildPackages to indirectly resolve such dependencies (e.g. referring to buildPackages.myScope.myPackage from within myScope).

I've come up with an unsatisfying but useful version of lib.makeScope which splices using the top-level pkgs${offset}${offset} attributes. As arguments, it takes a function for accessing its own result from the current scope, and a normal scope function:

{ lib, splicePackages, __splicedPackages, newScope }:

let
  compose = f: g: x: f (g x);

in {

  makeSplicedScope =
    let
      splicePackagesAt = ix: splicePackages (lib.mapAttrs (lib.const ix) {
        inherit (__splicedPackages)
          pkgsBuildBuild pkgsBuildHost pkgsBuildTarget
          /* pkgsHostHost not implemented */ pkgsHostTarget
          pkgsTargetTarget
          ;
      } // {
        pkgsHostHost = {};
      });

      makeSplicedScopeWith = newScope: ix: f:
        let
          self = f self // {
            newScope = scope: newScope (splicePackagesAt ix // scope);
            callPackage = self.newScope {};
            overrideScope = g: makeSplicedScopeWith newScope ix (lib.fixedPoints.extends g f);
            makeSplicedScope = ix': makeSplicedScopeWith self.newScope (compose ix' ix);
          };
        in self;

    in
      makeSplicedScopeWith newScope;

}

In the example usage below, foo and bar are spliced within myScope, so foo can use { bar }: { nativeBuildInputs = [ bar ]; } as it would in the top-level Nixpkgs scope.

let
  myScopeFn = self: with self; {
    foo = callPackage ./foo.nix {};
    bar = callPackage ./bar.nix {};
  };
in self: super: with self; {
  myScope = makeSplicedScope (x: x.myScope) myScopeFn;
}

I believe that the splicing of scopes can be accomplished in a way which does not require passing the scope's path from its parent's top-level, but I haven't found one yet.

Member access control

Features to organize scope members would allow for finer-grained access to members and more precise overriding. I'm not proposing that the definition of makeScope presented in this section should replace lib.makeScope. I'm only using it to illustrate some ideas.

The following definition of makeScope organizes scope members into three classes: public, protected, and private. Members defined at the scope function's top-level are public. Those specified within the _protected (resp. _private) attribute are protected (resp. private). The resulting scope attrset contains, at its top-level, all public members, the attributes added by makeScope (e.g. callPackage), and an attrset at _public (resp. _protected, _private) containing only the scope's public (resp. protected, private) members.

All members (public, protected, and private) are available to the scope via its callPackage function. However, only public and protected members are available via descendant scopes' callPackage functions.

{ lib }:

{

  makeScope = newScope: f:
    let
      raw = f self;
      _public = builtins.removeAttrs raw [ "_protected" "_private" ];
      _protected = raw._protected or {};
      _private = raw._private or {};
      extra = {
        newScope = scope: newScope ( _public // _protected // // scope);
        callPackage = newScope (_public // _protected // _private // extra);
        overrideScope = g: makeScope newScope (lib.fixedPoints.extends g f);
      };
      self = _public // extra // {
        inherit _public _protected _private;
      };
    in self;

}

Better organization of scope members prevents issues like #68525.

The example below illustrates another benefit of member access control:

{ makeScope, newScope }:

{

  myScope = makeScope newScope (self: with self; {

    _private.stdenv = callPackage ./my-stdenv.nix {};

    # foo uses modified stdenv
    foo = callPackage ./foo.nix {};

    myChildScope = makeScope newScope (self: with self; {
      # bar uses normal stdenv
      bar = callPackage ./bar.nix {};
    });

  });

}

Ability to override ancestor scopes

This feature is easier to explain. If scope attrsets provided access to their ancestors, perhaps via a _parent attribute, ancestors could be modified using their overrideScope attributes. This would allow for precise overriding in deeply-nested scenarios.

[FRidh]

Related issues:

• Add pkgs.overrideWithScope #44196 @nbp
• [RFC] Proposal: Engineering a Common "Plugin" Infrastructure #59344 @deliciouslytyped
• WIP lib: Better extensible attrsets #51213 @Ericson2314
• stage: use fix' to allow deep overriding of pkgs from within Nixpkgs #43828
• pythonPackages: Switch to lib.makeScope for overrideScope support #67422 @infinisil

[infinisil]

Splicing

Currently, scopes created with lib.makeScope are not spliced. If one member of a scope depends on another member of that same scope as a native build input, it cannot resolve that dependency directly through callPackage. As a workaround, I've been using buildPackages to indirectly resolve such dependencies (e.g. referring to buildPackages.myScope.myPackage from within myScope).

I still don't quite get what splicing is and what it's necessary for. What prevents these dependencies from being resolved through callPackage? In nixpkgs nobody is using buildPackages to set nativeBuildInputs either?

[nspin]

Splicing is mentioned but not explained in the manual. I provide a quick explanation here.

• Source: https://github.com/NixOS/nixpkgs/blob/9566ec034b9515ba2487d3625664109b59ca46f1/pkgs/top-level/splice.nix
• Relevant bits of manual:
  • https://nixos.org/nixpkgs/manual/#chap-cross
  • https://nixos.org/nixpkgs/manual/#ssec-stdenv-dependencies

Derivations expressed using stdenv have a build, host, and target platform. These attributes are, respectively, the platform on which a derivation is built, the platform on which it runs, and, for certain sorts of packages (e.g. a compilers), the platform which it targets.

When not cross-compiling, every derivation's build, host, and target platforms are the same. However, when cross-compiling, they differ. For example, a package which is cross-compiled from x86_64-linux to aarch64-linux, has (build, host, target) platforms at (x86_64-linux, aarch64-linux, aarch64-linux).

When I'm cross compiling a package which uses CMake and links against OpenSSL, I need to providemakeDerivation with a CMake derivation at (*, x86_64-linux, aarch64-linux) and an OpenSSL derivation at (x86_64-linux, aarch64-linux, *). Adjacent packages in the top-level Nixpkgs attrset have the same (build, host, target) as the package I'm expressing, so callPackage can provide me with the correct OpenSSL derivation. However, callPackage would provide me with a CMake that runs on aarch64-linux, which is not what I want.

One way to deal with this would be explicit references to buildPackages (e.g. nativeBuildInputs = [ buildPackages.cmake ]). This is a bit of a pain, and isn't convenient to override.

Instead, top-level Nixpkgs attrsets of every relevant (build, host, target) are "spliced" together into one attrset where each package has the extra attribute __spliced, which is an attrset with attributes buildBuild, buildHost, buildTarget, hostHost, hostTarget, and targetTarget. Each of those attributes holds the derivation of that package at the "offset" relative to the original derivation according to its name. The details here are confusing, but the point is, for a spliced package, buildPackages.cmake can be reached by cmake.__spliced.buildHost.

makeDerivation receives spliced packages via each of buildInputs, nativeBuildInputs, depsBuildBuild, etc., and retrieves the correct derivation using the .__spliced attribute.

(Note: currently .__spliced.buildHost and .__spliced.hostTarget are named .nativeDrv and .crossDrv respectively, but that's legacy and will change.)

[Ericson2314]

Wow, it's always impressive when someone has thoroughly reverse engineered your undocumented abomination—great job figuring out splicing!

As you point out the big question is whether we can avoid passing in the scopes own binding, which leads to odd behavior if the scope is, let's say, bound and overriden under a new name.

As Alan Kay might say "...late binding...". We usually solve these sorts of issues by delaying when the not is tied. Unfortunately, the solution here is pretty dirastic: we'd need package sets of functions rather than functions applied to the rest of the set via callPackage. Then in one big pass we'd crawl the thing tying the knot and adding sub package sets to the scope. Instead of functions we could use nix's "functors" so we could tell what was a sub package set and what is a package function.

(I'd write some example code but I'm on my phone.)

[nspin]

For what it's worth, I think splicing is a clever technique!

Thanks for your input. I'll give this some more thought.

[Ericson2314]

hehe it may be clever, but that doesn't make me hate it any less!

Good luck mulling it over :)

[edolstra]

Regarding splicing, I noticed that an evaluation like

NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 nix-instantiate --dry-run ~/Dev/nixpkgs/nixos/release-combined.nix -A nixos.tests.simple.x86_64-linux

calls the function merge in splice.nix 24627 times. The comment in splice.nix "for performance reasons, rather than uniformally splice in all cases, we only do so when pkgs and buildPackages are distinct" suggests that splice.nix should be a no-op when we're not cross-compiling, but apparently it's doing quite a lot. @Ericson2314 Is that expected/intended behaviour?