Package duplicate dependencies and overriding nixpkgs pkgs

[CMCDragonkai]

I found that when attempting to build a complex Python application where itself and its dependencies uses matplotlib, it is possible to get this problem:

Found duplicated packages in closure for dependency 'matplotlib':
  matplotlib 2.1.2 (/nix/store/38fz59rmqxq1l43y0k2v40d36vv6hr5w-python3.6-matplotlib-2.1.2/lib/python3.6/site-packages)
  matplotlib 2.1.2 (/nix/store/bcrnikk3vfsl7vxzgvhjnlw7rlc72kkj-python3.6-matplotlib-2.1.2/lib/python3.6/site-packages)

Package duplicates found in closure, see above. Usually this happens if two packages depend on different version of the same dependency.

The main reason is when you use:

matplotlib.override { enableQt = true; }

In your propagatedBuildInputs.
The problem is that your dependencies may rely on the default pkgs.matplotlib, whereas your top level package may actually expect to use the qt backend.

I found that the solution to this was to create an override at the nixpkgs level:

  let
    pkgs = (import <nixpkgs>) {
      config = {
        packageOverrides = pkgs: {
          python36 = pkgs.python36.override {
            packageOverrides = self: super: {
              matplotlib = super.matplotlib.override { enableQt = true; };
            };
          };
        };
      };
    };
  in
    with pkgs;

From there, every subsequent expression will reuse the same pkgs, and all packages will be using the overridden matplotlib.

My question here is, is this way that's expected to work? Is there no better way to do this? It seems overly verbose to achieve this. Furthermore, this requires you to have the nixpkgs function, instead of the pkgs attribute. I could not find a way to override the pkgs attribute instead of passing it as a the config attribute to the nixpkgs function.

[CMCDragonkai]

This PR seems related: #44196

[FRidh]

I found that when attempting to build a complex Python application where itself and its dependencies uses matplotlib, it is possible to get this problem:

What does the dependency tree look like?

The problem is that your dependencies may rely on the default pkgs.matplotlib, whereas your top level package may actually expect to use the qt backend.

Right, that's quite common and why we put this check in here, just to make sure you don't accidentally end up with multiple versions in a closure. Python doesn't support multiple versions of a package, so one should indeed override the value for the whole package set.

My question here is, is this way that's expected to work? Is there no better way to do this?

Almost. This is documented in the Nixpkgs manual, so I am not going to repeat it here.

It seems overly verbose to achieve this.

Yes, it is. Maybe it will get better with #44196.

[CMCDragonkai]

@FRidh thanks for the response.

I did find that PR.

Just regarding you saying there is an almost better way to do it, are you referring to:

{
  pkgs ? import ./pkgs.nix
}:
  let
    pkgs_ = pkgs // {
      config = pkgs.config // {
        packageOverrides = pkgs: {
          python36 = pkgs.python36.override {
            packageOverrides = self: super: {
              matplotlib = super.matplotlib.override { enableQt = true; };
            };
          };
        };
      };
    };
  in
    with pkgs_;

Because I find that my default.nix takes a pkgs instead of the nixpkgs function, and I wanted to override the pkgs without having to reapply the nixpkgs function.

[CMCDragonkai]

Oh that method doesn't actually work.

I can't find a way to override the package set, when I already have the package set.

[CMCDragonkai]

Here's what I would want ideally.

A default.nix:

{ pkgs ? import ./pkgs.nix }:
  let
    pkgs_ = pkgs.override { ... };
  in
    # do stuff with pkgs_ including passing the pkgs_ into subderivations
    # import ./someotherderivation.nix { pkgs = pkgs_; }

Where the pkgs.nix is:

import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/650d8c93a16333e9390db0269327bd2d47f02fc7.tar.gz) {}

But since I cannot find a way to override while having pkgs, the only working example I can get is something that is passed as a parameter to the nixpkgs function:

{ pkgs ? import ./nixpkgs.nix { config = import ./overrides.nix; } }:
  # do stuff with pkgs

Where the nixpkgs.nix is:

import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/650d8c93a16333e9390db0269327bd2d47f02fc7.tar.gz)

The problem with the above is that downstream derivations cannot pass their own pkgs without losing this specific derivation's overrides. Essentially the overrides becomes non-composable.

[CMCDragonkai]

I found that you can use pkgs.path. This was not recommended due to double imports. But appears to be the only way to propagate pkgs itself.