A safer inNixShell

[roberth]

Nix sets IN_NIX_SHELL so that Nix expressions and scripts can do something else when they are requested for a nix-shell. This works well when only one derivation does this, but, because it is a global variable, dependencies of the shell derivation will respond to this too.

This has caused unexpected behavior on countless occasions.

For the usual goal of dual-purposing a default.nix, you only want to influence the invocation of default.nix itself. It seems the we're in luck, because we do actually have a mechanism for that: autoCallFunction.

I propose to change nix-shell to always add the effect of --arg inNixShell true.
This way, Nix expressions change their behavior based on a variable that isn't forced into dependencies. It will allow Nixpkgs to migrate away from the environment variable and put an end to those impure surprises.

Refs #2797, NixOS/nixpkgs#9438
Another broken use case is extending haskellPackages with cabal2nix generated files.

[matthewbauer]

Getting rid of "IN_NIX_SHELL" usages is definitely good, but having different behavior for "nix-shell" and "nix-build" is itself an impurity (or at least a surprising inconsistency). It would be nice if we could rework things to not require knowing whether we are in a nix-shell or not. So perhaps this would work as a fix for now provided pureEval = false.