nix-shell --pure: non-POSIX locales do not work (by default)

[trofi]

TL;DR: glibcLocales seems to be required for many packages on nix to make localisation work at all

How to add them the nix way?

Issue description

Many nixpkgs lack glibcLocales dependency and as a result are subtly broken
in non-POSIX locale support. It looks like NixOS itself somehow avoids it but
not in sandbox (and not in nix on other OSes).

Steps to reproduce

I am happy user of ru_RU locale. Let's see how it should work first (nixos):

# good
$ LANG=ru_RU.UTF-8 date
Чт янв 12 01:28:17 +03 2017

Now how to break it in sandbox:

# bad
[sf@nixos:~]$ nix-shell --pure -p coreutils
[nix-shell:~]$ LANG=ru_RU.UTF-8 date
Thu Jan 12 01:31:22 +03 2017

And fix back:

# good again
[sf@nixos:~]$ nix-shell --pure -p coreutils -p glibcLocales
[nix-shell:~]$ LANG=ru_RU.UTF-8 date
Чт янв 12 01:32:11 +03 2017

The problem happens in other packages and tools: glibc, mc

# glibc's own 'locale -a' binary does not see any locales except C and POSIX onless glibcLocales is loaded.
[sf@nixos:~]$ nix-shell --pure -p glibc --command 'locale -a | wc -l'
2

Similar problem happens in nix installed on Gentoo where I did not find away
to teach 'date' how russian locale at all:

sf@siarheit-glaptop ~/git/nixpkgs $ LANG=ru_RU.UTF-8 date
Thu Jan 12 01:37:32 +03 2017

Nothing helps. Most desperate attempt was:

$ nix-env -iA nixpkgs.coreutils nixpkgs.glibcLocales

replacing old ‘coreutils-8.26’
installing ‘coreutils-8.26’
installing ‘glibc-locales-2.24’
building path(s) ‘/nix/store/83c0mlhpml6jbwhbllx85m6lys773y15-user-environment’
created 898 symlinks in user environment
sf@siarheit-glaptop ~/git/nixpkgs $ LANG=ru_RU.UTF-8 date
Thu Jan 12 01:44:04 +03 2017

I roughly understand why it does not work. My real question is: how to make most tools to see locales?
AFAIU the proper way is to add (optional?) glibcLocales dependency into all the packages:
glibc (for locale), coreutils (for date), mc (for charset collation so editor would be able to work with non-ascii).

Technical details

• System: NixOS or Gentoo + nixpkgs
• Nix version: nix-env (Nix) 1.11.5
• Nixpkgs version: "17.03pre98686.f88e2fb"

Thanks!

[trofi]

Having explored a bit more why nixos works i've found this glibc patch:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/glibc/nix-locale-archive.patch

[sf@nixos:~]$ echo $LOCALE_ARCHIVE
/run/current-system/sw/lib/locale/locale-archive

Basically it provides the following lookup order:

• first env:LOCALE_ARCHIVE_2_11
• then env:LOCALE_ARCHIVE
• then file:/nix/store/m71...155f-glibc-2.24/lib/locale/locale-archive

Thus I have my first workaround:

LOCALE_ARCHIVE=$HOME/.nix-profile/lib/locale/locale-archive

Is it intended search order though?

Suppose user will happen to have 2 binaries in env with different glibc versions: 2.11 and 2.24 without glibcLocales depend. There is no single correct LOCALE_ARCHIVE value.

To make things more robust it would make sense have a fallback-only env override. Say, in following order:

• first env:LOCALE_ARCHIVE_2_11
• then env:LOCALE_ARCHIVE
• then file:/nix/store/m71...155f-glibc-2.24/lib/locale/locale-archive
• [PROPOSED] then env:LOCALE_ARCHIVE_FALLBACK

Then it would be safer to set only env:LOCALE_ARCHIVE_FALLBACK for packages that omit glibcLocales dependency.

Thus my suggestions:

• extend installation doc (https://nixos.org/nixpkgs/manual/) to cover LOCALE_ARCHIVE envvar
• add low-priority LOCALE_ARCHIVE_FALLBACK envvar to glibc patch: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/glibc/nix-locale-archive.patch

WDYT?

[vcunat]

WDYT?

IMO the point of --pure is precisely not to pick up all the stuff you normally have in your environment, including path to whatever locale archive your system is using at that moment.

The package including all theoretically-supported locales is very big, so typically you don't want packages to depend on it (i.e. contain a reference to it).

[trofi]

One clarification: on non-nixos non-POSIX locales do not work at all unless unless user explicitly specifies LOCALE_ARCHIVE env var (it was my initial source of confusion).

Perhaps worth adding it to https://github.com/NixOS/nix/blob/master/scripts/nix-profile.sh.in

[trofi]

Adding @edolstra to validate if it makes sense to add LOCALE_ARCHIVE to nix-profile.sh

[vcunat]

Yes, that sounds good, at a quick glance, though I'm not certain of the top of my head whether the location for OS locale archive is the same on most distros (or easy to find in the script).

[Mic92]

A lot of python tests fail without this change.

[Nadrieril]

(triage) Any progress ? It seems like we are awaiting approval from @edolstra.

[qrilka]

Stumbled upon this seeing bad UTF-8 handling by mc installed with home-manager, fixed with setting $LOCALE_ARCHIVE_2_11 like in https://gist.github.com/peti/2c818d6cb49b0b0f2fd7c300f8386bc3

[qrilka]

also #6878 looks relevant

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[trofi]

From a bit of experience with NixOS I now see that it's somewhat expected for nixpkgs packages not to always work if packages rely on NixOS to provide global variables like locales or runpaths for mesa and friends. Users of nixpkgs without NixOS need to fiddle with environment quite a bit anyway. Locales are not very special here.

Closing as INVALID.