Nix cannot build aarch64-darwin Haskell binaries that don't require a nix store

[ParetoOptimalDev]

Describe the bug

When you download the ghc compiler with something like ghcup and compile a binary from aarch64-darwin, it produces a binary that links to normal apple system libraries.

Replicating this common use case doesn't seem possible on Nix due to the lack of truly static libraries on darwin.

Steps To Reproduce

Steps to reproduce the behavior:

$ nix build nixpkgs#pkgsStatic.shellcheck --print-out-paths
$ otool -L /path/to/that/shellcheck
... dynamic links into nix store aarch64-darwin users without nix won't have ...

Expected behavior

For pkgsStatic.shellcheck or Haskell programs using pkgsStatic.haskell.packages.ghcx to run on aarch64-darwin because they produce binaries that don't link into the nix store.

Notify maintainers

@cdepillabout @angerman @sternenseemann

Metadata

Apple m1 that is aarch64-darwin

TODO get actual nix-info output:

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-anyone-distributing-aarch64-darwin-haskell-binaries-sucessfully/27548/2

[angerman]

If you link dynamically you end up with all the libraries from Nixpkgs, which is to be expected. Now you can use install_name_tool, to copy the closure of dynamic library dependencies out of the nix store and update the relevant library identification paths. Using RPATH's appropriately you can build something where the executable and the dynamic libraries are next to each other.

Option two, you pass the relevant static flags, and ensure you link everything statically. This would give you a single executable similar to linux musl static builds.

Now, macOS (as BSDs) do not have stable sys call interfaces, and as such your "static" binary will always link dynamically to the libc on the respective system. You can simply patch up the linking to point to the system provided libc. The ones in nix would usually track the system provided close enough to just make that work.

In fact that is how I use nix as a substrate to build (mostly) statically linked macOS binaries.

[alyssais]

Now, macOS (as BSDs)

Nitpick: BSDs are all different operating systems, so you can't generalise about them. NetBSD has a stable syscall interface.

[ParetoOptimalDev]

Thank you for your help. I'm very new to both the static binaries and macOS world though, so I have some further questions.

If you link dynamically you end up with all the libraries from Nixpkgs, which is to be expected. Now you can use install_name_tool, to copy the closure of dynamic library dependencies out of the nix store and update the relevant library identification paths. Using RPATH's appropriately you can build something where the executable and the dynamic libraries are next to each other.

I'lll look more into RPATH and install_name_tool to try and understand how to do this.

Option two, you pass the relevant static flags, and ensure you link everything statically. This would give you a single executable similar to linux musl static builds.

Looking up a bit on this method I found haskell4nix's "creating statically linked binaries" that says:

It’s important to realize, however, that most system libraries in Nix are built as shared libraries only, i.e. there is just no static library available that Cabal could link!

I guess you somehow point to the macOS ones?

Now, macOS (as BSDs) do not have stable sys call interfaces, and as such your "static" binary will always link dynamically to the libc on the respective system. You can simply patch up the linking to point to the system provided libc. The ones in nix would usually track the system provided close enough to just make that work.

Thanks for the clarification. I have heard "static linking doesn't work on macOS" but never gotten a good explanation of what exactly that means. I believe they were referencing what you are talking about here.

In fact that is how I use nix as a substrate to build (mostly) statically linked macOS binaries.

Do you have any public examples of code doing this I could try building/understanding?

[ParetoOptimalDev]

I believe I found an example of your first approach:

{
   # ...
        # "static" binary for distribution
        # on linux this is actually a real fully static binary
        # on macos this has everything except libcxx, libsystem and libiconv
        # statically linked. we can be confident that these three will always
        # be provided in a well known location by macos itself.
        hevmRedistributable = let
          grep = "${pkgs.gnugrep}/bin/grep";
          otool = "${pkgs.darwin.binutils.bintools}/bin/otool";
          install_name_tool = "${pkgs.darwin.binutils.bintools}/bin/install_name_tool";
        in if pkgs.stdenv.isLinux
        then pkgs.haskell.lib.dontCheck hevmUnwrapped
        else pkgs.runCommand "stripNixRefs" {} ''
          mkdir -p $out/bin
          cp ${pkgs.haskell.lib.dontCheck hevmUnwrapped}/bin/hevm $out/bin/

          # get the list of dynamic libs from otool and tidy the output
          libs=$(${otool} -L $out/bin/hevm | tail -n +2 | sed 's/^[[:space:]]*//' | cut -d' ' -f1)

          # get the paths for libcxx and libiconv
          cxx=$(echo "$libs" | ${grep} '^/nix/store/.*-libcxx')
          iconv=$(echo "$libs" | ${grep} '^/nix/store/.*-libiconv')

          # rewrite /nix/... library paths to point to /usr/lib
          chmod 777 $out/bin/hevm
          ${install_name_tool} -change "$cxx" /usr/lib/libc++.1.dylib $out/bin/hevm
          ${install_name_tool} -change "$iconv" /usr/lib/libiconv.dylib $out/bin/hevm
          chmod 555 $out/bin/hevm
        '';
# ...
}

https://github.com/ethereum/hevm/blob/main/flake.nix#L99