Help relocatability of libraries/executables in the JLL wrappers

[giordano]

We have several cases where we need to set some environment variables at runtime (or using withenv) to relocate the libraries/executables:

• for git we need to set GIT_EXEC_PATH and GIT_SSL_CAINFO, see Git_jll: Can't clone over HTTPS Yggdrasil#284 and Git_jll doesn't work with HTTP remotes? Yggdrasil#452
• for ncurses we need to set TERMINFO or TERMINFINO_DIRS, see Cannot initialize Ncurses in macOS Yggdrasil#455
• for Tk/Tcl we need to set TCL_LIBRARY and TK_LIBRARY, see [TCL/TK] .tcl files not copied to artifacts Yggdrasil#462
• for OpenMPI we need to set OPAL_PREFIX, see openmpi launcher doesn't work Yggdrasil#390
• for Gtk we need to set a bunch of environment variables, see the __init__ function of Gtk.jl
• for Fontconfig we need to set FONTCONFIG_FILE, see the __init__ functions of Fontconfig.jl and Cairo.jl

We usually delegate the Julia packages calling the libraries to set these variables, but this has a couple of drawbacks:

• all packages calling the JLL package need to do it, see the example of Cairo.jl and Fontconfig.jl, both using Fontconfig_jll
• in some cases, the Julia packages doesn't need to know what JLL package it's using. This is for example the case of MPI.jl reported by @simonbyrne in openmpi launcher doesn't work Yggdrasil#390

Maybe we should allow setting these variables directly in the JLL wrappers, to alleviate the burden of the developers of the high-level interface of the package.

The problem is that sometimes these variables need to be set around an executable -- so that we could add these to the already existing wrapper -- but sometimes we need to define the variable globally because it's used by a library, so deciding an interface to do this may not be very easy.

[Keno]

I'm wondering if it doesn't make sense to try and carry patches that make these kinds of shenanigans unnecessary.

[giordano]

In most cases patching the source code might work, but it's more burden on our side to prepare the patches and maintain them in the long term. However, for git we need to set

ENV["GIT_SSL_CAINFO"] = joinpath(dirname(Sys.BINDIR), "share", "julia", "cert.pem")

I don't think we can escape setting some sort of environment variable in this case.

[fingolfin]

It would be really useful to go beyond just allowing to set a few custom variables: how about making it possible to inject some arbitrary extra initialization code? E.g. for libraries like GMP, FLINT and others, one might want to initialize memory management functions. Right now, every caller of such a library has to set these (or hope somebody else did it).

[simonbyrne]

A very simple solution would be a way to specify a custom __init__ function for the _jll files (say as a quote block passed to build_tarballs)

[fingolfin]

Based on a comment by @staticfloat on PR #791, I had a look at which env var values are in fact in use. Here is what I got (based on @giordano's helpful list):

• for git (see Git_jll: Can't clone over HTTPS Yggdrasil#284 and Git_jll doesn't work with HTTP remotes? Yggdrasil#452)
  • GIT_EXEC_PATH = joinpath(Git_jll.artifact_dir, "libexec", "git-core")
  • GIT_SSL_CAINFO = joinpath(dirname(Sys.BINDIR), "share", "julia", "cert.pem")
• for ncurses we need to set one of these (see Cannot initialize Ncurses in macOS Yggdrasil#455)
  • TERMINFO = joinpath(Ncurses_jll.artifact_dir, "share", "terminfo")
  • TERMINFO_DIRS = joinpath(Ncurses_jll.artifact_dir, "share", "terminfo")
• for Tk/Tcl (see [TCL/TK] .tcl files not copied to artifacts Yggdrasil#462)
  • TCL_LIBRARY = joinpath(Tcl_jll.artifact_dir, "lib", "tcl8.6")
  • TK_LIBRARY = joinpath(Tk_jll.artifact_dir, "lib", "tk8.6")
• for OpenMPI (see openmpi launcher doesn't work Yggdrasil#390)
  • OPAL_PREFIX = OpenMPI_jll.artifact_dir
• for Fontconfig we need to set FONTCONFIG_FILE, see the __init__ functions of Fontconfig.jl and Cairo.jl
  • FONTCONFIG_FILE = joinpath(Fontconfig_jll.artifact_dir, "etc", "fonts", "fonts.conf")
• for Gtk we need to set a bunch of environment variables, see the init function of Gtk.jl
  • XDG_DATA_DIRS -- complex value, depends on multiple JLLs
  • GDK_PIXBUF_MODULE_FILE -- depends on a mutable artifact
  • GDK_PIXBUF_MODULEDIR
• Xorg_xkeyboard_config
  • XKB_CONFIG_ROOT (but only on Linux and FreeBSD?)

Based on the above, it seems that setting env variables to a path relative to the artifact_dir would cover most
cases; the exceptions in the list above being

• GIT_SSL_CAINFO, which is set to a path relative to the Julia install path
• Gtk.jl, which however all in all seems out of scope here anyway

There is of course a good chance that there are other relevant env vars out there which
also don't fit into the "relative to artifact_dir" scheme, but it still seems as if
adding a way to specify such env vars be set in JLLs would be useful.

That said, it might not be completely without issue: e.g. setting TERMINFO could have
ripple effects: it would e.g. affect subprocesses launched by Julia, which then might
break (because those subprocesses might run executable which are linked against a different
copy of ncurses and which expect require a different TERMINFO value to work right).
So I am not 100% sure how to resolve that (or perhaps nothing needs to be resolved and
I am just overthinking it?)

[vchuravy]

I am encountering this in PMIx_jll and pprte_jll which both copy OpenMPI MCA approach. It is again an environment variable relative to the artifact_dir.

[simonbyrne]

init_block was added in #791, it appears to be used in a couple of recipes in Yggdrasil.