Creating a binary that links against Julia

[fingolfin]

For various reasons, I need to create a binary build for a binary that links against Julia resp. libjulia (to interact with the Julia GC, but that should be irrelevant). This requires Julia >= 1.2

My problem is that in the build environment, there is no Julia installed, and in particular no Julia headers, so I can't do that. Is there any way around this? I realize this is probably a relatively rare thing to do, so perhaps BinaryBuilder does not cover it yet? If it does not, is there are at least principle interest in supporting this? (If so, I'd be inclined to try and help implement it).

Not sure if it is relevant, but note that I am currently using BinaryBuilder 0.1.4, because that's the version being shipped. But I see that there are tons of changes in this repository compared to that, which makes me wonder what I should be using.

[giordano]

You'd probably need to use Julia as dependency. There is a builder in Yggdrasil, but it probably works for a handful of platforms and it's for Julia 1.0.3.

In general, if you're looking for a builder you should checkout Yggdrasil, and contribute there a new one

[fingolfin]

That Julia builder looks like it builds another copy of Julia, though? That's not sufficient; it is vital to link against the same Julia that is loading "my" binary.

Yggdrasil looks like it doesn't play with the current release of BinaryBuilder (0.1.4) though, or does it?

[giordano]

Yggdrasil looks like it doesn't play with the current release of BinaryBuilder (0.1.4) though, or does it?

Yeah, sorry, forgot to reply on this. Yggdrasil is using master version of BinaryBuilder.jl, which requires Julia 1.3-rc4 or later

[barche]

That Julia builder looks like it builds another copy of Julia, though? That's not sufficient; it is vital to link against the same Julia that is loading "my" binary.

Why is that, exactly? We're using those binaries to build the C++ part of CxxWrap, and since it is using dynamic linking it works fine with libjulia 1.0 - 1.3, even if at compile time only the 1.0 version was available.

[fingolfin]

Sorry, with "link" I was referring to "dynamic link at runtime", not "creating the binary during the build stage".

I didn't realize that I could use that "Julia builder" and then later on link against the actual "main" Julia. If that's possible, then this does again sound like a potentially viable venue -- thank you for the hint! I'll take a look at how CxxWrap does it.

However, we do use APIs that only were added in Julia 1.1 (via a PR by us), so we need that at the latest. We also access relatively low-level parts of the GC directly, so the question is whether that part of the ABI stayed compatible from Julia 1.1 to 1.2 resp. 1.3.

[fingolfin]

All in all, I am still very much confused what version of BinaryBuilder.jl to target. The one in master seems to be radically different from the 0.1.4 release; at the same time, it also seems to be heavily in flux? Should I still use it?

[giordano]

All in all, I am still very much confused what version of BinaryBuilder.jl to target. The one in master seems to be radically different from the 0.1.4 release;

The latest stable version is very old and almost no one is using it. The code on master is very much different from that version, but the only big differences in terms of builders are how to specify products (they're now a Vector, instead of a function returning a Vector) and dependencies (just use the name of a JLL package, instead of the URL of a remote build.jl script).

at the same time, it also seems to be heavily in flux?

Most of the new development happened in PR #441, but after that PR has been merged the code has been quite stable. Many of the changes that we had since then are about tweaking CI in Yggdrasil and don't affect users.

Should I still use it?

Yes, as long as you're willing to use Julia 1.3.0-rc4 or later.

[barche]

However, we do use APIs that only were added in Julia 1.1 (via a PR by us), so we need that at the latest. We also access relatively low-level parts of the GC directly, so the question is whether that part of the ABI stayed compatible from Julia 1.1 to 1.2 resp. 1.3.

Yes, updating the builder to get all the new point releases in should fix that. I'm kind of surprised we didn't hit problems yet with CxxWrap, so I'd probably also use the updated versions when available just to be safe.

[fingolfin]

Is there a way to specify multiple versions in a builder? Or does this require duplicating the builder (possible with an include to reduce code duplication)?

I noticed that the Julia builder has lots of commented out dependencies. Wonder what the story is there...

[barche]

I think the best way would be to sequentially generate new versions of the builder, the JLL packages that are generated are versioned and can be referred to by version in the build_tarballs.jl dependencies section. The version is also in the Yggdrasil release download path, so you could craft a legacy script for Julia versions before 1.3 that still use the binaries built using the artifacts system.

A nice overview of the new system is here: https://julialang.org/blog/2019/11/artifacts

[fingolfin]

Ok, that sounds interesting - but is it limited to Julia 1.3, or available in 1.1 & 1.2?

[barche]

As far as I understand, you can build the 1.1 and 1.2 binaries on Yggdrasil, but you will need to manually add a build.jl to download them on those versions, since it seems the new BinaryBuilder doesn't generate the build.jl anymore. The generated files end up in a repository in https://github.com/JuliaBinaryWrappers

[fingolfin]

Note that I actually need this for specific Julia versions; experimentally, binaries using some of the lower level Julia "APIs" (they are not really official APIs, I am afraid, just internal parts of the Julia kernel) seem to differ enough at times to make a binary built against, say, Julia 1.2 crash with 1.3 and vice versa.

[fingolfin]

So it seems @barche solved this for CxxWrap for now by using ArchiveSource to download Julia 1.3.1 / 1.3.0 binaries inside of L/libcxxwrap_julia/build_tarballs.jl. For macOS and Windows, binaries from https://github.com/Gnimuc/JuliaBuilder are used (see also JuliaPackaging/Yggdrasil#321)

That's an interesting approach, and I will try it out. Though it won't solve my use case completely (at least not with extra effort, as there the Julia version also affects my binaries... Perhaps I can solve this by downloading and installing multiple Julia versions when building, and then building my package N times, once for each supported Julia version, and bundling this together into one binary blob... (I won't really need N versions of the code, luckily: all but one C/C++ source file are independent of the Julia version; I could link them all in a single shared library, and then have N shared libs which have the Julia version as part of their SONAME, and which contain the code from that one remaining C file, plus a link to the shared lib with everything else).

Of course that also means I have to make sure to quickly update it when new (major) Julia releases are made... sigh

A simpler variant would be to make a binary build which only consists of this shared library that links in all but one file; and then compile that one file on the user's machine. Alas, if I do that, I really could just as well compile everything on the user's machine, and don't both with a binary builder, as the time savings would be minuscule (the whole project compilers in 20-30 seconds on a semi-current laptop), and all the hassle with building on the user's machine would still be there, and it's not really more or less work with 1 vs. 100 C files...

[staticfloat]

Just BTW, we have a Julia_jll now.

[fingolfin]

Thanks for the heads up (though I was aware already). Unfortunately it does not yet solve the issue for me, as there are binary incompatibilities in the low-level Julia kernel "interface" between e.g. Julia 1.2 and 1.3 (I have not yet tested with 1.3 vs. 1.4).

To be clear: I put "interface" in quotation part because what we use is not really part of any official Julia kernel API/ABI, I think, so I want to stress that this is not a complaint, merely an observation -- but the thing is, we are using the Julia GC instead of our own GC for this project (to facilitate better integration of Julia and GAP objects) and thus e.g. need to interact with TLS. So it's not surprising somewhere something in the Julia kernel changes from time to time that means we have to recompile. On the upside, the code which depends on Julia is restricted to a single C source file with just ~1000 lines of text (including comments), so it's not too bad. I've even thought about just recompiling that file -- but then the user still needs a working C compiler and Julia header files, so not that much is gained... sigh

Anyway, that means that using Julia_jll is not quite enough for us -- we'd still need a way to make the JLL choose the dylib it loads not just on the arch, but also on the version of Julia (and of course then we also need to build the relevant binaries against each supported Julia version). That should be doable with a customized version of the code generating JLLs (I don't think it'd be useful for the general public, so I don't think trying to get this into BinaryProvider/BinaryBuilder/Yggdrasil would make sense).

Alternatively (and preferably), perhaps we can figure out a way to avoid or workaround this binary incompatibility; but for that, we first need to figure out what exactly causes it, something I simply didn't get around yet (it wasn't very important given that even knowing it would not have solved anything; but now that Julia_jll is there, it is of interest again). Perhaps @rbehrends can help me with that.

[staticfloat]

@fingolfin is your issue not solved by creating a build recipe that links against a particular version of Julia, then building that recipe for all versions of Julia_jll? (Right now that only includes 1.4, but it will soon contain 1.5)

E.g. you can have a LibFoo_LinksAgainstJulia_jll, and release many simultaneous versions of it, one that is restricted to be compatible with Julia 1.4.X, one that is restricted to be compatible with Julia 1.5X, etc.... It's a pain, but it should work.

[fingolfin]

Yes that ought to work. I wasn't aware this was possible

[fingolfin]

@staticfloat I have two questions on this (well, probably more once I actually try doing it):

1. How can/should I instruct BinaryBuilder to emit e.g. 'julia = ~1.4 in the generated Project.toml? I tried to use a Dependency with PackageSpec(name="julia", version="~1.4") but that gives an error. After reading the Pkg.jl source code, I found the undocumented function Pkg.Types.semver_spec, so I guess I could use PackageSpec(name="julia", version=Pkg.Types.semver_spec("~1.4")) but I wonder if there is a better, documented and sanctioned way to do this?

2. Can I implement these package variants similar to what e.g. Coint-OR does, i.e., have on Yggdrasil a subdir G/GAP/ with a file common.jl which then is included by G/GAP/GAP-julia1.3/build_tarballs.jl and so on? The build_tarballs.jl then really only would contain the package name and the Julia version and then include("../common.jl") (I am not sure if the Azure Pipeline scripts handling updates would work with that, though; so perhaps for updates I would have to make some trivial modification to all G/GAP/*/build_tarballs.jl files, but that's of course no problem).

[staticfloat]

1. We are going to need to teach BinaryBuilder to treat the compat bounds on Julia_jll as applying to julia. Right now, it's always setting it to 1.0, but that should not be too difficult to change. You'll need to thread the Julia_jll dependency in through functions until it reaches here: https://github.com/JuliaPackaging/BinaryBuilder.jl/blob/master/src/AutoBuild.jl#L1447

Note that the dependencies that are being passed in are excluding BuildDependencies, of which Julia_jll should be one. I suggest adding a new kwarg to register_jll() called julia_dependency that defaults to something like Pkg.Types.PackageSpec(name="julia", version=Pkg.Types.VersionSpec("1.0")). Then, you can scan the list of dependencies, see if there is a Julia_jll anywhere, and if there is, you can extract its version spec, then insert it into the julia_dependency kwarg being passed to register_jll().

2. Yes, I think this would be a great way to do it.

[fingolfin]

Any particular reason why julia_dependency should be a PackageSpec and not just a VersionSpec?

I'll look into make a PR for BinaryBuilder.jl then.

BTW, we figured out what causes the dependency on the specific Julia version in our C code (which was written to integrate with the Julia GC): Unfortunately we need to access part of Julia's TLS, and the members of jl_ptls_t are changing their positions often (e.g. from 1.2 to 1.3 and also from 1.4 to 1.5; by chance not from 1.3 to 1.4). I've been working on reducing these, but we probably will still end up submitting a PR for Julia tomorrow that adds 1-3 tiny getter functions which would allow us to remove the dependency on the Julia version once those APIs are available, so that the number of GAP_jll variants wouldn't keep growing indefinitely.

[staticfloat]

Any particular reason why julia_dependency should be a PackageSpec and not just a VersionSpec?

Nope, do whatever makes the most sense to you.

but we probably will still end up submitting a PR for Julia tomorrow that adds 1-3 tiny getter functions which would allow us to remove the dependency on the Julia version once those APIs are available

Yep, that seems like the proper fix.

[fingolfin]

I've submitted JuliaLang/julia#36064 with the accessor functions. Any feedback welcome. I'll now look into adding Julia versioning to BinaryBuilder

[fingolfin]

We have julia_compat now and libjulia_jll, so I am happy to close this now.