{lang}[foss/2020a] Julia v1.5.1 #11180

boegel · 2020-08-28T21:43:54Z

(created using eb --new-pr)

boegel · 2020-08-28T22:00:13Z

I may rework this into a custom easyblock, it's a bit much in an easyconfig.

Still need to look into support for installing Julia packages as extensions, which will involve an easyblock anyway...

boegel · 2020-08-28T22:47:57Z

Test report by @boegel
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in this PR)
node3404.kirlia.os - Linux centos linux 7.8.2003, x86_64, Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz (cascadelake), Python 2.7.5
See https://gist.github.com/5b994ef1af31d41e26dc740be0dc7b9f for a full test report.

Micket · 2020-08-28T23:12:40Z

easybuild/easyconfigs/j/Julia/Julia-1.5.1-foss-2020a.eb

+    ('cURL', '7.69.1'),
+    ('zlib', '1.2.11'),
+    ('SuiteSparse', '5.7.1', '-METIS-5.1.0'),
+]


Should we list deps we leave to Julia to do it's own thing with? Based on my incomplete, broken, attempt, I identified:
LLVM
OpenLibm
utf8proc
libuv
libgit2
p7zip
dSFMT

We could, see https://github.com/JuliaLang/julia/blob/master/doc/build/build.md#required-build-tools-and-external-libraries (some are only needed on Windows)

We could probably could use these external as well if we want:

OpenLibm & utf8proc: No patches (though this is more or less just Julia specific anyway)

p7zip: Patches seem to CVE related, nothing julia specific

libunwind: Patches seem to be ARM specific

libgit2: Required patch to make ssh-agent problems non-fatal. Probably required, but I suppose it wouldn't hurt elsewhere, so we could possibly include that. It's been proposed for inclusion upstream (but without response from git devs).

Note: As Julia is transitioning more towards their BinaryBuilder tool where Julia modules can depend on pre-built "Artifacts", called "JLL"s, there is some potential conflict there if we try to control the versions to much and pulling in other libraries. Hard to say how it's all going to pan out.

@Micket "libunwind: Patches seem to be ARM specific" is interesting. Could you expand on this?
We see Mesa and probably more stuff fail on arm due to libunwind: easybuilders/easybuild-easyblocks#2153. Is libunwind on arm something we can fix? I see arch and others ship it and use it for Mesa: https://archlinuxarm.org/packages/aarch64/mesa

@terjekv This comment was just based on the patchset i found in julia: https://github.com/JuliaLang/julia/blob/master/deps/patches/ (loooks like a very very minor patch though). Also the yggdrasil package stuff: https://github.com/JuliaPackaging/Yggdrasil/tree/master/L/LibUnwind/bundled/patches. That's all i know.

Micket · 2020-08-28T23:13:52Z

easybuild/easyconfigs/j/Julia/Julia-1.5.1-foss-2020a.eb

+    ('Python', '3.8.2'),
+    ('Perl', '5.30.2'),
+    ('pkg-config', '0.29.2'),
+    ('PCRE2', '10.34'),


Is PCRE2 really a build dep? I suppose, maybe it statically links? But we use PCRE2 as a runtime dep in R for example.

I saw it listed in the Makefile as a build dep.

Micket · 2020-08-28T23:15:00Z

easybuild/easyconfigs/j/Julia/Julia-1.5.1-foss-2020a.eb

+    ('pkg-config', '0.29.2'),
+    ('PCRE2', '10.34'),
+    ('patchelf', '0.12'),
+]


In my incomplete attempt, I also marked gawk as required as a builddep for some reason. Maybe?

Jup, that's listed as a required build dep, but I didn't include it here, we should.

Micket · 2020-08-29T23:44:08Z

easybuild/easyconfigs/j/Julia/Julia-1.5.1-foss-2020a.eb

+    ('cURL', '7.69.1'),
+    ('zlib', '1.2.11'),
+    ('SuiteSparse', '5.7.1', '-METIS-5.1.0'),
+]


We could probably could use these external as well if we want:

OpenLibm & utf8proc: No patches (though this is more or less just Julia specific anyway)

p7zip: Patches seem to CVE related, nothing julia specific

libunwind: Patches seem to be ARM specific

libgit2: Required patch to make ssh-agent problems non-fatal. Probably required, but I suppose it wouldn't hurt elsewhere, so we could possibly include that. It's been proposed for inclusion upstream (but without response from git devs).

Note: As Julia is transitioning more towards their BinaryBuilder tool where Julia modules can depend on pre-built "Artifacts", called "JLL"s, there is some potential conflict there if we try to control the versions to much and pulling in other libraries. Hard to say how it's all going to pan out.

Micket · 2020-08-29T23:44:31Z

easybuild/easyconfigs/j/Julia/Julia-1.5.1-foss-2020a.eb

+    ('zlib', '1.2.11'),
+    ('SuiteSparse', '5.7.1', '-METIS-5.1.0'),
+]
+


Add osdeps for libssh2 (and mbedtls?)

@boegel ping?

bartoldeman · 2020-10-16T15:19:42Z

Here is an example of weird things that happen with Arpack and this Julia:

$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.1 (2020-08-25)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> import Pkg

julia> Pkg.add("Arpack")
   Updating registry at `~/.julia/registries/General`

   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
  Installed OpenBLAS_jll ───────────────── v0.3.10+0
  Installed Arpack_jll ─────────────────── v3.5.0+3
  Installed CompilerSupportLibraries_jll ─ v0.3.3+0
  Installed Arpack ─────────────────────── v0.4.0
Downloading artifact: OpenBLAS
Downloading artifact: Arpack
Downloading artifact: CompilerSupportLibraries
Updating `~/.julia/environments/v1.5/Project.toml`
  [7d9fca2a] + Arpack v0.4.0
Updating `~/.julia/environments/v1.5/Manifest.toml`
  [7d9fca2a] + Arpack v0.4.0
  [68821587] + Arpack_jll v3.5.0+3
  [e66e0078] + CompilerSupportLibraries_jll v0.3.3+0
  [4536629a] + OpenBLAS_jll v0.3.10+0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode

julia> using Arpack
[ Info: Precompiling Arpack [7d9fca2a-8960-54d3-9f78-7d1dccf2cb97]

julia> using LinearAlgebra

julia> A = Diagonal(1:4);

julia> eigs(A, nev = 2, which=:SM);

signal (11): Segmentation fault
in expression starting at none:0
page_metadata at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/gc.h:448 [inlined]
jl_gc_pool_alloc at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/gc.c:1153
jl_gc_alloc_ at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/julia_internal.h:277 [inlined]
jl_gc_alloc at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/gc.c:3150
_new_array_ at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/array.c:94 [inlined]
_new_array at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/array.c:162 [inlined]
jl_alloc_array_1d at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/array.c:433
jl_get_excstack at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/stackwalk.c:341
#catch_stack#47 at ./error.jl:144
catch_stack at ./error.jl:144 [inlined]
catch_stack at ./error.jl:144 [inlined]
eval_user_input at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:146
repl_backend_loop at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:195
start_repl_backend at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:180
#run_repl#37 at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:292
run_repl at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
#806 at ./client.jl:399
jfptr_YY.806_61660 at /home/oldeman/.local/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/julia/1.5.1/lib/julia/sys.so (unknown line)
jl_apply at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/julia.h:1690 [inlined]
do_apply at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/builtins.c:655
jl_f__apply_latest at /tmp/oldeman/avx2/Julia/1.5.1/foss-2020a/julia-1.5.1/src/builtins.c:705
#invokelatest#1 at ./essentials.jl:710 [inlined]
invokelatest at ./essentials.jl:709 [inlined]
run_main_repl at ./client.jl:383
exec_options at ./client.jl:313
_start at ./client.jl:506
jfptr__start_55536 at /home/oldeman/.local/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/julia/1.5.1/lib/julia/sys.so (unknown line)
unknown function (ip: 0x4017c2)
unknown function (ip: 0x4013f2)
__libc_start_main at /cvmfs/soft.computecanada.ca/gentoo/2020/lib64/libc.so.6 (unknown line)
unknown function (ip: 0x401499)
Allocations: 12245049 (Pool: 12241546; Big: 3503); GC: 13
Segmentation fault

Note the

  Installed OpenBLAS_jll ───────────────── v0.3.10+0
  Installed Arpack_jll ─────────────────── v3.5.0+3
  Installed CompilerSupportLibraries_jll ─ v0.3.3+0
  Installed Arpack ─────────────────────── v0.4.0

it in fact installs another OpenBLAS, in my case ~/.julia/artifacts/a599e8c9eff1f2d74ebe955ff5d78c4da00c1b69/lib/libopenblas64_.0.3.10.so

we have a GCCcore-9.3.0 Julia installed (https://github.com/ComputeCanada/easybuild-easyconfigs/blob/computecanada-master/easybuild/easyconfigs/j/Julia/Julia-1.5.2-GCCcore-9.3.0.eb) and it doesn't have this issue.. it still insists on downloading a new openblas library but it's now compatible with what Julia was compiled with in the first place.

(Yes -- the issues with Julia are a bit like those with Conda, just to a somewhat lesser extend)

akesandgren · 2023-10-23T08:42:26Z

Closing this PR, since it (only) uses deprecated toolchain versions,
see https://docs.easybuild.io/policies/toolchains .

adding easyconfigs: Julia-1.5.1-foss-2020a.eb

5892685

boegel added the update label Aug 28, 2020

boegel added this to the 4.x milestone Aug 28, 2020

Micket reviewed Aug 28, 2020

View reviewed changes

Micket reviewed Aug 29, 2020

View reviewed changes

bartoldeman mentioned this pull request Oct 16, 2020

{lang}[GCCcore/9.3.0] Julia v1.5.2 #11522

Closed

akesandgren closed this Oct 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{lang}[foss/2020a] Julia v1.5.1 #11180

{lang}[foss/2020a] Julia v1.5.1 #11180

boegel commented Aug 28, 2020

boegel commented Aug 28, 2020

boegel commented Aug 28, 2020

Micket Aug 28, 2020

boegel Aug 29, 2020

Micket Aug 29, 2020 •

edited

Loading

terjekv Oct 21, 2020

Micket Oct 21, 2020

Micket Aug 28, 2020

boegel Aug 29, 2020

Micket Aug 28, 2020

boegel Aug 29, 2020

Micket Aug 29, 2020 •

edited

Loading

Micket Aug 29, 2020

verdurin Sep 9, 2020

bartoldeman commented Oct 16, 2020

akesandgren commented Oct 23, 2023

{lang}[foss/2020a] Julia v1.5.1 #11180

{lang}[foss/2020a] Julia v1.5.1 #11180

Conversation

boegel commented Aug 28, 2020

boegel commented Aug 28, 2020

boegel commented Aug 28, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Micket Aug 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Micket Aug 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bartoldeman commented Oct 16, 2020

akesandgren commented Oct 23, 2023

Micket Aug 29, 2020 •

edited

Loading

Micket Aug 29, 2020 •

edited

Loading