Rethink package cache

[maleadt]

For performance reasons, we want to cache ~/.julia/compiled across the package tests for a single Julia build (this reduces PkgEval run time with multiple hours). This implies that we also need to cache ~/.julia/packages, because otherwise installing packages afresh results in an mtime that's more recent than the compilecache, invalidating the precompilation file.

However, there's a couple of problems with caching ~/.julia/packages. First of all, it's prone to corruption (as we've seen a couple of times in the past months). Secondly, caching packages that mutate their files introduces nondeterminism. The prime example of that is packages with a deps/build.jl -- once we've cached a package, the build step won't re-run. That's problematic with packages like Conda.jl which set-up ~/.julia/conda during their build step.

One solution that comes to mind is to cache and verify in two steps. Layered depots are great for that:

• mount the cache at /usr/share/julia/packages
• install new packages in ~/.julia/packages
• after testing, remove uncacheable packages (modified tree-hash, or containing a deps/build.jl)
• finally, sync ~/.julia/packages to /usr/share/julia/packages

Sadly, there's something breaking this approach: precompilation images contain the path to the sources they compiled, and get invalidated when that changes: https://github.com/JuliaLang/julia/blob/d12ac369a4be6ae7833fb95c644cea4531764e31/base/loading.jl#L2247-L2250

Any thoughts to get out of this situation? I was thinking about introducing an env var that disables the above check (or even rewriting .ji files, but that seems tricky). Or does anybody have any other solutions in mind?

cc @vtjnash @KristofferC

[DilumAluthge]

path to the sources they compiled, and get invalidated when that changes:

Would this be fixed by JuliaLang/julia#45215?

[maleadt]

Maybe, I'm not sure what would happen when precompiling a package from ~/.julia/packages that depends on a previously-cached package from the global depot at /usr/share/julia (i.e. which paths exactly are recorded in the ji files). It probably would, though.

[Keno]

Since we're running inside a sandbox anyway, could you just use an overlayfs?

[maleadt]

Ah yes that's a good thought. We don't have the permissions to do so anymore once in the sandbox, so that would require some plumbing to get Sandbox.jl to do that while setting-up the environment.