Alternating pass/fail for Mamba.jl

[brian-j-smith]

As of late, Mamba.jl seems to be alternating between passing and failing on the PackageEvaluator tests (see partial log below). The failures appears to be due to a LoadError: ArgumentError: Module DualNumbers not found in current path. DualNumbers is not a direct dependency of Mamba. Any thoughts on what a fix might be? Thanks!

Version and Status History
Julia v0.4

2016-09-28 to 2016-09-29, v0.9.2, Tests pass.
2016-09-22 to 2016-09-27, v0.9.2, Tests fail.
2016-09-21 to 2016-09-21, v0.9.2, Tests pass.
2016-09-12 to 2016-09-20, v0.9.2, Tests fail.
2016-09-11 to 2016-09-11, v0.9.2, Tests pass.
...

Julia v0.5

2016-09-27 to 2016-09-29, v0.10.0, Tests fail.
2016-09-26 to 2016-09-26, v0.10.0, Tests pass.
2016-09-24 to 2016-09-25, v0.10.0, Tests fail.
2016-09-23 to 2016-09-23, v0.10.0, Tests pass.
2016-09-21 to 2016-09-22, v0.10.0, Tests fail.
...

[tkelman]

Thanks for opening this. Related discussion is in #138. I think this is a problem with base precompilation, or maybe a package that precedes you in the same VM is trying to do something unreliable with optional dependencies. Since your package starts with M, it runs early in the MZ VM so hopefully this should be possible to come up with a set of steps to reproduce outside of pkgeval and report on base. Setting a temporary JULIA_PKGDIR and looping through add, test, rm should capture most of what pkgeval does.

[bdeonovic]

This is still an issue. @tkelman Can you give a bit more detail on how to pinpoint the culprit. If I understand correctly, packages starting with "M" through packages starting with "Z" are tested on same VM? So I should make temporary JULIA_PKGDIR, add, test, rm each package that comes before Mamba alphabetically? Does that sound right?

How will this help pinpoint culprit? Lots of packages don't pass tests before Mamba.

[tkelman]

It's not a matter of which other packages fail their tests. It's which packages doing precompilation before Mamba cause Mamba to fail.

[bdeonovic]

so I should make temporary JULIA_PKGDIR,

1. Identify which packages before Mamba have the precompile turned on loop over these
2. add, test, rm package,
3. test Mamba

[tkelman]

would also have to include anything where dependencies have precompilation enabled

[bdeonovic]

seems like it would be easier to just check all packages before Mamba. Can you confirm this is only for packages starting with "M"?

[tkelman]

yes https://github.com/JuliaCI/PackageEvaluator.jl/blob/master/scripts/setup.sh#L137

[bdeonovic]

@tkelman I just tried the above described procedure and I was not able to pinpoint a troublesome package; Mamba tests were able to start after adding, testing, and removing each package that came before Mamba alphabetically.

Whats next?

[tkelman]

What OS, distro, environment did you do this in? And version of Julia? Maybe try running the vagrant provisioning then run the packages up to mamba inside the VM interactively?

[bdeonovic]

I did this in JuliaBox which runs Ubuntu 14.04 on Julia version 0.5. What is vagrant provisioning?

[tkelman]

vagrant is a cli wrapper around various VM tools, often virtualbox. the scripts here set up package dependencies in a different way than what juliabox would have. could you post the script you ran?

[bdeonovic]

I manually ran the following commands in the REPL

pkg="MachineLearning"; Pkg.add(pkg); Pkg.test(pkg); Pkg.rm(pkg); Pkg.add("Mamba"); Pkg.test("Mamba")

I would wait to see if the Mamba tests started, then I would force them to stop (ctrl + c) as the Mamba tests take quite a long time and the PkgEvaluator issue seems to occur before any tests run. Then I would move on to the next package.

[tkelman]

it might not be a single package causing this, but a series of multiple. either way it looks highly repeatable within the specific environment and commands that pkgeval runs, at least the past few weeks have all been failures

[tkelman]

Also note that the sorting is maybe different than what you thought it was, I believe things run in the order they're listed at http://pkg.julialang.org/ (so capital second letters before lowercase a) nevermind, checking the timestamps on the logs now I'm not so sure

[tkelman]

Mamba has at least settled down for the last few months.

On 0.5.2 and later, precompilation should better handle the missing dependency problem from other packages doing fragile conditional imports, and mark a cache file as stale if things it depends on aren't installed any more

[brian-j-smith]

Thanks Tony!