subprocess.check_call + Rebar in Cowboy and Elli

[ashawnbandy]

When run-test hits either the Cowboy or Elli test, it will either hang or time-out (depending on the version of benchmarker.py being used). run-test calls setup_erlang.py#start(args) and successfully executes all lines up to line 9 (see commit #4219cb3441). Line 9 is apparently called but never returns. This may be an intermittent issue but has persisted for at least the last seven days across several installations of the test harness and also appeared while @michaelhixson was testing a few weeks ago.

Additional facts:

1. The command being issued on line 9, "./rebar get-deps" runs successfully on the command line.
2. The code at http://pastebin.com/PptZBHFG runs without error.

[ashawnbandy]

One additional thing, I changed setup_util.py (which is called in line 6) so that the file write is both unbuffered and is explicitly closed. No effect.

[bhauer]

For reference: Round 7 preview data

[tcrossland]

I haven't been able to reproduce the issue on EC2:

toolset/run-tests.py -s localhost -c localhost -i key.pem -u ubuntu --test cowboy

Results here: http://pastebin.com/L50nyiVE

[bhauer]

Thanks for taking a look at this @tcrossland. I'll run this by @ashawnbandy to see what we can make of it.

[tcrossland]

Just to confirm, I haven't been able to reproduce the issue with elli either. For what it's worth, if the problem reoccurs, I'd suggest enabling verbose output on rebar

rebar -vvv get-deps

Good luck and thanks for putting these benchmarks together!

[bhauer]

@tcrossland Thanks so much for taking a look! We'll see if the verbose mode helps and if we can figure this out quickly.

If not, we'll defer this until after Round 7 since it may be something very specific to our hardware environment.

[ashawnbandy]

@tcrossland and @bhauer I certainly appreciate you taking a look at this as well - it's a puzzle I would love to see solved because I couldn't even get my finger in the crack to open it up.

To add a couple of comments to what I wrote above:

1. The problem is somewhat intermittent for us. One of the previous developers had the same issue but when I picked up the project, we had at least one successful run of cowboy and elli and I suspect there were several. The issue cropped up both on a newly installed toolset and one that had been run in the past.
2. The problem shows up on both Ubuntu 12.04 LTS and 13.04 - knowing that you were able to run it successfully on Ubuntu 12.04 is a great piece of information. As Brian mentioned, I don't think this rules out something particular with our hardware or environment, but simply that we probably don't need to look at differences in kernel or system library versions as a likely source.
3. Adding verbosity was one of the things that I gave a shot. The issue is that setup_erlang.py hits line 9 but produces no output even with -vvv and never returns. The curious thing, mentioned above, is that running these commands on the bash prompt works with no issues.

[tcrossland]

Another idea. I just noticed the rebar binary in the framework tests is a little old (7 months). It might be worth modifying the install scripts to build rebar from the latest source: https://github.com/rebar/rebar.

[hamiltont]

Fixed in master - See travis. Neither test passes, but it seems to be due to syntax errors and not anything systematic