lib/repository2: fix parallel pull synchronisation

[jonboulle]

The parallel pull refactoring in
47f2cb9 introduced an unfortunate and
subtle race condition. Because i) we were asynchronously starting the
fetching of each layer 1 (which implies that the fetch is added to
the progress bar asynchronously, via getLayerV2 2 3), and ii)
the progressutil package has no guards against calling PrintAndWait and
AddCopy in indeterminate order 4, it was possible for us to enter
the PrintAndWait loop 5 6 before all of the layer fetches had
actually been added to it. Then, in the event that the first layer was
particularly fast, the CopyProgressPrinter could actually think it was
done 7 before the other layer(s) had finished. In this situation,
docker2aci would happily continue forward and try to generate an ACI
from each layer 8, and for any layer(s) that had not actually
finished downloading, the GenerateACI->writeACI->tarball.Walk call
9 would be operating on a partially written file and hence result
in the errors we've been seeing ("unexpected EOF", "corrupt input
before offset 45274", and so forth).

A great case to reproduce this is the docker:busybox image because of
its particular characteristics: two layers, one very small (32B) and one
relatively larger (676KB) layer. Typical output looks like the following

% $(exit 0); while [ $? -eq 0 ]; do ./bin/docker2aci docker://busybox; done

Downloading sha256:385e281300c [===============================] 676 KB
/ 676 KB
Downloading sha256:a3ed95caeb0 [===============================]     32
B / 32 B

Generated ACI(s):
library-busybox-latest.aci

Downloading sha256:a3ed95caeb0 [==============================]      32
B / 32 B
Downloading sha256:385e281300c [==============================]  676 KB
/ 676 KB

Generated ACI(s):
library-busybox-latest.aci

Downloading sha256:a3ed95caeb0 [===================================] 32
B / 32 B
Error: conversion error: error generating ACI: error writing ACI: Error
reading tar entry: unexpected EOF

Note that i) the order in which the layers are registered with the
progress printer is indeterminate, and ii) every failure case (observed
so far) is when the small layer is retrieved first, and the stdout contains no
progress output at all from retrieving the other layer. This indeed
suggests that the progress printer returns before realising the second
layer is still being retrieved.

Tragically, @dgonyeo's test case 10 probably gives a false sense
of security (i.e. it cannot reproduce this issue), most likely
because the dgonyeo/test image contains a number of layers (5) of
varying sizes, and I suspect it's much less likely for this particular
race to occur.

Almost certainly fixes #166 - with this patch I'm unable to reproduce.

[jonboulle]

coreos/pkg#61, coreos/pkg#62 and coreos/pkg#63 are non-critical for this but would be nice to have in asap too.

[alban]

LGTM