-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CI: Optimize CircleCI using a local docker registry instead docker save/load #136
Conversation
I guess if it works, cool? I have no context for evaluating this apart from "Will the test pass". |
Yup, it seems to work, looking at the layer digests and manifests. My only concern is that it seems to keep slowly growing. - I'm looking right now at how to garbage collect unused layers. |
21c4369
to
f0dd7bc
Compare
Nice - this https://stackoverflow.com/a/57989780 worked out locally. Testing now on Circle. |
Working on Circle too! - https://app.circleci.com/jobs/github/oesteban/smriprep/941/parallel-runs/0/steps/0-107 The step to archive and propagate the workspace went back below 5m as in the early runs where the registry was still clean. If tests pass, we can go ahead :) |
Yass! Almost 1GB of difference after garbage collection:
|
Okay, the full workflow is now under 30 min! - Merging. |
…ve/load Follow the pattern proposed in nipreps/smriprep#136
…ve/load Follow the pattern proposed in nipreps/smriprep#136
Seems like replacing the docker save/load approach with a local registry run on the same machine allows us to make the main build job ~8min lighter and smoke tests builds ~6min lighter.
This is probably not a big deal time-wise, but I think this approach is more reliable than the docker save/load, and will escalate better if CircleCI opens up to having service jobs or some such.
This PR also revises the artifacts being collected - which were too many for successful runs (including the work directory and freesurfer folder).
The idea is to test how this works for sMRIPrep and then propagate to fMRIPrep and other projects if things look good.