Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Optimize CircleCI using a local docker registry instead docker save/load #136

Merged
merged 2 commits into from
Nov 8, 2019

Conversation

oesteban
Copy link
Member

@oesteban oesteban commented Nov 8, 2019

Seems like replacing the docker save/load approach with a local registry run on the same machine allows us to make the main build job ~8min lighter and smoke tests builds ~6min lighter.

This is probably not a big deal time-wise, but I think this approach is more reliable than the docker save/load, and will escalate better if CircleCI opens up to having service jobs or some such.

This PR also revises the artifacts being collected - which were too many for successful runs (including the work directory and freesurfer folder).

The idea is to test how this works for sMRIPrep and then propagate to fMRIPrep and other projects if things look good.

@oesteban oesteban requested review from effigies and mgxd November 8, 2019 18:42
@effigies
Copy link
Member

effigies commented Nov 8, 2019

I guess if it works, cool? I have no context for evaluating this apart from "Will the test pass".

@oesteban
Copy link
Member Author

oesteban commented Nov 8, 2019

Yup, it seems to work, looking at the layer digests and manifests. My only concern is that it seems to keep slowly growing. - I'm looking right now at how to garbage collect unused layers.

@oesteban oesteban force-pushed the maint/optimize-circle branch from 21c4369 to f0dd7bc Compare November 8, 2019 19:06
@oesteban
Copy link
Member Author

oesteban commented Nov 8, 2019

Nice - this https://stackoverflow.com/a/57989780 worked out locally. Testing now on Circle.

@oesteban
Copy link
Member Author

oesteban commented Nov 8, 2019

Working on Circle too! - https://app.circleci.com/jobs/github/oesteban/smriprep/941/parallel-runs/0/steps/0-107

The step to archive and propagate the workspace went back below 5m as in the early runs where the registry was still clean.

If tests pass, we can go ahead :)

@oesteban
Copy link
Member Author

oesteban commented Nov 8, 2019

Yass! Almost 1GB of difference after garbage collection:

@oesteban
Copy link
Member Author

oesteban commented Nov 8, 2019

Okay, the full workflow is now under 30 min! - Merging.

@oesteban oesteban merged commit fe1cb25 into nipreps:master Nov 8, 2019
@oesteban oesteban deleted the maint/optimize-circle branch November 8, 2019 20:35
oesteban added a commit to oesteban/sdcflows that referenced this pull request Nov 8, 2019
oesteban added a commit to nipreps/sdcflows that referenced this pull request Nov 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants