Collapse run command experiment #7935
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Our Python docker builds are slow... a typical CI test run for Python takes ~21 minutes, of which the first ~10 minutes are spent building the image. This results in slower local development, slow CI test suites, and slow deployments. The main culprit is `pyenv install` which under the covers downloads the python source and then compiles it locally. Profiling showed that the download was quick, so even though `pyenv` supports `aria2c`, there's not much to be gained there. Unfortunately, a quick look at the `pyenv` issue tracker showed [there's no way to to pass pre-compiled artifacts to `pyenv`](https://github.com/orgs/pyenv/discussions/1872). For a long time we've bandied about the idea of switching from `pyenv` to downloading pre-compiled Pythons.However, we use `pyenv local` + `pyenv exec` throughout our Ruby code for switching to different Python versions. So we thought that it'd take a week or more to fully migrate away from `pyenv`. Today I had to rebuild the python image multiple times, and got so annoyed that I decided to poke at it a bit. It turns out that `pyenv` is simply a shim layer, and as long as `/usr/local/.pyenv/versions/<x.y.z>/bin` exists, it will happily pass commands to anything in that folder. So I was able to come up with an intermediate solution that speeds the builds up drastically without requiring a large code refactor. Running this locally results in the Python download/install/build step going from ~500 seconds all the way down to ~33 seconds, a savings of nearly 8 minutes. Given that a full CI run of the python test suite previously took ~21 minutes, this cuts it by 1/3.
|
Threw this up as a quick experiment because I was curious, but doesn't look like any change: a1c1474 One build was 1:40, and the other 2:03, vs 1:45 previously for two builds w/o this optimization, so both are within spitting distance of one another from a time perf perspective... Also, I realized that one other side-effect of splitting by layers is how it impacts cache hits when doing "cache from" in other images... for our use case there's zero impact since we never cache from these images. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Experiment on #7934 to see if collapsing the
RUNinstructions to a singleRUNcommand is more performant...