Same cache for all builds of a job

[auchri]

Is it possible to use the same cache for all builds of a job?

We have a lot of dependencies and so we decided to create an extra job which build the dependencies and store it in the cache.
All other (following) builds of that job should use the dependencies in the cache.

[BanzaiMan]

As explained in https://docs.travis-ci.com/user/caching/#Caches-and-build-matrices, the jobs within a build matrix would generally have different caches in order to minimize the chance of cache corruption.

All other (following) builds of that job should use the dependencies in the cache.

The jobs in a build matrix are independently queued. This makes it hard to predict the order in which jobs run; it is possible to limit the concurrency to 1 and list the "cache job" in the first slot, but even then, it is not guaranteed to run first.

[evelikov]

Rewording the question a bit:
How can we share the locally build dependencies across jobs in the build matrix ?
IMHO it's something definitely worth exploring, if it's not possible at the moment.

In one example It's causing a ~2 mins penalty for each job, while nearly all jobs are in the 4-7 mins range.
Would be amazing if we can avoid that extra ~2 mins.

[BanzaiMan]

Pointing to the same cache across jobs within a build matrix dramatically increases the likelihood of cache corruption. This is going to cause you many more problems than it "solves".

[BanzaiMan]

See #4393 for a history about this.

[BanzaiMan]

Dependencies among the jobs have been asked before, I believe, though I can't find a relevant issue at the moment.

[auchri]

Maybe an environment variable to override the name of the cache should be introduced.

It can work this way:
In the before_cache section, this variable can be defined. Your cache script checks if the variable is defined. If it is defined, the cache with the name will be fetched and the name is also used to store the cache. If the variable is not defined, the cache works the way it works now.

[BanzaiMan]

Sorry, but I don't think that would alleviate the concerns I mentioned earlier. Sharing caches among the jobs within a build matrix greatly increases the likelihood of cache corruption. Once you have a corrupt cache, any time savings you may get by sharing the cache will go out the window, and then you will have to spend time troubleshooting the broken builds. I don't think this is going to be in your best interest.

[auchri]

Sorry, but I don't think that would alleviate the concerns I mentioned earlier. Sharing caches among the jobs within a build matrix greatly increases the likelihood of cache corruption.

Yes, but it's the responsibility of the users if they set the environment variable.

[nurupo]

@BanzaiMan I disagree, Travis has Stage Builds, which allows you to build jobs sequentially, i.e. job2 won't start before job1 is finished, which removes the cache corruption. Making job1 building deps and caching them, and then having job2 retrieving job1's cache and using those pre-built deps is very useful, especially when you have one of job1 -- a job that builds dependencies of you project, but 10 of job2 -- jobs that must use those dependencies, then it's a big time saver, as you don't have to build the deps 10 times but just 1. It also helps when the combining runtime of job1+job2 would exceed 50 minutes, meaning that you can't even combine them into one job to begin with. These are just two of the benefits of cache sharing.

[evelikov]

@nurupo true build stages help a lot and IMHO it's a good way forward. Keep in mind:

• this issue was reported before build stages were introduced
• the lack of [locally] synchronising files across stages minimises the usefulness of the feature

[BanzaiMan]

If you are guaranteed that there is at most one job running for the repository (not just a build), then there is no concern for cache corruption. There are various ways to make this happen. I don't think that is a concern here.

At the same time, having an option to override the name (while it increases the chances of cache corruption dramatically) does seem useful.

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[auchri]

@BanzaiMan Please reopen this issue.

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue in 24 hours. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[auchri]

@BanzaiMan is it already fixed?

[nurupo]

@auchri You can share cache between different jobs, but in order to do so those jobs must have exactly the same configuration, specifically env, os, distro, and maybe more should be the same. However, script can be different, so one job might have script: bash .travis/build.sh stage1 but another script: bash .travis/build.sh stage2, i.e. if you really need the jobs to have different env variables, which you can't if you want to share the cache between them, you could make them script arguments instead and run export "$1"="$2" in the script or something, which would give it the similar effect as using env.

As mentioned before, because jobs run concurrently, if two concurrent jobs share the same cache, only one of them would be saved -- the slowest job would override the cache of the fastest job. That's why if you want multiple jobs to write into the cache, you must put them in separate Stages, so that they are run sequentially and don't overwrite each other, e.g. job2 in stage2 would get the result of the cache created by job1 in stage1. However, if all you do is read from the cache and don't write anything into it, then it's fine to have jobs run in parallel doing the reading, one job would overwrite the cache of the other, but since none of them changed it the cache would be the same no matter which job would overwrite it.

There is an unsolved issue with sharing cache though -- concurrent builds. The cache is also shared across builds, so if you have build1 and build2 runing concurrently, then one build might mess up the cache of another build, especially if you have 2 or more stages in each build sharing the cache, e.g. build1 stage1 would prepare the cache for build1 stage2, and it in turn would prepare it for build1 stage3, but it could be overwritten by build2 stage1, so build1 stage3 could end up using the cache of build2 stage1 which is undesirable and might not even work properly. You can detect if a cache was overwritten by another build by storing $TRAVIS_BUILD_ID in a file in the cache during stage1 and comparing it to $TRAVIS_BUILD_ID in stage2 and 3 -- it should stay the same if no overwrites by another build happened, but there is nothing you can do about the overwrites even if you detect them, because even if you fail the job when detecting that the cache is from another build, your failed job's cache would be stored (apparently there is no way to make Travis not store the cache if the job fails) and it will overwrite the cache of the other build which might have advanced several stages in cache-building by now, so you would undo all of those advances of the other build which might break that other build. So in the end you end up with both builds being broken as each of them overwrote the cache of another, at different stages too.

So yeah, concurrent builds are a problem, so you might want to try avoid them. If I remember right, each git branch has its own cache namespace, even PRs do, so those are only concurrent builds of the same branch that you should be concerned about. And it's only an issue if one of those builds changes the cache. In practice it's a lot less of an issue than it sounds, it rarely happens, in fact I have several projects share cache between multiple stages and I have yet to run into the issue of concurrent builds overwriting the cache. Perhaps because we don't push code that updates/invalidates the cache that often? (Well, I think we ran into it once in 12 months of use, but simply deleting the cache and restarting the build, all of which you can do from Travis-CI Web UI, solved the issue).

For example, we use shared cache in our qTox repo for Windows builds, here is a sample build: https://travis-ci.org/qTox/qTox/builds/406142611 and here is travis.yml https://github.com/qTox/qTox/blob/59b5578c7bffc56f6227c60bfcb38f97d39ec8d9/.travis.yml

Windows Stage1 checks if cache is intact by checking if hash file containing the sha256 of the build file exists and if the hash matches, if not then there is either no cache or the build script changed in which case we want to rebuild our cache -- it will rm -rf * all the cache and will build Qt5 and openssl, sotring them in the cache. It won't store hash file in the cache though. (If hash file matched initially, this job does nothing.)

Windows Stage2 checks if cache is intact by checking if hash file containing the sha256 of the build file exists and if the hash matches, if not then we want to rebuild our cache -- it keeps the existing cache intact as stage1 already added things to it, and it builds various other libs and stores them in the cache in addition to what stage1 has stored. This time it also stores hash file with the hash of the build script. (If hash file matched initially, this job does nothing.)

Windows Stage3 builds our own software, qTox, using all the deps built by stage1 and stage2. Stage3 uses cache in read-only way, so there are two jobs running in parallel that use it -- a job for the release build and a job for the debug build.

That's what we do for i686. We do the same for x86_64, we even re-use the same stages for it so that i686 and x86_86 builds would run in parallel and finish faster than if ran sequentionally, but we intentionally separate the caches of i686 and x86_64 builds by using different env -- env: WINDOWS_BUILD_ARCH_CACHE_TRICK_VARIABLE=i686 and env: WINDOWS_BUILD_ARCH_CACHE_TRICK_VARIABLE=x86_64, so that we don't have two jobs running in parallel writing to the same cache, not to mention there is no point in sharing i686 and x86_64 caches in our case, there is simply nothing that can be used by i686 build that was created for x86_64 build and vice versa.

Sharing cache is a very useful feature, we wouldn't be able to build qTox without it as building Qt5 alone sometimes takes around 45 minutes, and Travis terminates at 50 minutes, so thanks to Travis-CI team for it!

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue in 7 days. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please respond before the issue is closed, or open a new one after. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues