Speed up Github Actions CI

[levskaya]

Also enable more parallelism in CI tests.

[codecov-commenter]

Codecov Report

Merging #2635 (20cca7b) into main (3b27a69) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2635   +/-   ##
=======================================
  Coverage   80.53%   80.53%           
=======================================
  Files          50       50           
  Lines        5313     5313           
=======================================
  Hits         4279     4279           
  Misses       1034     1034           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[levskaya]

So this shaves off some time (maybe 3-5 min on average?) from our CI tests by using a larger 16-core test-runner, esp. due to faster pytype runs.

However we spend about half the time for all the pytest/doctest/pytype runs on just re-installing dependencies. I've tried caching the downloaded packages, but I feel like we might save more time by caching the installed packages - that would save another 2-3 min per run. See this interesting comment about caching the installed packages... maybe we should think about doing something like that to speed this up further.

Side note: We can also soon drop the 3.7 tests in favor of 3.10 tests. We may also want to add a mypy typecheck job as well.

[levskaya]

Addresses #2620

[levskaya]

NB: for caching e.g. a full virtualenv directory, the usual recommendation is to hash on the requirements.txt file, and to pin versions so as to be specific... however that's annoying - maybe we can invalidate caches every week using the date instead.

[levskaya]

OK I updated this to directly cache the virtual-environment directory post-installation so we don't waste any time installing all of our extensive set of deps for every single test. As part of the cache key I added the week-number and year, so that the caches will automatically invalidate every week, so that we're testing our unpinned dependencies at their newest versions. (Note that caches can be manually deleted under actions -> management -> caches)

Github supports 10G of caches per repo, our installed venv comes in at ~2.2Gb currently, allowing us to store enough for our 3 python version testing matrix.

So far the faster runners and caching drop our CI testing time: 17-19min --> 6m 15s

[levskaya]

The last major issue is that the codecov uploader can often take up to 1 minute to upload... this is horrible, and I don't think delaying every single CI test by ~30-60s is really worth it for a simple coverage indicator that none of us even probably look at.

[andsteing]

What's the recommended way to launch a Github build that forces an uncached environment?
(This might be useful to test dependency interplay at HEAD).

[levskaya]

You can just delete the caches from the Actions submenu. We could also probably figure out some way to signal against using caches, but in practice just deleting them is probably enough.

[marcvanzee]

Nice, thanks for adding this! Out of curiosity: do you have an idea how much of the speed improvement is due to the caching and how much is due to the improved pytype testing parallelism?

[levskaya]

Nice, thanks for adding this! Out of curiosity: do you have an idea how much of the speed improvement is due to the caching and how much is due to the improved pytype testing parallelism?

Kinda 50% / 50% - pytype is slow and really benefits from a larger machine (as do the rest of the tests) - but we were also wasting >3min or more for every test runner re-installing everything from scratch.