Skip to content

Commit

Permalink
internal/ci: refactor base caching pattern
Browse files Browse the repository at this point in the history 
We currently parameterise the setupGoActionsCaches template with an
expression that guards proctected branches. However this concept is
already well defined in base via isProtectedBranch. Therefore, remove
the #protectedBranchExpr parameter.

At the same time hoist some logic around the clearing of the testcache
as part of the setup of the Go caches. This is otherwise repeated in
every trybot workflow at least.

For some workflows that want to reuse the Go cache template but _not_
clear the testcache. We solve this with two parameters: readonly and
cleanTestCache. readonly explicitly states that we a workflow only wants
to restore from the cache. cleanTestCache is defaulted based on the
inverse of readonly, but is still guarded on us running on a protected
branch.

Signed-off-by: Paul Jolly <paul@myitcv.io>
Change-Id: Ia100955abdb8e6db3b89cbd04e1c914c51cd514c
Reviewed-on: https://review.gerrithub.io/c/cue-lang/cue/+/551922
Unity-Result: CUEcueckoo <cueckoo@cuelang.org>
TryBot-Result: CUEcueckoo <cueckoo@cuelang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
  • Loading branch information
@myitcv
myitcv committed Mar 30, 2023
1 parent 79aceb7 commit 92b28c4
Show file tree
Hide file tree
Showing 2 changed files with 64 additions and 40 deletions.
90 changes: 63 additions & 27 deletions internal/ci/base/github.cue
View file
Original file line number Diff line number Diff line change
Expand Up @@ -124,11 +124,17 @@ curlGitHubAPI: #"""
"""#

setupGoActionsCaches: {
// #protectedBranchExpr is a GitHub expression
// (https://docs.github.com/en/actions/learn-github-actions/expressions)
// that evaluates to true if the workflow is running for a commit against a
// protected branch.
#protectedBranchExpr: string
// #readonly determines whether we ever want to write the cache back. The
// writing of a cache back (for any given cache key) should only happen on a
// protected branch. But running a workflow on a protected branch often
// implies that we want to skip the cache to ensure we catch flakes early.
// Hence the concept of clearing the testcache to ensure we catch flakes
// early can be defaulted based on #readonly. In the general case the two
// concepts are orthogonal, hence they are kept as two parameters, even
// though in our case we could get away with a single parameter that
// encapsulates our needs.
#readonly: *false | bool
#cleanTestCache: *!#readonly | bool

let goModCacheDirID = "go-mod-cache-dir"
let goCacheDirID = "go-cache-dir"
Expand All @@ -138,6 +144,21 @@ setupGoActionsCaches: {
// that participate in Go caching.
let cacheDirs = [ "${{ steps.\(goModCacheDirID).outputs.dir }}/cache/download", "${{ steps.\(goCacheDirID).outputs.dir }}"]

let cacheRestoreKeys = "${{ runner.os }}-${{ matrix.go-version }}"

let cacheStep = json.#step & {
with: {
path: strings.Join(cacheDirs, "\n")

// GitHub actions caches are immutable. Therefore, use a key which is
// unique, but allow the restore to fallback to the most recent cache.
// The result is then saved under the new key which will benefit the
// next build. Restore keys are only set if the step is restore.
key: "\(cacheRestoreKeys)-${{ github.run_id }}"
"restore-keys": cacheRestoreKeys
}
}

// pre is the list of steps required to establish and initialise the correct
// caches for Go-based workflows.
[
Expand All @@ -153,34 +174,49 @@ setupGoActionsCaches: {
id: goCacheDirID
run: #"echo "dir=$(go env GOCACHE)" >> ${GITHUB_OUTPUT}"#
},
for _, v in [
{
if: #protectedBranchExpr

// Only if we are not running in readonly mode do we want a step that
// uses actions/cache (read and write). Even then, the use of the write
// step should be predicated on us running on a protected branch. Because
// it's impossible for anything else to write such a cache.
if !#readonly {
cacheStep & {
if: isProtectedBranch
uses: "actions/cache@v3"
},
{
if: "! \(#protectedBranchExpr)"
uses: "actions/cache/restore@v3"
},
] {
v & json.#step & {
with: {
path: strings.Join(cacheDirs, "\n")

// GitHub actions caches are immutable. Therefore, use a key which is
// unique, but allow the restore to fallback to the most recent cache.
// The result is then saved under the new key which will benefit the
// next build
key: "${{ runner.os }}-${{ matrix.go-version }}-${{ github.run_id }}"
"restore-keys": "${{ runner.os }}-${{ matrix.go-version }}"
}
}
},

cacheStep & {
// If we are readonly, there is no condition on when we run this step.
// It should always be run, becase there is no alternative. But if we
// are not readonly, then we need to predicate this step on us not
// being on a protected branch.
if !#readonly {
if: "! \(isProtectedBranch)"
}

uses: "actions/cache/restore@v3"
},

if #cleanTestCache {
// All tests on protected branches should skip the test cache. The
// canonical way to do this is with -count=1. However, we want the
// resulting test cache to be valid and current so that subsequent CLs
// in the trybot repo can leverage the updated cache. Therefore, we
// instead perform a clean of the testcache.
//
// Critically we only want to do this in the main repo, not the trybot
// repo.
json.#step & {
if: "github.repository == '\(githubRepositoryPath)' && (\(isProtectedBranch) || github.ref == 'refs/heads/\(testDefaultBranch)')"
run: "go clean -testcache"
}
},
]
}

// #isProtectedBranch is an expression that evaluates to true if the
// job is running as a result of pushing to one of _#protectedBranchPatterns.
// isProtectedBranch is an expression that evaluates to true if the
// job is running as a result of pushing to one of protectedBranchPatterns.
// It would be nice to use the "contains" builtin for simplicity,
// but array literals are not yet supported in expressions.
isProtectedBranch: {
Expand Down
14 changes: 1 addition & 13 deletions internal/ci/github/trybot.cue
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,25 +37,13 @@ workflows: trybot: _repo.bashWorkflow & {
strategy: _testStrategy
"runs-on": "${{ matrix.os }}"

let goCaches = _repo.setupGoActionsCaches & {#protectedBranchExpr: _repo.isProtectedBranch, _}

steps: [
for v in _repo.checkoutCode {v},
_repo.installGo,

// cachePre must come after installing Node and Go, because the cache locations
// are established by running each tool.
for v in goCaches {v},

// All tests on protected branches should skip the test cache.
// The canonical way to do this is with -count=1. However, we
// want the resulting test cache to be valid and current so that
// subsequent CLs in the trybot repo can leverage the updated
// cache. Therefore, we instead perform a clean of the testcache.
json.#step & {
if: "github.repository == '\(_repo.githubRepositoryPath)' && (\(_repo.isProtectedBranch) || github.ref == 'refs/heads/\(_repo.testDefaultBranch)')"
run: "go clean -testcache"
},
for v in _repo.setupGoActionsCaches {v},

_repo.earlyChecks & {
// These checks don't vary based on the Go version or OS,
Expand Down

0 comments on commit 92b28c4

Please sign in to comment.