feat(k8s): flexible buildKit cache configuration with mode=max support

[stefreak]

What this PR does / why we need it:

This is a result of a pairing session between @TimBeyer and @stefreak

Two main goals here:

• add support for mode=max and enable it by default if the registry supports it
• make the buildkit cache configuration more flexible

We achieved them by introducing a cache key within the clusterBuildkit config of the kubernetes provider.

The default is the following configuration

clusterBuildkit:
  cache:
    - type: registry

(mode is implicitly auto by default)

If you use gcr or aws ecr, this will effectively be equivalent to the current behaviour of garden, using an inline cache:

clusterBuildkit:
  cache:
    - type: registry
      mode: inline

# translates to
# --output type=image,"name=registry/image:v-xxxxx,registry/image:_buildcache",push=true
# --import-cache type=registry,ref=registry/image:_buildcache
# --export-cache type=inline

and if you use any other registry, the default mode auto is equivalent to:

clusterBuildkit:
  cache:
    - type: registry
      mode: max

# translates to
# --output type=image,"name=registry/image:v-xxxxx",push=true
# --import-cache type=registry,ref=registry/image:_buildcache
# --export-cache type=registry,ref=registry/image:_buildcache,mode=max

more advanced configurations are now possible as well. example:

clusterBuildkit:
  cache:
    - type: registry
      tag: _buildcache-${camelCase(git.branch)}
    - type: registry
      tag: _buildcache-main
      export: false

# translates to the following if you are in a git branch named `feature-branch`
# --output type=image,"name=registry/image:v-xxxxx",push=true
# --import-cache type=registry,ref=registry/image:_buildcache-featureBranch
# --import-cache type=registry,ref=registry/image:_buildcache-main
# --export-cache type=registry,ref=registry/image:_buildcache-featureBranch,mode=max
# (or the inline equivalent if you're using gcr or aws ecr)

which will result in a better cache hit rate: if you are developing in a new feature branch, you will use the cache of the main branch for the first build, and export the cache to a feature branch specific tag. From then on, you will use the feature branch specific tag for consequent builds. This makes sure that other builds don't pollute the feature branch build cache, and your feature branch does not pollute the master branch build cache. The idea for this configuration originates from a discussion with @antoinelyset from Slite: #3219 (comment)

Furthermore, it is now possible to configure cache registries that deviate from the deploymentRegistry, like so:

clusterBuildkit:
  cache:
    - type: registry
      registry:
        hostname: cacheRegistry
        namespace: namespace

# translates to the following
# --output type=image,"name=deploymentRegistry/image:v-xxxxx",push=true
# --import-cache type=registry,ref=cacheRegistry/image:_buildcache
# --export-cache type=registry,ref=cacheRegistry/image:_buildcache,mode=max

Currently the only supported type is registry, but the way to articulate the cache makes it possible to add other cache types as well in future pull requests.

Which issue(s) this PR fixes:

Fixes #3219
Fixes #3110

Special notes for your reviewer:

• There is a TODO in the joi configuration. We couldn't get the defaults working without repeating them. Help appreciated
• We want to add support for cache registries (e.g. to use GCHR only for the build cache, and AWS ECR for the images)
• We need to handle the "insecure" for in-cluster registries properly. TODO

[stefreak]

The test failure seems to be unrelated to this change.

[edvald]

Had some comments and suggestions, but overall this is great and a very welcome improvement 👏

I'd love to see more documentation in the in-cluster building guide around this as well, as opposed to just documenting in the reference. A suggested good practice in the guide would make a lot of sense imo.

[edvald]

It may be unnecessary, not sure, but we could use parseImageId to narrow this detection to the hostname part specifically.

[stefreak]

Same with https://github.com/garden-io/garden/pull/3239/files#r978832787

the test for in-cluster registry will fail.... all the code looks very complex, I thought about adding port support but I'd be very afraid of breaking something else with that

diff --git a/core/src/plugins/kubernetes/container/build/buildkit.ts b/core/src/plugins/kubernetes/container/build/buildkit.ts
index 980b9378..37e418a9 100644
--- a/core/src/plugins/kubernetes/container/build/buildkit.ts
+++ b/core/src/plugins/kubernetes/container/build/buildkit.ts
@@ -37,6 +37,7 @@ import { getRunningDeploymentPod, millicpuToString, megabytesToString, usingInCl
 import { PodRunner } from "../../run"
 import { prepareSecrets } from "../../secrets"
 import { ContainerModuleOutputs } from "../../../container/container"
+import { containerHelpers } from "../../../container/helpers"

 export const buildkitImageName = "gardendev/buildkit:v0.9.3-1"
 export const buildkitDeploymentName = "garden-buildkit"
@@ -322,13 +323,15 @@ export const getSupportedCacheMode = (
     return cache.mode
   }

+  const host = containerHelpers.parseImageId(deploymentImageName).host!
+
   // Detect AWS ECR
-  if (deploymentImageName.includes(".dkr.ecr.")) {
+  if (host.includes(".dkr.ecr.")) {
     return "inline"
   }

   // Detect gcr.io
-  if (deploymentImageName.includes("gcr.io")) {
+  if (host.includes("gcr.io")) {
     return "inline"
   }

[stefreak]

@edvald thank you for the review! I tried to address all the comments, and when I didn't I tried to explain why.

See you after vacation 👍 🍹