[proposal] Reuse charts between releases with OCIRepository

[stefanprodan]

I propose we offer an alternative solution to the current HelmRepository/HelmChart APIs for users which deploy the same chart version across clusters with various values. And for users that want to use immutable charts by pinning a chart to its OCI digest.

Using an OCIRepository would allow users to create thousands of release in an efficient manner with close to zero CPU & memory usage for source-controller. Compared to the current approach where for each HelmRelease, source-controller needs to parse the infamous index.yaml and generate a chart per release, reusing an OCIRepository will enable running Flux at massive scale with no overhead. Even for when Helm charts and Helm repos are not reused, OCIRepository is still more efficient, instead of having to create two objects for each HelmRelease, a HelmRepository (OCI or HTTP) and a HelmChart, users would create one OCIRepository, less pressure on etcd, easier to debug for users. Using an OCIRepository means Flux no longer depends on Helm SDK for pulling charts, it would allow pinning on digests and have immutable charts.

The only downside when using an OCIRepository is that users can no longer swap the values.yaml from within a chart with a diffrent file e.g. values-production.yaml.

HelmRelease.spec.chartRef

We'll introduce an optional field for referencing a Helm chart artifact produced by an OCIRepository. Any number of HelmReleases could refer to the same OCIRepository. When the OCIRepository detects a new chart version in the container registry, all Helm releases will be upgraded by helm-controller.

Example HelmReleases that use the same chart version fetched from a container registry:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo-internal
  namespace: podinfo
spec:
  chartRef:
    kind: OCIRepository
    name: podinfo-chart
    namespace: podinfo
  values:
    ingress:
      enabled: false
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo-external
  namespace: podinfo
spec:
  chartRef:
    kind: OCIRepository
    name: podinfo-chart
    namespace: podinfo
  values:
    ingress:
      enabled: true

Example OCIRepository that scans the container registry for new versions and pulls the latest stable version of a chart:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo-chart
  namespace: podinfo
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/charts/podinfo
  ref:
    semver: '*'
  layerSelector:
    mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
    operation: copy

Example OCIRepository that pins a chart to its immutable digest (something that Helm doesn't support):

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo-chart
  namespace: podinfo
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/charts/podinfo
  ref:
    digest: 'sha256:642383f56ccb529e3f658d40312d01b58d9bc6caeef653da43e58d1afe88982a'
  layerSelector:
    mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
    operation: copy