Add helm.sh/resource-policy: keep to CRDs to prevent deletion

[Danil-Grigorev]

What this PR does / why we need it:

When CAPI Operator chart used as a dependency, the removal of the chart hosting the dependency causes CRDs to be removed. This happens because CRDs are generated as a part of base and therefore are managed by helm 3. Helm docs define keep policy as:

skip deleting this resource when a helm operation (such as helm uninstall, helm upgrade or helm rollback) would result in its deletion

Setting this annotation should not prevent CRD upgrades, when resource is changed only.

Typically this removal happens after the main deployment is removed, and before any or all of the *Provider resources are removed. Provider resources has a finalizer, nothing can remove at this point, so the deletion hangs and later fails. Happens because CRD removal requires all resources it defines to be cleaned up before being removed itself.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

[netlify]

✅ Deploy Preview for kubernetes-sigs-cluster-api-operator ready!

Name	Link
🔨 Latest commit	5a19c98
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-cluster-api-operator/deploys/6629053b4097850007351410
😎 Deploy Preview	https://deploy-preview-500--kubernetes-sigs-cluster-api-operator.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[Fedosin]

For chart we have a dedicated section: https://github.com/kubernetes-sigs/cluster-api-operator/tree/main/config/chart
Maybe it's better to put your patch there, since it affects the chart only?

[Danil-Grigorev]

I added it there @Fedosin, although in this case an additional annotation makes difference for helm only anyway. Helm does not care what is on resource, it needs this annotation in the chart resources, so it can’t do anything regardless of the placement

[Danil-Grigorev]

I performed some testing of the helm upgrade, everything works with this change, so the docs are not wrong. Unless the resource is removed, this annotation changes nothing.

[alexander-demicev]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexander-demicev

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [alexander-demicev]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[furkatgofurov7]

/lgtm
/hold

To avoid merging, in case @Fedosin has some more questions, otherwise unhold

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 6f5e401c8eba3469de079022fc70627cb7eb575b

[Fedosin]

/lgtm
/hold cancel

[Danil-Grigorev]

/cherry-pick release-0.10

[k8s-infra-cherrypick-robot]

@Danil-Grigorev: new pull request created: #503

In response to this:

/cherry-pick release-0.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.