Core: reduce memory footprint

[benashz]

Previously, VSO was caching and registering full object watchers on K8s Secrets. While watching K8s Secrets is necessary for automated remediation in the case where a destination secret is deleted from the cluster, doing so can result in OOM conditions for the operator, since each Secret's data contributes to the operator's total memory.

This change does the following:

• disables the caching K8s Secrets in the manager's client.
• only watches for Secret metadata changes.

Remaining tasks:

• Add integration regression tests - see Add OOM integration test #984

Reproduction steps:
Run:

kind create cluster --name oom
helm install --wait --namespace vault-secrets-operator-system --create-namespace vault-secrets-operator hashicorp/vault-secrets-operator
kubectl create namespace oom-secrets
dd if=/dev/urandom of=1m.random bs=1m count=1
# should be enough to trigger the OOM killer
for n in {0..90}; do kubectl create --namespace oom-secrets secret generic sec-$n --from-file /tmp/1m.random ; done

Wait for the manager to be OOM killed

Test:

make docker-build load-docker-image KIND_CLUSTER_NAME=oom
helm upgrade --set controller.manager.image.tag=0.0.0-dev --namespace vault-secrets-operator-system --create-namespace vault-secrets-operator hashicorp/vault-secrets-operator

Wait for VSO to come up, it should not be OOM killed

[tvoran]

The doc string for WatchesMetadata makes it sound like we should use metav1.PartialObjectMetadata{} instead of corev1.Secret{} for Get's where possible. I'm wondering if we need to do that for functions like getSecretExists()?

[benashz]

The doc string for WatchesMetadata makes it sound like we should use metav1.PartialObjectMetadata{} instead of corev1.Secret{} for Get's where possible. I'm wondering if we need to do that for functions like getSecretExists()?

Yeah, that might make those calls a bit lighter weight. We can probably address that in follow-on work.