If mutation webhook created before controller, controller pod will fail to be created

[lubronzhan]

Context

  Warning  FailedCreate      3s (x12 over 15s)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "mimg.kb.io": Post "https://harbor-automation-4k8s-webhook-service.harbor-day2-op.svc:443/mutate-image-path?timeout=30s": dial tcp 100.68.145.242:443: connect: connection refused

This is because once the mutationWebhookConfiguration is deployed, our api server will check all pods creation, including our controller pod. API server will call our webhook server. But since our controller pod contains the webhooks server, so it needs to come up first before apiserver could trigger webhook. This will becomes a deadlock if we deploy our controller after the mutationWebhookConfiguration.

Related issue kubernetes-sigs/kustomize#821

Reproduce steps

make deploy

Possible solution

1. Add labels our namespace, so webhook controller could skip namespaces with those labels
   Drawback: k8s related or other system level namespaces needs labels as well, there could be potential issue
2. Webhook controller only checks namespaces with specific labels
   Drawback: we will need to modify namespaces labels, except the annotations on them.