TestCertManagerAddon flakes occasionally

[rainest]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Some test runs for tests-and-coverage fail with:

 === CONT  TestCertManagerAddon
    certmanager_test.go:25: 
        	Error Trace:	certmanager_test.go:25
        	Error:      	Received unexpected error:
        	            	failed to deploy YAML STDOUT=() STDERR=(Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.96.126.96:443: connect: connection refused
        	            	): exit status 1
        	Test:       	TestCertManagerAddon
--- FAIL: TestCertManagerAddon (86.36s)

Re-running usually fixes this.

Expected Behavior

Test should succeed consistently with the same code.

Steps To Reproduce

Unknown. Probably a mild race condition somewhere where we need to wait a bit or confirm readiness of some service before trying some action.

Kong Kubernetes Testing Framework Version

No response

Kubernetes version

No response

Anything else?

No response

[shaneutt]

The current logic intends to wait for the webhook deployment pod availability, so this report seems to suggest an issue in upstream in that we can't depend on that availability as there is some very brief window where it can be available but also not actually ready? Testing this locally I couldn't trigger it myself, so I assume the window must be quite limited. I've created #165 to resolve this, which adds a job to wait for HTTP requests to the webhook to actually succeed over the cluster network before considering the addon "ready".