[BUG] Warning before server is ready, possibly a sign for a broken load balancer setup.

[oglok]

What happened:

Logs show a persistent warning:

W1104 11:20:59.283644   45618 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.122.249:34706, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.

eventhought openshift and oauth apiservers are in ready state.

What you expected to happen:

If Aggregated API servers are ready, those warnings should not appear.

How to reproduce it (as minimally and precisely as possible):

1. microshift run

Anything else we need to know?:

After a bit of investigation, we have applied the following patch:

https://github.com/redhat-et/microshift/blob/main/scripts/rebase_patches/0004_b301080e0639_UPSTREAM_carry_create-termination-events.patch

which should be used by the root (kube) apiserver, while the openshift-apiserver and oauth-apiserver should be using this patch:

openshift/kubernetes-apiserver@888e3d5

However, go.mod has a replace line to point k8s.io/apiserver to openshift/kubernetes-apiserver. We need to patch of try to split dependencies. (I think)

Environment:

• Microshift version (use microshift version): main branch
• Hardware configuration: Libvirt VM
• OS (e.g: cat /etc/os-release): Fedora 34
• Kernel (e.g. uname -a): Linux fedora 5.11.12-300.fc34.x86_64 Init #1 SMP Wed Apr 7 16:31:13 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Relevant Logs

W1104 11:23:16.595813   45618 patch_genericapiserver.go:123] Request to "/apis/apps.openshift.io/v1" (source IP 192.168.122.249:34782, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
W1104 11:23:16.600575   45618 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.122.249:34734, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
W1104 11:23:16.601055   45618 patch_genericapiserver.go:123] Request to "/apis/user.openshift.io/v1" (source IP 192.168.122.249:53360, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
W1104 11:23:16.602803   45618 patch_genericapiserver.go:123] Request to "/apis/route.openshift.io/v1" (source IP 192.168.122.249:34726, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
W1104 11:23:16.603180   45618 patch_genericapiserver.go:123] Request to "/apis/project.openshift.io/v1" (source IP 192.168.122.249:34756, user agent "microshift/v1.21.1 (linux/amd64) kub

[fzdarsky]

Duplicate of #249.