Invalid KongIngress caused service downtime

[dpiddock]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Validating webhook allowed invalid KongIngress object to be created with upstream.hash_on not defined. This caused KIC to submit invalid configuration to Kong which returned an error. When the kong pods restarted this resulted in full service downtime.

Expected Behavior

Either:

• validating webhook rejects invalid KongIngress
• KIC or Kong handle the invalid configuration without total downtime

Steps To Reproduce

Example configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
      - image: jmalloc/echo-server:latest
        name: echo
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: test
  annotations:
    konghq.com/override: test
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  selector:
    app: echo
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    konghq.com/override: test
  name: test
spec:
  ingressClassName: kong-gateway
  rules:
    - http:
        paths:
          - backend:
              service:
                name: test
                port:
                  number: 8080
            path: /
            pathType: ImplementationSpecific
---
apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name: test
upstream:
  hash_on_query_arg: sid
  hash_fallback: ip

1. Apply sample config
2. Check logs of kong ingress controller. Note that Kong returns an error for posting new config to /config
3. Rollout restart Kong deployment
4. Downtime for services

Kong Ingress Controller version

v2.10.0

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.9", GitCommit:"a1a87a0a2bcd605820920c6b0e618a8ab7d117d4", GitTreeState:"clean", BuildDate:"2023-04-12T12:16:51Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.9-eks-0a21954", GitCommit:"eb82cd845d007ae98d215744675dcf7ff024a5a3", GitTreeState:"clean", BuildDate:"2023-04-15T00:37:59Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}

Anything else?

No response

[HsinHeng]

As We implement Sticky-Session using KongIngress, we encounter the same issue.

apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name: sticky-session
upstream:
  algorithm: "consistent-hashing"
  hash_on: "cookie"
  hash_on_cookie: "KONGINGRESSCOOKIE"
  hash_on_cookie_path: "/"
  hash_fallback: "header"
  hash_fallback_header: "x-forwarded-for"

Error Log:
time=“2023-11-08T07:53:18Z” level=error msg=“could not update kong admin” error=“performing update for https://192.168.182.230:8444/ failed: failed posting new config to /config: got status code 400” subsystem=dataplane-synchronizer

When we remove hash_fallback & hash_fallback_header will be fine.

Kong Ingress Contollrer Version: 2.9.3
Kong: 3.2