From 4ef76e251e6e8e6761fbab8d091792a57d90a1b8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Grzegorz=20Burzy=C5=84ski?= Date: Mon, 10 Jun 2024 16:54:44 +0200 Subject: [PATCH] do not disable webhook --- .../high-availability/fallback-config.md | 358 +++++++++++++++--- 1 file changed, 301 insertions(+), 57 deletions(-) diff --git a/app/_src/kubernetes-ingress-controller/guides/high-availability/fallback-config.md b/app/_src/kubernetes-ingress-controller/guides/high-availability/fallback-config.md index 3afe91fe18e7..d0aa1f6b48d3 100644 --- a/app/_src/kubernetes-ingress-controller/guides/high-availability/fallback-config.md +++ b/app/_src/kubernetes-ingress-controller/guides/high-availability/fallback-config.md @@ -135,19 +135,21 @@ First, we'll demonstrate the default behavior of the Fallback Configuration feat and their dependants from the configuration. To test the Fallback Configuration, make sure your {{site.kic_product_name}} instance is running with -the Fallback Configuration feature, diagnostics server enabled, and the Admission Webhook server disabled -(that will enable us to slip a broken configuration to the controller): +the Fallback Configuration feature and diagnostics server enabled. ```bash helm upgrade --install kong kong/ingress -n kong \ --set ingressController.env.feature_gates=FallbackConfiguration=true \ - --set ingressController.admissionWebhook.enabled=false + --set ingressController.env.dump_config=true ``` In the example, we'll consider a situation where: -1. We have two `HTTPRoute`s pointing to the same `Service` and working correctly. -2. One of the `HTTPRoute`s gets broken because of referencing an invalid `KongPlugin`. +1. We have two `HTTPRoute`s pointing to the same `Service`. One of `HTTPRoute`s is configured with `KongPlugin`s providing +authentication and base rate-limiting. Everything works as expected. +2. We add one more rate-limiting `KongPlugin` that is to be associated with the second `HTTPRoute` and a specific `KongConsumer` +so that it can be rate-limited in a different way than the base rate-limiting, but we forget associate the `KongConsumer` +with the `KongPlugin`. It results in the `HTTPRoute` being broken because of duplicated rate-limiting plugins. #### Deploying valid configuration @@ -162,7 +164,7 @@ service/echo created deployment.apps/echo created ``` -Next, let's deploy the `HTTPRoute`s: +Next, let's deploy the `HTTPRoute`s. `route-b` will refer three `KongPlugin`s (`key-auth`, `rate-limit-base,` `rate-limit-consumer`): ```bash echo 'apiVersion: gateway.networking.k8s.io/v1 @@ -185,6 +187,8 @@ apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: route-b + annotations: + konghq.com/plugins: key-auth, rate-limit-base, rate-limit-consumer spec: parentRefs: - name: kong @@ -205,6 +209,66 @@ httproute.gateway.networking.k8s.io/route-a created httproute.gateway.networking.k8s.io/route-b created ``` +Let's also create the `KongPlugin`s: + +```bash +echo 'apiVersion: configuration.konghq.com/v1 +kind: KongPlugin +metadata: + name: key-auth +plugin: key-auth +--- +apiVersion: configuration.konghq.com/v1 +kind: KongPlugin +metadata: + name: rate-limit-base +plugin: rate-limiting +config: + second: 1 + policy: local +--- +apiVersion: configuration.konghq.com/v1 +kind: KongPlugin +metadata: + name: rate-limit-consumer +plugin: rate-limiting +config: + second: 5 + policy: local' | kubectl apply -f - +``` + +The results should look like this: +```text +kongplugin.configuration.konghq.com/key-auth created +kongplugin.configuration.konghq.com/rate-limit-base created +kongplugin.configuration.konghq.com/rate-limit-consumer created +``` + +And finally, let's create the `KongConsumer` with credentials and the `rate-limit-consumer` `KongPlugin` associated: + +```bash +echo 'apiVersion: v1 +kind: Secret +metadata: + name: bob-key-auth + labels: + konghq.com/credential: key-auth +stringData: + key: bob-password +--- +apiVersion: configuration.konghq.com/v1 +kind: KongConsumer +metadata: + name: bob + annotations: + konghq.com/plugins: rate-limit-consumer + kubernetes.io/ingress.class: kong +username: bob +credentials: +- bob-key-auth +' | kubectl apply -f - +``` + #### Verifying routes are functional Let's ensure that the `HTTPRoute`s are working as expected: @@ -230,8 +294,10 @@ In namespace default. With IP address 192.168.194.13. ``` +Authenticated requests with the valid `apikey` header on the `route-b` should be accepted: + ```bash -curl -i $PROXY_IP/route-b +curl -i $PROXY_IP/route-b -H apikey:bob-password ``` The results should look like this: @@ -251,44 +317,52 @@ In namespace default. With IP address 192.168.194.13. ``` -#### Introducing a breaking change to the configuration - -Now, let's introduce a breaking change in the `route-b` by attaching a broken `KongPlugin` to it. - -Create a `KongPlugin` with an invalid configuration: +While the requests without the `apikey` header should be rejected: ```bash -echo 'apiVersion: configuration.konghq.com/v1 -kind: KongPlugin -metadata: - name: key-auth -plugin: key-auth -config: - # Should be 'key_names', not 'keys'. - keys: ["key"]' | kubectl apply -f - +curl -i $PROXY_IP/route-b ``` The results should look like this: ```text -kongplugin.configuration.konghq.com/key-auth created +HTTP/1.1 401 Unauthorized +Content-Type: application/json; charset=utf-8 +Connection: keep-alive +WWW-Authenticate: Key realm="kong" +Content-Length: 96 +X-Kong-Response-Latency: 0 +Server: kong/3.6.0 +X-Kong-Request-Id: 520c396c6c32b0400f7c33531b7f9b2c + +{ + "message":"No API key found in request", + "request_id":"520c396c6c32b0400f7c33531b7f9b2c" +} ``` -Now, let's attach the `KongPlugin` to `route-b`: +#### Introducing a breaking change to the configuration + +Now, let's simulate a situation where we introduce a breaking change to the configuration. We'll remove the `rate-limit-consumer` +`KongPlugin` from the `KongConsumer` so that the `route-b` will now have two `rate-limiting` plugins associated with it, +which will cause it to break. ```bash -kubectl annotate httproute route-b konghq.com/plugins=key-auth + kubectl annotate kongconsumer bob konghq.com/plugins- ``` The results should look like this: ```text -httproute.gateway.networking.k8s.io/route-b patched +kongconsumer.configuration.konghq.com/bob annotated ``` #### Verifying the broken route was excluded -This will cause the `route-b` to break. Let's verify this: +This will cause the `route-b` to break as there are two `KongPlugin`s using the same type (`rate-limiting`). We expect +the route to be excluded from the configuration. + +Let's verify this: ```bash curl -i $PROXY_IP/route-b @@ -313,8 +387,7 @@ X-Kong-Request-Id: 209a6b14781179103528093188ed4008 #### Inspecting diagnostic endpoints -The route is not configured because the Fallback Configuration mechanism has excluded the broken `KongPlugin` and its -dependants, including the `HTTPRoute`, from the configuration. +The route is not configured because the Fallback Configuration mechanism has excluded the broken `HTTPRoute`. We can verify this by inspecting the diagnostic endpoint: @@ -331,11 +404,11 @@ Content-Type: application/json { "brokenObjects": [ - {"kind": "KongPlugin", "name": "key-auth", "namespace": "default"}, + {"kind": "KongPlugin", "name": "rate-limit-consumer", "namespace": "default"}, {"kind": "HTTPRoute", "name": "route-b", "namespace": "default"} ], "excludedObjects": [ - {"kind": "KongPlugin", "name": "key-auth", "namespace": "default"}, + {"kind": "KongPlugin", "name": "rate-limit-consumer", "namespace": "default"}, {"kind": "HTTPRoute", "name": "route-b", "namespace": "default"} ] } @@ -369,7 +442,7 @@ With IP address 192.168.194.13. What's more, we're still able to update the correct `HTTPRoute` without any issues. Let's modify `route-a`'s path: ```bash -kubectl patch httproute route-a --type merge -p '{"spec":{"rules":[{"matches":[{"path":{"type":"PathPrefix","value":"/route-a-modified"}}],"backendRefs":[{"name":"echo","port":80}]}]}}' +kubectl patch httproute route-a --type merge -p '{"spec":{"rules":[{"matches":[{"path":{"type":"PathPrefix","value":"/route-a-modified"}}],"backendRefs":[{"name":"echo","port":1027}]}]}}' ``` The results should look like this: @@ -412,23 +485,29 @@ valid version. To demonstrate this, we'll use the same setup as in the default m ```bash helm upgrade --install kong kong/ingress -n kong \ --set ingressController.env.feature_gates=FallbackConfiguration=true \ ---set ingressController.env.use_last_valid_config_for_fallback=true \ ---set ingressController.admissionWebhook.enabled=false +--set ingressController.env.use_last_valid_config_for_fallback=true ``` -#### Detaching the broken KongPlugin +#### Attaching the plugin back As this mode of operation leverages the last valid Kubernetes objects' cache state, we need to make sure that -we begin with a clean slate, allowing {{site.kic_product_name}} to store it. -Let's detach the broken `KongPlugin` from `route-b` so we get an entirely valid configuration: +we begin with a clean slate, allowing {{site.kic_product_name}} to store it. + +{:.note} +> **Note:** {{site.kic_product_name}} stores the last valid Kubernetes objects' cache state in memory. It is not persisted +> across restarts. That means that if you've got broken objects in the configuration that were backfilled using the +> last valid version, after a restart the last valid version will be lost, effectively excluding these objects from the +> configuration. + +Let's remove one `KongPlugin` so we get an entirely valid configuration: ```bash -kubectl annotate --overwrite httproute route-b konghq.com/plugins- +kubectl annotate kongconsumer bob konghq.com/plugins=rate-limit-consumer ``` The results should look like this: ```text -httproute.gateway.networking.k8s.io/route-b annotated +kongconsumer.configuration.konghq.com/bob annotated ``` #### Verifying both routes are operational again @@ -437,7 +516,6 @@ Now, let's verify that both `HTTPRoute`s are operational back again: ```bash curl -i $PROXY_IP/route-a-modified -curl -i $PROXY_IP/route-b ``` For both, the results should look like this: @@ -457,23 +535,45 @@ In namespace default. With IP address 192.168.194.13. ``` +```bash +curl -i $PROXY_IP/route-b +``` + +The results should look like this: +```text +HTTP/1.1 401 Unauthorized +Content-Type: application/json; charset=utf-8 +Connection: keep-alive +WWW-Authenticate: Key realm="kong" +Content-Length: 96 +X-Kong-Response-Latency: 0 +Server: kong/3.6.0 +X-Kong-Request-Id: 0bc94b381edeb52f5a41e23a260afe40 + +{ + "message":"No API key found in request", + "request_id":"0bc94b381edeb52f5a41e23a260afe40" +} +``` + #### Breaking the route again -As we've verified that both `HTTPRoute`s are operational, let's break `route-b` again by attaching the broken `KongPlugin`: +As we've verified that both `HTTPRoute`s are operational, let's break `route-b` again by removing the `rate-limit-consumer` +`KongPlugin` from the `KongConsumer`: ```bash -kubectl annotate --overwrite httproute route-b konghq.com/plugins=key-auth +kubectl annotate kongconsumer bob konghq.com/plugins- ``` The results should look like this: ```text -httproute.gateway.networking.k8s.io/route-b annotated +kongconsumer.configuration.konghq.com/bob annotated ``` #### Verifying the broken route was backfilled Backfilling the broken `HTTPRoute` with its last valid version should have restored the route to its last valid working -state. That means we should be able to access `route-b`, but with no `KongPlugin` attached: +state. That means we should be able to access `route-b` as before the breaking change: ```bash curl -i $PROXY_IP/route-b @@ -482,19 +582,20 @@ curl -i $PROXY_IP/route-b The results should look like this: ```text -HTTP/1.1 200 OK -Content-Type: text/plain; charset=utf-8 -Content-Length: 137 +HTTP/1.1 401 Unauthorized +Date: Mon, 10 Jun 2024 14:00:38 GMT +Content-Type: application/json; charset=utf-8 Connection: keep-alive -X-Kong-Upstream-Latency: 2 -X-Kong-Proxy-Latency: 0 -Via: kong/3.6.0 -X-Kong-Request-Id: 0d91bf2d355ede4d2b01c3306886c043 +WWW-Authenticate: Key realm="kong" +Content-Length: 96 +X-Kong-Response-Latency: 5 +Server: kong/3.6.0 +X-Kong-Request-Id: 4604f84de6ed0b1a9357e935da5cea2c -Welcome, you are connected to node orbstack. -Running on Pod echo-74c66b778-szf8f. -In namespace default. -With IP address 192.168.194.13. +{ + "message":"No API key found in request", + "request_id":"4604f84de6ed0b1a9357e935da5cea2c" +} ``` #### Inspecting diagnostic endpoints @@ -513,19 +614,162 @@ Content-Type: application/json { "brokenObjects": [ - {"kind": "KongPlugin", "name": "key-auth", "namespace": "default"}, + {"kind": "KongPlugin", "name": "rate-limit-consumer", "namespace": "default"}, {"kind": "HTTPRoute", "name": "route-b", "namespace": "default"} - ] + ], "excludedObjects": [ - {"kind": "KongPlugin", "name": "key-auth", "namespace": "default"}, + {"kind": "KongPlugin", "name": "rate-limit-consumer", "namespace": "default"}, {"kind": "HTTPRoute", "name": "route-b", "namespace": "default"} - ], + ] "backfilledObjects": [ + {"kind": "KongPlugin", "name": "rate-limit-consumer", "namespace": "default"}, {"kind": "HTTPRoute", "name": "route-b", "namespace": "default"} + {"kind": "KongConsumer", "name": "bob", "namespace": "default"} ] } ``` +As `rate-limit-consumer` and `route-b` were reported back as broken by the {{site.base_gateway}}, they were excluded from +the configuration. However, the Fallback Configuration mechanism backfilled them with their last valid version, restoring +the route to its working state. You may notice that also the `KongConsumer` was backfilled - +this is because the `KongConsumer` was depending on the `rate-limit-consumer` plugin in the last valid state. + +{:.note} +> **Note:** The Fallback Configuration mechanism will attempt to backfill all the broken objects along with their direct +> and indirect dependants. The dependencies are resolved based on the last valid Kubernetes objects' cache state. + +#### Modifying the affected objects + +As we're now relying on the last valid version of the broken objects and their dependants, we won't be able to effectively +modify them until we fix the problems. Let's try and add another key for the `bob` `KongConsumer`: + +Create a new `Secret` with a new key: + +```bash +echo 'apiVersion: v1 +kind: Secret +metadata: + name: bob-key-auth-new + labels: + konghq.com/credential: key-auth +stringData: + key: bob-new-password' | kubectl apply -f - +``` + +The results should look like this: +```text +secret/bob-key-auth-new created +``` + +Associate the new `Secret` with the `KongConsumer`: + +```bash +kubectl patch kongconsumer bob --type merge -p '{"credentials":["bob-key-auth", "bob-key-auth-new"]}' +``` + +The results should look like this: +```text +kongconsumer.configuration.konghq.com/bob patched +``` + +The change won't be effective as the `HTTPRoute` and `KongPlugin` are still broken. We can verify this by trying to access the +`route-b` with the new key: + +```bash +curl -i $PROXY_IP/route-b -H apikey:bob-new-password +``` + +The results should look like this: + +```text +HTTP/1.1 401 Unauthorized +Content-Type: application/json; charset=utf-8 +Connection: keep-alive +Content-Length: 81 +X-Kong-Response-Latency: 2 +Server: kong/3.6.0 +X-Kong-Request-Id: 4c706c7e4e06140e56453b22e169df0a + +{ + "message":"Unauthorized", + "request_id":"4c706c7e4e06140e56453b22e169df0a" +} +``` + +#### Modifying the working route + +On the other hand, we can still modify the working `HTTPRoute`: + +```bash +kubectl patch httproute route-a --type merge -p '{"spec":{"rules":[{"matches":[{"path":{"type":"PathPrefix","value":"/route-a-modified-again"}}],"backendRefs":[{"name":"echo","port":1027}]}]}}' +``` + +The results should look like this: +```text +httproute.gateway.networking.k8s.io/route-a patched +``` + +Let's verify the updated `HTTPRoute`: + +```bash +curl -i $PROXY_IP/route-a-modified-again +``` + +The results should look like this: +```text +HTTP/1.1 200 OK +Content-Type: text/plain; charset=utf-8 +Content-Length: 136 +Connection: keep-alive +X-Kong-Upstream-Latency: 2 +X-Kong-Proxy-Latency: 0 +Via: kong/3.6.0 +X-Kong-Request-Id: 4369f15cf27cf16f5a2c82061b8d3950 + +Welcome, you are connected to node orbstack. +Running on Pod echo-bf9d56995-r8c86. +In namespace default. +With IP address 192.168.194.8. +``` + +#### Fixing the broken route + +To fix the broken `HTTPRoute`, we need to associate the `rate-limit-consumer` `KongPlugin` back with the `KongConsumer`: + +```bash +kubectl annotate kongconsumer bob konghq.com/plugins=rate-limit-consumer +``` + +This should unblock the changes we've made to the `bob-key-auth` `Secret`. Let's verify this by accessing the `route-b` +with the new key: + +```bash +curl -i $PROXY_IP/route-b -H apikey:bob-new-password +``` + +The results should look like this now: + +```text +HTTP/1.1 200 OK +Content-Type: text/plain; charset=utf-8 +Content-Length: 136 +Connection: keep-alive +X-RateLimit-Limit-Second: 5 +RateLimit-Limit: 5 +RateLimit-Remaining: 4 +RateLimit-Reset: 1 +X-RateLimit-Remaining-Second: 4 +X-Kong-Upstream-Latency: 2 +X-Kong-Proxy-Latency: 2 +Via: kong/3.6.0 +X-Kong-Request-Id: 183ecc2973f16529a314ca5bf205eb73 + +Welcome, you are connected to node orbstack. +Running on Pod echo-bf9d56995-r8c86. +In namespace default. +With IP address 192.168.194.8. +``` + ### Inspecting the Fallback Configuration process Each time {{site.kic_product_name}} successfully applies a fallback configuration, it emits a Kubernetes Event