Support for generating partial/fallback configuration when Kong rejects entities

[czeslavo]

Is there an existing issue for this?

• I have searched the existing issues

Does this enhancement require public documentation?

• I have added an Acceptance Criteria item for adding and/or adjusting public documentation (if applicable)

Problem Statement

Kubernetes Ingress Controller in DB-less mode uses Kong’s Admin API POST /config endpoint to populate Kong with configuration entities like Services, Routes, Plugins, etc. The whole configuration is a result of translating Kubernetes resources to Kong entities. No split of the structure happens - it’s treated as one undividable piece and is sent as such to Kong. KIC doesn’t keep a complete history of configuration changes. It can only keep the “last known good” configuration in memory (#4048) or fetch it from any Kong instance it configures (#4264). In KIC 2.11, the strategy for handling invalid configuration is divided into the following stages:

1. Admission webhook validates Kubernetes objects to satisfy schema and some basic rules that are viable to be verified in isolation for a single object (no cross-object validation). [Only if Admission webhook is enabled]
2. KIC catches an issue during the translation process (cross-object issues or schema violations not caught by the admission webhook). In such case, KIC can omit an affected object which will make subsequent configuration updates possible. Kubernetes Events are emitted to inform about issues.
3. KIC doesn’t catch an issue but Kong responds to POST /config with an error. KIC tags generated Kong entities with Kubernetes resources’ metadata. It’s returned in the error response, allowing KIC to generate Kubernetes Events associated with the faulty objects. No subsequent updates are allowed until the faulty objects are removed or fixed. The last known good configuration is used.

The third stage is most notably painful in scenarios where users configure their Kubernetes resources in an ApiOps/GitOps manner. There’s no way to ensure in a CI pipeline that the configuration that was applied with kubectl apply -f was correct due to the asynchronous nature of the 2nd and 3rd stages. Once the problem is discovered in the 3rd stage, there’s no other way to recover than removing or fixing the faulty object. It can be painful if the pipeline is shared between multiple teams (e.g., an organization with a single cluster/Gateway and multiple teams governing their own services configuration) as a mistake of one team can affect the whole organization, halting the possibility of changing the Gateway’s configuration without looking into Kubernetes Events or KIC logs and taking actions to fix them. In other words: as Kubernetes resources belonging to different teams do not rely on each other, users expect team’s A resources’ validation errors to not affect team’s B resources from being updated.

As a result of fixing this issue, we limit the blast radius that one broken entity in the configuration can make, allowing users to still modify other parts of the configuration without an urgent need to fix the mistake.

Proposed Solution

The proposed solution is described in the design doc.

Additional information

No response

Acceptance Criteria

• If an independent part of the configuration sent to the Gateway by KIC is not accepted, KIC can drop this piece of config and successfully apply the rest as a whole
• No user intervention is needed for the above to happen
• A user can inspect what entities were removed/backfilled during the process (either via raw dumped configuration or a visual graph representation)
• Feature is well documented in official docs

Tasks

Preview Give feedback

1. Refactor BuildKongConfig to accept a Kubernetes store snapshot as input #5928
   priority/high release/required +2
   
2. Make a new snapshot of the store only when it differs from previous snapshot #5996
   priority/high release/required +2
   
3. Implement conversion from a Kubernetes store snapshot to a graph #5929
   3 of 3
   priority/high release/required +2
   
4. Automated test ensuring ResolveDependencies is implemented for all supported object types #6026
   release/required +1
   
5. Implement building a fallback Kubernetes store snapshot #5930
   priority/high release/required +2
   
6. Integrate building and pushing fallback configuration into KongClient.Update #5931
   release/required +1
   
7. Expose debug endpoints allowing inspection of the fallback configuration #5932
   release/required +1
   
8. Implement backfilling broken objects from last valid cache state #6096
   release/required +1
   
9. Generate cache snapshot before building current config only if cache changed #6080
   release/required +1
   
10. Emit k8s events for fallback configuration #6081
    release/required +1
    
11. Expose fallback configuration metrics #6082
    release/required +1
    
12. Ensure that all supported Kubernetes resources fallback is covered in tests #5933
    2 of 2
    release/required +1
    
13. Write an article explaining how the fallback configuration feature works #5934
    area/docs release/required +2
    
14. Fallback configuration exploratory testing #6091
    release/required +1