From 0d031729b7676da179bc4bda7609153796bf338d Mon Sep 17 00:00:00 2001 From: Joe Betz Date: Tue, 4 Feb 2025 11:40:16 -0500 Subject: [PATCH 1/2] Propose a runtime verification check --- .../4153-declarative-validation/README.md | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/keps/sig-api-machinery/4153-declarative-validation/README.md b/keps/sig-api-machinery/4153-declarative-validation/README.md index 50ab35ff824..e9bee28ea3e 100644 --- a/keps/sig-api-machinery/4153-declarative-validation/README.md +++ b/keps/sig-api-machinery/4153-declarative-validation/README.md @@ -1212,6 +1212,28 @@ See [Unit Test Instrumentation](#unit-test-instrumentation) for unit testing pla - `k8s.io/apiserver/pkg/cel/library`: `10/02/2023` - `69` - `k8s.io/apiextensions-apiserver/pkg/apiserver/schema/cel/model`: `10/02/2023` - `83.3` +##### Runtime verification testing + +In addition to unit and fuzz tests, we will offer a means of running declarative validation in a "shadow mode" +such that the presence of mismatches between declarative validation and hand written validation can +be safely checked against production workloads. + +When a `DeclarativeValidationShadowMetrics` feature gate is enabled, the following will be collected for each validation operation: + +A. Errors from running all hand written validation +B. Errors from running only hand written validation for non-converted validations (using validation opts) +C. Errors from running declarative validation + +This data will be used to check if A-B == C. That is, the declarative validation errors should be equivalent to the errors that hand written validation produces for all validation that has been converted to declarative. + +If the errors do not match, a 'declarative-validation-mismatch' metric will be incremented and information +about the mismatch will be written to the apiserver's logs. + +This can then be used to minimize risk when rolling out Declarative Validation in production, by following these steps: +- Enable `DeclarativeValidationShadowMetrics` +- Soak for a desired duration across some number of clusters +- Check the metrics to ensure no mismatches have been found + ##### Integration tests