Run "Kubernetes Conformance Test" across all providers

[surajssd]

We should run conformance tests with each release and post the results for each provider. This helps us identify gaps and work towards closing them.

More information on conformance test: https://github.com/cncf/k8s-conformance

---

I think the first task is to identify what parts of the conformance test suite we should run? Then once identified we should create a script that can be run pointing to each provider.

[surajssd]

While testing, I came across some issues and I am fixing them upstream:

• kata-webhook-docs: Fix outdated script names kata-containers/tests#5665
• Dont modify sonobuoy pods kata-containers/tests#5666
• kata-webhook: Cleanup kata-webhook deployment kata-containers/tests#5668
• e2e_conformance: Add support for runtime class kata-containers/tests#5670
• versions: Update sonobuoy to latest v0.56.16 kata-containers/tests#5673
• k8s-conformance: Add support to run tests in parallel kata-containers/tests#5676

[surajssd]

Here is how to run it for a k8s cluster with CAA installed on it:

git clone https://github.com/kata-containers/tests
cd tests/

export RUNTIME_CLASS=kata-remote
kubectl create cm kata-webhook --from-literal runtime_class=$RUNTIME_CLASS

export KUBECONFIG=$HOME/.kube/config
export WAIT_TIME="10080"

./integration/kubernetes/e2e_conformance/run.sh

[surajssd]

In my first run I had following failures:

• [sig-node] Pods should run through the lifecycle of Pods and PodStatus [Conformance]
• [sig-storage] Projected downwardAPI should update labels on modification [NodeConformance] [Conformance]
• [sig-storage] ConfigMap optional updates should be reflected in volume [NodeConformance] [Conformance]
• [sig-storage] ConfigMap should be consumable from pods in volume with mappings as non-root [NodeConformance] [Conformance]
• [sig-node] InitContainer [NodeConformance] should not start app containers if init containers fail on a RestartAlways pod [Conformance]
• [sig-storage] Downward API volume should provide node allocatable (memory) as default memory limit if the limit is not set [NodeConformance] [Conformance]
• [sig-node] Kubelet when scheduling an agnhost Pod with hostAliases should write entries to /etc/hosts [NodeConformance] [Conformance]

[surajssd]

So in the second run I skipped the above and ran it again and now following more tests failed:

• [sig-scheduling] SchedulerPredicates [Serial] validates that there exists conflict between pods with same hostPort and protocol but one using 0.0.0.0 hostIP [Conformance]
• [sig-scheduling] SchedulerPreemption [Serial] PreemptionExecutionPath runs ReplicaSets to verify preemption running path [Conformance]
• [sig-storage] Projected configMap should be consumable from pods in volume with mappings [NodeConformance] [Conformance]
• [sig-storage] Secrets optional updates should be reflected in volume [NodeConformance] [Conformance]
• [sig-node] Container Runtime blackbox test on terminated container should report termination message from file when pod succeeds and TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance]
• [sig-storage] Projected downwardAPI should update labels on modification [NodeConformance] [Conformance]

---

Here is the summary of the above run: https://gist.github.com/surajssd/b8263d82086ca18cb0b6dc9a416f678a

---

Steps to recreate the above run:

• Deploy CAA
• Checkout this code: https://github.com/surajssd/tests/tree/suraj-run-e2e-may-30
• Run the following commands:

export RUNTIME_CLASS=kata-remote
kubectl create cm kata-webhook --from-literal runtime_class=$RUNTIME_CLASS

export KUBECONFIG=$HOME/.kube/config
export WAIT_TIME="10080"
export E2E_PARALLEL=true

./integration/kubernetes/e2e_conformance/run.sh

[surajssd]

Each time I am seeing different results! Not sure what is wrong, I think dissecting each failing test and looking closely as to what should be ignored should be considered for peer pods.