E2E testing a KIC built from HEAD by CI. (#869)

fixes #694 

- makes CI build a local KIC image from HEAD
- if on `main` or `next`: pushes to bintray. Can (or, maybe, should) be extended to all pushes
- spins up a local microk8s cluster
- pushes the local KIC to the local image registry
- runs an instance of KIC
- runs a set of tests: applying a bunch of manifests, waiting (open loop - `sleep 6`) for KIC to do its work, asserting on `curl` results
    - There are several possible ways to mitigate the `sleep` but none of them is perfect:
        - watch the `status` field of created resources
        - watch KIC logs for a successful/failed sync
        - watch the Kong Admin API `/config` endpoint

* test(integration): implement harness and basic tests

* test(integration): run on ci

* chore(test): improve test cleanup

* chore(test): switch from microk8s to kind

* chore(makefile): define target `integration-test`

* test(integration): patch instead of sed, disable anonymous reports

* test(e2e): replace kubectl patch with kustomization

* test(e2e): bump kubectl wait timeout to account for slow pulls

* test(e2e): remove unused PROXY_IP variable

* test(e2e): write README

* test(e2e): add shebang to verify.sh files

* test(e2e): gitignore leftover stuff

.github/workflows/build-test.yaml

@@ -0,0 +1,80 @@
+name: Test
+
+on: [push]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Setup go
+      uses: actions/setup-go@v2
+      with:
+        go-version: '^1.15'
+
+    - name: Cache Go modules
+      uses: actions/cache@v1
+      with:
+        path: ~/go/pkg/mod
+        key: ${{ runner.os }}-build-codegen-${{ hashFiles('**/go.sum') }}
+        restore-keys: |
+          ${{ runner.os }}-go
+
+    - name: 'Check out the repo'
+      uses: actions/checkout@v2
+
+    - name: 'make container'
+      run: |
+        make REGISTRY=local IMGNAME=kong-ingress-controller TAG=ci container
+
+    - run: docker save local/kong-ingress-controller:ci > image.tar
+
+    - uses: actions/upload-artifact@v2
+      with:
+        name: image
+        path: image.tar
+
+  push-public:
+    needs: build
+    runs-on: ubuntu-latest
+    if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/next'
+
+    steps:
+    - uses: actions/download-artifact@v2
+      with:
+        name: image
+
+    - name: 'Arrange: Push the Kong Operator image to the public registry'
+      env:
+        BINTRAY_KEY: ${{  secrets.BINTRAY_KEY }}
+        BINTRAY_USER: ${{  secrets.BINTRAY_USER }}
+      run: |
+        set -e
+
+        REPO="kong-docker-kubernetes-ingress-controller.bintray.io"
+        REF="$(git rev-parse --short HEAD)"
+
+        DOCKER_TAG="$REPO/master:$REF"
+        docker load < image.tar
+        docker tag local/kong-ingress-controller:ci "$DOCKER_TAG"
+        echo "${BINTRAY_KEY}" | docker login -u "${BINTRAY_USER}" ${REPO} --password-stdin
+        docker push "$DOCKER_TAG"
+
+  test-k8s:
+    needs: build
+    runs-on: ubuntu-latest
+    steps:
+    - name: 'Arrange: Check out the repo'
+      uses: actions/checkout@v2
+
+    - uses: actions/download-artifact@v2
+      with:
+        name: image
+
+    - name: 'Arrange: Push the image to the local registry'
+      run: |
+        docker load < image.tar
+        docker tag local/kong-ingress-controller:ci localhost:32000/kong-ingress-controller:ci
+
+    - name: 'Test: Run integration tests'
+      run: |
+        env KIC_IMAGE=local/kong-ingress-controller:ci ./test/integration/test.sh

.gitignore

@@ -37,3 +37,7 @@ kong-ingress-controller
 
 # ignore vendor tree
 vendor
+
+# integration test residuals
+kind
+kubeconfig-test-cluster

CONTRIBUTING.md

@@ -41,7 +41,7 @@ patch](#submitting-a-patch) section.
 
 ## Dependencies
 
-The build uses dependencies are managed by [go modules](https://blog.golang.org/using-go-modules) 
+The build uses dependencies are managed by [go modules](https://blog.golang.org/using-go-modules)
 
 ## Running in dev mode
 
@@ -126,6 +126,8 @@ $ cd $GOPATH/src/github.com/kong/kubernetes-ingress-controller
 $ make test
 ```
 
+To run integration tests, see the [integration test readme](test/integration/README.md).
+
 ## Releasing
 
 Makefile will produce a release binary, as shown above. To publish this

Makefile

@@ -50,3 +50,7 @@ container:
 .PHONY: run
 run:
 	./hack/dev/start.sh ${DB} ${RUN_VERSION}
+
+.PHONY: integration-test
+integration-test: container
+	KIC_IMAGE="${IMAGE}:${TAG}" ./test/integration/test.sh

test/integration/README.md

@@ -0,0 +1,68 @@
+## E2E test
+
+The end-to-end (E2E) integration test performs the following:
+
+1. Start up an ephemeral Kubernetes cluster (KIND)
+1. Install a container registry (localhost:5000)
+1. Push the Kong Ingress Controller image to localhost:5000/kic:local
+1. Run Kong Ingress Controller in the local Kubernetes cluster
+1. Start up example services (echo and httpbin)
+1. Run test cases (`/test/integration/cases/*`):
+    1. Apply manifests from `.yaml` files present in the test case directory
+		1. Wait for KIC to apply the configuration
+		1. Run ./verify.sh to verify assertions
+		1. Delete manifests created in the apply step above.
+
+### How to run the test
+
+```bash
+make integration-test
+```
+
+This command builds a KIC Docker image and runs the test against that image.
+
+It is possible to run the test against any prebuilt KIC image, skipping the build step:
+
+```bash
+env KIC_IMAGE=some-kic-image:tag ./test/integration/test.sh
+```
+
+### Troubleshooting
+
+If you want to troubleshoot a specific test case, here's how to do it:
+
+#### Run tests with `SKIP_TEARDOWN=yes`
+By passing `SKIP_TEARDOWN=yes` to the test you can inspect the test environment after failure, and run certain test cases manually:
+
+```bash
+make SKIP_TEARDOWN=yes integration-test
+# or
+env KIC_IMAGE=some-kic-image:tag SKIP_TEARDOWN=yes ./test/integration/test.sh
+```
+
+#### Access the test cluster with `kubectl`
+
+After the test invocation command with `SKIP_TEARDOWN=yes` set terminates, KIND will continue running in the cluster. You can access it using `./kubeconfig-test-cluster` as kubeconfig:
+
+```bash
+kubectl --kubeconfig=./kubeconfig-test-cluster get pods --all-namespaces
+```
+
+#### Run a test case manually
+
+You can run a test case manually:
+```bash
+kubectl --kubeconfig=./kubeconfig-test-cluster port-forward -n kong svc/kong-proxy "27080:80" "27443:443"
+
+# in a separate terminal window:
+kubectl --kubeconfig=./kubeconfig-test-cluster apply -f ./test/integration/cases/01-https
+env SUT_HTTP_HOST=127.0.0.1:27080 SUT_HTTPS_HOST=127.0.0.1:27443 ./test/integration/cases/01-https/verify.sh
+kubectl --kubeconfig=./kubeconfig-test-cluster delete -f ./test/integration/cases/01-https
+```
+
+#### Manually tear down the test cluster
+
+At the end of the debugging session, you can tear down the environment like this:
+```bash
+docker rm -f test-cluster-control-plane test-local-registry
+```

test/integration/cases/00-ingress-match-type/ingress.yaml

@@ -0,0 +1,44 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: echo
+  annotations:
+    kubernetes.io/ingress.class: kong
+spec:
+  rules:
+  - http:
+      paths:
+      - backend:
+          service:
+            name: echo
+            port:
+              number: 8080
+        path: /foo
+        pathType: Prefix
+  - http:
+      paths:
+      - backend:
+          service:
+            name: echo
+            port:
+              number: 8080
+        path: /bar1
+        pathType: Exact
+  - http:
+      paths:
+      - backend:
+          service:
+            name: echo
+            port:
+              number: 8080
+        path: /bar2/
+        pathType: Exact
+  - http:
+      paths:
+      - backend:
+          service:
+            name: echo
+            port:
+              number: 8080
+        path: "/baz+/"
+        pathType: ImplementationSpecific

test/integration/cases/00-ingress-match-type/verify.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+set -ex
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/)" == 404 ]
+
+# Match type: Prefix
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/foo)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/foo/)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/fooo)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/fooo/)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/fooo/xxx)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/foo/xxx)" == 200 ]
+
+# Match type: Exact
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar1)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar1/)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar1/xxx)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar2)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar2/)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar2/xxx)" == 404 ]
+
+# Match type: ImplementationSpecific
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/baz)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/baz/)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bazzz/)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bazzz/xxx)" == 200 ]

test/integration/cases/01-https/ingress.yaml

@@ -0,0 +1,37 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: echo-http
+  annotations:
+    kubernetes.io/ingress.class: kong
+    konghq.com/protocols: http
+spec:
+  rules:
+  - http:
+      paths:
+      - backend:
+          service:
+            name: echo
+            port:
+              number: 8080
+        path: /foo
+        pathType: Prefix
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: echo-https
+  annotations:
+    kubernetes.io/ingress.class: kong
+    konghq.com/protocols: https
+spec:
+  rules:
+  - http:
+      paths:
+      - backend:
+          service:
+            name: echo
+            port:
+              number: 8080
+        path: /bar
+        pathType: Prefix

test/integration/cases/01-https/verify.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+set -ex
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/foo)" == 200 ]
+[ "$(curl -k -sw '%{http_code}' -o /dev/null https://$SUT_HTTPS_HOST/foo)" == 200 ]
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar)" == 426 ]
+[ "$(curl -k -sw '%{http_code}' -o /dev/null https://$SUT_HTTPS_HOST/bar)" == 200 ]
+

test/integration/cases/02-ingress-apiversions/ingress.yaml

@@ -0,0 +1,32 @@
+apiVersion: networking.k8s.io/v1beta1
+kind: Ingress
+metadata:
+  name: echo-networkingv1beta1
+  annotations:
+    kubernetes.io/ingress.class: kong
+spec:
+  rules:
+  - http:
+      paths:
+      - backend:
+          serviceName: echo
+          servicePort: 8080
+        path: /bar
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: echo-networkingv1
+  annotations:
+    kubernetes.io/ingress.class: kong
+spec:
+  rules:
+  - http:
+      paths:
+      - backend:
+          service:
+            name: echo
+            port:
+              number: 8080
+        path: /baz
+        pathType: Prefix

test/integration/cases/02-ingress-apiversions/verify.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+set -ex
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/)" == 404 ]
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar/)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/baz/)" == 200 ]
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/qux/)" == 404 ]