Automate Kubernetes cluster QA tests

[emidoots]

Today our release process https://github.com/sourcegraph/sourcegraph/issues/13374 indicates you should create a Kubernetes cluster, deploy Sourcegraph on it, run the e2e test initializer on it, and run the regression test suite on it:

• New Sourcegraph Kubernetes cluster:
  • Run the initializer on a new Sourcegraph Kubernetes cluster.
  • Run the regression test suite.

There has to be a better way we can reduce the number of manual steps here, maybe using Pulumni or some other form of scripting for this. Let's investigate and find possible opportunities, even if we don't get around to actually implementing them in 3.21.

[pecigonzalo]

Isnt this already done by https://github.com/sourcegraph/deploy-sourcegraph/blob/master/.buildkite/integration-test.sh? While its not part of our main pipeline, I believe that does smoke testing.
This is not to say we cant improve that process.

[emidoots]

@pecigonzalo No, that doesn't run the e2e regression (QA) tests as the release checklist currently mandates. But that could be a sufficient replacement perhaps, given we run the QA tests against server. https://sourcegraph.slack.com/archives/CJX299FGE/p1603129425002100?thread_ts=1603129247.499700&cid=CJX299FGE

[pecigonzalo]

I was misled by the title as it said "smoke tests" which is what I though that repo was running as it just pings endpoints

[emidoots]

You're right, the title is misleading. Updating.

[daxmc99]

I synced with Uwe and came out with some takeaways:

• Adding additional automation to the release process is likely somewhat self defeating. These tests break often and require human intervention.
• If it can be automated, then it should be run as part of CI (like the docker e2e tests are). Or a subset of the tests could be run to reduce the break-fix work come release time.

[daxmc99]

I think this issue should be extended to include steps beyond smoke tests. Adding additional cluster validation seems fairly simple to do.

If we want to close this, I can finish up the work to only perform smoke tests.

This will be scoped to only basic cluster smoke tests, ideas around using the src-cli validate commands will be broken down into new issues.

This will be carried over onto the next iteration. We are going to run regression & e2e tests the same way we run docker-compose. There might be some duplicated work but K8s is one of our production deployment models and should have the same level of testing as docker-compose.

• These tests will run on the sourcegraph/sourcegraph repo. We already run other e2e tests here and that will increase the visibility of these tests.
• By default, it will use the head of deploy-sourcegraph UNLESS its a tagged branch, then it will use the equivalent tagged branch on the sourcegraph repo
• Update release process to add tagging of deploy-sourcegraph to the release process.