Test Dev Fleet within released Rancher

[manno]

Rancher could be (nightly, rc, release).

How to install Rancher with our fleet build?

• switching the rancher-charts branch with a helm value at installation time
  • we'd need a write token for test-fleet in rancher/charts
  • cannot use a different server, because of Add ChartDefaultURL to support custom rancher-charts repo in testing rancher#40281
• using airgap mode ("bundled") and building our own repo to install with dev fleet
  • setup a minimal, local helm chart repo
  • unless we run rancher outside the cluster, with a local repo dir, we need to ADD our repo over the clone in the rancher/rancher image
• Add another layer to the Rancher image, which contains our bundled fleet version in the airgap repo

Running outside the cluster, without the rancher helm chart, could be interesting for vscode&integration tests
E2e CI probably needs the modified image (or a branch in rancher/charts).

What are existing upgrade tests in Fleet:

• Upgrade Fleet in Rancher to Latest Release - red
  Upgrade fleet in specific Rancher versions to latest fleet release and run MC tests
• E2E Rancher Fleet - green (last run months ago)
  Upgrade fleet in latest Rancher to dev version and run MC tests
• Upgrade Fleet in Rancher - disabled

How effective are our existing rancher integration tests?

• Only testing simple, short-lived deployments
• Existing multi-cluster tests
• Upgrade from one Rancher version to the next?
• Installation with special options?
• Does Fleet work in previous Rancher versions, too? (fleet-upgrade-in-rancher.yaml had a nice matrix)
• ...

Problem

• Current tests only upgrade after the installation. This relies on a modified helm chart, with a very high version number.
  Do we observe downgrades to the released fleet chart?
• Are current test runners sufficient? Move the tests to the AKS or GKE workflow?
• Tests could use our 'test infra' with registry and git server?
• Tests could profit from version pinning PR

[prabalsharma]

Elemental has CI to install RM- we can reuse that as 1st part of this pipeline work, i.e Fleet with Rancher Manager

[manno]

• re-open PR for custom git server - see rancher/rancher #43530
• use in a github action to validate PRs

[weyfonk]

From Rancher 2.9 onwards, it will be possible to configure alternative chart repositories, which should ease testing of any Fleet commit within Rancher.

Additional work, with WIP actions to release Fleet to a test charts repo for testing and using that to deploy Rancher through GCP-hosted Github runners, is available on branch test-fleet-in-latest-rancher.

[manno]

We're waiting for Github Action private runners.

[weyfonk]

We need to request access to a non-containerised runner, see this.
Latest progress available on this branch.

[weyfonk]

Non-containerised runners now work, but registering a downstream cluster times out in CI, although it works locally.
We'd need to be able to know more about the state of Rancher on the runner to understand what prevents downstream cluster registration from working.

Could we run a pipeline against a fork (yes, physically possible), using a non-containerised runner (question sent to EIO in that regard, as per-repo authorisation is needed) with tmate (enabled temporarily until we find and fix the root cause of the issue)?

[weyfonk]

• Update docs/testing_fleet_in_rancher.md