Templated arm64 integration and e2e workflows for main and release-3.5

[jmhbnz]

Follow-on from #16151 to setup scheduled workflows for arm64 nightly integration and e2e suites against the release-3.5 branch.

Arm64 integration and e2e suites now adopt a shared re-usable workflow template which is slightly customised for main or release-3.5.

Fixes #15832

[jmhbnz]

Note for reviewers: Workflows have been tested on: pull_request and were green for both main and release-3.5:

• e2e: https://github.com/etcd-io/etcd/actions/runs/5409330146
• integration: https://github.com/etcd-io/etcd/actions/runs/5409330130

[serathius]

tldr; we need to update the release process to incorporate those workflow failures as a signal.

I think we are stomping into new ground. Periodic workflows themselves are not enough to guarantee that next etcd release will work on ARM. They will run, fail but not result in any action if they fail. We need to make them somehow actionable.

To paint the picture. Example of what I mean is process that K8s has. First a dashboard that gives clear view of all periodic tests for all supported versions. Dashboard for v1.24 testing I own https://k8s-testgrid.appspot.com/sig-instrumentation-tests#kind-json-logging-1.24
If there is a repeated test failure, an owner of such test scenario will be notified by email. Independently the K8s release team is monitoring same test failures on their dashboard https://k8s-testgrid.appspot.com/sig-release-1.27-blocking#gce-cos-k8sstable1-default. If a particular test failure blocks a release, they will ping the test owner to fix it.
Before making a release they check that all runs are green.

I think to make the periodic workflows failures be a signal for release we need to adapt to a process similar to this. Of course we will not try to rollout TestGrid ourselves, but as etcd joins Kubernetes it would be nice to have some integration. We already have a release process and release owners defined, we just need to make results from those periodics a clear signal for our releases.

[jmhbnz]

I think to make the periodic workflows failures be a signal for release we need to adapt to a process similar to this. Of course we will not try to rollout TestGrid ourselves, but as etcd joins Kubernetes it would be nice to have some integration. We already have a release process and release owners defined, we just need to make results from those periodics a clear signal for our releases.

Completely agree. Additionally I am intending to scale up our runners in future to have multiple on each box (we have tons of compute idle on each box) so that we can actually run some of these workflows on each pr also.

Edit: The scaling of runners to enable arm64 signal on pr is tracked under: #15951

[jmhbnz]

Hey @serathius, @ahrtr - Can you please take another look at this? I'm keen to see how these tests will behave against 3.5.

[ahrtr]

LGTM

Thanks @jmhbnz

I think we can discuss the adapting the similar K8s process in a separate session. Leave to @serathius to approve and merge this PR.