drone: Replace tag build pipelines with GHA workflows #35235
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I did a test run yesterday: https://drone.platform.teleport.sh/gravitational/teleport/31266 - this is the tag build running all steps on GHA. I have kicked off another test run now since I've made minor ordering changes since yesterday. I don't think that can cause any regression, but double-checking anyway. Will link the drone run when it's completed. |
fheinecke
approved these changes
Nov 30, 2023
Unroll the loops over architectures, fips and package type and just list all the pipelines explicitly. This makes it a bit easier to understand the combinations and to replace them with GitHub actions workflows. Running `make dronegen` results in no changes to `.drone.yml`.
Convert some of the linux-based tag build pipelines to run on GitHub
Actions. The following pipelines have been converted:
build-linux-amd64
build-linux-amd64-centos7
build-linux-amd64-centos7-fips
build-linux-386
build-linux-arm
build-linux-amd64-deb
build-linux-amd64-deb-fips
build-linux-amd64-centos7-rpm
build-linux-amd64-centos7-fips-rpm
build-linux-386-deb
build-linux-386-rpm
build-linux-arm-deb
build-linux-arm-rpm
The GHA workflows builds tarballs as well as deb/rpm packages in the one
workflow, so the `-deb` and `-rpm` pipelines will need to be manually
removed from `.drone.yml`. The amd64 centos7 and non-centos7 pipelines
have been combined as they were calling the same `make` target
duplicating work. The amd64 build is always done on centos7. As a
result, we do not name the pipeline with -centos7 any more, but we do
still specify it as the build.assets `Makefile` still has a centos7
target which is called, and we do still release an asset named with
"centos7".
Still remaining of the linux-based tag build pipelines are the arm64
pipelines which are already converted using a different workflow and the
non-native windows build.
Convert the build-linux-arm64 and push-build-linux-arm64 pipelines to use the common `release-linux` workflow instead of the arm64-specific `release-linux-arm64` workflow. This aligns it with the other linux build pipelines and allows us to get rid of the specific workflow. The pipelines for building the arm64 rpm and deb packages have not been generated by dronegen for some time now - since the arm64 build was converted to GitHub Actions. The OS packages were still built as dronegen does not remove pipelines, so the existing pipelines from before the GHA migration remained in `.drone.yml` and continued to run. These os packaging pipelines will be manually removed in a subsequent commit.
fb602b8 to
3d7cecf
Compare
|
The tag build almost passed: https://drone.platform.teleport.sh/gravitational/teleport/31318. I'm going to ignore this and merge. Everything worked as expected and the previous test run passed. I have since added |
Generate a pipeline for calling a GitHub Actions workflow to generate the legacy AMIs. There were two existing manually added pipelines - `build-oss-amis` and `build-ent-amis` - that are replaced by this. The new pipeline needs to be manually added and the old ones manually removed.
Replace the `teleport-container-images-branch-tag` workflow that builds the legacy and operator OCI images with a call to the GitHub Actions workflow that does the same on GitHub Actions. This requires the manual addition of the `build-oci` pipeline and manual removal of the `teleport-container-images-branch-tag` pipeline, followed by running `make dronegen` to flesh out `build-oci` and sign .drone.yml.
Remove the now-unused functions and vars after converting pipelines to calling GitHub Actions instead of running stuff on Drone.
Update .drone.yml by running `make dronegen` to update the following
pipelines to call GitHub Actions to build instead of building on Drone:
build-linux-amd64
build-linux-amd64-fips
build-linux-386
build-linux-arm64
build-linux-arm
Add two new pipelines for building AMIs and OCIs on GHA:
build-legacy-amis
build-oci
Remove the following pipelines as the build of deb/rpm packages are done
within the above pipelines on GitHub Actions now and the ami/oci
pipelines have been replaced:
build-linux-amd64-deb
build-linux-amd64-fips-deb
build-linux-amd64-centos7-rpm
build-linux-amd64-centos7-fips-rpm
build-linux-386-deb
build-linux-386-rpm
build-linux-arm64-deb
build-linux-arm64-rpm
build-linux-arm-deb
build-linux-arm-rpm
build-oss-amis
build-ent-amis
teleport-container-images-branch-tag
Remove the following pipelines as AMD64 builds are always centos7 builds, but
we were just doing it twice. No need for these any more, as the GHA workflow
will build the release artifacts for these with the centos7 targets:
build-linux-amd64-centos7
build-linux-amd64-centos7-fips
The pipelines were added/removed using the following script, followed by
`make dronegen`:
AWK_SCRIPT='
/^---$/ { printf "%s", accumulator; accumulator = "" }
/^---$/ || accumulator { accumulator = accumulator $0 "\n" }
/^name: / {
drop = $2 == to_remove
if ($2 == before && to_add) {
printf "---\nname: %s\n", to_add
}
if (!drop) { printf "%s", accumulator }
accumulator = ""
next
}
!drop && !accumulator { print }
ENDFILE { printf "%s", accumulator }'
toremove=(
build-linux-amd64-{centos7,centos7-fips}
build-linux-amd64-{deb,fips-deb,centos7-rpm,centos7-fips-rpm}
build-linux-386-{deb,rpm}
build-linux-arm64-{deb,rpm}
build-linux-arm-{deb,rpm}
build-{oss,ent}-amis
teleport-container-images-branch-tag
)
add_before=build-buildboxes
toadd=(
build-legacy-amis
build-oci
)
for pipeline in "${toremove[@]}"; do
gawk -i inplace -v to_remove=$pipeline "$AWK_SCRIPT" .drone.yml
done
for pipeline in "${toadd[@]}"; do
gawk -i inplace -v to_add=$pipeline -v before=$add_before "$AWK_SCRIPT" .drone.yml
done
3d7cecf to
22646ab
Compare
This was referenced Dec 8, 2023
This was referenced Dec 10, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replace the remaining tag build pipelines that are currently built on
drone with calls to GitHub Actions workflows instead. Also update the
build-linux-arm64 pipeline to call the common release-linux workflow
instead of the arm64-specific release-linux-arm64 workflow.
The following pipelines were updated:
The following pipelines were added:
The following pipelines were removed:
The deb/rpm pipelines are no longer needed as the GHA workflow for
building linux release builds these OS packages. The centos7 pipelines
are not needed as they generate the same artifacts as the non-centos7
pipelines (this removes some redundant build steps). The oss/ent ami
pipelines were manually added before and the pair of them have been
replaced with a single GHA workflow. The container images pipeline is
replaced with the build-oci pipeline, building the legacy OCIs and the
operator OCIs.
Issue: #20729
This PR is broken up into reviewable commits that perform the
transformations in small steps. Feel free to review in one big bang, but
I think it's easier to review the individual commits.
Note: This PR supercedes #32894, which never got merged due to some
messy dependencies between the repos. That's been cleared up so I've
folded it into this PR which adds the linux-arm64 change, legacy-amis
and build-oci pipelines.