Move CI from Travis to GitHub Workflows

[DasSkelett]

Motivation

While Travis is a great tool, it has its problematic times. Currently almost every CI fails in one instance because of some network problems.
We already discussed this in the Discord a bit, and I think no one was really bound to Travis and the move to GitHub Workflows was generally supported.

Changes

This PR removes the .travis.yml file from the repository, so Travis is no longer triggered on merges.

Instead, I've created three workflow files according to the documentation.

1. The standard build workflow.

   • It is triggered with pull requests, to be more precise, with the opened, synchronize and reopened events. This is adjustable.
   • It contains to so-called jobs: I called them build and build_NetCore.
   • The build job:

     • The build job contains a matrix: a list of Mono versions to use and a list of build configurations. It does one pass for each possible configuration + mono combination (just like Travis did).
       The mono versions in the list are 5.20, 6.4, 6.6 and 6.8. So I removed 5.16 and 6.0 and added 6.8 again compared to our current Travis config. I'm happy to change the versions, this was basically just what I filled in to have something for this workflow.

     • It first removes the preinstalled Mono version (currently 6.4 on Ubuntu 18.04 workers, see this page)
       Then it adds the Mono repository for the Mono version of this pass, and installs mono-complete. This step, as on Travis, takes the longest (~10 minutes). Unfortunately apt packages are not cacheable.

     • Then it installs libcurl4-openssl-dev and xvfb.

     • Afterwards it tries to restore two caches for NuGet packages I've set up. They are keyed with the hash of the packages.config and all .csproj files, so every change to those files (potentially changing our dependencies) invalidates them. In this case the packages are just downloaded as usual before build.

     • The it's build time! Same command as before, nothing's changed there.

     • The ./build test+only command has a xvfb-run prefixed, this is needed because I couldn't get xvfb to run as a service as it did in Travis. But I don't think that's a problem.

     • If anything failed before, we get a nice message in the Discord. Note: This runs for PRs! So yes, we will get notifications for CI fails in PRs. This is easily changeable if it's too spammy.

   • The build_NetCore job:
     • No Mono versions specified, because we are building with dotnet. But the configuration matrix is there and contains Debug_NetCore and Release_NetCore.
     • We use an action maintained by GitHub itself to setup the NetCore environment with the right version, currently 3.1.1 (runtime) / 3.1.101 (SDK).
     • Same es before: Restore caches, build, test, send failure to Discord.

2. The deploy workflow

   • It runs only on pushes to master.
   • It runs with only one Mono version, the one we use for releases too. This simplifies a lot, and those nasty ifs for the deployment like in Travis are not needed. Currently Mono 5.20, not changed due to the well-known issues...
   • Same as before: Caches, build, test.
   • Plus, build the NetKAN Docker container and push it to the Hub. Fortunately we decided to implement this entirely in Cake, so we only have to set the right secrets (see below).
   • To push ckan.exe and netkan.exe to S3 I needed to do a bit more. Travis has a built in S3 support. Fortunately there was a third-party GH Action that claims to do this. I'm not sure if we want to keep the --delete option:

   "--delete permanently deletes files in the S3 bucket that are not present in the latest version of your repository/build."

   • Finally the Discord notification again. We will get this one regardless of job status, but this is not a problem, because this one is only run for master-pushes for one single Mono version.
     I couldn't really test this workflow, because it needs secrets set for the Docker Hub, to redeploy our Infra services and for the S3.

3. The release workflow:

   • Triggered when new client releases are created.
   • Installs one single Mono version, this is used to compile the release binaries.
   • Builds the macOS, deb and rpm packages in release configuration.
   • Get some data from the GH API needed to upload the files: upload_url and tag_name (i.e. the CKAN version)
   • Now the next part is unfortunately pretty repetitive, because we need to call the upload-release-asset action for every single file we want to upload...
     It uploads the ckan.exe, AutoUpdater.exe, ckan_*.deb, CKAN.dmg and ckan-*.rpm.
   • When everything is done uploading, we get a notification 🎉

You can click around in my repo to see what it looks like: https://github.com/DasSkelett/CKAN/actions
Examples:
Overview for one run of build for a PR: https://github.com/DasSkelett/CKAN/actions/runs/36093128
And how it looks on the corresponding PR page: DasSkelett#7
The log output of a release run after creating a new CKAN release: https://github.com/DasSkelett/CKAN/runs/428138462?check_suite_focus=true

Secrets

We (as in someone with admin right in this repository) need to set some secrets for this to work:

• DISCORD_WEBHOOK
• DOCKERHUB_PASSWORD
• DOCKERHUB_USERNAME maybe? Was not a secret in Travis, but could make sense.
• AWS_SECRET_ACCESS_KEY
• AWS_ACCESS_KEY_ID maybe? Was not a secret in Travis, but could make sense.

Future additions

We can extend this more in the future if we want.
It's really easiy to also upload artifacts to GitHub itself during actions. This could replace the S3 (or exist alongside) and we would have the artifacts more accessible here in the repository. We could upload an artifact during the deploy workflow (the one that is run after merges/pushes to master).
They are downlaodable via the API: https://developer.github.com/v3/actions/artifacts/

[DasSkelett]

Someone also needs to allow workflows in the repository settings :P

[techman83]

I'll spend some time looking at this during the week (have a full weekend ahead), but in principle I think this is a great idea. Travis was really good for the promotion of CI/CD, but there is a lot of competition now. Between being bought out and shedding a lot of staff, I wonder if the writing is on the wall for them. Though the service seems to be moving forwards again, as it had stagnated for some time.

[politas]

Travis sends emails on build failures, too. Do we want that going forward?

[politas]

The release work flow ... [i]nstalls one single Mono version, this is used to compile the release binaries.

This will speed up the release process massively!

[DasSkelett]

Travis sends emails on build failures, too. Do we want that going forward?

I also got emails from GitHub for failures when testing this workflow in my own repository. A quick search indicates that this is controlled in the user settings.

The release work flow ... [i]nstalls one single Mono version, this is used to compile the release binaries.

This will speed up the release process massively!

It's still not as much as I hoped, because installing Mono still takes around 10 minutes, but it doesn't have to wait for the previous builds to start (like Travis always started them in batches of three or something).
The same applies for the build workflow, all 8 possible matrix combinations plus the two NetCore builds start simultaneously, so it does finish faster (~10-12 minutes with GH workflow vs 20-30 minutes with Travis).

Theoretically we could speed up both the workflows running with a single Mono version even more by using the pre-installed, but currently 6.4.0 is preinstalled and there's the problem with Mono6 binaries not working on Windows...

[DasSkelett]

Replaced by #3085