Switch to Azure Pipelines

[ofek]

Motivation

Consolidate CI providers. To be clear, Azure Pipelines was chosen for this monorepo (and a few other projects) over Travis, AppVeyor, Circle, and Gitlab.

What does this PR do?

Implements 2 pipelines:

• Changes: This triggers on every pull request that does not modify the base package and will always spin up a Linux (Ubuntu 16.04) and Windows (Windows Server 2019) VM. Any modified integrations will be tested on the main platform(s) they support.
• All: This triggers on every commit to master or any pull request that modifies the base package and will test every integration in its own runner.

Some integrations require additional set up such as the installation of system dependencies. As we only want these extra steps to occur when necessary, there is a set-up-integrations.yml template ran for every job that will detect what needs to be done and execute the appropriate scripts in .azure-pipelines/scripts/<integration>. As integrations may need different set up on different platforms, all scripts live under a directory named after the platform. All scripts in the directory will be executed in lexicographical order.

AZP reference entrypoint: https://aka.ms/tfbuild

Future

• I'll set up Codecov in a separate PR (it doesn't work OOTB)
• After the release freeze is over next week I'll merge this, enable the pipelines, and remove the current CI
• Get more runners
• Enable for integrations-extras and some other repos
• Implement caching when it's out of beta and there are Python examples
• We can optimize the Changes (pull request) pipeline by having a single job (probably self-hosted so it starts instantly) that spins up a runner for each changed integration instead of running each sequentially in one (or two) runners

Additional Notes

Pinging @martinwoodward @willsmythe @johnterickson @brcrista @jeremyepling @chrispat @infin8x since I noticed you helped a few other OSS projects switch (CPython, Rust, etc.). I tried to write idiomatic AZP configuration and made heavy use of templating (kudos btw, it's really powerful). Please let me know if this looks alright to you 😃

I also had a few questions that I couldn't quite find in the docs, maybe you know:

1. Can you restart individual failed jobs instead of the whole build?
2. Can you set the priority of pipelines? Say we have 50 runners and we make a commit to master that queues up a job/runner for all 120+ integrations and then a pull request is opened. Is there a way to ensure any newly available runners are allocated to the pull request before the master pipeline?
3. Are there plans to add options for different Microsoft-hosted machine capabilities on a per-job basis? We are particularly interested in higher memory and access to GPUs.
4. Do OSS projects that pay for more runners/parallel jobs keep the initial 10 e.g. to get 11 must one pay for 1 or 11?

[martinwoodward]

Very cool. Will take a look and let you know if I have any specific feedback, but looking fantastic so far. the templating stuff is definitely the way to go when you are tying to keep things maintainable.

While I'm v happy to see you using Pipelines, wanted to make sure you'd seen the latest GitHub Actions announcement? You should recognize a lot of the patterns and YAM syntax from your work with Pipelines so far and find it easy to adopt if it was interesting. If you signed up for the Beta than @ethomson would be a great person for you to work with on the GitHub actions end if that is interesting.

Trying to answer some of the questions to be best of my knowledge below:

1. I don't know of a way - perhaps the other folks might?
2. I don't think so. Machines are controlled at the pool level but with the hosted pool you tend to only have one level of access. What we do with large builds on self-hosted infrastructure is change the pool that they are in based on how much resource we want to give to them.
3. At the moment, the only way to do this is to self-host an image which has the memory or GPU that you need. Understood the reasons for this ask though, it's something we've definitely heard and makes sense but it's not yet on the immediate short term back-log
4. Yes, the 10 free are yours regardless of if you get more. We might be able to help add some here above the 10 you get free but why don't you take a look at GitHub Actions first and then see if this is something that you still want to do?

[brcrista]

Can you restart individual failed jobs instead of the whole build?

When you set up, did you install Azure Pipelines from the GitHub Marketplace? Then on the "Checks" tab in this PR, you will see each job that ran and can rerun failed ones.

You can check by going to Azure Pipelines and clicking "Edit" for your pipeline. If you are not using the Azure Pipelines GitHub app, it will pop up a banner with a one-click that will switch you over.

[ofek]

@martinwoodward

Thanks for answering my questions! I'm glad this looks alright to you.

As for GitHub Actions, I did this work right before the announcement so it wasn't an option at the time. Though from what I've seen it looks amazing! I think we'll continue with Azure Pipelines for now until it's a bit more fleshed out and the concurrent job limit is raised. Honestly, it's quite comforting GitHub Actions is (roughly) built atop Azure Pipelines, so we know it won't go away anytime soon 😄

Also, we would definitely appreciate any additional runners that could be spared. That is extremely generous, thank you very much!!!

@brcrista

Thanks! We did indeed set up the app but I can't find a way to do as you said on non-PR/master jobs like: https://dev.azure.com/ofekmeister/integrations-core/_build/results?buildId=249&view=logs&j=d74967ea-228d-5634-fe5f-5bd874cd0575

[brcrista]

Oh, outside of PRs, I don't think that feature is implemented.

[ofek]

We're switching today, thanks very much everyone!

[ofek]

Hello again everyone! Sorry for the bother but we ran into an issue.

We tried to purchase more parallel jobs but they are not getting applied to our public projects (which is our main use case). There is support ticket 119091924002451 open about this.

Can one of you please help with this, perhaps @martinwoodward?

[martinwoodward]

Ah - thanks ofek. Let me dig into that.

[martinwoodward]

Getting your open source job limit increased to 35. Should be increased soon.

[ofek]

@martinwoodward Thank you very much for fixing that! So going forward, will more purchases be automatically applied to our public projects or would we have to submit a ticket each time?

We plan to buy more either tonight or tomorrow morning since, as anticipated, our queue is still quite large:

[martinwoodward]

Do you want to drop me a mail with more details? - martinwo@microsoft.com