Gradle build-tools integration tests are slow and fail often

[mark-vieira]

We've had a high number of CI failures related to build-tools integration tests lately. Many of these tend to do with timeouts... of course.

The other issue here is that these tests are just horribly slow. They actually account for the bulk of one of our pull request checks.

This simply should not be the case as most PRs never change this stuff and most developers could care less about the result of these tests. We need to reevaluate what these tests are doing to see a) what is making them so damn slow and b) are they even providing any reasonable benefit. One thing we might consider is ripping testing of this project completely out of the general developer workflow and creating a parallel Jenkins build specifically for this stuff that triggers only on VCS changes to buildSrc/.

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

I think the buildexample plugins IT needs to be removed. It should be a separate type of test, done within each plugin example. and it shouldn't run tests, it should only run compilation because it's all about whether the project will build once it is extracted into its own project. Right now we are running all the example plugin tests (including integ test clusters) twice, but due to dependency ordering, the first time is within this monster "example plugins IT" and it has zero visibility into what actually goes wrong.

[rjernst]

Additionally, I think there should not be any dependency within build-tools on a built version of ES. All the integ tests need to use dummy/mock distributions/tools and such to test eg whether testclusters works. These need to be lightweight tests that get us 95% of the way to have confidence the build infrastructure will work. If we have gaps, they will be covered by the actual build running.

[mark-vieira]

These need to be lightweight tests that get us 95% of the way to have confidence the build infrastructure will work.

Absolutely. We need to mock out as much of this stuff as possible for these tests. Right now they are way too heavy. We are essentially running a ton of mini ES builds as part of this test suite.

[alpar-t]

Build example plugins also tests that build-tools works when applied to an external project.
This is also something we will have to cover when we remove it.

[alpar-t]

After giving it more taught, i think we do need to run tests for example plugins, or at least for one of them, we don't have a way to test that the infrastructure to do so ( cluster formation, testclsuters, etc ) works when used in an external project.

[mark-vieira]

I'm fine with keeping the coverage in some form. But it should be outside of normal of the normal CI intake pipeline so we aren't paying that cost for the majority of work that doesn't affect build-tools.

[alpar-t]

I'm not entirely convinced. One could make a point for a number of other features or plugins resulting in each plugin having it's own CI setup.

[rjernst]

I agree with @mark-vieira this should be a separate CI job. This is not something likely to break very often. We can have simple integ tests that check the build tools work as expected which are run always, that use fake projects in testkit, mimicking external behavior, and have a separate test task for running the example plugins builds in this way. This is similar to how we have packagingTest that is not run for most builds. The new test task can also be triggered on PRs by using the :Core/Infra/Build label, like how we trigger full packaging tests for :Core/Infra/Packaging PRs.

[mark-vieira]

One could make a point for a number of other features or plugins resulting in each plugin having it's own CI setup.

You say this in jest, but honestly I would be in favor of moving this direction and away from such a monolithic CI setup as we have now 😉

[mark-vieira]

The new test task can also be triggered on PRs by using the :Core/Infra/Build label, like how we trigger full packaging tests for :Core/Infra/Packaging PRs.

👍

[alpar-t]

Closing this as we switched to unit tests