Flaky test comments #754

v1v · 2020-10-06T12:54:24Z

What does this PR do?

Support flaky test reporting, for such it's required to:

lookForGitHubIssues to query the open GitHub issues that got a match with the given test-failures.
githubCreateIssue to return the issue that has been created.
enable errorDetails for the test errors.
Create templates for creating GH issues
Create template for the GH comment for flakey tests.
gh wrapper.
githubIssues to query all the open GitHub issues with the gh wrapper.

Important

This particular process will create GitHub issues if no match with the labels flaky-test,ci-reported.
The look up will filter all the issues that are not open and don't have the labels flaky-test,ci-reported.
The test-failure name match with the GitHub issues are based on the title with the filter contains.
gh has been configured to retrieve up to 200 GH issues.

Why is it important?

Provide more information when tests are failing and whether they were reported as flakey too.

Related issues

Relates to #753

Tasks

UTs for vars/githubCreateIssue.groovy
lookForGitHubIssues
Resilience if issue creation fails, to retry once more

Screenshots

When there are matching flakey tests:

When there is an issue creation regarding the flakey test:

Tests

ITs for gh

apmmachine · 2020-10-06T13:00:13Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [Pull request #754 updated]
Start Time: 2020-10-13T14:09:39.616+0000
Duration: 8 min 25 sec

Test stats 🧪

Test	Results
Failed	0
Passed	834
Skipped	18
Total	852

v1v · 2020-10-06T15:36:58Z

resources/flakey-github-comment-markdown.template

@@ -0,0 +1,17 @@
+## :bug: Flaky test report


An example of the output from the UTs:

v1v · 2020-10-06T15:37:01Z

resources/flakey-github-issue.template

@@ -0,0 +1,17 @@
+## Flaky Test


An example of the UTs output

v1v · 2020-10-06T16:31:05Z

jenkins run the tests please

…orting * upstream/master: docs: update CHANGELOG.md [maven-release-plugin] prepare for next development iteration [maven-release-plugin] prepare release v1.1.148 Fix formatting in flake report (#763) slack message for no push events (#762) ci: add workers to the infra test (#759) Update github-comment-markdown.template (#758) [DOCS] Add badge details (#757)

v1v · 2020-10-12T16:31:06Z

vars/githubCreateIssue.groovy

+    def value = sh(label: 'Create GitHub issue', script: "hub issue create ${title} ${description} ${assign} ${labels} ${milestone}", returnStdout: returnStdout).trim()
+    if(returnStdout) {
+      return value
+    }


Probably we don't need this but always returning the stdout

v1v · 2020-10-12T17:11:08Z

There is a dependency regarding the gh in the jenkins instance, since the notifyBuildResult runs in node('master || metal || immutable'){, waiting for the infra team to confirm if we can do that. So far, there is a PR to support gh in the workers, but it might not be enough for our use case.

The reason for using gh was to avoid the pagination when using the github api

cachedout

💯 Nice work!

resources/flakey-github-issue.template

resources/scripts/generate-build-data.sh

cachedout · 2020-10-13T07:25:07Z

src/co/elastic/NotificationManager.groovy

-      githubPrComment(message: msg, commentFile: 'flakey.id')
+
+    def labels = 'flaky-test,ci-reported'
+    def tests = lookForGitHubIssues(flakeyList: ret, labelsFilter: labels.split(','))


We are mixing two spellings here. We have flakey and flaky. I think we should standarize on flaky wherever possible.

cachedout · 2020-10-13T07:25:29Z

src/co/elastic/NotificationManager.groovy

+      def issue = v
+      if (!v?.trim()) {
+        def issueDescription = buildTemplate([
+          "template": 'flakey-github-issue.template',


Same comment above flaky versus flakey

…orting * upstream/master: Return PR URL with githubCreatePullRequest step (#765)

Co-authored-by: cachedout <mike.place@elastic.co>

…-library into feature/flaky-reporting * 'feature/flaky-reporting' of github.com:v1v/apm-pipeline-library: Apply suggestions from code review

v1v · 2020-10-13T12:19:44Z

I just updated all the new references from flakey to flaky :)

Flakey test comments

a54d6c9

v1v self-assigned this Oct 6, 2020

v1v added the automation label Oct 6, 2020

v1v added 7 commits October 6, 2020 14:13

Add flakey issue generation

ff60eb8

Support stacktrace

e8870e7

Fix leftovers

e89b99d

Mock lookForGitHubIssues and UTs

2212784

Add support for stacktraces if they match the flakey test

a52bf71

Add stacktrace for the tests-errors file only

7ab01dd

Add UTs

9e3890e

v1v commented Oct 6, 2020

View reviewed changes

resources/flakey-github-issue.template Outdated

@@ -0,0 +1,17 @@

## Flaky Test

Copy link

Member Author

v1v Oct 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An example of the UTs output

v1v added 2 commits October 6, 2020 16:56

UTs for githubCreateIssue

dc6f28d

Add labels filter

5e7d451

v1v added the enhancement New feature or request label Oct 6, 2020

v1v added 4 commits October 12, 2020 16:10

Support gh with ITs and UTs

a707b48

Support githubIssues step with UTs

290d843

Support lookForGitHubIssues with UTs

9af09fa

v1v marked this pull request as ready for review October 12, 2020 16:27

v1v requested a review from a team October 12, 2020 16:27

Update docs

f22443a

botelastic bot added groovy master labels Oct 12, 2020

v1v commented Oct 12, 2020

View reviewed changes

cachedout reviewed Oct 13, 2020

View reviewed changes

kuisathaverat approved these changes Oct 13, 2020

View reviewed changes

v1v and others added 5 commits October 13, 2020 12:59

Merge remote-tracking branch 'upstream/master' into feature/flaky-rep…

e8324c0

…orting * upstream/master: Return PR URL with githubCreatePullRequest step (#765)

Use flaky instead flakey in the new steps

7f40246

Apply suggestions from code review

c48d480

Co-authored-by: cachedout <mike.place@elastic.co>

Merge branch 'feature/flaky-reporting' of github.com:v1v/apm-pipeline…

97374ab

…-library into feature/flaky-reporting * 'feature/flaky-reporting' of github.com:v1v/apm-pipeline-library: Apply suggestions from code review

Use flaky instead flakey in the new steps

540dba0

v1v changed the title ~~Flakey test comments~~ Flaky test comments Oct 13, 2020

Merge branch 'master' into feature/flaky-reporting

4959399

Merge branch 'master' into feature/flaky-reporting

1dab66d

cachedout approved these changes Oct 14, 2020

View reviewed changes

v1v merged commit 74cf485 into elastic:master Oct 14, 2020

v1v deleted the feature/flaky-reporting branch October 14, 2020 10:18

This was referenced Oct 15, 2020

[CI][flaky] reporting for PRs in GitHub elastic/beats#21853

Merged

[CI] enable test reporting for flaky results elastic/apm-agent-php#241

Merged

v1v mentioned this pull request Dec 8, 2020

[ci][flaky] reporting elastic/apm-agent-nodejs#1910

Closed

1 task

v1v mentioned this pull request Dec 17, 2020

[ci][flaky] reporting elastic/apm-agent-java#1563

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flaky test comments #754

Flaky test comments #754

v1v commented Oct 6, 2020 •

edited

Loading

apmmachine commented Oct 6, 2020 •

edited

Loading

Build stats

Test stats 🧪

v1v Oct 6, 2020

v1v Oct 6, 2020

v1v commented Oct 6, 2020

v1v Oct 12, 2020

v1v commented Oct 12, 2020

cachedout left a comment

cachedout Oct 13, 2020

cachedout Oct 13, 2020

v1v commented Oct 13, 2020

Flaky test comments #754

Flaky test comments #754

Conversation

v1v commented Oct 6, 2020 • edited Loading

What does this PR do?

Important

Why is it important?

Related issues

Tasks

Screenshots

Tests

apmmachine commented Oct 6, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

v1v Oct 6, 2020

Choose a reason for hiding this comment

v1v Oct 6, 2020

Choose a reason for hiding this comment

v1v commented Oct 6, 2020

v1v Oct 12, 2020

Choose a reason for hiding this comment

v1v commented Oct 12, 2020

cachedout left a comment

Choose a reason for hiding this comment

cachedout Oct 13, 2020

Choose a reason for hiding this comment

cachedout Oct 13, 2020

Choose a reason for hiding this comment

v1v commented Oct 13, 2020

v1v commented Oct 6, 2020 •

edited

Loading

apmmachine commented Oct 6, 2020 •

edited

Loading