Display test failures in the gradle check status report.

[dblock]

Signed-off-by: Daniel (dB.) Doubrovkine dblock@amazon.com

Description

Display test failures in the gradle check status report.
I tested this by hard-coding a few failed build URIs, so hopefully this doesn't break everything, but to actually open the PR we need a pull request target to work.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6756/
• CommitID: 44a6755
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6760/
• CommitID: 4c41428
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/6755/
• CommitID: b18fe69

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6758/
• CommitID: 7f2e175
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6759/
• CommitID: ac53738
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[reta]

@dblock I think we could use community provided test reporters, fe [1], what do you think?

[1] https://github.com/marketplace/actions/test-reporter

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6761/
• CommitID: 731aa1f
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6764/
• CommitID: 2e79652
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6767/
• CommitID: 4e80ab8
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6769/
• CommitID: 560f26f
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dblock]

@dblock I think we could use community provided test reporters, fe [1], what do you think?

[1] https://github.com/marketplace/actions/test-reporter

Yeah that's a good idea. Would need to fetch the test results from gradle check to use, and this is not very reassuring:

java-junit (Experimental)
Support for [JUnit](https://junit.org/) XML is experimental - should work but it was not extensively tested. To have code annotations working properly, it's required your directory structure matches the package name. This is due to the fact Java stack traces don't contain a full path to the source file. Some heuristic was necessary to figure out the mapping between the line in the stack trace and an actual source file.

[andrross]

I've tested the curl / jq command and it seems to work. Is there any way to test this end-to-end without just committing it to main?

[andrross]

Maybe use uniq -c to get a count of failures?

[dblock]

I wanted to display the actual failures instead.

[andrross]

-c will give you both the failure and the count:

curl -s "https://build.ci.opensearch.org/job/gradle-check/6876/testReport/api/json?tree=suites\[cases\[status,className,name\]\]" | jq -r '.. | objects | select(.status=="FAILED",.status=="REGRESSION") | (.className + "." + .name)' | uniq -c
      2 org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests.testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections
      4 org.opensearch.index.mapper.GeoPointFieldMapperTests.testLatLonInArrayMoreThanThreeValues

[andrross]

Actually, I'd add | sort -n -r | head -n 10 to the command as well to put the most failures at the top and have a limit so we don't spam the correspondence in the case things go badly.

[dblock]

Added. Nice bash-fu @andrross :)

[andrross]

Also I agree there are probably better ways to do this but this seems like a quick and easy improvement that can be done now

[dblock]

@andrross This PR is just a lazy feature, you can click on the gradle check and see the test failures. So I don't feel particularly compelled to merge it, but if you think it's useful, maybe you can convince @reta of its benefits? :)

[reta]

@andrross This PR is just a lazy feature, you can click on the gradle check and see the test failures. So I don't feel particularly compelled to merge it, but if you think it's useful, maybe you can convince @reta of its benefits? :)

haha @dblock , I am absolutely not against it, just wanted to make integration of such a feature as simple as possible (using available Github actions but it seems like we need to allow external actions on repository explicitly)

[andrross]

The test-reporter action seems to put the results on the github action screen (which you'd have to click to get to) and doesn't seems like it would add much value given that we have a handy link to the test results anyway. Am I misinterpreting that?

I like that this solution ends up being very in-your-face about the flaky failures and you don't have to click anything to see what failed. Might be more motivation to help drive these down :)

[reta]

The test-reporter action seems to put the results on the github action screen (which you'd have to click to get to) and doesn't seems like it would add much value given that we have a handy link to the test results anyway. Am I misinterpreting that?

I like that this solution ends up being very in-your-face about the flaky failures and you don't have to click anything to see what failed. Might be more motivation to help drive these down :)

@andrross fair enough, let's give it a try?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6923/
• CommitID: 8dc990c
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6925/
• CommitID: 45ed16a
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/6926/
• CommitID: a07bbb2