java_test does not report about skipped tests

[or-shachar]

Problem:

When running bazel java_test target with junit test with annotation @Ignore - the test doesn't show up in the resulting test.xml

Report:

Simple test class:

// MySimpleTest.java
package com.example; 
 
import org.junit.Ignore;
import org.junit.Test; 
 
public class MySimpleTest {

    @Test
    public void happyTest(){
    	System.out.println("I am a happy test");
    } 

    @Test
    @Ignore("not ready yet")
    public void skippedTest() throws RuntimeException{
        throw new RuntimeException("Just skip me please");
    } 
} 

Running with java_test target.

I made a repro here:

# go to new folder
git clone --single-branch -b java_skipped_issue git@github.com:or-shachar/bazel-sample-repo.git .
bazel test //src/test/java/com/example:javatest
cat bazel-testlogs/src/test/java/com/example/javatest/test.xml

result will show

<?xml version='1.0' encoding='UTF-8'?>
<testsuites>
  <testsuite name='com.example.MySimpleTest' timestamp='1970-01-04T12:08:21.048Z' hostname='localhost' tests='2' failures='0' errors='0' time='0.004' package='' id='0'>
    <properties />
    <testcase name='happyTest' classname='com.example.MySimpleTest' time='0.004' />
    <testcase name='skippedTest' classname='com.example.MySimpleTest' time='0.0' />
    <system-out />
    <system-err />
  </testsuite>
</testsuites>

(The skipped testcase is listed with time 0 and is not counted as skipped)

OTH if you run mvn clean install you'd get:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" name="com.example.MySimpleTest" time="0.067" tests="2" errors="0" skipped="1" failures="0">
  <properties>
    ...
  </properties>
  <testcase name="happyTest" classname="com.example.MySimpleTest" time="0.002"/>
  <testcase name="skippedTest" classname="com.example.MySimpleTest" time="0">
    <skipped message="not ready yet"/>
  </testcase>
</testsuite>

(the skipped testcase is marked with skipped and is counted as skipped)

OS I tested on:

OS X and Linux

Releases:

release 0.19.1
release 0.18.1

Additional info

The bug propagates to rules_scala Specs2Junit test runner (cc: @ittaiz )

[or-shachar]

cc: @hmemcpy

[ittaiz]

@lberki @iirina can you help here?

[lberki]

I just parachuted into this, but it looks like we are explicitly not writing the <skipped/> tag for some reason?

https://github.com/bazelbuild/bazel/blob/master/src/java_tools/junitrunner/java/com/google/testing/junit/runner/model/AntXmlResultWriter.java#L161

@cushon , is this the correct diagnosis?

[cushon]

Maybe, I'm not very familiar with this code. There's some related discussion about test reporting here: #3201

[gmishkin]

Reading some of these discussions and seeing that updates were made however none were to generate the <skipped> element. I'm thinking of taking a crack at making it write that child element only for tests that are actually skipped, i.e. TestResult.Status.SKIPPED, not FILTERED (--test_filter?) or SUPPRESSED (@Ignored? I know that was the original purpose of this issue; I can also open a new one).

Would that be accepted?

[or-shachar]

@cushon @lberki - Is there any objection on adding support for writing the skipped xml node?

If not - we might get someone from our side to implement it.

@gmishkin - have you made any progress around it?

[gmishkin]

@or-shachar I haven't started on it yet because we decided that we didn't really care about this until there can be changes made to our internal CI that can alert us when a certain percentage of tests suddenly start being skipped. (this was inspired by a configuration mistake we made outside of Bazel that caused a bunch of tests to be skipped).