[CI] EnableSecurityOnBasicLicenseIT.testSecuritySetup failure

[dimitris-athanasiou]

Build scan: https://gradle-enterprise.elastic.co/s/ususuu432yrsu

Repro line:

./gradlew ':x-pack:plugin:security:qa:basic-enable-security:integTestRunner' --tests "org.elasticsearch.xpack.security.EnableSecurityOnBasicLicenseIT.testSecuritySetup" -Dtests.seed=EB6BC4DAFFD9A916 -Dtests.security.manager=true -Dtests.locale=be-BY -Dtests.timezone=America/Araguaina -Druntime.java=8

Reproduces locally?: No

Failure history:

It's been failing once in a while in the past 6 months.

https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-6M,mode:quick,to:now))&_a=(columns:!(_source),index:b646ed00-7efc-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:testSecuritySetup),sort:!(process.time-start,desc))

Failure excerpt:

org.elasticsearch.xpack.security.EnableSecurityOnBasicLicenseIT > testSecuritySetup FAILED
    org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:43207], URI [/_license], status line [HTTP/1.1 404 Not Found]
    { }
        at __randomizedtesting.SeedInfo.seed([EB6BC4DAFFD9A916:960DB491A93081CF]:0)
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
        at org.elasticsearch.test.rest.ESRestTestCase.getAsMap(ESRestTestCase.java:1222)
        at org.elasticsearch.xpack.security.EnableSecurityOnBasicLicenseIT.checkBasicLicenseType(EnableSecurityOnBasicLicenseIT.java:101)
        at org.elasticsearch.xpack.security.EnableSecurityOnBasicLicenseIT.testSecuritySetup(EnableSecurityOnBasicLicenseIT.java:68)

[elasticmachine]

Pinging @elastic/es-security (:Security/Security)

[ywangd]

I think root cause of this failure should be the same as #59157.

[martijnvg]

Another instance of this test failure occurred: https://gradle-enterprise.elastic.co/s/m4fphdcfdujz4/tests/:x-pack:plugin:security:qa:basic-enable-security:integTestRunner/org.elasticsearch.xpack.security.EnableSecurityOnBasicLicenseIT/testSecuritySetup#1

I think that #60498 doesn't fix this failure? Because a ResponseException isn't retried by assertBusy(...) in checkLicenseType() method, so in the case that a 404 is returned when the cluster isn't ready yet, the test still fails immediately.

[ywangd]

You are right @martijnvg I had a follow up PR (#60573) to throw an AssertionError on ResponseException. The failure you linked is before the PR, so it should be fixed by now. Backports are still pending.

[martijnvg]

Thanks @ywangd!