[Flaky Test] Configuration is not available after an update for more than 10 seconds

[peternied]

Many tests are seeing failures like the following

org.opensearch.OpenSearchException: java.util.concurrent.TimeoutException: Timeout after 10SECONDS while retrieving configuration for [CONFIG](index=.opendistro_security)

Here is an example list of tests that are impacted

Tests with failures:
 - org.opensearch.security.TransportUserInjectorIntegTest.testSecurityUserInjection
 - org.opensearch.security.dlic.rest.api.legacy.LegacyNodesDnApiTests.testNodesDnApiWithPermissions
 - org.opensearch.security.multitenancy.test.TenancyDefaultTenantTests.testDefaultTenantUpdate
 - org.opensearch.security.multitenancy.test.TenancyMultitenancyEnabledTests.testMultitenancyDisabled_endToEndTest

Research done by @willyborankin that seems to indicate there is a cleanup/setup issue

I played a bit with tests and found out that if you run first TenancyDefaultTenantTests and after that TenancyPrivateTenantEnabledTests the second one will fail in JetBrains. If you re-run tests afterwards they all green. It looks like a bug in the SingleClusterTest class I will try to take a look deeper.

#2557 (comment)

Looks like this could be impacting many different pull requests:

• Add score based password verification #2557
• [Backport 2.x] update security-analytics roles to add correlation engine apis #2735
• [Manual Backport 2.x] Manually backports auth tokens, service accounts, and multi tenancy changes #2737

[peternied]

@willyborankin Would you be willing to take this issue on? If so please assign it to yourself, otherwise let me know and I'll figure out what to do next

[willyborankin]

Sure. I can't assigned myself since I'm not a maintainer.

[peternied]

Thanks, I've assigned you the issue

I can't assigned myself since I'm yet not a maintainer.

Fixed that for you

[stephen-crawford]

Hi @willyborankin and @peternied, please let me know if you need a hand getting this sorted. I know @willyborankin is very capable but I am also happy to help if needed :)

[willyborankin]

@peternied and @scrawfor99 found out the root cause of the test flakiness. The reason is that tennacy rest endpoint implementation. It updates the sec index on each node and after the index has been updated the listener attached to the index sends client notifications to other nodes :-). I fixed it but still has some questions around REST end point itself. I will open PR most probably today.

[peternied]

@willyborankin Is this resolved?

• Fix multitency config update #2758

[willyborankin]

@peternied Yes I think I can close it now right?