Fix flaky model tests

[ryanbogan]

Description

Since ModelDaoTests extends KNNSingleNodeTestCase, a node is started at the beginning of each test and stopped at the end of each test. The TrainingJobClusterStateListenerClass contains a clusterChanged method that is called each time a new cluster is started. Within this method, a separate thread runs through all existing models and marks any that are stuck in training as failed. This can result in a scenario that causes flaky tests.

In this scenario, a node is started at the beginning of a test, calling the clusterChanged method. The test creates a model (or multiple), verifies the expected behavior, and then stops the node. Meanwhile, the other thread is in the process of looping through any existing models and potentially updating their model state parameter. If the node stops before this process is complete, an exception will be thrown saying that the model index (.opensearch-knn-models) does not exist.

To fix the flaky test situation, the TrainingJobClusterStateListener class has been mocked to no-op the clusterChanged method in the setup for the class. This means that there will be no separate thread accessing the models for the entire ModelDaoTests class. In the teardown after the class, the mocked static is closed so it will not affect any other test classes.

Issues Resolved

#1376
#1413

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (2b963b4) 84.92% compared to head (c23a1e7) 84.84%.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1429      +/-   ##
============================================
- Coverage     84.92%   84.84%   -0.08%     
+ Complexity     1274     1273       -1     
============================================
  Files           167      167              
  Lines          5186     5186              
  Branches        491      491              
============================================
- Hits           4404     4400       -4     
- Misses          574      577       +3     
- Partials        208      209       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.