Handle exceptions gracefully when delete non-existent resources during integ test resource clean up

[weijia-aws]

Description

When integ test runs, the test will create model, index, pipeline, etc., then perform some assertions, finally clean up the resources.

public void processor_integrationTest() {
    try {
        // load model
        // create pipeline/index
        // do tests/asserts
    } finally {
        // cleanup pipeline, index, model
    }
}

If somehow resource creation fail and receive an exception, the test will enter the finally block and try to delete the resources. Since these resources don't exist, exception will be thrown when try to delete them, see issue 1091. I was able to reproduce the exception by following #1093 (comment). So in this case, we should gracefully handle such NOT FOUND exceptions

Related Issues

Resolves #1098

Related issue: #1091 and #1093

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[heemin32]

Can we expand the scope of this PR to shift the responsibility for resource cleanup from individual tests to the framework? This way, each test wouldn’t need to handle resource cleanup individually.

[vibrantvarun]

Can we expand the scope of this PR to shift the responsibility for resource cleanup from individual tests to the framework? This way, each test wouldn’t need to handle resource cleanup individually.

@heemin32 I would suggest not to do that. As Integ tests are meant to be run in isolation mode. With resource cleanup at framework level, it will create issues where test will start sharing resources with each other and can cause critical issues to be skipped at integ test level because the resource might be created by some other method and used by some other method.

[vibrantvarun]

LGTM

[minalsha]

#1154 (comment)

+1 to Varun's comment here.

[heemin32]

@heemin32 I would suggest not to do that. As Integ tests are meant to be run in isolation mode. With resource cleanup at framework level, it will create issues where test will start sharing resources with each other and can cause critical issues to be skipped at integ test level because the resource might be created by some other method and used by some other method.

Have you seen any such issue in k-nn repo where the resource clean up is handling in framework level? The resource sharing issue can happen when clean up is happening in individual test level and developer missed to clean up the resource in a test.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.65%. Comparing base (e8ed3a4) to head (e296bc6).
Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1154      +/-   ##
============================================
- Coverage     81.72%   81.65%   -0.08%     
+ Complexity     2494     1245    -1249     
============================================
  Files           186       93      -93     
  Lines          8426     4213    -4213     
  Branches       1428      714     -714     
============================================
- Hits           6886     3440    -3446     
+ Misses         1000      501     -499     
+ Partials        540      272     -268     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[heemin32]

LGTM. Thanks!

[heemin32]

Test is failing. Please take a look.

[weijia-aws]

Test is failing. Please take a look.

qa tests are failing because: testAgainstNewCluster depends on resources that created in testAgainstOldCluster, however when testAgainstOldCluster finishes, all resources are cleaned up, resulting in testAgainstNewCluster tests fail with java.lang.NullPointerException or Resource not found exception. In order to fix this, we will need to re-create resources in testAgainstNewCluster tasks

[weijia-aws]

In order to fix this, we will need to re-create resources in testAgainstNewCluster tasks

We can also modify the cleanUp method to not delete resources if running against old cluster?

[weijia-aws]

We can also modify the cleanUp method to not delete resources if running against old cluster?

I think we should do this, as this is how the current behavior is, and requires less code changes comparing to re-create resources

[heemin32]

We can also modify the cleanUp method to not delete resources if running against old cluster?

I think we should do this, as this is how the current behavior is, and requires less code changes comparing to re-create resources

Recreation of resource break the purpose of bwc test where we are validating resource created in old cluster still works in new cluster.

How about exposing a method which tell if sub class want to clean up resource or not? Then, for bwc test, we can skip deletion of resources and delete the resources manually as we do today.

Unresolved comments

[heemin32]

LGTM. Thanks!

[heemin32]

The failure in bwc is a known issue.
#1142

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1154-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 7dc84b50d3b1727cafb4b07342df9ddff2561ffd
# Push it to GitHub
git push --set-upstream origin backport/backport-1154-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1154-to-2.x.