Failing test(s): TestAccNetappVolume_netapp*

[SarahFrench]

Impacted tests

• TestAccNetappVolume_netappVolumeBasicExample_update
• TestAccNetappbackup_netappBackupExample
• TestAccNetappbackup_netappBackupFull_update

Affected Resource(s)

• google_netapp_backup_vault

Failure rates

• 100% since June/July 2024 across the 3 tests

Message(s)

    vcr_utils.go:152: Step 11/16 error: Error running apply: exit status 1
        Error: Error waiting to create backupVault: Error waiting for Creating backupVault: Error code 3, message: bad request error: "Error creating backup vault - Creation of multiple backup vaults is not supported for a project within a region."
          with google_netapp_backup_vault.backup-vault,
          on terraform_plugin_test.tf line 37, in resource "google_netapp_backup_vault" "backup-vault":
          37: resource "google_netapp_backup_vault" "backup-vault" {

Same for all 3 named tests

Nightly build test history

• https://hashicorp.teamcity.com/test/714502481071555483?currentProjectId=TerraformProviders_GoogleCloud_GOOGLE_NIGHTLYTESTS&branch=refs%2Fheads%2Fnightly-test&expandTestHistoryChartSection=true
• https://hashicorp.teamcity.com/test/8777096052812546395?currentProjectId=TerraformProviders_GoogleCloud_GOOGLE_NIGHTLYTESTS&branch=refs%2Fheads%2Fnightly-test&expandTestHistoryChartSection=true
• https://hashicorp.teamcity.com/test/6527364086859666331?currentProjectId=TerraformProviders_GoogleCloud_GOOGLE_NIGHTLYTESTS&branch=refs%2Fheads%2Fnightly-test&expandTestHistoryChartSection=true

b/357636823

[SarahFrench]

The example used to generate TestAccNetappbackup_netappBackupExample uses us-central1, and this is the default region used in all tests.

We could make different tests use different regions.

[Mehul3217]

Can we please get the list of projects that are being used for above tests, looks like some projects requires cleaning up of resources.

[trodge]

Any cleanup for these resources should be handled by automated sweepers. I agree with @SarahFrench that the most likely solution is going to be putting each test in its own region, unless we know that the sweepers are not working.

[melinath]

@Mehul3217 responded internally on b/357940045

[Mehul3217]

We have raised a PR for manual sweepers: GoogleCloudPlatform/magic-modules#11467

[ScottSuarez]

Doesn't appear to have been resolved with my change.. reopening.. The bug is also tracked internally by the service team.

[Mehul3217]

It should get resolved now, we have cleaned up the required resources.

[SarahFrench]

It looks like these two tests are racing to be the first to provision a backup vault in the region:

And this test is consistently failing

All the failures are still Error creating backup vault - Creation of multiple backup vaults is not supported for a project within a region

[SarahFrench]

Even if some resources needed cleaning up in the past, there will be a problem if tests running at the same time are trying to provision a backup vault in the same region and project. Either tests need to use different regions (have them hardcoded into the test) or we need to ensure the tests provisioning these resources cannot run in parallel.

[Mehul3217]

Ack, working on it

created a PR: GoogleCloudPlatform/magic-modules#11939

[ScottSuarez]

Ack, working on it

created a PR: GoogleCloudPlatform/magic-modules#11939

If you are using new regions not being sweeped could you add them to the sweeper please?
https://github.com/modular-magician/terraform-provider-google/blob/c2cf67f225401493c6d11470be3d6df8747ae816/google/services/netapp/resource_netapp_backup_vault_sweeper.go#L41

[Mehul3217]

Ack, working on it
created a PR: GoogleCloudPlatform/magic-modules#11939

If you are using new regions not being sweeped could you add them to the sweeper please? https://github.com/modular-magician/terraform-provider-google/blob/c2cf67f225401493c6d11470be3d6df8747ae816/google/services/netapp/resource_netapp_backup_vault_sweeper.go#L41

Have not added any new locations

[melinath]

looks like these are now passing relatively consistently - marking this ticket as closed

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.