storage: fix google_storage_bucket 429 Error

[BBBmau]

Fixes hashicorp/terraform-provider-google#18132

We add in a 429 Retry Predicate for when users run into quota limit issues when attempting to provision a high amount of storage buckets in a short time.

Still in draft due to needing to fix a 409 error:

│ Error: googleapi: Error 409: Your previous request to create the named bucket succeeded and you already own it., conflict
│ 
│   with module.bucket[64].google_storage_bucket.standard,
│   on module/main.tf line 3, in resource "google_storage_bucket" "standard":
│    3: resource "google_storage_bucket" "standard" {

Release Note Template for Downstream PRs (will be copied)

storage: fix 429 error in `google_storage_bucket` resource

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 1 file changed, 2 insertions(+), 1 deletion(-))
google-beta provider: Diff ( 1 file changed, 2 insertions(+), 1 deletion(-))

[modular-magician]

Tests analytics

Total tests: 107
Passed tests: 98
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

• storage

$\textcolor{green}{\textsf{All tests passed!}}$

View the build log

[BBBmau]

It appears that the 409 error comes from a GET not being performed which would check whether or not the bucket creation been has successful or not.

The request below has been performed on buckets that show up in state, this is followed after a POST of the bucket creation

 ---[ REQUEST ]---------------------------------------
 GET /storage/v1/b/mau-my-bucket-20240515-4?alt=json&prettyPrint
 Host: storage.googleapis.com
 User-Agent: google-api-go-client/0.5 Terraform/1.8.0-alpha20240
 X-Goog-Api-Client: gl-go/1.20.11 gdcl/0.152.0
 Accept-Encoding: gzip

It was found in the logs that buckets that are missing the GET request would respond with the panic below:

│ Error: timeout while waiting for state to become 'success' (timeout: 1m0s)
│ 
│   with module.bucket[123].google_storage_bucket.standard,
│   on module/main.tf line 3, in resource "google_storage_bucket" "standard":
│    3: resource "google_storage_bucket" "standard" {

[BBBmau]

After some testing this morning it looks to be working flawlessly when attempting to provision 1000 buckets:

Marking as ready for review in order to get some feedback on this solution since it's very unclear what changed when testing from yesterday to testing from today.

The tfconfig used for working on this can be found below:
main.tf

provider "google" {
  project = "INSERT-PROJECT-ID"
  region  = "us-central1"
  credentials = file("${path.module}/key.json")
}

module "bucket" {
    count = 1000
    source = "./module"
    index = count.index
}

module/main.tf

variable "index" {}

resource "google_storage_bucket" "standard" {
  name = "mau-2-my-bucket-20240828-${var.index}"
  location = "us-central1"
  force_destroy = true
  public_access_prevention = "enforced"
  uniform_bucket_level_access = true
}

Let me know your thoughts @ScottSuarez !

[github-actions]

@ScottSuarez This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

[ScottSuarez]

This seems reasonable to me!