IT: BigQuery Exceeded rate limits

[zhumin8]

Observed consistent error on multiple retries on Mar 8 with message starting:

BigQuery Exceeded rate limits: too many table update operations for this table.

#2655 (comment)
#2662 (comment) - ci log and log
Also observed in multiple nightly integration ci Mar 8 , Mar 9, and a few other pr presubmits

Then on Mar 11, this error did not show up on rerun.

longer snippet of log:

Error: 9:661 [ERROR] Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 14.79 s <<< FAILURE! -- in com.google.cloud.spring.bigquery.integration.outbound.BigQueryFileMessageHandlerIntegrationTests
Error: 9:662 [ERROR] com.google.cloud.spring.bigquery.integration.outbound.BigQueryFileMessageHandlerIntegrationTests.testLoadFile_cancel -- Time elapsed: 4.071 s <<< ERROR!
com.google.cloud.bigquery.BigQueryException: Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.deleteTable(HttpBigQueryRpc.java:381)
	at com.google.cloud.bigquery.BigQueryImpl$10.call(BigQueryImpl.java:593)
	at com.google.cloud.bigquery.BigQueryImpl$10.call(BigQueryImpl.java:590)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.bigquery.BigQueryImpl.delete(BigQueryImpl.java:589)
	at com.google.cloud.spring.bigquery.integration.outbound.BigQueryFileMessageHandlerIntegrationTests.setup(BigQueryFileMessageHandlerIntegrationTests.java:74)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
DELETE https://bigquery.googleapis.com/bigquery/v2/projects/spring-cloud-gcp-ci/datasets/test_dataset/tables/test_table
***
  "code": 403,
  "errors": [
    ***
      "domain": "usageLimits",
      "message": "Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
      "reason": "rateLimitExceeded"
    ***
  ],
  "message": "Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
  "status": "PERMISSION_DENIED"
***
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.deleteTable(HttpBigQueryRpc.java:378)
	... 10 more

Error: 9:664 [ERROR] com.google.cloud.spring.bigquery.integration.outbound.BigQueryFileMessageHandlerIntegrationTests.testLoadFile_sync -- Time elapsed: 1.144 s <<< ERROR!
com.google.cloud.bigquery.BigQueryException: Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.deleteTable(HttpBigQueryRpc.java:381)
	at com.google.cloud.bigquery.BigQueryImpl$10.call(BigQueryImpl.java:593)
	at com.google.cloud.bigquery.BigQueryImpl$10.call(BigQueryImpl.java:590)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.bigquery.BigQueryImpl.delete(BigQueryImpl.java:589)
	at com.google.cloud.spring.bigquery.integration.outbound.BigQueryFileMessageHandlerIntegrationTests.setup(BigQueryFileMessageHandlerIntegrationTests.java:74)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
DELETE https://bigquery.googleapis.com/bigquery/v2/projects/spring-cloud-gcp-ci/datasets/test_dataset/tables/test_table
***
  "code": 403,
  "errors": [
    ***
      "domain": "usageLimits",
      "message": "Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
      "reason": "rateLimitExceeded"
    ***
  ],
  "message": "Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
  "status": "PERMISSION_DENIED"
***
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.deleteTable(HttpBigQueryRpc.java:378)
	... 10 more

08:15:09:664 [INFO] Running com.google.cloud.spring.bigquery.core.BigQueryTemplateIntegrationTests
08:15:09.658 [main] WARN org.springframework.boot.test.json.DuplicateJsonObjectContextCustomizerFactory$DuplicateJsonObjectContextCustomizer -- 

Found multiple occurrences of org.json.JSONObject on the class path:

	jar:file:/home/runner/.m2/repository/org/json/json/20231013/json-20231013.jar!/org/json/JSONObject.class
	jar:file:/home/runner/.m2/repository/com/vaadin/external/google/android-json/0.0.20131108.vaadin1/android-json-0.0.20131108.vaadin1.jar!/org/json/JSONObject.class

You may wish to exclude one of them to ensure predictable runtime behavior

[zhumin8]

duplicate of #2665, closing this one to merge.