Nightly ITs: bigquery IT fails due to rate limit exceeded

[diegomarquezp]

This may be a flake - I'm retrying the IT to confirm. This IT has failed a few times.

From https://github.com/GoogleCloudPlatform/spring-cloud-gcp/actions/runs/8229523346/job/22500895100

Error: 1:230 [ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 20.18 s <<< FAILURE! -- in com.google.cloud.spring.bigquery.integration.outbound.BigQueryFileMessageHandlerIntegrationTests
Error: 1:231 [ERROR] com.google.cloud.spring.bigquery.integration.outbound.BigQueryFileMessageHandlerIntegrationTests.testLoadFile_cancel -- Time elapsed: 4.512 s <<< ERROR!
com.google.cloud.bigquery.BigQueryException: Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.deleteTable(HttpBigQueryRpc.java:381)
	at com.google.cloud.bigquery.BigQueryImpl$10.call(BigQueryImpl.java:593)
	at com.google.cloud.bigquery.BigQueryImpl$10.call(BigQueryImpl.java:590)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.bigquery.BigQueryImpl.delete(BigQueryImpl.java:589)
	at com.google.cloud.spring.bigquery.integration.outbound.BigQueryFileMessageHandlerIntegrationTests.setup(BigQueryFileMessageHandlerIntegrationTests.java:74)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden

[JoeWang1127]

Reading the error log, this line caused the error:

spring-cloud-gcp/spring-cloud-gcp-bigquery/src/test/java/com/google/cloud/spring/bigquery/integration/outbound/BigQueryFileMessageHandlerIntegrationTests.java

Line 74 in 4621c5a

this.bigquery.delete(TableId.of(DATASET_NAME, TABLE_NAME));

However, the implementation of delete method is already run with exponential backoff retry settings.

I can increase the total # of retries in

spring-cloud-gcp/spring-cloud-gcp-bigquery/src/test/java/com/google/cloud/spring/bigquery/core/BigQueryTestConfiguration.java

Line 63 in 4621c5a

BigQueryOptions bigQueryOptions =

@meltsufin do you have other ideas about this issue or do you think increase the # of retries is suffice?

[meltsufin]

Can we just increase the quota?

[JoeWang1127]

Can we just increase the quota?

The problem is the test is reaching a short-term limit, I don't think increase quota will solve it.

Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas

[meltsufin]

Is it a configurable limit?

[JoeWang1127]

According to Data manipulation language, the rate limit for delete is 25 statements every 10 seconds.

[meltsufin]

Got it, then I think increasing retries makes sense. Thanks!

[burkedavison]

Note that this is a rate limit, not a quota limit. (See description of the difference here: https://cloud.google.com/bigquery/docs/troubleshoot-quotas#overview)

This issue continues to cause problems with PRs.

[burkedavison]

Alternative solution: rework bigquery IT to not use the same bigquery table for the unit tests.