Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fix pipeline failure due to bigquery Provided Schema does not match Table. #6344

Merged
merged 2 commits into from
Dec 26, 2023

Conversation

zhumin8
Copy link
Contributor

@zhumin8 zhumin8 commented Dec 26, 2023

fix kokoro job failing with error message:

+ bq load --autodetect --project_id=cloud-java-metrics --source_format=CSV client_library_versions.cloud_java_client_library_release_dates cloud_java_client_library_release_dates.csv
Upload complete.Waiting on bqjob_r28424c53fa9642fd_0000018b47445529_1 ... (0s) Current status: RUNNING                                                                                      Waiting on bqjob_r28424c53fa9642fd_0000018b47445529_1 ... (0s) Current status: DONE  

BigQuery error in load operation: Error processing job
'cloud-java-metrics:bqjob_r28424c53fa9642fd_0000018b47445529_1': Provided Schema
does not match Table cloud-java-metrics:client_library_versions.cloud_java_clien
t_library_release_dates. Cannot add fields (field: string_field_0)
Failure details:
- It looks like you are appending to an existing table with
autodetect enabled. Disabling autodetect may resolve this.

Relevant gcloud documentation: bq load

  • remove --autodetect and use default false so destination table schema is used.
  • --skip_leading_rows=1 since no need to infer schema as above.
  • add --null_marker="-" to deal with empty release dates from source, e.g. for google-cloud-accessapproval:1.0.10

image

Manual script run and successfully run script to upload data.

@zhumin8 zhumin8 requested a review from a team as a code owner December 26, 2023 20:47
@product-auto-label product-auto-label bot added the size: xs Pull request size is extra small. label Dec 26, 2023
@zhumin8 zhumin8 added the kokoro:run Add this label to force Kokoro to re-run the tests. label Dec 26, 2023
@zhumin8 zhumin8 requested a review from burkedavison December 26, 2023 20:54
@yoshi-kokoro yoshi-kokoro removed the kokoro:run Add this label to force Kokoro to re-run the tests. label Dec 26, 2023
@zhumin8 zhumin8 merged commit 17c18f1 into main Dec 26, 2023
28 checks passed
@zhumin8 zhumin8 deleted the fix-create-versions branch December 26, 2023 21:35
@release-please release-please bot mentioned this pull request Dec 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size: xs Pull request size is extra small.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants