Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix parsing of optional mode field in BigQuery Result Schema #26786

Merged
merged 1 commit into from
Oct 1, 2022

Conversation

patricker
Copy link
Contributor

@patricker patricker commented Sep 29, 2022

closes: #26785

The code currently expects mode to be present in the returned Schema object from BigQuery, but according to the official docs, only name and type are required to be present. mode is listed as optional.

https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema

I updated the code to make mode optional, with a default of NULLABLE which matches the docs:

The default value is NULLABLE.

I improved the unit tests to account for more possible scenarios.

@potiuk potiuk merged commit cee610a into apache:main Oct 1, 2022
@patricker patricker deleted the BigQueryHookOptionalSchemaFields branch October 1, 2022 14:50
@kevingoss2
Copy link

kevingoss2 commented Feb 9, 2023

I am still having this issue with the latest Airflow (2.5.1) and Google providers (8.8.0).
File "/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/providers/common/sql/hooks/sql.py", line 349, in run
self._run_command(cur, sql_statement, parameters)
File "/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/providers/common/sql/hooks/sql.py", line 380, in _run_command
cur.execute(sql_statement)
File "/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2697, in execute
description = _format_schema_for_description(query_results["schema"])
File "/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 3006, in _format_schema_for_description
field["mode"] == "NULLABLE",
KeyError: 'mode'

@patricker
Copy link
Contributor Author

patricker commented Feb 9, 2023

@kevingoss2 All I can think is that you somehow aren't actually using Google Providers 8.8.0? Looking at the code for the 8.8.0 Tag, the line numbers/code in the error don't line up with that version at all: https://github.com/apache/airflow/blob/providers-google/8.8.0/airflow/providers/google/cloud/hooks/bigquery.py#L2992

@kevingoss2
Copy link

kevingoss2 commented Feb 9, 2023

@patricker You are correct. That version of the code does not have the error. The issue is that when I do a pip freeze on the docker container it shows 8.8.0. Will try to see what is going on. Thanks for the response!

Note: I figured out why my dependency was getting the wrong version and all is working now. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BigQueryHook Requires Optional Field When Parsing Results Schema
3 participants