Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix incorrect query in BigQueryAsyncHook.create_job_for_partition_get #44225

Conversation

sean-rose
Copy link
Contributor

  • The table ID column in INFORMATION_SCHEMA.PARTITIONS is named table_name, not table_id (currently when these queries run they fail with the error "Unrecognized name: table_id").
  • The table ID string value needs to be quoted in the SQL.

related: #37655


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

  * The table ID column in `INFORMATION_SCHEMA.PARTITIONS` is named `table_name`, not `table_id`.
  * The table ID string value needs to be quoted in the SQL.
@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Nov 20, 2024
sean-rose added a commit to mozilla/bigquery-etl that referenced this pull request Nov 20, 2024
… reschedule mode.

There's a bug in those sensors when they run in deferrable mode where the secondary queries they run to detect partitions fail, which is being fixed in apache/airflow#44225.
github-merge-queue bot pushed a commit to mozilla/bigquery-etl that referenced this pull request Nov 20, 2024
… reschedule mode (bug 1932180) (#6542)

* Temporarily change `BigQueryTablePartitionExistenceSensor`s to run in reschedule mode.

There's a bug in those sensors when they run in deferrable mode where the secondary queries they run to detect partitions fail, which is being fixed in apache/airflow#44225.

* Fix `test_dag_with_bigquery_table_sensors`.
@sean-rose
Copy link
Contributor Author

@eladkal since you reviewed #37655 would you mind reviewing this as well? (this is a fix for issues with #37655)

@potiuk potiuk merged commit da94686 into apache:main Nov 22, 2024
64 checks passed
@sean-rose sean-rose deleted the fix-BigQueryAsyncHook.create_job_for_partition_get branch November 22, 2024 23:43
LefterisXefteris pushed a commit to LefterisXefteris/airflow that referenced this pull request Jan 5, 2025
…t`. (apache#44225)

* The table ID column in `INFORMATION_SCHEMA.PARTITIONS` is named `table_name`, not `table_id`.
  * The table ID string value needs to be quoted in the SQL.
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
…t`. (apache#44225)

* The table ID column in `INFORMATION_SCHEMA.PARTITIONS` is named `table_name`, not `table_id`.
  * The table ID string value needs to be quoted in the SQL.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants