-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PostgresToGCSOperator doesn't convert DATE correctly #8387
Comments
The proposed change looks good. I think it's worth testing this problem with system tests. A system test is a type of test that uses an example DAG and communicates with a real service. Example DAGs for operators for GCP are available: Would you like to work on PR? Apache Airflow is an open-source project without paid technical support. Each problem is solved by community members - other Airflow users. If this issue is important to you, it is best to take action yourself. I hope this information will be helpful for you and I look forward to your PR. |
I will take a look! |
Hello, |
@vijaya-lakshmi-venkatraman yes - assigned to you |
@vijaya-lakshmi-venkatraman - Thanks for taking this one over! I had started looking at it, but was unclear on proceeding given that it seems the same code in |
Recently #20807 was merged. I would say that probably the best thing to do is to implement similar generic operator for Gcs |
#22536 might have solved this. |
Normally it does indeed solve this issue. In my tests I could export and then load into bigquery each of these different field types. |
What happened:
Time datatypes are treated as datetime in the operator this create problems when dumping to csv (so can be loaded to BigQuery) because the data will be date type yet bigquery will expect datetime format.
How to reproduce it:
dump DATE column to csv using the operator and create a table from it in bigquery. The column will not be of DATE format.
Anything else we need to know:
probably need fix at https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/postgres_to_gcs.py#L46 changing it from
TIMESTAMP
toDATE
as:https://github.com/psycopg/psycopg2/blob/master/psycopg/pgtypes.h#L41
and make some adjustments in
convert_type
function.The text was updated successfully, but these errors were encountered: