Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gzip parameter of sftp_to_gcs operator is never passed to the GCS hook #20552

Closed
2 tasks done
qgallet opened this issue Dec 29, 2021 · 2 comments · Fixed by #20553
Closed
2 tasks done

gzip parameter of sftp_to_gcs operator is never passed to the GCS hook #20552

qgallet opened this issue Dec 29, 2021 · 2 comments · Fixed by #20553
Assignees
Labels
area:providers kind:bug This is a clearly a bug

Comments

@qgallet
Copy link
Contributor

qgallet commented Dec 29, 2021

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

List of versions is the one provided by Cloud Composer image composer-1.17.7-airflow-2.1.4.
(cf. https://cloud.google.com/composer/docs/concepts/versioning/composer-versions#images )

Relevant here is apache-airflow-providers-google==5.1.0

Apache Airflow version

2.1.4

Deployment

Composer

Deployment details

No response

What happened

Found on version 2.1.4 and reproduced locally on main branch (v2.2.3).

When using SFTPToGCSOperator with gzip=True, no compression is actually performed, the files are copied/moved as-is to GCS.

This happens because the gzip parameter isn't passed to the GCS Hook upload() call which then defaults to False.

What you expected to happen

I expect the files to be compressed when gzip=True.

How to reproduce

Create any SFTPToGCSOperator with gzip=True and upload a file.

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@qgallet qgallet added area:providers kind:bug This is a clearly a bug labels Dec 29, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Dec 29, 2021

Thanks for opening your first issue here! Be sure to follow the issue template!

@qgallet
Copy link
Contributor Author

qgallet commented Dec 29, 2021

Thanks for the quick reply and merge @potiuk, closing the issue

@qgallet qgallet closed this as completed Dec 29, 2021
@potiuk potiuk linked a pull request Dec 29, 2021 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers kind:bug This is a clearly a bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant