We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
google
List of versions is the one provided by Cloud Composer image composer-1.17.7-airflow-2.1.4. (cf. https://cloud.google.com/composer/docs/concepts/versioning/composer-versions#images )
composer-1.17.7-airflow-2.1.4
Relevant here is apache-airflow-providers-google==5.1.0
apache-airflow-providers-google==5.1.0
2.1.4
Composer
No response
Found on version 2.1.4 and reproduced locally on main branch (v2.2.3).
When using SFTPToGCSOperator with gzip=True, no compression is actually performed, the files are copied/moved as-is to GCS.
SFTPToGCSOperator
gzip=True
This happens because the gzip parameter isn't passed to the GCS Hook upload() call which then defaults to False.
gzip
upload()
False
I expect the files to be compressed when gzip=True.
Create any SFTPToGCSOperator with gzip=True and upload a file.
The text was updated successfully, but these errors were encountered:
Thanks for opening your first issue here! Be sure to follow the issue template!
Sorry, something went wrong.
Thanks for the quick reply and merge @potiuk, closing the issue
qgallet
Successfully merging a pull request may close this issue.
Apache Airflow Provider(s)
google
Versions of Apache Airflow Providers
List of versions is the one provided by Cloud Composer image
composer-1.17.7-airflow-2.1.4
.(cf. https://cloud.google.com/composer/docs/concepts/versioning/composer-versions#images )
Relevant here is
apache-airflow-providers-google==5.1.0
Apache Airflow version
2.1.4
Deployment
Composer
Deployment details
No response
What happened
Found on version 2.1.4 and reproduced locally on main branch (v2.2.3).
When using
SFTPToGCSOperator
withgzip=True
, no compression is actually performed, the files are copied/moved as-is to GCS.This happens because the
gzip
parameter isn't passed to the GCS Hookupload()
call which then defaults toFalse
.What you expected to happen
I expect the files to be compressed when
gzip=True
.How to reproduce
Create any
SFTPToGCSOperator
withgzip=True
and upload a file.Anything else
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: