Status of testing Providers that were prepared on June 18, 2021

[potiuk]

I have a kind request for all the contributors to the latest provider packages release.
Could you help us to test the RC versions of the providers and let us know in the comment,
if the issue is addressed there.

Providers that need testing

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 2.0.0rc2

• Add test_connection method to Airbyte hook (#16236): @msumit
• Fix hooks extended from http hook (#16109): @msumit

Provider amazon: 2.0.0rc2

• read timestamp from Cloudwatch events (#15173): @codenamestif
• remove retry for now (#16150): @zachliu
• Remove the not-allow-trailing-slash rule on S3_hook (#15609): @Isaacwhyuenac
• Add support of capacity provider strategy for ECSOperator (#15848): @codenamestif
• Update copy command for s3 to redshift (#16241): @sunki-hong
• Make job name check optional in SageMakerTrainingOperator (#16327): @olivermeyer
• Add AWS DMS replication task operators (#15850): @Visya
• Fix S3 Select payload join (#16189): @TAKEDA-Takashi
• Fix spacing in AwsBatchWaitersHook docstring (#15839): @jlaneve
• MongoToS3Operator failed when running with a single query (not aggregate pipeline) (#15680): @amatellanes
• fix: AwsGlueJobOperator change order of args for load_file (#16216): @avocadomaster
• Fix S3ToFTPOperator (#13796): @JavierLopezT

Provider apache.spark: 2.0.0rc2

• Make SparkSqlHook use Connection (#15794): @uranusjr

Provider cncf.kubernetes: 2.0.0rc2

• Add KPO pod-template-file jinja template support. (#15942): @Dr-Denzy
• Save pod name to xcom for KubernetesPodOperator (#15755): @junnplus
• Bug Fix Pod-Template Affinity Ignored due to empty Affinity K8S Object (#15787): @jpyen
• Bug Pod Template File Values Ignored (#16095): @jpyen
• Fix issue with parsing error logs in the KPO (#15638): @dimberman
• Fix unsuccessful KubernetesPod final_state call when is_delete_operator_pod=True (#15490): @MatthewRBruce

Provider dingding: 2.0.0rc2

• Fix hooks extended from http hook (#16109): @msumit

Provider discord: 2.0.0rc2

• Fix hooks extended from http hook (#16109): @msumit

Provider docker: 2.0.0rc2

• Replace DockerOperator's 'volumes' arg for 'mounts' (#15843): @uranusjr

Provider elasticsearch: 2.0.1rc1

• Remove support jinja templated log_id in elasticsearch (#16465): @jedcunningham
• Support remote logging in elasticsearch with filebeat 7 (#14625): @jedcunningham
• Support non-https elasticsearch external links (#16489): @jedcunningham
• Fix external elasticsearch logs link (#16357): @jedcunningham
• Fix elasticsearch external log link with json_format (#16467): @jedcunningham

Provider google: 4.0.0rc2

• Move plyvel to google provider extra (#15812): @dstandish
• Fixes AzureFileShare connection extras (#16388): @potiuk
• Add extra links for google dataproc (#10343): @yesemsanthoshkumar
• Add link to Oracle Connection Docs (#15632): @sunkickr
• pass wait_for_done parameter down to _DataflowJobsController (#15541): @dejii
• Update Google Ads hook (#15266): @jacobhjkim
• Implement BigQuery Table Schema Update Operator (#15367): @thejens
• Add BigQueryToMsSqlOperator (#15422): @subkanthi
• Fix: GCS To BigQuery source_object (#16160): @tegardp
• FIX: unnecessary downloads in GCSToLocalFilesystemOperator (#16171): @p-kachalov
• Fix bigquery type error when export format is parquet (#16027): @sunki-hong
• Fix argument ordering and type of bucket and object (#15738): @sjvanrossum
• Fix sql_to_gcs docstring lint error (#15730): @natanweinberger
• Ensure mysql_to_gcs fully compatible with MySQL and BigQuery for datetime-related values (#15026): @tianjianjiang
• Fix deprecation warnings location in google provider (#16403): @ashb

Provider hashicorp: 2.0.0rc2

• Sanitize end of line character when loading token from a file (vault) (#16407): @mmenarguezpear

Provider http: 2.0.0rc2

• Update SimpleHttpOperator to take auth type object (#15605): @fredthomsen
• HttpHook. Use request factory and respect defaults (#14701): @ngaranko

Provider microsoft.azure: 3.0.0rc2

• Fixes AzureFileShare connection extras (#16388): @potiuk
• Add link to Oracle Connection Docs (#15632): @sunkickr
• Fix WasbHook.delete_file broken when using prefix (#15637): @monti-python
• Fix colon spacing in AzureDataExplorerHook docstring (#15841): @jlaneve
• fix wasb remote logging when blob already exists (#16280): @flolas

Provider odbc: 2.0.0rc2

• OdbcHook returns None. Related to #15016 issue. (#15510): @Goodkat
• Fix OdbcHook handling of port (#15772): @dstandish

Provider opsgenie: 2.0.0rc2

• Fix hooks extended from http hook (#16109): @msumit

Provider oracle: 2.0.0rc2

• Add optional result handler to database hooks (#15581): @malthe
• [Oracle] Add port to DSN (#15589): @malthe
• Add link to Oracle Connection Docs (#15632): @sunkickr

Provider papermill: 2.0.0rc2

• Emit error on duplicated DAG ID (#15302): @uranusjr

Provider plexus: 2.0.0rc2

• Removes arrow higher limits for plexus provider (#16026): @potiuk

Provider postgres: 2.0.0rc2

• PostgresHook: deepcopy connection to avoid mutating connection obj (#15412): @zhzhang
• Avoid passing aws_conn_id as conn_args for psycopg2.connect (#16100): @gabrielsyapse

Provider samba: 2.0.0rc2

• Add support for extra parameters to samba client (#16115): @malthe

Provider sftp: 2.0.0rc2

• Depreciate private_key_pass extra param and rename to private_key_passphrase (#14028): @pgillet

Provider slack: 4.0.0rc2

• Fix hooks extended from http hook (#16109): @msumit

Provider snowflake: 2.0.0rc2

• Add template_fields to S3ToSnowflake operator (#15926): @nlecoy
• Allow S3ToSnowflakeOperator to omit schema (#15817): @uranusjr
• Added ability for Snowflake to attribute usage to Airflow by adding an application parameter (#16420): @sfc-gh-madkins
• fix: restore parameters support when sql passed to SnowflakeHook as str (#16102): @grassten

Provider ssh: 2.0.0rc2

• Fixed #9963: Display explicit error in case UID has no actual username (#15212): @andrewgodwin

Providers for which previous RC was fully tested (thanks @levyitay !)

Provider qubole: 2.0.0rc2

• Qubole Hook Does Not Support 'include_headers' (#15598): @levyitay
• Feature qubole hook support headers (#15683): @levyitay
• Feature qubole hook support headers (#15615): @levyitay

Providers that do not need testing

Those are providers that were either doc-only or had changes that do not require testing.

• Provider apache.beam: 3.0.0rc2
• Provider apache.cassandra: 2.0.0rc2
• Provider apache.druid: 2.0.0rc2
• Provider apache.hdfs: 2.0.0rc2
• Provider apache.hive: 2.0.0rc2
• Provider apache.kylin: 2.0.0rc2
• Provider apache.livy: 2.0.0rc2
• Provider apache.pig: 2.0.0rc2
• Provider apache.pinot: 2.0.0rc2
• Provider apache.sqoop: 2.0.0rc2
• Provider asana: 1.0.0rc2
• Provider celery: 2.0.0rc2
• Provider cloudant: 2.0.0rc2
• Provider databricks: 2.0.0rc2
• Provider datadog: 2.0.0rc2
• Provider exasol: 2.0.0rc2
• Provider facebook: 2.0.0rc2
• Provider ftp: 2.0.0rc2
• Provider grpc: 2.0.0rc2
• Provider imap: 2.0.0rc2
• Provider jdbc: 2.0.0rc2
• Provider jenkins: 2.0.0rc2
• Provider jira: 2.0.0rc2
• Provider microsoft.mssql: 2.0.0rc2
• Provider microsoft.winrm: 2.0.0rc2
• Provider mongo: 2.0.0rc2
• Provider mysql: 2.0.0rc2
• Provider neo4j: 2.0.0rc2
• Provider openfaas: 2.0.0rc2
• Provider pagerduty: 2.0.0rc2
• Provider presto: 2.0.0rc2
• Provider redis: 2.0.0rc2
• Provider salesforce: 3.0.0rc2
• Provider segment: 2.0.0rc2
• Provider sendgrid: 2.0.0rc2
• Provider singularity: 2.0.0rc2
• Provider sqlite: 2.0.0rc2
• Provider tableau: 2.0.0rc2
• Provider telegram: 2.0.0rc2
• Provider trino: 2.0.0rc2
• Provider vertica: 2.0.0rc2
• Provider yandex: 2.0.0rc2
• Provider zendesk: 2.0.0rc2

[potiuk]

This would be great it we could have confirmation for most of the providers by Friday afternoon, so that I could release them then :)

[potiuk]

@jmelot Also a kind request for you - it's the first release of Asana provider, so if you could test it as well and let us know in the comment, it would be great: https://pypi.org/project/apache-airflow-providers-asana/1.0.0rc1

[zhzhang]

Tested postgres connection changes on a Resdhift cluster, looks good to me.

[Goodkat]

Tested odbc connection, it works as expected with boolean parameters now:
#15016 (comment)

[jmelot]

Tested Asana provider, all seems to be well!

[nlecoy]

I've been using the fix of #15926 for work and everything looks fine! Also, I tested #15817 and its working too.

[pavelhlushchanka]

The package description says that the provider can be installed on top of existing airflow 2.*. I'm installing amazon provider and it requires airflow 2.1.

[potiuk]

The package description says that the provider can be installed on top of existing airflow 2.*. I'm installing amazon provider and it requires airflow 2.1.

Yep. We have an enty in changelog about it with warning. Do you think it's not enough ?

[pavelhlushchanka]

I tested #15173 and #15848. Works fine.

[pavelhlushchanka]

The package description says that the provider can be installed on top of existing airflow 2.*. I'm installing amazon provider and it requires airflow 2.1.

Yep. We have an enty in changelog about it with warning. Do you think it's not enough ?

@potiuk I believe it's a bit confusing. If such info is added, then it should be precise. But it's up to you.

[potiuk]

@potiuk I believe it's a bit confusing. If such info is added, then it should be precise. But it's up to you.

I will ask others what they think. I am inclined to release new rc for that together with elasticsearch (but let's continue testing).

[amatellanes]

I tested #15680 and it works fine 👍

[levyitay]

tested Qubole changes, looks good

[flolas]

Tested #16280 in prod env with k8s, it works fine.

[potiuk]

@codenamestif (and others) - yeah it's too misleading indeed. I cancelled the vote and we have updated/fixed a few more things in the meantime. I will keep the test status for those providers that's been tested and will update the issue with the new changes later today when I release new RCs./

[potiuk]

This will be the new README. Note 2.1+ and apache-airflow<=2.1.0 dependency first on the list.

Package apache-airflow-providers-amazon
------------------------------------------------------

Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).


Release: 2.0.0

Provider package
----------------

This is a provider package for ``amazon`` provider. All classes for this provider package
are in ``airflow.providers.amazon`` python package.

Installation
------------

You can install this package on top of an existing airflow 2.1+ installation via
``pip install apache-airflow-providers-amazon``

PIP requirements
----------------

==================  ====================
PIP package         Version required
==================  ====================
``apache-airflow``  ``>=2.1.0``
``boto3``           ``>=1.15.0,<1.18.0``
``watchtower``      ``~=1.0.6``
==================  ====================

[fredthomsen]

Tested SimpleHttpOperator for #15605. All good.

[potiuk]

As new package RCs are released, I encourage those from the list who have not done so, to double-check if the packages are working fine with their changes. The voting time has been prolonged to Tue 22 Jun 21:29:59 CEST 2021.

Looking forward to your thumb-ups :)

[TAKEDA-Takashi]

Tested #16189. Works fine.

[dejii]

Tested #15541. Works as expected.

[potiuk]

Hey everyone. Just a kind ping here.

Due to update in READMEs, the voting on the packages have been prolonged to Tuesday so we have two more days for testing.

Thanks to @dejii @TAKEDA-Takashi @flolas @levyitay @codenamestif @amatellanes @nlecoy @jmelot @Goodkat @zhzhang for testing their changes (you rock!).

It would be great if we hear from others as well before the PMC gets to vote :). @MatthewRBruce @JavierLopezT @mmenarguezpear @monti-python @junnplus @codenamestif @gabrielsyapse @uranusjr @Dr-Denzy @jpyen @jacobhjkim @olivermeyer @zachliu @thejens @p-kachalov
@sunki-hong @sjvanrossum @subkanthi @avocadomaster @jlaneve @sunkickr @malthe @yesemsanthoshkumar @natanweinberger @jedcunningham @Visya @andrewgodwin
@dimberman @msumit @grassten @pgillet @tianjianjiang @dstandish @Isaacwhyuenac @sfc-gh-madkins @ngaranko @ashb @tegardp

[Isaacwhyuenac]

Tested #15609. It works fine

[MatthewRBruce]

Tested #15490 - Fix unsuccessful KubernetesPod final_state call when is_delete_operator_pod=True. Works as expected.

[olivermeyer]

#16327 works as expected.

[monti-python]

#15637 works as expected as well

[natanweinberger]

#15730 is just a whitespace linting fix 👍

However, #15728 and #15667 touched the same file around the same time (also for minor changes) and aren't listed here, should they be included?

[potiuk]

However, #15728 and #15667 touched the same file around the same time (also for minor changes) and aren't listed here, should they be included?

Auto-apply is specifically excluded as it was a change that applied to all providers (and main reason for this backwards-incompatible release) but it has been tested before. The other one is a docstring only (and those are excluded from changelog/testing).

[jedcunningham]

Tested elasticsearch 2.0.1rc1 (note: rc1, not 2 as linked in the description), looks good. Do note that some of the fixes/features only work with core changes coming in 2.1.1, but the new provider does still work with earlier 2.x versions.

[potiuk]

Tested elasticsearch 2.0.1rc1 (note: rc1, not 2 as linked in the description), looks good. Do note that some of the fixes/features only work with core changes coming in 2.1.1, but the new provider does still work with earlier 2.x versions.

Thanks @jedcunningham ! corrected the link to keep good track of it.

[tianjianjiang]

@potiuk I've tried to allocate some resource for testing #15026 but unfortunately not happening soon enough. However, at least the situation I've experienced before is coded in https://github.com/apache/airflow/pull/15026/files#diff-ba677556962f02c90a0e6933525d9d3f795e2d44a3d2229de1db9d1b68a067eb and since the datetime-related change is abstract while its upstream has only one change set of docstring lint (#15730), it is highly likely #15026 will work.

[potiuk]

Kind ping again. Last chance to do some testing before we release.

@JavierLopezT @mmenarguezpear @junnplus @codenamestif @gabrielsyapse @uranusjr @Dr-Denzy @jpyen @jacobhjkim @zachliu @thejens @p-kachalov @sunki-hong @sjvanrossum @subkanthi @avocadomaster @jlaneve @sunkickr @malthe @yesemsanthoshkumar @Visya @andrewgodwin @dimberman @msumit @grassten @pgillet @dstandish @sfc-gh-madkins @ngaranko @ashb @tegardp

[JavierLopezT]

Hello. Regarding #13796, I did make a quick test before merging the PR (the file was there and was not empty) but I can't make a deeper test due to health reasons. I am sorry

[ngaranko]

#14701 works as expected.

[potiuk]

Hello. Regarding #13796, I did make a quick test before merging the PR (the file was there and was not empty) but I can't make a deeper test due to health reasons. I am sorry

No worries! thanks for your contributions so far @JavierLopezT !

[potiuk]

Callng for last vote so last chance to test before we release.
@mmenarguezpear @junnplus @codenamestif @gabrielsyapse @uranusjr @Dr-Denzy @jpyen @jacobhjkim @zachliu @thejens @p-kachalov @sunki-hong @sjvanrossum @subkanthi @avocadomaster @jlaneve @sunkickr @malthe @yesemsanthoshkumar @Visya @andrewgodwin @dimberman @msumit @grassten @pgillet @dstandish @sfc-gh-madkins @ashb @tegardp