Status of testing Providers that were prepared on February 09, 2022

[potiuk]

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 3.0.0rc2

• Rename params to cloudformation_parameter in CloudFormation operators. (#20989): @potiuk
• [SQSSensor] Add opt-in to disable auto-delete messages (#21159): @LaPetiteSouris
• Create a generic operator SqlToS3Operator and deprecate the MySqlToS3Operator. (#20807): @mariotaddeucci
• Move some base_aws logging from info to debug level (#20858): @o-nikolas
• Adds support for optional kwargs in the EKS Operators (#20819): @ferruzzi
• AwsAthenaOperator: do not generate client_request_token if not provided (#20854): @XD-DENG
• Add more SQL template fields renderers (#21237): @josh-fell
• Add conditional template_fields_renderers check for new SQL lexers (#21403): @josh-fell
• fix: cloudwatch logs fetch logic (#20814): @ayushchauhan0811
• Fix all Amazon Provider MyPy errors (#20935): @potiuk
• Bug fix in AWS glue operator related to num_of_dpus #19787 (#21353): @Ritika-Singhal
• Fix safe type cast error in aws mysql_to_s3 (#21277): @SasanAhmadi
• Alleviate import warning for EmrClusterLink in deprecated AWS module (#21195): @josh-fell
• Rename amazon EMR hook name (#20767): @vinitpayal
• Standardize AWS SQS classes names (#20732): @eladkal
• Standardize AWS Batch naming (#20369): @ferruzzi
• Standardize AWS Redshift naming (#20374): @ferruzzi
• Standardize DynamoDB naming (#20360): @ferruzzi
• Standardize AWS ECS naming (#20332): @ferruzzi
• Refactor operator links to not create ad hoc TaskInstances (#21285): @josh-fell
• eks_hook log level fatal -> FATAL (#21427): @frankcash

Provider apache.druid: 2.3.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell

Provider apache.hive: 2.2.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell
• Add conditional template_fields_renderers check for new SQL lexers (#21403): @josh-fell

Provider apache.spark: 2.1.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell
• Add optional features in providers. (#21074): @potiuk

Provider apache.sqoop: 2.1.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell

Provider cncf.kubernetes: 3.0.2rc2

• Add missed deprecations for cncf (#20031): @dimon222

Provider docker: 2.4.1rc2

• Fixes Docker xcom functionality (#21175): @ferruzzi
• Fix docker behaviour with byte lines returned (#21429): @potiuk

Provider exasol: 2.1.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell

Provider google: 6.4.0rc2

• [Part 1]: Add hook for integrating with Google Calendar (#20542): @rsg17
• Add encoding parameter to GCSToLocalFilesystemOperator to fix #20901 (#20919): @danneaves-ee
• batch as templated field in DataprocCreateBatchOperator (#20905): @wsmolak
• Make timeout Optional for wait_for_operation (#20981): @MaksYermak
• Add more SQL template fields renderers (#21237): @josh-fell
• Create CustomJob and Datasets operators for Vertex AI service (#21253): @MaksYermak
• Support to upload file to Google Shared Drive (#21319): @lihan
• Cloudsql import links fix. (#21199): @subkanthi
• Fix BigQueryDataTransferServiceHook.get_transfer_run() request parameter (#21293): @zaimy
• 🐛 (BigQueryHook) fix compatibility with sqlalchemy engine (#19508): @david30907d
• Refactor operator links to not create ad hoc TaskInstances (#21285): @josh-fell

Provider http: 2.0.3rc2

• Split out confusing path combination logic to separate method (#21247): @malthe

Provider imap: 2.2.0rc2

• Add "use_ssl" option to IMAP connection (#20441): @feluelle

Provider jdbc: 2.1.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell
• Add optional features in providers. (#21074): @potiuk

Provider microsoft.azure: 3.6.0rc2

• Add optional features in providers. (#21074): @potiuk
• Refactor operator links to not create ad hoc TaskInstances (#21285): @josh-fell

Provider microsoft.mssql: 2.1.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell
• Add conditional template_fields_renderers check for new SQL lexers (#21403): @josh-fell

Provider microsoft.psrp: 1.1.0rc2

• PSRP improvements (#19806): @malthe

Provider mysql: 2.2.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell
• Add conditional template_fields_renderers check for new SQL lexers (#21403): @josh-fell
• Refactor vertica_to_mysql to make it more 'mypy' friendly (#20618): @potiuk

Provider oracle: 2.2.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell
• [Oracle] Oracle Hook - automatically set current_schema when defined in Connection (#19084): @mehmax
• Fix handling of Oracle bindvars in stored procedure call when parameters are not provided (#20720): @malthe

Provider postgres: 3.0.0rc2

• Replaces the usage of postgres:// with postgresql:// (#21205): @potiuk
• Add more SQL template fields renderers (#21237): @josh-fell
• Add conditional template_fields_renderers check for new SQL lexers (#21403): @josh-fell

Provider qubole: 2.1.0rc2

• Add optional features in providers. (#21074): @potiuk
• Add Qubole how to documentation (#20058): @kazanzhy

Provider slack: 4.2.0rc2

• Return slack api call response in slack_hook (#21107): @pingzh
• Fix template_fields type to have MyPy friendly Sequence type (#20571): @potiuk

Provider snowflake: 2.5.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell
• Fix #21096: Support boolean in extra__snowflake__insecure_mode (#21155): @mik-laj

Provider sqlite: 2.1.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell

Provider ssh: 2.4.0rc2

• Add a retry with wait interval for SSH operator (#14489): @Gaurang033
• Add banner_timeout feature to SSH Hook/Operator (#21262): @potiuk
• Add a retry with wait interval for SSH operator #14489 (#19981): @nshetty15
• Delay the creation of ssh proxy until get_conn() (#20474): @derkuci

Provider tableau: 2.1.4rc2

• Squelch more deprecation warnings (#21003): @uranusjr

Provider vertica: 2.1.0rc2

• Add more SQL template fields renderers (#21237): @josh-fell

Provider zendesk: 3.0.0rc2

• Switch from zdesk to zenpy in ZendeskHook (#21349): @pierrejeambrun

[david30907d]

sure, will test it on weekend

[frankcash]

I was able to verify #21427 has been resolved in this RC

[mariotaddeucci]

#20807 all good o/

[dstandish]

@potiuk any reason elasticsearch not included?
there was this change: #21261
too late i guess?

[potiuk]

@potiuk any reason elasticsearch not included? there was this change: #21261 too late i guess?

AAAAH good eye @dstandish. Thanks!

This one was actually a mistake of mine. I classified as "doc-only" change in RC1 (the only change was that it had :rtype removed from the docs). That triggered a mistake when I released an RC2 - because I re-classified it the same (but there were your changes merged since).

I will continue with the release and wil re-release it after this voting (I will also take a look if I had not made similar RC1-> RC2 mistake).

[dstandish]

really it's @jedcunningham's eyes that are due any credit

but thank you very much @potiuk

[pierrejeambrun]

Hi,

I tested all the methods for the new ZendeskHook, everything is working fine.

#21349 ✔️

[potiuk]

Cool. Loooking forward for more comments. We have time till ~ end of weekend (I will release on Sunday/Monday morning if I get enough votes)

[kazanzhy]

Changes in Qubole provider was in documentation (#20058) and typing (#21074).
There are no code changes, therefore I'm pretty sure everything is OK.

[derkuci]

#20474 has been running in our system without issues.

[eladkal]

@potiuk why the new GitHub provider is missing?
#21076

[SasanAhmadi]

@potiuk, I checked #21277 has been resolved in this RC.

[MaksYermak]

@potiuk I´ve checked #20981 and #21253 it works fine

[raphaelauv]

docker: 2.4.1rc2 is working correctly 👍

[nshetty15]

@potiuk #19981 is working fine without issues.

[potiuk]

@potiuk why the new GitHub provider is missing? #21076

Becasue there is really nothing to test there ;) . If there is an error there, it's not a regression :). But seriously - we also release GitHub one, it's just not added to the list here...

But srsly: @pateash if you could test this as well it would be awesome: https://pypi.org/project/apache-airflow-providers-github/1.0.0rc2/

Thanks @eladkal for vigilance!

[pateash]

@potiuk why the new GitHub provider is missing? #21076

Becasue there is really nothing to test there ;) . If there is an error there, it's not a regression :). But seriously - we also release GitHub one, it's just not added to the list here...

But srsly: @pateash if you could test this as well it would be awesome: https://pypi.org/project/apache-airflow-providers-github/1.0.0rc2/

Sure @potiuk, let me look into it.

[mehmax]

Hi @potiuk
I tested #19084 in our environment. In some cases the backwards compatibility is not sufficient.
Improvement see PR #21524

[josh-fell]

#21195, #21285, and #21403 Look good

#21237 Looks good except for an inconsequential typo in the template_fields_renderers dict for HiveOperator in the Apache Hive provider.

TIL that the logic which chooses the renderer will, cleverly, default to the template field name if it cannot find said template field in the template_fields_renderers dict. For HiveOperator the key in template_fields_renderers is "sql" when it should be "hql". Luckily "hql" is itself a renderer so there is no impact on the visual rendering in the UI. @potiuk I created #21525 to fix the typo.

[potiuk]

I tested #19084 in our environment. In some cases the backwards compatibility is not sufficient.
Improvement see PR #21524

Cool. Will release it with elasticsearch as RC3 then (the longest Provider release ever :) )

[potiuk]

Tested:

• Add GCSToTrinoOperator #21704 across the board
• Rename params to cloudformation_parameter in CloudFormation operators. #20989 - tested that it is nicely "non-backwards compatible" for params passed (i.e. complains about cloudformation_parameters missing).
  Replaces the usage of postgres:// with postgresql:// #21205 -> postgresql

• Fixes Docker xcom functionality #21175 and Fix docker behaviour with byte lines returned #21429 with slightly modified example to @raphaelauv :) (Polish people would understand)
  

Both #14489 and #21262 with SSH (both banner timeout and retry present in the log):

[2022-02-11, 19:23:10 UTC] {transport.py:1873} ERROR - Exception (client): Error reading SSH protocol banner
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR - Traceback (most recent call last):
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR -   File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 2270, in _check_banner
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR -     buf = self.packetizer.readline(timeout)
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR -   File "/usr/local/lib/python3.7/site-packages/paramiko/packet.py", line 380, in readline
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR -     buf += self._read_timeout(timeout)
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR -   File "/usr/local/lib/python3.7/site-packages/paramiko/packet.py", line 622, in _read_timeout
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR -     raise socket.timeout()
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR - socket.timeout
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR - 
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR - During handling of the above exception, another exception occurred:
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR - 
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR - Traceback (most recent call last):
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR -   File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 2093, in run
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR -     self._check_banner()
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR -   File "/usr/local/lib/python3.7/site-packages/paramiko/transport.py", line 2275, in _check_banner
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR -     "Error reading SSH protocol banner" + str(e)
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR - paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
[2022-02-11, 19:23:10 UTC] {transport.py:1871} ERROR - 
[2022-02-11, 19:23:10 UTC] {ssh.py:314} INFO - Failed to connect. Sleeping before retry attempt 1
[2022-02-11, 19:23:14 UTC] {ssh.py:314} INFO - Failed to connect. Sleeping before retry attempt 2

[subkanthi]

#21199 is only a documentation change and I see the change in the tag.

[potiuk]

Still some chances for testing as we are waiting for votes. So feel free :)

[pateash]

Hi @potiuk @eladkal ,

apache-airflow-provider-github v1.0.0rc2 working as expected, I have validated against airflow v2.2.3

[potiuk]

Cool!

[david30907d]

#19008 has a bug, my bad 😅
already sent a PR and hope I can pass the CI #21542

[potiuk]

#19008 has a bug, my bad sweat_smile already sent a PR and hope I can pass the CI #21542

Ah . Cool that you checked. I will exclude trino from the release too and release those (and rc3 will include trino as well).

[eladkal]

Ah . Cool that you checked. I will exclude trino from the release too and release those (and rc3 will include trino as well).

I think @david30907d meant #19508 not #19008.
its about google provider not trino

[potiuk]

Ah . Cool that you checked. I will exclude trino from the release too and release those (and rc3 will include trino as well).

I think @david30907d meant #19508 not #19008. its about google provider not trino

Yeah. I realized it when approved it :)

[potiuk]

Glad that you checked @eladkal :)

[david30907d]

Oh sorry i have fat-fingers 😅

[potiuk]

Thanks for testing. All providers except oracle, elasticsearch, google from that batch have been released. Proceeding with rc3 for the three.