Status of testing Providers that were prepared on November 30, 2021

[potiuk]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you help us to test the RC versions of the providers and let us know in the comment,
if the issue is addressed there.

Providers that need testing

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 2.5.0rc1

• Adding support for using client_type API for interacting with EC2 and support for using filters (#9011): @VijayantSoni
• Do not check for S3 key before attempting download (#19504): @dstandish
• MySQLToS3Operator actually allow writing parquet files to s3. (#19094): @guotongfei
• Amazon provider remove deprecation, second try (#19815): @uranusjr
• Catch AccessDeniedException in AWS Secrets Manager Backend (#19324): @john-jac

Provider apache.druid: 2.1.0rc1

• Fix DruidOperator rendering problem (#19420): @humit0
• Adding missing max_ingestion_time to DruidOperator docstring (#18693): @josh-fell
• Add guide for Apache Druid operators (#18527): @Jasonnor

Provider apache.hdfs: 2.2.0rc1

• hdfs provider: fix HA support for webhdfs (#19711): @Aakcht

Provider apache.hive: 2.1.0rc1

• hive provider: restore HA support for metastore (#19777): @Aakcht

Provider apache.spark: 2.0.2rc1

• fix bug of SparkSql Operator log going to infinite loop. (#19449): @huage1994

Provider cncf.kubernetes: 2.2.0rc1

• Added namespace as a template field in the KPO. (#19718): @ReadytoRocc
• Decouple name randomization from name kwarg (#19398): @ferruzzi
• Checking event.status.container_statuses before filtering (#19713): @ulisesojeda
• Coalesce extra params to None in KubernetesHook (#19694): @dstandish
• Change to correct type in KubernetesPodOperator (#19459): @DorConnecteam

Provider databricks: 2.1.0rc1

• Databricks: add more methods to represent run state information (#19723): @alexott
• Databricks - allow Azure SP authentication on other Azure clouds (#19722): @alexott
• Databricks: allow to specify PAT in Password field (#19585): @alexott
• Databricks jobs 2.1 (#19544): @christophergrant
• Update Databricks API from 2.0 to 2.1 (#19412): @Mokubyow
• Authentication with AAD tokens in Databricks provider (#19335): @alexott
• Fixup string concatenations (#19099): @blag

The Databricks provider will be removed from this round due to #20036.

Provider google: 6.2.0rc1

• Added wait mechanizm to the DataprocJobSensor to avoid 509 errors when Job is not available (#19740): @lwyszomi
• Add support in GCP connection for reading key from Secret Manager (#19164): @keze
• Add dataproc metastore operators (#18945): @wojsamjan
• Add support of path parameter for GCloud Storage Transfer Service operators (#17446): @eskarimov
• Move bucket_name validation out of __init__ in Google Marketing Platform operators (#19383): @josh-fell
• Create dataproc serverless spark batches operator (#19248): @MaksYermak
• updates pipeline_timeout CloudDataFusionStartPipelineOperator (#18773): @matsavage
• Support impersonation_chain parameter in the GKEStartPodOperator (#19518): @lwyszomi
• Fix badly merged impersonation in GKEPodOperator (#19696): @potiuk

Provider influxdb: 1.1.0rc1

• Add influxdb operator (#19356): @subkanthi

Provider microsoft.azure: 3.4.0rc1

• Remove unnecessary connection form customizations in Azure (#19595): @josh-fell
• Update Azure modules to comply with AIP-21 (#19431): @josh-fell
• Remove host from hidden fields in WasbHook (#19475): @josh-fell
• use DefaultAzureCredential if login not provided for Data Factory (#19079): @sdebruyn
• Fix argument error in AzureContainerInstancesOperator (#19668): @chinwobble

Provider postgres: 2.4.0rc1

• 19489 - Pass client_encoding for postgres connections (#19827): @subkanthi
• Amazon provider remove deprecation, second try (#19815): @uranusjr

Provider salesforce: 3.4.0rc1

• Do not require all extras for SalesforceHook (#19530): @alexbegg

Provider sftp: 2.3.0rc1

• Add test_connection method for sftp hook (#19609): @cmarteepants

Provider snowflake: 2.3.1rc1

• Remove duplicate get_connection in SnowflakeHook (#19543): @harryplumer

Providers that do not need testing

We also had a number of providers that had documentation-only change (mostly around examples). Those need no testing.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[harryplumer]

snowflake 2.3.1rc1 looks good!

[eskarimov]

#17446 works as expected

[lwyszomi]

google 6.2.0rc1 works fine

[Aakcht]

#19711 and #19777 work as expected, so waiting for the release :)

[raphaelauv]

Checking event.status.container_statuses before filtering (#19713) -> work

[chinwobble]

#19668 tested and working with apache-airflow-providers-microsoft-azure==3.4.0rc1

[josh-fell]

Verified #18693, #19431, #19475, #19595, and #19383

Thanks again @potiuk!

[potiuk]

Thanks for those who tested. Today is last day to do some testing :)

[uranusjr]

cc @dimon222 for #19815

[alexott]

I have capacity to test only over the weekend...

[subkanthi]

Tested #19489 and #19356

[potiuk]

@alexott - good we need to more votes so still some time :)

[alexott]

Tested #19722, #19585, #19335, #19412.
Unfortunately there is a logical error in calculation of the expiration time that may lead to authentication error when executing long-running jobs, just submitted a very small PR to fix it: #20036 - it would be nice to include it into release

[potiuk]

Tested #19722, #19585, #19335, #19412. Unfortunately there is a logical error in calculation of the expiration time that may lead to authentication error when executing long-running jobs, just submitted a very small PR to fix it: #20036 - it would be nice to include it into release

Thanks!

I can remove it from the release and make ad-hoc rc2 for databricks only. This is how we deal with it when we find problems with a candidate provider.

[humit0]

Tested #19420

[potiuk]

Thanks everyone who helped. I am preparing Databricks relase separately now!