Status of testing Providers that were prepared on October 13, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider alibaba: 2.6.0rc1

• Consolidate hook management in AnalyticDBSparkSensor (#34435): @hussein-awala
• Consolidate hook management in AnalyticDBSparkBaseOperator (#34434): @hussein-awala
• Deprecate get_hook in OSSKeySensor and use hook instead (#34426): @hussein-awala

Provider amazon: 8.8.0rc1

• Implements AwsBaseOperator and AwsBaseSensor (#34784): @Taragolis
• Extend hooks arguments into AwsBaseWaiterTrigger (#34884): @Taragolis
• Allow setup endpoint_url per-service in AWS Connection (#34593): @Taragolis
• Include AWS Lambda execution logs to task logs (#34692): @Taragolis
• respect soft_fail argument when exception is raised for amazon sensors (#34134): @Lee-W
• do not fail operator if we cannot find logs (#34570): @vandonr-amz
• Respect 'soft_fail' argument when running BatchSensors (#34592): @utkarsharma2
• Respect 'soft_fail' argument when running SqsSensor (#34569): @utkarsharma2
• Respect 'soft_fail' argument when running EcsBaseSensor (#34596): @utkarsharma2
• Respect 'soft_fail' argument when running SageMakerBaseSensor (#34565): @utkarsharma2
• Respect soft_fail parameter in S3KeysUnchangedSensor and S3KeySensor (#34550): @utkarsharma2
• Respect soft_fail parameter in LambdaFunctionStateSensor (#34551): @utkarsharma2
• Respect soft_fail parameter in AthenaSensor (#34553): @utkarsharma2
• Respect soft_fail parameter in QuickSightSensor (#34555): @utkarsharma2
• Respect soft_fail parameter in GlacierJobOperationSensor (#34557): @utkarsharma2
• Respect soft_fail parameter in Glue*Sensor (#34559): @utkarsharma2
• Respect soft_fail parameter in StepFunctionExecutionSensor (#34560): @utkarsharma2
• Refactor consolidate import from io in providers (#34378): @eumiro
• Upgrade watchtower to 3.0.1 (#25019) (#34747): @cBiscuitSurprise
• Refactor import from collections (#34406): @eumiro
• Clarify Amazon Lambda invocation and sensing (#34653): @Taragolis
• Refactor multiple equals to contains in providers (#34441): @eumiro
• Rename "bucket" to "gcs_bucket" in GCSToS3Operator (#33031): @hankehly
• Remove duplicate asgiref dependency in Amazon Provider (#34580): @Taragolis
• Update operator_extra_links property (#34506): @asherkhb
• sagemaker.py spell error: tranform change to transform (#34445): @yanfangli85
• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis
• Use AirflowProviderDeprecationWarning in the deprecated decorator in Amazon provider (#34488): @hussein-awala
• Use AirflowProviderDeprecrationWarning in EMR Operators (#34453): @frankcash
• Deprecate get_hook in DataSyncOperator and use hook instead (#34427): @hussein-awala

Provider apache.beam: 5.3.0rc1

• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider apache.druid: 3.6.0rc1

• Allow passing context to DruidDbApiHook (#34603): @saulius

Provider apache.flink: 1.2.0rc1

• respect soft_fail argument when exception is raised for flink sensors (#34476): @Lee-W

Provider apache.hdfs: 4.2.0rc1

• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider apache.hive: 6.2.0rc1

• Consolidate hook management in HiveOperator (#34430): @hussein-awala

Provider apache.kafka: 1.2.0rc1

• Fix typos (double words and it's/its) (#33623): @eumiro
• Fixes the Kafka provider's max message limit error (#32926) (#33321): @aritra24
• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider apache.livy: 3.6.0rc1

• Consolidate hook management in LivyOperator (#34431): @hussein-awala

Provider arangodb: 2.3.0rc1

• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider celery: 3.4.0rc1

• respect soft_fail argument when exception is raised for celery sensors (#34474): @Lee-W

Provider cncf.kubernetes: 7.7.0rc1

• Fix parsing KubernetesPodOperator multiline logs (#34412): @hussein-awala
• Fix KubernetesPodTrigger startup timeout (#34579): @dzhigimont
• Fix Pod not being removed after istio-sidecar is removed (#34500): @Owen-CH-Leung
• Remove duplicated logs by reusing PodLogsConsumer (#34127): @dkim010
• warn level for deprecated set to stacklevel 2 (#34530): @daxmahoney
• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider common.sql: 1.8.0rc1

• respect soft_fail argument when exception is raised for sql sensors (#34199): @Lee-W
• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider databricks: 4.6.0rc1

• DatabricksHook ClusterState (#34643): @Seokyun-Ha
• Respect soft_fail parameter in DatabricksSqlSensor (#34544): @utkarsharma2
• Respect 'soft_fail' argument when running DatabricksPartitionSensor (#34517): @utkarsharma2
• Decode response in f-string (#34518): @Taragolis
• Use aiohttp.BasicAuth instead of HTTPBasicAuth for aiohttp session in databricks hook (#34590): @hussein-awala
• Update list_jobs function in DatabricksHook to token-based pagination (#33472): @oleksiidav

Provider datadog: 3.4.0rc1

• respect soft_fail argument when exception is raised for datadog sensors (#34472): @Lee-W

Provider dbt.cloud: 3.4.0rc1

• Remove useless print from dbt operator (#34322): @hussein-awala

Provider docker: 3.8.0rc1

• expose ulimits host_config-option (#34284): @kahlstrm
• Deprecate get_hook method in DockerOperator (#34432): @hussein-awala

Provider elasticsearch: 5.1.0rc1

• Remove unused argument metadata from es_read and make clearly private (#34790): @dstandish
• Improve intelligibility of end_of_log determination (#34788): @dstandish
• Replace try/except/pass by contextlib.suppress in ElasticSearch provider (#34251): @hussein-awala

Provider github: 2.4.0rc1

• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider google: 10.10.0rc1

• improvement: introduce proejct_id in BigQueryIntervalCheckOperator (#34573): @Zhenye-Na
• respect soft_fail argument when exception is raised for google sensors (#34501): @Lee-W
• Fix GCSToGoogleDriveOperator and gdrive system tests (#34545): @moiseenkov
• Fix LookerHook serialize missing 1 argument error (#34678): @cjonesy
• Fix Dataform system tests (#34329): @bkossakowska
• Refactor DataFusionInstanceLink usage (#34514): @moiseenkov
• Use airflow.models.dag.DAG in Google Provider examples (#34614): @Taragolis
• Deprecate Life Sciences Operator and Hook (#34549): @fdemiane
• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider http: 4.6.0rc1

• respect soft_fail argument when exception is raised for http sensors (#34391): @Lee-W

Provider jenkins: 3.4.0rc1

• respect soft_fail argument when exception is raised for jenkins sensors (#34475): @Lee-W
• Consolidate hook management in JenkinsJobTriggerOperator (#34433): @hussein-awala

Provider microsoft.azure: 8.0.0rc1

• Bump azure-mgmt-containerinstance (#34738): @pankajastro
• Upgrade azure-mgmt-datafactory in microsift azure provider (#34040): @pankajastro
• Add subnet_ids param in AzureContainerInstancesOperator (#34850): @pankajastro
• allow providing credentials through keyword argument in AzureKeyVaultBackend (#34706): @Lee-W
• Name params while invoking ClientSecretCredential (#34732): @irajankumar
• respect soft_fail argument when exception is raised for microsoft-azure sensors (#34494): @Lee-W
• Error handling for when Azure container log cannot be read in properly. (#34627): @krisfur
• Fix hardcoded container name in remote logging option for Azure Blob Storage (#32779): @Adaverse
• Consolidate hook management in AzureBatchOperator (#34437): @hussein-awala
• Consolidate hook management in AzureDataExplorerQueryOperator (#34436): @hussein-awala

Provider openlineage: 1.1.1rc1

• openlineage: adjust log levels (#34801): @JDarDagran

Provider papermill: 3.4.0rc1

• Remove Airflow 2.3 compat from PapermillOperator (#34675): @Taragolis

Provider postgres: 5.7.0rc1

• Add get_ui_field_behaviour() method to PostgresHook (#34811): @pubudu91

Provider presto: 5.2.0rc1

• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider samba: 4.3.0rc1

• Feature: Add Google Cloud Storage to Samba operator (#34369): @ealebed

Provider slack: 8.2.0rc1

• Slack: use default_conn_name by default (#34548): @Khrol

Provider snowflake: 5.1.0rc1

• Decode response in f-string (#34518): @Taragolis
• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

Provider trino: 5.4.0rc1

• Use airflow.exceptions.AirflowException in providers (#34511): @Taragolis

All users involved in the PRs:
@Zhenye-Na @pubudu91 @pankajastro @krisfur @Taragolis @kahlstrm @Khrol @moiseenkov @utkarsharma2 @saulius @irajankumar @ealebed @eumiro @hussein-awala @cjonesy @dstandish @daxmahoney @Seokyun-Ha
@oleksiidav @aritra24 @cBiscuitSurprise @dkim010 @frankcash @hankehly @asherkhb @bkossakowska @Lee-W @Owen-CH-Leung @fdemiane @vandonr-amz @JDarDagran @dzhigimont @Adaverse @yanfangli85

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[Lee-W]

Tested with our examples DAGs with only one expected fail due to azure provider breaking change.

[ealebed]

apache-airflow-providers-samba==4.3.0rc1 has been tested with our examples DAGs (on staging Airflow and test Google Cloud Composer environments)

[pubudu91]

Tested the postgres provider 5.7.0rc1 locally. The fix done is available in it.

[Owen-CH-Leung]

checked apache-airflow-providers-cncf-kubernetes==7.7.0rc1. The fix is working as expected.

[utkarsharma2]

Checked my changes, and they seem to be working fine. Thanks, Utkarsh Sharma

…

On Sun, Oct 15, 2023 at 1:01 PM Owen Leung ***@***.***> wrote: checked apache-airflow-providers-cncf-kubernetes==7.7.0rc1. The fix is working as expected. — Reply to this email directly, view it on GitHub <#34930 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADDLAHIDCF7DE6NJ3B7OA73X7OGLPANCNFSM6AAAAAA57TO654> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hankehly]

#33031 is working on amazon-8.8.0rc1 😄
Here are the commands I used to setup my environment / run my checks if it helps anyone.

breeze
pip install apache-airflow-providers-amazon==8.8.0rc1
pip list | grep amazon  # check version

# Run unit tests
pytest tests/providers/amazon/aws/transfers/test_gcs_to_s3.py

# Run system tests
export AWS_ACCESS_KEY_ID=***
export AWS_SECRET_ACCESS_KEY=***
export AWS_DEFAULT_REGION=***
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
export GCP_PROJECT_ID=***
pytest --system amazon tests/system/providers/amazon/aws/example_gcs_to_s3.py

[Adaverse]

Tested #32779, it works as expected.

[aritra24]

Tested out #33321, works as expected

[saulius]

Tested #34603 in dev environment, works as expected.

[moiseenkov]

#34545 and #34514 work as expected

[JDarDagran]

#34801 tested, works as expected

[Taragolis]

• Use airflow.exceptions.AirflowException in providers #34511: I've checked that packages in this release doesn't use lazy import AirflowException anymore
• Remove Airflow 2.3 compat from PapermillOperator #34675: Check that operator doesn't have any serialization problem, as well as parameters templates after the changes
• Include AWS Lambda execution logs to task logs #34692: works as expected
• Allow setup endpoint_url per-service in AWS Connection #34593: works as expected, test on couple services include STS
• Implements AwsBaseOperator and AwsBaseSensor #34784: No side effect found, this classes don't use right now
• Extend hooks arguments into AwsBaseWaiterTrigger #34884: Couldn't found side effect in existed triggers, additional parameters doesn't provided from community Operators yet
• Remove duplicate asgiref dependency in Amazon Provider #34580: Amazon Provider package's metadata contain asgiref as dependency only once

---

• Use airflow.models.dag.DAG in Google Provider examples #34614: System test/examples only
• Clarify Amazon Lambda invocation and sensing #34653: Doc only

---

• Decode response in f-string #34518: Hard to check because I do not have access to any Databriks or Snowflake environment, should not be a problem since this code executed only on Hooks/Operator's fatal error

[oleksiidav]

I trust the test coverage of #33472 since the changes are pretty trivial. Let's ship it 💪

[hussein-awala]

I checked if all my changes are present, and I tested some of them, all looks good 👍

[asherkhb]

Confirming Amazon 8.8.0rc1 resolves the issues related to my changes 👍

[Khrol]

slack: 8.2.0rc1

Tested locally. LGTM.

[eladkal]

Thank you everyone.
Providers are released
I invite everyone to help improve providers for the next release, a list of open issues can be found here.

[Seokyun-Ha]

I Checked #34643 works as expected!

Sorry for late response 🙏

[krisfur]

All good with #34627 (apologies for the very late response, holidays)