Status of testing Providers that were prepared on March 23, 2022

[potiuk]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 3.2.0rc1

• Add more filter to s3 hook list_key (#22231): @sunank200
• ImapAttachmentToS3Operator: fix it, update sample dag and update doc (#22351): @vincbeck

Provider apache.beam: 3.3.0rc1

• Add recipe for BeamRunGoPipelineOperator (#22296): @pierrejeambrun

Provider cncf.kubernetes: 3.1.2rc1

• Fix "run_id" k8s and elasticsearch compatibility with Airflow 2.1 (#22385): @potiuk
• Remove RefreshConfiguration workaround for K8s token refreshing (#20759): @dstandish

Provider databricks: 2.5.0rc1

• Operator for updating Databricks Repos (#22278): @alexott

Provider docker: 2.5.2rc1

• Correct multiple_outputs param descriptions mentioning lists/tuples (#22371): @josh-fell

Provider elasticsearch: 3.0.2rc1

• Fix "run_id" k8s and elasticsearch compatibility with Airflow 2.1 (#22385): @potiuk

Provider google: 6.7.0rc1

• Add dataflow_default_options to templated_fields (#22367): @NomNomGit
• Add LocalFilesystemToGoogleDriveOperator (#22219): @ulsc
• Add timeout and retry to the BigQueryInsertJobOperator (#22395): @lwyszomi
• Fix #21989 indentation. A test is added to confirm job is executed on… (#22302): @yathit
• [FIX] typo doc of gcs operator (#22290): @raphaelauv

Provider postgres: 4.1.0rc1

• Add ability to pass config parameters to postgres operator (#21551): @victorphoenix3

Provider snowflake: 2.6.0rc1

• Add support for private key in connection for Snowflake (#22266): @mik-laj

The guidelines on how to test providers can be found in

Verify providers by contributors

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[alexott]

I don't know if it makes sense to release Databricks provider after #22422 is merged?

[raphaelauv]

#22385: @potiuk -> WORK

airlfow 2.1.4 with apache-airflow-providers-cncf-kubernetes==3.1.1

give

[2022-03-23 09:56:34,086] {taskinstance.py:1252} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=kubernetes_dag
AIRFLOW_CTX_TASK_ID=task-one
AIRFLOW_CTX_EXECUTION_DATE=2022-03-14T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-03-14T00:00:00+00:00
[2022-03-23 09:56:34,096] {kubernetes_pod.py:571} ERROR - 'NoneType' object has no attribute 'metadata'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 368, in execute
    self.pod_request_obj = self.build_pod_request_obj(context)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 524, in build_pod_request_obj
    labels = self._get_ti_pod_labels(context)
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 286, in _get_ti_pod_labels
    run_id = getattr(ti, 'run_id') or context['run_id']
AttributeError: 'TaskInstance' object has no attribute 'run_id'

airflow 2.1.4 with apache-airflow-providers-cncf-kubernetes==3.1.2rc1

give

[2022-03-23 09:58:46,424] {taskinstance.py:1252} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=kubernetes_dag
AIRFLOW_CTX_TASK_ID=task-one
AIRFLOW_CTX_EXECUTION_DATE=2022-03-13T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-03-13T00:00:00+00:00
[2022-03-23 09:58:46,428] {kubernetes_pod.py:525} INFO - Creating pod airflow-test-pod.186f4469675a401aa74c57c002ee8be4 with labels: {'dag_id': 'kubernetes_dag', 'task_id': 'task-one', 'run_id': 'scheduled__2022-03-13T0000000000-eef165281', 'try_number': '2'}
[2022-03-23 09:58:46,485] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.186f4469675a401aa74c57c002ee8be4
[2022-03-23 09:58:47,496] {pod_manager.py:159} WARNING - Pod not yet started: airflow-test-pod.186f4469675a401aa74c57c002ee8be4
[2022-03-23 09:58:48,517] {pod_manager.py:199} INFO - toto

[potiuk]

I don't know if it makes sense to release Databricks provider after #22422 is merged?

This is a really a bugfix relase with all those things that were merged till yesterday (we have to re-release all packages due to #22380 . We will have another release in about two weeks with only changed providers. I am not sure how fast #22422 is going to be merged, but I do not want to wait with this release as it caused pain already to a number of people and I prefer to relave the pain ASAP.

Also there might be some bugs found in the new change that will make whole databricks provider to be removed from the release and I want to make sure "all" providers are released ASAP due to the#22380 bug.

[potiuk]

#22385: @potiuk -> WORK

Thanks @raphaelauv! Glad to hear. Next to #22380 this one the one I want to have released asap :) (@alexott - this was also the reason I do not want to wait for any other merges).

[alexott]

Checked #22278 - works as expected

[vincbeck]

#22351 Works!

[josh-fell]

#22371 Looks good.

[pierrejeambrun]

#22296 is mostly documentation and looks good. (The docker image was build locally and is working) 👍

[potiuk]

Few more hourse do do more testing :)

[ulsc]

#22219 works.

[potiuk]

Thanks everyone. All providers are out !