Add @task.kubernetes taskflow decorator

[fletchjeff]

Adds the @task.kubernetes decorator to run arbitrary python code in a DAG in a k8s pod using the KubernetesPodOperator.

closes: #19135

This fixes some of the changes requested with pull request 21077. Shout out to @subkanthi who did most of the work, I just made the final requested changes.

The python_kubernetes_script.py and python_kubernetes_script.jinja2 are now in the airflow/providers/cncf/kubernetes folder.

Co-authored-by: subkanthi subkanthi@gmail.com

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[uranusjr]

This docstring is duplicated; there’s already one near the top, so I took the chance to remove it.

[kaxil]

Thanks for making this cross the finish line @fletchjeff

[kaxil]

@fletchjeff @uranusjr Can one of you look at the failing tests please:

• https://github.com/apache/airflow/runs/7804776296?check_suite_focus=true
• https://github.com/apache/airflow/runs/7804775881?check_suite_focus=true

Feels related since those are provider failures only

[fletchjeff]

Hi @kaxil , both of those failures seem to be related to snowflake connector.

Unknown warnings generated:
  
  /usr/local/lib/python3.7/site-packages/snowflake/connector/options.py:99:
You have an incompatible version of 'pyarrow' installed (5.0.0), please install a version that adheres to: 
'pyarrow<8.1.0,>=8.0.0; extra == "pandas"'

I see similar messages in the airflow/scripts/in_container/verify_providers.py file in KNOWN_DEPRECATED_MESSAGES e.g.

    (
        "You have an incompatible version of 'pyarrow' installed (6.0.1), please install a version that "
        "adheres to: 'pyarrow<8.1.0,>=8.0.0; extra == \"pandas\"'",
        "snowflake",
    )

[kaxil]

Hi @kaxil , both of those failures seem to be related to snowflake connector.

Unknown warnings generated:
  
  /usr/local/lib/python3.7/site-packages/snowflake/connector/options.py:99:
You have an incompatible version of 'pyarrow' installed (5.0.0), please install a version that adheres to: 
'pyarrow<8.1.0,>=8.0.0; extra == "pandas"'

I see similar messages in the airflow/scripts/in_container/verify_providers.py file in KNOWN_DEPRECATED_MESSAGES e.g.

    (
        "You have an incompatible version of 'pyarrow' installed (6.0.1), please install a version that "
        "adheres to: 'pyarrow<8.1.0,>=8.0.0; extra == \"pandas\"'",
        "snowflake",
    )

Should be fixed by #25697 -- I have rebased the PR 🤞

[boring-cyborg]

Awesome work, congrats on your first merged pull request!

[kaxil]

@fletchjeff Thanks for staying on top of this 🎉

[potiuk]

wooooooohoooo 🎉

[fletchjeff]

Thank you all!! Super excited to get to use this now :)