Rework Airflow page to include guidance on Kubernetes without kedro-airflow-k8s

[astrojuanlu]

Description

The maintainers have mentioned that they'd rather move users towards kedro-airflow

Documentation page (if applicable)

Context

[DimedS]

I think it makes sense to guide our users on how to run Kedro projects on Airflow with execution in separate containers on Kubernetes. Currently, the only guide available was the now-deprecated kedro-airflow-k8s plugin.

It seems worthwhile to do a small spike to explore the best approach. We might need to modify the kedro-airflow plugin or, at the very least, update the documentation to explain how to use kedro-airflow with kedro-docker and how to adjust the DAG to work with Kubernetes.

This will become even more important when we implement node grouping in Airflow (issue #962), as it will make running these groups in separate environments more convenient.

[astrojuanlu]

It's a good point. I'll reframe the scope of this issue.