pass volumes/volume-mounts and env-vars through to gitsync containers

[adwk67]

Description

See https://github.com/orgs/stackabletech/discussions/49.
Also part of stackabletech/issues#548.

This is to enable SSH-usage using volumes/volumeMounts/envOverrides (which should be available anyway): incorporating this into the CRD is not part of this ticket.

CI tests 🟢 https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/136/

OKD 🟢

--- PASS: kuttl (806.00s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/smoke_airflow-2.8.1_openshift-true_executor-kubernetes (215.72s)
        --- PASS: kuttl/harness/cluster-operation_airflow-latest-2.8.1_openshift-true (269.89s)
        --- PASS: kuttl/harness/logging_airflow-2.8.1_openshift-true (188.91s)
        --- PASS: kuttl/harness/orphaned-resources_airflow-latest-2.8.1_openshift-true (176.65s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.8.1_ldap-authentication-server-verification-tls_openshift-true_executor-kubernetes (195.68s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-2.8.1_openshift-true_executor-kubernetes (223.35s)
        --- PASS: kuttl/harness/mount-dags-configmap_airflow-latest-2.8.1_openshift-true_executor-kubernetes (144.86s)
        --- PASS: kuttl/harness/resources_airflow-latest-2.8.1_openshift-true (127.48s)

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

Preview Give feedback

1. Changes are OpenShift compatible
2. Helm chart can be installed and deployed operator works
3. Integration tests passed (adapted gitsync test)
4. Changes need to be "offline" compatible

Reviewer

Preview Give feedback

1. Code contains useful comments
2. Code contains useful logging statements
3. (Integration-)Test cases added (adapted gitsync test)
4. Documentation added or updated. Follows the style guide.
5. Changelog updated
6. Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Preview Give feedback

1. Feature Tracker has been updated
2. Proper release label has been added
3. Roadmap has been updated

[razvan]

Left some comments. Happy to approve after you respond.

[PaulienVa]

@adwk67 Thank you lots for merging this so fast! when will this be released?

[adwk67]

You can use the nightly version of the operator chart already (version e.g. airflow-operator:0.0.0-dev). The next release (24.7) is due by the end of next month.

[lfrancke]

Is there anything to document here?

[adwk67]

This was a bugfix as they should have been passed through anyway. Now the code hopefully matches the docs :)

[PaulienVa]

@adwk67 we are testing the nightly atm and it is not working yet.

We are connecting the git-sync container through https, so we need to use some certificates.
Again; for the webserver and the scheduler it works perfectly fine. However when starting off a DAG with a kubernetesExecutor the property http.sslCAInfo configured like documented here, using the gitSyncConf in the AirflowCluster Resource does not get propagated through the git-sync container of my kubernetes Executor. Hence my DAGs cannot connect to git and do not run.

Is it possible to fix this in the shortest terms; this still is an inconsistent state of the operator. (webserver and scheduler working fine, executors not)

[razvan]

@adwk67 we are testing the nightly atm and it is not working yet.

Hey, are looking into this.

Would you mind in the meantime, opening a new issue and providing the exact definition of the AirflowCluster object ?

Thank you.

[PaulienVa]

Will do! Thank you!