Add suse_default_stack feature flag

It switches the default_stack from cflinuxfs3 to sle15.

#945 (comment)
specifies the expected functionality:

=== Diego ===

* suse_buildpacks=false and suse_default_stack=false

SUSE buildpacks are not installed, but the SUSE rootfs still is. We
can discuss if the rootfs should be removed as well in the future, but
in v2.2.x it has to stay for backwards compatibility.

However, buildpacks can be added back later via cf create-buildpack,
but stacks cannot, so maybe both stacks should always be available.

* suse_buildpacks=true and suse_default_stack=false (current default)

SUSE buildpacks would be installed, but not used by default because
the default stack remains cflinuxfs3.

* suse_buildpacks=false and suse_default_stack=true

This doesn't make much sense because a stack without buildpacks
doesn't work. I would suggest to throw an error because the
probability of user error is very high in this case.

* suse_buildpacks=true and suse_default_stack=true (default in v2.3+)

SUSE buildpacks installed and used by default.

===  Eirini ===

* suse_buildpacks=false and suse_default_stack=false

Not supported because there is no cflinuxfs3 Eirini image. Throws an error.

* suse_buildpacks=true and suse_default_stack=false (current default)

Not supported because there is no cflinuxfs3 Eirini image. In v2.2.x
it silently makes the SUSE stack the default. In v2.3+ it should throw
an error.

* suse_buildpacks=false and suse_default_stack=true

This doesn't make much sense because a stack without buildpacks
doesn't work. I would suggest to throw an error because the
probability of user error is very high in this case.

* suse_buildpacks=true and suse_default_stack=true (default in v2.3+)

SUSE buildpacks installed and used by default.

(cherry picked from commit 8aea6c2)

deploy/helm/kubecf/assets/operations/instance_groups/api.yaml

@@ -1,3 +1,24 @@
+# TODO(jandubois) Starting with v2.3.0 enabling eirini will no longer automatically switch to the suse stack.
+{{- if or .Values.features.suse_default_stack.enabled .Values.features.eirini.enabled }}
+
+{{- if not .Values.features.suse_buildpacks.enabled }}
+{{- fail "feature.suse_buildpacks.enabled must be true when features.suse_default_stack.enabled is true" }}
+{{- end }}
+
+- type: replace
+  path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/default_stack?
+  value: sle15
+- type: replace
+  path: /instance_groups/name=cc-worker/jobs/name=cloud_controller_worker/properties/cc/default_stack?
+  value: sle15
+- type: replace
+  path: /instance_groups/name=scheduler/jobs/name=cloud_controller_clock/properties/cc/default_stack?
+  value: sle15
+- type: replace
+  path: /instance_groups/name=scheduler/jobs/name=cc_deployment_updater/properties/cc/default_stack?
+  value: sle15
+{{- end }}
+
 {{- if .Values.features.suse_buildpacks.enabled }}
 - type: replace
   path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/stacks/-

deploy/helm/kubecf/assets/operations/instance_groups/eirini-helm.yaml

@@ -2,9 +2,18 @@
 - type: replace
   path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/opi?/enabled?
   value: true
-- type: replace
-  path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/default_stack?
-  value: "sle15"
+
+# TODO(jandubois) This check is disabled until v2.3.0 for backwards compatibility reasons.
+# {{- if not .Values.features.suse_default_stack.enabled }}
+# {{- fail "feature.suse_default_stack.enabled must be true to use Eirini" }}
+# {{- end }}
+
+# TODO(jandubois) This check becomes redundant once the previous check is enabled because
+# TODO(jandubois) the api.yaml file already checks that suse_buildpacks are enabled when
+# TODO(jandubois) suse_default_stack is set.
+{{- if not .Values.features.suse_buildpacks.enabled }}
+{{- fail "feature.suse_buildpacks.enabled must be true to use Eirini" }}
+{{- end }}
 
 - type: replace
   path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/opi?/url?

deploy/helm/kubecf/values.yaml

@@ -237,6 +237,7 @@ settings:
 
 features:
   eirini:
+    # When eirini is enabled, both suse_default_stack and suse_buildpacks must be enabled as well.
     enabled: false
     registry:
       service:
@@ -249,6 +250,8 @@ features:
       key: ~
     annotations: {}
     labels: {}
+  suse_default_stack:
+    enabled: false
   suse_buildpacks:
     enabled: true
   autoscaler: