Add suse_default_stack feature flag

It switches the default_stack from cflinuxfs3 to sle15.

#945 (comment)
specifies the expected functionality:

=== Diego ===

* suse_buildpacks=false and suse_default_stack=false

SUSE buildpacks are not installed, but the SUSE rootfs still is. We
can discuss if the rootfs should be removed as well in the future, but
in v2.2.x it has to stay for backwards compatibility.

However, buildpacks can be added back later via cf create-buildpack,
but stacks cannot, so maybe both stacks should always be available.

* suse_buildpacks=true and suse_default_stack=false (current default)

SUSE buildpacks would be installed, but not used by default because
the default stack remains cflinuxfs3.

* suse_buildpacks=false and suse_default_stack=true

This doesn't make much sense because a stack without buildpacks
doesn't work. I would suggest to throw an error because the
probability of user error is very high in this case.

* suse_buildpacks=true and suse_default_stack=true (default in v2.3+)

SUSE buildpacks installed and used by default.

===  Eirini ===

* suse_buildpacks=false and suse_default_stack=false

Not supported because there is no cflinuxfs3 Eirini image. Throws an error.

* suse_buildpacks=true and suse_default_stack=false (current default)

Not supported because there is no cflinuxfs3 Eirini image. In v2.2.x
it silently makes the SUSE stack the default. In v2.3+ it should throw
an error.

* suse_buildpacks=false and suse_default_stack=true

This doesn't make much sense because a stack without buildpacks
doesn't work. I would suggest to throw an error because the
probability of user error is very high in this case.

* suse_buildpacks=true and suse_default_stack=true (default in v2.3+)

SUSE buildpacks installed and used by default.

(cherry picked from commit 24b9c1f)

deploy/helm/kubecf/assets/operations/instance_groups/api.yaml

@@ -1,3 +1,18 @@
+{{- if .Values.features.suse_default_stack.enabled }}
+{{- $cc_jobs := list "cloud_controller_ng" }}
+{{- $cc_jobs = append $cc_jobs "cloud_controller_worker" }}
+{{- $cc_jobs = append $cc_jobs "cloud_controller_clock" }}
+{{- $cc_jobs = append $cc_jobs "cc_deployment_updater" }}
+{{- range $job := $cc_jobs }}
+- type: replace
+   path: /instance_groups/name=api/jobs/name={{ $job }}/properties/cc/default_stack?
+   value: sle15
+{{- end }}
+{{- if not .Values.features.suse_buildpacks.enabled }}
+{{- fail "feature.suse_buildpacks.enabled must be true when features.suse_default_stack.enabled is set" }}
+{{- end }}
+{{- end }}
+
 {{- if .Values.features.suse_buildpacks.enabled }}
 - type: replace
   path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/stacks/-

deploy/helm/kubecf/assets/operations/instance_groups/eirini-helm.yaml

@@ -2,6 +2,16 @@
 - type: replace
   path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/opi?/enabled?
   value: true
+# XXX This check is disabled until v2.3.0 for backwards compatibility reasons.
+# {{- if not .Values.features.suse_default_stack.enabled }}
+# {{- fail "feature.suse_default_stack.enabled must be true to use Eirini" }}
+# {{- end }}
+# XXX This check becomes redundant once the previous check is enabled because
+# XXX the api.yaml file already checks that suse_buildpacks are enabled when
+# XXX suse_default_stack is set.
+{{- if not .Values.features.suse_buildpacks.enabled }}
+{{- fail "feature.suse_buildpacks.enabled must be true to use Eirini" }}
+{{- end }}
 - type: replace
   path: /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/cc/default_stack?
   value: "sle15"

deploy/helm/kubecf/values.yaml

@@ -237,6 +237,7 @@ settings:
 
 features:
   eirini:
+    # When eirini is enabled, both suse_default_stack and suse_buildpacks must be enabled as well.
     enabled: false
     registry:
       service:
@@ -249,6 +250,8 @@ features:
       key: ~
     annotations: {}
     labels: {}
+  suse_default_stack:
+    enabled: false
   suse_buildpacks:
     enabled: true
   autoscaler: