Vendor 3rd-party charts and use ACR images (#305)

* Vendor 3rd-party charts and use ACR images

Move 3rd-party container resources into ACR.

* fixup

* Update docs/development/deploying.md

Co-authored-by: Tom Augspurger <taugspurger@microsoft.com>

---------

Co-authored-by: Tom Augspurger <taugspurger@microsoft.com>

deployment/bin/deploy

@@ -215,22 +215,25 @@ if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
     echo "===== Argo ======="
     echo "=================="
 
-    helm upgrade --install argo-workflows argo/argo-workflows \
+    helm upgrade --install argo-workflows helm/vendored/argo-workflows-0.41.8.tgz \
         -n pc \
         --create-namespace \
         -f "helm/argo-values.yaml" \
         --wait \
-        --timeout 2m0s
+        --timeout 2m0s \
+        --debug
 
     echo "=================="
     echo "===== KEDA ======="
     echo "=================="
 
-    helm upgrade --install keda kedacore/keda \
+    helm upgrade --install keda helm/vendored/keda-2.14.2.tgz \
         -n keda \
         --create-namespace \
+        -f helm/keda-values.yaml \
         --wait \
-        --timeout 2m0s
+        --timeout 2m0s \
+        --debug
 
     # TODO: Figure out how to apply to set this with helm
     echo "Adding KEDA secret"
@@ -283,7 +286,7 @@ if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
     echo "== nginx-ingress =="
     echo "==================="
 
-    helm upgrade --install nginx-ingress helm/ingress-nginx-4.8.3.tgz \
+    helm upgrade --install nginx-ingress helm/vendored/ingress-nginx-4.8.3.tgz \
         -n pc \
         --create-namespace \
         --set controller.replicaCount=1 \

deployment/bin/lib

@@ -97,13 +97,6 @@ function cluster_login() {
 function setup_helm() {
     # Set the helm context to the same as the kubectl context
     export KUBE_CONTEXT=$(kubectl config current-context)
-
-    # Add repos
-    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
-    helm repo add jetstack https://charts.jetstack.io
-    helm repo add argo https://argoproj.github.io/argo-helm
-    helm repo add kedacore https://kedacore.github.io/charts
-    helm repo update
 }
 
 function full_setup() {
@@ -151,4 +144,4 @@ function get_cidr_range() {
     runnerIpAddress=$(curl -s https://ifconfig.me/all.json | jq -r ".ip_addr")
     IFS='.' read -r -a ip_parts <<< "$runnerIpAddress"
     echo "${ip_parts[0]}.${ip_parts[1]}.0.0/16"
-}
+}

deployment/helm/argo-values.yaml

@@ -1,7 +1,19 @@
 server:
   baseHref: /argo/
   secure: false
-  # extraArgs:
-  # - --auth-mode=server
   serviceAccount:
     name: pctasks-sa
+  image:
+    registry: pccomponentstest.azurecr.io
+    repository: argoproj/argocli
+    tag: v3.5.7
+controller:
+  image:
+    registry: pccomponentstest.azurecr.io
+    repository: argoproj/workflow-controller
+    tag: v3.5.7
+executor:
+  image:
+    registry: pccomponentstest.azurecr.io
+    repository: argoproj/argoexec
+    tag: v3.5.7

deployment/helm/keda-values.yaml

@@ -0,0 +1,15 @@
+global:
+  registry: pccomponentstest.azurecr.io
+image:
+  keda:
+    registry: pccomponentstest.azurecr.io
+    repository: kedacore/keda
+    tag: 2.14.0
+  metricsApiServer:
+    registry: pccomponentstest.azurecr.io
+    repository: kedacore/keda-metrics-apiserver
+    tag: 2.14.0
+  webhooks:
+    registry: pccomponentstest.azurecr.io
+    repository: kedacore/keda-admission-webhooks
+    tag: 2.14.0

docs/development/deploying.md

@@ -10,6 +10,63 @@ by `deploy/docker-compose.yaml`.
 There are resources that PCTasks depends on that are not deployed itself, and need to be managed out-of-band. This can be done by creating
 the resources manually through the Azure Portal or through separate terraform processes.
 
+### 3rd party charts and images
+
+For compliance reasons, Microsoft services must use charts and images that are under the control of the service team, typically within ACR or MCR. In PCTasks, these 3rd party charts are vendored into the `deployment/helm` directory, and the images are imported into our internally managed ACR (where they don't already exist in MCR).
+
+#### Nginx-Ingress
+
+The chart can be brought into the `deployment/helm/vendored` directory by running the following command:
+
+```console
+cd deployment/helm/vendored
+helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
+helm pull ingress-nginx/ingress-nginx --version <a.b.c>
+```
+
+The image is in MCR, so can be used directly from there and specified in the CLI options to helm in the `deploy` script.
+
+#### Argo Workflows
+
+The chart can be brought into the `deployment/helm/vendored` directory by running the following command:
+
+```console
+cd deployment/helm/vendored
+helm repo add argo https://argoproj.github.io/argo-helm
+helm pull argo/argo-workflows --version 3.5.7
+```
+
+The images can be imported into your ACR by running the following command:
+
+```console
+az acr login --name <your-acr-name>
+az acr import -n pccomponentstest --source quay.io/argoproj/argocli:v3.5.7 -t argoproj/argocli:v3.5.7 --subscription "Planetary Computer Test"
+az acr import -n pccomponentstest --source quay.io/argoproj/workflow-controller:v3.5.7 -t argoproj/workflow-controller:v3.5.7 --subscription "Planetary Computer Test"
+az acr import -n pccomponentstest --source quay.io/argoproj/argoexec:v3.5.7 -t argoproj/argoexec:v3.5.7 --subscription "Planetary Computer Test"
+```
+
+The image and tag values are specified in the `argo-values.yaml` file and used during installs.
+
+#### KEDA
+
+The chart can be brought into the `deployment/helm/vendored` directory by running the following command:
+
+```console
+cd deployment/helm/vendored
+helm repo add kedacore <https://kedacore.github.io/charts>
+helm pull kedacore/keda --version 2.14.2
+```
+
+The images can be imported into your ACR by running the following command:
+
+```console
+az acr import -n pccomponentstest --source ghcr.io/kedacore/keda-admission-webhooks:2.14.0 -t kedacore/keda-admission-webhooks:2.14.0  --subscription "Planetary Computer Test"
+az acr import -n pccomponentstest --source ghcr.io/kedacore/keda-metrics-apiserver:2.14.0 -t kedacore/keda-metrics-apiserver:2.14.0 --subscription "Planetary Computer Test"
+az acr import -n pccomponentstest --source ghcr.io/kedacore/keda:2.14.0 -t kedacore/keda:2.14.0 --subscription "Planetary Computer Test"
+```
+
+The image and tag values are specified in the `keda-values.yaml` file and used during installs.
+
 ### Deployment Service principal
 
 You'll need a service principal that has sufficient permissions to deploy Azure resources, including creating resource groups and assigning IAM roles.