move2kube workflow using ssh keys instead of token

rhdhorchestrator · Jan 11, 2024 · ee80ec6 · ee80ec6
1 parent 62e560f
commit ee80ec6
Show file tree

Hide file tree

Showing 24 changed files with 238 additions and 581 deletions.
diff --git a/m2k/README.md b/m2k/README.md
@@ -30,7 +30,12 @@ Should output
 ```
 namespace/m2k created
 ```
+
 #### 1. Move2Kube
+move2kube needs to have the ssh keys in the `.ssh` folder in order to be able to clone git repository using ssh:
+```bash
+kubectl create secret generic sshkeys --from-file=id_rsa=${HOME}/.ssh/id_rsa --from-file=id_rsa.pub=${HOME}/.ssh/id_rsa.pub
+```
 To run properly, a move2kube instance must be running in the cluster, or at least reachable from the cluster:
 ```bash
 kubectl apply -f k8s/move2kube.yaml
@@ -78,6 +83,15 @@ Then
 ```bash
 eval $(minikube docker-env)
 ```
+
+We need to use `initContainers` in our Knative services, we have to tell Knative to enable that feature:
+```bash
+  kubectl patch configmap/config-features \
+    -n knative-serving \
+    --type merge \
+    -p '{"data":{kubernetes.podspec-init-containers: "enabled"}}'
+  ```
+
 Then generate the `broker` (and other workflow related Knative resources) by running the following command from `m2k/serverless-workflow-m2k`:
 ```bash
 cd serverless-workflow-m2k
@@ -104,7 +118,6 @@ Should output
 ```
 trigger.eventing.knative.dev/error-event-type-trigger-serverless-workflow-m2k created
 trigger.eventing.knative.dev/transformation-saved-event-type-trigger-serverless-workflow-m2k created
-trigger.eventing.knative.dev/plan-created-event-type-trigger-serverless-workflow-m2k created
 broker.eventing.knative.dev/default created
 ```
 
@@ -120,7 +133,7 @@ kubectl -n m2k patch ksvc serverless-workflow-m2k --type merge -p '{
                {
                   "name":"serverless-workflow-m2k",
                   "imagePullPolicy": "Always",
-                  "image":"quay.io/orchestrator/serverless-workflow-m2k:1.0.0-SNAPSHOT",
+                  "image":"quay.io/orchestrator/serverless-workflow-m2k:2.0.0-SNAPSHOT",
                   "env":[
                      {
                         "name":"MOVE2KUBE_URL",
@@ -152,16 +165,20 @@ Should output
 quay.io/orchestrator/serverless-workflow-m2k                                    <none>         cd2e0498ee70   4 minutes ago   487MB
 ```
 #### 4. M2K Knative functions and GC
-* [m2k-service.yaml](k8s/m2k-service.yaml) will deploy 2 kservices that will spin-up the functions when an event is received
+* [m2k-service.yaml](k8s/m2k-service.yaml) will deploy the Knative service that will spin-up the functions when an event is received
 * [m2k-trigger.yaml](k8s/m2k-trigger.yaml) will deploy the triggers related to the expected event and to which the kservice subscribes and rely on to get started
 * [knative-gc.yaml](k8s%2Fknative-gc.yaml) will setup the GC to keep only 3 revisions in the cluster
-From the root folder of the project, first create the Knative services:
+
+As we are using ssh keys to interact with the git repo (ie: bitbucket), similarly to what we have done when deploying the `move2kube` instance, we need to create secrets in the `m2k` namespace containing the keys:
+```bash
+kubectl create -n m2k secret generic sshkeys --from-file=id_rsa=${HOME}/.ssh/id_rsa --from-file=id_rsa.pub=${HOME}/.ssh/id_rsa.pub
+```
+* From the root folder of the project, first create the Knative services:
 ```bash
 kubectl -n m2k apply -f k8s/m2k-service.yaml 
 ```
 Should output
 ```
-service.serving.knative.dev/m2k-create-plan-func created
 service.serving.knative.dev/m2k-save-transformation-func created
 ```
 Next, the Knative Garbage Collector:
@@ -178,7 +195,6 @@ kubectl -n m2k apply -f k8s/m2k-trigger.yaml
 ```
 Should output
 ```
-trigger.eventing.knative.dev/m2k-create-plan-event created
 trigger.eventing.knative.dev/m2k-save-transformation-event created
 ```
 You will notice that the environment variable `EXPORTED_FUNC` is set for each Knative service: this variable defines which function is expose in the service.
@@ -189,7 +205,6 @@ kubectl -n m2k get ksvc
 ```
 ```
 NAME                           URL                                                               LATESTCREATED                     LATESTREADY                       READY   REASON
-m2k-create-plan-func           http://m2k-create-plan-func.m2k.10.110.165.153.sslip.io           m2k-create-plan-func-v1           m2k-create-plan-func-v1           True    
 m2k-save-transformation-func   http://m2k-save-transformation-func.m2k.10.110.165.153.sslip.io   m2k-save-transformation-func-v1   m2k-save-transformation-func-v1   True    
 serverless-workflow-m2k        http://serverless-workflow-m2k.m2k.10.110.165.153.sslip.io        serverless-workflow-m2k-00002     serverless-workflow-m2k-00002     True  
 ```
@@ -205,7 +220,7 @@ curl -X POST -H 'Content-Type: application/json'  serverless-workflow-m2k.m2k.sv
 "repo": "https://bitbucket.org/<repo path>", 
 "sourceBranch": "master",
 "targetBranch": "mk2-swf",
-"token": "<bitbucket auth token>",
+"token": "<optional, bitbucket token with read/write rights, otherwise will use ssh key>",
 "workspaceId": "816fea47-84e6-43b4-81c8-9a7462cf9e1e",
 "projectId": "fc411095-4b3c-499e-8590-7ac09d89d5fc",
 "notification": {
@@ -223,7 +238,6 @@ Then you can monitor the Knative functions pods being created:
 Every 2.0s: kubectl -n m2k get pods                                                                                                                                                              fedora: Fri Oct 13 11:33:22 2023
 
 NAME                                                          READY   STATUS    RESTARTS      AGE
-m2k-create-plan-func-v1-deployment-6d87766bdb-d7hkd           2/2     Running   0             45s
 m2k-save-transformation-func-v1-deployment-545dc45cfc-rsdls   2/2     Running   0             23s
 serverless-workflow-m2k-00002-deployment-58fb774d6c-xxwg2     2/2     Running   0             55s
 ```
@@ -245,7 +259,7 @@ If the timeout expires while the workflow is down, as the jobs service is sendin
     --type merge \
     -p '{"data":{"registries-skipping-tag-resolving":"quay.io"}}'
   ```
-* You can use the Integration tests `SaveTransformationFunctionIT` and `CreatePlanFunctionIT` to debug the code
+* You can use the Integration tests `SaveTransformationFunctionIT` to debug the code
 * If there is a `SinkBinding` generated you need to patch it as the namespace of the broker is not correctly set:
 ```bash
 kubectl patch SinkBinding/sb-serverless-workflow-m2k \

diff --git a/m2k/design.svg b/m2k/design.svg
diff --git a/m2k/k8s/m2k-service.yaml b/m2k/k8s/m2k-service.yaml
@@ -1,42 +1,54 @@
 apiVersion: serving.knative.dev/v1
 kind: Service
-metadata:
-  name: m2k-create-plan-func
-spec:
-  template:
-    metadata:
-      name: m2k-create-plan-func-v1
-    spec:
-      containers:
-        - image: quay.io/orchestrator/m2k-kfunc:latest
-          imagePullPolicy: Always
-          env:
-            - name: EXPORTED_FUNC
-              value: createPlan
-          name: user-container
-          readinessProbe:
-            successThreshold: 1
-            tcpSocket:
-              port: 0
-
----
-apiVersion: serving.knative.dev/v1
-kind: Service
 metadata:
   name: m2k-save-transformation-func
 spec:
   template:
     metadata:
       name: m2k-save-transformation-func-v1
     spec:
+      initContainers:
+        - name: volume-mount-hack
+          image: busybox
+          command: [ "sh", "-c", "cp /root/.ssh/id_rsa /etc/pre-install/. && chown 185 /etc/pre-install/id_rsa" ]
+          volumeMounts:
+            - name: ssh-priv-key
+              mountPath: "/root/.ssh/id_rsa"
+              subPath: id_rsa
+              readOnly: true
+            - name: pre-install
+              mountPath: /etc/pre-install
       containers:
-        - image: quay.io/orchestrator/m2k-kfunc:latest
+        - image: quay.io/orchestrator/m2k-kfunc:2.0.0-SNAPSHOT
           imagePullPolicy: Always
           env:
             - name: EXPORTED_FUNC
               value: saveTransformation
+            - name: SSH_PRIV_KEY_PATH
+              value: /home/jboss/.ssh/id_rsa
           name: user-container
+          volumeMounts:
+            - name: pre-install
+              readOnly: true
+              mountPath: "/home/jboss/.ssh/id_rsa"
+              subPath: id_rsa
+            - name: ssh-pub-key
+              readOnly: true
+              mountPath: "/home/jboss/.ssh/id_rsa.pub"
+              subPath: id_rsa.pub
+
           readinessProbe:
             successThreshold: 1
             tcpSocket:
               port: 0
+      volumes:
+        - name: ssh-priv-key
+          secret:
+            secretName: sshkeys
+            defaultMode: 384
+        - name: ssh-pub-key
+          secret:
+            secretName: sshkeys
+        - name: pre-install
+          emptyDir: { }
+
diff --git a/m2k/k8s/m2k-trigger.yaml b/m2k/k8s/m2k-trigger.yaml
@@ -1,20 +1,5 @@
 apiVersion: eventing.knative.dev/v1
 kind: Trigger
-metadata:
-  name: m2k-create-plan-event
-spec:
-  broker: default
-  filter:
-    attributes:
-      type: create-plan
-  subscriber:
-    ref:
-      apiVersion: serving.knative.dev/v1
-      kind: Service
-      name: m2k-create-plan-func
----
-apiVersion: eventing.knative.dev/v1
-kind: Trigger
 metadata:
   name: m2k-save-transformation-event
 spec:

diff --git a/m2k/k8s/move2kube.yaml b/m2k/k8s/move2kube.yaml
@@ -13,9 +13,33 @@ spec:
     spec:
       containers:
         - name: move2kube
-          image: quay.io/konveyor/move2kube-ui:latest
+          image: quay.io/orchestrator/move2kube-ui:latest
           ports:
             - containerPort: 8080
+          env:
+            - name: SSH_AUTH_SOCK
+              value: /tmp/unix-socket
+          volumeMounts:
+            - name: ssh-priv-key
+              readOnly: true
+              mountPath: "/root/.ssh/id_rsa"
+              subPath: id_rsa
+            - name: ssh-pub-key
+              readOnly: true
+              mountPath: "/root/.ssh/id_rsa.pub"
+              subPath: id_rsa.pub
+          lifecycle:
+            postStart:
+              exec:
+                command: [ "/bin/sh", "-c", "ssh-agent -a /tmp/unix-socket && ssh-add" ]
+      volumes:
+        - name: ssh-priv-key
+          secret:
+            secretName: sshkeys
+            defaultMode: 384
+        - name: ssh-pub-key
+          secret:
+            secretName: sshkeys
 ---
 apiVersion: v1
 kind: Service
@@ -26,4 +50,4 @@ spec:
     - port: 8080
       protocol: TCP
   selector:
-    app: move2kube
+    app: move2kube
diff --git a/m2k/m2k-func/README.md b/m2k/m2k-func/README.md
@@ -1,18 +1,11 @@
 # m2k-kfunc Project
 This projects implements the Knative functions that will interact with Move2Kube instance and Github in order to prepare and save the transformations.
 
-* CreatePlaning: 
-  * Triggered by the event `create-plan`
-  * First, this function will download the archive of the requested branch
-  * Then it will upload this archive to the requested Move2kube project under the provided workspace in order to create a planning
-  * Will send events:
-    * `plan_created_event_type` if success
-    * `error_event_type` if any error
 * SaveTransformationOutput:
   * Triggered by the event `save-transformation`
   * This function will first retrieve the transformation output archive from the Move2Kube project
   * Then it will create a new branch based on the provided input in the provided BitBucket repo
-  * Finally, it will un-archive the previously downloaded file, commit the change and push them to BitBucket using the provided token
+  * Finally, it will un-archive the previously downloaded file, commit the change and push them to BitBucket using the token if provided, otherwise the ssh keys will be used
   * Will send events:
     * `transformation_saved` if success
     * `error` if any error
@@ -42,7 +35,7 @@ mvn clean install
 ## Build image
 To build the image, run:
 ```bash
- docker build -t quay.io/orchestrator/m2k-kfunc -f src/main/docker/Dockerfile.jvm .
+ docker build -t quay.io/orchestrator/m2k-kfunc:2.0.0-SNAPSHOT -f src/main/docker/Dockerfile.jvm .
 ```
 
 ## Run it
@@ -78,7 +71,6 @@ kubectl -n m2k apply -f k8s/m2k-service.yaml
 ```
 Should output
 ```
-service.serving.knative.dev/m2k-create-plan-func created
 service.serving.knative.dev/m2k-save-transformation-func created
 ```
 Finally the triggers
@@ -87,7 +79,6 @@ kubectl -n m2k apply -f k8s/m2k-trigger.yaml
 ```
 Should output
 ```
-trigger.eventing.knative.dev/m2k-create-plan-event created
 trigger.eventing.knative.dev/m2k-save-transformation-event created
 ```
 You will notice that the environment variable `EXPORTED_FUNC` is set for each Knative service: this variable defines which function is expose in the service.
@@ -117,7 +108,6 @@ kubectl -n m2k get ksvc
 Should output
 ```
 NAME                           URL                                                               LATESTCREATED                     LATESTREADY                       READY   REASON
-m2k-create-plan-func           http://m2k-create-plan-func.m2k.10.110.165.153.sslip.io           m2k-create-plan-func-v1           m2k-create-plan-func-v1           True    
 m2k-save-transformation-func   http://m2k-save-transformation-func.m2k.10.110.165.153.sslip.io   m2k-save-transformation-func-v1   m2k-save-transformation-func-v1   True    
 ```
 ### Use it
@@ -128,48 +118,8 @@ kubectl run fedora --rm --image=fedora -i --tty -- bash
 
 1. Go to `http://<move2kubeUI-URL>/` and create a new workspace and a new project inside this workspace.
 
-2. To create a plan, send the following request from a place that can reach the broker deployed in the cluster:
-```bash
-curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/m2k/default"\
- -X POST\
-    -H "Ce-Id: 1234"\
-    -H "Ce-Specversion: 1.0"\
-    -H "Ce-Type: create-plan"\
-    -H "Ce-Source: curl"\
-    -H "Content-Type: application/json"\
-    -d '{"gitRepo": "<repo>", 
-    "branch": "<branch to use when generating archive>",
-    "token": "<optional, bitbucket token with read/write rights>",
-    "workspaceId": "<ID of the workspace previously created>",
-    "projectId": "<ID of the project previously created>",
-    "workflowCallerId": "<string, represents the ID of the SWF calling>"
-    }'
-```
-The URL `http://broker-ingress.knative-eventing.svc.cluster.local/m2k/default` is formatted as follow: `http://broker-ingress.knative-eventing.svc.cluster.local/<namespace>/<broker name>`. If you were to change the namespace or the name of the broker, the URL should be updated accordingly.
-
-To get this URL, run
-```bash
-kubectl get broker -n m2k 
-```
-Should output
-```
-NAME      URL                                                                    AGE    READY   REASON
-default   http://broker-ingress.knative-eventing.svc.cluster.local/m2k/default   107s   True    
-```
-
-You can find the workspace and project IDs in the URL path, ie: `http://localhost:8080/workspaces/<workspace ID>/projects/<project ID>`
-
-You should see a new pod created for the create plan service:
-```bash
-kubectl get pods -n m2k 
-```
-Should output
-```
-NAME                                                         READY   STATUS    RESTARTS   AGE
-m2k-create-plan-func-v1-deployment-5d6c4b6cb9-fkp9s          2/2     Running   0          6s
-```
-
-3. Now you can go to `http://<move2kubeUI-URL>/workspaces/<workspaceID>/projects/<projectID>` and start the transformation. 
+2. Create a plan by upload an archive (ie: zip file) containing a git repo (see https://move2kube.konveyor.io/tutorials/ui for more details)
+3. Then start the transformation. 
 You should be asked to answer some questions, once this is done, the transformation output should be generated.
 
 4. To save a transformation output, send the following request from a place that can reach the broker deployed in the cluster:
@@ -183,7 +133,7 @@ curl -v "http://broker-ingress.knative-eventing.svc.cluster.local/m2k/default"\
     -H "Content-Type: application/json"\
     -d '{"gitRepo": "<repo>", 
     "branch": "<branch to which save the transformation output>",
-    "token": "<BitBucket token with write rights>",
+    "token": "<optional, bitbucket token with read/write rights, otherwise will use ssh key>",
     "workspaceId": "<ID of the workspace previously created>",
     "projectId": "<ID of the project previously created>",
     "transformId": "<ID of the transformation previously created>",
@@ -203,15 +153,19 @@ m2k-save-transformation-func-v1-deployment-76859dc76-h7856   2/2     Running   0
 
 After few minutes, the pods will automatically scale down if no new event is received.
 
+The URL `http://broker-ingress.knative-eventing.svc.cluster.local/m2k/default` is formatted as follow: `http://broker-ingress.knative-eventing.svc.cluster.local/<namespace>/<broker name>`. If you were to change the namespace or the name of the broker, the URL should be updated accordingly.
+
+To get this URL, run
+```bash
+kubectl get broker -n m2k 
+```
+Should output
+```
+NAME      URL                                                                    AGE    READY   REASON
+default   http://broker-ingress.knative-eventing.svc.cluster.local/m2k/default   107s   True    
+```
 
 ## Related Guides
 
 - Funqy HTTP Binding ([guide](https://quarkus.io/guides/funqy-http)): HTTP Binding for Quarkus Funqy framework
 
-## Provided Code
-
-### Funqy HTTP
-
-Start your Funqy functions using HTTP
-
-[Related guide section...](https://quarkus.io/guides/funqy-http#get-query-parameter-mapping)