Incompatibility with PSPs using read-only allowed host paths

[louisblin]

Summary

Argo Workflows seems to be incompatible with Pod Security Policies (PSPs) specifying read-only allowed host paths. As mentioned in this thread, the init / wait containers surrounding a workflow container mount the same volumes as the main container, but without respecting their read/write mode. For environments using PSPs with read-only allowed host paths, it becomes impossible to run workflows that use volume mounts (as sidecar containers will violate the PSP).

Looking at the code mentioned above, the author claims that mounts need to be read/write to allow overlapping mount paths. However, the main container will already need to mount paths in read/write mode if they overlap, so I don't see why the sidecar containers needs to modify this setting. Am I missing something or could we simply remove this override?

Argo Workflows install: version v3.2.6, on a vanilla k8s cluster with the K8SAPI executor.

Diagnostics

To replicate the issue, here's a sample PSP with a read-only allowed host path:

$ kubectl apply -f - <<EOF
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: readonly-hostpaths
spec:
  allowedHostPaths:
  - pathPrefix: /etc/ssl/certs/
    readOnly: true
  volumes:
  - '*'

  runAsUser:
    rule: RunAsAny
  runAsGroup:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
EOF

And a workflow that will violate it:

$ argo submit - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: entrypoint
  templates:
  - name: entrypoint
    script:
      image: alpine
      source: |
        ls -lha /etc/ssl/certs/
      volumeMounts:
      - mountPath: /etc/ssl/certs/
        name: ssl-certs
        readOnly: true
    volumes:
    - hostPath:
        path: /etc/ssl/certs/
      name: ssl-certs
EOF

After inspecting the workflow in the UI, you get the following message, which indicates that the sidecar container is trying to mount the ssl-certs volume in write mode.

pods "hello-world-bq8pl" is forbidden: unable to validate against any pod security policy: [spec.containers[0].volumeMounts[0].readOnly: Invalid value: false: must be read-only]

This can be confirmed by disabling the PSP and inspecting the pod spec that is created.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[sarabala1979]

@louisblin you use podSpecPatch as workaround to set appropriate values. Can you try and let us know?

[louisblin]

Thanks for the hint @sarabala1979!

It does work, but the patch is particularly unreadable and I can't expect my users to understand this. Problems are:

• Custom patch for each combination of volume mounts
• Patch size grows with the number of mounts
• Makes an assumption about the magic /mainctrfs path prefix

Here's what it looks like:

podSpecPatch: |
  {"containers":[{"name":"wait","volumeMounts":[{"mountPath":"/mainctrfs/etc/ssl/certs","name":"ssl-certs","readOnly":true}]}],"initContainers":[{"name":"init","volumeMounts":[{"mountPath":"/mainctrfs/etc/ssl/certs","name":"ssl-certs","readOnly":true}]}]}

I would very much like to avoid doing this. Is there any reason why the init / wait containers can't mount these volumes using the same mode as the main container? It would be great to remove this line unless there's a good reason for keeping it.

[louisblin]

Any thoughts on removing the line I mentioned, @sarabala1979 or @alexec?

[alexec]

Would you be interested in investigating and submitting a PR to see if the change works?

[louisblin]

Absolutely, I'll give that a try!

[louisblin]

Would you be interested in investingg and submitting a PR to see if the change works?

All tests are passing in #8128. Do you feel this needs more validation?