[emissary] pod hangs running as non-root

[Bobgy]

Summary

What happened/what you expected to happen?

I ran the following workflow with emissary executor:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: hw
  templates:
  - name: hw
    container:
      image: <any-image-that-defaults-to-non-root-user>
      command: [sh, -c, "echo hello"]

Result:
The pod hangs forever.
logs for main container: standard_init_linux.go:211: exec user process caused "permission denied"
main container errors and completes with this error message.
However, wait container keeps running forever.

Analysis:
because argoexec in /var/run/argo/argoexec is in a volume and not executable from non-root user as expected, somehow wait container cannot know argoexec failed to start because of permission issue.

Expected behavior:
I think the failure is expected, but it'll be drastically better user experience if argo could give a meaningful error message and do not hang forever.

Idea to improve:
let the wait container wait for a heartbeat signal from argoexec, if the signal is not heard for a configured time range (maybe 5min~10min, because we need to wait for pulling the image).
If the signal is not heard, exit early and say sth is wrong with argoexec.

Context

I'm trying to upgrade KFP to argo v3.1 and switch to emissary executor: kubeflow/pipelines#5922

Diagnostics

👀 Yes! We need all of your diagnostics, please make sure you add it all, otherwise we'll go around in circles asking you for it:

What Kubernetes provider are you using?
GKE

What version of Argo Workflows are you running?
v3.1.0

What executor are you running? Docker/K8SAPI/Kubelet/PNS/Emissary
emissary

Did this work in a previous version? I.e. is it a regression?
No, emissary is new, but it works with Docker executor before.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[Bobgy]

Besides the bug report, do we have any existing examples for rootless pods in a workflow with emissary executor?

[alexec]

Interesting. I want the emissary to run as non-root, there is a test for this, but does not capture this case. I think this is what is happening:

• The binary is copied from the init container to the shared volume (by the init container) with chmod 500 (i.e. u+rx).
• The main container starts, the user ID is set by the image itself, so it cannot read the file.

I think there is an easy fix, copy with a+rx (i.e. 0o555) instead by change this line:

https://github.com/argoproj/argo-workflows/blob/master/workflow/executor/emissary/binary.go#L20

This does not increase security attack surface, as it does not add write permissions to the file. The file is in the public domain.

@Bobgy this is a pretty easy change, would you like to submit a PR?

[Bobgy]

Sure, I can give it a try. Agree your proposed solution feels a lot simpler and can cover the case we want.

[Bobgy]

Just to confirm, do other files in /var/run/argo need to have the same permission?

[alexec]

Just to confirm, do other files in /var/run/argo need to have the same permission?

Yes. They all need to be a+r at least.

[Bobgy]

After fix in the PR, we still hit error messages like:

Error: failed to read template: open /var/run/argo/template: permission denied
failed to read template: open /var/run/argo/template: permission denied

(this is an improvement, at least the Pod doesn't hang forever)

TODO: change all files in /var/run to be readable, as commented by #6238 (comment)

[Bobgy]

@alexec can you help me reopen this issue, so that I can keep adding PRs fixing this use-case?

[Bobgy]

Fixed by #6304