fix: prevent stdout from disappearing in script templates. Fixes #11330

[boiledfroginthewell]

Fixes #11330

Motivation

Outputs of the executed command are asynchronously read and copied to cmd.Stdout. cmd.Stdout must be closed after copies are done otherwise command outputs are gone.

As described below, however, copies can never be end because sub-processes can keep running and can write to stdout indefinitely.

argo-workflows/workflow/executor/os-specific/signal_darwin.go

Lines 27 to 38 in 6166464

	func Wait(process *os.Process) error {
	// We must copy the behaviour of Kubernetes in how we handle sub-processes.
	// Kubernetes only waits on PID 1, not on any sub-process that process might fork.
	// The only way for those forked processes to run in the background is to background the
	// sub-process by calling Process.Release.
	// Background processes always become zombies when they exit.
	// Because the sub-process is now running in the background it will become a zombie,
	// so we must wait for it.
	// Because we run the process in the background, we cannot Process.Wait for it to get the exit code.
	// Instead, we can reap it to get the exit code
	pid := process.Pid
	if err := process.Release(); err != nil {

Modifications

Wait() is added to wait the copy.
WaitDelay is also added. It will close the pipes (stdout/stderr) of the executed command to stop sub-processes writing any more texts.

Verification

Tests were added.
I also ran the following command for more than an hour.

$ docker run \
    --rm -it \
    --cpus 0.5 \
    -e ARGO_CONTAINER_NAME=main \
    -e ARGO_INCLUDE_SCRIPT_OUTPUT=true \
    -v $PWD/dist/argoexec:/var/run/argo/argoexec \
    busybox:1.35 \
    sh
$ echo '{}' > /var/run/argo/template
$ while :; do
    rm -f /var/run/argo/ctr/main/*
    < /dev/null /var/run/argo/argoexec emissary --loglevel info --log-format text -- echo hello
    ls -l /var/run/argo/ctr/main
    [ ! -s /var/run/argo/ctr/main/stdout ] && break
    [ ! -s /var/run/argo/ctr/main/combined ] && break
done

[terrytangyuan]

@alexec Could you help review this when you get a chance?

[sonbui00]

I got a fail test

--- FAIL: TestSimpleStartCloser (0.43s)
    /Users/sonbui/go/src/github.com/argoproj/argo-workflows/workflow/executor/os-specific/command_test.go:31: 
        	Error Trace:	/Users/sonbui/go/src/github.com/argoproj/argo-workflows/workflow/executor/os-specific/command_test.go:31
        	Error:      	Not equal: 
        	            	expected: "A123456789B123456789C123456789D123456789E123456789"
        	            	actual  : "-n A123456789B123456789C123456789D123456789E123456789\n"

From man page

Most notably, the builtin echo in sh(1) does not accept the -n option.  Consult the builtin(1) manual page.