test/end-to-end/core:183: oc status output fragmented across oc run --attach

[stevekuznetsov]

Seeing this in AWS-based image.

$ oc run cli-with-token --attach --image=openshift/origin:${TAG} --restart=Never -- cli status --loglevel=4 > ${LOG_DIR}/cli-with-token.log 2>&1
...
Jun 13 15:59:11 Running test/end-to-end/core.sh:183: executing 'cat '/tmp/openshift/test-end-to-end-docker//logs/cli-with-token.log'' expecting success and text 'In project test'...
Jun 13 15:59:11 FAILURE after 0.010s: test/end-to-end/core.sh:183: executing 'cat '/tmp/openshift/test-end-to-end-docker//logs/cli-with-token.log'' expecting success and text 'In project test': the output content test failed
Jun 13 15:59:11 Standard output from the command:
Jun 13 15:59:11 Waiting for pod test/cli-with-token to be running, status is Pending, pod ready: false
Jun 13 15:59:11 I0613 19:59:08.888906       1 factory.go:129] Using in-cluster configuration
Jun 13 15:59:11 I0613 19:59:08.916543       1 factory.go:129] Using in-cluster configuration
Jun 13 15:59:11 I0613 19:59:08.918458       1 factory.go:129] Using in-cluster configuration
Jun 13 15:59:11 pod/cli-with-token runs openshift/origin:4148579
Jun 13 15:59:11 2 warnings identified, use 'oc status -v' to see details.
Jun 13 15:59:11 
Jun 13 15:59:11 There was no error output from the command.

We see some of the output from oc status ... but not all of it?

@deads2k @ncdc thoughts?

[mfojtik]

This seems to be more like a CLI issue. @Kargakis can you look this up? If it turns to be CLI and not platform specific, please re-assign to @fabianofranz

[stevekuznetsov]

I'm not sure if it's CLI -- I guess it's possible there are cases where oc status doesn't print everything it needs to, but perhaps it's an oc run --attach problem

[stevekuznetsov]

Confirm I've seen this more than once, same log.

[smarterclayton]

Before or after rebase? Looks like before.

[mfojtik]

@smarterclayton i don't think this is related to the other attach flake... but rebase could make this worse. any suggestion who would be the best candidate to look it up?

[ncdc]

I was seeing this before the rebase, using go-dockerclient.

[liggitt]

@ncdc output of attach still being seen as truncated at times (see https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/2734/console). is there a known race between a process exiting and the output being sent?

[ncdc]

It is known, yes. I don't know why it's happening.

[0xmichalis]

@stevekuznetsov is this related to #9371?

[stevekuznetsov]

@Kargakis not that I can tell, no. In this case output is missing and it is missing from the command

[0xmichalis]

The test has been refactored ever since - can we close this?

[stevekuznetsov]

I'm not seeing any refactors here, what do you mean?

[0xmichalis]

ugh, nvmd

[liggitt]

we commented out the test. #11240 is open it re-enable it. @ncdc, did we pick up that vendored fix yet?

[ncdc]

@liggitt no, and I imagine it will be a while. Kube has to switch from using the now-deprecated docker/engine-api back to docker/docker, and then Origin will have to pick up the change as well.

[liggitt]

Do we consider logs to be critical enough function that we need to do extraordinary things to fix this before we release?

[smarterclayton]

If logs aren't reliable you can't script with them. They were previously scriptable, so that's a problem.

…

On Mon, Jan 23, 2017 at 9:49 AM, Jordan Liggitt ***@***.***> wrote: Do we consider logs to be critical enough function that we need to do extraordinary things to fix this before we release? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#9309 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p97p4pNFLzgBu8sJ7o4nYIlvM5pvks5rVL3qgaJpZM4I0s_v> .

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[stevekuznetsov]

Looks like a manifestation of our favorite journald bug

/close