feat(controller): Retry archiving later on error. Fixes #3786 #3862

alexec · 2020-08-25T20:22:41Z

Checklist:

Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
The title of the PR is (a) conventional, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
I've signed the CLA.
I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
My builds are green. Try syncing with master if they are not.
My organization is added to USERS.md.

Fixes #3786

alexec · 2020-08-28T22:26:08Z

TestEventOnWorkflowSuccess

alexec · 2020-08-28T22:47:16Z

TestEventOnWorkflowSuccess

alexec · 2020-08-28T23:16:07Z

TestEventOnNodeFail

alexec · 2020-09-04T15:51:38Z

FAIL: TestFunctionalSuite/TestEventOnNodeFail

alexec · 2020-09-04T15:53:43Z

Event occurring out of order:

m controller | �[mtime="2020-09-04T01:39:52Z" level=info msg="Marking workflow completed" namespace=argo workflow=failed-step-event-wf7tt
�[32m controller | �[mtime="2020-09-04T01:39:52Z" level=info msg="Marking workflow as pending archiving" namespace=argo workflow=failed-step-event-wf7tt
�[32m controller | �[mtime="2020-09-04T01:39:52Z" level=info msg="Checking daemoned children of " namespace=argo workflow=failed-step-event-wf7tt
�[32m controller | �[mtime="2020-09-04T01:39:52Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"failed-step-event-wf7tt\", UID:\"6e23f453-b98b-4836-8fb9-f13447d91c44\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"3869\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowFailed' failed with exit code 1"
�[32m controller | �[mtime="2020-09-04T01:39:52Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"failed-step-event-wf7tt\", UID:\"6e23f453-b98b-4836-8fb9-f13447d91c44\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"3869\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowNodeFailed' Failed node failed-step-event-wf7tt: failed with exit code 1"
�[32m controller | �[mtime="2020-09-04T01:39:52Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=3873 workflow=failed-step-event-wf7tt
�[32m controller | �[mtime="2020-09-04T01:39:53Z" level=info msg="Labeled pod completed" namespace=argo pod=failed-step-event-wf7tt
�[32m controller | �[mtime="2020-09-04T01:39:53Z" level=info msg="archiving workflow" namespace=argo uid=6e23f453-b98b-4836-8fb9-f13447d91c44 workflow=failed-step-event-wf7tt
�[32m controller | �[mtime="2020-09-04T01:39:53Z" level=info msg="archiving workflow" namespace=argo uid=6e23f453-b98b-4836-8fb9-f13447d91c44 workflow=failed-step-event-wf7tt

alexec · 2020-09-04T16:12:58Z

Added diagnostics to print when we send a message.

alexec · 2020-09-06T17:20:51Z

test/e2e/argo_server_test.go

  entrypoint: run-archie
  templates:
    - name: run-archie
      container:
        image: argoproj/argosay:v2`).
 		When().
 		SubmitWorkflow().
-		WaitForWorkflow().
+		WaitForWorkflow(fixtures.ToBeArchived).


add message

alexec · 2020-09-06T17:21:44Z

test/e2e/functional_test.go

-				assert.Equal(t, "failed with exit code 1", e.Message)
+			2,
+			func(t *testing.T, es []corev1.Event) {
+				for _, e := range es {


these now appear out of order - which I'm guessing is something to do with Goroutine scheduling

util/sync/key_lock.go

workflow/controller/controller.go

sarabala1979 · 2020-09-16T17:56:36Z

@alexec Can we set up some time to go through changes?

alexec · 2020-09-17T22:18:33Z

@sarabala1979 ready for review.

sarabala1979

LGTM, The workflow key lock will be refactored in controller level parallelism PR

alexec added 4 commits August 25, 2020 13:22

feat(controller): Retry archiving later on error.

9244d5b

fix bug, update test

bd772b0

lint

1552dde

fix e2e test

82e438e

alexec changed the title ~~feat(controller): Retry archiving later on error.~~ feat(controller): Retry archiving later on error. Fixes #3837 Aug 25, 2020

alexec added 4 commits August 25, 2020 15:47

archiving

a526bab

archiving

e3dfbc3

archiving

c97aecf

archiving

888c1c7

alexec marked this pull request as ready for review August 26, 2020 21:03

sarabala1979 mentioned this pull request Aug 27, 2020

Some workflows not appearing in workflow archive. #3880

Closed

alexec marked this pull request as draft August 27, 2020 19:24

alexec added 4 commits August 27, 2020 13:36

Merge branch 'master' into archiving

9e6c332

archiving

f909332

archiving

c697109

archiving

e76a8fe

alexec added this to the v2.11 milestone Aug 28, 2020

alexec added the epic: scaling part 2 label Aug 28, 2020

alexec marked this pull request as ready for review August 28, 2020 20:57

Merge branch 'master' into archiving

2d35836

Merge branch 'master' into archiving

d80a8ed

Merge branch 'master' into archiving

0ac474c

alexec added the P3 label Aug 31, 2020

alexec modified the milestones: v2.11, v2.12 Aug 31, 2020

Merge branch 'master' into archiving

21ab69c

Merge branch 'master' into archiving

dfcaf54

Merge branch 'master' into archiving

1a24538

alexec added 2 commits September 4, 2020 10:15

archiving

bdf757c

archiving

57589b6

alexec mentioned this pull request Sep 4, 2020

feat(controller): Panic or error on mis-matched resource version #3949

Merged

6 tasks

alexec added 2 commits September 4, 2020 15:32

Merge branch 'master' into archiving

5e1e7bf

archiving

30cbf90

alexec marked this pull request as ready for review September 6, 2020 17:20

alexec commented Sep 6, 2020

View reviewed changes

util/sync/key_lock.go Outdated Show resolved Hide resolved

alexec assigned sarabala1979 Sep 10, 2020

archiving

2acba8b

alexec requested a review from jessesuen as a code owner September 15, 2020 00:44

alexec added 2 commits September 14, 2020 17:44

master

60f2903

Merge branch 'master' into archiving

94ce8ea

alexec mentioned this pull request Sep 15, 2020

stuck with deleted pods #3857

Closed

sarabala1979 reviewed Sep 16, 2020

View reviewed changes

workflow/controller/controller.go Show resolved Hide resolved

alexec added 2 commits September 16, 2020 14:52

updated pkg

f9b9017

archiving

12bf782

sarabala1979 approved these changes Sep 18, 2020

View reviewed changes

alexec merged commit 5461d54 into argoproj:master Sep 18, 2020

alexec deleted the archiving branch September 18, 2020 17:07

alexec added epic: reliability and removed epic: scaling part 2 labels Sep 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(controller): Retry archiving later on error. Fixes #3786 #3862

feat(controller): Retry archiving later on error. Fixes #3786 #3862

alexec commented Aug 25, 2020 •

edited

Loading

alexec commented Aug 28, 2020

alexec commented Aug 28, 2020

alexec commented Aug 28, 2020

alexec commented Sep 4, 2020

alexec commented Sep 4, 2020

alexec commented Sep 4, 2020

alexec Sep 6, 2020

alexec Sep 6, 2020

sarabala1979 commented Sep 16, 2020

alexec commented Sep 17, 2020

sarabala1979 left a comment

feat(controller): Retry archiving later on error. Fixes #3786 #3862

feat(controller): Retry archiving later on error. Fixes #3786 #3862

Conversation

alexec commented Aug 25, 2020 • edited Loading

alexec commented Aug 28, 2020

alexec commented Aug 28, 2020

alexec commented Aug 28, 2020

alexec commented Sep 4, 2020

alexec commented Sep 4, 2020

alexec commented Sep 4, 2020

alexec Sep 6, 2020

Choose a reason for hiding this comment

alexec Sep 6, 2020

Choose a reason for hiding this comment

sarabala1979 commented Sep 16, 2020

alexec commented Sep 17, 2020

sarabala1979 left a comment

Choose a reason for hiding this comment

alexec commented Aug 25, 2020 •

edited

Loading