Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scale test failed because ingress wasn't deleted #219

Closed
nicksardo opened this issue Apr 16, 2018 · 2 comments
Closed

Scale test failed because ingress wasn't deleted #219

nicksardo opened this issue Apr 16, 2018 · 2 comments

Comments

@nicksardo
Copy link
Contributor

Test run: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-ingress-gce-e2e-scale/258
GLBC log: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-ingress-gce-e2e-scale/258/artifacts/e2e-258-68260c-master/glbc.log
Build log: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-ingress-gce-e2e-scale/258/build-log.txt
Test code: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/network/scale/ingress.go

Seems that ing-scale-45 was not deleted.

Timeline

build.log:

I0415 11:31:50.938] Apr 15 11:31:50.919: INFO: Cleaning up ingresses...

glbc.log

I0415 11:31:50.920575       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-1 deleted, enqueueing
I0415 11:31:50.925053       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-4 deleted, enqueueing
I0415 11:31:50.937525       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-3 deleted, enqueueing
I0415 11:31:50.937558       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-2 deleted, enqueueing
I0415 11:31:50.951442       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-0 deleted, enqueueing
I0415 11:31:50.951470       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-5 deleted, enqueueing
I0415 11:31:50.966010       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-6 deleted, enqueueing
...
I0415 11:31:53.171198       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-60 deleted, enqueueing
I0415 11:31:53.221113       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-98 deleted, enqueueing
I0415 11:31:53.270698       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-92 deleted, enqueueing
I0415 11:31:53.321062       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-51 deleted, enqueueing
I0415 11:31:53.371148       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-99 deleted, enqueueing

ing-scale-45 was not in the above list of event handler logs.

Instead, ing-scale-45 continues to be periodically enqueued. However, it didn't actually get synced until 12:20.

I0415 11:39:42.262865       1 controller.go:147] Periodic enqueueing of e2e-tests-ingress-scale-c4p8n/ing-scale-45
...
I0415 11:49:42.263056       1 controller.go:147] Periodic enqueueing of e2e-tests-ingress-scale-c4p8n/ing-scale-45
...
I0415 11:59:42.263368       1 controller.go:147] Periodic enqueueing of e2e-tests-ingress-scale-c4p8n/ing-scale-45
...
I0415 12:09:42.263533       1 controller.go:147] Periodic enqueueing of e2e-tests-ingress-scale-c4p8n/ing-scale-45
...
I0415 12:19:42.263882       1 controller.go:147] Periodic enqueueing of e2e-tests-ingress-scale-c4p8n/ing-scale-45
...

I0415 12:52:29.439280       1 controller.go:138] Ingress e2e-tests-ingress-scale-c4p8n/ing-scale-45 deleted, enqueueing
I0415 12:52:29.439327       1 controller.go:240] Syncing e2e-tests-ingress-scale-c4p8n/ing-scale-45
I0415 12:52:29.439823       1 controller.go:260] Ingress "e2e-tests-ingress-scale-c4p8n/ing-scale-45" no longer exists, triggering GC
I0415 12:52:29.439856       1 l7s.go:182] GCing loadbalancer e2e-tests-ingress-scale-c4p8n-ing-scale-45--12b3b819e0bddc89
I0415 12:52:29.440530       1 l7s.go:133] Deleting lb e2e-tests-ingress-scale-c4p8n-ing-scale-45--12b3b819e0bddc89
I0415 12:52:29.440571       1 l7.go:845] Deleting global forwarding rule k8s-fw-e2e-tests-ingress-scale-c4p8n-ing-scale-45--12b3b819e0b0
I0415 12:52:33.142799       1 l7.go:852] Deleting global forwarding rule k8s-fws-e2e-tests-ingress-scale-c4p8n-ing-scale-45--12b3b819e00
I0415 12:52:37.390834       1 l7.go:859] Deleting static IP k8s-fw-e2e-tests-ingress-scale-c4p8n-ing-scale-45--12b3b819e0b0(35.190.76.229)
I0415 12:52:41.381293       1 l7.go:866] Deleting target https proxy k8s-tps-e2e-tests-ingress-scale-c4p8n-ing-scale-45--12b3b819e00
I0415 12:52:44.706582       1 l7.go:876] Deleting sslcert k8s-ssl-77dd24d67214d362-72e064981ed4f559--12b3b819e0bddc89
I0415 12:52:47.222874       1 l7.go:889] Deleting target http proxy k8s-tp-e2e-tests-ingress-scale-c4p8n-ing-scale-45--12b3b819e0b0
I0415 12:52:50.163649       1 l7.go:896] Deleting url map k8s-um-e2e-tests-ingress-scale-c4p8n-ing-scale-45--12b3b819e0b0
I0415 12:52:52.905790       1 backends.go:510] Deleting backend service k8s-be-30181--12b3b819e0bddc89
I0415 12:52:56.330013       1 healthchecks.go:194] Deleting health check k8s-be-30181--12b3b819e0bddc89

This correlates to the namespace deletion in kube-apiserver.

I0415 12:52:29.407968       1 wrap.go:42] DELETE /apis/extensions/v1beta1/namespaces/e2e-tests-ingress-scale-c4p8n/ingresses: (3.367637ms) 200 [[kube-controller-manager/v1.11.0 (linux/amd64) kubernetes/ee4d90a/system:serviceaccount:kube-system:namespace-controller] [::1]:46634]

Kube-apiserver has no record of the specific ingress being deleted at 11:31. Interestingly, build.log has no record of the ingress deletion failing.

cc @MrHohn @rramkumar1
@nicksardo
Copy link
Contributor Author

Looks like a race condition in the test code...

@MrHohn MrHohn mentioned this issue Apr 19, 2018
Merged
@nicksardo
Copy link
Contributor Author

Fixed by the awesome @MrHohn
/close

k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Apr 24, 2018
Merge pull request #62829 from MrHohn/e2e-inrgess-scale-race
1aa67be 
Automatic merge from submit-queue (batch tested with PRs 62495, 63003, 62829, 62151, 62002). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

[e2e ingress-gce] Fix race condition for appending services and ingresses

**What this PR does / why we need it**:
From kubernetes/ingress-gce#219, how the test appends services and ingresses is not thread-safe. This PR fixes it by using channel.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #kubernetes/ingress-gce#219

**Special notes for your reviewer**:
/assign @nicksardo 
I picked this up as it is fun :) Hoping you haven't worked on it yet.

**Release note**:

```release-note
NONE
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nicksardo @k8s-ci-robot