Write an E2E test that stresses route creation.

[markusthoemmes]

We already have an E2E test that tests route creation. However, we see a large amount of "503: no healthy upstream" directly after creating a route. See #2464 for more in-depth information and analysis on that.

Instead of dealing with failures across the board, I'd like to come up with a test that stresses this scenario specifically. We can then either work to fix that test for good or live with the 503s once we root caused them.

For the meantime, I'd like to propose to retry these errors throughout our test suite as proposed in #3286.

[vagababov]

Scale-from-zero test seems to do that exactly. So, we have the test.

[vagababov]

So with my changes today on my bigass GKE cluster I am able to pass the test reliably at 50 and reliably fail it with 503 at 60 services (ScaleFromZero performance test).

[markusthoemmes]

I don't think the scale-from-zero test stresses what I'm after here.

I'd like to come up with a test that creates a new route and immediately hammers it (quicker than the usual 1 sec retry) and reports how many 404s/503s happen before it eventually works. We should be able to eventually enable that test and not need retries throughout our tests. That's the goal at least.

/assign

[markusthoemmes]

Not working on this

/unassign

[tcnghia]

/close

We have scale/scale-N tests that aren't flakey now.

[knative-prow-robot]

@tcnghia: Closing this issue.

In response to this:

/close

We have scale/scale-N tests that aren't flakey now.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.