-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance Tests Optimization: Measure impact of changes #45175
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dmsnell
added
the
Exclude from Metrics
When compiling metrics or statistics about the project, this PR should be ignored.
label
Oct 20, 2022
Size Change: 0 B Total Size: 1.3 MB ℹ️ View Unchanged
|
dmsnell
force-pushed
the
perf-tests/without-changes
branch
from
November 1, 2022 23:11
465b5ac
to
14b8ffe
Compare
dmsnell
force-pushed
the
perf-tests/without-changes
branch
3 times, most recently
from
November 17, 2022 00:28
4cd7f22
to
a538b7b
Compare
dmsnell
force-pushed
the
perf-tests/without-changes
branch
from
November 17, 2022 16:25
a538b7b
to
827b24f
Compare
dmsnell
force-pushed
the
perf-tests/without-changes
branch
from
November 17, 2022 22:49
827b24f
to
6118e60
Compare
Closing for now since active work on the performance tests performance is on hold. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Exclude from Metrics
When compiling metrics or statistics about the project, this PR should be ignored.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
This branch is intended to follow
trunk
without the set of changes meant improve the runtime performance of the Performance Test CI workflow.trunk
as of this test is 20.6mintrunk
and 76 test runs in this branch (there are more intrunk
because the tests finish faster and I'm able to get more data in the same amount of time)trunk
is 2.6min vs. without the changes at 7.5min, so not only has the average test time reduced but also the extrema for the test runs has also reduced and the runtimes are now more consistent.Results
refspecs
#45780In order to eliminate some bias introduced by variations in test run durations over time these numbers reflect only the 30 most recent test runs.
Nov. 1
I'm rebasing all branches and am going to let them re-gather data to see if any clearer patterns emerge. Will be ignoring all test run data before Nov. 1 to see if there were effects due to getting too far behind
trunk
.Why
By comparing the runtimes in this branch against those in
trunk
we can measure the impact of the overall optimization project while deploying improvements incrementally. Running this over time will not only eliminate sampling bias and noise, but it will more importantly help us to eliminate the impact of other changes on the performance test suite which might be conflated with changes we're making to the suite itself (e.g. if someone adds a new set of performance tests to the suite which would slow them down).