Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

πŸ‘©β€πŸŒΎ Relax flaky performance test #640

Merged
merged 2 commits into from
Feb 22, 2021

Conversation

chapulina
Copy link
Contributor

🦟 Bug fix

Addresses #429

Summary

The test rarely fails, but when it does, it looks like it's just during the 1st iteration, where there's only one matching entity and one non-matching, with a difference of a few nanoseconds:

147: [ RUN      ] EntityComponentManagerPerfrormance.Each
147: /var/lib/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-bionic-amd64/ign-gazebo/test/performance/each.cc:128: Failure
147: Expected: (cacheEntityAvg) < (cachelessEntityAvg), actual: 417.82 vs 386.29
147: Matching Entity Count =		1
147: Nonmatching Entity Count =	1
147: Each Iterations =		100
147: Cache total =			41782 ns
147: Cache avg per iter =		417.81999999999999 ns
147: Cache avg per iter*entity =	417.81999999999999 ns
147: Cacheless total =		38629 ns
147: Cacheless avg per iter=		386.29000000000002 ns
147: Cacheless avg per iter*entity=	386.29000000000002 ns
147: 
147: [  FAILED  ] EntityComponentManagerPerfrormance.Each (3329 ms)

I think that this isn't unexpected, the performance advantages of the cached Each call are most noticeable with more entities. The lower the number of entities, the more the test may be affected by random variation on the CI machine at the time the test is running.

So I think it's reasonable to relax the test a bit.

Checklist

  • Signed all commits for DCO
  • Added tests
  • Updated documentation (as needed)
  • Updated migration guide (as needed)
  • codecheck passed (See contributing)
  • All tests passed (See test coverage)
  • While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge


https://github.com/osrf/buildfarmer/issues/156

Signed-off-by: Louise Poubel <louise@openrobotics.org>
@chapulina chapulina added the tests Broken or missing tests / testing infra label Feb 19, 2021
@github-actions github-actions bot added the 🏰 citadel Ignition Citadel label Feb 19, 2021
@codecov
Copy link

codecov bot commented Feb 19, 2021

Codecov Report

Merging #640 (39290be) into ign-gazebo3 (f8c9050) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##           ign-gazebo3     #640   +/-   ##
============================================
  Coverage        77.71%   77.71%           
============================================
  Files              210      210           
  Lines            11615    11615           
============================================
  Hits              9027     9027           
  Misses            2588     2588           

Continue to review full report at Codecov.

Legend - Click here to learn more
Ξ” = absolute <relative> (impact), ΓΈ = not affected, ? = missing data
Powered by Codecov. Last update f8c9050...39290be. Read the comment docs.

@chapulina chapulina merged commit 7b4b258 into ign-gazebo3 Feb 22, 2021
@chapulina chapulina deleted the chapulina/3/perf branch February 22, 2021 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏰 citadel Ignition Citadel tests Broken or missing tests / testing infra
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants