-
Notifications
You must be signed in to change notification settings - Fork 615
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use pytest-rng
generated seeds for stochastic tests
#6435
Conversation
Hello. You may have forgotten to update the changelog!
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #6435 +/- ##
=======================================
Coverage 99.38% 99.39%
=======================================
Files 452 452
Lines 42789 42787 -2
=======================================
- Hits 42527 42526 -1
+ Misses 262 261 -1 ☔ View full report in Codecov by Sentry. |
4fd9637
to
66ff813
Compare
pytest-rng
generated seeds for stochastic tests [WIP]pytest-rng
generated seeds for stochastic tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me. We're still maintaining a "global" seeding behaviour in the sense that we're globally setting a value that affects local seeds. I'm a big fan of this approach, and it's making our tests much more thread safe. I'm happy to approve this PR in its current state, no blocking comments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for this whole work @astralcai ! It's time for us to get rid of magic power in our previous testings involving any random sampling; I do think we are on the correct track, although probably more perspectives from another senior instead of me should be included.
Also do we have methods to track the traces of salts? Like is it possible to collect the salts for those false negativities.
Co-authored-by: Mudit Pandey <mudit.pandey@xanadu.ai>
Follow up PR to #6435, fixing seeds for more tests. See https://github.com/PennyLaneAI/pennylane/actions/runs/11558049719/job/32170809496
Follow up PR to #6435, fixing seeds for more tests. See https://github.com/PennyLaneAI/pennylane/actions/runs/11558049719/job/32170809496
**Context:** The PennyLane CI suffers from sporadic failures due to stochasticity of test cases. In order to save CI runtime, we have seeded many tests. However, this approach introduces the risk of a test case passing due to a special seed, potentially hiding bugs that could have been discovered. Therefore, it is beneficial to periodically update the seeds to review such tests. **Description of the Change:** - The test suite uses `pytest-rng` to handle seed generation. A `seed` fixture is now available to use for every test case. If seeding is required, please use the seed provided by `pytest-rng` ```python def test_some_function(..., seed): dev = qml.device("default.qubit", seed=seed) ``` - A `local_salt` test marker has been added that allows you to modify the seed locally for a particular test case if the generated seed happens to make your test fail. ```python @pytest.mark.local_salt(42) def test_some_function(..., seed): dev = qml.device("default.qubit", seed=seed) ``` - All tests that uses local seeds has been updated to use the new seeding approach. - An `rng_salt` has been added to `pytest.ini`. This controls the seed generation for the entire test suite. We want to periodically change this `rng_salt` which updates all the seeds across the test suite. - The fixture that sets the global seed for every test case has been removed, and another fixture is added that restores the global seed after each test in case the test modifies the global seed. **Benefits:** We will be able to catch bugs hidden in tests that are passing only because of a magic seed that was set locally. **Possible Drawbacks:** This is a chore to maintain this seed an ensure that all tests pass with the new salt when it is updated. **Related Shortcut Story:** [sc-74294] --------- Co-authored-by: Mudit Pandey <mudit.pandey@xanadu.ai>
Follow up PR to #6435, fixing seeds for more tests. See https://github.com/PennyLaneAI/pennylane/actions/runs/11558049719/job/32170809496
Context:
The PennyLane CI suffers from sporadic failures due to stochasticity of test cases. In order to save CI runtime, we have seeded many tests. However, this approach introduces the risk of a test case passing due to a special seed, potentially hiding bugs that could have been discovered. Therefore, it is beneficial to periodically update the seeds to review such tests.
Description of the Change:
pytest-rng
to handle seed generation. Aseed
fixture is now available to use for every test case. If seeding is required, please use the seed provided bypytest-rng
local_salt
test marker has been added that allows you to modify the seed locally for a particular test case if the generated seed happens to make your test fail.All tests that uses local seeds has been updated to use the new seeding approach.
An
rng_salt
has been added topytest.ini
. This controls the seed generation for the entire test suite. We want to periodically change thisrng_salt
which updates all the seeds across the test suite.The fixture that sets the global seed for every test case has been removed, and another fixture is added that restores the global seed after each test in case the test modifies the global seed.
Benefits:
We will be able to catch bugs hidden in tests that are passing only because of a magic seed that was set locally.
Possible Drawbacks:
This is a chore to maintain this seed an ensure that all tests pass with the new salt when it is updated.
Related Shortcut Story:
[sc-74294]