nargo test passes or fails arbitrarily with no changes to the code

[harshnambiar]

Aim

While adding tests to this repository, I ran nargo test twice. Once it failed, and once it didn't. I had made no changes to the code between the nargo test commands.

Expected Behavior

The tests should ideally both pass.

Bug

However, only once it did. When I kept running nargo test again and again, the test always subsequently passed. However, if someone runs it just once, they might wrongly interpret that a test is failing.

To Reproduce

1. clone the repository mentioned above
2. cd fraction
3. nargo test

Installation Method

None

Nargo Version

0.7.1

Additional Context

No response

Would you like to submit a PR for this Issue?

No

Support Needs

No response

[jfecher]

I was able to replicate this on master. Running 10 times test_sum_negative in the repository failed twice and test_diff failed once.

[jfecher]

I've confirmed that this does not occur with --experimental-ssa (the tests always pass with it). So it seems there is some nondeterministic part of the old SSA code. Because it is only occurring in the old ssa code that we'll be removing anyway, I'm going to leave this issue until the experimental ssa is switched on in Noir by default. Until then, you can use the --experimental-ssa flag as a workaround.

[TomAFrench]

Closing as the legacy SSA code is now no longer supported.

[harshnambiar]

So I'm on 0.9.0 and the same library tests are still causing this problem. Here is the terminal output for 2 consecutive runs with no changes inbetween:

`ritik@HP-EliteBook-8460w:~/hello_aztec/fraction$ nargo test
Running 18 test functions...
Testing test_cube...
ok
Testing test_diff...
failed
Testing test_sum...
ok
Testing test_integer...
ok
Testing test_square...
ok
Testing test_ceiling...
ok
Testing test_globals...
ok
Testing test_ceiling2...
ok
Testing test_compare...
ok
Testing test_floor2...
ok
Testing test_reduce...
ok
Testing test_sum_negative...
failed
Testing test_not_integer...
ok
Testing test_floor...
ok
Testing test_zero_neutrality...
ok
Testing test_inv...
ok
Testing test_mul...
ok
Testing test_div...
ok
Error: 2 tests failed

Location:
crates/nargo_cli/src/cli/mod.rs:74:5
ritik@HP-EliteBook-8460w:~/hello_aztec/fraction$ nargo test
Running 18 test functions...
Testing test_floor2...
ok
Testing test_diff...
ok
Testing test_floor...
ok
Testing test_inv...
ok
Testing test_integer...
ok
Testing test_not_integer...
ok
Testing test_ceiling...
ok
Testing test_cube...
ok
Testing test_zero_neutrality...
ok
Testing test_sum...
ok
Testing test_ceiling2...
ok
Testing test_div...
ok
Testing test_mul...
ok
Testing test_sum_negative...
ok
Testing test_compare...
ok
Testing test_reduce...
ok
Testing test_square...
ok
Testing test_globals...
ok
All tests passed
`

Is it ok to reopen this? @TomAFrench

[TomAFrench]

Yep, if you're still experiencing this then we should have an issue open for it.

[harshnambiar]

Same steps to reproduce it as before. nargo test in the fraction folder should do it.

[TomAFrench]

I can reproduce on 0.9.0 but not on the nightly release. Can you check if it's fixed for you on that?

[jfecher]

0.10.0 will be the first release with the legacy SSA code removed (and the old --experimental-ssa flag made the default), so it is expected that this is still failing on 0.9.0.

[TomAFrench]

Ok, thanks Jake. I'm going to close this issue in that case. @harshnambiar Feel free to reopen if you experience this on nightly or 0.10.0 when it is released.