fc.assert pass with path in the config

[maciejmyslinski]

Hey!
My property-based test fails (it takes about 800ms, this is ok ✅)
When I copy seed and provide it to fc.assert, it fails (again, it's ok ✅)
When I copy both seed and path and provide it to fc.assert, test passes (it takes about 2ms, it's bad 😢 ❌)

I looks like this:

fc.assert(
  fc.asyncProperty(
    fc.commands(commands),
    async cmds => {
      const getInitialState = () => ({ model, real })
      await fc.asyncModelRun(getInitialState, cmds)
    }
  ),
  {
    seed: 1842107356,
    // path: '31:10' <–– when this is uncommented, the issue occurs.
  }
)

[dubzzz]

It is unfortunately a known issue, fc.commands cannot be replayed for the moment.

In the documentation: https://github.com/dubzzz/fast-check/blob/master/documentation/Arbitraries.md
WARNING: fc.commands cannot be replayed for the moment

To be able to shrink those, the path would have to contain lots of additional details :/

[dubzzz]

Here is what happens when you shrink fc.commands:

Let's consider the selected commands are [c0, c1, c2, c3, c4, c5, c6] .
During the execution c2 was not executed because it does not fulfill its preconditions and commands c5 and c6 neither because c4 found a bug.

If we were using fc.array to generate our commands - it was the case initially - we would not have been able to take into account that some commands are not relevant because not ran.

fc.commands in the contrary takes that into account, instead of shrinking [c0, c1, c2, c3, c4, c5, c6] , it shrinks [c0, c1, c3, c4] . In addition it knows that c4 raised a red flag, so it should always keep it in the shrinking process.

The path as defined today, cannot give fc.commands enough details to be able to know which commands were executed, failed.. That is the main reason why it is not working properly.

For the moment, I have no nice way to make it work, except relying on fc.array if you really care about having a working path. I would say that today it is a won't fix but I keep the point in mind as it is also something I would be interested in.

[dubzzz]

I should come with some kind of workaround solution in the next minor version of fast-check. I have an idea that should work fine for this purpose.

Stay tuned :)

[dubzzz]

PR #294 should solve the issue. Feel free to re-open if it does not.

(not released yet - I will let you know as soon as it gets released)