sequential scenario run on parallel context

[sambathkumar-sekar]

🤔 What's the problem you're trying to solve?

I know that cucumber scenarios should be isolated from one another in terms of test data and other aspects. but in lot of real life scenarios, the test data is very limited and should be re-used across scenarios in a feature file. previous Cucumber6-Junit4 platform was able to run only the feature files in parallel but not the scenarios. this actually helped in test executions with limited test data across scenarios. But with Cucumber7-Junit5 platform, by default all the scenarios are executed in parallel in available threads which breaks the previous tests with limited test data and there is no provision to configure the behavior. I know there are already few discussions going on like Discussion on sequential scenario runs where contributor @mpkorstanje mentioned it is the expected behavior and on exceptional situation we can use resource locking mechanisms which results in adding new tags in all feature files.

✨ What's your proposed solution?

I have made changes on the cucumber-junit-platform-engine, with which we can configure the scenarios to run in sequence but features in parallel. can be merged with a PR if accepted.

I was able to do this by configuring individual getExecutionMode() layer for NodeDescriptor & PickleDescriptor classes and adding a configuration property in Constants

can I create a PR for this?

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[mpkorstanje]

I assume that you're using a fixed number of threads to ensure parallel execution does not consume more resources then available?

If so, bear in mind that unlike Surefire executing JUnit 4 in parallel, JUnit 5 does not guarantee that there will be a maximum number of threads in the span of a suites execution, or even that there is a maximum number of threads alive concurrently. This follows from the underlying fork-join pool implementation, but also see the second note in the JUnit User Guide on parallel execution and some of the implications discussed in #2512 and linked JUnit 5 issues. It has already been proven to be a problem when using web drivers.

This is a problem that can be worked around, but the work around is non-trivial either. So the whole setup would be kinda janky.

As a more elegant solution have you considered using one or more object pools to store and reuse the limited resources? I do not believe it to be to difficult to retrofit an existing project with an object pool.

I'm also a little apprehensive that once parallel execution of features becomes an explicitly supported option, there will be more requests for features that would facilitate it such as for example feature hooks. This is definitely not a direction the Cucumber project as a whole wants to go.

So ideally I would like to push people toward a more comprehensive solution such as object pools to keep the complexity of the framework down.

Still, given the right use case, I'm open to considering adding a same thread execution mode for features, but currently I don't believe it is the right solution to your problem.

[sambathkumar-sekar]

Thanks for Responding!

on the usage of object pools:

So my features and scenarios looks below,

And our typical parallel execution with Cucumber6-Junit4 looks like below

we have very limited user credentials (we cannot have more users on the system) and we have allocated one user per feature file. each scenario should definitely starts with logging into the app and ends with logging out. even if I create an object pool, then each scenario should wait till the execution of the previous scenario to get the user as the user is available only when logged out - this would be the same case as running scenarios in sequence without the overhead to configure the object pool in a multi threaded context.

we have hundreds of features like this. we have benefitted from running features in parallel but not scenarios where in the execution time was around an hour for 500 scenarios. if we aren't able to configure scenarios to run sequentially then we can't use parallelism and looking at 7-8 hours of execution time.

And for this "Surefire executing JUnit 4 in parallel, JUnit 5 does not guarantee that there will be a maximum number of threads in the span of a suites execution, or even that there is a maximum number of threads alive concurrently" --- I'm aware of this issue and currently using the workaround of having a custom fixed strategy(ParallelExecutionConfigurationStrategy) implementation.

I believe lot of people who are having test data constraints can utilize this Parallel Features - Sequential Scenarios if implemented

[mpkorstanje]

I'm having some trouble solving an apparent contradiction.

we have very limited user credentials (we cannot have more users on the system)
we have hundreds of features like this

It doesn't seem like your user credentials are limited. With hundreds of feature files there must be hundreds of users. Rather it seems there may be another constraint.

I would guess that you've manually created one user for each feature file and are using a Background to log the user in before each scenario.

Is that correct?

If so, have you considered placing all user credential in the object pool and hand them out to any scenario rather then tie them to a specific feature file?

This should also speed up your execution. Currently you are limited by the slowest feature file. This could be the slowest scenario.

[sambathkumar-sekar]

For this

I would guess that you've manually created one user for each feature file and are using a Background to log the user in before each scenario.

Yes, that's exactly what I'm doing

To give you a more info on the below

It doesn't seem like your user credentials are limited. With hundreds of feature files there must be hundreds of users. Rather it seems there may be another constraint.

If so, have you considered placing all user credential in the object pool and hand them out to any scenario rather then tie them to a specific feature file?

I cannot utilize the user created for one feature into another. Each feature is mapped with a Retail Store and a store can have only one user per role. And the system does not support concurrent logins with the same credentials. this is where the scenarios in the feature should be using the same credentials (we use top level user for most scenarios as he has all the privileges) .

[mpkorstanje]

Thanks for explaining.

Between your use case, backwards compatibility with JUnit 4 and the existence of the cucumber-jvm-parallel-plugin I reckon there is sufficient reason to add this feature. So please feel free to send a pull request that adds cucumber.execution.execution-mode.feature property with values same-thread and concurrent. Please also ensure the documentation is updated and test coverage is added.

And to manage expectations, I won't be able to dedicate much time to Cucumber in the next few weeks. It may be a while before your pull request is reviewed, merged and released.

[sambathkumar-sekar]

Thanks @mpkorstanje. I will create a PR with changes and updated documentation.

[mpkorstanje]

Fixed by #2604