Executing tests in Quarkus 3.13+ are extremely slow

[mschorsch]

Describe the bug

Since the introduction of the annotation @WithTestResource(value = OidcWiremockTestResource.class), tests are executed extremely slowly because the application is started for each test class. If @WithTestResource(value = OidcWiremockTestResource.class, restrictToAnnotatedClass = false) is used, the tests are fast again.

Note: The constant restarts then also show the classloader problems which then lead to the OOMs (#42303, #42355).

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

1. Download slow-tests-reproducer.zip
2. ./gradlew clean build -> Extremly slow
3. Replace all @WithTestResource(value = OidcWiremockTestResource.class) with @WithTestResource(value = OidcWiremockTestResource.class, restrictToAnnotatedClass = false)
4. ./gradlew clean build -> Fast again

Output of uname -a or ver

No response

Output of java -version

Java 21

Quarkus version or git rev

Quarkus 3.13.1

Build tool (ie. output of mvnw --version or gradlew --version)

Gradle 8.8

Additional information

No response

[mschorsch]

@gsmet @geoand might be interested

[geoand]

The slowness is also likely being exarcebated by the fact that under yet to be determined cases, we are leaking ClassLoaders

[geoand]

@edeandrea I wonder if we need to make it clearer in the docs that @WithTestResource (by default) causes a restart which can slow down the testsuite (regardless of the ClassLoader leaks which we can hopefully fix)

[mschorsch]

@edeandrea I wonder if we need to make it clearer in the docs that @WithTestResource (by default) causes a restart which can slow down the testsuite (regardless of the ClassLoader leaks which we can hopefully fix)

I read this to mean that this issue is not a bug but the expected (default) behavior of annotation @WithTestResource. Is that correct?

[geoand]

Yes, absolutely, but it seems like we need to make that clearer in the docs :)

[gsmet]

I think we need more than something in the Javadoc, we also need to document it very clearly in the doc.

And we should also make it clear in the migration guide.

We already had two reports of this and we are in the middle of the summer.

[geoand]

Sure yeah, all that makes sense.

@edeandrea are you up for it?

[gsmet]

BTW, @mschorsch you confirm that you didn't use quarkus update? Because I actually made a recipe that should keep the original intent of QuarkusTestResource when upgrading.

[mschorsch]

BTW, @mschorsch you confirm that you didn't use quarkus update? Because I actually made a recipe that should keep the original intent of QuarkusTestResource when upgrading.

Yes we do not use quarkus update, it doesn't work for us, but I haven't investigated this further.

[gsmet]

@mschorsch another small question to make sure I fully understand this: you had @QuarkusTestResource set up without restriction on one class and you changed it to @WithTestResource with restriction on many classes? And that's when things started to get slow, right?

[mschorsch]

@mschorsch another small question to make sure I fully understand this: you had @QuarkusTestResource set up without restriction on one class and you changed it to @WithTestResource with restriction on many classes? And that's when things started to get slow, right?

Yes, we used QuarkusTestResource and then simply replaced the annotation with WithTestResource but since the behavior of the annotation changed implicitly (restrictToAnnotatedClass was implicitly changed from false to true) we got several problems:

• OOMs
• slow tests

Edit: To make it fully clear, we changed in all our tests @QuarkusTestResource(OidcWiremockTestResource::class) to @WithTestResource(OidcWiremockTestResource::class).

[edeandrea]

Edit: To make it fully clear, we changed in all our tests @QuarkusTestResource(OidcWiremockTestResource::class) to @WithTestResource(OidcWiremockTestResource::class).

If you want to be 100% backwards compatible with @QuarkusTestResource(OidcWiremockTestResource::class), use @WithTestResource(value = OidcWIremockTestResource::class, restrictToAnnotatedClass = false). That will give you what you had before.

[gsmet]

Also you need only one annotation if you're doing this ^

[mschorsch]

My question has been answered and the behavior of WithTestResource has been documented in #42406.

[HerrDerb]

For the sake of completeness, what was the intention to change this behaviour?

[geoand]

The old default behavior was super confusing to anyone that was not aware of it.
The fact that an annotation added to a specific test affected the entire testsuite was very counter intuitive and was recognized very early on as a mistake that would have to be rectified at some point