Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MemberToMetricMappings is not cleaned on app redeploy/shutdown #420

Closed
spyrkob opened this issue Mar 23, 2021 · 5 comments
Closed

MemberToMetricMappings is not cleaned on app redeploy/shutdown #420

spyrkob opened this issue Mar 23, 2021 · 5 comments
Milestone

Comments

@spyrkob
Copy link
Contributor

spyrkob commented Mar 23, 2021

Renaming a metric name (eg @counted(name="mymetric2") to @counted(name="mymetric3")) and redeploying the application results in:

13:28:24,595 ERROR [io.undertow.request] (default task-1) UT005023: Exception handling request to /test/rest: org.jboss.resteasy.spi.UnhandledException: java.lang.IllegalStateException: SRMET00002: No metric of type counter and ID MetricID{name='org.redhat.set.metrics.TestBean.mymetric2', tags=[]} found in registry
	at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:82)
	at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:346)
	at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:193)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:457)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229)
	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135)
	at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:358)

This seems to be introduced in 3.0.0 - the MetricRegistries#cleanUp method is no longer called on application shutdown, which results in stale mappings in MemberToMetricMappings.

@spyrkob
Copy link
Contributor Author

spyrkob commented Mar 23, 2021

I was not able to test this with the main branch, so apologies if this is already fixed.

I opened a possible fix (building on #390) - #421. I think a simpler fix could be removing the Application registry like MetricRegistries#cleanUp used to do it 2.4.x, but I don't know if that's desired.

@jmartisk
Copy link
Member

I was not able to test this with the main branch

Let's not worry too much about main now, it's switching to Micrometer and WildFly won't be consuming that for quite some time (until it bubbles up to the MicroProfile spec) and it will use a different approach for registries (we will need to think it though too, but that's a different story).

removing the Application registry like MetricRegistries#cleanUp used to do it 2.4.x

To be honest I don't really know how exactly this is made to work in WildFly - AFAIK, there is only one application registry per WildFly instance, not per deployment, so dropping the application registry would drop metrics for all deployments?! So is (was) MetricRegistries#cleanUp really being used during undeploy? (On the SmallRye side, I don't recall any change related to this method between 2.4 and 3.0, it could have been a change on the WF side, but I don't have enough insight into the metrics subsystem to understand it ;) )

Anyway, your fix looks good to me. It removes a deployment's mappings along with its metrics without touching stuff from other deployments. If you verified that it fixes the WF problem, I'm OK with merging it
I suppose you'll also want a new 3.0.x release with it, right?

@spyrkob
Copy link
Contributor Author

spyrkob commented Mar 25, 2021

Thanks @jmartisk!

AFAIK, there is only one application registry per WildFly instance, not per deployment, so dropping the application registry would drop metrics for all deployments?!

I just tested this with two apps and that seems to be the case :)

On the SmallRye side, I don't recall any change related to this method between 2.4 and 3.0, it could have been a change on the WF side

AFAIU this is caused by changes in MetricCdiInjectionExtension#registerMetrics - in 2.4 obtaining the Application registry required CDI creating MetricRegistries bean, that was later disposed off during undeploy calling cleanUp method.

I suppose you'll also want a new 3.0.x release with it, right?

Yes I think so. If there is a release scheduled in future it can probably wait for it though.

@jmartisk
Copy link
Member

I merged your fix, thanks.
I'll do a 3.0.3 release today - there's no release schedule, I just release when something important gets in :)

@jmartisk jmartisk added this to the 3.0.3 milestone Mar 26, 2021
@jmartisk
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants