JMX InstanceAlreadyExistsException when setting server.contextPath + dev tools + cloud + actuator

[davidmelia]

Hi,

I'm having some funnies with the dev tools reload and MBean registration.

Within eclipse, if you create a Spring Boot 1.4.1 Starter Project with the following dependencies: Actuator, Cloud Boostrap, Web & DevTools.

Then add server.contextPath=/webapp to application.properties.

If you trigger a dev tools reload (I just put a comment in the application.properties) then you get the following error

org.springframework.context.ApplicationContextException: Failed to register LiveBeansView MBean; nested exception is javax.management.InstanceAlreadyExistsException: DefaultDomain:application=
    at org.springframework.context.support.LiveBeansView.registerApplicationContext(LiveBeansView.java:72) ~[spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:880) ~[spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544) ~[spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
    at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:134) ~[spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
    at org.springframework.cloud.bootstrap.BootstrapApplicationListener.bootstrapServiceContext(BootstrapApplicationListener.java:138) ~[spring-cloud-context-1.1.3.RELEASE.jar:1.1.3.RELEASE]
    at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:84) ~[spring-cloud-context-1.1.3.RELEASE.jar:1.1.3.RELEASE]
    at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:62) ~[spring-cloud-context-1.1.3.RELEASE.jar:1.1.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:166) ~[spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:138) ~[spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:121) ~[spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE]
    at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:68) ~[spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
    at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54) ~[spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
    at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:337) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175) [spring-boot-1.4.1.RELEASE.jar:1.4.1.RELEASE]
    at com.example.SpringDevtoolsIssue2Application.main(SpringDevtoolsIssue2Application.java:11) [classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_101]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_101]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
    at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) [spring-boot-devtools-1.4.1.RELEASE.jar:1.4.1.RELEASE]
Caused by: javax.management.InstanceAlreadyExistsException: DefaultDomain:application=
    at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) ~[na:1.8.0_101]
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898) ~[na:1.8.0_101]
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966) ~[na:1.8.0_101]
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) ~[na:1.8.0_101]
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) ~[na:1.8.0_101]
    at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) ~[na:1.8.0_101]
    at org.springframework.context.support.LiveBeansView.registerApplicationContext(LiveBeansView.java:68) ~[spring-context-4.3.3.RELEASE.jar:4.3.3.RELEASE]
    ... 24 common frames omitted

If I remote debug LiveBeansView then within registerApplicationContext > applicationContext.getApplicationName() returns "" but unregisterApplicationContext > applicationContext.getApplicationName() returns "/webapp" i.e from server.contextPath. Therefore you get a "Failed to unregister LiveBeansView MBean" because it can't find it.

Thanks

N.B Interestingly if I try and turn off JMX (spring.jmx.enabled=false) this has no effect.

[snicoll]

Could it be something that STS does on your app? That MBean rings a bell, ping @kdvolder

[davidmelia]

Hi, Just a couple more details.

• You don't need to include actuator as you get the bug anyway.
• If you remove the spring cloud dependency the problem goes away i.e. registerApplicationContext > applicationContext.getApplicationName() returns "/webapp"
• I'm using STS 3.8.2.RELEASE

Thanks

[kdvolder]

N.B Interestingly if I try and turn off JMX (spring.jmx.enabled=false) this has no effect.

If you use 'Run As Spring Boot App' in STS, STS turns it on automatically, this could explain why you can't turn it off. The system properties that STS sets on launch probably take priority over what you are doing.

You can disable this in STS launch config editor. You can open this, for example, from Boot Dashboard: select the project or launch config in Boot Dash and click 'pencil' icon. Make sure that both Enable Live Bean Support and Enable Life Cycle Management are unchecked. If either of these is checked STS will set a few com.sun.management.jmx.remote.* sys properties to enable JMX. (Hover mouse over the checkboxes to see precisely which props STS will set to enable JMX)

Unchecking these you will loose some functionality in STS/BootDash but most of it will still be functional.

[davidmelia]

Hi @kdvolder ,

thanks for the reply. Turning those STS properties off does solve my issue however I assume this is still a bug in that, with a fresh Spring Boot App + Dev Tools + Cloud + server.contextPath then LiveBeansView sets up an MBean with applicationName="" but tries to destroy an MBean with applicationName=${server.contextPath} (in my case this is '/webapp') which does not exist?

Thanks

[kdvolder]

I assume this is still a bug

I tend to agree. Some wires seem to get crossed somewhere when using all this stuff together. This shouldn't happen. I'm just not sure how to fix it, or even who should fix it (STS vs spring-boot vs spring-cloud versus devtools).

If there's some way for STS to enable JMX differently and avoid the problem I'd be happy to fix it on STS side, but I'm not sure what would have to be done.

Or perhaps its easier for spring-boot or spring-cloud or devtools to do something a little differently. @snicoll what do you think?

[kdvolder]

Just in case this is relevant. STS sets these to enable JMX:

      -Dcom.sun.management.jmxremote
      -Dcom.sun.management.jmxremote.port=<jmxPort>
      -Dcom.sun.management.jmxremote.authenticate=false
      -Dcom.sun.management.jmxremote.ssl=false

Additionally sets this to enable spring boot admin bean:

      -Dspring.application.admin.enabled=true

And this to enable 'live bean' graph information:

      -Dspring.liveBeansView.mbeanDomain

[snicoll]

Ok so applicationContext.getApplicationName() is probably changed too late when the application starts. The MBean is registered with the default application name and we're changing it after the MBean has actually been registered. Any change to applicationName should be done before the context is actually refreshed.

LiveBeansView is registered in finsihRefresh which is quite late though.

[snicoll]

@dsyer @spencergibb does Spring Cloud change the application context's name? If so, it looks like we need to change something either here or in Spring cloud to do it earlier in the lifecycle.

[dsyer]

Spring Cloud does not change the ApplicationContext name (as far as I know, but I didn't even know it had a name). But it does introduce a parent context. Maybe that fubar?

[dsyer]

The problem is in the ordering of calls to LiveBeansView.unregisterApplicationContext(). The parent is registered first in LiveBeansView and then deregistered first as well. Something should reverse the order, otherwise you are trying to unregister one with the wrong name.

The devtools Restarter is only attempting to close one context and it has only rudimentary knowledge of the hierarchy, which is probably fine, but it was a design choice that we already made for some reason to close the parent (probably to support exactly this use case - a hierarchy created by a SpringApplicationBuilder). The LiveBeansView gets the signal too late because LiveBeansView.unregisterApplicationContext() is called before the ContextClosedEvent is fired in AbstractApplicationContext.doClose().

[cabbonizio]

I tried which @kdvolder recommended (disabling Live Bean/Lifecycle Management) and this does get me back to LiveReload support however I do get an error on startup which makes sense:

It still starts up and whenever I make a change to anything in classpath it automatically reloads successfully. I recently upgraded to latest Spring Boot version and upgraded STS to latest close to the same time, so not sure which one was the culprit. Seems like it's more related to the STS tools when running as "Spring Boot App".

[snicoll]

I've created an issue in the framework that I'll fix right the way. This should fix this issue as well. But I guess we need a separate issue to figure out why devtools stops the parent before the main application context.

[snicoll]

The issue in the framework is fixed. I am flagging this one as duplicate, pending the upgrade to Spring Framework 4.3.4 (#7213). You can also give that a try once the snapshot has been published.

[tbotalla]

For anyone that comes to this issue, in my case using Intellij IDEA the problem was that when I ran the Spring Boot application I had the checkbox "Enable JMX agent" checked.
Disabling that fixed the problem