Initialization blocked by multi-threaded event publishing [SPR-16357]

[spring-projects-issues]

Mark Paluch opened SPR-16357 and commented

Publishing events can block the container initialization in conjunction with event publishing from a different thread.

This can happen if events are published from a bean constructor while its bean is instantiated and events get published in a different thread while the constructed bean awaits completion of event publishing.

Consider following code:

@Component
class Foo {

	Foo(ApplicationEventPublisher publisher) throws Exception {

		Thread t = new Thread(() -> publisher.publishEvent(new Object()));
		t.start();
		t.join();
	}
}

The code above publishes an event in an other thread than the constructing thread while awaiting completion before the constructor progresses.

It leads to a thread state like:

"Thread-19@5373" daemon prio=5 tid=0x22 nid=NA waiting for monitor entry
  java.lang.Thread.State: BLOCKED
	 waiting for main@1315 to release lock on <0x17bd> (a java.util.concurrent.ConcurrentHashMap)
	  at org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:189)
	  at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:133)
	  at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:399)
	  at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:353)
	  at com.example.demo.Foo$1.run()

"main@1315" prio=5 tid=0xf nid=NA waiting (Thread calling initialization code)
  java.lang.Thread.State: WAITING
	 blocks Thread-19@5373
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.Thread.join(Thread.java:1252)
	  at java.lang.Thread.join(Thread.java:1326)
	  at com.example.demo.Foo.<init>
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
	  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	  at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)
	  at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
	  at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:271)
	  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1270)
	  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1127)
	  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
	  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
	  at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
	  at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$87.1534495070.getObject(Unknown Source:-1)
	  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
	  - locked <0x17bd> (a java.util.concurrent.ConcurrentHashMap)
	  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
	  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
	  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:758)

The example originates from reactive code in which a constructor is used to call initialization code using .block() for synchronization, see DATAMONGO-1841.

---

Affects: 5.0 GA

Issue Links:

• DATAMONGO-1841 Reactive mongodb hangs on block() when used in component initialization
• Asynchronous initialization of beans during startup [SPR-14920] #19487 Asynchronous initialization of beans during startup
• Parallel bean initialization during startup [SPR-8767] #13410 Parallel bean initialization during startup

0 votes, 8 watchers

[spring-projects-issues]

Juergen Hoeller commented

This is a tough one since it relates to the general challenges of parallel bean initialization. At this point, we generally don't support custom threads triggering (singleton) bean initialization while the main thread is waiting on them.

[spring-projects-issues]

Brian Clozel commented

Apart from the general challenge, I wanted to point out that the driver for this issue is the following: being able to call initialization code during application startup using Mono / Flux types. Maybe there is a way to solve that use case without tackling that hard issue.

[spring-projects-issues]

Sébastien Deleuze commented

I faced the issue described by Brian Clozel in my reference Kotlin + Spring Reactive application https://github.com/mixitconf/mixit, and I could not find a satisfying solution, so I would like to know if we could improve that. I need to initialize data in my MongoDB instance with ReactiveMongoTemplate and I need these data to be in the database before serving requests to users and before running my integration tests.

@Component
class DatabaseInitializer(private val userRepository: UserRepository, ...) {

    @PostConstruct
    fun init() {
        userRepository.initData()
        /...
    }
}

@Repository
class UserRepository(private val template: ReactiveMongoTemplate, ...) {

    private val logger = LoggerFactory.getLogger(this.javaClass)

    fun initData() {
        if (count().block() == 0L) {
            val usersResource = ClassPathResource("data/users.json")
            val users: List<User> = objectMapper.readValue(usersResource.inputStream)
            users.forEach { save(it).block() }
            logger.info("Users data initialization complete")
        }
    }
}

With Boot 2.0.2 it was working (by luck I think), but if I upgrade to Boot 2.0.6 or 2.1.1, it is blocking indefinitely at users.forEach level. I tried to change this to chain reactive operators and use a single block() but same issue. CommandLineRunner is not usable here since it will not ensure the data are in the database before running the application.

I quote mp911de explanation:

So what happened there is, that we emit application events Emission is synchronous and while waiting on a Mono.block() to complete, events wait being dispatched. Dispatch is blocked by a synchronized block in the application context (synchronized on the event dispatcher, it was, I think)
The synchronized block was entered by the bean initializer that is currently executing Mono.block() (that’s where the circle meets its start)

The only solution we have here is to use a blocking MongoTemplate, but I have hard time requiring injecting a second bean with a different API and infrastructure just for data initialization.

I don't know if we could find a way to avoid this deadlock or support Mono<Void> return type for this kind of Reactive use cases, but it seems to me we are missing something here. Any thoughts?

[spring-projects-issues]

Dave Syer commented

I think CommandLineRunner is a reasonable compromise - it certainly should work with integration tests, even if technically there is a window where the app is listening on port 8080 but not able to serve accurate data. If it doesn't work, then I consider that a bug as well.

[spring-projects-issues]

Brian Clozel commented

Looking at Juergen Hoeller's comments in #21025, I don't think we'll have a reactive way of publishing/processing events - for the same reason, supporting Mono<Void> is probably not an option?

[spring-projects-issues]

Sébastien Deleuze commented

Dave Syer I was mistaken by spring-projects/spring-boot#7656 where application and command line runners were initially moved after ready event, then it was rollbacked. I did some test with my MiXiT application, and I can confirm the data are created before my integration tests. That said using CommandLineRunner breaks a test so I need to dig into it more in detail to understand if that can be used as a reliable replacement or not.

Also I still think that a lot of users will be spend lot of time stuck on the @PostConstruct issue, and it seems we don't have an answer for that pretty common use case at Spring Framework level, so maybe worth to discuss for Spring 5.2?

[spring-projects-issues]

Sébastien Deleuze commented

I confirm CommandLineRunner is not a good fit as a general solution for this. In addition to being a Boot construct, in my application the web router which specifies @DependsOn("databaseInitializer") needs the data to be created in the database to generate routes at startup based on these data. It was working with @PostConstruct but it does not with CommandLineRunner so that's another indication something like #19487 is needed.

[ganeshkanyal]

@jhoeller - I am using spring 5.2.8 and still seeing this issue. Any updates on the fix?

Our project is trying to load the application cache using multiple parallel threads during @PostConstruct of the Cacheloader bean. If any of the threads has some issue while loading the cache we publish the issue as an events . A listener listens to those events and log the issue to the external system.

I am seeing deadlock while cache loader threads trying to publish the events using spring ApplicationEventpublisher.

[jhoeller]

With #25799 having been addressed years ago and #23501 revised for 6.2 now, there should not be any blocking for asynchronous event publication anymore. #25799 made it work fine for existing listeners, #23501 makes it work for listener beans that need to be initialized yet as well (unless there is a conflict among the dependencies created for that listener).