Skip to content
This repository was archived by the owner on Nov 20, 2024. It is now read-only.

Kafka Health actuator indicator fail randomly #404

Closed
Delorien84 opened this issue Jul 4, 2018 · 1 comment
Closed

Kafka Health actuator indicator fail randomly #404

Delorien84 opened this issue Jul 4, 2018 · 1 comment
Assignees
@garyrussell
Labels
backport-2.0.x bug
Milestone
2.1.0.M2

Comments

@Delorien84
Copy link

Delorien84 commented Jul 4, 2018 •
edited
Loading

Hi,
I have simple service, that produce messages to Kafka topic using spring cloud stream. The service is added into Consul service discovery. Consul indicate, that the service is randomly failed. Consul use /actuator/health endpoint to indicate if service is in healthy state. Some call to the service return 503 with detail

HTTP GET http://172.31.135.146:9080/actuator/health: 503 Service Unavailable Output: 
{
  "status":"DOWN", 
  "binders": {
    "status":"DOWN",
    "details":{
      "kafka":{
        "status":"DOWN",
        "details":{
          "healthIndicator": {
            "status":"DOWN",
            "details": {
              "error":"java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access"
             }
           }
         }
       }
     }
   },
  ....
}

It seems that this happened in case parallel call is done on /actuator/health endpoint.

Using Spring Boot 2.0.3 and Spring Cloud Finchley.RELEASE
@garyrussell garyrussell added the bug label Jul 4, 2018
@sabbyanandan sabbyanandan added this to the 2.1.0.M2 milestone Jul 9, 2018
@garyrussell
Copy link
Contributor

Broken by #231

garyrussell added a commit to garyrussell/spring-cloud-stream-binder-kafka that referenced this issue Jul 9, 2018
@garyrussell
spring-atticGH-404: Synchronize shared consumer
9b678f0 
Fixes spring-attic#404

The fix for issue spring-attic#231
added a shared consumer but the consumer is not thread safe. Add synchronization.

Also, a timeout was added to the `KafkaBinderHealthIndicator` but not to the
`KafkaBinderMetrics` which has a similar shared consumer; add a timeout there.
@garyrussell garyrussell mentioned this issue Jul 9, 2018
Closed
@sobychacko sobychacko removed the in pr label Jul 11, 2018
sobychacko pushed a commit to sobychacko/spring-cloud-stream-binder-kafka that referenced this issue Jul 11, 2018
@garyrussell @sobychacko
spring-atticGH-404: Synchronize shared consumer
725d2a0 
Fixes spring-attic#404

The fix for issue spring-attic#231
added a shared consumer but the consumer is not thread safe. Add synchronization.

Also, a timeout was added to the `KafkaBinderHealthIndicator` but not to the
`KafkaBinderMetrics` which has a similar shared consumer; add a timeout there.
@nhomble nhomble mentioned this issue Aug 16, 2018
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@garyrussell garyrussell

Labels
backport-2.0.x bug
Milestone
2.1.0.M2
Development

Successfully merging a pull request may close this issue.

GH-404: Synchronize shared consumer garyrussell/spring-cloud-stream-binder-kafka
4 participants
@garyrussell @sobychacko @Delorien84 @sabbyanandan