reconnect_backoff_max_ms does not work in case Kafka cluster drops

[georgehdd]

When the Kafka cluster goes down, the KafkaClient will try to reconnect forever and will not fail after the reconnect_backoff_max_ms have elapsed.

Reproduce: Run the following code while Kafka is available:

from kafka import KafkaConsumer
consumer = KafkaConsumer("some_topic", bootstrap_servers="kafkabroker:9092")
for message in consumer:
    print("Received Message")

Then while it's running, take Kafka down.

Expected behavior: After reconnect_backoff_max_ms have elapsed, the client should fail / an exception should be thrown.

Actual behavior: for message in consumer: is stuck infinitely

[dpkp]

Apologies if the documentation is confusing. This is the expected behavior. The purpose of this maximum is to put some reasonable cap on the reconnect time so that once a cluster recovers, clients will be expected to initiate a reconnect within this maximum time. Otherwise, if we allowed exponential backoff with no maximum, a cluster could recover and a reconnect timeout may have grown to multiple days or weeks -- meaning the client would stay disconnected despite cluster recovery.

       reconnect_backoff_max_ms (int): The maximum amount of time in
            milliseconds to wait when reconnecting to a broker that has
            repeatedly failed to connect. If provided, the backoff per host
            will increase exponentially for each consecutive connection
            failure, up to this maximum. To avoid connection storms, a
            randomization factor of 0.2 will be applied to the backoff
            resulting in a random range between 20% below and 20% above
            the computed value. Default: 1000.

[georgehdd]

But why is there no notification to the application using the SDK that it's stuck in a reconnect loop?

[deepaktammali]

Is there a way to get notified or identify when the connection breaks and the client keeps infinitely reconnecting?