Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

logsize and lag are missing from secure (SSL) kafka 10.2 #410

Open
bigdata4u opened this issue Jul 17, 2017 · 10 comments
Open

logsize and lag are missing from secure (SSL) kafka 10.2 #410

bigdata4u opened this issue Jul 17, 2017 · 10 comments

Comments

@bigdata4u
Copy link

my Kafka cluster is SSL enabled (NO SASL or ACL).

image

I am getting below error in the kafka manager log

[info] k.m.a.KafkaManagerActor - Updating internal state... java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99) at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:129) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:120) at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:99) at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:83) at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:149) at kafka.manager.actor.cluster.OffsetCache$$anonfun$19$$anonfun$21$$anonfun$22.apply(KafkaStateActor.scala:415) at kafka.manager.actor.cluster.OffsetCache$$anonfun$19$$anonfun$21$$anonfun$22.apply(KafkaStateActor.scala:412) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [error] k.m.a.c.OffsetCachePassive - [topic=test_7] An error has occurred while getting topic offsets from broker List((BrokerIdentity(1001,localhost,9092,-1,false),8), (BrokerIdentity(1001,localhost,9092,-1,false),4), (BrokerIdentity(1001,localhost,9092,-1,false),9), (BrokerIdentity(1001,localhost,9092,-1,false),5), (BrokerIdentity(1001,localhost,9092,-1,false),6), (BrokerIdentity(1001,localhost,9092,-1,false),1), (BrokerIdentity(1001,localhost,9092,-1,false),0), (BrokerIdentity(1001,localhost,9092,-1,false),2), (BrokerIdentity(1001,localhost,9092,-1,false),7), (BrokerIdentity(1001,localhost,9092,-1,false),3)) java.io.EOFException: null at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99) ~[org.apache.kafka.kafka-clients-0.10.0.1.jar:na] at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:129) ~[org.apache.kafka.kafka_2.11-0.10.0.1.jar:na] at kafka.network.BlockingChannel.receive(BlockingChannel.scala:120) ~[org.apache.kafka.kafka_2.11-0.10.0.1.jar:na] at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:99) ~[org.apache.kafka.kafka_2.11-0.10.0.1.jar:na] at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:83) ~[org.apache.kafka.kafka_2.11-0.10.0.1.jar:na] at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:149) ~[org.apache.kafka.kafka_2.11-0.10.0.1.jar:na] at kafka.manager.actor.cluster.OffsetCache$$anonfun$19$$anonfun$21$$anonfun$22.apply(KafkaStateActor.scala:415) ~[kafka-manager.kafka-manager-1.3.3.8-sans-externalized.jar:na] at kafka.manager.actor.cluster.OffsetCache$$anonfun$19$$anonfun$21$$anonfun$22.apply(KafkaStateActor.scala:412) ~[kafka-manager.kafka-manager-1.3.3.8-sans-externalized.jar:na] at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.11.8.jar:na] [info] k.m.a.c.BrokerViewCacheActor - Updating broker view...
@tanuj83
Copy link

tanuj83 commented Jul 18, 2017

I added truststore in consumer config on KM but even after that don't see any offset information and consumer information

@johnjang
Copy link

johnjang commented Jul 19, 2017 •
edited
Loading

I thought (correct me if I am wrong) KM did not support consumer information for ssl enabled clusters yet. How did you configure your configurations to see those results? Do you mind sharing me how you configured your consumer.properties with ssl enabled cluster?

@tanuj83
Copy link

tanuj83 commented Jul 21, 2017

@johnjang it is working by changing order of SSL and PLAINTEXT, more information in #310

@patelh
Copy link
Collaborator

patelh commented Aug 2, 2017

Use latest version, now you can configure security protocol to use per cluster.

@cah-jeffgraham
Copy link

cah-jeffgraham commented Aug 7, 2017 •
edited
Loading

I'd like to get KM to work with my SSL secured cluster. I'm using the latest 1.3.3.11 version and now see the new Security Protocol dropdown. I'm still not seeing KF consumer groups and even tried changing conf/consumer.properties to:

security.protocol=ssl
ssl.truststore.location=/root/cert/kafka.client.truststore.jks
ssl.truststore.password=foo

Yet still not seeing consumers. I am receiving the log error 'key not found' below so perhaps I'm missing a step?

[info] k.m.a.c.OffsetCachePassive - Adding consumers path children cache listener...
[error] k.m.a.c.package$TryLogErrorHelper - Failed to create consumer for offset topic for cluster Kafka
java.util.NoSuchElementException: key not found: PLAINTEXT
	at scala.collection.MapLike$class.default(MapLike.scala:228) ~[org.scala-lang.scala-library-2.11.8.jar:na]
	at scala.collection.AbstractMap.default(Map.scala:59) ~[org.scala-lang.scala-library-2.11.8.jar:na]
	at scala.collection.MapLike$class.apply(MapLike.scala:141) ~[org.scala-lang.scala-library-2.11.8.jar:na]
	at scala.collection.AbstractMap.apply(Map.scala:59) ~[org.scala-lang.scala-library-2.11.8.jar:na]
	at kafka.manager.actor.cluster.KafkaManagedOffsetCache$$anonfun$12.apply(KafkaStateActor.scala:195) ~[kafka-manager.kafka-manager-1.3.3.11-sans-externalized.jar:na]
	at kafka.manager.actor.cluster.KafkaManagedOffsetCache$$anonfun$12.apply(KafkaStateActor.scala:194) ~[kafka-manager.kafka-manager-1.3.3.11-sans-externalized.jar:na]
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[org.scala-lang.scala-library-2.11.8.jar:na]
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[org.scala-lang.scala-library-2.11.8.jar:na]
	at scala.collection.Iterator$class.foreach(Iterator.scala:893) ~[org.scala-lang.scala-library-2.11.8.jar:na]
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) ~[org.scala-lang.scala-library-2.11.8.jar:na]
[info] k.m.a.c.KafkaManagedOffsetCache - KafkaManagedOffsetCache shut down for cluster Kafka

@bigdata4u
Copy link
Author

bigdata4u commented Aug 7, 2017 •
edited
Loading

@patelh Nice Work, But my issue still remains the same. I reviewed your code changes, merged on Aug 1. It seems this tool always needed a PLAINTEXT port to communicate with Kafka. All its test cases are looks like that.
My problem is happening because this tool is using SimpleConsumer which makes an unsecured socket connection and in the case when Kafka is only supporting.

@patelh
Copy link
Collaborator

patelh commented Aug 8, 2017

@bigdata4u do you know offhand if the new consumer allows us to get log size ?

@johnjang
Copy link

johnjang commented Aug 10, 2017 •
edited
Loading

Would be awesome if we can use this for a ssl secure cluster (without plaintext port open) :(
Right now, using other scripts and utils to mock what KM is doing in command line. Only reason we are not using this is because lack of support on ssl secure cluster (yet).
If only I knew scala, would have tried to help or something rather than just waiting with my fingers crossed.

Edit:
"Use latest version, now you can configure security protocol to use per cluster."
I wasn't able to find any documentation related to this. Can someone point me on configuration examples of some sort? Do I just need to point to my ssl certs on consumer.properties file?

Should have read the files before asking this. Just added the certs/keys to consumer.properties. Working fine so far for me.

@sparrovv
Copy link

Hi @johnjang I might have a very similar problem to what you had. Would you mind having a look at this one: #425 . I'm wondering whether I made some silly configuration error or is it a genuine issue with kafka-manager.

@sshantveer
Copy link

@johnjang I was able to follow you and run kafka-manager with certs information provided in consumer.properties file.
Issue I am facing is, it works when the Kafka is setup to have both SSL and PLAINTEXT listeners.

  • If I remove PLAINTEXT listener from my KAFKA configuratio i.e to have only SSL listener enabled.
    Kafka manager is not able to get the consumer lag information. And error message is not helpful either

  • [ERROR] - from kafka.manager.actor.cluster.KafkaStateActor in kafka-manager-system-akka.actor.default-dispatcher-2 KafkaTopicOffsetGetter exception | WARNING arguments left: 1

There is an issue opened for this error message - https://github.com/yahoo/kafka-manager/issues/471

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@sparrovv @patelh @sshantveer @johnjang @bigdata4u @cah-jeffgraham @tanuj83