Connect Directly to Kafka Brokers

[kevinswarner]

My company has provided a Kafka environment in AWS, but our DevOps team insists that we should be connecting directly to the Kafka Broker (e.g. 9092). I am able to connect with this approach using kafkacat, but cannot with node-kafka.

The following works...

kafkacat -b my-amazon-remote-url:9092 -L

But, when I run my code using kafka-node, it does not work.

const kafkaNode = require('kafka-node');

const client = new kafkaNode.Client(
 'my-amazon-remote-url:9092',
  'test-client'
);

const producer = new kafkaNode.Producer(client);

producer.on('error', (err) => {
  console.log('NEVER GETS HERE', err);
});

producer.on('ready', () => {
  console.log('NEVER GETS HERE');
});

1. Does kafka-node support connecting to the 9092 ports?
2. kafkacat CAN connect this way, so is this just something that has not been implemented in kafka-node or is it simply the wrong way to connect to Kafka? Why is my DevOps team insisting that we use the 9092 port type connection? Would there be a reason?
3. Even when I attempt to connect in the above example using the zookeeper port, I still cannot connect to the remote version.

const client = new kafkaNode.Client(
 'my-amazon-remote-url:2181',
  'test-client'
);
...

Still no luck, although the above does work on my local Kafka instance (installed on Mac OSX following standard instructions on Kafka.org.)

I appreciate any help on this issue? Thank you!

Kevin

[hyperlink]

Hi @kevinswarner Kafka is turning away from zookeeper so I've been working on a local branch with a kafka only Client. Unfortunately, it's not complete yet. I'll try to get some more time to work on it.

[vidhill]

Hello,
It looks like I might have a similar requirement, might I ask how you're getting on with your branch?

Would you need some help?

[hyperlink]

Thanks for the offer @vidhill.

I've been making good progress and pushing updates to kafka-client. It's usable and it passes the tests I've written but I haven't tested in any deployed environments yet. I would like to get more testing and feedback from others before submitting a PR.

To test

1. Point kafka-node in package.json to SOHU-Co/kafka-node#kafka-client
2. If using a ConsumerGroup change host to kafkaHost

[vidhill]

Ok, I will give that a go, it may be a few days before I get back around to this task, but I will report back when I have tried it.
Many thanks

[krystianity]

I will also test this during the week and add some feedback on friday (update: 16.06.17).

[hyperlink]

Anybody had a chance to run the new Kafka Client?

[khushbookatta]

Hi ,
I am also facing problem in connecting to client.
But I am able to connect to zookeeper port 2181
But devops team suggest to connect to kafka 9092
How to achieve it with this module.
I have tried the above solution but it didn't work

[hyperlink]

if you are instantiating your own client use KafkaClient instead of the Client. If using the GroupConsumer specify a kafkaHost instead of host. This should be covered in the README.

[khushbookatta]

Below is the code I am using through kafkaClient , Still no success , I get an error saying connection refused
import kafka from 'kafka-node';
const Producer = kafka.Producer;
const client = new kafka.KafkaClient('10.3.100.196:9092');
const producer = new Producer(client);
producer.on('ready', () => {
producer.send(payloads, (err) => {
if (err) {
console.log(err);
}
});
});
producer.on('error', (err) => {
console.log(err);
});
});

[hyperlink]

@khushbookatta in the README for KafkaClient states:

Constructor accepts an single options object (see below)

So try

new kafka.KafkaClient({kafkaHost: '10.3.100.196:9092'});

[Zala]

Can you add an example to the KafkaClient README? For me it wasn't obvious that arguments have to be passed as an object (altough now that I saw your example above, I also noticed that it says in the Notable Differences Constructor accepts an single options object (see below)). I spent way to much time on making this connection to the broker work :)

[khushbookatta]

Thanks for your help . It worked for me . It was mentioned clearly in the readMe file but still I missed it some how, in the hussel of other things. I can add an example in the README file . Thanks Khushboo Katta Get Outlook for Android<https://aka.ms/ghei36> From: Zala Herga Sent: Friday, November 10, 7:55 PM Subject: Re: [SOHU-Co/kafka-node] Connect Directly to Kafka Brokers (#666) To: SOHU-Co/kafka-node Cc: Khushboo Katta, Mention Can you add an example to the KafkaClient README? For me it wasn't obvious that arguments have to be passed as an object (altough now that I saw your example above, I also noticed that it says in the Notable Differences Constructor accepts an single options object (see below)). I spent way to much time on making this connection to the broker work :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#666 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ATIWesreKOLr8wKm1kT5hCTzrwGxgijlks5s1FzDgaJpZM4NRPhu>.

[engomar2003]

this code worked for me
Broker brokerForPartition1 = new Broker("ip",9092);
GlobalPartitionInformation partitionInfo = new GlobalPartitionInformation("Topic name");
partitionInfo.addPartition(0, brokerForPartition1);
BrokerHosts temp = new StaticHosts(partitionInfo);