You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the k8s environment with hostNetwork enabled, there will be multiple pods running on the same host, and the ip of these pods will be the same.
Assume these pods are taskmanager as a flink cluster and the flink cluster is exclusive to a job. At this time, taskmanagers running on the same host will most likely have the same pid. And because each taskmanager has the same parallelism, so the number of tube consumers is also the same, and the self-incrementing id will also have the same probability at this time.
When the parallelism of the tube source is relatively high, it is easy for multiple consumers to be created in parallel at the same time. At this time, the timestamp is likely to have the same value, which makes the tube consumer id duplicate. In the end, the tube server has been unable to wait for enough consumers so that the flink job cannot consume any data.
Therefore, it is hoped that the accuracy of the consumer id can be improved to avoid the above situation.
Hello @Reo-LEI, thank you for opening your first issue in InLong 🧡 We will respond as soon as possible ⏳
If this is a bug report, please provide screenshots or error logs for us to reproduce your issue, so we can do our best to fix it.
If you have any questions in the meantime, you can also ask us on the InLong Discussions 🔍
gosonzhang
changed the title
[Improve] Improve the precision of tube consumer id
[Improve][TubeMQ] Improve the precision of tube consumer id
Jun 26, 2023
Description
inlong/inlong-tubemq/tubemq-client/src/main/java/org/apache/inlong/tubemq/client/consumer/BaseMessageConsumer.java
Lines 589 to 594 in 017e400
In the k8s environment with hostNetwork enabled, there will be multiple pods running on the same host, and the ip of these pods will be the same.
Assume these pods are taskmanager as a flink cluster and the flink cluster is exclusive to a job. At this time, taskmanagers running on the same host will most likely have the same pid. And because each taskmanager has the same parallelism, so the number of tube consumers is also the same, and the self-incrementing id will also have the same probability at this time.
When the parallelism of the tube source is relatively high, it is easy for multiple consumers to be created in parallel at the same time. At this time, the timestamp is likely to have the same value, which makes the tube consumer id duplicate. In the end, the tube server has been unable to wait for enough consumers so that the flink job cannot consume any data.
Therefore, it is hoped that the accuracy of the consumer id can be improved to avoid the above situation.
InLong Component
InLong TubeMQ
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: