Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve][TubeMQ] Improve the precision of tube consumer id #8321

Closed
1 of 2 tasks
Reo-LEI opened this issue Jun 26, 2023 · 5 comments · Fixed by #8325
Closed
1 of 2 tasks

[Improve][TubeMQ] Improve the precision of tube consumer id #8321

Reo-LEI opened this issue Jun 26, 2023 · 5 comments · Fixed by #8325

Comments

@Reo-LEI
Copy link

Reo-LEI commented Jun 26, 2023

Description

StringBuilder strBuffer = new StringBuilder(256)
.append(this.consumerConfig.getConsumerGroup())
.append("_").append(AddressUtils.getLocalAddress())
.append("-").append(pidName)
.append("-").append(System.currentTimeMillis())
.append("-").append(consumerCounter.incrementAndGet());

In the k8s environment with hostNetwork enabled, there will be multiple pods running on the same host, and the ip of these pods will be the same.

Assume these pods are taskmanager as a flink cluster and the flink cluster is exclusive to a job. At this time, taskmanagers running on the same host will most likely have the same pid. And because each taskmanager has the same parallelism, so the number of tube consumers is also the same, and the self-incrementing id will also have the same probability at this time.

When the parallelism of the tube source is relatively high, it is easy for multiple consumers to be created in parallel at the same time. At this time, the timestamp is likely to have the same value, which makes the tube consumer id duplicate. In the end, the tube server has been unable to wait for enough consumers so that the flink job cannot consume any data.

Therefore, it is hoped that the accuracy of the consumer id can be improved to avoid the above situation.

InLong Component

InLong TubeMQ

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

@github-actions
Copy link

Hello @Reo-LEI, thank you for opening your first issue in InLong 🧡 We will respond as soon as possible ⏳
If this is a bug report, please provide screenshots or error logs for us to reproduce your issue, so we can do our best to fix it.
If you have any questions in the meantime, you can also ask us on the InLong Discussions 🔍

@Reo-LEI
Copy link
Author

Reo-LEI commented Jun 26, 2023

@gosonzhang Please take a look of this issue, thx~

@gosonzhang
Copy link
Contributor

@Reo-LEI, thanks for your report

@gosonzhang
Copy link
Contributor

Maybe replacing milliseconds with nanoseconds and adding random numbers would be a better way

@gosonzhang
Copy link
Contributor

I'll push a PR to fix this, thanks @Reo-LEI

@gosonzhang gosonzhang changed the title [Improve] Improve the precision of tube consumer id [Improve][TubeMQ] Improve the precision of tube consumer id Jun 26, 2023
@healchow healchow added this to the 1.8.0 milestone Jun 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants