Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add stats for bulk_shard sizes/lantency/qps #50536

Closed
zhichen opened this issue Dec 31, 2019 · 5 comments
Closed

add stats for bulk_shard sizes/lantency/qps #50536

zhichen opened this issue Dec 31, 2019 · 5 comments
Labels
:Data Management/Stats Statistics tracking and retrieval APIs >enhancement Team:Data Management Meta label for data/management team

Comments

@zhichen
Copy link
Contributor

zhichen commented Dec 31, 2019

In the high-concurrency write log scenario, we found that hot_threads is always on the translog.sync operation. After a long investigation, I found that the application client often consumes several Kafka topics at the same time to form a bulkrequest and write it to Elasticsearch. Each topic corresponds to an index, resulting in a very small bulk quest to each shard, so the translog sync operation is very frequent .
This problem can be alleviated by setting translog async. In order to quickly locate the problem, I think it is necessary to add stats for bulk per shard, including bulk_shard_sizes, bulk_shard_lantency, and bulk_shard_qps.

@cbuescher cbuescher added :Data Management/Stats Statistics tracking and retrieval APIs >enhancement labels Jan 2, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Stats)

@zhichen
Copy link
Contributor Author

zhichen commented Jan 3, 2020

We have a lot of users who have this problem, so this demand is urgent for us.
If it is necessary to add stats for bulk per shard, which is a key indicator for writing, May I start working on this issue?

@dakrone
Copy link
Member

dakrone commented Jan 3, 2020

@zhichen sure if you would like to work on this, go for it. Let me know if you have any questions.

@zhichen
Copy link
Contributor Author

zhichen commented Jan 15, 2020

@dadoonet Could you help me review this pr. (52208)

probakowski pushed a commit that referenced this issue Apr 20, 2020
* Add Bulk stats track the bulk sizes per shard and the time spent on the bulk shard request (#50536)(#47345)
probakowski pushed a commit to probakowski/elasticsearch that referenced this issue Apr 20, 2020
* Add Bulk stats track the bulk sizes per shard and the time spent on the bulk shard request (elastic#50536)(elastic#47345)
@rjernst rjernst added the Team:Data Management Meta label for data/management team label May 4, 2020
@dakrone
Copy link
Member

dakrone commented May 17, 2024

This has been open for quite a while, and we haven't made much progress on this due to focus in other areas. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed.

@dakrone dakrone closed this as not planned Won't fix, can't repro, duplicate, stale May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Stats Statistics tracking and retrieval APIs >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

5 participants