-
Notifications
You must be signed in to change notification settings - Fork 25k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add stats for bulk_shard sizes/lantency/qps #50536
Comments
Pinging @elastic/es-core-features (:Core/Features/Stats) |
We have a lot of users who have this problem, so this demand is urgent for us. |
@zhichen sure if you would like to work on this, go for it. Let me know if you have any questions. |
* Add Bulk stats track the bulk sizes per shard and the time spent on the bulk shard request (elastic#50536)(elastic#47345)
This has been open for quite a while, and we haven't made much progress on this due to focus in other areas. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed. |
In the high-concurrency write log scenario, we found that hot_threads is always on the translog.sync operation. After a long investigation, I found that the application client often consumes several Kafka topics at the same time to form a bulkrequest and write it to Elasticsearch. Each topic corresponds to an index, resulting in a very small bulk quest to each shard, so the translog sync operation is very frequent .
This problem can be alleviated by setting translog async. In order to quickly locate the problem, I think it is necessary to add stats for bulk per shard, including bulk_shard_sizes, bulk_shard_lantency, and bulk_shard_qps.
The text was updated successfully, but these errors were encountered: