Add two consumer benchmark #149
Conversation
259ff65 to
98f17fa
Compare
|
Previous CI failed with timeout but there is not much information, just the following: I've disabled all but one test to check that one works correctly and reduced pollInterval from default (100) to 10ms |
|
@FranzBusch, @felixschlegel just a kind reminder. |
cc54b75 to
34e7b4d
Compare
|
Just to note: I could not adjust thresholds and absolute values that they could adequately fit CI results. Seems that depending on machine hardware or its load, results difference may be up to several times faster or slower. |
|
I think CI should only check things like mallocs or syscalls which are deterministic unless we have a dedicated CI runner. Then for local tests other metrics can be used. |
|
Hi @hassila Unfortunately, mallocs also depend on CPU as well in a tricky way. If there is more powerful machine (or less loaded one), it leads to less queues in librdkafka/AsyncStream thus less mallocs and memory consumption (as e.g. librdkafka seems re-use less messages). While if machine is less powerful (or maybe loaded with other tasks) it leads to higher number of elements in queue. It is still not a lot of elements but relatively it differs by tens of percents. Unfortunately, after several days (in background) trying to tune benchmark results, librdkafka and swift kafka gave no luck. Let me know if you have any particular ideas to try adjust test in CI, I am happy to try them. |
* Feature: expose librdkafka statistics as swift metrics (swift-server#92) * introduce statistics for producer * add statistics to new consumer with events * fix some artefacts * adjust to KeyRefreshAttempts * draft: statistics with metrics * make structures internal * Update Sources/Kafka/Configuration/KafkaConfiguration+Metrics.swift Co-authored-by: Felix Schlegel <fefefe152@gmail.com> * Update Sources/Kafka/Configuration/KafkaConsumerConfiguration.swift Co-authored-by: Felix Schlegel <fefefe152@gmail.com> * Update Sources/Kafka/Configuration/KafkaConfiguration+Metrics.swift Co-authored-by: Felix Schlegel <fefefe152@gmail.com> * Update Sources/Kafka/Configuration/KafkaConfiguration+Metrics.swift Co-authored-by: Felix Schlegel <fefefe152@gmail.com> * address review comments * formatting * map gauges in one place * move json mode as rd kafka statistics, misc renaming + docc * address review comments * remove import Metrics * divide producer/consumer configuration * apply swiftformat * fix code after conflicts * fix formatting --------- Co-authored-by: Felix Schlegel <fefefe152@gmail.com> * Add benchmark infratructure without actual tests (swift-server#146) * add benchmark infratructure without actual test * apply swiftformat * fix header in sh file * use new async seq methods * Update to latest librdkafka & add a define for RAND_priv_bytes (swift-server#148) Co-authored-by: Franz Busch <f.busch@apple.com> * exit from consumer batch loop when no more messages left (swift-server#153) * Lower requirements for consumer state machine (swift-server#154) * lower requirements for kafka consumer * add twin test for kafka producer * defer source.finish (swift-server#157) * Add two consumer benchmark (swift-server#149) * benchmark for consumer * attempty to speedup benchmarks * check CI works for one test * enable one more test * try to lower poll interval * adjust max duration of test * remain only manual commit test * check if commit is the reason for test delays * try all with schedule commit * revert max test time to 5 seconds * dockerfiles * test set threasholds * create dummy thresholds from ci results * disable benchmark in CI * add header * add stable metrics * update thresholds to stable metrics only * try use '1' instead of 'true' * adjust thresholds to CI results (as temporary measure) * set 20% threshold.. * move arc to unstable metrics * try use 'true' in quotes for CI * try reduce number of messages for more reliable results * try upgrade bench * disable benchmark in CI * Update librdkafka for BoringSSL (swift-server#162) * chore(patch): [sc-8379] use returned error (swift-server#163) * [producer message] Allow optional key for initializer (swift-server#164) Co-authored-by: Harish Yerra <hyerra@apple.com> * Allow groupID to be specified when assigning partition (swift-server#161) * Allow groupID to be specified when assigning partition Motivation: A Consumer Group can provide a lot of benefits even if the dynamic loadbalancing features are not used. Modifications: Allow for an optional GroupID when creating a partition consumer. Result: Consumer Groups can now be used when manual assignment is used. * fix format --------- Co-authored-by: Ómar Kjartan Yasin <omarkj@apple.com> Co-authored-by: blindspotbounty <127803250+blindspotbounty@users.noreply.github.com> Co-authored-by: Franz Busch <f.busch@apple.com> * Wrap rd_kafka_consumer_poll into iterator (use librdkafka embedded backpressure) (swift-server#158) * remove message sequence * test consumer with implicit rebalance * misc + format * remove artefact * don't check a lot of messages * fix typo * slow down first consumer to lower message to fit CI timeout * remove helpers * use exact benchmark version to avoid missing thresholds error (as no thresholds so far) * add deprecated marks for backpressure, change comment for future dev * address comments --------- Co-authored-by: Felix Schlegel <fefefe152@gmail.com> Co-authored-by: Axel Andersson <axel@ordo.one> Co-authored-by: Franz Busch <f.busch@apple.com> Co-authored-by: Samuel M <samuel.mn77@yahoo.com> Co-authored-by: Harish Yerra <hyerra@gmail.com> Co-authored-by: Harish Yerra <hyerra@apple.com> Co-authored-by: Omar Yasin <omarkj@gmail.com> Co-authored-by: Ómar Kjartan Yasin <omarkj@apple.com>
This PR contains:
@_spiannotation.finishedstate andwaitForNewMessages()callThe following results are for this baseline (1000 messages):