Go back to polling for the kafka onramp

[mfelsche]

Pull request

Description

• Do not use the async StreamConsumer and MessageStream for the kafka onramp anymore. It is hanging if there is no message in the kafka queue, hindering the runtime from consuming control-plane and CB events. This PR uses the BaseConsumer and on each pull_event it is polling kafka for a new message.
• Make the poll_interval configurable for the kafka onramp. People might want to set it to lower values if they want to be informed faster of incoming messages when their queue has been empty before.
• Add the retry_failed_events setting (default true), that if set to false, will not seek back to the offset of the failed event it received, but keep on going. This means those failed events remain uncommitted and will only ever be retried if tremor restarts.
• Add the cb offramp that sends back CB events as requested in the event metadata or payload. E.g. {"cb": "ack"} Mostly for testing onramp/operator behaviour with CB events.
• Make the kafka onramp fail upon initialization if it couldnt connect to the cluster or couldnt subscribe to all configured topics.

Related

• Related docs PR

Checklist

• The RFC, if required, has been submitted and approved
• Any user-facing impact of the changes is reflected in docs.tremor.rs
• The code is tested
• Use of unsafe code is reasoned about in a comment
• Update CHANGELOG.md appropriately, recording any changes, bug fixes or other observable changes in behavior

[coveralls]

Coverage increased (+0.04%) to 81.541% when pulling d96a9b2 on kafka-polling into 05d0d5f on main.

[Licenser]

I love how this simplifies the kafka source! ❤️ 👍