KafkaConsumer.commitAsync

[felixschlegel]

Motivation:

Having KafkaConsumer.commitSync be async is not always convenient as
it suspends the KafkaConsumer.messages read loop and can therefore
lower throughput.

This PR introduces a new method KafkaConsumer.commitAsync that allows
users who don't care about the result of the commit to commit in a
"fire-and-forget" manner.

Modifications:

• new method KafkaConsumer.commitAsync
• rename KafkaConsumer.StateMachine.commitSync to
  KafkaConsumer.StateMachine.commit to serve both commitSync and
  commitAsync
• add new test for KafkaConsumer.commitAsync

[blindspotbounty]

That make sense to have async method, however it is especially a problem with fixed poll interval. If commits are often and poll interval is adaptive, that should not be a problem (unless it is a rare operation).
We bumped into that problem in particular (#128) and made polls adaptive, that helps a lot and in most cases not require async commit.

[FranzBusch]

@felixschlegel Can you elaborate the use-case for the async commit here a bit more? When would you want to use this while doing manual commit management?

[felixschlegel]

@felixschlegel Can you elaborate the use-case for the async commit here a bit more? When would you want to use this while doing manual commit management?

I don't know if this is the answer you're looking for, but generally manual commits allow the user to decide when a message is marked as processed.

The problem with commitSync is that it adds a lot of latency to the read loop as it waits for the commit to be complete before moving on.

With commitAsync you just trigger a commit without handling the result of the commit (although we could add this with a callback-based approach) to get higher throughput while still having the freedom to decide when a message is considered processed.

In the worst case, if commitAsync fails, either the offset of message i is committed by committing the offset of message i + 1 or the offset of the message is not committed at all, which could lead to another consumer reading the message again.

[FranzBusch]

Thanks for explaining @felixschlegel. One last comment

What about naming these two methods commit() async throws and triggerCommit() throws?

[FranzBusch]

@felixschlegel Can you check why the CI is failing?

[FranzBusch]

Can we align the name here with triggerCommit and commit please

[FranzBusch]

Okay sorry for the bike shedding but we probably should call this scheduleCommit which has a bit better semantics. Can you rename the public and internal methods?