[KRaft] Support for ZooKeeper-less Kafka / Kraft / KIP-500

[DanArlowski]

Please consider adding support for strimzi using kafka new Quorum system and removal of zookeeper
link

This is still not 'production ready' as stated by confluence, but starting to support it from now as beta feature might be easier for when this mode will be up and running

[scholzj]

I think this is fairly complicated. While there might be some value in supporting it for testing etc. But there are also many challenges:

• Maintaining both modes with one of them not being stable would cost a lot of time and effort which can be probably used in a better way.
• We would not be able to guarantee any API stability and backwards compatibility. So such mode would anyway not work for any long running clusters but only for shirt tests.

[scholzj]

Triaged on 28. 4. 2022: We should keep this issue to track the work on support for ZooKeeper-less Kafka / Kraft / KIP-500.

[elvis-cai]

do we plan to support upgrade from cluster strimzi managed with zookeeper added to zookeeper-less mode using strimzi kafka operator in future release?
(Background context: we are looking at migrating to strimzi, not sure if we need to wait for kraft supported in prod or not. )

[scholzj]

do we plan to support upgrade from cluster strimzi managed with zookeeper added to zookeeper-less mode using strimzi kafka operator in future release?

We definitely plan to support this. But AFAIK this is still work in progress in Kafka itself. So it remains to be seen how exactly it works, how easy it will be etc.

[eddy-geek]

X-Ref KIP-866 ZooKeeper to KRaft Migration, targetted for Kafka 3.4 (end-'22), according to KIP-833: Mark KRaft as Production Ready

[markusthoemmes]

Since there is alpha-level support for this now, what state is this in on the strimzi side? As in: When Kafka 3.3 launches (I guess that should be soon'ish) and Strimzi 0.32 releases, is KRaft mode usable or would you then still recommend against it? I'm using strimzi to only bootstrap the cluster for me, no additional features are in-use right now.

[scholzj]

I'm afraid we did not had much time to work on ZooKeeper-less Kafka over the summer 😢. I hope to find some more time for it after I finish the User Operator redesign to improve tis scalability and hopefully others will as well.

That said, I think Kafka 3.3 is still missing some important features (I'm not 100% sure if the timeline changed, but at some point it was planned that there will be no JBOD support or SCRAM-SHA support) and upgrade path from ZooKeeper. I think calling it production ready for new clusters is a bit strange. It means that we would need to maintain two parallel code paths with guaranteed upgrades etc. for possibly a long time. So, TBH, I hoped we would have much more progress at this point in time and be more prepared for ZooKeeper removal. But as a my personal opinion - I would be probably very reluctant to call anything at this stage production ready anyway.

The current prototype is really intended mainly for development. To work on things such as health checks, topic operator etc. It is not really intended for production use.

If you want to talk more about the missing parts or the plans, feel free to let me know.

[markusthoemmes]

Thanks for the insights! I guess I'll just be a little less eager and roll with Zookeeper then, for now.

[justinjoseph89]

@scholzj just curious, since kafka 3.3 is out and stated that its production ready , Is there any considerations for strimzi to support this feature (atleast for new clusters) ? I can understand from the [docs ] ,(https://strimzi.io/docs/operators/0.32.0/configuring.html#ref-operator-use-kraft-feature-gate-str) I can see that its not production ready, can we expect this soon?

[scholzj]

TBH, I do not expect we will have anything production ready until all the pieces are done in Kafka - which I think according to the current plan should be Kafka 3.5. Maintaining the different code paths in parallel would be quite complicated and would require a lot of resources. So I'm not sure we would be able to guarantee future upgrades etc. So we rather than that focus on supporting the final thing and the migration from ZooKeeper to KRaft when fully available.

[RafalSkolasinski]

As Kafka 3.5 supposedly brings support for SCRAm with KRaft - how long after its release we can realistically expect this to become available in Strimzi?

[scholzj]

SCRAM itself? I suspect that might be supported fairly quickly. Possibly in Strimzi 0.36. But the details might depend on the exact timing of the Kafka release (I guess 3.5 is expected to have first RC only late this week or next week). There will be still some important gaps -> as you see from the KIP the work is now planned un to next year and Kafka 3.7.

[Padmabishek]

Any news on when KRAFT support will be available with the operator ?

[scholzj]

Please check the release notes and the "What's new" videos which we publish for different releases for updates on the progress. Also keep in mind that KRaft is not really production ready in Kafka itself. It is still missing key features such as JBOD, migration is not yet ready, and the controller-only nodes are not really operationalized.

[scholzj]

FYI: The known issues and limitations in Strimzi should be tracked under the KRaft label. This does not necessarily include all Kafka limitations, just the things to work on in Strimzi.

[ppatierno]

The core work of the KRaft support is now done and available in the main branch.
Furthermore, the ZooKeeper to KRaft migration is supported as well.
We already have some small separated issues for other smaller things related to KRaft.
For this reason, I am going to close this one.