[TRACKING] deployKF 0.1.5 / Kubeflow Pipelines V2

[thesuperzapper]

This is a tracking issue for Kubeflow 1.8 support, which will be included in the 0.2.0 release of deployKF.

The main tasks we are planning are:

1. Update the deploykf-core apps which are forks of Kubeflow components to use 1.8 versions
2. Update all kubeflow-tools to use their Kubeflow 1.8 versions (including KFP v2)
3. Update Kyverno to 1.11 (if possible, so we are not stuck on exactly 1.10.0)
4. Ensure all container images work on ARM64 clusters (including Apple Silicon)

We will initially release a 0.2.0-alpha.0 and if that goes well, we will release a full 0.2.0 version.

[samzong]

Will version 0.2 support running on MacOS? arm M1 chip

[thesuperzapper]

Will version 0.2 support running on MacOS? arm M1 chip

@samzong yes, and with native containers, not through emulation!

[thesuperzapper]

A quick update, we will probably have to stick with Kyverno 1.10.0 (at least initially, for deployKF 0.2.0). I ran into a critical bug with Kyverno 1.11.1, which prevents the "restart deployments when secrets are updated" feature from working:

• [Bug] mutate rules with targets fail to apply for trigger resources with names > 63 characters kyverno/kyverno#9134

However, while I was working on Kyverno, I found a number of places to optimize the ClusterPolicies, to reduce load on Kyverno (as it has to intercept all resource update events on the cluster).

[Hammad-NobleAI]

I'm looking into potentially using deploykf, but we really need access to the v2 api... do you have a rough estimate of when 0.2 might be released?

[thesuperzapper]

@Hammad-NobleAI sorry for the delay here, I am also very eager to get Kubeflow Pipelines 2.0 officially supported in deployKF.

My current plan is this:

1. KFP V2 will actually be in deployKF 0.1.5 (because it's fully backward compatible).
2. There are some upstream issues with KFP v2 that prevent external object stores from working properly.
3. I have made a temporary fork of Kubeflow Pipelines which will be used in deployKF 0.1.5 (which fixes these issues until they are upstream).

I hope to release deployKF 0.1.5 (with Kubeflow Pipelines 2.1.0) in the next week.

[Hammad-NobleAI]

Thanks, appreciate it :)

[tom-pavz]

@thesuperzapper Any updates on when we can expect a deployKF release that fully supports KFP 2.x? I understand there were some upstream issues with KFP itself preventing this, I am curious if the ability to specify tolerations on KFP tasks (like we could in V1 with task.add_toleration()) is included? Aka, do you expect that to be a functionality that is in the next version of deployKF? If not, any ideas when that would be supported?

And relatedly, when can we expect a deployKF release that fully supports KServe?

Basically, we are extremely interested in moving our current deployment from AWS Labs to deployKF but we need these things to be in place to do so, so just curious about timing.

[thesuperzapper]

@tom-pavz I have a PR that adds support for Kubeflow Pipelines 2.1.0:

• feat: update to Kubeflow Pipelines 2.1.0 #122

That PR should be fully backward compatible with KFP v1 pipelines. So unless I find something wrong with it, that will be in deployKF 0.1.5, along with a few other backward-compatible fixes within the next few days.

However, they just released KFP 2.2.0, so I need to review if it makes any sense to update it to that (although my current preference is to do that in a later release).

---

Regarding KServe, you can already manually deploy KServe alongside deployKF, but we are planning a more direct integration. Please follow #129, and raise any ideas you have there so we can work out the best solution.

[tom-pavz]

@thesuperzapper Gotcha, thanks for the update!

[thesuperzapper]

I am going to close this as we have now shipped deployKF 0.1.5, which has support for Kubeflow Pipelines V2.

Specifically, it includes a patched version Kubeflow Pipelines 2.1.0 that is fully backward compatible with V1 pipelines (which is why we did not make a minor version bump to 0.2.0).

We will update much more quickly to the Kubeflow 1.9.0 release when it becomes GA, as I was much more involved upstream.