[feature] Configurable KFP Artifact types while storing to Minio

[rado-parrak]

Feature Area

/area sdk

What feature would you like to see?

We are trying to marry "KF Pipelines + KF Artifacts + KF Serving" in a nice use-case where we train a (sklearn) model using KFP, store it as an output artifact to the artifact store of pipelines (minio) and then deploy that model directly from the artifact store using KFServing.

What is the use case or pain point?

The problem we are facing is that the output artifacts are by default stored as .tgz , so even a .joblib blob model object is still wrapped into .tgz at creation time of the artifact in minio. On the other end, the default KF serving application server expects a .joblib file, not a .tgz.

There are at least two ways to tackle this issue:
(1) On the KFP side, by adding some kind of flag that would allow to dump some artifacts to minio uncompressed (i.e. directly as .joblib in our case) - something we discussed with @eterna2 already in the Kubeflow Slack, or
(2) On the KFServing side, by adding a functionality to the storage -initiazer such that it can unpack the .tgz during the inference service creation time. This suggestion was made in this ticket.

---

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[eterna2]

Not that familiar with kfserving, but with respect to #5453.

Can I assume there are 2 separate issues?

• for model retrieval, kfserving only support local model or http/https model ([Discuss] Should KFServing Component use InputPath? #5453)
• for remote model artifacts, it expects an unarchived model binary? (This ticket)

[rado-parrak]

Yes indeed, these are two separate issues, stemming from 2 separate use cases. Thx.

[juangon]

Love this idea. I am new to Kubeflow and was implementing a "normal" pipeline but looks like model serving from another pipeline step that trains model isn't well supported, and found this issue.

[eterna2]

#1640 probably can solve this issue w/o changing the sdk.

[Bobgy]

FYI, v2 compatible mode does not compress artifacts by default already.
https://www.kubeflow.org/docs/components/pipelines/sdk/v2/

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[yangyang919]

Hi @rado-parrak , may I ask eventually how your team solve this case? We meet the exact same case, and are not quite sure which way to proceed...thanks!

[rado-parrak]

Hi @yangyang919, we actually did not solve it at the time. I left the project a couple of months ago, so I do not know if there was a resolution later. Sorry for being a dead end.

[yangyang919]

FYI, v2 compatible mode does not compress artifacts by default already. https://www.kubeflow.org/docs/components/pipelines/sdk/v2/

Hi @Bobgy , could you point me where I can find this setting? Coz I found the model still gets compressed after pipeline running

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.