Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(backend): Add support for importing models stored in the Modelcar format (sidecar) #11606

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mprahl
Copy link
Contributor

@mprahl mprahl commented Feb 8, 2025

Description of your changes:

This allows dsl.import to leverage Modelcar container images in an OCI repository. This works by having an init container prepull the image and then adding a sidecar container when the launcher container is running. The Modelcar container adds a symlink to its /models directory in an emptyDir volume that is accessible by the launcher container. Once the launcher is done running the user code, it stops the Modelcar containers.

This approach has the benefit of leveraging image pull secrets configured on the Kubernetes cluster rather than require separate credentials for importing the artifact. Additionally, no data is copied to the emptyDir volume, so the storage cost is just pulling the Modelcar container image on the Kubernetes worker node.

Note that once Kubernetes supports OCI images as volume mounts for several releases, consider replacing the init container with that approach.

This also adds a new environment variable of PIPELINE_RUN_AS_USER to set the runAsUser on all pods created by Argo Workflows.

Resolves:
#11584

Checklist:

Copy link

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mprahl
Copy link
Contributor Author

mprahl commented Feb 10, 2025

The last step that I'm aware of that needs to happen is to have the launcher SIGHUP the sleep infinity rather than the Argo Exec process.

@mprahl mprahl force-pushed the import-as-oci-sidecar branch 2 times, most recently from 651abd9 to 584edf2 Compare February 10, 2025 19:56
@mprahl mprahl marked this pull request as ready for review February 10, 2025 20:18
@mprahl mprahl changed the title WIP: feat(backend): Add support for importing models stored in the Modelcar format (sidecar) feat(backend): Add support for importing models stored in the Modelcar format (sidecar) Feb 10, 2025
@google-oss-prow google-oss-prow bot requested a review from HumairAK February 10, 2025 20:18
@mprahl mprahl force-pushed the import-as-oci-sidecar branch from 584edf2 to a6f4c94 Compare February 11, 2025 14:01
@mprahl mprahl force-pushed the import-as-oci-sidecar branch 4 times, most recently from b692729 to a380cf3 Compare February 12, 2025 18:42
@mprahl mprahl force-pushed the import-as-oci-sidecar branch from a380cf3 to f94dd85 Compare February 12, 2025 18:46
@google-oss-prow google-oss-prow bot added size/XL and removed size/L labels Feb 12, 2025
@mprahl mprahl force-pushed the import-as-oci-sidecar branch 2 times, most recently from 16e8225 to 1d0fb3f Compare February 12, 2025 19:40
This allows dsl.import to leverage Modelcar container images in an OCI
repository. This works by having an init container prepull the image and
then adding a sidecar container when the launcher container is running.
The Modelcar container adds a symlink to its /models directory in an
emptyDir volume that is accessible by the launcher container. Once the
launcher is done running the user code, it stops the Modelcar
containers.

This approach has the benefit of leveraging image pull secrets
configured on the Kubernetes cluster rather than require separate
credentials for importing the artifact. Additionally, no data is copied
to the emptyDir volume, so the storage cost is just pulling the Modelcar
container image on the Kubernetes worker node.

Note that once Kubernetes supports OCI images as volume mounts for
several releases, consider replacing the init container with that
approach.

This also adds a new environment variable of PIPELINE_RUN_AS_USER to
set the runAsUser on all pods created by Argo Workflows.

Resolves:
kubeflow#11584

Signed-off-by: mprahl <mprahl@users.noreply.github.com>
@mprahl mprahl force-pushed the import-as-oci-sidecar branch from 1d0fb3f to 8a0b0ec Compare February 12, 2025 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant