Distributed Workloads v0.1.0 Release

[anishasthana]

This updates the Distributed Workloads manifests (CodeFlare and Ray) to the latest as part of the v0.1.0 release. Release notes can be found at https://github.com/opendatahub-io/distributed-workloads/releases/tag/v0.1.0.

cc @KPostOffice @Maxusmusti @MichaelClifford @tedhtchang @jbusche @LaVLaS

[LaVLaS]

Execution of the CI tests is most likely blocked due to an outage in openshift-ci.

[anishasthana]

/retest

[anishasthana]

/retest

[jbusche]

Testing looked good - I tried it utilizing this kfdef applied to the 0.8.0 ODH operator:

apiVersion: kfdef.apps.kubeflow.org/v1
kind: KfDef
metadata:
  name: codeflare-stack
  namespace: opendatahub
spec:
  applications:
  # CodeFlare
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: codeflare-stack
    name: codeflare-stack
  # KubeRay
  - kustomizeConfig:
      repoRef:
        name: manifests
        path: ray/operator
    name: ray-operator
  repos:
    # ODH Core component manifests
  - name: manifests
    #uri: https://github.com/opendatahub-io/odh-manifests/tarball/master
    uri: https://github.com/anishasthana/odh-manifests/tarball/dw_0.1.1

[jbusche]

LGTM

[tedhtchang]

I didn't actually run a e2e test. The change looks good to me. One question, do we need to specify the new kuberay version somewhere?

      parameters:
      - name: odh-kuberay-operator-controller-image
         value: docker.io/kuberay/operator:v0.6.0

in
kfdef/codeflare-stack-kfdef.yaml
tests/resources/codeflare-stack/codeflare-stack-kfdef.yaml
kfdef/ray-minimal-kfdef.yaml

[anishasthana]

/rebase

[dimakis]

LGTM, tested with a batch job and checked all versions. Everything working as expected.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dimakis, jbusche
Once this PR has been reviewed and has the lgtm label, please ask for approval from anishasthana. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[anishasthana]

/retest

[openshift-ci]

New changes are detected. LGTM label has been removed.

[anishasthana]

/retest

[LaVLaS]

@anishasthana Can you verify the test install script to ensure that the codeflare-operator install is inline with the distributed workloads feature set in this PR?

[LaVLaS]

@anishasthana Attached is the pod log from the failed jupyter notebook run during your test
odh-manifests-PR910-jupyter-nb-kube-3aadmin-0-jupyter-nb-kube-3aadmin.log

[anishasthana]

/retest

[anishasthana]

/retest

[anishasthana]

/retest

[anishasthana]

The reason for this is that the ray dashboard takes a few seconds to become accessible, so there is a gap between wait_ready and the job submission being usable. The short term solution is to simply add a small sleep before submitting the job, and the long-term solution (for next codeflare release) will likely involve waiting longer in wait_ready for the dashboard to be accessible, or adding better messaging in this edge case

We will create a follow-on issue for the wait_ready condition. Related, the notebook pods end up restarting almost immediately and then ending up in a crashloop at a different (earlier) point in the notebook. This makes it much harder to debug as the "useful" logs only showed up in the first instance of the crashloop -- every other instance fails due to an unrelated (but not critical) issue.

Thanks @MichaelClifford @Maxusmusti

[anishasthana]

/retest

[openshift-ci]

@anishasthana: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/odh-manifests-e2e	4fb8b89	link	true	/test odh-manifests-e2e
ci/prow/411-odh-manifests-e2e	4fb8b89	link	true	/test 411-odh-manifests-e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[anishasthana]

/retest

[LaVLaS]

Merging this since it appears to have resolved the Distributed Workloads test. The modelmesh test is failing and will be fixed in #918 based on an offline conversation by @VedantMahabaleshwarkar