Skip to content

Commit

Permalink
Another 2.8 backport changes (#1348)
Browse files Browse the repository at this point in the history
This bundle contains following changes backported from `master` to
`releases/2.8.0`:

* 007b6af (#1326)
* 6b93683 (#1310)
* 655ac3b (#1329)
* e10f8fb (#1340)
* 01b755e (#1343)

CI: tested smoke run only with `rhods/job/rhods-smoke/5057/` against
RHOAI 2.8.1 RC2.

----

I HAVEN'T included following changes yet since these doesn't affect us
directly now (but we should probably backport them for the long-term
future):

* 389a773 (#1336)
* de721a3 (#1275)
* 24c43cd (#1334)
  • Loading branch information
jstourac authored Apr 8, 2024
2 parents 4088985 + 1085b0a commit ca8e4a8
Show file tree
Hide file tree
Showing 9 changed files with 222 additions and 44 deletions.
7 changes: 0 additions & 7 deletions ods_ci/tasks/Resources/Provisioning/Hive/OSP/create_fips.sh
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,6 @@ osp_dashboard="$(openstack catalog show keystone -c endpoints -c name -c type \

echo "Connected to Openstack: ${osp_dashboard}"

echo "Cleaning unused Floating IPs in Openstack Cloud '$OSP_CLOUD' (before creating new IPs in Network '$OSP_NETWORK')"
openstack floating ip list --status DOWN -c 'Floating IP Address' -f value | xargs -n1 -r --verbose openstack floating ip delete || rc=$?
if [[ -n "$rc" ]] ; then
echo -e "Failure [$rc] cleaning unused floating IPs"
exit ${rc:+$rc}
fi

echo "Allocating a floating IP for cluster's API"
cmd=(openstack floating ip create --description "$CLUSTER_NAME API" -f value -c floating_ip_address "$OSP_NETWORK")
echo "${cmd[@]}"
Expand Down
9 changes: 5 additions & 4 deletions ods_ci/tests/Resources/CLI/ModelServing/llm.resource
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ Compile Inference Service YAML
${mode}= Set Variable ${DSC_KSERVE_MODE}
ELSE
${mode}= Get KServe Default Deployment Mode From DSC
END
END
Log message=Using defaultDeploymentMode set in the DSC: ${mode}
END

Expand Down Expand Up @@ -388,7 +388,8 @@ Compile Deploy And Query LLM model
... namespace=${namespace}
Wait For Pods To Be Ready label_selector=serving.kserve.io/inferenceservice=${isvc_name}
... namespace=${namespace}
IF ${IS_KSERVE_RAW} Start Port-forwarding namespace=${namespace} model_name=${model_name}
${pod_name}= Get Pod Name namespace=${namespace} label_selector=serving.kserve.io/inferenceservice=${isvc_name}
IF ${IS_KSERVE_RAW} Start Port-forwarding namespace=${namespace} pod_name=${pod_name}
Query Model Multiple Times isvc_name=${isvc_name} model_name=${model_name}
... n_times=${n_queries} namespace=${namespace} query_idx=${query_idx}
... validate_response=${validate_response} protocol=${protocol}
Expand Down Expand Up @@ -755,8 +756,8 @@ Get KServe Default Deployment Mode From DSC
RETURN ${mode}

Start Port-forwarding
[Arguments] ${namespace} ${model_name} ${process_alias}=llm-query-process
${process}= Start Process oc -n ${namespace} port-forward svc/${model_name}-predictor 8033:80
[Arguments] ${namespace} ${pod_name} ${process_alias}=llm-query-process
${process}= Start Process oc -n ${namespace} port-forward pod/${pod_name} 8033:8033
... alias=${process_alias} stderr=STDOUT shell=yes
Process Should Be Running ${process}
sleep 7s
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
name: caikit
containers:
- name: kserve-container
image: quay.io/modh/text-generation-inference@sha256:9f18a63cd84b084c3cbf15534f22d5ba6916d9501298abcc03271d26ebf5cdfb
image: quay.io/modh/text-generation-inference@sha256:b87d83c65c9c5897d8a6881a160f5c65b9d7ba1d8a27bdc1ee229e60af654a9c
command: ["text-generation-launcher"]
args: ["--model-name=/mnt/models/artifacts/"]
env:
Expand All @@ -23,7 +23,7 @@ spec:
## Note: cannot add readiness/liveness probes to this container because knative will refuse them.
# multi-container probing will be available after https://github.com/knative/serving/pull/14853 is merged
- name: transformer-container
image: quay.io/modh/caikit-tgis-serving@sha256:0fd3584362e8780aed922fe124c8829c1c7df9d55590ba2ae76bb6aef0155c1f
image: quay.io/modh/caikit-tgis-serving@sha256:444bca43c99bfc4b961c926f5f10c556488613912f5e333011e98b3407d76d00
env:
- name: RUNTIME_LOCAL_MODELS_DIR
value: /mnt/models
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
name: caikit
containers:
- name: kserve-container
image: quay.io/modh/text-generation-inference@sha256:9f18a63cd84b084c3cbf15534f22d5ba6916d9501298abcc03271d26ebf5cdfb
image: quay.io/modh/text-generation-inference@sha256:b87d83c65c9c5897d8a6881a160f5c65b9d7ba1d8a27bdc1ee229e60af654a9c
command: ["text-generation-launcher"]
args: ["--model-name=/mnt/models/artifacts/"]
env:
Expand All @@ -21,7 +21,7 @@ spec:
# cpu: 8
# memory: 16Gi
- name: transformer-container
image: quay.io/modh/caikit-tgis-serving@sha256:0fd3584362e8780aed922fe124c8829c1c7df9d55590ba2ae76bb6aef0155c1f
image: quay.io/modh/caikit-tgis-serving@sha256:444bca43c99bfc4b961c926f5f10c556488613912f5e333011e98b3407d76d00
env:
- name: TRANSFORMERS_CACHE
value: /tmp/transformers_cache
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ spec:
name: pytorch
containers:
- name: kserve-container
image: quay.io/modh/text-generation-inference@sha256:e4d24fd401fd4eb89b49b4ab07e0c08389384d4a672b240e98a03ad7f9ef1c85
image: quay.io/modh/text-generation-inference@sha256:b87d83c65c9c5897d8a6881a160f5c65b9d7ba1d8a27bdc1ee229e60af654a9c
command: ["text-generation-launcher"]
args:
- "--model-name=/mnt/models/"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
*** Settings ***
Documentation Test Cases to verify Trusted CA Bundle support
Library Collections
Resource ../../../../Resources/Page/OCPDashboard/OCPDashboard.resource
Suite Setup Suite Setup
Suite Teardown Suite Teardown


*** Variables ***
${OPERATOR_NS} ${OPERATOR_NAMESPACE}
${RHOAI_OPERATOR_DEPLOYMENT_NAME} rhods-operator
${TEST_NS} test-trustedcabundle
${DSCI_NAME} default-dsci
${TRUSTED_CA_BUNDLE_CONFIGMAP} odh-trusted-ca-bundle
${CUSTOM_CA_BUNDLE} test-example-custom-ca-bundle
${IS_PRESENT} 0
${IS_NOT_PRESENT} 1


*** Test Cases ***
Validate Trusted CA Bundles State Managed
[Documentation] The purpose of this test case is to validate Trusted CA Bundles when in state Managed
... With Trusted CA Bundles Managed, ConfigMap odh-trusted-ca-bundle is expected to be created in
... each non-reserved namespace.
[Tags] Operator Smoke ODS-2638 TrustedCABundle-Managed
Wait Until Keyword Succeeds 2 min 0 sec
... Is Resource Present project ${TEST_NS} ${TEST_NS} ${IS_PRESENT}

Wait Until Keyword Succeeds 3 min 0 sec
... Is Resource Present ConfigMap ${TRUSTED_CA_BUNDLE_CONFIGMAP} ${TEST_NS} ${IS_PRESENT}

# Check that ConfigMap contains key "ca-bundle.crt"
Wait Until Keyword Succeeds 3 min 0 sec
... Check ConfigMap Contains CA Bundle Key ${TRUSTED_CA_BUNDLE_CONFIGMAP} ca-bundle.crt ${TEST_NS}

Set Custom CA Bundle Value In DSCI ${DSCI_NAME} ${CUSTOM_CA_BUNDLE} ${OPERATOR_NS}
Wait Until Keyword Succeeds 2 min 0 sec
... Is CA Bundle Value Present ${TRUSTED_CA_BUNDLE_CONFIGMAP} ${CUSTOM_CA_BUNDLE} ${TEST_NS} ${IS_PRESENT}

[Teardown] Restore DSCI Trusted CA Bundle Settings

Validate Trusted CA Bundles State Unmanaged
[Documentation] The purpose of this test case is to validate Trusted CA Bundles when in state Unmanaged
... With Trusted CA Bundles Unmanaged, ConfigMap odh-trusted-ca-bundle will not be managed by the operator.
[Tags] Operator Smoke ODS-2638 TrustedCABundle-Unmanaged
Set Trusted CA Bundle Management State ${DSCI_NAME} Unmanaged ${OPERATOR_NS}

# Trusted CA Bundle managementStatus 'Unmanaged' should NOT result in bundle being overwirtten by operator
Set Custom CA Bundle Value On ConfigMap
... ${TRUSTED_CA_BUNDLE_CONFIGMAP} random-ca-bundle-value ${TEST_NS} 5s
Wait Until Keyword Succeeds 1 min 0 sec
... Is CA Bundle Value Present ${TRUSTED_CA_BUNDLE_CONFIGMAP} random-ca-bundle-value ${TEST_NS} ${IS_PRESENT}

[Teardown] Restore DSCI Trusted CA Bundle Settings

Validate Trusted CA Bundles State Removed
[Documentation] The purpose of this test case is to validate Trusted CA Bundles when in state Removed
... With Trusted CA Bundles Removed, all odh-trusted-ca-bundle ConfigMaps will be removed.
[Tags] Operator Smoke ODS-2638 TrustedCABundle-Removed
Set Trusted CA Bundle Management State ${DSCI_NAME} Removed ${OPERATOR_NS}

# Check that odh-trusted-ca-bundle has been 'Removed'
Wait Until Keyword Succeeds 3 min 0 sec
... Is Resource Present ConfigMap ${TRUSTED_CA_BUNDLE_CONFIGMAP} ${TEST_NS} ${IS_NOT_PRESENT}

[Teardown] Restore DSCI Trusted CA Bundle Settings


*** Keywords ***
Suite Setup
[Documentation] Suite Setup
RHOSi Setup
Wait Until Operator Ready ${RHOAI_OPERATOR_DEPLOYMENT_NAME} ${OPERATOR_NS}
Wait For DSCI Ready State ${DSCI_NAME} ${OPERATOR_NS}
Create Namespace In Openshift ${TEST_NS}

Suite Teardown
[Documentation] Suite Teardown
Delete Namespace From Openshift ${TEST_NS}
RHOSi Teardown

Restore DSCI Trusted CA Bundle Settings
[Documentation] Restore DSCI Trusted CA Bundle settings to original tate
Set Custom CA Bundle Value In DSCI ${DSCI_NAME} '' ${OPERATOR_NS}
Set Trusted CA Bundle Management State ${DSCI_NAME} Managed ${OPERATOR_NS}

Wait Until Operator Ready
[Documentation] Checks if operator is available/ready
[Arguments] ${operator_name} ${namespace}
${rc} ${output}= Run And Return Rc And Output
... oc wait --timeout=2m --for condition=available -n ${namespace} deploy/${operator_name}
Should Be Equal "${rc}" "0" msg=${output}

Is Resource Present
[Documentation] Check if resource is present in namespace
[Arguments] ${resource} ${resource_name} ${namespace} ${expected_result}
${rc} ${output}= Run And Return Rc And Output
... oc get ${resource} ${resource_name} -n ${namespace}
Should Be Equal "${rc}" "${expected_result}" msg=${output}

Is CA Bundle Value Present
[Documentation] Check if the ConfigtMap contains Custom CA Bundle value
[Arguments] ${config_map} ${custom_ca_bundle_value} ${namespace} ${expected_result}
${rc} ${output}= Run And Return Rc And Output
... oc get configmap ${config_map} -n ${namespace} -o yaml | grep ${custom_ca_bundle_value}
Should Be Equal "${rc}" "${expected_result}" msg=${output}

Wait For DSCI Ready State
[Documentation] Checks that DSCI reconciled succesfully
[Arguments] ${dsci} ${namespace}
${rc} ${output}= Run And Return Rc And Output
... oc wait --timeout=3m --for jsonpath='{.status.conditions[].reason}'=ReconcileCompleted -n ${namespace} dsci ${dsci}
Should Be Equal "${rc}" "0" msg=${output}

Check ConfigMap Contains CA Bundle Key
[Documentation] Checks that ConfigMap contains CA Bundle
[Arguments] ${config_map} ${ca_bundle_name} ${namespace}
${rc} ${output}= Run And Return Rc And Output
... oc get configmap ${config_map} -n ${namespace} -o yaml | grep ${ca_bundle_name}
Should Be Equal "${rc}" "0" msg=${output}

Create Namespace In Openshift
[Documentation] Create a new namespace if it does not already exist
[Arguments] ${namespace}
${rc} ${output}= Run And Return Rc And Output oc get project ${namespace}
IF ${rc} != 0
${rc}= Run And Return Rc oc new-project ${namespace}
Should Be Equal "${rc}" "0" msg=${output}
END

Delete Namespace From Openshift
[Documentation] Delete namespace from opneshift
[Arguments] ${namespace}
${rc} ${output}= Run And Return Rc And Output oc delete project ${namespace}
Should Be Equal "${rc}" "0" msg=${output}

Set Custom CA Bundle Value In DSCI
[Documentation] Set Custom CA Bundle value in DSCI
[Arguments] ${DSCI} ${custom_ca_bundle_value} ${namespace}
${rc} ${output}= Run And Return Rc And Output
... oc patch DSCInitialization/${DSCI} -n ${namespace} -p '{"spec":{"trustedCABundle":{"customCABundle":"${custom_ca_bundle_value}"}}}' --type merge
Should Be Equal "${rc}" "0" msg=${output}

Set Custom CA Bundle Value On ConfigMap
[Documentation] Set Custom CA Bundle value in ConfigMap
[Arguments] ${config_map} ${custom_ca_bundle_value} ${namespace} ${reconsile_wait_time}
Log message=IN Here:${config_map} ${custom_ca_bundle_value} ${namespace} level=INFO
${rc} ${output}= Run And Return Rc And Output
... oc patch ConfigMap/${config_map} -n ${namespace} -p '{"data":{"odh-ca-bundle.crt":"${custom_ca_bundle_value}"}}' --type merge
Should Be Equal "${rc}" "0" msg=${output}

# Allow operator time to reconsile
Sleep ${reconsile_wait_time}

Set Trusted CA Bundle Management State
[Documentation] Change DSCI Management state to one of Managed/Unmanaged/Removed
[Arguments] ${DSCI} ${management_state} ${namespace}
${rc} ${output}= Run And Return Rc And Output
... oc patch DSCInitialization/${DSCI} -n ${namespace} -p '{"spec":{"trustedCABundle":{"managementState":"${management_state}"}}}' --type merge
Should Be Equal "${rc}" "0" msg=${output}
Loading

0 comments on commit ca8e4a8

Please sign in to comment.