Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RHOAIENG-2508 ODS-2600 Validate DSC creates all Serverless CRs #1259

Merged
merged 20 commits into from
Apr 4, 2024
Merged

RHOAIENG-2508 ODS-2600 Validate DSC creates all Serverless CRs #1259

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e15eb4d
RHOAIENG-2508 ODS-2600 Validate DSC creates all Serverless CRs
asanzgom Mar 6, 2024
132594f
Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…
asanzgom Mar 7, 2024
bd5184c
removed refactorings
asanzgom Mar 7, 2024
af7b4d8
change use for upstream/downstream + refactoring vars
asanzgom Mar 7, 2024
de5d3e9
Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…
asanzgom Mar 7, 2024
8c813bb
switched serverless operator check order
asanzgom Mar 7, 2024
46b2b76
removed Extract Errors From Logs Keyword
asanzgom Mar 7, 2024
8b1f734
Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…
asanzgom Mar 7, 2024
e0fc0be
Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…
asanzgom Mar 7, 2024
edafa97
changed read DSC Conditions Keyword name
asanzgom Mar 7, 2024
9c34b0d
removed unnecessary operator namespace var assignment
asanzgom Mar 7, 2024
b46accf
Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…
asanzgom Apr 1, 2024
ae0d917
Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…
asanzgom Apr 2, 2024
43f0a2d
Added ProductBug Tag + moved keywords to Resources + renamed keywords
asanzgom Apr 2, 2024
53e9465
Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…
asanzgom Apr 3, 2024
e68dd21
Refactored Check For Errors On Pod Logs Keyword to be more generic
asanzgom Apr 3, 2024
a699a90
Renamed keyword to Verify Pod Logs Do Not Contain
asanzgom Apr 3, 2024
35c9761
changed log message for Verify Pod Logs Do Not Contain keyword
asanzgom Apr 3, 2024
1c68c23
Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…
asanzgom Apr 4, 2024
4261ec6
changed keyword to wait for DSC Reconciled
asanzgom Apr 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions ods_ci/tests/Resources/OCP.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,3 +200,29 @@
[Arguments] ${crd_fullname}
Wait Until Keyword Succeeds 15 times 5s
... Oc Get kind=CustomResourceDefinition name=${crd_fullname}

Resource Should Exist
[Documentation] Check resource existson a namespace
[Arguments] ${resource} ${resource_name} ${namespace}
${rc} ${out}= Run and Return Rc And Output

Check warning

Code scanning / Robocop

Keyword name '{{ keyword_name }}' does not follow case convention Warning test

Keyword name 'Run and Return Rc And Output' does not follow case convention
... oc get ${resource} ${resource_name} -n ${namespace}
Should Be Equal "${rc}" "0" msg=${out}

Resource Status Should Be
[Documentation] Check Resource Status
[Arguments] ${oc_get} ${resource} ${expected_status}
${status}= Run
... ${oc_get}
Log ${status} console=True
Should Be Equal ${status} ${expected_status} msg=${resource} is not in Ready status

Check For Errors On Operator Logs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you modify this keyword (in this PR or a follow-up on) so it can be used on any pod and not just the operator pod?

Verify Pod Logs Does Not Contain
[Arguments]    ${pod_name}  ${label_selector}  ${all_containers}=${FALSE}  ${container_name}=${EMPTY}    ${namespace}    ${since}  ${since-time}  ${regex_pattern}

And use the oc command instead of Oc Get. If you run oc logs -h you'll see that it has most of the parameters

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok Jorge I modify it

[Documentation] Checks there are no errors on Operator Logs
[Arguments] ${operator_name} ${operator_namespace} ${regex_pattern}
${data} Run keyword Oc Get kind=Pod namespace=${operator_namespace} label_selector=name=rhods-operator
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
${pod_logs}= Oc Get Pod Logs name=${operator_name} namespace=${operator_namespace} container=rhods-operator
${match_list} Get Regexp Matches ${pod_logs} ${regex_pattern}

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '=' but got '' instead
${entry_msg} Remove Duplicates ${match_list}
Fixed Show fixed Hide fixed

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '=' but got '' instead
${length} Get Length ${entry_msg}
Fixed Show fixed Hide fixed

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '=' but got '' instead
IF ${length} != ${0} FAIL Check logs, there are some ERROR entries present.
Fixed Show fixed Hide fixed
... ELSE Log message=No ERROR logs found on Pod. level=INFO
Fixed Show fixed Hide fixed
2 changes: 1 addition & 1 deletion ods_ci/tests/Tests/100__deploy/130__operators/130__rhods_operator/130__rhods_operator.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Verify RHODS operator information
[Tags] ODS-498 ODS-624 Sanity ProductBug Tier1
Open Installed Operators Page
#Select All Projects
Wait Until Keyword Succeeds 10 times 5s Click On Searched Operator Red Hat OpenShift Data Science #robocop: disable
Wait Until Keyword Succeeds 10 times 5s Click On Searched Operator Red Hat OpenShift AI #robocop: disable
${link_elements}= Get WebElements xpath=//*[@class="co-clusterserviceversion-details__field"]//a
#Temporary List to hold the url for the verification
${temp_list} Create List
Expand Down
72 changes: 72 additions & 0 deletions .../Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
*** Settings ***
Documentation Test Cases to verify Serverless installation
Library Collections
Library SeleniumLibrary
Library OpenShiftLibrary
Resource ../../../../Resources/Page/OCPDashboard/OCPDashboard.resource
Resource ../../../../Resources/OCP.resource
Suite Setup Suite Setup
Suite Teardown Suite Teardown


*** Variables ***
${SERVERLESS_APPNAME} Red Hat OpenShift Serverless
${SERVERLESS_OPERATOR_NAME} Red Hat OpenShift Serverless
${KNATIVESERVING_NS} knative-serving
${ISTIO_NS} istio-system
${regex_pattern} ERROR
Fixed Show fixed Hide fixed

Check warning

Code scanning / Robocop

Section variable '{{ variable_name }}' name should be uppercase Warning test

Section variable '${regex_pattern}' name should be uppercase


Check warning

Code scanning / Robocop

Invalid number of empty lines between sections ({{ empty_lines }}/{{ allowed_empty_lines }}) Warning test

Invalid number of empty lines between sections (1/2)
*** Test Cases ***
Validate DSC creates all Serverless CRs

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' has too many keywords inside ({{ keyword_count }}/{{ max_allowed_count }}) Warning test

Test case 'Validate DSC creates all Serverless CRs' has too many keywords inside (12/10)
[Documentation] The purpose of this Test Case is to validate the creation
... of Serverless Custom Resources
... ProductBug: RHOAIENG-4358
[Tags] Operator ODS-2600 ProductBug
Assign Vars According To Product ${PRODUCT}
Check And Install Operator in Openshift ${SERVERLESS_APPNAME} ${SERVERLESS_OPERATOR_NAME}
Fixed Show fixed Hide fixed
asanzgom marked this conversation as resolved.
Show resolved Hide resolved
Fixed Show fixed Hide fixed
Check And Install Operator in Openshift ${OPERATOR_APPNAME} ${OPERATOR_NAME}
Fixed Show fixed Hide fixed
jgarciao marked this conversation as resolved.
Show resolved Hide resolved
Fixed Show fixed Hide fixed
Resource Should Exist KnativeServing knative-serving ${KNATIVESERVING_NS}
Resource Status Should Be oc get KnativeServing knative-serving -n ${KNATIVESERVING_NS} -o json | jq '.status.conditions[] | select(.type=="Ready") | .status' KnativeServing "True" # robocop: disable
Resource Should Exist Gateway knative-ingress-gateway ${KNATIVESERVING_NS}
Resource Should Exist Gateway knative-local-gateway ${KNATIVESERVING_NS}
Resource Should Exist Service knative-local-gateway ${ISTIO_NS}
Resource Should Exist deployment controller ${KNATIVESERVING_NS}
Wait For Pods Numbers 2 namespace=${KNATIVESERVING_NS}
... label_selector=app.kubernetes.io/component=controller timeout=120
${pod_names}= Get Pod Names ${KNATIVESERVING_NS} app.kubernetes.io/component=controller
Verify Containers Have Zero Restarts ${pod_names} ${KNATIVESERVING_NS}
${podname}= Get Pod Name ${OPERATOR_NAMESPACE} label_selector=name=rhods-operator
Check For Errors On Operator Logs ${podname} ${OPERATOR_NAMESPACE} ${regex_pattern}
Read DSC Conditions ${KNATIVESERVING_NS} default-dsc


*** Keywords ***
Suite Setup
[Documentation] Suite Setup
Set Library Search Order SeleniumLibrary
RHOSi Setup

Suite Teardown
[Documentation] Suite Teardown
Close All Browsers
RHOSi Teardown

Assign Vars According To Product
jgarciao marked this conversation as resolved.
Show resolved Hide resolved
[Documentation] Assign vars related to product
[Arguments] ${product}
IF "${product}" == "RHODS"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be ${PRODUCT}, as it is a global variable

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Jorge, I change it

Set Suite Variable ${OPERATOR_APPNAME} Red Hat OpenShift AI

Check notice

Code scanning / Robocop

{{ set_variable_keyword }} can be replaced with VAR Note test

Set Suite Variable can be replaced with VAR
Set Suite Variable ${OPERATOR_NAME} Red Hat OpenShift AI
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed

Check notice

Code scanning / Robocop

{{ set_variable_keyword }} can be replaced with VAR Note test

Set Suite Variable can be replaced with VAR
ELSE IF "${product}" == "ODH"
Set Suite Variable ${OPERATOR_APPNAME} Open Data Hub Operator

Check notice

Code scanning / Robocop

{{ set_variable_keyword }} can be replaced with VAR Note test

Set Suite Variable can be replaced with VAR
Set Suite Variable ${OPERATOR_NAME} Open Data Hub Operator
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed

Check notice

Code scanning / Robocop

{{ set_variable_keyword }} can be replaced with VAR Note test

Set Suite Variable can be replaced with VAR
END

Read DSC Conditions
[Documentation] Reads all DSC conditions
[Arguments] ${namespace} ${dsc_name}
${rc} ${out}= Run And Return Rc And Output
... oc get DataScienceCluster ${dsc_name} -n ${namespace} -o jsonpath='{.status.conditions[].reason}'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How exactly we check the condition here? 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is meant to be displayed as it is very prone to errors depending on the installation

Copy link
Member

@jstourac jstourac Mar 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the status of the components isn't stable... this is something I think that operator could also improve... the statuses are regularly changing even in case that all is fine (Ready vs Unknown). Again, I am new to the OpenShift world, so my assumption can be wrong.

But wanted to say that this doesn't check anything... it just prints some info... so if that is desired and won't be changed, please rename the keyword appropriately so it's not confusing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just realized that we are checking the condition of just one component - then... shouldn't we want it to be in ReconcileReady state? It's just one component here, it should be ready and working.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should Be Equal As Integers ${rc} ${0} are you sure this is actually checking what we want? it would check that the the oc get command has worked without error (i.e., status.conditions[].reason exists in the DSC).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed the keyword to wait for Conditions on DSC to be Reconciled successfully or timeout

Should Be Equal As Integers ${rc} ${0}
Log ${out} console=${out}
2 changes: 1 addition & 1 deletion ods_ci/tests/Tests/400__ods_dashboard/400__ods_dashboard.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -640,7 +640,7 @@ Check And Uninstall Operator In Openshift
IF ${actual_no_of_operator} == ${expected_number_operator}
Uninstall Operator ${operator_name}
ELSE
FAIL Only ${actual_no_of_operator} ${operator_name} is found in Opearatorhub
FAIL Only ${actual_no_of_operator} ${operator_name} is found in Operatorhub

END
END
Expand Down
Loading