RHOAIENG-2508 ODS-2600 Validate DSC creates all Serverless CRs #1259

asanzgom · 2024-03-06T16:04:25Z

RHOAIENG-2508
The goal of this Test Case is to automate manual testing on the Platform scrum team

ods_ci/tests/Tests/400__ods_dashboard/400__ods_dashboard.robot

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

+
+
+*** Test Cases ***
+Validate DSC creates all Serverless CRs


...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

ods_ci/tests/Resources/Page/OCPDashboard/InstalledOperators/InstalledOperators.robot

github-actions · 2024-03-06T16:08:16Z

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %
455	0	0	455	100

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

asanzgom · 2024-03-06T17:08:04Z

Triggered rhods-ci-pr-test/2573

Executed successfully

…Serverless_CRs

ods_ci/tests/Resources/Page/OCPDashboard/InstalledOperators/InstalledOperators.robot

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

…Serverless_CRs

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

+    [Documentation]    Assign vars related to product
+    [Arguments]    ${product}
+    IF    "${product}" == "RHODS"
+        Set Suite Variable    ${OPERATOR_APPNAME}     Red Hat OpenShift AI


...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

+        Set Suite Variable    ${OPERATOR_NAME}    Red Hat OpenShift AI
+        Set Suite Variable    ${OPERATOR_NAMESPACE}    redhat-ods-operator
+    ELSE IF    "${product}" == "ODH"
+        Set Suite Variable    ${OPERATOR_APPNAME}  Open Data Hub Operator


...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

asanzgom · 2024-03-07T11:05:10Z

Triggered rhods-ci-pr-test/2576

Executed successfully

jstourac

I put a few comments, Adrian. TBH, I would welcome if somebody more experienced with the whole our TS review this too.

jstourac · 2024-03-07T12:21:28Z

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

+    [Documentation]    Checks that all conditions Reconciled Succesfully
+    [Arguments]    ${namespace}    ${dsc_name}
+    ${rc}    ${out}=    Run And Return Rc And Output
+    ...    oc get DataScienceCluster ${dsc_name} -n ${namespace} -o jsonpath='{.status.conditions[].reason}'


How exactly we check the condition here? 🤔

This is meant to be displayed as it is very prone to errors depending on the installation

Yeah, the status of the components isn't stable... this is something I think that operator could also improve... the statuses are regularly changing even in case that all is fine (Ready vs Unknown). Again, I am new to the OpenShift world, so my assumption can be wrong.

But wanted to say that this doesn't check anything... it just prints some info... so if that is desired and won't be changed, please rename the keyword appropriately so it's not confusing.

Just realized that we are checking the condition of just one component - then... shouldn't we want it to be in ReconcileReady state? It's just one component here, it should be ready and working.

Should Be Equal As Integers ${rc} ${0} are you sure this is actually checking what we want? it would check that the the oc get command has worked without error (i.e., status.conditions[].reason exists in the DSC).

changed the keyword to wait for Conditions on DSC to be Reconciled successfully or timeout

jstourac · 2024-03-07T12:22:07Z

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

+    ${pod_logs}=    Oc Get Pod Logs  name=${operator_name}  namespace=${operator_namespace}  container=rhods-operator
+    ${error_present}=    Run Keyword And Return Status    Should Contain    ${pod_logs}    ERROR
+    IF    ${error_present}
+        Log    message=Check Pod Logs, ERROR level logs found.    level=WARN


Just warning here? Errors in operator logs are just fine?

There are ERROR level logs at the start because of Reconciliation sync, then it is fine

Yes, so shouldn't we limit to check logs after some time when the operator is stable? Otherwise, do you plan to open these test results even when they are green and search for the content of this?

Also, it would be nice if operator would be smart enough to try to understand the context and reduce emitting errors at the beginning when components are starting. But I'm new to the OpenShift world, so my assumption may be just wrong.

@zdtsw This is something that maybe could be improved on the operator side
In the meanwhile, we need a test in place that is not prone to errors

which part need to be done in operator?

Not printing ERROR logs due to Reconcile if afterward everything goes fine
e.g.

2024-03-07T10:12:03Z ERROR Reconciler error {"controller": "secret-generator-controller", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"dashboard-oauth-config","namespace":"redhat-ods-applications"}, "namespace": "redhat-ods-applications", "name": "dashboard-oauth-config", "reconcileID": "65761c63-8b59-44bc-8a57-4471f6163df0", "error": "Secret \"dashboard-oauth-config-generated\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /remote-source/operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /remote-source/operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /remote-source/operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235

I see this error everytime Operator gets installed

@asanzgom regarding this test - so no plan to wait until operator is stable and check only the portion of the logs after that?

Also, looks like it just prints this warning message but doesn't print any actual error in the log... it's kind of useless, isn't it? 🤔

We need to check the Operator's Pod Logs for errors. There is one Test Case that could be refactored into a reusable Keyword for this
ods_ci/tests/Tests/100__deploy/130__operators/130__rhods_operator/135__rhods_operator_logs_verification.robot

I agree with Jan's points.
Check For Errors On Operator Logs so I would expect it failing if any error is found. I understand that ERRORS at the startup are accepted/expected, but what is the benefit we got from having this check?

Refactored to fail when ERROR reg ex is found on logs and added ProductBug Tag

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

asanzgom · 2024-03-07T12:50:05Z

I put a few comments, Adrian. TBH, I would welcome if somebody more experienced with the whole our TS review this too.

@jstourac This TS is new

…Serverless_CRs

ods_ci/tests/Resources/OCP.resource

+Resource Should Exist
+    [Documentation]    Check resource existson a namespace
+    [Arguments]       ${resource}     ${resource_name}    ${namespace}
+    ${rc}    ${out}=     Run and Return Rc And Output


ods_ci/tests/Resources/OCP.resource

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

jgarciao · 2024-04-03T08:01:31Z

ods_ci/tests/Resources/OCP.resource

+    Log    ${status}    console=True
+    Should Be Equal    ${status}    ${expected_status}   msg=${resource} is not in Ready status
+
+Check For Errors On Operator Logs


Can you modify this keyword (in this PR or a follow-up on) so it can be used on any pod and not just the operator pod?

Verify Pod Logs Does Not Contain [Arguments] ${pod_name} ${label_selector} ${all_containers}=${FALSE} ${container_name}=${EMPTY} ${namespace} ${since} ${since-time} ${regex_pattern}

And use the oc command instead of Oc Get. If you run oc logs -h you'll see that it has most of the parameters

ok Jorge I modify it

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

jgarciao · 2024-04-03T08:13:42Z

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

+Assign Vars According To Product
+    [Documentation]    Assign vars related to product
+    [Arguments]    ${product}
+    IF    "${product}" == "RHODS"


It should be ${PRODUCT}, as it is a global variable

Thanks Jorge, I change it

…Serverless_CRs

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

+*** Variables ***
+${KNATIVESERVING_NS}    knative-serving
+${ISTIO_NS}     istio-system
+${regex_pattern}       ERROR


...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

+${ISTIO_NS}     istio-system
+${regex_pattern}       ERROR
+${LABEL_SELECTOR}    name=rhods-operator
+


...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

+    [Arguments]    ${PRODUCT}
+    IF    "${PRODUCT}" == "RHODS"
+        Set Suite Variable    ${OPERATOR_APPNAME}     Red Hat OpenShift AI
+        Set Suite Variable    ${OPERATOR_NAME}    Red Hat OpenShift AI


ods_ci/tests/Resources/OCP.resource

+    [Arguments]    ${pod_name}    ${pod_namespace}    ${regex_pattern}    ${container}
+    ${pod_logs}=    Oc Get Pod Logs  name=${pod_name}  namespace=${pod_namespace}  container=${container}
+    ${match_list} 	 Get Regexp Matches   ${pod_logs}     ${regex_pattern}
+    ${entry_msg}      Remove Duplicates      ${match_list}


ods_ci/tests/Resources/OCP.resource

+    ${pod_logs}=    Oc Get Pod Logs  name=${pod_name}  namespace=${pod_namespace}  container=${container}
+    ${match_list} 	 Get Regexp Matches   ${pod_logs}     ${regex_pattern}
+    ${entry_msg}      Remove Duplicates      ${match_list}
+    ${length}         Get Length   ${entry_msg}


ods_ci/tests/Resources/OCP.resource

+    IF   ${length} != ${0}    FAIL    Pod ${pod_name} logs should not contain regexp ${regex_pattern}. Matching log entry: ${entry_msg} 
+    ...    ELSE    Log    message=Pod ${pod_name} logs does not contain regexp ${regex_pattern}    level=INFO


ods_ci/tests/Resources/OCP.resource

+    ${match_list} 	 Get Regexp Matches   ${pod_logs}     ${regex_pattern}
+    ${entry_msg}      Remove Duplicates      ${match_list}
+    ${length}         Get Length   ${entry_msg}
+    IF   ${length} != ${0}    FAIL    Pod ${pod_name} logs should not contain regexp ${regex_pattern}. Matching log entry: ${entry_msg} 


…Serverless_CRs

sonarqubecloud · 2024-04-04T10:39:17Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
2.6% Duplication on New Code

See analysis details on SonarCloud

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot

+        Set Suite Variable    ${OPERATOR_NAME}    Red Hat OpenShift AI
+    ELSE IF    "${PRODUCT}" == "ODH"
+        Set Suite Variable    ${OPERATOR_APPNAME}  Open Data Hub Operator
+        Set Suite Variable    ${OPERATOR_NAME}    Open Data Hub Operator


RHOAIENG-2508 ODS-2600 Validate DSC creates all Serverless CRs

e15eb4d

asanzgom added needs testing Needs to be tested in Jenkins new test New test(s) added (PR will be listed in release-notes) labels Mar 6, 2024

asanzgom requested review from jgarciao, zdtsw, ykaliuta, mattmahoneyrh, ajaypratap003, jstourac, VaishnaviHire and tarukumar March 6, 2024 16:04

asanzgom self-assigned this Mar 6, 2024

github-advanced-security bot found potential problems Mar 6, 2024

View reviewed changes

zdtsw reviewed Mar 6, 2024

View reviewed changes

Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…

132594f

…Serverless_CRs

jstourac reviewed Mar 7, 2024

View reviewed changes

ods_ci/tests/Resources/Page/OCPDashboard/InstalledOperators/InstalledOperators.robot Outdated Show resolved Hide resolved

jstourac reviewed Mar 7, 2024

View reviewed changes

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot Outdated Show resolved Hide resolved

removed refactorings

bd5184c

asanzgom requested a review from FedeAlonso March 7, 2024 09:39

asanzgom and others added 2 commits March 7, 2024 11:47

change use for upstream/downstream + refactoring vars

af7b4d8

Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…

de5d3e9

…Serverless_CRs

github-advanced-security bot found potential problems Mar 7, 2024

View reviewed changes

asanzgom requested review from jstourac and zdtsw March 7, 2024 11:05

jstourac reviewed Mar 7, 2024

View reviewed changes

switched serverless operator check order

8c813bb

asanzgom and others added 2 commits April 2, 2024 12:41

Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…

ae0d917

…Serverless_CRs

Added ProductBug Tag + moved keywords to Resources + renamed keywords

43f0a2d

asanzgom dismissed stale reviews from jstourac and jgarciao via 43f0a2d April 2, 2024 10:43

github-advanced-security bot found potential problems Apr 2, 2024

View reviewed changes

asanzgom requested review from bdattoma, jgarciao and jstourac April 2, 2024 13:53

mattmahoneyrh previously approved these changes Apr 2, 2024

View reviewed changes

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot Outdated Show resolved Hide resolved

...sts/Tests/100__deploy/130__operators/140__serverless_operator/140__serverless_operator.robot Outdated Show resolved Hide resolved

jgarciao requested changes Apr 3, 2024

View reviewed changes

Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…

53e9465

…Serverless_CRs

asanzgom dismissed mattmahoneyrh’s stale review via 53e9465 April 3, 2024 13:03

Refactored Check For Errors On Pod Logs Keyword to be more generic

e68dd21

github-advanced-security bot found potential problems Apr 3, 2024

View reviewed changes

Renamed keyword to Verify Pod Logs Do Not Contain

a699a90

asanzgom requested review from mattmahoneyrh and jgarciao April 3, 2024 14:12

changed log message for Verify Pod Logs Do Not Contain keyword

35c9761

github-advanced-security bot found potential problems Apr 3, 2024

View reviewed changes

jgarciao previously approved these changes Apr 4, 2024

View reviewed changes

asanzgom and others added 2 commits April 4, 2024 12:20

Merge branch 'master' into feature/ODS-2600_Validate_DSC_creates_all_…

1c68c23

…Serverless_CRs

changed keyword to wait for DSC Reconciled

4261ec6

asanzgom dismissed jgarciao’s stale review via 4261ec6 April 4, 2024 10:38

github-advanced-security bot found potential problems Apr 4, 2024

View reviewed changes

bdattoma approved these changes Apr 4, 2024

View reviewed changes

asanzgom requested a review from jgarciao April 4, 2024 11:02

jgarciao approved these changes Apr 4, 2024

View reviewed changes

asanzgom merged commit cdc6695 into red-hat-data-services:master Apr 4, 2024
10 checks passed

jstourac mentioned this pull request Apr 10, 2024

Items to backport to releases/2.8.0 branch #1370

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RHOAIENG-2508 ODS-2600 Validate DSC creates all Serverless CRs #1259

RHOAIENG-2508 ODS-2600 Validate DSC creates all Serverless CRs #1259

asanzgom commented Mar 6, 2024 •

edited

Loading

github-actions bot commented Mar 6, 2024 •

edited

Loading

asanzgom commented Mar 6, 2024 •

edited

Loading

asanzgom commented Mar 7, 2024

jstourac left a comment

jstourac Mar 7, 2024

asanzgom Mar 7, 2024

jstourac Mar 7, 2024 •

edited

Loading

asanzgom Mar 7, 2024

jstourac Mar 7, 2024

bdattoma Mar 8, 2024

asanzgom Apr 4, 2024

jstourac Mar 7, 2024

asanzgom Mar 7, 2024

jstourac Mar 7, 2024

asanzgom Mar 7, 2024

zdtsw Mar 7, 2024

asanzgom Mar 7, 2024

jstourac Mar 7, 2024

asanzgom Mar 7, 2024

bdattoma Mar 8, 2024

asanzgom Apr 2, 2024

asanzgom commented Mar 7, 2024

jgarciao Apr 3, 2024

asanzgom Apr 3, 2024

jgarciao Apr 3, 2024

asanzgom Apr 3, 2024

sonarqubecloud bot commented Apr 4, 2024

		IF ${length} != ${0} FAIL Pod ${pod_name} logs should not contain regexp ${regex_pattern}. Matching log entry: ${entry_msg}
		... ELSE Log message=Pod ${pod_name} logs does not contain regexp ${regex_pattern} level=INFO

RHOAIENG-2508 ODS-2600 Validate DSC creates all Serverless CRs #1259

RHOAIENG-2508 ODS-2600 Validate DSC creates all Serverless CRs #1259

Conversation

asanzgom commented Mar 6, 2024 • edited Loading

github-actions bot commented Mar 6, 2024 • edited Loading

Robot Results

asanzgom commented Mar 6, 2024 • edited Loading

asanzgom commented Mar 7, 2024

jstourac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jstourac Mar 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asanzgom commented Mar 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Apr 4, 2024

Quality Gate passed

asanzgom commented Mar 6, 2024 •

edited

Loading

github-actions bot commented Mar 6, 2024 •

edited

Loading

asanzgom commented Mar 6, 2024 •

edited

Loading

jstourac Mar 7, 2024 •

edited

Loading