Reduce Cypress Mock Test Execution Time (#3579)

* Initial WIP version of resource creation test

* Experimental changes to poll the UI for updates

* Working version if resource is present

* increase card timeout and delete active wait

* Added changes to find namespace from variables

* Final changes to read variables, cleaned up utils

* Small change to a comment

* Dummy change to trigger mocks

* Save changes on cypress-RHOAIENG-12649

* Changed file directories and names as requested on a PR comment

* Saving changes to current branch

* Additional directory/file name changes

* Additional changes to save

* Resolving timeout issue breaking mock tests, also resolved latest PR comments

* Further changes for this test

* Changes to revert the exist method appended to getCardView.

* Fixed linting

* Linting fixes

* Final comments added

* Fixed merge conflict

* Small change to page object name

* dummy commit

* Removed RHOAI bug workaround

* Removed comments

* Last comment change

* Testing changes to parallelism in Mock tests

* Fixed syntax

* Removed parallel matrix from entire file, made changes to Cypress execution command

* Created a new job for Mocks, which allows to control threads and parallelism

* Naming change

* Syntax fix

* Removed reference to cypress cloud and added headless execution

* Indentation fix

* changes to split files

* Reverting to previous version which executed tests in parallel. Cleanup required but phase 1 is complete.

* Changes to artifacts

* 1 Container to test single execution speed with a split job

* Removed redundent test, added logic to split mocks up and run them in 5 containers

* Fixed syntax issue

* Moved Code Coverage to correct position

* Makes fixes to kick tests off

* More tweaks to make tests execute

* Tweaked command to launch server which I omitted last time - d'oh

* Tweak to the execution comman

* Further tweaking the execution command

* More tweaks

* Changes to Get test files to work out pathing issue

* Pathing debugging and removed Unit tests for debugging

* Removed unit test req

* Further pathing debugging

* Further pathing debugging part 2

* Further pathing debugging part 3

* List contents of mocked dir

* Further pathing changes

* Added recursive check

* Syntax fix

* Change to Cypress execution command

* Container debug 1

* Container debug 2

* Container debug 3

* Container debug 3

* Container debug 5

* Spec file debugger 1

* Spec file debugger 2

* Spec file debugger 3

* Spec file debugger 4

* Spec file debugger 5

* Spec file debugger 6

* Spec file debugger 7

* Spec file debugger 8

* Spec file debugger 9

* Spec file debugger 10

* Spec file debugger 11

* Spec file debugger 12

* Spec file debugger 13

* Spec file debugger 14

* Spec file debugger 15

* Spec file debugger 16

* Spec file debugger 17

* Spec file debugger 18

* Removed references to CYPRESS_

* Amalgamated Gage's changes into this PR

* Indention changes

* Fixed linting issues and added another change from Gage's PR

* Started again with Gage's changes as the base

* Reverted CI command changes

* Added steps to amalgamate results of Unit and Cypress tests. Also added Codecov back.

* Changed unit tests to generate coverage results

* Removed covereage list

* Additional changes to Unit-tests

* Made a change to exclude mock and test dir from Codecov, also made a change only look for json and lcov files

* Further exclusion of Pages directory used in Page Objects

* Dummy commit to retest codecov

* Further change to exclude tests/pages and mocks from codecov

* Additional Codecov changes, merged results and uploaded

* Added backend changes and made changes to combinining

* Chaniging Cypress Mock execution command

* Chaniging Cypress coverage command to accept args

* Amending cypress:run:mock:coverage again

* Additional changes to amend coverage dir

* second fix to coverage

* Another potential fix for merging coverage reports

* Further potential fixes for combine coverage report

* Changes made to coverage report generation

* Tweaking coverage output

* Additional changes to add coverage-final.json to the necessary steps

* Change to specify coverage-final for codecov

* Hopefully final change to upload artifact to codecov

* Hopefully final change to upload artifact to codecov 2

* Removed directory upload to codecov

* Added back in verbose file viewer

* Experiment with merged coverage

* Experiment with merged coverage

* Improvements to backend test code coverage generation

* Added backend test coverage script

* Added Jest coverage also to backend

* Added Artifact Cleanup steps

* Removed redundent upload artifact step

* Added a wrapper job to report status to Github

* PR Comments

* Added script for linting

* Corrected test:lint script name

* Updated pipeline with new linter

* Changing linting

* Changing linting 2

* Changing linting 3

* Changing linting 4

* Changing linting 5

* Updated linting command name

* Missed dir nav

* Removed redundant commands

* Removed redundent test lint command from root dir

---------

Co-authored-by: Fede Alonso <fealonso@redhat.com>

.github/workflows/test.yml

@@ -1,7 +1,7 @@
 name: Test
 on: [push, pull_request]
 jobs:
-  Tests:
+  Setup:
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -12,51 +12,227 @@ jobs:
         uses: actions/setup-node@v4.1.0
         with:
           node-version: ${{ matrix.node-version }}
-      - name: Node.js modules cache, repository
+      - name: Node.js modules cache
         uses: actions/cache@v4
-        id: repo-cache
+        id: modules-cache
         with:
-          path: ${{ github.workspace }}/node_modules
-          key: ${{ runner.os }}-${{ matrix.node-version }}-modules-repo-${{ hashFiles('**/package-lock.json') }}
-          restore-keys: |
-            ${{ runner.os }}-${{ matrix.node-version }}-modules-repo
-      - name: Node.js modules cache, backend
+          path: |
+            ~/.cache/Cypress
+            ${{ github.workspace }}/node_modules
+            ${{ github.workspace }}/backend/node_modules
+            ${{ github.workspace }}/frontend/node_modules
+          key: ${{ runner.os }}-${{ matrix.node-version }}-modules-${{ hashFiles('**/package-lock.json') }}
+      - name: Install dependencies
+        if: steps.modules-cache.outputs.cache-hit != 'true'
+        run: npm install
+      - name: Check for uncomitted changes
+        run: git diff --exit-code
+
+  Lint:
+    needs: Setup
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Restore cache
         uses: actions/cache@v4
-        id: backend-cache
         with:
-          path: ${{ github.workspace }}/backend/node_modules
-          key: ${{ runner.os }}-${{ matrix.node-version }}-modules-backend-${{ hashFiles('**/backend/package-lock.json') }}
-          restore-keys: |
-            ${{ runner.os }}-${{ matrix.node-version }}-modules-backend
-      - name: Node.js modules cache, frontend
+          path: |
+            ~/.cache/Cypress
+            ${{ github.workspace }}/node_modules
+            ${{ github.workspace }}/backend/node_modules
+            ${{ github.workspace }}/frontend/node_modules
+          key: ${{ runner.os }}-18.x-modules-${{ hashFiles('**/package-lock.json') }}
+      - name: Run linting and formatting checks
+        run: npm run lint:frontend:backend
+
+  Unit-Tests:
+    needs: Setup
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Restore cache
         uses: actions/cache@v4
-        id: frontend-cache
         with:
           path: |
             ~/.cache/Cypress
+            ${{ github.workspace }}/node_modules
+            ${{ github.workspace }}/backend/node_modules
             ${{ github.workspace }}/frontend/node_modules
-          key: ${{ runner.os }}-${{ matrix.node-version }}-modules-frontend-${{ hashFiles('**/frontend/package-lock.json') }}
-          restore-keys: |
-            ${{ runner.os }}-${{ matrix.node-version }}-modules-frontend
-      - name: Install Node.js packages
-        if: ${{ steps.repo-cache.outputs.cache-hit != 'true' || steps.backend-cache.outputs.cache-hit != 'true' || steps.frontend-cache.outputs.cache-hit != 'true' }}
-        run: npm install
-      - name: Check for uncomitted changes
+          key: ${{ runner.os }}-18.x-modules-${{ hashFiles('**/package-lock.json') }}
+      - name: Run backend unit tests with coverage
+        run: |
+          npm run test:backend:unit-coverage
+          mkdir -p ./backend/coverage
+          if [ -d "./backend/jest-coverage" ]; then
+            cp -R ./backend/jest-coverage/* ./backend/coverage/
+          elif [ -d "./jest-coverage" ]; then
+            cp -R ./jest-coverage/* ./backend/coverage/
+          fi
+      - name: Cleanup backend interim artifacts
+        run: |
+          rm -rf ./backend/jest-coverage
+          rm -rf ./jest-coverage
+      - name: Run frontend unit tests with coverage
+        run: |
+          npm run test:frontend:unit-coverage
+          mkdir -p ./frontend/coverage
+          if [ -d "./frontend/jest-coverage" ]; then
+            cp -R ./frontend/jest-coverage/* ./frontend/coverage/
+          elif [ -d "./jest-coverage" ]; then
+            cp -R ./jest-coverage/* ./frontend/coverage/
+          fi
+      - name: Cleanup frontend interim artifacts
+        run: |
+          rm -rf ./frontend/jest-coverage
+          rm -rf ./jest-coverage
+      - name: Upload frontend unit test coverage
+        uses: actions/upload-artifact@v4
+        with:
+          name: unit-coverage
+          path: ./frontend/coverage
+          if-no-files-found: error
+      - name: Upload backend unit test coverage
+        uses: actions/upload-artifact@v4
+        with:
+          name: backend-coverage
+          path: ./backend/coverage
+          if-no-files-found: warn
+
+  Get-Test-Groups:
+    runs-on: ubuntu-latest
+    outputs:
+      test-groups: ${{ steps.set-groups.outputs.test-groups }}
+    steps:
+      - uses: actions/checkout@v4
+      - id: set-groups
+        shell: bash
         run: |
-          git diff --exit-code
-      - name: Test backend & test frontend with code coverage
-        if: ${{ success() }}
-        run: npm run test:backend && npm run test:frontend:coverage
-      - name: Upload coverage reports to Codecov
+          set -x  # Enable debug mode to see each command
+          
+          echo "Checking directory structure..."
+          ls -la frontend/src/__tests__/cypress/cypress/tests/ || echo "Base test directory not found"
+          
+          if [ -d "frontend/src/__tests__/cypress/cypress/tests/mocked" ]; then
+            echo "Found mocked tests directory"
+            
+            # Get directories and create JSON array - force compact output with -c
+            DIRS=$(cd frontend/src/__tests__/cypress/cypress/tests/mocked && \
+                  find . -maxdepth 1 -mindepth 1 -type d -printf "%f\n" | \
+                  jq -R . | jq -c -s .)
+            
+            # Output in the correct format for GitHub Actions
+            echo "test-groups=$DIRS" >> "$GITHUB_OUTPUT"
+            
+            echo "Generated test groups:"
+            cat "$GITHUB_OUTPUT"
+          else
+            echo "No mocked tests directory found, using default"
+            echo "test-groups=[\"default\"]" >> "$GITHUB_OUTPUT"
+          fi
+
+  Cypress-Mock-Tests:
+    needs: [Setup, Get-Test-Groups, Unit-Tests, Lint]
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        test-group: ${{ fromJson(needs.Get-Test-Groups.outputs.test-groups) }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: Restore cache
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cache/Cypress
+            ${{ github.workspace }}/node_modules
+            ${{ github.workspace }}/backend/node_modules
+            ${{ github.workspace }}/frontend/node_modules
+          key: ${{ runner.os }}-18.x-modules-${{ hashFiles('**/package-lock.json') }}
+      - name: Run Cypress Mock tests
+        run: |
+          if [ "${{ matrix.test-group }}" == "default" ]; then
+            npm run test:cypress-ci:coverage
+          else
+            npm run test:cypress-ci:coverage -- --spec "src/__tests__/cypress/cypress/tests/mocked/${{ matrix.test-group }}/**/*"
+          fi
+        working-directory: ./frontend
+      - name: Upload Cypress Mock results
+        uses: actions/upload-artifact@v4
+        with:
+          name: cypress-results-${{ matrix.test-group }}
+          path: ./frontend/src/__tests__/cypress/results/mocked
+      - name: Upload Cypress coverage
+        uses: actions/upload-artifact@v4
+        with:
+          name: cypress-coverage-${{ matrix.test-group }}
+          path: ./frontend/src/__tests__/cypress/coverage
+
+  Combine-Results-and-Upload:
+    needs: [Unit-Tests, Cypress-Mock-Tests] 
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Download all artifacts
+        uses: actions/download-artifact@v4
+        with:
+          path: all-artifacts
+      - name: Install dependencies
+        run: npm install nyc --no-save
+      - name: Combine coverage reports
+        run: |
+          mkdir -p ./coverage
+          cp all-artifacts/unit-coverage/coverage-final.json ./coverage/unit-coverage-final.json || true
+          [ -f all-artifacts/backend-coverage/coverage-final.json ] && cp all-artifacts/backend-coverage/coverage-final.json ./coverage/backend-coverage-final.json || true
+          find all-artifacts -name "cypress-coverage-*" -type d | while read dir; do
+          group_name=$(basename "$dir")
+          cp "$dir/coverage-final.json" "./coverage/${group_name}-coverage-final.json" || true
+          done
+          echo "Combined coverage files:"
+          ls -R ./coverage
+      - name: Merge coverage reports
+        run: |
+          npx nyc merge ./coverage ./coverage/merged-coverage.json
+          echo "Cleaning up merged coverage file..."
+          jq 'del(.[] | select(.statementMap == {}))' ./coverage/merged-coverage.json > ./coverage/cleaned-coverage.json
+          mv ./coverage/cleaned-coverage.json ./coverage/merged-coverage.json
+      - name: Generate coverage report
+        run: |
+          mkdir -p ./coverage/report
+          npx nyc report --reporter=html --reporter=text-summary --temp-directory ./coverage -t ./coverage --report-dir ./coverage/report
+          cp ./coverage/unit-coverage-final.json ./coverage/combined-coverage-final.json || true
+      - name: Upload combined results
+        uses: actions/upload-artifact@v4
+        with:
+          name: combined-coverage-results
+          path: ./coverage/combined-coverage-final.json
+      - name: Cleanup interim coverage files
+        run: |
+          rm -rf ./coverage/unit-coverage-final.json
+          rm -rf ./coverage/backend-coverage-final.json
+          find ./coverage -name "*-coverage-final.json" -type f -delete 
+      - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v4.6.0
         with:
           token: ${{ secrets.CODECOV_TOKEN }}
           fail_ci_if_error: false
-          name: frontend
-          directory: ./frontend/coverage
+          name: frontend-and-backend
+          file: ./coverage/merged-coverage.json 
+          disable_search: true
+          directory: ./coverage
           verbose: true
-      - uses: actions/upload-artifact@v4
-        if: ${{ always() }}
-        with:
-          name: cypress-results
-          path: ./frontend/src/__tests__/cypress/results/
+
+  Tests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        node-version: [18.x]
+    needs:
+      - Setup
+      - Lint
+      - Unit-Tests
+      - Get-Test-Groups
+      - Cypress-Mock-Tests
+      - Combine-Results-and-Upload
+    steps:
+      - name: Verify all jobs succeeded
+        run: echo "All required jobs have successfully completed for Node.js ${{ matrix.node-version }}."

backend/package.json

@@ -26,6 +26,8 @@
     "build": "run-s build:clean tsc:prod",
     "build:clean": "rimraf ./dist",
     "test": "run-s test:lint test:type-check test:jest",
+    "test:unit-coverage": "run-s test:lint test:type-check test:jest:coverage",
+    "test:jest:coverage": "rimraf jest-coverage && jest --silent --coverage --coverageReporters=json --coverageReporters=lcov",
     "test:lint": "eslint --max-warnings 0 --ext .json,.js,.ts ./src",
     "test:fix": "eslint --ext .json,.js,.ts ./src --fix",
     "test:type-check": "tsc --noEmit",

frontend/package.json

@@ -35,14 +35,15 @@
     "test:jest": "jest",
     "test:jest:coverage": "rimraf jest-coverage && jest --silent --coverage --coverageReporters=json --coverageReporters=lcov",
     "test:unit": "npm run test:jest -- --silent",
+    "test:unit-coverage": "run-s test:lint test:type-check test:jest:coverage",
     "test:cypress-ci": "npx concurrently -P -k -s first \"npm run cypress:server:build && npm run cypress:server\" \"npx wait-on tcp:127.0.0.1:9001 && npm run cypress:run:mock -- {@}\" -- ",
-    "test:cypress-ci:coverage": "npx concurrently -P -k -s first \"npm run cypress:server:build:coverage && npm run cypress:server\" \"npx wait-on tcp:127.0.0.1:9001 && npm run cypress:run:mock:coverage\"",
+    "test:cypress-ci:coverage": "npx concurrently -P -k -s first \"npm run cypress:server:build:coverage && npm run cypress:server\" \"npx wait-on tcp:127.0.0.1:9001 && npm run cypress:run:mock:coverage -- {@}\" -- ",
     "coverage:merge": "rimraf coverage && istanbul-merge --out coverage/coverage-final.json jest-coverage/coverage-final.json src/__tests__/cypress/coverage/coverage-final.json && nyc report --reporter html -t coverage --report-dir coverage/report && nyc report --reporter json-summary -t coverage --report-dir coverage",
     "cypress:open": "cypress open --project src/__tests__/cypress",
     "cypress:run": "cypress run -b chrome --project src/__tests__/cypress",
     "cypress:open:mock": "CY_MOCK=1 CY_WS_PORT=9002 npm run cypress:open -- ",
     "cypress:run:mock": "CY_MOCK=1 CY_WS_PORT=9002 npm run cypress:run -- ",
-    "cypress:run:mock:coverage": "rimraf src/__tests__/cypress/coverage src/__tests__/cypress/.nyc_output && CY_COVERAGE=true npm run cypress:run:mock",
+    "cypress:run:mock:coverage": "rimraf src/__tests__/cypress/coverage src/__tests__/cypress/.nyc_output && CY_COVERAGE=true npm run cypress:run:mock -- {@}",
     "cypress:open:record": "CY_RECORD=1 npm run cypress:open",
     "cypress:run:record": "CY_RECORD=1 npm run cypress:run && npm run cypress:format",
     "cypress:server:build": "ODH_DIST_DIR=./public-cypress POLL_INTERVAL=9999999 FAST_POLL_INTERVAL=9999999 WS_HOSTNAME=localhost:9002 npm run build",

package.json

@@ -40,8 +40,12 @@
     "test:backend": "cd ./backend && npm run test",
     "test:frontend": "cd ./frontend && npm run test",
     "test:frontend:coverage": "cd ./frontend && npm run test:coverage",
+    "test:frontend:unit-coverage": "cd ./frontend && npm run test:unit-coverage",
+    "test:backend:unit-coverage": "cd ./backend && npm run test:unit-coverage",
+
     "test:cypress-ci": "cd ./frontend && npm run test:cypress-ci -- ",
     "test:unit": "cd ./frontend && npm run test:unit",
+    "lint:frontend:backend": "cd ./backend && npm run test:lint && cd ../frontend && npm run test:lint",
     "test:fix": "run-s test:fix-backend test:fix-frontend",
     "test:fix-backend": "cd ./backend && npm run test:fix",
     "test:fix-frontend": "cd ./frontend && npm run test:fix"