re-enable e2e UI tests on CI #1961
Conversation
9e80c76 to
3b94acb
Compare
There was a problem hiding this comment.
component is not referenced in either repo
| runs-on: ubuntu-latest | ||
| name: "End to end tests - Grafana: ${{ matrix.grafana-image-tag }}" | ||
| strategy: | ||
| matrix: | ||
| grafana-image-tag: | ||
| - 8.5.22 |
There was a problem hiding this comment.
OnCall does not work on this version and we may want to consider updating the dependencies object in the plugin.json to reflect this:
| @@ -620,8 +620,7 @@ | |||
| } | |||
| ], | |||
| "dependencies": { | |||
| "grafanaDependency": ">=8.3.2", | |||
| "grafanaVersion": "8.3", | |||
There was a problem hiding this comment.
dependencies.grafanaVersion is deprecated according to the docs
- run on several more grafana versions - update grafanaDependency version in plugin.json
4dac1fc to
cb82a75
Compare
| @@ -14,7 +14,7 @@ const config: PlaywrightTestConfig = { | |||
| testDir: './integration-tests', | |||
| globalSetup: './integration-tests/globalSetup.ts', | |||
| /* Maximum time one test can run for. */ | |||
| timeout: 90 * 1000, | |||
| timeout: 60 * 1000, | |||
There was a problem hiding this comment.
60s should be plenty of time for most tests, if we put this timeout too long, it'll take forever for the CI job to fail (especially with retries enabled).
There is currently one test, "we can create an oncall schedule + receive an alert", that genuinely takes longer (because it creates a ton of different objects). This test has already been marked w/ test.slow() which increases the timeout value by 3x just for this test.
| # - 8.5.22 | ||
| # - 9.0.0 | ||
| # - 9.1.0 |
…na/oncall into jorlando/turn-e2e-tests-back-on-ci
…na/oncall into jorlando/turn-e2e-tests-back-on-ci
| @@ -408,8 +423,7 @@ jobs: | |||
| GRAFANA_PASSWORD: oncall | |||
| MAILSLURP_API_KEY: ${{ secrets.MAILSLURP_API_KEY }} | |||
| working-directory: ./grafana-plugin | |||
| # -x = exit command after first failing test | |||
| run: yarn test:integration -x | |||
There was a problem hiding this comment.
don't exit on first failed test as this defeats the purpose of allowing the retries to get around tests which are currently flaky
…na/oncall into jorlando/turn-e2e-tests-back-on-ci
#1692 is still open. This PR is not an ideal approach, but it's a quick win while we wait for that issue to be resolved. By retrying failing tests up to 3 times, we _should_ be fine to re-enable these on CI. If a test is failing > 3 times, there's likely a legitimate issue occuring.
#1692 is still open. This PR is not an ideal approach, but it's a quick win while we wait for that issue to be resolved.
By retrying failing tests up to 3 times, we should be fine to re-enable these on CI. If a test is failing > 3 times, there's likely a legitimate issue occuring.