forked from elastic/kibana
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[scout] adding test helper
@kbn/scout-oblt package and uptate onboa…
…rding tests (elastic#209761) ## Summary `@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test helpers specifically designed to test `Observability` applications in Kibana. All Oblt plugins should only import from `@kbn/scout-oblt` Its primary goal is to simplify the test development experience for teams working on `Observability` plugins by providing custom Playwright fixtures, page objects, and utilities tailored for Observability-related testing scenarios. Contributing: - when Fixture/Page Object is sharable across all Solutions and Platform (`fleetApi` fixture), it should be added in `@kbn/scout` - when Fixture/Page Object is Oblt-specific but is shared across tests under the multiple plugins (`OnboardingHome` page), it should be added in `@kbn/scout-oblt` - when Fixture/Page Object is only used in a single plugin (`onboarding` internal APIs ?), it should be added in this plugin. I also re-worked existing tests with few ideas in mind: - Scout is **e2e testing tool** and should target primary e2e test scenarios; We have _API integration tests_ to test multiple short scenarios for APIs behavior (response, status code) and _jest/React testing library_ to test components in isolation (elements rendering, fields validation). Doing all the testing with e2e tool like Playwright will dramatically affect cost efficiency and stability of tests, but also slows overall CI execution and PRs delivery. The goal is to follow testing pyramid and keep in mind its principles. - We on purpose spin up new browser context for each `test` block to make sure our **tests are independent**. Having too many short `test` blocks in the file significantly slows down the execution: every block triggers browser context, saml authentication, adding/removing Fleet integrations (each call up to 2 seconds) and other beforeEach/afterEach hooks. Real browser-based testing is expensive. It is not about putting every step into 1 `test` block, but also not a Jest unit-test-style design. When it is possible to group similar actions on the same page and if it is a part of the same user flow - we should do it. It also doesn't bring the testing value repeating the same UI steps multiple times in different scenarios. _Our CI costs are critical to cut when it is possible_ - Avoid **nesting describe** blocks: it complicates test readability and also complicates for CI bot to properly skip the failing block (it will skip the top level one). We encourage **Scout parallel test execution** based on running test spec files in multiple workers, not the `test` blocks within the same file. Having too many `test` blocks in the same file will be slowly run in the single thread and in case of flakiness, it means Team lose more test coverage than they probably expect. Before (**59** test blocks - **8-8.5 min** per distro): <img width="1709" alt="Screenshot 2025-02-08 at 18 01 40" src="https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab" /> After (**15** test blocks - **3.5-4 min** per distro): <img width="1578" alt="Screenshot 2025-02-10 at 18 14 42" src="https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120" /> For reviewers: updated tests are possible to run in 2 parallel workers against the same Kibana/ES instance and run time is dropping to **2.5-3 min** 🚀 . It is up to UX-Logs team to decide if you want to keep parallel run (new tests can be added either to parallel or sequential run) <img width="1578" alt="Screenshot 2025-02-11 at 12 14 30" src="https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41" /> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Loading branch information
1 parent
000e913
commit bd13e82
Showing
41 changed files
with
1,086 additions
and
1,183 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
71 changes: 71 additions & 0 deletions
71
packages/kbn-scout/src/playwright/fixtures/worker/apis/fleet/index.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the "Elastic License | ||
| * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side | ||
| * Public License v 1"; you may not use this file except in compliance with, at | ||
| * your election, the "Elastic License 2.0", the "GNU Affero General Public | ||
| * License v3.0 only", or the "Server Side Public License, v 1". | ||
| */ | ||
|
|
||
| import { measurePerformanceAsync } from '../../../../../common'; | ||
| import { coreWorkerFixtures } from '../../core_fixtures'; | ||
|
|
||
| export interface FleetApiFixture { | ||
| integration: { | ||
| install: (name: string) => Promise<void>; | ||
| delete: (name: string) => Promise<void>; | ||
| }; | ||
| } | ||
|
|
||
| /** | ||
| * This fixture provides a helper to interact with the Fleet API. | ||
| */ | ||
| export const fleetApiFixture = coreWorkerFixtures.extend<{}, { fleetApi: FleetApiFixture }>({ | ||
| fleetApi: [ | ||
| async ({ kbnClient, log }, use) => { | ||
| const fleetApiHelper = { | ||
| integration: { | ||
| install: async (name: string) => { | ||
| await measurePerformanceAsync( | ||
| log, | ||
| `fleetApi.integration.install [${name}]`, | ||
| async () => { | ||
| await kbnClient.request({ | ||
| method: 'POST', | ||
| path: `/api/fleet/epm/custom_integrations`, | ||
| body: { | ||
| force: true, | ||
| integrationName: name, | ||
| datasets: [ | ||
| { name: `${name}.access`, type: 'logs' }, | ||
| { name: `${name}.error`, type: 'metrics' }, | ||
| { name: `${name}.warning`, type: 'logs' }, | ||
| ], | ||
| }, | ||
| }); | ||
| } | ||
| ); | ||
| }, | ||
|
|
||
| delete: async (name: string) => { | ||
| await measurePerformanceAsync( | ||
| log, | ||
| `fleetApi.integration.delete [${name}]`, | ||
| async () => { | ||
| await kbnClient.request({ | ||
| method: 'DELETE', | ||
| path: `/api/fleet/epm/packages/${name}`, | ||
| ignoreErrors: [400], | ||
| }); | ||
| } | ||
| ); | ||
| }, | ||
| }, | ||
| }; | ||
|
|
||
| log.serviceLoaded('fleetApi'); | ||
| await use(fleetApiHelper); | ||
| }, | ||
| { scope: 'worker' }, | ||
| ], | ||
| }); |
21 changes: 21 additions & 0 deletions
21
packages/kbn-scout/src/playwright/fixtures/worker/apis/index.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the "Elastic License | ||
| * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side | ||
| * Public License v 1"; you may not use this file except in compliance with, at | ||
| * your election, the "Elastic License 2.0", the "GNU Affero General Public | ||
| * License v3.0 only", or the "Server Side Public License, v 1". | ||
| */ | ||
|
|
||
| import { mergeTests } from 'playwright/test'; | ||
| import { FleetApiFixture, fleetApiFixture } from './fleet'; | ||
|
|
||
| export const apiFixtures = mergeTests(fleetApiFixture); | ||
|
|
||
| export interface ApiFixtures { | ||
| fleetApi: FleetApiFixture; | ||
| } | ||
|
|
||
| export interface ApiParallelWorkerFixtures { | ||
| fleetApi: FleetApiFixture; | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
45 changes: 45 additions & 0 deletions
45
x-pack/solutions/observability/packages/kbn-scout-oblt/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # @kbn/scout-oblt | ||
|
|
||
| `@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test helpers specifically designed for `Observability` products in Kibana. | ||
|
|
||
| Its primary goal is to simplify the test development experience for teams working on `Observability` plugins by providing custom Playwright fixtures, page objects, and utilities tailored for Observability-related testing scenarios. | ||
|
|
||
| ### Table of Contents | ||
|
|
||
| 1. Folder Structure | ||
| 2. How to Use | ||
| 3. Contributing | ||
|
|
||
| ### Folder Structure | ||
|
|
||
| The `@kbn/scout-oblt` structure includes the following key directories and files: | ||
|
|
||
| ``` | ||
| x-pack/solutions/observability/packages/kbn-scout-oblt/ | ||
| ├── src/ | ||
| │ ├── playwright/ | ||
| │ │ └── fixtures | ||
| │ │ │ └── test/ | ||
| │ │ │ │ └── // Observability test-scope fixtures | ||
| │ │ │ └── worker/ | ||
| │ │ │ │ └── // Observability worker-scope fixtures | ||
| │ │ │ └── single_thread_fixtures.ts | ||
| │ │ │ └── parallel_run_fixtures.ts | ||
| │ │ │ └── index.ts | ||
| │ │ └── page_objects/ | ||
| │ │ │ └── // Observability pages | ||
| │ └── index.ts | ||
| ├── package.json | ||
| ├── tsconfig.json | ||
| ``` | ||
|
|
||
| ### How to use | ||
|
|
||
| ``` | ||
| import { test } from '@kbn/scout-oblt'; | ||
| test('verifies Observability Home loads', async ({ page, pageObjects }) => { | ||
| await pageObjects.onboardingHome.goto(); | ||
| expect(await page.title()).toContain('Observability'); | ||
| }); | ||
| ``` |
38 changes: 38 additions & 0 deletions
38
x-pack/solutions/observability/packages/kbn-scout-oblt/index.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| export { test, spaceTest } from './src/playwright'; | ||
| export type { | ||
| ObltPageObjects, | ||
| ObltTestFixtures, | ||
| ObltWorkerFixtures, | ||
| ObltParallelTestFixtures, | ||
| ObltParallelWorkerFixtures, | ||
| } from './src/playwright'; | ||
|
|
||
| // re-export from @kbn/scout | ||
| export { | ||
| expect, | ||
| tags, | ||
| createPlaywrightConfig, | ||
| createLazyPageObject, | ||
| ingestTestDataHook, | ||
| } from '@kbn/scout'; | ||
|
|
||
| export type { | ||
| EsClient, | ||
| KbnClient, | ||
| KibanaUrl, | ||
| ScoutLogger, | ||
| ScoutPage, | ||
| PageObjects, | ||
| ScoutServerConfig, | ||
| ScoutTestConfig, | ||
| ScoutPlaywrightOptions, | ||
| ScoutTestOptions, | ||
| Locator, | ||
| } from '@kbn/scout'; |
12 changes: 12 additions & 0 deletions
12
x-pack/solutions/observability/packages/kbn-scout-oblt/jest.config.js
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| module.exports = { | ||
| preset: '@kbn/test/jest_node', | ||
| rootDir: '../../../../..', | ||
| roots: ['<rootDir>/x-pack/solutions/observability/packages/kbn-scout-oblt'], | ||
| }; |
6 changes: 6 additions & 0 deletions
6
x-pack/solutions/observability/packages/kbn-scout-oblt/kibana.jsonc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| { | ||
| "type": "test-helper", | ||
| "id": "@kbn/scout-oblt", | ||
| "owner": "@elastic/appex-qa", | ||
| "devOnly": true | ||
| } |
6 changes: 6 additions & 0 deletions
6
x-pack/solutions/observability/packages/kbn-scout-oblt/package.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| { | ||
| "name": "@kbn/scout-oblt", | ||
| "private": true, | ||
| "version": "1.0.0", | ||
| "license": "Elastic License 2.0" | ||
| } |
10 changes: 10 additions & 0 deletions
10
x-pack/solutions/observability/packages/kbn-scout-oblt/src/playwright/fixtures/index.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| export { test } from './single_thread_fixtures'; | ||
| export { spaceTest } from './parallel_run_fixtures'; | ||
| export * from './types'; |
Oops, something went wrong.