[kbn-scout] Add Synthtrace as a fixture #210505

rmyz · 2025-02-11T09:09:20Z

Summary

This PR adds synthtrace clients to scout as a test fixture, so you can use it in your test to generate data.

The clients added were apmSynthtraceEsClient, infraSynthtraceEsClient and otelSynthtraceEsClient.

How to use them in parallel tests

As synthtrace ingests data into our indices, and sequential runs would be the perfect way to introduce flakiness in our tests, there is a better way to ingest data, using a hook, at the setup phase with globalSetup.
We need to create a global_setup.ts file and link it into our playwright config.
Then we can use something like

async function globalSetup(config: FullConfig) {
  const data = {
    apm: [
      opbeans({
        from: new Date(start).getTime(),
        to: new Date(end).getTime(),
      }),
    ],
    infra: [
      generateHosts({
        from: new Date(start).toISOString(),
        to: new Date(end).toISOString(),
      }),
    ],
    otel: [
      sendotlp({
        from: new Date(start).getTime(),
        to: new Date(end).getTime(),
      }),
    ],
  };

  return ingestSynthtraceDataHook(config, data);
}

Each key (apm, infra, otel) accepts an array of generators.

How to use them in sequential tests

Warning

This should not be the standard behaviour, we should embrace parallelism and use sequential testing when there is no other way.

apmSynthtraceEsClient

 test.before(
    async ({ apmSynthtraceEsClient }) => {
      await apmSynthtraceEsClient.index(
        opbeans({
          from: new Date(start).getTime(),
          to: new Date(end).getTime(),
        })
      );
    }
  );

opbeans file used in the example.

otelSynthtraceEsClient

 test.before(
    async ({otelSynthtraceEsClient }) => {
      await otelSynthtraceEsClient.index(
        sendotlp({
          from: new Date(start).getTime(),
          to: new Date(end).getTime(),
        })
      );
    }
  );

sendotlp file which will create the data.

infraSynthtraceEsClient

 test.before(
    async ({ infraSynthtraceEsClient }) => {
      await infraSynthtraceEsClient.index(
        generateHosts({
          from: new Date(start).toISOString(),
          to: new Date(end).toISOString(),
        })
      );
    }
  );

generateHosts file used to generate data.

rmyz · 2025-02-11T09:09:48Z

/ci

rmyz · 2025-02-11T09:21:32Z

/ci

packages/kbn-scout/src/playwright/fixtures/test/synthtrace/index.ts

rmyz · 2025-02-11T10:52:49Z

/ci

jennypavlova

Q: is there a way to test atm or do we need the fixture from the other draft (I tested there but wanted to check here if it's possible - it should work the same so I can just copy it from there, just wanted to fist check with you if we want to test it again)

packages/kbn-scout/src/playwright/fixtures/test/synthtrace/index.ts

rmyz · 2025-02-12T10:18:32Z

Q: is there a way to test atm or do we need the fixture from the other draft (I tested there but wanted to check here if it's possible - it should work the same so I can just copy it from there, just wanted to fist check with you if we want to test it again)

It should be the same, the only change I made is for the naming, so the synthtrace clients that we expose are: apmSynthtraceEsClient, infraSynthtraceEsClient and otelSynthtraceEsClient.

jennypavlova

The code LGTM 💯 Thanks for the fixes!

rmyz · 2025-02-12T10:47:33Z

Tested parallelism using this playwright config as it was already configured, and created a few tests using synthtrace, works fine ✅

rmyz · 2025-02-12T11:09:25Z

Tested all clients and they are working ✅

elasticmachine · 2025-02-12T11:19:02Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

dmlemeshko

Hey @rmyz , nice work!

I left questions to better understand the use case and hopefully make few improvements.

packages/kbn-scout/src/playwright/fixtures/test/synthtrace/index.ts

packages/kbn-scout/src/playwright/fixtures/parallel_run_fixtures.ts

crespocarlos

Thanks for the changes! just a couple questions

crespocarlos · 2025-02-14T13:40:40Z

packages/kbn-scout/src/playwright/global_hooks/synthtrace_ingestion.ts

+      if (Object.hasOwn(data, key)) {
+        const typedKey = key as keyof SynthtraceIngestionData;
+        if (data[typedKey].length > 0) {
+          const client = await getClient(typedKey, esClient, kbnUrl, scoutConfig.auth, log);


nit: getSynthtraceClient to make this name more specific.

crespocarlos · 2025-02-14T13:42:32Z

packages/kbn-scout/src/playwright/global_hooks/synthtrace_ingestion.ts

+    const kbnUrl = scoutConfig.hosts.kibana;
+
+    for (const key of Object.keys(data)) {
+      if (Object.hasOwn(data, key)) {


Do we need to check if Object.hasOwn(data, key)? It's already iterating over data keys.

removed, not needed

crespocarlos

LGTM. thanks for applying the suggestions

miloszmarcinkowski

I'm sorry for the late review. Nice job with synthtrace 🚀

miloszmarcinkowski · 2025-02-14T15:32:36Z

packages/kbn-scout/src/playwright/global_hooks/synthtrace_ingestion.ts

+            })
+          );
+        } catch (e) {
+          log.debug(`[setup] error ingesting ${key} synthtrace data`, e);


Suggested change

log.debug(`[setup] error ingesting ${key} synthtrace data`, e);

log.error(`[setup] error ingesting ${key} synthtrace data`, e);

nit: it can be logged as error here

elasticmachine · 2025-02-14T18:53:07Z

💚 Build Succeeded

Buildkite Build
Commit: e15551e

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/scout`	93	99	+6

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/scout`	14	15	+1

Unknown metric groups

API count

id	before	after	diff
`@kbn/scout`	383	389	+6

History

💔 Build #276413 failed 8da80e3
💚 Build #276007 succeeded ea99d35
💚 Build #275944 succeeded ced8cf9
💚 Build #275498 succeeded 88a527a
💛 Build #275054 was flaky 84191a7

cc @rmyz

kibanamachine · 2025-02-15T11:44:08Z

Starting backport for target branches: 8.x, 9.0

https://github.com/elastic/kibana/actions/runs/13344704434

kibanamachine · 2025-02-15T11:44:09Z

Starting backport for target branches: 8.x, 9.0

https://github.com/elastic/kibana/actions/runs/13344704463

## Summary Closes elastic#210340 This PR adds synthtrace clients to scout as a test fixture, so you can use it in your test to generate data. The clients added were `apmSynthtraceEsClient`, `infraSynthtraceEsClient` and `otelSynthtraceEsClient`. ## How to use them in parallel tests As `synthtrace` ingests data into our indices, and sequential runs would be the perfect way to introduce flakiness in our tests, there is a better way to ingest data, using a hook, at the setup phase with `globalSetup`. We need to create a `global_setup.ts` file and link it into our playwright config. Then we can use something like ``` async function globalSetup(config: FullConfig) { const data = { apm: [ opbeans({ from: new Date(start).getTime(), to: new Date(end).getTime(), }), ], infra: [ generateHosts({ from: new Date(start).toISOString(), to: new Date(end).toISOString(), }), ], otel: [ sendotlp({ from: new Date(start).getTime(), to: new Date(end).getTime(), }), ], }; return ingestSynthtraceDataHook(config, data); } ``` Each key (apm, infra, otel) accepts an array of generators. ## How to use them in sequential tests > [!WARNING] > This should not be the standard behaviour, we should embrace parallelism and use sequential testing when there is no other way. ### apmSynthtraceEsClient ```ts test.before( async ({ apmSynthtraceEsClient }) => { await apmSynthtraceEsClient.index( opbeans({ from: new Date(start).getTime(), to: new Date(end).getTime(), }) ); } ); ``` [opbeans file](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/fixtures/synthtrace/opbeans.ts) used in the example. ### otelSynthtraceEsClient ```ts test.before( async ({otelSynthtraceEsClient }) => { await otelSynthtraceEsClient.index( sendotlp({ from: new Date(start).getTime(), to: new Date(end).getTime(), }) ); } ); ``` [sendotlp file](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/fixtures/synthtrace/sendotlp.ts) which will create the data. ### infraSynthtraceEsClient ```ts test.before( async ({ infraSynthtraceEsClient }) => { await infraSynthtraceEsClient.index( generateHosts({ from: new Date(start).toISOString(), to: new Date(end).toISOString(), }) ); } ); ``` [generateHosts file](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/inventory/e2e/cypress/e2e/alert_count/generate_data.ts#L82) used to generate data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit e21c5d0)

kibanamachine · 2025-02-15T11:49:42Z

💔 Some backports could not be created

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts You might need to backport the following PRs to 8.x: - [scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761) - [Scout] add maps test (#204607)
✅	9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 210505

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-02-15T11:49:42Z

💔 Some backports could not be created

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts You might need to backport the following PRs to 8.x: - [scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761) - [Scout] add maps test (#204607)
✅	9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 210505

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `9.0`: - [[kbn-scout] Add Synthtrace as a fixture (#210505)](#210505)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Sergi Romeu <sergi.romeu@elastic.co>

kibanamachine · 2025-02-18T08:26:16Z

Starting backport for target branches: 8.x, 9.0

https://github.com/elastic/kibana/actions/runs/13386246157

## Summary Closes elastic#210340 This PR adds synthtrace clients to scout as a test fixture, so you can use it in your test to generate data. The clients added were `apmSynthtraceEsClient`, `infraSynthtraceEsClient` and `otelSynthtraceEsClient`. ## How to use them in parallel tests As `synthtrace` ingests data into our indices, and sequential runs would be the perfect way to introduce flakiness in our tests, there is a better way to ingest data, using a hook, at the setup phase with `globalSetup`. We need to create a `global_setup.ts` file and link it into our playwright config. Then we can use something like ``` async function globalSetup(config: FullConfig) { const data = { apm: [ opbeans({ from: new Date(start).getTime(), to: new Date(end).getTime(), }), ], infra: [ generateHosts({ from: new Date(start).toISOString(), to: new Date(end).toISOString(), }), ], otel: [ sendotlp({ from: new Date(start).getTime(), to: new Date(end).getTime(), }), ], }; return ingestSynthtraceDataHook(config, data); } ``` Each key (apm, infra, otel) accepts an array of generators. ## How to use them in sequential tests > [!WARNING] > This should not be the standard behaviour, we should embrace parallelism and use sequential testing when there is no other way. ### apmSynthtraceEsClient ```ts test.before( async ({ apmSynthtraceEsClient }) => { await apmSynthtraceEsClient.index( opbeans({ from: new Date(start).getTime(), to: new Date(end).getTime(), }) ); } ); ``` [opbeans file](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/fixtures/synthtrace/opbeans.ts) used in the example. ### otelSynthtraceEsClient ```ts test.before( async ({otelSynthtraceEsClient }) => { await otelSynthtraceEsClient.index( sendotlp({ from: new Date(start).getTime(), to: new Date(end).getTime(), }) ); } ); ``` [sendotlp file](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/fixtures/synthtrace/sendotlp.ts) which will create the data. ### infraSynthtraceEsClient ```ts test.before( async ({ infraSynthtraceEsClient }) => { await infraSynthtraceEsClient.index( generateHosts({ from: new Date(start).toISOString(), to: new Date(end).toISOString(), }) ); } ); ``` [generateHosts file](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/inventory/e2e/cypress/e2e/alert_count/generate_data.ts#L82) used to generate data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit e21c5d0)

kibanamachine · 2025-02-18T08:31:48Z

💔 Some backports could not be created

Status	Branch	Result
✅	8.x
❌	9.0	Cherrypick failed because the selected commit (`e21c5d0`) is empty. It looks like the commit was already backported in #211351

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 210505

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.x`: - [[kbn-scout] Add Synthtrace as a fixture (#210505)](#210505)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) \n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n\n\nCo-authored-by: Sergi Romeu <sergi.romeu@elastic.co>"}},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/210505","number":210505,"mergeCommit":{"message":"[kbn-scout] Add Synthtrace as a fixture (#210505)\n\n## Summary\n\nCloses #210340\n\nThis PR adds synthtrace clients to scout as a test fixture, so you can\nuse it in your test to generate data.\n\nThe clients added were `apmSynthtraceEsClient`,\n`infraSynthtraceEsClient` and `otelSynthtraceEsClient`.\n\n## How to use them in parallel tests\n\nAs `synthtrace` ingests data into our indices, and sequential runs would\nbe the perfect way to introduce flakiness in our tests, there is a\nbetter way to ingest data, using a hook, at the setup phase with\n`globalSetup`.\nWe need to create a `global_setup.ts` file and link it into our\nplaywright config.\nThen we can use something like\n```\nasync function globalSetup(config: FullConfig) {\n const data = {\n apm: [\n opbeans({\n from: new Date(start).getTime(),\n to: new Date(end).getTime(),\n }),\n ],\n infra: [\n generateHosts({\n from: new Date(start).toISOString(),\n to: new Date(end).toISOString(),\n }),\n ],\n otel: [\n sendotlp({\n from: new Date(start).getTime(),\n to: new Date(end).getTime(),\n }),\n ],\n };\n\n return ingestSynthtraceDataHook(config, data);\n}\n```\nEach key (apm, infra, otel) accepts an array of generators.\n\n## How to use them in sequential tests\n> [!WARNING] \n> This should not be the standard behaviour, we should embrace\nparallelism and use sequential testing when there is no other way.\n\n### apmSynthtraceEsClient\n```ts\n test.before(\n async ({ apmSynthtraceEsClient }) => {\n await apmSynthtraceEsClient.index(\n opbeans({\n from: new Date(start).getTime(),\n to: new Date(end).getTime(),\n })\n );\n }\n );\n```\n[opbeans\nfile](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/fixtures/synthtrace/opbeans.ts)\nused in the example.\n\n### otelSynthtraceEsClient\n```ts\n test.before(\n async ({otelSynthtraceEsClient }) => {\n await otelSynthtraceEsClient.index(\n sendotlp({\n from: new Date(start).getTime(),\n to: new Date(end).getTime(),\n })\n );\n }\n );\n```\n[sendotlp\nfile](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/fixtures/synthtrace/sendotlp.ts)\nwhich will create the data.\n\n### infraSynthtraceEsClient\n```ts\n test.before(\n async ({ infraSynthtraceEsClient }) => {\n await infraSynthtraceEsClient.index(\n generateHosts({\n from: new Date(start).toISOString(),\n to: new Date(end).toISOString(),\n })\n );\n }\n );\n```\n[generateHosts\nfile](https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/inventory/e2e/cypress/e2e/alert_count/generate_data.ts#L82)\nused to generate data.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"e21c5d0e9175ffd1bea0ad78ffe26cb973cc489f"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: Sergi Romeu <sergi.romeu@elastic.co>

feat: add synthtrace to scout

8cf3b53

rmyz self-assigned this Feb 11, 2025

rmyz added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting apm:synthtrace APM Synthtrace Topics Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team test:scout labels Feb 11, 2025

rmyz mentioned this pull request Feb 11, 2025

feat: add synthtrace rmyz/kibana#1

Closed

10 tasks

[CI] Auto-commit changed files from 'node scripts/yarn_deduplicate'

240a855

jennypavlova reviewed Feb 11, 2025

View reviewed changes

packages/kbn-scout/src/playwright/fixtures/test/synthtrace/index.ts Outdated Show resolved Hide resolved

rmyz added 3 commits February 11, 2025 11:28

fix types

853b80e

feat: add infra client

6b5c7fd

improve typings

84191a7

jennypavlova reviewed Feb 11, 2025

View reviewed changes

rmyz added 2 commits February 12, 2025 10:56

PR comment fixes

307095b

Merge branch 'main' into 210340-scout-add-synthtrace-fixture

9ea808d

jennypavlova approved these changes Feb 12, 2025

View reviewed changes

rmyz marked this pull request as ready for review February 12, 2025 11:18

rmyz requested a review from a team as a code owner February 12, 2025 11:19

fix

88a527a

dmlemeshko reviewed Feb 12, 2025

View reviewed changes

rmyz added 2 commits February 12, 2025 16:44

address PR comments

b0a2cdb

feat: add synthtrace ingestion hook

9742d27

Merge branch 'main' into 210340-scout-add-synthtrace-fixture

3cd18cd

crespocarlos reviewed Feb 14, 2025

View reviewed changes

solve PR comments

8da80e3

crespocarlos approved these changes Feb 14, 2025

View reviewed changes

rmyz enabled auto-merge (squash) February 14, 2025 14:23

miloszmarcinkowski approved these changes Feb 14, 2025

View reviewed changes

Merge branch 'main' into 210340-scout-add-synthtrace-fixture

e15551e

rmyz merged commit e21c5d0 into elastic:main Feb 14, 2025
9 checks passed

kibanamachine added the v9.1.0 label Feb 14, 2025

dmlemeshko added v9.0.0 v8.19.0 backport:version Backport to applied version labels and removed backport:skip This commit does not require backporting labels Feb 15, 2025

kibanamachine mentioned this pull request Feb 15, 2025

[9.0] [kbn-scout] Add Synthtrace as a fixture (#210505) #211351

Merged

rmyz deleted the 210340-scout-add-synthtrace-fixture branch February 17, 2025 09:20

dmlemeshko added backport:version Backport to applied version labels and removed backport:version Backport to applied version labels labels Feb 18, 2025

kibanamachine mentioned this pull request Feb 18, 2025

[8.x] [kbn-scout] Add Synthtrace as a fixture (#210505) #211524

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[kbn-scout] Add Synthtrace as a fixture #210505

[kbn-scout] Add Synthtrace as a fixture #210505

rmyz commented Feb 11, 2025 •

edited by kibanamachine

Loading

rmyz commented Feb 11, 2025

rmyz commented Feb 11, 2025

rmyz commented Feb 11, 2025

jennypavlova left a comment

rmyz commented Feb 12, 2025

jennypavlova left a comment

rmyz commented Feb 12, 2025

rmyz commented Feb 12, 2025

elasticmachine commented Feb 12, 2025

dmlemeshko left a comment

crespocarlos left a comment

crespocarlos Feb 14, 2025

rmyz Feb 14, 2025

crespocarlos Feb 14, 2025

rmyz Feb 14, 2025

crespocarlos left a comment

miloszmarcinkowski left a comment

miloszmarcinkowski Feb 14, 2025

elasticmachine commented Feb 14, 2025

API count

kibanamachine commented Feb 15, 2025

kibanamachine commented Feb 15, 2025

kibanamachine commented Feb 15, 2025

kibanamachine commented Feb 15, 2025

kibanamachine commented Feb 18, 2025

kibanamachine commented Feb 18, 2025

	log.debug(`[setup] error ingesting ${key} synthtrace data`, e);
	log.error(`[setup] error ingesting ${key} synthtrace data`, e);

[kbn-scout] Add Synthtrace as a fixture #210505

[kbn-scout] Add Synthtrace as a fixture #210505

Conversation

rmyz commented Feb 11, 2025 • edited by kibanamachine Loading

Summary

How to use them in parallel tests

How to use them in sequential tests

apmSynthtraceEsClient

otelSynthtraceEsClient

infraSynthtraceEsClient

rmyz commented Feb 11, 2025

rmyz commented Feb 11, 2025

rmyz commented Feb 11, 2025

jennypavlova left a comment

Choose a reason for hiding this comment

rmyz commented Feb 12, 2025

jennypavlova left a comment

Choose a reason for hiding this comment

rmyz commented Feb 12, 2025

rmyz commented Feb 12, 2025

elasticmachine commented Feb 12, 2025

dmlemeshko left a comment

Choose a reason for hiding this comment

crespocarlos left a comment

Choose a reason for hiding this comment

crespocarlos Feb 14, 2025

Choose a reason for hiding this comment

rmyz Feb 14, 2025

Choose a reason for hiding this comment

crespocarlos Feb 14, 2025

Choose a reason for hiding this comment

rmyz Feb 14, 2025

Choose a reason for hiding this comment

crespocarlos left a comment

Choose a reason for hiding this comment

miloszmarcinkowski left a comment

Choose a reason for hiding this comment

miloszmarcinkowski Feb 14, 2025

Choose a reason for hiding this comment

elasticmachine commented Feb 14, 2025

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

API count

History

kibanamachine commented Feb 15, 2025

kibanamachine commented Feb 15, 2025

kibanamachine commented Feb 15, 2025

💔 Some backports could not be created

Manual backport

Questions ?

kibanamachine commented Feb 15, 2025

💔 Some backports could not be created

Manual backport

Questions ?

kibanamachine commented Feb 18, 2025

kibanamachine commented Feb 18, 2025

💔 Some backports could not be created

Manual backport

Questions ?

rmyz commented Feb 11, 2025 •

edited by kibanamachine

Loading