[SLO] [Alerting] deployment agnostic slo burn rate rule tests (elasti…

…c#187924) Addresses elastic#179549 Relates to elastic#183113 ## Update Since the Appex QA team has taken on deployment agnostic tests, a lot of the original implementation of this PR has changed. Now that the Appex QA team has provided a current directly to write deployment agnostic tests, the burn rate rule tests have been moved here. To finish onboarding the burn rate rule test to this new framework, the following was done. 1. Add an `oblt.stateful.config.ts` file to complement the existing `oblt.serverless.config.ts` file to ensure the tests are run in CI 2. Ensure our test config is added to the buildkite pipepline 3. Add the alerting service to the new `deployment_agnostic/services` directory. 4. Port the tests over to the new `deployment_agnostic` directory To run serverless ``` node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.serverless.config.ts --grep="Burn rate rule" ``` To run stateful ``` node scripts/functional_tests_server --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts node scripts/functional_test_runner --config x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts --grep="Burn rate rule" ``` For context, I've kept the history from the original PR description below. ## 🍒 History A new type of config file will be allowed for API integration and functional tests within the `x-pack/test` folder, using a pattern of `*.serverless.config.ts` — these config files will specify configuration needed to run a set of tests in a serverless deployment context. FTR tests already make use of the `it.tags(['my-tag-1', 'my-tag-2'])` pattern, and so we would like to stay with that pattern rather than introducing a new type of mocha tag in the it block's "description" as it was introduced in this [POC](elastic#183113). The difference with the previous PR in terms of tagging is that we use `suiteTags` instead of `mochaOps` Adding following in the config files: **serverless config** ``` suiteTags: { include: ['serverless'], }, ``` **ess config** ``` suiteTags: { include: ['ess'], }, ``` and then adding `this.tags(['serverless', 'ess'])` in the test suite instructs the test runner to run the same test suite in both environments. In order to keep things simple, we stay with the current skip approach, which means that flaky tests will be skipped for all environments by appending .skip() to the suite or to specific test cases. ## Description - This PR uses `suiteTags` for tagging the tests appropriately. We decide through following labels in which environment the tests are going to be executed: - **@ess**: Runs in an ESS environment (on-prem installation) as part of the CI validation on PRs. - **@serverless**: Runs in a serverless environment. - It introduces a new folder `x-pack/test/observability_solution_api_integration` which will serve as a centralized location for all tests by obs-ux-management team that must be run in Serverless and ESS environments. A list of all tests can be found in the R&D [issue](elastic#179549) - Within this folder, there is a "**config**" subdirectory that stores base configurations specific to both the Serverless and ESS environments. These configurations build upon the base configuration provided by test_serverless and api_integration, incorporating additional settings such as environment variables and tagging options. - The file `x-pack/test/observability_solution_api_integration/test_suites/alerting/burn_rate/burn_rate_rule.ts` is functional in both Serverless and ESS - It removes the existing burn rate rule from `x-pack/test_serverless/api_integration/test_suites/observability/alerting/burn_rate/burn_rate_rule.ts` - The `alertingApi` and `sloApi` services are moved to `test/api_integration` servers In the screenshot below you can see the `test_suites` folder structure, after having migrated the current slo burn rate rule. We recommend having an `alerting` and `slo` subfolders. Rest observability apps could be added as another subfolder under test_suites. As part of this PR, the `alerting > burn_rate` subfolders are created. <img width="376" alt="Screenshot 2024-05-13 at 09 21 28" src="https://github.com/elastic/kibana/assets/2852703/3ccaf0a5-1443-4bad-ad06-daa347488bf1"> ## How to run locally You can navigate into the new `observability_solution_api_integration` folder and use following commands to run the tests in serverless and ess environments accordingly. You can find more information in the README file of the observability_solution_api_integration folder. ``` cd x-pack/test/observability_solution_api_integration // SERVERLESS npm run alerting_burn_rate:server:serverless npm run alerting_burn_rate:runner:serverless // ESS npm run alerting_burn_rate:server:ess npm run alerting_burn_rate:runner:ess ``` ## CI - It includes a new entry in the `ftr_configs.yml` to execute the newly added tests in the pipeline. - It involves the addition of `suiteTags` in both serverless/config.base.ts and ess/config.base.ts. In the case of serverless, it includes **@serverless** while excluding **@skipInServerless**. Similarly, for ess, it includes **@ess** and excludes **@skipInEss**. ## Quality Gates and MKI pipeline The Platform team will support config files within `x-pack/test` folder with a pattern of `*.serverless.config.ts`, so these tests will be included in Kibana's Quality gates and will be run against a real MKI environment. --------- Co-authored-by: Dominique Belcher <dominique.clarke@elastic.co> Co-authored-by: Dominique Clarke <doclarke71@gmail.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co> Co-authored-by: Robert Oskamp <traeluki@gmail.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
stephmilovic · Aug 21, 2024 · e3d6cf6 · e3d6cf6
1 parent 9c35c15
commit e3d6cf6
Show file tree

Hide file tree

Showing 12 changed files with 268 additions and 65 deletions.
diff --git a/.buildkite/ftr_oblt_stateful_configs.yml b/.buildkite/ftr_oblt_stateful_configs.yml
@@ -50,5 +50,5 @@ enabled:
   - x-pack/test/observability_ai_assistant_functional/enterprise/config.ts
   - x-pack/test/profiling_api_integration/cloud/config.ts
   - x-pack/test/functional/apps/apm/config.ts
-  # stateful config files that run deployment-agnostic tests
+  # stateful configs that run deployment-agnostic tests
   - x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -1142,7 +1142,9 @@ x-pack/test/observability_ai_assistant_functional @elastic/obs-ai-assistant
 /x-pack/test_serverless/**/test_suites/observability/custom_threshold_rule/ @elastic/obs-ux-management-team
 /x-pack/test_serverless/**/test_suites/observability/slos/ @elastic/obs-ux-management-team
 /x-pack/test_serverless/api_integration/test_suites/observability/es_query_rule @elastic/obs-ux-management-team
-/x-pack/test_serverless/api_integration/test_suites/observability/burn_rate_rule @elastic/obs-ux-management-team
+/x-pack/test/api_integration/deployment_agnostic/apis/observability/alerting/burn_rate_rule @elastic/obs-ux-management-team
+/x-pack/test/api_integration/deployment_agnostic/services/alerting_api @elastic/obs-ux-management-team
+/x-pack/test/api_integration/deployment_agnostic/services/slo_api @elastic/obs-ux-management-team
 /x-pack/test_serverless/**/test_suites/observability/infra/ @elastic/obs-ux-infra_services-team
 
 # Elastic Stack Monitoring

diff --git a/...vability/burn_rate_rule/burn_rate_rule.ts → .../observability/alerting/burn_rate_rule.ts b/...vability/burn_rate_rule/burn_rate_rule.ts → .../observability/alerting/burn_rate_rule.ts
@@ -4,46 +4,41 @@
  * 2.0; you may not use this file except in compliance with the Elastic License
  * 2.0.
  */
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License
- * 2.0; you may not use this file except in compliance with the Elastic License
- * 2.0.
- */
 
 import { cleanup, Dataset, generate, PartialConfig } from '@kbn/data-forge';
+import { RoleCredentials, InternalRequestHeader } from '@kbn/ftr-common-functional-services';
 import expect from '@kbn/expect';
-import { FtrProviderContext } from '../../../ftr_provider_context';
-import { InternalRequestHeader, RoleCredentials } from '../../../../shared/services';
+import { DeploymentAgnosticFtrProviderContext } from '../../../ftr_provider_context';
 
-export default function ({ getService }: FtrProviderContext) {
+export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
   const esClient = getService('es');
-  const supertest = getService('supertest');
   const esDeleteAllIndices = getService('esDeleteAllIndices');
+  const samlAuth = getService('samlAuth');
+  const supertestWithoutAuth = getService('supertestWithoutAuth');
   const logger = getService('log');
   const alertingApi = getService('alertingApi');
   const dataViewApi = getService('dataViewApi');
   const sloApi = getService('sloApi');
-  const svlUserManager = getService('svlUserManager');
-  const svlCommonApi = getService('svlCommonApi');
-  let roleAuthc: RoleCredentials;
-  let internalReqHeader: InternalRequestHeader;
+  const config = getService('config');
+  const isServerless = config.get('serverless');
+  const expectedConsumer = isServerless ? 'observability' : 'slo';
 
   describe('Burn rate rule', () => {
     const RULE_TYPE_ID = 'slo.rules.burnRate';
     const DATA_VIEW = 'kbn-data-forge-fake_hosts.fake_hosts-*';
     const RULE_ALERT_INDEX = '.alerts-observability.slo.alerts-default';
-
     const ALERT_ACTION_INDEX = 'alert-action-slo';
     const DATA_VIEW_ID = 'data-view-id';
     let dataForgeConfig: PartialConfig;
     let dataForgeIndices: string[];
     let actionId: string;
     let ruleId: string;
+    let adminRoleAuthc: RoleCredentials;
+    let internalHeaders: InternalRequestHeader;
 
     before(async () => {
-      roleAuthc = await svlUserManager.createM2mApiKeyWithRoleScope('admin');
-      internalReqHeader = svlCommonApi.getInternalRequestHeader();
+      adminRoleAuthc = await samlAuth.createM2mApiKeyWithRoleScope('admin');
+      internalHeaders = samlAuth.getInternalRequestHeader();
       dataForgeConfig = {
         schedule: [
           {
@@ -60,37 +55,49 @@ export default function ({ getService }: FtrProviderContext) {
         indexing: { dataset: 'fake_hosts' as Dataset, eventsPerCycle: 1, interval: 10000 },
       };
       dataForgeIndices = await generate({ client: esClient, config: dataForgeConfig, logger });
-      await alertingApi.waitForDocumentInIndex({ indexName: DATA_VIEW, docCountTarget: 360 });
+      await alertingApi.waitForDocumentInIndex({
+        indexName: DATA_VIEW,
+        docCountTarget: 360,
+      });
       await dataViewApi.create({
+        roleAuthc: adminRoleAuthc,
         name: DATA_VIEW,
         id: DATA_VIEW_ID,
         title: DATA_VIEW,
       });
-      roleAuthc = await svlUserManager.createM2mApiKeyWithRoleScope('admin');
     });
 
     after(async () => {
-      await supertest.delete(`/api/alerting/rule/${ruleId}`).set(internalReqHeader);
-      await supertest.delete(`/api/actions/connector/${actionId}`).set(internalReqHeader);
+      await supertestWithoutAuth
+        .delete(`/api/alerting/rule/${ruleId}`)
+        .set(adminRoleAuthc.apiKeyHeader)
+        .set(internalHeaders);
+      await supertestWithoutAuth
+        .delete(`/api/actions/connector/${actionId}`)
+        .set(adminRoleAuthc.apiKeyHeader)
+        .set(internalHeaders);
       await esClient.deleteByQuery({
         index: '.kibana-event-log-*',
         query: { term: { 'rule.id': ruleId } },
         conflicts: 'proceed',
       });
       await dataViewApi.delete({
+        roleAuthc: adminRoleAuthc,
         id: DATA_VIEW_ID,
       });
-      await supertest.delete('/api/observability/slos/my-custom-id').set(internalReqHeader);
-
+      await supertestWithoutAuth
+        .delete('/api/observability/slos/my-custom-id')
+        .set(adminRoleAuthc.apiKeyHeader)
+        .set(internalHeaders);
       await esDeleteAllIndices([ALERT_ACTION_INDEX, ...dataForgeIndices]);
       await cleanup({ client: esClient, config: dataForgeConfig, logger });
-      await svlUserManager.invalidateM2mApiKeyWithRoleScope(roleAuthc);
+      await samlAuth.invalidateM2mApiKeyWithRoleScope(adminRoleAuthc);
     });
 
     describe('Rule creation', () => {
       it('creates rule successfully', async () => {
         actionId = await alertingApi.createIndexConnector({
-          roleAuthc,
+          roleAuthc: adminRoleAuthc,
           name: 'Index Connector: Slo Burn rate API test',
           indexName: ALERT_ACTION_INDEX,
         });
@@ -119,13 +126,13 @@ export default function ({ getService }: FtrProviderContext) {
             },
             groupBy: '*',
           },
-          roleAuthc
+          adminRoleAuthc
         );
 
         const dependencyRule = await alertingApi.createRule({
-          roleAuthc,
+          roleAuthc: adminRoleAuthc,
           tags: ['observability'],
-          consumer: 'observability',
+          consumer: expectedConsumer,
           name: 'SLO Burn Rate rule - Dependency',
           ruleTypeId: RULE_TYPE_ID,
           schedule: {
@@ -196,9 +203,9 @@ export default function ({ getService }: FtrProviderContext) {
         });
 
         const createdRule = await alertingApi.createRule({
-          roleAuthc,
+          roleAuthc: adminRoleAuthc,
           tags: ['observability'],
-          consumer: 'observability',
+          consumer: expectedConsumer,
           name: 'SLO Burn Rate rule',
           ruleTypeId: RULE_TYPE_ID,
           schedule: {
@@ -279,7 +286,7 @@ export default function ({ getService }: FtrProviderContext) {
 
       it('should be active', async () => {
         const executionStatus = await alertingApi.waitForRuleStatus({
-          roleAuthc,
+          roleAuthc: adminRoleAuthc,
           ruleId,
           expectedStatus: 'active',
         });
@@ -299,9 +306,9 @@ export default function ({ getService }: FtrProviderContext) {
       });
 
       it('should find the created rule with correct information about the consumer', async () => {
-        const match = await alertingApi.findRule(roleAuthc, ruleId);
+        const match = await alertingApi.findRule(ruleId, adminRoleAuthc);
         expect(match).not.to.be(undefined);
-        expect(match.consumer).to.be('observability');
+        expect(match.consumer).to.be(expectedConsumer);
       });
     });
   });

diff --git a/x-pack/test/api_integration/deployment_agnostic/apis/observability/alerting/index.ts b/x-pack/test/api_integration/deployment_agnostic/apis/observability/alerting/index.ts
@@ -0,0 +1,14 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { DeploymentAgnosticFtrProviderContext } from '../../../ftr_provider_context';
+
+export default function ({ loadTestFile }: DeploymentAgnosticFtrProviderContext) {
+  describe('Slo - Burn rate rule', () => {
+    loadTestFile(require.resolve('./burn_rate_rule'));
+  });
+}
diff --git a/x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.index.ts b/x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.index.ts
@@ -12,5 +12,6 @@ export default function ({ loadTestFile }: DeploymentAgnosticFtrProviderContext)
     loadTestFile(require.resolve('../../apis/console'));
     loadTestFile(require.resolve('../../apis/core'));
     loadTestFile(require.resolve('../../apis/painless_lab'));
+    loadTestFile(require.resolve('../../apis/observability/alerting'));
   });
 }
diff --git a/x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.index.ts b/x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.index.ts
@@ -7,8 +7,9 @@
 
 import { DeploymentAgnosticFtrProviderContext } from '../../ftr_provider_context';
 
-export default function ({}: DeploymentAgnosticFtrProviderContext) {
+export default function ({ loadTestFile }: DeploymentAgnosticFtrProviderContext) {
   describe('apis', () => {
     // load new oblt deployment-agnostic test here
+    loadTestFile(require.resolve('../../apis/observability/alerting'));
   });
 }
diff --git a/x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts b/x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts
@@ -10,6 +10,6 @@ import { createStatefulTestConfig } from '../../default_configs/stateful.config.
 export default createStatefulTestConfig({
   testFiles: [require.resolve('./oblt.index.ts')],
   junit: {
-    reportName: 'Observibility Stateful - Deployment-agnostic API Integration Tests',
+    reportName: 'Stateful Observability - Deployment-agnostic API Integration Tests',
   },
 });