Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Adds serverlessQA tag to the API tests #180773

Merged
merged 19 commits into from
Apr 23, 2024
Merged

[Security Solution] Adds serverlessQA tag to the API tests #180773

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
973d5da
adds serverlessQA tag to API tests
MadameSheema Apr 15, 2024
ec2c07b
aling tags with Cypress tests
MadameSheema Apr 15, 2024
77fa125
Merge branch 'main' into api/serverlessQA
MadameSheema Apr 15, 2024
08b8ebc
Merge branch 'main' into api/serverlessQA
MadameSheema Apr 15, 2024
b052b7f
adds missing file
MadameSheema Apr 15, 2024
dae1ce9
fixes
MadameSheema Apr 16, 2024
2137e9e
more fixes
MadameSheema Apr 16, 2024
3980615
adjustments
MadameSheema Apr 16, 2024
39270f7
adds tags information to the readme
MadameSheema Apr 17, 2024
0cc8b13
Applying Yara's comment: we need to add serverlessQA to the suites t…
MadameSheema Apr 17, 2024
37d3b3a
Merge branch 'main' into api/serverlessQA
MadameSheema Apr 17, 2024
cfdaefb
Update .buildkite/pipelines/security_solution/api_integration_serverl…
MadameSheema Apr 18, 2024
dc14e41
Update x-pack/test/security_solution_api_integration/README.md
MadameSheema Apr 18, 2024
6ff7457
Update .buildkite/pipelines/security_solution/api_integration_serverl…
MadameSheema Apr 18, 2024
78760a6
Merge branch 'main' into api/serverlessQA
MadameSheema Apr 18, 2024
1a6dc9a
updates readme
MadameSheema Apr 22, 2024
7dae577
updates package.json
MadameSheema Apr 22, 2024
3e1dd97
updating tags
MadameSheema Apr 22, 2024
0ac31e6
Merge branch 'main' into api/serverlessQA
MadameSheema Apr 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
adds tags information to the readme
  • Loading branch information
@MadameSheema
MadameSheema committed Apr 17, 2024
commit 39270f750cd91383844886957bd20dfe5eaf8839
13 changes: 11 additions & 2 deletions x-pack/test/security_solution_api_integration/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,19 @@ we have introduced the `detection_response` directory to consolidate all the int

- In this directory, Mocha tagging is utilized to assign tags to specific test suites and individual test cases. This tagging system enables the ability to selectively apply tags to test suites and test cases, facilitating the exclusion of specific test cases within a test suite as needed.

- There are three primary tags that have been defined: @ess, @serverless, and @skipInServerless

- Test suites and cases are prefixed with specific tags to determine their execution in particular environments or to exclude them from specific environments.

- We are using the following tags:
* `@ess`: Runs in an ESS environment (on-prem installation) as part of the CI validation on PRs.

* `@serverless`: Runs in the first quality gate and in the periodic pipeline.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only other mention of "first quality gate" is in the cypress README; I think it's worth defining these terms in some detail since they're not going to be self-evident to everyone.

Same thing for "periodic pipeline," "second quality gate," and "all quality gates."

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @rylnd everything is already documented.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema it's wonderful that it's documented, but how does someone reading this README find those definitions? You shared links to quality gates and periodic pipeline with me individually, but that process does not scale, so I'm proposing that we somehow reference that documentation here.

If we are reluctant/unable to share those docs because they're internal, I think that might be indication that this README either covers too much or is in the wrong place; our public documentation shouldn't reference internal processes that the reader has no insight on.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ to @rylnd's comment

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, share it with you in private due to being internal documentation, tbh I don't know what is best, thoughts? What do you prefer?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the ideal approach in my mind would be:

  1. Internal documentation associating the cypress tags to our CI/QA infrastructure/processes
  2. Public documentation with definitions appropriate to those not involved in our operations

I'm imagining a contributor who's written a cypress test for a feature they'd like to add; I would not expect them to know/care about any of these tags. As part of the review process, an Elastician would tell them which tags need to be added to their tests.

Generally: if we can't provide the reader with enough information to determine whether they should use the tag, I don't think that it's helpful to mention it at all.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a passing mention of the tags as "related to internal release processes" would be a way to both define the tags but also deemphasize them, here?


* `@serverlessQA`: Runs in the second quality gate.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tag seems to be strongly tied to the concept of a "release" (as evidenced in the command names, and this PR's description), but that isn't really reflected in either the README here nor in the tag name itself.

Would @serverlessRelease be a more accurate and descriptive tag name?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As part of the release, the tests might be executed as part of 2 other quality gates and we might have some on the second one that we don't want to execute it on the third one.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO referencing the "gate order" might be the most straightforward way to express this; it's not intuitive that QA represents "second gate," nor what the other gates are.


* `@skipInEss`: Skipped for ESS environment.

* `@skipInServerless`: Skipped for all quality gates and periodic pipeline.

ex:
```
describe('@serverless @ess create_rules', () => { ==> tests in this suite will run in both Ess and Serverless
Expand Down