Skip to content

Commit

Permalink
Include skipped (not omitted) tests in reports
Browse files Browse the repository at this point in the history 
This fixes #1041 [1].

[1] #1041
  • Loading branch information
@badeball
badeball committed Jun 23, 2024
1 parent f343cc3 commit 1d9ce1c
Show file tree
Hide file tree
Showing 6 changed files with 436 additions and 0 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

All notable changes to this project will be documented in this file.

## Unreleased

- Include skipped (not omitted) tests in reports, fixes [#1041](https://github.com/badeball/cypress-cucumber-preprocessor/issues/1041).

## v20.0.7

- Updated all dependencies, fixes [#1198](https://github.com/badeball/cypress-cucumber-preprocessor/issues/1198).
Expand Down
147 changes: 147 additions & 0 deletions features/pretty_output.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,153 @@ Feature: pretty output
Given a step
"""

Rule: it should include results of skipped (not omitted) tests
Scenario: first scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
@skip
Scenario: first scenario
Given a step
Scenario: second scenario
Given a step
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And the output should contain
"""
Feature: a feature # cypress/e2e/a.feature:1
@skip
Scenario: first scenario # cypress/e2e/a.feature:3
Given a step
- skipped
Scenario: second scenario # cypress/e2e/a.feature:5
Given a step
Scenario: third scenario # cypress/e2e/a.feature:7
Given a step
"""

Scenario: middle scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
Scenario: first scenario
Given a step
@skip
Scenario: second scenario
Given a step
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And the output should contain
"""
Feature: a feature # cypress/e2e/a.feature:1
Scenario: first scenario # cypress/e2e/a.feature:2
Given a step
@skip
Scenario: second scenario # cypress/e2e/a.feature:5
Given a step
- skipped
Scenario: third scenario # cypress/e2e/a.feature:7
Given a step
"""

Scenario: last scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
Scenario: first scenario
Given a step
Scenario: second scenario
Given a step
@skip
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And the output should contain
"""
Feature: a feature # cypress/e2e/a.feature:1
Scenario: first scenario # cypress/e2e/a.feature:2
Given a step
Scenario: second scenario # cypress/e2e/a.feature:4
Given a step
@skip
Scenario: third scenario # cypress/e2e/a.feature:7
Given a step
- skipped
"""

Scenario: all scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
@skip
Scenario: first scenario
Given a step
@skip
Scenario: second scenario
Given a step
@skip
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And the output should contain
"""
Feature: a feature # cypress/e2e/a.feature:1
@skip
Scenario: first scenario # cypress/e2e/a.feature:3
Given a step
- skipped
@skip
Scenario: second scenario # cypress/e2e/a.feature:6
Given a step
- skipped
@skip
Scenario: third scenario # cypress/e2e/a.feature:9
Given a step
- skipped
"""

@network
Rule: it should handle reloads gracefully in a multitude of scenarios

Expand Down
87 changes: 87 additions & 0 deletions features/reporters/json.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -387,6 +387,93 @@ Feature: JSON formatter
Then it fails
And the JSON report should contain an image attachment for what appears to be a screenshot

Rule: it should include results of skipped (not omitted) tests
Scenario: first scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
@skip
Scenario: first scenario
Given a step
Scenario: second scenario
Given a step
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And there should be a JSON output similar to "fixtures/skipped-first-scenario.json"

Scenario: middle scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
Scenario: first scenario
Given a step
@skip
Scenario: second scenario
Given a step
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And there should be a JSON output similar to "fixtures/skipped-second-scenario.json"

Scenario: last scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
Scenario: first scenario
Given a step
Scenario: second scenario
Given a step
@skip
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And there should be a JSON output similar to "fixtures/skipped-third-scenario.json"

Scenario: all scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
@skip
Scenario: first scenario
Given a step
@skip
Scenario: second scenario
Given a step
@skip
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And there should be a JSON output similar to "fixtures/skipped-all-scenarios.json"

Rule: failing Cypress hooks outside of the plugins control should discontinue the report
Scenario: failing before hook
Given a file named "cypress/e2e/a.feature" with:
Expand Down
87 changes: 87 additions & 0 deletions features/reporters/messages.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,93 @@ Feature: messages report
Then it fails
And the messages report should contain an image attachment for what appears to be a screenshot

Rule: it should include results of skipped (not omitted) tests
Scenario: first scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
@skip
Scenario: first scenario
Given a step
Scenario: second scenario
Given a step
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And there should be a messages similar to "fixtures/skipped-first-scenario.ndjson"

Scenario: middle scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
Scenario: first scenario
Given a step
@skip
Scenario: second scenario
Given a step
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And there should be a messages similar to "fixtures/skipped-second-scenario.ndjson"

Scenario: last scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
Scenario: first scenario
Given a step
Scenario: second scenario
Given a step
@skip
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And there should be a messages similar to "fixtures/skipped-third-scenario.ndjson"

Scenario: all scenario skipped
Given a file named "cypress/e2e/a.feature" with:
"""
Feature: a feature
@skip
Scenario: first scenario
Given a step
@skip
Scenario: second scenario
Given a step
@skip
Scenario: third scenario
Given a step
"""
And a file named "cypress/support/step_definitions/steps.js" with:
"""
const { Given } = require("@badeball/cypress-cucumber-preprocessor");
Given("a step", function () {});
"""
When I run cypress
Then it passes
And there should be a messages similar to "fixtures/skipped-all-scenarios.ndjson"

Rule: failing Cypress hooks outside of the plugins control should discontinue the report
Scenario: failing before hook
Given a file named "cypress/e2e/a.feature" with:
Expand Down
23 changes: 23 additions & 0 deletions features/step_definitions/html_steps.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,3 +100,26 @@ Then(
assert(AccordionItemPanel);
}
);

Then(
"the HTML report should display {int} {string} scenario(s)",
async function (this: ICustomWorld, n: number, status: string) {
const dom = await JSDOM.fromFile(
path.join(this.tmpDir, "cucumber-report.html"),
{ runScripts: "dangerously" }
);

const li = await findByText(
dom.window.document.documentElement,
new RegExp(`\\d+ ${status}`),
{
selector: "li",
}
);

// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
const actual = parseInt(li.textContent!, 10);

assert.equal(actual, n);
}
);
Loading

0 comments on commit 1d9ce1c

Please sign in to comment.