Cucumber report bug on retried test not reporting

[camanderson]

Current behavior

Currently, the willBeRetried flag is set with this line:

willBeRetried: currentRetry < retries,

however if a test is retried, but succeeds before currentRetry is >= retries that test will not be reported in the cucumber-reports.json and look like this:
[]

Desired behavior

The desired behavior is to have the retried test be reported in the cucumber-reports.json regardless of how many retries are remaining. I believe the fix would be as simple as changing the referenced line 832 in create-tests.ts to this:

willBeRetried: this.currentTest?.state === "passed" ? false : currentRetry < retries

this should result in retried tests being added to the cucumber-report.json looking like this:

[
  {
    "description": "",
    "elements": [
      {
        "description": "",
        "id": "retry-test;fail-first-pass-second",
        "keyword": "Scenario",
        "line": 2,
        "name": "Fail first Pass second",
        "steps": [
          {
            "arguments": [],
            "keyword": "Given ",
            "line": 3,
            "name": "it succeeds the second time",
            "match": {
              "location": "not available:0"
            },
            "result": {
              "status": "passed",
              "duration": 9000000
            }
          }
        ],
        "tags": [],
        "type": "scenario"
      }
    ],
    "id": "retry-test",
    "line": 1,
    "keyword": "Feature",
    "name": "Retry test",
    "tags": [],
    "uri": "cypress/integration/retry.feature"
  }
]

Test code to reproduce

Here is some code to reproduce the issue
cypress.config.js swap retries between 1 (good) and 2 (bad) to see the bug

const { defineConfig } = require("cypress");
const createBundler = require("@bahmutov/cypress-esbuild-preprocessor");
const { addCucumberPreprocessorPlugin } = require("@badeball/cypress-cucumber-preprocessor");
const {createEsbuildPlugin} = require("@badeball/cypress-cucumber-preprocessor/esbuild");

module.exports = defineConfig({
  e2e: {
    retries: 1,
    specPattern: "**/*.feature",
    async setupNodeEvents(
      on,
      config
    ){
      // This is required for the preprocessor to be able to generate JSON reports after each run, and more,
      await addCucumberPreprocessorPlugin(on, config);

      on(
        "file:preprocessor",
        createBundler({
          plugins: [createEsbuildPlugin(config)],
        })
      );

      // Make sure to return the config object as it might have been modified by the plugin.
      return config;
    },
  },
});

package.json

{
  "name": "reportingenv",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "cypress open",
    "headless": "cypress run"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@badeball/cypress-cucumber-preprocessor": "^16.0.0",
    "@bahmutov/cypress-esbuild-preprocessor": "^2.2.0",
    "@deepakvishwakarma/cucumber-json-formatter": "^0.0.3",
    "cypress": "^12.8.1",
    "cypress-xpath": "^2.0.1"
  },
  "cypress-cucumber-preprocessor": {
    "nonGlobalStepDefinitions": true,
    "stepDefinitions": [
      "cypress/integration/**/*.js"
    ],
    "filterSpecs": true,
    "omitFiltered": true,
    "json": {
      "enabled": true,
      "output": "reports/cucumber-report.json"
    }
  }
}

steps.js

'use strict';

const { Given } = require('@badeball/cypress-cucumber-preprocessor');

let count = 0;
Given('it succeeds the second time', () => {
  count++;
  if (count === 1) {
    cy.log(`Fail on count ${count}`);
    throw new Error('fail on purpose');
  }
  cy.log(`Succeed on count ${count}`);
});

retry.feature

Feature: Retry test
  Scenario: Fail first Pass second
    Given it succeeds the second time

Versions

• Cypress version: 12.8.1
• Preprocessor version: 16.0.0
• Node version: 16.17.0

Checklist

• I've read the FAQ.
• I've read Instructions for logging issues.
• I'm not using cypress-cucumber-preprocessor@4.3.1 (package name has changed and it is no longer the most recent version, see #689).

[deanhaleem]

Just want to add, that willBeRetried condition for v16 means here in @cucumber/cucumber won't necessarily be false, and thus won't be included in the report.

In v15 when willBeRetried was always false, that actually resulted in both the failure and success being included in the json report, which was also not desired behavior at least for us

[badeball]

I've fixed this with v16.0.1.