[Tests-Only] Reporting expected failures when running a single feature

[swoichha]

Description

Reporting log when running just a single feature or single scenario that are listed in expected-failure.txt. Previously we used to get the log about scenarios that were actually running and also for scenarios that were not running but were listed in expected-failure.

Related Issue

• Fixes Reporting of unexpected failures when running just a single feature #37815

How Has This Been Tested?

• 🤖

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Database schema changes (next release will require increase of minor version instead of patch)
• Breaking change (fix or feature that would cause existing functionality to change)
• Technical debt
• Tests only (no source changes)

Checklist:

• Code changes
• Unit tests added
• Acceptance tests added
• Documentation ticket raised:
• Changelog item, see TEMPLATE

[phil-davis]

Should this be the other way around? We want to add the to ACTUAL_UNEXPECTED_PASS if it is the BEHAT_FEATURE.

Also,

• I think BEHAT_FEATURE is set to something like tests/acceptance/features/apiComments/comments.feature:42 but the "unexpected" array will have just apiComments/comments.feature:42 in it. See code that calculates SUITE_SCENARIO from the end of a path.
• it needs to not match entries like apiComments/comments.feature:421 - the last * here is probably not needed.

It might be easier to use the bash regex-match operator =~ - then you can match strings exactly at the start or end. See examples in run.sh that use =~

Or actually I think that after extracting the ``apiComments/comments.feature:42` part from BEHAT_FEATURE then you can check for an exact match - no need for "glob *" or regex.

[swoichha]

After BEHAT_FEATURE is set to apiComments/comments.feature:42 I think we can directly check for an exact match using== by removing the first and last * from *"${BEHAT_FEATURE}"* ]]

[codecov]

Codecov Report

Merging #37852 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #37852   +/-   ##
=========================================
  Coverage     64.75%   64.75%           
  Complexity    19403    19403           
=========================================
  Files          1285     1285           
  Lines         75816    75816           
  Branches       1336     1336           
=========================================
  Hits          49094    49094           
  Misses        26328    26328           
  Partials        394      394           

Flag	Coverage Δ	Complexity Δ
#javascript	54.06% <ø> (ø)	0.00 <ø> (ø)
#phpunit	65.93% <ø> (ø)	19403.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c2c8a5f...4e2381d. Read the comment docs.

[kiranparajuli589]

lgtm