Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tests-Only] remove UI tests from expected failures if rerun is successful #37849

Merged
merged 1 commit into from
Aug 26, 2020

Conversation

phil-davis
Copy link
Contributor

Description

https://drone.owncloud.com/owncloud/activity/1410/31/10

--- Failed scenarios:

    /var/www/owncloud/testrunner/apps/activity/tests/acceptance/features/webUIActivityTags/tags.feature:17
    /var/www/owncloud/testrunner/apps/activity/tests/acceptance/features/webUIActivityTags/tags.feature:18
    /var/www/owncloud/testrunner/apps/activity/tests/acceptance/features/webUIActivityTags/tags.feature:19
    /var/www/owncloud/testrunner/apps/activity/tests/acceptance/features/webUIActivityTags/tags.feature:20
    /var/www/owncloud/testrunner/apps/activity/tests/acceptance/features/webUIActivityTags/tags.feature:21
    /var/www/owncloud/testrunner/apps/activity/tests/acceptance/features/webUIActivityTags/tags.feature:22

25 scenarios (19 passed, 6 failed)
234 steps (204 passed, 6 failed, 24 skipped)

Then all those 6 scenarios were rerun and passed.

But the first 5 scenarios were still reported as unexpected failures:

runsh: Total unexpected failed scenarios throughout the test run:
webUIActivityTags/tags.feature:17
webUIActivityTags/tags.feature:18
webUIActivityTags/tags.feature:19
webUIActivityTags/tags.feature:20
webUIActivityTags/tags.feature:21

They were not removed from the UNEXPECTED_FAILED_SCENARIOS array because the value of SUITE_SCENARIO was not being recalculated each time in the rerun-failed-scenarios loop.

Move the calculation of SUITE_SCENARIO so that it is always done in the rerun-failed-scenarios loop.

How Has This Been Tested?

CI

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Database schema changes (next release will require increase of minor version instead of patch)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Technical debt
  • Tests only (no source changes)

Checklist:

  • Code changes
  • Unit tests added
  • Acceptance tests added
  • Documentation ticket raised:
  • Changelog item, see TEMPLATE

@codecov
Copy link

codecov bot commented Aug 26, 2020

Codecov Report

Merging #37849 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master   #37849   +/-   ##
=========================================
  Coverage     64.75%   64.75%           
  Complexity    19403    19403           
=========================================
  Files          1285     1285           
  Lines         75816    75816           
  Branches       1336     1336           
=========================================
  Hits          49096    49096           
  Misses        26326    26326           
  Partials        394      394           
Flag Coverage Δ Complexity Δ
#javascript 54.06% <ø> (ø) 0.00 <ø> (ø)
#phpunit 65.93% <ø> (ø) 19403.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 119f23d...461daa3. Read the comment docs.

@phil-davis phil-davis force-pushed the fix-cleanup-of-failed-scenarios-after-rerun branch from 461daa3 to f840e9d Compare August 26, 2020 07:47
@phil-davis phil-davis merged commit cf6b81a into master Aug 26, 2020
@phil-davis phil-davis deleted the fix-cleanup-of-failed-scenarios-after-rerun branch August 26, 2020 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants