Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AO3-6646 Fix flaky works tests #4672

Merged
merged 1 commit into from
Dec 1, 2023

Conversation

Bilka2
Copy link
Contributor

@Bilka2 Bilka2 commented Nov 26, 2023

Issue

https://otwarchive.atlassian.net/browse/AO3-6646

Intermittent test failures in features/works:

Example run, also listed in wiki: features/works/work_share.feature:68 # Scenario: Share URL should not be used for post-login redirect

expected: "/works/1"
     got: "/users/MadUser" (using ==) (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:263:in `/^(?:|I )should be on (.+)$/'
features/works/work_share.feature:84:in `Then I should be on the "Blabla" work page'

Example run, also listed in wiki: features/works/work_browse.feature:120 # Scenario: Kudos link from from work browsing leads to full work page

expected to find text "Kudos: 1" in "Awesome Work by y4dcnk5s (zd5t42d31) Fandoms: Testing Not Rated No Archive Warnings Apply No category Complete Work 25 Nov 2023 Tags No Archive Warnings Apply Language: English Words: 10 Chapters: 2/2 Hits: 0" (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:123:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `block in with_scope'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:122:in `/^(?:|I )should see "([^"]*)"(?: within "([^"]*)")?$/'
./features/step_definitions/work_steps.rb:762:in `"I should see {string} within the work blurb of {string}"'
features/works/work_browse.feature:131:in `Then I should see "Kudos: 1" within the work blurb of "Awesome Work"'

Example run: features/works/work_browse.feature:136 # Scenario: Comments link from from work browsing leads to full work page

expected to find text "Comments: 1" in "Awesome Work by glzm1uvl (r4afnswf3) Fandoms: Testing Not Rated No Archive Warnings Apply No category Complete Work 25 Nov 2023 Tags No Archive Warnings Apply Language: English Words: 10 Chapters: 2/2 Hits: 0" (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:123:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `block in with_scope'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:122:in `/^(?:|I )should see "([^"]*)"(?: within "([^"]*)")?$/'
./features/step_definitions/work_steps.rb:762:in `"I should see {string} within the work blurb of {string}"'
features/works/work_browse.feature:146:in `Then I should see "Comments: 1" within the work blurb of "Awesome Work"'

Example run: features/works/work_edit.feature:7 # Scenario: You can't edit a work unless you're logged in and it's your work

expected not to find text "testuser," in "Main Content While we've done our best to make the core functionality of this site accessible without javascript, it will work better with it enabled. Please consider turning it on! Example Archive beta User Navigation Hi, testuser! My Dashboard My Works My Bookmarks My History My Preferences Post New Work Import Work Log Out Site Navigation Fandoms All Fandoms TV Shows Uncategorized Fandoms Browse Works Bookmarks Tags Collections Search Works Bookmarks Tags People About About Us News FAQ Wrangling Guidelines Donate or Volunteer Search Works Work Search: tip: lex m/m (mature OR explicit) Work was successfully updated. Actions Add Chapter Edit Edit Tags Entire Work Next Chapter → Chapter Index Chapter Index 1. Chapter 1 2. Chapter 2 Full-page index Edit Bookmark Cancel Bookmark Comments Share Download AZW3 EPUB MOBI PDF HTML Work Header Rating: Not Rated Archive Warning: No Warnings Fandom: first fandom Character: Persone A Additional Tags: new tag Language: English Stats: Published:2010-04-30Completed:2023-11-25Words:7Chapters:2/2Kudos:4Bookmarks:1Hits:1,001 First work testuser, testymctesty (testuser) Chapter Management Edit Chapter Chapter 1 Chapter by testuser Chapter Text first chapter new content Actions ↑ Top Next Chapter → Edit Bookmark Kudos 4 guests left kudos on this work! Comments Post Comment Comment as testuser testy tester_pseud testymctesty (Plain text with limited HTML ?) Comment 10000 characters left Bookmark Bookmark × testuser testy tester_pseud testymctesty save a bookmark! Write Comments Notes The creator's summary is added automatically. Plain text with limited HTML ? 5000 characters left Your tags The creator's tags are added automatically. Comma separated, 100 characters per tag Add to collections Choose Type and Post Private bookmark Rec Footer About the Archive Site Map Diversity Statement Terms of Service DMCA Policy Contact Us Policy Questions & Abuse Reports Technical Support & Feedback Development Known Issues GPL by the OTW" (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:182:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:180:in `/^(?:|I )should not see "([^"]*)"(?: within "([^"]*)")?$/'
features/works/work_edit.feature:82:in `And I should not see "testuser,"'

Example run: features/works/work_edit_multiple.feature:7 # Scenario: I can delete multiple works at once

expected not to find text "Glorious" in "Main Content While we've done our best to make the core functionality of this site accessible without javascript, it will work better with it enabled. Please consider turning it on! Example Archive beta User Navigation Hi, author! My Dashboard My History My Preferences Post New Work Import Work Log Out Site Navigation Fandoms All Fandoms Uncategorized Fandoms Browse Works Bookmarks Tags Collections Search Works Bookmarks Tags People About About Us News FAQ Wrangling Guidelines Donate or Volunteer Search Works Work Search: tip: \"sherlock (tv)\" m/m NOT \"sherlock holmes/john watson\" Choices Dashboard Profile Preferences Skins Pitch Works (3) Drafts (0) Series (0) Bookmarks (0) Collections (0) Catch Inbox (0) Statistics History Subscriptions Switch Sign-ups (0) Assignments (0) Claims (0) Related Works (0) Gifts (0) 3 Works by author Navigation and Actions Edit Works Works in Collections Works Filters Listing Works Lovely by author Fandoms: Steven Universe Not Rated No Archive Warnings Apply Other Complete Work 25 Nov 2023 Tags No Archive Warnings ApplyScary tag Language: English Words: 6 Chapters: 1/1 Hits: 0 Author Actions Edit Edit Tags Excellent by author Fandoms: No rating Choose Not To Use Archive Warnings No category Complete Work 25 Nov 2023 Tags Language: English Words: 6 Chapters: 0/1 Hits: 0 Author Actions Edit Edit Tags Add Chapter Glorious by author Fandoms: No rating Choose Not To Use Archive Warnings No category Complete Work 25 Nov 2023 Tags Language: English Words: 6 Chapters: 0/1 Hits: 0 Author Actions Edit Edit Tags Add Chapter Filters Filter results: Submit Sort by Author Title Date Posted Date Updated Word Count Hits Kudos Comments Bookmarks Include ? Include Ratings Not Rated (3) Include Warnings No Archive Warnings Apply (3) Include Categories Other (3) Include Fandoms Include Characters Include Relationships Include Additional Tags Other tags to include Exclude ? Exclude Ratings Not Rated (3) Exclude Warnings No Archive Warnings Apply (3) Exclude Categories Other (3) Exclude Fandoms Exclude Characters Exclude Relationships Exclude Additional Tags Other tags to exclude More Options Crossovers Include crossovers Exclude crossovers Show only crossovers Completion Status All works Complete works only Works in progress only Word Count From To Date Updated From To Search within results ? Language English Submit Clear Filters Top of Work Index Footer About the Archive Site Map Diversity Statement Terms of Service DMCA Policy Contact Us Policy Questions & Abuse Reports Technical Support & Feedback Development Known Issues GPL by the OTW" (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:182:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:180:in `/^(?:|I )should not see "([^"]*)"(?: within "([^"]*)")?$/'
features/works/work_edit_multiple.feature:27:in `Then I should not see "Glorious"'

Example run: features/works/work_browse.feature:11 # Scenario: If works in a listing exceed the maximum search result count,

expected to find text "3 - 4 of 5 Works" in "Main Content While we've done our best to make the core functionality of this site accessible without javascript, it will work better with it enabled. Please consider turning it on! Example Archive beta User Navigation Hi, testuser! My Dashboard My History My Preferences Post New Work Import Work Log Out Site Navigation Fandoms All Fandoms Uncategorized Fandoms Browse Works Bookmarks Tags Collections Search Works Bookmarks Tags People About About Us News FAQ Wrangling Guidelines Donate or Volunteer Search Works Work Search: tip: \"uchiha sasuke/uzumaki naruto\" angst kudos>10 3 - 4 of 4 Works in Aggressive Retsuko Navigation and Actions Works Bookmarks Filters RSS Feed Pages Navigation ← Previous 1 2 Next → Listing Works Whatever 2 by testuser Fandoms: Aggressive Retsuko Not Rated No Archive Warnings Apply Other Complete Work 25 Nov 2023 Tags No Archive Warnings ApplyScary tag Language: English Words: 6 Chapters: 1/1 Hits: 0 Author Actions Edit Edit Tags Whatever 1 by testuser Fandoms: Aggressive Retsuko Not Rated No Archive Warnings Apply Other Complete Work 25 Nov 2023 Tags No Archive Warnings ApplyScary tag Language: English Words: 6 Chapters: 1/1 Hits: 0 Author Actions Edit Edit Tags Filters Filter results: Submit Sort by Author Title Date Posted Date Updated Word Count Hits Kudos Comments Bookmarks Include ? Include Ratings Not Rated (4) Include Warnings No Archive Warnings Apply (4) Include Categories Other (4) Include Fandoms Aggressive Retsuko (4) Include Characters Include Relationships Include Additional Tags Other tags to include Exclude ? Exclude Ratings Not Rated (4) Exclude Warnings No Archive Warnings Apply (4) Exclude Categories Other (4) Exclude Fandoms Aggressive Retsuko (4) Exclude Characters Exclude Relationships Exclude Additional Tags Other tags to exclude More Options Crossovers Include crossovers Exclude crossovers Show only crossovers Completion Status All works Complete works only Works in progress only Word Count From To Date Updated From To Search within results ? Language English Submit Clear Filters Top of Work Index Pages Navigation ← Previous 1 2 Next → Footer About the Archive Site Map Diversity Statement Terms of Service DMCA Policy Contact Us Policy Questions & Abuse Reports Technical Support & Feedback Development Known Issues GPL by the OTW" (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:123:in `block (2 levels) in <top (required)>'
./features/step_definitions/web_steps.rb:6:in `with_scope'
./features/step_definitions/web_steps.rb:122:in `/^(?:|I )should see "([^"]*)"(?: within "([^"]*)")?$/'
features/works/work_browse.feature:32:in `Then I should see "3 - 4 of 5 Works"'

Purpose

Fix flaky works tests by adding delays in a bunch of places to make sure the caches are expired and in one case, works are posted in the correct order.

For the Share URL should not be used for post-login redirect test, the failures are not obviously related to caching. It's checking that it redirects to a work URL, however it redirects to the user page instead. If I manually test on production, it also redirects to the user page. Based on the test name, #3815 (comment) and https://otwarchive.atlassian.net/browse/AO3-5907?focusedCommentId=356446, this is acceptable. The intention is to check that it doesn't redirect to the share page. So I modified the test to check that it's not the share URL, without caring for which actual URL (work or user) it ends up being.

Testing Instructions

No manual QA required. 50 runs of the tests all passing: once and again.

Credit

Bilka (he/him)

@github-actions github-actions bot added Scope: Tests Only Only changes automated tests or test configuration Awaiting Review labels Nov 26, 2023
@sarken sarken changed the title Fix flaky works tests AO3-6646 Fix flaky works tests Nov 27, 2023
@sarken sarken merged commit 68ed2eb into otwcode:master Dec 1, 2023
25 checks passed
@Bilka2 Bilka2 deleted the intermittent-failure-works-feature branch December 1, 2023 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Reviewed: Ready to Merge Scope: Tests Only Only changes automated tests or test configuration
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants