Flaky spec: Unable to find visible select box "residence_document_type" that is not disabled

[bertocq]

Warning!

This spec probably only fails between 00:00 and 02:00 am Madrid time

What

Tests that fail randomly are called "flakies", this one seems to be one:

Randomized seed: 63774

Failure:

    2) Residence Assigned officers Document number is copied from the census API
     Failure/Error: select 'DNI', from: 'residence_document_type'
     Capybara::ElementNotFound:
       Unable to find visible select box "residence_document_type" that is not disabled
     # ./spec/features/officing/residence_spec.rb:54:in `block (3 levels) in <top (required)>'
  3) Residence Assigned officers Verify voter
     Failure/Error: select 'DNI', from: 'residence_document_type'
     Capybara::ElementNotFound:
       Unable to find visible select box "residence_document_type" that is not disabled
     # ./spec/features/officing/residence_spec.rb:40:in `block (3 levels) in <top (required)>'
  4) Residence Assigned officers Error on Census (document number)
     Failure/Error: select 'DNI', from: 'residence_document_type'
     Capybara::ElementNotFound:
       Unable to find visible select box "residence_document_type" that is not disabled
     # ./spec/features/officing/residence_spec.rb:80:in `block (3 levels) in <top (required)>'
  6) Residence Assigned officers Error on Census (year of birth)
     Failure/Error: select 'DNI', from: 'residence_document_type'
     Capybara::ElementNotFound:
       Unable to find visible select box "residence_document_type" that is not disabled
     # ./spec/features/officing/residence_spec.rb:101:in `block (3 levels) in <top (required)>'

How

• Explain why the test is flaky, or under which conditions/scenario it fails randomly
• Explain why your PR fixes it
• Create a backport PR to consul/consul when the fixing PR is approved

Tips for flaky hunting

Random values issues

If the problem comes from randomly generated values, running multiple times a single spec could help you reproduce the failure by running at your command line:

for run in {1..10}
do
  bin/rspec ./spec/features/budgets/investments_spec.rb:256
done

You can also try running a single spec in Travis:
Add option :focus to the spec and push your branch to Github, for example:

scenario 'Show', :focus do

But remember to remove that :focus changes afterwards when submitting your PR changes!

Test order issues

Running specs in the order they failed may discover that the problem is that a previous test sets an state in the test environment that makes our flaky fail/pass. Tests should be independent from the rest.

After executing rspec you can see the seed used, add it as an option to rspec, for example:
bin/rspec --seed 55638 (check Randomized seed value at beginning of issue)

Other things to watch for

• Time related issues (current time, two time or date comparisons with miliseconds/time when its not needed)
• https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests

[raul-fuentes]

I'll do this one

[voodoorai2000]

This seems to be a time zone related issue, it is still happening in a build ran at ~01:00
https://travis-ci.org/AyuntamientoMadrid/consul/jobs/362377792#L1609

[raul-fuentes]

I check the time zone issue and do the backport

[raul-fuentes]

This backport is not needed, this code is already meged into master

[javierv]

@raul-fuentes I'm very sorry I hadn't noticed this issue was already open! These flaky specs should be fixed by consul#2712, which was originally opened to fix consul#2520 and consul#2521.

[javierm]

All these failures have happened again on a build run at ~01:00 A.M.: https://travis-ci.org/javierm/consul/jobs/422313695

However, they now a different error message:

1) Residence Assigned officers Error on Census (year of birth)
     Failure/Error:
       expect(page).to have_selector('#new_residence .small-12.medium-6',
                                     text: 'Document type')
       expected to find visible css "#new_residence .small-12.medium-6" with text "Document type" but there were no matches
     # ./spec/features/officing/residence_spec.rb:125:in `block (3 levels) in <top (required)>'
  2) Residence Assigned officers Document number is copied from the census API
     Failure/Error:
       expect(page).to have_selector('#new_residence .small-12.medium-6',
                                     text: 'Document type')
       expected to find visible css "#new_residence .small-12.medium-6" with text "Document type" but there were no matches
     # ./spec/features/officing/residence_spec.rb:64:in `block (3 levels) in <top (required)>'
  3) Residence Assigned officers Error on Census (document number)
     Failure/Error:
       expect(page).to have_selector('#new_residence .small-12.medium-6',
                                     text: 'Document type')
       expected to find visible css "#new_residence .small-12.medium-6" with text "Document type" but there were no matches
     # ./spec/features/officing/residence_spec.rb:99:in `block (3 levels) in <top (required)>'
  4) Residence Assigned officers Verify voter
     Failure/Error:
       expect(page).to have_selector('#new_residence .small-12.medium-6',
                                     text: 'Document type')
       expected to find visible css "#new_residence .small-12.medium-6" with text "Document type" but there were no matches
     # ./spec/features/officing/residence_spec.rb:45:in `block (3 levels) in <top (required)>'
  5) Residence Assigned officers Error on verify
     Failure/Error:
       within("#new_residence") do
         click_button "Validate document"
       end
     Capybara::ElementNotFound:
       Unable to find visible css "#new_residence"
     # ./spec/features/officing/residence_spec.rb:86:in `block (3 levels) in <top (required)>'

[javierm]

Closed via consul#2712 and #1625.