Some tests flake only if test runner's browser loses focus (or run headlessly)

[blunderdome]

Current behavior:

We have a number of tests which:

• Always pass when run in Chrome 76 with the browser in focus the whole time
• Always pass when run in Electron 61 with the browser in focus the whole time
• Often fail when run in Chrome 76 without the browser in focus
• Often fail when run in Electron 61 without the browser in focus
• Often fail when run headless

Desired behavior:

Browser does not need to stay in focus; tests also pass headlessly.

Steps to reproduce: (app code and test code)

it('is just an example', function () {
  cy.visit('https://noredink.com/logout')
  cy.contains('Log In').click()
  cy.findField('Email or username').type('example_user')
  cy.findField('Password').type('easypassword')
  cy.contains('button', 'Log in').click()
})

Here is a simple case - logging into our production app. This passes if you keep the browser in focus but sometimes fails if it loses focus (or is run headlessly):

Video: in focus and passing

Video: out of focus and failing

We have other examples but they involve the admin side of our site and we probably shouldn't share them here. If you need more examples, let me know and we'll try to find something else that's appropriate to share.

Versions

Cypress: 3.4.1
OS: macOS Mojave 10.14.5
Browser(s): Chrome 76 and Electron 61

Other

#1892 may be relevant.

[jennifer-shehane]

This has been something @bkucera has been working on. It is not an easy issue to solve.

Thanks for providing a reproducible issue. I wasn't able to reproduce it in the few short times I ran this - could you explain what 'often fails' looks like? Do we have to run the test 5 times to see the error? 100 times?

Test code:

it('is just an example', function () {
  cy.visit('https://noredink.com/logout')
  cy.contains('Log In').click()
  cy.get('#Nri-Ui-TextInput-Email-or-username').type('example_user')
  cy.get("#Nri-Ui-TextInput-Password").type('easypassword')
  cy.contains('button', 'Log in').click()
})

[blunderdome]

Hmm, I'm not sure. For me it fails almost every time, but a colleague said it was working for him, so there may be something about our setup or what we're doing that isn't exactly the same.

The tiny example I gave runs pretty quickly, so if you don't switch focus away from the browser right away, it will already have passed. You could have it wait for a few seconds if you want to make sure you've switched focus away before it gets to that login menu.

Let me know if there's anything else about my browser, computer, or config that would be helpful to share ... can also jump on a call with someone if it would be helpful to see it live.

[vesper8]

not sure if related but I am also noticing that tests seem to randomly fail when running headless but run perfectly and consistently when run with the --headed flag. And it's not the same tests that randomly fail, it's quite random, but only happens when running headless.

not to mention that tests complete slower (sometimes MUCH slower) when running headless

[kuceb]

@blunderdome I think your cy.findField command must be sensitive to window focus, and that's the reason your test fails. We also can't run your code for that reason. Is that command from https://github.com/testing-library/cypress-testing-library ?

try using the code @jennifer-shehane wrote, and see if the issue goes away.

[fr0]

I am experiencing something like this related to the focusout event. If the browser window has focus while the test is running, it passes. Otherwise, it fails.

Specifically, I have a popup that is hidden when it loses focus (when it receives focusout). Is there a known workaround for this?

[kuceb]

@fr0 this should have been fixed in 3.4.1, could you give a reproducible?

[fr0]

@fr0 this should have been fixed in 3.4.1, could you give a reproducible?

Sure, here you go: https://github.com/fr0/cypress-focus-test

[ginna-baker]

This is a big issue for us testing our accessibility features, particular the "Skip to Main Content" feature in which the button is only visible on focus.

[jennifer-shehane]

Another example, this UI also seems to have some logic listening to some sort of 'focus' event.

failing spec:

it("Navigates to help article with mobile header menu", () => {
  cy.viewport("iphone-x");
  cy.visit("https://help.doterra.com/");
  cy.get('[data-aura-class="cHC_StartingCategories"]', { timeout: 25000 });
  cy.get('[data-aura-class="cDT_HeaderMobileMenu"]')
    .click();
  cy.get('.dtds-chevron-arrow-right[data-value="section-Help Mobile"]')
    .click();
  // this assertion fails if the window is not focused
  cy.get('.mobile-menu-carousel').should('be.visible')
});

[boda-sh]

Same issue here with the .click() method, only the error message is different and this problem doesn't exist with Electron 78

Behaviour

• Always pass when run in Chrome 80 with the browser in focus the whole time
• Always pass when run in Electron 78 with the browser in focus the whole time
• Always pass when run in Electron 78 without the browser in focus
• Often fail when run in Chrome 80 without the browser in focus
• Often fail when run in CI/headless mode

Steps to reproduce

Simplified code with problem

• Codesandbox link

Cypress test code

describe("Cypress fails when not in focus", () => {
  it("Try to click on the modal text button", () => {
	// deployed from codesandbox
    cy.visit("https://csb-lu86t-40qtfxquh.now.sh/");

    // gives us time to minimize (de-focus) the Cypress window
    cy.wait(5000);

    cy.get("[data-cy=show-modal]").click();
    
    // error here!
    cy.get("[data-cy=modal-text]").click();
  });
});

Error message (only with Chrome 80 not in focus or CI/headless)

CypressError: Timed out retrying: cy.click() failed because this element is not
visible:

<button data-cy="modal-text">i used ...</button>

This element '<button>' is not visible because its ancestor has 'position: fixed'
CSS property and it is overflowed by other elements. How about scrolling to the
element with cy.scrollIntoView()?

Fix this problem, or use {force: true} to disable error checking.

https://on.cypress.io/element-cannot-be-interacted-with

Versions
Cypress: 3.8.2
OS: macOS Catalina 10.15.4
Browser(s): Chrome 80 and Electron 78

[einfallstoll]

@bodazhao I have one test case which often fails in CI/headless with the same setup as you do:

• Select element, click it to show modal
• Select element within modal, click it
• Error, because it's invisible

[juanvillegas]

Is there any way to approach this? We are having issues with native checkboxes, if the window doesn't have focus the checkboxes aren't found by the cy.get() command.

[cmraible]

I've been experiencing similar "flakiness" in my tests, only when the window is in the background or minimized. The tests that fail all assert on visibility or click elements like Modals or Dropdowns, and they fail with an error stating that the element has opacity set to 0 (and thus is not visible). This seems to jive with the other examples in the thread, which all seem to be Modals/Dropdowns or similar.

From Mozilla:

requestAnimationFrame() calls are paused in most browsers when running in background tabs or hidden <iframe>s in order to improve performance and battery life.

Note that the Cypress test runner embeds the application being tested in an iframe, so the above optimization would apply here.

I suspect that Chrome is pausing/throttling calls to requestAnimationFrame() when the Cypress window is backgrounded/minimized/occluded, and thus not rendering these items.

I'm sorry that I don't have a solution for this, but I'm pretty confident this is at least one of the causes. Chrome also throttles JS timers in background tabs, but there is a flag to disable this, which I tried, but didn't see any improvement.

A note on reproducibility: Chromium just introduced occlusion tracking on Windows for Chrome 87, but has supported it for some time on macOS and some other operating systems. This may cause the iframe on Windows to still be considered "visible" even if it was covered by another window. This may be why @jennifer-shehane couldn't reproduce this in August 2019, assuming she was using a Windows machine. I don't have a windows machine to test, but theoretically you could reproduce by opening a new tab in the test runner to force the runner to the background.

[jennifer-shehane]

• I can't reproduce this example in Chrome 87 on any version of Cypress: Some tests flake only if test runner's browser loses focus (or run headlessly) #5023 (comment)
• I was never able to reproduce this example: Some tests flake only if test runner's browser loses focus (or run headlessly) #5023 (comment)

I have a PR open to disable backgrounding occluded windows in Chrome. This could fix some of these issues for Chrome that people are encountering: #9640

• This is reproducible in 6.1.0 and is not fixed by occluded PR: Some tests flake only if test runner's browser loses focus (or run headlessly) #5023 (comment)
• This is reproducible in 6.1.0 and is not fixed by occluded PR: https://github.com/fr0/cypress-focus-test

As for the comment on requestAnimationFrame, I'd have to research that more. We do already pass some chrome flags concerning throttling

  '--disable-background-timer-throttling',
  '--disable-renderer-backgrounding',
  '--disable-renderer-throttling',

[cmraible]

@jennifer-shehane thanks for the prompt response. I put together a little test repo to:

A) reproduce the failure reliably (on my machine at least)
B) confirm that requestAnimationFrame() is being throttled/paused altogether within Cypress

Instructions to reproduce and demonstrate the requestAnimationFrame() throttling are in the README.

The --disable-backgrounding-occluded-windows flag sounds promising -- maybe that will fix this? Not sure how/if I can test this myself...

Thanks again for all that you do, Cypress is dope 👍

[cmraible]

Chiming in: 6.2.1 fixed this completely for me

Thanks @jennifer-shehane and team

[adamalton]

I've just upgraded to 6.3.0, but unfortunately that didn't fix it for me.

My use case essentially boils down to:

document.addEventListener('keydown', function (event) {
    if (document.querySelector('[data-some-specific-selector]:focus') && event.code === 'Space') {
        doSomething();
    }
});

With a Cypress test which does:

cy.get('[data-some-specific-selector]').focus().type(' ');
// Some kind of assertion to check that doSomething was called

And it runs fine when Cypress stays in the foreground, but if I switch to another application while the test is running then doSomething is never called. When Cypress is run headless it seems to be intermittent.

I hope that's useful. Thank you for looking at this, I appreciate the effort being put into tackling this issue.

[manueledones]

I'm on Windows, Cypress 7.1.0, Electron 89, I'm doing a cy.contains('String').click(); if the electron window has focus it works perfectly, but if it hasn't (i.g I'm focusing my IDE) it doesn't click, no error is logged

Here is the div i'm clicking on:
<div _ngcontent-txt-c47="" tabindex="-1" class="suggestion ng-star-inserted" ng-reflect-ng-class="[object Object]"> String </div>

With cy.contains('String').click({ force: true }); it works even if the electron window is not focused :)

[jennifer-shehane]

We should look at what CDP does when you set "Emulate a focused page" - this seems to have fixed some of our internal issues with focus.

[kevingcfcb88]

Is there any workaround for this? I'm doing a drag & drop feature and when I run the tests in interactive mode with the focus on the tests browser windows, all pass, but if I run with the window minimized or test from console always fail the same tests when I start to draw the line. Below is the test in question:

             cy.get("body")
                .click({force: true})
                .tab()
                .tab()
                .focus()
                .type(" ")
                .blur({force: true})
                .then($item => {
                    cy.get("body")
                        .tab({shift: true})
                        .tab({shift: true})
                        .type(" ")
                        .then($element => {
                            cy.get("svg>line").should("have.length", 1);
                        });
                });

[nyghtly-derek]

I've encountered a similar problem while testing an Angular JS application:

scenario	result
npx cypress run	pass
npx cypress open with focus	pass
npx cypress open without focus	fail

[Mayank-jain34]

Hi Team, is there any update on this. We are also running into the same issue. Thanks!

[charleshimmer]

@nyghtly-derek similar issue for but with React app. We are finding focus is lose causing the app to be in an incorrect state if focus is on another app other than the cypress browser. Running headless sometimes has the same issue for us though.

cypress@8.5.0