Cypress 14 CLI Mode: Element hanging

[Ognengineer]

Current behavior

When running Cypress tests in CLI mode for version 14.0.0, there are recurring issues with elements not being found within the specified timeout period. This results in AssertionError messages indicating that certain elements cannot be located in the DOM or should not be visible.

The problem appears to be related to timing or rendering issues when executing tests in CLI mode, which may differ from the behaviour observed in the interactive mode. This inconsistency suggests there could be underlying issues with how elements are rendered or how the application responds to commands in this mode.
Note: that this is not the case with Cypress 13.x.x

I would need additional investigation to identify the root cause and potential solutions for these element not found or hanging errors. @jennifer-shehane attention to this matter would be greatly appreciated.

linking the comment on the mentioned issue #25806 (comment)

Desired behavior

All elements that are supposed to be present in the DOM should be found and interactable within the specified timeout period. This includes elements like buttons, icons, and input fields that are essential for the tests to proceed.

Test code to reproduce

https://github.com/Ognengineer/Test-Cypress-14-pre-release

Cypress Version

https://cdn.cypress.io/beta/npm/14.0.0/linux-x64/document-domain-binary-ce19f25e1b0b5cf626535b31894c4adc377bee08/cypress.tgz

Node version

18.20.1

Operating System

Azure agent running on linux

Debug Logs

Other

No response

[jennifer-shehane]

@Ognengineer Thanks for opening the issue! I've been on the lookout for this since you mentioned it. We'll investigate.

[AtofStryker]

@Ognengineer I cloned your reproduction and gave the tests both a run in open mode with Electron. I am getting the same result in 13.17.0. Is there something else I need to be doing?

[Ognengineer]

Tbh I quickly created this based of some failed test on the real project which I can't share. For me in the Open mode the shared repo test pass, where in the cli --headed the test fail, tested only with cy14. Currently, on the project we are on cypress 13.16.1 binaries build with the fix for same origin match for cookie jar. Let me test it with 13.17.0 and come back to you with the outcome.

Edit: I will need to do more investigation I guess however sharing the outcome of the tests from the two versions.

[mschile]

Hi @Ognengineer 👋🏼, would you be willing to try a pre-release binary from this commit to see if it fixes your issue? Thanks!

[Ognengineer]

Hi @Ognengineer 👋🏼, would you be willing to try a pre-release binary from this commit to see if it fixes your issue? Thanks!

Hi @mschile, unfortunately, the issue is still there with the suggested build, and now suspect that it might be memory-related. I'm super busy on the project and privately atm and unable to do some investigation and debugging.

[MetRonnie]

I think we are also seeing this with our menu.cy.js test here: https://github.com/cylc/cylc-ui/actions/runs/12961795106/job/36157517061?pr=2047#step:4:824

With the Cypress 14 upgrade, the Electron tests now take 3 times as long as before (also 3x longer than Chrome which seems fine on Cypress 14)

Run time comparison Cypress 13 vs 14

Spec	Cypress 13.x	Cypress 14.0
alert.cy.js	00:03	00:05
analysis.cy.js	00:23	00:53
aotf.cy.js	00:08	00:48
dashboard.cy.js	00:05	00:05
drawer.cy.js	00:03	00:03
editRuntimeForm.cy.js	00:09	00:25
gantt.cy.js	00:05	00:10
graph.cy.js	00:17	01:02
graphiql.cy.js	00:02	00:02
gscan.cy.js	00:10	00:33
header.cy.js	00:02	00:12
info.cy.js	00:01	00:01
layout.cy.js	00:01	00:01
log.cy.js	00:10	00:27
menu.cy.js	00:07	00:48
mutation.cy.js	00:10	00:51
table.cy.js	00:09	00:34
toolbar.cy.js	00:04	00:12
tree.cy.js	00:13	00:16
url.cy.js	00:01	00:01
userprofile.cy.js	00:08	00:27
workflowsTable.cy.js	00:02	00:05
workflowservice.cy.js	00:05	00:07
workspace.cy.js	00:08	00:38
All specs passed!	02:59	08:57

[ryanthemanuel]

@Ognengineer we fixed a few visibility issues in 14.0.1. I am going to go ahead and close this as resolved. If you are still having issues, please reopen this issue or create a new issue.

@MetRonnie I have verified that there is a separate issue involving your dramatic performance degradation. I will open a separate issue for that. It appears to be related to the Electron version we are using. Specifically it seems like v-menu's are taking a lot longer to dismiss on linux and headless.

[Ognengineer]

@ryanthemanuel you are right, I've just done a run with --browser chrome and all the tests were passing again. I am happy that we know the root cause