Cypress 7.2.0 still runs slower than 6.9.1 (disregarding screen size increase)

[coryhouse]

Current behavior

My tests complete in 7.5 minutes with Cypress 5.6. They take 9.5 minutes with Cypress 6.9.1. They take nearly 29 minutes with Cypress 7. The tests run equally fast when I use Cypress open. But the different times above occur when I use Cypress run. The run command is much slower on Cypress 7 with both Chrome and Electron.

Desired behavior

Cypress tests should run equally fast on all versions, whether I use run or open.

Test code to reproduce

I can't share the code, but it's a medium-sized React app with 66 Cypress tests.

Versions

[bahmutov]

Cypress v7.2.0?

[DannyFeliz]

Which version are you using? In version 7.2.0 they fixed it

See #15853 and #16030

[coryhouse]

Ah I was on 7.1 so will try upgrading and report back. Thanks!

[jennifer-shehane]

There were some performance issues that we addressed in 7.2.0. Please try updating to the latest version of Cypress.

Additionally, if you are running Cypress in a headless browser, the default screen size was increased to 1920x1080 which may introduce extra time. You could try reducing the screen size to the default size prior to 7.0 (1280x720) to see if the performance is related to the increase in screen size. Instructions here.

[coryhouse]

Thanks for the reply @jennifer-shehane! I upgraded to 7.2 and it runs in 15 minutes. That's still twice as slow as 5.6.0, with the viewport set to 1440 x 660 on both versions.

Summary:
5.6.0: 7.5 min
6.9.1: 9.5 min
7.0.1: 29 min
7.2.0: 15 min

These times are all using cypress run, video: false, viewport 1440 x 660. Oddly, cypress open is much faster (only 1.5 mins) on all versions. Why is run so much slower than open?

Regardless, I'm glad to see 7.2 improved things, but I'm going to stick with 5.6 for now since it's still twice as fast with run.

[aminbenselim]

nitpick: Did you upgrade to 7.2.0 or to 7.0.2. The improvements are part of the minor release and would like to know if 7.2.0 actually improves speed from your side

[coryhouse]

@amine-benselim I used to 7.2 for the v7 results I posted above. Sorry, had a typo in that comment which I just fixed.

[jennifer-shehane]

@coryhouse Could you provide a repo that demonstrates the slower performance? It may be particular to the tests written or the application under test that we'd need to track down. Thanks!

[scrumvisualize]

I too agree with @coryhouse when I updated 7.1.0 cypress version and cypress run was facing serious performance issues. Usually my test run completed in 25 minutes and the 7.1.0 version run took more than 70 minutes to complete the test run and I have seen most of my test gets failed. So I have to revert back to 6.6.0 version. But now I wish to go back to older version of cypress, like 5.6.0 as @coryhouse said, but I can't really as many of tests are using cy.intercept() command config setting s like "experimentalSourceRewriting": true,

[DamienCassou]

I confirm that 7.2.0 is very slow for me:

• 6.9.1 : 9 minutes
• 7.2.0 : 24 minutes

[jennifer-shehane]

Need reproducible example

We'll need a reproducible example of the performance issues that show a difference between 6.9.1 and 7.2.0 (comparing the exact same screen size during the run).

We definitely want to address all performance issues, but can't do it without a repo provided to reproduce on our side. Thanks.

[ghost]

I can't share a reproducible example but our test suite is still twice as slow when upgrading from 5.2.0 to 7.2.0. We are using the same screen size, 640x360.

[estefafdez]

Same here with the new version 7.3.0

288 tests:

• version 6.9.1: 1:32:00 h
• version 7.3.0: 2:00:58 h

[redReno]

Same here as well. I'm in the process of checking which operations take the longest, but generally operations involving cy.visit, click(), cy.contains and cy.intercept seem to be affected the most. We're on 6.6.0 and every version after that run significantly slower, especially in our Jenkins environment.

[jennifer-shehane]

I want to be very clear that the screen size is not the viewport size. Setting viewportHeight and viewportWidth will only change the application under test within the Cypress runner. Changing the viewportHeight and viewportWidth will never affect the performance. This is not going to change the screen size so do not change these values.

The screen size represents the virtual machine's screen, the size that screenshots and videos will be taken at. You have to follow the instructions here to change the screen size.

[coryhouse]

Thanks for clarifying @jennifer-shehane! However, the instructions you linked to appear to be out of date, correct?

In this comment, you said the default screen size is now 1920x1080. But the instructions you linked to say the default is 1280x720. Which is right?

[jennifer-shehane]

@coryhouse Yes 😞 I opened a PR to update the docs here: cypress-io/cypress-documentation#3925

[coryhouse]

So @jennifer-shehane would the screen size setting effect speed even if I have screenshots and video disabled?

[michalscreen]

Hey guys,
Unfortunately, I can confirm that 7.3.0 is slower than 6.9.0 (27 min vs 49 min to run our tests).

I have been trying to revert cypress version to 6.9.0 but there seems to be a problem:

Cypress Version: 6.9.0
npm ERR! The Cypress App could not be downloaded.
npm ERR! 
npm ERR! Does your workplace require a proxy to be used to access the Internet? If so, you must configure the HTTP_PROXY environment variable before downloading Cypress. Read more: https://on.cypress.io/proxy-configuration
npm ERR! 
npm ERR! Otherwise, please check network connectivity and try again:
npm ERR! 
npm ERR! ----------
npm ERR! 
npm ERR! URL: https://download.cypress.io/desktop/6.9.0?platform=darwin&arch=x64
npm ERR! Error: Failed downloading the Cypress binary.

It's possible to revert to 6.9.1 tho

[redReno]

@jennifer-shehane I tried changing the headless screen size to the old default and while the tests still are slower it's not as impactful. It's closer to a 10-20% increase in execution time rather than 50-100% as before. Still noticeable, but now it's workable in case we decide to upgrade. But it would be great if it were more straight-forward to change the screen size, with the cypress.json file or an environment variable.

[jennifer-shehane]

@coryhouse We'd like to narrow down the variables for performance issues, if videos and screenshots are disabled, that would help narrow it down potentially.

@redReno I agree it would be much easier to have an easily configurable value.

We'll be looking into the video issues in general in regards to performance soon.

[ottopaulsen]

I am seeing more than 20% increased testing time on a Vue+Vuetify app with 28 tests, when video is turned off. (No snapshots are created.) With video on, the increase is > 50%. I am comparing Cypress 7.3.0 with 6.8.0.

[jennifer-shehane]

@ottopaulsen Have you tried reducing the screen size from the instructions here?

[jennifer-shehane]

@ottopaulsen Or can you provide a reproducible example we can run?

[ottopaulsen]

If I reduce the screen size to 600x800, I get a good effect. Then the video hardly affects the time. However, it is still much slower than 6.8. And 600x800 is not very useful. With 1400x1200 it is still very slow.

Sorry, I cannot share the code.

[sandinmyjoints]

FWIW, changing the screen size back to the pre-7 setting as @jennifer-shehane suggested has made a big difference for us.

[pranavj1001]

The following steps can be used to reproduce the performance issue on cypress 7.2.0

1. You can use kitchen sink examples here. We were able to replicate with actions.spec.js
2. We did not face a performance issue on Mac OS so please make sure you run it on Windows OS
3. Please do not use a powerful Windows machine, instead think of a low power CI level machine
   • We were able to replicate this on an EC2 Windows machine that was a virtual machine and had the following specs
     • CPU: AMD EPYC 7571 (2 virtual cores)
     • RAM: 2GB
4. We ran the following command

   cypress run -b chrome --spec cypress\integration\examples\actions.spec.js --headless
5. Results that we found on the above-mentioned machine are
   • 7.3.0 - chrome 90 - headless - default screen size
     • actions.spec.js took 1 min 33 seconds
   • 7.2.0 - chrome 90 - headless - default screen size
     • actions.spec.js took 1 min 15 seconds
   • 6.8.0 - chrome 90 - headless - default screen size
     • actions.spec.js took 40 seconds

If more information is required then please let me know.

[egrubbs]

I ran these two versions of Cypress on Ubuntu 18.04.

        Cypress package version: 7.3.0                         Cypress package version: 6.9.1
        Cypress binary version: 7.3.0                          Cypress binary version: 6.9.1
        Electron version: 12.0.0-beta.14                       Electron version: 11.3.0
        Bundled Node version: 14.15.1                          Bundled Node version: 12.18.3

Options	Cypress 7.3.0	Cypress 6.9.1
--headless --config video=false	00:31	00:17
--headless --browser chromium --config video=false	00:27	00:18

This is the test that I benchmarked, When I commented out toMatchImageSnapshot(), it had no effect on the time taken.

  it('should fail to sign in', function () {
    cy.log('1. Logging in');
    cy.visit('/', {
      retryOnNetworkFailure: true,
    });
    cy.get('[placeholder="User Name"] input.dx-texteditor-input').type('not-a-real-user');
    cy.get('[placeholder="Password"] input.dx-texteditor-input[type="Password"]').type('mypass');
    cy.contains('.dx-button-text', 'Login').click();
    cy.log('2. Verifying that failed login message is displayed');
    cy.contains('Unable to log in with provided credentials');
    cy.log('3. Comparing snapshot of .dx-card element on login page');
    cy.get('.dx-card').toMatchImageSnapshot({
      threshold: 0.05,
    });
  });

Here is my cypress.json config file:

{
  "chromeWebSecurity": false,
  "video": true,
  "viewportWidth": 1600,
  "viewportHeight": 1200,
  "trashAssetsBeforeRuns": false,
  "defaultCommandTimeout": 15000,
  "execTimeout": 60000,
  "timeout": 8000,
  "taskTimeout": 90000,
  "pageLoadTimeout": 60000,
  "requestTimeout": 9000,
  "responseTimeout": 30000,
  "reporter": "cypress/reporters/nunit",
  "reporterOptions": {
    "output": "cypress-nunit-result.xml"
  },
  "baseUrl": "http://127.0.0.1:4300",
  "restServerUrl": "http://localhost:8300",
  "ignoreTestFiles": [
    "**/__snapshots__/*",
    "**/__image_snapshots__/*"
  ],
  "cypress-plugin-snapshots": {
    "serverPort": 3000
  }
}

[fearhq]

@jennifer-shehane I have experimented with this issue and Cypress 7.5.0. What I have noticed is that it takes significantly more time to load the page with 7.x (tested 7.5.0) than it does with 6.x (tested 6.6.0), when using cypress run

We have vue 2 app, with roughly 40 or so js chunks that get loaded, and although the contents are cached, the network tab says it still takes a few seconds to load these files under 7.x, whereas in 6.6.0 this is done in under a second.

I was able to see this behavior with cypress run --headed and a quick inspect to see the network tab.
It really seems there is a regression in cypress when loading dependencies within a page, which is exacerbated when the page has many small javascript files (real-world projects)

Same behavior under chrome or electron, not a problem with cypress open

[jennifer-shehane]

There's an issue that's been identified concerning cy.visit() taking longer in 7.2.0 versus 7.1.0. The load event takes longer to fire, although this doesn't appear to be for every website. #16671 Perhaps some of the issues mentioned in this thread are related.

It would be helpful to identify exactly where the increased time is being seen at. I think we've been focusing on video performance since that was a large change, but I don't think this has been clearly identified as the cause of a large increase in time in any case.

I also think that the performance increase is likely dependent on some aspect of the test being run, which is why a reproducible example is important.

I'm going to try to recreate the kitchensink example mentioned above in a docker container or Windows VM with reduced resources.

[fearhq]

It really seems like something is slowing down the loading of ressources, and having many small ressources makes things much worse. Perhaps it is possible to create a test case of a simple page with 50 external javascript files?

[jennifer-shehane]

Following through on what @pranavj1001 suggested, I ran the kitchensink actions.spec.js in a Windows VM with limited resources (2CPU / 4gb memory).

Some thoughts in general from this, but it needs more investigation:

1. This is not enough CPU resources to run Cypress with video. From this and other experiments I've run (2CPU/7gb memory), having Cypress run with this setup results in unusable videos. They're all frozen or choppy. It also increases video processing time at the least. I haven't verified that it increases runtime of the specs, but I imagine this is possible since the CPU is maxed.
2. There may be an increase in runtime between 6.8.0 and 7.5.0 (current version), even when reducing the screen size back to pre-7.0 size. It seems like turning off video may reduce the runtime even more (although others mentioned this doesn't matter)? But I'm not seeing a really significant difference with this example. It seems like the screen size increase is the biggest factor.

Browser	Ver	headed?	OS	Cy	runtime	processing	size
Chrome	90	headless	Windows	7.5.0	35	-	video: false
Chrome	90	headless	Windows	7.5.0	41	11secs	1280x1024
Chrome	90	headless	Windows	7.5.0	1:12	36secs	1920x1080
Chrome	90	headless	Windows	7.4.0	1:36	52secs	1920x1080
Chrome	90	headless	Windows	7.3.0	1:29	32secs	1920x1080
Chrome	90	headless	Windows	7.2.0	1:22	36secs	1920x1080
Chrome	90	headless	Windows	7.1.0	1:12	32secs	1920x1080
Chrome	90	headless	Windows	7.0.1	1:32	44secs	1920x1080
Chrome	90	headless	Windows	7.0.0	1:26	43secs	1920x1080
Chrome	90	headless	Windows	6.8.0	35	8 secs	1280x1024
Chrome	90	headless	Windows	6.8.0	38	-	video: false

[jennifer-shehane]

For reference, this is the timing on my Mac (6CPU, 16gb memory) running the same kitchensink actions.spec.js file as above.

Browser	Ver	headed?	OS	Cy	runtime	processing	size
Chrome	90	headless	Windows	7.5.0	15sec	8secs	1280x1024
Chrome	90	headless	Windows	7.5.0	16sec	8secs	1280x1024
Chrome	90	headless	Windows	6.8.0	16sec	4secs	1280x1024
Chrome	90	headless	Windows	6.8.0	15sec	4secs	1280x1024

The optimal time for this specfile to run is ~15secs, with ~8secs video processing time. So even the baseline 6.8.0 runs on the underpowered Windows machine were not as performant as they should be.

I would suggest everyone run their spec files first on a high resources machine, to get a baseline of how long a spec file should run for - then compare this against your CI run. If the CI is running slower, I would add more resources to the machine to see if that improves runtime.

That being said, we are evaluating ways to help in situations where Cypress is run on underpowered machines because the reason behind slow/slower runs is not something that is immediately obvious today or clear to diagnose.

---

Takeaway

We will be reverting the screen size back to 1280x720 in our next breaking release. You can read more about this decision here: #16853. We believe this is the sole contributing factor to increased runtime (outside of this visit issue: #16671).

Need a reproducible example

We need a reproducible example showing clear evidence of a slowdown of runtime/processing time between 6.8.0 and 7.2.0 that has reduced the screen size back to 1280x720.

We'll have to close this issue if this is not provided. We've spent a lot of time running performance metrics and can't find any evidence outside of these observations so far.

[jennifer-shehane]

We've also identified that there's an issue in Chrome 89 regarding video frames that would slow down runs when recording video. This will be addressed. Please follow: #16152

This issue will be closed as there are other issues open now to address the underlying slower runs between 6.9.1 and 7.2.0:

• Decrease screen size back to 1280x720 Make default screen size for headless runs to be 1280x720 #16853
• Chrome 89 frames slowdown Address performance issues with video recording in Chrome 89+/Electron 12+ #16152

Please follow these issues for updates, we're aiming for 8.0 to release these fixes.

[nnelson-cen65665]

Same here cy v6.5.0 tests run 25m update to 7.7.0 tests run 1h 44m on Jenkins CI pipeline. Screen size 1440x900. Angular app.

[jennifer-shehane]

Please upgrade to Cypress 8.0.0 as this has reduced the screen size to 1280x720 and may fix many of your issues.

[nikolas-nelson]

Please upgrade to Cypress 8.0.0 as this has reduced the screen size to 1280x720 and may fix many of your issues.

updated to 8.0.0 but still 2x slower then 6.5.0 with video recording on

[ottopaulsen]

I just upgraded from 6.8 to 8.3, and my performance problems seem to have been solved now. Thanks!