Cypress run hangs for hours on Jenkins with no output, no errors, and no logs

[Muhammadehsan2019]

@jennifer-shehane
@bahmutov
@brian-mann

I noticed that the Cypress run stalls and hangs for hours in Jenkins. The scheduled job runs on Jenkins every day at 8:00 pm for the duration of approximately 30 minutes, but I noticed that this 30 minutes job was running indefinitely for over 12 hours in Jenkins with no errors, and no logs. All I could do is to stop the run and rerun the build.
Please provide a solution for this!!!

Additional Info:

• Operating System: Windows 10
• Cypress Version: 7.0.1
• Browser: Chrome
• Run Mode: Headless
• Chrome Version: 89

[tommasomarchionni]

I resolved the same problem in Azure pipeline, reducing screen size to 1024x720. See Set screen size when running headless for details.
From Cypress 7.0.0 the default screen size is 1980x1080 (before was 1024x720), with this screen size Azure pipeline randomly hangs for hours...

[Muhammadehsan2019]

@tommasomarchionni
Thanks very much for the recommandation. I have changed the screen size and reduced it to 1024x720 but the issue still exists. still hanging with Jenkins.

@jennifer-shehane
@bahmutov
@brian-mann
I am still waiting for your responses, if Cypress dev team can't help then who to reach out? please help!!!

[jennifer-shehane]

Hey @Muhammadehsan2019, can you narrow down what has changed in your project within the time that your tests ran successfully in Jenkins and the time that it began hanging? Did you update Cypress?

Narrowing that down will likely point to the issue.

[Muhammadehsan2019]

@jennifer-shehane
Nothing has changed in the project other than Cypress version. I have just integrated Cypress with Jenkins and it is the first time running Cypress tests in Jenkins. Cypress was updated to 7.0.1

[mtc3bmtc3b]

I'm seeing the same behavior. However, I am on Linux, though also running headless running via Jenkins. This started for us when we moved from 6.8.0 to 7.1.0. We changed from route() to intercept() at the same time.

One thing that I've noticed if that I run top on the build machine when it is stalled, Cypress is still out there, still consuming CPU, and 1 process claims it is using 46g of virtual memory. With video enabled, my CPU usage is substantially higher (about 3x higher) and the behavior is the same. I tried lowering the screen size per @tommasomarchionni, and the behavior is unchanged (and my top is similar to what it showed prior to changing the screen size).

Result: top - 02:12:47 up 8 days, 4:16, 0 users, load average: 0.90, 0.93, 1.01
Tasks: 120 total, 2 running, 118 sleeping, 0 stopped, 0 zombie
%Cpu(s): 21.2 us, 11.2 sy, 0.0 ni, 67.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8009048 total, 288412 free, 1777824 used, 5942812 buff/cache
KiB Swap: 524284 total, 326652 free, 197632 used. 5904888 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20318 user1-+ 20 0 46.8g 504972 49784 R 42.1 6.3 294:15.85 Cypress
20084 user1-+ 20 0 6033328 177808 57256 S 36.8 2.2 191:02.95 Cypress
20246 user1-+ 20 0 606484 64996 46484 S 10.5 0.8 72:43.37 Cypress
4581 user1-+ 20 0 160060 2028 1476 R 5.3 0.0 0:00.07 top

[mtc3bmtc3b]

I turned on Cypress DEBUG to see what it was doing after it should have completed.

I get a bunch of lines like this:
2021-04-22T09:10:48.314Z cypress:server:browsers:electron debugger: sending Page.screencastFrameAck with params { sessionId: 1 }
2021-04-22T09:10:48.314Z cypress:server:browsers:electron debugger: received response to Page.screencastFrameAck: {}

Occasionally, it will dump out some profile data:
2021-04-22T09:10:57.024Z cypress-verbose:server:util:process_profiler all Cypress-launched processes: [
(bunch of profile stuff with CPU usage and whatnot)

And periodically I see:
2021-04-22T09:10:57.987Z engine:socket writing ping packet - expecting pong within 5000ms
2021-04-22T09:10:57.987Z engine:socket sending packet "ping" (undefined)
2021-04-22T09:10:57.987Z engine:socket flushing buffer to transport
2021-04-22T09:10:57.987Z engine:ws writing "2"

Followed after a while by:
2021-04-22T09:11:02.995Z engine:ws closing
2021-04-22T09:11:02.996Z socket.io:client client close with reason ping timeout
2021-04-22T09:11:02.996Z socket.io:socket closing socket - reason ping timeout
2021-04-22T09:11:03.007Z cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 43660 }

It will do that endlessly, never exiting.

The last apparently legitimate output I see looks like this:

2021-04-22T09:10:48.296Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'AttachPlainTextStreamFn' }
2021-04-22T09:10:48.296Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'InterceptResponse' }
Thu, 22 Apr 2021 09:10:48 GMT cypress:net-stubbing:server:intercept-response InterceptResponse { req: { url: '/api/v1/persons?esnUserId=&asOf=2021-04-22' }, request: undefined }
2021-04-22T09:10:48.297Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'PatchExpressSetHeader' }
2021-04-22T09:10:48.297Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'SetInjectionLevel' }
2021-04-22T09:10:48.297Z cypress:server:server-base Getting remote state: { auth: null, props: { port: '443', tld: 'com', domain: 'site' }, origin: 'https://.site.com', strategy: 'http', visiting: false, domainName: 'site.com', fileServer: null }
2021-04-22T09:10:48.297Z cypress:network:cors Parsed URL { port: '443', tld: 'com', domain: 'site' }
2021-04-22T09:10:48.297Z cypress:proxy:http:response-middleware injection levels: { isInitial: false, wantsInjection: false, wantsSecurityRemoved: false }
2021-04-22T09:10:48.298Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'OmitProblematicHeaders' }
2021-04-22T09:10:48.298Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'MaybePreventCaching' }
2021-04-22T09:10:48.298Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'MaybeStripDocumentDomainFeaturePolicy' }
2021-04-22T09:10:48.298Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'CopyCookiesFromIncomingRes' }
2021-04-22T09:10:48.298Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'MaybeSendRedirectToClient' }
2021-04-22T09:10:48.298Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'CopyResponseStatusCode' }
2021-04-22T09:10:48.299Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'ClearCyInitialCookie' }
2021-04-22T09:10:48.299Z cypress:server:server-base Getting remote state: { auth: null, props: { port: '443', tld: 'com', domain: 'site' }, origin: 'https://****.site.com', strategy: 'http', visiting: false, domainName: 'site.com', fileServer: null }
2021-04-22T09:10:48.299Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'MaybeEndWithEmptyBody' }
2021-04-22T09:10:48.299Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'MaybeInjectHtml' }
2021-04-22T09:10:48.299Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'MaybeRemoveSecurity' }
2021-04-22T09:10:48.299Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'GzipBody' }
2021-04-22T09:10:48.299Z cypress:proxy:http Running middleware { stage: 'IncomingResponse', middlewareName: 'SendResponseBodyToClient' }
Thu, 22 Apr 2021 09:10:48 GMT compression gzip compression

[jennifer-shehane]

@Muhammadehsan2019 What Cypress version were you previously using? Is there still hanging when using the old cy.route() in the new version?

We are working on resolving a few performance issues introduced in the last few versions, so it's likely concerned with these performance issues.

[Muhammadehsan2019]

@jennifer-shehane
I was using 6.9.0, and when locally running Cypress on headless mode there was no issue with it, but when we moved to 7.0.0 and above the hang started even locally not only in Jenkins. Running the test on headed mode from the command line is very slow compared to running from the test runner.
I use cy.Intercept() since cy.route() is deprecated in the 6.0.0 and above...

[jelyck]

Similar issue here. Jobs hangs for hours in middle of test case without any output. As far as I can tell the issue started to appear after upgrading cypress from 7.0.0 to cypress 7.1.0. In Jenkins we have linux slaves running docker cypress/included:7.1.0. The test case that hangs is using cy.intercept to wait and check API response content. After going back to 7.0.0 it works fine.

[jennifer-shehane]

There are some performance issues introduced in recent versions of Cypress, please see those issues for relevant updates. We're working on them.

• [7.0.0] Performance regression from 6.8 (over 3x slower in our test suite) #15853
• Performance regression in CI from 6.6 -> 6.7 (and 6.8/7.0) #15779
• Address performance issues with video recording in Chrome 89+/Electron 12+ #16152
• Cypress hangs in Azure when running headless in 7.0+ #15973
• Test Timing out on Github Actions - Cypress 7.0  #15990

If you are running Cypress in a headless browser, please try reducing the screen size to the default size prior to 7.0 (1280x720) to see if the new issues are related to the increase in screen size. Instructions here.

If that doesn't resolve the issue, I suggest downgrading to the last known version of Cypress that worked well.

We'll be closing this issue as there are no reproducible actions. For now we need to focus on the performance issues where we're given clear reproducible examples.