Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cypress freezes in Docker when running in Firefox #502

Closed
mj-jackson opened this issue Apr 15, 2021 · 6 comments
Closed

Cypress freezes in Docker when running in Firefox #502

mj-jackson opened this issue Apr 15, 2021 · 6 comments

Comments

@mj-jackson
Copy link

Hi, I was using Cypress 7.0.0 and the docker image cypress/included was working. After updating to 7.1.0 the docker container started to "freeze" on random tests. I tried to downgrade to 7.0.1 = didn't work, downgrading to 7.0.0 made it work again.

I say it "freezed" because there are no outputs to the console anymore and the container never finishes.

After running it with DEBUG=cypress:* the outputs stop at some point.
Every Minute or so it updates and shows "current & mean memory and CPU usage by process group" looking a little like this:

cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 56106 } +22s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │        pids        │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'Firefox'     │      3       │ '816, 1107, 1161'  │   69.54    │     30.24      │  450.29  │    625.48    │   759.52    │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │        '19'        │    0.31    │     10.42      │  226.39  │    226.62    │    288.3    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │ '27, 184, 28, 219' │     0      │       0        │  208.7   │    189.64    │    208.7    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │       '252'        │     0      │      0.44      │  129.18  │    134.32    │   142.04    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │    '1280, 1281'    │     0      │       0        │   3.4    │     3.36     │     3.4     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      11      │        '-'         │   69.85    │     35.98      │ 1017.96  │   1052.79    │   1326.67   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │        pids        │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'Firefox'     │      3       │ '816, 1107, 1161'  │    0.81    │     25.33      │  448.81  │    596.03    │   759.52    │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │        '19'        │    0.18    │      8.96      │  226.8   │    226.64    │    288.3    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │ '27, 184, 28, 219' │     0      │       0        │  208.7   │    192.36    │    208.7    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │       '252'        │     0      │      0.36      │  129.18  │    133.47    │   142.04    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │    '1309, 1310'    │     0      │       0        │   3.39   │     3.36     │     3.4     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      11      │        '-'         │    0.98    │     30.98      │ 1016.88  │   1047.66    │   1326.67   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s

Then after waiting for like 10 Minutes I got this

cypress:server:server-base Got CONNECT request from firefox.settings.services.mozilla.com:443 +6m
  cypress:https-proxy Writing browserSocket connection headers { url: 'firefox.settings.services.mozilla.com:443', headLength: 0, headers: { 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0', 'proxy-connection': 'keep-alive', connection: 'keep-alive', host: 'firefox.settings.services.mozilla.com:443' } } +6m
  cypress:https-proxy Got first head bytes { url: 'firefox.settings.services.mozilla.com:443', head: "\x16\x03\x01\x02\x00\x01\x00\x01�\x03\x03�/\x01��9 �ݘ��\x1E, 3bc}�\x11\x12}.u>��K��`�ڍ\x11\x10�" } +6ms
  cypress:https-proxy Making intercepted connection to 36787 +1ms
  cypress:network:connect successfully connected { opts: { port: 36787, host: 'localhost', getDelayMsForRetry: [Function: getDelayForRetry] }, iteration: 0 } +6m
  cypress:https-proxy received upstreamSocket callback for request { port: 36787, hostname: 'localhost', err: undefined } +5ms
  cypress:server:util:socket_allowed allowing socket { localPort: 53150 } +6m
  cypress:proxy:http Entering stage { stage: 'IncomingRequest' } +6m
  cypress:proxy:http Running middleware { stage: 'IncomingRequest', middlewareName: 'LogRequest' } +0ms
  cypress:proxy:http:request-middleware proxying request { req: { method: 'GET', proxiedUrl: 'https://firefox.settings.services.mozilla.com/v1/buckets/monitor/collections/changes/records?', headers: { host: 'firefox.settings.services.mozilla.com', 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0', accept: '*/*', 'accept-language': 'en-US,en;q=0.5', 'accept-encoding': 'gzip, deflate, br', connection: 'keep-alive' } } } +6m
  cypress:proxy:http Running middleware { stage: 'IncomingRequest', middlewareName: 'MaybeEndRequestWithBufferedResponse' } +1ms
  cypress:proxy:http Running middleware { stage: 'IncomingRequest', middlewareName: 'InterceptRequest' } +0ms
  cypress:net-stubbing:server:intercept-request intercepting request { requestId: 'interceptedRequest230', req: { url: '/v1/buckets/monitor/collections/changes/records?' } } +6m
  cypress:net-stubbing:server:util sending event to driver { eventName: 'before:request', data: { eventId: 'event231', subscription: { eventName: 'before:request', await: true, routeId: '1618506135594-55' }, requestId: 'interceptedRequest230', data: { headers: [Object], url: 'https://firefox.settings.services.mozilla.com/v1/buckets/monitor/collections/changes/records?', method: 'GET', httpVersion: '1.1', body: '' } } } +4ms

After that the current memory and CPU usage is displayed again. Waiting didn't resolve anything. I have to manually stop the container.

I tested for about an hour and I intend to test more, but it seems to be happening with firefox pretty consistently for me. My test with electron was successful (meaning it finished as expected).

Since it will "freeze" on random tests I cannot really give any code to reproduce. Most tests that I saw it freeze didn't do anything crazy. Just some intercepts, visits, shoulds and maybe a type here and there. But I can give some code on how I start cypress in docker.

I basically create a docker image with the following code

FROM cypress/included:7.1.0
USER root
COPY cypress.json /cypress.json
COPY cypress /cypress

Copying the files was done before I was in the project, but mounting the directory via docker run -v didn't have any impact.
After creating the image I run the container
docker run --name cy-e2e imagename --browser firefox --config video=false

@jelyck
Copy link

jelyck commented May 18, 2021

Same behaviour here. Only seen in firefox.

Not working in 7.1.0 or later (just tested 7.3.0). Not sure about 7.0.1.

Latest working revision is 7.0.0.

Nothing random for me.

Env:
Linux
Jenkins
Docker
cypress/included:7.3.0 (using --browser firefox)

Test case:

    it('should login successfully', () => {
      cy.get('[data-cy=account-button]').should('exist').click();
      cy.get('input[name="username"]').clear().type(username);
      cy.get('input[name="password"]').clear().type(password);
  
      cy.intercept({
        method: Cypress.env('endpoints').auth.sessions.method,
        url: Cypress.env('endpoints').auth.sessions.url,
      }).as('session');
  
      cy.intercept({
        method: Cypress.env('endpoints').auth.users.method,
        url: Cypress.env('endpoints').auth.users.url,
      }).as('users');
  
      cy.get('#login').click();
  
      cy.wait(['@session', '@users']).spread((authReqInt, userReqInt) => {
        expect(authReqInt.response.statusCode).to.equal(200);
        expect(authReqInt.response.body).to.not.be.null;
        expect(userReqInt.response.statusCode).to.equal(200);
        expect(userReqInt.response.body).to.not.be.null;
      });
  
      cy.get('[data-cy=welcome-message]').should('exist');
    });

Note: The same code is executed in a previous test which PASS. Second time the code is executed Cypress hangs.

Note2: Works without any issues when running cypress normally outside of docker.

Suspected changes:

Log file (using cypress/included:7.3.0):
docker-firefox-hangs.txt

@jennifer-shehane jennifer-shehane changed the title Cypress freezes in Docker Cypress freezes in Docker when running in Firefox May 18, 2021
@jelyck
Copy link

jelyck commented May 19, 2021

Some more information:

Name + Tag Base image
cypress/included:7.0.0 cypress/browsers:node12.18.3-chrome89-ff86
cypress/included:7.0.1 cypress/browsers:node14.16.0-chrome89-ff77
cypress/included:7.1.0 cypress/browsers:node14.16.0-chrome89-ff77
cypress/included:7.2.0 cypress/browsers:node14.16.0-chrome89-ff77
cypress/included:7.3.0 cypress/browsers:node14.16.0-chrome89-ff77

Seems like the Firefox version used in cypress/included:7.0.1 compared to cypress/included:7.0.0 has been stepped backwards. Maybe an unintentional mistake that was done during the node14 uplift? A cypress/browsers build with Firefox version 86 also exists.

Name + Tag Base image Chrome Firefox Edge
cypress/browsers:node14.16.0-chrome89-ff77 cypress/base:14.16.0 89.0.4389.72 77.0  
cypress/browsers:node14.16.0-chrome89-ff86 cypress/base:14.16.0 89.0.4389.72 86.0.1

Not sure if the older Firefox version could cause the issue with hanging Cypress tests. The issue also seems to be related to shm.

This fix was also added in 7.0.1 and as far as I can see should only affect electron and chrome browsers... However, as seen in the log something called "electron-shared" is running and I'm not sure if --disable-dev-shm-usage could affect firefox.

Also, running docker with increased shm-size=2gb seems to work in Firefox. That is:

$ docker run -it --shm-size=2gb -v $PWD:/e2e -w /e2e cypress/included:7.3.0

With increased shm-size Cypress runs normally without freezing.

@jelyck
Copy link

jelyck commented May 19, 2021

Some more info.

I have tested some different versions.


Versions Freeze/Working
cypress/browsers:node14.16.0-chrome89-ff86 + Cypress 7.3.0 Working
cypress/browsers:node14.16.0-chrome89-ff86 + Cypress 7.0.0 Working
cypress/browsers:node14.16.0-chrome89-ff86 + Cypress 7.3.0 + --shm-size=2gb Working
cypress/browsers:node14.16.0-chrome89-ff77 + Cypress 7.3.0 Freeze
cypress/browsers:node14.16.0-chrome89-ff77 + Cypress 7.0.0 Freeze
cypress/browsers:node14.16.0-chrome89-ff77 + Cypress 7.3.0 + --shm-size=2gb Working
cypress/browsers:node14.16.0-chrome89-ff77 + Cypress 7.0.0 + --shm-size=2gb Working
cypress/included:7.0.0 (using ff86) Working
cypress/included:7.0.1 (using ff77) Freeze
cypress/included:7.3.0 (using ff77) Freeze
cypress/included:7.3.0 (using ff77) + --shm-size=2gb Working

So the issue seems to be due to ff77 running in docker with default shm-size.

I suggest updating cypress/included with cypress/browsers:node14.16.0-chrome89-ff86, same Firefox version as in cypress/included:7.0.0, and update the documentation regarding shm-size and why increasing it solves the issue.

cypress-io/cypress#350 (comment) mentions that there is no longer a need to specify --ipc=host or increase /dev/shm size for Chrome and Electron. However, seems like issues still exists with Firefox. Even though ff86 seems OK there might be issues in future versions.

@jennifer-shehane
Copy link
Member

I'm going to move this issue to the docker repo since it's specific to the docker images.

@jennifer-shehane jennifer-shehane transferred this issue from cypress-io/cypress Jun 15, 2021
@emilyrohrbough
Copy link
Member

@jenslycke Can you provide verbose logs with cypress* (without the :)?

@mjhenkes
Copy link
Member

Since our current supported browser versions are 86 and above, i'm going to go ahead and close this issue.
https://docs.cypress.io/guides/guides/launching-browsers#Browser-versions-supported

@mjhenkes mjhenkes closed this as not planned Won't fix, can't repro, duplicate, stale Jan 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants