Docs site and binary install from yarn can be very slow from Korea

[paulfalgout]

• Operating System: OSX
• Cypress Version: Any

Current behavior:

Sometimes cypress can take up to 30 minutes to download

How to reproduce:

Use a blazing fast internet connection in Korea

Additional Info (images, stack traces, etc)

Docs site after a few minutes (still no fonts)

Download

[21:47:48]  Downloading Cypress      21% 4487s [title changed]\n[21:47:48]  Downloading Cypress      21% 4489s [title changed]\n[21:47:49]  Downloading Cypress      21% 4491s [title changed]\n[21:47:50]  Downloading Cypress      21% 4495s [title changed]\n[21:47:51]  Downloading Cypress      21% 4498s [title changed]\n[21:47:51]  Downloading Cypress      21% 4500s [title changed]\n[21:47:51]  Downloaded Cypress      [title changed]\n[21:47:51]  Downloaded Cypress      [completed]\n[21:47:51]  Unzipping Cypress       [started]\n[21:47:51]  Unzipping Cypress       [failed]\n[21:47:51] → The Cypress App could not be unzipped.\n\nSearch for an existing issue or open a GitHub issue at\n\nhttps://github.com/cypress-io/cypress/issues\n----------\n\nError: end of central directory record signature not found\n----------\n\nPlatform: darwin (15.6.0)\nCypress Version: 1.4.1\nThe Cypress App could not be unzipped

The cypress docs site takes minutes to load docs sometimes. This isn't 100% consistent, but it seems to be becoming a more frequent issue. While I can generally use some of the docs during the loading, if I have to re-yarn sometimes it stops me in my tracks as a download can take 30 minutes and fail to unzip. I can't say that this is always the case, but it is frequent and it is, as far as I can tell, unique to the cypress.io server, as I can't think of another site I experience this with. I've also had this experience at home as well as various coffee shop wifi.

[brian-mann]

Both the docs + the binary are hosted on s3 and then served to Cloudflare edge locations. They should have data centers right near you https://www.cloudflare.com/network/

Although they did have some incidents lately. Caching is hard :-/

[paulfalgout]

It seems to happen a lot.. I'm only installing and reading the docs seasonally, so I don't know how consistent the problem is, but it's been slow for at least 2 days now. I'm not sure if there's anything to really do about it, but the struggle is real.

[mtibben]

Also very slow from Australia

[mtibben]

Ideally the downloaded binaries would be cached on the local filesystem as per #1233

[maxime1992]

I asked for CI how to avoid downloading again and again the binaries and @bahmutov told me to put the node_modules folder in cache, which is not really 100% OK in my opinion (but I ended up with something similar, caching the yarn cache folder).

Anyway, my point here is that @mtibben is right, if we can cache node_modules on CI, we don't want to have to save it manually locally. Having some cache on local filesystem would be 🎉 🎉 🎉

😄

[bahmutov]

I am not sure I understand the problem then. Every CI provides a way to cache node_modules folder - otherwise you would need to install dependencies every time the build runs which is slow. Cypress unzips itself in node_modules folder. If CI caches that folder as it should it will only have to download Cypress once. Am I missing something here?

…

Sent from my iPhone

On Jan 30, 2018, at 17:55, Maxime ***@***.***> wrote: I asked for CI how to avoid downloading again and again the binaries and @bahmutov told me to put the node_modules folder in cache, which is not really 100% OK in my opinion (but I ended up with something similar, caching the yarn cache folder). Anyway, my point here is that @mtibben is right, if we can cache node_modules on CI, we don't want to have to save it manually locally. Having some cache on local filesystem would be 🎉 🎉 🎉 😄 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mtibben]

The npm local cache directory is normally ~/.npm/.

node_modules is not a cache directory, it's your installed packages for the current project. While some CI systems will keep it around, it more than likely removed for a few reasons

1. the node_modules directory is inside your project directory, so a git clean blows it away
2. If the CI is parallelising builds between machines / directories, you'll often get a clean tree, or a version of node_modules which is out of date or broken
3. historically npm has been really bad at keeping node_modules consistent. The first thing you do when you run into npm issues is just delete the node_modules directory and npm install again.

So this gets painful with cypress even locally when you have multiple projects using it, flaky npm, and slow download speeds.

I'm guessing you're using the free tier of Cloudflare? I've seen some pretty poor performance on the free tier from some countries (particularly Australia 😢 )

If cypress can cache the binary via ~/.npm, that would be 💯 💯 💯

[maxime1992]

@bahmutov CI is not the main problem here but as @mtibben said

historically npm has been really bad at keeping node_modules consistent

And often, on our dev machine we end up removing the whole node_modules.
And for example, I'm using yarn and I've defined a cache folder so when I run yarn install on a project that I've previously fetched dependencies, but removed the node_modules for some reason, yarn install is still super fast thanks to cache. BUT cypress have to be downloaded again so instead of few seconds it can take up to minutes according to internet speed because cypress is downloaded again and not cached on my machine.

Does that make sense?

EDIT:

I just made a test on a large angular-cli project.
I've already yarn install it, so every packages are in yarn's cache.

With Cypress dependency in the project:

rm -rf node_modules && time yarn install --frozen-lockfile

yarn install --frozen-lockfile 31.23s user 8.21s system 75% cpu 52.559 total

Without Cypress dependency in the project:

rm -rf node_modules && time yarn install --frozen-lockfile

yarn install --frozen-lockfile 13.95s user 2.36s system 133% cpu 12.184 total

And honestly this morning internet is unsually faster than usual. Otherwise it can take up to 3mn.

If Cypress was cached on my machine it would have been closer to the ~12s instead of ~52s or more 😺

[bahmutov]

I see, thank you for clarifying this. We rarely experience node_modules corruption (using exact dependency versions helps a lot), so blowing away this folder happens infrequently.

We have support for installing binary from a given location: https://docs.cypress.io/guides/getting-started/installing-cypress.html#Environment-Variables which you can cache anywhere. That should make install work faster. For example if you cache folder /foo and store downloaded cypress.zip there you can install it later

CYPRESS_BINARY_VERSION=/foo/cypress.zip npm install cypress

Same works if you set the CYPRESS_BINARY_VERSION variable and just run npm install too.

This still assumes manual step of downloading cypress.zip and putting it somewhere. If someone wants to add support for saving the cypress.zip in ~/.npm that would be great. Relevant code is in https://github.com/cypress-io/cypress/blob/develop/cli/lib/tasks/download.js - I think saving downloaded file into ~/.npm folder could be added as intermediate location.

[maxime1992]

cache folder /foo and store downloaded cypress.zip there

This still assumes manual step of downloading cypress.zip and putting it somewhere

It wouldn't be that easy in fact. Because we'd have to download multiple versions of Cypress and when installing we'd need to copy the good one from the cache.

Can't promise anything I have a lot to do these days but I'll try to find some time and dig into the code to see if I can come up with a PR.

[maxime1992]

@bahmutov I did try something :)

But now, how am I supposed to try what I've done?

I asked on Gitter but I do not have any solid lead right now:

Any help on that would be really appreciated thanks 😄

[maxime1992]

Thanks to @jennifer-shehane I've understood how to get started with all of that and I'm giving a try =)
Cheers

[paulfalgout]

S3 alone would magnitudes faster than cloudflare

[maxime1992]

What should be done here is a proper caching so you don't have to download cypress anymore except for update. Take a look to this PR #1254 and brian-mann proposal :)

EDIT: my bad you've already commented there

[bahmutov]

I will work on this in #1300

[paulfalgout]

Worth noting switching to cloudflare's DNS seemingly solved this entirely.
Nope. Just had a good day. back to cypress: [22:57:34] Downloading Cypress 4% 4871s [title

[kuceb]

Fixed in #1300, released in 3.0

[paulfalgout]

This isn't fixed. The download and docs are still abysmally slow. Yes, I can cache the app.. great. Every time there's a new release I still have to take an afternoon to download it.

[paulfalgout]

seems to be fixed.. maybe cloudflare fixed it?