Bug: running 2 (maybe more) yarn commands on different projects the same time fails #683

neophob · 2016-10-11T18:10:05Z

I installed yarn the first time, then i ran yarnon project A, seconds later I ran yarn on project B.

Both installation failed, project A output:

yarn install v0.15.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error Couldn't find a package.json (or bower.json) file in /Users/USERNAME/.yarn-cache/npm-yargs-3.10.0
    at /usr/local/lib/node_modules/yarnpkg/lib-legacy/config.js:363:13
    at next (native)
    at step (/usr/local/lib/node_modules/yarnpkg/node_modules/babel-runtime/helpers/asyncToGenerator.js:17:30)
    at /usr/local/lib/node_modules/yarnpkg/node_modules/babel-runtime/helpers/asyncToGenerator.js:28:20
    at run (/usr/local/lib/node_modules/yarnpkg/node_modules/core-js/library/modules/es6.promise.js:87:22)
    at /usr/local/lib/node_modules/yarnpkg/node_modules/core-js/library/modules/es6.promise.js:100:28
    at flush (/usr/local/lib/node_modules/yarnpkg/node_modules/core-js/library/modules/_microtask.js:18:9)
    at nextTickCallbackWith0Args (node.js:415:9)
    at process._tickCallback (node.js:344:13)

project B output:

yarn install v0.15.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error Couldn't find a package.json (or bower.json) file in /Users/USERNAME/.yarn-cache/npm-yargs-4.8.1
    at /usr/local/lib/node_modules/yarnpkg/lib-legacy/config.js:363:13
    at next (native)
    at step (/usr/local/lib/node_modules/yarnpkg/node_modules/babel-runtime/helpers/asyncToGenerator.js:17:30)
    at /usr/local/lib/node_modules/yarnpkg/node_modules/babel-runtime/helpers/asyncToGenerator.js:28:20
    at run (/usr/local/lib/node_modules/yarnpkg/node_modules/core-js/library/modules/es6.promise.js:87:22)
    at /usr/local/lib/node_modules/yarnpkg/node_modules/core-js/library/modules/es6.promise.js:100:28
    at flush (/usr/local/lib/node_modules/yarnpkg/node_modules/core-js/library/modules/_microtask.js:18:9)
    at nextTickCallbackWith0Args (node.js:415:9)
    at process._tickCallback (node.js:344:13)

When I run those commands sequentially it works as expected.

versions:

node 4.5
yarn 0.15.1
osx 10.11.6

The text was updated successfully, but these errors were encountered:

moneytree-doug · 2016-10-12T04:00:30Z

Confirmed.

$ node -v
v5.7.1
$ yarn -v

Edit: Now I am just stuck in this state.

timche · 2016-10-21T13:21:29Z

This bug has to be resolved in order to make Lerna compatible with Yarn, because Lerna spawns multiple concurrent processes of npm install.

moneytree-doug · 2016-10-24T06:01:40Z

@timche Thanks for the update. I'll check it out again tonight.

steelbrain · 2016-10-24T06:39:28Z

@moneytree-doug I think you might have misunderstood what he said. He didn't say that it has been resolved but rather has to be resolved :)

moneytree-doug · 2016-10-24T06:41:07Z

@steelbrain Haha, whoops!

yarnpkg/yarn#683

darabos · 2016-11-07T15:19:32Z

I ended up with a partially written file in ~/.yarn-cache due to such a collision in parallel invocation from Jenkins tests. (gulp-jshint/src/reporters/fail.js was truncated to 512 bytes.) yarn would then successfully run, just copying the broken file from .yarn-cache to the destination directory. 😩

diegov · 2016-11-10T13:05:21Z

Adding my case here since we're in a similar situation, with multiple yarn instances running concurrently. In my case the error I got was
Trace: Error: https://registry.yarnpkg.com/@types/node/-/node-6.0.46.tgz: EEXIST: file already exists, mkdir '/home/administrator/.yarn-cache/npm-@types/node-6.0.46' at Error (native)

STRML · 2016-11-10T19:31:33Z

Seeing this as well if Jenkins happens to run multiple builds at the same time.

The error is very short:

Trace:
  Error: http://x.x.x.x:xxxx/autoprefixer/-/autoprefixer-6.5.3.tgz: ENOENT: no such file or directory, utime '/var/lib/jenkins/.yarn-cache/npm-autoprefixer-6.5.3/package.json'
      at Error (native)

marc-guenther · 2016-12-19T12:30:29Z

It seems the intended "solution" is to use --mutex option to yarn:

https://yarnpkg.com/en/docs/cli/#toc-concurrency-and-mutex

This effectively turns off concurrency completely, which I find rather ridiculous, but maybe preferable to complete crash?

Also, the default location of the mutex is NOT the location of the cache (which would make sense, since it is access to this cache that it is guarding) but the current working directory, where you are probably alone anyway.

So you have to come up with your own unique location of the mutex file, and agree on that with all your other users, otherwise it won't help.

Apart from a global mutex being an "interesting" solution to implement a concurrency safe cache, one can only wonder why this isn't enabled by default, and with a sensible default location as well...

STRML · 2016-12-19T13:10:28Z

There's - - mutex network, which works globally. Good for use in CI environments but unfortunately undocumented.

…

On Dec 19, 2016 6:30 AM, "marc-guenther" ***@***.***> wrote: It seems the intended "solution" is to use --mutex option to yarn: - https://yarnpkg.com/en/docs/cli/#toc-concurrency-and-mutex This effectively *turns off concurrency completely*, which I find rather ridiculous, but maybe preferable to complete crash? Also, the *default location* of the mutex is NOT the location of the cache (which would make sense, since it is access to this cache that it is guarding) but the *current working directory*, where you are probably alone anyway. So you have to come up with your own unique location of the mutex file, and agree on that with all your other users, otherwise it won't help. Apart from a global mutex being an "interesting" solution to implement a concurrency safe cache, one can only wonder why this isn't enabled by default, and with a sensible default location as well... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#683 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABJFPz_goDBKU9WuHogr_boD_Uhack0vks5rJnjtgaJpZM4KT7sH> .

davewasmer · 2016-12-19T18:49:29Z

Looks like the --mutex network option is documented here: https://yarnpkg.com/en/docs/cli/#toc-concurrency-and-mutex

marc-guenther · 2016-12-19T19:08:32Z

Isn't that the link I posted? :)

--mutex network is similar to --mutex file:/tmp/.yarn-mutex in that it is a global lock on the host. You still have to specify it everywhere you run yarn. Also the port number is way easier to collide with something existing, so I would prefer the file.

We have some projects that set their own cache location, and as such would not be affected by the concurrency issue. Setting this option globally would mean they cannot run in parallel to others anymore.

None of this would be a problem if the --mutex option would have sensible defaults or the cache would be concurrency safe in the first place.

scthi · 2017-01-20T07:56:35Z

It would be great to set the default behaviour via env variable. So one doesn't need to specify it on every task of the CI server.

References: yarnpkg/yarn#683

* chore(lerna): Fix lerna packaging and travis config * Moves generator into packages folder * Moves containers into packages folder * Fix lerna travis build * Fixes travis * Fixes yarn issues References: yarnpkg/yarn#683

jgreen210 · 2017-01-27T18:29:08Z

It should be possible to allow for concurrency (on at least UNIX) here if each yarn processes creates a cache entry in a directory with a randomized name and then creates a symlink to it. Races between processes should then be harmless (just leaving an orphaned directory around if lose). Need the symlinks since directory renames aren't atomic.

http://stackoverflow.com/questions/307437/moving-a-directory-atomically

It would be simpler if could use files in the cache, not directories, but maybe that's not viable or performant.

Creating node_modules/ dirs is almost always going to be on the critical path on test servers running things concurrently, so allowing concurrency here would be desirable.

Putting all the cache entries in one directory will fail to scale properly with a sufficiently large number of npm package versions, so might want to split across multiple directories too (directory creation should be easy to make work concurrently).

How would existing caches be migrated to a new format safely? Could use a new independent cache dir, subdir or naming scheme, though this would make people download all cache entries again.

If don't add extra directories, could allow a mixture of dirs and symlinks. If a dir already exists, nothing would want to replace it with a symlink, so there's no timing issue.

If do add extra sub directories, could symlink to the old top-level directories where necessary.

jamischarles · 2017-02-08T00:04:22Z

I get this exact same error, but my use case is slightly different:

On Jenkins, I get this message with Yarn > 0.19.0 but NOT 0.18.1.

I'm only running 1 Jenkins job, and I clean the workspace each time.

npm install -g yarn

These options appear to have no effect on the outcome:
yarn install --ignore-engines --cache-folder .yarn_cache --mutex file:/usr/local/share/.yarn-mutex
yarn install --ignore-engines
./build-tools/node_modules/.bin/yarn install --ignore-engines

Fails 0.20.0

00:02:03.218 yarn install v0.20.0
00:02:03.505 [1/4] Resolving packages...
00:02:06.490 [2/4] Fetching packages...
                                                                                                    
00:02:48.882 error Couldn't find a package.json file in ".yarn_cache/npm-lodash-3.10.1-5bf45e8e49ba4189e17d482789dfd15bd140b7b6"

Fails 0.19.0

00:01:50.156 yarn install v0.19.0
00:01:50.413 [1/4] Resolving packages...
00:01:52.091 [2/4] Fetching packages...
00:02:07.750 error An unexpected error occurred: "http://npm.paypal.com/lodash/-/lodash-3.10.1.tgz: ENOENT: no such file or directory, lstat '/workspace/p2pnodeweb-release-jamis/.yarn_cache/npm-lodash-3.10.1-5bf45e8e49ba4189e17d482789dfd15bd140b7b6/collection/map.js'".
00:02:07.750 info If you think this is a bug, please open a bug report with the information provided in "/workspace/jamis/yarn-error.log".
00:02:07.750 info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

Passes 0.18.1

00:01:34.276 yarn install v0.18.1
00:01:35.084 [1/4] Resolving packages...
00:01:40.234 [2/4] Fetching packages...
                                                                                                    
                                                                                                    
00:03:02.051 warning fsevents@1.0.17: The platform "linux" is incompatible with this module.
00:03:02.051 info "fsevents@1.0.17" is an optional dependency and failed compatibility check. Excluding it from installation.
00:03:02.058 [3/4] Linking dependencies...

modosc · 2017-02-28T17:01:13Z

we see this too, even if using the mutex option. if any attempt at using yarn to install simultaneously occurs, the cache is forever screwed and no installation will go forward (even tried yarn cache clear, npm cache clean, etc). if package.json or yarn.lock is updated there's a chance a subsequent install will run.

this is super annoying, is there any way to debug this?

* yarnpkg/yarn#683

See yarnpkg/yarn#683

MartinHaeusler · 2017-09-27T09:21:58Z

This problem needs to be adressed, badly. Running parallel jenkins builds that invoke yarn is highly problematic at the moment. We tried mutexes, we tried to redirect the yarn cache to the build directory, but nothing helped. There seems to be some part of yarn that always accesses a global location, and that access isn't safe for concurrency.

arcanis · 2017-09-27T10:04:03Z

Which version are you using? Starting from the 1.0, we made a few changes on the mutex codebase that might solve your issues.

MartinHaeusler · 2017-09-27T11:20:27Z

We're on 1.0.2. We are running Jenkins declarative Pipelines, which call Gradle, which in turn uses the Gradle Node Plugin in order to execute the yarn tasks .

The yarn install task in my gradle looks like this:

task yarnInstall(type: YarnTask, dependsOn: yarnSetup){
    def cacheFolder = file("${project.buildDir}/yarn-cache")
    logger.lifecycle("Yarn Cache Dir: " + cacheFolder.getAbsolutePath());
    // run yarn with these arguments
    args = ['install', '--cache-folder ' + cacheFolder.getAbsolutePath(), '--mutex network:31997']
}

So what is executed is yarn install --cache-folder <something local to the build> --mutex network:31997. And still, I get the following error when more than one build runs in parallel:

An unexpected error occurred: "https://registry.yarnpkg.com/core-js/-/core-js-2.5.0.tgz: ENOENT: no such file or directory, utime '/home/user/.cache/yarn/v1/npm-core-js-2.5.0-569c050918be6486b3837552028ae0466b717086/library/fn/math/rad-per-deg.js'" (the offending JS file varies). The interesting part is that yarn attempts to open something in /home/user/..., even though --yarn-cache is set to something else.

I currently cannot eliminate the possibility that the gradle node plugin has a bug, or is interfering in any way here. However, as Yarn integrates with all popular build servers except for Jenkins, we have little choice. An official yarn gradle plugin would be nice too.

arcanis · 2017-09-27T11:53:17Z

I think there's an error in your code. Have you tried using

args = ['install', '--cache-folder', cacheFolder.getAbsolutePath(), '--mutex', 'network:31997']

instead of

args = ['install', '--cache-folder ' + cacheFolder.getAbsolutePath(), '--mutex network:31997']

?

MartinHaeusler · 2017-09-27T12:14:50Z

... interesting point. Will give it a try, thanks!

BYK · 2017-09-27T12:17:01Z

@MartinHaeusler as a workaround you may also try using a separate cache folder for each invocation. That would trade space for time (mutex makes other instances wait for each other).

MartinHaeusler · 2017-09-27T12:18:59Z

@BYK yes, true. We used --cache-folder when we realized that --mutex wasn't doing what we intended on our Jenkins. But maybe it was just my configuration that was wrong all along (as @arcanis pointed out). I will investigate this further.

MartinHaeusler · 2017-09-27T12:34:33Z

Unfortunately the fix proposed by @arcanis (even though it was a nice hint!) did not change the situation.

arcanis · 2017-09-27T12:41:44Z

That's really weird 😯 We use Yarn internally at fb, and as you can imagine we do have a fair amount of parallel tests running with the network mutex, so I have no idea why it doesn't work on your side.

MartinHaeusler · 2017-09-27T12:43:56Z

I am suspecting the gradle node plugin by now which runs our yarn tasks to be at fault here... This issue will be one tough nut to crack.

BYK · 2017-09-27T12:56:19Z

@MartinHaeusler may be run Yarn with --verbose to see how processes interact with each other with the mutex?

MartinHaeusler · 2017-09-27T12:58:51Z

Nice suggestion, I'll give it a shot. Somehow I suspect that either the gradle node plugin does not properly pass the --mutex option to yarn, overrides it somehow, or yarn ignores it. Looking at the verbose logs should provide some insight.

MartinHaeusler · 2017-09-27T13:11:25Z

By the way, apparently I'm not the only one struggling with this problem on Jenkins:
Ticket 2629

See yarnpkg/yarn#683

BYK · 2017-10-17T13:55:47Z

I feel like we can close this. Objections? @MartinHaeusler @neophob or anyone else?

jgreen210 · 2017-10-17T14:22:12Z

Being able to corrupt the cache if run yarn concurrently with itself using its default options is a bug. Using a sensible default for --mutex (i.e. a per-cache-folder path) or reimplementing cache so that it doesn't need a mutex would solve this (see my above comments for how this might be possible).

marc-guenther · 2017-10-17T14:32:25Z

Close this? Has it been fixed in a different ticket? Cause I haven't seen anything here. What's the fix?

arcanis · 2017-10-17T16:28:39Z

Using a sensible default for --mutex (i.e. a per-cache-folder path) or reimplementing cache so that it doesn't need a mutex would solve this (see my above comments for how this might be possible).

Correct, and we will happily review and merge any PR that would help with that.

Close this? Has it been fixed in a different ticket? Cause I haven't seen anything here. What's the fix?

Using the --mutex flag solves the issue reported in the OP (running two Yarn instances concurrently will not corrupt the cache anymore). At this point what you're asking seem to be an improvement over this, which deserves its own issue (but as you might have guessed by my previous statement, we would very much prefer a PR over an issue ;).

Spongman · 2017-11-20T20:34:22Z

ah, the best kind of QA: sweep it under the rug ;-)

marc-guenther · 2017-11-20T21:23:30Z

Yes, it is completely ridiculous. First ignore the bug for a year, then suddenly close it without any fix, because there is a horrible workaround (which existed the entire time!), which everyone has to enable manually every time. Ah, and most important, don't bother to update the documentation.

arcanis · 2017-11-20T22:46:36Z

Yarn is an open-source and community-driven project, which means that by design you're just as welcome to contribute to it as you are to use it. The documentation is here, the source code here. If you feel strongly about this feature, please feel free to give it a try - we'll happily spend time reviewing it and giving you pointers to unblock you if you have questions.

BYK · 2017-11-21T14:13:57Z

@Spongman @marc-guenther I am sorry that you feel this way about the issue, and Yarn in general. That said we have documented this command line flag some time ago with yarnpkg/website#280. I am not claiming it is the most visible or best description but at least it is there, searchable, and anyone who feels differently is welcome to try improving it. It is actually best for people like you who feel frustrated and not served well by the project to take a stab at improving docs because you know exactly where the pain is.

Regarding fixing the root cause of the issue, which is unsafe concurrent read/write access to a global cache, I agree that this should be solved. That said, anyone who tried to fix this would also know that it is not an easy problem to solve, especially if you don't want to compromise performance. Moreover, concurrent Yarn instances do not seem to be the common case since people usually work on a single project at a time (at least install dependencies one project at a time) and CI systems are isolating builds (with some exceptions). This is why the issue hasn't received much attention from the core maintainers. That said if you feel strongly about it and if you have a good solution in mind, we would be more than happy to assist you throughout the contribution process and champion your pull request if you decide to submit one.

Thanks a lot for sharing your concerns again. All being said I'd also appreciate if you keep in mind that we are all working towards better software here and the best way to get there is collaboration instead of blaming.

I'd love to continue this discussion without pinging everyone on this thread continuously so feel free to join our Discord server and ping me directly or on any of the channels. I'd love to hear more about why you feel your contributions are not welcome and do our best to resolve them ❤️

sebmck added the cat-bug label Oct 11, 2016

sebmck mentioned this issue Oct 15, 2016

Feature: Allow for concurrency #1019

Closed

timche mentioned this issue Oct 19, 2016

[Feature] Switch to Yarn lerna/lerna#371

Closed

wyze mentioned this issue Oct 25, 2016

Concurrent install messes up #1418

Closed

shanewilson added a commit to knitjs/knit that referenced this issue Nov 4, 2016

🐛 sync publish tasks

16e4a27

yarnpkg/yarn#683

tmair mentioned this issue Nov 7, 2016

Can't run two instances of Yarn simultaneously: ENOENT: no such file or directory in .yarn-cache #1275

Closed

ukilon-okta mentioned this issue Jan 5, 2017

Yarn fails getting a package from an artifactory server. or from disk. #2398

Closed

talend-glorieux added a commit to Talend/ui that referenced this issue Jan 24, 2017

Fixes yarn issues

e916950

References: yarnpkg/yarn#683

squadette mentioned this issue Feb 12, 2017

Using yarn instead of npm shakacode/react_on_rails#715

Merged

viliusl mentioned this issue Feb 25, 2017

parallel bootstrap wix-incubator/octopus#10

Closed

evocateur mentioned this issue Mar 7, 2017

No such file error when running bootstrap with yarn as npmClient lerna/lerna#650

Closed

omniscient mentioned this issue Mar 8, 2017

Yarn can not use private registries #2541

Closed

jdx pushed a commit to heroku/cli-engine that referenced this issue Aug 25, 2017

Force jest to run in band again due to yarn bug

45e067e

* yarnpkg/yarn#683

ketan added a commit to gocd/gocd that referenced this issue Sep 20, 2017

Use a mutex to prevent multiple yarn processes from installing

73e2d3d

See yarnpkg/yarn#683

jdubois added a commit to jhipster/generator-jhipster that referenced this issue Sep 27, 2017

Add a mutex on Yarn to try to fix the Travis build corruptions

0447f95

See yarnpkg/yarn#683

jdubois mentioned this issue Sep 27, 2017

Upgrade to Yarn 1.0 jhipster/generator-jhipster#6362

Closed

mmdriley mentioned this issue Oct 13, 2017

Fix Yarn errors during build pulumi/pulumi#410

Merged

arcanis closed this as completed Oct 17, 2017

vjpr mentioned this issue Nov 17, 2017

Concurrent installs pnpm/pnpm#941

Closed

yarnpkg deleted a comment from Spongman Nov 21, 2017

yarnpkg locked and limited conversation to collaborators Nov 21, 2017

Bug: running 2 (maybe more) yarn commands on different projects the same time fails #683

Bug: running 2 (maybe more) yarn commands on different projects the same time fails #683

Comments

neophob commented Oct 11, 2016

moneytree-doug commented Oct 12, 2016 • edited Loading

timche commented Oct 21, 2016

moneytree-doug commented Oct 24, 2016

steelbrain commented Oct 24, 2016

moneytree-doug commented Oct 24, 2016

darabos commented Nov 7, 2016 • edited Loading

diegov commented Nov 10, 2016

STRML commented Nov 10, 2016

marc-guenther commented Dec 19, 2016

STRML commented Dec 19, 2016 via email

davewasmer commented Dec 19, 2016

marc-guenther commented Dec 19, 2016 • edited Loading

scthi commented Jan 20, 2017

jgreen210 commented Jan 27, 2017 • edited Loading

jamischarles commented Feb 8, 2017

modosc commented Feb 28, 2017

MartinHaeusler commented Sep 27, 2017

arcanis commented Sep 27, 2017

MartinHaeusler commented Sep 27, 2017

arcanis commented Sep 27, 2017

MartinHaeusler commented Sep 27, 2017

BYK commented Sep 27, 2017

MartinHaeusler commented Sep 27, 2017

MartinHaeusler commented Sep 27, 2017

arcanis commented Sep 27, 2017

MartinHaeusler commented Sep 27, 2017

BYK commented Sep 27, 2017

MartinHaeusler commented Sep 27, 2017

MartinHaeusler commented Sep 27, 2017

BYK commented Oct 17, 2017

jgreen210 commented Oct 17, 2017

marc-guenther commented Oct 17, 2017

arcanis commented Oct 17, 2017 • edited Loading

Spongman commented Nov 20, 2017

marc-guenther commented Nov 20, 2017

arcanis commented Nov 20, 2017 • edited Loading

BYK commented Nov 21, 2017

moneytree-doug commented Oct 12, 2016 •

edited

Loading

darabos commented Nov 7, 2016 •

edited

Loading

marc-guenther commented Dec 19, 2016 •

edited

Loading

jgreen210 commented Jan 27, 2017 •

edited

Loading

arcanis commented Oct 17, 2017 •

edited

Loading

arcanis commented Nov 20, 2017 •

edited

Loading