"bundled": true added to package-lock.json in lerna + npm monorepo

[mcous]

What Renovate type are you using?
Renovate GitHub App

Describe the bug
In a lerna + npm monorepo using file: dependencies, Renovate introduces lockfile changes that are different from running npm install locally. The most common difference I'm seeing is that Renovate will replace {..., "resolved": "...", "integrity": "...", ...} with {..., "bundled": "true", ..."}.

Did you see anything helpful in debug logs?
I'm happy to check them out with some guidance, but at the moment I don't know what to look for.

To Reproduce
I could try to get a minimal repro repository with a little bit of guidance, if that seems helpful!

For now I'm observing this with almost every Renovate PR to tracespace/tracespace (e.g. tracespace/tracespace#177; I think the renovate job ID is 5444517)

1. Pull down a renovate branch from tracespace/tracespace
2. Run npm install
3. Observe lockfile changes

Expected behavior
Lockfile should not have changes compared to npm install

Additional context
From browsing the issues, this is possibly related to #2294, #3317, #2388, and #2389

I pulled down the branch in the tracespace PR listed above on both macOS and Ubuntu running npm v6.9.0. I did not end up with the same lockfile on both OS's (Ubuntu diff was a superset of the macOS one with some "optional": trues added), so - as mentioned in the threads above - there's definitely still some npm weirdness with cross-platform lockfile generation. Crucially, though, both OS's reverted all the Renovate "bundled": true changes.

Also, thanks for this very cool tool! I'm excited to have automated dependency bumping

[rarkins]

Thanks for reporting. It seems that npm install —package-lock-only still has a bug. I have a fix that can be applied soon.

[rarkins]

@mcous I've made some changes that I think resolve this problem. In short, npm has a bug around file: references and --package-lock-only, so we need to run a full npm install instead in such cases.

Please alert me if you notice any problems remaining and I'll need to dig deeper with reproduction. FYI you can retry/regenerate any PR by ticking the rebase/retry checkbox in the PR body.

[mcous]

Thanks for the quick resolution @rarkins! I'll keep an eye out and post results here

[mcous]

Just had a new one come in (tracespace/tracespace#190) and unfortunately the issue still appears to be present - "bundled": true is showing up in place of "resoved": ... and "integrity": ...

Weirdly enough, when I pull the branch down locally (macOS, npm v6.9.0), I have to run npm i twice before the installation/lockfile settle into a steady state

[abernix]

@mcous Thanks for referencing #3317 from this; it made it easy for me to find this and I noticed it while investigating a commit which added a "bundled": true for an unrelated package renovation: apollographql/apollo-tooling@e1567c3, which seems to align with the bug reference above.

Also, it's unfortunate that this is broken, but I do think the npm team is working hard on resolving issues related to this but thank you @rarkins for looking into it, nevertheless.

[rarkins]

@abernix actually, npm just laid off the (recently hired) staff member who had been assigned responsibility for pushing out new npm CLI releases, so I'm not sure we'll see lots of CLI fixes for a while.

Speaking of npm releases, I am wondering if it's the latest version of npm that could have broken things with local references. We didn't have anyone noticing problems for so long and then now there's at least the two of you.

[rarkins]

Here's a log extract from one of tracespace's recent lock file updates in the app:

Spawning lerna with npm to create lock files
Using lerna version ^3.13.1"
npm i -g -C ~/.npm/lerna@^3.13.1 lerna@^3.13.1 && npm install --ignore-scripts  --no-audit && ~/.npm/lerna@^3.13.1/bin/lerna bootstrap -- --ignore-scripts  --no-audit
npm stdout:
	/home/ubuntu/.npm/lerna@^3.13.1/bin/lerna -> /home/ubuntu/.npm/lerna@^3.13.1/lib/node_modules/lerna/cli.js
	+ lerna@3.13.1
	updated 2 packages in 130.731s
	added 1771 packages from 1728 contributors in 86.09s
	added 10 packages from 6 contributors in 38.236s
npm stderr:
	npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.7 (node_modules/fsevents):
	npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.7: wanted {\"os\":\"darwin\",\"arch\":\"any\"} (current: {\"os\":\"linux\",\"arch\":\"x64\"})

	info cli using local version of lerna
	lerna notice cli v3.13.1
	lerna info bootstrap root only
	npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.7 (node_modules/fsevents):
	npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.7: wanted {\"os\":\"darwin\",\"arch\":\"any\"} (current: {\"os\":\"linux\",\"arch\":\"x64\"})

When I run it locally against a fork:

Using lerna version ^3.13.1 (repository=renovate-tests/tracespace, dependencies=semver, branch=renovate/semver-5.x)
npm i -g -C ~/.npm/lerna@^3.13.1 lerna@^3.13.1 && npm install --ignore-scripts  --no-audit && ~/.npm/lerna@^3.13.1/bin/lerna bootstrap -- --ignore-scripts  --no-audit
npm stdout:
       /Users/rhys/.npm/lerna@^3.13.1/bin/lerna -> /Users/rhys/.npm/lerna@^3.13.1/lib/node_modules/lerna/cli.js
       + lerna@3.13.1
       updated 1 package in 3.697s
       removed 1 package and updated 3 packages in 5.501s
       up to date in 5.249s
npm stderr:
       info cli using local version of lerna
       lerna notice cli v3.13.1
       lerna info bootstrap root only`

[rarkins]

I can now reproduce when running locally/forked using the Renovate docker image: docker run -e RENOVATE_TOKEN renovate/renovate --log-level=debug --include-forks=true renovate-tests/tracespace

[rarkins]

Testing with renovate@14.39.1 (last release to use npm 6.8.0) doesn't solve it

[rarkins]

@mcous could you try ticking the rebase/retry checkbox on some of your PRs? I think the problem may be resolved now in the app. Example from my fork: https://github.com/renovate-tests/tracespace/pull/7/files

It seems that lerna was not using the version of npm we bundled with Renovate and instead was using the default (buggy) npm installed with Node.js 10

[mcous]

@rarkins thanks! That explanation makes sense. I just ticked the rebase box for a couple PRs and will keep an eye on them

[abernix]

@rarkins Is there something we need to do in order to adopt the fix for this? Or is this just something that would occur going forward?

We're getting plagued with peculiarities like this https://github.com/apollographql/apollo-tooling/pull/1150/files. Is there a specific way we should be indicating the version of npm to be used with Lerna?

(Sorry, I know you're not Lerna support, I just wasn't aware we had control over the version of npm that Renovate would use in a lerna monorepo).