Wishlist on managing package manager versions in a Node.js application

[trivikr]

Is your feature request related to a problem? Please describe.

There are several discussions about managing package manager version in the past.

At the time of writing, i.e. 2024-07-30, Node.js ships experimental corepack which allows pinning the packageManager field in package.json. It's being used in at least tens of thousands of applications searchable in public code.

There has been asks to make corepack stable since May 2022 nodejs/corepack#104
The PR to enable yarn/pnpm corepack binaries by default in nodejs/node#51886, has moved from most approvals to most declines. There's an open PR to remove corepack too at nodejs/node#51981

The tweets about corepack often go viral, like ones by Matt https://twitter.com/search?q=from%3Amattpocockuk%20corepack&src=typed_query, and then some discussions happen Twitter threads which aren't documented.

Here is some wishlist I would like to share:

• As a Node.js user, I want an built-in way to manage my package manager version, like corepack.
• As a corepack user, I don't want my application which specified packageManager field to break.
• As a Node.js user, I'm open to experiment with alternatives, like devEngines.

Describe the solution you'd like

Based on requirements, and consensus in devEngines proposal as of July 2024, here is what a good solution may look like:

• Do not break applications which use packageManager field.
• Implement devEngines specification, and provide an easy built-in way for users to use it. This can be corepack or something else. It doesn't matter as long as it ships with Node.js and is easy to use.

Over time, if there's data that one specification is better than other, then add doc/runtime deprecations and provide a migration path to the preferred one.

Describe alternatives you've considered

• Switch to installing corepack from npm.
• Switch to a different runtime, like Deno/Bun.

Additional context

Originally posted as a tweet in https://twitter.com/trivikram/status/1818303053902307479.
A GitHub issue was created as per response from @wesleytodd

[trivikr]

I'm not too happy with the existing title Wishlist on managing package manager versions in a Node.js application, as it's too generic. Do suggest a better one in the comments, if you come across one.

[alexaka1]

Describe alternatives you've considered

• Switch to installing corepack from npm.

I think this outcome is the one that should be avoided. Why? Bad defaults.
I spend way too much time finding people npm i yarn at work and online, and then I always have to spend 30 minutes explaining to them what yarn is in 2024, what corepack is and why This is the way.
People don't know what corepack is, because it's not enabled by default.
The breaking changes of Yarn 2 have left behind a generation's worth of outdated new code, just because the proper way of installing Yarn 2+ is not via NPM, so people have no idea that a yarn newer than 1.x even exists, or how to install is, because the NPM package makes no mention of it (afaik). Upon RTFM-ing the docs, they are asked to run some command, and some mentions of the binary maybe not even being available on your Node install. This is just mental overhead, and a bad default.

Most people statistically never switch away from defaults, and arguing or urging people to do so causes frustration and a bad experience. Bad defaults cause all sorts of problems, from security, privacy, financial, and so on. So the onus is on Node to provide a good experience. And imo Corepack is a good experience. I have not heard of devEngines until now, it looks alright. I really like the shim approach of corepack, but to each their own. My only concern is, that node version managers are a solved problem. Nvm, fnm are the two recommended by Node, in fact fnm specifically solves this problem by exposing a setting to the user, to always enable corepack on new Node installs, or the very aggressive --use-on-cd flag to switch node versions. However, that is yet another setting and software that devs need to 1) know about, 2) remember to turn on/use, because it is not the default. I believe this is a worthy enough breaking change, to not worry about old Node version. I.e. from Node XX you have to specify what NPM version you need, you cannot run npm commands in your project unless you have someProperty set in your package.json. I think worrying about seamless upgrades from Node 0.4 to Node 24 is futile. Microsoft is known for going all in on backwards compatibility and has famously bad defaults, that are vulnerable to all kinds of attacks, in both Entra ID and Windows. Speaking of...

This whole npm i corepack or nver install workflow is very Edge-like. On a fresh Windows you open Edge just to download Chrome and then never use Edge again. While Edge is fighting tooth and nail to stop you from doing this. It gives a bad vibe for NPM to act this way, plus just more churn for developers to dislike the newer (and inarguably better) solutions to NPM.

(sorry for the long rant)

[wesleytodd]

Don't be sorry for the rant, this is good feedback. Feedback which I think nearly everyone agrees on the goals of. I think the breakdown occurred in a very different way than is made out above though. All that work on devEngines was more about trying to find something that could reach consensus which had been impossible previously. Something that would have unblocked the objections to corepack's implementation details (the shims were not the only issues).

If we had a group of corepack maintainers willing to engage on addressing the concerns I personally would love to see us all be able to move forward with the better UX provided by corepack. It just cannot be in a way that forces participation in the way it's implementation does today. And it cannot be without requirements for the package managers (like sourcing the binaries from a trusted source, another of the objections).

The last attempt I made to bring these groups together resulted in this PR which made some headway and has resulted in some great changes to the downloads page to feature better instructions and featuring the package managers. I have been way to busy to keep up with this stuff lately, but I have hopes that this sign of progress will be enough to get over the historical baggage. If you are interested in being a part of helping to resolve this please join in on the conversations and help us reach a more balanced consensus agreement. The more voices we have helping push this toward that the better.

[alexaka1]

I was going to listen in on the august team meeting, but it was cancelled, which prompted the above post. And unfortunately GitHub doesn't notify me when a subscribed issue (this one) is mentioned somewhere else. Or I just completely missed the 3 other team meetings since then.
I would be happy to chime in, as a corepack user.
I don't remember where the discussions were happening though, and as far as I remember, Node's concerns were not addressed by the required party (Corepack maintainers). Although personally I would challenge some concerns like the trusted source requirement. But in that long thread I didn't feel like there was any more room to chime in. That was the vibe I got from it.

[ljharb]

fwiw I saw your comment elsewhere; npmjs.org is indeed more special than every single other option; it's the "most favored package registry" for the foreseeable future.

[alexaka1]

Why? Like I have no problem Node/NPM using npmjs.org by default, like for example how Debian has their own package registry too. But I can use my own, or any compatible registry I want, no? The open option to use 3rd party registries is contradictory to npmjs.org being special. I don't get this. It's the default, no argument there, but more special?

[wesleytodd]

I am not sure if it is a fruitful line of discussion, but I think @ljharb meant that it is special as the ultimate source of truth for such a large majority of packages that it is defacto "special". Not that anything you said is untrue, it just also is a not really important to the discussion at hand.

Sorry about the meeting things, it is quite hard to organize these things with volunteers and as such it means it is up to individuals to find/follow the right repos. Not a great setup, but I could also recommend using the OpenJS public calendar which lists all of these.

As for the threads in node core, I agree adding more to them now is unlikely to be helpful. I had attempted to move them here where we could better manage them without sending notifications to all the people following all activity on that repo (reduce the workload to keep up for many) but that did not seem to pan out.

Although personally I would challenge some concerns like the trusted source requirement

Challenging that concern is valid. I am sure there are compromise points which could have been reached around that. As someone who has helped manage the infra for this stuff I personally hold strongly that corepack is designed to enable a decision that is bad for most people and that the defaults are dangerous. If this was a userland project I would just move on with life, but the problem arises when the runtime is pushing this feature as the default experience. Hence all of the strong feedback and dissent.

[alexaka1]

corepack is designed to enable a decision that is bad for most people and that the defaults are dangerous

Yeah, this where my counter concern comes from. Imo corepack is good, and it solves an important problem of managing package manager versions in a team. So seeing an almost hostile stance towards it from the Node team, made me very concerned (the Remove corepack PR out of nowhere).

Have you tried to contact (I hate to say) JS influencers? Like Theo Browne, Matt Pocock, Thdxr. I know Theo is a big corepack fan, reaching out to them could get you to reach a bigger audience of devs. 99.9% of devs don't follow any of this, they don't even know this is a concern you guys have. You could get a lot of insight through this audience, on e.g. why they use corepack.

Speaking for myself, I use Node more than I'd like to, and I have never seen any of your surveys, conversations about this prior to Theo's video about corepack's removal. By that point I was all in on corepack and have always moved teams I worked with onto it, because imo it is the nvm of package managers. And it makes me cringe whenever someone installs Node from the .msi, because managing node versions is a solved problem, and this tool made managing package managers a solved problem for me.

[wesleytodd]

I agree it solves an important problem, but that is separate from everyone's stated concerns about how it is solving that problem.

I am not sure what we would gain from asking individual users who are only one of the constituencies the project serves. The different constituencies the project serves can have very different priorities, but what ships with core needs to at least be non-objectionable to most (or hopefully all) of them. Can you help explain what we would gain from speaking to those folks about this issue?

As you point out, watching new users fumble with the install is a problem and if you compare the download page of today to what it was before we started this, I hope you agree this work to understand and better document the ways to get setup have been very helpful. I still don't believe we are across the finish line, but I also firmly believe the project made the right decision not turning it on by default.

[alexaka1]

Oh it has shipped already I see. Yes I even liked it before the changes you linked from days ago, the fact that the default option is nvm was very welcome by me (Although I became a fnm guy a year ago, sorry ljharb :P). I wish the big green button on the landing page would also default to a version manager, and not just note it below. The in your face CTA should guide towards a version manager. But I accept that as my personal preference. The dedicated /download page is very nice.

As for what benefit to contacting creators would bring. I think it would help cool the waters and reduce hostility between node and its users. Right now the sentiment is very much Big NPM being anti-competitive even though you would argue that is not the case. But that is not the perception, and having a bigger reach on this (or other issues) helps mitigate that.
Tbh I'm not familiar with what constituencies you're talking about, but people (devs) use node, not companies, and if people are not happy with it or where it's going (i.e. node 24 removing corepack entirely with no drop in replacement) then users will start leaving. Speaking for myself I would be very unhappy if corepack got removed. At that point I would start looking at node alternatives and urge my team to move off of Node in favor of other tools.

[wesleytodd]

I wish...

All of this stuff is great feedback which might be better done in an issue or PR to the website so the folks who work on that see it (not sure if they follow along here).

I think it would help cool the waters and reduce hostility between node and its users.

I have seen multiple issues I am deeply involved in get attention from some of these folks and nothing they have done ever helps "reduce hostility". I have reached out to multiple folks who work on this stuff in the past offering to help clear up the message and make sure it is accurate, not once have any of them engaged in a conversation about it before publishing.

I would love if the reality were different, but that is not the reality we are in.

Tbh I'm not familiar with what constituencies you're talking about, but people (devs) use node, not companies

We have a few docs on this you can check out:

(edit early send! oops)

https://github.com/nodejs/next-10/blob/main/CONSTITUENCIES.md
https://github.com/nodejs/next-10/blob/main/CONSTITUENCY-NEEDS.md
https://github.com/nodejs/node/blob/main/doc/contributing/technical-values.md