🙋 Automatically install missing dependencies, part 2

[devongovett]

Now that #306 has landed, it would be awesome to take this even further. Currently we automatically install dependencies loaded by the internal localRequire helper. These are things like compilers (e.g. typescript, coffeescript), and plugins (e.g. postcss plugins, parcel plugins, etc.).

It would be super cool to take this even further and transparently install any npm dependencies you require in your code for you. e.g. if you add require('react') in your code, and it's not installed, Parcel can install it for you.

Thoughts?

[devongovett]

cc @DeMoorJasper

Also, looks like there is a webpack plugin that does something similar https://github.com/webpack-contrib/npm-install-webpack-plugin

[brandon93s]

So, to make sure I understand....

Instead of this:

$ git clone https://github.com/project/repo.git
$ yarn install
$ yarn start

All I would need is this:

$ git clone https://github.com/project/repo.git
$ yarn start

Assume yarn start points to parcel {entry} in these examples.

[devongovett]

Sure, or you have parcel already started, and change your code to import a new dependency. No need to ctrl+C and install it anymore, parcel will just install it automatically when it detects the import.

[shawwn]

I get the temptation to make Parcel be completely automatic, but you have to be careful to weigh the risks. Parcel will have a good run for a few years. Why will Parcel fail? Because the codebase becomes too bloated and hard to change, just like Webpack. I created a fork of Parcel called Parsley, and it was incredibly simple because we have no configuration. We don't depend on any environment variables or console settings that users would expect to maintain across the fork. Automatically installing dependencies is one of those things that seems cool at first, but when you think about it, what's being saved? The users are slightly less annoyed for a few seconds? Typing `yarn add foo --dev` is too hard? Worse, when it causes problems, it will cause problems in a huge way. Users will run into strange mysterious errors that they'll have to google to get answers for. "Why is parcel foo.html failing for this Vue project?" "Oh, yeah, vue-hot-reload-api fails to install due to some bug in yarn. Really annoying, just run yarn add vue-hot-reload-api --dev manually." Combine this with the fact that we're spinning our wheels on problems that don't really matter. We could be focusing on impactful work such as extending the plugin system so that people have access to the core APIs from plugins. They can't right now. There's no way to override the behavior of the methods in Bundler. But we're focusing our thought and energy into automatically installing dependencies instead, when no one really wants that aside from a few enthusiasts. And those users are important, but they're not the cream of our crop: disgruntled Webpack users. The bigger our codebase becomes, the harder it will be to deal with concerns that sprout up regarding all of these features. These features have to be maintained, and they have to work, and we have to respond to issues and PRs related to them. Codebase grows as O(n^2) with the number of lines, because lines tend to communicate exponentially with other parts of the codebase. It's not strictly O(n^2), but it certainly isn't O(n). The cost of adding code outweighs the slight convenience of doing this. That said, it's a team decision. :) If everyone on the team wants this, then *shrug* go for it. But it'll be the #1 thing users will want to turn off. (If Parcel doesn't add a CLI flag to turn it off, I'd stop using Parcel. But that just means I now have to remember to run `parcel --no-install --no-cache index.html` for development all the time now, which is quickly becoming hilarious.)

…

On Thu, Dec 21, 2017 at 11:14 PM, Devon Govett ***@***.***> wrote: Now that #306 <#306> has landed, it would be awesome to take this even further. Currently we automatically install dependencies loaded by the internal localRequire helper. These are things like compilers (e.g. typescript, coffeescript), and plugins (e.g. postcss plugins, parcel plugins, etc.). It would be super cool to take this even further and transparently install any npm dependencies you require in your code for you. e.g. if you add require('react') in your code, and it's not installed, Parcel can install it for you. Thoughts? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#376>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADo8HEj0pmj6l0W9wUtnyqET15ybYD7ks5tCzqpgaJpZM4RKpxg> .

[shawwn]

That came out a bit more rant-y than I wanted, but the gist is that if someone wants to set up react, it takes them ten seconds to install the dependencies.

If someone wants to use react without Parcel, it takes them ten minutes to configure Webpack.

That's why Parcel is valuable. Saving ten seconds at the cost of maintainability and extensibility seems like a bad tradeoff, but maybe users feel differently. I don't know.

[devongovett]

Parcel is all about developer experience. Anything we can do, no matter how small, to save developers time is a win in my book. Developer hours add up quick!

We'll be working on things like improving plugins etc. in good time. Feel free to discuss how we can improve other aspects of Parcel in separate issues. There is room for everyone here.

[brandon93s]

@shawwn - It will be interesting to see how this plays out. The project is moving at breakneck pace at the moment, it's hard to say what things will look like tomorrow, nonetheless next week at the moment. I suspect if there is community outcry for or against a feature like this, the project will react accordingly. Let's see how the community feels about it.

From triaging issues that come in, I can say this will fix the occasional "what do I install" question.

I agree auto-install may feel a bit gimmicky on the surface right now, but I'd like to give it a chance to see what it becomes and how the community adopts it.

[shawwn]

Mm, fair point. But please ship a CLI option to turn it off. :)

…

On Thu, Dec 21, 2017 at 11:35 PM, Brandon Smith ***@***.***> wrote: @shawwn <https://github.com/shawwn> - It will be interesting to see how this plays out. The project is moving at breakneck pace at the moment, it's hard to say what things will look like tomorrow, nonetheless next week at the moment. I suspect if there is community outcry for or against a feature like this, the project will react accordingly. Let's see how the community feels about it. From triaging issues that come in, I can say this will fix the occasional "what do I install" question. I agree auto-install may feel a bit *gimmicky* on the surface right now, but I'd like to give it a chance to see what it becomes and how the community adopts it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADo8CTGZDLViD5duEFFJDHCmSanp1Dbks5tCz--gaJpZM4RKpxg> .

[brandon93s]

Sure, or you have parcel already started, and change your code to import a new dependency. No need to ctrl+C and install it anymore, parcel will just install it automatically when it detects the import.

Open question here would be whether or not it shows up in your package.json dependencies "automagically" or not. Would be pretty wild if I just add require('lodash') to a file and it installs it and adds the dependency for me. That sounds like a developer experience I can get behind. Your focus becomes the code you're writing, and nothing else at that point.

[devongovett]

@brandon93s That's exactly how it works now for compilers. e.g. import a .ts file and typescript is automatically installed and saved into devDependencies.

[shawwn]

If we do this, do we need to support letting the user write `require('coffeescript:dev')` in their projects so that Parcel installs `coffeescript` as a dev dependency? We can get away with that with localRequire. But here, we're talking about the general case where people want to avoid typing `yarn add foo` altogether. And if we want to do that, we might need a way to express "require this or install it as a dev dependency" vs "require this or install it as a regular dependency." This ties in with updating `require` to be a bit more extensible, e.g. passing query parameters like `import "./foo.Vue?ssr"` to do server-side rendering, or `import "./foo.css?scoped"` to use CSS scoping. That's "implicit configuration" -- the code defines the configuration -- but it's also handy for one-off scenarios. (Such as `require("coffeescript?dev")`, actually.) I don't know if this makes sense or not, but there you go.

…

On Thu, Dec 21, 2017 at 11:40 PM, Brandon Smith ***@***.***> wrote: Sure, or you have parcel already started, and change your code to import a new dependency. No need to ctrl+C and install it anymore, parcel will just install it automatically when it detects the import. Open question here would be whether or not it shows up in your package.json dependencies "automagically" or not. Would be pretty wild if I just add require('lodash') to a file and it installs it and adds the dependency for me. That sounds like a developer experience I can get behind. Your focus becomes the code you're writing, and nothing else at that point. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADo8AWuPv8fzo6WPe4Th2oMLZ8WJs-0ks5tC0DJgaJpZM4RKpxg> .

[devongovett]

Not really a fan of overloading the require function or import syntax. but yeah, that's a valid concern.

[shawwn]

Meh, just make it install everything as regular dependencies then. It's not strictly correct, but it will work in all cases, which is more important. People can manually move dependencies to dev if they want to, later on.

…

On Fri, Dec 22, 2017 at 12:02 AM, Devon Govett ***@***.***> wrote: Not really a fan of overloading the require function or import syntax. but yeah, that's a valid concern. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADo8OIJbteQhRmAfL2P_pyVa7y8Db3Sks5tC0X_gaJpZM4RKpxg> .

[DeMoorJasper]

Why would there be a cli option to turn it off?
If you install the dependencies up front then it wouldn't even care to look to install them?
You can just still use it the same way @shawwn
And about the errors it's pretty straightforward it says it fails to autoinstall so than u just run the command in very restricted business environments or wathever, we could even extend this warning to literally give them the command.

[shawwn]

Because I don't want parcel doing magical things for me. I want to know exactly which deps are added to my project and whether they're installed as dev deps or real deps. And the only way to ensure that is to have control: to tell parcel "please stop trying to hold my hand."

…

On Fri, Dec 22, 2017 at 3:18 AM, Jasper De Moor ***@***.***> wrote: Why would there be a cli option to turn it off? If you install the dependencies up front then it wouldn't even care to look to install them? You can just still use it the same way @shawwn <https://github.com/shawwn> And about the errors it's pretty straightforward it says it fails to autoinstall so than u just run the command in very restricted business environments or wathever, we could even extend this warning to literally give them the command. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADo8MoQWIxsgFX7mm6v-PyNnbP24no4ks5tC3PQgaJpZM4RKpxg> .

[shawwn]

(It's similar to the reason I run with --no-cache: The cache isn't perfect, and there are many cases where it fails. Those cases cost more time than the small convenience afforded.) On Fri, Dec 22, 2017 at 3:31 AM, Shawn Presser <notifications@github.com> wrote:

…

Because I don't want parcel doing magical things for me. I want to know exactly which deps are added to my project and whether they're installed as dev deps or real deps. And the only way to ensure that is to have control: to tell parcel "please stop trying to hold my hand." On Fri, Dec 22, 2017 at 3:18 AM, Jasper De Moor ***@***.***> wrote: > Why would there be a cli option to turn it off? > If you install the dependencies up front then it wouldn't even care to > look to install them? > You can just still use it the same way @shawwn < https://github.com/shawwn> > And about the errors it's pretty straightforward it says it fails to > autoinstall so than u just run the command in very restricted business > environments or wathever, we could even extend this warning to literally > give them the command. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#376# issuecomment-353556881>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AADo8MoQWIxsgFX7mm6v- PyNnbP24no4ks5tC3PQgaJpZM4RKpxg> > . > — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADo8Pi5rXmhZavMpSQ34ZiDmCjyecD0ks5tC3b6gaJpZM4RKpxg> .

[DeMoorJasper]

'Those cases cost more time than the small convenience afforded.' Imo clearing out the cache once everytime u change a config doesn't cost much time. (Keeping in mind building speed goes down from a few secs to millisecs thanks to this...)
And same for dragging dependencies to dev-dependencies or vice versa, would save more time than it costs as far as i know.

But everyone has his own way of developing and u can't please everyone i guess...

[shawwn]

No, I'm saying caching straight-up *fails* a lot of the time. Like, a file changes, and the change never shows up. This is especially true during plugin development. If parcel had the mindset "Well, caching is so wonderful, I'm going to force everyone to use it," then I wouldn't use it because caching would interfere with my primary task: making new plugins. Ditto for this automatic dependency installation stuff. I need to see what's going on and be aware of all the changes to my projects. This is both a personal quirk and a reason my projects tend not to do unexpected things: if I know everything that's happening, I can usually minimize the number of bugs I introduce. Meaning I need to shut off the auto installer.

…

On Fri, Dec 22, 2017 at 3:52 AM, Jasper De Moor ***@***.***> wrote: 'Those cases cost more time than the small convenience afforded.' Imo clearing out the cache once everytime u change a config doesn't cost much time. (Keeping in mind building speed goes down from a few secs to millisecs thanks to this...) And same for dragging dependencies to dev-dependencies or vice versa, would save more time than it costs as far as i know. But everyone has his own way of developing and u can't please everyone i guess... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADo8DZo8IgVke_eis6yzy0VkaDoI7pwks5tC3vwgaJpZM4RKpxg> .

[rondonjon]

@shawwn is raising some very good points here IMO. I agree that this whole auto-installation thing may be nice at first sight but extends the scope of this project into fields that should be left to actual package managers. Users should be kept in control of their dependency management, especially in work environments. At the same time, the freed-up resources could be spent on key features of a better bundler.

[jdanyow]

I wonder if this will cause any issues in monorepos managed by lerna, yarn, workspaces, etc.

[devongovett]

@jdanyow it triggers the install at the nearest directory containing a package.json to the file where the import occurred.

[shawwn]

Won't this break yarn's lockfile then? Consider importing node_modules/foo/index.js Since node_modules/foo/package.json exists, it sounds like you're saying parcel will run an install at node_modules/foo Actually, maybe this will work fine – maybe yarn walks upwards until it finds a lockfile. I'm not sure.

…

On Sat, Dec 30, 2017 at 10:16 PM, Devon Govett ***@***.***> wrote: @jdanyow <https://github.com/jdanyow> it triggers the install at the nearest directory containing a package.json to the file where the import occurred. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#376 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADo8P7D7AOg8Cj-y4xVnmjx2TvqENOkks5tFwqegaJpZM4RKpxg> .

[DeMoorJasper]

@shawwn it'll prefer yarn over package.json so if a yarn file is present in the tree it'll take that one

[jamiebuilds]

Can we be sure to have an option to disable this behavior? Especially in CI environments

[davidnagli]

I’m a really big fan of this idea. I think we should go ahead and start thinking through how we can try implementing it.

Some questions we need to answer first:

1. Should we run this automagic install in production build mode (parcel build command)?
2. How are we implementing this!?? Here’s a few options:
   a) During the build process when we are resolving JS dependencies, if we come accross a dependency we can’t resolve (and it’s not a local file), then we attempt to npm install it (or use yarn if available)
   b) We run as usual, if we fail with an error, we go back and try to figure out which dependencies to install to make the error go away (dirty, but potentially necessary in some cases)
3. How do we tell apart imports of npm modules and local files?
   a) I’m pretty sure we can just check if the first characters are ./ since I think those are the required way to signify a local import, but we definitely need to research the proper way to detect this and cover any/all edge cases
4. Should we also cover installing dependencies that tools along the way need?
   a) For example, if babel needs to install a plugin/preset
   b) How would we detect this? Maybe we need to read the error output from Babel? Maybe we can figure out which installs it’ll need ahead of time by also parsing .babelrc?

The only SUPER important question here is number 2. Once we get that out of the way, we can begin implementing this feature and keep discussions about the rest of the points parallel to development.

[dpikt]

For future reference: flag to disable this behavior is --no-autoinstall

[DeMoorJasper]

@dpikt fyi this is documented in the docs: https://parceljs.org/cli.html#disable-autoinstall

[yumindeckard]

With all due respect, sir, you should try to suppress the impulse of making decisions for the users. You can give them the options and if they need it, they can just turn it on.