Import from node_modules

[joas8211]

I could not import Netlify CMS from node_modules and recognized that this is a larger issue then just this one case. This has been also noticed in #164 with THREE.js and in #42 generally.

I will use D3.js for more repoducable examples of this issue. The steps to reproduce is as follows.

1. Create bare Plenti site.

$ plenti new site plenti-import --bare
$ cd plenti-import

2. Install D3.js via NPM.

$ npm install --save d3

3. Edit layouts/content/index.svelte as follows:

<script>
    import * as d3 from 'd3';
</script>

4. Try to build and serve the site.

$ plenti serve

The error that appers in terminal is this:

open public/spa/web_modules/d3: no such file or directoryno such file or directory (File: /home/runner/work/plenti/plenti/cmd/build.go, Line: 145)

It seems that Plenti doesn't even try to look for the module in node_modules and copy it to web_modules folder automatically as was somewhat suggested in #42 (comment).

[jimafisk]

Thanks for the detailed report @joas8211! The main issue here is I wrote an internal tool called "gopack" (similar to snowpack) to handle npm modules, but it's not very smart. Currently it only copies npm packages that ship with .mjs files for esm support and converts imports to full paths that the browser understands. Gopack has other problems and I've been wanting to remove it entirely. Luckily esbuild is written in Go and has an API we can use and it has an esm transform option (https://esbuild.github.io/api/#format-esm), so I think it's about time I look into swapping these for better support.

Our problem in the three.js issue (#164) is similar, but might be slightly different, as it appears to be related to trying to convert this dependency to a component signature based on the error I get in a test repo:

Can't render htmlComponent: ReferenceError: layouts_content__assets_scene_js_createScene is not defined

For more details on the thought process of converting named imports to component signatures see: #62 (comment)

Lastly, I don't think this is a problem, but also want to note to myself that I simplified gopack not too long ago: ac3a8ed

[jimafisk]

I was playing around with this a bit and esbuild does not resolve import paths and appears that's by design. We potentially could write an esbuild plugin using the onResolve callback, just not sure if that offers a huge advantage over the regexp we're currently using in gopack. Maybe a hybrid approach using esbuild to transform node_modules that don't already support esm and gopack to fix the import paths so the browser can read them is the way to go (although I'd prefer to remove gopack completely if someone knows of an existing way to do this without nodejs). Just need to make sure I understand the different module resolver rules for npm.

[jimafisk]

From https://the-guild.dev/blog/support-nodejs-esm

To add support ESM for Node.js, you have two alternatives:

1. build your library into ESM Compatible modules with the extension ".mjs", and keep the CommonJS version with the standard ".js" extension

2. build your library into ESM Compatible modules with the extension ".js", set "type": "module", and the CommonJS version of your modules with the ".cjs" extension.

Clearly using the ".mjs" extension is the cleaner solution, and everything should work just fine.

Gopack is only accounting for approach 1 above (using .mjs extensions), but projects like d3 appear to be taking approach 2. We could go through each node_module, check if its package.json is setting "type": "module" and if so, copying the .js files to public/spa/web_modules. Doing this without shaking out unused packages is a bad idea for performance, but might the easiest way to get something working for now.

[jimafisk]

Hi @joas8211 - Following up on our discussion earlier, I've done some work around NPM imports. For now I've opted to make improvements to the internal "gopack" tool so we can continue using ESM for all dependencies. I haven't done a ton of testing, and NPM module resolution is quite complex, so I'm sure there are some cases that I haven't accounted for. I was able to get a basic D3 example up and running through:

• Demo: https://jimafisk.github.io/plenti-d3
• Repo: https://github.com/jimafisk/plenti-d3

Note: you have to upgrade your version of Plenti to v0.4.28 get this to work